Bazaar checkout is taking more than 1/2 for a 7MB vc++ project with 40 commits

Is this with pure bzr, or bzr-svn or some other foreign branch plugin?

The first thing to try would be to run 'bzr -Dhpss checkout...' which will log network operations and the time they take into bazaar.log; you can then post them into this question.

The bazaar log file is called '.bzr.log', 'bzr version' will tell you where it's located.

1. This is pure bzr. I checkout from bzr command line shell as well as bzr explorer, both resulted with same amount of time.
2. I have copy pasted the interesting part from the bzr.log. Let me know any other information is needed for this problem.

Here is the interesting part from the .bzr.log

Mon 2010-11-15 19:13:25 +0530
0.130 bazaar version: 2.1.1
0.130 bzr arguments: [u'qsubprocess', u'--bencode', u'l8:checkout55:sftp://siv@10.0.0.24/bzrrepo/brengine/Trail-Branch/lark69:C:\\Documents and Settings\\Administrator\\Desktop\\Bzr-Repo\\Release\\larke']
0.150 looking for plugins in C:/Documents and Settings/Administrator/Application Data/bazaar/2.0/plugins
0.150 looking for plugins in C:/Program Files/Bazaar/plugins
0.370 encoding stdout as osutils.get_user_encoding() 'cp1252'
0.561 bazaar version: 2.1.1
0.561 bzr arguments: [u'checkout', u'sftp://siv@10.0.0.24/bzrrepo/Trail-Branch/lark', u'C:\\Documents and Settings\\Administrator\\Desktop\\Bzr-Repo\\Release\\lark']
0.571 encoding stdout as osutils.get_user_encoding() 'cp1252'
0.711 bzr-svn: using Subversion 1.6.6 ()
0.741 falling back to default implementation
0.741 failed to load system host keys: [Errno 2] No such file or directory: 'C:\\Documents and Settings\\Administrator/.ssh/known_hosts'
[ 3784] 2010-11-15 19:13:26.486 INFO: Connected (version 2.0, client OpenSSH_5.3p1)
[ 3784] 2010-11-15 19:13:31.104 INFO: Authentication (password) successful!
[ 3784] 2010-11-15 19:13:31.714 INFO: Secsh channel 1 opened.
[ 3784] 2010-11-15 19:13:31.734 INFO: [chan 1] Opened sftp connection (server version 3)
6.299 creating repository in file:///C:/Documents%20and%20Settings/Administrator/Desktop/Bzr-Repo/Release/lark/.bzr/.
6.339 creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x0105BCB0> in file:///C:/Documents%20and%20Settings/Administrator/Desktop/Bzr-Repo/Release/lark/.bzr/
6.509 Using fetch logic to copy between CHKInventoryRepository('sftp://siv@10.0.0.24/bzrrepo/.bzr/repository/')(RepositoryFormat2a()) and CHKInventoryRepository('file:///C:/Documents%20and%20Settings/Administrator/Desktop/Bzr-Repo/Release/lark/.bzr/repository/')(RepositoryFormat2a())
6.509 fetch up to rev {administrator@cn-jass-devel-20101111063853-prwf808w1t57yc4n}
18.717 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80F70>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800
24.695 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80F70>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800
30.754 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80F70>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800
36.853 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80F70>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800
42.972 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80F70>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800
49.071 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80F70>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800
55.219 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80F70>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800
[ 2908] 2010-11-15 19:15:43.483 INFO: Discarding watch for C:\Documents and Settings\Administrator\Desktop\Bzr-Repo - it has expired
184.676 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
303.727 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
425.722 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
545.064 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
663.093 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
786.751 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
919.783 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
1044.742 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
1168.751 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
1293.730 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
1406.853 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
1534.066 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
1659.707 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80890>, 2223175, 209260276) to an LRUSizeCache failed. value 440276687 is too big to fit in a the cache with size 41943040 52428800
1676.611 opening working tree 'C:/Documents and Settings/Administrator/Desktop/Bzr-Repo/Release/lark'
1681.538 Transferred: 2760148KiB (1647.4K/s r:2760148K w:0K)
1681.538 return code 0
7030.114 opening working tree 'C:/Documents and Settings/Administrator/Desktop/Bzr-Repo/Release/lark'
[ 2908] 2010-11-15 19:41:42.686 INFO: added path to watch 'C:\Documents and Settings\Administrator\Desktop\Bzr-Repo\Release\lark' - now watching 1 locations
[ 2908] 2010-11-15 19:41:48.453 INFO: Client disconnected from pipe - closing connection
[ 2908] 2010-11-15 21:34:31.933 INFO: Discarding watch for C:\Documents and Settings\Administrator\Desktop\Bzr-Repo\Release\lark - it has expired
14442.747 opening working tree 'C:/Documents and Settings/Administrator/Desktop/Bzr-Repo/cncli'
[ 2908] 2010-11-15 21:45:14.197 INFO: added path to watch 'C:\Documents and Settings\Administrator\Desktop\Bzr-Repo\cncli' - now watching 1 locations
14443.648 opening working tree 'C:/Documents and Settings/Administrator/Desktop/Bzr-Repo/compression-devel'
[ 2908] 2010-11-15 21:45:15.017 INFO: added path to watch 'C:\Documents and Settings\Administrator\Desktop\Bzr-Repo' - now watching 1 locations
14444.099 opening working tree 'C:/Documents and Settings/Administrator/Desktop/Bzr-Repo/Vembu-cli-branch'
14446.222 opening working tree 'C:/Documents and Settings/Administrator/Desktop/Bzr-Repo/Release/lark'
[ 2908] 2010-11-15 21:56:01.634 INFO: Discarding watch for C:\Documents and Settings\Administrator\Desktop\Bzr-Repo - it has expired
[ 2908] 2010-11-15 22:56:26.496 INFO: dirwatcher stopped

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...
> 18.717 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x01A80F70>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800

^- This seems to be the key. Something got added which was very large
(51MB), which is causing us to not cache it, and then download it
repeatedly. I don't fully understand why we would need it repeatedly,
but not cache it.

...

> 1676.611 opening working tree 'C:/Documents and Settings/Administrator/Desktop/Bzr-Repo/Release/lark'
> 1681.538 Transferred: 2760148KiB (1647.4K/s r:2760148K w:0K)

^- Transferring 2.7GB is going to take a while, even at 1.6MB/s.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzi9RYACgkQJdeBCYSNAAONbgCfeX2mO4Te6nuVqTXryMwI4OM/
IEIAnAr9dSLMHDmzDAFEgqhafGH/5UWA
=jPLx
-----END PGP SIGNATURE-----

Actually the branch size is not more than 8MB. I do not understand how it is showing 2.7G? How can I get more information on this? I checked each of the commits(after which this problem started) that went in to the branch and I do not see any thing unusual there. What else can I check?

Can you just please run 'bzr info' in the branch (and paste the output), and also check the size of /bzrrepo/brengine/Trail-Branch/lark on the server? On Windows, use the file properties dialog; on Unix use 'du -s' in that directory.

cnadmin@droid:/bzrrepo/brengine/Trail-Branch/lark$ bzr info
Repository branch (format: 2a)
Location:
  shared repository: /home/cnadmin/data/bzrrepo
  repository branch: .

Related branches:
  parent branch: /home/cnadmin/data/bzrrepo/brengine/Trail-Branch

du -s of a checked out branch gives 39640.

It is a shared repo. I am not sure whether giving the size of /bzrrepo/brengine/Trail-Branch/lark would make sense - the size at that path is 44k.

Size at the /bzrrepo/.bzr is at 239 MB which is where I guess all the changes are recorded.

The shared repo has other project maintained there as well, so the 239MB could not be from this project alone.

I did a further analysis on the repo by checking out individual commits. I saw all the commits till the revision 1 has this error

6.176 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x368d710>, 1698005, 13431386) to an LRUSizeCache failed. value 51228920 is too big to fit in a the cache with size 41943040 52428800

How can I find out which file is creating the problem? Is having a binary file(Ex: suo file in vc++) as part of the commit a problem? Just to make it clear, the above mentioned error is getting printed even for the first revision. I will do some additional testing to identify the problem. Any advice on how to solve this issue?

Update: Updating the bzr to 2.2.1 did not change anything in the problem behavior.

Update:

I simply copied the repo to a Debian Squeeze system. This problem did not occur. So the problem could with the Ubuntu package. I moving my repo to a Debian Server.