Git benchmarks at OpenOffice.org wiki

Git benchmarks at OpenOffice.org wiki

[Jakub Narebski]

OpenOffice.org is looking for a new SCM (Software Configuration 
Management) tool, or at least was on Friday, 19 Jan 2007;
see: http://blogs.sun.com/GullFOSS/entry/openoffice_org_scm

One of the SCMs considered is Git. One of others is Subversion.
There is a functional git tree with the entire OOo history for testing 
purposes that can be found at: http://go-oo.org/git.

What I am concerned about is some of git benchmark results at Git page 
on OpenOffice.org wiki:
  http://wiki.services.openoffice.org/wiki/Git#Comparison
Actually it is comparison with CVS and Subversion, although most 
benchmarks are done only for git.


In 'Size of data on the server' git has CVS beat hands down: 1.3G vs 
8.5G for sources, 591M vs 1.1G for third party. I think it is similar
for Subversion. I hope that repository is fully packed: IIRC the Mozilla
CVS repository import was about 0.6GB pack file, not 1.3GB.


The problem is with 'Size of checkout': to start working in repository
one needs 1.4G (sources) and 98M (third party) for CVS checkout (it is
1.5G for sources for Subversion checkout). Ordinary for distributed SCM
you would need size of repository + size of sources (working area), 
which is 2.8G for sources and 688M for third party stuff files you can 
hack on + the history]. This makes some prefer to go centralized SCM 
route, i.e. Subversion as replacement for CVS (+ CWS, ChildWorkSpace).

What might help here is splitting repository into current (e.g. from
OOo 2.0) and historical part, and / or using shallow clone. Implementing 
partial checkouts, i.e. checking out only part of working area (and 
using 'theirs' strategy for merging not-checked-out part for merges) 
would help. Splitting repository into submodules, and submodule
support -- it depends on organization of OOo sources, would certainly 
help for third party stuff repository.

'Checkout time' (which should be renamed to 'Initial checkout time'),
in which git also loses with 130 minutes (Linux, 2MBit DSL) [from 
go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from 
go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux, 
2MBit DSL, with compression (-z 6)) for CVS, and  60 Minutes (Windows, 
34Mbit Line) for Subversion, would also be helped by the above.


What I'm really concerned about is branch switch and merging branches,
when one of the branches is an old one (e.g. unxsplash branch), which 
takes 3min (!) according to the benchmark. 13-25sec for commit is also 
bit long, but BRANCH SWITCHING which takes 3 MINUTES!? There is no 
comparison benchmark for CVS or Subversion, though...

Comparison / benchmark lacks some crucial info, like what computer was 
used (CPU, RAM, HDD), what filesystem was used, git version etc. It 
does have commands used for tests (benchmarks).

Could you confirm (or deny) those results? go-oo.org uses git 1.4.3.4;
was there some improvement or bugfix related to the speed of checkout?

-- 
Jakub Narebski
ShadeHawk on #git
Poland

Re: Git benchmarks at OpenOffice.org wiki

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> What might help here is splitting repository into current (e.g. from
> OOo 2.0) and historical part, and / or using shallow clone.

Yes, depending on where you cut off and how reasonable the
project history is.

> Implementing 
> partial checkouts, i.e. checking out only part of working area (and 
> using 'theirs' strategy for merging not-checked-out part for merges) 
> would help.

Partial checkouts, perhaps, "theirs", NO.

Consider that you are working on the tip with partial checkout.
Somebody has a bugfix that is applicable to all of ancient, old,
maintenance and current codebase.  Naturally you would want the
bugfix to be applied to ancient, merge it to old, and then
maintenance and then current (the last one is what you are
working on).

What happens if you actually pull ancient when you are partially
checked out and use "theirs"?

> Splitting repository into submodules, and submodule
> support -- it depends on organization of OOo sources, would certainly 
> help for third party stuff repository.

This is probably the most sane way.

Re: Git benchmarks at OpenOffice.org wiki

[Andy Parkins]

On Tuesday 2007 May 01, Jakub Narebski wrote:

> In 'Size of data on the server' git has CVS beat hands down: 1.3G vs
> 8.5G for sources, 591M vs 1.1G for third party. I think it is similar
> for Subversion. I hope that repository is fully packed: IIRC the Mozilla
> CVS repository import was about 0.6GB pack file, not 1.3GB.

I'm fairly sure it's not.  If so that would also affect the speed of 
operations wouldn't it?

I also doubt the subversion checkout size - subversion keeps a pristine copy 
of the HEAD file - so a subversion checkout is usually over twice the size of 
the source tree.

> takes 3min (!) according to the benchmark. 13-25sec for commit is also
> bit long

I wonder if they are measuring the time for the generation of the commit 
message or something?  Or perhaps by using "git-commit -a" is causing a check 
of the whole tree for changed files?

> Comparison / benchmark lacks some crucial info, like what computer was
> used (CPU, RAM, HDD), what filesystem was used, git version etc. It
> does have commands used for tests (benchmarks).

I'd also like to see some of the numbers for the other systems, I tried to use 
subversion with the linux kernel once and got fed up waiting for it to do 
anything.  I suspect the reason numbers aren't shown for the others is that 
they haven't finished yet :-)

> Could you confirm (or deny) those results? go-oo.org uses git 1.4.3.4;
> was there some improvement or bugfix related to the speed of checkout?

Wasn't there a recent change that made repacking after a clone unnecessary?  
That would certainly reduce the checkout size.



Andy

-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: Git benchmarks at OpenOffice.org wiki

[Julian Phillips]

On Wed, 2 May 2007, Andy Parkins wrote:

> On Tuesday 2007 May 01, Jakub Narebski wrote:
>
>> In 'Size of data on the server' git has CVS beat hands down: 1.3G vs
>> 8.5G for sources, 591M vs 1.1G for third party. I think it is similar
>> for Subversion. I hope that repository is fully packed: IIRC the Mozilla
>> CVS repository import was about 0.6GB pack file, not 1.3GB.
>
> I'm fairly sure it's not.  If so that would also affect the speed of
> operations wouldn't it?

A fully packed clone of the OOo git repo was indeed 1.3G, and the entrire 
checkout + repo was indeed 8.5G (using git 1.5.1.2).

Took about 46m to clone on a server with decent bandwith, ~5.5m user time, 
~1.5m system.

>
> I also doubt the subversion checkout size - subversion keeps a pristine copy
> of the HEAD file - so a subversion checkout is usually over twice the size of
> the source tree.
>
>> takes 3min (!) according to the benchmark. 13-25sec for commit is also
>> bit long
>
> I wonder if they are measuring the time for the generation of the commit
> message or something?  Or perhaps by using "git-commit -a" is causing a check
> of the whole tree for changed files?
>
>> Comparison / benchmark lacks some crucial info, like what computer was
>> used (CPU, RAM, HDD), what filesystem was used, git version etc. It
>> does have commands used for tests (benchmarks).
>
> I'd also like to see some of the numbers for the other systems, I tried to use
> subversion with the linux kernel once and got fed up waiting for it to do
> anything.  I suspect the reason numbers aren't shown for the others is that
> they haven't finished yet :-)
>
>> Could you confirm (or deny) those results? go-oo.org uses git 1.4.3.4;
>> was there some improvement or bugfix related to the speed of checkout?
>
> Wasn't there a recent change that made repacking after a clone unnecessary?
> That would certainly reduce the checkout size.

Not from the numbers that are quoted it won't, they are fully packed 
sizes.

-- 
Julian

  ---
"Do you have blacks, too?"

George W. Bush
To Brazilian president Fernando Cardoso
November 8, 2001
Washington, D.C.

Re: Git benchmarks at OpenOffice.org wiki

[Johannes Schindelin]

Hi,

On Tue, 1 May 2007, Jakub Narebski wrote:

> 'Checkout time' (which should be renamed to 'Initial checkout time'),
> in which git also loses with 130 minutes (Linux, 2MBit DSL) [from 
> go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from 
> go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux, 
> 2MBit DSL, with compression (-z 6)) for CVS, and  60 Minutes (Windows, 
> 34Mbit Line) for Subversion, would also be helped by the above.

FWIW I can confirm the number "100min".

Something I realized with pain is that the refs/ directory is 24MB big. 
Yep. Really. They have 3464 heads and 2639 tags. I suspect that this is 
the reason why.

Will play with it.

Ciao,
Dscho

Re: Git benchmarks at OpenOffice.org wiki

[Andy Parkins]

On Wednesday 2007 May 02, Julian Phillips wrote:

> A fully packed clone of the OOo git repo was indeed 1.3G, and the entrire
> checkout + repo was indeed 8.5G (using git 1.5.1.2).

I'm more confused now then.  I assumed the figures were accurate, but they 
cannot be:

                               CVS      git      SVN
Size of data on the server     8.5G     1.3G     n/a
Size of checkout               1.4G     2.8G     1.5G

I don't doubt the 1.3G on the server - and assume that is fully packed.  The 
checkout sizes are suspicious though.  Is that 2.8G packed?
 - If it is, then we can deduce that this is a repo+source size, since the
   server is packed size+0 therefore the size of the source tree is
    2.8G - 1.3G = 1.5G
   In which case the other figures are wrong:
    - CVS checkout is 1.4G - impossible, the source tree is 1.5G. And where is
      the overhead of the CVS directories which would make it more than 1.5G?
    - SVN checkout overhead is always _at least_ the size of the source tree 
      because it keeps a pristine copy of HEAD.  If the source tree is 1.5G,
      then this figure should be at least 3G.
 - If it is not, then we're back to "I don't believe that git was packed"

Something smells fishy here - either the source tree size is included in some, 
but not in others or the git repository wasn't packed.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: Git benchmarks at OpenOffice.org wiki

[Jakub Narebski]

Hi!

On 5/2/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Tue, 1 May 2007, Jakub Narebski wrote:
>
> > 'Checkout time' (which should be renamed to 'Initial checkout time'),
> > in which git also loses with 130 minutes (Linux, 2MBit DSL) [from
> > go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from
> > go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux,
> > 2MBit DSL, with compression (-z 6)) for CVS, and  60 Minutes (Windows,
> > 34Mbit Line) for Subversion, would also be helped by the above.
>
> FWIW I can confirm the number "100min".
>
> Something I realized with pain is that the refs/ directory is 24MB big.
> Yep. Really. They have 3464 heads and 2639 tags. I suspect that this is
> the reason why.

Then packed refs would certainly help with speed and a bit with size.

-- 
Jakub Narebski

Re: Git benchmarks at OpenOffice.org wiki

[Jan Holesovsky]

Hi Jakub,

On Tuesday 01 May 2007 23:46, Jakub Narebski wrote:

> OpenOffice.org is looking for a new SCM (Software Configuration
> Management) tool, or at least was on Friday, 19 Jan 2007;
> see: http://blogs.sun.com/GullFOSS/entry/openoffice_org_scm
>
> One of the SCMs considered is Git. One of others is Subversion.
> There is a functional git tree with the entire OOo history for testing
> purposes that can be found at: http://go-oo.org/git.
>
> What I am concerned about is some of git benchmark results at Git page
> on OpenOffice.org wiki:
>   http://wiki.services.openoffice.org/wiki/Git#Comparison
> Actually it is comparison with CVS and Subversion, although most
> benchmarks are done only for git.

I did the git numbers, so if they are wrong - blame me :-)  I am also curious
about the SVN numbers, because the SVN conversion [from my point of view]
cheats a lot.  From what I know, it does not contain the historical branches
(yes, the >3000 of them that are in the git tree), and if I understood that
correctly, instead of history in the branches, they commit just
'integration commits' [one commit for all the changes in the branch] which
breaks 'svn blame' completely.

Unfortunately, I did not have a chance to try the SVN tree yet to see it
myself to prove this true or false :-(

> In 'Size of data on the server' git has CVS beat hands down: 1.3G vs
> 8.5G for sources, 591M vs 1.1G for third party. I think it is similar
> for Subversion. I hope that repository is fully packed: IIRC the Mozilla
> CVS repository import was about 0.6GB pack file, not 1.3GB.
>
> The problem is with 'Size of checkout': to start working in repository
> one needs 1.4G (sources) and 98M (third party) for CVS checkout (it is
> 1.5G for sources for Subversion checkout). Ordinary for distributed SCM
> you would need size of repository + size of sources (working area),
> which is 2.8G for sources and 688M for third party stuff files you can
> hack on + the history]. This makes some prefer to go centralized SCM
> route, i.e. Subversion as replacement for CVS (+ CWS, ChildWorkSpace).

Considering the size OOo needs for build (>8G without languages),
the ~1.4G overhead for history is very well bearable.  I am surprised about
the 100M overhead for SVN as well - from my experience it is usually about
the size of the project itself; but maybe they improved something in SVN
in the meantime.

> What might help here is splitting repository into current (e.g. from
> OOo 2.0) and historical part,

No, I don't want this ;-)

> and / or using shallow clone. Implementing 
> partial checkouts, i.e. checking out only part of working area (and
> using 'theirs' strategy for merging not-checked-out part for merges)
> would help. Splitting repository into submodules, and submodule
> support -- it depends on organization of OOo sources, would certainly
> help for third party stuff repository.

We should better split the OOo sources; it's a process that already started
[UNO runtime environment vs. OOo without URE], and I proposed some more
changes already.

> 'Checkout time' (which should be renamed to 'Initial checkout time'),
> in which git also loses with 130 minutes (Linux, 2MBit DSL) [from
> go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from
> go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux,
> 2MBit DSL, with compression (-z 6)) for CVS, and  60 Minutes (Windows,
> 34Mbit Line) for Subversion, would also be helped by the above.

Good point, and I already changed the page in the morning.  I also added the
checkout time that I got over a fast line [it was 44min].

> What I'm really concerned about is branch switch and merging branches,
> when one of the branches is an old one (e.g. unxsplash branch), which
> takes 3min (!) according to the benchmark. 13-25sec for commit is also
> bit long, but BRANCH SWITCHING which takes 3 MINUTES!? There is no
> comparison benchmark for CVS or Subversion, though...

I am really curious about the SVN tree.  As I said, I did not see it yet.
There is just some info about it here:
http://wiki.services.openoffice.org/wiki/SVNMigration, but I cannot check it
now, the Wiki is down :-(

> Comparison / benchmark lacks some crucial info, like what computer was
> used (CPU, RAM, HDD), what filesystem was used, git version etc. It
> does have commands used for tests (benchmarks).

For the git tests, it was:

CPU: AMD Athlon(tm) 64 Processor 3200+

RAM: 1G RAM

Disk (info from bonnie):
              ---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec %CPU
one    1*2000 37819 77.6 44296 16.8 16982  5.1 35203 63.9 45915  6.6  152.4  0.4

Filesystem: ext3

> Could you confirm (or deny) those results? go-oo.org uses git 1.4.3.4;
> was there some improvement or bugfix related to the speed of checkout?

Regards,
Jan

Re: Git benchmarks at OpenOffice.org wiki

[Julian Phillips]

On Wed, 2 May 2007, Andy Parkins wrote:

> On Wednesday 2007 May 02, Julian Phillips wrote:
>
>> A fully packed clone of the OOo git repo was indeed 1.3G, and the entrire
>> checkout + repo was indeed 8.5G (using git 1.5.1.2).


oops, meant 2.7G not 8.5G there ... sorry, was working from memory.

jp3@electron: ooo(unxsplash)>du -sh .git
1.3G    .git
jp3@electron: ooo(unxsplash)>du -sh .
2.7G    .
jp3@electron: ooo(unxsplash)>ls .git/objects/
info  pack


> I'm more confused now then.  I assumed the figures were accurate, but they
> cannot be:
>
>                               CVS      git      SVN
> Size of data on the server     8.5G     1.3G     n/a
> Size of checkout               1.4G     2.8G     1.5G
>
> I don't doubt the 1.3G on the server - and assume that is fully packed.  The
> checkout sizes are suspicious though.  Is that 2.8G packed?
> - If it is, then we can deduce that this is a repo+source size, since the
>   server is packed size+0 therefore the size of the source tree is
>    2.8G - 1.3G = 1.5G

the difference between 2.7G and 2.8G may be due to filesystem difference?

>   In which case the other figures are wrong:
>    - CVS checkout is 1.4G - impossible, the source tree is 1.5G. And where is
>      the overhead of the CVS directories which would make it more than 1.5G?
>    - SVN checkout overhead is always _at least_ the size of the source tree
>      because it keeps a pristine copy of HEAD.  If the source tree is 1.5G,
>      then this figure should be at least 3G.

I was wondering about the subversion figures too ...

> - If it is not, then we're back to "I don't believe that git was packed"
>
> Something smells fishy here - either the source tree size is included in some,
> but not in others or the git repository wasn't packed.

1.3G is the packed size ...

jp3@electron: ooo(unxsplash)>ls -sh .git/objects/pack/
total 1.3G
  37M pack-87efcac9bcb117328e8a1b0c1b42c88c3603c5b7.idx
1.2G pack-87efcac9bcb117328e8a1b0c1b42c88c3603c5b7.pack

-- 
Julian

  ---
To err is humor.

Re: Git benchmarks at OpenOffice.org wiki

[Johannes Schindelin]

Hi,

On Wed, 2 May 2007, Jan Holesovsky wrote:

> On Tuesday 01 May 2007 23:46, Jakub Narebski wrote:
> 
> > What I am concerned about is some of git benchmark results at Git page
> > on OpenOffice.org wiki:
> >   http://wiki.services.openoffice.org/wiki/Git#Comparison
> > Actually it is comparison with CVS and Subversion, although most
> > benchmarks are done only for git.
> 
> I did the git numbers, so if they are wrong - blame me :-)

Good to have you here!

> > 'Checkout time' (which should be renamed to 'Initial checkout time'),
> > in which git also loses with 130 minutes (Linux, 2MBit DSL) [from
> > go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from
> > go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux,
> > 2MBit DSL, with compression (-z 6)) for CVS, and  60 Minutes (Windows,
> > 34Mbit Line) for Subversion, would also be helped by the above.
> 
> Good point, and I already changed the page in the morning.  I also added the
> checkout time that I got over a fast line [it was 44min].

It took me longer here, but the reason might be that my "local" repository 
is on NFS, due to quota on the machine.

> > What I'm really concerned about is branch switch and merging branches,
> > when one of the branches is an old one (e.g. unxsplash branch), which
> > takes 3min (!) according to the benchmark. 13-25sec for commit is also
> > bit long, but BRANCH SWITCHING which takes 3 MINUTES!? There is no
> > comparison benchmark for CVS or Subversion, though...

I imagine that might be related to the vast amount of remote branches. 
IIRC we do not pack them with git-gc, and ext3 is not that good with big 
directories (remember: 3464 branches!).

Maybe oprofile knows a bit more where the hotspots are.

Ciao,
Dscho

Re: Git benchmarks at OpenOffice.org wiki

[Jan Holesovsky]

Hi Andy,

On Wednesday 02 May 2007 12:58, Andy Parkins wrote:

> On Wednesday 2007 May 02, Julian Phillips wrote:
> > A fully packed clone of the OOo git repo was indeed 1.3G, and the entrire
> > checkout + repo was indeed 8.5G (using git 1.5.1.2).
>
> I'm more confused now then.  I assumed the figures were accurate, but they
> cannot be:
>
>                                CVS      git      SVN
> Size of data on the server     8.5G     1.3G     n/a
> Size of checkout               1.4G     2.8G     1.5G
>
> I don't doubt the 1.3G on the server - and assume that is fully packed. 
> The checkout sizes are suspicious though.  Is that 2.8G packed?
>  - If it is, then we can deduce that this is a repo+source size, since the
>    server is packed size+0 therefore the size of the source tree is
>     2.8G - 1.3G = 1.5G
>    In which case the other figures are wrong:
>     - CVS checkout is 1.4G - impossible, the source tree is 1.5G. And where
> is the overhead of the CVS directories which would make it more than 1.5G?

Unfortunately I don't have the _exact_ numbers here any more so I cannot prove 
it ;-) - but this is a rounding problem [CVS checkout is slightly more than 
1.4G].  Similarly, overhead of of CVS directories is 0 when we count in 
gigabytes.

> - SVN checkout overhead is always _at least_ the size of the source tree
> because it keeps a pristine copy of HEAD.  If the source tree is 1.5G, then
> this figure should be at least 3G.

Yes, this surprises me as well.  I've heard about some improvements in the 
recent SVN, but 0.1M sounds very small.

>  - If it is not, then we're back to "I don't believe that git was packed"

It was, IIRC with 'git-repack -a -d -f'.

> Something smells fishy here - either the source tree size is included in
> some, but not in others or the git repository wasn't packed.

As I wrote, I am looking forward to seeing the SVN tree myself for further 
testing.

Regards,
Jan

Re: Git benchmarks at OpenOffice.org wiki

[Jan Holesovsky]

Hi Johannes,

On Wednesday 02 May 2007 12:24, Johannes Schindelin wrote:

> On Tue, 1 May 2007, Jakub Narebski wrote:
> > 'Checkout time' (which should be renamed to 'Initial checkout time'),
> > in which git also loses with 130 minutes (Linux, 2MBit DSL) [from
> > go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from
> > go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux,
> > 2MBit DSL, with compression (-z 6)) for CVS, and  60 Minutes (Windows,
> > 34Mbit Line) for Subversion, would also be helped by the above.
>
> FWIW I can confirm the number "100min".
>
> Something I realized with pain is that the refs/ directory is 24MB big.
> Yep. Really. They have 3464 heads and 2639 tags. I suspect that this is
> the reason why.

I should probably produce even a tree where would be the merged branches 
deleted, right...

Regards,
Jan

Re: Git benchmarks at OpenOffice.org wiki

[Johannes Schindelin]

Hi,

On Wed, 2 May 2007, Jakub Narebski wrote:

> On 5/2/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > On Tue, 1 May 2007, Jakub Narebski wrote:
> > 
> > > 'Checkout time' (which should be renamed to 'Initial checkout time'),
> > > in which git also loses with 130 minutes (Linux, 2MBit DSL) [from
> > > go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from
> > > go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux,
> > > 2MBit DSL, with compression (-z 6)) for CVS, and  60 Minutes (Windows,
> > > 34Mbit Line) for Subversion, would also be helped by the above.
> > 
> > FWIW I can confirm the number "100min".
> > 
> > Something I realized with pain is that the refs/ directory is 24MB big.
> > Yep. Really. They have 3464 heads and 2639 tags. I suspect that this is
> > the reason why.
> 
> Then packed refs would certainly help with speed and a bit with size.

Indeed for size: du -h reported 11 megabyte for the tags directory. After 
packing them, a 265 kilobyte file is left. Of course, git-show-ref now 
becomes a speed demon again.

Ciao,
Dscho

Re: Git benchmarks at OpenOffice.org wiki

[Andy Parkins]

On Wednesday 2007 May 02, Julian Phillips wrote:

> oops, meant 2.7G not 8.5G there ... sorry, was working from memory.

Not a problem.  That fixes one ambiguity:
  2.7G - 1.3G = 1.4G 
Which is the same as the CVS checkout size.  Both the CVS and git figures are 
now consistent:
                                CVS      git      SVN
Size of data on the server     8.5G     1.3G      n/a
Size of checkout               1.4G     2.7G     1.5G
Overhead in checkout             0G     1.3G     0.1G

So that only leaves the subversion number as being suspicious.

> the difference between 2.7G and 2.8G may be due to filesystem difference?

Could be I suppose.  Although, in that case CVS should have suffered the same 
because the disparity was in the source tree size.  Packed git shouldn't 
suffer any filesystem overhead (relatively) because the majority of it's 
space is taken up by one large pack file (which of course only suffers file 
system overhead once).

> I was wondering about the subversion figures too ...

I've just checked using subversion 1.4.2 and the .svn/text-base/*.svn-base 
files are all uncompressed copies of the working tree files.  Doesn't look 
like anythings changed in the pristine copy department.

> jp3@electron: ooo(unxsplash)>ls -sh .git/objects/pack/
> total 1.3G
>   37M pack-87efcac9bcb117328e8a1b0c1b42c88c3603c5b7.idx
> 1.2G pack-87efcac9bcb117328e8a1b0c1b42c88c3603c5b7.pack

Thanks for your help.  It's all looking more consistent to me now; only the 
subversion figures seem wrong.

I wonder when they're going to get timing numbers for the non-git systems.  
That must be a monster of a repository for them to deal with.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: Git benchmarks at OpenOffice.org wiki

[Andy Parkins]

On Wednesday 2007 May 02, Jan Holesovsky wrote:

> Unfortunately I don't have the _exact_ numbers here any more so I cannot
> prove it ;-) - but this is a rounding problem [CVS checkout is slightly
> more than 1.4G].  Similarly, overhead of of CVS directories is 0 when we
> count in gigabytes.

0.1G would have been an awfully big rounding error.  Regardless, Julian has 
put me right on that - the git checked out size was actually 2.7GB - this 
then lines up with the CVS figures.

> > - SVN checkout overhead is always _at least_ the size of the source tree
> > because it keeps a pristine copy of HEAD.  If the source tree is 1.5G,
> > then this figure should be at least 3G.
>
> Yes, this surprises me as well.  I've heard about some improvements in the
> recent SVN, but 0.1M sounds very small.

Very much so - I've tried with a 1.4.2 and my own small repository and the 
pristine copies are stored uncompressed as always.  0.1G now sounds plain 
wrong.  Maybe there are some switches I should be using to svn checkout.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: Git benchmarks at OpenOffice.org wiki

[Petr Baudis]

On Wed, May 02, 2007 at 04:24:24PM CEST, Jan Holesovsky wrote:
> > What might help here is splitting repository into current (e.g. from
> > OOo 2.0) and historical part,
> 
> No, I don't want this ;-)

Are you sure? Using the graft mechanism, Git can make this very easy and
almost transparent for the user - when he clones he gets no history but
he can use say some simple vendor-provided script to download the
historical packfile and graft it to the 'current' tree. After that, the
graft acts completely transparently and it 'seems' like the history
goes on continuously from OOo prehistory up to the latest commit.

Besides, in case you discover a year later that the conversion was
broken in some places etc., you can just fix this, re-run the conversion
and simply regraft your history to point at the 'new' historical commit,
without affecting your current development and commit ids at all. For
this reason alone, I'd seriously consider grafting history separately
when migrating any non-trivial project from other SCM to Git.

Then again, due to the sheer tree sizes etc., I'm not sure how much
would throwing the history away actually reduce the packfile size.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
		-- Samuel Beckett

Re: Git benchmarks at OpenOffice.org wiki

[Johannes Schindelin]

Hi,

On Wed, 2 May 2007, Jan Holesovsky wrote:

> On Wednesday 02 May 2007 12:24, Johannes Schindelin wrote:
> 
> > On Tue, 1 May 2007, Jakub Narebski wrote:
> > > 'Checkout time' (which should be renamed to 'Initial checkout time'),
> > > in which git also loses with 130 minutes (Linux, 2MBit DSL) [from
> > > go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from
> > > go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux,
> > > 2MBit DSL, with compression (-z 6)) for CVS, and  60 Minutes (Windows,
> > > 34Mbit Line) for Subversion, would also be helped by the above.
> >
> > FWIW I can confirm the number "100min".
> >
> > Something I realized with pain is that the refs/ directory is 24MB big.
> > Yep. Really. They have 3464 heads and 2639 tags. I suspect that this is
> > the reason why.
> 
> I should probably produce even a tree where would be the merged branches 
> deleted, right...

FWIW, I just deleted all branches except for one, packed the tags, and did 
a local clone (via NFS, urgh) _without_ checking the files out.

Now it takes 25 minutes vs 50 minutes before (in an _extremely_ 
unscientific test, mind you).

So, this issue is worth looking at, probably.

Ciao,
Dscho

Re: Git benchmarks at OpenOffice.org wiki

[Jan Holesovsky]

Hi Pasky,

On Wednesday 02 May 2007 18:15, Petr Baudis wrote:

> On Wed, May 02, 2007 at 04:24:24PM CEST, Jan Holesovsky wrote:
> > > What might help here is splitting repository into current (e.g. from
> > > OOo 2.0) and historical part,
> >
> > No, I don't want this ;-)
>
> Are you sure? Using the graft mechanism, Git can make this very easy and
> almost transparent for the user - when he clones he gets no history but
> he can use say some simple vendor-provided script to download the
> historical packfile and graft it to the 'current' tree. After that, the
> graft acts completely transparently and it 'seems' like the history
> goes on continuously from OOo prehistory up to the latest commit.

Interesting, I did not know that it is possible to do it so that it appears 
transparently; this would be indeed a tremendous win - we could start nearly 
from scratch ;-)

Please - where could I find more info?  Like what does the script have to do, 
etc.

> Besides, in case you discover a year later that the conversion was
> broken in some places etc., you can just fix this, re-run the conversion
> and simply regraft your history to point at the 'new' historical commit,
> without affecting your current development and commit ids at all. For
> this reason alone, I'd seriously consider grafting history separately
> when migrating any non-trivial project from other SCM to Git.
>
> Then again, due to the sheer tree sizes etc., I'm not sure how much
> would throwing the history away actually reduce the packfile size.

Thanks a lot,
Jan

Re: Git benchmarks at OpenOffice.org wiki

[Petr Baudis]

  Hi,

On Wed, May 02, 2007 at 06:27:51PM CEST, Jan Holesovsky wrote:
> On Wednesday 02 May 2007 18:15, Petr Baudis wrote:
> 
> > On Wed, May 02, 2007 at 04:24:24PM CEST, Jan Holesovsky wrote:
> > > > What might help here is splitting repository into current (e.g. from
> > > > OOo 2.0) and historical part,
> > >
> > > No, I don't want this ;-)
> >
> > Are you sure? Using the graft mechanism, Git can make this very easy and
> > almost transparent for the user - when he clones he gets no history but
> > he can use say some simple vendor-provided script to download the
> > historical packfile and graft it to the 'current' tree. After that, the
> > graft acts completely transparently and it 'seems' like the history
> > goes on continuously from OOo prehistory up to the latest commit.
> 
> Interesting, I did not know that it is possible to do it so that it appears 
> transparently; this would be indeed a tremendous win - we could start nearly 
> from scratch ;-)
> 
> Please - where could I find more info?  Like what does the script have to do, 
> etc.

  you can see an example script at

	http://repo.or.cz/w/elinks.git?a=blob;f=contrib/grafthistory.sh

and I have tried vainly few times to get a similar script to the kernel
too

	http://lists.zerezo.com/linux-kernel/msg6599002.html

that can use both wget and curl and will also download tag refs for the
history.

  The format of the grafts file itself (.git/info/grafts) is pretty
simple (just one-graft-per-line where you first say the commit id and
then the parent commit(s) to be drafted onto it), please see
Documentation/repository-layout.txt for details.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
		-- Samuel Beckett

Re: Git benchmarks at OpenOffice.org wiki

[Petr Baudis]

  Hi,

On Wed, May 02, 2007 at 06:37:15PM CEST, Petr Baudis wrote:
>   you can see an example script at
> 
> 	http://repo.or.cz/w/elinks.git?a=blob;f=contrib/grafthistory.sh

  by the way, this script goes back to very ancient Git times, maybe by
now git-fetch could be convinced to do all the hard work for you.
Actually, maybe just something (totally untested) like

	git remote add -f historical {http,git}://historical_repository_url
	cat <<EOF >>.git/info/grafts
	... the graft specs go here ...
	EOF

might work prefectly fine nowadays that git keeps the remote refs in a
separate namespace tidily. This way you don't have to care about all the
manual wgetting, ls-remote magic etc. The downside is that this is
available only since git-1.5.0 (Debian stable has older version; maybe
even newer git version is required, I'm not sure).

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
		-- Samuel Beckett

Re: Git benchmarks at OpenOffice.org wiki

[Julian Phillips]

On Wed, 2 May 2007, Andy Parkins wrote:

> On Wednesday 2007 May 02, Julian Phillips wrote:
>
>> oops, meant 2.7G not 8.5G there ... sorry, was working from memory.
>
> Not a problem.  That fixes one ambiguity:
>  2.7G - 1.3G = 1.4G
> Which is the same as the CVS checkout size.  Both the CVS and git figures are
> now consistent:
>                                CVS      git      SVN
> Size of data on the server     8.5G     1.3G      n/a
> Size of checkout               1.4G     2.7G     1.5G
> Overhead in checkout             0G     1.3G     0.1G

Except that it's 2.8G, I forgot I had switched branch.  I switched to the 
unxsplash branch, and _that_ is 2.7G checked out.

(du -s .) - (du -s .git) = 1.49G

-- 
Julian

  ---
"Consider a spherical bear, in simple harmonic motion..."
 		-- Professor in the UCB physics department

Re: Git benchmarks at OpenOffice.org wiki

[Junio C Hamano]

Andy Parkins <andyparkins@gmail.com> writes:

>     - SVN checkout overhead is always _at least_ the size of the source tree 
>       because it keeps a pristine copy of HEAD.  If the source tree is 1.5G,
>       then this figure should be at least 3G.

Could it be that there is a mode in svn checkout that allows
pristine to be hardlinked to the working tree copies?  It
requires an editor that can be told to break hardlinks when
making modifications (and the user obviously needs to know about
it), but to save 1.5G it is worth it and if _I_ were hacking on
SVN that would be an obvious optimization to add.

Re: Git benchmarks at OpenOffice.org wiki

[Jakub Narebski]

Jan Holesovsky wrote:
> On Tuesday 01 May 2007 23:46, Jakub Narebski wrote:
> 
>> What I am concerned about is some of git benchmark results at Git page
>> on OpenOffice.org wiki:
>>   http://wiki.services.openoffice.org/wiki/Git#Comparison

>> The problem is with 'Size of checkout': to start working in repository
>> one needs 1.4G (sources) and 98M (third party) for CVS checkout (it is
>> 1.5G for sources for Subversion checkout). Ordinary for distributed SCM
>> you would need size of repository + size of sources (working area),
>> which is 2.8G for sources and 688M for third party stuff files you can
>> hack on + the history]. This makes some prefer to go centralized SCM
>> route, i.e. Subversion as replacement for CVS (+ CWS, ChildWorkSpace).
> 
> Considering the size OOo needs for build (>8G without languages),
> the ~1.4G overhead for history is very well bearable.  I am surprised about
> the 100M overhead for SVN as well - from my experience it is usually about
> the size of the project itself; but maybe they improved something in SVN
> in the meantime.

I think the supposition that SVN uses hardlinks for pristine copy
of sources (HEAD version) seems probable; then there it is 100M overhead
plus size of changed files, and of course this tricks works only on
filesystems which support hardlinks, and assumes either hardlinks being
COW-links (copy-on-write) or editor behaving.
 
>> What might help here is splitting repository into current (e.g. from
>> OOo 2.0) and historical part,
> 
> No, I don't want this ;-)

I forgot to add there is possible to graft historical repository to the
current work repository, resulting in full history available. For example
Linux kernel repository has backported from BK historical repository, and
there is grafts file which connect those two repositories.

>> and / or using shallow clone. 

git-clone(1):

--depth <depth>::
        Create a 'shallow' clone with a history truncated to the
        specified number of revs.  A shallow repository has
        number of limitations (you cannot clone or fetch from
        it, nor push from nor into it), but is adequate if you
        want to only look at near the tip of a large project
        with a long history, and would want to send in a fixes
        as patches.

It is possible that those limitations will be lifted in the future
(if possible), so there is alternate possibility to reduce needed
disk space for git checkout. But certainly this is not for everybody.

>> Implementing  
>> partial checkouts, i.e. checking out only part of working area (...)

The problem with implementing this feature (you can do partial checkout
using low level commands, but this feature is not implemented [yet?]
per se) is with doing merge on part which is not checked out. Might
not be a problem for OOo; but this might be also not needed for OOo.
Sometimes submodules are better, sometimes partial checkout is the
only way: see below.

>> Splitting repository into submodules, and submodule 
>> support -- it depends on organization of OOo sources, would certainly
>> help for third party stuff repository.
> 
> We should better split the OOo sources; it's a process that already started
> [UNO runtime environment vs. OOo without URE], and I proposed some more
> changes already.

In my opinion each submodule should be able to compile and test by
itself. You can go X.Org route with splitting sources into modules...
or you can make use of the new submodules support (currently plumbing
level, i.e. low level commands), aka. gitlinks.

The submodules support makes it possible to split sources into
independent modules (parts), which can be developed independently,
and which you can download (clone, fetch) or not, while making it
possible to bind it all together into one superproject.

See (somewhat not up to date) http://git.or.cz/gitwiki/SubprojectSupport
page on git wiki.

>> What I'm really concerned about is branch switch and merging branches,
>> when one of the branches is an old one (e.g. unxsplash branch), which
>> takes 3min (!) according to the benchmark. 13-25sec for commit is also
>> bit long, but BRANCH SWITCHING which takes 3 MINUTES!? There is no
>> comparison benchmark for CVS or Subversion, though...

By the way, the time to switch branch should be proportional to number
of changed files, which you can get with "git diff --summary unxsplash
HEAD". Or to be more realistic to checkout some old version
(some old tag), as usually branches which got merged in are deleted
(or even never got published). For example when bisecting some bug:
Subversion doesn't have bisect, does it?

I wonder if running "git pack-refs" would help this benchmark...

-- 
Jakub Narebski
Poland

Re: Git benchmarks at OpenOffice.org wiki

[Florian Weimer]

* Jakub Narebski:

> The problem is with 'Size of checkout': to start working in repository
> one needs 1.4G (sources) and 98M (third party) for CVS checkout (it is
> 1.5G for sources for Subversion checkout).

The text bases for Subversion really should take another 1.4 GiB.
As a result, Subversion should be closer to 3 GiB.

> What might help here is splitting repository into current (e.g. from
> OOo 2.0) and historical part, and / or using shallow clone.

You could also split along project boundaries, but this is probably
too political.

> What I'm really concerned about is branch switch and merging branches,
> when one of the branches is an old one (e.g. unxsplash branch), which 
> takes 3min (!) according to the benchmark. 13-25sec for commit is also 
> bit long, but BRANCH SWITCHING which takes 3 MINUTES!?

IIRC, GIT accesses every file in the tree, not just the ones that need
updating.  How many files were actually updated when you changed
branches in your experiment?

Re: Git benchmarks at OpenOffice.org wiki

[Johannes Schindelin]

Hi,

On Thu, 3 May 2007, Florian Weimer wrote:

> * Jakub Narebski:
> 
> > What I'm really concerned about is branch switch and merging branches, 
> > when one of the branches is an old one (e.g. unxsplash branch), which 
> > takes 3min (!) according to the benchmark. 13-25sec for commit is also 
> > bit long, but BRANCH SWITCHING which takes 3 MINUTES!?
> 
> IIRC, GIT accesses every file in the tree, not just the ones that need
> updating.  How many files were actually updated when you changed
> branches in your experiment?

No. Git does not access every file, but rather all stats. That is a huge 
difference. And it should not take _that_ long for ~64000 files. Granted, 
it will cause a substantial delay, but not in the range of minutes.

Ciao,
Dscho

Re: Git benchmarks at OpenOffice.org wiki

[Robin Rosenberg]

torsdag 03 maj 2007 skrev Johannes Schindelin:
> Hi,
> 
> On Thu, 3 May 2007, Florian Weimer wrote:
> 
> > * Jakub Narebski:
> > 
> > > What I'm really concerned about is branch switch and merging branches, 
> > > when one of the branches is an old one (e.g. unxsplash branch), which 
> > > takes 3min (!) according to the benchmark. 13-25sec for commit is also 
> > > bit long, but BRANCH SWITCHING which takes 3 MINUTES!?
> > 
> > IIRC, GIT accesses every file in the tree, not just the ones that need
> > updating.  How many files were actually updated when you changed
> > branches in your experiment?
> 
> No. Git does not access every file, but rather all stats. That is a huge 
> difference. And it should not take _that_ long for ~64000 files. Granted, 
> it will cause a substantial delay, but not in the range of minutes.

It's worse... On my laptop the switch took ~ten minutes, not three. 
A diff --stat takes over six minutes!! For reference, dd:in the pack 
file with my disk takes ~50 seconds.

The reason is simple. I have a lousy one gigabyte RAM only, while 
git wants 1.7GB virtual to do the diff-stat.  and 800 MB resident. The swap is having a party,

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  2 1861632  14108    428 126816   70  347   605   647  594 1041 11  2 74 13
 0  2 1861204  12096    420 125724 3096    8  3096    24  625 1171  5  1  0 94
 0  2 1860896  18972    404 115836 3524  292  3524   292  671 1474  7  4  0 89
 0  2 1860820  18668    364 113736 3556  784  3556   784  669 1384  7  5  0 88
 0  3 1860420  19692    300 109904 3008  180  3156   180  684 1325  8  5  0 87
 0  3 1860184  18560    300 108596 3316  232  3396   232  643 1246  8  4  0 88
 0  2 1859856  21808    292 103744 2108   32  2356    32  637 1319  9  1  0 90

-- robin

Re: Git benchmarks at OpenOffice.org wiki

[Martin Langhoff]

[resend - correcting a couple of typos and addressing git@vger
correctly - apologies]

On 5/3/07, Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
> The reason is simple. I have a lousy one gigabyte RAM only, while
> git wants 1.7GB virtual to do the diff-stat.  and 800 MB resident. The swap is having a party,

That is true, unfortunately. git will fly if it can fit its working
set plus the kernel stat cache for your working tree in memory. And
the underlying assumption is that for large trees you'll have gobs of
RAM. If things don't fit, it does get rather slow...

But... just to put things in perspective, how long does it take to
*compile* that checkout on that same laptop. I remember reading
instructions to the tune of "don't even try to compile this with less
than 4GB RAM, a couple of CPUs and 12hs". Those were for the OSX build
IIRC.

Ah - it's moved to the general instructions: "Building OOo takes some
time (approx 10-12 hours on standard desktop PC) ":
http://wiki.services.openoffice.org/wiki/Building_OpenOffice.org#Starting_the_real_build

So I don't think anyone working on projects the size of the kernel or
OO.org is going to be happy with 1GB RAM.

cheers


m

Re: [tools-dev] Re: Git benchmarks at OpenOffice.org wiki

[Jan Holesovsky]

Hi Jakub,

On Thursday 03 May 2007 01:30, Jakub Narebski wrote:

> >> What might help here is splitting repository into current (e.g. from
> >> OOo 2.0) and historical part,
> >
> > No, I don't want this ;-)
>
> I forgot to add there is possible to graft historical repository to the
> current work repository, resulting in full history available. For example
> Linux kernel repository has backported from BK historical repository, and
> there is grafts file which connect those two repositories.

Yes, grafting sounds really very promising! - I did not know about it.

> >> and / or using shallow clone.
>
> git-clone(1):
>
> --depth <depth>::
>         Create a 'shallow' clone with a history truncated to the
>         specified number of revs.  A shallow repository has
>         number of limitations (you cannot clone or fetch from
>         it, nor push from nor into it), but is adequate if you
>         want to only look at near the tip of a large project
>         with a long history, and would want to send in a fixes
>         as patches.
>
> It is possible that those limitations will be lifted in the future
> (if possible), so there is alternate possibility to reduce needed
> disk space for git checkout. But certainly this is not for everybody.

It's probably too tight limitation for regular developers; for random hackers 
contributing a patch or two it could be a choice, right.

> > We should better split the OOo sources; it's a process that already
> > started [UNO runtime environment vs. OOo without URE], and I proposed
> > some more changes already.
>
> In my opinion each submodule should be able to compile and test by
> itself. You can go X.Org route with splitting sources into modules...

Indeed, this is the case of URE - it is supposed to run by separately & be 
used even by other projects than OOo.

> or you can make use of the new submodules support (currently plumbing
> level, i.e. low level commands), aka. gitlinks.

And this would be interesting for the translations, I guess...

> The submodules support makes it possible to split sources into
> independent modules (parts), which can be developed independently,
> and which you can download (clone, fetch) or not, while making it
> possible to bind it all together into one superproject.
>
> See (somewhat not up to date) http://git.or.cz/gitwiki/SubprojectSupport
> page on git wiki.

... but will have a better look; thanks for the pointer!

> Subversion doesn't have bisect, does it?

>From what I know, it does not.

Thank you and others for all the input!

Last question: what is the status of the Win32 support?  I got a full clone 
using the Cygwin git 1.5.0 [it took 6hrs 20min on a Xen virtual machine; I 
have to try it with real hardware], MinGW version did not work for me too 
well :-(  Are there any other options?  Is 
http://git.or.cz/gitwiki/WindowsInstall up-to-date?

Regards,
Jan

Re: [tools-dev] Re: Git benchmarks at OpenOffice.org wiki

[Alex Riesen]

On 5/3/07, Jan Holesovsky <kendy@suse.cz> wrote:
> Last question: what is the status of the Win32 support?

It kind of works. Performance is horrible, but still better
than almost everything comparable (and there isn't anything
comparable). You have to be very careful not to push it
(them, actually: cygwin and windows) too hard: it is quick
to fall over taking down the whole machine with it (yes,
avoid Ctrl-C at all costs).
The repos have always been recoverable for me, though.

> I got a full clone using the Cygwin git 1.5.0 [it took 6hrs 20min
> on a Xen virtual machine; I have to try it with real hardware],
> MinGW version did not work for me too well :-(
>  Are there any other options?

Avoid Win32 if possible, work somewhere in a sane environment,
using windows for testing, if you have to.

> Is http://git.or.cz/gitwiki/WindowsInstall up-to-date?

Yes.

Re: [tools-dev] Re: Git benchmarks at OpenOffice.org wiki

[Johannes Sixt]

Jan Holesovsky wrote:
> Last question: what is the status of the Win32 support?  I got a full clone
> using the Cygwin git 1.5.0 [it took 6hrs 20min on a Xen virtual machine; I
> have to try it with real hardware], MinGW version did not work for me too
> well :-(

I'd really like to hear what you exactly mean by "MinGW version did not
work too well". I use it in production. (But then, I'm the one who
publishes the MinGW port ;)

-- Hannes

Re: Git benchmarks at OpenOffice.org wiki

[Jakub Narebski]

Robin Rosenberg wrote:
> torsdag 03 maj 2007 skrev Johannes Schindelin:
>> On Thu, 3 May 2007, Florian Weimer wrote:
>>> * Jakub Narebski:
>>> 
>>>> What I'm really concerned about is branch switch and merging branches, 
>>>> when one of the branches is an old one (e.g. unxsplash branch), which 
>>>> takes 3min (!) according to the benchmark. 13-25sec for commit is also 
>>>> bit long, but BRANCH SWITCHING which takes 3 MINUTES!?
>>> 
>>> IIRC, GIT accesses every file in the tree, not just the ones that need
>>> updating.  How many files were actually updated when you changed
>>> branches in your experiment?
>> 
>> No. Git does not access every file, but rather all stats. That is a huge 
>> difference. And it should not take _that_ long for ~64000 files. Granted, 
>> it will cause a substantial delay, but not in the range of minutes.
> 
> It's worse... On my laptop the switch took ~ten minutes, not three. 
> A diff --stat takes over six minutes!! For reference, dd:in the pack 
> file with my disk takes ~50 seconds.
> 
> The reason is simple. I have a lousy one gigabyte RAM only, while 
> git wants 1.7GB virtual to do the diff-stat, and 800 MB resident.
> The swap is having a party, 

That is nice to know where the culprit is: 197000 files and 24000
directories (132 projects), i.e. huge tree and not enogh memory.
This is yet another reason for splitting OOo repository into subprojects.
I do wonder if git can be more conservative in memory usage...

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [tools-dev] Re: Git benchmarks at OpenOffice.org wiki

[Jakub Narebski]

On Thu, May 02, 2007, Jan Holesovsky wrote:
> On Thursday 03 May 2007 01:30, Jakub Narebski wrote:

>>> We should better split the OOo sources; it's a process that already
>>> started [UNO runtime environment vs. OOo without URE], and I proposed
>>> some more changes already.
>>
>> In my opinion each submodule should be able to compile and test by
>> itself. You can go X.Org route with splitting sources into modules...
> 
> Indeed, this is the case of URE - it is supposed to run by separately & be 
> used even by other projects than OOo.
> 
>> or you can make use of the new submodules support (currently plumbing
>> level, i.e. low level commands), aka. gitlinks.
> 
> And this would be interesting for the translations, I guess...
> 
>> The submodules support makes it possible to split sources into
>> independent modules (parts), which can be developed independently,
>> and which you can download (clone, fetch) or not, while making it
>> possible to bind it all together into one superproject.

By the way, even without submodule support, which for now is plumbing
level only, it would be possible to pull separate subprojects into main
project, like git repository does now with gitk repository, and with
git-gui repository. The latter is merged putting git-gui files in separate
directory in git.git repository, via using 'subtree' merge strategy.

Submodules / subprojects are something similar to Subversion svn:externals
done right.

-- 
Jakub Narebski
Poland

Re: Git benchmarks at OpenOffice.org wiki

[Linus Torvalds]

On Wed, 2 May 2007, Jakub Narebski wrote:
> On 5/2/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > 
> > Something I realized with pain is that the refs/ directory is 24MB big.
> > Yep. Really. They have 3464 heads and 2639 tags. I suspect that this is
> > the reason why.
> 
> Then packed refs would certainly help with speed and a bit with size.

Btw, this reminds me: we really should start out clones with a fully 
packed set of refs. It seems stupid to get the refs in one go, and then 
explode them into thousands of files.

A trivial patch is to just do

	git pack-refs --all --prune

in the "git-clone.sh" script rather than force people to do it themselves, 
but we really probably shouldn't have ever even unpacked them in the first 
place. That is kind of stupid, but especially since that thing is written 
in shell, it's hard to do anything smarter.

Of course, I don't know what the hell openoffice is doing with that many 
branches and tags, but I guess it's a normal result of having used CVS/SVN 
- you want to tag every single merge you do, and all branches stay around 
forever, because you can never merge them back and get rid of them.

It's always sad to see the crap that is CVS, and how bad decisions in CVS 
end up resulting in pain downstream.

		Linus

Re: Git benchmarks at OpenOffice.org wiki

[Robin Rosenberg]

torsdag 03 maj 2007 skrev Martin Langhoff:
> [resend - correcting a couple of typos and addressing git@vger
> correctly - apologies]
> 
> On 5/3/07, Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
> > The reason is simple. I have a lousy one gigabyte RAM only, while
> > git wants 1.7GB virtual to do the diff-stat.  and 800 MB resident. The swap is having a party,
> 
> That is true, unfortunately. git will fly if it can fit its working
> set plus the kernel stat cache for your working tree in memory. And
> the underlying assumption is that for large trees you'll have gobs of
> RAM. If things don't fit, it does get rather slow...
> 
> But... just to put things in perspective, how long does it take to
> *compile* that checkout on that same laptop. I remember reading
> instructions to the tune of "don't even try to compile this with less
> than 4GB RAM, a couple of CPUs and 12hs". Those were for the OSX build
> IIRC.
No idea. I wouldn't try it without distcc and ccache anyway which makes the
capabilities of this particular machine less relevant. 

> Ah - it's moved to the general instructions: "Building OOo takes some
> time (approx 10-12 hours on standard desktop PC) ":
> http://wiki.services.openoffice.org/wiki/Building_OpenOffice.org#Starting_the_real_build
> 
> So I don't think anyone working on projects the size of the kernel or
> OO.org is going to be happy with 1GB RAM.

The kernel 2.6 repo isn't in the same ball park wrt to size.  Hacking the kernel is quite fine
on this machine and even smaller though the first compile takes some time. Having more 
is always fun though. 

Consider another huge project like Eclipse. Similar operations take a loong time (not anywhere
near the eons that CVS need, but...)  and building Eclipse with 1GB i very reasonable so sheer
project size does not per se demand powerful computers. KDE is another huge project that
is reasonable to build with 1GB. The first time is somewhat painful, but rebuilding is not.

-- robin

Re: Git benchmarks at OpenOffice.org wiki

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> A trivial patch is to just do
>
> 	git pack-refs --all --prune
>
> in the "git-clone.sh" script rather than force people to do it themselves, 
> but we really probably shouldn't have ever even unpacked them in the first 
> place. That is kind of stupid, but especially since that thing is written 
> in shell, it's hard to do anything smarter.

It is not just being in shell.

Although I do agree that the initial clone is special, I would
rather make clone just a thin wrapper to fetch that also happens
to perform necessary initial setup.

Keeping fetched and updated refs in core and write a packed refs
out in one go in git-fetch--tool (and later, git-fetch all in C)
would be much simpler if we do not have to worry about existing
refs (aka "git clone" special case); I am not sure if packing
refs is desirable in general for incremental "git-fetch".

Re: Git benchmarks at OpenOffice.org wiki

[Linus Torvalds]

On Mon, 7 May 2007, Junio C Hamano wrote:
> 
> Keeping fetched and updated refs in core and write a packed refs
> out in one go in git-fetch--tool (and later, git-fetch all in C)
> would be much simpler if we do not have to worry about existing
> refs (aka "git clone" special case); I am not sure if packing
> refs is desirable in general for incremental "git-fetch".

Fair enough. It's true that for the general case of "git fetch", it's much 
less obvious how to keep things packed.

So maybe the right thing really *is* to just add the

	git pack-refs --all --prune

to the git-clone wrapper.

		Linus