Re: dumb transports not being welcomed..

[Linus Torvalds <torvalds@osdl.org>]

On Wed, 14 Sep 2005, Johannes Schindelin wrote:
> 
> What I see when fetching all heads (thanks to Junio, this is one call to 
> git-fetch now), where all but origin are up to date, is that it takes a 
> very long time. Swapping kicks in, and top tells me that 26.6% of the 
> memory is occupied by git-rev-list (The server has 128M, with 1G swap, and 
> I am unfortunately not the only user of this machine).

Ok. As mentioned, I've not looked at memory usage. The machines I play 
with tend to have 2GB or more, simply because bk needed at least 1GB to be 
nice and cached on the kernel ;)

Git has needed less than bk, so I've not cared ;)

> I fail to see why it should need those amounts of memory. (I tested this 
> over the ssh protocol, which should essentially do the same as git-daemon, 
> right?) After all, the merge point between the branches should be marked 
> uninteresting after one single step from each of my private branches.

One of the issues is that git-rev-list will (for example) keep track of 
the commit messages too for every commit. That in itself can be a lot of 
stuff, depending on how active the tree is and how large the messages are.

Now, that should be easy enough to fix (parse_commit() normally saves the 
buffer it parses into "commit->buffer", so we'd just need to do something 
like

	if (!verbose_header && commit->buffer) {
		free(commit->buffer);
		commit->buffer = NULL;
	}

for each commit.

But for --objects, the bigger memory pressure is that it needs to track 
the "struct object" for every single object when it generates the 
reference tracking. And THAT tends to be expensive. The object lists are 
also not very space-efficient (ie one small allocation for each list 
entry).

We could probably make objects/lists more space-efficient.

> I also see other strange things like packing 0 objects, and packing >0 
> objects after just having fetched from that repository. Hopefully I will 
> have time to look into that (and understand the code to begin with).

Well, the "packing 0 objects" should be normal. I'm surprised at the ">0" 
case after a fetch: the packign is _not_ guaranteed to be exact, but if 
you have the exact same state as (or a superset of) the other end, you 
should always see a zero.

		Linus