Re: "git-send-pack"

[Linus Torvalds <torvalds@osdl.org>]

On Fri, 1 Jul 2005, Mike Taht wrote:
> 
> You are getting closer and closer to where something like bitTorrent or 
> a multicast protocol makes sense. The problem isn't just the number of 
> outstanding commit objects but the number of machines and developers 
> that want to grab those commits at the same time.

I don't think so. First off, I don't think the decision is kernel- 
specific, in the sense that I at least use git for sparse and git itself 
too, so the solution should make sense for small projects as well.

Also, even for the kernel, the total dataset right now (after three months
or whatever) is a 60MB pack. It's not like we're sending DVD's or even
CD's worth of data around - we're sending the equivalent of 20MB per
_month_. That's really not a lot of data. You could easily keep up with a 
slow modem.

Also, the number of people involved isn't _that_ big. We're talking a few
thousand people who actively would update their trees for a big project,
and many smaller projects have anything from a couple to maybe a hundred. 
A few mirrors, and you don't have any problem.

So I think that the problem is actually not that big, and we just need to
find an acceptable format. Quite frankly, it might be perfectly acceptable
for kernel.org to run a simple packing script once a week which packs
everything into one single file, and even if that means that the mirrors
will have to re-get everything once a week, that actually sounds 
acceptable.

It's obviously a _stupid_ way to handle the rsync problem, so there's 
bound to be some cleaner solution, but the point is that we can probably 
make mirroring acceptable even with a really really stupid approach. I'd 
be a bit ashamed of just how ugly it is, but it would likely _work_ fine.
You'd create 52 pack-files in a year, but each pack-file is likely just
ten megabytes each. 

Oh, each pack-file should also be associated with the list of "refs" that
were used to generate that pack-file, so make that 104 files per project
year (but the list of "refs" would usually be something small, like

	refs/heads/master       4a89a04f1ee21a7c1f4413f1ad7dcfac50ff9b63
	refs/tags/v2.6.11       5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
	refs/tags/v2.6.11-tree  5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
	refs/tags/v2.6.12       26791a8bcf0e6d33f43aef7682bdb555236d56de
	refs/tags/v2.6.12-rc2   9e734775f7c22d2f89943ad6c745571f1930105f
	refs/tags/v2.6.12-rc3   0397236d43e48e821cce5bbe6a80a1a56bb7cc3a
	refs/tags/v2.6.12-rc4   ebb5573ea8beaf000d4833735f3e53acb9af844c
	refs/tags/v2.6.12-rc5   06f6d9e2f140466eeb41e494e14167f90210f89d
	refs/tags/v2.6.12-rc6   701d7ecec3e0c6b4ab9bb824fd2b34be4da63b7e
	refs/tags/v2.6.13-rc1   733ad933f62e82ebc92fed988c7f0795e64dea62

which was trivially generated from my current tree with

	for i in refs/*/*; do echo -ne $i"\t"; cat $i; done

so now you can use the refs associated with the previous pack-file as the 
list of refs you're _not_ interested in, and the current list of refs as 
the list you _are_ interested in, and generate the new pack-file.

Generating the pack-file would literally be something like

	obj=$(git-rev-parse $(cut -f2 new-list) --not $(cut -f2 old-list))
	git-rev-list $obj | git-pack-objects --stdin > new-pack

so a few one-liners like this, run from a cron-job once a week, should
just do it.

		Linus