Re: kernel.org and GIT tree rebuilding

[Linus Torvalds <torvalds@osdl.org>]

On Sat, 25 Jun 2005, Jeff Garzik wrote:
> 
> The problem is kernel.org mirroring, not individual pushes and pulls, 
> really.
> 
> Would git-pack be the best solution for mirroring a bunch of git trees?

No guarantees, but here's a rough plan:

 - I just committed a fairly trivial change to add a "--objects" flag to 
   git-rev-list, which allows you to basically say "I want to see the 
   difference not just in commit ID's, but also trees and blobs"

   What does that mean? It means that in a mirroring schenario, you can, 
   for each git tree, do:

	(a) On the slave:
		cat .git/refs/*/* | sort | uniq > slave-ref-list

	(b) On the master:
		cat .git/refs/*/* | sort | uniq > master-ref-list

	(c) On the master:

		cmp $master-ref-list $slave-ref-list && exit 1
		list=$(cat master-ref-list)
		for i in $(cat slave-ref-list)
		do
			list=$list ^$i
		done
		git-rev-list --objects $list

   and now that "git-rev-list" will list every object that needs to be 
   copied from the master to the slave. No need to read huge directories 
   etc, you get the list computed for you.

yeah, it clearly needs some refining to be useful, but I think you can
kind of see how it would work.

Now, the secondary advantage of this is that once you don't use rsync as
the mirroring method, you can now change the filesystem object database
layout. In particular, the packing thing that Chris Mason was working on
at some point suddenly becomes a lot more viable.

(In fact, more than that. You can make a single packed blob for all
"historical" objects, and that also gives you an efficient archive format
- if you're not required to have the full filesystem layout, you could
have a much more efficient packing that you basically do once a week or
something, so that you only keep the last week in the regular "one file
per object" format).

		Linus