Git transfer protocols (was: Re: Git-commits mailing list feed)

[Mike Taht <mike.taht@timesys.com>]

Jan Harkes wrote:

> rsync works fine for now, but people are already looking at implementing
> smarter (more efficient) ways to synchronize git repositories by
> grabbing missing commits, and from there fetching any missing tree and
> file blobs. However there is no such linkage to discover missing tag
> objects, only a full rsync would be able to get them and for that it has
> to send the name of every object in the repository to the other side to
> check for any missing ones.

I think that one reason why rsync is inefficient for git is that it 
appears to need an acknowledgement after every file. (at least, that's 
what what the rhythm of the packets looked like when I sniffed it 
earlier, I don't know anything else about it) For a series of very small 
files this interacts badly with tcp's flow control mechanisms. Perhaps 
rsync could be modified for a "sliding file acknowledgement window".

Most "swarming protocols" (e.g BitTorrent, eDonkey) work well for one 
big file shared among multiple hosts, but poorly for lots of small files.

*Nothing* out there matches the simplicity of git's sha1 filename 
length... but

Something like robcast or fcast/flute might be of interest:

http://www.inrialpes.fr/planete/people/roca/mcl/mcl_in_short.html

Or one of the multicast netnews experiments:

"mcntp" http://mcntp.sourceforge.net/
"newscaster" http://www.dmn.tzi.org/en/newscaster.html

lastly, Monotone has it's own "netsync" protocol
(via http://www.venge.net/monotone/faq.html)

"[netsync] is a bi-directional pipelined protocol for synchronizing 
collections using a tree of hashed indices. It allows any copy of 
monotone to function as either a client or a server, and rapidly 
synchronize or half-synchronize (push / pull) their database with 
another user. It is somewhat similar in flavor to rsync or Unison, in 
that it quickly and idempotently synchronizes information across the 
network without needing to store any local state; however, it is much 
more efficient than these protocols."



-- 

Mike Taht