Re: Handling large files with GIT

[Junio C Hamano <junkio@cox.net>]

Linus Torvalds <torvalds@osdl.org> writes:

> Side note: the original explicit git "delta" objects by Nicolas Pitre 
> would have handled this large-file-case much more gracefully. 

True.

> The pack-files had absolutely huge advantages, though, so I think we (I) 
> did the right thing there in making the delta code only a very specific 
> special case..

Well the blame for ripping that out falls on me, actually...

> It is possible that we could re-introduce the "explicit delta" object, 
> though (it's not incompatible with also doing pack-files, it's just that 
> pack-files made 99% of all the arguments for an explicit delta go away).

I do not remember we had 'rev-list --objects' support for Nico's
explicit delta object chains.  If we didn't that would be a new
development that needs to be done to resurrect it.  I know
pack-objects never had support for it so obviously that needs to
be added as well.  Probably explicit delta objects should always
be packed in full without spending cost to find delta candidates.

Personally I feel that post-1.2.0 would be a good time to start
looking at enhancing the pack generation chain, rev-list piped
to pack-objects.  This "large files" use case is helped by
less self-contained packs while "shallow clone" use case
we discussed earlier is helped by more self-contained packs (we
had a discussion long time ago on this and I think we have the
code to do so [*1*]). 

An addition to pack-objects is needed to make it capable to read
a list of objects that we do not want to include in the
resulting pack but can be used as base objects for delitified.

BTW, as to the "shallow clone", I changed my mind and am
inclined to agree with Johannes that handling cut-offs
differently from grafts is easier for dealing with later "give
me more history" operation, so I am planning to chuck my jc/clone
topic branch that I have included in the proposed updates so
far.

[Footnote]

*1* http://article.gmane.org/gmane.comp.version-control.git/5779