Re: [PATCH 4/5] fast-export: Introduce --inline-blobs

[Drew Northup <drew.northup@maine.edu>]

On Thu, 2011-01-20 at 10:20 +0530, Ramkumar Ramachandra wrote:
> Hi,
> 
> Jonathan Nieder writes:
> > Junio C Hamano wrote:
> > > Ramkumar Ramachandra <artagnon@gmail.com> writes:
> > > Just thinking aloud, but is it possible to write a filter that converts an
> > > arbitrary G-F-I stream with referenced blobs into a G-F-I stream without
> > > referenced blobs by inlining all the blobs?
> > 
> > to avoid complexity in the svn fast-import backend itself.
> > (Complicating detail: such a filter would presumably take responsibility
> > for --export-marks, so it might want a way to retrieve commit marks
> > from its downstream.)
> 
> This filter will need to persist every blob for the entire lifetime of
> the program. We can't possibly do it in-memory, so we have to find
> some way to persist them on-disk and retrieve them very
> quickly. Jonathan suggested using something like ToyoCabinet earlier-
> I'll start working and see what I come up with.

Is it worth including the extra dependency? Most systems that I'm in
frequent contact with already have some lightweight BDB implementation
already. I don't currently know of any with TokyoCabinet (or
KyotoCabinet for that matter) already in place. Besides, if all you're
doing is persisting blobs that you're likely to write out to disk
eventually anyway you might as well just do so once you have them and
keep an "index" (not to be confused with the Git Index, just lacking a
better word right now) of what you have in some standard in-memory
format (a heap?). From there you can build each commit into the Git
Index in the proper order once you have the required parts for
each--perhaps even re-using the blobs you've already dumped to disk
(mv'ing them or something).

Granted, there's a good likelihood that I'm missing something here, but
I don't see the point of adding external complexity (beyond what you're
currently stuck dealing with).

-- 
-Drew Northup
________________________________________________
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59