Re: [ANNOUNCE] git_fast_filter

[Elijah Newren <newren@gmail.com>]

Hi,

On Wed, Apr 8, 2009 at 3:45 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> On Tue, 7 Apr 2009, Elijah Newren wrote:
>
>> Just thought I'd make this available, in case there's others with niche
>> needs that find it useful...
>
> Have you seen
>
>        http://thread.gmane.org/gmane.comp.version-control.git/52323
>
> I was rather disappointed that skimo left the patch series in a rather
> half-useful state.

No, I had not.  Looks useful, though it appears to be missing a number
of options from git-filter-branch, such as --subdirectory-filter,
--tree-filter, and --prune-empty.  I'm guessing that's related to your
"half-useful" comment?

I was particularly interested in --tree-filter for my rewrite, since I
needed it (to remove all cvs keywords and a few other touch-ups) and
it was the slowest of the filters I'd be using.  Problem is, in a
repository with 40,000 commits and 70,000 files in the latest commits,
--tree-filter is unacceptably slow.  On average, a commit would have
about 35,000 files (assuming approximately linear growth over the
commit history), meaning that I'd have to modify 40,000 x 35,000 files
= 1,400,000,000 files[1].  However, on average, less than a dozen
files changed per commit, so there are less than 40,000 x 12 = 480,000
unique files that actually need to be rewritten.  git-fast-export
provides (and git-fast-import expects) just those half million files,
and rewriting half a million files instead of 1.4 billion files is the
difference between a 45 minute rewrite and a 3 month one.

I didn't see a way to easily avoid the 1.4 billion file rewrite using
git-filter-branch (or git-rewrite-commits had I known about it), and
writing something to parse and modify git-fast-export output seemed
like the easiest solution.  Perhaps I could have written some fancy
index-filter script that recorded original and modified file sha1sums
somewhere and used that to only check out certain files and rewrite
them, but such an idea hadn't occurred to me (and I'm not sure it
would have been the better route even if it had).  Maybe there's
something I missed that would have made this easy, though?


Elijah

[1] For simplicity, I'm ignoring the 'binary' files that should not
have any cvs-keyword unmunging performed on them.  However, it does
present an issue, particularly with extra process forks, since you
need to determine which files are safe to modify.  I used libmagic
(the library behind the unix 'file' command) to avoid the need to run
'file' repeatedly.