Re: SHA1 collision in production repo?! (probably not)

[Jeff King <peff@peff.net>]

On Tue, Sep 12, 2017 at 06:18:32PM +0200, Lars Schneider wrote:

> we are seeing this now in Git 2.14.1:
> 
> ...
> error: inflate: data stream error (unknown compression method)
> error: unable to unpack 7b513f98a66ef9488e516e7abbc246438597c6d5 header
> error: inflate: data stream error (unknown compression method)
> error: unable to unpack 7b513f98a66ef9488e516e7abbc246438597c6d5 header
> fatal: loose object 7b513f98a66ef9488e516e7abbc246438597c6d5 (stored in .git/objects/7b/513f98a66ef9488e516e7abbc246438597c6d5) is corrupt
> fatal: The remote end hung up unexpectedly
> 
> I guess this means your fix [1] works properly :-)

Oh, good. :)

> At some point I will try to explore a retry mechanism for these cases.

I don't think we can generally retry loose-object failures. We use
copies from packs first, and then loose. So a corrupt loose can fallback
to another pack or to loose, but not the other way around (because we
would never look at the loose if we had a good copy elsewhere).

Though in your particular case, if I recall, you're receiving the object
over the network and the corrupted copy is in the way. So right now the
recovery process is:

  1. Notice the commit message.

  2. Run git-fsck to notice that we don't really[1] need the object.

  3. Run `rm .git/objects/7b/513f...`

  4. Re-run the fetch.

But in theory we should be able to say "oh, we don't _really_ have that,
the collision test isn't necessary" and then overwrite it. I actually
thought that's what would happen now (has_sha1_file() would return an
error), but I guess for what we need, it just does a stat() and calls it
a day, not realizing we ought to be overwriting.

-Peff

[1] git-fsck will actually complain if reflogs point to the object, and
    we can always expire those in a corrupted repo. So possibly what you
    want to know is whether it's reachable from actual refs. Of course
    this whole check is optional. If the object's corrupted, it's
    corrupted. But I get nervous calling `rm` on something that _could_
    be precious (say it's just a single-bit error that could be
    recovered). But if you have a known-good copy incoming, that's less
    of an issue.