Segfault during fsck and spreading brokenness

Segfault during fsck and spreading brokenness

[Rich]

I was encouraged by a couple of people on stackoverflow to post to this 
list. Apols in advance if it's not the right place.


I encountered a Segfault during fsck on a damaged bare repo (probably 
due to a powercut. Possibly during an operation, although not sure):

git --version
git version 2.11.0

git fsck --full -v
Checking HEAD link
Checking object directory
Checking tree 11bbc847cf1b4422b3e37830a9eac2e7af6559de
Checking tree 11be4abeb20314de6145dfc0e6180807a74c03dc
--->8 snip 8<---------------------------------------------------------------
Checking tree 14a4423e86f06c7ad75bf391d138e0cf7790508f
Checking tree 147aeaec72b2f29bf1813494c942fbce497be679
zsh: segmentation fault  git fsck --full -v

Host: Debian 10. Linux 4.9.0-13-amd64 #1 SMP Debian 4.9.228-1 
(2020-07-05) x86_64 GNU/Linux

FS: BTRFS


I learnt about the damage while trying to `git push`  from my dev 
working tree. I think I did a `git pull` when the `git push` failed. The 
result of this was that my *local* repo was also damaged: there was an 
empty file created that would stop git operations and on deleting that I 
got a page full of errors.

Original SO question: 
https://stackoverflow.com/questions/64044995/can-one-corrupt-git-repo-corrupt-another?noredirect=1


Unfortunately I don't think I'm at liberty to give access to the repo as 
it possibly contains sensitive info/personal data.

I have solved the problem for myself (took a morning to do it!) but 
reporting here in case it's useful to do so. I use git all the time and 
I'm grateful for it. I'm not a systems-level coder, so not sure what use 
I'll be, but happy to answer any Qs or try things on the broken copies.


Thanks,

Rich

Re: Segfault during fsck and spreading brokenness

[Jeff King]

On Fri, Sep 25, 2020 at 04:48:39PM +0100, Rich wrote:

> I was encouraged by a couple of people on stackoverflow to post to this
> list. Apols in advance if it's not the right place.
>
> I encountered a Segfault during fsck on a damaged bare repo (probably due to
> a powercut. Possibly during an operation, although not sure):

Thanks for the report. This is definitely the right place.

> git --version
> git version 2.11.0
> 
> git fsck --full -v
> Checking HEAD link
> Checking object directory
> Checking tree 11bbc847cf1b4422b3e37830a9eac2e7af6559de
> Checking tree 11be4abeb20314de6145dfc0e6180807a74c03dc
> --->8 snip 8<---------------------------------------------------------------
> Checking tree 14a4423e86f06c7ad75bf391d138e0cf7790508f
> Checking tree 147aeaec72b2f29bf1813494c942fbce497be679
> zsh: segmentation fault  git fsck --full -v

It's tough to say from this where the problem might be. If you still
have the broken repo and can reproduce, two things that might help:

  - trying with a more recent version of Git; we've fixed quite a few
    segfaults around corrupted data in the past few years

  - getting a backtrace; if you can build Git from source, the simplest
    thing is just running "gdb --args /path/to/your/git fsck --full -v",
    waiting for it to crash, and then running "bt" in the debugger

> I learnt about the damage while trying to `git push`  from my dev working
> tree. I think I did a `git pull` when the `git push` failed. The result of
> this was that my *local* repo was also damaged: there was an empty file
> created that would stop git operations and on deleting that I got a page
> full of errors.

That's concerning; push/pull generally try hard not to let corruption
spread. What transport do you use between the two servers? The usual
git-over-ssh and git-over-http protocols should be pretty resilient, but
I would not be surprised if the old dumb-http protocol, which just
downloads remote files wholesale, would be confused by a zero-length
loose object file or similar.

-Peff

Re: Segfault during fsck and spreading brokenness

[Rich]

Hi Jeff,

Thanks for a friendly and welcoming response (I was a bit scared posting 
here I suppose because the project's creator has a rep for not suffering 
fools!).

I copied the repo that causes a segfault on 2.11.0 to my laptop which 
runs 2.25.1 and I'm pleased to report that there's no segfault. I've 
posted the errors it does show below this message in case they're useful.

>   What transport do you use between the two servers? The usual
> git-over-ssh and git-over-http protocols should be pretty resilient, but
> I would not be surprised if the old dumb-http protocol, which just
> downloads remote files wholesale, would be confused by a zero-length
> loose object file or similar.

The two servers are connected by SSH. As in:

% git config remote.origin.url sshalias:path/to/repo

where sshalias is defined in my .ssh/config file.

To test again,I made a bare clone of the fixed repo on my 2.25.1 host. I 
still had some dangling trees/blobs so I ran the following to make sure 
we were all clean:

git reflog expire --expire-unreachable=now --all
git gc --prune=now
git fsck --full


Then I added the broken repo as a remote (ssh) repo and ran fetch, which 
crashed (presumably correctly):

% git fetch broke
error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 
is empty
error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 
is empty
fatal: loose object 3e9c7cd338d2ea3603d0c23b4b56902a04937833 (stored in 
./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833) is corrupt
fatal: the remote end hung up unexpectedly

I re-ran fsck and it was still all clean. So 2.25.1 seems safe.I 
repeated the same procedure on the 2.11.0 version (Debian 10 host), and 
the same thing happened.

So I'm not sure what happened with the two repos getting broken at once; 
can't repeat it.

Let's hope that this was something that was fixed as you say!

Thanks,
Rich




## Errors shown when running fsck on 2.25.1 on the broken repo:

git --version
git version
% git fsck --full
error: object file ./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c 
is empty
error: unable to mmap 
./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c: No such file or 
directory
error: 142d73242bba1e4aee89a887ec37d3b1ef00461c: object corrupt or 
missing: ./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c
error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 
is empty
error: unable to mmap 
./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833: No such file or 
directory
error: 3e9c7cd338d2ea3603d0c23b4b56902a04937833: object corrupt or 
missing: ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833
error: object file ./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9 
is empty
error: unable to mmap 
./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9: No such file or 
directory
error: 45aaa8e78abffa54ee2a620ef453d2b73443a3f9: object corrupt or 
missing: ./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9
error: object file ./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4 
is empty
error: unable to mmap 
./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4: No such file or 
directory
error: 51c79dda59372c0bd44457fbf810c66a6b184bb4: object corrupt or 
missing: ./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4
error: object file ./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52 
is empty
error: unable to mmap 
./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52: No such file or 
directory
error: 9292c90c7f01b5f9252a4631071d8ed39b121d52: object corrupt or 
missing: ./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52
error: object file ./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4 
is empty
error: unable to mmap 
./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4: No such file or 
directory
error: a01c6d43bb04b86b8263848c9d8daa20d4a68be4: object corrupt or 
missing: ./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4
error: object file ./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b 
is empty
error: unable to mmap 
./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b: No such file or 
directory
error: c6480c5f0be01c6860a17123bef79aa3fb98dd8b: object corrupt or 
missing: ./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b
error: object file ./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d 
is empty
error: unable to mmap 
./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d: No such file or 
directory
error: e7e8f34edee57ed77b240fa024df2f472dd9776d: object corrupt or 
missing: ./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d
Checking object directories: 100% (256/256), done.
Checking objects: 100% (310949/310949), done.
error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 
is empty
error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 
is empty
fatal: loose object 3e9c7cd338d2ea3603d0c23b4b56902a04937833 (stored in 
./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833) is corrupt