I have gone and done a bad thing - malformed tree objects

I have gone and done a bad thing - malformed tree objects

[Jason Pyeron]

I was trying to "do stuff" using hash-object -t tree --stdin -w, but I accidentally created trees where other trees were marked as blobs. They were dangling and not connected to any actual commits on my branches.

After gc and fsck clean ups, everything reports well...

Except:

$ GIT_TRACE=1 git cat-file --batch-all-objects --batch=objecttype
20:41:33.323399 git.c:442               trace: built-in: git cat-file --batch-all-objects --batch=objecttype
objecttype
fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?

Where to start on this? I think the unused malformed tree referring to that object got put in to a pack. No idea how to find the malformed tree's id.

For that fact, cat-file views the object properly:

$ git cat-file -p 00009623a06b8dea7c151542fc789539599c07d0
100644 blob e465d57c345e2dcb117b5a30f9272b7fc5ec77cd    .p-truncated-names-because-they-are-not-needed-in-this-email
100755 blob 7f16c1d4cbb75cf7bd635970a2588ced6ccea8ad    Ap
040000 tree 5261c0a3f3b4c688a082c3c5eaf03f8039bf153c    CA
100644 blob 188c0d0541523016352b6851e0f7200c18a372e6    CM
100644 blob c8b040ec356b21fcc06911c544149dc6f5d5b861    CM
100644 blob e441983f0fd4d57fb7bf640de31f728529f12c29    CM
100644 blob fd06c9c6ad662e099341f4e0a05b272c6370e64b    CM
100644 blob d433fb05ebca807f4487ae4cecf48ec3b66cce78    CM
100755 blob 4b1b049b83bcc7821a7b62977124bfcaa024d960    CM
040000 tree 150d60813c913ec9a178c4230b18fbda84edc2af    RE
100644 blob ec54be851b811bff55a2034886a683969ef39880    ac
040000 tree f541fe035a345b5b3fcf84c83c64135899adadda    ad
040000 tree 80f08d7c97c88da471495a7e93e1466a737a7d29    au
040000 tree 7130737ca26ee3f1ac7d437bd8123d024c1cbfeb    cf
100755 blob 9637152c0a45a7ea716c5c05a7a3a68e2f655bc2    cm
100755 blob bde5db875d0c5d2bc6874ac62bff026bafd91914    co
100755 blob 308678b6e4f4aec8c03c6ced43750c84f8a0659d    cr
040000 tree 1657e8a77ed7430e4845b60cc1e1fb277b385d8a    da
100644 blob ed2eb8b686d008b895c52ddfca2ef2692174a722    de
100755 blob 6fe10b9f4d0df0d0c357b7ddee56e5612143c6b9    de
040000 tree 58a3cb399b25cf54f3ec92d52b79e3f081614d68    de
100755 blob f09ba30b1d5be4015dbcd8e8cec37449fae504e1    er
040000 tree 951642bc2f183e984e1879eab68f3414c16e535a    fa
100755 blob 5b796b1747de423f1fb81f976c860f50c3b1d02a    fi
100755 blob c2074af9676af85beb9f08bb1a5339b14d6ee485    ge
100755 blob 22ca54226a6c8e2bb2c9eb6d0db1731553e4b5ae    gl
100755 blob 57060dc44448e5b6c14a0df59bf81050a8464472    he
040000 tree af5e7c654837923730d15cc051f843d42ef84b56    hr
040000 tree 0a0f5e8c1b2968d516b501f7995d6e85897b26ab    im
040000 tree 6abe0be44fad7a58c0035010137431d9c581aabc    in
100755 blob 156ee5024cd8fb361552e873b843a8d90b861fb6    in
100644 blob 3371f55d5057ea4f4d29eae0184dfe17ca2919ff    jb
040000 tree eef927bf134d3c3777e7c02f6e81f3de06be0bdc    li
100755 blob 8d0cfbb604a92c2b24b9761efc5320f851c96ac6    lo
100755 blob 25efc4fd596e651d22c4bfdd8d694acefb5aa004    lo
100755 blob 92b3f44a17c12485923cf0143938c6f2490bced9    lo
100755 blob af171c4a7b37c1763d05275ddeeb994fd6ec5d01    ma
100755 blob 4047be1da363bf55fa9773218665f723d4d9941d    mc
100755 blob 0a28fb9de61bb5fb9de7876514dabbbc35b8d444    me
040000 tree b9b374e4925ece4b5781ef70c6590fab8a85b0f5    mi
100755 blob 6bb288db5b11856ea79577b52d57bf8b0c8ea422    po
040000 tree 2e18b952c4c796afdcdf4d41230c2e8a4212a8b9    re
040000 tree 002c945c849e2b976ae193481ad299181322663f    re
040000 tree 40dd932e877ad3aee47628277d9e75c53284dea5    re
100755 blob ea520d8ccda4207e732c3549d4c18d1ae0e6136b    sc
040000 tree 9a7014ee2249be58fd79bbb1a76f8ef94ef1af8e    sc
100644 blob dfbbb48c461b7739c211ae84a7b1355a03a0bd5e    se
040000 tree cdb9bdb60dc857cb04e707582852e700cc5c1d63    se
100755 blob 87542658ec3f45076443a278f7aac8cdcdc649cd    se
100755 blob bc50a5c1ba42b95abef98c31c36ad1c154b480b4    si
100644 blob 6e96fb004b88d5537bec9fcfeab581253886a61f    sq
100644 blob 6e96fb004b88d5537bec9fcfeab581253886a61f    sq
040000 tree a614f05126c49e0ad2c35243907d3d7300db85a2    te
040000 tree 1f7e4a51903a860fb4710060aa035beda35e18df    te
100755 blob ebb1a7029de2062a00b851463922560ed1a369ae    te
100644 blob e794743d0bf1c99b3ba211bdcf5d60e8261aa8f5    te
100644 blob da41b803b2a86c90417b3ed1f7fda1b9524fa897    te
100755 blob 43803c936c25f29895dc1c78a8f1ebcd10ab77b8    te
040000 tree 042ca993b1f44d4cd7abd846f669c9b0ee9d5545    tr
040000 tree de84b6f6c8a562a001a492efe640c43e158b2dd8    tr
100755 blob 90dbabed658c0cbf9f74eed9eff6da0ef6155464    un
100644 blob ebc83b2afb7bed3c360b7cc5da8750ab3038a2ee    vd
100644 blob 7c689ba4641860b103a6afda8dc2c0cc803eabdf    xl

Thanks in advance, and sorry for my silliness.

-Jason

Re: I have gone and done a bad thing - malformed tree objects

[Junio C Hamano]

"Jason Pyeron" <jpyeron@pdinc.us> writes:

> I was trying to "do stuff" using hash-object -t tree --stdin -w,
> but I accidentally created trees where other trees were marked as
> blobs. They were dangling and not connected to any actual commits
> on my branches.
>
> After gc and fsck clean ups, everything reports well...
>
> Except:
>
> $ GIT_TRACE=1 git cat-file --batch-all-objects --batch=objecttype

gc and fsck may not have pruned the dangling object yet, but
--batch-all-objects is a request to enumerate objects that exist in
the repository, regardless of their reachability from any ref.

Perhaps "git prune --expire=now" would get rid of it?

RE: I have gone and done a bad thing - malformed tree objects

[Jason Pyeron]

> From: Junio C Hamano
> Sent: Tuesday, July 28, 2020 8:52 PM
> 
> "Jason Pyeron" writes:
> 
> > I was trying to "do stuff" using hash-object -t tree --stdin -w,
> > but I accidentally created trees where other trees were marked as
> > blobs. They were dangling and not connected to any actual commits
> > on my branches.
> >
> > After gc and fsck clean ups, everything reports well...
> >
> > Except:
> >
> > $ GIT_TRACE=1 git cat-file --batch-all-objects --batch=objecttype
> 
> gc and fsck may not have pruned the dangling object yet, but
> --batch-all-objects is a request to enumerate objects that exist in
> the repository, regardless of their reachability from any ref.
> 
> Perhaps "git prune --expire=now" would get rid of it?

Both that and

git -c gc.reflogExpire=now -c gc.reflogExpireUnreachable=now   -c gc.rerereresolved=now -c gc.rerereunresolved=now   -c gc.pruneExpire=now -c gc.worktreePruneExpire=now gc --prune=now --aggressive

leave it in.

Re: I have gone and done a bad thing - malformed tree objects

[Junio C Hamano]

"Jason Pyeron" <jpyeron@pdinc.us> writes:

>> gc and fsck may not have pruned the dangling object yet, but
>> --batch-all-objects is a request to enumerate objects that exist in
>> the repository, regardless of their reachability from any ref.
>> 
>> Perhaps "git prune --expire=now" would get rid of it?
>
> Both that and
>
> git -c gc.reflogExpire=now -c gc.reflogExpireUnreachable=now   -c gc.rerereresolved=now -c gc.rerereunresolved=now   -c gc.pruneExpire=now -c gc.worktreePruneExpire=now gc --prune=now --aggressive
>
> leave it in.

If the cruft has already been stored in a packfile, then prune would
not touch it.  "git repack -a -d && git prune --expire=now" would be
the next thing to do.

RE: I have gone and done a bad thing - malformed tree objects

[Jason Pyeron]

> From: Junio C Hamano
> Sent: Wednesday, July 29, 2020 2:10 PM
> 
> "Jason Pyeron" writes:
> 
> >> gc and fsck may not have pruned the dangling object yet, but
> >> --batch-all-objects is a request to enumerate objects that exist in
> >> the repository, regardless of their reachability from any ref.
> >>
> >> Perhaps "git prune --expire=now" would get rid of it?
> >
> > Both that and
> >
> > git -c gc.reflogExpire=now -c gc.reflogExpireUnreachable=now   -c gc.rerereresolved=now -c
> gc.rerereunresolved=now   -c gc.pruneExpire=now -c gc.worktreePruneExpire=now gc --prune=now --
> aggressive
> >
> > leave it in.
> 
> If the cruft has already been stored in a packfile, then prune would
> not touch it.  "git repack -a -d && git prune --expire=now" would be
> the next thing to do.

$ git repack -a -d && git prune --expire=now
Enumerating objects: 327236, done.
Counting objects: 100% (327125/327125), done.
Delta compression using up to 8 threads
Compressing objects: 100% (104728/104728), done.
Writing objects: 100% (327125/327125), done.
Total 327125 (delta 205244), reused 326116 (delta 204678), pack-reused 0

$ git cat-file --batch-all-objects --batch=objecttype
fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?

Grrrrr....


$ mkdir bak/objects/pack -p

$ mv objects/pack/* bak/objects/pack/

$ for i in bak/objects/pack/*.pack; do git unpack-objects < $i ; done
Unpacking objects: 100% (111/111), 142.57 KiB | 41.00 KiB/s, done.
Unpacking objects: 100% (303/303), 1.16 MiB | 182.00 KiB/s, done.
Unpacking objects: 100% (327125/327125), 1.39 GiB | 158.00 KiB/s, done.

$ git -c gc.reflogExpire=now -c gc.reflogExpireUnreachable=now   -c gc.rerereresolved=now -c gc.rerereunresolved=now   -c gc.pruneExpire=now -c gc.worktreePruneExpire=now gc --prune=now --aggressive
Enumerating objects: 327314, done.
Counting objects: 100% (327314/327314), done.
Delta compression using up to 8 threads
Compressing objects: 100% (309595/309595), done.
Writing objects: 100% (327314/327314), done.
Selecting bitmap commits: 33000, done.
Building bitmaps: 100% (311/311), done.
Total 327314 (delta 170421), reused 0 (delta 0), pack-reused 0
Removing duplicate objects: 100% (256/256), done.

$ ls -Gg objects/pack/
total 1620740
-r--r-----+ 1    1979274 Jul 30 19:50 pack-45501d1b4064e03537158f14ea1cd40158424086.bitmap
-r--r-----+ 1    9165864 Jul 30 17:00 pack-45501d1b4064e03537158f14ea1cd40158424086.idx
-r--r-----+ 1 1648486198 Jul 30 17:00 pack-45501d1b4064e03537158f14ea1cd40158424086.pack

$ git fsck
Checking object directories: 100% (256/256), done.
warning in tree 64dee778f5cb68573709e04d5bdd0f391c002599: zeroPaddedFilemode: contains zero-padded file modes
Checking objects: 100% (327314/327314), done.
Checking connectivity: 327540, done.

jpyeron@blackfat /projects/disa-cmis/cmis.git
$ git cat-file -p 64dee778f5cb68573709e04d5bdd0f391c002599
040000 tree cee1ca9232b589fd8721cbe8c9a2910413f5607e    00
040000 tree d8e1f140eaa746fe358e5eb8c3d1573b432158c2    01
040000 tree 5b1b2826f56a9d7ce4ae4040cdf809bded41557c    02
...
040000 tree 49d3bcf3bf2dcaac80c12a6075b16ef6d067e807    fd
040000 tree 64a163e509b2c061927133eacc68826df202f933    fe
040000 tree 6572fafefd58be1364e7a8dc8230a13d0fd1c9b2    ff

$ git cat-file --batch-all-objects --batch=objecttype
objecttype
fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?

Double Grrrrr.....


$ mv bak bak2

$ mkdir bak/objects/pack -p

$ mv -v objects/pack/* bak/objects/pack/
'objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.bitmap' -> 'bak/objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.bitmap'
'objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.idx' -> 'bak/objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.idx'
'objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.pack' -> 'bak/objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.pack'

$ for i in bak/objects/pack/*.pack; do git unpack-objects < $i ; done
Unpacking objects: 100% (327314/327314), 1.53 GiB | 179.00 KiB/s, done.

$ git fsck --lost-found --no-reflogs
warning in tree 64dee778f5cb68573709e04d5bdd0f391c002599: zeroPaddedFilemode: contains zero-padded file modes
Checking object directories: 100% (256/256), done.
Checking connectivity: 326145, done.
dangling tree 9e6a1003b4999aacfe522160843059484350f34d
dangling tree ff42e36dc8727cb5c664ba42add1b42e70de209d
dangling blob d761c46714331a15dbcef84fd9af92678cd3c79d
dangling tree cfdf35b71ec160a5cf37cb52647f766dee0a3737
dangling tree 78dbd6e5a37fea6b708952a00e8bf1d1eb3eb5b5
dangling tree 798377228d213226f3b068aeb346b84c4df916a1
dangling tree d4d84773d2523d4d3290ebb669fbc37a3f16808d
dangling tree fa1de9e451e8f51f51f31ba15d1f0d941be0e51e
missing commit 37b7f95fd6b1a015559d0a1dfb0df8f0524fc660
dangling commit e3300a5773d39735a8930303f58411fc145eff93
dangling tree 9e3bba142e03a3b9f08a58ff88385e030553540d
dangling blob e5474ae6ae5fde0d0dfe83a32fa72b3c71601317
dangling tree ba486ad81526bc6bfed81a292ad4713a170f9cc6
dangling commit 8cba1b102d044d4098087c4b5fed1f02f3b63ad7
dangling tree 90657cf7213475972835035494edfa0ae2fea3fc
dangling commit e0d29cbbdf18cbaf6b23781f08f199d9bd11b7ef
dangling commit d22f7d69cb188b067c1ae385537aac67063a0bcd
dangling tree b7f16f536f3c5f6b0642fb2cae2e969da4c81f22

$ git cat-file --batch-all-objects --batch=objecttype
objecttype
fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?

I'm running fsck a second time with no packs to see what happens. Thoughts?

-Jason

Re: I have gone and done a bad thing - malformed tree objects

[Jeff King]

On Fri, Jul 31, 2020 at 07:05:42PM -0400, Jason Pyeron wrote:

> > If the cruft has already been stored in a packfile, then prune would
> > not touch it.  "git repack -a -d && git prune --expire=now" would be
> > the next thing to do.
> 
> $ git repack -a -d && git prune --expire=now
> Enumerating objects: 327236, done.
> Counting objects: 100% (327125/327125), done.
> Delta compression using up to 8 threads
> Compressing objects: 100% (104728/104728), done.
> Writing objects: 100% (327125/327125), done.
> Total 327125 (delta 205244), reused 326116 (delta 204678), pack-reused 0
> 
> $ git cat-file --batch-all-objects --batch=objecttype
> fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?

That should be dropping everything that isn't reachable. I'd suggest to
expire reflogs, though it looks like you've also tried "git gc" with
reflog expiration. Does removing .git/logs entirely help?

If not, are you sure it isn't actually reachable from your history? What
does:

  git rev-list --all --objects | grep 00009623a06

say? If no hits, does adding --reflogs to the command-line change it?

We also consider blobs in the index reachable. I don't recall offhand
whether that applies to trees mentioned by the cache-trees extension. I
don't _think_ that would apply to your broken tree, since they'd have
been generated by Git itself, but possibly removing .git/index (if this
isn't a bare repo) would help?

-Peff

RE: I have gone and done a bad thing - malformed tree objects

[Jason Pyeron]

> From: Jeff King
> Sent: Friday, July 31, 2020 7:15 PM
> 
> On Fri, Jul 31, 2020 at 07:05:42PM -0400, Jason Pyeron wrote:
> 
> > > If the cruft has already been stored in a packfile, then prune would
> > > not touch it.  "git repack -a -d && git prune --expire=now" would be
> > > the next thing to do.
> >
> > $ git repack -a -d && git prune --expire=now
> > Enumerating objects: 327236, done.
> > Counting objects: 100% (327125/327125), done.
> > Delta compression using up to 8 threads
> > Compressing objects: 100% (104728/104728), done.
> > Writing objects: 100% (327125/327125), done.
> > Total 327125 (delta 205244), reused 326116 (delta 204678), pack-reused 0
> >
> > $ git cat-file --batch-all-objects --batch=objecttype
> > fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?
> 
> That should be dropping everything that isn't reachable. I'd suggest to
> expire reflogs, though it looks like you've also tried "git gc" with
> reflog expiration. Does removing .git/logs entirely help?
> 
> If not, are you sure it isn't actually reachable from your history? What
> does:
> 
>   git rev-list --all --objects | grep 00009623a06

$ git rev-list --all --objects | grep 00009623a06
00009623a06b8dea7c151542fc789539599c07d0 src/htdocs
(it is still running...)

But that is an expected result, I will be back at work on Sunday.


> 
> say? If no hits, does adding --reflogs to the command-line change it?
> 
> We also consider blobs in the index reachable. I don't recall offhand
> whether that applies to trees mentioned by the cache-trees extension. I
> don't _think_ that would apply to your broken tree, since they'd have
> been generated by Git itself, but possibly removing .git/index (if this
> isn't a bare repo) would help?
> 
> -Peff

Re: I have gone and done a bad thing - malformed tree objects

[Jeff King]

On Fri, Jul 31, 2020 at 08:01:58PM -0400, Jason Pyeron wrote:

> > That should be dropping everything that isn't reachable. I'd suggest to
> > expire reflogs, though it looks like you've also tried "git gc" with
> > reflog expiration. Does removing .git/logs entirely help?
> > 
> > If not, are you sure it isn't actually reachable from your history? What
> > does:
> > 
> >   git rev-list --all --objects | grep 00009623a06
> 
> $ git rev-list --all --objects | grep 00009623a06
> 00009623a06b8dea7c151542fc789539599c07d0 src/htdocs
> (it is still running...)
> 
> But that is an expected result, I will be back at work on Sunday.

So it sounds like it's still reachable, and you'd need to rewrite
history to get rid of it. Or is that object OK, and it's a containing
tree that mentions it with the wrong mode the problem? In that case,
same question: is the containing tree reachable?

-Peff

RE: I have gone and done a bad thing - malformed tree objects

[Jason Pyeron]

> -----Original Message-----
> From: Jeff King
> Sent: Friday, July 31, 2020 9:45 PM
> 
> On Fri, Jul 31, 2020 at 08:01:58PM -0400, Jason Pyeron wrote:
> 
> > > That should be dropping everything that isn't reachable. I'd suggest to
> > > expire reflogs, though it looks like you've also tried "git gc" with
> > > reflog expiration. Does removing .git/logs entirely help?
> > >
> > > If not, are you sure it isn't actually reachable from your history? What
> > > does:
> > >
> > >   git rev-list --all --objects | grep 00009623a06
> >
> > $ git rev-list --all --objects | grep 00009623a06
> > 00009623a06b8dea7c151542fc789539599c07d0 src/htdocs
> > (it is still running...)

$ git rev-list --all --objects | grep 00009623a06
00009623a06b8dea7c151542fc789539599c07d0 src/htdocs

$ git rev-list --all --objects --reflog | grep 00009623a06
00009623a06b8dea7c151542fc789539599c07d0 src/htdocs

No unexpected results, just the correct tree pointing to the 00009623a06 as a tree.

> >
> > But that is an expected result, I will be back at work on Sunday.
> 
> So it sounds like it's still reachable, and you'd need to rewrite
> history to get rid of it. Or is that object OK, and it's a containing
> tree that mentions it with the wrong mode the problem? In that case,
> same question: is the containing tree reachable?

Backing up to the beginning.

There has always been a tree with a tree entry for that blob - which is reachable.

I then created a tree manually, in it I added that tree id as a blob reference when it should have been a tree. (e.g. 100644 blob 00009623a06b8dea7c151542fc789539599c07d0 00009623a06b8dea7c151542fc789539599c07d0.blob) 

I realized my mistake, dropped the commit referring to my butchered tree object (reset to new correct commit with correct tree).

The tree I created should no longer be reachable.

$ mv logs logs.bak

$ git cat-file --batch-all-objects --batch=objecttype --unordered --allow-unknown-type
objecttype
fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?

$ git cat-file -t 00009623a06b8dea7c151542fc789539599c07d0
tree

$ git cat-file -s 00009623a06b8dea7c151542fc789539599c07d0
2375

$ echo -e 'import zlib\nfrom hashlib import sha1\ndecompressed_contents=zlib.decompress(open("objects/00/009623a06b8dea7c151542fc789539599c07d0", "rb").read())\nprint sha1(decompressed_contents).hexdigest()\nprint decompressed_contents[0:19]' | python | hexdump -C
00000000  30 30 30 30 39 36 32 33  61 30 36 62 38 64 65 61  |00009623a06b8dea|
00000010  37 63 31 35 31 35 34 32  66 63 37 38 39 35 33 39  |7c151542fc789539|
00000020  35 39 39 63 30 37 64 30  0a 74 72 65 65 20 32 33  |599c07d0.tree 23|
00000030  37 35 00 31 30 30 36 34  34 20 2e 70 0a           |75.100644 .p.|
0000003d

$ git cat-file -p 00009623a06b8dea7c151542fc789539599c07d0 | cut -c -55 | head
100644 blob e465d57c345e2dcb117b5a30f9272b7fc5ec77cd    .p
100755 blob 7f16c1d4cbb75cf7bd635970a2588ced6ccea8ad    Ap
040000 tree 5261c0a3f3b4c688a082c3c5eaf03f8039bf153c    CA
100644 blob 188c0d0541523016352b6851e0f7200c18a372e6    CM
100644 blob c8b040ec356b21fcc06911c544149dc6f5d5b861    CM
100644 blob e441983f0fd4d57fb7bf640de31f728529f12c29    CM
100644 blob fd06c9c6ad662e099341f4e0a05b272c6370e64b    CM
100644 blob d433fb05ebca807f4487ae4cecf48ec3b66cce78    CM
100755 blob 4b1b049b83bcc7821a7b62977124bfcaa024d960    CM
040000 tree 150d60813c913ec9a178c4230b18fbda84edc2af    RE