Re: [PATCH] midx: disable replace objects

[Taylor Blau <me@ttaylorr.com>]

On Sun, Apr 07, 2024 at 01:11:47PM +0000, blanet via GitGitGadget wrote:
> From: Xing Xin <xingxin.xx@bytedance.com>
>
> We observed a series of clone failures arose in a specific set of
> repositories after we fully enabled the MIDX bitmap feature within our
> Codebase service. These failures were accompanied with error messages
> such as:
>
>   fatal: did not receive expected object ...
>   fatal: fetch-pack: invalid index-pack output
>
> Temporarily disabling the MIDX feature eliminated the reported issues.
> After some investigation we found that all repositories experiencing
> failures contain replace references, which seem to be improperly
> acknowledged by the MIDX bitmap generation logic.

I was suspicious that this might be related to the MIDX or MIDX bitmap,
but noticed something curious upon digging in. Applying the following on
top of your patch:

--- 8< ---
diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh
index 5e4cdef6a8..8543f8d097 100755
--- a/t/t5326-multi-pack-bitmaps.sh
+++ b/t/t5326-multi-pack-bitmaps.sh
@@ -451,9 +451,7 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' '
 		git rev-list --objects --no-object-names $B |sort >expected &&

 		git replace $A $C &&
-		git repack -ad &&
-		git multi-pack-index write --bitmap &&
-		git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&
+		git rev-list --objects --no-object-names $B |sort >actual &&
 		test_cmp expected actual
 	)
 '
--- >8 ---

, I can still produce the failure that you are seeing here. So I suspect
that while it's entirely possible that there is a bug in the MIDX/bitmap
code, that this test is not exercising it.

I think the first step to demonstrate a bug in the MIDX/bitmap machinery
would be to provide a reproducer that fails only when using a MIDX
and/or bitmap.

> @@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv,
>  	};
>  	struct option *options = parse_options_concat(builtin_multi_pack_index_options, common_opts);
>
> +	disable_replace_refs();
> +

Supposing for a moment that this issue is in the MIDX, we know that
regardless of what replace refs might be in place, the MIDX should only
be storing the objects that are in the packs being indexed, not the
objects which are their replacements.

Are we storing objects in the MIDX that are replacements? Looking
at midx.c::fill_pack_entry(), I think the answer is "no", since we're
looking up packed objects by calling nth_packed_object_id(), which is
just a table read into the .idx, all of which is beneath the level of
replace refs.

> @@ -434,6 +434,30 @@ test_expect_success 'tagged commits are selected for bitmapping' '
>  	)
>  '
>
> +test_expect_success 'do not follow replace objects for MIDX bitmap' '
> +	rm -fr repo &&
> +	git init repo &&
> +	test_when_finished "rm -fr repo" &&
> +	(
> +		cd repo &&
> +
> +		test_commit A &&
> +		A=$(git rev-parse HEAD) &&

It's possible that much of this will be moot if the current test gets
rewritten, but here are a couple of suggestions for writing tests in
Git's suite:

- test_commit will create a tag for you, so there is no need to store
  "$A", "$B", and "$C".

> +		test_commit B &&
> +		B=$(git rev-parse HEAD) &&
> +		git checkout --orphan=orphan $A &&
> +		test_commit orphan &&
> +		C=$(git rev-parse HEAD) &&
> +		git rev-list --objects --no-object-names $B |sort >expected &&

- We do not allow Git invocations on the left-hand side of a pipe, since
  doing so will squelch its exit code. Instead, try:

    git rev-list --objects --no-object-names B >expect.raw &&
    sort expect.raw >expect &&

Thanks,
Taylor