From: Jonathan Tan <jonathantanmy@google.com>
To: me@ttaylorr.com
Cc: git@vger.kernel.org, gitster@pobox.com, stolee@gmail.com,
Jonathan Tan <jonathantanmy@google.com>
Subject: Re: [PATCH v3 2/9] midx.c: make changing the preferred pack safe
Date: Thu, 20 Jan 2022 10:08:43 -0800 [thread overview]
Message-ID: <20220120180843.3103044-1-jonathantanmy@google.com> (raw)
In-Reply-To: <7d20c13f8b48d2aef45c2c8c40efb6ecdb865aa8.1641320129.git.me@ttaylorr.com>
Taylor Blau <me@ttaylorr.com> writes:
> The previous patch demonstrates a bug where a MIDX's auxiliary object
> order can become out of sync with a MIDX bitmap.
>
> This is because of two confounding factors:
>
> - First, the object order is stored in a file which is named according
> to the multi-pack index's checksum, and the MIDX does not store the
> object order. This means that the object order can change without
> altering the checksum.
>
> - But the .rev file is moved into place with finalize_object_file(),
> which link(2)'s the file into place instead of renaming it. For us,
> that means that a modified .rev file will not be moved into place if
> MIDX's checksum was unchanged.
>
> The fix here is two-fold. First, we need to stop linking the file into
> place and instead rename it. It's likely we were using
> finalize_object_file() instead of a pure rename() because the former
> also adjusts shared permissions. But that is unnecessary, because we
> already do so in write_rev_file_order(), so rename alone is safe.
>
> But we also need to make the MIDX's checksum change in some way when the
> preferred pack changes without altering the set of packs stored in a
> MIDX to prevent a race where the new .rev file is moved into place
> before the MIDX is updated. Here, you'd get the opposite effect: reading
> old bitmaps with the new object order.
I think the main issue is the first confounding factor you listed above:
even if we didn't have the other confounding factor, that issue alone is
enough to motivate the entire patch set. Likewise, as Junio said [1], I
don't think we need to switch to rename() if we make the checksum
different, so the fix is one-fold, not two-fold. For what it's worth, I
switched back to finalize_object_file() and ran the tests, and they all
pass.
So I would simplify the commit message to just talk about the checksum
issue. (This is definitely not a blocker for merging, though - others
might find the additional context helpful.)
The code up to this patch (apart from the rename()) looks good.
[1] https://lore.kernel.org/git/xmqqtue54iop.fsf@gitster.g/
next prev parent reply other threads:[~2022-01-20 18:08 UTC|newest]
Thread overview: 87+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-08 19:26 [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order Taylor Blau
2021-12-08 19:26 ` [PATCH 1/2] t5326: demonstrate bitmap corruption after permutation Taylor Blau
2021-12-08 19:26 ` [PATCH 2/2] midx.c: make changing the preferred pack safe Taylor Blau
2021-12-08 19:30 ` [PATCH 0/2] midx: prevent bitmap corruption when permuting pack order Derrick Stolee
2021-12-08 19:55 ` Jeff King
2021-12-10 18:36 ` Taylor Blau
2021-12-10 22:31 ` Taylor Blau
2021-12-11 1:39 ` Taylor Blau
2021-12-13 14:00 ` Derrick Stolee
2021-12-13 14:31 ` Taylor Blau
2021-12-14 1:55 ` [PATCH v2 0/8] " Taylor Blau
2021-12-14 1:55 ` [PATCH v2 1/8] t5326: demonstrate bitmap corruption after permutation Taylor Blau
2021-12-14 1:55 ` [PATCH v2 2/8] midx.c: make changing the preferred pack safe Taylor Blau
2021-12-14 1:55 ` [PATCH v2 3/8] pack-revindex.c: instrument loading on-disk reverse index Taylor Blau
2021-12-14 1:55 ` [PATCH v2 4/8] t5326: drop unnecessary setup Taylor Blau
2021-12-14 1:55 ` [PATCH v2 5/8] t5326: extract `test_rev_exists` Taylor Blau
2021-12-20 18:33 ` Derrick Stolee
2022-01-04 15:33 ` Taylor Blau
2021-12-14 1:55 ` [PATCH v2 6/8] t5326: move tests to t/lib-bitmap.sh Taylor Blau
2021-12-14 1:55 ` [PATCH v2 7/8] t/lib-bitmap.sh: parameterize tests over reverse index source Taylor Blau
2021-12-14 1:55 ` [PATCH v2 8/8] midx: read `RIDX` chunk when present Taylor Blau
2021-12-20 18:42 ` Derrick Stolee
2022-01-04 15:21 ` Taylor Blau
2021-12-15 19:46 ` [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order Junio C Hamano
2021-12-15 21:37 ` Taylor Blau
2021-12-15 22:17 ` Junio C Hamano
2021-12-15 22:55 ` Junio C Hamano
2021-12-20 18:51 ` Derrick Stolee
2021-12-20 19:52 ` Taylor Blau
2021-12-20 20:09 ` Derrick Stolee
2021-12-15 22:58 ` Junio C Hamano
2021-12-15 23:01 ` Taylor Blau
2022-01-04 18:15 ` [PATCH v3 0/9] " Taylor Blau
2022-01-04 18:15 ` [PATCH v3 1/9] t5326: demonstrate bitmap corruption after permutation Taylor Blau
2022-01-20 17:55 ` Jonathan Tan
2022-01-20 22:11 ` Taylor Blau
2022-01-20 22:41 ` Junio C Hamano
2022-01-20 22:46 ` Taylor Blau
2022-01-24 17:40 ` Jonathan Tan
2022-01-04 18:15 ` [PATCH v3 2/9] midx.c: make changing the preferred pack safe Taylor Blau
2022-01-14 21:35 ` Junio C Hamano
2022-01-14 21:43 ` Junio C Hamano
2022-01-15 0:59 ` Taylor Blau
2022-01-15 6:27 ` Junio C Hamano
2022-01-20 18:08 ` Jonathan Tan [this message]
2022-01-20 22:13 ` Taylor Blau
2022-01-04 18:15 ` [PATCH v3 3/9] pack-revindex.c: instrument loading on-disk reverse index Taylor Blau
2022-01-20 18:15 ` Jonathan Tan
2022-01-20 22:18 ` Taylor Blau
2022-01-24 17:53 ` Jonathan Tan
2022-01-04 18:15 ` [PATCH v3 4/9] t5326: drop unnecessary setup Taylor Blau
2022-01-04 18:15 ` [PATCH v3 5/9] t5326: extract `test_rev_exists` Taylor Blau
2022-01-04 18:15 ` [PATCH v3 6/9] t5326: move tests to t/lib-bitmap.sh Taylor Blau
2022-01-04 18:15 ` [PATCH v3 7/9] t/lib-bitmap.sh: parameterize tests over reverse index source Taylor Blau
2022-01-24 19:15 ` Jonathan Tan
2022-01-25 21:40 ` Taylor Blau
2022-01-26 21:00 ` Jonathan Tan
2022-01-04 18:16 ` [PATCH v3 8/9] midx: read `RIDX` chunk when present Taylor Blau
2022-01-24 19:27 ` Jonathan Tan
2022-01-25 21:45 ` Taylor Blau
2022-01-26 21:28 ` Jonathan Tan
2022-01-04 18:16 ` [PATCH v3 9/9] pack-bitmap.c: gracefully fallback after opening pack/MIDX Taylor Blau
2022-01-24 19:29 ` Jonathan Tan
2022-01-25 21:46 ` Taylor Blau
2022-01-25 22:40 ` [PATCH v4 0/9] midx: prevent bitmap corruption when permuting pack order Taylor Blau
2022-01-25 22:41 ` [PATCH v4 1/9] t5326: demonstrate bitmap corruption after permutation Taylor Blau
2022-01-26 15:01 ` Ævar Arnfjörð Bjarmason
2022-01-26 20:18 ` Taylor Blau
2022-01-25 22:41 ` [PATCH v4 2/9] midx.c: make changing the preferred pack safe Taylor Blau
2022-01-25 22:41 ` [PATCH v4 3/9] pack-revindex.c: instrument loading on-disk reverse index Taylor Blau
2022-01-26 15:03 ` Ævar Arnfjörð Bjarmason
2022-01-25 22:41 ` [PATCH v4 4/9] t5326: drop unnecessary setup Taylor Blau
2022-01-25 22:41 ` [PATCH v4 5/9] t5326: extract `test_rev_exists` Taylor Blau
2022-01-26 15:04 ` Ævar Arnfjörð Bjarmason
2022-01-26 20:19 ` Taylor Blau
2022-01-25 22:41 ` [PATCH v4 6/9] t5326: move tests to t/lib-bitmap.sh Taylor Blau
2022-01-25 22:41 ` [PATCH v4 7/9] t/lib-bitmap.sh: parameterize tests over reverse index source Taylor Blau
2022-01-25 22:41 ` [PATCH v4 8/9] midx: read `RIDX` chunk when present Taylor Blau
2022-01-26 15:10 ` Ævar Arnfjörð Bjarmason
2022-01-26 20:23 ` Taylor Blau
2022-01-25 22:41 ` [PATCH v4 9/9] pack-bitmap.c: gracefully fallback after opening pack/MIDX Taylor Blau
2022-01-26 15:08 ` Ævar Arnfjörð Bjarmason
2022-01-26 17:50 ` [PATCH v4 0/9] midx: prevent bitmap corruption when permuting pack order Ævar Arnfjörð Bjarmason
2022-01-26 20:24 ` Taylor Blau
2022-01-27 17:15 ` Jonathan Tan
2022-02-24 22:50 ` Taylor Blau
2022-01-27 14:13 ` Derrick Stolee
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220120180843.3103044-1-jonathantanmy@google.com \
--to=jonathantanmy@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=me@ttaylorr.com \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).