git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Taylor Blau <me@ttaylorr.com>
To: git@vger.kernel.org
Cc: peff@peff.net
Subject: [PATCH 2/4] pack-bitmap.c: propagate namehash values from existing bitmaps
Date: Tue, 7 Sep 2021 17:17:58 -0400	[thread overview]
Message-ID: <fa9f5633f6a0955c045168c80d4d9d7987a8327d.1631049462.git.me@ttaylorr.com> (raw)
In-Reply-To: <cover.1631049462.git.me@ttaylorr.com>

When an old bitmap exists while writing a new one, we load it and build
a "reposition" table which maps bit positions of objects from the old
bitmap to their respective positions in the new bitmap. This can help
when we encounter a commit which was selected in both the old and new
bitmap, since we only need to permute its bit (not recompute it from
scratch).

We do not, however, repurpose existing namehash values in the case of
the hash-cache extension. There has been thus far no good reason to do
so, since all of the namehash values for objects in the new bitmap would
be populated during the traversal that was just performed by
pack-objects when generating single-pack reachability bitmaps.

But this isn't the case for multi-pack bitmaps, which are written via
`git multi-pack-index write --bitmap` and do not perform any traversal.
In this case all namehash values are set to zero, but we don't even
bother to check the `pack.writeBitmapHashcache` option anyway, so it
fails to matter.

There are two approaches we could take to fill in non-zero hash-cache
values:

  - have either the multi-pack-index builtin run its own
    traversal to attempt to fill in some values, or let a hypothetical
    caller (like `pack-objects` when `repack` eventually drives the
    `multi-pack-index` builtin) fill in the values they found during
    their traversal

  - or copy any existing namehash values that were stored in an
    existing bitmap to their corresponding positions in the new bitmap

In a system where a repository is generally repacked with `git repack
--geometric=<d>` and occasionally repacked with `git repack -a`, the
hash-cache coverage will tend towards all objects.

Since populating the hash-cache is additive (i.e., doing so only helps
our delta search), any intermediate lack of full coverage is just fine.
So let's start by just propagating any values from the existing
hash-cache if we see one.

The next patch will respect the `pack.writeBitmapHashcache` option while
writing MIDX bitmaps, and then test this new behavior.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 pack-bitmap.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/pack-bitmap.c b/pack-bitmap.c
index e44af36933..cb876e7e9d 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -1815,18 +1815,20 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,
 	for (i = 0; i < num_objects; ++i) {
 		struct object_id oid;
 		struct object_entry *oe;
+		uint32_t index_pos;
 
 		if (bitmap_is_midx(bitmap_git))
-			nth_midxed_object_oid(&oid,
-					      bitmap_git->midx,
-					      pack_pos_to_midx(bitmap_git->midx, i));
+			index_pos = pack_pos_to_midx(bitmap_git->midx, i);
 		else
-			nth_packed_object_id(&oid, bitmap_git->pack,
-					     pack_pos_to_index(bitmap_git->pack, i));
+			index_pos = pack_pos_to_index(bitmap_git->pack, i);
+		nth_bitmap_object_oid(bitmap_git, &oid, index_pos);
 		oe = packlist_find(mapping, &oid);
 
-		if (oe)
+		if (oe) {
 			reposition[i] = oe_in_pack_pos(mapping, oe) + 1;
+			if (bitmap_git->hashes && !oe->hash)
+				oe->hash = get_be32(bitmap_git->hashes + index_pos);
+		}
 	}
 
 	return reposition;
-- 
2.33.0.96.g73915697e6


  parent reply	other threads:[~2021-09-07 21:18 UTC|newest]

Thread overview: 55+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-07 21:17 [PATCH 0/4] pack-bitmap: permute existing namehash values Taylor Blau
2021-09-07 21:17 ` [PATCH 1/4] t/helper/test-bitmap.c: add 'dump-hashes' mode Taylor Blau
2021-09-08  1:37   ` Ævar Arnfjörð Bjarmason
2021-09-08  2:24     ` Taylor Blau
2021-09-07 21:17 ` Taylor Blau [this message]
2021-09-07 21:18 ` [PATCH 3/4] midx.c: respect 'pack.writeBitmapHashcache' when writing bitmaps Taylor Blau
2021-09-08  1:40   ` Ævar Arnfjörð Bjarmason
2021-09-08  2:28     ` Taylor Blau
2021-09-09  8:18       ` Ævar Arnfjörð Bjarmason
2021-09-09  9:34         ` Ævar Arnfjörð Bjarmason
2021-09-09 14:55           ` Taylor Blau
2021-09-09 15:50             ` Ævar Arnfjörð Bjarmason
2021-09-09 16:23               ` Taylor Blau
2021-09-09 14:47         ` Taylor Blau
2021-09-13  0:38   ` Junio C Hamano
2021-09-14  1:15     ` Taylor Blau
2021-09-07 21:18 ` [PATCH 4/4] t5326: test propagating hashcache values Taylor Blau
2021-09-08  1:46   ` Ævar Arnfjörð Bjarmason
2021-09-08  2:30     ` Taylor Blau
2021-09-17  8:56       ` Ævar Arnfjörð Bjarmason
2021-09-17 17:32         ` Taylor Blau
2021-09-17 19:22           ` Ævar Arnfjörð Bjarmason
2021-09-13  0:46   ` Junio C Hamano
2021-09-14  1:12     ` Taylor Blau
2021-09-14  2:05       ` Junio C Hamano
2021-09-14  5:11         ` Taylor Blau
2021-09-14  5:17           ` Taylor Blau
2021-09-14  5:27           ` Jeff King
2021-09-14  5:31             ` Taylor Blau
2021-09-14  5:23         ` Jeff King
2021-09-14  5:49           ` Junio C Hamano
2021-09-14 22:05 ` [PATCH v2 0/7] pack-bitmap: permute existing namehash values Taylor Blau
2021-09-14 22:06   ` [PATCH v2 1/7] t/helper/test-bitmap.c: add 'dump-hashes' mode Taylor Blau
2021-09-14 22:06   ` [PATCH v2 2/7] pack-bitmap.c: propagate namehash values from existing bitmaps Taylor Blau
2021-09-14 22:06   ` [PATCH v2 3/7] midx.c: respect 'pack.writeBitmapHashcache' when writing bitmaps Taylor Blau
2021-09-14 22:06   ` [PATCH v2 4/7] p5326: create missing 'perf-tag' tag Taylor Blau
2021-09-16 22:36     ` Jeff King
2021-09-17  4:14       ` Taylor Blau
2021-09-14 22:06   ` [PATCH v2 5/7] p5326: don't set core.multiPackIndex unnecessarily Taylor Blau
2021-09-16 22:38     ` Jeff King
2021-09-14 22:06   ` [PATCH v2 6/7] p5326: generate pack bitmaps before writing the MIDX bitmap Taylor Blau
2021-09-16 22:45     ` Jeff King
2021-09-17  4:20       ` Taylor Blau
2021-09-14 22:06   ` [PATCH v2 7/7] t5326: test propagating hashcache values Taylor Blau
2021-09-16 22:49     ` Jeff King
2021-09-16 22:52   ` [PATCH v2 0/7] pack-bitmap: permute existing namehash values Jeff King
2021-09-17 21:21 ` [PATCH v3 " Taylor Blau
2021-09-17 21:21   ` [PATCH v3 1/7] t/helper/test-bitmap.c: add 'dump-hashes' mode Taylor Blau
2021-09-17 21:21   ` [PATCH v3 2/7] pack-bitmap.c: propagate namehash values from existing bitmaps Taylor Blau
2021-09-17 21:21   ` [PATCH v3 3/7] midx.c: respect 'pack.writeBitmapHashcache' when writing bitmaps Taylor Blau
2021-09-17 21:21   ` [PATCH v3 4/7] p5326: create missing 'perf-tag' tag Taylor Blau
2021-09-17 21:21   ` [PATCH v3 5/7] p5326: don't set core.multiPackIndex unnecessarily Taylor Blau
2021-09-17 21:21   ` [PATCH v3 6/7] p5326: generate pack bitmaps before writing the MIDX bitmap Taylor Blau
2021-09-17 21:21   ` [PATCH v3 7/7] t5326: test propagating hashcache values Taylor Blau
2021-09-17 22:12   ` [PATCH v3 0/7] pack-bitmap: permute existing namehash values Jeff King

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=fa9f5633f6a0955c045168c80d4d9d7987a8327d.1631049462.git.me@ttaylorr.com \
    --to=me@ttaylorr.com \
    --cc=git@vger.kernel.org \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).