From: Patrick Steinhardt <ps@pks.im>
To: git@vger.kernel.org
Cc: Taylor Blau <me@ttaylorr.com>,
peff@peff.net, dstolee@microsoft.com,
Junio C Hamano <gitster@pobox.com>
Subject: [PATCH v4 10/10] repack: disable writing bitmaps when doing a local repack
Date: Fri, 14 Apr 2023 08:02:12 +0200 [thread overview]
Message-ID: <03503f9fc19523ce999f2baa20120c3f178f8d1a.1681452028.git.ps@pks.im> (raw)
In-Reply-To: <cover.1681452028.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 6746 bytes --]
In order to write a bitmap, we need to have full coverage of all objects
that are about to be packed. In the traditional non-multi-pack-index
world this meant we need to do a full repack of all objects into a
single packfile. But in the new multi-pack-index world we can get away
with writing bitmaps when we have multiple packfiles as long as the
multi-pack-index covers all objects.
This is not always the case though. When asked to perform a repack of
local objects, only, then we cannot guarantee to have full coverage of
all objects regardless of whether we do a full repack or a repack with a
multi-pack-index. The end result is that writing the bitmap will fail in
both worlds:
$ git multi-pack-index write --stdin-packs --bitmap <packfiles
warning: Failed to write bitmap index. Packfile doesn't have full closure (object 1529341d78cf45377407369acb0f4ff2b5cdae42 is missing)
error: could not write multi-pack bitmap
Now there are two different ways to fix this. The first one would be to
amend git-multi-pack-index(1) to disable writing bitmaps when we notice
that we don't have full object coverage.
- We don't have enough information in git-multi-pack-index(1) in
order to tell whether the local repository _should_ have full
coverage. Because even when connected to an alternate object
directory, it may be the case that we still have all objects
around in the main object database.
- git-multi-pack-index(1) is quite a low-level tool. Automatically
disabling functionality that it was asked to provide does not feel
like the right thing to do.
We can easily fix it at a higher level in git-repack(1) though. When
asked to only include local objects via `-l` and when connected to an
alternate object directory then we will override the user's ask and
disable writing bitmaps with a warning. This is similar to what we do in
git-pack-objects(1), where we also disable writing bitmaps in case we
omit an object from the pack.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
builtin/repack.c | 12 ++++++++++++
object-file.c | 6 ++++++
object-store.h | 1 +
t/t7700-repack.sh | 17 +++++++++++++++++
t/t7703-repack-geometric.sh | 27 +++++++++++++++++++++++++++
5 files changed, 63 insertions(+)
diff --git a/builtin/repack.c b/builtin/repack.c
index 5768aeb1f3..e2abcea2ee 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -887,6 +887,18 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
if (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)
die(_(incremental_bitmap_conflict_error));
+ if (write_bitmaps && po_args.local && has_alt_odb(the_repository)) {
+ /*
+ * When asked to do a local repack, but we have
+ * packfiles that are inherited from an alternate, then
+ * we cannot guarantee that the multi-pack-index would
+ * have full coverage of all objects. We thus disable
+ * writing bitmaps in that case.
+ */
+ warning(_("disabling bitmap writing, as some objects are not being packed"));
+ write_bitmaps = 0;
+ }
+
if (write_midx && write_bitmaps) {
struct strbuf path = STRBUF_INIT;
diff --git a/object-file.c b/object-file.c
index 8fab8dbe80..3ad11c683b 100644
--- a/object-file.c
+++ b/object-file.c
@@ -946,6 +946,12 @@ void prepare_alt_odb(struct repository *r)
r->objects->loaded_alternates = 1;
}
+int has_alt_odb(struct repository *r)
+{
+ prepare_alt_odb(r);
+ return !!r->objects->odb->next;
+}
+
/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
static int freshen_file(const char *fn)
{
diff --git a/object-store.h b/object-store.h
index 82201ec3e7..574f16140f 100644
--- a/object-store.h
+++ b/object-store.h
@@ -56,6 +56,7 @@ KHASH_INIT(odb_path_map, const char * /* key: odb_path */,
struct object_directory *, 1, fspathhash, fspatheq)
void prepare_alt_odb(struct repository *r);
+int has_alt_odb(struct repository *r);
char *compute_alternate_path(const char *path, struct strbuf *err);
struct object_directory *find_odb(struct repository *r, const char *obj_dir);
typedef int alt_odb_fn(struct object_directory *, void *);
diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh
index 4aabe98139..faa739eeb9 100755
--- a/t/t7700-repack.sh
+++ b/t/t7700-repack.sh
@@ -106,6 +106,23 @@ test_expect_success SYMLINKS '--local keeps packs when alternate is objectdir '
test_cmp expect actual
'
+test_expect_success '--local disables writing bitmaps when connected to alternate ODB' '
+ test_when_finished "rm -rf shared member" &&
+
+ git init shared &&
+ git clone --shared shared member &&
+ (
+ cd member &&
+ test_commit "object" &&
+ GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adl --write-bitmap-index 2>err &&
+ cat >expect <<-EOF &&
+ warning: disabling bitmap writing, as some objects are not being packed
+ EOF
+ test_cmp expect err &&
+ test_path_is_missing .git/objects/pack-*.bitmap
+ )
+'
+
test_expect_success 'packed obs in alt ODB are repacked even when local repo is packless' '
mkdir alt_objects/pack &&
mv .git/objects/pack/* alt_objects/pack &&
diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh
index be8fe78448..00f28fb558 100755
--- a/t/t7703-repack-geometric.sh
+++ b/t/t7703-repack-geometric.sh
@@ -418,4 +418,31 @@ test_expect_success '--geometric -l with non-intact geometric sequence across OD
test_cmp expected-objects actual-objects
'
+test_expect_success '--geometric -l disables writing bitmaps with non-local packfiles' '
+ test_when_finished "rm -fr shared member" &&
+
+ git init shared &&
+ test_commit_bulk -C shared --start=1 1 &&
+
+ git clone --shared shared member &&
+ test_commit_bulk -C member --start=2 1 &&
+
+ # When performing a geometric repack with `-l` while connected to an
+ # alternate object database that has a packfile we do not have full
+ # coverage of objects. As a result, we expect that writing the bitmap
+ # will be disabled.
+ git -C member repack -l --geometric=2 --write-midx --write-bitmap-index 2>err &&
+ cat >expect <<-EOF &&
+ warning: disabling bitmap writing, as some objects are not being packed
+ EOF
+ test_cmp expect err &&
+ test_path_is_missing member/.git/objects/pack/multi-pack-index-*.bitmap &&
+
+ # On the other hand, when we repack without `-l`, we should see that
+ # the bitmap gets created.
+ git -C member repack --geometric=2 --write-midx --write-bitmap-index 2>err &&
+ test_must_be_empty err &&
+ test_path_is_file member/.git/objects/pack/multi-pack-index-*.bitmap
+'
+
test_done
--
2.40.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2023-04-14 6:02 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-04-04 11:08 [PATCH] repack: fix geometric repacking with gitalternates Patrick Steinhardt
2023-04-04 18:55 ` Taylor Blau
2023-04-04 19:00 ` Taylor Blau
2023-04-05 7:08 ` Patrick Steinhardt
2023-04-10 15:06 ` Derrick Stolee
2023-04-10 23:49 ` Taylor Blau
2023-04-11 17:13 ` Patrick Steinhardt
2023-04-11 21:13 ` Taylor Blau
2023-04-12 9:37 ` Patrick Steinhardt
2023-04-11 17:06 ` Patrick Steinhardt
2023-04-11 17:26 ` Patrick Steinhardt
2023-04-11 21:14 ` Taylor Blau
2023-04-10 23:29 ` Taylor Blau
2023-04-12 10:22 ` [PATCH v2 0/8] " Patrick Steinhardt
2023-04-12 10:22 ` [PATCH v2 1/8] midx: fix segfault with no packs and invalid preferred pack Patrick Steinhardt
2023-04-12 17:56 ` Taylor Blau
2023-04-13 9:28 ` Patrick Steinhardt
2023-04-12 10:22 ` [PATCH v2 2/8] repack: fix trying to use preferred pack in alternates Patrick Steinhardt
2023-04-12 18:37 ` Taylor Blau
2023-04-13 9:31 ` Patrick Steinhardt
2023-04-12 10:22 ` [PATCH v2 3/8] repack: fix generating multi-pack-index with only non-local packs Patrick Steinhardt
2023-04-12 20:39 ` Taylor Blau
2023-04-12 10:22 ` [PATCH v2 4/8] pack-objects: fix error when packing same pack twice Patrick Steinhardt
2023-04-12 21:33 ` Taylor Blau
2023-04-12 10:22 ` [PATCH v2 5/8] pack-objects: fix error when same packfile is included and excluded Patrick Steinhardt
2023-04-12 21:52 ` Taylor Blau
2023-04-12 10:22 ` [PATCH v2 6/8] pack-objects: extend test coverage of `--stdin-packs` with alternates Patrick Steinhardt
2023-04-12 10:22 ` [PATCH v2 7/8] repack: honor `-l` when calculating pack geometry Patrick Steinhardt
2023-04-12 23:56 ` Junio C Hamano
2023-04-13 5:11 ` Junio C Hamano
2023-04-13 6:41 ` Patrick Steinhardt
2023-04-12 10:23 ` [PATCH v2 8/8] repack: disable writing bitmaps when doing a local geometric repack Patrick Steinhardt
2023-04-12 22:01 ` Taylor Blau
2023-04-13 9:54 ` Patrick Steinhardt
2023-04-13 10:14 ` Patrick Steinhardt
2023-04-12 22:02 ` [PATCH v2 0/8] repack: fix geometric repacking with gitalternates Taylor Blau
2023-04-13 11:16 ` [PATCH v3 00/10] " Patrick Steinhardt
2023-04-13 11:16 ` [PATCH v3 01/10] midx: fix segfault with no packs and invalid preferred pack Patrick Steinhardt
2023-04-13 13:49 ` Derrick Stolee
2023-04-13 11:16 ` [PATCH v3 02/10] repack: fix trying to use preferred pack in alternates Patrick Steinhardt
2023-04-13 11:16 ` [PATCH v3 03/10] repack: fix generating multi-pack-index with only non-local packs Patrick Steinhardt
2023-04-13 11:16 ` [PATCH v3 04/10] pack-objects: split out `--stdin-packs` tests into separate file Patrick Steinhardt
2023-04-13 11:16 ` [PATCH v3 05/10] pack-objects: fix error when packing same pack twice Patrick Steinhardt
2023-04-13 11:16 ` [PATCH v3 06/10] pack-objects: fix error when same packfile is included and excluded Patrick Steinhardt
2023-04-13 11:16 ` [PATCH v3 07/10] pack-objects: extend test coverage of `--stdin-packs` with alternates Patrick Steinhardt
2023-04-13 11:16 ` [PATCH v3 08/10] t/helper: allow chmtime to print verbosely without modifying mtime Patrick Steinhardt
2023-04-13 11:16 ` [PATCH v3 09/10] repack: honor `-l` when calculating pack geometry Patrick Steinhardt
2023-04-13 13:59 ` Derrick Stolee
2023-04-13 14:13 ` Patrick Steinhardt
2023-04-13 15:40 ` Junio C Hamano
2023-04-13 11:16 ` [PATCH v3 10/10] repack: disable writing bitmaps when doing a local repack Patrick Steinhardt
2023-04-13 14:54 ` [PATCH v3 00/10] repack: fix geometric repacking with gitalternates Derrick Stolee
2023-04-14 2:03 ` Junio C Hamano
2023-04-14 5:42 ` Patrick Steinhardt
2023-04-14 6:01 ` [PATCH v4 " Patrick Steinhardt
2023-04-14 6:01 ` [PATCH v4 01/10] midx: fix segfault with no packs and invalid preferred pack Patrick Steinhardt
2023-04-14 6:01 ` [PATCH v4 02/10] repack: fix trying to use preferred pack in alternates Patrick Steinhardt
2023-04-14 6:01 ` [PATCH v4 03/10] repack: fix generating multi-pack-index with only non-local packs Patrick Steinhardt
2023-04-14 6:01 ` [PATCH v4 04/10] pack-objects: split out `--stdin-packs` tests into separate file Patrick Steinhardt
2023-04-14 6:01 ` [PATCH v4 05/10] pack-objects: fix error when packing same pack twice Patrick Steinhardt
2023-04-14 6:01 ` [PATCH v4 06/10] pack-objects: fix error when same packfile is included and excluded Patrick Steinhardt
2023-04-14 6:01 ` [PATCH v4 07/10] pack-objects: extend test coverage of `--stdin-packs` with alternates Patrick Steinhardt
2023-04-14 6:02 ` [PATCH v4 08/10] t/helper: allow chmtime to print verbosely without modifying mtime Patrick Steinhardt
2023-04-14 6:02 ` [PATCH v4 09/10] repack: honor `-l` when calculating pack geometry Patrick Steinhardt
2023-04-14 6:02 ` Patrick Steinhardt [this message]
2023-04-14 13:23 ` [PATCH v4 00/10] repack: fix geometric repacking with gitalternates Derrick Stolee
2023-04-14 17:29 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=03503f9fc19523ce999f2baa20120c3f178f8d1a.1681452028.git.ps@pks.im \
--to=ps@pks.im \
--cc=dstolee@microsoft.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=me@ttaylorr.com \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).