git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Patrick Steinhardt <ps@pks.im>
To: git@vger.kernel.org
Cc: Taylor Blau <me@ttaylorr.com>, peff@peff.net, dstolee@microsoft.com
Subject: [PATCH] repack: fix geometric repacking with gitalternates
Date: Tue, 4 Apr 2023 13:08:33 +0200	[thread overview]
Message-ID: <a07ed50feeec4bfc3e9736bf493b9876896bcdd2.1680606445.git.ps@pks.im> (raw)

[-- Attachment #1: Type: text/plain, Size: 4757 bytes --]

Performing geometric repacking with repositories that have alternate
object directories set up is causing errors in different scenarios.

- Repacking fails when the repository linked to the object
  directory does not have any objects on its own.

  ```
  $ git init shared
  $ git -C shared commit --allow-empty --message something
  $ git clone --shared shared member
  $ git -C member repack --geometric=2 --write-midx
  Nothing new to pack.
  warning: unknown preferred pack: 'pack-3e1a94a8dc9bb4defb0d98ce2ebf325312d76362.pack'
  error: multi-pack-index died of signal 7
  ```

- Repacking fails when the repository linked to the object
  directory has the exact same packfile as the linked-to object
  directory.

  ```
  $ git init shared
  $ git -C shared commit --allow-empty --message something
  $ git -C shared repack -Ad
  $ cp -r shared member
  $ realpath shared/.git/objects >member/.git/objects/info/alternates
  $ git -C member repack --geometric 2 --write-midx
  fatal: could not find pack 'pack-d404037a861afe456e07a1aefb3655150f1299f0.pack'
  ```

Both issues have the same underlying root cause, which is that geometric
repacks don't honor whether packfiles are local or not. As a result,
they will try to include packs part of the alternate object directory
and then at a later point fail to locate them as they do not exist in
the object directory of the repository we're about to repack.

Skip over packfiles that aren't local. This will cause geometric repacks
to never include packfiles of its alternates.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/repack.c            |  6 ++++
 t/t7703-repack-geometric.sh | 59 +++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+)

diff --git a/builtin/repack.c b/builtin/repack.c
index 87f73c8923..c6d12fa4bd 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -333,6 +333,12 @@ static void init_pack_geometry(struct pack_geometry **geometry_p,
 	geometry = *geometry_p;
 
 	for (p = get_all_packs(the_repository); p; p = p->next) {
+		/*
+		 * We don't want to repack packfiles which are not part of the
+		 * primary object database.
+		 */
+		if (!p->pack_local)
+			continue;
 		if (!pack_kept_objects) {
 			/*
 			 * Any pack that has its pack_keep bit set will appear
diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh
index 8821fbd2dd..9f8bc663e4 100755
--- a/t/t7703-repack-geometric.sh
+++ b/t/t7703-repack-geometric.sh
@@ -281,4 +281,63 @@ test_expect_success '--geometric with pack.packSizeLimit' '
 	)
 '
 
+packed_objects() {
+	git verify-pack -v "$@" >tmp-object-list &&
+	sed -n -e "s/^\([0-9a-f][0-9a-f]*\).*\(commit\|tree\|blob\|tag\).*/\1/p" <tmp-object-list &&
+	rm -f tmp-object-list
+}
+
+test_expect_success '--geometric with no local objects creates no pack' '
+	git init shared &&
+	test_when_finished "rm -fr shared" &&
+	test_commit -C shared "shared" &&
+	git -C shared repack -Ad &&
+
+	# Set up the member repository so that it does not have
+	# any objects on its own.
+	git clone --shared shared member &&
+	test_when_finished "rm -fr member" &&
+
+	git -C member repack --geometric 2 --write-midx &&
+	find member/.git/objects/pack -type f >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success '--geometric does not include shared packfiles' '
+	git init shared &&
+	test_when_finished "rm -fr shared" &&
+	test_commit -C shared "shared" &&
+	git -C shared repack -Ad &&
+
+	git clone --shared shared member &&
+	test_when_finished "rm -fr member" &&
+	git -C member commit --allow-empty --message "not-shared" &&
+	git -C member repack --geometric 2 --write-midx &&
+
+	# We expect the created packfile to only contain the new commit.
+	packed_objects member/.git/objects/pack/pack-*.idx >actual &&
+	git -C member rev-parse HEAD >expect &&
+	test_cmp expect actual
+'
+
+test_expect_success '--geometric with same packfile in shared repository' '
+	git init shared &&
+	test_when_finished "rm -fr shared" &&
+	test_commit -C shared "shared" &&
+	git -C shared repack -Ad &&
+
+	# Prepare the member repository so that it got the exact same packfile
+	# as the shared repository and set up gitalternates.
+	cp -r shared member &&
+	test_when_finished "rm -fr member" &&
+	test-tool path-utils real_path shared/.git/objects >member/.git/objects/info/alternates &&
+	find shared/.git/objects -type f >expect &&
+
+	# After repacking, contents of the member repository should not have
+	# changed.
+	git -C member repack --geometric 2 --write-midx 2>error &&
+	find shared/.git/objects -type f >actual &&
+	test_cmp expect actual
+'
+
 test_done
-- 
2.40.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

             reply	other threads:[~2023-04-04 11:10 UTC|newest]

Thread overview: 67+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-04-04 11:08 Patrick Steinhardt [this message]
2023-04-04 18:55 ` [PATCH] repack: fix geometric repacking with gitalternates Taylor Blau
2023-04-04 19:00   ` Taylor Blau
2023-04-05  7:08   ` Patrick Steinhardt
2023-04-10 15:06     ` Derrick Stolee
2023-04-10 23:49       ` Taylor Blau
2023-04-11 17:13         ` Patrick Steinhardt
2023-04-11 21:13           ` Taylor Blau
2023-04-12  9:37             ` Patrick Steinhardt
2023-04-11 17:06       ` Patrick Steinhardt
2023-04-11 17:26         ` Patrick Steinhardt
2023-04-11 21:14           ` Taylor Blau
2023-04-10 23:29     ` Taylor Blau
2023-04-12 10:22 ` [PATCH v2 0/8] " Patrick Steinhardt
2023-04-12 10:22   ` [PATCH v2 1/8] midx: fix segfault with no packs and invalid preferred pack Patrick Steinhardt
2023-04-12 17:56     ` Taylor Blau
2023-04-13  9:28       ` Patrick Steinhardt
2023-04-12 10:22   ` [PATCH v2 2/8] repack: fix trying to use preferred pack in alternates Patrick Steinhardt
2023-04-12 18:37     ` Taylor Blau
2023-04-13  9:31       ` Patrick Steinhardt
2023-04-12 10:22   ` [PATCH v2 3/8] repack: fix generating multi-pack-index with only non-local packs Patrick Steinhardt
2023-04-12 20:39     ` Taylor Blau
2023-04-12 10:22   ` [PATCH v2 4/8] pack-objects: fix error when packing same pack twice Patrick Steinhardt
2023-04-12 21:33     ` Taylor Blau
2023-04-12 10:22   ` [PATCH v2 5/8] pack-objects: fix error when same packfile is included and excluded Patrick Steinhardt
2023-04-12 21:52     ` Taylor Blau
2023-04-12 10:22   ` [PATCH v2 6/8] pack-objects: extend test coverage of `--stdin-packs` with alternates Patrick Steinhardt
2023-04-12 10:22   ` [PATCH v2 7/8] repack: honor `-l` when calculating pack geometry Patrick Steinhardt
2023-04-12 23:56     ` Junio C Hamano
2023-04-13  5:11       ` Junio C Hamano
2023-04-13  6:41         ` Patrick Steinhardt
2023-04-12 10:23   ` [PATCH v2 8/8] repack: disable writing bitmaps when doing a local geometric repack Patrick Steinhardt
2023-04-12 22:01     ` Taylor Blau
2023-04-13  9:54       ` Patrick Steinhardt
2023-04-13 10:14         ` Patrick Steinhardt
2023-04-12 22:02   ` [PATCH v2 0/8] repack: fix geometric repacking with gitalternates Taylor Blau
2023-04-13 11:16 ` [PATCH v3 00/10] " Patrick Steinhardt
2023-04-13 11:16   ` [PATCH v3 01/10] midx: fix segfault with no packs and invalid preferred pack Patrick Steinhardt
2023-04-13 13:49     ` Derrick Stolee
2023-04-13 11:16   ` [PATCH v3 02/10] repack: fix trying to use preferred pack in alternates Patrick Steinhardt
2023-04-13 11:16   ` [PATCH v3 03/10] repack: fix generating multi-pack-index with only non-local packs Patrick Steinhardt
2023-04-13 11:16   ` [PATCH v3 04/10] pack-objects: split out `--stdin-packs` tests into separate file Patrick Steinhardt
2023-04-13 11:16   ` [PATCH v3 05/10] pack-objects: fix error when packing same pack twice Patrick Steinhardt
2023-04-13 11:16   ` [PATCH v3 06/10] pack-objects: fix error when same packfile is included and excluded Patrick Steinhardt
2023-04-13 11:16   ` [PATCH v3 07/10] pack-objects: extend test coverage of `--stdin-packs` with alternates Patrick Steinhardt
2023-04-13 11:16   ` [PATCH v3 08/10] t/helper: allow chmtime to print verbosely without modifying mtime Patrick Steinhardt
2023-04-13 11:16   ` [PATCH v3 09/10] repack: honor `-l` when calculating pack geometry Patrick Steinhardt
2023-04-13 13:59     ` Derrick Stolee
2023-04-13 14:13       ` Patrick Steinhardt
2023-04-13 15:40       ` Junio C Hamano
2023-04-13 11:16   ` [PATCH v3 10/10] repack: disable writing bitmaps when doing a local repack Patrick Steinhardt
2023-04-13 14:54   ` [PATCH v3 00/10] repack: fix geometric repacking with gitalternates Derrick Stolee
2023-04-14  2:03   ` Junio C Hamano
2023-04-14  5:42     ` Patrick Steinhardt
2023-04-14  6:01 ` [PATCH v4 " Patrick Steinhardt
2023-04-14  6:01   ` [PATCH v4 01/10] midx: fix segfault with no packs and invalid preferred pack Patrick Steinhardt
2023-04-14  6:01   ` [PATCH v4 02/10] repack: fix trying to use preferred pack in alternates Patrick Steinhardt
2023-04-14  6:01   ` [PATCH v4 03/10] repack: fix generating multi-pack-index with only non-local packs Patrick Steinhardt
2023-04-14  6:01   ` [PATCH v4 04/10] pack-objects: split out `--stdin-packs` tests into separate file Patrick Steinhardt
2023-04-14  6:01   ` [PATCH v4 05/10] pack-objects: fix error when packing same pack twice Patrick Steinhardt
2023-04-14  6:01   ` [PATCH v4 06/10] pack-objects: fix error when same packfile is included and excluded Patrick Steinhardt
2023-04-14  6:01   ` [PATCH v4 07/10] pack-objects: extend test coverage of `--stdin-packs` with alternates Patrick Steinhardt
2023-04-14  6:02   ` [PATCH v4 08/10] t/helper: allow chmtime to print verbosely without modifying mtime Patrick Steinhardt
2023-04-14  6:02   ` [PATCH v4 09/10] repack: honor `-l` when calculating pack geometry Patrick Steinhardt
2023-04-14  6:02   ` [PATCH v4 10/10] repack: disable writing bitmaps when doing a local repack Patrick Steinhardt
2023-04-14 13:23   ` [PATCH v4 00/10] repack: fix geometric repacking with gitalternates Derrick Stolee
2023-04-14 17:29     ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a07ed50feeec4bfc3e9736bf493b9876896bcdd2.1680606445.git.ps@pks.im \
    --to=ps@pks.im \
    --cc=dstolee@microsoft.com \
    --cc=git@vger.kernel.org \
    --cc=me@ttaylorr.com \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).