From: Patrick Steinhardt <ps@pks.im>
To: git@vger.kernel.org
Cc: Derrick Stolee <stolee@gmail.com>
Subject: [PATCH 00/15] refs: introduce `--auto` to pack refs as needed
Date: Mon, 18 Mar 2024 11:52:22 +0100 [thread overview]
Message-ID: <cover.1710706118.git.ps@pks.im> (raw)
[-- Attachment #1: Type: text/plain, Size: 4220 bytes --]
Hi,
right now, it is left to the caller of `refs_pack_refs()` to decide
whether or not to repack refs and how exactly to do it. This is
inefficient at times given that the caller typically has no idea whether
or not the refdb is close to optimal already. So even if the refdb is
already in a close-to-optimal state we would end up repacking all of it.
This patch series aims to address this shortcoming. It introduces a new
flag `PACK_REFS_AUTO` that tells the ref backend to repack refs as
needed. For the "files" backend we don't honor this flag yet and thus
end up behaving the exact same as if that flag wasn't set. But for the
"reftable" backend we will use the same auto-compaction algorithm that
is already used during writes to the refdb. Thus, in most of the cases
it wouldn't actually do anything except when the refdb is suboptimally
packed.
Eventually we'll probably also want to wire up heuristics for the
"files" backend to honor the `PACK_REFS_AUTO` flag. Using something like
a ratio of packed-refs size/number of loose refs might be viable. I
punted on that though as I feared that it might lead to bikeshedding and
thus distract from the main goal of this topic, which is to prepare the
ref code and relevant commands to perform optimizations as required. I'm
happy to add such a patch to this series though in case anybody feels
strongly about this.
The `PACK_REFS_AUTO` flag is exposed via a new `git pack-refs --auto`
flag. It is wired up in both `git gc --auto` and `git maintenance run
--auto`.
The series is structured as follows:
- Patches 1 - 5: Bugfixes and improvements for the reftable-specific
compaction code. These are issues that I've found while working on
this series.
- Patches 6 - 8: Refactorings to drop the `PACK_REFS_ALL` flag,
which isn't actually used by the ref backends anymore and confused
me multiple times.
- Patches 9 - 15: Handling of `PACK_REFS_ALL` in git-pack-refs(1),
git-gc(1) and git-maintenance(1).
The patch series is built on top of 2953d95d40 (The eighth batch,
2024-03-15) with "ps/reftable-stack-tempfile" at 60c4c42515
(reftable/stack: register compacted tables as tempfiles, 2024-03-07)
merged into it due to a merge conflict in "reftable/stack.c".
Patrick
PS: I polished this patch series while traveling and am still out of
office until Thursday. I'll thus only get to respond to threads that
are waiting for my input at the end of this week.
Patrick Steinhardt (15):
reftable/stack: fix error handling in `reftable_stack_init_addition()`
reftable/error: discern locked/outdated errors
reftable/stack: use error codes when locking fails during compaction
reftable/stack: gracefully handle failed auto-compaction due to locks
refs/reftable: print errors on compaction failure
t/helper: drop pack-refs wrapper
refs: move `struct pack_refs_opts` to where it's used
refs: remove `PACK_REFS_ALL` flag
refs/reftable: expose auto compaction via new flag
builtin/pack-refs: release allocated memory
builtin/pack-refs: introduce new "--auto" flag
builtin/gc: move `struct maintenance_run_opts`
t6500: extract objects with "17" prefix
builtin/gc: forward git-gc(1)'s `--auto` flag when packing refs
builtin/gc: pack refs when using `git maintenance run --auto`
Documentation/git-pack-refs.txt | 15 +++++-
builtin/gc.c | 86 +++++++++++++++++++--------------
builtin/pack-refs.c | 31 +++++++-----
refs.h | 20 ++++----
refs/reftable-backend.c | 11 ++++-
reftable/error.c | 4 +-
reftable/reftable-error.h | 5 +-
reftable/stack.c | 34 +++++++------
reftable/stack_test.c | 2 +-
t/helper/test-ref-store.c | 20 --------
t/oid-info/hash-info | 12 +++++
t/t0601-reffiles-pack-refs.sh | 30 ++++++++++--
t/t0610-reftable-basics.sh | 79 ++++++++++++++++++++++++++++++
t/t6500-gc.sh | 30 +++---------
14 files changed, 255 insertions(+), 124 deletions(-)
base-commit: f94fee98b456ef39a5f85fff5802f4e538a4dc7b
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next reply other threads:[~2024-03-18 10:52 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-18 10:52 Patrick Steinhardt [this message]
2024-03-18 10:52 ` [PATCH 01/15] reftable/stack: fix error handling in `reftable_stack_init_addition()` Patrick Steinhardt
2024-03-20 21:50 ` Karthik Nayak
2024-03-25 9:10 ` Patrick Steinhardt
2024-03-18 10:52 ` [PATCH 02/15] reftable/error: discern locked/outdated errors Patrick Steinhardt
2024-03-18 10:52 ` [PATCH 03/15] reftable/stack: use error codes when locking fails during compaction Patrick Steinhardt
2024-03-20 22:14 ` Karthik Nayak
2024-03-25 9:10 ` Patrick Steinhardt
2024-03-18 10:52 ` [PATCH 04/15] reftable/stack: gracefully handle failed auto-compaction due to locks Patrick Steinhardt
2024-03-20 22:22 ` Karthik Nayak
2024-03-25 9:10 ` Patrick Steinhardt
2024-03-18 10:52 ` [PATCH 05/15] refs/reftable: print errors on compaction failure Patrick Steinhardt
2024-03-18 10:52 ` [PATCH 06/15] t/helper: drop pack-refs wrapper Patrick Steinhardt
2024-03-18 10:52 ` [PATCH 07/15] refs: move `struct pack_refs_opts` to where it's used Patrick Steinhardt
2024-03-18 10:52 ` [PATCH 08/15] refs: remove `PACK_REFS_ALL` flag Patrick Steinhardt
2024-03-18 10:53 ` [PATCH 09/15] refs/reftable: expose auto compaction via new flag Patrick Steinhardt
2024-03-18 10:53 ` [PATCH 10/15] builtin/pack-refs: release allocated memory Patrick Steinhardt
2024-03-20 23:23 ` Karthik Nayak
2024-03-25 9:10 ` Patrick Steinhardt
2024-03-18 10:53 ` [PATCH 11/15] builtin/pack-refs: introduce new "--auto" flag Patrick Steinhardt
2024-03-18 10:53 ` [PATCH 12/15] builtin/gc: move `struct maintenance_run_opts` Patrick Steinhardt
2024-03-18 10:53 ` [PATCH 13/15] t6500: extract objects with "17" prefix Patrick Steinhardt
2024-03-18 10:53 ` [PATCH 14/15] builtin/gc: forward git-gc(1)'s `--auto` flag when packing refs Patrick Steinhardt
2024-03-20 23:56 ` Karthik Nayak
2024-03-25 9:10 ` Patrick Steinhardt
2024-03-18 10:53 ` [PATCH 15/15] builtin/gc: pack refs when using `git maintenance run --auto` Patrick Steinhardt
2024-03-20 23:59 ` Karthik Nayak
2024-03-25 9:10 ` Patrick Steinhardt
2024-03-20 19:30 ` [PATCH 00/15] refs: introduce `--auto` to pack refs as needed Karthik Nayak
2024-03-25 9:10 ` Patrick Steinhardt
2024-03-25 10:02 ` [PATCH v2 " Patrick Steinhardt
2024-03-25 10:02 ` [PATCH v2 01/15] reftable/stack: fix error handling in `reftable_stack_init_addition()` Patrick Steinhardt
2024-03-25 10:02 ` [PATCH v2 02/15] reftable/error: discern locked/outdated errors Patrick Steinhardt
2024-03-25 10:02 ` [PATCH v2 03/15] reftable/stack: use error codes when locking fails during compaction Patrick Steinhardt
2024-03-25 10:02 ` [PATCH v2 04/15] reftable/stack: gracefully handle failed auto-compaction due to locks Patrick Steinhardt
2024-03-25 11:22 ` Patrick Steinhardt
2024-03-25 10:02 ` [PATCH v2 05/15] refs/reftable: print errors on compaction failure Patrick Steinhardt
2024-03-25 10:02 ` [PATCH v2 06/15] t/helper: drop pack-refs wrapper Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 07/15] refs: move `struct pack_refs_opts` to where it's used Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 08/15] refs: remove `PACK_REFS_ALL` flag Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 09/15] refs/reftable: expose auto compaction via new flag Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 10/15] builtin/pack-refs: release allocated memory Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 11/15] builtin/pack-refs: introduce new "--auto" flag Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 12/15] builtin/gc: move `struct maintenance_run_opts` Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 13/15] t6500: extract objects with "17" prefix Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 14/15] builtin/gc: forward git-gc(1)'s `--auto` flag when packing refs Patrick Steinhardt
2024-03-25 10:03 ` [PATCH v2 15/15] builtin/gc: pack refs when using `git maintenance run --auto` Patrick Steinhardt
2024-03-25 11:23 ` [PATCH v2 00/15] refs: introduce `--auto` to pack refs as needed Karthik Nayak
2024-03-25 16:56 ` Junio C Hamano
-- strict thread matches above, loose matches on Subject: below --
2024-03-18 22:14 [PATCH " Han-Wen Nienhuys
2024-03-19 16:10 ` Patrick Steinhardt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1710706118.git.ps@pks.im \
--to=ps@pks.im \
--cc=git@vger.kernel.org \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).