From: Jeff King <peff@peff.net>
To: Taylor Blau <me@ttaylorr.com>
Cc: git@vger.kernel.org
Subject: Re: [PATCH 0/3] pack-objects: simplify add_objects_in_unpacked_packs()
Date: Mon, 30 Aug 2021 16:58:34 -0400 [thread overview]
Message-ID: <YS1GemF3aKTe//Tn@coredump.intra.peff.net> (raw)
In-Reply-To: <cover.1630291682.git.me@ttaylorr.com>
On Sun, Aug 29, 2021 at 10:48:43PM -0400, Taylor Blau wrote:
> This short series is extracted from mine and Peff's work on cruft packs. These
> three patches focus on cleaning up add_objects_in_unpacked_packs(), which is
> used to implement `git repack -k`.
>
> The pay-off for this clean-up is significant, though: we net -50 lines of code,
> and the result is much more readable, at least in my opinion.
>
> The changes are described in detail in the patch messages, but essentially we
> are replacing a loop over get_all_packs() with for_each_packed_object() after
> adding a couple of new flags necessary to make the switch. And once we are done
> with that, the third patch removes a bit from the object flag allocation table.
Thanks for extracting these from that other work. I gave them all an
extra read-through and didn't find anything wrong.
> builtin/pack-objects.c | 85 ++++++------------------------------------
> object-store.h | 6 +++
> object.h | 1 -
> packfile.c | 6 +++
> 4 files changed, 24 insertions(+), 74 deletions(-)
Of course I love this diffstat, and I _really_ love dropping the one-off
bit from the flag allocation table. But I wanted to also give some
numbers for the lookup_unknown_object() part of the final patch.
Those unknown objects cost 72 bytes of heap each (the same size as a
commit, since it's the biggest struct and the unknown object is a
union). We've seen some real-world cases where there are 40M+
unreachable objects. So that's almost 3GB of wasted RAM during
pack-objects just to store those "did I see it already" bits. :)
Of course, we are already spending 96 bytes per object in "struct
object_entry", but at least that's doing more useful stuff. :)
(And the more obvious question is: why not delete those objects. The
answer thus far has been: because git's pruning is racy, and thus we
don't run it as part of our automatic maintenance).
-Peff
next prev parent reply other threads:[~2021-08-30 20:59 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-30 2:48 [PATCH 0/3] pack-objects: simplify add_objects_in_unpacked_packs() Taylor Blau
2021-08-30 2:48 ` [PATCH 1/3] object-store.h: teach for_each_packed_object to ignore kept packs Taylor Blau
2021-08-30 2:48 ` [PATCH 2/3] builtin/pack-objects.c: simplify add_objects_in_unpacked_packs() Taylor Blau
2021-09-02 23:04 ` Jeff King
2021-09-03 2:33 ` Taylor Blau
2021-08-30 2:48 ` [PATCH 3/3] builtin/pack-objects.c: remove duplicate hash lookup Taylor Blau
2021-08-30 6:39 ` [PATCH 0/3] pack-objects: simplify add_objects_in_unpacked_packs() Junio C Hamano
2021-08-30 20:58 ` Jeff King [this message]
2021-08-30 21:30 ` Taylor Blau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YS1GemF3aKTe//Tn@coredump.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=me@ttaylorr.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).