From: Jeff King <peff@peff.net>
To: Taylor Blau <me@ttaylorr.com>
Cc: Teng Long <dyroneteng@gmail.com>,
avarab@gmail.com, derrickstolee@github.com, git@vger.kernel.org,
gitster@pobox.com, tenglong.tl@alibaba-inc.com
Subject: Re: [PATCH v3 1/2] pack-bitmap.c: remove unnecessary "open_pack_index()" calls
Date: Mon, 14 Nov 2022 17:31:18 -0500 [thread overview]
Message-ID: <Y3LBtl+x2rCzsoIh@coredump.intra.peff.net> (raw)
In-Reply-To: <Y3K9zVx7L9K85NrE@nand.local>
On Mon, Nov 14, 2022 at 05:14:37PM -0500, Taylor Blau wrote:
> OK, so with 10K packs, we see about a 1.6-fold improvement, which is
> definitely substantial.
>
> On a fresh clone of git.git, repeating your experiment with only 1K
> packs (which is definitely a number of packs that GitHub sees in
> under-maintained repositories), the runtime goes from 25.3ms -> 20.9ms
> on my machine, or about a 1.2-fold improvement.
>
> So definitely smaller, but even at 1/10th the number of packs from your
> experiment, still noticeable.
Interesting. I had tried it initially with 1000 and the improvements were
much smaller. I just did it again, though, and got the same 20% speedup.
I'm not sure what I screwed up earlier (I may have been confused by the
timestamp/sorting issue; I only realized it was important midway through
looking into this).
> > - this probably isn't helping anybody much in the real world, as
> > evidenced by the contortions I had to go through to set up the
> > situation (and which would be made much better by repacking, which
> > would also speed up non-bitmap operations).
>
> Per above, I'm not sure I totally agree ;-). 1K packs is definitely an
> extreme amount of packs, but not out-of-this-world. It probably would
> show up in carefully-picked graphs, but not in "overall git rev-list
> time" or something as broad/noisy as that.
Yeah, I agree that 1k is a lot more compelling. The big impractical
thing I think is that if the bitmapped pack is older (and it usually
is), then we'd often open all the other packs anyway:
- if the start of the traversal is in the bitmapped pack, then we
fruitlessly open each of the others looking for the object (since
the bitmapped one will come last in the reverse-chronological
sorting)
- if it isn't in the bitmapped pack, then we'll end up opening all
those other packs anyway to fill out the bitmap (since by definition
it can't be included in the on-disk bitmaps)
So I'd be surprised if it ever mattered in the real world. Though again,
I think the new code is less surprising in general, and could matter if
we changed other things (e.g., if we prioritized lookups in a pack with
a .bitmap).
-Peff
next prev parent reply other threads:[~2022-11-14 22:32 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-26 7:09 [PATCH 0/1] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-08-26 7:09 ` [PATCH 1/1] " Teng Long
2022-08-26 16:34 ` [PATCH 0/1] " Junio C Hamano
2022-08-29 2:48 ` Teng Long
2022-10-26 21:42 ` Taylor Blau
2022-10-26 23:19 ` Ævar Arnfjörð Bjarmason
2022-10-31 13:20 ` Teng Long
2022-10-27 20:45 ` Jeff King
2022-10-30 18:42 ` Taylor Blau
2022-10-31 12:22 ` [PATCH 0/1] pack-bitmap.c: avoid exposing absolute paths Taylor Blau <me@ttaylorr.com> writes: Teng Long
2022-11-02 5:37 ` [PATCH 0/1] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-11-02 7:54 ` Jeff King
2022-11-02 13:52 ` Teng Long
2022-10-31 13:13 ` Teng Long
2022-11-03 1:00 ` Taylor Blau
2022-11-02 9:20 ` Ævar Arnfjörð Bjarmason
2022-11-02 13:04 ` Teng Long
2022-11-02 12:56 ` [PATCH v2 " Teng Long
2022-11-02 12:56 ` [PATCH v2 1/1] " Teng Long
2022-11-03 1:16 ` Taylor Blau
2022-11-03 9:35 ` Teng Long
2022-11-05 0:35 ` Taylor Blau
2022-11-03 1:21 ` [PATCH v2 0/1] " Taylor Blau
2022-11-03 8:42 ` Teng Long
2022-11-04 3:17 ` [PATCH v3 0/2] " Teng Long
2022-11-04 3:17 ` [PATCH v3 1/2] " Teng Long
2022-11-04 22:11 ` Taylor Blau
2022-11-04 3:17 ` [PATCH v3 2/2] pack-bitmap.c: remove unnecessary "open_pack_index()" calls Teng Long
2022-11-04 22:09 ` Taylor Blau
2022-11-04 22:13 ` [PATCH v3 0/2] pack-bitmap.c: avoid exposing absolute paths Taylor Blau
2022-11-10 7:10 ` Teng Long
2022-11-10 7:10 ` [PATCH v3 1/2] pack-bitmap.c: remove unnecessary "open_pack_index()" calls Teng Long
2022-11-14 22:03 ` Jeff King
2022-11-14 22:14 ` Taylor Blau
2022-11-14 22:31 ` Jeff King [this message]
2022-11-14 22:50 ` Taylor Blau
2022-11-10 7:10 ` [PATCH v3 2/2] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-11-11 22:26 ` [PATCH v3 0/2] " Taylor Blau
2022-11-14 22:23 ` Jeff King
2022-11-17 14:19 ` Teng Long
2022-11-17 15:03 ` Jeff King
2022-11-17 21:57 ` Taylor Blau
2022-11-21 3:27 ` Teng Long
2022-11-21 12:16 ` [PATCH v4 0/4] " Teng Long
2022-11-21 12:16 ` [PATCH v4 1/4] pack-bitmap.c: remove unnecessary "open_pack_index()" calls Teng Long
2022-11-21 12:16 ` [PATCH v4 2/4] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-11-21 12:16 ` [PATCH v4 3/4] pack-bitmap.c: break out of the bitmap loop early if not tracing Teng Long
2022-11-21 23:27 ` Junio C Hamano
2022-11-28 13:09 ` Teng Long
2022-11-21 12:16 ` [PATCH v4 4/4] pack-bitmap.c: trace bitmap ignore logs when midx-bitmap is found Teng Long
2022-11-21 19:09 ` Jeff King
2022-11-21 23:29 ` Junio C Hamano
2022-11-28 12:29 ` Teng Long
2022-11-28 12:37 ` Teng Long
2022-11-29 1:27 ` Jeff King
2022-11-29 13:14 ` Teng Long
2022-11-21 19:04 ` [PATCH v4 0/4] pack-bitmap.c: avoid exposing absolute paths Jeff King
2022-11-28 12:48 ` Teng Long
2022-11-28 14:09 ` [PATCH v5 " Teng Long
2022-11-28 14:09 ` [PATCH v5 1/4] pack-bitmap.c: remove unnecessary "open_pack_index()" calls Teng Long
2022-11-28 14:09 ` [PATCH v5 2/4] pack-bitmap.c: avoid exposing absolute paths Teng Long
2022-11-28 14:09 ` [PATCH v5 3/4] pack-bitmap.c: break out of the bitmap loop early if not tracing Teng Long
2022-11-28 23:26 ` Taylor Blau
2022-11-29 13:17 ` Teng Long
2022-11-28 14:09 ` [PATCH v5 4/4] pack-bitmap.c: trace bitmap ignore logs when midx-bitmap is found Teng Long
2022-11-28 23:30 ` [PATCH v5 0/4] pack-bitmap.c: avoid exposing absolute paths Taylor Blau
2022-11-29 13:21 ` Teng Long
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y3LBtl+x2rCzsoIh@coredump.intra.peff.net \
--to=peff@peff.net \
--cc=avarab@gmail.com \
--cc=derrickstolee@github.com \
--cc=dyroneteng@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=me@ttaylorr.com \
--cc=tenglong.tl@alibaba-inc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).