From: Jeff King <peff@peff.net>
To: git@vger.kernel.org
Cc: Stefan Beller <sbeller@google.com>, Junio C Hamano <gitster@pobox.com>
Subject: [PATCH v2 0/6] reuse on-disk deltas for fetches with bitmaps
Date: Tue, 21 Aug 2018 15:06:22 -0400 [thread overview]
Message-ID: <20180821190622.GA30301@sigill.intra.peff.net> (raw)
In-Reply-To: <20180817205427.GA19580@sigill.intra.peff.net>
On Fri, Aug 17, 2018 at 04:54:27PM -0400, Jeff King wrote:
> This series more aggressively reuses on-disk deltas to serve fetches
> when reachability bitmaps tell us a more complete picture of what the
> client has. That saves server CPU and results in smaller packs. See the
> final patch for numbers and more discussion.
Here's a v2, with just a few cosmetic fixes to address the comments on
v1 (range-diff below).
[1/6]: t/perf: factor boilerplate out of test_perf
[2/6]: t/perf: factor out percent calculations
[3/6]: t/perf: add infrastructure for measuring sizes
[4/6]: t/perf: add perf tests for fetches from a bitmapped server
[5/6]: pack-bitmap: save "have" bitmap from walk
[6/6]: pack-objects: reuse on-disk deltas for thin "have" objects
builtin/pack-objects.c | 28 +++++++----
pack-bitmap.c | 25 +++++++++-
pack-bitmap.h | 7 +++
pack-objects.c | 19 ++++++++
pack-objects.h | 20 +++++++-
t/perf/README | 25 ++++++++++
t/perf/aggregate.perl | 69 ++++++++++++++++++++++------
t/perf/p5311-pack-bitmaps-fetch.sh | 45 ++++++++++++++++++
t/perf/perf-lib.sh | 74 +++++++++++++++++++-----------
9 files changed, 260 insertions(+), 52 deletions(-)
create mode 100755 t/perf/p5311-pack-bitmaps-fetch.sh
1: 89fa0ec8d8 ! 1: 3e1b94d7d6 pack-bitmap: save "have" bitmap from walk
@@ -69,6 +69,8 @@
+
+ if (!bitmap_git)
+ return 0; /* no bitmap loaded */
++ if (!bitmap_git->result)
++ BUG("failed to perform bitmap walk before querying");
+ if (!bitmap_git->haves)
+ return 0; /* walk had no "haves" */
+
2: f7ca0d59e3 ! 2: b8b2416aac pack-objects: reuse on-disk deltas for thin "have" objects
@@ -12,7 +12,7 @@
However, this misses some opportunities. Modulo some special
cases like shallow or partial clones, we know that every
object reachable from the "haves" could be a preferred base.
- We don't use them all for two reasons:
+ We don't use all of them for two reasons:
1. It's expensive to traverse the whole history and
enumerate all of the objects the other side has.
@@ -100,15 +100,16 @@
The second is that the rest of the code assumes that any
reused delta will point to another "struct object_entry" as
- its base. But by definition, we don't have such an entry!
+ its base. But of course the case we are interested in here
+ is the one where don't have such an entry!
I looked at a number of options that didn't quite work:
- - we could use a different flag for reused deltas. But it's
- not a single bit for "I'm being reused". We have to
- actually store the oid of the base, which is normally
- done by pointing to the existing object_entry. And we'd
- have to modify all the code which looks at deltas.
+ - we could use a flag to signal a reused delta, but it's
+ not a single bit. We have to actually store the oid of
+ the base, which is normally done by pointing to the
+ existing object_entry. And we'd have to modify all the
+ code which looks at deltas.
- we could add the reused bases to the end of the existing
object_entry array. While this does create some extra
@@ -173,7 +174,7 @@
static int depth = 50;
static int delta_search_threads;
static int pack_to_stdout;
-+static int thin = 0;
++static int thin;
static int num_preferred_base;
static struct progress *progress_state;
next prev parent reply other threads:[~2018-08-21 19:06 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-17 20:54 [PATCH 0/6] reuse on-disk deltas for fetches with bitmaps Jeff King
2018-08-17 20:55 ` [PATCH 1/6] t/perf: factor boilerplate out of test_perf Jeff King
2018-08-17 20:55 ` [PATCH 2/6] t/perf: factor out percent calculations Jeff King
2018-08-17 20:56 ` [PATCH 3/6] t/perf: add infrastructure for measuring sizes Jeff King
2018-08-17 20:57 ` [PATCH 4/6] t/perf: add perf tests for fetches from a bitmapped server Jeff King
2018-08-17 20:59 ` [PATCH 5/6] pack-bitmap: save "have" bitmap from walk Jeff King
2018-08-17 22:39 ` Stefan Beller
2018-08-17 22:45 ` Jeff King
2018-08-17 21:06 ` [PATCH 6/6] pack-objects: reuse on-disk deltas for thin "have" objects Jeff King
2018-08-17 22:57 ` Stefan Beller
2018-08-17 23:32 ` Jeff King
2018-08-20 21:03 ` Junio C Hamano
2018-08-20 21:42 ` Jeff King
2018-08-21 19:06 ` Jeff King [this message]
2018-08-21 19:08 ` [PATCH v2 0/6] reuse on-disk deltas for fetches with bitmaps Jeff King
2018-08-21 19:34 ` Junio C Hamano
2018-08-21 19:51 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180821190622.GA30301@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=sbeller@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).