From: Patrick Steinhardt <ps@pks.im>
To: git@vger.kernel.org
Cc: "Jeff King" <peff@peff.net>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Junio C Hamano" <gitster@pobox.com>,
"Derrick Stolee" <stolee@gmail.com>,
"René Scharfe" <l.s.r@web.de>
Subject: [PATCH v3 0/7] Speed up mirror-fetches with many refs
Date: Wed, 1 Sep 2021 15:09:36 +0200 [thread overview]
Message-ID: <cover.1630501732.git.ps@pks.im> (raw)
In-Reply-To: <cover.1629452412.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 5048 bytes --]
Hi,
this is the third version of my patch series to speed up mirror-fetches
with many refs. This patch series applies on top of master with
ps/connectivity-optim merged into it.
There's only some smallish changes based on Stolee's feedback (thanks
for that!):
- A small typo in 1/7.
- A confict fix in 4/7 required now because it's based on master
instead of directly on my merged topic.
- I've adjusted patch 5/7 such that I don't have to re-touch the
logic in 6/7.
Patrick
Patrick Steinhardt (7):
fetch: speed up lookup of want refs via commit-graph
fetch: avoid unpacking headers in object existence check
connected: refactor iterator to return next object ID directly
fetch-pack: optimize loading of refs via commit graph
fetch: refactor fetch refs to be more extendable
fetch: merge fetching and consuming refs
fetch: avoid second connectivity check if we already have all objects
builtin/clone.c | 8 ++---
builtin/fetch.c | 74 +++++++++++++++++++++++-------------------
builtin/receive-pack.c | 17 ++++------
connected.c | 15 +++++----
connected.h | 2 +-
fetch-pack.c | 12 ++++---
6 files changed, 67 insertions(+), 61 deletions(-)
Range-diff against v2:
1: 4a819a6830 ! 1: 8214f04971 fetch: speed up lookup of want refs via commit-graph
@@ Commit message
that we repeatedly need to unpack object headers for each of the
referenced objects.
- Speed this up by opportunistcally trying to resolve object IDs via the
+ Speed this up by opportunistically trying to resolve object IDs via the
commit graph. We only do so for any refs which are not in "refs/tags":
more likely than not, these are going to be a commit anyway, and this
lets us avoid having to unpack object headers completely in case the
2: 81ebadabe8 = 2: 991a27cb82 fetch: avoid unpacking headers in object existence check
3: 98e981ced9 = 3: ba834803ab connected: refactor iterator to return next object ID directly
4: 6311203f08 ! 4: 99d3316d48 fetch-pack: optimize loading of refs via commit graph
@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object
while (1) {
if (oid_object_info_extended(the_repository, oid, &info,
-@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object_id *oid,
- }
-
- if (type == OBJ_COMMIT) {
-- struct commit *commit = lookup_commit(the_repository, oid);
-+ commit = lookup_commit(the_repository, oid);
- if (!commit || repo_parse_commit(the_repository, commit))
- return NULL;
- return commit;
5: 56a9158ac3 ! 5: d64888e072 fetch: refactor fetch refs to be more extendable
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
trace2_region_enter("fetch", "fetch_refs", the_repository);
ret = transport_fetch_refs(transport, ref_map);
trace2_region_leave("fetch", "fetch_refs", the_repository);
-+ if (ret) {
-+ transport_unlock_pack(transport);
-+ return ret;
-+ }
++ if (ret)
++ goto out;
}
- if (!ret)
- /*
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
- * time to update refs to reference the new objects.
- */
- return 0;
-- transport_unlock_pack(transport);
-- return ret;
+
+ /*
+ * Keep the new pack's ".keep" file around to allow the caller
+ * time to update refs to reference the new objects.
+ */
-+ return 0;
++ return ret;
++
++out:
+ transport_unlock_pack(transport);
+ return ret;
}
-
- /* Update local refs based on the ref values fetched from a remote */
6: 31d9f72edf ! 6: 56ecbfc9c3 fetch: merge fetching and consuming refs
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
/*
@@ builtin/fetch.c: static int fetch_refs(struct transport *transport, struct ref *ref_map)
- trace2_region_enter("fetch", "fetch_refs", the_repository);
- ret = transport_fetch_refs(transport, ref_map);
- trace2_region_leave("fetch", "fetch_refs", the_repository);
-- if (ret) {
-- transport_unlock_pack(transport);
-- return ret;
-- }
-+ if (ret)
-+ goto out;
+ goto out;
}
- /*
- * Keep the new pack's ".keep" file around to allow the caller
- * time to update refs to reference the new objects.
- */
-- return 0;
+- return ret;
+-
+-out:
+- transport_unlock_pack(transport);
+- return ret;
-}
-
-/* Update local refs based on the ref values fetched from a remote */
7: 84e39c847f = 7: c342fc0c69 fetch: avoid second connectivity check if we already have all objects
--
2.33.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2021-09-01 13:19 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-20 10:08 [PATCH 0/6] Speed up mirror-fetches with many refs Patrick Steinhardt
2021-08-20 10:08 ` [PATCH 1/6] fetch: speed up lookup of want refs via commit-graph Patrick Steinhardt
2021-08-20 14:27 ` Derrick Stolee
2021-08-20 17:18 ` Junio C Hamano
2021-08-23 6:46 ` Patrick Steinhardt
2021-08-25 14:12 ` Derrick Stolee
2021-08-20 10:08 ` [PATCH 2/6] fetch: avoid unpacking headers in object existence check Patrick Steinhardt
2021-08-25 23:44 ` Ævar Arnfjörð Bjarmason
2021-08-20 10:08 ` [PATCH 3/6] connected: refactor iterator to return next object ID directly Patrick Steinhardt
2021-08-20 14:32 ` Derrick Stolee
2021-08-20 17:43 ` Junio C Hamano
2021-08-20 17:43 ` René Scharfe
2021-08-23 6:47 ` Patrick Steinhardt
2021-08-20 10:08 ` [PATCH 4/6] fetch-pack: optimize loading of refs via commit graph Patrick Steinhardt
2021-08-20 14:37 ` Derrick Stolee
2021-08-20 10:08 ` [PATCH 5/6] fetch: refactor fetch refs to be more extendable Patrick Steinhardt
2021-08-20 14:41 ` Derrick Stolee
2021-08-20 10:08 ` [PATCH 6/6] fetch: avoid second connectivity check if we already have all objects Patrick Steinhardt
2021-08-20 14:47 ` Derrick Stolee
2021-08-23 6:52 ` Patrick Steinhardt
2021-08-20 14:50 ` [PATCH 0/6] Speed up mirror-fetches with many refs Derrick Stolee
2021-08-21 0:09 ` Junio C Hamano
2021-08-24 10:36 ` [PATCH v2 0/7] " Patrick Steinhardt
2021-08-24 10:36 ` [PATCH v2 1/7] fetch: speed up lookup of want refs via commit-graph Patrick Steinhardt
2021-08-25 14:16 ` Derrick Stolee
2021-08-24 10:37 ` [PATCH v2 2/7] fetch: avoid unpacking headers in object existence check Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 3/7] connected: refactor iterator to return next object ID directly Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 4/7] fetch-pack: optimize loading of refs via commit graph Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 5/7] fetch: refactor fetch refs to be more extendable Patrick Steinhardt
2021-08-25 14:19 ` Derrick Stolee
2021-09-01 12:48 ` Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 6/7] fetch: merge fetching and consuming refs Patrick Steinhardt
2021-08-25 14:26 ` Derrick Stolee
2021-09-01 12:49 ` Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 7/7] fetch: avoid second connectivity check if we already have all objects Patrick Steinhardt
2021-08-24 22:48 ` [PATCH v2 0/7] Speed up mirror-fetches with many refs Junio C Hamano
2021-08-25 6:04 ` Patrick Steinhardt
2021-08-25 14:27 ` Derrick Stolee
2021-09-01 13:09 ` Patrick Steinhardt [this message]
2021-09-01 13:09 ` [PATCH v3 1/7] fetch: speed up lookup of want refs via commit-graph Patrick Steinhardt
2021-09-01 13:09 ` [PATCH v3 2/7] fetch: avoid unpacking headers in object existence check Patrick Steinhardt
2021-09-01 13:09 ` [PATCH v3 3/7] connected: refactor iterator to return next object ID directly Patrick Steinhardt
2021-09-01 13:09 ` [PATCH v3 4/7] fetch-pack: optimize loading of refs via commit graph Patrick Steinhardt
2021-09-01 13:09 ` [PATCH v3 5/7] fetch: refactor fetch refs to be more extendable Patrick Steinhardt
2021-09-01 13:10 ` [PATCH v3 6/7] fetch: merge fetching and consuming refs Patrick Steinhardt
2021-09-01 13:10 ` [PATCH v3 7/7] fetch: avoid second connectivity check if we already have all objects Patrick Steinhardt
2021-09-01 19:58 ` [PATCH v3 0/7] Speed up mirror-fetches with many refs Junio C Hamano
2021-09-08 0:08 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1630501732.git.ps@pks.im \
--to=ps@pks.im \
--cc=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=l.s.r@web.de \
--cc=peff@peff.net \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).