From: Patrick Steinhardt <ps@pks.im>
To: git@vger.kernel.org
Cc: Derrick Stolee <derrickstolee@github.com>
Subject: [PATCH v2 5/5] refs/files-backend: optimize reading of symbolic refs
Date: Tue, 1 Mar 2022 10:33:54 +0100 [thread overview]
Message-ID: <b6eca63d3b9d9ff60778d6dabe13e4d087f2ff63.1646127015.git.ps@pks.im> (raw)
In-Reply-To: <cover.1646127015.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 4272 bytes --]
When reading references via `files_read_raw_ref()` we always consult
both the loose reference, and if that wasn't found, we also consult the
packed-refs file. While this makes sense to read a generic reference, it
is wasteful in the case where we only care about symbolic references:
the packed-refs backend does not support them, and thus it cannot ever
return one for us.
Special-case reading of symbolic references for the files backend such
that we always skip asking the packed-refs backend.
We use `refs_read_symbolic_ref()` extensively to determine whether we
need to skip updating local symbolic references during a fetch, which is
why the change results in a significant speedup when doing fetches in
repositories with huge numbers of references. The following benchmark
executes a mirror-fetch in a repository with about 2 million references
via `git fetch --prune --no-write-fetch-head +refs/*:refs/*`:
Benchmark 1: HEAD~
Time (mean ± σ): 68.372 s ± 2.344 s [User: 65.629 s, System: 8.786 s]
Range (min … max): 65.745 s … 70.246 s 3 runs
Benchmark 2: HEAD
Time (mean ± σ): 60.259 s ± 0.343 s [User: 61.019 s, System: 7.245 s]
Range (min … max): 60.003 s … 60.649 s 3 runs
Summary
'HEAD' ran
1.13 ± 0.04 times faster than 'HEAD~'
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
refs/files-backend.c | 34 ++++++++++++++++++++++++++++------
1 file changed, 28 insertions(+), 6 deletions(-)
diff --git a/refs/files-backend.c b/refs/files-backend.c
index f3428a9f12..0457ecdb42 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -338,9 +338,9 @@ static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs)
return refs->loose;
}
-static int files_read_raw_ref(struct ref_store *ref_store, const char *refname,
- struct object_id *oid, struct strbuf *referent,
- unsigned int *type, int *failure_errno)
+static int read_ref_internal(struct ref_store *ref_store, const char *refname,
+ struct object_id *oid, struct strbuf *referent,
+ unsigned int *type, int *failure_errno, int skip_packed_refs)
{
struct files_ref_store *refs =
files_downcast(ref_store, REF_STORE_READ, "read_raw_ref");
@@ -381,7 +381,7 @@ static int files_read_raw_ref(struct ref_store *ref_store, const char *refname,
if (lstat(path, &st) < 0) {
int ignore_errno;
myerr = errno;
- if (myerr != ENOENT)
+ if (myerr != ENOENT || skip_packed_refs)
goto out;
if (refs_read_raw_ref(refs->packed_ref_store, refname, oid,
referent, type, &ignore_errno)) {
@@ -425,7 +425,8 @@ static int files_read_raw_ref(struct ref_store *ref_store, const char *refname,
* ref is supposed to be, there could still be a
* packed ref:
*/
- if (refs_read_raw_ref(refs->packed_ref_store, refname, oid,
+ if (skip_packed_refs ||
+ refs_read_raw_ref(refs->packed_ref_store, refname, oid,
referent, type, &ignore_errno)) {
myerr = EISDIR;
goto out;
@@ -470,6 +471,27 @@ static int files_read_raw_ref(struct ref_store *ref_store, const char *refname,
return ret;
}
+static int files_read_raw_ref(struct ref_store *ref_store, const char *refname,
+ struct object_id *oid, struct strbuf *referent,
+ unsigned int *type, int *failure_errno)
+{
+ return read_ref_internal(ref_store, refname, oid, referent, type, failure_errno, 0);
+}
+
+static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refname,
+ struct strbuf *referent)
+{
+ struct object_id oid;
+ int failure_errno, ret;
+ unsigned int type;
+
+ ret = read_ref_internal(ref_store, refname, &oid, referent, &type, &failure_errno, 1);
+ if (ret)
+ return ret;
+
+ return !(type & REF_ISSYMREF);
+}
+
int parse_loose_ref_contents(const char *buf, struct object_id *oid,
struct strbuf *referent, unsigned int *type,
int *failure_errno)
@@ -3286,7 +3308,7 @@ struct ref_storage_be refs_be_files = {
files_ref_iterator_begin,
files_read_raw_ref,
- NULL,
+ files_read_symbolic_ref,
files_reflog_iterator_begin,
files_for_each_reflog_ent,
--
2.35.1
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2022-03-01 9:34 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-23 12:35 [PATCH 0/5] fetch: more optimizations for mirror fetches Patrick Steinhardt
2022-02-23 12:35 ` [PATCH 1/5] upload-pack: look up "want" lines via commit-graph Patrick Steinhardt
2022-02-23 14:13 ` Derrick Stolee
2022-03-01 8:43 ` Patrick Steinhardt
2022-03-01 9:24 ` Patrick Steinhardt
2022-03-02 18:53 ` Derrick Stolee
2022-02-23 12:35 ` [PATCH 2/5] fetch: avoid lookup of commits when not appending to FETCH_HEAD Patrick Steinhardt
2022-02-23 14:18 ` Derrick Stolee
2022-03-01 8:44 ` Patrick Steinhardt
2022-02-23 12:35 ` [PATCH 3/5] refs: add ability for backends to special-case reading of symbolic refs Patrick Steinhardt
2022-02-23 12:35 ` [PATCH 4/5] remote: read symbolic refs via `refs_read_symbolic_ref()` Patrick Steinhardt
2022-02-23 12:35 ` [PATCH 5/5] refs/files-backend: optimize reading of symbolic refs Patrick Steinhardt
2022-03-01 9:33 ` [PATCH v2 0/5] fetch: more optimizations for mirror fetches Patrick Steinhardt
2022-03-01 9:33 ` [PATCH v2 1/5] upload-pack: look up "want" lines via commit-graph Patrick Steinhardt
2022-03-01 9:33 ` [PATCH v2 2/5] fetch: avoid lookup of commits when not appending to FETCH_HEAD Patrick Steinhardt
2022-03-01 9:33 ` [PATCH v2 3/5] refs: add ability for backends to special-case reading of symbolic refs Patrick Steinhardt
2022-03-01 9:33 ` [PATCH v2 4/5] remote: read symbolic refs via `refs_read_symbolic_ref()` Patrick Steinhardt
2022-03-01 9:33 ` Patrick Steinhardt [this message]
2022-03-01 22:02 ` [PATCH v2 0/5] fetch: more optimizations for mirror fetches Junio C Hamano
2022-03-02 18:54 ` Derrick Stolee
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b6eca63d3b9d9ff60778d6dabe13e4d087f2ff63.1646127015.git.ps@pks.im \
--to=ps@pks.im \
--cc=derrickstolee@github.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).