From: Jeff King <peff@peff.net>
To: git@vger.kernel.org
Cc: "Taylor Blau" <me@ttaylorr.com>,
"Derrick Stolee" <dstolee@microsoft.com>,
"Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
Subject: [PATCH] list-objects: don't queue root trees unless revs->tree_objects is set
Date: Wed, 11 Sep 2019 21:11:37 -0400 [thread overview]
Message-ID: <20190912011137.GA23412@sigill.intra.peff.net> (raw)
In-Reply-To: <20190912001846.GA31370@sigill.intra.peff.net>
On Wed, Sep 11, 2019 at 08:18:46PM -0400, Jeff King wrote:
> > That creates an interesting problem for commits that have _already_ been
> > parsed using the commit graph. Their commit->object.parsed flag is set,
> > their commit->graph_pos is set, but their commit->maybe_tree may still
> > be NULL. When somebody later calls repo_get_commit_tree(), we see that
> > we haven't loaded the tree oid yet and try to get it from the commit
> > graph. But since it has been freed, we segfault!
>
> I was surprised we ever called repo_get_commit_tree() at all, since
> we're literally just traversing commits here. It looks like
> list-objects.c is very happy to queue pending trees for each commit,
> even if we're just going to throw them away when we get to
> process_tree()! I wonder if could be checking revs->tree_objects here
> and saving ourselves some work.
Indeed, this seems to help quite a bit in the commit-graph case. I think
it's worth doing (and is independent of the other patch).
-- >8 --
Subject: list-objects: don't queue root trees unless revs->tree_objects is set
When traverse_commit_list() processes each commit, it queues the
commit's root tree in the pending array. Then, after all commits are
processed, it calls traverse_trees_and_blobs() to walk over the pending
list, calling process_tree() on each. But if revs->tree_objects is not
set, process_tree() just exists immediately!
We can save ourselves some work by not even bothering to queue these
trees in the first place. There are a few subtle points to make:
- we also detect commits with a NULL tree pointer here. But this isn't
an interesting check for broken commits, since the lookup_tree()
we'd have done during commit parsing doesn't actually check that we
have the tree on disk. So we're not losing any robustness.
- besides queueing, we also set the NOT_USER_GIVEN flag on the tree
object. This is used by the traverse_commit_list_filtered() variant.
But if we're not exploring trees, then we won't actually care about
this flag, which is used only inside process_tree() code-paths.
- queueing trees eventually leads to us queueing blobs, too. But we
don't need to check revs->blob_objects here. Even in the current
code, we still wouldn't find those blobs, because we'd never open up
the tree objects to list their contents.
- the user-visible impact to the caller is minimal. The pending trees
are all cleared by the time the function returns anyway, by
traverse_trees_and_blobs(). We do call a show_commit() callback,
which technically could be looking at revs->pending during the
callback. But it seems like a rather unlikely thing to do (if you
want the tree of the current commit, then accessing the tree struct
member is a lot simpler).
So this should be safe to do. Let's look at the benefits:
[before]
Benchmark #1: git -C linux rev-list HEAD >/dev/null
Time (mean ± σ): 7.651 s ± 0.021 s [User: 7.399 s, System: 0.252 s]
Range (min … max): 7.607 s … 7.683 s 10 runs
[after]
Benchmark #1: git -C linux rev-list HEAD >/dev/null
Time (mean ± σ): 7.593 s ± 0.023 s [User: 7.329 s, System: 0.264 s]
Range (min … max): 7.565 s … 7.634 s 10 runs
Not too impressive, but then we're really just avoiding sticking a
pointer into a growable array. But still, I'll take a free 0.75%
speedup.
Let's try it after running "git commit-graph write":
[before]
Benchmark #1: git -C linux rev-list HEAD >/dev/null
Time (mean ± σ): 1.458 s ± 0.011 s [User: 1.199 s, System: 0.259 s]
Range (min … max): 1.447 s … 1.481 s 10 runs
[after]
Benchmark #1: git -C linux rev-list HEAD >/dev/null
Time (mean ± σ): 1.126 s ± 0.023 s [User: 896.5 ms, System: 229.0 ms]
Range (min … max): 1.106 s … 1.181 s 10 runs
Now that's more like it. We saved over 22% of the total time. Part of
that is because the runtime is shorter overall, but the absolute
improvement is also much larger. What's going on?
When we fill in a commit struct using the commit graph, we don't bother
to set the tree pointer, and instead lazy-load it when somebody calls
get_commit_tree(). So we're not only skipping the pointer write to the
pending queue, but we're skipping the lazy-load of the tree entirely.
Signed-off-by: Jeff King <peff@peff.net>
---
list-objects.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/list-objects.c b/list-objects.c
index b5651ddd5b..c837bcaca8 100644
--- a/list-objects.c
+++ b/list-objects.c
@@ -370,7 +370,9 @@ static void do_traverse(struct traversal_context *ctx)
* an uninteresting boundary commit may not have its tree
* parsed yet, but we are not going to show them anyway
*/
- if (get_commit_tree(commit)) {
+ if (!ctx->revs->tree_objects)
+ ; /* do not bother loading tree */
+ else if (get_commit_tree(commit)) {
struct tree *tree = get_commit_tree(commit);
tree->object.flags |= NOT_USER_GIVEN;
add_pending_tree(ctx->revs, tree);
--
2.23.0.663.gbe3d117559
next prev parent reply other threads:[~2019-09-12 1:11 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-12 0:04 [PATCH] upload-pack: disable commit graph more gently for shallow traversal Jeff King
2019-09-12 0:18 ` Jeff King
2019-09-12 1:11 ` Jeff King [this message]
2019-09-12 1:19 ` [PATCH] list-objects: don't queue root trees unless revs->tree_objects is set Jeff King
2019-09-12 12:31 ` Derrick Stolee
2019-09-12 16:52 ` Junio C Hamano
2019-09-12 22:34 ` Jeff King
2019-09-13 18:05 ` Junio C Hamano
2019-09-12 2:08 ` [PATCH] upload-pack: disable commit graph more gently for shallow traversal Taylor Blau
2019-09-12 14:03 ` Jeff King
2019-09-12 2:07 ` Taylor Blau
2019-09-12 11:06 ` SZEDER Gábor
2019-09-12 14:01 ` Jeff King
2019-09-12 12:46 ` Derrick Stolee
2019-09-12 13:59 ` Jeff King
2019-09-12 12:23 ` Derrick Stolee
2019-09-12 14:23 ` Jeff King
2019-09-12 19:27 ` Derrick Stolee
2019-09-12 14:41 ` [PATCH v2] upload-pack commit graph segfault fix Jeff King
2019-09-12 14:43 ` Jeff King
2019-09-12 14:44 ` [PATCH v2 1/2] commit-graph: bump DIE_ON_LOAD check to actual load-time Jeff King
2019-09-12 19:30 ` Derrick Stolee
2019-09-12 14:44 ` [PATCH v2 2/2] upload-pack: disable commit graph more gently for shallow traversal Jeff King
2019-09-13 13:26 ` Derrick Stolee
2019-09-12 16:56 ` [PATCH v2] upload-pack commit graph segfault fix Taylor Blau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190912011137.GA23412@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=dstolee@microsoft.com \
--cc=git@vger.kernel.org \
--cc=me@ttaylorr.com \
--cc=pclouds@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).