From: "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: gitster@pobox.com, git@jeffhostetler.com, me@ttaylorr.com,
chakrabortyabhradeep79@gmail.com,
Derrick Stolee <derrickstolee@github.com>,
Derrick Stolee <derrickstolee@github.com>
Subject: [PATCH v2] pack-bitmap: remove trace2 region from hot path
Date: Mon, 26 Sep 2022 13:17:57 +0000 [thread overview]
Message-ID: <pull.1365.v2.git.1664198277250.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.1365.git.1663938034607.gitgitgadget@gmail.com>
From: Derrick Stolee <derrickstolee@github.com>
The trace2 region around the call to lazy_bitmap_for_commit() in
bitmap_for_commit() was added in 28cd730680d (pack-bitmap: prepare to
read lookup table extension, 2022-08-14). While adding trace2 regions is
typically helpful for tracking performance, this method is called
possibly thousands of times as a commit walk explores commit history
looking for a matching bitmap. When trace2 output is enabled, this
region is emitted many times and performance is throttled by that
output.
For now, remove these regions entirely.
This is a critical path, and it would be valuable to measure that the
time spent in bitmap_for_commit() does not increase when using the
commit lookup table. The best way to do that would be to use a mechanism
that sums the time spent in a region and reports a single value at the
end of the process. This technique was introduced but not merged by [1]
so maybe this example presents some justification to revisit that
approach.
[1] https://lore.kernel.org/git/pull.1099.v2.git.1640720202.gitgitgadget@gmail.com/
To help with the 'git blame' output in this region, add a comment that
warns against adding a trace2 region. Delete a test from t5310 that used
that trace output to check that this lookup optimization was activated.
To create this kind of test again in the future, the stopwatch traces
mentioned earlier could be used as a signal that we activated this code
path.
Helpedy-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
[For 2.38.0] pack-bitmap: remove trace2 region from hot path
I noticed this while trying to backport the Abhradeep's lookup table
work into GitHub's fork. Something went wrong in that process, causing
this region to significantly slow down. It turns out that slow down does
not reproduce on current 'master', which is good. I must have missed
something while I was backporting.
Regardless, the use of trace2_region_enter() here should be removed or
replaced. For the sake of 2.38.0, this simple removal is safe enough.
However, to really dig into what was happening I had to construct a
rebase [2] of Jeff's trace2 stopwatch work, then apply changes on top
[3] that could replace this region with trace2_timer_*() methods.
[2]
https://github.com/git/git/compare/master...derrickstolee:trace2-stopwatch
[3]
https://github.com/derrickstolee/git/compare/trace2-stopwatch...bitmap-trace2
As a separate discussion, it might be worth revisiting that stopwatch
work so we have it available as a tool when doing these kinds of
investigations.
Updates in v2
=============
* I removed the test that was checking for trace2 regions. In [3], I
played around with using trace2_counter() to replace that check, so
we could reinstate the test without the performance drop from using
repeated trace2_region() calls.
* I squashed in Junio's comment change.
Thanks, -Stolee
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1365%2Fderrickstolee%2Fbitmap-remove-trace2-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1365/derrickstolee/bitmap-remove-trace2-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1365
Range-diff vs v1:
1: 987827eaf90 ! 1: 8ef63c9e32f pack-bitmap: remove trace2 region from hot path
@@ Commit message
[1] https://lore.kernel.org/git/pull.1099.v2.git.1640720202.gitgitgadget@gmail.com/
+ To help with the 'git blame' output in this region, add a comment that
+ warns against adding a trace2 region. Delete a test from t5310 that used
+ that trace output to check that this lookup optimization was activated.
+ To create this kind of test again in the future, the stopwatch traces
+ mentioned earlier could be used as a signal that we activated this code
+ path.
+
+ Helpedy-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
## pack-bitmap.c ##
@@ pack-bitmap.c: struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap
return NULL;
- trace2_region_enter("pack-bitmap", "reading_lookup_table", the_repository);
++ /* this is a fairly hot codepath - no trace2_region please */
/* NEEDSWORK: cache misses aren't recorded */
bitmap = lazy_bitmap_for_commit(bitmap_git, commit);
- trace2_region_leave("pack-bitmap", "reading_lookup_table", the_repository);
if (!bitmap)
return NULL;
return lookup_stored_bitmap(bitmap);
+
+ ## t/t5310-pack-bitmaps.sh ##
+@@ t/t5310-pack-bitmaps.sh: test_expect_success 'verify writing bitmap lookup table when enabled' '
+ grep "\"label\":\"writing_lookup_table\"" trace2
+ '
+
+-test_expect_success 'lookup table is actually used to traverse objects' '
+- git repack -adb &&
+- GIT_TRACE2_EVENT="$(pwd)/trace3" \
+- git rev-list --use-bitmap-index --count --all &&
+- grep "\"label\":\"reading_lookup_table\"" trace3
+-'
+-
+ test_expect_success 'truncated bitmap fails gracefully (lookup table)' '
+ test_config pack.writebitmaphashcache false &&
+ git repack -adb &&
pack-bitmap.c | 3 +--
t/t5310-pack-bitmaps.sh | 7 -------
2 files changed, 1 insertion(+), 9 deletions(-)
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9a208abc1fd..a2bbbbd811d 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -830,10 +830,9 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,
if (!bitmap_git->table_lookup)
return NULL;
- trace2_region_enter("pack-bitmap", "reading_lookup_table", the_repository);
+ /* this is a fairly hot codepath - no trace2_region please */
/* NEEDSWORK: cache misses aren't recorded */
bitmap = lazy_bitmap_for_commit(bitmap_git, commit);
- trace2_region_leave("pack-bitmap", "reading_lookup_table", the_repository);
if (!bitmap)
return NULL;
return lookup_stored_bitmap(bitmap);
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index 7e50f8e7653..6d693eef82f 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -455,13 +455,6 @@ test_expect_success 'verify writing bitmap lookup table when enabled' '
grep "\"label\":\"writing_lookup_table\"" trace2
'
-test_expect_success 'lookup table is actually used to traverse objects' '
- git repack -adb &&
- GIT_TRACE2_EVENT="$(pwd)/trace3" \
- git rev-list --use-bitmap-index --count --all &&
- grep "\"label\":\"reading_lookup_table\"" trace3
-'
-
test_expect_success 'truncated bitmap fails gracefully (lookup table)' '
test_config pack.writebitmaphashcache false &&
git repack -adb &&
base-commit: 1b3d6e17fe83eb6f79ffbac2f2c61bbf1eaef5f8
--
gitgitgadget
next prev parent reply other threads:[~2022-09-26 14:53 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-23 13:00 [PATCH] pack-bitmap: remove trace2 region from hot path Derrick Stolee via GitGitGadget
2022-09-23 17:05 ` Junio C Hamano
2022-09-23 17:25 ` Junio C Hamano
2022-09-23 17:36 ` Junio C Hamano
2022-09-23 18:07 ` Derrick Stolee
2022-09-23 19:17 ` Taylor Blau
2022-09-26 13:17 ` Derrick Stolee via GitGitGadget [this message]
2022-09-26 15:01 ` [PATCH v2] " Ævar Arnfjörð Bjarmason
2022-09-26 17:31 ` Derrick Stolee
2022-09-27 18:37 ` [PATCH] " Jeff Hostetler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=pull.1365.v2.git.1664198277250.gitgitgadget@gmail.com \
--to=gitgitgadget@gmail.com \
--cc=chakrabortyabhradeep79@gmail.com \
--cc=derrickstolee@github.com \
--cc=git@jeffhostetler.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=me@ttaylorr.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).