From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>, "Jeff King" <peff@peff.net>,
"Johannes Sixt" <j6t@kdbg.org>,
"Øystein Walle" <oystwa@gmail.com>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH 7/8] Makefile: stop having command-list.h depend on a wildcard
Date: Wed, 20 Oct 2021 20:39:58 +0200 [thread overview]
Message-ID: <patch-7.8-0c6f9b80d3b-20211020T183533Z-avarab@gmail.com> (raw)
In-Reply-To: <cover-0.8-00000000000-20211020T183533Z-avarab@gmail.com>
Change the dependency graph for "command-list.h" so that it's not
re-built in its entirety every time one file in Documentation/git*.txt
changes. This makes the generation of the command-list.h use a pattern
similar to that established in 8650c6298c1 (doc lint: make "lint-docs"
non-.PHONY, 2021-10-15) for the "lint-docs" target.
This change replaces the monolithic "generate-cmdlist.sh" invocation
with a dependency chain like:
Documentation/git-add.txt ->
.build/command-list.h.d/git-add.gen (and "git-add.txt" intermediary) ->
command-list.h
I.e. we'll now generate intermediary files when extracting the "NAME"
section the documentation. Here the 6th line of
"Documentation/git.txt" will be extracted to
".build/command-list.h.d/git-add.gen.txt":
git-add - Add file contents to the index
We'll then create an intermediate
".build/command-list.h.d/git-add.gen" (leading "\t" stripped):
./generate-cmdlist.sh --entry-only .build/command-list.h.d/git-add.gen.txt
{ "git-add", N_("Add file contents to the index"), 0 | CAT_mainporcelain | CAT_worktree },
The "command-list.h" itself can the be made by having
"generate-cmdlist.sh" emit only the header section, followed by:
LC_ALL=sort .build/command-list.h.d/*.gen
Unfortunately we can't drop the old code completely due to the CMake
integration, see 061c2240b1b (Introduce CMake support for configuring
Git, 2020-06-12). It will keep using the older and slower script.
With this the initial creation of the command-list.h is a bit slower
with -j1, but around 2x as fast -j8[1]. The real benefit comes from
the more common case of an incremental build, say when only
"Documentation/git-add.txt" was updated after a "pull". There we're
4-5x as fast with this new method[2].
The benefit of optimizing this is because this file is very frequently
re-generated, e.g. for "git rebase -i --exec 'make git'" with
"git-add.txt" modified we're around 1.7 times as fast[3]. That target
will need to re-make "git" (via "help.o") due to help.c's use of
command-list.h.
In terms of implementation: I'm adding $(QUIET) while I'm at it, which
is here so we don't quiet the equivalent of trace output under V=1,
this could be used in other places that use "@cmd" to quiet "cmd"
output.
Some of the dependencies between "command-list.h" and
"$(COMMAND_LIST_GEN)" are redundant, considering that the former
depends on the latter. I'm sticking to listing dependencies mentioned
in the rule itself, e.g. "command-list.h" itself calls
"generate-cmdlist.sh", so I list the dependency even though it would
get it via a recursive dependency.
These rules can be lazy about leaving behind files on error thanks to
the .DELETE_ON_ERROR flag, see 7b76d6bf221 (Makefile: add and use the
".DELETE_ON_ERROR" flag, 2021-06-29), or in the case of the
"*.txt.gen" files because we'll unconditionally clobber them anyway if
the relevant source file is touched, so we can leave them for "make
clean".
1. Piped through: <cmd> 2>/dev/null | grep -P -v "^\s*$"; ditto below:
hyperfine -s basic -L j ", -j8" -L s ,.old -p 'rm -rf .build command-list.h' 'make{j} -f Makefile{s} command-list.h'
Benchmark #1: make -f Makefile command-list.h
Time (mean ± σ): 769.3 ms ± 90.4 ms [User: 892.4 ms, System: 98.8 ms]
Range (min … max): 665.6 ms … 941.9 ms 10 runs
Benchmark #2: make -j8 -f Makefile command-list.h
Time (mean ± σ): 212.5 ms ± 45.0 ms [User: 954.4 ms, System: 136.9 ms]
Range (min … max): 187.6 ms … 326.2 ms 11 runs
Benchmark #3: make -f Makefile.old command-list.h
Time (mean ± σ): 515.0 ms ± 70.5 ms [User: 613.8 ms, System: 110.9 ms]
Range (min … max): 407.6 ms … 603.9 ms 10 runs
Benchmark #4: make -j8 -f Makefile.old command-list.h
Time (mean ± σ): 474.9 ms ± 62.7 ms [User: 569.7 ms, System: 106.0 ms]
Range (min … max): 407.0 ms … 556.8 ms 10 runs
Summary
'make -j8 -f Makefile command-list.h' ran
2.24 ± 0.56 times faster than 'make -j8 -f Makefile.old command-list.h'
2.42 ± 0.61 times faster than 'make -f Makefile.old command-list.h'
3.62 ± 0.88 times faster than 'make -f Makefile command-list.h'
2. hyperfine -s basic -L j ", -j8" -L s ,.old -p 'touch Documentation/git-add.txt' 'make{j} -f Makefile{s} command-list.h'
Benchmark #1: make -f Makefile command-list.h
Time (mean ± σ): 94.6 ms ± 26.7 ms [User: 83.0 ms, System: 14.1 ms]
Range (min … max): 79.4 ms … 186.0 ms 15 runs
Benchmark #2: make -j8 -f Makefile command-list.h
Time (mean ± σ): 80.3 ms ± 10.5 ms [User: 81.3 ms, System: 12.9 ms]
Range (min … max): 76.9 ms … 127.6 ms 37 runs
Benchmark #3: make -f Makefile.old command-list.h
Time (mean ± σ): 348.9 ms ± 174.5 ms [User: 415.3 ms, System: 70.6 ms]
Range (min … max): 66.0 ms … 550.3 ms 43 runs
Benchmark #4: make -j8 -f Makefile.old command-list.h
Time (mean ± σ): 406.7 ms ± 157.1 ms [User: 473.4 ms, System: 78.3 ms]
Range (min … max): 67.4 ms … 560.4 ms 34 runs
Summary
'make -j8 -f Makefile command-list.h' ran
1.18 ± 0.37 times faster than 'make -f Makefile command-list.h'
4.34 ± 2.25 times faster than 'make -f Makefile.old command-list.h'
5.07 ± 2.07 times faster than 'make -j8 -f Makefile.old command-list.h'
3. hyperfine -s basic -L j " -j8" -L s ,.old -p 'touch Documentation/git-add.txt' 'make{j} -f Makefile{s} git'
Benchmark #1: make -j8 -f Makefile git
Time (mean ± σ): 484.2 ms ± 53.5 ms [User: 253.2 ms, System: 83.5 ms]
Range (min … max): 400.7 ms … 580.5 ms 10 runs
Benchmark #2: make -j8 -f Makefile.old git
Time (mean ± σ): 836.7 ms ± 37.3 ms [User: 729.2 ms, System: 157.1 ms]
Range (min … max): 774.2 ms … 885.6 ms 10 runs
Summary
'make -j8 -f Makefile git' ran
1.73 ± 0.21 times faster than 'make -j8 -f Makefile.old git'
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
.gitignore | 1 +
Makefile | 52 +++++++++++++++++++++++++++++++++++++++------
generate-cmdlist.sh | 36 +++++++++++++++++++++++++------
3 files changed, 77 insertions(+), 12 deletions(-)
diff --git a/.gitignore b/.gitignore
index 054249b20a8..dc03a988b16 100644
--- a/.gitignore
+++ b/.gitignore
@@ -198,6 +198,7 @@
*.[aos]
*.o.json
*.py[co]
+.build/
.depend/
*.gcda
*.gcno
diff --git a/Makefile b/Makefile
index 381bed2c1d2..ce4cce57eb8 100644
--- a/Makefile
+++ b/Makefile
@@ -1954,6 +1954,7 @@ endif
ifneq ($(findstring s,$(MAKEFLAGS)),s)
ifndef V
+ QUIET = @
QUIET_CC = @echo ' ' CC $@;
QUIET_AR = @echo ' ' AR $@;
QUIET_LINK = @echo ' ' LINK $@;
@@ -2242,12 +2243,50 @@ config-list.h: generate-configlist.sh
config-list.h: Documentation/*config.txt Documentation/config/*.txt
$(QUIET_GEN)$(SHELL_PATH) ./generate-configlist.sh >$@
-command-list.h: generate-cmdlist.sh command-list.txt
-
-command-list.h: $(wildcard Documentation/git*.txt)
- $(QUIET_GEN)$(SHELL_PATH) ./generate-cmdlist.sh \
- $(patsubst %,--exclude-program %,$(EXCLUDED_PROGRAMS)) \
- command-list.txt >$@
+EXCLUDED_COMMAND_LIST =
+EXCLUDED_COMMAND_LIST += git
+EXCLUDED_COMMAND_LIST += git-bisect-lk2009
+EXCLUDED_COMMAND_LIST += git-credential-cache--daemon
+EXCLUDED_COMMAND_LIST += git-fsck-objects
+EXCLUDED_COMMAND_LIST += git-init-db
+EXCLUDED_COMMAND_LIST += git-mergetool--lib
+EXCLUDED_COMMAND_LIST += git-remote-ext
+EXCLUDED_COMMAND_LIST += git-remote-fd
+EXCLUDED_COMMAND_LIST += git-sh-i18n--envsubst
+EXCLUDED_COMMAND_LIST += git-submodules
+EXCLUDED_COMMAND_LIST += git-tools
+EXCLUDED_COMMAND_LIST += git-version
+EXCLUDED_COMMAND_LIST += git-web--browse
+EXCLUDED_COMMAND_LIST += gitweb.conf
+
+EXCLUDED_TXT += $(patsubst %,Documentation/%.txt,$(EXCLUDED_PROGRAMS) $(EXCLUDED_COMMAND_LIST))
+COMMAND_LIST_TXT_DEP = $(filter-out $(EXCLUDED_TXT), $(wildcard Documentation/git*.txt))
+
+COMMAND_LIST_GEN = $(patsubst Documentation/%.txt,.build/command-list.h.d/%.gen,$(COMMAND_LIST_TXT_DEP))
+
+.build:
+ $(QUIET)mkdir .build
+.build/command-list.h.d: | .build
+ $(QUIET)mkdir -p .build/command-list.h.d
+
+# We must depend on .build/command-list.h as an "order-only"
+# prerequisite, its mtime will change when these targets run.
+$(COMMAND_LIST_GEN): | .build/command-list.h.d
+$(COMMAND_LIST_GEN): command-list.txt
+$(COMMAND_LIST_GEN): generate-cmdlist.sh
+$(COMMAND_LIST_GEN): .build/command-list.h.d/%.gen: Documentation/%.txt
+ $(QUIET)grep "^$(patsubst .build/command-list.h.d/%.gen,%,$@) " command-list.txt >$@.txt && \
+ ./generate-cmdlist.sh --entry-only $@.txt >$@
+
+command-list.h: $(COMMAND_LIST_GEN)
+command-list.h: generate-cmdlist.sh
+command-list.h: command-list.txt
+ $(QUIET_GEN){ \
+ $(SHELL_PATH) ./generate-cmdlist.sh --header-only command-list.txt && \
+ echo "static struct cmdname_help command_list[] = {" && \
+ LC_ALL=C sort $(COMMAND_LIST_GEN) && \
+ echo "};"; \
+ } >$@
hook-list.h: generate-hooklist.sh Documentation/githooks.txt
$(QUIET_GEN)$(SHELL_PATH) ./generate-hooklist.sh >$@
@@ -3238,6 +3277,7 @@ clean: profile-clean coverage-clean cocciclean
$(RM) $(SP_OBJ)
$(RM) $(HCC)
$(RM) -r bin-wrappers $(dep_dirs) $(compdb_dir) compile_commands.json
+ $(RM) -r .build/
$(RM) -r po/build/
$(RM) *.pyc *.pyo */*.pyc */*.pyo $(GENERATED_H) $(ETAGS_TARGET) tags cscope*
$(RM) -r .dist-tmp-dir .doc-tmp-dir
diff --git a/generate-cmdlist.sh b/generate-cmdlist.sh
index a1ab2b1f077..2bc528e8cae 100755
--- a/generate-cmdlist.sh
+++ b/generate-cmdlist.sh
@@ -15,7 +15,7 @@ get_category_line () {
}
category_list () {
- command_list "$1" |
+ grep -v '^#' "$1" |
cut -c 40- |
tr ' ' '\012' |
grep -v '^$' |
@@ -23,13 +23,14 @@ category_list () {
}
get_synopsis () {
+ head -n 10 "Documentation/$1.txt" |
sed -n '
/^NAME/,/'"$1"'/H
${
x
s/.*'"$1"' - \(.*\)/N_("\1")/
p
- }' "Documentation/$1.txt"
+ }'
}
define_categories () {
@@ -61,16 +62,12 @@ define_category_names () {
}
print_command_list () {
- echo "static struct cmdname_help command_list[] = {"
-
- command_list "$1" |
while read cmd rest
do
printf " { \"$cmd\", $(get_synopsis $cmd), 0"
printf " | CAT_%s" $(echo "$rest" | get_category_line)
echo " },"
done
- echo "};"
}
exclude_programs=
@@ -81,6 +78,19 @@ do
shift
done
+header_only=
+case $1 in
+--entry-only)
+ shift
+ print_command_list $1 <"$1"
+ exit 0
+ ;;
+--header-only)
+ shift
+ header_only=t
+ ;;
+esac
+
echo "/* Automatically generated by generate-cmdlist.sh */
struct cmdname_help {
const char *name;
@@ -92,4 +102,18 @@ define_categories "$1"
echo
define_category_names "$1"
echo
+
+if test -n "$header_only"
+then
+ exit 0
+fi
+
+# The old compatibility mode for CMmake. See 061c2240b1b (Introduce
+# CMake support for configuring Git, 2020-06-12)
+echo "static struct cmdname_help command_list[] = {"
+grep -v \
+ -e '^#' \
+ -e '^git-fsck-objects ' \
+ "$1" |
print_command_list "$1"
+echo "};"
--
2.33.1.1338.g20da966911a
next prev parent reply other threads:[~2021-10-20 18:40 UTC|newest]
Thread overview: 87+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-24 13:16 Why the Makefile is so eager to re-build & re-link Ævar Arnfjörð Bjarmason
2021-06-24 15:16 ` Jeff King
2021-06-24 15:28 ` Ævar Arnfjörð Bjarmason
2021-06-24 21:30 ` Johannes Sixt
2021-06-25 8:34 ` Ævar Arnfjörð Bjarmason
2021-06-25 9:01 ` Ævar Arnfjörð Bjarmason
2021-06-29 2:13 ` Jeff King
2021-10-20 18:39 ` [PATCH 0/8] Makefile: make command-list.h 2-5x as fast with -jN Ævar Arnfjörð Bjarmason
2021-10-20 18:39 ` [PATCH 1/8] command-list.txt: sort with "LC_ALL=C sort" Ævar Arnfjörð Bjarmason
2021-10-20 18:39 ` [PATCH 2/8] generate-cmdlist.sh: trivial whitespace change Ævar Arnfjörð Bjarmason
2021-10-20 18:39 ` [PATCH 3/8] generate-cmdlist.sh: spawn fewer processes Ævar Arnfjörð Bjarmason
2021-10-20 18:39 ` [PATCH 4/8] generate-cmdlist.sh: don't call get_categories() from category_list() Ævar Arnfjörð Bjarmason
2021-10-20 18:39 ` [PATCH 5/8] generate-cmdlist.sh: run "grep | sort", not "sort | grep" Ævar Arnfjörð Bjarmason
2021-10-20 18:39 ` [PATCH 6/8] generate-cmdlist.sh: replace for loop by printf's auto-repeat feature Ævar Arnfjörð Bjarmason
2021-10-21 14:42 ` Jeff King
2021-10-21 16:25 ` Jeff King
2021-10-20 18:39 ` Ævar Arnfjörð Bjarmason [this message]
2021-10-21 14:45 ` [PATCH 7/8] Makefile: stop having command-list.h depend on a wildcard Jeff King
2021-10-21 18:24 ` Junio C Hamano
2021-10-21 22:46 ` Øystein Walle
2021-10-20 18:39 ` [PATCH 8/8] Makefile: assert correct generate-cmdlist.sh output Ævar Arnfjörð Bjarmason
2021-10-20 20:35 ` [PATCH 0/8] Makefile: make command-list.h 2-5x as fast with -jN Jeff King
2021-10-20 21:31 ` Taylor Blau
2021-10-20 23:14 ` Ævar Arnfjörð Bjarmason
2021-10-20 23:46 ` Jeff King
2021-10-21 0:48 ` Ævar Arnfjörð Bjarmason
2021-10-21 2:20 ` Taylor Blau
2021-10-22 12:37 ` Ævar Arnfjörð Bjarmason
2021-10-21 14:34 ` Jeff King
2021-10-21 22:34 ` Junio C Hamano
2021-10-22 10:51 ` Ævar Arnfjörð Bjarmason
2021-10-22 18:31 ` Jeff King
2021-10-22 20:50 ` Ævar Arnfjörð Bjarmason
2021-10-21 5:39 ` Eric Sunshine
2021-10-22 19:36 ` [PATCH v2 00/10] Makefile: make generate-cmdlist.sh much faster Ævar Arnfjörð Bjarmason
2021-10-22 19:36 ` [PATCH v2 01/10] command-list.txt: sort with "LC_ALL=C sort" Ævar Arnfjörð Bjarmason
2021-10-25 18:29 ` Junio C Hamano
2021-10-25 21:22 ` Ævar Arnfjörð Bjarmason
2021-10-25 21:26 ` Junio C Hamano
2021-10-22 19:36 ` [PATCH v2 02/10] generate-cmdlist.sh: trivial whitespace change Ævar Arnfjörð Bjarmason
2021-10-22 19:36 ` [PATCH v2 03/10] generate-cmdlist.sh: spawn fewer processes Ævar Arnfjörð Bjarmason
2021-10-22 19:36 ` [PATCH v2 04/10] generate-cmdlist.sh: don't call get_categories() from category_list() Ævar Arnfjörð Bjarmason
2021-10-22 19:36 ` [PATCH v2 05/10] generate-cmdlist.sh: run "grep | sort", not "sort | grep" Ævar Arnfjörð Bjarmason
2021-10-22 19:36 ` [PATCH v2 06/10] generate-cmdlist.sh: replace for loop by printf's auto-repeat feature Ævar Arnfjörð Bjarmason
2021-10-22 19:36 ` [PATCH v2 07/10] generate-cmdlist.sh: stop sorting category lines Ævar Arnfjörð Bjarmason
2021-10-25 16:39 ` Jeff King
2021-10-22 19:36 ` [PATCH v2 08/10] generate-cmdlist.sh: do not shell out to "sed" Ævar Arnfjörð Bjarmason
2021-10-25 16:46 ` Jeff King
2021-10-25 17:52 ` Jeff King
2021-10-22 19:36 ` [PATCH v2 09/10] generate-cmdlist.sh: replace "grep' invocation with a shell version Ævar Arnfjörð Bjarmason
2021-10-23 22:19 ` Junio C Hamano
2021-10-23 22:26 ` Junio C Hamano
2021-10-22 19:36 ` [PATCH v2 10/10] generate-cmdlist.sh: replace "cut", "tr" and "grep" with pure-shell Ævar Arnfjörð Bjarmason
2021-10-23 22:26 ` Junio C Hamano
2021-10-22 21:20 ` [PATCH v2 00/10] Makefile: make generate-cmdlist.sh much faster Taylor Blau
2021-10-23 22:34 ` Junio C Hamano
2021-10-25 16:57 ` Jeff King
2021-11-05 14:07 ` [PATCH v3 00/10] generate-cmdlist.sh: make it (and "make") run faster Ævar Arnfjörð Bjarmason
2021-11-05 14:07 ` [PATCH v3 01/10] command-list.txt: sort with "LC_ALL=C sort" Ævar Arnfjörð Bjarmason
2021-11-05 22:45 ` Junio C Hamano
2021-11-06 4:26 ` Ævar Arnfjörð Bjarmason
2021-11-08 19:18 ` Junio C Hamano
2021-11-05 14:08 ` [PATCH v3 02/10] generate-cmdlist.sh: trivial whitespace change Ævar Arnfjörð Bjarmason
2021-11-05 14:08 ` [PATCH v3 03/10] generate-cmdlist.sh: spawn fewer processes Ævar Arnfjörð Bjarmason
2021-11-05 22:47 ` Junio C Hamano
2021-11-06 4:23 ` Ævar Arnfjörð Bjarmason
2021-11-05 14:08 ` [PATCH v3 04/10] generate-cmdlist.sh: don't call get_categories() from category_list() Ævar Arnfjörð Bjarmason
2021-11-05 14:08 ` [PATCH v3 05/10] generate-cmdlist.sh: run "grep | sort", not "sort | grep" Ævar Arnfjörð Bjarmason
2021-11-05 14:08 ` [PATCH v3 06/10] generate-cmdlist.sh: replace for loop by printf's auto-repeat feature Ævar Arnfjörð Bjarmason
2021-11-05 14:08 ` [PATCH v3 07/10] generate-cmdlist.sh: stop sorting category lines Ævar Arnfjörð Bjarmason
2021-11-05 14:08 ` [PATCH v3 08/10] generate-cmdlist.sh: do not shell out to "sed" Ævar Arnfjörð Bjarmason
2021-11-05 14:08 ` [PATCH v3 09/10] generate-cmdlist.sh: replace "grep' invocation with a shell version Ævar Arnfjörð Bjarmason
2021-11-05 14:08 ` [PATCH v3 10/10] generate-cmdlist.sh: don't parse command-list.txt thrice Ævar Arnfjörð Bjarmason
2021-06-25 21:17 ` Why the Makefile is so eager to re-build & re-link Felipe Contreras
2021-06-29 5:04 ` Eric Sunshine
2021-06-24 23:35 ` Øystein Walle
2021-06-24 23:39 ` Øystein Walle
2021-06-25 0:11 ` Ævar Arnfjörð Bjarmason
2021-07-02 11:58 ` [PATCH] Documentation/Makefile: don't re-build on 'git version' changes Ævar Arnfjörð Bjarmason
2021-07-02 15:53 ` Junio C Hamano
2021-07-03 11:58 ` Ævar Arnfjörð Bjarmason
2021-07-05 19:48 ` Junio C Hamano
2021-07-03 1:05 ` Felipe Contreras
2021-07-03 12:03 ` Ævar Arnfjörð Bjarmason
2021-07-03 18:56 ` Felipe Contreras
2021-07-05 19:38 ` Junio C Hamano
2021-07-06 22:25 ` Felipe Contreras
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=patch-7.8-0c6f9b80d3b-20211020T183533Z-avarab@gmail.com \
--to=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=j6t@kdbg.org \
--cc=oystwa@gmail.com \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).