git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>,
	"Phillip Wood" <phillip.wood123@gmail.com>,
	"Jeff King" <peff@peff.net>, "Dan Jacques" <dnj@google.com>,
	"Eric Wong" <e@80x24.org>, "Jonathan Nieder" <jrnieder@gmail.com>,
	"Mike Hommey" <mh@glandium.org>,
	"Đoàn Trần Công Danh" <congdanhqx@gmail.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH v4 21/23] Makefile: disable GNU make built-in wildcard rules
Date: Wed, 17 Nov 2021 11:20:20 +0100	[thread overview]
Message-ID: <patch-v4-21.23-dd569a59c74-20211117T101807Z-avarab@gmail.com> (raw)
In-Reply-To: <cover-v4-00.23-00000000000-20211117T101807Z-avarab@gmail.com>

Override built-in rules of GNU make that use a wildcard target. This
speeds things up significantly as we don't need to stat() so many
files just in case we'd be able to retrieve their contents from RCS or
SCCS. See [1] for an old mailing list discussion about how to disable
these.

This gives us a noticeable speedup on a no-op run:

    $ git hyperfine -L rev HEAD~1,HEAD~0 -s 'make -j8 all NO_TCLTK=Y' 'make NO_TCLTK=Y' --warmup 10 -M 10
    Benchmark 1: make NO_TCLTK=Y' in 'HEAD~1
      Time (mean ± σ):     182.2 ms ±   4.1 ms    [User: 146.8 ms, System: 49.5 ms]
      Range (min … max):   179.6 ms … 193.4 ms    10 runs

    Benchmark 2: make NO_TCLTK=Y' in 'HEAD~0
      Time (mean ± σ):     167.9 ms ±   1.3 ms    [User: 127.8 ms, System: 55.6 ms]
      Range (min … max):   166.1 ms … 170.4 ms    10 runs

    Summary
      'make NO_TCLTK=Y' in 'HEAD~0' ran
        1.09 ± 0.03 times faster than 'make NO_TCLTK=Y' in 'HEAD~1'

Running the same except with 'strace -c -S calls make' as the
benchmark command shows (under --show-output) that we went from ~7716
syscalls to ~7519, mostly a reduction in [l]stat().

We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early,
adding a "-r" variant to the above benchmark shows that it may be 1.01
or so faster (but in my tests, always with a much bigger error
bar). But doing so is a much bigger hammer, since it will disable all
built-in rules, some (all?) of which can be seen with:

    make -f/dev/null -p | grep -v -e ^# -e ^$

We may have something that relies on them, so let's go for the more
isolated optimization here that gives us most or all of the wins.

1. https://lists.gnu.org/archive/html/help-make/2002-11/msg00063.html

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 shared.mak | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/shared.mak b/shared.mak
index 2d5d5b2fb9b..20109e6a90e 100644
--- a/shared.mak
+++ b/shared.mak
@@ -1,3 +1,14 @@
+### Remove GNU make implicit rules
+
+## This speeds things up since we don't need to look for and stat() a
+## "foo.c,v" every time a rule referring to "foo.c" is in play. See
+## "make -p -f/dev/null | grep ^%::'".
+%:: %,v
+%:: RCS/%,v
+%:: RCS/%
+%:: s.%
+%:: SCCS/s.%
+
 ### Flags affecting all rules
 
 # A GNU make extension since gmake 3.72 (released in late 1994) to
-- 
2.34.0.796.g2c87ed6146a


  parent reply	other threads:[~2021-11-17 10:21 UTC|newest]

Thread overview: 107+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-06 21:03 [PATCH 00/16] Makefiles: dependency correctness & speedup Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 01/16] Makefile: don't set up "perl/build" rules under NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 02/16] Makefile: clean perl/build/ even with NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-09 23:10   ` Junio C Hamano
2021-11-06 21:03 ` [PATCH 03/16] Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 04/16] Makefile: move Perl-only variable assignments under !NO_PERL Ævar Arnfjörð Bjarmason
2021-11-09 23:22   ` Junio C Hamano
2021-11-06 21:03 ` [PATCH 05/16] Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph Ævar Arnfjörð Bjarmason
2021-11-07  1:51   ` Eric Sunshine
2021-11-06 21:03 ` [PATCH 06/16] Makefile: don't have Perl over-depend on GIT-BUILD-OPTIONS Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 07/16] Makefile: create a GIT-PYTHON-DEFINES, like "PERL" Ævar Arnfjörð Bjarmason
2021-11-07  1:55   ` Eric Sunshine
2021-11-06 21:03 ` [PATCH 08/16] Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 09/16] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 10/16] Makefile: move $(comma), $(empty) and $(space) to shared.mak Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 11/16] Makefile: re-add and use the "shellquote" macros Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 12/16] Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 13/16] Makefile: add "$(QUIET)" boilerplate to shared.mak Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 14/16] Makefile: use $(wspfx) for $(QUIET...) in shared.mak Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 15/16] Makefiles: add and use wildcard "mkdir -p" template Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 16/16] Makefile: don't invoke msgfmt with --statistics Ævar Arnfjörð Bjarmason
2021-11-12 21:48 ` [PATCH v2 00/18] Makefiles: dependency correctness & speedup Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 01/18] Makefile: don't invoke msgfmt with --statistics Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 02/18] Makefile: don't set up "perl/build" rules under NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 03/18] Makefile: use "=" not ":=" for po/* and perl/* Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 04/18] Makefile: clean perl/build/ even with NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 05/18] Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 06/18] Makefile: guard Perl-only variable assignments Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 07/18] Makefile: change "ifndef NO_PERL" to "ifdef NO_PERL" Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 08/18] Makefile: adjust Perl-related comments & whitespace Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 09/18] Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 10/18] Makefile: create a GIT-PYTHON-DEFINES, like "PERL" Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 11/18] Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 12/18] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 13/18] Makefile: move $(comma), $(empty) and $(space) to shared.mak Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 14/18] Makefile: re-add and use the "shellquote" macros Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 15/18] Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} Ævar Arnfjörð Bjarmason
2021-11-12 21:49   ` [PATCH v2 16/18] Makefile: add "$(QUIET)" boilerplate to shared.mak Ævar Arnfjörð Bjarmason
2021-11-12 21:49   ` [PATCH v2 17/18] Makefile: use $(wspfx) for $(QUIET...) in shared.mak Ævar Arnfjörð Bjarmason
2021-11-12 21:49   ` [PATCH v2 18/18] Makefiles: add and use wildcard "mkdir -p" template Ævar Arnfjörð Bjarmason
2021-11-16 12:00   ` [PATCH v3 00/23] Makefile: dependency fixes, make noop runtime ~1.4x faster Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 01/23] Makefile: don't invoke msgfmt with --statistics Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 02/23] Makefile: don't set up "perl/build" rules under NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 03/23] Makefile: use "=" not ":=" for po/* and perl/* Ævar Arnfjörð Bjarmason
2021-11-17  1:57       ` Mike Hommey
2021-11-16 12:00     ` [PATCH v3 04/23] Makefile: clean perl/build/ even with NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 05/23] Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR Ævar Arnfjörð Bjarmason
2021-11-17  2:01       ` Mike Hommey
2021-11-17  9:20         ` Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 06/23] Makefile: guard Perl-only variable assignments Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 07/23] Makefile: change "ifndef NO_PERL" to "ifdef NO_PERL" Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 08/23] Makefile: adjust Perl-related comments & whitespace Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 09/23] Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 10/23] Makefile: create a GIT-PYTHON-DEFINES, like "PERL" Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 11/23] Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 12/23] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 13/23] Makefile: move $(comma), $(empty) and $(space) to shared.mak Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 14/23] Makefile: re-add and use the "shellquote" macros Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 15/23] Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} Ævar Arnfjörð Bjarmason
2021-11-17  2:19       ` Mike Hommey
2021-11-17  9:24         ` Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 16/23] Makefile: add "$(QUIET)" boilerplate to shared.mak Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 17/23] Makefile: use $(wspfx) for $(QUIET...) in shared.mak Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 18/23] Makefiles: add and use wildcard "mkdir -p" template Ævar Arnfjörð Bjarmason
2021-11-17  2:51       ` Mike Hommey
2021-11-17  9:26         ` Ævar Arnfjörð Bjarmason
2021-11-17  9:39           ` Mike Hommey
2021-11-17 11:52             ` Ævar Arnfjörð Bjarmason
2021-11-18  0:00               ` Mike Hommey
2021-11-18 13:05                 ` Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 19/23] Makefile: correct the dependency graph of hook-list.h Ævar Arnfjörð Bjarmason
2021-11-17  2:52       ` Mike Hommey
2021-11-16 12:00     ` [PATCH v3 20/23] Makefile: use $(file) I/O instead of "FORCE" when possible Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 21/23] Makefile: disable GNU make built-in wildcard rules Ævar Arnfjörð Bjarmason
2021-11-17  3:00       ` Mike Hommey
2021-11-17  9:47         ` Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 22/23] Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 23/23] Makefile: move ".SUFFIXES" rule to shared.mak Ævar Arnfjörð Bjarmason
2021-11-17 10:19     ` [PATCH v4 00/23] Makefile: dependency fixes, make noop runtime ~1.4x faster Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 01/23] Makefile: don't invoke msgfmt with --statistics Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 02/23] Makefile: don't set up "perl/build" rules under NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 03/23] Makefile: use "=" not ":=" for po/* and perl/* Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 04/23] Makefile: clean perl/build/ even with NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 05/23] Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 06/23] Makefile: guard Perl-only variable assignments Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 07/23] Makefile: change "ifndef NO_PERL" to "ifdef NO_PERL" Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 08/23] Makefile: adjust Perl-related comments & whitespace Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 09/23] Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 10/23] Makefile: create a GIT-PYTHON-DEFINES, like "PERL" Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 11/23] Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 12/23] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 13/23] Makefile: move $(comma), $(empty) and $(space) to shared.mak Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 14/23] Makefile: re-add and use the "shellquote" macros Ævar Arnfjörð Bjarmason
2021-12-13 23:09         ` Junio C Hamano
2021-12-14  0:31           ` Junio C Hamano
2021-11-17 10:20       ` [PATCH v4 15/23] Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} Ævar Arnfjörð Bjarmason
2021-11-19  6:45         ` Junio C Hamano
2021-11-19  6:56           ` Junio C Hamano
2021-11-19  7:30             ` Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 16/23] Makefile: add "$(QUIET)" boilerplate to shared.mak Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 17/23] Makefile: use $(wspfx) for $(QUIET...) in shared.mak Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 18/23] Makefiles: add and use wildcard "mkdir -p" template Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 19/23] Makefile: correct the dependency graph of hook-list.h Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 20/23] Makefile: use $(file) I/O instead of "FORCE" when possible Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` Ævar Arnfjörð Bjarmason [this message]
2021-11-17 10:20       ` [PATCH v4 22/23] Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 23/23] Makefile: move ".SUFFIXES" rule to shared.mak Ævar Arnfjörð Bjarmason

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=patch-v4-21.23-dd569a59c74-20211117T101807Z-avarab@gmail.com \
    --to=avarab@gmail.com \
    --cc=congdanhqx@gmail.com \
    --cc=dnj@google.com \
    --cc=e@80x24.org \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jrnieder@gmail.com \
    --cc=mh@glandium.org \
    --cc=peff@peff.net \
    --cc=phillip.wood123@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).