git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>,
	"Phillip Wood" <phillip.wood123@gmail.com>,
	"Jeff King" <peff@peff.net>, "Dan Jacques" <dnj@google.com>,
	"Eric Wong" <e@80x24.org>, "Jonathan Nieder" <jrnieder@gmail.com>,
	"Mike Hommey" <mh@glandium.org>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH v3 20/23] Makefile: use $(file) I/O instead of "FORCE" when possible
Date: Tue, 16 Nov 2021 13:00:20 +0100	[thread overview]
Message-ID: <patch-v3-20.23-567ad5c3ebc-20211116T114334Z-avarab@gmail.com> (raw)
In-Reply-To: <cover-v3-00.23-00000000000-20211116T114334Z-avarab@gmail.com>

Provide an alternate implementation of the recently added
"TRACK_template" which uses GNU make 4.2 features. If the 'file'
function is available we don't need to shell out at all to check if
our tracked files change.

We need to use the intermediate TRACK_template calling a
TRACK_template_eval within the generated rule so that we don't eagerly
fleshen these when "make" reads the file.

This doesn't make the runtime faster on my system, but helps to cut
down on the noise of things we shell out for
unconditionally. I.e. running "make" with "SHELL_PATH='sh -x'" (twice,
so we pick up the setting) shows than a no-op run went from emitting:

    $ SHELL_PATH='sh -x' make -j8 >/dev/null 2>&1; SHELL_PATH='sh -x' make 2>&1 |wc -l
    124

To:

    $ SHELL_PATH='sh -x' make -j8 >/dev/null 2>&1; SHELL_PATH='sh -x' make 2>&1 |wc -l
    95

That 124 to 95 number is a rough approximation of how many times we
shell out. "strace -f -c" similarly shows that we went from 8798 to
8466 syscalls. So this brings us further along in the goal of making
"make" do as little as possible when it's got nothing to re-build (see
[2]).

1. https://lore.kernel.org/git/874kdn1j6i.fsf@evledraar.gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 shared.mak | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/shared.mak b/shared.mak
index 363138a5577..4ee0bb7a13d 100644
--- a/shared.mak
+++ b/shared.mak
@@ -8,6 +8,22 @@
 #    info make --index-search=.DELETE_ON_ERROR
 .DELETE_ON_ERROR:
 
+### GNU Make version detection
+# We don't care about "release" versions like the "90" in "3.99.90"
+MAKE_VERSION_MAJOR = $(word 1,$(subst ., ,$(MAKE_VERSION)))
+MAKE_VERSION_MINOR = $(word 2,$(subst ., ,$(MAKE_VERSION)))
+
+# The oldest supported version of GNU make is 3-something. So "not v3"
+# is a future-proof way to ask "is it modern?"
+ifneq ($(MAKE_VERSION_MAJOR),3)
+# $(file >[...]) and $(file >>[...]) is in 4.0...
+MAKE_HAVE_FILE_WRITE = Need version 4.0 or later (released in late 2013)
+# .. but we need 4.2 for $(file <[...])
+ifneq ($(filter-out 0 1,$(MAKE_VERSION_MINOR)),)
+MAKE_HAVE_FILE_READ = Need version 4.2 or later (released in mid-2016)
+endif
+endif
+
 ### Quoting helpers
 
 ## Quote a ' inside a '': FOO='$(call shq,$(BAR))'
@@ -99,6 +115,10 @@ endef
 
 ## Template for making a GIT-SOMETHING, which changes if a
 ## TRACK_SOMETHING variable changes.
+##
+## This is the slower version used on GNU make <4.2.
+ifndef MAKE_HAVE_FILE_READ
+
 define TRACK_template
 .PHONY: FORCE
 $(1): FORCE
@@ -111,3 +131,41 @@ $(1): FORCE
 		echo "$$$$FLAGS" >$(1); \
 	fi
 endef
+
+endif # !MAKE_HAVE_FILE_READ
+
+## A TRACK_template template compatible with the one above. Uses
+## features of GNU make >=4.2 to avoid shelling out for this "hot"
+## "FORCE" logic.
+##
+## Since version >=4.2 can do both "I" and "O" in I/O with using
+## $(file <)/$(file >) we read the GIT-SOMETHING file into a variable
+## with the former, and if it's different from our expected value
+## write it out with the latter.
+ifdef MAKE_HAVE_FILE_READ
+
+define TRACK_template_eval
+$(1)_WRITE =
+$(1)_EXISTS = $(wildcard $(1))
+ifeq ($$($(1)_EXISTS),)
+$(1)_WRITE = new
+else
+$(1)_CONTENT = $(file <$(1))
+ifeq ($$($(1)_CONTENT),$($(2)))
+$(1)_WRITE = same
+else
+$(1)_WRITE = changed
+endif
+endif
+ifneq ($$($(1)_WRITE),same)
+$$(info $$(wspfx) $(1) parameters ($$($(1)_WRITE)))
+$$(file >$(1),$($(2)))
+endif
+endef # TRACK_template_eval
+
+define TRACK_template
+$(1):
+	$$(eval $$(call TRACK_template_eval,$(1),$(2)))
+endef
+
+endif # MAKE_HAVE_FILE_READ
-- 
2.34.0.795.g1e9501ab396


  parent reply	other threads:[~2021-11-16 12:06 UTC|newest]

Thread overview: 107+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-06 21:03 [PATCH 00/16] Makefiles: dependency correctness & speedup Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 01/16] Makefile: don't set up "perl/build" rules under NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 02/16] Makefile: clean perl/build/ even with NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-09 23:10   ` Junio C Hamano
2021-11-06 21:03 ` [PATCH 03/16] Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 04/16] Makefile: move Perl-only variable assignments under !NO_PERL Ævar Arnfjörð Bjarmason
2021-11-09 23:22   ` Junio C Hamano
2021-11-06 21:03 ` [PATCH 05/16] Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph Ævar Arnfjörð Bjarmason
2021-11-07  1:51   ` Eric Sunshine
2021-11-06 21:03 ` [PATCH 06/16] Makefile: don't have Perl over-depend on GIT-BUILD-OPTIONS Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 07/16] Makefile: create a GIT-PYTHON-DEFINES, like "PERL" Ævar Arnfjörð Bjarmason
2021-11-07  1:55   ` Eric Sunshine
2021-11-06 21:03 ` [PATCH 08/16] Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 09/16] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 10/16] Makefile: move $(comma), $(empty) and $(space) to shared.mak Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 11/16] Makefile: re-add and use the "shellquote" macros Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 12/16] Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 13/16] Makefile: add "$(QUIET)" boilerplate to shared.mak Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 14/16] Makefile: use $(wspfx) for $(QUIET...) in shared.mak Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 15/16] Makefiles: add and use wildcard "mkdir -p" template Ævar Arnfjörð Bjarmason
2021-11-06 21:03 ` [PATCH 16/16] Makefile: don't invoke msgfmt with --statistics Ævar Arnfjörð Bjarmason
2021-11-12 21:48 ` [PATCH v2 00/18] Makefiles: dependency correctness & speedup Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 01/18] Makefile: don't invoke msgfmt with --statistics Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 02/18] Makefile: don't set up "perl/build" rules under NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 03/18] Makefile: use "=" not ":=" for po/* and perl/* Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 04/18] Makefile: clean perl/build/ even with NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 05/18] Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 06/18] Makefile: guard Perl-only variable assignments Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 07/18] Makefile: change "ifndef NO_PERL" to "ifdef NO_PERL" Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 08/18] Makefile: adjust Perl-related comments & whitespace Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 09/18] Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 10/18] Makefile: create a GIT-PYTHON-DEFINES, like "PERL" Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 11/18] Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 12/18] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 13/18] Makefile: move $(comma), $(empty) and $(space) to shared.mak Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 14/18] Makefile: re-add and use the "shellquote" macros Ævar Arnfjörð Bjarmason
2021-11-12 21:48   ` [PATCH v2 15/18] Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} Ævar Arnfjörð Bjarmason
2021-11-12 21:49   ` [PATCH v2 16/18] Makefile: add "$(QUIET)" boilerplate to shared.mak Ævar Arnfjörð Bjarmason
2021-11-12 21:49   ` [PATCH v2 17/18] Makefile: use $(wspfx) for $(QUIET...) in shared.mak Ævar Arnfjörð Bjarmason
2021-11-12 21:49   ` [PATCH v2 18/18] Makefiles: add and use wildcard "mkdir -p" template Ævar Arnfjörð Bjarmason
2021-11-16 12:00   ` [PATCH v3 00/23] Makefile: dependency fixes, make noop runtime ~1.4x faster Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 01/23] Makefile: don't invoke msgfmt with --statistics Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 02/23] Makefile: don't set up "perl/build" rules under NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 03/23] Makefile: use "=" not ":=" for po/* and perl/* Ævar Arnfjörð Bjarmason
2021-11-17  1:57       ` Mike Hommey
2021-11-16 12:00     ` [PATCH v3 04/23] Makefile: clean perl/build/ even with NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 05/23] Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR Ævar Arnfjörð Bjarmason
2021-11-17  2:01       ` Mike Hommey
2021-11-17  9:20         ` Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 06/23] Makefile: guard Perl-only variable assignments Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 07/23] Makefile: change "ifndef NO_PERL" to "ifdef NO_PERL" Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 08/23] Makefile: adjust Perl-related comments & whitespace Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 09/23] Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 10/23] Makefile: create a GIT-PYTHON-DEFINES, like "PERL" Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 11/23] Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 12/23] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 13/23] Makefile: move $(comma), $(empty) and $(space) to shared.mak Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 14/23] Makefile: re-add and use the "shellquote" macros Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 15/23] Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} Ævar Arnfjörð Bjarmason
2021-11-17  2:19       ` Mike Hommey
2021-11-17  9:24         ` Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 16/23] Makefile: add "$(QUIET)" boilerplate to shared.mak Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 17/23] Makefile: use $(wspfx) for $(QUIET...) in shared.mak Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 18/23] Makefiles: add and use wildcard "mkdir -p" template Ævar Arnfjörð Bjarmason
2021-11-17  2:51       ` Mike Hommey
2021-11-17  9:26         ` Ævar Arnfjörð Bjarmason
2021-11-17  9:39           ` Mike Hommey
2021-11-17 11:52             ` Ævar Arnfjörð Bjarmason
2021-11-18  0:00               ` Mike Hommey
2021-11-18 13:05                 ` Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 19/23] Makefile: correct the dependency graph of hook-list.h Ævar Arnfjörð Bjarmason
2021-11-17  2:52       ` Mike Hommey
2021-11-16 12:00     ` Ævar Arnfjörð Bjarmason [this message]
2021-11-16 12:00     ` [PATCH v3 21/23] Makefile: disable GNU make built-in wildcard rules Ævar Arnfjörð Bjarmason
2021-11-17  3:00       ` Mike Hommey
2021-11-17  9:47         ` Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 22/23] Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) Ævar Arnfjörð Bjarmason
2021-11-16 12:00     ` [PATCH v3 23/23] Makefile: move ".SUFFIXES" rule to shared.mak Ævar Arnfjörð Bjarmason
2021-11-17 10:19     ` [PATCH v4 00/23] Makefile: dependency fixes, make noop runtime ~1.4x faster Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 01/23] Makefile: don't invoke msgfmt with --statistics Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 02/23] Makefile: don't set up "perl/build" rules under NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 03/23] Makefile: use "=" not ":=" for po/* and perl/* Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 04/23] Makefile: clean perl/build/ even with NO_PERL=Y Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 05/23] Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 06/23] Makefile: guard Perl-only variable assignments Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 07/23] Makefile: change "ifndef NO_PERL" to "ifdef NO_PERL" Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 08/23] Makefile: adjust Perl-related comments & whitespace Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 09/23] Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 10/23] Makefile: create a GIT-PYTHON-DEFINES, like "PERL" Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 11/23] Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 12/23] Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 13/23] Makefile: move $(comma), $(empty) and $(space) to shared.mak Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 14/23] Makefile: re-add and use the "shellquote" macros Ævar Arnfjörð Bjarmason
2021-12-13 23:09         ` Junio C Hamano
2021-12-14  0:31           ` Junio C Hamano
2021-11-17 10:20       ` [PATCH v4 15/23] Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} Ævar Arnfjörð Bjarmason
2021-11-19  6:45         ` Junio C Hamano
2021-11-19  6:56           ` Junio C Hamano
2021-11-19  7:30             ` Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 16/23] Makefile: add "$(QUIET)" boilerplate to shared.mak Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 17/23] Makefile: use $(wspfx) for $(QUIET...) in shared.mak Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 18/23] Makefiles: add and use wildcard "mkdir -p" template Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 19/23] Makefile: correct the dependency graph of hook-list.h Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 20/23] Makefile: use $(file) I/O instead of "FORCE" when possible Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 21/23] Makefile: disable GNU make built-in wildcard rules Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 22/23] Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) Ævar Arnfjörð Bjarmason
2021-11-17 10:20       ` [PATCH v4 23/23] Makefile: move ".SUFFIXES" rule to shared.mak Ævar Arnfjörð Bjarmason

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=patch-v3-20.23-567ad5c3ebc-20211116T114334Z-avarab@gmail.com \
    --to=avarab@gmail.com \
    --cc=dnj@google.com \
    --cc=e@80x24.org \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jrnieder@gmail.com \
    --cc=mh@glandium.org \
    --cc=peff@peff.net \
    --cc=phillip.wood123@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).