From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: * X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=1.5 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,LIST_MIRROR_RECEIVED,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by dcvr.yhbt.net (Postfix) with ESMTP id 02B9E1F852 for ; Fri, 25 Feb 2022 09:05:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238894AbiBYJFd (ORCPT ); Fri, 25 Feb 2022 04:05:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238887AbiBYJFX (ORCPT ); Fri, 25 Feb 2022 04:05:23 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D9E57DABE for ; Fri, 25 Feb 2022 01:04:51 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id p4so1255382wmg.1 for ; Fri, 25 Feb 2022 01:04:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=py7ySvWb+/RxODcD4K9363GlcZj1CcmPCbpQncMStBM=; b=oPsWghlTUCLwKh0PrKctl0YB10jk7j1kdlYjLJqiEo5oI/IvmFnNvv4t0XgDYY70A5 61FYFQs6usKwqc4LhrcA3ld7x9CmYWVHNEdQQPV3yHuyTnMVTXR3lyJq2G/x4qoL8ySf hKZpNQ7MnXkggAg3/mefwo3ZD8HtgeJ/o5rnZ7cSAK0Pja6q1LqSpGnNDeSRumbTLj3l b1yfje5M4jNJdm/ALivDYZ29DJWfVxHZDuig+DTqNoIAzm2mwaxqDhgfcmmokPQKV96+ TXpoywVldmC0QwBAGQKeksvsVE2ja+oy8DGGS3CSlo8AestEgBnwrJwm2f3BEjlVKzR9 hN3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=py7ySvWb+/RxODcD4K9363GlcZj1CcmPCbpQncMStBM=; b=NgKuN5kksDcqdk785vD3Z2LypB+eVAE7ON1fuDEGgqAYzhkIvRkclcw1ZB7OF+usb1 o2uCt9Rty3FP1C/t+U4UY5ipSrzVCTs5Qy8lZiD2gKOyM86Hckx0tpEOS8BJUywskDgC 27ZsZ2o7352XqoKccRhJQl3cbU3h12zKvxG4QncWBgpjkskjCIvgW1rxMtzwVAit6BcT I0ttFxkGvwhmUR6Lg1HSMH54OhMgA0PdpBsiSVdWsf9bUTBRwnv0uvI05KPV1HYhbxp3 LdLft1hGF9I9S8GYm++3rxgWEB7OVRM1LqNMxqnlz1aXw5upmfScWgCYIhtPUY9r9UMY FdTw== X-Gm-Message-State: AOAM532FLYEwlPbCDe3LmLITWnHpF78W9m0ZPfg3hr91ov1M95PSaMtd wHiWTjQIiJfRTX9mbKxYT923LmUXJwKtsg== X-Google-Smtp-Source: ABdhPJyzwxKFuCkK6y6b5AxxS+l11xNUvKoCXpuEHGwZXlLxOzCcOjibkS/V6V8dsMfyU6j4Ryqs5g== X-Received: by 2002:a1c:2946:0:b0:37b:d710:f565 with SMTP id p67-20020a1c2946000000b0037bd710f565mr1829420wmp.10.1645779888683; Fri, 25 Feb 2022 01:04:48 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l38-20020a05600c1d2600b0037e9090fb1esm5636657wms.24.2022.02.25.01.04.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Feb 2022 01:04:48 -0800 (PST) From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= To: git@vger.kernel.org Cc: Junio C Hamano , Phillip Wood , Jeff King , Dan Jacques , Eric Wong , Jonathan Nieder , Mike Hommey , =?UTF-8?q?=C4=90o=C3=A0n=20Tr=E1=BA=A7n=20C=C3=B4ng=20Danh?= , Victoria Dye , Eric Sunshine , Taylor Blau , =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Subject: [PATCH v3 9/9] Makefiles: add and use wildcard "mkdir -p" template Date: Fri, 25 Feb 2022 10:04:35 +0100 Message-Id: X-Mailer: git-send-email 2.35.1.1175.gf9e1b23ea35 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a template to do the "mkdir -p" of $(@D) (the parent dir of $@) for us, and use it for the "make lint-docs" targets I added in 8650c6298c1 (doc lint: make "lint-docs" non-.PHONY, 2021-10-15). As seen in 4c64fb5aad9 (Documentation/Makefile: fix lint-docs mkdir dependency, 2021-10-26) maintaining these manual lists of parent directory dependencies is fragile, in addition to being obviously verbose. I used this pattern at the time because I couldn't find another method than "order-only" prerequisites to avoid doing a "mkdir -p $(@D)" for every file being created, which as noted in [1] would be significantly slower. But as it turns out we can use this neat trick of only doing a "mkdir -p" if the $(wildcard) macro tells us the path doesn't exist. A re-run of a performance test similar to that noted downthread of [1] in [2] shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]): $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs' Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1 Time (mean ± σ): 2.914 s ± 0.062 s [User: 2.449 s, System: 0.489 s] Range (min … max): 2.834 s … 3.020 s 10 runs Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0 Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s] Range (min … max): 2.229 s … 2.397 s 10 runs Summary 'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran 1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1' So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets. This method of creating parent directories is explicitly racy in that we don't know if we're going to say always create a "foo" followed by a "foo/bar" under parallelism, or skip the "foo" because we created "foo/bar" first. In this case it doesn't matter for anything except that we aren't guaranteed to get the same number of rules firing when running make in parallel. 1. https://lore.kernel.org/git/211028.861r45y3pt.gmgdl@evledraar.gmail.com/ 2. https://lore.kernel.org/git/211028.86o879vvtp.gmgdl@evledraar.gmail.com/ 3. https://gitlab.com/avar/git-hyperfine/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/Makefile | 25 +++---------------------- Makefile | 12 +++++++----- shared.mak | 17 +++++++++++++++++ 3 files changed, 27 insertions(+), 27 deletions(-) diff --git a/Documentation/Makefile b/Documentation/Makefile index 0f4ebdeda8a..1eb9192dae8 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -434,25 +434,11 @@ quick-install-html: require-htmlrepo print-man1: @for i in $(MAN1_TXT); do echo $$i; done -## Lint: Common -.build: - $(QUIET)mkdir $@ -.build/lint-docs: | .build - $(QUIET)mkdir $@ - ## Lint: gitlink -.build/lint-docs/gitlink: | .build/lint-docs - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/howto: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ -.build/lint-docs/gitlink/config: | .build/lint-docs/gitlink - $(QUIET)mkdir $@ LINT_DOCS_GITLINK = $(patsubst %.txt,.build/lint-docs/gitlink/%.ok,$(HOWTO_TXT) $(DOC_DEP_TXT)) -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/howto -$(LINT_DOCS_GITLINK): | .build/lint-docs/gitlink/config $(LINT_DOCS_GITLINK): lint-gitlink.perl $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_GITLINK)$(PERL_PATH) lint-gitlink.perl \ $< \ $(HOWTO_TXT) $(DOC_DEP_TXT) \ @@ -463,23 +449,18 @@ $(LINT_DOCS_GITLINK): .build/lint-docs/gitlink/%.ok: %.txt lint-docs-gitlink: $(LINT_DOCS_GITLINK) ## Lint: man-end-blurb -.build/lint-docs/man-end-blurb: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_END_BLURB = $(patsubst %.txt,.build/lint-docs/man-end-blurb/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_END_BLURB): | .build/lint-docs/man-end-blurb $(LINT_DOCS_MAN_END_BLURB): lint-man-end-blurb.perl $(LINT_DOCS_MAN_END_BLURB): .build/lint-docs/man-end-blurb/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANEND)$(PERL_PATH) lint-man-end-blurb.perl $< >$@ .PHONY: lint-docs-man-end-blurb -lint-docs-man-end-blurb: $(LINT_DOCS_MAN_END_BLURB) ## Lint: man-section-order -.build/lint-docs/man-section-order: | .build/lint-docs - $(QUIET)mkdir $@ LINT_DOCS_MAN_SECTION_ORDER = $(patsubst %.txt,.build/lint-docs/man-section-order/%.ok,$(MAN_TXT)) -$(LINT_DOCS_MAN_SECTION_ORDER): | .build/lint-docs/man-section-order $(LINT_DOCS_MAN_SECTION_ORDER): lint-man-section-order.perl $(LINT_DOCS_MAN_SECTION_ORDER): .build/lint-docs/man-section-order/%.ok: %.txt + $(call mkdir_p_parent_template) $(QUIET_LINT_MANSEC)$(PERL_PATH) lint-man-section-order.perl $< >$@ .PHONY: lint-docs-man-section-order lint-docs-man-section-order: $(LINT_DOCS_MAN_SECTION_ORDER) diff --git a/Makefile b/Makefile index 9f93ee1532c..549ca6e7a5c 100644 --- a/Makefile +++ b/Makefile @@ -2745,7 +2745,8 @@ all:: $(MOFILES) endif po/build/locale/%/LC_MESSAGES/git.mo: po/%.po - $(QUIET_MSGFMT)mkdir -p $(dir $@) && $(MSGFMT) -o $@ $< + $(call mkdir_p_parent_template) + $(QUIET_MSGFMT)$(MSGFMT) -o $@ $< LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm) LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL)) @@ -2761,15 +2762,16 @@ NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS)) endif perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES - $(QUIET_GEN)mkdir -p $(dir $@) && \ + $(call mkdir_p_parent_template) + $(QUIET_GEN) \ sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \ -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \ -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \ < $< > $@ perl/build/man/man3/Git.3pm: perl/Git.pm - $(QUIET_GEN)mkdir -p $(dir $@) && \ - pod2man $< $@ + $(call mkdir_p_parent_template) + $(QUIET_GEN)pod2man $< $@ $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) $(QUIET_GEN)$(RM) $@+ && \ @@ -2903,7 +2905,7 @@ test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $( all:: $(TEST_PROGRAMS) $(test_bindir_programs) bin-wrappers/%: wrap-for-bin.sh - @mkdir -p bin-wrappers + $(call mkdir_p_parent_template) $(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \ -e 's|@@BUILD_DIR@@|$(shell pwd)|' \ -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \ diff --git a/shared.mak b/shared.mak index 6e0f103b65f..7ba04049c8a 100644 --- a/shared.mak +++ b/shared.mak @@ -59,6 +59,8 @@ ifndef V QUIET = @ QUIET_GEN = @echo $(wspfx_SQ) GEN $@; + QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D); + ## Used in "Makefile" QUIET_CC = @echo $(wspfx_SQ) CC $@; QUIET_AR = @echo $(wspfx_SQ) AR $@; @@ -90,3 +92,18 @@ ifndef V export V endif endif + +### Templates + +## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@ +## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not +## needed. +## +## Is racy, but in a good way; we might redundantly (and safely) +## "mkdir -p" when running in parallel, but won't need to exhaustively create +## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a +## "a/prefix/dir/file". This can instead be inserted at the start of +## the "a/prefix/dir/file" rule. +define mkdir_p_parent_template +$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D))) +endef -- 2.35.1.1175.gf9e1b23ea35