Re: Why the Makefile is so eager to re-build & re-link

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Fri, Jun 25 2021, Øystein Walle wrote:

> Hi, Ævar
>
>> {command,config}-list.h (and in-flight, my hook-list.h): Every time
>> you touch a Documentation/git-*.txt we need to re-generate these, and
>> since their mtime changes we re-compile and re-link all the way up to
>> libgit and our other tools.
>> 
>> I think the best solution here is to make the generate-*.sh
>> shellscripts faster (just one takes ~300ms of nested shellscripting,
>> just to grep out the first few lines of every git-*.txt, in e.g. Perl
>> or a smarter awk script this would be <5ms).
>> 
>> Then we make those FORCE, but most of the time the config or command
>> summary (or list of hooks) doesn't change, so we don't need to
>> replace the file.
>
> One possible technique to fix this is to generate to a temporary file, and copy
> the temporary file over the real file if it's actually different. I see the
> Makefile already does create a temporary file, so how about something like
> this:
>
> diff --git a/Makefile b/Makefile
> index 93664d6714..9b2f081a2a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2216,7 +2216,8 @@ command-list.h: generate-cmdlist.sh command-list.txt
>  command-list.h: $(wildcard Documentation/git*.txt)
>  	$(QUIET_GEN)$(SHELL_PATH) ./generate-cmdlist.sh \
>  		$(patsubst %,--exclude-program %,$(EXCLUDED_PROGRAMS)) \
> -		command-list.txt >$@+ && mv $@+ $@
> +		command-list.txt >$@+ && \
> +		if ! cmp -s $@ $@+; then mv $@+ $@; fi
>  
>  SCRIPT_DEFINES = $(SHELL_PATH_SQ):$(DIFF_SQ):$(GIT_VERSION):\
>  	$(localedir_SQ):$(NO_CURL):$(USE_GETTEXT_SCHEME):$(SANE_TOOL_PATH_SQ):\
>
>
> This seems to work fine from my basic testing. I think it can even be written
> more terse as `&& ! cmp ... &&` but that looks a bit weird to me. In any case
> it looks like it can easily be added the other relevant places in the Makefile
> too.

I see your downthread "oops", no worries. Just to elaborate slightly on
the comment I had in [1] (and as you've doubtless discovered) then yes,
this works the first time:
    
    $ touch Documentation/git-bisect.txt
    $ make -k -j $(nproc) git
        GEN command-list.h

Without your change we'd have also had to recompile "git", but the
problem is that if you run and re-run that we'll keep making
command-list.h again and again, which is "make" getting stuck on,
because:

    $ make -d -k -j $(nproc) git 2>&1 | grep newer.*command-list.h
        Prerequisite 'Documentation/git-bisect.txt' is newer than target 'command-list.h'.

Hence the "lowest level" comment in [1], i.e. you can only use this
"cmp" trick for things that don't themselves have dependencies.

Which at the end of the day is why make is both wonderful, and why it
sucks.

It sometimes gets hard to force your "circular" dependency graph into
the "square peg" of files and their mtimes, but it's also great for
simplicity that everyone can't write their own custom "this is what it
means for my dependencies to have changed" function (which in practice
would break way more in the hands of most programmers, including yours
truly, than files & their mtimes).

E.g. in this case we don't *actually* depend on Documentation/git*.txt
for the purposes of re-making things all the way up the graph, we depend
on the tiny bit of data we extract from those files, basically just the
NAME section at the very beginning.

The only way to represent that information in a way that "make"
understands would be to add another intermediate step where we extract
exactly that information out into intermediate files with a script, and
then in turn depend on those files.

Which actually, might not be such a bad idea as a solution to this
particular problem.

I.e. just to have individual generated versions of
git-bisect.txt.just-what-command-list.h-needs. We do something vaguely
similar with Documentation/build-docdep.perl, which is another glorious
hack of trying to get make's dependency graph to behave as we'd like.

As a practical matter it wouldn't get you very far except for doc-only
changes, since you'd still have the version.c problem, which also
changes on every (code) commit.

1. http://lore.kernel.org/git/87y2azyzer.fsf@evledraar.gmail.com