Re: [PATCH 0/8] Makefile: make command-list.h 2-5x as fast with -jN

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Wed, Oct 20 2021, Taylor Blau wrote:

> On Thu, Oct 21, 2021 at 02:48:24AM +0200, Ævar Arnfjörð Bjarmason wrote:
>> >> Per Eric's Sunshine's upthread comments an awk and Perl implementation
>> >> were both considered before[1].
>> >
>> > Ah sorry, I thought it was just a perl one that had been the
>> > show-stopper. I hadn't noticed the awk one. However, the point of my
>> > patch was to use perl if available, and fall back otherwise. Maybe
>> > that's too ugly, but it does address the concern with Eric's
>> > implementation.
>>
>> I think carrying two implementations is worse than just having the one
>> slightly slower one.
>
> I have no opinion on whether or not assuming that awk or Perl exists and
> can be relied upon during the build is reasonable or not. It seems like
> the former might be a slightly safer assumption than the latter, but in
> all honesty it seems like both are always likely to be around.
>
> In any case, I think the point was that we could improve upon Peff's
> patch by just having a single implementation done in awk. And when I
> wrote that I definitely was in the mindset of being able to rely on awk
> during compilation.
>
>> >> I.e. I think if you e.g. touch Documentation/git-a*.txt with this series
>> >> with/without this awk version the difference in runtime is within the
>> >> error bars. I.e. making the loop faster isn't necessary. It's better to
>> >> get to a point where make can save you from doing all/most of the work
>> >> by checking modification times, rather than making an O(n) loop faster.
>> >
>> > FWIW, I don't agree with this paragraph at all. Parallelizing or reusing
>> > partial results is IMHO inferior to just making things faster.
>>
>> I agree with you in the general case, but for something that's consumed
>> by a make dependency graph I find it easier to debug things if
>> e.g. changing git-add.txt results in a change to git-add.gen, which is
>> then cat'd together.
>>
>> IOW if we had a sufficiently fast C compiler I think I'd still prefer
>> make's existing rules over some equivalent of:
>>
>>     cat *.c | super-fast-cc
>>
>> Since similar to how the *.sp files depend on the the *.o files now,
>> declaring the dependency graph allows you to easily add more built
>> things.
>
> This seems like an unfair comparison to me. I might be more sympathetic
> if we were generating a more complicated artifact by running
> generate-cmdlist.sh, but its inputs and outputs seem very well defined
> (and non-complicated) to me.

They are? a foo.o to foo.o input is relatively uncomplicated, and you
can discover the exact dependencies with 3rd party tools, like the GCC
and Clang switches we use generate the .depends dirs[1].

Whereas with the custom shellscripts that have for-loops of their own
like generate-cmdlist.sh what it depends on exactly is relatively opaque
to you until you read the shellscript.

I guess it's a matter of taste, but if you run this with/without this
series:

    touch Documentation/git-a*.txt; time make -j1 command-list.h --debug=b V=1

You'll see that before we'd spot that e.g. git-add.txt changed, but
we'll run one target in response to that at the end.

So it's just like what you'd get when you make %.o from %.c to produce a
"program" that links all those %.o together at the end.

So I do think it's a fair comparison, if anything it's unfair to this
series, because as noted you can discover these dependency independently
with GCC etc. for C code. But for a custom *.txt format with an ad-hoc
*.sh to parse it there's no such aid available.

1. $ cat .depend/help.o.d 
help.o: help.c cache.h git-compat-util.h compat/bswap.h wildmatch.h \
 banned.h strbuf.h hashmap.h hash.h repository.h path.h sha1dc_git.h \
 sha1collisiondetection/lib/sha1.h sha256/block/sha256.h list.h advice.h \
 gettext.h convert.h string-list.h trace.h trace2.h pack-revindex.h \
 oid-array.h mem-pool.h config.h builtin.h commit.h object.h tree.h \
 decorate.h gpg-interface.h pretty.h commit-slab.h commit-slab-decl.h \
 commit-slab-impl.h exec-cmd.h run-command.h thread-utils.h strvec.h \
 levenshtein.h help.h command-list.h column.h version.h refs.h \
 parse-options.h prompt.h
[...]