[PATCH 0/5] Misc Coccinelle-related improvements

[PATCH 0/5] Misc Coccinelle-related improvements

[SZEDER Gábor]

Just a couple of minor Coccinelle-related improvements:

  - The first two patches are small cleanups.

  - The last three could make life perhaps just a tad bit easier for
    devs running 'make coccicheck'.


SZEDER Gábor (5):
  coccinelle: mark the 'coccicheck' make target as .PHONY
  coccinelle: use $(addsuffix) in 'coccicheck' make target
  coccinelle: exclude sha1dc source files from static analysis
  coccinelle: put sane filenames into output patches
  coccinelle: extract dedicated make target to clean Coccinelle's
    results

 Makefile | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

-- 
2.18.0.408.g42635c01bc

[PATCH 1/5] coccinelle: mark the 'coccicheck' make target as .PHONY

[SZEDER Gábor]

The 'coccicheck' target doesn't create a file with the same name, so
mark it as .PHONY.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Makefile b/Makefile
index e4b503d259..27bfc196dd 100644
--- a/Makefile
+++ b/Makefile
@@ -2685,6 +2685,8 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
 	fi
 coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci))
 
+.PHONY: coccicheck
+
 ### Installation rules
 
 ifneq ($(filter /%,$(firstword $(template_dir))),)
-- 
2.18.0.408.g42635c01bc

Re: [PATCH 1/5] coccinelle: mark the 'coccicheck' make target as .PHONY

[Derrick Stolee]

On 7/23/2018 9:50 AM, SZEDER Gábor wrote:
> The 'coccicheck' target doesn't create a file with the same name, so
> mark it as .PHONY.
>
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
>   Makefile | 2 ++
>   1 file changed, 2 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index e4b503d259..27bfc196dd 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2685,6 +2685,8 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
>   	fi
>   coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci))
>   
> +.PHONY: coccicheck
> +
>   ### Installation rules
>   
>   ifneq ($(filter /%,$(firstword $(template_dir))),)

I did not know about phony targets [1] so thanks for teaching me 
something today.

[1] https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html

Re: [PATCH 1/5] coccinelle: mark the 'coccicheck' make target as .PHONY

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> The 'coccicheck' target doesn't create a file with the same name, so
> mark it as .PHONY.
>
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---

Good.  It is customary to do so immediately before the target, not
after a blank line, though.

>  Makefile | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index e4b503d259..27bfc196dd 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2685,6 +2685,8 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
>  	fi
>  coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci))
>  
> +.PHONY: coccicheck
> +
>  ### Installation rules
>  
>  ifneq ($(filter /%,$(firstword $(template_dir))),)

[PATCH 2/5] coccinelle: use $(addsuffix) in 'coccicheck' make target

[SZEDER Gábor]

The dependencies of the 'coccicheck' make target are listed with the
help of the $(patsubst) make function, which in this case doesn't do
any pattern substitution, but only adds the '.patch' suffix.

Use the shorter and more idiomatic $(addsuffix) make function instead.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 27bfc196dd..8f509576e9 100644
--- a/Makefile
+++ b/Makefile
@@ -2683,7 +2683,7 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
 	then \
 		echo '    ' SPATCH result: $@; \
 	fi
-coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci))
+coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci))
 
 .PHONY: coccicheck
 
-- 
2.18.0.408.g42635c01bc

Re: [PATCH 2/5] coccinelle: use $(addsuffix) in 'coccicheck' make target

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> The dependencies of the 'coccicheck' make target are listed with the
> help of the $(patsubst) make function, which in this case doesn't do
> any pattern substitution, but only adds the '.patch' suffix.
>
> Use the shorter and more idiomatic $(addsuffix) make function instead.

Makes sense.

>
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index 27bfc196dd..8f509576e9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2683,7 +2683,7 @@ C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
>  	then \
>  		echo '    ' SPATCH result: $@; \
>  	fi
> -coccicheck: $(patsubst %.cocci,%.cocci.patch,$(wildcard contrib/coccinelle/*.cocci))
> +coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci))
>  
>  .PHONY: coccicheck

[PATCH 3/5] coccinelle: exclude sha1dc source files from static analysis

[SZEDER Gábor]

sha1dc is an external library, that we carry in-tree for convenience
or grab as a submodule, so there is no use in applying our semantic
patches to its source files.

Therefore, exclude sha1dc's source files from Coccinelle's static
analysis.

This change also makes the static analysis somewhat faster: presumably
because of the heavy use of repetitive macro declarations, applying
the semantic patches 'array.cocci' and 'swap.cocci' to 'sha1dc/sha1.c'
takes over half a minute each on my machine, which amounts to about a
third of the runtime of applying these two semantic patches to the
whole git source tree.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 Makefile | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 8f509576e9..73e2d16926 100644
--- a/Makefile
+++ b/Makefile
@@ -2666,10 +2666,16 @@ check: command-list.h
 	fi
 
 C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
-%.cocci.patch: %.cocci $(C_SOURCES)
+ifdef DC_SHA1_SUBMODULE
+COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
+else
+COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
+endif
+
+%.cocci.patch: %.cocci $(COCCI_SOURCES)
 	@echo '    ' SPATCH $<; \
 	ret=0; \
-	for f in $(C_SOURCES); do \
+	for f in $(COCCI_SOURCES); do \
 		$(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \
 			{ ret=$$?; break; }; \
 	done >$@+ 2>$@.log; \
-- 
2.18.0.408.g42635c01bc

Re: [PATCH 3/5] coccinelle: exclude sha1dc source files from static analysis

[Eric Sunshine]

On Mon, Jul 23, 2018 at 9:51 AM SZEDER Gábor <szeder.dev@gmail.com> wrote:
> sha1dc is an external library, that we carry in-tree for convenience
> or grab as a submodule, so there is no use in applying our semantic
> patches to its source files.
>
> Therefore, exclude sha1dc's source files from Coccinelle's static
> analysis.
>
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
> diff --git a/Makefile b/Makefile
> @@ -2666,10 +2666,16 @@ check: command-list.h
> +ifdef DC_SHA1_SUBMODULE
> +COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
> +else
> +COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
> +endif

Can't you just filter out both of these unconditionally without
worrying about DC_SHA1_SUBMODULE?

Re: [PATCH 3/5] coccinelle: exclude sha1dc source files from static analysis

[SZEDER Gábor]

On Mon, Jul 23, 2018 at 8:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> On Mon, Jul 23, 2018 at 9:51 AM SZEDER Gábor <szeder.dev@gmail.com> wrote:
> > sha1dc is an external library, that we carry in-tree for convenience
> > or grab as a submodule, so there is no use in applying our semantic
> > patches to its source files.
> >
> > Therefore, exclude sha1dc's source files from Coccinelle's static
> > analysis.
> >
> > Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> > ---
> > diff --git a/Makefile b/Makefile
> > @@ -2666,10 +2666,16 @@ check: command-list.h
> > +ifdef DC_SHA1_SUBMODULE
> > +COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
> > +else
> > +COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
> > +endif
>
> Can't you just filter out both of these unconditionally without
> worrying about DC_SHA1_SUBMODULE?

I'm not sure what you mean by that.  Like this perhaps?

  COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(filter-out
sha1dc/%,$(C_SOURCES)))

While it's only a single line, I don't think it's any easier on the
eyes.

Re: [PATCH 3/5] coccinelle: exclude sha1dc source files from static analysis

[Eric Sunshine]

On Mon, Jul 23, 2018 at 2:44 PM SZEDER Gábor <szeder.dev@gmail.com> wrote:
> On Mon, Jul 23, 2018 at 8:28 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > On Mon, Jul 23, 2018 at 9:51 AM SZEDER Gábor <szeder.dev@gmail.com> wrote:
> > > +ifdef DC_SHA1_SUBMODULE
> > > +COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
> > > +else
> > > +COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
> > > +endif
> >
> > Can't you just filter out both of these unconditionally without
> > worrying about DC_SHA1_SUBMODULE?
>
> I'm not sure what you mean by that.  Like this perhaps?
>
>   COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(filter-out
> sha1dc/%,$(C_SOURCES)))
>
> While it's only a single line, I don't think it's any easier on the
> eyes.

I wasn't worried about readability or one or two lines (indeed, you
could still do the filtering over two statements).

What I meant was that sha1dc/ contains files whether DC_SHA1_SUBMODULE
is defined or not. If the idea of this change is that there's no point
in having Coccinelle check those foreign, imported files (and waste
time in the process), then I was thinking that you'd want to omit
sha1dc/* regardless of whether DC_SHA1_SUBMODULE is defined.

Looking more closely at the Makefile, however, I see that C_SOURCES
holds only one or the other of sha1dc/* or
sha1collisiondetection/lib/*, so my concern is unfounded, which
explains why my question confused you.

[PATCH 4/5] coccinelle: put sane filenames into output patches

[SZEDER Gábor]

Coccinelle outputs its suggested transformations as patches, whose
header looks something like this:

  --- commit.c
  +++ /tmp/cocci-output-19250-7ae78a-commit.c

Note the lack of 'diff --opts <old> <new>' line, the differing number
of path components on the --- and +++ lines, and the nonsensical
filename on the +++ line.  'patch -p0' can still apply these patches,
as it takes the filename to be modified from the --- line.  Alas, 'git
apply' can't, because it takes the filename from the +++ line, and
then complains about the nonexisting file.

Pass the '--patch .' options to Coccinelle via the SPATCH_FLAGS 'make'
variable, as it seems to make it generate proper context diff patches,
with the header starting with a 'diff ...' line and containing sane
filenames.  The resulting 'contrib/coccinelle/*.cocci.patch' files
then can be applied both with 'git apply' and 'patch' (even without
'-p0').

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 73e2d16926..72ea29df4e 100644
--- a/Makefile
+++ b/Makefile
@@ -564,7 +564,7 @@ SPATCH = spatch
 export TCL_PATH TCLTK_PATH
 
 SPARSE_FLAGS =
-SPATCH_FLAGS = --all-includes
+SPATCH_FLAGS = --all-includes --patch .
 
 
 
-- 
2.18.0.408.g42635c01bc

Re: [PATCH 4/5] coccinelle: put sane filenames into output patches

[Derrick Stolee]

On 7/23/2018 9:50 AM, SZEDER Gábor wrote:
> Coccinelle outputs its suggested transformations as patches, whose
> header looks something like this:
>
>    --- commit.c
>    +++ /tmp/cocci-output-19250-7ae78a-commit.c
>
> Note the lack of 'diff --opts <old> <new>' line, the differing number
> of path components on the --- and +++ lines, and the nonsensical
> filename on the +++ line.  'patch -p0' can still apply these patches,
> as it takes the filename to be modified from the --- line.  Alas, 'git
> apply' can't, because it takes the filename from the +++ line, and
> then complains about the nonexisting file.
>
> Pass the '--patch .' options to Coccinelle via the SPATCH_FLAGS 'make'
> variable, as it seems to make it generate proper context diff patches,
> with the header starting with a 'diff ...' line and containing sane
> filenames.  The resulting 'contrib/coccinelle/*.cocci.patch' files
> then can be applied both with 'git apply' and 'patch' (even without
> '-p0').
>
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
>   Makefile | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index 73e2d16926..72ea29df4e 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -564,7 +564,7 @@ SPATCH = spatch
>   export TCL_PATH TCLTK_PATH
>   
>   SPARSE_FLAGS =
> -SPATCH_FLAGS = --all-includes
> +SPATCH_FLAGS = --all-includes --patch .

Thank you for this! These garbage filenames cause significant pain when 
trying to apply the suggested patch.

[PATCH 5/5] coccinelle: extract dedicated make target to clean Coccinelle's results

[SZEDER Gábor]

Sometimes I want to remove only Coccinelle's results, but keep all
other build artifacts left after my usual 'make all man' build.  This
new 'cocciclean' make target will allow just that.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 Makefile | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 72ea29df4e..aee629cbaf 100644
--- a/Makefile
+++ b/Makefile
@@ -2903,7 +2903,10 @@ profile-clean:
 	$(RM) $(addsuffix *.gcda,$(addprefix $(PROFILE_DIR)/, $(object_dirs)))
 	$(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs)))
 
-clean: profile-clean coverage-clean
+cocciclean:
+	$(RM) contrib/coccinelle/*.cocci.patch*
+
+clean: profile-clean coverage-clean cocciclean
 	$(RM) *.res
 	$(RM) $(OBJECTS)
 	$(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB)
@@ -2915,7 +2918,6 @@ clean: profile-clean coverage-clean
 	$(RM) -r $(GIT_TARNAME) .doc-tmp-dir
 	$(RM) $(GIT_TARNAME).tar.gz git-core_$(GIT_VERSION)-*.tar.gz
 	$(RM) $(htmldocs).tar.gz $(manpages).tar.gz
-	$(RM) contrib/coccinelle/*.cocci.patch*
 	$(MAKE) -C Documentation/ clean
 ifndef NO_PERL
 	$(MAKE) -C gitweb clean
@@ -2931,7 +2933,7 @@ endif
 	$(RM) GIT-USER-AGENT GIT-PREFIX
 	$(RM) GIT-SCRIPT-DEFINES GIT-PERL-DEFINES GIT-PERL-HEADER GIT-PYTHON-VARS
 
-.PHONY: all install profile-clean clean strip
+.PHONY: all install profile-clean cocciclean clean strip
 .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
 .PHONY: FORCE cscope
 
-- 
2.18.0.408.g42635c01bc

Re: [PATCH 0/5] Misc Coccinelle-related improvements

[Derrick Stolee]

On 7/23/2018 9:50 AM, SZEDER Gábor wrote:
> Just a couple of minor Coccinelle-related improvements:
>
>    - The first two patches are small cleanups.
>
>    - The last three could make life perhaps just a tad bit easier for
>      devs running 'make coccicheck'.

I appreciate your focus on making 'make coccicheck' easier to use!

Reviewed-by: Derrick Stolee <dstolee@microsoft.com>

Re: [PATCH 0/5] Misc Coccinelle-related improvements

[Duy Nguyen]

On Mon, Jul 23, 2018 at 3:51 PM SZEDER Gábor <szeder.dev@gmail.com> wrote:
>
> Just a couple of minor Coccinelle-related improvements:
>
>   - The first two patches are small cleanups.
>
>   - The last three could make life perhaps just a tad bit easier for
>     devs running 'make coccicheck'.

I'm not a heavy cocci user and probably won't spot any corner case
problems. With that in mind, I've read this though and the series
looks good to me.
-- 
Duy

Re: [PATCH 0/5] Misc Coccinelle-related improvements

[René Scharfe]

Am 23.07.2018 um 15:50 schrieb SZEDER Gábor:
> Just a couple of minor Coccinelle-related improvements:
> 
>    - The first two patches are small cleanups.
> 
>    - The last three could make life perhaps just a tad bit easier for
>      devs running 'make coccicheck'.
> 
> 
> SZEDER Gábor (5):
>    coccinelle: mark the 'coccicheck' make target as .PHONY
>    coccinelle: use $(addsuffix) in 'coccicheck' make target
>    coccinelle: exclude sha1dc source files from static analysis
>    coccinelle: put sane filenames into output patches
>    coccinelle: extract dedicated make target to clean Coccinelle's
>      results

These patches look good to me.  Thanks a lot!

René

Re: [PATCH 0/5] Misc Coccinelle-related improvements

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> Just a couple of minor Coccinelle-related improvements:
>
>   - The first two patches are small cleanups.
>
>   - The last three could make life perhaps just a tad bit easier for
>     devs running 'make coccicheck'.

Thanks.  All 5 patches make sense.

Queued.

[PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[SZEDER Gábor]

So, I have this PoC patch below on top of this patch series for some
months now; actually this series was all fallout from working on this
patch.  It makes 'make -j<N> coccicheck' much faster (speedup of over
99% :) in some scenarios that, in my experience, occur rather
frequently when working on Coccinelle semantic patches (as opposed to
occasionally running 'make coccicheck' to see whether there are any
regressions).

However, it's not ready for inclusion as it is, because,
unfortunately, the improvements are not without some disadvantages, as
explained in the second half of the commit message below.

Anyway, I'm sending this out, because I don't see how I could make it
any better, but other contributors working on semantic patches might
still find it useful to keep this in their own fork, even in its
current form.  And perhaps someone will come up with a brilliant idea
to address those disadvantages...

Also available at:

  https://github.com/szeder/git make-coccicheck-finegrained


  -- >8 --

Subject: [PATCH] [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

When running 'make coccicheck', each semantic patch is applied to all
source files in a separate shell for loop.  This approach has the
following disadvantages:

  1. Even if you only modified a single C source file, that shell loop
     will still iterate over all source files and apply the semantic
     patches to all of them, wasting a lot of time.

  2. If you apply only a single semantic patch (either implicitly,
     after you modified only one of them (and the results of the
     previous 'make coccicheck' are still there), or explicitly, by
     running e.g. 'make contrib/coccinelle/array.cocci.patch'), then
     you won't be able to exploit multiple CPU cores to speed up the
     operation, because that shell loop will iterate over all source
     files sequentially.

  3. 'make coccicheck' can only use as many parallel jobs as the
     number of semantic patches, so even if you have more available
     CPU cores than that, you can't exploit them all to speed up this
     operation.

  4. During 'make coccicheck' there is only a single

       SPATCH contrib/coccinelle/<whatever>.cocci

     line (or as many lines as the number of parallel jobs) of output
     when starting to apply each semantic patch, and then comes a long
     silence without any indication of progress, because applying some
     of our semantic patches to all source files can take over a
     minute.  This can trick developers new to semantic patches into
     thinking that something went wrong, hung, ended up in an endless
     loop, etc.  It certainly confused my the first time.

Let's add a bit of Makefile metaprogramming to generate finer-grained
make targets applying one semantic patch to only a single source file,
and specify these as dependencies of the targets applying one semantic
patch to all source files.  This way that shell loop can be avoided,
semantic patches will only be applied to changed source files, and the
same semantic patch can be applied in parallel to multiple source
files.  The only remaining sequential part is aggregating the
suggested transformations from the individual targets into a single
patch file, which is comparatively cheap (especially since ideally
there aren't any suggestions).

This change brings spectacular speedup in the scenario described in
point (1) above.  When the results of a previous 'make coccicheck' are
there, the time needed to run

  touch apply.c ; time make -j4 coccicheck

went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and
'apply.c' is the second longest source file in our codebase.  In the
scenario in point (2), running

  touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck

went from 56.364s to 35.883s, which is ~36% speedup.

All the above timings are best-of-five on a machine with 2 physical
and 2 logical cores.  I don't have the hardware to bring any numbers
for point (3).  The time needed to run 'make -j4 coccicheck' in a
clean state didn't change, it's ~3m42s both with and without this
patch.

Unfortunately, this new approach has some disadvantages compared to
the current situation:

  - [RFC]
    With this patch 'make coccicheck's output will look like this
    ('make' apparently doesn't iterate over semantic patches in
    lexicographical order):

      SPATCH commit.cocci              abspath.c
      SPATCH commit.cocci              advice.c
      <... lots of lines ...>
      SPATCH array.cocci               http-walker.c
      SPATCH array.cocci               remote-curl.c

    which means that the number of lines in the output grows from
    "Nr-of-semantic-patches" to "Nr-of-semantic-patches *
    Nr-of-source-files".

    Now, while this certainly addresses point (4) above, it can be
    considered too much, and I'm not sure that (4) is that big an
    issue to justify this much output.

    OTOH, I couldn't yet figure out a way to print a single 'SPATCH
    contrib/coccinelle/<whatever>.cocci' line at the beginning of
    applying that semantic patch without triggering unnecessary work,
    effectively killing most of the runtime benefits.  Or to somehow
    iterate over source files in the outer loop and over semantic
    patches in the inner loop, so we could have one output line per
    file.

  - [RFC]
    The overhead of applying a semantic patch to all source files
    became larger.  'make coccicheck' currently runs only one shell
    process and creates two output files for each semantic patch.
    With this patch it will run approx.  "Nr-of-semantic-patches *
    Nr-of-source-files" shell processes and create twice as many
    output files.

    This overhead can be quantified by measuring the runtime of:

      make -j4 SPATCH=true coccicheck

    and this patch increases it from 0.142s to 1.479s.  While this is
    indeed an order of magnitude increase, it's still negligible
    compared to the "real" runtime of 3m42s.

    So the increased overhead doesn't seem to matter on Linux, but I
    would expect that it's worse on OSX and Windows; though I'm not
    sure whether Coccinelle (with all its OCaml dependencies) works on
    those platforms in the first place.

  - [RFC]
    This approach uses $(eval), which we haven't used in any of our
    Makefiles yet.  I wonder whether it's portable enough.  And it's
    ugly and complicated.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 Makefile                      | 52 ++++++++++++++++++++++++-----------
 contrib/coccinelle/.gitignore |  3 +-
 2 files changed, 38 insertions(+), 17 deletions(-)

diff --git a/Makefile b/Makefile
index d616c04125..f516dd93d1 100644
--- a/Makefile
+++ b/Makefile
@@ -2674,25 +2674,44 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
 else
 COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
 endif
+COCCI_SEM_PATCHES = $(wildcard contrib/coccinelle/*.cocci)
 
-%.cocci.patch: %.cocci $(COCCI_SOURCES)
-	@echo '    ' SPATCH $<; \
-	ret=0; \
-	for f in $(COCCI_SOURCES); do \
-		$(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \
-			{ ret=$$?; break; }; \
-	done >$@+ 2>$@.log; \
-	if test $$ret != 0; \
+define cocci_template
+$(cocci_sem_patch)_dirs := $$(addprefix $(cocci_sem_patch).patches/,$$(sort $$(dir $$(COCCI_SOURCES))))
+
+$$($(cocci_sem_patch)_dirs):
+	@mkdir -p $$@
+
+# e.g. 'contrib/coccinelle/strbuf.cocci.patches/builtin/commit.c.patch'
+# Applies one semantic patch to _one_ source file.
+$(cocci_sem_patch).patches/%.patch: % $(cocci_sem_patch)
+	@printf '     SPATCH %-25s %s\n' $$(notdir $(cocci_sem_patch)) $$<; \
+	if ! $$(SPATCH) --sp-file $(cocci_sem_patch) $$< $$(SPATCH_FLAGS) >$$@ 2>$$@.log; \
 	then \
-		cat $@.log; \
+		rm -f $$@; \
+		cat $$@.log; \
 		exit 1; \
-	fi; \
-	mv $@+ $@; \
-	if test -s $@; \
-	then \
-		echo '    ' SPATCH result: $@; \
 	fi
-coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci))
+
+# e.g. 'contrib/coccinelle/strbuf.cocci.patch'
+# Applies one semantic patch to _all_ source files.
+$(cocci_sem_patch).patch: $(cocci_sem_patch) $$($(cocci_sem_patch)_dirs) $$(patsubst %,$(cocci_sem_patch).patches/%.patch,$(COCCI_SOURCES))
+	@>$$@+; \
+	for f in $$(filter %.patch,$$^); do \
+		if test -s $$$$f; \
+		then \
+			cat $$$$f >>$$@+; \
+		fi \
+	done; \
+	mv $$@+ $$@; \
+	if test -s $$@; then \
+		echo '    ' SPATCH result: $$@; \
+	fi
+endef
+
+$(foreach cocci_sem_patch,$(COCCI_SEM_PATCHES),$(eval $(cocci_template)))
+
+coccicheck: $(addsuffix .patch,$(COCCI_SEM_PATCHES))
 
 .PHONY: coccicheck
 
@@ -2907,7 +2926,8 @@ profile-clean:
 	$(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs)))
 
 cocciclean:
-	$(RM) contrib/coccinelle/*.cocci.patch*
+	$(RM) contrib/coccinelle/*.cocci.patch
+	$(RM) -r contrib/coccinelle/*.cocci.patches/
 
 clean: profile-clean coverage-clean cocciclean
 	$(RM) *.res
diff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore
index d3f29646dc..7ae6ffa983 100644
--- a/contrib/coccinelle/.gitignore
+++ b/contrib/coccinelle/.gitignore
@@ -1 +1,2 @@
-*.patch*
+*.cocci.patch
+*.cocci.patches/
-- 
2.18.0.408.g42635c01bc

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[René Scharfe]

Am 02.08.2018 um 13:55 schrieb SZEDER Gábor:
> Let's add a bit of Makefile metaprogramming to generate finer-grained
> make targets applying one semantic patch to only a single source file,
> and specify these as dependencies of the targets applying one semantic
> patch to all source files.  This way that shell loop can be avoided,
> semantic patches will only be applied to changed source files, and the
> same semantic patch can be applied in parallel to multiple source
> files.  The only remaining sequential part is aggregating the
> suggested transformations from the individual targets into a single
> patch file, which is comparatively cheap (especially since ideally
> there aren't any suggestions).
> 
> This change brings spectacular speedup in the scenario described in
> point (1) above.  When the results of a previous 'make coccicheck' are
> there, the time needed to run
> 
>    touch apply.c ; time make -j4 coccicheck
> 
> went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and
> 'apply.c' is the second longest source file in our codebase.  In the
> scenario in point (2), running
> 
>    touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck
> 
> went from 56.364s to 35.883s, which is ~36% speedup.

Awesome!

> Unfortunately, this new approach has some disadvantages compared to
> the current situation:
> 
>    - [RFC]
>      With this patch 'make coccicheck's output will look like this
>      ('make' apparently doesn't iterate over semantic patches in
>      lexicographical order):
> 
>        SPATCH commit.cocci              abspath.c
>        SPATCH commit.cocci              advice.c
>        <... lots of lines ...>
>        SPATCH array.cocci               http-walker.c
>        SPATCH array.cocci               remote-curl.c
> 
>      which means that the number of lines in the output grows from
>      "Nr-of-semantic-patches" to "Nr-of-semantic-patches *
>      Nr-of-source-files".

The output is mostly interesting to see if something is happening, to
see if some semantic patch generated a non-empty diff and to get
error details.  We have ca. 400 .c files and 8 .cocci files -- that
means the lines would fill up my scroll-back buffer.  Hmm.

Would it be possible to have a phony target (or directory) per source
file that just prints a line and depends on the individual file+cocci
targets?  (I don't know much make, you probably thought about it
already.)

Or to select one random .cocci file (let's say the first one) and
only print SPATCH lines for this one (without mentioning its name)?
(A dirty hack..)

> ---
>   Makefile                      | 52 ++++++++++++++++++++++++-----------
>   contrib/coccinelle/.gitignore |  3 +-
>   2 files changed, 38 insertions(+), 17 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index d616c04125..f516dd93d1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2674,25 +2674,44 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
>   else
>   COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
>   endif
> +COCCI_SEM_PATCHES = $(wildcard contrib/coccinelle/*.cocci)
>   
> -%.cocci.patch: %.cocci $(COCCI_SOURCES)
> -	@echo '    ' SPATCH $<; \
> -	ret=0; \
> -	for f in $(COCCI_SOURCES); do \
> -		$(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \
> -			{ ret=$$?; break; }; \
> -	done >$@+ 2>$@.log; \
> -	if test $$ret != 0; \
> +define cocci_template
> +$(cocci_sem_patch)_dirs := $$(addprefix $(cocci_sem_patch).patches/,$$(sort $$(dir $$(COCCI_SOURCES))))
> +
> +$$($(cocci_sem_patch)_dirs):
> +	@mkdir -p $$@
> +
> +# e.g. 'contrib/coccinelle/strbuf.cocci.patches/builtin/commit.c.patch'
> +# Applies one semantic patch to _one_ source file.
> +$(cocci_sem_patch).patches/%.patch: % $(cocci_sem_patch)
> +	@printf '     SPATCH %-25s %s\n' $$(notdir $(cocci_sem_patch)) $$<; \
> +	if ! $$(SPATCH) --sp-file $(cocci_sem_patch) $$< $$(SPATCH_FLAGS) >$$@ 2>$$@.log; \
>   	then \
> -		cat $@.log; \
> +		rm -f $$@; \
> +		cat $$@.log; \
>   		exit 1; \
> -	fi; \
> -	mv $@+ $@; \
> -	if test -s $@; \
> -	then \
> -		echo '    ' SPATCH result: $@; \
>   	fi
> -coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci))
> +
> +# e.g. 'contrib/coccinelle/strbuf.cocci.patch'
> +# Applies one semantic patch to _all_ source files.
> +$(cocci_sem_patch).patch: $(cocci_sem_patch) $$($(cocci_sem_patch)_dirs) $$(patsubst %,$(cocci_sem_patch).patches/%.patch,$(COCCI_SOURCES))

Are the dependencies correct (before and after the patch)?  E.g. are all
semantic patches reapplied after cache.h was changed?  I think we ignore
header files currently, and that becomes more of a problem if the check
becomes more fine-grained.

René

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Jeff King]

On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote:

> Let's add a bit of Makefile metaprogramming to generate finer-grained
> make targets applying one semantic patch to only a single source file,
> and specify these as dependencies of the targets applying one semantic
> patch to all source files.  This way that shell loop can be avoided,
> semantic patches will only be applied to changed source files, and the
> same semantic patch can be applied in parallel to multiple source
> files.  The only remaining sequential part is aggregating the
> suggested transformations from the individual targets into a single
> patch file, which is comparatively cheap (especially since ideally
> there aren't any suggestions).
> 
> This change brings spectacular speedup in the scenario described in
> point (1) above.  When the results of a previous 'make coccicheck' are
> there, the time needed to run
> 
>   touch apply.c ; time make -j4 coccicheck
> 
> went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and
> 'apply.c' is the second longest source file in our codebase.  In the
> scenario in point (2), running
> 
>   touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck
> 
> went from 56.364s to 35.883s, which is ~36% speedup.

I really like this direction. The slowness of coccicheck is one of the
things that has prevented me from running it more frequently. And I'm a
big fan of breaking steps down as much as possible into make targets. It
lets make do its job (avoiding repeated work and parallelizing).

> All the above timings are best-of-five on a machine with 2 physical
> and 2 logical cores.  I don't have the hardware to bring any numbers
> for point (3).  The time needed to run 'make -j4 coccicheck' in a
> clean state didn't change, it's ~3m42s both with and without this
> patch.

On a 40-core (20+20) machine, doing "make -j40 coccicheck" from scratch
went from:

  real	1m25.520s
  user	5m41.492s
  sys	0m26.776s

to:

  real	0m24.300s
  user	14m35.084s
  sys	0m50.108s

I was surprised by the jump in CPU times. Doing "make -j1 coccicheck"
with your patch results in:

  real	5m34.887s
  user	5m5.620s
  sys	0m19.908s

so it's really the parallelizing that seems to be to blame (possibly
because this CPU boosts from 2.3Ghz to 3.0Ghz, and we're only using 8
threads in the first example).

>   - [RFC]
>     With this patch 'make coccicheck's output will look like this
>     ('make' apparently doesn't iterate over semantic patches in
>     lexicographical order):
> 
>       SPATCH commit.cocci              abspath.c
>       SPATCH commit.cocci              advice.c
>       <... lots of lines ...>
>       SPATCH array.cocci               http-walker.c
>       SPATCH array.cocci               remote-curl.c
> 
>     which means that the number of lines in the output grows from
>     "Nr-of-semantic-patches" to "Nr-of-semantic-patches *
>     Nr-of-source-files".

IMHO this is not really that big a deal. We are doing useful work for
each line, so to me it's just more eye candy (and it's really satisfying
to watch it zip by on the 40-core machine ;) ).

What if we inverted the current loop? That is, right now we iterate over
the cocci patches at the Makefile level, and then for each target we
iterate over the giant list of source files. Instead, we could teach the
Makefile to iterate over the source files, with a target for each, and
then hit each cocci patch inside there.

That would give roughly the same output as a normal build. But moreover,
I wonder if we could make things faster by actually combining the cocci
files into a single set of rules to be applied to each source file. That
would mean invoking spatch 1/8 as much. It does give fewer opportunities
for "make" to reuse work, but that only matters when the cocci files
change (which is much less frequent than source files changing).

Doing:

  cat contrib/coccinelle/*.cocci >mega.cocci
  make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci

gives me:

  real	0m17.573s
  user	10m56.724s
  sys	0m11.548s

And that should show an improvement on more normal-sized systems, too,
because we really are eliminating some of the startup overhead.

The other obvious downside is that you don't get individual patches for
each class of transformation. Do we care? Certainly for a routine "make
coccicheck" I primarily want to know:

  - is there something that needs fixing?

  - give me the patch for all fixes

So I wonder if we'd want to have that be the default, and then perhaps
optionally give some targets to let people run single scripts (or not;
they could probably just run spatch themselves).

>   - [RFC]
>     The overhead of applying a semantic patch to all source files
>     became larger.  'make coccicheck' currently runs only one shell
>     process and creates two output files for each semantic patch.
>     With this patch it will run approx.  "Nr-of-semantic-patches *
>     Nr-of-source-files" shell processes and create twice as many
>     output files.

Doing the "one big .cocci" would help with this, too (and again puts us
in the same ballpark as a compile).

>   - [RFC]
>     This approach uses $(eval), which we haven't used in any of our
>     Makefiles yet.  I wonder whether it's portable enough.  And it's
>     ugly and complicated.

I looked into this a long time ago for another set of Makefile patches I
was considering. $(eval) was added to GNU make in 3.80, released in
2002. Which is probably fine by now.

If it isn't, one more exotic option would be to push the coccicheck
stuff into its own Makefile, and just run it via recursive make. Then
anybody doing a vanilla build can do so even with an antique make, but
you could only "make coccicheck" with something from the last 16 years
(but good luck getting ocaml running there).

I suspect if we go with the one-spatch-per-source route, though, that we
could do this just with regular make rules.

-Peff

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Jeff King]

On Thu, Aug 02, 2018 at 02:01:55PM -0400, Jeff King wrote:

> I suspect if we go with the one-spatch-per-source route, though, that we
> could do this just with regular make rules.

Yeah, it's pretty straightforward:

diff --git a/Makefile b/Makefile
index d616c0412..86fdcf567 100644
--- a/Makefile
+++ b/Makefile
@@ -2674,15 +2674,17 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
 else
 COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
 endif
+COCCI_COMBINED = contrib/coccinelle/combined.cocci
+COCCI_SEM_PATCHES = $(filter-out $(COCCI_COMBINED), $(wildcard contrib/coccinelle/*.cocci))
 
-%.cocci.patch: %.cocci $(COCCI_SOURCES)
+$(COCCI_COMBINED): $(COCCI_SEM_PATCHES)
+	cat $^ >$@+
+	mv $@+ $@
+
+$(COCCI_COMBINED).patches/%.patch: % $(COCCI_COMBINED)
 	@echo '    ' SPATCH $<; \
-	ret=0; \
-	for f in $(COCCI_SOURCES); do \
-		$(SPATCH) --sp-file $< $$f $(SPATCH_FLAGS) || \
-			{ ret=$$?; break; }; \
-	done >$@+ 2>$@.log; \
-	if test $$ret != 0; \
+	mkdir -p $(dir $@) || exit 1; \
+	if ! $(SPATCH) --sp-file $(COCCI_COMBINED) $< $(SPATCH_FLAGS) >$@+ 2>$@.log; \
 	then \
 		cat $@.log; \
 		exit 1; \
@@ -2692,7 +2694,8 @@ endif
 	then \
 		echo '    ' SPATCH result: $@; \
 	fi
-coccicheck: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.cocci))
+
+coccicheck: $(patsubst %, $(COCCI_COMBINED).patches/%.patch, $(COCCI_SOURCES))
 
 .PHONY: coccicheck
 
@@ -2907,7 +2910,7 @@ profile-clean:
 	$(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs)))
 
 cocciclean:
-	$(RM) contrib/coccinelle/*.cocci.patch*
+	$(RM) -r contrib/coccinelle/*.cocci.patches
 
 clean: profile-clean coverage-clean cocciclean
 	$(RM) *.res

I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in
most of the targets, and that would let people do individual:

  make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck

The default would just be the concatenated version.

-Peff

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Jonathan Nieder]

Hi,

Jeff King wrote:
> On Thu, Aug 02, 2018 at 02:01:55PM -0400, Jeff King wrote:

>> I suspect if we go with the one-spatch-per-source route, though, that we
>> could do this just with regular make rules.
>
> Yeah, it's pretty straightforward:
> 
> diff --git a/Makefile b/Makefile
> index d616c0412..86fdcf567 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2674,15 +2674,17 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
>  else
>  COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
>  endif
> +COCCI_COMBINED = contrib/coccinelle/combined.cocci

I like this approach.

[...]
> I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in
> most of the targets, and that would let people do individual:
> 
>   make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck

The issue here is that the dependencies for foo.cocci become
unreliable, so I'd rather have a separate target for that (e.g.
depending on FORCE) if we go that way.

Thanks,
Jonathan

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Jeff King]

On Thu, Aug 02, 2018 at 11:21:44PM -0700, Jonathan Nieder wrote:

> > diff --git a/Makefile b/Makefile
> > index d616c0412..86fdcf567 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -2674,15 +2674,17 @@ COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
> >  else
> >  COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
> >  endif
> > +COCCI_COMBINED = contrib/coccinelle/combined.cocci
> 
> I like this approach.

I was pretty pleased with myself, too, but I had a lingering doubt about
whether just cat-ing the files was legitimate. It sounds from the
response elsewhere that it's not (but just happens to work now for out
limited case). But it also sounds like there may be even better options.

> > I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in
> > most of the targets, and that would let people do individual:
> > 
> >   make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck
> 
> The issue here is that the dependencies for foo.cocci become
> unreliable, so I'd rather have a separate target for that (e.g.
> depending on FORCE) if we go that way.

Can you be more specific? I don't see how it's unreliable, unless you
mean that anything relying on "coccicheck" would depend on the exact
value of COCCI_PATCH.

But it may all be moot anyway, based no the responses elsewhere in the
thread.

-Peff

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Jonathan Nieder]

Hi,

Jeff King wrote:
> On Thu, Aug 02, 2018 at 11:21:44PM -0700, Jonathan Nieder wrote:
>> Jeff King wrote:

>>> I guess you could even replace "COCCI_COMBINED" with "COCCI_PATCH" in
>>> most of the targets, and that would let people do individual:
>>>
>>>   make COCCI_PATCH=contrib/coccinelle/foo.cocci coccicheck
>>
>> The issue here is that the dependencies for foo.cocci become
>> unreliable, so I'd rather have a separate target for that (e.g.
>> depending on FORCE) if we go that way.
>
> Can you be more specific? I don't see how it's unreliable, unless you
> mean that anything relying on "coccicheck" would depend on the exact
> value of COCCI_PATCH.

Yes, that's what I mean: changing which COCCI_PATCH to try would need
to trigger re-runs.

> But it may all be moot anyway, based no the responses elsewhere in the
> thread.

Sure.

Thanks,
Jonathan

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Eric Sunshine]

On Thu, Aug 2, 2018 at 2:02 PM Jeff King <peff@peff.net> wrote:
> On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote:
> >     This approach uses $(eval), which we haven't used in any of our
> >     Makefiles yet.  I wonder whether it's portable enough.  And it's
> >     ugly and complicated.
>
> I looked into this a long time ago for another set of Makefile patches I
> was considering. $(eval) was added to GNU make in 3.80, released in
> 2002. Which is probably fine by now.

For the record, MacOS developer tools are stuck at GNU make 3.81 (from 2006).

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Jeff King]

On Thu, Aug 02, 2018 at 03:46:08PM -0400, Eric Sunshine wrote:

> On Thu, Aug 2, 2018 at 2:02 PM Jeff King <peff@peff.net> wrote:
> > On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote:
> > >     This approach uses $(eval), which we haven't used in any of our
> > >     Makefiles yet.  I wonder whether it's portable enough.  And it's
> > >     ugly and complicated.
> >
> > I looked into this a long time ago for another set of Makefile patches I
> > was considering. $(eval) was added to GNU make in 3.80, released in
> > 2002. Which is probably fine by now.
> 
> For the record, MacOS developer tools are stuck at GNU make 3.81 (from 2006).

Thanks, that's interesting to know. Fortunately from my past research
that means that it should have both $(eval) and $(call).

If we follow my suggestions here, we don't need to care for now, but I
won't be surprised if it comes up again at some point.

-Peff

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Ævar Arnfjörð Bjarmason]

On Thu, Aug 02 2018, Jeff King wrote:

> On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote:
>
>> Let's add a bit of Makefile metaprogramming to generate finer-grained
>> make targets applying one semantic patch to only a single source file,
>> and specify these as dependencies of the targets applying one semantic
>> patch to all source files.  This way that shell loop can be avoided,
>> semantic patches will only be applied to changed source files, and the
>> same semantic patch can be applied in parallel to multiple source
>> files.  The only remaining sequential part is aggregating the
>> suggested transformations from the individual targets into a single
>> patch file, which is comparatively cheap (especially since ideally
>> there aren't any suggestions).
>>
>> This change brings spectacular speedup in the scenario described in
>> point (1) above.  When the results of a previous 'make coccicheck' are
>> there, the time needed to run
>>
>>   touch apply.c ; time make -j4 coccicheck
>>
>> went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and
>> 'apply.c' is the second longest source file in our codebase.  In the
>> scenario in point (2), running
>>
>>   touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck
>>
>> went from 56.364s to 35.883s, which is ~36% speedup.
>
> I really like this direction. The slowness of coccicheck is one of the
> things that has prevented me from running it more frequently. And I'm a
> big fan of breaking steps down as much as possible into make targets. It
> lets make do its job (avoiding repeated work and parallelizing).

Yeah, this is great. Also, CC-ing some of the recent contributors to
linux.git's coccinelle, perhaps they're interested / have comments.

>> All the above timings are best-of-five on a machine with 2 physical
>> and 2 logical cores.  I don't have the hardware to bring any numbers
>> for point (3).  The time needed to run 'make -j4 coccicheck' in a
>> clean state didn't change, it's ~3m42s both with and without this
>> patch.
>
> On a 40-core (20+20) machine, doing "make -j40 coccicheck" from scratch
> went from:
>
>   real	1m25.520s
>   user	5m41.492s
>   sys	0m26.776s
>
> to:
>
>   real	0m24.300s
>   user	14m35.084s
>   sys	0m50.108s
>
> I was surprised by the jump in CPU times. Doing "make -j1 coccicheck"
> with your patch results in:
>
>   real	5m34.887s
>   user	5m5.620s
>   sys	0m19.908s
>
> so it's really the parallelizing that seems to be to blame (possibly
> because this CPU boosts from 2.3Ghz to 3.0Ghz, and we're only using 8
> threads in the first example).
>
>>   - [RFC]
>>     With this patch 'make coccicheck's output will look like this
>>     ('make' apparently doesn't iterate over semantic patches in
>>     lexicographical order):
>>
>>       SPATCH commit.cocci              abspath.c
>>       SPATCH commit.cocci              advice.c
>>       <... lots of lines ...>
>>       SPATCH array.cocci               http-walker.c
>>       SPATCH array.cocci               remote-curl.c
>>
>>     which means that the number of lines in the output grows from
>>     "Nr-of-semantic-patches" to "Nr-of-semantic-patches *
>>     Nr-of-source-files".
>
> IMHO this is not really that big a deal. We are doing useful work for
> each line, so to me it's just more eye candy (and it's really satisfying
> to watch it zip by on the 40-core machine ;) ).

FWIW on the 8-cpu box I usually test on this went from 2m30s to 1m50s,
so not a huge improvement in time, but nice to have the per-file
progress.

> What if we inverted the current loop? That is, right now we iterate over
> the cocci patches at the Makefile level, and then for each target we
> iterate over the giant list of source files. Instead, we could teach the
> Makefile to iterate over the source files, with a target for each, and
> then hit each cocci patch inside there.
>
> That would give roughly the same output as a normal build. But moreover,
> I wonder if we could make things faster by actually combining the cocci
> files into a single set of rules to be applied to each source file. That
> would mean invoking spatch 1/8 as much. It does give fewer opportunities
> for "make" to reuse work, but that only matters when the cocci files
> change (which is much less frequent than source files changing).
>
> Doing:
>
>   cat contrib/coccinelle/*.cocci >mega.cocci
>   make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci
>
> gives me:
>
>   real	0m17.573s
>   user	10m56.724s
>   sys	0m11.548s
>
> And that should show an improvement on more normal-sized systems, too,
> because we really are eliminating some of the startup overhead.
>
> The other obvious downside is that you don't get individual patches for
> each class of transformation. Do we care? Certainly for a routine "make
> coccicheck" I primarily want to know:
>
>   - is there something that needs fixing?
>
>   - give me the patch for all fixes
>
> So I wonder if we'd want to have that be the default, and then perhaps
> optionally give some targets to let people run single scripts (or not;
> they could probably just run spatch themselves).
>
>>   - [RFC]
>>     The overhead of applying a semantic patch to all source files
>>     became larger.  'make coccicheck' currently runs only one shell
>>     process and creates two output files for each semantic patch.
>>     With this patch it will run approx.  "Nr-of-semantic-patches *
>>     Nr-of-source-files" shell processes and create twice as many
>>     output files.
>
> Doing the "one big .cocci" would help with this, too (and again puts us
> in the same ballpark as a compile).
>
>>   - [RFC]
>>     This approach uses $(eval), which we haven't used in any of our
>>     Makefiles yet.  I wonder whether it's portable enough.  And it's
>>     ugly and complicated.
>
> I looked into this a long time ago for another set of Makefile patches I
> was considering. $(eval) was added to GNU make in 3.80, released in
> 2002. Which is probably fine by now.
>
> If it isn't, one more exotic option would be to push the coccicheck
> stuff into its own Makefile, and just run it via recursive make. Then
> anybody doing a vanilla build can do so even with an antique make, but
> you could only "make coccicheck" with something from the last 16 years
> (but good luck getting ocaml running there).
>
> I suspect if we go with the one-spatch-per-source route, though, that we
> could do this just with regular make rules.
>
> -Peff

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 9773 bytes --]



On Thu, 2 Aug 2018, Ævar Arnfjörð Bjarmason wrote:

>
> On Thu, Aug 02 2018, Jeff King wrote:
>
> > On Thu, Aug 02, 2018 at 01:55:22PM +0200, SZEDER Gábor wrote:
> >
> >> Let's add a bit of Makefile metaprogramming to generate finer-grained
> >> make targets applying one semantic patch to only a single source file,
> >> and specify these as dependencies of the targets applying one semantic
> >> patch to all source files.  This way that shell loop can be avoided,
> >> semantic patches will only be applied to changed source files, and the
> >> same semantic patch can be applied in parallel to multiple source
> >> files.

This was already possible.  Make coccicheck is not supposed to be used
with -j, but rather with J=n.  That tells Coccinelle to parallelize the
treatment of the files internally.  In this case, the semantic patch is
only parsed once, and then n worker processes are forked to treat the
different files.

>  The only remaining sequential part is aggregating the
> >> suggested transformations from the individual targets into a single
> >> patch file, which is comparatively cheap (especially since ideally
> >> there aren't any suggestions).
> >>
> >> This change brings spectacular speedup in the scenario described in
> >> point (1) above.  When the results of a previous 'make coccicheck' are
> >> there, the time needed to run
> >>
> >>   touch apply.c ; time make -j4 coccicheck
> >>
> >> went from 3m42s to 1.848s, which is just over 99% speedup, yay!, and
> >> 'apply.c' is the second longest source file in our codebase.  In the
> >> scenario in point (2), running
> >>
> >>   touch contrib/coccinelle/array.cocci ; time make -j4 coccicheck
> >>
> >> went from 56.364s to 35.883s, which is ~36% speedup.
> >
> > I really like this direction. The slowness of coccicheck is one of the
> > things that has prevented me from running it more frequently. And I'm a
> > big fan of breaking steps down as much as possible into make targets. It
> > lets make do its job (avoiding repeated work and parallelizing).
>
> Yeah, this is great. Also, CC-ing some of the recent contributors to
> linux.git's coccinelle, perhaps they're interested / have comments.

I have extended the list of recipients with Nicolas Palix and the
Coccinelle mailing list.  In particular, Nicolas should comment on any
changes.

>
> >> All the above timings are best-of-five on a machine with 2 physical
> >> and 2 logical cores.  I don't have the hardware to bring any numbers
> >> for point (3).  The time needed to run 'make -j4 coccicheck' in a
> >> clean state didn't change, it's ~3m42s both with and without this
> >> patch.
> >
> > On a 40-core (20+20) machine, doing "make -j40 coccicheck" from scratch
> > went from:
> >
> >   real	1m25.520s
> >   user	5m41.492s
> >   sys	0m26.776s
> >
> > to:
> >
> >   real	0m24.300s
> >   user	14m35.084s
> >   sys	0m50.108s

By default, when the -j option is not used, make coccicheck causes
Coccinelle to run on all of the cores of the machine.  In my experience,
on a laptop (2 physical cores with hyperthreading), this is basically OK.
And on a server, it is a disaster.  On a machine with 80 physical cores
and hyperthreading, make coccicheck will run each instance of Coccinelle
such that it parallelizes on 160 cores.  But in reality, there is not much
difference between 20 and 40 cores, and after 40 cores the performance
starts to degrade.  So basically, using more than half of the physical
cores on each socket is a loss.


> >
> > I was surprised by the jump in CPU times. Doing "make -j1 coccicheck"
> > with your patch results in:
> >
> >   real	5m34.887s
> >   user	5m5.620s
> >   sys	0m19.908s
> >
> > so it's really the parallelizing that seems to be to blame (possibly
> > because this CPU boosts from 2.3Ghz to 3.0Ghz, and we're only using 8
> > threads in the first example).
> >
> >>   - [RFC]
> >>     With this patch 'make coccicheck's output will look like this
> >>     ('make' apparently doesn't iterate over semantic patches in
> >>     lexicographical order):
> >>
> >>       SPATCH commit.cocci              abspath.c
> >>       SPATCH commit.cocci              advice.c
> >>       <... lots of lines ...>
> >>       SPATCH array.cocci               http-walker.c
> >>       SPATCH array.cocci               remote-curl.c
> >>
> >>     which means that the number of lines in the output grows from
> >>     "Nr-of-semantic-patches" to "Nr-of-semantic-patches *
> >>     Nr-of-source-files".
> >
> > IMHO this is not really that big a deal. We are doing useful work for
> > each line, so to me it's just more eye candy (and it's really satisfying
> > to watch it zip by on the 40-core machine ;) ).
>
> FWIW on the 8-cpu box I usually test on this went from 2m30s to 1m50s,
> so not a huge improvement in time, but nice to have the per-file
> progress.

Coccinelle already does this when make coccicheck is used with J=n.  It
makes a subdirectory for the semantic patch that it is currently working
on, and this subdirectory contains stderr.n files that contain HANDLING
and the name of the file being treated.  It also prints information about
performance bottlenecks.  I think that make coccicheck turns off all this
reporting by default, but you can get it back with SPFLAGS="--quiet"

> > What if we inverted the current loop? That is, right now we iterate over
> > the cocci patches at the Makefile level, and then for each target we
> > iterate over the giant list of source files. Instead, we could teach the
> > Makefile to iterate over the source files, with a target for each, and
> > then hit each cocci patch inside there.
> >
> > That would give roughly the same output as a normal build. But moreover,
> > I wonder if we could make things faster by actually combining the cocci
> > files into a single set of rules to be applied to each source file. That
> > would mean invoking spatch 1/8 as much. It does give fewer opportunities
> > for "make" to reuse work, but that only matters when the cocci files
> > change (which is much less frequent than source files changing).
> >
> > Doing:
> >
> >   cat contrib/coccinelle/*.cocci >mega.cocci
> >   make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci

There was already a COCCI=foo.cocci argument to focus on a single semantic
patch.

I'm surprised that the above cat command would work.  Semantic patch rules
have names, and Coccinelle will not be happy isf two rules have the same
name.  Some may also have variables declared in initializers, although
perhaps the ones in the kernel don't do this.  Causing these variables to
be shared would not have a good effect.

Putting everything together can also go counter to the optimizations that
Coccinelle provides.  You can speed up spatch a lot on rules that mention
specific functions by running id-utils on your code base in advance (sh
coccinelle/scripts/idutils_index.sh).  The if you have one semantic patch
that only applies to files that contain foo and another that only applies
to files that contain bar, then each will only be applied to its
respective files.  If you don't have an id-utils index this optimization
will be done by Coccinelle by first scanning files for foo and bar, but
the index is obviously much faster.  If your semantic patch can be
relavant to files that contain foo or bar, then the rules from the foo
semantic patch (which could be very slow) will also be uselessly applied
to the bar files.  Whether this is relevant in practice depends on the
specific semantic patches of course.

> >
> > gives me:
> >
> >   real	0m17.573s
> >   user	10m56.724s
> >   sys	0m11.548s
> >
> > And that should show an improvement on more normal-sized systems, too,
> > because we really are eliminating some of the startup overhead.
> >
> > The other obvious downside is that you don't get individual patches for
> > each class of transformation. Do we care? Certainly for a routine "make
> > coccicheck" I primarily want to know:
> >
> >   - is there something that needs fixing?
> >
> >   - give me the patch for all fixes
> >
> > So I wonder if we'd want to have that be the default, and then perhaps
> > optionally give some targets to let people run single scripts (or not;
> > they could probably just run spatch themselves).
> >
> >>   - [RFC]
> >>     The overhead of applying a semantic patch to all source files
> >>     became larger.  'make coccicheck' currently runs only one shell
> >>     process

Not sure to understand this.  To my understanding it runs as many shell
processes as there are cores on the machine.

julia

> and creates two output files for each semantic patch.
> >>     With this patch it will run approx.  "Nr-of-semantic-patches *
> >>     Nr-of-source-files" shell processes and create twice as many
> >>     output files.
> >
> > Doing the "one big .cocci" would help with this, too (and again puts us
> > in the same ballpark as a compile).
> >
> >>   - [RFC]
> >>     This approach uses $(eval), which we haven't used in any of our
> >>     Makefiles yet.  I wonder whether it's portable enough.  And it's
> >>     ugly and complicated.
> >
> > I looked into this a long time ago for another set of Makefile patches I
> > was considering. $(eval) was added to GNU make in 3.80, released in
> > 2002. Which is probably fine by now.
> >
> > If it isn't, one more exotic option would be to push the coccicheck
> > stuff into its own Makefile, and just run it via recursive make. Then
> > anybody doing a vanilla build can do so even with an antique make, but
> > you could only "make coccicheck" with something from the last 16 years
> > (but good luck getting ocaml running there).
> >
> > I suspect if we go with the one-spatch-per-source route, though, that we
> > could do this just with regular make rules.
> >
> > -Peff
>

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Jonathan Nieder]

Hi,

Julia Lawall wrote:

> This was already possible.  Make coccicheck is not supposed to be used
> with -j, but rather with J=n.  That tells Coccinelle to parallelize the
> treatment of the files internally.  In this case, the semantic patch is
> only parsed once, and then n worker processes are forked to treat the
> different files.

Thanks for this hint.

I wonder if we can make this happen automatically under suitable
conditions.  We can parse -j<num> out of MAKEFLAGS and convert it into a
J=<num> argument, but that only solves half the problem: the "make"
parent process would think that the coccinelle run only deserves to
occupy one job slot.

Technically we could do all this using a wrapper that pretends to be a
submake <https://www.gnu.org/software/make/manual/html_node/Job-Slots.html>
(prefixing the command with '+', parsing jobserver options from the
MAKEFLAGS envvar) but it gets ugly.

It's likely that the best we can do is just to advertise J more
prominently.

[...]
>> On Thu, Aug 02 2018, Jeff King wrote:

>>>   cat contrib/coccinelle/*.cocci >mega.cocci
>>>   make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci
>
> There was already a COCCI=foo.cocci argument to focus on a single semantic
> patch.
>
> I'm surprised that the above cat command would work.  Semantic patch rules
> have names, and Coccinelle will not be happy isf two rules have the same
> name.

Yes, Git's semantic patches (in contrib/coccinelle) tend to be
relatively undemanding.

>       Some may also have variables declared in initializers, although
> perhaps the ones in the kernel don't do this.  Causing these variables to
> be shared would not have a good effect.

... oh!  You're thinking of the Linux kernel.

It looks like Linux's scripts/coccicheck has a lot we can crib from.
That's where the J envvar and automatic parallelism you mentioned are
implemented, too.

So it sounds to me like at a minimum we should use all of that. ;-)

Thanks again for the pointers.

Sincerely,
Jonathan

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Julia Lawall]

On Thu, 2 Aug 2018, Jonathan Nieder wrote:

> Hi,
>
> Julia Lawall wrote:
>
> > This was already possible.  Make coccicheck is not supposed to be used
> > with -j, but rather with J=n.  That tells Coccinelle to parallelize the
> > treatment of the files internally.  In this case, the semantic patch is
> > only parsed once, and then n worker processes are forked to treat the
> > different files.
>
> Thanks for this hint.
>
> I wonder if we can make this happen automatically under suitable
> conditions.  We can parse -j<num> out of MAKEFLAGS and convert it into a
> J=<num> argument,

It does happen automatically, in the sense that the default with no -j or
J=option is to use all the cores on the machine.

With -j, it seems to have to default to one core internally:

d7059ca0147adcd495f3c5b41f260e1ac55bb679

Also, if it didn't do that, it would end up with the product of the -j
option and the number of cores on the machine, which would give very bad
performance.

> but that only solves half the problem: the "make"
> parent process would think that the coccinelle run only deserves to
> occupy one job slot.

Yes, it seems to be intended to be run by itself.

> Technically we could do all this using a wrapper that pretends to be a
> submake <https://www.gnu.org/software/make/manual/html_node/Job-Slots.html>
> (prefixing the command with '+', parsing jobserver options from the
> MAKEFLAGS envvar) but it gets ugly.
>
> It's likely that the best we can do is just to advertise J more
> prominently.
>
> [...]
> >> On Thu, Aug 02 2018, Jeff King wrote:
>
> >>>   cat contrib/coccinelle/*.cocci >mega.cocci
> >>>   make -j40 coccicheck COCCI_SEM_PATCHES=mega.cocci
> >
> > There was already a COCCI=foo.cocci argument to focus on a single semantic
> > patch.
> >
> > I'm surprised that the above cat command would work.  Semantic patch rules
> > have names, and Coccinelle will not be happy isf two rules have the same
> > name.
>
> Yes, Git's semantic patches (in contrib/coccinelle) tend to be
> relatively undemanding.
>
> >       Some may also have variables declared in initializers, although
> > perhaps the ones in the kernel don't do this.  Causing these variables to
> > be shared would not have a good effect.
>
> ... oh!  You're thinking of the Linux kernel.
>
> It looks like Linux's scripts/coccicheck has a lot we can crib from.
> That's where the J envvar and automatic parallelism you mentioned are
> implemented, too.
>
> So it sounds to me like at a minimum we should use all of that. ;-)

Yes.

julia

Re: [PoC] coccinelle: make Coccinelle-related make targets more fine-grained

[Julia Lawall]

By the way, you can see my performance numbers here:

https://www.usenix.org/system/files/conference/atc18/atc18-lawall.pdf

Page 8, figures 4 and 5.

julia