[PATCH 0/2] Makefile: run coccicheck on all non-upstream sources

[PATCH 0/2] Makefile: run coccicheck on all non-upstream sources

[Denton Liu]

Before, when we ran coccicheck, it would only run on files that are
currently being compiled. However, this leaves us with a blindspot where
Windows-only sources are not checked since Coccinelle does not run on
Windows.

This patchset addresses this by making the "coccicheck" target run
against all source files in the repository, except for source files that
are pulled from some upstream source.

This patchset should serve as a continuation of the original discussion
about running coccicheck on all sources[1].

These patches depend on "ds/midx-expire-repack" and "dl/compat-cleanup".

[1]: https://public-inbox.org/git/nycvar.QRO.7.76.6.1905031127170.45@tvgsbejvaqbjf.bet/


Denton Liu (2):
  Makefile: define UPSTREAM_SOURCES
  Makefile: run coccicheck on more source files

 Makefile | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

-- 
2.23.0.248.g3a9dd8fb08

[PATCH 1/2] Makefile: define UPSTREAM_SOURCES

[Denton Liu]

After looking through the source files in compat/ and investigating the
files' content and/or its Git history, I've determined the list of files
that were copied from an upstream. Place the names of these files into
the UPSTREAM_SOURCES variable in the Makefile.

In addition, add the sha1collisiondetection/ and sha1dc/ sources as well
since they are also imported from upstream.

In a future commit, this variable will be used to determine which files
are excluded when running the "coccicheck" target.

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Makefile b/Makefile
index ad71ae1219..708df2c289 100644
--- a/Makefile
+++ b/Makefile
@@ -598,6 +598,7 @@ SCRIPT_SH =
 SCRIPT_LIB =
 TEST_BUILTINS_OBJS =
 TEST_PROGRAMS_NEED_X =
+UPSTREAM_SOURCES =
 
 # Having this variable in your environment would break pipelines because
 # you cause "cd" to echo its destination to stdout.  It can also take
@@ -1146,6 +1147,15 @@ BUILTIN_OBJS += builtin/verify-tag.o
 BUILTIN_OBJS += builtin/worktree.o
 BUILTIN_OBJS += builtin/write-tree.o
 
+UPSTREAM_SOURCES += compat/inet_ntop.c
+UPSTREAM_SOURCES += compat/inet_pton.c
+UPSTREAM_SOURCES += compat/obstack.%
+UPSTREAM_SOURCES += compat/nedmalloc/%
+UPSTREAM_SOURCES += compat/poll/%
+UPSTREAM_SOURCES += compat/regex/%
+UPSTREAM_SOURCES += sha1collisiondetection/%
+UPSTREAM_SOURCES += sha1dc/%
+
 GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)
 EXTLIBS =
 
-- 
2.23.0.248.g3a9dd8fb08

[PATCH 2/2] Makefile: run coccicheck on more source files

[Denton Liu]

Before, when running the "coccicheck" target, only the source files
which were being compiled would have been checked by Coccinelle.
However, just because we aren't compiling a source file doesn't mean we
have to exclude it from analysis. This will allow us to catch more
mistakes, in particular ones that affect Windows-only sources since
Coccinelle currently runs only on Linux.

Make the "coccicheck" target run on all C sources except for those that
are taken from some upstream. We don't want to patch these files since
we want them to be as close to upstream as possible so that it'll be
easier to pull in upstream updates.

This results in the following sources now being checked:

* block-sha1/sha1.c
* compat/access.c
* compat/basename.c
* compat/fileno.c
* compat/gmtime.c
* compat/hstrerror.c
* compat/memmem.c
* compat/mingw.c
* compat/mkdir.c
* compat/mkdtemp.c
* compat/mmap.c
* compat/msvc.c
* compat/pread.c
* compat/precompose_utf8.c
* compat/qsort.c
* compat/setenv.c
* compat/sha1-chunked.c
* compat/snprintf.c
* compat/stat.c
* compat/strcasestr.c
* compat/strdup.c
* compat/strtoimax.c
* compat/strtoumax.c
* compat/unsetenv.c
* compat/win32/dirent.c
* compat/win32/path-utils.c
* compat/win32/pthread.c
* compat/win32/syslog.c
* compat/win32/trace2_win32_process_info.c
* compat/win32mmap.c
* compat/winansi.c
* ppc/sha1.c

This also results in the following source now being excluded:

* compat/obstack.c

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/Makefile b/Makefile
index 708df2c289..d468b7c9c4 100644
--- a/Makefile
+++ b/Makefile
@@ -2802,12 +2802,8 @@ check: command-list.h
 		exit 1; \
 	fi
 
-C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
-ifdef DC_SHA1_SUBMODULE
-COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
-else
-COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
-endif
+FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
+COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES))
 
 %.cocci.patch: %.cocci $(COCCI_SOURCES)
 	@echo '    ' SPATCH $<; \
-- 
2.23.0.248.g3a9dd8fb08

Re: [PATCH 2/2] Makefile: run coccicheck on more source files

[SZEDER Gábor]

On Tue, Sep 10, 2019 at 12:44:31AM -0700, Denton Liu wrote:
> Before, when running the "coccicheck" target, only the source files
> which were being compiled would have been checked by Coccinelle.
> However, just because we aren't compiling a source file doesn't mean we
> have to exclude it from analysis. This will allow us to catch more
> mistakes, in particular ones that affect Windows-only sources since
> Coccinelle currently runs only on Linux.
> 
> Make the "coccicheck" target run on all C sources except for those that
> are taken from some upstream. We don't want to patch these files since
> we want them to be as close to upstream as possible so that it'll be
> easier to pull in upstream updates.
> 
> This results in the following sources now being checked:
> 
> * block-sha1/sha1.c
> * compat/access.c
> * compat/basename.c
> * compat/fileno.c
> * compat/gmtime.c
> * compat/hstrerror.c
> * compat/memmem.c
> * compat/mingw.c

When applying 'array.cocci' to 'compat/mingw.c' Coccinelle 1.0.4 (the
version shipped in Ubuntu 16.04, i.e. what is used in our static
analysis CI job) suggests to use the COPY_ARRAY macro instead of
memcpy():

diff -u -p a/compat/mingw.c b/compat/mingw.c
--- a/compat/mingw.c
+++ b/compat/mingw.c
@@ -1265,7 +1265,7 @@ static wchar_t *make_environment_block(c
 		}
 
 		ALLOC_ARRAY(result, size);
-		memcpy(result, wenv, size * sizeof(*wenv));
+		COPY_ARRAY(result, wenv, size);
 		FreeEnvironmentStringsW(wenv);
 		return result;
 	}
@@ -1309,7 +1309,7 @@ static wchar_t *make_environment_block(c
 			continue;
 
 		size = wcslen(array[i]) + 1;
-		memcpy(p, array[i], size * sizeof(*p));
+		COPY_ARRAY(p, array[i], size);
 		p += size;
 	}
 	*p = L'\0';

This patch series should apply these suggestions in preparatory
patches to make sure that our CI builds still pass.

> * compat/mkdir.c
> * compat/mkdtemp.c
> * compat/mmap.c
> * compat/msvc.c
> * compat/pread.c
> * compat/precompose_utf8.c
> * compat/qsort.c
> * compat/setenv.c
> * compat/sha1-chunked.c
> * compat/snprintf.c
> * compat/stat.c
> * compat/strcasestr.c
> * compat/strdup.c
> * compat/strtoimax.c
> * compat/strtoumax.c
> * compat/unsetenv.c
> * compat/win32/dirent.c
> * compat/win32/path-utils.c
> * compat/win32/pthread.c
> * compat/win32/syslog.c
> * compat/win32/trace2_win32_process_info.c
> * compat/win32mmap.c
> * compat/winansi.c
> * ppc/sha1.c
> 
> This also results in the following source now being excluded:
> 
> * compat/obstack.c

Why mention this file here, and, more importantly, why mention _only_
this file here?  According to the previous patch, there should be a
few more excluded files...

> Signed-off-by: Denton Liu <liu.denton@gmail.com>
> ---
>  Makefile | 8 ++------
>  1 file changed, 2 insertions(+), 6 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 708df2c289..d468b7c9c4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2802,12 +2802,8 @@ check: command-list.h
>  		exit 1; \
>  	fi
>  
> -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
> -ifdef DC_SHA1_SUBMODULE
> -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
> -else
> -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
> -endif
> +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> +COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES))
>  
>  %.cocci.patch: %.cocci $(COCCI_SOURCES)
>  	@echo '    ' SPATCH $<; \
> -- 
> 2.23.0.248.g3a9dd8fb08
> 

Re: [PATCH 2/2] Makefile: run coccicheck on more source files

[Denton Liu]

On Tue, Sep 10, 2019 at 03:28:13PM +0200, SZEDER Gábor wrote:
> On Tue, Sep 10, 2019 at 12:44:31AM -0700, Denton Liu wrote:
> > Before, when running the "coccicheck" target, only the source files
> > which were being compiled would have been checked by Coccinelle.
> > However, just because we aren't compiling a source file doesn't mean we
> > have to exclude it from analysis. This will allow us to catch more
> > mistakes, in particular ones that affect Windows-only sources since
> > Coccinelle currently runs only on Linux.
> > 
> > Make the "coccicheck" target run on all C sources except for those that
> > are taken from some upstream. We don't want to patch these files since
> > we want them to be as close to upstream as possible so that it'll be
> > easier to pull in upstream updates.
> > 
> > This results in the following sources now being checked:
> > 
> > * block-sha1/sha1.c
> > * compat/access.c
> > * compat/basename.c
> > * compat/fileno.c
> > * compat/gmtime.c
> > * compat/hstrerror.c
> > * compat/memmem.c
> > * compat/mingw.c
> 
> When applying 'array.cocci' to 'compat/mingw.c' Coccinelle 1.0.4 (the
> version shipped in Ubuntu 16.04, i.e. what is used in our static
> analysis CI job) suggests to use the COPY_ARRAY macro instead of
> memcpy():
> 
> diff -u -p a/compat/mingw.c b/compat/mingw.c
> --- a/compat/mingw.c
> +++ b/compat/mingw.c
> @@ -1265,7 +1265,7 @@ static wchar_t *make_environment_block(c
>  		}
>  
>  		ALLOC_ARRAY(result, size);
> -		memcpy(result, wenv, size * sizeof(*wenv));
> +		COPY_ARRAY(result, wenv, size);
>  		FreeEnvironmentStringsW(wenv);
>  		return result;
>  	}
> @@ -1309,7 +1309,7 @@ static wchar_t *make_environment_block(c
>  			continue;
>  
>  		size = wcslen(array[i]) + 1;
> -		memcpy(p, array[i], size * sizeof(*p));
> +		COPY_ARRAY(p, array[i], size);
>  		p += size;
>  	}
>  	*p = L'\0';
> 
> This patch series should apply these suggestions in preparatory
> patches to make sure that our CI builds still pass.

This patch series depends on the "dl/compat-cleanup" branch, which
should've already made these changes.

> 
> > * compat/mkdir.c
> > * compat/mkdtemp.c
> > * compat/mmap.c
> > * compat/msvc.c
> > * compat/pread.c
> > * compat/precompose_utf8.c
> > * compat/qsort.c
> > * compat/setenv.c
> > * compat/sha1-chunked.c
> > * compat/snprintf.c
> > * compat/stat.c
> > * compat/strcasestr.c
> > * compat/strdup.c
> > * compat/strtoimax.c
> > * compat/strtoumax.c
> > * compat/unsetenv.c
> > * compat/win32/dirent.c
> > * compat/win32/path-utils.c
> > * compat/win32/pthread.c
> > * compat/win32/syslog.c
> > * compat/win32/trace2_win32_process_info.c
> > * compat/win32mmap.c
> > * compat/winansi.c
> > * ppc/sha1.c
> > 
> > This also results in the following source now being excluded:
> > 
> > * compat/obstack.c
> 
> Why mention this file here, and, more importantly, why mention _only_
> this file here?  According to the previous patch, there should be a
> few more excluded files...

I guess this commit message is pretty weak since this only really
applies to my system. I'll mention in a reroll that this was for a
standard build with no additional flags on Arch Linux to be specific.

I got these lists by dumping the contents of $(COCCI_SOURCES) before and
after applying the patch and, with my "standard" build, it seems like
the only upstream file that was being checked before was
compat/obstack.c.

I don't have build options like, for example, NO_REGEX enabled so the
regex stuff wasn't being built in the first place but for users that
might want to run coccicheck on such a platform (do they even exist?),
this would benefit them.

> 
> > Signed-off-by: Denton Liu <liu.denton@gmail.com>
> > ---
> >  Makefile | 8 ++------
> >  1 file changed, 2 insertions(+), 6 deletions(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index 708df2c289..d468b7c9c4 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -2802,12 +2802,8 @@ check: command-list.h
> >  		exit 1; \
> >  	fi
> >  
> > -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
> > -ifdef DC_SHA1_SUBMODULE
> > -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
> > -else
> > -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
> > -endif
> > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> > +COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES))
> >  
> >  %.cocci.patch: %.cocci $(COCCI_SOURCES)
> >  	@echo '    ' SPATCH $<; \
> > -- 
> > 2.23.0.248.g3a9dd8fb08
> > 

Re: [PATCH 2/2] Makefile: run coccicheck on more source files

[SZEDER Gábor]

On Tue, Sep 10, 2019 at 12:44:31AM -0700, Denton Liu wrote:
> Make the "coccicheck" target run on all C sources except for those that
> are taken from some upstream. We don't want to patch these files since
> we want them to be as close to upstream as possible so that it'll be
> easier to pull in upstream updates.

> diff --git a/Makefile b/Makefile
> index 708df2c289..d468b7c9c4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2802,12 +2802,8 @@ check: command-list.h
>  		exit 1; \
>  	fi
>  
> -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
> -ifdef DC_SHA1_SUBMODULE
> -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
> -else
> -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
> -endif
> +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))

Hrm, so this uses FIND_SOURCE_FILES, which first attempts to run 'git
ls-files' and if that fails it falls back to run 'find'.

Unfortunately, the output of the two slightly differ: 'git ls-files'
prints 'abspath.c advice.c alias.c ...' why 'find' prints
'./upload-pack.c ./unpack-trees.c ./gpg-interface.c ...'.  Now, while
the order of files doesn't matter, the './' prefix does, because:

> +COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES))

Here the paths/patterns in UPSTREAM_SOURCES don't have that './'
prefix, and thus won't match and won't filter out any of the upstream
files that they are supposed to.  IOW, if someone runs 'make
coccicheck' on a system without Git installed, then Coccinelle will
check all upstream sources as well, and will e.g. suggest using
COPY_ARRAY in 'compat/regex/regexec.c'.

Now, running 'make coccicheck' on a git.git clone without Git
installed might look like quite a pathological case on the first
sight, but I would argue that it is not that pathological: e.g.
consider someone running the recent Coccinelle version in a small-ish
Docker image containing just enough to run 'make coccicheck', but not
Git.  (yeah, you guessed right, I am that someone ;)

I don't know how to convince 'find' to omit that './' prefix from each
listed file, and in a portable way at that.  Piping its output through
'sed' or even 'cut' easily takes care of it, though.

>  %.cocci.patch: %.cocci $(COCCI_SOURCES)
>  	@echo '    ' SPATCH $<; \
> -- 
> 2.23.0.248.g3a9dd8fb08
> 

[PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources

[Denton Liu]

Before, when we ran coccicheck, it would only run on files that are
currently being compiled. However, this leaves us with a blindspot where
Windows-only sources are not checked since Coccinelle does not run on
Windows.

This patchset addresses this by making the "coccicheck" target run
against all source files in the repository, except for source files that
are pulled from some upstream source.

This patchset should serve as a continuation of the original discussion
about running coccicheck on all sources[1].

These patches depend on "ds/midx-expire-repack" and "dl/compat-cleanup".

[1]: https://public-inbox.org/git/nycvar.QRO.7.76.6.1905031127170.45@tvgsbejvaqbjf.bet/


Denton Liu (3):
  Makefile: define THIRD_PARTY_SOURCES
  Makefile: strip leading ./ in $(FIND_SOURCE_FILES)
  Makefile: run coccicheck on more source files

 Makefile | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

Range-diff against v1:
1:  bc3cd4637a < -:  ---------- Makefile: define UPSTREAM_SOURCES
-:  ---------- > 1:  3878cb2598 Makefile: define THIRD_PARTY_SOURCES
-:  ---------- > 2:  239ffa690c Makefile: strip leading ./ in $(FIND_SOURCE_FILES)
2:  f62b0c7d17 ! 3:  89b7e17469 Makefile: run coccicheck on more source files
    @@ Commit message
         Coccinelle currently runs only on Linux.
     
         Make the "coccicheck" target run on all C sources except for those that
    -    are taken from some upstream. We don't want to patch these files since
    -    we want them to be as close to upstream as possible so that it'll be
    -    easier to pull in upstream updates.
    +    are taken from some third-party source. We don't want to patch these
    +    files since we want them to be as close to upstream as possible so that
    +    it'll be easier to pull in upstream updates.
     
    -    This results in the following sources now being checked:
    +    When running a build on Arch Linux with no additional flags provided,
    +    after applying this patch, the following sources are now checked:
     
         * block-sha1/sha1.c
         * compat/access.c
    @@ Makefile: check: command-list.h
     -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
     -endif
     +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
    -+COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES))
    ++COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
      
      %.cocci.patch: %.cocci $(COCCI_SOURCES)
      	@echo '    ' SPATCH $<; \
-- 
2.23.0.37.g745f681289

[PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES

[Denton Liu]

After looking through the source files in compat/ and investigating the
files' content and/or its Git history, I've determined the list of files
that were copied from a third-party source. Place the names of these
files into the THIRD_PARTY_SOURCES variable in the Makefile.

In addition, add the sha1collisiondetection/ and sha1dc/ sources as well
since they are also imported from a third-party source.

In a future commit, this variable will be used to determine which files
are excluded when running the "coccicheck" target.

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Makefile b/Makefile
index ad71ae1219..b88b42d7ed 100644
--- a/Makefile
+++ b/Makefile
@@ -598,6 +598,7 @@ SCRIPT_SH =
 SCRIPT_LIB =
 TEST_BUILTINS_OBJS =
 TEST_PROGRAMS_NEED_X =
+THIRD_PARTY_SOURCES =
 
 # Having this variable in your environment would break pipelines because
 # you cause "cd" to echo its destination to stdout.  It can also take
@@ -1146,6 +1147,15 @@ BUILTIN_OBJS += builtin/verify-tag.o
 BUILTIN_OBJS += builtin/worktree.o
 BUILTIN_OBJS += builtin/write-tree.o
 
+THIRD_PARTY_SOURCES += compat/inet_ntop.c
+THIRD_PARTY_SOURCES += compat/inet_pton.c
+THIRD_PARTY_SOURCES += compat/obstack.%
+THIRD_PARTY_SOURCES += compat/nedmalloc/%
+THIRD_PARTY_SOURCES += compat/poll/%
+THIRD_PARTY_SOURCES += compat/regex/%
+THIRD_PARTY_SOURCES += sha1collisiondetection/%
+THIRD_PARTY_SOURCES += sha1dc/%
+
 GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)
 EXTLIBS =
 
-- 
2.23.0.37.g745f681289

Re: [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES

[Junio C Hamano]

Denton Liu <liu.denton@gmail.com> writes:

> After looking through the source files in compat/ and investigating the
> files' content and/or its Git history, I've determined the list of files
> that were copied from a third-party source. Place the names of these
> files into the THIRD_PARTY_SOURCES variable in the Makefile.

Please make sure readers won't get fooled into thinking these are
filenames.  They are meant to be used in $(filter-out) as patterns
to match the names of third-party source files.

Also, the paragraph before that talks about "I did this", but
claiming credit is much less important than why we are doing this in
the first place, which you forgot to say (perhaps because you were
too deeply into the task and the motivation was too obvious to you).

So, if I were doing this patch, I'd justify the whole thing as such:

	Some files in our codebase are borrowed from other projects,
	and minimally updated to suit our own needs.  We'd sometimes
	need to tell our own sources and these third-party sources
	apart for management purposes (e.g. we may want to be less
	strict about coding style and other issues on third-party
	files).

	Define the $(MAKE) variable THIRD_PARTY_SOURCES that can be
	used to match names of third-party sources.


>  
> +THIRD_PARTY_SOURCES += compat/inet_ntop.c
> +THIRD_PARTY_SOURCES += compat/inet_pton.c
> +THIRD_PARTY_SOURCES += compat/obstack.%
> +THIRD_PARTY_SOURCES += compat/nedmalloc/%
> +THIRD_PARTY_SOURCES += compat/poll/%
> +THIRD_PARTY_SOURCES += compat/regex/%
> +THIRD_PARTY_SOURCES += sha1collisiondetection/%
> +THIRD_PARTY_SOURCES += sha1dc/%

Before this block, please explain that this is meant to be a list of
patterns suitable for $(filter-out) and friends in a comment.

Thanks.

[PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES)

[Denton Liu]

Currently, $(FIND_SOURCE_FILES) has two modes: if `git ls-files` is
present, it will use that to enumerate the files in the repository; else
it will use `$(FIND) .` to enumerate the files in the directory.

There is a subtle difference between these two methods, however. With
ls-files, filenames don't have a leading `./` while with $(FIND), they
do. This does not currently pose a problem but in a future patch, we
will be using `filter-out` to process the list of files with the
assumption that there is no prefix.

Unify the two possible invocations in $(FIND_SOURCE_FILES) by using sed
to remove the `./` prefix in the $(FIND) case.

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index b88b42d7ed..e2c693440b 100644
--- a/Makefile
+++ b/Makefile
@@ -2610,6 +2610,7 @@ FIND_SOURCE_FILES = ( \
 		-o \( -name 'trash*' -type d -prune \) \
 		-o \( -name '*.[hcS]' -type f -print \) \
 		-o \( -name '*.sh' -type f -print \) \
+		| sed -e 's|^\./||' \
 	)
 
 $(ETAGS_TARGET): FORCE
-- 
2.23.0.37.g745f681289

Re: [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES)

[Junio C Hamano]

Denton Liu <liu.denton@gmail.com> writes:

> Unify the two possible invocations in $(FIND_SOURCE_FILES) by using sed
> to remove the `./` prefix in the $(FIND) case.

> diff --git a/Makefile b/Makefile
> index b88b42d7ed..e2c693440b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2610,6 +2610,7 @@ FIND_SOURCE_FILES = ( \
>  		-o \( -name 'trash*' -type d -prune \) \
>  		-o \( -name '*.[hcS]' -type f -print \) \
>  		-o \( -name '*.sh' -type f -print \) \
> +		| sed -e 's|^\./||' \
>  	)
>  
>  $(ETAGS_TARGET): FORCE

An extra process and piping is a bit unfortunate, but this is purely
a fallback position where it does not matter too much to expend
one-shot overhead; it would only be needed until the user has a
working "git ls-files" and starts hacking on our codebase, as this
is used only by the xtags targets, which are only for developers.

Thanks.  Will queue.

Re: [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES)

[SZEDER Gábor]

On Thu, Sep 12, 2019 at 10:28:32AM -0700, Denton Liu wrote:
> Currently, $(FIND_SOURCE_FILES) has two modes: if `git ls-files` is
> present, it will use that to enumerate the files in the repository; else
> it will use `$(FIND) .` to enumerate the files in the directory.
> 
> There is a subtle difference between these two methods, however. With
> ls-files, filenames don't have a leading `./` while with $(FIND), they
> do. This does not currently pose a problem but in a future patch, we
> will be using `filter-out` to process the list of files with the
> assumption that there is no prefix.
> 
> Unify the two possible invocations in $(FIND_SOURCE_FILES) by using sed
> to remove the `./` prefix in the $(FIND) case.
> 
> Signed-off-by: Denton Liu <liu.denton@gmail.com>
> ---
>  Makefile | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/Makefile b/Makefile
> index b88b42d7ed..e2c693440b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2610,6 +2610,7 @@ FIND_SOURCE_FILES = ( \
>  		-o \( -name 'trash*' -type d -prune \) \
>  		-o \( -name '*.[hcS]' -type f -print \) \
>  		-o \( -name '*.sh' -type f -print \) \
> +		| sed -e 's|^\./||' \

Ramsay pointed out to me (in an off-list email) that the list of
library header files is generated in a similar way, and then those
leading './' have to be removed later.  See how the LIB_H and CHK_HDRS
variables are assigned.

I think it would be great to apply the same treatment there, too, to
eliminate those './' prefixes right at the source.

[PATCH v2 3/3] Makefile: run coccicheck on more source files

[Denton Liu]

Before, when running the "coccicheck" target, only the source files
which were being compiled would have been checked by Coccinelle.
However, just because we aren't compiling a source file doesn't mean we
have to exclude it from analysis. This will allow us to catch more
mistakes, in particular ones that affect Windows-only sources since
Coccinelle currently runs only on Linux.

Make the "coccicheck" target run on all C sources except for those that
are taken from some third-party source. We don't want to patch these
files since we want them to be as close to upstream as possible so that
it'll be easier to pull in upstream updates.

When running a build on Arch Linux with no additional flags provided,
after applying this patch, the following sources are now checked:

* block-sha1/sha1.c
* compat/access.c
* compat/basename.c
* compat/fileno.c
* compat/gmtime.c
* compat/hstrerror.c
* compat/memmem.c
* compat/mingw.c
* compat/mkdir.c
* compat/mkdtemp.c
* compat/mmap.c
* compat/msvc.c
* compat/pread.c
* compat/precompose_utf8.c
* compat/qsort.c
* compat/setenv.c
* compat/sha1-chunked.c
* compat/snprintf.c
* compat/stat.c
* compat/strcasestr.c
* compat/strdup.c
* compat/strtoimax.c
* compat/strtoumax.c
* compat/unsetenv.c
* compat/win32/dirent.c
* compat/win32/path-utils.c
* compat/win32/pthread.c
* compat/win32/syslog.c
* compat/win32/trace2_win32_process_info.c
* compat/win32mmap.c
* compat/winansi.c
* ppc/sha1.c

This also results in the following source now being excluded:

* compat/obstack.c

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/Makefile b/Makefile
index e2c693440b..7c88e0606f 100644
--- a/Makefile
+++ b/Makefile
@@ -2803,12 +2803,8 @@ check: command-list.h
 		exit 1; \
 	fi
 
-C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
-ifdef DC_SHA1_SUBMODULE
-COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
-else
-COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
-endif
+FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
 
 %.cocci.patch: %.cocci $(COCCI_SOURCES)
 	@echo '    ' SPATCH $<; \
-- 
2.23.0.37.g745f681289

Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files

[Junio C Hamano]

Denton Liu <liu.denton@gmail.com> writes:

> +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))

The former is somewhat misnamed.  FIND_SOURCE_FILES is *not* a list
of source files---it is a procedure to list source files to its
standard output.  FIND_C_SOUCRES sounds as if it is a similar
procedure, which would be implemented much like

	FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p'

but that is not what you did and that is not what you want to have.
Perhaps call it FOUND_C_SOURCES?

I wonder if we can get rid of FIND_SOURCE_FILES that is a mere
procedure and replace its use with a true list of source files.
Would it make the result more pleasant to work with?

Perhaps something like the attached patch, (which would come before
this entire thing as a clean-up, and removing the need for 2/3)?

I dunno.

Using a procedure whose output is fed to xargs has an advantage that
a platform with very short command line limit can still work with
many source files, but the way you create and use COCCI_SOURCES in
this patch would defeat that advantage anyway, so perhaps we can get
away with an approach like this.  Having a list of things in $(MAKE)
variable has a longer-term benefit that we could exploit more
parallelism if we wanted to, too.

 Makefile | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/Makefile b/Makefile
index f9255344ae..9dddd0e88c 100644
--- a/Makefile
+++ b/Makefile
@@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
 	$(QUIET_GEN)mkdir -p $(dir $@) && \
 	pod2man $< $@
 
-FIND_SOURCE_FILES = ( \
+SOURCE_FILES = $(patsubst ./%,%,$(shell \
 	git ls-files \
 		'*.[hcS]' \
 		'*.sh' \
@@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \
 		-o \( -name 'trash*' -type d -prune \) \
 		-o \( -name '*.[hcS]' -type f -print \) \
 		-o \( -name '*.sh' -type f -print \) \
-	)
+	))
 
 $(ETAGS_TARGET): FORCE
 	$(RM) $(ETAGS_TARGET)
-	$(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET)
+	etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES)
 
 tags: FORCE
 	$(RM) tags
-	$(FIND_SOURCE_FILES) | xargs ctags -a
+	ctags -a $(SOURCE_FILES)
 
 cscope:
 	$(RM) cscope*
-	$(FIND_SOURCE_FILES) | xargs cscope -b
+	cscope -b $(SOURCE_FILES)
 
 ### Detect prefix changes
 TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\

Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files

[SZEDER Gábor]

On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote:
> Denton Liu <liu.denton@gmail.com> writes:
> 
> > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
> 
> The former is somewhat misnamed.  FIND_SOURCE_FILES is *not* a list
> of source files---it is a procedure to list source files to its
> standard output.  FIND_C_SOUCRES sounds as if it is a similar
> procedure, which would be implemented much like
> 
> 	FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p'
> 
> but that is not what you did and that is not what you want to have.
> Perhaps call it FOUND_C_SOURCES?
> 
> I wonder if we can get rid of FIND_SOURCE_FILES that is a mere
> procedure and replace its use with a true list of source files.
> Would it make the result more pleasant to work with?
> 
> Perhaps something like the attached patch, (which would come before
> this entire thing as a clean-up, and removing the need for 2/3)?
> 
> I dunno.
> 
> Using a procedure whose output is fed to xargs has an advantage that
> a platform with very short command line limit can still work with
> many source files, but the way you create and use COCCI_SOURCES in
> this patch would defeat that advantage anyway,

COCCI_SOURCES is only used as an input to 'xargs', so that advantage
is not defeated.

> so perhaps we can get
> away with an approach like this.  Having a list of things in $(MAKE)
> variable has a longer-term benefit that we could exploit more
> parallelism if we wanted to, too.
> 
>  Makefile | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index f9255344ae..9dddd0e88c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
>  	$(QUIET_GEN)mkdir -p $(dir $@) && \
>  	pod2man $< $@
>  
> -FIND_SOURCE_FILES = ( \
> +SOURCE_FILES = $(patsubst ./%,%,$(shell \
>  	git ls-files \
>  		'*.[hcS]' \
>  		'*.sh' \
> @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \
>  		-o \( -name 'trash*' -type d -prune \) \
>  		-o \( -name '*.[hcS]' -type f -print \) \
>  		-o \( -name '*.sh' -type f -print \) \
> -	)
> +	))
>  
>  $(ETAGS_TARGET): FORCE
>  	$(RM) $(ETAGS_TARGET)
> -	$(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET)
> +	etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES)
>  
>  tags: FORCE
>  	$(RM) tags
> -	$(FIND_SOURCE_FILES) | xargs ctags -a
> +	ctags -a $(SOURCE_FILES)
>  
>  cscope:
>  	$(RM) cscope*
> -	$(FIND_SOURCE_FILES) | xargs cscope -b
> +	cscope -b $(SOURCE_FILES)
>  
>  ### Detect prefix changes
>  TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\
> 
> 
> 
> 

Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files

[Denton Liu]

On Fri, Sep 13, 2019 at 01:49:52PM +0200, SZEDER Gábor wrote:
> On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote:
> > Denton Liu <liu.denton@gmail.com> writes:
> > 
> > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
> > 
> > The former is somewhat misnamed.  FIND_SOURCE_FILES is *not* a list
> > of source files---it is a procedure to list source files to its
> > standard output.  FIND_C_SOUCRES sounds as if it is a similar
> > procedure, which would be implemented much like
> > 
> > 	FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p'
> > 
> > but that is not what you did and that is not what you want to have.
> > Perhaps call it FOUND_C_SOURCES?
> > 
> > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere
> > procedure and replace its use with a true list of source files.
> > Would it make the result more pleasant to work with?
> > 
> > Perhaps something like the attached patch, (which would come before
> > this entire thing as a clean-up, and removing the need for 2/3)?
> > 
> > I dunno.
> > 
> > Using a procedure whose output is fed to xargs has an advantage that
> > a platform with very short command line limit can still work with
> > many source files, but the way you create and use COCCI_SOURCES in
> > this patch would defeat that advantage anyway,
> 
> COCCI_SOURCES is only used as an input to 'xargs', so that advantage
> is not defeated.

I think it still does matter; the relevant snippet is as follows:

	if ! echo $(COCCI_SOURCES) | xargs $$limit \
		$(SPATCH) --sp-file $< $(SPATCH_FLAGS) \
		>$@+ 2>$@.log; \

which means that a really big COCCI_SOURCES could exceed the limit.

That being said, COCCI_SOURCES should be smaller than the future
SOURCE_FILES variable since we're only taking %.c files (and filtering
out some of them too!).

I dunno, either. I'm mostly in favour of this change since it makes a
lot of sense to keep lists in make variables if possible as opposed to
command invocations. I guess worst case, if someone complains in the
future, we can always change it back.

> 
> > so perhaps we can get
> > away with an approach like this.  Having a list of things in $(MAKE)
> > variable has a longer-term benefit that we could exploit more
> > parallelism if we wanted to, too.
> > 
> >  Makefile | 10 +++++-----
> >  1 file changed, 5 insertions(+), 5 deletions(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index f9255344ae..9dddd0e88c 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
> >  	$(QUIET_GEN)mkdir -p $(dir $@) && \
> >  	pod2man $< $@
> >  
> > -FIND_SOURCE_FILES = ( \
> > +SOURCE_FILES = $(patsubst ./%,%,$(shell \
> >  	git ls-files \
> >  		'*.[hcS]' \
> >  		'*.sh' \
> > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \
> >  		-o \( -name 'trash*' -type d -prune \) \
> >  		-o \( -name '*.[hcS]' -type f -print \) \
> >  		-o \( -name '*.sh' -type f -print \) \
> > -	)
> > +	))
> >  
> >  $(ETAGS_TARGET): FORCE
> >  	$(RM) $(ETAGS_TARGET)
> > -	$(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET)
> > +	etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES)
> >  
> >  tags: FORCE
> >  	$(RM) tags
> > -	$(FIND_SOURCE_FILES) | xargs ctags -a
> > +	ctags -a $(SOURCE_FILES)
> >  
> >  cscope:
> >  	$(RM) cscope*
> > -	$(FIND_SOURCE_FILES) | xargs cscope -b
> > +	cscope -b $(SOURCE_FILES)
> >  
> >  ### Detect prefix changes
> >  TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\
> > 
> > 
> > 
> > 

Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files

[SZEDER Gábor]

On Fri, Sep 13, 2019 at 10:14:01AM -0700, Denton Liu wrote:
> On Fri, Sep 13, 2019 at 01:49:52PM +0200, SZEDER Gábor wrote:
> > On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote:
> > > Denton Liu <liu.denton@gmail.com> writes:
> > > 
> > > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> > > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
> > > 
> > > The former is somewhat misnamed.  FIND_SOURCE_FILES is *not* a list
> > > of source files---it is a procedure to list source files to its
> > > standard output.  FIND_C_SOUCRES sounds as if it is a similar
> > > procedure, which would be implemented much like
> > > 
> > > 	FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p'
> > > 
> > > but that is not what you did and that is not what you want to have.
> > > Perhaps call it FOUND_C_SOURCES?
> > > 
> > > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere
> > > procedure and replace its use with a true list of source files.
> > > Would it make the result more pleasant to work with?
> > > 
> > > Perhaps something like the attached patch, (which would come before
> > > this entire thing as a clean-up, and removing the need for 2/3)?
> > > 
> > > I dunno.
> > > 
> > > Using a procedure whose output is fed to xargs has an advantage that
> > > a platform with very short command line limit can still work with
> > > many source files, but the way you create and use COCCI_SOURCES in
> > > this patch would defeat that advantage anyway,
> > 
> > COCCI_SOURCES is only used as an input to 'xargs', so that advantage
> > is not defeated.
> 
> I think it still does matter; the relevant snippet is as follows:
> 
> 	if ! echo $(COCCI_SOURCES) | xargs $$limit \
> 		$(SPATCH) --sp-file $< $(SPATCH_FLAGS) \
> 		>$@+ 2>$@.log; \
> 
> which means that a really big COCCI_SOURCES could exceed the limit.

Oh, you're both right.

> That being said, COCCI_SOURCES should be smaller than the future
> SOURCE_FILES variable since we're only taking %.c files (and filtering
> out some of them too!).

We could also argue that Coccinelle only runs on platforms that have a
reasonably large command line arg limit, and the number of our source
files is way below that, so it won't matter in the foreseeable future.

(Furthermore, 'echo' is often a shell builtin command, and I don't
think that the platform's argument size limit applies to them.  At
least the 'echo' of dash, Bash, ksh, ksh93, mksh, and BusyBox sh can
deal with at least 10 million arguments; the platform limit is
somewhere around 147k)

> > > diff --git a/Makefile b/Makefile
> > > index f9255344ae..9dddd0e88c 100644
> > > --- a/Makefile
> > > +++ b/Makefile
> > > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
> > >  	$(QUIET_GEN)mkdir -p $(dir $@) && \
> > >  	pod2man $< $@
> > >  
> > > -FIND_SOURCE_FILES = ( \
> > > +SOURCE_FILES = $(patsubst ./%,%,$(shell \
> > >  	git ls-files \
> > >  		'*.[hcS]' \
> > >  		'*.sh' \
> > > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \
> > >  		-o \( -name 'trash*' -type d -prune \) \
> > >  		-o \( -name '*.[hcS]' -type f -print \) \
> > >  		-o \( -name '*.sh' -type f -print \) \
> > > -	)
> > > +	))
> > >  
> > >  $(ETAGS_TARGET): FORCE
> > >  	$(RM) $(ETAGS_TARGET)
> > > -	$(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET)
> > > +	etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES)
> > >  
> > >  tags: FORCE
> > >  	$(RM) tags
> > > -	$(FIND_SOURCE_FILES) | xargs ctags -a
> > > +	ctags -a $(SOURCE_FILES)
> > >  
> > >  cscope:
> > >  	$(RM) cscope*
> > > -	$(FIND_SOURCE_FILES) | xargs cscope -b
> > > +	cscope -b $(SOURCE_FILES)

Here, however, the list of source files is passed as argument to
non-builtin commands, that also might be used on
cmdline-arg-limit-challenged platforms.

Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files

[Denton Liu]

On Fri, Sep 13, 2019 at 08:00:14PM +0200, SZEDER Gábor wrote:
> On Fri, Sep 13, 2019 at 10:14:01AM -0700, Denton Liu wrote:
> > On Fri, Sep 13, 2019 at 01:49:52PM +0200, SZEDER Gábor wrote:
> > > On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote:
> > > > Denton Liu <liu.denton@gmail.com> writes:
> > > > 
> > > > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> > > > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
> > > > 
> > > > The former is somewhat misnamed.  FIND_SOURCE_FILES is *not* a list
> > > > of source files---it is a procedure to list source files to its
> > > > standard output.  FIND_C_SOUCRES sounds as if it is a similar
> > > > procedure, which would be implemented much like
> > > > 
> > > > 	FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p'
> > > > 
> > > > but that is not what you did and that is not what you want to have.
> > > > Perhaps call it FOUND_C_SOURCES?
> > > > 
> > > > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere
> > > > procedure and replace its use with a true list of source files.
> > > > Would it make the result more pleasant to work with?
> > > > 
> > > > Perhaps something like the attached patch, (which would come before
> > > > this entire thing as a clean-up, and removing the need for 2/3)?
> > > > 
> > > > I dunno.
> > > > 
> > > > Using a procedure whose output is fed to xargs has an advantage that
> > > > a platform with very short command line limit can still work with
> > > > many source files, but the way you create and use COCCI_SOURCES in
> > > > this patch would defeat that advantage anyway,
> > > 
> > > COCCI_SOURCES is only used as an input to 'xargs', so that advantage
> > > is not defeated.
> > 
> > I think it still does matter; the relevant snippet is as follows:
> > 
> > 	if ! echo $(COCCI_SOURCES) | xargs $$limit \
> > 		$(SPATCH) --sp-file $< $(SPATCH_FLAGS) \
> > 		>$@+ 2>$@.log; \
> > 
> > which means that a really big COCCI_SOURCES could exceed the limit.
> 
> Oh, you're both right.
> 
> > That being said, COCCI_SOURCES should be smaller than the future
> > SOURCE_FILES variable since we're only taking %.c files (and filtering
> > out some of them too!).
> 
> We could also argue that Coccinelle only runs on platforms that have a
> reasonably large command line arg limit, and the number of our source
> files is way below that, so it won't matter in the foreseeable future.

Good point.

> 
> (Furthermore, 'echo' is often a shell builtin command, and I don't
> think that the platform's argument size limit applies to them.  At
> least the 'echo' of dash, Bash, ksh, ksh93, mksh, and BusyBox sh can
> deal with at least 10 million arguments; the platform limit is
> somewhere around 147k)
> 
> > > > diff --git a/Makefile b/Makefile
> > > > index f9255344ae..9dddd0e88c 100644
> > > > --- a/Makefile
> > > > +++ b/Makefile
> > > > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
> > > >  	$(QUIET_GEN)mkdir -p $(dir $@) && \
> > > >  	pod2man $< $@
> > > >  
> > > > -FIND_SOURCE_FILES = ( \
> > > > +SOURCE_FILES = $(patsubst ./%,%,$(shell \
> > > >  	git ls-files \
> > > >  		'*.[hcS]' \
> > > >  		'*.sh' \
> > > > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \
> > > >  		-o \( -name 'trash*' -type d -prune \) \
> > > >  		-o \( -name '*.[hcS]' -type f -print \) \
> > > >  		-o \( -name '*.sh' -type f -print \) \
> > > > -	)
> > > > +	))
> > > >  
> > > >  $(ETAGS_TARGET): FORCE
> > > >  	$(RM) $(ETAGS_TARGET)
> > > > -	$(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET)
> > > > +	etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES)
> > > >  
> > > >  tags: FORCE
> > > >  	$(RM) tags
> > > > -	$(FIND_SOURCE_FILES) | xargs ctags -a
> > > > +	ctags -a $(SOURCE_FILES)
> > > >  
> > > >  cscope:
> > > >  	$(RM) cscope*
> > > > -	$(FIND_SOURCE_FILES) | xargs cscope -b
> > > > +	cscope -b $(SOURCE_FILES)
> 
> Here, however, the list of source files is passed as argument to
> non-builtin commands, that also might be used on
> cmdline-arg-limit-challenged platforms.
> 

After doing a bit of research, I think that I agree with you. It seems
like the max command-line length for CMD on Windows is 8191 characters.

However, after running the following,

	$ git ls-files '*.[hcS]' '*.sh' ':!*[tp][0-9][0-9][0-9][0-9]*' ':!contrib' | wc -c
	   12779

we can see that the command-line length would definitely exceed the max
length so xargs would be required. As a result, we should probably just
keep the existing xargs invocations.

Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

>> Using a procedure whose output is fed to xargs has an advantage that
>> a platform with very short command line limit can still work with
>> many source files, but the way you create and use COCCI_SOURCES in
>> this patch would defeat that advantage anyway,
>
> COCCI_SOURCES is only used as an input to 'xargs', so that advantage
> is not defeated.

It is passed as a command line argument to "echo", that pipes to
xargs; I would not say it is taking advantage of "xargs" to lift the
command line length limit, as it first needs to convince the shell
to feed all of them to the "echo" that is upstream of "xargs".

As you mentioned elsewhere, LIB_H already uses the same approach as
I outlined in the message you are responding to (i.e. "don't define
a procedure to produce lines to the standard output in a $(MAKE)
variable--instead make the variable to hold the list itself"), so I
suspect that we are almost on the same page?

[PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources

[Denton Liu]

Hi all, I spent the weekend brooding over this and I opted to not
convert $(FIND_SOURCE_FILES) into a list of filenames. I put the
justification for it in 4/4 so, hopefully, it'll make sense for someone
who might want to do the same change in the future.


Before, when we ran coccicheck, it would only run on files that are
currently being compiled. However, this leaves us with a blindspot where
Windows-only sources are not checked since Coccinelle does not run on
Windows.

This patchset addresses this by making the "coccicheck" target run
against all source files in the repository, except for source files that
are pulled from some upstream source.

This patchset should serve as a continuation of the original discussion
about running coccicheck on all sources[1].

These patches depend on "ds/midx-expire-repack" and "dl/compat-cleanup".

[1]: https://public-inbox.org/git/nycvar.QRO.7.76.6.1905031127170.45@tvgsbejvaqbjf.bet/


Denton Liu (4):
  Makefile: strip leading ./ in $(LIB_H)
  Makefile: define THIRD_PARTY_SOURCES
  Makefile: strip leading ./ in $(FIND_SOURCE_FILES)
  Makefile: run coccicheck on more source files

 Makefile | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

Range-diff against v2:
-:  ---------- > 1:  e7fa5e1df7 Makefile: strip leading ./ in $(LIB_H)
1:  72b4cc68f5 ! 2:  a7c855a973 Makefile: define THIRD_PARTY_SOURCES
    @@ Metadata
      ## Commit message ##
         Makefile: define THIRD_PARTY_SOURCES
     
    -    After looking through the source files in compat/ and investigating the
    -    files' content and/or its Git history, I've determined the list of files
    -    that were copied from a third-party source. Place the names of these
    -    files into the THIRD_PARTY_SOURCES variable in the Makefile.
    +    Some files in our codebase are borrowed from other projects, and
    +    minimally updated to suit our own needs. We'd sometimes need to tell
    +    our own sources and these third-party sources apart for management
    +    purposes (e.g. we may want to be less strict about coding style and
    +    other issues on third-party files).
     
    -    In addition, add the sha1collisiondetection/ and sha1dc/ sources as well
    -    since they are also imported from a third-party source.
    -
    -    In a future commit, this variable will be used to determine which files
    -    are excluded when running the "coccicheck" target.
    +    Define the $(MAKE) variable THIRD_PARTY_SOURCES that can be used to
    +    match names of third-party sources.
     
         Signed-off-by: Denton Liu <liu.denton@gmail.com>
    -    Signed-off-by: Junio C Hamano <gitster@pobox.com>
     
      ## Makefile ##
     @@ Makefile: SCRIPT_SH =
    @@ Makefile: BUILTIN_OBJS += builtin/verify-tag.o
      BUILTIN_OBJS += builtin/worktree.o
      BUILTIN_OBJS += builtin/write-tree.o
      
    ++# THIRD_PARTY_SOURCES is a list of patterns compatible with
    ++# the $(filter) and $(filter-out) family of functions
     +THIRD_PARTY_SOURCES += compat/inet_ntop.c
     +THIRD_PARTY_SOURCES += compat/inet_pton.c
     +THIRD_PARTY_SOURCES += compat/obstack.%
2:  a51acfec66 ! 3:  71861cbdf8 Makefile: strip leading ./ in $(FIND_SOURCE_FILES)
    @@ Commit message
         to remove the `./` prefix in the $(FIND) case.
     
         Signed-off-by: Denton Liu <liu.denton@gmail.com>
    -    Signed-off-by: Junio C Hamano <gitster@pobox.com>
     
      ## Makefile ##
     @@ Makefile: FIND_SOURCE_FILES = ( \
3:  75c36bc9a1 ! 4:  0ecfcd8858 Makefile: run coccicheck on more source files
    @@ Commit message
     
         * compat/obstack.c
     
    +    Instead of generating $(FOUND_C_SOURCES) from a
    +    `$(shell $(FIND_SOURCE_FILES))` invocation, an alternative design was
    +    considered which involved converting $(FIND_SOURCE_FILES) into
    +    $(SOURCE_FILES) which would hold a list of filenames from the
    +    $(FIND_SOURCE_FILES) invocation. We would simply filter `%.c` files into
    +    $(ALL_C_SOURCES). $(SOURCE_FILES) would then be passed directly to the
    +    etags, ctags and cscope commands. We can see from the following
    +    invocation
    +
    +            $ git ls-files '*.[hcS]' '*.sh' ':!*[tp][0-9][0-9][0-9][0-9]*' ':!contrib' | wc -c
    +               12779
    +
    +    that the number of characters in this list would pose a problem on
    +    platforms with short command-line length limits (such as CMD which has a
    +    max of 8191 characters). As a result, we don't perform this change.
    +
    +    However, we can see that the same issue may apply when running
    +    Coccinelle since $(COCCI_SOURCES) is also a list of filenames:
    +
    +            if ! echo $(COCCI_SOURCES) | xargs $$limit \
    +                    $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \
    +                    >$@+ 2>$@.log; \
    +
    +    This is justified since platforms that support Coccinelle generally have
    +    reasonably long command-line length limits and so we are safe for the
    +    foreseeable future.
    +
         Signed-off-by: Denton Liu <liu.denton@gmail.com>
    -    Signed-off-by: Junio C Hamano <gitster@pobox.com>
     
      ## Makefile ##
     @@ Makefile: check: command-list.h
    @@ Makefile: check: command-list.h
     -else
     -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
     -endif
    -+FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
    -+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
    ++FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
    ++COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
      
      %.cocci.patch: %.cocci $(COCCI_SOURCES)
      	@echo '    ' SPATCH $<; \
-- 
2.23.0

[PATCH v3 1/4] Makefile: strip leading ./ in $(LIB_H)

[Denton Liu]

Currently, $(LIB_H) is generated from two modes: if `git ls-files` is
present, it will use that to enumerate the files in the repository; else
it will use `$(FIND) .` to enumerate the files in the directory.

There is a subtle difference between these two methods, however. With
ls-files, filenames don't have a leading `./` while with $(FIND), they
do. This results in $(CHK_HDRS) having to substitute out the leading
`./` before it uses $(LIB_H).

Unify the two possible values in $(LIB_H) by using patsubst to remove the
`./` prefix at its definition.

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index ad71ae1219..ea77198247 100644
--- a/Makefile
+++ b/Makefile
@@ -818,12 +818,12 @@ VCSSVN_LIB = vcs-svn/lib.a
 
 GENERATED_H += command-list.h
 
-LIB_H := $(sort $(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \
+LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \
 	$(FIND) . \
 	-name .git -prune -o \
 	-name t -prune -o \
 	-name Documentation -prune -o \
-	-name '*.h' -print))
+	-name '*.h' -print)))
 
 LIB_OBJS += abspath.o
 LIB_OBJS += advice.o
@@ -2769,7 +2769,7 @@ EXCEPT_HDRS := $(GEN_HDRS) compat/% xdiff/%
 ifndef GCRYPT_SHA256
 	EXCEPT_HDRS += sha256/gcrypt.h
 endif
-CHK_HDRS = $(filter-out $(EXCEPT_HDRS),$(patsubst ./%,%,$(LIB_H)))
+CHK_HDRS = $(filter-out $(EXCEPT_HDRS),$(LIB_H))
 HCO = $(patsubst %.h,%.hco,$(CHK_HDRS))
 
 $(HCO): %.hco: %.h FORCE
-- 
2.23.0

[PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES

[Denton Liu]

Some files in our codebase are borrowed from other projects, and
minimally updated to suit our own needs. We'd sometimes need to tell
our own sources and these third-party sources apart for management
purposes (e.g. we may want to be less strict about coding style and
other issues on third-party files).

Define the $(MAKE) variable THIRD_PARTY_SOURCES that can be used to
match names of third-party sources.

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/Makefile b/Makefile
index ea77198247..49839579ec 100644
--- a/Makefile
+++ b/Makefile
@@ -598,6 +598,7 @@ SCRIPT_SH =
 SCRIPT_LIB =
 TEST_BUILTINS_OBJS =
 TEST_PROGRAMS_NEED_X =
+THIRD_PARTY_SOURCES =
 
 # Having this variable in your environment would break pipelines because
 # you cause "cd" to echo its destination to stdout.  It can also take
@@ -1146,6 +1147,17 @@ BUILTIN_OBJS += builtin/verify-tag.o
 BUILTIN_OBJS += builtin/worktree.o
 BUILTIN_OBJS += builtin/write-tree.o
 
+# THIRD_PARTY_SOURCES is a list of patterns compatible with
+# the $(filter) and $(filter-out) family of functions
+THIRD_PARTY_SOURCES += compat/inet_ntop.c
+THIRD_PARTY_SOURCES += compat/inet_pton.c
+THIRD_PARTY_SOURCES += compat/obstack.%
+THIRD_PARTY_SOURCES += compat/nedmalloc/%
+THIRD_PARTY_SOURCES += compat/poll/%
+THIRD_PARTY_SOURCES += compat/regex/%
+THIRD_PARTY_SOURCES += sha1collisiondetection/%
+THIRD_PARTY_SOURCES += sha1dc/%
+
 GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)
 EXTLIBS =
 
-- 
2.23.0

Re: [PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES

[Junio C Hamano]

Denton Liu <liu.denton@gmail.com> writes:

> +# THIRD_PARTY_SOURCES is a list of patterns compatible with
> +# the $(filter) and $(filter-out) family of functions

That defines the format, but does it convey what they want to
achieve to the readers?  "... to catch the list of source files
we borrowed from elsewhere" or something, perhaps?

> +THIRD_PARTY_SOURCES += compat/inet_ntop.c
> +THIRD_PARTY_SOURCES += compat/inet_pton.c
> +THIRD_PARTY_SOURCES += compat/obstack.%
> +THIRD_PARTY_SOURCES += compat/nedmalloc/%
> +THIRD_PARTY_SOURCES += compat/poll/%
> +THIRD_PARTY_SOURCES += compat/regex/%
> +THIRD_PARTY_SOURCES += sha1collisiondetection/%
> +THIRD_PARTY_SOURCES += sha1dc/%
> +
>  GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)
>  EXTLIBS =

[PATCH] fixup! Makefile: define THIRD_PARTY_SOURCES

[Denton Liu]

* expand the comment to show what it's used for

* alphabetical ordering fix

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 522342a46c..617c25d16b 100644
--- a/Makefile
+++ b/Makefile
@@ -1147,12 +1147,15 @@ BUILTIN_OBJS += builtin/verify-tag.o
 BUILTIN_OBJS += builtin/worktree.o
 BUILTIN_OBJS += builtin/write-tree.o
 
-# THIRD_PARTY_SOURCES is a list of patterns compatible with
-# the $(filter) and $(filter-out) family of functions
+# THIRD_PARTY_SOURCES is a list of patterns compatible with the
+# $(filter) and $(filter-out) family of functions. They specify source
+# files which are taken from some third-party source where we want to be
+# less strict about issues such as coding style so we don't diverge from
+# upstream unnecessarily (making merging in future changes easier).
 THIRD_PARTY_SOURCES += compat/inet_ntop.c
 THIRD_PARTY_SOURCES += compat/inet_pton.c
-THIRD_PARTY_SOURCES += compat/obstack.%
 THIRD_PARTY_SOURCES += compat/nedmalloc/%
+THIRD_PARTY_SOURCES += compat/obstack.%
 THIRD_PARTY_SOURCES += compat/poll/%
 THIRD_PARTY_SOURCES += compat/regex/%
 THIRD_PARTY_SOURCES += sha1collisiondetection/%
-- 
2.23.0

[PATCH v3 3/4] Makefile: strip leading ./ in $(FIND_SOURCE_FILES)

[Denton Liu]

Currently, $(FIND_SOURCE_FILES) has two modes: if `git ls-files` is
present, it will use that to enumerate the files in the repository; else
it will use `$(FIND) .` to enumerate the files in the directory.

There is a subtle difference between these two methods, however. With
ls-files, filenames don't have a leading `./` while with $(FIND), they
do. This does not currently pose a problem but in a future patch, we
will be using `filter-out` to process the list of files with the
assumption that there is no prefix.

Unify the two possible invocations in $(FIND_SOURCE_FILES) by using sed
to remove the `./` prefix in the $(FIND) case.

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 49839579ec..20eb5c5c42 100644
--- a/Makefile
+++ b/Makefile
@@ -2612,6 +2612,7 @@ FIND_SOURCE_FILES = ( \
 		-o \( -name 'trash*' -type d -prune \) \
 		-o \( -name '*.[hcS]' -type f -print \) \
 		-o \( -name '*.sh' -type f -print \) \
+		| sed -e 's|^\./||' \
 	)
 
 $(ETAGS_TARGET): FORCE
-- 
2.23.0

[PATCH v3 4/4] Makefile: run coccicheck on more source files

[Denton Liu]

Before, when running the "coccicheck" target, only the source files
which were being compiled would have been checked by Coccinelle.
However, just because we aren't compiling a source file doesn't mean we
have to exclude it from analysis. This will allow us to catch more
mistakes, in particular ones that affect Windows-only sources since
Coccinelle currently runs only on Linux.

Make the "coccicheck" target run on all C sources except for those that
are taken from some third-party source. We don't want to patch these
files since we want them to be as close to upstream as possible so that
it'll be easier to pull in upstream updates.

When running a build on Arch Linux with no additional flags provided,
after applying this patch, the following sources are now checked:

* block-sha1/sha1.c
* compat/access.c
* compat/basename.c
* compat/fileno.c
* compat/gmtime.c
* compat/hstrerror.c
* compat/memmem.c
* compat/mingw.c
* compat/mkdir.c
* compat/mkdtemp.c
* compat/mmap.c
* compat/msvc.c
* compat/pread.c
* compat/precompose_utf8.c
* compat/qsort.c
* compat/setenv.c
* compat/sha1-chunked.c
* compat/snprintf.c
* compat/stat.c
* compat/strcasestr.c
* compat/strdup.c
* compat/strtoimax.c
* compat/strtoumax.c
* compat/unsetenv.c
* compat/win32/dirent.c
* compat/win32/path-utils.c
* compat/win32/pthread.c
* compat/win32/syslog.c
* compat/win32/trace2_win32_process_info.c
* compat/win32mmap.c
* compat/winansi.c
* ppc/sha1.c

This also results in the following source now being excluded:

* compat/obstack.c

Instead of generating $(FOUND_C_SOURCES) from a
`$(shell $(FIND_SOURCE_FILES))` invocation, an alternative design was
considered which involved converting $(FIND_SOURCE_FILES) into
$(SOURCE_FILES) which would hold a list of filenames from the
$(FIND_SOURCE_FILES) invocation. We would simply filter `%.c` files into
$(ALL_C_SOURCES). $(SOURCE_FILES) would then be passed directly to the
etags, ctags and cscope commands. We can see from the following
invocation

	$ git ls-files '*.[hcS]' '*.sh' ':!*[tp][0-9][0-9][0-9][0-9]*' ':!contrib' | wc -c
	   12779

that the number of characters in this list would pose a problem on
platforms with short command-line length limits (such as CMD which has a
max of 8191 characters). As a result, we don't perform this change.

However, we can see that the same issue may apply when running
Coccinelle since $(COCCI_SOURCES) is also a list of filenames:

	if ! echo $(COCCI_SOURCES) | xargs $$limit \
		$(SPATCH) --sp-file $< $(SPATCH_FLAGS) \
		>$@+ 2>$@.log; \

This is justified since platforms that support Coccinelle generally have
reasonably long command-line length limits and so we are safe for the
foreseeable future.

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 Makefile | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/Makefile b/Makefile
index 20eb5c5c42..522342a46c 100644
--- a/Makefile
+++ b/Makefile
@@ -2805,12 +2805,8 @@ check: command-list.h
 		exit 1; \
 	fi
 
-C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ))
-ifdef DC_SHA1_SUBMODULE
-COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES))
-else
-COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES))
-endif
+FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
 
 %.cocci.patch: %.cocci $(COCCI_SOURCES)
 	@echo '    ' SPATCH $<; \
-- 
2.23.0

Re: [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources

[Junio C Hamano]

Queued; thanks.

Re: [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources

[SZEDER Gábor]

On Mon, Sep 16, 2019 at 01:57:14PM -0700, Junio C Hamano wrote:
> Queued; thanks.

Nit: it would be nicer to queue this series on top of
'dl/compat-cleanup', because 'make coccicheck' errors out suggesting
transformations in 'compat/mingw.c' that have only been applied in
that series.

Re: [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> On Mon, Sep 16, 2019 at 01:57:14PM -0700, Junio C Hamano wrote:
>> Queued; thanks.
>
> Nit: it would be nicer to queue this series on top of
> 'dl/compat-cleanup', because 'make coccicheck' errors out suggesting
> transformations in 'compat/mingw.c' that have only been applied in
> that series.

Yikes, yes I remember that you pointed it out already, yet I
forgot.  Will correct.

Thanks.