* [PATCH 0/2] Makefile: run coccicheck on all non-upstream sources @ 2019-09-10 7:44 Denton Liu 2019-09-10 7:44 ` [PATCH 1/2] Makefile: define UPSTREAM_SOURCES Denton Liu ` (2 more replies) 0 siblings, 3 replies; 29+ messages in thread From: Denton Liu @ 2019-09-10 7:44 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Before, when we ran coccicheck, it would only run on files that are currently being compiled. However, this leaves us with a blindspot where Windows-only sources are not checked since Coccinelle does not run on Windows. This patchset addresses this by making the "coccicheck" target run against all source files in the repository, except for source files that are pulled from some upstream source. This patchset should serve as a continuation of the original discussion about running coccicheck on all sources[1]. These patches depend on "ds/midx-expire-repack" and "dl/compat-cleanup". [1]: https://public-inbox.org/git/nycvar.QRO.7.76.6.1905031127170.45@tvgsbejvaqbjf.bet/ Denton Liu (2): Makefile: define UPSTREAM_SOURCES Makefile: run coccicheck on more source files Makefile | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) -- 2.23.0.248.g3a9dd8fb08 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH 1/2] Makefile: define UPSTREAM_SOURCES 2019-09-10 7:44 [PATCH 0/2] Makefile: run coccicheck on all non-upstream sources Denton Liu @ 2019-09-10 7:44 ` Denton Liu 2019-09-10 7:44 ` [PATCH 2/2] Makefile: run coccicheck on more source files Denton Liu 2019-09-12 17:28 ` [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources Denton Liu 2 siblings, 0 replies; 29+ messages in thread From: Denton Liu @ 2019-09-10 7:44 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano After looking through the source files in compat/ and investigating the files' content and/or its Git history, I've determined the list of files that were copied from an upstream. Place the names of these files into the UPSTREAM_SOURCES variable in the Makefile. In addition, add the sha1collisiondetection/ and sha1dc/ sources as well since they are also imported from upstream. In a future commit, this variable will be used to determine which files are excluded when running the "coccicheck" target. Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/Makefile b/Makefile index ad71ae1219..708df2c289 100644 --- a/Makefile +++ b/Makefile @@ -598,6 +598,7 @@ SCRIPT_SH = SCRIPT_LIB = TEST_BUILTINS_OBJS = TEST_PROGRAMS_NEED_X = +UPSTREAM_SOURCES = # Having this variable in your environment would break pipelines because # you cause "cd" to echo its destination to stdout. It can also take @@ -1146,6 +1147,15 @@ BUILTIN_OBJS += builtin/verify-tag.o BUILTIN_OBJS += builtin/worktree.o BUILTIN_OBJS += builtin/write-tree.o +UPSTREAM_SOURCES += compat/inet_ntop.c +UPSTREAM_SOURCES += compat/inet_pton.c +UPSTREAM_SOURCES += compat/obstack.% +UPSTREAM_SOURCES += compat/nedmalloc/% +UPSTREAM_SOURCES += compat/poll/% +UPSTREAM_SOURCES += compat/regex/% +UPSTREAM_SOURCES += sha1collisiondetection/% +UPSTREAM_SOURCES += sha1dc/% + GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) EXTLIBS = -- 2.23.0.248.g3a9dd8fb08 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH 2/2] Makefile: run coccicheck on more source files 2019-09-10 7:44 [PATCH 0/2] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-10 7:44 ` [PATCH 1/2] Makefile: define UPSTREAM_SOURCES Denton Liu @ 2019-09-10 7:44 ` Denton Liu 2019-09-10 13:28 ` SZEDER Gábor 2019-09-10 14:18 ` SZEDER Gábor 2019-09-12 17:28 ` [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources Denton Liu 2 siblings, 2 replies; 29+ messages in thread From: Denton Liu @ 2019-09-10 7:44 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Before, when running the "coccicheck" target, only the source files which were being compiled would have been checked by Coccinelle. However, just because we aren't compiling a source file doesn't mean we have to exclude it from analysis. This will allow us to catch more mistakes, in particular ones that affect Windows-only sources since Coccinelle currently runs only on Linux. Make the "coccicheck" target run on all C sources except for those that are taken from some upstream. We don't want to patch these files since we want them to be as close to upstream as possible so that it'll be easier to pull in upstream updates. This results in the following sources now being checked: * block-sha1/sha1.c * compat/access.c * compat/basename.c * compat/fileno.c * compat/gmtime.c * compat/hstrerror.c * compat/memmem.c * compat/mingw.c * compat/mkdir.c * compat/mkdtemp.c * compat/mmap.c * compat/msvc.c * compat/pread.c * compat/precompose_utf8.c * compat/qsort.c * compat/setenv.c * compat/sha1-chunked.c * compat/snprintf.c * compat/stat.c * compat/strcasestr.c * compat/strdup.c * compat/strtoimax.c * compat/strtoumax.c * compat/unsetenv.c * compat/win32/dirent.c * compat/win32/path-utils.c * compat/win32/pthread.c * compat/win32/syslog.c * compat/win32/trace2_win32_process_info.c * compat/win32mmap.c * compat/winansi.c * ppc/sha1.c This also results in the following source now being excluded: * compat/obstack.c Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 708df2c289..d468b7c9c4 100644 --- a/Makefile +++ b/Makefile @@ -2802,12 +2802,8 @@ check: command-list.h exit 1; \ fi -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) -ifdef DC_SHA1_SUBMODULE -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) -else -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) -endif +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) +COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) @echo ' ' SPATCH $<; \ -- 2.23.0.248.g3a9dd8fb08 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH 2/2] Makefile: run coccicheck on more source files 2019-09-10 7:44 ` [PATCH 2/2] Makefile: run coccicheck on more source files Denton Liu @ 2019-09-10 13:28 ` SZEDER Gábor 2019-09-10 16:07 ` Denton Liu 2019-09-10 14:18 ` SZEDER Gábor 1 sibling, 1 reply; 29+ messages in thread From: SZEDER Gábor @ 2019-09-10 13:28 UTC (permalink / raw) To: Denton Liu Cc: Git Mailing List, Johannes Schindelin, Jeff King, Junio C Hamano On Tue, Sep 10, 2019 at 12:44:31AM -0700, Denton Liu wrote: > Before, when running the "coccicheck" target, only the source files > which were being compiled would have been checked by Coccinelle. > However, just because we aren't compiling a source file doesn't mean we > have to exclude it from analysis. This will allow us to catch more > mistakes, in particular ones that affect Windows-only sources since > Coccinelle currently runs only on Linux. > > Make the "coccicheck" target run on all C sources except for those that > are taken from some upstream. We don't want to patch these files since > we want them to be as close to upstream as possible so that it'll be > easier to pull in upstream updates. > > This results in the following sources now being checked: > > * block-sha1/sha1.c > * compat/access.c > * compat/basename.c > * compat/fileno.c > * compat/gmtime.c > * compat/hstrerror.c > * compat/memmem.c > * compat/mingw.c When applying 'array.cocci' to 'compat/mingw.c' Coccinelle 1.0.4 (the version shipped in Ubuntu 16.04, i.e. what is used in our static analysis CI job) suggests to use the COPY_ARRAY macro instead of memcpy(): diff -u -p a/compat/mingw.c b/compat/mingw.c --- a/compat/mingw.c +++ b/compat/mingw.c @@ -1265,7 +1265,7 @@ static wchar_t *make_environment_block(c } ALLOC_ARRAY(result, size); - memcpy(result, wenv, size * sizeof(*wenv)); + COPY_ARRAY(result, wenv, size); FreeEnvironmentStringsW(wenv); return result; } @@ -1309,7 +1309,7 @@ static wchar_t *make_environment_block(c continue; size = wcslen(array[i]) + 1; - memcpy(p, array[i], size * sizeof(*p)); + COPY_ARRAY(p, array[i], size); p += size; } *p = L'\0'; This patch series should apply these suggestions in preparatory patches to make sure that our CI builds still pass. > * compat/mkdir.c > * compat/mkdtemp.c > * compat/mmap.c > * compat/msvc.c > * compat/pread.c > * compat/precompose_utf8.c > * compat/qsort.c > * compat/setenv.c > * compat/sha1-chunked.c > * compat/snprintf.c > * compat/stat.c > * compat/strcasestr.c > * compat/strdup.c > * compat/strtoimax.c > * compat/strtoumax.c > * compat/unsetenv.c > * compat/win32/dirent.c > * compat/win32/path-utils.c > * compat/win32/pthread.c > * compat/win32/syslog.c > * compat/win32/trace2_win32_process_info.c > * compat/win32mmap.c > * compat/winansi.c > * ppc/sha1.c > > This also results in the following source now being excluded: > > * compat/obstack.c Why mention this file here, and, more importantly, why mention _only_ this file here? According to the previous patch, there should be a few more excluded files... > Signed-off-by: Denton Liu <liu.denton@gmail.com> > --- > Makefile | 8 ++------ > 1 file changed, 2 insertions(+), 6 deletions(-) > > diff --git a/Makefile b/Makefile > index 708df2c289..d468b7c9c4 100644 > --- a/Makefile > +++ b/Makefile > @@ -2802,12 +2802,8 @@ check: command-list.h > exit 1; \ > fi > > -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) > -ifdef DC_SHA1_SUBMODULE > -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > -else > -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > -endif > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) > +COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES)) > > %.cocci.patch: %.cocci $(COCCI_SOURCES) > @echo ' ' SPATCH $<; \ > -- > 2.23.0.248.g3a9dd8fb08 > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 2/2] Makefile: run coccicheck on more source files 2019-09-10 13:28 ` SZEDER Gábor @ 2019-09-10 16:07 ` Denton Liu 0 siblings, 0 replies; 29+ messages in thread From: Denton Liu @ 2019-09-10 16:07 UTC (permalink / raw) To: SZEDER Gábor Cc: Git Mailing List, Johannes Schindelin, Jeff King, Junio C Hamano On Tue, Sep 10, 2019 at 03:28:13PM +0200, SZEDER Gábor wrote: > On Tue, Sep 10, 2019 at 12:44:31AM -0700, Denton Liu wrote: > > Before, when running the "coccicheck" target, only the source files > > which were being compiled would have been checked by Coccinelle. > > However, just because we aren't compiling a source file doesn't mean we > > have to exclude it from analysis. This will allow us to catch more > > mistakes, in particular ones that affect Windows-only sources since > > Coccinelle currently runs only on Linux. > > > > Make the "coccicheck" target run on all C sources except for those that > > are taken from some upstream. We don't want to patch these files since > > we want them to be as close to upstream as possible so that it'll be > > easier to pull in upstream updates. > > > > This results in the following sources now being checked: > > > > * block-sha1/sha1.c > > * compat/access.c > > * compat/basename.c > > * compat/fileno.c > > * compat/gmtime.c > > * compat/hstrerror.c > > * compat/memmem.c > > * compat/mingw.c > > When applying 'array.cocci' to 'compat/mingw.c' Coccinelle 1.0.4 (the > version shipped in Ubuntu 16.04, i.e. what is used in our static > analysis CI job) suggests to use the COPY_ARRAY macro instead of > memcpy(): > > diff -u -p a/compat/mingw.c b/compat/mingw.c > --- a/compat/mingw.c > +++ b/compat/mingw.c > @@ -1265,7 +1265,7 @@ static wchar_t *make_environment_block(c > } > > ALLOC_ARRAY(result, size); > - memcpy(result, wenv, size * sizeof(*wenv)); > + COPY_ARRAY(result, wenv, size); > FreeEnvironmentStringsW(wenv); > return result; > } > @@ -1309,7 +1309,7 @@ static wchar_t *make_environment_block(c > continue; > > size = wcslen(array[i]) + 1; > - memcpy(p, array[i], size * sizeof(*p)); > + COPY_ARRAY(p, array[i], size); > p += size; > } > *p = L'\0'; > > This patch series should apply these suggestions in preparatory > patches to make sure that our CI builds still pass. This patch series depends on the "dl/compat-cleanup" branch, which should've already made these changes. > > > * compat/mkdir.c > > * compat/mkdtemp.c > > * compat/mmap.c > > * compat/msvc.c > > * compat/pread.c > > * compat/precompose_utf8.c > > * compat/qsort.c > > * compat/setenv.c > > * compat/sha1-chunked.c > > * compat/snprintf.c > > * compat/stat.c > > * compat/strcasestr.c > > * compat/strdup.c > > * compat/strtoimax.c > > * compat/strtoumax.c > > * compat/unsetenv.c > > * compat/win32/dirent.c > > * compat/win32/path-utils.c > > * compat/win32/pthread.c > > * compat/win32/syslog.c > > * compat/win32/trace2_win32_process_info.c > > * compat/win32mmap.c > > * compat/winansi.c > > * ppc/sha1.c > > > > This also results in the following source now being excluded: > > > > * compat/obstack.c > > Why mention this file here, and, more importantly, why mention _only_ > this file here? According to the previous patch, there should be a > few more excluded files... I guess this commit message is pretty weak since this only really applies to my system. I'll mention in a reroll that this was for a standard build with no additional flags on Arch Linux to be specific. I got these lists by dumping the contents of $(COCCI_SOURCES) before and after applying the patch and, with my "standard" build, it seems like the only upstream file that was being checked before was compat/obstack.c. I don't have build options like, for example, NO_REGEX enabled so the regex stuff wasn't being built in the first place but for users that might want to run coccicheck on such a platform (do they even exist?), this would benefit them. > > > Signed-off-by: Denton Liu <liu.denton@gmail.com> > > --- > > Makefile | 8 ++------ > > 1 file changed, 2 insertions(+), 6 deletions(-) > > > > diff --git a/Makefile b/Makefile > > index 708df2c289..d468b7c9c4 100644 > > --- a/Makefile > > +++ b/Makefile > > @@ -2802,12 +2802,8 @@ check: command-list.h > > exit 1; \ > > fi > > > > -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) > > -ifdef DC_SHA1_SUBMODULE > > -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > > -else > > -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > > -endif > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) > > +COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES)) > > > > %.cocci.patch: %.cocci $(COCCI_SOURCES) > > @echo ' ' SPATCH $<; \ > > -- > > 2.23.0.248.g3a9dd8fb08 > > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH 2/2] Makefile: run coccicheck on more source files 2019-09-10 7:44 ` [PATCH 2/2] Makefile: run coccicheck on more source files Denton Liu 2019-09-10 13:28 ` SZEDER Gábor @ 2019-09-10 14:18 ` SZEDER Gábor 1 sibling, 0 replies; 29+ messages in thread From: SZEDER Gábor @ 2019-09-10 14:18 UTC (permalink / raw) To: Denton Liu Cc: Git Mailing List, Johannes Schindelin, Jeff King, Junio C Hamano On Tue, Sep 10, 2019 at 12:44:31AM -0700, Denton Liu wrote: > Make the "coccicheck" target run on all C sources except for those that > are taken from some upstream. We don't want to patch these files since > we want them to be as close to upstream as possible so that it'll be > easier to pull in upstream updates. > diff --git a/Makefile b/Makefile > index 708df2c289..d468b7c9c4 100644 > --- a/Makefile > +++ b/Makefile > @@ -2802,12 +2802,8 @@ check: command-list.h > exit 1; \ > fi > > -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) > -ifdef DC_SHA1_SUBMODULE > -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) > -else > -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) > -endif > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) Hrm, so this uses FIND_SOURCE_FILES, which first attempts to run 'git ls-files' and if that fails it falls back to run 'find'. Unfortunately, the output of the two slightly differ: 'git ls-files' prints 'abspath.c advice.c alias.c ...' why 'find' prints './upload-pack.c ./unpack-trees.c ./gpg-interface.c ...'. Now, while the order of files doesn't matter, the './' prefix does, because: > +COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES)) Here the paths/patterns in UPSTREAM_SOURCES don't have that './' prefix, and thus won't match and won't filter out any of the upstream files that they are supposed to. IOW, if someone runs 'make coccicheck' on a system without Git installed, then Coccinelle will check all upstream sources as well, and will e.g. suggest using COPY_ARRAY in 'compat/regex/regexec.c'. Now, running 'make coccicheck' on a git.git clone without Git installed might look like quite a pathological case on the first sight, but I would argue that it is not that pathological: e.g. consider someone running the recent Coccinelle version in a small-ish Docker image containing just enough to run 'make coccicheck', but not Git. (yeah, you guessed right, I am that someone ;) I don't know how to convince 'find' to omit that './' prefix from each listed file, and in a portable way at that. Piping its output through 'sed' or even 'cut' easily takes care of it, though. > %.cocci.patch: %.cocci $(COCCI_SOURCES) > @echo ' ' SPATCH $<; \ > -- > 2.23.0.248.g3a9dd8fb08 > ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources 2019-09-10 7:44 [PATCH 0/2] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-10 7:44 ` [PATCH 1/2] Makefile: define UPSTREAM_SOURCES Denton Liu 2019-09-10 7:44 ` [PATCH 2/2] Makefile: run coccicheck on more source files Denton Liu @ 2019-09-12 17:28 ` Denton Liu 2019-09-12 17:28 ` [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES Denton Liu ` (3 more replies) 2 siblings, 4 replies; 29+ messages in thread From: Denton Liu @ 2019-09-12 17:28 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Before, when we ran coccicheck, it would only run on files that are currently being compiled. However, this leaves us with a blindspot where Windows-only sources are not checked since Coccinelle does not run on Windows. This patchset addresses this by making the "coccicheck" target run against all source files in the repository, except for source files that are pulled from some upstream source. This patchset should serve as a continuation of the original discussion about running coccicheck on all sources[1]. These patches depend on "ds/midx-expire-repack" and "dl/compat-cleanup". [1]: https://public-inbox.org/git/nycvar.QRO.7.76.6.1905031127170.45@tvgsbejvaqbjf.bet/ Denton Liu (3): Makefile: define THIRD_PARTY_SOURCES Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Makefile: run coccicheck on more source files Makefile | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) Range-diff against v1: 1: bc3cd4637a < -: ---------- Makefile: define UPSTREAM_SOURCES -: ---------- > 1: 3878cb2598 Makefile: define THIRD_PARTY_SOURCES -: ---------- > 2: 239ffa690c Makefile: strip leading ./ in $(FIND_SOURCE_FILES) 2: f62b0c7d17 ! 3: 89b7e17469 Makefile: run coccicheck on more source files @@ Commit message Coccinelle currently runs only on Linux. Make the "coccicheck" target run on all C sources except for those that - are taken from some upstream. We don't want to patch these files since - we want them to be as close to upstream as possible so that it'll be - easier to pull in upstream updates. + are taken from some third-party source. We don't want to patch these + files since we want them to be as close to upstream as possible so that + it'll be easier to pull in upstream updates. - This results in the following sources now being checked: + When running a build on Arch Linux with no additional flags provided, + after applying this patch, the following sources are now checked: * block-sha1/sha1.c * compat/access.c @@ Makefile: check: command-list.h -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) -endif +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) -+COCCI_SOURCES = $(filter-out $(UPSTREAM_SOURCES),$(FIND_C_SOURCES)) ++COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) @echo ' ' SPATCH $<; \ -- 2.23.0.37.g745f681289 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES 2019-09-12 17:28 ` [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources Denton Liu @ 2019-09-12 17:28 ` Denton Liu 2019-09-12 21:42 ` Junio C Hamano 2019-09-12 17:28 ` [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu ` (2 subsequent siblings) 3 siblings, 1 reply; 29+ messages in thread From: Denton Liu @ 2019-09-12 17:28 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano After looking through the source files in compat/ and investigating the files' content and/or its Git history, I've determined the list of files that were copied from a third-party source. Place the names of these files into the THIRD_PARTY_SOURCES variable in the Makefile. In addition, add the sha1collisiondetection/ and sha1dc/ sources as well since they are also imported from a third-party source. In a future commit, this variable will be used to determine which files are excluded when running the "coccicheck" target. Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/Makefile b/Makefile index ad71ae1219..b88b42d7ed 100644 --- a/Makefile +++ b/Makefile @@ -598,6 +598,7 @@ SCRIPT_SH = SCRIPT_LIB = TEST_BUILTINS_OBJS = TEST_PROGRAMS_NEED_X = +THIRD_PARTY_SOURCES = # Having this variable in your environment would break pipelines because # you cause "cd" to echo its destination to stdout. It can also take @@ -1146,6 +1147,15 @@ BUILTIN_OBJS += builtin/verify-tag.o BUILTIN_OBJS += builtin/worktree.o BUILTIN_OBJS += builtin/write-tree.o +THIRD_PARTY_SOURCES += compat/inet_ntop.c +THIRD_PARTY_SOURCES += compat/inet_pton.c +THIRD_PARTY_SOURCES += compat/obstack.% +THIRD_PARTY_SOURCES += compat/nedmalloc/% +THIRD_PARTY_SOURCES += compat/poll/% +THIRD_PARTY_SOURCES += compat/regex/% +THIRD_PARTY_SOURCES += sha1collisiondetection/% +THIRD_PARTY_SOURCES += sha1dc/% + GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) EXTLIBS = -- 2.23.0.37.g745f681289 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES 2019-09-12 17:28 ` [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES Denton Liu @ 2019-09-12 21:42 ` Junio C Hamano 0 siblings, 0 replies; 29+ messages in thread From: Junio C Hamano @ 2019-09-12 21:42 UTC (permalink / raw) To: Denton Liu Cc: Git Mailing List, Johannes Schindelin, SZEDER Gábor, Jeff King Denton Liu <liu.denton@gmail.com> writes: > After looking through the source files in compat/ and investigating the > files' content and/or its Git history, I've determined the list of files > that were copied from a third-party source. Place the names of these > files into the THIRD_PARTY_SOURCES variable in the Makefile. Please make sure readers won't get fooled into thinking these are filenames. They are meant to be used in $(filter-out) as patterns to match the names of third-party source files. Also, the paragraph before that talks about "I did this", but claiming credit is much less important than why we are doing this in the first place, which you forgot to say (perhaps because you were too deeply into the task and the motivation was too obvious to you). So, if I were doing this patch, I'd justify the whole thing as such: Some files in our codebase are borrowed from other projects, and minimally updated to suit our own needs. We'd sometimes need to tell our own sources and these third-party sources apart for management purposes (e.g. we may want to be less strict about coding style and other issues on third-party files). Define the $(MAKE) variable THIRD_PARTY_SOURCES that can be used to match names of third-party sources. > > +THIRD_PARTY_SOURCES += compat/inet_ntop.c > +THIRD_PARTY_SOURCES += compat/inet_pton.c > +THIRD_PARTY_SOURCES += compat/obstack.% > +THIRD_PARTY_SOURCES += compat/nedmalloc/% > +THIRD_PARTY_SOURCES += compat/poll/% > +THIRD_PARTY_SOURCES += compat/regex/% > +THIRD_PARTY_SOURCES += sha1collisiondetection/% > +THIRD_PARTY_SOURCES += sha1dc/% Before this block, please explain that this is meant to be a list of patterns suitable for $(filter-out) and friends in a comment. Thanks. ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) 2019-09-12 17:28 ` [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-12 17:28 ` [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES Denton Liu @ 2019-09-12 17:28 ` Denton Liu 2019-09-12 18:18 ` Junio C Hamano 2019-09-13 12:05 ` SZEDER Gábor 2019-09-12 17:28 ` [PATCH v2 3/3] Makefile: run coccicheck on more source files Denton Liu 2019-09-16 19:23 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu 3 siblings, 2 replies; 29+ messages in thread From: Denton Liu @ 2019-09-12 17:28 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Currently, $(FIND_SOURCE_FILES) has two modes: if `git ls-files` is present, it will use that to enumerate the files in the repository; else it will use `$(FIND) .` to enumerate the files in the directory. There is a subtle difference between these two methods, however. With ls-files, filenames don't have a leading `./` while with $(FIND), they do. This does not currently pose a problem but in a future patch, we will be using `filter-out` to process the list of files with the assumption that there is no prefix. Unify the two possible invocations in $(FIND_SOURCE_FILES) by using sed to remove the `./` prefix in the $(FIND) case. Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/Makefile b/Makefile index b88b42d7ed..e2c693440b 100644 --- a/Makefile +++ b/Makefile @@ -2610,6 +2610,7 @@ FIND_SOURCE_FILES = ( \ -o \( -name 'trash*' -type d -prune \) \ -o \( -name '*.[hcS]' -type f -print \) \ -o \( -name '*.sh' -type f -print \) \ + | sed -e 's|^\./||' \ ) $(ETAGS_TARGET): FORCE -- 2.23.0.37.g745f681289 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) 2019-09-12 17:28 ` [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu @ 2019-09-12 18:18 ` Junio C Hamano 2019-09-13 12:05 ` SZEDER Gábor 1 sibling, 0 replies; 29+ messages in thread From: Junio C Hamano @ 2019-09-12 18:18 UTC (permalink / raw) To: Denton Liu Cc: Git Mailing List, Johannes Schindelin, SZEDER Gábor, Jeff King Denton Liu <liu.denton@gmail.com> writes: > Unify the two possible invocations in $(FIND_SOURCE_FILES) by using sed > to remove the `./` prefix in the $(FIND) case. > diff --git a/Makefile b/Makefile > index b88b42d7ed..e2c693440b 100644 > --- a/Makefile > +++ b/Makefile > @@ -2610,6 +2610,7 @@ FIND_SOURCE_FILES = ( \ > -o \( -name 'trash*' -type d -prune \) \ > -o \( -name '*.[hcS]' -type f -print \) \ > -o \( -name '*.sh' -type f -print \) \ > + | sed -e 's|^\./||' \ > ) > > $(ETAGS_TARGET): FORCE An extra process and piping is a bit unfortunate, but this is purely a fallback position where it does not matter too much to expend one-shot overhead; it would only be needed until the user has a working "git ls-files" and starts hacking on our codebase, as this is used only by the xtags targets, which are only for developers. Thanks. Will queue. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) 2019-09-12 17:28 ` [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu 2019-09-12 18:18 ` Junio C Hamano @ 2019-09-13 12:05 ` SZEDER Gábor 1 sibling, 0 replies; 29+ messages in thread From: SZEDER Gábor @ 2019-09-13 12:05 UTC (permalink / raw) To: Denton Liu Cc: Git Mailing List, Johannes Schindelin, Jeff King, Junio C Hamano, Ramsay Jones On Thu, Sep 12, 2019 at 10:28:32AM -0700, Denton Liu wrote: > Currently, $(FIND_SOURCE_FILES) has two modes: if `git ls-files` is > present, it will use that to enumerate the files in the repository; else > it will use `$(FIND) .` to enumerate the files in the directory. > > There is a subtle difference between these two methods, however. With > ls-files, filenames don't have a leading `./` while with $(FIND), they > do. This does not currently pose a problem but in a future patch, we > will be using `filter-out` to process the list of files with the > assumption that there is no prefix. > > Unify the two possible invocations in $(FIND_SOURCE_FILES) by using sed > to remove the `./` prefix in the $(FIND) case. > > Signed-off-by: Denton Liu <liu.denton@gmail.com> > --- > Makefile | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/Makefile b/Makefile > index b88b42d7ed..e2c693440b 100644 > --- a/Makefile > +++ b/Makefile > @@ -2610,6 +2610,7 @@ FIND_SOURCE_FILES = ( \ > -o \( -name 'trash*' -type d -prune \) \ > -o \( -name '*.[hcS]' -type f -print \) \ > -o \( -name '*.sh' -type f -print \) \ > + | sed -e 's|^\./||' \ Ramsay pointed out to me (in an off-list email) that the list of library header files is generated in a similar way, and then those leading './' have to be removed later. See how the LIB_H and CHK_HDRS variables are assigned. I think it would be great to apply the same treatment there, too, to eliminate those './' prefixes right at the source. ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 3/3] Makefile: run coccicheck on more source files 2019-09-12 17:28 ` [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-12 17:28 ` [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES Denton Liu 2019-09-12 17:28 ` [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu @ 2019-09-12 17:28 ` Denton Liu 2019-09-12 18:40 ` Junio C Hamano 2019-09-16 19:23 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu 3 siblings, 1 reply; 29+ messages in thread From: Denton Liu @ 2019-09-12 17:28 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Before, when running the "coccicheck" target, only the source files which were being compiled would have been checked by Coccinelle. However, just because we aren't compiling a source file doesn't mean we have to exclude it from analysis. This will allow us to catch more mistakes, in particular ones that affect Windows-only sources since Coccinelle currently runs only on Linux. Make the "coccicheck" target run on all C sources except for those that are taken from some third-party source. We don't want to patch these files since we want them to be as close to upstream as possible so that it'll be easier to pull in upstream updates. When running a build on Arch Linux with no additional flags provided, after applying this patch, the following sources are now checked: * block-sha1/sha1.c * compat/access.c * compat/basename.c * compat/fileno.c * compat/gmtime.c * compat/hstrerror.c * compat/memmem.c * compat/mingw.c * compat/mkdir.c * compat/mkdtemp.c * compat/mmap.c * compat/msvc.c * compat/pread.c * compat/precompose_utf8.c * compat/qsort.c * compat/setenv.c * compat/sha1-chunked.c * compat/snprintf.c * compat/stat.c * compat/strcasestr.c * compat/strdup.c * compat/strtoimax.c * compat/strtoumax.c * compat/unsetenv.c * compat/win32/dirent.c * compat/win32/path-utils.c * compat/win32/pthread.c * compat/win32/syslog.c * compat/win32/trace2_win32_process_info.c * compat/win32mmap.c * compat/winansi.c * ppc/sha1.c This also results in the following source now being excluded: * compat/obstack.c Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index e2c693440b..7c88e0606f 100644 --- a/Makefile +++ b/Makefile @@ -2803,12 +2803,8 @@ check: command-list.h exit 1; \ fi -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) -ifdef DC_SHA1_SUBMODULE -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) -else -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) -endif +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) @echo ' ' SPATCH $<; \ -- 2.23.0.37.g745f681289 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files 2019-09-12 17:28 ` [PATCH v2 3/3] Makefile: run coccicheck on more source files Denton Liu @ 2019-09-12 18:40 ` Junio C Hamano 2019-09-13 11:49 ` SZEDER Gábor 0 siblings, 1 reply; 29+ messages in thread From: Junio C Hamano @ 2019-09-12 18:40 UTC (permalink / raw) To: Denton Liu Cc: Git Mailing List, Johannes Schindelin, SZEDER Gábor, Jeff King Denton Liu <liu.denton@gmail.com> writes: > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES)) The former is somewhat misnamed. FIND_SOURCE_FILES is *not* a list of source files---it is a procedure to list source files to its standard output. FIND_C_SOUCRES sounds as if it is a similar procedure, which would be implemented much like FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p' but that is not what you did and that is not what you want to have. Perhaps call it FOUND_C_SOURCES? I wonder if we can get rid of FIND_SOURCE_FILES that is a mere procedure and replace its use with a true list of source files. Would it make the result more pleasant to work with? Perhaps something like the attached patch, (which would come before this entire thing as a clean-up, and removing the need for 2/3)? I dunno. Using a procedure whose output is fed to xargs has an advantage that a platform with very short command line limit can still work with many source files, but the way you create and use COCCI_SOURCES in this patch would defeat that advantage anyway, so perhaps we can get away with an approach like this. Having a list of things in $(MAKE) variable has a longer-term benefit that we could exploit more parallelism if we wanted to, too. Makefile | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index f9255344ae..9dddd0e88c 100644 --- a/Makefile +++ b/Makefile @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm $(QUIET_GEN)mkdir -p $(dir $@) && \ pod2man $< $@ -FIND_SOURCE_FILES = ( \ +SOURCE_FILES = $(patsubst ./%,%,$(shell \ git ls-files \ '*.[hcS]' \ '*.sh' \ @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \ -o \( -name 'trash*' -type d -prune \) \ -o \( -name '*.[hcS]' -type f -print \) \ -o \( -name '*.sh' -type f -print \) \ - ) + )) $(ETAGS_TARGET): FORCE $(RM) $(ETAGS_TARGET) - $(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET) + etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES) tags: FORCE $(RM) tags - $(FIND_SOURCE_FILES) | xargs ctags -a + ctags -a $(SOURCE_FILES) cscope: $(RM) cscope* - $(FIND_SOURCE_FILES) | xargs cscope -b + cscope -b $(SOURCE_FILES) ### Detect prefix changes TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\ ^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files 2019-09-12 18:40 ` Junio C Hamano @ 2019-09-13 11:49 ` SZEDER Gábor 2019-09-13 17:14 ` Denton Liu 2019-09-13 17:38 ` Junio C Hamano 0 siblings, 2 replies; 29+ messages in thread From: SZEDER Gábor @ 2019-09-13 11:49 UTC (permalink / raw) To: Junio C Hamano Cc: Denton Liu, Git Mailing List, Johannes Schindelin, Jeff King On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote: > Denton Liu <liu.denton@gmail.com> writes: > > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES)) > > The former is somewhat misnamed. FIND_SOURCE_FILES is *not* a list > of source files---it is a procedure to list source files to its > standard output. FIND_C_SOUCRES sounds as if it is a similar > procedure, which would be implemented much like > > FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p' > > but that is not what you did and that is not what you want to have. > Perhaps call it FOUND_C_SOURCES? > > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere > procedure and replace its use with a true list of source files. > Would it make the result more pleasant to work with? > > Perhaps something like the attached patch, (which would come before > this entire thing as a clean-up, and removing the need for 2/3)? > > I dunno. > > Using a procedure whose output is fed to xargs has an advantage that > a platform with very short command line limit can still work with > many source files, but the way you create and use COCCI_SOURCES in > this patch would defeat that advantage anyway, COCCI_SOURCES is only used as an input to 'xargs', so that advantage is not defeated. > so perhaps we can get > away with an approach like this. Having a list of things in $(MAKE) > variable has a longer-term benefit that we could exploit more > parallelism if we wanted to, too. > > Makefile | 10 +++++----- > 1 file changed, 5 insertions(+), 5 deletions(-) > > diff --git a/Makefile b/Makefile > index f9255344ae..9dddd0e88c 100644 > --- a/Makefile > +++ b/Makefile > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm > $(QUIET_GEN)mkdir -p $(dir $@) && \ > pod2man $< $@ > > -FIND_SOURCE_FILES = ( \ > +SOURCE_FILES = $(patsubst ./%,%,$(shell \ > git ls-files \ > '*.[hcS]' \ > '*.sh' \ > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \ > -o \( -name 'trash*' -type d -prune \) \ > -o \( -name '*.[hcS]' -type f -print \) \ > -o \( -name '*.sh' -type f -print \) \ > - ) > + )) > > $(ETAGS_TARGET): FORCE > $(RM) $(ETAGS_TARGET) > - $(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET) > + etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES) > > tags: FORCE > $(RM) tags > - $(FIND_SOURCE_FILES) | xargs ctags -a > + ctags -a $(SOURCE_FILES) > > cscope: > $(RM) cscope* > - $(FIND_SOURCE_FILES) | xargs cscope -b > + cscope -b $(SOURCE_FILES) > > ### Detect prefix changes > TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\ > > > > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files 2019-09-13 11:49 ` SZEDER Gábor @ 2019-09-13 17:14 ` Denton Liu 2019-09-13 18:00 ` SZEDER Gábor 2019-09-13 17:38 ` Junio C Hamano 1 sibling, 1 reply; 29+ messages in thread From: Denton Liu @ 2019-09-13 17:14 UTC (permalink / raw) To: SZEDER Gábor Cc: Junio C Hamano, Git Mailing List, Johannes Schindelin, Jeff King On Fri, Sep 13, 2019 at 01:49:52PM +0200, SZEDER Gábor wrote: > On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote: > > Denton Liu <liu.denton@gmail.com> writes: > > > > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) > > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES)) > > > > The former is somewhat misnamed. FIND_SOURCE_FILES is *not* a list > > of source files---it is a procedure to list source files to its > > standard output. FIND_C_SOUCRES sounds as if it is a similar > > procedure, which would be implemented much like > > > > FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p' > > > > but that is not what you did and that is not what you want to have. > > Perhaps call it FOUND_C_SOURCES? > > > > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere > > procedure and replace its use with a true list of source files. > > Would it make the result more pleasant to work with? > > > > Perhaps something like the attached patch, (which would come before > > this entire thing as a clean-up, and removing the need for 2/3)? > > > > I dunno. > > > > Using a procedure whose output is fed to xargs has an advantage that > > a platform with very short command line limit can still work with > > many source files, but the way you create and use COCCI_SOURCES in > > this patch would defeat that advantage anyway, > > COCCI_SOURCES is only used as an input to 'xargs', so that advantage > is not defeated. I think it still does matter; the relevant snippet is as follows: if ! echo $(COCCI_SOURCES) | xargs $$limit \ $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ >$@+ 2>$@.log; \ which means that a really big COCCI_SOURCES could exceed the limit. That being said, COCCI_SOURCES should be smaller than the future SOURCE_FILES variable since we're only taking %.c files (and filtering out some of them too!). I dunno, either. I'm mostly in favour of this change since it makes a lot of sense to keep lists in make variables if possible as opposed to command invocations. I guess worst case, if someone complains in the future, we can always change it back. > > > so perhaps we can get > > away with an approach like this. Having a list of things in $(MAKE) > > variable has a longer-term benefit that we could exploit more > > parallelism if we wanted to, too. > > > > Makefile | 10 +++++----- > > 1 file changed, 5 insertions(+), 5 deletions(-) > > > > diff --git a/Makefile b/Makefile > > index f9255344ae..9dddd0e88c 100644 > > --- a/Makefile > > +++ b/Makefile > > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm > > $(QUIET_GEN)mkdir -p $(dir $@) && \ > > pod2man $< $@ > > > > -FIND_SOURCE_FILES = ( \ > > +SOURCE_FILES = $(patsubst ./%,%,$(shell \ > > git ls-files \ > > '*.[hcS]' \ > > '*.sh' \ > > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \ > > -o \( -name 'trash*' -type d -prune \) \ > > -o \( -name '*.[hcS]' -type f -print \) \ > > -o \( -name '*.sh' -type f -print \) \ > > - ) > > + )) > > > > $(ETAGS_TARGET): FORCE > > $(RM) $(ETAGS_TARGET) > > - $(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET) > > + etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES) > > > > tags: FORCE > > $(RM) tags > > - $(FIND_SOURCE_FILES) | xargs ctags -a > > + ctags -a $(SOURCE_FILES) > > > > cscope: > > $(RM) cscope* > > - $(FIND_SOURCE_FILES) | xargs cscope -b > > + cscope -b $(SOURCE_FILES) > > > > ### Detect prefix changes > > TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\ > > > > > > > > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files 2019-09-13 17:14 ` Denton Liu @ 2019-09-13 18:00 ` SZEDER Gábor 2019-09-13 21:38 ` Denton Liu 0 siblings, 1 reply; 29+ messages in thread From: SZEDER Gábor @ 2019-09-13 18:00 UTC (permalink / raw) To: Denton Liu Cc: Junio C Hamano, Git Mailing List, Johannes Schindelin, Jeff King On Fri, Sep 13, 2019 at 10:14:01AM -0700, Denton Liu wrote: > On Fri, Sep 13, 2019 at 01:49:52PM +0200, SZEDER Gábor wrote: > > On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote: > > > Denton Liu <liu.denton@gmail.com> writes: > > > > > > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) > > > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES)) > > > > > > The former is somewhat misnamed. FIND_SOURCE_FILES is *not* a list > > > of source files---it is a procedure to list source files to its > > > standard output. FIND_C_SOUCRES sounds as if it is a similar > > > procedure, which would be implemented much like > > > > > > FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p' > > > > > > but that is not what you did and that is not what you want to have. > > > Perhaps call it FOUND_C_SOURCES? > > > > > > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere > > > procedure and replace its use with a true list of source files. > > > Would it make the result more pleasant to work with? > > > > > > Perhaps something like the attached patch, (which would come before > > > this entire thing as a clean-up, and removing the need for 2/3)? > > > > > > I dunno. > > > > > > Using a procedure whose output is fed to xargs has an advantage that > > > a platform with very short command line limit can still work with > > > many source files, but the way you create and use COCCI_SOURCES in > > > this patch would defeat that advantage anyway, > > > > COCCI_SOURCES is only used as an input to 'xargs', so that advantage > > is not defeated. > > I think it still does matter; the relevant snippet is as follows: > > if ! echo $(COCCI_SOURCES) | xargs $$limit \ > $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ > >$@+ 2>$@.log; \ > > which means that a really big COCCI_SOURCES could exceed the limit. Oh, you're both right. > That being said, COCCI_SOURCES should be smaller than the future > SOURCE_FILES variable since we're only taking %.c files (and filtering > out some of them too!). We could also argue that Coccinelle only runs on platforms that have a reasonably large command line arg limit, and the number of our source files is way below that, so it won't matter in the foreseeable future. (Furthermore, 'echo' is often a shell builtin command, and I don't think that the platform's argument size limit applies to them. At least the 'echo' of dash, Bash, ksh, ksh93, mksh, and BusyBox sh can deal with at least 10 million arguments; the platform limit is somewhere around 147k) > > > diff --git a/Makefile b/Makefile > > > index f9255344ae..9dddd0e88c 100644 > > > --- a/Makefile > > > +++ b/Makefile > > > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm > > > $(QUIET_GEN)mkdir -p $(dir $@) && \ > > > pod2man $< $@ > > > > > > -FIND_SOURCE_FILES = ( \ > > > +SOURCE_FILES = $(patsubst ./%,%,$(shell \ > > > git ls-files \ > > > '*.[hcS]' \ > > > '*.sh' \ > > > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \ > > > -o \( -name 'trash*' -type d -prune \) \ > > > -o \( -name '*.[hcS]' -type f -print \) \ > > > -o \( -name '*.sh' -type f -print \) \ > > > - ) > > > + )) > > > > > > $(ETAGS_TARGET): FORCE > > > $(RM) $(ETAGS_TARGET) > > > - $(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET) > > > + etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES) > > > > > > tags: FORCE > > > $(RM) tags > > > - $(FIND_SOURCE_FILES) | xargs ctags -a > > > + ctags -a $(SOURCE_FILES) > > > > > > cscope: > > > $(RM) cscope* > > > - $(FIND_SOURCE_FILES) | xargs cscope -b > > > + cscope -b $(SOURCE_FILES) Here, however, the list of source files is passed as argument to non-builtin commands, that also might be used on cmdline-arg-limit-challenged platforms. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files 2019-09-13 18:00 ` SZEDER Gábor @ 2019-09-13 21:38 ` Denton Liu 0 siblings, 0 replies; 29+ messages in thread From: Denton Liu @ 2019-09-13 21:38 UTC (permalink / raw) To: SZEDER Gábor Cc: Junio C Hamano, Git Mailing List, Johannes Schindelin, Jeff King On Fri, Sep 13, 2019 at 08:00:14PM +0200, SZEDER Gábor wrote: > On Fri, Sep 13, 2019 at 10:14:01AM -0700, Denton Liu wrote: > > On Fri, Sep 13, 2019 at 01:49:52PM +0200, SZEDER Gábor wrote: > > > On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote: > > > > Denton Liu <liu.denton@gmail.com> writes: > > > > > > > > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) > > > > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES)) > > > > > > > > The former is somewhat misnamed. FIND_SOURCE_FILES is *not* a list > > > > of source files---it is a procedure to list source files to its > > > > standard output. FIND_C_SOUCRES sounds as if it is a similar > > > > procedure, which would be implemented much like > > > > > > > > FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p' > > > > > > > > but that is not what you did and that is not what you want to have. > > > > Perhaps call it FOUND_C_SOURCES? > > > > > > > > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere > > > > procedure and replace its use with a true list of source files. > > > > Would it make the result more pleasant to work with? > > > > > > > > Perhaps something like the attached patch, (which would come before > > > > this entire thing as a clean-up, and removing the need for 2/3)? > > > > > > > > I dunno. > > > > > > > > Using a procedure whose output is fed to xargs has an advantage that > > > > a platform with very short command line limit can still work with > > > > many source files, but the way you create and use COCCI_SOURCES in > > > > this patch would defeat that advantage anyway, > > > > > > COCCI_SOURCES is only used as an input to 'xargs', so that advantage > > > is not defeated. > > > > I think it still does matter; the relevant snippet is as follows: > > > > if ! echo $(COCCI_SOURCES) | xargs $$limit \ > > $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ > > >$@+ 2>$@.log; \ > > > > which means that a really big COCCI_SOURCES could exceed the limit. > > Oh, you're both right. > > > That being said, COCCI_SOURCES should be smaller than the future > > SOURCE_FILES variable since we're only taking %.c files (and filtering > > out some of them too!). > > We could also argue that Coccinelle only runs on platforms that have a > reasonably large command line arg limit, and the number of our source > files is way below that, so it won't matter in the foreseeable future. Good point. > > (Furthermore, 'echo' is often a shell builtin command, and I don't > think that the platform's argument size limit applies to them. At > least the 'echo' of dash, Bash, ksh, ksh93, mksh, and BusyBox sh can > deal with at least 10 million arguments; the platform limit is > somewhere around 147k) > > > > > diff --git a/Makefile b/Makefile > > > > index f9255344ae..9dddd0e88c 100644 > > > > --- a/Makefile > > > > +++ b/Makefile > > > > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm > > > > $(QUIET_GEN)mkdir -p $(dir $@) && \ > > > > pod2man $< $@ > > > > > > > > -FIND_SOURCE_FILES = ( \ > > > > +SOURCE_FILES = $(patsubst ./%,%,$(shell \ > > > > git ls-files \ > > > > '*.[hcS]' \ > > > > '*.sh' \ > > > > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \ > > > > -o \( -name 'trash*' -type d -prune \) \ > > > > -o \( -name '*.[hcS]' -type f -print \) \ > > > > -o \( -name '*.sh' -type f -print \) \ > > > > - ) > > > > + )) > > > > > > > > $(ETAGS_TARGET): FORCE > > > > $(RM) $(ETAGS_TARGET) > > > > - $(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET) > > > > + etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES) > > > > > > > > tags: FORCE > > > > $(RM) tags > > > > - $(FIND_SOURCE_FILES) | xargs ctags -a > > > > + ctags -a $(SOURCE_FILES) > > > > > > > > cscope: > > > > $(RM) cscope* > > > > - $(FIND_SOURCE_FILES) | xargs cscope -b > > > > + cscope -b $(SOURCE_FILES) > > Here, however, the list of source files is passed as argument to > non-builtin commands, that also might be used on > cmdline-arg-limit-challenged platforms. > After doing a bit of research, I think that I agree with you. It seems like the max command-line length for CMD on Windows is 8191 characters. However, after running the following, $ git ls-files '*.[hcS]' '*.sh' ':!*[tp][0-9][0-9][0-9][0-9]*' ':!contrib' | wc -c 12779 we can see that the command-line length would definitely exceed the max length so xargs would be required. As a result, we should probably just keep the existing xargs invocations. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files 2019-09-13 11:49 ` SZEDER Gábor 2019-09-13 17:14 ` Denton Liu @ 2019-09-13 17:38 ` Junio C Hamano 1 sibling, 0 replies; 29+ messages in thread From: Junio C Hamano @ 2019-09-13 17:38 UTC (permalink / raw) To: SZEDER Gábor Cc: Denton Liu, Git Mailing List, Johannes Schindelin, Jeff King SZEDER Gábor <szeder.dev@gmail.com> writes: >> Using a procedure whose output is fed to xargs has an advantage that >> a platform with very short command line limit can still work with >> many source files, but the way you create and use COCCI_SOURCES in >> this patch would defeat that advantage anyway, > > COCCI_SOURCES is only used as an input to 'xargs', so that advantage > is not defeated. It is passed as a command line argument to "echo", that pipes to xargs; I would not say it is taking advantage of "xargs" to lift the command line length limit, as it first needs to convince the shell to feed all of them to the "echo" that is upstream of "xargs". As you mentioned elsewhere, LIB_H already uses the same approach as I outlined in the message you are responding to (i.e. "don't define a procedure to produce lines to the standard output in a $(MAKE) variable--instead make the variable to hold the list itself"), so I suspect that we are almost on the same page? ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources 2019-09-12 17:28 ` [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources Denton Liu ` (2 preceding siblings ...) 2019-09-12 17:28 ` [PATCH v2 3/3] Makefile: run coccicheck on more source files Denton Liu @ 2019-09-16 19:23 ` Denton Liu 2019-09-16 19:23 ` [PATCH v3 1/4] Makefile: strip leading ./ in $(LIB_H) Denton Liu ` (4 more replies) 3 siblings, 5 replies; 29+ messages in thread From: Denton Liu @ 2019-09-16 19:23 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Hi all, I spent the weekend brooding over this and I opted to not convert $(FIND_SOURCE_FILES) into a list of filenames. I put the justification for it in 4/4 so, hopefully, it'll make sense for someone who might want to do the same change in the future. Before, when we ran coccicheck, it would only run on files that are currently being compiled. However, this leaves us with a blindspot where Windows-only sources are not checked since Coccinelle does not run on Windows. This patchset addresses this by making the "coccicheck" target run against all source files in the repository, except for source files that are pulled from some upstream source. This patchset should serve as a continuation of the original discussion about running coccicheck on all sources[1]. These patches depend on "ds/midx-expire-repack" and "dl/compat-cleanup". [1]: https://public-inbox.org/git/nycvar.QRO.7.76.6.1905031127170.45@tvgsbejvaqbjf.bet/ Denton Liu (4): Makefile: strip leading ./ in $(LIB_H) Makefile: define THIRD_PARTY_SOURCES Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Makefile: run coccicheck on more source files Makefile | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) Range-diff against v2: -: ---------- > 1: e7fa5e1df7 Makefile: strip leading ./ in $(LIB_H) 1: 72b4cc68f5 ! 2: a7c855a973 Makefile: define THIRD_PARTY_SOURCES @@ Metadata ## Commit message ## Makefile: define THIRD_PARTY_SOURCES - After looking through the source files in compat/ and investigating the - files' content and/or its Git history, I've determined the list of files - that were copied from a third-party source. Place the names of these - files into the THIRD_PARTY_SOURCES variable in the Makefile. + Some files in our codebase are borrowed from other projects, and + minimally updated to suit our own needs. We'd sometimes need to tell + our own sources and these third-party sources apart for management + purposes (e.g. we may want to be less strict about coding style and + other issues on third-party files). - In addition, add the sha1collisiondetection/ and sha1dc/ sources as well - since they are also imported from a third-party source. - - In a future commit, this variable will be used to determine which files - are excluded when running the "coccicheck" target. + Define the $(MAKE) variable THIRD_PARTY_SOURCES that can be used to + match names of third-party sources. Signed-off-by: Denton Liu <liu.denton@gmail.com> - Signed-off-by: Junio C Hamano <gitster@pobox.com> ## Makefile ## @@ Makefile: SCRIPT_SH = @@ Makefile: BUILTIN_OBJS += builtin/verify-tag.o BUILTIN_OBJS += builtin/worktree.o BUILTIN_OBJS += builtin/write-tree.o ++# THIRD_PARTY_SOURCES is a list of patterns compatible with ++# the $(filter) and $(filter-out) family of functions +THIRD_PARTY_SOURCES += compat/inet_ntop.c +THIRD_PARTY_SOURCES += compat/inet_pton.c +THIRD_PARTY_SOURCES += compat/obstack.% 2: a51acfec66 ! 3: 71861cbdf8 Makefile: strip leading ./ in $(FIND_SOURCE_FILES) @@ Commit message to remove the `./` prefix in the $(FIND) case. Signed-off-by: Denton Liu <liu.denton@gmail.com> - Signed-off-by: Junio C Hamano <gitster@pobox.com> ## Makefile ## @@ Makefile: FIND_SOURCE_FILES = ( \ 3: 75c36bc9a1 ! 4: 0ecfcd8858 Makefile: run coccicheck on more source files @@ Commit message * compat/obstack.c + Instead of generating $(FOUND_C_SOURCES) from a + `$(shell $(FIND_SOURCE_FILES))` invocation, an alternative design was + considered which involved converting $(FIND_SOURCE_FILES) into + $(SOURCE_FILES) which would hold a list of filenames from the + $(FIND_SOURCE_FILES) invocation. We would simply filter `%.c` files into + $(ALL_C_SOURCES). $(SOURCE_FILES) would then be passed directly to the + etags, ctags and cscope commands. We can see from the following + invocation + + $ git ls-files '*.[hcS]' '*.sh' ':!*[tp][0-9][0-9][0-9][0-9]*' ':!contrib' | wc -c + 12779 + + that the number of characters in this list would pose a problem on + platforms with short command-line length limits (such as CMD which has a + max of 8191 characters). As a result, we don't perform this change. + + However, we can see that the same issue may apply when running + Coccinelle since $(COCCI_SOURCES) is also a list of filenames: + + if ! echo $(COCCI_SOURCES) | xargs $$limit \ + $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ + >$@+ 2>$@.log; \ + + This is justified since platforms that support Coccinelle generally have + reasonably long command-line length limits and so we are safe for the + foreseeable future. + Signed-off-by: Denton Liu <liu.denton@gmail.com> - Signed-off-by: Junio C Hamano <gitster@pobox.com> ## Makefile ## @@ Makefile: check: command-list.h @@ Makefile: check: command-list.h -else -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) -endif -+FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) -+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES)) ++FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) ++COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) @echo ' ' SPATCH $<; \ -- 2.23.0 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 1/4] Makefile: strip leading ./ in $(LIB_H) 2019-09-16 19:23 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu @ 2019-09-16 19:23 ` Denton Liu 2019-09-16 19:23 ` [PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES Denton Liu ` (3 subsequent siblings) 4 siblings, 0 replies; 29+ messages in thread From: Denton Liu @ 2019-09-16 19:23 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Currently, $(LIB_H) is generated from two modes: if `git ls-files` is present, it will use that to enumerate the files in the repository; else it will use `$(FIND) .` to enumerate the files in the directory. There is a subtle difference between these two methods, however. With ls-files, filenames don't have a leading `./` while with $(FIND), they do. This results in $(CHK_HDRS) having to substitute out the leading `./` before it uses $(LIB_H). Unify the two possible values in $(LIB_H) by using patsubst to remove the `./` prefix at its definition. Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index ad71ae1219..ea77198247 100644 --- a/Makefile +++ b/Makefile @@ -818,12 +818,12 @@ VCSSVN_LIB = vcs-svn/lib.a GENERATED_H += command-list.h -LIB_H := $(sort $(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \ +LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \ $(FIND) . \ -name .git -prune -o \ -name t -prune -o \ -name Documentation -prune -o \ - -name '*.h' -print)) + -name '*.h' -print))) LIB_OBJS += abspath.o LIB_OBJS += advice.o @@ -2769,7 +2769,7 @@ EXCEPT_HDRS := $(GEN_HDRS) compat/% xdiff/% ifndef GCRYPT_SHA256 EXCEPT_HDRS += sha256/gcrypt.h endif -CHK_HDRS = $(filter-out $(EXCEPT_HDRS),$(patsubst ./%,%,$(LIB_H))) +CHK_HDRS = $(filter-out $(EXCEPT_HDRS),$(LIB_H)) HCO = $(patsubst %.h,%.hco,$(CHK_HDRS)) $(HCO): %.hco: %.h FORCE -- 2.23.0 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES 2019-09-16 19:23 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-16 19:23 ` [PATCH v3 1/4] Makefile: strip leading ./ in $(LIB_H) Denton Liu @ 2019-09-16 19:23 ` Denton Liu 2019-09-16 20:56 ` Junio C Hamano 2019-09-16 19:23 ` [PATCH v3 3/4] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu ` (2 subsequent siblings) 4 siblings, 1 reply; 29+ messages in thread From: Denton Liu @ 2019-09-16 19:23 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Some files in our codebase are borrowed from other projects, and minimally updated to suit our own needs. We'd sometimes need to tell our own sources and these third-party sources apart for management purposes (e.g. we may want to be less strict about coding style and other issues on third-party files). Define the $(MAKE) variable THIRD_PARTY_SOURCES that can be used to match names of third-party sources. Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/Makefile b/Makefile index ea77198247..49839579ec 100644 --- a/Makefile +++ b/Makefile @@ -598,6 +598,7 @@ SCRIPT_SH = SCRIPT_LIB = TEST_BUILTINS_OBJS = TEST_PROGRAMS_NEED_X = +THIRD_PARTY_SOURCES = # Having this variable in your environment would break pipelines because # you cause "cd" to echo its destination to stdout. It can also take @@ -1146,6 +1147,17 @@ BUILTIN_OBJS += builtin/verify-tag.o BUILTIN_OBJS += builtin/worktree.o BUILTIN_OBJS += builtin/write-tree.o +# THIRD_PARTY_SOURCES is a list of patterns compatible with +# the $(filter) and $(filter-out) family of functions +THIRD_PARTY_SOURCES += compat/inet_ntop.c +THIRD_PARTY_SOURCES += compat/inet_pton.c +THIRD_PARTY_SOURCES += compat/obstack.% +THIRD_PARTY_SOURCES += compat/nedmalloc/% +THIRD_PARTY_SOURCES += compat/poll/% +THIRD_PARTY_SOURCES += compat/regex/% +THIRD_PARTY_SOURCES += sha1collisiondetection/% +THIRD_PARTY_SOURCES += sha1dc/% + GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) EXTLIBS = -- 2.23.0 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES 2019-09-16 19:23 ` [PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES Denton Liu @ 2019-09-16 20:56 ` Junio C Hamano 2019-09-16 22:00 ` [PATCH] fixup! " Denton Liu 0 siblings, 1 reply; 29+ messages in thread From: Junio C Hamano @ 2019-09-16 20:56 UTC (permalink / raw) To: Denton Liu Cc: Git Mailing List, Johannes Schindelin, SZEDER Gábor, Jeff King Denton Liu <liu.denton@gmail.com> writes: > +# THIRD_PARTY_SOURCES is a list of patterns compatible with > +# the $(filter) and $(filter-out) family of functions That defines the format, but does it convey what they want to achieve to the readers? "... to catch the list of source files we borrowed from elsewhere" or something, perhaps? > +THIRD_PARTY_SOURCES += compat/inet_ntop.c > +THIRD_PARTY_SOURCES += compat/inet_pton.c > +THIRD_PARTY_SOURCES += compat/obstack.% > +THIRD_PARTY_SOURCES += compat/nedmalloc/% > +THIRD_PARTY_SOURCES += compat/poll/% > +THIRD_PARTY_SOURCES += compat/regex/% > +THIRD_PARTY_SOURCES += sha1collisiondetection/% > +THIRD_PARTY_SOURCES += sha1dc/% > + > GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) > EXTLIBS = ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH] fixup! Makefile: define THIRD_PARTY_SOURCES 2019-09-16 20:56 ` Junio C Hamano @ 2019-09-16 22:00 ` Denton Liu 0 siblings, 0 replies; 29+ messages in thread From: Denton Liu @ 2019-09-16 22:00 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano * expand the comment to show what it's used for * alphabetical ordering fix Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 522342a46c..617c25d16b 100644 --- a/Makefile +++ b/Makefile @@ -1147,12 +1147,15 @@ BUILTIN_OBJS += builtin/verify-tag.o BUILTIN_OBJS += builtin/worktree.o BUILTIN_OBJS += builtin/write-tree.o -# THIRD_PARTY_SOURCES is a list of patterns compatible with -# the $(filter) and $(filter-out) family of functions +# THIRD_PARTY_SOURCES is a list of patterns compatible with the +# $(filter) and $(filter-out) family of functions. They specify source +# files which are taken from some third-party source where we want to be +# less strict about issues such as coding style so we don't diverge from +# upstream unnecessarily (making merging in future changes easier). THIRD_PARTY_SOURCES += compat/inet_ntop.c THIRD_PARTY_SOURCES += compat/inet_pton.c -THIRD_PARTY_SOURCES += compat/obstack.% THIRD_PARTY_SOURCES += compat/nedmalloc/% +THIRD_PARTY_SOURCES += compat/obstack.% THIRD_PARTY_SOURCES += compat/poll/% THIRD_PARTY_SOURCES += compat/regex/% THIRD_PARTY_SOURCES += sha1collisiondetection/% -- 2.23.0 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v3 3/4] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) 2019-09-16 19:23 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-16 19:23 ` [PATCH v3 1/4] Makefile: strip leading ./ in $(LIB_H) Denton Liu 2019-09-16 19:23 ` [PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES Denton Liu @ 2019-09-16 19:23 ` Denton Liu 2019-09-16 19:23 ` [PATCH v3 4/4] Makefile: run coccicheck on more source files Denton Liu 2019-09-16 20:57 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Junio C Hamano 4 siblings, 0 replies; 29+ messages in thread From: Denton Liu @ 2019-09-16 19:23 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Currently, $(FIND_SOURCE_FILES) has two modes: if `git ls-files` is present, it will use that to enumerate the files in the repository; else it will use `$(FIND) .` to enumerate the files in the directory. There is a subtle difference between these two methods, however. With ls-files, filenames don't have a leading `./` while with $(FIND), they do. This does not currently pose a problem but in a future patch, we will be using `filter-out` to process the list of files with the assumption that there is no prefix. Unify the two possible invocations in $(FIND_SOURCE_FILES) by using sed to remove the `./` prefix in the $(FIND) case. Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/Makefile b/Makefile index 49839579ec..20eb5c5c42 100644 --- a/Makefile +++ b/Makefile @@ -2612,6 +2612,7 @@ FIND_SOURCE_FILES = ( \ -o \( -name 'trash*' -type d -prune \) \ -o \( -name '*.[hcS]' -type f -print \) \ -o \( -name '*.sh' -type f -print \) \ + | sed -e 's|^\./||' \ ) $(ETAGS_TARGET): FORCE -- 2.23.0 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v3 4/4] Makefile: run coccicheck on more source files 2019-09-16 19:23 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu ` (2 preceding siblings ...) 2019-09-16 19:23 ` [PATCH v3 3/4] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu @ 2019-09-16 19:23 ` Denton Liu 2019-09-16 20:57 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Junio C Hamano 4 siblings, 0 replies; 29+ messages in thread From: Denton Liu @ 2019-09-16 19:23 UTC (permalink / raw) To: Git Mailing List Cc: Johannes Schindelin, SZEDER Gábor, Jeff King, Junio C Hamano Before, when running the "coccicheck" target, only the source files which were being compiled would have been checked by Coccinelle. However, just because we aren't compiling a source file doesn't mean we have to exclude it from analysis. This will allow us to catch more mistakes, in particular ones that affect Windows-only sources since Coccinelle currently runs only on Linux. Make the "coccicheck" target run on all C sources except for those that are taken from some third-party source. We don't want to patch these files since we want them to be as close to upstream as possible so that it'll be easier to pull in upstream updates. When running a build on Arch Linux with no additional flags provided, after applying this patch, the following sources are now checked: * block-sha1/sha1.c * compat/access.c * compat/basename.c * compat/fileno.c * compat/gmtime.c * compat/hstrerror.c * compat/memmem.c * compat/mingw.c * compat/mkdir.c * compat/mkdtemp.c * compat/mmap.c * compat/msvc.c * compat/pread.c * compat/precompose_utf8.c * compat/qsort.c * compat/setenv.c * compat/sha1-chunked.c * compat/snprintf.c * compat/stat.c * compat/strcasestr.c * compat/strdup.c * compat/strtoimax.c * compat/strtoumax.c * compat/unsetenv.c * compat/win32/dirent.c * compat/win32/path-utils.c * compat/win32/pthread.c * compat/win32/syslog.c * compat/win32/trace2_win32_process_info.c * compat/win32mmap.c * compat/winansi.c * ppc/sha1.c This also results in the following source now being excluded: * compat/obstack.c Instead of generating $(FOUND_C_SOURCES) from a `$(shell $(FIND_SOURCE_FILES))` invocation, an alternative design was considered which involved converting $(FIND_SOURCE_FILES) into $(SOURCE_FILES) which would hold a list of filenames from the $(FIND_SOURCE_FILES) invocation. We would simply filter `%.c` files into $(ALL_C_SOURCES). $(SOURCE_FILES) would then be passed directly to the etags, ctags and cscope commands. We can see from the following invocation $ git ls-files '*.[hcS]' '*.sh' ':!*[tp][0-9][0-9][0-9][0-9]*' ':!contrib' | wc -c 12779 that the number of characters in this list would pose a problem on platforms with short command-line length limits (such as CMD which has a max of 8191 characters). As a result, we don't perform this change. However, we can see that the same issue may apply when running Coccinelle since $(COCCI_SOURCES) is also a list of filenames: if ! echo $(COCCI_SOURCES) | xargs $$limit \ $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ >$@+ 2>$@.log; \ This is justified since platforms that support Coccinelle generally have reasonably long command-line length limits and so we are safe for the foreseeable future. Signed-off-by: Denton Liu <liu.denton@gmail.com> --- Makefile | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 20eb5c5c42..522342a46c 100644 --- a/Makefile +++ b/Makefile @@ -2805,12 +2805,8 @@ check: command-list.h exit 1; \ fi -C_SOURCES = $(patsubst %.o,%.c,$(C_OBJ)) -ifdef DC_SHA1_SUBMODULE -COCCI_SOURCES = $(filter-out sha1collisiondetection/%,$(C_SOURCES)) -else -COCCI_SOURCES = $(filter-out sha1dc/%,$(C_SOURCES)) -endif +FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) @echo ' ' SPATCH $<; \ -- 2.23.0 ^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources 2019-09-16 19:23 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu ` (3 preceding siblings ...) 2019-09-16 19:23 ` [PATCH v3 4/4] Makefile: run coccicheck on more source files Denton Liu @ 2019-09-16 20:57 ` Junio C Hamano 2019-09-17 8:18 ` SZEDER Gábor 4 siblings, 1 reply; 29+ messages in thread From: Junio C Hamano @ 2019-09-16 20:57 UTC (permalink / raw) To: Denton Liu Cc: Git Mailing List, Johannes Schindelin, SZEDER Gábor, Jeff King Queued; thanks. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources 2019-09-16 20:57 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Junio C Hamano @ 2019-09-17 8:18 ` SZEDER Gábor 2019-09-17 16:13 ` Junio C Hamano 0 siblings, 1 reply; 29+ messages in thread From: SZEDER Gábor @ 2019-09-17 8:18 UTC (permalink / raw) To: Junio C Hamano Cc: Denton Liu, Git Mailing List, Johannes Schindelin, Jeff King On Mon, Sep 16, 2019 at 01:57:14PM -0700, Junio C Hamano wrote: > Queued; thanks. Nit: it would be nicer to queue this series on top of 'dl/compat-cleanup', because 'make coccicheck' errors out suggesting transformations in 'compat/mingw.c' that have only been applied in that series. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources 2019-09-17 8:18 ` SZEDER Gábor @ 2019-09-17 16:13 ` Junio C Hamano 0 siblings, 0 replies; 29+ messages in thread From: Junio C Hamano @ 2019-09-17 16:13 UTC (permalink / raw) To: SZEDER Gábor Cc: Denton Liu, Git Mailing List, Johannes Schindelin, Jeff King SZEDER Gábor <szeder.dev@gmail.com> writes: > On Mon, Sep 16, 2019 at 01:57:14PM -0700, Junio C Hamano wrote: >> Queued; thanks. > > Nit: it would be nicer to queue this series on top of > 'dl/compat-cleanup', because 'make coccicheck' errors out suggesting > transformations in 'compat/mingw.c' that have only been applied in > that series. Yikes, yes I remember that you pointed it out already, yet I forgot. Will correct. Thanks. ^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2019-09-17 16:13 UTC | newest] Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-10 7:44 [PATCH 0/2] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-10 7:44 ` [PATCH 1/2] Makefile: define UPSTREAM_SOURCES Denton Liu 2019-09-10 7:44 ` [PATCH 2/2] Makefile: run coccicheck on more source files Denton Liu 2019-09-10 13:28 ` SZEDER Gábor 2019-09-10 16:07 ` Denton Liu 2019-09-10 14:18 ` SZEDER Gábor 2019-09-12 17:28 ` [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-12 17:28 ` [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES Denton Liu 2019-09-12 21:42 ` Junio C Hamano 2019-09-12 17:28 ` [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu 2019-09-12 18:18 ` Junio C Hamano 2019-09-13 12:05 ` SZEDER Gábor 2019-09-12 17:28 ` [PATCH v2 3/3] Makefile: run coccicheck on more source files Denton Liu 2019-09-12 18:40 ` Junio C Hamano 2019-09-13 11:49 ` SZEDER Gábor 2019-09-13 17:14 ` Denton Liu 2019-09-13 18:00 ` SZEDER Gábor 2019-09-13 21:38 ` Denton Liu 2019-09-13 17:38 ` Junio C Hamano 2019-09-16 19:23 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu 2019-09-16 19:23 ` [PATCH v3 1/4] Makefile: strip leading ./ in $(LIB_H) Denton Liu 2019-09-16 19:23 ` [PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES Denton Liu 2019-09-16 20:56 ` Junio C Hamano 2019-09-16 22:00 ` [PATCH] fixup! " Denton Liu 2019-09-16 19:23 ` [PATCH v3 3/4] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu 2019-09-16 19:23 ` [PATCH v3 4/4] Makefile: run coccicheck on more source files Denton Liu 2019-09-16 20:57 ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Junio C Hamano 2019-09-17 8:18 ` SZEDER Gábor 2019-09-17 16:13 ` Junio C Hamano
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).