git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / Atom feed
From: Denton Liu <liu.denton@gmail.com>
To: SZEDER Gábor <szeder.dev@gmail.com>
Cc: Junio C Hamano <gitster@pobox.com>,
	Git Mailing List <git@vger.kernel.org>,
	Johannes Schindelin <Johannes.Schindelin@gmx.de>,
	Jeff King <peff@peff.net>
Subject: Re: [PATCH v2 3/3] Makefile: run coccicheck on more source files
Date: Fri, 13 Sep 2019 14:38:18 -0700
Message-ID: <20190913213818.GA88412@dentonliu-ltm.internal.salesforce.com> (raw)
In-Reply-To: <20190913180014.GQ32087@szeder.dev>

On Fri, Sep 13, 2019 at 08:00:14PM +0200, SZEDER Gábor wrote:
> On Fri, Sep 13, 2019 at 10:14:01AM -0700, Denton Liu wrote:
> > On Fri, Sep 13, 2019 at 01:49:52PM +0200, SZEDER Gábor wrote:
> > > On Thu, Sep 12, 2019 at 11:40:36AM -0700, Junio C Hamano wrote:
> > > > Denton Liu <liu.denton@gmail.com> writes:
> > > > 
> > > > > +FIND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> > > > > +COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FIND_C_SOURCES))
> > > > 
> > > > The former is somewhat misnamed.  FIND_SOURCE_FILES is *not* a list
> > > > of source files---it is a procedure to list source files to its
> > > > standard output.  FIND_C_SOUCRES sounds as if it is a similar
> > > > procedure, which would be implemented much like
> > > > 
> > > > 	FIND_C_SOURCES = $(FIND_SOURCE_FILES) | sed -n -e '/\.c$/p'
> > > > 
> > > > but that is not what you did and that is not what you want to have.
> > > > Perhaps call it FOUND_C_SOURCES?
> > > > 
> > > > I wonder if we can get rid of FIND_SOURCE_FILES that is a mere
> > > > procedure and replace its use with a true list of source files.
> > > > Would it make the result more pleasant to work with?
> > > > 
> > > > Perhaps something like the attached patch, (which would come before
> > > > this entire thing as a clean-up, and removing the need for 2/3)?
> > > > 
> > > > I dunno.
> > > > 
> > > > Using a procedure whose output is fed to xargs has an advantage that
> > > > a platform with very short command line limit can still work with
> > > > many source files, but the way you create and use COCCI_SOURCES in
> > > > this patch would defeat that advantage anyway,
> > > 
> > > COCCI_SOURCES is only used as an input to 'xargs', so that advantage
> > > is not defeated.
> > 
> > I think it still does matter; the relevant snippet is as follows:
> > 
> > 	if ! echo $(COCCI_SOURCES) | xargs $$limit \
> > 		$(SPATCH) --sp-file $< $(SPATCH_FLAGS) \
> > 		>$@+ 2>$@.log; \
> > 
> > which means that a really big COCCI_SOURCES could exceed the limit.
> 
> Oh, you're both right.
> 
> > That being said, COCCI_SOURCES should be smaller than the future
> > SOURCE_FILES variable since we're only taking %.c files (and filtering
> > out some of them too!).
> 
> We could also argue that Coccinelle only runs on platforms that have a
> reasonably large command line arg limit, and the number of our source
> files is way below that, so it won't matter in the foreseeable future.

Good point.

> 
> (Furthermore, 'echo' is often a shell builtin command, and I don't
> think that the platform's argument size limit applies to them.  At
> least the 'echo' of dash, Bash, ksh, ksh93, mksh, and BusyBox sh can
> deal with at least 10 million arguments; the platform limit is
> somewhere around 147k)
> 
> > > > diff --git a/Makefile b/Makefile
> > > > index f9255344ae..9dddd0e88c 100644
> > > > --- a/Makefile
> > > > +++ b/Makefile
> > > > @@ -2584,7 +2584,7 @@ perl/build/man/man3/Git.3pm: perl/Git.pm
> > > >  	$(QUIET_GEN)mkdir -p $(dir $@) && \
> > > >  	pod2man $< $@
> > > >  
> > > > -FIND_SOURCE_FILES = ( \
> > > > +SOURCE_FILES = $(patsubst ./%,%,$(shell \
> > > >  	git ls-files \
> > > >  		'*.[hcS]' \
> > > >  		'*.sh' \
> > > > @@ -2599,19 +2599,19 @@ FIND_SOURCE_FILES = ( \
> > > >  		-o \( -name 'trash*' -type d -prune \) \
> > > >  		-o \( -name '*.[hcS]' -type f -print \) \
> > > >  		-o \( -name '*.sh' -type f -print \) \
> > > > -	)
> > > > +	))
> > > >  
> > > >  $(ETAGS_TARGET): FORCE
> > > >  	$(RM) $(ETAGS_TARGET)
> > > > -	$(FIND_SOURCE_FILES) | xargs etags -a -o $(ETAGS_TARGET)
> > > > +	etags -a -o $(ETAGS_TARGET) $(SOURCE_FILES)
> > > >  
> > > >  tags: FORCE
> > > >  	$(RM) tags
> > > > -	$(FIND_SOURCE_FILES) | xargs ctags -a
> > > > +	ctags -a $(SOURCE_FILES)
> > > >  
> > > >  cscope:
> > > >  	$(RM) cscope*
> > > > -	$(FIND_SOURCE_FILES) | xargs cscope -b
> > > > +	cscope -b $(SOURCE_FILES)
> 
> Here, however, the list of source files is passed as argument to
> non-builtin commands, that also might be used on
> cmdline-arg-limit-challenged platforms.
> 

After doing a bit of research, I think that I agree with you. It seems
like the max command-line length for CMD on Windows is 8191 characters.

However, after running the following,

	$ git ls-files '*.[hcS]' '*.sh' ':!*[tp][0-9][0-9][0-9][0-9]*' ':!contrib' | wc -c
	   12779

we can see that the command-line length would definitely exceed the max
length so xargs would be required. As a result, we should probably just
keep the existing xargs invocations.

  reply index

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-10  7:44 [PATCH 0/2] Makefile: run coccicheck on all non-upstream sources Denton Liu
2019-09-10  7:44 ` [PATCH 1/2] Makefile: define UPSTREAM_SOURCES Denton Liu
2019-09-10  7:44 ` [PATCH 2/2] Makefile: run coccicheck on more source files Denton Liu
2019-09-10 13:28   ` SZEDER Gábor
2019-09-10 16:07     ` Denton Liu
2019-09-10 14:18   ` SZEDER Gábor
2019-09-12 17:28 ` [PATCH v2 0/3] Makefile: run coccicheck on all non-upstream sources Denton Liu
2019-09-12 17:28   ` [PATCH v2 1/3] Makefile: define THIRD_PARTY_SOURCES Denton Liu
2019-09-12 21:42     ` Junio C Hamano
2019-09-12 17:28   ` [PATCH v2 2/3] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu
2019-09-12 18:18     ` Junio C Hamano
2019-09-13 12:05     ` SZEDER Gábor
2019-09-12 17:28   ` [PATCH v2 3/3] Makefile: run coccicheck on more source files Denton Liu
2019-09-12 18:40     ` Junio C Hamano
2019-09-13 11:49       ` SZEDER Gábor
2019-09-13 17:14         ` Denton Liu
2019-09-13 18:00           ` SZEDER Gábor
2019-09-13 21:38             ` Denton Liu [this message]
2019-09-13 17:38         ` Junio C Hamano
2019-09-16 19:23   ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Denton Liu
2019-09-16 19:23     ` [PATCH v3 1/4] Makefile: strip leading ./ in $(LIB_H) Denton Liu
2019-09-16 19:23     ` [PATCH v3 2/4] Makefile: define THIRD_PARTY_SOURCES Denton Liu
2019-09-16 20:56       ` Junio C Hamano
2019-09-16 22:00         ` [PATCH] fixup! " Denton Liu
2019-09-16 19:23     ` [PATCH v3 3/4] Makefile: strip leading ./ in $(FIND_SOURCE_FILES) Denton Liu
2019-09-16 19:23     ` [PATCH v3 4/4] Makefile: run coccicheck on more source files Denton Liu
2019-09-16 20:57     ` [PATCH v3 0/4] Makefile: run coccicheck on all non-upstream sources Junio C Hamano

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190913213818.GA88412@dentonliu-ltm.internal.salesforce.com \
    --to=liu.denton@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=peff@peff.net \
    --cc=szeder.dev@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org list mirror (unofficial, one of many)

Archives are clonable:
	git clone --mirror http://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox