git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / Atom feed
* Weird behavior with git grep --recurse-submodules
@ 2019-07-08  8:14 Daniel Zaoui
  2019-07-10  6:43 ` Matheus Tavares Bernardino
  0 siblings, 1 reply; 14+ messages in thread
From: Daniel Zaoui @ 2019-07-08  8:14 UTC (permalink / raw)
  To: git

Hi guys,

I work with submodules and use git grep a lot.

I noted that when it is invoked used with --recurse-submodules, the result is not as expected for the submodules. I get submodules results as if no files were modified (like --cached option) although I would expect results taking into account the modifications.

Expected behavior:
git grep --recurse-submodules string:
- git grep string // search into main repo
- for each submodule, git grep string // search into submodule

Actual behavior:
git grep --recurse-submodules string:
- git grep string // search into main repo
- for each submodule, git grep --cached string // search into submodule

Do you get the same behavior? Am I doing something wrong? Was I understandable :-)? Is it a bug?

git --version: git version 2.22.0
uname -a: Linux daniel 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019 x86_64 GNU/Linux

Thanks
Daniel

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Weird behavior with git grep --recurse-submodules
  2019-07-08  8:14 Weird behavior with git grep --recurse-submodules Daniel Zaoui
@ 2019-07-10  6:43 ` Matheus Tavares Bernardino
  2019-07-10 11:14   ` Johannes Schindelin
  0 siblings, 1 reply; 14+ messages in thread
From: Matheus Tavares Bernardino @ 2019-07-10  6:43 UTC (permalink / raw)
  To: Daniel Zaoui; +Cc: git, Brandon Williams

On Mon, Jul 8, 2019 at 5:22 AM Daniel Zaoui <jackdanielz@eyomi.org> wrote:
>
> Hi guys,

Hi, Daniel

> I work with submodules and use git grep a lot.
>
> I noted that when it is invoked used with --recurse-submodules, the result is not as expected for the submodules. I get submodules results as if no files were modified (like --cached option) although I would expect results taking into account the modifications.
>
> Expected behavior:
> git grep --recurse-submodules string:
> - git grep string // search into main repo
> - for each submodule, git grep string // search into submodule
>
> Actual behavior:
> git grep --recurse-submodules string:
> - git grep string // search into main repo
> - for each submodule, git grep --cached string // search into submodule
>
> Do you get the same behavior? Am I doing something wrong? Was I understandable :-)? Is it a bug?

It seems git-grep was taking into account the worktree modifications
in submodules before f9ee2fc ("grep: recurse in-process using 'struct
repository'", 02-08-2017). I'm not sure, thought, if this behavior
change was a bug during the conversion or a project decision.

CC-ing Brandon, in case he has other inputs

> git --version: git version 2.22.0
> uname -a: Linux daniel 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019 x86_64 GNU/Linux
>
> Thanks
> Daniel

Best,
Matheus

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Weird behavior with git grep --recurse-submodules
  2019-07-10  6:43 ` Matheus Tavares Bernardino
@ 2019-07-10 11:14   ` Johannes Schindelin
  2019-07-16 18:10     ` Daniel Zaoui
  0 siblings, 1 reply; 14+ messages in thread
From: Johannes Schindelin @ 2019-07-10 11:14 UTC (permalink / raw)
  To: Matheus Tavares Bernardino; +Cc: Brandon Williams, Daniel Zaoui, git

[cC:ing Brandon via his current email address, as per .mailmap]


On Wed, 10 Jul 2019, Matheus Tavares Bernardino wrote:

> On Mon, Jul 8, 2019 at 5:22 AM Daniel Zaoui <jackdanielz@eyomi.org> wrote:
> >
> > Hi guys,
>
> Hi, Daniel
>
> > I work with submodules and use git grep a lot.
> >
> > I noted that when it is invoked used with --recurse-submodules, the result is not as expected for the submodules. I get submodules results as if no files were modified (like --cached option) although I would expect results taking into account the modifications.
> >
> > Expected behavior:
> > git grep --recurse-submodules string:
> > - git grep string // search into main repo
> > - for each submodule, git grep string // search into submodule
> >
> > Actual behavior:
> > git grep --recurse-submodules string:
> > - git grep string // search into main repo
> > - for each submodule, git grep --cached string // search into submodule
> >
> > Do you get the same behavior? Am I doing something wrong? Was I understandable :-)? Is it a bug?
>
> It seems git-grep was taking into account the worktree modifications
> in submodules before f9ee2fc ("grep: recurse in-process using 'struct
> repository'", 02-08-2017). I'm not sure, thought, if this behavior
> change was a bug during the conversion or a project decision.
>
> CC-ing Brandon, in case he has other inputs
>
> > git --version: git version 2.22.0
> > uname -a: Linux daniel 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019 x86_64 GNU/Linux
> >
> > Thanks
> > Daniel
>
> Best,
> Matheus
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Weird behavior with git grep --recurse-submodules
  2019-07-10 11:14   ` Johannes Schindelin
@ 2019-07-16 18:10     ` Daniel Zaoui
  2019-07-29 20:27       ` Matheus Tavares Bernardino
  0 siblings, 1 reply; 14+ messages in thread
From: Daniel Zaoui @ 2019-07-16 18:10 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: Matheus Tavares Bernardino, Brandon Williams, git

Hi Matheus,

Thank you for your response.

I really hope the change Brandon made is not a project decision. At least, it does seem to me like a bug.

How do you recommend me to solve this issue? Is there some place where I can check if some bug ticket has been created on this matter? I didn't find anything in the mailing list archives about this.

BR
Daniel

On Wed, 10 Jul 2019 13:14:18 +0200 (CEST)
Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

> [cC:ing Brandon via his current email address, as per .mailmap]
> 
> 
> On Wed, 10 Jul 2019, Matheus Tavares Bernardino wrote:
> 
>  [...]  
>  [...]  
>  [...]  
>  [...]  
>  [...]  
>  [...]  
>  [...]  


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Weird behavior with git grep --recurse-submodules
  2019-07-16 18:10     ` Daniel Zaoui
@ 2019-07-29 20:27       ` Matheus Tavares Bernardino
  2019-07-30 16:53         ` [GSoC][PATCH] grep: fix worktree case in submodules Matheus Tavares
  2019-08-03 23:39         ` Weird behavior with git grep --recurse-submodules Brandon Williams
  0 siblings, 2 replies; 14+ messages in thread
From: Matheus Tavares Bernardino @ 2019-07-29 20:27 UTC (permalink / raw)
  To: Daniel Zaoui; +Cc: Johannes Schindelin, Brandon Williams, git

On Tue, Jul 16, 2019 at 3:09 PM Daniel Zaoui <jackdanielz@eyomi.org> wrote:
>
> Hi Matheus,

Hi, Daniel

I'm sorry, your last message went to my spam folder for some reason :(

> Thank you for your response.
>
> I really hope the change Brandon made is not a project decision. At least, it does seem to me like a bug.
>
> How do you recommend me to solve this issue? Is there some place where I can check if some bug ticket has been created on this matter? I didn't find anything in the mailing list archives about this.

I think I manage to solve the bug with this[1] patch. I'm going to ask
my GSoC mentors to take a look and then I'll send it to the mailing
list :) Thanks for reporting it.

Best,
Matheus

[1]: https://github.com/matheustavares/git/commit/37aeeb3ab86b5dfebfaafeb98d34e379341a529d

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [GSoC][PATCH] grep: fix worktree case in submodules
  2019-07-29 20:27       ` Matheus Tavares Bernardino
@ 2019-07-30 16:53         ` Matheus Tavares
  2019-07-30 20:04           ` Junio C Hamano
  2019-07-31 15:35           ` [GSoC][PATCH v2] " Matheus Tavares
  2019-08-03 23:39         ` Weird behavior with git grep --recurse-submodules Brandon Williams
  1 sibling, 2 replies; 14+ messages in thread
From: Matheus Tavares @ 2019-07-30 16:53 UTC (permalink / raw)
  To: git
  Cc: Christian Couder, Olga Telezhnaya, kernel-usp, jackdanielz,
	Antonio Ospite, Stefan Beller,
	Nguyễn Thái Ngọc Duy, Jonathan Tan,
	Brandon Williams, Junio C Hamano

Running git-grep with --recurse-submodules results in a cached grep for
the submodules even when --cached is not used. This makes all
modifications in submodules' tracked files be always ignored when
grepping. Solve that making git-grep respect the cached option when
invoking grep_cache() inside grep_submodule(). Also, add tests to
ensure that the desired behavior is performed.

Reported-by: Daniel Zaoui <jackdanielz@eyomi.org>
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
 builtin/grep.c                     | 10 ++++++----
 t/t7814-grep-recurse-submodules.sh | 21 +++++++++++++++++++++
 2 files changed, 27 insertions(+), 4 deletions(-)

diff --git a/builtin/grep.c b/builtin/grep.c
index 560051784e..2699001fbd 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -403,7 +403,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 static int grep_submodule(struct grep_opt *opt,
 			  const struct pathspec *pathspec,
 			  const struct object_id *oid,
-			  const char *filename, const char *path)
+			  const char *filename, const char *path, int cached)
 {
 	struct repository subrepo;
 	struct repository *superproject = opt->repo;
@@ -475,7 +475,7 @@ static int grep_submodule(struct grep_opt *opt,
 		strbuf_release(&base);
 		free(data);
 	} else {
-		hit = grep_cache(&subopt, pathspec, 1);
+		hit = grep_cache(&subopt, pathspec, cached);
 	}
 
 	repo_clear(&subrepo);
@@ -523,7 +523,8 @@ static int grep_cache(struct grep_opt *opt,
 			}
 		} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&
 			   submodule_path_match(repo->index, pathspec, name.buf, NULL)) {
-			hit |= grep_submodule(opt, pathspec, NULL, ce->name, ce->name);
+			hit |= grep_submodule(opt, pathspec, NULL, ce->name,
+					      ce->name, cached);
 		} else {
 			continue;
 		}
@@ -598,7 +599,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 			free(data);
 		} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {
 			hit |= grep_submodule(opt, pathspec, &entry.oid,
-					      base->buf, base->buf + tn_len);
+					      base->buf, base->buf + tn_len,
+					      1); /* ignored */
 		}
 
 		strbuf_setlen(base, old_baselen);
diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh
index a11366b4ce..946f91fa57 100755
--- a/t/t7814-grep-recurse-submodules.sh
+++ b/t/t7814-grep-recurse-submodules.sh
@@ -408,4 +408,25 @@ test_expect_success 'grep --recurse-submodules with submodules without .gitmodul
 	test_cmp expect actual
 '
 
+reset_and_clean () {
+	git reset --hard &&
+	git clean -fd &&
+	git submodule foreach --recursive 'git reset --hard' &&
+	git submodule foreach --recursive 'git clean -fd'
+}
+
+test_expect_success 'grep --recurse-submodules without --cached considers worktree modifications' '
+	reset_and_clean &&
+	echo "A modified line in submodule" >>submodule/a &&
+	echo "submodule/a:A modified line in submodule" >expect &&
+	git grep --recurse-submodules "A modified line in submodule" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'grep --recurse-submodules with --cached ignores worktree modifications' '
+	reset_and_clean &&
+	echo "A modified line in submodule" >>submodule/a &&
+	test_must_fail git grep --recurse-submodules --cached "A modified line in submodule" >actual 2>&1 &&
+	test_must_be_empty actual
+'
 test_done
-- 
2.22.0


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [GSoC][PATCH] grep: fix worktree case in submodules
  2019-07-30 16:53         ` [GSoC][PATCH] grep: fix worktree case in submodules Matheus Tavares
@ 2019-07-30 20:04           ` Junio C Hamano
  2019-07-30 22:02             ` Christian Couder
  2019-07-30 23:40             ` Matheus Tavares Bernardino
  2019-07-31 15:35           ` [GSoC][PATCH v2] " Matheus Tavares
  1 sibling, 2 replies; 14+ messages in thread
From: Junio C Hamano @ 2019-07-30 20:04 UTC (permalink / raw)
  To: Matheus Tavares
  Cc: git, Christian Couder, Olga Telezhnaya, kernel-usp, jackdanielz,
	Antonio Ospite, Stefan Beller,
	Nguyễn Thái Ngọc Duy, Jonathan Tan,
	Brandon Williams

Matheus Tavares <matheus.bernardino@usp.br> writes:

> @@ -475,7 +475,7 @@ static int grep_submodule(struct grep_opt *opt,
>  		strbuf_release(&base);
>  		free(data);
>  	} else {
> -		hit = grep_cache(&subopt, pathspec, 1);
> +		hit = grep_cache(&subopt, pathspec, cached);
>  	}

Interesting.  It appears to me that this has always searched in
submodule index and never working tree.  I am not sure if there was
any specific reason to avoid looking into the working tree files.

Passing the 'cached' bit down from grep_cache() does look like a
sensible way to go.

> @@ -523,7 +523,8 @@ static int grep_cache(struct grep_opt *opt,
>  			}
>  		} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&
>  			   submodule_path_match(repo->index, pathspec, name.buf, NULL)) {
> -			hit |= grep_submodule(opt, pathspec, NULL, ce->name, ce->name);
> +			hit |= grep_submodule(opt, pathspec, NULL, ce->name,
> +					      ce->name, cached);
>  		} else {
>  			continue;
>  		}
> @@ -598,7 +599,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
>  			free(data);
>  		} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {
>  			hit |= grep_submodule(opt, pathspec, &entry.oid,
> -					      base->buf, base->buf + tn_len);
> +					      base->buf, base->buf + tn_len,
> +					      1); /* ignored */

The trailing comment is misleading.  In the context of reviewing
this patch, we can probably tell it applies only to that "1", but
if you read only the postimage, the "ignored" comment looks as if
the call itself is somehow ignored by somebody unspecified.  It is
not clear at all that it is only about the final parameter.

If you must...

		hit |= grep_submodule(opt, pathspec, &entry.oid,
				      base->buf, base->buf + tn_len,
				      1 /* ignored */);

... is a reasonable way to write it.

> diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh
> index a11366b4ce..946f91fa57 100755
> --- a/t/t7814-grep-recurse-submodules.sh
> +++ b/t/t7814-grep-recurse-submodules.sh
> @@ -408,4 +408,25 @@ test_expect_success 'grep --recurse-submodules with submodules without .gitmodul
>  	test_cmp expect actual
>  '
>  
> +reset_and_clean () {
> +	git reset --hard &&
> +	git clean -fd &&
> +	git submodule foreach --recursive 'git reset --hard' &&
> +	git submodule foreach --recursive 'git clean -fd'

Do we need two separate foreach, instread of a single one that does
both reset and clean (i.e. "git reset --hard && git clean -f -d")?


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [GSoC][PATCH] grep: fix worktree case in submodules
  2019-07-30 20:04           ` Junio C Hamano
@ 2019-07-30 22:02             ` Christian Couder
  2019-07-31 15:57               ` Junio C Hamano
  2019-07-30 23:40             ` Matheus Tavares Bernardino
  1 sibling, 1 reply; 14+ messages in thread
From: Christian Couder @ 2019-07-30 22:02 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: Matheus Tavares, git, Olga Telezhnaya, kernel-usp, jackdanielz,
	Antonio Ospite, Stefan Beller,
	Nguyễn Thái Ngọc Duy, Jonathan Tan,
	Brandon Williams

On Tue, Jul 30, 2019 at 10:04 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Matheus Tavares <matheus.bernardino@usp.br> writes:

> > @@ -598,7 +599,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
> >                       free(data);
> >               } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {
> >                       hit |= grep_submodule(opt, pathspec, &entry.oid,
> > -                                           base->buf, base->buf + tn_len);
> > +                                           base->buf, base->buf + tn_len,
> > +                                           1); /* ignored */
>
> The trailing comment is misleading.  In the context of reviewing
> this patch, we can probably tell it applies only to that "1", but
> if you read only the postimage, the "ignored" comment looks as if
> the call itself is somehow ignored by somebody unspecified.  It is
> not clear at all that it is only about the final parameter.
>
> If you must...
>
>                 hit |= grep_submodule(opt, pathspec, &entry.oid,
>                                       base->buf, base->buf + tn_len,
>                                       1 /* ignored */);

Yeah, I suggested adding an "/* ignored */" comment, but I was indeed
thinking about something like this.

> ... is a reasonable way to write it.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [GSoC][PATCH] grep: fix worktree case in submodules
  2019-07-30 20:04           ` Junio C Hamano
  2019-07-30 22:02             ` Christian Couder
@ 2019-07-30 23:40             ` Matheus Tavares Bernardino
  1 sibling, 0 replies; 14+ messages in thread
From: Matheus Tavares Bernardino @ 2019-07-30 23:40 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: git, Christian Couder, Olga Telezhnaya, Kernel USP, Daniel Zaoui,
	Antonio Ospite, Stefan Beller,
	Nguyễn Thái Ngọc Duy, Jonathan Tan,
	Brandon Williams

On Tue, Jul 30, 2019 at 5:04 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Matheus Tavares <matheus.bernardino@usp.br> writes:
>
> > @@ -475,7 +475,7 @@ static int grep_submodule(struct grep_opt *opt,
> >               strbuf_release(&base);
> >               free(data);
> >       } else {
> > -             hit = grep_cache(&subopt, pathspec, 1);
> > +             hit = grep_cache(&subopt, pathspec, cached);
> >       }
>
> Interesting.  It appears to me that this has always searched in
> submodule index and never working tree.  I am not sure if there was
> any specific reason to avoid looking into the working tree files.

It seems that git-grep was taking the worktree into account before
f9ee2fc ("grep: recurse in-process using 'struct repository'",
02-08-2017). So maybe it was just a mistake during the in-process
conversion.

> Passing the 'cached' bit down from grep_cache() does look like a
> sensible way to go.
>
> > @@ -523,7 +523,8 @@ static int grep_cache(struct grep_opt *opt,
> >                       }
> >               } else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&
> >                          submodule_path_match(repo->index, pathspec, name.buf, NULL)) {
> > -                     hit |= grep_submodule(opt, pathspec, NULL, ce->name, ce->name);
> > +                     hit |= grep_submodule(opt, pathspec, NULL, ce->name,
> > +                                           ce->name, cached);
> >               } else {
> >                       continue;
> >               }
> > @@ -598,7 +599,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
> >                       free(data);
> >               } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {
> >                       hit |= grep_submodule(opt, pathspec, &entry.oid,
> > -                                           base->buf, base->buf + tn_len);
> > +                                           base->buf, base->buf + tn_len,
> > +                                           1); /* ignored */
>
> The trailing comment is misleading.  In the context of reviewing
> this patch, we can probably tell it applies only to that "1", but
> if you read only the postimage, the "ignored" comment looks as if
> the call itself is somehow ignored by somebody unspecified.  It is
> not clear at all that it is only about the final parameter.
>
> If you must...
>
>                 hit |= grep_submodule(opt, pathspec, &entry.oid,
>                                       base->buf, base->buf + tn_len,
>                                       1 /* ignored */);
>
> ... is a reasonable way to write it.

Right, thanks.

> > diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh
> > index a11366b4ce..946f91fa57 100755
> > --- a/t/t7814-grep-recurse-submodules.sh
> > +++ b/t/t7814-grep-recurse-submodules.sh
> > @@ -408,4 +408,25 @@ test_expect_success 'grep --recurse-submodules with submodules without .gitmodul
> >       test_cmp expect actual
> >  '
> >
> > +reset_and_clean () {
> > +     git reset --hard &&
> > +     git clean -fd &&
> > +     git submodule foreach --recursive 'git reset --hard' &&
> > +     git submodule foreach --recursive 'git clean -fd'
>
> Do we need two separate foreach, instread of a single one that does
> both reset and clean (i.e. "git reset --hard && git clean -f -d")?

Indeed! Thanks. I'll send a v2 addressing both comments.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [GSoC][PATCH v2] grep: fix worktree case in submodules
  2019-07-30 16:53         ` [GSoC][PATCH] grep: fix worktree case in submodules Matheus Tavares
  2019-07-30 20:04           ` Junio C Hamano
@ 2019-07-31 15:35           ` " Matheus Tavares
  2019-08-01  3:13             ` [GSoC][PATCH v3] " Matheus Tavares
  1 sibling, 1 reply; 14+ messages in thread
From: Matheus Tavares @ 2019-07-31 15:35 UTC (permalink / raw)
  To: git
  Cc: Christian Couder, Olga Telezhnaya, kernel-usp, jackdanielz,
	Junio C Hamano

Running git-grep with --recurse-submodules results in a cached grep for
the submodules even when --cached is not used. This makes all
modifications in submodules' tracked files be always ignored when
grepping. Solve that making git-grep respect the cached option when
invoking grep_cache() inside grep_submodule(). Also, add tests to
ensure that the desired behavior is performed.

Reported-by: Daniel Zaoui <jackdanielz@eyomi.org>
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
Changes in v2:
- repositioned the '/* ignored */' comment to avoid ambiguity
- joined `git clean` and `git reset` invokations in a single `git
  submodule foreach`. 

travis build: https://travis-ci.org/matheustavares/git/builds/565749070

builtin/grep.c                     | 10 ++++++----
 t/t7814-grep-recurse-submodules.sh | 20 ++++++++++++++++++++
 2 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/builtin/grep.c b/builtin/grep.c
index 560051784e..d9866dd936 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -403,7 +403,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 static int grep_submodule(struct grep_opt *opt,
 			  const struct pathspec *pathspec,
 			  const struct object_id *oid,
-			  const char *filename, const char *path)
+			  const char *filename, const char *path, int cached)
 {
 	struct repository subrepo;
 	struct repository *superproject = opt->repo;
@@ -475,7 +475,7 @@ static int grep_submodule(struct grep_opt *opt,
 		strbuf_release(&base);
 		free(data);
 	} else {
-		hit = grep_cache(&subopt, pathspec, 1);
+		hit = grep_cache(&subopt, pathspec, cached);
 	}
 
 	repo_clear(&subrepo);
@@ -523,7 +523,8 @@ static int grep_cache(struct grep_opt *opt,
 			}
 		} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&
 			   submodule_path_match(repo->index, pathspec, name.buf, NULL)) {
-			hit |= grep_submodule(opt, pathspec, NULL, ce->name, ce->name);
+			hit |= grep_submodule(opt, pathspec, NULL, ce->name,
+					      ce->name, cached);
 		} else {
 			continue;
 		}
@@ -598,7 +599,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 			free(data);
 		} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {
 			hit |= grep_submodule(opt, pathspec, &entry.oid,
-					      base->buf, base->buf + tn_len);
+					      base->buf, base->buf + tn_len,
+					      1 /* ignored */);
 		}
 
 		strbuf_setlen(base, old_baselen);
diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh
index a11366b4ce..edb64690e6 100755
--- a/t/t7814-grep-recurse-submodules.sh
+++ b/t/t7814-grep-recurse-submodules.sh
@@ -408,4 +408,24 @@ test_expect_success 'grep --recurse-submodules with submodules without .gitmodul
 	test_cmp expect actual
 '
 
+reset_and_clean () {
+	git reset --hard &&
+	git clean -fd &&
+	git submodule foreach --recursive 'git reset --hard && git clean -fd'
+}
+
+test_expect_success 'grep --recurse-submodules without --cached considers worktree modifications' '
+	reset_and_clean &&
+	echo "A modified line in submodule" >>submodule/a &&
+	echo "submodule/a:A modified line in submodule" >expect &&
+	git grep --recurse-submodules "A modified line in submodule" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'grep --recurse-submodules with --cached ignores worktree modifications' '
+	reset_and_clean &&
+	echo "A modified line in submodule" >>submodule/a &&
+	test_must_fail git grep --recurse-submodules --cached "A modified line in submodule" >actual 2>&1 &&
+	test_must_be_empty actual
+'
 test_done
-- 
2.22.0


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [GSoC][PATCH] grep: fix worktree case in submodules
  2019-07-30 22:02             ` Christian Couder
@ 2019-07-31 15:57               ` Junio C Hamano
  2019-08-01  3:08                 ` Matheus Tavares Bernardino
  0 siblings, 1 reply; 14+ messages in thread
From: Junio C Hamano @ 2019-07-31 15:57 UTC (permalink / raw)
  To: Christian Couder
  Cc: Matheus Tavares, git, Olga Telezhnaya, kernel-usp, jackdanielz,
	Antonio Ospite, Stefan Beller,
	Nguyễn Thái Ngọc Duy, Jonathan Tan,
	Brandon Williams

Christian Couder <christian.couder@gmail.com> writes:

> On Tue, Jul 30, 2019 at 10:04 PM Junio C Hamano <gitster@pobox.com> wrote:
>>
>> Matheus Tavares <matheus.bernardino@usp.br> writes:
>
>> > @@ -598,7 +599,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
>> >                       free(data);
>> >               } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {
>> >                       hit |= grep_submodule(opt, pathspec, &entry.oid,
>> > -                                           base->buf, base->buf + tn_len);
>> > +                                           base->buf, base->buf + tn_len,
>> > +                                           1); /* ignored */
>>
>> The trailing comment is misleading.  In the context of reviewing
>> this patch, we can probably tell it applies only to that "1", but
>> if you read only the postimage, the "ignored" comment looks as if
>> the call itself is somehow ignored by somebody unspecified.  It is
>> not clear at all that it is only about the final parameter.
>>
>> If you must...
>>
>>                 hit |= grep_submodule(opt, pathspec, &entry.oid,
>>                                       base->buf, base->buf + tn_len,
>>                                       1 /* ignored */);
>
> Yeah, I suggested adding an "/* ignored */" comment, but I was indeed
> thinking about something like this.
>
>> ... is a reasonable way to write it.

Thanks.  In this case, I am not sure if the comment here really
helps.  If anything, shouldn't there be a comment near the top of
grep_submodule() that says 'cached bit is meaningful only when you
feed an empty oid, aka "not grepping inside a tree object"'?


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [GSoC][PATCH] grep: fix worktree case in submodules
  2019-07-31 15:57               ` Junio C Hamano
@ 2019-08-01  3:08                 ` Matheus Tavares Bernardino
  0 siblings, 0 replies; 14+ messages in thread
From: Matheus Tavares Bernardino @ 2019-08-01  3:08 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: Christian Couder, git, Olga Telezhnaya, Kernel USP, Daniel Zaoui,
	Antonio Ospite, Stefan Beller,
	Nguyễn Thái Ngọc Duy, Jonathan Tan,
	Brandon Williams

On Wed, Jul 31, 2019 at 12:57 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Christian Couder <christian.couder@gmail.com> writes:
>
> > On Tue, Jul 30, 2019 at 10:04 PM Junio C Hamano <gitster@pobox.com> wrote:
> >>
> >> Matheus Tavares <matheus.bernardino@usp.br> writes:
> >
> >> > @@ -598,7 +599,8 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
> >> >                       free(data);
> >> >               } else if (recurse_submodules && S_ISGITLINK(entry.mode)) {
> >> >                       hit |= grep_submodule(opt, pathspec, &entry.oid,
> >> > -                                           base->buf, base->buf + tn_len);
> >> > +                                           base->buf, base->buf + tn_len,
> >> > +                                           1); /* ignored */
> >>
> >> The trailing comment is misleading.  In the context of reviewing
> >> this patch, we can probably tell it applies only to that "1", but
> >> if you read only the postimage, the "ignored" comment looks as if
> >> the call itself is somehow ignored by somebody unspecified.  It is
> >> not clear at all that it is only about the final parameter.
> >>
> >> If you must...
> >>
> >>                 hit |= grep_submodule(opt, pathspec, &entry.oid,
> >>                                       base->buf, base->buf + tn_len,
> >>                                       1 /* ignored */);
> >
> > Yeah, I suggested adding an "/* ignored */" comment, but I was indeed
> > thinking about something like this.
> >
> >> ... is a reasonable way to write it.
>
> Thanks.  In this case, I am not sure if the comment here really
> helps.  If anything, shouldn't there be a comment near the top of
> grep_submodule() that says 'cached bit is meaningful only when you
> feed an empty oid, aka "not grepping inside a tree object"'?

Right, it makes sense. I'll add that.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [GSoC][PATCH v3] grep: fix worktree case in submodules
  2019-07-31 15:35           ` [GSoC][PATCH v2] " Matheus Tavares
@ 2019-08-01  3:13             ` " Matheus Tavares
  0 siblings, 0 replies; 14+ messages in thread
From: Matheus Tavares @ 2019-08-01  3:13 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: git, Christian Couder, Olga Telezhnaya, kernel-usp, jackdanielz

Running git-grep with --recurse-submodules results in a cached grep for
the submodules even when --cached is not used. This makes all
modifications in submodules' tracked files be always ignored when
grepping. Solve that making git-grep respect the cached option when
invoking grep_cache() inside grep_submodule(). Also, add tests to
ensure that the desired behavior is performed.

Reported-by: Daniel Zaoui <jackdanielz@eyomi.org>
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
Changes in v3:
- Replaced the "\* ignored *\" comment by a more meaningful note at the
  top of grep_submodule()

builtin/grep.c                     | 13 +++++++++----
 t/t7814-grep-recurse-submodules.sh | 20 ++++++++++++++++++++
 2 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/builtin/grep.c b/builtin/grep.c
index 560051784e..df8cdecdae 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -400,10 +400,14 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 		     struct tree_desc *tree, struct strbuf *base, int tn_len,
 		     int check_attr);
 
+/*
+ * The cached bit is only meaningful when a NULL oid is given (i.e. when not
+ * grepping inside a tree object).
+ */
 static int grep_submodule(struct grep_opt *opt,
 			  const struct pathspec *pathspec,
 			  const struct object_id *oid,
-			  const char *filename, const char *path)
+			  const char *filename, const char *path, int cached)
 {
 	struct repository subrepo;
 	struct repository *superproject = opt->repo;
@@ -475,7 +479,7 @@ static int grep_submodule(struct grep_opt *opt,
 		strbuf_release(&base);
 		free(data);
 	} else {
-		hit = grep_cache(&subopt, pathspec, 1);
+		hit = grep_cache(&subopt, pathspec, cached);
 	}
 
 	repo_clear(&subrepo);
@@ -523,7 +527,8 @@ static int grep_cache(struct grep_opt *opt,
 			}
 		} else if (recurse_submodules && S_ISGITLINK(ce->ce_mode) &&
 			   submodule_path_match(repo->index, pathspec, name.buf, NULL)) {
-			hit |= grep_submodule(opt, pathspec, NULL, ce->name, ce->name);
+			hit |= grep_submodule(opt, pathspec, NULL, ce->name,
+					      ce->name, cached);
 		} else {
 			continue;
 		}
@@ -598,7 +603,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 			free(data);
 		} else if (recurse_submodules && S_ISGITLINK(entry.mode)) {
 			hit |= grep_submodule(opt, pathspec, &entry.oid,
-					      base->buf, base->buf + tn_len);
+					      base->buf, base->buf + tn_len, 1);
 		}
 
 		strbuf_setlen(base, old_baselen);
diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh
index a11366b4ce..edb64690e6 100755
--- a/t/t7814-grep-recurse-submodules.sh
+++ b/t/t7814-grep-recurse-submodules.sh
@@ -408,4 +408,24 @@ test_expect_success 'grep --recurse-submodules with submodules without .gitmodul
 	test_cmp expect actual
 '
 
+reset_and_clean () {
+	git reset --hard &&
+	git clean -fd &&
+	git submodule foreach --recursive 'git reset --hard && git clean -fd'
+}
+
+test_expect_success 'grep --recurse-submodules without --cached considers worktree modifications' '
+	reset_and_clean &&
+	echo "A modified line in submodule" >>submodule/a &&
+	echo "submodule/a:A modified line in submodule" >expect &&
+	git grep --recurse-submodules "A modified line in submodule" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'grep --recurse-submodules with --cached ignores worktree modifications' '
+	reset_and_clean &&
+	echo "A modified line in submodule" >>submodule/a &&
+	test_must_fail git grep --recurse-submodules --cached "A modified line in submodule" >actual 2>&1 &&
+	test_must_be_empty actual
+'
 test_done
-- 
2.22.0


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Weird behavior with git grep --recurse-submodules
  2019-07-29 20:27       ` Matheus Tavares Bernardino
  2019-07-30 16:53         ` [GSoC][PATCH] grep: fix worktree case in submodules Matheus Tavares
@ 2019-08-03 23:39         ` Brandon Williams
  1 sibling, 0 replies; 14+ messages in thread
From: Brandon Williams @ 2019-08-03 23:39 UTC (permalink / raw)
  To: Matheus Tavares Bernardino; +Cc: Daniel Zaoui, Johannes Schindelin, git

On Mon, Jul 29, 2019 at 1:27 PM Matheus Tavares Bernardino
<matheus.bernardino@usp.br> wrote:
>
> On Tue, Jul 16, 2019 at 3:09 PM Daniel Zaoui <jackdanielz@eyomi.org> wrote:
> >
> > Hi Matheus,
>
> Hi, Daniel
>
> I'm sorry, your last message went to my spam folder for some reason :(
>
> > Thank you for your response.
> >
> > I really hope the change Brandon made is not a project decision. At least, it does seem to me like a bug.
> >

Since I'm the one who originally implemented the --recurse-submodules
feature to git grep back in
0281e487f (grep: optionally recurse into submodules, 2016-12-16) and
then subsequently changed
the implementation to do the recursion in process in  f9ee2fc ("grep:
recurse in-process using
'struct repository'", 02-08-2017) I can say it was definitely a bug I
introduced during the conversion
as I don't believe I would have intended to introduce that change of
behavior (especially without
documenting it).

> > How do you recommend me to solve this issue? Is there some place where I can check if some bug ticket has been created on this matter? I didn't find anything in the mailing list archives about this.
>
> I think I manage to solve the bug with this[1] patch. I'm going to ask
> my GSoC mentors to take a look and then I'll send it to the mailing
> list :) Thanks for reporting it.
>
> Best,
> Matheus
>
> [1]: https://github.com/matheustavares/git/commit/37aeeb3ab86b5dfebfaafeb98d34e379341a529d

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, back to index

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-08  8:14 Weird behavior with git grep --recurse-submodules Daniel Zaoui
2019-07-10  6:43 ` Matheus Tavares Bernardino
2019-07-10 11:14   ` Johannes Schindelin
2019-07-16 18:10     ` Daniel Zaoui
2019-07-29 20:27       ` Matheus Tavares Bernardino
2019-07-30 16:53         ` [GSoC][PATCH] grep: fix worktree case in submodules Matheus Tavares
2019-07-30 20:04           ` Junio C Hamano
2019-07-30 22:02             ` Christian Couder
2019-07-31 15:57               ` Junio C Hamano
2019-08-01  3:08                 ` Matheus Tavares Bernardino
2019-07-30 23:40             ` Matheus Tavares Bernardino
2019-07-31 15:35           ` [GSoC][PATCH v2] " Matheus Tavares
2019-08-01  3:13             ` [GSoC][PATCH v3] " Matheus Tavares
2019-08-03 23:39         ` Weird behavior with git grep --recurse-submodules Brandon Williams

git@vger.kernel.org list mirror (unofficial, one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox