How to find out which gitignore blocks my git-add

How to find out which gitignore blocks my git-add

[Gonzo]

After doing a "git add file" I get the message:

"The following paths are ignored by one of your .gitignore files:
..."

Is there an easy way to find out which line in which gitignore file 
blocks this add?
Would this be a viable addition to "git add -v"?

g

Re: How to find out which gitignore blocks my git-add

[Jeff King]

On Fri, Feb 06, 2009 at 10:38:45AM +0100, Gonzo wrote:

> After doing a "git add file" I get the message:
>
> "The following paths are ignored by one of your .gitignore files:
> ..."
>
> Is there an easy way to find out which line in which gitignore file  
> blocks this add?

No, I don't think so.

> Would this be a viable addition to "git add -v"?

I think it might be useful to be able to get this information. However,
rather than coupling it with "git add", it might make more sense to have
a separate way to query "is this being ignored, and if so, by what
pattern". Then you could use that tool to generally debug your
.gitignore patterns.

I'm not sure how painful it would be to implement. You'd probably have
to record and pass back that information from dir.c:excluded, which is
where we decide whether or not a file is ignored (most of the code calls
it "excluded", but it is the same concept). Want to take a stab at
writing a patch?

-Peff

Re: How to find out which gitignore blocks my git-add

[Sitaram Chamarty]

On 2009-02-06, Jeff King <peff@peff.net> wrote:
> On Fri, Feb 06, 2009 at 10:38:45AM +0100, Gonzo wrote:
>> Is there an easy way to find out which line in which gitignore file  
>> blocks this add?
>
> No, I don't think so.
>
>> Would this be a viable addition to "git add -v"?
>
> I think it might be useful to be able to get this information. However,
> rather than coupling it with "git add", it might make more sense to have
> a separate way to query "is this being ignored, and if so, by what
> pattern". Then you could use that tool to generally debug your
> .gitignore patterns.

maybe GIT_TRACE could print that as well, in some way?

Re: How to find out which gitignore blocks my git-add

[Jeff King]

On Sat, Feb 07, 2009 at 01:33:43AM +0000, Sitaram Chamarty wrote:

> > I think it might be useful to be able to get this information. However,
> > rather than coupling it with "git add", it might make more sense to have
> > a separate way to query "is this being ignored, and if so, by what
> > pattern". Then you could use that tool to generally debug your
> > .gitignore patterns.
> 
> maybe GIT_TRACE could print that as well, in some way?

It could, but I think reusing GIT_TRACE isn't a good idea. Currently it
traces _just_ exec information, so I wouldn't want to pollute that with
this information. But yes, you could trigger it through an environment
variable, which would let us dump at the lowest level.

I spent a few minutes checking this out, and it looks to be a little
more complex than I had hoped, just because we don't have all of the
information in the same place at the same time.

A toy patch series follows; see 2/2 for a description of why it doesn't
work like you might hope. I'm not too interested in trying to deal with
the refactoring that would be required to do it right. But maybe
somebody else is.

-Peff

[PATCH 1/2] refactor exclude handling

[Jeff King]

The excluded function uses the static helper excluded_1 to
perform the inner loop over all of the exclude patterns. The
helper just tells us whether the path is included, excluded,
or undecided.

However, it may be useful to know _which_ pattern was
triggered. So let's pass out the entire exclude match, which
contains the status information we were already passing out.

Further patches can make use of this.

Signed-off-by: Jeff King <peff@peff.net>
---
Just a cleanup for the next patch.

 dir.c |   25 ++++++++++++++-----------
 1 files changed, 14 insertions(+), 11 deletions(-)

diff --git a/dir.c b/dir.c
index cfd1ea5..0ea81b7 100644
--- a/dir.c
+++ b/dir.c
@@ -291,9 +291,10 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)
 }
 
 /* Scan the list and let the last match determines the fate.
- * Return 1 for exclude, 0 for include and -1 for undecided.
+ * Returns the exclude_list element which matched, or NULL for
+ * undecided.
  */
-static int excluded_1(const char *pathname,
+static struct exclude *excluded_1(const char *pathname,
 		      int pathlen, const char *basename, int *dtype,
 		      struct exclude_list *el)
 {
@@ -303,7 +304,6 @@ static int excluded_1(const char *pathname,
 		for (i = el->nr - 1; 0 <= i; i--) {
 			struct exclude *x = el->excludes[i];
 			const char *exclude = x->pattern;
-			int to_exclude = x->to_exclude;
 
 			if (x->flags & EXC_FLAG_MUSTBEDIR) {
 				if (*dtype == DT_UNKNOWN)
@@ -316,14 +316,14 @@ static int excluded_1(const char *pathname,
 				/* match basename */
 				if (x->flags & EXC_FLAG_NOWILDCARD) {
 					if (!strcmp(exclude, basename))
-						return to_exclude;
+						return x;
 				} else if (x->flags & EXC_FLAG_ENDSWITH) {
 					if (x->patternlen - 1 <= pathlen &&
 					    !strcmp(exclude + 1, pathname + pathlen - x->patternlen + 1))
-						return to_exclude;
+						return x;
 				} else {
 					if (fnmatch(exclude, basename, 0) == 0)
-						return to_exclude;
+						return x;
 				}
 			}
 			else {
@@ -342,16 +342,16 @@ static int excluded_1(const char *pathname,
 
 				if (x->flags & EXC_FLAG_NOWILDCARD) {
 					if (!strcmp(exclude, pathname + baselen))
-						return to_exclude;
+						return x;
 				} else {
 					if (fnmatch(exclude, pathname+baselen,
 						    FNM_PATHNAME) == 0)
-					    return to_exclude;
+					    return x;
 				}
 			}
 		}
 	}
-	return -1; /* undecided */
+	return NULL;
 }
 
 int excluded(struct dir_struct *dir, const char *pathname, int *dtype_p)
@@ -363,8 +363,11 @@ int excluded(struct dir_struct *dir, const char *pathname, int *dtype_p)
 
 	prep_exclude(dir, pathname, basename-pathname);
 	for (st = EXC_CMDL; st <= EXC_FILE; st++) {
-		switch (excluded_1(pathname, pathlen, basename,
-				   dtype_p, &dir->exclude_list[st])) {
+		struct exclude *x = excluded_1(pathname, pathlen, basename,
+					dtype_p, &dir->exclude_list[st]);
+		if (!x)
+			continue;
+		switch (x->to_exclude) {
 		case 0:
 			return 0;
 		case 1:
-- 
1.6.1.2.552.g1682c.dirty

[PATCH 2/2] give exclude mechanism a debug option

[Jeff King]

Users can set GIT_DEBUG_IGNORE in the environment to get the
exclusion mechanism to dump to stderr files mentioned in
.gitignore along with the pattern that matched. The output
looks something like:

  foo.c: exclude: *.c

This implementation has several shortcomings that make it
unsuitable for inclusion:

  1. Doing it as a debug environment variable is hack-ish.
     A nicer interface would be a .gitignore equivalent of
     "git check-attr".

  2. If you ask for "foo/bar", and "foo/" is ignored, the
     output will show only "foo: exclude: foo". This is an
     artifact of the calling interface: you don't ask "is
     foo/bar excluded", but rather while recursing through
     "foo/" you ask "should I bother even recursing into
     foo?". So the exclusion code never even knows that you
     might have cared about foo/bar in the first place.

  3. There is no indication of where patterns came from. We
     could specify whether it came from the command-line,
     from per-directory files, or from another file. But what
     is most interesting is the actual _filename_ that it
     came from. I.e., something like:

       sub/foo.c: exclude: sub/.gitignore: *.c

     But that information seems to have been forgotten by
     the time we are actually doing excludes.

Signed-off-by: Jeff King <peff@peff.net>
---
In addition to the problems above, this is hardly tested. ;)

 dir.c |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/dir.c b/dir.c
index 0ea81b7..1052496 100644
--- a/dir.c
+++ b/dir.c
@@ -8,6 +8,7 @@
 #include "cache.h"
 #include "dir.h"
 #include "refs.h"
+#include "quote.h"
 
 struct path_simplify {
 	int len;
@@ -354,6 +355,24 @@ static struct exclude *excluded_1(const char *pathname,
 	return NULL;
 }
 
+static int debug_ignore(void)
+{
+	static int ignore = -1;
+	if (ignore == -1) {
+		const char *env = getenv("GIT_DEBUG_IGNORE");
+		ignore = env ? git_config_bool("GIT_DEBUG_IGNORE", env) : 0;
+	}
+	return ignore;
+}
+
+static void show_ignore(const char *path, struct exclude *x)
+{
+	quote_c_style(path, NULL, stderr, 0);
+	fprintf(stderr, ": %s: %.*s\n",
+		(x->to_exclude ? "exclude" : "include"),
+		x->patternlen, x->pattern);
+}
+
 int excluded(struct dir_struct *dir, const char *pathname, int *dtype_p)
 {
 	int pathlen = strlen(pathname);
@@ -367,6 +386,8 @@ int excluded(struct dir_struct *dir, const char *pathname, int *dtype_p)
 					dtype_p, &dir->exclude_list[st]);
 		if (!x)
 			continue;
+		if (debug_ignore())
+			show_ignore(pathname, x);
 		switch (x->to_exclude) {
 		case 0:
 			return 0;
-- 
1.6.1.2.552.g1682c.dirty

Re: [PATCH 2/2] give exclude mechanism a debug option

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Users can set GIT_DEBUG_IGNORE in the environment to get the
> exclusion mechanism to dump to stderr files mentioned in
> .gitignore along with the pattern that matched. The output
> looks something like:
>
>   foo.c: exclude: *.c
>
> This implementation has several shortcomings that make it
> unsuitable for inclusion:
>
>   1. Doing it as a debug environment variable is hack-ish.
>      A nicer interface would be a .gitignore equivalent of
>      "git check-attr".

Correct.

>   2. If you ask for "foo/bar", and "foo/" is ignored, the
>      output will show only "foo: exclude: foo". This is an
>      artifact of the calling interface: you don't ask "is
>      foo/bar excluded", but rather while recursing through
>      "foo/" you ask "should I bother even recursing into
>      foo?". So the exclusion code never even knows that you
>      might have cared about foo/bar in the first place.

I do not see why it is a problem.  It exactly is what you want to know,
isn't it?

>   3. There is no indication of where patterns came from. We
>      could specify whether it came from the command-line,
>      from per-directory files, or from another file. But what
>      is most interesting is the actual _filename_ that it
>      came from. I.e., something like:
>
>        sub/foo.c: exclude: sub/.gitignore: *.c
>
>      But that information seems to have been forgotten by
>      the time we are actually doing excludes.

For this, you would need to add an element to exclude_stack structure to
record the human readable name of the exclude source in prep_exclude().
Once you find the matched element using your new excluded_1() mechanism
introduced by patch 1/2, you can find which stack in dir->exclude_stack
the match element belongs to, and map it back the human readable source
name recorded in the exclude_stack.
.

Re: [PATCH 2/2] give exclude mechanism a debug option

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

>>   1. Doing it as a debug environment variable is hack-ish.
>>      A nicer interface would be a .gitignore equivalent of
>>      "git check-attr".
>
> Correct.

I suspect that ls-files would be the good match for it.  Wasn't the
exclude mechanism originally done for that command to begin with?

Re: [PATCH 2/2] give exclude mechanism a debug option

[Jeff King]

On Fri, Feb 06, 2009 at 11:42:45PM -0800, Junio C Hamano wrote:

> Junio C Hamano <gitster@pobox.com> writes:
> 
> >>   1. Doing it as a debug environment variable is hack-ish.
> >>      A nicer interface would be a .gitignore equivalent of
> >>      "git check-attr".
> >
> > Correct.
> 
> I suspect that ls-files would be the good match for it.  Wasn't the
> exclude mechanism originally done for that command to begin with?

I think you are envisioning something like "which files are excluded,
and what patterns matched them?" which ls-files is perhaps a good fit
for. I was imagining something more like "which patterns are affecting
this particular path?" which you could feed the output of a failed "git
add" to.

-Peff

Re: [PATCH 2/2] give exclude mechanism a debug option

[Jeff King]

On Fri, Feb 06, 2009 at 11:38:38PM -0800, Junio C Hamano wrote:

> >   2. If you ask for "foo/bar", and "foo/" is ignored, the
> >      output will show only "foo: exclude: foo". This is an
> >      artifact of the calling interface: you don't ask "is
> >      foo/bar excluded", but rather while recursing through
> >      "foo/" you ask "should I bother even recursing into
> >      foo?". So the exclusion code never even knows that you
> >      might have cared about foo/bar in the first place.
> 
> I do not see why it is a problem.  It exactly is what you want to know,
> isn't it?

Because I would expect "git check-ignore foo/bar | grep ^foo/bar:" to
tell me if and how foo/bar is being excluded. But I have to instead
check for ^foo and ^foo/bar.

Not a big deal, perhaps. But it just seems like counter-intuitive output
for a query about how a specific path is excluded.

-Peff

Re: [PATCH 2/2] give exclude mechanism a debug option

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Fri, Feb 06, 2009 at 11:38:38PM -0800, Junio C Hamano wrote:
>
>> >   2. If you ask for "foo/bar", and "foo/" is ignored, the
>> >      output will show only "foo: exclude: foo". This is an
>> >      artifact of the calling interface: you don't ask "is
>> >      foo/bar excluded", but rather while recursing through
>> >      "foo/" you ask "should I bother even recursing into
>> >      foo?". So the exclusion code never even knows that you
>> >      might have cared about foo/bar in the first place.
>> 
>> I do not see why it is a problem.  It exactly is what you want to know,
>> isn't it?
>
> Because I would expect "git check-ignore foo/bar | grep ^foo/bar:" to
> tell me if and how foo/bar is being excluded. But I have to instead
> check for ^foo and ^foo/bar.

Sorry, I do not understand why you need the downstream pipe that filters
using grep to begin with.

Shouldn't "git check-ignore dir/path" talk about the exclude entries that
caused dir/path to be excluded and no other patterns?  And if the reason
dir/path is exclude is because there was a higher level entry to exclude
dir/, the output would say so.  If there are unrelated exclude entries
that exclude dir's neighbour dir2 or dir/path's neighbour dir/path2, they
do not affect if dir/path is excluded, so check-ignore wouldn't show, no?

Re: [PATCH 2/2] give exclude mechanism a debug option

[Jeff King]

On Sat, Feb 07, 2009 at 01:45:44PM -0800, Junio C Hamano wrote:

> > Because I would expect "git check-ignore foo/bar | grep ^foo/bar:" to
> > tell me if and how foo/bar is being excluded. But I have to instead
> > check for ^foo and ^foo/bar.
> 
> Sorry, I do not understand why you need the downstream pipe that filters
> using grep to begin with.

Sorry, I should have been more clear. The grep was meant to simulate
what my eyes and brain are doing. That is, if I ask "what are patterns
affecting foo/bar?", I expect to see "foo/bar" in the output.

For asking about one path, it may not be that big a deal to see that the
output matches to the input. But if I feed a hundred paths to
check-ignore, I expect to be able to scan the output for the input I put
in.

> Shouldn't "git check-ignore dir/path" talk about the exclude entries that
> caused dir/path to be excluded and no other patterns?  And if the reason

Yes, I would think so. Which maybe means using the current "exclude"
code is not appropriate. Because it is really about traversing the
directory structure, excluding as we go. I don't know that you can
directly ask it about a particular path.

-Peff

Re: [PATCH 2/2] give exclude mechanism a debug option

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Sat, Feb 07, 2009 at 01:45:44PM -0800, Junio C Hamano wrote:
>
>> > Because I would expect "git check-ignore foo/bar | grep ^foo/bar:" to
>> > tell me if and how foo/bar is being excluded. But I have to instead
>> > check for ^foo and ^foo/bar.
>> 
>> Sorry, I do not understand why you need the downstream pipe that filters
>> using grep to begin with.
>
> Sorry, I should have been more clear. The grep was meant to simulate
> what my eyes and brain are doing. That is, if I ask "what are patterns
> affecting foo/bar?", I expect to see "foo/bar" in the output.

Oh, then the output format would, just like "grep" given more than one
file tells which file it found the match in, talk about which path the
output entry talks about when given more than one path.

Something like:

	$ git check-ignore foo/bar xyzzy~ hello.c
	foo/bar: foo/.gitignore:4: ??r
        xyzzy~: .git/info/excludes:22: *~

that says "foo/bar is ignored because line 4 of foo/.gitignore tells us to
ignore any three-letter filename that ends with r, xyzzy~ is ignored
because .git/info/excludes tells us to ignore everything that ends with
tilde on line 22, and hello.c is not ignored at all."

Re: [PATCH 2/2] give exclude mechanism a debug option

[Jeff King]

On Sun, Feb 08, 2009 at 01:50:27AM -0800, Junio C Hamano wrote:

> Oh, then the output format would, just like "grep" given more than one
> file tells which file it found the match in, talk about which path the
> output entry talks about when given more than one path.
> 
> Something like:
> 
> 	$ git check-ignore foo/bar xyzzy~ hello.c
> 	foo/bar: foo/.gitignore:4: ??r
>         xyzzy~: .git/info/excludes:22: *~

Right, exactly. What I was trying to say in my original message was that
my toy implementation _didn't_ do that, and it is a downside.

-Peff

Re: How to find out which gitignore blocks my git-add

[Sitaram Chamarty]

On 2009-02-07, Jeff King <peff@peff.net> wrote:
> A toy patch series follows; see 2/2 for a description of why it doesn't
> work like you might hope. I'm not too interested in trying to deal with
> the refactoring that would be required to do it right. But maybe
> somebody else is.

I was thinking it could simply be a separate utility in
contrib for starters, perhaps not even written in C.  I
don't mind volunteering to write one in shell+the usual
toolchain (and I'll test it on msysgit too, because I have
Windows users).  My C days are a bit behind me, sadly...

Re: How to find out which gitignore blocks my git-add

[Jeff King]

On Sat, Feb 07, 2009 at 12:44:27PM +0000, Sitaram Chamarty wrote:

> On 2009-02-07, Jeff King <peff@peff.net> wrote:
> > A toy patch series follows; see 2/2 for a description of why it doesn't
> > work like you might hope. I'm not too interested in trying to deal with
> > the refactoring that would be required to do it right. But maybe
> > somebody else is.
> 
> I was thinking it could simply be a separate utility in
> contrib for starters, perhaps not even written in C.  I
> don't mind volunteering to write one in shell+the usual
> toolchain (and I'll test it on msysgit too, because I have
> Windows users).  My C days are a bit behind me, sadly...

Sure, that might be an easy way to start. But in the long term,
having a separate implementation handling .gitignore parsing and
rules may diverge from what git is doing. OTOH, in theory those
rules are pretty well set in stone since users are depending on
them.

-Peff

Re: How to find out which gitignore blocks my git-add

[Bisani, Alok]

Gonzo <gonzo <at> gonsolo.de> writes:

> 
> After doing a "git add file" I get the message:
> 
> "The following paths are ignored by one of your .gitignore files:
> ..."
> 
> Is there an easy way to find out which line in which gitignore file 
> blocks this add?
> Would this be a viable addition to "git add -v"?
> 
> g
> 

It would be nice if a -v option also lists which pattern caused
 it to exclude.