1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

[Misty De Meo]

Hi,

Gitignore parsing no longer seems to work properly in git 1.8.3.

One of my repositories has the following gitignore:

/*
!/.gitignore
!/Library/
!/CONTRIBUTING.md
!/README.md
!/SUPPORTERS.md
!/bin
/bin/*
!/bin/brew
!/share/man/man1/brew.1
.DS_Store
/Library/LinkedKegs
/Library/PinnedKegs
/Library/Taps
/Library/Formula/.gitignore

In 1.8.2.3 and earlier, this works as expected. However, in 1.8.3 I'm
seeing every file in /bin/ being marked as an untracked file.

I asked about this in #git, and was told that the culprit was the
regex support; apparently recompiling without regex support fixes the
specific gitignore issue. However, this doesn't seem to have been
reported anywhere on the mailing list that I can see. I was also told
that the issue is OS X-specific, and doesn't happen on other
platforms.

Thanks,
Misty De Meo

Re: 1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

[Øystein Walle]

Misty De Meo <misty <at> brew.sh> writes:

> 
> Hi,
> 
> Gitignore parsing no longer seems to work properly in git 1.8.3.
> 
> One of my repositories has the following gitignore:
> 
> /*
> !/.gitignore
> !/Library/
> !/CONTRIBUTING.md
> !/README.md
> !/SUPPORTERS.md
> !/bin
> /bin/*
> !/bin/brew
> !/share/man/man1/brew.1
> .DS_Store
> /Library/LinkedKegs
> /Library/PinnedKegs
> /Library/Taps
> /Library/Formula/.gitignore
> 
> In 1.8.2.3 and earlier, this works as expected. However, in 1.8.3 I'm
> seeing every file in /bin/ being marked as an untracked file.
> 
> I asked about this in #git, and was told that the culprit was the
> regex support; apparently recompiling without regex support fixes the
> specific gitignore issue. However, this doesn't seem to have been
> reported anywhere on the mailing list that I can see. I was also told
> that the issue is OS X-specific, and doesn't happen on other
> platforms.
> 
> Thanks,
> Misty De Meo
> 

I see a similar problem using e.g. the following .gitignore to exclude
everything except C source files and header files:

    *
    !*/
    !*.c
    !*.h

In Git 1.8.3 'git status' will show other files as untracked while in
Git 1.8.2.3 I don't have that problem. I bisected to find that the
offending commit is v1.8.2.1-402-g95c6f27. 

I am not on OSX, however, but on Linux (Ubuntu 12.04 and RHEL 5.8) so
this may be a separate issue. I've also gotten the impression that this
is intentional. In any case I cannot create a .gitignore that achieves
the same for both older and newer versions of Git.

Best regards,
Øystein Walle

Re: 1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

[Junio C Hamano]

Øystein Walle <oystwa@gmail.com> writes:

> Misty De Meo <misty <at> brew.sh> writes:
>
>> 
>> Hi,
>> 
>> Gitignore parsing no longer seems to work properly in git 1.8.3.
>> 
>> One of my repositories has the following gitignore:
>> 
>> /*
>> !/.gitignore
>> !/Library/
>> !/CONTRIBUTING.md
>> !/README.md
>> !/SUPPORTERS.md
>> !/bin
>> /bin/*
>> !/bin/brew
>> !/share/man/man1/brew.1
>> .DS_Store
>> /Library/LinkedKegs
>> /Library/PinnedKegs
>> /Library/Taps
>> /Library/Formula/.gitignore
>> 
>> In 1.8.2.3 and earlier, this works as expected. However, in 1.8.3 I'm
>> seeing every file in /bin/ being marked as an untracked file.
>> 
>> I asked about this in #git, and was told that the culprit was the
>> regex support; apparently recompiling without regex support fixes the
>> specific gitignore issue. However, this doesn't seem to have been
>> reported anywhere on the mailing list that I can see. I was also told
>> that the issue is OS X-specific, and doesn't happen on other
>> platforms.
>> 
>> Thanks,
>> Misty De Meo
>> 
>
> I see a similar problem using e.g. the following .gitignore to exclude
> everything except C source files and header files:
>
>     *
>     !*/
>     !*.c
>     !*.h
>
> In Git 1.8.3 'git status' will show other files as untracked while in
> Git 1.8.2.3 I don't have that problem. I bisected to find that the
> offending commit is v1.8.2.1-402-g95c6f27. 
>
> I am not on OSX, however, but on Linux (Ubuntu 12.04 and RHEL 5.8) so
> this may be a separate issue. I've also gotten the impression that this
> is intentional. In any case I cannot create a .gitignore that achieves
> the same for both older and newer versions of Git.

Thanks; 95c6f27 is from Karsten and I think Duy is the most clueful
one in this area in the discussion that had this change; I am
forwarding this message to them.

Cf. http://thread.gmane.org/gmane.comp.version-control.git/218440/focus=221289

Re: 1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

[Duy Nguyen]

On Wed, May 29, 2013 at 12:54 AM, Misty De Meo <misty@brew.sh> wrote:
> Hi,
>
> Gitignore parsing no longer seems to work properly in git 1.8.3.
>
> One of my repositories has the following gitignore:
>
> /*
> !/.gitignore
> !/Library/
> !/CONTRIBUTING.md
> !/README.md
> !/SUPPORTERS.md
> !/bin
> /bin/*
> !/bin/brew
> !/share/man/man1/brew.1
> .DS_Store
> /Library/LinkedKegs
> /Library/PinnedKegs
> /Library/Taps
> /Library/Formula/.gitignore
>
> In 1.8.2.3 and earlier, this works as expected. However, in 1.8.3 I'm
> seeing every file in /bin/ being marked as an untracked file.

The changes in this area since 1.8.2.3 seem to be Karsten's (I'm not
blaming, just wanted to narrow down the problem). The patterns of
interest seem to be

!/bin
/bin/*
!/bin/brew

Without "!/bin" v1.8.3 seems to behave the same as v1.8.2.3. Can you verify it?

> I asked about this in #git, and was told that the culprit was the
> regex support; apparently recompiling without regex support fixes the
> specific gitignore issue. However, this doesn't seem to have been
> reported anywhere on the mailing list that I can see. I was also told
> that the issue is OS X-specific, and doesn't happen on other
> platforms.

Puzzled. regex has nothing to do with gitignore (glob does). How do
you recompile without regex support? Setting NO_REGEX? I'm on Linux
btw, no chance of touching OS X.
--
Duy

Re: 1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

[Duy Nguyen]

On Wed, May 29, 2013 at 10:41 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> The changes in this area since 1.8.2.3 seem to be Karsten's (I'm not
> blaming, just wanted to narrow down the problem). The patterns of
> interest seem to be
>
> !/bin
> /bin/*
> !/bin/brew
>
> Without "!/bin" v1.8.3 seems to behave the same as v1.8.2.3.

Karsten, the block "/* Abort if the directory is excluded */" in
prep_exclude() seems to cause this. I think it goes through the
exclude patterns, hits "!/bin", believes the patterns do not make
sense in this context and throws all away. I think Øystein's case
falls into the same path. Commenting out the block seems to gain the
old behavior back (and probably breaks other stuff). Contrary to what
Junio said, I'm clueless about this. I wanted to read your series
through and eventually gave up. I think I now have the motivation to
look at it again this weekend.
--
Duy

Re: 1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

[David Aguilar]

On Tue, May 28, 2013 at 9:19 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Wed, May 29, 2013 at 10:41 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>> The changes in this area since 1.8.2.3 seem to be Karsten's (I'm not
>> blaming, just wanted to narrow down the problem). The patterns of
>> interest seem to be
>>
>> !/bin
>> /bin/*
>> !/bin/brew
>>
>> Without "!/bin" v1.8.3 seems to behave the same as v1.8.2.3.
>
> Karsten, the block "/* Abort if the directory is excluded */" in
> prep_exclude() seems to cause this. I think it goes through the
> exclude patterns, hits "!/bin", believes the patterns do not make
> sense in this context and throws all away. I think Øystein's case
> falls into the same path. Commenting out the block seems to gain the
> old behavior back (and probably breaks other stuff). Contrary to what
> Junio said, I'm clueless about this. I wanted to read your series
> through and eventually gave up. I think I now have the motivation to
> look at it again this weekend.

The very first patch in the da/darwin series in "next" is one possible fix.

See 29de20504e9790785fe1698300755323f74972aa

    Makefile: fix default regex settings on Darwin

    t0070-fundamental.sh fails on Mac OS X 10.8:

        $ uname -a
        Darwin lustrous 12.2.0 Darwin Kernel Version 12.2.0:
        Sat Aug 25 00:48:52 PDT 2012;
        root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

        $ ./t0070-fundamental.sh -v
        fatal: regex bug confirmed: re-build git with NO_REGEX=1

    Fix it by using Git's regex library.

Does that patch also fix this issue?

Karsten, you mentioned that compiling without regex support fixes it for you.
Does the above commit in git.git's "next" branch fix it too?

If so, I think it would be worth merging this early part into a "maint" release.
--
David

Re: 1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

[Karsten Blees]

Am 29.05.2013 06:19, schrieb Duy Nguyen:
> On Wed, May 29, 2013 at 10:41 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>> The changes in this area since 1.8.2.3 seem to be Karsten's (I'm not
>> blaming, just wanted to narrow down the problem). The patterns of
>> interest seem to be
>>
>> !/bin
>> /bin/*
>> !/bin/brew
>>
>> Without "!/bin" v1.8.3 seems to behave the same as v1.8.2.3.
> 
> Karsten, the block "/* Abort if the directory is excluded */" in
> prep_exclude() seems to cause this. I think it goes through the
> exclude patterns, hits "!/bin", believes the patterns do not make
> sense in this context and throws all away.

Yes, I forgot to check the "!" flag to determine if the directory is really excluded. I'll prepare a patch + test case for this.

@Øystein: in the meantime, could you check if this fixes the problem for you?

--- 8< ---
diff --git a/dir.c b/dir.c
index a5926fb..13858fe 100644
--- a/dir.c
+++ b/dir.c
@@ -821,6 +821,9 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)
 				dir->basebuf, stk->baselen - 1,
 				dir->basebuf + current, &dt);
 			dir->basebuf[stk->baselen - 1] = '/';
+			if (dir->exclude &&
+			    dir->exclude->flags & EXC_FLAG_NEGATIVE)
+				dir->exclude = NULL;
 			if (dir->exclude) {
 				dir->basebuf[stk->baselen] = 0;
 				dir->exclude_stack = stk;
-- 

Re: 1.8.3 - gitignore not being parsed correctly on OS X; regex support is broken?

[Øystein Walle]

Karsten Blees <karsten.blees <at> gmail.com> writes:

>  <at> Øystein: in the meantime, could you check if this fixes the problem 
for you?
> 
> --- 8< ---
> diff --git a/dir.c b/dir.c
> index a5926fb..13858fe 100644
> --- a/dir.c
> +++ b/dir.c
>  <at>  <at>  -821,6 +821,9  <at>  <at>  static void prep_exclude(struct 
dir_struct *dir, const char *base, int baselen)
>  				dir->basebuf, stk->baselen - 1,
>  				dir->basebuf + current, &dt);
>  			dir->basebuf[stk->baselen - 1] = '/';
> +			if (dir->exclude &&
> +			    dir->exclude->flags & EXC_FLAG_NEGATIVE)
> +				dir->exclude = NULL;
>  			if (dir->exclude) {
>  				dir->basebuf[stk->baselen] = 0;
>  				dir->exclude_stack = stk;

Hi, Karsten

I applied your fix on v1.8.3 on both systems I mentioned earlier and
from my tests the issue I reported is fixed.

Thank you very much! :)

Regards
Øsse

[PATCH] dir.c: fix ignore processing within not-ignored directories

[Karsten Blees]

As of 95c6f271 "dir.c: unify is_excluded and is_path_excluded APIs", the
is_excluded API no longer recurses into directories that match an ignore
pattern, and returns the directory's ignored state for all contained paths.

This is OK for normal ignore patterns, i.e. ignoring a directory affects
the entire contents recursively.

Unfortunately, this also "works" for negated ignore patterns ('!dir'), i.e.
the entire contents is "not-ignored" recursively, regardless of ignore
patterns that match the contents directly.

In prep_exclude, skip recursing into a directory only if it is really
ignored (i.e. the ignore pattern is not negated).

Signed-off-by: Karsten Blees <blees@dcon.de>
---

Also available here:
https://github.com/kblees/git/tree/kb/ignore-within-not-ignored-dir
git pull git://github.com/kblees/git.git kb/ignore-within-not-ignored-dir

 dir.c                              |  3 +++
 t/t3001-ls-files-others-exclude.sh | 18 ++++++++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/dir.c b/dir.c
index a5926fb..13858fe 100644
--- a/dir.c
+++ b/dir.c
@@ -821,6 +821,9 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)
 				dir->basebuf, stk->baselen - 1,
 				dir->basebuf + current, &dt);
 			dir->basebuf[stk->baselen - 1] = '/';
+			if (dir->exclude &&
+			    dir->exclude->flags & EXC_FLAG_NEGATIVE)
+				dir->exclude = NULL;
 			if (dir->exclude) {
 				dir->basebuf[stk->baselen] = 0;
 				dir->exclude_stack = stk;
diff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh
index 4e3735f..f0421c0 100755
--- a/t/t3001-ls-files-others-exclude.sh
+++ b/t/t3001-ls-files-others-exclude.sh
@@ -175,6 +175,24 @@ test_expect_success 'negated exclude matches can override previous ones' '
 	grep "^a.1" output
 '
 
+test_expect_success 'excluded directory overrides content patterns' '
+
+	git ls-files --others --exclude="one" --exclude="!one/a.1" >output &&
+	if grep "^one/a.1" output
+	then
+		false
+	fi
+'
+
+test_expect_success 'negated directory doesn'\''t affect content patterns' '
+
+	git ls-files --others --exclude="!one" --exclude="one/a.1" >output &&
+	if grep "^one/a.1" output
+	then
+		false
+	fi
+'
+
 test_expect_success 'subdirectory ignore (setup)' '
 	mkdir -p top/l1/l2 &&
 	(
-- 
1.8.3.8097.gdc25f02.dirty

Re: [PATCH] dir.c: fix ignore processing within not-ignored directories

[Duy Nguyen]

On Thu, May 30, 2013 at 3:32 AM, Karsten Blees <karsten.blees@gmail.com> wrote:
> As of 95c6f271 "dir.c: unify is_excluded and is_path_excluded APIs", the
> is_excluded API no longer recurses into directories that match an ignore
> pattern, and returns the directory's ignored state for all contained paths.
>
> This is OK for normal ignore patterns, i.e. ignoring a directory affects
> the entire contents recursively.
>
> Unfortunately, this also "works" for negated ignore patterns ('!dir'), i.e.
> the entire contents is "not-ignored" recursively, regardless of ignore
> patterns that match the contents directly.
>
> In prep_exclude, skip recursing into a directory only if it is really
> ignored (i.e. the ignore pattern is not negated).
>
> Signed-off-by: Karsten Blees <blees@dcon.de>

I think I've got a hang on the "unify" patch now.

Reviewed-by: Duy Nguyen <pclouds@gmail.com>


> diff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh
> +test_expect_success 'excluded directory overrides content patterns' '
> +
> +       git ls-files --others --exclude="one" --exclude="!one/a.1" >output &&
> +       if grep "^one/a.1" output

Actually I think this is a shortcoming of gitignore. You ask to
"exclude one except one/a.1" and one/a.1 should show up. '!' is
designed from day one to deal with the other way around ("include one
except one/a.1"). And it's arguable (and it was in the mail archive)
that if you already exclude "one", we should never ever descend there
to pick up "!a.1" from one/.gitignore. But we should do it with
already collected patterns, at least if we detect there are negated
patterns following the pattern that excludes a directory, e.g.
!one/a.1 or even !*.c. For the latter case, the user can always move
"!*.c" up before "one" if they don't want git to misinterpret and
descend in every excluded directory.

> +       then
> +               false
> +       fi
> +'

Nit pick, maybe this instead?

test_must_fail grep "^one/a.1" output

> +
> +test_expect_success 'negated directory doesn'\''t affect content patterns' '
> +
> +       git ls-files --others --exclude="!one" --exclude="one/a.1" >output &&
> +       if grep "^one/a.1" output
> +       then
> +               false
> +       fi
> +'

Same.
--
Duy

Re: [PATCH] dir.c: fix ignore processing within not-ignored directories

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

>> +       then
>> +               false
>> +       fi
>> +'
>
> Nit pick, maybe this instead?
>
> test_must_fail grep "^one/a.1" output

Neither.

	! grep "^one/a.1" output

The second bullet point in the "Don't" section of t/README may want
to be updated to clarify that test_must_fail is to test Git, and we
do not mean to catch segfaults by platform tools like grep.

Re: [PATCH] dir.c: fix ignore processing within not-ignored directories

[Karsten Blees]

Am 02.06.2013 21:25, schrieb Junio C Hamano:
> Duy Nguyen <pclouds@gmail.com> writes:
> 
>>> +       then
>>> +               false
>>> +       fi
>>> +'
>>
>> Nit pick, maybe this instead?
>>
>> test_must_fail grep "^one/a.1" output
> 
> Neither.
> 
> 	! grep "^one/a.1" output
> 

Nice. I actually tried "!" but without the space - which didn't work so I took the other t3001 tests as example...

[PATCH] t/README: test_must_fail is for testing Git

[Junio C Hamano]

When a test wants to make sure there is no <string> in an output
file, we should just say "! grep string output"; "test_must_fail"
is there only to test Git command and catch unusual deaths we know
about (e.g. segv) as an error, not as an expected failure.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 t/README | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/t/README b/t/README
index e669bb3..35b3c5c 100644
--- a/t/README
+++ b/t/README
@@ -324,6 +324,9 @@ Don't:
    use 'test_must_fail git cmd'.  This will signal a failure if git
    dies in an unexpected way (e.g. segfault).
 
+   On the other hand, don't use test_must_fail for running regular
+   platform commands; just use '! cmd'.
+
  - use perl without spelling it as "$PERL_PATH". This is to help our
    friends on Windows where the platform Perl often adds CR before
    the end of line, and they bundle Git with a version of Perl that

Re: [PATCH] t/README: test_must_fail is for testing Git

[Philip Oakley]

From: "Junio C Hamano" <gitster@pobox.com>
Sent: Tuesday, June 04, 2013 5:50 PM
> When a test wants to make sure there is no <string> in an output
> file, we should just say "! grep string output";

Small nit: It took me two readings of the commit message to correctly 
parse this break point. The flowing together of the two parts with the 
semicolon fooled me. Separate them?

>      "test_must_fail"
> is there only to test Git command and catch unusual deaths we know
> about (e.g. segv) as an error, not as an expected failure.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
> t/README | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/t/README b/t/README
> index e669bb3..35b3c5c 100644
> --- a/t/README
> +++ b/t/README
> @@ -324,6 +324,9 @@ Don't:
>    use 'test_must_fail git cmd'.  This will signal a failure if git
>    dies in an unexpected way (e.g. segfault).
>
> +   On the other hand, don't use test_must_fail for running regular
> +   platform commands; just use '! cmd'.
> +
>  - use perl without spelling it as "$PERL_PATH". This is to help our
>    friends on Windows where the platform Perl often adds CR before
>    the end of line, and they bundle Git with a version of Perl that
> --

Philip 

Re: [PATCH] t/README: test_must_fail is for testing Git

[Junio C Hamano]

"Philip Oakley" <philipoakley@iee.org> writes:

> From: "Junio C Hamano" <gitster@pobox.com>
> Sent: Tuesday, June 04, 2013 5:50 PM
>> When a test wants to make sure there is no <string> in an output
>> file, we should just say "! grep string output";
>
> Small nit: It took me two readings of the commit message to correctly
> parse this break point. The flowing together of the two parts with the
> semicolon fooled me. Separate them?
>
>>      "test_must_fail"
>> is there only to test Git command and catch unusual deaths we know
>> about (e.g. segv) as an error, not as an expected failure.

Thanks.  Does this read better?

    t/README: test_must_fail is for testing Git
    
    When a test wants to make sure there is no <string> in an output
    file, we should just say "! grep string output".
    
    "test_must_fail" is there only to test Git command and catch unusual
    deaths we know about (e.g. segv) as an error, not as an expected
    failure.  "test_must_fail grep string output" is unnecessary, as
    we are not making sure the system binaries do not dump core or
    anything like that.
    
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

Re: [PATCH] t/README: test_must_fail is for testing Git

[Philip Oakley]

From: "Junio C Hamano" <gitster@pobox.com>
Sent: Tuesday, June 04, 2013 9:49 PM
> "Philip Oakley" <philipoakley@iee.org> writes:
>
>> From: "Junio C Hamano" <gitster@pobox.com>
>> Sent: Tuesday, June 04, 2013 5:50 PM
>>> When a test wants to make sure there is no <string> in an output
>>> file, we should just say "! grep string output";
>>
>> Small nit: It took me two readings of the commit message to correctly
>> parse this break point. The flowing together of the two parts with 
>> the
>> semicolon fooled me. Separate them?
>>
>>>      "test_must_fail"
>>> is there only to test Git command and catch unusual deaths we know
>>> about (e.g. segv) as an error, not as an expected failure.
>
> Thanks.  Does this read better?

Yes.  Thanks.

>
>    t/README: test_must_fail is for testing Git
>
>    When a test wants to make sure there is no <string> in an output
>    file, we should just say "! grep string output".
>
>    "test_must_fail" is there only to test Git command and catch 
> unusual
>    deaths we know about (e.g. segv) as an error, not as an expected
>    failure.  "test_must_fail grep string output" is unnecessary, as
>    we are not making sure the system binaries do not dump core or
>    anything like that.
>
>    Signed-off-by: Junio C Hamano <gitster@pobox.com>
> --

Philip