git 2.16.0 segfaults on clone of specific repo

git 2.16.0 segfaults on clone of specific repo

[Александр Булаев]

I found that git 2.16.0 segfaults on clone of vim-colorschemes repo.
See the log below.

alexbool@alexbool-osx ~/Documents> lldb -- git clone
https://github.com/flazz/vim-colorschemes.git
(lldb) target create "git"
Current executable set to 'git' (x86_64).
(lldb) settings set -- target.run-args  "clone"
"https://github.com/flazz/vim-colorschemes.git"
(lldb) run
Process 25643 launched: '/usr/local/bin/git' (x86_64)
Cloning into 'vim-colorschemes'...
remote: Counting objects: 1457, done.
remote: Total 1457 (delta 0), reused 0 (delta 0), pack-reused 1457
Receiving objects: 100% (1457/1457), 1.43 MiB | 289.00 KiB/s, done.
Resolving deltas: 100% (424/424), done.
Process 25643 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason =
EXC_BAD_ACCESS (code=1, address=0x48)
    frame #0: 0x00000001000f7c3d git`ce_match_stat_basic + 263
git`ce_match_stat_basic:
->  0x1000f7c3d <+263>: movq   0x48(%rax), %rsi
    0x1000f7c41 <+267>: movl   $0x14, %edx
    0x1000f7c46 <+272>: movq   %r14, %rdi
    0x1000f7c49 <+275>: callq  0x1001659fa               ; symbol stub
for: memcmp
Target 0: (git) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason =
EXC_BAD_ACCESS (code=1, address=0x48)
  * frame #0: 0x00000001000f7c3d git`ce_match_stat_basic + 263
    frame #1: 0x00000001000f7ae8 git`ie_match_stat + 108
    frame #2: 0x00000001000bbe07 git`checkout_entry + 266
    frame #3: 0x0000000100144ff9 git`unpack_trees + 2317
    frame #4: 0x0000000100017e13 git`cmd_clone + 5733
    frame #5: 0x0000000100001ea3 git`handle_builtin + 532
    frame #6: 0x0000000100001954 git`cmd_main + 263
    frame #7: 0x0000000100079a28 git`main + 80
    frame #8: 0x00007fff64d76115 libdyld.dylib`start + 1
(lldb) ^D
alexbool@alexbool-osx ~/Documents> git --version
git version 2.16.0

OS: macOS 10.13.2
git installed from homebrew

RE: git 2.16.0 segfaults on clone of specific repo

[Randall S. Becker]

On January 18, 2018 3:56 PM, Aleks wrote:
> I found that git 2.16.0 segfaults on clone of vim-colorschemes repo.
<snip>

Just tested on NonStop NSE and works fine here. Just an FYI now that we're on 2.16.0.

Cheers,
Randall

-- Brief whoami:
  NonStop developer since approximately NonStop(211288444200000000)
  UNIX developer since approximately 421664400
-- In my real life, I talk too much.

Re: git 2.16.0 segfaults on clone of specific repo

[Eric Sunshine]

[+cc:brian]

On Thu, Jan 18, 2018 at 3:55 PM, Александр Булаев
<aleks.bulaev@gmail.com> wrote:
> I found that git 2.16.0 segfaults on clone of vim-colorschemes repo.
>
> (lldb) run
> Process 25643 launched: '/usr/local/bin/git' (x86_64)
> Cloning into 'vim-colorschemes'...
> remote: Counting objects: 1457, done.
> remote: Total 1457 (delta 0), reused 0 (delta 0), pack-reused 1457
> Receiving objects: 100% (1457/1457), 1.43 MiB | 289.00 KiB/s, done.
> Resolving deltas: 100% (424/424), done.
> Process 25643 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason =
> EXC_BAD_ACCESS (code=1, address=0x48)

I can confirm that this crashes on MacOS; it does not crash on Linux or BSD.

git-bisect places blame on eb0ccfd7f5 (Switch empty tree and blob
lookups to use hash abstraction, 2017-11-12).

Re: git 2.16.0 segfaults on clone of specific repo

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2669 bytes --]

On Thu, Jan 18, 2018 at 07:15:56PM -0500, Eric Sunshine wrote:
> [+cc:brian]
> 
> On Thu, Jan 18, 2018 at 3:55 PM, Александр Булаев
> <aleks.bulaev@gmail.com> wrote:
> > I found that git 2.16.0 segfaults on clone of vim-colorschemes repo.
> >
> > (lldb) run
> > Process 25643 launched: '/usr/local/bin/git' (x86_64)
> > Cloning into 'vim-colorschemes'...
> > remote: Counting objects: 1457, done.
> > remote: Total 1457 (delta 0), reused 0 (delta 0), pack-reused 1457
> > Receiving objects: 100% (1457/1457), 1.43 MiB | 289.00 KiB/s, done.
> > Resolving deltas: 100% (424/424), done.
> > Process 25643 stopped
> > * thread #1, queue = 'com.apple.main-thread', stop reason =
> > EXC_BAD_ACCESS (code=1, address=0x48)
> 
> I can confirm that this crashes on MacOS; it does not crash on Linux or BSD.
> 
> git-bisect places blame on eb0ccfd7f5 (Switch empty tree and blob
> lookups to use hash abstraction, 2017-11-12).

I unfortunately don't have a macOS system to test with, and I've
compiled with both gcc and clang on my Debian system and, as you
mentioned, it doesn't fail there.

I have a guess about what the problem might be.  Can you try this patch
and see if it fixes things?

-- >8 --
From 10b690241619a452634b31fbc5ccd054a4f6e5ec Mon Sep 17 00:00:00 2001
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Date: Sun, 14 Jan 2018 18:26:29 +0000
Subject: [PATCH] repository: pre-initialize hash algo pointer

There are various git subcommands (among them, clone) which don't set up
the repository but end up needing to have information about the hash
algorithm in use.  In the future, we can add a command line option for
this or read it from the configuration, but until we're ready to expose
that functionality to the user, simply initialize the repository
structure to use the current hash algorithm, SHA-1.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 repository.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/repository.c b/repository.c
index 998413b8bb..f66fcb1342 100644
--- a/repository.c
+++ b/repository.c
@@ -5,7 +5,7 @@
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
+	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
 };
 struct repository *the_repository = &the_repo;
 
-- 
2.16.0.rc2.280.g09355b536d
-- >8 --
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: git 2.16.0 segfaults on clone of specific repo

[Eric Sunshine]

On Thu, Jan 18, 2018 at 9:47 PM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> On Thu, Jan 18, 2018 at 07:15:56PM -0500, Eric Sunshine wrote:
>> On Thu, Jan 18, 2018 at 3:55 PM, Александр Булаев
>> <aleks.bulaev@gmail.com> wrote:
>> > I found that git 2.16.0 segfaults on clone of vim-colorschemes repo.
>>
>> I can confirm that this crashes on MacOS; it does not crash on Linux or BSD.
>>
>> git-bisect places blame on eb0ccfd7f5 (Switch empty tree and blob
>> lookups to use hash abstraction, 2017-11-12).
>
> I unfortunately don't have a macOS system to test with, and I've
> compiled with both gcc and clang on my Debian system and, as you
> mentioned, it doesn't fail there.
>
> I have a guess about what the problem might be.  Can you try this patch
> and see if it fixes things?

That does fix the crash. Thanks for the quick diagnosis.

Can the commit message go into more detail as to why this was crashing
(or your speculation about why)? Perhaps give more detail about what
'clone' is doing that led to the crash.

> -- >8 --
> From 10b690241619a452634b31fbc5ccd054a4f6e5ec Mon Sep 17 00:00:00 2001
> From: "brian m. carlson" <sandals@crustytoothpaste.net>
> Date: Sun, 14 Jan 2018 18:26:29 +0000
> Subject: [PATCH] repository: pre-initialize hash algo pointer
>
> There are various git subcommands (among them, clone) which don't set up
> the repository but end up needing to have information about the hash
> algorithm in use.  In the future, we can add a command line option for
> this or read it from the configuration, but until we're ready to expose
> that functionality to the user, simply initialize the repository
> structure to use the current hash algorithm, SHA-1.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  repository.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/repository.c b/repository.c
> index 998413b8bb..f66fcb1342 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -5,7 +5,7 @@
>
>  /* The main repository */
>  static struct repository the_repo = {
> -       NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
> +       NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
>  };
>  struct repository *the_repository = &the_repo;
>
> --

RE: git 2.16.0 segfaults on clone of specific repo

[Randall S. Becker]

On January 18, 2018 10:06 PM, Eric Sunshine wrote:
> On Thu, Jan 18, 2018 at 9:47 PM, brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > On Thu, Jan 18, 2018 at 07:15:56PM -0500, Eric Sunshine wrote:
> >> On Thu, Jan 18, 2018 at 3:55 PM, Александр Булаев
> >> <aleks.bulaev@gmail.com> wrote:
> >> > I found that git 2.16.0 segfaults on clone of vim-colorschemes repo.
> >>
> >> I can confirm that this crashes on MacOS; it does not crash on Linux or
> BSD.
> >>
> >> git-bisect places blame on eb0ccfd7f5 (Switch empty tree and blob
> >> lookups to use hash abstraction, 2017-11-12).
> >
> > I unfortunately don't have a macOS system to test with, and I've
> > compiled with both gcc and clang on my Debian system and, as you
> > mentioned, it doesn't fail there.
> >
> > I have a guess about what the problem might be.  Can you try this
> > patch and see if it fixes things?
> 
> That does fix the crash. Thanks for the quick diagnosis.
> 
> Can the commit message go into more detail as to why this was crashing (or
> your speculation about why)? Perhaps give more detail about what 'clone' is
> doing that led to the crash.

I'm curious as to why this worked on my platform, given how it tends to get annoyed with NULL in the wrong place.

> 
> > -- >8 --
> > From 10b690241619a452634b31fbc5ccd054a4f6e5ec Mon Sep 17 00:00:00
> 2001
> > From: "brian m. carlson" <sandals@crustytoothpaste.net>
> > Date: Sun, 14 Jan 2018 18:26:29 +0000
> > Subject: [PATCH] repository: pre-initialize hash algo pointer
> >
> > There are various git subcommands (among them, clone) which don't set
> > up the repository but end up needing to have information about the
> > hash algorithm in use.  In the future, we can add a command line
> > option for this or read it from the configuration, but until we're
> > ready to expose that functionality to the user, simply initialize the
> > repository structure to use the current hash algorithm, SHA-1.
> >
> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> > ---
> >  repository.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/repository.c b/repository.c index 998413b8bb..f66fcb1342
> > 100644
> > --- a/repository.c
> > +++ b/repository.c
> > @@ -5,7 +5,7 @@
> >
> >  /* The main repository */
> >  static struct repository the_repo = {
> > -       NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index,
> NULL, 0, 0
> > +       NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
> > + &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
> >  };
> >  struct repository *the_repository = &the_repo;
> >
> > --

Re: git 2.16.0 segfaults on clone of specific repo

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]

On Thu, Jan 18, 2018 at 10:06:10PM -0500, Eric Sunshine wrote:
> > I have a guess about what the problem might be.  Can you try this patch
> > and see if it fixes things?
> 
> That does fix the crash. Thanks for the quick diagnosis.
> 
> Can the commit message go into more detail as to why this was crashing
> (or your speculation about why)? Perhaps give more detail about what
> 'clone' is doing that led to the crash.

Sure.  I ran into this as I was expanding the hash structure abstraction
into my next series.  I'll send a follow-up patch with a more
descriptive answer.

I'm still extremely puzzled as to why this doesn't fail on Linux.  If
it's failing in this case, it should very, very clearly fail all the
time we access an empty blob or an empty tree.  I've tried with gcc and
two versions of clang, using -fno-lto, with address sanitizer, with -O0,
and so on.  I'd really like to catch this kind of issue sooner in the
future if I can figure out how to reproduce it.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

[PATCH] repository: pre-initialize hash algo pointer

[brian m. carlson]

There are various git subcommands (among them, clone) which don't set up
the repository (that is, they lack RUN_SETUP or RUN_SETUP_GENTLY) but
end up needing to have information about the hash algorithm in use.
Because the hash algorithm is part of struct repository and it's only
initialized in repository setup, we can end up dereferencing a NULL
pointer in some cases if we call one of these subcommands and look up
the empty blob or empty tree values.

In the future, we can add a command line option for this or read it from
the configuration, but until we're ready to expose that functionality to
the user, simply initialize the repository structure to use the current
hash algorithm, SHA-1.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 repository.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

I'm still quite mystified as to why this is working on Linux and not
macOS, but I can only guess that compilers are just very advanced and
have somehow concluded that we would clearly never dereference a NULL
pointer, so they picked the only non-NULL value.

I haven't included a test because I have no way to reproduce the issue.
This patch is the first from a series I'm working on where I do expand
the use of the hash struct and therefore caused a segfault on clone, so
I can imagine what's going on without having a way to prove it affects
this particular case.

If someone with access to macOS can provide a test, I'd be very
grateful.

My apologies for the error and inconvenience.

diff --git a/repository.c b/repository.c
index 998413b8bb..f66fcb1342 100644
--- a/repository.c
+++ b/repository.c
@@ -5,7 +5,7 @@
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
+	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
 };
 struct repository *the_repository = &the_repo;
 

Re: git 2.16.0 segfaults on clone of specific repo

[Duy Nguyen]

On Fri, Jan 19, 2018 at 10:40 AM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> On Thu, Jan 18, 2018 at 10:06:10PM -0500, Eric Sunshine wrote:
>> > I have a guess about what the problem might be.  Can you try this patch
>> > and see if it fixes things?
>>
>> That does fix the crash. Thanks for the quick diagnosis.
>>
>> Can the commit message go into more detail as to why this was crashing
>> (or your speculation about why)? Perhaps give more detail about what
>> 'clone' is doing that led to the crash.
>
> Sure.  I ran into this as I was expanding the hash structure abstraction
> into my next series.  I'll send a follow-up patch with a more
> descriptive answer.
>
> I'm still extremely puzzled as to why this doesn't fail on Linux.  If
> it's failing in this case, it should very, very clearly fail all the
> time we access an empty blob or an empty tree.  I've tried with gcc and
> two versions of clang, using -fno-lto, with address sanitizer, with -O0,
> and so on.  I'd really like to catch this kind of issue sooner in the
> future if I can figure out how to reproduce it.

I think it's file system related, case-insensitive one in particular.

The call trace posted at the beginning of this thread should never
trigger for an initial clone. You only check if an _existing_ entry
matches what you are about to checkout when you switch trees. For this
to happen at clone time, I suppose you have to checkout entry "A",
then re-checkout "A" again. Which can only happen on case-insensitive
file systems and a case-sensitive repo where the second "A" might
actually be "a".

vim-colorschemes.git has these two entries, which meets one of the
conditions, I suppose macos meets the other

colors/darkblue.vim
colors/darkBlue.vim

On Linux, after I hacked all over the place to force ce_match_stat()
to eventually call index_fd() which in turns must use one of the
hashing function, I got a crash.
-- 
Duy

Re: git 2.16.0 segfaults on clone of specific repo

[Eric Sunshine]

On Fri, Jan 19, 2018 at 12:31:58PM +0700, Duy Nguyen wrote:
> On Fri, Jan 19, 2018 at 10:40 AM, brian m. carlson
> > I'm still extremely puzzled as to why this doesn't fail on Linux.  If
> > it's failing in this case, it should very, very clearly fail all the
> > time we access an empty blob or an empty tree.[...]
> 
> I think it's file system related, case-insensitive one in particular.
> 
> The call trace posted at the beginning of this thread should never
> trigger for an initial clone. You only check if an _existing_ entry
> matches what you are about to checkout when you switch trees. For this
> to happen at clone time, I suppose you have to checkout entry "A",
> then re-checkout "A" again. Which can only happen on case-insensitive
> file systems and a case-sensitive repo where the second "A" might
> actually be "a".
> [...]
> On Linux, after I hacked all over the place to force ce_match_stat()
> to eventually call index_fd() which in turns must use one of the
> hashing function, I got a crash.

Nice detective work. This particular manifestation is caught by the
following test which fails without brian's patch on MacOS (and
presumably Windows) and succeeds with it. On Linux and BSD, it will of
course succeed always, so I'm not sure how much practical value it
has.

--- >8 ---
hex2oct() {
	perl -ne 'printf "\\%03o", hex for /../g'
}

test_expect_success 'clone on case-insensitive fs' '
	o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
	t=$(printf "100644 X\0${o}100644 x\0${o}" |
		   git hash-object -w -t tree --stdin) &&
	c=$(git commit-tree -m bogus $t) &&
	git update-ref refs/heads/bogus $c &&
	git clone -b bogus . bogus
'
--- >8 ---

(hex2oct lifted from t1007/t1450)

Re: [PATCH] repository: pre-initialize hash algo pointer

[Eric Sunshine]

On Thu, Jan 18, 2018 at 11:18 PM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> There are various git subcommands (among them, clone) which don't set up
> the repository (that is, they lack RUN_SETUP or RUN_SETUP_GENTLY) but
> end up needing to have information about the hash algorithm in use.
> Because the hash algorithm is part of struct repository and it's only
> initialized in repository setup, we can end up dereferencing a NULL
> pointer in some cases if we call one of these subcommands and look up
> the empty blob or empty tree values.
>
> In the future, we can add a command line option for this or read it from
> the configuration, but until we're ready to expose that functionality to
> the user, simply initialize the repository structure to use the current
> hash algorithm, SHA-1.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> I'm still quite mystified as to why this is working on Linux and not
> macOS, but I can only guess that compilers are just very advanced and
> have somehow concluded that we would clearly never dereference a NULL
> pointer, so they picked the only non-NULL value.

Now that we know (due to Duy's excellent detective work[1]) that the
trigger is files with names differing only in case on case-insensitive
filesystems, the commit message can be updated appropriately.

> I haven't included a test because I have no way to reproduce the issue.
> This patch is the first from a series I'm working on where I do expand
> the use of the hash struct and therefore caused a segfault on clone, so
> I can imagine what's going on without having a way to prove it affects
> this particular case.
>
> If someone with access to macOS can provide a test, I'd be very
> grateful.

Done. Find the test here[2]. It fails without your fix on MacOS and
(presumably) Windows, and succeeds with the fix.

> My apologies for the error and inconvenience.

[1]: https://public-inbox.org/git/CACsJy8BTFm_0sv=roL1OKKW=1DyU3vqD50NKyHg3KQ7G+mAepQ@mail.gmail.com/
[2]: https://public-inbox.org/git/20180119074001.GA55929@flurp.local/

Re: git 2.16.0 segfaults on clone of specific repo

[Duy Nguyen]

On Fri, Jan 19, 2018 at 2:40 PM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> On Fri, Jan 19, 2018 at 12:31:58PM +0700, Duy Nguyen wrote:
>> On Fri, Jan 19, 2018 at 10:40 AM, brian m. carlson
>> > I'm still extremely puzzled as to why this doesn't fail on Linux.  If
>> > it's failing in this case, it should very, very clearly fail all the
>> > time we access an empty blob or an empty tree.[...]
>>
>> I think it's file system related, case-insensitive one in particular.
>>
>> The call trace posted at the beginning of this thread should never
>> trigger for an initial clone. You only check if an _existing_ entry
>> matches what you are about to checkout when you switch trees. For this
>> to happen at clone time, I suppose you have to checkout entry "A",
>> then re-checkout "A" again. Which can only happen on case-insensitive
>> file systems and a case-sensitive repo where the second "A" might
>> actually be "a".
>> [...]
>> On Linux, after I hacked all over the place to force ce_match_stat()
>> to eventually call index_fd() which in turns must use one of the
>> hashing function, I got a crash.
>
> Nice detective work.

And not stepping back to think for a bit. I realized now that if I
just mounted a vfat filesystem, I could have verified it much quicker.
It does crash on linux with similar stack trace.

(gdb) bt
#0  0x000000000055809f in is_empty_blob_sha1 (sha1=0x8fa6d4 "\236") at
cache.h:1055
#1  0x0000000000558acd in ce_match_stat_basic (ce=0x8fa690,
st=0x7fffffffcd20) at read-cache.c:272
#2  0x0000000000558c78 in ie_match_stat (istate=0x8e9fc0 <the_index>,
ce=0x8fa690, st=0x7fffffffcd20, options=5) at read-cache.c:342
#3  0x0000000000501e01 in checkout_entry (ce=0x8fa690,
state=0x7fffffffcea0, topath=0x0) at entry.c:424
#4  0x00000000005c8ea0 in check_updates (o=0x7fffffffd060) at unpack-trees.c:383
#5  0x00000000005cb2bb in unpack_trees (len=1, t=0x7fffffffd010,
o=0x7fffffffd060) at unpack-trees.c:1382
#6  0x00000000004214ca in checkout (submodule_progress=1) at builtin/clone.c:750
#7  0x00000000004229ce in cmd_clone (argc=1, argv=0x7fffffffd840,
prefix=0x0) at builtin/clone.c:1191
#8  0x0000000000405846 in run_builtin (p=0x89a908 <commands+456>,
argc=2, argv=0x7fffffffd840) at git.c:346
#9  0x0000000000405af7 in handle_builtin (argc=2, argv=0x7fffffffd840)
at git.c:554
#10 0x0000000000405c8f in run_argv (argcp=0x7fffffffd6fc,
argv=0x7fffffffd6f0) at git.c:606
#11 0x0000000000405e31 in cmd_main (argc=2, argv=0x7fffffffd840) at git.c:683
#12 0x00000000004a3231 in main (argc=3, argv=0x7fffffffd838) at common-main.c:43

> This particular manifestation is caught by the
> following test which fails without brian's patch on MacOS (and
> presumably Windows) and succeeds with it. On Linux and BSD, it will of
> course succeed always, so I'm not sure how much practical value it
> has.

There's a travis job running "on windows" (I don't what it really
means) so it might help actually. This vim-colorschemes repository has
shown up in git mailing list before, I think. We probably should just
add it as part of our regression tests ;-)
-- 
Duy

Re: git 2.16.0 segfaults on clone of specific repo

[Eric Sunshine]

On Fri, Jan 19, 2018 at 3:22 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Fri, Jan 19, 2018 at 2:40 PM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> On Fri, Jan 19, 2018 at 12:31:58PM +0700, Duy Nguyen wrote:
>>> On Linux, after I hacked all over the place to force ce_match_stat()
>>> to eventually call index_fd() which in turns must use one of the
>>> hashing function, I got a crash.
>>
>> Nice detective work.
>
> And not stepping back to think for a bit. I realized now that if I
> just mounted a vfat filesystem, I could have verified it much quicker.
> It does crash on linux with similar stack trace.
> [...]
>> This particular manifestation is caught by the
>> following test which fails without brian's patch on MacOS (and
>> presumably Windows) and succeeds with it. On Linux and BSD, it will of
>> course succeed always, so I'm not sure how much practical value it
>> has.
>
> There's a travis job running "on windows" (I don't what it really
> means) so it might help actually. This vim-colorschemes repository has
> shown up in git mailing list before, I think. We probably should just
> add it as part of our regression tests ;-)

And, Dscho does run the test suite on Windows regularly, so perhaps
this test does have some practical value.

Re: git 2.16.0 segfaults on clone of specific repo

[Todd Zullinger]

Eric Sunshine wrote:
> Nice detective work. This particular manifestation is caught by the
> following test which fails without brian's patch on MacOS (and
> presumably Windows) and succeeds with it. On Linux and BSD, it will of
> course succeed always, so I'm not sure how much practical value it
> has.

The CASE_INSENSITIVE_FS prereq could be used to avoid
running the test on systems where it won't provide much
value, couldn't it?

> --- >8 ---
> hex2oct() {
> 	perl -ne 'printf "\\%03o", hex for /../g'
> }
> 
> test_expect_success 'clone on case-insensitive fs' '
> 	o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
> 	t=$(printf "100644 X\0${o}100644 x\0${o}" |
> 		   git hash-object -w -t tree --stdin) &&
> 	c=$(git commit-tree -m bogus $t) &&
> 	git update-ref refs/heads/bogus $c &&
> 	git clone -b bogus . bogus
> '
> --- >8 ---
> 
> (hex2oct lifted from t1007/t1450)

-- 
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Money frees you from doing things you dislike. Since I dislike doing
nearly everything, money is handy.
    -- Groucho Marx

Re: [PATCH] repository: pre-initialize hash algo pointer

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

>> I'm still quite mystified as to why this is working on Linux and not
>> macOS, but I can only guess that compilers are just very advanced and
>> have somehow concluded that we would clearly never dereference a NULL
>> pointer, so they picked the only non-NULL value.
>
> Now that we know (due to Duy's excellent detective work[1]) that the
> trigger is files with names differing only in case on case-insensitive
> filesystems, the commit message can be updated appropriately.

Thanks.  Let me apply the following and do a 2.16.1, hopefully by
the end of day or mid tomorrow at the latest.  Test to protect the
fix can come as a separate follow-up patch.

-- >8 --
Subject: [PATCH] repository: pre-initialize hash algo pointer

There are various git subcommands (among them, clone) which don't set up
the repository (that is, they lack RUN_SETUP or RUN_SETUP_GENTLY) but
end up needing to have information about the hash algorithm in use.
Because the hash algorithm is part of struct repository and it's only
initialized in repository setup, we can end up dereferencing a NULL
pointer in some cases if we call one of these subcommands and look up
the empty blob or empty tree values.

A "git clone" of a project that has two paths that differ only in
case suffers from this if it is run on a case insensitive platform.
When the command attempts to check out one of these two paths after
checking out the other one, the checkout codepath needs to see if
the version that is already on the filesystem (which should not
happen if the FS were case sensitive), and it needs to exercise the
hashing code.

In the future, we can add a command line option for this or read it
from the configuration, but until we're ready to expose that
functionality to the user, simply initialize the repository
structure to use the current hash algorithm, SHA-1.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 repository.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/repository.c b/repository.c
index 998413b8bb..f66fcb1342 100644
--- a/repository.c
+++ b/repository.c
@@ -5,7 +5,7 @@
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
+	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
 };
 struct repository *the_repository = &the_repo;
 
-- 
2.16.0-204-gc262421c89

Re: [PATCH] repository: pre-initialize hash algo pointer

[Eric Sunshine]

On Fri, Jan 19, 2018 at 2:24 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Eric Sunshine <sunshine@sunshineco.com> writes:
>> Now that we know (due to Duy's excellent detective work[1]) that the
>> trigger is files with names differing only in case on case-insensitive
>> filesystems, the commit message can be updated appropriately.
>
> Thanks.  Let me apply the following and do a 2.16.1, hopefully by
> the end of day or mid tomorrow at the latest.  Test to protect the
> fix can come as a separate follow-up patch.
>
> -- >8 --
> Subject: [PATCH] repository: pre-initialize hash algo pointer
> [...]
> A "git clone" of a project that has two paths that differ only in
> case suffers from this if it is run on a case insensitive platform.
> When the command attempts to check out one of these two paths after
> checking out the other one, the checkout codepath needs to see if
> the version that is already on the filesystem (which should not
> happen if the FS were case sensitive), and it needs to exercise the
> hashing code.

Thanks, the amended commit message makes the reason for the patch more
concrete. There does seem to be a bit of a grammatical issue, however,
which makes it difficult to parse. Namely, "already on the filesystem
(...)" probably was meant to say "already on the filesystem (...) is
{something}".

Re: [PATCH] repository: pre-initialize hash algo pointer

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

> On Fri, Jan 19, 2018 at 2:24 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Eric Sunshine <sunshine@sunshineco.com> writes:
>>> Now that we know (due to Duy's excellent detective work[1]) that the
>>> trigger is files with names differing only in case on case-insensitive
>>> filesystems, the commit message can be updated appropriately.
>>
>> Thanks.  Let me apply the following and do a 2.16.1, hopefully by
>> the end of day or mid tomorrow at the latest.  Test to protect the
>> fix can come as a separate follow-up patch.
>>
>> -- >8 --
>> Subject: [PATCH] repository: pre-initialize hash algo pointer
>> [...]
>> A "git clone" of a project that has two paths that differ only in
>> case suffers from this if it is run on a case insensitive platform.
>> When the command attempts to check out one of these two paths after
>> checking out the other one, the checkout codepath needs to see if
>> the version that is already on the filesystem (which should not
>> happen if the FS were case sensitive), and it needs to exercise the
>> hashing code.
>
> Thanks, the amended commit message makes the reason for the patch more
> concrete. There does seem to be a bit of a grammatical issue, however,
> which makes it difficult to parse. Namely, "already on the filesystem
> (...)" probably was meant to say "already on the filesystem (...) is
> {something}".

Indeed it is incomplete sentence.  The codepath wants to check if
the one on the filesystem is the same as the one that is being
checked out and exercises the hashing code at that point.

Thansk.

Re: git 2.16.0 segfaults on clone of specific repo

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

On Fri, Jan 19, 2018 at 02:40:02AM -0500, Eric Sunshine wrote:
> Nice detective work. This particular manifestation is caught by the
> following test which fails without brian's patch on MacOS (and
> presumably Windows) and succeeds with it. On Linux and BSD, it will of
> course succeed always, so I'm not sure how much practical value it
> has.
> 
> --- >8 ---
> hex2oct() {
> 	perl -ne 'printf "\\%03o", hex for /../g'
> }
> 
> test_expect_success 'clone on case-insensitive fs' '
> 	o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
> 	t=$(printf "100644 X\0${o}100644 x\0${o}" |
> 		   git hash-object -w -t tree --stdin) &&
> 	c=$(git commit-tree -m bogus $t) &&
> 	git update-ref refs/heads/bogus $c &&
> 	git clone -b bogus . bogus
> '
> --- >8 ---

I'd argue that it's a worthwhile test to have, since it will fail on
those systems where it's going to be a problem.  Furthermore, people do
run the tests (as does Travis) on case-insensitive file systems during
the development cycle, so if we break something in the future, someone
will notice while we're still in the development cycle.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: [PATCH] repository: pre-initialize hash algo pointer

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

On Fri, Jan 19, 2018 at 11:24:24AM -0800, Junio C Hamano wrote:
> Eric Sunshine <sunshine@sunshineco.com> writes:
> > Now that we know (due to Duy's excellent detective work[1]) that the
> > trigger is files with names differing only in case on case-insensitive
> > filesystems, the commit message can be updated appropriately.
> 
> Thanks.  Let me apply the following and do a 2.16.1, hopefully by
> the end of day or mid tomorrow at the latest.  Test to protect the
> fix can come as a separate follow-up patch.

This looks good with Eric's suggested changes.

Again, my apologies for the breakage and the inconvenience involved.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: git 2.16.0 segfaults on clone of specific repo

[Eric Sunshine]

On Fri, Jan 19, 2018 at 5:31 PM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> On Fri, Jan 19, 2018 at 02:40:02AM -0500, Eric Sunshine wrote:
>> Nice detective work. This particular manifestation is caught by the
>> following test which fails without brian's patch on MacOS (and
>> presumably Windows) and succeeds with it. On Linux and BSD, it will of
>> course succeed always, so I'm not sure how much practical value it
>> has.
>
> I'd argue that it's a worthwhile test to have, since it will fail on
> those systems where it's going to be a problem.  Furthermore, people do
> run the tests (as does Travis) on case-insensitive file systems during
> the development cycle, so if we break something in the future, someone
> will notice while we're still in the development cycle.

Oh, I agree. My original question of its practical value was based
upon my belief that the full test suite is very rarely run on MacOS
(partly because there are so few Git developers on MacOS and partly
because it runs so slowly on the platform, though not nearly as slowly
as on Windows). However, as soon as I hit "Send", it hit me that the
problem would also manifest on Windows, and we know that Dscho runs
the tests regularly on Windows, so I changed my mind. I did consider
Travis but wasn't sure if it was testing on case-insensitive
filesystems.

Are you planning on submitting the test as a proper patch as follow-up
to [1]? I couldn't quite decide in which test script it should reside.

[1]: https://public-inbox.org/git/xmqqr2qlps7r.fsf@gitster.mtv.corp.google.com/

Re: git 2.16.0 segfaults on clone of specific repo

[Eric Sunshine]

On Fri, Jan 19, 2018 at 12:25 PM, Todd Zullinger <tmz@pobox.com> wrote:
> Eric Sunshine wrote:
>> Nice detective work. This particular manifestation is caught by the
>> following test which fails without brian's patch on MacOS (and
>> presumably Windows) and succeeds with it. On Linux and BSD, it will of
>> course succeed always, so I'm not sure how much practical value it
>> has.
>
> The CASE_INSENSITIVE_FS prereq could be used to avoid
> running the test on systems where it won't provide much
> value, couldn't it?

As mentioned in [1], my original questioning of the practical value of
the test was tied to the belief that the full test suite is probably
rarely run on MacOS, and it wasn't until I pressed "Send" that it hit
me that it would manifest on Windows as well, and that Dscho does test
on Windows regularly. Therefore, the test could have value.

At any rate, the test potentially could catch some sort of future
regression even on case-sensitive filesystems, therefore, it would be
better not to hide it behind CASE_INSENSITIVE_FS.

[1]: https://public-inbox.org/git/CAPig+cQmWqQWQrRQHHn=3hn6UFzJxT=9d5kKnJht_dt8sCgwkQ@mail.gmail.com/

Re: git 2.16.0 segfaults on clone of specific repo

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

On Fri, Jan 19, 2018 at 07:15:45PM -0500, Eric Sunshine wrote:
> Oh, I agree. My original question of its practical value was based
> upon my belief that the full test suite is very rarely run on MacOS
> (partly because there are so few Git developers on MacOS and partly
> because it runs so slowly on the platform, though not nearly as slowly
> as on Windows). However, as soon as I hit "Send", it hit me that the
> problem would also manifest on Windows, and we know that Dscho runs
> the tests regularly on Windows, so I changed my mind. I did consider
> Travis but wasn't sure if it was testing on case-insensitive
> filesystems.
> 
> Are you planning on submitting the test as a proper patch as follow-up
> to [1]? I couldn't quite decide in which test script it should reside.

Sure, I can.  Since you wrote it, may I have permission to add your
sign-off?

I'll try to set up a temporary FAT device and verify that I can
reproduce the problem on Linux as well.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: git 2.16.0 segfaults on clone of specific repo

[Eric Sunshine]

On Fri, Jan 19, 2018 at 7:23 PM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> On Fri, Jan 19, 2018 at 07:15:45PM -0500, Eric Sunshine wrote:
>> Are you planning on submitting the test as a proper patch as follow-up
>> to [1]? I couldn't quite decide in which test script it should reside.
>
> Sure, I can.  Since you wrote it, may I have permission to add your
> sign-off?

You may...

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>

> I'll try to set up a temporary FAT device and verify that I can
> reproduce the problem on Linux as well.

Re: [PATCH] repository: pre-initialize hash algo pointer

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Eric Sunshine <sunshine@sunshineco.com> writes:
>
>>> I'm still quite mystified as to why this is working on Linux and not
>>> macOS, but I can only guess that compilers are just very advanced and
>>> have somehow concluded that we would clearly never dereference a NULL
>>> pointer, so they picked the only non-NULL value.
>>
>> Now that we know (due to Duy's excellent detective work[1]) that the
>> trigger is files with names differing only in case on case-insensitive
>> filesystems, the commit message can be updated appropriately.
>
> Thanks.  Let me apply the following and do a 2.16.1, hopefully by
> the end of day or mid tomorrow at the latest.  Test to protect the
> fix can come as a separate follow-up patch.

I actually changed my mind and decided not to rush 2.16.1 with just
this change.  After all, even though it is better not to crash in
such a problematic case, a "successful" clone of such a project on
such a filesystem cannot be sanely used *anyway*, so in that sense
the "fix" would stop "clone" from segfaulting but the resulting
repository would still be "wrong" (e.g. "git status" immediately
after clone would likely to say that the working tree is already
dirty and missing one of the two paths, or something silly like
that).

Re: [PATCH] repository: pre-initialize hash algo pointer

[Eric Sunshine]

On Sat, Jan 20, 2018 at 2:01 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Junio C Hamano <gitster@pobox.com> writes:
>> Eric Sunshine <sunshine@sunshineco.com> writes:
>>> Now that we know (due to Duy's excellent detective work[1]) that the
>>> trigger is files with names differing only in case on case-insensitive
>>> filesystems, the commit message can be updated appropriately.
>>
>> Thanks.  Let me apply the following and do a 2.16.1, hopefully by
>> the end of day or mid tomorrow at the latest.  Test to protect the
>> fix can come as a separate follow-up patch.
>
> I actually changed my mind and decided not to rush 2.16.1 with just
> this change.  After all, even though it is better not to crash in
> such a problematic case, a "successful" clone of such a project on
> such a filesystem cannot be sanely used *anyway*, so in that sense
> the "fix" would stop "clone" from segfaulting but the resulting
> repository would still be "wrong" (e.g. "git status" immediately
> after clone would likely to say that the working tree is already
> dirty and missing one of the two paths, or something silly like
> that).

The counter-argument is that filenames differing only in case do not
necessarily render a project unusable on case-insensitive filesystems.
For instance, if the problematic files exist only in a project's test
suite or in some platform-specific resource directory, the project
itself may still be perfectly usable for a person cloning a project
merely to build and use it (not develop it). However, "clone" crashing
_does_ render the project unusable for the same person.

The crash[1] when cloning 'tcell' is an example of a project which is
still 100% usable on case-insensitive filesystems despite
case-conflicting filenames. Each of the conflicting file pairs [2] has
identical content, so everything works as intended.

Case-conflicting filenames also do not necessarily make it impossible
to do development work on a project. I have, myself, on several
occasions cloned such projects on MacOS to make improvements and track
down and/or fix bugs in parts of the projects not impacted by the
case-conflicting filenames.

Footnotes:

[1]: https://public-inbox.org/git/20180119180855.GA98561@smm.local/

[2]: case-conflicting files in tcell's bundled terminfo database:
    terminfo/database/32/2621A
    terminfo/database/32/2621a
    terminfo/database/68/hp2621A
    terminfo/database/68/hp2621a
    terminfo/database/68/hp70092A
    terminfo/database/68/hp70092a

[PATCH] t: add clone test for files differing only in case

[brian m. carlson]

We recently introduced a regression in cloning repositories onto
case-insensitive file systems where the repository contains multiple
files differing only in case.  In such a case, we would segfault.  This
segfault has already been fixed (repository: pre-initialize hash algo
pointer), but it's not the first time similar problems have arisen.
Introduce a test to catch this case and ensure the behavior does not
regress.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5601-clone.sh | 13 +++++++++++++
 1 file changed, 13 insertions(+)

I've verified that the test does fail without the patch on a vfat file
system.  However, many other tests also fail on a vfat file system on
Linux, so unfortunately that doesn't look like a viable testing strategy
going forward.

I didn't include an object ID for the commit referenced simply because I
didn't see one yet and I didn't want to insert a local one that wouldn't
work for anyone else.

diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index 0f895478f0..53b2dda9d2 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -611,4 +611,17 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' '
 	git -C replay.git index-pack -v --stdin <tmp.pack
 '
 
+hex2oct() {
+	perl -ne 'printf "\\%03o", hex for /../g'
+}
+
+test_expect_success 'clone on case-insensitive fs' '
+	o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
+	t=$(printf "100644 X\0${o}100644 x\0${o}" |
+		git hash-object -w -t tree --stdin) &&
+	c=$(git commit-tree -m bogus $t) &&
+	git update-ref refs/heads/bogus $c &&
+	git clone -b bogus . bogus
+'
+
 test_done

Re: [PATCH] t: add clone test for files differing only in case

[Eric Sunshine]

On Sat, Jan 20, 2018 at 3:33 PM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> We recently introduced a regression in cloning repositories onto
> case-insensitive file systems where the repository contains multiple
> files differing only in case.  In such a case, we would segfault.  This
> segfault has already been fixed (repository: pre-initialize hash algo
> pointer), but it's not the first time similar problems have arisen.
> Introduce a test to catch this case and ensure the behavior does not
> regress.

I guess you'd probably want a "From: Eric Sunshine <sunshine@sunshineco.com>"
header before this paragraph.

The patch itself looks correct... ;-)

> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
> index 0f895478f0..53b2dda9d2 100755
> --- a/t/t5601-clone.sh
> +++ b/t/t5601-clone.sh
> @@ -611,4 +611,17 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' '
>         git -C replay.git index-pack -v --stdin <tmp.pack
>  '
>
> +hex2oct() {
> +       perl -ne 'printf "\\%03o", hex for /../g'
> +}
> +
> +test_expect_success 'clone on case-insensitive fs' '
> +       o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
> +       t=$(printf "100644 X\0${o}100644 x\0${o}" |
> +               git hash-object -w -t tree --stdin) &&
> +       c=$(git commit-tree -m bogus $t) &&
> +       git update-ref refs/heads/bogus $c &&
> +       git clone -b bogus . bogus
> +'
> +
>  test_done

Re: [PATCH] t: add clone test for files differing only in case

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
> index 0f895478f0..53b2dda9d2 100755
> --- a/t/t5601-clone.sh
> +++ b/t/t5601-clone.sh
> @@ -611,4 +611,17 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' '
>  	git -C replay.git index-pack -v --stdin <tmp.pack
>  '
>  
> +hex2oct() {
> +	perl -ne 'printf "\\%03o", hex for /../g'
> +}
> +
> +test_expect_success 'clone on case-insensitive fs' '
> +	o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
> +	t=$(printf "100644 X\0${o}100644 x\0${o}" |
> +		git hash-object -w -t tree --stdin) &&
> +	c=$(git commit-tree -m bogus $t) &&
> +	git update-ref refs/heads/bogus $c &&
> +	git clone -b bogus . bogus
> +'
> +
>  test_done

Hmm, I seem to be seeing a failure from this thing:

    expecting success:
            o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
            t=$(printf "100644 X\0${o}100644 x\0${o}" |
                    git hash-object -w -t tree --stdin) &&
            c=$(git commit-tree -m bogus $t) &&
            git update-ref refs/heads/bogus $c &&
            git clone -b bogus . bogus

    fatal: repository '.' does not exist

even on a case sensitive platform.

Re: [PATCH] t: add clone test for files differing only in case

[Eric Sunshine]

On Sun, Jan 21, 2018 at 2:33 AM, Junio C Hamano <gitster@pobox.com> wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
>> +test_expect_success 'clone on case-insensitive fs' '
>> +     o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
>> +     t=$(printf "100644 X\0${o}100644 x\0${o}" |
>> +             git hash-object -w -t tree --stdin) &&
>> +     c=$(git commit-tree -m bogus $t) &&
>> +     git update-ref refs/heads/bogus $c &&
>> +     git clone -b bogus . bogus
>> +'
>> +
>>  test_done
>
> Hmm, I seem to be seeing a failure from this thing:
>
>     expecting success:
>             o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
>             t=$(printf "100644 X\0${o}100644 x\0${o}" |
>                     git hash-object -w -t tree --stdin) &&
>             c=$(git commit-tree -m bogus $t) &&
>             git update-ref refs/heads/bogus $c &&
>             git clone -b bogus . bogus
>
>     fatal: repository '.' does not exist
>
> even on a case sensitive platform.

Yep. In pretty much any other test script, this would work (it was
developed in a stand-alone script), but t5601 (which nukes .git as its
first action) isn't the most friendly place.

Re: [PATCH] t: add clone test for files differing only in case

[Eric Sunshine]

On Sun, Jan 21, 2018 at 02:46:15AM -0500, Eric Sunshine wrote:
> On Sun, Jan 21, 2018 at 2:33 AM, Junio C Hamano <gitster@pobox.com> wrote:
> > "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> >> +test_expect_success 'clone on case-insensitive fs' '
> >> +     o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
> >> +     t=$(printf "100644 X\0${o}100644 x\0${o}" |
> >> +             git hash-object -w -t tree --stdin) &&
> >> +     c=$(git commit-tree -m bogus $t) &&
> >> +     git update-ref refs/heads/bogus $c &&
> >> +     git clone -b bogus . bogus
> >> +'
> >
> > Hmm, I seem to be seeing a failure from this thing:
> >     fatal: repository '.' does not exist
> > even on a case sensitive platform.
> 
> Yep. In pretty much any other test script, this would work (it was
> developed in a stand-alone script), but t5601 (which nukes .git as its
> first action) isn't the most friendly place.

Here's a re-roll which fixes that problem (and has a slightly
re-written commit message.

--- >8 ---
From: Eric Sunshine <sunshine@sunshineco.com>
Subject: [PATCH] t5601-clone: test case-conflicting files on case-insensitive
 filesystem

A recently introduced regression caused a segfault at clone time on
case-insensitive filesystems when filenames differing only in case are
present. This bug has already been fixed (repository: pre-initialize
hash algo pointer, 2018-01-18), but it's not the first time similar
problems have arisen. Therefore, introduce a test to catch this case and
protect against future regressions.

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
---
 t/t5601-clone.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index 0f895478f0..2d1490f631 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -611,4 +611,21 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' '
 	git -C replay.git index-pack -v --stdin <tmp.pack
 '
 
+hex2oct() {
+	perl -ne 'printf "\\%03o", hex for /../g'
+}
+
+test_expect_success 'clone on case-insensitive fs' '
+	git init icasefs &&
+	(
+		cd icasefs
+		o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
+		t=$(printf "100644 X\0${o}100644 x\0${o}" |
+			git hash-object -w -t tree --stdin) &&
+		c=$(git commit-tree -m bogus $t) &&
+		git update-ref refs/heads/bogus $c &&
+		git clone -b bogus . bogus
+	)
+'
+
 test_done
-- 
2.16.0.312.g896df04e46

Re: [PATCH] t: add clone test for files differing only in case

[Duy Nguyen]

On Sun, Jan 21, 2018 at 3:33 AM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> We recently introduced a regression in cloning repositories onto
> case-insensitive file systems where the repository contains multiple
> files differing only in case.  In such a case, we would segfault.  This
> segfault has already been fixed (repository: pre-initialize hash algo
> pointer), but it's not the first time similar problems have arisen.
> Introduce a test to catch this case and ensure the behavior does not
> regress.
>
> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5601-clone.sh | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> I've verified that the test does fail without the patch on a vfat file
> system.  However, many other tests also fail on a vfat file system on
> Linux, so unfortunately that doesn't look like a viable testing strategy
> going forward.

vfat is not very UNIXy. I suspect we need a bunch of tweaks that
cygwin/mingw uses to make the test suite pass. There are other
case-insensitive filesystems on linux though. JFS (with "mkfs.jfs -O"
to make it case-insensitive) seems to pass the test suite and crash
git without your fix so it's a good candidate if someone wants to test
this behavior on linux in future. HFS+ fails some unicode test in
t0050 and I didn't check further.

> +test_expect_success 'clone on case-insensitive fs' '

We have CASE_INSENSITIVE_FS prereq. Should we use it here? I know it
does not harm running this test on case-sensitive filesystem, but the
prereq could be useful for grepping.

> +       o=$(git hash-object -w --stdin </dev/null | hex2oct) &&
> +       t=$(printf "100644 X\0${o}100644 x\0${o}" |
> +               git hash-object -w -t tree --stdin) &&
> +       c=$(git commit-tree -m bogus $t) &&
> +       git update-ref refs/heads/bogus $c &&
> +       git clone -b bogus . bogus
> +'
> +
>  test_done
-- 
Duy

Re: [PATCH] t: add clone test for files differing only in case

[Eric Sunshine]

On Sun, Jan 21, 2018 at 6:50 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Sun, Jan 21, 2018 at 3:33 AM, brian m. carlson
>> +test_expect_success 'clone on case-insensitive fs' '
>
> We have CASE_INSENSITIVE_FS prereq. Should we use it here? I know it
> does not harm running this test on case-sensitive filesystem, but the
> prereq could be useful for grepping.

I'd rather not hide it behind the CASE_INSENSITIVE_FS[1] prerequisite
since the test potentially could catch some sort of future regression
even on case-sensitive filesystems.

[1]: Todd Zullinger suggested the same:
https://public-inbox.org/git/CAPig+cSRN1zHc=zsO1Y_aQ_eO+sbsd0cq5iZ9hYz3ruK_E-0Dw@mail.gmail.com/

Re: [PATCH] t: add clone test for files differing only in case

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

On Sun, Jan 21, 2018 at 03:07:28AM -0500, Eric Sunshine wrote:
> On Sun, Jan 21, 2018 at 02:46:15AM -0500, Eric Sunshine wrote:
> > Yep. In pretty much any other test script, this would work (it was
> > developed in a stand-alone script), but t5601 (which nukes .git as its
> > first action) isn't the most friendly place.
> 
> Here's a re-roll which fixes that problem (and has a slightly
> re-written commit message.

Yes, this looks good.  I did indeed intend to forge your From line on
the message and actually test that it was fixed, but only tested it on
the vfat partition, not my ext4 partition, where, of course, I got
distracted by the various other failures.  Sigh.

Thanks for fixing this up.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]