* git 2.16.0 segfaults on clone of specific repo @ 2018-01-18 20:55 Александр Булаев 2018-01-18 21:03 ` Randall S. Becker 2018-01-19 0:15 ` Eric Sunshine 0 siblings, 2 replies; 33+ messages in thread From: Александр Булаев @ 2018-01-18 20:55 UTC (permalink / raw) To: git I found that git 2.16.0 segfaults on clone of vim-colorschemes repo. See the log below. alexbool@alexbool-osx ~/Documents> lldb -- git clone https://github.com/flazz/vim-colorschemes.git (lldb) target create "git" Current executable set to 'git' (x86_64). (lldb) settings set -- target.run-args "clone" "https://github.com/flazz/vim-colorschemes.git" (lldb) run Process 25643 launched: '/usr/local/bin/git' (x86_64) Cloning into 'vim-colorschemes'... remote: Counting objects: 1457, done. remote: Total 1457 (delta 0), reused 0 (delta 0), pack-reused 1457 Receiving objects: 100% (1457/1457), 1.43 MiB | 289.00 KiB/s, done. Resolving deltas: 100% (424/424), done. Process 25643 stopped * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x48) frame #0: 0x00000001000f7c3d git`ce_match_stat_basic + 263 git`ce_match_stat_basic: -> 0x1000f7c3d <+263>: movq 0x48(%rax), %rsi 0x1000f7c41 <+267>: movl $0x14, %edx 0x1000f7c46 <+272>: movq %r14, %rdi 0x1000f7c49 <+275>: callq 0x1001659fa ; symbol stub for: memcmp Target 0: (git) stopped. (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x48) * frame #0: 0x00000001000f7c3d git`ce_match_stat_basic + 263 frame #1: 0x00000001000f7ae8 git`ie_match_stat + 108 frame #2: 0x00000001000bbe07 git`checkout_entry + 266 frame #3: 0x0000000100144ff9 git`unpack_trees + 2317 frame #4: 0x0000000100017e13 git`cmd_clone + 5733 frame #5: 0x0000000100001ea3 git`handle_builtin + 532 frame #6: 0x0000000100001954 git`cmd_main + 263 frame #7: 0x0000000100079a28 git`main + 80 frame #8: 0x00007fff64d76115 libdyld.dylib`start + 1 (lldb) ^D alexbool@alexbool-osx ~/Documents> git --version git version 2.16.0 OS: macOS 10.13.2 git installed from homebrew ^ permalink raw reply [flat|nested] 33+ messages in thread
* RE: git 2.16.0 segfaults on clone of specific repo 2018-01-18 20:55 git 2.16.0 segfaults on clone of specific repo Александр Булаев @ 2018-01-18 21:03 ` Randall S. Becker 2018-01-19 0:15 ` Eric Sunshine 1 sibling, 0 replies; 33+ messages in thread From: Randall S. Becker @ 2018-01-18 21:03 UTC (permalink / raw) To: 'Александр Булаев', git On January 18, 2018 3:56 PM, Aleks wrote: > I found that git 2.16.0 segfaults on clone of vim-colorschemes repo. <snip> Just tested on NonStop NSE and works fine here. Just an FYI now that we're on 2.16.0. Cheers, Randall -- Brief whoami: NonStop developer since approximately NonStop(211288444200000000) UNIX developer since approximately 421664400 -- In my real life, I talk too much. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-18 20:55 git 2.16.0 segfaults on clone of specific repo Александр Булаев 2018-01-18 21:03 ` Randall S. Becker @ 2018-01-19 0:15 ` Eric Sunshine 2018-01-19 2:47 ` brian m. carlson 1 sibling, 1 reply; 33+ messages in thread From: Eric Sunshine @ 2018-01-19 0:15 UTC (permalink / raw) To: Александр Булаев Cc: Git List, brian m. carlson [+cc:brian] On Thu, Jan 18, 2018 at 3:55 PM, Александр Булаев <aleks.bulaev@gmail.com> wrote: > I found that git 2.16.0 segfaults on clone of vim-colorschemes repo. > > (lldb) run > Process 25643 launched: '/usr/local/bin/git' (x86_64) > Cloning into 'vim-colorschemes'... > remote: Counting objects: 1457, done. > remote: Total 1457 (delta 0), reused 0 (delta 0), pack-reused 1457 > Receiving objects: 100% (1457/1457), 1.43 MiB | 289.00 KiB/s, done. > Resolving deltas: 100% (424/424), done. > Process 25643 stopped > * thread #1, queue = 'com.apple.main-thread', stop reason = > EXC_BAD_ACCESS (code=1, address=0x48) I can confirm that this crashes on MacOS; it does not crash on Linux or BSD. git-bisect places blame on eb0ccfd7f5 (Switch empty tree and blob lookups to use hash abstraction, 2017-11-12). ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 0:15 ` Eric Sunshine @ 2018-01-19 2:47 ` brian m. carlson 2018-01-19 3:06 ` Eric Sunshine 0 siblings, 1 reply; 33+ messages in thread From: brian m. carlson @ 2018-01-19 2:47 UTC (permalink / raw) To: Eric Sunshine Cc: Александр Булаев, Git List [-- Attachment #1: Type: text/plain, Size: 2669 bytes --] On Thu, Jan 18, 2018 at 07:15:56PM -0500, Eric Sunshine wrote: > [+cc:brian] > > On Thu, Jan 18, 2018 at 3:55 PM, Александр Булаев > <aleks.bulaev@gmail.com> wrote: > > I found that git 2.16.0 segfaults on clone of vim-colorschemes repo. > > > > (lldb) run > > Process 25643 launched: '/usr/local/bin/git' (x86_64) > > Cloning into 'vim-colorschemes'... > > remote: Counting objects: 1457, done. > > remote: Total 1457 (delta 0), reused 0 (delta 0), pack-reused 1457 > > Receiving objects: 100% (1457/1457), 1.43 MiB | 289.00 KiB/s, done. > > Resolving deltas: 100% (424/424), done. > > Process 25643 stopped > > * thread #1, queue = 'com.apple.main-thread', stop reason = > > EXC_BAD_ACCESS (code=1, address=0x48) > > I can confirm that this crashes on MacOS; it does not crash on Linux or BSD. > > git-bisect places blame on eb0ccfd7f5 (Switch empty tree and blob > lookups to use hash abstraction, 2017-11-12). I unfortunately don't have a macOS system to test with, and I've compiled with both gcc and clang on my Debian system and, as you mentioned, it doesn't fail there. I have a guess about what the problem might be. Can you try this patch and see if it fixes things? -- >8 -- From 10b690241619a452634b31fbc5ccd054a4f6e5ec Mon Sep 17 00:00:00 2001 From: "brian m. carlson" <sandals@crustytoothpaste.net> Date: Sun, 14 Jan 2018 18:26:29 +0000 Subject: [PATCH] repository: pre-initialize hash algo pointer There are various git subcommands (among them, clone) which don't set up the repository but end up needing to have information about the hash algorithm in use. In the future, we can add a command line option for this or read it from the configuration, but until we're ready to expose that functionality to the user, simply initialize the repository structure to use the current hash algorithm, SHA-1. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> --- repository.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/repository.c b/repository.c index 998413b8bb..f66fcb1342 100644 --- a/repository.c +++ b/repository.c @@ -5,7 +5,7 @@ /* The main repository */ static struct repository the_repo = { - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0 + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0 }; struct repository *the_repository = &the_repo; -- 2.16.0.rc2.280.g09355b536d -- >8 -- -- brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 867 bytes --] ^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 2:47 ` brian m. carlson @ 2018-01-19 3:06 ` Eric Sunshine 2018-01-19 3:39 ` Randall S. Becker ` (2 more replies) 0 siblings, 3 replies; 33+ messages in thread From: Eric Sunshine @ 2018-01-19 3:06 UTC (permalink / raw) To: brian m. carlson, Eric Sunshine, Александр Булаев, Git List On Thu, Jan 18, 2018 at 9:47 PM, brian m. carlson <sandals@crustytoothpaste.net> wrote: > On Thu, Jan 18, 2018 at 07:15:56PM -0500, Eric Sunshine wrote: >> On Thu, Jan 18, 2018 at 3:55 PM, Александр Булаев >> <aleks.bulaev@gmail.com> wrote: >> > I found that git 2.16.0 segfaults on clone of vim-colorschemes repo. >> >> I can confirm that this crashes on MacOS; it does not crash on Linux or BSD. >> >> git-bisect places blame on eb0ccfd7f5 (Switch empty tree and blob >> lookups to use hash abstraction, 2017-11-12). > > I unfortunately don't have a macOS system to test with, and I've > compiled with both gcc and clang on my Debian system and, as you > mentioned, it doesn't fail there. > > I have a guess about what the problem might be. Can you try this patch > and see if it fixes things? That does fix the crash. Thanks for the quick diagnosis. Can the commit message go into more detail as to why this was crashing (or your speculation about why)? Perhaps give more detail about what 'clone' is doing that led to the crash. > -- >8 -- > From 10b690241619a452634b31fbc5ccd054a4f6e5ec Mon Sep 17 00:00:00 2001 > From: "brian m. carlson" <sandals@crustytoothpaste.net> > Date: Sun, 14 Jan 2018 18:26:29 +0000 > Subject: [PATCH] repository: pre-initialize hash algo pointer > > There are various git subcommands (among them, clone) which don't set up > the repository but end up needing to have information about the hash > algorithm in use. In the future, we can add a command line option for > this or read it from the configuration, but until we're ready to expose > that functionality to the user, simply initialize the repository > structure to use the current hash algorithm, SHA-1. > > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> > --- > repository.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/repository.c b/repository.c > index 998413b8bb..f66fcb1342 100644 > --- a/repository.c > +++ b/repository.c > @@ -5,7 +5,7 @@ > > /* The main repository */ > static struct repository the_repo = { > - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0 > + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0 > }; > struct repository *the_repository = &the_repo; > > -- ^ permalink raw reply [flat|nested] 33+ messages in thread
* RE: git 2.16.0 segfaults on clone of specific repo 2018-01-19 3:06 ` Eric Sunshine @ 2018-01-19 3:39 ` Randall S. Becker 2018-01-19 3:40 ` brian m. carlson 2018-01-19 4:18 ` [PATCH] repository: pre-initialize hash algo pointer brian m. carlson 2 siblings, 0 replies; 33+ messages in thread From: Randall S. Becker @ 2018-01-19 3:39 UTC (permalink / raw) To: 'Eric Sunshine', 'brian m. carlson', 'Александр Булаев', 'Git List' On January 18, 2018 10:06 PM, Eric Sunshine wrote: > On Thu, Jan 18, 2018 at 9:47 PM, brian m. carlson > <sandals@crustytoothpaste.net> wrote: > > On Thu, Jan 18, 2018 at 07:15:56PM -0500, Eric Sunshine wrote: > >> On Thu, Jan 18, 2018 at 3:55 PM, Александр Булаев > >> <aleks.bulaev@gmail.com> wrote: > >> > I found that git 2.16.0 segfaults on clone of vim-colorschemes repo. > >> > >> I can confirm that this crashes on MacOS; it does not crash on Linux or > BSD. > >> > >> git-bisect places blame on eb0ccfd7f5 (Switch empty tree and blob > >> lookups to use hash abstraction, 2017-11-12). > > > > I unfortunately don't have a macOS system to test with, and I've > > compiled with both gcc and clang on my Debian system and, as you > > mentioned, it doesn't fail there. > > > > I have a guess about what the problem might be. Can you try this > > patch and see if it fixes things? > > That does fix the crash. Thanks for the quick diagnosis. > > Can the commit message go into more detail as to why this was crashing (or > your speculation about why)? Perhaps give more detail about what 'clone' is > doing that led to the crash. I'm curious as to why this worked on my platform, given how it tends to get annoyed with NULL in the wrong place. > > > -- >8 -- > > From 10b690241619a452634b31fbc5ccd054a4f6e5ec Mon Sep 17 00:00:00 > 2001 > > From: "brian m. carlson" <sandals@crustytoothpaste.net> > > Date: Sun, 14 Jan 2018 18:26:29 +0000 > > Subject: [PATCH] repository: pre-initialize hash algo pointer > > > > There are various git subcommands (among them, clone) which don't set > > up the repository but end up needing to have information about the > > hash algorithm in use. In the future, we can add a command line > > option for this or read it from the configuration, but until we're > > ready to expose that functionality to the user, simply initialize the > > repository structure to use the current hash algorithm, SHA-1. > > > > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> > > --- > > repository.c | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/repository.c b/repository.c index 998413b8bb..f66fcb1342 > > 100644 > > --- a/repository.c > > +++ b/repository.c > > @@ -5,7 +5,7 @@ > > > > /* The main repository */ > > static struct repository the_repo = { > > - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, > NULL, 0, 0 > > + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, > > + &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0 > > }; > > struct repository *the_repository = &the_repo; > > > > -- ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 3:06 ` Eric Sunshine 2018-01-19 3:39 ` Randall S. Becker @ 2018-01-19 3:40 ` brian m. carlson 2018-01-19 5:31 ` Duy Nguyen 2018-01-19 4:18 ` [PATCH] repository: pre-initialize hash algo pointer brian m. carlson 2 siblings, 1 reply; 33+ messages in thread From: brian m. carlson @ 2018-01-19 3:40 UTC (permalink / raw) To: Eric Sunshine Cc: Александр Булаев, Git List [-- Attachment #1: Type: text/plain, Size: 1158 bytes --] On Thu, Jan 18, 2018 at 10:06:10PM -0500, Eric Sunshine wrote: > > I have a guess about what the problem might be. Can you try this patch > > and see if it fixes things? > > That does fix the crash. Thanks for the quick diagnosis. > > Can the commit message go into more detail as to why this was crashing > (or your speculation about why)? Perhaps give more detail about what > 'clone' is doing that led to the crash. Sure. I ran into this as I was expanding the hash structure abstraction into my next series. I'll send a follow-up patch with a more descriptive answer. I'm still extremely puzzled as to why this doesn't fail on Linux. If it's failing in this case, it should very, very clearly fail all the time we access an empty blob or an empty tree. I've tried with gcc and two versions of clang, using -fno-lto, with address sanitizer, with -O0, and so on. I'd really like to catch this kind of issue sooner in the future if I can figure out how to reproduce it. -- brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 867 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 3:40 ` brian m. carlson @ 2018-01-19 5:31 ` Duy Nguyen 2018-01-19 7:40 ` Eric Sunshine 0 siblings, 1 reply; 33+ messages in thread From: Duy Nguyen @ 2018-01-19 5:31 UTC (permalink / raw) To: brian m. carlson, Eric Sunshine, Александр Булаев, Git List On Fri, Jan 19, 2018 at 10:40 AM, brian m. carlson <sandals@crustytoothpaste.net> wrote: > On Thu, Jan 18, 2018 at 10:06:10PM -0500, Eric Sunshine wrote: >> > I have a guess about what the problem might be. Can you try this patch >> > and see if it fixes things? >> >> That does fix the crash. Thanks for the quick diagnosis. >> >> Can the commit message go into more detail as to why this was crashing >> (or your speculation about why)? Perhaps give more detail about what >> 'clone' is doing that led to the crash. > > Sure. I ran into this as I was expanding the hash structure abstraction > into my next series. I'll send a follow-up patch with a more > descriptive answer. > > I'm still extremely puzzled as to why this doesn't fail on Linux. If > it's failing in this case, it should very, very clearly fail all the > time we access an empty blob or an empty tree. I've tried with gcc and > two versions of clang, using -fno-lto, with address sanitizer, with -O0, > and so on. I'd really like to catch this kind of issue sooner in the > future if I can figure out how to reproduce it. I think it's file system related, case-insensitive one in particular. The call trace posted at the beginning of this thread should never trigger for an initial clone. You only check if an _existing_ entry matches what you are about to checkout when you switch trees. For this to happen at clone time, I suppose you have to checkout entry "A", then re-checkout "A" again. Which can only happen on case-insensitive file systems and a case-sensitive repo where the second "A" might actually be "a". vim-colorschemes.git has these two entries, which meets one of the conditions, I suppose macos meets the other colors/darkblue.vim colors/darkBlue.vim On Linux, after I hacked all over the place to force ce_match_stat() to eventually call index_fd() which in turns must use one of the hashing function, I got a crash. -- Duy ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 5:31 ` Duy Nguyen @ 2018-01-19 7:40 ` Eric Sunshine 2018-01-19 8:22 ` Duy Nguyen ` (2 more replies) 0 siblings, 3 replies; 33+ messages in thread From: Eric Sunshine @ 2018-01-19 7:40 UTC (permalink / raw) To: Duy Nguyen Cc: brian m. carlson, Александр Булаев, Git List On Fri, Jan 19, 2018 at 12:31:58PM +0700, Duy Nguyen wrote: > On Fri, Jan 19, 2018 at 10:40 AM, brian m. carlson > > I'm still extremely puzzled as to why this doesn't fail on Linux. If > > it's failing in this case, it should very, very clearly fail all the > > time we access an empty blob or an empty tree.[...] > > I think it's file system related, case-insensitive one in particular. > > The call trace posted at the beginning of this thread should never > trigger for an initial clone. You only check if an _existing_ entry > matches what you are about to checkout when you switch trees. For this > to happen at clone time, I suppose you have to checkout entry "A", > then re-checkout "A" again. Which can only happen on case-insensitive > file systems and a case-sensitive repo where the second "A" might > actually be "a". > [...] > On Linux, after I hacked all over the place to force ce_match_stat() > to eventually call index_fd() which in turns must use one of the > hashing function, I got a crash. Nice detective work. This particular manifestation is caught by the following test which fails without brian's patch on MacOS (and presumably Windows) and succeeds with it. On Linux and BSD, it will of course succeed always, so I'm not sure how much practical value it has. --- >8 --- hex2oct() { perl -ne 'printf "\\%03o", hex for /../g' } test_expect_success 'clone on case-insensitive fs' ' o=$(git hash-object -w --stdin </dev/null | hex2oct) && t=$(printf "100644 X\0${o}100644 x\0${o}" | git hash-object -w -t tree --stdin) && c=$(git commit-tree -m bogus $t) && git update-ref refs/heads/bogus $c && git clone -b bogus . bogus ' --- >8 --- (hex2oct lifted from t1007/t1450) ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 7:40 ` Eric Sunshine @ 2018-01-19 8:22 ` Duy Nguyen 2018-01-19 8:28 ` Eric Sunshine 2018-01-19 17:25 ` Todd Zullinger 2018-01-19 22:31 ` brian m. carlson 2 siblings, 1 reply; 33+ messages in thread From: Duy Nguyen @ 2018-01-19 8:22 UTC (permalink / raw) To: Eric Sunshine Cc: brian m. carlson, Александр Булаев, Git List On Fri, Jan 19, 2018 at 2:40 PM, Eric Sunshine <sunshine@sunshineco.com> wrote: > On Fri, Jan 19, 2018 at 12:31:58PM +0700, Duy Nguyen wrote: >> On Fri, Jan 19, 2018 at 10:40 AM, brian m. carlson >> > I'm still extremely puzzled as to why this doesn't fail on Linux. If >> > it's failing in this case, it should very, very clearly fail all the >> > time we access an empty blob or an empty tree.[...] >> >> I think it's file system related, case-insensitive one in particular. >> >> The call trace posted at the beginning of this thread should never >> trigger for an initial clone. You only check if an _existing_ entry >> matches what you are about to checkout when you switch trees. For this >> to happen at clone time, I suppose you have to checkout entry "A", >> then re-checkout "A" again. Which can only happen on case-insensitive >> file systems and a case-sensitive repo where the second "A" might >> actually be "a". >> [...] >> On Linux, after I hacked all over the place to force ce_match_stat() >> to eventually call index_fd() which in turns must use one of the >> hashing function, I got a crash. > > Nice detective work. And not stepping back to think for a bit. I realized now that if I just mounted a vfat filesystem, I could have verified it much quicker. It does crash on linux with similar stack trace. (gdb) bt #0 0x000000000055809f in is_empty_blob_sha1 (sha1=0x8fa6d4 "\236") at cache.h:1055 #1 0x0000000000558acd in ce_match_stat_basic (ce=0x8fa690, st=0x7fffffffcd20) at read-cache.c:272 #2 0x0000000000558c78 in ie_match_stat (istate=0x8e9fc0 <the_index>, ce=0x8fa690, st=0x7fffffffcd20, options=5) at read-cache.c:342 #3 0x0000000000501e01 in checkout_entry (ce=0x8fa690, state=0x7fffffffcea0, topath=0x0) at entry.c:424 #4 0x00000000005c8ea0 in check_updates (o=0x7fffffffd060) at unpack-trees.c:383 #5 0x00000000005cb2bb in unpack_trees (len=1, t=0x7fffffffd010, o=0x7fffffffd060) at unpack-trees.c:1382 #6 0x00000000004214ca in checkout (submodule_progress=1) at builtin/clone.c:750 #7 0x00000000004229ce in cmd_clone (argc=1, argv=0x7fffffffd840, prefix=0x0) at builtin/clone.c:1191 #8 0x0000000000405846 in run_builtin (p=0x89a908 <commands+456>, argc=2, argv=0x7fffffffd840) at git.c:346 #9 0x0000000000405af7 in handle_builtin (argc=2, argv=0x7fffffffd840) at git.c:554 #10 0x0000000000405c8f in run_argv (argcp=0x7fffffffd6fc, argv=0x7fffffffd6f0) at git.c:606 #11 0x0000000000405e31 in cmd_main (argc=2, argv=0x7fffffffd840) at git.c:683 #12 0x00000000004a3231 in main (argc=3, argv=0x7fffffffd838) at common-main.c:43 > This particular manifestation is caught by the > following test which fails without brian's patch on MacOS (and > presumably Windows) and succeeds with it. On Linux and BSD, it will of > course succeed always, so I'm not sure how much practical value it > has. There's a travis job running "on windows" (I don't what it really means) so it might help actually. This vim-colorschemes repository has shown up in git mailing list before, I think. We probably should just add it as part of our regression tests ;-) -- Duy ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 8:22 ` Duy Nguyen @ 2018-01-19 8:28 ` Eric Sunshine 0 siblings, 0 replies; 33+ messages in thread From: Eric Sunshine @ 2018-01-19 8:28 UTC (permalink / raw) To: Duy Nguyen Cc: brian m. carlson, Александр Булаев, Git List, Johannes Schindelin, Johannes Schindelin On Fri, Jan 19, 2018 at 3:22 AM, Duy Nguyen <pclouds@gmail.com> wrote: > On Fri, Jan 19, 2018 at 2:40 PM, Eric Sunshine <sunshine@sunshineco.com> wrote: >> On Fri, Jan 19, 2018 at 12:31:58PM +0700, Duy Nguyen wrote: >>> On Linux, after I hacked all over the place to force ce_match_stat() >>> to eventually call index_fd() which in turns must use one of the >>> hashing function, I got a crash. >> >> Nice detective work. > > And not stepping back to think for a bit. I realized now that if I > just mounted a vfat filesystem, I could have verified it much quicker. > It does crash on linux with similar stack trace. > [...] >> This particular manifestation is caught by the >> following test which fails without brian's patch on MacOS (and >> presumably Windows) and succeeds with it. On Linux and BSD, it will of >> course succeed always, so I'm not sure how much practical value it >> has. > > There's a travis job running "on windows" (I don't what it really > means) so it might help actually. This vim-colorschemes repository has > shown up in git mailing list before, I think. We probably should just > add it as part of our regression tests ;-) And, Dscho does run the test suite on Windows regularly, so perhaps this test does have some practical value. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 7:40 ` Eric Sunshine 2018-01-19 8:22 ` Duy Nguyen @ 2018-01-19 17:25 ` Todd Zullinger 2018-01-20 0:23 ` Eric Sunshine 2018-01-19 22:31 ` brian m. carlson 2 siblings, 1 reply; 33+ messages in thread From: Todd Zullinger @ 2018-01-19 17:25 UTC (permalink / raw) To: Eric Sunshine Cc: Duy Nguyen, brian m. carlson, Александр Булаев, Git List Eric Sunshine wrote: > Nice detective work. This particular manifestation is caught by the > following test which fails without brian's patch on MacOS (and > presumably Windows) and succeeds with it. On Linux and BSD, it will of > course succeed always, so I'm not sure how much practical value it > has. The CASE_INSENSITIVE_FS prereq could be used to avoid running the test on systems where it won't provide much value, couldn't it? > --- >8 --- > hex2oct() { > perl -ne 'printf "\\%03o", hex for /../g' > } > > test_expect_success 'clone on case-insensitive fs' ' > o=$(git hash-object -w --stdin </dev/null | hex2oct) && > t=$(printf "100644 X\0${o}100644 x\0${o}" | > git hash-object -w -t tree --stdin) && > c=$(git commit-tree -m bogus $t) && > git update-ref refs/heads/bogus $c && > git clone -b bogus . bogus > ' > --- >8 --- > > (hex2oct lifted from t1007/t1450) -- Todd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Money frees you from doing things you dislike. Since I dislike doing nearly everything, money is handy. -- Groucho Marx ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 17:25 ` Todd Zullinger @ 2018-01-20 0:23 ` Eric Sunshine 0 siblings, 0 replies; 33+ messages in thread From: Eric Sunshine @ 2018-01-20 0:23 UTC (permalink / raw) To: Todd Zullinger Cc: Duy Nguyen, brian m. carlson, Александр Булаев, Git List On Fri, Jan 19, 2018 at 12:25 PM, Todd Zullinger <tmz@pobox.com> wrote: > Eric Sunshine wrote: >> Nice detective work. This particular manifestation is caught by the >> following test which fails without brian's patch on MacOS (and >> presumably Windows) and succeeds with it. On Linux and BSD, it will of >> course succeed always, so I'm not sure how much practical value it >> has. > > The CASE_INSENSITIVE_FS prereq could be used to avoid > running the test on systems where it won't provide much > value, couldn't it? As mentioned in [1], my original questioning of the practical value of the test was tied to the belief that the full test suite is probably rarely run on MacOS, and it wasn't until I pressed "Send" that it hit me that it would manifest on Windows as well, and that Dscho does test on Windows regularly. Therefore, the test could have value. At any rate, the test potentially could catch some sort of future regression even on case-sensitive filesystems, therefore, it would be better not to hide it behind CASE_INSENSITIVE_FS. [1]: https://public-inbox.org/git/CAPig+cQmWqQWQrRQHHn=3hn6UFzJxT=9d5kKnJht_dt8sCgwkQ@mail.gmail.com/ ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 7:40 ` Eric Sunshine 2018-01-19 8:22 ` Duy Nguyen 2018-01-19 17:25 ` Todd Zullinger @ 2018-01-19 22:31 ` brian m. carlson 2018-01-20 0:15 ` Eric Sunshine 2 siblings, 1 reply; 33+ messages in thread From: brian m. carlson @ 2018-01-19 22:31 UTC (permalink / raw) To: Eric Sunshine Cc: Duy Nguyen, Александр Булаев, Git List [-- Attachment #1: Type: text/plain, Size: 1287 bytes --] On Fri, Jan 19, 2018 at 02:40:02AM -0500, Eric Sunshine wrote: > Nice detective work. This particular manifestation is caught by the > following test which fails without brian's patch on MacOS (and > presumably Windows) and succeeds with it. On Linux and BSD, it will of > course succeed always, so I'm not sure how much practical value it > has. > > --- >8 --- > hex2oct() { > perl -ne 'printf "\\%03o", hex for /../g' > } > > test_expect_success 'clone on case-insensitive fs' ' > o=$(git hash-object -w --stdin </dev/null | hex2oct) && > t=$(printf "100644 X\0${o}100644 x\0${o}" | > git hash-object -w -t tree --stdin) && > c=$(git commit-tree -m bogus $t) && > git update-ref refs/heads/bogus $c && > git clone -b bogus . bogus > ' > --- >8 --- I'd argue that it's a worthwhile test to have, since it will fail on those systems where it's going to be a problem. Furthermore, people do run the tests (as does Travis) on case-insensitive file systems during the development cycle, so if we break something in the future, someone will notice while we're still in the development cycle. -- brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 867 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-19 22:31 ` brian m. carlson @ 2018-01-20 0:15 ` Eric Sunshine 2018-01-20 0:23 ` brian m. carlson 0 siblings, 1 reply; 33+ messages in thread From: Eric Sunshine @ 2018-01-20 0:15 UTC (permalink / raw) To: brian m. carlson, Eric Sunshine, Duy Nguyen, Александр Булаев, Git List On Fri, Jan 19, 2018 at 5:31 PM, brian m. carlson <sandals@crustytoothpaste.net> wrote: > On Fri, Jan 19, 2018 at 02:40:02AM -0500, Eric Sunshine wrote: >> Nice detective work. This particular manifestation is caught by the >> following test which fails without brian's patch on MacOS (and >> presumably Windows) and succeeds with it. On Linux and BSD, it will of >> course succeed always, so I'm not sure how much practical value it >> has. > > I'd argue that it's a worthwhile test to have, since it will fail on > those systems where it's going to be a problem. Furthermore, people do > run the tests (as does Travis) on case-insensitive file systems during > the development cycle, so if we break something in the future, someone > will notice while we're still in the development cycle. Oh, I agree. My original question of its practical value was based upon my belief that the full test suite is very rarely run on MacOS (partly because there are so few Git developers on MacOS and partly because it runs so slowly on the platform, though not nearly as slowly as on Windows). However, as soon as I hit "Send", it hit me that the problem would also manifest on Windows, and we know that Dscho runs the tests regularly on Windows, so I changed my mind. I did consider Travis but wasn't sure if it was testing on case-insensitive filesystems. Are you planning on submitting the test as a proper patch as follow-up to [1]? I couldn't quite decide in which test script it should reside. [1]: https://public-inbox.org/git/xmqqr2qlps7r.fsf@gitster.mtv.corp.google.com/ ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-20 0:15 ` Eric Sunshine @ 2018-01-20 0:23 ` brian m. carlson 2018-01-20 0:27 ` Eric Sunshine 0 siblings, 1 reply; 33+ messages in thread From: brian m. carlson @ 2018-01-20 0:23 UTC (permalink / raw) To: Eric Sunshine Cc: Duy Nguyen, Александр Булаев, Git List [-- Attachment #1: Type: text/plain, Size: 1144 bytes --] On Fri, Jan 19, 2018 at 07:15:45PM -0500, Eric Sunshine wrote: > Oh, I agree. My original question of its practical value was based > upon my belief that the full test suite is very rarely run on MacOS > (partly because there are so few Git developers on MacOS and partly > because it runs so slowly on the platform, though not nearly as slowly > as on Windows). However, as soon as I hit "Send", it hit me that the > problem would also manifest on Windows, and we know that Dscho runs > the tests regularly on Windows, so I changed my mind. I did consider > Travis but wasn't sure if it was testing on case-insensitive > filesystems. > > Are you planning on submitting the test as a proper patch as follow-up > to [1]? I couldn't quite decide in which test script it should reside. Sure, I can. Since you wrote it, may I have permission to add your sign-off? I'll try to set up a temporary FAT device and verify that I can reproduce the problem on Linux as well. -- brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 867 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: git 2.16.0 segfaults on clone of specific repo 2018-01-20 0:23 ` brian m. carlson @ 2018-01-20 0:27 ` Eric Sunshine 0 siblings, 0 replies; 33+ messages in thread From: Eric Sunshine @ 2018-01-20 0:27 UTC (permalink / raw) To: brian m. carlson, Eric Sunshine, Duy Nguyen, Александр Булаев, Git List On Fri, Jan 19, 2018 at 7:23 PM, brian m. carlson <sandals@crustytoothpaste.net> wrote: > On Fri, Jan 19, 2018 at 07:15:45PM -0500, Eric Sunshine wrote: >> Are you planning on submitting the test as a proper patch as follow-up >> to [1]? I couldn't quite decide in which test script it should reside. > > Sure, I can. Since you wrote it, may I have permission to add your > sign-off? You may... Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> > I'll try to set up a temporary FAT device and verify that I can > reproduce the problem on Linux as well. ^ permalink raw reply [flat|nested] 33+ messages in thread
* [PATCH] repository: pre-initialize hash algo pointer 2018-01-19 3:06 ` Eric Sunshine 2018-01-19 3:39 ` Randall S. Becker 2018-01-19 3:40 ` brian m. carlson @ 2018-01-19 4:18 ` brian m. carlson 2018-01-19 7:54 ` Eric Sunshine 2018-01-20 20:33 ` [PATCH] t: add clone test for files differing only in case brian m. carlson 2 siblings, 2 replies; 33+ messages in thread From: brian m. carlson @ 2018-01-19 4:18 UTC (permalink / raw) To: Junio C Hamano Cc: git, Eric Sunshine, Александр Булаев There are various git subcommands (among them, clone) which don't set up the repository (that is, they lack RUN_SETUP or RUN_SETUP_GENTLY) but end up needing to have information about the hash algorithm in use. Because the hash algorithm is part of struct repository and it's only initialized in repository setup, we can end up dereferencing a NULL pointer in some cases if we call one of these subcommands and look up the empty blob or empty tree values. In the future, we can add a command line option for this or read it from the configuration, but until we're ready to expose that functionality to the user, simply initialize the repository structure to use the current hash algorithm, SHA-1. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> --- repository.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) I'm still quite mystified as to why this is working on Linux and not macOS, but I can only guess that compilers are just very advanced and have somehow concluded that we would clearly never dereference a NULL pointer, so they picked the only non-NULL value. I haven't included a test because I have no way to reproduce the issue. This patch is the first from a series I'm working on where I do expand the use of the hash struct and therefore caused a segfault on clone, so I can imagine what's going on without having a way to prove it affects this particular case. If someone with access to macOS can provide a test, I'd be very grateful. My apologies for the error and inconvenience. diff --git a/repository.c b/repository.c index 998413b8bb..f66fcb1342 100644 --- a/repository.c +++ b/repository.c @@ -5,7 +5,7 @@ /* The main repository */ static struct repository the_repo = { - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0 + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0 }; struct repository *the_repository = &the_repo; ^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH] repository: pre-initialize hash algo pointer 2018-01-19 4:18 ` [PATCH] repository: pre-initialize hash algo pointer brian m. carlson @ 2018-01-19 7:54 ` Eric Sunshine 2018-01-19 19:24 ` Junio C Hamano 2018-01-20 20:33 ` [PATCH] t: add clone test for files differing only in case brian m. carlson 1 sibling, 1 reply; 33+ messages in thread From: Eric Sunshine @ 2018-01-19 7:54 UTC (permalink / raw) To: brian m. carlson Cc: Junio C Hamano, Git List, Александр Булаев On Thu, Jan 18, 2018 at 11:18 PM, brian m. carlson <sandals@crustytoothpaste.net> wrote: > There are various git subcommands (among them, clone) which don't set up > the repository (that is, they lack RUN_SETUP or RUN_SETUP_GENTLY) but > end up needing to have information about the hash algorithm in use. > Because the hash algorithm is part of struct repository and it's only > initialized in repository setup, we can end up dereferencing a NULL > pointer in some cases if we call one of these subcommands and look up > the empty blob or empty tree values. > > In the future, we can add a command line option for this or read it from > the configuration, but until we're ready to expose that functionality to > the user, simply initialize the repository structure to use the current > hash algorithm, SHA-1. > > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> > --- > I'm still quite mystified as to why this is working on Linux and not > macOS, but I can only guess that compilers are just very advanced and > have somehow concluded that we would clearly never dereference a NULL > pointer, so they picked the only non-NULL value. Now that we know (due to Duy's excellent detective work[1]) that the trigger is files with names differing only in case on case-insensitive filesystems, the commit message can be updated appropriately. > I haven't included a test because I have no way to reproduce the issue. > This patch is the first from a series I'm working on where I do expand > the use of the hash struct and therefore caused a segfault on clone, so > I can imagine what's going on without having a way to prove it affects > this particular case. > > If someone with access to macOS can provide a test, I'd be very > grateful. Done. Find the test here[2]. It fails without your fix on MacOS and (presumably) Windows, and succeeds with the fix. > My apologies for the error and inconvenience. [1]: https://public-inbox.org/git/CACsJy8BTFm_0sv=roL1OKKW=1DyU3vqD50NKyHg3KQ7G+mAepQ@mail.gmail.com/ [2]: https://public-inbox.org/git/20180119074001.GA55929@flurp.local/ ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] repository: pre-initialize hash algo pointer 2018-01-19 7:54 ` Eric Sunshine @ 2018-01-19 19:24 ` Junio C Hamano 2018-01-19 21:48 ` Eric Sunshine ` (2 more replies) 0 siblings, 3 replies; 33+ messages in thread From: Junio C Hamano @ 2018-01-19 19:24 UTC (permalink / raw) To: Eric Sunshine Cc: brian m. carlson, Nguyễn Thái Ngọc Duy, Git List, Александр Булаев Eric Sunshine <sunshine@sunshineco.com> writes: >> I'm still quite mystified as to why this is working on Linux and not >> macOS, but I can only guess that compilers are just very advanced and >> have somehow concluded that we would clearly never dereference a NULL >> pointer, so they picked the only non-NULL value. > > Now that we know (due to Duy's excellent detective work[1]) that the > trigger is files with names differing only in case on case-insensitive > filesystems, the commit message can be updated appropriately. Thanks. Let me apply the following and do a 2.16.1, hopefully by the end of day or mid tomorrow at the latest. Test to protect the fix can come as a separate follow-up patch. -- >8 -- Subject: [PATCH] repository: pre-initialize hash algo pointer There are various git subcommands (among them, clone) which don't set up the repository (that is, they lack RUN_SETUP or RUN_SETUP_GENTLY) but end up needing to have information about the hash algorithm in use. Because the hash algorithm is part of struct repository and it's only initialized in repository setup, we can end up dereferencing a NULL pointer in some cases if we call one of these subcommands and look up the empty blob or empty tree values. A "git clone" of a project that has two paths that differ only in case suffers from this if it is run on a case insensitive platform. When the command attempts to check out one of these two paths after checking out the other one, the checkout codepath needs to see if the version that is already on the filesystem (which should not happen if the FS were case sensitive), and it needs to exercise the hashing code. In the future, we can add a command line option for this or read it from the configuration, but until we're ready to expose that functionality to the user, simply initialize the repository structure to use the current hash algorithm, SHA-1. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com> --- repository.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/repository.c b/repository.c index 998413b8bb..f66fcb1342 100644 --- a/repository.c +++ b/repository.c @@ -5,7 +5,7 @@ /* The main repository */ static struct repository the_repo = { - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0 + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0 }; struct repository *the_repository = &the_repo; -- 2.16.0-204-gc262421c89 ^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH] repository: pre-initialize hash algo pointer 2018-01-19 19:24 ` Junio C Hamano @ 2018-01-19 21:48 ` Eric Sunshine 2018-01-19 22:25 ` Junio C Hamano 2018-01-19 23:14 ` brian m. carlson 2018-01-20 7:01 ` Junio C Hamano 2 siblings, 1 reply; 33+ messages in thread From: Eric Sunshine @ 2018-01-19 21:48 UTC (permalink / raw) To: Junio C Hamano Cc: brian m. carlson, Nguyễn Thái Ngọc Duy, Git List, Александр Булаев On Fri, Jan 19, 2018 at 2:24 PM, Junio C Hamano <gitster@pobox.com> wrote: > Eric Sunshine <sunshine@sunshineco.com> writes: >> Now that we know (due to Duy's excellent detective work[1]) that the >> trigger is files with names differing only in case on case-insensitive >> filesystems, the commit message can be updated appropriately. > > Thanks. Let me apply the following and do a 2.16.1, hopefully by > the end of day or mid tomorrow at the latest. Test to protect the > fix can come as a separate follow-up patch. > > -- >8 -- > Subject: [PATCH] repository: pre-initialize hash algo pointer > [...] > A "git clone" of a project that has two paths that differ only in > case suffers from this if it is run on a case insensitive platform. > When the command attempts to check out one of these two paths after > checking out the other one, the checkout codepath needs to see if > the version that is already on the filesystem (which should not > happen if the FS were case sensitive), and it needs to exercise the > hashing code. Thanks, the amended commit message makes the reason for the patch more concrete. There does seem to be a bit of a grammatical issue, however, which makes it difficult to parse. Namely, "already on the filesystem (...)" probably was meant to say "already on the filesystem (...) is {something}". ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] repository: pre-initialize hash algo pointer 2018-01-19 21:48 ` Eric Sunshine @ 2018-01-19 22:25 ` Junio C Hamano 0 siblings, 0 replies; 33+ messages in thread From: Junio C Hamano @ 2018-01-19 22:25 UTC (permalink / raw) To: Eric Sunshine Cc: brian m. carlson, Nguyễn Thái Ngọc Duy, Git List, Александр Булаев Eric Sunshine <sunshine@sunshineco.com> writes: > On Fri, Jan 19, 2018 at 2:24 PM, Junio C Hamano <gitster@pobox.com> wrote: >> Eric Sunshine <sunshine@sunshineco.com> writes: >>> Now that we know (due to Duy's excellent detective work[1]) that the >>> trigger is files with names differing only in case on case-insensitive >>> filesystems, the commit message can be updated appropriately. >> >> Thanks. Let me apply the following and do a 2.16.1, hopefully by >> the end of day or mid tomorrow at the latest. Test to protect the >> fix can come as a separate follow-up patch. >> >> -- >8 -- >> Subject: [PATCH] repository: pre-initialize hash algo pointer >> [...] >> A "git clone" of a project that has two paths that differ only in >> case suffers from this if it is run on a case insensitive platform. >> When the command attempts to check out one of these two paths after >> checking out the other one, the checkout codepath needs to see if >> the version that is already on the filesystem (which should not >> happen if the FS were case sensitive), and it needs to exercise the >> hashing code. > > Thanks, the amended commit message makes the reason for the patch more > concrete. There does seem to be a bit of a grammatical issue, however, > which makes it difficult to parse. Namely, "already on the filesystem > (...)" probably was meant to say "already on the filesystem (...) is > {something}". Indeed it is incomplete sentence. The codepath wants to check if the one on the filesystem is the same as the one that is being checked out and exercises the hashing code at that point. Thansk. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] repository: pre-initialize hash algo pointer 2018-01-19 19:24 ` Junio C Hamano 2018-01-19 21:48 ` Eric Sunshine @ 2018-01-19 23:14 ` brian m. carlson 2018-01-20 7:01 ` Junio C Hamano 2 siblings, 0 replies; 33+ messages in thread From: brian m. carlson @ 2018-01-19 23:14 UTC (permalink / raw) To: Junio C Hamano Cc: Eric Sunshine, Nguyễn Thái Ngọc Duy, Git List, Александр Булаев [-- Attachment #1: Type: text/plain, Size: 802 bytes --] On Fri, Jan 19, 2018 at 11:24:24AM -0800, Junio C Hamano wrote: > Eric Sunshine <sunshine@sunshineco.com> writes: > > Now that we know (due to Duy's excellent detective work[1]) that the > > trigger is files with names differing only in case on case-insensitive > > filesystems, the commit message can be updated appropriately. > > Thanks. Let me apply the following and do a 2.16.1, hopefully by > the end of day or mid tomorrow at the latest. Test to protect the > fix can come as a separate follow-up patch. This looks good with Eric's suggested changes. Again, my apologies for the breakage and the inconvenience involved. -- brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 867 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] repository: pre-initialize hash algo pointer 2018-01-19 19:24 ` Junio C Hamano 2018-01-19 21:48 ` Eric Sunshine 2018-01-19 23:14 ` brian m. carlson @ 2018-01-20 7:01 ` Junio C Hamano 2018-01-20 9:38 ` Eric Sunshine 2 siblings, 1 reply; 33+ messages in thread From: Junio C Hamano @ 2018-01-20 7:01 UTC (permalink / raw) To: Eric Sunshine Cc: brian m. carlson, Nguyễn Thái Ngọc Duy, Git List, Александр Булаев Junio C Hamano <gitster@pobox.com> writes: > Eric Sunshine <sunshine@sunshineco.com> writes: > >>> I'm still quite mystified as to why this is working on Linux and not >>> macOS, but I can only guess that compilers are just very advanced and >>> have somehow concluded that we would clearly never dereference a NULL >>> pointer, so they picked the only non-NULL value. >> >> Now that we know (due to Duy's excellent detective work[1]) that the >> trigger is files with names differing only in case on case-insensitive >> filesystems, the commit message can be updated appropriately. > > Thanks. Let me apply the following and do a 2.16.1, hopefully by > the end of day or mid tomorrow at the latest. Test to protect the > fix can come as a separate follow-up patch. I actually changed my mind and decided not to rush 2.16.1 with just this change. After all, even though it is better not to crash in such a problematic case, a "successful" clone of such a project on such a filesystem cannot be sanely used *anyway*, so in that sense the "fix" would stop "clone" from segfaulting but the resulting repository would still be "wrong" (e.g. "git status" immediately after clone would likely to say that the working tree is already dirty and missing one of the two paths, or something silly like that). ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] repository: pre-initialize hash algo pointer 2018-01-20 7:01 ` Junio C Hamano @ 2018-01-20 9:38 ` Eric Sunshine 0 siblings, 0 replies; 33+ messages in thread From: Eric Sunshine @ 2018-01-20 9:38 UTC (permalink / raw) To: Junio C Hamano Cc: brian m. carlson, Nguyễn Thái Ngọc Duy, Git List, Александр Булаев On Sat, Jan 20, 2018 at 2:01 AM, Junio C Hamano <gitster@pobox.com> wrote: > Junio C Hamano <gitster@pobox.com> writes: >> Eric Sunshine <sunshine@sunshineco.com> writes: >>> Now that we know (due to Duy's excellent detective work[1]) that the >>> trigger is files with names differing only in case on case-insensitive >>> filesystems, the commit message can be updated appropriately. >> >> Thanks. Let me apply the following and do a 2.16.1, hopefully by >> the end of day or mid tomorrow at the latest. Test to protect the >> fix can come as a separate follow-up patch. > > I actually changed my mind and decided not to rush 2.16.1 with just > this change. After all, even though it is better not to crash in > such a problematic case, a "successful" clone of such a project on > such a filesystem cannot be sanely used *anyway*, so in that sense > the "fix" would stop "clone" from segfaulting but the resulting > repository would still be "wrong" (e.g. "git status" immediately > after clone would likely to say that the working tree is already > dirty and missing one of the two paths, or something silly like > that). The counter-argument is that filenames differing only in case do not necessarily render a project unusable on case-insensitive filesystems. For instance, if the problematic files exist only in a project's test suite or in some platform-specific resource directory, the project itself may still be perfectly usable for a person cloning a project merely to build and use it (not develop it). However, "clone" crashing _does_ render the project unusable for the same person. The crash[1] when cloning 'tcell' is an example of a project which is still 100% usable on case-insensitive filesystems despite case-conflicting filenames. Each of the conflicting file pairs [2] has identical content, so everything works as intended. Case-conflicting filenames also do not necessarily make it impossible to do development work on a project. I have, myself, on several occasions cloned such projects on MacOS to make improvements and track down and/or fix bugs in parts of the projects not impacted by the case-conflicting filenames. Footnotes: [1]: https://public-inbox.org/git/20180119180855.GA98561@smm.local/ [2]: case-conflicting files in tcell's bundled terminfo database: terminfo/database/32/2621A terminfo/database/32/2621a terminfo/database/68/hp2621A terminfo/database/68/hp2621a terminfo/database/68/hp70092A terminfo/database/68/hp70092a ^ permalink raw reply [flat|nested] 33+ messages in thread
* [PATCH] t: add clone test for files differing only in case 2018-01-19 4:18 ` [PATCH] repository: pre-initialize hash algo pointer brian m. carlson 2018-01-19 7:54 ` Eric Sunshine @ 2018-01-20 20:33 ` brian m. carlson 2018-01-21 1:19 ` Eric Sunshine ` (2 more replies) 1 sibling, 3 replies; 33+ messages in thread From: brian m. carlson @ 2018-01-20 20:33 UTC (permalink / raw) To: Junio C Hamano Cc: git, Eric Sunshine, Александр Булаев We recently introduced a regression in cloning repositories onto case-insensitive file systems where the repository contains multiple files differing only in case. In such a case, we would segfault. This segfault has already been fixed (repository: pre-initialize hash algo pointer), but it's not the first time similar problems have arisen. Introduce a test to catch this case and ensure the behavior does not regress. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> --- t/t5601-clone.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+) I've verified that the test does fail without the patch on a vfat file system. However, many other tests also fail on a vfat file system on Linux, so unfortunately that doesn't look like a viable testing strategy going forward. I didn't include an object ID for the commit referenced simply because I didn't see one yet and I didn't want to insert a local one that wouldn't work for anyone else. diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 0f895478f0..53b2dda9d2 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -611,4 +611,17 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' ' git -C replay.git index-pack -v --stdin <tmp.pack ' +hex2oct() { + perl -ne 'printf "\\%03o", hex for /../g' +} + +test_expect_success 'clone on case-insensitive fs' ' + o=$(git hash-object -w --stdin </dev/null | hex2oct) && + t=$(printf "100644 X\0${o}100644 x\0${o}" | + git hash-object -w -t tree --stdin) && + c=$(git commit-tree -m bogus $t) && + git update-ref refs/heads/bogus $c && + git clone -b bogus . bogus +' + test_done ^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH] t: add clone test for files differing only in case 2018-01-20 20:33 ` [PATCH] t: add clone test for files differing only in case brian m. carlson @ 2018-01-21 1:19 ` Eric Sunshine 2018-01-21 7:33 ` Junio C Hamano 2018-01-21 11:50 ` Duy Nguyen 2 siblings, 0 replies; 33+ messages in thread From: Eric Sunshine @ 2018-01-21 1:19 UTC (permalink / raw) To: brian m. carlson Cc: Junio C Hamano, Git List, Александр Булаев On Sat, Jan 20, 2018 at 3:33 PM, brian m. carlson <sandals@crustytoothpaste.net> wrote: > We recently introduced a regression in cloning repositories onto > case-insensitive file systems where the repository contains multiple > files differing only in case. In such a case, we would segfault. This > segfault has already been fixed (repository: pre-initialize hash algo > pointer), but it's not the first time similar problems have arisen. > Introduce a test to catch this case and ensure the behavior does not > regress. I guess you'd probably want a "From: Eric Sunshine <sunshine@sunshineco.com>" header before this paragraph. The patch itself looks correct... ;-) > Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> > --- > diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh > index 0f895478f0..53b2dda9d2 100755 > --- a/t/t5601-clone.sh > +++ b/t/t5601-clone.sh > @@ -611,4 +611,17 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' ' > git -C replay.git index-pack -v --stdin <tmp.pack > ' > > +hex2oct() { > + perl -ne 'printf "\\%03o", hex for /../g' > +} > + > +test_expect_success 'clone on case-insensitive fs' ' > + o=$(git hash-object -w --stdin </dev/null | hex2oct) && > + t=$(printf "100644 X\0${o}100644 x\0${o}" | > + git hash-object -w -t tree --stdin) && > + c=$(git commit-tree -m bogus $t) && > + git update-ref refs/heads/bogus $c && > + git clone -b bogus . bogus > +' > + > test_done ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] t: add clone test for files differing only in case 2018-01-20 20:33 ` [PATCH] t: add clone test for files differing only in case brian m. carlson 2018-01-21 1:19 ` Eric Sunshine @ 2018-01-21 7:33 ` Junio C Hamano 2018-01-21 7:46 ` Eric Sunshine 2018-01-21 11:50 ` Duy Nguyen 2 siblings, 1 reply; 33+ messages in thread From: Junio C Hamano @ 2018-01-21 7:33 UTC (permalink / raw) To: brian m. carlson Cc: git, Eric Sunshine, Александр Булаев "brian m. carlson" <sandals@crustytoothpaste.net> writes: > diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh > index 0f895478f0..53b2dda9d2 100755 > --- a/t/t5601-clone.sh > +++ b/t/t5601-clone.sh > @@ -611,4 +611,17 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' ' > git -C replay.git index-pack -v --stdin <tmp.pack > ' > > +hex2oct() { > + perl -ne 'printf "\\%03o", hex for /../g' > +} > + > +test_expect_success 'clone on case-insensitive fs' ' > + o=$(git hash-object -w --stdin </dev/null | hex2oct) && > + t=$(printf "100644 X\0${o}100644 x\0${o}" | > + git hash-object -w -t tree --stdin) && > + c=$(git commit-tree -m bogus $t) && > + git update-ref refs/heads/bogus $c && > + git clone -b bogus . bogus > +' > + > test_done Hmm, I seem to be seeing a failure from this thing: expecting success: o=$(git hash-object -w --stdin </dev/null | hex2oct) && t=$(printf "100644 X\0${o}100644 x\0${o}" | git hash-object -w -t tree --stdin) && c=$(git commit-tree -m bogus $t) && git update-ref refs/heads/bogus $c && git clone -b bogus . bogus fatal: repository '.' does not exist even on a case sensitive platform. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] t: add clone test for files differing only in case 2018-01-21 7:33 ` Junio C Hamano @ 2018-01-21 7:46 ` Eric Sunshine 2018-01-21 8:07 ` Eric Sunshine 0 siblings, 1 reply; 33+ messages in thread From: Eric Sunshine @ 2018-01-21 7:46 UTC (permalink / raw) To: Junio C Hamano Cc: brian m. carlson, Git List, Александр Булаев On Sun, Jan 21, 2018 at 2:33 AM, Junio C Hamano <gitster@pobox.com> wrote: > "brian m. carlson" <sandals@crustytoothpaste.net> writes: >> +test_expect_success 'clone on case-insensitive fs' ' >> + o=$(git hash-object -w --stdin </dev/null | hex2oct) && >> + t=$(printf "100644 X\0${o}100644 x\0${o}" | >> + git hash-object -w -t tree --stdin) && >> + c=$(git commit-tree -m bogus $t) && >> + git update-ref refs/heads/bogus $c && >> + git clone -b bogus . bogus >> +' >> + >> test_done > > Hmm, I seem to be seeing a failure from this thing: > > expecting success: > o=$(git hash-object -w --stdin </dev/null | hex2oct) && > t=$(printf "100644 X\0${o}100644 x\0${o}" | > git hash-object -w -t tree --stdin) && > c=$(git commit-tree -m bogus $t) && > git update-ref refs/heads/bogus $c && > git clone -b bogus . bogus > > fatal: repository '.' does not exist > > even on a case sensitive platform. Yep. In pretty much any other test script, this would work (it was developed in a stand-alone script), but t5601 (which nukes .git as its first action) isn't the most friendly place. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] t: add clone test for files differing only in case 2018-01-21 7:46 ` Eric Sunshine @ 2018-01-21 8:07 ` Eric Sunshine 2018-01-21 19:55 ` brian m. carlson 0 siblings, 1 reply; 33+ messages in thread From: Eric Sunshine @ 2018-01-21 8:07 UTC (permalink / raw) To: Junio C Hamano Cc: brian m. carlson, Git List, Александр Булаев On Sun, Jan 21, 2018 at 02:46:15AM -0500, Eric Sunshine wrote: > On Sun, Jan 21, 2018 at 2:33 AM, Junio C Hamano <gitster@pobox.com> wrote: > > "brian m. carlson" <sandals@crustytoothpaste.net> writes: > >> +test_expect_success 'clone on case-insensitive fs' ' > >> + o=$(git hash-object -w --stdin </dev/null | hex2oct) && > >> + t=$(printf "100644 X\0${o}100644 x\0${o}" | > >> + git hash-object -w -t tree --stdin) && > >> + c=$(git commit-tree -m bogus $t) && > >> + git update-ref refs/heads/bogus $c && > >> + git clone -b bogus . bogus > >> +' > > > > Hmm, I seem to be seeing a failure from this thing: > > fatal: repository '.' does not exist > > even on a case sensitive platform. > > Yep. In pretty much any other test script, this would work (it was > developed in a stand-alone script), but t5601 (which nukes .git as its > first action) isn't the most friendly place. Here's a re-roll which fixes that problem (and has a slightly re-written commit message. --- >8 --- From: Eric Sunshine <sunshine@sunshineco.com> Subject: [PATCH] t5601-clone: test case-conflicting files on case-insensitive filesystem A recently introduced regression caused a segfault at clone time on case-insensitive filesystems when filenames differing only in case are present. This bug has already been fixed (repository: pre-initialize hash algo pointer, 2018-01-18), but it's not the first time similar problems have arisen. Therefore, introduce a test to catch this case and protect against future regressions. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> --- t/t5601-clone.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 0f895478f0..2d1490f631 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -611,4 +611,21 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' ' git -C replay.git index-pack -v --stdin <tmp.pack ' +hex2oct() { + perl -ne 'printf "\\%03o", hex for /../g' +} + +test_expect_success 'clone on case-insensitive fs' ' + git init icasefs && + ( + cd icasefs + o=$(git hash-object -w --stdin </dev/null | hex2oct) && + t=$(printf "100644 X\0${o}100644 x\0${o}" | + git hash-object -w -t tree --stdin) && + c=$(git commit-tree -m bogus $t) && + git update-ref refs/heads/bogus $c && + git clone -b bogus . bogus + ) +' + test_done -- 2.16.0.312.g896df04e46 ^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH] t: add clone test for files differing only in case 2018-01-21 8:07 ` Eric Sunshine @ 2018-01-21 19:55 ` brian m. carlson 0 siblings, 0 replies; 33+ messages in thread From: brian m. carlson @ 2018-01-21 19:55 UTC (permalink / raw) To: Eric Sunshine Cc: Junio C Hamano, Git List, Александр Булаев [-- Attachment #1: Type: text/plain, Size: 877 bytes --] On Sun, Jan 21, 2018 at 03:07:28AM -0500, Eric Sunshine wrote: > On Sun, Jan 21, 2018 at 02:46:15AM -0500, Eric Sunshine wrote: > > Yep. In pretty much any other test script, this would work (it was > > developed in a stand-alone script), but t5601 (which nukes .git as its > > first action) isn't the most friendly place. > > Here's a re-roll which fixes that problem (and has a slightly > re-written commit message. Yes, this looks good. I did indeed intend to forge your From line on the message and actually test that it was fixed, but only tested it on the vfat partition, not my ext4 partition, where, of course, I got distracted by the various other failures. Sigh. Thanks for fixing this up. -- brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 867 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] t: add clone test for files differing only in case 2018-01-20 20:33 ` [PATCH] t: add clone test for files differing only in case brian m. carlson 2018-01-21 1:19 ` Eric Sunshine 2018-01-21 7:33 ` Junio C Hamano @ 2018-01-21 11:50 ` Duy Nguyen 2018-01-21 18:47 ` Eric Sunshine 2 siblings, 1 reply; 33+ messages in thread From: Duy Nguyen @ 2018-01-21 11:50 UTC (permalink / raw) To: brian m. carlson Cc: Junio C Hamano, Git Mailing List, Eric Sunshine, Александр Булаев On Sun, Jan 21, 2018 at 3:33 AM, brian m. carlson <sandals@crustytoothpaste.net> wrote: > We recently introduced a regression in cloning repositories onto > case-insensitive file systems where the repository contains multiple > files differing only in case. In such a case, we would segfault. This > segfault has already been fixed (repository: pre-initialize hash algo > pointer), but it's not the first time similar problems have arisen. > Introduce a test to catch this case and ensure the behavior does not > regress. > > Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> > --- > t/t5601-clone.sh | 13 +++++++++++++ > 1 file changed, 13 insertions(+) > > I've verified that the test does fail without the patch on a vfat file > system. However, many other tests also fail on a vfat file system on > Linux, so unfortunately that doesn't look like a viable testing strategy > going forward. vfat is not very UNIXy. I suspect we need a bunch of tweaks that cygwin/mingw uses to make the test suite pass. There are other case-insensitive filesystems on linux though. JFS (with "mkfs.jfs -O" to make it case-insensitive) seems to pass the test suite and crash git without your fix so it's a good candidate if someone wants to test this behavior on linux in future. HFS+ fails some unicode test in t0050 and I didn't check further. > +test_expect_success 'clone on case-insensitive fs' ' We have CASE_INSENSITIVE_FS prereq. Should we use it here? I know it does not harm running this test on case-sensitive filesystem, but the prereq could be useful for grepping. > + o=$(git hash-object -w --stdin </dev/null | hex2oct) && > + t=$(printf "100644 X\0${o}100644 x\0${o}" | > + git hash-object -w -t tree --stdin) && > + c=$(git commit-tree -m bogus $t) && > + git update-ref refs/heads/bogus $c && > + git clone -b bogus . bogus > +' > + > test_done -- Duy ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH] t: add clone test for files differing only in case 2018-01-21 11:50 ` Duy Nguyen @ 2018-01-21 18:47 ` Eric Sunshine 0 siblings, 0 replies; 33+ messages in thread From: Eric Sunshine @ 2018-01-21 18:47 UTC (permalink / raw) To: Duy Nguyen Cc: brian m. carlson, Junio C Hamano, Git Mailing List, Александр Булаев On Sun, Jan 21, 2018 at 6:50 AM, Duy Nguyen <pclouds@gmail.com> wrote: > On Sun, Jan 21, 2018 at 3:33 AM, brian m. carlson >> +test_expect_success 'clone on case-insensitive fs' ' > > We have CASE_INSENSITIVE_FS prereq. Should we use it here? I know it > does not harm running this test on case-sensitive filesystem, but the > prereq could be useful for grepping. I'd rather not hide it behind the CASE_INSENSITIVE_FS[1] prerequisite since the test potentially could catch some sort of future regression even on case-sensitive filesystems. [1]: Todd Zullinger suggested the same: https://public-inbox.org/git/CAPig+cSRN1zHc=zsO1Y_aQ_eO+sbsd0cq5iZ9hYz3ruK_E-0Dw@mail.gmail.com/ ^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2018-01-21 19:55 UTC | newest] Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-01-18 20:55 git 2.16.0 segfaults on clone of specific repo Александр Булаев 2018-01-18 21:03 ` Randall S. Becker 2018-01-19 0:15 ` Eric Sunshine 2018-01-19 2:47 ` brian m. carlson 2018-01-19 3:06 ` Eric Sunshine 2018-01-19 3:39 ` Randall S. Becker 2018-01-19 3:40 ` brian m. carlson 2018-01-19 5:31 ` Duy Nguyen 2018-01-19 7:40 ` Eric Sunshine 2018-01-19 8:22 ` Duy Nguyen 2018-01-19 8:28 ` Eric Sunshine 2018-01-19 17:25 ` Todd Zullinger 2018-01-20 0:23 ` Eric Sunshine 2018-01-19 22:31 ` brian m. carlson 2018-01-20 0:15 ` Eric Sunshine 2018-01-20 0:23 ` brian m. carlson 2018-01-20 0:27 ` Eric Sunshine 2018-01-19 4:18 ` [PATCH] repository: pre-initialize hash algo pointer brian m. carlson 2018-01-19 7:54 ` Eric Sunshine 2018-01-19 19:24 ` Junio C Hamano 2018-01-19 21:48 ` Eric Sunshine 2018-01-19 22:25 ` Junio C Hamano 2018-01-19 23:14 ` brian m. carlson 2018-01-20 7:01 ` Junio C Hamano 2018-01-20 9:38 ` Eric Sunshine 2018-01-20 20:33 ` [PATCH] t: add clone test for files differing only in case brian m. carlson 2018-01-21 1:19 ` Eric Sunshine 2018-01-21 7:33 ` Junio C Hamano 2018-01-21 7:46 ` Eric Sunshine 2018-01-21 8:07 ` Eric Sunshine 2018-01-21 19:55 ` brian m. carlson 2018-01-21 11:50 ` Duy Nguyen 2018-01-21 18:47 ` Eric Sunshine
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).