'git stash list' => Segmentation fault

'git stash list' => Segmentation fault

[Brendan Boerner]

Ubuntu 14.04

I was using some prior v2.x using linuxbrew, everything was fine, took
an update, this started happening.

I verified also happens using the 2.21.0 from the Ubuntu 14.04 PPA
(this reproduction was using that build, not linuxbrew build).

bboerner@myhost:~/funproject/Src/bash$ git --version
git version 2.21.0

bboerner@myhost:~/funproject/Src/bash$ git stash list
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
stash@{0}: On devel: NYFL optimization
stash@{1}: On GL285: gl285

Regards,
Brendan

Re: 'git stash list' => Segmentation fault

[Johannes Schindelin]

Hi Brendan,

On Fri, 24 May 2019, Brendan Boerner wrote:

> Ubuntu 14.04
>
> I was using some prior v2.x using linuxbrew, everything was fine, took
> an update, this started happening.
>
> I verified also happens using the 2.21.0 from the Ubuntu 14.04 PPA
> (this reproduction was using that build, not linuxbrew build).
>
> bboerner@myhost:~/funproject/Src/bash$ git --version
> git version 2.21.0
>
> bboerner@myhost:~/funproject/Src/bash$ git stash list
> Segmentation fault (core dumped)
> Segmentation fault (core dumped)
> Segmentation fault (core dumped)
> stash@{0}: On devel: NYFL optimization
> stash@{1}: On GL285: gl285

Unfortunately, this leaves *very* little in the way of actionable
evidence. Would it be possible for you to share a tar'red up copy of the
.git/ directory? Or do you find that this happens for all of your stashes?
What is the output when you run this with `GIT_TRACE=1`?

Ciao,
Johannes

Re: 'git stash list' => Segmentation fault

[Jeff King]

On Mon, May 27, 2019 at 08:47:56PM +0200, Johannes Schindelin wrote:

> > bboerner@myhost:~/funproject/Src/bash$ git stash list
> > Segmentation fault (core dumped)
> > Segmentation fault (core dumped)
> > Segmentation fault (core dumped)
> > stash@{0}: On devel: NYFL optimization
> > stash@{1}: On GL285: gl285
> 
> Unfortunately, this leaves *very* little in the way of actionable
> evidence. Would it be possible for you to share a tar'red up copy of the
> .git/ directory? Or do you find that this happens for all of your stashes?
> What is the output when you run this with `GIT_TRACE=1`?

Alternatively, if you cannot share the repo, a backtrace from gdb might
help. There should be a core file either in the working directory, or in
the ".git/" directory. Try "gdb git /path/to/core" and then "bt" to get
the backtrace.

-Peff

Re: 'git stash list' => Segmentation fault

[Brendan Boerner]

Yes, providing the repo isn't an option.

Thanks for the suggestions, here's what I can provide. Unfortunately
not a lot in the stack trace.

Do you need the whole repo or would bits of it suffice e.g. index?

bboerner@myhost:~/funproject/Src$ GIT_TRACE=1 git stash list
14:52:22.633388 git.c:670               trace: exec: git-stash list
14:52:22.633448 run-command.c:643       trace: run_command: git-stash list
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
14:52:24.904494 git.c:419               trace: built-in: git rev-parse --git-dir
14:52:24.944879 git.c:419               trace: built-in: git rev-parse
--git-path objects
14:52:24.949126 git.c:419               trace: built-in: git rev-parse
--show-prefix
14:52:24.951414 git.c:419               trace: built-in: git rev-parse
--show-toplevel
14:52:24.953447 git.c:419               trace: built-in: git rev-parse
--git-path index
14:52:24.955670 git.c:419               trace: built-in: git config
--get-colorbool color.interactive
14:52:24.957435 git.c:419               trace: built-in: git config
--get-color color.interactive.help 'red bold'
14:52:24.958753 git.c:419               trace: built-in: git config
--get-color  reset
14:52:24.960461 git.c:419               trace: built-in: git rev-parse
--verify --quiet refs/stash
14:52:25.040053 git.c:419               trace: built-in: git log
'--format=%gd: %gs' -g --first-parent -m refs/stash --
14:52:26.436274 run-command.c:643       trace: run_command: unset
GIT_PAGER_IN_USE; LESS=FRX LV=-c less
stash@{0}: On devel: NYFL optimization
stash@{1}: On GL285: gl285


(gdb) bt
#0  0x00007fd321805fdf in ?? ()
#1  0x0000000000000000 in ?? ()

Regards,
Brendan

On Tue, May 28, 2019 at 1:34 AM Jeff King <peff@peff.net> wrote:
>
> On Mon, May 27, 2019 at 08:47:56PM +0200, Johannes Schindelin wrote:
>
> > > bboerner@myhost:~/funproject/Src/bash$ git stash list
> > > Segmentation fault (core dumped)
> > > Segmentation fault (core dumped)
> > > Segmentation fault (core dumped)
> > > stash@{0}: On devel: NYFL optimization
> > > stash@{1}: On GL285: gl285
> >
> > Unfortunately, this leaves *very* little in the way of actionable
> > evidence. Would it be possible for you to share a tar'red up copy of the
> > .git/ directory? Or do you find that this happens for all of your stashes?
> > What is the output when you run this with `GIT_TRACE=1`?
>
> Alternatively, if you cannot share the repo, a backtrace from gdb might
> help. There should be a core file either in the working directory, or in
> the ".git/" directory. Try "gdb git /path/to/core" and then "bt" to get
> the backtrace.
>
> -Peff

Re: 'git stash list' => Segmentation fault

[Jeff King]

On Tue, May 28, 2019 at 02:56:13PM -0500, Brendan Boerner wrote:

> Unfortunately not a lot in the stack trace.
> [...]
> (gdb) bt
> #0  0x00007fd321805fdf in ?? ()
> #1  0x0000000000000000 in ?? ()

I'll say. :)

I'd suspect either:

  1. Your git isn't built with debugging symbols. Is it possible to
     build from source? The default CFLAGS if you just run "make" should
     be enough, then you can run it straight out of the build directory
     as "$YOUR_BUILD_DIR/bin-wrappers/git".

  2. It's not the main git binary that's segfaulting. Most commands are
     built-ins of that binary these days, but a few are not. We should
     be able to figure out which command is having a problem from the
     trace output, but...

> 14:52:22.633448 run-command.c:643       trace: run_command: git-stash list
> Segmentation fault (core dumped)
> Segmentation fault (core dumped)
> Segmentation fault (core dumped)

I'd have expected to see more sub-commands here. And it's weird that
there are _three_ segfaults. Which version of Git is this? In the
upcoming release stash is written in C, but from the trace output it
looks like this is the old shell-script version (which is what I was
assuming, but I just want to double check).

Is it possible that your shell is segfaulting? Does "gdb /bin/sh core"
reveal anything more useful (a long shot, I know; it probably doesn't
have symbols either)?

> Do you need the whole repo or would bits of it suffice e.g. index?

Probably the whole thing. "stash list" wouldn't generally look at
the index at all. It should just be doing a "git log" on the reflog
entries found in .git/logs/refs/stash. But...

> 14:52:25.040053 git.c:419               trace: built-in: git log
> '--format=%gd: %gs' -g --first-parent -m refs/stash --
> 14:52:26.436274 run-command.c:643       trace: run_command: unset
> GIT_PAGER_IN_USE; LESS=FRX LV=-c less
> stash@{0}: On devel: NYFL optimization
> stash@{1}: On GL285: gl285

We can see that the "log" command actually works! The segfaults seem not
to derail the command from actually performing its task. So what
processes are actually segfaulting?

Maybe "strace -o trace.out -f git stash list" would be instructive. You
should be able to find the segfaulting process in that trace, and then
can grep for that PID to see what it was doing (especially what command
was exec'd, and possibly what it was doing right before the segfault).

I can also help picking through the strace output if you're comfortable
sending it off-list (it's likely to be much too large to send on-list
anyway).

-Peff

Re: 'git stash list' => Segmentation fault

[Brendan Boerner]

Hey Jeff,

Lot of good tips for debugging for me to work on - I'll work on that.

Both to avoid spamming the list and if sending large files e.g. strace
is it ok if I continue correspondence with you directly? (And
Johannes?)

Thanks!
Brendan

On Tue, May 28, 2019 at 3:40 PM Jeff King <peff@peff.net> wrote:
>
> On Tue, May 28, 2019 at 02:56:13PM -0500, Brendan Boerner wrote:
>
> > Unfortunately not a lot in the stack trace.
> > [...]
> > (gdb) bt
> > #0  0x00007fd321805fdf in ?? ()
> > #1  0x0000000000000000 in ?? ()
>
> I'll say. :)
>
> I'd suspect either:
>
>   1. Your git isn't built with debugging symbols. Is it possible to
>      build from source? The default CFLAGS if you just run "make" should
>      be enough, then you can run it straight out of the build directory
>      as "$YOUR_BUILD_DIR/bin-wrappers/git".
>
>   2. It's not the main git binary that's segfaulting. Most commands are
>      built-ins of that binary these days, but a few are not. We should
>      be able to figure out which command is having a problem from the
>      trace output, but...
>
> > 14:52:22.633448 run-command.c:643       trace: run_command: git-stash list
> > Segmentation fault (core dumped)
> > Segmentation fault (core dumped)
> > Segmentation fault (core dumped)
>
> I'd have expected to see more sub-commands here. And it's weird that
> there are _three_ segfaults. Which version of Git is this? In the
> upcoming release stash is written in C, but from the trace output it
> looks like this is the old shell-script version (which is what I was
> assuming, but I just want to double check).
>
> Is it possible that your shell is segfaulting? Does "gdb /bin/sh core"
> reveal anything more useful (a long shot, I know; it probably doesn't
> have symbols either)?
>
> > Do you need the whole repo or would bits of it suffice e.g. index?
>
> Probably the whole thing. "stash list" wouldn't generally look at
> the index at all. It should just be doing a "git log" on the reflog
> entries found in .git/logs/refs/stash. But...
>
> > 14:52:25.040053 git.c:419               trace: built-in: git log
> > '--format=%gd: %gs' -g --first-parent -m refs/stash --
> > 14:52:26.436274 run-command.c:643       trace: run_command: unset
> > GIT_PAGER_IN_USE; LESS=FRX LV=-c less
> > stash@{0}: On devel: NYFL optimization
> > stash@{1}: On GL285: gl285
>
> We can see that the "log" command actually works! The segfaults seem not
> to derail the command from actually performing its task. So what
> processes are actually segfaulting?
>
> Maybe "strace -o trace.out -f git stash list" would be instructive. You
> should be able to find the segfaulting process in that trace, and then
> can grep for that PID to see what it was doing (especially what command
> was exec'd, and possibly what it was doing right before the segfault).
>
> I can also help picking through the strace output if you're comfortable
> sending it off-list (it's likely to be much too large to send on-list
> anyway).
>
> -Peff

Re: 'git stash list' => Segmentation fault

[Jeff King]

On Tue, May 28, 2019 at 03:53:51PM -0500, Brendan Boerner wrote:

> Lot of good tips for debugging for me to work on - I'll work on that.
> 
> Both to avoid spamming the list and if sending large files e.g. strace
> is it ok if I continue correspondence with you directly? (And
> Johannes?)

In general, please keep substantive responses on the list, so everybody
can follow along. Adding to the list volume is fine. :)

But certainly if there are sensitive bits, feel free to send them
directly off-list.

-Peff

Re: 'git stash list' => Segmentation fault

[Brendan Boerner]

Thanks Jeff.

To answer your previous question I'm using git v2.21.0. I'm using the
Ubuntu 14.04 PPA as well as Linuxbew (also v2.21.0).

Your help helped me narrow this down to linuxbrew. Take it out of my
path no segfault.

Uninstall linuxbrew git, linuxbrew still in PATH, use PPA git, segfault.

Narrowed down further to be the "gettext" package - remove it from
linuxbrew and PPA git stash list => no segfault.

I'll ping the Linuxbrew folks.

Thanks for you help!
Brendan

On Tue, May 28, 2019 at 3:56 PM Jeff King <peff@peff.net> wrote:
>
> On Tue, May 28, 2019 at 03:53:51PM -0500, Brendan Boerner wrote:
>
> > Lot of good tips for debugging for me to work on - I'll work on that.
> >
> > Both to avoid spamming the list and if sending large files e.g. strace
> > is it ok if I continue correspondence with you directly? (And
> > Johannes?)
>
> In general, please keep substantive responses on the list, so everybody
> can follow along. Adding to the list volume is fine. :)
>
> But certainly if there are sensitive bits, feel free to send them
> directly off-list.
>
> -Peff

Re: 'git stash list' => Segmentation fault

[Jeff King]

On Tue, May 28, 2019 at 04:29:13PM -0500, Brendan Boerner wrote:

> To answer your previous question I'm using git v2.21.0. I'm using the
> Ubuntu 14.04 PPA as well as Linuxbew (also v2.21.0).
> 
> Your help helped me narrow this down to linuxbrew. Take it out of my
> path no segfault.
> 
> Uninstall linuxbrew git, linuxbrew still in PATH, use PPA git, segfault.
> 
> Narrowed down further to be the "gettext" package - remove it from
> linuxbrew and PPA git stash list => no segfault.
> 
> I'll ping the Linuxbrew folks.

Ah, that makes sense. You're indeed running the shell version, and it's
invoking "gettext" for messages, which is probably what's segfaulting.
And that explains the multiple segfaults and the fact that they didn't
actually cause the overall operation to fail.

So probably nothing here for Git to fix. Thanks for following up with
the resolution!

-Peff

Re: 'git stash list' => Segmentation fault

[Brendan Boerner]

Hey Jeff,

Yes, if I install gettext build from source envsubst and gettext no
segfaults. tset from ncurses segfaults as well unless I install from
source.

So weird breakage which manifested as a red herring. :-(

Thanks for your help helping me sort it out.

Regards,
Brendan



On Tue, May 28, 2019 at 4:39 PM Jeff King <peff@peff.net> wrote:
>
> On Tue, May 28, 2019 at 04:29:13PM -0500, Brendan Boerner wrote:
>
> > To answer your previous question I'm using git v2.21.0. I'm using the
> > Ubuntu 14.04 PPA as well as Linuxbew (also v2.21.0).
> >
> > Your help helped me narrow this down to linuxbrew. Take it out of my
> > path no segfault.
> >
> > Uninstall linuxbrew git, linuxbrew still in PATH, use PPA git, segfault.
> >
> > Narrowed down further to be the "gettext" package - remove it from
> > linuxbrew and PPA git stash list => no segfault.
> >
> > I'll ping the Linuxbrew folks.
>
> Ah, that makes sense. You're indeed running the shell version, and it's
> invoking "gettext" for messages, which is probably what's segfaulting.
> And that explains the multiple segfaults and the fact that they didn't
> actually cause the overall operation to fail.
>
> So probably nothing here for Git to fix. Thanks for following up with
> the resolution!
>
> -Peff