From: Phil Hord <phil.hord@gmail.com>
To: Git <git@vger.kernel.org>
Cc: matheus.bernardino@usp.br, Stefan Beller <stefanbeller@gmail.com>,
Jonathan Tan <jonathantanmy@google.com>,
Derrick Stolee <dstolee@microsoft.com>
Subject: RFC - concurrency causes segfault in git grep since 2.26.0
Date: Thu, 24 Sep 2020 19:36:06 -0700 [thread overview]
Message-ID: <CABURp0r_VPNeU5ugbJDQo=hV_aOit7W-pf9nhDF9Og=vVJG8tw@mail.gmail.com> (raw)
Hi list. It's been a while.
I can reliably reproduce a segfault in git grep when searching the
object DB. It is caused by a race when threads > 2.
I have a patch that fixes the problem, but I don't know exactly why.
Can someone else explain it and/or offer a better solution?
====
diff --git packfile.c packfile.c
index e69012e7f2..52b7b54aeb 100644
--- packfile.c
+++ packfile.c
@@ -1812,7 +1812,9 @@ void *unpack_entry(struct repository *r, struct
packed_git *p, off_t obj_offset,
if (!base)
continue;
+ obj_read_lock();
delta_data = unpack_compressed_entry(p, &w_curs,
curpos, delta_size);
+ obj_read_unlock();
if (!delta_data) {
error("failed to unpack compressed delta "
====
My notes thus far:
I have seen it fail with `--threads 3`, but it fails very reliably for
me with `--threads 20`.
My repo is large (7GB) which gives grep enough to do. I'm not able to
make it fail on git.git; it seems like it's too small to run into this
problem.
The problem bisects to 1d1729caeb. It was somewhat hard to find this,
though, because git grep disables --threads when reading from the
object store until another patch later in the same series.
====
$ git bisect run sh -c \
"sed -i -e 's/list.nr || cached || show_in_pager/list.nr ||
show_in_pager/' builtin/grep.c || exit 129;
make -j 30 || exit 125;
git checkout HEAD -- builtin/grep.c ;
./bin-wrappers/git -C ~/git/purity grep --threads 20 --cached
'target-regex' || exit 1"
1d1729caebd41b340dd8dd61057f613da4df526c is the first bad commit
commit 1d1729caebd41b340dd8dd61057f613da4df526c
Author: Matheus Tavares <matheus.bernardino@usp.br>
Date: Wed Jan 15 23:39:54 2020 -0300
grep: replace grep_read_mutex by internal obj read lock
Here's what I'm testing with:
$ ./bin-wrappers/git --version
git version 2.28.0.585.ge1cfff6765
I have found that the crash always happens in cache_or_unpack_entry.
The thread that segfaults is always in patch_delta, and there are
always at least two threads inside unpack_compressed_entry at the time
of the failure.
I was able to get it to crash with just three threads one time. I
copied a log with the stack trace of all (4) threads during that run
to a gist [1].
Another log showing four different segfaults when run with 40 threads
is also in that gist [1].
[1] https://gist.github.com/phord/02e84d003688baa493b978110225d443
next reply other threads:[~2020-09-25 2:36 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-25 2:36 Phil Hord [this message]
2020-09-25 5:52 ` RFC - concurrency causes segfault in git grep since 2.26.0 Matheus Tavares
2020-09-25 19:53 ` Phil Hord
2020-09-28 16:50 ` [PATCH 0/2] Fix race condition and memory leak in delta base cache Matheus Tavares
2020-09-28 16:50 ` [PATCH 1/2] packfile: fix race condition on unpack_entry() Matheus Tavares
2020-09-28 18:05 ` Junio C Hamano
2020-09-28 16:50 ` [PATCH 2/2] packfile: fix memory leak in add_delta_base_cache() Matheus Tavares
2020-09-28 18:22 ` Junio C Hamano
2020-09-29 0:01 ` [PATCH v2 0/2] Fix race condition and memory leak in delta base cache Matheus Tavares
2020-09-29 0:01 ` [PATCH v2 1/2] packfile: fix race condition on unpack_entry() Matheus Tavares
2020-10-02 20:06 ` Phil Hord
2020-09-29 0:01 ` [PATCH v2 2/2] packfile: fix memory leak in add_delta_base_cache() Matheus Tavares
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CABURp0r_VPNeU5ugbJDQo=hV_aOit7W-pf9nhDF9Og=vVJG8tw@mail.gmail.com' \
--to=phil.hord@gmail.com \
--cc=dstolee@microsoft.com \
--cc=git@vger.kernel.org \
--cc=jonathantanmy@google.com \
--cc=matheus.bernardino@usp.br \
--cc=stefanbeller@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).