From: Jeff King <peff@peff.net>
To: Michael Haggerty <mhagger@alum.mit.edu>
Cc: git@vger.kernel.org, Johan Herland <johan@herland.net>
Subject: Re: another packed-refs race
Date: Mon, 6 May 2013 18:18:28 -0400 [thread overview]
Message-ID: <20130506221828.GA19851@sigill.intra.peff.net> (raw)
In-Reply-To: <20130506184122.GA23568@sigill.intra.peff.net>
On Mon, May 06, 2013 at 02:41:22PM -0400, Jeff King wrote:
> That is a weaker guarantee, and I think we can provide it with:
>
> 1. Load all loose refs into cache for a particular enumeration.
>
> 2. Make sure the packed-refs cache is up-to-date (by checking its
> stat() information and reloading if necessary).
>
> 3. Run the usual iteration over the loose/packed ref caches.
This does seem to work in my experiments. With stock git, I can trigger
the race reliably with:
# base load, in one terminal, as before
git init -q repo &&
cd repo &&
git commit -q --allow-empty -m one &&
one=`git rev-parse HEAD` &&
git commit -q --allow-empty -m two &&
two=`git rev-parse HEAD` &&
sha1=$one &&
while true; do
# this re-creates the loose ref in .git/refs/heads/master
if test "$sha1" = "$one"; then
sha1=$two
else
sha1=$one
fi &&
git update-ref refs/heads/master $sha1 &&
# we can remove packed-refs safely, as we know that
# its only value is now stale. Real git would not do
# this, but we are simulating the case that "master"
# simply wasn't included in the last packed-refs file.
rm -f .git/packed-refs &&
# and now we repack, which will create an up-to-date
# packed-refs file, and then delete the loose ref
git pack-refs --all --prune
done
# in another terminal, enumerate and make sure we never miss the ref
cd repo &&
while true; do
refs=`git.compile for-each-ref --format='%(refname)'`
echo "==> $refs"
test -z "$refs" && break
done
It usually takes about 30 seconds to hit a problem, though I measured
failures at up to 90 seconds. With the patch below (on top of the one I
posted the other day, which refreshes the packed-refs cache in
get_packed_refs), it has been running fine for 15 minutes.
The "noop_each_fn" is a little gross. I could also just reimplement the
recursion from do_for_each_ref_in_dir (except we don't care about flags,
trim, base, etc, and would just be calling get_ref_dir recursively).
It's a slight repetition of code, but it would be less subtle than what
I have written below (which uses a no-op callback for the side effect
that it primes the loose ref cache). Which poison do you prefer?
diff --git a/refs.c b/refs.c
index 45a7ee6..59ae7e4 100644
--- a/refs.c
+++ b/refs.c
@@ -1363,19 +1363,38 @@ static int do_for_each_ref(const char *submodule, const char *base, each_ref_fn
for_each_rawref(warn_if_dangling_symref, &data);
}
+static int noop_each_fn(const char *ref, const unsigned char *sha1, int flags,
+ void *data)
+{
+ return 0;
+}
+
static int do_for_each_ref(const char *submodule, const char *base, each_ref_fn fn,
int trim, int flags, void *cb_data)
{
struct ref_cache *refs = get_ref_cache(submodule);
- struct ref_dir *packed_dir = get_packed_refs(refs);
- struct ref_dir *loose_dir = get_loose_refs(refs);
+ struct ref_dir *packed_dir;
+ struct ref_dir *loose_dir;
int retval = 0;
- if (base && *base) {
- packed_dir = find_containing_dir(packed_dir, base, 0);
+ /*
+ * Prime the loose ref cache; we must make sure the packed ref cache is
+ * uptodate after we read the loose refs in order to avoid race
+ * conditions with a simultaneous "pack-refs --prune".
+ */
+ loose_dir = get_loose_refs(refs);
+ if (base && *base)
loose_dir = find_containing_dir(loose_dir, base, 0);
+ if (loose_dir) {
+ sort_ref_dir(loose_dir);
+ do_for_each_ref_in_dir(loose_dir, 0, base, noop_each_fn, 0, 0,
+ NULL);
}
+ packed_dir = get_packed_refs(refs);
+ if (base && *base)
+ packed_dir = find_containing_dir(packed_dir, base, 0);
+
if (packed_dir && loose_dir) {
sort_ref_dir(packed_dir);
sort_ref_dir(loose_dir);
next prev parent reply other threads:[~2013-05-06 22:18 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-03 8:38 another packed-refs race Jeff King
2013-05-03 9:26 ` Johan Herland
2013-05-03 17:28 ` Jeff King
2013-05-03 18:26 ` Jeff King
2013-05-03 21:02 ` Johan Herland
2013-05-06 12:12 ` Michael Haggerty
2013-05-06 18:44 ` Jeff King
2013-05-03 21:21 ` Jeff King
2013-05-06 12:03 ` Michael Haggerty
2013-05-06 18:41 ` Jeff King
2013-05-06 22:18 ` Jeff King [this message]
2013-05-07 4:32 ` Michael Haggerty
2013-05-07 4:44 ` Jeff King
2013-05-07 8:03 ` Michael Haggerty
2013-05-07 2:36 ` [PATCH 0/4] fix packed-refs races Jeff King
2013-05-07 2:38 ` [PATCH 1/4] resolve_ref: close race condition for packed refs Jeff King
2013-05-12 22:56 ` Michael Haggerty
2013-05-16 3:47 ` Jeff King
2013-05-16 5:50 ` Michael Haggerty
2013-05-12 23:26 ` Michael Haggerty
2013-06-11 14:26 ` [PATCH 0/4] Fix a race condition when reading loose refs Michael Haggerty
2013-06-11 14:26 ` [PATCH 1/4] resolve_ref_unsafe(): extract function handle_missing_loose_ref() Michael Haggerty
2013-06-11 14:26 ` [PATCH 2/4] resolve_ref_unsafe(): handle the case of an SHA-1 within loop Michael Haggerty
2013-06-11 14:26 ` [PATCH 3/4] resolve_ref_unsafe(): nest reference-reading code in an infinite loop Michael Haggerty
2013-06-11 14:26 ` [PATCH 4/4] resolve_ref_unsafe(): close race condition reading loose refs Michael Haggerty
2013-06-12 8:04 ` Jeff King
2013-06-13 8:22 ` Thomas Rast
2013-06-14 7:17 ` Michael Haggerty
2013-06-11 20:57 ` [PATCH 0/4] Fix a race condition when " Junio C Hamano
2013-05-07 2:39 ` [PATCH 2/4] add a stat_validity struct Jeff King
2013-05-13 2:29 ` Michael Haggerty
2013-05-13 3:00 ` [RFC 0/2] Separate stat_data from cache_entry Michael Haggerty
2013-05-13 3:00 ` [RFC 1/2] Extract a struct " Michael Haggerty
2013-05-13 3:00 ` [RFC 2/2] add a stat_validity struct Michael Haggerty
2013-05-13 5:10 ` [RFC 0/2] Separate stat_data from cache_entry Junio C Hamano
2013-05-16 3:51 ` [PATCH 2/4] add a stat_validity struct Jeff King
2013-05-07 2:43 ` [PATCH 3/4] get_packed_refs: reload packed-refs file when it changes Jeff King
2013-05-07 2:54 ` [PATCH 0/2] peel_ref cleanups changes Jeff King
2013-05-07 2:56 ` [PATCH 1/2] peel_ref: rename "sha1" argument to "peeled" Jeff King
2013-05-07 3:06 ` [PATCH 2/2] peel_ref: refactor for safety with simultaneous update Jeff King
2013-05-09 19:18 ` [PATCH 3/4] get_packed_refs: reload packed-refs file when it changes Eric Sunshine
2013-05-13 2:43 ` Michael Haggerty
2013-05-07 2:51 ` [PATCH 4/4] for_each_ref: load all loose refs before packed refs Jeff King
2013-05-07 6:40 ` [PATCH 0/4] fix packed-refs races Junio C Hamano
2013-05-07 14:19 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130506221828.GA19851@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=johan@herland.net \
--cc=mhagger@alum.mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).