From: Nasser Grainawi <nasser@codeaurora.org>
To: Derrick Stolee <stolee@gmail.com>
Cc: Andreas Krey <a.krey@gmx.de>,
git@vger.kernel.org, "peff@peff.net" <peff@peff.net>
Subject: Re: Avoid race condition between fetch and repack/gc?
Date: Mon, 16 Mar 2020 11:17:12 -0600 [thread overview]
Message-ID: <06992130-5109-4180-AB26-315AAF536788@codeaurora.org> (raw)
In-Reply-To: <759f4b3b-28a7-c002-ae51-5991bf9ad211@gmail.com>
> On Mar 16, 2020, at 6:10 AM, Derrick Stolee <stolee@gmail.com> wrote:
>
> On 3/16/2020 4:23 AM, Andreas Krey wrote:
>> Hi all,
>>
>> we occasionally seeing things like this:
>>
>> | DEBUG: 11:25:20: git -c advice.fetchShowForcedUpdates=false fetch --no-show-forced-updates -q --prune
>
> I'm happy to see these options. I hope they are helping you!
>
>> | Warning: Permanently added '[socgit.$company.com]:7999' (RSA) to the list of known hosts.
>> | remote: fatal: packfile ./objects/pack/pack-20256f2be3bd51b57e519a9f2a4d3df09f231952.pack cannot be accessed
> This _could_ mean a lot of things, but....
>
>> | error: git upload-pack: git-pack-objects died with error.
>> | fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
>> | remote: aborting due to possible repository corruption on the remote side.
>> | fatal: protocol error: bad pack header
>>
>> and when you look in the server repository there is a new packfile dated just around
>> that time. It looks like the fetch tries to access a packfile that it assumes to exist,
>> but the GC on the server throws it away just in that moment, and thus upload-pack fails.
>
> ...your intuition about repacking seems accurate. The important part of the
> race condition is likely that the server process read and holds a read handle
> on the .idx file, but when looking for the object contents it tries to open
> the .pack file which was deleted.
>
[snip]
>
>> Is there a way to avoid this?
>>
>> Should there be, like git repack waiting a bit before deleting old packfiles?
>
> This all depends on how you are managing your server. It is likely that you
> could create your own maintenance that handles this for you.
>
> The "git multi-pack-index (expire|repack)" cycle is built to prevent this sort
> of issue, but is not yet integrated well with reachability bitmaps. You likely
> require the bitmaps to keep your server performance, so that may not be a way
> forward for you.
We manage this on our servers with a repack wrapper that first creates hard links for all packfiles into a objects/pack/preserved dir and then we have patches on top of JGit [1] that actually know how to recover objects from that dir when the original pack is removed by repacking. It’s worked quite well for us for a couple years now and should be compatible with/without bitmaps (haven’t specifically tested) and any pack/repacking strategy.
[1] https://git.eclipse.org/r/122288
next prev parent reply other threads:[~2020-03-16 17:17 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-03-16 8:23 Avoid race condition between fetch and repack/gc? Andreas Krey
2020-03-16 12:10 ` Derrick Stolee
2020-03-16 17:17 ` Nasser Grainawi [this message]
2020-03-16 17:27 ` Jeff King
2020-03-16 23:40 ` Bryan Turner
2020-03-17 18:41 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=06992130-5109-4180-AB26-315AAF536788@codeaurora.org \
--to=nasser@codeaurora.org \
--cc=a.krey@gmx.de \
--cc=git@vger.kernel.org \
--cc=peff@peff.net \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).