Re: [PATCH 12/12] t0612: add tests to exercise Git/JGit reftable compatibility

[Patrick Steinhardt <ps@pks.im>]

[-- Attachment #1: Type: text/plain, Size: 2690 bytes --]

On Sat, Apr 06, 2024 at 03:03:20PM +0200, Han-Wen Nienhuys wrote:
> On Fri, Apr 5, 2024 at 11:41 AM Patrick Steinhardt <ps@pks.im> wrote:
> > > > +test_expect_success 'JGit repository can be read by CGit' '
> > > > +       test_when_finished "rm -rf repo" &&
> > > > +       # JGit does not provide a way to create a reftable-enabled repository.
> > >
> > > You can do "jgit init && jgit convert-refstorage --format=reftable"
> >
> > Perfect, thanks for this hint!
> 
> on a tangent: I wrote this a long time ago, and it does no locking,
> which probably leads to unpleasant surprises if it is run in parallel
> with other ref updates. What would be a foolproof way to stop other
> processes from stomping over the repo while the conversion is ongoing?
> Maybe create a packed-refs.lock file?

Yes, this is indeed a problem that I have been thinking about quite a
lot recently as I want to introduce migration tooling in the next
release cycle. I was quite happy to learn that you've got something in
JGit and was hoping that you found a way to avoid races. Too bad you
seem to have the same issue as I do right now.

In any case, creating a "packed-refs.lock" is not sufficient. While it
does protect against ref deletion and repacking refs, it doesn't protect
against ref creation and updates. And in fact there is no way to guard
against these at all as all that the "files" backend does is to create
or update a single loose file without taking any central lock.

I was briefly thinking about a "migration" repository extension. Once
the migration starts we add "extensions.ref-migration" to the repo,
which is an extension that no client knows how to handle. Consequently,
while the extension is set, any _new_ Git processes would bail out and
refuse to update the repository.

That only solves part of the problem though. What it doesn't protect
against is already-running Git processes that have already read the
config.

There are of course very hacky ways to get closer to the goal. E.g.:

    1. Repack all refs.

    2. Delete the now-empty "refs/" hierarchy.

    3. Create a file "refs" with invalid contents.

    4. Lock the "packed-refs" file.

    5. Lock the "HEAD" file.

Now you can be sure that there is no way to create or update normal
refs. But it's still incomplete as pseudo-refs can be written to. And it
does feel very fragile overall, especially if you need to roll back the
migration due to whatever reason.

So I dunno, until now I didn't yet find any way to reliably avoid racy
ref updates during a conversion. If you or anybody else got some ideas
I'd be very happy to hear about them.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]