Re: Bug: All git operations fail when .git contains a non-existent gitdir

[David Aguilar <davvid@gmail.com>]

On Thu, Jul 22, 2021 at 6:16 AM Tom Cook <tom.k.cook@gmail.com> wrote:
>
> The easiest way to reproduce it is this:
>
> $ mkdir test
> $ cd test
> $ echo "gitdir: /foo/bar" > .git
> $ git ls-remote https://github.com/torvalds/linux
>
> We happen to use overlay mounts in our build system in a way that maps
> a git submodule from one place to another so that its "gitdir" is
> invalid and then attempt a `git ls-remote` from that location which
> unexpectedly fails.  But the above reproduces the problem well enough.

I'm making a guess about your setup. To me it sounds like tweaking the
overlayfs or bind mounts to also map the .git directory of the parent
repository (at just the right relative location inside the build
environment) is the right approach towards making the .git-file links
happy.

Submodules point to the parent repository's .git/modules/ directory
these days so poking those paths through into the mapped file system
is the solution that's going to lead to the least amount of
hard-to-diagnose issues in the future.

The .git-file uses relative paths so it is possible to remap and
relocate them as you're doing, but it's going to require another mount
in the right relative location to do it.

It seems doable, even for most edge cases. For example if you end up
mapping some/deeply/nested/submod into /shallow inside your
container(?) and the .git-file contains
"../../../../.git/modules/submod", that might seem impossible to
resolve from a "/shallow" directory, but actually it's fine.

The solution in that situation is to map the parent repo's .git/ dir
to the "/.git" filesystem root inside the build environment. Unix
treats "/../../../../../../" as equivalent to "/" so if the .git-file
has a bunch of "../../../" parent-dir traversing going on then it'll
still work from a one-level deep "/shallow" directory. The parent git
repo directory would be expected to be found at "/.git" (with a
"modules" directory inside it) because the parent directory of "/" is
also "/".

This should mesh right in with the setup you described because you're
already using overlay mounts to synthesize a filesystem.

There are edge cases where this approach breaks down, though. One of
these cases is submodules inside submodules. Luckily it doesn't sound
like you have that problem.

There are patches now that special-case being able to run ls-remote
<url> in a broken setup, but the real fix is to extend the build
environment's filesystem view so that the submodules have access to
the underlying .git/modules/ storage.

Hopefully it's just a --mount or --volume docker(?) command-line flag
that has to be extended to poke the parent .git/ repository contents
into the environment. That's the avenue I would explore if I were
already down a path of using mount tricks.

I hope that helps and maybe sparks an idea that you can run with. I
certainly don't mean it to be a dismissive, "go change your setup"
kind of advice, but it would be interesting to know more details since
it seems like achieving correctness is going to require doing that
based on the limited information in this thread about the specifics of
the setup.

Also, I wouldn't be surprised if, even after the "git ls-remote"
robustness fixes are applied, you'll just stumble onto the next git
command that can only be resolved by mapping in the parent repo's .git
directory.

--
David