Re: [PATCH v5 0/4] cache parent project's gitdir in submodules

[Emily Shaffer <emilyshaffer@google.com>]

On Sun, Nov 07, 2021 at 08:24:43PM -0500, Derrick Stolee wrote:
> 
> On 11/4/2021 7:49 PM, Emily Shaffer wrote:
> 
> > The only real change here is a slight semantics change to map from
> > <submodule gitdir> to <superproject common git dir>. In every case
> > *except* for when the superproject has a worktree, this changes nothing.
> > For the case when the superproject has a worktree, this means that now
> > submodules will refer to the general superproject common dir (e.g. no
> > worktree-specific refs or configs or whatnot).
> > 
> > I *think* that because a submodule should exist in the context of the
> > common dir, not the worktree gitdir, that is ok. However, it does mean
> > it would be difficult to do something like sharing a config specific to
> > the worktree (the initial goal of this series).
> > 
> > $ROOT/.git
> > $ROOT/.git/config.superproject <- shared by $ROOT/.git/modules/sub
> > $ROOT/.git/modules/sub <- points to $ROOT/.git
> > $ROOT/.git/worktrees/wt
> > $ROOT/.git/worktrees/wt/config.superproject <- contains a certain config-based pre-commit hook
> > 
> > If the submodule only knows about the common dir, that is tough, because
> > the submodule would basically have to guess which worktree it's in from
> > its own path. There would be no way for '$WT/sub' to inherit
> > '$ROOT/.git/worktrees/wt/config.superproject'.
> > 
> > That said... right now, we don't support submodules in worktrees very
> > well at all. A submodule in a worktree will get a brand new gitdir in
> > $ROOT/.git/worktrees/modules/ (and that brand new gitdir would point to
> > the super's common dir). So I think we can punt on this entire question
> > until we teach submodules and worktrees to play more gracefully together
> > (it's on my long list...),
> 
> (I omit a portion that will be discussed later.)
> 
> > Or, to summarize the long ramble above: "this is still kind of weird
> > with worktrees, but let's fix it later when we fix worktrees more
> > thoroughly".
> 
> I'm concerned about punting here, because making a messy situation worse
> is unlikely to have a clean way out. Could we set up a design that works
> with superproject worktrees?
> 
> You mentioned that submodules cannot have worktrees. At least, you said
> that 'absorbgitdirs' does not allow them. Could those subprojects still
> exist and be registered as submodules without using that command?
> 
> What I'm trying to hint at is that if the submodules can't have
> worktrees, then maybe we could make their 'config.worktree' files be
> relative to the superproject worktrees. Then, these submodules could
> point to the commondir in their base config and _also_ to the worktree
> gitdir in their config.worktree.
> 
> The issue that is immediately obvious here is that my definition is
> circular: we need to know the superproject worktree in order to discover
> the config.worktree which contains the information about the superproject
> worktree.
> 
> > and at that time we can probably introduce a
> > pointer from $ROOT/.git/modules/sub/worktrees/wt/ to
> > $ROOT/.git/worktrees/wt/....
> 
> Your idea here appears to assume that if the superproject has worktrees,
> then the submodule is divided into worktrees in an exact correspondence.
> This would allow the submodule's config.worktree to point to the
> superproject's worktree (or possibly it could be inferred from the
> submodule's worktree relative to the submodule's commondir).
> 
> This seems like an interesting way forward, but requires changing how
> 'git absorbgitdirs' works, along with changes to 'git worktree' or other
> submodule commands when the submodule first appears during a 'git checkout'
> in a worktree. I imagine there are a lot of "gotchas" here. It is worth
> spending some time imagining how to create this setup and/or enforce it
> as submodules are added in the lifecycle of a repository, if only to
> validate the config design presented by this series.

Yeah, I think we may be overthinking it, especially with the concerns
about common dir vs. gitdir. More specifically - I think we accidentally
did the right thing in the previous iteration by using the gitdir :)

I think we can probably put it pretty simply:
submodule.superprojectGitDir should point from the most local gitdir of
the submodule to the most local gitdir of the superproject.

Luckily there are not so many permutations to worry about here.

Super doesn't have worktrees, sub doesn't have worktrees:
.git/
  modules/
    sub/
      config <- contains a pointer to ".git/"
  config

Super doesn't have worktrees, sub does have worktrees (and as you
suggest above, right now this would have to be created carefully and
manually, but later we probably want this to Just Work):
.git/
  modules/
    sub/
      config
      config.worktree <- contains a pointer to ".git/"
      worktrees/
        sub-wt/
	config <- contains a pointer to ".git/"
  config

Super has worktrees, sub doesn't have worktrees:
Actually, I think in the future this might not be possible, if we want
to make `git worktree add --recurse-submodules` work gracefully (and I
do want that). But in the interim, in practice it looks like this:
.git/
  modules/
    sub/
      config <- contains a pointer to ".git/"
  worktrees/
    super-wt/
      modules/
        sub/
	  config <- contains a pointer to ".git/worktrees/super-wt"
      config
  config
This case is pretty weird anyway, because in order for a submodule to be
present in multiple worktrees of the superproject, the submodule itself
needs to either have multiple worktrees or multiple repos. But on the
flip side, today it's impossible for a single submodule gitdir to need
to know about more superproject worktrees than the one it was
initted/whatever into.

Both super and sub have worktrees:
And this won't exist until we have graceful support of `git worktree add
--recurse-submodules` or with some manual effort, now.
.git/
  modules/
    sub/
      worktrees/
        sub-wt/
	  config <- contains a pointer to ".git/worktrees/super-wt/"
      config
      config.worktree <- contains a pointer to ".git/"
  worktrees/
    super-wt/
      config
  config
  config.worktree

I think this will give us access to both the worktree gitdir *and* the
common gitdir:

  ~/git/.git/worktrees/git-second [GIT_DIR!]$ git rev-parse --git-common-dir
  /usr/local/google/home/emilyshaffer/git/.git

So that means from any submodule, we can determine:
 - submodule's gitdir (from the .git link in the submodule wt)
 - submodule's common dir (from existing commands)
 - gitdir of superproject which submodule inhabits (from the config in
   the submodule's gitdir, or the submodule's config.worktree)
 - common dir of superproject (from existing commands + prior config)

The upshot to me, then, means that we should be 1) making sure to get
the path to the gitdir, not the common dir, of the superproject; and 2)
using helpers to write to the worktree config, not to the local config,
of the submodule. In other words, we want to avoid the following:

.git/
  modules/
    sub/
      worktrees/
        wt/
	  config
      config <- "submodule.superprojectGitDir = ../../../.." as written by the worktree

Will take a look at the rest of the comments too, but this sounds like a
reasonable approach to me.

 - Emily

> 
> Thanks,
> -Stolee