git submodule: 3 modules same git repo?

git submodule: 3 modules same git repo?

[Joakim Tjernlund]

I am trying to create 3 submodules from the same git repo, each pointing to a different branch.
Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3
times, I want one clone and then have the 3 submodules to point to different commits.

Is this possible? If not, could it be added?

  Jocke

[PATCH 1/2] store submodule in common dir

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 path.c | 1 +
 1 file changed, 1 insertion(+)
 
On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
>
> I am trying to create 3 submodules from the same git repo, each pointing to a different branch.
> Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3
> times, I want one clone and then have the 3 submodules to point to different commits.
>
> Is this possible? If not, could it be added?

yup.

According to recent discussions, it would be just this patch.
(plus some unspecified amount of work, TBD).

I thought about proposing something proper later, but here is the WIP patch.

Thanks,
Stefan 

diff --git a/path.c b/path.c
index 34f0f98349a..64c9821b834 100644
--- a/path.c
+++ b/path.c
@@ -115,6 +115,7 @@ static struct common_dir common_list[] = {
 	{ 1, 1, 1, "logs/HEAD" },
 	{ 0, 1, 1, "logs/refs/bisect" },
 	{ 0, 1, 0, "lost-found" },
+	{ 0, 1, 0, "modules" },
 	{ 0, 1, 0, "objects" },
 	{ 0, 1, 0, "refs" },
 	{ 0, 1, 1, "refs/bisect" },
-- 
2.18.0.865.gffc8e1a3cd6-goog

Re: [PATCH 1/2] store submodule in common dir

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  path.c | 1 +
>  1 file changed, 1 insertion(+)
>  
> On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
>>
>> I am trying to create 3 submodules from the same git repo, each pointing to a different branch.
>> Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3
>> times, I want one clone and then have the 3 submodules to point to different commits.
>>
>> Is this possible? If not, could it be added?
>
> yup.
>
> According to recent discussions, it would be just this patch.
> (plus some unspecified amount of work, TBD).
>
> I thought about proposing something proper later, but here is the WIP patch.
>
> Thanks,
> Stefan 

My understanding of what Joakim wants to do is to have a top-level
project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4
and kernel/v2.6, each of which is a submodule that houses these
versions of Linux kernel source, but only clone Linus's repository
(as the up-to-late tree has all the necessary history to check out
these past development tracks).  And that should be doable with
just the main checkout, without any additional worktree (it's just
the matter of having .git/modules/kernel%2fv2.6/ directory pointed
by two symlinks from .git/modules/kernel%2fv2.[24], or something
like that).

Isn't "common_dir" stuff used to decide if each of separate
"worktree" instance (of the superproject) shares ".git/$stuff"
with each other?

Unless I am grossly misinterpreting the original question, I fail to
see how changing .git/modules to be shared across worktrees possibly
affects anything.  I am puzzled...

>
> diff --git a/path.c b/path.c
> index 34f0f98349a..64c9821b834 100644
> --- a/path.c
> +++ b/path.c
> @@ -115,6 +115,7 @@ static struct common_dir common_list[] = {
>  	{ 1, 1, 1, "logs/HEAD" },
>  	{ 0, 1, 1, "logs/refs/bisect" },
>  	{ 0, 1, 0, "lost-found" },
> +	{ 0, 1, 0, "modules" },
>  	{ 0, 1, 0, "objects" },
>  	{ 0, 1, 0, "refs" },
>  	{ 0, 1, 1, "refs/bisect" },

Re: [PATCH 1/2] store submodule in common dir

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> My understanding of what Joakim wants to do is to have a top-level
> project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4
> and kernel/v2.6, each of which is a submodule that houses these
> versions of Linux kernel source, but only clone Linus's repository
> (as the up-to-late tree has all the necessary history to check out
> these past development tracks).  And that should be doable with
> just the main checkout, without any additional worktree (it's just
> the matter of having .git/modules/kernel%2fv2.6/ directory pointed
> by two symlinks from .git/modules/kernel%2fv2.[24], or something
> like that).

Actually I take the last part of that back.  When thought naively
about, it may appear that it should be doable, but because each of
the modules/* directory in the top-level project has to serve as the
$GIT_DIR for each submodule checkout, and the desire is to have
these three directories to have checkout of three different
branches, a single directory under modules/. that is shared among
three submodules would *not* work---they must have separate index,
HEAD, etc.

Theoretically we should be able to make modules/kernel%2fv2.[24]
additional "worktree"s of modules/kernel%2fv2.6, but given that
these are all "bare" repositories without an attached working tree,
I am not sure how that would supposed to work.  Thinking about
having multiple worktrees on a single bare repository makes me head
spin and ache X-<;-)

Re: [PATCH 1/2] store submodule in common dir

[Stefan Beller]

On Tue, Aug 14, 2018 at 4:20 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > My understanding of what Joakim wants to do is to have a top-level
> > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4
> > and kernel/v2.6, each of which is a submodule that houses these
> > versions of Linux kernel source, but only clone Linus's repository
> > (as the up-to-late tree has all the necessary history to check out
> > these past development tracks).  And that should be doable with
> > just the main checkout, without any additional worktree (it's just
> > the matter of having .git/modules/kernel%2fv2.6/ directory pointed
> > by two symlinks from .git/modules/kernel%2fv2.[24], or something
> > like that).
>
> Actually I take the last part of that back.  When thought naively
> about, it may appear that it should be doable, but because each of
> the modules/* directory in the top-level project has to serve as the
> $GIT_DIR for each submodule checkout, and the desire is to have
> these three directories to have checkout of three different
> branches, a single directory under modules/. that is shared among
> three submodules would *not* work---they must have separate index,
> HEAD, etc.
>
> Theoretically we should be able to make modules/kernel%2fv2.[24]
> additional "worktree"s of modules/kernel%2fv2.6, but given that
> these are all "bare" repositories without an attached working tree,
> I am not sure how that would supposed to work.  Thinking about
> having multiple worktrees on a single bare repository makes me head
> spin and ache X-<;-)

Well the worktree feature would do all this in a safe manner, but the existing
feature of just cloning the submodules with a reference pointer to another
repository at least dedupes some of the object store, although warnings
need to be read carefully.

Regarding the worktree, I would think we'd want to have

  git worktree --recurse-submodules [list, add, etc]

that would do a sensible thing for each action on the worktrees,
but you seem to propose to have one of the three submodules
the main worktree and the other two would be just worktrees of
the first.

  I guess you could just

* init/update one of the submodules
* add their worktrees manually pointed to where
  the 2nd and 3rd submodule would go.
* make a symbolic link from
  ln -s .git/modules/<1st>/worktrees/<2nd> .git/modules/<2nd>
  ln -s .git/modules/<1st>/worktrees/<3rd> .git/modules/<3rd>

as then submodule operations should "just work" and by having the
worktrees in-place where the submodules are, we'd also have
all the protection against overzealous garbage collection, too.

Stefan

Re: [PATCH 1/2] store submodule in common dir

[Jonathan Nieder]

Junio C Hamano wrote:

> Theoretically we should be able to make modules/kernel%2fv2.[24]
> additional "worktree"s of modules/kernel%2fv2.6, but given that
> these are all "bare" repositories without an attached working tree,
> I am not sure how that would supposed to work.  Thinking about
> having multiple worktrees on a single bare repository makes me head
> spin and ache X-<;-)

Agreed about spinning head.  This is why I suggested at [1] that
anyone intereseted in this start with description of their proposed
design, which would have three benefits:

- after implementation, it would document the intent behind whatever
  we choose to do, hopefully saving people debugging or improving this
  code some head spinning

- it would allow subject matter experts on-list to suggest refinements
  and simplifications

- it would avoid the interested contributor going too far down a blind
  alley in case their initial proposal has a fatal flaw

I also agree with the "theoretically we should be able to make it
work".  As described in [1], I think most of this is work we're going
to have to do eventually, as part of properly supporting multiple
worktrees for a superproject.  But I don't want to set wrong
expectations: this will be hard.

Thanks,
Jonathan

[1] https://public-inbox.org/git/20180814231049.GH142615@aiede.svl.corp.google.com/

Re: [PATCH 1/2] store submodule in common dir

[Joakim Tjernlund]

On Tue, 2018-08-14 at 16:20 -0700, Junio C Hamano wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > My understanding of what Joakim wants to do is to have a top-level
> > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4
> > and kernel/v2.6, each of which is a submodule that houses these
> > versions of Linux kernel source, but only clone Linus's repository
> > (as the up-to-late tree has all the necessary history to check out
> > these past development tracks).  And that should be doable with
> > just the main checkout, without any additional worktree (it's just
> > the matter of having .git/modules/kernel%2fv2.6/ directory pointed
> > by two symlinks from .git/modules/kernel%2fv2.[24], or something
> > like that).
> 
> Actually I take the last part of that back.  When thought naively
> about, it may appear that it should be doable, but because each of
> the modules/* directory in the top-level project has to serve as the
> $GIT_DIR for each submodule checkout, and the desire is to have
> these three directories to have checkout of three different
> branches, a single directory under modules/. that is shared among
> three submodules would *not* work---they must have separate index,
> HEAD, etc.
> 
> Theoretically we should be able to make modules/kernel%2fv2.[24]
> additional "worktree"s of modules/kernel%2fv2.6, but given that
> these are all "bare" repositories without an attached working tree,
> I am not sure how that would supposed to work.  Thinking about
> having multiple worktrees on a single bare repository makes me head
> spin and ache X-<;-)

You nailed it ! :)
My head spins just reading this so I think I got my answer. I can be done
but will be tricky to impl. 
I will keep an eye on how submodules develops, sure would be a welcome feature.

 Jocke

Re: [PATCH 1/2] store submodule in common dir

[Joakim Tjernlund]

On Wed, 2018-08-15 at 10:28 +0200, Joakim Tjernlund wrote:
> On Tue, 2018-08-14 at 16:20 -0700, Junio C Hamano wrote:
> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > 
> > 
> > Junio C Hamano <gitster@pobox.com> writes:
> > 
> > > My understanding of what Joakim wants to do is to have a top-level
> > > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4
> > > and kernel/v2.6, each of which is a submodule that houses these
> > > versions of Linux kernel source, but only clone Linus's repository
> > > (as the up-to-late tree has all the necessary history to check out
> > > these past development tracks).  And that should be doable with
> > > just the main checkout, without any additional worktree (it's just
> > > the matter of having .git/modules/kernel%2fv2.6/ directory pointed
> > > by two symlinks from .git/modules/kernel%2fv2.[24], or something
> > > like that).
> > 
> > Actually I take the last part of that back.  When thought naively
> > about, it may appear that it should be doable, but because each of
> > the modules/* directory in the top-level project has to serve as the
> > $GIT_DIR for each submodule checkout, and the desire is to have
> > these three directories to have checkout of three different
> > branches, a single directory under modules/. that is shared among
> > three submodules would *not* work---they must have separate index,
> > HEAD, etc.
> > 
> > Theoretically we should be able to make modules/kernel%2fv2.[24]
> > additional "worktree"s of modules/kernel%2fv2.6, but given that
> > these are all "bare" repositories without an attached working tree,
> > I am not sure how that would supposed to work.  Thinking about
> > having multiple worktrees on a single bare repository makes me head
> > spin and ache X-<;-)
> 
> You nailed it ! :)
> My head spins just reading this so I think I got my answer. I can be done
> but will be tricky to impl. 
> I will keep an eye on how submodules develops, sure would be a welcome feature.
> 
>  Jocke

On a related note, it would be great if git could support sparse checkout/clone
for submodules, now one have to manually add .git/info/sparse-checkout.
If sparse clone and sparse checkout could be saved in the submodule, then there
would be no problem with 3 copies of the same repo in my submodules.

 Jocke

Re: [PATCH 1/2] store submodule in common dir

[Duy Nguyen]

On Wed, Aug 15, 2018 at 1:20 AM Junio C Hamano <gitster@pobox.com> wrote:
> Theoretically we should be able to make modules/kernel%2fv2.[24]
> additional "worktree"s of modules/kernel%2fv2.6, but given that
> these are all "bare" repositories without an attached working tree,
> I am not sure how that would supposed to work.  Thinking about
> having multiple worktrees on a single bare repository makes me head
> spin and ache X-<;-)

I could only read the mail subject this morning and thought a bit
about it on the train, and came to the same conclusion that "git
worktree" is the way to go.

The bareness should not affect this at all.  If you meant core.bare,
it can only affect the main repo. But what I'm thinking is repos with
only linked worktrees and no main one. So when you "submodule update"
a repo, you "clone -n", then "git worktree add" to put a worktree in
place. The repo is strictly speaking not bare, it just does not have a
main worktree.

The main HEAD should be detached so that it does not block any
worktree from checking out branches. If we could get away without the
main HEAD, that would be great, but HEAD is part of the repo signature
so it has to stay (and we may end up keeping more objects for this
HEAD even after every other heads don't need the same objects
anymore).

This also solves the problem with mupltiple worktrees on a repo with
submodules, where we have the same problems (if I add a new worktree
of the superproject, where do the new submodules go?). In this case
ideally all worktrees should have separate ref namespaces to maintain
the "separate repo" view that we currently have, but I guess we can
live with sharing refs for a while.

And simply sharing $GIT_DIR/modules won't work. For starter, it breaks
existing setups. What I've wanted to do is adding a shared "common"
directory. Then you could have "common/modules" (among other future
common stuff), which is shared across all worktrees, but you don't
have to add extra share rules since it's already covered by the
"common" rule.
-- 
Duy

Re: [PATCH 1/2] store submodule in common dir

[Stefan Beller]

On Tue, Aug 14, 2018 at 4:04 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Stefan Beller <sbeller@google.com> writes:
>
> > Signed-off-by: Stefan Beller <sbeller@google.com>
> > ---
> >  path.c | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
> >>
> >> I am trying to create 3 submodules from the same git repo, each pointing to a different branch.
> >> Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3
> >> times, I want one clone and then have the 3 submodules to point to different commits.
> >>
> >> Is this possible? If not, could it be added?
> >
> > yup.
> >
> > According to recent discussions, it would be just this patch.
> > (plus some unspecified amount of work, TBD).
> >
> > I thought about proposing something proper later, but here is the WIP patch.
> >
> > Thanks,
> > Stefan
>
> My understanding of what Joakim wants to do is to have a top-level
> project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4
> and kernel/v2.6, each of which is a submodule that houses these
> versions of Linux kernel source, but only clone Linus's repository
> (as the up-to-late tree has all the necessary history to check out
> these past development tracks).  And that should be doable with
> just the main checkout, without any additional worktree (it's just
> the matter of having .git/modules/kernel%2fv2.6/ directory pointed
> by two symlinks from .git/modules/kernel%2fv2.[24], or something
> like that).

Ah! I misunderstood due to fast reading.

For that I think you are interested in the feature added in d92a39590d1
(Add --reference option to git submodule., 2009-05-04), i.e.
both the update and add command take the --reference flag
that can be pointed at another repository such as an outside
clone of these three submodules, so some deduplication will
be performed.

> Isn't "common_dir" stuff used to decide if each of separate
> "worktree" instance (of the superproject) shares ".git/$stuff"
> with each other?
>
> Unless I am grossly misinterpreting the original question, I fail to
> see how changing .git/modules to be shared across worktrees possibly
> affects anything.  I am puzzled...

I did misunderstand grossly.

Stefan

Re: git submodule: 3 modules same git repo?

[Jonathan Nieder]

Hi,

Joakim Tjernlund wrote:

> I am trying to create 3 submodules from the same git repo, each pointing to a different branch.
> Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3
> times, I want one clone and then have the 3 submodules to point to different commits.
>
> Is this possible? If not, could it be added?

It's an interesting case.  You're not the only one that has wanted
something like this: for example, "repo" (a similar tool with some
differences) included the change [1] (repo: Support multiple branches
for the same project, 2014-01-30) for this kind of application.

In practice, it's a bit messy.  To allow switching to a branch where
the submodule is not present, we store the submodule .git directory
in the superproject's .git/modules/<submodule name> directory.  This is
an ordinary .git directory, with files like

	HEAD
	config
	packed-refs
	[etc]

The "git worktree" tool allows having multiple worktrees for a single
Git repository.  Each worktree has its own HEAD.  So the layout would
look something like

	.git/modules/<submodule name>/
		config
		packed-refs
		[etc]
		worktrees/
			<branch1>/
				HEAD
			<branch2>/
				HEAD

By piggy-backing on the "git worktree" feature, this should work
great, but it will take some work to teach Git to set it up.

A side note: as Stefan (cc-ed) mentioned, there is another, related
interaction between submodules and "git worktree".  Namely: I might want
to have multiple worktrees for a single superproject Git repository.
In that case, the layout might look something like

	.git/
		HEAD
		config
		packed-refs
		[etc]
		modules/
			<submodule name>/
				HEAD
				config
				packed-refs
				...
		worktrees/
			<branch1>/
				HEAD
			<branch2>/
				HEAD

The patch that Stefan sent heads in this direction, but it has a
problem: if the submodule is checked out in both worktrees, then they
cannot have the same HEAD.  So to support that goal well would also
require supporting the goal you've described as a side effect: each
submodule would need to have multiple worktrees, at least one per
worktree of the superproject.

Sorry, that got a bit contorted after a while.  If you'd be interested
in pursuing this, I'd be happy to review any thoughts you have (for
example if you write a brief design doc).

My experience from interacting with the feature [1] is that it is easy
to make mistakes when trying to support this kind of case.  (repo had
some bugs due to, for example, multiple checkouts of a repository
trying to run "git prune" at the same time.)  The "git worktree"
feature should be a good foundation to build on that avoids some of
the problems encountered there.

Thanks,
Jonathan

[1] https://gerrit-review.googlesource.com/c/git-repo/+/50715/