[PATCH] new-workdir: Never try to recurse into submodules on the initial checkout.

[PATCH] new-workdir: Never try to recurse into submodules on the initial checkout.

[Marc Branchaud]

The new workdir is empty before the checkout, so attempts to recurse into
a non-existent submodule directory fail.

Signed-off-by: Marc Branchaud <marcnarc@xiplink.com>
---

Until the worktree command supports submodules I've gone back to using the
git-new-workdir script, but it fails if my config has
submdodule.recurse=true.

With this patch, the checkout succeeds and the workdir has empty submodules,
which is the script's normal behaviour.

		M.

 contrib/workdir/git-new-workdir | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir
index 888c34a521..5de1dc3c58 100755
--- a/contrib/workdir/git-new-workdir
+++ b/contrib/workdir/git-new-workdir
@@ -102,4 +102,4 @@ trap - $siglist
 
 # checkout the branch (either the same as HEAD from the original repository,
 # or the one that was asked for)
-git checkout -f $branch
+git -c submodule.recurse=false checkout -f $branch
-- 
2.20.1.1.gfb6d716d28

Re: [PATCH] new-workdir: Never try to recurse into submodules on the initial checkout.

[Jonathan Nieder]

Hi,

Marc Branchaud wrote:

> The new workdir is empty before the checkout, so attempts to recurse into
> a non-existent submodule directory fail.
>
> Signed-off-by: Marc Branchaud <marcnarc@xiplink.com>
> ---

Thanks for reporting.  Can you describe the error message when it fails
here?

> Until the worktree command supports submodules I've gone back to using the
> git-new-workdir script, but it fails if my config has
> submdodule.recurse=true.

Oh, dear.  In general, the project does a better job at supporting "git
worktree" than "git new-workdir", but I don't blame you about this.

Noting locally as another vote for getting submodules to play well with
worktrees soon.

[...]
>  contrib/workdir/git-new-workdir | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir
> index 888c34a521..5de1dc3c58 100755
> --- a/contrib/workdir/git-new-workdir
> +++ b/contrib/workdir/git-new-workdir
> @@ -102,4 +102,4 @@ trap - $siglist
> 
>  # checkout the branch (either the same as HEAD from the original repository,
>  # or the one that was asked for)
> -git checkout -f $branch
> +git -c submodule.recurse=false checkout -f $branch

nit: can this use "git checkout --no-recurse-submodules" instead
of -c?

In general, we tend to recommend that kind of option instead of
--config in scripts.

Thanks,
Jonathan

Re: [PATCH] new-workdir: Never try to recurse into submodules on the initial checkout.

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

>> The new workdir is empty before the checkout, so attempts to recurse into
>> a non-existent submodule directory fail.
>>
>> Signed-off-by: Marc Branchaud <marcnarc@xiplink.com>
>> ---
>
> Thanks for reporting.  Can you describe the error message when it fails
> here?
>
>> Until the worktree command supports submodules I've gone back to using the
>> git-new-workdir script, but it fails if my config has
>> submdodule.recurse=true.
>
> Oh, dear.  In general, the project does a better job at supporting "git
> worktree" than "git new-workdir", but I don't blame you about this.
>
> Noting locally as another vote for getting submodules to play well with
> worktrees soon.
>
> [...]
>>  contrib/workdir/git-new-workdir | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir
>> index 888c34a521..5de1dc3c58 100755
>> --- a/contrib/workdir/git-new-workdir
>> +++ b/contrib/workdir/git-new-workdir
>> @@ -102,4 +102,4 @@ trap - $siglist
>> 
>>  # checkout the branch (either the same as HEAD from the original repository,
>>  # or the one that was asked for)
>> -git checkout -f $branch
>> +git -c submodule.recurse=false checkout -f $branch
>
> nit: can this use "git checkout --no-recurse-submodules" instead
> of -c?
>
> In general, we tend to recommend that kind of option instead of
> --config in scripts.

I am not sure if either approach makes sense.  Wouldn't the ideal
endgame to allow recursive checkout if the user wants to have it,
but not enable it by default?

Stepping back a bit, if the user has recursive checkout configured
somewhere valid for this repository (or worktree), shouldn't the
initial checkout also recurse and do a "submodule init" if that is
necessary before doing so?

IOW, at the point in that script where we call "git checkout -f", if
we changed it to "git checkout --recurse-submodules -f", what breaks
and why?  Shouldn't it succeed instead?

Re: [PATCH] new-workdir: Never try to recurse into submodules on the initial checkout.

[Jonathan Nieder]

Hi,

Junio C Hamano wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>> Marc Branchaud wrote:

>>> The new workdir is empty before the checkout, so attempts to recurse into
>>> a non-existent submodule directory fail.
>>>
>>> Signed-off-by: Marc Branchaud <marcnarc@xiplink.com>
>>> ---
>>
>> Thanks for reporting.  Can you describe the error message when it fails
>> here?
[...]
> IOW, at the point in that script where we call "git checkout -f", if
> we changed it to "git checkout --recurse-submodules -f", what breaks
> and why?  Shouldn't it succeed instead?

I think that's a similar question to the one I asked.

But I have a good guess about what goes wrong.  It's related to the same
issue as the "git worktree" problem Marc described.

Inside the superproject's $GIT_DIR, we see

	config
	modules/
		a/
			config
		b/
			config
	...

The question is what to do with the modules/ directory when you have
multiple working directories making use of the refs and objects from
this $GIT_DIR.

In general, the most useful answer is that the additional working
directories should make use for modules/ from this $GIT_DIR as well.
After all, each submodule has its own refs and objects, and the same
motivation that pushes us to share the refs and objects from the
superproject would drive us to share them in the submodules as well.

However, if you do this in the most naive way, it will not work.  In
the config file, there is a core.worktree setting that ensures that
commands run from a submodule affect the correct working directory.
Which worktree should it point to?  All of them.

There's still an obvious "most useful" answer: each submodule should
contain its own worktrees/ directory with metadata specific to each
worktree.  This should work fine and is the future work that Marc and
I alluded to.  Let me call it (*), for later reference.

Anything done today is papering over the sad truth that that future
work (*) has not been done yet.

contrib/workdir is currently naive about all this: it does *not*
symlink across the modules/ directory, so each workdir gets its own
independent copy of all the submodules.  Which kind of defeats the
point of this kind of setup.

That said, it's better than nothing at all, which is why Marc proposes
making it not attempt to check out the submodules right away, instead
permitting the user to make the best of things.  I suppose another
thing that is missing is a warning message to ensure the user knows
that once (*) arrives, they need to be ready for it.  (Or not: this is
contrib/workdir, and there would be no need to make use of it in place
of "git worktree" once that moment arrives.)

To reiterate, this is all about papering over (*) not having been
done.

Marc, did I understand correctly?

Thanks,
Jonathan

Re: [PATCH] new-workdir: Never try to recurse into submodules on the initial checkout.

[Marc Branchaud]

On 2019-01-14 4:34 p.m., Jonathan Nieder wrote:
> Hi,
> 
> Marc Branchaud wrote:
> 
>> The new workdir is empty before the checkout, so attempts to recurse into
>> a non-existent submodule directory fail.
>>
>> Signed-off-by: Marc Branchaud <marcnarc@xiplink.com>
>> ---
> 
> Thanks for reporting.  Can you describe the error message when it fails
> here?

The error is:

fatal: exec '--super-prefix=external/submodule/': cd to 
'external/submodule' failed: No such file or directory

The created workdir has only the .git directory.  The .git/HEAD file 
contains the expected ref, so the workdir repo's status simply shows 
that everything has been deleted.


Note that git-worktree also fails when submodule.recurse=true, with the 
same error:

# git worktree add ~/Code/foo/test-worktree
Preparing worktree (new branch 'test-worktree')
fatal: exec '--super-prefix=external/submodule/': cd to 
'external/submodule' failed: No such file or directory
error: Submodule 'external/submodule' could not be updated.
error: Submodule 'external/submodule' cannot checkout new HEAD.
fatal: Could not reset index file to revision 'HEAD'.

I had assumed that this was simply an aspect of submodules not working, 
so I was holding off reporting it until more of the submodule support 
was complete.

>> Until the worktree command supports submodules I've gone back to using the
>> git-new-workdir script, but it fails if my config has
>> submdodule.recurse=true.
> 
> Oh, dear.  In general, the project does a better job at supporting "git
> worktree" than "git new-workdir", but I don't blame you about this.
> 
> Noting locally as another vote for getting submodules to play well with
> worktrees soon.
> 
> [...]
>>   contrib/workdir/git-new-workdir | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir
>> index 888c34a521..5de1dc3c58 100755
>> --- a/contrib/workdir/git-new-workdir
>> +++ b/contrib/workdir/git-new-workdir
>> @@ -102,4 +102,4 @@ trap - $siglist
>>
>>   # checkout the branch (either the same as HEAD from the original repository,
>>   # or the one that was asked for)
>> -git checkout -f $branch
>> +git -c submodule.recurse=false checkout -f $branch
> 
> nit: can this use "git checkout --no-recurse-submodules" instead
> of -c?
> 
> In general, we tend to recommend that kind of option instead of
> --config in scripts.

--no-recurse-submodules does work.  I'll send a v2.

		M.

> Thanks,
> Jonathan
> 

Re: [PATCH] new-workdir: Never try to recurse into submodules on the initial checkout.

[Marc Branchaud]

On 2019-01-14 4:58 p.m., Jonathan Nieder wrote:
> Hi,
> 
> Junio C Hamano wrote:
>> Jonathan Nieder <jrnieder@gmail.com> writes:
>>> Marc Branchaud wrote:
> 
>>>> The new workdir is empty before the checkout, so attempts to recurse into
>>>> a non-existent submodule directory fail.
>>>>
>>>> Signed-off-by: Marc Branchaud <marcnarc@xiplink.com>
>>>> ---
>>>
>>> Thanks for reporting.  Can you describe the error message when it fails
>>> here?
> [...]
>> IOW, at the point in that script where we call "git checkout -f", if
>> we changed it to "git checkout --recurse-submodules -f", what breaks
>> and why?  Shouldn't it succeed instead?
> 
> I think that's a similar question to the one I asked.
> 
> But I have a good guess about what goes wrong.  It's related to the same
> issue as the "git worktree" problem Marc described.
> 
> Inside the superproject's $GIT_DIR, we see
> 
> 	config
> 	modules/
> 		a/
> 			config
> 		b/
> 			config
> 	...
> 
> The question is what to do with the modules/ directory when you have
> multiple working directories making use of the refs and objects from
> this $GIT_DIR.
> 
> In general, the most useful answer is that the additional working
> directories should make use for modules/ from this $GIT_DIR as well.
> After all, each submodule has its own refs and objects, and the same
> motivation that pushes us to share the refs and objects from the
> superproject would drive us to share them in the submodules as well.
> 
> However, if you do this in the most naive way, it will not work.  In
> the config file, there is a core.worktree setting that ensures that
> commands run from a submodule affect the correct working directory.
> Which worktree should it point to?  All of them.
> 
> There's still an obvious "most useful" answer: each submodule should
> contain its own worktrees/ directory with metadata specific to each
> worktree.  This should work fine and is the future work that Marc and
> I alluded to.  Let me call it (*), for later reference.
> 
> Anything done today is papering over the sad truth that that future
> work (*) has not been done yet.
> 
> contrib/workdir is currently naive about all this: it does *not*
> symlink across the modules/ directory, so each workdir gets its own
> independent copy of all the submodules.  Which kind of defeats the
> point of this kind of setup.
> 
> That said, it's better than nothing at all, which is why Marc proposes
> making it not attempt to check out the submodules right away, instead
> permitting the user to make the best of things.  I suppose another
> thing that is missing is a warning message to ensure the user knows
> that once (*) arrives, they need to be ready for it.  (Or not: this is
> contrib/workdir, and there would be no need to make use of it in place
> of "git worktree" once that moment arrives.)
> 
> To reiterate, this is all about papering over (*) not having been
> done.
> 
> Marc, did I understand correctly?

Yup!

I just hope to keep git-new-workdir hobbling along until worktree can 
fully replace it.

I agree with Junio about what should happen when submodule.recurse=true. 
  But that work should be done in worktree instead of git-new-workdir.

		M.

> Thanks,
> Jonathan
> 

[PATCHv2] new-workdir: Never try to recurse into submodules on the initial checkout.

[Marc Branchaud]

The new workdir is empty before the checkout, so attempts to recurse into
a non-existent submodule directory fail.

Signed-off-by: Marc Branchaud <marcnarc@xiplink.com>
---

Changed to use --no-recurse-submodules instead of -c submodule.recurse=false,
as Jonathan suggested.

		M.

 contrib/workdir/git-new-workdir | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir
index 888c34a521..d88765e73f 100755
--- a/contrib/workdir/git-new-workdir
+++ b/contrib/workdir/git-new-workdir
@@ -102,4 +102,4 @@ trap - $siglist
 
 # checkout the branch (either the same as HEAD from the original repository,
 # or the one that was asked for)
-git checkout -f $branch
+git checkout --no-recurse-submodules -f $branch
-- 
2.20.1.1.gfb6d716d28

Re: [PATCH] new-workdir: Never try to recurse into submodules on the initial checkout.

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> That said, it's better than nothing at all, which is why Marc proposes
> making it not attempt to check out the submodules right away, instead
> permitting the user to make the best of things.

OK.  Then I do agree with the hardcoded "--no-recurse" given to "git
checkout" at the last step in the code.

Thanks, both.