[PATCH] add test for bug in git-mv with nested submodules

[PATCH] add test for bug in git-mv with nested submodules

[Heiko Voigt]

When using git-mv with a submodule it will detect that and update the
paths for its configurations (.gitmodules, worktree and gitfile). This
does not work for nested submodules where a user renames the root
submodule.

We discovered this fact when working on on-demand fetch for renamed
submodules. Lets add a test to document.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
 t/t7001-mv.sh | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
index e365d1f..39f8aed 100755
--- a/t/t7001-mv.sh
+++ b/t/t7001-mv.sh
@@ -491,4 +491,13 @@ test_expect_success 'moving a submodule in nested directories' '
 	test_cmp actual expect
 '
 
+test_expect_failure 'moving nested submodules' '
+	git commit -am "cleanup commit" &&
+	git submodule add ./. sub_nested &&
+	git commit -m "add sub_nested" &&
+	git submodule update --init --recursive &&
+	git mv sub_nested sub_nested_moved &&
+	git status
+'
+
 test_done
-- 
2.0.0.274.g6b2cd91

Re: [PATCH] add test for bug in git-mv with nested submodules

[Stefan Beller]

On Thu, Aug 17, 2017 at 3:34 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> When using git-mv with a submodule it will detect that and update the
> paths for its configurations (.gitmodules, worktree and gitfile). This
> does not work for nested submodules where a user renames the root
> submodule.
>
> We discovered this fact when working on on-demand fetch for renamed
> submodules. Lets add a test to document.
>
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
>  t/t7001-mv.sh | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> index e365d1f..39f8aed 100755
> --- a/t/t7001-mv.sh
> +++ b/t/t7001-mv.sh
> @@ -491,4 +491,13 @@ test_expect_success 'moving a submodule in nested directories' '
>         test_cmp actual expect
>  '
>
> +test_expect_failure 'moving nested submodules' '
> +       git commit -am "cleanup commit" &&
> +       git submodule add ./. sub_nested &&

If possible, I would avoid adding the repo itself
as a submodule as it is unrealistic in the wild.

While it may be ok for the test here, later down the road
other tests making use of it it may become an issue with
the URL of the submodule.

> +       git commit -m "add sub_nested" &&
> +       git submodule update --init --recursive &&
> +       git mv sub_nested sub_nested_moved &&
> +       git status
> +'
> +
>  test_done
> --
> 2.0.0.274.g6b2cd91
>

Re: [PATCH] add test for bug in git-mv with nested submodules

[Heiko Voigt]

On Thu, Aug 17, 2017 at 12:05:56PM -0700, Stefan Beller wrote:
> On Thu, Aug 17, 2017 at 3:34 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > When using git-mv with a submodule it will detect that and update the
> > paths for its configurations (.gitmodules, worktree and gitfile). This
> > does not work for nested submodules where a user renames the root
> > submodule.
> >
> > We discovered this fact when working on on-demand fetch for renamed
> > submodules. Lets add a test to document.
> >
> > Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> > ---
> >  t/t7001-mv.sh | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> > index e365d1f..39f8aed 100755
> > --- a/t/t7001-mv.sh
> > +++ b/t/t7001-mv.sh
> > @@ -491,4 +491,13 @@ test_expect_success 'moving a submodule in nested directories' '
> >         test_cmp actual expect
> >  '
> >
> > +test_expect_failure 'moving nested submodules' '
> > +       git commit -am "cleanup commit" &&
> > +       git submodule add ./. sub_nested &&
> 
> If possible, I would avoid adding the repo itself
> as a submodule as it is unrealistic in the wild.
> 
> While it may be ok for the test here, later down the road
> other tests making use of it it may become an issue with
> the URL of the submodule.

I just copied the shortcut that they were adding themselfes as submodule
in 'setup submodule'. The whole setup of submodules in this test is like
this. This way we already had a nested submodule structure which I could
just add.

I agree that this is unrealistic so I can change that in the test I am
adding. But from what I have seen, this shortcut is taken in quite some
places when dealing with submodules.

Cheers Heiko

Re: [PATCH] add test for bug in git-mv with nested submodules

[Stefan Beller]

> I just copied the shortcut that they were adding themselfes as submodule
> in 'setup submodule'. The whole setup of submodules in this test is like
> this. This way we already had a nested submodule structure which I could
> just add.
>
> I agree that this is unrealistic so I can change that in the test I am
> adding. But from what I have seen, this shortcut is taken in quite some
> places when dealing with submodules.

Please do not make it worse.
Once upon a time (late '16 IIRC) I had a series floating on the
list removing all occurrences, but there were issues with the
series and it did not land.

[PATCH v2] add test for bug in git-mv for recursive submodules

[Heiko Voigt]

When using git-mv with a submodule it will detect that and update the
paths for its configurations (.gitmodules, worktree and gitfile). This
does not work for recursive submodules where a user renames the root
submodule.

We discovered this fact when working on on-demand fetch for renamed
submodules. Lets add a test to document.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
On Fri, Aug 18, 2017 at 12:04:03PM -0700, Stefan Beller wrote:
> > I just copied the shortcut that they were adding themselfes as submodule
> > in 'setup submodule'. The whole setup of submodules in this test is like
> > this. This way we already had a nested submodule structure which I could
> > just add.
> >
> > I agree that this is unrealistic so I can change that in the test I am
> > adding. But from what I have seen, this shortcut is taken in quite some
> > places when dealing with submodules.
> 
> Please do not make it worse.
> Once upon a time (late '16 IIRC) I had a series floating on the
> list removing all occurrences, but there were issues with the
> series and it did not land.

Took a little while but here is a more clean patch creating individual
submodules for the nesting.

Cheers Heiko

 t/t7001-mv.sh | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
index e365d1ff77..cbc5fb37fe 100755
--- a/t/t7001-mv.sh
+++ b/t/t7001-mv.sh
@@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
 	test_cmp actual expect
 '
 
+test_expect_failure 'moving nested submodules' '
+	git commit -am "cleanup commit" &&
+	mkdir sub_nested_nested &&
+	(cd sub_nested_nested &&
+		touch nested_level2 &&
+		git init &&
+		git add . &&
+		git commit -m "nested level 2"
+	) &&
+	mkdir sub_nested &&
+	(cd sub_nested &&
+		touch nested_level1 &&
+		git init &&
+		git add . &&
+		git commit -m "nested level 1"
+		git submodule add ../sub_nested_nested &&
+		git commit -m "add nested level 2"
+	) &&
+	git submodule add ./sub_nested nested_move &&
+	git commit -m "add nested_move" &&
+	git submodule update --init --recursive &&
+	git mv nested_move sub_nested_moved &&
+	git status
+'
+
 test_done
-- 
2.14.1.145.gb3622a4

Re: [PATCH v2] add test for bug in git-mv for recursive submodules

[Junio C Hamano]

Heiko Voigt <hvoigt@hvoigt.net> writes:

> When using git-mv with a submodule it will detect that and update the
> paths for its configurations (.gitmodules, worktree and gitfile). This
> does not work for recursive submodules where a user renames the root
> submodule.
>
> We discovered this fact when working on on-demand fetch for renamed
> submodules. Lets add a test to document.
>
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
> On Fri, Aug 18, 2017 at 12:04:03PM -0700, Stefan Beller wrote:
>> > I just copied the shortcut that they were adding themselfes as submodule
>> > in 'setup submodule'. The whole setup of submodules in this test is like
>> > this. This way we already had a nested submodule structure which I could
>> > just add.
>> >
>> > I agree that this is unrealistic so I can change that in the test I am
>> > adding. But from what I have seen, this shortcut is taken in quite some
>> > places when dealing with submodules.
>> 
>> Please do not make it worse.
>> Once upon a time (late '16 IIRC) I had a series floating on the
>> list removing all occurrences, but there were issues with the
>> series and it did not land.
>
> Took a little while but here is a more clean patch creating individual
> submodules for the nesting.
>
> Cheers Heiko

Thanks.  Stefan, does this look good to you now?

It is not quite clear which step is expected to fail with the
current code by reading the test or the proposed log message.  Does
"mv" refuse to work and we do not get to run "status", or does
"status" report a failure, or do we fail well before that?

The log message that only says "This does not work when ..." is not
helpful in figuring it out, either.  Something like "This does not
work and fails to update the paths for its configurations" or
whatever that describes "what actually happens" (in contrast to
"what ought to happen", which you described clearly) should be
there.  

Description on how you happened to have discovered the issue feels a
lot less relevant compared to that, and it is totally useless if it
is unclear what the issue is in the first place.

>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
>
> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> index e365d1ff77..cbc5fb37fe 100755
> --- a/t/t7001-mv.sh
> +++ b/t/t7001-mv.sh
> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
>  	test_cmp actual expect
>  '
>  
> +test_expect_failure 'moving nested submodules' '
> +	git commit -am "cleanup commit" &&
> +	mkdir sub_nested_nested &&
> +	(cd sub_nested_nested &&
> +		touch nested_level2 &&
> +		git init &&
> +		git add . &&
> +		git commit -m "nested level 2"
> +	) &&
> +	mkdir sub_nested &&
> +	(cd sub_nested &&
> +		touch nested_level1 &&
> +		git init &&
> +		git add . &&
> +		git commit -m "nested level 1"
> +		git submodule add ../sub_nested_nested &&
> +		git commit -m "add nested level 2"
> +	) &&
> +	git submodule add ./sub_nested nested_move &&
> +	git commit -m "add nested_move" &&
> +	git submodule update --init --recursive &&
> +	git mv nested_move sub_nested_moved &&
> +	git status
> +'
> +
>  test_done

Re: [PATCH v2] add test for bug in git-mv for recursive submodules

[Stefan Beller]

>> Took a little while but here is a more clean patch creating individual
>> submodules for the nesting.
>>
>> Cheers Heiko

Thanks for writing this test!

>
> Thanks.  Stefan, does this look good to you now?

Yes, though there are nits below.

> It is not quite clear which step is expected to fail with the
> current code by reading the test or the proposed log message.  Does
> "mv" refuse to work and we do not get to run "status", or does
> "status" report a failure, or do we fail well before that?

git-mv failing seems like a new possibility without incurring
another process spawn with the new repository object.
(Though then we could also just fix the recursed submodule)

> The log message that only says "This does not work when ..." is not
> helpful in figuring it out, either.  Something like "This does not
> work and fails to update the paths for its configurations" or
> whatever that describes "what actually happens" (in contrast to
> "what ought to happen", which you described clearly) should be
> there.
>
> Description on how you happened to have discovered the issue feels a
> lot less relevant compared to that, and it is totally useless if it
> is unclear what the issue is in the first place.
>
>>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
>>  1 file changed, 25 insertions(+)
>>
>> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
>> index e365d1ff77..cbc5fb37fe 100755
>> --- a/t/t7001-mv.sh
>> +++ b/t/t7001-mv.sh
>> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
>>       test_cmp actual expect
>>  '
>>
>> +test_expect_failure 'moving nested submodules' '
>> +     git commit -am "cleanup commit" &&
>> +     mkdir sub_nested_nested &&
>> +     (cd sub_nested_nested &&

We seem to have different styles for nested shell. I prefer

  outside command &&
  (
      first nested command here &&
      ...

as that aligns indentation to the nesting level. I have seen
the style you use a lot in the  test suite, and we do not have
a guideline in Documentation/CodingGuidelines, so I do not
complain too loudly. ;)


>> +             touch nested_level2 &&
>> +             git init &&
>> +             git add . &&
>> +             git commit -m "nested level 2"
>> +     ) &&
>> +     mkdir sub_nested &&
>> +     (cd sub_nested &&
>> +             touch nested_level1 &&
>> +             git init &&
>> +             git add . &&
>> +             git commit -m "nested level 1"
>> +             git submodule add ../sub_nested_nested &&
>> +             git commit -m "add nested level 2"
>> +     ) &&
>> +     git submodule add ./sub_nested nested_move &&
>> +     git commit -m "add nested_move" &&
>> +     git submodule update --init --recursive &&

So far a nice setup!

>> +     git mv nested_move sub_nested_moved &&

This is the offending command that produces the bug,
as it will break most subsequent commands, such as

>> +     git status

git-status is one of the basic commands. Without
status to function, I think it is hard to recover your repo without
a lot of in-depth knowledge of Git (submodules).

I wonder if git-status should complain more gracefully
and fallback to one of --ignore-submodules={dirty, all},
that actually still works.

Maybe we could introduce a new default mode for this
flag, that is "none-except-on-error", though this sounds
as if we're fixing symptoms instead of the root cause.

Re: [PATCH v2] add test for bug in git-mv for recursive submodules

[Heiko Voigt]

On Mon, Sep 18, 2017 at 01:03:32PM -0700, Stefan Beller wrote:
> >> Took a little while but here is a more clean patch creating individual
> >> submodules for the nesting.
> >>
> >> Cheers Heiko
> 
> Thanks for writing this test!

No worries. :)

> > Thanks.  Stefan, does this look good to you now?
> 
> Yes, though there are nits below.
> 
> > It is not quite clear which step is expected to fail with the
> > current code by reading the test or the proposed log message.  Does
> > "mv" refuse to work and we do not get to run "status", or does
> > "status" report a failure, or do we fail well before that?
> 
> git-mv failing seems like a new possibility without incurring
> another process spawn with the new repository object.
> (Though then we could also just fix the recursed submodule)

It is mv that fails to update everything necessary when using it with
recursively nested submodules. So the git-mv command does not report a
failure here. As an interim fix it could maybe report an error when
encountering nested submodules but the real fix would be to teach it to
recursively spawn the appropriate git-mv commands.

> > The log message that only says "This does not work when ..." is not
> > helpful in figuring it out, either.  Something like "This does not
> > work and fails to update the paths for its configurations" or
> > whatever that describes "what actually happens" (in contrast to
> > "what ought to happen", which you described clearly) should be
> > there.
> >
> > Description on how you happened to have discovered the issue feels a
> > lot less relevant compared to that, and it is totally useless if it
> > is unclear what the issue is in the first place.

Sorry about being a bit brief here. How about dropping that information
how I discovered the bug then and change the commit message to something
like this:

    add test for bug in git-mv for recursive submodules

    When using git-mv with a submodule it will detect that and update
    the paths for its configurations (.gitmodules, worktree and
    gitfile). This does not work in case it encounters nested
    submodules. In that case it only updates the configurations for the
    submodule directly underneath the superproject and fails to update
    the paths for the submodules nested more deeply. This in turn leads
    to the symptom that git status reports that it can not chdir to the
    nested submodule in its old location.

    Lets add a test to document.

?

> >>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
> >>  1 file changed, 25 insertions(+)
> >>
> >> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> >> index e365d1ff77..cbc5fb37fe 100755
> >> --- a/t/t7001-mv.sh
> >> +++ b/t/t7001-mv.sh
> >> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
> >>       test_cmp actual expect
> >>  '
> >>
> >> +test_expect_failure 'moving nested submodules' '
> >> +     git commit -am "cleanup commit" &&
> >> +     mkdir sub_nested_nested &&
> >> +     (cd sub_nested_nested &&
> 
> We seem to have different styles for nested shell. I prefer
> 
>   outside command &&
>   (
>       first nested command here &&
>       ...
> 
> as that aligns indentation to the nesting level. I have seen
> the style you use a lot in the  test suite, and we do not have
> a guideline in Documentation/CodingGuidelines, so I do not
> complain too loudly. ;)

Yeah we have some different styles it seems ;) So here some reasoning
behind my style:

I actually would agree on your style if 'first nested command' was any
arbitrary command but when I use my style it is always when I use a
nested shell for changing into some directory, doing something there and
then being able to return to the previous directory by closing the nested
shell. So for me the 'cd somewhere' belongs to the brackets similarly
like a condition definition belongs to the if it is used with.

> >> +             touch nested_level2 &&
> >> +             git init &&
> >> +             git add . &&
> >> +             git commit -m "nested level 2"
> >> +     ) &&
> >> +     mkdir sub_nested &&
> >> +     (cd sub_nested &&
> >> +             touch nested_level1 &&
> >> +             git init &&
> >> +             git add . &&
> >> +             git commit -m "nested level 1"
> >> +             git submodule add ../sub_nested_nested &&
> >> +             git commit -m "add nested level 2"
> >> +     ) &&
> >> +     git submodule add ./sub_nested nested_move &&
> >> +     git commit -m "add nested_move" &&
> >> +     git submodule update --init --recursive &&
> 
> So far a nice setup!

Thanks.

> >> +     git mv nested_move sub_nested_moved &&
> 
> This is the offending command that produces the bug,
> as it will break most subsequent commands, such as

Yes.

> >> +     git status
> 
> git-status is one of the basic commands. Without
> status to function, I think it is hard to recover your repo without
> a lot of in-depth knowledge of Git (submodules).
> 
> I wonder if git-status should complain more gracefully
> and fallback to one of --ignore-submodules={dirty, all},
> that actually still works.
> 
> Maybe we could introduce a new default mode for this
> flag, that is "none-except-on-error", though this sounds
> as if we're fixing symptoms instead of the root cause.

I think we should rather fix the root cause. For me git-mv is actually
breaking the repository and as described above one possible interim
solution for me would be for 'git-mv' to error out and tell the user
that it does currently not work on recursively nested submodules.

Cheers Heiko