How to checkout a revision that contains a deleted submodule?

How to checkout a revision that contains a deleted submodule?

[Luke Diamand]

Maybe this is a FAQ, but I couldn't figure it out!

I have a repo which has a couple of submodules.

At some point in the past I deleted one of those submodules:

    git rm sub2
    git add -u
    git commit -m 'Deleting sub2'
    git push origin
    ...
    ... more commits and pushes...

Now I go and clone the head revision. This gives me a clone which has
nothing present in .git/modules/sub2.
    login on some other machine
    git clone git@my.repo:thing
    cd thing
    ls .git/modules
    <sub2 not present>

So when I go and checkout an old revision where sub2 is still around I get:
    git checkout oldrevision
    fatal: not a git repository: sub2/../.git/modules/sub2

What am I doing wrong?
What set of commands do I need to use to ensure that this will always
do the right thing?

Thanks
Luke

Re: How to checkout a revision that contains a deleted submodule?

[Luke Diamand]

On Sat, 19 Sep 2020 at 10:03, Luke Diamand <luke@diamand.org> wrote:
>
> Maybe this is a FAQ, but I couldn't figure it out!
>
> I have a repo which has a couple of submodules.
>
> At some point in the past I deleted one of those submodules:
>
>     git rm sub2
>     git add -u
>     git commit -m 'Deleting sub2'
>     git push origin
>     ...
>     ... more commits and pushes...
>
> Now I go and clone the head revision. This gives me a clone which has
> nothing present in .git/modules/sub2.
>     login on some other machine
>     git clone git@my.repo:thing
>     cd thing
>     ls .git/modules
>     <sub2 not present>
>
> So when I go and checkout an old revision where sub2 is still around I get:
>     git checkout oldrevision
>     fatal: not a git repository: sub2/../.git/modules/sub2
>
> What am I doing wrong?
> What set of commands do I need to use to ensure that this will always
> do the right thing?
>
> Thanks
> Luke

Replying to myself, adding Jens who added the section below.

This is a known bug:

https://git-scm.com/docs/git-rm

> BUGS
> ----
> Each time a superproject update removes a populated submodule
> (e.g. when switching between commits before and after the removal) a
> stale submodule checkout will remain in the old location. Removing the
> old directory is only safe when it uses a gitfile, as otherwise the
> history of the submodule will be deleted too. This step will be
> obsolete when recursive submodule update has been implemented.

I'm wondering what "recursive submodule update" is. If I do:

    git submodule update --checkout --force --remote --recursive

then those stale repos are still left lying around. I guess that's a
different kind of recursive?

Thanks
Luke

Re: How to checkout a revision that contains a deleted submodule?

[Kaartic Sivaraam]

On 20/09/20 3:14 pm, Luke Diamand wrote:
> On Sat, 19 Sep 2020 at 10:03, Luke Diamand <luke@diamand.org> wrote:
>>
>> Maybe this is a FAQ, but I couldn't figure it out!
>>
>> I have a repo which has a couple of submodules.
>>
>> At some point in the past I deleted one of those submodules:
>>
>>      git rm sub2
>>      git add -u
>>      git commit -m 'Deleting sub2'
>>      git push origin
>>      ...
>>      ... more commits and pushes...
>>
>> Now I go and clone the head revision. This gives me a clone which has
>> nothing present in .git/modules/sub2.
>>      login on some other machine
>>      git clone git@my.repo:thing
>>      cd thing
>>      ls .git/modules
>>      <sub2 not present>
>>
>> So when I go and checkout an old revision where sub2 is still around I get:
>>      git checkout oldrevision
>>      fatal: not a git repository: sub2/../.git/modules/sub2
>>
>> What am I doing wrong?
>> What set of commands do I need to use to ensure that this will always
>> do the right thing?
>>
>> Thanks
>> Luke
> 
> Replying to myself, adding Jens who added the section below.
> 
> This is a known bug:
> 
> https://git-scm.com/docs/git-rm
> 
>> BUGS
>> ----
>> Each time a superproject update removes a populated submodule
>> (e.g. when switching between commits before and after the removal) a
>> stale submodule checkout will remain in the old location. Removing the
>> old directory is only safe when it uses a gitfile, as otherwise the
>> history of the submodule will be deleted too. This step will be
>> obsolete when recursive submodule update has been implemented.
> 

I don't think that part of the documentation applies to your case. So,
I also don't think this is a known bug. As a matter of fact, I couldn't
reproduce this with the following:


 git init checkout-removed-submodule &&
 cd checkout-removed-submodule/ &&
 echo "Hello, world" >foo &&
 git add foo && git commit -m "Initial commit" &&
 git init ../submodule &&
 cd ../submodule/ &&
 echo "Foo bar" >foobar.txt &&
 git add foobar.txt && git commit -m "Foo bar baz" &&
 cd ../checkout-removed-submodule/ &&
 git submodule add ../submodule/ foobar &&
 git commit -m "Add foobar submodule" &&
 git rm foobar/ &&
 git commit -m "Remove foobar submodule" &&
 git checkout HEAD~ # Checking out the "Add foobar submodule" commit

I get:

 HEAD is now at 25270d8 Add foobar submodule


I also tried with a cloned version of that repository as follows:
 git clone /me/checkout-removed-submodule/ cloned-repo &&
 cd cloned-repo &&
 git co HEAD~

I get:

 HEAD is now at 25270d8 Add foobar submodule

So, I don't get any errors when I checkout a revision where the deleted
submodule is still around. There might other factors in play such as,

- the version of Git being used
- whether `--recurse-submodules` was passed to checkout
- the configuration of the submodule in .gitmodules

It would be great if you could share these and possibly other useful
information to help us identify why you get an error when checking out
the revision.

--
Sivaraam

Re: How to checkout a revision that contains a deleted submodule?

[Luke Diamand]

On Sun, 20 Sep 2020 at 19:02, Kaartic Sivaraam
<kaartic.sivaraam@gmail.com> wrote:
>
> On 20/09/20 3:14 pm, Luke Diamand wrote:
> > On Sat, 19 Sep 2020 at 10:03, Luke Diamand <luke@diamand.org> wrote:
> >>
> >> Maybe this is a FAQ, but I couldn't figure it out!
> >>
> >> I have a repo which has a couple of submodules.
> >>
> >> At some point in the past I deleted one of those submodules:
> >>
> >>      git rm sub2
> >>      git add -u
> >>      git commit -m 'Deleting sub2'
> >>      git push origin
> >>      ...
> >>      ... more commits and pushes...
> >>
> >> Now I go and clone the head revision. This gives me a clone which has
> >> nothing present in .git/modules/sub2.
> >>      login on some other machine
> >>      git clone git@my.repo:thing
> >>      cd thing
> >>      ls .git/modules
> >>      <sub2 not present>
> >>
> >> So when I go and checkout an old revision where sub2 is still around I get:
> >>      git checkout oldrevision
> >>      fatal: not a git repository: sub2/../.git/modules/sub2
> >>
> >> What am I doing wrong?
> >> What set of commands do I need to use to ensure that this will always
> >> do the right thing?
> >>
> >> Thanks
> >> Luke
> >
> > Replying to myself, adding Jens who added the section below.
> >
> > This is a known bug:
> >
> > https://git-scm.com/docs/git-rm
> >
> >> BUGS
> >> ----
> >> Each time a superproject update removes a populated submodule
> >> (e.g. when switching between commits before and after the removal) a
> >> stale submodule checkout will remain in the old location. Removing the
> >> old directory is only safe when it uses a gitfile, as otherwise the
> >> history of the submodule will be deleted too. This step will be
> >> obsolete when recursive submodule update has been implemented.
> >
>
> I don't think that part of the documentation applies to your case. So,
> I also don't think this is a known bug. As a matter of fact, I couldn't
> reproduce this with the following:
>
>
>  git init checkout-removed-submodule &&
>  cd checkout-removed-submodule/ &&
>  echo "Hello, world" >foo &&
>  git add foo && git commit -m "Initial commit" &&
>  git init ../submodule &&
>  cd ../submodule/ &&
>  echo "Foo bar" >foobar.txt &&
>  git add foobar.txt && git commit -m "Foo bar baz" &&
>  cd ../checkout-removed-submodule/ &&
>  git submodule add ../submodule/ foobar &&
>  git commit -m "Add foobar submodule" &&
>  git rm foobar/ &&
>  git commit -m "Remove foobar submodule" &&
>  git checkout HEAD~ # Checking out the "Add foobar submodule" commit
>
> I get:
>
>  HEAD is now at 25270d8 Add foobar submodule
>
>
> I also tried with a cloned version of that repository as follows:
>  git clone /me/checkout-removed-submodule/ cloned-repo &&
>  cd cloned-repo &&
>  git co HEAD~
>
> I get:
>
>  HEAD is now at 25270d8 Add foobar submodule
>
> So, I don't get any errors when I checkout a revision where the deleted
> submodule is still around. There might other factors in play such as,
>
> - the version of Git being used
> - whether `--recurse-submodules` was passed to checkout
> - the configuration of the submodule in .gitmodules
>
> It would be great if you could share these and possibly other useful
> information to help us identify why you get an error when checking out
> the revision.

Thanks for taking the time to answer!

I will see if I can come up with a test case.

Luke

Re: How to checkout a revision that contains a deleted submodule?

[Philippe Blain]

Hi Luke and Kaartic,

> Le 20 sept. 2020 à 14:02, Kaartic Sivaraam <kaartic.sivaraam@gmail.com> a écrit :
> 
> On 20/09/20 3:14 pm, Luke Diamand wrote:
>> On Sat, 19 Sep 2020 at 10:03, Luke Diamand <luke@diamand.org> wrote:
>>> 
>>> Maybe this is a FAQ, but I couldn't figure it out!
>>> 
>>> I have a repo which has a couple of submodules.
>>> 
>>> At some point in the past I deleted one of those submodules:
>>> 
>>>     git rm sub2
>>>     git add -u
>>>     git commit -m 'Deleting sub2'
>>>     git push origin
>>>     ...
>>>     ... more commits and pushes...
>>> 
>>> Now I go and clone the head revision. This gives me a clone which has
>>> nothing present in .git/modules/sub2.
>>>     login on some other machine
>>>     git clone git@my.repo:thing
>>>     cd thing
>>>     ls .git/modules
>>>     <sub2 not present>
>>> 
>>> So when I go and checkout an old revision where sub2 is still around I get:
>>>     git checkout oldrevision
>>>     fatal: not a git repository: sub2/../.git/modules/sub2
>>> 
>>> What am I doing wrong?
>>> What set of commands do I need to use to ensure that this will always
>>> do the right thing?
>>> 
>>> Thanks
>>> Luke
>> 
>> Replying to myself, adding Jens who added the section below.
>> 
>> This is a known bug:
>> 
>> https://git-scm.com/docs/git-rm
>> 
>>> BUGS
>>> ----
>>> Each time a superproject update removes a populated submodule
>>> (e.g. when switching between commits before and after the removal) a
>>> stale submodule checkout will remain in the old location. Removing the
>>> old directory is only safe when it uses a gitfile, as otherwise the
>>> history of the submodule will be deleted too. This step will be
>>> obsolete when recursive submodule update has been implemented.
>> 
> 
> I don't think that part of the documentation applies to your case.

I also don't think this part of the doc applies here. 


> So,
> I also don't think this is a known bug. As a matter of fact, I couldn't
> reproduce this with the following:
> 
> 
> git init checkout-removed-submodule &&
> cd checkout-removed-submodule/ &&
> echo "Hello, world" >foo &&
> git add foo && git commit -m "Initial commit" &&
> git init ../submodule &&
> cd ../submodule/ &&
> echo "Foo bar" >foobar.txt &&
> git add foobar.txt && git commit -m "Foo bar baz" &&
> cd ../checkout-removed-submodule/ &&
> git submodule add ../submodule/ foobar &&
> git commit -m "Add foobar submodule" &&
> git rm foobar/ &&
> git commit -m "Remove foobar submodule" &&
> git checkout HEAD~ # Checking out the "Add foobar submodule" commit

Yes. At this point "foobar" would be empty because '--recurse-submodules' was not used
on 'checkout'. Using `git checkout --recurse-submodules HEAD~` instead would populate it, 
and it would work correctly because the Git repository
of foobar does exist at .git/modules/foobar.

> I also tried with a cloned version of that repository as follows:

here let's make sure we re-checkout 'master' before cloning:
git checkout -

> git clone /me/checkout-removed-submodule/ cloned-repo &&
> cd cloned-repo &&
> git co HEAD~
> 
> I get:
> 
> HEAD is now at 25270d8 Add foobar submodule

I get the same thing, with or without '--recurse-submodules'.

However, if I you have the 'submodule.active' 
configuration set to '.', which is the case if you *cloned* with '--recurse-submodules',
and you then checkout with '--recurse-submodules',
then it fails as Luke describes:

git clone --recurse-submodules  checkout-removed-submodule cloned-repo 
cd cloned-repo &&
git co --recurse-submodules HEAD~
  fatal: not a git repository: ../.git/modules/foobar
  fatal: could not reset submodule index

This bug was reported earlier in May [1], and I suggested a couple ways
the experience could be improved.

I might add here that maybe a good idea would be that 'checkout'
be taught to try to clone the missing submodules if it does not find their 
repository at .git/modules.

Cheers,

Philippe.

[1] https://lore.kernel.org/git/20200501005432.h62dnpkx7feb7rto@glandium.org/T/#u

Re: How to checkout a revision that contains a deleted submodule?

[Luke Diamand]

On Tue, 22 Sep 2020 at 00:14, Philippe Blain
<levraiphilippeblain@gmail.com> wrote:
>
> Hi Luke and Kaartic,
>
> > Le 20 sept. 2020 à 14:02, Kaartic Sivaraam <kaartic.sivaraam@gmail.com> a écrit :
> >
> > On 20/09/20 3:14 pm, Luke Diamand wrote:
> >> On Sat, 19 Sep 2020 at 10:03, Luke Diamand <luke@diamand.org> wrote:
> >>>
> >>> Maybe this is a FAQ, but I couldn't figure it out!
> >>>
> >>> I have a repo which has a couple of submodules.
> >>>
> >>> At some point in the past I deleted one of those submodules:
> >>>
> >>>     git rm sub2
> >>>     git add -u
> >>>     git commit -m 'Deleting sub2'
> >>>     git push origin
> >>>     ...
> >>>     ... more commits and pushes...
> >>>
> >>> Now I go and clone the head revision. This gives me a clone which has
> >>> nothing present in .git/modules/sub2.
> >>>     login on some other machine
> >>>     git clone git@my.repo:thing
> >>>     cd thing
> >>>     ls .git/modules
> >>>     <sub2 not present>
> >>>
> >>> So when I go and checkout an old revision where sub2 is still around I get:
> >>>     git checkout oldrevision
> >>>     fatal: not a git repository: sub2/../.git/modules/sub2
> >>>
> >>> What am I doing wrong?
> >>> What set of commands do I need to use to ensure that this will always
> >>> do the right thing?
> >>>
> >>> Thanks
> >>> Luke
> >>
> >> Replying to myself, adding Jens who added the section below.
> >>
> >> This is a known bug:
> >>
> >> https://git-scm.com/docs/git-rm
> >>
> >>> BUGS
> >>> ----
> >>> Each time a superproject update removes a populated submodule
> >>> (e.g. when switching between commits before and after the removal) a
> >>> stale submodule checkout will remain in the old location. Removing the
> >>> old directory is only safe when it uses a gitfile, as otherwise the
> >>> history of the submodule will be deleted too. This step will be
> >>> obsolete when recursive submodule update has been implemented.
> >>
> >
> > I don't think that part of the documentation applies to your case.
>
> I also don't think this part of the doc applies here.
>
>
> > So,
> > I also don't think this is a known bug. As a matter of fact, I couldn't
> > reproduce this with the following:
> >
> >
> > git init checkout-removed-submodule &&
> > cd checkout-removed-submodule/ &&
> > echo "Hello, world" >foo &&
> > git add foo && git commit -m "Initial commit" &&
> > git init ../submodule &&
> > cd ../submodule/ &&
> > echo "Foo bar" >foobar.txt &&
> > git add foobar.txt && git commit -m "Foo bar baz" &&
> > cd ../checkout-removed-submodule/ &&
> > git submodule add ../submodule/ foobar &&
> > git commit -m "Add foobar submodule" &&
> > git rm foobar/ &&
> > git commit -m "Remove foobar submodule" &&
> > git checkout HEAD~ # Checking out the "Add foobar submodule" commit
>
> Yes. At this point "foobar" would be empty because '--recurse-submodules' was not used
> on 'checkout'. Using `git checkout --recurse-submodules HEAD~` instead would populate it,
> and it would work correctly because the Git repository
> of foobar does exist at .git/modules/foobar.
>
> > I also tried with a cloned version of that repository as follows:
>
> here let's make sure we re-checkout 'master' before cloning:
> git checkout -
>
> > git clone /me/checkout-removed-submodule/ cloned-repo &&
> > cd cloned-repo &&
> > git co HEAD~
> >
> > I get:
> >
> > HEAD is now at 25270d8 Add foobar submodule
>
> I get the same thing, with or without '--recurse-submodules'.
>
> However, if I you have the 'submodule.active'
> configuration set to '.', which is the case if you *cloned* with '--recurse-submodules',
> and you then checkout with '--recurse-submodules',
> then it fails as Luke describes:
>
> git clone --recurse-submodules  checkout-removed-submodule cloned-repo
> cd cloned-repo &&
> git co --recurse-submodules HEAD~
>   fatal: not a git repository: ../.git/modules/foobar
>   fatal: could not reset submodule index
>
> This bug was reported earlier in May [1], and I suggested a couple ways
> the experience could be improved.
>
> I might add here that maybe a good idea would be that 'checkout'
> be taught to try to clone the missing submodules if it does not find their
> repository at .git/modules.

I was thinking this myself. I think the code change might be quite
straightforward.

I added a test case for this problem here:

https://lore.kernel.org/git/20200921081537.15300-2-luke@diamand.org/

I think I could probably have a go at redoing this with an attempt at
a fix to get `git checkout` to clone missing submodules.

Luke

Re: How to checkout a revision that contains a deleted submodule?

[Philippe Blain]

Hi Luke, 

> Le 20 sept. 2020 à 05:44, Luke Diamand <luke@diamand.org> a écrit :
> 
> On Sat, 19 Sep 2020 at 10:03, Luke Diamand <luke@diamand.org> wrote:
>> 
>> Maybe this is a FAQ, but I couldn't figure it out!
>> 
>> I have a repo which has a couple of submodules.
>> 
>> At some point in the past I deleted one of those submodules:
>> 
>>    git rm sub2
>>    git add -u
>>    git commit -m 'Deleting sub2'
>>    git push origin
>>    ...
>>    ... more commits and pushes...
>> 
>> Now I go and clone the head revision. This gives me a clone which has
>> nothing present in .git/modules/sub2.
>>    login on some other machine
>>    git clone git@my.repo:thing
>>    cd thing
>>    ls .git/modules
>>    <sub2 not present>
>> 
>> So when I go and checkout an old revision where sub2 is still around I get:
>>    git checkout oldrevision
>>    fatal: not a git repository: sub2/../.git/modules/sub2
>> 
>> What am I doing wrong?
>> What set of commands do I need to use to ensure that this will always
>> do the right thing?
>> 
>> Thanks
>> Luke
> 
> Replying to myself, adding Jens who added the section below.
> 
> This is a known bug:
> 
> https://git-scm.com/docs/git-rm
> 
>> BUGS
>> ----
>> Each time a superproject update removes a populated submodule
>> (e.g. when switching between commits before and after the removal) a
>> stale submodule checkout will remain in the old location. Removing the
>> old directory is only safe when it uses a gitfile, as otherwise the
>> history of the submodule will be deleted too. This step will be
>> obsolete when recursive submodule update has been implemented.
> 
> I'm wondering what "recursive submodule update" is. If I do:
> 
>    git submodule update --checkout --force --remote --recursive
> 
> then those stale repos are still left lying around. I guess that's a
> different kind of recursive?
> 

I think Jens was referring to this project :
https://github.com/jlehmann/git-submod-enhancements/wiki/Recursive-submodule-checkout

of which some parts were implemented over the years,
a lot of them by Stefan Beller. 

In fact this part of the doc is stale and should be removed since `git checkout` now 
understands `--recurse-submodules` and will
automatically transform "old-style" subdmodules (i.e. with an embeded .git repository)
into "new style" submodules (with a gitfile) (see gitsubmodules(7), at [1], for more info)
when switching to a commit where the submodule is not present.

Cheers,
Philippe.