Add an option to automatically submodule update on checkout

Add an option to automatically submodule update on checkout

[Ciro Santilli]

This is a must if you are working with submodules, otherwise every git
checkout requires a git submodule update, and you forget it, and
things break, and you understand, and you go to stack overflow
questions http://stackoverflow.com/questions/22328053/why-doesnt-git-checkout-automatically-do-git-submodule-update-recursive
| http://stackoverflow.com/questions/4611512/is-there-a-way-to-make-git-pull-automatically-update-submodules
and you give up and create aliases :-)

RE: Add an option to automatically submodule update on checkout

[Randall S. Becker]

On May 6, 2017 4:38 AM Ciro Santilli wrote:
> This is a must if you are working with submodules, otherwise every git checkout requires a git submodule update,
> and you forget it, and things break, and you understand, and you go to stack overflow questions
> http://stackoverflow.com/questions/22328053/why-doesnt-git-checkout-automatically-do-git-submodule-update-recursive
> http://stackoverflow.com/questions/4611512/is-there-a-way-to-make-git-pull-automatically-update-submodules
> and you give up and create aliases :-)

I rather like the concept of supporting --recurse-submodules. The complexity is that the branches in all submodules all have to have compatible semantics when doing the checkout, which is by no means guaranteed. In the scenario where you are including a submodule from a third-party (very common - see gnulib), the branches likely won't be there, so you have a high probability of having the command fail or produce the same results as currently exists if you allow the checkout even with problems (another option?). If you have control of everything, then this makes sense.

Cheers,
Randall

Re: Add an option to automatically submodule update on checkout

[Stefan Beller]

On Mon, May 8, 2017 at 7:42 AM, Randall S. Becker
<rsbecker@nexbridge.com> wrote:
> On May 6, 2017 4:38 AM Ciro Santilli wrote:
>> This is a must if you are working with submodules, otherwise every git checkout requires a git submodule update,
>> and you forget it, and things break, and you understand, and you go to stack overflow questions
>> http://stackoverflow.com/questions/22328053/why-doesnt-git-checkout-automatically-do-git-submodule-update-recursive
>> http://stackoverflow.com/questions/4611512/is-there-a-way-to-make-git-pull-automatically-update-submodules
>> and you give up and create aliases :-)

The upcoming release (2.13) will have "git checkout --recurse-submodules",
which will checkout the submodules at the commit as recorded in the
superproject.

I plan to add an option "submodule.recurse" (name is subject to bikeshedding),
which would make the --recurse-submodules flag given by default for all commands
that support the flag. (Currently cooking we have reset
--recurse-submodules, already
existing there is push/pull).

> I rather like the concept of supporting --recurse-submodules. The complexity is that the branches in all submodules all have to have compatible semantics when doing the checkout, which is by no means guaranteed. In the scenario where you are including a submodule from a third-party (very common - see gnulib), the branches likely won't be there, so you have a high probability of having the command fail or produce the same results as currently exists if you allow the checkout even with problems (another option?). If you have control of everything, then this makes sense.

I am trying to give the use case of having control over everything
(or rather mixed) more thought as well,
e.g. "checkout --recurse-submodules -b <name>" may want to
create the branches in a subset of submodules as well.

Thanks,
Stefan

>
> Cheers,
> Randall
>

RE: Add an option to automatically submodule update on checkout

[Randall S. Becker]

On May 8, 2017 12:25 PM, Stefan Beller wrote:
>On Mon, May 8, 2017 at 7:42 AM, Randall S. Becker <rsbecker@nexbridge.com> wrote:
>> On May 6, 2017 4:38 AM Ciro Santilli wrote:
>>> This is a must if you are working with submodules, otherwise every 
>>> git checkout requires a git submodule update, and you forget it, and 
>>> things break, and you understand, and you go to stack overflow 
>>> questions 
>>> http://stackoverflow.com/questions/22328053/why-doesnt-git-checkout-a
>>> utomatically-do-git-submodule-update-recursive
>>> http://stackoverflow.com/questions/4611512/is-there-a-way-to-make-git
>>> -pull-automatically-update-submodules
>>> and you give up and create aliases :-)

> The upcoming release (2.13) will have "git checkout --recurse-submodules", which will checkout the submodules
> at the commit as recorded in the superproject.
> I plan to add an option "submodule.recurse" (name is subject to bikeshedding), which would make the --recurse-submodules
> flag given by default for all commands that support the flag. (Currently cooking we have reset --recurse-submodules, already
> existing there is push/pull).

Brilliant! 😊

>> I rather like the concept of supporting --recurse-submodules. The complexity is that the branches in all submodules all have to have compatible >>semantics when doing the checkout, which is by no means guaranteed. In the scenario where you are including a submodule from a third-party (very >>common - see gnulib), the branches likely won't be there, so you have a high probability of having the command fail or produce the same results as >>currently exists if you allow the checkout even with problems (another option?). If you have control of everything, then this makes sense.

>I am trying to give the use case of having control over everything (or rather mixed) more thought as well, e.g. "checkout --recurse-submodules -b ><name>" may want to create the branches in a subset of submodules as well.

I have to admit that I just assumed it would have to work that way this would not be particularly useful. However, in thinking about it, we might want to limit the depth of how far -b <name> takes effect. If the super module brings in submodules entirely within control of the development group, having -b <name> apply down to leaf submodules makes sense (in some policies). However, if some submodules span out to, say, gnulib, that might not make particular sense. Some downward limit might be appropriate. Perhaps, in the submodule ref, you might want to qualify it as <commit>:<ref> (but the impact of that is probably and admittedly pretty horrid). I hesitate to suggest a numeric limit, as that assumes that submodules are organized in a balanced tree - which is axiomatically unreasonable. Maybe something in .git/config, like

[branch "topic*"]
	submodules=a,b,c

But I suspect that would make things even more confusing.

Cheers,
Randall

Re: Add an option to automatically submodule update on checkout

[Brandon Williams]

On 05/08, Stefan Beller wrote:
> On Mon, May 8, 2017 at 7:42 AM, Randall S. Becker
> <rsbecker@nexbridge.com> wrote:
> > On May 6, 2017 4:38 AM Ciro Santilli wrote:
> >> This is a must if you are working with submodules, otherwise every
> >> git checkout requires a git submodule update, and you forget it,
> >> and things break, and you understand, and you go to stack overflow
> >> questions
> >> http://stackoverflow.com/questions/22328053/why-doesnt-git-checkout-automatically-do-git-submodule-update-recursive
> >> http://stackoverflow.com/questions/4611512/is-there-a-way-to-make-git-pull-automatically-update-submodules
> >> and you give up and create aliases :-)
> 
> The upcoming release (2.13) will have "git checkout
> --recurse-submodules", which will checkout the submodules at the
> commit as recorded in the superproject.
> 
> I plan to add an option "submodule.recurse" (name is subject to
> bikeshedding), which would make the --recurse-submodules flag given by
> default for all commands that support the flag. (Currently cooking we
> have reset --recurse-submodules, already existing there is push/pull).

Well pull not so much...it'll do a recursive fetch but not a recursive
merge/rebase.  That is something on the docket to get done in the next
couple months though.

> 
> > I rather like the concept of supporting --recurse-submodules. The
> > complexity is that the branches in all submodules all have to have
> > compatible semantics when doing the checkout, which is by no means
> > guaranteed. In the scenario where you are including a submodule from
> > a third-party (very common - see gnulib), the branches likely won't
> > be there, so you have a high probability of having the command fail
> > or produce the same results as currently exists if you allow the
> > checkout even with problems (another option?). If you have control
> > of everything, then this makes sense.
> 
> I am trying to give the use case of having control over everything (or
> rather mixed) more thought as well, e.g. "checkout
> --recurse-submodules -b <name>" may want to create the branches in a
> subset of submodules as well.
> 
> Thanks, Stefan
> 
> >
> > Cheers, Randall
> >

-- 
Brandon Williams

Re: Add an option to automatically submodule update on checkout

[Stefan Beller]

On Mon, May 8, 2017 at 9:46 AM, Randall S. Becker
<rsbecker@nexbridge.com> wrote:
> On May 8, 2017 12:25 PM, Stefan Beller wrote:
>>On Mon, May 8, 2017 at 7:42 AM, Randall S. Becker <rsbecker@nexbridge.com> wrote:
>>> On May 6, 2017 4:38 AM Ciro Santilli wrote:
>>>> This is a must if you are working with submodules, otherwise every
>>>> git checkout requires a git submodule update, and you forget it, and
>>>> things break, and you understand, and you go to stack overflow
>>>> questions
>>>> http://stackoverflow.com/questions/22328053/why-doesnt-git-checkout-a
>>>> utomatically-do-git-submodule-update-recursive
>>>> http://stackoverflow.com/questions/4611512/is-there-a-way-to-make-git
>>>> -pull-automatically-update-submodules
>>>> and you give up and create aliases :-)
>
>> The upcoming release (2.13) will have "git checkout --recurse-submodules", which will checkout the submodules
>> at the commit as recorded in the superproject.
>> I plan to add an option "submodule.recurse" (name is subject to bikeshedding), which would make the --recurse-submodules
>> flag given by default for all commands that support the flag. (Currently cooking we have reset --recurse-submodules, already
>> existing there is push/pull).
>
> Brilliant! 😊
>
>>> I rather like the concept of supporting --recurse-submodules. The complexity is that the branches in all submodules all have to have compatible >>semantics when doing the checkout, which is by no means guaranteed. In the scenario where you are including a submodule from a third-party (very >>common - see gnulib), the branches likely won't be there, so you have a high probability of having the command fail or produce the same results as >>currently exists if you allow the checkout even with problems (another option?). If you have control of everything, then this makes sense.
>
>>I am trying to give the use case of having control over everything (or rather mixed) more thought as well, e.g. "checkout --recurse-submodules -b ><name>" may want to create the branches in a subset of submodules as well.
>
> I have to admit that I just assumed it would have to work that way this would not be particularly useful. However, in thinking about it, we might want to limit the depth of how far -b <name> takes effect. If the super module brings in submodules entirely within control of the development group, having -b <name> apply down to leaf submodules makes sense (in some policies). However, if some submodules span out to, say, gnulib, that might not make particular sense. Some downward limit might be appropriate. Perhaps, in the submodule ref, you might want to qualify it as <commit>:<ref> (but the impact of that is probably and admittedly pretty horrid). I hesitate to suggest a numeric limit, as that assumes that submodules are organized in a balanced tree - which is axiomatically unreasonable. Maybe something in .git/config, like
>
> [branch "topic*"]
>         submodules=a,b,c
>
> But I suspect that would make things even more confusing.

I thought about having yet-another-flag in the .gitmodules file, which
states if the submodule is extern or internal.

[submodule "gnulib"]
    path=./gnulib
    external = true # implies no branch for checkout -b --recurse-submodules

I think there are a couple more situations where such "external" submodules
are treated differently, so maybe we'd want to think carefully about
the actual name
as different workflows would want to have different features for an
internal/external
submodule.

Thanks,
Stefan

RE: Add an option to automatically submodule update on checkout

[Randall S. Becker]

On May 8, 2017 12:55 PM, Stefan Beller wrote:
>On Mon, May 8, 2017 at 9:46 AM, Randall S. Becker <rsbecker@nexbridge.com> wrote:
>> On May 8, 2017 12:25 PM, Stefan Beller wrote:
>>>On Mon, May 8, 2017 at 7:42 AM, Randall S. Becker <rsbecker@nexbridge.com> wrote:
>>>> On May 6, 2017 4:38 AM Ciro Santilli wrote:
>>>>> This is a must if you are working with submodules, otherwise every 
>>>>> git checkout requires a git submodule update, and you forget it, 
>>>>> and things break, and you understand, and you go to stack overflow 
>>>>> questions 
>>>>> http://stackoverflow.com/questions/22328053/why-doesnt-git-checkout
>>>>> -a utomatically-do-git-submodule-update-recursive
>>>>> http://stackoverflow.com/questions/4611512/is-there-a-way-to-make-g
>>>>> it -pull-automatically-update-submodules
>>>>> and you give up and create aliases :-)
>>
>>> The upcoming release (2.13) will have "git checkout 
>>> --recurse-submodules", which will checkout the submodules at the commit as recorded in the superproject.
>>> I plan to add an option "submodule.recurse" (name is subject to 
>>> bikeshedding), which would make the --recurse-submodules flag given 
>>> by default for all commands that support the flag. (Currently cooking we have reset --recurse-submodules, already existing there is push/pull).
>>
>> Brilliant! 😊
>>
>>>> I rather like the concept of supporting --recurse-submodules. The complexity is that the branches in all submodules all have to have compatible >>>semantics when doing the checkout, which is by no means guaranteed. In the scenario where you are including a submodule from a third-party (very >>>common - see gnulib), the branches likely won't be there, so you have a high probability of having the command fail or produce the same results as >>>currently exists if you allow the checkout even with problems (another option?). If you have control of everything, then this makes sense.
>>
>>>I am trying to give the use case of having control over everything (or rather mixed) more thought as well, e.g. "checkout --recurse-submodules -b >><name>" may want to create the branches in a subset of submodules as well.
>>
>> I have to admit that I just assumed it would have to work that way 
>> this would not be particularly useful. However, in thinking about it, 
>> we might want to limit the depth of how far -b <name> takes effect. If 
>> the super module brings in submodules entirely within control of the 
>> development group, having -b <name> apply down to leaf submodules 
>> makes sense (in some policies). However, if some submodules span out 
>> to, say, gnulib, that might not make particular sense. Some downward 
>> limit might be appropriate. Perhaps, in the submodule ref, you might 
>> want to qualify it as <commit>:<ref> (but the impact of that is 
>> probably and admittedly pretty horrid). I hesitate to suggest a 
>> numeric limit, as that assumes that submodules are organized in a 
>> balanced tree - which is axiomatically unreasonable. Maybe something 
>> in .git/config, like
>>
>> [branch "topic*"]
>>         submodules=a,b,c
>>
>> But I suspect that would make things even more confusing.

>I thought about having yet-another-flag in the .gitmodules file, which states if the submodule is extern or internal.

>[submodule "gnulib"]
>    path=./gnulib
>    external = true # implies no branch for checkout -b --recurse-submodules

>I think there are a couple more situations where such "external" submodules are treated differently, so maybe we'd want to think carefully about the >actual name as different workflows would want to have different features for an internal/external submodule.

I didn't want to open up that one, but yes. That makes sense. However, I don't like overloading what "external" means or might mean in the future. Would you consider a distinct Boolean for that, like inherit-branch=true?

Cheers,
Randall

Re: Add an option to automatically submodule update on checkout

[Brandon Williams]

On 05/08, Randall S. Becker wrote:
> On May 8, 2017 12:55 PM, Stefan Beller wrote:
> >On Mon, May 8, 2017 at 9:46 AM, Randall S. Becker <rsbecker@nexbridge.com> wrote:
> >> On May 8, 2017 12:25 PM, Stefan Beller wrote:
> >>>On Mon, May 8, 2017 at 7:42 AM, Randall S. Becker <rsbecker@nexbridge.com> wrote:
> >>>> On May 6, 2017 4:38 AM Ciro Santilli wrote:
> >>>>> This is a must if you are working with submodules, otherwise every 
> >>>>> git checkout requires a git submodule update, and you forget it, 
> >>>>> and things break, and you understand, and you go to stack overflow 
> >>>>> questions 
> >>>>> http://stackoverflow.com/questions/22328053/why-doesnt-git-checkout
> >>>>> -a utomatically-do-git-submodule-update-recursive
> >>>>> http://stackoverflow.com/questions/4611512/is-there-a-way-to-make-g
> >>>>> it -pull-automatically-update-submodules
> >>>>> and you give up and create aliases :-)
> >>
> >>> The upcoming release (2.13) will have "git checkout 
> >>> --recurse-submodules", which will checkout the submodules at the commit as recorded in the superproject.
> >>> I plan to add an option "submodule.recurse" (name is subject to 
> >>> bikeshedding), which would make the --recurse-submodules flag given 
> >>> by default for all commands that support the flag. (Currently cooking we have reset --recurse-submodules, already existing there is push/pull).
> >>
> >> Brilliant! 😊
> >>
> >>>> I rather like the concept of supporting --recurse-submodules. The complexity is that the branches in all submodules all have to have compatible >>>semantics when doing the checkout, which is by no means guaranteed. In the scenario where you are including a submodule from a third-party (very >>>common - see gnulib), the branches likely won't be there, so you have a high probability of having the command fail or produce the same results as >>>currently exists if you allow the checkout even with problems (another option?). If you have control of everything, then this makes sense.
> >>
> >>>I am trying to give the use case of having control over everything (or rather mixed) more thought as well, e.g. "checkout --recurse-submodules -b >><name>" may want to create the branches in a subset of submodules as well.
> >>
> >> I have to admit that I just assumed it would have to work that way 
> >> this would not be particularly useful. However, in thinking about it, 
> >> we might want to limit the depth of how far -b <name> takes effect. If 
> >> the super module brings in submodules entirely within control of the 
> >> development group, having -b <name> apply down to leaf submodules 
> >> makes sense (in some policies). However, if some submodules span out 
> >> to, say, gnulib, that might not make particular sense. Some downward 
> >> limit might be appropriate. Perhaps, in the submodule ref, you might 
> >> want to qualify it as <commit>:<ref> (but the impact of that is 
> >> probably and admittedly pretty horrid). I hesitate to suggest a 
> >> numeric limit, as that assumes that submodules are organized in a 
> >> balanced tree - which is axiomatically unreasonable. Maybe something 
> >> in .git/config, like
> >>
> >> [branch "topic*"]
> >>         submodules=a,b,c
> >>
> >> But I suspect that would make things even more confusing.
> 
> >I thought about having yet-another-flag in the .gitmodules file, which states if the submodule is extern or internal.
> 
> >[submodule "gnulib"]
> >    path=./gnulib
> >    external = true # implies no branch for checkout -b --recurse-submodules
> 
> >I think there are a couple more situations where such "external" submodules are treated differently, so maybe we'd want to think carefully about the >actual name as different workflows would want to have different features for an internal/external submodule.
> 
> I didn't want to open up that one, but yes. That makes sense. However, I don't like overloading what "external" means or might mean in the future. Would you consider a distinct Boolean for that, like inherit-branch=true?

Something like that kind of already exists.  The 'branch' field.
Internal repos would most likely use the '.' value to indicate that the
submodules should track the superproject's branch.  While a value of say
'foo' would indicate that the submodule should always be on branch
'foo'; this could be used for external repositories.

> 
> Cheers,
> Randall
> 

-- 
Brandon Williams

Re: Add an option to automatically submodule update on checkout

[Stefan Beller]

On Mon, May 8, 2017 at 10:08 AM, Brandon Williams <bmwill@google.com> wrote:
>>
>> >[submodule "gnulib"]
>> >    path=./gnulib
>> >    external = true # implies no branch for checkout -b --recurse-submodules
>>
>> >I think there are a couple more situations where such "external" submodules are treated differently, so maybe we'd want to think carefully about the >actual name as different workflows would want to have different features for an internal/external submodule.
>>
>> I didn't want to open up that one, but yes. That makes sense. However, I don't like overloading what "external" means or might mean in the future. Would you consider a distinct Boolean for that, like inherit-branch=true?
>
> Something like that kind of already exists.  The 'branch' field.
> Internal repos would most likely use the '.' value to indicate that the
> submodules should track the superproject's branch.  While a value of say
> 'foo' would indicate that the submodule should always be on branch
> 'foo'; this could be used for external repositories.

so for external repos you'd keep the branch unset, such that
you strictly checkout the sha1 object into a detached HEAD.

Makes sense.

Thanks,
Stefan

Re: Add an option to automatically submodule update on checkout

[Junio C Hamano]

"Randall S. Becker" <rsbecker@nexbridge.com> writes:

> I have to admit that I just assumed it would have to work that way
> this would not be particularly useful. However, in thinking about
> it, we might want to limit the depth of how far -b <name> takes
> effect. If the super module brings in submodules entirely within
> control of the development group, having -b <name> apply down to
> leaf submodules makes sense (in some policies). However, if some
> submodules span out to, say, gnulib, that might not make
> particular sense.

I do not see a strong reason to avoid your own branches in "other
people's project" like this.

The submodule's upstream may be a project you have no control over,
but the repository you have locally is under your total control and
you can use any branch names to suit the need of your project as the
whole (i.e. the superproject and submodules bound to it).

The fact that local branch names are under your control and for your
own use is true even when you are not using submodules, by the way.

RE: Add an option to automatically submodule update on checkout

[Randall S. Becker]

On May 8, 2017 10:58 PM, Junio C Hamano wrote:
>"Randall S. Becker" <rsbecker@nexbridge.com> writes:
>> I have to admit that I just assumed it would have to work that way 
>> this would not be particularly useful. However, in thinking about it, 
>> we might want to limit the depth of how far -b <name> takes effect. If 
>> the super module brings in submodules entirely within control of the 
>> development group, having -b <name> apply down to leaf submodules 
>> makes sense (in some policies). However, if some submodules span out 
>> to, say, gnulib, that might not make particular sense.

>I do not see a strong reason to avoid your own branches in "other people's
project" like this.
>The submodule's upstream may be a project you have no control over, but the
repository you have locally is under your total control and you can use >any
branch names to suit the need of your project as the whole (i.e. the
superproject and submodules bound to it).

>The fact that local branch names are under your control and for your own
use is true even when you are not using submodules, by the way.

I agree with the technical aspects of this, but doing a checkout -b into
something like gnulib will pin the code you are using in that submodule to
whatever commit was referenced when you did the checkout. Example: In a
situation like that, I would want gnulib to stay on 'master'. It is my
opinion, FWIW, that this is a matter of policy or standards within the
organization using git that we should not be imposing one way or another. In
the current state of affairs (2.12.x), when I checkout, I make sure that
people are aware of which branch each submodule is on because git won't go
into the submodules - I'm fine with imposing that as a policy at present
because it takes positive action by the developers and I keep the master
branch on my own repositories locked down and it's obvious when they are
accidentally on it. But we're talking changing this so that checkout
branches can apply recursively. This changes the policy requirements so that
people have to further act to undo what git will do by default on recursion.
The policy will be at a high level the same (i.e., always make sure you know
what branch you are on in submodules), but the implementation of it will
need to be different (i.e., after you checkout recursive, go into each
submodule and undo what git just did by checking out the default branch on
some submodules ___ ___ ___, which depends on which super repository they
are using, is onerous for me to manage, and for my developers to remember to
do).

With Respect,
Randall