sudmodule.<name>.recurse ignored

sudmodule.<name>.recurse ignored

[Michael S. Tsirkin]

THE CONFIGURATION OF SUBMODULES
       Submodule operations can be configured using the following mechanisms (from highest to lowest precedence):

       •   The command line for those commands that support taking submodules as part of their pathspecs. Most commands have a boolean
           flag --recurse-submodules which specify whether to recurse into submodules. Examples are grep and checkout. Some commands
           take enums, such as fetch and push, where you can specify how submodules are affected.

       •   The configuration inside the submodule. This includes $GIT_DIR/config in the submodule, but also settings in the tree such
           as a .gitattributes or .gitignore files that specify behavior of commands inside the submodule.

           For example an effect from the submodule’s .gitignore file would be observed when you run git status
           --ignore-submodules=none in the superproject. This collects information from the submodule’s working directory by running
           status in the submodule while paying attention to the .gitignore file of the submodule.

           The submodule’s $GIT_DIR/config file would come into play when running git push --recurse-submodules=check in the
           superproject, as this would check if the submodule has any changes not published to any remote. The remotes are configured
           in the submodule as usual in the $GIT_DIR/config file.

       •   The configuration file $GIT_DIR/config in the superproject. Git only recurses into active submodules (see "ACTIVE
           SUBMODULES" section below).

           If the submodule is not yet initialized, then the configuration inside the submodule does not exist yet, so where to obtain
           the submodule from is configured here for example.

       •   The .gitmodules file inside the superproject. A project usually uses this file to suggest defaults for the upstream
           collection of repositories for the mapping that is required between a submodule’s name and its path.

           This file mainly serves as the mapping between the name and path of submodules in the superproject, such that the
           submodule’s Git directory can be located.

           If the submodule has never been initialized, this is the only place where submodule configuration is found. It serves as
           the last fallback to specify where to obtain the submodule from.


However, when we are talking about the recurse attribute, it is
not taken from .gitmodules - only command line and .git/config seem
to be consulted.
Is this a bug or a feature?


Thanks!

-- 
MST

Re: sudmodule.<name>.recurse ignored

[Ævar Arnfjörð Bjarmason]

On Fri, Oct 07 2022, Michael S. Tsirkin wrote:

> THE CONFIGURATION OF SUBMODULES
>        Submodule operations can be configured using the following mechanisms (from highest to lowest precedence):
>
>        •   The command line for those commands that support taking submodules as part of their pathspecs. Most commands have a boolean
>            flag --recurse-submodules which specify whether to recurse into submodules. Examples are grep and checkout. Some commands
>            take enums, such as fetch and push, where you can specify how submodules are affected.
>
>        •   The configuration inside the submodule. This includes $GIT_DIR/config in the submodule, but also settings in the tree such
>            as a .gitattributes or .gitignore files that specify behavior of commands inside the submodule.
>
>            For example an effect from the submodule’s .gitignore file would be observed when you run git status
>            --ignore-submodules=none in the superproject. This collects information from the submodule’s working directory by running
>            status in the submodule while paying attention to the .gitignore file of the submodule.
>
>            The submodule’s $GIT_DIR/config file would come into play when running git push --recurse-submodules=check in the
>            superproject, as this would check if the submodule has any changes not published to any remote. The remotes are configured
>            in the submodule as usual in the $GIT_DIR/config file.
>
>        •   The configuration file $GIT_DIR/config in the superproject. Git only recurses into active submodules (see "ACTIVE
>            SUBMODULES" section below).
>
>            If the submodule is not yet initialized, then the configuration inside the submodule does not exist yet, so where to obtain
>            the submodule from is configured here for example.
>
>        •   The .gitmodules file inside the superproject. A project usually uses this file to suggest defaults for the upstream
>            collection of repositories for the mapping that is required between a submodule’s name and its path.
>
>            This file mainly serves as the mapping between the name and path of submodules in the superproject, such that the
>            submodule’s Git directory can be located.
>
>            If the submodule has never been initialized, this is the only place where submodule configuration is found. It serves as
>            the last fallback to specify where to obtain the submodule from.
>
>
> However, when we are talking about the recurse attribute, it is
> not taken from .gitmodules - only command line and .git/config seem
> to be consulted.
> Is this a bug or a feature?

It's a (security) feature. We have had disussions[1] about how to safely
read in-repo config, but we are very far away from that becoming a
reality (if it ever happens).

If we read this from the .gitmodules the repo could change how you
interact with your submodules.

1. https://lore.kernel.org/git/YzXwZQbM69eNJfm7@nand.local/

Re: sudmodule.<name>.recurse ignored

[Michael S. Tsirkin]

On Fri, Oct 07, 2022 at 01:09:39PM +0200, Ævar Arnfjörð Bjarmason wrote:
> 
> On Fri, Oct 07 2022, Michael S. Tsirkin wrote:
> 
> > THE CONFIGURATION OF SUBMODULES
> >        Submodule operations can be configured using the following mechanisms (from highest to lowest precedence):
> >
> >        •   The command line for those commands that support taking submodules as part of their pathspecs. Most commands have a boolean
> >            flag --recurse-submodules which specify whether to recurse into submodules. Examples are grep and checkout. Some commands
> >            take enums, such as fetch and push, where you can specify how submodules are affected.
> >
> >        •   The configuration inside the submodule. This includes $GIT_DIR/config in the submodule, but also settings in the tree such
> >            as a .gitattributes or .gitignore files that specify behavior of commands inside the submodule.
> >
> >            For example an effect from the submodule’s .gitignore file would be observed when you run git status
> >            --ignore-submodules=none in the superproject. This collects information from the submodule’s working directory by running
> >            status in the submodule while paying attention to the .gitignore file of the submodule.
> >
> >            The submodule’s $GIT_DIR/config file would come into play when running git push --recurse-submodules=check in the
> >            superproject, as this would check if the submodule has any changes not published to any remote. The remotes are configured
> >            in the submodule as usual in the $GIT_DIR/config file.
> >
> >        •   The configuration file $GIT_DIR/config in the superproject. Git only recurses into active submodules (see "ACTIVE
> >            SUBMODULES" section below).
> >
> >            If the submodule is not yet initialized, then the configuration inside the submodule does not exist yet, so where to obtain
> >            the submodule from is configured here for example.
> >
> >        •   The .gitmodules file inside the superproject. A project usually uses this file to suggest defaults for the upstream
> >            collection of repositories for the mapping that is required between a submodule’s name and its path.
> >
> >            This file mainly serves as the mapping between the name and path of submodules in the superproject, such that the
> >            submodule’s Git directory can be located.
> >
> >            If the submodule has never been initialized, this is the only place where submodule configuration is found. It serves as
> >            the last fallback to specify where to obtain the submodule from.
> >
> >
> > However, when we are talking about the recurse attribute, it is
> > not taken from .gitmodules - only command line and .git/config seem
> > to be consulted.
> > Is this a bug or a feature?
> 
> It's a (security) feature. We have had disussions[1] about how to safely
> read in-repo config, but we are very far away from that becoming a
> reality (if it ever happens).
> 
> If we read this from the .gitmodules the repo could change how you
> interact with your submodules.
> 
> 1. https://lore.kernel.org/git/YzXwZQbM69eNJfm7@nand.local/

Hmm. Well we have a bunch of attributes there. Why this one? What did I miss?

-- 
MST

Re: sudmodule.<name>.recurse ignored

[Glen Choo]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> On Fri, Oct 07, 2022 at 01:09:39PM +0200, Ævar Arnfjörð Bjarmason wrote:
>> 
>> On Fri, Oct 07 2022, Michael S. Tsirkin wrote:
>> 
>> > THE CONFIGURATION OF SUBMODULES
>> >        Submodule operations can be configured using the following mechanisms (from highest to lowest precedence):
>> >
>> >        •   The command line for those commands that support taking submodules as part of their pathspecs. Most commands have a boolean
>> >            flag --recurse-submodules which specify whether to recurse into submodules. Examples are grep and checkout. Some commands
>> >            take enums, such as fetch and push, where you can specify how submodules are affected.
>> >
>> >        •   The configuration inside the submodule. This includes $GIT_DIR/config in the submodule, but also settings in the tree such
>> >            as a .gitattributes or .gitignore files that specify behavior of commands inside the submodule.
>> >
>> >            For example an effect from the submodule’s .gitignore file would be observed when you run git status
>> >            --ignore-submodules=none in the superproject. This collects information from the submodule’s working directory by running
>> >            status in the submodule while paying attention to the .gitignore file of the submodule.
>> >
>> >            The submodule’s $GIT_DIR/config file would come into play when running git push --recurse-submodules=check in the
>> >            superproject, as this would check if the submodule has any changes not published to any remote. The remotes are configured
>> >            in the submodule as usual in the $GIT_DIR/config file.
>> >
>> >        •   The configuration file $GIT_DIR/config in the superproject. Git only recurses into active submodules (see "ACTIVE
>> >            SUBMODULES" section below).
>> >
>> >            If the submodule is not yet initialized, then the configuration inside the submodule does not exist yet, so where to obtain
>> >            the submodule from is configured here for example.
>> >
>> >        •   The .gitmodules file inside the superproject. A project usually uses this file to suggest defaults for the upstream
>> >            collection of repositories for the mapping that is required between a submodule’s name and its path.
>> >
>> >            This file mainly serves as the mapping between the name and path of submodules in the superproject, such that the
>> >            submodule’s Git directory can be located.
>> >
>> >            If the submodule has never been initialized, this is the only place where submodule configuration is found. It serves as
>> >            the last fallback to specify where to obtain the submodule from.
>> >
>> >
>> > However, when we are talking about the recurse attribute, it is
>> > not taken from .gitmodules - only command line and .git/config seem
>> > to be consulted.
>> > Is this a bug or a feature?
>> 
>> It's a (security) feature. We have had disussions[1] about how to safely
>> read in-repo config, but we are very far away from that becoming a
>> reality (if it ever happens).
>> 
>> If we read this from the .gitmodules the repo could change how you
>> interact with your submodules.
>> 
>> 1. https://lore.kernel.org/git/YzXwZQbM69eNJfm7@nand.local/
>
> Hmm. Well we have a bunch of attributes there. Why this one? What did I miss?

I couldn't find the `submodule.<name>.recurse` you referenced in your
email subject, so I'll assume we're talking about `submodule.recurse`
here. A simple reason would be that `submodule.recurse` is not specific
to a single submodule so it does not come from .gitmodules. .gitmodules
is a way of providing metadata for that submodules need to work
correctly, e.g.:

- where to get the submodule from during a fetch/clone, e.g.
  `submodule.<name>.url`, `submodule.<name>.branch`
- information needed for submodules to work correctly, e.g.
  `submodule.<name>.path`
- default config that gets populated in .git/config via `git submodule
  init`, e.g. `submodule.<name>.update`,
  `submodule.<name>.fetchRecurseSubmodules`

With respect to config, the general stance is that .gitmodules is meant
to provide defaults for .git/config and _not_ to be read directly (see
Ævar's answer). There are some very limited cases where config is read
directly from .gitmodules, but that approach is very frowned upon and
it's unlikely that we'll add more of them in the future.

>
> -- 
> MST

Re: sudmodule.<name>.recurse ignored

[Junio C Hamano]

Glen Choo <chooglen@google.com> writes:

> - where to get the submodule from during a fetch/clone, e.g.
>   `submodule.<name>.url`, `submodule.<name>.branch`
> - information needed for submodules to work correctly, e.g.
>   `submodule.<name>.path`
> - default config that gets populated in .git/config via `git submodule
>   init`, e.g. `submodule.<name>.update`,
>   `submodule.<name>.fetchRecurseSubmodules`
>
> With respect to config, the general stance is that .gitmodules is meant
> to provide defaults for .git/config and _not_ to be read directly (see
> Ævar's answer).

I think that is a sensible way to think about this.

It is more like "suggestion made by the upstream", to be taken after
examining it, just like you should type "make" after examining the
Makefile.  At least enough to trust its contents ;-)

> There are some very limited cases where config is read
> directly from .gitmodules, but that approach is very frowned upon and
> it's unlikely that we'll add more of them in the future.

Yup, we should be careful and see if we can get rid of existing
uses.

Thanks.

Re: sudmodule.<name>.recurse ignored

[Michael S. Tsirkin]

On Fri, Oct 07, 2022 at 02:13:12PM -0700, Junio C Hamano wrote:
> Glen Choo <chooglen@google.com> writes:
> 
> > - where to get the submodule from during a fetch/clone, e.g.
> >   `submodule.<name>.url`, `submodule.<name>.branch`
> > - information needed for submodules to work correctly, e.g.
> >   `submodule.<name>.path`
> > - default config that gets populated in .git/config via `git submodule
> >   init`, e.g. `submodule.<name>.update`,
> >   `submodule.<name>.fetchRecurseSubmodules`
> >
> > With respect to config, the general stance is that .gitmodules is meant
> > to provide defaults for .git/config and _not_ to be read directly (see
> > Ævar's answer).
> 
> I think that is a sensible way to think about this.
> 
> It is more like "suggestion made by the upstream", to be taken after
> examining it, just like you should type "make" after examining the
> Makefile.  At least enough to trust its contents ;-)
> 
> > There are some very limited cases where config is read
> > directly from .gitmodules, but that approach is very frowned upon and
> > it's unlikely that we'll add more of them in the future.
> 
> Yup, we should be careful and see if we can get rid of existing
> uses.
> 
> Thanks.


Fundamentally the problem we encounter regularly is this:
	qemu is superproject, ui/keycodemapdb is subproject.


	I have a change on master setting the submodule commit:

	qemu$ git show master | grep +Sub
	+Subproject commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352

	and check it out:

	qemu$ git submodule update --init ui/keycodemapdb
	Submodule 'ui/keycodemapdb' (https://gitlab.com/qemu-project/keycodemapdb.git) registered for path 'ui/keycodemapdb'
	Submodule path 'ui/keycodemapdb': checked out '7381b9bfadd31c4c9e9a10b5bb5032f9189d4352'


	In another branch I have a different commit:

	qemu$ git show sub-foo  | grep +Sub
	+Subproject commit 57ba70da5312170883a3d622cd2aa3fd0e2ec7ae


	Now I switch branches and nothing happens, the submodule
	is marked as dirty:

	qemu$ git checkout sub-foo
	M       ui/keycodemapdb
	Switched to branch 'sub-foo'

	qemu$ (cd ui/keycodemapdb && git show | head -1)
	commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352


it is now very easy to generate incorrect code:
- git commit -a  will commit the change to submodule
- any build will use a mix of super and subproject that
  is completely untested


As a result people are basically saying avoid using submodules
but I am wondering whether git can be tweaked to do the
right thing by default.


I understand we can work around it by asking everyone to
create a correct config, but can't we make it DTRT by default
to reduce friction?


-- 
MST

Re: sudmodule.<name>.recurse ignored

[Glen Choo]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> Fundamentally the problem we encounter regularly is this:
> 	qemu is superproject, ui/keycodemapdb is subproject.
>
>
> 	I have a change on master setting the submodule commit:
>
> 	qemu$ git show master | grep +Sub
> 	+Subproject commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352
>
> 	and check it out:
>
> 	qemu$ git submodule update --init ui/keycodemapdb
> 	Submodule 'ui/keycodemapdb' (https://gitlab.com/qemu-project/keycodemapdb.git) registered for path 'ui/keycodemapdb'
> 	Submodule path 'ui/keycodemapdb': checked out '7381b9bfadd31c4c9e9a10b5bb5032f9189d4352'
>
>
> 	In another branch I have a different commit:
>
> 	qemu$ git show sub-foo  | grep +Sub
> 	+Subproject commit 57ba70da5312170883a3d622cd2aa3fd0e2ec7ae
>
>
> 	Now I switch branches and nothing happens, the submodule
> 	is marked as dirty:
>
> 	qemu$ git checkout sub-foo
> 	M       ui/keycodemapdb
> 	Switched to branch 'sub-foo'
>
> 	qemu$ (cd ui/keycodemapdb && git show | head -1)
> 	commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352
>
>
> it is now very easy to generate incorrect code:
> - git commit -a  will commit the change to submodule
> - any build will use a mix of super and subproject that
>   is completely untested
>
>
> As a result people are basically saying avoid using submodules
> but I am wondering whether git can be tweaked to do the
> right thing by default.
>

It sounds like you want submodule.recurse [1] :) With that enabled, your
"git checkout" should behave like "git checkout --recurse-submodules",
which should make ui/keycodemapdb check out the correct commit as long
as the submodule commit is present locally. If is is _not_ present
locally, you will have to run "git submodule update".

Unfortunately, you typically won't know whether the commit is present
before running the command. This is yet another of those things that are
painful with submodules.

[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-submodulerecurse

> I understand we can work around it by asking everyone to
> create a correct config, but can't we make it DTRT by default
> to reduce friction?

You might be interested in the proposed 'new' Submodule UX [2]; one of
the goals is to make manual submodule management via "git submodule"
unnecessary.

As a part of that, you should be able to set "submodule.recurse = true"
and have high confidence that all necessary submodules and submodule
commits are present. Work on that is still ongoing, but this situation
should have improved as of [3].

[2] https://lore.kernel.org/git/YHofmWcIAidkvJiD@google.com/
[3] https://lore.kernel.org/git/20220308001433.94995-1-chooglen@google.com/

>
>
> -- 
> MST

Re: sudmodule.<name>.recurse ignored

[Michael S. Tsirkin]

On Wed, Oct 12, 2022 at 09:56:45AM -0700, Glen Choo wrote:
> "Michael S. Tsirkin" <mst@redhat.com> writes:
> 
> > Fundamentally the problem we encounter regularly is this:
> > 	qemu is superproject, ui/keycodemapdb is subproject.
> >
> >
> > 	I have a change on master setting the submodule commit:
> >
> > 	qemu$ git show master | grep +Sub
> > 	+Subproject commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352
> >
> > 	and check it out:
> >
> > 	qemu$ git submodule update --init ui/keycodemapdb
> > 	Submodule 'ui/keycodemapdb' (https://gitlab.com/qemu-project/keycodemapdb.git) registered for path 'ui/keycodemapdb'
> > 	Submodule path 'ui/keycodemapdb': checked out '7381b9bfadd31c4c9e9a10b5bb5032f9189d4352'
> >
> >
> > 	In another branch I have a different commit:
> >
> > 	qemu$ git show sub-foo  | grep +Sub
> > 	+Subproject commit 57ba70da5312170883a3d622cd2aa3fd0e2ec7ae
> >
> >
> > 	Now I switch branches and nothing happens, the submodule
> > 	is marked as dirty:
> >
> > 	qemu$ git checkout sub-foo
> > 	M       ui/keycodemapdb
> > 	Switched to branch 'sub-foo'
> >
> > 	qemu$ (cd ui/keycodemapdb && git show | head -1)
> > 	commit 7381b9bfadd31c4c9e9a10b5bb5032f9189d4352
> >
> >
> > it is now very easy to generate incorrect code:
> > - git commit -a  will commit the change to submodule
> > - any build will use a mix of super and subproject that
> >   is completely untested
> >
> >
> > As a result people are basically saying avoid using submodules
> > but I am wondering whether git can be tweaked to do the
> > right thing by default.
> >
> 
> It sounds like you want submodule.recurse [1] :)

yes. unfortuntely there is no way to set it by default
for cloned repos. users have to remember to set it.

> With that enabled, your
> "git checkout" should behave like "git checkout --recurse-submodules",
> which should make ui/keycodemapdb check out the correct commit as long
> as the submodule commit is present locally. If is is _not_ present
> locally, you will have to run "git submodule update".
> 
> Unfortunately, you typically won't know whether the commit is present
> before running the command. This is yet another of those things that are
> painful with submodules.
> 
> [1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-submodulerecurse
> 
> > I understand we can work around it by asking everyone to
> > create a correct config, but can't we make it DTRT by default
> > to reduce friction?
> 
> You might be interested in the proposed 'new' Submodule UX [2]; one of
> the goals is to make manual submodule management via "git submodule"
> unnecessary.
> 
> As a part of that, you should be able to set "submodule.recurse = true"
> and have high confidence that all necessary submodules and submodule
> commits are present.

Hmm. How about only doing this for active submodules? Possible?

> Work on that is still ongoing, but this situation
> should have improved as of [3].
> 
> [2] https://lore.kernel.org/git/YHofmWcIAidkvJiD@google.com/
> [3] https://lore.kernel.org/git/20220308001433.94995-1-chooglen@google.com/
> 
> >
> >
> > -- 
> > MST