Re: [RFD/PATCH] submodule doc: describe where we can configure them

[Jonathan Nieder <jrnieder@gmail.com>]

Stefan Beller wrote:

> This is similar to the gitignore document, but doesn't mirror
> the current situation. It is rather meant to start a discussion for
> the right approach for mirroring repositories with submodules.

Ooh.

[...]
> --- a/Documentation/git-submodule.txt
> +++ b/Documentation/git-submodule.txt
> @@ -59,6 +59,22 @@ instead of treating the other project as a submodule. Directories
>  that come from both projects can be cloned and checked out as a whole
>  if you choose to go that route.
>  
> +Submodule operations can be configured using the following mechanisms
> +(from highest to lowest precedence):
> +
> + * the command line for those commands that support taking submodule specs.
> +
> + * the configuration file `$GIT_DIR/config`.
> +
> + * the configuration file `config` found in the `refs/submodule/config` branch.
> +   This can be used to overwrite the upstream configuration in the `.gitmodules`
> +   file without changing the history of the project.
> +   Useful options here are overwriting the base, where relative URLs apply to,
> +   when mirroring only parts of the larger collection of submodules.
> +
> + * the `.gitmodules` file inside the repository. A project usually includes this
> +   file to suggest defaults for the upstream collection of repositories.

(This documentation probably belongs in gitmodules(5) --- then,
git-submodule(1) could focus on command-line usage and point there for
configuration information.)

There are two aspects of this to be separated: what governs the behavior
of commands running locally, and where we get information about
submodules from a remote repository.

Local commands
--------------
The original submodule design was that local commands rely on
information from .git/config, and that information gets copied there
from .gitmodules when a submodule is initialized.  That way, a local
user can specify their preferred mirror or other options using some
straightforward 'git config' commands.

As a side effect, the settings in .git/config tell git which submodules
to pay attention to (which submodules were initialized).

When .gitmodules changes, the settings in .git/config are left alone,
since the end user *might* have manually set something up and we don't
want to trample on it.

This design is somewhat problematic for a few reasons:

- When I want to stop paying attention to a particular submodule and
  start paying attention to it again later, all my local settings are
  gone.

- When upstream adds a new submodule, I have to do the same manual
  work to change the options for that new submodule.

- When upstream changes submodule options (perhaps to fix a URL
  typo), I don't get those updates.

A fix is to use settings from .git/config when present and fall back
to .gitmodules when not.  I believe the submodule code has been slowly
moving in that direction for new features.  Perhaps we can do so for
existing features (like submodule.*.url) too.

An alternative would have been to introduce a .git/info/submodules
file that overrides settings from .gitmodules, analagous to
.git/info/excludes overriding .gitignore and .git/info/attributes
overriding .gitattributes.  We are already using .git/config for
this so that doesn't seem necessary.

Remote repositories
-------------------
The .gitmodules file has some odd properties as a place to put
configuration:

- it is versioned.  There is no way to change URLs in an old version
  of .gitmodules retroactively when a URL has changed.

- it is controlled by whoever writes history.  There is no way for me
  to change the URLs in my mirror of https://gerrit.googlesource.com/gerrit
  to match my mirror's different filesystem layout without producing
  my own history that diverges from the commits I am mirroring.

When the URLs in .gitmodules are relative URLs, this means that if
I mirror a superproject, I have to mirror all its submodules, too,
with the same layout.  It's not so easy for me to publish my copy
of the parent project and the one subproject I made changes in --- I
have to mirror everything.  In particular, this means I can't mirror
https://gerrit.googlesource.com/gerrit to github.

When the URLs in .gitmodules are absolute URLs, this means that if
I mirror a superproject, I cannot ask people consuming my mirror to
use my mirrors of child projects, too.  I cannot publish my copy of
the parent project and the one subproject I made changes in and
expect people to be able to "git clone --recurse-submodules" the
result successfully.

It is as though refs were stored in a .gitrefs file, with all the
attendant disadvantages, instead of being a separate component of
the repository that a particular repository owner can manipulate
without changing history.

To fix this, we could allow additional .gitmodules settings to be put
in another ref (perhaps something like "refs/repository/config" to allow
sharing additional repository-specific configuration in other files
within the same tree --- e.g., branch descriptions).  The semantics:

* If there is a gitmodules file in refs/repository/config in the
  repository I clone, then the submodule settings from it are stored
  locally somewhere that overrides .gitmodules.  Perhaps
  .git/info/<remotename>/gitmodules?

* Later fetches from the remote would also update this gitmodules
  file.

* Settings from this gitmodules file can be overridden locally
  using 'git config' until an explicit "git submodule sync" to
  override the local configuration.

What do you think?

If two different remotes provide conflicting values for a setting
in their gitmodules files, git would error out and ask the user
to intervene with a tie-breaking "git config" setting.

Thanks,
Jonathan