Re: [PATCHv2] builtin/clone: support submodule groups

[Stefan Beller <sbeller@google.com>]

+ cc Duy, Michael, who discussed the sparse checkout recently

On Wed, Nov 25, 2015 at 9:00 PM, Trevor Saunders <tbsaunde@tbsaunde.org> wrote:
> Seeing the recent sparse checkout discussion I realized it might be
> useful to have a similar sort of feature for sparse checkouts.  So say I
> had a mobile and desktop client in the same repo and wanted to be able to
> checkout the commoncode and one client without having to explicitly list
> all the paths I care about. It seems like UI wise you might want to use
> --group there too, or at least explaining the difference to users might
> be interesting, but maybe that's worrying way too much abouta possible
> future feature.

For reference what I was proposing (coverletter [RFC PATCH 0/5]
Submodule Groups):
-->8--
This is also available at
https://github.com/stefanbeller/git/tree/submodule-groups
It applies on top of the submodule-parallel-patch series I sent a few
minutes ago.

Consider having a real large software project in Git with each component
in a submodule (such as an operating system, Android, Debian, Fedora,
no toy OS such as https://github.com/gittup/gittup as that doesn't quite
demonstrate the scale of the problem).

If you have lots of submodules, you probably don't need all of them at once,
but you have functional units. Some submodules are absolutely required,
some are optional and only for very specific purposes.

This patch series adds meaning to a "groups" field in the .gitmodules file.

So you could have a .gitmodules file such as:

[submodule "gcc"]
        path = gcc
        url = git://...
        groups = default,devel
[submodule "linux"]
        path = linux
        url = git://...
        groups = default
[submodule "nethack"]
        path = nethack
        url = git://...
        groups = optional,games

and by this series you can work on an arbitrary subgroup of these
submodules such
using these commands:

    git clone --group default --group devel git://...
    # will clone the superproject and recursively
    # checkout any submodule being in at least one of the groups.

    git submodule add --group default --group devel git://... ..
    # will add a submodule, adding 2 submodule
    # groups to its entry in .gitmodule

    # as support for clone we want to have:
    git config submodule.groups default
    git submodule init --groups
    # will init all submodules from the default group

    # as support for clone we want to have:
    git config submodule.groups default
    git submodule update --groups
    # will update all submodules from the default group

Any feedback welcome, specially on the design level!
(Do we want to have it stored in the .gitmodules file? Do we want to have
the groups configured in .git/config as "submodule.groups", any other way
to make it future proof and extend the groups syntax?)
-->8--

I think the biggest advantage with the groups is to have it not depending on the
path. Consider your one repository containing both mobile and desktop code,
where you have a sparse checkout for mobile.

Now what happens if files are renamed, i.e. the leading directory?
As this change comes in from your dear coworker, who has no idea how
your .git/info/sparse-checkout looks like, it may happen that more files appear
in your worktree as your patterns did not cover the renamed case.
Or some file contents go missing as they are in one of the ignored paths.

This groups feature would solve that as the groups are not dependent on
the paths or the data itself. However the groups in this proposal are only
meant to be applied on a submodule level, not in a repository itself.

How would you do that?
I could imagine a file like .gitgroups (It should be part of the repository,
such that everybody talks about the same groups), with a similar syntax like
.gitattributes where some file patterns are assigned one or more groups.
And in either .git/config (as it is for the submodules here) or in a file
.git/info/groups (as the sparse checkouts do it in that directory)
you'd configure the groups you are interested in.

It would be cool to have the same mechanism for both sparse-group-checkout
and submodules, so I'd propose to use a config option in .git/config,
such as checkout-group which covers both submodules as well as the
pattern groups as specified by .gitgroups.

---
Beware, this is just a first shot spinning around some ideas.

Thanks,
Stefan