Re: [PATCH] FYI / RFC: submodules: introduce repo-like workflow

[Stefan Beller <sbeller@google.com>]

On Fri, Sep 28, 2018 at 12:26 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Thu, Sep 27 2018, Stefan Beller wrote:
>
> > Internally we have rolled out this as an experiment for
> > "submodules replacing the repo tool[1]". The repo tool is described as:
> >
> >     Repo unifies Git repositories when necessary, performs uploads to the
> >     Gerrit revision control system, and automates parts of the Android
> >     development workflow. Repo is not meant to replace Git, only to make
> >     it easier to work with Git in the context of Android. The repo command
> >     is an executable Python script that you can put anywhere in your path.
> >
> >     In working with the Android source files, you use Repo for
> >     across-network operations. For example, with a single Repo command you
> >     can download files from multiple repositories into your local working
> >     directory.
> >
> >     In most situations, you can use Git instead of Repo, or mix Repo and
> >     Git commands to form complex commands.
> >
> > [1] https://source.android.com/setup/develop/
>
> Some questions just out of curiosity, not for this patch in particular:
>
> Those docs seem to describe the situation without this patch, with this
> patch is the repo tool fully replaced?

Yeah I started by describing the status quo. Some points to add:

* The repo UX looks very similar to perforce, as that was used internally
   at Google at the time a lot. Git wasn't around for long in the days of
   early Android. Now that Git is well known, new grads joining are more likely
   to know Git than perforce.
* repo has no tests because it started as a small script
* maintenance/releases of repo are not ideal.

So yes, one of our long term goals is to replace repo with Git-only
(on the client side) workflow.

> How are you planning to migrate from repo to this on a repository-data
> basis

We don't plan to migrate the existing clients. You'd have to re-clone.

>, does gerrit also populate .gitmodules files appropriately, which
> your clone --recurse-submodules will pick up, but repo will just ignore,
> so you can use the two in parallel?

Gerrit has native support to update submodules in a superproject.
So if you submit code to a project that is a submodule of a superproject
the superproject updates its gitlink. (and if you use a topic based workflow
to submit to two projects, it will show up as an atomic update in the
superproject by having just one commit updating two gitlinks)

Also Gerrit has a plugin "supermanifest" [1], which tracks repo
manifests and mirrors changes from the manifest into the
superproject, e.g. adding a new project to the manifest will
add a new submodule to the superproject.

[1] https://gerrit.googlesource.com/plugins/supermanifest/

With both Gerrits internal superproject subscription and that plugin
it should be possible to use repo or git-submodule in parallel (on an
organisational level, i.e. you choose one of them, and your coworker
chooses the other one)

> Now "repo init -u" takes a URL to a manifest of repositories to stitch
> together, I've understood from past conversations (but am not sure) that
> this is used e.g. by downstream Android vendors so they can use what
> Google's using + whatever they have in-house, i.e. make the manifest the
> set of open source repos plus some (e.g. drivers specific to their
> device). How is that sort of workflow going to work where you
> (presumably) have do that via .gitmodules + commit entries in trees?

The manifest is tracked in its own manifest repo, so today they fork
that project, with the superproject you'd fork the superproject and modify
the .gitmodules file as needed.

> They run their own Gerrit install with some magic to sync back & forth?
>
> I assume that now the recursive "checkout" relies on all the origin/HEAD
> symbolic refs pointing to "master", but how is this going to deal with
> incorporating a repo whose main branch has a different name?

Change the name? Or pin it via sha1.

> E.g. "trunk" or "blead"? Perhaps some interaction with
> checkout.defaultRemote + submodule.<name>.branch=<NAME> could make "git
> checkout :mainbranch" DWYM.
>
> > * Fetching changes ("repo sync") needs to fetch all repositories, as there
> >   is no central place that tracks what has changed. In a superproject
> >   git fetch can determine which submodules need fetching.  In Androids
> >   case the daily change is only in a few repositories (think 10s), so
> >   migrating to a superproject would save an order of magnitude in fetch
> >   traffic for daily updates of developers.
>
> Interesting that in all this time with the reliance on a central server
> repo wasn't already asking some custom API "what repos changed since xyz
> time" to narrow that down, but hey, .gitmodules + commit entries in
> trees will do it for you.

It's complicated. Shawn really grew to dislike the repo tool and wanted
to have it gone as quickly as possible, so no hacks that extend its life. ;-)
I would prefer to keep that stance as any hack in there would grow into
a nightmare down the road.

> > * Sometimes when the dependencies are not on a clear repository boundary
> >   one would like to have git-bisect available across the different
> >   repositories, which repo cannot provide due to its design.
>
> I assume that you're not upgrading independently to e.g. every single
> linux commit, just stable releases, so does bisecting deal with knowing
> that e.g. a breakage occurred when linux.git was updated from v4.10 to
> v4.12, and then to go within the repo itself and bisect from there, or
> is that done manually?

We currently have no submodule support in bisect, but that is open to
future contribution. But with relations between submodules via the
superproject it is at least possible to bisect precisely, whereas in repo
the projects are unrelated to each other and it becomes a political
issue if A broke B or B broke A.

Thanks for taking time to read and think through the concept here,

Stefan