Re: [PATCHv2 0/2] Submodule shallow recommendation [WAS: Submodules: have a depth field in the .gitmodules file]

[Stefan Beller <sbeller@google.com>]

On Thu, May 26, 2016 at 12:16 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> On Thu, May 26, 2016 at 11:13 AM, Junio C Hamano <gitster@pobox.com> wrote:
>>> Stefan Beller <sbeller@google.com> writes:
>>>
>>>> Sometimes the history of a submodule is not considered important by
>>>> the projects upstream. To make it easier for downstream users, allow
>>>> a field 'submodule.<name>.depth' in .gitmodules, which can be used
>>>> to indicate the recommended depth.
>>>
>>> I have a design-level question.
>>>
>>> If your project consists of 600 submodules, among which 40 of them
>>> you would recommend making a shallow clone, are there traits, other
>>> than "most people would not care about its history", that are shared
>>> across these 40 subprojects?
>>
>> From my understanding these 40 subprojects are a large file storage
>> done different than Git LFS. In the repo world this was choosen to be
>> a separate repository, such that you had versioning available as the
>> large files change a few times (like a precompiled kernel for special
>> hardware, etc). And this is one of the missing pieces to translate the
>> current repo-tool workflow to submodules.
>
> I am not questioning the value of being able to say "this and that
> submodule need only a shallow copy".  I am trying to see
> "individually mark each and every such submodules" should be the
> primary interface to do so.
>
>> So you are proposing another layer of indirection, i.e. instead of giving
>> a pathspec with ":(attr:label-framework)" we would want to give a profile
>> which then has the pathspec plus additional information on shallowness
>> an such.
>
> I do not understand what you mean by "instead of giving a pathspec"
> at all.
>
> When you have 40 submodules, with your design the user has to
> sprinkle 40 lines of
>
>         shallow = true
>
> for each of [submodule "$n"] sections.  If there are other traits
> (see my first question) that are similar, they will have some other
> setting next to the "shallow = true" 40 times.  When a new submodule
> is added to that same class, they will again have to add these two
> lines.

I could not find out about another class or attribute matching like that.
Sure there are other attributes (Labels), but they don't match closely
these 40 submodules, but are arbitrary.

>
> I was wondering if a level of indirection that lets you say
> "submodules in this group all share 'shallow=true' (by default)"
> would improve that cumbersomeness.  When you add another similar,
> trait, you add that just once, not 40 times.  When you add another
> submodule, you say "this submodule is also in that group", without
> mentioning "shallow".
>
> We probably _need_ shallow=true at individual module level, if only
> to override the default given by such an indirection scheme.  So
> don't take the message you are responding to as "no, your design is
> not good, scrap it, and here is a better one".  It is more like "It
> would be a good first step, but have you considered where it will
> eventually lead us to?

Even if we decide we need it later, this doesn't seem to be a blocker
or contradicting such an enhancement.

> Would the more complete future look like
> this, and how well would this first step fit in that world?  Would
> it be a good escape hatch, or would it have to be deprecated?"

Looking at the Android open source project, we have 45 repos,
that are marked with clone-depth="1" , 37/45 are in a path starting
with "./prebuilts/", i.e. competing with Git LFS.
The remaining 8/45 seem to be device specific kernels.
37 of them (a different set than the prebuilts though) carry one label.

So with that data, I would expect that you would not need to mark a
submodule often as shallow, but it's quite obvious at creation time.
And specially at creation time it is easier to edit the module specific section
as opposed to adding the module in a profile or other indirect section.

I note, that we can extend to an indirection concept later if needed, though.

>
>> And you reinvented submodule groups. ;)
>> IIRC we had a discussion if we want to have the submodule groups
>> stored at each submodule or at a central "profile/group" setting.
>
> As I said, it was not my intention to get into that; I am not
> interested in the exact syntax, and I am not interested whether the
> pointer goes from group to individual modules (i.e. [group "bar"]
> says "foo" is one of its member modules), or individual modules have
> pointers to groups (i.e. "module [foo]" declares its membership in
> group "bar") at all.  I really do not care.
>
> What matters in my suggestion was, after you established that group
> "bar" exists, you can do:
>
>         [profile "framework"]
>                 shallow = "some notation that refers to group bar"
>
> so that you do not have to repeat "shallow = true" many times per
> submodule.

Ok, I just wonder how hard that  "some notation that refers to group bar"
is to get right. (Both on the design level for us as well as for users to
understand how to express themselves and then doing it right).

And I would not try to outsmart the user here? The repetition of
40 times "shallow" may be cumbersome, but only for creation,
not for maintaining!

Having more lines in the .gitmodules file is not a bad thing per se, though?
If you have these 600 submodules, you would search for the one you're
currently interested in anyway, so it doesn't matter if the average
number of lines
per modules is 3 or 5.

>
> By the way, I do not see the "profile" as about "submodules" or
> "submodule groups".  It is more about "Who am I?  What am I working
> on?  Give me an appropriate set of settings for the 600 submodules
> you have, with the knowledge that I am a framework person".

And I thought branches with different .gitmodule/.gitattribute files would
solve this.

Compare to the kernel development: a network person will use the
network maintainers branch to start developing; They will not get
Linus copy and set some magic to indicate they're a network person?

>
> grouping submodules would merely be one mechanism to help specify
> that.
>

Thanks for raising concerns, it actually made me think of more issues.

Thanks,
Stefan