Re: --progress option for git submodule update?

[Vitali Lovich <vlovich@gmail.com>]

> On Sep 9, 2015, at 4:40 PM, Stefan Beller <sbeller@google.com> wrote:
> 
> So submodules...
> 
> I am currently working on improving submodules (some basic performance
> improvements have been done, soon to be merged upstream, I currently
> try to get parallelism working for git fetch --recurse-submodules and for
> git submodule update eventually. I sent some early working patches for that,
> but I am doing a whole new redesign without threads now).
That sounds exciting.  Can’t wait.

> On Wed, Sep 9, 2015 at 3:52 PM, Vitali Lovich <vlovich@gmail.com> wrote:
>> Hi,
>> 
>> Git submodule doesn’t have a --progress option like regular clone/fetch does.  This means that it can hang a long time without output as it’s transferring data, particularly for large repositories.
> 
> For repositories with nested submodules it is impossible to estimate
> the progress because you don't know how many there are.
> Say you have a layout like:
> 
> A
> -> B
> -> C
> -> D
>    -> E
>    -> F
> 
> whereas each letter is a repository and B,C,D are submodules of A and
> E,F are submodules of D.
> So if D is not cloned yet, it looks like A has only 3 submodules, but
> in fact we need to update 5
> submodules.
I don’t think I’m asking for an overall --progress.  As you point out that is very difficult/an intractable problem.  I was thinking it would just report the progress for each submodule it encounters as it fetches/clones.
The added benefit to that is that if there’s a lot of submodules, an overall progress might get stuck at a long time at a given percentage whereas it’s less likely cloning just a single module would depending on the size of repositories.

>> This is problematic in automation scenarios where there can be upper-bounds on how long a process may run without any output (to protect against processes hanging for long periods of time without forward progress).
> 
> Maybe a better error-out-if-hanging would be better IMHO ?
> Another option would be to enumerate the submodules and give the
> currently estimated upper bound ?
That’s much more difficult & I’m still left with the original problem where I have to set a very conservative upper-bound which wastes valuable machine time & causes extra contention for automation resources.

> Doh! I see what you're missing now after rereading the email closely.
> You can add a --quiet option,
> but --verbose or --progress just errors out, but you want that as a
> possible argument for git clone
> inside the git submodule update code.
Yes exactly.

> 
> Thanks,
> Stefan
> 
>> 
>> I’m sure this has been asked for before but having this option would be really nice for automation system (like buildbot) to take advantage of.  The only alternative is a hacky solution to clone locally first with the —progress option
>> & then somehow set up the submodule to use the local clone as a reference.
>> 
>> Thanks,
>> Vitali--
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html