git@vger.kernel.org mailing list mirror (one of many)
 help / Atom feed
From: Stefan Beller <sbeller@google.com>
To: Martin Ågren <martin.agren@gmail.com>
Cc: git <git@vger.kernel.org>, Heiko Voigt <hvoigt@hvoigt.net>
Subject: Re: [PATCH 10/10] fetch: retry fetching submodules if sha1 were not fetched
Date: Thu, 9 Aug 2018 10:42:34 -0700
Message-ID: <CAGZ79kbkLDhiqYrd6epPVP7dLLR7AFApyas=_mgeCucngJONMg@mail.gmail.com> (raw)
In-Reply-To: <CAN0heSpsbYWzujzyteWuhto9DTXzvAkP+vt++d7ar3ob6Zx=Gg@mail.gmail.com>

On Thu, Aug 9, 2018 at 12:50 AM Martin Ågren <martin.agren@gmail.com> wrote:
>
> On 9 August 2018 at 00:17, Stefan Beller <sbeller@google.com> wrote:
> > Currently when git-fetch is asked to recurse into submodules, it dispatches
> > a plain "git-fetch -C <submodule-dir>" (and some submodule related options
> > such as prefix and recusing strategy, but) without any information of the
> > remote or the tip that should be fetched.
> >
> > This works surprisingly well in some workflows (such as using submodules
> > as a third party library), while not so well in other scenarios, such
> > as in a Gerrit topic-based workflow, that can tie together changes
> > (potentially across repositories) on the server side. One of the parts
> > of such a Gerrit workflow is to download a change when wanting to examine
> > it, and you'd want to have its submodule changes that are in the same
> > topic downloaded as well. However these submodule changes reside in their
> > own repository in their on ref (refs/changes/<int>).
>
> s/on/own/
>
> > Retry fetching a submodule if the object id that the superproject points
> > to, cannot be found.
> >
> > Note: This is an RFC and doesn't support fetching to FETCH_HEAD yet, but
> > only into a local branch. To make fetching into FETCH_HEAD work, we need
> > some refactoring in builtin/fetch.c to adjust the calls to
> > 'check_for_new_submodule_commits'.
> >
> > Signed-off-by: Stefan Beller <sbeller@google.com>
> > ---
>
> > diff --git a/submodule.c b/submodule.c
> > index ec7ea6f8c2d..6cbd0b1a470 100644
> > --- a/submodule.c
> > +++ b/submodule.c
> > @@ -1127,6 +1127,7 @@ struct submodule_parallel_fetch {
> >         int result;
> >
> >         struct string_list changed_submodule_names;
> > +       struct string_list retry;
> >  };
> >  #define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP }
>
> `retry` will effectively be `STRING_LIST_INIT_NODUP`, but making that
> explicit would be better and the next addition to the struct would be
> easier to get right.
>
> > +retry_next:
> > +       retry_it = string_list_pop(&spf->retry);
> > +       if (retry_it) {
> > +               struct strbuf submodule_prefix = STRBUF_INIT;
> > +               const struct submodule *sub =
> > +                               submodule_from_name(spf->r,
> > +                                                   &null_oid,
> > +                                                   retry_it->string);
> > +
> > +               child_process_init(cp);
> > +               cp->dir = get_submodule_git_dir(spf->r, sub->path);
> > +               if (!cp->dir)
> > +                       goto retry_next;
>
> So here you just drop the string list item. Since it's NODUP, and since
> the `util` pointers are owned elsewhere(?), this seems fine. Other uses
> of `string_list_pop()` might not be so straightforward.
>
> Just a thought, but rather than pop+if+goto, maybe
>
> while ((retry_it = )) {
>         ...
>         if (!cp->dir) continue;
>         ...
>         return 1;
> }

I really want to keep the retry list short and pruned, as this
function is called O(n) times with n the number of submodules
and the retry list will also be up to n.
And with that we'd run O(n^2) times into "if (!..) continue;".

When we use the 'pop-no-work items' logic, then we're still in O(n).

> I haven't commented on any of the submodule stuff, which is probably
> where you'd be most interested in comments. I don't use submodules, nor
> do I know the code that runs them.. I guess my comments are more "if
> those who know something about submodules find this series worthwhile,
> you might want to consider my comments as well".

Thanks for your comments! I'll try to think of another way to
represent this more easily in code.

Thanks,
Stefan

      reply index

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-08-08 22:17 [RFC PATCH 00/10] fetch: make sure submodule oids are fetched Stefan Beller
2018-08-08 22:17 ` [PATCH 01/10] string_list: print_string_list to use trace_printf Stefan Beller
2018-08-09 21:16   ` Junio C Hamano
2018-08-09 21:40     ` Stefan Beller
2018-08-08 22:17 ` [PATCH 02/10] string-list.h: add string_list_pop function Stefan Beller
2018-08-09  7:35   ` Martin Ågren
2018-08-09 21:29     ` Junio C Hamano
2018-08-09 21:41       ` Jeff King
2018-08-09 21:52         ` Stefan Beller
2018-08-09 21:56           ` Jeff King
2018-08-09 22:10             ` Stefan Beller
2018-08-08 22:17 ` [PATCH 03/10] sha1-array: provide oid_array_remove_if Stefan Beller
2018-08-09  7:39   ` Martin Ågren
2018-08-09 17:25     ` Stefan Beller
2018-08-09 19:24       ` Jeff King
2018-08-09 21:46         ` Junio C Hamano
2018-08-09 21:44   ` Junio C Hamano
2018-08-08 22:17 ` [PATCH 04/10] submodule.c: convert submodule_move_head new argument to object id Stefan Beller
2018-08-09 22:00   ` Junio C Hamano
2018-08-08 22:17 ` [PATCH 05/10] submodule.c: fix indentation Stefan Beller
2018-08-08 22:17 ` [PATCH 06/10] submodule.c: sort changed_submodule_names before searching it Stefan Beller
2018-08-08 22:17 ` [PATCH 07/10] submodule: move global changed_submodule_names into fetch submodule struct Stefan Beller
2018-08-08 22:17 ` [PATCH 08/10] submodule.c: do not copy around submodule list Stefan Beller
2018-08-08 22:17 ` [PATCH 09/10] submodule: fetch in submodules git directory instead of in worktree Stefan Beller
2018-08-08 22:17 ` [PATCH 10/10] fetch: retry fetching submodules if sha1 were not fetched Stefan Beller
2018-08-09  7:50   ` Martin Ågren
2018-08-09 17:42     ` Stefan Beller [this message]

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAGZ79kbkLDhiqYrd6epPVP7dLLR7AFApyas=_mgeCucngJONMg@mail.gmail.com' \
    --to=sbeller@google.com \
    --cc=git@vger.kernel.org \
    --cc=hvoigt@hvoigt.net \
    --cc=martin.agren@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org mailing list mirror (one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox