Re: warning: no common commits - slow pull

[Junio C Hamano <gitster@pobox.com>]

Len Brown <lenb@kernel.org> writes:

>> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
>>  * [new tag]         v2.6.25-rc2 -> v2.6.25-rc2
>> Updating 4ee29f6..101142c
>> Fast forward
>>  Makefile                               |    4 +-
> ...
>> [lenb@d975xbx2 linus (master)]$            
>> [lenb@d975xbx2 linus (master)]$ git --version
>> git version 1.5.4.1.122.gaa8d
>
> It still happens with latest git. (linus has declared -rc3 this time)

That's because nobody has touched anything in this area since your
last report.

But I now have a theory.

Next time this happens, could you run ls-remote for all four git
servers and compare them?

    $ LINUS=kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
    $ for i in 1 2 3 4; do git ls-remote git://git$i.$LINUS >git$i.txt; done

If my theory is correct, a workaround would be not to fetch from
git://git.kernel.org/ but from git://git$i.kernel.org/ for some value
of $i.

Now to the theory part.

Suppose the ancestry is like this:

         o (rc2)             o (rc3)
        /                   /
    ---A---o---o---o---o---B---o---o---o---C

And the event sequence is:

 (0) You are in sync with rc2 (have commits up to "A" and tag
     rc2);

 (1) Linus builds up to "B", tags rc3, pushes out to the master;

 (2) git1 and git2 mirror that history;

 (3) Linus builds up to "C", pushes out to the master;

 (4) git1 gets "C" but git2 mirror lags behind;

 (5) You fetch, which happens to first go to git1; you get "C"
     and learn that rc3 points at "B" that is locally reachable,
     but you do not have rc3 tag itself yet.  So git-fetch
     auto-follows (i.e. asks for "rc3" tag);

 (6) However, that goes over a separate connection, and you
     happen to hit git2, which does have "B" and rc3, but it
     does not have "C" yet.

 (7) You tell the other end "Hey, I have C", which git2 does not
     understand, because it hasn't seen the commit through the
     mirroring system yet.

Now, even if we are in the above situation, things _ought_ to work
(and I strongly suspect that the scripted git-fetch did work
correctly).  git2 would say "You have "C"?  I dunno that one, but keep
talking", and you will keep sending "I have this, this, this,...",
walking the ancestry chain down from C.  When you say "I also have B",
it would say "Aha, Ok, I heard enough to tell that rc3 tag is the only
thing I need to send to you".

However, if the current version of git-fetch has a bug in the
negotiation code when it auto-follows tags, that could throw this
conversation to compute what's common way off, and that is what I
suspect is happening.

For example, I notice that the list of old refs is reused (kept in the
transport structure) when it reconnects to a different instance of
git-upload-pack to auto-follow the tags, and never refreshed from the
actual remote end you are talking with.

This stale list of refs is passed all the way down to fetch_pack(),
representing _your_ idea of what they said they have, which does not
match the reality if you are connecting to a different host (via DNS
round robin).  Even if there is no DNS round robin issue, if an update
happened on the host between the time you fetched the refs from there
and you started a different instance of git-upload-pack for auto
follow the tags, the issue is the same (if you re-read the list of
refs from the new instance of upload-pack, you should be Ok, as the
upload-pack on the other end should be internally consistent).

Another possibile problem area I did not check is if the current
git-fetch takes care of clearing various mark bits used and left by
find_common() and get_rev() in builtin-fetch-pack.c during the main
transfer, before it initiates another round to auto follow the tags.
When we wrote fetch-pack the first time, we did not design it (most
importantly, find_common()) to be callable more than once, as cleaning
them was unnecessary overhead for the call-once-and-exit program.

The attached stops the stale set of refs from being used for common
ancestry computation.  It may or may not fix your issue, but at least
it should be more correct than what we currently have, I think.

diff --git a/builtin-fetch.c b/builtin-fetch.c
index ac335f2..c7cdd42 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -544,9 +544,16 @@ static int do_fetch(struct transport *transport,
 
 	fetch_map = ref_map;
 
-	/* if neither --no-tags nor --tags was specified, do automated tag
-	 * following ... */
+	/*
+	 * If neither --no-tags nor --tags was specified, do automated tag
+	 * following.
+	 */
 	if (tags == TAGS_DEFAULT && autotags) {
+		if (transport->remote_refs) {
+			struct ref *stale = (struct ref *)transport->remote_refs;
+			free_refs(stale);
+			transport->remote_refs = NULL;
+		}
 		ref_map = find_non_local_tags(transport, fetch_map);
 		if (ref_map) {
 			transport_set_option(transport, TRANS_OPT_DEPTH, "0");