"Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

"Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Stefan Beller]

So I ran "git fetch --all" inside git.git that you are all familiar with.
All fetches failed with a similar error as:
Fetching kernelorg
fatal: bad object HEAD
error: https://kernel.googlesource.com/pub/scm/git/git did not send
all necessary objects

error: Could not fetch kernelorg

Working with an older version of Git (2.13.0) gives me

$ GIT_TRACE=/u/trace GIT_TRACE_PACKET=/u/out git fetch git://github.com/git/git
From git://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD

After fiddling around and some printf debugging,
clean fsck(!)
I could not make sense of it, so bisecting

$ git bisect bad
d0c39a49ccb5dfe7feba4325c3374d99ab123c59 is the first bad commit
commit d0c39a49ccb5dfe7feba4325c3374d99ab123c59
Author: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date:   Wed Aug 23 19:36:59 2017 +0700

    revision.c: --all adds HEAD from all worktrees

    Unless single_worktree is set, --all now adds HEAD from all worktrees.

    Since reachable.c code does not use setup_revisions(), we need to call
    other_head_refs_submodule() explicitly there to have the same effect on
    "git prune", so that we won't accidentally delete objects needed by some
    other HEADs.

    A new FIXME is added because we would need something like

        int refs_other_head_refs(struct ref_store *, each_ref_fn, cb_data);

    in addition to other_head_refs() to handle it, which might require

        int get_submodule_worktrees(const char *submodule, int flags);

    It could be a separate topic to reduce the scope of this one.

    Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

And here we go, I do use worktrees nowadays!
$ git worktree list
/usr/local/google/home/sbeller/OSS/git                   117ddefdb4
(detached HEAD)
/u/git                                                   d0c39a49cc
(detached HEAD)
/usr/local/google/home/sbeller/OSS/git_origin_master     b14f27f917
(detached HEAD)
/usr/local/google/home/sbeller/OSS/submodule_remote_dot  3d9025bd69
(detached HEAD)

$ git show 3d9025bd69
fatal: ambiguous argument '3d9025bd69': unknown revision or path not
in the working tree.

ok, so I presume I just delete that working tree to "fix my copy of git"
sbeller@sbeller:/u/git$ rm -rf
/usr/local/google/home/sbeller/OSS/submodule_remote_dot
sbeller@sbeller:/u/git$ git worktree prune
sbeller@sbeller:/u/git$ git worktree list
/usr/local/google/home/sbeller/OSS/git                117ddefdb4 (detached HEAD)
/u/git                                                d0c39a49cc (detached HEAD)
/usr/local/google/home/sbeller/OSS/git_origin_master  b14f27f917 (detached HEAD)
$ git bisect reset
Previous HEAD position was d0c39a49cc... revision.c: --all adds HEAD
from all worktrees
HEAD is now at 660fb3dfa8... Sync with maint
$ make install
$ git fetch --all
# works fine!

Any idea which direction we need to heading for a real fix?

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Jeff King]

On Thu, Oct 19, 2017 at 06:43:57PM -0700, Stefan Beller wrote:

> So I ran "git fetch --all" inside git.git that you are all familiar with.
> All fetches failed with a similar error as:
> Fetching kernelorg
> fatal: bad object HEAD
> error: https://kernel.googlesource.com/pub/scm/git/git did not send
> all necessary objects
> 
> error: Could not fetch kernelorg

I ran into this a while back with worktrees, too. In my case I had a
worktree which was "owned" by multiple repositories[1]. One would write
the HEAD, but the other did not necessarily have those objects.

I don't quite recall how I ended up there. Doing "git worktree add"
multiple times does seem to complain about the directory already
existing. My case did involve relative symlinks in both repositories, so
maybe that confused things. Anyway, I can't seem to replicate it now.

> And here we go, I do use worktrees nowadays!
> $ git worktree list
> /usr/local/google/home/sbeller/OSS/git                   117ddefdb4
> (detached HEAD)
> /u/git                                                   d0c39a49cc
> (detached HEAD)
> /usr/local/google/home/sbeller/OSS/git_origin_master     b14f27f917
> (detached HEAD)
> /usr/local/google/home/sbeller/OSS/submodule_remote_dot  3d9025bd69
> (detached HEAD)
> 
> $ git show 3d9025bd69
> fatal: ambiguous argument '3d9025bd69': unknown revision or path not
> in the working tree.

So maybe you've ended up in the same situation and somehow "reused"
submodule_remote_dot.

But I think until that commit you mentioned (d0c39a49cc), "gc" in the
parent repo could drop objects accessible only from the worktree. So if
that happened _before_ d0c39a49cc, then at that point your worktree was
corrupted. And now of course further operations will complain.

> ok, so I presume I just delete that working tree to "fix my copy of git"
> sbeller@sbeller:/u/git$ rm -rf
> /usr/local/google/home/sbeller/OSS/submodule_remote_dot
> sbeller@sbeller:/u/git$ git worktree prune
> sbeller@sbeller:/u/git$ git worktree list
> /usr/local/google/home/sbeller/OSS/git                117ddefdb4 (detached HEAD)
> /u/git                                                d0c39a49cc (detached HEAD)
> /usr/local/google/home/sbeller/OSS/git_origin_master  b14f27f917 (detached HEAD)

So that's the right immediate workaround (and hoping there was nothing
valuable in that worktree!).

> Any idea which direction we need to heading for a real fix?

If my analysis above is correct, then it's already fixed. You just had
leftover corruption.

-Peff

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Stefan Beller]

On Thu, Oct 19, 2017 at 8:16 PM, Jeff King <peff@peff.net> wrote:
> On Thu, Oct 19, 2017 at 06:43:57PM -0700, Stefan Beller wrote:

>
> So that's the right immediate workaround (and hoping there was nothing
> valuable in that worktree!).

How would I know? ;)

Without the commit available I do not even know if the code there
is well reasoned about. (My commit messages usually tell my future
self how much thought I put into the code)

>
>> Any idea which direction we need to heading for a real fix?
>
> If my analysis above is correct, then it's already fixed. You just had
> leftover corruption.

Well fetching yesterday worked and the commit in question is from
8/23, the merge  8a044c7f1d56cef657be342e40de0795d688e882
occurred 9/18, so I suspect there is something else at play.
(I do not remember having a gc between yesterday and today.
Though maybe one in the background?)


I am curious how you can have a worktree owned by multiple
repositories [1] (?).

Thanks,
Stefan

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Jeff King]

On Thu, Oct 19, 2017 at 10:27:28PM -0700, Stefan Beller wrote:

> > If my analysis above is correct, then it's already fixed. You just had
> > leftover corruption.
> 
> Well fetching yesterday worked and the commit in question is from
> 8/23, the merge  8a044c7f1d56cef657be342e40de0795d688e882
> occurred 9/18, so I suspect there is something else at play.
> (I do not remember having a gc between yesterday and today.
> Though maybe one in the background?)

Even a gc between yesterday and today should have used the new code,
which would have been safe. So yeah, maybe it is something else
entirely.

> I am curious how you can have a worktree owned by multiple
> repositories [1] (?).

Sorry, I forgot my footnote. I saw this with my "ci" script:

  https://github.com/peff/git/blob/7905ff395adecdd2bb7ab045a24223dfb103e0e9/ci

I check out the contents of my "meta" branch as "Meta", and it contains
that script. It basically just waits for ref updates, then walks over
all the commits and runs "make test" on them in the background (caching
the results, thanks to the git-test[1] script). So I kick off "Meta/ci"
in a terminal and forget about it, and magically it builds my commits in
the background as I work.

It operates in a worktree inside the Meta directory (Meta/tmp-ci), so as
not to disturb what I'm doing. So far so good.

But I actually have _two_ clones of Git on my system. One on which I do
most of my work, and then the other which has the fork we use in
production at GitHub. I symlink the Meta directory from the first into
the latter, which means they both see the same worktree directory. And
somehow running "Meta/ci" in the second corrupted things.

I can get some funniness now, but I think it's mostly caused by the
script being confused about the worktree existing but not having access
to our branches. That's not a corruption, just a confusion. I _think_ I
had a bogus HEAD in the worktree at one point, but I may be
mis-remembering. I can't seem to trigger it now.

-Peff

[1] https://github.com/mhagger/git-test

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Stefan Beller]

On Thu, Oct 19, 2017 at 11:04 PM, Jeff King <peff@peff.net> wrote:
> On Thu, Oct 19, 2017 at 10:27:28PM -0700, Stefan Beller wrote:
>
>> > If my analysis above is correct, then it's already fixed. You just had
>> > leftover corruption.
>>
>> Well fetching yesterday worked and the commit in question is from
>> 8/23, the merge  8a044c7f1d56cef657be342e40de0795d688e882
>> occurred 9/18, so I suspect there is something else at play.
>> (I do not remember having a gc between yesterday and today.
>> Though maybe one in the background?)
>
> Even a gc between yesterday and today should have used the new code,
> which would have been safe. So yeah, maybe it is something else
> entirely.

Oh, yeah.

>
>> I am curious how you can have a worktree owned by multiple
>> repositories [1] (?).
>
> Sorry, I forgot my footnote. I saw this with my "ci" script:
>
>   https://github.com/peff/git/blob/7905ff395adecdd2bb7ab045a24223dfb103e0e9/ci
>
> I check out the contents of my "meta" branch as "Meta", and it contains
> that script. It basically just waits for ref updates, then walks over
> all the commits and runs "make test" on them in the background (caching
> the results, thanks to the git-test[1] script). So I kick off "Meta/ci"
> in a terminal and forget about it, and magically it builds my commits in
> the background as I work.
>
> It operates in a worktree inside the Meta directory (Meta/tmp-ci), so as
> not to disturb what I'm doing. So far so good.
>
> But I actually have _two_ clones of Git on my system. One on which I do
> most of my work, and then the other which has the fork we use in
> production at GitHub. I symlink the Meta directory from the first into
> the latter, which means they both see the same worktree directory. And
> somehow running "Meta/ci" in the second corrupted things.
>
> I can get some funniness now, but I think it's mostly caused by the
> script being confused about the worktree existing but not having access
> to our branches. That's not a corruption, just a confusion. I _think_ I
> had a bogus HEAD in the worktree at one point, but I may be
> mis-remembering. I can't seem to trigger it now.

Thanks for these insights.

I played around with Meta a bit, but I did not feel it would enhance
my workflow enough as I am not involved with any maintainance
of git using git.

The git-test from Michael sounds intriguing. Initially I put off using
it as I had my main working dir (or rather test dir) on a spinning
disk, still. Now I test in memory only, which is a lot faster, so I could
try if git-test can keep up with my local commit pace.

Thanks,
Stefan

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Kaartic Sivaraam]

On Fri, 2017-10-20 at 13:45 -0700, Stefan Beller wrote:
> 
> The git-test from Michael sounds intriguing. Initially I put off using
> it as I had my main working dir (or rather test dir) on a spinning
> disk, still. Now I test in memory only, which is a lot faster, so I could
> try if git-test can keep up with my local commit pace.
> 

Excuse my ignorance but I don't get your statement correctly. What do
you mean by "I test in memory only"? How do you do that?

---
Kaartic

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Jeff King]

On Sun, Oct 22, 2017 at 07:54:57PM +0530, Kaartic Sivaraam wrote:

> On Fri, 2017-10-20 at 13:45 -0700, Stefan Beller wrote:
> > 
> > The git-test from Michael sounds intriguing. Initially I put off using
> > it as I had my main working dir (or rather test dir) on a spinning
> > disk, still. Now I test in memory only, which is a lot faster, so I could
> > try if git-test can keep up with my local commit pace.
> > 
> 
> Excuse my ignorance but I don't get your statement correctly. What do
> you mean by "I test in memory only"? How do you do that?

I assume it means pointing "--root" at a tmpfs or ram disk in
GIT_TEST_OPTS. That makes the test suite run _way_ faster.

-Peff

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Stefan Beller]

On Sun, Oct 22, 2017 at 5:36 PM, Jeff King <peff@peff.net> wrote:
> On Sun, Oct 22, 2017 at 07:54:57PM +0530, Kaartic Sivaraam wrote:
>
>> On Fri, 2017-10-20 at 13:45 -0700, Stefan Beller wrote:
>> >
>> > The git-test from Michael sounds intriguing. Initially I put off using
>> > it as I had my main working dir (or rather test dir) on a spinning
>> > disk, still. Now I test in memory only, which is a lot faster, so I could
>> > try if git-test can keep up with my local commit pace.
>> >
>>
>> Excuse my ignorance but I don't get your statement correctly. What do
>> you mean by "I test in memory only"? How do you do that?
>
> I assume it means pointing "--root" at a tmpfs or ram disk in
> GIT_TEST_OPTS. That makes the test suite run _way_ faster.

Exactly. By memory I mean volatile RAM (as opposed to
memory on a spinning disk).

Using GIT_TEST_OPTS has had some issues (I remember vaguely
there was an inconsistency between the output of `make test` and prove),
so I put my entire working tree on a tmpfs, I run roughly this script
after booting my computer:

  sudo mount -t tmpfs -o size=16g tmpfs /u
  mkdir /u/git
  echo "gitdir:
/usr/local/google/home/sbeller/OSS/git/.git/worktrees/git"
>/u/git/.git
  git -C /u/git checkout -f HEAD

  cat <<EOF >/u/config.mak
  DEVELOPER=1
  DEVELOPERS=1
  CFLAGS += -g -O2
  CFLAGS += -DFLEX_ARRAY=2048
  #CFLAGS += -Wno-unused-value
  EOF

The downside (or upside?) is that there will be no files cluttering
my git directory, but any untracked file is wiped periodically. So if I want
to really preserve a thing I am forcing myself to put it somewhere else
than in an untracked file inside the git repo.

The test suite (excluding t9*) runs in less than 50 seconds on the ram
disk.

Thanks,
Stefan

>
> -Peff

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Kaartic Sivaraam]

On Monday 23 October 2017 11:07 PM, Stefan Beller wrote:
> Exactly. By memory I mean volatile RAM (as opposed to
> memory on a spinning disk).
> 
> Using GIT_TEST_OPTS has had some issues (I remember vaguely
> there was an inconsistency between the output of `make test` and prove),
> so I put my entire working tree on a tmpfs, I run roughly this script
> after booting my computer:
> 
>    sudo mount -t tmpfs -o size=16g tmpfs /u
>    mkdir /u/git
>    echo "gitdir:
> /usr/local/google/home/sbeller/OSS/git/.git/worktrees/git"
>> /u/git/.git
>    git -C /u/git checkout -f HEAD
> 
>    cat <<EOF >/u/config.mak
>    DEVELOPER=1
>    DEVELOPERS=1
>    CFLAGS += -g -O2
>    CFLAGS += -DFLEX_ARRAY=2048
>    #CFLAGS += -Wno-unused-value
>    EOF

Did I thank you for a good explanation? If not, thanks that was 
interesting and enlightening.

> The test suite (excluding t9*) runs in less than 50 seconds on the ram
> disk.
> 

BTW, this is what I call _way way_ faster. Unfortunately due to the 
limited configuration of my system, the test suite has following timing

     real    3m14.482s
     user    2m10.556s
     sys     1m12.328s

Anyways it's better than the timing I get for running it on a spinning 
disk which takes

     real    4m37.585s
     user    2m17.244s
     sys     1m37.136s

Thanks,
Kaartic

Re: "Cannot fetch git.git" (worktrees at fault? or origin/HEAD) ?

[Stefan Beller]

On Fri, Nov 3, 2017 at 2:32 AM, Kaartic Sivaraam
<kaartic.sivaraam@gmail.com> wrote:
> On Monday 23 October 2017 11:07 PM, Stefan Beller wrote:
>>
>> Exactly. By memory I mean volatile RAM (as opposed to
>> memory on a spinning disk).
>>
>> Using GIT_TEST_OPTS has had some issues (I remember vaguely
>> there was an inconsistency between the output of `make test` and prove),
>> so I put my entire working tree on a tmpfs, I run roughly this script
>> after booting my computer:
>>
>>    sudo mount -t tmpfs -o size=16g tmpfs /u
>>    mkdir /u/git
>>    echo "gitdir:
>> /usr/local/google/home/sbeller/OSS/git/.git/worktrees/git"
>>>
>>> /u/git/.git
>>
>>    git -C /u/git checkout -f HEAD
>>
>>    cat <<EOF >/u/config.mak
>>    DEVELOPER=1
>>    DEVELOPERS=1
>>    CFLAGS += -g -O2
>>    CFLAGS += -DFLEX_ARRAY=2048
>>    #CFLAGS += -Wno-unused-value
>>    EOF
>
>
> Did I thank you for a good explanation? If not, thanks that was interesting
> and enlightening.
>
>> The test suite (excluding t9*) runs in less than 50 seconds on the ram
>> disk.
>>

Just tested again, I meant to say 70s instead of 50s.


> BTW, this is what I call _way way_ faster. Unfortunately due to the limited
> configuration of my system, the test suite has following timing
>
>     real    3m14.482s
>     user    2m10.556s
>     sys     1m12.328s

This sounds to me as if it is running with just one thread
(because sys+user = real); I usually run with

    cd t
    prove --jobs 25 t[0-8][0-9]*.sh

The multithreading can be seen in the timing as well
    real 1m9.913s
    user 1m50.796s
    sys 0m54.092s
as user > real already.

If you run tests via 'make test' or 'cd t && make', you can also give
a --jobs <n>
to make it faster. I have no good answer for how many, but I have the impression
overloading the system is no big deal. (I only have a few cores in this machine,
4 or 6, but still run with --jobs 25; 'git grep sleep' returns a
couple of lines,
and such threads sleeping definitely don't need a CPU)

Thanks,
Stefan

Re: "Cannot fetch git.git" ( iworktrees at fault? or origin/HEAD) ?

[Kaartic Sivaraam]

On Fri, 2017-11-03 at 12:11 -0700, Stefan Beller wrote:
> On Fri, Nov 3, 2017 at 2:32 AM, Kaartic Sivaraam
> <kaartic.sivaraam@gmail.com> wrote:
> > BTW, this is what I call _way way_ faster. Unfortunately due to the limited
> > configuration of my system, the test suite has following timing
> > 
> >     real    3m14.482s
> >     user    2m10.556s
> >     sys     1m12.328s
> 
> This sounds to me as if it is running with just one thread
> (because sys+user = real); I usually run with
> 
>     cd t
>     prove --jobs 25 t[0-8][0-9]*.sh
> 
> The multithreading can be seen in the timing as well
>     real 1m9.913s
>     user 1m50.796s
>     sys 0m54.092s
> as user > real already.
> 
> If you run tests via 'make test' or 'cd t && make', you can also give
> a --jobs <n>
> to make it faster. I have no good answer for how many, but I have the impression
> overloading the system is no big deal. (I only have a few cores in this machine,
> 4 or 6, but still run with --jobs 25; 'git grep sleep' returns a
> couple of lines,
> and such threads sleeping definitely don't need a CPU)
> 

I usually use the following command to build and run the test suite
from the root of the working directory,

    make NO_GETTEXT=1 DEVELOPER=1 DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS="--timer --jobs 16" test

    I beleived that the DEFAULT_TST_TARGE and GIT_PROVE_OPTS options would
    be seen by the Makefile in t/ but didn't verify it. Your statement made
    me suspicious and I wanted to know whether my assumption was correct or
    not. So, I did this,


    1. I first tried running the above command while seeing how much time
    on the 'next' branch when it was pointing at,

    9a519c715 (Merge branch 'jk/rebase-i-exec-gitdir-fix' into next, 2017-11-02)

    First, I observed that at least part of my assumption held on observing
    that the output was similar to that of 'prove'. This shows that
    t/Makefile recognizes DEFAULT_TEST_TARGET.

    Next, to verify if it recognizes GIT_PROVE_OPTS I ran the above command
    with 'time' and it gave back the following statistics,

    real    	    4m55.707s
    user    	    2m29.924s
    sys    	    1m48.072s

    which is in line with the statistics for the spinning disk in the
    previous mail. (The time doesn't include the build time).

    (real)-(user+sys) = 38s (approx.)


    2. Now I switched to the t/ directory, I tried running the following,

    make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS="--timer --jobs 16"

    'time' gave the statistics for the above command as follows,

    real    	    5m19.523s
    user    	    2m26.764s
    sys    	    1m45.240s

    (real)-(user+sys) =     68s (approx.)
Not a big improvement from the previous case to assure that
GIT_PROVE_OPTS wasn't recognized there.


3. Staying in the t/ directory, I tried running the following just to
be pretty sure that the "--jobs" was indeed sent to 'prove',

    prove --jobs 16 t[0-9][0-9]*.sh

'time' gave the statistics for the above command as follows,

    real    	    4m49.241s
    user    	    2m29.220s
    sys    	    1m47.828s

    (real)-(user+sys) = 34s (approx.)

    This seems to be identical with the first case.

    So, it does seem to be a limitation of my system (Intel i3 with 2
    cores, FWIW).


    4. To see if things improve with a higher number of jobs I tried,

    prove --jobs 25 t[0-9][0-9]*.sh

    'time' gave the statistics for the above command as follows,

    real    	    5m21.229s
    user    	    2m25.704s
    sys    	    1m46.744s

        (real)-(user+sys) = 70s (approx.)

        This ensured the limitation once more!

        Anyways, thanks for the explanation and information which was
        enlightening. :-)

-- 
Kaartic