git@vger.kernel.org mailing list mirror (one of many)
 help / Atom feed
From: Junio C Hamano <gitster@pobox.com>
To: Takuto Ikuta <tikuta@chromium.org>
Cc: git@vger.kernel.org
Subject: Re: [PATCH v3] fetch-pack.c: use oidset to check existence of loose object
Date: Tue, 13 Mar 2018 10:53:10 -0700
Message-ID: <xmqqo9jr3lrt.fsf@gitster-ct.c.googlers.com> (raw)
In-Reply-To: <CALNjmMqBbZTFVRnc__wewWy3808n9J9xfgu8ZH-FWyWJKvqb9w@mail.gmail.com>

Takuto Ikuta <tikuta@chromium.org> writes:

>> During fetch, everything_local() tries to mark common part by
>> walking the refs the other side advertised upon the first contact,
>> so it is correct that the number of checks is not reduced in a fetch
>> that does not fetch many refs, but the number of remote-tracking refs
>> you have has no effect, so I doubt such a rephrasing would make the
>> description more understandable.  "When fetching from a repository
>> with large number of refs" is probably what you want to say, no?
>
> For refs existing in local repository, everything_local looks to be able to find
> corresponding object from packed or loose objects. And if it exists,
> I think, cost of lstat(2) is relatively smaller than other operations.
> But for remote refs, everything_local fails to find it from packed
> object (this check is fast)
> and it tries to find loose object by using lstat(2), and this fails and slow.
> My patch is to skip this lstat(2) to non-existing ref objects for repositories
> having large number of remote refs.

This still does not make sense to me, and I suspect that I am
misreading you.  In what sense are you using the word "repository"
and "remote refs"?

Imagine this situation.  I have a local repository A, I fetch from a
remote repository B but in my repository A, I do *not* use
remote-tracking refs to remember what the last values of refs at
repository B.  Now when I try to fetch a single ref from B into A,
many refs B advertises point at objects A has never heard about, and
that triggers many lstat(2) that yields ENOENT that is slow.  Your
patch is to optimize this so that we learn these objects do not
exist locally without running many lstat(2) to objects and helps
repositories (like my repository A) when fetching from a repository
with large number of refs (like the repository B).  It does not
matter how many "remote refs" receiving repository (e.g. my A) has,
and it does not matter how many "remote refs" sending repository
(e.g. my B) has---whether it is refs/remotes/origin/foo
(i.e. "remote") or refs/heads/foo (i.e. "local"), a ref at B that
points at an object that is missing at A will cause the same
lstat(2) at A without your change.

That is why I think "When fetching from a repository with large
number of refs" is what you meant, not "fetching into a repository
with large number of remote refs" nor "fetching from a repository
with large number of remote refs".

Thanks.

  reply index

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-03-08 12:06 [PATCH] " Takuto Ikuta
2018-03-08 17:19 ` René Scharfe
2018-03-09 13:42   ` Takuto Ikuta
2018-03-08 18:42 ` Junio C Hamano
2018-03-09 13:11   ` [PATCH v2 0/1] " Takuto Ikuta
2018-03-09 13:11     ` [PATCH v2 1/1] " Takuto Ikuta
2018-03-09 13:26       ` [PATCH v3] " Takuto Ikuta
2018-03-09 19:54         ` Junio C Hamano
2018-03-10 13:19           ` Takuto Ikuta
2018-03-13 17:53             ` Junio C Hamano [this message]
2018-03-14  6:26               ` Takuto Ikuta
2018-03-10 12:34         ` [PATCH v4] " Takuto Ikuta
2018-03-10 12:46           ` [PATCH v5] " Takuto Ikuta
2018-03-13 19:04             ` Junio C Hamano
2018-03-14  6:05           ` [PATCH v6] " Takuto Ikuta
2018-03-14  6:32             ` [PATCH v7] " Takuto Ikuta
2018-03-09 14:12   ` [PATCH] " Takuto Ikuta
2018-03-09 18:00     ` Junio C Hamano
2018-03-09 19:41       ` Junio C Hamano
2018-03-13 15:30   ` [PATCH] sha1_file: restore OBJECT_INFO_QUICK functionality Jonathan Tan

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=xmqqo9jr3lrt.fsf@gitster-ct.c.googlers.com \
    --to=gitster@pobox.com \
    --cc=git@vger.kernel.org \
    --cc=tikuta@chromium.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org mailing list mirror (one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox