From: Dave Borowitz <dborowitz@google.com>
To: Shawn Pearce <spearce@spearce.org>
Cc: Michael Haggerty <mhagger@alum.mit.edu>,
git <git@vger.kernel.org>, Jeff King <peff@peff.net>,
Junio C Hamano <gitster@pobox.com>
Subject: Re: reftable [v4]: new ref storage format
Date: Wed, 2 Aug 2017 08:20:44 -0400 [thread overview]
Message-ID: <CAD0k6qRTa6jSgEBBX1Ux5yg4QMMWPpyOGTa471cRhtzBaS-KjQ@mail.gmail.com> (raw)
In-Reply-To: <CAJo=hJv=zJvbzfAZwspxECXrnBJR4XfJbGZegsNUCx=6uheO2Q@mail.gmail.com>
On Tue, Aug 1, 2017 at 10:38 PM, Shawn Pearce <spearce@spearce.org> wrote:
>> Peff and I discussed off-list whether the lookup-by-SHA-1 feature is
>> so important in the first place. Currently, all references must be
>> scanned for the advertisement anyway,
>
> Not really. You can hide refs and allow-tip-sha1 so clients can fetch
> a ref even if it wasn't in the advertisement. We really want to use
> that wire protocol capability with Gerrit Code Review to hide the
> refs/changes/ namespace from the advertisement, but allow clients to
> fetch any of those refs if they send its current SHA-1 in a want line
> anyway.
>
> So a server could scan only the refs/{heads,tags}/ prefixes for the
> advertisement, and then leverage the lookup-by-SHA1 to verify other
> SHA-1s sent by the client.
>
>> so avoiding a second scan to vet
>> SHA-1s received from the client is at best going to reduce the effort
>> by a constant factor. Do you have numbers showing that this
>> optimization is worth it?
>
> No, but I don't think I need to do much to prove it. My 866k ref
> example advertisement right now is >62 MiB. If we do what I'm
> suggesting in the paragraphs above, the advertisement is ~51 KiB.
That being said, our bias towards minimizing the number of ref scans
is rooted in our experience where scanning 866k refs takes 5 seconds
to get the response from the storage backend into the git server.
Cutting ref scans from 2 to 1 (or 1 to 0) is a big deal in that case.
But that 5s number is based on our current, slow storage, not on
reftable. If migrating to reftable turns each 5s scan into a 400ms
scan, we might be able to live with that, even if we don't have fast
lookup by SHA-1.
>> OTOH a mythical protocol v2 might reduce the need to scan the
>> references for advertisement, so maybe this optimization will be more
>> helpful in the future?
I haven't been following the status of the proposal, but I was
assuming a client-speaks-first protocol would also imply the client
asking for refnames, not SHA-1s, in which case lookup by SHA-1 is no
longer relevant.
next prev parent reply other threads:[~2017-08-02 12:21 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-31 3:51 reftable [v4]: new ref storage format Shawn Pearce
2017-07-31 17:41 ` Dave Borowitz
2017-07-31 19:01 ` Stefan Beller
2017-07-31 23:05 ` Shawn Pearce
2017-07-31 19:42 ` Junio C Hamano
2017-07-31 23:43 ` Shawn Pearce
2017-08-01 16:08 ` Shawn Pearce
2017-08-01 6:41 ` Michael Haggerty
2017-08-01 20:23 ` Shawn Pearce
2017-08-02 0:49 ` Michael Haggerty
2017-08-01 23:27 ` Shawn Pearce
2017-08-01 23:54 ` Shawn Pearce
2017-08-02 1:51 ` Michael Haggerty
2017-08-02 2:38 ` Shawn Pearce
2017-08-02 9:28 ` Jeff King
2017-08-02 15:17 ` Shawn Pearce
2017-08-02 16:51 ` Junio C Hamano
2017-08-02 17:28 ` Jeff King
2017-08-02 12:20 ` Dave Borowitz [this message]
2017-08-02 17:18 ` Jeff King
2017-08-03 18:38 ` Michael Haggerty
2017-08-03 22:26 ` Shawn Pearce
2017-08-03 22:48 ` Michael Haggerty
2017-08-04 2:50 ` Shawn Pearce
2017-08-05 21:00 ` Shawn Pearce
2017-08-01 13:54 ` Dave Borowitz
2017-08-01 15:27 ` Shawn Pearce
2017-08-02 19:50 ` Junio C Hamano
2017-08-02 20:28 ` Jeff King
2017-08-03 22:17 ` Shawn Pearce
2017-08-03 1:50 ` Junio C Hamano
2017-08-03 2:21 ` Shawn Pearce
2017-08-03 2:36 ` Junio C Hamano
2017-08-02 19:54 ` Stefan Beller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAD0k6qRTa6jSgEBBX1Ux5yg4QMMWPpyOGTa471cRhtzBaS-KjQ@mail.gmail.com \
--to=dborowitz@google.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=mhagger@alum.mit.edu \
--cc=peff@peff.net \
--cc=spearce@spearce.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).