From: Jeff King <email@example.com> To: Shawn Pearce <firstname.lastname@example.org> Cc: Michael Haggerty <email@example.com>, git <firstname.lastname@example.org>, Junio C Hamano <email@example.com>, David Borowitz <firstname.lastname@example.org> Subject: Re: reftable [v4]: new ref storage format Date: Wed, 2 Aug 2017 13:28:45 -0400 [thread overview] Message-ID: <email@example.com> (raw) In-Reply-To: <CAJo=hJu1rud5pEZ93HDty1qyaCOHmwn89aEvPFe2ER0JD1ExwQ@mail.gmail.com> On Wed, Aug 02, 2017 at 08:17:29AM -0700, Shawn Pearce wrote: > > Just peeking at torvalds/linux, we have some objects with ~35K refs > > pointing to them (e.g., the v2.6.11 tag). > > Oy. I'll bet that every occurrence winds up in its own block due to > the layout of the namespace, and so the obj block list needs 35k > varint pointers. That requires a larger block size if it has any > chance of fitting into the reftable format. > > Another option is disable the obj table for these shared-object repos. > Its an optional part of the format and can be omitted if the reader > isn't likely to need to lookup by SHA-1, or is willing to pay the > brute force cost of scanning every ref. Yeah, sorry, I meant to write a few more paragraphs. I think refusing to generate the object table for these repos would be OK. We don't serve any user-facing operations out of them directly. I'm also open to the argument that they're simply insane. Most of the time we don't need them to be a real repository at all. They could exist as a bare "objects/" directory. It's only at repack time that we actually need to know which objects are reachable, so we could do a special repack that generates the list on the fly from the child repositories. -Peff  We actually disable the ".have" advertisements, because the cost of showing all of the shared-storage ref tips is too high. One thing I'd like to do is be able to advertise a subset of the alternate refs (if you're a fork of torvalds/linux, then share _just_ the refs from there). But with the current ref code, I can't even ask for a subset of the refs without paying the cost to walk all of them. That's one of the things I'd like to build on top of the mmap'd packed-refs solution (and naturally would work with reftables, too).  It's a bit more complicated than just knowing the list of reachable objects. We also want to know which ones are reachable from which fork, as we do some trickery to avoid creating deltas across forks. So we really do want the whole ref-list, and not something like a de-duped set of reachable tips. I don't think that makes a difference for anything we're discussing here, but just a bit of trivia in case somebody is thinking about the shared-storage problem space.
next prev parent reply other threads:[~2017-08-02 17:28 UTC|newest] Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-07-31 3:51 Shawn Pearce 2017-07-31 17:41 ` Dave Borowitz 2017-07-31 19:01 ` Stefan Beller 2017-07-31 23:05 ` Shawn Pearce 2017-07-31 19:42 ` Junio C Hamano 2017-07-31 23:43 ` Shawn Pearce 2017-08-01 16:08 ` Shawn Pearce 2017-08-01 6:41 ` Michael Haggerty 2017-08-01 20:23 ` Shawn Pearce 2017-08-02 0:49 ` Michael Haggerty 2017-08-01 23:27 ` Shawn Pearce 2017-08-01 23:54 ` Shawn Pearce 2017-08-02 1:51 ` Michael Haggerty 2017-08-02 2:38 ` Shawn Pearce 2017-08-02 9:28 ` Jeff King 2017-08-02 15:17 ` Shawn Pearce 2017-08-02 16:51 ` Junio C Hamano 2017-08-02 17:28 ` Jeff King [this message] 2017-08-02 12:20 ` Dave Borowitz 2017-08-02 17:18 ` Jeff King 2017-08-03 18:38 ` Michael Haggerty 2017-08-03 22:26 ` Shawn Pearce 2017-08-03 22:48 ` Michael Haggerty 2017-08-04 2:50 ` Shawn Pearce 2017-08-05 21:00 ` Shawn Pearce 2017-08-01 13:54 ` Dave Borowitz 2017-08-01 15:27 ` Shawn Pearce 2017-08-02 19:50 ` Junio C Hamano 2017-08-02 20:28 ` Jeff King 2017-08-03 22:17 ` Shawn Pearce 2017-08-03 1:50 ` Junio C Hamano 2017-08-03 2:21 ` Shawn Pearce 2017-08-03 2:36 ` Junio C Hamano 2017-08-02 19:54 ` Stefan Beller
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://vger.kernel.org/majordomo-info.html * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --subject='Re: reftable [v4]: new ref storage format' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).