git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / Atom feed
From: Elijah Newren <newren@gmail.com>
To: Jonathan Tan <jonathantanmy@google.com>
Cc: novalis@novalis.org, Kaushik Srenevasan <kaushik@twitter.com>,
	Git Mailing List <git@vger.kernel.org>
Subject: Re: [RFC] Extending git-replace
Date: Tue, 14 Jan 2020 14:46:17 -0800
Message-ID: <CABPp-BHBtwxk7B9LLZdCFBTOdTUCnxXyeMXG3XPavp96N+LKHg@mail.gmail.com> (raw)
In-Reply-To: <20200114215730.154601-1-jonathantanmy@google.com>

On Tue, Jan 14, 2020 at 1:57 PM Jonathan Tan <jonathantanmy@google.com> wrote:
>
> > > Missing promisor objects do not prevent fsck from passing - this is part
> > > of the original design (any packfiles we download from the specifically
> > > designated promisor remote are marked as such, and any objects that the
> > > objects in the packfile refer to are considered OK to be missing).
> >
> > Is there ever a risk that objects in the downloaded packfile come
> > across as deltas against other objects that are missing/excluded, or
> > does the partial clone machinery ensure that doesn't happen?  (Because
> > this was certainly the biggest pain-point with my "fake cheap clone"
> > hacks.)
>
> The server may send thin packs during a fetch or clone, but because the
> client runs index-pack (which calculates the hash of every object
> downloaded, necessitating having the full object, which in turn triggers
> fetches of any delta bases), this should not happen.

So if a user does a partial clone, filtering by blob size >= 1M, and
if they have several blobs of size just above and just below that
limit, then the partial clone will work but probably cause them to
still download several blobs above the limit size anyway?  (Which, if
I'm understanding correctly, happens because the blobs just smaller
than 1M likely will delta well against the blobs just larger than 1M.)

> But if you create the packfile in some other way and then manually set a
> fake promisor remote (as I perhaps too naively suggested) then the
> mechanism will attempt to fetch missing delta bases, which (I think) is
> not what you want.

Well, it's not optimal, but we're currently just dying with cryptic
errors whenever we have missing delta bases, and this happens whenever
we have an accidental fetch of older branches (although this does have
the nice side effect of notifying us of stray fetches in our CI
scripts).  Your promisor suggestion would at least permit gc's &
prunes if we use it in more places, so should be an improvement.  I
just wanted to verify whether this problem with delta bases would
remain.

> > > Currently, when a missing object is read, it is first fetched (there are
> > > some more details that I can go over if you have any specific
> > > questions). What you're suggesting here is to return a fake blob with
> > > wrong hash - I haven't looked at all the callers of read-object
> > > functions in detail, but I don't think all of them are ready for such a
> > > behavioral change.
> >
> > git-replace already took care of that for you and provides that
> > guarantee, modulo the --no-replace-objects & fsck & prune & fetch &
> > whatnot cases that ignore replace objects as Kaushik mentioned.  I
> > took advantage of this to great effect with my "fake cheap clone"
> > hacks.  Based in part on your other email where you made a suggestion
> > about promisors, I'm starting to think a pretty good first cut
> > solution might look like the following:
> >
> >   * user manually adds a bunch of replace refs to map the unwanted big
> > blobs to something else (e.g. a README about how the files were
> > stripped, or something similar to this)
> >   * a partial clone specification that says "exclude objects that are
> > referenced by replace refs"
> >   * add a fake promisor to the downloaded promisor pack so that if
> > anyone runs with --no-replace-objects or similar then they get an
> > error saying the specified objects don't exist and can't be
> > downloaded.
> >
> > Anyone see any obvious problems with this?
>
> Looking at the list of commands given in the original email (fsck,
> upload-pack, pack/unpack-objects, prune and index-pack), if we use a
> filter by blob size (instead of the partial clone specification
> suggested), this would satisfy the purposes of fsck and prune only.
>
> If we had a partial clone specification that excludes object referenced
> by replace refs, then upload-pack from this partial repository (and
> pack-objects) would work too.
>
> But there might be non-obvious problems that I haven't thought of.

Cool, sounds like it's at least worth investigating.  Maybe Kaushik is
interested, or maybe I consider throwing it on my backlog and coming
back to it in a year or two.  :-)

      reply index

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-14  5:33 Kaushik Srenevasan
2020-01-14  6:55 ` Elijah Newren
2020-01-14 19:11   ` Jonathan Tan
2020-01-16  3:30   ` Kaushik Srenevasan
2020-01-14 18:19 ` David Turner
2020-01-14 19:03   ` Jonathan Tan
2020-01-14 20:39     ` Elijah Newren
2020-01-14 21:57       ` Jonathan Tan
2020-01-14 22:46         ` Elijah Newren [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CABPp-BHBtwxk7B9LLZdCFBTOdTUCnxXyeMXG3XPavp96N+LKHg@mail.gmail.com \
    --to=newren@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=jonathantanmy@google.com \
    --cc=kaushik@twitter.com \
    --cc=novalis@novalis.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org list mirror (unofficial, one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git