git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / Atom feed
From: Jonathan Tan <jonathantanmy@google.com>
To: newren@gmail.com
Cc: kaushik@twitter.com, git@vger.kernel.org, jonathantanmy@google.com
Subject: Re: [RFC] Extending git-replace
Date: Tue, 14 Jan 2020 11:11:56 -0800
Message-ID: <20200114191156.122421-1-jonathantanmy@google.com> (raw)
In-Reply-To: <CABPp-BGy3qu_Rd4epore0wLyoh1fg0UH5EAV27shKJ=kLWX4FA@mail.gmail.com>

> 2) Some folks might be okay with a clone that won't pass fsck or
> prune, at least in special circumstances.  We're actually doing that
> on purpose to deal with one of our large repositories.  We don't
> provide that to normal developers, but we do use "cheap, fake clones"
> in our CI systems.  These slim clones have 99% of all objects, but
> happen to be missing the really big ones, resulting in only needing
> 1/7 of the time to download.  (And no, don't try to point out shallow
> clones to me.  I hate those things, they're an awful hack, *and* they
> don't work for us.  It's nice getting all commit history, all trees,
> and most blobs including all for at least the last two years while
> still saving lots of space.)
> 
> [For the curious, I did make a simple script to create these "cheap,
> fake clones" for repositories of interest.  See
> https://github.com/newren/sequester-old-big-blobs.  But they are
> definitely a hack with some sharp corners, with failing fsck and
> prunes only being part of the story.]

If you want to reduce the sharpness of the corners, it might be possible
to designate the pack with missing blobs as a promisor pack (add a
.promisor file - which is just like the .keep file except
s/keep/promisor/) and a fake promisor remote. That will make fsck and
repack (GC) work.

> 3) Back to your idea...
> 
> What you're proposing actually sounds very similar to partial clones,
> whose idea is to make it okay to download a subset of history.  The
> primary problems with partial clones are (a) they are still under
> development and are just experimental, (b) they are currently
> implemented with a "promisor" mode, meaning that if a command tries to
> run over any piece of missing data then the command pauses while the
> objects are downloaded from the server.  I want an offline mode (even
> if I'm online) where only explicit downloading from the server (clone,
> fetch, etc.) occurs.

David Turner had an idea of what could be done (instead of fetching) in
such an offline mode [1], so I replied there.

[1] https://lore.kernel.org/git/d4361b6d34513a3fdefa564d10269f60d4732208.camel@novalis.org/

> Instead of inventing yet another partial-clone-like thing, it'd be
> nice if your new mechanism could just be implemented in terms of
> partial clones, extending them as you need.  I don't like the idea of
> supporting multiple competing implementations of partial clones
> withing git.git, but if it's just some extensions of the existing
> capability then it sounds great.  But you may want to talk with
> Jonathan Tan if you want to go this route (cc'd), since he's the
> partial clone expert.

Ah, thanks for your kind words.

  reply index

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-14  5:33 Kaushik Srenevasan
2020-01-14  6:55 ` Elijah Newren
2020-01-14 19:11   ` Jonathan Tan [this message]
2020-01-16  3:30   ` Kaushik Srenevasan
2020-01-14 18:19 ` David Turner
2020-01-14 19:03   ` Jonathan Tan
2020-01-14 20:39     ` Elijah Newren
2020-01-14 21:57       ` Jonathan Tan
2020-01-14 22:46         ` Elijah Newren

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200114191156.122421-1-jonathantanmy@google.com \
    --to=jonathantanmy@google.com \
    --cc=git@vger.kernel.org \
    --cc=kaushik@twitter.com \
    --cc=newren@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org list mirror (unofficial, one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git