git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
* Bundles: Partial Clone & Shallow clone
@ 2020-10-27 21:55 Robin H. Johnson
  2020-10-30  8:19 ` Christian Couder
  0 siblings, 1 reply; 2+ messages in thread
From: Robin H. Johnson @ 2020-10-27 21:55 UTC (permalink / raw)
  To: Git Mailing List

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

Hi!

I was wondering if anybody has made a start towards easier ways to
create bundles of partial & shallow clones.

I'm looking at potential ways to improve the various snapshot &
incremental update mechanisms that we have available in Gentoo Linux's
tree distribution.

Right now, the various mechanisms available are:
- git tree WITHOUT generated metadata (git-daemon, smart-http)

Plus all of the following WITH generated metadata.
- git tree (git-daemon OR smart-http)
- daily full snapshots in several formats/layouts.
- deltas of full snapshots in some of the formats & layouts [1-day interval, no rollback]
- rsync:// tree [30-min intervals, no rollback]

Offline/Air-gapped use cases presently are expected to use the
snapshots+deltas, but the 1-day cadence is longer than desirable

Several of these formats cannot rollback unless they kept the old tree
or the snapshot it was generated from.

rsync:// performance is heavily impacted by network latency and large
number of small objects being transferred.

What I'd like to offer instead, is CDN-replicated bundles generated at
regular cadences, with the absolute minimal content; taken from the git
tree WITH generated metadata. The git tree would have tags for every
cadence point (ideally 30-minutes, with potential pruning of old tags).
The bundles would have GPG-signed checksums separately included, to
provide verification of the updates.

These come in two variants:
1. Daily full snapshots, equivalent to depth=1 clone. 
2. 30-min & daily incremental bundles, using partial clone (needs to
   include the new blobs that would be present when up to date, and
   knowledge of which files can be deleted).

This should let users load some consistent set of daily or 30-min bundle
onto their gitdir, and hop between those tags [they would not have
gaps].

If they did have gaps due to a missing bundle or wanting to go within
the cadence points, and were online, they could use the partial
mechanisms to fill in their tree as needed.

Right now, I can naively generate the snapshots by explicitly making a
new detached shallow clone & then generating a bundle of that. 

Incremental bundles are already possible, but presently include all
commits, trees & blobs between two points.

The bundles are already generally smaller than our prior snapshots and
deltas, but I'm looking to make the process easier and cover the
remaining gaps [if we have a high-change period, then the bundle winds
up bigger than other deltas, because of the intermediate blobs].

Ask 1. Ability to generate shallow bundle without the intermediate clone step.

Ask 2. For incremental bundles, ability to exclude blobs not needed by
       the latest commit (and it's tree).

Both of these I think would be possible by adding some variant on the
--filter mechanism to git-bundle.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: Bundles: Partial Clone & Shallow clone
  2020-10-27 21:55 Bundles: Partial Clone & Shallow clone Robin H. Johnson
@ 2020-10-30  8:19 ` Christian Couder
  0 siblings, 0 replies; 2+ messages in thread
From: Christian Couder @ 2020-10-30  8:19 UTC (permalink / raw)
  To: Robin H. Johnson; +Cc: Git Mailing List

Hi,

On Wed, Oct 28, 2020 at 10:39 PM Robin H. Johnson <robbat2@gentoo.org> wrote:
>
> Hi!
>
> I was wondering if anybody has made a start towards easier ways to
> create bundles of partial & shallow clones.

I don't think anybody has started working on this.

> I'm looking at potential ways to improve the various snapshot &
> incremental update mechanisms that we have available in Gentoo Linux's
> tree distribution.

The next Git Rev News edition
(https://github.com/git/git.github.io/blob/master/rev_news/drafts/edition-68.md),
which should be published today, has a summary of a discussion thread
(https://lore.kernel.org/git/CAHpGcMJqmUmCR_u3fxSVppzwBo8_6RJG5-WLrXAWXm+OQ6_3bA@mail.gmail.com/)
related to this that you might find interesting.

Best,
Christian.

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2020-10-30  8:19 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-27 21:55 Bundles: Partial Clone & Shallow clone Robin H. Johnson
2020-10-30  8:19 ` Christian Couder

git@vger.kernel.org list mirror (unofficial, one of many)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 git git/ https://public-inbox.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.version-control.git
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.version-control.git
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git