git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: Christian Couder <christian.couder@gmail.com>
To: git@vger.kernel.org
Cc: Junio C Hamano <gitster@pobox.com>, Jeff King <peff@peff.net>,
	Ben Peart <Ben.Peart@microsoft.com>,
	Jonathan Tan <jonathantanmy@google.com>,
	Jonathan Nieder <jrnieder@gmail.com>,
	Stefan Beller <sbeller@google.com>,
	Nguyen Thai Ngoc Duy <pclouds@gmail.com>,
	Mike Hommey <mh@glandium.org>,
	Lars Schneider <larsxschneider@gmail.com>,
	Eric Wong <e@80x24.org>,
	Christian Couder <chriscool@tuxfamily.org>,
	Jeff Hostetler <jeffhost@microsoft.com>,
	Eric Sunshine <sunshine@sunshineco.com>,
	Beat Bolli <dev+git@drbeat.li>
Subject: [PATCH v2 00/13] Many promisor remotes
Date: Tue, 22 Jan 2019 15:41:59 +0100	[thread overview]
Message-ID: <20190122144212.15119-1-chriscool@tuxfamily.org> (raw)

This path series is a follow up from the "remote odb" patch series
that I sent last year, which were a follow up from previous
series. See the links section for more information.

The goal of this patch series is to make it possible to have and to
fetch missing objects from multiple remotes instead of only one.

For now the fetch order is the order of the remotes in the config.

I selected the name "Promisor remote" over "Partial clone remote"
because it is shorter and because it is not only about cloning.

The existing extensions.partialclone is respected, but it is not
written in the config when a partial clone or fetch is made. Instead
remote.<name>.promisor is set to "true". This may create a
compatibility issue, but it makes it possible to start using many
promisor remotes by just cloning and fetching from different remotes
with partial clone filters. The compatibility issue could be resolved
in a future iteration by just setting extensions.partialclone instead
of remote.<name>.promisor the first time a promisor remote is used.

In general I have tried to change as few things as possible.

High level changes since the V1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  - added documentation patches
  
  - fixed issues when fetching from a new promisor remote
  
  - added a test to check that remote.<name>.promisor is set to "true"
    when fetching from a new promisor remote

  - renamed functions in promisor-remote.{c,h} to make the names more
    coherent and easier to understand

  - added 2 small typo fix patches that I sent separately

High level overview of this patch series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  - Patch 1/13 and 2/13:

These are 2 new small patches that I sent separately. I am adding them
here as the other patches were developed on top of them and it creates
a small conflict is these 2 patches are removed.

  - Patch 3/13:

This patch, which hasn't changed since V1, makes functions in
fetch-object.c return an error code, which is necessary to later tell
that they failed and try another promisor remote when there is more
than one. This could also just be seen as a fix to these functions.

  - Patch 4/13:

This introduces the minimum infrastructure for promisor remotes. There
has been a number of renames and improvements since V1.

  - Patch 5/13, 6/13 and 7/13:

This add a few missing bits in the promisor remote infrastructure that
will be needed in the following patches. There has been a number of
renames and improvements since V1.

  - Patch 8/13:

This replaces the previous interface to use only one promisor remote
defined in extensions.partialclone by the new interface created by the
previous patches. There has been a few renames since V1.

  - Patch 9/13:

This replaces the way a partial clone filter was handled by a new way
based on the previous patches that support more than one partial clone
filter. There has been a few renames since V1.

  - Patch 10/13:

This new patch removes the limitation in builtin/fetch.c to have only
one promisor remote.

  - Patch 11/13:

This adds test cases that shows that now more than one promisor remote
can be used and that remote.<name>.promisor is set to "true" when
fetching from a new promisor remote.

  - Patch 12/13 and 13/13:

These are new documentation patches, to explain how things can work
with more than one promisor remote.

Links
~~~~~

This patch series on GitHub:

V2: https://github.com/chriscool/git/commits/many-promisor-remotes
V1: https://github.com/chriscool/git/commits/many-promisor-remotes17

On the mailing list:

V1: https://public-inbox.org/git/20181211052746.16218-1-chriscool@tuxfamily.org/

This patch series is a follow up from the discussions related to
the remote odb V4 patch series:

https://public-inbox.org/git/20180802061505.2983-1-chriscool@tuxfamily.org/

Especially in:

https://public-inbox.org/git/CAP8UFD3nrhjANwNDqTwx5ZtnZNcnbAFqUN=u=LrvzuH4+3wQQA@mail.gmail.com/

I said that I would like to work on things in the following order:

  1) Teaching partial clone to attempt to fetch missing objects from
multiple remotes instead of only one using the order in the config.

  2) Simplifying the protocol for fetching missing objects so that it
can be satisfied by a lighter weight object storage system than a full
Git server.

  3) Making it possible to explicitly define an order in which the
remotes are accessed.

  4) Making the criteria for what objects can be missing more
aggressive, so that I can "git add" a large file and work with it
using Git without even having a second copy of that object in my local
object store.

And this patch series is about the 1).

The previous remote odb patch series on GitHub:

V5: https://github.com/chriscool/git/commits/remote-odb
V4: https://github.com/chriscool/git/commits/remote-odb5
V3: https://github.com/chriscool/git/commits/remote-odb3
V2: https://github.com/chriscool/git/commits/remote-odb2
V1: https://github.com/chriscool/git/commits/remote-odb1

Discussions related to previous versions of the odb patch series:

V4: https://public-inbox.org/git/20180802061505.2983-1-chriscool@tuxfamily.org/
V3: https://public-inbox.org/git/20180713174959.16748-1-chriscool@tuxfamily.org/
V2: https://public-inbox.org/git/20180630083542.20347-1-chriscool@tuxfamily.org/
V1: https://public-inbox.org/git/20180623121846.19750-1-chriscool@tuxfamily.org/


Christian Couder (13):
  fetch: fix extensions.partialclone name in error message
  partial-clone: add missing 'is' in doc
  fetch-object: make functions return an error code
  Add initial support for many promisor remotes
  promisor-remote: implement promisors_get_direct()
  promisor-remote: add promisor_remote_reinit()
  promisor-remote: use repository_format_partial_clone
  Use promisor_remote_get_direct() and has_promisor_remote()
  promisor-remote: parse remote.*.partialclonefilter
  builtin/fetch: remove unique promisor remote limitation
  t0410: test fetching from many promisor remotes
  partial-clone: add multiple remotes in the doc
  remote: add promisor and partial clone config to the doc

 Documentation/config/remote.txt           |   8 ++
 Documentation/technical/partial-clone.txt |  85 +++++++++----
 Makefile                                  |   1 +
 builtin/cat-file.c                        |   5 +-
 builtin/fetch.c                           |  31 ++---
 builtin/gc.c                              |   3 +-
 builtin/repack.c                          |   3 +-
 cache-tree.c                              |   3 +-
 connected.c                               |   3 +-
 fetch-object.c                            |  13 +-
 fetch-object.h                            |   4 +-
 list-objects-filter-options.c             |  51 ++++----
 list-objects-filter-options.h             |   3 +-
 packfile.c                                |   3 +-
 promisor-remote.c                         | 148 ++++++++++++++++++++++
 promisor-remote.h                         |  22 ++++
 sha1-file.c                               |  14 +-
 t/t0410-partial-clone.sh                  |  49 ++++++-
 t/t5601-clone.sh                          |   3 +-
 t/t5616-partial-clone.sh                  |   4 +-
 unpack-trees.c                            |   6 +-
 21 files changed, 365 insertions(+), 97 deletions(-)
 create mode 100644 promisor-remote.c
 create mode 100644 promisor-remote.h

-- 
2.20.1.322.gd6b9ae60d4


             reply	other threads:[~2019-01-22 14:42 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-22 14:41 Christian Couder [this message]
2019-01-22 14:42 ` [PATCH v2 01/13] fetch: fix extensions.partialclone name in error message Christian Couder
2019-01-22 14:42 ` [PATCH v2 02/13] partial-clone: add missing 'is' in doc Christian Couder
2019-01-22 14:42 ` [PATCH v2 03/13] fetch-object: make functions return an error code Christian Couder
2019-01-22 14:42 ` [PATCH v2 04/13] Add initial support for many promisor remotes Christian Couder
2019-01-22 14:42 ` [PATCH v2 05/13] promisor-remote: implement promisors_get_direct() Christian Couder
2019-01-22 14:42 ` [PATCH v2 06/13] promisor-remote: add promisor_remote_reinit() Christian Couder
2019-01-22 14:42 ` [PATCH v2 07/13] promisor-remote: use repository_format_partial_clone Christian Couder
2019-01-22 14:42 ` [PATCH v2 08/13] Use promisor_remote_get_direct() and has_promisor_remote() Christian Couder
2019-01-22 14:42 ` [PATCH v2 09/13] promisor-remote: parse remote.*.partialclonefilter Christian Couder
2019-01-22 14:42 ` [PATCH v2 10/13] builtin/fetch: remove unique promisor remote limitation Christian Couder
2019-01-22 14:42 ` [PATCH v2 11/13] t0410: test fetching from many promisor remotes Christian Couder
2019-01-22 14:42 ` [PATCH v2 12/13] partial-clone: add multiple remotes in the doc Christian Couder
2019-01-22 14:42 ` [PATCH v2 13/13] remote: add promisor and partial clone config to " Christian Couder

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190122144212.15119-1-chriscool@tuxfamily.org \
    --to=christian.couder@gmail.com \
    --cc=Ben.Peart@microsoft.com \
    --cc=chriscool@tuxfamily.org \
    --cc=dev+git@drbeat.li \
    --cc=e@80x24.org \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jeffhost@microsoft.com \
    --cc=jonathantanmy@google.com \
    --cc=jrnieder@gmail.com \
    --cc=larsxschneider@gmail.com \
    --cc=mh@glandium.org \
    --cc=pclouds@gmail.com \
    --cc=peff@peff.net \
    --cc=sbeller@google.com \
    --cc=sunshine@sunshineco.com \
    --subject='Re: [PATCH v2 00/13] Many promisor remotes' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org list mirror (unofficial, one of many)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 git git/ https://public-inbox.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.version-control.git
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.version-control.git
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git