git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: Christian Couder <christian.couder@gmail.com>
To: git@vger.kernel.org
Cc: Junio C Hamano <gitster@pobox.com>, Jeff King <peff@peff.net>,
	Ben Peart <Ben.Peart@microsoft.com>,
	Jonathan Tan <jonathantanmy@google.com>,
	Nguyen Thai Ngoc Duy <pclouds@gmail.com>,
	Mike Hommey <mh@glandium.org>,
	Lars Schneider <larsxschneider@gmail.com>,
	Eric Wong <e@80x24.org>,
	Christian Couder <chriscool@tuxfamily.org>,
	Jeff Hostetler <jeffhost@microsoft.com>
Subject: [PATCH v2 00/36] Promisor remotes and external ODB support
Date: Mon, 19 Mar 2018 14:31:11 +0100	[thread overview]
Message-ID: <20180319133147.15413-1-chriscool@tuxfamily.org> (raw)

This is a follow up from an early patch series at the beginning of the
year that started to integrate the jh/fsck-promisors patch series with
the external odb patch series.

A lot of things are different because the jh/fsck-promisors and
jh/partial-clone have been merged into master since the v1. So the
integration is much more complete now (though not fully complete), and
this integration happens quite early in the patch series.

This integration makes it possible to have many promisor and partial
clone remotes (instead of just one) with possibly different partial
clone filters, though that is not tested yet.

I am not sure that the "external odb" name is still the best, as the
promisor remotes are not quite external. So I am open to suggestions
about a new name.

This patch series does not include the last part of the original
external odb series which was about adding an `--inital-refspec`
option to `git clone`, as this might not be needed anymore and this
current series is long and complex enough.

High level overview of this patch series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In general the external odb tests are now numbered 05XX instead of
04XX, so that they don't conflict with promisor objects/partial clone
tests. Also there are a lot of changes since v1, so I don't think an
interdiff would be useful.

  - Patches 1/36 to 12/36:
  
These patches implement the external odb infrastructure as well as the
support for promisor objects and partial clone.

  - Patches 13/36 and 14/36:

These patches move over the promisor objects and partial clone code to
use the external odb mechanism. The result of 13/36 is that instead of
the "extensions.partialclone" config variable, a
"odb.<name>.promisorRemote" config variable is now used. The result of
14/36 is that instead of the "core.partialclonefilter" config
variable, a "odb.<name>.partialclonefilter" config variable is now
used.

  - Patches 15/36 to 36/36:

These patches implement the rest of the external odb mechanism. Patch
36/36 is a documentation patch that explains how this mechanism work,
though this patch needs to be updated.

Future work
~~~~~~~~~~~

  - add tests that show that one can now use more than one promisor remote
  - add fsck, gc and repack tests for script and process modes
  - see if partial clone filters could be used for other modes
  - update documentation patch
  - fix memory leaks
  - take another look at prepare_helper_command() [1]
  - take a look at changing the order of the have object lines [1]
  - take a look at what happens when a server creates a pack [2]
  - add tests with different kinds of external odbs

[1] https://public-inbox.org/git/ebf67bcc-3e17-3fda-9f56-dd152e7bf3af@jeffhostetler.com/
[2] https://public-inbox.org/git/8de3597a-01bd-a721-fffb-20769701d0af@jeffhostetler.com/

Links
~~~~~

This patch series on GitHub:

https://github.com/chriscool/git/commits/gl-small-promisor-external-odb-next

Peff started to work on external odbs some years ago:

http://thread.gmane.org/gmane.comp.version-control.git/206886/focus=207040
http://thread.gmane.org/gmane.comp.version-control.git/247171
http://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020

His work, which is not compile-tested any more:

https://github.com/peff/git/commits/jk/external-odb-wip

Initial discussions about external odbs:

http://thread.gmane.org/gmane.comp.version-control.git/288151/focus=295160

Version 1, 2, 3, 4, 5 and 6 of the external odbs series:

https://public-inbox.org/git/20160613085546.11784-1-chriscool@tuxfamily.org/
https://public-inbox.org/git/20160628181933.24620-1-chriscool@tuxfamily.org/
https://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/
https://public-inbox.org/git/20170620075523.26961-1-chriscool@tuxfamily.org/
https://public-inbox.org/git/20170803091926.1755-1-chriscool@tuxfamily.org/
https://public-inbox.org/git/20170916080731.13925-1-chriscool@tuxfamily.org/

Some of the discussions related to Ben Peart's work that is used by
this series:

https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/
https://public-inbox.org/git/20170322165220.5660-1-benpeart@microsoft.com/
https://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/

Version 1, 2, 3, 4, 5 and 6 of the external odbs series on GitHub:

https://github.com/chriscool/git/commits/gl-external-odb12
https://github.com/chriscool/git/commits/gl-external-odb22
https://github.com/chriscool/git/commits/gl-external-odb61
https://github.com/chriscool/git/commits/gl-external-odb239
https://github.com/chriscool/git/commits/gl-external-odb373
https://github.com/chriscool/git/commits/gl-external-odb411

Version 1 of this "Promisor remotes and external ODB support" series:

https://public-inbox.org/git/20180103163403.11303-1-chriscool@tuxfamily.org/

Version 1 of this "Promisor remotes and external ODB support" series on GitHub:

https://github.com/chriscool/git/commits/gl-small-promisor-external-odb12


Ben Peart (1):
  Add t0550 to test 'get_direct' mechanism

Christian Couder (35):
  Add initial external odb support
  sha1_file: add prepare_external_alt_odb()
  Add GIT_NO_EXTERNAL_ODB env variable
  external-odb: add has_external_odb()
  external-odb: implement external_odb_get_direct
  sha1_file: prepare for external odbs
  odb-helper: add 'enum odb_helper_type'
  external-odb: add external_odb_reinit()
  external-odb: add script mode support
  odb-helper: add 'script_mode' to 'struct odb_helper'
  odb-helper: add odb_helper_init() to send 'init' instruction
  external-odb: add external_odb_get_many_direct()
  Use external_odb_get_direct() and has_external_odb()
  Use odb.origin.partialclonefilter instead of core.partialclonefilter
  external-odb: add 'get_direct' support
  t0500: add 'put_raw_obj' instruction to odb-helper script
  external odb: add 'put_raw_obj' support
  external-odb: accept only blobs for now
  t0500: add test for external odb write support
  Add t0510 to test external ODB transfer
  lib-httpd: pass config file to start_httpd()
  lib-httpd: add upload.sh
  lib-httpd: add list.sh
  lib-httpd: add apache-e-odb.conf
  odb-helper: add odb_helper_get_raw_object()
  pack-objects: don't pack objects in external odbs
  Add t0520 to test transfer to HTTP external odb
  odb-helper: add init_object_process()
  Add t0560 to test passing git objects
  odb-helper: add put_object_process()
  Add t0570 to test passing raw objects
  odb-helper: add have_object_process()
  Add t0580 to test "have" capability and raw objects
  external-odb: use 'odb=magic' attribute to mark odb blobs
  Add Documentation/technical/external-odb.txt

 Documentation/technical/external-odb.txt |  342 +++++++
 Makefile                                 |    2 +
 builtin/cat-file.c                       |    5 +-
 builtin/fetch.c                          |   13 +-
 builtin/gc.c                             |    3 +-
 builtin/pack-objects.c                   |    4 +
 builtin/repack.c                         |    3 +-
 cache.h                                  |   20 +-
 connected.c                              |    3 +-
 environment.c                            |    5 +-
 external-odb.c                           |  250 +++++
 external-odb.h                           |   25 +
 list-objects-filter-options.c            |   46 +-
 list-objects-filter-options.h            |    3 +-
 odb-helper.c                             | 1109 ++++++++++++++++++++++
 odb-helper.h                             |   49 +
 packfile.c                               |    3 +-
 setup.c                                  |    7 +-
 sha1_file.c                              |  120 ++-
 t/lib-httpd.sh                           |    8 +-
 t/lib-httpd/apache-e-odb.conf            |  214 +++++
 t/lib-httpd/list.sh                      |   41 +
 t/lib-httpd/upload.sh                    |   45 +
 t/t0410-partial-clone.sh                 |   30 +-
 t/t0500-external-odb.sh                  |   83 ++
 t/t0510-transfer-e-odb.sh                |  147 +++
 t/t0520-transfer-http-e-odb.sh           |  145 +++
 t/t0550-read-object.sh                   |   28 +
 t/t0550/read-object                      |   68 ++
 t/t0560-read-object-git.sh               |   28 +
 t/t0560/read-object-git                  |   78 ++
 t/t0570-read-object-http-e-odb.sh        |  112 +++
 t/t0570/read-object-plain                |   83 ++
 t/t0580-read-object-have-http-e-odb.sh   |  112 +++
 t/t0580/read-object-plain-have           |  103 ++
 t/t5500-fetch-pack.sh                    |    4 +-
 t/t5601-clone.sh                         |    2 +-
 t/t5616-partial-clone.sh                 |    4 +-
 unpack-trees.c                           |    6 +-
 39 files changed, 3254 insertions(+), 99 deletions(-)
 create mode 100644 Documentation/technical/external-odb.txt
 create mode 100644 external-odb.c
 create mode 100644 external-odb.h
 create mode 100644 odb-helper.c
 create mode 100644 odb-helper.h
 create mode 100644 t/lib-httpd/apache-e-odb.conf
 create mode 100644 t/lib-httpd/list.sh
 create mode 100644 t/lib-httpd/upload.sh
 create mode 100755 t/t0500-external-odb.sh
 create mode 100755 t/t0510-transfer-e-odb.sh
 create mode 100755 t/t0520-transfer-http-e-odb.sh
 create mode 100755 t/t0550-read-object.sh
 create mode 100755 t/t0550/read-object
 create mode 100755 t/t0560-read-object-git.sh
 create mode 100755 t/t0560/read-object-git
 create mode 100755 t/t0570-read-object-http-e-odb.sh
 create mode 100755 t/t0570/read-object-plain
 create mode 100755 t/t0580-read-object-have-http-e-odb.sh
 create mode 100755 t/t0580/read-object-plain-have

-- 
2.17.0.rc0.37.g8f476fabe9


             reply	other threads:[~2018-03-19 13:32 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-03-19 13:31 Christian Couder [this message]
2018-03-19 13:31 ` [PATCH v2 01/36] Add initial external odb support Christian Couder
2018-03-19 13:31 ` [PATCH v2 02/36] sha1_file: add prepare_external_alt_odb() Christian Couder
2018-03-19 13:31 ` [PATCH v2 03/36] Add GIT_NO_EXTERNAL_ODB env variable Christian Couder
2018-03-19 13:31 ` [PATCH v2 04/36] external-odb: add has_external_odb() Christian Couder
2018-03-19 13:31 ` [PATCH v2 05/36] external-odb: implement external_odb_get_direct Christian Couder
2018-03-19 13:31 ` [PATCH v2 06/36] sha1_file: prepare for external odbs Christian Couder
2018-03-19 13:31 ` [PATCH v2 07/36] odb-helper: add 'enum odb_helper_type' Christian Couder
2018-03-19 13:31 ` [PATCH v2 08/36] external-odb: add external_odb_reinit() Christian Couder
2018-03-19 13:31 ` [PATCH v2 09/36] external-odb: add script mode support Christian Couder
2018-03-19 13:31 ` [PATCH v2 10/36] odb-helper: add 'script_mode' to 'struct odb_helper' Christian Couder
2018-03-19 13:31 ` [PATCH v2 11/36] odb-helper: add odb_helper_init() to send 'init' instruction Christian Couder
2018-03-19 13:31 ` [PATCH v2 12/36] external-odb: add external_odb_get_many_direct() Christian Couder
2018-03-19 13:31 ` [PATCH v2 13/36] Use external_odb_get_direct() and has_external_odb() Christian Couder
2018-03-19 13:31 ` [PATCH v2 14/36] Use odb.origin.partialclonefilter instead of core.partialclonefilter Christian Couder
2018-03-19 13:31 ` [PATCH v2 15/36] external-odb: add 'get_direct' support Christian Couder
2018-03-19 13:31 ` [PATCH v2 16/36] t0500: add 'put_raw_obj' instruction to odb-helper script Christian Couder
2018-03-19 13:31 ` [PATCH v2 17/36] external odb: add 'put_raw_obj' support Christian Couder
2018-03-19 13:31 ` [PATCH v2 18/36] external-odb: accept only blobs for now Christian Couder
2018-03-19 13:31 ` [PATCH v2 19/36] t0500: add test for external odb write support Christian Couder
2018-03-19 13:31 ` [PATCH v2 20/36] Add t0510 to test external ODB transfer Christian Couder
2018-03-19 13:31 ` [PATCH v2 21/36] lib-httpd: pass config file to start_httpd() Christian Couder
2018-03-19 13:31 ` [PATCH v2 22/36] lib-httpd: add upload.sh Christian Couder
2018-03-19 13:31 ` [PATCH v2 23/36] lib-httpd: add list.sh Christian Couder
2018-03-19 13:31 ` [PATCH v2 24/36] lib-httpd: add apache-e-odb.conf Christian Couder
2018-03-19 13:31 ` [PATCH v2 25/36] odb-helper: add odb_helper_get_raw_object() Christian Couder
2018-03-19 13:31 ` [PATCH v2 26/36] pack-objects: don't pack objects in external odbs Christian Couder
2018-03-19 13:31 ` [PATCH v2 27/36] Add t0520 to test transfer to HTTP external odb Christian Couder
2018-03-19 13:31 ` [PATCH v2 28/36] odb-helper: add init_object_process() Christian Couder
2018-03-19 13:31 ` [PATCH v2 29/36] Add t0550 to test 'get_direct' mechanism Christian Couder
2018-03-19 13:31 ` [PATCH v2 30/36] Add t0560 to test passing git objects Christian Couder
2018-03-19 13:31 ` [PATCH v2 31/36] odb-helper: add put_object_process() Christian Couder
2018-03-19 13:31 ` [PATCH v2 32/36] Add t0570 to test passing raw objects Christian Couder
2018-03-19 13:31 ` [PATCH v2 33/36] odb-helper: add have_object_process() Christian Couder
2018-03-19 13:31 ` [PATCH v2 34/36] Add t0580 to test "have" capability and raw objects Christian Couder
2018-03-19 13:31 ` [PATCH v2 35/36] external-odb: use 'odb=magic' attribute to mark odb blobs Christian Couder
2018-03-19 13:31 ` [PATCH v2 36/36] Add Documentation/technical/external-odb.txt Christian Couder
2018-03-19 19:13 ` [PATCH v2 00/36] Promisor remotes and external ODB support Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180319133147.15413-1-chriscool@tuxfamily.org \
    --to=christian.couder@gmail.com \
    --cc=Ben.Peart@microsoft.com \
    --cc=chriscool@tuxfamily.org \
    --cc=e@80x24.org \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jeffhost@microsoft.com \
    --cc=jonathantanmy@google.com \
    --cc=larsxschneider@gmail.com \
    --cc=mh@glandium.org \
    --cc=pclouds@gmail.com \
    --cc=peff@peff.net \
    --subject='Re: [PATCH v2 00/36] Promisor remotes and external ODB support' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).