git@vger.kernel.org mailing list mirror (one of many)
 help / Atom feed
From: Stefan Beller <sbeller@google.com>
To: Duy Nguyen <pclouds@gmail.com>
Cc: git <git@vger.kernel.org>
Subject: Re: submodule support in git-bundle
Date: Fri, 2 Nov 2018 10:08:51 -0700
Message-ID: <CAGZ79kbk0QdxTNWusieU=3DzTvxykmh-yTgNArTcM9USc5WWRw@mail.gmail.com> (raw)
In-Reply-To: <CACsJy8AVfkWsAP_-CNd5RSbQ=tyCzdw-ix1k5zwmAKaj+z1qXw@mail.gmail.com>

On Fri, Nov 2, 2018 at 9:10 AM Duy Nguyen <pclouds@gmail.com> wrote:
>
> I use git-bundle today and it occurs to me that if I want to use it to
> transfer part of a history that involves submodule changes, things
> aren't pretty. Has anybody given thought on how to do binary history
> transfer that contains changes from submodules?
>
> Since .bundle files are basically .pack files, i'm not sure if it's
> easy to bundle multiple pack files (one per repo)...

That is a really good discussion starter!

As bundles are modeled after the fetch protocol, I would
redirect the discussion there.

The new fetch protocol could support sending more than
one pack, which could be for both the superproject as
well as the relevant submodule updates (i.e. what is recorded
in the superproject) based on a new capability.

We at Google have given this idea some thought, but from a
different angle: As you may know currently Android uses the
repo tool, which we want to replace with Gits native submodules
eventually. The repo tool tests for each repository to clone if
there is a bundle file for that repository, such that instead of
cloning the repo, the bundle can be downloaded and then
a catch-up fetch can be performed. (This helps the Git servers
as well as the client, the bundle can be hosted on a CDN,
which is faster and cheaper than a git server for us).

So we've given some thought on extending the packfiles in the
fetch protocol to have some redirection to a CDN possible,
i.e. instead of sending bytes as is, you get more or less a "todo"
list, which might be
    (a) take the following bytes as is (current pack format)
    (b) download these other bytes from $THERE
        (possibly with a checksum)
once the stream of bytes is assembled, it will look like a regular
packfile with deltas etc.

This offloading-to-CDN (or "mostly resumable clone" in the
sense that the communication with the server is minimal, and
you get most of your data via resumable http range-requests)
sounds like complete offtopic, but is one of the requirements
for the repo to submodule migration, hence I came to speak of it.

Did you have other things in mind, on a higher level?
e.g. querying the bundle and creating submodule bundles
based off the superproject bundle? 'git bundle create' could
learn the --recurse-submodules option, which then produces
multiple bundle files without changing the file formats.

Stefan

  reply index

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-02 16:09 Duy Nguyen
2018-11-02 17:08 ` Stefan Beller [this message]
2018-11-02 18:34   ` Duy Nguyen
2018-11-02 19:00     ` Stefan Beller

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAGZ79kbk0QdxTNWusieU=3DzTvxykmh-yTgNArTcM9USc5WWRw@mail.gmail.com' \
    --to=sbeller@google.com \
    --cc=git@vger.kernel.org \
    --cc=pclouds@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org mailing list mirror (one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox