git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: Christian Couder <christian.couder@gmail.com>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>, Alireza <rezaxm@gmail.com>
Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org>,
	"brian m. carlson" <sandals@crustytoothpaste.net>,
	git <git@vger.kernel.org>
Subject: Partial clone demo for large files (Re: Why Git LFS is not a built-in feature)
Date: Wed, 18 Nov 2020 11:20:20 +0100	[thread overview]
Message-ID: <CAP8UFD35kk10FpUnPpiAUzTHJbm=SJ-76OTmkTwBstGFe3Zgdw@mail.gmail.com> (raw)
In-Reply-To: <87blfzg5qa.fsf@evledraar.gmail.com>

On Sat, Nov 14, 2020 at 7:25 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Sat, Nov 14 2020, Konstantin Ryabitsev wrote:
>
> > On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:
> >> Additionally, in many cases, projects can avoid the need for storing
> >> large files at all by using repository best practices, like not storing
> >> build products or binary dependencies in the repository and instead
> >> using an artifact server or a standard packaging system.  If possible,
> >> that will almost always provide a better experience than any solution
> >> for storing large files in the repository.
> >
> > Well, I would argue that if the goal is ongoing archival and easy
> > replication, then storing objects in a repository like git makes a lot
> > more sense than keeping them on a central server that may or may not be
> > there a few years down the line. Having large file support native in git
> > is a laudable goal and I quite often wish that it existed.
>
> That native support does exist right now in the form of partial clones,
> the packfile-uris support, core.bigFileThreshold etc.
>
> It's got a lot of rough edges currently, but if it's something you're
> interested in you should try it out and see if the subset of features
> that works well now is something that would work for you.

I have been working on a partial clone demo that stores large files on
an HTTP server:

https://gitlab.com/chriscool/partial-clone-demo/-/blob/master/http-promisor/demo.txt

It has a lot of rough edges indeed. Fetching from the HTTP server
promisor remote is very slow. For the last part you need a hacked Git.
The scripts have a lot of bugs and limitations and are not finished
(to say the least).

The goal for now is just to give people (especially product managers,
developers and managers inside GitLab) an outlook about how it could
work.

  reply	other threads:[~2020-11-18 10:23 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-13  9:45 Why Git LFS is not a built-in feature Alireza
2020-11-14  0:29 ` brian m. carlson
2020-11-14 16:27   ` Konstantin Ryabitsev
2020-11-14 18:20     ` Ævar Arnfjörð Bjarmason
2020-11-18 10:20       ` Christian Couder [this message]
2020-11-14 19:15     ` brian m. carlson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAP8UFD35kk10FpUnPpiAUzTHJbm=SJ-76OTmkTwBstGFe3Zgdw@mail.gmail.com' \
    --to=christian.couder@gmail.com \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=konstantin@linuxfoundation.org \
    --cc=rezaxm@gmail.com \
    --cc=sandals@crustytoothpaste.net \
    --subject='Partial clone demo for large files (Re: Why Git LFS is not a built-in feature)' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).