Why Git LFS is not a built-in feature

Why Git LFS is not a built-in feature

[Alireza]

Currently, having to set up git-lfs in each client and checking server
compatibility is a huge barrier for using it in the first place,
whilst it is generally a good practice to store large files in lfs.

As a consequence a lot of repos are not using it when they should.

Is there any reason that we don't have built-in support for such an
important feature?

Re: Why Git LFS is not a built-in feature

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2596 bytes --]

On 2020-11-13 at 09:45:52, Alireza wrote:
> Currently, having to set up git-lfs in each client and checking server
> compatibility is a huge barrier for using it in the first place,
> whilst it is generally a good practice to store large files in lfs.
> 
> As a consequence a lot of repos are not using it when they should.
> 
> Is there any reason that we don't have built-in support for such an
> important feature?

There are a couple reasons that it's not a built-in feature:

* First, there are several options in this space.  Git LFS is one,
  git-annex is another, and some people prefer to store large objects in
  the repository and use partial clone.  Git, as a project, tries to be
  flexible and meet the needs of various kinds of users without
  privileging one or another external tool.
* Git LFS is a complicated piece of software and it's currently written
  in Go, which is different from most of Git.  Re-implementing it in C
  would be burdensome, and there's little interest in maintaining Go
  software in the Git project.
* Git LFS uses a different protocol from Git, requiring additional
  configuration and a separate server-side component.
* The smudge and clean filter approach has some limitations, among them
  that users who don't have the external filter installed can commit
  uncleaned objects that then result in the working tree consistently
  being modified, even after git reset --hard.

It's my hope that the built-in support for partial clone will mature
enough to the point where that's a clear win and the need for external
tools isn't as great, since I think that will ultimately provide a
better experience for users.  Some people are already using it.  So in
some sense, we do have this as a built-in feature, maybe just not the
one you were expecting.

Additionally, in many cases, projects can avoid the need for storing
large files at all by using repository best practices, like not storing
build products or binary dependencies in the repository and instead
using an artifact server or a standard packaging system.  If possible,
that will almost always provide a better experience than any solution
for storing large files in the repository.

Finally, if you do want to use an external tool like Git LFS, it's
reasonably straightforward to specify a script to install and configure
the required dependencies for your project on each system so that
everything just works.  One popular location for this kind of path is
script/bootstrap.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Why Git LFS is not a built-in feature

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:
> Additionally, in many cases, projects can avoid the need for storing
> large files at all by using repository best practices, like not storing
> build products or binary dependencies in the repository and instead
> using an artifact server or a standard packaging system.  If possible,
> that will almost always provide a better experience than any solution
> for storing large files in the repository.

Well, I would argue that if the goal is ongoing archival and easy 
replication, then storing objects in a repository like git makes a lot 
more sense than keeping them on a central server that may or may not be 
there a few years down the line. Having large file support native in git 
is a laudable goal and I quite often wish that it existed.

-K

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Why Git LFS is not a built-in feature

[Ævar Arnfjörð Bjarmason]

On Sat, Nov 14 2020, Konstantin Ryabitsev wrote:

> On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:
>> Additionally, in many cases, projects can avoid the need for storing
>> large files at all by using repository best practices, like not storing
>> build products or binary dependencies in the repository and instead
>> using an artifact server or a standard packaging system.  If possible,
>> that will almost always provide a better experience than any solution
>> for storing large files in the repository.
>
> Well, I would argue that if the goal is ongoing archival and easy 
> replication, then storing objects in a repository like git makes a lot 
> more sense than keeping them on a central server that may or may not be 
> there a few years down the line. Having large file support native in git 
> is a laudable goal and I quite often wish that it existed.

That native support does exist right now in the form of partial clones,
the packfile-uris support, core.bigFileThreshold etc.

It's got a lot of rough edges currently, but if it's something you're
interested in you should try it out and see if the subset of features
that works well now is something that would work for you.

Re: Why Git LFS is not a built-in feature

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2187 bytes --]

On 2020-11-14 at 16:27:00, Konstantin Ryabitsev wrote:
> On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:
> > Additionally, in many cases, projects can avoid the need for storing
> > large files at all by using repository best practices, like not storing
> > build products or binary dependencies in the repository and instead
> > using an artifact server or a standard packaging system.  If possible,
> > that will almost always provide a better experience than any solution
> > for storing large files in the repository.
> 
> Well, I would argue that if the goal is ongoing archival and easy
> replication, then storing objects in a repository like git makes a lot
> more sense than keeping them on a central server that may or may not be
> there a few years down the line. Having large file support native in git
> is a laudable goal and I quite often wish that it existed.

Sure, and I think that's a different goal than the typical source code
or writing project (documentation, book, etc.) repository.  For example,
one can use Git repositories to do backups using the tool bup, which
actually works quite well but isn't a traditional use of Git.

The typical use case is that the user wants to store some reasonably
sized project on their local system and possibly also collaborate with
others for that, and with that goal, it makes sense to make the
repository not be absurdly large, since most developer systems don't
have tons of storage.  As Ævar and I mentioned, there are built-in
options for large files that make this use case more palatable with
native Git tooling.  But for this particular use case, it doesn't
logically make sense to store build assets, whether they're yours or
others', in this project repository.

If your goal is archival and replication, then a tool like bup might
meet your needs, or simply a large repository with many objects.  But in
that context, you'll likely have more storage, CPU, and memory available
to you and the need for large file support will look different (e.g.,
core.bigFileThreshold) or not be present at all.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Partial clone demo for large files (Re: Why Git LFS is not a built-in feature)

[Christian Couder]

On Sat, Nov 14, 2020 at 7:25 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Sat, Nov 14 2020, Konstantin Ryabitsev wrote:
>
> > On Sat, Nov 14, 2020 at 12:29:02AM +0000, brian m. carlson wrote:
> >> Additionally, in many cases, projects can avoid the need for storing
> >> large files at all by using repository best practices, like not storing
> >> build products or binary dependencies in the repository and instead
> >> using an artifact server or a standard packaging system.  If possible,
> >> that will almost always provide a better experience than any solution
> >> for storing large files in the repository.
> >
> > Well, I would argue that if the goal is ongoing archival and easy
> > replication, then storing objects in a repository like git makes a lot
> > more sense than keeping them on a central server that may or may not be
> > there a few years down the line. Having large file support native in git
> > is a laudable goal and I quite often wish that it existed.
>
> That native support does exist right now in the form of partial clones,
> the packfile-uris support, core.bigFileThreshold etc.
>
> It's got a lot of rough edges currently, but if it's something you're
> interested in you should try it out and see if the subset of features
> that works well now is something that would work for you.

I have been working on a partial clone demo that stores large files on
an HTTP server:

https://gitlab.com/chriscool/partial-clone-demo/-/blob/master/http-promisor/demo.txt

It has a lot of rough edges indeed. Fetching from the HTTP server
promisor remote is very slow. For the last part you need a hacked Git.
The scripts have a lot of bugs and limitations and are not finished
(to say the least).

The goal for now is just to give people (especially product managers,
developers and managers inside GitLab) an outlook about how it could
work.