Re: Reducing git size by building libgit.so

[Johannes Schindelin <Johannes.Schindelin@gmx.de>]

Hi Paul,

On Wed, 12 Jun 2019, Paul Smith wrote:

> On Tue, 2019-06-11 at 23:48 +0000, brian m. carlson wrote:
> > Also, some people install Git into their home directories, and a
> > shared library means that they'll have to use LD_LIBRARY_PATH (or
> > equivalent) to run Git.
>
> I don't have strong feeling about .so's although obviously less disk
> space used is always a good thing, everything else being equal.
>
> However, the above concern isn't actually an issue.  You can install
> the .so in a known location relative to the binaries, then link the
> binaries with an RPATH setting using $ORIGIN (or the equivalent on
> MacOS which does exist but I forget the name).

Hassles aside, you mentioned Linux and macOS. What about literally *all*
the other platforms we support? Like AIX, NonStop, HP/UX, etc?

Sure, you can hunt down all of them, and maybe even come up with a
workaround for platforms that do not have a $ORIGIN equivalent. You can
pile workaround on workaround all you want.

In the end, it seems to be a clear indicator that this is a complicator's
glove, and the only reasonably simple way forward would be to either leave
things as-are, or have an *opt-in* to build a shared libgit.

But.

And this is a really big but.

While you can try to document _all you want_ how libgit.so is not supposed
to be used as a library, how its API is not an API or at least not a
stable one, if you have _some_ experience with software development you
will know that it won't matter one bit. It _will_ be used, people _will_
complain, and it will turn out to simply not have been a good idea in the
first place.

> On Windows, DLLs are installed in the same directory as the binary,
> typically.
>
> Allowing relocatable binaries with .so dependencies without requiring
> LD_LIBRARY_PATH settings is a solved problem, to the best of my
> understanding.

You're probably right, as long as you restrict your view to mainstream
Operating Systems.

To put things into perspective, you might be interested in reading up on
https://github.com/git/git/commit/0f50c8e32c87 (Makefile: remove the
NO_R_TO_GCC_LINKER flag, 2019-05-17) and related commit history.

Sure, you could still argue that it is a "solved" problem. Where "solved"
is a different term than "desirable".

> One thing to think about is that runtime loading a .so can take some
> time if it has lots of public symbols.  If someone really wanted to do
> this, the ideal thing would be to make all symbols hidden except those
> needed by the binary front-ends and have those be very small shells
> that just had a very limited number of entry points into the .so.

That would fall squarely into the "pile on workaround on workaround"
category I mentioned above.

> Maybe for git this doesn't matter but for some projects I've worked on
> the time to dlopen() a library was a blocking issue that the above
> procedure solved nicely.

Sure, sometimes you cannot control whether it is an ill-designed `.so` you
need to consume.

As far as Git is concerned, this is not the case. At least when you look
at libgit.

When you look at libcurl, it is a different matter. But then, we do not
need to play RPATH games there: we expect it to be in the system's
preferred location.

BTW Duy hinted at problems with libcurl that made us split apart
`git-remote-https` from the main `git` executable. The full story is here:

1. The Linus complained about some "crazy" shared library loading behavior
   five months before Christmas 2009:

   https://public-inbox.org/git/alpine.LFD.2.01.0907241349390.3960@localhost.localdomain/

2. Daniel Barkalow was working on some "foreign VCS" support and thought
   that HTTPS/HTTP support could be handled via the same route, to avoid
   having to load libcurl for every Git operation no matter what:

   https://public-inbox.org/git/alpine.LNX.2.00.0907242242310.2147@iabervon.org/

3. Daniel then sent a patch series about two weeks later:

   https://public-inbox.org/git/alpine.LNX.2.00.0908050052390.2147@iabervon.org/

4. Those patches were accepted via cd03eebbfdae (Merge branch
   'db/vcs-helper', 2009-09-13)

So yes, I think that a patch or patch series to turn libgit.a into
libgit.so would need to be crafted *very* carefully, and _in the least_
offer a sound performance analysis in the commit messages.

It would obviously need to be proven beyond doubt that the startup time
does not deteriorate noticeably, otherwise the patch (series) would likely
be rejected.

Ciao,
Johannes