libgit2 status

libgit2 status

[greened]

What is the status of libgit2 WRT the overall git project?  I recall
that there was some discussion of basing bits of git on libgit2 once it
matures.

I ask because I'm starting a project to improve the abysmal speed of
git-subtree split.  It's unbearably slow at the moment and as far as I
can puzzle out it's due almost entirely to repeated subshell invocations
to run git commands.

I was planning on doing some experiments rewriting bits of git-subtree
using libgit2 but I want to make sure that that isn't wasted work.  It
appears to be exactly what I need to code bits of git-subtree natively.

Thoughts?

                        -Dave

Re: libgit2 status

[Andreas Ericsson]

On 08/24/2012 04:02 PM, greened@obbligato.org wrote:
> What is the status of libgit2 WRT the overall git project?  I recall
> that there was some discussion of basing bits of git on libgit2 once it
> matures.
> 
> I ask because I'm starting a project to improve the abysmal speed of
> git-subtree split.  It's unbearably slow at the moment and as far as I
> can puzzle out it's due almost entirely to repeated subshell invocations
> to run git commands.
> 
> I was planning on doing some experiments rewriting bits of git-subtree
> using libgit2 but I want to make sure that that isn't wasted work.  It
> appears to be exactly what I need to code bits of git-subtree natively.
> 
> Thoughts?
> 

libgit2 is now maintained by Vicent Marti, who was once a gsoc student.
He's employed by github and seems to spend most of his time working on
libgit2.

Politically, I'm not sure how keen the git community is on handing
over control to the core stuff of git to a commercial entity, but it
doesn't seem to be a dying project, so I'd say go ahead and do it.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: libgit2 status

[Vicent Marti]

On Sat, Aug 25, 2012 at 2:56 AM, Andreas Ericsson <ae@op5.se> wrote:
> Politically, I'm not sure how keen the git community is on handing
> over control to the core stuff of git to a commercial entity,

The development of libgit2 happens 100% in the open. I don't know what
"commercial entity" are you talking about, but there are several
companies and independent contributors working on the Library at the
moment.

Re: libgit2 status

[Nicolas Sebrecht]

The 25/08/12, Vicent Marti wrote:

> The development of libgit2 happens 100% in the open. I don't know what
> "commercial entity" are you talking about, but there are several
> companies and independent contributors working on the Library at the
> moment.

Right but as far as I'm aware of Junio had reserves about libgit2
integration into git due to issues making repositories broken. Though,
having libgit2 as git core would make libgit2 the the-facto standard
which would a *very* big plus.

Also, I guess that integration into git would mean more developers
contibuting for libgit2. Currently, issues seems to be a blocker for
integration. So, libgit2 might appear to be a marginal/risky alternative
for a long time which is sad.

[ I'm somewhat in the same situation of OP. ]

-- 
Nicolas Sebrecht

Re: libgit2 status

[Carlos Martín Nieto]

Nicolas Sebrecht <nicolas.s.dev@gmx.fr> writes:

> The 25/08/12, Vicent Marti wrote:
>
>> The development of libgit2 happens 100% in the open. I don't know what
>> "commercial entity" are you talking about, but there are several
>> companies and independent contributors working on the Library at the
>> moment.
>
> Right but as far as I'm aware of Junio had reserves about libgit2
> integration into git due to issues making repositories broken. Though,

The comment I saw about that was that at one point libgit2 had produced
broken trees; which is true, the algorithm for the almost-alphanumeric
sorting was slightly broken. This was fixed quite some time ago, which
he also mentioned in the same message.

> [ I'm somewhat in the same situation of OP. ]

If you wait for it to be perfect, it's never going to happen. If your
application would benefit, port it to libgit2 and report the issues you
find. That's the only way we can know of the odd edge cases and
improvements that we should make.

Note that the GitHub apps for Mac and Windows both use the Library to
perform parts of their job. Their new backend for the website is also
(going to be) based on libgit2.

I am also working on a project for a client involving the Library for
importing data and the only problem we've had is that we discovered an
edge case regarding symlinks and an assumption that one of the bindings
made wrt diffs, which is getting fixed.

   cmn

Re: libgit2 status

[Elia Pinto]

I know julio notes about libgit2. Anyway the rpm5 mantainer had
decided to integrate libgit2 recently. Jfi.

Regards

2012/8/25, Nicolas Sebrecht <nicolas.s.dev@gmx.fr>:
> The 25/08/12, Vicent Marti wrote:
>
>> The development of libgit2 happens 100% in the open. I don't know what
>> "commercial entity" are you talking about, but there are several
>> companies and independent contributors working on the Library at the
>> moment.
>
> Right but as far as I'm aware of Junio had reserves about libgit2
> integration into git due to issues making repositories broken. Though,
> having libgit2 as git core would make libgit2 the the-facto standard
> which would a *very* big plus.
>
> Also, I guess that integration into git would mean more developers
> contibuting for libgit2. Currently, issues seems to be a blocker for
> integration. So, libgit2 might appear to be a marginal/risky alternative
> for a long time which is sad.
>
> [ I'm somewhat in the same situation of OP. ]
>
> --
> Nicolas Sebrecht
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
Inviato dal mio dispositivo mobile

Re: libgit2 status

[Junio C Hamano]

Andreas Ericsson <ae@op5.se> writes:

> Politically, I'm not sure how keen the git community is on handing
> over control to the core stuff of git to a commercial entity, but it
> doesn't seem to be a dying project, so I'd say go ahead and do it.

I do not think commercial-ness of any entity comes into the picture.

The only three things that matter are license compatibility (I think
libgit2 licensed under GPLv2 + linkage exception is doing just fine
in that department), maturity and quality of it (it is in early
development phase), and the openness of the development process (it
could do better by finding ways to better interact with the
mainstream git development discussion that happens here in the
longer term).

And the last one should really be a "longer term" item.  It is more
important for its codebase to get mature and robust, and that can
only happen by various projects and products (e.g. GitHub for Mac)
using it to improve it.  I do not think "subtree" (or anything in
contrib/ for that matter) is part of "the core stuff of git", and do
not see a problem; such a move may help both subtree and libgit2.

Over a much longer timeperiod, I wouldn't be surprised if some "core
stuff" gets reimplemented on top of libgit2 and distributed as part
of the git-core.

There will be substantial integration and logistics hassles ahead of
us before that can happen, though.  E.g.  we could point at libgit2
as our submodule, but that is not the only way to make git depend on
libgit2; it could just be a Build-Depends like we depend on libz.
Looking at the build dependency of libgit2 itself, I do not think
tighter integration of the libgit2 itself into the git-core is not
likely to happen very soon, and also is not necessarily a good thing
to do.

Re: libgit2 status

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Looking at the build dependency of libgit2 itself, I do not think
> tighter integration of the libgit2 itself into the git-core is not
> likely to happen very soon, and also is not necessarily a good thing
> to do.

Obviously I meant "I think it is not likely to happen and is not
necessaryly a good thing".  Dumb double-negatives.

Re: libgit2 status

[dag]

Junio C Hamano <gitster@pobox.com> writes:

> And the last one should really be a "longer term" item.  It is more
> important for its codebase to get mature and robust, and that can
> only happen by various projects and products (e.g. GitHub for Mac)
> using it to improve it.  I do not think "subtree" (or anything in
> contrib/ for that matter) is part of "the core stuff of git", and do
> not see a problem; such a move may help both subtree and libgit2.
>
> Over a much longer timeperiod, I wouldn't be surprised if some "core
> stuff" gets reimplemented on top of libgit2 and distributed as part
> of the git-core.

I am hoping to move git-subtree into core once it performs a little
better and I've fixed a couple of bugs.  Will basing it on libgit2 delay
that process significantly?  Six months delay is no problem.  2 years
would be problematic.

I would be happy to be a guinea pig for libgit2 in order to improve it,
but I don't want to significantly impact git-subtree's move to core.
I'll have to figure out the right balance there given feedback.

                         -Dave

Re: libgit2 status

[Junio C Hamano]

<dag@cray.com> writes:

> I am hoping to move git-subtree into core once it performs a little
> better and I've fixed a couple of bugs.  Will basing it on libgit2 delay
> that process significantly?  Six months delay is no problem.  2 years
> would be problematic.
>
> I would be happy to be a guinea pig for libgit2 in order to improve it,
> but I don't want to significantly impact git-subtree's move to core.
> I'll have to figure out the right balance there given feedback.

I expect it will take some time for libgit2 to allow our Makefile to
start saying "LDFLAGS += -libgit2"; it will need to become as stable
and widespread as other libraries we depend on, e.g. -lz and -lcurl.

Re: libgit2 status

[dag]

Junio C Hamano <gitster@pobox.com> writes:

>> I would be happy to be a guinea pig for libgit2 in order to improve it,
>> but I don't want to significantly impact git-subtree's move to core.
>> I'll have to figure out the right balance there given feedback.
>
> I expect it will take some time for libgit2 to allow our Makefile to
> start saying "LDFLAGS += -libgit2"; it will need to become as stable
> and widespread as other libraries we depend on, e.g. -lz and -lcurl.

Well that's a chicken-and-egg problem, isn't it.  How will a library
become widespread unless something uses it?

Would it be enough to have libgit2 as an installable package in the
major distributions?

                                 -Dave

Re: libgit2 status

[Junio C Hamano]

<dag@cray.com> writes:

> Junio C Hamano <gitster@pobox.com> writes:
>
>>> I would be happy to be a guinea pig for libgit2 in order to improve it,
>>> but I don't want to significantly impact git-subtree's move to core.
>>> I'll have to figure out the right balance there given feedback.
>>
>> I expect it will take some time for libgit2 to allow our Makefile to
>> start saying "LDFLAGS += -libgit2"; it will need to become as stable
>> and widespread as other libraries we depend on, e.g. -lz and -lcurl.
>
> Well that's a chicken-and-egg problem, isn't it.  How will a library
> become widespread unless something uses it?

That something will not be the git core itself.  Otherwise we will
lose a stable reference implementation to catch its bugs.

Re: libgit2 status

[dag]

Junio C Hamano <gitster@pobox.com> writes:

>> Well that's a chicken-and-egg problem, isn't it.  How will a library
>> become widespread unless something uses it?
>
> That something will not be the git core itself.  Otherwise we will
> lose a stable reference implementation to catch its bugs.

Well, the whole question here is whether git-subtree can become part of
core if it is based on libgit2.  It boils down to what you mean by
"widespread," I guess.  Does "widespread" mean "available as a package
in major distributions," "installed by default in major distributions"
or something else?

                                 -Dave

Re: libgit2 status

[Nicolas Sebrecht]

The 27/08/12, Junio C Hamano wrote:
> <dag@cray.com> writes:
> 
> > Junio C Hamano <gitster@pobox.com> writes:
> >
> >>> I would be happy to be a guinea pig for libgit2 in order to improve it,
> >>> but I don't want to significantly impact git-subtree's move to core.
> >>> I'll have to figure out the right balance there given feedback.
> >>
> >> I expect it will take some time for libgit2 to allow our Makefile to
> >> start saying "LDFLAGS += -libgit2"; it will need to become as stable
> >> and widespread as other libraries we depend on, e.g. -lz and -lcurl.
> >
> > Well that's a chicken-and-egg problem, isn't it.  How will a library
> > become widespread unless something uses it?
> 
> That something will not be the git core itself.  Otherwise we will
> lose a stable reference implementation to catch its bugs.

This is exactly what I'm most afraid about. I tend to think it's a
chicken-and-egg problem, too.

Do you expect one big merge of a very stable libgit2 at some point?
Because, asking others to implement widely used tools on top of libgit2
in order to have it as stable/tested as curl is not going to happen, IMHO.

Otherwise, what about going with this optionnal "LDFLAGS += -libgit2"
ASAP with good disclaimer that it's only intended for development and
testing purpose? Then, git-core could slowly rely on functions of
libgit2, one after the other.


-- 
Nicolas Sebrecht

Re: libgit2 status

[dag]

Nicolas Sebrecht <nicolas.s.dev@gmx.fr> writes:

> Do you expect one big merge of a very stable libgit2 at some point?

I don't think there's any need to merge libgit2 into the git project
source.  As a library, it should be perfectly usable as a project of its
own, just like libcurl and libz.

> Otherwise, what about going with this optionnal "LDFLAGS += -libgit2"
> ASAP with good disclaimer that it's only intended for development and
> testing purpose? Then, git-core could slowly rely on functions of
> libgit2, one after the other.

This makes a lot of sense to me.

                            -David

Re: libgit2 status

[Junio C Hamano]

<dag@cray.com> writes:

> Nicolas Sebrecht <nicolas.s.dev@gmx.fr> writes:
>
>> Do you expect one big merge of a very stable libgit2 at some point?
>
> I don't think there's any need to merge libgit2 into the git project
> source.  As a library, it should be perfectly usable as a project of its
> own, just like libcurl and libz.
>
>> Otherwise, what about going with this optionnal "LDFLAGS += -libgit2"
>> ASAP with good disclaimer that it's only intended for development and
>> testing purpose? Then, git-core could slowly rely on functions of
>> libgit2, one after the other.
>
> This makes a lot of sense to me.

As I already said in my earlier message in $gmane/204305, I wouldn't
be surprised if some "core stuff" gets reimplemented on top of
libgit2 and distributed as part of the git-core.  But at this
moment, I see that just as a blip of possibility far in the future.

It would most likely start slowly, by adding lg-client/cat-file.c
that is linked with libgit2 when "make USE_LIBGIT2=YesPlease" was
asked for, and we compile the tried-and-true builtin/cat-file.c
otherwise ("cat-file" may actually not the most trivial first step,
though, but I think readers get the idea).

Even after most if not all commands have counterparts reimplemented
on libgit2, I do not think we can afford to drop any of the original
for a long time.  For that to happen, at the very least:

 - libgit2 must be available in major distros so that people can say
   "[yum|apt-get] install libgit2-dev"; and

 - the version of it packaged for major distros are recent and
   stable enough, so that we never have to say "distro X ships with
   libgit2 that is too old, so people on that distro have to compile
   it from the source."

which is the quality we expect from libraries we would depend on,
like -lz, -lcurl, etc.

It is OK if we have to conditionally compile out some of our code in
the periphery when libgit2 that is available on the platform is too
old (we allow it for -lcurl already).

But all of the above assumes one thing: the reimplementated result
does not suck ;-) which is a large unknown.

Re: libgit2 status

[Thiago Farina]

Just an addendum, the libfication would come from within the core git
developers, i.e, they start making the git library inside git itself.
The git project would provide the API, not an external project.

With some structure like:

include/git.h
src/git.c

...

whatever.

And doing the way Jeff did with argv_array for example and with rigid
style guide defining how new APIs should be written. Just my 2 cents.
Don't take it to seriously.

Re: libgit2 status

[Ramkumar Ramachandra]

Thiago Farina wrote:
> [...]
> With some structure like:
>
> include/git.h
> src/git.c
>
> ...
>
> whatever.
> [...]

Junio- is it reasonable to expect the directory-restructuring by 2.0?

Ram

Re: libgit2 status

[Junio C Hamano]

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> Thiago Farina wrote:
>> [...]
>> With some structure like:
>>
>> include/git.h
>> src/git.c
>>
>> ...
>>
>> whatever.
>> [...]
>
> Junio- is it reasonable to expect the directory-restructuring by 2.0?

I actually hate "include/git.h vs src/git.c"; you have distinction
between .c and .h already.

Re: libgit2 status

[dag]

Junio C Hamano <gitster@pobox.com> writes:

> I actually hate "include/git.h vs src/git.c"; you have distinction
> between .c and .h already.

+1

                          -David

Re: libgit2 status

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Ramkumar Ramachandra <artagnon@gmail.com> writes:
>
>> Thiago Farina wrote:
>>> [...]
>>> With some structure like:
>>>
>>> include/git.h
>>> src/git.c
>>>
>>> ...
>>>
>>> whatever.
>>> [...]
>>
>> Junio- is it reasonable to expect the directory-restructuring by 2.0?
>
> I actually hate "include/git.h vs src/git.c"; you have distinction
> between .c and .h already.

Having said that, I do not mind moving codeblocks around to make
some files purely library-ish while others purely commands.

Ideally, if you run

    $ nm --defined-only -g builtin/frotz.o

you should see nothing but "T cmd_frotz" (there are exceptions, most
notably, what the commands in the "log" family do are so close with
each other that builtin/log.o can define cmd_* for all of them).

    $ nm --defined-only -og builtin/*.o  | grep -v 'T cmd_'

a handful of offenders.  If these functions with external linkage
are truly useful across subcommands, we should move them to a more
library-ish location.

It may require splitting the bits that are too closely tied to the
external interface they are implementing from these functions,
generalizing only the core-ish logic from them, and moving them to a
more library-ish file as a preparatory step.  Such a library-ish file
may be created inside lib/ subdirectory from the get-go.

Until that kind of code restructure happens, I do not see much sense
in just moving files, e.g. renaming revision.c to src/revision.c or
lib/revision.c or somesuch.

Re: libgit2 status

[Thiago Farina]

On Fri, Oct 19, 2012 at 5:13 PM, Junio C Hamano <gitster@pobox.com> wrote:
> I actually hate "include/git.h vs src/git.c"; you have distinction
> between .c and .h already.
Which distinction are you talking about? This is not an issue of
header file versus source file, but a public header file to be
included by external projects versus internal header files that are
intended to be included only by git itself and that will probably live
under src/ directory.

Re: libgit2 status

[Andreas Ericsson]

On 10/19/2012 10:13 PM, Junio C Hamano wrote:
> Ramkumar Ramachandra <artagnon@gmail.com> writes:
> 
>> Thiago Farina wrote:
>>> [...]
>>> With some structure like:
>>>
>>> include/git.h
>>> src/git.c
>>>
>>> ...
>>>
>>> whatever.
>>> [...]
>>
>> Junio- is it reasonable to expect the directory-restructuring by 2.0?
> 
> I actually hate "include/git.h vs src/git.c"; you have distinction
> between .c and .h already.
> 

Agreed. The way libgit2 does it is to have "src/tag.[ch]", which are
for internal use, and then "src/include/tag.h" which is the published
version that others can use to write code against the tag library.
src/tag.h always includes src/include/tag.h, so no code needs to be
duplicated, but internal parts of the library can still use lower-
level stuff if it wants to. It's a good compromise when creating a
library from application code and there were no opaque types from
the start.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.