Git Miniconference at Plumbers

Git Miniconference at Plumbers

[Jon Loeliger]

Folks,

I have recently been enlisted by folks at the Linux Foundation to
help run a Miniconference on Git at the Plumbers Conference [*]
this fall.

We currently have both Junio Hamano and Josh Triplett signed
up to do talks.  Hopefully, though not confirmed yet, Junio
will give us a brief "State of the Git Union" to set the stage
for a few more presentations and some discussion about the
Future of Git.  Josh has volunteered to talk about a patch
series manager he's been developing.

Rumor also suggests that the Man In The	Bowtie might make
an appearance as well.	With luck, Mr. Bottomley might
offer a	speaking role as well!	We'll see.

I would like to solicit one or two more solid talks from
the community, either the Linux Kernel community or the
Git community proper.  If you are so interested, please
send me some mail.

Thanks,
jdl


[*] -- http://www.linuxplumbersconf.org/2016/

Re: Git Miniconference at Plumbers

[Jeff King]

On Tue, Sep 06, 2016 at 12:42:04PM -0500, Jon Loeliger wrote:

> I have recently been enlisted by folks at the Linux Foundation to
> help run a Miniconference on Git at the Plumbers Conference [*]
> this fall.

I see the conference runs for 4 days; I assume the Git portion will just
be one day. Do you know yet which day?

-Peff

Re: Git Miniconference at Plumbers

[Jon Loeliger]

So, like, Jeff King said:
> On Tue, Sep 06, 2016 at 12:42:04PM -0500, Jon Loeliger wrote:
> 
> > I have recently been enlisted by folks at the Linux Foundation to
> > help run a Miniconference on Git at the Plumbers Conference [*]
> > this fall.
> 
> I see the conference runs for 4 days; I assume the Git portion will just
> be one day.

Yes, and yes.  Likely even "a half day".

> Do you know yet which day?

No.  Sorry.

> -Peff

jdl

RE: Git Miniconference at Plumbers

[David Bainbridge]

Hi,

The subject matter of the conference looks really interesting but I am unlikely to be able to attend, unfortunately.

The subjects being covered like the current State of Git and the Future of Git, for example, deserve much wider exposure, and I would certainly appreciate hearing the thoughts of Junio and others.

Does anyone know whether the sessions will be recorded in any way?

Thanks

David

DAVID BAINBRIDGE
Product Manager SW Development

Ericsson
8500 Decarie
Montreal, H4P 2N2, Canada
david.bainbridge@ericsson.com
www.ericsson.com

-----Original Message-----
From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On Behalf Of Jon Loeliger
Sent: Monday, September 12, 2016 09:33
To: Jeff King <peff@peff.net>
Cc: git@vger.kernel.org
Subject: Re: Git Miniconference at Plumbers

So, like, Jeff King said:
> On Tue, Sep 06, 2016 at 12:42:04PM -0500, Jon Loeliger wrote:
> 
> > I have recently been enlisted by folks at the Linux Foundation to 
> > help run a Miniconference on Git at the Plumbers Conference [*] this 
> > fall.
> 
> I see the conference runs for 4 days; I assume the Git portion will 
> just be one day.

Yes, and yes.  Likely even "a half day".

> Do you know yet which day?

No.  Sorry.

> -Peff

jdl

Re: Git Miniconference at Plumbers

[Jon Loeliger]

So, like, David Bainbridge said:
> Hi,
> 
> The subject matter of the conference looks really interesting but I am
> unlikely to be able to attend, unfortunately.
> 
> The subjects being covered like the current State of Git and the
> Future of Git, for example, deserve much wider exposure, and I would
> certainly appreciate hearing the thoughts of Junio and others.

Indeed.

> Does anyone know whether the sessions will be recorded in any way?

I am uncertain about outright recording (digital video/audio),
but there will be at least summarizing notes taken and posted.
Anyone wishing to record the talks/discussions is likely welcome
to do so.

HTH,
jdl

Re: Git Miniconference at Plumbers

[Junio C Hamano]

Jon Loeliger <jdl@jdl.com> writes:

> So, like, David Bainbridge said:
>> Hi,
>> 
>> The subject matter of the conference looks really interesting but I am
>> unlikely to be able to attend, unfortunately.
>> 
>> The subjects being covered like the current State of Git and the
>> Future of Git, for example, deserve much wider exposure, and I would
>> certainly appreciate hearing the thoughts of Junio and others.
>
> Indeed.

You do not need to go to NM to _hear_ that.  Basically, I want us
not to have "big" plans that come from the top.

Now, you heard it ;-)

There are areas that we as Git community would want to address for
some audience that were discovered over the years, and that "some
audience" might even be a large population of Git users, but if that
does not have overlap with Kernel Plumbers, the Plumbers mini-conf
may not be a suitable venue for even mentioning them.  E.g. the
enhancement of the submodule subsystem to allow more end-user facing
commands to recurse into them; rearchitecting the index and redoing
the "sparse checkout" hack so that we can do narrow clones more
properly; supporting "huge objects" better in the object layer,
without having to resort to ugly hacks like GitLFS that will never
be part of the core Git.  These things may all be worth talking
about in wider Git setting, but some of them may be waste of time to
bring up in the Plumbers' venue.

The future of Git is shaped largely by end-user itches.  From my
point of view, Git people are going there primarily to find what
Kernel Plubmbers' itches are, and help assessing the workflow
improvements around Git the Plumbers are wishing for or designing
themselves by being there, because we are at the best position to
tell what kind of enhancement to Git is feasible and what is
unlikely to happen in the near term.

Re: Git Miniconference at Plumbers

[Jakub Narębski]

W dniu 12.09.2016 o 20:09, Jon Loeliger napisał:
> So, like, David Bainbridge said:

>> Does anyone know whether the sessions will be recorded in any way?
> 
> I am uncertain about outright recording (digital video/audio),
> but there will be at least summarizing notes taken and posted.
> Anyone wishing to record the talks/discussions is likely welcome
> to do so.

I think LWN.net usually covers Linux Plumbers Conference, see e.g.
https://lwn.net/Archives/ConferenceByYear/#2015-Linux_Plumbers_Conference

Hopefully they would cover it this year too.

HTH
-- 
Jakub Narębski

Re: Git Miniconference at Plumbers

[Lars Schneider]

> On 12 Sep 2016, at 21:11, Junio C Hamano <gitster@pobox.com> wrote:
> 
> [..]
> properly; supporting "huge objects" better in the object layer,
> without having to resort to ugly hacks like GitLFS that will never
> be part of the core Git. [...]

I agree with you that GitLFS is an ugly hack.

Some applications have test data, image assets, and other data sets that
need to be versioned along with the source code.

How would you deal with these kind of "huge objects" _today_?

Thanks,
Lars

Re: Git Miniconference at Plumbers

[Christian Couder]

On Tue, Sep 13, 2016 at 1:14 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
>> On 12 Sep 2016, at 21:11, Junio C Hamano <gitster@pobox.com> wrote:
>>
>> [..]
>> properly; supporting "huge objects" better in the object layer,
>> without having to resort to ugly hacks like GitLFS that will never
>> be part of the core Git. [...]
>
> I agree with you that GitLFS is an ugly hack.
>
> Some applications have test data, image assets, and other data sets that
> need to be versioned along with the source code.
>
> How would you deal with these kind of "huge objects" _today_?

I think that Junio was saying that this problem and other problems
like this one are indeed itches for some people, but maybe not for
kernel community.

About this specific problem, as you probably know, I started working
on adding support for external object databases, on top of some
previous work that Peff had started some years ago:

https://public-inbox.org/git/20160628181933.24620-1-chriscool@tuxfamily.org/

So if you want to better deal with huge objects in the near future,
you are welcome to help on this.

Re: Git Miniconference at Plumbers

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> Some applications have test data, image assets, and other data sets that
> need to be versioned along with the source code.
>
> How would you deal with these kind of "huge objects" _today_?

When you know that you'd find the answer to that question totally
uninteresting, why do you even bother to ask? ;-)

I don't, and if I had to, I would deal with them just like any other
objects.

A more interesting pair of questions to ask would be what the
fundamental requirement for an acceptable solution is, and what
solution within the constraint I would envision, if I were given a
group competent Git hackers and enough time to realize it.

The most important constraint is that any acceptable solution should
preserve the object identity.

And starting from a "I don't but if I had to..." repository that is
created in a dumb way, a solution that satisifies the constraint may
work like this, requiring enhancements to various parts of the
system:

 - The "upload-pack" protocol would allow the owner of a such
   repository and the party that "git clone"'s from there to
   negotiate:

    . what it means for a object to be "huge" (e.g. the owner may
      implicitly show the preference by marking a packfile as
      containing such "huge" objects, may configure that blobs that
      appear at paths that match certain glob pattern are "huge", or
      the sender and the receiver may say objects that are larger
      than X MB are "huge", etc.); and

    . what to do with "huge" objects (e.g. the receiver may ask for
      a full clone, or the receiver may ask to omit "huge" ones from
      the initial transfer)

 - The "upload-pack" protocol would give, in addition to the normal
   pack stream that conveys only non-"huge" objects, for each of
   "huge" objects that are not transferred, what its object name is
   and how it can later be retrieved.

 - Just like packing objects in packfiles was added as a different
   implementation to store objects in the object database that is
   better than storing them individually as loose object files,
   there will be a third way to store such "huge" object _in_ the
   object database, which may actually not _store_ them locally at
   all.  The local object store may merely have placeholders for
   them, in which instructions for how it can be acquired when
   necessary are stored.  The extra information sent over the
   "upload-pack" protocol for "huge" objects with the previous
   bullet-point are used to store these objects in this "third" way.

 - A new mechanism would allow such objects that are stored in this
   "third" way to be retrieved lazily or on-demand.

There are other enhancements whose necessity will fall naturally out
of such a lazy scheme outlined above.  E.g. "fsck" needs to learn
that the objects stored in the third way are considered to "exist"
but their actual contents is not expected to be verifiable until
they are retrieved. "send-pack" (i.e. running "git push" from a
repository cloned with the procedure outlined above) needs to treat
the objects stored in the third way differently (most likely, it
will fail a request for full-clone and send "not here, but you can
get it this way" for them).  Local operations that need more than
object names need to learn reasonable fallback behaviours to work
when the actual object contents are not yet available (e.g. all of
them may offer "this is not yet available; do you want to get
on-demand?" or there may even be "object.ondemand" configuration
option to skip the end-user interaction.  When on-demand retrieving
is not done, "git archive" may place a placeholder file in its
output that says "no data (yet) here", "git log --raw" may show the
object name but "git log -p" may say "insufficient data to produce a
patch", etc.) [*1*].

Because we start from the "object identity should not change", you
do not have to make a decision upfront when preparing the ultimate
source of the truth.  When you take a clone-network of a single
project as a whole, somebody needs to hold the entire set of objects
somewhere, and many of the repository in the clone-network may have
"huge" objects in the third "not here yet, here is how to get it"
form.  As the system improves, and as the networking and storage
technology changes, the definition of "huge" WILL change over time
and those repositories can turn the ones that used to be "huge" into
normal objects.

If you use approaches taken by various clean/smudge based current
crop of solutions [*2*], on the other hand, once you decide a blob
object is "huge" and needs to be replaced with a surrogate (to be
instantiated via the "clean" filter), the "huge" object _has_ to
stay in the surrogate form in the containing tree and you cannot
change the division between "huge" and "normal" ever without
rewriting the history.


[Footnote]

*1* Astute readers would realize that the utility of such a "third
    way" object storage mechanism is not limited to "keep and
    transfer huge objects lazily".  The same mechanism can say "not
    yet here, and there is no way for _you_ to retrieve the
    contents", which is an effective way to "obliterate" an object.

*2* I called them "hacks" because they are practical compromise that
    can be done with today's Git, while sidestepping harder problems
    that are needed to be solved to realize the solution outlined
    above.