The state of the object store series

The state of the object store series

[Stefan Beller]

There is an ongoing effort to remove global state currently and switch over
to pass around the relevant data structures; for most of it we end up passing
around 'the_repository' as it contains everything there is.

Merged into master:

fcb6df32546 Merge branch 'sb/oid-object-info'
a2cec42213c Merge branch 'sb/object-store-replace'
3a1ec60c43b Merge branch 'sb/packfiles-in-repository'
cf0b1793ead Merge branch 'sb/object-store'

Currently cooking:

8c69a7d7e80 Merge branch 'sb/object-store-grafts' into pu
42d32d07298 Merge branch 'sb/object-store-alloc' into jch

(both marked for "will merge to next" in the cooking report)

Floating on the mailing list, not cooking yet:

"sb/object-store-lookup"
https://public-inbox.org/git/20180613230522.55335-1-sbeller@google.com/
  This clashes with other series in flight (Stolees get_tree series; see
  https://public-inbox.org/git/709bd61c-70fc-a925-efba-58ab9be265c9@gmail.com/
  I'll rebase this series on top of that series once the currently cooking
  series stabilize)

"nd/kill-the_index"
https://public-inbox.org/git/20180616054157.32433-1-pclouds@gmail.com/
  This converts the_index to pass around index pointers instead of
the_repository;
  it fits into the theme, but Duys end goal differs from mine; he is
less submodule focused.

Work that still needs to be done:

"xx/object-store-commit-graph"
  Convert the commit graph to have no global state, but be part of the
  repository struct. I think this can go in parallel to
"sb/object-store-lookup",
  so I'll tackle that next. Thanks Stolee for looking ahead: There is only the
  commit graph itself as well as whether it has been prepared as a global
  variable. So this series will consist of passing around a repository struct
  for all those higher level functions that do not pass around the commit graph
  or parts of it.

"xx/finish-object-stores"
  This requires "xx/object-store-commit-graph" as well as
"sb/object-store-lookup";
  it will convert parse_commit[_gently] to take a repository argument and will
  finish the actual object store part. This might be optional for the goal of
  converting submodules, that I have in mind, but it sure is a nice finishing
  touch.

"xx/convert-revision-walking"
  This series aims to convert get_merge_bases(), in_merge_bases() and all its
  revision walking code to take a repository argument.

"xx/submodule-dont-use-alternates"
  Once "xx/convert-revision-walking" is in, convert the local
find_first_merges(),
  and convert all functions to drop function add_submodule_odb() and instead
  operate on the submodule repository instead of the_repository with the
  submodule objects added as an alternate.

Thanks,
Stefan

Re: The state of the object store series

[Jonathan Tan]

> Floating on the mailing list, not cooking yet:

One more is my bitmap one here:

https://public-inbox.org/git/cover.1528397984.git.jonathantanmy@google.com/

It's not in any branch yet, as far as I can tell, so I've just sent out
an e-mail letting Junio know [1].

[1] https://public-inbox.org/git/20180619222749.124671-1-jonathantanmy@google.com/

Re: The state of the object store series

[Stefan Beller]

On Tue, Jun 19, 2018 at 3:37 PM Jonathan Tan <jonathantanmy@google.com> wrote:
>
> > Floating on the mailing list, not cooking yet:
>
> One more is my bitmap one here:

Oh right. Thanks for writing the series!

When writing this I dabbled back and forth whether I only present the series
that are on the critical path to reach the submodule goal or to
include all of them
(see Duys series) and that is how I missed yours.

> https://public-inbox.org/git/cover.1528397984.git.jonathantanmy@google.com/
>
> It's not in any branch yet, as far as I can tell, so I've just sent out
> an e-mail letting Junio know [1].

Thanks!

>
> [1] https://public-inbox.org/git/20180619222749.124671-1-jonathantanmy@google.com/

Re: The state of the object store series

[Duy Nguyen]

On Wed, Jun 20, 2018 at 12:09 AM Stefan Beller <sbeller@google.com> wrote:
> "xx/convert-revision-walking"
>   This series aims to convert get_merge_bases(), in_merge_bases() and all its
>   revision walking code to take a repository argument.

We'll see who gets there first [1] ;-). It does not really matter (and
I suspect you may beat me to it) but I hope we all agree to pass
'struct repository *' in init_revisions(), or I should start reworking
that patch now.

[1] https://gitlab.com/pclouds/git/commit/a4a91e3815cf0e594d5c9999e0c862f5060e2c26
-- 
Duy

Re: The state of the object store series

[Derrick Stolee]

[My email client says I sent this two days ago, but it doesn't appear on 
public-inbox.org, so somehow it didn't land on-list (Stefan pointed out 
that the message was in HTML). Repeating the message now so I can 
reference it in another discussion. -Stolee]

On 6/19/2018 6:07 PM, Stefan Beller wrote:
> There is an ongoing effort to remove global state currently and switch over
> to pass around the relevant data structures; for most of it we end up passing
> around 'the_repository' as it contains everything there is.
>
> Merged into master:
>
> fcb6df32546 Merge branch 'sb/oid-object-info'
> a2cec42213c Merge branch 'sb/object-store-replace'
> 3a1ec60c43b Merge branch 'sb/packfiles-in-repository'
> cf0b1793ead Merge branch 'sb/object-store'
>
> Currently cooking:
>
> 8c69a7d7e80 Merge branch 'sb/object-store-grafts' into pu
> 42d32d07298 Merge branch 'sb/object-store-alloc' into jch
>
> (both marked for "will merge to next" in the cooking report)
>
> Floating on the mailing list, not cooking yet:
>
> "sb/object-store-lookup" 
> https://public-inbox.org/git/20180613230522.55335-1-sbeller@google.com/ 
> This clashes with other series in flight (Stolees get_tree series; see 
> https://public-inbox.org/git/709bd61c-70fc-a925-efba-58ab9be265c9@gmail.com/ 
> I'll rebase this series on top of that series once the currently 
> cooking series stabilize) "nd/kill-the_index" 
> https://public-inbox.org/git/20180616054157.32433-1-pclouds@gmail.com/ 
> This converts the_index to pass around index pointers instead of 
> the_repository; it fits into the theme, but Duys end goal differs from 
> mine; he is less submodule focused. Work that still needs to be done: "xx/object-store-commit-graph"
>    Convert the commit graph to have no global state, but be part of the
>    repository struct. I think this can go in parallel to
> "sb/object-store-lookup",
>    so I'll tackle that next. Thanks Stolee for looking ahead: There is only the
>    commit graph itself as well as whether it has been prepared as a global
>    variable. So this series will consist of passing around a repository struct
>    for all those higher level functions that do not pass around the commit graph
>    or parts of it.

There are some patches in ds/commit-graph-fsck that would be helpful for 
this effort, and may be easier for you to take.

Specifically, these ones:

  - commit-graph: load a root tree from specific graph
  - commit: force commit to parse from object database
  - commit-graph: parse commit from chosen graph

I'm waiting for sb/object-store-lookup to be in 'next' before I re-roll 
that branch. If you're not in a rush to send this series, perhaps wait 
for the next version here.

> "xx/finish-object-stores"
>    This requires "xx/object-store-commit-graph" as well as
> "sb/object-store-lookup";
>    it will convert parse_commit[_gently] to take a repository argument and will
>    finish the actual object store part. This might be optional for the goal of
>    converting submodules, that I have in mind, but it sure is a nice finishing
>    touch.
>
> "xx/convert-revision-walking"
>    This series aims to convert get_merge_bases(), in_merge_bases() and all its
>    revision walking code to take a repository argument.

I'm playing around with a series that groups most commit walks into one 
file, so they can share more code and we can more easily test their 
correctness and performance. I'll send an RFC sometime that will 
absolutely collide with this one, but I expect a full v1 patch will be 
based on this series.

Thanks,

-Stolee

Re: The state of the object store series

[Jonathan Tan]

> I'm waiting for sb/object-store-lookup to be in 'next' before I re-roll 
> that branch. If you're not in a rush to send this series, perhaps wait 
> for the next version here.

Did you mean another branch? I'm looking at
https://github.com/gitster/git right now, and I see
sb/object-store{,-alloc,-grafts,-replace} but not
sb/object-store-lookup.

(I'm asking this because I'm trying to wrap my head around the
interconnectedness of all our patches.)