[GSoC] Interest in "More Sparse Index Integrations" project idea

[GSoC] Interest in "More Sparse Index Integrations" project idea

[Shaoxuan Yuan]

Hi Stolee, Victoria, and Community,

The self-intro was done during my first practice patch [1].

I've been paying attention to the updates to the SoC 2022 ideas
page. And find your sparse index integrations project quite interesting.

It took me around an hour to read through this blog [2] written by
Stolee, and I felt
that although it's a relatively new and quite cool idea (seems a bit daunting),
I think now I have some basic grasp of it (just a distant big idea).

The most amazing part is the drastic improvement of speed when
sparse index is used on huge monorepo (not sure but sometimes "git
status" on git's
repo can make me feel slight lag). I can see this efficiency boost
is beneficial for almost everyone, especially those who operate on
huge monorepo.

And I think what makes me feel excited is that it actually connects to
many topics that I want to learn as a newbie to git community,
e.g. index, the new ORT merge strategy by Elijah (I didn't know the
default merge strategy is replaced if not for reading the blog),
and possibly many other commands' underlying magic (guess because
I can have the chance to integrate them with sparse index and learn more
at the same time).

Generally speaking, I feel that working on and researching about
the project opens up the gate to dive deeper into git and meet tons of fun
(I just finished skimming through the Pro Git book and can barely understand
what the blog is talking about, can't wait to see them in practice). Plus
it is also a good idea for a beginner to walk on a "decent paved-path", quoted
from Stolee.

And the main purpose of this message is to first, show my interest
about this project idea and second, use it as a thread so I can follow up
some questions during the research process.

Question: It is a good idea to start looking into the commands listed
in the idea doc [3] and make some practice patches about sparse index
integrations?

[1] https://lore.kernel.org/git/20220123060318.471414-1-shaoxuan.yuan02@gmail.com/
[2] https://github.blog/2021-11-10-make-your-monorepo-feel-small-with-gits-sparse-index/
[3] https://git.github.io/SoC-2022-Ideas/#more-sparse-index-integrations

-- 
Thanks & Regards,
Shaoxuan

Re: [GSoC] Interest in "More Sparse Index Integrations" project idea

[Derrick Stolee]

On 2/14/2022 7:44 AM, Shaoxuan Yuan wrote:
> Hi Stolee, Victoria, and Community,
> 
> The self-intro was done during my first practice patch [1].

Welcome!
 
> I've been paying attention to the updates to the SoC 2022 ideas
> page. And find your sparse index integrations project quite interesting.

...

> And the main purpose of this message is to first, show my interest
> about this project idea and second, use it as a thread so I can follow up
> some questions during the research process.
> 
> Question: It is a good idea to start looking into the commands listed
> in the idea doc [3] and make some practice patches about sparse index
> integrations?
> 
> [1] https://lore.kernel.org/git/20220123060318.471414-1-shaoxuan.yuan02@gmail.com/
> [2] https://github.blog/2021-11-10-make-your-monorepo-feel-small-with-gits-sparse-index/
> [3] https://git.github.io/SoC-2022-Ideas/#more-sparse-index-integrations
 
One thing you can do is look at the history of t1092, which is used by
every patch series that performs these integrations.

$ git log -20 --oneline --graph --show-pulls origin/master -- t/t1092-sparse-checkout-compatibility.sh
* 3c0e4178277 Merge branch 'ds/fetch-pull-with-sparse-index'
* 3a9a6ac5d9d t1092: replace 'read-cache --table' with 'ls-files --sparse'
* 78087097b8b ls-files: add --sparse option
* 5a4e0547e23 fetch/pull: use the sparse index
*   8d2c37320bd Merge branch 'ld/sparse-diff-blame'
|\  
| * add4c864b60 blame: enable and test the sparse index
| * 51ba65b5c35 diff: enable and test the sparse index
| * 338e2a9acc8 diff: replace --staged with --cached in t1092 tests
| *   ea6ae410bed Merge branch 'vd/sparse-reset' into ld/sparse-diff-blame
| |\  
* | \   e1d92882676 Merge branch 'ds/sparse-deep-pattern-checkout-fix'
|\ \ \  
| * | | 8c5de0d2657 unpack-trees: use traverse_path instead of name
| * | | 1b38efc7a03 t1092: add deeper changes during a checkout
* | | |   62329d336f5 Merge branch 'ds/trace2-regions-in-tests'
|\ \ \ \  
| * | | | 8c4cbad6a3a t/t*: remove custom GIT_TRACE2_EVENT_NESTING
| |/ / /  
* | | |   f0850875fd4 Merge branch 'vd/sparse-reset'
|\ \ \ \  
| | |_|/  
| |/| |   
| * | | 4d1cfc1351f reset: make --mixed sparse-aware
| * | | 20ec2d034cd reset: make sparse-aware (except --mixed)
| * | | 291d77eb3e2 reset: expand test coverage for sparse checkouts
| * | | 86609db9dae sparse-index: update command for expand/collapse test
| * | | 71471b2a7c7 reset: preserve skip-worktree bit in mixed reset

But of course, this doesn't show the full history of the work in
each of these branches, so you can do a history walk from each
merge to get a picture of the full series:

$ git log --oneline --graph 3c0e4178277 -6
*   3c0e4178277 Merge branch 'ds/fetch-pull-with-sparse-index'
|\  
| * 408c51f0b4b test-read-cache: remove --table, --expand options
| * c2a29405105 t1091/t3705: remove 'test-tool read-cache --table'
| * 3a9a6ac5d9d t1092: replace 'read-cache --table' with 'ls-files --sparse'
| * 78087097b8b ls-files: add --sparse option
| * 5a4e0547e23 fetch/pull: use the sparse index

$ git log --oneline --graph 8d2c37320bd -8
*   8d2c37320bd Merge branch 'ld/sparse-diff-blame'
|\  
| * add4c864b60 blame: enable and test the sparse index
| * 51ba65b5c35 diff: enable and test the sparse index
| * 338e2a9acc8 diff: replace --staged with --cached in t1092 tests
| * 44c7e62e51e repo-settings: prepare_repo_settings only in git repos
| * 27a443b8208 test-read-cache: set up repo after git directory
| * 0803f9c7cdf commit-graph: return if there is no git directory
| * e5b17bda8b6 git: ensure correct git directory setup with -h

These are just a couple examples of merged work in this area. Be
sure to look at the full commit messages and diffs to get a feel
for the area. Then, playing around with the list of commands in
the project idea document is a good idea.

There is also a series that is currently waiting to merge [4], for
even more examples.

[4] https://lore.kernel.org/git/pull.1109.v2.git.1641924306.gitgitgadget@gmail.com/

Thanks,
-Stolee

Re: [GSoC] Interest in "More Sparse Index Integrations" project idea

[Philippe Blain]

> But of course, this doesn't show the full history of the work in
> each of these branches, so you can do a history walk from each
> merge to get a picture of the full series:
> 
> $ git log --oneline --graph 3c0e4178277 -6
> *   3c0e4178277 Merge branch 'ds/fetch-pull-with-sparse-index'
> |\  

Or even better (you do not have to guess the number of commits in 
each series):

git log --oneline --graph 3c0e4178277^-

(see [1]). Why that syntax seems to completely ignore '--graph',
though is beyond me!

Philippe.

[1] https://git-scm.com/docs/gitrevisions#Documentation/gitrevisions.txt-emltrevgt-ltngtemegemHEAD-HEAD-2em

Re: [GSoC] Interest in "More Sparse Index Integrations" project idea

[Shaoxuan Yuan]

On Mon, Feb 14, 2022 at 10:38 PM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 2/14/2022 7:44 AM, Shaoxuan Yuan wrote:
> > Hi Stolee, Victoria, and Community,
> >
> > The self-intro was done during my first practice patch [1].
>
> Welcome!
>
> > I've been paying attention to the updates to the SoC 2022 ideas
> > page. And find your sparse index integrations project quite interesting.
>
> ...
>
> > And the main purpose of this message is to first, show my interest
> > about this project idea and second, use it as a thread so I can follow up
> > some questions during the research process.
> >
> > Question: It is a good idea to start looking into the commands listed
> > in the idea doc [3] and make some practice patches about sparse index
> > integrations?
> >
> > [1] https://lore.kernel.org/git/20220123060318.471414-1-shaoxuan.yuan02@gmail.com/
> > [2] https://github.blog/2021-11-10-make-your-monorepo-feel-small-with-gits-sparse-index/
> > [3] https://git.github.io/SoC-2022-Ideas/#more-sparse-index-integrations
>
> One thing you can do is look at the history of t1092, which is used by
> every patch series that performs these integrations.
>
> $ git log -20 --oneline --graph --show-pulls origin/master -- t/t1092-sparse-checkout-compatibility.sh
> * 3c0e4178277 Merge branch 'ds/fetch-pull-with-sparse-index'
> * 3a9a6ac5d9d t1092: replace 'read-cache --table' with 'ls-files --sparse'
> * 78087097b8b ls-files: add --sparse option
> * 5a4e0547e23 fetch/pull: use the sparse index
> *   8d2c37320bd Merge branch 'ld/sparse-diff-blame'
> |\
> | * add4c864b60 blame: enable and test the sparse index
> | * 51ba65b5c35 diff: enable and test the sparse index
> | * 338e2a9acc8 diff: replace --staged with --cached in t1092 tests
> | *   ea6ae410bed Merge branch 'vd/sparse-reset' into ld/sparse-diff-blame
> | |\
> * | \   e1d92882676 Merge branch 'ds/sparse-deep-pattern-checkout-fix'
> |\ \ \
> | * | | 8c5de0d2657 unpack-trees: use traverse_path instead of name
> | * | | 1b38efc7a03 t1092: add deeper changes during a checkout
> * | | |   62329d336f5 Merge branch 'ds/trace2-regions-in-tests'
> |\ \ \ \
> | * | | | 8c4cbad6a3a t/t*: remove custom GIT_TRACE2_EVENT_NESTING
> | |/ / /
> * | | |   f0850875fd4 Merge branch 'vd/sparse-reset'
> |\ \ \ \
> | | |_|/
> | |/| |
> | * | | 4d1cfc1351f reset: make --mixed sparse-aware
> | * | | 20ec2d034cd reset: make sparse-aware (except --mixed)
> | * | | 291d77eb3e2 reset: expand test coverage for sparse checkouts
> | * | | 86609db9dae sparse-index: update command for expand/collapse test
> | * | | 71471b2a7c7 reset: preserve skip-worktree bit in mixed reset
>
> But of course, this doesn't show the full history of the work in
> each of these branches, so you can do a history walk from each
> merge to get a picture of the full series:
>
> $ git log --oneline --graph 3c0e4178277 -6
> *   3c0e4178277 Merge branch 'ds/fetch-pull-with-sparse-index'
> |\
> | * 408c51f0b4b test-read-cache: remove --table, --expand options
> | * c2a29405105 t1091/t3705: remove 'test-tool read-cache --table'
> | * 3a9a6ac5d9d t1092: replace 'read-cache --table' with 'ls-files --sparse'
> | * 78087097b8b ls-files: add --sparse option
> | * 5a4e0547e23 fetch/pull: use the sparse index
>
> $ git log --oneline --graph 8d2c37320bd -8
> *   8d2c37320bd Merge branch 'ld/sparse-diff-blame'
> |\
> | * add4c864b60 blame: enable and test the sparse index
> | * 51ba65b5c35 diff: enable and test the sparse index
> | * 338e2a9acc8 diff: replace --staged with --cached in t1092 tests
> | * 44c7e62e51e repo-settings: prepare_repo_settings only in git repos
> | * 27a443b8208 test-read-cache: set up repo after git directory
> | * 0803f9c7cdf commit-graph: return if there is no git directory
> | * e5b17bda8b6 git: ensure correct git directory setup with -h
>
> These are just a couple examples of merged work in this area. Be
> sure to look at the full commit messages and diffs to get a feel
> for the area. Then, playing around with the list of commands in
> the project idea document is a good idea.

Thanks for the examples and suggestions! Working on them :)

>
> There is also a series that is currently waiting to merge [4], for
> even more examples.
>
> [4] https://lore.kernel.org/git/pull.1109.v2.git.1641924306.gitgitgadget@gmail.com/
>
> Thanks,
> -Stolee

-- 
Thanks & Regards,
Shaoxuan

Re: [GSoC] Interest in "More Sparse Index Integrations" project idea

[Shaoxuan Yuan]

On Tue, Feb 15, 2022 at 2:45 AM Philippe Blain
<levraiphilippeblain@gmail.com> wrote:
>
>
> > But of course, this doesn't show the full history of the work in
> > each of these branches, so you can do a history walk from each
> > merge to get a picture of the full series:
> >
> > $ git log --oneline --graph 3c0e4178277 -6
> > *   3c0e4178277 Merge branch 'ds/fetch-pull-with-sparse-index'
> > |\
>
> Or even better (you do not have to guess the number of commits in
> each series):
>
> git log --oneline --graph 3c0e4178277^-
>
> (see [1]). Why that syntax seems to completely ignore '--graph',
> though is beyond me!
>
> Philippe.
>
> [1] https://git-scm.com/docs/gitrevisions#Documentation/gitrevisions.txt-emltrevgt-ltngtemegemHEAD-HEAD-2em

Thanks ;-)

-- 
Thanks & Regards,
Shaoxuan

Re: [GSoC] Interest in "More Sparse Index Integrations" project idea

[Shaoxuan Yuan]

On Tue, Feb 15, 2022 at 2:45 AM Philippe Blain
<levraiphilippeblain@gmail.com> wrote:
> Or even better (you do not have to guess the number of commits in
> each series):
>
> git log --oneline --graph 3c0e4178277^-
>
> (see [1]). Why that syntax seems to completely ignore '--graph',
> though is beyond me!

I read the [1] and this "<rev>^-" syntax seems to exclude the first
parent of <rev>.
In another word, it only gets the part of history within the checked-out
new branch, i.e. "This is typically useful for merge commits where you
can just pass <commit>^- to get all the commits in the branch that was
merged in merge commit <commit> (including <commit> itself)."

It seems that the checked-out branch has a linear clean history,
and that's why it looks like "completely ignore '--graph'". Actually, it does
not ignore "--graph", e.g.

with "--graph":

* 3c0e417827 Merge branch 'ds/fetch-pull-with-sparse-index'
* 408c51f0b4 test-read-cache: remove --table, --expand options
* c2a2940510 t1091/t3705: remove 'test-tool read-cache --table'
* 3a9a6ac5d9 t1092: replace 'read-cache --table' with 'ls-files --sparse'
* 78087097b8 ls-files: add --sparse option
* 5a4e0547e2 fetch/pull: use the sparse index

w/out "--graph":

3c0e417827 Merge branch 'ds/fetch-pull-with-sparse-index'
408c51f0b4 test-read-cache: remove --table, --expand options
c2a2940510 t1091/t3705: remove 'test-tool read-cache --table'
3a9a6ac5d9 t1092: replace 'read-cache --table' with 'ls-files --sparse'
78087097b8 ls-files: add --sparse option
5a4e0547e2 fetch/pull: use the sparse index

Look at the asterisks prefixing every commit in the with "--graph" part.
If you init a new repo and add a few commits to form a linear history,
you will get a similar result by "git log --oneline --graph".

> [1] https://git-scm.com/docs/gitrevisions#Documentation/gitrevisions.txt-emltrevgt-ltngtemegemHEAD-HEAD-2em

-- 
Thanks & Regards,
Shaoxuan