[GSoC] My project blog

[GSoC] My project blog

[Matheus Tavares Bernardino]

Hi, everyone

I've been posting about the progress of my GSoC project at
https://matheustavares.gitlab.io/gsoc/ I should have sent the link
earlier but, please, let me know if you have any comments on my
previous posts.

You can read my latest update here:
https://matheustavares.gitlab.io/posts/week-7-race-conditions-on-delta-base-cache
I've been working to allow parallel decompression when reading objects
for git-grep.

Thanks,
Matheus

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

Hi, everyone

I just posted a new update about my GSoC project here:
https://matheustavares.gitlab.io/posts/week-8-a-working-parallel-inflation
Please, feel free to leave any comments. I've finally solved the race
condition problem and now we have a working parallel inflation :) But
there's still some tasks to be done on it before I have a proper
patchset.

Thanks,
Matheus

Re: [GSoC] My project blog

[Christian Couder]

Hi Matheus,

On Sun, Jul 14, 2019 at 9:38 AM Matheus Tavares Bernardino
<matheus.bernardino@usp.br> wrote:
>
> I just posted a new update about my GSoC project here:
> https://matheustavares.gitlab.io/posts/week-8-a-working-parallel-inflation
> Please, feel free to leave any comments. I've finally solved the race
> condition problem and now we have a working parallel inflation :) But
> there's still some tasks to be done on it before I have a proper
> patchset.

Great that you solved the race condition!

I agree with the tasks you list in the next steps. I would even
suggest that, right after splitting the changes into logical patches,
you ask your mentors (Olga and me) to review it, and then send it as
an RFC patch series to the mailing list. When you do that though,
please list the other tasks and limitations of your patch series in
the cover letter.

Thanks,
Christian.

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

On Sun, Jul 14, 2019 at 8:00 AM Christian Couder
<christian.couder@gmail.com> wrote:
>
> Hi Matheus,
>
> On Sun, Jul 14, 2019 at 9:38 AM Matheus Tavares Bernardino
> <matheus.bernardino@usp.br> wrote:
> >
> > I just posted a new update about my GSoC project here:
> > https://matheustavares.gitlab.io/posts/week-8-a-working-parallel-inflation
> > Please, feel free to leave any comments. I've finally solved the race
> > condition problem and now we have a working parallel inflation :) But
> > there's still some tasks to be done on it before I have a proper
> > patchset.
>
> Great that you solved the race condition!
>
> I agree with the tasks you list in the next steps. I would even
> suggest that, right after splitting the changes into logical patches,
> you ask your mentors (Olga and me) to review it, and then send it as
> an RFC patch series to the mailing list. When you do that though,
> please list the other tasks and limitations of your patch series in
> the cover letter.

OK, I will do that.

Thanks,
Matheus

> Thanks,
> Christian.

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

Hi, everyone

Unfortunately, not quite much to report this week :( I'm attending
DebConf until July 28th but I'm trying to conciliate that with the
work on my project. You can see my brief report of last week here:
https://matheustavares.gitlab.io/posts/week-9-an-rfc-on-parallel-inflation

As always, every comment is highly appreciated.

Thanks,
Matheus

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

Hi, everyone

I just posted an update about my project here:
https://matheustavares.gitlab.io/posts/week-10-a-bug-in-git-grep-submodules
This one is focused on a patch to correct a bug at git-grep
--recurse-submodules and some tests for the parallel inflation
patchset. Please, feel free to leave any comments.

Thanks,
Matheus

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

Hi,

Here's my report from last week:
https://matheustavares.gitlab.io/posts/week-11-wip-grep-protecting-textconv-and-submodules
I'm working to protect the operations I left behind on the first
version of the patchset[1]. And for that, I used a lot of the code Duy
provided[2] me as an example in the early days of this project. The
race conditions are now majorly gone, but the patches still need some
refactoring and there are still some problems to overcome.

Thanks,
Matheus

[1]: https://public-inbox.org/git/cover.1563570204.git.matheus.bernardino@usp.br/
[2]: https://gitlab.com/snippets/1834613#L3

Re: [GSoC] My project blog

[Christian Couder]

Hi Matheus,

On Tue, Aug 6, 2019 at 4:54 AM Matheus Tavares Bernardino
<matheus.bernardino@usp.br> wrote:
>
> Here's my report from last week:
> https://matheustavares.gitlab.io/posts/week-11-wip-grep-protecting-textconv-and-submodules

Thank you for another great report!

> I'm working to protect the operations I left behind on the first
> version of the patchset[1]. And for that, I used a lot of the code Duy
> provided[2] me as an example in the early days of this project. The
> race conditions are now majorly gone, but the patches still need some
> refactoring and there are still some problems to overcome.

This looks good to me! I think you are now way past the point where
you know much more about this than me, but anyway the following might
give you some ideas or perhaps in some other ways help you:

  - About "Try to remove the obj_read_lock", which is the last point
on your TODO list, it doesn't look necessary before you submit a patch
series, unless there is no performance improvement without it. You
could, as far as I can tell, do it in a later patch series.

  - I guess " --recurse-submodules" is not used very often with git
grep. For example I never use it on the Git repo even when I perhaps
should. So if it would simplify things to just disable the
improvements you made when " --recurse-submodules" is used, then you
might consider doing that. You could later send another patch series
that would enable your improvements when " --recurse-submodules" is
used.

Thanks,
Christian.

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

On Tue, Aug 6, 2019 at 4:16 AM Christian Couder
<christian.couder@gmail.com> wrote:
>
> Hi Matheus,
>
> On Tue, Aug 6, 2019 at 4:54 AM Matheus Tavares Bernardino
> <matheus.bernardino@usp.br> wrote:
> >
> > Here's my report from last week:
> > https://matheustavares.gitlab.io/posts/week-11-wip-grep-protecting-textconv-and-submodules
>
> Thank you for another great report!
>
> > I'm working to protect the operations I left behind on the first
> > version of the patchset[1]. And for that, I used a lot of the code Duy
> > provided[2] me as an example in the early days of this project. The
> > race conditions are now majorly gone, but the patches still need some
> > refactoring and there are still some problems to overcome.
>
> This looks good to me! I think you are now way past the point where
> you know much more about this than me, but anyway the following might
> give you some ideas or perhaps in some other ways help you:
>
>   - About "Try to remove the obj_read_lock", which is the last point
> on your TODO list, it doesn't look necessary before you submit a patch
> series, unless there is no performance improvement without it. You
> could, as far as I can tell, do it in a later patch series.
>
>   - I guess " --recurse-submodules" is not used very often with git
> grep. For example I never use it on the Git repo even when I perhaps
> should. So if it would simplify things to just disable the
> improvements you made when " --recurse-submodules" is used, then you
> might consider doing that. You could later send another patch series
> that would enable your improvements when " --recurse-submodules" is
> used.

Thanks for the suggestions, Christian. So, I think I could finish and
send the patchset without the optimization for --recurse-submodules
and --textconv, while I don't finish support for them, right? And I
think 'obj_read_lock' may be removed in a future patch, as you said,
because we already have good speedup without removing it. Finally,
should I also add a "NOTES" section to Documentation/git-grep.txt
saying which options --threads is not compatible with?

Thanks,
Matheus

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

Hi, everyone

I just posted a new update on my project:
https://matheustavares.gitlab.io/posts/simplified-version-of-parallel-inflation

This week I sent a simplified version of the series I was working on
last week (which would still take some time to finish). And I'm
working to improve it with support to --textconv and
--recurse-submodules in the added optimizations. I've been also
working on a patch to make git-grep stop adding submodules to the
alternates list.

Thanks,
Matheus

Re: [GSoC] My project blog

[Christian Couder]

Hi Matheus,

On Tue, Aug 13, 2019 at 6:06 AM Matheus Tavares Bernardino
<matheus.bernardino@usp.br> wrote:
>
> I just posted a new update on my project:
> https://matheustavares.gitlab.io/posts/simplified-version-of-parallel-inflation

Great blog post as usual!

> This week I sent a simplified version of the series I was working on
> last week (which would still take some time to finish). And I'm
> working to improve it with support to --textconv and
> --recurse-submodules in the added optimizations. I've been also
> working on a patch to make git-grep stop adding submodules to the
> alternates list.

I don't know what refactoring you think is needed on your patch:

https://github.com/matheustavares/git/commit/05536a503b3315d36ad13276af7728adf674ed51

I think it's worth sending to the list with or without addressing the
small nits that I commented on.

Thanks,
Christian.

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

Hi, everyone

I just posted the penultimate report on my project:
https://matheustavares.gitlab.io/posts/going-for-a-too-big-step This
week I’ve been working on a v2 of threaded git-grep w/ parallel
inflation, to allow threads when grepping submodules. I also tried
some more optimizations along the way.

As always, any comments/suggestions will be highly appreciated.

Thanks,
Matheus

Re: [GSoC] My project blog

[Olga Telezhnaya]

вт, 20 авг. 2019 г. в 07:59, Matheus Tavares Bernardino
<matheus.bernardino@usp.br>:
>
> Hi, everyone
>
> I just posted the penultimate report on my project:
> https://matheustavares.gitlab.io/posts/going-for-a-too-big-step This
> week I’ve been working on a v2 of threaded git-grep w/ parallel
> inflation, to allow threads when grepping submodules. I also tried
> some more optimizations along the way.

Thank you for great blog post! You have done so many things, it is impressive.

I absolutely agree with your plans to structure the job and split it
to smaller pieces, it should definitely help. Moreover, if you want,
you can make something like a documentation and delegate some parts.

I hope you enjoyed this summer :) Thank you for your readiness to
continue contributing after GSoC. It is not mandatory, you do not have
to finish the project, but if you want - all the community will be so
glad to see you as the contributor.

Thank you so much,
Olga

>
> As always, any comments/suggestions will be highly appreciated.
>
> Thanks,
> Matheus

Re: [GSoC] My project blog

[Christian Couder]

Hi Matheus,

On Tue, Aug 20, 2019 at 1:28 PM Olga Telezhnaya
<olyatelezhnaya@gmail.com> wrote:
>
> вт, 20 авг. 2019 г. в 07:59, Matheus Tavares Bernardino
> <matheus.bernardino@usp.br>:
> >
> > I just posted the penultimate report on my project:
> > https://matheustavares.gitlab.io/posts/going-for-a-too-big-step This
> > week I’ve been working on a v2 of threaded git-grep w/ parallel
> > inflation, to allow threads when grepping submodules. I also tried
> > some more optimizations along the way.
>
> Thank you for great blog post! You have done so many things, it is impressive.

Yeah, great blog post again!

> I absolutely agree with your plans to structure the job and split it
> to smaller pieces, it should definitely help. Moreover, if you want,
> you can make something like a documentation and delegate some parts.

I also agree with the above. I think it would be really nice for
example if we could have something in Documentation/technical/
describing how multi-threading in `git grep` works and how it could be
improved. (You can of course copy paste from your blog.)

About static function-scope variables I wonder if we could have one
day some .h and .c files with only thread safe functions in them.

> I hope you enjoyed this summer :) Thank you for your readiness to
> continue contributing after GSoC. It is not mandatory, you do not have
> to finish the project, but if you want - all the community will be so
> glad to see you as the contributor.

Thank you for your work and for considering continuing after the GSoC,
Christian.

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

On Tue, Aug 20, 2019 at 9:16 AM Christian Couder
<christian.couder@gmail.com> wrote:
>
> Hi Matheus,
>
> On Tue, Aug 20, 2019 at 1:28 PM Olga Telezhnaya
> <olyatelezhnaya@gmail.com> wrote:
> >
> > вт, 20 авг. 2019 г. в 07:59, Matheus Tavares Bernardino
> > <matheus.bernardino@usp.br>:
> > >
> > > I just posted the penultimate report on my project:
> > > https://matheustavares.gitlab.io/posts/going-for-a-too-big-step This
> > > week I’ve been working on a v2 of threaded git-grep w/ parallel
> > > inflation, to allow threads when grepping submodules. I also tried
> > > some more optimizations along the way.
> >
> > Thank you for great blog post! You have done so many things, it is impressive.
>
> Yeah, great blog post again!

Thank you both for the always encouraging comments :)

> > I absolutely agree with your plans to structure the job and split it
> > to smaller pieces, it should definitely help. Moreover, if you want,
> > you can make something like a documentation and delegate some parts.
>
> I also agree with the above. I think it would be really nice for
> example if we could have something in Documentation/technical/
> describing how multi-threading in `git grep` works and how it could be
> improved. (You can of course copy paste from your blog.)

Hm, good idea! I would have to revisit the important points and better
formulate them, but that could hopefully help in future optimizations
to git-grep.

> About static function-scope variables I wonder if we could have one
> day some .h and .c files with only thread safe functions in them.

Yes, that would probably require quite some work but would definitely
be very helpful!

> > I hope you enjoyed this summer :) Thank you for your readiness to
> > continue contributing after GSoC. It is not mandatory, you do not have
> > to finish the project, but if you want - all the community will be so
> > glad to see you as the contributor.
>
> Thank you for your work and for considering continuing after the GSoC,

Thank you both, again. It's been a great summer of code! (although
it's winter here haha) I do hope to finish the project and keep
contributing with whatever I can :)

Best,
Matheus

Re: [GSoC] My project blog

[Matheus Tavares Bernardino]

Hi, everyone

Just wanted to share with you the final report on my GSoC project:
https://matheustavares.gitlab.io/posts/gsoc-final-report  I plan to
continue the project (as it isn't finished yet) and keep contributing
with whatever I can :) Thank you all for the support during this SoC!

Best,
Matheus