Request for testing v2.19.0-rc0 *with builtin stash/rebase*

Request for testing v2.19.0-rc0 *with builtin stash/rebase*

[Johannes Schindelin]

Team,

while this mail talks about Git for Windows, please keep in mind that we
try *very* hard to keep Git for Windows' master working correctly not only
on Windows but also on macOS and Linux.

I, for one, run Git built from Git for Windows' `master` branch in my
Linux VMs all the time.

As all of you know by now, the fact that Git was pretty much designed to
work well on Linux is not exactly helping Git for Windows. (Or for that
matter, macOS: Git is substantially slower on macOS than on Linux when
running on the same hardware.) The quickest pay-off comes from converting
Unix shell scripts (which are designed to spawn processes, whereas Windows
is more optimized for multi-threaded applications).

For that reason, I was delighted to see that our Google Summer of Code
pushed pretty hard in that direction. And I could not help myself so I had
to test how much faster things got. Here is the result of my first, really
quick and dirty test:

		without builtin stash/rebase	with builtin stash/rebase
t3400 (rebase)		1m27s				32s
t3404 (rebase -i)	13m15s				3m59s
t3903 (stash)		8m37s				1m18s

What can I say? Even if the numbers are off by as much as 10%, these are
impressive improvements: keep in mind that there is a lot of churn going
on in the test suite because it is itself implemented in Unix shell
scripts (and hence I won't even bother to try more correct performance
benchmarking because that is simply not possible when Unix shell scripts
are in the equation). So the speed improvements of the stash/rebase
commands are *even higher* than this.

So I really, really, really want those builtins in Git for Windows
v2.19.0. At *least* as an option.

Therefore, after creating a pre-release of Git for Windows corresponding
to Git v2.19.0-rc0, I created another one (note the .2 at the end), *with*
those new builtins:
https://github.com/git-for-windows/git/releases/tag/v2.19.0-rc0.windows.2

I would like to ask you for help in dog-fooding this. In particular if you
are a heavy stash/rebase user (like I am), it would be helpful to really
kick those tires.

Of course, those are only Windows binaries on that web page, but it should
be easy to compile from that tag on Linux and macOS. I could also build a
macOS installer and add it to that pre-release, is there interest in that?

Currently I am uncertain whether I should spend the time to reinstate the
old scripted `git stash` guarded by `stash.useBuiltin` and the old
scripted `git rebase -i` (guarded by the same `rebase.useBuiltin` that
guards the scripted `git rebase`), as a fall-back (or make the new
builtins opt-ins instead of opt-outs).

So far, I have not encountered any serious bug, but then, I did not really
have a chance to use it yet (I installed it, of course, but I have not
done any serious rebasing yet). So your help will be crucial in
determining where I need to spend time.

Thanks,
Johannes

Re: [git-for-windows] Request for testing v2.19.0-rc0 *with builtin stash/rebase*

[Johannes Schindelin]

Team,

On Fri, 24 Aug 2018, Johannes Schindelin wrote:

> while this mail talks about Git for Windows, please keep in mind that we
> try *very* hard to keep Git for Windows' master working correctly not only
> on Windows but also on macOS and Linux.
> 
> I, for one, run Git built from Git for Windows' `master` branch in my
> Linux VMs all the time.
> 
> As all of you know by now, the fact that Git was pretty much designed to
> work well on Linux is not exactly helping Git for Windows. (Or for that
> matter, macOS: Git is substantially slower on macOS than on Linux when
> running on the same hardware.) The quickest pay-off comes from converting
> Unix shell scripts (which are designed to spawn processes, whereas Windows
> is more optimized for multi-threaded applications).
> 
> For that reason, I was delighted to see that our Google Summer of Code
> pushed pretty hard in that direction. And I could not help myself so I had
> to test how much faster things got. Here is the result of my first, really
> quick and dirty test:
> 
> 		without builtin stash/rebase	with builtin stash/rebase
> t3400 (rebase)		1m27s				32s
> t3404 (rebase -i)	13m15s				3m59s
> t3903 (stash)		8m37s				1m18s
> 
> What can I say? Even if the numbers are off by as much as 10%, these are
> impressive improvements: keep in mind that there is a lot of churn going
> on in the test suite because it is itself implemented in Unix shell
> scripts (and hence I won't even bother to try more correct performance
> benchmarking because that is simply not possible when Unix shell scripts
> are in the equation). So the speed improvements of the stash/rebase
> commands are *even higher* than this.
> 
> So I really, really, really want those builtins in Git for Windows
> v2.19.0. At *least* as an option.
> 
> Therefore, after creating a pre-release of Git for Windows corresponding
> to Git v2.19.0-rc0, I created another one (note the .2 at the end), *with*
> those new builtins:
> https://github.com/git-for-windows/git/releases/tag/v2.19.0-rc0.windows.2
> 
> I would like to ask you for help in dog-fooding this. In particular if you
> are a heavy stash/rebase user (like I am), it would be helpful to really
> kick those tires.
> 
> Of course, those are only Windows binaries on that web page, but it should
> be easy to compile from that tag on Linux and macOS. I could also build a
> macOS installer and add it to that pre-release, is there interest in that?
> 
> Currently I am uncertain whether I should spend the time to reinstate the
> old scripted `git stash` guarded by `stash.useBuiltin` and the old
> scripted `git rebase -i` (guarded by the same `rebase.useBuiltin` that
> guards the scripted `git rebase`), as a fall-back (or make the new
> builtins opt-ins instead of opt-outs).
> 
> So far, I have not encountered any serious bug, but then, I did not really
> have a chance to use it yet (I installed it, of course, but I have not
> done any serious rebasing yet). So your help will be crucial in
> determining where I need to spend time.

Thomas Gummerer pointed out just how dangerously I want to live, so I did
implement the `stash.useBuiltin` option (and extended the
`rebase.useBuiltin` option to also cover `rebase -i`), and made the use of
those builtins an opt-in, i.e. still default to the scripted versions.

You can see my progress in this PR (which I mainly opened to have this
tested on Windows, macOS and Linux):
https://github.com/git-for-windows/git/pull/1800

Ciao,
Johannes

Re: Request for testing v2.19.0-rc0 *with builtin stash/rebase*

[Bryan Turner]

On Fri, Aug 24, 2018 at 5:14 AM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> For that reason, I was delighted to see that our Google Summer of Code
> pushed pretty hard in that direction. And I could not help myself so I had
> to test how much faster things got. Here is the result of my first, really
> quick and dirty test:
>
>                 without builtin stash/rebase    with builtin stash/rebase
> t3400 (rebase)          1m27s                           32s
> t3404 (rebase -i)       13m15s                          3m59s
> t3903 (stash)           8m37s                           1m18s
>
> What can I say? Even if the numbers are off by as much as 10%, these are
> impressive improvements: keep in mind that there is a lot of churn going
> on in the test suite because it is itself implemented in Unix shell
> scripts (and hence I won't even bother to try more correct performance
> benchmarking because that is simply not possible when Unix shell scripts
> are in the equation). So the speed improvements of the stash/rebase
> commands are *even higher* than this.

Thanks for taking the time to make these pre-releases available. I
appreciate the effort. And the same to Junio, for always posting
release candidates. We rely on them heavily to find changes that might
cause issues before admins start upgrading in the wild and find them
for us.

I downloaded both the rc0.1 and rc0.2 builds, as well as 2.18.0, and
ran them all through Bitbucket Server's test suite a few times (to
ensure warm disk for comparable numbers). I added support for some
"simple" rebase cases a few releases ago, so we have a set of tests
that verify the rebase behaviors we use. (We don't use git stash, so
we don't have any tests in our suite for that.)

Running our entire Git test suite (~1,600 tests) against Git for
Windows 2.18.0 takes ~5 minutes, and 2.19.0-rc0.1 produced an almost
identical duration. Running our tests against rc0.2 cut the duration
down to 4 minutes. There were no test failures on either pre-release
build.

To try and get a better sense of the rebase performance improvement
specifically, I filtered down to a set of 14 specific tests which use
it and ran those. On Git 2.18, those 14 tests take just over 19
seconds. On 2.19.0-rc0.2, they take just over 8 seconds.

When they do ship, whether it's in 2.19 (by default or optional) or
later, the changes definitely offer some significant performance wins.

Thanks again, to everyone involved, for all the effort that went into
designing, implementing, reviewing and releasing these improvements.
As someone who develops under Windows most of the time, they make a
big difference in my day to day work. And that's not to mention all
the Bitbucket Server and Bitbucket Data Center users who will enjoy a
snappier experience as these changes make their way out into the wild.

Best regards,
Bryan Turner

Re: Request for testing v2.19.0-rc0 *with builtin stash/rebase*

[Johannes Schindelin]

Hi Bryan,

On Fri, 24 Aug 2018, Bryan Turner wrote:

> On Fri, Aug 24, 2018 at 5:14 AM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> >
> > For that reason, I was delighted to see that our Google Summer of Code
> > pushed pretty hard in that direction. And I could not help myself so I had
> > to test how much faster things got. Here is the result of my first, really
> > quick and dirty test:
> >
> >                 without builtin stash/rebase    with builtin stash/rebase
> > t3400 (rebase)          1m27s                           32s
> > t3404 (rebase -i)       13m15s                          3m59s
> > t3903 (stash)           8m37s                           1m18s
> >
> > What can I say? Even if the numbers are off by as much as 10%, these are
> > impressive improvements: keep in mind that there is a lot of churn going
> > on in the test suite because it is itself implemented in Unix shell
> > scripts (and hence I won't even bother to try more correct performance
> > benchmarking because that is simply not possible when Unix shell scripts
> > are in the equation). So the speed improvements of the stash/rebase
> > commands are *even higher* than this.
> 
> Thanks for taking the time to make these pre-releases available. I
> appreciate the effort. And the same to Junio, for always posting
> release candidates. We rely on them heavily to find changes that might
> cause issues before admins start upgrading in the wild and find them
> for us.
> 
> I downloaded both the rc0.1 and rc0.2 builds, as well as 2.18.0, and
> ran them all through Bitbucket Server's test suite a few times (to
> ensure warm disk for comparable numbers). I added support for some
> "simple" rebase cases a few releases ago, so we have a set of tests
> that verify the rebase behaviors we use. (We don't use git stash, so
> we don't have any tests in our suite for that.)
> 
> Running our entire Git test suite (~1,600 tests) against Git for
> Windows 2.18.0 takes ~5 minutes, and 2.19.0-rc0.1 produced an almost
> identical duration. Running our tests against rc0.2 cut the duration
> down to 4 minutes. There were no test failures on either pre-release
> build.
> 
> To try and get a better sense of the rebase performance improvement
> specifically, I filtered down to a set of 14 specific tests which use
> it and ran those. On Git 2.18, those 14 tests take just over 19
> seconds. On 2.19.0-rc0.2, they take just over 8 seconds.
> 
> When they do ship, whether it's in 2.19 (by default or optional) or
> later, the changes definitely offer some significant performance wins.
> 
> Thanks again, to everyone involved, for all the effort that went into
> designing, implementing, reviewing and releasing these improvements.
> As someone who develops under Windows most of the time, they make a
> big difference in my day to day work. And that's not to mention all
> the Bitbucket Server and Bitbucket Data Center users who will enjoy a
> snappier experience as these changes make their way out into the wild.

Thank you for the thorough testing!

As a heads-up: after Thomas Gummerer offered some cautious thoughts, I
decided to work on the stash.useBuiltin support after all: the Git for
Windows v2.19.0 installer will have builtin stash and rebase as two
experimental options, both defaulting to off (to protect not only Git for
Windows' users, but also the maintainer's inbox).

Likewise, portable Git and MinGit will support stash.useBuiltin and
rebase.useBuiltin, defaulting to `false` for both.

Thanks,
Johannes