[PATCH] mingw: use OpenSSL's SHA-1 routines

[PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Schindelin]

From: Jeff Hostetler <jeffhost@microsoft.com>

Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines.
This improves performance on SHA1 operations on Intel processors.

OpenSSL 1.0.2 has made considerable performance improvements and
support the Intel hardware acceleration features.  See:
https://software.intel.com/en-us/articles/improving-openssl-performance
https://software.intel.com/en-us/articles/intel-sha-extensions

To test this I added/staged a single file in a gigantic
repository having a 450MB index file.  The code in read-cache.c
verifies the header SHA as it reads the index and computes a new
header SHA as it writes out the new index.  Therefore, in this test
the SHA code must process 900MB of data.  Testing was done on an
Intel I7-4770 CPU @ 3.40GHz (Intel64, Family 6, Model 60) CPU.

The block-sha1 version averaged 5.27 seconds.
The OpenSSL    version averaged 4.50 seconds.

================================================================

$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk

real    0m5.207s
user    0m0.000s
sys     0m0.250s

$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk

real    0m5.362s
user    0m0.015s
sys     0m0.234s

$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk

real    0m5.300s
user    0m0.016s
sys     0m0.250s

$ echo xxx >> project.mk
$ time /e/blk_sha/bin/git.exe add project.mk

real    0m5.216s
user    0m0.000s
sys     0m0.250s

================================================================
$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk

real    0m4.431s
user    0m0.000s
sys     0m0.250s

$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk

real    0m4.478s
user    0m0.000s
sys     0m0.265s

$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk

real    0m4.690s
user    0m0.000s
sys     0m0.250s

$ echo xxx >> project.mk
$ time /e/openssl/bin/git.exe add project.mk

real    0m4.420s
user    0m0.000s
sys     0m0.234s

================================================================

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
Published-As: https://github.com/dscho/git/releases/tag/mingw-openssl-sha1-v1
Fetch-It-Via: git fetch https://github.com/dscho/git mingw-openssl-sha1-v1

 config.mak.uname | 1 -
 1 file changed, 1 deletion(-)

diff --git a/config.mak.uname b/config.mak.uname
index 447f36ac2e..a07936da8b 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -515,7 +515,6 @@ ifneq (,$(findstring MINGW,$(uname_S)))
 	OBJECT_CREATION_USES_RENAMES = UnfortunatelyNeedsTo
 	NO_REGEX = YesPlease
 	NO_PYTHON = YesPlease
-	BLK_SHA1 = YesPlease
 	ETAGS_TARGET = ETAGS
 	NO_INET_PTON = YesPlease
 	NO_INET_NTOP = YesPlease

base-commit: 6e3a7b3398559305c7a239a42e447c21a8f39ff8
-- 
2.11.1.windows.1

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Junio C Hamano]

Johannes Schindelin <johannes.schindelin@gmx.de> writes:

> From: Jeff Hostetler <jeffhost@microsoft.com>
>
> Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines.
> This improves performance on SHA1 operations on Intel processors.
> ...
>
> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---

Nice.  Will queue as jh/mingw-openssl-sha1 topic; it is a bit too
late for today's integration cycle to be merged to 'next', but let's
have this by the end of the week in 'master'.

Thanks.

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Sixt]

Am 10.02.2017 um 00:41 schrieb Junio C Hamano:
> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
>
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines.
>> This improves performance on SHA1 operations on Intel processors.
>> ...
>>
>> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>> ---
>
> Nice.  Will queue as jh/mingw-openssl-sha1 topic; it is a bit too
> late for today's integration cycle to be merged to 'next', but let's
> have this by the end of the week in 'master'.

Please don't rush this through. I didn't have a chance to cross-check 
the patch; it will have to wait for Monday. I would like to address 
Peff's concerns about additional runtime dependencies.

-- Hannes

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> Am 10.02.2017 um 00:41 schrieb Junio C Hamano:
>> ...
>> Nice.  Will queue as jh/mingw-openssl-sha1 topic; it is a bit too
>> late for today's integration cycle to be merged to 'next', but let's
>> have this by the end of the week in 'master'.
>
> Please don't rush this through. I didn't have a chance to cross-check
> the patch; it will have to wait for Monday. I would like to address
> Peff's concerns about additional runtime dependencies.

OK.  Will mark it as "Will cook in 'next'" for now.

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Schindelin]

Hi Hannes,

On Sat, 11 Feb 2017, Johannes Sixt wrote:

> Am 10.02.2017 um 00:41 schrieb Junio C Hamano:
> > Johannes Schindelin <johannes.schindelin@gmx.de> writes:
> >
> > > From: Jeff Hostetler <jeffhost@microsoft.com>
> > >
> > > Use OpenSSL's SHA-1 routines rather than builtin block-sha1
> > > routines.  This improves performance on SHA1 operations on Intel
> > > processors.
> > > ...
> > >
> > > Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
> > > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> > > ---
> >
> > Nice.  Will queue as jh/mingw-openssl-sha1 topic; it is a bit too
> > late for today's integration cycle to be merged to 'next', but let's
> > have this by the end of the week in 'master'.
> 
> Please don't rush this through. I didn't have a chance to cross-check the
> patch; it will have to wait for Monday. I would like to address Peff's
> concerns about additional runtime dependencies.

I never meant this to be fast-tracked into git.git. We have all the time
in our lives to get this in, as Git for Windows already carries this patch
for a while, and shall continue to do so.

Ciao,
Dscho

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Sixt]

Am 13.02.2017 um 18:16 schrieb Johannes Schindelin:
> On Sat, 11 Feb 2017, Johannes Sixt wrote:
>> Am 10.02.2017 um 00:41 schrieb Junio C Hamano:
>>> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
>>>
>>>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>>>
>>>> Use OpenSSL's SHA-1 routines rather than builtin block-sha1
>>>> routines.  This improves performance on SHA1 operations on Intel
>>>> processors.
>>>> ...
>>>>
>>>> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
>>>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>>>> ---
>>>
>>> Nice.  Will queue as jh/mingw-openssl-sha1 topic; it is a bit too
>>> late for today's integration cycle to be merged to 'next', but let's
>>> have this by the end of the week in 'master'.
>>
>> Please don't rush this through. I didn't have a chance to cross-check the
>> patch; it will have to wait for Monday. I would like to address Peff's
>> concerns about additional runtime dependencies.
>
> I never meant this to be fast-tracked into git.git. We have all the time
> in our lives to get this in, as Git for Windows already carries this patch
> for a while, and shall continue to do so.

I've been working with this patch for the past few days, and I did not 
notice any disadvantage during interactive work even though there is a 
new dependency on libcrypto.dll.

Here are some unscientific numbers collected during test suite runs:

bash -c "time make -j4 -k test"

with this patch:

real    34m47.242s
user    9m55.827s
sys     25m20.483s

without this patch:

real    34m2.330s
user    9m56.556s
sys     25m5.520s

It looks like BLK_SHA1 has some advantage, but I would not count on 
these figures too much. (I certainly did not sit idly in front of the 
workstation during these tests, for example. That may have skewed the 
numbers somewhat.) (And, no, I'm not going to measure best-of-five 
timings, not even best-of-two. ;)

In summary: Interactive response times do not decline noticably. I do 
not object the patch.

-- Hannes

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Jeff King]

On Thu, Feb 09, 2017 at 11:27:49PM +0100, Johannes Schindelin wrote:

> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines.
> This improves performance on SHA1 operations on Intel processors.
> 
> OpenSSL 1.0.2 has made considerable performance improvements and
> support the Intel hardware acceleration features.  See:
> https://software.intel.com/en-us/articles/improving-openssl-performance
> https://software.intel.com/en-us/articles/intel-sha-extensions
> 
> To test this I added/staged a single file in a gigantic
> repository having a 450MB index file.  The code in read-cache.c
> verifies the header SHA as it reads the index and computes a new
> header SHA as it writes out the new index.  Therefore, in this test
> the SHA code must process 900MB of data.  Testing was done on an
> Intel I7-4770 CPU @ 3.40GHz (Intel64, Family 6, Model 60) CPU.

I think this is only half the story. A heavy-sha1 workload is faster,
which is good. But one of the original reasons to prefer blk-sha1 (at
least on Linux) is that resolving libcrypto.so symbols takes a
non-trivial amount of time. I just timed it again, and it seems to be
consistently 1ms slower to run "git rev-parse --git-dir" on my machine
(from the top-level of a repo).

1ms is mostly irrelevant, but it adds up on scripted workloads that
start a lot of git processes. Whether it's a net win or not depends on
how much sha1 computation you do in your workload versus how many
processes you start.

I don't know what that means for Windows, though. My impression is that
process startup is so painfully slow there that the link time may just
be lost in the noise. It may just always be a win there. So not really
an objection to your patch, but something you may want to consider.

(Of course, it would in theory be possible to have the best of both
worlds either by static-linking openssl, or by teaching block-sha1 the
same optimizations, but both of those are obviously much more complex).

-Peff

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Schindelin]

Hi Peff,

On Fri, 10 Feb 2017, Jeff King wrote:

> On Thu, Feb 09, 2017 at 11:27:49PM +0100, Johannes Schindelin wrote:
> 
> > From: Jeff Hostetler <jeffhost@microsoft.com>
> > 
> > Use OpenSSL's SHA-1 routines rather than builtin block-sha1 routines.
> > This improves performance on SHA1 operations on Intel processors.
> > 
> > OpenSSL 1.0.2 has made considerable performance improvements and
> > support the Intel hardware acceleration features.  See:
> > https://software.intel.com/en-us/articles/improving-openssl-performance
> > https://software.intel.com/en-us/articles/intel-sha-extensions
> > 
> > To test this I added/staged a single file in a gigantic repository
> > having a 450MB index file.  The code in read-cache.c verifies the
> > header SHA as it reads the index and computes a new header SHA as it
> > writes out the new index.  Therefore, in this test the SHA code must
> > process 900MB of data.  Testing was done on an Intel I7-4770 CPU @
> > 3.40GHz (Intel64, Family 6, Model 60) CPU.
> 
> I think this is only half the story. A heavy-sha1 workload is faster,
> which is good. But one of the original reasons to prefer blk-sha1 (at
> least on Linux) is that resolving libcrypto.so symbols takes a
> non-trivial amount of time. I just timed it again, and it seems to be
> consistently 1ms slower to run "git rev-parse --git-dir" on my machine
> (from the top-level of a repo).
> 
> 1ms is mostly irrelevant, but it adds up on scripted workloads that
> start a lot of git processes.

You know my answer to that. If scripting slows things down, we should
avoid it in production code. As it is, scripting slows us down. Therefore
I work slowly but steadily to get rid of scripting where it hurts most.

Ciao,
Dscho

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Jeff King]

On Fri, Feb 10, 2017 at 04:49:02PM +0100, Johannes Schindelin wrote:

> > I think this is only half the story. A heavy-sha1 workload is faster,
> > which is good. But one of the original reasons to prefer blk-sha1 (at
> > least on Linux) is that resolving libcrypto.so symbols takes a
> > non-trivial amount of time. I just timed it again, and it seems to be
> > consistently 1ms slower to run "git rev-parse --git-dir" on my machine
> > (from the top-level of a repo).
> > 
> > 1ms is mostly irrelevant, but it adds up on scripted workloads that
> > start a lot of git processes.
> 
> You know my answer to that. If scripting slows things down, we should
> avoid it in production code. As it is, scripting slows us down. Therefore
> I work slowly but steadily to get rid of scripting where it hurts most.

Well, yes. My question is more "what does it look like on normal Git
workloads?". Are you trading off an optimization for your giant 450MB
index workload (which _also_ could be fixed by trying do the slow
operation less, rather than micro-optimizing it) in a way that hurts
people working with more normal sized repos?

For instance, "make BLK_SHA1=Yes test" is measurably faster for me than
"make BLK_SHA1= test".

I'm open to the argument that it doesn't matter in practice for normal
git users. But it's not _just_ scripting. It depends on the user's
pattern of invoking git commands (and how expensive the symbol
resolution is on their system).

-Peff

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Sixt]

Am 10.02.2017 um 17:04 schrieb Jeff King:
> On Fri, Feb 10, 2017 at 04:49:02PM +0100, Johannes Schindelin wrote:
>
>>> I think this is only half the story. A heavy-sha1 workload is faster,
>>> which is good. But one of the original reasons to prefer blk-sha1 (at
>>> least on Linux) is that resolving libcrypto.so symbols takes a
>>> non-trivial amount of time. I just timed it again, and it seems to be
>>> consistently 1ms slower to run "git rev-parse --git-dir" on my machine
>>> (from the top-level of a repo).
>>>
>>> 1ms is mostly irrelevant, but it adds up on scripted workloads that
>>> start a lot of git processes.
>>
>> You know my answer to that. If scripting slows things down, we should
>> avoid it in production code. As it is, scripting slows us down. Therefore
>> I work slowly but steadily to get rid of scripting where it hurts most.
>
> Well, yes. My question is more "what does it look like on normal Git
> workloads?". Are you trading off an optimization for your giant 450MB
> index workload (which _also_ could be fixed by trying do the slow
> operation less, rather than micro-optimizing it) in a way that hurts
> people working with more normal sized repos?
>
> For instance, "make BLK_SHA1=Yes test" is measurably faster for me than
> "make BLK_SHA1= test".

The patch does add a new runtime dependency on libcrypto.dll in my 
environment. I would be surprised if it does not also with your modern 
build tools.

I haven't had time to compare test suite runtimes.

> I'm open to the argument that it doesn't matter in practice for normal
> git users. But it's not _just_ scripting. It depends on the user's
> pattern of invoking git commands (and how expensive the symbol
> resolution is on their system).

It can be argued that in normal interactive use, it is hard to notice 
that another DLL is loaded. Don't forget, though, that on Windows it is 
not only the pure time to resolve the entry points, but also that 
typically virus scanners inspect every executable file that is loaded, 
which adds another share of time.

I'll use the patch for daily work for a while to see whether it hurts.

-- Hannes

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> The patch does add a new runtime dependency on libcrypto.dll in my
> environment. I would be surprised if it does not also with your modern
> build tools.
>
> I haven't had time to compare test suite runtimes.
>
>> I'm open to the argument that it doesn't matter in practice for normal
>> git users. But it's not _just_ scripting. It depends on the user's
>> pattern of invoking git commands (and how expensive the symbol
>> resolution is on their system).
>
> It can be argued that in normal interactive use, it is hard to notice
> that another DLL is loaded. Don't forget, though, that on Windows it
> is not only the pure time to resolve the entry points, but also that
> typically virus scanners inspect every executable file that is loaded,
> which adds another share of time.
>
> I'll use the patch for daily work for a while to see whether it hurts.

Thanks.

I need to ask an unrelated question at a bit higher level, though.

I have been operating under the assumption that everybody on Windows
who builds Git works off of Dscho's Git for Windows tree, and
patches that are specific to Windows from Dscho's are sent to me via
the list only after they have been in Git for Windows and proven to
help Windows users in the wild.  

The consequence of these two assumptions is that I would feel safe
to treat Windows specific changes that do not touch generic part of
the codebase from Dscho just like updates from any other subsystem
maintainers (any git-svn thing from Eric, any gitk thing from Paul,
any p4 thing Luke and Lars are both happy with, etc.).

You seem to be saying that the first of the two assumptions does not
hold.  Should I change my expectations while queuing Windows specific
patches from Dscho?

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Sixt]

Am 13.02.2017 um 20:42 schrieb Junio C Hamano:
> I have been operating under the assumption that everybody on Windows
> who builds Git works off of Dscho's Git for Windows tree, and
> patches that are specific to Windows from Dscho's are sent to me via
> the list only after they have been in Git for Windows and proven to
> help Windows users in the wild.
>
> The consequence of these two assumptions is that I would feel safe
> to treat Windows specific changes that do not touch generic part of
> the codebase from Dscho just like updates from any other subsystem
> maintainers (any git-svn thing from Eric, any gitk thing from Paul,
> any p4 thing Luke and Lars are both happy with, etc.).
>
> You seem to be saying that the first of the two assumptions does not
> hold.  Should I change my expectations while queuing Windows specific
> patches from Dscho?

Your first assumption is incorrect as far as I am concerned. I build 
from your tree plus some topics. During -rc period, I build off of 
master; after a release, I build off of next. I merge some of the topics 
that you carry in pu when I find them interesting or when I suspect them 
to regress on Windows. Then I carry around a few additional patches that 
the public has never seen, and these days I also merge Dscho's rebase-i 
topic.

-- Hannes

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Schindelin]

Hi,

On Mon, 13 Feb 2017, Johannes Sixt wrote:

> Am 13.02.2017 um 20:42 schrieb Junio C Hamano:
> > I have been operating under the assumption that everybody on Windows
> > who builds Git works off of Dscho's Git for Windows tree, and patches
> > that are specific to Windows from Dscho's are sent to me via the list
> > only after they have been in Git for Windows and proven to help
> > Windows users in the wild.
> >
> > The consequence of these two assumptions is that I would feel safe to
> > treat Windows specific changes that do not touch generic part of the
> > codebase from Dscho just like updates from any other subsystem
> > maintainers (any git-svn thing from Eric, any gitk thing from Paul,
> > any p4 thing Luke and Lars are both happy with, etc.).
> >
> > You seem to be saying that the first of the two assumptions does not
> > hold.  Should I change my expectations while queuing Windows specific
> > patches from Dscho?
> 
> Your first assumption is incorrect as far as I am concerned. I build
> from your tree plus some topics. During -rc period, I build off of
> master; after a release, I build off of next. I merge some of the topics
> that you carry in pu when I find them interesting or when I suspect them
> to regress on Windows.  Then I carry around a few additional patches
> that the public has never seen, and these days I also merge Dscho's
> rebase-i topic.

In addition, you build from a custom MINGW/MSys1 setup, correct?

Ciao,
Johannes

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Sixt]

Am 13.02.2017 um 23:38 schrieb Johannes Schindelin:
> In addition, you build from a custom MINGW/MSys1 setup, correct?

Correct. Specifically, I use the build tools from "msysgit" times, but 
build outside the premanufactured build environement; i.e., the 
"THIS_IS_MSYSGIT" section in config.mak.uname is not used.

-- Hannes

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> It can be argued that in normal interactive use, it is hard to notice
> that another DLL is loaded. Don't forget, though, that on Windows it
> is not only the pure time to resolve the entry points, but also that
> typically virus scanners inspect every executable file that is loaded,
> which adds another share of time.
>
> I'll use the patch for daily work for a while to see whether it hurts.

Please ping this thread again when you have something to add.  For
now, I'll demote this patch from 'next' to 'pu' when we rewind and
rebuild 'next' post 2.12 release.

Thanks.

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Johannes Sixt]

Am 24.02.2017 um 22:54 schrieb Junio C Hamano:
> Johannes Sixt <j6t@kdbg.org> writes:
>> I'll use the patch for daily work for a while to see whether it hurts.
>
> Please ping this thread again when you have something to add.  For
> now, I'll demote this patch from 'next' to 'pu' when we rewind and
> rebuild 'next' post 2.12 release.

As I already said in [1], I have no objection. I've used the patch in 
production for some time now and did not notice any slowdowns.

[1] 
https://public-inbox.org/git/0080edd6-a515-2fe9-6266-b6f6bbedfdde@kdbg.org/

-- Hannes

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> Am 24.02.2017 um 22:54 schrieb Junio C Hamano:
>> Johannes Sixt <j6t@kdbg.org> writes:
>>> I'll use the patch for daily work for a while to see whether it hurts.
>>
>> Please ping this thread again when you have something to add.  For
>> now, I'll demote this patch from 'next' to 'pu' when we rewind and
>> rebuild 'next' post 2.12 release.
>
> ... I've used the patch in
> production for some time now and did not notice any slowdowns.

Thanks.  Dscho obviously thinks this is the right thing for Windows,
and you agree.  That's more than sufficient votes to make me feel
safe ;-)

Thanks.

Re: [PATCH] mingw: use OpenSSL's SHA-1 routines

[Linus Torvalds]

On Thu, Feb 9, 2017 at 9:02 PM, Jeff King <peff@peff.net> wrote:
>
> I think this is only half the story. A heavy-sha1 workload is faster,
> which is good. But one of the original reasons to prefer blk-sha1 (at
> least on Linux) is that resolving libcrypto.so symbols takes a
> non-trivial amount of time. I just timed it again, and it seems to be
> consistently 1ms slower to run "git rev-parse --git-dir" on my machine
> (from the top-level of a repo).

Yes. It's also a horrible plain to profile those things.

Avoiding openssl was a great thing, because it avoided a lot of crazy overhead.

I suspect that most of the openssl win comes from using the actual SHA
instructions on modern CPU's. Because last I looked, the hand-coded
assembly simply wasn't that much faster.

We could easily have some x86-specific library that just does "use SHA
instructions if you have them, use blk-sha1 otherwise".

Of course, if we end up using the collision checking SHA libraries
this is all moot anyway.

                 Linus