From: Johannes Schindelin <Johannes.Schindelin@gmx.de> To: Ævar Arnfjörð Bjarmason <firstname.lastname@example.org> Cc: "brian m. carlson" <email@example.com>, Adam Langley <firstname.lastname@example.org>, Jeff King <email@example.com>, Mike Hommey <firstname.lastname@example.org>, Brandon Williams <email@example.com>, Linus Torvalds <firstname.lastname@example.org>, Jonathan Nieder <email@example.com>, Git Mailing List <firstname.lastname@example.org>, Stefan Beller <email@example.com>, Jonathan Tan <firstname.lastname@example.org>, Junio Hamano <email@example.com> Subject: Re: Which hash function to use, was Re: RFC: Another proposed hash function transition plan Date: Fri, 16 Jun 2017 15:24:19 +0200 (CEST) Message-ID: <alpine.DEB.188.8.131.526161438470.4200@virtualbox> (raw) In-Reply-To: <firstname.lastname@example.org> [-- Attachment #1: Type: text/plain, Size: 7027 bytes --] Hi, On Fri, 16 Jun 2017, Ævar Arnfjörð Bjarmason wrote: > On Fri, Jun 16 2017, brian m. carlson jotted: > > > On Fri, Jun 16, 2017 at 01:36:13AM +0200, Ævar Arnfjörð Bjarmason wrote: > > > >> So I don't follow the argument that we shouldn't weigh future HW > >> acceleration highly just because you can't easily buy a laptop today > >> with these features. > >> > >> Aside from that I think you've got this backwards, it's AMD that's > >> adding SHA acceleration to their high-end Ryzen chips but Intel is > >> starting at the lower end this year with Goldmont which'll be in > >> lower-end consumer devices. If you read the github issue I linked > >> to upthread you can see that the cryptopp devs already tested > >> their SHA accelerated code on a consumer Celeron recently. > >> > >> I don't think Intel has announced the SHA extensions for future Xeon > >> releases, but it seems given that they're going to have it there as > >> well. Have there every been x86 extensions that aren't eventually > >> portable across the entire line, or that they've ended up removing > >> from x86 once introduced? > >> > >> In any case, I think by the time we're ready to follow-up the current > >> hash refactoring efforts with actually changing the hash > >> implementation many of us are likely to have laptops with these > >> extensions, making this easy to test. > > > > I think you underestimate the life of hardware and software. I have > > servers running KVM development instances that have been running since > > at least 2012. Those machines are not scheduled for replacement > > anytime soon. > > > > Whatever we deploy within the next year is going to run on existing > > hardware for probably a decade, whether we want it to or not. Most of > > those machines don't have acceleration. > > To clarify, I'm not dismissing the need to consider existing hardware > without these acceleration functions or future processors without them. > I don't think that makes any sense, we need to keep those in mind. > > I was replying to a bit in your comment where you (it seems to me) were > making the claim that we shouldn't consider the HW acceleration of > certain hash functions either. Yes, I also had the impression that it stressed the status quo quite a bit too much. We know for a fact that SHA-256 acceleration is coming to consumer CPUs. We know of no plans for any of the other mentioned hash functions to hardware-accelerate them in consumer CPUs. And remember: for those who are affected most (humongous monorepos, source code hosters), upgrading hardware is less of an issue than having a secure hash function for the rest of us. And while I am really thankful that Adam chimed in, I think he would agree that BLAKE2 is a purposefully weakened version of BLAKE, for the benefit of speed (with the caveat that one of my experts disagrees that BLAKE2b would be faster than hardware-accelerated SHA-256). And while BLAKE has seen roughly equivalent cryptanalysis as Keccak (which became SHA-3), BLAKE2 has not. That makes me *very* uneasy about choosing BLAKE2. > > Furthermore, you need a reasonably modern crypto library to get hardware > > acceleration. OpenSSL has only recently gained support for it. RHEL 7 > > does not currently support it, and probably never will. That OS is > > going to be around for the next 6 years. > > > > If we're optimizing for performance, I don't want to optimize for the > > latest, greatest machines. Those machines are going to outperform > > everything else either way. I'd rather optimize for something which > > performs well on the whole everywhere. There are a lot of developers > > who have older machines, for cost reasons or otherwise. > > We have real data showing that the intersection between people who care > about the hash slowing down and those who can't afford the latest > hardware is pretty much nil. > > I.e. in 2.13.0 SHA-1 got slower, and pretty much nobody noticed or cared > except Johannes Schindelin, myself & Christian Couder. This is because > in practice hashing only becomes a bottleneck on huge monorepos that > need to e.g. re-hash the contents of a huge index. Indeed. I am still concerned about that. As you mention, though, it really only affects users of ginormous monorepos, and of course source code hosters. The jury's still out on how much it impacts my colleagues, by the way. I have no doubt that Visual Studio Team Services, GitHub and Atlassian will eventually end up with FPGAs for hash computation. So that's that. Side note: BLAKE is actually *not* friendly to hardware acceleration, I have been told by one cryptography expert. In contrast, the Keccak team claims SHA3-256 to be the easiest to hardware-accelerate, making it "a green cryptographic primitive": http://keccak.noekeon.org/is_sha3_slow.html > > Here are some stats (cycles/byte for long messages): > > > > SHA-256 BLAKE2b > > Ryzen 1.89 3.06 > > Knight's Landing 19.00 5.65 > > Cortex-A72 1.99 5.48 > > Cortex-A57 11.81 5.47 > > Cortex-A7 28.19 15.16 > > > > In other words, BLAKE2b performs well uniformly across a wide variety of > > architectures even without acceleration. I'd rather tell people that > > upgrading to a new hash algorithm is a performance win either way, not > > just if they have the latest hardware. > > Yup, all of those need to be considered, although given my comment above > about big repos a 40% improvement on Ryzen (a processor likely to be > used for big repos) stands out, where are those numbers from, and is > that with or without HW accel for SHA-256 on Ryzen? When it comes to BLAKE2, I would actually strongly suggest to consider the amount of attempts to break it. Or rather, how much less attention it got than, say, SHA-256. In any case, I have been encouraged to stress the importance of "crypto-agility", i.e. the ability to switch to another algorithm when the current one gets broken "enough". And I am delighted that that is exactly the direction we are going. In other words, even if I still think (backed up by the experts on whose knowledge I lean heavily to form my opinions) that SHA-256 would be the best choice for now, it should be relatively easy to offer BLAKE2b support for (and by [*1*]) those who want it. Ciao, Dscho Footnote *1*: I say that the support for BLAKE2b should come from those parties who desire it also because it is not as ubiquituous as SHA-256. Hence, it would add the burden of having a performant and reasonably bug-free implementation in Git's source tree. IIUC OpenSSL added BLAKE2b support only in OpenSSL 1.1.0, the 1.0.2 line (which is still in use in many places, e.g. Git for Windows' SDK) does not, meaning: Git's implementation would be the one *everybody* relies on, with *no* fall-back.
next prev parent reply index Thread overview: 113+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-03-04 1:12 Jonathan Nieder 2017-03-05 2:35 ` Linus Torvalds 2017-03-06 0:26 ` brian m. carlson 2017-03-06 18:24 ` Brandon Williams 2017-06-15 10:30 ` Which hash function to use, was " Johannes Schindelin 2017-06-15 11:05 ` Mike Hommey 2017-06-15 13:01 ` Jeff King 2017-06-15 16:30 ` Ævar Arnfjörð Bjarmason 2017-06-15 19:34 ` Johannes Schindelin 2017-06-15 21:59 ` Adam Langley 2017-06-15 22:41 ` brian m. carlson 2017-06-15 23:36 ` Ævar Arnfjörð Bjarmason 2017-06-16 0:17 ` brian m. carlson 2017-06-16 6:25 ` Ævar Arnfjörð Bjarmason 2017-06-16 13:24 ` Johannes Schindelin [this message] 2017-06-16 17:38 ` Adam Langley 2017-06-16 20:52 ` Junio C Hamano 2017-06-16 21:12 ` Junio C Hamano 2017-06-16 21:24 ` Jonathan Nieder 2017-06-16 21:39 ` Ævar Arnfjörð Bjarmason 2017-06-16 20:42 ` Jeff King 2017-06-19 9:26 ` Johannes Schindelin 2017-06-15 21:10 ` Mike Hommey 2017-06-16 4:30 ` Jeff King 2017-06-15 17:36 ` Brandon Williams 2017-06-15 19:20 ` Junio C Hamano 2017-06-15 19:13 ` Jonathan Nieder 2017-03-07 0:17 ` RFC v3: " Jonathan Nieder 2017-03-09 19:14 ` Shawn Pearce 2017-03-09 20:24 ` Jonathan Nieder 2017-03-10 19:38 ` Jeff King 2017-03-10 19:55 ` Jonathan Nieder 2017-09-28 4:43 ` [PATCH v4] technical doc: add a design doc for hash function transition Jonathan Nieder 2017-09-29 6:06 ` Junio C Hamano 2017-09-29 8:09 ` Junio C Hamano 2017-09-29 17:34 ` Jonathan Nieder 2017-10-02 8:25 ` Junio C Hamano 2017-10-02 19:41 ` Jason Cooper 2017-10-02 9:02 ` Junio C Hamano 2017-10-02 19:23 ` Jason Cooper 2017-10-03 5:40 ` Junio C Hamano 2017-10-03 13:08 ` Jason Cooper 2017-10-04 1:44 ` Junio C Hamano 2017-09-06 6:28 ` RFC v3: Another proposed hash function transition plan Junio C Hamano 2017-09-08 2:40 ` Junio C Hamano 2017-09-08 3:34 ` Jeff King 2017-09-11 18:59 ` Brandon Williams 2017-09-13 12:05 ` Johannes Schindelin 2017-09-13 13:43 ` demerphq 2017-09-13 22:51 ` Jonathan Nieder 2017-09-14 18:26 ` Johannes Schindelin 2017-09-14 18:40 ` Jonathan Nieder 2017-09-14 22:09 ` Johannes Schindelin 2017-09-13 23:30 ` Linus Torvalds 2017-09-14 18:45 ` Johannes Schindelin 2017-09-18 12:17 ` Gilles Van Assche 2017-09-18 22:16 ` Johannes Schindelin 2017-09-19 16:45 ` Gilles Van Assche 2017-09-29 13:17 ` Johannes Schindelin 2017-09-29 14:54 ` Joan Daemen 2017-09-29 22:33 ` Johannes Schindelin 2017-09-30 22:02 ` Joan Daemen 2017-10-02 14:26 ` Johannes Schindelin 2017-09-18 22:25 ` Jonathan Nieder 2017-09-26 17:05 ` Jason Cooper 2017-09-26 22:11 ` Johannes Schindelin 2017-09-26 22:25 ` [PATCH] technical doc: add a design doc for hash function transition Stefan Beller 2017-09-26 23:38 ` Jonathan Nieder 2017-09-26 23:51 ` RFC v3: Another proposed hash function transition plan Jonathan Nieder 2017-10-02 14:54 ` Jason Cooper 2017-10-02 16:50 ` Brandon Williams 2017-10-02 14:00 ` Jason Cooper 2017-10-02 17:18 ` Linus Torvalds 2017-10-02 19:37 ` Jeff King 2017-09-13 16:30 ` Jonathan Nieder 2017-09-13 21:52 ` Junio C Hamano 2017-09-13 22:07 ` Stefan Beller 2017-09-13 22:18 ` Jonathan Nieder 2017-09-14 2:13 ` Junio C Hamano 2017-09-14 15:23 ` Johannes Schindelin 2017-09-14 15:45 ` demerphq 2017-09-14 22:06 ` Johannes Schindelin 2017-09-13 22:15 ` Junio C Hamano 2017-09-13 22:27 ` Jonathan Nieder 2017-09-14 2:10 ` Junio C Hamano 2017-09-14 12:39 ` Johannes Schindelin 2017-09-14 16:36 ` Brandon Williams 2017-09-14 18:49 ` Jonathan Nieder 2017-09-15 20:42 ` Philip Oakley 2017-03-05 11:02 ` RFC: " David Lang [not found] ` <CA+dhYEXHbQfJ6KUB1tWS9u1MLEOJL81fTYkbxu4XO-i+379LPw@mail.gmail.com> 2017-03-06 9:43 ` Jeff King 2017-03-06 23:40 ` Jonathan Nieder 2017-03-07 0:03 ` Mike Hommey 2017-03-06 8:43 ` Jeff King 2017-03-06 18:39 ` Jonathan Tan 2017-03-06 19:22 ` Linus Torvalds 2017-03-06 19:59 ` Brandon Williams 2017-03-06 21:53 ` Junio C Hamano 2017-03-07 8:59 ` Jeff King 2017-03-06 18:43 ` Junio C Hamano 2017-03-07 18:57 ` Ian Jackson 2017-03-07 19:15 ` Linus Torvalds 2017-03-08 11:20 ` Ian Jackson 2017-03-08 15:37 ` Johannes Schindelin 2017-03-08 15:40 ` Johannes Schindelin 2017-03-20 5:21 ` Use base32? Jason Hennessey 2017-03-20 5:58 ` Michael Steuer 2017-03-20 8:05 ` Jacob Keller 2017-03-21 3:07 ` Michael Steuer 2017-03-13 9:24 ` RFC: Another proposed hash function transition plan The Keccak Team 2017-03-13 17:48 ` Jonathan Nieder 2017-03-13 18:34 ` ankostis 2017-03-17 11:07 ` Johannes Schindelin
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://vger.kernel.org/majordomo-info.html * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=alpine.DEB.184.108.40.2066161438470.4200@virtualbox \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
firstname.lastname@example.org list mirror (unofficial, one of many) Archives are clonable: git clone --mirror https://public-inbox.org/git git clone --mirror http://ou63pmih66umazou.onion/git git clone --mirror http://czquwvybam4bgbro.onion/git git clone --mirror http://hjrcffqmbrq6wope.onion/git Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.version-control.git nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git nntp://news.gmane.org/gmane.comp.version-control.git note: .onion URLs require Tor: https://www.torproject.org/ AGPL code for this site: git clone https://public-inbox.org/ public-inbox