[ruby-core:92614] [Ruby trunk Bug#15841] SegFault in OpenSSL::PKey::RSA#private_encrypt

[ruby-core:92614] [Ruby trunk Bug#15841] SegFault in OpenSSL::PKey::RSA#private_encrypt

[thekuwayama]

Issue #15841 has been reported by thekuwayama (tomoya kuwayama).

----------------------------------------
Bug #15841: SegFault in OpenSSL::PKey::RSA#private_encrypt
https://bugs.ruby-lang.org/issues/15841

* Author: thekuwayama (tomoya kuwayama)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby-trunk
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Hi.

I am writing code that gets rsa private_key using `OpenSSL::PKey::RSA#set_key`.
As a test, I tried to run following code, and got a crash report.

```
require 'openssl'

MODULUS = OpenSSL::BN.new('126914039353434453831661971268647447269232081862082764501010934367441434199199964254884893447916776634375786528636229937728173623541291144426274921409848997181513107190580453415730826852070626720125773687471242\
611642649234390348699947633571205684722799950579951120477619298143923772148965919919195784168283711', 10)
PUBLIC_EXPONENT = OpenSSL::BN.new('65537', 10)
PRIVATE_EXPONENT = OpenSSL::BN.new('341964495821065129936072986248372022243660770187105326365541869938588248782459643985676392231199635777382326886137241414828657902188760530546426203854726301121665061632837569847323878241274517300277489\
6102686920500059152100016165854694372963975060765003171003826784408362498480661236694500218201182323054913', 10)
PRIME1 = OpenSSL::BN.new('11952373024606947105152469897150254148042322654516052874548960228374163164391052864033557517269946782417764389875359650595699633451962690417812447456789781', 10)
PRIME2 = OpenSSL::BN.new('10618313124276675806272072098863521356129998721878748974728637357066521302704987846522920724710466419737573058767973827707394086143442677100153976677110531', 10)

rsa = OpenSSL::PKey::RSA.new
rsa.set_key(MODULUS, PUBLIC_EXPONENT, nil)# PRIVATE_EXPONENT)
rsa.set_factors(PRIME1, PRIME2)
puts rsa.private_encrypt('plaintext') if rsa.private?

```

I expected that

1. `rsa.private_encrypt` should not crash, raise RSAError.
2. if `rsa.set_key` had called without `d` argument, `rsa.private?` should return false.

This is the execute environment.

```
$ ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

with 2.7.0-dev too

```
$ ruby --version
ruby 2.7.0dev (2019-05-09 trunk 025206d0dd) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

Thanks.

---Files--------------------------------
ruby_2019-05-09-191920_MacBookPro.crash (39.1 KB)
ruby_2019-05-09-192040_MacBookPro.crash (39.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:92627] [Ruby trunk Bug#15841] SegFault in OpenSSL::PKey::RSA#private_encrypt

[Greg.mpls]

Issue #15841 has been updated by MSP-Greg (Greg L).


Interesting.  I've meant to create a repo that allowed one to write a test, and run it against MinGW, Linux, & OSX builds, using current Ruby builds (2.4 thru trunk).

All darwin18 builds passed, Linux Xenial builds failed, and MinGW builds passed.  Some use OpenSSL 1.1.1, others 1.0.2.

The test is here:
https://github.com/MSP-Greg/ruby-test/blob/15841/test/test_15841.rb

Travis:
https://travis-ci.org/MSP-Greg/ruby-test/builds/531454106

Appveyor:
https://ci.appveyor.com/project/MSP-Greg/ruby-test

Re this issue, when and where SEGV's are considered 'improper' is not something I'll weigh in on...

----------------------------------------
Bug #15841: SegFault in OpenSSL::PKey::RSA#private_encrypt
https://bugs.ruby-lang.org/issues/15841#change-77989

* Author: thekuwayama (tomoya kuwayama)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby-trunk
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Hi.

I am writing code that gets rsa private_key using `OpenSSL::PKey::RSA#set_key`.
As a test, I tried to run following code, and got a crash report.

```
require 'openssl'

MODULUS = OpenSSL::BN.new('126914039353434453831661971268647447269232081862082764501010934367441434199199964254884893447916776634375786528636229937728173623541291144426274921409848997181513107190580453415730826852070626720125773687471242\
611642649234390348699947633571205684722799950579951120477619298143923772148965919919195784168283711', 10)
PUBLIC_EXPONENT = OpenSSL::BN.new('65537', 10)
PRIVATE_EXPONENT = OpenSSL::BN.new('341964495821065129936072986248372022243660770187105326365541869938588248782459643985676392231199635777382326886137241414828657902188760530546426203854726301121665061632837569847323878241274517300277489\
6102686920500059152100016165854694372963975060765003171003826784408362498480661236694500218201182323054913', 10)
PRIME1 = OpenSSL::BN.new('11952373024606947105152469897150254148042322654516052874548960228374163164391052864033557517269946782417764389875359650595699633451962690417812447456789781', 10)
PRIME2 = OpenSSL::BN.new('10618313124276675806272072098863521356129998721878748974728637357066521302704987846522920724710466419737573058767973827707394086143442677100153976677110531', 10)

rsa = OpenSSL::PKey::RSA.new
rsa.set_key(MODULUS, PUBLIC_EXPONENT, nil)# PRIVATE_EXPONENT)
rsa.set_factors(PRIME1, PRIME2)
puts rsa.private_encrypt('plaintext') if rsa.private?

```

I expected that

1. `rsa.private_encrypt` should not crash, raise RSAError.
2. if `rsa.set_key` had called without `d` argument, `rsa.private?` should return false.

This is the execute environment.

```
$ ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

with 2.7.0-dev too

```
$ ruby --version
ruby 2.7.0dev (2019-05-09 trunk 025206d0dd) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

Thanks.

---Files--------------------------------
ruby_2019-05-09-191920_MacBookPro.crash (39.1 KB)
ruby_2019-05-09-192040_MacBookPro.crash (39.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:92946] [Ruby trunk Bug#15841] SegFault in OpenSSL::PKey::RSA#private_encrypt

[merch-redmine]

Issue #15841 has been updated by jeremyevans0 (Jeremy Evans).

Assignee set to rhenium (Kazuki Yamaguchi)

I can confirm that the bug also happens on OpenBSD 6.5 with LibreSSL 2.9.1.

I created a pull request with a fix: https://github.com/ruby/openssl/pull/255

----------------------------------------
Bug #15841: SegFault in OpenSSL::PKey::RSA#private_encrypt
https://bugs.ruby-lang.org/issues/15841#change-78322

* Author: thekuwayama (tomoya kuwayama)
* Status: Open
* Priority: Normal
* Assignee: rhenium (Kazuki Yamaguchi)
* Target version: 
* ruby -v: ruby-trunk
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Hi.

I am writing code that gets rsa private_key using `OpenSSL::PKey::RSA#set_key`.
As a test, I tried to run following code, and got a crash report.

```
require 'openssl'

MODULUS = OpenSSL::BN.new('126914039353434453831661971268647447269232081862082764501010934367441434199199964254884893447916776634375786528636229937728173623541291144426274921409848997181513107190580453415730826852070626720125773687471242\
611642649234390348699947633571205684722799950579951120477619298143923772148965919919195784168283711', 10)
PUBLIC_EXPONENT = OpenSSL::BN.new('65537', 10)
PRIVATE_EXPONENT = OpenSSL::BN.new('341964495821065129936072986248372022243660770187105326365541869938588248782459643985676392231199635777382326886137241414828657902188760530546426203854726301121665061632837569847323878241274517300277489\
6102686920500059152100016165854694372963975060765003171003826784408362498480661236694500218201182323054913', 10)
PRIME1 = OpenSSL::BN.new('11952373024606947105152469897150254148042322654516052874548960228374163164391052864033557517269946782417764389875359650595699633451962690417812447456789781', 10)
PRIME2 = OpenSSL::BN.new('10618313124276675806272072098863521356129998721878748974728637357066521302704987846522920724710466419737573058767973827707394086143442677100153976677110531', 10)

rsa = OpenSSL::PKey::RSA.new
rsa.set_key(MODULUS, PUBLIC_EXPONENT, nil)# PRIVATE_EXPONENT)
rsa.set_factors(PRIME1, PRIME2)
puts rsa.private_encrypt('plaintext') if rsa.private?

```

I expected that

1. `rsa.private_encrypt` should not crash, raise RSAError.
2. if `rsa.set_key` had called without `d` argument, `rsa.private?` should return false.

This is the execute environment.

```
$ ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

with 2.7.0-dev too

```
$ ruby --version
ruby 2.7.0dev (2019-05-09 trunk 025206d0dd) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

Thanks.

---Files--------------------------------
ruby_2019-05-09-191920_MacBookPro.crash (39.1 KB)
ruby_2019-05-09-192040_MacBookPro.crash (39.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:93274] [Ruby trunk Bug#15841] SegFault in OpenSSL::PKey::RSA#private_encrypt

[thekuwayama]

Issue #15841 has been updated by thekuwayama (tomoya kuwayama).


I creaated a PR, https://github.com/ruby/openssl/pull/258 , about remaining issues.

> 2. if rsa.set_key had called without d argument, rsa.private? should return false.


----------------------------------------
Bug #15841: SegFault in OpenSSL::PKey::RSA#private_encrypt
https://bugs.ruby-lang.org/issues/15841#change-78733

* Author: thekuwayama (tomoya kuwayama)
* Status: Open
* Priority: Normal
* Assignee: rhenium (Kazuki Yamaguchi)
* Target version: 
* ruby -v: ruby-trunk
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Hi.

I am writing code that gets rsa private_key using `OpenSSL::PKey::RSA#set_key`.
As a test, I tried to run following code, and got a crash report.

```
require 'openssl'

MODULUS = OpenSSL::BN.new('126914039353434453831661971268647447269232081862082764501010934367441434199199964254884893447916776634375786528636229937728173623541291144426274921409848997181513107190580453415730826852070626720125773687471242\
611642649234390348699947633571205684722799950579951120477619298143923772148965919919195784168283711', 10)
PUBLIC_EXPONENT = OpenSSL::BN.new('65537', 10)
PRIVATE_EXPONENT = OpenSSL::BN.new('341964495821065129936072986248372022243660770187105326365541869938588248782459643985676392231199635777382326886137241414828657902188760530546426203854726301121665061632837569847323878241274517300277489\
6102686920500059152100016165854694372963975060765003171003826784408362498480661236694500218201182323054913', 10)
PRIME1 = OpenSSL::BN.new('11952373024606947105152469897150254148042322654516052874548960228374163164391052864033557517269946782417764389875359650595699633451962690417812447456789781', 10)
PRIME2 = OpenSSL::BN.new('10618313124276675806272072098863521356129998721878748974728637357066521302704987846522920724710466419737573058767973827707394086143442677100153976677110531', 10)

rsa = OpenSSL::PKey::RSA.new
rsa.set_key(MODULUS, PUBLIC_EXPONENT, nil)# PRIVATE_EXPONENT)
rsa.set_factors(PRIME1, PRIME2)
puts rsa.private_encrypt('plaintext') if rsa.private?

```

I expected that

1. `rsa.private_encrypt` should not crash, raise RSAError.
2. if `rsa.set_key` had called without `d` argument, `rsa.private?` should return false.

This is the execute environment.

```
$ ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

with 2.7.0-dev too

```
$ ruby --version
ruby 2.7.0dev (2019-05-09 trunk 025206d0dd) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

Thanks.

---Files--------------------------------
ruby_2019-05-09-191920_MacBookPro.crash (39.1 KB)
ruby_2019-05-09-192040_MacBookPro.crash (39.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:93559] [Ruby master Bug#15841] SegFault in OpenSSL::PKey::RSA#private_encrypt

[thekuwayama]

Issue #15841 has been updated by thekuwayama (tomoya kuwayama).


Both of PR is merged and this issue is resolved, so this can be closed.

- https://github.com/ruby/openssl/pull/255
- https://github.com/ruby/openssl/pull/258

Thanks!

----------------------------------------
Bug #15841: SegFault in OpenSSL::PKey::RSA#private_encrypt
https://bugs.ruby-lang.org/issues/15841#change-79128

* Author: thekuwayama (tomoya kuwayama)
* Status: Open
* Priority: Normal
* Assignee: rhenium (Kazuki Yamaguchi)
* Target version: 
* ruby -v: ruby-trunk
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Hi.

I am writing code that gets rsa private_key using `OpenSSL::PKey::RSA#set_key`.
As a test, I tried to run following code, and got a crash report.

```
require 'openssl'

MODULUS = OpenSSL::BN.new('126914039353434453831661971268647447269232081862082764501010934367441434199199964254884893447916776634375786528636229937728173623541291144426274921409848997181513107190580453415730826852070626720125773687471242\
611642649234390348699947633571205684722799950579951120477619298143923772148965919919195784168283711', 10)
PUBLIC_EXPONENT = OpenSSL::BN.new('65537', 10)
PRIVATE_EXPONENT = OpenSSL::BN.new('341964495821065129936072986248372022243660770187105326365541869938588248782459643985676392231199635777382326886137241414828657902188760530546426203854726301121665061632837569847323878241274517300277489\
6102686920500059152100016165854694372963975060765003171003826784408362498480661236694500218201182323054913', 10)
PRIME1 = OpenSSL::BN.new('11952373024606947105152469897150254148042322654516052874548960228374163164391052864033557517269946782417764389875359650595699633451962690417812447456789781', 10)
PRIME2 = OpenSSL::BN.new('10618313124276675806272072098863521356129998721878748974728637357066521302704987846522920724710466419737573058767973827707394086143442677100153976677110531', 10)

rsa = OpenSSL::PKey::RSA.new
rsa.set_key(MODULUS, PUBLIC_EXPONENT, nil)# PRIVATE_EXPONENT)
rsa.set_factors(PRIME1, PRIME2)
puts rsa.private_encrypt('plaintext') if rsa.private?

```

I expected that

1. `rsa.private_encrypt` should not crash, raise RSAError.
2. if `rsa.set_key` had called without `d` argument, `rsa.private?` should return false.

This is the execute environment.

```
$ ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

with 2.7.0-dev too

```
$ ruby --version
ruby 2.7.0dev (2019-05-09 trunk 025206d0dd) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

Thanks.

---Files--------------------------------
ruby_2019-05-09-191920_MacBookPro.crash (39.1 KB)
ruby_2019-05-09-192040_MacBookPro.crash (39.1 KB)


-- 
https://bugs.ruby-lang.org/