[ruby-core:93274] [Ruby trunk Bug#15841] SegFault in OpenSSL::PKey::RSA#private_encrypt

[thekuwayama@gmail.com]

Issue #15841 has been updated by thekuwayama (tomoya kuwayama).


I creaated a PR, https://github.com/ruby/openssl/pull/258 , about remaining issues.

> 2. if rsa.set_key had called without d argument, rsa.private? should return false.


----------------------------------------
Bug #15841: SegFault in OpenSSL::PKey::RSA#private_encrypt
https://bugs.ruby-lang.org/issues/15841#change-78733

* Author: thekuwayama (tomoya kuwayama)
* Status: Open
* Priority: Normal
* Assignee: rhenium (Kazuki Yamaguchi)
* Target version: 
* ruby -v: ruby-trunk
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Hi.

I am writing code that gets rsa private_key using `OpenSSL::PKey::RSA#set_key`.
As a test, I tried to run following code, and got a crash report.

```
require 'openssl'

MODULUS = OpenSSL::BN.new('126914039353434453831661971268647447269232081862082764501010934367441434199199964254884893447916776634375786528636229937728173623541291144426274921409848997181513107190580453415730826852070626720125773687471242\
611642649234390348699947633571205684722799950579951120477619298143923772148965919919195784168283711', 10)
PUBLIC_EXPONENT = OpenSSL::BN.new('65537', 10)
PRIVATE_EXPONENT = OpenSSL::BN.new('341964495821065129936072986248372022243660770187105326365541869938588248782459643985676392231199635777382326886137241414828657902188760530546426203854726301121665061632837569847323878241274517300277489\
6102686920500059152100016165854694372963975060765003171003826784408362498480661236694500218201182323054913', 10)
PRIME1 = OpenSSL::BN.new('11952373024606947105152469897150254148042322654516052874548960228374163164391052864033557517269946782417764389875359650595699633451962690417812447456789781', 10)
PRIME2 = OpenSSL::BN.new('10618313124276675806272072098863521356129998721878748974728637357066521302704987846522920724710466419737573058767973827707394086143442677100153976677110531', 10)

rsa = OpenSSL::PKey::RSA.new
rsa.set_key(MODULUS, PUBLIC_EXPONENT, nil)# PRIVATE_EXPONENT)
rsa.set_factors(PRIME1, PRIME2)
puts rsa.private_encrypt('plaintext') if rsa.private?

```

I expected that

1. `rsa.private_encrypt` should not crash, raise RSAError.
2. if `rsa.set_key` had called without `d` argument, `rsa.private?` should return false.

This is the execute environment.

```
$ ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

with 2.7.0-dev too

```
$ ruby --version
ruby 2.7.0dev (2019-05-09 trunk 025206d0dd) [x86_64-darwin18]
$ ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.1.1b  26 Feb 2019
$ gem list openssl

*** LOCAL GEMS ***

openssl (default: 2.1.2)
```

Thanks.

---Files--------------------------------
ruby_2019-05-09-191920_MacBookPro.crash (39.1 KB)
ruby_2019-05-09-192040_MacBookPro.crash (39.1 KB)


-- 
https://bugs.ruby-lang.org/