[ruby-core:61614] [ruby-trunk - Bug #9659] [Open] crash in FIPS mode after unchecked algo->init_func failure

[ruby-core:61614] [ruby-trunk - Bug #9659] [Open] crash in FIPS mode after unchecked algo->init_func failure

[jared.jennings.ctr]

Issue #9659 has been reported by Jared Jennings.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659

* Author: Jared Jennings
* Status: Open
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627



-- 
https://bugs.ruby-lang.org/

[ruby-core:61615] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[jared.jennings.ctr]

Issue #9659 has been updated by Jared Jennings.


Now I see that `rb_digest_hash_init_func_t` (source:ext/digest/digest.h@43668#L20) is a typedef for a pointer to a function returning void. This complicates the patch: the typedef must be changed so init functions return an int, and the init functions in each digest algorithm implementation included in the digest extension must be changed slightly, to return a 1 for success or a 0 for failure, as the OpenSSL implementations they imitate claim to do.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-45885

* Author: Jared Jennings
* Status: Open
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627



-- 
https://bugs.ruby-lang.org/

[ruby-core:61657] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[jared.jennings.ctr]

Issue #9659 has been updated by Jared Jennings.


I changed the `rb_digest_hash_init_func` typedef from a return type of `void` to `int`, so that the return value of `MD5_Init` could be checked. I changed digest.c to check the return value of `algo->init_func`, which at the time of the crash seems to point at `MD5_Init`, and raise an exception if the function returns 0.

The interpreter still crashes. Running with `gdb` reveals that in my version of OpenSSL the `MD5_Init` function goes sort of like, `{ if (FIPS_mode() ...) { OpenSSLDie(..., "Digest MD5 forbidden in FIPS mode!"); } return private_MD5_Init(...); }`. `OpenSSLDie` goes on to call `abort`. There's no returning 0 for failure in this case.

On a further look at `md5(3)`, I see: "Applications should use the higher level functions `EVP_DigestInit(3)` etc. instead of calling the hash functions directly." Those functions should return a value to indicate failure rather than raising a signal: the `openssl` module was successfully modified to check their return value in #4944, to good effect.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-45916

* Author: Jared Jennings
* Status: Open
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627



-- 
https://bugs.ruby-lang.org/

[ruby-core:61735] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[jared.jennings.ctr]

Issue #9659 has been updated by Jared Jennings.

File 001-detect-digest-failure.patch added
File 002-builtin-indicate-digest-failure.patch added
File 003-digest-openssl-md5-use-evp-api.patch added

Attached are three cumulative patches against source:/trunk@45452.

The first, 001-detect-digest-failure, changes the prototypes of digest initialization and finalization functions in the digest extension to return int instead of void; changes digest.c to check the return value of the initialization function and raise an exception in case of failure; and bumps the digest API version from 2 to 3.

The second, 002-builtin-indicate-digest-failure, changes the built-in digest implementations so that their initialization and finalization functions return an int, 1 for success or 0 for failure, as the OpenSSL functions return.

The third, 003-digest-openssl-md5-use-evp-api, changes the OpenSSL implementation of the md5 algorithm to use functions from `openssl/evp.h` rather than `openssl/md5.h`. The old, deprecated `MD5_Init` function calls `abort(3)` if used in FIPS-compliant mode, killing the interpreter; the `EVP_DigestInit_ex` function returns 0 to indicate initialization failure instead.

With these patches:

~~~
[vagrant@localhost ruby]$ OPENSSL_FORCE_FIPS_MODE= ruby -v -rdigest -e "puts Digest::MD5.hexdigest('hi')" 
ruby 2.2.0dev (2014-03-27) [x86_64-linux]
-e:1:in `digest': Digest initialization failed. (RuntimeError)
	from -e:1:in `hexdigest'
	from -e:1:in `<main>'
~~~

I think further improvement is possible. Generally, it appears that the functions and types used in the builtin digest algorithm implementations are made to mirror the `MD5_*`, `RIPEMD160_*`, etc APIs from OpenSSL. Since I'm moving the `ossl` implementations to use the `EVP_*` API instead, I think the Right Thing to do here would be to change the builtins to mirror that newer API. If someone else agrees, I can produce the patches; until then, I have tried to make the smallest patches possible.

About 001, I don't know the consequences of bumping the digest API version, and I didn't provide any migration code that will make code written against the version-2 API work with the version-3 API. Also I don't know if the exception raised in the case of digest failure is the right class of exception.

003 only changes the ossl implementation of MD5, not any of the other algorithms. To keep the patch size down, I hardcoded the digest and block size constants. This isn't very DRY. The larger changes I alluded to above could fix it.

I don't know if tests need to be added for this code, but there are none in the patches.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-45975

* Author: Jared Jennings
* Status: Open
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:61736] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[jared.jennings.ctr]

Issue #9659 has been updated by Jared Jennings.


If any credit is given for finding this problem, it belongs to Joseph Yaworski; see https://tickets.puppetlabs.com/browse/PUP-1840.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-45976

* Author: Jared Jennings
* Status: Open
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:61737] [ruby-trunk - Bug #9659] [Feedback] crash in FIPS mode after unchecked algo->init_func failure

[nobu]

Issue #9659 has been updated by Nobuyoshi Nakada.

Status changed from Open to Feedback

I can't reproduce that assertion failure, with openssl 0.9.8y and 1.0.1f.
`OPENSSL_FIPS` needs to be defined, perhaps?

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-45977

* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:61742] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[jared.jennings.ctr]

Issue #9659 has been updated by Jared Jennings.


I've just compared the Debian and CentOS OpenSSL sources, and it looks like large parts of the FIPS functionality in OpenSSL that I've taken for granted are provided in CentOS/RHEL-specific patches. So you may not be able to duplicate the failure with stock OpenSSL, or on Debian or Ubuntu machines.

On my RHEL 6 machine, I needed the `dracut-fips` package installed, which contains the FIPS crypto module (sometimes it's called a "canister"); see https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Security_Guide/sect-Security_Guide-Federal_Standards_And_Regulations-Federal_Information_Processing_Standard.html. This was because the OpenSSL init function checked whether the FIPS module was installed, and it's distributed in this package. But the code to check this was part of the CentOS/RHEL patches.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-45982

* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:63730] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[v.ondruch]

Issue #9659 has been updated by Vit Ondruch.


Hi, can we please push this forward? Since the fixes proposed so far seems to break API/ABI, it would be nice to have fixes in upstream Ruby sooner than later. This would help incorporate this patch into future versions of RHEL/CentOS/Fedora or any other FIPS compliant system.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-47774

* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:63735] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[nobu]

Issue #9659 has been updated by Nobuyoshi Nakada.

Backport changed from 2.0.0: UNKNOWN, 2.1: UNKNOWN to 2.0.0: REQUIRED, 2.1: REQUIRED

https://github.com/nobu/ruby/compare/Bug%239659-digest-failure

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-47779

* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: REQUIRED, 2.1: REQUIRED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:63736] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[knu]

Issue #9659 has been updated by Akinori MUSHA.


The above set of patches looks good to me.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-47780

* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: REQUIRED, 2.1: REQUIRED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:63739] [ruby-trunk - Bug #9659] [Closed] crash in FIPS mode after unchecked algo->init_func failure

[nobu]

Issue #9659 has been updated by Nobuyoshi Nakada.

Status changed from Feedback to Closed
% Done changed from 0 to 100

Applied in changeset r46826.

----------
digest.c: raise exception on init failure

* ext/digest/digest.c: expect digest init and finish functions to
  indicate success or failure; raise exception on failure.
  [ruby-core:61614] [Bug #9659]

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-47783

* Author: Jared Jennings
* Status: Closed
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: REQUIRED, 2.1: REQUIRED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:63750] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[v.ondruch]

Issue #9659 has been updated by Vit Ondruch.


Thanks Nobu. Nonetheless, I don't think it is backportable (which was not necessarily the point :).

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-47791

* Author: Jared Jennings
* Status: Closed
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: REQUIRED, 2.1: REQUIRED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:63752] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[nobu]

Issue #9659 has been updated by Nobuyoshi Nakada.

Backport changed from 2.0.0: REQUIRED, 2.1: REQUIRED to 2.0.0: DONTNEED, 2.1: DONTNEED

Is EVP API is necessary?
I've reverted it because of segfaults on many platforms.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-47792

* Author: Jared Jennings
* Status: Closed
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: DONTNEED, 2.1: DONTNEED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:63757] [ruby-trunk - Bug #9659] [Feedback] crash in FIPS mode after unchecked algo->init_func failure

[usa]

Issue #9659 has been updated by Usaku NAKAMURA.

Status changed from Closed to Feedback

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-47794

* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: DONTNEED, 2.1: DONTNEED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:65833] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

[jared.jennings.ctr]

Issue #9659 has been updated by Jared Jennings.


Nobuyoshi Nakada wrote:
> Is EVP API necessary?

The EVP API has been recommended over the old digest-specific API for [almost fifteen years](http://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=4facdbb5fa9d791fc72dc78b9c3512ea1384df33#patch3). It seems that EVP might [automatically use hardware acceleration](http://stackoverflow.com/a/25978145) where possible. And if EVP is not used, Ruby crashes on the secure systems used by banks and governments, with no indication of which Ruby code caused the problem. 

Nobuyoshi Nakada wrote:
> I've reverted it because of segfaults on many platforms.

Since EVP is so old already, any problem is likely due somehow to my code. I'd like to fix this. Can you share any further details?

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-49573

* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: DONTNEED, 2.1: DONTNEED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/