[ruby-core:70465] [Ruby trunk - Bug #11466] [Open] Memory leak in win32 fill_random_bytes_syscall

[ruby-core:70465] [Ruby trunk - Bug #11466] [Open] Memory leak in win32 fill_random_bytes_syscall

[abudovski]

Issue #11466 has been reported by Alex Budovski.

----------------------------------------
Bug #11466: Memory leak in win32 fill_random_bytes_syscall
https://bugs.ruby-lang.org/issues/11466

* Author: Alex Budovski
* Status: Open
* Priority: Normal
* Assignee: 
* ruby -v: master
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN
----------------------------------------
Leak of 608 bytes in this stack:

~~~
+     608 (    608 -      0)      1 allocs	BackTrace9BD0704B
+       1 (      1 -      0)	BackTrace9BD0704B	allocations

	ntdll!RtlpCallInterceptRoutine+40 (d:\blue\minkernel\ntos\rtl\heappriv.h, 3625)
	ntdll!RtlAllocateHeap+79836 (d:\blue\minkernel\ntos\rtl\heap.c, 1892)
	rsaenh!InitUser+20 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 783)
	rsaenh!NTagLogonUser+167 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 950)
	rsaenh!CPAcquireContext+88 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 1230)
	cryptsp!CryptAcquireContextA+4A3 (d:\9147\ds\win32\ntcrypto\cryptsp\cryptapi.c, 884)
	rubyprov!fill_random_bytes_syscall+44 (e:\dev\ruby\random.c, 498)
	rubyprov!fill_random_bytes+1D (e:\dev\ruby\random.c, 542)
	rubyprov!fill_random_seed+2A (e:\dev\ruby\random.c, 557)
	rubyprov!init_randomseed+18 (e:\dev\ruby\random.c, 1446)
	rubyprov!Init_RandomSeed+3C (e:\dev\ruby\random.c, 1458)
	rubyprov!rb_call_inits+E (e:\dev\ruby\inits.c, 21)
	rubyprov!ruby_setup+C6 (e:\dev\ruby\eval.c, 63)
~~~

It looks like the fill_random_bytes_syscall isn't releasing the context on success?

~~~
static int
fill_random_bytes_syscall(void *seed, size_t size)
{
    static HCRYPTPROV perm_prov;
    HCRYPTPROV prov = perm_prov, old_prov;
    if (!prov) {
	if (!CryptAcquireContext(&prov, NULL, NULL, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
	    prov = (HCRYPTPROV)INVALID_HANDLE_VALUE;
	}
	old_prov = (HCRYPTPROV)ATOMIC_PTR_CAS(perm_prov, 0, prov);
	if (LIKELY(!old_prov)) { /* no other threads acquried */
	    if (prov != (HCRYPTPROV)INVALID_HANDLE_VALUE) {
		rb_gc_register_mark_object(Data_Wrap_Struct(0, 0, release_crypt, &perm_prov));
	    }
	}
	else {			/* another thread acquried */
	    if (prov != (HCRYPTPROV)INVALID_HANDLE_VALUE) {
		CryptReleaseContext(prov, 0);
	    }
	    prov = old_prov;
	}
    }
    if (prov == (HCRYPTPROV)INVALID_HANDLE_VALUE) return -1;
    CryptGenRandom(prov, size, seed);
    return 0;
}
~~~



-- 
https://bugs.ruby-lang.org/

[ruby-core:94491] [Ruby master Bug#11466] Memory leak in win32 fill_random_bytes_syscall

[merch-redmine]

Issue #11466 has been updated by jeremyevans0 (Jeremy Evans).

Status changed from Open to Closed

If `fill_random_bytes_syscall` were to leak memory, then this Ruby code should show the leak:

```ruby
i = 0; while true; puts i if (i+=1) % 10000000; SecureRandom.bytes(1) end
```

As this shows no leak even with with 600000000 iterations, I am going to guess the leak checker used is flawed, or maybe is a one-time leak of 608 bytes when the function is first called, not a leak per call, in which case it can be safely ignored.

----------------------------------------
Bug #11466: Memory leak in win32 fill_random_bytes_syscall
https://bugs.ruby-lang.org/issues/11466#change-80926

* Author: scorpion007 (Alex Budovski)
* Status: Closed
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: master
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN
----------------------------------------
Leak of 608 bytes in this stack:

~~~
+     608 (    608 -      0)      1 allocs	BackTrace9BD0704B
+       1 (      1 -      0)	BackTrace9BD0704B	allocations

	ntdll!RtlpCallInterceptRoutine+40 (d:\blue\minkernel\ntos\rtl\heappriv.h, 3625)
	ntdll!RtlAllocateHeap+79836 (d:\blue\minkernel\ntos\rtl\heap.c, 1892)
	rsaenh!InitUser+20 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 783)
	rsaenh!NTagLogonUser+167 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 950)
	rsaenh!CPAcquireContext+88 (d:\9147\ds\win32\ntcrypto\scp\ntagum.c, 1230)
	cryptsp!CryptAcquireContextA+4A3 (d:\9147\ds\win32\ntcrypto\cryptsp\cryptapi.c, 884)
	rubyprov!fill_random_bytes_syscall+44 (e:\dev\ruby\random.c, 498)
	rubyprov!fill_random_bytes+1D (e:\dev\ruby\random.c, 542)
	rubyprov!fill_random_seed+2A (e:\dev\ruby\random.c, 557)
	rubyprov!init_randomseed+18 (e:\dev\ruby\random.c, 1446)
	rubyprov!Init_RandomSeed+3C (e:\dev\ruby\random.c, 1458)
	rubyprov!rb_call_inits+E (e:\dev\ruby\inits.c, 21)
	rubyprov!ruby_setup+C6 (e:\dev\ruby\eval.c, 63)
~~~

It looks like the fill_random_bytes_syscall isn't releasing the context on success?

~~~
static int
fill_random_bytes_syscall(void *seed, size_t size)
{
    static HCRYPTPROV perm_prov;
    HCRYPTPROV prov = perm_prov, old_prov;
    if (!prov) {
	if (!CryptAcquireContext(&prov, NULL, NULL, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
	    prov = (HCRYPTPROV)INVALID_HANDLE_VALUE;
	}
	old_prov = (HCRYPTPROV)ATOMIC_PTR_CAS(perm_prov, 0, prov);
	if (LIKELY(!old_prov)) { /* no other threads acquried */
	    if (prov != (HCRYPTPROV)INVALID_HANDLE_VALUE) {
		rb_gc_register_mark_object(Data_Wrap_Struct(0, 0, release_crypt, &perm_prov));
	    }
	}
	else {			/* another thread acquried */
	    if (prov != (HCRYPTPROV)INVALID_HANDLE_VALUE) {
		CryptReleaseContext(prov, 0);
	    }
	    prov = old_prov;
	}
    }
    if (prov == (HCRYPTPROV)INVALID_HANDLE_VALUE) return -1;
    CryptGenRandom(prov, size, seed);
    return 0;
}
~~~



-- 
https://bugs.ruby-lang.org/