[ruby-core:60699] [ruby-trunk - Bug #9518] [Open] Objects in large arrays are leaked

[ruby-core:60699] [ruby-trunk - Bug #9518] [Open] Objects in large arrays are leaked

[charliesome]

Issue #9518 has been reported by Charlie Somerville.

----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518

* Author: Charlie Somerville
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
http://bugs.ruby-lang.org/

[ruby-core:60700] [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[charliesome]

Issue #9518 has been updated by Charlie Somerville.


Oops, the markdown rendering messed up the code I pasted above.

    a = [nil] * 131071
    loop { a << Object.new; a.pop }
    # process RSS stays stable

    a = [nil] * 131072
    loop { a << Object.new; a.pop }
    # process RSS grows quickly and never falls

----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518#change-45121

* Author: Charlie Somerville
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
http://bugs.ruby-lang.org/

[ruby-core:61065] [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[ko1]

Issue #9518 has been updated by Koichi Sasada.


Yes, you are right. WB (write barrier) strategy doesn't care such case.

How many such program?
If it is popular case, we can care about it.



----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518#change-45462

* Author: Charlie Somerville
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
http://bugs.ruby-lang.org/

[ruby-core:61068] Re: [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[Eric Wong]

The yahns HTTP server uses a long-lived fdmap array to map
Fixnum(fileno) -> IO connections.
This array exists prevent GC from sweeping IOs (because IOs are watched
by Linux epoll and not markable w/o an Array to store them).

64K+ connections (array size) is attainable with yahns.  The long-lived
fdmap array lifetime should not infect the short lived client
connections.

If I were to allow >=64K client connections on my public yahns instance,
clients will cause memory leaks/waste.

(yahns supports infinitely-lived connections, but in
 practice, clients connections tend to be short-lived).

[ruby-core:61069] [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[normalperson]

Issue #9518 has been updated by Eric Wong.


 The yahns HTTP server uses a long-lived fdmap array to map
 Fixnum(fileno) -> IO connections.
 This array exists prevent GC from sweeping IOs (because IOs are watched
 by Linux epoll and not markable w/o an Array to store them).
 
 64K+ connections (array size) is attainable with yahns.  The long-lived
 fdmap array lifetime should not infect the short lived client
 connections.
 
 If I were to allow >=64K client connections on my public yahns instance,
 clients will cause memory leaks/waste.
 
 (yahns supports infinitely-lived connections, but in
  practice, clients connections tend to be short-lived).

----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518#change-45465

* Author: Charlie Somerville
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
http://bugs.ruby-lang.org/

[ruby-core:61749] Re: [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[Eric Wong]

ko1: may I commit the removal of special case for large array/hash?

* gc.c (rb_gc_writebarrier): remove special case for large hash/array

http://bogomips.org/ruby.git/patch?id=40849ff1bc881
git://80x24.org/ruby.git gc-nomagic

I strongly believe there should no magic based on size.

[ruby-core:61750] [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[normalperson]

Issue #9518 has been updated by Eric Wong.


 ko1: may I commit the removal of special case for large array/hash?
 
 * gc.c (rb_gc_writebarrier): remove special case for large hash/array
 
 http://bogomips.org/ruby.git/patch?id=40849ff1bc881
 git://80x24.org/ruby.git gc-nomagic
 
 I strongly believe there should no magic based on size.

----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518#change-45989

* Author: Charlie Somerville
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
https://bugs.ruby-lang.org/

[ruby-core:61751] Re: [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[SASADA Koichi]

(2014/03/29 15:48), normalperson@yhbt.net wrote:
>  I strongly believe there should no magic based on size.

I need to ask tarui-san, who introduce this mechanism.

Tarui-san, could you give me a comment?

-- 
// SASADA Koichi at atdot dot net

[ruby-core:61752] [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[ko1]

Issue #9518 has been updated by Koichi Sasada.


 (2014/03/29 15:48), normalperson@yhbt.net wrote:
 >  I strongly believe there should no magic based on size.
 
 I need to ask tarui-san, who introduce this mechanism.
 
 Tarui-san, could you give me a comment?
 
 -- 
 // SASADA Koichi at atdot dot net

----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518#change-45990

* Author: Charlie Somerville
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
https://bugs.ruby-lang.org/

[ruby-core:62086] [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[tarui]

Issue #9518 has been updated by Masaya Tarui.


Sorry for such a late response.

I agree.

At the first time, I was introduled for reducing minor GC's long time
by rescan of huge size array.
But in real case, if there are huge size array, there may be other huge data too.
No matter how it do, it have to take time.

----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518#change-46250

* Author: Charlie Somerville
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
https://bugs.ruby-lang.org/

[ruby-core:62098] Re: [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[Eric Wong]

Thanks all! r45638

[ruby-core:62099] [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[normalperson]

Issue #9518 has been updated by Eric Wong.


 Thanks all! r45638

----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518#change-46259

* Author: Charlie Somerville
* Status: Closed
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
https://bugs.ruby-lang.org/

[ruby-core:62358] [ruby-trunk - Bug #9518] Objects in large arrays are leaked

[nagachika00]

Issue #9518 has been updated by Tomoyuki Chikanaga.

Backport changed from 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN to 1.9.3: DONTNEED, 2.0.0: DONTNEED, 2.1: DONE

r45638 was backported into ruby_2_1 at r45818.
according to #9796

----------------------------------------
Bug #9518: Objects in large arrays are leaked
https://bugs.ruby-lang.org/issues/9518#change-46517

* Author: Charlie Somerville
* Status: Closed
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]
* Backport: 1.9.3: DONTNEED, 2.0.0: DONTNEED, 2.1: DONE
----------------------------------------
a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766



-- 
https://bugs.ruby-lang.org/