[ruby-core:91837] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[ruby-core:91837] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[sam.saffron]

Issue #15667 has been reported by sam.saffron (Sam Saffron).

----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.





-- 
https://bugs.ruby-lang.org/

[ruby-core:91858] Re: [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[Eric Wong]

sam.saffron@gmail.com wrote:
> Feature #15667: Introduce malloc_trim(0) in full gc cycles
> https://bugs.ruby-lang.org/issues/15667

The patch looks OK with the current state of glibc malloc for
multithreaded use cases in Ruby.  However, this seems like it is
papering over an inefficiency in glibc malloc which needs to be
fixed.

Background: glibc uses sbrk for the main-thread arena.
  Sub-thread arenas use mmap by default; but limiting
  MALLOC_ARENA_MAX can case sub-threads to use the main(sbrk)
  arena.

Some notes:

1.  We need to know how performance is in single-threaded apps
    compared multi-threaded apps(*); since I'm not sure if
    single-threaded use was benchmarked.

    The part which concerns me is the systrim() call only affects the
    main (sbrk) arena...

    https://public-inbox.org/libc-alpha/801ba1f499/s/?b=malloc/malloc.c#n4817

    ...which is likely to cause a performance hit if the main thread
    does more allocations+frees down the line.

    Multi-thread servers like puma do not allocate much in the main thread
    once started, but `require' eats a lot of memory at startup[3].
    So the benefit of reduced memory use noted by Hongli could be coming
    from this.

    The MADV_DONTNEED loop above systrim releases memory back to
    the OS for non-main-thread arenas, too; but should not
    noticeably affect performance.  Maybe MADV_DONTNEED (or
    MADV_FREE) alone is enough for RSS savings?


2.  As for improving glibc...
    Maybe glibc is missing malloc_consolidate calls in some
    places which need them.  Or (more likely) there's places where
    it would be better to call mtrim(av, mp_.top_pad) instead of
    malloc_consolidate(av) to have glibc do trimming.


3.  Once a multi-threaded application is fully-loaded; perhaps a
    single call to "malloc_trim(0)" (via fiddle) is all that's
    necessary?  That would free the memory eaten by `require'
    in the main thread; and further GCs won't be affected.

    If this is successful in reducing memory use, that would
    mean the systrim() call I was concerned about in [1] was
    responsible for the savings.  And perhaps a single call
    is all that is necessary.


I no longer have mosh/ssh access to anything which can build or
test large codebases at a decent rate.  So it's not feasable for
me to work on glibc, ruby or any large codebase anymore.

[ruby-core:91841] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[sam.saffron]

Issue #15667 has been updated by sam.saffron (Sam Saffron).


Relevant code in glibc

https://github.com/bminor/glibc/blob/c2d8f0b704c2b828bcd8d517a2376c0240c73c09/malloc/malloc.c#L4448-L4539

https://github.com/bminor/glibc/blob/c2d8f0b704c2b828bcd8d517a2376c0240c73c09/malloc/malloc.c#L4805



----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667#change-77110

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.





-- 
https://bugs.ruby-lang.org/

[ruby-core:91889] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[mame]

Issue #15667 has been updated by mame (Yusuke Endoh).

File ruby_gc_malloc_trim.patch added

I created a patch.

> I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify.

Could you test the patch attached?

----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667#change-77212

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.



---Files--------------------------------
ruby_gc_malloc_trim.patch (1011 Bytes)


-- 
https://bugs.ruby-lang.org/

[ruby-core:91968] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[sam.saffron]

Issue #15667 has been updated by sam.saffron (Sam Saffron).


@mame / @carlos, absolutely, I just need a few more days here, mounting this kind of test is not trivial even with docker containers. 

----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667#change-77300

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.



---Files--------------------------------
ruby_gc_malloc_trim.patch (1011 Bytes)


-- 
https://bugs.ruby-lang.org/

[ruby-core:92027] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[sam.saffron]

Issue #15667 has been updated by sam.saffron (Sam Saffron).

File Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png added

@mame / @carlos attached is a screenshot of side by side testing on live traffic patterns

containers run multiple ruby processes (a few unicorn workers and a sidekiq worker)

standard14 = Ruby 2.6.2 + jemalloc
standard14_a = Ruby 2.6.2 + glibc malloc
standard14_b = Ruby 2.6.2 + glibc malloc + patch

My conclusions from these graphs:

- Memory is clearly down with the patch
- 99th percentile performance is slightly impacted
- cpu is very slightly higher
- jemalloc still fairs better than glibc even after the patch


I think I would support a slightly amended patch that only does the trim once every say 10 minutes (maybe even in a background thread), happy to test that out as well. 

That said... selfishly for Discourse this does not matter that much we will still stick with jemalloc cause memory is better and performance is better under jemalloc. 

For the wider ruby community though a safer default is very appealing. 

----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667#change-77352

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.



---Files--------------------------------
ruby_gc_malloc_trim.patch (1011 Bytes)
Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:92046] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[dennisb55]

Issue #15667 has been updated by bluz71 (Dennis B).


Thanks Sam, a very nice set of results.

Notice that 99th percentile Topic list was faster with the patch, whilst slower with Topic view. So I'm not sure we can say that the patch will always be slower on the worst runs.

Query, what is the version of jemalloc that you are using? One of the interesting observations in [#14759](https://bugs.ruby-lang.org/issues/14759#note-6) is the variance between jemalloc versions (say 3.6.0 vs 5.1.0).

----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667#change-77367

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.



---Files--------------------------------
ruby_gc_malloc_trim.patch (1011 Bytes)
Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:92060] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[philipp]

Issue #15667 has been updated by tessi (Philipp Tessenow).


FYI: To easy testing this change, I just pushed a small gem to rubygems `malloc_trim` (https://github.com/tessi/malloc_trim). This gives access to `malloc_trim` to ruby land to let us play with it without the need to re-compile ruby and/or deploying a custom patches ruby.

With `MallocTrim.enable_trimming`, there is also a built in way to run `malloc_trim(0)` after every GC MARK (the most relevant internal event I found to hook into). It probably calls malloc_trim somewhat too often -- I'd be happy for suggestions to find a better hook.

In any way, doing a manual `MallocTrim.trim` after a `GC.start` (e.g. between two rails requests) is still possible with this gem.

----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667#change-77383

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.



---Files--------------------------------
ruby_gc_malloc_trim.patch (1011 Bytes)
Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:92070] [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[sam.saffron]

Issue #15667 has been updated by sam.saffron (Sam Saffron).

File crash.png added

@tessi 

My tests were with 3.6.0, I will do a side by side now that I have all the infrastructure of 5 vs 3.6 and even tcmalloc. 

Nice to see that gem! Looking at my graphs I think best bang would just be to spin a thread that does trimming every 10-30 minutes or something, especially if you can release the GVL prior to calling it (provided this thing is thread safe, which @carlos should know)
 

----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667#change-77394

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.



---Files--------------------------------
ruby_gc_malloc_trim.patch (1011 Bytes)
Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB)
crash.png (85.9 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:92087] Re: [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[Waheed Al-Barghouthi]

[-- Attachment #1.1: Type: text/plain, Size: 2660 bytes --]

There is some decent amount of information in that article, it's a great
food for thought, for me I actually decided to use jmalloc for a production
service and the problem were specifically related to sidekiq workers not
releasing memory effectively when jobs are consumed.

Anyone have an idea if setting M_TRIM_THRESHOLD for a value like 40 or 80
bytes would actually help in this case, as I believe it should
automatically return memory to the system. I will try using its environment
variable and test if there are any improvement by just using that variable.

All in all, great job everyone.

On Mon, Apr 1, 2019 at 6:05 AM <sam.saffron@gmail.com> wrote:

> Issue #15667 has been updated by sam.saffron (Sam Saffron).
>
> File crash.png added
>
> @tessi
>
> My tests were with 3.6.0, I will do a side by side now that I have all the
> infrastructure of 5 vs 3.6 and even tcmalloc.
>
> Nice to see that gem! Looking at my graphs I think best bang would just be
> to spin a thread that does trimming every 10-30 minutes or something,
> especially if you can release the GVL prior to calling it (provided this
> thing is thread safe, which @carlos should know)
>
>
> ----------------------------------------
> Feature #15667: Introduce malloc_trim(0) in full gc cycles
> https://bugs.ruby-lang.org/issues/15667#change-77394
>
> * Author: sam.saffron (Sam Saffron)
> * Status: Open
> * Priority: Normal
> * Assignee:
> * Target version:
> ----------------------------------------
> Per Hongli's excellent article it looks like malloc_trim can help
> tremendously with memory bloat issues.
>
>
>
> https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming
>
>
> I would like to get this patch tested side-by-side at Discourse, GitHub
> and Shopify. If it looks good I think this is both a great candidate for
> 2.7 and and 2.4,2.5,2.6 backports.
>
>
> Will coordinate with Shopify and GitHub to see if we can get numbers
> posted here, I will run tests on a live Discourse instance over the next
> week and report numbers here.
>
> Koichi, what are your thoughts, to me this looks like an incredibly safe
> patch, the amount of work added to major GCs is tiny compared to the
> potential benefit, walking all pages is a very cheap operation.
>
>
>
> ---Files--------------------------------
> ruby_gc_malloc_trim.patch (1011 Bytes)
> Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB)
> crash.png (85.9 KB)
>
>
> --
> https://bugs.ruby-lang.org/
>
> Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
> <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>
>

[-- Attachment #1.2: Type: text/html, Size: 3740 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

[ruby-core:92088] Re: [Ruby trunk Feature#15667] Introduce malloc_trim(0) in full gc cycles

[Martin J. Dürst]

On 2019/04/01 18:28, Waheed Al-Barghouthi wrote:

> Anyone have an idea if setting M_TRIM_THRESHOLD for a value like 40 or 80
> bytes would actually help in this case, as I believe it should
> automatically return memory to the system. I will try using its environment
> variable and test if there are any improvement by just using that variable.

I'm not at all an expert in this area, but a value like 40 or 80 
(bytes!) sounds way, way too low. The default is something like 128*1024 
bytes, and that may be a bit high, but nobody would want a system call 
just to save 80 bytes.

Regards,   Martin.

> All in all, great job everyone.
> 
> On Mon, Apr 1, 2019 at 6:05 AM <sam.saffron@gmail.com> wrote:
> 
>> Issue #15667 has been updated by sam.saffron (Sam Saffron).
>>
>> File crash.png added
>>
>> @tessi
>>
>> My tests were with 3.6.0, I will do a side by side now that I have all the
>> infrastructure of 5 vs 3.6 and even tcmalloc.
>>
>> Nice to see that gem! Looking at my graphs I think best bang would just be
>> to spin a thread that does trimming every 10-30 minutes or something,
>> especially if you can release the GVL prior to calling it (provided this
>> thing is thread safe, which @carlos should know)
>>
>>
>> ----------------------------------------
>> Feature #15667: Introduce malloc_trim(0) in full gc cycles
>> https://bugs.ruby-lang.org/issues/15667#change-77394
>>
>> * Author: sam.saffron (Sam Saffron)
>> * Status: Open
>> * Priority: Normal
>> * Assignee:
>> * Target version:
>> ----------------------------------------
>> Per Hongli's excellent article it looks like malloc_trim can help
>> tremendously with memory bloat issues.

[ruby-core:94981] [Ruby master Feature#15667] Introduce malloc_trim(0) in full gc cycles

[dosadnizub]

Issue #15667 has been updated by dosadnizub (Borna Novak).


I'd just like to point out that calling malloc_trim(0) according to article explaining the feature - https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html - will not fix memory fragmentation - it'll just make the "used memory" number go down - the graph on https://www.joyfulbikeshedding.com/images/2019/os_heap_visualization_after_trim-b3e36496.png still means that an attempt to grab 15 pages of memory (~16kb) will fail even though there's 8MB of memory "free".

----------------------------------------
Feature #15667: Introduce malloc_trim(0) in full gc cycles
https://bugs.ruby-lang.org/issues/15667#change-81600

* Author: sam.saffron (Sam Saffron)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. 


https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming


I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. 


Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. 

Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation.



---Files--------------------------------
ruby_gc_malloc_trim.patch (1011 Bytes)
Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB)
crash.png (85.9 KB)


-- 
https://bugs.ruby-lang.org/