[ruby-core:87154] [Ruby trunk Feature#14718] Use jemalloc by default?

[dennisb55@hotmail.com]

Issue #14718 has been updated by bluz71 (Dennis B).


@mperham, @mame, @normalperson,

glibc version 2.10 (2009) malloc was changed to favour scalibility in preference to memory size as noted [here](https://udrepper.livejournal.com/20948.html). Basically Red Hat and Ulrich Drepper did this to favour large clients (big customers, government agencies) where RAM is plentiful and thread-contention was a real problem.

Prior to 2.10 it seems that `MAX_ARENA_MAX=1`.
After 2.10 for 32-bit systems it changed to `2 x core-count` and for 64-bit systems it changed to `8 x core-count`.

This change has since caused a lot of grief for customers in regards to memory usage and fragmentation as noted here:

- https://sourceware.org/bugzilla/show_bug.cgi?id=11261

- https://github.com/prestodb/presto/issues/8993

This has lead to many recommendations:

- https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior

- https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en

- https://stackoverflow.com/questions/26041117/growing-resident-memory-usage-rss-of-java-process/35610063#35610063

Languages such as Golang and Rust ship their own memory allocators (tcmalloc and jemalloc based respectively). Browsers such as Chrome and Firefox also ship and use  their own allocators (tcmalloc/PartitionAlloc and jemalloc respectively). These technologies have their own reasons for doing this, not always related to memory fragmentation (though that is quite important).

Long-lived Ruby processes are a victim of this 2009 change.

----------------------------------------
Feature #14718: Use jemalloc by default?
https://bugs.ruby-lang.org/issues/14718#change-72144

* Author: mperham (Mike Perham)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
I know Sam opened #9113 4 years ago to suggest this but I'm revisiting the topic to see if there's any movement here for Ruby 2.6 or 2.7.  I supply a major piece of Ruby infrastructure (Sidekiq) and I keep hearing over and over how Ruby is terrible with memory, a huge memory hog with their Rails apps.  My users switch to jemalloc and a miracle occurs: their memory usage drops massively.  Some data points:

https://twitter.com/brandonhilkert/status/987400365627801601
https://twitter.com/d_jones/status/989866391787335680
https://github.com/mperham/sidekiq/issues/3824#issuecomment-383072469

Redis moved to jemalloc many years ago and it solved all of their memory issues too.  Their conclusion: the glibc allocator "sucks really really hard". http://oldblog.antirez.com/post/everything-about-redis-24.html

This is a real pain point for the entire Rails community and would improve Ruby's reputation immensely if we can solve this problem.



-- 
https://bugs.ruby-lang.org/