From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-3.4 required=3.0 tests=AWL,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 6C1EA1F597 for ; Wed, 1 Aug 2018 00:14:38 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 74773120B25; Wed, 1 Aug 2018 09:14:35 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id C72D11209D4 for ; Wed, 1 Aug 2018 09:14:33 +0900 (JST) Received: by filter0033p3las1.sendgrid.net with SMTP id filter0033p3las1-27055-5B60FB66-6 2018-08-01 00:14:30.384858896 +0000 UTC m=+15204.357763537 Received: from herokuapp.com (ec2-54-197-75-100.compute-1.amazonaws.com [54.197.75.100]) by ismtpd0023p1mdw1.sendgrid.net (SG) with ESMTP id NKuTAEZJSiWl8eNKlFbK_g Wed, 01 Aug 2018 00:14:30.241 +0000 (UTC) Date: Wed, 01 Aug 2018 00:14:30 +0000 (UTC) From: sam.saffron@gmail.com To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 63614 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 14718 X-Redmine-Issue-Author: mperham X-Redmine-Sender: sam.saffron X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS4deGh5As0bHjrkLnUL5dt60V25p+N9sV/Ay7 aig9xjmshrKu8JG9bess5uAJ79qgRMQMZjAz2SeteA6wI8ZgaSYXqsc7AtrufctlAoNNZmad6G0Crt hD8qA4U+tDOuIksvDs72hFGKJsZT4j/j5qUJF4JyI7u3X+RfOEhkGPz+9w== X-ML-Name: ruby-core X-Mail-Count: 88238 Subject: [ruby-core:88238] [Ruby trunk Feature#14718] Use jemalloc by default? X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #14718 has been updated by sam.saffron (Sam Saffron). After spending a bit too much time thinking about this, I would like to recommend **against** and jemalloc related changes and instead to double down on Eric's https://bugs.ruby-lang.org/issues/14759 for the next release of Ruby (which I think should be backported to 2.5/2.4) As much as we like to think of jemalloc as a silver bullet of sorts... there are problems... In particular if you have THP enabled which most people do out of the box its behavior is not ideal despite all the fixes it has gotten over the years. My observation is that both MALLOC_ARENA_MAX=2 and tcmalloc can perform better if THP is on, both in 5.1 and 3.6.0. Getting the world to turn off THP is a hard job. Further to this jemalloc probably has too many arenas out of the box and should only need 2 or so for optimal perf. Since it is so hairy and so application specific my vote here is merge and backport https://bugs.ruby-lang.org/issues/14759 ASAP. We can teach people about jemalloc quirks elsewhere but Ruby does not need to go down this path. Better just work with glibc here. ![THP on jemalloc](https://discourse-cdn-sjc1.com/dev/uploads/default/original/2X/7/7409f18667a24ed6643457377e475a738164e952.png) ---------------------------------------- Feature #14718: Use jemalloc by default? https://bugs.ruby-lang.org/issues/14718#change-73261 * Author: mperham (Mike Perham) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- I know Sam opened #9113 4 years ago to suggest this but I'm revisiting the topic to see if there's any movement here for Ruby 2.6 or 2.7. I supply a major piece of Ruby infrastructure (Sidekiq) and I keep hearing over and over how Ruby is terrible with memory, a huge memory hog with their Rails apps. My users switch to jemalloc and a miracle occurs: their memory usage drops massively. Some data points: https://twitter.com/brandonhilkert/status/987400365627801601 https://twitter.com/d_jones/status/989866391787335680 https://github.com/mperham/sidekiq/issues/3824#issuecomment-383072469 Redis moved to jemalloc many years ago and it solved all of their memory issues too. Their conclusion: the glibc allocator "sucks really really hard". http://oldblog.antirez.com/post/everything-about-redis-24.html This is a real pain point for the entire Rails community and would improve Ruby's reputation immensely if we can solve this problem. ---Files-------------------------------- glibc_arena_2.png (7.23 KB) jemalloc.png (21.1 KB) glibc-arena-2.log (60.3 KB) glibc.log (62.3 KB) jemalloc-3-5.log (58.5 KB) glibc.png (9.03 KB) -- https://bugs.ruby-lang.org/