From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-3.2 required=3.0 tests=BAYES_00,FORGED_HOTMAIL_RCVD2, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 5EC891F597 for ; Tue, 31 Jul 2018 00:31:07 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 40542120B3A; Tue, 31 Jul 2018 09:31:06 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id 07C01120B38 for ; Tue, 31 Jul 2018 09:31:03 +0900 (JST) Received: by filter0041p3mdw1.sendgrid.net with SMTP id filter0041p3mdw1-5774-5B5FADC5-34 2018-07-31 00:31:01.520367102 +0000 UTC m=+439781.091446635 Received: from herokuapp.com (ec2-54-196-202-207.compute-1.amazonaws.com [54.196.202.207]) by ismtpd0004p1iad1.sendgrid.net (SG) with ESMTP id hrIlIg3kQfSYmNRR6r1mcw Tue, 31 Jul 2018 00:31:01.510 +0000 (UTC) Date: Tue, 31 Jul 2018 00:31:02 +0000 (UTC) From: dennisb55@hotmail.com To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 63597 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 14718 X-Redmine-Issue-Author: mperham X-Redmine-Sender: bluz71 X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS6de+iC92dnHJRU7/ldM2rov93KYPjXe/12kz ZH95Si747+R+cnsHplvwMyRf/dQiVi/bvonU+0U2vxh+mRSk/5OIQ/hcqXZBLcvG8KywTjiGQ5wmCD Z3RwiNpcH5Iwcxkl7C0BuITDdpx6I2+DYaLaTH7I7i6BXmLBrqnmQ8NdUA== X-ML-Name: ruby-core X-Mail-Count: 88223 Subject: [ruby-core:88223] [Ruby trunk Feature#14718] Use jemalloc by default? X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #14718 has been updated by bluz71 (Dennis B). davidtgoldblatt (David Goldblatt) wrote: > I don't think this benchmark is a useful way to compare performance between versions 3 and 5 of jemalloc. In between them was the advent of time-based purging, where the allocator waits for a while (by default, around 10 seconds) before returning memory to the OS. That purging occurs (well, in the default but not recommended case where background threads are disabled) on an operation counter tick. By having low activity for a short program lifetime, it effectively disables returning memory to the OS, in a way that wouldn't actually persist if you were running a more realistic program. Even outside of that, we've started using MADV_FREE, which gives the kernel the right to reclaim memory, but in such a way that it will only do so if the machine is actually under memory pressure; this won't show up in test runs unless you're trying to make it happen. > > You could verify this by setting dirty decay and muzzy decay to 0 in the MALLOC_CONF environment variable (i.e. MALLOC_CONF="dirty_decay_ms:0,muzzy_decay_ms:0", unless I've typoed something). Very interesting indeed. I suspect many of us were not aware of that, I certainly wasn't. All the micro-testing conducted here and at [#14759](https://bugs.ruby-lang.org/issues/14759) indicated a large regression in jemalloc RSS behaviour post 3.X series. This is a difficult one. I wonder if we need to transistion across to Noah Gibbs Rails benchmark? Nevertheless this usage graph posted by Micke Perham is indicative of what many users experience with long-lived production Ruby applications on Linux: ![jemalloc](https://www.mikeperham.com/images/jemalloc.jpg) ---------------------------------------- Feature #14718: Use jemalloc by default? https://bugs.ruby-lang.org/issues/14718#change-73242 * Author: mperham (Mike Perham) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- I know Sam opened #9113 4 years ago to suggest this but I'm revisiting the topic to see if there's any movement here for Ruby 2.6 or 2.7. I supply a major piece of Ruby infrastructure (Sidekiq) and I keep hearing over and over how Ruby is terrible with memory, a huge memory hog with their Rails apps. My users switch to jemalloc and a miracle occurs: their memory usage drops massively. Some data points: https://twitter.com/brandonhilkert/status/987400365627801601 https://twitter.com/d_jones/status/989866391787335680 https://github.com/mperham/sidekiq/issues/3824#issuecomment-383072469 Redis moved to jemalloc many years ago and it solved all of their memory issues too. Their conclusion: the glibc allocator "sucks really really hard". http://oldblog.antirez.com/post/everything-about-redis-24.html This is a real pain point for the entire Rails community and would improve Ruby's reputation immensely if we can solve this problem. ---Files-------------------------------- glibc_arena_2.png (7.23 KB) jemalloc.png (21.1 KB) glibc-arena-2.log (60.3 KB) glibc.log (62.3 KB) jemalloc-3-5.log (58.5 KB) glibc.png (9.03 KB) -- https://bugs.ruby-lang.org/