From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 4D8401F51C for ; Sat, 19 May 2018 14:04:00 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 10E93120915; Sat, 19 May 2018 23:03:56 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id 88BAE120912 for ; Sat, 19 May 2018 23:03:53 +0900 (JST) Received: by filter0001p3iad2.sendgrid.net with SMTP id filter0001p3iad2-8577-5B002EC3-26 2018-05-19 14:03:47.340820824 +0000 UTC Received: from herokuapp.com (ec2-107-22-7-168.compute-1.amazonaws.com [107.22.7.168]) by ismtpd0008p1iad1.sendgrid.net (SG) with ESMTP id sVPH9dbTSk64yH2Wa9nWgw Sat, 19 May 2018 14:03:47.184 +0000 (UTC) Date: Sat, 19 May 2018 14:03:48 +0000 (UTC) From: mame@ruby-lang.org To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 62556 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 14759 X-Redmine-Issue-Author: normalperson X-Redmine-Sender: mame X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS7RAHFZ2ueXWF8+bUA23XcYwp+dV9cuk8UhX2 09+DVW/+w6MjQJoKgzB5d3UeiPhNarJ6q77gA8GjoFBgndZawE7C7d6K9yDSTa7O2Uc1dAEXpkbLoL en+k8iFiOQGVkh72+q2/K5BlpO1KMuxGPXOnSgdkAr5uSu2b14/vcmmY7w== X-ML-Name: ruby-core X-Mail-Count: 87193 Subject: [ruby-core:87193] [Ruby trunk Feature#14759] [PATCH] set M_ARENA_MAX for glibc malloc X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #14759 has been updated by mame (Yusuke Endoh). I tried to change Mike's script to use I/O, and I've created a script that works best with glibc with no MALLOC_ARENA_MAX specified. # glibc (default) $ time ./miniruby frag2.rb VmRSS: 852648 kB real 0m26.191s # glibc with MALLOC_ARENA_MAX=2 $ time MALLOC_ARENA_MAX=2 ./miniruby frag2.rb VmRSS: 1261032 kB real 0m29.072s # jemalloc 3.6.0 (shipped with Ubuntu) $ time LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so ./miniruby frag2.rb VmRSS: 966624 kB real 1m6.730s $ ./miniruby -v ruby 2.6.0dev (2018-05-19) [x86_64-linux] As you see, default glibc is the fastest and most memory-efficient. Comparing to that, glibc with MALLOC_ARENA_MAX=2 uses 1.5x memory, and jemalloc 3.6.0 is twice slow. I'm unsure what happens. I guess it is difficult to discuss this issue without glibc/jemalloc hackers. ``` # frag2.rb THREAD_COUNT = (ARGV[0] || "10").to_i File.write("tmp.txt", "x" * 1024 * 64) Threads = [] THREAD_COUNT.times do Threads << Thread.new do a = [] 100_000.times do a << open("tmp.txt") {|f| f.read } a.shift if a.size >= 1600 end end end Threads.each {|th| th.join } IO.foreach("/proc/#{$$}/status") do |line| print line if line =~ /VmRSS/ end if RUBY_PLATFORM =~ /linux/ ``` ---------------------------------------- Feature #14759: [PATCH] set M_ARENA_MAX for glibc malloc https://bugs.ruby-lang.org/issues/14759#change-72182 * Author: normalperson (Eric Wong) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- Not everybody benefits from jemalloc and the extra download+install time is not always worth it. Lets make the user experience for glibc malloc users better, too. Personally, I prefer using M_ARENA_MAX=1 (via MALLOC_ARENA_MAX env) myself, but there is currently a performance penalty for that. gc.c (Init_GC): set M_ARENA_MAX=2 for glibc malloc glibc malloc creates too many arenas and leads to fragmentation. Given the existence of the GVL, clamping to two arenas seems to be a reasonable trade-off for performance and memory usage. Some users (including myself for several years, now) prefer only one arena, now, so continue to respect users' wishes when MALLOC_ARENA_MAX is set. Thanks to Mike Perham for the reminder [ruby-core:86843] This doesn't seem to conflict with jemalloc, so it should be safe for all glibc-using systems. ---Files-------------------------------- 0001-gc.c-Init_GC-set-M_ARENA_MAX-2-for-glibc-malloc.patch (1.46 KB) -- https://bugs.ruby-lang.org/