From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 5E68D1F609 for ; Thu, 29 Nov 2018 01:56:10 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id EF105120E34; Thu, 29 Nov 2018 10:56:06 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id D365F120D73 for ; Thu, 29 Nov 2018 10:56:04 +0900 (JST) Received: by filter0178p3mdw1.sendgrid.net with SMTP id filter0178p3mdw1-4957-5BFF4731-B 2018-11-29 01:56:01.649550849 +0000 UTC m=+1144517.781484784 Received: from herokuapp.com (ec2-54-90-189-182.compute-1.amazonaws.com [54.90.189.182]) by ismtpd0032p1iad2.sendgrid.net (SG) with ESMTP id iRs6gS9PSWmqECSicpwdOQ for ; Thu, 29 Nov 2018 01:56:01.588 +0000 (UTC) Date: Thu, 29 Nov 2018 01:56:02 +0000 (UTC) From: tenderlove@ruby-lang.org To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 65544 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 15349 X-Redmine-Issue-Author: tenderlovemaking X-Redmine-Sender: tenderlovemaking X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS4Oe8FuvwncD6NiLrwvumCv6DapvPAYESEaZS Ji5+2+CjSU9xnfF9KBFMHx7t+AvZVtMZSmgXXB+sh5ip1Abnwk7q1UljAtluVeQSQs1UScUY4cNgux P8vZeO0IvxYIjD0Cptp0MD5WYV6GrkUjBCe+g6N9DUWtfpBf3fevgMNJGA== X-ML-Name: ruby-core X-Mail-Count: 90147 Subject: [ruby-core:90147] [Ruby trunk Feature#15349] Use a shared array for the `duparray` instruction X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #15349 has been updated by tenderlovemaking (Aaron Patterson). normalperson (Eric Wong) wrote: > tenderlove@ruby-lang.org wrote: > > I was going to apply this patch, but it modifies `rb_ary_resurrect` and I wanted to double check. `rb_str_resurrect` will create shared strings, so it seems OK for `rb_ary_resurrect` to create shared arrays, IMO. > > It's probably fine; I would not expect us to have bugs in our > shared array handling code at this point. Ok, I'll apply the patch. > > Does anyone have opinions on this? If not, I'll just apply the patch. > > > > (Also I noticed trunk seems to allocate a lot more memory at boot) > > Looks like transient heap (only checked miniruby, so no gems loaded). > Disabling transient heap (below) seems to help, but real-world results > would likely be different: I tried disabling theap and re-running the test. It looks like disabling theap brings the memory usage back to what Ruby 2.5 does: ![remove theap](https://user-images.githubusercontent.com/3124/49194017-49fe8280-f336-11e8-8131-dd97ba4922bf.png) I tried a similar test, but booting a Rails application. It looks like theap just adds a constant overhead to the application: ![theap, no theap, Ruby 2.5 on Rails](https://user-images.githubusercontent.com/3124/49194094-9944b300-f336-11e8-8b54-6bfec63b78db.png) Disabling theap makes memory usage on 2.6 *lower* than 2.5, where enabling theap keeps it constantly above 2.5 levels. I'll open another ticket for this issue. ---------------------------------------- Feature #15349: Use a shared array for the `duparray` instruction https://bugs.ruby-lang.org/issues/15349#change-75263 * Author: tenderlovemaking (Aaron Patterson) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- In this example code: ~~~ ruby def foo [1, 2, 3, 4] end ~~~ The array literal uses a duparray instruction. Before this patch, rb_ary_resurrect would malloc and memcpy a new array buffer. This patch changes rb_ary_resurrect to use ary_make_partial so that the new array object shares the underlying buffer with the array stored in the instruction sequences. Before this patch, the new array object is not shared: ~~~ $ ruby -r objspace -e'p ObjectSpace.dump([1, 2, 3, 4])' "{\"address\":\"0x7fa2718372d0\", \"type\":\"ARRAY\", \"class\":\"0x7fa26f8b0010\", \"length\":4, \"memsize\":72, \"flags\":{\"wb_protected\":true}}\n" ~~~ After this patch: ~~~ $ ./ruby -r objspace -e'p ObjectSpace.dump([1, 2, 3, 4])' "{\"address\":\"0x7f9a76883638\", \"type\":\"ARRAY\", \"class\":\"0x7f9a758af900\", \"length\":4, \"shared\":true, \"references\":[\"0x7f9a768837c8\"], \"memsize\":40, \"flags\":{\"wb_protected\":true}}\n" ~~~ I wrote a test program: ~~~ ruby def foo [1, 2, 3, 4] end list = [] 10000.times { list << foo } GC.start puts "ready #{$$}" system "malloc_history #{$$} -allEvents > #{RUBY_VERSION}-#{ARGV[0]}.log" # MacOS specific ~~~ This test program uses a MacOS specific tool to get a list of all malloc / free calls (should be able to do the same with valgrind, I just don't know the command). I compared trunk, trunk + this patch, and Ruby 2.5.3. All tests disable RubyGems. Here is a graph of the results: ![smallish array results](https://user-images.githubusercontent.com/3124/49109227-09700d80-f23f-11e8-9597-d87f15b69dbf.png) The X axis is sample number, and the Y axis is total bytes the process is using at that sample. Each sample is a call to malloc, so the closer the line is to the origin (0, 0), the better (it means fewer samples and less live memory). After this patch there are fewer calls to malloc than in either trunk or 2.5 for this test program. If I modify the test program to be like this: ~~~ ruby def foo [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ] end list = [] 10000.times { list << foo } GC.start puts "ready #{$$}" system "malloc_history #{$$} -allEvents > #{RUBY_VERSION}-#{ARGV[0]}.log" ~~~ The difference in memory usage becomes more pronounced: ![large array alloc](https://user-images.githubusercontent.com/3124/49114573-62df3900-f24d-11e8-99c4-957daadd7628.png) I noticed that strings already use this technique: ~~~ $ ruby -v --disable-gems -robjspace -e'p ObjectSpace.dump("abcdefghijklmnopqrstuvwxyz")' ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-darwin18] "{\"address\":\"0x00007fad18051528\", \"type\":\"STRING\", \"class\":\"0x00007fad180d3c08\", \"shared\":true, \"encoding\":\"UTF-8\", \"references\":[\"0x00007fad18051640\"], \"memsize\":40, \"flags\":{\"wb_protected\":true}}\n" ~~~ I was going to apply this patch, but it modifies `rb_ary_resurrect` and I wanted to double check. `rb_str_resurrect` will create shared strings, so it seems OK for `rb_ary_resurrect` to create shared arrays, IMO. Does anyone have opinions on this? If not, I'll just apply the patch. (Also I noticed trunk seems to allocate a lot more memory at boot) Thanks! ---Files-------------------------------- 0001-Use-a-shared-array-for-the-duparray-instruction.patch (1.56 KB) -- https://bugs.ruby-lang.org/