From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=AWL,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id E39151F4B4 for ; Thu, 1 Oct 2020 21:16:17 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 213B1120B0D; Fri, 2 Oct 2020 06:15:37 +0900 (JST) Received: from xtrwkhkc.outbound-mail.sendgrid.net (xtrwkhkc.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id 24896120B09 for ; Fri, 2 Oct 2020 06:15:34 +0900 (JST) Received: by filterdrecv-p3mdw1-5dd6bc5999-gcr92 with SMTP id filterdrecv-p3mdw1-5dd6bc5999-gcr92-20-5F764713-6A 2020-10-01 21:16:03.606788532 +0000 UTC m=+861453.437097319 Received: from herokuapp.com (unknown) by geopod-ismtpd-6-1 (SG) with ESMTP id Pu2t0sTPRZOAU1enSWhl0g for ; Thu, 01 Oct 2020 21:16:03.554 +0000 (UTC) Date: Thu, 01 Oct 2020 21:16:03 +0000 (UTC) From: eregontp@gmail.com Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 76105 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Feature X-Redmine-Issue-Id: 17176 X-Redmine-Issue-Author: tenderlovemaking X-Redmine-Sender: Eregon X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?KippOI8ZHtTweq7XfQzW93937kJ4QNWwSBuHnaMEcr3odQ98N13baHDgCI9kxU?= =?us-ascii?Q?LIVSnqb11jmu1IWCrfhqntI8IDACkN544FCTAb1?= =?us-ascii?Q?pWZuD7oa8uh8Wz8azcqPe4Q51B5abjIyJqhVy+B?= =?us-ascii?Q?ieX6s5cYfAHkfn860SImfXBV+wsvWd+WHtLPCQr?= =?us-ascii?Q?cjk7zS367Aj5WOZb6dL9ZwbHUdQE0gHSjH8aSOX?= =?us-ascii?Q?VkRTXfdwQGhW54t3k=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 100263 Subject: [ruby-core:100263] [Ruby master Feature#17176] GC.enable_autocompact / GC.disable_autocompact X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #17176 has been updated by Eregon (Benoit Daloze). tenderlovemaking (Aaron Patterson) wrote in #note-5: > ko1 (Koichi Sasada) wrote in #note-4: > > Another idea is GC.enable(compact: true/false). > > I like this better than `GC.auto_compact = true/false` because we can add more options. I'll change to use that. One potential issue is that this pattern `GC.disable; begin; ...; ensure; GC.enable; end` will unintentionally lose the `compact` flag. I'm not sure it matters too much because (I hope) this pattern is rare. Another downside is it makes it more complicated to know if auto-compaction can be set (need to check arity, and won't work for the second flag vs `GC.respond_to?(:auto_compact=)`. ---------------------------------------- Feature #17176: GC.enable_autocompact / GC.disable_autocompact https://bugs.ruby-lang.org/issues/17176#change-87850 * Author: tenderlovemaking (Aaron Patterson) * Status: Open * Priority: Normal ---------------------------------------- Hi, I'd like to make compaction automatic eventually. As a first step, I would like to introduce two functions: * GC.enable_autocompact * GC.disable_autocompact One function enables auto compaction, the other one disables it. Automatic compaction is *disabled* by default. When it is enabled it will happen only on every major GC. I've made a pull request here: https://github.com/ruby/ruby/pull/3547 This patch makes _object movement_ happen at the same time as page sweep. When one page finishes sweeping, that page is filled. ## Sweep + Move Phase During sweep, we keep a pointer to the current sweeping page. This pointer is kept in [`heap->sweeping_page`](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4817). At the beginning of sweep, this is the *first* element of the heap's linked list. At the same time, the compaction process points at the *last* page in the heap, and that is stored in `heap->compact_cursor` [here](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L5023). Incremental sweeping sweeps one page at a time in the [`gc_page_sweep` function](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4624). At the end of that function, we call [`gc_fill_swept_page`](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4738-L4742). `gc_fill_swept_page` fills the page that was just swept and moves the movement cursor towards the sweeping cursor. When the sweeping cursor and the movement cursor meet, sweeping is paused, and references are updated. This can happen in 2 ways, the sweeping cursor "runs in to the moving cursor" which is [here](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4634-L4644). Or the moving cursor runs in to the sweep cursor which happens [here](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4425-L4430). Either way, the sweep step is paused and references are updated. ## Reference Updating Reference updating hasn't changed, but since reference updating happens before the GC finishes a cycle, it must take in to account garbage objects [here](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L8971-L8977). ## Read Barrier During the sweep phase, some objects may touch other objects. For example, `T_CLASS` [must remove itself from a parent class](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L2769-L2770). ```ruby class A; end class B < A; end const_set(:B, nil) ``` When `B` is freed, it must remove itself from `A`'s subclasses. But what if `A` moved? To fix this, I've introduced a read barrier. The read barrier protects `heap_page_body` using `mprotect`. If something tries to read from the page, an exception will occur and we can move all objects back to the page (invalidate the movement). The lock function is [here](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4321-L4335). The unlock function is [here](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4337-L4351). It uses `sigaction` to catch the exception [here](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4514-L4530). ## Cross Platform `mprotect` and `sigaction` are not cross platform, they doesn't work on Windows. On Windows the read barrier uses exception handlers that are built in to Windows. I implemented them [here](https://github.com/ruby/ruby/blob/8a4d8fa0ea463d44486bf2447ea9830593768fd7/gc.c#L4471-L4503). The read barrier seems to work on all platforms we're testing. ## Statistics `GC.stat(:compact_count)` contains the number of times compaction has happened, so we can write things like this: ```ruby GC.enable_autocompact cc = GC.stat(:compact_count) list = [] loop do 500.times { list << Object.new } break if cc < GC.stat(:compact_count) end p GC.stat(:compact_count) ``` We can check when the read barrier is triggered with `GC.stat(:read_barrier_faults)` I've also added `GC.latest_compact_info` so you can see what types of objects moved and how many. For example: ``` [aaron@tc-lan-adapter ~/g/ruby (autocompact)]$ cat test.rb list = [] 500.times { list << Object.new Object.new Object.new } GC.enable_autocompact count = GC.stat :compact_count loop do list << Object.new break if GC.stat(:compact_count) > count end p GC.latest_compact_info [aaron@tc-lan-adapter ~/g/ruby (autocompact)]$ make runruby ./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems ./test.rb {:considered=>{:T_OBJECT=>408}, :moved=>{:T_OBJECT=>408}} [aaron@tc-lan-adapter ~/g/ruby (autocompact)]$ ``` ## Recap New methods: * GC.enable_autocompact * GC.disable_autocompact * GC.last_compact_info New statistics in `GC.stat`: * GC.stat(:read_barrier_faults) Diff is here: https://github.com/ruby/ruby/pull/3547 -- https://bugs.ruby-lang.org/