[ruby-core:102071] [Ruby master Bug#17538] Assertion failure when rincgc is turned off

[peter@peterzhu.ca]

Issue #17538 has been reported by peterzhu2118 (Peter Zhu).

----------------------------------------
Bug #17538: Assertion failure when rincgc is turned off
https://bugs.ruby-lang.org/issues/17538

* Author: peterzhu2118 (Peter Zhu)
* Status: Open
* Priority: Normal
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN
----------------------------------------
[Pull request on GitHub](https://github.com/ruby/ruby/pull/4064).

When compiling with `-DUSE_RINCGC=0` (turn off rincgc) and `-DRGENGC_CHECK_MODE=1` (turn on assertions), there is an assertion error when running multiple ractors.

Reproduction script (on macOS 11.1):

```ruby
rs = (1..30).map do |i|
  Ractor.new(i) do |i|
    "r#{i}"
  end
end
```

Error:

```
Assertion Failed: ../gc.c:5193:gc_sweep_step:gc_mode(objspace) == gc_mode_sweeping ? heap->free_pages != NULL : 1
```

This happens because another ractor may be using this page during sweeping, so it appears like the page has free slots but the freelist of the page is empty.

For example, let's say we have pages P1, P2 and ractors R1, R2. Say R1 is using P1 and R2 is using P2. When P2 runs out of space, GC is started. But if nothing is swept in P1 and P1 still has space (in the ractor cache in R1), then `free_slots > 0` is true for P1 but `heap->free_pages == NULL` because the freelist of P2 is null.



-- 
https://bugs.ruby-lang.org/