[ruby-core:110708] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack

[ruby-core:110708] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack

[smcgivern (Sean McGivern)]

Issue #19122 has been reported by smcgivern (Sean McGivern).

----------------------------------------
Misc #19122: Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack
https://bugs.ruby-lang.org/issues/19122

* Author: smcgivern (Sean McGivern)
* Status: Open
* Priority: Normal
* Assignee: ioquatix (Samuel Williams)
----------------------------------------
I'd like to propose that Ruby stops using MADV_FREE when freeing a Fiber's stack, and switches to using MADV_DONTNEED even when MADV_FREE is supported.

MADV_FREE is used in one place in the Ruby codebase, when freeing the stack of a freed Fiber: https://git.ruby-lang.org/ruby.git/tree/cont.c#n683

The comment for `fiber_pool_stack_free` says:

```c
// We advise the operating system that the stack memory pages are no longer being used.
// This introduce some performance overhead but allows system to relaim memory when there is pressure.
```

Where possible (i.e. on Linux 4.5 and later), `fiber_pool_stack_free` uses `MADV_FREE` over `MADV_DONTNEED`. This has the side effect that memory statistics such as RSS will not reduce until and unless the OS actually reclaims that memory. If that doesn't happen, then the reported memory usage via RSS will be much higher than the 'real' memory usage.

If this was pervasive throughtout the Ruby codebase then that would be one thing, but currently this is just for Fiber. This means that:

1. A program that doesn't use Fiber will have somewhat reliable RSS statistics on recent Linux.
2. A program that heavily uses Fiber (such as something using Async::HTTP) will see an inflated RSS statistic.

Go made a similar change to the one I'm proposing here for similar reasons: https://github.com/golang/go/issues/42330

> While `MADV_FREE` is somewhat faster than `MADV_DONTNEED`, it doesn't affect many of the statistics that `MADV_DONTNEED` does until the memory is actually reclaimed. This generally leads to poor user experience, like confusing stats in `top` and other monitoring tools; and bad integration with management systems that respond to memory usage.
> [...]
> I propose we change the default to prefer `MADV_DONTNEED` over `MADV_FREE`, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of `MADV_FREE` ultimately doesn't meet our needs.

As an aside, MADV_FREE was not used in Ruby 3.1 (https://bugs.ruby-lang.org/issues/19101), and I haven't found any bugs filed about this behaviour other than that one.



-- 
https://bugs.ruby-lang.org/

[ruby-core:110709] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack

[smcgivern (Sean McGivern)]

Issue #19122 has been updated by smcgivern (Sean McGivern).


@ioquatix apologies for the direct assignment; you just seemed like the person who has the most knowledge and investment in the current state of Fiber.

----------------------------------------
Misc #19122: Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack
https://bugs.ruby-lang.org/issues/19122#change-100048

* Author: smcgivern (Sean McGivern)
* Status: Open
* Priority: Normal
* Assignee: ioquatix (Samuel Williams)
----------------------------------------
I'd like to propose that Ruby stops using MADV_FREE when freeing a Fiber's stack, and switches to using MADV_DONTNEED even when MADV_FREE is supported.

MADV_FREE is used in one place in the Ruby codebase, when freeing the stack of a freed Fiber: https://git.ruby-lang.org/ruby.git/tree/cont.c#n683

The comment for `fiber_pool_stack_free` says:

```c
// We advise the operating system that the stack memory pages are no longer being used.
// This introduce some performance overhead but allows system to relaim memory when there is pressure.
```

Where possible (i.e. on Linux 4.5 and later), `fiber_pool_stack_free` uses `MADV_FREE` over `MADV_DONTNEED`. This has the side effect that memory statistics such as RSS will not reduce until and unless the OS actually reclaims that memory. If that doesn't happen, then the reported memory usage via RSS will be much higher than the 'real' memory usage.

If this was pervasive throughtout the Ruby codebase then that would be one thing, but currently this is just for Fiber. This means that:

1. A program that doesn't use Fiber will have somewhat reliable RSS statistics on recent Linux.
2. A program that heavily uses Fiber (such as something using Async::HTTP) will see an inflated RSS statistic.

Go made a similar change to the one I'm proposing here for similar reasons: https://github.com/golang/go/issues/42330

> While `MADV_FREE` is somewhat faster than `MADV_DONTNEED`, it doesn't affect many of the statistics that `MADV_DONTNEED` does until the memory is actually reclaimed. This generally leads to poor user experience, like confusing stats in `top` and other monitoring tools; and bad integration with management systems that respond to memory usage.
> [...]
> I propose we change the default to prefer `MADV_DONTNEED` over `MADV_FREE`, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of `MADV_FREE` ultimately doesn't meet our needs.

As an aside, MADV_FREE was not used in Ruby 3.1 (https://bugs.ruby-lang.org/issues/19101), and I haven't found any bugs filed about this behaviour other than that one.



-- 
https://bugs.ruby-lang.org/

[ruby-core:110830] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack

[ioquatix (Samuel Williams)]

Issue #19122 has been updated by ioquatix (Samuel Williams).


I don't have a strong opinion about this, but I'm generally against loosing performance.

Maybe it can be controlled using environment variable.

----------------------------------------
Misc #19122: Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack
https://bugs.ruby-lang.org/issues/19122#change-100188

* Author: smcgivern (Sean McGivern)
* Status: Open
* Priority: Normal
* Assignee: ioquatix (Samuel Williams)
----------------------------------------
I'd like to propose that Ruby stops using MADV_FREE when freeing a Fiber's stack, and switches to using MADV_DONTNEED even when MADV_FREE is supported.

MADV_FREE is used in one place in the Ruby codebase, when freeing the stack of a freed Fiber: https://git.ruby-lang.org/ruby.git/tree/cont.c#n683

The comment for `fiber_pool_stack_free` says:

```c
// We advise the operating system that the stack memory pages are no longer being used.
// This introduce some performance overhead but allows system to relaim memory when there is pressure.
```

Where possible (i.e. on Linux 4.5 and later), `fiber_pool_stack_free` uses `MADV_FREE` over `MADV_DONTNEED`. This has the side effect that memory statistics such as RSS will not reduce until and unless the OS actually reclaims that memory. If that doesn't happen, then the reported memory usage via RSS will be much higher than the 'real' memory usage.

If this was pervasive throughtout the Ruby codebase then that would be one thing, but currently this is just for Fiber. This means that:

1. A program that doesn't use Fiber will have somewhat reliable RSS statistics on recent Linux.
2. A program that heavily uses Fiber (such as something using Async::HTTP) will see an inflated RSS statistic.

Go made a similar change to the one I'm proposing here for similar reasons: https://github.com/golang/go/issues/42330

> While `MADV_FREE` is somewhat faster than `MADV_DONTNEED`, it doesn't affect many of the statistics that `MADV_DONTNEED` does until the memory is actually reclaimed. This generally leads to poor user experience, like confusing stats in `top` and other monitoring tools; and bad integration with management systems that respond to memory usage.
> [...]
> I propose we change the default to prefer `MADV_DONTNEED` over `MADV_FREE`, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of `MADV_FREE` ultimately doesn't meet our needs.

As an aside, MADV_FREE was not used in Ruby 3.1 (https://bugs.ruby-lang.org/issues/19101), and I haven't found any bugs filed about this behaviour other than that one.



-- 
https://bugs.ruby-lang.org/

[ruby-core:111007] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack

[smcgivern (Sean McGivern)]

Issue #19122 has been updated by smcgivern (Sean McGivern).


ioquatix (Samuel Williams) wrote in #note-2:
> I don't have a strong opinion about this, but I'm generally against loosing performance.
> 
> Maybe it can be controlled using environment variable.

Having this user-controllable works for me, although I don't know about the Ruby project's general stance on toggles like this.

@ioquatix is https://github.com/socketry/falcon-benchmark the right test suite to run here? I'm not sure if that's what you ran in https://bugs.ruby-lang.org/issues/15997#note-19, or if that was slightly different.

I'm happy to try this out with the various options here to quantify the current state - just point me in the right direction :-)

----------------------------------------
Misc #19122: Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack
https://bugs.ruby-lang.org/issues/19122#change-100257

* Author: smcgivern (Sean McGivern)
* Status: Open
* Priority: Normal
* Assignee: ioquatix (Samuel Williams)
----------------------------------------
I'd like to propose that Ruby stops using MADV_FREE when freeing a Fiber's stack, and switches to using MADV_DONTNEED even when MADV_FREE is supported.

MADV_FREE is used in one place in the Ruby codebase, when freeing the stack of a freed Fiber: https://git.ruby-lang.org/ruby.git/tree/cont.c#n683

The comment for `fiber_pool_stack_free` says:

```c
// We advise the operating system that the stack memory pages are no longer being used.
// This introduce some performance overhead but allows system to relaim memory when there is pressure.
```

Where possible (i.e. on Linux 4.5 and later), `fiber_pool_stack_free` uses `MADV_FREE` over `MADV_DONTNEED`. This has the side effect that memory statistics such as RSS will not reduce until and unless the OS actually reclaims that memory. If that doesn't happen, then the reported memory usage via RSS will be much higher than the 'real' memory usage.

If this was pervasive throughtout the Ruby codebase then that would be one thing, but currently this is just for Fiber. This means that:

1. A program that doesn't use Fiber will have somewhat reliable RSS statistics on recent Linux.
2. A program that heavily uses Fiber (such as something using Async::HTTP) will see an inflated RSS statistic.

Go made a similar change to the one I'm proposing here for similar reasons: https://github.com/golang/go/issues/42330

> While `MADV_FREE` is somewhat faster than `MADV_DONTNEED`, it doesn't affect many of the statistics that `MADV_DONTNEED` does until the memory is actually reclaimed. This generally leads to poor user experience, like confusing stats in `top` and other monitoring tools; and bad integration with management systems that respond to memory usage.
> [...]
> I propose we change the default to prefer `MADV_DONTNEED` over `MADV_FREE`, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of `MADV_FREE` ultimately doesn't meet our needs.

As an aside, MADV_FREE was not used in Ruby 3.1 (https://bugs.ruby-lang.org/issues/19101), and I haven't found any bugs filed about this behaviour other than that one.



-- 
https://bugs.ruby-lang.org/

[ruby-core:113630] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack

[ioquatix (Samuel Williams) via ruby-core]

Issue #19122 has been updated by ioquatix (Samuel Williams).


https://github.com/ruby/ruby/pull/7855 should enable you to test different advice. Just bit shift it to the left to enable any `MADV_...` value to be specified.

----------------------------------------
Misc #19122: Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack
https://bugs.ruby-lang.org/issues/19122#change-103271

* Author: smcgivern (Sean McGivern)
* Status: Open
* Priority: Normal
* Assignee: ioquatix (Samuel Williams)
----------------------------------------
I'd like to propose that Ruby stops using MADV_FREE when freeing a Fiber's stack, and switches to using MADV_DONTNEED even when MADV_FREE is supported.

MADV_FREE is used in one place in the Ruby codebase, when freeing the stack of a freed Fiber: https://git.ruby-lang.org/ruby.git/tree/cont.c#n683

The comment for `fiber_pool_stack_free` says:

```c
// We advise the operating system that the stack memory pages are no longer being used.
// This introduce some performance overhead but allows system to relaim memory when there is pressure.
```

Where possible (i.e. on Linux 4.5 and later), `fiber_pool_stack_free` uses `MADV_FREE` over `MADV_DONTNEED`. This has the side effect that memory statistics such as RSS will not reduce until and unless the OS actually reclaims that memory. If that doesn't happen, then the reported memory usage via RSS will be much higher than the 'real' memory usage.

If this was pervasive throughtout the Ruby codebase then that would be one thing, but currently this is just for Fiber. This means that:

1. A program that doesn't use Fiber will have somewhat reliable RSS statistics on recent Linux.
2. A program that heavily uses Fiber (such as something using Async::HTTP) will see an inflated RSS statistic.

Go made a similar change to the one I'm proposing here for similar reasons: https://github.com/golang/go/issues/42330

> While `MADV_FREE` is somewhat faster than `MADV_DONTNEED`, it doesn't affect many of the statistics that `MADV_DONTNEED` does until the memory is actually reclaimed. This generally leads to poor user experience, like confusing stats in `top` and other monitoring tools; and bad integration with management systems that respond to memory usage.
> [...]
> I propose we change the default to prefer `MADV_DONTNEED` over `MADV_FREE`, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of `MADV_FREE` ultimately doesn't meet our needs.

As an aside, MADV_FREE was not used in Ruby 3.1 (https://bugs.ruby-lang.org/issues/19101), and I haven't found any bugs filed about this behaviour other than that one.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:113652] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack

[ioquatix (Samuel Williams) via ruby-core]

Issue #19122 has been updated by ioquatix (Samuel Williams).


If you want to use a specific mode (OS specific), you can do this:

On Linux, find the mode, e.g. `MADV_DONTNEED = 6` (https://github.com/torvalds/linux/blob/933174ae28ba72ab8de5b35cb7c98fc211235096/arch/alpha/include/uapi/asm/mman.h#L50).

Shift it one bit to the left (i.e. multiply by 2) `= 12`.

Then run Ruby like this:

```
> RUBY_SHARED_FIBER_POOL_FREE_STACKS=12 ruby
```

This will force the use of `MADV_DONTNEED`. However, no validation is done, so the OS may not accept it. So it will also emit a warning that this behaviour is OS dependent and may crash your Ruby interpreter.

With this in place, you should be able to test your code.

As for making this the default, I suppose we could consider it but we'd need to confirm the performance impact.

----------------------------------------
Misc #19122: Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack
https://bugs.ruby-lang.org/issues/19122#change-103295

* Author: smcgivern (Sean McGivern)
* Status: Open
* Priority: Normal
* Assignee: ioquatix (Samuel Williams)
----------------------------------------
I'd like to propose that Ruby stops using MADV_FREE when freeing a Fiber's stack, and switches to using MADV_DONTNEED even when MADV_FREE is supported.

MADV_FREE is used in one place in the Ruby codebase, when freeing the stack of a freed Fiber: https://git.ruby-lang.org/ruby.git/tree/cont.c#n683

The comment for `fiber_pool_stack_free` says:

```c
// We advise the operating system that the stack memory pages are no longer being used.
// This introduce some performance overhead but allows system to relaim memory when there is pressure.
```

Where possible (i.e. on Linux 4.5 and later), `fiber_pool_stack_free` uses `MADV_FREE` over `MADV_DONTNEED`. This has the side effect that memory statistics such as RSS will not reduce until and unless the OS actually reclaims that memory. If that doesn't happen, then the reported memory usage via RSS will be much higher than the 'real' memory usage.

If this was pervasive throughtout the Ruby codebase then that would be one thing, but currently this is just for Fiber. This means that:

1. A program that doesn't use Fiber will have somewhat reliable RSS statistics on recent Linux.
2. A program that heavily uses Fiber (such as something using Async::HTTP) will see an inflated RSS statistic.

Go made a similar change to the one I'm proposing here for similar reasons: https://github.com/golang/go/issues/42330

> While `MADV_FREE` is somewhat faster than `MADV_DONTNEED`, it doesn't affect many of the statistics that `MADV_DONTNEED` does until the memory is actually reclaimed. This generally leads to poor user experience, like confusing stats in `top` and other monitoring tools; and bad integration with management systems that respond to memory usage.
> [...]
> I propose we change the default to prefer `MADV_DONTNEED` over `MADV_FREE`, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of `MADV_FREE` ultimately doesn't meet our needs.

As an aside, MADV_FREE was not used in Ruby 3.1 (https://bugs.ruby-lang.org/issues/19101), and I haven't found any bugs filed about this behaviour other than that one.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:113702] [Ruby master Misc#19122] Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack

[smcgivern (Sean McGivern) via ruby-core]

Issue #19122 has been updated by smcgivern (Sean McGivern).


ioquatix (Samuel Williams) wrote in #note-5:
> If you want to use a specific mode (OS specific), you can do this:
> 
> On Linux, find the mode, e.g. `MADV_DONTNEED = 6` (https://github.com/torvalds/linux/blob/933174ae28ba72ab8de5b35cb7c98fc211235096/arch/alpha/include/uapi/asm/mman.h#L50).
> 
> Shift it one bit to the left (i.e. multiply by 2) `= 12`.
> 
> Then run Ruby like this:
> 
> ```
> > RUBY_SHARED_FIBER_POOL_FREE_STACKS=12 ruby
> ```

Thanks for adding this!
 
> As for making this the default, I suppose we could consider it but we'd need to confirm the performance impact. I would prefer to have high performance by default. RSS is a poor measurement IMHO. What about USS and PSS? Are they impacted the same way? Can we subtract the usage that can be later dropped?

I mean, as I said in the original description, this is the only part of Ruby that works this way, which makes it surprising. If usage of fibers becomes more common in the wild, or `MADV_FREE` gets used in other areas where Ruby frees memory, then I would predict more reports noting this behaviour. Other languages have reverted their usage of `MADV_FREE` for similar reasons.

We can construct another metric that gives us exactly what we want (for cAdvisor, we'd need something like https://github.com/google/cadvisor/issues/3197), but not through PSS or USS; both of those track RSS closely here, at least in my tests. We need to look at either `active_anon + inactive_anon` (for a cgroups `memory.stat` file), or `LazyFree` plus some other things I'm not sure about (for a process's `smaps_rollup`). This won't be the default in most monitoring tools, or in `top` et al, so as I said above, that wouldn't necessarily stop issues coming in.

----------------------------------------
Misc #19122: Use MADV_DONTNEED instead of MADV_FREE when freeing a Fiber's stack
https://bugs.ruby-lang.org/issues/19122#change-103348

* Author: smcgivern (Sean McGivern)
* Status: Open
* Priority: Normal
* Assignee: ioquatix (Samuel Williams)
----------------------------------------
I'd like to propose that Ruby stops using MADV_FREE when freeing a Fiber's stack, and switches to using MADV_DONTNEED even when MADV_FREE is supported.

MADV_FREE is used in one place in the Ruby codebase, when freeing the stack of a freed Fiber: https://git.ruby-lang.org/ruby.git/tree/cont.c#n683

The comment for `fiber_pool_stack_free` says:

```c
// We advise the operating system that the stack memory pages are no longer being used.
// This introduce some performance overhead but allows system to relaim memory when there is pressure.
```

Where possible (i.e. on Linux 4.5 and later), `fiber_pool_stack_free` uses `MADV_FREE` over `MADV_DONTNEED`. This has the side effect that memory statistics such as RSS will not reduce until and unless the OS actually reclaims that memory. If that doesn't happen, then the reported memory usage via RSS will be much higher than the 'real' memory usage.

If this was pervasive throughtout the Ruby codebase then that would be one thing, but currently this is just for Fiber. This means that:

1. A program that doesn't use Fiber will have somewhat reliable RSS statistics on recent Linux.
2. A program that heavily uses Fiber (such as something using Async::HTTP) will see an inflated RSS statistic.

Go made a similar change to the one I'm proposing here for similar reasons: https://github.com/golang/go/issues/42330

> While `MADV_FREE` is somewhat faster than `MADV_DONTNEED`, it doesn't affect many of the statistics that `MADV_DONTNEED` does until the memory is actually reclaimed. This generally leads to poor user experience, like confusing stats in `top` and other monitoring tools; and bad integration with management systems that respond to memory usage.
> [...]
> I propose we change the default to prefer `MADV_DONTNEED` over `MADV_FREE`, to favor user-friendliness and minimal surprise over performance. I think it's become clear that Linux's implementation of `MADV_FREE` ultimately doesn't meet our needs.

As an aside, MADV_FREE was not used in Ruby 3.1 (https://bugs.ruby-lang.org/issues/19101), and I haven't found any bugs filed about this behaviour other than that one.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/