From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 703BA1F66E for ; Thu, 20 Aug 2020 13:38:28 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 97542120A58; Thu, 20 Aug 2020 22:37:50 +0900 (JST) Received: from xtrwkhkc.outbound-mail.sendgrid.net (xtrwkhkc.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id 631E1120A00 for ; Thu, 20 Aug 2020 22:37:47 +0900 (JST) Received: by filterdrecv-p3iad2-5c98798b7-lvbn8 with SMTP id filterdrecv-p3iad2-5c98798b7-lvbn8-19-5F3E7CC3-46 2020-08-20 13:38:11.518474127 +0000 UTC m=+594522.001078792 Received: from herokuapp.com (unknown) by ismtpd0100p1mdw1.sendgrid.net (SG) with ESMTP id wxMdJhjRQ0aM0ypDAPlT_A for ; Thu, 20 Aug 2020 13:38:11.440 +0000 (UTC) Date: Thu, 20 Aug 2020 13:38:11 +0000 (UTC) From: samuel@oriontransfer.net Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 75486 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Feature X-Redmine-Issue-Id: 16786 X-Redmine-Issue-Author: ioquatix X-Redmine-Sender: ioquatix X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?cjxb6GWHefMLoR50bkJBcGo6DRiDl=2FNYcMZdY+Wj30RS1vVQ0x6F45WyGTqvhi?= =?us-ascii?Q?SlLLMDoga5KrBKrkQcrc48dDjNbrrlF6YG8PgsH?= =?us-ascii?Q?FFz3nXFSsoWHbNksCOmA5J9Vr28gCeplfttRyhp?= =?us-ascii?Q?hUynXW9XSAZ+r109pDS1wzQFxIog+SuzJduUCAp?= =?us-ascii?Q?ahafKM21R+tbX8Y=2FTCr4XC5ps+vGfAClhKCtQtA?= =?us-ascii?Q?P7tGS4XL4KDv+9wWw=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 99657 Subject: [ruby-core:99657] [Ruby master Feature#16786] Light-weight scheduler for improved concurrency. X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #16786 has been updated by ioquatix (Samuel Williams). Here is new PR: https://github.com/ruby/ruby/pull/3434 > [NEED TO CHANGE] doc/fiber.rdoc We will update it. > [NEED TO CHANGE] enter/exit_blocking_region callback Removed. However, I would like to add, that if we do not have some kind of support for finding blocking behaviour in Ruby code, it will be trouble for developers. So, more work is required here. > [MAYBE NEED TO CHANGE] API names I agree, some parts can be improved. So, I'll do review and continue to think about it during development. > [NEED TO CHANGE] Fiber After we discussed, I felt `Fiber.schedule{}` is good compromise. @matz can you comment if you are happy with this? I'm running informal poll to see community feedback: https://twitter.com/ioquatix/status/1295720461003882498 > [Suggestion] Callbacks It's good idea. So, I discussed many points around this topic with @nobu, @mame, @ko1, and I got the general feedback: - @ko1 said it was surprising we have public interface for waiting on raw file descriptor. - @nobu wondered if we can introduce something like `rb_io_wait(VALUE io, VALUE events)`. After many discussion, I'm leaning more towards NOT exposing file descriptor anywhere. It's CRuby's problem internally, and we have some options to solve it. SO: - I added `rb_io_wait(VALUE io, ...)`. - I reworked some existing methods to use it. - I made single entry point to scheduler: `Scheduler#io_wait(...)`. - I use `IO.for_fd` internally to handle raw file descriptors. I've also updated the naming convention for hooks. The name format is `{class}_{method}`. > I guess running Scheduler#run by the programmer explicitly is better than implicit invocation on any threads (1 line at the end of thread is not a cost, I guess) We discussed. I'm following Crystal method. I think it's good. @ko1 was concerned, how to disable scheduler, but you can assign nil if you want to cancel run at exit. We could also have separate method, e.g. `Scheduler#run_at_exit`. I'll think about it more and I'll also make the current implementation more consistent. > [Suggestion] nonblocking IO Completely agree. As this is a work in progress, documentation is not completed yet. Finally, I'd like to bring attention to a few more exciting points of the new PR. We are working on support for Linux io_uring and Windows IOCP. It's going to be important to shape the scheduler interface based on these modern interfaces rather than "legacy" epoll/kqueue. @dsh0416 has been working on implementing a compatible scheduler implementation and we have discussed requirements for efficient `io_uring`. It turns out we need to add hooks for `read` and `write`, and this allows us to implement zero-copy file-system and network I/O using event loop. (There is some related discussion here: https://bugs.ruby-lang.org/issues/17059) One exciting area of this is the multitude of system calls that `io_uring` supports, including `fallocate`, `madvise`, and so on. While we would not aim for that initially, it looks to be a bright future in terms of handling these operations in a non-blocking fashion. Such interfaces would be introduced, e.g. `Scheduler#io_fallocate` or similar. It's hard to anticipate all design issues, but so far I feel confident we are in a good position to leverage these interfaces (and more) as they become available/are supported. Linux itself is introducing more `io_uring` system calls, and not all of them would be supported by the given system (depending on Kernel version for example), so the C extension would need to handle this. ---------------------------------------- Feature #16786: Light-weight scheduler for improved concurrency. https://bugs.ruby-lang.org/issues/16786#change-87140 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- # Abstract We propose to introduce a light weight fiber scheduler, to improve the concurrency of Ruby code with minimal changes. # Background We have been discussing and considering options to improve Ruby scalability for several years. More context can be provided by the following discussions: - https://bugs.ruby-lang.org/issues/14736 - https://bugs.ruby-lang.org/issues/13618 The final Ruby Concurrency report provides some background on the various issues considered in the latest iteration: https://www.codeotaku.com/journal/2020-04/ruby-concurrency-final-report/index # Proposal We propose to introduce the following concepts: - A `Scheduler` interface which provides hooks for user-supplied event loops. - Non-blocking `Fiber` which can invoke the scheduler when it would otherwise block. ## Scheduler The per-thread fiber scheduler interface is used to intercept blocking operations. A typical implementation would be a wrapper for a gem like EventMachine or Async. This design provides separation of concerns between the event loop implementation and application code. It also allows for layered schedulers which can perform instrumentation, enforce constraints (e.g. during testing) and provide additional logging. You can see a [sample implementation here](https://github.com/socketry/async/pull/56). ```ruby class Scheduler # Wait for the given file descriptor to become readable. def wait_readable(io) end # Wait for the given file descriptor to become writable. def wait_writable(io) end # Wait for the given file descriptor to match the specified events within # the specified timeout. # @param event [Integer] a bit mask of +IO::WAIT_READABLE+, # `IO::WAIT_WRITABLE` and `IO::WAIT_PRIORITY`. # @param timeout [#to_f] the amount of time to wait for the event. def wait_any(io, events, timeout) end # Sleep the current task for the specified duration, or forever if not # specified. # @param duration [#to_f] the amount of time to sleep. def wait_sleep(duration = nil) end # The Ruby virtual machine is going to enter a system level blocking # operation. def enter_blocking_region end # The Ruby virtual machine has completed the system level blocking # operation. def exit_blocking_region end # Intercept the creation of a non-blocking fiber. def fiber(&block) Fiber.new(blocking: false, &block) end # Invoked when the thread exits. def run # Implement event loop here. end end ``` A thread has a non-blocking fiber scheduler. All blocking operations on non-blocking fibers are hooked by the scheduler and the scheduler can switch to another fiber. If any mutex is acquired by a fiber, then a scheduler is not called; the same behaviour as blocking Fiber. Schedulers can be written in Ruby. This is a desirable property as it allows them to be used in different implementations of Ruby easily. To enable non-blocking fiber switching on blocking operations: - Specify a scheduler: `Thread.current.scheduler = Scheduler.new`. - Create several non-blocking fibers: `Fiber.new(blocking:false) {...}`. - As the main fiber exits, `Thread.current.scheduler.run` is invoked which begins executing the event loop until all fibers are finished. ### Time/Duration Arguments Tony Arcieri suggested against using floating point values for time/durations, because they can accumulate rounding errors and other issues. He has a wealth of experience in this area so his advice should be considered carefully. However, I have yet to see these issues happen in an event loop. That being said, round tripping between `struct timeval` and `double`/`VALUE` seems a bit inefficient. One option is to have an opaque argument that responds to `to_f` as well as potentially `seconds` and `microseconds` or some other such interface (could be opaque argument supported by `IO.select` for example). ### File Descriptor Arguments Because of the public C interface we may need to support a specific set of wrappers for CRuby. ```c int rb_io_wait_readable(int); int rb_io_wait_writable(int); int rb_wait_for_single_fd(int fd, int events, struct timeval *tv); ``` One option is to introduce hooks specific to CRuby: ```ruby class Scheduler # Wrapper for rb_io_wait_readable(int) C function. def wait_readable_fd(fd) wait_readable(::IO.from_fd(fd, autoclose: false)) end # Wrapper for rb_io_wait_readable(int) C function. def wait_writable_fd(fd) wait_writable(::IO.from_fd(fd, autoclose: false)) end # Wrapper for rb_wait_for_single_fd(int) C function. def wait_for_single_fd(fd, events, duration) wait_any(::IO.from_fd(fd, autoclose: false), events, duration) end end ``` Alternatively, in CRuby, it may be possible to map from `fd` -> `IO` instance. Most C schedulers only care about file descriptor, so such a mapping will introduce a small performance penalty. In addition, most C level schedulers will not care about `IO` instance. ## Non-blocking Fiber We propose to introduce per-fiber flag `blocking: true/false`. A fiber created by `Fiber.new(blocking: true)` (the default `Fiber.new`) becomes a "blocking Fiber" and has no changes from current Fiber implementation. This includes the root fiber. A fiber created by `Fiber.new(blocking: false)` becomes a "non-blocking Fiber" and it will be scheduled by the per-thread scheduler when the blocking operations (blocking I/O, sleep, and so on) occurs. ```ruby Fiber.new(blocking: false) do puts Fiber.current.blocking? # false # May invoke `Thread.scheduler&.wait_readable`. io.read(...) # May invoke `Thread.scheduler&.wait_writable`. io.write(...) # Will invoke `Thread.scheduler&.wait_sleep`. sleep(n) end.resume ``` Non-blocking fibers also supports `Fiber#resume`, `Fiber#transfer` and `Fiber.yield` which are necessary to create a scheduler. ### Fiber Method We also introduce a new method which simplifes the creation of these non-blocking fibers: ```ruby Fiber do puts Fiber.current.blocking? # false end ``` This method invokes `Scheduler#fiber(...)`. The purpose of this method is to allow the scheduler to internally decide the policy for when to start the fiber, and whether to use symmetric or asymmetric fibers. If no scheduler is specified, it is a error: `RuntimeError.new("No scheduler is available")`. In the future we may expand this to support some kind of default scheduler. ## Non-blocking I/O `IO#nonblock` is an existing interface to control whether I/O uses blocking or non-blocking system calls. We can take advantage of this: - `IO#nonblock = false` prevents that particular IO from utilising the scheduler. This should be the default for `stderr`. - `IO#nonblock = true` enables that particular IO to utilise the scheduler. We should enable this where possible. As proposed by Eric Wong, we believe that making I/O non-blocking by default is the right approach. We have expanded his work in the current implementation. By doing this, when the user writes `Fiber do ... end` they are guaranteed the best possible concurrency possible, without any further changes to code. As an example, one of the tests shows `Net::HTTP.get` being used in this way with no further modifications required. To support this further, consider the counterpoint, that `Net::HTTP.get(..., blocking: false)` is required for concurrent requests. Library code may not expose the relevant options, sevearly limiting the user's ability to improve concurrency, even if that is what they desire. # Implementation We have an evolving implementation here: https://github.com/ruby/ruby/pull/3032 which we will continue to update as the proposal changes. # Evaluation This proposal provides the hooks for scheduling fibers. With regards to performance, there are several things to consider: - The impact of the scheduler design on non-concurrent workloads. We believe it's acceptable. - The impact of the scheduler design on concurrent workloads. Our results are promising. - The impact of different event loops on throughput and latency. We have independent tests which confirm the scalability of the approach. We can control for the first two in this proposal, and depending on the design we may help or hinder the wrapper implementation. In the tests, we provide a basic implementation using `IO.select`. As this proposal is finalised, we will introduce some basic benchmarks using this approach. # Discussion The following points are good ones for discussion: - Handling of file descriptors vs `IO` instances. - Handling of time/duration arguments. - General design and naming conventions. - Potential platform issues (e.g. CRuby vs JRuby vs TruffleRuby, etc). The following is planned to be described by @eregon in another design document: - Semantics of non-blocking mutex (e.g. `Mutex.new(blocking: false)` or some other approach). In the future we hope to extend the scheduler to handle other blocking operations, including name resolution, file I/O (by `io_uring`) and others. We may need to introduce additional hooks. If these hooks are not defined on the scheduler implementation, we will revert back to the blocking implementation where possible. -- https://bugs.ruby-lang.org/