From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-3.1 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_MED,SPF_PASS,T_RP_MATCHES_RCVD shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 0EA9220401 for ; Thu, 15 Jun 2017 20:28:15 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id CDD7012078F; Fri, 16 Jun 2017 05:28:12 +0900 (JST) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by neon.ruby-lang.org (Postfix) with ESMTPS id DAB95120788 for ; Fri, 16 Jun 2017 05:28:08 +0900 (JST) Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id EC59620401; Thu, 15 Jun 2017 20:28:06 +0000 (UTC) Date: Thu, 15 Jun 2017 20:28:06 +0000 From: Eric Wong To: ruby-core@ruby-lang.org Message-ID: <20170615202806.GA32346@whir> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-ML-Name: ruby-core X-Mail-Count: 81695 Subject: [ruby-core:81695] Re: [Ruby trunk Feature#13618] [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" samuel@oriontransfer.org wrote: > Thanks for your detailed reply. It's impressive and useful > that you have such a good knowledge of these issues. No problem. > I spent some time just thinking about this issue, and how this > feature tries to solve the problem in Ruby. > > On the one hand, I'm fundamentally opposed to increasing the > surface area of Ruby when it could be done by writing a gem. > This has a massive upstream cost, affecting both JRuby and > Rubinius. While I appreciate what you are saying w.r.t. > maximising usage, I feel like building this into Ruby will > cause stagnation of progress long term - one solution for all > problems isn't always ideal. Seeing initiatives like > stdgems.org only reinforces how I feel about this. I understood something it was already decided by matz and ko1 to do something along the lines of auto-Fiber. Though I can't find ko1's original message in the archives, it's mostly quoted in in my reply to him: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/80531 I should note some languages like Go, Erlang, Haskell, and the afore-mentioned Crystal all have lightweight threading along these lines in the core language. In their current state, Fibers are much less useful than the equivalents in those languages; while native Threads are too expensive. Something in between Fibers and Threads seems desirable; maybe we can give auto-Fiber another (short) name; but I'm not sure it's necessary. I was also influenced to explore lightweight threading in a rack-devel thread and the responses James Tucker wrote to me: Subject: big responses to slow clients: Rack vs PSGI It's somewhere in https://groups.google.com/group/rack-devel but that requires JS; so I can't view or link to it using w3m :< > Generally speaking - I really appreciate the work that's been > done here. I also feel like you've reinvented nio4r, async and > a bunch of other stuff, at a very low level, without as much > testing, compatibility, etc. That's a fair point about less testing and compatibility. But, I think there is more code using normal Ruby stdlib that can automatically take advantage of these changes so we'll be able to nail down any problems quickly. On a technical level, I consider the design of libev (used by nio4r and async) too limited in that it does not take advantage of thread-safety baked into kqueue and epoll. Thinking in terms of "events vs. threads" too limiting. As I've said before; combining them is advantageous because both have their uses. kqueue is an thread-friendly queue, so is epoll. This feels like the microkernel vs monolithic kernel debate, too. On one level, isolation and compartmentalization provided by micro-kernels is appealing; but the ease-of-development of a monolith allowed Linux to become the kernel for nearly everything, from tiny IoT devices to giant supercomputers. And that doesn't preclude things like loadable modules and FUSE for userspace filesystems from being useful, despite core filesystem drivers being bundled with Linux. So I think `async` can still be supported as an alternative for Ruby; but the bundled implementation can benefit more from tighter integration into the core. A more recent example might be git; which included high-level non-essential "porcelain" tools early on in addition to the core "plumbing". Initially, it was intended that separately maintained wrappers such as "cogito", would implement the porcelain UI bits and git would remain low-level plumbing. That ended up making both development and usage more complicated. Eventually git swallowed up most of the cogito functionality and cogito was abandoned. git also ended up with bundled functionality that would've been separately packaged in other VCSes, including import/export tools for email, CVS, SVN, etc. The most relevant example from git might be the bundling of libxdiff in git, allowing optimizations and tweaks not possible with an external diff. However, GIT_EXTERNAL_DIFF still remains supported for less-common use cases. On a non-technical level: Finally, this (ruby-core) is one of the few places I can still contribute to in the Ruby world. All other relevant Ruby projects requires running non-Free software (including JS) and having to abide accept Terms-of-Service set by a corporation. Fwiw, I agree with Rubinius philosophy of implementing more of Ruby in Ruby and would rather contribute to that; but the above is a huge factor in why I went on to work on C Ruby, instead. (the other major factor is I strongly prefer C to C++). > Ideally, we could move all socket related code into a gem - > perhaps that's already on the cards e.g. stdgems. Once that's > done, fixing issues like `exceptions: false` would be easier > since it can be versioned. Maybe that'll be done, too, but not my call. But what about IO.pipe, `backtick`, and IO.popen? > I was thinking about how we could expose this to Ruby - and > ideally, I think we should add two functions: > > `IO.wait_for_single_fd` and `IO.wait_for_pid`. The C functions > rb_wait_for_single_fd and rb_waitpid would invoke these > functions, and these functions would implement the current > logic of the current C functions. It probably makes sense to > think in more detail how these functions should work - e.g. > `wait_for_multiple_fds` (or `select`), or something more > elaborate. Maybe.. I guess we already have IO#wait_*able in io/wait; and Process.wait*/IO.select is already possible to override and that would have the same effect. We'd also have to expose the optional read/write buffering + encoding conversion and make that accessible to pure Ruby. It would make C Ruby feel closer to Rubinius and that would be nice :) I'm not sure how feasable it would be; to introduce more Ruby-visible APIs to implement this. And I think exposing more APIs to handle FDs directly is a mistake in the presence of native threads. My proposed C API prefers "int *fd" and "rb_io_t" to deal with close notification handling. Multithreaded programs recycle FDs frequently and internal APIs need to be prepared to deal with that. The implementation I proposed also takes advantage of some C-only optimizations such as reading/writing to memory across Fiber stack boundaries: something which cannot be done with higher-level APIs . Similar optimizations already landed for thread_sync.c (Mutex/Queue) as well as IO#close in trunk. Again, designing user-visible APIs is most difficult and ruby-core have to think most about long-term support and consequences. So the difficulty of changing/adding APIs is: 1) internal C API (easiest) 2) public C API (difficult) 3) Ruby API (most difficult) So, I've mainly done 1) and made minimal additions to 3). Only changes to 2) are to internal behavior, so use from C extensions remains the same. > Then, we could allow things like `async` and `auto-fibers` to > extend Ruby's IO system to provide a policy for blocking IO. > `auto-fibers` could be implemented as a gem with a C > extension. > > What do you think? I guess this is meant for matz and ko1. We could actually have that today; and I guess you already have that with `async`. All the IO methods are well-documented and you can even ignore/override the existing IO buffering if you override all the methods by monkey patching core classes. Heck, you may even go as far as to never allocate rb_io_t if you override IO.open/IO.pipe/*Socket.new/... and replace them with your own class. What I think is (or at least ought to be) irrelevant. I only give matz and ko1 another option to choose from. We can wait for matz and ko1 to decide what to do, maybe they'll discuss this at: https://bugs.ruby-lang.org/projects/ruby/wiki/DevelopersMeeting20170616Japan I certainly won't attend meetings or try to influence anybody using anything besides plain-text messages, here.