From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-3.9 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 3C9FC1F6AC for ; Thu, 5 Jul 2018 07:48:27 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 5AB86120BE2; Thu, 5 Jul 2018 16:48:24 +0900 (JST) Received: from o1678916x28.outbound-mail.sendgrid.net (o1678916x28.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id E2BDE120AA5 for ; Thu, 5 Jul 2018 16:48:22 +0900 (JST) Received: by filter0044p3mdw1.sendgrid.net with SMTP id filter0044p3mdw1-28713-5B3DCD43-2F 2018-07-05 07:48:19.950019317 +0000 UTC Received: from herokuapp.com (ec2-54-242-189-103.compute-1.amazonaws.com [54.242.189.103]) by ismtpd0004p1iad1.sendgrid.net (SG) with ESMTP id iVulqRGqRQ-LJIngbB3WFQ Thu, 05 Jul 2018 07:48:19.922 +0000 (UTC) Date: Thu, 05 Jul 2018 07:48:20 +0000 (UTC) From: samuel@oriontransfer.net To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 63194 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 14736 X-Redmine-Issue-Author: ioquatix X-Redmine-Sender: ioquatix X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS7NfwWi7V1HHYDIF0o7KRjYBNSAJTjmaRCycE sS1GqVlLmqYuWSCa/li/QFWjXnyS6VhS3v1RTzCLhTyvlSdm6yJTj/8oTGL4t4R/dRXx6GMxUAByr7 mw3Z1O4aOBwZ7we9q4/gIlcw+SmUaB2yspewpbL3zucSBZlFiacXryw3Ew== X-ML-Name: ruby-core X-Mail-Count: 87807 Subject: [ruby-core:87807] [Ruby trunk Feature#14736] Thread selector for flexible cooperative fiber based concurrency X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #14736 has been updated by ioquatix (Samuel Williams). For the first case, you naturally can't call Fiber.yield in that context... but this works: ``` #!/usr/bin/env ruby require 'fiber' fiber = Fiber.new do def aga; yield 1; Fiber.yield 4; yield 8; end puts to_enum(:aga).to_a end value = fiber.resume puts "Got #{value}" while fiber.alive? fiber.resume end # Prints: # Got 4 # 1 # 8 ``` This also works: ``` #!/usr/bin/env ruby require 'fiber' fiber = Fiber.new do def aga; yield 1; Fiber.yield 4; yield 8; end e = to_enum(:aga) puts e.next puts e.next puts e.next end value = fiber.resume puts "Got #{value.inspect}" while fiber.alive? fiber.resume end # 1 # 4 # 8 # Got nil ``` The semantics of `Fiber.yield` from within the enumerator block depend on how it's being used. That's not something I was aware of. I see that it fails if you try to do this: ``` #!/usr/bin/env ruby $LOAD_PATH << File.expand_path("../../lib", __dir__) require 'async' require 'async/io/stream' require 'async/io/host_endpoint' require 'async/io/protocol/line' class Lines < Async::IO::Protocol::Line def each return to_enum unless block_given? while line = read_line yield line end end end input = Lines.new( Async::IO::Stream.new( Async::IO::Generic.new($stdin) ) ) Async.run do # This works: # input.each do |line| # puts "... #{line}" # end # This doesn't: enumerator = input.each while line = enumerator.next puts "... #{line}" end end ``` I can imagine, if you don't know the underlying task is asynchronous, that this might cause some frustration, fortunately the error is reasonably clear in this instance. The problem is the nested fiber is not created in an `Async::Task`. I believe that with this PR in place, it would be possible to avoid this, because ALL Fiber created on the thread with a selector is asynchronous, so it shouldn't be a problem. I am interested in working further on this PR, and this is an interesting test case. Perhaps I will see if it can work or not. ---------------------------------------- Feature #14736: Thread selector for flexible cooperative fiber based concurrency https://bugs.ruby-lang.org/issues/14736#change-72824 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- Ruby concurrency can be greatly enhanced by concurrent, deterministic IO. Fibers have been shown many times to be a great abstraction for this purpose. The retain normal code flow and don't require any kind of Thread synchronisation. They are enjoyable to write code with because you don't have to concern yourself with thread synchronisation or other complicated issues. The basic idea is that if some operation would block, it yields the Fiber, and other Fibers within the thread can continue to execute. There are a number of ways to implement this. Here is a proof of concept to amend the existing `rb_io_wait_readable`/`rb_io_wait_writable`. https://github.com/ruby/ruby/pull/1870 This design minimally affects the Ruby implementation and allows flexibility for selector implementation. With a small amount of work, we can support EventMachine (65 million downloads), NIO4r (21 million downloads). It would be trivial to back port. This PR isn't complete but I am seeking feedback. If it's a good idea, I will do my best to see it through to completion, including support for EventMachine and NIO4r. ---Files-------------------------------- port_scanner_threadlet.rb (925 Bytes) -- https://bugs.ruby-lang.org/