From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-2.9 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_MED,SPF_PASS,T_RP_MATCHES_RCVD shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id A67EF1F404 for ; Sat, 27 Jan 2018 03:45:34 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id BE7EC120AA9; Sat, 27 Jan 2018 12:45:31 +0900 (JST) Received: from o1678916x28.outbound-mail.sendgrid.net (o1678916x28.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id 387E8120A83 for ; Sat, 27 Jan 2018 12:45:29 +0900 (JST) Received: by filter0012p3mdw1.sendgrid.net with SMTP id filter0012p3mdw1-3166-5A6BF5D6-3 2018-01-27 03:45:26.120414552 +0000 UTC Received: from herokuapp.com (ec2-54-80-24-225.compute-1.amazonaws.com [54.80.24.225]) by ismtpd0006p1iad1.sendgrid.net (SG) with ESMTP id ChnvqoSoRPmIfgqaGCmP3w Sat, 27 Jan 2018 03:45:26.057 +0000 (UTC) Date: Sat, 27 Jan 2018 03:45:26 +0000 (UTC) From: danieldasilvaferreira@gmail.com To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 60400 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 13618 X-Redmine-Issue-Author: normalperson X-Redmine-Issue-Assignee: normalperson X-Redmine-Sender: dsferreira X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS5/4Q4Zv2mERlucAAXJi1EeHh67uz22mIXrWj peWB++jpxO5EQlhaRAkD985dIa37aiOehcLnfxSUDEBKUiH9Z4sHXVfcEoVIbNKPWkkZPkfQeNajvD P9zB5gRv1jSitk8MAuzDdZoAcyJdNGwINufAZF2A27c1iQmys63Y6DLwlw== X-ML-Name: ruby-core X-Mail-Count: 85144 Subject: [ruby-core:85144] [Ruby trunk Feature#13618] [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #13618 has been updated by dsferreira (Daniel Ferreira). normalperson (Eric Wong) wrote: > I'm not sure what you're talking about. I suppose nothing is reliable Let me try to explain what I think about the async subject in ruby land using a different story: For me there is ruby core and there is ruby. I'm a ruby kind of guy like most of ruby developers. I like to use ruby and I like to use it as it is given to us by ruby core. I prefer to build my own tools in top of ruby core rather than using external libraries/gems. That with the assumption that ruby core will not break backwards compatibility. If there is something I really dislike is to fix broken code due to dependency issues. Ruby core is the rock solid foundation I rely upon for the developments I design and implement. I started in ruby land with rails like most of us. As years passed by I went more and more to other territories. So I believe my story it is a very common story: The ruby developer that starts at the very high level with rails. With time becomes progressively more and more familiar with the low level concepts of programming. Gets to understand the underlying concepts behind the frameworks and starts to grasp at last the ruby essence. And here I am now speaking with you guys. Ruby core. The lower level by excellence. It is fascinating to go through discussions like this one. The craft of ruby landscape for the future. Technically speaking I'm learning a lot but I'm not prepared yet to give my contribution at that level. The contribution I believe I can give is this view I'm speaking about. The daily user that sometimes struggle to find the right paths for the problems in hands. --- Ruby developers like my self (I imagine there will be more that feel this way) are very much impacted by the opinions of ruby core team members. Specially top team members like Koichi. We can call it the teacher - student dichotomy. When Koichi referring to threads functionality in ruby land writes and says: > "But most (many? some? a few?) of ruby programmer (including me) can not write correct code I believe." I do listen. People listen. (Koichi sentence here is just an handy reference example (sorry Koichi), from the many I have read throughout this many years and many of those comments are here, embedded in redmine issues). These sentences have a very big impact. I as a programmer aim to write and develop correct code. If there is an area that I do not feel comfortable with then I study it, play with it but that is it. I will not put my job and my company in jeopardy just to show some cool stuff to the team. Ruby programmer not ruby core hacker remember? How many ruby developers develop a http server or know the internals of at least one? (Just as an example of different levels of developer seniority.) Unicorn or passenger or thin or puma... are black boxes for the most of us. And yes there are bugs and our applications are impacted by them. That is the ecosystem and it is good like that. It will not change. Somehow there are people that feel happy playing in dangerous zones like threads and fibers (See previous Koichi reference. We know you have a "slightly" different opinion). Us, mere mortals, just would like to be able to do our daily work at least without compromising. Although I would like to use libraries to play with my actors without worrying to much I can't. Knowing that they are dangerous zones tells me I must worry still. So, why not use Akka and live happy ever after? In Akka land everyone happily uses actors. I never heard any reference telling people to be careful about a given issue. Maybe the issues exist but what you read is that Akka is the solution for all your problems in async world. I don't want to use Akka but I know that ruby is losing developers every day because of situations like this one I'm referring here. Ruby desperately needs to resolve once and for all this situation. The key word for me here is a clear message that could say with confidence: "Ruby is rock solid for async because..." If we don't succeed to pass this message to the world of programming ruby will slowly be replaced by other languages. Parallelism and concurrency and async will be everywhere in the future. I took the decision to express this thoughts in this conversation because I love ruby and I want to help ruby become better. In my opinion: We need to create the foundations for a post ruby 3 future in ruby land where async is the standard for the many and not the exception for the few. That is my vision. Many Thanks, Daniel ---------------------------------------- Feature #13618: [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid https://bugs.ruby-lang.org/issues/13618#change-69880 * Author: normalperson (Eric Wong) * Status: Assigned * Priority: Normal * Assignee: normalperson (Eric Wong) * Target version: ---------------------------------------- ``` auto fiber schedule for rb_wait_for_single_fd and rb_waitpid Implement automatic Fiber yield and resume when running rb_wait_for_single_fd and rb_waitpid. The Ruby API changes for Fiber are named after existing Thread methods. main Ruby API: Fiber#start -> enable auto-scheduling and run Fiber until it automatically yields (due to EAGAIN/EWOULDBLOCK) The following behave like their Thread counterparts: Fiber.start - Fiber.new + Fiber#start (prelude.rb) Fiber#join - run internal scheduler until Fiber is terminated Fiber#value - ditto Fiber#run - like Fiber#start (prelude.rb) Right now, it takes over rb_wait_for_single_fd() and rb_waitpid() function if the running Fiber is auto-enabled (cont.c::rb_fiber_auto_sched_p) Changes to existing functions are minimal. New files (all new structs and relations should be documented): iom.h - internal API for the rest of RubyVM (incomplete?) iom_internal.h - internal header for iom_(select|epoll|kqueue).h iom_epoll.h - epoll-specific pieces iom_kqueue.h - kqueue-specific pieces iom_select.h - select-specific pieces iom_pingable_common.h - common code for iom_(epoll|kqueue).h iom_common.h - common footer for iom_(select|epoll|kqueue).h Changes to existing data structures: rb_thread_t.afrunq - list of fibers to auto-resume rb_vm_t.iom - Ruby I/O Manager (rb_iom_t) :) Besides rb_iom_t, all the new structs are stack-only and relies extensively on ccan/list for branch-less, O(1) insert/delete. As usual, understanding the data structures first should help you understand the code. Right now, I reuse some static functions in thread.c, so thread.c includes iom_(select|epoll|kqueue).h TODO: Hijack other blocking functions (IO.select, ...) I am using "double" for timeout since it is more convenient for arithmetic like parts of thread.c. Most platforms have good FP, I think. Also, all "blocking" functions (rb_iom_wait*) will have timeout support. ./configure gains a new --with-iom=(select|epoll|kqueue) switch libkqueue: libkqueue support is incomplete; corner cases are not handled well: 1) multiple fibers waiting on the same FD 2) waiting for both read and write events on the same FD Bugfixes to libkqueue may be necessary to support all corner cases. Supporting these corner cases for native kqueue was challenging, even. See comments on iom_kqueue.h and iom_epoll.h for nuances. Limitations Test script I used to download a file from my server: ----8<--- require 'net/http' require 'uri' require 'digest/sha1' require 'fiber' url = 'http://80x24.org/git-i-forgot-to-pack/objects/pack/pack-97b25a76c03b489d4cbbd85b12d0e1ad28717e55.idx' uri = URI(url) use_ssl = "https" == uri.scheme fibs = 10.times.map do Fiber.start do cur = Fiber.current.object_id # XXX getaddrinfo() and connect() are blocking # XXX resolv/replace + connect_nonblock Net::HTTP.start(uri.host, uri.port, use_ssl: use_ssl) do |http| req = Net::HTTP::Get.new(uri) http.request(req) do |res| dig = Digest::SHA1.new res.read_body do |buf| dig.update(buf) #warn "#{cur} #{buf.bytesize}\n" end warn "#{cur} #{dig.hexdigest}\n" end end warn "done\n" :done end end warn "joining #{Time.now}\n" fibs[-1].join(4) warn "joined #{Time.now}\n" all = fibs.dup warn "1 joined, wait for the rest\n" until fibs.empty? fibs.each(&:join) fibs.keep_if(&:alive?) warn fibs.inspect end p all.map(&:value) Fiber.new do puts 'HI' end.run.join ``` ---Files-------------------------------- 0001-auto-fiber-schedule-for-rb_wait_for_single_fd-and-rb.patch (82.8 KB) -- https://bugs.ruby-lang.org/