From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-2.9 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_MED,SPF_PASS,T_RP_MATCHES_RCVD shortcircuit=no autolearn=no autolearn_force=no version=3.4.0 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id C843C1F576 for ; Sat, 3 Feb 2018 09:34:05 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 806711209E4; Sat, 3 Feb 2018 18:34:03 +0900 (JST) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by neon.ruby-lang.org (Postfix) with ESMTPS id 4DE54120999 for ; Sat, 3 Feb 2018 18:33:59 +0900 (JST) Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id F12181F576; Sat, 3 Feb 2018 09:33:56 +0000 (UTC) Date: Sat, 3 Feb 2018 09:33:56 +0000 From: Eric Wong To: ruby-core@ruby-lang.org Message-ID: <20180203093356.GA4353@whir> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-ML-Name: ruby-core X-Mail-Count: 85362 Subject: [ruby-core:85362] Re: [Ruby trunk Feature#13618] [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" sam.saffron@gmail.com wrote: > Issue #13618 has been updated by sam.saffron (Sam Saffron). > > > I'm leaning towards Thread::Green, so existing users can do > > s/Thread.new/Thread::Green.new/ in many cases. > > Yes I think this works the problem though is that people will > expect this to work like green threads, meaning they also > should auto-yield regularly. You should be allowed to have > two green threads doing expensive computations. One tight loop > in a reactor now and you blow up everything (unlike normal threads) Good point. rb_thread_call_without_gvl could be used to migrate work to a thread pool (so we end up with M:N threads), and maybe that's not horrible as a default behavior. Data migration across native threads would hurt locality-wise for short-lived tasks (e.g. rb_stat), though... Fwiw, I was planning on adding a hinting mechanism to rb_thread_call_without_gvl anyways later on (for GC, maybe); but hints could be added to prevent/encourage migration based on the expected duration/bottleneck of the function. > This would mean you would have to pull in the 1.8 scheduler > or something. But then this stops being a proper reactor :(. > > I guess this is the underlying reason you just wanted to call this auto > yielding fibers instead of threads to start with. Right, the predictability of not having a timer switch threads automatically is appealing, sometimes. Having rb_thread_call_without_gvl become a scheduling point of some sort for green threads would be fine, however, since all callers already assume a context switch will happen. > A question for Matz and Koichi is if they expect the scheduler from 1.8 > to be brought back, if this is "safe" by default and "opt-in" for unsafe. > > > > I expect PG to be able to benefit from rb_wait_for_single_fd when > > using sockets. I know mysql2 uses rb_wait_for_single_fd, at least. > > I am not sure about this, libpq abstracts all of this stuff away from you > this is why Sean G wrote a complete binary protocol implementation in rust, > to gain control. pg gem does not use rb_wait_for_single_fd it just releases gvl. > > We have to make sure there is some sort of path forward with Postgres here > it is a huge issue. I don't know how expensive it is to parse the Pg protocol; but I remember in the 1.8 days pg was one of the few gems to use rb_thread_select and it played nicely with 1.8 green threads. Can't say I know Pg well these days, it's been over a decade since I used it with Ruby. > MiniRacer is CPU bound its basically packaging libv8 into > Ruby. I am on the fence here On one hand it would be nice to > auto yield so we feel reduced GVL pain and Ruby code can run > while v8 does it's thing. On the other hand the semantics of > one thread at 100% suddenly becomes 2 threads at 100% is not > ideal. Hard to decide. How long does it release the GVL for? The thread pool / workqueue idea I mentioned above might be a good fit for this if the communications overhead can masked by the length of the task. Nothing wrong with 2 threads at 100% if they're getting work done faster than 1 thread at 100%. I wouldn't want a native pool to be used for something like getaddrinfo, however, that's hugely inefficient (but exactly what getaddrinfo_a does internally in glibc).