From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-4.1 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 7FB661F404 for ; Tue, 14 Aug 2018 17:47:13 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 5027B12091D; Wed, 15 Aug 2018 02:47:11 +0900 (JST) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by neon.ruby-lang.org (Postfix) with ESMTPS id C9B4D120912 for ; Wed, 15 Aug 2018 02:47:04 +0900 (JST) Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id C99D71F404; Tue, 14 Aug 2018 17:47:02 +0000 (UTC) Date: Tue, 14 Aug 2018 17:47:02 +0000 From: Eric Wong To: ruby-core@ruby-lang.org Message-ID: <20180814174702.GA32360@dcvr> References: <20180814004235.hj3rokgsxoy77fza@dcvr> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-ML-Name: ruby-core X-Mail-Count: 88484 Subject: [ruby-core:88484] Re: [Ruby trunk Feature#13618] [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Koichi Sasada wrote: > On 2018/08/14 9:42, Eric Wong wrote: > > I am thinking of adding preemption support to this feature for > > compatibility with 1.8 > > So that "auto-fiber" proposal is to provide green threads like Ruby 1.8? Yes; this was always the idea. > Like: > ``` > model 1: Userlevel Thread > Same as traditional ruby thread. > ``` > in thread.c comment (I wrote 13 years ago!). > > I don't against this idea, but I think it is hard to select these options by > Ruby programmers. I think changing Thread implementation model from native > thread (1:1 model) to green thread mode (1:N model) is better for Ruby > programmers. No, I want to keep current Thread 1:1 model because it is useful for filesystem operations and some CPU/memory intensive tasks (zlib). Changing "Thread" now to 1:N would also break code written for 1.9..2.5 1:N model is good for most network operations (high latency, low throughput). > To change them, we need to discuss pros. and cons. of them carefully. There > are several good points (the biggest advantage of 1:1 model is friendly for > outer libraries) but are bad points (1:1 model has performance penalties, > and recent glibc malloc arena issues and so on). Agreed. > I don't think it is a good idea to choose such internal implementation by > Ruby programmers. ... easy? I think it is necessary to give that control to programmers. We can educate them on pros and cons of each and to use them in combination. ``` 1:1 + C-ext parallelism for SMP systems (example: zlib) + filesystem parallelism (no non-blocking I/O on FS ops) + external library compatibility + compatibility with 1.9..2.5 code - high memory use (malloc arenas, kernel structs) - kernel resource limitations (Process::RLIMIT_NPROC) 1:N + C10K problem (or C100K :P) + compatibility with old 1.8 designs + low memory and resource use (malloc and kernel never sees extra data structures) - cannot take advantage of SMP or multiple filesystems alone ``` The key is a programmer may combine 1:1 and 1:N for different parts of the program flow. So where the program is bottlenecked on filesystem, it can rely on 1:1 behavior, but when the program is waiting on slow network traffic, it can rely on 1:N behavior For example: I have 4 filesystems, I might have 32 native threads (8 threads/FS to keep kernel IO scheduler busy). But I will still serve 100K network clients with 100K 1:N threads and that can even use 1 native thread.