From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: poffice@blade.nagaokaut.ac.jp Delivered-To: poffice@blade.nagaokaut.ac.jp Received: from kankan.nagaokaut.ac.jp (smtp.nagaokaut.ac.jp [133.44.2.24]) by blade.nagaokaut.ac.jp (Postfix) with ESMTP id BB7691BA000C for ; Tue, 9 May 2017 11:54:34 +0900 (JST) Received: from voscc.nagaokaut.ac.jp (voscc.nagaokaut.ac.jp [133.44.1.100]) by kankan.nagaokaut.ac.jp (Postfix) with ESMTP id 96CF0B5D89B for ; Tue, 9 May 2017 12:38:14 +0900 (JST) Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by voscc.nagaokaut.ac.jp (Postfix) with ESMTP id EE80A18CC81A for ; Tue, 9 May 2017 12:38:14 +0900 (JST) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 31B3012074C; Tue, 9 May 2017 12:38:13 +0900 (JST) X-Original-To: ruby-core@ruby-lang.org Delivered-To: ruby-core@ruby-lang.org Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by neon.ruby-lang.org (Postfix) with ESMTPS id 504DC1206B2 for ; Tue, 9 May 2017 12:38:09 +0900 (JST) Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 9EE3F1FC44; Tue, 9 May 2017 03:38:06 +0000 (UTC) Date: Tue, 9 May 2017 03:38:06 +0000 From: Eric Wong To: ruby-core@ruby-lang.org Message-ID: <20170509033806.GA27973@starla> References: <20170402023514.GB30476@dcvr> <76459664-9857-4244-7d43-79b24e737efc@atdot.net> <20170403044254.GA16328@starla> <20170508003315.GA3789@starla> <38090d10-c6a1-5097-66af-130275d773ea@atdot.net> <2b47c736-08d8-095b-0454-2dd0b1020b03@atdot.net> <20170508030120.GB24763@starla> <94ca8f9a-7001-12d3-323d-8c5751569c51@atdot.net> <20170508063633.GA6821@starla> <4a83bbeb-b22b-61ec-a03f-657746843431@atdot.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4a83bbeb-b22b-61ec-a03f-657746843431@atdot.net> X-ML-Name: ruby-core X-Mail-Count: 81044 Subject: [ruby-core:81044] Re: [ruby-cvs:65407] normal:r58236 (trunk): thread.c: comments on M:N threading [ci skip] X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" SASADA Koichi wrote: > On 2017/05/08 15:36, Eric Wong wrote: > > Maybe; if we can avoid GVL and introduce more parallelism. > > > > However, I think having one epoll/kqueue FD is better for a > > whole process; maybe one epoll/kqueue per-core (not per-thread) > > at maximum. > > > > I can easily imagine Ruby doing 100 native threads in one process > > (8 cores, 10-20 rotational disks, 2 SSD), but 20000-30000 fibers. > > could you elaborate more? 100 epoll threads are not effective? > Honestly, I have no experience to use epoll/kqueue. 100 epoll FDs is a waste of FDs; especially since it is common to have a 1024 FD limit. I already feel bad about timer thread taking up two FDs; but maybe epoll/kevent can cut reduce that. In the kernel, every "struct eventpoll" + "struct file" in Linux is at least 400 bytes of unswappable kernel memory. Anyways, I've contributed bugfixes to both epoll in the Linux kernel and also to libkqueue (userspace emulation lib); and use them both in several projects in and outside of Ruby. > # context switching to another topic > > > Side note: First, I would like to make fibers smaller. > > Right now rb_fiber_t stores all of the rb_thread_t > > struct, but not all fields get used. I started to work on > > splitting out to a new struct rb_thread_context_t earlier: > ... > > The end goal is to avoid storing all of rb_thread_t inside > > rb_context_t/rb_fiber_t; and only store rb_thread_context_t. > > That should reduce memory overhead and maybe make switching > > faster. > > This is what my goal of Ruby 2.5 (2017) I proposed to my company. If you > do it, it's great (and I achieved one of my job :)). > > My plan is almost similar but I want to introduce something like > `mrb_state` which passed to all mruby functions as first argument. > > Do you want to commit this patch before your final goal (lightweight > fiber switching)? > > > FYI: my plan. > > (1) Make separate execution context and make fiber switching lightweight > * before 2.5 > * You named `rb_thread_context_t`, but I don't want to name `thread` > so I planned to name it `execution_context` and so on (a bit longer) OK, I can rename my work-in-progress patch with s/rb_thread_context_t/rb_execution_context_t/ and commit later tonight. > (2-1: extend Fiber) Add Fiber scheduler like you are thinking. > (2-2: toward Guild) Add `execution_context` as first argument to > all C APIs > * to keep compatibility, we need to introduce new prefix `rbX_...` > for new APIs which receive first argument. > * On mruby, `mrb_state` is passed to all of APIs. We need to consider > the passed information carefully. OK, that sounds good. Also, can you take a look at implementing [Feature #13434] "better method definition in C API" for rbX_*? You are more knowledgeable in VM + method definition area; while I can work on the epoll/kqueue parts for I/O scheduling. (but I will need to schedule my own human time to work on Ruby :)