From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-3.1 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_MED,RP_MATCHES_RCVD,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id D0B2F2027C for ; Thu, 13 Jul 2017 22:31:15 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 8ABE51207F5; Fri, 14 Jul 2017 07:31:13 +0900 (JST) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by neon.ruby-lang.org (Postfix) with ESMTPS id C230E1207CC for ; Fri, 14 Jul 2017 07:31:10 +0900 (JST) Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 180642027C; Thu, 13 Jul 2017 22:31:09 +0000 (UTC) Date: Thu, 13 Jul 2017 22:31:08 +0000 From: Eric Wong To: ruby-core@ruby-lang.org Message-ID: <20170713223108.GA13429@untitled> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-ML-Name: ruby-core X-Mail-Count: 82040 Subject: [ruby-core:82040] Re: [Ruby trunk Feature#13618] [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" ko1@atdot.net wrote: > sorry for long absent about this topic. it is hard task (hard > to start writing up because of problem difficulties and my > English skil ;p ) to summarize about this topic. No problem, thank you for summarizing. > I try to write step by step. > > ---- > > # Discussion at last developers meeting > > ## Thread/Fiber switch safety > > Koichi: (repeat my opinion about difficulty of thread/fiber safety) > > akr: providing better synchronize mechanism (such as go-lang > has) and encouraging safe parallel computation seems better. > > Koichi: It is one possible solution but my position is "if > people can shoot their foot, people will shoot". I think your approach is too cautious. We already have many dangerous things in Ruby, even in single-threaded code. For example: File.read, IO#read, IO#gets are all dangerous with no size limit: they can cause out-of-memory or swapping on gigantic inputs, leading to DoS. Fork and inadvertant sharing of open files/sockets can also cause problems. And there are also pathological Regexp which can cause unbound CPU usage. > Matz: I don't like to force people to use lock and so on. > > (the point is Matz doesn't reject "-safe" approach) > > ## Introduce restriction > > (The following idea is not available at last meeting (only > part of idea I showed)) > > Koichi: > > The problem of this feature is mind gap using auto-fiber user > and script writer. This is same as thread-safety. Person A > consider the code is auto-fiber safe, and other person B (can > be same as A) write a code without auto-fiber safety, then it > will be problem. > > In general, most of existing libraries are not auto-fiber safe > code (it doesn't mean most of libraries are not auto-fiber > safe. Many code are auto-fiber safe without any care). Right; most code does not have to care; and all these dangers already exist with native Threads. > If we can know a code (and code called by this code) is > auto-fiber safe, we can use auto-fiber in safe. > > There are three type of code. > > * (1) don't care about auto-fiber > * (2) auto-fiber aware code (assume switching is not allowed at the beginning) > * (3) auto-fiber aware code (don't care it is allowed or not allowed to switch) > > There are three types of status. > > * (a) can't switch > * (b) can enable to switch, but don't switch > * (c) can switch > > in matrix > > ``` > can switch / can enable switch > (a) can't / can't > (b) can't / can > (c) can / ?? > ``` > > matrix with (1-3) and (a-c) > > ``` > (a) (b) (c) > (1) OK NG NG > (2) OK OK NG > (3) OK(*1) OK(*1) OK > ``` > > (1)-(b) and (1)-(c) is not accepted because other method called from this code can switch the context. > (2)-(c) is also unacceptable because the beginning of code is not auto-fiber aware. > > *1) Possible problem: (3) can introduce dead-lock problem because it can stop forever. Perhaps holding Mutex lock should disable auto-fiber switching. This should prevent deadlocks, I think. Existing code has Mutexes, so I'm not sure how they should interact with auto-Fiber. I agree with Matz that we should discourage locking, so I guess disabling auto-Fiber switch while Mutex is held is the most straightforward solution. > Normal threads start from (a). Auto-fibers start from (b). > They are written in (1), (2) and (3). Maybe (2) is written for > auto fiber top-lelvel. This code will call some async methods > which can change context. > > My proposal is, to write down explicitly of (1) to (3) and (a) > to (c) in program. > > At the meeting, I proposed non-matured keywords(-like) to control them. > (and just now I don't have good syntax for it yet) > > akira: If we introduce such keywords, we need to rewrite all > of code if we want to use auto-fiber web application request > handler (for example, we need to rewrite Rails to run on > auto-fiber based rack server). > > Matz: it is unacceptable to introduce huge rewriting for existing code. I agree completely with akira's observation and Matz's opinion of this. > (IMO (not appeared in last meeting) we need to rewrite all of > code even if we don't introduce keywords to make sure the > auto-fiber safety) I don't agree with this. A lot of code is already auto-fiber safe because they are written with GVL+Threads in mind. (see my original Net::HTTP example); and we also have a lot of code (webrick, net/*) which worked fine with green Threads in 1.8 Worst case is we release GVL in a native Thread and forget to yield to other Fibers in the same Thread. However, that is already a problem with existing code when run inside Fibers (e.g. getaddrinfo, IO operations on NFS/slow-disk, ...) I am working on making rb_thread_fd_select auto-fiber aware, too. (done for iom_select/iom_epoll, working on iom_kqueue) > Matz and I discussed about this issue, and we conclude that it > is too early to introduce this feature on Ruby 2.5. OK, I will continue to work on implementation improvements and keep patches rebased to trunk. > I want to consider this issue further. auto-fiber based guild > is one possibility, this mean we can introduce object > isolation and context switching each other. Do you think this is in the 2.5 timeline? Thank you.