[ruby-core:81050] Re: [ruby-cvs:65407] normal:r58236 (trunk): thread.c: comments on M:N threading [ci skip]

[SASADA Koichi <ko1@atdot.net>]

On 2017/05/09 14:12, Eric Wong wrote:
> SASADA Koichi <ko1@atdot.net> wrote:
>> On 2017/05/09 12:38, Eric Wong wrote:
>>> 100 epoll FDs is a waste of FDs; especially since it is common
>>> to have a 1024 FD limit.  I already feel bad about timer thread
>>> taking up two FDs; but maybe epoll/kevent can cut reduce that.
>> 1024 soft limit and 4096 hard limit is an issue. However, if we employ
>>
>>> I can easily imagine Ruby doing 100 native threads in one process
>>> (8 cores, 10-20 rotational disks, 2 SSD), but 20000-30000 fibers.
>> 20000-30000 fibers, it is also problem if they have corresponding fds.
>> So that I think people increase this limit upto 65K, don't?
> Yes, for people that run 20000-30000 fibers maybe it is not a
> problem to have 100 epoll FD...
> 
> However, for existing apps like puma, webrick and net/http-based
> scripts: they can spawn dozens/hundreds of threads and only use
> one socket per thread.  It is a waste to use epoll/kqueue to
> watch a few number of FD per thread (ppoll is more appropriate
> for watching a single FD).

I see. 1000 fds -> 500 fds (with per-thread epoll) is bad.

> On the contrary; software like nginx and cmogstored watch
> thousands of FDs with a single epoll|kqueue FD.
> 
>>> In the kernel, every "struct eventpoll" + "struct file" in
>>> Linux is at least 400 bytes of unswappable kernel memory.
>> 400B * 100 = 40KB. Is it problem? I have no knowledge to evaluate this
>> size (10 pages seems not so small, I guess).
> I'd rather not use that much memory and save whereever possible.

On the other hand, aggressive I/O request can conflict by multi-thread
app. But current ruby threads don't run in parallel, so that it seems no
problem (hopefully). It seems can cause problem on parallel running
Guilds (but not available now).

-- 
// SASADA Koichi at atdot dot net