[ruby-core:81514] Re: [Ruby trunk Feature#13618] [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid

[Eric Wong <normalperson@yhbt.net>]

ko1@atdot.net wrote:
> My opinion is opposite. I think "For human being using threading is too hard to use correctly" or "Rubyist shouldn't care about threading difficulties". I agree my opinion is extreme and many "advanced" programmers like Eric can write correct thread programs. But most (many? some? a few?) of ruby programmer (including me) can not write correct code I believe.

I do not believe I can write correct code of any type, actually.
Everything I write; even trivial single-threaded scripts has bugs.

On the other hand, my likelyhood of introducing bugs seems
nearly identical across any environment and programming models.
However, having less/simpler code (and less dependencies) seems
to result in fewer bugs, in my experience.

> (In addition: I heard some advanced programmers say "people can write". I doubt because it is something survivor bias)

Yes.

> (recent days I fixed rubygems' threading problem it is difficult to reproduce)
> 
> I often use this metaphor: It is like GC jtrategy. If people can manage object lifetime, it is faster than using GC (at some case. Some case GC is more faster than manual memory management). However we choose GC because we want to concentrate on writing application code.

Right.  However, it seems choosing "easier" strategies means
less focus on overall design, leading to more problems down the line.

Since around 2010; I believe unicorn caused major, irreparable
damage to Rack ecosystem by promoting single-threaded design and
having a SIGKILL timeout feature.  unicorn made Rubyists stop
caring to fix concurrency bugs and do proper timeouts.

Nowadays Rack apps are both too buggy AND use too much memory :<

I know some people disagree with my assessment of unicorn;
but I prefer to hate everything I've done: it's easier to
find improvements that way :)

> I agree auto-fibers is safer than threads. In my mind:
> 
> ```
> danger <-> safe (this is my opinion)
> 
>    parallel threads (JRuby, ...) > concurrent threads (MRI) >>
>    auto-fibers (full-auto)       > auto-fiber (restricted) >>
>    Guild                         > single thread

Agree.  So maybe we can design API for "auto-fiber (restricted)"?

> But auto-fiber can introduce accident and it should be not so frequent, and it is difficult to reproduce. This means it is difficult to debug.
> 
> Ruby has many pit falls to shoot our own legs (meta-programming features, open class and so on) but they are deterministic (at most of case).

Yes.  I think these (along too much code + dependencies) cause
more problems than concurrency bugs.

> normalperson (Eric Wong) wrote:
> >  Yes; we will document all switch points in RDoc and NEWS,
> >  of course (maybe write a separate doc/auto-fiber.rdoc)
> 
> My point is, if method "foo" is switching point, then any method can call "foo" (bar, and baz, the caller of bar, ...) should be noted. Maybe it is impossible to complete because of Ruby's dynamic nature.

Right.  Maybe that is a lot of documentation...

What if the API were the opposite of Thread.exclusive/Mutex#synchronize?
Perhaps:

    Fiber.new do
      Fiber.auto do
         # enable auto-fiber inside this block
      end
      # disable auto-fiber again
    end

Maybe Fiber.exclusive can disable Fiber.auto temporarily:

    Fiber.new do
      Fiber.auto do
        # enable auto-fiber

        Fiber.exclusive do
          # temporarily disable auto-fiber
        end
        # enable auto-fiber again
        ...
      end
    end

Fiber.auto/Fiber.exclusive would be no-ops unless inside
a Fiber.new block...

But maybe that is too much code and nesting levels;
so I still like Fiber.start more.

> >  I would like to use existing stdlib (net/*, webrick, drb, ...)
> >  as much as possible without modifications.  That means many
> >  existing Ruby libraries can work transparently.
> 
> I understand your point.

Thanks; that is my biggest wish for this feature.

Anyways, I will leave matz, you and others deal with final API
decisions.