[ruby-core:99044] [Ruby master Bug#17007] SystemStackError when using super inside Module included and lexically inside refinement

[ruby-core:99044] [Ruby master Bug#17007] SystemStackError when using super inside Module included and lexically inside refinement

[eregontp]

Issue #17007 has been reported by Eregon (Benoit Daloze).

----------------------------------------
Bug #17007: SystemStackError when using super inside Module included and lexically inside refinement
https://bugs.ruby-lang.org/issues/17007

* Author: Eregon (Benoit Daloze)
* Status: Open
* Priority: Normal
* Assignee: shugo (Shugo Maeda)
* ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN
----------------------------------------
```ruby
class C
  def foo
    ["C"]
  end
end

refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include Module.new {
      def foo
        ["M"] + super
      end
    }
  end
end

using refinement
p C.new.foo
```

gives

```
$ ruby bug_refine_super.rb
Traceback (most recent call last):
	10920: from bug_refine_super.rb:22:in `<main>'
	10919: from bug_refine_super.rb:10:in `foo'
	10918: from bug_refine_super.rb:15:in `foo'
	10917: from bug_refine_super.rb:10:in `foo'
	10916: from bug_refine_super.rb:15:in `foo'
	10915: from bug_refine_super.rb:10:in `foo'
	10914: from bug_refine_super.rb:15:in `foo'
	10913: from bug_refine_super.rb:10:in `foo'
	 ... 10908 levels...
	    4: from bug_refine_super.rb:15:in `foo'
	    3: from bug_refine_super.rb:10:in `foo'
	    2: from bug_refine_super.rb:15:in `foo'
	    1: from bug_refine_super.rb:10:in `foo'
bug_refine_super.rb:15:in `foo': stack level too deep (SystemStackError)
```

OTOH defining the module lexically outside of `refine` works:
```ruby
m = Module.new {
  def foo
    ["M"] + super
  end
}
refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include m
  end
end

# result: ["R", "M", "C"]
```



-- 
https://bugs.ruby-lang.org/

[ruby-core:99119] [Ruby master Bug#17007] SystemStackError when using super inside Module included and lexically inside refinement

[merch-redmine]

Issue #17007 has been updated by jeremyevans0 (Jeremy Evans).


After more analysis, I've determined that this issue is due to CREF handling in method lookup in `search_refined_method`.  It fails in the lexical case because the module is affected by the refinement (refinement blocks have an implicit `using` of the refinement themselves, I guess).

You can reproduce the SystemStackError outside of the lexical case by having the refinement activated at the point of method definition:

```ruby
class C
  def foo
    ["C"]
  end
end

module M; end

refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include M
  end
end

using refinement
M.define_method(:foo){["M"] + super()}
p C.new.foo
```

You can fix the issue by skipping any refined method during super lookup if the current method also uses the same refinement.  I implemented this in a pull request: https://github.com/ruby/ruby/pull/3309.

----------------------------------------
Bug #17007: SystemStackError when using super inside Module included and lexically inside refinement
https://bugs.ruby-lang.org/issues/17007#change-86494

* Author: Eregon (Benoit Daloze)
* Status: Open
* Priority: Normal
* Assignee: shugo (Shugo Maeda)
* ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN
----------------------------------------
```ruby
class C
  def foo
    ["C"]
  end
end

refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include Module.new {
      def foo
        ["M"] + super
      end
    }
  end
end

using refinement
p C.new.foo
```

gives

```
$ ruby bug_refine_super.rb
Traceback (most recent call last):
	10920: from bug_refine_super.rb:22:in `<main>'
	10919: from bug_refine_super.rb:10:in `foo'
	10918: from bug_refine_super.rb:15:in `foo'
	10917: from bug_refine_super.rb:10:in `foo'
	10916: from bug_refine_super.rb:15:in `foo'
	10915: from bug_refine_super.rb:10:in `foo'
	10914: from bug_refine_super.rb:15:in `foo'
	10913: from bug_refine_super.rb:10:in `foo'
	 ... 10908 levels...
	    4: from bug_refine_super.rb:15:in `foo'
	    3: from bug_refine_super.rb:10:in `foo'
	    2: from bug_refine_super.rb:15:in `foo'
	    1: from bug_refine_super.rb:10:in `foo'
bug_refine_super.rb:15:in `foo': stack level too deep (SystemStackError)
```

OTOH defining the module lexically outside of `refine` works:
```ruby
m = Module.new {
  def foo
    ["M"] + super
  end
}
refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include m
  end
end

# result: ["R", "M", "C"]
```



-- 
https://bugs.ruby-lang.org/

[ruby-core:99359] [Ruby master Bug#17007] SystemStackError when using super inside Module included and lexically inside refinement

[eregontp]

Issue #17007 has been updated by Eregon (Benoit Daloze).


Thanks for the fix!
BTW this changes the behavior on a new spec, is that intended? (result is [:A, :C] instead of [:A, :LOCAL, :C] on < 2.8)
https://github.com/ruby/spec/commit/b0da11b52560860e844470d145acee0ff4d4acea?w=1

----------------------------------------
Bug #17007: SystemStackError when using super inside Module included and lexically inside refinement
https://bugs.ruby-lang.org/issues/17007#change-86759

* Author: Eregon (Benoit Daloze)
* Status: Closed
* Priority: Normal
* Assignee: shugo (Shugo Maeda)
* ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN
----------------------------------------
```ruby
class C
  def foo
    ["C"]
  end
end

refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include Module.new {
      def foo
        ["M"] + super
      end
    }
  end
end

using refinement
p C.new.foo
```

gives

```
$ ruby bug_refine_super.rb
Traceback (most recent call last):
	10920: from bug_refine_super.rb:22:in `<main>'
	10919: from bug_refine_super.rb:10:in `foo'
	10918: from bug_refine_super.rb:15:in `foo'
	10917: from bug_refine_super.rb:10:in `foo'
	10916: from bug_refine_super.rb:15:in `foo'
	10915: from bug_refine_super.rb:10:in `foo'
	10914: from bug_refine_super.rb:15:in `foo'
	10913: from bug_refine_super.rb:10:in `foo'
	 ... 10908 levels...
	    4: from bug_refine_super.rb:15:in `foo'
	    3: from bug_refine_super.rb:10:in `foo'
	    2: from bug_refine_super.rb:15:in `foo'
	    1: from bug_refine_super.rb:10:in `foo'
bug_refine_super.rb:15:in `foo': stack level too deep (SystemStackError)
```

OTOH defining the module lexically outside of `refine` works:
```ruby
m = Module.new {
  def foo
    ["M"] + super
  end
}
refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include m
  end
end

# result: ["R", "M", "C"]
```



-- 
https://bugs.ruby-lang.org/

[ruby-core:99362] [Ruby master Bug#17007] SystemStackError when using super inside Module included and lexically inside refinement

[merch-redmine]

Issue #17007 has been updated by jeremyevans0 (Jeremy Evans).


Eregon (Benoit Daloze) wrote in #note-4:
> Thanks for the fix!
> BTW this changes the behavior on a new spec, is that intended? (result is [:A, :C] instead of [:A, :LOCAL, :C] on < 2.8)
> https://github.com/ruby/spec/commit/b0da11b52560860e844470d145acee0ff4d4acea?w=1

It's a consequence of the fix (skips the currently activated refinement during super), but I wouldn't say it is intended.  Ideally, the fix would only affect looping cases.  Unfortunately, I'm not sure if there is a better way to detect whether looping during super would occur.  I'm also not sure whether the approach I committed can detect all cases of looping (there may be other ways to introduce looping during super).

Maybe the refinements spec needs to be changed if we want to forbid looping, spelling out how to handle super in modules included in refinements where the refinements are activated at the point of super call.

----------------------------------------
Bug #17007: SystemStackError when using super inside Module included and lexically inside refinement
https://bugs.ruby-lang.org/issues/17007#change-86762

* Author: Eregon (Benoit Daloze)
* Status: Closed
* Priority: Normal
* Assignee: shugo (Shugo Maeda)
* ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN
----------------------------------------
```ruby
class C
  def foo
    ["C"]
  end
end

refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include Module.new {
      def foo
        ["M"] + super
      end
    }
  end
end

using refinement
p C.new.foo
```

gives

```
$ ruby bug_refine_super.rb
Traceback (most recent call last):
	10920: from bug_refine_super.rb:22:in `<main>'
	10919: from bug_refine_super.rb:10:in `foo'
	10918: from bug_refine_super.rb:15:in `foo'
	10917: from bug_refine_super.rb:10:in `foo'
	10916: from bug_refine_super.rb:15:in `foo'
	10915: from bug_refine_super.rb:10:in `foo'
	10914: from bug_refine_super.rb:15:in `foo'
	10913: from bug_refine_super.rb:10:in `foo'
	 ... 10908 levels...
	    4: from bug_refine_super.rb:15:in `foo'
	    3: from bug_refine_super.rb:10:in `foo'
	    2: from bug_refine_super.rb:15:in `foo'
	    1: from bug_refine_super.rb:10:in `foo'
bug_refine_super.rb:15:in `foo': stack level too deep (SystemStackError)
```

OTOH defining the module lexically outside of `refine` works:
```ruby
m = Module.new {
  def foo
    ["M"] + super
  end
}
refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include m
  end
end

# result: ["R", "M", "C"]
```



-- 
https://bugs.ruby-lang.org/

[ruby-core:100075] [Ruby master Bug#17007] SystemStackError when using super inside Module included and lexically inside refinement

[merch-redmine]

Issue #17007 has been updated by jeremyevans0 (Jeremy Evans).

Status changed from Closed to Assigned

To restore compatibility with Ruby 2.7 and fix #17182, I've reverted commit:eeef16e190cdabc2ba474622720f8e3df7bac43b at commit:179384a66862d5ef7413b6f4850b97d0becf4ec9.

@shugo, I would appreciate your input here on how refinements should be handled in this case.

----------------------------------------
Bug #17007: SystemStackError when using super inside Module included and lexically inside refinement
https://bugs.ruby-lang.org/issues/17007#change-87628

* Author: Eregon (Benoit Daloze)
* Status: Assigned
* Priority: Normal
* Assignee: shugo (Shugo Maeda)
* ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN
----------------------------------------
```ruby
class C
  def foo
    ["C"]
  end
end

refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include Module.new {
      def foo
        ["M"] + super
      end
    }
  end
end

using refinement
p C.new.foo
```

gives

```
$ ruby bug_refine_super.rb
Traceback (most recent call last):
	10920: from bug_refine_super.rb:22:in `<main>'
	10919: from bug_refine_super.rb:10:in `foo'
	10918: from bug_refine_super.rb:15:in `foo'
	10917: from bug_refine_super.rb:10:in `foo'
	10916: from bug_refine_super.rb:15:in `foo'
	10915: from bug_refine_super.rb:10:in `foo'
	10914: from bug_refine_super.rb:15:in `foo'
	10913: from bug_refine_super.rb:10:in `foo'
	 ... 10908 levels...
	    4: from bug_refine_super.rb:15:in `foo'
	    3: from bug_refine_super.rb:10:in `foo'
	    2: from bug_refine_super.rb:15:in `foo'
	    1: from bug_refine_super.rb:10:in `foo'
bug_refine_super.rb:15:in `foo': stack level too deep (SystemStackError)
```

OTOH defining the module lexically outside of `refine` works:
```ruby
m = Module.new {
  def foo
    ["M"] + super
  end
}
refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include m
  end
end

# result: ["R", "M", "C"]
```



-- 
https://bugs.ruby-lang.org/

[ruby-core:101637] [Ruby master Bug#17007] SystemStackError when using super inside Module included and lexically inside refinement

[shugo]

Issue #17007 has been updated by shugo (Shugo Maeda).


jeremyevans0 (Jeremy Evans) wrote in #note-6:
> To restore compatibility with Ruby 2.7 and fix #17182, I've reverted commit:eeef16e190cdabc2ba474622720f8e3df7bac43b at commit:179384a66862d5ef7413b6f4850b97d0becf4ec9.
> 
> @shugo, I would appreciate your input here on how refinements should be handled in this case.

I think the change is acceptable, but it may be better to prohibit include/prepend in refinement modules in future versions to avoid such implementation difficulties.


----------------------------------------
Bug #17007: SystemStackError when using super inside Module included and lexically inside refinement
https://bugs.ruby-lang.org/issues/17007#change-89428

* Author: Eregon (Benoit Daloze)
* Status: Assigned
* Priority: Normal
* Assignee: shugo (Shugo Maeda)
* ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN
----------------------------------------
```ruby
class C
  def foo
    ["C"]
  end
end

refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include Module.new {
      def foo
        ["M"] + super
      end
    }
  end
end

using refinement
p C.new.foo
```

gives

```
$ ruby bug_refine_super.rb
Traceback (most recent call last):
	10920: from bug_refine_super.rb:22:in `<main>'
	10919: from bug_refine_super.rb:10:in `foo'
	10918: from bug_refine_super.rb:15:in `foo'
	10917: from bug_refine_super.rb:10:in `foo'
	10916: from bug_refine_super.rb:15:in `foo'
	10915: from bug_refine_super.rb:10:in `foo'
	10914: from bug_refine_super.rb:15:in `foo'
	10913: from bug_refine_super.rb:10:in `foo'
	 ... 10908 levels...
	    4: from bug_refine_super.rb:15:in `foo'
	    3: from bug_refine_super.rb:10:in `foo'
	    2: from bug_refine_super.rb:15:in `foo'
	    1: from bug_refine_super.rb:10:in `foo'
bug_refine_super.rb:15:in `foo': stack level too deep (SystemStackError)
```

OTOH defining the module lexically outside of `refine` works:
```ruby
m = Module.new {
  def foo
    ["M"] + super
  end
}
refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include m
  end
end

# result: ["R", "M", "C"]
```



-- 
https://bugs.ruby-lang.org/

[ruby-core:105740] [Ruby master Bug#17007] SystemStackError when using super inside Module included and lexically inside refinement

[jeremyevans0 (Jeremy Evans)]

Issue #17007 has been updated by jeremyevans0 (Jeremy Evans).

Status changed from Assigned to Closed

Refinement#include is now deprecated and will be removed in Ruby 3.2.

----------------------------------------
Bug #17007: SystemStackError when using super inside Module included and lexically inside refinement
https://bugs.ruby-lang.org/issues/17007#change-94239

* Author: Eregon (Benoit Daloze)
* Status: Closed
* Priority: Normal
* Assignee: shugo (Shugo Maeda)
* ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN
----------------------------------------
```ruby
class C
  def foo
    ["C"]
  end
end

refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include Module.new {
      def foo
        ["M"] + super
      end
    }
  end
end

using refinement
p C.new.foo
```

gives

```
$ ruby bug_refine_super.rb
Traceback (most recent call last):
	10920: from bug_refine_super.rb:22:in `<main>'
	10919: from bug_refine_super.rb:10:in `foo'
	10918: from bug_refine_super.rb:15:in `foo'
	10917: from bug_refine_super.rb:10:in `foo'
	10916: from bug_refine_super.rb:15:in `foo'
	10915: from bug_refine_super.rb:10:in `foo'
	10914: from bug_refine_super.rb:15:in `foo'
	10913: from bug_refine_super.rb:10:in `foo'
	 ... 10908 levels...
	    4: from bug_refine_super.rb:15:in `foo'
	    3: from bug_refine_super.rb:10:in `foo'
	    2: from bug_refine_super.rb:15:in `foo'
	    1: from bug_refine_super.rb:10:in `foo'
bug_refine_super.rb:15:in `foo': stack level too deep (SystemStackError)
```

OTOH defining the module lexically outside of `refine` works:
```ruby
m = Module.new {
  def foo
    ["M"] + super
  end
}
refinement = Module.new do
  R = refine C do
    def foo
      ["R"] + super
    end

    include m
  end
end

# result: ["R", "M", "C"]
```



-- 
https://bugs.ruby-lang.org/