[ruby-core:100943] [Ruby master Feature#17276] Ripper stops tokenizing after keyword as a method parameter

[nov@yo.rim.or.jp]

Issue #17276 has been updated by no6v (Nobuhiro IMAI).


> Support raise_errors keyword for Ripper.{lex,tokenize,sexp,sexp_raw}
> 
> Implements [Feature #17276]

Thanks for your clarification and implementation.
(it seems that those two lines are same :)
https://github.com/ruby/ruby/blob/cd0877a93e91fecb3066984b3fa2a762e6977caf/test/ripper/test_lexer.rb#L150-L151

`Ripper::Lexer#{lex,tokenize}` seem to accept second or more calls to return the rest of code as tokens.

```
$ cat src.rb
def req(true) end
def opt(true=0) end
def rest(*true) end
def keyrest(**true) end
def block(&true) end
->true{}
->true=0{}
->*true{}
->**true{}
->&true{}
$ cat l.rb
require "ripper"
lexer = Ripper::Lexer.new(ARGF.read)
until (tokens = lexer.tokenize).empty?
  p tokens
end
$ ruby l.rb src.rb
["def", " ", "req", "(", "true", ")"]
[" ", "end", "\n", "def", " ", "opt", "(", "true", "=", "0", ")"]
[" ", "end", "\n", "def", " ", "rest", "(", "*", "true", ")"]
[" ", "end", "\n", "def", " ", "keyrest", "(", "**", "true", ")"]
[" ", "end", "\n", "def", " ", "block", "(", "&", "true", ")"]
[" ", "end", "\n", "->", "true", "{"]
["}", "\n", "->", "true", "=", "0", "{"]
["}", "\n", "->", "*", "true", "{"]
["}", "\n", "->", "**", "true", "{"]
["}", "\n", "->", "&", "true", "{"]
["}", "\n"]
```

`Ripper::Lexer#lex` is as well. Concatenated those tokens is what I exactly wanted.
I would prefer `Ripper.{lex,tokenize}` returning fully parsed tokens.

----------------------------------------
Feature #17276: Ripper stops tokenizing after keyword as a method parameter
https://bugs.ruby-lang.org/issues/17276#change-88598

* Author: no6v (Nobuhiro IMAI)
* Status: Closed
* Priority: Normal
----------------------------------------
Although these are obviously syntax errors at this time, the following
codes cannot be tokenized correctly by `Ripper.tokenize`.

```
$ cat src.rb
def req(true) end
def opt(true=0) end
def rest(*true) end
def keyrest(**true) end
def block(&true) end
->true{}
->true=0{}
->*true{}
->**true{}
->&true{}
$ ruby -rripper -vlne 'p Ripper.tokenize($_)' src.rb
ruby 3.0.0dev (2020-10-21T00:24:47Z master da25affdac) [x86_64-linux]
["def", " ", "req", "(", "true", ")"]
["def", " ", "opt", "(", "true", "=", "0", ")"]
["def", " ", "rest", "(", "*", "true", ")"]
["def", " ", "keyrest", "(", "**", "true", ")"]
["def", " ", "block", "(", "&", "true", ")"]
["->", "true", "{"]
["->", "true", "=", "0", "{"]
["->", "*", "true", "{"]
["->", "**", "true", "{"]
["->", "&", "true", "{"]
```

`end` and `}` are not shown in result.

This seems to prevent `irb` from determining the continuity of the input.
See: https://github.com/ruby/irb/issues/38




-- 
https://bugs.ruby-lang.org/