[ruby-core:100908] [Ruby master Feature#17276] Ripper stops tokenizing after keyword as a method parameter

[merch-redmine@jeremyevans.net]

Issue #17276 has been updated by jeremyevans0 (Jeremy Evans).


Eregon (Benoit Daloze) wrote in #note-2:
> jeremyevans0 (Jeremy Evans) wrote in #note-1:
> > Maybe we could support keyword arguments in `Ripper.lex` and `Ripper.tokenize` to raise SyntaxError for errors?  Here's a pull request for that approach: https://github.com/ruby/ruby/pull/3774
> 
> I agree it would be nice.
> 
> Do you think the same would be possible for `Ripper.sexp/sexp_raw`?

Yes, the same is possible with `Ripper.sexp/sexp_raw`.  I've updated the pull request to handle those as well.

----------------------------------------
Feature #17276: Ripper stops tokenizing after keyword as a method parameter
https://bugs.ruby-lang.org/issues/17276#change-88560

* Author: no6v (Nobuhiro IMAI)
* Status: Open
* Priority: Normal
----------------------------------------
Although these are obviously syntax errors at this time, the following
codes cannot be tokenized correctly by `Ripper.tokenize`.

```
$ cat src.rb
def req(true) end
def opt(true=0) end
def rest(*true) end
def keyrest(**true) end
def block(&true) end
->true{}
->true=0{}
->*true{}
->**true{}
->&true{}
$ ruby -rripper -vlne 'p Ripper.tokenize($_)' src.rb
ruby 3.0.0dev (2020-10-21T00:24:47Z master da25affdac) [x86_64-linux]
["def", " ", "req", "(", "true", ")"]
["def", " ", "opt", "(", "true", "=", "0", ")"]
["def", " ", "rest", "(", "*", "true", ")"]
["def", " ", "keyrest", "(", "**", "true", ")"]
["def", " ", "block", "(", "&", "true", ")"]
["->", "true", "{"]
["->", "true", "=", "0", "{"]
["->", "*", "true", "{"]
["->", "**", "true", "{"]
["->", "&", "true", "{"]
```

`end` and `}` are not shown in result.

This seems to prevent `irb` from determining the continuity of the input.
See: https://github.com/ruby/irb/issues/38




-- 
https://bugs.ruby-lang.org/