[ruby-core:92650] [Ruby trunk Bug#15839] mixed encoding heredoc should be a syntax error regardless the order

[merch-redmine@jeremyevans.net]

Issue #15839 has been updated by jeremyevans0 (Jeremy Evans).

File mixed-encoding-heredoc-fix.patch added

Heredocs are parsed line-by-line, and mixed encoding is
already detected if it is on the same line:

```ruby
#encoding: cp932

p <<-STR
\xe9\x9d\u1234
STR
# UTF-8 mixed within Windows-31J source
# \xe9\x9d\u1234
# syntax error, unexpected end-of-input, expecting tSTRING_CONTENT or tSTRING_DBEG or tSTRING_DVAR or tSTRING_END
```

In order to handle mixed content on separate lines, we need to
keep track of the temporary encoding of the string, which was
previously done via a local variable in `tokadd_string`. The
attached patch adds a second `rb_encoding **` argument to
`tokadd_string` for keeping track of the temporary encoding,
so that `here_document` can store the value between lines.

----------------------------------------
Bug #15839: mixed encoding heredoc should be a syntax error regardless the order
https://bugs.ruby-lang.org/issues/15839#change-78012

* Author: nobu (Nobuyoshi Nakada)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: 
* Backport: 2.4: REQUIRED, 2.5: REQUIRED, 2.6: REQUIRED
----------------------------------------
This heredoc isn't a syntax error,
```ruby
#encoding: cp932

p <<-STR
\xe9\x9d
\u1234
STR
```

whereas this is.
```ruby
#encoding: cp932
"
\xe9\x9d
\u1234
"
```

---Files--------------------------------
mixed-encoding-heredoc-fix.patch (5.28 KB)


-- 
https://bugs.ruby-lang.org/