* [ruby-core:91129] [Ruby trunk Bug#15543] rb_str_set_len should clear code range
[not found] <redmine.issue-15543.20190116163539@ruby-lang.org>
@ 2019-01-16 16:35 ` ruby
2019-01-16 23:27 ` [ruby-core:91133] " nobu
` (3 subsequent siblings)
4 siblings, 0 replies; 5+ messages in thread
From: ruby @ 2019-01-16 16:35 UTC (permalink / raw)
To: ruby-core
Issue #15543 has been reported by nirvdrum (Kevin Menard).
----------------------------------------
Bug #15543: rb_str_set_len should clear code range
https://bugs.ruby-lang.org/issues/15543
* Author: nirvdrum (Kevin Menard)
* Status: Open
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Calling `rb_str_set_len` on a `String` could alter the code range. I think this hasn't been much of an issue because of pure luck rather than anything that was deliberately designed. If called on a string that already has a `CR_UNKNOWN` code range, there's no problem because the correct values will lazily calculated.
An example invocation that could be problematic, using helper methods for writing C API specs from the Ruby Spec Suite, looks like:
```
@str = "abcdefghij"[0..-1]
@s.rb_str_set_len(@str, 1)
@str.should == "a"
@str.force_encoding(Encoding::UTF_8)
@str.valid_encoding?.should == true
@s.RSTRING_PTR_set(@str, 1, 'B'.ord)
@s.RSTRING_PTR_set(@str, 2, 0x80)
@s.rb_str_set_len(@str, 3)
p @str
@str.valid_encoding?.should == false # This line fails because the cached code range hasn't been updated.
```
The first call to `valid_encoding?` forces the code range to be calculated for the string and it's determined to be `CR_7BIT`. Then `rb_str_set_len` is called, simulating the splitting the bytes of a valid UTF-8 multi-byte character. Here, the code range is still cached as `CR_7BIT`, but the byte sequence is invalid for UTF-8.
I think the fix is a simple matter of clearing the code range in `rb_str_set_len`, but I'd appreciate a review of my analysis.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
* [ruby-core:91133] [Ruby trunk Bug#15543] rb_str_set_len should clear code range
[not found] <redmine.issue-15543.20190116163539@ruby-lang.org>
2019-01-16 16:35 ` [ruby-core:91129] [Ruby trunk Bug#15543] rb_str_set_len should clear code range ruby
@ 2019-01-16 23:27 ` nobu
2019-01-17 0:03 ` [ruby-core:91134] " eregontp
` (2 subsequent siblings)
4 siblings, 0 replies; 5+ messages in thread
From: nobu @ 2019-01-16 23:27 UTC (permalink / raw)
To: ruby-core
Issue #15543 has been updated by nobu (Nobuyoshi Nakada).
Status changed from Open to Rejected
It isn't an issue of `rb_str_set_len`.
The string isn't valid encoding by `@s.RSTRING_PTR_set(@str, 2, 0x80)`.
You should call `rb_str_modify` before modifying a string object.
----------------------------------------
Bug #15543: rb_str_set_len should clear code range
https://bugs.ruby-lang.org/issues/15543#change-76364
* Author: nirvdrum (Kevin Menard)
* Status: Rejected
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Calling `rb_str_set_len` on a `String` could alter the code range. I think this hasn't been much of an issue because of pure luck rather than anything that was deliberately designed. If called on a string that already has a `CR_UNKNOWN` code range, there's no problem because the correct values will lazily calculated.
An example invocation that could be problematic, using helper methods for writing C API specs from the Ruby Spec Suite, looks like:
```
@str = "abcdefghij"[0..-1]
@s.rb_str_set_len(@str, 1)
@str.should == "a"
@str.force_encoding(Encoding::UTF_8)
@str.valid_encoding?.should == true
@s.RSTRING_PTR_set(@str, 1, 'B'.ord)
@s.RSTRING_PTR_set(@str, 2, 0x80)
@s.rb_str_set_len(@str, 3)
p @str
@str.valid_encoding?.should == false # This line fails because the cached code range hasn't been updated.
```
The first call to `valid_encoding?` forces the code range to be calculated for the string and it's determined to be `CR_7BIT`. Then `rb_str_set_len` is called, simulating the splitting the bytes of a valid UTF-8 multi-byte character. Here, the code range is still cached as `CR_7BIT`, but the byte sequence is invalid for UTF-8.
I think the fix is a simple matter of clearing the code range in `rb_str_set_len`, but I'd appreciate a review of my analysis.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
* [ruby-core:91134] [Ruby trunk Bug#15543] rb_str_set_len should clear code range
[not found] <redmine.issue-15543.20190116163539@ruby-lang.org>
2019-01-16 16:35 ` [ruby-core:91129] [Ruby trunk Bug#15543] rb_str_set_len should clear code range ruby
2019-01-16 23:27 ` [ruby-core:91133] " nobu
@ 2019-01-17 0:03 ` eregontp
2019-01-18 2:43 ` [ruby-core:91157] " nobu
2019-01-20 14:44 ` [ruby-core:91193] " eregontp
4 siblings, 0 replies; 5+ messages in thread
From: eregontp @ 2019-01-17 0:03 UTC (permalink / raw)
To: ruby-core
Issue #15543 has been updated by Eregon (Benoit Daloze).
I think few people know about this, and `rb_str_modify` isn't documented in doc/extension.rdoc.
I guess some C extensions ignore this requirement and could have bugs because of it.
Isn't it simpler and safer to clear the coderange, as it's done for, e.g., rb_str_resize() ?
----------------------------------------
Bug #15543: rb_str_set_len should clear code range
https://bugs.ruby-lang.org/issues/15543#change-76365
* Author: nirvdrum (Kevin Menard)
* Status: Rejected
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Calling `rb_str_set_len` on a `String` could alter the code range. I think this hasn't been much of an issue because of pure luck rather than anything that was deliberately designed. If called on a string that already has a `CR_UNKNOWN` code range, there's no problem because the correct values will lazily calculated.
An example invocation that could be problematic, using helper methods for writing C API specs from the Ruby Spec Suite, looks like:
```
@str = "abcdefghij"[0..-1]
@s.rb_str_set_len(@str, 1)
@str.should == "a"
@str.force_encoding(Encoding::UTF_8)
@str.valid_encoding?.should == true
@s.RSTRING_PTR_set(@str, 1, 'B'.ord)
@s.RSTRING_PTR_set(@str, 2, 0x80)
@s.rb_str_set_len(@str, 3)
p @str
@str.valid_encoding?.should == false # This line fails because the cached code range hasn't been updated.
```
The first call to `valid_encoding?` forces the code range to be calculated for the string and it's determined to be `CR_7BIT`. Then `rb_str_set_len` is called, simulating the splitting the bytes of a valid UTF-8 multi-byte character. Here, the code range is still cached as `CR_7BIT`, but the byte sequence is invalid for UTF-8.
I think the fix is a simple matter of clearing the code range in `rb_str_set_len`, but I'd appreciate a review of my analysis.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
* [ruby-core:91157] [Ruby trunk Bug#15543] rb_str_set_len should clear code range
[not found] <redmine.issue-15543.20190116163539@ruby-lang.org>
` (2 preceding siblings ...)
2019-01-17 0:03 ` [ruby-core:91134] " eregontp
@ 2019-01-18 2:43 ` nobu
2019-01-20 14:44 ` [ruby-core:91193] " eregontp
4 siblings, 0 replies; 5+ messages in thread
From: nobu @ 2019-01-18 2:43 UTC (permalink / raw)
To: ruby-core
Issue #15543 has been updated by nobu (Nobuyoshi Nakada).
`rb_str_set_len` is not the true problem.
Should `RSTRING_PTR` make the object unshared and clear the code range?
Or enclose it only for the core and prohibit in extension libraries?
Hmmm, the last choice may be the best, I think.
----------------------------------------
Bug #15543: rb_str_set_len should clear code range
https://bugs.ruby-lang.org/issues/15543#change-76388
* Author: nirvdrum (Kevin Menard)
* Status: Rejected
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Calling `rb_str_set_len` on a `String` could alter the code range. I think this hasn't been much of an issue because of pure luck rather than anything that was deliberately designed. If called on a string that already has a `CR_UNKNOWN` code range, there's no problem because the correct values will lazily calculated.
An example invocation that could be problematic, using helper methods for writing C API specs from the Ruby Spec Suite, looks like:
```
@str = "abcdefghij"[0..-1]
@s.rb_str_set_len(@str, 1)
@str.should == "a"
@str.force_encoding(Encoding::UTF_8)
@str.valid_encoding?.should == true
@s.RSTRING_PTR_set(@str, 1, 'B'.ord)
@s.RSTRING_PTR_set(@str, 2, 0x80)
@s.rb_str_set_len(@str, 3)
p @str
@str.valid_encoding?.should == false # This line fails because the cached code range hasn't been updated.
```
The first call to `valid_encoding?` forces the code range to be calculated for the string and it's determined to be `CR_7BIT`. Then `rb_str_set_len` is called, simulating the splitting the bytes of a valid UTF-8 multi-byte character. Here, the code range is still cached as `CR_7BIT`, but the byte sequence is invalid for UTF-8.
I think the fix is a simple matter of clearing the code range in `rb_str_set_len`, but I'd appreciate a review of my analysis.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
* [ruby-core:91193] [Ruby trunk Bug#15543] rb_str_set_len should clear code range
[not found] <redmine.issue-15543.20190116163539@ruby-lang.org>
` (3 preceding siblings ...)
2019-01-18 2:43 ` [ruby-core:91157] " nobu
@ 2019-01-20 14:44 ` eregontp
4 siblings, 0 replies; 5+ messages in thread
From: eregontp @ 2019-01-20 14:44 UTC (permalink / raw)
To: ruby-core
Issue #15543 has been updated by Eregon (Benoit Daloze).
nobu (Nobuyoshi Nakada) wrote:
> `rb_str_set_len` is not the true problem.
> Should `RSTRING_PTR` make the object unshared and clear the code range?
That sounds safer, because indeed as soon as the C code can access the `char*` it can change it.
> Or enclose it only for the core and prohibit in extension libraries?
I'm not sure what you mean?
Do you mean removing rb_str_set_len() or RSTRING_PTR from the public C-API?
Or do you mean in core, a special version of RSTRING_PTR would be used?
Or something else?
----------------------------------------
Bug #15543: rb_str_set_len should clear code range
https://bugs.ruby-lang.org/issues/15543#change-76425
* Author: nirvdrum (Kevin Menard)
* Status: Rejected
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Calling `rb_str_set_len` on a `String` could alter the code range. I think this hasn't been much of an issue because of pure luck rather than anything that was deliberately designed. If called on a string that already has a `CR_UNKNOWN` code range, there's no problem because the correct values will lazily calculated.
An example invocation that could be problematic, using helper methods for writing C API specs from the Ruby Spec Suite, looks like:
```
@str = "abcdefghij"[0..-1]
@s.rb_str_set_len(@str, 1)
@str.should == "a"
@str.force_encoding(Encoding::UTF_8)
@str.valid_encoding?.should == true
@s.RSTRING_PTR_set(@str, 1, 'B'.ord)
@s.RSTRING_PTR_set(@str, 2, 0x80)
@s.rb_str_set_len(@str, 3)
p @str
@str.valid_encoding?.should == false # This line fails because the cached code range hasn't been updated.
```
The first call to `valid_encoding?` forces the code range to be calculated for the string and it's determined to be `CR_7BIT`. Then `rb_str_set_len` is called, simulating the splitting the bytes of a valid UTF-8 multi-byte character. Here, the code range is still cached as `CR_7BIT`, but the byte sequence is invalid for UTF-8.
I think the fix is a simple matter of clearing the code range in `rb_str_set_len`, but I'd appreciate a review of my analysis.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-01-20 14:44 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <redmine.issue-15543.20190116163539@ruby-lang.org>
2019-01-16 16:35 ` [ruby-core:91129] [Ruby trunk Bug#15543] rb_str_set_len should clear code range ruby
2019-01-16 23:27 ` [ruby-core:91133] " nobu
2019-01-17 0:03 ` [ruby-core:91134] " eregontp
2019-01-18 2:43 ` [ruby-core:91157] " nobu
2019-01-20 14:44 ` [ruby-core:91193] " eregontp
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).