[ruby-core:110636] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[ruby-core:110636] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[chrisseaton (Chris Seaton)]

Issue #19108 has been reported by chrisseaton (Chris Seaton).

----------------------------------------
Bug #19108: Format routines like pack blindly treat a string as ASCII-encoded
https://bugs.ruby-lang.org/issues/19108

* Author: chrisseaton (Chris Seaton)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Format routines like pack and unpack blindly treat a string as ASCII-encoded, even if they aren't ASCII or ASCII-compatible.

I tried to construct code that was misleading using ASCII-incompatible-encodings but couldn't do it in practice (no ASCII-incompatible encodings have a pack directive ASCII byte that is encoded as a printable character.)

But I could demonstrate at least some strange behaviour:

```
p ['foo'].pack('u').encoding # => #<Encoding:US-ASCII>
p ['foo'].pack('u'.encode('UTF-32BE')).encoding # => #<Encoding:ASCII-8BIT>
```

This is because the NUL characters in the second one (which aren't really NUL characters - they're part of the directive characters) explicitly trigger the encoding to change to binary.

There is a warning, but the warning is only for unexpected directives. How about disallowing or warning for non-ascii compatible format strings?



-- 
https://bugs.ruby-lang.org/

[ruby-core:110639] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[chrisseaton (Chris Seaton)]

Issue #19108 has been updated by chrisseaton (Chris Seaton).


Possibly we should raise an exception if the string is not `ascii_only?`

----------------------------------------
Bug #19108: Format routines like pack blindly treat a string as ASCII-encoded
https://bugs.ruby-lang.org/issues/19108#change-99977

* Author: chrisseaton (Chris Seaton)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Format routines like pack and unpack blindly treat a string as ASCII-encoded, even if they aren't ASCII or ASCII-compatible.

I tried to construct code that was misleading using ASCII-incompatible-encodings but couldn't do it in practice (no ASCII-incompatible encodings have a pack directive ASCII byte that is encoded as a printable character.)

But I could demonstrate at least some strange behaviour:

```
p ['foo'].pack('u').encoding # => #<Encoding:US-ASCII>
p ['foo'].pack('u'.encode('UTF-32BE')).encoding # => #<Encoding:ASCII-8BIT>
```

This is because the NUL characters in the second one (which aren't really NUL characters - they're part of the directive characters) explicitly trigger the encoding to change to binary.

There is a warning, but the warning is only for unexpected directives. How about disallowing or warning for non-ascii compatible format strings?



-- 
https://bugs.ruby-lang.org/

[ruby-core:110640] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[byroot (Jean Boussier)]

Issue #19108 has been updated by byroot (Jean Boussier).


I agree that at the very least the `unknown pack directive` warning should be made non-verbose (displayed even with `$VERBOSE=false`, and would make sense as `ArgumentError`.



----------------------------------------
Bug #19108: Format routines like pack blindly treat a string as ASCII-encoded
https://bugs.ruby-lang.org/issues/19108#change-99978

* Author: chrisseaton (Chris Seaton)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Format routines like pack and unpack blindly treat a string as ASCII-encoded, even if they aren't ASCII or ASCII-compatible.

I tried to construct code that was misleading using ASCII-incompatible-encodings but couldn't do it in practice (no ASCII-incompatible encodings have a pack directive ASCII byte that is encoded as a printable character.)

But I could demonstrate at least some strange behaviour:

```
p ['foo'].pack('u').encoding # => #<Encoding:US-ASCII>
p ['foo'].pack('u'.encode('UTF-32BE')).encoding # => #<Encoding:ASCII-8BIT>
```

This is because the NUL characters in the second one (which aren't really NUL characters - they're part of the directive characters) explicitly trigger the encoding to change to binary.

There is a warning, but the warning is only for unexpected directives. How about disallowing or warning for non-ascii compatible format strings?



-- 
https://bugs.ruby-lang.org/

[ruby-core:110834] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[Eregon (Benoit Daloze)]

Issue #19108 has been updated by Eregon (Benoit Daloze).


Agreed, I think it should be ArgumentError since it's otherwise silently ignoring characters in the pack format string.
A non-verbose warning is better than the current state if ArgumentError is deemed too incompatible.

Here is real case where the silent warning caused confusion for `[1].pack('<L')`: https://github.com/oracle/truffleruby/issues/2791

----------------------------------------
Bug #19108: Format routines like pack blindly treat a string as ASCII-encoded
https://bugs.ruby-lang.org/issues/19108#change-100191

* Author: chrisseaton (Chris Seaton)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Format routines like pack and unpack blindly treat a string as ASCII-encoded, even if they aren't ASCII or ASCII-compatible.

I tried to construct code that was misleading using ASCII-incompatible-encodings but couldn't do it in practice (no ASCII-incompatible encodings have a pack directive ASCII byte that is encoded as a printable character.)

But I could demonstrate at least some strange behaviour:

```
p ['foo'].pack('u').encoding # => #<Encoding:US-ASCII>
p ['foo'].pack('u'.encode('UTF-32BE')).encoding # => #<Encoding:ASCII-8BIT>
```

This is because the NUL characters in the second one (which aren't really NUL characters - they're part of the directive characters) explicitly trigger the encoding to change to binary.

There is a warning, but the warning is only for unexpected directives. How about disallowing or warning for non-ascii compatible format strings?



-- 
https://bugs.ruby-lang.org/

[ruby-core:110850] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[nobu (Nobuyoshi Nakada)]

Issue #19108 has been updated by nobu (Nobuyoshi Nakada).


chrisseaton (Chris Seaton) wrote in #note-1:
> Possibly we should raise an exception if the string is not `ascii_only?`

I think you want to mean "if the string is not ASCII-compatible".

https://github.com/ruby/ruby/pull/6785

----------------------------------------
Bug #19108: Format routines like pack blindly treat a string as ASCII-encoded
https://bugs.ruby-lang.org/issues/19108#change-100208

* Author: chrisseaton (Chris Seaton)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Format routines like pack and unpack blindly treat a string as ASCII-encoded, even if they aren't ASCII or ASCII-compatible.

I tried to construct code that was misleading using ASCII-incompatible-encodings but couldn't do it in practice (no ASCII-incompatible encodings have a pack directive ASCII byte that is encoded as a printable character.)

But I could demonstrate at least some strange behaviour:

```
p ['foo'].pack('u').encoding # => #<Encoding:US-ASCII>
p ['foo'].pack('u'.encode('UTF-32BE')).encoding # => #<Encoding:ASCII-8BIT>
```

This is because the NUL characters in the second one (which aren't really NUL characters - they're part of the directive characters) explicitly trigger the encoding to change to binary.

There is a warning, but the warning is only for unexpected directives. How about disallowing or warning for non-ascii compatible format strings?



-- 
https://bugs.ruby-lang.org/

[ruby-core:110857] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[chrisseaton (Chris Seaton)]

Issue #19108 has been updated by chrisseaton (Chris Seaton).


> I think you want to mean "if the string is not ASCII-compatible".

Can you explain why?

I think a string is only a valid pack format string if it is `ascii_only?` - if it isn't `ascii_only?` then there is a silent warning and the output encoding is changed. We're proposing raising an error up front if the string is not `ascii_only?`.

----------------------------------------
Bug #19108: Format routines like pack blindly treat a string as ASCII-encoded
https://bugs.ruby-lang.org/issues/19108#change-100218

* Author: chrisseaton (Chris Seaton)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Format routines like pack and unpack blindly treat a string as ASCII-encoded, even if they aren't ASCII or ASCII-compatible.

I tried to construct code that was misleading using ASCII-incompatible-encodings but couldn't do it in practice (no ASCII-incompatible encodings have a pack directive ASCII byte that is encoded as a printable character.)

But I could demonstrate at least some strange behaviour:

```
p ['foo'].pack('u').encoding # => #<Encoding:US-ASCII>
p ['foo'].pack('u'.encode('UTF-32BE')).encoding # => #<Encoding:ASCII-8BIT>
```

This is because the NUL characters in the second one (which aren't really NUL characters - they're part of the directive characters) explicitly trigger the encoding to change to binary.

There is a warning, but the warning is only for unexpected directives. How about disallowing or warning for non-ascii compatible format strings?



-- 
https://bugs.ruby-lang.org/

[ruby-core:110864] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[alanwu (Alan Wu)]

Issue #19108 has been updated by alanwu (Alan Wu).


Checking `ascii_only?` would reject non-ascii comments which are fine:

```ruby
p [2, 89].pack(<<~PACK)
  C # 🚗
  c
PACK
p [2, 89].pack('Cc')
# Same output
```

----------------------------------------
Bug #19108: Format routines like pack blindly treat a string as ASCII-encoded
https://bugs.ruby-lang.org/issues/19108#change-100224

* Author: chrisseaton (Chris Seaton)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Format routines like pack and unpack blindly treat a string as ASCII-encoded, even if they aren't ASCII or ASCII-compatible.

I tried to construct code that was misleading using ASCII-incompatible-encodings but couldn't do it in practice (no ASCII-incompatible encodings have a pack directive ASCII byte that is encoded as a printable character.)

But I could demonstrate at least some strange behaviour:

```
p ['foo'].pack('u').encoding # => #<Encoding:US-ASCII>
p ['foo'].pack('u'.encode('UTF-32BE')).encoding # => #<Encoding:ASCII-8BIT>
```

This is because the NUL characters in the second one (which aren't really NUL characters - they're part of the directive characters) explicitly trigger the encoding to change to binary.

There is a warning, but the warning is only for unexpected directives. How about disallowing or warning for non-ascii compatible format strings?



-- 
https://bugs.ruby-lang.org/

[ruby-core:111117] [Ruby master Bug#19108] Format routines like pack blindly treat a string as ASCII-encoded

[matz (Yukihiro Matsumoto)]

Issue #19108 has been updated by matz (Yukihiro Matsumoto).


Template strings should be ASCII compatible, exceptions otherwise.

Matz.


----------------------------------------
Bug #19108: Format routines like pack blindly treat a string as ASCII-encoded
https://bugs.ruby-lang.org/issues/19108#change-100391

* Author: chrisseaton (Chris Seaton)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Format routines like pack and unpack blindly treat a string as ASCII-encoded, even if they aren't ASCII or ASCII-compatible.

I tried to construct code that was misleading using ASCII-incompatible-encodings but couldn't do it in practice (no ASCII-incompatible encodings have a pack directive ASCII byte that is encoded as a printable character.)

But I could demonstrate at least some strange behaviour:

```
p ['foo'].pack('u').encoding # => #<Encoding:US-ASCII>
p ['foo'].pack('u'.encode('UTF-32BE')).encoding # => #<Encoding:ASCII-8BIT>
```

This is because the NUL characters in the second one (which aren't really NUL characters - they're part of the directive characters) explicitly trigger the encoding to change to binary.

There is a warning, but the warning is only for unexpected directives. How about disallowing or warning for non-ascii compatible format strings?



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/