[ruby-core:54563] [ruby-trunk - Bug #8323][Open] no conversion by "bom|utf-8"

[ruby-core:54563] [ruby-trunk - Bug #8323][Open] no conversion by "bom|utf-8"

[nobu (Nobuyoshi Nakada)]

Issue #8323 has been reported by nobu (Nobuyoshi Nakada).

----------------------------------------
Bug #8323: no conversion by "bom|utf-8"
https://bugs.ruby-lang.org/issues/8323

Author: nobu (Nobuyoshi Nakada)
Status: Open
Priority: Normal
Assignee: naruse (Yui NARUSE)
Category: M17N
Target version: current: 2.1.0
ruby -v: 2.1.0
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN


openなどのモード指定にBOMつきのUTFを指定できますが、そこで指定された外部エンコーディングと内部エンコーディングが同じ場合、実際のエンコーディングに関わらず変換が行われません。
BOM指定に付加されたエンコーディングは、実際のエンコーディングではなくてあくまでフォールバックにすぎないので、変換に関してはBOMによって検出されたエンコーディングに従うべきと思われます。


-- 
http://bugs.ruby-lang.org/

[ruby-core:54564] [ruby-trunk - Bug #8323] no conversion by "bom|utf-8"

[nobu (Nobuyoshi Nakada)]

Issue #8323 has been updated by nobu (Nobuyoshi Nakada).

Description updated

Sorry, miss-post to ruby-core.
----------------------------------------
Bug #8323: no conversion by "bom|utf-8"
https://bugs.ruby-lang.org/issues/8323#change-38881

Author: nobu (Nobuyoshi Nakada)
Status: Open
Priority: Normal
Assignee: naruse (Yui NARUSE)
Category: M17N
Target version: current: 2.1.0
ruby -v: 2.1.0
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN


Mode spec in open and etc. accepts BOM-prefix UTF encoding, however if the external and internal encodings given there are same no conversion take place regardless the actual external encoding.
Since the encoding prefixed with "BOM" is not a real encoding, but just a fallback, the conversion should honor the detected encoding by BOM.



-- 
http://bugs.ruby-lang.org/

[ruby-core:54566] [ruby-trunk - Bug #8323] no conversion by "bom|utf-8"

[nobu (Nobuyoshi Nakada)]

Issue #8323 has been updated by nobu (Nobuyoshi Nakada).

File 0001-io.c-conversion-from-bom-encoding.patch added

A patch attached
----------------------------------------
Bug #8323: no conversion by "bom|utf-8"
https://bugs.ruby-lang.org/issues/8323#change-38883

Author: nobu (Nobuyoshi Nakada)
Status: Open
Priority: Normal
Assignee: naruse (Yui NARUSE)
Category: M17N
Target version: current: 2.1.0
ruby -v: 2.1.0
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN


Mode spec in open and etc. accepts BOM-prefix UTF encoding, however if the external and internal encodings given there are same no conversion take place regardless the actual external encoding.
Since the encoding prefixed with "BOM" is not a real encoding, but just a fallback, the conversion should honor the detected encoding by BOM.



-- 
http://bugs.ruby-lang.org/

[ruby-core:54569] [ruby-trunk - Bug #8323] no conversion by "bom|utf-8"

[naruse (Yui NARUSE)]

Issue #8323 has been updated by naruse (Yui NARUSE).


The patch doesn't work on following case:

%  ./ruby -e'IO.write"p","a";open("p","r:BOM|utf-8:utf-8"){|f|p f.read.size}'
-e:1:in `read': code converter not found (UTF-8 to UTF-8) (Encoding::ConverterNotFoundError)
        from -e:1:in `block in <main>'
        from -e:1:in `open'
        from -e:1:in `<main>'
----------------------------------------
Bug #8323: no conversion by "bom|utf-8"
https://bugs.ruby-lang.org/issues/8323#change-38887

Author: nobu (Nobuyoshi Nakada)
Status: Open
Priority: Normal
Assignee: naruse (Yui NARUSE)
Category: M17N
Target version: current: 2.1.0
ruby -v: 2.1.0
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN


Mode spec in open and etc. accepts BOM-prefix UTF encoding, however if the external and internal encodings given there are same no conversion take place regardless the actual external encoding.
Since the encoding prefixed with "BOM" is not a real encoding, but just a fallback, the conversion should honor the detected encoding by BOM.



-- 
http://bugs.ruby-lang.org/

[ruby-core:54571] [ruby-trunk - Bug #8323] no conversion by "bom|utf-8"

[nobu (Nobuyoshi Nakada)]

Issue #8323 has been updated by nobu (Nobuyoshi Nakada).

File 0001-io.c-conversion-from-bom-encoding.patch added

Updated.
----------------------------------------
Bug #8323: no conversion by "bom|utf-8"
https://bugs.ruby-lang.org/issues/8323#change-38890

Author: nobu (Nobuyoshi Nakada)
Status: Open
Priority: Normal
Assignee: naruse (Yui NARUSE)
Category: M17N
Target version: current: 2.1.0
ruby -v: 2.1.0
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN


Mode spec in open and etc. accepts BOM-prefix UTF encoding, however if the external and internal encodings given there are same no conversion take place regardless the actual external encoding.
Since the encoding prefixed with "BOM" is not a real encoding, but just a fallback, the conversion should honor the detected encoding by BOM.



-- 
http://bugs.ruby-lang.org/

[ruby-core:54574] [ruby-trunk - Bug #8323] no conversion by "bom|utf-8"

[naruse (Yui NARUSE)]

Issue #8323 has been updated by naruse (Yui NARUSE).


nobu (Nobuyoshi Nakada) wrote:
> Updated.

OK, commit please.
----------------------------------------
Bug #8323: no conversion by "bom|utf-8"
https://bugs.ruby-lang.org/issues/8323#change-38897

Author: nobu (Nobuyoshi Nakada)
Status: Open
Priority: Normal
Assignee: naruse (Yui NARUSE)
Category: M17N
Target version: current: 2.1.0
ruby -v: 2.1.0
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN


Mode spec in open and etc. accepts BOM-prefix UTF encoding, however if the external and internal encodings given there are same no conversion take place regardless the actual external encoding.
Since the encoding prefixed with "BOM" is not a real encoding, but just a fallback, the conversion should honor the detected encoding by BOM.



-- 
http://bugs.ruby-lang.org/

[ruby-core:54576] [ruby-trunk - Bug #8323] no conversion by "bom|utf-8"

[nobu (Nobuyoshi Nakada)]

Issue #8323 has been updated by nobu (Nobuyoshi Nakada).

Backport changed from 1.9.3: UNKNOWN, 2.0.0: UNKNOWN to 1.9.3: REQUIRED, 2.0.0: REQUIRED


----------------------------------------
Bug #8323: no conversion by "bom|utf-8"
https://bugs.ruby-lang.org/issues/8323#change-38901

Author: nobu (Nobuyoshi Nakada)
Status: Closed
Priority: Normal
Assignee: naruse (Yui NARUSE)
Category: M17N
Target version: current: 2.1.0
ruby -v: 2.1.0
Backport: 1.9.3: REQUIRED, 2.0.0: REQUIRED


Mode spec in open and etc. accepts BOM-prefix UTF encoding, however if the external and internal encodings given there are same no conversion take place regardless the actual external encoding.
Since the encoding prefixed with "BOM" is not a real encoding, but just a fallback, the conversion should honor the detected encoding by BOM.



-- 
http://bugs.ruby-lang.org/