* [ruby-dev:47369] [ruby-trunk - Bug #8445][Open] IO.open and IO#set_enconding does not support :fallback option
@ 2013-05-24 13:03 pjmtdw (Haruhiro Yoshimoto)
2013-05-25 14:50 ` [ruby-dev:47373] [ruby-trunk - Bug #8445][Assigned] " naruse (Yui NARUSE)
2014-01-30 6:17 ` [ruby-dev:47934] [ruby-trunk - Bug #8445] " shibata.hiroshi
0 siblings, 2 replies; 3+ messages in thread
From: pjmtdw (Haruhiro Yoshimoto) @ 2013-05-24 13:03 UTC (permalink / raw
To: ruby developers list
Issue #8445 has been reported by pjmtdw (Haruhiro Yoshimoto).
----------------------------------------
Bug #8445: IO.open and IO#set_enconding does not support :fallback option
https://bugs.ruby-lang.org/issues/8445
Author: pjmtdw (Haruhiro Yoshimoto)
Status: Open
Priority: Normal
Assignee:
Category:
Target version:
ruby -v: trunk(ruby 2.1.0dev)
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN
RubyDoc says that IO.open and IO#set_encoding supports optional argument defined in String#encode.
http://ruby-doc.org/core-2.0/IO.html#method-c-new-label-Options
In fact, :invalid, :undef and :replace works as expected.
However, :fallback option does not work neither for IO.open and IO#set_encoding.
Following is the example code which does not work.
f(x) is never called even if hoge.txt contains non convertible character.
File.open("./hoge.txt","r:Shift_JIS:utf-8", :fallback => lambda{|x|f(x)}){|f|
...
}
File.open("./hoge.txt"){|f|
f.set_encoding("Shift_JIS","utf-8",:fallback => lambda{|x|f(x)})
...
}
I Think this is because fill_cbuf() in io.c calls rb_econv_convert() from transcode.c directly.
On the other hand, fallback_func is called in transcode_loop(), which is called by str_encode().
Since transcode_loop() also calls rb_econv_convert(), I wrote a small patch which moves some codes from
transcode_loop() to rb_econv_convert() to fix the problem.
The attached file is the patch. Hope this helps.
--
http://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 3+ messages in thread
* [ruby-dev:47373] [ruby-trunk - Bug #8445][Assigned] IO.open and IO#set_enconding does not support :fallback option
2013-05-24 13:03 [ruby-dev:47369] [ruby-trunk - Bug #8445][Open] IO.open and IO#set_enconding does not support :fallback option pjmtdw (Haruhiro Yoshimoto)
@ 2013-05-25 14:50 ` naruse (Yui NARUSE)
2014-01-30 6:17 ` [ruby-dev:47934] [ruby-trunk - Bug #8445] " shibata.hiroshi
1 sibling, 0 replies; 3+ messages in thread
From: naruse (Yui NARUSE) @ 2013-05-25 14:50 UTC (permalink / raw
To: ruby developers list
Issue #8445 has been updated by naruse (Yui NARUSE).
Category set to M17N
Status changed from Open to Assigned
Assignee set to akr (Akira Tanaka)
Target version set to current: 2.1.0
Is this intentional or just unexpected inconsistency?
----------------------------------------
Bug #8445: IO.open and IO#set_enconding does not support :fallback option
https://bugs.ruby-lang.org/issues/8445#change-39528
Author: pjmtdw (Haruhiro Yoshimoto)
Status: Assigned
Priority: Normal
Assignee: akr (Akira Tanaka)
Category: M17N
Target version: current: 2.1.0
ruby -v: trunk(ruby 2.1.0dev)
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN
RubyDoc says that IO.open and IO#set_encoding supports optional argument defined in String#encode.
http://ruby-doc.org/core-2.0/IO.html#method-c-new-label-Options
In fact, :invalid, :undef and :replace works as expected.
However, :fallback option does not work neither for IO.open and IO#set_encoding.
Following is the example code which does not work.
f(x) is never called even if hoge.txt contains non convertible character.
File.open("./hoge.txt","r:Shift_JIS:utf-8", :fallback => lambda{|x|f(x)}){|f|
...
}
File.open("./hoge.txt"){|f|
f.set_encoding("Shift_JIS","utf-8",:fallback => lambda{|x|f(x)})
...
}
I Think this is because fill_cbuf() in io.c calls rb_econv_convert() from transcode.c directly.
On the other hand, fallback_func is called in transcode_loop(), which is called by str_encode().
Since transcode_loop() also calls rb_econv_convert(), I wrote a small patch which moves some codes from
transcode_loop() to rb_econv_convert() to fix the problem.
The attached file is the patch. Hope this helps.
--
http://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 3+ messages in thread
* [ruby-dev:47934] [ruby-trunk - Bug #8445] IO.open and IO#set_enconding does not support :fallback option
2013-05-24 13:03 [ruby-dev:47369] [ruby-trunk - Bug #8445][Open] IO.open and IO#set_enconding does not support :fallback option pjmtdw (Haruhiro Yoshimoto)
2013-05-25 14:50 ` [ruby-dev:47373] [ruby-trunk - Bug #8445][Assigned] " naruse (Yui NARUSE)
@ 2014-01-30 6:17 ` shibata.hiroshi
1 sibling, 0 replies; 3+ messages in thread
From: shibata.hiroshi @ 2014-01-30 6:17 UTC (permalink / raw
To: ruby-dev
Issue #8445 has been updated by Hiroshi SHIBATA.
Target version changed from 2.1.0 to current: 2.2.0
----------------------------------------
Bug #8445: IO.open and IO#set_enconding does not support :fallback option
https://bugs.ruby-lang.org/issues/8445#change-44779
* Author: Haruhiro Yoshimoto
* Status: Assigned
* Priority: Normal
* Assignee: Akira Tanaka
* Category: M17N
* Target version: current: 2.2.0
* ruby -v: trunk(ruby 2.1.0dev)
* Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN
----------------------------------------
RubyDoc says that IO.open and IO#set_encoding supports optional argument defined in String#encode.
http://ruby-doc.org/core-2.0/IO.html#method-c-new-label-Options
In fact, :invalid, :undef and :replace works as expected.
However, :fallback option does not work neither for IO.open and IO#set_encoding.
Following is the example code which does not work.
f(x) is never called even if hoge.txt contains non convertible character.
File.open("./hoge.txt","r:Shift_JIS:utf-8", :fallback => lambda{|x|f(x)}){|f|
...
}
File.open("./hoge.txt"){|f|
f.set_encoding("Shift_JIS","utf-8",:fallback => lambda{|x|f(x)})
...
}
I Think this is because fill_cbuf() in io.c calls rb_econv_convert() from transcode.c directly.
On the other hand, fallback_func is called in transcode_loop(), which is called by str_encode().
Since transcode_loop() also calls rb_econv_convert(), I wrote a small patch which moves some codes from
transcode_loop() to rb_econv_convert() to fix the problem.
The attached file is the patch. Hope this helps.
---Files--------------------------------
support-fallback-for-io.patch (3.9 KB)
--
http://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2014-01-30 6:36 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-05-24 13:03 [ruby-dev:47369] [ruby-trunk - Bug #8445][Open] IO.open and IO#set_enconding does not support :fallback option pjmtdw (Haruhiro Yoshimoto)
2013-05-25 14:50 ` [ruby-dev:47373] [ruby-trunk - Bug #8445][Assigned] " naruse (Yui NARUSE)
2014-01-30 6:17 ` [ruby-dev:47934] [ruby-trunk - Bug #8445] " shibata.hiroshi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).