[ruby-dev:50734] [Ruby trunk Bug#15468] Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string

ruby-dev (Japanese) list archive (unofficial mirror)
 help / color / mirror / Atom feed

* [ruby-dev:50734] [Ruby trunk Bug#15468] Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
       [not found] <redmine.issue-15468.20181227044528@ruby-lang.org>
@ 2018-12-27  4:45 ` eitoball
  2019-01-10 17:00 ` [ruby-dev:50759] " rafaelmfranca
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 4+ messages in thread
From: eitoball @ 2018-12-27  4:45 UTC (permalink / raw)
  To: ruby-dev

Issue #15468 has been reported by eitoball (Eito Katagiri).

----------------------------------------
Bug #15468: Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
https://bugs.ruby-lang.org/issues/15468

* Author: eitoball (Eito Katagiri)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
ruby-2.6.0で大きなJSON文字列をNet::HTTPでPOSTすると以下のように NoMothodError が発生するようになりました。
~~~
$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
~~~

理由は、Net::Protocol::BufferedIO#write0で、String#[]で送信されていない文字列を求める際に文字列にマルチバイト文字が含まれているとIO#write_nonblockで返ってくる送信されたバイト数が文字数を超えるため、nilになるためだと考えています。https://github.com/ruby/ruby/pull/2058 を送ってみました。



-- 
https://bugs.ruby-lang.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [ruby-dev:50759] [Ruby trunk Bug#15468] Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
       [not found] <redmine.issue-15468.20181227044528@ruby-lang.org>
  2018-12-27  4:45 ` [ruby-dev:50734] [Ruby trunk Bug#15468] Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string eitoball
@ 2019-01-10 17:00 ` rafaelmfranca
  2019-01-12 21:03 ` [ruby-dev:50760] " naruse
  2019-01-23 21:41 ` [ruby-dev:50769] " samuel
  3 siblings, 0 replies; 4+ messages in thread
From: rafaelmfranca @ 2019-01-10 17:00 UTC (permalink / raw)
  To: ruby-dev

Issue #15468 has been updated by rafaelfranca (Rafael França).


Can we get this issue closed and the with right backport field values?

----------------------------------------
Bug #15468: Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
https://bugs.ruby-lang.org/issues/15468#change-76226

* Author: eitoball (Eito Katagiri)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
ruby-2.6.0で大きなJSON文字列をNet::HTTPでPOSTすると以下のように NoMothodError が発生するようになりました。
~~~
$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
~~~

理由は、Net::Protocol::BufferedIO#write0で、String#[]で送信されていない文字列を求める際に文字列にマルチバイト文字が含まれているとIO#write_nonblockで返ってくる送信されたバイト数が文字数を超えるため、nilになるためだと考えています。https://github.com/ruby/ruby/pull/2058 を送ってみました。



-- 
https://bugs.ruby-lang.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [ruby-dev:50760] [Ruby trunk Bug#15468] Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
       [not found] <redmine.issue-15468.20181227044528@ruby-lang.org>
  2018-12-27  4:45 ` [ruby-dev:50734] [Ruby trunk Bug#15468] Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string eitoball
  2019-01-10 17:00 ` [ruby-dev:50759] " rafaelmfranca
@ 2019-01-12 21:03 ` naruse
  2019-01-23 21:41 ` [ruby-dev:50769] " samuel
  3 siblings, 0 replies; 4+ messages in thread
From: naruse @ 2019-01-12 21:03 UTC (permalink / raw)
  To: ruby-dev

Issue #15468 has been updated by naruse (Yui NARUSE).

Backport changed from 2.4: DONTNEED, 2.5: DONTNEED, 2.6: REQUIRED to 2.4: DONTNEED, 2.5: DONTNEED, 2.6: DONE

ruby_2_6 r66799 merged revision(s) 66582.

----------------------------------------
Bug #15468: Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
https://bugs.ruby-lang.org/issues/15468#change-76269

* Author: eitoball (Eito Katagiri)
* Status: Closed
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: DONTNEED, 2.5: DONTNEED, 2.6: DONE
----------------------------------------
ruby-2.6.0で大きなJSON文字列をNet::HTTPでPOSTすると以下のように NoMothodError が発生するようになりました。
~~~
$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
~~~

理由は、Net::Protocol::BufferedIO#write0で、String#[]で送信されていない文字列を求める際に文字列にマルチバイト文字が含まれているとIO#write_nonblockで返ってくる送信されたバイト数が文字数を超えるため、nilになるためだと考えています。https://github.com/ruby/ruby/pull/2058 を送ってみました。



-- 
https://bugs.ruby-lang.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [ruby-dev:50769] [Ruby trunk Bug#15468] Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
       [not found] <redmine.issue-15468.20181227044528@ruby-lang.org>
                   ` (2 preceding siblings ...)
  2019-01-12 21:03 ` [ruby-dev:50760] " naruse
@ 2019-01-23 21:41 ` samuel
  3 siblings, 0 replies; 4+ messages in thread
From: samuel @ 2019-01-23 21:41 UTC (permalink / raw)
  To: ruby-dev

Issue #15468 has been updated by ioquatix (Samuel Williams).


Can we please fast track the release of this? It's breaking a ton of stuff.

Stepping back a little bit from this issue, I think we see a picture of Ruby where:

- Handling binary data can cause bugs like this (mismatch between binary and encoded string).
- Reading and writing strings is tricky, e.g. https://bugs.ruby-lang.org/issues/13626
- There are some odd edge cases which can cause problems, e.g. https://bugs.ruby-lang.org/issues/14975
- Reading and writing can put pressure on GC even though no additional strings should be allocated, see  https://bugs.ruby-lang.org/issues/15560 for ideas to fix this.

I feel like some improvements to handling binary data could be a very useful addition to Ruby.

I welcome further discussion here and on the above issues so that we can avoid issues like this in the future. Ideally, we provide APIs such that this kind of bug is impossible.

----------------------------------------
Bug #15468: Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
https://bugs.ruby-lang.org/issues/15468#change-76480

* Author: eitoball (Eito Katagiri)
* Status: Closed
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
* Backport: 2.4: DONTNEED, 2.5: DONTNEED, 2.6: DONE
----------------------------------------
ruby-2.6.0で大きなJSON文字列をNet::HTTPでPOSTすると以下のように NoMothodError が発生するようになりました。
~~~
$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
~~~

理由は、Net::Protocol::BufferedIO#write0で、String#[]で送信されていない文字列を求める際に文字列にマルチバイト文字が含まれているとIO#write_nonblockで返ってくる送信されたバイト数が文字数を超えるため、nilになるためだと考えています。https://github.com/ruby/ruby/pull/2058 を送ってみました。



-- 
https://bugs.ruby-lang.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2019-01-23 21:41 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <redmine.issue-15468.20181227044528@ruby-lang.org>
2018-12-27  4:45 ` [ruby-dev:50734] [Ruby trunk Bug#15468] Net::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string eitoball
2019-01-10 17:00 ` [ruby-dev:50759] " rafaelmfranca
2019-01-12 21:03 ` [ruby-dev:50760] " naruse
2019-01-23 21:41 ` [ruby-dev:50769] " samuel

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).