From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-3.1 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, FORGED_GMAIL_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id A683B1F453 for ; Mon, 11 Feb 2019 18:32:39 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 7A84312140E; Tue, 12 Feb 2019 03:32:35 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id 0178A1213A0 for ; Tue, 12 Feb 2019 03:32:32 +0900 (JST) Received: by filter0124p3las1.sendgrid.net with SMTP id filter0124p3las1-20838-5C61BFBE-27 2019-02-11 18:32:30.54228761 +0000 UTC m=+354.426402109 Received: from herokuapp.com (ec2-52-90-27-246.compute-1.amazonaws.com [52.90.27.246]) by ismtpd0025p1mdw1.sendgrid.net (SG) with ESMTP id tmgx7HWUQNOdiLTgnhtqpg for ; Mon, 11 Feb 2019 18:32:30.434 +0000 (UTC) Date: Mon, 11 Feb 2019 18:32:32 +0000 (UTC) From: josh.nw@gmail.com To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 66963 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 14972 X-Redmine-Issue-Author: joshc X-Redmine-Sender: joshc X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS4x4UU3f/94YnVnPYcxYgveChbiV65d/Il27w +w/3SakdQd1VSur3qjummIoXiGTO45JCmAbiVoOSnwcoO8AP2l/qIQbec9cdAAJC6ACLc1uB1/aeuE Jkscpsci4Ic9h1BM3rOdLlhQ3SbFBuaM0pzfTG6nAo+zCr5TjKREL1oEKw== X-ML-Name: ruby-core X-Mail-Count: 91510 Subject: [ruby-core:91510] [Ruby trunk Bug#14972] Net::HTTP inconsistently raises EOFError when peer closes the connection X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #14972 has been updated by joshc (Josh C). > It is to resume with the partially downloaded result. Doesn't `Net::HTTPResponse#read_body` raise if called more than once? How can the caller resume? ---------------------------------------- Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connection https://bugs.ruby-lang.org/issues/14972#change-76770 * Author: joshc (Josh C) * Status: Open * Priority: Normal * Assignee: * Target version: 2.7 * ruby -v: ruby 2.4.3p205 (2017-12-14 revision 61247) [x86_64-darwin15] * Backport: 2.3: DONTNEED, 2.4: DONTNEED, 2.5: DONTNEED ---------------------------------------- If chunked transfer encoding is used, and the peer closes the connection while the caller is reading data, then the `Net::HTTP::Response#read_body` method will raise `EOFError`. If chunked transfer encoding is not used (and an explicit `Content-Length` is used instead), the `read_body` method swallows the `EOFError` exception. I would expect `read_body` to raise EOFError if it reads fewer than `Content-Length` bytes. The current behavior is explained by the `ignore_eof` parameter in https://github.com/ruby/ruby/blob/v2_4_3/lib/net/http/response.rb#L284-L301. However, RFC 7230 section 3.3.3 https://tools.ietf.org/html/rfc7230#section-3.3.3 says: ~~~ 5. If a valid Content-Length header field is present without Transfer-Encoding, its decimal value defines the expected message body length in octets. If the sender closes the connection or the recipient times out before the indicated number of octets are received, the recipient MUST consider the message to be incomplete and close the connection. ~~~ As it is now, if chunked encoding is not used, then the caller is unaware when the response body is truncated. In order to detect it, the caller must count the number of bytes read until `Content-Length` is reached. However, that means you can't use ruby's automatic decompression, because `Content-Length` is the number of compressed bytes, while `read_body` yields chunks of uncompressed data. Here's sample code to reproduce. Run the following http server. Note chunked is currently false, but can be toggled. ~~~ require 'webrick' server = WEBrick::HTTPServer.new :Port => 8000 trap 'INT' do server.shutdown end # toggle this chunked = false server.mount_proc '/' do |req, res| res.status = 200 res['Content-Type'] = 'text/plain' str = "0123456789" * 10000 res.body = str if chunked res.chunked = true else res['Content-Length'] = str.length end end server.start ~~~ Run the following http client code. In order to simulate a closed connection, the block raises EOFError. ~~~ require 'net/http' require 'uri' uri = URI("http://localhost:8000/") Net::HTTP.start(uri.host, uri.port) do |http| http.request_get(uri.path) do |response| response.read_body do |chunk| puts "Read #{chunk.length} bytes" raise EOFError.new("whoops") end end end puts "EOF was silently caught" ~~~ When chunked encoding is used, the exception is properly raised. I believe ruby is retrying the request because `GET` is idempotent: ~~~ $ ruby --version ruby 2.4.3p205 (2017-12-14 revision 61247) [x86_64-darwin15] $ ruby client.rb Content-Length: Transfer-Encoding: chunked Read 16377 bytes Content-Length: Transfer-Encoding: chunked Read 16377 bytes client.rb:11:in `block (3 levels) in
': whoops (EOFError) from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/protocol.rb:429:in `call_block' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/protocol.rb:420:in `<<' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/protocol.rb:122:in `read' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http/response.rb:322:in `read_chunked' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http/response.rb:286:in `block in read_body_0' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http/response.rb:278:in `inflater' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http/response.rb:283:in `read_body_0' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http/response.rb:204:in `read_body' from client.rb:9:in `block (2 levels) in
' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http.rb:1455:in `block in transport_request' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http/response.rb:165:in `reading_body' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http.rb:1454:in `transport_request' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http.rb:1416:in `request' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http.rb:1317:in `request_get' from client.rb:6:in `block in
' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http.rb:877:in `start' from /usr/local/opt/rbenv/versions/2.4.3/lib/ruby/2.4.0/net/http.rb:608:in `start' from client.rb:5:in `
' ~~~ When chunked encoding is not used, the exception is not raised: ~~~ ruby client.rb Content-Length: 100000 Transfer-Encoding: Read 0 bytes EOF was silently caught ~~~ I verified the behavior exists as far back as ruby 1.9.3p551. It was introduced in https://github.com/ruby/ruby/commit/cdc7602379c9d911983db2c044d69ac417869266#diff-8c2ab8e0fb4f052e1d95ab6334e192c1R949. -- https://bugs.ruby-lang.org/