From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id E25D120A1E for ; Fri, 7 Dec 2018 11:39:33 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 0A57A120FEE; Fri, 7 Dec 2018 20:39:31 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id 1B4BB120FEE for ; Fri, 7 Dec 2018 20:39:26 +0900 (JST) Received: by filter0172p3mdw1.sendgrid.net with SMTP id filter0172p3mdw1-30880-5C0A5BEA-14 2018-12-07 11:39:22.344876791 +0000 UTC m=+213811.673902900 Received: from herokuapp.com (ec2-54-211-34-192.compute-1.amazonaws.com [54.211.34.192]) by ismtpd0018p1iad2.sendgrid.net (SG) with ESMTP id 0dXMsOPzQHObYoPfbVuZOg Fri, 07 Dec 2018 11:39:22.313 +0000 (UTC) Date: Fri, 07 Dec 2018 11:39:23 +0000 (UTC) From: me@dmitry.it To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 65758 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 15356 X-Redmine-Issue-Author: zunda X-Redmine-Sender: dm1try X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS7IjPK54OpBOqQ9c99xV5ZrbQt1P2MfnAfo1T fAiIhKekaDFL3kydeD1/PPCpAX0VYZDUBtBQPgpk68BgqABJnzCOHWWE3bB8v4yRs11GUD8VFdyqEU H8xpYiu0vl6nBOmF/gkfZz2l2SMO+rxnn3sF X-ML-Name: ruby-core X-Mail-Count: 90366 Subject: [ruby-core:90366] [Ruby trunk Bug#15356] Rack::Deflater on Rails responds with no data on Ruby 2.6.0 X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #15356 has been updated by dm1try (Dmitry Dedov). zunda (zunda an) wrote: > I'd like to report that a Rails app with `Rack::Deflater` enabled responds with a zero-length body on Ruby 2.6.0, to a request with `Accept-encoding: gzip` request header. It seems to happen at least on `x86_64-linux` (Ubuntu 18.04.1; see below for procedure for a reproduction) as well as on `universal.x86_64-darwin17` (macOS High Sierra 10.13.6; tested about a half year ago). I suspect some kind of change in behavior of pipe or named pipe is causing this. for me it looks like this is not related to the non-blocking I/O at all. the actual problem here: > $ sed -ie '/config.load_defaults 5.2/a config.middleware.use Rack::Deflater' \ > config/application.rb this will insert Deflater to the end of middleware stack: ``` bundle exec rake middleware ... use Rack::ETag use Rack::Deflater use Rack::TempfileReaper run Deflater::Application.routes ``` and the problem can be reproduced without rails by running: ```ruby use Rack::ETag use Rack::Deflater run ->(env) { [200, {}, ["test"]] } ``` though the resulting issue lies in the interaction between those middlewares: ```ruby # ETag def digest_body(body) parts = [] digest = nil body.each do |part| parts << part (digest ||= Digest::SHA256.new) << part unless part.empty? end [digest && digest.hexdigest.byteslice(0, 32), parts] end # Deflater ... def each(&block) @writer = block gzip =::Zlib::GzipWriter.new(self) gzip.mtime = @mtime @body.each { |part| gzip.write(part) gzip.flush } ensure gzip.close @writer = nil end def write(data) @writer.call(data) end ... ``` see the `write` callback; after https://bugs.ruby-lang.org/projects/ruby-trunk/repository/ruby-git/revisions/a55abcc0ca6f628fc05304f81e5a044d65ab4a68 the intermediate data is cleared on each write callback ```c rb_funcall(gz->io, id_write, 1, str); rb_str_resize(str, 0); ``` in the result, ETag middleware will have an array of empty strings as body parts anyway, using wrong position of Deflater will lead to unexpected behavior: - different ETag value is calculated for the same body on each reqeuest(because Deflater uses timestamp for packing) - in addition, after the mentioned patch, an empty body is returned(see above; looks like this is by design, though I'm not sure. should client code duplicate strings on write callback?) it looks like a popular suggestion is to put Deflater right after `ActionDispatch::Static`: ```ruby config.middleware.insert_after ActionDispatch::Static, Rack::Deflater ``` this should solve the actual problem. ---------------------------------------- Bug #15356: Rack::Deflater on Rails responds with no data on Ruby 2.6.0 https://bugs.ruby-lang.org/issues/15356#change-75476 * Author: zunda (zunda an) * Status: Closed * Priority: Normal * Assignee: * Target version: * ruby -v: ruby 2.6.0dev (2018-11-28 trunk 66081) [x86_64-linux] * Backport: 2.4: UNKNOWN, 2.5: UNKNOWN ---------------------------------------- I'd like to report that a Rails app with `Rack::Deflater` enabled responds with a zero-length body on Ruby 2.6.0, to a request with `Accept-encoding: gzip` request header. It seems to happen at least on `x86_64-linux` (Ubuntu 18.04.1; see below for procedure for a reproduction) as well as on `universal.x86_64-darwin17` (macOS High Sierra 10.13.6; tested about a half year ago). I suspect some kind of change in behavior of pipe or named pipe is causing this. ## 1: Prepare a Rails app, enable Rack::Deflater, and run the server ~~~ $ rbenv install 2.6.0-dev $ mkdir ruby26-rails-deflater $ cd ruby26-rails-deflater $ echo 2.6.0-dev > .ruby-version $ ruby --version ruby 2.6.0dev (2018-11-28 trunk 66081) [x86_64-linux] $ cat <<_END > Gemfile source 'https://rubygems.org' gem 'rails', '~> 5.2.1' _END $ bundle install --path=vendor/bundle $ bundle exec rails new test-deflater $ cd test-deflater $ sed -ie '/config.load_defaults 5.2/a config.middleware.use Rack::Deflater' \ config/application.rb $ bundle exec rails s ~~~ ## 2: Send a request >From another terminal, run the curl command. The `0` in the last line represents the first chunked response body with zero-length. I expect to see the "Yay! You're on Rails!" response body. ~~~ $ curl -H 'Accept-encoding: gzip' -v http://localhost:3000 --raw * Rebuilt URL to: http://localhost:3000/ * Trying 127.0.0.1... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 3000 (#0) > GET / HTTP/1.1 > Host: localhost:3000 > User-Agent: curl/7.58.0 > Accept: */* > Accept-encoding: gzip > < HTTP/1.1 200 OK < X-Frame-Options: SAMEORIGIN < X-XSS-Protection: 1; mode=block < X-Content-Type-Options: nosniff < X-Download-Options: noopen < X-Permitted-Cross-Domain-Policies: none < Referrer-Policy: strict-origin-when-cross-origin < Content-Type: text/html; charset=utf-8 < Vary: Accept-Encoding < Content-Encoding: gzip < ETag: W/"cd54cb5f8d93b7800b9e76460c5fe915" < Cache-Control: max-age=0, private, must-revalidate < Content-Security-Policy: script-src 'unsafe-inline'; style-src 'unsafe-inline' < X-Request-Id: e671eca8-f6ff-4599-b241-ea00fd864f56 < X-Runtime: 0.033015 < Transfer-Encoding: chunked < 0 * Connection #0 to host localhost left intact ~~~ Puma logs doesn't show anything out of ordinary: ~~~ Started GET "/" for 127.0.0.1 at 2018-11-28 09:09:44 -1000 Processing by Rails::WelcomeController#index as */* Rendering /home/zunda/.rbenv/versions/2.6.0-dev/lib/ruby/gems/2.6.0/gems/railties-5.2.1.1/lib/rails/templates/rails/welcome/index.html.erb Rendered /home/zunda/.rbenv/versions/2.6.0-dev/lib/ruby/gems/2.6.0/gems/railties-5.2.1.1/lib/rails/templates/rails/welcome/index.html.erb (5.1ms) Completed 200 OK in 17ms (Views: 10.8ms | ActiveRecord: 0.0ms) ~~~ Puma responds as expected without compression: ~~~ $ curl -s http://localhost:3000 | tail -8

Rails version: 5.2.1.1
Ruby version: 2.6.0 (x86_64-linux)

~~~ -- https://bugs.ruby-lang.org/