From mboxrd@z Thu Jan 1 00:00:00 1970 Delivered-To: chneukirchen@gmail.com Received: by 10.229.190.149 with SMTP id di21cs78278qcb; Mon, 25 Apr 2011 18:16:37 -0700 (PDT) Return-Path: Received-SPF: pass (google.com: domain of rack-devel+bncCNSYpqGTFhDyudjtBBoEcgMrnQ@googlegroups.com designates 10.142.50.11 as permitted sender) client-ip=10.142.50.11; Authentication-Results: mr.google.com; spf=pass (google.com: domain of rack-devel+bncCNSYpqGTFhDyudjtBBoEcgMrnQ@googlegroups.com designates 10.142.50.11 as permitted sender) smtp.mail=rack-devel+bncCNSYpqGTFhDyudjtBBoEcgMrnQ@googlegroups.com; dkim=pass header.i=rack-devel+bncCNSYpqGTFhDyudjtBBoEcgMrnQ@googlegroups.com Received: from mr.google.com ([10.142.50.11]) by 10.142.50.11 with SMTP id x11mr38043wfx.18.1303780595894 (num_hops = 1); Mon, 25 Apr 2011 18:16:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=beta; h=domainkey-signature:x-beenthere:received-spf:date:from:to:subject :message-id:references:mime-version:in-reply-to:user-agent :x-original-sender:x-original-authentication-results:reply-to :precedence:mailing-list:list-id:x-google-group-id:list-post :list-help:list-archive:sender:list-subscribe:list-unsubscribe :content-type:content-disposition; bh=nfDgeCBABoH5K7Q1wWoCe3dkDp8w6vFffXKZLOByUrM=; b=WXA+JbF9xbty4IZZj1645CCUok2C1tDVbbrxABtH4soSYaqQs3C5z1S+usgAer2sbx p/jKyc59HuKGKZ2I8Dd+A6iMPGzC0Bt1jJVHYCZD166ejZ2XOe6Bs+VsZjIZhlLkhrAs /TONWpk0Ceixa6NIqnal2t8J5a9h8nXXKykUM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlegroups.com; s=beta; h=x-beenthere:received-spf:date:from:to:subject:message-id:references :mime-version:in-reply-to:user-agent:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-google-group-id:list-post:list-help:list-archive:sender :list-subscribe:list-unsubscribe:content-type:content-disposition; b=uaa6DaBHaujYifWa3Ih8QuRLh1HnD0YGqHP2zqQL1wvYFGO3Qh2pmJmU/eH0HKfq/B uVae1lZZeUnP8rxs/DPxbMEJ/94KmFqCUgHDB7pr2rq9KIg8g9oXSOJuxQhNt4ijfFuR idF14IyfYKi6PkC3y8lUHBM0UKXzjvxBRjt94= Received: by 10.142.50.11 with SMTP id x11mr8644wfx.18.1303780594194; Mon, 25 Apr 2011 18:16:34 -0700 (PDT) X-BeenThere: rack-devel@googlegroups.com Received: by 10.68.8.3 with SMTP id n3ls2459386pba.3.gmail; Mon, 25 Apr 2011 18:16:33 -0700 (PDT) Received: by 10.68.30.194 with SMTP id u2mr9544pbh.54.1303780593087; Mon, 25 Apr 2011 18:16:33 -0700 (PDT) Received: by 10.68.30.194 with SMTP id u2mr9543pbh.54.1303780593072; Mon, 25 Apr 2011 18:16:33 -0700 (PDT) Received: from dcvr.yhbt.net ([64.71.152.64]) by gmr-mx.google.com with ESMTP id o5si2725588pbb.3.2011.04.25.18.16.31; Mon, 25 Apr 2011 18:16:31 -0700 (PDT) Received-SPF: error (google.com: error in processing during lookup of normalperson@yhbt.net: DNS timeout) client-ip=64.71.152.64; Received: from localhost (unknown [127.0.2.5]) by dcvr.yhbt.net (Postfix) with ESMTP id 949B71F6AF; Tue, 26 Apr 2011 01:16:28 +0000 (UTC) Date: Mon, 25 Apr 2011 18:16:28 -0700 From: Eric Wong To: rack-devel@googlegroups.com Subject: Re: Does Rack-servers block until the whole request body has been read? Message-ID: <20110426011628.GA25804@dcvr.yhbt.net> References: <31236863.1318.1302679061651.JavaMail.geo-discussion-forums@yqlq3> MIME-Version: 1.0 In-Reply-To: <31236863.1318.1302679061651.JavaMail.geo-discussion-forums@yqlq3> User-Agent: Mutt/1.5.18 (2008-05-17) X-Original-Sender: normalperson@yhbt.net X-Original-Authentication-Results: gmr-mx.google.com; spf=temperror (google.com: error in processing during lookup of normalperson@yhbt.net: DNS timeout) smtp.mail=normalperson@yhbt.net Reply-To: rack-devel@googlegroups.com Precedence: list Mailing-list: list rack-devel@googlegroups.com; contact rack-devel+owners@googlegroups.com List-ID: X-Google-Group-Id: 486215384060 List-Post: , List-Help: , List-Archive: Sender: rack-devel@googlegroups.com List-Subscribe: , List-Unsubscribe: , Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Daniel Abrahamsson wrote: > I am working on a server that deals with large file uploads (currently built > upon Rails, with a Metal taking care of the uploads). According to the > rack-specification: > "... handler developers must buffer the input data into some rewindable > object if the underlying input stream is not rewindable". The current Rack spec requires rewindability, but I think the requirement will go away for Rack 2.0 > For Passenger and Mongrel, this means a temp file is created for the request > (at least for requests as big as those I am dealing with). > > Now, my question is, does Rack wait until all data from the client has been > read to a tempfile before passing on control to the Rack app (in this case, > Rails)? Or does it pass on control directly, and block if the application > reads data faster than the client is sending it? It depends on the Rack webserver. I can confirm Mongrel waits until everthing is sent. Disclaimer: I'm the BFDL for both Unicorn and Rainbows! Both Unicorn[1] and Rainbows![3] can pass control directly and block inside the app if the app is faster than the client. Unicorn reads off the socket as data comes in and "tee"'s off the input to a temporary file for rewindability. However, you can disable the temporary file backing store by setting the "rewindable_input false" option in the Unicorn config file[2]. Setting "rewindable_input false" violates the Rack spec, but can save you a lot of filesystem I/O if you handle large files. Unless all your clients are fast (on the same LAN), Unicorn requires nginx in front of it and nginx reads everything before sending it to Unicorn), so "rewindable_input false" doesn't do much, however... Rainbows! is based on Unicorn, but it is designed to handle slow clients. It offers many concurrency options, and some of them offer rack.input streaming like Unicorn does. I recommend ThreadSpawn/ThreadPool for the greatest compatibility if you want input streaming[4]. > After some initial testing, it appears to me that the former is the case. > Can anyone give me advice on how to test this? Perhaps it depends on the web > server used? Perhaps on the framework? The framework also matters. You'll want to audit all your layers and make sure they don't call methods like "rewind" or "size" on the env["rack.input"] object. I once wrote an app (which I haven't touched in a long time, but probably still works) to do upload progress with Rack + Rainbows! based on the streaming rack.input support I have: http://upr.bogomips.org/ [1] http://unicorn.bogomips.org/ [2] http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-rewindable_input [3] http://rainbows.rubyforge.org/ [4] http://rainbows.rubyforge.org/Summary.html -- Eric Wong