From mboxrd@z Thu Jan 1 00:00:00 1970 Delivered-To: chneukirchen@gmail.com Received: by 10.76.159.169 with SMTP id xd9csp56943oab; Thu, 17 Jul 2014 16:10:04 -0700 (PDT) Return-Path: Received-SPF: pass (google.com: domain of rack-devel+bncBD75LW742ECRBS5PUGPAKGQEDVXANZY@googlegroups.com designates 10.50.111.232 as permitted sender) client-ip=10.50.111.232 Authentication-Results: mr.google.com; spf=pass (google.com: domain of rack-devel+bncBD75LW742ECRBS5PUGPAKGQEDVXANZY@googlegroups.com designates 10.50.111.232 as permitted sender) smtp.mail=rack-devel+bncBD75LW742ECRBS5PUGPAKGQEDVXANZY@googlegroups.com; dkim=pass header.i=@googlegroups.com X-Received: from mr.google.com ([10.50.111.232]) by 10.50.111.232 with SMTP id il8mr322722igb.6.1405638603694 (num_hops = 1); Thu, 17 Jul 2014 16:10:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20120806; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :x-original-sender:x-original-authentication-results:reply-to :precedence:mailing-list:list-id:list-post:list-help:list-archive :sender:list-subscribe:list-unsubscribe:content-type; bh=QTIkL9MU+NZ54Pj77PBosCaeND1Xriv5vdH17ihfS2E=; b=bjqdK9S/M8nQTJNGu2cydFSsn59VWAFXYojr/2HmvFJM6TFmBJZCNTRtGe0hyTrJXS SkMBeYYSw8PXod+ae5YriJ93WrKb5tkI8ZXnmKRR80CjR5q+vobinW8TwAejWQikZQx5 jMNTulNEFndYzmJ1nB5s2kOs9AEN7IKCcuj5F2P1qhgmxoxeOOBJUic3CfsaRLYDYEOA l1lCEEtHntHnRKgKUZF2Un/Vu1RGFkF/VS8T2v92UPGkty3SAdd8ZEFTPqLeWOKM/jd5 82eOVzUUg+4lifMFWuLpJVdHSik+sdvJ+CfQt59SL76ttHsLXOL1W+cWc18pqTPJHt1v TfKQ== X-Received: by 10.50.111.232 with SMTP id il8mr39257igb.6.1405638603605; Thu, 17 Jul 2014 16:10:03 -0700 (PDT) X-BeenThere: rack-devel@googlegroups.com Received: by 10.51.17.100 with SMTP id gd4ls1939264igd.21.gmail; Thu, 17 Jul 2014 16:10:02 -0700 (PDT) X-Received: by 10.66.144.7 with SMTP id si7mr354543pab.32.1405638602859; Thu, 17 Jul 2014 16:10:02 -0700 (PDT) Received: from mail-oa0-x229.google.com (mail-oa0-x229.google.com [2607:f8b0:4003:c02::229]) by gmr-mx.google.com with ESMTPS id h10si47015igq.1.2014.07.17.16.10.02 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 17 Jul 2014 16:10:02 -0700 (PDT) Received-SPF: pass (google.com: domain of jftucker@gmail.com designates 2607:f8b0:4003:c02::229 as permitted sender) client-ip=2607:f8b0:4003:c02::229; Received: by mail-oa0-f41.google.com with SMTP id j17so1843378oag.14 for ; Thu, 17 Jul 2014 16:10:02 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.60.42.129 with SMTP id o1mr696486oel.66.1405638602552; Thu, 17 Jul 2014 16:10:02 -0700 (PDT) Received: by 10.76.71.197 with HTTP; Thu, 17 Jul 2014 16:10:02 -0700 (PDT) In-Reply-To: <20140717223454.GA22447@dcvr.yhbt.net> References: <20140717223454.GA22447@dcvr.yhbt.net> Date: Thu, 17 Jul 2014 16:10:02 -0700 Message-ID: Subject: Re: rational for rewind() From: James Tucker To: rack-devel@googlegroups.com X-Original-Sender: jftucker@gmail.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jftucker@gmail.com designates 2607:f8b0:4003:c02::229 as permitted sender) smtp.mail=jftucker@gmail.com; dkim=pass header.i=@gmail.com; dmarc=pass (p=NONE dis=NONE) header.from=gmail.com Reply-To: rack-devel@googlegroups.com Precedence: list Mailing-list: list rack-devel@googlegroups.com; contact rack-devel+owners@googlegroups.com List-ID: X-Google-Group-Id: 486215384060 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , Content-Type: multipart/alternative; boundary=001a11c20daa96976204fe6bbdef --001a11c20daa96976204fe6bbdef Content-Type: text/plain; charset=UTF-8 On Thu, Jul 17, 2014 at 3:34 PM, Eric Wong wrote: > (resending my original message for the archives, original seems lost, > Torsten already got this privately) > > Torsten Robitzki wrote: > > Hello, > > I'm implementing a C++ comet web server, that (tries) to implement rack > to > > adapt ruby applications. Currently I'm reading a body very naively put > the > > body into a ruby String and wrap it with a StringIO to provide the > > rack.input for the implementation. As I'm going to use the server for > > uploading images, I would like to implement a real, stream-like object to > > circumvent the need to buffer the POST body before handing it to the > > application. > > unicorn implements input like tee(1) doing lazy, rewindable buffering: > http://unicorn.bogomips.org/Unicorn/TeeInput.html This is the approach I would take. I'm not sure if TeeInput supports it, but for a generic server supporting websocket type use cases, I'd add a discardable buffer API too, so you can "hijack" the input stream and release any stale resources. > Now, I've read the rack specs and read about rewind(). To implement > > rewind(), I would have to store the whole body, even when the upstream > > application just calculates some kind of checksum on the body, or uploads > > it to s3. What's the rational behind this part of the specification? Is > it > > possible to not implement rewind() and to tell applications that need > > rewind(), to keep there own copy of the body, in case it is needed? > > I don't know the rationale, but I suspect it's to make life easier for > application/API authors to know the data will exist for the lifetime > of the request/response cycle. > > But I think having rewind as part of the spec sucks. Agreed. I generally think a lot of concerns that used to be in middleware should not be so (e.g. chunking and content-length), but this concern actually could be handled by middleware so apps that want it can add it in the appropriate place in the chain. This would remove the responsibility from the server authors, which opens up a lot of potential optimizations and flexibility. We also lack a lot of potentially useful security considerations in this area (i.e. both static and dynamic configurable limits). > I seem to recall > there was hope of getting rid of it for Rack 2 (if it ever happens). > Yup. > However, since I'm not sure if Rack 2 will happen, rewind can already > be disabled and break spec in servers I implement: > > unicorn has a "rewindable_input " option (default=true): > http://unicorn.bogomips.org/Unicorn/Configurator.html > > yahns is a tri-state: "input_buffering <:lazy|true|false>" option: > http://yahns.yhbt.net/yahns_config.txt (default=true) > > Fwiw, rewindable_input(true) in unicorn is equivalent to > input_buffering(:lazy) in yahns. This is because unicorn expects nginx > to buffer for slow clients, whereas yahns does not require nginx and > buffers asynchronously even with trickling clients. > This is the approach I recommend today. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "Rack Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to rack-devel+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "Rack Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to rack-devel+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/d/optout. --001a11c20daa96976204fe6bbdef Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Thu, Jul 17, 2014 at 3:34 PM, Eric Wong <e@80x24.org> w= rote:
(resending my original message for the archi= ves, original seems lost,
=C2=A0Torsten already got this privately)

Torsten Robitzki <Torsten@Robitzki.de> wrote:
> Hello,
> I'm implementing a C++ comet web server, that (tries) to implement= rack to
> adapt ruby applications. Currently I'm reading a body very naively= put the
> body into a ruby String and wrap it with a StringIO to provide the
> rack.input for the implementation. As I'm going to use the server = for
> uploading images, I would like to implement a real, stream-like object= to
> circumvent the need to buffer the POST body before handing it to the > application.

unicorn implements input like tee(1) doing lazy, rewindable buffering= :
=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://unicorn.bogomips.org/Unicorn/TeeInp= ut.html

This is the approach I would ta= ke. I'm not sure if TeeInput supports it, but for a generic server supp= orting websocket type use cases, I'd add a discardable buffer API too, = so you can "hijack" the input stream and release any stale resour= ces.

> Now, I've read the rack specs and read about rewind(). To implemen= t
> rewind(), I would have to store the whole body, even when the upstream=
> application just calculates some kind of checksum on the body, or uplo= ads
> it to s3. What's the rational behind this part of the specificatio= n? Is it
> possible to not implement rewind() and to tell applications that need<= br> > rewind(), to keep there own copy of the body, in case it is needed?
I don't know the rationale, but I suspect it's to make life e= asier for
application/API authors to know the data will exist for the lifetime
of the request/response cycle.

But I think having rewind as part of the spec sucks.

<= /div>
Agreed. I generally think a lot of concerns that used to be in mi= ddleware should not be so (e.g. chunking and content-length), but this conc= ern actually could be handled by middleware so apps that want it can add it= in the appropriate place in the chain. This would remove the responsibilit= y from the server authors, which opens up a lot of potential optimizations = and flexibility. We also lack a lot of potentially useful security consider= ations in this area (i.e. both static and dynamic configurable limits).
=C2=A0
=C2=A0I seem to recall
there was hope of getting rid of it for Rack 2 (if it ever happens).

Yup.
=C2=A0
However, since I'm not sure if Rack 2 will happen, rewind can already be disabled and break spec in servers I implement:

unicorn has a "rewindable_input <true|false>" option (defau= lt=3Dtrue):
=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://unicorn.bogomips.org/Unicorn/Co= nfigurator.html

yahns is a tri-state: "input_buffering <:lazy|true|false>" = option:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://yahns.yhbt.net/yahns_config.txt (default= =3Dtrue)

Fwiw, rewindable_input(true) in unicorn is equivalent to
input_buffering(:lazy) in yahns. =C2=A0This is because unicorn expects ngin= x
to buffer for slow clients, whereas yahns does not require nginx and
buffers asynchronously even with trickling clients.
This is the approach I recommend today.
=C2=A0
=

--

---
You received this message because you are subscribed to the Google Groups &= quot;Rack Development" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to rack-d= evel+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups &= quot;Rack Development" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to rack-dev= el+unsubscribe@googlegroups.com.
For more options, visit http= s://groups.google.com/d/optout.
--001a11c20daa96976204fe6bbdef--