Re: Not cleaning up tempfiles for multipart?

[Eric Wong <normalperson@yhbt.net>]

Charles Oliver Nutter <headius@headius.com> wrote:
> On Sun, Mar 7, 2010 at 7:12 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > I guess this deals with wonky multipart uploads that browsers still
> > generate these days[1].  Ugh, yeah, it's nasty...
> > The comment below in _call is very important.
> >
> >
> > 1.  Ensure all tempfiles created by Rack go into an array in env,
> >    probably env["rack.tempfiles"]:
> >
> >    tempfile = Tempfile.new("foo")
> >    (env["rack.tempfiles"] ||= []) << tempfile
> 
> I'm with you so far...
> 
> > 2.  Have a middleware wrap everything, including the response body:
> 
> I'm a bit new to the middleware thing. Does this mean everyone would
> have to configure middleware on their setups to get this
> tempfile-closing behavior? And your comment below...does that mean
> there are situations where this wouldn't work?
> 
> Closing and removing tempfiles when they're no longer needed should be
> the default behavior, not something you have to configure. It's a bug
> to not close and remove them (or at least, a bug to keep creating new
> ones and expecting GC to clean everything up eventually).

I would expect major frameworks to stick this into the default
middleware stack as a convenience, but some users want a more bare-bones
Rack stack can still opt out.  I see it as needless overhead as the vast
majority of HTTP requests do not create tempfiles.

I don't make decisions for Rack, though.

<snip>

> >      # the Rack server should call this (when we're the body)
> >      def close
> >        tempfiles = env["rack.tempfiles"]
> >        if tempfiles
> >          tempfiles.each { |tmp| tmp.close! rescue nil }
> >        end
> >      end
> 
> By "the Rack server should call this" do you mean that if the server
> doesn't call this, tempfiles Rack creates will not be cleaned up until
> GC runs?

Yes, a Rack-compliant server should call body.close if it responds to
body.close at the end of the response cycle for every individual
request.

I put this in the wrapped body because body.each {} could be relying on
the tempfiles to generate the response.  body.close is the absolute last
action for any Rack application.

> Shouldn't Rack itself be guaranteeing it doesn't leave garbage files around?
> 
> >      # wrap the normal application call, saving env
> >      def _call(env)
> >        self.env = env
> >
> >        # XXX VERY IMPORTANT:
> >        # you need to ensure env stays the same throughout the request,
> >        # some middlewares overwrite/replace it instead of merge!-ing into it
> >        status, headers, body = app.call(env)
> >        self.body = body
> >        [ status, headers, self ]
> >      end
> >    end
> 
> Same question as above...can badly-written middleware now cause
> tempfiles to linger?

Yes, it might be safer to do this before app.call above:

   env["rack.tempfiles"] ||= []

That way if env gets replaced down the stack, e.g. via:

   app.call(env.merge("foo.hello" => "world"))
   # env.merge! would be correct above

any use of env["rack.tempfiles"] will still point to the same array.
Well, almost...

Then again some bad middleware could do:
   env["rack.tempfiles"] += [ tmp_a, tmp_b ]

Instead of what they _should_ do:
   env["rack.tempfiles"].concat([ tmp_a, tmp_b ])

So yes, discourage future middleware authors from replacing
the rack.tempfiles array.

-- 
Eric Wong