Bloat?

Bloat?

[Matt Todd]

[-- Attachment #1: Type: text/plain, Size: 635 bytes --]

I was talking to a friend of mine yesterday and he mentioned that thought
the Rack package itself seemed to be slightly bloated by things like Basic
Auth et al. He mentioned two things I thought were interesting and I wanted
to get your feedback on it:

1. Like Merb, Rack probably could benefit from using a core and more
separation of functionality, and
2. Rack core should only include what's necessary for Rack::Lint to validate
a basic application at minimum.

Thoughts?

Matt

-- 
Matt Todd
Highgroove Studios
www.highgroove.com
cell: 404-314-2612
blog: maraby.org

Scout - Web Monitoring and Reporting Software
www.scoutapp.com

[-- Attachment #2: Type: text/html, Size: 885 bytes --]

Re: Bloat?

[Yehuda Katz]

[-- Attachment #1: Type: text/plain, Size: 1747 bytes --]

I think Rack::Request and Rack::Response are convenient, but they need 
to undergo a fairly extensive performance audit if they are to remain in 
rack-core.

You might have seen that I did a bunch of performance work on Rails 3, 
and I got a lot of improvement by moving pieces of Rack::Response down 
into ActionDispatch::Response and rewriting them for performance. For 
instance, if people want a HeaderHash, they should probably have to 
specify that. Right now, there's no way to super to 
Rack::Response#initialize and avoid creating a fairly expensive HeaderHash.

My approach would be that Rack::Response would be extremely simple and 
performant, and that if people wanted to add special functionality to 
it, they should do it themselves (and possibly call super). I'd be happy 
to supply patches if people were interested, but I'm not sure if 
everything could be 100% compatible with the existing Response. Which 
frameworks are using Rack::Response?

-- Yehdua

Matt Todd wrote:
> I was talking to a friend of mine yesterday and he mentioned that 
> thought the Rack package itself seemed to be slightly bloated by 
> things like Basic Auth et al. He mentioned two things I thought were 
> interesting and I wanted to get your feedback on it:
>
> 1. Like Merb, Rack probably could benefit from using a core and more 
> separation of functionality, and
> 2. Rack core should only include what's necessary for Rack::Lint to 
> validate a basic application at minimum.
>
> Thoughts?
>
> Matt
>
> -- 
> Matt Todd
> Highgroove Studios
> www.highgroove.com <http://www.highgroove.com>
> cell: 404-314-2612
> blog: maraby.org <http://maraby.org>
>
> Scout - Web Monitoring and Reporting Software
> www.scoutapp.com <http://www.scoutapp.com>

[-- Attachment #2: Type: text/html, Size: 2318 bytes --]

Re: Bloat?

[James Tucker]

[-- Attachment #1: Type: text/plain, Size: 3835 bytes --]

raggi@mbk: ~/dev/ext/rack % du -sh lib/
296K	lib/

And...

raggi@mbk: ~/dev/ext/rack % g grep autoload
bin/rackup:autoload :Rack, 'rack'
lib/rack.rb:# All modules meant for use in your application are  
<tt>autoload</tt>ed here,
lib/rack.rb:  autoload :Builder, "rack/builder"
lib/rack.rb:  autoload :Cascade, "rack/cascade"
lib/rack.rb:  autoload :Chunked, "rack/chunked"
lib/rack.rb:  autoload :CommonLogger, "rack/commonlogger"
lib/rack.rb:  autoload :ConditionalGet, "rack/conditionalget"
lib/rack.rb:  autoload :Const, "rack/constants"
lib/rack.rb:  autoload :ContentLength, "rack/content_length"
lib/rack.rb:  autoload :ContentType, "rack/content_type"
lib/rack.rb:  autoload :File, "rack/file"
lib/rack.rb:  autoload :Deflater, "rack/deflater"
lib/rack.rb:  autoload :Directory, "rack/directory"
lib/rack.rb:  autoload :ForwardRequest, "rack/recursive"
lib/rack.rb:  autoload :Handler, "rack/handler"
lib/rack.rb:  autoload :Head, "rack/head"
lib/rack.rb:  autoload :Lint, "rack/lint"
lib/rack.rb:  autoload :Lock, "rack/lock"
lib/rack.rb:  autoload :MethodOverride, "rack/methodoverride"
lib/rack.rb:  autoload :Mime, "rack/mime"
lib/rack.rb:  autoload :Recursive, "rack/recursive"
lib/rack.rb:  autoload :Reloader, "rack/reloader"
lib/rack.rb:  autoload :ShowExceptions, "rack/showexceptions"
lib/rack.rb:  autoload :ShowStatus, "rack/showstatus"
lib/rack.rb:  autoload :Static, "rack/static"
lib/rack.rb:  autoload :URLMap, "rack/urlmap"
lib/rack.rb:  autoload :Utils, "rack/utils"
lib/rack.rb:  autoload :MockRequest, "rack/mock"
lib/rack.rb:  autoload :MockResponse, "rack/mock"
lib/rack.rb:  autoload :Request, "rack/request"
lib/rack.rb:  autoload :Response, "rack/response"
lib/rack.rb:    autoload :Basic, "rack/auth/basic"
lib/rack.rb:    autoload :AbstractRequest, "rack/auth/abstract/request"
lib/rack.rb:    autoload :AbstractHandler, "rack/auth/abstract/handler"
lib/rack.rb:    autoload :OpenID, "rack/auth/openid"
lib/rack.rb:      autoload :MD5, "rack/auth/digest/md5"
lib/rack.rb:      autoload :Nonce, "rack/auth/digest/nonce"
lib/rack.rb:      autoload :Params, "rack/auth/digest/params"
lib/rack.rb:      autoload :Request, "rack/auth/digest/request"
lib/rack.rb:    autoload :Cookie, "rack/session/cookie"
lib/rack.rb:    autoload :Pool, "rack/session/pool"
lib/rack.rb:    autoload :Memcache, "rack/session/memcache"
lib/rack.rb:    autoload :Camping, "rack/adapter/camping"
lib/rack/handler.rb:    autoload :CGI, "rack/handler/cgi"
lib/rack/handler.rb:    autoload :FastCGI, "rack/handler/fastcgi"
lib/rack/handler.rb:    autoload :Mongrel, "rack/handler/mongrel"
lib/rack/handler.rb:    autoload :EventedMongrel, "rack/handler/ 
evented_mongrel"
lib/rack/handler.rb:    autoload :SwiftipliedMongrel, "rack/handler/ 
swiftiplied_mongrel"
lib/rack/handler.rb:    autoload :WEBrick, "rack/handler/webrick"
lib/rack/handler.rb:    autoload :LSWS, "rack/handler/lsws"
lib/rack/handler.rb:    autoload :SCGI, "rack/handler/scgi"
lib/rack/handler.rb:    autoload :Thin, "rack/handler/thin"


I don't think we have a bloat problem. I think you're judging bloat  
the wrong way.


On 13 Aug 2009, at 18:04, Matt Todd wrote:

> I was talking to a friend of mine yesterday and he mentioned that  
> thought the Rack package itself seemed to be slightly bloated by  
> things like Basic Auth et al. He mentioned two things I thought were  
> interesting and I wanted to get your feedback on it:
>
> 1. Like Merb, Rack probably could benefit from using a core and more  
> separation of functionality, and
> 2. Rack core should only include what's necessary for Rack::Lint to  
> validate a basic application at minimum.
>
> Thoughts?
>
> Matt
>
> -- 
> Matt Todd
> Highgroove Studios
> www.highgroove.com
> cell: 404-314-2612
> blog: maraby.org
>
> Scout - Web Monitoring and Reporting Software
> www.scoutapp.com


[-- Attachment #2: Type: text/html, Size: 5344 bytes --]

Re: Bloat?

[Joshua Peek]

Here are a few candidates I'd like to see moved into rack-contrib.

Adapter::Camping
Auth::OpenID
ContentType
Recursive
Session::Memcache

On Thu, Aug 13, 2009 at 12:04 PM, Matt Todd<chiology@gmail.com> wrote:
> I was talking to a friend of mine yesterday and he mentioned that thought
> the Rack package itself seemed to be slightly bloated by things like Basic
> Auth et al. He mentioned two things I thought were interesting and I wanted
> to get your feedback on it:
> 1. Like Merb, Rack probably could benefit from using a core and more
> separation of functionality, and
> 2. Rack core should only include what's necessary for Rack::Lint to validate
> a basic application at minimum.
> Thoughts?
> Matt
> --
> Matt Todd
> Highgroove Studios
> www.highgroove.com
> cell: 404-314-2612
> blog: maraby.org
>
> Scout - Web Monitoring and Reporting Software
> www.scoutapp.com
>



-- 
Joshua Peek

Re: Bloat?

[James Tucker]

[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]


On 13 Aug 2009, at 18:04, Matt Todd wrote:

> I was talking to a friend of mine yesterday and he mentioned that  
> thought the Rack package itself seemed to be slightly bloated by  
> things like Basic Auth et al. He mentioned two things I thought were  
> interesting and I wanted to get your feedback on it:
>
> 1. Like Merb, Rack probably could benefit from using a core and more  
> separation of functionality, and

On this point, I have massive trouble agreeing. In recent months, the  
disparity of supported versions of thin/rack/sinatra has been a big  
problem for some folks (integrity mainly). Splitting up into a crazy  
number of gems causes a lot of management overhead and sysadmin /  
deployment crap that I almost never see done right. The only good  
solution is the recently released bundler system by wycats, halogrium  
and carllerche. Requiring something like that to "get by" is far far  
far worse in my opinion.

We're not as fat as you might think, because stuff you don't touch is  
never loaded. Do you have a genuine use case to drop 100K or so out of  
core, by adding ~500K (or more) to deployment overhead of *each*  
composite part?

> 2. Rack core should only include what's necessary for Rack::Lint to  
> validate a basic application at minimum.
>
> Thoughts?
>
> Matt
>
> -- 
> Matt Todd
> Highgroove Studios
> www.highgroove.com
> cell: 404-314-2612
> blog: maraby.org
>
> Scout - Web Monitoring and Reporting Software
> www.scoutapp.com


[-- Attachment #2: Type: text/html, Size: 1956 bytes --]

Re: Bloat?

[James Tucker]

On 13 Aug 2009, at 18:49, Joshua Peek wrote:

>
> Here are a few candidates I'd like to see moved into rack-contrib.
>
> Adapter::Camping
> Auth::OpenID
> ContentType
> Recursive
> Session::Memcache

Whilst I sort of agree, doesn't this just shift the bloat problem  
somewhere else?

>
> On Thu, Aug 13, 2009 at 12:04 PM, Matt Todd<chiology@gmail.com> wrote:
>> I was talking to a friend of mine yesterday and he mentioned that  
>> thought
>> the Rack package itself seemed to be slightly bloated by things  
>> like Basic
>> Auth et al. He mentioned two things I thought were interesting and  
>> I wanted
>> to get your feedback on it:
>> 1. Like Merb, Rack probably could benefit from using a core and more
>> separation of functionality, and
>> 2. Rack core should only include what's necessary for Rack::Lint to  
>> validate
>> a basic application at minimum.
>> Thoughts?
>> Matt
>> --
>> Matt Todd
>> Highgroove Studios
>> www.highgroove.com
>> cell: 404-314-2612
>> blog: maraby.org
>>
>> Scout - Web Monitoring and Reporting Software
>> www.scoutapp.com
>>
>
>
>
> -- 
> Joshua Peek

Re: Bloat?

[Adrian Madrid]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]

I don't think that rack is bloated considering the bloat is only
self-inflicted (autoload as mentioned) but Yahuda brings another point:
performance. I would love to have response more performant for everyone to
use.

Adrian Madrid
My eBiz, Developer
3082 W. Maple Loop Dr
Lehi, UT 84043
801-341-3824


On Thu, Aug 13, 2009 at 11:50, James Tucker <jftucker@gmail.com> wrote:

>
>
> On 13 Aug 2009, at 18:49, Joshua Peek wrote:
>
>
>> Here are a few candidates I'd like to see moved into rack-contrib.
>>
>> Adapter::Camping
>> Auth::OpenID
>> ContentType
>> Recursive
>> Session::Memcache
>>
>
> Whilst I sort of agree, doesn't this just shift the bloat problem somewhere
> else?
>
>
>
>> On Thu, Aug 13, 2009 at 12:04 PM, Matt Todd<chiology@gmail.com> wrote:
>>
>>> I was talking to a friend of mine yesterday and he mentioned that thought
>>> the Rack package itself seemed to be slightly bloated by things like
>>> Basic
>>> Auth et al. He mentioned two things I thought were interesting and I
>>> wanted
>>> to get your feedback on it:
>>> 1. Like Merb, Rack probably could benefit from using a core and more
>>> separation of functionality, and
>>> 2. Rack core should only include what's necessary for Rack::Lint to
>>> validate
>>> a basic application at minimum.
>>> Thoughts?
>>> Matt
>>> --
>>> Matt Todd
>>> Highgroove Studios
>>> www.highgroove.com
>>> cell: 404-314-2612
>>> blog: maraby.org
>>>
>>> Scout - Web Monitoring and Reporting Software
>>> www.scoutapp.com
>>>
>>>
>>
>>
>> --
>> Joshua Peek
>>
>
>

[-- Attachment #2: Type: text/html, Size: 2537 bytes --]

Re: Bloat?

[James Tucker]

[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]


On 13 Aug 2009, at 18:55, Adrian Madrid wrote:

> I don't think that rack is bloated considering the bloat is only  
> self-inflicted (autoload as mentioned) but Yahuda brings another  
> point: performance. I would love to have response more performant  
> for everyone to use.

As long as it's profiled off of real style applications, and profile  
guided, sure, that would be lovely.

>
>
> Adrian Madrid
> My eBiz, Developer
> 3082 W. Maple Loop Dr
> Lehi, UT 84043
> 801-341-3824
>
>
> On Thu, Aug 13, 2009 at 11:50, James Tucker <jftucker@gmail.com>  
> wrote:
>
>
> On 13 Aug 2009, at 18:49, Joshua Peek wrote:
>
>
> Here are a few candidates I'd like to see moved into rack-contrib.
>
> Adapter::Camping
> Auth::OpenID
> ContentType
> Recursive
> Session::Memcache
>
> Whilst I sort of agree, doesn't this just shift the bloat problem  
> somewhere else?
>
>
>
> On Thu, Aug 13, 2009 at 12:04 PM, Matt Todd<chiology@gmail.com> wrote:
> I was talking to a friend of mine yesterday and he mentioned that  
> thought
> the Rack package itself seemed to be slightly bloated by things like  
> Basic
> Auth et al. He mentioned two things I thought were interesting and I  
> wanted
> to get your feedback on it:
> 1. Like Merb, Rack probably could benefit from using a core and more
> separation of functionality, and
> 2. Rack core should only include what's necessary for Rack::Lint to  
> validate
> a basic application at minimum.
> Thoughts?
> Matt
> --
> Matt Todd
> Highgroove Studios
> www.highgroove.com
> cell: 404-314-2612
> blog: maraby.org
>
> Scout - Web Monitoring and Reporting Software
> www.scoutapp.com
>
>
>
>
> -- 
> Joshua Peek
>
>


[-- Attachment #2: Type: text/html, Size: 2927 bytes --]

Re: Bloat?

[Ryan Tomayko]

On Thu, Aug 13, 2009 at 10:04 AM, Matt Todd<chiology@gmail.com> wrote:
> I was talking to a friend of mine yesterday and he mentioned that thought
> the Rack package itself seemed to be slightly bloated by things like Basic
> Auth et al. He mentioned two things I thought were interesting and I wanted
> to get your feedback on it:
>
> 1. Like Merb, Rack probably could benefit from using a core and more
> separation of functionality, and

Matt, that's just crazy talk.

> 2. Rack core should only include what's necessary for Rack::Lint to validate
> a basic application at minimum.

You actually don't need Rack the library to write Rack apps. The Rack
SPEC uses only core Ruby data structures to define the basic protocol.
Rack would include only Rack::Lint at that point.

Thanks,
Ryan

Re: Bloat?

[Matt Todd]

[-- Attachment #1: Type: text/plain, Size: 2481 bytes --]

Great feedback!
1. Lot of disagreement here, and lots of valid points. To clarify, the point
wasn't runtime bloat (which is not a problem with Rack) but with the overall
package bloat. I tend to agree that runtime bloat is much more severe than
package bloat, and that Rack is hardly bloated even considering.

The question needs to be asked, though... are there things in Rack's library
that could be better elsewhere? I mention rack-more, and many don't like
this, which is fine. I think of rack-contrib to be much like rack-more
already, and like that some things are there instead of in Rack's core
library.

2. The point of this was to elaborate on the question "are there things in
Rack's library that could be better elsewhere?" by starting with the
minimum. Rack::Lint is, as Ryan points at, the wrong starting point because
we still want to support middleware and all of the server handlers and
framework adapters as well as Rack::Request and Rack::Response.

I think Yehuda understood this with his recommendations on improvements to
these last two. What else can we minimize and defer to the implementers
using Rack instead of doing it ourselves?

Should Rack concern itself with much more than a minimal implementation
covering these handlers, adapters, request/response interfaces, and
middleware functionality?

No dictating or anything quite so grave here, just raising the discussion.

Cheers!

Matt



On Thu, Aug 13, 2009 at 3:01 PM, Ryan Tomayko <r@tomayko.com> wrote:

>
> On Thu, Aug 13, 2009 at 10:04 AM, Matt Todd<chiology@gmail.com> wrote:
> > I was talking to a friend of mine yesterday and he mentioned that thought
> > the Rack package itself seemed to be slightly bloated by things like
> Basic
> > Auth et al. He mentioned two things I thought were interesting and I
> wanted
> > to get your feedback on it:
> >
> > 1. Like Merb, Rack probably could benefit from using a core and more
> > separation of functionality, and
>
> Matt, that's just crazy talk.
>
> > 2. Rack core should only include what's necessary for Rack::Lint to
> validate
> > a basic application at minimum.
>
> You actually don't need Rack the library to write Rack apps. The Rack
> SPEC uses only core Ruby data structures to define the basic protocol.
> Rack would include only Rack::Lint at that point.
>
> Thanks,
> Ryan
>



-- 
Matt Todd
Highgroove Studios
www.highgroove.com
cell: 404-314-2612
blog: maraby.org

Scout - Web Monitoring and Reporting Software
www.scoutapp.com

[-- Attachment #2: Type: text/html, Size: 3366 bytes --]

Re: Bloat?

[Christian Neukirchen]

Joshua Peek <josh@joshpeek.com> writes:

> Here are a few candidates I'd like to see moved into rack-contrib.
>
> Adapter::Camping

This one is gone when Camping 2.0 is released anyway.

> Auth::OpenID
> Session::Memcache

Alright.

> ContentType
> Recursive

These are so small and general purpose that I'd rather keep them in core.

-- 
Christian Neukirchen  <chneukirchen@gmail.com>  http://chneukirchen.org

Re: Bloat?

[James Tucker]

On 13 Aug 2009, at 20:19, Matt Todd wrote:

> Great feedback!
>
> 1. Lot of disagreement here, and lots of valid points. To clarify,  
> the point wasn't runtime bloat (which is not a problem with Rack)  
> but with the overall package bloat. I tend to agree that runtime  
> bloat is much more severe than package bloat, and that Rack is  
> hardly bloated even considering.
>
> The question needs to be asked, though... are there things in Rack's  
> library that could be better elsewhere? I mention rack-more, and  
> many don't like this, which is fine. I think of rack-contrib to be  
> much like rack-more already, and like that some things are there  
> instead of in Rack's core library.

Certainly, and I think the key question is usage frequency. The  
important consideration is that running two gems is significantly more  
overhead than a single gem. The observation is that if the most common  
use case after such changes involves running two or more gems, then  
the changes have introduced storage, packaging, maintenance, and  
runtime bloat. Loading gems (on 1.8) is not at all 'bloat free', in  
fact, if you check, you'll find that the gemspec details when loaded,  
live for quite an extended period of time in the application boot and  
run cycle. I wrote about this back in 2007, and to date, it has not  
changed (for 1.8). On 1.9 the gem_prelude shim can aid the situation,  
however, when there are gems in the system that use file discovery  
plugin loads, then all these gains are lost, too.

> 2. The point of this was to elaborate on the question "are there  
> things in Rack's library that could be better elsewhere?" by  
> starting with the minimum. Rack::Lint is, as Ryan points at, the  
> wrong starting point because we still want to support middleware and  
> all of the server handlers and framework adapters as well as  
> Rack::Request and Rack::Response.
>
> I think Yehuda understood this with his recommendations on  
> improvements to these last two. What else can we minimize and defer  
> to the implementers using Rack instead of doing it ourselves?

I agree, however, as I've otherwise said, it's also important to  
consider real world profiles for such improvements. If the real  
improvements are minor, but the change in complexity is high, then  
time might be better spent donating to the core language than to micro- 
optimising frameworks that are currently very simple and easy to  
understand and maintain.

> Should Rack concern itself with much more than a minimal  
> implementation covering these handlers, adapters, request/response  
> interfaces, and middleware functionality?

I can only re-iterate the common use case statement, and I won't claim  
to know what that is, but I will make keen the observation that "most  
users" don't tend to be signed up to every mailinglist, so polling for  
such information may be difficult - this aspect may require further  
thought and discussion.

> No dictating or anything quite so grave here, just raising the  
> discussion.

Absolutely, discussion is worthwhile.

Re: Bloat?

[Kyle Drake]

I like having a rich set of rack components available, and I think
splitting it up would just make it harder for newcomers to use Rack
because they would have to dig around for the parts needed to make it
useful. If these files all got loaded on runtime I would agree with
you, but as long as they aren't all used by default I'm content with
keeping all the components in one gem install. One of my favorite
things about Rack was getting a ton of cool toys to play with out of
the box, it's one of the principal things that really sold me on it.

-Kyle

On Thu, Aug 13, 2009 at 10:04 AM, Matt Todd<chiology@gmail.com> wrote:
> I was talking to a friend of mine yesterday and he mentioned that thought
> the Rack package itself seemed to be slightly bloated by things like Basic
> Auth et al. He mentioned two things I thought were interesting and I wanted
> to get your feedback on it:
> 1. Like Merb, Rack probably could benefit from using a core and more
> separation of functionality, and
> 2. Rack core should only include what's necessary for Rack::Lint to validate
> a basic application at minimum.
> Thoughts?
> Matt
> --
> Matt Todd
> Highgroove Studios
> www.highgroove.com
> cell: 404-314-2612
> blog: maraby.org
>
> Scout - Web Monitoring and Reporting Software
> www.scoutapp.com
>