PATH_INFO spec (with regard to ";")

PATH_INFO spec (with regard to ";")

[Eric Wong]

Hi all,

I've been notified privately that my changes for PATH_INFO in Unicorn
0.95.2 (which also got into Thin) may not be completely kosher, but I'm
also asking for the Rack team to clarify PATH_INFO for HTTP parser
implementers.


Upon further reading (and also of the
related-but-not-necessarily-true-for-Rack RFC 3875 section 4.1.5),
I came across this:

   Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
   contain path-segment parameters.

First off, Rack already directly contradicts the "the PATH_INFO is not
URL-encoded" part, so Unicorn conforms to Rack specs over RFC 3875.

*But* Rack does not address the "cannot contain path-segment parameters"
part at all.  So I (and probably a few other people) would like
clarification on how to handle PATH_INFO when it comes to ";"


Things to keep in mind:

  * URI.parse keeps ";" in URI::HTTP#path
    This point may not be relevant to us, as PATH_INFO and
    URI::HTTP#path should not necessarily be treated as equals

  * WEBrick keeps ";" in PATH_INFO

  * PEP333 (which Rack is based on) does not go into this level of
    detail regarding PATH_INFO and path segments

  * PATH_INFO in Rack appears to be based on CGI/1.1 (RFC 3875)

  * Again, Rack already contradicts the URL encoding rules of RFC 3875
    for PATH_INFO, so there is precedence for Rack contradicting more
    of RFC 3875...

  * Rack::Request#full_path only looks at PATH_INFO + QUERY_STRING,
    this means many Rack applications may never see the ";" parts
    if Thin and Unicorn revert to old behavior.

  * Rack does not require REQUEST_URI, this is an extension Unicorn
    and Thin both carried over from Mongrel.

  * None of the official rack/rack-contrib middleware use REQUEST_URI


Of course, in the grand scheme of things, hardly anybody uses ";" in
paths.  Yay for rare corner cases making our lives difficult.

-- 
Eric Wong

Re: PATH_INFO spec (with regard to ";")

[James Tucker]

[-- Attachment #1: Type: text/plain, Size: 2489 bytes --]


On 10 Dec 2009, at 22:30, Eric Wong wrote:

> Hi all,
> 
> I've been notified privately that my changes for PATH_INFO in Unicorn
> 0.95.2 (which also got into Thin) may not be completely kosher, but I'm
> also asking for the Rack team to clarify PATH_INFO for HTTP parser
> implementers.
> 
> 
> Upon further reading (and also of the
> related-but-not-necessarily-true-for-Rack RFC 3875 section 4.1.5),
> I came across this:
> 
>   Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
>   contain path-segment parameters.
> 
> First off, Rack already directly contradicts the "the PATH_INFO is not
> URL-encoded" part, so Unicorn conforms to Rack specs over RFC 3875.
> 
> *But* Rack does not address the "cannot contain path-segment parameters"
> part at all.  So I (and probably a few other people) would like
> clarification on how to handle PATH_INFO when it comes to ";"
> 
> 
> Things to keep in mind:
> 
>  * URI.parse keeps ";" in URI::HTTP#path
>    This point may not be relevant to us, as PATH_INFO and
>    URI::HTTP#path should not necessarily be treated as equals
> 
>  * WEBrick keeps ";" in PATH_INFO
> 
>  * PEP333 (which Rack is based on) does not go into this level of
>    detail regarding PATH_INFO and path segments
> 
>  * PATH_INFO in Rack appears to be based on CGI/1.1 (RFC 3875)
> 
>  * Again, Rack already contradicts the URL encoding rules of RFC 3875
>    for PATH_INFO, so there is precedence for Rack contradicting more
>    of RFC 3875...
> 
>  * Rack::Request#full_path only looks at PATH_INFO + QUERY_STRING,
>    this means many Rack applications may never see the ";" parts
>    if Thin and Unicorn revert to old behavior.
> 
>  * Rack does not require REQUEST_URI, this is an extension Unicorn
>    and Thin both carried over from Mongrel.
> 
>  * None of the official rack/rack-contrib middleware use REQUEST_URI

* http://twistedmatrix.com/trac/ticket/1361

* http://jira.codehaus.org/browse/JETTY-980 (although not directly relevant, may be food for thought)

* http://ksi.cpsc.ucalgary.ca/archives/WWW-TALK/www-talk-1994q1.messages/3.html (ancient history)

* http://code.google.com/p/googleappengine/issues/detail?id=1340

> Of course, in the grand scheme of things, hardly anybody uses ";" in
> paths.  Yay for rare corner cases making our lives difficult.

Amen, because I feel like I want to say "be configurable", given all of the above.

> 
> -- 
> Eric Wong


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3679 bytes --]

Re: PATH_INFO spec (with regard to ";")

[Iñaki Baz Castillo]

El Viernes, 11 de Diciembre de 2009, James Tucker escribió:
> > Of course, in the grand scheme of things, hardly anybody uses ";" in
> > paths.  Yay for rare corner cases making our lives difficult.
> 
> Amen, because I feel like I want to say "be configurable", given all of the
>  above.

I'm coding a XCAP server using Rack. XCAP is basically XPath over HTTP in 
which there are HTTP URI's like the following:

 /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d

After reading RFC for URI and HTTP I think that the ";" is valid into the 
above URI. I also asked in a XCAP maillist and received same conclusion.

Please tell me if you consider the ";" in the above URI wrong.

Regards.



-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: PATH_INFO spec (with regard to ";")

[James Tucker]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]


On 11 Dec 2009, at 00:19, Iñaki Baz Castillo wrote:

> El Viernes, 11 de Diciembre de 2009, James Tucker escribió:
>>> Of course, in the grand scheme of things, hardly anybody uses ";" in
>>> paths.  Yay for rare corner cases making our lives difficult.
>> 
>> Amen, because I feel like I want to say "be configurable", given all of the
>> above.
> 
> I'm coding a XCAP server using Rack. XCAP is basically XPath over HTTP in 
> which there are HTTP URI's like the following:
> 
> /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d
> 
> After reading RFC for URI and HTTP I think that the ";" is valid into the 
> above URI. I also asked in a XCAP maillist and received same conclusion.
> 
> Please tell me if you consider the ";" in the above URI wrong.

"Wrong" is hard to define with archaic protocol specifications that do not match "common" usage patterns, which is why I suggested it should be configurable. I would probably suggest to you that you make your client configurable to escape it's semi-colon characters in the uri that are not part of query syntax.

> 
> Regards.
> 
> 
> 
> -- 
> Iñaki Baz Castillo <ibc@aliax.net>


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3679 bytes --]

Re: PATH_INFO spec (with regard to ";")

[Iñaki Baz Castillo]

El Domingo, 13 de Diciembre de 2009, James Tucker escribió:
> On 11 Dec 2009, at 00:19, Iñaki Baz Castillo wrote:
> > El Viernes, 11 de Diciembre de 2009, James Tucker escribió:
> >>> Of course, in the grand scheme of things, hardly anybody uses ";" in
> >>> paths.  Yay for rare corner cases making our lives difficult.
> >>
> >> Amen, because I feel like I want to say "be configurable", given all of
> >> the above.
> >
> > I'm coding a XCAP server using Rack. XCAP is basically XPath over HTTP in
> > which there are HTTP URI's like the following:
> >
> > /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d
> >
> > After reading RFC for URI and HTTP I think that the ";" is valid into the
> > above URI. I also asked in a XCAP maillist and received same conclusion.
> >
> > Please tell me if you consider the ";" in the above URI wrong.
> 
> "Wrong" is hard to define with archaic protocol specifications that do not
>  match "common" usage patterns, which is why I suggested it should be
>  configurable. I would probably suggest to you that you make your client
>  configurable to escape it's semi-colon characters in the uri that are not
>  part of query syntax.

Yes, modifications in the client could be an option. However I cannot force to 
every XCAP clients to escape ";" when it's allowed as per URI BNF grammar.


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: PATH_INFO spec (with regard to ";")

[macournoyer]

"not completely kosher" as in security issue? Should I revert this
change in Thin?

On Dec 10, 5:30 pm, Eric Wong <normalper...@yhbt.net> wrote:
> Hi all,
>
> I've been notified privately that my changes for PATH_INFO in Unicorn
> 0.95.2 (which also got into Thin) may not be completely kosher, but I'm
> also asking for the Rack team to clarify PATH_INFO for HTTP parser
> implementers.
>
> Upon further reading (and also of the
> related-but-not-necessarily-true-for-Rack RFC 3875 section 4.1.5),
> I came across this:
>
>    Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
>    contain path-segment parameters.
>
> First off, Rack already directly contradicts the "the PATH_INFO is not
> URL-encoded" part, so Unicorn conforms to Rack specs over RFC 3875.
>
> *But* Rack does not address the "cannot contain path-segment parameters"
> part at all.  So I (and probably a few other people) would like
> clarification on how to handle PATH_INFO when it comes to ";"
>
> Things to keep in mind:
>
>   * URI.parse keeps ";" in URI::HTTP#path
>     This point may not be relevant to us, as PATH_INFO and
>     URI::HTTP#path should not necessarily be treated as equals
>
>   * WEBrick keeps ";" in PATH_INFO
>
>   * PEP333 (which Rack is based on) does not go into this level of
>     detail regarding PATH_INFO and path segments
>
>   * PATH_INFO in Rack appears to be based on CGI/1.1 (RFC 3875)
>
>   * Again, Rack already contradicts the URL encoding rules of RFC 3875
>     for PATH_INFO, so there is precedence for Rack contradicting more
>     of RFC 3875...
>
>   * Rack::Request#full_path only looks at PATH_INFO + QUERY_STRING,
>     this means many Rack applications may never see the ";" parts
>     if Thin and Unicorn revert to old behavior.
>
>   * Rack does not require REQUEST_URI, this is an extension Unicorn
>     and Thin both carried over from Mongrel.
>
>   * None of the official rack/rack-contrib middleware use REQUEST_URI
>
> Of course, in the grand scheme of things, hardly anybody uses ";" in
> paths.  Yay for rare corner cases making our lives difficult.
>
> --
> Eric Wong

Re: PATH_INFO spec (with regard to ";")

[Eric Wong]

macournoyer <macournoyer@gmail.com> wrote:
> "not completely kosher" as in security issue? Should I revert this
> change in Thin?

Not a security issue, but it violates the CGI/1.1 RFC which Rack is
somewhat based on (but also not tied to).

-- 
Eric Wong

Re: PATH_INFO spec (with regard to ";")

[Iñaki Baz Castillo]

El Domingo, 13 de Diciembre de 2009, James Tucker escribió:
> "Wrong" is hard to define with archaic protocol specifications that do not
>  match "common" usage patterns

I've re-checked URI/HTTP grammar and the following URI (containing semicolon) 
is valid for me (perhaps I'm wrong):

  /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d

So, why do you say "archaic protocol specifications that do not match 'common' 
usage patterns"?
I expect that HTTP is not just for web servers.

But as I said I could be wrong. Please if you (or someone) can verify me that 
the semicolon is invalid according to URI/HTTP BNF grammar, then I would 
accept it.

Regards.


-- 
Iñaki Baz Castillo <ibc@aliax.net>

Re: PATH_INFO spec (with regard to ";")

[James Tucker]

On 14 Dec 2009, at 00:44, Iñaki Baz Castillo wrote:

> El Domingo, 13 de Diciembre de 2009, James Tucker escribió:
>> "Wrong" is hard to define with archaic protocol specifications that do not
>> match "common" usage patterns
> 
> I've re-checked URI/HTTP grammar and the following URI (containing semicolon) 
> is valid for me (perhaps I'm wrong):
> 
>  /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d

This appears to be legal according the aggregate grammars in Appendix A of both 2396 and 3986. I don't have time to track down all of the relevant HTTP and CGI rfcs right now, but bug me later and I'll check those too.

> So, why do you say "archaic protocol specifications that do not match 'common' 
> usage patterns"?

Things change, indeed between 2396 and 3986 there were quite some changes to the way segments are split in URIs, however, it does not affect this particular issue.

> I expect that HTTP is not just for web servers.

No comment.

> But as I said I could be wrong. Please if you (or someone) can verify me that 
> the semicolon is invalid according to URI/HTTP BNF grammar, then I would 
> accept it.

According to the URI grammars in RFCs 2396 and 3986 it's fine. As I said above, I have not checked the HTTP and CGI grammars, which may well be different / more restrictive.

> 
> Regards.
> 
> 
> -- 
> Iñaki Baz Castillo <ibc@aliax.net>

Re: PATH_INFO spec (with regard to ";")

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> I'm coding a XCAP server using Rack. XCAP is basically XPath over HTTP in 
> which there are HTTP URI's like the following:
> 
>  /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d
> 
> After reading RFC for URI and HTTP I think that the ";" is valid into the 
> above URI. I also asked in a XCAP maillist and received same conclusion.
> 
> Please tell me if you consider the ";" in the above URI wrong.

Hi Iñaki,

You're missing the point, it's not that ";" is right or wrong.  It's
"right" in that it's always been _accepted_ by the Mongrel family of
HTTP parsers.  That's not the issue.

The issue is just whether or not the PATH_INFO variable can contain ";"
(a path-segment parameter) in it.  RFC 3875 section 4.1.5 states it
cannot:

   Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
   contain path-segment parameters.

However, Rack is not governed by RFC 3875 and already deviates from
it when it comes to the "PATH_INFO is not URL-encoded".

-- 
Eric Wong

Re: PATH_INFO spec (with regard to ";")

[Iñaki Baz Castillo]

El Lunes, 14 de Diciembre de 2009, Eric Wong escribió:
> Iñaki Baz Castillo <ibc@aliax.net> wrote:
> > I'm coding a XCAP server using Rack. XCAP is basically XPath over HTTP in
> > which there are HTTP URI's like the following:
> >
> >  /auid/global/mydoc.xml/~~/ruleset/rule%5b@id=%22sip:dom.org;p=abc%22%5d
> >
> > After reading RFC for URI and HTTP I think that the ";" is valid into the
> > above URI. I also asked in a XCAP maillist and received same conclusion.
> >
> > Please tell me if you consider the ";" in the above URI wrong.
> 
> Hi Iñaki,
> 
> You're missing the point, it's not that ";" is right or wrong.  It's
> "right" in that it's always been _accepted_ by the Mongrel family of
> HTTP parsers.  That's not the issue.
> 
> The issue is just whether or not the PATH_INFO variable can contain ";"
> (a path-segment parameter) in it.  RFC 3875 section 4.1.5 states it
> cannot:
> 
>    Unlike a URI path, the PATH_INFO is not URL-encoded, and cannot
>    contain path-segment parameters.
> 
> However, Rack is not governed by RFC 3875 and already deviates from
> it when it comes to the "PATH_INFO is not URL-encoded".

Ok, thanks a lot for the clarification.




-- 
Iñaki Baz Castillo <ibc@aliax.net>