Hi Tobi,

Both %c3%a9 and %e9 could be valid -- it depends on the content of the page
and what the server is prepared to accept. In short, a standards-conforming
browser will choose which encoding to use based (mainly) on the form's
accept-charset attribute. You can read the details here:

http://www.w3.org/TR/html5/association-of-controls-and-forms.html#url-encoded-form-data

Also, note that while UTF-8 is indeed a character encoding, "Unicode" is a
standard, not a character encoding itself. As such, it doesn't make sense to
talk about é being "encoded in Unicode". It is true that é is the Unicode
code point U+00E9, but code points are independent of encodings (they are
just an abstract numeric identifiers for a particular characters). You
*can*say that é is encoded in ISO-8859-1 (and other common encodings)
as hex E9,
however. (See here for more:
http://www.joelonsoftware.com/articles/Unicode.html)

So, with a form that specifies accept-charset="utf-8", é would be escaped
as %c3%a9. With a form that specifies accept-charset="iso-8859-1", it would
be escaped as %e9. Leaving it unescaped is technically invalid.

In order to process the data correctly, the server must know what encoding
the incoming URL parameters are encoded in, typically either by convention
or via HTTP headers. Rack has a deficiency here, in that
Rack::Utils.unescape does not allow you to specify the encoding -- only
UTF-8 is supported. The underlying API used by Rack,
URI.decode_www_form_component, does allow you to specify the encoding, so if
that matters to you, you might have to use that directly. (Though as was
discovered recently, it has a rexexp that is vulnerable to catastrophic
backtracking.)

John

On Mon, Jul 25, 2011 at 7:22 AM, Tobias Bielohlawek <tobi@soundcloud.com>wrote:

> I have a question on correct behavior of handling unicode encoded
> query params.
>
> Let's take character 'é', in UTF-8 that's encoded '%c3%a9 ' in unicode
> '%e9'. As an real life example, we're seeing those URLs:
>
> 1. not escaped: -> http://soundcloud.com/search?q%5Bfulltext%5D=café<http://soundcloud.com/search?q%5Bfulltext%5D=caf%C3%A9>
> 2. UTF8 escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%c3%a9
> 3. or Unicode escaped ->
> http://soundcloud.com/search?q%5Bfulltext%5D=caf%E9
>
> Using rack 1.3.1, the first two cases are processed correct, but the
> latter fails with error 'incorrect UTF-8 byte sequence'. Which is
> correct behavior at first place as it's
> not UTF-8 but the unicode.
>
> I'm no wondering what's best solution here? Why not be smart and give
> it another run to unescape the URL assuming it's unicode encoded?
>
> I checked other players, e.g. Twitter has same error (but fails
> silently)
> http://twitter.com/search/é
> http://twitter.com/search/%c3%a9
> http://twitter.com/search/%E9
>
> but not Google, it does it this way:
> http://www.google.de/search?q=é
> http://www.google.de/search?q=%c3%a9
> http://www.google.de/search?q=%E9
>
>
> Any advice is very much appreciated. Patch is comming..
>
> Thx - Tobi
>
>
>