Hey John

thank you very much for you comprehensive answer. Good you pointed me to ISO-8859-1, stupid I didn't thought of that - that confused me most. So the request params we are seeing are indeed ISO-8859-1 encoded but without any  "accept-charset" param given. For those cases we don't wont to fail, but be smart and try to encode ISO-8859-1:

So on error I try decode again, but with Encoding::ISO8859_1, followed , by iconv to get it to UTF-8
 
str = URI.decode_www_form_component("%E9", Encoding::ISO8859_1)
Iconv.conv("utf-8", "ISO-8859-1", str)

That seams to do the trick. Thanks!

Tobi



On 25.07.2011, at 20:48, John Firebaugh wrote:

Hi Tobi,

Both %c3%a9 and %e9 could be valid -- it depends on the content of the page and what the server is prepared to accept. In short, a standards-conforming browser will choose which encoding to use based (mainly) on the form's accept-charset attribute. You can read the details here:

http://www.w3.org/TR/html5/association-of-controls-and-forms.html#url-encoded-form-data

Also, note that while UTF-8 is indeed a character encoding, "Unicode" is a standard, not a character encoding itself. As such, it doesn't make sense to talk about é being "encoded in Unicode". It is true that é is the Unicode code point U+00E9, but code points are independent of encodings (they are just an abstract numeric identifiers for a particular characters). You can say that é is encoded in ISO-8859-1 (and other common encodings) as hex E9, however. (See here for more: http://www.joelonsoftware.com/articles/Unicode.html)

So, with a form that specifies accept-charset="utf-8", é would be escaped as %c3%a9. With a form that specifies accept-charset="iso-8859-1", it would be escaped as %e9. Leaving it unescaped is technically invalid.

In order to process the data correctly, the server must know what encoding the incoming URL parameters are encoded in, typically either by convention or via HTTP headers. Rack has a deficiency here, in that Rack::Utils.unescape does not allow you to specify the encoding -- only UTF-8 is supported. The underlying API used by Rack,


URI.decode_www_form_component, does allow you to specify the encoding, so if that matters to you, you might have to use that directly. (Though as was discovered recently, it has a rexexp that is vulnerable to catastrophic backtracking.)

John

On Mon, Jul 25, 2011 at 7:22 AM, Tobias Bielohlawek <tobi@soundcloud.com> wrote:
I have a question on correct behavior of handling unicode encoded
query params.

Let's take character 'é', in UTF-8 that's encoded '%c3%a9 ' in unicode
'%e9'. As an real life example, we're seeing those URLs:

1. not escaped: -> http://soundcloud.com/search?q%5Bfulltext%5D=café
2. UTF8 escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%c3%a9
3. or Unicode escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%E9

Using rack 1.3.1, the first two cases are processed correct, but the
latter fails with error 'incorrect UTF-8 byte sequence'. Which is
correct behavior at first place as it's
not UTF-8 but the unicode.

I'm no wondering what's best solution here? Why not be smart and give
it another run to unescape the URL assuming it's unicode encoded?

I checked other players, e.g. Twitter has same error (but fails
silently)
http://twitter.com/search/é
http://twitter.com/search/%c3%a9
http://twitter.com/search/%E9

but not Google, it does it this way:
http://www.google.de/search?q=é
http://www.google.de/search?q=%c3%a9
http://www.google.de/search?q=%E9


Any advice is very much appreciated. Patch is comming..

Thx - Tobi





Cheers,
  Tobi

–––––––––––––––––
Tobias Bielohlawek
Developer, SoundCloud

Mail & gtalk: tobi@soundcloud.com
Rosenthalerstraße 13, 10119 Berlin, Germany

What is SoundCloud? 
http://soundcloud.com/tour

Send me a track?
http://soundcloud.com/hopit/dropbox

Limited registered at Company House, Cardiff, UK. Registered Office: London, UK. Company Number 6343600
Managing Director: Alexander Ljung
Local Branch Office Berlin, Germany: AG Charlottenburg, HRB 110657B