How to handle unicode escaped params?

How to handle unicode escaped params?

[Tobias Bielohlawek]

I have a question on correct behavior of handling unicode encoded
query params.

Let's take character 'é', in UTF-8 that's encoded '%c3%a9 ' in unicode
'%e9'. As an real life example, we're seeing those URLs:

1. not escaped: -> http://soundcloud.com/search?q%5Bfulltext%5D=café
2. UTF8 escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%c3%a9
3. or Unicode escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%E9

Using rack 1.3.1, the first two cases are processed correct, but the
latter fails with error 'incorrect UTF-8 byte sequence'. Which is
correct behavior at first place as it's
not UTF-8 but the unicode.

I'm no wondering what's best solution here? Why not be smart and give
it another run to unescape the URL assuming it's unicode encoded?

I checked other players, e.g. Twitter has same error (but fails
silently)
http://twitter.com/search/é
http://twitter.com/search/%c3%a9
http://twitter.com/search/%E9

but not Google, it does it this way:
http://www.google.de/search?q=é
http://www.google.de/search?q=%c3%a9
http://www.google.de/search?q=%E9


Any advice is very much appreciated. Patch is comming..

Thx - Tobi

Re: How to handle unicode escaped params?

[John Firebaugh]

[-- Attachment #1: Type: text/plain, Size: 3129 bytes --]

Hi Tobi,

Both %c3%a9 and %e9 could be valid -- it depends on the content of the page
and what the server is prepared to accept. In short, a standards-conforming
browser will choose which encoding to use based (mainly) on the form's
accept-charset attribute. You can read the details here:

http://www.w3.org/TR/html5/association-of-controls-and-forms.html#url-encoded-form-data

Also, note that while UTF-8 is indeed a character encoding, "Unicode" is a
standard, not a character encoding itself. As such, it doesn't make sense to
talk about é being "encoded in Unicode". It is true that é is the Unicode
code point U+00E9, but code points are independent of encodings (they are
just an abstract numeric identifiers for a particular characters). You
*can*say that é is encoded in ISO-8859-1 (and other common encodings)
as hex E9,
however. (See here for more:
http://www.joelonsoftware.com/articles/Unicode.html)

So, with a form that specifies accept-charset="utf-8", é would be escaped
as %c3%a9. With a form that specifies accept-charset="iso-8859-1", it would
be escaped as %e9. Leaving it unescaped is technically invalid.

In order to process the data correctly, the server must know what encoding
the incoming URL parameters are encoded in, typically either by convention
or via HTTP headers. Rack has a deficiency here, in that
Rack::Utils.unescape does not allow you to specify the encoding -- only
UTF-8 is supported. The underlying API used by Rack,
URI.decode_www_form_component, does allow you to specify the encoding, so if
that matters to you, you might have to use that directly. (Though as was
discovered recently, it has a rexexp that is vulnerable to catastrophic
backtracking.)

John

On Mon, Jul 25, 2011 at 7:22 AM, Tobias Bielohlawek <tobi@soundcloud.com>wrote:

> I have a question on correct behavior of handling unicode encoded
> query params.
>
> Let's take character 'é', in UTF-8 that's encoded '%c3%a9 ' in unicode
> '%e9'. As an real life example, we're seeing those URLs:
>
> 1. not escaped: -> http://soundcloud.com/search?q%5Bfulltext%5D=café<http://soundcloud.com/search?q%5Bfulltext%5D=caf%C3%A9>
> 2. UTF8 escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%c3%a9
> 3. or Unicode escaped ->
> http://soundcloud.com/search?q%5Bfulltext%5D=caf%E9
>
> Using rack 1.3.1, the first two cases are processed correct, but the
> latter fails with error 'incorrect UTF-8 byte sequence'. Which is
> correct behavior at first place as it's
> not UTF-8 but the unicode.
>
> I'm no wondering what's best solution here? Why not be smart and give
> it another run to unescape the URL assuming it's unicode encoded?
>
> I checked other players, e.g. Twitter has same error (but fails
> silently)
> http://twitter.com/search/é
> http://twitter.com/search/%c3%a9
> http://twitter.com/search/%E9
>
> but not Google, it does it this way:
> http://www.google.de/search?q=é
> http://www.google.de/search?q=%c3%a9
> http://www.google.de/search?q=%E9
>
>
> Any advice is very much appreciated. Patch is comming..
>
> Thx - Tobi
>
>
>

[-- Attachment #2: Type: text/html, Size: 4840 bytes --]

Re: How to handle unicode escaped params?

[Tobias Bielohlawek | SoundCloud]

[-- Attachment #1: Type: text/plain, Size: 4253 bytes --]

Hey John

thank you very much for you comprehensive answer. Good you pointed me to ISO-8859-1, stupid I didn't thought of that - that confused me most. So the request params we are seeing are indeed ISO-8859-1 encoded but without any  "accept-charset" param given. For those cases we don't wont to fail, but be smart and try to encode ISO-8859-1:

So on error I try decode again, but with Encoding::ISO8859_1, followed , by iconv to get it to UTF-8
 
str = URI.decode_www_form_component("%E9", Encoding::ISO8859_1)
Iconv.conv("utf-8", "ISO-8859-1", str)

That seams to do the trick. Thanks!

Tobi



On 25.07.2011, at 20:48, John Firebaugh wrote:

> Hi Tobi,
> 
> Both %c3%a9 and %e9 could be valid -- it depends on the content of the page and what the server is prepared to accept. In short, a standards-conforming browser will choose which encoding to use based (mainly) on the form's accept-charset attribute. You can read the details here:
> 
> http://www.w3.org/TR/html5/association-of-controls-and-forms.html#url-encoded-form-data
> 
> Also, note that while UTF-8 is indeed a character encoding, "Unicode" is a standard, not a character encoding itself. As such, it doesn't make sense to talk about é being "encoded in Unicode". It is true that é is the Unicode code point U+00E9, but code points are independent of encodings (they are just an abstract numeric identifiers for a particular characters). You can say that é is encoded in ISO-8859-1 (and other common encodings) as hex E9, however. (See here for more: http://www.joelonsoftware.com/articles/Unicode.html)
> 
> So, with a form that specifies accept-charset="utf-8", é would be escaped as %c3%a9. With a form that specifies accept-charset="iso-8859-1", it would be escaped as %e9. Leaving it unescaped is technically invalid.
> 
> In order to process the data correctly, the server must know what encoding the incoming URL parameters are encoded in, typically either by convention or via HTTP headers. Rack has a deficiency here, in that Rack::Utils.unescape does not allow you to specify the encoding -- only UTF-8 is supported. The underlying API used by Rack,


> URI.decode_www_form_component, does allow you to specify the encoding, so if that matters to you, you might have to use that directly. (Though as was discovered recently, it has a rexexp that is vulnerable to catastrophic backtracking.)
> 
> John
> 
> On Mon, Jul 25, 2011 at 7:22 AM, Tobias Bielohlawek <tobi@soundcloud.com> wrote:
> I have a question on correct behavior of handling unicode encoded
> query params.
> 
> Let's take character 'é', in UTF-8 that's encoded '%c3%a9 ' in unicode
> '%e9'. As an real life example, we're seeing those URLs:
> 
> 1. not escaped: -> http://soundcloud.com/search?q%5Bfulltext%5D=café
> 2. UTF8 escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%c3%a9
> 3. or Unicode escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%E9
> 
> Using rack 1.3.1, the first two cases are processed correct, but the
> latter fails with error 'incorrect UTF-8 byte sequence'. Which is
> correct behavior at first place as it's
> not UTF-8 but the unicode.
> 
> I'm no wondering what's best solution here? Why not be smart and give
> it another run to unescape the URL assuming it's unicode encoded?
> 
> I checked other players, e.g. Twitter has same error (but fails
> silently)
> http://twitter.com/search/é
> http://twitter.com/search/%c3%a9
> http://twitter.com/search/%E9
> 
> but not Google, it does it this way:
> http://www.google.de/search?q=é
> http://www.google.de/search?q=%c3%a9
> http://www.google.de/search?q=%E9
> 
> 
> Any advice is very much appreciated. Patch is comming..
> 
> Thx - Tobi
> 
> 
> 


Cheers,
  Tobi

–––––––––––––––––
Tobias Bielohlawek
Developer, SoundCloud

Mail & gtalk: tobi@soundcloud.com
Rosenthalerstraße 13, 10119 Berlin, Germany

What is SoundCloud? 
http://soundcloud.com/tour

Send me a track?
http://soundcloud.com/hopit/dropbox

Limited registered at Company House, Cardiff, UK. Registered Office: London, UK. Company Number 6343600
Managing Director: Alexander Ljung
Local Branch Office Berlin, Germany: AG Charlottenburg, HRB 110657B



[-- Attachment #2: Type: text/html, Size: 7978 bytes --]

Re: How to handle unicode escaped params?

[Evgeni Dzhelyov]

I think that why Rails 3 appends the tick character (utf8=✓) into the
forms, to force the browser to send the form data into utf-8.

On Tue, Jul 26, 2011 at 12:07 PM, Tobias Bielohlawek | SoundCloud
<tobi@soundcloud.com> wrote:
> Hey John
> thank you very much for you comprehensive answer. Good you pointed me to
> ISO-8859-1, stupid I didn't thought of that - that confused me most. So the
> request params we are seeing are indeed ISO-8859-1 encoded but without
> any  "accept-charset" param given. For those cases we don't wont to fail,
> but be smart and try to encode ISO-8859-1:
> So on error I try decode again, but with Encoding::ISO8859_1, followed , by
> iconv to get it to UTF-8
>
> str = URI.decode_www_form_component("%E9", Encoding::ISO8859_1)
> Iconv.conv("utf-8", "ISO-8859-1", str)
> That seams to do the trick. Thanks!
> Tobi
>
>
> On 25.07.2011, at 20:48, John Firebaugh wrote:
>
> Hi Tobi,
> Both %c3%a9 and %e9 could be valid -- it depends on the content of the page
> and what the server is prepared to accept. In short, a standards-conforming
> browser will choose which encoding to use based (mainly) on the form's
> accept-charset attribute. You can read the details here:
> http://www.w3.org/TR/html5/association-of-controls-and-forms.html#url-encoded-form-data
> Also, note that while UTF-8 is indeed a character encoding, "Unicode" is a
> standard, not a character encoding itself. As such, it doesn't make sense to
> talk about é being "encoded in Unicode". It is true that é is the Unicode
> code point U+00E9, but code points are independent of encodings (they are
> just an abstract numeric identifiers for a particular characters). You can
> say that é is encoded in ISO-8859-1 (and other common encodings) as hex E9,
> however. (See here for
> more: http://www.joelonsoftware.com/articles/Unicode.html)
> So, with a form that specifies accept-charset="utf-8", é would be escaped
> as %c3%a9. With a form that specifies accept-charset="iso-8859-1", it would
> be escaped as %e9. Leaving it unescaped is technically invalid.
> In order to process the data correctly, the server must know what encoding
> the incoming URL parameters are encoded in, typically either by convention
> or via HTTP headers. Rack has a deficiency here, in that
> Rack::Utils.unescape does not allow you to specify the encoding -- only
> UTF-8 is supported. The underlying API used by Rack,
>
> URI.decode_www_form_component, does allow you to specify the encoding, so if
> that matters to you, you might have to use that directly. (Though as was
> discovered recently, it has a rexexp that is vulnerable to catastrophic
> backtracking.)
>
> John
> On Mon, Jul 25, 2011 at 7:22 AM, Tobias Bielohlawek <tobi@soundcloud.com>
> wrote:
>>
>> I have a question on correct behavior of handling unicode encoded
>> query params.
>>
>> Let's take character 'é', in UTF-8 that's encoded '%c3%a9 ' in unicode
>> '%e9'. As an real life example, we're seeing those URLs:
>>
>> 1. not escaped: -> http://soundcloud.com/search?q%5Bfulltext%5D=café
>> 2. UTF8 escaped -> http://soundcloud.com/search?q%5Bfulltext%5D=caf%c3%a9
>> 3. or Unicode escaped ->
>> http://soundcloud.com/search?q%5Bfulltext%5D=caf%E9
>>
>> Using rack 1.3.1, the first two cases are processed correct, but the
>> latter fails with error 'incorrect UTF-8 byte sequence'. Which is
>> correct behavior at first place as it's
>> not UTF-8 but the unicode.
>>
>> I'm no wondering what's best solution here? Why not be smart and give
>> it another run to unescape the URL assuming it's unicode encoded?
>>
>> I checked other players, e.g. Twitter has same error (but fails
>> silently)
>> http://twitter.com/search/é
>> http://twitter.com/search/%c3%a9
>> http://twitter.com/search/%E9
>>
>> but not Google, it does it this way:
>> http://www.google.de/search?q=é
>> http://www.google.de/search?q=%c3%a9
>> http://www.google.de/search?q=%E9
>>
>>
>> Any advice is very much appreciated. Patch is comming..
>>
>> Thx - Tobi
>>
>>
>
>
>
> Cheers,
>   Tobi
> –––––––––––––––––
> Tobias Bielohlawek
> Developer, SoundCloud
>
> Mail & gtalk: tobi@soundcloud.com
> Rosenthalerstraße 13, 10119 Berlin, Germany
>
> What is SoundCloud?
> http://soundcloud.com/tour
>
> Send me a track?
> http://soundcloud.com/hopit/dropbox
>
> Limited registered at Company House, Cardiff, UK. Registered Office: London,
> UK. Company Number 6343600
> Managing Director: Alexander Ljung
> Local Branch Office Berlin, Germany: AG Charlottenburg, HRB 110657B
>
>
>