Re: Rack environment encoding

[Hongli Lai <hongli@phusion.nl>]

On Sep 12, 8:00 pm, James Tucker <jftuc...@gmail.com> wrote:
> I'm not sure, but I think they don't expect them to be utf-8, they actually expect them to be compatible with literals.

Yes. I'm fine with US-ASCII for PATH_INFO and friends as long as
comment #16 in the bug report doesn't result in breakage anymore.


> > If the app does
> > something like
> >  some_utf8_string + env['PATH_INFO']
> > then Ruby 1.9 will complain with an incompatible encoding error.
>
> On your system.

No. Specifically, it breaks in Phusion Passenger because we set the
encoding of the entire environment hash to binary, regardless of the
system encoding, exactly to prevent data loss as you've mentioned
earlier. However setting everything to binary results in breakages as
described in the bug report which is the reason why I proposed setting
some things to UTF-8/ASCII/whatever and other things to binary.


> Rails does a lot of work on the /client side/ to try and ensure it receives UTF-8, and tries to enforce UTF-8 elsewhere. Rack can't enforce this as it doesn't operate client side (build forms). It's also worth noting that rails accepts a percentile use case hit here, whereby it makes no attempt to expect full support for encodings that can't round-trip through unicode. For them this is sensible, and maybe it might be for us, but this is why I need particularly CP932 users to actually pay attention here. Until I hear from someone who deals with these issues in the real world, I cannot defer to the advice "just use unicode". Alas, one of the larger issues here is that I don't speak the languages required to actually track down most of these users, so I need help from people who do. I hope there's someone on this list proactive enough to do this, or knows someone to call on.

Woah, I think we have a misunderstanding here. I started this thread
to discuss what env['something'].encoding should return. Whether
env['something'] actually contains UTF-8 data is a different
discussion.

To re-iterate: the problem that we're running into is that
env['something'].encoding always returns #<Encoding: binary> in
Phusion Passenger, even if env['something'] contains valid UTF-8 data.
Should env['something'] - assuming it contains valid UTF-8 data or
ASCII data or whatever - have its #encoding return #<Encoding: UTF-8>?


Of course, the easiest way to solve this problem is to mandate all
Rack web servers to set the encoding to binary have the frameworks
deal with conversions.