Threading Bug Causing Corrupted File Response

Threading Bug Causing Corrupted File Response

[Sunny Hirai]

There appears to be a threading related bug where if you try and load
the same file twice at the same time, you get an error that flashes in
Firefox before displaying the image that says:

  The image "path/to/image.png" cannot be displayed, because it
contains errors.

Since this was working before, I'm guessing the bug was introduced in
1.1 or at least a recent-ish release (not exactly sure when the bug
crawled in).

Although I was using a custom file mapper, this bug still happens in
Rack's file mapper. I can replicate by doing:

app = Rack::Builder.new do
  run Rack::File.new( '.' )
end

If I hit the file twice quickly, Firefox will flash the error. It will
retry immediately so the image eventually comes up but there are twice
as many requests made.

The bug can be replicated using both Mongrel and WEBrick.

Sunny Hirai

Re: Threading Bug Causing Corrupted File Response

[Sunny Hirai]

Okay, narrowed it down a little. It only happens with Rack::Builder.

Actually, the error is hard to reproduce when going directly to the
web service because everything happens fast. We put in a proxy to
simulate slow connections and that's what is causing the problems to
show up more.

Sunny

On Jan 25, 7:44 pm, Sunny Hirai <thesu...@gmail.com> wrote:
> There appears to be a threading related bug where if you try and load
> the same file twice at the same time, you get an error that flashes in
> Firefox before displaying the image that says:
>
>   The image "path/to/image.png" cannot be displayed, because it
> contains errors.
>
> Since this was working before, I'm guessing the bug was introduced in
> 1.1 or at least a recent-ish release (not exactly sure when the bug
> crawled in).
>
> Although I was using a custom file mapper, this bug still happens in
> Rack's file mapper. I can replicate by doing:
>
> app = Rack::Builder.new do
>   run Rack::File.new( '.' )
> end
>
> If I hit the file twice quickly, Firefox will flash the error. It will
> retry immediately so the image eventually comes up but there are twice
> as many requests made.
>
> The bug can be replicated using both Mongrel and WEBrick.
>
> Sunny Hirai

Re: Threading Bug Causing Corrupted File Response

[Sunny Hirai]

Oops, sorry. It is not Builder. I can get the errors for it after all.

Sunny

On Jan 25, 7:53 pm, Sunny Hirai <thesu...@gmail.com> wrote:
> Okay, narrowed it down a little. It only happens with Rack::Builder.
>
> Actually, the error is hard to reproduce when going directly to the
> web service because everything happens fast. We put in a proxy to
> simulate slow connections and that's what is causing the problems to
> show up more.
>
> Sunny
>
> On Jan 25, 7:44 pm, Sunny Hirai <thesu...@gmail.com> wrote:
>
> > There appears to be a threading related bug where if you try and load
> > the same file twice at the same time, you get an error that flashes in
> > Firefox before displaying the image that says:
>
> >   The image "path/to/image.png" cannot be displayed, because it
> > contains errors.
>
> > Since this was working before, I'm guessing the bug was introduced in
> > 1.1 or at least a recent-ish release (not exactly sure when the bug
> > crawled in).
>
> > Although I was using a custom file mapper, this bug still happens in
> > Rack's file mapper. I can replicate by doing:
>
> > app = Rack::Builder.new do
> >   run Rack::File.new( '.' )
> > end
>
> > If I hit the file twice quickly, Firefox will flash the error. It will
> > retry immediately so the image eventually comes up but there are twice
> > as many requests made.
>
> > The bug can be replicated using both Mongrel and WEBrick.
>
> > Sunny Hirai
>
>

Re: Threading Bug Causing Corrupted File Response

[Sunny Hirai]

Just an update.

This problem does not seem to happen on Linux and only on Windows.

Note that it is also not a :num_processor limit that it is hitting as
it is set to 950.

I may be giving up on fixing this for now since it won't affect our
production environment, just my personal development environment.
Still, it feels like it should be fixed.

Sunny

On Jan 25, 8:10 pm, Sunny Hirai <thesu...@gmail.com> wrote:
> Oops, sorry. It is not Builder. I can get the errors for it after all.
>
> Sunny
>
> On Jan 25, 7:53 pm, Sunny Hirai <thesu...@gmail.com> wrote:
>
> > Okay, narrowed it down a little. It only happens with Rack::Builder.
>
> > Actually, the error is hard to reproduce when going directly to the
> > web service because everything happens fast. We put in a proxy to
> > simulate slow connections and that's what is causing the problems to
> > show up more.
>
> > Sunny
>
> > On Jan 25, 7:44 pm, Sunny Hirai <thesu...@gmail.com> wrote:
>
> > > There appears to be a threading related bug where if you try and load
> > > the same file twice at the same time, you get an error that flashes in
> > > Firefox before displaying the image that says:
>
> > >   The image "path/to/image.png" cannot be displayed, because it
> > > contains errors.
>
> > > Since this was working before, I'm guessing the bug was introduced in
> > > 1.1 or at least a recent-ish release (not exactly sure when the bug
> > > crawled in).
>
> > > Although I was using a custom file mapper, this bug still happens in
> > > Rack's file mapper. I can replicate by doing:
>
> > > app = Rack::Builder.new do
> > >   run Rack::File.new( '.' )
> > > end
>
> > > If I hit the file twice quickly, Firefox will flash the error. It will
> > > retry immediately so the image eventually comes up but there are twice
> > > as many requests made.
>
> > > The bug can be replicated using both Mongrel and WEBrick.
>
> > > Sunny Hirai
>
>

Re: Threading Bug Causing Corrupted File Response

[James Tucker]

[-- Attachment #1: Type: text/plain, Size: 1182 bytes --]


On 28 Jan 2010, at 01:22, Sunny Hirai wrote:

> Just an update.
> 
> This problem does not seem to happen on Linux and only on Windows.

Platform specific, good clue.

> Note that it is also not a :num_processor limit that it is hitting as
> it is set to 950.

num_processors is not the only limit in effect, there's also socket, select and file handle limits. under 1.8.x on win32 presently, i believe ruby is pretty limited, 256 handles if i remember correctly (this needs checking).

> I may be giving up on fixing this for now since it won't affect our
> production environment, just my personal development environment.
> Still, it feels like it should be fixed.

When you describe the problem, you say "flashed and then reloaded", are you sure it's not just a close connection effect due to a refresh?

Where are you seeing the "error"?

You can quite easily test this by getting a copy of curl[1] and spawning a bunch of curl processes in parallel. Log the results, and save the files downloaded, then hash check all the files, and correlate response data. If some of the files are corrupted, please post your script that demonstrates the error.

HTH

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3679 bytes --]