[ruby-core:69827] [Ruby trunk - Bug #11322] [Open] OpenUri: RuntimeError: HTTP redirection loop

[ruby-core:69827] [Ruby trunk - Bug #11322] [Open] OpenUri: RuntimeError: HTTP redirection loop

[tobias.preuss+ruby-lang]

Issue #11322 has been reported by Tobias Preuss.

----------------------------------------
Bug #11322: OpenUri: RuntimeError: HTTP redirection loop
https://bugs.ruby-lang.org/issues/11322

* Author: Tobias Preuss
* Status: Open
* Priority: Normal
* Assignee: Akira Tanaka
* ruby -v: ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN
----------------------------------------
Trying to download [this file](http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip) from [this website](http://www.london.ca/city-hall/open-data/Pages/Open-Data-Data-Catalogue.aspx) with [`OpenUri`](http://ruby-doc.org/stdlib-2.2.2/libdoc/open-uri/rdoc/OpenURI.html) fails with the runtime error "HTTP redirection loop".
Here is how I can reproduce the error:

~~~
> require 'open-uri'
 => true

> open('http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip')
RuntimeError: HTTP redirection loop: http://apps.london.ca/uniquesig87fdc01fb86ce6f0fd235c713015d7d7/uniquesig0/InternalSite/StartApp.asp?resource_id=837A134B9EC24A2197B6AF5745B6CA55&login_type=0&site_name=appstrunk&secure=0&orig_url=http%3a%2f%2fapps.london.ca%2fOpenData%2fShapeFiles_Zipped%2f2010_skateboard_parks_shp.zip
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:232:in `open_loop'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:150:in `open_uri'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:716:in `open'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:34:in `open'
	from (irb):2
	from /home/john/.rvm/rubies/ruby-2.2.2/bin/irb:11:in `<main>'
~~~



-- 
https://bugs.ruby-lang.org/

[ruby-core:69831] [Ruby trunk - Bug #11322] OpenUri: RuntimeError: HTTP redirection loop

[0x0dea+redmine]

Issue #11322 has been updated by D.E. Akers.


The problem is not specific to `OpenURI`:

    $ curl -L http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip
    curl: (47) Maximum (50) redirects followed

It seems the `302 Object Moved` handler on this server has not been properly configured; it expects the previous request to have set a few cookies and simply sends the client back if it doesn't find them.

`OpenURI` appears to be incapable of handling such a circumstance, but `Net::HTTP` can and isn't that much more complex. I've presented below a demonstration of how you might go about orchestrating the "handshake" in order to successfully obtain the file.

``` ruby
conn = Net::HTTP.new 'apps.london.ca'
file = '/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip'
resp = conn.get file

cookie = resp.get_fields('Set-Cookie').map { |c| c.split(';')[0] }.join(';')
resp   = conn.get file, 'Cookie' => cookie
File.write File.basename(file), resp.body
```

----------------------------------------
Bug #11322: OpenUri: RuntimeError: HTTP redirection loop
https://bugs.ruby-lang.org/issues/11322#change-53229

* Author: Tobias Preuss
* Status: Open
* Priority: Normal
* Assignee: Akira Tanaka
* ruby -v: ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN
----------------------------------------
Trying to download [this file](http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip) from [this website](http://www.london.ca/city-hall/open-data/Pages/Open-Data-Data-Catalogue.aspx) with [`OpenUri`](http://ruby-doc.org/stdlib-2.2.2/libdoc/open-uri/rdoc/OpenURI.html) fails with the runtime error "HTTP redirection loop".
Here is how I can reproduce the error:

~~~
> require 'open-uri'
 => true

> open('http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip')
RuntimeError: HTTP redirection loop: http://apps.london.ca/uniquesig87fdc01fb86ce6f0fd235c713015d7d7/uniquesig0/InternalSite/StartApp.asp?resource_id=837A134B9EC24A2197B6AF5745B6CA55&login_type=0&site_name=appstrunk&secure=0&orig_url=http%3a%2f%2fapps.london.ca%2fOpenData%2fShapeFiles_Zipped%2f2010_skateboard_parks_shp.zip
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:232:in `open_loop'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:150:in `open_uri'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:716:in `open'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:34:in `open'
	from (irb):2
	from /home/john/.rvm/rubies/ruby-2.2.2/bin/irb:11:in `<main>'
~~~



-- 
https://bugs.ruby-lang.org/

[ruby-core:69840] [Ruby trunk - Bug #11322] OpenUri: RuntimeError: HTTP redirection loop

[tobias.preuss+ruby-lang]

Issue #11322 has been updated by Tobias Preuss.


Dear D.E. Akers: Your workaround works like a charm. Thank you very much.

----------------------------------------
Bug #11322: OpenUri: RuntimeError: HTTP redirection loop
https://bugs.ruby-lang.org/issues/11322#change-53240

* Author: Tobias Preuss
* Status: Open
* Priority: Normal
* Assignee: Akira Tanaka
* ruby -v: ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN
----------------------------------------
Trying to download [this file](http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip) from [this website](http://www.london.ca/city-hall/open-data/Pages/Open-Data-Data-Catalogue.aspx) with [`OpenUri`](http://ruby-doc.org/stdlib-2.2.2/libdoc/open-uri/rdoc/OpenURI.html) fails with the runtime error "HTTP redirection loop".
Here is how I can reproduce the error:

~~~
> require 'open-uri'
 => true

> open('http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip')
RuntimeError: HTTP redirection loop: http://apps.london.ca/uniquesig87fdc01fb86ce6f0fd235c713015d7d7/uniquesig0/InternalSite/StartApp.asp?resource_id=837A134B9EC24A2197B6AF5745B6CA55&login_type=0&site_name=appstrunk&secure=0&orig_url=http%3a%2f%2fapps.london.ca%2fOpenData%2fShapeFiles_Zipped%2f2010_skateboard_parks_shp.zip
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:232:in `open_loop'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:150:in `open_uri'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:716:in `open'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:34:in `open'
	from (irb):2
	from /home/john/.rvm/rubies/ruby-2.2.2/bin/irb:11:in `<main>'
~~~



-- 
https://bugs.ruby-lang.org/

[ruby-core:74828] [Ruby trunk Bug#11322] OpenUri: RuntimeError: HTTP redirection loop

[nutaksho]

Issue #11322 has been updated by Eugene Chaikin.


i've had a similar issue with `open('http://www.replayjeans.com/us/shop/product/women/jumpers-knitwear/neoprene-printed-sweatshirt/pc/48/c/61/sc/-1/1962')`
which i solved modifying D.E. Akers workaround a bit:

~~~

url = 'http://www.replayjeans.com/us/shop/product/women/jumpers-knitwear/neoprene-printed-sweatshirt/pc/48/c/61/sc/-1/1962'
uri = URI(url)
res = Net::HTTP.get_response(uri)
cookie = res['Set-Cookie']
req = Net::HTTP::Get.new(uri)
req['Cookie'] = cookie
res = Net::HTTP.start(uri.hostname, uri.port) { |http| http.request(req) }

~~~

However there is another issue that i can't resolve.
Some requests don't end up in a redirection loop, and `open url` returns not the page i expect.
If I apply redirect loop workaround for this case, i get the correct page.
Tried to google but no avail so far.

----------------------------------------
Bug #11322: OpenUri: RuntimeError: HTTP redirection loop
https://bugs.ruby-lang.org/issues/11322#change-57953

* Author: Tobias Preuss
* Status: Open
* Priority: Normal
* Assignee: Akira Tanaka
* ruby -v: ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN
----------------------------------------
Trying to download [this file](http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip) from [this website](http://www.london.ca/city-hall/open-data/Pages/Open-Data-Data-Catalogue.aspx) with [`OpenUri`](http://ruby-doc.org/stdlib-2.2.2/libdoc/open-uri/rdoc/OpenURI.html) fails with the runtime error "HTTP redirection loop".
Here is how I can reproduce the error:

~~~
> require 'open-uri'
 => true

> open('http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip')
RuntimeError: HTTP redirection loop: http://apps.london.ca/uniquesig87fdc01fb86ce6f0fd235c713015d7d7/uniquesig0/InternalSite/StartApp.asp?resource_id=837A134B9EC24A2197B6AF5745B6CA55&login_type=0&site_name=appstrunk&secure=0&orig_url=http%3a%2f%2fapps.london.ca%2fOpenData%2fShapeFiles_Zipped%2f2010_skateboard_parks_shp.zip
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:232:in `open_loop'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:150:in `open_uri'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:716:in `open'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:34:in `open'
	from (irb):2
	from /home/john/.rvm/rubies/ruby-2.2.2/bin/irb:11:in `<main>'
~~~



-- 
https://bugs.ruby-lang.org/

[ruby-core:94488] [Ruby master Feature#11322] OpenUri: RuntimeError: HTTP redirection loop

[merch-redmine]

Issue #11322 has been updated by jeremyevans0 (Jeremy Evans).

Backport deleted (2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN)
ruby -v deleted (ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux])
Tracker changed from Bug to Feature
File open_uri-redirect-cookie-11322.patch added

I don't think this is a bug.  As 0x0dea (D.E. Akers) pointed out, other programs work the same way.  However, I think cookie handling in open_uri could be a useful feature.  Attached is a patch that implements the necessary support.

I used this Roda app to test the cookie redirection support:

```ruby
require 'roda'

Roda.plugin :cookies
Roda.route do |r|
  r.root do
    if r.cookies['foo'] == 'bar'
      'Success!'
    else
      response.set_cookie('foo', 'bar')
      r.redirect '/'
    end
  end
end

run Roda
```

----------------------------------------
Feature #11322: OpenUri: RuntimeError: HTTP redirection loop
https://bugs.ruby-lang.org/issues/11322#change-80923

* Author: tbsprs (Tobias Preuss)
* Status: Assigned
* Priority: Normal
* Assignee: akr (Akira Tanaka)
* Target version: 
----------------------------------------
Trying to download [this file](http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip) from [this website](http://www.london.ca/city-hall/open-data/Pages/Open-Data-Data-Catalogue.aspx) with [`OpenUri`](http://ruby-doc.org/stdlib-2.2.2/libdoc/open-uri/rdoc/OpenURI.html) fails with the runtime error "HTTP redirection loop".
Here is how I can reproduce the error:

~~~
> require 'open-uri'
 => true

> open('http://apps.london.ca/OpenData/ShapeFiles_Zipped/2010_skateboard_parks_shp.zip')
RuntimeError: HTTP redirection loop: http://apps.london.ca/uniquesig87fdc01fb86ce6f0fd235c713015d7d7/uniquesig0/InternalSite/StartApp.asp?resource_id=837A134B9EC24A2197B6AF5745B6CA55&login_type=0&site_name=appstrunk&secure=0&orig_url=http%3a%2f%2fapps.london.ca%2fOpenData%2fShapeFiles_Zipped%2f2010_skateboard_parks_shp.zip
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:232:in `open_loop'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:150:in `open_uri'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:716:in `open'
	from /home/john/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/open-uri.rb:34:in `open'
	from (irb):2
	from /home/john/.rvm/rubies/ruby-2.2.2/bin/irb:11:in `<main>'
~~~

---Files--------------------------------
open_uri-redirect-cookie-11322.patch (2.3 KB)


-- 
https://bugs.ruby-lang.org/