* [ruby-core:95588] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memories compared to other libraries
[not found] <redmine.issue-16284.20191030121415@ruby-lang.org>
@ 2019-10-30 12:14 ` milovan.zogovic
2019-10-30 14:10 ` [ruby-core:95589] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memory " mame
` (2 subsequent siblings)
3 siblings, 0 replies; 4+ messages in thread
From: milovan.zogovic @ 2019-10-30 12:14 UTC (permalink / raw)
To: ruby-core
Issue #16284 has been reported by zogash (Milovan Zogovic).
----------------------------------------
Bug #16284: Net/HTTP consuming multiple times more memories compared to other libraries
https://bugs.ruby-lang.org/issues/16284
* Author: zogash (Milovan Zogovic)
* Status: Open
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content.
Here is the script to reproduce the issue:
```
gem 'excon'
require 'uri'
require 'net/http'
require 'excon'
uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4')
def print_memory
mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i
puts "Memory: #{mem/1024} MB"
end
25.times do
case ARGV[0]
when 'EXCON' then Excon.get(uri.to_s)
when 'NET' then Net::HTTP.get_response(uri)
end
print_memory
end
```
The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, `Net/HTTP` ends up almost 300MB of ram, while `Excon` only 60MB. Here is the complete result of RAM size the process used after each iteration:
```
> Excon
Memory: 41 MB
Memory: 60 MB
Memory: 95 MB
Memory: 58 MB
Memory: 95 MB
Memory: 49 MB
Memory: 55 MB
Memory: 51 MB
Memory: 71 MB
Memory: 66 MB
Memory: 58 MB
Memory: 65 MB
Memory: 89 MB
Memory: 76 MB
Memory: 83 MB
Memory: 64 MB
Memory: 75 MB
Memory: 98 MB
Memory: 64 MB
Memory: 58 MB
Memory: 63 MB
Memory: 69 MB
Memory: 75 MB
Memory: 70 MB
Memory: 54 MB
> Net::HTTP
Memory: 51 MB
Memory: 81 MB
Memory: 104 MB
Memory: 129 MB
Memory: 146 MB
Memory: 166 MB
Memory: 163 MB
Memory: 144 MB
Memory: 163 MB
Memory: 161 MB
Memory: 169 MB
Memory: 187 MB
Memory: 204 MB
Memory: 221 MB
Memory: 238 MB
Memory: 255 MB
Memory: 272 MB
Memory: 289 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 317 MB
Memory: 299 MB
Memory: 294 MB
```
I have tried with ruby 2.7.0-preview2 and the issue is present there as well.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 4+ messages in thread* [ruby-core:95589] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memory compared to other libraries
[not found] <redmine.issue-16284.20191030121415@ruby-lang.org>
2019-10-30 12:14 ` [ruby-core:95588] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memories compared to other libraries milovan.zogovic
@ 2019-10-30 14:10 ` mame
2019-10-31 9:33 ` [ruby-core:95605] " milovan.zogovic
2019-12-16 7:31 ` [ruby-core:96258] " mame
3 siblings, 0 replies; 4+ messages in thread
From: mame @ 2019-10-30 14:10 UTC (permalink / raw)
To: ruby-core
Issue #16284 has been updated by mame (Yusuke Endoh).
Thank you for the report. Unfortunately, I cannot reproduce the issue. The result on my Linux machine is completely opposite against yours.
```
$ ruby -v
ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
$ ruby t.rb EXCON
Memory: 60 MB
Memory: 81 MB
Memory: 104 MB
Memory: 83 MB
Memory: 107 MB
Memory: 124 MB
Memory: 92 MB
Memory: 109 MB
Memory: 126 MB
Memory: 148 MB
Memory: 150 MB
Memory: 154 MB
Memory: 136 MB
Memory: 136 MB
Memory: 153 MB
Memory: 170 MB
Memory: 189 MB
Memory: 191 MB
Memory: 192 MB
Memory: 192 MB
Memory: 159 MB
Memory: 159 MB
Memory: 159 MB
Memory: 175 MB
Memory: 193 MB
$ ruby t.rb NET
Memory: 49 MB
Memory: 61 MB
Memory: 62 MB
Memory: 79 MB
Memory: 96 MB
Memory: 97 MB
Memory: 99 MB
Memory: 104 MB
Memory: 80 MB
Memory: 97 MB
Memory: 97 MB
Memory: 104 MB
Memory: 63 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
Memory: 85 MB
```
----------------------------------------
Bug #16284: Net/HTTP consuming multiple times more memory compared to other libraries
https://bugs.ruby-lang.org/issues/16284#change-82377
* Author: zogash (Milovan Zogovic)
* Status: Open
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content.
Here is the script to reproduce the issue:
```
gem 'excon'
require 'uri'
require 'net/http'
require 'excon'
uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4')
def print_memory
mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i
puts "Memory: #{mem/1024} MB"
end
25.times do
case ARGV[0]
when 'EXCON' then Excon.get(uri.to_s)
when 'NET' then Net::HTTP.get_response(uri)
end
print_memory
end
```
The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, `Net/HTTP` ends up almost 300MB of ram, while `Excon` only 60MB. Here is the complete result of RAM size the process used after each iteration:
```
> Excon
Memory: 41 MB
Memory: 60 MB
Memory: 95 MB
Memory: 58 MB
Memory: 95 MB
Memory: 49 MB
Memory: 55 MB
Memory: 51 MB
Memory: 71 MB
Memory: 66 MB
Memory: 58 MB
Memory: 65 MB
Memory: 89 MB
Memory: 76 MB
Memory: 83 MB
Memory: 64 MB
Memory: 75 MB
Memory: 98 MB
Memory: 64 MB
Memory: 58 MB
Memory: 63 MB
Memory: 69 MB
Memory: 75 MB
Memory: 70 MB
Memory: 54 MB
> Net::HTTP
Memory: 51 MB
Memory: 81 MB
Memory: 104 MB
Memory: 129 MB
Memory: 146 MB
Memory: 166 MB
Memory: 163 MB
Memory: 144 MB
Memory: 163 MB
Memory: 161 MB
Memory: 169 MB
Memory: 187 MB
Memory: 204 MB
Memory: 221 MB
Memory: 238 MB
Memory: 255 MB
Memory: 272 MB
Memory: 289 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 317 MB
Memory: 299 MB
Memory: 294 MB
```
I have tried with ruby 2.7.0-preview2 and the issue is present there as well.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 4+ messages in thread* [ruby-core:95605] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memory compared to other libraries
[not found] <redmine.issue-16284.20191030121415@ruby-lang.org>
2019-10-30 12:14 ` [ruby-core:95588] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memories compared to other libraries milovan.zogovic
2019-10-30 14:10 ` [ruby-core:95589] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memory " mame
@ 2019-10-31 9:33 ` milovan.zogovic
2019-12-16 7:31 ` [ruby-core:96258] " mame
3 siblings, 0 replies; 4+ messages in thread
From: milovan.zogovic @ 2019-10-31 9:33 UTC (permalink / raw)
To: ruby-core
Issue #16284 has been updated by zogash (Milovan Zogovic).
Hey Yusuke, thanks for checking this out. I have tried this out on my friends machines and one of them has exactly the same results as I, but other has the results that are matching yours. All three of us are using MacBooks (although different models). However, running it in Docker (`FROM ruby:2.6.5`) it gave us consistent results where Excon is actually consuming more memory. Furthermore, I have also tried running this on Heroku dyno and here are the results:
> Net/HTTP
Memory: 106 MB
Memory: 136 MB
Memory: 124 MB
Memory: 124 MB
Memory: 141 MB
Memory: 175 MB
Memory: 120 MB
Memory: 149 MB
Memory: 166 MB
Memory: 183 MB
Memory: 149 MB
Memory: 149 MB
Memory: 149 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
Memory: 162 MB
> EXCON
Memory: 117 MB
Memory: 125 MB
Memory: 152 MB
Memory: 139 MB
Memory: 166 MB
Memory: 182 MB
Memory: 200 MB
Memory: 147 MB
Memory: 164 MB
Memory: 181 MB
Memory: 148 MB
Memory: 164 MB
Memory: 181 MB
Memory: 201 MB
Memory: 201 MB
Memory: 207 MB
Memory: 223 MB
Memory: 239 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
Memory: 240 MB
----------------------------------------
Bug #16284: Net/HTTP consuming multiple times more memory compared to other libraries
https://bugs.ruby-lang.org/issues/16284#change-82394
* Author: zogash (Milovan Zogovic)
* Status: Open
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content.
Here is the script to reproduce the issue:
```
gem 'excon'
require 'uri'
require 'net/http'
require 'excon'
uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4')
def print_memory
mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i
puts "Memory: #{mem/1024} MB"
end
25.times do
case ARGV[0]
when 'EXCON' then Excon.get(uri.to_s)
when 'NET' then Net::HTTP.get_response(uri)
end
print_memory
end
```
The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, `Net/HTTP` ends up almost 300MB of ram, while `Excon` only 60MB. Here is the complete result of RAM size the process used after each iteration:
```
> Excon
Memory: 41 MB
Memory: 60 MB
Memory: 95 MB
Memory: 58 MB
Memory: 95 MB
Memory: 49 MB
Memory: 55 MB
Memory: 51 MB
Memory: 71 MB
Memory: 66 MB
Memory: 58 MB
Memory: 65 MB
Memory: 89 MB
Memory: 76 MB
Memory: 83 MB
Memory: 64 MB
Memory: 75 MB
Memory: 98 MB
Memory: 64 MB
Memory: 58 MB
Memory: 63 MB
Memory: 69 MB
Memory: 75 MB
Memory: 70 MB
Memory: 54 MB
> Net::HTTP
Memory: 51 MB
Memory: 81 MB
Memory: 104 MB
Memory: 129 MB
Memory: 146 MB
Memory: 166 MB
Memory: 163 MB
Memory: 144 MB
Memory: 163 MB
Memory: 161 MB
Memory: 169 MB
Memory: 187 MB
Memory: 204 MB
Memory: 221 MB
Memory: 238 MB
Memory: 255 MB
Memory: 272 MB
Memory: 289 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 317 MB
Memory: 299 MB
Memory: 294 MB
```
I have tried with ruby 2.7.0-preview2 and the issue is present there as well.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 4+ messages in thread* [ruby-core:96258] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memory compared to other libraries
[not found] <redmine.issue-16284.20191030121415@ruby-lang.org>
` (2 preceding siblings ...)
2019-10-31 9:33 ` [ruby-core:95605] " milovan.zogovic
@ 2019-12-16 7:31 ` mame
3 siblings, 0 replies; 4+ messages in thread
From: mame @ 2019-12-16 7:31 UTC (permalink / raw)
To: ruby-core
Issue #16284 has been updated by mame (Yusuke Endoh).
Status changed from Open to Closed
Thanks. I think this is not a bug. Closing this issue.
I guess there may be room for improvement in Net::HTTP (and Excon?), though. Investigation and a patch are welcome.
----------------------------------------
Bug #16284: Net/HTTP consuming multiple times more memory compared to other libraries
https://bugs.ruby-lang.org/issues/16284#change-83152
* Author: zogash (Milovan Zogovic)
* Status: Closed
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content.
Here is the script to reproduce the issue:
```
gem 'excon'
require 'uri'
require 'net/http'
require 'excon'
uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4')
def print_memory
mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i
puts "Memory: #{mem/1024} MB"
end
25.times do
case ARGV[0]
when 'EXCON' then Excon.get(uri.to_s)
when 'NET' then Net::HTTP.get_response(uri)
end
print_memory
end
```
The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, `Net/HTTP` ends up almost 300MB of ram, while `Excon` only 60MB. Here is the complete result of RAM size the process used after each iteration:
```
> Excon
Memory: 41 MB
Memory: 60 MB
Memory: 95 MB
Memory: 58 MB
Memory: 95 MB
Memory: 49 MB
Memory: 55 MB
Memory: 51 MB
Memory: 71 MB
Memory: 66 MB
Memory: 58 MB
Memory: 65 MB
Memory: 89 MB
Memory: 76 MB
Memory: 83 MB
Memory: 64 MB
Memory: 75 MB
Memory: 98 MB
Memory: 64 MB
Memory: 58 MB
Memory: 63 MB
Memory: 69 MB
Memory: 75 MB
Memory: 70 MB
Memory: 54 MB
> Net::HTTP
Memory: 51 MB
Memory: 81 MB
Memory: 104 MB
Memory: 129 MB
Memory: 146 MB
Memory: 166 MB
Memory: 163 MB
Memory: 144 MB
Memory: 163 MB
Memory: 161 MB
Memory: 169 MB
Memory: 187 MB
Memory: 204 MB
Memory: 221 MB
Memory: 238 MB
Memory: 255 MB
Memory: 272 MB
Memory: 289 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 317 MB
Memory: 299 MB
Memory: 294 MB
```
I have tried with ruby 2.7.0-preview2 and the issue is present there as well.
--
https://bugs.ruby-lang.org/
^ permalink raw reply [flat|nested] 4+ messages in thread