[ruby-core:95588] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memories compared to other libraries

[milovan.zogovic@gmail.com]

Issue #16284 has been reported by zogash (Milovan Zogovic).

----------------------------------------
Bug #16284: Net/HTTP consuming multiple times more memories compared to other libraries
https://bugs.ruby-lang.org/issues/16284

* Author: zogash (Milovan Zogovic)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content. 

Here is the script to reproduce the issue:

```
gem 'excon'

require 'uri'
require 'net/http'
require 'excon'

uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4')

def print_memory
  mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i
  puts "Memory: #{mem/1024} MB"
end

25.times do
  case ARGV[0]
  when 'EXCON' then Excon.get(uri.to_s)
  when 'NET' then Net::HTTP.get_response(uri)
  end
  print_memory
end

```

The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, `Net/HTTP` ends up almost 300MB of ram, while `Excon` only 60MB. Here is the complete result of RAM size the process used after each iteration:

```
> Excon
Memory: 41 MB
Memory: 60 MB
Memory: 95 MB
Memory: 58 MB
Memory: 95 MB
Memory: 49 MB
Memory: 55 MB
Memory: 51 MB
Memory: 71 MB
Memory: 66 MB
Memory: 58 MB
Memory: 65 MB
Memory: 89 MB
Memory: 76 MB
Memory: 83 MB
Memory: 64 MB
Memory: 75 MB
Memory: 98 MB
Memory: 64 MB
Memory: 58 MB
Memory: 63 MB
Memory: 69 MB
Memory: 75 MB
Memory: 70 MB
Memory: 54 MB

> Net::HTTP
Memory: 51 MB
Memory: 81 MB
Memory: 104 MB
Memory: 129 MB
Memory: 146 MB
Memory: 166 MB
Memory: 163 MB
Memory: 144 MB
Memory: 163 MB
Memory: 161 MB
Memory: 169 MB
Memory: 187 MB
Memory: 204 MB
Memory: 221 MB
Memory: 238 MB
Memory: 255 MB
Memory: 272 MB
Memory: 289 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 317 MB
Memory: 299 MB
Memory: 294 MB
```

I have tried with ruby 2.7.0-preview2 and the issue is present there as well.






-- 
https://bugs.ruby-lang.org/