From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, FORGED_GMAIL_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id AD96D1F4C0 for ; Thu, 31 Oct 2019 09:34:00 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 49AD4120A22; Thu, 31 Oct 2019 18:33:50 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id 5A2E11209E4 for ; Thu, 31 Oct 2019 18:33:47 +0900 (JST) Received: by filter0096p3las1.sendgrid.net with SMTP id filter0096p3las1-12874-5DBAAA7B-7F 2019-10-31 09:33:48.046298236 +0000 UTC m=+206513.549877012 Received: from herokuapp.com (unknown [54.210.198.90]) by ismtpd0007p1iad2.sendgrid.net (SG) with ESMTP id 0jr-cHQKRI63gIeT6DJIuA for ; Thu, 31 Oct 2019 09:33:47.957 +0000 (UTC) Date: Thu, 31 Oct 2019 09:33:48 +0000 (UTC) From: milovan.zogovic@gmail.com Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 71198 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 16284 X-Redmine-Issue-Author: zogash X-Redmine-Sender: zogash X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?Go7D1029O7yxiGxsq+rXd5xPeVzcIKUvwos555c0tpBIgF1NkTezXEjN=2FyaTQ4?= =?us-ascii?Q?pJaI0jsh8WqYxrZB0QUbDZgfVSz7HwcGPMa+VA5?= =?us-ascii?Q?mn2tmX8FQOXDjfGuqhrNE9MF7312eVgfDinOL8k?= =?us-ascii?Q?g6EBtcotH9y4nhHcnINmifgtf7ognVP0HCMcZq4?= =?us-ascii?Q?xElMJbavtkKubvIFuXHHFae1RqOUXEpp0OQ=3D=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 95605 Subject: [ruby-core:95605] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memory compared to other libraries X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #16284 has been updated by zogash (Milovan Zogovic). Hey Yusuke, thanks for checking this out. I have tried this out on my friends machines and one of them has exactly the same results as I, but other has the results that are matching yours. All three of us are using MacBooks (although different models). However, running it in Docker (`FROM ruby:2.6.5`) it gave us consistent results where Excon is actually consuming more memory. Furthermore, I have also tried running this on Heroku dyno and here are the results: > Net/HTTP Memory: 106 MB Memory: 136 MB Memory: 124 MB Memory: 124 MB Memory: 141 MB Memory: 175 MB Memory: 120 MB Memory: 149 MB Memory: 166 MB Memory: 183 MB Memory: 149 MB Memory: 149 MB Memory: 149 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB Memory: 162 MB > EXCON Memory: 117 MB Memory: 125 MB Memory: 152 MB Memory: 139 MB Memory: 166 MB Memory: 182 MB Memory: 200 MB Memory: 147 MB Memory: 164 MB Memory: 181 MB Memory: 148 MB Memory: 164 MB Memory: 181 MB Memory: 201 MB Memory: 201 MB Memory: 207 MB Memory: 223 MB Memory: 239 MB Memory: 240 MB Memory: 240 MB Memory: 240 MB Memory: 240 MB Memory: 240 MB Memory: 240 MB Memory: 240 MB ---------------------------------------- Bug #16284: Net/HTTP consuming multiple times more memory compared to other libraries https://bugs.ruby-lang.org/issues/16284#change-82394 * Author: zogash (Milovan Zogovic) * Status: Open * Priority: Normal * Assignee: * Target version: * ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content. Here is the script to reproduce the issue: ``` gem 'excon' require 'uri' require 'net/http' require 'excon' uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4') def print_memory mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i puts "Memory: #{mem/1024} MB" end 25.times do case ARGV[0] when 'EXCON' then Excon.get(uri.to_s) when 'NET' then Net::HTTP.get_response(uri) end print_memory end ``` The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, `Net/HTTP` ends up almost 300MB of ram, while `Excon` only 60MB. Here is the complete result of RAM size the process used after each iteration: ``` > Excon Memory: 41 MB Memory: 60 MB Memory: 95 MB Memory: 58 MB Memory: 95 MB Memory: 49 MB Memory: 55 MB Memory: 51 MB Memory: 71 MB Memory: 66 MB Memory: 58 MB Memory: 65 MB Memory: 89 MB Memory: 76 MB Memory: 83 MB Memory: 64 MB Memory: 75 MB Memory: 98 MB Memory: 64 MB Memory: 58 MB Memory: 63 MB Memory: 69 MB Memory: 75 MB Memory: 70 MB Memory: 54 MB > Net::HTTP Memory: 51 MB Memory: 81 MB Memory: 104 MB Memory: 129 MB Memory: 146 MB Memory: 166 MB Memory: 163 MB Memory: 144 MB Memory: 163 MB Memory: 161 MB Memory: 169 MB Memory: 187 MB Memory: 204 MB Memory: 221 MB Memory: 238 MB Memory: 255 MB Memory: 272 MB Memory: 289 MB Memory: 306 MB Memory: 306 MB Memory: 306 MB Memory: 306 MB Memory: 317 MB Memory: 299 MB Memory: 294 MB ``` I have tried with ruby 2.7.0-preview2 and the issue is present there as well. -- https://bugs.ruby-lang.org/