From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-4.1 required=3.0 tests=AWL,BAYES_00, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 4D31C1F463 for ; Mon, 16 Dec 2019 07:31:22 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 93B58120AA8; Mon, 16 Dec 2019 16:31:07 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id 44EB1120AA7 for ; Mon, 16 Dec 2019 16:31:05 +0900 (JST) Received: by filter0191p3mdw1.sendgrid.net with SMTP id filter0191p3mdw1-26111-5DF732BB-23 2019-12-16 07:31:07.865480178 +0000 UTC m=+307314.016811097 Received: from herokuapp.com (unknown [18.212.126.58]) by ismtpd0038p1iad2.sendgrid.net (SG) with ESMTP id JZ6CZ302TbuJUJcUdNnlVw for ; Mon, 16 Dec 2019 07:31:07.848 +0000 (UTC) Date: Mon, 16 Dec 2019 07:31:07 +0000 (UTC) From: mame@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 71913 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 16284 X-Redmine-Issue-Author: zogash X-Redmine-Sender: mame X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?EJh2gqwnyqXtd++xo=2FinyA1V0bXouTB4FkWnzNiKb4+55nNo2qyOv8OBMSP7gG?= =?us-ascii?Q?yIJDGXVme7q9JpWAq=2FLkvn7POkXWZU4PfpbIHp6?= =?us-ascii?Q?AwtB62gJH0mg6n=2FU8ZGaa6yTiwQCuVhrFI=2Fx=2F=2Fc?= =?us-ascii?Q?XGs0PRHqn1R+hZ3Uc8x7WFGPaouMIx4wNCnA1qX?= =?us-ascii?Q?KaYnJeapcP47T5wZ19Uw89QwG++BU8a38tg=3D=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 96258 Subject: [ruby-core:96258] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memory compared to other libraries X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #16284 has been updated by mame (Yusuke Endoh). Status changed from Open to Closed Thanks. I think this is not a bug. Closing this issue. I guess there may be room for improvement in Net::HTTP (and Excon?), though. Investigation and a patch are welcome. ---------------------------------------- Bug #16284: Net/HTTP consuming multiple times more memory compared to other libraries https://bugs.ruby-lang.org/issues/16284#change-83152 * Author: zogash (Milovan Zogovic) * Status: Closed * Priority: Normal * Assignee: * Target version: * ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content. Here is the script to reproduce the issue: ``` gem 'excon' require 'uri' require 'net/http' require 'excon' uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4') def print_memory mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i puts "Memory: #{mem/1024} MB" end 25.times do case ARGV[0] when 'EXCON' then Excon.get(uri.to_s) when 'NET' then Net::HTTP.get_response(uri) end print_memory end ``` The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, `Net/HTTP` ends up almost 300MB of ram, while `Excon` only 60MB. Here is the complete result of RAM size the process used after each iteration: ``` > Excon Memory: 41 MB Memory: 60 MB Memory: 95 MB Memory: 58 MB Memory: 95 MB Memory: 49 MB Memory: 55 MB Memory: 51 MB Memory: 71 MB Memory: 66 MB Memory: 58 MB Memory: 65 MB Memory: 89 MB Memory: 76 MB Memory: 83 MB Memory: 64 MB Memory: 75 MB Memory: 98 MB Memory: 64 MB Memory: 58 MB Memory: 63 MB Memory: 69 MB Memory: 75 MB Memory: 70 MB Memory: 54 MB > Net::HTTP Memory: 51 MB Memory: 81 MB Memory: 104 MB Memory: 129 MB Memory: 146 MB Memory: 166 MB Memory: 163 MB Memory: 144 MB Memory: 163 MB Memory: 161 MB Memory: 169 MB Memory: 187 MB Memory: 204 MB Memory: 221 MB Memory: 238 MB Memory: 255 MB Memory: 272 MB Memory: 289 MB Memory: 306 MB Memory: 306 MB Memory: 306 MB Memory: 306 MB Memory: 317 MB Memory: 299 MB Memory: 294 MB ``` I have tried with ruby 2.7.0-preview2 and the issue is present there as well. -- https://bugs.ruby-lang.org/