From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, FORGED_GMAIL_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 7B0591F4C0 for ; Wed, 30 Oct 2019 12:14:30 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 9963F120970; Wed, 30 Oct 2019 21:14:19 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id CE673120954 for ; Wed, 30 Oct 2019 21:14:16 +0900 (JST) Received: by filter0084p3iad2.sendgrid.net with SMTP id filter0084p3iad2-18587-5DB97E97-99 2019-10-30 12:14:15.754121018 +0000 UTC m=+133045.235249139 Received: from herokuapp.com (unknown [18.215.166.206]) by ismtpd0003p1iad2.sendgrid.net (SG) with ESMTP id nmkR_gmHSYC87aj1Xy9vsA for ; Wed, 30 Oct 2019 12:14:15.637 +0000 (UTC) Date: Wed, 30 Oct 2019 12:14:15 +0000 (UTC) From: milovan.zogovic@gmail.com Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 71181 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 16284 X-Redmine-Issue-Author: zogash X-Redmine-Sender: zogash X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?Go7D1029O7yxiGxsq+rXd5xPeVzcIKUvwos555c0tpBj8vfBuTqGZ9U=2F1wg32j?= =?us-ascii?Q?Tj9LoX6ykqEJ9SokAQtIcAsYZmGcHDOtU=2FGcnjG?= =?us-ascii?Q?psEkFWkQOsm9jlJu40sYp8Zd9dAlNDA+NbfyV4e?= =?us-ascii?Q?CHgiRBcpfShf0QV8=2FKJqunHwR=2Fc5jGae87CWNLL?= =?us-ascii?Q?ae+6TNosVFr8Zd=2FEquvjfqe40Bp61=2FRMM4g=3D=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 95588 Subject: [ruby-core:95588] [Ruby master Bug#16284] Net/HTTP consuming multiple times more memories compared to other libraries X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #16284 has been reported by zogash (Milovan Zogovic). ---------------------------------------- Bug #16284: Net/HTTP consuming multiple times more memories compared to other libraries https://bugs.ruby-lang.org/issues/16284 * Author: zogash (Milovan Zogovic) * Status: Open * Priority: Normal * Assignee: * Target version: * ruby -v: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content. Here is the script to reproduce the issue: ``` gem 'excon' require 'uri' require 'net/http' require 'excon' uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4') def print_memory mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i puts "Memory: #{mem/1024} MB" end 25.times do case ARGV[0] when 'EXCON' then Excon.get(uri.to_s) when 'NET' then Net::HTTP.get_response(uri) end print_memory end ``` The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, `Net/HTTP` ends up almost 300MB of ram, while `Excon` only 60MB. Here is the complete result of RAM size the process used after each iteration: ``` > Excon Memory: 41 MB Memory: 60 MB Memory: 95 MB Memory: 58 MB Memory: 95 MB Memory: 49 MB Memory: 55 MB Memory: 51 MB Memory: 71 MB Memory: 66 MB Memory: 58 MB Memory: 65 MB Memory: 89 MB Memory: 76 MB Memory: 83 MB Memory: 64 MB Memory: 75 MB Memory: 98 MB Memory: 64 MB Memory: 58 MB Memory: 63 MB Memory: 69 MB Memory: 75 MB Memory: 70 MB Memory: 54 MB > Net::HTTP Memory: 51 MB Memory: 81 MB Memory: 104 MB Memory: 129 MB Memory: 146 MB Memory: 166 MB Memory: 163 MB Memory: 144 MB Memory: 163 MB Memory: 161 MB Memory: 169 MB Memory: 187 MB Memory: 204 MB Memory: 221 MB Memory: 238 MB Memory: 255 MB Memory: 272 MB Memory: 289 MB Memory: 306 MB Memory: 306 MB Memory: 306 MB Memory: 306 MB Memory: 317 MB Memory: 299 MB Memory: 294 MB ``` I have tried with ruby 2.7.0-preview2 and the issue is present there as well. -- https://bugs.ruby-lang.org/