From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ruby-core-bounces@ruby-lang.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN: AS4713 221.184.0.0/13
X-Spam-Status: No, score=-3.5 required=3.0 tests=AWL,BAYES_00,
	HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_PASS
	shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0
Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75])
	by dcvr.yhbt.net (Postfix) with ESMTP id 682A11F406
	for <normalperson@yhbt.net>; Fri, 11 May 2018 04:26:35 +0000 (UTC)
Received: from neon.ruby-lang.org (localhost [IPv6:::1])
	by neon.ruby-lang.org (Postfix) with ESMTP id 7D6F6120A75;
	Fri, 11 May 2018 13:26:33 +0900 (JST)
Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64])
 by neon.ruby-lang.org (Postfix) with ESMTPS id 8288D120A72
 for <ruby-core@ruby-lang.org>; Fri, 11 May 2018 13:26:29 +0900 (JST)
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
 by dcvr.yhbt.net (Postfix) with ESMTP id 89C3B1F406;
 Fri, 11 May 2018 04:26:27 +0000 (UTC)
Date: Fri, 11 May 2018 04:26:27 +0000
From: Eric Wong <normalperson@yhbt.net>
To: ruby-core@ruby-lang.org
Message-ID: <20180511042627.GA18507@dcvr>
References: <redmine.issue-14745.20180509125009@ruby-lang.org>
 <redmine.issue-14745.20180509125009.1d902e8a4fa3dc01@ruby-lang.org>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <redmine.issue-14745.20180509125009.1d902e8a4fa3dc01@ruby-lang.org>
X-ML-Name: ruby-core
X-Mail-Count: 86987
Subject: [ruby-core:86987] Re: [Ruby trunk Bug#14745] High memory usage when
 using String#replace with IO.copy_stream
X-BeenThere: ruby-core@ruby-lang.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: Ruby developers <ruby-core@ruby-lang.org>
List-Id: Ruby developers <ruby-core.ruby-lang.org>
List-Unsubscribe: <https://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>, 
 <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
List-Post: <mailto:ruby-core@ruby-lang.org>
List-Help: <mailto:ruby-core-request@ruby-lang.org?subject=help>
List-Subscribe: <https://lists.ruby-lang.org/cgi-bin/mailman/listinfo/ruby-core>, 
 <mailto:ruby-core-request@ruby-lang.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: ruby-core-bounces@ruby-lang.org
Sender: "ruby-core" <ruby-core-bounces@ruby-lang.org>

janko.marohnic@gmail.com wrote:
> ~~~ ruby
>   def read(length, outbuf)
>     chunk = @io.read(length)
> 
>     if chunk
>       outbuf.replace chunk
>       chunk.clear

Elaborating on my previous comment,  chunk.clear is a no-op in
this case because the shared frozen string is still in play and
now used by outbuf.

>     else
>       outbuf.clear

Likewise, the final #clear is also no-op because the
hidden+frozen string is still floating around waiting to be
GC-ed.

>     end
> 
>     outbuf unless outbuf.empty?
>   end
> ~~~

I don't think you need IO.copy_stream or IO#write to trigger, even.

~~~ ruby
io = FakeIO.new("a" * 50*1024*1024) # 50MB
buf = ''.b
while z = io.read(16384, buf)
end
~~~

I tried playing with shortening object lifetimes, but I guess GC
is not aggressive enough by default to matter.