From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: poffice@blade.nagaokaut.ac.jp Delivered-To: poffice@blade.nagaokaut.ac.jp Received: from kankan.nagaokaut.ac.jp (kankan.nagaokaut.ac.jp [133.44.2.24]) by blade.nagaokaut.ac.jp (Postfix) with ESMTP id CA8C91940A34 for ; Wed, 17 Jun 2015 08:10:56 +0900 (JST) Received: from funfun.nagaokaut.ac.jp (smtp.nagaokaut.ac.jp [133.44.2.201]) by kankan.nagaokaut.ac.jp (Postfix) with ESMTP id 5A34EB5D8CB for ; Wed, 17 Jun 2015 08:33:22 +0900 (JST) Received: from funfun.nagaokaut.ac.jp (localhost.nagaokaut.ac.jp [127.0.0.1]) by funfun.nagaokaut.ac.jp (Postfix) with ESMTP id 3C79C97A827 for ; Wed, 17 Jun 2015 08:33:23 +0900 (JST) X-Virus-Scanned: amavisd-new at nagaokaut.ac.jp Authentication-Results: funfun.nagaokaut.ac.jp (amavisd-new); dkim=fail (1024-bit key) reason="fail (message has been altered)" header.d=sendgrid.me Received: from funfun.nagaokaut.ac.jp ([127.0.0.1]) by funfun.nagaokaut.ac.jp (funfun.nagaokaut.ac.jp [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id we7xVC05eden for ; Wed, 17 Jun 2015 08:33:23 +0900 (JST) Received: from voscc.nagaokaut.ac.jp (voscc.nagaokaut.ac.jp [133.44.1.100]) by funfun.nagaokaut.ac.jp (Postfix) with ESMTP id 1178497A826 for ; Wed, 17 Jun 2015 08:33:23 +0900 (JST) Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by voscc.nagaokaut.ac.jp (Postfix) with ESMTP id 1B13295243A for ; Wed, 17 Jun 2015 08:33:21 +0900 (JST) Received: from [221.186.184.76] (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id C580412046D; Wed, 17 Jun 2015 08:33:19 +0900 (JST) X-Original-To: ruby-core@ruby-lang.org Delivered-To: ruby-core@ruby-lang.org Received: from o10.shared.sendgrid.net (o10.shared.sendgrid.net [173.193.132.135]) by neon.ruby-lang.org (Postfix) with ESMTPS id 0A775120458 for ; Wed, 17 Jun 2015 08:33:15 +0900 (JST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.me; h=from:to:references:subject:mime-version:content-type:content-transfer-encoding:list-id; s=smtpapi; bh=qtaKqbdy/3GIUsD4bIgLZ8QmOJk=; b=y4Tc0hgFvjX09Xs+yl lGLa5QWm0MJuDs41H5hmcy+ejxHCYhF804C0BlWTLD27iPuJeJpfBUY3r0HTSzlp ocnzo7xVa7vmQVQarC/cBzrSFSW4i2KJt12zeqbZImHmL20PVc0XmOICps3wNcA+ Aane33SJ/FK5IMBBBC4na8+lA= Received: by filter0560p1mdw1.sendgrid.net with SMTP id filter0560p1mdw1.1613.5580B23617 2015-06-16 23:33:10.593027741 +0000 UTC Received: from herokuapp.com (ec2-54-80-18-132.compute-1.amazonaws.com [54.80.18.132]) by ismtpd-022 (SG) with ESMTP id 14dfeb824db.5273.f4901 Tue, 16 Jun 2015 23:33:10 +0000 (UTC) Date: Tue, 16 Jun 2015 23:33:10 +0000 From: luke.gru@gmail.com To: ruby-core@ruby-lang.org Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Redmine-MailingListIntegration-Message-Ids: 44152 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 11264 X-Redmine-Issue-Author: luke-gru X-Redmine-Sender: luke-gru X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: ync6xU2WACa70kv/Ymy4QrNMhiuLXJG8OTL2vJD1yS4g9H6hCIKOEeTMcthWYXMrbijmqytqtoEtNo QCazb//a9cCrlZt+whc5WUeeQ5vVYljgX1Uxr7KmTH3J/vEV0PZVsnZzVmEgNErEPrb2BZFTYmeszP XU6UT3DqwOYXDtHKhbGtH76qE4RsPfiubvvQ X-ML-Name: ruby-core X-Mail-Count: 69622 Subject: [ruby-core:69622] [Ruby trunk - Bug #11264] Memory leak in JSON stdlib ext (JSON generation) X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #11264 has been updated by Luke Gruber. Thanks for getting back to me. That's true, but my guess is the most common leak scenario would be when calling a method that's basically an extension point to the library itself (such as `to_json`. At least if you leak the internals of the buffer, it won't be too many bytes per leak :) Otherwise you would have to wrap the whole 'cState_partial_generate' method in a rescue block. As the library stands now, lots of methods that raise will result in a leak, not just `to_json`: Object#respond_to?(:to_json) Object#to_s Array#[] Hash#keys Hash#[] String#encode But the chances of those methods raising errors are much lower, I think. Should we continue this here or on https://github.com/flori/json/issues/251 ? I don't know what the right approach is for std libs that are maintained separately as gems. ---------------------------------------- Bug #11264: Memory leak in JSON stdlib ext (JSON generation) https://bugs.ruby-lang.org/issues/11264#change-52965 * Author: Luke Gruber * Status: Third Party's Issue * Priority: Normal * Assignee: * ruby -v: 2.2-head * Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN ---------------------------------------- Hi, I'm not sure if this is a bug, or just undocumented behaviour, but here's a script to reproduce the memory leak: ~~~ruby require 'json' class MyClass def to_json(*) "a" * 1048576 # 1 megabytes of chars end end class MyOther def to_json(*) raise "OMG" end end 1000.times do |i| # will leak up to ~ 4 gigs puts i JSON.dump([MyClass.new, MyClass.new, MyClass.new, MyOther.new]) rescue nil end ~~~ What's happening is that the C extension is iterating over the array to eventually dump it out to JSON. It's going through the array in order, appending to the `fbuffer` as needed. The problem is that that the API extension point of adding a `to_json` method to a class (or object), without wrapping the code in some sort of 'begin...rescue , free(buffer), re-raise' block results in the buffer never being freed. Normally this isn't too bad, except if a lot of data was appended to the buffer before the error got raised. To test it against normal behaviour in the above script, take out the offending `MyOther.new` in the array. It should run much more smoothly this way :) Note that since the `fbuffer`s aren't GC marked (not that they should be), it isn't possible to trace this leak using `GC.stat`. Once again, not sure if this is a bug or if we should never raise errors from custom `to_json` methods (ie: always wrap them in a begin... rescue block. Thanks, I also reported this to the JSON gem maintainer here: https://github.com/flori/json/issues/251 -- https://bugs.ruby-lang.org/