From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=AWL,BAYES_00, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS, UNPARSEABLE_RELAY shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id E96751F66E for ; Tue, 18 Aug 2020 18:57:44 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id B1CBF120A20; Wed, 19 Aug 2020 03:57:08 +0900 (JST) Received: from xtrwkhkc.outbound-mail.sendgrid.net (xtrwkhkc.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id 65903120A1F for ; Wed, 19 Aug 2020 03:57:06 +0900 (JST) Received: by filterdrecv-p3las1-5785d94c6b-4nt5m with SMTP id filterdrecv-p3las1-5785d94c6b-4nt5m-19-5F3C249D-51 2020-08-18 18:57:34.028534574 +0000 UTC m=+440882.291282493 Received: from herokuapp.com (unknown) by ismtpd0016p1iad2.sendgrid.net (SG) with ESMTP id E0g9iqWFSMS5dw2_5KPwBw for ; Tue, 18 Aug 2020 18:57:33.884 +0000 (UTC) Date: Tue, 18 Aug 2020 18:57:34 +0000 (UTC) From: tenderlove@ruby-lang.org Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 75459 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Bug X-Redmine-Issue-Id: 17023 X-Redmine-Issue-Author: larskanis X-Redmine-Issue-Assignee: tenderlovemaking X-Redmine-Sender: tenderlovemaking X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?lWh=2FbSnEjJFRTUHRj45oOQBme8zohdTU9K3TxrDAbWKlhr24VXh=2Fle0lUhyphb?= =?us-ascii?Q?eft2gYVHgPqFYyP+9=2F0IB0HZc8rYEyeyMdgp7W0?= =?us-ascii?Q?LJf=2F5ZY3ZJMITrERnlh=2FG3gZ0wlCOAV=2F=2FIbjvd2?= =?us-ascii?Q?OCniH0fOnLxzYSAdC9E0aIrgComp4PaUtenSpPh?= =?us-ascii?Q?pZGreOqfyK1mFQDm0RKNfPrjbuVJYGj=2FNS0qb4I?= =?us-ascii?Q?fDlTahjH82JvyxADs=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 99631 Subject: [ruby-core:99631] [Ruby master Bug#17023] How to prevent String memory to be relocated in ruby-ffi X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #17023 has been updated by tenderlovemaking (Aaron Patterson). larskanis (Lars Kanis) wrote in #note-19: > @tenderlovemaking The issue in FFI is still unsolved. Could you please have a look at the questions above? Or may I open a new issue for discussion? Hi Lars, Sorry I totally missed your last email. The main issue with solution 2 is that there is no place to store that information. The GC doesn't keep any persistent data about objects between (major) GC cycles. To support 2 we would have to store the data in the string itself, and I'm not sure if we have header bits available to do that. The second problem with solution 2 is that this is not a problem unique just to strings. If an FFI extension holds a reference to any VALUE (maybe by stuffing it in a void* or something) and doesn't rb_gc_mark the value, the GC will assume it is safe to move that object. Fiddle has the same problem, so I introduced a "pinning reference" here: https://github.com/ruby/fiddle/pull/44 All it does is hold a reference to the object and call `rb_gc_mark` on the reference at mark time. I think adding something like ^^^ to the FFI gem and asking users to wrap objects they know will hold references is the safest solution. As you mention, this issue only impacts objects that are stored off the stack, so it should be pretty rare for anyone to need this. Another idea I had was to figure out some way to have FFI automatically wrap constants with the above proxy object, but I didn't get that far. ---------------------------------------- Bug #17023: How to prevent String memory to be relocated in ruby-ffi https://bugs.ruby-lang.org/issues/17023#change-87112 * Author: larskanis (Lars Kanis) * Status: Closed * Priority: Normal * Assignee: tenderlovemaking (Aaron Patterson) * ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux] * Backport: 2.5: DONTNEED, 2.6: DONTNEED, 2.7: DONE ---------------------------------------- [ruby-ffi](https://github.com/ffi/ffi) allows to pass String objects to C by using the `:string` argument type. This way the string memory returned by `RSTRING_PTR` is passed to the C function. The user has to ensure on Ruby level that the string isn't GC'ed - as long as it is used on C level. That's the contract and this worked with all past ruby versions, but ruby-2.7 introduced `GC.compact`, which can relocate strings to another memory location. This example shows the situation and that the string is relocated although it is still referenced in ruby code: ```ruby File.write "string-relocate.c", <<-EOC static char *g_str; void set(char* str) { g_str = str; } char* get() { return g_str; } EOC system "gcc -shared -fPIC string-relocate.c -o string-relocate.so" require 'ffi' class Foo extend FFI::Library ffi_lib File.expand_path('string-relocate.so') attach_function :set, [:string], :void attach_function :get, [], :string def initialize(count) proc {} # necessary to trigger relocation a = "a" * count set(a) GC.verify_compaction_references(toward: :empty, double_heap: true) puts "get(#{count}): #{get} (should be: #{a})" end end Foo.new(23) Foo.new(24) ``` The output looks like so on ruby-2.7.1: ``` get(23): (should be: aaaaaaaaaaaaaaaaaaaaaaa) get(24): aaaaaaaaaaaaaaaaaaaaaaaa (should be: aaaaaaaaaaaaaaaaaaaaaaaa) ``` So using `GC.compact` while a string parameter is in use, both on Ruby and on C level, can cause invalid memory access. How can this prevented? A C extension is expected to use `rb_gc_mark()` in order to pin the VALUE to a memory location. But I couldn't find a way to pin a `VALUE` at the time the argument is passed to the C function, which is the only point in time ruby-ffi has access to it. ---Files-------------------------------- string-relocate.rb (653 Bytes) 0001-Only-marked-objects-should-be-considered-movable.patch (1.23 KB) -- https://bugs.ruby-lang.org/