From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=3.0 tests=AWL,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 95FC61F5AE for ; Tue, 21 Jul 2020 20:19:01 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 57B5F120A57; Wed, 22 Jul 2020 05:18:28 +0900 (JST) Received: from xtrwkhkc.outbound-mail.sendgrid.net (xtrwkhkc.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id 5B9A5120A50 for ; Wed, 22 Jul 2020 05:18:26 +0900 (JST) Received: by filterdrecv-p3iad2-5b55dcd864-l2x6n with SMTP id filterdrecv-p3iad2-5b55dcd864-l2x6n-19-5F174DAD-2 2020-07-21 20:18:53.010410638 +0000 UTC m=+2171371.642160741 Received: from herokuapp.com (unknown) by geopod-ismtpd-1-0 (SG) with ESMTP id AQMi-QTkTzq2F5WYleL1rQ for ; Tue, 21 Jul 2020 20:18:52.888 +0000 (UTC) Date: Tue, 21 Jul 2020 20:18:53 +0000 (UTC) From: larskanis@gmail.com Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 75043 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Bug X-Redmine-Issue-Id: 17023 X-Redmine-Issue-Author: larskanis X-Redmine-Issue-Assignee: tenderlovemaking X-Redmine-Sender: larskanis X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?+E+TGAYxgAqNvT2Hc3jgthCyK1YJRpYNPBZWWqZMzpV8P4CBv5VrtXRfh=2FK9xd?= =?us-ascii?Q?cjkqhLHYnVyN9MpzECT+d6UwKQprDNFOblDtLbl?= =?us-ascii?Q?9qboLDQod2W=2F7bybrtxnskIfHGuNlsU4nqimY4t?= =?us-ascii?Q?T33AFTQj6wccrRSVog8jzfhfbcLahbmtvdDNrGl?= =?us-ascii?Q?UJYIQfGHuRfZsrb28M7ndZ6dE4PWMST11mier=2Fh?= =?us-ascii?Q?A6jAwJMYF56Uob9iE=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 99254 Subject: [ruby-core:99254] [Ruby master Bug#17023] How to prevent String memory to be relocated in ruby-ffi X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #17023 has been updated by larskanis (Lars Kanis). Sorry for not responding earlier! Unfortunately this issue still remains. So it is not closed and the backported commit doesn't fix it in any way. I tried Aarons hint to change all passed strings to non-embedded content. The result is here: https://github.com/ffi/ffi/pull/811 Unfortunately this adds a significant slowdown (up to 70%) and is not suitable for frozen strings. But frozen strings are equally relocated. These both issues makes the PR a no-go. Regarding `rb_gc_register_address()`: Enqueuing every string value passed to C would keep them alive forever, but we don't know how long the string is in use by the C library. We'd need some kind of finalizer hook to unregister them. But the finalizer is not called until `rb_gc_unregister_address()` was called. Regarding `ffi_register_global`: This could solve the issue, but establishes a new API which is mostly superfluous and breaks the current contract. Both causes (`GC.compact` and long string usage in C) are rarely used in combination. But only if they're used together, this new call would be needed. This is quite hard to explain to people. We would need a function that pins the VALUE for it's further lifetime, but without keeping it alive. ---------------------------------------- Bug #17023: How to prevent String memory to be relocated in ruby-ffi https://bugs.ruby-lang.org/issues/17023#change-86638 * Author: larskanis (Lars Kanis) * Status: Closed * Priority: Normal * Assignee: tenderlovemaking (Aaron Patterson) * ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux] * Backport: 2.5: DONTNEED, 2.6: DONTNEED, 2.7: DONE ---------------------------------------- [ruby-ffi](https://github.com/ffi/ffi) allows to pass String objects to C by using the `:string` argument type. This way the string memory returned by `RSTRING_PTR` is passed to the C function. The user has to ensure on Ruby level that the string isn't GC'ed - as long as it is used on C level. That's the contract and this worked with all past ruby versions, but ruby-2.7 introduced `GC.compact`, which can relocate strings to another memory location. This example shows the situation and that the string is relocated although it is still referenced in ruby code: ```ruby File.write "string-relocate.c", <<-EOC static char *g_str; void set(char* str) { g_str = str; } char* get() { return g_str; } EOC system "gcc -shared -fPIC string-relocate.c -o string-relocate.so" require 'ffi' class Foo extend FFI::Library ffi_lib File.expand_path('string-relocate.so') attach_function :set, [:string], :void attach_function :get, [], :string def initialize(count) proc {} # necessary to trigger relocation a = "a" * count set(a) GC.verify_compaction_references(toward: :empty, double_heap: true) puts "get(#{count}): #{get} (should be: #{a})" end end Foo.new(23) Foo.new(24) ``` The output looks like so on ruby-2.7.1: ``` get(23): (should be: aaaaaaaaaaaaaaaaaaaaaaa) get(24): aaaaaaaaaaaaaaaaaaaaaaaa (should be: aaaaaaaaaaaaaaaaaaaaaaaa) ``` So using `GC.compact` while a string parameter is in use, both on Ruby and on C level, can cause invalid memory access. How can this prevented? A C extension is expected to use `rb_gc_mark()` in order to pin the VALUE to a memory location. But I couldn't find a way to pin a `VALUE` at the time the argument is passed to the C function, which is the only point in time ruby-ffi has access to it. ---Files-------------------------------- string-relocate.rb (653 Bytes) 0001-Only-marked-objects-should-be-considered-movable.patch (1.23 KB) -- https://bugs.ruby-lang.org/