From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-2.9 required=3.0 tests=AWL,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 87C4F1F5AE for ; Wed, 22 Jul 2020 09:46:10 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 8C38E1209E5; Wed, 22 Jul 2020 18:45:34 +0900 (JST) Received: from xtrwkhkc.outbound-mail.sendgrid.net (xtrwkhkc.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id 51E831209D9 for ; Wed, 22 Jul 2020 18:45:33 +0900 (JST) Received: by filterdrecv-p3las1-7754f7d4cc-fqqgm with SMTP id filterdrecv-p3las1-7754f7d4cc-fqqgm-19-5F180AD7-64 2020-07-22 09:46:00.001234886 +0000 UTC m=+2302341.180920085 Received: from herokuapp.com (unknown) by ismtpd0091p1iad2.sendgrid.net (SG) with ESMTP id zrpqm1DtQ2ilFYyMVonz3w for ; Wed, 22 Jul 2020 09:45:59.890 +0000 (UTC) Date: Wed, 22 Jul 2020 09:46:00 +0000 (UTC) From: larskanis@gmail.com Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 75051 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Bug X-Redmine-Issue-Id: 17023 X-Redmine-Issue-Author: larskanis X-Redmine-Issue-Assignee: tenderlovemaking X-Redmine-Sender: larskanis X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?+E+TGAYxgAqNvT2Hc3jgthCyK1YJRpYNPBZWWqZMzpVnJxe0Fd=2FegKlixUW9ns?= =?us-ascii?Q?77EaHmVL9uB2k1oyc2PGqX8y0LXMuhCb3NGl527?= =?us-ascii?Q?D+dP1e7sLrKlZKUJ159=2FW6Fy0T3IvK9IOsLm+0j?= =?us-ascii?Q?QiFHqT+IkLuu+Up9L6273UVz6=2FUw2JaHhbatARE?= =?us-ascii?Q?sg=2FyjLZ9uD+gTexOAbbDfn1sBReEfxdFLV34Bhk?= =?us-ascii?Q?q9FiRQZuv3VN4ixUE=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 99262 Subject: [ruby-core:99262] [Ruby master Bug#17023] How to prevent String memory to be relocated in ruby-ffi X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #17023 has been updated by larskanis (Lars Kanis). @Hanmac The string must be kept as a ruby object in an instance variable or constant, etc. and must not be modified as long as it's in use in the C library. Similar on the C side the string must not be modified and freeing it is not allowed. The details are the same since 10 years and are described here: https://github.com/ffi/ffi/wiki/Core-Concepts#memory-management Now ruby-2.7+ breaks this contract in some cases by moving the string value around. Raw String usage for `:pointer` or `:string` arguments is the fastest way to transfer data buffers from ruby to C. `FFI::MemoryPointer` is a more flexible alternative, but is way slower. ---------------------------------------- Bug #17023: How to prevent String memory to be relocated in ruby-ffi https://bugs.ruby-lang.org/issues/17023#change-86648 * Author: larskanis (Lars Kanis) * Status: Closed * Priority: Normal * Assignee: tenderlovemaking (Aaron Patterson) * ruby -v: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux] * Backport: 2.5: DONTNEED, 2.6: DONTNEED, 2.7: DONE ---------------------------------------- [ruby-ffi](https://github.com/ffi/ffi) allows to pass String objects to C by using the `:string` argument type. This way the string memory returned by `RSTRING_PTR` is passed to the C function. The user has to ensure on Ruby level that the string isn't GC'ed - as long as it is used on C level. That's the contract and this worked with all past ruby versions, but ruby-2.7 introduced `GC.compact`, which can relocate strings to another memory location. This example shows the situation and that the string is relocated although it is still referenced in ruby code: ```ruby File.write "string-relocate.c", <<-EOC static char *g_str; void set(char* str) { g_str = str; } char* get() { return g_str; } EOC system "gcc -shared -fPIC string-relocate.c -o string-relocate.so" require 'ffi' class Foo extend FFI::Library ffi_lib File.expand_path('string-relocate.so') attach_function :set, [:string], :void attach_function :get, [], :string def initialize(count) proc {} # necessary to trigger relocation a = "a" * count set(a) GC.verify_compaction_references(toward: :empty, double_heap: true) puts "get(#{count}): #{get} (should be: #{a})" end end Foo.new(23) Foo.new(24) ``` The output looks like so on ruby-2.7.1: ``` get(23): (should be: aaaaaaaaaaaaaaaaaaaaaaa) get(24): aaaaaaaaaaaaaaaaaaaaaaaa (should be: aaaaaaaaaaaaaaaaaaaaaaaa) ``` So using `GC.compact` while a string parameter is in use, both on Ruby and on C level, can cause invalid memory access. How can this prevented? A C extension is expected to use `rb_gc_mark()` in order to pin the VALUE to a memory location. But I couldn't find a way to pin a `VALUE` at the time the argument is passed to the C function, which is the only point in time ruby-ffi has access to it. ---Files-------------------------------- string-relocate.rb (653 Bytes) 0001-Only-marked-objects-should-be-considered-movable.patch (1.23 KB) -- https://bugs.ruby-lang.org/