From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS17314 8.43.84.0/22 X-Spam-Status: No, score=-3.3 required=3.0 tests=AWL,BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,PDS_RDNS_DYNAMIC_FP,RCVD_IN_DNSWL_MED,RDNS_DYNAMIC, SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 559441F8C6 for ; Wed, 14 Jul 2021 14:14:15 +0000 (UTC) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 2EE9F3985811 for ; Wed, 14 Jul 2021 14:14:14 +0000 (GMT) Received: from hedgehog.birch.relay.mailchannels.net (hedgehog.birch.relay.mailchannels.net [23.83.209.81]) by sourceware.org (Postfix) with ESMTPS id 051723860C37; Wed, 14 Jul 2021 14:14:01 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 051723860C37 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=gotplt.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gotplt.org X-Sender-Id: dreamhost|x-authsender|siddhesh@gotplt.org Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id DE1F05A30E3; Wed, 14 Jul 2021 14:13:56 +0000 (UTC) Received: from pdx1-sub0-mail-a5.g.dreamhost.com (100-96-27-225.trex-nlb.outbound.svc.cluster.local [100.96.27.225]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 195255A30CD; Wed, 14 Jul 2021 14:13:56 +0000 (UTC) X-Sender-Id: dreamhost|x-authsender|siddhesh@gotplt.org Received: from pdx1-sub0-mail-a5.g.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384) by 100.96.27.225 (trex/6.3.3); Wed, 14 Jul 2021 14:13:56 +0000 X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|siddhesh@gotplt.org X-MailChannels-Auth-Id: dreamhost X-Obese-Trouble: 30fce3aa0c75e62e_1626272036524_3026049808 X-MC-Loop-Signature: 1626272036523:2269769204 X-MC-Ingress-Time: 1626272036523 Received: from pdx1-sub0-mail-a5.g.dreamhost.com (localhost [127.0.0.1]) by pdx1-sub0-mail-a5.g.dreamhost.com (Postfix) with ESMTP id CBA528B02B; Wed, 14 Jul 2021 07:13:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gotplt.org; h=subject:to :cc:references:from:message-id:date:mime-version:in-reply-to :content-type:content-transfer-encoding; s=gotplt.org; bh=bxvu6m wvnnQ4MpYFavKmBeVYIN4=; b=LuPiKlmU/0CBfFBnlDsTBHpLXcUkij1nBRVML0 teq+VCQIqdXvADpu2r5e4eLEGrh9yLk8P53uZDEI1XFt9FC6h5FtU/4RUkED3MWQ EiGIW6Ti16COitM/YO28Odn9Hm/ElUeche3mzi6xWIX/pL0HAbFNNGS8mUYz3l/P VE11A= Received: from [192.168.1.139] (unknown [1.186.101.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: siddhesh@gotplt.org) by pdx1-sub0-mail-a5.g.dreamhost.com (Postfix) with ESMTPSA id 134748AFE3; Wed, 14 Jul 2021 07:13:52 -0700 (PDT) Subject: Re: [PATCH v8 03/10] Remove __morecore and __default_morecore To: Guillaume Morin References: <20210713073845.504356-1-siddhesh@sourceware.org> <20210713073845.504356-4-siddhesh@sourceware.org> <9925362a-d278-a0ad-f504-ae08ca93f628@gotplt.org> <20210714125415.GA24678@bender.morinfr.org> X-DH-BACKEND: pdx1-sub0-mail-a5 From: Siddhesh Poyarekar Message-ID: <45ba79b4-02e6-fae5-ae9b-db6c8a01aecb@gotplt.org> Date: Wed, 14 Jul 2021 19:43:47 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: <20210714125415.GA24678@bender.morinfr.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fweimer@redhat.com, Siddhesh Poyarekar , libc-alpha@sourceware.org Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org Sender: "Libc-alpha" On 7/14/21 6:24 PM, Guillaume Morin wrote: > On 14 Jul 12:31, Siddhesh Poyarekar wrote: > I replied on the bug report becuase I had not seen this message. > > But basically, this breaks https://github.com/libhugetlbfs/libhugetlbfs/ > which is old (started in 2006) and commonly used library. The library > is plugging its own morecore() implementation to use hugetlb pages to > back the malloc heap. Thanks, AFAICT, libhugetlbfs only uses __morecore and not the rest of the interfaces. The thing is, __morecore and friends have been deprecated for a year and building anything with __morecore ought to give the deprecation warning. > This is definitely not a win for all libhugetlbfs users. We have no > equivalent solution. You're asking to find us an entire new malloc > implementation if there is one (or write one?). We have no way of keep > using glibc's malloc and hugetlb. I am not even sure there exists > an equivalent replacement. > > I understand there is some security concern about malloc hooks but then > why not allow morecore() substitution with a properly > documented/supported interface? Most users already LD_PRELOAD > libhugetlbfs so that would be an easy fix. I don't think we would like to export a different interface to do the same thing because it still doesn't allow us to clamp down further on the system malloc implementation. Besides, the morecore hooking interface is severely limited because it only affects the main arena. Non-main arenas have their own allocation techniques and it doesn't really make sense to have an interface to control just the main arena. > You're claiming that it's subtly broken. I'd like to understand how and > why this can't be fixed. Afaik people have been using libhugetlbfs's > morecore() in production for 15+ years without any issue. > > The only issue I am aware of is the one I reported about 5 years ago > (along with a reproducer and a patch). This problem can only be reached > if trimming is enabled in the morecore implementation (it's disabled in > libhugetlbfs for this very reason). There is only the one known issue that you reported but that may well be because the interface isn't well tested. There have been many changes to malloc over the years and I cannot say for certain that the idea that morecore could be anything other than brk has been consistently considered. We don't have tests to verify that either. Further, morecore limits its influence to the main arena and it is possible that future changes could break this in ways that we cannot anticipate today. Basically this is technical debt we've accumulated from the days when malloc assumed single threaded programs and was incrementally hacked to add in multi-threaded support. We'd like to clean this up. The malloc code in glibc is quite general purpose and has little dependency on libc internals. It may well be possible for you to copy the source files over into libhugetlbfs to implement the interposer within libhugetlbfs that does exactly what you need. In fact, that may even allow you to extend hugetlbfs support to non-main arenas. Siddhesh