From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS17314 8.43.84.0/22 X-Spam-Status: No, score=-4.2 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 922131F8C6 for ; Fri, 10 Sep 2021 11:03:00 +0000 (UTC) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 5CD2F385841B for ; Fri, 10 Sep 2021 11:02:59 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5CD2F385841B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1631271779; bh=jp1VO+9Kg2o5Z+I8VIFYLDzOJkR5jmuzu1wlu1e9MD4=; h=To:Subject:References:Date:In-Reply-To:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=WEj+EAbJP12+Y9m9zJNTAQcv2WH5Odm1W9pGnCW6szMUZuE2dWIaEV+bdYQ5vE/Ke r11CcXvb5EC1ILViANx5nkOsVVxc+LWqU09x1UGFmANR0MqHQ5BhMg0ybZQRRKRaQ6 JgxwxZjyo+N0tguw+4uIacaa1l6h1icM6eVEiAfs= Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id 3BA06384A8B3 for ; Fri, 10 Sep 2021 11:01:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3BA06384A8B3 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-492-pXG7HYSDMZiaSjCjTV-qdg-1; Fri, 10 Sep 2021 07:01:51 -0400 X-MC-Unique: pXG7HYSDMZiaSjCjTV-qdg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CCABA1800D41; Fri, 10 Sep 2021 11:01:49 +0000 (UTC) Received: from oldenburg.str.redhat.com (unknown [10.39.195.4]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3507451F00; Fri, 10 Sep 2021 11:01:48 +0000 (UTC) To: Huang Shijie Subject: Re: [PATCH] Add LD_NUMA_REPLICATION for glibc References: <20210903121434.12162-1-shijie@os.amperecomputing.com> <87bl5a6ul2.fsf@oldenburg.str.redhat.com> Date: Fri, 10 Sep 2021 13:01:46 +0200 In-Reply-To: (Huang Shijie's message of "Thu, 9 Sep 2021 10:19:09 +0000") Message-ID: <878s04k82t.fsf@oldenburg.str.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Florian Weimer via Libc-alpha Reply-To: Florian Weimer Cc: zwang@amperecomputing.com, Huang Shijie via Libc-alpha , patches@amperecomputing.com Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org Sender: "Libc-alpha" * Huang Shijie: > Hi Florian, > On Fri, Sep 03, 2021 at 08:28:57AM +0200, Florian Weimer wrote: >> * Huang Shijie via Libc-alpha: >> >> > This patch adds LD_NUMA_REPLICATION which influences the linkage of shared libraries at run time. >> > >> > If LD_NUMA_REPLICATION is set for program foo like this: >> > #LD_NUMA_REPLICATION=1 ./foo >> > >> > At the time ld.so mmaps the shared libraries, it will uses >> > mmap(, c->prot | PROT_WRITE, MAP_COPY | MAP_FILE | MAP_POPULATE,) >> > for them, and the mmap will trigger COW(copy on write) for the shared >> > libraries at the NUMA node which the program `foo` runs. After the >> > COW, the foo will have a copy of the shared library segment(mmap >> > covered) which belong to the same NUMA node. >> > >> > So when enable LD_NUMA_REPLICATION, it will consume more memory, >> > but it will reduce the remote-access in NUMA. >> >> I think the kernel could do this in a much better way, avoiding >> duplicating the pages within the same NUMA node. > > https://marc.info/?l=linux-kernel&m=163070220429222&w=2 > Since Linus did not think it a good choice to do it in kernel, > glibc is the only place to do it now. > So could you please re-evaluate this patch? The name of the environment variable is quite misleading. It should refer to MAP_POPULATE, not NUMA. As far as I can tell, it does not necessarily have the desired effect for multi-threaded applications (if some threads end up running on other NUMA nodes). And it would be helpful to have some performance numbers. And I wonder if a FUSE file system could do better, by making one backing copy per NUMA node instead of one copy per process. >> The other issue is the temporary RWX mapping, which does not >> interoperate well with some security hardening features. > Could you please tell me in detail? I am confused at it. Some environments block mapping files with PROT_WRITE | PROT_EXEC. Thanks, Florian