From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS17314 8.43.84.0/22 X-Spam-Status: No, score=-3.1 required=3.0 tests=AWL,BAYES_00,BODY_8BITS, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, NICE_REPLY_A,RCVD_IN_DNSWL_MED,RDNS_DYNAMIC,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 697331F5AE for ; Thu, 17 Jun 2021 19:44:13 +0000 (UTC) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 99505398B8B8 for ; Thu, 17 Jun 2021 19:44:12 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 99505398B8B8 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1623959052; bh=lodeCpu9iYYt2LBtchfPEKR2QoiVmE81k8Ls1AMJVro=; h=Subject:To:References:Date:In-Reply-To:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=D/+sDMom80qizozOyyrBdpB0asWlHzap+wty3GkjW9opIYgInfCukwD+46qiSbxa8 9bTfVCFua2kFkkrm+oVBV8CaLAGsChw85bzELfktMjSfDcFn9P7j7DLwOv2ZsGbE4R Vj4KlWpukidS/WChgiEhmg9DVMJxD+fXydbDyto8= Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by sourceware.org (Postfix) with ESMTPS id 66BBC39960F2 for ; Thu, 17 Jun 2021 19:42:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 66BBC39960F2 Received: by mail-qt1-x82a.google.com with SMTP id p21so5713270qtw.6 for ; Thu, 17 Jun 2021 12:42:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=lodeCpu9iYYt2LBtchfPEKR2QoiVmE81k8Ls1AMJVro=; b=tFw+GHeZ3XnyvrvSy0eRH4NPDw7eDjGtqA9yJJcv8mV74YzaUBXupT5JcppxSXqd00 eQt87eVfzAKjdyqeJ4xvVzCA775PtosUuf85+0myBXiMuom+BTkPRp85quf4kLpaWzT5 gKD4jh8BMqUdewW9akQgXmP2DlU7gk6C0XUAIE997L371WWrxTly5iLjGWWgvbECn0Om bEhvo5iAQGIzqgMtYluMuY2I0VSjBNMN5Cqq7sj2ScLln5Es4UETCLjmNyLTRTSlY8yM prWbbpJLp8Xt/uh7mGzLLyuAkD1JXfHKU4W7arNeXoavWKOqDfhpLaCUuxIc6aSlXv/P ihRA== X-Gm-Message-State: AOAM531MgXZVmNf2kj9h1BLCxJjcmHmjYQQXabrG0n/1dPbcp+E1WjpO GmSGw0lC7QrMxI4S7pu5thMucw== X-Google-Smtp-Source: ABdhPJzxNliHTGWtjFqwIPgVBYJX1hvuaoAT04RT4Nvc0+Z8kOjZqo++/ZvwuNsveP27rYJjIywhLw== X-Received: by 2002:ac8:5895:: with SMTP id t21mr6865083qta.365.1623958926911; Thu, 17 Jun 2021 12:42:06 -0700 (PDT) Received: from [192.168.1.4] ([177.194.59.218]) by smtp.gmail.com with ESMTPSA id o14sm1180607qtx.62.2021.06.17.12.42.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 17 Jun 2021 12:42:06 -0700 (PDT) Subject: Re: A collection of LD_AUDIT bugs that are important for tools (with better formatting for this list) To: John Mellor-Crummey , libc-alpha@sourceware.org References: <8A8FF420-8316-4A22-AC4D-DA1F2D5625A5@rice.edu> Message-ID: <2fc830b9-35da-9b94-369f-4df683078a5c@linaro.org> Date: Thu, 17 Jun 2021 16:42:03 -0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <8A8FF420-8316-4A22-AC4D-DA1F2D5625A5@rice.edu> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Adhemerval Zanella via Libc-alpha Reply-To: Adhemerval Zanella Cc: Florian Weimer Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org Sender: "Libc-alpha" On 16/06/2021 14:55, John Mellor-Crummey via Libc-alpha wrote: > I was encouraged to notify this list about several LD_AUDIT bugs that > have significant impact on performance tools that we are developing > for Linux in general and US Department of Energy (DOE) parallel > supercomputers in particular. > > My team develops the HPCToolkit performance tools > (https://hpctoolkit.org, https://github.com/HPCToolkit/hpctoolkit) > under funding from the DOE. We have been modifying our measurement > subsystem to interpose itself between an application and the OS using > glibc’s LD_AUDIT capability. > > For such tools to succeed, we need many of LD_AUDIT’s features to > work. Over the last eight months, we identified six bugs on systems we > use. We believe that four are still problems in upstream glibc and two > may be fixed (as noted). > > We would like confirmed fixes for the following bugs in upstream > glibc: Thank you very much to approach us about the current state of audit interface from the user point of view. There are some recent discussion started by Ben Coyote Woodard some time ago that has stalled. I checked the issues you listed below with master (I can't really comment their status regarding the usual HPC distributions). > > ---------------------------------------------------------- > Priority | Issue > ————————————————————————————— > VERY | When using an auditor, there is an unacceptable > HIGH | performance degradation of over 10x for PLT > | calls to small procedures even when neither > | la_pltenter or la_pltexit is present. The fix provided by https://sourceware.org/legacy-ml/libc-alpha/2013-05/msg00888.html seems to fix, I am not sure why it has not been applied. It also seems to not show any regressions. I will respin the patch and check if we can add a testcase. > ---------------------------------------------------------- > HIGH | When auditing, a dlmopen of a shared library > | causes a SEGV. This does not show on master, I will need to pinpoint exactly which are the commits that actually fixed it. > ---------------------------------------------------------- > HIGH | la_symbind isn't always called when appropriate. > | We observed that glibc 2.26 calls la_symbind > | when appropriate; glibc 2.28 does not. I am not sure how to proper fix it, maybe an option is just to disable bind-now if we have any audit module enabled. We will need to discuss the possible implications. (btw your auditor-tests/symbind does not enable bind-now with either LD_BIND_NOW=1 or -Wl,-z,now). > ---------------------------------------------------------- > HIGH | glibc does not save all necessary registers > | (e.g. X8 - the indirect result register, truncated > | SIMD registers) when auditing on aarch64 since > | the beginning of time. This has been already discussed on the maillist: * NEON support: we have a proposed [1] patch that addresses it by extending the export struct used. Although the patch seems to fix the issue described by BZ#26643 it is still incomplete: it would required to bump LAV_CURRENT for aarch64 (and add a arch-specific way to override it), add tests, and check on how to handle possible incompatbilities. From a briefly chat with other glibc maintainer, we can just set that audit version lower LAV_CURRENT are just unsupported. * SVE support: as indicated by Szabolcs SVE calls are marked with the STO_AARCH64_VARIANT_PCS and thus explicit not supported by dynamic loader. To support SVE, it would require save/restore all registers (but pass down arg and retval registers to pltentry/exit callbacks according to the base PCS). Another option would be to use different LAV_CURRENT version, one for NEON and SVE with 128-bits and another for SVE larger than 256-bits. This also has performance implications. [1] https://patchwork.sourceware.org/project/glibc/patch/20200923160448.2321909-1-woodard@redhat.com/ [2] https://sourceware.org/pipermail/libc-alpha/2020-September/117822.html > ---------------------------------------------------------- > LOW | When auditing, a dlopen of a shared library > | that uses R_X86_64_TLSDESC causes a SEGV. This > | is reportedly fixed in glibc 2.34. It should be fixed by BZ#27137 (8f7e09f4dbdb5c815a18b8285fbc5d5d7bc17d86), however it has regressed by 03e187a41d9. We need the following fix and we definitely need a regression tests for this (I will probably use your auditor-test) as base: diff --git a/elf/dl-open.c b/elf/dl-open.c index d2240d8747..d2638ebf05 100644 --- a/elf/dl-open.c +++ b/elf/dl-open.c @@ -771,7 +771,7 @@ dl_open_worker (void *a) { struct link_map *libc_map = GL(dl_ns)[args->nsid].libc_map; #ifdef SHARED - bool initial = libc_map->l_ns == LM_ID_BASE; + bool initial = libc_map != NULL ? libc_map->l_ns == LM_ID_BASE : false; #else /* In the static case, there is only one namespace, but it contains a secondary libc (the primary libc is statically > ---------------------------------------------------------- > LOW | An auditor added to an executable at link time > | with --audit=auditor.so and noted in the DT_AUDIT > | entry of the dynamic section is not called at > | runtime. This is reportedly fixed in glibc 2.32. > ---------------------------------------------------------- > This has also fixed by BZ#24943 on glibc 2.32 > > > A repository of reproducers for these bugs can be found here: > https://github.com/hpctoolkit/auditor-tests. Thanks, this is really helpful. > > A detailed writeup of everything known about each of these bugs, > including links to Red Hat and Sourceware Bugzilla entries, if any are > known to exist, can be found here: > https://docs.google.com/document/d/1dVaDBdzySecxQqD6hLLzDrEF18M1UtjDna9gL5BWWI0/edit?usp=sharing > And you are correct about your assessment on the document, we *really* need more extensible tests for audit interface. It would be really helpful if we could add more tests to exercise the real world usage from tools that rely on audit modules. > Technical stakeholders for platforms that are HPCToolkit’s principal > targets under DOE funding, especially the exascale computing program: > > ---------------------------------------------------------- > Stakeholder | Why > ---------------------------------------------------------- > Intel | Prime contractor on Aurora exascale system at > | Argonne National Laboratory > ---------------------------------------------------------- > IBM | Prime contractor and processor vendor for Summit > | and Sierra supercomputers at Oak Ridge National > | Laboratory and Lawrence Livermore National > | Laboratory. > ---------------------------------------------------------- > ARM | Stakeholder who wants all ARM Linux platforms > | to succeed, including Sandia National Laboratory's > | Astra supercomputer and SUNY Stony Brook's > | A64FX-based Ookami. > ---------------------------------------------------------- > AMD | Processor vendor for Frontier and El Capitan > | exascale supercomputers at Oak Ridge and Lawrence > | Livermore National Laboratories. > ---------------------------------------------------------- > SuSE | Linux distribution provider for Cray systems to be > | delivered to Oak Ridge and Lawrence Livermore > | National Laboratories and the A64FX-based system > | installed at SUNY Stony Brook. > ---------------------------------------------------------- > Red Hat | Linux distribution provider for Oak Ridge > | National Laboratory s Summit, Lawrence Livermore > | National Laboratory s Sierra. > ---------------------------------------------------------- > Cray | Prime contractor and system vendor for Oak Ridge > | and Lawrence Livermore National Laboratories, > | and SUNY Stony Brook; system vendor for Argonne > | National Laboratory. > ---------------------------------------------------------- > > > For reference, here is a pointer to the portion of our tool that uses > the LD_AUDIT interface: > https://github.com/HPCToolkit/hpctoolkit/blob/master/src/tool/hpcrun/audit/auditor.c > > Here are some of the capabilities of LD_AUDIT that we need to work and why: > > - We use LD_AUDIT’s la_objopen and la_objclose to track what objects are > in an application’s address space so that our measurement subsystem > can unwind the call stack when a profiling signal is > received. Tracking libraries by wrapping dlopen is problematic for > several reasons. For instance, a wrapper would need to implement RPATH > and RUNPATH semantics because glibc does not provide an alternate > dlopen interface (like _dlsym) so that a wrapper can provide the > return address in the requesting library as an argument which glibc > needs to determine the R_PATH and RUNPATH to use when trying to find > the library and its dependencies. It is not clear to me what kind of extensible interface for audit you are asking here or if la_objopen/la_objclose is already suffice. > > - We want to use LD_AUDIT’s la_symbind32 and la_symbind64 to interpose > wrappers around key functions, e.g. pthread_create. This enables a > tool to intercept functions invoked through pointers obtained with > dlsym, which preloaded wrappers can’t do. (Note: We don’t use > la_symbind for interposition yet, but we plan to when it works > everywhere.) > > - We need auditing to work when an application or a tool library (e.g., > Intel’s GT-Pin) opens a shared library with dlmopen. > > - We need auditing to work when opening a dynamic library with TLS > dialect gnu2 relocations on x86_64 (R_X86_64_TLSDESC). We don’t have > any special interest in such relocations; at present, they cause a > SEGV when auditing and that must be avoided. This should be fixed minus the regression above. > > - We want to add an auditor to an application at link time, noted in the > DT_AUDIT entry of the dynamic section. Loading the DT_AUDIT entry as a > program is launched enables our profiler to be injected into an > application’s address space without a wrapper script that sets > LD_AUDIT and LD_PRELOAD. So if I understood correctly you are asking something like a DT_FILTER for DT_AUDIT? > > - LD_AUDIT needs to work on aarch64, which is an important target for > our tools. The fact that _dl_runtime_profile does not save register x8 > (the indirect result register) is often fatal for applications, which > makes LD_AUDIT unusable for any purpose on aarch64. > > - LD_AUDIT needs to support auditing of inter-object calls on aarch64 > when SVE registers are in use. > > As a final thing to consider: we understand that there is a tension > between security and auditability. We are concerned that changes being > considered for security may compromise observability for tools. For > tools, we would need a way to authorize full observability even in the > cases when that may theoretically reduce security. Perhaps setting > DT_AUDIT could be considered as authorizing full observability. > > -- > John Mellor-Crummey Professor > Dept of Computer Science Rice University > email: johnmc@rice.edu phone: 713-348-5179 >