From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.7 required=3.0 tests=AWL,BAYES_00,BODY_8BITS, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 892881F5AE for ; Wed, 16 Jun 2021 17:55:40 +0000 (UTC) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7D224396C80A for ; Wed, 16 Jun 2021 17:55:39 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7D224396C80A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1623866139; bh=up2Q4qUNHtCYGWmhdcyE2fQEb1znfvidznsKCpvhbBc=; h=Subject:Date:To:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=QAb+F1Chl/uglLridqvI/R1yFKTrs5HgASYIhyq3bQbZ1WnCOftKMZO2DCIL/kYUJ tmnNTiRDlr1J7gMzLpmiQa2lkWKubmgZi4Kgto2sVAhlvFg0dHZ9lqY9mUCPmaGgT6 OiCIIR8O6gNE/YDom60d/TqJtcLWWUh7g6IlvrkY= Received: from mx0a-0010f301.pphosted.com (mx0a-0010f301.pphosted.com [148.163.149.254]) by sourceware.org (Postfix) with ESMTPS id 4DC0F38618BB for ; Wed, 16 Jun 2021 17:55:19 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 4DC0F38618BB Received: from pps.filterd (m0102857.ppops.net [127.0.0.1]) by mx0b-0010f301.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15GHriqC019225 for ; Wed, 16 Jun 2021 12:55:18 -0500 Received: from mh2.mail.rice.edu (mh2.mail.rice.edu [128.42.201.21]) by mx0b-0010f301.pphosted.com with ESMTP id 3973fmh2q7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 16 Jun 2021 12:55:18 -0500 Received-X: from mh2.mail.rice.edu (localhost [127.0.0.1]) by mh2.mail.rice.edu (Postfix) with ESMTP id 7BC132528E4 for ; Wed, 16 Jun 2021 12:55:17 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by mh2.mail.rice.edu (Postfix) with ESMTP id 7AAC42528E1; Wed, 16 Jun 2021 12:55:17 -0500 (CDT) X-Virus-Scanned: by amavis-2.12.1 at mh2.mail.rice.edu, auth channel Received: from mh2.mail.rice.edu ([127.0.0.1]) by localhost (mh2.mail.rice.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id R6s0i-MPP7CQ; Wed, 16 Jun 2021 12:55:08 -0500 (CDT) Received: from [192.168.50.203] (c-98-200-175-18.hsd1.tx.comcast.net [98.200.175.18]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: johnmc@rice.edu) by mh2.mail.rice.edu (Postfix) with ESMTPSA id 8076C2178A1; Wed, 16 Jun 2021 12:55:08 -0500 (CDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) Subject: A collection of LD_AUDIT bugs that are important for tools (with better formatting for this list) Message-Id: <8A8FF420-8316-4A22-AC4D-DA1F2D5625A5@rice.edu> Date: Wed, 16 Jun 2021 12:55:07 -0500 To: libc-alpha@sourceware.org X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Proofpoint-GUID: 5T0N6WW5ElL5oaP0xdjlHS8h0w4vzHsA X-Proofpoint-ORIG-GUID: 5T0N6WW5ElL5oaP0xdjlHS8h0w4vzHsA X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-06-16_11:2021-06-15, 2021-06-16 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 bulkscore=0 phishscore=0 spamscore=0 adultscore=0 suspectscore=0 impostorscore=0 lowpriorityscore=0 clxscore=1015 mlxscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106160103 X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: John Mellor-Crummey via Libc-alpha Reply-To: John Mellor-Crummey Cc: John Mellor-Crummey Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org Sender: "Libc-alpha" I was encouraged to notify this list about several LD_AUDIT bugs that have significant impact on performance tools that we are developing for Linux in general and US Department of Energy (DOE) parallel supercomputers in particular. My team develops the HPCToolkit performance tools (https://hpctoolkit.org, https://github.com/HPCToolkit/hpctoolkit) under funding from the DOE. We have been modifying our measurement subsystem to interpose itself between an application and the OS using glibc=E2=80=99s LD_AUDIT capability. For such tools to succeed, we need many of LD_AUDIT=E2=80=99s features = to work. Over the last eight months, we identified six bugs on systems we use. We believe that four are still problems in upstream glibc and two may be fixed (as noted). We would like confirmed fixes for the following bugs in upstream glibc: ---------------------------------------------------------- Priority | Issue=20 = =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94 VERY | When using an auditor, there is an unacceptable HIGH | performance degradation of over 10x for PLT | calls to small procedures even when neither | la_pltenter or la_pltexit is present. ---------------------------------------------------------- HIGH | When auditing, a dlmopen of a shared library | causes a SEGV. ---------------------------------------------------------- HIGH | la_symbind isn't always called when appropriate. | We observed that glibc 2.26 calls la_symbind | when appropriate; glibc 2.28 does not. ---------------------------------------------------------- HIGH | glibc does not save all necessary registers | (e.g. X8 - the indirect result register, truncated | SIMD registers) when auditing on aarch64 since | the beginning of time. ---------------------------------------------------------- LOW | When auditing, a dlopen of a shared library | that uses R_X86_64_TLSDESC causes a SEGV. This | is reportedly fixed in glibc 2.34. ---------------------------------------------------------- LOW | An auditor added to an executable at link time | with --audit=3Dauditor.so and noted in the DT_AUDIT | entry of the dynamic section is not called at | runtime. This is reportedly fixed in glibc 2.32. ---------------------------------------------------------- A repository of reproducers for these bugs can be found here: https://github.com/hpctoolkit/auditor-tests. A detailed writeup of everything known about each of these bugs, including links to Red Hat and Sourceware Bugzilla entries, if any are known to exist, can be found here: = https://docs.google.com/document/d/1dVaDBdzySecxQqD6hLLzDrEF18M1UtjDna9gL5= BWWI0/edit?usp=3Dsharing Technical stakeholders for platforms that are HPCToolkit=E2=80=99s = principal targets under DOE funding, especially the exascale computing program: ---------------------------------------------------------- Stakeholder | Why ---------------------------------------------------------- Intel | Prime contractor on Aurora exascale system at | Argonne National Laboratory ---------------------------------------------------------- IBM | Prime contractor and processor vendor for Summit | and Sierra supercomputers at Oak Ridge National | Laboratory and Lawrence Livermore National | Laboratory. =20 ---------------------------------------------------------- ARM | Stakeholder who wants all ARM Linux platforms | to succeed, including Sandia National Laboratory's | Astra supercomputer and SUNY Stony Brook's | A64FX-based Ookami. ---------------------------------------------------------- AMD | Processor vendor for Frontier and El Capitan | exascale supercomputers at Oak Ridge and Lawrence | Livermore National Laboratories. =20 ---------------------------------------------------------- SuSE | Linux distribution provider for Cray systems to be | delivered to Oak Ridge and Lawrence Livermore | National Laboratories and the A64FX-based system | installed at SUNY Stony Brook. =20 ---------------------------------------------------------- Red Hat | Linux distribution provider for Oak Ridge | National Laboratory s Summit, Lawrence Livermore | National Laboratory s Sierra. ---------------------------------------------------------- Cray | Prime contractor and system vendor for Oak Ridge | and Lawrence Livermore National Laboratories, | and SUNY Stony Brook; system vendor for Argonne | National Laboratory. ---------------------------------------------------------- For reference, here is a pointer to the portion of our tool that uses the LD_AUDIT interface: = https://github.com/HPCToolkit/hpctoolkit/blob/master/src/tool/hpcrun/audit= /auditor.c Here are some of the capabilities of LD_AUDIT that we need to work and = why: - We use LD_AUDIT=E2=80=99s la_objopen and la_objclose to track what = objects are in an application=E2=80=99s address space so that our measurement = subsystem can unwind the call stack when a profiling signal is received. Tracking libraries by wrapping dlopen is problematic for several reasons. For instance, a wrapper would need to implement = RPATH and RUNPATH semantics because glibc does not provide an alternate dlopen interface (like _dlsym) so that a wrapper can provide the return address in the requesting library as an argument which glibc needs to determine the R_PATH and RUNPATH to use when trying to find the library and its dependencies. - We want to use LD_AUDIT=E2=80=99s la_symbind32 and la_symbind64 to = interpose wrappers around key functions, e.g. pthread_create. This enables a tool to intercept functions invoked through pointers obtained with dlsym, which preloaded wrappers can=E2=80=99t do. (Note: We don=E2=80=99= t use la_symbind for interposition yet, but we plan to when it works everywhere.) - We need auditing to work when an application or a tool library (e.g., Intel=E2=80=99s GT-Pin) opens a shared library with dlmopen. - We need auditing to work when opening a dynamic library with TLS dialect gnu2 relocations on x86_64 (R_X86_64_TLSDESC). We don=E2=80=99t= have any special interest in such relocations; at present, they cause a SEGV when auditing and that must be avoided. - We want to add an auditor to an application at link time, noted in = the DT_AUDIT entry of the dynamic section. Loading the DT_AUDIT entry as = a program is launched enables our profiler to be injected into an application=E2=80=99s address space without a wrapper script that = sets LD_AUDIT and LD_PRELOAD. - LD_AUDIT needs to work on aarch64, which is an important target for our tools. The fact that _dl_runtime_profile does not save register = x8 (the indirect result register) is often fatal for applications, which makes LD_AUDIT unusable for any purpose on aarch64. - LD_AUDIT needs to support auditing of inter-object calls on aarch64 when SVE registers are in use. As a final thing to consider: we understand that there is a tension between security and auditability. We are concerned that changes being considered for security may compromise observability for tools. For tools, we would need a way to authorize full observability even in the cases when that may theoretically reduce security. Perhaps setting DT_AUDIT could be considered as authorizing full observability. -- John Mellor-Crummey Professor Dept of Computer Science Rice University email: johnmc@rice.edu phone: 713-348-5179