From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS17314 8.43.84.0/22 X-Spam-Status: No, score=-3.6 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RDNS_DYNAMIC,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 11F301F8C6 for ; Mon, 13 Sep 2021 13:47:21 +0000 (UTC) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 297E53857C6C for ; Mon, 13 Sep 2021 13:47:20 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 297E53857C6C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1631540840; bh=x5Jv5PkA1OhNe/CvaoUeOInFKSl1JzjKJCvLdKDLwQU=; h=To:Subject:Date:References:In-Reply-To:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=I+XJcxU87KSosPBDmx0cTHV8C4lNhbGBUJUE/AA9iFuppTzZe3YZ+e2TB1lIhp1mu FWcGSH8RBgdJRE+c9OPpCFrXdA+ZOQTJbp/w3apOKNruB78ehPPA+KAPSymqkxcsCR 1scYyNbQe3NLVhMcRAzMjm65pOWPh93pC3Mw7g1Y= Received: from esa4.fujitsucc.c3s2.iphmx.com (esa4.fujitsucc.c3s2.iphmx.com [68.232.151.214]) by sourceware.org (Postfix) with ESMTPS id C8075385841A for ; Mon, 13 Sep 2021 13:46:58 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C8075385841A X-IronPort-AV: E=McAfee;i="6200,9189,10105"; a="47003532" X-IronPort-AV: E=Sophos;i="5.85,290,1624287600"; d="scan'208";a="47003532" Received: from mail-os2jpn01lp2059.outbound.protection.outlook.com (HELO JPN01-OS2-obe.outbound.protection.outlook.com) ([104.47.92.59]) by ob1.fujitsucc.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Sep 2021 22:46:57 +0900 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OeJdrA46Wu+I1//GM1VvmgkSnhe5V+NwNdsjzOTLCcukEdG3eH/UDU8hOfmfFbpi7/jRU0xz/gZF0k6FRdp1AZ0Z7hbFBDTXxDG/ct0qxBVwf6UCH/QU+PFuOs5atGZu2tva0+aM5srVoGHDKhbiIRe12a4c7YijkdS3duy+q+Y3x7JM9mHS7v9elJedo/DK+9u+kamd6TrLT4albkAQ1r3TuDfLL76zvpTtQuvLGbuB0Vqlxn7xQKvxtlLNqLH5Wg9UuLxezTSdyU2DyvxruZiaGhWE9uc2TS7k2milG63nIr0CK825JAm7xWzfFY8bCLKGPzO4oIgH1JapwSiXpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=x5Jv5PkA1OhNe/CvaoUeOInFKSl1JzjKJCvLdKDLwQU=; b=ACDSBtpeN1aelCBYUKFdIKabbQT3m+cBWabuS5bdqRb3st8/Ze229Vsh1u8+9EcMShl1LE1a4dXOVmdRYFCkumjJGhkSwUiaLnBOQ+vE/NkcxVbf44ivRZFJbb7XUVMexcglQWo5Xmwu1JPssZMYMXX5ZWoi/qGZF4SZLml/1MJjWaCu/FvzsmMUkC0XWcMlbozvVmZ1gXph/kwJRzJO6Vc6TXVbHufTaDqsxBNpINCo6/z5ze4jjjKBsB3caE860icL9Z0JZPEhP+6f0qzUbljtzgtMF2ZJaC0nRKIyG8xJZUskg6V4P1CMJDai14et7yzBCewVHPfncClkHSHLHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fujitsu.com; dmarc=pass action=none header.from=fujitsu.com; dkim=pass header.d=fujitsu.com; arc=none Received: from TYAPR01MB6025.jpnprd01.prod.outlook.com (2603:1096:402:36::13) by TYAPR01MB5322.jpnprd01.prod.outlook.com (2603:1096:404:803c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.18; Mon, 13 Sep 2021 13:46:54 +0000 Received: from TYAPR01MB6025.jpnprd01.prod.outlook.com ([fe80::cde6:2645:63a1:51b]) by TYAPR01MB6025.jpnprd01.prod.outlook.com ([fe80::cde6:2645:63a1:51b%9]) with mapi id 15.20.4500.018; Mon, 13 Sep 2021 13:46:54 +0000 To: Siddhesh Poyarekar , "libc-alpha@sourceware.org" Subject: Re: [PATCH v3 4/5] benchtests: Fix validate_benchout.py exceptions Thread-Topic: [PATCH v3 4/5] benchtests: Fix validate_benchout.py exceptions Thread-Index: AQHXic7Bp/iXfsAbo02ySKj9XMNU3quhjwcAgAACOgCAAKOrSw== Date: Mon, 13 Sep 2021 13:46:54 +0000 Message-ID: References: <20210805074733.433430-1-naohirot@fujitsu.com> <20210805075144.433644-1-naohirot@fujitsu.com> <2f0293c7-d0b2-0a3b-1ff9-48127837d3c5@gotplt.org> In-Reply-To: <2f0293c7-d0b2-0a3b-1ff9-48127837d3c5@gotplt.org> Accept-Language: en-001, ja-JP, en-US Content-Language: aa X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_a7295cc1-d279-42ac-ab4d-3b0f4fece050_Enabled=True; MSIP_Label_a7295cc1-d279-42ac-ab4d-3b0f4fece050_SiteId=a19f121d-81e1-4858-a9d8-736e267fd4c7; MSIP_Label_a7295cc1-d279-42ac-ab4d-3b0f4fece050_SetDate=2021-09-13T13:46:53.562Z; MSIP_Label_a7295cc1-d279-42ac-ab4d-3b0f4fece050_Name=FUJITSU-RESTRICTED; MSIP_Label_a7295cc1-d279-42ac-ab4d-3b0f4fece050_ContentBits=0; MSIP_Label_a7295cc1-d279-42ac-ab4d-3b0f4fece050_Method=Standard; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 10a109d5-e9f8-4430-d1a2-08d976bcf256 x-ms-traffictypediagnostic: TYAPR01MB5322: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7691; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: AtvGZpJtc6Qih1JCfkHsKa3Aq/YK+q7E30KR9Q/7h2SazQ/dJFZ5sHbw5Wsf97Mdxk/1tmGyD8m3b7xjxO+kMjAzEAZyiJOrBS70nuzIjTSU3F23lVGp8CTmeEvIJJnhFLBQlj6T3BwFTVVuKOMjKdew64WBRtXEyNJITVj4tAxEK8jX3ck9g8H6Pvu4t8dIAN5R4H9Rdd8QHhEdtyLYHCvNe6KhNVVGRtvwtRaJ8pjYHKPSpE+Y3xdkZCFezh/W2qpqVc1b9599yvmQ+MAgDC8wuJtzPLDF1N9XVBDzlSf8frTuKIONhS14b78cgogDL6npsAyDooq7z1lbN5XrvaRKSkq1B5gFtiLk6LFdQBM9SeQI1Iwz0hPLhOFvJqULlP9SnE9BW4IEYIqHIgKhCvnW7pZCZimngODmkoJCkeLs85J9W2131yfP5QqLipB3XrEZJRbPOEVh7lDZC8N4DSqbf+YyUyncuY0yIRJGGqJ+7UjexmjnYMvRTA185rfutRytQ/u18TcKu96duCUrOR8ZJ0uXvpcJi4ubMRrXstUWXkaz/ugEf8fzuEcQKmvQBpVFWES5lJLQOIARPt7wq0x2jtv9eiXVazcC2b4T9mwQ0hwUGxBB+fQQL+qvnoG99Vxf4ElHAkvvZj4y6ujl22Ht+xAWrP1Ms/0QAoQWvLCHgdRtb3b7hqvlvYigSSSj4GB5uMgElM5FkNHn9OHFpcB3MxJODfVRbKs4VbUSCD/NtZp2Eb+9feBkXSgqRMpu7vFbwrjAjCvEcXPxqWpT3zs/q+GjinCQkfASvOJxEP4= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:TYAPR01MB6025.jpnprd01.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(396003)(136003)(346002)(39860400002)(366004)(110136005)(122000001)(316002)(86362001)(15650500001)(85182001)(38100700002)(966005)(8676002)(8936002)(71200400001)(5660300002)(478600001)(33656002)(186003)(66556008)(76116006)(55016002)(52536014)(83380400001)(66446008)(64756008)(7696005)(66946007)(66476007)(9686003)(38070700005)(6506007)(53546011)(2906002)(26005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-2022-jp?B?VFlzRU1heW0vTjYrZ1dqL0YxTzRMaHRjVzIrWWRhV3hHbFZLYk9IMHFC?= =?iso-2022-jp?B?M2E3eSt6a2xTQzBmNUs0cnVNMlo3eUozcVcyd1pCc29qY3VhbjBmT2xJ?= =?iso-2022-jp?B?QzRvNGFGS1kzTklUNXB5OGF1ZGFQUVYyaTVnUzNoQ1FEYUxxUXM1RWVX?= =?iso-2022-jp?B?QmpVMHhqbXc3M3RaNVU3QUZiUm14a1ppU2JDbmVvZ25BWGw3TmFmbUJC?= =?iso-2022-jp?B?RTRTTGh0b1RFK0RQQkJCc1pNeGd3THpUb2lPQXlsdlIzeXJIbWh3SXVT?= =?iso-2022-jp?B?em1iajNZeUk0WUd2bHA0UjhzMm5kbzY0Q05hU2tyRWlLSUdqMUxBV25i?= =?iso-2022-jp?B?Ym10SmxXV0VKSmx2Vmt4L2RSdnJOM256VWxPaERuWjlLZERUclEwZ3ZD?= =?iso-2022-jp?B?TmxDd2hDMW14SHNWbWZ5RFVMLzVCNFdDMmxUcGJRb2VmM1ROanBDRzdw?= =?iso-2022-jp?B?VnRTbVBtVElWL3laam9aMTNreG8xRGtpZGQ1eGl2S2pkNHp5ODJOZi9q?= =?iso-2022-jp?B?RS9lbU0xbWF2NEhrUHRFOHQ5YlYrV1haRTdkdXhvQks3VWcveENPK1Ax?= =?iso-2022-jp?B?QlZtRS8zR1JZejN0VCsrWDVaWkRuUkdDM2FrR2NWbWtzN3dQb2NBYmpS?= =?iso-2022-jp?B?SXMwOWY5SlB1UWJoOVlIOVhBOThWcGhKUkdRU3BUdTczTjBQZENzMjdM?= =?iso-2022-jp?B?VU9YMWpOY2o0aFhkZThrN09aS2Z4TldzclZPOVo5dmJYemhDUWZKbEV4?= =?iso-2022-jp?B?ZWt6UUFKTTRGSGxZT2lhM1l1MDhya1pJOXdiWCsxMiswckxCeWF3UXBm?= =?iso-2022-jp?B?ajh0R0xrU28rUExScklBdXBTblRRSDJ5VjlETDl3OGlMeUNrQk1kaU5l?= =?iso-2022-jp?B?MkdUczhDV2ROVkkrN0JjVTBiOWt2aEtxcWI1Ymw0VXVlbE8xcklmVDhs?= =?iso-2022-jp?B?YTV5eW8vaGVKMTd0YjF4RGV1WnRVK0FoUWxkU1dmUlduSHZCTWt5VEsv?= =?iso-2022-jp?B?VDdmQjFDdmtBYkkxU2FkN210ODNrSU5xK2phWFVFWW5mSmRFd21nWm04?= =?iso-2022-jp?B?bUwxMVpDeWpxbnJ3cmhZU0xPbGV6c1hJVk5kMHBxNnpWMGFpS280dXYw?= =?iso-2022-jp?B?NWcxeVlBL1NJVFZlbUs3Umo4L2xCUEIxb2ZtUVdXdUFvbWlRaWhYWDMv?= =?iso-2022-jp?B?bWpIZ0NZZFh2dDBjVkM2VmRuc2xZMkE0RlFzMWJpNVRXT1JZN0pUS1U4?= =?iso-2022-jp?B?YmU0WlBvOHorR3JBb3FHMy91VmFjY044dUhKdTY5Z2dwSll6cWtUM2RK?= =?iso-2022-jp?B?OXdLT2RzVFJxaG9wU09xOUZwYm82UHRMa0g1OHZTVWVOZ3JPaFA5MWJy?= =?iso-2022-jp?B?RkxKY1p4ei9xUnkzR3Q1QmtsM2hkUStPampQSTJZMnREYWxGdzZ4Q2pX?= =?iso-2022-jp?B?QUVSeHBDbWNXbU4wcEVMZm44UCtSV01xaXpEZ0NjVnE4TEMrbVFQdlR3?= =?iso-2022-jp?B?cW5mNE81eWxzM09FUDd3dExiclRpMmhmamV3NFVha0lJUWYwRGIxelZB?= =?iso-2022-jp?B?ZmdIY2JEMzZFWWh2N3NKZlJjaE5HaU9qY1creFg1NWsyUmhCWmtHV0wr?= =?iso-2022-jp?B?dHdBZ3kraGovclkyZDNqNTcvSVRnaExSanQrUGxXclJXSkJGMHEyNGRM?= =?iso-2022-jp?B?cXc4bGovbFRXbUdYMGFIa3J5QUpjK05EWm5SM1crVEY2WHFha0lOay9Y?= =?iso-2022-jp?B?dWNIWDZNaXB3RVVXTkJZUkMwMmU3aldvY2FpVHpXZmRqejZ3K29lVVpK?= =?iso-2022-jp?B?VlpsTWdOYk85YlJRdXFZamlQdlBCN2ZOMkM2TzIwQlFCUGJ2VkdFM1Za?= =?iso-2022-jp?B?eXJzV3V6b1NuUHFDNGhtUFEzY1R3PQ==?= Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: fujitsu.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: TYAPR01MB6025.jpnprd01.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 10a109d5-e9f8-4430-d1a2-08d976bcf256 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Sep 2021 13:46:54.0624 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: XcJr20Fv6r/Pp3vKoI6+6C7Kj1dH0VG/2vkYnaiP3bBPgKko+mnBrICaa9VMO23Ssjs//PWXSfVXld88Bh2S9w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYAPR01MB5322 X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: naohirot--- via Libc-alpha Reply-To: "naohirot@fujitsu.com" Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org Sender: "Libc-alpha" Hi Siddhesh,=0A= =0A= Thank you for the two merges and this review.=0A= =0A= > From: Siddhesh Poyarekar =0A= > Sent: Monday, 13 September 2021 12:50=0A= > On 9/13/21 9:12 AM, Siddhesh Poyarekar wrote:=0A= > >> --- a/benchtests/scripts/import_bench.py=0A= > >> +++ b/benchtests/scripts/import_bench.py=0A= > >> @@ -104,7 +104,10 @@ def do_for_all_timings(bench, callback):=0A= > >> """=0A= > >> for func in bench['functions'].keys():=0A= > >> for k in bench['functions'][func].keys():=0A= > >> - if 'timings' not in bench['functions'][func][k].keys():= =0A= > >> + try:=0A= > >> + if 'timings' not in bench['functions'][func][k].keys(= ):=0A= > >> + continue=0A= > >> + except AttributeError:=0A= > >> continue=0A= > >=0A= > > When do you get an AttributeError here?=0A= > >=0A= > =0A= > OK the one possibility I can think of is when=0A= > bench['functions'][func][k] is None. This implies the existence of a=0A= > benchmark output that has a function variant without any inputs and=0A= > hence, without any benchmark data. That should be invalid, in which=0A= > case the benchmark should be fixed, not the validator.=0A= =0A= AttributeError unconditionally occurs with a correct JSON benchout=0A= file such as below because the code=0A= "bench['functions'][func][k].keys()" is either "bench-variant",=0A= "ifunc", or "results" that doesn't have keys()."=0A= =0A= $ ~/glibc/benchtests/scripts/validate_benchout.py bench-memcpy.out \=0A= ~/glibc/benchtests/scripts/benchout_strings.schema.json=0A= Traceback (most recent call last):=0A= File "/home/naohirot/work/github/glibc/benchtests/scripts/validate_bencho= ut.py", line 86, in =0A= sys.exit(main(sys.argv[1:]))=0A= File "/home/naohirot/work/github/glibc/benchtests/scripts/validate_bencho= ut.py", line 69, in main=0A= bench.parse_bench(args[0], args[1])=0A= File "/home/naohirot/work/github/glibc/benchtests/scripts/import_bench.py= ", line 139, in parse_bench=0A= do_for_all_timings(bench, lambda b, f, v:=0A= File "/home/naohirot/work/github/glibc/benchtests/scripts/import_bench.py= ", line 107, in do_for_all_timings=0A= if 'timings' not in bench['functions'][func][k].keys():=0A= AttributeError: 'str' object has no attribute 'keys'=0A= =0A= $ cat bench-memcpy.out=0A= 1 {=0A= 2 "timing_type": "hp_timing",=0A= 3 "functions": {=0A= 4 "memcpy": {=0A= 5 "bench-variant": "default",=0A= 6 "ifuncs": ["generic_memcpy", "__memcpy_thunderx", "__memcpy_thunderx= 2", "__memcpy_falkor", "__memcpy_simd", "__memcpy_a64fx", "__memcpy_generic= "],=0A= 7 "results": [=0A= 8 {=0A= 9 "length": 1,=0A= 10 "align1": 0,=0A= 11 "align2": 0,=0A= 12 "dst > src": 0,=0A= 13 "timings": [10.9326, 11.0449, 11.5515, 13.5693, 11.5198, 6.77368, = 11.5259]=0A= 14 },=0A= ...=0A= =0A= I found out that the implementation is not right, and sent V4 patch [1].=0A= Please find it.=0A= =0A= [1] https://sourceware.org/pipermail/libc-alpha/2021-September/130915.html= =0A= =0A= Thanks.=0A= Naohiro=0A= =0A=