From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS17314 8.43.84.0/22 X-Spam-Status: No, score=-3.6 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RDNS_DYNAMIC,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 05E261F8C8 for ; Mon, 13 Sep 2021 13:45:19 +0000 (UTC) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 2C46A3857413 for ; Mon, 13 Sep 2021 13:45:18 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2C46A3857413 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1631540718; bh=WFsXuI/heZWJw2aOC9nkG28ic7dbMpWR8CGd5xJrwkQ=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=lbhPlyjHIIQ6bgHCHrpCzzO2sxoi/GmRMbU3CyKyigXtYFnNTmmjKzdmzbdzx7SWB Y7vimT3kctJSdCbhWh++xtyUZLmEf44+w9ugrzwVT95n+jAvRurZ+AUD7hoBwji/Fr zvynto4EsDy6qfwI3bd3/w4sC+jTIyDQxj880LUA= Received: from esa5.hc1455-7.c3s2.iphmx.com (esa5.hc1455-7.c3s2.iphmx.com [68.232.139.130]) by sourceware.org (Postfix) with ESMTPS id 779FB385740B for ; Mon, 13 Sep 2021 13:44:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 779FB385740B IronPort-SDR: IoGx9lYKfnZN4X0IBv+vS7RZ4QuC4ZK0OWCpO/BGGn2nk6VAqK7v1hYH7w76YMgwGX7u5N1h+9 JYFQXkjxDycthXXLs14IHEwBB7kbhAitrzKs1d1iTZnhDYyxMwh0hGfCIsUk9dsIYwwEhZ0G8C mMjij4GtpY4HAoErAcf0KNa9ManciyV+jcFiAjcSOaQfOI4+JUgAnfKQnuxrrdHh8fxF80q1Ez VlxSfKPFc6iZcrFfgKcoYx04SJiZGfYt+8a0Y57zlkGYVQQiXMBw6NZdifJumh3Xp849PTF7Mk aMr5RZBPPk6fOwGr3G2RX53T X-IronPort-AV: E=McAfee;i="6200,9189,10105"; a="44396525" X-IronPort-AV: E=Sophos;i="5.85,290,1624287600"; d="scan'208";a="44396525" Received: from unknown (HELO oym-r4.gw.nic.fujitsu.com) ([210.162.30.92]) by esa5.hc1455-7.c3s2.iphmx.com with ESMTP; 13 Sep 2021 22:44:44 +0900 Received: from oym-m1.gw.nic.fujitsu.com (oym-nat-oym-m1.gw.nic.fujitsu.com [192.168.87.58]) by oym-r4.gw.nic.fujitsu.com (Postfix) with ESMTP id 73EBAE0AE0 for ; Mon, 13 Sep 2021 22:44:42 +0900 (JST) Received: from m3050.s.css.fujitsu.com (msm.b.css.fujitsu.com [10.134.21.208]) by oym-m1.gw.nic.fujitsu.com (Postfix) with ESMTP id 935F3C520E for ; Mon, 13 Sep 2021 22:44:41 +0900 (JST) Received: from bionic.lxd (unknown [10.126.53.116]) by m3050.s.css.fujitsu.com (Postfix) with ESMTP id 7572AAD; Mon, 13 Sep 2021 22:44:41 +0900 (JST) To: Siddhesh Poyarekar , libc-alpha@sourceware.org Subject: [PATCH v4] benchtests: Fix validate_benchout.py exceptions Date: Mon, 13 Sep 2021 13:44:12 +0000 Message-Id: <20210913134412.751102-1-naohirot@fujitsu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <2f0293c7-d0b2-0a3b-1ff9-48127837d3c5@gotplt.org> References: <2f0293c7-d0b2-0a3b-1ff9-48127837d3c5@gotplt.org> X-TM-AS-GCONF: 00 X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Naohiro Tamura via Libc-alpha Reply-To: Naohiro Tamura Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org Sender: "Libc-alpha" This patch fixed validate_benchout.py two exceptions, AttributeError if benchout_strings.schema.json is specified and json.decoder.JSONDecodeError if benchout is not JSON. AttributeError unconditionally occurs with a correct JSON benchout file such as below because the code "bench['functions'][func][k].keys()" is either "bench-variant", "ifunc", or "results" that doesn't have keys()." $ ~/glibc/benchtests/scripts/validate_benchout.py bench-memcpy.out \ ~/glibc/benchtests/scripts/benchout_strings.schema.json Traceback (most recent call last): File "/home/naohirot/work/github/glibc/benchtests/scripts/validate_benchout.py", line 86, in sys.exit(main(sys.argv[1:])) File "/home/naohirot/work/github/glibc/benchtests/scripts/validate_benchout.py", line 69, in main bench.parse_bench(args[0], args[1]) File "/home/naohirot/work/github/glibc/benchtests/scripts/import_bench.py", line 139, in parse_bench do_for_all_timings(bench, lambda b, f, v: File "/home/naohirot/work/github/glibc/benchtests/scripts/import_bench.py", line 107, in do_for_all_timings if 'timings' not in bench['functions'][func][k].keys(): AttributeError: 'str' object has no attribute 'keys' $ cat bench-memcpy.out 1 { 2 "timing_type": "hp_timing", 3 "functions": { 4 "memcpy": { 5 "bench-variant": "default", 6 "ifuncs": ["generic_memcpy", "__memcpy_thunderx", "__memcpy_thunderx2", "__memcpy_falkor", "__memcpy_simd", "__memcpy_a64fx", "__memcpy_generic"], 7 "results": [ 8 { 9 "length": 1, 10 "align1": 0, 11 "align2": 0, 12 "dst > src": 0, 13 "timings": [10.9326, 11.0449, 11.5515, 13.5693, 11.5198, 6.77368, 11.5259] 14 }, ... --- benchtests/scripts/import_bench.py | 17 +++++++++++------ benchtests/scripts/validate_benchout.py | 6 +++++- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/benchtests/scripts/import_bench.py b/benchtests/scripts/import_bench.py index a799b4e1b7dc..f5e67570d4c5 100644 --- a/benchtests/scripts/import_bench.py +++ b/benchtests/scripts/import_bench.py @@ -101,13 +101,18 @@ def do_for_all_timings(bench, callback): Args: bench: The benchmark object callback: The callback function + Raises: + validator.exceptions.ValidationError: if 'timings' key not found """ for func in bench['functions'].keys(): for k in bench['functions'][func].keys(): - if 'timings' not in bench['functions'][func][k].keys(): - continue - - callback(bench, func, k) + if k == 'results': + for r in range(len(bench['functions'][func][k])): + if 'timings' not in bench['functions'][func][k][r].keys(): + raise validator.exceptions.ValidationError( + "'timings' key not found") + else: + callback(bench, func, k, r) def compress_timings(points): @@ -136,6 +141,6 @@ def parse_bench(filename, schema_filename): with open(filename, 'r') as benchfile: bench = json.load(benchfile) validator.validate(bench, schema) - do_for_all_timings(bench, lambda b, f, v: - b['functions'][f][v]['timings'].sort()) + do_for_all_timings(bench, lambda b, f, v, r: + b['functions'][f][v][r]['timings'].sort()) return bench diff --git a/benchtests/scripts/validate_benchout.py b/benchtests/scripts/validate_benchout.py index 47df33ed0252..00d5fa0ee5eb 100755 --- a/benchtests/scripts/validate_benchout.py +++ b/benchtests/scripts/validate_benchout.py @@ -73,11 +73,15 @@ def main(args): except bench.validator.ValidationError as e: return print_and_exit("Invalid benchmark output: %s" % e.message, - os.EX_DATAERR) + os.EX_DATAERR) except bench.validator.SchemaError as e: return print_and_exit("Invalid schema: %s" % e.message, os.EX_DATAERR) + except json.decoder.JSONDecodeError as e: + return print_and_exit("Benchmark output in %s is not JSON." % args[0], + os.EX_DATAERR) + print("Benchmark output in %s is valid." % args[0]) return os.EX_OK -- 2.17.1