From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-3.1 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org [IPv6:2604:1380:45e3:2400::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 4D2061F451 for ; Tue, 16 Jan 2024 23:18:47 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=pobox.com header.i=@pobox.com header.a=rsa-sha256 header.s=sasl header.b=pxA67Qyw; dkim-atps=neutral Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 00ED2293424 for ; Tue, 16 Jan 2024 23:18:46 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8AC841BC33; Tue, 16 Jan 2024 23:18:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=pobox.com header.i=@pobox.com header.b="pxA67Qyw" Received: from pb-smtp20.pobox.com (pb-smtp20.pobox.com [173.228.157.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0985B67C5B for ; Tue, 16 Jan 2024 23:18:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.228.157.52 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705447117; cv=none; b=ZahQXNPQufVQ4lrkuhZe1c2Tl5CieXQ0e+hoV+Ioq/v1iq78VUfEAWDYlg9cja+e7LXHJoiZMZunFOstCAGe4reZG5q3GYH4pXjsKbQ4zo9s8F/mjtMaSiebVrud/SnJKTuHw6O4EOx0ST+G+wzmgd8uYT7OHFRzf+h3fKM3wa0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705447117; c=relaxed/simple; bh=AajQ8FyMIvvA+hwCNmAutGPp3rOPCMbQmolDFpS+okE=; h=Received:DKIM-Signature:Received:Received:From:To:Cc:Subject: In-Reply-To:References:Date:Message-ID:User-Agent:MIME-Version: Content-Type:X-Pobox-Relay-ID; b=QTvhm+J8vXT4TGXwkJOtsk4WE8SEEALVcw/9tPn9+ylIw6srG0o9HLEgQbuqjyTSY5tG639wDMmTvjVdoesQwYUlAu9rYrnjQZUrGW7+sfPs/Aaopb/kz5Lks1A9GPJrcEQo5B6gx26eMtUMAWOBEUdGiQwlI7xjmnX8kKxnI34= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=pobox.com; spf=pass smtp.mailfrom=pobox.com; dkim=pass (1024-bit key) header.d=pobox.com header.i=@pobox.com header.b=pxA67Qyw; arc=none smtp.client-ip=173.228.157.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=pobox.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pobox.com Received: from pb-smtp20.pobox.com (unknown [127.0.0.1]) by pb-smtp20.pobox.com (Postfix) with ESMTP id 7F6652801B; Tue, 16 Jan 2024 18:18:34 -0500 (EST) (envelope-from junio@pobox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=pobox.com; h=from:to:cc :subject:in-reply-to:references:date:message-id:mime-version :content-type; s=sasl; bh=AajQ8FyMIvvA+hwCNmAutGPp3rOPCMbQmolDFp S+okE=; b=pxA67Qywk2ekLlOdf2bUT0RJJ4TQowh8FtNMT9FTHoDOqH13rbL/D9 CGmS4Wj9D9jIFXG8gSYa7Fe5OEkRdD4RsRLMEXwkDU9dLTp28PJvjRoK+bd2jraC mlkjYAncImZwGnPWfpSOIrLg/LBrkQBQtYIn4CiUeJ4/kmirMpCjE= Received: from pb-smtp20.sea.icgroup.com (unknown [127.0.0.1]) by pb-smtp20.pobox.com (Postfix) with ESMTP id 777682801A; Tue, 16 Jan 2024 18:18:34 -0500 (EST) (envelope-from junio@pobox.com) Received: from pobox.com (unknown [34.125.200.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pb-smtp20.pobox.com (Postfix) with ESMTPSA id ADF7A28019; Tue, 16 Jan 2024 18:18:30 -0500 (EST) (envelope-from junio@pobox.com) From: Junio C Hamano To: Josh Steadmon Cc: git@vger.kernel.org, johannes.schindelin@gmx.de, peff@peff.net, phillip.wood@dunelm.org.uk Subject: Re: [RFC PATCH 2/4] test-tool run-command testsuite: support unit tests In-Reply-To: <5ecbc976e6216b941e760e096e166ab432ee7784.1705443632.git.steadmon@google.com> (Josh Steadmon's message of "Tue, 16 Jan 2024 14:22:59 -0800") References: <5ecbc976e6216b941e760e096e166ab432ee7784.1705443632.git.steadmon@google.com> Date: Tue, 16 Jan 2024 15:18:29 -0800 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain X-Pobox-Relay-ID: 8FB01004-B4C5-11EE-9553-F515D2CDFF5E-77302942!pb-smtp20.pobox.com Josh Steadmon writes: > Teach the testsuite runner in `test-tool run-command testsuite` how to > run unit tests, by adding two new flags: > > First, "--(no-)run-in-shell" allows the test-tool to exec the unit-test > binaries directly, rather than trying to interpret them as shell > scripts. Makes perfect sense. > Second "--(no-)require-shell-test-pattern" bypasses the check that the > test filenames match the expected t####-*.sh pattern. This one I am not so sure. Do we still have situations where erroring out when fed a non t[0-9][0-9][0-9][0-9]-*.sh script is problematic? IOW, do we need to keep it as conditional? ... goes and looks ... Ah, this variable/option is misnamed and that is what invited the above nonsense question out of me. The logic this option disables does not "require" (and error out if the requirement is not met); it is used in a loop over "ls *" and "filtering" files out that are not the numbered scripts. But that confusion makes me wonder if non-script side of tests would also want some filtering in the longer run, even if the directory we feed to "test-tool run" happens to contain nothing but what we want to run right now. I wonder if we instead want a variable that holds a pattern used to match programs readdir() discovers and skip those that do not match the pattern? Its default value may be something like "t[0-9][0-9][0-9][0-9]-*.sh" but alternatively you can give, say, "*" to pass everything, or something like that. > With these changes, you can now use test-tool to run the unit tests: > $ make > $ cd t/unit-tests/bin > $ ../../helper/test-tool run-command testsuite --no-run-in-shell \ > --no-require-shell-test-pattern This makes me wonder why we want to do the readdir() loop ourselves. Instead of saying --no-require-shell-test-pattern there, wouldn't it be simpler to say "*" right there, and have testsuite() run the test programs named from the command line? But that is orthogonal to the enhancement we have here.