From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by dcvr.yhbt.net (Postfix) with ESMTP id 4A7AB1F545 for ; Sat, 1 Jul 2023 01:03:37 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=pobox.com header.i=@pobox.com header.a=rsa-sha256 header.s=sasl header.b=WGVc8S4g; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229585AbjGABD1 (ORCPT ); Fri, 30 Jun 2023 21:03:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229447AbjGABDW (ORCPT ); Fri, 30 Jun 2023 21:03:22 -0400 Received: from pb-smtp1.pobox.com (pb-smtp1.pobox.com [64.147.108.70]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 619C72D52 for ; Fri, 30 Jun 2023 18:03:21 -0700 (PDT) Received: from pb-smtp1.pobox.com (unknown [127.0.0.1]) by pb-smtp1.pobox.com (Postfix) with ESMTP id 7C91D1A3C3C; Fri, 30 Jun 2023 21:03:20 -0400 (EDT) (envelope-from junio@pobox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=DaEoRTNGaickOSohT6EWmCuxO3kzs7w3xJvQj+ 3d3Ew=; b=WGVc8S4gkUZzBZltxnYJ/tUdTLs82je3lWOlLAYgg+dGKIyb3fln/6 uGPQJD3QHdnGnLP3FqkBuTyzLU9raRI2ajA+ixeiAJdEjuz/ZXb2oOm/J3RJglSV 5Oxgk3lkQpEzEX140alIOxdNPuCmWNz9Nx/tJhvlliBHS+6sHzVss= Received: from pb-smtp1.nyi.icgroup.com (unknown [127.0.0.1]) by pb-smtp1.pobox.com (Postfix) with ESMTP id 735EC1A3C3B; Fri, 30 Jun 2023 21:03:20 -0400 (EDT) (envelope-from junio@pobox.com) Received: from pobox.com (unknown [35.233.135.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pb-smtp1.pobox.com (Postfix) with ESMTPSA id D7C881A3C3A; Fri, 30 Jun 2023 21:03:19 -0400 (EDT) (envelope-from junio@pobox.com) From: Junio C Hamano To: Josh Steadmon Cc: git@vger.kernel.org, szeder.dev@gmail.com, phillip.wood123@gmail.com, chooglen@google.com, avarab@gmail.com, sandals@crustytoothpaste.net, calvinwan@google.com Subject: Re: [PATCH v4] unit tests: Add a project plan document References: <20230517-unit-tests-v2-v2-0-8c1b50f75811@google.com> <0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com> Date: Fri, 30 Jun 2023 18:03:18 -0700 In-Reply-To: <0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com> (Josh Steadmon's message of "Fri, 30 Jun 2023 15:51:19 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Pobox-Relay-ID: 11B52598-17AB-11EE-A238-C65BE52EC81B-77302942!pb-smtp1.pobox.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Josh Steadmon writes: > In our current testing environment, we spend a significant amount of > effort crafting end-to-end tests for error conditions that could easily > be captured by unit tests (or we simply forgo some hard-to-setup and > rare error conditions). Describe what we hope to accomplish by > implementing unit tests, and explain some open questions and milestones. > Discuss desired features for test frameworks/harnesses, and provide a > preliminary comparison of several different frameworks. > > Coauthored-by: Calvin Wan > Signed-off-by: Calvin Wan > Signed-off-by: Josh Steadmon > --- > TODOs remaining: > - List rough priorities across comparison dimensions > - Group dimensions into sensible categories > - Discuss pre-existing harnesses for the current test suite > - Discuss harness vs. framework features, particularly for parallelism > - Figure out how to evaluate frameworks on additional OSes such as *BSD > and NonStop > - Add more discussion about desired features (particularly mocking) > - Add dimension for test timing > - Evaluate remaining missing comparison table entries Listing these explicitly here is very much appreciated. Thanks.