[ANNOUNCE] Sharness v1.2.0

[ANNOUNCE] Sharness v1.2.0

[Felipe Contreras]

Hello,

Version 1.2.0 of Sharness [1] -- the test harness library derived from
Git's test lib -- is released.

This release contains a few features from upstream and many
improvements.

 * Moved functions into lib-sharness/functions.sh
 * Add simple example
 * Add support for zsh
 * Clarify the purpose of `aggregate-results.sh`
 * Add `SHARNESS_TEST_OUTDIR`
 * Add `SHARNESS_TEST_NB`
 * Add `-x` option for tracing commands
 * Add vim syntax file
 * Minor bugfixes

Many thanks to all that contributed to this release:

     1  Albert Chu
     1  Chris Dunlap
    39  Christian Couder
     2  Eric Curtin
    72  Felipe Contreras
     1  Marcel Schilling
     1  Rafael Ascensão
     1  Scott Bronson
     1  Tristan Le Guern

The previous release was v1.1.0 in September 2018.

To find more information about the release, including download
locations, check the GitHub Release [3].

To find more information about Sharness, you can check the
Sharness site [4].

Christian Couder has decided to transfer the ownership of the
project [2] to me, Felipe Contreras.

Thank you Christian for keeping the project afloat these years and your
many contributions.

Enjoy.

[1] https://github.com/felipec/sharness
[2] https://github.com/felipec/sharness/commit/00d30002218928d11d27d7c308909baf6da6da8b
[3] https://github.com/felipec/sharness/releases/tag/v1.2.0
[4] https://felipec.github.io/sharness/

-- 
Felipe Contreras

New attempt to export test-lib from Git, maybe Sharness2?

[Jiang Xin]

From: Jiang Xin <zhiyou.jx@alibaba-inc.com>

On Mon, 20 Mar 2023 18:15:17 -0600, Felipe Contreras wrote:
> Hello,
> 
> Version 1.2.0 of Sharness [1] -- the test harness library derived from
> Git's test lib -- is released. 

I like Sharness which make it possible to reuse the test framework of
Git. I try to use sharness in my projects and I'm having fun writing
test cases using familiar shell scripts.

But the test framework used by Sharpness is from Git v1.7.9 and is
quite old. The absence of test helper API and command options such as
"--run=..." is quite inconvenient for me. Furthermore, the lack of
lint tools results in many errors in my test cases (e.g., missing
"&&" between statements).

It's not easy to upgrade sharness to the latest test framework of Git.
So I decide to start a new project. The new project is named test-lib,
see:

  * https://github.com/jiangxin/test-lib

Some of my projects have upgraded the test framework from sharkness to
test-lib:

 * git-po-helper: https://github.com/git-l10n/git-po-helper/tree/main/test
 * git-repo-go: https://github.com/alibaba/git-repo-go/tree/master/test

I wonder if we can start Sharness2 based on this solution. See the
README of the test-lib project for details:


diff --git a/README.md b/README.md
new file mode 100644
index 0000000000..4cc529ab73
--- /dev/null
+++ b/README.md
@@ -0,0 +1,159 @@
+Test Lib From Git Core
+======================
+
+Test-lib is a test framework developed by Junio and is specifically
+designed for the Git project. It allows us to write a test suite
+using shell script, which contains a collection of test cases. The
+output of each test suite is presented in TAP ([Test Anything
+Protocol]) format. We can use test-lib or any other TAP harness
+programs (e.g., prove) to run and analyze the output of the test
+suites.
+
+In order to reuse the git test framework into other projects,
+the [sharness project] made a successful attempt. However, it is
+based on an outdated version of git (v1.7.9), which results in
+bugs and missing new features. For example:
+
+ * Commit d88785e424 (test-lib: set `BASH_XTRACEFD` automatically,
+   2016-05-11) and commit a5bf824f3b4d (t: prevent '-x' tracing
+   from interfering with test helpers' stderr, 2018-02-25) of the Git
+   project addressed bugs when we run test suites with "-x" option.
+
+ * Commit 0445e6f0a1 (test-lib: '--run' to run only specific tests,
+   2014-04-30) provided better control of the set of tests to run.
+
+ * Commit 92b269f5c5 (test-lib: turn on `GIT_TEST_CHAIN_LINT`
+   by default, 2015-04-22) turned on chain-lint by default to
+   prevent the accidental omission of "&&" between statements in
+   test cases. Additionally, Git offers more linter tools such as
+   "chainlint.pl" to help write correct test cases.
+
+ * The latest version of the Git project includes numerous test
+   helpers that are not present in sharness. These helpers
+   provide a more comprehensive and efficient testing developing
+   tools. E.g.: `test_bool_env`, `test_cmp_bin`, `test_commit`,
+   `test_config`, `test_env`, `test_file_not_empty`, `test_file_size`,
+   `test_line_count`, `test_oid`, `test_path_exists`,
+   `test_path_is_executable`, `test_path_is_missing`, `test_tick`,
+   `write_script`, etc.
+
+In order to reuse the latest test framework of the Git project and
+easy to maintain, use the following strategies:
+
+1. Use [git-filter-repo] to export test-lib related files and their
+   commit histories from the Git project. The generated tailored
+   commits are saved in the branch named "git-test-lib".
+
+2. The test-lib test framework relies on a helper program named
+   "test-tool", which is written in C. To use test-lib without the
+   need for C compilation, re-implemented part of the "test-tool"
+   subcommands in Python.
+
+3. Apply a series of patches to test-lib so that it can be easily
+   reused in other projects. Some patches are borrowed from Sharness.
+
+4. This project is a by-product of the Git project. For each Git
+   release, we will re-run step 1 to export the latest test
+   framework from Git projects which may include new files we
+   missed in older versions. The generated commit history will
+   overwrite the "git-test-lib" branch, and the master branch will
+   rebase on it.
+
+
+Install test-lib
+----------------
+
+To create test suites in shell scripts powered by test-lib, you can
+follow these steps:
+
+1. Set up a directory (such as "test") to save test suites and
+   files of test-lib.
+
+        $ mkdir test
+
+2. Clone or copy the test-lib repository inside the test directory.
+
+        $ cd test
+        $ git clone https://github.com/jiangixn/test-lib lib
+
+4. Copy files from the example test directory.
+
+        $ cp lib/test-example/.gitignore .
+        $ cp lib/test-example/.gitattributes .
+        $ cp lib/test-example/Makefile .
+
+4. Start writing a test suite powered by test-lib, make sure it
+   sources the "test-lib.sh" file from test-lib. Refer to the example
+   test suite (e.g. "test/t0001-test-tool-chmtime.sh") to write
+   your own test suite:
+
+        test_description='My first test suite'
+
+        . lib/test-lib.sh
+
+
+Usage of test-lib
+-----------------
+
+As for how to use test-lib to write, run and manage your test suites,
+please see the documentation [README.git] for reference.
+
+
+Filtering test-lib from the Git project
+---------------------------------------
+
+"test-lib" is part of the Git project and is stored in the "t/"
+subdirectory. We use "git-filter-repo" to export test-lib related
+files to the root directory of this project. We repeat this
+periodically to save historical commits of test-lib in the
+"git-test-lib" branch of this project. The master branch of this
+project will merge with or rebase onto the "git-test-lib" branch.
+As how to filter and export test-lib from the Git project, see the
+steps below.
+
+1. Make a fresh clone of the Git project before filtering.
+
+        $ git clone --single-branch --no-tags \
+          https://github.com/git/git.git \
+          git-test-lib
+
+        $ cd git-test-lib
+
+2. Use git-filter-repo to filter and export test-lib.
+
+        $ git filter-repo \
+          --preserve-commit-encoding \
+          --prune-degenerate always \
+          --path COPYING \
+          --path shared.mak \
+          --path t/.gitattributes \
+          --path t/.gitignore \
+          --path t/Makefile \
+          --path t/README \
+          --path t/aggregate-results.sh \
+          --path t/chainlint/ \
+          --path t/chainlint.pl \
+          --path t/check-non-portable-shell.pl \
+          --path t/oid-info/ \
+          --path t/perf/.gitignore \
+          --path t/perf/Makefile \
+          --path t/perf/README \
+          --path t/perf/aggregate.perl \
+          --path t/perf/config \
+          --path t/perf/min_time.perl \
+          --path t/perf/perf-lib.sh \
+          --path t/perf/p0000-perf-lib-sanity.sh \
+          --path t/perf/run \
+          --path t/test-lib-functions.sh \
+          --path t/test-lib-github-workflow-markup.sh \
+          --path t/test-lib-junit.sh \
+          --path t/test-lib.sh \
+          --path t/lib-subtest.sh \
+          --path t/t0000-basic.sh \
+          --path-rename t/:
+
+
+[Test Anything Protocol]: http://testanything.org/
+[sharness project]: https://github.com/felipec/sharness
+[git-filter-repo]: https://github.com/newren/git-filter-repo
+[README.git]: ./README.git
-- 
2.42.0.rc2.21.gd1f87c2148

Re: New attempt to export test-lib from Git, maybe Sharness2?

[Junio C Hamano]

Jiang Xin <worldhello.net@gmail.com> writes:

> It's not easy to upgrade sharness to the latest test framework of Git.

So?

> So I decide to start a new project. The new project is named test-lib,
> see:
>
>   * https://github.com/jiangxin/test-lib
>
> Some of my projects have upgraded the test framework from sharkness to
> test-lib:
>
>  * git-po-helper: https://github.com/git-l10n/git-po-helper/tree/main/test
>  * git-repo-go: https://github.com/alibaba/git-repo-go/tree/master/test
>
> I wonder if we can start Sharness2 based on this solution. See the
> README of the test-lib project for details:

Is it a viable option to stick to the name "test-lib" (or possibly,
"git-test-lib" to make it more prominent to say where it came from)?

If you do not plan to coordinate with those who work on (the remnant
of) the original sharness based on an ancient version of our test
framework, and do not plan to actively transition its users to your
version, it is less confusing if you named yours differently, as it
avoids hinting that your version is a successor of theirs.

I am not sure if reusing the history of our project verbatim using
filter-repo is really a good way to help those who are interested in
the test framework, by the way.  We make changes for our own purpose
and as a part of such an effort we may touch the test framework to
make it easier to test the changes we made, e.g.,

https://github.com/jiangxin/test-lib/commit/0d5db66ef2b9d8ed5bcee9a0167672dc88b1b026

and unedited filter-repo result will describe such a commit
primarily to explain why the changes in the commit was made on Git
side.  Most of the changes described in the resulting commit message
are discarded by filter-repo and the resulting history becomes hard
to follow.

Re: New attempt to export test-lib from Git, maybe Sharness2?

[Jiang Xin]

On Sun, Dec 10, 2023 at 2:59 AM Junio C Hamano <gitster@pobox.com> wrote:
> Is it a viable option to stick to the name "test-lib" (or possibly,
> "git-test-lib" to make it more prominent to say where it came from)?

Will rename the project to "git-test-lib".

> If you do not plan to coordinate with those who work on (the remnant
> of) the original sharness based on an ancient version of our test
> framework, and do not plan to actively transition its users to your
> version, it is less confusing if you named yours differently, as it
> avoids hinting that your version is a successor of theirs.
>
> I am not sure if reusing the history of our project verbatim using
> filter-repo is really a good way to help those who are interested in
> the test framework, by the way.

If all historical commits were squashed, it would be difficult to
track future changes of test-lib in the Git project. Futhermore, who
will be the author of the squashed commit?

> and unedited filter-repo result will describe such a commit
> primarily to explain why the changes in the commit was made on Git
> side.  Most of the changes described in the resulting commit message
> are discarded by filter-repo and the resulting history becomes hard
> to follow.

One solution is to add notes in the README file and users can refer to
the commit history of the git project.