git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Matheus Tavares <matheus.bernardino@usp.br>
Cc: git@vger.kernel.org, christian.couder@gmail.com,
	git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de
Subject: AIX failures on parallel checkout (new in v2.32.0-rc*)
Date: Wed, 26 May 2021 20:36:43 +0200	[thread overview]
Message-ID: <87eedtmj7w.fsf@evledraar.gmail.com> (raw)
In-Reply-To: <667777053ab853ef2f94b7ba4ab39f7241d2b49c.1620145501.git.matheus.bernardino@usp.br>


On Tue, May 04 2021, Matheus Tavares wrote:

> Add tests to populate the working tree during clone and checkout using
> sequential and parallel mode, to confirm that they produce identical
> results. Also test basic checkout mechanics, such as checking for
> symlinks in the leading directories and the abidance to --force.
>
> Note: some helper functions are added to a common lib file which is only
> included by t2080 for now. But they will also be used by other
> parallel-checkout tests in the following patches.
>
> Co-authored-by: Jeff Hostetler <jeffhost@microsoft.com>
> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
> ---
>  t/lib-parallel-checkout.sh          |  42 +++++
>  t/t2080-parallel-checkout-basics.sh | 229 ++++++++++++++++++++++++++++
>  2 files changed, 271 insertions(+)
>  create mode 100644 t/lib-parallel-checkout.sh
>  create mode 100755 t/t2080-parallel-checkout-basics.sh
>
> diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh
> new file mode 100644
> index 0000000000..f60b22ef34
> --- /dev/null
> +++ b/t/lib-parallel-checkout.sh
> @@ -0,0 +1,42 @@
> +# Helpers for tests invoking parallel-checkout
> +
> +set_checkout_config () {
> +	if test $# -ne 2
> +	then
> +		BUG "usage: set_checkout_config <workers> <threshold>"
> +	fi &&
> +
> +	test_config_global checkout.workers $1 &&
> +	test_config_global checkout.thresholdForParallelism $2
> +}
> +
> +# Run "${@:2}" and check that $1 checkout workers were used
> +test_checkout_workers () {
> +	if test $# -lt 2
> +	then
> +		BUG "too few arguments to test_checkout_workers"
> +	fi &&
> +
> +	local expected_workers=$1 &&
> +	shift &&
> +
> +	local trace_file=trace-test-checkout-workers &&
> +	rm -f "$trace_file" &&
> +	GIT_TRACE2="$(pwd)/$trace_file" "$@" &&
> +
> +	local workers=$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l) &&
> +	test $workers -eq $expected_workers &&
> +	rm "$trace_file"
> +}
> +
> +# Verify that both the working tree and the index were created correctly
> +verify_checkout () {
> +	if test $# -ne 1
> +	then
> +		BUG "usage: verify_checkout <repository path>"
> +	fi &&
> +
> +	git -C "$1" diff-index --ignore-submodules=none --exit-code HEAD -- &&
> +	git -C "$1" status --porcelain >"$1".status &&
> +	test_must_be_empty "$1".status
> +}
> diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh
> new file mode 100755
> index 0000000000..7087818550
> --- /dev/null
> +++ b/t/t2080-parallel-checkout-basics.sh
> @@ -0,0 +1,229 @@
> +#!/bin/sh
> +
> +test_description='parallel-checkout basics
> +
> +Ensure that parallel-checkout basically works on clone and checkout, spawning
> +the required number of workers and correctly populating both the index and the
> +working tree.
> +'
> +
> +TEST_NO_CREATE_REPO=1
> +. ./test-lib.sh
> +. "$TEST_DIRECTORY/lib-parallel-checkout.sh"
> +
> +# Test parallel-checkout with a branch switch containing a variety of file
> +# creations, deletions, and modifications, involving different entry types.
> +# The branches B1 and B2 have the following paths:
> +#
> +#      B1                 B2
> +#  a/a (file)         a   (file)
> +#  b   (file)         b/b (file)
> +#
> +#  c/c (file)         c   (symlink)
> +#  d   (symlink)      d/d (file)
> +#
> +#  e/e (file)         e   (submodule)
> +#  f   (submodule)    f/f (file)
> +#
> +#  g   (submodule)    g   (symlink)
> +#  h   (symlink)      h   (submodule)
> +#
> +# Additionally, the following paths are present on both branches, but with
> +# different contents:
> +#
> +#  i   (file)         i   (file)
> +#  j   (symlink)      j   (symlink)
> +#  k   (submodule)    k   (submodule)
> +#
> +# And the following paths are only present in one of the branches:
> +#
> +#  l/l (file)         -
> +#  -                  m/m (file)
> +#
> +test_expect_success 'setup repo for checkout with various types of changes' '
> +	git init sub &&
> +	(
> +		cd sub &&
> +		git checkout -b B2 &&
> +		echo B2 >file &&
> +		git add file &&
> +		git commit -m file &&
> +
> +		git checkout -b B1 &&
> +		echo B1 >file &&
> +		git add file &&
> +		git commit -m file
> +	) &&
> +
> +	git init various &&
> +	(
> +		cd various &&
> +
> +		git checkout -b B1 &&
> +		mkdir a c e &&
> +		echo a/a >a/a &&
> +		echo b >b &&
> +		echo c/c >c/c &&
> +		test_ln_s_add c d &&
> +		echo e/e >e/e &&
> +		git submodule add ../sub f &&
> +		git submodule add ../sub g &&
> +		test_ln_s_add c h &&
> +
> +		echo "B1 i" >i &&
> +		test_ln_s_add c j &&
> +		git submodule add -b B1 ../sub k &&
> +		mkdir l &&
> +		echo l/l >l/l &&
> +
> +		git add . &&
> +		git commit -m B1 &&
> +
> +		git checkout -b B2 &&
> +		git rm -rf :^.gitmodules :^k &&
> +		mkdir b d f &&
> +		echo a >a &&
> +		echo b/b >b/b &&
> +		test_ln_s_add b c &&
> +		echo d/d >d/d &&
> +		git submodule add ../sub e &&
> +		echo f/f >f/f &&
> +		test_ln_s_add b g &&
> +		git submodule add ../sub h &&
> +
> +		echo "B2 i" >i &&
> +		test_ln_s_add b j &&
> +		git -C k checkout B2 &&
> +		mkdir m &&
> +		echo m/m >m/m &&
> +
> +		git add . &&
> +		git commit -m B2 &&
> +
> +		git checkout --recurse-submodules B1
> +	)
> +'
> +
> +for mode in sequential parallel sequential-fallback
> +do
> +	case $mode in
> +	sequential)          workers=1 threshold=0 expected_workers=0 ;;
> +	parallel)            workers=2 threshold=0 expected_workers=2 ;;
> +	sequential-fallback) workers=2 threshold=100 expected_workers=0 ;;
> +	esac
> +
> +	test_expect_success "$mode checkout" '
> +		repo=various_$mode &&
> +		cp -R various $repo &&
> +
> +		# The just copied files have more recent timestamps than their
> +		# associated index entries. So refresh the cached timestamps
> +		# to avoid an "entry not up-to-date" error from `git checkout`.
> +		# We only have to do this for the submodules as `git checkout`
> +		# will already refresh the superproject index before performing
> +		# the up-to-date check.
> +		#
> +		git -C $repo submodule foreach "git update-index --refresh" &&
> +
> +		set_checkout_config $workers $threshold &&
> +		test_checkout_workers $expected_workers \
> +			git -C $repo checkout --recurse-submodules B2 &&
> +		verify_checkout $repo
> +	'
> +done
> +
> +for mode in parallel sequential-fallback
> +do
> +	case $mode in
> +	parallel)            workers=2 threshold=0 expected_workers=2 ;;
> +	sequential-fallback) workers=2 threshold=100 expected_workers=0 ;;
> +	esac
> +
> +	test_expect_success "$mode checkout on clone" '
> +		repo=various_${mode}_clone &&
> +		set_checkout_config $workers $threshold &&
> +		test_checkout_workers $expected_workers \
> +			git clone --recurse-submodules --branch B2 various $repo &&
> +		verify_checkout $repo
> +	'
> +done
> +
> +# Just to be paranoid, actually compare the working trees' contents directly.
> +test_expect_success 'compare the working trees' '
> +	rm -rf various_*/.git &&
> +	rm -rf various_*/*/.git &&
> +
> +	# We use `git diff` instead of `diff -r` because the latter would
> +	# follow symlinks, and not all `diff` implementations support the
> +	# `--no-dereference` option.
> +	#
> +	git diff --no-index various_sequential various_parallel &&
> +	git diff --no-index various_sequential various_parallel_clone &&
> +	git diff --no-index various_sequential various_sequential-fallback &&
> +	git diff --no-index various_sequential various_sequential-fallback_clone
> +'
> +
> +# Currently, each submodule is checked out in a separated child process, but
> +# these subprocesses must also be able to use parallel checkout workers to
> +# write the submodules' entries.
> +test_expect_success 'submodules can use parallel checkout' '
> +	set_checkout_config 2 0 &&
> +	git init super &&
> +	(
> +		cd super &&
> +		git init sub &&
> +		test_commit -C sub A &&
> +		test_commit -C sub B &&
> +		git submodule add ./sub &&
> +		git commit -m sub &&
> +		rm sub/* &&
> +		test_checkout_workers 2 git checkout --recurse-submodules .
> +	)
> +'
> +
> +test_expect_success 'parallel checkout respects --[no]-force' '
> +	set_checkout_config 2 0 &&
> +	git init dirty &&
> +	(
> +		cd dirty &&
> +		mkdir D &&
> +		test_commit D/F &&
> +		test_commit F &&
> +
> +		rm -rf D &&
> +		echo changed >D &&
> +		echo changed >F.t &&
> +
> +		# We expect 0 workers because there is nothing to be done
> +		test_checkout_workers 0 git checkout HEAD &&
> +		test_path_is_file D &&
> +		grep changed D &&
> +		grep changed F.t &&
> +
> +		test_checkout_workers 2 git checkout --force HEAD &&
> +		test_path_is_dir D &&
> +		grep D/F D/F.t &&
> +		grep F F.t
> +	)
> +'
> +
> +test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading dirs' '
> +	set_checkout_config 2 0 &&
> +	git init symlinks &&
> +	(
> +		cd symlinks &&
> +		mkdir D untracked &&
> +		# Commit 2 files to have enough work for 2 parallel workers
> +		test_commit D/A &&
> +		test_commit D/B &&
> +		rm -rf D &&
> +		ln -s untracked D &&
> +
> +		test_checkout_workers 2 git checkout --force HEAD &&
> +		! test -h D &&
> +		grep D/A D/A.t &&
> +		grep D/B D/B.t
> +	)
> +'
> +
> +test_done

I haven't dug into why but these tests fail on AIX (on the fairly easy
to get access to GCC farm boxes):

E.g. for ./t2080-parallel-checkout-basics.sh:
    
    [...]
    Branch 'B2' set up to track remote branch 'B2' from 'origin'.
    Switched to a new branch 'B2'
    + mkdir m
    + echo m/m
    + 1> m/m
    + git add .
    + git commit -m B2
    [B2 176cae8] B2
     Author: A U Thor <author@example.com>
     19 files changed, 17 insertions(+), 17 deletions(-)
     create mode 100644 a
     delete mode 100644 a/a
     delete mode 100644 b
     create mode 100644 b/b
     create mode 120000 c
     delete mode 100644 c/c
     delete mode 120000 d
     create mode 100644 d/d
     rename f => e (100%)
     delete mode 100644 e/e
     create mode 100644 f/f
     mode change 160000 => 120000 g
     mode change 120000 => 160000 h
     delete mode 100644 l/l
     create mode 100644 m/m
    + git checkout --recurse-submodules B1
    Switched to branch 'B1'
    ok 1 - setup repo for checkout with various types of changes
    
    expecting success of 2080.2 'sequential checkout':
                    repo=various_$mode &&
                    cp -R various $repo &&
    
                    # The just copied files have more recent timestamps than their
                    # associated index entries. So refresh the cached timestamps
                    # to avoid an "entry not up-to-date" error from `git checkout`.
                    # We only have to do this for the submodules as `git checkout`
                    # will already refresh the superproject index before performing
                    # the up-to-date check.
                    #
                    git -C $repo submodule foreach "git update-index --refresh" &&
    
                    set_checkout_config $workers $threshold &&
                    test_checkout_workers $expected_workers \
                            git -C $repo checkout --recurse-submodules B2 &&
                    verify_checkout $repo
    
    + repo=various_sequential
    + cp -R various various_sequential
    + git -C various_sequential submodule foreach git update-index --refresh
    Entering 'f'
    Entering 'g'
    Entering 'k'
    + set_checkout_config 1 0
    + test_checkout_workers 0 git -C various_sequential checkout --recurse-submodules B2
    error: Your local changes to the following files would be overwritten by checkout:
            d
            h
            j
    Please commit your changes or stash them before you switch branches.
    Aborting
    error: last command exited with $?=1
    not ok 2 - sequential checkout
    #
    #                       repo=various_$mode &&
    #                       cp -R various $repo &&
    #
    #                       # The just copied files have more recent timestamps than their
    #                       # associated index entries. So refresh the cached timestamps
    #                       # to avoid an "entry not up-to-date" error from `git checkout`.
    #                       # We only have to do this for the submodules as `git checkout`
    #                       # will already refresh the superproject index before performing
    #                       # the up-to-date check.
    #                       #
    #                       git -C $repo submodule foreach "git update-index --refresh" &&
    #
    #                       set_checkout_config $workers $threshold &&
    #                       test_checkout_workers $expected_workers \
    #                               git -C $repo checkout --recurse-submodules B2 &&
    #                       verify_checkout $repo
    #
    
The test suite has a lot of breakages on AIX that I haven't looked at,
but this particular one is new, so I thought I'd send a quick poke about
it...

  reply	other threads:[~2021-05-26 18:40 UTC|newest]

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-22 15:17 [PATCH 0/7] Parallel Checkout (part 3) Matheus Tavares
2021-04-22 15:17 ` [PATCH 1/7] make_transient_cache_entry(): optionally alloc from mem_pool Matheus Tavares
2021-04-22 15:17 ` [PATCH 2/7] builtin/checkout.c: complete parallel checkout support Matheus Tavares
2021-04-23 16:19   ` Derrick Stolee
2021-04-26 21:54     ` Matheus Tavares Bernardino
2021-04-22 15:17 ` [PATCH 3/7] checkout-index: add " Matheus Tavares
2021-04-23 18:32   ` Derrick Stolee
2021-04-26 22:30     ` Matheus Tavares Bernardino
2021-04-22 15:17 ` [PATCH 4/7] parallel-checkout: add tests for basic operations Matheus Tavares
2021-04-23 19:18   ` Derrick Stolee
2021-04-27  2:30     ` Matheus Tavares Bernardino
2021-04-22 15:17 ` [PATCH 5/7] parallel-checkout: add tests related to path collisions Matheus Tavares
2021-04-22 15:17 ` [PATCH 6/7] parallel-checkout: add tests related to .gitattributes Matheus Tavares
2021-04-23 19:48   ` Derrick Stolee
2021-04-22 15:17 ` [PATCH 7/7] ci: run test round with parallel-checkout enabled Matheus Tavares
2021-04-23 19:56   ` Derrick Stolee
2021-04-30 21:40 ` [PATCH v2 0/8] Parallel Checkout (part 3) Matheus Tavares
2021-04-30 21:40   ` [PATCH v2 1/8] make_transient_cache_entry(): optionally alloc from mem_pool Matheus Tavares
2021-05-01 17:06     ` Christian Couder
2021-05-03 14:11       ` Matheus Tavares Bernardino
2021-04-30 21:40   ` [PATCH v2 2/8] builtin/checkout.c: complete parallel checkout support Matheus Tavares
2021-05-01 17:08     ` Christian Couder
2021-05-03 14:21       ` Matheus Tavares Bernardino
2021-04-30 21:40   ` [PATCH v2 3/8] checkout-index: add " Matheus Tavares
2021-05-01 17:08     ` Christian Couder
2021-05-03 14:22       ` Matheus Tavares Bernardino
2021-04-30 21:40   ` [PATCH v2 4/8] parallel-checkout: add tests for basic operations Matheus Tavares
2021-04-30 21:40   ` [PATCH v2 5/8] parallel-checkout: add tests related to path collisions Matheus Tavares
2021-05-02  7:59     ` Torsten Bögershausen
2021-05-03 14:58       ` Matheus Tavares Bernardino
2021-04-30 21:40   ` [PATCH v2 6/8] t0028: extract encoding helpers to lib-encoding.sh Matheus Tavares
2021-04-30 21:40   ` [PATCH v2 7/8] parallel-checkout: add tests related to .gitattributes Matheus Tavares
2021-04-30 21:40   ` [PATCH v2 8/8] ci: run test round with parallel-checkout enabled Matheus Tavares
2021-05-02 10:12   ` [PATCH v2 0/8] Parallel Checkout (part 3) Torsten Bögershausen
2021-05-03 15:01     ` Matheus Tavares Bernardino
2021-05-04 16:27   ` [PATCH v3 " Matheus Tavares
2021-05-04 16:27     ` [PATCH v3 1/8] make_transient_cache_entry(): optionally alloc from mem_pool Matheus Tavares
2021-05-04 16:27     ` [PATCH v3 2/8] builtin/checkout.c: complete parallel checkout support Matheus Tavares
2021-05-05 13:55       ` Derrick Stolee
2021-05-04 16:27     ` [PATCH v3 3/8] checkout-index: add " Matheus Tavares
2021-05-04 16:27     ` [PATCH v3 4/8] parallel-checkout: add tests for basic operations Matheus Tavares
2021-05-26 18:36       ` Ævar Arnfjörð Bjarmason [this message]
2021-05-26 22:01         ` AIX failures on parallel checkout (new in v2.32.0-rc*) Matheus Tavares Bernardino
2021-05-26 23:00           ` Junio C Hamano
2021-05-04 16:27     ` [PATCH v3 5/8] parallel-checkout: add tests related to path collisions Matheus Tavares
2021-05-04 16:27     ` [PATCH v3 6/8] t0028: extract encoding helpers to lib-encoding.sh Matheus Tavares
2021-05-04 16:27     ` [PATCH v3 7/8] parallel-checkout: add tests related to .gitattributes Matheus Tavares
2021-05-04 16:27     ` [PATCH v3 8/8] ci: run test round with parallel-checkout enabled Matheus Tavares
2021-05-05 13:57     ` [PATCH v3 0/8] Parallel Checkout (part 3) Derrick Stolee
2021-05-06  0:40       ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87eedtmj7w.fsf@evledraar.gmail.com \
    --to=avarab@gmail.com \
    --cc=christian.couder@gmail.com \
    --cc=git@jeffhostetler.com \
    --cc=git@vger.kernel.org \
    --cc=matheus.bernardino@usp.br \
    --cc=stolee@gmail.com \
    --cc=tboegi@web.de \
    --subject='Re: AIX failures on parallel checkout (new in v2.32.0-rc*)' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org list mirror (unofficial, one of many)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 git git/ https://public-inbox.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.version-control.git
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.version-control.git
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git