git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
* [PATCH] [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs
@ 2020-07-21 16:12 SZEDER Gábor
  2020-07-21 16:23 ` Eric Sunshine
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: SZEDER Gábor @ 2020-07-21 16:12 UTC (permalink / raw)
  To: git; +Cc: SZEDER Gábor

In our test suite, when 'git p4' invokes a Git command as a
subprocesses, then it should run the 'git' binary we are testing.
Unfortunately, this is not the case in the 'linux-clang' and
'linux-gcc' jobs on Travis CI, where 'git p4' runs the system
'/usr/bin/git' instead.

Travis CI's default Linux image includes 'pyenv', and all Python
invocations that involve PATH lookup go through 'pyenv', e.g. our
'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as
PYTHON_PATH, which in turn will invoke '/usr/bin/python3'.  Alas, the
'pyenv' version included in this image is buggy, and prepends the
directory containing the Python binary to PATH even if that is a
system directory already in PATH near the end.  Consequently, 'git p4'
in those jobs ends up with its PATH starting with '/usr/bin', and then
runs '/usr/bin/git'.

So remove 'pyenv' in Travis CI's Linux jobs to prevet it from
interfering with our 'git p4' tests.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---

Notes:
    This issue triggered test failures in all 'git p4' test scripts in the
    recent GIT_TEST_DEFAULT_HASH=sha256 test runs, because the system Git
    doesn't understand the 'objectformat' extension, e.g.:
    
      https://travis-ci.org/github/git/git/jobs/710159470#L3267
    
    This is not an issue in Travis CI's macOS jobs or on Azure Pipelines,
    because they don't use 'pyenv'.
    
    However, perhaps removing 'pyenv' is not the best solution here.
    
    We set PYTHON_PATH in 'ci/lib.sh', which is sourced at the beginning
    of (almost) all of our CI scripts.  Consequently, in these jobs we
    first run 'ci/install-dependencies.sh', which sources 'ci/lib.sh',
    assigns PYTHON_PATH=/opt/pyenv/shims/python3 (which is never used),
    and removes 'pyenv', and then run 'ci/run-build-and-test.sh', which
    sources 'ci/lib.sh' and assigns PYTHON_PATH=/usr/bin/python3 (which is
    then used in the build process to set 'pit p4's shebang line).  Not
    really nice, is it.
    
    Alternatively, we could avoid the PATH lookup and thus the bogus
    'pyenv' by explicitly using '/usr/bin/python{2,3}'.  The Linux images
    in both Travis CI and Azure Pipelines are standard Ubuntu images, so I
    think we can safely rely on these Python paths.

 ci/install-dependencies.sh | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
index 0229a77f7d..b4bdcbcba2 100755
--- a/ci/install-dependencies.sh
+++ b/ci/install-dependencies.sh
@@ -36,6 +36,14 @@ linux-clang|linux-gcc)
 		tar --extract --gunzip --file "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
 		cp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .
 	popd
+
+	if test true = "$TRAVIS" &&
+	   pyenv_root=$(pyenv root)
+	then
+		# pyenv in Travis CI's current default (xenial) Linux
+		# image messes up PATH for 'git p4'.
+		sudo rm -rf "$pyenv_root"
+	fi
 	;;
 osx-clang|osx-gcc)
 	export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1
-- 
2.28.0.rc1.319.g12ef029c8a


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs
  2020-07-21 16:12 [PATCH] [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs SZEDER Gábor
@ 2020-07-21 16:23 ` Eric Sunshine
  2020-07-21 20:34 ` Junio C Hamano
  2020-07-21 22:20 ` [PATCH] [RFC] travis-ci: remove bogus 'pyenv' " brian m. carlson
  2 siblings, 0 replies; 7+ messages in thread
From: Eric Sunshine @ 2020-07-21 16:23 UTC (permalink / raw)
  To: SZEDER Gábor; +Cc: Git List

On Tue, Jul 21, 2020 at 12:12 PM SZEDER Gábor <szeder.dev@gmail.com> wrote:
> In our test suite, when 'git p4' invokes a Git command as a
> subprocesses, then it should run the 'git' binary we are testing.
> Unfortunately, this is not the case in the 'linux-clang' and
> 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system
> '/usr/bin/git' instead.
>
> Travis CI's default Linux image includes 'pyenv', and all Python
> invocations that involve PATH lookup go through 'pyenv', e.g. our
> 'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as
> PYTHON_PATH, which in turn will invoke '/usr/bin/python3'.  Alas, the
> 'pyenv' version included in this image is buggy, and prepends the
> directory containing the Python binary to PATH even if that is a
> system directory already in PATH near the end.  Consequently, 'git p4'
> in those jobs ends up with its PATH starting with '/usr/bin', and then
> runs '/usr/bin/git'.
>
> So remove 'pyenv' in Travis CI's Linux jobs to prevet it from

s/prevet/prevent/

> interfering with our 'git p4' tests.
>
> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
> @@ -36,6 +36,14 @@ linux-clang|linux-gcc)
> +
> +       if test true = "$TRAVIS" &&
> +          pyenv_root=$(pyenv root)
> +       then
> +               # pyenv in Travis CI's current default (xenial) Linux
> +               # image messes up PATH for 'git p4'.

I wonder if this comment should give more precise detail about exactly
how it "messes up" PATH.

> +               sudo rm -rf "$pyenv_root"
> +       fi

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs
  2020-07-21 16:12 [PATCH] [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs SZEDER Gábor
  2020-07-21 16:23 ` Eric Sunshine
@ 2020-07-21 20:34 ` Junio C Hamano
  2020-07-23 21:38   ` [PATCH v2] ci: use absolute PYTHON_PATH " SZEDER Gábor
  2020-07-21 22:20 ` [PATCH] [RFC] travis-ci: remove bogus 'pyenv' " brian m. carlson
  2 siblings, 1 reply; 7+ messages in thread
From: Junio C Hamano @ 2020-07-21 20:34 UTC (permalink / raw)
  To: SZEDER Gábor; +Cc: git

SZEDER Gábor <szeder.dev@gmail.com> writes:

>     Alternatively, we could avoid the PATH lookup and thus the bogus
>     'pyenv' by explicitly using '/usr/bin/python{2,3}'.  The Linux images
>     in both Travis CI and Azure Pipelines are standard Ubuntu images, so I
>     think we can safely rely on these Python paths.

That sounds quite sensible, actually.  The CI environment being
"special" in the sense that we have certain control over it makes
this rather straight-forward.




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs
  2020-07-21 16:12 [PATCH] [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs SZEDER Gábor
  2020-07-21 16:23 ` Eric Sunshine
  2020-07-21 20:34 ` Junio C Hamano
@ 2020-07-21 22:20 ` brian m. carlson
  2 siblings, 0 replies; 7+ messages in thread
From: brian m. carlson @ 2020-07-21 22:20 UTC (permalink / raw)
  To: SZEDER Gábor; +Cc: git

On 2020-07-21 at 16:12:25, SZEDER Gábor wrote:
> In our test suite, when 'git p4' invokes a Git command as a
> subprocesses, then it should run the 'git' binary we are testing.
> Unfortunately, this is not the case in the 'linux-clang' and
> 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system
> '/usr/bin/git' instead.
> 
> Travis CI's default Linux image includes 'pyenv', and all Python
> invocations that involve PATH lookup go through 'pyenv', e.g. our
> 'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as
> PYTHON_PATH, which in turn will invoke '/usr/bin/python3'.  Alas, the
> 'pyenv' version included in this image is buggy, and prepends the
> directory containing the Python binary to PATH even if that is a
> system directory already in PATH near the end.  Consequently, 'git p4'
> in those jobs ends up with its PATH starting with '/usr/bin', and then
> runs '/usr/bin/git'.

Ouch.  So we're testing git-p4, but not the git binary underlying it.
Fixing this definitely seems like a good idea.
-- 
brian m. carlson: Houston, Texas, US

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH v2] ci: use absolute PYTHON_PATH in the Linux jobs
  2020-07-21 20:34 ` Junio C Hamano
@ 2020-07-23 21:38   ` SZEDER Gábor
  2020-07-23 22:31     ` Junio C Hamano
  0 siblings, 1 reply; 7+ messages in thread
From: SZEDER Gábor @ 2020-07-23 21:38 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git, brian m. carlson, Eric Sunshine, SZEDER Gábor

In our test suite, when 'git p4' invokes a Git command as a
subprocesses, then it should run the 'git' binary we are testing.
Unfortunately, this is not the case in the 'linux-clang' and
'linux-gcc' jobs on Travis CI, where 'git p4' runs the system
'/usr/bin/git' instead.

Travis CI's default Linux image includes 'pyenv', and all Python
invocations that involve PATH lookup go through 'pyenv', e.g. our
'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as
PYTHON_PATH, which in turn will invoke '/usr/bin/python3'.  Alas, the
'pyenv' version included in this image is buggy, and prepends the
directory containing the Python binary to PATH even if that is a
system directory already in PATH near the end.  Consequently, 'git p4'
in those jobs ends up with its PATH starting with '/usr/bin', and then
runs '/usr/bin/git'.

So use the absolute paths '/usr/bin/python{2,3}' explicitly when
setting PYTHON_PATH in those Linux jobs to avoid the PATH lookup and
thus the bogus 'pyenv' from interfering with our 'git p4' tests.
Don't bother with special-casing Travis CI: while this issue doesn't
affect the corresponding Linux jobs on GitHub Actions, both CI systems
use Ubuntu LTS-based images, so we can safely rely on these Python
paths.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---

Junio, I see that you picked up the first/RFC version and applied it
on top of v2.26.2.  This patch won't work on v2.26.2, because its 'git
p4' is not compatible with python3 yet.

 ci/lib.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ci/lib.sh b/ci/lib.sh
index ff24c547c8..3eefec500d 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -184,9 +184,9 @@ linux-clang|linux-gcc)
 	if [ "$jobname" = linux-gcc ]
 	then
 		export CC=gcc-8
-		MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)"
+		MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3"
 	else
-		MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)"
+		MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python2"
 	fi
 
 	export GIT_TEST_HTTPD=true
-- 
2.28.0.rc2.319.g2e16345e2b


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH v2] ci: use absolute PYTHON_PATH in the Linux jobs
  2020-07-23 21:38   ` [PATCH v2] ci: use absolute PYTHON_PATH " SZEDER Gábor
@ 2020-07-23 22:31     ` Junio C Hamano
  2020-07-24 13:58       ` SZEDER Gábor
  0 siblings, 1 reply; 7+ messages in thread
From: Junio C Hamano @ 2020-07-23 22:31 UTC (permalink / raw)
  To: SZEDER Gábor; +Cc: git, brian m. carlson, Eric Sunshine

SZEDER Gábor <szeder.dev@gmail.com> writes:

> In our test suite, when 'git p4' invokes a Git command as a
> subprocesses, then it should run the 'git' binary we are testing.
> Unfortunately, this is not the case in the 'linux-clang' and
> 'linux-gcc' jobs on Travis CI, where 'git p4' runs the system
> '/usr/bin/git' instead.
>
> Travis CI's default Linux image includes 'pyenv', and all Python
> invocations that involve PATH lookup go through 'pyenv', e.g. our
> 'PYTHON_PATH=$(which python3)' sets '/opt/pyenv/shims/python3' as
> PYTHON_PATH, which in turn will invoke '/usr/bin/python3'.  Alas, the
> 'pyenv' version included in this image is buggy, and prepends the
> directory containing the Python binary to PATH even if that is a
> system directory already in PATH near the end.  Consequently, 'git p4'
> in those jobs ends up with its PATH starting with '/usr/bin', and then
> runs '/usr/bin/git'.
>
> So use the absolute paths '/usr/bin/python{2,3}' explicitly when
> setting PYTHON_PATH in those Linux jobs to avoid the PATH lookup and
> thus the bogus 'pyenv' from interfering with our 'git p4' tests.
> Don't bother with special-casing Travis CI: while this issue doesn't
> affect the corresponding Linux jobs on GitHub Actions, both CI systems
> use Ubuntu LTS-based images, so we can safely rely on these Python
> paths.

Good.  This does not need to touch pyenv but take advantage of the
fact that we know where Python 2 and 3 are.  Nice.

>
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
>
> Junio, I see that you picked up the first/RFC version and applied it
> on top of v2.26.2.  This patch won't work on v2.26.2, because its 'git
> p4' is not compatible with python3 yet.

Thanks.  Which means that we do not need to touch maint track (not
that we'd be merging the SHA-256 topic down to the maint track
anyway).

Will queue.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH v2] ci: use absolute PYTHON_PATH in the Linux jobs
  2020-07-23 22:31     ` Junio C Hamano
@ 2020-07-24 13:58       ` SZEDER Gábor
  0 siblings, 0 replies; 7+ messages in thread
From: SZEDER Gábor @ 2020-07-24 13:58 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git, brian m. carlson, Eric Sunshine

On Thu, Jul 23, 2020 at 03:31:56PM -0700, Junio C Hamano wrote:
> SZEDER Gábor <szeder.dev@gmail.com> writes:
> > Junio, I see that you picked up the first/RFC version and applied it
> > on top of v2.26.2.  This patch won't work on v2.26.2, because its 'git
> > p4' is not compatible with python3 yet.
> 
> Thanks.  Which means that we do not need to touch maint track 

Yeah, that system Git is v2.21.x, as far as I remember, but apparently
'git p4' doesn't use any shiny new features since then, because all
its test succeeded even with that old version.

> (not
> that we'd be merging the SHA-256 topic down to the maint track
> anyway).

It might be worth applying the final SHA-256 topic on top of this,
though, so Travis CI builds of that branch on its own can succeed.


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2020-07-24 13:58 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-21 16:12 [PATCH] [RFC] travis-ci: remove bogus 'pyenv' in the Linux jobs SZEDER Gábor
2020-07-21 16:23 ` Eric Sunshine
2020-07-21 20:34 ` Junio C Hamano
2020-07-23 21:38   ` [PATCH v2] ci: use absolute PYTHON_PATH " SZEDER Gábor
2020-07-23 22:31     ` Junio C Hamano
2020-07-24 13:58       ` SZEDER Gábor
2020-07-21 22:20 ` [PATCH] [RFC] travis-ci: remove bogus 'pyenv' " brian m. carlson

git@vger.kernel.org list mirror (unofficial, one of many)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 git git/ https://public-inbox.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.version-control.git
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.version-control.git
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git