[PATCH] ci: install P4 from package to fix build error

[PATCH] ci: install P4 from package to fix build error

[SZEDER Gábor]

To test 'git-p4' in the Linux Clang and GCC build jobs we used to
install the 'p4' and 'p4d' binaries by directly downloading those
binaries from a Perforce filehost.  This has worked just fine ever
since we started using Travis CI [1], but during the last day or so
that filehost appeared to be gone: while its hostname still resolves,
the host doesn't seem to reply to any download request, it doesn't
even refuse the connection, and eventually our build jobs time out
[2].

Now, this might be just a temporary glitch, but I'm afraid that it
isn't.  The "Helix Core Server Administrator Guide" [3] describes two
ways to install these binaries on Linux, and none of them mentions the
filehost that we've been downloading from in the past:

  - non-package installation: open the website's download page in your
    web browser, select OS and platform, click on the download link,
    and eventually you get a .tar.gz archive containing, among other
    things, the necessary 'p4' and 'p4d' binaries.

    Although we could use the URL of this archive to download it in
    our CI scripts with 'wget', nobody said that that URL remains
    stable, and we would still need to extract the archive and copy
    the binaries to $PATH.

  - package installation for various distros, including Ubuntu 16.04
    (i.e. the Ubuntu version used both in our Travis CI and Azure
    Pipelines builds): add a package repository and its pubkey,
    'apt-get update && apt-get install', and ready to go.

Let's install P4 from the package repository, because this approach
seems to be simpler and more future proof.

Note that we used to install an old P4 version (2016.2) in the Linux
build jobs, but with this change we'll install the most recent version
available in the Perforce package repository (currently 2019.1).

[1] 522354d70f (Add Travis CI support, 2015-11-27).
[2] https://travis-ci.org/git/git/jobs/581429927#L422
[3] https://www.perforce.com/manuals/p4sag/Content/P4SAG/chapter.install.html

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 ci/install-dependencies.sh | 14 +++++---------
 ci/lib.sh                  |  8 +++-----
 2 files changed, 8 insertions(+), 14 deletions(-)

diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
index 8cc72503cb..0df48365dc 100755
--- a/ci/install-dependencies.sh
+++ b/ci/install-dependencies.sh
@@ -5,27 +5,23 @@
 
 . ${0%/*}/lib.sh
 
-P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION
 LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION
 
 case "$jobname" in
 linux-clang|linux-gcc)
+	wget -qO - https://package.perforce.com/perforce.pubkey | sudo apt-key add -
+	echo "deb http://package.perforce.com/apt/ubuntu xenial release" >perforce.list
+	sudo mv perforce.list /etc/apt/sources.list.d/
 	sudo apt-add-repository -y "ppa:ubuntu-toolchain-r/test"
 	sudo apt-get -q update
-	sudo apt-get -q -y install language-pack-is libsvn-perl apache2
+	sudo apt-get -q -y install language-pack-is libsvn-perl apache2 \
+		helix-p4d
 	case "$jobname" in
 	linux-gcc)
 		sudo apt-get -q -y install gcc-8
 		;;
 	esac
 
-	mkdir --parents "$P4_PATH"
-	pushd "$P4_PATH"
-		wget --quiet "$P4WHENCE/bin.linux26x86_64/p4d"
-		wget --quiet "$P4WHENCE/bin.linux26x86_64/p4"
-		chmod u+x p4d
-		chmod u+x p4
-	popd
 	mkdir --parents "$GIT_LFS_PATH"
 	pushd "$GIT_LFS_PATH"
 		wget --quiet "$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
diff --git a/ci/lib.sh b/ci/lib.sh
index 44db2d5cbb..efcccfee6f 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -162,17 +162,15 @@ linux-clang|linux-gcc)
 
 	export GIT_TEST_HTTPD=YesPlease
 
-	# The Linux build installs the defined dependency versions below.
-	# The OS X build installs much more recent versions, whichever
+	# The Linux build installs the defined dependency version below.
+	# The OS X build installs much more recent version, whichever
 	# were recorded in the Homebrew database upon creating the OS X
 	# image.
 	# Keep that in mind when you encounter a broken OS X build!
-	export LINUX_P4_VERSION="16.2"
 	export LINUX_GIT_LFS_VERSION="1.5.2"
 
-	P4_PATH="$HOME/custom/p4"
 	GIT_LFS_PATH="$HOME/custom/git-lfs"
-	export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH"
+	export PATH="$GIT_LFS_PATH:$PATH"
 	;;
 osx-clang|osx-gcc)
 	if [ "$jobname" = osx-gcc ]
-- 
2.23.0.331.g4e51dcdf11

Re: [PATCH] ci: install P4 from package to fix build error

[SZEDER Gábor]

On Fri, Sep 06, 2019 at 12:27:11PM +0200, SZEDER Gábor wrote:
> To test 'git-p4' in the Linux Clang and GCC build jobs we used to
> install the 'p4' and 'p4d' binaries by directly downloading those
> binaries from a Perforce filehost.  This has worked just fine ever
> since we started using Travis CI [1], but during the last day or so
> that filehost appeared to be gone: while its hostname still resolves,
> the host doesn't seem to reply to any download request, it doesn't
> even refuse the connection, and eventually our build jobs time out
> [2].
> 
> Now, this might be just a temporary glitch, but I'm afraid that it
> isn't.

Well, now would you believe it, while I was testing this patch (I even
made a gitgitgadget PR to run it on Azure Pipelines! :) and touching
up its log message the good old Perforce filehost sprang back to life,
and the CI build jobs now succeed again even without this patch.

> Let's install P4 from the package repository, because this approach
> seems to be simpler and more future proof.
> 
> Note that we used to install an old P4 version (2016.2) in the Linux
> build jobs, but with this change we'll install the most recent version
> available in the Perforce package repository (currently 2019.1).

So I'm not quite sure whether we really want this patch.  It depends
on how important it is to test 'git-p4' with an old P4 version, but I
don't really have an opinion on that.

Re: [PATCH] ci: install P4 from package to fix build error

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 1916 bytes --]

Hi,


On Fri, 6 Sep 2019, SZEDER Gábor wrote:

> On Fri, Sep 06, 2019 at 12:27:11PM +0200, SZEDER Gábor wrote:
> > To test 'git-p4' in the Linux Clang and GCC build jobs we used to
> > install the 'p4' and 'p4d' binaries by directly downloading those
> > binaries from a Perforce filehost.  This has worked just fine ever
> > since we started using Travis CI [1], but during the last day or so
> > that filehost appeared to be gone: while its hostname still resolves,
> > the host doesn't seem to reply to any download request, it doesn't
> > even refuse the connection, and eventually our build jobs time out
> > [2].
> >
> > Now, this might be just a temporary glitch, but I'm afraid that it
> > isn't.
>
> Well, now would you believe it, while I was testing this patch (I even
> made a gitgitgadget PR to run it on Azure Pipelines! :) and touching
> up its log message the good old Perforce filehost sprang back to life,
> and the CI build jobs now succeed again even without this patch.

Sorry for being so slow with granting you access to GitGitGadget. FWIW
_anybody_ who already was granted access can issue `/allow` commands, it
is not just me.

> > Let's install P4 from the package repository, because this approach
> > seems to be simpler and more future proof.
> >
> > Note that we used to install an old P4 version (2016.2) in the Linux
> > build jobs, but with this change we'll install the most recent version
> > available in the Perforce package repository (currently 2019.1).
>
> So I'm not quite sure whether we really want this patch.  It depends
> on how important it is to test 'git-p4' with an old P4 version, but I
> don't really have an opinion on that.

I'd rather have that patch. It seems to be a much better idea to use the
package management system than to rely on one host, especially when said
host already displayed hiccups.

Ciao,
Dscho

Re: [PATCH] ci: install P4 from package to fix build error

[SZEDER Gábor]

On Tue, Sep 10, 2019 at 12:51:01AM +0200, Johannes Schindelin wrote:
> On Fri, 6 Sep 2019, SZEDER Gábor wrote:
> 
> > On Fri, Sep 06, 2019 at 12:27:11PM +0200, SZEDER Gábor wrote:
> > > To test 'git-p4' in the Linux Clang and GCC build jobs we used to
> > > install the 'p4' and 'p4d' binaries by directly downloading those
> > > binaries from a Perforce filehost.  This has worked just fine ever
> > > since we started using Travis CI [1], but during the last day or so
> > > that filehost appeared to be gone: while its hostname still resolves,
> > > the host doesn't seem to reply to any download request, it doesn't
> > > even refuse the connection, and eventually our build jobs time out
> > > [2].
> > >
> > > Now, this might be just a temporary glitch, but I'm afraid that it
> > > isn't.
> >
> > Well, now would you believe it, while I was testing this patch (I even
> > made a gitgitgadget PR to run it on Azure Pipelines! :) and touching
> > up its log message the good old Perforce filehost sprang back to life,
> > and the CI build jobs now succeed again even without this patch.
> 
> Sorry for being so slow with granting you access to GitGitGadget. FWIW
> _anybody_ who already was granted access can issue `/allow` commands, it
> is not just me.

No problem; I was only interested in the results of the Azure
Pipelines build, and that seemed to go well even without access.

> > > Let's install P4 from the package repository, because this approach
> > > seems to be simpler and more future proof.
> > >
> > > Note that we used to install an old P4 version (2016.2) in the Linux
> > > build jobs, but with this change we'll install the most recent version
> > > available in the Perforce package repository (currently 2019.1).
> >
> > So I'm not quite sure whether we really want this patch.  It depends
> > on how important it is to test 'git-p4' with an old P4 version, but I
> > don't really have an opinion on that.
> 
> I'd rather have that patch. It seems to be a much better idea to use the
> package management system than to rely on one host, especially when said
> host already displayed hiccups.

Well, I'm not so sure.  As far as I remember this was the first time
that this Perforce filehost was inaccessible and a simple "Restart
job" could not rectify the situation, because it was inaccessible for
about a day or more.  OTOH, transient errors or timeouts from 'apt-get
update' or 'install' from the official Ubuntu package repositories are
not uncommon (at least on Travis CI), although in those cases it's
usually enough to just restart the errored job.

Re: [PATCH] ci: install P4 from package to fix build error

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

Hi,

On Tue, 10 Sep 2019, SZEDER Gábor wrote:

> On Tue, Sep 10, 2019 at 12:51:01AM +0200, Johannes Schindelin wrote:
> > On Fri, 6 Sep 2019, SZEDER Gábor wrote:
> >
> > > On Fri, Sep 06, 2019 at 12:27:11PM +0200, SZEDER Gábor wrote:
>
> > > > Let's install P4 from the package repository, because this
> > > > approach seems to be simpler and more future proof.
> > > >
> > > > Note that we used to install an old P4 version (2016.2) in the
> > > > Linux build jobs, but with this change we'll install the most
> > > > recent version available in the Perforce package repository
> > > > (currently 2019.1).
> > >
> > > So I'm not quite sure whether we really want this patch.  It
> > > depends on how important it is to test 'git-p4' with an old P4
> > > version, but I don't really have an opinion on that.
> >
> > I'd rather have that patch. It seems to be a much better idea to use
> > the package management system than to rely on one host, especially
> > when said host already displayed hiccups.
>
> Well, I'm not so sure.  As far as I remember this was the first time
> that this Perforce filehost was inaccessible and a simple "Restart
> job" could not rectify the situation, because it was inaccessible for
> about a day or more.

Right.

> OTOH, transient errors or timeouts from 'apt-get update' or 'install'
> from the official Ubuntu package repositories are not uncommon (at
> least on Travis CI), although in those cases it's usually enough to
> just restart the errored job.

My impression precisely. I trust the Ubuntu package servers to have
transient, _short-lived_ problems ;-)

Ciao,
Dscho

Re: [PATCH] ci: install P4 from package to fix build error

[SZEDER Gábor]

On Wed, Sep 11, 2019 at 05:12:31PM +0200, Johannes Schindelin wrote:
> On Tue, 10 Sep 2019, SZEDER Gábor wrote:
> 
> > On Tue, Sep 10, 2019 at 12:51:01AM +0200, Johannes Schindelin wrote:
> > > On Fri, 6 Sep 2019, SZEDER Gábor wrote:
> > >
> > > > On Fri, Sep 06, 2019 at 12:27:11PM +0200, SZEDER Gábor wrote:
> >
> > > > > Let's install P4 from the package repository, because this
> > > > > approach seems to be simpler and more future proof.
> > > > >
> > > > > Note that we used to install an old P4 version (2016.2) in the
> > > > > Linux build jobs, but with this change we'll install the most
> > > > > recent version available in the Perforce package repository
> > > > > (currently 2019.1).
> > > >
> > > > So I'm not quite sure whether we really want this patch.  It
> > > > depends on how important it is to test 'git-p4' with an old P4
> > > > version, but I don't really have an opinion on that.
> > >
> > > I'd rather have that patch. It seems to be a much better idea to use
> > > the package management system than to rely on one host, especially
> > > when said host already displayed hiccups.
> >
> > Well, I'm not so sure.  As far as I remember this was the first time
> > that this Perforce filehost was inaccessible and a simple "Restart
> > job" could not rectify the situation, because it was inaccessible for
> > about a day or more.
> 
> Right.
> 
> > OTOH, transient errors or timeouts from 'apt-get update' or 'install'
> > from the official Ubuntu package repositories are not uncommon (at
> > least on Travis CI), although in those cases it's usually enough to
> > just restart the errored job.
> 
> My impression precisely. I trust the Ubuntu package servers to have
> transient, _short-lived_ problems ;-)

Heh, that's true, but note that we won't install P4 from the Ubuntu
package servers, but from the Perforce package repository, so I doubt
we can expect any improvements in this regard.

OTOH, instead of issuing only two HTTP requests to download those two
binaries we'll issue a total of 9 requests (1 pubkey, 2 package lists,
6 packages) that could go wrong.

I'm undecided, and at the very least the proposed commit message
should be updated.