* Travis not looking so good
@ 2019-05-30 19:32 Johannes Schindelin
2019-06-01 0:41 ` brian m. carlson
0 siblings, 1 reply; 18+ messages in thread
From: Johannes Schindelin @ 2019-05-30 19:32 UTC (permalink / raw)
To: SZEDER Gábor; +Cc: git
[-- Attachment #1: Type: text/plain, Size: 655 bytes --]
Hi Gábor,
do you have any idea why Travis is failing like this in the macOS/gcc
job?
> +case "$jobname" in
> +brew link gcc@8
> Error: No such keg: /usr/local/Cellar/gcc@8
> The command "ci/install-dependencies.sh" failed and exited with 1 during .
I usually only look at the Azure Pipelines (which gives me plenty enough
to do, what with pu's individual branches being tested individually), but
couldn't fail to notice that *all* four branches (maint, master, next and
pu) fail in Travis' macOS/gcc job (and only there, the Azure Pipelines are
all green):
https://github.com/git/git/branches/all
What's going on?
Ciao,
Dscho
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Travis not looking so good 2019-05-30 19:32 Travis not looking so good Johannes Schindelin @ 2019-06-01 0:41 ` brian m. carlson 2019-06-02 11:22 ` SZEDER Gábor 0 siblings, 1 reply; 18+ messages in thread From: brian m. carlson @ 2019-06-01 0:41 UTC (permalink / raw) To: Johannes Schindelin; +Cc: SZEDER Gábor, git [-- Attachment #1: Type: text/plain, Size: 1452 bytes --] On 2019-05-30 at 19:32:41, Johannes Schindelin wrote: > Hi Gábor, > > do you have any idea why Travis is failing like this in the macOS/gcc > job? > > > +case "$jobname" in > > +brew link gcc@8 > > Error: No such keg: /usr/local/Cellar/gcc@8 > > The command "ci/install-dependencies.sh" failed and exited with 1 during . > > I usually only look at the Azure Pipelines (which gives me plenty enough > to do, what with pu's individual branches being tested individually), but > couldn't fail to notice that *all* four branches (maint, master, next and > pu) fail in Travis' macOS/gcc job (and only there, the Azure Pipelines are > all green): > > https://github.com/git/git/branches/all > > What's going on? I'm certainly not an expert on macOS, so I could be getting things wrong here, but it looks like we don't ever explicitly install the gcc@8 package from Homebrew. It may have happened to be installed automatically by Travis in the past, but they may have upgraded now that GCC 9 has been released. I suspect if we want to use GCC 8, we need to explicitly install it by using "brew install gcc@8", or we can just pick the latest released GCC by using "brew install gcc" if we like that better. We will still need to do "brew link gcc" (or "gcc@8"), since I suspect Homebrew won't auto-link it since macOS provides a gcc binary. -- brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 868 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Travis not looking so good 2019-06-01 0:41 ` brian m. carlson @ 2019-06-02 11:22 ` SZEDER Gábor 2019-06-26 20:35 ` Taylor Blau 0 siblings, 1 reply; 18+ messages in thread From: SZEDER Gábor @ 2019-06-02 11:22 UTC (permalink / raw) To: brian m. carlson, Johannes Schindelin, git On Sat, Jun 01, 2019 at 12:41:43AM +0000, brian m. carlson wrote: > On 2019-05-30 at 19:32:41, Johannes Schindelin wrote: > > Hi Gábor, > > > > do you have any idea why Travis is failing like this in the macOS/gcc > > job? > > > > > +case "$jobname" in > > > +brew link gcc@8 > > > Error: No such keg: /usr/local/Cellar/gcc@8 > > > The command "ci/install-dependencies.sh" failed and exited with 1 during . > > > > I usually only look at the Azure Pipelines (which gives me plenty enough > > to do, what with pu's individual branches being tested individually), but > > couldn't fail to notice that *all* four branches (maint, master, next and > > pu) fail in Travis' macOS/gcc job (and only there, the Azure Pipelines are > > all green): > > > > https://github.com/git/git/branches/all > > > > What's going on? The usual: Homebrew desperately tries to be overly clever and helpful, but ends up being dumb and annoying. :) I was hoping that this issue will just solve itself, like several other brew breakages in the past, but apparently it won't... > I'm certainly not an expert on macOS, so I could be getting things wrong > here, but it looks like we don't ever explicitly install the gcc@8 > package from Homebrew. It may have happened to be installed > automatically by Travis in the past, but they may have upgraded now that > GCC 9 has been released. gcc@8 is still installed in the Travis CI macOS image we use, but it is version 8.2. It's still perfectly usable, but: When installing dependencies on macOS, we first explicitly run 'brew update', which spends over two minutes to update itself and information about the available packages. This way it learns about GCC 8.3, and after that gcc@8 refers to v8.3, and, unfortunately, 'brew' is just too dumb to be able to do anything with the still installed 8.2 package, and the subsequent 'brew link gcc@8' fails. But hey, even 'brew uninstall gcc@8' fails with the same error after that, which is simply unacceptable from a package manager in my book. > I suspect if we want to use GCC 8, we need to explicitly install it by > using "brew install gcc@8", or we can just pick the latest released GCC > by using "brew install gcc" if we like that better. We will still need > to do "brew link gcc" (or "gcc@8"), since I suspect Homebrew won't > auto-link it since macOS provides a gcc binary. Yeah, installing gcc@8 or gcc works. Back in 2c8921db2b (travis-ci: build with the right compiler, 2019-01-17) I opted for simply linking the already installed gcc@8 package, because GCC is big, installing it takes time, and the macOS build jobs have always been prone to exceeding the time limit. (Note that these packages provide 'gcc-8' and 'gcc-9' binaries, not 'gcc', and after 'brew install'-ing them we won't need an additional link step (I'm not sure why linking is necessary with the gcc@8 package already installed in the Travis CI image).) Another possibilities are: - Running 'brew link gcc@8' before 'brew update' works: https://travis-ci.org/szeder/git/jobs/540027012#L139 - Not running 'brew update' at all works as well: https://travis-ci.org/szeder/git/jobs/514960153#L179 However, it's not as simple as removing the 'brew update' command from our ci script, because 'brew' is so very helpful that it implicitly runs update for us on the next 'brew install' anyway... we have to 'export HOMEBREW_NO_AUTO_UPDATE=1' first to disable that. As an additional benefit, disabling 'brew update' shaves off over 2mins from the ~4mins currently spent on installing dependencies [1]. I've been running my somewhat customized builds on Travis CI [2] without 'brew update' for about a month now, no ill effects yet. Initially I was worried that without 'brew update' a subsequent 'brew install <pkg>' might fail, if the package repositories contain only a newer version of 'pkg'. I haven't seen any such issues yet, and my builds can still 'brew install' Git-LFS 2.5.2 and Perforce 18.1, although their current versions are 2.7.2 and 19.1, respectively. [1] And disabling the recently introduced auto cleanup feature saves some more time, and installing all dependencies in the macOS builds can be done in ~20-30secs: https://travis-ci.org/szeder/git/jobs/514960153#L92 [2] https://travis-ci.org/szeder/git-cooking-topics-for-travis-ci/branches ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Travis not looking so good 2019-06-02 11:22 ` SZEDER Gábor @ 2019-06-26 20:35 ` Taylor Blau 2019-06-27 13:23 ` SZEDER Gábor 0 siblings, 1 reply; 18+ messages in thread From: Taylor Blau @ 2019-06-26 20:35 UTC (permalink / raw) To: SZEDER Gábor; +Cc: brian m. carlson, Johannes Schindelin, git Hi Gábor, On Sun, Jun 02, 2019 at 01:22:39PM +0200, SZEDER Gábor wrote: > On Sat, Jun 01, 2019 at 12:41:43AM +0000, brian m. carlson wrote: > > On 2019-05-30 at 19:32:41, Johannes Schindelin wrote: > > > Hi Gábor, > > > > > > do you have any idea why Travis is failing like this in the macOS/gcc > > > job? > > > > > > > +case "$jobname" in > > > > +brew link gcc@8 > > > > Error: No such keg: /usr/local/Cellar/gcc@8 > > > > The command "ci/install-dependencies.sh" failed and exited with 1 during . > > > > > > I usually only look at the Azure Pipelines (which gives me plenty enough > > > to do, what with pu's individual branches being tested individually), but > > > couldn't fail to notice that *all* four branches (maint, master, next and > > > pu) fail in Travis' macOS/gcc job (and only there, the Azure Pipelines are > > > all green): > > > > > > https://github.com/git/git/branches/all > > > > > > What's going on? > > The usual: Homebrew desperately tries to be overly clever and helpful, > but ends up being dumb and annoying. :) > > I was hoping that this issue will just solve itself, like several > other brew breakages in the past, but apparently it won't... I have noticed this as well on my own fork's TravisCI builds. > > I suspect if we want to use GCC 8, we need to explicitly install it by > > using "brew install gcc@8", or we can just pick the latest released GCC > > by using "brew install gcc" if we like that better. We will still need > > to do "brew link gcc" (or "gcc@8"), since I suspect Homebrew won't > > auto-link it since macOS provides a gcc binary. > > Yeah, installing gcc@8 or gcc works. Back in 2c8921db2b (travis-ci: > build with the right compiler, 2019-01-17) I opted for simply linking > the already installed gcc@8 package, because GCC is big, installing it > takes time, and the macOS build jobs have always been prone to > exceeding the time limit. (Note that these packages provide 'gcc-8' > and 'gcc-9' binaries, not 'gcc', and after 'brew install'-ing them we > won't need an additional link step (I'm not sure why linking is > necessary with the gcc@8 package already installed in the Travis CI > image).) I wrote something like this up in [1] before I realized that you had your own patches in [2]. This did fix things, but it's kind of awkward in the sense that we're not really "installing" anything (in fact, the patch in [1] incorrectly indicates that we are), but instead nudging it after it discovers v8.3. > Another possibilities are: > > - Running 'brew link gcc@8' before 'brew update' works: > > https://travis-ci.org/szeder/git/jobs/540027012#L139 > > - Not running 'brew update' at all works as well: > > https://travis-ci.org/szeder/git/jobs/514960153#L179 I'd be just as happy to do something similar to what I did as really either of these. Getting rid of 'brew update' entirely would make me happiest, since it takes a *long* time for one of these to complete. But, I'd almost prefer explicitly running 'brew install gcc@8' to running 'brew link gcc@8' before 'brew update'. The later seems fragile and awfully prone to break, especially when we are just doing it to try and work around a Homebrew quirk. If you don't have any plans to send your patches upstream, and feel OK running 'brew install', let me know and I will send my patch in [1]. Thanks, Taylor [1]: https://github.com/ttaylorr/git/commit/a20e34d143a4a15b6b15ccb5bfb996fab9551b76 [2]: https://github.com/szeder/git/commit/ca29709d09a440d98857efb2575a3f92feaab29f ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Travis not looking so good 2019-06-26 20:35 ` Taylor Blau @ 2019-06-27 13:23 ` SZEDER Gábor 2019-06-27 16:46 ` Junio C Hamano 0 siblings, 1 reply; 18+ messages in thread From: SZEDER Gábor @ 2019-06-27 13:23 UTC (permalink / raw) To: Taylor Blau; +Cc: brian m. carlson, Johannes Schindelin, git On Wed, Jun 26, 2019 at 03:35:59PM -0500, Taylor Blau wrote: > Hi Gábor, > > On Sun, Jun 02, 2019 at 01:22:39PM +0200, SZEDER Gábor wrote: > > On Sat, Jun 01, 2019 at 12:41:43AM +0000, brian m. carlson wrote: > > > On 2019-05-30 at 19:32:41, Johannes Schindelin wrote: > > > > Hi Gábor, > > > > > > > > do you have any idea why Travis is failing like this in the macOS/gcc > > > > job? > > > > > > > > > +case "$jobname" in > > > > > +brew link gcc@8 > > > > > Error: No such keg: /usr/local/Cellar/gcc@8 > > > > > The command "ci/install-dependencies.sh" failed and exited with 1 during . > > > > > > > > I usually only look at the Azure Pipelines (which gives me plenty enough > > > > to do, what with pu's individual branches being tested individually), but > > > > couldn't fail to notice that *all* four branches (maint, master, next and > > > > pu) fail in Travis' macOS/gcc job (and only there, the Azure Pipelines are > > > > all green): > > > > > > > > https://github.com/git/git/branches/all > > > > > > > > What's going on? > > > > The usual: Homebrew desperately tries to be overly clever and helpful, > > but ends up being dumb and annoying. :) > > > > I was hoping that this issue will just solve itself, like several > > other brew breakages in the past, but apparently it won't... > > I have noticed this as well on my own fork's TravisCI builds. > > > > I suspect if we want to use GCC 8, we need to explicitly install it by > > > using "brew install gcc@8", or we can just pick the latest released GCC > > > by using "brew install gcc" if we like that better. We will still need > > > to do "brew link gcc" (or "gcc@8"), since I suspect Homebrew won't > > > auto-link it since macOS provides a gcc binary. > > > > Yeah, installing gcc@8 or gcc works. Back in 2c8921db2b (travis-ci: > > build with the right compiler, 2019-01-17) I opted for simply linking > > the already installed gcc@8 package, because GCC is big, installing it > > takes time, and the macOS build jobs have always been prone to > > exceeding the time limit. Oh, and now I recall that simply running 'brew install gcc@8' back then (or running it before 'brew update' nowadays) errored out with something along the lines of "gcc@8 is already installed, you only have to link it". > > (Note that these packages provide 'gcc-8' > > and 'gcc-9' binaries, not 'gcc', and after 'brew install'-ing them we > > won't need an additional link step (I'm not sure why linking is > > necessary with the gcc@8 package already installed in the Travis CI > > image).) > > I wrote something like this up in [1] before I realized that you had > your own patches in [2]. This did fix things, but it's kind of awkward > in the sense that we're not really "installing" anything (in fact, the > patch in [1] incorrectly indicates that we are), but instead nudging it > after it discovers v8.3. > > > > Another possibilities are: > > > > - Running 'brew link gcc@8' before 'brew update' works: > > > > https://travis-ci.org/szeder/git/jobs/540027012#L139 > > > > - Not running 'brew update' at all works as well: > > > > https://travis-ci.org/szeder/git/jobs/514960153#L179 > > I'd be just as happy to do something similar to what I did as really > either of these. Getting rid of 'brew update' entirely would make me > happiest, since it takes a *long* time for one of these to complete. I would love to see 'brew update' go away, partly because it takes so long, and partly because the update of Homebrew itself broke our builds once or twice in the past (though they usually sorted out the breakage in a few days) [1]. However, we've always used the macOS build jobs as "build and test with the latest and greatest", i.e. they install the latest available Perforce and Git-LFS. To keep up with this tradition we'd need to run 'brew update' and in turn would need to 'brew install gcc'. [1] See e.g. a1ccaedd62 (travis-ci: make the OSX build jobs' 'brew update' more quiet, 2019-02-02) or https://public-inbox.org/git/20180907032002.23366-1-szeder.dev@gmail.com/T/#u > But, I'd almost prefer explicitly running 'brew install gcc@8' to > running 'brew link gcc@8' before 'brew update'. The later seems fragile > and awfully prone to break, especially when we are just doing it to try > and work around a Homebrew quirk. > > If you don't have any plans to send your patches upstream, and feel OK > running 'brew install', let me know and I will send my patch in [1]. > > Thanks, > Taylor > > [1]: https://github.com/ttaylorr/git/commit/a20e34d143a4a15b6b15ccb5bfb996fab9551b76 > [2]: https://github.com/szeder/git/commit/ca29709d09a440d98857efb2575a3f92feaab29f ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Travis not looking so good 2019-06-27 13:23 ` SZEDER Gábor @ 2019-06-27 16:46 ` Junio C Hamano 2019-06-29 17:01 ` SZEDER Gábor 0 siblings, 1 reply; 18+ messages in thread From: Junio C Hamano @ 2019-06-27 16:46 UTC (permalink / raw) To: SZEDER Gábor; +Cc: Taylor Blau, brian m. carlson, Johannes Schindelin, git SZEDER Gábor <szeder.dev@gmail.com> writes: > However, we've always used the macOS build jobs as "build and test > with the latest and greatest", i.e. they install the latest available > Perforce and Git-LFS. To keep up with this tradition we'd need to run > 'brew update' and in turn would need to 'brew install gcc'. > > [1] See e.g. a1ccaedd62 (travis-ci: make the OSX build jobs' 'brew > update' more quiet, 2019-02-02) or > > https://public-inbox.org/git/20180907032002.23366-1-szeder.dev@gmail.com/T/#u Is the reason why you did not submit your bonus patch [*1*] in the series at https://public-inbox.org/git/20190614100059.13540-1-szeder.dev@gmail.com/ because it goes the opposite way, i.e. "build and test with whatever happens to be in the image"? Unless what happens to come in the image at travis-ci.org is hopelessly outdated and does not match what normal users run, isn't it is better to test with "whatever happens to be there" than not to test at all due to missing compiler? [Reference] *1* ... what I picked up from your branch reproduced here https://public-inbox.org/git/xmqqy324t4g0.fsf@gitster-ct.c.googlers.com/ It is missing explanation and sign-off, but if it lets the build jobs run, even in a tad stale environment, it may be worth resurrecting until those who want macOS port working can come up with a real "with the latest and greatest" alternative. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Travis not looking so good 2019-06-27 16:46 ` Junio C Hamano @ 2019-06-29 17:01 ` SZEDER Gábor 2019-07-03 10:47 ` [PATCH 1/2] ci: don't update Homebrew SZEDER Gábor 0 siblings, 1 reply; 18+ messages in thread From: SZEDER Gábor @ 2019-06-29 17:01 UTC (permalink / raw) To: Junio C Hamano; +Cc: Taylor Blau, brian m. carlson, Johannes Schindelin, git On Thu, Jun 27, 2019 at 09:46:51AM -0700, Junio C Hamano wrote: > SZEDER Gábor <szeder.dev@gmail.com> writes: > > > However, we've always used the macOS build jobs as "build and test > > with the latest and greatest", i.e. they install the latest available > > Perforce and Git-LFS. To keep up with this tradition we'd need to run > > 'brew update' and in turn would need to 'brew install gcc'. > > > > [1] See e.g. a1ccaedd62 (travis-ci: make the OSX build jobs' 'brew > > update' more quiet, 2019-02-02) or > > > > https://public-inbox.org/git/20180907032002.23366-1-szeder.dev@gmail.com/T/#u > > Is the reason why you did not submit your bonus patch [*1*] in the series at > > https://public-inbox.org/git/20190614100059.13540-1-szeder.dev@gmail.com/ > > because it goes the opposite way, i.e. "build and test with whatever > happens to be in the image"? Basically yes... with the other factor being that when I'm not particularly happy with any of the possible solutions for an issue, then it tends to end up on a back burner and forgotten for a while... > Unless what happens to come in the image at travis-ci.org is > hopelessly outdated and does not match what normal users run, We explicitly specify which macOS image we want to use in our builds on Travis CI, see 2000ac9fbf (travis-ci: switch to Xcode 10.1 macOS image, 2019-01-17), so it's up to us to change that. Travis CI's current default macOS image is still Xcode 9.4, as it was at the time of 2000ac9fbf, and the newest is 10.2, which now comes with GCC 8.3 properly preinstalled (i.e. no 'brew link gcc@8' necessary). > isn't > it is better to test with "whatever happens to be there" than not to > test at all due to missing compiler? Better, sure... Right? I'm not sure. > [Reference] > > *1* ... what I picked up from your branch reproduced here > > https://public-inbox.org/git/xmqqy324t4g0.fsf@gitster-ct.c.googlers.com/ > > It is missing explanation and sign-off, And that patch should be split into two, of course: setting HOMEBREW_NO_AUTO_UPDATE first to fix the build, and then set HOMEBREW_NO_INSTALL_CLEANUP separately for a bit of additional speedup. > but if it lets the build > jobs run, even in a tad stale environment, it may be worth > resurrecting until those who want macOS port working can come up > with a real "with the latest and greatest" alternative. 'brew update && brew install gcc && export CC=gcc-9' will do it... but then we'll still spend that ~2.5 minutes spent on updating Homebrew itself, and there's the possibility that Homebrew breaks, and eventually GCC 10 will be released and we'll need to update that CC variable. Anyway, will look into this again later during the weekend. ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/2] ci: don't update Homebrew 2019-06-29 17:01 ` SZEDER Gábor @ 2019-07-03 10:47 ` SZEDER Gábor 2019-07-03 10:47 ` [PATCH 2/2] ci: disable Homebrew's auto cleanup SZEDER Gábor ` (2 more replies) 0 siblings, 3 replies; 18+ messages in thread From: SZEDER Gábor @ 2019-07-03 10:47 UTC (permalink / raw) To: Junio C Hamano Cc: Johannes Schindelin, brian m. carlson, Taylor Blau, git, SZEDER Gábor Lately our GCC macOS build job on Travis CI has been erroring out while installing dependencies with: +brew link gcc@8 Error: No such keg: /usr/local/Cellar/gcc@8 The command "ci/install-dependencies.sh" failed and exited with 1 during . Now, while gcc@8 is still pre-installed (but not linked) and would be perfectly usable in the Travis CI macOS image we use [1], it's at version 8.2. However, when installing dependencies we first explicitly run 'brew update', which spends over two minutes to update itself and information about the available packages, and it learns about GCC 8.3. After that point gcc@8 exclusively refers to v8.3, and, unfortunately, 'brew' is just too dumb to be able to do anything with the still installed 8.2 package, and the subsequent 'brew link gcc@8' fails. (Even 'brew uninstall gcc@8' fails with the same error!) Don't run 'brew update' to keep the already installed GCC 8.2 'brew link'-able. Note that in addition we have to 'export HOMEBREW_NO_AUTO_UPDATE=1' first, because 'brew' is so very helpful that it would implicitly run update for us on the next 'brew install <pkg>' otherwise. Disabling 'brew update' has additional benefits: - It shaves off 2-3mins from the ~4mins currently spent on installing dependencies, and the macOS build jobs have always been prone to exceeding the time limit on Travis CI. - Our builds won't suddenly break because of the occasional Homebrew breakages [2]. The drawback is that we'll be stuck with slightly older versions of the packages that we install via Homebrew (Git-LFS 2.5.2 and Perforce 2018.1; they are currently at 2.7.2 and 2019.1, respectively). We might want to reconsider this decision as time goes on and/or switch to a more recent macOS image as they become available. [1] 2000ac9fbf (travis-ci: switch to Xcode 10.1 macOS image, 2019-01-17) [2] See e.g. a1ccaedd62 (travis-ci: make the OSX build jobs' 'brew update' more quiet, 2019-02-02) or https://public-inbox.org/git/20180907032002.23366-1-szeder.dev@gmail.com/T/#+u Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> --- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 7f6acdd803..7f546c8552 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -34,7 +34,7 @@ linux-clang|linux-gcc) popd ;; osx-clang|osx-gcc) - brew update >/dev/null + export HOMEBREW_NO_AUTO_UPDATE=1 # Uncomment this if you want to run perf tests: # brew install gnu-time test -z "$BREW_INSTALL_PACKAGES" || -- 2.22.0.621.ge52941b842.dirty ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 2/2] ci: disable Homebrew's auto cleanup 2019-07-03 10:47 ` [PATCH 1/2] ci: don't update Homebrew SZEDER Gábor @ 2019-07-03 10:47 ` SZEDER Gábor 2019-07-03 11:49 ` Johannes Schindelin 2019-07-03 12:26 ` Thomas Braun 2019-07-06 16:16 ` [PATCH] ci/lib.sh: update a comment about installed P4 and Git-LFS versions SZEDER Gábor 2019-07-09 16:32 ` [PATCH 1/2] ci: don't update Homebrew Taylor Blau 2 siblings, 2 replies; 18+ messages in thread From: SZEDER Gábor @ 2019-07-03 10:47 UTC (permalink / raw) To: Junio C Hamano Cc: Johannes Schindelin, brian m. carlson, Taylor Blau, git, SZEDER Gábor Lately Homebrew learned to automagically clean up information about outdated packages during other 'brew' commands, which might be useful for the avarage user, but is a waste of time in CI build jobs, because the next build jobs will start from the exact same image containing the same outdated packages anyway. Export HOMEBREW_NO_AUTO_UPDATE=1 to disable this auto cleanup feature, shaving off about 20-30s from the time needed to install dependencies in our macOS build jobs on Travis CI. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> --- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 7f546c8552..8cc72503cb 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -34,7 +34,7 @@ linux-clang|linux-gcc) popd ;; osx-clang|osx-gcc) - export HOMEBREW_NO_AUTO_UPDATE=1 + export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1 # Uncomment this if you want to run perf tests: # brew install gnu-time test -z "$BREW_INSTALL_PACKAGES" || -- 2.22.0.621.ge52941b842.dirty ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 2/2] ci: disable Homebrew's auto cleanup 2019-07-03 10:47 ` [PATCH 2/2] ci: disable Homebrew's auto cleanup SZEDER Gábor @ 2019-07-03 11:49 ` Johannes Schindelin 2019-07-03 12:26 ` Thomas Braun 1 sibling, 0 replies; 18+ messages in thread From: Johannes Schindelin @ 2019-07-03 11:49 UTC (permalink / raw) To: SZEDER Gábor; +Cc: Junio C Hamano, brian m. carlson, Taylor Blau, git [-- Attachment #1: Type: text/plain, Size: 640 bytes --] Hi, On Wed, 3 Jul 2019, SZEDER Gábor wrote: > Lately Homebrew learned to automagically clean up information about > outdated packages during other 'brew' commands, which might be useful > for the avarage user, but is a waste of time in CI build jobs, because > the next build jobs will start from the exact same image containing > the same outdated packages anyway. > > Export HOMEBREW_NO_AUTO_UPDATE=1 to disable this auto cleanup feature, > shaving off about 20-30s from the time needed to install dependencies > in our macOS build jobs on Travis CI. I am very much in favor of both of these patches. Thanks, Dscho ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/2] ci: disable Homebrew's auto cleanup 2019-07-03 10:47 ` [PATCH 2/2] ci: disable Homebrew's auto cleanup SZEDER Gábor 2019-07-03 11:49 ` Johannes Schindelin @ 2019-07-03 12:26 ` Thomas Braun 2019-07-03 13:04 ` SZEDER Gábor 1 sibling, 1 reply; 18+ messages in thread From: Thomas Braun @ 2019-07-03 12:26 UTC (permalink / raw) To: SZEDER Gábor, Junio C Hamano Cc: Johannes Schindelin, brian m. carlson, Taylor Blau, git Am 03.07.2019 um 12:47 schrieb SZEDER Gábor: > Lately Homebrew learned to automagically clean up information about > outdated packages during other 'brew' commands, which might be useful > for the avarage user, but is a waste of time in CI build jobs, because > the next build jobs will start from the exact same image containing > the same outdated packages anyway. > > Export HOMEBREW_NO_AUTO_UPDATE=1 to disable this auto cleanup feature, The patch below adds HOMEBREW_NO_INSTALL_CLEANUP so maybe you meant HOMEBREW_NO_INSTALL_CLEANUP here as well? > shaving off about 20-30s from the time needed to install dependencies > in our macOS build jobs on Travis CI. > > Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> > --- > ci/install-dependencies.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh > index 7f546c8552..8cc72503cb 100755 > --- a/ci/install-dependencies.sh > +++ b/ci/install-dependencies.sh > @@ -34,7 +34,7 @@ linux-clang|linux-gcc) > popd > ;; > osx-clang|osx-gcc) > - export HOMEBREW_NO_AUTO_UPDATE=1 > + export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1 > # Uncomment this if you want to run perf tests: > # brew install gnu-time > test -z "$BREW_INSTALL_PACKAGES" || > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/2] ci: disable Homebrew's auto cleanup 2019-07-03 12:26 ` Thomas Braun @ 2019-07-03 13:04 ` SZEDER Gábor 2019-07-03 16:58 ` Junio C Hamano 0 siblings, 1 reply; 18+ messages in thread From: SZEDER Gábor @ 2019-07-03 13:04 UTC (permalink / raw) To: Thomas Braun Cc: Junio C Hamano, Johannes Schindelin, brian m. carlson, Taylor Blau, git On Wed, Jul 03, 2019 at 02:26:21PM +0200, Thomas Braun wrote: > Am 03.07.2019 um 12:47 schrieb SZEDER Gábor: > > Lately Homebrew learned to automagically clean up information about > > outdated packages during other 'brew' commands, which might be useful > > for the avarage user, but is a waste of time in CI build jobs, because > > the next build jobs will start from the exact same image containing > > the same outdated packages anyway. > > > > Export HOMEBREW_NO_AUTO_UPDATE=1 to disable this auto cleanup feature, > > The patch below adds HOMEBREW_NO_INSTALL_CLEANUP so maybe you meant > HOMEBREW_NO_INSTALL_CLEANUP here as well? Thanks, good catch. Apparently copy-pasted the wrong env var from the patch to the commit message... > > shaving off about 20-30s from the time needed to install dependencies > > in our macOS build jobs on Travis CI. > > > > Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> > > --- > > ci/install-dependencies.sh | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh > > index 7f546c8552..8cc72503cb 100755 > > --- a/ci/install-dependencies.sh > > +++ b/ci/install-dependencies.sh > > @@ -34,7 +34,7 @@ linux-clang|linux-gcc) > > popd > > ;; > > osx-clang|osx-gcc) > > - export HOMEBREW_NO_AUTO_UPDATE=1 > > + export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1 > > # Uncomment this if you want to run perf tests: > > # brew install gnu-time > > test -z "$BREW_INSTALL_PACKAGES" || > > > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 2/2] ci: disable Homebrew's auto cleanup 2019-07-03 13:04 ` SZEDER Gábor @ 2019-07-03 16:58 ` Junio C Hamano 0 siblings, 0 replies; 18+ messages in thread From: Junio C Hamano @ 2019-07-03 16:58 UTC (permalink / raw) To: SZEDER Gábor Cc: Thomas Braun, Johannes Schindelin, brian m. carlson, Taylor Blau, git SZEDER Gábor <szeder.dev@gmail.com> writes: > On Wed, Jul 03, 2019 at 02:26:21PM +0200, Thomas Braun wrote: >> Am 03.07.2019 um 12:47 schrieb SZEDER Gábor: >> > Lately Homebrew learned to automagically clean up information about >> > outdated packages during other 'brew' commands, which might be useful >> > for the avarage user, but is a waste of time in CI build jobs, because >> > the next build jobs will start from the exact same image containing >> > the same outdated packages anyway. >> > >> > Export HOMEBREW_NO_AUTO_UPDATE=1 to disable this auto cleanup feature, >> >> The patch below adds HOMEBREW_NO_INSTALL_CLEANUP so maybe you meant >> HOMEBREW_NO_INSTALL_CLEANUP here as well? > > Thanks, good catch. Apparently copy-pasted the wrong env var from the > patch to the commit message... Applied an obvious rewording manually while queuing. Thanks. commit af8ed04778781f2013d26cc450173190fb5876d6 Author: SZEDER Gábor <szeder.dev@gmail.com> Date: Wed Jul 3 12:47:48 2019 +0200 ci: disable Homebrew's auto cleanup Lately Homebrew learned to automagically clean up information about outdated packages during other 'brew' commands, which might be useful for the avarage user, but is a waste of time in CI build jobs, because the next build jobs will start from the exact same image containing the same outdated packages anyway. Export HOMEBREW_NO_INSTALL_CLEANUP=1 to disable this auto cleanup feature, shaving off about 20-30s from the time needed to install dependencies in our macOS build jobs on Travis CI. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH] ci/lib.sh: update a comment about installed P4 and Git-LFS versions 2019-07-03 10:47 ` [PATCH 1/2] ci: don't update Homebrew SZEDER Gábor 2019-07-03 10:47 ` [PATCH 2/2] ci: disable Homebrew's auto cleanup SZEDER Gábor @ 2019-07-06 16:16 ` SZEDER Gábor 2019-07-06 16:21 ` [PATCH v1.1] " SZEDER Gábor 2019-07-09 16:32 ` [PATCH 1/2] ci: don't update Homebrew Taylor Blau 2 siblings, 1 reply; 18+ messages in thread From: SZEDER Gábor @ 2019-07-06 16:16 UTC (permalink / raw) To: Junio C Hamano Cc: Johannes Schindelin, brian m. carlson, Taylor Blau, git, SZEDER Gábor A comment 'ci/lib.sh' claims that the "OS X build installs the latest available versions" of P4 and Git-LFS, but since 02373e56bd (ci: don't update Homebrew, 2019-07-03) that's no longer the case, as it will install the versions which were recorded in the image's Homebrew database when the image was created. Update this comment accordingly. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> --- ci/lib.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 288a5b3884..0c7171a173 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -163,8 +163,10 @@ linux-clang|linux-gcc) export GIT_TEST_HTTPD=YesPlease # The Linux build installs the defined dependency versions below. - # The OS X build installs the latest available versions. Keep that - # in mind when you encounter a broken OS X build! + # The OS X build installs much more recent versions, whichever + # were recorded in the Homebrew database upon creating the OS X + # image. + # Keep that in mind when you encounter a broken OS X build! export LINUX_P4_VERSION="16.2" export LINUX_GIT_LFS_VERSION="1.5.2" -- 2.22.0.667.g5c3548c1fe ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v1.1] ci/lib.sh: update a comment about installed P4 and Git-LFS versions 2019-07-06 16:16 ` [PATCH] ci/lib.sh: update a comment about installed P4 and Git-LFS versions SZEDER Gábor @ 2019-07-06 16:21 ` SZEDER Gábor 2019-07-08 10:00 ` Johannes Schindelin 0 siblings, 1 reply; 18+ messages in thread From: SZEDER Gábor @ 2019-07-06 16:21 UTC (permalink / raw) To: Junio C Hamano Cc: Johannes Schindelin, brian m. carlson, Taylor Blau, git, SZEDER Gábor A comment in 'ci/lib.sh' claims that the "OS X build installs the latest available versions" of P4 and Git-LFS, but since 02373e56bd (ci: don't update Homebrew, 2019-07-03) that's no longer the case, as it will install the versions which were recorded in the image's Homebrew database when the image was created. Update this comment accordingly. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> --- Sigh... Same patch as before, but corrected a grammar error in the commit message. ci/lib.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 288a5b3884..0c7171a173 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -163,8 +163,10 @@ linux-clang|linux-gcc) export GIT_TEST_HTTPD=YesPlease # The Linux build installs the defined dependency versions below. - # The OS X build installs the latest available versions. Keep that - # in mind when you encounter a broken OS X build! + # The OS X build installs much more recent versions, whichever + # were recorded in the Homebrew database upon creating the OS X + # image. + # Keep that in mind when you encounter a broken OS X build! export LINUX_P4_VERSION="16.2" export LINUX_GIT_LFS_VERSION="1.5.2" -- 2.22.0.667.g5c3548c1fe ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH v1.1] ci/lib.sh: update a comment about installed P4 and Git-LFS versions 2019-07-06 16:21 ` [PATCH v1.1] " SZEDER Gábor @ 2019-07-08 10:00 ` Johannes Schindelin 0 siblings, 0 replies; 18+ messages in thread From: Johannes Schindelin @ 2019-07-08 10:00 UTC (permalink / raw) To: SZEDER Gábor; +Cc: Junio C Hamano, brian m. carlson, Taylor Blau, git [-- Attachment #1: Type: text/plain, Size: 1418 bytes --] Hi Gábor, On Sat, 6 Jul 2019, SZEDER Gábor wrote: > A comment in 'ci/lib.sh' claims that the "OS X build installs the > latest available versions" of P4 and Git-LFS, but since 02373e56bd > (ci: don't update Homebrew, 2019-07-03) that's no longer the case, as > it will install the versions which were recorded in the image's > Homebrew database when the image was created. > > Update this comment accordingly. > > Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> ACK, Dscho > --- > > Sigh... Same patch as before, but corrected a grammar error in the > commit message. > > ci/lib.sh | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/ci/lib.sh b/ci/lib.sh > index 288a5b3884..0c7171a173 100755 > --- a/ci/lib.sh > +++ b/ci/lib.sh > @@ -163,8 +163,10 @@ linux-clang|linux-gcc) > export GIT_TEST_HTTPD=YesPlease > > # The Linux build installs the defined dependency versions below. > - # The OS X build installs the latest available versions. Keep that > - # in mind when you encounter a broken OS X build! > + # The OS X build installs much more recent versions, whichever > + # were recorded in the Homebrew database upon creating the OS X > + # image. > + # Keep that in mind when you encounter a broken OS X build! > export LINUX_P4_VERSION="16.2" > export LINUX_GIT_LFS_VERSION="1.5.2" > > -- > 2.22.0.667.g5c3548c1fe > > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/2] ci: don't update Homebrew 2019-07-03 10:47 ` [PATCH 1/2] ci: don't update Homebrew SZEDER Gábor 2019-07-03 10:47 ` [PATCH 2/2] ci: disable Homebrew's auto cleanup SZEDER Gábor 2019-07-06 16:16 ` [PATCH] ci/lib.sh: update a comment about installed P4 and Git-LFS versions SZEDER Gábor @ 2019-07-09 16:32 ` Taylor Blau 2019-07-10 22:33 ` Junio C Hamano 2 siblings, 1 reply; 18+ messages in thread From: Taylor Blau @ 2019-07-09 16:32 UTC (permalink / raw) To: SZEDER Gábor Cc: Junio C Hamano, Johannes Schindelin, brian m. carlson, Taylor Blau, git Hi Gábor, On Wed, Jul 03, 2019 at 12:47:47PM +0200, SZEDER Gábor wrote: > Lately our GCC macOS build job on Travis CI has been erroring out > while installing dependencies with: > > +brew link gcc@8 > Error: No such keg: /usr/local/Cellar/gcc@8 > The command "ci/install-dependencies.sh" failed and exited with 1 during . Thanks for working on this. I think that the patch below is ultimately a better approach than what we had discussed in a previous thread [1]. > Now, while gcc@8 is still pre-installed (but not linked) and would be > perfectly usable in the Travis CI macOS image we use [1], it's at > version 8.2. However, when installing dependencies we first > explicitly run 'brew update', which spends over two minutes to update > itself and information about the available packages, and it learns > about GCC 8.3. After that point gcc@8 exclusively refers to v8.3, > and, unfortunately, 'brew' is just too dumb to be able to do anything > with the still installed 8.2 package, and the subsequent 'brew link > gcc@8' fails. (Even 'brew uninstall gcc@8' fails with the same > error!) > > Don't run 'brew update' to keep the already installed GCC 8.2 'brew > link'-able. Note that in addition we have to 'export > HOMEBREW_NO_AUTO_UPDATE=1' first, because 'brew' is so very helpful > that it would implicitly run update for us on the next 'brew install > <pkg>' otherwise. > > Disabling 'brew update' has additional benefits: > > - It shaves off 2-3mins from the ~4mins currently spent on > installing dependencies, and the macOS build jobs have always been > prone to exceeding the time limit on Travis CI. > > - Our builds won't suddenly break because of the occasional Homebrew > breakages [2]. This is sensible. We'll rely on the most-recent version of the Homebrew packages that are known by the macOS image _from Travis_, ignoring any changes that have happened in Homebrew core since the image snapshot. Thanks also for paying attention to setting 'HOMEBREW_NO_AUTO_UPDATE' in the environment. This will harden any other existing or future uses of 'brew' against accidentally spending a bunch of time trying to update itself (and perhaps breaking a subsequent 'brew link', if it should happen before that). > The drawback is that we'll be stuck with slightly older versions of > the packages that we install via Homebrew (Git-LFS 2.5.2 and Perforce > 2018.1; they are currently at 2.7.2 and 2019.1, respectively). We > might want to reconsider this decision as time goes on and/or switch > to a more recent macOS image as they become available. I think that this may in fact be better than what we have now. If we find ourselves wanting a newer version of, say, Git LFS, then I think we'd be benefited by upgrading the image itself. Perhaps we should look into that shortly, although I'm not aware of any urgency to do so (at least as it pertains to Git LFS). > [1] 2000ac9fbf (travis-ci: switch to Xcode 10.1 macOS image, > 2019-01-17) > > [2] See e.g. a1ccaedd62 (travis-ci: make the OSX build jobs' 'brew > update' more quiet, 2019-02-02) or > > https://public-inbox.org/git/20180907032002.23366-1-szeder.dev@gmail.com/T/#+u > > Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> > --- > ci/install-dependencies.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh > index 7f6acdd803..7f546c8552 100755 > --- a/ci/install-dependencies.sh > +++ b/ci/install-dependencies.sh > @@ -34,7 +34,7 @@ linux-clang|linux-gcc) > popd > ;; > osx-clang|osx-gcc) > - brew update >/dev/null > + export HOMEBREW_NO_AUTO_UPDATE=1 > # Uncomment this if you want to run perf tests: > # brew install gnu-time > test -z "$BREW_INSTALL_PACKAGES" || > -- > 2.22.0.621.ge52941b842.dirty Thanks, Taylor [1]: https://public-inbox.org/git/20190602112239.GO951@szeder.dev ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/2] ci: don't update Homebrew 2019-07-09 16:32 ` [PATCH 1/2] ci: don't update Homebrew Taylor Blau @ 2019-07-10 22:33 ` Junio C Hamano 0 siblings, 0 replies; 18+ messages in thread From: Junio C Hamano @ 2019-07-10 22:33 UTC (permalink / raw) To: Taylor Blau; +Cc: SZEDER Gábor, Johannes Schindelin, brian m. carlson, git Taylor Blau <me@ttaylorr.com> writes: > Hi Gábor, > > On Wed, Jul 03, 2019 at 12:47:47PM +0200, SZEDER Gábor wrote: >> Lately our GCC macOS build job on Travis CI has been erroring out >> while installing dependencies with: >> >> +brew link gcc@8 >> Error: No such keg: /usr/local/Cellar/gcc@8 >> The command "ci/install-dependencies.sh" failed and exited with 1 during . > > Thanks for working on this. I think that the patch below is ultimately a > better approach than what we had discussed in a previous thread [1]. > ... > I think that this may in fact be better than what we have now. If we > find ourselves wanting a newer version of, say, Git LFS, then I think > we'd be benefited by upgrading the image itself. Perhaps we should look > into that shortly, although I'm not aware of any urgency to do so (at > least as it pertains to Git LFS). Thanks all. Let's fast-track this fix down to 'maint'. We left the macOS build broken for too long. ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2019-07-10 22:33 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-05-30 19:32 Travis not looking so good Johannes Schindelin 2019-06-01 0:41 ` brian m. carlson 2019-06-02 11:22 ` SZEDER Gábor 2019-06-26 20:35 ` Taylor Blau 2019-06-27 13:23 ` SZEDER Gábor 2019-06-27 16:46 ` Junio C Hamano 2019-06-29 17:01 ` SZEDER Gábor 2019-07-03 10:47 ` [PATCH 1/2] ci: don't update Homebrew SZEDER Gábor 2019-07-03 10:47 ` [PATCH 2/2] ci: disable Homebrew's auto cleanup SZEDER Gábor 2019-07-03 11:49 ` Johannes Schindelin 2019-07-03 12:26 ` Thomas Braun 2019-07-03 13:04 ` SZEDER Gábor 2019-07-03 16:58 ` Junio C Hamano 2019-07-06 16:16 ` [PATCH] ci/lib.sh: update a comment about installed P4 and Git-LFS versions SZEDER Gábor 2019-07-06 16:21 ` [PATCH v1.1] " SZEDER Gábor 2019-07-08 10:00 ` Johannes Schindelin 2019-07-09 16:32 ` [PATCH 1/2] ci: don't update Homebrew Taylor Blau 2019-07-10 22:33 ` Junio C Hamano
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).