git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Johannes Schindelin <Johannes.Schindelin@gmx.de>
To: Junio C Hamano <gitster@pobox.com>
Cc: Lars Schneider <larsxschneider@gmail.com>,
	git@vger.kernel.org, peff@peff.net, bmwill@google.com
Subject: Re: What's cooking in git.git (Apr 2017, #04; Wed, 19)
Date: Mon, 24 Apr 2017 16:19:48 +0200 (CEST)	[thread overview]
Message-ID: <alpine.DEB.2.20.1704241609300.3480@virtualbox> (raw)
In-Reply-To: <xmqqfugy4pnx.fsf@gitster.mtv.corp.google.com>

Hi Junio,

On Sun, 23 Apr 2017, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > Part of the reason is that you push out all of the branches in one go,
> > typically at the very end of your work day. The idea of Continuous
> > Integration is a little orthogonal to that style, suggesting to build
> > & test whenever new changes come into the integration branch.
> >
> > As a consequence, my original setup was a little overloaded: the VM
> > sat idle most of the time, and when you pushed, it was overloaded.
> 
> I do not see pushing out all them in one go is making the problem worse
> for you, though.

Oh no, you don't see that? Then let me spell it out a little more
clearly: when you push out four branches at the same time, the same
Virtual Machine that hosts all of the build agents has to build each and
everyone of them, then run the entire test suite.

As I have pointed out at several occasions (but I was probably complaining
too much about it, so you probably ignored it), the test suite uses shell
scripting a lot, and as a consequence it is really, really slow on
Windows. Meaning that even on a high-end VM, it typically takes 1.5 hours
to run the test suite. That's without SVN tests.

So now we have up to four build agents banging at the same CPU and RAM,
competing for resources. Now it takes more like 2-3 hours to run the
entire build & test.

The situation usually gets a little worse, even: you sometimes push out
several iterations of `pu` in relatively rapid succession, "rapid" being
relative to the time taken by the builds.

That means that there are sometimes four jobs still hogging the VM when
the next request to build & test `pu` arrives, and sometimes there is
another one queued before the first job finishes.

Naturally, the last two jobs will have started barely before Travis
decides that it waited long enough (3 hours) to call it quits.

To answer your implied question: the situation would be much, much better
if the branches with more time in-between.

But as I said, I understand that it would be asking you way too much to
change your process that seems to work well for you.

> As of this writing, master..pu counts 60+ first-parent merges.
> Instead of pushing out the final one at the end of the day, I could
> push out after every merge.  Behind the scenes, because some topics
> are extended or tweaked while I read the list discussion, the number
> of merges I am doing during a day is about twice or more than that
> before I reach the final version for the day.  
> 
> Many issues can be noticed locally even before the patches hit a
> topic, before the topic gets merged to 'pu', or before the tentative
> 'pu' is pushed out, and breakage at each of these points can be
> locally corrected without bothering external test setups.  I've been
> assuming that pushing out all in one go at the end will help
> reducing the load at external test setups.

Pushing out only four updates at the end of the day is probably better
than pushing after every merge, for sure.

Ciao,
Dscho

  reply	other threads:[~2017-04-24 14:20 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-04-20  5:37 What's cooking in git.git (Apr 2017, #04; Wed, 19) Junio C Hamano
2017-04-20  9:59 ` Duy Nguyen
2017-04-20 15:35   ` Jeff King
2017-04-20 22:51     ` Junio C Hamano
2017-04-20 22:46   ` Junio C Hamano
     [not found] ` <D61D47BD-9750-4FB6-892E-013504E03738@gmail.com>
2017-04-20 13:24   ` Johannes Schindelin
2017-04-20 16:56     ` Brandon Williams
2017-04-20 23:18       ` Brandon Williams
2017-04-21  0:56         ` Junio C Hamano
2017-04-20 22:58   ` Junio C Hamano
2017-04-21  9:50     ` Johannes Schindelin
2017-04-21 12:29       ` Christian Couder
2017-04-22 11:48         ` Johannes Schindelin
2017-04-22 17:32           ` Christian Couder
2017-04-24 14:08             ` Johannes Schindelin
2017-04-22 13:37         ` Johannes Sixt
2017-04-24 14:24           ` Johannes Schindelin
2017-04-24 16:34             ` Philip Oakley
2017-04-25  2:17               ` Christian Couder
2017-04-25  2:00           ` Christian Couder
2017-04-25  5:51             ` Johannes Sixt
2017-04-25  6:52               ` Junio C Hamano
2017-04-25 18:26                 ` Johannes Sixt
2017-04-24  0:25       ` Junio C Hamano
2017-04-24 14:19         ` Johannes Schindelin [this message]
2017-04-24 15:18           ` Ævar Arnfjörð Bjarmason
2017-04-25  0:56             ` Junio C Hamano
2017-04-20 15:32 ` Lars Schneider
2017-04-20 22:52   ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.DEB.2.20.1704241609300.3480@virtualbox \
    --to=johannes.schindelin@gmx.de \
    --cc=bmwill@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=larsxschneider@gmail.com \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).