git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Jeff King <peff@peff.net>
To: git@vger.kernel.org
Subject: [PATCH 0/12] push progress reporting and keepalives
Date: Fri, 15 Jul 2016 06:25:06 -0400	[thread overview]
Message-ID: <20160715102506.GA23164@sigill.intra.peff.net> (raw)

If you push a large number of objects, the server side may have to chew
CPU for a long time processing the input (delta resolution, connectivity
check, and whatever any hooks choose to do). During this time, you get
no feedback that anything is happening, unless the hooks feel like
writing something to stderr.  For a repository the size of linux.git, a
full push from scratch can take several minutes.

This is annoying and confusing to the user, who wonders if the
connection has hung. But it can also cause problems on systems that have
other timeouts (e.g., firewalls dropping TCP sessions, or web proxies
dropping requests that produce no response within a certain time).

This patch series adds two new features:

 1. Progress reporting for the CPU-intensive parts of receive-pack.

 2. A keepalive mechanism similar to what we use in upload-pack
    (basically sending zero-length packets on sideband 1 while the client
    is waiting for us to speak).

Both are enabled for any client which speaks the sideband protocol.
Existing versions of git handle the new behavior just fine (the progress
reporting is easy, because they were expecting stderr messages anyway;
the keepalive works because the demuxer just relays zero bytes back to
send-pack).

I also tested with both JGit and libgit2 clients, and both seem to
handle the zero-length packets just fine.

There are unfortunately no automated tests, as it's hard to simulate the
effect. My manual testing involved inserting "sleep" statements into
index-pack (and hooks with manual sleeps), and then using "strace" to
confirm that the keepalives were sent.

  [01/12]: check_everything_connected: always pass --quiet to rev-list
  [02/12]: rev-list: add optional progress reporting
  [03/12]: check_everything_connected: convert to argv_array
  [04/12]: check_everything_connected: use a struct with named options
  [05/12]: check_connected: relay errors to alternate descriptor
  [06/12]: check_connected: add progress flag
  [07/12]: clone: use a real progress meter for connectivity check
  [08/12]: index-pack: add flag for showing delta-resolution progress
  [09/12]: receive-pack: turn on index-pack resolving progress
  [10/12]: receive-pack: relay connectivity errors to sideband
  [11/12]: receive-pack: turn on connectivity progress
  [12/12]: receive-pack: send keepalives during quiet periods

-Peff

             reply	other threads:[~2016-07-15 10:25 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-07-15 10:25 Jeff King [this message]
2016-07-15 10:26 ` [PATCH 01/12] check_everything_connected: always pass --quiet to rev-list Jeff King
2016-07-15 10:28 ` [PATCH 02/12] rev-list: add optional progress reporting Jeff King
2016-07-15 18:00   ` Junio C Hamano
2016-07-16  1:23     ` Jeff King
2016-07-15 10:28 ` [PATCH 03/12] check_everything_connected: convert to argv_array Jeff King
2016-07-15 10:30 ` [PATCH 04/12] check_everything_connected: use a struct with named options Jeff King
2016-07-15 18:13   ` Junio C Hamano
2016-07-16  1:24     ` Jeff King
2016-07-15 10:32 ` [PATCH 05/12] check_connected: relay errors to alternate descriptor Jeff King
2016-07-15 18:19   ` Junio C Hamano
2016-07-16  1:27     ` Jeff King
2016-07-15 10:32 ` [PATCH 06/12] check_connected: add progress flag Jeff King
2016-07-15 10:33 ` [PATCH 07/12] clone: use a real progress meter for connectivity check Jeff King
2016-07-15 10:34 ` [PATCH 08/12] index-pack: add flag for showing delta-resolution progress Jeff King
2016-07-15 10:35 ` [PATCH 09/12] receive-pack: turn on index-pack resolving progress Jeff King
2016-07-15 10:36 ` [PATCH 10/12] receive-pack: relay connectivity errors to sideband Jeff King
2016-07-15 10:36 ` [PATCH 11/12] receive-pack: turn on connectivity progress Jeff King
2016-07-15 10:43 ` [PATCH 12/12] receive-pack: send keepalives during quiet periods Jeff King
2016-07-15 17:24   ` Stefan Beller
2016-07-16  7:56     ` Jeff King
2016-07-19  5:28       ` Stefan Beller
2016-07-19 10:07         ` Jeff King
2016-07-19 16:05           ` Stefan Beller
2016-07-20 13:28             ` Jeff King
2016-07-15 19:18   ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160715102506.GA23164@sigill.intra.peff.net \
    --to=peff@peff.net \
    --cc=git@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).