From: Jeff King <peff@peff.net>
To: "SZEDER Gábor" <szeder.dev@gmail.com>
Cc: Derrick Stolee <stolee@gmail.com>,
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>,
git@vger.kernel.org, Junio C Hamano <gitster@pobox.com>,
Derrick Stolee <dstolee@microsoft.com>
Subject: Re: [PATCH 1/1] upload-pack: fix race condition in error messages
Date: Wed, 4 Sep 2019 01:04:42 -0400 [thread overview]
Message-ID: <20190904050441.GB6488@sigill.intra.peff.net> (raw)
In-Reply-To: <20190830121005.GI8571@szeder.dev>
On Fri, Aug 30, 2019 at 02:10:05PM +0200, SZEDER Gábor wrote:
> On Fri, Aug 30, 2019 at 12:06:30AM +0200, SZEDER Gábor wrote:
> > On Thu, Aug 29, 2019 at 11:58:18PM +0200, SZEDER Gábor wrote:
> > > On Thu, Aug 29, 2019 at 10:38:05AM -0400, Jeff King wrote:
> > > > So any fixes there have to happen on the client side. I am still
> > > > confused about why the client is writing in this case, per the argument
> > > > in 014ade7484 (upload-pack: send ERR packet for non-tip objects,
> > > > 2019-04-13). It would be nice to use GIT_TRACE_PACKET to see what it's
> > > > trying to write, but I still haven't been able to reproduce the issue.
> > >
> > > It's the "done" line:
> > >
> > > + cat trace-packet
> [...]
> > > packet: upload-pack> 0000
> > > packet: fetch> done
> > >
> > > In the avarage successful run that "fetch> done" pkt-line is
> > > immediately after the "fetch> 0000".
Thanks for all of your persistent digging on this. I had forgotten about
the "done" packet, but it explains all of the symptoms we've seen.
> So instead of immediately die()int after write_in_full() returned an
> error, fetch should first try to read all incoming packets in the hope
> that the remote did send an ERR packet before it died, and then die
> with the error in that packet, or fall back to the current generic
> error message if there is no ERR packet (e.g. remote segfaulted or
> something similarly horrible). This fixes the test failure with that
> strategically-placed sleep() in 'fetch-pack.c'.
>
> https://travis-ci.org/szeder/git/jobs/578778749#L2689
>
> Alas, passing a 'reader' to a function called send_request() doesn't
> look quite right, does it... And I'm not sure about the stateless
> communication, it still uses write_or_die().
And thank you for putting this patch together. I had taken a stab at it
a while ago, but got discouraged by figuring out at which layer to add
the "reader" info (I had envisioned it much lower in packet_write(), but
it is clear from your patch that fetch-pack does most of its own
writing).
I agree passing around the reader is a bit weird; I wonder if we should
be representing the full-duplex connection more clearly as a single
struct. But I suspect that creates other headaches, and what you have
here doesn't look _too_ bad. As you note, it probably doesn't cover all
code paths, but it at least fixes some of them, and gives us a template
for addressing the others.
> } else {
> - if (write_in_full(fd, buf->buf, buf->len) < 0)
> + if (write_in_full(fd, buf->buf, buf->len) < 0) {
> + int save_errno = errno;
> + /*
> + * Read everything the remote has sent to us.
> + * If there is an ERR packet, then the loop die()s
> + * with the received error message.
> + * If we reach EOF without seeing an ERR, then die()
> + * with a generic error message, most likely "Broken
> + * pipe".
> + */
> + while (packet_reader_read(reader) != PACKET_READ_EOF);
> + errno = save_errno;
> die_errno(_("unable to write to remote"));
> + }
One unfortunate thing here is that we could block indefinitely in
packet_reader_read(). That shouldn't happen, I don't think, but since
this is an error case where we've been cutoff, anything's possible.
We maybe could get away with using non-blocking I/O. We're looking for
an ERR packet the other side sent us _before_ it hung up, so in theory
we've have received the data before any FIN packet (or EOF on a pipe).
But I'm wary of introducing new races there.
It might be enough to put in an actual timer, waiting for an ERR packet,
EOF, or something like 5 seconds. Or maybe I'm just being overly
paranoid.
-Peff
next prev parent reply other threads:[~2019-09-04 5:04 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-28 1:43 [PATCH 0/1] upload-pack: fix race condition in t5516 Derrick Stolee via GitGitGadget
2019-08-28 1:43 ` [PATCH 1/1] upload-pack: fix race condition in error messages Derrick Stolee via GitGitGadget
2019-08-28 9:34 ` SZEDER Gábor
2019-08-28 14:54 ` Jeff King
2019-08-28 15:39 ` Jeff King
2019-08-28 16:15 ` SZEDER Gábor
2019-08-29 12:58 ` Derrick Stolee
2019-08-29 14:13 ` Jeff King
2019-08-29 14:27 ` Derrick Stolee
2019-08-29 14:38 ` Jeff King
2019-08-29 21:58 ` SZEDER Gábor
2019-08-29 22:06 ` SZEDER Gábor
2019-08-30 12:10 ` SZEDER Gábor
2019-09-04 5:04 ` Jeff King [this message]
2019-09-10 12:08 ` SZEDER Gábor
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190904050441.GB6488@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=dstolee@microsoft.com \
--cc=git@vger.kernel.org \
--cc=gitgitgadget@gmail.com \
--cc=gitster@pobox.com \
--cc=stolee@gmail.com \
--cc=szeder.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).