git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
* t5559 breaks with apache 2.4.55
@ 2023-01-22  8:00 Jeff King
  2023-01-22 16:33 ` Todd Zullinger
  0 siblings, 1 reply; 4+ messages in thread
From: Jeff King @ 2023-01-22  8:00 UTC (permalink / raw)
  To: git

I noticed that the test suite now fails after upgrading from apache
2.4.54 to 2.4.55 (the latter of which just hit debian unstable). The
problem is with the http2 tests, specifically t5559.30, where we send a
large fetch negotiation over http2. The output from curl is (including
some bits of tracing):

  == Info: Received 101, Switching to HTTP/2
  == Info: Using HTTP2, server supports multiplexing
  == Info: Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  == Info: Closing connection 1
  error: RPC failed; HTTP 101 curl 16 Error in the HTTP2 framing layer

Bisecting within apache's Git repo, the culprit is their 9767274b88,
which says:

  mod_http2: version 2.0.10 of the module, synchronizing changes
  with the gitgub version. This is a partial rewrite of how connections
  and streams are handled.

which seems like a plausible source. But the diff is 8000 lines. It may
be possible to bisect within the mod_http2 source itself, but I haven't
tried it yet.

It's also not 100% clear that it's an apache bug. We could be doing
something weird with git-http-backend, or curl might be doing something
wrong. Though I tend to doubt it, given the simplicity of the CGI
interface on the server side and the fact that curl was working reliably
with older versions of apache.

So I haven't reported the bug further yet. But I thought I'd post this
here before anybody else wastes time digging in the same hole.

-Peff

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: t5559 breaks with apache 2.4.55
  2023-01-22  8:00 t5559 breaks with apache 2.4.55 Jeff King
@ 2023-01-22 16:33 ` Todd Zullinger
  2023-01-26 11:39   ` Jeff King
  0 siblings, 1 reply; 4+ messages in thread
From: Todd Zullinger @ 2023-01-22 16:33 UTC (permalink / raw)
  To: Jeff King; +Cc: git

Jeff King wrote:
> I noticed that the test suite now fails after upgrading from apache
> 2.4.54 to 2.4.55 (the latter of which just hit debian unstable). The
> problem is with the http2 tests, specifically t5559.30, where we send a
> large fetch negotiation over http2. The output from curl is (including
> some bits of tracing):
> 
>   == Info: Received 101, Switching to HTTP/2
>   == Info: Using HTTP2, server supports multiplexing
>   == Info: Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
>   == Info: Closing connection 1
>   error: RPC failed; HTTP 101 curl 16 Error in the HTTP2 framing layer
> 
> Bisecting within apache's Git repo, the culprit is their 9767274b88,
> which says:
> 
>   mod_http2: version 2.0.10 of the module, synchronizing changes
>   with the gitgub version. This is a partial rewrite of how connections
>   and streams are handled.
> 
> which seems like a plausible source. But the diff is 8000 lines. It may
> be possible to bisect within the mod_http2 source itself, but I haven't
> tried it yet.
> 
> It's also not 100% clear that it's an apache bug. We could be doing
> something weird with git-http-backend, or curl might be doing something
> wrong. Though I tend to doubt it, given the simplicity of the CGI
> interface on the server side and the fact that curl was working reliably
> with older versions of apache.
> 
> So I haven't reported the bug further yet. But I thought I'd post this
> here before anybody else wastes time digging in the same hole.

FWIW, I think this is the same issue we discussed about 2
months back, in <Y4fUntdlc1mqwad5@pobox.com>¹.

I haven't done much else with it since then.  It's almost
surely either an apache httpd/mod_http2 or curl issue.  If I
had to bet, I'd say mod_http2.  (But then, it could be curl
and just has yet to be exposed widely because not many are
using the current mod_http2 code.)

¹ https://lore.kernel.org/git/Y4fUntdlc1mqwad5@pobox.com/

-- 
Todd

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: t5559 breaks with apache 2.4.55
  2023-01-22 16:33 ` Todd Zullinger
@ 2023-01-26 11:39   ` Jeff King
  2023-01-29  6:35     ` Jeff King
  0 siblings, 1 reply; 4+ messages in thread
From: Jeff King @ 2023-01-26 11:39 UTC (permalink / raw)
  To: Todd Zullinger; +Cc: git

On Sun, Jan 22, 2023 at 11:33:07AM -0500, Todd Zullinger wrote:

> > So I haven't reported the bug further yet. But I thought I'd post this
> > here before anybody else wastes time digging in the same hole.
> 
> FWIW, I think this is the same issue we discussed about 2
> months back, in <Y4fUntdlc1mqwad5@pobox.com>¹.
> 
> I haven't done much else with it since then.  It's almost
> surely either an apache httpd/mod_http2 or curl issue.  If I
> had to bet, I'd say mod_http2.  (But then, it could be curl
> and just has yet to be exposed widely because not many are
> using the current mod_http2 code.)
> 
> ¹ https://lore.kernel.org/git/Y4fUntdlc1mqwad5@pobox.com/

Ah, I somehow completely forgot about that issue. Despite being one of
the two participants on the thread.

Yeah, after seeing that, I'm quite sure this is a mod_http2 issue. It
would be nice to bisect within the mod_http2 history to find the
culprit, but I'd first have to figure out how to build standalone apache
modules. ;)

I may try to poke at it later if I have time. It might also be worth
submitting a bug report to the mod_http2 folks. I'd hope to have a more
compact reproduction, but it does at least seem to fail reliably for me
(not even racily).

-Peff

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: t5559 breaks with apache 2.4.55
  2023-01-26 11:39   ` Jeff King
@ 2023-01-29  6:35     ` Jeff King
  0 siblings, 0 replies; 4+ messages in thread
From: Jeff King @ 2023-01-29  6:35 UTC (permalink / raw)
  To: Todd Zullinger; +Cc: git

On Thu, Jan 26, 2023 at 06:39:42AM -0500, Jeff King wrote:

> Yeah, after seeing that, I'm quite sure this is a mod_http2 issue. It
> would be nice to bisect within the mod_http2 history to find the
> culprit, but I'd first have to figure out how to build standalone apache
> modules. ;)

This turned out to be quite painless, so I bisected using the source at:

  https://github.com/icing/mod_h2

Unfortunately, it's not super helpful for identifying the problem.  The
culprit turns out to be 16ffed9692b, which has a 450-line diff. The
commit message is:

  * refactored stream response handling to reflect the different phases
    (response/data/trailers) more clearly and help resolving cpu busy loops.

But that does at least give me more confidence that the bug is in
mod_http2, and isn't, say, some intentional behavior change there that
happens to trigger a bug in curl.

I opened an issue here: https://github.com/icing/mod_h2/issues/243

So we'll see if that helps.

I re-read your earlier thread, and the version problems you have don't
quite line up. I think you were having issues with mod_http2 2.0.9, but
the older version (1.5.19) worked. I was OK with 2.0.9, but upgrading to
2.0.10 broke things. I'd hate to have to disable t5559 by default; it
does catch some cases that would trigger in the real-world (since real
sites like GitHub are using http2). But if it's too unreliable in
practice, that might be the path of least resistance.

-Peff

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2023-01-29  6:35 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-22  8:00 t5559 breaks with apache 2.4.55 Jeff King
2023-01-22 16:33 ` Todd Zullinger
2023-01-26 11:39   ` Jeff King
2023-01-29  6:35     ` Jeff King

Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).