Re: [RFC PATCH 4/4] t/lib-httpd: sort log based on timestamp to avoid occasional failure

[Jeff King <peff@peff.net>]

On Thu, Jun 14, 2018 at 11:27:03AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> >>     Another alternative is to simply accept that these tests are racy, and
> >>     that the resulting test failures are rare enough that it isn't worth
> >>     the complexity of the workaround, but adding a comment to the affected
> >>     tests warning about the raciness is sufficient.  (But I wrote this
> >>     when I first saw and tracked down this failure; since then I observed
> >>     it four more times... :)
> >
> > It's definitely bugged me. I'd be happy to see some solution. I've been
> > close to suggesting that reading apache logs is simply not robust, and
> > we should focus our tests on the git-visible state changes (e.g., seeing
> > successful requests, updated refs, etc).
> 
> Hmph, that certainly is "checking only the things that matter",
> which is desirable.

The existing tests that look at the apache logs are very white-box, and
come from Shawn's original smart-http series. I suspect it was as much
about sanity-checking the implementation then as it was about protecting
against regressions.

So it's possible that these tests have simply outlived their usefulness.

OTOH, they might catch non-functional problems, like if we started
sending redundant requests. But then, they are not very comprehensive,
as they only cover a few basic situations (so for example, for a while
we were sending extra auth-related requests, but I don't think these
tests detected that).

-Peff