about summary refs log tree commit homepage
path: root/t/httpd-corner.t
DateCommit message (Collapse)
2023-12-16tests: quiet uninitialized warnings on CentOS 7.x
Test::More distributed with Perl 5.16.3 on CentOS 7.x expects the `$how_many' argument for `skip' and warns when its uninitialized, so quiet that warning down.
2023-12-13tests: attempt compatibility w/ busybox lsof
BusyBox lsof(1) ignores the `-p PID' argument and shows the open files for every process it knows about. BusyBox lsof also lacks the `NODE' column of the non-BusyBox implementation, so we'll rely on /proc/PID/fd/ in those cases since the deleted file checks are Linux-only and it's common to have procfs is mounted on /proc on Linux.
2023-11-03treewide: use ->close to call ProcessIO->CLOSE
This will open the door for us to drop `tie' usage from ProcessIO completely in favor of OO method dispatch. While OO method dispatches (e.g. `$fh->close') are slower than normal subroutine calls, it hardly matters in this case since process teardown is a fairly rare operation and we continue to use `close($fh)' for Maildir writes.
2023-10-25drop psgi_return, httpd/async and GetlineBody
Now that psgi_yield is used everywhere, the more complex psgi_return and it's helper bits can be removed. We'll also fix some outdated comments now that everything on psgi_return has switched to psgi_yield. GetlineResponse replaces GetlineBody and does a better job of isolating generic PSGI-only code.
2023-10-03daemon: enable SO_ACCEPTFILTER on NetBSD
NetBSD 5.0+ has accept filter support from FreeBSD; and I I think we can assume all NetBSD is 5.0+ (released in 2009) nowadays if we're already depending on Perl 5.12 from 2010.
2023-01-30use Net::SSLeay (OpenSSL) for SHA-(1|256) if installed
On my x86-64 machine, OpenSSL SHA-256 is nearly twice as fast as the Digest::SHA implementation from Perl, most likely due to an optimized assembly implementation. SHA-1 is a few percent faster, too.
2022-12-12tests: replace select/usleep calls with tick()
This makes it easier to identify places in tests which cause unnecessary slowdowns doing busy waits.
2022-09-29treewide: use --globoff with curl(1)
curl 7.29.0 (on CentOS 7.x) seems to mishandle square-bracketed IPv6 addresses, at least. Furthermore, we don't actually need nor use the globbing in curl for lei when forwarding requests from the lei command-line. lei has its own globbing and `--globoff' behavior for externals and none of it is intended for curl.
2022-08-02daemon: support per-listener env, .psgi, out, err
This allows memory savings by allowing multiple, completely unrelated-PSGI apps to run within the same process as IMAP, NNTP, and POP3.
2021-10-20httpd: reject requests with spaces in header names
Malicious clients may attempt HTTP request smuggling this way. This doesn't affect our current code as we only look for exact matches, but it could affect other servers behind a to-be-implemented reverse proxy built around our -httpd. This doesn't affect users behind varnish at all, nor the HTTPS/HTTP reverse proxy I use (I don't know about nginx), but could be passed through by other reverse proxies. This change is only needed for HTTP::Parser::XS which most users probably use. Users of the pure Perl parser (via PLACK_HTTP_PARSER_PP=1) already hit 400 errors in this case, so this makes the common XS case consistent with the pure Perl case. cf. https://www.mozilla.org/en-US/security/advisories/mfsa2006-33/
2021-10-01daemon: make SO_ACCEPTFILTER a shared variable
Constant subroutines use more memory and there's no need to optimize it for inlining since it's only used at startup.
2021-09-13tests: add require_cmd, require curl when needed
t/v2mirror.t and t/lei-mirror.t are now skipped when curl is missing (instead of failing in appropriate places). A bunch of which() checks are updated to use require_cmd to avoid explicitly loading Spawn.
2021-03-17tests: show lsof output on deleted-file-check failures
This may help track down some occasional test failures I'm seeing.
2021-02-08tests: favor IPv6
IPv4 gets plenty of real-world coverage, and apparently there's Debian buildd hosts which lack IPv4(*). So ensure everything can work on IPv6 and not cause problems for odd setups. (*) https://bugs.debian.org/979432
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2020-07-06qspawn: learn to gzip streaming responses
This will allow us to gzip responses generated by cgit and any other CGI programs or long-lived streaming responses we may spawn.
2020-05-03t/httpd-corner.t: fix uninitialized warning
Current versions of Perl don't warn when vec() is given `undef' as its first arg, but Perl 5.10.1 does, at least. Fixes: c7b4cbdadf3116a0 ("t/httpd-corner: improve reliability and diagnostics")
2020-04-20testcommon: spawn-aware system() and qx[] workalikes
Barely noticeable on Linux, but this gives a 1-2% speedup on a FreeBSD 11.3 VM and lets us use built-in redirects rather than relying on /bin/sh.
2020-04-16t/httpd-corner: improve reliability and diagnostics
The graceful-shutdown-on-PUT test is unreliable because we can't rely on a FIFO as we do with the GET tests. So increase the delay to 100ms since that seems enough on my system even with CONFIG_HZ=100. Add a timeout and backtrace to the $check_self sub to help with further diagnostics while we're at it, too. It would be nice if there were a portable syscall tracing mechanism we could attach to the -httpd process to make the test more determistic...
2020-04-15t/httpd-corner.t: relax read-after-failed-write handling
I've observed FreeBSD 11.2 read(2) having one of three behaviors after a failed write(2) on a socket: 1) returning number of bytes read 2) failing with ECONNRESET 3) returning with EOF 1) is the most common, and I've only seen 1) on Linux. It may be possible to use SO_LINGER or shutdown(2) to ensure 1) always happens, but SO_LINGER behavior seems inconsistent across OSes, especially with non-blocking sockets. Since these tests are corner-cases where we're dealing with broken/malicious clients, lets continue spending the least amount of syscalls protecting ourselves in the daemon and instead make the client-side test code tolerate more socket implementations.
2020-04-09triewyde: ficks soem speling errrors
Dikshunarees R gude!
2020-03-25qspawn: handle ENOENT (and other errors on exec)
As sqlite3(1) and other executables may become unavailable or uninstalled while a daemon runs, we need to gracefully handle errors in those cases.
2020-03-25qspawn: reinstate filter support, add gzip filter
We'll be supporting gzipped from sqlite3(1) dumps for altid files in future commits. In the future (and if we survive), we may replace Plack::Middleware::Deflater with our own GzipFilter to work better with asynchronous responses without relying on memory-intensive anonymous subs.
2020-03-19http: fix RFC conformance w.r.t. message length
We need to favor "Transfer-Encoding: chunked" over the value of the Content-Length header. We should also reject bogus, duplicate and/or unreasonable values for both these, since they can trigger unexpected behavior when combined with other HTTP parsers in proxies such as varnish, nginx, haproxy, etc... See RFC 7230 (and RFC 2616) for more details: https://tools.ietf.org/html/rfc7230 https://www.rfc-editor.org/errata_search.php?rfc=7230
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2020-01-28avoid relying on IO::Handle/IO::File autoload
Perl 5.14+ gained the ability to autoload IO::File (and IO::Handle) on missing methods, so relying on this breaks under 5.10.1. There's no reason to load IO::File or IO::Handle when built-in perlops work fine and are even a hair faster.
2020-01-28daemon: provide TCP_DEFER_ACCEPT for Perl <5.14
Socket::TCP_DEFER_ACCEPT() did not appear in the Socket module distributed with Perl until 5.14, despite it being available since Linux 2.4.
2019-12-30spawn: allow passing GLOB handles for redirects
We can save callers the trouble of {-hold} and {-dev_null} refs as well as the trouble of calling fileno().
2019-12-24testcommon: add require_mods method and use it
This cuts down on lines of code in individual test cases and fixes some misnamed error messages by using "$0" consistently. This will also provide us with a method of swapping out dependencies which provide equivalent functionality (e.g "Xapian" SWIG can replace "Search::Xapian" XS bindings).
2019-12-19tests: move t/common.perl to PublicInbox::TestCommon
We want to be able to use run_script with *.t files, so t/common.perl putting subs into the top-level "main" namespace won't work. Instead, make it a module which uses Exporter like other libraries.
2019-12-19t/*.t: avoid sharing "my" variables in subs
These usages of file-local global variables make the *.t files incompatible with run_script(). Instead, use anonymous subs, "our", or pass the parameter as appropriate.
2019-11-24tests: use File::Temp->newdir instead of tempdir()
We'll also introduce a tmpdir() API to give tempdirs consistent names.
2019-11-24t/common: start_script replaces spawn_listener
We can shave several hundred milliseconds off tests which spawn daemons by preloading and avoiding startup time for common modules which are already loaded in the parent process. This also gives ENV{TAIL} support to all tests which support daemons which log to stdout/stderr.
2019-11-24t/httpd-corner: wait for worker process death
We need to ensure the worker process is terminated before starting a new connection, so leave a persistent HTTP/1.1 connection open and wait for the SIGKILL to take effect and drop the client.
2019-11-15t/common: move unix_server to t/httpd-corner.t
unix_server() is not commonly used, only t/httpd-corner.t uses it and most HTTP tests use TCP since most HTTP libraries only support TCP.
2019-11-08t/*.t: disable nntpd/httpd worker processes in most tests
And explicitly test for respawning in t/httpd-corner.t There's no need to have an extra entries in the process table for most tests we run, since that's not what we're testing.
2019-11-08t/httpd-corner.t: get rid of IPC::Run for running curl
We already load PublicInbox::Spawn, so there's no need to add another dependency to make life difficult for potential contributors.
2019-11-08t/httpd-corner.t: drop unnecessary bytes:: for length()
We don't need to force byte semantics for a buffer we clearly create (via ->read) with byte semantics. Since we didn't "use bytes" in t/httpd-corner.t, it was inadvertantly made available by IPC::Run (which goes away, next).
2019-11-04t/httpd-corner.t: check for curl(1) errors in big async test
curl(1) can fail and we need to invalidate the test in the rare case it fails.
2019-09-17t/httpd-corner.t: don't fail lsof test if stdin is a pipe (try #2)
Actually do the redirect properly
2019-09-17t/httpd-corner.t: don't fail lsof test if stdin is a pipe
We don't want the stdin from the test runner to accidentally cause this test to fail.
2019-09-15t/httpd-corner: use which() sub for detecting curl(1)
We already import `which' for lsof(8), so we might as well use it to detect curl(1), too.
2019-09-14t/httpd-corner: check for leaking FDs and pipes
-W0 (no workers) should not create any pipes on its own, and we shouldn't have any deleted FDs if no clients are connected. This can find if leaks which may be triggered by PublicInbox::HTTP (and not Qspawn or GitHTTPBackend).
2019-09-09tests: add tcp_connect() helper
IO::Socket::INET->new is rather verbose with the options hash, extract it into a standalone sub
2019-07-04qspawn: retry sysread when parsing headers, too
We need to ensure the BIN_DETECT (8000 byte) check in ViewVCS can be handled properly when sending giant files. Otherwise, EPOLLET won't notify us, again, and responses can get stuck. While we're at it, bump up the read-size up to 4096 bytes so we make fewer trips to the kernel.
2019-06-30tests: common tcp_server and unix_server helpers
IO::Socket:*->new options are verbose and we can save a bunch of code by putting this into t/common.perl, since the related spawn_listener stuff is already there.
2019-06-24daemon: use FreeBSD accept filters on non-NNTP
Similar to TCP_DEFER_ACCEPT on Linux, FreeBSD has a 'dataready' accept filter which we can use to reduce wakeups when doing TLS negotiation or plain HTTP. There's also a 'httpready' which we can use for plain HTTP connections.
2019-06-24daemon: set TCP_DEFER_ACCEPT on everything but NNTP
This Linux-specific option can save us some wakeups during the TLS negotiation phase, and it can help with ordinary HTTP, too. Plain NNTP (and in the future, POP3) are the only things which require the server send messages, first.
2019-06-20t/httpd-corner: ensure chunk payload read doesn't overreach
It never has, AFAIK, but I'm making some changes to this code in another branch and nearly introduced a bug where it would be overreading and discarding the pipelined request.
2019-06-20t/httpd-corner.t: fix braino :x
Plack is for Perl, Rack is for Ruby; this a Perl project :x