about summary refs log tree commit homepage
path: root/lib/PublicInbox/HTTP.pm
DateCommit message (Collapse)
2018-03-27http: fix modification of read-only value
This fails in the rare case we get a partial send() on "\r\n" when writing chunked HTTP responses out.
2018-02-07update copyrights for 2018
Using update-copyrights from gnulib While we're at it, use the SPDX identifier for AGPL-3.0+ to ease mechanical processing.
2017-01-04http: remove weaken usage, reduce anonsub capture scope
Avoiding weaken here is no more dangerous than the existing circular refs (e.g. psgix.io) we create and manage throughout the lifetime of the connection. So, trust ourselves to maintain the data structure properly and avoid triggering extra memory usage. While we're at it, avoid having anonymous subroutines capture more variables than necessary to simplify reference auditing.
2017-01-04http: fix spelling error
Oops. And we'll be fixing circular references from now...
2016-12-25http: fix clobbering of $null_io
Oops, this would be disatrous if we started handling bigger request bodies or slow clients. Fixes: c008654229a9 ("avoid IO::File for anonymous temporary files")
2016-11-26avoid IO::File for anonymous temporary files
We do not need to import IO::File into the main programs since Perl 5.8+ supports literal "undef" for generating anonymous temporary file handles.
2016-08-05http: do not allow bad getline+close responses to kill us
PSGI applications (like our WWW :P) can fail unpredictability, but lets try to avoid bringing the entire process down when this happens.
2016-07-08http: drop extra newline in error message
We already add the extra newline when we call print.
2016-07-07http: additional info for write failures
There was a spurious test failure in t/httpd-corner.t which I have not been able to reproduce.
2016-07-07inbox: cleanup and consolidate object weakening
This fixes some layering violations and consolidates the cleanup into the inbox object itself. Keeping in mind weakening does not work at all without our PSGI server.
2016-06-25http: cork chunked responses for small savings
This only affects Linux users with MSG_MORE support. We can avoid extra TCP overhead for sub-optimal chunk sizes by using MSG_MORE even with chunk trailers under Linux. This breaks real-time apps which require <= 200ms latency for streaming small packets (e.g. implementing "tail -F"), but the public-inbox WWW code does not (and will never) do such things.
2016-06-24http: always yield on getline/body
We want to maximize fairness for large responses which may download the entire mbox.
2016-06-19http: constrain getline/close responses by time
This allows us to yield control to other clients gracefully if getline takes too long to generate a chunk. This is more expensive but should not cost a syscall on modern 64-bit systems.
2016-06-19http: avoid recursion when hitting write count limit
Use the EvCleanup::asap handler to reschedule our writes after yielding to other clients.
2016-05-30http: yield body->getline running time
We cannot let a client monopolize the single-threaded server even if it can drain the socket buffer faster than we can emit data. While we're at it, acknowledge the this behavior (which happens naturally) in httpd/async. The same idea is present in NNTP for the long_response code. This is the HTTP followup to: commit 0d0fde0bff97 ("nntp: introduce long response API for streaming") commit 79d8bfedcdd2 ("nntp: avoid signals for long responses")
2016-05-28http: clarify comments about layering violation
It's a low priority, but acknowledge it.
2016-05-27http: avoid circular reference for getline responses
Lightly tested, this seems to work when mass-aborting responses. Will still need to automate the testing...
2016-05-24http: fix various race conditions
We no longer override Danga::Socket::event_write and instead re-enable reads by queuing up another callback in the $close response callback. This is necessary because event_write may not be completely done writing a response, only the existing buffered data. Furthermore, the {closed} field can almost be set at any time when writing, so we must check it before acting on pipelined requests as well as during write callbacks in more().
2016-05-24standardize timer-related event-loop code
Standardize the code we have in place to avoid creating too many timer objects. We do not need exact timers for things that don't need to be run ASAP, so we can play things fast and loose to avoid wasting power with unnecessary wakeups. We only need two classes of timers: * asap - run this on the next loop tick, after operating on @Danga::Socket::ToClose to close remaining sockets * later - run at some point in the future. It could be as soon as immediately (like "asap"), and as late as 60s into the future. In the future, we support an "emergency" switch to fire "later" timers immediately.
2016-05-23http: avoid uninitialized variable
Oops, really gotta start checking logs in tests :x Fixes: bb38f0fcce739 ("http: chunk in the server, not middleware")
2016-05-23http: chunk in the server, not middleware
Since PSGI does not require Transfer-Encoding: chunked or Content-Length, we cannot expect random apps we host to chunk their responses. Thus, to improve interoperability, chunk at the HTTP layer like other PSGI servers do. I'm chosing a more syscall-intensive method (via multiple send(...MSG_MORE) for now to reduce copy + packet overhead.
2016-05-22http: rework async_pass support
Unfortunately, the original design did not work because middleware can wrap the response body and make `async_pass' invisible to HTTP.pm
2016-05-22http: pass reference to Danga::Socket::write
This can avoid an expensive copy for big strings.
2016-05-22http: fix typo: write_buf => write_buf_size
Otherwise, we get deep recursion as we keep calling recursively on giant responses
2016-05-21http: support async_pass for Danga::Socket
This will allow us to minimize buffering after we wait (possibly a long time) for readability. This also greatly reduces the amount of Danga::Socket-specific knowledge we have in our PSGI code, making it easier for others to understand.
2016-05-21http: reduce over-buffering for getline responses
By switching to a "pull"-based I/O model for reading application responses, we should be able to throttle buffering to slow clients more effectively and avoid wasting precious RAM. This will also allow us to more Danga::Socket-specific knowledge out of the PSGI application and keep it confined to PublicInbox::HTTP.
2016-05-17http: release resources when idle
This lets us release old git processes so unlinked packs (leftover from repacking) can be released. This may also be helpful for Xapian as indices get rebuilt for tuning. For SQLite (msgmap), the there may be no benefit besides reducing FD pressure. Followup changes will unify the Inbox and NewsGroup classes and allow better code-sharing between NNTP and HTTP classes (as well as the planned POP3 class).
2016-05-03http: move empty string check into write callback
This empty string check is for middlewares such as Deflater which may write empty strings, not for direct real callers of Danga::Socket who (presumably) know what they're doing.
2016-05-02http: remove needless binmode call
Unnecessary on *nix, and we won't support systems which do insane things.
2016-05-01daemon: reduce timer-related allocations
We can reduce the allocation and overhead needed for Danga::Socket timers for immediately-executed responses by combining identical timers and reducing anonymous sub creation.
2016-04-29http: avoid lseek if no input
This saves us a system call for common GET/HEAD requests with no upload body.
2016-04-29http: improve error handling for aborted responses
We need to abort connections properly if a response is prematurely truncated. This includes problems with serving static files, since a clumsy admin or broken FS could return truncated responses and inadvertently leave a client waiting (since the client saw "Content-Length" in the header and expected a certain length).
2016-04-29http: avoid corking on "Content-Length: 0" response
We must use a normal write instead of send(.., MSG_MORE) when writing responses of "Content-Length: 0" to avoid the corking effect MSG_MORE provides. We only want to cork headers if we will send a non-empty body. Fixes: c3eeaf664cf0 ("http: clarify intent for persistence") This needs a proper test.
2016-04-06http: clarify intent for persistence
We don't actually need to know if a response is chunked or what the actual Content-Length is; we just need to know if the PSGI app properly terminated the response so we can handle persistent connections.
2016-04-04http: fix condition for detecting persistence
Oops, we need to watch out for how we handle operator precedence and ensure responses without a Content-Length or "Transfer-Encoding: chunked" header will always disconnect after writing.
2016-03-12http: use Plack::HTTPParser for HTTP parsing
This allows us to reduce installation dependencies while retaining performance as it favors HTTP::Parser::XS when it is installed and available. PLACK_HTTP_PARSER_PP may be set to 1 to force a pure Perl parser for testing.
2016-03-12http: prevent zero-byte writes
Plack::Middleware::Deflater (and perhaps other middleware) triggers zero-byte writes which wastes syscalls when they get passed to Danga::Socket. This may also trigger problems when we introduce TLS support in the future.
2016-03-06http: reject excessively large HTTP request bodies
We cannot risk using all of a users' disk space buffering gigantic requests. Use the defaults git gives us since we primarily host git repositories.
2016-03-06http: ensure errors are printable before PSGI env
We cannot rely on a client socket having a PSGI env before headers are fully-parsed as we seek to avoid storing hashes for idle clients. Sso print errors to the psgi.errors value which belongs to the httpd listener, instead.
2016-03-06http: reject excessive headers
HTTP::Parser::XS::PP does not reject excessively large headers like the XS version. Ensure we reject headers over 16K since public-inbox should never need such large request headers.
2016-03-03daemon: introduce host_with_port for identifying sockets
This allows us to share more code between daemons and avoids having to make additional syscalls for preparing REMOTE_HOST and REMOTE_PORT in the PSGI env in -httpd. This will also make supporting HTTP (and NNTP) over Unix sockets easier in a future commit.
2016-03-01http: better error handling for EMFILE/ENFILE
Better to throw the error back to the client ASAP if we're out-of-descriptors. We will need to implement idle client expiration for long-lived HTTP connections.
2016-02-29favor procedural calls for most private functions
This makes for better compile-time checking and also helps document which calls are private for HTTP and NNTP. While we're at it, use IO::Handle::* functions procedurally, too, since we know we're working with native glob handles.
2016-02-29http: avoid needless time2str calls
Checking the time is nearly free on modern systems with vDSO/vsyscall/similar while sprintf is always expensive.
2016-02-29http: document event_write usage
It may not be obvious where we are when we enter the event_write callback. Hopefully this clarifies things.
2016-02-29http: error check for sysseek on input
Just in case we screwed up somewhere, we need to match up syswrite to sysseek and we also favor procedural calls for native types.
2016-02-28http: support graceful shutdown like nntp
HTTP responses may be long-running or requests may be slow or pipelined. Ensure we don't kill them off prematurely.
2016-02-27http: simplify offset calculation slightly
No point in comparing an empty string; length() is only potentially expensive on big strings.
2016-02-25httpd: fill in missing Danga::Socket callbacks
Danga::Socket will die on us if we hit the base implementations.
2016-02-23initial public-inbox-httpd implemenation
This is meant to provide an easy starting point for server admins. It provides a basic HTTP server for admins unfamiliar with configuring PSGI applications as well as being an identical interface for management as our nntpd implementation. This HTTP server may also be a generic Plack/PSGI server for existing Plack/PSGI applications.