about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2020-07-06view: eml_entry: reduce parameters
We can save stack space and simplify subroutine calls, here.
2020-07-06ssearchview: /$INBOX/?q=$QUERY&x=t uses async blobs
Another 10% or so speedup when displaying full messages off search results.
2020-07-06feed: /$INBOX/new.html fetches blobs asynchronously
Once again this speeds another endpoint up 10% or so.
2020-07-06feed: generate_i: eliminate pointless loop
$ctx->{msgs} won't ever contain undef values.
2020-07-06view: update /$INBOX/$MSGID/T/ to be async
Another 10% or so speedup in a frequently-hit endpoint.
2020-07-06view: /$INBOX/$MSGID/t/ reads blobs asynchronously
Once again, this shows a ~10% speedup with multi-message threads in xt/httpd-async-stream.t regardless of whether TEST_JOBS is 1 or 100.
2020-07-06view: make /$INBOX/$MSGID/ permalink async
This will allow -httpd to handle other requusts if waiting on an HDD seek or git to decode a blob.
2020-07-06wwwstream: subclass off GzipFilter
This makes WwwStream closer to MboxGz and WwwAtomStream and will eventually allow us to follow the same patterns.
2020-07-06wwwstream: use parent.pm and no warnings
parent.pm is leaner than base and we'll rely on `-w' for warnings during development.
2020-07-06remove unused/redundant zlib-related imports
Z_FINISH is the default for Compress::Raw::Zlib::Deflate->flush, anyways, so there's no reason to import it. And none of C::R::Z is needed in WwwText now that gzf_maybe handles it all.
2020-07-06www: start making gzipfilter the parent response class
Virtually all of our responses are going to be gzipped, anyways. This will allow us to utilize zlib as a buffering layer and share common code for async blob retrieval responses. To streamline this and allow GzipFilter to be a parent class, we'll replace the NoopFilter with a similar CompressNoop class which emulates the two Compress::Raw::Zlib::Deflate methods we use. This drops a bunch of redundant code and will hopefully make upcoming WwwStream changes easier to reason about.
2020-07-06wwwstream: reduce blob fetch paths for ->getline
This will make it easier to support asynchronous blob retrievals. The `$ctx->{nr}' counter is no longer implicitly supplied since many users didn't care for it, so stack overhead is slightly reduced.
2020-07-06wwwstream: reduce object graph depth
Like with WwwAtomStream and MboxGz, we can bless the existing $ctx object directly to avoid allocating a new hashref. We'll also switch from "->" to "::" to reduce stack utilization.
2020-07-06wwwatomstream: support async blob fetch
This allows -httpd to handle other requests while waiting for git to retrieve and decode blobs. We'll also break apart t/psgi_v2.t further to ensure tests run against -httpd in addition to generic PSGI testing. Using xt/httpd-async-stream.t to test against clones of meta@public-inbox.org shows a 10-12% performance improvement with the following env: TEST_JOBS=1000 TEST_CURL_OPT=--compressed TEST_ENDPOINT=new.atom
2020-07-06wwwatomstream: reuse $ctx as $self
No need to deepen our object graph, here.
2020-07-06wwwatomstream: use PublicInbox::Inbox->modified for feed_updated
stat(2) on the inboxdir is unlikely to be correct, now that msgmap truncates its journal (rather than unlinking it).
2020-07-06wwwatomstream: simplify feed_update callers
We always return Z (UTC) times, anyways, so we'll always use gmtime() on the seconds-after-the-epoch.
2020-07-06mbox: async blob fetch for "single message" raw mboxrd
This restores gzip-by-default behavior for /$INBOX/$MSGID/raw endpoints for all indexed inboxes. Unindexed v1 inboxes will remain uncompressed, for now.
2020-07-06mboxgz: reduce hash depth
We can bless $ctx directly into a MboxGz object to reduce hash lookups and allocations.
2020-07-06mboxgz: do asynchronous git blob retrievals
This lets the -httpd worker process make better use of time instead of waiting for git-cat-file to respond. With 4 jobs in the new test case against a clone of <https://public-inbox.org/meta/>, a speedup of 10-12% is shown. Even a single job shows a 2-5% improvement on an SSD.
2020-07-06qspawn: learn to gzip streaming responses
This will allow us to gzip responses generated by cgit and any other CGI programs or long-lived streaming responses we may spawn.
2020-07-06wwwstatic: support gzipped directory listings
This will allow others to mimic our award-winning homepage design without needing to rely on Plack::Middleware::Deflater or varnish to compress responses.
2020-07-06mbox: remove html_oneshot import
It's no longer needed, we no longer show a runtime error for zlib being missing, as zlib is a hard requirement. Fixes: a318e758129d616b ("make zlib-related modules a hard dependency")
2020-07-06{gzip,noop}filter: ->zmore returns undef, always
This simplifies callers, as witnessed by the change to WwwListing. It adds overhead to NoopFilter, but NoopFilter should see little use as nearly all HTTP clients request gzip.
2020-07-06gzipfilter: replace Compress::Raw::Deflate usages
The new ->zmore and ->zflush APIs make it possible to replace existing verbose usages of Compress::Raw::Deflate and simplify buffering logic for streaming large gzipped data. One potentially user visible change is we now break the mbox.gz response on zlib failures, instead of silently continuing onto the next message. zlib only seems to fail on OOM, which should be rare; so it's ideal we drop the connection anyways.
2020-07-06wwwlisting: use GzipFilter for HTML
The changes to GzipFilter here may be beneficial for building HTML and XML responses in other places, too.
2020-07-06www: need: use WwwStream::html_oneshot
It'll give us a nicer HTML header and footer.
2020-07-06wwwtext: switch to html_oneshot
No point in streaming a tiny response via ->getline, but we may stream to a gzipped buffer, later.
2020-07-06wwwtext: gzip text/plain responses, as well
Most of our plain-text responses are config files big enough to warrant compression.
2020-07-06www*stream: gzip ->getline responses
Our most common endpoints deserve to be gzipped.
2020-07-06wwwstream: oneshot: perform gzip without middleware
Plack::Middleware::Deflater forces us to use a memory-intensive closure. Instead, work towards building compressed strings in memory to reduce the overhead of buffering large HTML output.
2020-07-06gzipfilter: minor cleanups
We currently don't use bytes::length in ->write, so there's no need to `use bytes'. Favor `//=' to describe the intent of the conditional assignment since the C::R::Z::Deflate object is always truthy. Also use the local $gz variable to avoid unnecessary {gz} hash lookups.
2020-07-06git_async_cat: unref pipes on EOF from git->cleanup
We avoided a managed circular reference in 10ee3548084c125f but introduced a pipe FD leak, instead. So handle the EOF we get when the "git cat-file --batch" process exits and closes its stdout FD. v2: remove ->close entirely. PublicInbox::Git->cleanup handles all cleanup. This prevents us from inadvertantly deleting the {async_cat} field associated with a different pipe than the one GAC is monitoring. Fixes: 10ee3548084c125f ("git_async_cat: remove circular reference")
2020-07-06inboxidle: warn on inotify_add_watch failures
inotify_add_watch(2), open(2), stat(2) may all fail due to permissions errors, especially when running -nntpd/-imapd as `nobody' as recommended.
2020-07-05watch: don't burn CPU on IDLE failures
Network connections fail and need to be detected sooner rather than later during IDLE to avoid backtrace floods. In case the IDLE process dies completely, don't respawn right away, either, to avoid entering a respawn loop. There's also a typo fix :P
2020-07-02spawn: drop unused sys/uio.h include
We no longer use writev(2) in pi_fork_exec to emit errors.
2020-07-02overidx: document why we don't use SQLite WAL
I was wondering about this myself the other day and had to read up on it. So make a note of it for future readers.
2020-07-02tests: add use/require statements for TEST_RUN_MODE=0
The default (and fast) TEST_RUN_MODE=2 preloads most modules, but TEST_RUN_MODE=0 is more realistic and can catch some problems which may show up in real-world use.
2020-07-02watch: retry signals to kill IDLE and polling processes
To ensure reliable signal delivery in Perl, it seems we need to repeatedly signal processes which aren't using signalfd (or EVFILT_SIGNAL) with our event loop.
2020-07-02spawn: modernize with parent.pm, drop warnings.pm
parent.pm is smaller than base.pm, and we'll also move towards relying on `-w' (or not) to toggle process-wide warnings during development.
2020-07-02spawn: make @RLIMITS an array
Making the RLIMITS list a function doesn't allow constant folding, so just make it an array accessible to other modules.
2020-07-02inboxidle: avoid per-inbox anonymous subs
Anonymous subs cost over 5K each on x86-64. So prefer the less-recommended-but-still-documented way of using Linux::Inotify2::watch to register watchers. This also updates FakeInotify to detect modifications correctly when used on systems with neither IO::KQueue nor Linux::Inotify2.
2020-06-30watch: make waitpid() synchronous for Maildir scans
Maildir scanning still happens in the main process. Scanning dozens of Maildirs is still time-consuming and monopolizes the event loop during WatchMaildir::event_step. This can cause cause zombies to accumulate before Sigfd::event_step triggers DS::reap_pids.
2020-06-30spawn: unblock SIGCHLD in subprocess
Subprocess we spawn may want to use SIGCHLD for themselves. This also ensures we restore default signal handlers in the pure Perl version.
2020-06-30watch: ensure SIGCHLD works in forked children
In case our git or spam checker subprocesses spawn subprocesses of their own. We'll also ensure signal handlers are properly setup before unblocking them.
2020-06-30watch: show path for warnings from spam messages
It could be useful to see warnings generated for known problematic messages just as it is for possibly non-problematic ones.
2020-06-30watch: check for duplicates in ->over before spamcheck
It's cheaper to check for duplicates than run `spamc' repeatedly when rechecking. We already do this for v1 with by using the "ls" command with fast-import, but v2 requires checking against over.sqlite3.
2020-06-28watch: simplify internal structures
We won't be attempting to reuse Mail::IMAPConnections used to check authentication info, for now, so stop storing $self->{mics}. We can also combine $poll initialization for IMAP and NNTP to avoid data structure duplication. Furthermore, rely on autovivification to create {idle_pids} and {poll_pids}.
2020-06-28imaptracker: use flock(2) around writes
SQLite only issues non-blocking F_SETLK ops (not F_SETLKW) and retries failures using a configurable busy_timeout. SQLite's busy loop sleeps for a millisecond and retries the lock until the configured busy_timeout is hit. Trying to set ->sqlite_busy_timeout to larger values (e.g. 30000 milliseconds) still leads to failure when running the new stress test with 8 processes with TMPDIR on a 7200 RPM HDD. Inspection of SQLite source reveals there's no built-in way to use F_SETLKW, so tack on the existing flock(2) support we use to synchronize git + SQLite + Xapian for inbox writing. We use flock(2) instead of POSIX fcntl(2) locks since Perl doesn't provide a way to manipulate "struct flock" portably.
2020-06-28watch: support ~/.netrc via Net::Netrc
While git-credential-netrc exists in git.git contrib/, it may not be widely known or installed. Net::Netrc is already a standard part of most (if not all) Perl installations, so use it directly if available.