about summary refs log tree commit homepage
DateCommit message (Collapse)
2020-07-17import: use common capitalization for filtering headers
In case this ends up in the same process as Mbox::msg_hdr, it can reduce memory use by sharing the cache key in PublicInbox::Eml::re_memo
2020-07-17drop binmode usage
We only support Unix-like platforms where binmode (":raw") is the default anyways, and v5.10 semantics means it won't do unicode_strings (unlike v5.12). So save some lines of code.
2020-07-17v2: use v5.10.1, parent.pm, drop warnings
The "5.010_001" form was for Perl 5.6, which I doubt anybody would attempt; so favor "v5.10.1" as it is more readable to humans. Prefer "parent" to "base" since the former is lighter. We'll also rely on warnings from "-w" globally (or not) instead of via "use". We'll also update "use" statements to reflect what's actually used by V2Writable.
2020-07-17t/import: quiet warning, clobber variable
The eval in key2sub via t/run.perl ("make check-run") won't trigger the warning, but running "prove -bvw t/import.t" directly, does. In any case, ensure the contents of this variable doesn't linger across runs.
2020-07-17doc: add some recommendations around slow HDDs
grok-pull is still painful with serialization on an old USB 2.0 HDD, but at least it can finish with flock(1) and disabling parallelization. While parallel "git fetch" doesn't seem so bad, slow seeks are exacerbated by parallel reads in Xapian. That means some updates can take days instead of hours. The same updates take only seconds or minutes on an SSD.
2020-07-17config: reject `\n' in `inboxdir'
"\n" and other characters requiring quoting and/or escaping in in $GIT_DIR/objects/info/alternates was not supported in git 2.11 and earlier; nor does it seem supported at all in libgit2. This will allow us to support sharing git-cat-file or similar endpoints across multiple inboxes via alternates. This breaks an existing use case for anybody wacky enough to put `\n' in the `inboxdir' pathname; but I doubt this affects anybody.
2020-07-14doc: release notes and version info updates
Update release notes with some features in the 1.6 timeline. We'll note the version availability of some command-line options, it may help users who are reading the latest documentation online but running older versions.
2020-07-14over+msgmap: do not store filename after DBI->connect
SQLite already knows the filename internally, so avoid having it as a long-lived Perl SV to save some bytes when there's many inboxes and open DBs.
2020-07-14nntpd+imapd: detect unlinked msgmap
While it's even less common to experience a replaced msgmap.sqlite3 file, BOFHs may do the darndest things. This is another step towards reducing the number of needless wakeups we need to do in long-lived read-only daemons.
2020-07-14over: unset sqlite_unicode attribute
None of the human-readable strings stored in over.sqlite3 require UTF-8. Message-IDs do not, nor do the compressed Subject IDs (sid) we use for Subject-based threading. And the `ddd' (doc-data-deflated) column is of course binary data. This frees us of having to use SQL_BLOB for the `ddd', column, and will open the door for us to use dbh_new for Msgmap, too.
2020-07-14xapcmd: delay over->check_inodes trigger
We must not trigger wakeups on InboxIdle users until after we've renamed all files into place. Otherwise, the InboxIdle caller may just reopen the old (soon-to-be-unlinked) file. This fixes occasional test failures in t/nntpd.t Fixes: f977826a17f8735e ("lock: reduce inotify wakeups")
2020-07-13xt/mem-imapd-tls: avoid EMFILE in -imapd process
Test::More dups standard FDs and may create FDs for other purposes. run_mode => 0 lets us rely on FD_CLOEXEC to ensure -imapd has enough FDs to accept all incoming connections at the cost of higher (one-off) startup time.
2020-07-13imap: SEARCH fails more gracefully in non-slice mailbox
Instead of returning "BAD program fault", just give the standard "BAD search not available"... message we show for mailbox slices.
2020-07-10wwwatomstream: avoid uninitialized warnings for $email
As in Import, we'll fall back to Sender: if From: is missing, and use the primary_address of the inboxes to indicate the total absence of those fields.
2020-07-10imap: avoid warnings on non-slice mailboxes
Non-slice mailboxes never have messages themselves, so we must not assume a message exists when sending untagged EXISTS messages.
2020-07-10viewvcs: stop checking unused "B" query parameter
The resulting OID ("oid_b") is a required arg and part of $env->{PATH_INFO}, instead; so it's never part of an optional query parameter.
2020-07-10hval: to_filename: return `undef' instead of empty string
Returning an empty string for a filename makes no sense, so instead return `undef' so the caller can setup a fallback using the "//" operator. This fixes uninitialized variable warnings because split() on an empty string returns `undef', which caused to_filename to warn on s// and tr// ops.
2020-07-10viewvcs: allow "0" as a path name
This means we need to filter out "" from query parameters. While we're at it, update comments for the WWW endpoint.
2020-07-10doc: standards: link IMAP capabilities and response codes
We'll be implementing some IMAP search/threading extensions in IMAP and providing analogues over HTTP via JMAP.
2020-07-07t/spawn: fix test reliability
Since Perl doesn't internally use a self-pipe for sleep/select/poll/etc, wake up every 10ms to ensure it can see the SIGCHLD; since neither signalfd nor EVFILT_SIGNAL are always available. Fixes: 761baa2a300e4268 ("spawn: unblock SIGCHLD in subprocess")
2020-07-07mda: use PublicInbox::Address to extract emails
I'm not sure why this wasn't done in Jun/July 2016 when I was working on PublicInbox::Address to replace the DoS-vulnerable Email::Address. Nowadays, PublicInbox::Address allows using Email::Address::XS which should be fast and robust.
2020-07-06doc/technical/whyperl: note Perl 7 announcement
Right now[1] the Perl upstream plan is to maintain 5 compatibility in Perl 7 for at least 5 years[1], and perhaps drop it when Perl 8 comes along. That said, distros may pick it and maintain 5 on their own given the vast amounts of perfectly good legacy code out there. [1] http://nntp.perl.org/group/perl.perl5.porters/257817 [2] http://nntp.perl.org/group/perl.perl5.porters/257565
2020-07-06doc/technical/whyperl: reword bit around installed docs
I originally proposed this rewording to address Leah's comment but forgot to squash it in :x Link: https://public-inbox.org/meta/20200408221741.GA10142@dcvr/ Cc: Leah Neukirchen <leah@vuxu.org>
2020-07-06doc: daemon: update documentation around Inline::C
`~/.cache/public-inbox/inline-c' is supported, nowadays for convenience, but Inline::C usage will remain opt-in.
2020-07-06git: use v5.10.1, parent.pm and Time::HiRes::stat
parent.pm is leaner than base.pm, and Time::HiRes::stat is more accurate, so take advantage of these Perl 5.10+-isms since it's been over a year since we left 5.8 behind.
2020-07-06gzipfilter: check http->{forward} for client disconnects
We actually don't do anything with {env} or {'psgix.io'} on client aborts, so checking the truthiness of '{forward}' is necessary.
2020-07-06daemon: warn on missing blobs
Since -edit and -purge should be rare and TOCTOU around them rarer still; missing {blobs} could be indicative of a real bug elsewhere. Warn on them. And I somehow ended up with 3 different field names for Inbox objects. Perhaps they'll be made consistent in the future.
2020-07-06gzipfilter: drop HTTP connection on bugs or data corruption
While all the {async_next} callbacks needed eval guards anyways because of DS->write, {async_eml} callbacks did not. Ensure any bugs in our code or data corruption result in termination of the HTTP connection, so as not to leave clients hanging on a response which never comes or is mangled in some way.
2020-07-06wwwattach: support async blob retrievals
We can reuse some of the GzipFilter infrastructure used by other WWW components to handle slow blob retrieval, here. The difference from previous changes is we don't decide on the 200 status code until we've retrieved the blob and found the attachment. While we're at it, ensure we can compress text attachment responses once again, since all text attachments are served as text/plain.
2020-07-06wwwtext: simplify gzf_maybe use
gzf_maybe always returns a GzipFilter object, even if it uses CompressNoop. We can also use ->zflush instead of ->translate(undef) here for the final bit.
2020-07-06view: simplify eml_entry callers further
This simplifies the primary callers of eml_entry while only making mknews.perl worse.
2020-07-06www: update internal docs
We no longer favor getline+close for streaming PSGI responses when using public-inbox-httpd. We still support it for other PSGI servers, though.
2020-07-06wwwstream: eliminate ::response, use html_oneshot
All of our streaming responses use ::aresponse, now, and our synchronous responses use html_oneshot. So there's no need for the old WwwStream::response.
2020-07-06view: /$INBOX/$MSGID/t/: avoid extra hash lookup in eml case
We can build and buffer the HTML <head> section once the first non-ghost message in a thread is loaded, so there's no need to perform an extra check on $ctx->{nr} once the $eml is ready.
2020-07-06view: eml_entry: reduce parameters
We can save stack space and simplify subroutine calls, here.
2020-07-06ssearchview: /$INBOX/?q=$QUERY&x=t uses async blobs
Another 10% or so speedup when displaying full messages off search results.
2020-07-06feed: /$INBOX/new.html fetches blobs asynchronously
Once again this speeds another endpoint up 10% or so.
2020-07-06feed: generate_i: eliminate pointless loop
$ctx->{msgs} won't ever contain undef values.
2020-07-06view: update /$INBOX/$MSGID/T/ to be async
Another 10% or so speedup in a frequently-hit endpoint.
2020-07-06view: /$INBOX/$MSGID/t/ reads blobs asynchronously
Once again, this shows a ~10% speedup with multi-message threads in xt/httpd-async-stream.t regardless of whether TEST_JOBS is 1 or 100.
2020-07-06view: make /$INBOX/$MSGID/ permalink async
This will allow -httpd to handle other requusts if waiting on an HDD seek or git to decode a blob.
2020-07-06wwwstream: subclass off GzipFilter
This makes WwwStream closer to MboxGz and WwwAtomStream and will eventually allow us to follow the same patterns.
2020-07-06wwwstream: use parent.pm and no warnings
parent.pm is leaner than base and we'll rely on `-w' for warnings during development.
2020-07-06remove unused/redundant zlib-related imports
Z_FINISH is the default for Compress::Raw::Zlib::Deflate->flush, anyways, so there's no reason to import it. And none of C::R::Z is needed in WwwText now that gzf_maybe handles it all.
2020-07-06www: start making gzipfilter the parent response class
Virtually all of our responses are going to be gzipped, anyways. This will allow us to utilize zlib as a buffering layer and share common code for async blob retrieval responses. To streamline this and allow GzipFilter to be a parent class, we'll replace the NoopFilter with a similar CompressNoop class which emulates the two Compress::Raw::Zlib::Deflate methods we use. This drops a bunch of redundant code and will hopefully make upcoming WwwStream changes easier to reason about.
2020-07-06wwwstream: reduce blob fetch paths for ->getline
This will make it easier to support asynchronous blob retrievals. The `$ctx->{nr}' counter is no longer implicitly supplied since many users didn't care for it, so stack overhead is slightly reduced.
2020-07-06wwwstream: reduce object graph depth
Like with WwwAtomStream and MboxGz, we can bless the existing $ctx object directly to avoid allocating a new hashref. We'll also switch from "->" to "::" to reduce stack utilization.
2020-07-06wwwatomstream: support async blob fetch
This allows -httpd to handle other requests while waiting for git to retrieve and decode blobs. We'll also break apart t/psgi_v2.t further to ensure tests run against -httpd in addition to generic PSGI testing. Using xt/httpd-async-stream.t to test against clones of meta@public-inbox.org shows a 10-12% performance improvement with the following env: TEST_JOBS=1000 TEST_CURL_OPT=--compressed TEST_ENDPOINT=new.atom
2020-07-06xt/httpd-async-stream: allow more options
We want to be able to parallelize and stress test more endpoints and toggle `--compressed' and possibly other options in curl.
2020-07-06wwwatomstream: reuse $ctx as $self
No need to deepen our object graph, here.