about summary refs log tree commit homepage
DateCommit message (Collapse)
2020-09-10solver: break apart inbox blob retrieval
To avoid hogging the event loop in public-inbox-httpd when many candidate messages match, we'll separate the steps to ensure fairness on slow storage.
2020-09-10solver: check one git coderepo and inbox at a time
With public-inbox-httpd, this mitigates the effect of slow git blob storage with multiple coderepos configured for an inbox. It's still synchronous for now (and may need to remain that way for ->last_check_err), but no longer monopolizes the event loop when checking multiple coderepos. We don't yet support multi-inbox scanning, yet; but this also prepares us for a future where we do. We'll also support >=40 char blob OIDs in preparation for future git SHA-256 support, too.
2020-09-10wwwlisting: avoid hogging event loop
By using the just-introduced ConfigIter class. And make ManifestJsGz a subclass of it to reduce duplication.
2020-09-10extmsg: prevent cross-inbox matches from hogging event loop
With many inboxes, checking multiple SQLite repos will be slow and time-consuming, so ensure we can schedule it fairly between multiple inboxes.
2020-09-10t/cgi.t: show stderr on failures
This helped me diagnose an error I would've introduced in the next commit.
2020-09-10config: split out iterator into separate object
We will need to allow simultaneous iterators on the same config object, since we'll need this for ExtMsg, NNTPD, WwwListing, NewsWWW, and other places.
2020-09-10config: flatten each_inbox and iterate_start args
In Perl, we can simplify callers by passing a single array all the way down the stack instead of a single array ref which needs to be expanded every call.
2020-09-10www: manifest.js.gz generation no longer hogs event loop
It's still as slow as before with hundreds/thousands of inboxes, but at least it's fair. Future changes will allow it to be cached and memoized with persistent HTTP servers.
2020-09-10use "\&" where possible when referring to subroutines
"*foo" is ambiguous in that it may refer to a bareword file handle; so we'll use it where we can without triggering warnings. PublicInbox::TestCommon::run_script_exit required dropping the prototype, however. We'll also future-proof by dropping "use warnings" in Cgit.pm and use the less-ambiguous "//=" in Inbox.pm while we're in the area.
2020-09-10solver: drop warnings, modernize use v5.10.1, use SEEK_SET
With Perl upstream preparing to deprecate things, we'll move towards only enabling warnings during development via shebang and stop enabling them via "use". We'll also favor "use v5.10.1" over the Perl 5.6-compatible "use 5.010_001", since our code base never worked on 5.6. Finally, were also importing SEEK_SET without using it, just use it for readability since we can't avoid loading Fcntl in other places and it'll get constant-folded, anyways.
2020-09-10xt/solver: test with public-inbox-httpd, too
We'll be making changes to solver to make it even fairer to slow clients on slow storage. Ensure we test with public-inbox-httpd-specific codepaths, since the generic PSGI code paths are rare in production use.
2020-09-10wwwtext: config comment improvements
Use the full URL of the inbox being mirrored to reduce ambiguity (instead of just the inbox name). Using asymmetric quotes (e.g `foo') improves readability for me in that it's more obvious when a quote begins and ends. It also lights up fewer pixels and reduces visual noise compared to double-quotes. We'll also reflow the `mainrepo' vs `inboxdir' comment slightly to emphasize the word `instead'.
2020-09-10wwwtext: don't blindly quote "git clone" destination
Save screen space and light up fewer pixels to reduce visual noise.
2020-09-10wwwtext: describe the use of `coderepo' entries
The `solver' feature is not very obvious, give potential users a hint about it.
2020-09-10nntp: fix cross-newsgroup Message-ID lookups
We cannot blindly use the selected newsgroup for HEAD/ARTICLE/BODY requests using Message-ID, since those commands look across all newsgroups; not just the selected one (if any). So stuff a reference to the Inbox object into $smsg. We can reduce args passed into set_nntp_headers() and msg_hdr_write(), too. Fixes: 0e6ceff37fc38f28 ("nntp: support slow blob retrievals")
2020-09-09wwwstream: fix "Atom feed" link
Oops, I wanted to stop escaping double-quotes with `qq()' but used `q()' instead :x Fixes: 2f61828fcb727e51 ("www: make mirror instructions more prominent")
2020-09-09contrib/css: limit <a> coloring to links, only
We don't want <a> tags without href= attributes to be colored, since the `<a id=mirror>' tag in the HTML footer is intended as an anchor destination for `<a href=#mirror>' link at the top.
2020-09-09www: make mirror instructions more prominent
In order to fight the misconception that public-inboxes are centralized, anchor "#mirror" to the clone instructions and place an emphasis on "mirror", not just cloning. While we're at it, better describe multi-epoch -V2 inboxes, since some users do not seem to realize epochs consist of different data.
2020-09-03v2writable: reuse read-only shard counting code
We'll also fix the read-only code to ensure we notice missing Xapian shards, since gaps would throw off our expectation that Xapian document IDs and NNTP article numbers are interchangeable.
2020-09-03overidx: document column uses
This may be useful for keeping our heads on straight dealing with IMAP, NNTP, JMAP, etc.
2020-09-03wwwaltid: drop unused sqlite3_missing function
It's inlined into the main function, which we'll shorten slightly with the defined-or (`//') operator. Also noticed and fixed a mismatched HTML tag.
2020-09-03imap: drop old, pre-Parse::RecDescent search parser
We switched to Parse::RecDescent during development and left some dead code behind.
2020-09-03search: remove {over_ro} field
Only inbox accesses the read-only {over}, now, instead of going through ->search. This simplifies our object graph and avoids potentially redundant FDs and DB handles pointing to the same over.sqlite3 file.
2020-09-03search: replace ->query with ->mset
Nearly all of the search uses in the production code rely on a Xapian mset iterator being returned (instead of an array of $smsg objects). So default to returning the mset and move the burden of smsg array conversion into the test cases.
2020-09-03tests: add "use strict" and declare v5.10.1 compatibility
strict.pm helped me find a typo in an upcoming recent change, so ensure we use it since it does more good than harm. We'll also take the opportunity here to declare v5.10.1 compatibility level to future-proof against Perl incompatibilities.
2020-09-03search: remove special case for blank query
The special case (if any) belongs at a higher-level, and this is another step towards removing {over_ro}-dependence in our Search object.
2020-09-03use more idiomatic internal API for ->over access
{over_ro} being a part of the Search object is a historical oddity which will go away, soon. Lets start removing its use in tests and rarely-used helper scripts.
2020-09-03disambiguate OverIdx and Over by field name
We'll use {oidx} as the common field name for the read-write OverIdx, here, to disambiguate it from the read-only {over} field. This hopefully makes it clearer which code paths are read-only and which are read-write.
2020-09-03msgmap: note how we use ->created_at
It'll likely be used in the future for JMAP, detached indices, and maybe other things.
2020-09-02t/run: Perl future proofing
Bareword file handles outside of STD(IN|OUT|ERR) seem to be on the chopping block for Perl 8. We'll also "use v5.10.1" to guard against future incompatibilities.
2020-09-02init+convert: create non-existing directory hierarchies
Following "git init" as an example, we'll create every parent path up to the one specified, instead of attempting to continue on when Cwd::abs_path returns `undef'.
2020-09-02doc: remove B<> (bold) markup from the remaining POD
B<> decreases readability of the POD source and is of dubious usefulness in the man page.
2020-09-02watch: add --help/-h support
And avoid unnecessary POD markup in the man page.
2020-09-02config: use defined-or (//) in a few places
Just some golfing to reduce scrolling and hopefully readability.
2020-09-02mda+learn: add --help / -h support
"use Getopt::Long" doesn't seem too slow on a hot page cache, and it's probably used frequently enough to be in cache. We'll also start reducing the amount of markup in the .pod and favoring verbatim text in documentation for readability in source form, since the bold text seems excessive.
2020-09-02daemon: support --help/-h in -httpd/imapd/nntpd
For consistency with other commands, though the protocol-specific options should refer users to the manpage.
2020-09-02script/*: fold $usage into $help, support `-h' instead of -?
`-h' doesn't conflict with anything, and some users (including git users) may be more accustomed to using it rather than the rarely-seen-outside-of-Getopt::Long `-?' switch. We can also rely on the GetOptions() function to emit a proper error message instead of just "bad command-line args".
2020-09-02edit+purge: support `--help' and `-h' like other commands
And while we're at it, note edit is *destructive* to encourage reading the fine manual.
2020-09-02admin: improve minimum version text
"inboxes 1 inboxes not supported by ..." was non-sensical. Now it'll show "-V1 inbox not supported by ...", instead.
2020-09-02script/*: set executable bit on -learn and -imapd
It's useful to mark they're meant to be executable, even if the shebang is useless.
2020-09-02t/v2dupindex: test indexing mirrors with duplicate messages
While it's not a known problem, our deduplicating logic may change in the future; or a BOFH could be manually injecting duplicate messages directly into the git epoch repositories. Ensure indexing in mirrors doesn't break when there's duplicates. This is in preparation for detached indices for multi-inbox search.
2020-09-02index: check for xapian-compact when using --compact
Otherwise, users may be frustrated to discover it missing a long indexing run.
2020-09-01replace ParentPipe with EOFpipe
ParentPipe was a subset of EOFpipe, except EOFpipe correctly accounts for theoretical(*) spurious wakeups on the pipe. (*) AFAIK, spurious wakeups are/were more likely on TCP sockets due to checksum failures, something that's not a problem on local pipes. We're also not sharing pipes like we do with listen sockets on accept(2), so there's no chance of another process grabbing bytes (unless we have bugs in our code).
2020-09-01ds: avoid unnecessary timer for waitpid
It doesn't seem necessary, since we won't call dwaitpid() until we see an EOF.
2020-09-01watch: use EOFpipe to reduce dwaitpid wakeups
It's a bit inefficient to use a pipe, here. However, using dwaitpid() on a process that's not expected to exit soon is also inefficient as it causes excessive wakeups as most of our inbox-writing code expects synchronous waitpid(). This only affects -watch instances configured for NNTP and IMAP clients.
2020-09-01ds: avoid excessive queueing when reaping PIDs
We should not enqueue reap_pids() to run more than once per EventLoop iteration. We'll start reformatting reap_pids to tabs, too, since we're no longer Danga::Socket. We should also be able to remove timer usage for reaping down-the-line once we stop abusing dwaitpid() in -watch.
2020-09-01watch: comments and tiny cleanups
Get rid of an unused variable, prefix a warning and try to better document control flow around various callbacks.
2020-09-01watch: block signals before fork on non-signalfd/kevent systems
In case there's non-Linux or BSD users w/o IO::KQueue, we shouldn't let signal handlers fire in the child processes. The child processes always assumed signals were blocked by the parent, so no changes were necessary, there.
2020-09-01watch: avoid unnecessary spawning on spam removals
This should further mitigate lock contention problems when -watch is configured to watch on a Maildir for spam while performing a large NNTP import. There is now a small risk a message won't get removed because if it's in the current (uncommitted) fast-import batch, but unlikely given the batch size is now only 10 messages. If a that small window is hit, flipping the \Seen flag (e.g. marking it unread, and then read again) will trigger another removal attempt via IMAP or Maildir.
2020-09-01watch: log signal activities to STDERR
Sometimes it may not be apparent when/if a signal is processed, this hopefully improves the situation. We'll also change the process title when we're quitting to better inform users.