about summary refs log tree commit homepage
path: root/lib/PublicInbox/Config.pm
DateCommit message (Collapse)
2021-03-17extindex: add some validation and config knobs for WWW
We'll try to share a bit more configuration with extindex entries for WWW PSGI usage.
2021-03-11config: use '-f' key to store config file pathname
This fixes ->urlmatch use from lei, which already sets '-f'. I noticed this because imap.$URL.compress was ignored in my lei config file.
2021-02-04tests: guard against missing DBD::SQLite
The features we use for SharedKV could probably be implemented with GDBM_File or SDBM_File, but that doesn't seem worth it at the moment since we depend on SQLite elsewhere.
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2020-12-23config: config_fh_parse: micro-optimize harder
Instead of relying on split() and a regexp, we'll drop split() entirely and rely on index() + two substr() calls to operate on fixed strings. This brings PublicInbox::Config->new time down from 0.98s down to 0.84s.
2020-12-23config: config_fh_parse: micro-optimize
We can avoid a slow regexp capture and instead and rely on rindex + substr to extract the section from the config file. Then we use the defined-or-assignment (//=) operator combined with the documented return value of `push' to ensure @section_order is unique without repeating a hash lookup. Finally, we avoid short-lived variables inside the loop and declare them subroutine-wide to knock a teeny bit of allocation time. Combined, these optimizations bring the ~1.22s PublicInbox::Config->new time down to ~0.98s with 50K inboxes.
2020-12-23config: git_config_dump: pre-compile RE for split
It appears the Perl split() operator is not optimized for fixed strings at all. With this change, PublicInbox::Config->new (w/o ->fill_all) time is reduced from 1.81s to 1.22s on a config file with 50K inboxes.
2020-12-23config: _fill: inbox name extraction optimization
Using substr() instead of a string copy + s// substitution here reduces ->fill_all from 4.00s to 3.88s with 50K inboxes on my workstation.
2020-12-21use rel2abs_collapsed when loading Inbox objects
We need to canonicalize paths for inboxes which do not have a newsgroup defined, otherwise ->eidx_key matches can fail in unexpected ways.
2020-12-20config: eliminate unnecessary join call up front
We can rely on implicit join in string interpolation on die() iff needed. And just creating the arrayref up front to avoid an extra backslash seems nicer at the moment.
2020-12-20inboxidle: remove needless check for {inboxdir}
->each_inbox will never attempt to iterate an object without {inboxdir}, and simplify + short-circuit the corresponding code
2020-12-09rename {pi_config} fields to {pi_cfg}
{pi_config} may be confused with the documented `PI_CONFIG' environment variable, and we'll favor vowel-removal to be consistent with our usage of object references. The `pi_' prefix may stay in some places, for now; since a separate namespace may come into this codebase for local/private client-tooling. For InboxIdle, we'll also remove an invalid comment about holding a reference to the PublicInbox::Config object, too.
2020-12-05isearch: emulate per-inbox search with ->ALL
Using "eidx_key:" boolean prefix to limit results to a given inbox, we can use ->ALL to emulate and replace per-Inbox xap15/[0-9] search indices. With this change, the presence of "extindex.all.topdir" in the $PI_CONFIG will cause the WWW code to use that extindex and ignore per-inbox Xapian DBs in xap15/[0-9]. Unfortunately IMAP search still requires old per-inbox indices, for now. Mapping extindex Xapian docids to per-Inbox UIDs and vice-versa is proving tricky. Fortunately, IMAP search is rarely used and optional. The RFCs don't specify expensive phrase search, either, so `indexlevel=medium' can be used in per-inbox Xapian indices to save space. For primarily WWW (and future JMAP) users; this should result in significant disk space, FD, and page cache footprint savings for large instances with many inboxes and many cross-posted messages.
2020-12-05newswww: use ->ALL to avoid O(n) inbox scan
We can avoid doing a Message-ID lookup on every single inbox by using ->ALL to scan its over.sqlite3 DB. This mimics NNTP behavior and picks the first message indexed, though redirecting to /all/$MESSAGE_ID/ could be done. With the current lore.kernel.org set of inboxes (~140), this provides a 10-40% speedup depending on inbox ordering.
2020-12-01nntpd: move {newsgroup} name check to config
With 50K newsgroups in the config file, this doesn't slow down `PublicInbox::Config->new->fill_all' any measurable amount on my busy old workstation. This should prevent invalid newsgroup names from getting into into extindex and catch user errors sooner, rather than later. v2: - delete {newsgroup} if invalid to avoid ->nntp_url link - simplify -imapd and explain remaining check
2020-11-28nntpd: share {groups} hash with {-by_newsgroup} in Config
There's no need to duplicate a potentially large hash, but we can keep the inexpensive shortcut to it. We may eventually drop the {groups} shortcut if it's no longer useful.
2020-11-24manifest: support faster generation via [extindex "all"]
For a mirror of lore.kernel.org with >140 inboxes, this speeds up manifest.js.gz generation from ~1s to 40ms on my HW. This is still unacceptable when dealing with thousands of inboxes, but gets us closer to where we need to be.
2020-11-24move JSON module portability into PublicInbox::Config
We'll be using JSON in MiscIdx and MiscSearch, and PublicInbox::Config seems like an appropriate place to put it.
2020-11-08extsearch: rename -eindex to -extindex
Upon "eindex" rhymes with "reindex", which could be confusing; so name the command and config prefix to use "extindex" which is hopefully less confusing.
2020-11-07extsearch: wire up remaining Inbox-like methods for WWW
This lets us pretend an ExtSearch object is an Inbox object in most of the existing WWW code.
2020-09-22mda: match List-Id insensitively
This follows -watch commit b70473ab8296d31ebb600adb4fa8fe0ac5935ca8 to match List-Id headers case-insensitively. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20200921180152.uyqluod7qxbwqubo@chatter.i7.local/
2020-09-20config: warn on multiple values for some fields
Our code doesn't support multi-values for these, and having unexpected arrays leads to unexpected results (e.g. showing stuff like "ARRAY(0xDEADBEEFADD12E55)" in user interfaces). So warn and only use the last value (matching git-config(1) behavior without `--get-all').
2020-09-10config: split out iterator into separate object
We will need to allow simultaneous iterators on the same config object, since we'll need this for ExtMsg, NNTPD, WwwListing, NewsWWW, and other places.
2020-09-10config: flatten each_inbox and iterate_start args
In Perl, we can simplify callers by passing a single array all the way down the stack instead of a single array ref which needs to be expanded every call.
2020-09-02config: use defined-or (//) in a few places
Just some golfing to reduce scrolling and hopefully readability.
2020-08-07index: v2: --sequential-shard option
This gives better page cache utilization for Xapian indexing on slow storage by improving locality for random I/O activity on the Xapian DB. Instead of doing a single-pass to index both SQLite and Xapian; this indexes them separately. The first pass is identical to indexlevel=basic: it indexes both over.sqlite3 and msgmap.sqlite3. Subsequent passes only operate on a single Xapian shard for documents belonging to that shard. Given enough shards, each individual shard can be made small enough to fit into the kernel page cache and avoid HDD seeks for read activity. Doing rough tests with a busy system with a 7200 RPM HDD with ext4, full indexing of LKML (9 epochs) goes from ~80 hours (-j0) to ~30 hours (-j8) with 16GB RAM with 7 shards configured and fsync(2) disabled (--no-sync) and `--batch-size=10m'.
2020-07-17config: reject `\n' in `inboxdir'
"\n" and other characters requiring quoting and/or escaping in in $GIT_DIR/objects/info/alternates was not supported in git 2.11 and earlier; nor does it seem supported at all in libgit2. This will allow us to support sharing git-cat-file or similar endpoints across multiple inboxes via alternates. This breaks an existing use case for anybody wacky enough to put `\n' in the `inboxdir' pathname; but I doubt this affects anybody.
2020-06-28config: support ->urlmatch method for -watch
Since we have IMAP client support in -watch; make sure per-URL settings are familiar to git users by taking advantage of git's URL matching abilities. This requires git 1.8.5+, which most users ought to have (though base CentOS 7 is on 1.8.3).
2020-06-13imap: require ".$UID_MIN-$UID_END" suffix
Finish up the IMAP-only portion of iterative config reloading, which allows us to create all sub-ranges of an inbox up front. The InboxIdler still uses ->each_inbox which will struggle with 100K inboxes. Having messages in the top-level newsgroup name of an inbox will still waste bandwidth for clients which want to do full syncs once there's a rollover to a new 50K range. So instead, make every inbox accessible exclusively via 50K slices in the form of "$NEWSGROUP.$UID_MIN-$UID_END". This introduces the DummyInbox, which makes $NEWSGROUP and every parent component a selectable, empty inbox. This aids navigation with mutt and possibly other MUAs. Finally, the xt/perf-imap-list maintainer test is broken, now, so remove it. The grep perlfunc is already proven effective, and we'll have separate tests for mocking out ~100k inboxes.
2020-06-13imap: start doing iterative config reloading
This will be used to prevent reloading a giant config with tens/hundreds of thousands of inboxes from blocking the event loop.
2020-04-20watchmaildir: support multiple watchheader values
The watchheader key supports only a single value. Supporting multiple watchheader values was mentioned in discussion [1] of 8d3e3bd8 (doc: explain publicinbox.<name>.watchheader, 2019-10-09), and it wasn't clear if there was a need. One scenario in which matching multiple headers would be convenient is when someone wants to set up public-inbox archives for some small projects but does _not_ want to run mailing lists for them, instead allowing others to follow the project by any of the pull mechanisms. Using a common underlying address, an address alias for each project is configured via a third-party email provider, with messages for each alias being exposed as a separate public-inbox archive. In this setup, messages for an inbox cannot be selected by a List-ID header but can be identified by the inbox's address in either the To or Cc header. To support such a use case, update the watchheader handling to consider multiple values, accepting a message if it matches any value. While selecting a message based on matching _any_ rather than _all_ values is motivated by the above scenario, it's worth noting that the "any" behavior is consistent with how multiple listid config values are handled. [1] https://public-inbox.org/meta/20191010085118.r3amey4cayazfycb@dcvr/
2020-03-29config: Honor gitconfig includes
This allows for a setup where a central config file for the web server includes per-user config files.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2020-02-01config: assume multiple cgit URLs, too
Since we support inboxes with multiple URLs and multiple infourls to reduce reliance on SPOFs, we'll do the same with cgit URLs.
2020-01-13config: do not slurp entire cgitrc at once
cgitrc files can have hundreds or thousands of lines in them and slurping them into memory is a waste. "while (<$fh>)" only reads one line at a time, whereas "for (<$fh>)" reads the entire contents of the file into a temporary array.
2020-01-11spawn (and thus popen_rd) die on failure
Most spawn and popen_rd callers die on failure to spawn, anyways, and some are missing checks entirely. This saves us a bunch of verbose error-checking code in callers. This also makes popen_rd more consistent, since it already dies on pipe creation failures.
2020-01-06admin: do not lazy-load Inbox or Config packages
No point in lazy-loading these, since they're always loaded anyways and would not have portability problems on systems with minimal dependencies.
2020-01-02config: support multi-value inbox.*.*url
Since the beginning of this project, we've implicitly supported inboxes with multiple URLs by relying on the Host: header sent by the client ($env->{HTTP_HOST}). We now offer the option to explicitly configure multiple URLs for every inbox along with the ability to do a best-effort match for matching hostnames.
2019-12-27config: each_inbox: pass user arg to callback
Another place where we can replace anonymous subs with named subs by passing a user-supplied arg.
2019-10-16config: remove redundant inboxdir check
This was causing compatibility problems for old configs when using public-inbox-nntpd.
2019-10-16config: support "inboxdir" in addition to "mainrepo"
"mainrepo" ws a bad name and artifact from the early days when I intended for there to be a "spamrepo" (now just the ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be especially confusing, since v2 needs at least two git repositories (epoch + all.git) to function and we shouldn't confuse users by having them point to a git repository for v2. Much of our documentation already references "INBOX_DIR" for command-line arguments, so use "inboxdir" as the git-config(1)-friendly variant for that. "mainrepo" remains supported indefinitely for compatibility. Users may need to revert to old versions, or may be referring to old documentation and must not be forced to change config files to account for this change. So if you're using "mainrepo" today, I do NOT recommend changing it right away because other bugs can lurk. Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
2019-10-15config: allow "0" as a valid mainrepo path
It's probably wrong to use relative path names, but things are all relative these days anyways with shared and networked FSes.
2019-10-15config: avoid unnecessary '||' use
'//' is available in Perl 5.10+ which allows `0' and `""' (empty string) to remain unclobbered. We also don't need '||=' for initializing our internal caches.
2019-10-15config: simplify lookup* methods
This ensures we always process inboxes in section order and reduces the amount of code we have to maintain for each lookup. Avoiding the cost of inboxes object creation is not worth the code overhead; and we can implement a config cache via Storable easily for large configs and -mda users.
2019-10-15config: we always have {-section_order}
Rewrite a bunch of tests to use ordered input (emulating "git config -l" output) so we can always walk sections in the order they were given in the config file.
2019-10-15Config.pm: Add support for mailing list information
The world has turned since I first started following mailing lists and to my surprise every mailing list that I am subscribed to properly sets the "List-ID:" mailing list header. So instead of doing something clever and flexible I am adding support for looking up public inbox mailing lists by their mailing list name. That makes the work needed for each email trivial and easy to understand. - Parse the "List-ID:" header. - Lookup in the configuration which mailbox is connected to that "List-ID:" - Deliver the mail to that mailbox. To that end this change enhances PublicInbox to have an additional mailbox configuration parameter "listid" that holds the mailing list name. A method is added to the PublicInbox config object called lookup_list_id that given a mailing list name will return the PublicInbox in the configuration that is configured to handle that mailing list. Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com> [ew: avoid autovivification of $ibx->{listid} for t/config.t]
2019-09-30config: use NUL-delimited git-config(1) output
This allows us to deal with newlines in config values, since git-config(1) acquired "-z" support in git v1.5.3. I'm not sure if it's actually useful in our case, but maybe some multi-line texts could be added. And newlines in path names are super useful!
2019-09-18config: boolean handling matches git-config(1)
We need to handle arbitrary integers and case-insensitive variations of human words to match git-config(1) behavior, since that's what users would expect given we use config files parseable by git-config(1).
2019-09-09run update-copyrights from gnulib for 2019
2019-06-04config: do not accept non-ASCII digits in cgitrc params
cgit uses atoi(3), and now we can retain compatibility.