about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2020-08-27over*: use v5.10.1, drop warnings
v5.10.1 lets us use the lighter parent.pm instead of base.pm, and we'll rely on the shebang to enable warnings (or not). While we're in the area, drop a no-longer-necessary import for PublicInbox::Search, since OverIdx doesn't require search.
2020-08-27over: recent: remove expensive COUNT query
As noted in commit 87dca6d8d5988c5eb54019cca342450b0b7dd6b7 ("www: rework query responses to avoid COUNT in SQLite"), COUNT on many rows is expensive on big SQLite DBs. We've already stopped using that code path long ago in WWW while -imapd and -nntpd never used it. So we'll adjust our remaining test cases to not need it, either.
2020-08-27over: rename ->disconnect to ->dbh_close
Since we got rid of over->connect, `disconnect' no longer pairs with it. So name it after the `close(2)' syscall it ultimately issues.
2020-08-27over: rename ->connect method to ->dbh
`->connect' is confused with the perlfunc for the `connect(2)' syscall, and also `DBI->connect'. Since SQLite doesn't use sockets, the word "connect" needlessly confuses me. Give it a short name to match the field name we use for it, which also matches the variable name used by the DBI(3pm) and DBD::SQLite(3pm) manpages.
2020-08-26v2writable: compatibility with SWIG Xapian binding
The SWIG binding won't auto-convert IV/UV to PV like the XS Search::Xapian binding would, so workaround that shortcoming for now. Fixes: a367ec1b15a2458 ("mbox: disable "&t" on existing Xapian until full reindex")
2020-08-26over+msgmap: respect WAL journal_mode if set
WAL actually seems to have ideal locking characteristics given concurrency problems I'm experiencing with --reindex running in parallel with expensive read-only SQLite queries: <https://public-inbox.org/meta/20200825001204.GA840@dcvr/> Unfortunately, we cannot blindly use WAL while preserving compatibility with existing setups nor our guarantees that read-only daemons are indeed "read-only". However, respect an user's the choice to set WAL on their own if they're comfortable with giving -nntpd/-httpd/-imapd processes write permission to the directory storing SQLite DBs.
2020-08-26msgmap: use "CREATE TABLE IF NOT EXISTS"
It's fewer queries and matches what we do in OverIdx.
2020-08-26over: skip nodatacow on the journal
This file gets truncated anyhow, so it won't fragment.
2020-08-25searchidx: croak for Xapian DB open failure
croak() can give more context on the failure, and setting `PERL5OPT=-MCarp=verbose' can force a stacktrace.
2020-08-23index: --sequential-shard checkpoints after each shard
There's no reason we'd want Xapian to defer flushing once we've indexed everything belonging to a particular shard.
2020-08-23mbox: disable "&t" on existing Xapian until full reindex
Expanding threads via over.sqlite3 for mbox.gz downloads without Xapian effectively collapsing on the THREADID column leads to repeated messages getting downloaded. To avoid that situation, use a "has_threadid" Xapian metadata flag that's only set on --reindex (and brand new Xapian DBs). This allows admins to upgrade WWW or do --reindex in any order; without worrying about users eating up bandwidth and CPU cycles.
2020-08-23search: support downloading mboxes results with full thread
Finally, the addition of THREADID for collapsing results in Xapian lets us emulate the "mairix --threads" feature. That is, instead of returning only the matching messages, the entire thread is included in the downloaded mbox.gz This requires a "public-inbox-index --reindex" to be usable.
2020-08-23searchidx: index THREADID in Xapian
This is the `tid' column from over.sqlite3; and will be used for IMAP and JMAP search (among other things).
2020-08-23searchidx: put all shard-related stuff in SearchIdxShard.pm
We'll also rename the /^remote_/ prefix to "shard_", since remote implies the process is on a different host. These methods only pass messages to a child process on the same host OR perform operations within the same process.
2020-08-23searchidxshard: clear $msgref buffer properly
Merely assigning `undef' to a scalar does not free the underlying buffer memory of a scalar.
2020-08-22searchview: fix mbox.gz downloads for lynx users
Unlike w3m and links, the lynx browser seems to require a `name' attribute for `<input type=submit>' elements. Maybe some other browsers do, too. The `name' attribute for submit elements doesn't seem to cause any harm for w3m or links, users, either; despite not (AFAIK) being part of historical or current HTML specs.
2020-08-20search: add mset_to_artnums method
We can avoid importing mdocid() in several places by using this method, simplifying callers.
2020-08-20init+index: support --skip-docdata for Xapian
Since we no longer read document data from Xapian, allow users to opt-out of storing it. This breaks compatibility with previous releases of public-inbox, but gives us a ~1.5% space savings on Xapian storage (and associated I/O and page cache pressure reduction).
2020-08-20smsg: remove from_mitem
We no longer read docdata.glass from anywhere in our code base. Some adjustments were needed to t/search.t to deal with the Xapian::WritableDatabase committing at different times, since our ->query is avoided from PublicInbox::SearchIdx to avoid needing a {over_ro} field.
2020-08-20mbox: avoid Xapian docdata in search results
Another place where we can reduce kernel page cache overhead by hitting over.sqlite3 instead of docdata.glass.
2020-08-20extmsg: avoid using Xapian docdata
Once again, over.sqlite3 contains everything necessary for Message-ID resolution. Also, Xapian may be completely unnecessary with the advent of over.sqlite3, but that's for another time.
2020-08-20searchview: convert nested and Atom display to over.sqlite3
git blob retrieval dominates on these, "&x=t" (nested) is roughly the same due to increased overhead for ->get_percent storage balancing out the mass-loading from SQLite. Atom "&x=A" is sped up slightly and uses less memory in the long-lived response.
2020-08-20searchview: speed up search summary by ~10%
Instead of loading one article at-a-time from over.sqlite3, we can use SQL to mass-load IN (?,?, ...) all results with a single SQLite query. Despite SQLite being in-process and having no network latency, the reduction in SQL query executions from loading multiple rows at once speeds things up significantly. We'll keep the over->get_art optimizations from the previous commit, since it still speeds up long-lived responses, slightly.
2020-08-20searchview: use over.sqlite3 instead of Xapian docdata
This is a step towards improving kernel page cache hit rates by relying on over.sqlite3 for document data instead of Xapian. Some micro-optimization to over->get_art was required to maintain performance.
2020-08-20smsg: reduce utf8::decode call sites
Both callers of load_from_data call utf8::decode, so just do utf8::decode in load_from_data.
2020-08-20search: make qparse_new an internal function
We'll probably be reusing it from another package in a future commit.
2020-08-20searchquery: split off from searchview
Since this was already a separate package, split it off into its own file since SearchView may not handle inbox groups.
2020-08-20search: export mdocid subroutine
No need to have awkward globrefs for this.
2020-08-20search: improve comments around constants
We'll probably be adding more value columns like THREADID to sort on.
2020-08-20www: reduce long-lived PublicInbox::Search references
While this is unlikely to be a problem in current practice, keeping Xapian DBs open for long responses can interfere with free space recovery after -compact. In the future, it will interfere with inbox search grouping and lead to unexpected results.
2020-08-20xapcmd: simplify {reindex} parameter passing
No need to localize it, here, since we can just refer to it in the `$opt' hashref. Hopefully this improves readability for others like it does for me. I sometimes wonder if the concept of a stack in high-level languages is even necessary...
2020-08-20search: v2: ensure shards are numerically sorted
This seems required to correctly get the NNTP article number from Xapian docid on combined Xapian DBs. The default (ASCII-betical) sorting was only acceptable for -imapd users until somebody hit 11 (or more) shards, which is a rare case.
2020-08-20admin: progress shows the inbox being indexed
This is helpful with --all, or when multiple inboxes are being indexed.
2020-08-19v2writable: show newline after "indexing all of .. " message
Otherwise things get very confusing when verbosity is enabled :x
2020-08-19smsg: handle wide characters in raw mail headers
There may be messages in the wild with wide characters in headers which aren't non-RFC2047 encoded. Assume UTF-8 so those fields can round trip through over.sqlite3. This doesn't affect docdata.glass in Xapian, but it does affect how over.sqlite3 stores the same deflated info.
2020-08-13v2writable: remove IdxStack import
We use IdxStack via log2stack() from SearchIdx, now.
2020-08-13xcpdb: wire up new index options and --help
--sequential-shard also disables the copy parallelism (--jobs), so it can be useful for systems unable to handle parallel random I/O but still want many shards. There was a missing "use strict", too, which is fixed.
2020-08-13admin: don't warn when --jobs exceeds shards
Established tools like make(1), prove(1) and xargs(1) don't warn when the desired parallelism level can't be met, either.
2020-08-13xapcmd: reduce CPU idling when shards exceeds job count
In case there's unbalanced shards AND we're limiting parallelism while using many shards, spawn the next task in the queue ASAP once a task is done, instead of waiting for all tasks to finish before spawning the next batch. Unbalanced shards probably isn't a big issue for most users; however many smaller shards with few jobs can be useful for HDD users to reduce the effect of random writes.
2020-08-13xapcmd: simplify sub reference
We don't need to fully-qualify when referring to subs in the same namespace, nor do we need make a SCALAR ref only to dereference it (Yes, still learning Perl :x)
2020-08-10convert: support new -index options
Converting v1 inboxes from v2 can be a painful experience on HDD. Some of the new options in the CLI or config file make it less painful.
2020-08-10searchidx: use singular `$opt' for consistency with v2
The rest of our indexing code uses `$opt' instead of `$opts'.
2020-08-10index: cleanup internal variables
Move away from hard-to-read alllowercase naming and favor snake_case or separated-by-dashes. We'll keep `--indexlevel' as-is for now, since it's been around for several releases; but we'll support `--index-level' in the CLI and update our documentation in a few months. We'll also clarify that publicInbox.indexMaxSize is only intended for -index, and not -watch or -mda.
2020-08-10admin: use a generic variable name
We parse other options, too, not just --max-size
2020-08-10avoid File::Temp::tempfile in more places
We can use open(..., undef) natively in Perl in t/import.t In places where we need a pathname, the File::Temp OO API gives us auto-unlinking for free.
2020-08-10msgmap: tmp_clone: simplify + meaningful filename
Trying to use the newer ->sqlite_backup_to_dbh method doesn't seem worth it, as we'll have to support DBD::SQLite <= 1.60 another decade or more. Dumping 'msgmap-XXXXXXX' into $INBOX_DIR can appear a bit confusing to users, so give it a "mm_tmp-$PID-XXXXXXXX" name to emphasize it's a temporary file tied to a given PID. We also don't want to penalize read-only daemons with loading File::Temp, so do it lazily.
2020-08-10index+xcpdb: improve SIG{INT,TERM,HUP,PIPE} behavior
-index now invokes ->DESTROY like xcpdb does, which is necessary to cleanup $INBOX_DIR/msgmap-XXXXXXX files. We'll also exit with the expected values for various signals by adding 128 as described in <https://www.tldp.org/LDP/abs/html/exitcodes.html> -xcpdb now terminates worker processes and xapian-compact(1) invocations when prematurely killed, too.
2020-08-10index: --sequential-shard works incrementally
We should never reindex all data in Xapian unless --reindex is specified on the command-line. This means users who put publicInbox.indexSequentialShard in their config file won't have to put up with a full reindex at every invocation, only when they specify --reindex. We'll also cleanup the progress output to not emit non-sensical ranges where the starting number is higher than the end.
2020-08-09favor `getconf _NPROCESSORS_ONLN` over GNU nproc
getconf(1) itself is POSIX, while `_NPROCESSORS_ONLN' is not. However, FreeBSD (tested 11.4 and 12.1) and glibc (tested CentOS 7.x and Debian 10.x) both support `getconf _NPROCESSORS_ONLN'. GNU coreutils (and thus `nproc' or `gnproc') are not installed by default on the *BSDs, so we'll try the option most likely to exist on both glibc and *BSDs out-of-the-box.
2020-08-08dir_idle: require Perl 5.22+ for kqueue
IO::KQueue requires us to use fileno(DIRHANDLE) for setting up kqueue watches. This use of fileno() is only supported since Perl 5.22, so BSD users on older Perl will have to fall back to old polling. This affects users of -watch, currently; but will affect other read-only Xapian users soon.