about summary refs log tree commit homepage
path: root/lib/PublicInbox/Over.pm
DateCommit message (Collapse)
2021-08-28move ->ids_after from mm to over
Since we favor ->over in WWW and IMAP, move this method to ->over to reduce open files in common cases. This fixes the /$EXTINDEX_NAME/all.mbox.gz endpoint for extindex entries (which may get expensive...).
2021-06-03lei import: speed up kw updates for old IMAP messages
On a 4-core CPU, this speeds up "lei import" on a largish IMAP inbox with 75K messages from ~21 minutes down to 40s. Parallelizing with the new LeiImportKw WQ worker class gives a near-linear speedup and brought the runtime down to ~5:40. The new idx_fid_uid index on the "fid" and "uid" columns of blob2num in mail_sync.sqlite3 brought us the final speedup. An additional index on over.sqlite3#xref3(oidbin) did not help, since idx_nntp already exists and speeds up the new ->oidbin_exists internal API. I initially experimented with a separate "lei import-kw" command but decided against it since it's useless outside of IMAP+JMAP and would require extra cognitive overhead for both users and hackers. So LeiImportKw is just a WQ worker used by "lei import" and not its own user-visible command. v2: fix ikw_done_wait arg handling (ugh, confusing API :x)
2021-03-21lei import: vivify external-only messages
Keyword storage for external-only messages was preventing messages from being explicitly imported. Teach lei_store to vivify keyword-only entries into fully-indexed messages on import.
2021-03-21lei q: support vmd for external-only messages
"lei q" now preserves changes per-message keywords across invocations when it's --output (Maildir or mbox) is reused (with or without --augment). In the future, these changes will be monitored via inotify, EVFILT_VNODE or IMAP IDLE, too. Unfortunately, this currently prevents "lei import" from ever importing a message that's in an external. That will be fixed in a future change.
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2020-12-17extindex: support --rethread and content bifurcation
--rethread is useful for dealing with bugs and behaves just like it does with current inboxes. This is in case our content deduplication logic changes for whatever reason and causes previously merged messages to be considered "different". As with v2, this won't allow us to merge messages in a way that allows deduplicating messages which were previously considered different, but v2 inboxes do not allow that, either. In other words, this makes the --reindex and --rethread switches of -extindex match the behavior of v2 -index.
2020-12-17over: sort xref3 by xnum if ibx_id repeats
While unlikely to happen, it may be possible for messages from the same inbox to get indexed multiple times. Provide consistent results in this case for ease-of-testing.
2020-12-08over: gracefully show invalid ibx_id
While "public-inbox-extindex --gc" invocations try to ensure proper ordering, it is still possible for users to change the `inboxes' tables via sqlite3(1) or similar means. So show a "missing://ibx_id=$ibx_id" placeholder to avoid undefined variable warnings. URLs such as "imaps://..." will eventually be supported as eidx_keys, so having a URL-like "missing://" as a placeholder probably makes sense.
2020-11-07over: store xref3 data in over.sqlite3
We may not end up storing xref3 data in Xapian, actually. This will make indexlevel=basic possible, and along with --sequential-shard indexing support for slow storage. Making oidmap a separate table seems unnecessary, too, so fold it into the xref3 table since it's unlikely a git blob will be responsible for multiple xref3 rows.
2020-08-28www: improve navigation around contemporary threads
Sometimes it's useful to quickly get to threads and messages which are contemporaries of the current thread/message being focused on. This hopefully improves navigation by making: a) the top line (where $INBOX_DIR/description) is shown a link to the latest topics in search results and per-thread/per-message views. b) providing a link to contemporaries ("~YYYY-MM-DD") at around the thread overview skeleton area for per-thread and per-message views
2020-08-27over*: use v5.10.1, drop warnings
v5.10.1 lets us use the lighter parent.pm instead of base.pm, and we'll rely on the shebang to enable warnings (or not). While we're in the area, drop a no-longer-necessary import for PublicInbox::Search, since OverIdx doesn't require search.
2020-08-27over: recent: remove expensive COUNT query
As noted in commit 87dca6d8d5988c5eb54019cca342450b0b7dd6b7 ("www: rework query responses to avoid COUNT in SQLite"), COUNT on many rows is expensive on big SQLite DBs. We've already stopped using that code path long ago in WWW while -imapd and -nntpd never used it. So we'll adjust our remaining test cases to not need it, either.
2020-08-27over: rename ->disconnect to ->dbh_close
Since we got rid of over->connect, `disconnect' no longer pairs with it. So name it after the `close(2)' syscall it ultimately issues.
2020-08-27over: rename ->connect method to ->dbh
`->connect' is confused with the perlfunc for the `connect(2)' syscall, and also `DBI->connect'. Since SQLite doesn't use sockets, the word "connect" needlessly confuses me. Give it a short name to match the field name we use for it, which also matches the variable name used by the DBI(3pm) and DBD::SQLite(3pm) manpages.
2020-08-26over+msgmap: respect WAL journal_mode if set
WAL actually seems to have ideal locking characteristics given concurrency problems I'm experiencing with --reindex running in parallel with expensive read-only SQLite queries: <https://public-inbox.org/meta/20200825001204.GA840@dcvr/> Unfortunately, we cannot blindly use WAL while preserving compatibility with existing setups nor our guarantees that read-only daemons are indeed "read-only". However, respect an user's the choice to set WAL on their own if they're comfortable with giving -nntpd/-httpd/-imapd processes write permission to the directory storing SQLite DBs.
2020-08-26over: skip nodatacow on the journal
This file gets truncated anyhow, so it won't fragment.
2020-08-23search: support downloading mboxes results with full thread
Finally, the addition of THREADID for collapsing results in Xapian lets us emulate the "mairix --threads" feature. That is, instead of returning only the matching messages, the entire thread is included in the downloaded mbox.gz This requires a "public-inbox-index --reindex" to be usable.
2020-08-23searchidx: index THREADID in Xapian
This is the `tid' column from over.sqlite3; and will be used for IMAP and JMAP search (among other things).
2020-08-20searchview: speed up search summary by ~10%
Instead of loading one article at-a-time from over.sqlite3, we can use SQL to mass-load IN (?,?, ...) all results with a single SQLite query. Despite SQLite being in-process and having no network latency, the reduction in SQL query executions from loading multiple rows at once speeds things up significantly. We'll keep the over->get_art optimizations from the previous commit, since it still speeds up long-lived responses, slightly.
2020-08-20searchview: use over.sqlite3 instead of Xapian docdata
This is a step towards improving kernel page cache hit rates by relying on over.sqlite3 for document data instead of Xapian. Some micro-optimization to over->get_art was required to maintain performance.
2020-08-20smsg: reduce utf8::decode call sites
Both callers of load_from_data call utf8::decode, so just do utf8::decode in load_from_data.
2020-08-08support setting No_COW on Perl <5.22
fileno(DIRHANDLE) only works on Perl 5.22+, so we need to use dirfd(3) ourselves from Inline::C (or rely on chattr(1) being installed). While we're at it, rename `set_nodatacow' to `nodatacow_fd' for consistency with `nodatacow_dir'.
2020-07-29searchidx: disable CoW for SQLite and Xapian under btrfs
SQLite and Xapian files are written randomly, thus they become fragmented under btrfs with copy-on-write. This leads to noticeable performance problems (and probably ENOSPC) as these files get big. lore/git (v2, <1GB) indexes around 20% faster with this on an ancient SSD. lore/lkml seems to be taking forever and I'll probably cancel it to save wear on my SSD. Unfortunately, disabling CoW also means disabling checksumming (and compression), so we'll be careful to only set the No_COW attribute on regeneratable data. We want to keep CoW (and checksums+compression) on git storage because current ref storage is neither checksummed nor compressed, and git streams pack output.
2020-07-25index+xcpdb: support --no-sync flag
This allows us to speed up indexing operations to SQLite and Xapian. Unfortunately, it doesn't affect operations using `xapian-compact' and the compactor API, since that doesn't seem to support Xapian::DB_NO_SYNC, yet.
2020-07-14over+msgmap: do not store filename after DBI->connect
SQLite already knows the filename internally, so avoid having it as a long-lived Perl SV to save some bytes when there's many inboxes and open DBs.
2020-07-14over: unset sqlite_unicode attribute
None of the human-readable strings stored in over.sqlite3 require UTF-8. Message-IDs do not, nor do the compressed Subject IDs (sid) we use for Subject-based threading. And the `ddd' (doc-data-deflated) column is of course binary data. This frees us of having to use SQL_BLOB for the `ddd', column, and will open the door for us to use dbh_new for Msgmap, too.
2020-06-27imap: EXAMINE: avoid potential race conditions
We need to rely on num_highwater for UIDNEXT since the highest `num' stored in over.sqlite3 may be rolled back if the most recent messages were spam. We also need to load the uo2m immediately on EXAMINE to ensure EXISTS responses are always consistent with regard to future updates.
2020-06-13nntpd+imapd: detect replaced over.sqlite3
For v1 inboxes (and possibly v2 in the future, for VACUUM), public-inbox-compact replaces over.sqlite3 with a new file. This currently doesn't need an extra inotify watch descriptor (or FD for kevent) at the moment, so it can coexist nicely for systems w/o IO::KQueue or Linux::Inotify2.
2020-06-13over: uid_range: remove LIMIT
The IMAP code already limits the range to UID_SLICE (50K), so that's about 1.6MB of of IVs for an ephemeral allocation that won't live beyond one iteration of the event loop.
2020-06-13imap: STATUS/EXAMINE: rely on SQLite overview
We can get exact values for EXISTS, UIDNEXT using SQLite rather than calculating off $ibx->mm->max ourselves. Furthermore, $ibx->mm is less useful than $ibx->over for IMAP (and for our read-only daemons in general) so do not depend on $ibx->mm outside of startup/reload to save FDs and reduce kernel page cache footprint.
2020-06-13over: get_art: use dbh->prepare_cached
This speeds up xt/imapd-validate.t by around 10% when used with an abandoned patch to remove ->query_xover. We may also depend on this further if we abandon storing doc_data in Xapian to save disk space.
2020-06-13imap: allow UID range search on timestamps
Since it seems somewhat common for IMAP clients to limit searches by sent Date: or INTERNALDATE, we can rely on the NNTP/WWW-optimized overview DB. For other queries, we'll have to depend on the Xapian DB.
2020-03-22rename PublicInbox::SearchMsg => PublicInbox::Smsg
Since the introduction of over.sqlite3, SearchMsg is not tied to our search functionality in any way, so stop confusing ourselves and future hackers by just calling it "PublicInbox::Smsg". Add a missing "use" in ExtMsg while we're at it.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2020-02-04over: simplify read-only vs read-write checking
No need to call ref() and do a string comparison. Add some extra tests using the {ReadOnly} attribute in DBI.pm.
2019-09-09run update-copyrights from gnulib for 2019
2019-01-08over: cull unneeded fields for get_thread
On a certain ugly /$INBOX/$MESSAGE_ID/T/ endpoint with 1000 messages in the thread, this cuts memory usage from 2.5M to 1.9M (which still isn't great, but it's a start).
2018-08-05view: distinguish strict and loose thread matches
The "loose" (Subject:-based) thread matching yields too many hits for some common subjects (e.g. "[GIT] Networking" on LKML) and causes thread skeletons to not show the current messages. Favor strict matches in the query and only add loose matches if there's space. While working on this, I noticed the backwards --reindex walk breaks `tid' on v1 repositories, at least. That bug was hidden by the Subject: match logic and not discovered until now. It will be fixed separately. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2018-06-19Tweak over.sqlite3 queries for sqlite < 3.8
The query planner in sqlite3 < 3.8 is not very clever, so when it sees num mentioned in the query filter, it decides not to use the fast idx_ts index and goes for the much slower autoindex. CentOS-7 still has sqlite-3.7, so loading the http landing page of a very large archive (LKML) was taking over 18 seconds, as oppposed to milliseconds on a system with sqlite-3.8 and above: $ time sqlite3 -line over.sqlite3 'SELECT ts,ds,ddd FROM over \ WHERE num > 0 ORDER BY ts DESC LIMIT 1000;' > /dev/null real 0m19.610s user 0m17.805s sys 0m1.805s $ sqlite3 -line over.sqlite3 'EXPLAIN QUERY PLAN SELECT ts,ds,ddd \ FROM over WHERE num > 0 ORDER BY ts DESC LIMIT 1000;' selectid = 0 order = 0 from = 0 detail = SEARCH TABLE over USING INDEX sqlite_autoindex_over_1 (num>?) (~250000 rows) However, if we slightly tweak the query per SQlite recommendations [1] by adding + to the num filter, we force it to use the correct index and see much faster performance: $ time sqlite3 -line over.sqlite3 'SELECT ts,ds,ddd FROM over \ WHERE +num > 0 ORDER BY ts DESC LIMIT 1000;' > /dev/null real 0m0.007s user 0m0.005s sys 0m0.002s $ sqlite3 -line over.sqlite3 'EXPLAIN QUERY PLAN SELECT ts,ds,ddd \ FROM over WHERE +num > 0 ORDER BY ts DESC LIMIT 1000;' selectid = 0 order = 0 from = 0 detail = SCAN TABLE over USING INDEX idx_ts (~1464303 rows) This appears to be the only place where this is needed in order to avoid running into this issue. As far as I can tell, this change has no impact on systems running newer sqlite3 (>= 3.8). .. [1] https://sqlite.org/optoverview.html#disqualifying_where_clause_terms_using_unary_ Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2018-04-18ensure SQLite and Xapian files respect core.sharedRepository
We can't have files with permissions inconsistent with what's in git objects.
2018-04-07over: avoid excessive SELECT
No need to read what we don't need into the Perl process. Fix some broken capitalization while we're at it.
2018-04-07psgi: ensure /$INBOX/$MESSAGE_ID/T/ endpoint is chronological
We only need to call get_thread beyond 1000 messages for fetching entire mboxes. It's probably too much for the HTML display otherwise.
2018-04-06www: favor reading more from SQLite, and less from Xapian
Favor simpler internal APIs this time around, this cuts a fair amount of code out and takes another step towards removing Xapian as a dependency for v2 repos.
2018-04-03mbox: remove remaining OFFSET usage in SQLite
We can use id_batch in the common case to speed up full mbox retrievals. Gigantic msets are still a problem, but will be fixed in future commits.
2018-04-03view: avoid offset during pagination
OFFSET in SQLite gets painful to deal with. Instead, rely on timestamps (from Received:) for pagination. This also sets us up for more precise Date searching in case we want it.
2018-04-03nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster
While SQLite is faster than Xapian for some queries we use, it sucks at handling OFFSET. Fortunately, we do not need offsets when retrieving sorted results and can bake it into the query. For inbox.comp.version-control.git (v1 Xapian), XOVER and XHDR are over 20x faster.
2018-04-02over: speedup get_thread by avoiding JOIN
JOIN operations on SQLite can be disasterously slow. This reduces per-message pages with the thread overview at the bottom of those pages from over 800ms to ~60ms. In comparison, the v1 code took around 70-80ms using Xapian on my machine.
2018-04-02www: rework query responses to avoid COUNT in SQLite
In many cases, we do not care about the total number of messages. It's a rather expensive operation in SQLite (Xapian only provides an estimate). For LKML, this brings top-level /$INBOX/ loading time from ~375ms to around 60ms on my system. Days ago, this operation was taking 800-900ms(!) for me before introducing the SQLite overview DB.
2018-04-02replace Xapian skeleton with SQLite overview DB
This ought to provide better performance and scalability which is less dependent on inbox size. Xapian does not seem optimized for some queries used by the WWW homepage, Atom feeds, XOVER and NEWNEWS NNTP commands. This can actually make Xapian optional for NNTP usage, and allow more functionality to work without Xapian installed. Indexing performance was extremely bad at first, but DBI::Profile helped me optimize away problematic queries.