Date | Commit message (Collapse) |
|
When retrieving loose (Subject) matches for a thread, we wanted
the most recent matches in reverse chronological order.
However, when displaying the /T/ endpoint generating the thread
skeleton, we prefer ascending chronological order to match the
flow of the conversation.
Reported-by: Askar Safin <safinaskar@gmail.com>
Link: https://public-inbox.org/meta/CAPnZJGAqsh8ZhPaCAy5M2NZVNcWrr_Hr94t32VXiyiTXwD9jRQ@mail.gmail.com/
|
|
These may've been causing strange errors[1] in t/imapd.t from
the -watch daemon, such as:
Cannot copy to HASH in scalar assignment ../PublicInbox/Over.pm
in the Over->dbh() sub. I've only noticed this failure on
FreeBSD 13.2 (Perl 5.32.1, DBD::SQLite 1.72 (bundled SQLite
3.39.4), DBI 1.643) so far, so it could also be something to do
with the versions used and/or memory layout differences
with libc or build toolchain.
|
|
We must not filter out seen docids from the mset; but only with
the result of over->expand_thread.
|
|
This allows filtering the contents of any existing thread using
a search query. It uses the existing THREADID column in Xapian
so we can internally add a Xapian OP_FILTER to the results.
This new functionality is orthogonal to the existing `t=1'
parameter which gives mairix-style thread expansion. It doesn't
make sense to use `t=1' with this functionality, but it's not
disallowed, either.
The indentation change in Over->next_by_mid is to ensure
DBI->prepare_cached can share across both ->next_by_mid
and ->mid2tid.
I also noticed the existing regex for `POST /$INBOX/?x=m&q=' was
allowing extra characters. With an added \z, it's now as strict
was originally intended and AFAIK nothing was generating invalid
URLs for it
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/aaniyhk7wfm4e6m5mbukcrhevzoc6ftctyrfwvmz4fkykwwtlj@mverfng6ytas/T/
|
|
There's no need to create two intermediate arrays when we can
modify the existing arrayref.
|
|
btrfs is Linux-only at the moment (and likely to remain that way
for practical purposes). So rely on Linux ABI stability and use
the `syscall' and `ioctl' perlops rather than relying on Inline::C.
Inline::C (and gcc||clang) are monstrous dependencies which we
can't expect users to have.
This makes supporting new architectures more difficult, but new
architectures come along rarely and this reduces the burden for
the majority of Linux users on popular architectures (while
still avoiding the distribution of pre-built binaries).
Link: https://public-inbox.org/meta/YbCPWGaJEkV6eWfo@codewreck.org/
|
|
This required some tweaking of xref3 indices in over.sqlite3,
but the end result is it brings no-op "--reindex --fast --all"
checks down to roughly 20 minutes (from 30-40 minutes) on
lore/all.
This is faster because a bunch of small SQLite queries are still
slower en-mass than a bunch of perlops. Despite the lack of IPC
overhead, crossing .so boundaries and repeating lookups over
btrees is still slower than doing the same with Perl hash tables.
|
|
We'll set nodatacow when detecting existing but empty
files, and also their directories in more cases (for
auxiliary -wal, -journal, -shm files). Hopefully
this keeps performance reasonable on CoW FSes.
|
|
Since we favor ->over in WWW and IMAP, move this method to
->over to reduce open files in common cases.
This fixes the /$EXTINDEX_NAME/all.mbox.gz endpoint for extindex
entries (which may get expensive...).
|
|
On a 4-core CPU, this speeds up "lei import" on a largish IMAP
inbox with 75K messages from ~21 minutes down to 40s.
Parallelizing with the new LeiImportKw WQ worker class gives a
near-linear speedup and brought the runtime down to ~5:40.
The new idx_fid_uid index on the "fid" and "uid" columns of
blob2num in mail_sync.sqlite3 brought us the final speedup.
An additional index on over.sqlite3#xref3(oidbin) did not help,
since idx_nntp already exists and speeds up the new ->oidbin_exists
internal API.
I initially experimented with a separate "lei import-kw" command
but decided against it since it's useless outside of IMAP+JMAP
and would require extra cognitive overhead for both users and
hackers. So LeiImportKw is just a WQ worker used by "lei import"
and not its own user-visible command.
v2: fix ikw_done_wait arg handling (ugh, confusing API :x)
|
|
Keyword storage for external-only messages was preventing
messages from being explicitly imported. Teach lei_store
to vivify keyword-only entries into fully-indexed messages
on import.
|
|
"lei q" now preserves changes per-message keywords across
invocations when it's --output (Maildir or mbox) is reused
(with or without --augment).
In the future, these changes will be monitored via inotify,
EVFILT_VNODE or IMAP IDLE, too.
Unfortunately, this currently prevents "lei import" from ever
importing a message that's in an external. That will be fixed
in a future change.
|
|
Using "make update-copyrights" after setting GNULIB_PATH in my
config.mak
|
|
--rethread is useful for dealing with bugs and behaves
just like it does with current inboxes.
This is in case our content deduplication logic changes for
whatever reason and causes previously merged messages to be
considered "different". As with v2, this won't allow us to
merge messages in a way that allows deduplicating messages which
were previously considered different, but v2 inboxes do not
allow that, either.
In other words, this makes the --reindex and --rethread
switches of -extindex match the behavior of v2 -index.
|
|
While unlikely to happen, it may be possible for messages
from the same inbox to get indexed multiple times. Provide
consistent results in this case for ease-of-testing.
|
|
While "public-inbox-extindex --gc" invocations try to ensure
proper ordering, it is still possible for users to change
the `inboxes' tables via sqlite3(1) or similar means. So
show a "missing://ibx_id=$ibx_id" placeholder to avoid undefined
variable warnings.
URLs such as "imaps://..." will eventually be supported as
eidx_keys, so having a URL-like "missing://" as a placeholder
probably makes sense.
|
|
We may not end up storing xref3 data in Xapian, actually.
This will make indexlevel=basic possible, and along with
--sequential-shard indexing support for slow storage.
Making oidmap a separate table seems unnecessary, too, so
fold it into the xref3 table since it's unlikely a git blob
will be responsible for multiple xref3 rows.
|
|
Sometimes it's useful to quickly get to threads and messages
which are contemporaries of the current thread/message being
focused on. This hopefully improves navigation by making:
a) the top line (where $INBOX_DIR/description) is shown
a link to the latest topics in search results and
per-thread/per-message views.
b) providing a link to contemporaries ("~YYYY-MM-DD") at
around the thread overview skeleton area for per-thread
and per-message views
|
|
v5.10.1 lets us use the lighter parent.pm instead of base.pm,
and we'll rely on the shebang to enable warnings (or not).
While we're in the area, drop a no-longer-necessary import for
PublicInbox::Search, since OverIdx doesn't require search.
|
|
As noted in commit 87dca6d8d5988c5eb54019cca342450b0b7dd6b7
("www: rework query responses to avoid COUNT in SQLite"),
COUNT on many rows is expensive on big SQLite DBs.
We've already stopped using that code path long ago in WWW
while -imapd and -nntpd never used it. So we'll adjust our
remaining test cases to not need it, either.
|
|
Since we got rid of over->connect, `disconnect' no longer pairs
with it. So name it after the `close(2)' syscall it ultimately
issues.
|
|
`->connect' is confused with the perlfunc for the `connect(2)'
syscall, and also `DBI->connect'. Since SQLite doesn't use
sockets, the word "connect" needlessly confuses me. Give
it a short name to match the field name we use for it, which
also matches the variable name used by the DBI(3pm) and
DBD::SQLite(3pm) manpages.
|
|
WAL actually seems to have ideal locking characteristics given
concurrency problems I'm experiencing with --reindex running
in parallel with expensive read-only SQLite queries:
<https://public-inbox.org/meta/20200825001204.GA840@dcvr/>
Unfortunately, we cannot blindly use WAL while preserving
compatibility with existing setups nor our guarantees that
read-only daemons are indeed "read-only".
However, respect an user's the choice to set WAL on their
own if they're comfortable with giving -nntpd/-httpd/-imapd
processes write permission to the directory storing SQLite DBs.
|
|
This file gets truncated anyhow, so it won't fragment.
|
|
Finally, the addition of THREADID for collapsing results
in Xapian lets us emulate the "mairix --threads" feature.
That is, instead of returning only the matching messages,
the entire thread is included in the downloaded mbox.gz
This requires a "public-inbox-index --reindex" to be usable.
|
|
This is the `tid' column from over.sqlite3; and will be used for
IMAP and JMAP search (among other things).
|
|
Instead of loading one article at-a-time from over.sqlite3, we
can use SQL to mass-load IN (?,?, ...) all results with a single
SQLite query. Despite SQLite being in-process and having no
network latency, the reduction in SQL query executions from
loading multiple rows at once speeds things up significantly.
We'll keep the over->get_art optimizations from the previous
commit, since it still speeds up long-lived responses, slightly.
|
|
This is a step towards improving kernel page cache hit rates by
relying on over.sqlite3 for document data instead of Xapian.
Some micro-optimization to over->get_art was required to
maintain performance.
|
|
Both callers of load_from_data call utf8::decode, so just
do utf8::decode in load_from_data.
|
|
fileno(DIRHANDLE) only works on Perl 5.22+, so we need to use
dirfd(3) ourselves from Inline::C (or rely on chattr(1) being
installed).
While we're at it, rename `set_nodatacow' to `nodatacow_fd'
for consistency with `nodatacow_dir'.
|
|
SQLite and Xapian files are written randomly, thus they become
fragmented under btrfs with copy-on-write. This leads to
noticeable performance problems (and probably ENOSPC) as these
files get big.
lore/git (v2, <1GB) indexes around 20% faster with this on an
ancient SSD. lore/lkml seems to be taking forever and I'll
probably cancel it to save wear on my SSD.
Unfortunately, disabling CoW also means disabling checksumming
(and compression), so we'll be careful to only set the No_COW
attribute on regeneratable data. We want to keep CoW (and
checksums+compression) on git storage because current ref
storage is neither checksummed nor compressed, and git streams
pack output.
|
|
This allows us to speed up indexing operations to SQLite
and Xapian.
Unfortunately, it doesn't affect operations using
`xapian-compact' and the compactor API, since that doesn't seem
to support Xapian::DB_NO_SYNC, yet.
|
|
SQLite already knows the filename internally, so avoid having it
as a long-lived Perl SV to save some bytes when there's many
inboxes and open DBs.
|
|
None of the human-readable strings stored in over.sqlite3
require UTF-8. Message-IDs do not, nor do the compressed
Subject IDs (sid) we use for Subject-based threading. And the
`ddd' (doc-data-deflated) column is of course binary data.
This frees us of having to use SQL_BLOB for the `ddd', column,
and will open the door for us to use dbh_new for Msgmap, too.
|
|
We need to rely on num_highwater for UIDNEXT since the
highest `num' stored in over.sqlite3 may be rolled back
if the most recent messages were spam.
We also need to load the uo2m immediately on EXAMINE to ensure
EXISTS responses are always consistent with regard to future
updates.
|
|
For v1 inboxes (and possibly v2 in the future, for VACUUM),
public-inbox-compact replaces over.sqlite3 with a new file.
This currently doesn't need an extra inotify watch descriptor
(or FD for kevent) at the moment, so it can coexist nicely for
systems w/o IO::KQueue or Linux::Inotify2.
|
|
The IMAP code already limits the range to UID_SLICE (50K),
so that's about 1.6MB of of IVs for an ephemeral allocation
that won't live beyond one iteration of the event loop.
|
|
We can get exact values for EXISTS, UIDNEXT using SQLite
rather than calculating off $ibx->mm->max ourselves.
Furthermore, $ibx->mm is less useful than $ibx->over for IMAP
(and for our read-only daemons in general) so do not depend on
$ibx->mm outside of startup/reload to save FDs and reduce kernel
page cache footprint.
|
|
This speeds up xt/imapd-validate.t by around 10% when used with
an abandoned patch to remove ->query_xover. We may also depend
on this further if we abandon storing doc_data in Xapian to save
disk space.
|
|
Since it seems somewhat common for IMAP clients to limit
searches by sent Date: or INTERNALDATE, we can rely on
the NNTP/WWW-optimized overview DB.
For other queries, we'll have to depend on the Xapian DB.
|
|
Since the introduction of over.sqlite3, SearchMsg is not tied to
our search functionality in any way, so stop confusing ourselves
and future hackers by just calling it "PublicInbox::Smsg".
Add a missing "use" in ExtMsg while we're at it.
|
|
I didn't wait until September to do it, this year!
|
|
No need to call ref() and do a string comparison. Add some
extra tests using the {ReadOnly} attribute in DBI.pm.
|
|
|
|
On a certain ugly /$INBOX/$MESSAGE_ID/T/ endpoint with 1000
messages in the thread, this cuts memory usage from 2.5M to 1.9M
(which still isn't great, but it's a start).
|
|
The "loose" (Subject:-based) thread matching yields too many
hits for some common subjects (e.g. "[GIT] Networking" on LKML)
and causes thread skeletons to not show the current messages.
Favor strict matches in the query and only add loose matches
if there's space.
While working on this, I noticed the backwards --reindex walk
breaks `tid' on v1 repositories, at least. That bug was hidden
by the Subject: match logic and not discovered until now. It
will be fixed separately.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
The query planner in sqlite3 < 3.8 is not very clever, so when it sees
num mentioned in the query filter, it decides not to use the fast idx_ts
index and goes for the much slower autoindex. CentOS-7 still has
sqlite-3.7, so loading the http landing page of a very large archive
(LKML) was taking over 18 seconds, as oppposed to milliseconds on a
system with sqlite-3.8 and above:
$ time sqlite3 -line over.sqlite3 'SELECT ts,ds,ddd FROM over \
WHERE num > 0 ORDER BY ts DESC LIMIT 1000;' > /dev/null
real 0m19.610s
user 0m17.805s
sys 0m1.805s
$ sqlite3 -line over.sqlite3 'EXPLAIN QUERY PLAN SELECT ts,ds,ddd \
FROM over WHERE num > 0 ORDER BY ts DESC LIMIT 1000;'
selectid = 0
order = 0
from = 0
detail = SEARCH TABLE over USING INDEX sqlite_autoindex_over_1 (num>?) (~250000 rows)
However, if we slightly tweak the query per SQlite recommendations [1]
by adding + to the num filter, we force it to use the correct index
and see much faster performance:
$ time sqlite3 -line over.sqlite3 'SELECT ts,ds,ddd FROM over \
WHERE +num > 0 ORDER BY ts DESC LIMIT 1000;' > /dev/null
real 0m0.007s
user 0m0.005s
sys 0m0.002s
$ sqlite3 -line over.sqlite3 'EXPLAIN QUERY PLAN SELECT ts,ds,ddd \
FROM over WHERE +num > 0 ORDER BY ts DESC LIMIT 1000;'
selectid = 0
order = 0
from = 0
detail = SCAN TABLE over USING INDEX idx_ts (~1464303 rows)
This appears to be the only place where this is needed in order to avoid
running into this issue.
As far as I can tell, this change has no impact on systems running newer
sqlite3 (>= 3.8).
.. [1] https://sqlite.org/optoverview.html#disqualifying_where_clause_terms_using_unary_
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We can't have files with permissions inconsistent with what's
in git objects.
|
|
No need to read what we don't need into the Perl process.
Fix some broken capitalization while we're at it.
|
|
We only need to call get_thread beyond 1000 messages for
fetching entire mboxes. It's probably too much for the HTML
display otherwise.
|