about summary refs log tree commit homepage
path: root/t/lei_xsearch.t
DateCommit message (Collapse)
2021-08-11lei: attempt to canonicalize away "/../" pathnames
As documented, File::Spec->canonpath does not canonicalize "/../". While we want to do our best to preserve symlinks in pathnames, leaving "/../" can mislead our inotify|kqueue usage.
2021-03-21lei q: trim JSON output
Stop showing `docid' since it's not useful with shards. `bytes' and `lines' are probably noise, but maybe could be visible in some "fuller" view. v2: t/lei_xsearch: fix warnings from {docid} removal
2021-03-21lei: tie ALE lifetime to config file
This should make a future change to "lei import" work more nicely, since we'll be needing ALE to vivify external-only messages upon explicit "lei import".
2021-03-21lei: All Local Externals: bare git dir for alternates
This will be used for keyword (and label) storage for externals. We'll be using this to ensure we don't redundantly auto-import messages into lei/store if they're already in a local external (they can still be imported explicitly via "lei import").
2021-03-15t/lei_xsearch: use create_inbox
This ends up being significantly faster when confined to a slow TMPDIR.
2021-03-15t/lei_xsearch: use create_inbox
Less code and noticeably faster when TMPDIR is on an SSD
2021-03-13searchidx: fix -Lmedium for IDs and filenames
This fixes "m:", "l:", "f:", "t:", "c:", "dfn:", and "n:" search prefixes under indexlevel=medium when mixed with indexlevel=full inboxish. We need positional data for Message-IDs, List-Id, email addresses and filenames for exact matches, though we still want to support wildcards. Fortunately the storage cost is still small as these prefixes tend to be small compared to message bodies. These are NOT boolean terms since wildcard support and partial matching is desired.
2021-02-04tests: guard against missing DBD::SQLite
The features we use for SharedKV could probably be implemented with GDBM_File or SDBM_File, but that doesn't seem worth it at the moment since we depend on SQLite elsewhere.
2021-01-23lei: move external vivification to xsearch
This seems like a better place to put it given upcoming URI support, which starts in this commit.
2021-01-12lei query + pagination sorta working
Parallelism and interactivity with pager + SIGPIPE needs work; but results are shown and phrase search works without shell users having to apply Xapian quoting rules on top of standard shell quoting.
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2021-01-01lei: rename "extinbox" => "external"
The words "extinbox" and "extindex" are too close and easy to confuse with the other. Rename "extinbox" to "external", since these could be IMAP, JMAP or other non-public-inbox search APIs. Link: https://public-inbox.org/meta/20201226112649.GB6226@dcvr/
2020-12-31lei_xsearch: cross-(inbox|extindex) search
While a single extindex combines multiple inboxes into a single search index, extindex still requires up-front indexing on items which can be searched. XSearch has no on-disk footprint itself and uses Xapian DBs of existing publicinbox and extindex ("extinbox") exclusively. XSearch still suffers from the multi-shard Xapian scalability problems which led to the creation of extindex, but I expect the number of shards to remain relatively low. I envision users hosting public-inbox instances on their workstations will only have two extindex combined by this, one read-only extindex for serving public archives, and one read-write extindex managed by LeiStore for private mail.