about summary refs log tree commit homepage
path: root/t/lei.t
DateCommit message (Collapse)
2021-09-19lei config --edit: use controlling terminal
As with "lei edit-search", "lei config --edit" may spawn an interactive editor which works best from the terminal running script/lei. So implement LeiConfig as a superclass of LeiEditSearch so the two commands can share the same verification hooks and retry logic.
2021-09-13tests: add require_cmd, require curl when needed
t/v2mirror.t and t/lei-mirror.t are now skipped when curl is missing (instead of failing in appropriate places). A bunch of which() checks are updated to use require_cmd to avoid explicitly loading Spawn.
2021-09-12lei sucks: allow it to work without SQLite
And try to improve the message about Inline::C while we're at it, since Socket::Msghdr isn't widely-packaged, yet.
2021-08-19lei q: make --save the default
Since "lei up" is more often useful than not and incurs neglible overhead; enable --save by default and allow --no-save to work. This also fixes a long-standing when overwriting --output destinations with saved searches: dedupe data from previous searches are reset and no longer influences the new (changed) search, so results no longer go missing if two sequential invocations of "lei q --save" point to the same --output.
2021-04-28lei: avoid close(STD{IN,OUT,ERR}) in oneshot mode
This seems to fix the occasional "make check-run" failures I've been chasing. Some parts of our code assumes we can close($lei->{1}) and similar, which causes IO::Handle::autoflush to behave badly when STDOUT is the "select"-ed FH of the Perl process. Since oneshot mode is (hopefully) the uncommon case, we'll just accept the cost of extra FDs and minimize differences between lei in oneshot vs daemon mode.
2021-04-22lei import|convert: drop --no-kw aliases
Supporting --no-keywords and --no-flags aliases is harmful if users end up assuming "keywords:" and "flags:" are valid search prefixes (they're not).
2021-04-13lei q: start wiring up saved search
This will have a over.sqlite3 for content-based deduplication. It may exhibit ibxish methods, so serving a read-only (or even R/W) IMAP or instance or displaying HTML isn't outside the realm of possibility.
2021-04-01lei sucks: sub-command to aid bug reporting
It's a bit of an Easter egg, though it's not possible to hide those in Free Software... Anyways, it doesn't cost us an entry in %CMD of LEI.pm and anybody frustrated enough with lei just might type "lei sucks" on the command-line :>
2021-03-23lei: support -c <name>=<value> to overrides
It's a bit nasty, but seems to mostly work for debugging IMAP and NNTP commands.
2021-03-19lei: disallow "\n" in local externals paths
git 2.11 and earlier could not handle git directories with newlines in them, nor does libgit2 support them. Followup-to: d87dd0e679587043 ("config: reject `\n' in `inboxdir'")
2021-03-04lei q: import flags when clobbering/augmenting Maildirs
This will eventually be supported for other mail stores, but Maildir is the easiest to test and support, here. This lets us avoid a situation where flag changes get lost between search results.
2021-02-23lei: support "-C" to chdir in all sub commands
We'll also support "-C" at the end of most commands to give users a little more flexibility when building command-lines. This conflicts with "lei daemon-kill -CHLD", so that's special-cased since "-C" makes no sense with daemon-kill, anyways. Unlike "git show", the to-be-implemented "lei show" will diverge and enable "--find-copies[=<n>]" by default, so "-C[<n>]" won't be necessary.
2021-02-22t/lei*: drop $lei->(...) sub
lei() and lei_ok() are superior since they offer prototype checks and lei_ok() adds another check + description DRY-ness. The $lei sub was only bound to a variable since it was in t/lei.t and named subs don't work well with the key2sub() wrapper.
2021-02-10lei: replace "I:"-prefixed info messages with "#"
The "#" is what TAP <https://testanything.org/> uses, which is also consistent with what our (and many other) test suites emit.
2021-02-07lei: remove --mua-cmd alias for --mua
While "mua-cmd" may be more accurate, nobody is expected to type 4 extra characters. It's a needless ambiguity with no precedence or prior art to follow. Link: https://public-inbox.org/meta/20210206090119.GA14519@dcvr/
2021-02-07tests: split out lei-daemon.t from lei.t
This makes it easier for hackers to find daemon-specific tests and forces us to always test both daemon and oneshot mode.
2021-02-07t/lei-externals: split out into separate test
This is still overloaded with "lei q" stuff, but that's somewhat inevitable.
2021-02-07tests: add test_lei wrapper, split out t/lei-import.t
This will make it easier to maintain and test lei going forward, we need to be testing against existing read-only daemons. We'll also save ourselves some boilerplate by exporting all the Test::More methods directly in TestCommon We'll start using this by splitting out the latest "lei import" tests into its own file.
2021-02-07lei: fix completion of --no-kw / --no-keywords
We did not complete --no-* flags properly when multiple options are allowed.
2021-02-07lei: favor "keywords" over "flags", test --no-kw
JMAP brain says "keywords", IMAP brain says "flags"; JMAP brain wins today. Since "keywords" is a bit long, support "kw" as a shortcut since there's no conflict and "kw:" will be our search prefix for looking up messages by keyword.
2021-02-05lei import: initial implementation
Only tested with .eml files so far, but Maildir + IMAP will be supported.
2021-02-04t/lei: skip "lei q" tests on missing dependencies
... for now. It's probably possible to just use send() recv() without CMSG_* eventually.
2021-02-04lei q: support reading queries from stdin
This will be useful on shared machines when a user doesn't want search queries visible to other users looking at the ps(1) output or similar.
2021-02-04lei add-external: completion for existing URL basenames
Given the presence of one external on a certain host or prefix path, it's logical other inboxes would share a common prefix. For bash users, attempt to complete that using the "-o nospace" option of bash
2021-02-04lei: propagate curl errors, improve internal consistency
IO::Uncompress::Gunzip seems to be losing $? when closing PublicInbox::ProcessPipe. To workaround this, do a synchronous waitpid ourselves to force proper $? reporting update tests to use the new --only feature for testing invalid URLs. This improves internal code consistency by having {pkt_op} parse the same ASCII-only protocol script/lei understands. We no longer pass {sock} to worker processes at all, further reducing FD pressure on per-user limits.
2021-02-03lei: q: shell completion for --(include|exclude|only)
Because .onion URLs names are long!
2021-02-03lei q: emit progress and counting via PktOp
Sometimes it can be confusing for "lei q" to finish writing to a Maildir|mbox and not know if it did anything. So show some per-external progress and stats. These can be disabled via the new --quiet/-q switch. We differ slightly from mairix(1) here, as we use stderr instead of stdout for reporting totals (and we support parallel queries from various sources).
2021-01-29lei: complete option switch args
And add tests for existing completion cases
2021-01-26lei q: demangle and quiet curl output
curl(1) writes to stderr one byte-at-a-time (presumably for the progress bar). This ends up being unreadable on my terminal when parallel processes are trying to write error messages. So instead, we'll capture the output to a file and run 'tail -f' on it if --verbose is enabled. Since HTTP 404s from non-existent results are a common response, we'll ignore them and stay silent, matching behavior of local searches.
2021-01-26lei: reinstate JSON smsg output deduplication
This was accidentally clobbered completely in ("lei q: fix JSON overview with remote externals"). There are now more tests to prevent future regressions.
2021-01-24lei q: honor --no-local to force remote searches
This can be useful for testing remote behavior, or for augmenting local results. It'll also be possible to explicitly include/exclude externals via CLI switches (once names are decided).
2021-01-24lei add-external: don't allow non-existent directories
At least not yet, though we may support mirroring via git.
2021-01-23lei: support remote externals
Via curl(1), since that lets us easily use tor on a per-connection basis via LD_PRELOAD (torsocks) or proxy. We'll eventually support more curl options which can allow users to get past firewalls and deal with other odd network configurations.
2021-01-22lei: forget-external support with canonicalization
For proper matching, we'll do a better job canonicalizing URLs and path names for matching. Of course, users may edit the file outside of lei, so ensure we try both the canonicalized and as-is form provided by the user. I also don't think we'll need to store externals info in MiscIdx; just the config file is fine.
2021-01-21lei: dump and clear errors.log in daemon mode
Inspired by "dmesg -c", this should help users report bugs and avoids eating up $XDG_RUNTIME_DIR. Once lei is ready for release, hopefully the need for this should be few an far between, but shit happens.
2021-01-21lei: allow more mbox inode types
We may attempt to write an mbox to any terminal, block, or character device, not just regular files and FIFOs/pipes. The only thing that is known to not work is a directory. Sockets may be possible with some OSes (e.g. Plan 9) or filesystems. This fixes t/lei.t on FreeBSD 11.x
2021-01-21lei_overview: start implementing format detection
We'll need it for IMAP support, at least. Proper mbox family detection will be expensive, so deal with it later.
2021-01-21lei: test some likely errors due to misuse
Because user errors happen...
2021-01-21t/lei: fix double-running of socket test with oneshot
We split out t/lei-oneshot.t and t/lei.t so it's easier to isolate run-mode specific bugs and behavior and there's no reason to rerun the socket daemon tests.
2021-01-21lei q: fix augment of compressed mailboxes
We need to delay writing out the mailbox until the compressor process is up and running, so have startq wait a bit. This means we must create the pipe early and hand it off to the workers before augmenting, despite spawning the gzip/pigz/xz/bzip2 process after augment is complete.
2021-01-18lei: q: results output to Maildir and mbox* working
All the augment and deduplication stuff seems to be working based on unit tests. OpPipe is a nice general addition that will probably make future state machines easier.
2021-01-15lei: pass FD to CWD via cmsg, use fchdir on server
Perl chdir() automatically does fchdir(2) if given a file or directory handle since 5.8.8/5.10.0, so we can safely rely on it given our 5.10.1+ requirement. This means we no longer have to waste several milliseconds loading the Cwd.so and making stat() calls to ensure ENV{PWD} is correct and usable in the server. It also lets us work in directories that are no longer accessible via pathname.
2021-01-14lei: do not unlink socket path at exit
This matches existing -httpd/-nntpd/-imapd daemon behavior. From what I can recall, it is less racy for the process doing bind(2) to unlink it if stale.
2021-01-14lei: test SIGPIPE, stop xsearch workers on client abort
The new test ensures consistency between oneshot and client/daemon users. Cancelling an in-progress result now also stops xsearch workers to avoid wasted CPU and I/O. Note the lei->atfork_child_wq usage changes, it is to workaround a bug in Perl 5: http://nntp.perl.org/group/perl.perl5.porters/258784 <CAHhgV8hPbcmkzWizp6Vijw921M5BOXixj4+zTh3nRS9vRBYk8w@mail.gmail.com> This switches the internal protocol to use SOCK_SEQPACKET AF_UNIX sockets to prevent merging messages from the daemon to client to run pager and kill/exit the client script.
2021-01-12lei_xsearch: transfer 4 FDs internally, drop IO::FDPass
It's easier to make the code more generic by transferring all four FDs (std(in|out|err) + socket) instead of omitting stdin. We'll be reading from stdin on some imports, and possibly outputting to stdout, so omitting stdin now would needlessly complicate things. The differences with IO::FDPass "1" code paths and the "4" code paths used by Inline::C and Socket::MsgHdr are far too much to support and test at the moment.
2021-01-12cmd_ipc: send FDs with buffer payload
For another step in in syscall reduction, we'll support transferring 3 FDs and a buffer with a single sendmsg/recvmsg syscall using Socket::MsgHdr if available. Beyond script/lei itself, this will be used for internal IPC between search backends (perhaps with SOCK_SEQPACKET). There's a chance this could make it to the public-facing daemons, too. This adds an optional dependency on the Socket::MsgHdr package, available as libsocket-msghdr-perl on Debian-based distros (but not CentOS 7.x and FreeBSD 11.x, at least). Our Inline::C version in PublicInbox::Spawn remains the last choice for script/lei due to the high startup time, and IO::FDPass remains supported for non-Debian distros. Since the socket name prefix changes from 3 to 4, we'll also take this opportunity to make the argv+env buffer transfer less error-prone by relying on argc instead of designated delimiters.
2021-01-12lei query + pagination sorta working
Parallelism and interactivity with pager + SIGPIPE needs work; but results are shown and phrase search works without shell users having to apply Xapian quoting rules on top of standard shell quoting.
2021-01-06lei: use client env as-is, drop daemon-env command
There may be subtle misbehaviours when mixing the existing daemon env and the client-supplied env. Just do the simplest thing and use the client env as-is. We'll also start the ->event_step callback since we'll need to remember some things for long-lived commands.
2021-01-04lei: prefer IO::FDPass over our Inline::C recv_3fds
While our recv_3fds() implementation is more efficient syscall-wise, loading Inline takes nearly 50ms on my machine even after Inline::C memoizes the build. The current ~20ms in the fast path is barely acceptable to me, and 50ms would be unusable. Eventually, script/lei may invoke tcc(1) or cc(1) directly in the fast path, but it needs @INC for the slow path, at least. We'll encode the number of FDs into the socket name allow parallel installations, for now.
2021-01-03send and receive all 3 FDs at once
We'll always be transferring stdin, stdout, and stderr together for lei. Perhaps I lack imagination or foresight, but I can't think of a reason to send more or less FDs.