about summary refs log tree commit homepage
path: root/xt
DateCommit message (Collapse)
2022-09-10xt: fold perf-obfuscate into perf-msgview, future-proof
perf-obfuscate was close enough to perf-msgview that it only required setting the `obfuscate' field of the inbox. Then update perf-msgview to account for upcoming internal changes. The current use of {obuf} and concat ops results in excessive scratchpad space and I may be able to even get speedups by avoiding concat ops.
2022-09-04prepare HTML rendering maintainer tests for upcoming changes
There'll be a number of upcoming changes to HTML rendering of messages to hopefully reduce memory usage and speedups by writing out to the gzip buffer earlier. Update the tests now so it'll be easier to test before and after results.
2022-08-23xt/solver: improve diagnostics
I'm making some tweaks to solver and will probably find extra output useful, and also update to v5.12 while we're at it.
2022-07-30tests: maintainer test for using mpop
This ought to be a good stress test to ensure our POP3 implementation works against the POP3 client I've found.
2022-07-24add xt/mem-nntpd-tls maintainer test
This ensures memory usage is reasonable when DEFLATE and TLS are enabled. It's also our only coverage for NNTP COMPRESS since Net::NNTP has yet to implement compression support: https://rt.cpan.org/Public/Bug/Display.html?id=129967
2022-07-24xt/mem-imapd-tls: update aliases to DSdeflate subs
Fixes: 23af251dd607c4e7 (imap+nntp: share COMPRESS implementation, 2022-07-23)
2022-07-23imap+nntp: share COMPRESS implementation
Their code was nearly identical to begin with, so save some memory in -netd and disk space for all of our tarball/distro users, at least. And I seem to have used multiple inheritance successfully, here, maybe...
2021-10-28xt/net_writer_imap: test "lei convert" w/ IMAP source
I just did a double-take and nearly thought authentication was broken while reading LeiConvert.pm. Add a comment in LeiConvert.pm to clarify things, too.
2021-10-22lei export-kw: don't recreate deleted IMAP folders
In case an IMAP folder is deleted, just set an error and ignore it rather than creating an empty folder which we attempt to export keywords to for non-existent messages.
2021-10-13xt/perf-msgview: drop unnecessary use_ok
require_mods covers it, and we're not testing Plack itself.
2021-10-08git: use async_wait_all everywhere
Some code paths may use maximum size checks, so ensure any checks are waited on, too.
2021-10-06xt/perf-msgview: modernize, support TEST_BLOB
This helped me quickly reproduce a bug in Encode[1] and will help me determine performance implications of workarounds for the aforementioned bug. [1] https://rt.cpan.org/Public/Bug/Display.html?id=139622
2021-10-01ds: simplify signalfd use
Since signalfd is often combined with our event loop, give it a convenient API and reduce the code duplication required to use it. EventLoop is replaced with ::event_loop to allow consistent parameter passing and avoid needlessly passing the package name on stack. We also avoid exporting SFD_NONBLOCK since it's the only flag we support. There's no sense in having the memory overhead of a constant function when it's in cold code.
2021-09-27xt/net_writer_imap: env knobs for compress/debug/proxy
It can be useful to test with some of these, but we can't enable them universally for all servers (and debug + compress is gross)
2021-09-19xt: add fsck script over over.sqlite3
I'm not sure what caused it, but I've noticed two missing messages that failed from "lei up" on an https:// external; and I've also seen some duplicates in the past (which I think I fixed...).
2021-09-14uri_imap: handle '/' as an IMAP hierarchy separator
Untested at the moment(*), but we were inadvertantly truncating mailbox names with '/' due to our work-in-progress handling of "/;UID=$NUM" parameter. (*) strangely, my dovecot instance doesn't allow '/' by default, so the change to xt/net_writer-imap.t is untested. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210914175025.eq7s2shkc323itaf@meerkat.local/
2021-09-07lei up: support --all for IMAP folders
Since "lei up" is expected to be a heavily-used command, better support for IMAP seems like a reasonable idea. This is inefficient since we waste an IMAP(S) TCP connection since it dies when an auth-only LeiUp worker process dies, but it's better than not working at all, right now.
2021-09-07xt/net_writer_imap: test "lei up" on single IMAP output
That's the minimum, at least...
2021-08-28get rid of unnecessary bytes::length usage
The only place where we could return wide characters with -httpd was the raw $INBOX_DIR/description text, which is now converted to octets. All daemon (HTTP/NNTP/IMAP) sockets are opened in binary mode, so length() and bytes::length() are equivalent on reads. For socket writes, any non-octet data would warn about wide characters and we are strict in warnings with test_httpd. All gzipped buffers are also octets, as is PublicInbox::Eml->body, and anything from PerlIO objects ("git cat-file --batch" output, filesystems), so bytes::length was unnecessary in all those places.
2021-05-23lei export-kw: relax IMAP URL matching
It's unreasonable to expect UIDVALIDITY= to be specified in command-line arguments. We'll also check for cases without "$USER@" or ";AUTH=", since we accept those forms on the command-line.
2021-05-23lei export-kw: support exporting keywords to IMAP
We support writing to IMAP stores in other places (just like Maildir), and it's actually less complex for us to write to IMAP. Neither usability nor performance is ideal, but usability will be addressed in the next commit to relax CLI argument checking. Performance is poor due to the synchronous Mail::IMAPClient API and will need to be addressed with pipelining sometime further in the future.
2021-05-04treewide: update to v3 Tor onions
v2 onions are insecure, deprecated and going away. v3 names are unfortunately longer and more difficult to remember, but should be more resistant to attack than v2 ones.
2021-05-03net_writer: use "FLAGS.SILENT" to set keywords
Instead of "+FLAGS.SILENT" which merely adds to the keywords. We store all keywords together, so it's unlikely we will rely on the "+FLAGS.SILENT" or "-FLAGS.SILENT".
2021-05-01xt/lei-onion-convert: test for NNTP+IMAP onions
These tests require a running Tor instance (defaulting to 127.0.0.1:9050) and Internet connectivity, but otherwise work pretty well.
2021-04-30net_reader: Net::NNTP --proxy=socks5h:// support
Since Net::NNTP doesn't support Socket or RawSocket options/accessors like Mail::IMAPClient does; we must perform localized @ISA manipulation and massage Net::NNTP into using IO::Socket::Socks rather than IO::Socket::IP. This is a bit fragile, but Net::Cmd and Net::NNTP rarely change; and I keep an eye on them, anyways.
2021-04-22lei: flesh out `forwarded' kw support for Maildir and IMAP
Maildir and IMAP can both handle `forwarded'. Ensure we don't lose `forwarded' when reading from stores which do not support it, but ensure we can set it when reading from IMAP and Maildir stores.
2021-04-20lei-sigpipe: update and move test from xt => t
We have "lei import" and better test infrastructure for lei, now, so we can more easily test SIGPIPE without relying on an already-configured instance.
2021-04-11www: do not obfuscate addresses in URLs
As they are likely Message-IDs. If an email address ends up in a URL, then it's likely public, so there's even less reason to obfuscate that particular address. [km: add xt/perf-obfuscate.t] [ew: modernize perf test (5.10.1), use diag instead of print] This version of the patch avoids the massive slowdown noted by Kyle in <https://public-inbox.org/meta/87wnt9or6t.fsf@kyleam.com/>. Performance remains roughly the same, if not slightly faster (which may be due to me testing this on a busy server). Results from xt/perf-obfuscate.t against 6078 messages on a local mirror of <https://public-inbox.org/meta/>: before: 6.67 usr + 0.04 sys = 6.71 CPU after: 6.64 usr + 0.04 sys = 6.68 CPU Reported-by: Kyle Meyer <kyle@kyleam.com> Helped-by: Kyle Meyer <kyle@kyleam.com> Link: https://public-inbox.org/meta/87a6q8p5qa.fsf@kyleam.com/
2021-04-03xt/lei-auth-fail: test more failure cases
Because failures are often overlooked, unfortunately.
2021-03-31lei: fix IMAP auth failure handling
We must use the $ops hashref returned by lei->workers_start, since it's modified to include extra handlers for auth failures and whatnot. Fixes: 954581b8e575966a ("lei: simplify PktOp callers")
2021-03-28test_common: require_mods bundles
This makes it easier to manage test dependencies on systems where optional stuff isn't installed. This fixes some lei tests which didn't check for Plack before starting -httpd, and ensures Parse::RecDescent is available for -imapd in case Mail::IMAPClient stops using it.
2021-03-19xt/create-many-inboxes: adjust for detect_nproc, no fsync
detect_nproc is in the IPC module, now; and we can safely disable fsync when creating test data. And "modernize" up to 5.10.1 while we're at it. The use fsync was causing this to run for hours instead of minutes since I forgot to use eatmydata.
2021-03-12msg_part_text: discover text in application/octet-stream
Some poorly-configured MUAs will send application/octet-stream even for text-only attachments. We can't make expect all MUAs are configured with proper MIME types, and there is plenty of historical mail that falls into this unfortunate criteria. v2: simplify the check and ensures returned text is Perl "utf8"
2021-03-10watch: IMAP: ignore \Deleted and \Draft messages
This matches existing Maildir behavior, as trash and draft messages have little reason to be exposed publicly.
2021-03-09lei q: remove angle brackets around Message-IDs
They're unnecessary visual noise, and angle brackets don't always work as intended when going through Xapian's query parser. Since we already use "m:" and "refs:" instead of the actual header names, it should be obvious we're at liberty to abbreviate such things Link: https://public-inbox.org/meta/20210304184348.GA19350@dcvr/
2021-03-05search: use "z:" instead of "bytes:" prefix
So far, searching by size has never been publicly documented, and IMHO, of questionable utility. In any case, "z:" is what mairix(1) uses, so it may be familiar to existing mairix users (I've never used this prefix myself). So far, this prefix is only used internally in tests and in auto-translated queries from IMAP; thus this incompatible change is unlikely to affect anyone.
2021-03-04lei q: support --import-augment for IMAP
IMAP is similar to Maildir and we can now preserve keyword updates done on IMAP folders.
2021-02-26lei convert: support IMAP output and "-F eml" inputs
eml ("message/rfc822" MIME type) is supported by "lei import", so it probably makes sense to support via convert, at least for tests. And IMAP support is supported in "lei q -o $MFOLDER", so this only required renaming {nrd} => {net} and initializing outputs before augment preparation (creating the IMAP folder)
2021-02-21net_reader: use and accept URIimap objects in more places
This flexibility should save us some code down-the-line.
2021-02-21lei q: support IMAP/IMAPS --output destinations
Augment (and dedupe) aren't parallel, yet, so its more sensitive to high-latency networks.
2021-02-19URIimap: overload "" to ->as_string
This interpolation is used by the upstream URI package and we rely on it elsewhere for HTTP(S) URIs, so save ourselves some surprises down the line.
2021-02-19net_writer: start implementing IMAP write support
Requiring TEST_IMAP_WRITE_URL to be set to a writable IMAP server URL isn't ideal, but it works for now until we have time to setup a mock dovecot/cyrus/etc... instance for testing.
2021-02-19tests: require Mail::IMAPClient for IMAP tests
All of our current IMAP code relies on Mail::IMAPClient at the moment, so ensure we skip those tests on systems without that module.
2021-02-18lei: check for IMAP auth errors
We need to ensure authentication failures and error codes get propagated to the parent process(es) properly. v2: update MANIFEST v3: LeiAuth.pm ->_lei_cfg bit moved to a previous commit
2021-02-08tests: favor IPv6
IPv4 gets plenty of real-world coverage, and apparently there's Debian buildd hosts which lack IPv4(*). So ensure everything can work on IPv6 and not cause problems for odd setups. (*) https://bugs.debian.org/979432
2021-02-07ipc: wq_do => wq_io_do
We will have a ->wq_do that doesn't pass FDs for I/O.
2021-02-03lei q: do not leave temporary files after oneshot exit
Avoid on-stack shortcuts which may prevent destructors from firing since we're not inside the event loop. We'll also tidy up the unlink mechanism in LeiOverview while we're at it.
2021-02-03lei q: emit progress and counting via PktOp
Sometimes it can be confusing for "lei q" to finish writing to a Maildir|mbox and not know if it did anything. So show some per-external progress and stats. These can be disabled via the new --quiet/-q switch. We differ slightly from mairix(1) here, as we use stderr instead of stdout for reporting totals (and we support parallel queries from various sources).
2021-02-01sharedkv: use lock_for_scope_fast
This allows us to avoid repeated open() and close() syscalls and speeds up the new xt/stress-sharedkv.t maintainer test by roughly 7%.
2021-01-26use defined-or in a few more places
Mainly around fork() calls, but some nearby places as well.