about summary refs log tree commit homepage
path: root/t
DateCommit message (Collapse)
2021-09-17lei refresh-mail-sync: implicitly remove missing folders
There's no point in keeping mail_sync.sqlite3 entries around if the folder is gone. We do keep saved-search configs around, however, since somebody may decide to blow away a search and start over.
2021-09-17lei refresh-mail-sync: replace prune-mail-sync
Merely pruning mail synchronization information was insufficient for Maildir: renames are common in Maildir and we need to detect them after-the-fact when lei-daemon isn't running. Running this command could make "lei index" far more useful... v2: close R/O mail_sync.sqlite3 dbh before fork Keeping the DB file handle open across fork can cause bad things to happen even if we don't use it since sqlite3 itself still knows about it (but doesn't know Perl code doesn't know about it).
2021-09-16inbox: streamline ->nntp_url
We no longer waste a precious hash slot for a per-Inbox {nntpserver} if it's only configured globally for all inboxes.
2021-09-15fetch|clone|--mirror: shorten paths for progress output
The full pathname for "curl -o ..." was too noisy and confusing. Reduce confusion by adding the ".tmp" suffix and relying on "-C". We'll also avoid displaying "-C" in run_reap() and rely on "--git-dir=" with "git fetch" to display progress for users.
2021-09-15clone|fetch|--mirror: add convenience $INBOX_DIR/Makefile
Since the beginning of time, I've been dropping Makefiles in $INBOX_DIR (and above hiearchies) to organize groups of commands. make(1) is widely available in various flavors and a familiar tool for our target audience. It is easy to run in the right directory, typically has built-in shell completion, and doesn't silently ignore errors by default like Bourne shell.
2021-09-15fetch: support --exit-code switch
As noted in the new manpage entry, this is useful for avoiding public-inbox-index invocations when there's nothing to update. We use 127 to match "grok-pull", and also because it doesn't conflict with any of the current curl(1) exit codes.
2021-09-15multi_git: hoist out common epoch/alternates handling
IMHO, this greatly improves code sharing and organization between v2, extindex, and lei/store. Common git-related logic for these is lightly-refactored and easier to reason about. The impetus for this big change was to ensure inboxes created+managed by public-inbox-{clone,fetch} could have alternates and configs setup properly without depending on SQLite (via V2Writable). This change does that while making old code shorter and better factored.
2021-09-14uri_imap: fix ->uidvalidity and ->uid w/ `/' separator
Again, we were failing to account for '/' use in mailbox names :x Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210914210547.akdp4cqmwaheayp5@meerkat.local/
2021-09-14uri_imap: handle '/' as an IMAP hierarchy separator
Untested at the moment(*), but we were inadvertantly truncating mailbox names with '/' due to our work-in-progress handling of "/;UID=$NUM" parameter. (*) strangely, my dovecot instance doesn't allow '/' by default, so the change to xt/net_writer-imap.t is untested. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210914175025.eq7s2shkc323itaf@meerkat.local/
2021-09-14t/run: TEST_LEI_DAEMON_PERSIST: die if pid changes
While persisting lei-daemon across different test cases isn't the default anymore, we can notice problems more quickly if the daemon PID changes since the daemon gets auto-restarted after failures.
2021-09-13lei up: fix --mua with single output
Oops :x Fixes: b584a53f053a7629 ("lei up: support --all for IMAP folders")
2021-09-13use POSIX::PIPE_BUF (instead of _POSIX_PIPE_BUF)
PIPE_BUF accounts for Linux being 4096 (and presumably other OSes differing), while _POSIX_PIPE_BUF is the minimum 512 value.
2021-09-13tests: add require_cmd, require curl when needed
t/v2mirror.t and t/lei-mirror.t are now skipped when curl is missing (instead of failing in appropriate places). A bunch of which() checks are updated to use require_cmd to avoid explicitly loading Spawn.
2021-09-12fetch: drop 304 Not Modified support, simplify comparisons
Timestamp comparisons only have 1 second granularity, which isn't nearly enough for our test cases, and probably not for real world use for "git send-email" bursts and fast SMTP servers. We'll continue to check modification times inside the manifest, though, in case an extremely rare SHA-1 collision is found...
2021-09-12fetch: fix and test v2 epoch detection
It was also totally broken by the change to use manifest.js.gz for v1 :x Fixes: ffb7fbda6869db4b ("fetch: use manifest.js.gz for v1")
2021-09-12fetch: fix half-baked v1 manifest.js.gz handling
The v1 code path was totally half-baked after the change to use manifest.js.gz :x Fixes: ffb7fbda6869db4b ("fetch: use manifest.js.gz for v1")
2021-09-12lei sucks: allow it to work without SQLite
And try to improve the message about Inline::C while we're at it, since Socket::Msghdr isn't widely-packaged, yet.
2021-09-12t/lei-*.t: guard setup_public_inboxes with test_lei
This ensures tests are skipped properly if SQLite or Xapian are missing and don't bail out.
2021-09-12init: set a useful description
"Unnamed repository" for v1 inboxes was misleading, and having a non-existent description for v2 was equally annoying, so set a short description based on the primary address. We remove descriptions when setting up new test inboxes to preserve the behavior of the t/lei-mirror.t test case.
2021-09-12import: do not write a "description" file
The default value is worthless to us and git functions fine without the file. public-inbox-init will create a useful one in the next change.
2021-09-12clone|lei_mirror: write description in mirrors
Instead of generic "Unnamed repository" or "missing" messages, show "mirror of $URL" since it seems like a better default when creating a mirror.
2021-09-12new public-inbox-{clone,fetch} commands
Setting up and maintaining git-only mirrors of v2 inboxes is complex since multiple commands are required to clone and fetch into epochs. Unlike grokmirror, these commands do not require any configuration. Instead, they rely on existing git config files and work like "git clone --mirror" and "git fetch", respectively. Like grokmirror, they use manifest.js.gz, but only on a per-inbox basis so users won't have to clone every inbox of a large instance nor edit config files to include/exclude inboxes they're interested in.
2021-09-12www: use ->ALL for per-inbox manifest.js.gz, too
With 11 epochs on LKML, the lkml/manifest.js.gz response time goes from around 60ms to around 10ms, a significant improvement. And improve test coverage while we're at it.
2021-09-11lei q|lcat: support "-f reply" output format
When composing replies in "git format-patch" cover letters, I'd been relying on "lei q -f text ...", but that still requires several steps to make it suitable for composing a reply: * s/^/> / to quote the body * drop existing In-Reply-To+References * s/^Message-ID:/In-Reply-To:/; * add an attribute line ... "lei q -f reply" takes care of most of that and users will only have to trim "From " lines, unnecessary results and over-quoted text (and trimming is likely less error-prone than doing all the steps above manually). This should also be a good replacement for "git format-patch --in-reply-to=...", since copying long Message-IDs can be error-prone (and this lets you include quoted text in replies).
2021-09-11lei: fix handling of broken lei.saved-search config files
lei shouldn't become unusable if a config file is invalid. Instead, show the "git config" stderr and attempt to continue gracefully. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210910141157.6u5adehpx7wftkor@meerkat.local/
2021-09-10lei add-external --mirror: quiet unlink error on ENOENT
If the mirror.done file doesn't exist for unlink, it's because we already got another error, so don't confuse users by noting an unlink error since the ENOENT is expected in the face of other errors.
2021-09-10lei add-external --mirror: deduce paths for PSGI mount prefixes
The current manifest.js.gz generation in WWW doesn't account for PSGI mount prefixes (and grokmirror 1.x appears to work fine). In other words, <https://yhbt.net/lore/lkml/manifest.js.gz> currently has keys like "/lkml/git/0.git" and not "/lore/lkml/git/0.git" where "/lore" is the PSGI mount prefix. This works fine with the prefix accounted for in my grokmirror (1.x) repos.conf like this: site = https://yhbt.net/lore/ manifest = https://yhbt.net/lore/manifest.js.gz Adding the PSGI mount prefix in manifest.js.gz is probably not desirable since it would force the prefix into the locally cloned path by grokmirror, and all the cloned directories would have the remote PSGI mount prefix prepended to the toplevel. So, "lei add-external --mirror" needs to account for PSGI mount prefixes by deducing the prefix based on available keys in the manifest.js.gz hash table.
2021-09-10t/git.t: quiet intentional git-rev-parse failure
It can get confusing, especially when running non-parallel "make test" Link: https://public-inbox.org/meta/20210909210138.ssiv5tri65mf4l4o@meerkat.local/
2021-09-08lei q|up: fix write counter for v2
It's a bit confusing to see "0 written to ..." when we actually wrote something.
2021-09-04lei_to_mail+mbox_reader: fix handling of empty/bogus emails
We may be handling invalid mboxes, so just return no objects in that case. While "lei q" on HTTP(S) externals expects a gzipped mboxrd, there's always a chance something else gzipped can be sent to us. There's also changes to lei_to_mail to better handle emails which lack a body and/or headers (e.g. t/solve/bare.patch) Link: https://public-inbox.org/meta/20210903151500.h72mzcpqixgtytjs@meerkat.local/
2021-09-03lei up --all: avoid double-close on shared STDOUT
This is merely to avoid perl setting errors internally which were not user visible. The double-close wasn't a problem in practice since we open a new file hanlde for the mbox or mbox.gz anyways, so the new t/lei-up.t test case shows no regressions nor fixes.
2021-09-03lei: dump errors to syslog, and not to CLI
Dumping errors from the previous run can often get lost, so just spew to syslog since it's a standard place to put errors that don't make it to a client. Note: we don't rely on $SIG{__WARN__} since some of the Net:: stuff will write directly to STDERR (as will external processes).
2021-09-02tests: "make check-run" favors reliability over speed
Sharing a single lei-daemon across multiple processes still exhibits reliability problems, and reliably checking lei-daemon's inotify internals seems impossible without. Even without lei-daemon sharing, "make check-run" is a few seconds faster than "make check" for me.
2021-09-02t/lei-auto-watch: improve test reliability
On slower systems, even a 100ms delay may not be enough; so loop and retry in hopes of an early exit for faster systems.
2021-09-02lei: propagate keyword changes from lei/store
This works with existing inotify/EVFILT_VNODE functionality to propagate changes made from one Maildir to another Maildir. I chose the lei/store worker process to handle this since propagating changes back into lei-daemon on a massive scale could lead to dead-locking while both processes are attempting to write to each other. Eliminating IPC overhead is a nice side effect, but could hurt performance if Maildirs are slow. The code for "lei export-kw" is significantly revamped to match the new code used in the "lei/store" daemon. It should be more correct w.r.t. corner-cases and stale entries, but perhaps better tests need to be written. squashed: t/lei-auto-watch: increase delay for FreeBSD kevent My FreeBSD VM seems to need longer for this test than inotify under Linux, likely because the kevent support code needs to be more complicated.
2021-09-02lei_mail_sync: do not use transactions
For lei-index to work in parallel with MUA access and upcoming inotify-based updates, mail_sync.sqlite3 needs to always be up-to-date to read-only worker processes (ahead of everything else). So rely on the default auto-commit behavior and hope SQLite WAL can reduce some of the overheads involved with writes.
2021-09-01extindex: --gc removes messages from over, too
While messages from removed inboxes were removed from Xapian search, --gc failed to remove messages from over.sqlite3 entirely. They no longer show up in the topic summary view. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/20210830201723.dehoul4y6gpqf2cp@nitro.local/
2021-08-31lei_mail_sync: set_src uses binary OIDs
Another step towards moving more of our internals to use binary OIDs to avoid needless conversions before hitting disk.
2021-08-31t/lei-watch: avoid race between glob + readlink
Open file handles in lei-daemon may be unstable so we need to account for readlink() returning undef.
2021-08-30www: move mirror instructions to /text/
This makes the mirroring and code retrieval instructions less obstructive. Relying on WwwText means we only use our Linkify module to make hrefs of full URLs; making relative and shortened hrefs off-limits; hopefully this isn't too much of a problem. coderepo information remains duplicated on every page since (IMHO) coderepos are an important feature; but nobody besides me has ever bothered to configure coderepos, so I suppose it's fine... Suggested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210826132747.6gxuwnhftyf7c6hp@nitro.local/
2021-08-28www_listing: fix odd "locate inbox" cases
Searching inboxes with an empty query no longer gives 500 errors due to Xapian. Also, improve the error message when no inboxes match, since saying no inboxes exist yet is wrong.
2021-08-28www_listing: show ->ALL at top of HTML listing
It's a special case and we can show it in the HTML display without affecting manifest.js.gz generation.
2021-08-28move ->ids_after from mm to over
Since we favor ->over in WWW and IMAP, move this method to ->over to reduce open files in common cases. This fixes the /$EXTINDEX_NAME/all.mbox.gz endpoint for extindex entries (which may get expensive...).
2021-08-28www_text: fix example config snippet for extindex
extindex doesn't use the same config stuff as normal "publicinbox" entries, so we'll need a separate function for them.
2021-08-28get rid of unnecessary bytes::length usage
The only place where we could return wide characters with -httpd was the raw $INBOX_DIR/description text, which is now converted to octets. All daemon (HTTP/NNTP/IMAP) sockets are opened in binary mode, so length() and bytes::length() are equivalent on reads. For socket writes, any non-octet data would warn about wide characters and we are strict in warnings with test_httpd. All gzipped buffers are also octets, as is PublicInbox::Eml->body, and anything from PerlIO objects ("git cat-file --batch" output, filesystems), so bytes::length was unnecessary in all those places.
2021-08-19lei q: make --save the default
Since "lei up" is more often useful than not and incurs neglible overhead; enable --save by default and allow --no-save to work. This also fixes a long-standing when overwriting --output destinations with saved searches: dedupe data from previous searches are reset and no longer influences the new (changed) search, so results no longer go missing if two sequential invocations of "lei q --save" point to the same --output.
2021-08-17view: remove mbox.gz and Atom from topic view
This declutters the topic view since these links seem rarely used. Atom and mbox.gz links probably make most sense when users have read the HTML and decide the topic is worth following or downloading. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210816154444.sj3ks2sikq3x2ywx@nitro.local/
2021-08-16t/run.perl: fix "make check-run" on FreeBSD 11.x
Persistent lei-daemon still leads to ECONNRESET client errors on FreeBSD, and maxing out the kern.ipc.soacceptqueue sysctl (as documented in the FreeBSD listen(2) manpage) doesn't seem to help. "make check-run" is still 4-5s faster than "make check" on my FreeBSD VM even after this change, so it's still a worthwhile improvement.
2021-08-11lei: attempt to canonicalize away "/../" pathnames
As documented, File::Spec->canonpath does not canonicalize "/../". While we want to do our best to preserve symlinks in pathnames, leaving "/../" can mislead our inotify|kqueue usage.
2021-08-11lei_saved_search: canonicalized relative save paths
Storing relative paths with '..' in them can be expensive to resolve when running 'lei up', so prefer storing canonicalized absolute paths. We only do this for paths with '..' in them, though, since this can lose symlink info.