about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-09-06lei_auth: simplify users
There's no need to alias net_merge_all in each WQ class which uses LeiAuth, `$obj->$sub' works even when `$sub' is a fully-qualified subroutine name with `::' in it. perlobj(1) documents it under "Method Call Variations".
2021-09-06lei_auth: remove net_merge_done1 step
It turns out this step is unnecessary, since SOCK_SEQPACKET ordering is guaranteed and we know wq_broadcast calls will always be handled sequentially.
2021-09-06lei_auth: diagram for current behavior
Before making potentially major changes, lets clarify readers' understanding of how LeiAuth currently works.
2021-09-06lei_search: xsmsg_vmd: retry_reopen properly
The deeper eval was preventing retry_reopen from retrying with readers and writers working in parallel: FOO=imaps://example.com/INBOX.huge lei lcat $FOO -f mboxcl | lei tag -F mboxcl +L:bar - Fixes: c7bcfe6cd6648ff0 ("lei: diagnostics for /Document \d+ not found/ errors")
2021-09-06net_reader: don't approve/reject credentials w/o "fill"
Credentials sourced via ~/.netrc should not be written to git-credential.
2021-09-04lei_to_mail+mbox_reader: fix handling of empty/bogus emails
We may be handling invalid mboxes, so just return no objects in that case. While "lei q" on HTTP(S) externals expects a gzipped mboxrd, there's always a chance something else gzipped can be sent to us. There's also changes to lei_to_mail to better handle emails which lack a body and/or headers (e.g. t/solve/bare.patch) Link: https://public-inbox.org/meta/20210903151500.h72mzcpqixgtytjs@meerkat.local/
2021-09-03lei: fix read/write IMAP access
xt/net_writer-imap.t was completely broken in recent months and I completely forgot this test. net->add_url still only accepts bare scalars (and not scalar refs), so we must set that up properly. Furthermore, our changes to do FLAGS-only synchronization in lei of old messages was causing us to not handle FLAGS properly for the test.
2021-09-03lei_xsearch: avoid false-positives on externals w/ L: and kw:
We need to use LeiSearch->qparse_new to handle (and filter out) "L:" and "kw:" search prefixes to avoid hitting false positives when externals are involved. Unfortunately, this doesn't work for remote HTTP(S) externals, but those aren't enabled by default.
2021-09-03lei inspect: support reading eml from --stdin
This can be useful inside mutt since I was diagnosing why a label ("L:$FOO") search was giving me a false-positive search result...
2021-09-03lei up --all: avoid double-close on shared STDOUT
This is merely to avoid perl setting errors internally which were not user visible. The double-close wasn't a problem in practice since we open a new file hanlde for the mbox or mbox.gz anyways, so the new t/lei-up.t test case shows no regressions nor fixes.
2021-09-03lei: use lei->lms in place of lse->lms in a few places
We can golf out some code and refcounts this way.
2021-09-03lei: ->child_error less error-prone
I was calling "child_error(1, ...)" in a few places where I meant to be calling "child_error(1 << 8, ...)" and inadvertantly triggering SIGHUP in script/lei. Since giving a zero exit code to child_error makes no sense, just allow falsy values to default to 1 << 8.
2021-09-03lei/store: quiet down link(2) warnings
ENOENT can be too common due to timing and concurrent access from MUAs and "lei export-kw", and other mail synchronization tools (e.g. mbsync and offlineimap).
2021-09-03lei: dump errors to syslog, and not to CLI
Dumping errors from the previous run can often get lost, so just spew to syslog since it's a standard place to put errors that don't make it to a client. Note: we don't rely on $SIG{__WARN__} since some of the Net:: stuff will write directly to STDERR (as will external processes).
2021-09-02www: handle name-only publicinbox.*.url entries
Apparently URLs can be configured relatively for HTTP(S) setups, attempt to support them when linking to cross-posted messages. This also fixes the top-row (mirror/help/color/Atom feed) links in /$INBOX_URL/$EXTMSG_MSGID/T/ (and /t/) URLs. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210902191239.cmbxlmjqcsmdzmqp@meerkat.local/
2021-09-02tests: "make check-run" favors reliability over speed
Sharing a single lei-daemon across multiple processes still exhibits reliability problems, and reliably checking lei-daemon's inotify internals seems impossible without. Even without lei-daemon sharing, "make check-run" is a few seconds faster than "make check" for me.
2021-09-02t/lei-auto-watch: improve test reliability
On slower systems, even a 100ms delay may not be enough; so loop and retry in hopes of an early exit for faster systems.
2021-09-02lei: propagate keyword changes from lei/store
This works with existing inotify/EVFILT_VNODE functionality to propagate changes made from one Maildir to another Maildir. I chose the lei/store worker process to handle this since propagating changes back into lei-daemon on a massive scale could lead to dead-locking while both processes are attempting to write to each other. Eliminating IPC overhead is a nice side effect, but could hurt performance if Maildirs are slow. The code for "lei export-kw" is significantly revamped to match the new code used in the "lei/store" daemon. It should be more correct w.r.t. corner-cases and stale entries, but perhaps better tests need to be written. squashed: t/lei-auto-watch: increase delay for FreeBSD kevent My FreeBSD VM seems to need longer for this test than inotify under Linux, likely because the kevent support code needs to be more complicated.
2021-09-02lei_input: set and prepare watches early
This will be needed as we track changes in real-time, especially for "lei index" since there's no storage involved.
2021-09-02lei_mail_sync: do not use transactions
For lei-index to work in parallel with MUA access and upcoming inotify-based updates, mail_sync.sqlite3 needs to always be up-to-date to read-only worker processes (ahead of everything else). So rely on the default auto-commit behavior and hope SQLite WAL can reduce some of the overheads involved with writes.
2021-09-01extindex: --gc removes messages from over, too
While messages from removed inboxes were removed from Xapian search, --gc failed to remove messages from over.sqlite3 entirely. They no longer show up in the topic summary view. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/20210830201723.dehoul4y6gpqf2cp@nitro.local/
2021-09-01extindex: document --all and indexlevel=basic interaction
Cache is expensive, so help users save on storage by documenting this behavior.
2021-08-31lei up: only show finmsg in top-level lei-daemon
->DESTROY can get triggered in child processes, which unnecessarily duplicates messages queued up for display when lei spawns extra workers.
2021-08-31lei/store: correctly delete entries from over
Some of these errors were inadvertantly lost due to delayed error reporting in the past.
2021-08-31lei: fix error reporting from lei/store -> lei-daemon
We must set autoflush to ensure timely notification of clients; and lei-daemon must not block when waiting on reads in case of spurious wakeups. Finally, if no clients are connected to lei-daemon, write to syslog to ensure the error is visible.
2021-08-31lei_mail_sync: set_src uses binary OIDs
Another step towards moving more of our internals to use binary OIDs to avoid needless conversions before hitting disk.
2021-08-31lei: refresh watches before MUA spawn for Maildir
If we possibly just wrote or created a Maildir, ensure it's monitored by the lei watch mechanism.
2021-08-31lei note-event: always flush changes on daemon exit
Because the timer may not fire in time before daemon shutdown.
2021-08-31t/lei-watch: avoid race between glob + readlink
Open file handles in lei-daemon may be unstable so we need to account for readlink() returning undef.
2021-08-31lei_mail_sync: make rename_folder more robust
We need to account for past canonicalization errors and deal with cases which violate uniqueness constraints in mail_sync.sqlite3
2021-08-31lei_mail_sync: simplify group2folders
No need to loop when we can rely on grep.
2021-08-31lei prune-mail-sync: handle --all (no args)
This still needs tests, but I noticed "--all" w/o "local" or "remote" was not working correctly since split() returned an empty array.
2021-08-31lei_mail_sync: forget_folder: simplify code
No need to bump refcounts of {dbh} nor declare extra variables for a rarely-called function.
2021-08-31www_listing: add note about mirroring information
Perhaps this can be expanded to include grokmirror information in the future. For now, just give a hint about the "mirror" link for each inbox.
2021-08-31www_text/mirror: spell out "external index" and "public inbox"
"extindex" and "public-inbox" are project-specific terms which are probably unsuitable for folks who are seeing this for the first time. Use "public inbox" when referring to actual public inboxes, since "public-inbox" is merely the name for this particular implementation and others have adopted the same concept (IMHO the concept is more important than any particular implementation).
2021-08-31www_stream: extra link to mirroring information in the footer
This may be redundant with the "mirror" link at the top right, but maybe people will miss one. Properly capitalize the "Code repositories" text while we're at it. Link: https://public-inbox.org/20210828175827.rgzwqbn7brl56oej@nitro.local/ Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-08-30www_stream: description header links to top $INBOX_URL
Making the inbox description link back to the most recent per-inbox topics from text/ and $OID/s/ URLs seems useful, rather than keeping the description up there. Followup-to: 6c853f5256f3a324 ("www: improve navigation around contemporary threads")
2021-08-30www: move mirror instructions to /text/
This makes the mirroring and code retrieval instructions less obstructive. Relying on WwwText means we only use our Linkify module to make hrefs of full URLs; making relative and shortened hrefs off-limits; hopefully this isn't too much of a problem. coderepo information remains duplicated on every page since (IMHO) coderepos are an important feature; but nobody besides me has ever bothered to configure coderepos, so I suppose it's fine... Suggested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210826132747.6gxuwnhftyf7c6hp@nitro.local/
2021-08-28www: avoid potential auto-vivification on ibx->{url}
This may fix problems with the "all" link disappearing. Link: https://public-inbox.org/meta/CAMwyc-Tw=v5yT1U1U66GSwwTK8OJXv8_YDu-=oXbZO3tHSnYWw@mail.gmail.com/
2021-08-28www_listing: fix odd "locate inbox" cases
Searching inboxes with an empty query no longer gives 500 errors due to Xapian. Also, improve the error message when no inboxes match, since saying no inboxes exist yet is wrong.
2021-08-28www_listing: show ->ALL at top of HTML listing
It's a special case and we can show it in the HTML display without affecting manifest.js.gz generation.
2021-08-28move ->ids_after from mm to over
Since we favor ->over in WWW and IMAP, move this method to ->over to reduce open files in common cases. This fixes the /$EXTINDEX_NAME/all.mbox.gz endpoint for extindex entries (which may get expensive...).
2021-08-28www_text: add coderepo config support for extindex
At least manually configured coderepos "just work" for extindex, though it probably could be automatic and inherited from the publicinbox configs.
2021-08-28config: do not parse altid for extindex
There's currently no support for altid with extindex, and there's likely no legacy precedent for using altid like there is with single public-inboxes.
2021-08-28www_text: fix example config snippet for extindex
extindex doesn't use the same config stuff as normal "publicinbox" entries, so we'll need a separate function for them.
2021-08-28www: avoid incorrect instructions for extindex
There's no way to clone an extindex, since there's no git storage associated with them. So attempt to link to the HTML listing of public-inboxes, instead.
2021-08-28www_stream: sh-friendly .onion URLs wrapping
The long v3 .onion URL was causing havoc on small mobile displays, so extract "hostname" into a variable which can still used as a Bourne shell snippet. While we're at it, include "torsocks" in the git command used for .onion URLs since that's the (near)-universal wrapper for Tor-ifying things (like git) which are dynamically linked to libc. Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210816163654.c6gfzuezhji4l6s7@nitro.local/
2021-08-28ds: use bytes::substr and bytes::length module-wide for now
The use of substr within IO::Handle->write may not be correct if we have wide characters, so handle it ourselves. bytes.pm usage is probably better fixed in PublicInbox::NNTP, but the effort required is higher, so we'll just keep bytes in DS for now.
2021-08-28get rid of unnecessary bytes::length usage
The only place where we could return wide characters with -httpd was the raw $INBOX_DIR/description text, which is now converted to octets. All daemon (HTTP/NNTP/IMAP) sockets are opened in binary mode, so length() and bytes::length() are equivalent on reads. For socket writes, any non-octet data would warn about wide characters and we are strict in warnings with test_httpd. All gzipped buffers are also octets, as is PublicInbox::Eml->body, and anything from PerlIO objects ("git cat-file --batch" output, filesystems), so bytes::length was unnecessary in all those places.
2021-08-26wwwlisting: support global CSS in HTML view
Since CSS can be overridden by a static webserver on a per-inbox basis, we need a similar pattern to deal with the instance-wide WwwListing HTML. "/+/" probably won't conflict with any current nor future public inbox names. I don't think it'll cause problems with common linkifiers or URL extractors, either (and it's unlikely anybody would want to share URLs of just the CSS in a plain text(-like) format).