about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-09-12new public-inbox-{clone,fetch} commands
Setting up and maintaining git-only mirrors of v2 inboxes is complex since multiple commands are required to clone and fetch into epochs. Unlike grokmirror, these commands do not require any configuration. Instead, they rely on existing git config files and work like "git clone --mirror" and "git fetch", respectively. Like grokmirror, they use manifest.js.gz, but only on a per-inbox basis so users won't have to clone every inbox of a large instance nor edit config files to include/exclude inboxes they're interested in.
2021-09-12lei_mirror: fix error message
We're using rename(2) rather than link(2)
2021-09-12lei_mirror: simplify error reporting
Slowly transitioning to using die() more, which hopefully improves code reusability between lei and non-lei parts of our code.
2021-09-12www: use ->ALL for per-inbox manifest.js.gz, too
With 11 epochs on LKML, the lkml/manifest.js.gz response time goes from around 60ms to around 10ms, a significant improvement. And improve test coverage while we're at it.
2021-09-12manifest.js.gz: avoid long-lived per-epoch cat-file processes
When generating per-inbox manifests, we were forgetting to cleanup per-epoch "git cat-file --batch" processes. Our previous method of generating modified times was also stupidly inefficient, so replace the pipeline with a single "git for-each-ref" invocation.
2021-09-11lei q|lcat: support "-f reply" output format
When composing replies in "git format-patch" cover letters, I'd been relying on "lei q -f text ...", but that still requires several steps to make it suitable for composing a reply: * s/^/> / to quote the body * drop existing In-Reply-To+References * s/^Message-ID:/In-Reply-To:/; * add an attribute line ... "lei q -f reply" takes care of most of that and users will only have to trim "From " lines, unnecessary results and over-quoted text (and trimming is likely less error-prone than doing all the steps above manually). This should also be a good replacement for "git format-patch --in-reply-to=...", since copying long Message-IDs can be error-prone (and this lets you include quoted text in replies).
2021-09-11lei: normalize whitespace in remote queries
Having redundant "+" in URLs is ugly and can hurt cacheability of queries. Even with "quoted phrase searches", Xapian seems unaffected by redundant spaces, so just normalize the ASCII white spaces to ' ' (%20) when fed via STDIN or saved-search config file. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210910141157.6u5adehpx7wftkor@meerkat.local/
2021-09-11lei: pass client stderr to git-config in more places
This should improve the users' chances of seeing errors in various git config files we use.
2021-09-11lei: fix handling of broken lei.saved-search config files
lei shouldn't become unusable if a config file is invalid. Instead, show the "git config" stderr and attempt to continue gracefully. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210910141157.6u5adehpx7wftkor@meerkat.local/
2021-09-10INSTALL: depend on URI rather than URI::Escape
As far as I can tell, URI::Escape has always been a part of the `URI' package (aka "distribution" on CPAN) and not distributed separately (unlike URI::Escape::XS). So avoid confusing users with `URI::Escape' and just document `URI' instead. Along the same lines, we depend on the `Plack' package rather than Plack::Util or Plack::Builder, after all.
2021-09-10lei up: only delay non-zero "# $NR written to ..."
"# 0 written to $FOLDER" messages aren't important to the user, so we can show them in real time and allow them to be lost in the terminal scroll. When >0 messages are written to a folder, we'll show them last so a user will know which folders to open with their MUA.
2021-09-10doc: lei-index manpage
It's a pretty incomplete command, so it's important to document its incompleteness.
2021-09-10lei: do not read ~/.netrc by default
Since ~/.netrc isn't widely used by most (if any) NNTP and IMAP clients, we won't read it by default for lei. AFAIK, ~/.netrc is mainly by FTP clients (e.g. ftp(1) and lftp(1)). wget uses it by default for HTTP(S) (and FTP), but curl does not. To avoid breaking stable release use cases, public-inbox-watch continues to read ~/.netrc by default. The --netrc switch is supported by all existing lei commands which may use curl.
2021-09-10lei: split out @net_opt for curl/torsocks use
IMAP and NNTP connections share some curl(1) options for TLS, IPv4/IPv6, or netrc, etc...
2021-09-10lei_query: fix comment about %lei2curl commands
Just a typo.
2021-09-10lei add-external --mirror: quiet unlink error on ENOENT
If the mirror.done file doesn't exist for unlink, it's because we already got another error, so don't confuse users by noting an unlink error since the ENOENT is expected in the face of other errors.
2021-09-10lei add-external --mirror: deduce paths for PSGI mount prefixes
The current manifest.js.gz generation in WWW doesn't account for PSGI mount prefixes (and grokmirror 1.x appears to work fine). In other words, <https://yhbt.net/lore/lkml/manifest.js.gz> currently has keys like "/lkml/git/0.git" and not "/lore/lkml/git/0.git" where "/lore" is the PSGI mount prefix. This works fine with the prefix accounted for in my grokmirror (1.x) repos.conf like this: site = https://yhbt.net/lore/ manifest = https://yhbt.net/lore/manifest.js.gz Adding the PSGI mount prefix in manifest.js.gz is probably not desirable since it would force the prefix into the locally cloned path by grokmirror, and all the cloned directories would have the remote PSGI mount prefix prepended to the toplevel. So, "lei add-external --mirror" needs to account for PSGI mount prefixes by deducing the prefix based on available keys in the manifest.js.gz hash table.
2021-09-10gcf2: die if pkg-config is missing
We can't link properly to libgit2 without pkg-config telling us which libraries and headers to use.
2021-09-10t/git.t: quiet intentional git-rev-parse failure
It can get confusing, especially when running non-parallel "make test" Link: https://public-inbox.org/meta/20210909210138.ssiv5tri65mf4l4o@meerkat.local/
2021-09-10test_common: print diagnostic for missing dependencies
The "#" prefix should prevent it from being too alarming if a dependency is expected to be missing, but still useful if a dependency is misconfigured.
2021-09-09lei up: print messages before disconnecting
Closing the socket for script/lei needs to be done AFTER the final message(s) are printed.
2021-09-09net_reader: support Mail::IMAPClient Ignoresizeerrors
Some proprietary servers may do wacky things and give the wrong size, so Mail::IMAPClient has a knob for this which we can expose to users to workaround this.
2021-09-09net_reader: improve naming of common args
IMHO this makes things easier-to-follow than before.
2021-09-09net_reader: combine Net::NNTP and IMAPClient args
Since these are keyed by IMAP and NNTP URIs which can never conflict, it simplifies our internals to keep them in one big hash since we'll add POP3 and JMAP client support.
2021-09-09net_reader: imap_opt => cfg_opt
Since this our internal IMAP options are keyed by URI section, there's no need to have separate hashes for NNTP and IMAP options since they URI already distinguishes them. This will make future changes to support POP3 and JMAP and arg caching with lei/store easier.
2021-09-09net_reader: nntp_opt => cfg_opt
Since this our internal NNTP options are keyed by URI section, there's no need to have separate hashes for NNTP and IMAP options since they URI already distinguishes them. This will make future changes to support POP3 and JMAP and arg caching with lei/store easier.
2021-09-09net_reader: preserve memoized IMAPClient arg for SOCKS
Multiple invocations of mic_new may happen in long-lived processes, so do not let mic_new make irreversible changes to the cached args when using a SOCKS proxy.
2021-09-09net_reader: set IMAPClient Keepalive flag late
Since we always enable SO_KEEPALIVE unconditionally, having it in {mic_arg} leads to unnecessary IPC overhead and memory use.
2021-09-09net_reader: do not set "SSL" fields for non-TLS
This will save a little bit of memory and IPC I/O for users connecting to localhost and the majority of Tor .onions.
2021-09-09lei prune-mail-sync: ignore missing locations
"lei prune-mail-sync --all" shouldn't abort if a location isn't available, and maybe it should prune harder...
2021-09-09lei_input: provide hint for bare "L:" and "kw:"
I just made this mistake running "lei import" myself, so I figure giving a hint makes sense, here.
2021-09-09lei-rm: add man page, support LeiInput args
-F/--in-format and --lock=TYPE(S) are easily supported by all classes using LeiInput.
2021-09-08news_www: favor ->ALL URL when using ->ALL
This allows us to link to threads spread across multiple inboxes. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210907140954.4rlh6pn5fz4ljkxp@meerkat.local/
2021-09-08doc: acknowledge the MMDF mailbox format
While I don't currently see a point in supporting MMDF, we'll still acknowledge it since mutt actually supports it. Expand a bit on MH while we're at it, since MH seems at least relevant.
2021-09-08lei q|up: fix write counter for v2
It's a bit confusing to see "0 written to ..." when we actually wrote something.
2021-09-07news_www: avoid uninitialized variables
PATH_INFO may not have enough slashes for newsgroup name in the URL at all, so ensure we don't try to further process requests which have no chance of having a newsgroup name.
2021-09-07doc: lei-*.pod: update to Tor v3 .onion address
We missed a few when new documentation came in, and there's no going back to v2 onions. Followup-to: 0b15dfc58ceaecdc ("treewide: update to v3 Tor onions")
2021-09-07lei up: support --all for IMAP folders
Since "lei up" is expected to be a heavily-used command, better support for IMAP seems like a reasonable idea. This is inefficient since we waste an IMAP(S) TCP connection since it dies when an auth-only LeiUp worker process dies, but it's better than not working at all, right now.
2021-09-07lei: dump and clear log at exit
This may be helpful for diagnosing errors in case we missed any.
2021-09-07xt/net_writer_imap: test "lei up" on single IMAP output
That's the minimum, at least...
2021-09-06lei_auth: simplify users
There's no need to alias net_merge_all in each WQ class which uses LeiAuth, `$obj->$sub' works even when `$sub' is a fully-qualified subroutine name with `::' in it. perlobj(1) documents it under "Method Call Variations".
2021-09-06lei_auth: remove net_merge_done1 step
It turns out this step is unnecessary, since SOCK_SEQPACKET ordering is guaranteed and we know wq_broadcast calls will always be handled sequentially.
2021-09-06lei_auth: diagram for current behavior
Before making potentially major changes, lets clarify readers' understanding of how LeiAuth currently works.
2021-09-06lei_search: xsmsg_vmd: retry_reopen properly
The deeper eval was preventing retry_reopen from retrying with readers and writers working in parallel: FOO=imaps://example.com/INBOX.huge lei lcat $FOO -f mboxcl | lei tag -F mboxcl +L:bar - Fixes: c7bcfe6cd6648ff0 ("lei: diagnostics for /Document \d+ not found/ errors")
2021-09-06net_reader: don't approve/reject credentials w/o "fill"
Credentials sourced via ~/.netrc should not be written to git-credential.
2021-09-04lei_to_mail+mbox_reader: fix handling of empty/bogus emails
We may be handling invalid mboxes, so just return no objects in that case. While "lei q" on HTTP(S) externals expects a gzipped mboxrd, there's always a chance something else gzipped can be sent to us. There's also changes to lei_to_mail to better handle emails which lack a body and/or headers (e.g. t/solve/bare.patch) Link: https://public-inbox.org/meta/20210903151500.h72mzcpqixgtytjs@meerkat.local/
2021-09-03lei: fix read/write IMAP access
xt/net_writer-imap.t was completely broken in recent months and I completely forgot this test. net->add_url still only accepts bare scalars (and not scalar refs), so we must set that up properly. Furthermore, our changes to do FLAGS-only synchronization in lei of old messages was causing us to not handle FLAGS properly for the test.
2021-09-03lei_xsearch: avoid false-positives on externals w/ L: and kw:
We need to use LeiSearch->qparse_new to handle (and filter out) "L:" and "kw:" search prefixes to avoid hitting false positives when externals are involved. Unfortunately, this doesn't work for remote HTTP(S) externals, but those aren't enabled by default.
2021-09-03lei inspect: support reading eml from --stdin
This can be useful inside mutt since I was diagnosing why a label ("L:$FOO") search was giving me a false-positive search result...
2021-09-03lei up --all: avoid double-close on shared STDOUT
This is merely to avoid perl setting errors internally which were not user visible. The double-close wasn't a problem in practice since we open a new file hanlde for the mbox or mbox.gz anyways, so the new t/lei-up.t test case shows no regressions nor fixes.