Date | Commit message (Collapse) |
|
Setting up and maintaining git-only mirrors of v2 inboxes is
complex since multiple commands are required to clone and fetch
into epochs.
Unlike grokmirror, these commands do not require any
configuration. Instead, they rely on existing git config files
and work like "git clone --mirror" and "git fetch",
respectively.
Like grokmirror, they use manifest.js.gz, but only on a
per-inbox basis so users won't have to clone every inbox of a
large instance nor edit config files to include/exclude inboxes
they're interested in.
|
|
We're using rename(2) rather than link(2)
|
|
Slowly transitioning to using die() more, which hopefully
improves code reusability between lei and non-lei parts of our
code.
|
|
With 11 epochs on LKML, the lkml/manifest.js.gz response time
goes from around 60ms to around 10ms, a significant improvement.
And improve test coverage while we're at it.
|
|
When generating per-inbox manifests, we were forgetting to
cleanup per-epoch "git cat-file --batch" processes. Our
previous method of generating modified times was also stupidly
inefficient, so replace the pipeline with a single
"git for-each-ref" invocation.
|
|
When composing replies in "git format-patch" cover letters,
I'd been relying on "lei q -f text ...", but that still requires
several steps to make it suitable for composing a reply:
* s/^/> / to quote the body
* drop existing In-Reply-To+References
* s/^Message-ID:/In-Reply-To:/;
* add an attribute line
...
"lei q -f reply" takes care of most of that and users will
only have to trim "From " lines, unnecessary results and
over-quoted text (and trimming is likely less error-prone
than doing all the steps above manually).
This should also be a good replacement for
"git format-patch --in-reply-to=...", since copying long
Message-IDs can be error-prone (and this lets you include
quoted text in replies).
|
|
Having redundant "+" in URLs is ugly and can hurt cacheability
of queries. Even with "quoted phrase searches", Xapian seems
unaffected by redundant spaces, so just normalize the ASCII
white spaces to ' ' (%20) when fed via STDIN or saved-search
config file.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210910141157.6u5adehpx7wftkor@meerkat.local/
|
|
This should improve the users' chances of seeing errors in
various git config files we use.
|
|
lei shouldn't become unusable if a config file is invalid.
Instead, show the "git config" stderr and attempt to continue
gracefully.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210910141157.6u5adehpx7wftkor@meerkat.local/
|
|
As far as I can tell, URI::Escape has always been a part of the
`URI' package (aka "distribution" on CPAN) and not distributed
separately (unlike URI::Escape::XS). So avoid confusing users
with `URI::Escape' and just document `URI' instead.
Along the same lines, we depend on the `Plack' package rather
than Plack::Util or Plack::Builder, after all.
|
|
"# 0 written to $FOLDER" messages aren't important to the
user, so we can show them in real time and allow them to
be lost in the terminal scroll. When >0 messages are
written to a folder, we'll show them last so a user
will know which folders to open with their MUA.
|
|
It's a pretty incomplete command, so it's important to document
its incompleteness.
|
|
Since ~/.netrc isn't widely used by most (if any) NNTP and IMAP
clients, we won't read it by default for lei. AFAIK, ~/.netrc
is mainly by FTP clients (e.g. ftp(1) and lftp(1)). wget uses
it by default for HTTP(S) (and FTP), but curl does not.
To avoid breaking stable release use cases, public-inbox-watch
continues to read ~/.netrc by default.
The --netrc switch is supported by all existing lei commands
which may use curl.
|
|
IMAP and NNTP connections share some curl(1) options for TLS,
IPv4/IPv6, or netrc, etc...
|
|
Just a typo.
|
|
If the mirror.done file doesn't exist for unlink, it's because
we already got another error, so don't confuse users by noting
an unlink error since the ENOENT is expected in the face of
other errors.
|
|
The current manifest.js.gz generation in WWW doesn't account for
PSGI mount prefixes (and grokmirror 1.x appears to work fine).
In other words, <https://yhbt.net/lore/lkml/manifest.js.gz>
currently has keys like "/lkml/git/0.git" and not
"/lore/lkml/git/0.git" where "/lore" is the PSGI mount prefix.
This works fine with the prefix accounted for in my grokmirror
(1.x) repos.conf like this:
site = https://yhbt.net/lore/
manifest = https://yhbt.net/lore/manifest.js.gz
Adding the PSGI mount prefix in manifest.js.gz is probably not
desirable since it would force the prefix into the locally
cloned path by grokmirror, and all the cloned directories
would have the remote PSGI mount prefix prepended to the
toplevel.
So, "lei add-external --mirror" needs to account for PSGI
mount prefixes by deducing the prefix based on available keys
in the manifest.js.gz hash table.
|
|
We can't link properly to libgit2 without pkg-config telling
us which libraries and headers to use.
|
|
It can get confusing, especially when running non-parallel "make test"
Link: https://public-inbox.org/meta/20210909210138.ssiv5tri65mf4l4o@meerkat.local/
|
|
The "#" prefix should prevent it from being too alarming if
a dependency is expected to be missing, but still useful if a
dependency is misconfigured.
|
|
Closing the socket for script/lei needs to be done AFTER the
final message(s) are printed.
|
|
Some proprietary servers may do wacky things and give the
wrong size, so Mail::IMAPClient has a knob for this which
we can expose to users to workaround this.
|
|
IMHO this makes things easier-to-follow than before.
|
|
Since these are keyed by IMAP and NNTP URIs which can never
conflict, it simplifies our internals to keep them in one big
hash since we'll add POP3 and JMAP client support.
|
|
Since this our internal IMAP options are keyed by URI section,
there's no need to have separate hashes for NNTP and IMAP
options since they URI already distinguishes them.
This will make future changes to support POP3 and JMAP and
arg caching with lei/store easier.
|
|
Since this our internal NNTP options are keyed by URI section,
there's no need to have separate hashes for NNTP and IMAP
options since they URI already distinguishes them.
This will make future changes to support POP3 and JMAP and
arg caching with lei/store easier.
|
|
Multiple invocations of mic_new may happen in long-lived
processes, so do not let mic_new make irreversible changes
to the cached args when using a SOCKS proxy.
|
|
Since we always enable SO_KEEPALIVE unconditionally, having it
in {mic_arg} leads to unnecessary IPC overhead and memory use.
|
|
This will save a little bit of memory and IPC I/O for users
connecting to localhost and the majority of Tor .onions.
|
|
"lei prune-mail-sync --all" shouldn't abort if a location
isn't available, and maybe it should prune harder...
|
|
I just made this mistake running "lei import" myself, so
I figure giving a hint makes sense, here.
|
|
-F/--in-format and --lock=TYPE(S) are easily supported by
all classes using LeiInput.
|
|
This allows us to link to threads spread across multiple inboxes.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210907140954.4rlh6pn5fz4ljkxp@meerkat.local/
|
|
While I don't currently see a point in supporting MMDF, we'll
still acknowledge it since mutt actually supports it. Expand a
bit on MH while we're at it, since MH seems at least relevant.
|
|
It's a bit confusing to see "0 written to ..." when we actually
wrote something.
|
|
PATH_INFO may not have enough slashes for newsgroup name in the
URL at all, so ensure we don't try to further process requests
which have no chance of having a newsgroup name.
|
|
We missed a few when new documentation came in, and there's no
going back to v2 onions.
Followup-to: 0b15dfc58ceaecdc ("treewide: update to v3 Tor onions")
|
|
Since "lei up" is expected to be a heavily-used command,
better support for IMAP seems like a reasonable idea.
This is inefficient since we waste an IMAP(S) TCP connection
since it dies when an auth-only LeiUp worker process dies, but
it's better than not working at all, right now.
|
|
This may be helpful for diagnosing errors in case we missed any.
|
|
That's the minimum, at least...
|
|
There's no need to alias net_merge_all in each WQ class
which uses LeiAuth, `$obj->$sub' works even when `$sub'
is a fully-qualified subroutine name with `::' in it.
perlobj(1) documents it under "Method Call Variations".
|
|
It turns out this step is unnecessary, since SOCK_SEQPACKET
ordering is guaranteed and we know wq_broadcast calls will
always be handled sequentially.
|
|
Before making potentially major changes, lets clarify readers'
understanding of how LeiAuth currently works.
|
|
The deeper eval was preventing retry_reopen from retrying
with readers and writers working in parallel:
FOO=imaps://example.com/INBOX.huge
lei lcat $FOO -f mboxcl | lei tag -F mboxcl +L:bar -
Fixes: c7bcfe6cd6648ff0 ("lei: diagnostics for /Document \d+ not found/ errors")
|
|
Credentials sourced via ~/.netrc should not be written to
git-credential.
|
|
We may be handling invalid mboxes, so just return no objects in
that case. While "lei q" on HTTP(S) externals expects a gzipped
mboxrd, there's always a chance something else gzipped can be
sent to us.
There's also changes to lei_to_mail to better handle emails
which lack a body and/or headers (e.g. t/solve/bare.patch)
Link: https://public-inbox.org/meta/20210903151500.h72mzcpqixgtytjs@meerkat.local/
|
|
xt/net_writer-imap.t was completely broken in recent months and
I completely forgot this test. net->add_url still only accepts
bare scalars (and not scalar refs), so we must set that up
properly. Furthermore, our changes to do FLAGS-only
synchronization in lei of old messages was causing us to not
handle FLAGS properly for the test.
|
|
We need to use LeiSearch->qparse_new to handle (and filter out)
"L:" and "kw:" search prefixes to avoid hitting false positives
when externals are involved. Unfortunately, this doesn't work
for remote HTTP(S) externals, but those aren't enabled by
default.
|
|
This can be useful inside mutt since I was diagnosing why
a label ("L:$FOO") search was giving me a false-positive
search result...
|
|
This is merely to avoid perl setting errors internally which
were not user visible. The double-close wasn't a problem in
practice since we open a new file hanlde for the mbox or
mbox.gz anyways, so the new t/lei-up.t test case shows no
regressions nor fixes.
|