Date | Commit message (Collapse) |
|
There's no need to alias net_merge_all in each WQ class
which uses LeiAuth, `$obj->$sub' works even when `$sub'
is a fully-qualified subroutine name with `::' in it.
perlobj(1) documents it under "Method Call Variations".
|
|
It turns out this step is unnecessary, since SOCK_SEQPACKET
ordering is guaranteed and we know wq_broadcast calls will
always be handled sequentially.
|
|
Before making potentially major changes, lets clarify readers'
understanding of how LeiAuth currently works.
|
|
The deeper eval was preventing retry_reopen from retrying
with readers and writers working in parallel:
FOO=imaps://example.com/INBOX.huge
lei lcat $FOO -f mboxcl | lei tag -F mboxcl +L:bar -
Fixes: c7bcfe6cd6648ff0 ("lei: diagnostics for /Document \d+ not found/ errors")
|
|
Credentials sourced via ~/.netrc should not be written to
git-credential.
|
|
We may be handling invalid mboxes, so just return no objects in
that case. While "lei q" on HTTP(S) externals expects a gzipped
mboxrd, there's always a chance something else gzipped can be
sent to us.
There's also changes to lei_to_mail to better handle emails
which lack a body and/or headers (e.g. t/solve/bare.patch)
Link: https://public-inbox.org/meta/20210903151500.h72mzcpqixgtytjs@meerkat.local/
|
|
xt/net_writer-imap.t was completely broken in recent months and
I completely forgot this test. net->add_url still only accepts
bare scalars (and not scalar refs), so we must set that up
properly. Furthermore, our changes to do FLAGS-only
synchronization in lei of old messages was causing us to not
handle FLAGS properly for the test.
|
|
We need to use LeiSearch->qparse_new to handle (and filter out)
"L:" and "kw:" search prefixes to avoid hitting false positives
when externals are involved. Unfortunately, this doesn't work
for remote HTTP(S) externals, but those aren't enabled by
default.
|
|
This can be useful inside mutt since I was diagnosing why
a label ("L:$FOO") search was giving me a false-positive
search result...
|
|
This is merely to avoid perl setting errors internally which
were not user visible. The double-close wasn't a problem in
practice since we open a new file hanlde for the mbox or
mbox.gz anyways, so the new t/lei-up.t test case shows no
regressions nor fixes.
|
|
We can golf out some code and refcounts this way.
|
|
I was calling "child_error(1, ...)" in a few places where I meant
to be calling "child_error(1 << 8, ...)" and inadvertantly
triggering SIGHUP in script/lei. Since giving a zero exit code
to child_error makes no sense, just allow falsy values to
default to 1 << 8.
|
|
ENOENT can be too common due to timing and concurrent access
from MUAs and "lei export-kw", and other mail synchronization
tools (e.g. mbsync and offlineimap).
|
|
Dumping errors from the previous run can often get lost, so just
spew to syslog since it's a standard place to put errors that
don't make it to a client. Note: we don't rely on $SIG{__WARN__}
since some of the Net:: stuff will write directly to STDERR
(as will external processes).
|
|
Apparently URLs can be configured relatively for HTTP(S) setups,
attempt to support them when linking to cross-posted messages.
This also fixes the top-row (mirror/help/color/Atom feed) links
in /$INBOX_URL/$EXTMSG_MSGID/T/ (and /t/) URLs.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210902191239.cmbxlmjqcsmdzmqp@meerkat.local/
|
|
Sharing a single lei-daemon across multiple processes still
exhibits reliability problems, and reliably checking
lei-daemon's inotify internals seems impossible without.
Even without lei-daemon sharing, "make check-run" is a few
seconds faster than "make check" for me.
|
|
On slower systems, even a 100ms delay may not be enough;
so loop and retry in hopes of an early exit for faster
systems.
|
|
This works with existing inotify/EVFILT_VNODE functionality to
propagate changes made from one Maildir to another Maildir.
I chose the lei/store worker process to handle this since
propagating changes back into lei-daemon on a massive scale
could lead to dead-locking while both processes are attempting
to write to each other. Eliminating IPC overhead is a nice
side effect, but could hurt performance if Maildirs are slow.
The code for "lei export-kw" is significantly revamped to match
the new code used in the "lei/store" daemon. It should be more
correct w.r.t. corner-cases and stale entries, but perhaps
better tests need to be written.
squashed:
t/lei-auto-watch: increase delay for FreeBSD kevent
My FreeBSD VM seems to need longer for this test than inotify
under Linux, likely because the kevent support code needs to be
more complicated.
|
|
This will be needed as we track changes in real-time, especially
for "lei index" since there's no storage involved.
|
|
For lei-index to work in parallel with MUA access and upcoming
inotify-based updates, mail_sync.sqlite3 needs to always be
up-to-date to read-only worker processes (ahead of everything
else). So rely on the default auto-commit behavior and hope
SQLite WAL can reduce some of the overheads involved with
writes.
|
|
While messages from removed inboxes were removed from Xapian
search, --gc failed to remove messages from over.sqlite3
entirely. They no longer show up in the topic summary view.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/20210830201723.dehoul4y6gpqf2cp@nitro.local/
|
|
Cache is expensive, so help users save on storage by documenting
this behavior.
|
|
->DESTROY can get triggered in child processes, which
unnecessarily duplicates messages queued up for display
when lei spawns extra workers.
|
|
Some of these errors were inadvertantly lost due to delayed
error reporting in the past.
|
|
We must set autoflush to ensure timely notification of clients;
and lei-daemon must not block when waiting on reads in case of
spurious wakeups.
Finally, if no clients are connected to lei-daemon, write to
syslog to ensure the error is visible.
|
|
Another step towards moving more of our internals to use binary
OIDs to avoid needless conversions before hitting disk.
|
|
If we possibly just wrote or created a Maildir, ensure it's
monitored by the lei watch mechanism.
|
|
Because the timer may not fire in time before daemon shutdown.
|
|
Open file handles in lei-daemon may be unstable so we need to
account for readlink() returning undef.
|
|
We need to account for past canonicalization errors and deal
with cases which violate uniqueness constraints in
mail_sync.sqlite3
|
|
No need to loop when we can rely on grep.
|
|
This still needs tests, but I noticed "--all" w/o "local" or
"remote" was not working correctly since split() returned
an empty array.
|
|
No need to bump refcounts of {dbh} nor declare extra variables
for a rarely-called function.
|
|
Perhaps this can be expanded to include grokmirror information
in the future. For now, just give a hint about the "mirror"
link for each inbox.
|
|
"extindex" and "public-inbox" are project-specific terms which
are probably unsuitable for folks who are seeing this for the
first time.
Use "public inbox" when referring to actual public inboxes,
since "public-inbox" is merely the name for this particular
implementation and others have adopted the same concept (IMHO
the concept is more important than any particular
implementation).
|
|
This may be redundant with the "mirror" link at the top right,
but maybe people will miss one. Properly capitalize the
"Code repositories" text while we're at it.
Link: https://public-inbox.org/20210828175827.rgzwqbn7brl56oej@nitro.local/
Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Making the inbox description link back to the most recent
per-inbox topics from text/ and $OID/s/ URLs seems useful,
rather than keeping the description up there.
Followup-to: 6c853f5256f3a324 ("www: improve navigation around contemporary threads")
|
|
This makes the mirroring and code retrieval instructions less
obstructive. Relying on WwwText means we only use our Linkify
module to make hrefs of full URLs; making relative and shortened
hrefs off-limits; hopefully this isn't too much of a problem.
coderepo information remains duplicated on every page since
(IMHO) coderepos are an important feature; but nobody besides me
has ever bothered to configure coderepos, so I suppose it's
fine...
Suggested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210826132747.6gxuwnhftyf7c6hp@nitro.local/
|
|
This may fix problems with the "all" link disappearing.
Link: https://public-inbox.org/meta/CAMwyc-Tw=v5yT1U1U66GSwwTK8OJXv8_YDu-=oXbZO3tHSnYWw@mail.gmail.com/
|
|
Searching inboxes with an empty query no longer gives 500 errors
due to Xapian. Also, improve the error message when no inboxes
match, since saying no inboxes exist yet is wrong.
|
|
It's a special case and we can show it in the HTML display
without affecting manifest.js.gz generation.
|
|
Since we favor ->over in WWW and IMAP, move this method to
->over to reduce open files in common cases.
This fixes the /$EXTINDEX_NAME/all.mbox.gz endpoint for extindex
entries (which may get expensive...).
|
|
At least manually configured coderepos "just work"
for extindex, though it probably could be automatic
and inherited from the publicinbox configs.
|
|
There's currently no support for altid with extindex, and
there's likely no legacy precedent for using altid like there is
with single public-inboxes.
|
|
extindex doesn't use the same config stuff as normal
"publicinbox" entries, so we'll need a separate function
for them.
|
|
There's no way to clone an extindex, since there's no git
storage associated with them. So attempt to link to the
HTML listing of public-inboxes, instead.
|
|
The long v3 .onion URL was causing havoc on small mobile
displays, so extract "hostname" into a variable which can
still used as a Bourne shell snippet.
While we're at it, include "torsocks" in the git command used
for .onion URLs since that's the (near)-universal wrapper for
Tor-ifying things (like git) which are dynamically linked to
libc.
Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210816163654.c6gfzuezhji4l6s7@nitro.local/
|
|
The use of substr within IO::Handle->write may not be correct if
we have wide characters, so handle it ourselves.
bytes.pm usage is probably better fixed in PublicInbox::NNTP,
but the effort required is higher, so we'll just keep bytes in
DS for now.
|
|
The only place where we could return wide characters with -httpd
was the raw $INBOX_DIR/description text, which is now converted
to octets.
All daemon (HTTP/NNTP/IMAP) sockets are opened in binary mode,
so length() and bytes::length() are equivalent on reads. For
socket writes, any non-octet data would warn about wide characters
and we are strict in warnings with test_httpd.
All gzipped buffers are also octets, as is PublicInbox::Eml->body,
and anything from PerlIO objects ("git cat-file --batch" output,
filesystems), so bytes::length was unnecessary in all those places.
|
|
Since CSS can be overridden by a static webserver on a per-inbox
basis, we need a similar pattern to deal with the instance-wide
WwwListing HTML. "/+/" probably won't conflict with any current
nor future public inbox names.
I don't think it'll cause problems with common linkifiers or URL
extractors, either (and it's unlikely anybody would want to
share URLs of just the CSS in a plain text(-like) format).
|