| Date | Commit message (Collapse) |
|---|
|
The good news (compared to lei) is we only have to worry about
imports and don't care about the filename nor keywords, so it's
immune to .mh_sequences writing inconsistencies across MH
implementations and sequence number packing.
We still assume the writer will write the mail file with one of:
* rename(2) to create the final sequence number filename
* a single write(2) if not relying on rename(2)
mlmmj and mutt satisfy these requirements. Python's Lib/mailbox.py
may, I'm not sure...
|
|
This is a step towards improving the out-of-the-box experience
in achieving notifications without XS, extra downloads, and .so
loading + runtime mmap overhead.
This also fixes loongarch support of all Linux syscalls due to
a bad regexp :x
All the reachable Linux architectures listed at
<https://portal.cfarm.net/machines/list/> should be supported.
At the moment, there appears to be no reachable sparc* Linux
machines available to cfarm users.
Fixes: b0e5093aa3572a86 (syscall: add support for riscv64, 2022-08-11)
|
|
For users hosting read-only mirrors (via clone|fetch) and feeding
inboxes via -watch
|
|
List-Unsubscribe headers with unique identifiers (such as those
generated by our examples/unsubscribe.milter) should not
end up in public archives. Add a new config knob to strip
List-Unsubscribe headers if they have the
`List-Unsubscribe-Post: List-Unsubscribe=One-Click'
header.
Unfortunately, this breaks DKIM signatures if the signature
covers either of these List-Unsubscribe* headers. However,
breaking DKIM is the lesser evil compared to any archive reader
being able to stop archival by an independent archivist.
As much as I would like this to be the default, it probably
affects few users at the moment since very few mailing lists
use unique identifiers in List-Unsubscribe (but that number
has grown, recently).
|
|
It's a needless branch to maintain exclusively for our tests.
The `git config -l' output isn't pleasant to write in tests,
anyways. So just use heredocs to write git configs in their
native format rather than emulate the output of `git config -l'.
This does make the test suite do more work with temporary files
and process invocations, but it doesn't seem very measurable
when testing on tmpfs (TMPDIR=/dev/shm).
We'll make a minor improvement to TestCommon::tmpdir by allowing
it to return a single value (which I suspect we can rely on in
more places since File::Temp::Dir overloads stringification).
|
|
public-inbox-watch, lei-daemon, the master process of
public-inbox-(netd|httpd|imapd|nntpd|pop3d),
and the (mostly) Perl implementation of XapHelper do not
have many FDs to watch so epoll|kqueue end up being overkill.
Of course, *BSDs already have separate kqueue FDs emulating
signalfd and/or inotify, even.
In other words, only the worker processes of
public-inbox-(netd|httpd|imapd|nntpd|pop3d) are expected
to see C10K (or C100K) types of traffic where epoll|kqueue
shine.
Perhaps lei could benefit from epoll/kqueue on some virtual users
IMAP/JMAP system one day; as could -watch with many IMAP IDLE
folders; but we'll probably add a knob if/when it comes to that.
|
|
This allows us to eliminate the workaround of respawning
`git cat-file', too :x
|
|
This allows us to avoid repeatedly using memory-intensive
anonymous subs in CodeSearchIdx where the callback is assigned
frequently. Anonymous subs are known to leak memory in old
Perls (e.g. 5.16.3 in enterprise distros) and still expensive in
newer Perls. So favor the (\&subroutine, @args) form which
allows us to eliminate anonymous subs going forward.
Only CodeSearchIdx takes advantage of the new API at the moment,
since it's the biggest repeat user of post-loop callback
changes.
Getting rid of the subroutine and relying on a global `our'
variable also has two advantages:
1) Perl warnings can detect typos at compile-time, whereas the
(now gone) method could only detect errors at run-time.
2) `our' variable assignment can be `local'-ized to a scope
|
|
|
|
Since signalfd is often combined with our event loop, give it a
convenient API and reduce the code duplication required to use it.
EventLoop is replaced with ::event_loop to allow consistent
parameter passing and avoid needlessly passing the package name
on stack.
We also avoid exporting SFD_NONBLOCK since it's the only flag we
support. There's no sense in having the memory overhead of a
constant function when it's in cold code.
|
|
Using "make update-copyrights" after setting GNULIB_PATH in my
config.mak
|
|
{pi_config} may be confused with the documented `PI_CONFIG'
environment variable, and we'll favor vowel-removal to be
consistent with our usage of object references.
The `pi_' prefix may stay in some places, for now; since a
separate namespace may come into this codebase for local/private
client-tooling.
For InboxIdle, we'll also remove an invalid comment about
holding a reference to the PublicInbox::Config object, too.
|
|
This is no longer limited to Maildirs now that IMAP and NNTP
support exist; so give it a shorter name.
|
|
Since we store all watched directory names as keys in %mdmap,
there should be no need to keep an array of those directories
around.
t/watch_maildir*.t required changes to remove trained spam.
Once we've trained something as spam, there shouldn't be
a need to rescan it.
|
|
We can get rid of the janky wannabe
self-using-a-directory-instead-of-pipe thing we needed to
workaround Filesys::Notify::Simple being blocking.
For existing Maildir users, this should be more robust and
immune to missed wakeups for signalfd and kqueue-enabled
systems; as well as being immune to BOFHs clearing $TMPDIR
and preventing notifications from firing.
The IMAP IDLE code still uses normal Perl signals, so it's still
vulnerable to missed wakeups. That will be addressed in future
commits.
|
|
Since we already use inotify and EVFILT_VNODE (kqueue)
in -imapd, we might as well use them directly in -watch,
too.
This will allow public-inbox-watch to use PublicInbox::DS
for timers to watch newsgroups/mailboxes and have saner
signal handling in future commits.
|
|
The old check was ineffective since we process the spam folder
config before ham inboxes; and would only fail when attempting
to treat the scalar "watchspam" string as an array ref.
|
|
We no longer load or use Email::MIME outside of comparison
tests.
|
|
Instead, favor PublicInbox::MIME->new for non-attachment emails.
We may support alternatives to Email::MIME down the line.
We'll still keep Email::MIME->create to deal with attachments,
for now, but there's also a fair amount of test duplication
we should eliminate, later.
|
|
Allowing ->init_bare to be used as a method saves some
keystrokes, and we can save a little bit of time on systems with
our vfork(2)-enabled spawn().
This also sets us up for future improvements where we can
avoid spawning a process at all.
|
|
I didn't wait until September to do it, this year!
|
|
This cuts down on lines of code in individual test cases and
fixes some misnamed error messages by using "$0" consistently.
This will also provide us with a method of swapping out
dependencies which provide equivalent functionality (e.g
"Xapian" SWIG can replace "Search::Xapian" XS bindings).
|
|
We want to be able to use run_script with *.t files, so
t/common.perl putting subs into the top-level "main" namespace
won't work. Instead, make it a module which uses Exporter
like other libraries.
|
|
We'll also introduce a tmpdir() API to give tempdirs
consistent names.
|
|
The "strict" pragma makes code easier to debug, and we had
undeclared variables as a result in t/watch_maildir_v2.t.
So use it everywhere to be consistent with the rest of our
code.
|
|
"mainrepo" ws a bad name and artifact from the early days when I
intended for there to be a "spamrepo" (now just the
ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be
especially confusing, since v2 needs at least two git
repositories (epoch + all.git) to function and we shouldn't
confuse users by having them point to a git repository for v2.
Much of our documentation already references "INBOX_DIR" for
command-line arguments, so use "inboxdir" as the
git-config(1)-friendly variant for that.
"mainrepo" remains supported indefinitely for compatibility.
Users may need to revert to old versions, or may be referring
to old documentation and must not be forced to change config
files to account for this change.
So if you're using "mainrepo" today, I do NOT recommend changing
it right away because other bugs can lurk.
Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
|
|
Rewrite a bunch of tests to use ordered input (emulating
"git config -l" output) so we can always walk sections in
the order they were given in the config file.
|
|
|
|
Remove redundant slashes while we're at it.
|
|
I had to dig through commit history for this and we should
better document our tests (along with everything else).
|
|
Using update-copyrights from gnulib
While we're at it, use the SPDX identifier for AGPL-3.0+ to
ease mechanical processing.
|
|
We need to ensure new messages are being processed
fairly during full rescans, so have the ->scan subroutine
yield and reschedule itself. Additionally, having a
long-running task inside the signal handler is dangerous
and subject to reentrancy bugs.
Due to the limitations of the Filesys::Notify::Simple interface,
we cannot rely on multiplexing I/O interfaces (select, IO::Poll,
Danga::Socket, etc...) for this. Forking a separate process
was considered, but it is more expensive for a mostly-idle
process.
So, we use a variant of the "self-pipe trick" via inotify (or
whatever Filesys::Notify::Simple gives us). Instead of writing
to our own pipe, we write to a file in our own temporary
directory watched by Filesys::Notify::Simple to trigger events
in signal handlers.
|
|
Probably better than bloating our own API with configurable
warning streams and such...
|
|
Mailing lists I watch and mirror may not have the best spam
filtering, and an extra layer should not hurt.
|
|
And improve documentation for existing dependencies, too.
|
|
Only mark seen messages as spam, otherwise it could be
too aggressive and cause problems or over training.
We wouldn't want a wayward FIFO ruining our day, either :)
|
|
We can support spam removal by watching a special "spam"
Maildir, too. We can run public-inbox-learn as a separate
step, and that command will be improved to support
auto-learning, too.
|
|
This should be portable despite the intended use of this
directory being non-portable.
|