| Date | Commit message (Collapse) |
|---|
|
It's cheaper to check for duplicates than run `spamc'
repeatedly when rechecking. We already do this for
v1 with by using the "ls" command with fast-import,
but v2 requires checking against over.sqlite3.
|
|
We won't be attempting to reuse Mail::IMAPConnections used to
check authentication info, for now, so stop storing
$self->{mics}.
We can also combine $poll initialization for IMAP and NNTP
to avoid data structure duplication. Furthermore, rely on
autovivification to create {idle_pids} and {poll_pids}.
|
|
SQLite only issues non-blocking F_SETLK ops (not F_SETLKW) and
retries failures using a configurable busy_timeout. SQLite's
busy loop sleeps for a millisecond and retries the lock until
the configured busy_timeout is hit.
Trying to set ->sqlite_busy_timeout to larger values (e.g. 30000
milliseconds) still leads to failure when running the new stress
test with 8 processes with TMPDIR on a 7200 RPM HDD.
Inspection of SQLite source reveals there's no built-in way to
use F_SETLKW, so tack on the existing flock(2) support we use to
synchronize git + SQLite + Xapian for inbox writing. We use
flock(2) instead of POSIX fcntl(2) locks since Perl doesn't
provide a way to manipulate "struct flock" portably.
|
|
While git-credential-netrc exists in git.git contrib/, it may
not be widely known or installed. Net::Netrc is already a
standard part of most (if not all) Perl installations, so use it
directly if available.
|
|
Git.pm may not be installed on some systems; or some users have
multiple Perl installations and Git.pm is not available to the
Perl running -watch. Accomodate both those types of users by
providing our own "git credential" wrapper.
|
|
In case output is redirected to a pipe, ensure stdout and stderr
are always unbuffered, as -watch may go long periods without
any output to fill up buffers.
|
|
Since we use the non-ref scalar URL in many error messages,
favor keeping the unblessed URL in the long-lived process.
This avoids showing "snews://" to users who've specified
"nntps://" URLs, since "nntps" is IANA-registered nowadays and
what we show in our documentation, while "snews" was just a
draft the URI package picked up decades ago.
|
|
This is similar to IMAP support, but only supports polling.
Automatic altid support is not yet supported, yet; but may
be in the future.
v2: small grammar fix by Kyle Meyer
Link: https://public-inbox.org/meta/87sgeg5nxf.fsf@kyleam.com/
|
|
Existing use of the $ENV{TAIL} relied on parsing --std{out,err},
which was only usable for read-only daemons. However, -watch
doesn't use PublicInbox::Daemon code(*), so attempt to figure
out redirects.
(*) -watch won't able to run as a daemon in cases when
git-credential prompts for IMAP/NNTP passwords.
PublicInbox::Daemon is also designed for read-only
parallelism where all worker processes are the same.
Any subprocesses spawned by -watch are to do specific
tasks for a particular set of inboxes.
|
|
We may just modify PublicInbox::Config->urlmatch in the future
to support git <1.8.5, but I wonder if there's enough users on
git <1.8.5 to justify it.
|
|
Since we store all watched directory names as keys in %mdmap,
there should be no need to keep an array of those directories
around.
t/watch_maildir*.t required changes to remove trained spam.
Once we've trained something as spam, there shouldn't be
a need to rescan it.
|
|
Some users will find it useful to merge several Maildir or
IMAP mailboxes into one public-inbox. Let them do it, since
we've always supported multi-address inboxes.
|
|
If ->idle_done was handled successfully, we can just
let normal ->DESTROY disconnect and avoid ugly backtraces
when a user hits Ctrl-C to take down the process group.
|
|
IMAP allows retrieving multiple messages with a single command,
and Mail::IMAPClient supports that. Unfortunately, it means we
slurp multiple messages into memory at once. This option allows
users to trade off memory usage to reduce network round-trips.
Ideally, we'd support pipelining; but AFAIK no widely installed
Perl IMAP library supports it.
|
|
With different polling intervals, multiple processes may
simultaneously write to IMAPtracker. This ought to reduce
SQLite busy waiting and contention issues when importing
many inboxes in parallel.
|
|
It's not used anywhere since the IMAPTracker object doesn't
disconnect and reconnect. If we ever need the filename,
{dbh}->sqlite_db_filename may be used.
Cc: Eric W. Biederman <ebiederm@xmission.com>
|
|
Passing a $url parameter to every function was error-prone
and having {url} field for a short-lived object is appropriate.
This matches the version of IMAPTracker posted by
Eric W. Biederman on 2020-05-15 at:
https://public-inbox.org/meta/87ftc0c3r4.fsf_-_@x220.int.ebiederm.org/
The version I originally imported was based on the one
posted on 2019-10-09:
https://public-inbox.org/meta/874l0i9vhc.fsf_-_@x220.int.ebiederm.org/
Cc: Eric W. Biederman <ebiederm@xmission.com>
|
|
This allows callers to avoid creating expensive closures.
We no longer pass the `$now' value to callers, as none of
the callers used it.
|
|
For mailboxes with many gaps in the UID sequence,
performing a UID SEARCH beforehand can reduce the
number of articles to fetch.
However, the downside to this is we may end up with
an arbitrarly large list of UIDs from the server.
|
|
This fixes cases where watch is handling both Maildirs and IMAP
connections. While we're at it, close open directories in the
IMAP children to save FDs.
|
|
Since we have IMAP client support in -watch; make sure per-URL
settings are familiar to git users by taking advantage of git's
URL matching abilities.
This requires git 1.8.5+, which most users ought to have
(though base CentOS 7 is on 1.8.3).
|
|
Not all IMAP servers support IDLE, and IDLE may be prohibitively
expensive for some IMAP servers with many inboxes. So allow
configuring a imap.$IMAP_URL.pollInterval=SECONDS to poll
mailboxes.
We'll also need to poll for NNTP servers in the future.
|
|
We can avoid synchronous `waitpid(-1, 0)' and save a process
when simultaneously watching Maildirs.
One DS bug is fixed: ->Reset needs to clear the DS $in_loop flag
in forked children so dwaitpid() fails and allows git processes
to be reaped synchronously. TestCommon also calls DS->Reset
when spawning new processes, since t/imapd.t uses DS->EventLoop
while waiting on -watch to write.
|
|
Since the removal of pseudo-hash support in Perl 5.10, the
"fields" module no longer provides the space or speed benefits
it did in 5.8. It also does not allow for compile-time checks,
only run-time checks.
To me, the extra developer overhead in maintaining "use fields"
args has become a hassle. None of our non-DS-related code uses
fields.pm, nor do any of our current dependencies. In fact,
Danga::Socket (which DS was originally forked from) and its
subclasses are the only fields.pm users I've ever encountered in
the wild. Removing fields may make our code more approachable
to other Perl hackers.
So stop using fields.pm and locked hashes, but continue to
document what fields do for non-trivial classes.
|
|
We can get rid of the janky wannabe
self-using-a-directory-instead-of-pipe thing we needed to
workaround Filesys::Notify::Simple being blocking.
For existing Maildir users, this should be more robust and
immune to missed wakeups for signalfd and kqueue-enabled
systems; as well as being immune to BOFHs clearing $TMPDIR
and preventing notifications from firing.
The IMAP IDLE code still uses normal Perl signals, so it's still
vulnerable to missed wakeups. That will be addressed in future
commits.
|
|
Since we already use inotify and EVFILT_VNODE (kqueue)
in -imapd, we might as well use them directly in -watch,
too.
This will allow public-inbox-watch to use PublicInbox::DS
for timers to watch newsgroups/mailboxes and have saner
signal handling in future commits.
|
|
We need to detect link(2) and rename(2) in other apps
writing to the Maildir.
We'll be removing the Filesys::Notify::Simple from -watch
in favor of using IO::KQueue or Linux::Inotify2 directly.
Ensure non-inotify emulations can support everything we
expect for Maildir writers.
|
|
Only servers with IDLE are supported, for now. Polling will
be needed since users may need to watch many inboxes with
a few active connections due to IMAP server limitations.
|
|
We'll be supporting the IMAP URL scheme described in RFC 5092
for -watch, so add this module to fill in what the `URI' package
lacks.
|
|
The old check was ineffective since we process the spam folder
config before ham inboxes; and would only fail when attempting
to treat the scalar "watchspam" string as an array ref.
|
|
It's too deeply indented, and we will be using it for IMAP, too.
|
|
Respect XDG_DATA_HOME to avoid cluttering ~/.public-inbox/.
Existing users of ~/.public-inbox/imap.sqlite3 will remain
supported, but the preference for new data is to use
~/.local/share and other paths standardized by XDG.
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
This removes the need to delete from an imap mailbox when
downloading it's messages.
[ew: minor style changes]
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Otherwise, we may never know what went wrong.
|
|
This will allow us to use InboxIdle on empty/unindexed v1 inboxes.
|
|
Apparently, using $1 from an octet string still results in a
multi-byte string. Thus we need to perform utf8::encode after
the regexp character match to ensure wide characters don't get
passed to encode_base64.
This fixes a bug in which caused -watch to crash when using
PublicInbox::Filter::SubjectTag to remove "[list prefix]"
tags from Subject: lines.
I only found this bug because the proposed -watch updates for
NNTP/IMAP support introduced a possible bug which triggered a
full rescan of old archives:
https://public-inbox.org/meta/20200627100400.9871-1-e@yhbt.net/
|
|
Since we'll have an IMAP server released soon, maybe imaps://
and imap:// URLs can become popular.
news:// is defined with nntp:// in RFC 5538, and we can at least
support the news:// form in rendered HTML. snews:// may appear
in old mail archives, too, so we'll attempt to support it in
case clients do.
|
|
We need to rely on num_highwater for UIDNEXT since the
highest `num' stored in over.sqlite3 may be rolled back
if the most recent messages were spam.
We also need to load the uo2m immediately on EXAMINE to ensure
EXISTS responses are always consistent with regard to future
updates.
|
|
Clients which are NOT in an IDLE state still need to be
notified of message existence. Unlike the EXPUNGE response,
untagged EXISTS responses seem to be allowed at any time
according to RFC 3501.
We'll also perform uo2m_extend on the NOOP command, since
NOOP is the recommended command for message polling.
|
|
This quiets warnings from IMAP::fetch_blob (called via
long_response) failing to access `$self->{ibx}->git'
because ->{ibx} gets deleted by IMAP::close.
|
|
While this circular reference was carefully managed to not leak
memory; it was still triggering a warning at -imapd/-nntpd
shutdown due to the EPOLL_CTL_DEL op failing after the $Epoll FD
gets closed.
So remove the circular reference by providing a ref to `undef',
instead.
|
|
We can reduce the amount of platform-specific code by always
relying on IN_MODIFY/NOTE_WRITE notifications from lock release.
This reduces the number of times our read-only daemons will
need to wake up when -watch sees no-op message changes
(e.g. replied, seen, recent flag changes).
|
|
This fixes a bug on FreeBSD 11 here -nntpd + TEST_RUN_MODE=2
(default) was occasionally causing failures in t/v2writable.t
due to the kqueue descriptor being auto-closed by the OS on fork.
|
|
I used find(1) here for debugging. The "make check-run" test
target needs to be updated to make stderr spew more obvious.
|
|
For archivists with only newer mail archives, this option allows
reserving reserve NNTP article numbers for yet-to-be-archived
old messages. Indexers will need to be updated to support this
feature in future commits.
-V1 inboxes will now be initialized with SQLite and Xapian
support if this option is used, or if --indexlevel= is
specified.
|
|
Since V2 uses multiple git repositories, stop using
the word "repo" when referring to inboxes.
|
|
On a powerful (by my standards) machine with 16GB RAM and an
7200 RPM HDD marketed for "enterprise" use, indexing a 8.1G (in
git) LKML snapshot from Sep 2019 did not finish after 7 days
with the default number (3) of Xapian shards (`--jobs=4') and
`--batch-size=10m'.
Indexing starts off fast, but progressively get slower as
contents of the inbox (including Xapian + SQLite DBs) could no
longer be cached by the kernel. Once the on-disk size
increased, HDD seek contention between the Xapian shard workers
slowed the process down to a crawl.
With a single shard, it still took around 3.5 days to index on
the HDD. That's not good, but it's far better than not
finishing after 7 days. So allow unfortunate HDD users to
easily specify a single shard on public-inbox-init.
For reference, a freshly TRIM-ed low-end TLC SSD on the SATA II
bus on the same machine indexes that same snapshot of LKML in
~7 hours with 3 shards and the same 10m batch size. In the past,
a higher-end consumer grade MLC SSDs on similar hardware indexed
a similarly sized-data set in ~4 hours.
|
|
There's no need to loop when the first iteration guarantees
a `return'.
|
|
We also need to check for git 2.6 earlier in each test case,
before any other TAP output is emitted to avoid confusing the
TAP consumers.
|
|
Net::NNTP does not support older IO::Socket::SSL. 1.94 on
CentOS 7.x fails HTTPS and IMAPS tests, too.
cf. https://rt.cpan.org/Ticket/Display.html?id=100529
|