| Date | Commit message (Collapse) |
|---|
|
It'll give us a nicer HTML header and footer.
|
|
No point in streaming a tiny response via ->getline,
but we may stream to a gzipped buffer, later.
|
|
Most of our plain-text responses are config files
big enough to warrant compression.
|
|
Our most common endpoints deserve to be gzipped.
|
|
Plack::Middleware::Deflater forces us to use a memory-intensive
closure. Instead, work towards building compressed strings in
memory to reduce the overhead of buffering large HTML output.
|
|
We currently don't use bytes::length in ->write, so there's no
need to `use bytes'. Favor `//=' to describe the intent of the
conditional assignment since the C::R::Z::Deflate object is
always truthy. Also use the local $gz variable to avoid
unnecessary {gz} hash lookups.
|
|
We avoided a managed circular reference in 10ee3548084c125f
but introduced a pipe FD leak, instead. So handle the EOF
we get when the "git cat-file --batch" process exits and
closes its stdout FD.
v2: remove ->close entirely. PublicInbox::Git->cleanup
handles all cleanup. This prevents us from inadvertantly
deleting the {async_cat} field associated with a different
pipe than the one GAC is monitoring.
Fixes: 10ee3548084c125f ("git_async_cat: remove circular reference")
|
|
inotify_add_watch(2), open(2), stat(2) may all fail due to
permissions errors, especially when running -nntpd/-imapd
as `nobody' as recommended.
|
|
Network connections fail and need to be detected sooner rather
than later during IDLE to avoid backtrace floods. In case the
IDLE process dies completely, don't respawn right away, either,
to avoid entering a respawn loop.
There's also a typo fix :P
|
|
We no longer use writev(2) in pi_fork_exec to emit errors.
|
|
I was wondering about this myself the other day and had to read
up on it. So make a note of it for future readers.
|
|
The default (and fast) TEST_RUN_MODE=2 preloads most modules,
but TEST_RUN_MODE=0 is more realistic and can catch some
problems which may show up in real-world use.
|
|
To ensure reliable signal delivery in Perl, it seems we need to
repeatedly signal processes which aren't using signalfd (or
EVFILT_SIGNAL) with our event loop.
|
|
parent.pm is smaller than base.pm, and we'll also move
towards relying on `-w' (or not) to toggle process-wide
warnings during development.
|
|
Making the RLIMITS list a function doesn't allow constant
folding, so just make it an array accessible to other modules.
|
|
Anonymous subs cost over 5K each on x86-64. So prefer the
less-recommended-but-still-documented way of using
Linux::Inotify2::watch to register watchers.
This also updates FakeInotify to detect modifications correctly
when used on systems with neither IO::KQueue nor
Linux::Inotify2.
|
|
Maildir scanning still happens in the main process. Scanning
dozens of Maildirs is still time-consuming and monopolizes the
event loop during WatchMaildir::event_step. This can cause
cause zombies to accumulate before Sigfd::event_step triggers
DS::reap_pids.
|
|
Subprocess we spawn may want to use SIGCHLD for themselves.
This also ensures we restore default signal handlers
in the pure Perl version.
|
|
In case our git or spam checker subprocesses spawn
subprocesses of their own. We'll also ensure signal
handlers are properly setup before unblocking them.
|
|
It could be useful to see warnings generated for known problematic
messages just as it is for possibly non-problematic ones.
|
|
It's cheaper to check for duplicates than run `spamc'
repeatedly when rechecking. We already do this for
v1 with by using the "ls" command with fast-import,
but v2 requires checking against over.sqlite3.
|
|
We won't be attempting to reuse Mail::IMAPConnections used to
check authentication info, for now, so stop storing
$self->{mics}.
We can also combine $poll initialization for IMAP and NNTP
to avoid data structure duplication. Furthermore, rely on
autovivification to create {idle_pids} and {poll_pids}.
|
|
SQLite only issues non-blocking F_SETLK ops (not F_SETLKW) and
retries failures using a configurable busy_timeout. SQLite's
busy loop sleeps for a millisecond and retries the lock until
the configured busy_timeout is hit.
Trying to set ->sqlite_busy_timeout to larger values (e.g. 30000
milliseconds) still leads to failure when running the new stress
test with 8 processes with TMPDIR on a 7200 RPM HDD.
Inspection of SQLite source reveals there's no built-in way to
use F_SETLKW, so tack on the existing flock(2) support we use to
synchronize git + SQLite + Xapian for inbox writing. We use
flock(2) instead of POSIX fcntl(2) locks since Perl doesn't
provide a way to manipulate "struct flock" portably.
|
|
While git-credential-netrc exists in git.git contrib/, it may
not be widely known or installed. Net::Netrc is already a
standard part of most (if not all) Perl installations, so use it
directly if available.
|
|
Git.pm may not be installed on some systems; or some users have
multiple Perl installations and Git.pm is not available to the
Perl running -watch. Accomodate both those types of users by
providing our own "git credential" wrapper.
|
|
Since we use the non-ref scalar URL in many error messages,
favor keeping the unblessed URL in the long-lived process.
This avoids showing "snews://" to users who've specified
"nntps://" URLs, since "nntps" is IANA-registered nowadays and
what we show in our documentation, while "snews" was just a
draft the URI package picked up decades ago.
|
|
This is similar to IMAP support, but only supports polling.
Automatic altid support is not yet supported, yet; but may
be in the future.
v2: small grammar fix by Kyle Meyer
Link: https://public-inbox.org/meta/87sgeg5nxf.fsf@kyleam.com/
|
|
Existing use of the $ENV{TAIL} relied on parsing --std{out,err},
which was only usable for read-only daemons. However, -watch
doesn't use PublicInbox::Daemon code(*), so attempt to figure
out redirects.
(*) -watch won't able to run as a daemon in cases when
git-credential prompts for IMAP/NNTP passwords.
PublicInbox::Daemon is also designed for read-only
parallelism where all worker processes are the same.
Any subprocesses spawned by -watch are to do specific
tasks for a particular set of inboxes.
|
|
We may just modify PublicInbox::Config->urlmatch in the future
to support git <1.8.5, but I wonder if there's enough users on
git <1.8.5 to justify it.
|
|
Since we store all watched directory names as keys in %mdmap,
there should be no need to keep an array of those directories
around.
t/watch_maildir*.t required changes to remove trained spam.
Once we've trained something as spam, there shouldn't be
a need to rescan it.
|
|
Some users will find it useful to merge several Maildir or
IMAP mailboxes into one public-inbox. Let them do it, since
we've always supported multi-address inboxes.
|
|
If ->idle_done was handled successfully, we can just
let normal ->DESTROY disconnect and avoid ugly backtraces
when a user hits Ctrl-C to take down the process group.
|
|
IMAP allows retrieving multiple messages with a single command,
and Mail::IMAPClient supports that. Unfortunately, it means we
slurp multiple messages into memory at once. This option allows
users to trade off memory usage to reduce network round-trips.
Ideally, we'd support pipelining; but AFAIK no widely installed
Perl IMAP library supports it.
|
|
With different polling intervals, multiple processes may
simultaneously write to IMAPtracker. This ought to reduce
SQLite busy waiting and contention issues when importing
many inboxes in parallel.
|
|
It's not used anywhere since the IMAPTracker object doesn't
disconnect and reconnect. If we ever need the filename,
{dbh}->sqlite_db_filename may be used.
Cc: Eric W. Biederman <ebiederm@xmission.com>
|
|
Passing a $url parameter to every function was error-prone
and having {url} field for a short-lived object is appropriate.
This matches the version of IMAPTracker posted by
Eric W. Biederman on 2020-05-15 at:
https://public-inbox.org/meta/87ftc0c3r4.fsf_-_@x220.int.ebiederm.org/
The version I originally imported was based on the one
posted on 2019-10-09:
https://public-inbox.org/meta/874l0i9vhc.fsf_-_@x220.int.ebiederm.org/
Cc: Eric W. Biederman <ebiederm@xmission.com>
|
|
This allows callers to avoid creating expensive closures.
We no longer pass the `$now' value to callers, as none of
the callers used it.
|
|
For mailboxes with many gaps in the UID sequence,
performing a UID SEARCH beforehand can reduce the
number of articles to fetch.
However, the downside to this is we may end up with
an arbitrarly large list of UIDs from the server.
|
|
This fixes cases where watch is handling both Maildirs and IMAP
connections. While we're at it, close open directories in the
IMAP children to save FDs.
|
|
Since we have IMAP client support in -watch; make sure per-URL
settings are familiar to git users by taking advantage of git's
URL matching abilities.
This requires git 1.8.5+, which most users ought to have
(though base CentOS 7 is on 1.8.3).
|
|
Not all IMAP servers support IDLE, and IDLE may be prohibitively
expensive for some IMAP servers with many inboxes. So allow
configuring a imap.$IMAP_URL.pollInterval=SECONDS to poll
mailboxes.
We'll also need to poll for NNTP servers in the future.
|
|
We can avoid synchronous `waitpid(-1, 0)' and save a process
when simultaneously watching Maildirs.
One DS bug is fixed: ->Reset needs to clear the DS $in_loop flag
in forked children so dwaitpid() fails and allows git processes
to be reaped synchronously. TestCommon also calls DS->Reset
when spawning new processes, since t/imapd.t uses DS->EventLoop
while waiting on -watch to write.
|
|
Since the removal of pseudo-hash support in Perl 5.10, the
"fields" module no longer provides the space or speed benefits
it did in 5.8. It also does not allow for compile-time checks,
only run-time checks.
To me, the extra developer overhead in maintaining "use fields"
args has become a hassle. None of our non-DS-related code uses
fields.pm, nor do any of our current dependencies. In fact,
Danga::Socket (which DS was originally forked from) and its
subclasses are the only fields.pm users I've ever encountered in
the wild. Removing fields may make our code more approachable
to other Perl hackers.
So stop using fields.pm and locked hashes, but continue to
document what fields do for non-trivial classes.
|
|
We can get rid of the janky wannabe
self-using-a-directory-instead-of-pipe thing we needed to
workaround Filesys::Notify::Simple being blocking.
For existing Maildir users, this should be more robust and
immune to missed wakeups for signalfd and kqueue-enabled
systems; as well as being immune to BOFHs clearing $TMPDIR
and preventing notifications from firing.
The IMAP IDLE code still uses normal Perl signals, so it's still
vulnerable to missed wakeups. That will be addressed in future
commits.
|
|
Since we already use inotify and EVFILT_VNODE (kqueue)
in -imapd, we might as well use them directly in -watch,
too.
This will allow public-inbox-watch to use PublicInbox::DS
for timers to watch newsgroups/mailboxes and have saner
signal handling in future commits.
|
|
We need to detect link(2) and rename(2) in other apps
writing to the Maildir.
We'll be removing the Filesys::Notify::Simple from -watch
in favor of using IO::KQueue or Linux::Inotify2 directly.
Ensure non-inotify emulations can support everything we
expect for Maildir writers.
|
|
Only servers with IDLE are supported, for now. Polling will
be needed since users may need to watch many inboxes with
a few active connections due to IMAP server limitations.
|
|
We'll be supporting the IMAP URL scheme described in RFC 5092
for -watch, so add this module to fill in what the `URI' package
lacks.
|
|
The old check was ineffective since we process the spam folder
config before ham inboxes; and would only fail when attempting
to treat the scalar "watchspam" string as an array ref.
|
|
It's too deeply indented, and we will be using it for IMAP, too.
|