about summary refs log tree commit homepage
path: root/lib/PublicInbox/DirIdle.pm
DateCommit message (Collapse)
2023-12-29pure Perl inotify support
This is a step towards improving the out-of-the-box experience in achieving notifications without XS, extra downloads, and .so loading + runtime mmap overhead. This also fixes loongarch support of all Linux syscalls due to a bad regexp :x All the reachable Linux architectures listed at <https://portal.cfarm.net/machines/list/> should be supported. At the moment, there appears to be no reachable sparc* Linux machines available to cfarm users. Fixes: b0e5093aa3572a86 (syscall: add support for riscv64, 2022-08-11)
2023-11-02treewide: use ->close method rather than CORE::close
It's easier-to-read and should open the door for us to get rid of `tie' for ProcessIO without performance penalties for more frequently-used perlop calls and ability to do `stat' directly on the object instead of the awkward `tied' thing.
2023-10-04lei: close DirIdle (inotify) early at daemon shutdown
We don't want FS activity to delay lei-daemon shutdown.
2023-03-29inotify: wrap with informative error message
As encountered by Louis DeLosSantos, Linux inotify is capped by a lesser-known limit than the standard RLIMIT_NOFILE (`ulimit -n`) value. Give the user a hint about the fs.inotify.max_user_instances sysctl knob on EMFILE, since EMFILE alone may mislead users into thinking they've hit the (typically higher) RLIMIT_NOFILE limit. I can test this on my system using: perl -I lib -MPublicInbox::Inotify -E \ 'my @x = map { PublicInbox::Inotify->new } (1..128)' But I hesitate to include it in the test suite since triggering the limit can cause unrelated processes to fail. Link: https://public-inbox.org/meta/CAE6jdTo8iQfNM9Yuk0Dwi-ARMxmQxX-onL8buXcQ9Ze3r0hKrg@mail.gmail.com/ Reported-by: Louis DeLosSantos <louis.delos@gmail.com>
2022-11-28switch inotify/kevent stuff to v5.12
Another tiny step towards an eventual startup time improvements by avoiding strict.pm
2021-10-22dir_idle: treat IN_MOVED_FROM as a gone event
Whether an MUA uses rename(2) or link(2)+unlink(2) combination should not matter to us. We should be able to handle both cases.
2021-10-16dir_idle: do not add watches in ->new
There's no savings in having two ways to add watches to an inotify nor kqueue descriptor.
2021-10-04{dir,inbox}idle: use level-triggered epoll
Both read(2) on inotify and kevent(2) return a finite amount of events. Let the kernel notify us again in cases where we'd need to retry instead of looping ourselves. This can prevent missed/delayed notifications while still ensuring fairness in busy event loops.
2021-08-04lei: close inotify FD in forked child
Linux::Inotify2 2.3+ includes an ->fh method to give us the ability to safely close an FD without hitting EBADF (and automatically use FD_CLOEXEC). We'll still need a new wrapper class (LI2Wrap) to handle it for users of old versions, though. Link: http://lists.schmorp.de/pipermail/perl/2021q3/thread.html
2021-07-22lei: auto-refresh watches in config, cancel missing
This makes behavior less surprising on restarts as we no longer lose state on restarts, so there's no need to manually run "lei add-watch" to re-enable watches. This also allows us to transparently handle changes if somebody edits the lei config file directly or via git-config(1).
2021-07-22lei: start implementing inotify Maildir support
This allows lei to automatically note keyword (message flag) changes made to a Maildir and propagate it into lei/store: lei add-watch --state=tag-ro /path/to/Maildir This doesn't persist across restarts, yet. In the future, it will be applied automatically to "lei q" output Maildirs by default (with an option to disable it). State values of tag-rw, index-<ro|rw>, import-<ro|rw> will all be supported for Maildir. This represents a fairly major internal change that's fairly intrusive, but the whole daemon-oriented design was to facilitate being able to automatically monitor (and propagate) Maildir/IMAP flag changes.
2021-05-15dir_idle: support IN_DELETE_SELF|IN_MOVE_SELF, too
We'll treat IN_MOVE_SELF as IN_DELETE_SELF since there doesn't seem to be a reliable way to distinguish them with FakeInotify, nor know the new name with kevent.
2021-05-15dir_idle: detect files which are gone
lei now makes use of this to clean up after unlinked sockets with less delay. This will also be used to maintain mail_sync.sqlite3.
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2020-08-08dir_idle: require Perl 5.22+ for kqueue
IO::KQueue requires us to use fileno(DIRHANDLE) for setting up kqueue watches. This use of fileno() is only supported since Perl 5.22, so BSD users on older Perl will have to fall back to old polling. This affects users of -watch, currently; but will affect other read-only Xapian users soon.
2020-08-01improve error handling on import fork / lock failures
v?fork failures seems to be the cause of locks not getting released in -watch. Ensure lock release doesn't get skipped in ->done for both v1 and v2 inboxes. We also need to do everything we can to ensure DB handles, pipes and processes get released even in the face of failure. While we're at it, make failures around `git update-server-info' non-fatal, since smart HTTP seems more popular anyways. v2 changes: - spawn: show failing command - ensure waitpid is synchronous for inotify events - teardown all fast-import processes on exception, not just the failing one - beef up lock_release error handling - release lock on fast-import spawn failure
2020-07-02inboxidle: avoid per-inbox anonymous subs
Anonymous subs cost over 5K each on x86-64. So prefer the less-recommended-but-still-documented way of using Linux::Inotify2::watch to register watchers. This also updates FakeInotify to detect modifications correctly when used on systems with neither IO::KQueue nor Linux::Inotify2.
2020-06-28ds: remove fields.pm usage
Since the removal of pseudo-hash support in Perl 5.10, the "fields" module no longer provides the space or speed benefits it did in 5.8. It also does not allow for compile-time checks, only run-time checks. To me, the extra developer overhead in maintaining "use fields" args has become a hassle. None of our non-DS-related code uses fields.pm, nor do any of our current dependencies. In fact, Danga::Socket (which DS was originally forked from) and its subclasses are the only fields.pm users I've ever encountered in the wild. Removing fields may make our code more approachable to other Perl hackers. So stop using fields.pm and locked hashes, but continue to document what fields do for non-trivial classes.
2020-06-28watch: remove Filesys::Notify::Simple dependency
Since we already use inotify and EVFILT_VNODE (kqueue) in -imapd, we might as well use them directly in -watch, too. This will allow public-inbox-watch to use PublicInbox::DS for timers to watch newsgroups/mailboxes and have saner signal handling in future commits.