* [PATCH 34/34] watch: simplify internal structures
2020-06-27 10:03 6% [PATCH 00/34] watch: add IMAP and NNTP support Eric Wong
@ 2020-06-27 10:04 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-06-27 10:04 UTC (permalink / raw)
To: meta
We won't be attempting to reuse Mail::IMAPConnections used to
check authentication info, for now, so stop storing
$self->{mics}.
We can also combine $poll initialization for IMAP and NNTP
to avoid data structure duplication. Furthermore, rely on
autovivification to create {idle_pids} and {poll_pids}.
---
lib/PublicInbox/WatchMaildir.pm | 42 +++++++++++----------------------
1 file changed, 14 insertions(+), 28 deletions(-)
diff --git a/lib/PublicInbox/WatchMaildir.pm b/lib/PublicInbox/WatchMaildir.pm
index 377548a2ad5..94b03ab36e3 100644
--- a/lib/PublicInbox/WatchMaildir.pm
+++ b/lib/PublicInbox/WatchMaildir.pm
@@ -509,8 +509,7 @@ sub watch_imap_idle_1 ($$$) {
my $mic;
local $0 = $uri->mailbox." $sec";
until ($self->{quit}) {
- $mic //= delete($self->{mics}->{$sec}) //
- PublicInbox::IMAPClient->new(%$mic_arg);
+ $mic //= PublicInbox::IMAPClient->new(%$mic_arg);
my $err = imap_fetch_all($self, $mic, $url);
$err //= imap_idle_once($self, $mic, $intvl, $url);
if ($err && !$self->{quit}) {
@@ -534,7 +533,6 @@ sub watch_atfork_child ($) {
sub watch_atfork_parent ($) {
my ($self) = @_;
_done_for_now($self);
- $self->{mics} = {}; # going to be forking, so disconnect
}
sub imap_idle_reap { # PublicInbox::DS::dwaitpid callback
@@ -652,8 +650,8 @@ sub poll_fetch_reap { # PublicInbox::DS::dwaitpid callback
[$self, $intvl, $urls]);
}
-sub watch_imap_init ($) {
- my ($self) = @_;
+sub watch_imap_init ($$) {
+ my ($self, $poll) = @_;
eval { require PublicInbox::IMAPClient } or
die "Mail::IMAPClient is required for IMAP:\n$@\n";
eval { require PublicInbox::IMAPTracker } or
@@ -663,14 +661,13 @@ sub watch_imap_init ($) {
# make sure we can connect and cache the credentials in memory
$self->{mic_arg} = {}; # schema://authority => IMAPClient->new args
- my $mics = $self->{mics} = {}; # schema://authority => IMAPClient obj
+ my $mics = {}; # schema://authority => IMAPClient obj
for my $url (sort keys %{$self->{imap}}) {
my $uri = PublicInbox::URIimap->new($url);
$mics->{uri_section($uri)} //= mic_for($self, $url, $mic_args);
}
my $idle = []; # [ [ url1, intvl1 ], [url2, intvl2] ]
- my $poll = {}; # intvl_seconds => [ url1, url2 ]
for my $url (keys %{$self->{imap}}) {
my $uri = PublicInbox::URIimap->new($url);
my $sec = uri_section($uri);
@@ -684,18 +681,9 @@ sub watch_imap_init ($) {
}
}
if (scalar @$idle) {
- $self->{idle_pids} = {};
$self->{idle_todo} = $idle;
PublicInbox::DS::requeue($self); # ->event_step to fork
}
- return unless scalar keys %$poll;
- $self->{poll_pids} //= {};
-
- # poll all URLs for a given interval sequentially
- while (my ($intvl, $urls) = each %$poll) {
- PublicInbox::DS::add_timer(0, \&poll_fetch_fork,
- [$self, $intvl, $urls]);
- }
}
# flesh out common NNTP-specific data structures
@@ -897,8 +885,8 @@ sub nntp_fetch_all ($$$) {
$err;
}
-sub watch_nntp_init ($) {
- my ($self) = @_;
+sub watch_nntp_init ($$) {
+ my ($self, $poll) = @_;
eval { require Net::NNTP } or
die "Net::NNTP is required for NNTP:\n$@\n";
eval { require PublicInbox::IMAPTracker } or
@@ -911,28 +899,26 @@ sub watch_nntp_init ($) {
for my $url (sort keys %{$self->{nntp}}) {
nn_for($self, $url, $nn_args);
}
- my $poll = {}; # intvl_seconds => [ url1, url2 ]
for my $url (keys %{$self->{nntp}}) {
my $uri = uri_new($url);
my $sec = uri_section($uri);
my $intvl = $self->{nntp_opt}->{$sec}->{pollInterval};
push @{$poll->{$intvl || 120}}, $url;
}
- $self->{poll_pids} //= {};
-
- # poll all URLs for a given interval sequentially
- while (my ($intvl, $urls) = each %$poll) {
- PublicInbox::DS::add_timer(0, \&poll_fetch_fork,
- [$self, $intvl, $urls]);
- }
}
sub watch {
my ($self, $sig, $oldset) = @_;
$self->{oldset} = $oldset;
$self->{sig} = $sig;
- watch_imap_init($self) if $self->{imap};
- watch_nntp_init($self) if $self->{nntp};
+ my $poll = {}; # intvl_seconds => [ url1, url2 ]
+ watch_imap_init($self, $poll) if $self->{imap};
+ watch_nntp_init($self, $poll) if $self->{nntp};
+ while (my ($intvl, $urls) = each %$poll) {
+ # poll all URLs for a given interval sequentially
+ PublicInbox::DS::add_timer(0, \&poll_fetch_fork,
+ [$self, $intvl, $urls]);
+ }
watch_fs_init($self) if $self->{mdre};
PublicInbox::DS->SetPostLoopCallback(sub {});
PublicInbox::DS->EventLoop until $self->{quit};
^ permalink raw reply related [relevance 7%]
* [PATCH 00/34] watch: add IMAP and NNTP support
@ 2020-06-27 10:03 6% Eric Wong
2020-06-27 10:04 7% ` [PATCH 34/34] watch: simplify internal structures Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-06-27 10:03 UTC (permalink / raw)
To: meta
Some fairly major changes to -watch. Filesys::Notify::Simple is
no longer used, and -watch now uses inotify, signalfd or kevent
like the read-only daemons.
Credentials are handled via Net::Netrc (Perl standard library)
or "git-credential", so we do no password storage on our own.
NNTP (and non-IDLE IMAP) may allow more parallelization in the
future.
One significant project-wide change is getting rid of "use
fields". It gets in my way more than it helps, and it's
probably alien to a fair amount of Perl hackers. AFAIK, it's
never really been popular outside of Danga::Socket-based
projects.
Eric W. Biederman (1):
IMAPTracker: Add a helper to track our place in reading imap mailboxes
Eric Wong (33):
inboxwritable: ensure ssoma.lock exists on init
inbox: warn on ->on_inbox_unlock exception
imaptracker: use ~/.local/share/public-inbox/imap.sqlite3
watchmaildir: hoist out compile_watchheaders
watchmaildir: fix check for spam vs ham inbox conflicts
URI IMAP support
watch: preliminary IMAP support
kqnotify|fake_inotify: detect Maildir write ops
watch: remove Filesys::Notify::Simple dependency
watch: use signalfd for Maildir watching
ds: remove fields.pm usage
watch: wire up IMAP IDLE reapers to DS
watch: support IMAP polling
config: support ->urlmatch method for -watch
watch: stop importers before forking
watch: use UID SEARCH to avoid empty UID FETCH
ds: add_timer: allow passing arg to callback.
imaptracker: add {url} field to reduce args
imaptracker: drop {dbname} field
watch: avoid long transaction when writing to IMAPTracker
watch: support imap.fetchBatchSize parameter
watch: imap: be quieter about disconnecting on quit
watch: support multiple watch: directives per-inbox
watch: remove {mdir} array
watch: just use ->urlmatch
testcommon: $ENV{TAIL} supports non-@ARGV redirects
watch: add NNTP support
watch: show user-specified URL consistently.
watch: enable autoflush for STDOUT and STDERR
watch: use our own "git credential" wrapper
watch: support ~/.netrc via Net::Netrc
imaptracker: use flock(2) around writes
watch: simplify internal structures
Documentation/public-inbox-watch.pod | 3 +-
INSTALL | 8 -
MANIFEST | 11 +
Makefile.PL | 4 -
ci/deps.perl | 1 -
lib/PublicInbox/Config.pm | 21 +-
lib/PublicInbox/DS.pm | 29 +-
lib/PublicInbox/Daemon.pm | 19 +-
lib/PublicInbox/DirIdle.pm | 49 ++
lib/PublicInbox/FakeInotify.pm | 56 +-
lib/PublicInbox/GitAsyncCat.pm | 4 +-
lib/PublicInbox/GitCredential.pm | 55 ++
lib/PublicInbox/HTTP.pm | 23 +-
lib/PublicInbox/HTTPD/Async.pm | 22 +-
lib/PublicInbox/IMAP.pm | 19 +-
lib/PublicInbox/IMAPTracker.pm | 82 +++
lib/PublicInbox/In2Tie.pm | 13 +
lib/PublicInbox/Inbox.pm | 1 +
lib/PublicInbox/InboxIdle.pm | 20 +-
lib/PublicInbox/InboxWritable.pm | 3 +
lib/PublicInbox/KQNotify.pm | 38 +-
lib/PublicInbox/Listener.pm | 8 +-
lib/PublicInbox/NNTP.pm | 12 +-
lib/PublicInbox/NNTPdeflate.pm | 5 +-
lib/PublicInbox/ParentPipe.pm | 8 +-
lib/PublicInbox/Sigfd.pm | 21 +-
lib/PublicInbox/TestCommon.pm | 40 +-
lib/PublicInbox/URIimap.pm | 113 +++
lib/PublicInbox/WatchMaildir.pm | 998 +++++++++++++++++++++++----
script/public-inbox-watch | 33 +-
t/config.t | 18 +
t/dir_idle.t | 6 +
t/fake_inotify.t | 45 ++
t/imap_tracker.t | 54 ++
t/imapd.t | 74 ++
t/kqnotify.t | 41 ++
t/nntpd.t | 52 ++
t/uri_imap.t | 65 ++
t/watch_filter_rubylang.t | 2 +-
t/watch_imap.t | 21 +
t/watch_maildir.t | 96 ++-
t/watch_maildir_v2.t | 4 +-
t/watch_multiple_headers.t | 2 +-
t/watch_nntp.t | 17 +
xt/mem-imapd-tls.t | 18 +-
45 files changed, 1944 insertions(+), 290 deletions(-)
create mode 100644 lib/PublicInbox/DirIdle.pm
create mode 100644 lib/PublicInbox/GitCredential.pm
create mode 100644 lib/PublicInbox/IMAPTracker.pm
create mode 100644 lib/PublicInbox/URIimap.pm
create mode 100644 t/dir_idle.t
create mode 100644 t/fake_inotify.t
create mode 100644 t/imap_tracker.t
create mode 100644 t/kqnotify.t
create mode 100644 t/uri_imap.t
create mode 100644 t/watch_imap.t
create mode 100644 t/watch_nntp.t
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-06-27 10:03 6% [PATCH 00/34] watch: add IMAP and NNTP support Eric Wong
2020-06-27 10:04 7% ` [PATCH 34/34] watch: simplify internal structures Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).