* [PATCH 03/34] IMAPTracker: Add a helper to track our place in reading imap mailboxes
2020-06-27 10:03 4% [PATCH 00/34] watch: add IMAP and NNTP support Eric Wong
@ 2020-06-27 10:03 6% ` Eric Wong
0 siblings, 0 replies; 5+ results
From: Eric Wong @ 2020-06-27 10:03 UTC (permalink / raw)
To: meta; +Cc: Eric W. Biederman
From: "Eric W. Biederman" <ebiederm@xmission.com>
This removes the need to delete from an imap mailbox when
downloading it's messages.
[ew: minor style changes]
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
MANIFEST | 1 +
lib/PublicInbox/IMAPTracker.pm | 61 ++++++++++++++++++++++++++++++++++
2 files changed, 62 insertions(+)
create mode 100644 lib/PublicInbox/IMAPTracker.pm
diff --git a/MANIFEST b/MANIFEST
index 3e7d4cc0e29..42a00d74344 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -132,6 +132,7 @@ lib/PublicInbox/Hval.pm
lib/PublicInbox/IMAP.pm
lib/PublicInbox/IMAPClient.pm
lib/PublicInbox/IMAPD.pm
+lib/PublicInbox/IMAPTracker.pm
lib/PublicInbox/IMAPdeflate.pm
lib/PublicInbox/IMAPsearchqp.pm
lib/PublicInbox/Import.pm
diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
new file mode 100644
index 00000000000..c7da422b725
--- /dev/null
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -0,0 +1,61 @@
+# Copyright (C) 2018-2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+package PublicInbox::IMAPTracker;
+use strict;
+use DBI;
+use DBD::SQLite;
+use PublicInbox::Config;
+
+sub create_tables ($) {
+ my ($dbh) = @_;
+
+ $dbh->do(<<'');
+CREATE TABLE IF NOT EXISTS imap_last (
+ url VARCHAR PRIMARY KEY NOT NULL,
+ uid_validity INTEGER NOT NULL,
+ uid INTEGER NOT NULL,
+ UNIQUE (url)
+)
+
+}
+
+sub dbh_new ($) {
+ my ($dbname) = @_;
+ my $dbh = DBI->connect("dbi:SQLite:dbname=$dbname", '', '', {
+ AutoCommit => 1,
+ RaiseError => 1,
+ PrintError => 0,
+ sqlite_use_immediate_transaction => 1,
+ });
+ $dbh->{sqlite_unicode} = 1;
+ $dbh->do('PRAGMA journal_mode = TRUNCATE');
+ create_tables($dbh);
+ $dbh;
+}
+
+sub get_last ($$) {
+ my ($self, $url) = @_;
+ my $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+SELECT uid_validity, uid FROM imap_last WHERE url = ?
+
+ $sth->execute($url);
+ $sth->fetchrow_array;
+}
+
+sub update_last ($$$$) {
+ my ($self, $url, $validity, $last) = @_;
+ my $sth = $self->{dbh}->prepare_cached(<<'');
+INSERT OR REPLACE INTO imap_last (url, uid_validity, uid)
+VALUES (?, ?, ?)
+
+ $sth->execute($url, $validity, $last);
+}
+
+sub new {
+ my ($class) = @_;
+ my $dbname = PublicInbox::Config->config_dir() . "/imap.sqlite3";
+ my $dbh = dbh_new($dbname);
+ bless { dbname => $dbname, dbh => $dbh }, $class;
+}
+
+1;
^ permalink raw reply related [relevance 6%]
* [PATCH 00/34] watch: add IMAP and NNTP support
@ 2020-06-27 10:03 4% Eric Wong
2020-06-27 10:03 6% ` [PATCH 03/34] IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric Wong
0 siblings, 1 reply; 5+ results
From: Eric Wong @ 2020-06-27 10:03 UTC (permalink / raw)
To: meta
Some fairly major changes to -watch. Filesys::Notify::Simple is
no longer used, and -watch now uses inotify, signalfd or kevent
like the read-only daemons.
Credentials are handled via Net::Netrc (Perl standard library)
or "git-credential", so we do no password storage on our own.
NNTP (and non-IDLE IMAP) may allow more parallelization in the
future.
One significant project-wide change is getting rid of "use
fields". It gets in my way more than it helps, and it's
probably alien to a fair amount of Perl hackers. AFAIK, it's
never really been popular outside of Danga::Socket-based
projects.
Eric W. Biederman (1):
IMAPTracker: Add a helper to track our place in reading imap mailboxes
Eric Wong (33):
inboxwritable: ensure ssoma.lock exists on init
inbox: warn on ->on_inbox_unlock exception
imaptracker: use ~/.local/share/public-inbox/imap.sqlite3
watchmaildir: hoist out compile_watchheaders
watchmaildir: fix check for spam vs ham inbox conflicts
URI IMAP support
watch: preliminary IMAP support
kqnotify|fake_inotify: detect Maildir write ops
watch: remove Filesys::Notify::Simple dependency
watch: use signalfd for Maildir watching
ds: remove fields.pm usage
watch: wire up IMAP IDLE reapers to DS
watch: support IMAP polling
config: support ->urlmatch method for -watch
watch: stop importers before forking
watch: use UID SEARCH to avoid empty UID FETCH
ds: add_timer: allow passing arg to callback.
imaptracker: add {url} field to reduce args
imaptracker: drop {dbname} field
watch: avoid long transaction when writing to IMAPTracker
watch: support imap.fetchBatchSize parameter
watch: imap: be quieter about disconnecting on quit
watch: support multiple watch: directives per-inbox
watch: remove {mdir} array
watch: just use ->urlmatch
testcommon: $ENV{TAIL} supports non-@ARGV redirects
watch: add NNTP support
watch: show user-specified URL consistently.
watch: enable autoflush for STDOUT and STDERR
watch: use our own "git credential" wrapper
watch: support ~/.netrc via Net::Netrc
imaptracker: use flock(2) around writes
watch: simplify internal structures
Documentation/public-inbox-watch.pod | 3 +-
INSTALL | 8 -
MANIFEST | 11 +
Makefile.PL | 4 -
ci/deps.perl | 1 -
lib/PublicInbox/Config.pm | 21 +-
lib/PublicInbox/DS.pm | 29 +-
lib/PublicInbox/Daemon.pm | 19 +-
lib/PublicInbox/DirIdle.pm | 49 ++
lib/PublicInbox/FakeInotify.pm | 56 +-
lib/PublicInbox/GitAsyncCat.pm | 4 +-
lib/PublicInbox/GitCredential.pm | 55 ++
lib/PublicInbox/HTTP.pm | 23 +-
lib/PublicInbox/HTTPD/Async.pm | 22 +-
lib/PublicInbox/IMAP.pm | 19 +-
lib/PublicInbox/IMAPTracker.pm | 82 +++
lib/PublicInbox/In2Tie.pm | 13 +
lib/PublicInbox/Inbox.pm | 1 +
lib/PublicInbox/InboxIdle.pm | 20 +-
lib/PublicInbox/InboxWritable.pm | 3 +
lib/PublicInbox/KQNotify.pm | 38 +-
lib/PublicInbox/Listener.pm | 8 +-
lib/PublicInbox/NNTP.pm | 12 +-
lib/PublicInbox/NNTPdeflate.pm | 5 +-
lib/PublicInbox/ParentPipe.pm | 8 +-
lib/PublicInbox/Sigfd.pm | 21 +-
lib/PublicInbox/TestCommon.pm | 40 +-
lib/PublicInbox/URIimap.pm | 113 +++
lib/PublicInbox/WatchMaildir.pm | 998 +++++++++++++++++++++++----
script/public-inbox-watch | 33 +-
t/config.t | 18 +
t/dir_idle.t | 6 +
t/fake_inotify.t | 45 ++
t/imap_tracker.t | 54 ++
t/imapd.t | 74 ++
t/kqnotify.t | 41 ++
t/nntpd.t | 52 ++
t/uri_imap.t | 65 ++
t/watch_filter_rubylang.t | 2 +-
t/watch_imap.t | 21 +
t/watch_maildir.t | 96 ++-
t/watch_maildir_v2.t | 4 +-
t/watch_multiple_headers.t | 2 +-
t/watch_nntp.t | 17 +
xt/mem-imapd-tls.t | 18 +-
45 files changed, 1944 insertions(+), 290 deletions(-)
create mode 100644 lib/PublicInbox/DirIdle.pm
create mode 100644 lib/PublicInbox/GitCredential.pm
create mode 100644 lib/PublicInbox/IMAPTracker.pm
create mode 100644 lib/PublicInbox/URIimap.pm
create mode 100644 t/dir_idle.t
create mode 100644 t/fake_inotify.t
create mode 100644 t/imap_tracker.t
create mode 100644 t/kqnotify.t
create mode 100644 t/uri_imap.t
create mode 100644 t/watch_imap.t
create mode 100644 t/watch_nntp.t
^ permalink raw reply [relevance 4%]
* [PATCH 1/2] IMAPTracker: Add a helper to track our place in reading imap mailboxes
@ 2020-05-15 21:00 6% ` Eric W. Biederman
0 siblings, 0 replies; 5+ results
From: Eric W. Biederman @ 2020-05-15 21:00 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
By tracking which messages in an imap mailbox have been already downloaded
they imap messages do not need to be deleted immediately.
It is recommended to implement one connection per IMAP mailbox,
so this tracker caches the entire url for a mailbox at creation time.
If you need to process multiple IMAP mailboxes you will need multiple trackers.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
lib/PublicInbox/IMAPTracker.pm | 73 ++++++++++++++++++++++++++++++++++
1 file changed, 73 insertions(+)
create mode 100644 lib/PublicInbox/IMAPTracker.pm
diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
new file mode 100644
index 000000000000..d445555c0fd9
--- /dev/null
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -0,0 +1,73 @@
+# Copyright (C) 2018 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+package PublicInbox::IMAPTracker;
+use strict;
+use warnings;
+use DBI;
+use DBD::SQLite;
+use PublicInbox::Config;
+
+sub create_tables ($)
+{
+ my ($dbh) = @_;
+
+ $dbh->do(<<'');
+CREATE TABLE IF NOT EXISTS imap_last (
+ url VARCHAR PRIMARY KEY NOT NULL,
+ uid_validity INTEGER NOT NULL,
+ uid INTEGER NOT NULL,
+ UNIQUE (url)
+)
+
+}
+
+
+sub dbh_new ($)
+{
+ my ($dbname) = @_;
+ my $dbh = DBI->connect("dbi:SQLite:dbname=$dbname", '', '', {
+ AutoCommit => 1,
+ RaiseError => 1,
+ PrintError => 0,
+ ReadOnly => 0,
+ sqlite_use_immediate_transaction => 1,
+ });
+ $dbh->{sqlite_unicode} = 1;
+ $dbh->do('PRAGMA journal_mode = TRUNCATE');
+ $dbh->do('PRAGMA cache_size = 80000');
+ create_tables($dbh);
+ $dbh;
+}
+
+sub get_last($)
+{
+ my ($self) = @_;
+ my $dbh = $self->{dbh};
+
+ my $sth = $dbh->prepare_cached(<<'', undef, 1);
+SELECT uid_validity, uid FROM imap_last WHERE url = ?
+
+ $sth->execute($self->{url});
+ $sth->fetchrow_array;
+}
+
+sub update_last($$$)
+{
+ my ($self, $validity, $last) = @_;
+ my $dbh = $self->{dbh};
+ my $sth = $dbh->prepare_cached(<<'');
+INSERT OR REPLACE INTO imap_last (url, uid_validity, uid)
+VALUES (?, ?, ?)
+
+ $sth->execute($self->{url}, $validity, $last);
+}
+
+sub new ($) {
+ my ($class, $url) = @_;
+ my $dbname = PublicInbox::Config->config_dir() . "/imap.sqlite3";
+ my $dbh = dbh_new($dbname);
+ bless { dbname => $dbname, dbh => $dbh, url => $url }, $class;
+}
+
+1;
--
2.20.1
^ permalink raw reply related [relevance 6%]
* [PATCH 4/4] IMAPTracker: Add a helper to track our place in reading imap mailboxes
2019-10-09 8:15 5% ` [PATCH 0/4] Various bits to support import_imap_mailbox Eric W. Biederman
@ 2019-10-09 8:25 7% ` Eric W. Biederman
0 siblings, 0 replies; 5+ results
From: Eric W. Biederman @ 2019-10-09 8:25 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Date: Fri, 27 Jul 2018 20:54:27 -0500
This removes the need to delete from an imap mailbox when downloading it's messages.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
This is simple and potentially very useful.
lib/PublicInbox/IMAPTracker.pm | 73 ++++++++++++++++++++++++++++++++++
1 file changed, 73 insertions(+)
create mode 100644 lib/PublicInbox/IMAPTracker.pm
diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
new file mode 100644
index 000000000000..65241eeebf51
--- /dev/null
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -0,0 +1,73 @@
+# Copyright (C) 2018 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+package PublicInbox::IMAPTracker;
+use strict;
+use warnings;
+use DBI;
+use DBD::SQLite;
+use PublicInbox::Config;
+
+sub create_tables ($)
+{
+ my ($dbh) = @_;
+
+ $dbh->do(<<'');
+CREATE TABLE IF NOT EXISTS imap_last (
+ url VARCHAR PRIMARY KEY NOT NULL,
+ uid_validity INTEGER NOT NULL,
+ uid INTEGER NOT NULL,
+ UNIQUE (url)
+)
+
+}
+
+
+sub dbh_new ($)
+{
+ my ($dbname) = @_;
+ my $dbh = DBI->connect("dbi:SQLite:dbname=$dbname", '', '', {
+ AutoCommit => 1,
+ RaiseError => 1,
+ PrintError => 0,
+ ReadOnly => 0,
+ sqlite_use_immediate_transaction => 1,
+ });
+ $dbh->{sqlite_unicode} = 1;
+ $dbh->do('PRAGMA journal_mode = TRUNCATE');
+ $dbh->do('PRAGMA cache_size = 80000');
+ create_tables($dbh);
+ $dbh;
+}
+
+sub get_last($$)
+{
+ my ($self, $url) = @_;
+ my $dbh = $self->{dbh};
+
+ my $sth = $dbh->prepare_cached(<<'', undef, 1);
+SELECT uid_validity, uid FROM imap_last WHERE url = ?
+
+ $sth->execute($url);
+ $sth->fetchrow_array;
+}
+
+sub update_last($$$$)
+{
+ my ($self, $url, $validity, $last) = @_;
+ my $dbh = $self->{dbh};
+ my $sth = $dbh->prepare_cached(<<'');
+INSERT OR REPLACE INTO imap_last (url, uid_validity, uid)
+VALUES (?, ?, ?)
+
+ $sth->execute($url, $validity, $last);
+}
+
+sub new {
+ my ($class) = @_;
+ my $dbname = PublicInbox::Config->config_dir() . "/imap.sqlite3";
+ my $dbh = dbh_new($dbname);
+ bless { dbname => $dbname, dbh => $dbh }, $class;
+}
+
+1;
--
2.23.0
^ permalink raw reply related [relevance 7%]
* [PATCH 0/4] Various bits to support import_imap_mailbox
@ 2019-10-09 8:15 5% ` Eric W. Biederman
2019-10-09 8:25 7% ` [PATCH 4/4] IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric W. Biederman
0 siblings, 1 reply; 5+ results
From: Eric W. Biederman @ 2019-10-09 8:15 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong,
These should all of my generic patches to support my import_imap_mailbox
script. The really important patch that adds to the support for List-ID
to public inbox configuration file I have already sent.
I haven't written tests and I get the following test failure when I run
make test
> t/config.t ................. 1/?
> # Failed test 'lookup matches expected output'
> # at t/config.t line 26.
> # Structures begin differing at:
> # $got->{listid} = ARRAY(0x55c1d4e3b6a8)
> # $expected->{listid} = Does not exist
>
> # Failed test 'lookup matches expected output for test'
> # at t/config.t line 42.
> # Structures begin differing at:
> # $got->{listid} = ARRAY(0x55c1d508d8d0)
> # $expected->{listid} = Does not exist
> # Looks like you failed 2 tests of 69.
> t/config.t ................. Dubious, test returned 2 (wstat 512, 0x200)
I haven't looked into what is happening there.
Eric Biederman (4):
PublicInbox::Import Smuggle a raw message into add
PublicInbox::Config: Process mailboxes in sorted order
Config.pm: Add support for looking up repos by their directories
IMAPTracker: Add a helper to track our place in reading imap mailboxes
lib/PublicInbox/Config.pm | 19 ++++++++++-
lib/PublicInbox/IMAPTracker.pm | 73 ++++++++++++++++++++++++++++++++++++++++++
lib/PublicInbox/Import.pm | 9 +++---
3 files changed, 96 insertions(+), 5 deletions(-)
Eric
^ permalink raw reply [relevance 5%]
Results 1-5 of 5 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-10-07 22:13 Do I need multiple publicinbox.<name>.address values? Alyssa Ross
2019-10-08 0:10 ` Eric Wong
2019-10-08 12:18 ` Eric W. Biederman
2019-10-08 22:11 ` Eric Wong
2019-10-08 22:24 ` Eric W. Biederman
2019-10-08 22:41 ` Eric Wong
2019-10-09 7:58 ` Eric W. Biederman
2019-10-09 8:15 5% ` [PATCH 0/4] Various bits to support import_imap_mailbox Eric W. Biederman
2019-10-09 8:25 7% ` [PATCH 4/4] IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric W. Biederman
2019-10-29 14:40 I have figured out IMAP IDLE Eric W. Biederman
2020-05-13 19:31 ` Eric Wong
2020-05-13 21:48 ` Eric W. Biederman
2020-05-13 22:17 ` Eric Wong
2020-05-14 12:32 ` Eric W. Biederman
2020-05-15 21:00 6% ` [PATCH 1/2] IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric W. Biederman
2020-06-27 10:03 4% [PATCH 00/34] watch: add IMAP and NNTP support Eric Wong
2020-06-27 10:03 6% ` [PATCH 03/34] IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).