user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Subject: [PATCH] msgmap: mid_insert: use plain "INSERT" to detect duplicates
Date: Mon, 24 Jun 2019 23:38:09 +0000
Message-ID: <20190624233809.1721-1-e@80x24.org> (raw)
In-Reply-To: <878strvusz.fsf@xmission.com>

"INSERT OR IGNORE" still bumps the auto-increment counter in
SQLite, which causes gaps to appear in NNTP article numbering.

This bug appeared in v2 repos where V2Writable may call ->add
repeatedly on the same message.  This bug is apparent with
public-inbox-watch and work-in-progress IMAP watchers which may
rescan and (attempt to) reinsert the same message on mailbox
changes.

Most uses of public-inbox-mda were not affected, unless the
same message is actually delivered multiple times to the mda.
v1 is not affected, either, since deduplication is only based
on Message-ID and msgmap never sees the duplicate.

Reported-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
 lib/PublicInbox/Msgmap.pm | 4 ++--
 t/msgmap.t                | 3 +++
 t/v2writable.t            | 2 ++
 3 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/Msgmap.pm b/lib/PublicInbox/Msgmap.pm
index 0035c9e3..5a89b85a 100644
--- a/lib/PublicInbox/Msgmap.pm
+++ b/lib/PublicInbox/Msgmap.pm
@@ -126,9 +126,9 @@ sub mid_insert {
 	my ($self, $mid) = @_;
 	my $dbh = $self->{dbh};
 	my $sth = $dbh->prepare_cached(<<'');
-INSERT OR IGNORE INTO msgmap (mid) VALUES (?)
+INSERT INTO msgmap (mid) VALUES (?)
 
-	return if $sth->execute($mid) == 0;
+	return unless eval { $sth->execute($mid) };
 	my $num = $dbh->last_insert_id(undef, undef, 'msgmap', 'num');
 	$self->num_highwater($num) if defined($num);
 	$num;
diff --git a/t/msgmap.t b/t/msgmap.t
index 4dddd0a8..2d018219 100644
--- a/t/msgmap.t
+++ b/t/msgmap.t
@@ -30,6 +30,9 @@ $@ = undef;
 my $ret = $d->mid_insert('a@b');
 is($ret, undef, 'duplicate mid_insert in undef result');
 is($d->num_for('a@b'), $mid2num{'a@b'}, 'existing number not clobbered');
+my $max = (sort(keys %num2mid))[-1];
+is($d->mid_insert('ok@unique'), $max + 1,
+	'got expected num after failing mid_insert');
 
 foreach my $n (keys %num2mid) {
 	is($d->mid_for($n), $num2mid{$n}, "num:$n maps correctly");
diff --git a/t/v2writable.t b/t/v2writable.t
index 88df2d64..8f32fbe5 100644
--- a/t/v2writable.t
+++ b/t/v2writable.t
@@ -118,6 +118,8 @@ if ('ensure git configs are correct') {
 	$mime->header_set('References', '<zz-mid@b>');
 	ok($im->add($mime), 'message with multiple Message-ID');
 	$im->done;
+	my ($total, undef) = $ibx->over->recent;
+	is($ibx->mm->num_highwater, $total, 'got expected highwater value');
 	my $srch = $ibx->search;
 	my $mset1 = $srch->reopen->query('m:abcde@1', { mset => 1 });
 	is($mset1->size, 1, 'message found by first MID');
-- 
EW


  parent reply index

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-24 15:59 Q: Did you do something to message number recently? Eric W. Biederman
2019-06-24 16:34 ` Eric Wong
2019-06-24 17:33   ` Eric Wong
2019-06-24 22:56     ` Eric W. Biederman
2019-06-24 23:42       ` Eric Wong
2019-06-25 12:01         ` Eric W. Biederman
2019-06-25 17:51           ` Eric Wong
2019-06-24 23:38 ` Eric Wong [this message]
2019-06-25  3:48   ` [PATCH] msgmap: mid_insert: use plain "INSERT" to detect duplicates Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190624233809.1721-1-e@80x24.org \
    --to=e@80x24.org \
    --cc=ebiederm@xmission.com \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git