user/dev discussion of public-inbox itself
 help / color / Atom feed
* [PATCH] nntp: filter out duplicate Message-IDs for leafnode
@ 2019-06-14  0:53 Eric Wong
  0 siblings, 0 replies; only message in thread
From: Eric Wong @ 2019-06-14  0:53 UTC (permalink / raw)
  To: meta

It's the unfortunate reality that there are some clients which
reuse Message-IDs (in which we generate + use another) or set
multiple Message-IDs on their own.  While the v2 format
addresses that, NNTP clients such as leafnode are not always
prepared to deal with that case.

So, ensure NNTP clients only see a single Message-ID, and
show the others as 'X-Alt-Message-ID'.
---
 lib/PublicInbox/NNTP.pm | 16 ++++++++++++++++
 t/nntpd.t               | 17 +++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index d409e78..8a31b91 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -438,6 +438,22 @@ sub set_nntp_headers ($$$$$) {
 	# reason.  We'll fake the shortest one possible.
 	$hdr->header_set('Path', 'y');
 
+	# leafnode (and maybe other NNTP clients) have trouble dealing
+	# with v2 messages which have multiple Message-IDs (either due
+	# to our own content-based dedupe or buggy git-send-email versions).
+	my @mids = $hdr->header('Message-ID');
+	if (scalar(@mids) > 1) {
+		my $mid0 = "<$mid>";
+		$hdr->header_set('Message-ID', $mid0);
+		my @alt = $hdr->header('X-Alt-Message-ID');
+		my %seen = map { $_ => 1 } (@alt, $mid0);
+		foreach my $m (@mids) {
+			next if $seen{$m}++;
+			push @alt, $m;
+		}
+		$hdr->header_set('X-Alt-Message-ID', @alt);
+	}
+
 	# clobber some
 	my $xref = xref($self, $ng, $n, $mid);
 	$hdr->header_set('Xref', $xref);
diff --git a/t/nntpd.t b/t/nntpd.t
index cce21ee..a95fb6f 100644
--- a/t/nntpd.t
+++ b/t/nntpd.t
@@ -250,6 +250,23 @@ Date: Fri, 02 Oct 1993 00:00:00 +0000
 		my $expect = qr/\AMessage-ID: /i . qr/\Q<$long_hdr>\E/;
 		ok(scalar(grep(/$expect/, @$hdr)), 'Message-ID not folded');
 		ok(scalar(grep(/^Path:/, @$hdr)), 'Path: header found');
+
+		# it's possible for v2 messages to have 2+ Message-IDs,
+		# but leafnode can't handle it
+		if ($version != 1) {
+			my @mids = ("<$long_hdr>", '<2mid@wtf>');
+			$for_leafnode->header_set('Message-ID', @mids);
+			$for_leafnode->body_set('not-a-dupe');
+			my $warn = '';
+			$SIG{__WARN__} = sub { $warn .= join('', @_) };
+			$im->add($for_leafnode);
+			$im->done;
+			like($warn, qr/reused/, 'warned for reused MID');
+			$hdr = $n->head('<2mid@wtf>');
+			my @hmids = grep(/\AMessage-ID: /i, @$hdr);
+			is(scalar(@hmids), 1, 'Single Message-ID in header');
+			like($hmids[0], qr/: <2mid\@wtf>/, 'got expected mid');
+		}
 	}
 
 	# pipelined requests:
-- 
EW


^ permalink raw reply	[flat|nested] only message in thread

only message in thread, back to index

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-14  0:53 [PATCH] nntp: filter out duplicate Message-IDs for leafnode Eric Wong

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git