user/dev discussion of public-inbox itself
 help / color / Atom feed
* interop with leafnode
@ 2018-10-17 16:28 Dave Taht
  2018-10-27  0:10 ` Eric Wong
  2019-06-13  7:00 ` Eric Wong
  0 siblings, 2 replies; 9+ messages in thread
From: Dave Taht @ 2018-10-17 16:28 UTC (permalink / raw)
  To: meta


Jon Corbet turned me onto his nntp server instance which archives a
bunch of mailing lists and leverages public-inbox.

I excitedly accessed it directly from gnus. Oh, joy, bliss, rapture!
Blazing fast, useful at a glance.

In a paroxysm of overenthusiasm for the the good ole days I then setup a
leafnode real quick and pointed it at his instance so I could replicate...

For most (not all) traffic, leafnode does this:

> Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
> considering articles 200621 - 200643
> Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article 200621
> - no Path:  found

:(

I'm really glad to hear of this project. It sounds like more of the
right way to do this is via a local git?

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: interop with leafnode
  2018-10-17 16:28 interop with leafnode Dave Taht
@ 2018-10-27  0:10 ` Eric Wong
  2018-10-27  0:33   ` Dave Taht
  2019-05-05 21:40   ` Eric Wong
  2019-06-13  7:00 ` Eric Wong
  1 sibling, 2 replies; 9+ messages in thread
From: Eric Wong @ 2018-10-27  0:10 UTC (permalink / raw)
  To: Dave Taht; +Cc: meta

Dave Taht <dave@taht.net> wrote:
> I excitedly accessed it directly from gnus. Oh, joy, bliss, rapture!
> Blazing fast, useful at a glance.
> 
> In a paroxysm of overenthusiasm for the the good ole days I then setup a
> leafnode real quick and pointed it at his instance so I could replicate...
> 
> For most (not all) traffic, leafnode does this:
> 
> > Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
> > considering articles 200621 - 200643
> > Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article 200621
> > - no Path:  found

I guess there's some bug or missing commands in public-inbox-nntpd
which leafnode relies on.  Sorry I haven't had time to investigate
or do much with public-inbox, lately (but maybe you could :)

> :(
> 
> I'm really glad to hear of this project. It sounds like more of the
> right way to do this is via a local git?

Yes, git clone/fetch + running your own public-inbox-nntpd instance
and using slrn/gnus/mutt(w/ nntp patch)/sylpheed/thunderbird all
seem to work.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: interop with leafnode
  2018-10-27  0:10 ` Eric Wong
@ 2018-10-27  0:33   ` Dave Taht
  2018-10-27  0:43     ` Eric Wong
  2019-05-05 21:40   ` Eric Wong
  1 sibling, 1 reply; 9+ messages in thread
From: Dave Taht @ 2018-10-27  0:33 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta


Eric Wong <e@80x24.org> writes:

> Dave Taht <dave@taht.net> wrote:
>> I excitedly accessed it directly from gnus. Oh, joy, bliss, rapture!
>> Blazing fast, useful at a glance.
>> 
>> In a paroxysm of overenthusiasm for the the good ole days I then
>> setup a
>> leafnode real quick and pointed it at his instance so I could
>> replicate...
>> 
>> For most (not all) traffic, leafnode does this:
>> 
>> > Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
>> > considering articles 200621 - 200643
>> > Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article
>> > 200621
>> > - no Path:  found
>
> I guess there's some bug or missing commands in public-inbox-nntpd
> which leafnode relies on.  Sorry I haven't had time to investigate
> or do much with public-inbox, lately (but maybe you could :)
>
>> :(
>> 
>> I'm really glad to hear of this project. It sounds like more of the
>> right way to do this is via a local git?
>
> Yes, git clone/fetch + running your own public-inbox-nntpd instance
> and using slrn/gnus/mutt(w/ nntp patch)/sylpheed/thunderbird all
> seem to work.

I will try to modernize. 

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: interop with leafnode
  2018-10-27  0:33   ` Dave Taht
@ 2018-10-27  0:43     ` Eric Wong
  2018-10-27 16:07       ` Dave Taht
  0 siblings, 1 reply; 9+ messages in thread
From: Eric Wong @ 2018-10-27  0:43 UTC (permalink / raw)
  To: Dave Taht; +Cc: meta

Dave Taht <dave@taht.net> wrote:
> I will try to modernize. 

It's probably only missing a few commands to support leafnode
or other stuff (e.g. NNTP replication with innd).  I just don't
have experience with leafnode, and lately I've been fighting
with hardware problems, bills and taxes :<

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: interop with leafnode
  2018-10-27  0:43     ` Eric Wong
@ 2018-10-27 16:07       ` Dave Taht
  0 siblings, 0 replies; 9+ messages in thread
From: Dave Taht @ 2018-10-27 16:07 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong <e@80x24.org> writes:

> Dave Taht <dave@taht.net> wrote:
>> I will try to modernize. 
>
> It's probably only missing a few commands to support leafnode
> or other stuff (e.g. NNTP replication with innd).  I just don't
> have experience with leafnode, and lately I've been fighting
> with hardware problems, bills and taxes :<

Back in the day I used to maintain an innd instance, too.

I am tempted to start up a news instance again, just because
the traffic rate is now so low and because I miss things like comp.arch

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: interop with leafnode
  2018-10-27  0:10 ` Eric Wong
  2018-10-27  0:33   ` Dave Taht
@ 2019-05-05 21:40   ` Eric Wong
  1 sibling, 0 replies; 9+ messages in thread
From: Eric Wong @ 2019-05-05 21:40 UTC (permalink / raw)
  To: Dave Taht; +Cc: meta

Eric Wong <e@80x24.org> wrote:
> I guess there's some bug or missing commands in public-inbox-nntpd
> which leafnode relies on.  Sorry I haven't had time to investigate
> or do much with public-inbox, lately (but maybe you could :)

I still haven't tried leafnode, but I just remembered I fixed an
NNTP bug a while back to give it compatibility with Alpine:

https://public-inbox.org/meta/20190116045802.ikm67b4cwigso3tc@dcvr/

commit d66aa534a4a7506cfc5cfab49d1e09f8db8be3dd
("nntp: header responses use CRLF consistently")

Perhaps it's the same bug you hit with leafnode?

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: interop with leafnode
  2018-10-17 16:28 interop with leafnode Dave Taht
  2018-10-27  0:10 ` Eric Wong
@ 2019-06-13  7:00 ` Eric Wong
  2019-06-13 17:52   ` Dave Taht
  1 sibling, 1 reply; 9+ messages in thread
From: Eric Wong @ 2019-06-13  7:00 UTC (permalink / raw)
  To: Dave Taht; +Cc: meta

Dave Taht <dave@taht.net> wrote:
> For most (not all) traffic, leafnode does this:
> 
> > Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
> > considering articles 200621 - 200643
> > Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article 200621
> > - no Path:  found

So yeah, I Cc-ed you on <20190613065413.4705-1-e@80x24.org>.
Apparently leafnode doesn't need any real Path:, and just a
single byte seems to work.

However, I've also noticed leafnode (finally tried it) also
fails if a Message-ID is too long and spans multiple lines
(those were from outlook.com, go figure).

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: interop with leafnode
  2019-06-13  7:00 ` Eric Wong
@ 2019-06-13 17:52   ` Dave Taht
  2019-06-13 20:46     ` [PATCH] nntp: ensure Message-ID is not folded for leafnode Eric Wong
  0 siblings, 1 reply; 9+ messages in thread
From: Dave Taht @ 2019-06-13 17:52 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong <e@80x24.org> writes:

> Dave Taht <dave@taht.net> wrote:
>> For most (not all) traffic, leafnode does this:
>> 
>> > Oct 16 22:08:01 spaceheater fetchnews[3045]: lwn.kernel.lkml:
>> > considering articles 200621 - 200643
>> > Oct 16 22:08:01 spaceheater fetchnews[3045]: Discarding article
>> > 200621
>> > - no Path:  found
>
> So yeah, I Cc-ed you on <20190613065413.4705-1-e@80x24.org>.
> Apparently leafnode doesn't need any real Path:, and just a
> single byte seems to work.
>
> However, I've also noticed leafnode (finally tried it) also
> fails if a Message-ID is too long and spans multiple lines
> (those were from outlook.com, go figure).

Thanks! Is there a way to reformat that?

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH] nntp: ensure Message-ID is not folded for leafnode
  2019-06-13 17:52   ` Dave Taht
@ 2019-06-13 20:46     ` Eric Wong
  0 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2019-06-13 20:46 UTC (permalink / raw)
  To: Dave Taht; +Cc: meta

Dave Taht <dave@taht.net> wrote:
> Eric Wong <e@80x24.org> writes:
> > However, I've also noticed leafnode (finally tried it) also
> > fails if a Message-ID is too long and spans multiple lines
> > (those were from outlook.com, go figure).
> 
> Thanks! Is there a way to reformat that?

Yeah, this seems to work:

-----------8<------------
Subject: [PATCH] nntp: ensure Message-ID is not folded for leafnode

Leafnode cannot handle Message-ID headers which are too long and
require folding via Email::Simple::Header.  Since there are
already many of these messages in git with the header already
folded, we need to handle the unfolding when emitting the
message via NNTP.

As far as we know, Leafnode is the only client software
incapable of handling this case.
---
 lib/PublicInbox/NNTP.pm |  7 +++++++
 t/nntpd.t               | 20 ++++++++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 1d07220..d409e78 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -519,6 +519,13 @@ sub _header ($) {
 	my $hdr = $_[0]->header_obj->as_string;
 	utf8::encode($hdr);
 	$hdr =~ s/(?<!\r)\n/\r\n/sg;
+
+	# for leafnode compatibility, we need to ensure Message-ID headers
+	# are only a single line.  We can't subclass Email::Simple::Header
+	# and override _default_fold_at in here, either; since that won't
+	# affect messages already in the archive.
+	$hdr =~ s/^(Message-ID:)[ \t]*\r\n[ \t]+([^\r]+)\r\n/$1 $2\r\n/igsm;
+
 	$hdr
 }
 
diff --git a/t/nntpd.t b/t/nntpd.t
index c73cc12..cce21ee 100644
--- a/t/nntpd.t
+++ b/t/nntpd.t
@@ -231,6 +231,26 @@ EOF
 		ok($date >= $t0, 'valid date after start');
 		ok($date <= $t1, 'valid date before stop');
 	}
+	if ('leafnode interop') {
+		my $for_leafnode = PublicInbox::MIME->new(<<"");
+From: longheader\@example.com
+To: $addr
+Subject: none
+Date: Fri, 02 Oct 1993 00:00:00 +0000
+
+		my $long_hdr = 'for-leafnode-'.('y'x200).'@example.com';
+		$for_leafnode->header_set('Message-ID', "<$long_hdr>");
+		$im->add($for_leafnode);
+		$im->done;
+		if ($version == 1) {
+			my $s = PublicInbox::SearchIdx->new($mainrepo, 1);
+			$s->index_sync;
+		}
+		my $hdr = $n->head("<$long_hdr>");
+		my $expect = qr/\AMessage-ID: /i . qr/\Q<$long_hdr>\E/;
+		ok(scalar(grep(/$expect/, @$hdr)), 'Message-ID not folded');
+		ok(scalar(grep(/^Path:/, @$hdr)), 'Path: header found');
+	}
 
 	# pipelined requests:
 	{
-- 
EW

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, back to index

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-17 16:28 interop with leafnode Dave Taht
2018-10-27  0:10 ` Eric Wong
2018-10-27  0:33   ` Dave Taht
2018-10-27  0:43     ` Eric Wong
2018-10-27 16:07       ` Dave Taht
2019-05-05 21:40   ` Eric Wong
2019-06-13  7:00 ` Eric Wong
2019-06-13 17:52   ` Dave Taht
2019-06-13 20:46     ` [PATCH] nntp: ensure Message-ID is not folded for leafnode Eric Wong

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox