Re: NNTP COMPRESS clients? RFC 8054

Re: NNTP COMPRESS clients? RFC 8054

[Eric Wong]

Julien ÉLIE <julien@trigofacile.com> wrote:
> Hi Eric,
> 
> Eric Wong wrote:
> > I've prepared a patch to the
> > Perl Net::NNTP module and am hoping I got everything right...
> > I've only lightly tested it against news.gmane.org which runs inn:
> > 
> >    https://rt.cpan.org/Ticket/Display.html?id=129967
> 
> I've just taken the time to have a look at it.  A few comments below:

Hi Julien,

Fwiw, I was more interested in making sure Net::NNTP client was
right OK in that RT ticket:

	git clone https://80x24.org/perl-libnet.git
	(nntp-compress branch)

But it looks like you've looked at the public-inbox-nntpd server
code at `git clone https://public-inbox.org/', but I appreciate
the comments either way :)

+Cc: meta@public-inbox.org since it's on topic, there..

> +	LINE_MAX => 512, # RFC 977 section 2.3
> 
> Now "RFC 3977 Section 3.1" :)

Right, though I think a lot of the Net::NNTP client code (and
maybe a lot of other clients) is still RFC 977, so I'd like to
retain compatibility (and I'm still catching up and missing
some commands)

> +	$err == Z_OK or die "Failed to initialize zlib deflate stream: $err";
> 
> Shouldn't a 403 response code be offered?  The connection then goes on,
> uncompressed.

Right, the inflate stream now returns 403:

  https://public-inbox.org/meta/20190707070831.fafviqilsywuc5bw@dcvr/

I also made a another change to do a single deflate stream at
startup to save a bunch of memory at the expense of reduced
compression.  If that fails, the rest of the NNTP server
continues w/o that capability:

  https://public-inbox.org/meta/20190705225339.5698-5-e@80x24.org/

> +# RSS usage for 10K idle-but-did-something NNTP clients on 64-bit:
> +#   TLS + DEFLATE :  1.8 GB  (MemLevel=9, 1.2 GB with MemLevel=8)
> +#   TLS only      :  <200MB
> +#   plain         :   <50MB
> 
> How did you test those 10k connections?  (with your test suite
> nntpd-validate.t?)

It's not part of the test suite, yet.   I do notice a drastic
memory difference on the server depending on whether the client
is using Danga::Socket or PublicInbox::DS(*)

Will report back on that later (though maybe much later...)

(*) - the latter is a gutted, API-unstable fork of Danga::Socket
      which allows use of EPOLLONESHOT/EV_DISPATCH

> +	# nnrpd (INN) and Compress::Raw::Zlib favor MemLevel=9,
> +	# but the zlib C library and git use MemLevel=8
> +	# as the default.  Using 8 drops our memory use with 10K
> +	# TLS clients from 1.8 GB to 1.2 GB, but...
> +	# FIXME: sometimes clients fail with 8, so we use 9
> +	# -MemLevel => 9,
> 
> Strange that clients fail with mem level 8.  Seems like a bug with such
> clients.
> Hopefully nnrpd uses mem level 9 anyway.

Not a bug in clients; it was a stupid bug on the server not
realizing Deflate->new would return an array when used inside []:

  https://public-inbox.org/meta/20190707070831.fafviqilsywuc5bw@dcvr/

> +	# Net::NNTP emit different line-endings depending on TLS or not...:
> +	$data =~ tr/\r//d;
> 
> Shouldn't it be fixed in Net::NNTP?

Maybe, but it's not a big issue AFAIK.

I'm still traumatized from dealing with method resolution
confusion while implementing COMPRESS on the client side :x
I documented some of this the other day at:

  https://rt.cpan.org/Public/Bug/Display.html?id=129966#txn-1854816

It'll be up to the libnet maintainer(s) on whether to fix it.

> > I also implemented it server-side in public-inbox-nntpd[1],
> > which powers news.public-inbox.org, but there's a danger I'm
> > implementing something wrong on both ends in that case :x
> 
> Time to implement it in nntp.perl.org then :)
> It runs Colobus, an NNTP news server written in Perl by Ask Bjørn Hansen.

My plan is to keep public-inbox-nntpd read-only; since (AFAIK)
NNTP lacks good spam filtering and most news->mail gateways get
shut down because of that.  But it might be easy for Ask or
anybody else to implement posting if they desire...

I also don't want to hassle him or anybody at perl.org with
migrating, given given how little load it sees:

  http://nntp.perl.org/group/perl.perl5.porters/255487

Re: NNTP COMPRESS clients? RFC 8054

[Julien ÉLIE]

Hi Eric,

> 	git clone https://80x24.org/perl-libnet.git
> 	(nntp-compress branch)

https://80x24.org/perl-libnet.git/commit/?h=nntp-compress&id=6f1c2c717f6e2858e5d62073c47efe11e597e067

"sysread may call SSL_read, and SSL_read may buffer extra
data in userspace which can be returned via sysread without
making another read(2) syscall.  This makes it it possible for
select() to block indefinitely since select() only knows about
buffers in the kernel, not userspace."

Nice catch!  Hopefully it will be fixed in Net::NNTP.
For your information, INN also suffered from that annoying bug in 
earlier 2.4.x versions.  That was why we finally by-passed the use of 
select() because it generated dead connections.  And also implemented a 
timeout alarm signal around SSL_read().
   https://inn.eyrie.org/trac/changeset/7755
   https://inn.eyrie.org/trac/changeset/7836



https://80x24.org/perl-libnet.git/commit/?h=nntp-compress&id=0ab5c363a69702e586b7ef06be3ebff3e78656b7

+  croak("NNTP connection already compressed ($comp)") if $comp;
+  croak("$alg not supported (only 'DEFLATE')") if $alg ne 'DEFLATE';

Note that these messages have different response codes:  502 for the 
first one, 503 for the last one.



+  can_deflate() or die $nodeflate_warn;
and
+  $err == Z_OK or die "Inflate->new failed: $err\n";
and
+  $err == Z_OK or die "Deflate->new failed: $err\n";

It should be a 403 response code, and then the connections goes on 
uncompressed (same remark found in your patch against public-inbox-nntpd).



> My plan is to keep public-inbox-nntpd read-only; since (AFAIK)
> NNTP lacks good spam filtering

Isn't Cleanfeed good enough for your needs?
   http://www.mixmin.net/cleanfeed/

I am also aware of a use of SpamAssassin with INN.  Maybe it would 
better fit your expectations?

Installation guide in French:
 
https://usenet-fr.redatomik.org/index.php/Installer_et_configurer_un_serveur_de_nouvelles#Filtrer_le_spam_avec_SpamAssassin



> and most news->mail gateways get shut down because of that.

Too bad :-(



> I also don't want to hassle him or anybody at perl.org with
> migrating, given how little load it sees:
> 
>    http://nntp.perl.org/group/perl.perl5.porters/255487

I see...  Too bad too :-(

-- 
Julien ÉLIE

« Ta remise sur pied lui a fait perdre la tête ! » (Astérix)

Re: NNTP COMPRESS clients? RFC 8054

[Eric Wong]

Julien ÉLIE <julien@trigofacile.com> wrote:
> Hi Eric,
> 
> > 	git clone https://80x24.org/perl-libnet.git
> > 	(nntp-compress branch)

(oops, totally forgot about this for a few months :x)
<snip>

> https://80x24.org/perl-libnet.git/commit/?h=nntp-compress&id=0ab5c363a69702e586b7ef06be3ebff3e78656b7
> 
> +  croak("NNTP connection already compressed ($comp)") if $comp;
> +  croak("$alg not supported (only 'DEFLATE')") if $alg ne 'DEFLATE';
> 
> Note that these messages have different response codes:  502 for the first
> one, 503 for the last one.

I'm not sure if faking NNTP server error codes in a client is
appropriate if the failure is entirely on the client side(*).

Fwiw, Net::NNTP currently dies if ->starttls is repeated,
so I'm following that behavior.

(*) LWP fakes HTTP error codes, and it annoyed me to no end
    back in the day...

> and
> +  $err == Z_OK or die "Inflate->new failed: $err\n";
> and
> +  $err == Z_OK or die "Deflate->new failed: $err\n";
> 
> It should be a 403 response code, and then the connections goes on
> uncompressed (same remark found in your patch against public-inbox-nntpd).

Also, I most perl code (including core) just croaks/dies to
raise an exception on allocation failure.  Bugs or ABI
mismatches in Compress::Raw::Zlib leading to failure are
probably to rare to do anything with besides croak/die.

> > My plan is to keep public-inbox-nntpd read-only; since (AFAIK)
> > NNTP lacks good spam filtering
> 
> Isn't Cleanfeed good enough for your needs?
>   http://www.mixmin.net/cleanfeed/
> 
> I am also aware of a use of SpamAssassin with INN.  Maybe it would better
> fit your expectations?

*shrug*  I also wouldn't know how NNTP <-> mail would work for a
decentralized workflow where everybody is expected to To/Cc one
another and merely agree on some mailing list addresses as a
logging/archival channel (this is how Linux kernel and git
development works, at least).

I wouldn't want NNTP servers to become open relays, either...

I still think email has the best chance of being a federated
identity provider for Free software development.