user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <>
To: Julien ÉLIE <>
Subject: Re: NNTP COMPRESS clients? RFC 8054
Date: Mon, 15 Jul 2019 01:25:46 +0000
Message-ID: <20190715012546.ljpdiavoesixy3sp@whir> (raw)
In-Reply-To: <>

Julien ÉLIE <> wrote:
> Hi Eric,
> Eric Wong wrote:
> > I've prepared a patch to the
> > Perl Net::NNTP module and am hoping I got everything right...
> > I've only lightly tested it against which runs inn:
> > 
> >
> I've just taken the time to have a look at it.  A few comments below:

Hi Julien,

Fwiw, I was more interested in making sure Net::NNTP client was
right OK in that RT ticket:

	git clone
	(nntp-compress branch)

But it looks like you've looked at the public-inbox-nntpd server
code at `git clone', but I appreciate
the comments either way :)

+Cc: since it's on topic, there..

> +	LINE_MAX => 512, # RFC 977 section 2.3
> Now "RFC 3977 Section 3.1" :)

Right, though I think a lot of the Net::NNTP client code (and
maybe a lot of other clients) is still RFC 977, so I'd like to
retain compatibility (and I'm still catching up and missing
some commands)

> +	$err == Z_OK or die "Failed to initialize zlib deflate stream: $err";
> Shouldn't a 403 response code be offered?  The connection then goes on,
> uncompressed.

Right, the inflate stream now returns 403:

I also made a another change to do a single deflate stream at
startup to save a bunch of memory at the expense of reduced
compression.  If that fails, the rest of the NNTP server
continues w/o that capability:

> +# RSS usage for 10K idle-but-did-something NNTP clients on 64-bit:
> +#   TLS + DEFLATE :  1.8 GB  (MemLevel=9, 1.2 GB with MemLevel=8)
> +#   TLS only      :  <200MB
> +#   plain         :   <50MB
> How did you test those 10k connections?  (with your test suite
> nntpd-validate.t?)

It's not part of the test suite, yet.   I do notice a drastic
memory difference on the server depending on whether the client
is using Danga::Socket or PublicInbox::DS(*)

Will report back on that later (though maybe much later...)

(*) - the latter is a gutted, API-unstable fork of Danga::Socket
      which allows use of EPOLLONESHOT/EV_DISPATCH

> +	# nnrpd (INN) and Compress::Raw::Zlib favor MemLevel=9,
> +	# but the zlib C library and git use MemLevel=8
> +	# as the default.  Using 8 drops our memory use with 10K
> +	# TLS clients from 1.8 GB to 1.2 GB, but...
> +	# FIXME: sometimes clients fail with 8, so we use 9
> +	# -MemLevel => 9,
> Strange that clients fail with mem level 8.  Seems like a bug with such
> clients.
> Hopefully nnrpd uses mem level 9 anyway.

Not a bug in clients; it was a stupid bug on the server not
realizing Deflate->new would return an array when used inside []:

> +	# Net::NNTP emit different line-endings depending on TLS or not...:
> +	$data =~ tr/\r//d;
> Shouldn't it be fixed in Net::NNTP?

Maybe, but it's not a big issue AFAIK.

I'm still traumatized from dealing with method resolution
confusion while implementing COMPRESS on the client side :x
I documented some of this the other day at:

It'll be up to the libnet maintainer(s) on whether to fix it.

> > I also implemented it server-side in public-inbox-nntpd[1],
> > which powers, but there's a danger I'm
> > implementing something wrong on both ends in that case :x
> Time to implement it in then :)
> It runs Colobus, an NNTP news server written in Perl by Ask Bjørn Hansen.

My plan is to keep public-inbox-nntpd read-only; since (AFAIK)
NNTP lacks good spam filtering and most news->mail gateways get
shut down because of that.  But it might be easy for Ask or
anybody else to implement posting if they desire...

I also don't want to hassle him or anybody at with
migrating, given given how little load it sees:

       reply index

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20190706044942.wmgjwbfy4vwntogu@dcvr>
     [not found] ` <>
2019-07-15  1:25   ` Eric Wong [this message]
2019-07-15 20:17     ` Julien ÉLIE
2019-09-21 20:35       ` Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190715012546.ljpdiavoesixy3sp@whir \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:

 note: .onion URLs require Tor:

AGPL code for this site: git clone