From a3c054cbbf2a51be121b3237c5d223acc5c8a2f4 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Fri, 5 Jul 2019 22:53:39 +0000 Subject: nntp: reduce memory overhead of zlib Using Z_FULL_FLUSH at the right places in our event loop, it appears we can share a single zlib deflate context across ALL clients in a process. The zlib deflate context is the biggest factor in per-client memory use, so being able to share that across many clients results in a large memory savings. With 10K idle-but-did-something NNTP clients connected to a single process on a 64-bit system, TLS+DEFLATE used around 1.8 GB of RSS before this change. It now uses around 300 MB. TLS via IO::Socket::SSL alone uses <200MB in the same situation, so the actual memory reduction is over 10x. This makes compression less efficient and bandwidth increases around 45% in informal testing, but it's far better than no compression at all. It's likely around the same level of compression gzip gives on the HTTP side. Security implications with TLS? I don't know, but I don't really care, either... public-inbox-nntpd doesn't support authentication and it's up to the client to enable compression. It's not too different than Varnish caching gzipped responses on the HTTP side and having responses go to multiple HTTPS clients. --- lib/PublicInbox/DS.pm | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) (limited to 'lib/PublicInbox/DS.pm') diff --git a/lib/PublicInbox/DS.pm b/lib/PublicInbox/DS.pm index 586c47cd..b16b1896 100644 --- a/lib/PublicInbox/DS.pm +++ b/lib/PublicInbox/DS.pm @@ -553,7 +553,9 @@ sub msg_more ($$) { return 0; } } - $self->write(\($_[1])); + + # don't redispatch into NNTPdeflate::write + PublicInbox::DS::write($self, \($_[1])); } sub epwait ($$) { -- cgit v1.2.3-24-ge0c7