about summary refs log tree commit homepage
path: root/lib/PublicInbox/NNTPdeflate.pm
DateCommit message (Collapse)
2019-07-14nntpdeflate: reduce overhead of idle clients
We don't need to keep an empty buffer around in the common case when a client is sending us completely inflatable requests and we're able to read them in one go. This only seems to save about 2M with 10K NNTPS clients using COMPRESS, so it's not a huge win, but better than nothing.
2019-07-13nntpdeflate: stop relying on SUPER for ->do_read
We won't need further layering after enabling compression. So be explicit about which sub we're calling when we hit ->do_read from NNTP and eliminate the need for the comment.
2019-07-07nntp: improve error reporting for COMPRESS
Add some checks for errors at initialization, though there's not much that can be done with ENOMEM-type errors aside from dropping clients. We can also get rid of the scary FIXME for MemLevel=8. It was a stupid error on my part in the original per-client deflate stream implementation calling C::R::Z::{Inflate,Deflate} in array context and getting the extra dualvar error code as a string result, causing the {zin}/{zout} array refs to have extra array elements.
2019-07-06nntp: reduce memory overhead of zlib
Using Z_FULL_FLUSH at the right places in our event loop, it appears we can share a single zlib deflate context across ALL clients in a process. The zlib deflate context is the biggest factor in per-client memory use, so being able to share that across many clients results in a large memory savings. With 10K idle-but-did-something NNTP clients connected to a single process on a 64-bit system, TLS+DEFLATE used around 1.8 GB of RSS before this change. It now uses around 300 MB. TLS via IO::Socket::SSL alone uses <200MB in the same situation, so the actual memory reduction is over 10x. This makes compression less efficient and bandwidth increases around 45% in informal testing, but it's far better than no compression at all. It's likely around the same level of compression gzip gives on the HTTP side. Security implications with TLS? I don't know, but I don't really care, either... public-inbox-nntpd doesn't support authentication and it's up to the client to enable compression. It's not too different than Varnish caching gzipped responses on the HTTP side and having responses go to multiple HTTPS clients.
2019-07-06nntp: support COMPRESS DEFLATE per RFC 8054
This is only tested so far with my patches to Net::NNTP at: https://rt.cpan.org/Ticket/Display.html?id=129967 Memory use in C10K situations is disappointing, but that's the nature of compression. gzip compression over HTTPS does have the advantage of not keeping zlib streams open when clients are idle, at the cost of worse compression.