From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 395EB1F4C0; Tue, 22 Oct 2019 19:11:45 +0000 (UTC) Date: Tue, 22 Oct 2019 19:11:44 +0000 From: Eric Wong To: Konstantin Ryabitsev Cc: meta@public-inbox.org Subject: Re: how's memory usage on public-inbox-httpd? Message-ID: <20191022191144.GB697@dcvr> References: <20181201194429.d5aldesjkb56il5c@dcvr> <20190606190455.GA17362@chatter.i7.local> <20191016221045.GA6828@dcvr> <20191018192352.GH25456@chatter.i7.local> <20191019001144.GA20824@dcvr> <20191022172830.GB4960@chatter.i7.local> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20191022172830.GB4960@chatter.i7.local> List-Id: Konstantin Ryabitsev wrote: > On Sat, Oct 19, 2019 at 12:11:44AM +0000, Eric Wong wrote: > >> It's been definitely dramatically better. We keep adding lists to > >> lore, so I > >> haven't really been able to watch memory usage after a long period of daemon > >> uptime, but it's never really gone very much above 1GB. In fact, we're > >> downgrading lore to a smaller instance in the near future since we don't > >> need to worry about running out of RAM any more. > > > >Cool, but 1GB is still an order of magnitude worse that what > >I'd expect :< I remember Email::MIME had huge explosions with > >some 30MB+ spam messages: > > https://public-inbox.org/meta/20190609083918.gfr2kurah7f2hysx@dcvr/ > > (maybe gmime can help) > > > >Depending on your storage speed/latency, more RAM can still help > >significantly with Xapian. The NVME stuff has amazing numbers, > >but my mobos are too old and I'm still stuck on SATA 2. > > My goal is to rework lore.kernel.org significantly. Currently, it's a > single system hosted at AWS that both receives mail and serves the > archives, but I would actually like to split it into two: > > - the archiver that just generates git repositories but serves no > traffic (probably running directly on mail.kernel.org). > - several front-ends that replicate repositories from the archiver and > provide http/nntp access, probably reusing mirrors.edge.kernel.org > nodes that run from us-west, us-east, eu-west and ap-east. > > That should provide both redundancy and better geographic availability > of the service. This requires some testing first to ensure that > grokmirror hooks and reindexing works reliably for replicated repo > collections. Yeah, I noticed some mirror indexing bugs at: https://public-inbox.org/meta/20191016211415.GA6084@dcvr/ But patches 4/3 and 5/3 seem to be doing everything right and I expect the series to be merged soon. I'm also planning to dogfood a mirror of lore off some of my .onions, soon. > >Is nntpd better? That only uses Email::Simple and not MIME; > >so less explosions. > > The number of people using nntp is several orders of magnitude lower, so > I'm not sure it's a good metric for anything. Hopefully nntp usage goes up over time.