From: Eric Wong <firstname.lastname@example.org> To: Konstantin Ryabitsev <email@example.com> Cc: firstname.lastname@example.org Subject: Re: how's memory usage on public-inbox-httpd? Date: Tue, 22 Oct 2019 19:11:44 +0000 Message-ID: <20191022191144.GB697@dcvr> (raw) In-Reply-To: <20191022172830.GB4960@chatter.i7.local> Konstantin Ryabitsev <email@example.com> wrote: > On Sat, Oct 19, 2019 at 12:11:44AM +0000, Eric Wong wrote: > >> It's been definitely dramatically better. We keep adding lists to > >> lore, so I > >> haven't really been able to watch memory usage after a long period of daemon > >> uptime, but it's never really gone very much above 1GB. In fact, we're > >> downgrading lore to a smaller instance in the near future since we don't > >> need to worry about running out of RAM any more. > > > >Cool, but 1GB is still an order of magnitude worse that what > >I'd expect :< I remember Email::MIME had huge explosions with > >some 30MB+ spam messages: > > https://public-inbox.org/meta/20190609083918.gfr2kurah7f2hysx@dcvr/ > > (maybe gmime can help) > > > >Depending on your storage speed/latency, more RAM can still help > >significantly with Xapian. The NVME stuff has amazing numbers, > >but my mobos are too old and I'm still stuck on SATA 2. > > My goal is to rework lore.kernel.org significantly. Currently, it's a > single system hosted at AWS that both receives mail and serves the > archives, but I would actually like to split it into two: > > - the archiver that just generates git repositories but serves no > traffic (probably running directly on mail.kernel.org). > - several front-ends that replicate repositories from the archiver and > provide http/nntp access, probably reusing mirrors.edge.kernel.org > nodes that run from us-west, us-east, eu-west and ap-east. > > That should provide both redundancy and better geographic availability > of the service. This requires some testing first to ensure that > grokmirror hooks and reindexing works reliably for replicated repo > collections. Yeah, I noticed some mirror indexing bugs at: https://public-inbox.org/meta/20191016211415.GA6084@dcvr/ But patches 4/3 and 5/3 seem to be doing everything right and I expect the series to be merged soon. I'm also planning to dogfood a mirror of lore off some of my .onions, soon. > >Is nntpd better? That only uses Email::Simple and not MIME; > >so less explosions. > > The number of people using nntp is several orders of magnitude lower, so > I'm not sure it's a good metric for anything. Hopefully nntp usage goes up over time.
next prev parent reply index Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-12-01 19:44 Eric Wong 2019-06-06 19:04 ` Konstantin Ryabitsev 2019-06-06 20:37 ` Eric Wong 2019-06-06 21:45 ` Konstantin Ryabitsev 2019-06-06 22:10 ` Eric Wong 2019-06-06 22:19 ` Konstantin Ryabitsev 2019-06-06 22:29 ` Eric Wong 2019-06-10 10:09 ` [RFC] optionally support glibc malloc_info via SIGCONT Eric Wong 2019-06-09 8:39 ` how's memory usage on public-inbox-httpd? Eric Wong 2019-06-12 17:08 ` Eric Wong 2019-06-06 20:54 ` Eric Wong 2019-10-16 22:10 ` Eric Wong 2019-10-18 19:23 ` Konstantin Ryabitsev 2019-10-19 0:11 ` Eric Wong 2019-10-22 17:28 ` Konstantin Ryabitsev 2019-10-22 19:11 ` Eric Wong [this message] 2019-10-28 23:24 ` Eric Wong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://public-inbox.org/README * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20191022191144.GB697@dcvr \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
user/dev discussion of public-inbox itself Archives are clonable: git clone --mirror http://public-inbox.org/meta git clone --mirror http://czquwvybam4bgbro.onion/meta git clone --mirror http://hjrcffqmbrq6wope.onion/meta git clone --mirror http://ou63pmih66umazou.onion/meta Example config snippet for mirrors Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta nntp://news.gmane.io/gmane.mail.public-inbox.general note: .onion URLs require Tor: https://www.torproject.org/ AGPL code for this site: git clone https://public-inbox.org/public-inbox.git