From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.4 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-qt1-x843.google.com (mail-qt1-x843.google.com [IPv6:2607:f8b0:4864:20::843]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 266A61F462 for ; Thu, 6 Jun 2019 21:45:14 +0000 (UTC) Received: by mail-qt1-x843.google.com with SMTP id d23so27071qto.2 for ; Thu, 06 Jun 2019 14:45:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Ot0vWFM+plawoWvDi4CP2fIle1wTXcqNhWAHVpsRgOk=; b=Eyl4Uz2zKJkftpe9FA+X2hzp/McA+6b9NXFFEvZK+xTlTt4xlBDcMJ6wRJqhVl7ZT4 p4FV39+0CIVrjZ4LzqbkkJvayaIq+5lgNzwQI9ROYkbCntbns6uRrAqapc1Ifv+/VwmO TP5aubg6adlDXScEzlbSkZyJlRkafe+bMlDpk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=Ot0vWFM+plawoWvDi4CP2fIle1wTXcqNhWAHVpsRgOk=; b=tO3yRdzpoeMXfGGeyOuy7BuV76mNBnsagTRwWAwtQmn4b6bW1NcSbg1+egrwl/Uc5q BiuVISkTRZoZX/ooetlpOiaGckK5I/N6nM244iqtQIIN0QnCy1HXrfOx+1XvmXXvyHFu tIoXxqCYOz90GuDCx34V7PvIwp39yLifF6KvxD+Uuj23F1bMrnL97NK4vYTbnSD427GV 8Mn7vLZH1DBIvR7j6omkS41ZaFpUOYD6pJ/+D68neooe4eiRi7TbQbjmsSi51P4pQHtj E96WAyLHaBjK4LW6l4SreHUwhmn/nUTXrbALEacydW0O3b4fTXoXTK/6gKefVpPXy7+h YhYA== X-Gm-Message-State: APjAAAXZhPT6mk4VnS4BVOr4ly+Xjsnb02y2wsjzIqCKV27RRdPnjRhT ks1TD6e6F+i+q47Kuh54RRQSwQ== X-Google-Smtp-Source: APXvYqzZ/ItuVNuHXo5u0FqiH+bL1trbTDsL6iyypgEj4Mna5BpzG7JGYWh8TLaEkBR84+1b5O4irg== X-Received: by 2002:ad4:5345:: with SMTP id v5mr28980902qvs.103.1559857512154; Thu, 06 Jun 2019 14:45:12 -0700 (PDT) Received: from chatter.i7.local (192-0-228-88.cpe.teksavvy.com. [192.0.228.88]) by smtp.gmail.com with ESMTPSA id p62sm60188qkc.42.2019.06.06.14.45.11 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Thu, 06 Jun 2019 14:45:11 -0700 (PDT) Date: Thu, 6 Jun 2019 17:45:09 -0400 From: Konstantin Ryabitsev To: Eric Wong Cc: meta@public-inbox.org Subject: Re: how's memory usage on public-inbox-httpd? Message-ID: <20190606214509.GA4087@chatter.i7.local> References: <20181201194429.d5aldesjkb56il5c@dcvr> <20190606190455.GA17362@chatter.i7.local> <20190606203752.7wpdla5ynemjlshs@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline In-Reply-To: <20190606203752.7wpdla5ynemjlshs@dcvr> User-Agent: Mutt/1.11.4 (2019-03-13) List-Id: On Thu, Jun 06, 2019 at 08:37:52PM +0000, Eric Wong wrote: >Do you have commit 7d02b9e64455831d3bda20cd2e64e0c15dc07df5? >("view: stop storing all MIME objects on large threads") >That was most significant. Yes. We're running 743ac758 with a few cherry-picked patches on top of that (like epoch roll-over fix). >Otherwise it's probably a combination of several things... >httpd and nntpd both supports streaming, arbitrarily large >endpoints (all.mbox.gz, and /T/, /t/, /t.mbox.gz threads with >thousands of messages, giant NNTP BODY/ARTICLE ranges). > >All those endpoints should detect backpressure from a slow >client (varnish/nginx in your case) using the ->getline method. Wouldn't that spike up and down? The size I'm seeing stays pretty constant without any significant changes across requests. >Also, are you only using the default of -W/--worker-process=1 >on a 16-core machine? Just checked public-inbox-httpd(8), the >-W switch is documented :) You can use SIGTTIN/TTOU to >increase, decrease workers w/o restarting, too. D'oh, yes... though it's not been a problem yet. :) I'm not sure I want to bump that up, though, if that means we're going to have multiple 19GB-sized processes instead of one. :) >Do you have any stats on the number of simultaneous connections >public-inbox-httpd/nginx/varnish handles (and logging of that >info at peek)? (perhaps running "ss -tan" periodically)(*) We don't collect that info, but I'm not sure it's the number of concurrent connections that's the culprit, as there is no fluctuation in RSS size based on the number of responses. To answer the questions in your follow-up: It would appear to be all in anon memory. Mem_usage [1] reports: # ./Mem_usage 18275 Backed by file: Executable r-x 16668 Write/Exec (jump tables) rwx 0 RO data r-- 106908 Data rw- 232 Unreadable --- 94072 Unknown 0 Anonymous: Writable code (stack) rwx 0 Data (malloc, mmap) rw- 19988892 RO data r-- 0 Unreadable --- 0 Unknown 12 I've been looking at lsof -p of that process and I see sqlite and xapian showing up and disappearing. The lkml ones are being accessed almost all the time, but even there I see them showing up with different FD entries, so they are being closed and reopened properly. Hope this helps. -K .. [1] https://elinux.org/images/d/d3/Mem_usage