From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id E17F81F463; Thu, 12 Sep 2019 00:05:41 +0000 (UTC) Date: Thu, 12 Sep 2019 00:05:41 +0000 From: Eric Wong To: Konstantin Ryabitsev Cc: meta@public-inbox.org Subject: Re: httpd 502s [was: trying to figure out 100% CPU usage in nntpd...] Message-ID: <20190912000541.gikjoimbdeahh7lx@whir> References: <20190908104518.11919-1-e@80x24.org> <20190908105243.GA15983@dcvr> <20190909100500.GA9452@pure.paranoia.local> <20190909175340.u5aq4ztfzukko7zb@dcvr> <20190910083820.GA8018@pure.paranoia.local> <20190910181224.urhyoo6av7mhjs67@dcvr> <20190911022215.GA309@dcvr> <20190911102436.GA21959@pure.paranoia.local> <20190911171250.vqqpaeb7sn34hv3s@dcvr> <20190911173628.GA14147@pure.paranoia.local> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20190911173628.GA14147@pure.paranoia.local> List-Id: Konstantin Ryabitsev wrote: > On Wed, Sep 11, 2019 at 05:12:50PM +0000, Eric Wong wrote: > > Konstantin Ryabitsev wrote: > > > To give some more data points, downgrading to f4f0a3be still shows a > > > number of /tmp/PerlIO* (deleted) entries, but the number of pipes stays > > > the same over time. If I switch to the latest master, the number of > > > broken pipes grows steadily following each git pull (in the hundreds > > > after 10 minutes of running). > > > > Thanks for that info, did those deleted entries eventually go > > away (perhaps after several minutes)? > > They don't appear to go away -- since 10:23 UTC earlier today, they > accumulated over 2,400 entries: > > # ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | wc -l > 2427 > # ls -al /proc/{2103,2104,2105,2106}/fd | grep pipe | wc -l > 26 > > Curiously, I also have this datapoint that may or may not be making > things more confusing. :) > > # ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | awk '{print $8}' | sort | uniq -c > 695 10:24 > 356 10:27 > 843 14:45 > 175 14:46 > 6 14:50 > 372 17:19 > 4 17:20 > 11 17:23 > > So, they appear to show up in chunks and hang around together. Any other cronjobs or events which may fire around then? Anything in nginx error logs? I've never seen anything like this. It should not matter as HTTP should be HTTP; but I've always had -httpd talking to varnish (mainly out of laziness, but I've never used nginx with public-inbox myself, but also ease of documentation since the varnish-4.vcl example teaches Varnish to only cache GET and HEAD requests). > When setting up lore, I also considered serving git via git-http-backend > instead of passing it via public-inbox-httpd -- the only reason we > didn't try that was because it complicated SELinux bits a bit. Can you > think of a reason why we shouldn't server git requests directly via > git-http-backend? I didn't realize nginx added the ability to spawn CGI processes, (but I haven't really looked at nginx this decade). public-inbox-httpd has the qspawn/limiter stuff and the same 32-process default limit as git-daemon. I actually limit it to 1-2 processes depending on the size of inbox given the tiny VPS which powers public-inbox.org. It's all documented in the -config manpage ("NAMED LIMITER"/"httpbackendmax"). My favorite feature is being able to configure different inboxes for different queues, so clone requests for large inboxes won't starve requests to small inboxes. My goal is to allow -httpd to do HTTPS and proxy to varnish soonish (eliminating the need for nginx or similar), and eventually eliminate the need for varnish entirely. > Thanks for your help. No problem; thanks for the extra info. Will try to finish my current (non-public) tasks so I can dig into this ASAP.