Re: httpd 502s [was: trying to figure out 100% CPU usage in nntpd...]

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Wed, Sep 11, 2019 at 05:12:50PM +0000, Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > > To give some more data points, downgrading to f4f0a3be still shows a
> > > number of /tmp/PerlIO* (deleted) entries, but the number of pipes stays
> > > the same over time. If I switch to the latest master, the number of
> > > broken pipes grows steadily following each git pull (in the hundreds
> > > after 10 minutes of running).
> > 
> > Thanks for that info, did those deleted entries eventually go
> > away (perhaps after several minutes)?
> 
> They don't appear to go away -- since 10:23 UTC earlier today, they
> accumulated over 2,400 entries:
> 
> # ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | wc -l
> 2427
> # ls -al /proc/{2103,2104,2105,2106}/fd | grep pipe | wc -l
> 26
> 
> Curiously, I also have this datapoint that may or may not be making
> things more confusing. :)
> 
> # ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | awk '{print $8}' | sort | uniq -c
>     695 10:24
>     356 10:27
>     843 14:45
>     175 14:46
>       6 14:50
>     372 17:19
>       4 17:20
>      11 17:23
> 
> So, they appear to show up in chunks and hang around together.

Any other cronjobs or events which may fire around then?

Anything in nginx error logs?

I've never seen anything like this.  It should not matter as
HTTP should be HTTP; but I've always had -httpd talking to varnish
(mainly out of laziness, but I've never used nginx with
public-inbox myself, but also ease of documentation since
the varnish-4.vcl example teaches Varnish to only cache GET and
HEAD requests).

> When setting up lore, I also considered serving git via git-http-backend
> instead of passing it via public-inbox-httpd -- the only reason we
> didn't try that was because it complicated SELinux bits a bit. Can you
> think of a reason why we shouldn't server git requests directly via
> git-http-backend?

I didn't realize nginx added the ability to spawn CGI processes,
(but I haven't really looked at nginx this decade).

public-inbox-httpd has the qspawn/limiter stuff and the same
32-process default limit as git-daemon.  I actually limit it to
1-2 processes depending on the size of inbox given the tiny VPS
which powers public-inbox.org.  It's all documented in the -config
manpage ("NAMED LIMITER"/"httpbackendmax").  My favorite feature
is being able to configure different inboxes for different queues,
so clone requests for large inboxes won't starve requests to
small inboxes.

My goal is to allow -httpd to do HTTPS and proxy to varnish
soonish (eliminating the need for nginx or similar),
and eventually eliminate the need for varnish entirely.

> Thanks for your help.

No problem; thanks for the extra info.  Will try to finish my
current (non-public) tasks so I can dig into this ASAP.