user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: httpd 502s [was: trying to figure out 100% CPU usage in nntpd...]
Date: Thu, 12 Sep 2019 00:05:41 +0000
Message-ID: <20190912000541.gikjoimbdeahh7lx@whir> (raw)
In-Reply-To: <20190911173628.GA14147@pure.paranoia.local>

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Wed, Sep 11, 2019 at 05:12:50PM +0000, Eric Wong wrote:
> > Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > > To give some more data points, downgrading to f4f0a3be still shows a
> > > number of /tmp/PerlIO* (deleted) entries, but the number of pipes stays
> > > the same over time. If I switch to the latest master, the number of
> > > broken pipes grows steadily following each git pull (in the hundreds
> > > after 10 minutes of running).
> > 
> > Thanks for that info, did those deleted entries eventually go
> > away (perhaps after several minutes)?
> 
> They don't appear to go away -- since 10:23 UTC earlier today, they
> accumulated over 2,400 entries:
> 
> # ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | wc -l
> 2427
> # ls -al /proc/{2103,2104,2105,2106}/fd | grep pipe | wc -l
> 26
> 
> Curiously, I also have this datapoint that may or may not be making
> things more confusing. :)
> 
> # ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | awk '{print $8}' | sort | uniq -c
>     695 10:24
>     356 10:27
>     843 14:45
>     175 14:46
>       6 14:50
>     372 17:19
>       4 17:20
>      11 17:23
> 
> So, they appear to show up in chunks and hang around together.

Any other cronjobs or events which may fire around then?

Anything in nginx error logs?

I've never seen anything like this.  It should not matter as
HTTP should be HTTP; but I've always had -httpd talking to varnish
(mainly out of laziness, but I've never used nginx with
public-inbox myself, but also ease of documentation since
the varnish-4.vcl example teaches Varnish to only cache GET and
HEAD requests).

> When setting up lore, I also considered serving git via git-http-backend
> instead of passing it via public-inbox-httpd -- the only reason we
> didn't try that was because it complicated SELinux bits a bit. Can you
> think of a reason why we shouldn't server git requests directly via
> git-http-backend?

I didn't realize nginx added the ability to spawn CGI processes,
(but I haven't really looked at nginx this decade).

public-inbox-httpd has the qspawn/limiter stuff and the same
32-process default limit as git-daemon.  I actually limit it to
1-2 processes depending on the size of inbox given the tiny VPS
which powers public-inbox.org.  It's all documented in the -config
manpage ("NAMED LIMITER"/"httpbackendmax").  My favorite feature
is being able to configure different inboxes for different queues,
so clone requests for large inboxes won't starve requests to
small inboxes.

My goal is to allow -httpd to do HTTPS and proxy to varnish
soonish (eliminating the need for nginx or similar),
and eventually eliminate the need for varnish entirely.

> Thanks for your help.

No problem; thanks for the extra info.  Will try to finish my
current (non-public) tasks so I can dig into this ASAP.

  reply index

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-08 10:45 [PATCH] nntp: regexp always consumes rbuf if "\n" exists Eric Wong
2019-09-08 10:52 ` trying to figure out 100% CPU usage in nntpd Eric Wong
2019-09-09 10:05   ` Konstantin Ryabitsev
2019-09-09 17:53     ` Eric Wong
2019-09-10  8:38       ` Konstantin Ryabitsev
2019-09-10 18:12         ` Eric Wong
2019-09-11  2:22           ` httpd 502s [was: trying to figure out 100% CPU usage in nntpd...] Eric Wong
2019-09-11 10:24             ` Konstantin Ryabitsev
2019-09-11 17:12               ` Eric Wong
2019-09-11 17:36                 ` Konstantin Ryabitsev
2019-09-12  0:05                   ` Eric Wong [this message]
2019-09-12  2:49                     ` Eric Wong
2019-09-12  8:35                       ` Eric Wong
2019-09-12 11:37                         ` Konstantin Ryabitsev
2019-09-13  3:12                           ` Eric Wong
2019-09-13  7:03                             ` Eric Wong
2019-09-13  9:01                             ` Eric Wong
2019-09-13 18:07                             ` Konstantin Ryabitsev
2019-09-14  5:25                               ` Eric Wong
2019-09-11  9:44           ` trying to figure out 100% CPU usage in nntpd Konstantin Ryabitsev
2019-09-11 17:12             ` Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190912000541.gikjoimbdeahh7lx@whir \
    --to=e@80x24.org \
    --cc=konstantin@linuxfoundation.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox