user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
To: Eric Wong <e@80x24.org>
Cc: meta@public-inbox.org
Subject: Re: httpd 502s [was: trying to figure out 100% CPU usage in nntpd...]
Date: Wed, 11 Sep 2019 13:36:28 -0400
Message-ID: <20190911173628.GA14147@pure.paranoia.local> (raw)
In-Reply-To: <20190911171250.vqqpaeb7sn34hv3s@dcvr>

On Wed, Sep 11, 2019 at 05:12:50PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > To give some more data points, downgrading to f4f0a3be still shows a
> > number of /tmp/PerlIO* (deleted) entries, but the number of pipes stays
> > the same over time. If I switch to the latest master, the number of
> > broken pipes grows steadily following each git pull (in the hundreds
> > after 10 minutes of running).
> 
> Thanks for that info, did those deleted entries eventually go
> away (perhaps after several minutes)?

They don't appear to go away -- since 10:23 UTC earlier today, they
accumulated over 2,400 entries:

# ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | wc -l
2427
# ls -al /proc/{2103,2104,2105,2106}/fd | grep pipe | wc -l
26

Curiously, I also have this datapoint that may or may not be making
things more confusing. :)

# ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | awk '{print $8}' | sort | uniq -c
    695 10:24
    356 10:27
    843 14:45
    175 14:46
      6 14:50
    372 17:19
      4 17:20
     11 17:23

So, they appear to show up in chunks and hang around together.

For nginx configs, this is the relevant part:

server {
  listen *:80 default_server;

  proxy_buffering off;

  location ~ ^(.*/(HEAD|info/refs|objects/info/[^/]+|git-(upload|receive)-pack))$ {

    proxy_pass            http://localhost:8080;
    proxy_read_timeout    90;
    proxy_connect_timeout 90;
    proxy_redirect        off;

    proxy_set_header      Host $host;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header      X-Forwarded-Proto $http_x_forwarded_proto;
    proxy_set_header      X-Forwarded-Port $server_port;

  }
}

When setting up lore, I also considered serving git via git-http-backend
instead of passing it via public-inbox-httpd -- the only reason we
didn't try that was because it complicated SELinux bits a bit. Can you
think of a reason why we shouldn't server git requests directly via
git-http-backend?

Thanks for your help.

-K

  reply index

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-08 10:45 [PATCH] nntp: regexp always consumes rbuf if "\n" exists Eric Wong
2019-09-08 10:52 ` trying to figure out 100% CPU usage in nntpd Eric Wong
2019-09-09 10:05   ` Konstantin Ryabitsev
2019-09-09 17:53     ` Eric Wong
2019-09-10  8:38       ` Konstantin Ryabitsev
2019-09-10 18:12         ` Eric Wong
2019-09-11  2:22           ` httpd 502s [was: trying to figure out 100% CPU usage in nntpd...] Eric Wong
2019-09-11 10:24             ` Konstantin Ryabitsev
2019-09-11 17:12               ` Eric Wong
2019-09-11 17:36                 ` Konstantin Ryabitsev [this message]
2019-09-12  0:05                   ` Eric Wong
2019-09-12  2:49                     ` Eric Wong
2019-09-12  8:35                       ` Eric Wong
2019-09-12 11:37                         ` Konstantin Ryabitsev
2019-09-13  3:12                           ` Eric Wong
2019-09-13  7:03                             ` Eric Wong
2019-09-13  9:01                             ` Eric Wong
2019-09-13 18:07                             ` Konstantin Ryabitsev
2019-09-14  5:25                               ` Eric Wong
2019-09-11  9:44           ` trying to figure out 100% CPU usage in nntpd Konstantin Ryabitsev
2019-09-11 17:12             ` Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190911173628.GA14147@pure.paranoia.local \
    --to=konstantin@linuxfoundation.org \
    --cc=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox