Re: httpd 502s [was: trying to figure out 100% CPU usage in nntpd...]

[Konstantin Ryabitsev <konstantin@linuxfoundation.org>]

On Wed, Sep 11, 2019 at 05:12:50PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > To give some more data points, downgrading to f4f0a3be still shows a
> > number of /tmp/PerlIO* (deleted) entries, but the number of pipes stays
> > the same over time. If I switch to the latest master, the number of
> > broken pipes grows steadily following each git pull (in the hundreds
> > after 10 minutes of running).
> 
> Thanks for that info, did those deleted entries eventually go
> away (perhaps after several minutes)?

They don't appear to go away -- since 10:23 UTC earlier today, they
accumulated over 2,400 entries:

# ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | wc -l
2427
# ls -al /proc/{2103,2104,2105,2106}/fd | grep pipe | wc -l
26

Curiously, I also have this datapoint that may or may not be making
things more confusing. :)

# ls -al /proc/{2103,2104,2105,2106}/fd | grep deleted | awk '{print $8}' | sort | uniq -c
    695 10:24
    356 10:27
    843 14:45
    175 14:46
      6 14:50
    372 17:19
      4 17:20
     11 17:23

So, they appear to show up in chunks and hang around together.

For nginx configs, this is the relevant part:

server {
  listen *:80 default_server;

  proxy_buffering off;

  location ~ ^(.*/(HEAD|info/refs|objects/info/[^/]+|git-(upload|receive)-pack))$ {

    proxy_pass            http://localhost:8080;
    proxy_read_timeout    90;
    proxy_connect_timeout 90;
    proxy_redirect        off;

    proxy_set_header      Host $host;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header      X-Forwarded-Proto $http_x_forwarded_proto;
    proxy_set_header      X-Forwarded-Port $server_port;

  }
}

When setting up lore, I also considered serving git via git-http-backend
instead of passing it via public-inbox-httpd -- the only reason we
didn't try that was because it complicated SELinux bits a bit. Can you
think of a reason why we shouldn't server git requests directly via
git-http-backend?

Thanks for your help.

-K