From: Eric Wong <firstname.lastname@example.org> To: Konstantin Ryabitsev <email@example.com> Cc: firstname.lastname@example.org Subject: Re: httpd 502s [was: trying to figure out 100% CPU usage in nntpd...] Date: Fri, 13 Sep 2019 03:12:12 +0000 Message-ID: <20190913031212.dkbtvj3ij6qqm6im@dcvr> (raw) In-Reply-To: <20190912113758.GB29277@pure.paranoia.local> Konstantin Ryabitsev <email@example.com> wrote: > On Thu, Sep 12, 2019 at 08:35:03AM +0000, Eric Wong wrote: > > Eric Wong <firstname.lastname@example.org> wrote: > > > One more thing, are you running any extra middlewares in the > > > .psgi file? Thanks. > > No, it's just vanilla what comes with the source. OK, and Perl 5.16.3 from CentOS 7? (4:5.16.3-294.el7_6 RPM) > > That's probably not it, I suspected the non-fileno path was > > being hit, but I just tested a debug change on top of > > b7cfd5aeff4b6b316b61b327af9c144776d77225 (branch: "unlink") > > ("tmpfile: support O_APPEND and use it in DS::tmpio") > > to fake the presence of a middleware wrapping psgi.input. > > I sent you a dump of lsof -p of all 4 processes after about 20 minutes > of running. For another data point, the daemon was running in > SELinux-permissive mode, to make sure unlinks aren't failing because of > any permission errors. It looks like there's a Perl reference leak (cycle) of some sort holding on to FDs, since you have lots of input files and pipes, yet only one established IPv4 connection. And the inodes encoded into the filenames don't point to the connected socket, even.... However, I'm not able to reproduce it on my CentOS 7 VM which has nginx 1.12.2. I don't think nginx is a factor in this since public-inbox-httpd is clearly not holding TCP sockets open, even. Not at all familiar with SELinux, but I'm just using the defaults CentOS comes with and running both nginx + public-inbox-httpd as a regular user. That "if (0..." GitHTTPBackend patch definitely isn't needed for testing anymore and only makes FD exhaustion happen sooner. > Let me know if you would like any further info. If there's a reference leak somewhere, this could also be part of the high memory use you showed us a few months ago. Dunno if you had many FDs back then. I could see about adding explicit close() calls in a few places, but that would make a corresponding memory leak harder-to-notice, even. I pushed out two patches to the "unlink" branch which may be able to reproduce the issue on your end (I see nothing out of the ordinary on my slow CentOS 7 VM or Debian machines/VMs) * [PATCH] t/httpd-corner: check for leaking FDs and pipes * [RFC] t/git-http-backend: add MANY_CLONE test # no pipes should be present in -httpd with -W0 prove -lv t/httpd-corner.t # unrelated note: there's 4 pipes from -W1 (the default), # but I think 2 can be closed, actually... GIANT_GIT_DIR=/path/to/git.git MANY_CLONE=1 prove -lv t/git-http-backend.t If those updated test cases can't reproduce the problem, can you reproduce this on erol or any other machines? perhaps with a different Perl? Thanks.
next prev parent reply index Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-09-08 10:45 [PATCH] nntp: regexp always consumes rbuf if "\n" exists Eric Wong 2019-09-08 10:52 ` trying to figure out 100% CPU usage in nntpd Eric Wong 2019-09-09 10:05 ` Konstantin Ryabitsev 2019-09-09 17:53 ` Eric Wong 2019-09-10 8:38 ` Konstantin Ryabitsev 2019-09-10 18:12 ` Eric Wong 2019-09-11 2:22 ` httpd 502s [was: trying to figure out 100% CPU usage in nntpd...] Eric Wong 2019-09-11 10:24 ` Konstantin Ryabitsev 2019-09-11 17:12 ` Eric Wong 2019-09-11 17:36 ` Konstantin Ryabitsev 2019-09-12 0:05 ` Eric Wong 2019-09-12 2:49 ` Eric Wong 2019-09-12 8:35 ` Eric Wong 2019-09-12 11:37 ` Konstantin Ryabitsev 2019-09-13 3:12 ` Eric Wong [this message] 2019-09-13 7:03 ` Eric Wong 2019-09-13 9:01 ` Eric Wong 2019-09-13 18:07 ` Konstantin Ryabitsev 2019-09-14 5:25 ` Eric Wong 2019-09-11 9:44 ` trying to figure out 100% CPU usage in nntpd Konstantin Ryabitsev 2019-09-11 17:12 ` Eric Wong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://public-inbox.org/README * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190913031212.dkbtvj3ij6qqm6im@dcvr \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
user/dev discussion of public-inbox itself Archives are clonable: git clone --mirror http://public-inbox.org/meta git clone --mirror http://czquwvybam4bgbro.onion/meta git clone --mirror http://hjrcffqmbrq6wope.onion/meta git clone --mirror http://ou63pmih66umazou.onion/meta Example config snippet for mirrors Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta nntp://news.gmane.io/gmane.mail.public-inbox.general note: .onion URLs require Tor: https://www.torproject.org/ AGPL code for this site: git clone https://public-inbox.org/public-inbox.git