about summary refs log tree commit
AgeCommit message (Collapse)AuthorFilesLines
2019-09-26httpd: disable Deflater middleware by default on Perl <5.18 p516-leakEric Wong1-1/+7
Testing with perl-5.16.3-294.el7_6 RPM package on RHEL/CentOS 7, the Deflater middleware triggers a leak when used in conjunction with our push-based responses from PublicInbox::Qspawn. I could not find another solution to workaround the memory leak in this case, and I could not find a specific leak fix in the perl5180delta manpage[1] which looked like it would solve our problem. Attempting to workaround the issue proved futile. Using internal Deflater-specific keys to prevent deflating in GitHTTPBackend and Qspawn did not solve the problem: $env->{"plack.skip-deflater"} = 1; $env->{"psgix.no-compress"} = 1; Nor did forcing an invalid encoding via "git fetch": git -c http.extraheader=Accept-Encoding:gzap fetch So this appears to be a problem with Plack::Util::response_cb somewhere. This does NOT appear to be a problem with ref() leaking as in DS::next_tick[2], since I couldn't find where Plack::Middleware::Deflater or Plack::Util::response_cb would be calling ref() on a blessed reference to trigger a leak. Also, oddly enough, the ref() use for backwards compatibility at the top of PublicInbox::GitHTTPBackend::serve does NOT seem to trigger a leak on 5.16.3 due to [2]: # XXX compatibility... ugh, can we stop supporting this? $git = PublicInbox::Git->new($git) unless ref($git); [1] https://perldoc.perl.org/perl5180delta.html [2] https://rt.perl.org/Public/Bug/Display.html?id=114340
2019-09-25ds: workaround a memory leak in Perl 5.16.xEric Wong1-3/+6
The perl-5.16.3-294.el7_6 RPM package on RHEL/CentOS 7 is affected by a memory leak in Perl when calling `ref' on blessed references. This resulted in a very slow leak that manifests more quickly with a nonstop "git fetch" loop. Use Scalar::Util::blessed to work around the issue. Tested overnight on a CentOS 7 VM. cf. https://rt.perl.org/Public/Bug/Display.html?id=114340
2019-09-24spawnpp: use absolute path for execEric Wong1-2/+4
We support "-env" to clear the environment with spawn(), which causes test failures but no runtime failures (since "-env" isn't used anywhere in our real code) Reported-and-tested-by: Alyssa Ross <hi@alyssa.is>
2019-09-22feed: remove unused $cmt->{-html_url} fieldEric Wong1-3/+0
It was never used, and will not be needed.
2019-09-22doc: update HACKING and TODO with a few itemsEric Wong2-6/+5
Inline::C seems alright, so we might use it more since it still allows end users to quickly make changes. Our performance on rotational disks is also terrible, and could be improved...
2019-09-20wwwatomstream: fix per-feed <id>Eric Wong1-12/+16
We were emitting the same "<id>mailto:name@domain</id>" tag for every feed (but not per-feed entry). This could cause feed readers to mistake the top (news.atom) feed for other feeds (search results, or per-thread feeds). This is technically a breaking change for people relying on per-thread or per-query feeds, but the only alternative is to remain broken for anybody trying to follow multiple feeds off the same inbox.
2019-09-18config: boolean handling matches git-config(1)Eric Wong2-4/+33
We need to handle arbitrary integers and case-insensitive variations of human words to match git-config(1) behavior, since that's what users would expect given we use config files parseable by git-config(1).
2019-09-18doc: add release notes directoryEric Wong5-0/+362
The v1.2.0 is a work-in-progress, while the others are copied out of our mail archives. Eventually, a NEWS file will be generated from these emails and distributed in the release tarball. There'll also be an Atom feed for the website reusing our feed generation code.
2019-09-17t/httpd-corner.t: don't fail lsof test if stdin is a pipe (try #2)Eric Wong1-3/+2
Actually do the redirect properly
2019-09-17t/httpd-corner.t: don't fail lsof test if stdin is a pipeEric Wong1-1/+5
We don't want the stdin from the test runner to accidentally cause this test to fail.
2019-09-17http: remove unnecessary deleteEric Wong1-3/+1
Only removing $http->{env} is needed to prevent circular references. $env->{'psgix.io'} does not need to be deleted since $env will no longer have any references to it when ->close returns.
2019-09-17http: drop unused `$env' variable after deleteEric Wong1-1/+1
And explain why we need to do that delete in a comment.
2019-09-17qspawn: improve variable naming and commentingEric Wong2-8/+10
Naming $start_cb consistently helps avoid confusing new readers, and some comments will help with understanding flow
2019-09-17qspawn: shorten lifetime of circular referencesEric Wong1-19/+9
All of these circular references are designed to clear themselves, but these will make actual errors from Devel::Cycle easier-to-spot. The circular reference in the limiter {run_queue} is not a real problem, but we can avoid storing the circular reference until we actually need to spawn the child, reducing the size of the Qspawn object while it's in the queue, slightly. We also do not need to have redundant checks to spawn new processes, we should only spawn new processes when they're ->start-ed or after waitpid reaps them.
2019-09-17qspawn: log errors for generic PSGI server usersEric Wong1-4/+3
Generic PSGI servers have $env->{'psgi.errors'}, too, so ensure they can log errors.
2019-09-17qspawn: remove return value from ->finishEric Wong2-7/+11
We don't use the return value in real code since we do waitpid asynchronously, now. So simplify our runtime code at the cost of making our test slighly more complex.
2019-09-15doc: update config manpage for "publicinbox.grokmanifest"Eric Wong1-3/+32
It's a bit of an esoteric option, but maybe somebody out there can find it useful.
2019-09-15qspawn: shorten lifetime of environ and opts argsEric Wong1-1/+6
We don't need to hold onto the subprocess environ and redirects/options for popen_rd after spawning the child process. I do not expect this to fix problem of leaking unlinked regular file descriptors (which I still can't reproduce), and it definitely does not fix the problem of leaking pipe descriptors (which I also can't reproduce). This will save an FD sooner on non-public-inbox-httpd servers which give a non-FD $env->{'psgi.input'}, however Regardless, it's good to free up memory resources in our own process ASAP we're done using them.
2019-09-15qspawn: clarify and improve error handlingEric Wong1-11/+27
EINTR should not happen when using non-blocking sockets like we do in our daemons, but maybe some OSes allow it to happen and edge-triggered notifications won't notify us again. So always retry immediately on EINTR without relying on kqueue or epoll to notify us, and log any other unrecoverable errors which may happen while we're at it.
2019-09-15t/httpd-corner: use which() sub for detecting curl(1)Eric Wong1-9/+2
We already import `which' for lsof(8), so we might as well use it to detect curl(1), too.
2019-09-14doc: update nntpd with NNTPS and STARTTLS examplesEric Wong5-7/+59
NNTPS and STARTTLS seems to be working for several months without incident on news.public-inbox.org, so consider it a success and maybe others can try using it. HTTPS technically works, too, but isn't documented at the moment since I can't recommend production deployments without varnish protecting it.
2019-09-14t/httpd-corner: check for leaking FDs and pipesEric Wong1-0/+8
-W0 (no workers) should not create any pipes on its own, and we shouldn't have any deleted FDs if no clients are connected. This can find if leaks which may be triggered by PublicInbox::HTTP (and not Qspawn or GitHTTPBackend).
2019-09-14qspawn: remove unused WNOHANG importEric Wong1-1/+0
We rely on DS to do waitpid with WNOHANG, now, and the non-DS code path won't use WNOHANG.
2019-09-14httpd/async: improve naming and commentsEric Wong2-12/+29
Rename the {cleanup} field to {end}, since it's similar to END {} and is consistent with the variable in Qspawn.pm And document how certain subs get called, since we have many subs named "new" and "close".
2019-09-14githttpbackend: use REMOTE_ADDR for deleted identifierEric Wong1-1/+1
REMOTE_HOST is not set by us (it is the reverse DNS name) of REMOTE_ADDR, and there's few better ways to kill HTTP server performance than to use standard name resolution APIs like getnameinfo(3).
2019-09-14tmpfile: support O_APPEND and use it in DS::tmpioEric Wong2-14/+8
Might as well share some code for temporary file creation
2019-09-14tmpfile: give temporary files meaningful namesEric Wong7-8/+55
Although we always unlink temporary files, give them a meaningful name so that we can we can still make sense of the pre-unlink name when using lsof(8) or similar tools on Linux.
2019-09-14qspawn: simplify by using PerlIO::scalarEric Wong1-20/+4
I didn't know PerlIO::scalar existed until a few months ago, but it's been distributed with Perl since 5.8 and doesn't seem to be split out into it's own package on any distro.
2019-09-14admin: warn and ignore inaccessible inboxesEric Wong1-1/+8
For whatever reason, inbox directories can go missing temporarily or permanently. Tell the admin about them and continue on our way.
2019-09-12solvergit: don't drop update-index stdin with qspawnEric Wong1-1/+1
It's possible for Qspawn callers to be deferred, in which case we must ensure we don't cause the temporary file used for stdin to become unref-ed and closed. This can be a problem when we exceed the default Qspawn limiter of 32 concurrent processes for "git update-index".
2019-09-09doc daemon: note the --listen directive is not always requiredEric Wong1-0/+3
Users of socket activation don't need it, and hopefully other init systems support it, too.
2019-09-09doc edit: move =for comment after itemEric Wong1-2/+2
Quiets down pod2man complaining
2019-09-09run update-copyrights from gnulib for 2019Eric Wong151-151/+151
2019-09-09doc config: document indexlevel directiveEric Wong1-0/+17
It was never documented, before.
2019-09-09tests: add tcp_connect() helperEric Wong9-38/+25
IO::Socket::INET->new is rather verbose with the options hash, extract it into a standalone sub
2019-09-08nntp: regexp always consumes rbuf if "\n" existsEric Wong1-1/+1
We don't want to get hung into a state where we see "\n" via index(), yet cannot consume rbuf in the while loop. So tweak the regexp to ensure we always consume rbuf. I suspect this is what causes occasional 100% CPU usage of -nntpd, but reproducing it's been difficult..
2019-09-08nntp: fix redundant CRLF from "LISTGROUP GROUP RANGE"Eric Wong1-1/+1
Since Net::NNTP::listgroup doesn't support the range parameter, I had to test this manually and noticed extra CRLF were emitted.
2019-07-14nntpdeflate: reduce overhead of idle clientsEric Wong1-5/+8
We don't need to keep an empty buffer around in the common case when a client is sending us completely inflatable requests and we're able to read them in one go. This only seems to save about 2M with 10K NNTPS clients using COMPRESS, so it's not a huge win, but better than nothing.
2019-07-13TODO: remove done items, add some moreEric Wong1-3/+4
It never ends...
2019-07-13nntp: support optional [range] arg in LISTGROUPEric Wong3-14/+28
RFC3977 LISTGROUP allows a [range] arg after [group], and supporting it allows NNTP support in neomutt to work again. Tested with NeoMutt 20170113 (1.7.2) on Debian stretch (oldstable)
2019-07-13nntp: fix LIST OVERVIEW.FMT ordering and formatEric Wong2-2/+9
RFC3977 8.4.2 mandates the order of non-standard headers to be after the first seven standard headers/metadata; so "Xref:" must appear after "Lines:"|":lines". Additionally, non-required header names must be followed by ":full". Cc: Jonathan Corbet <corbet@lwn.net> Reported-by: Urs Jan├čen <E1hmKBw-0008Bq-8t@akw>
2019-07-13nntpdeflate: stop relying on SUPER for ->do_readEric Wong1-2/+2
We won't need further layering after enabling compression. So be explicit about which sub we're calling when we hit ->do_read from NNTP and eliminate the need for the comment.
2019-07-12nntp: clear local timer on idle client expiryEric Wong1-4/+2
We need to ensure further timers can be registered if there's currently no idle clients.
2019-07-10http|nntp: avoid recursion inside ->writeEric Wong2-33/+34
In HTTP.pm, we can use the same technique NNTP.pm uses with long_response with the $long_cb callback and avoid storing $pull in the per-client structure at all. We can also reuse the same logic to push the callback into wbuf from NNTP. This does NOT introduce a new circular reference, but documents it more clearly.
2019-07-10solver: remove redundant spawn importsEric Wong1-1/+0
We're using Qspawn, now
2019-07-08http|nntp: "use PublicInbox::DS" instead of ->importEric Wong2-2/+2
Relying on "use" to import during BEGIN means we get to take advantage of prototype checking of function args during the rest of the compilation phase.
2019-07-08httpd: (cleanup) use reference instead of *globEric Wong1-4/+1
*glob notation isn't always necessary, and there's no need to disable 'once' warnings, this way.
2019-07-08daemon: use POSIX and WNOHANG more idiomaticallyEric Wong1-2/+2
No point in uglifying our code since we need the POSIX module in many places, anyways.
2019-07-08Merge remote-tracking branch 'origin/nntp-compress'Eric Wong9-13/+404
* origin/nntp-compress: nntp: improve error reporting for COMPRESS nntp: reduce memory overhead of zlib nntp: support COMPRESS DEFLATE per RFC 8054 nntp: move LINE_MAX constant to the top nntp: use msg_more as a method
2019-07-08ds: use WNOHANG with waitpid if inside event loopEric Wong4-24/+111
While we're usually not stuck waiting on waitpid after seeing a pipe EOF or even triggering SIGPIPE in the process (e.g. git-http-backend) we're reading from, it MAY happen and we should be careful to never hang the daemon process on waitpid calls. v2: use "eq" for string comparison against 'DEFAULT'