* [PATCH 21/21] doc: note optional BSD::Resource use
@ 2021-02-01 8:28 6% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2021-02-01 8:28 UTC (permalink / raw)
To: meta
We've actually been capable of using this since 2019(*) in our
spawn code for PSGI limiters. And it's been used since 2016 in
our tests. It's a dependency of SpamAssassin, and Danga::Socket
used it, too.
(*) commit 721368cd04bfbd03c0d9173fff633ae34f16409a
("spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE")
---
Documentation/public-inbox-config.pod | 2 +-
INSTALL | 6 ++++++
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/Documentation/public-inbox-config.pod b/Documentation/public-inbox-config.pod
index 4a97fe3b..904af804 100644
--- a/Documentation/public-inbox-config.pod
+++ b/Documentation/public-inbox-config.pod
@@ -335,7 +335,7 @@ limiter with a low max value; while smaller inboxes can use
the default limiter.
C<RLIMIT_*> keys may be set to enforce resource limits for
-a particular limiter.
+a particular limiter (L<BSD::Resource(3pm)> is required).
Default named-limiters are prefixed with "-". Currently,
the "-cgit" named limiter is reserved for instances spawning
diff --git a/INSTALL b/INSTALL
index de871b1a..36d89814 100644
--- a/INSTALL
+++ b/INSTALL
@@ -86,6 +86,7 @@ Numerous optional modules are likely to be useful as well:
- Inline::C deb: libinline-c-perl
pkg: pkg-Inline-C
+ rpm: perl-Inline (or perl-Inline-C)
(speeds up process spawning on Linux,
see public-inbox-daemon(8))
@@ -100,6 +101,11 @@ Numerous optional modules are likely to be useful as well:
rpm: perl-ParseRecDescent
(optional, for public-inbox-imapd(1))
+- BSD::Resource deb: libbsd-resource-perl
+ pkg: p5-BSD-Resource
+ rpm: perl-BSD-Resource
+ (optional, for PSGI limiters
+ see public-inbox-config(5))
- Plack::Middleware::ReverseProxy deb: libplack-middleware-reverseproxy-perl
pkg: p5-Plack-Middleware-ReverseProxy
^ permalink raw reply related [relevance 6%]
* [PATCH 00/13] support parsing cgitrc and spawning cgit
@ 2019-03-12 4:00 7% Eric Wong
2019-03-12 4:00 6% ` [PATCH 04/13] spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2019-03-12 4:00 UTC (permalink / raw)
To: meta
Parsing an existing cgitrc reduces the amount of work some for
admins (who already maintain cgit instances) by allowing them to
skip the tedious setup of of setting up [coderepo "..."]
sections.
We currently do not support "scan-path", "project-list" or
macros in cgitrc processing, yet. So it's expected that
"repo.url" and "repo.path" be configured in the cgitrc for each
code repo. (I started using cgit in 2008 before cgit supported
path scanning, and never updated my setup :x)
Since cgit does not serve smart HTTP fetch/clone, we can
intercept requests for those and route such requests to
git-http-backend(1) using the same mechanisms we use for
serving inboxes.
Eric Wong (13):
git: add "commit_title" method
viewvcs: preliminary support for showing non-blobs
viewvcs: match 8000-byte lookup for git
spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE
support publicinbox.cgitrc directive
githttpbackend: move more psgi.input handling into subroutine
githttpbackend: check for other errors and relax CRLF check
spawn: support absolute paths
cgit: support running cgit as a standalone CGI
www: wire up cgit as a 404 handler if cgitrc is configured
qspawn: wire up RLIMIT_* handling to limiters
cgit: use a dedicated named limiter
spawn: require soft and hard entries in RLIMIT_* handling
Documentation/public-inbox-config.pod | 44 +++++++++++++-
MANIFEST | 2 +
examples/cgit.psgi | 29 +++++++++
lib/PublicInbox/Cgit.pm | 88 +++++++++++++++++++++++++++
lib/PublicInbox/Config.pm | 61 +++++++++++++++++--
lib/PublicInbox/Git.pm | 20 +++++-
lib/PublicInbox/GitHTTPBackend.pm | 29 ++++-----
lib/PublicInbox/Qspawn.pm | 41 ++++++++++++-
lib/PublicInbox/SolverGit.pm | 14 +++--
lib/PublicInbox/Spawn.pm | 40 ++++++++++--
lib/PublicInbox/SpawnPP.pm | 9 ++-
lib/PublicInbox/ViewVCS.pm | 34 ++++++++++-
lib/PublicInbox/WWW.pm | 20 +++++-
t/solver_git.t | 9 ++-
t/spawn.t | 19 ++++++
15 files changed, 416 insertions(+), 43 deletions(-)
create mode 100644 examples/cgit.psgi
create mode 100644 lib/PublicInbox/Cgit.pm
--
EW
^ permalink raw reply [relevance 7%]
* [PATCH 04/13] spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE
2019-03-12 4:00 7% [PATCH 00/13] support parsing cgitrc and spawning cgit Eric Wong
@ 2019-03-12 4:00 6% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2019-03-12 4:00 UTC (permalink / raw)
To: meta
We'll be spawning cgit and git-diff, which can take gigantic
amounts of CPU time and/or heap given the right (ermm... wrong)
input. Limit the damage that large/expensive diffs can cause.
---
lib/PublicInbox/Spawn.pm | 43 ++++++++++++++++++++++++++++++++++----
lib/PublicInbox/SpawnPP.pm | 9 ++++++--
t/spawn.t | 18 ++++++++++++++++
3 files changed, 64 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/Spawn.pm b/lib/PublicInbox/Spawn.pm
index 91a3c12..8ea255a 100644
--- a/lib/PublicInbox/Spawn.pm
+++ b/lib/PublicInbox/Spawn.pm
@@ -22,6 +22,8 @@ our @EXPORT_OK = qw/which spawn popen_rd/;
my $vfork_spawn = <<'VFORK_SPAWN';
#include <sys/types.h>
#include <sys/uio.h>
+#include <sys/time.h>
+#include <sys/resource.h>
#include <unistd.h>
#include <alloca.h>
#include <signal.h>
@@ -74,11 +76,12 @@ static void xerr(const char *msg)
* whatever we'll need in the future.
* Be sure to update PublicInbox::SpawnPP if this changes
*/
-int public_inbox_fork_exec(int in, int out, int err,
- SV *file, SV *cmdref, SV *envref)
+int pi_fork_exec(int in, int out, int err,
+ SV *file, SV *cmdref, SV *envref, SV *rlimref)
{
AV *cmd = (AV *)SvRV(cmdref);
AV *env = (AV *)SvRV(envref);
+ AV *rlim = (AV *)SvRV(rlimref);
const char *filename = SvPV_nolen(file);
pid_t pid;
char **argv, **envp;
@@ -99,12 +102,27 @@ int public_inbox_fork_exec(int in, int out, int err,
pid = vfork();
if (pid == 0) {
int sig;
+ I32 i, max;
REDIR(in, 0);
REDIR(out, 1);
REDIR(err, 2);
for (sig = 1; sig < NSIG; sig++)
signal(sig, SIG_DFL); /* ignore errors on signals */
+
+ max = av_len(rlim);
+ for (i = 0; i < max; i += 3) {
+ struct rlimit rl;
+ SV **res = av_fetch(rlim, i, 0);
+ SV **soft = av_fetch(rlim, i + 1, 0);
+ SV **hard = av_fetch(rlim, i + 2, 0);
+
+ rl.rlim_cur = SvIV(*soft);
+ rl.rlim_max = SvIV(*hard);
+ if (setrlimit(SvIV(*res), &rl) < 0)
+ xerr("sertlimit");
+ }
+
/*
* don't bother unblocking, we don't want signals
* to the group taking out a subprocess
@@ -145,7 +163,7 @@ if (defined $vfork_spawn) {
unless (defined $vfork_spawn) {
require PublicInbox::SpawnPP;
no warnings 'once';
- *public_inbox_fork_exec = *PublicInbox::SpawnPP::public_inbox_fork_exec
+ *pi_fork_exec = *PublicInbox::SpawnPP::pi_fork_exec
}
# n.b. we never use absolute paths with this
@@ -182,7 +200,24 @@ sub spawn ($;$$) {
my $in = $opts->{0} || 0;
my $out = $opts->{1} || 1;
my $err = $opts->{2} || 2;
- my $pid = public_inbox_fork_exec($in, $out, $err, $f, $cmd, \@env);
+ my $rlim = [];
+
+ foreach my $l (qw(RLIMIT_CPU RLIMIT_CORE RLIMIT_DATA)) {
+ defined(my $v = $opts->{$l}) or next;
+ my ($soft, $hard);
+ if (ref($v)) {
+ ($soft, $hard) = @$v;
+ } else {
+ $soft = $hard = $v;
+ }
+ my $r = eval "require BSD::Resource; BSD::Resource::$l();";
+ unless (defined $r) {
+ warn "$l undefined by BSD::Resource: $@\n";
+ next;
+ }
+ push @$rlim, $r, $soft, $hard;
+ }
+ my $pid = pi_fork_exec($in, $out, $err, $f, $cmd, \@env, $rlim);
$pid < 0 ? undef : $pid;
}
diff --git a/lib/PublicInbox/SpawnPP.pm b/lib/PublicInbox/SpawnPP.pm
index 743db22..8692b76 100644
--- a/lib/PublicInbox/SpawnPP.pm
+++ b/lib/PublicInbox/SpawnPP.pm
@@ -9,8 +9,8 @@ use warnings;
use POSIX qw(dup2 :signal_h);
# Pure Perl implementation for folks that do not use Inline::C
-sub public_inbox_fork_exec ($$$$$$) {
- my ($in, $out, $err, $f, $cmd, $env) = @_;
+sub pi_fork_exec ($$$$$$) {
+ my ($in, $out, $err, $f, $cmd, $env, $rlim) = @_;
my $old = POSIX::SigSet->new();
my $set = POSIX::SigSet->new();
$set->fillset or die "fillset failed: $!";
@@ -22,6 +22,11 @@ sub public_inbox_fork_exec ($$$$$$) {
$pid = -1;
}
if ($pid == 0) {
+ while (@$rlim) {
+ my ($r, $soft, $hard) = splice(@$rlim, 0, 3);
+ BSD::Resource::setrlimit($r, $soft, $hard) or
+ warn "failed to set $r=[$soft,$hard]\n";
+ }
if ($in != 0) {
dup2($in, 0) or die "dup2 failed for stdin: $!";
}
diff --git a/t/spawn.t b/t/spawn.t
index db3f2dc..5abedc9 100644
--- a/t/spawn.t
+++ b/t/spawn.t
@@ -92,6 +92,24 @@ use PublicInbox::Spawn qw(which spawn popen_rd);
isnt($?, 0, '$? set properly: '.$?);
}
+SKIP: {
+ eval {
+ require BSD::Resource;
+ defined(BSD::Resource::RLIMIT_CPU())
+ } or skip 'BSD::Resource::RLIMIT_CPU missing', 3;
+ my ($r, $w);
+ pipe($r, $w) or die "pipe: $!";
+ my $cmd = ['sh', '-c', 'while true; do :; done'];
+ my $opt = { RLIMIT_CPU => [ 1, 1 ], RLIMIT_CORE => 0, 1 => fileno($w) };
+ my $pid = spawn($cmd, undef, $opt);
+ close $w or die "close(w): $!";
+ my $rset = '';
+ vec($rset, fileno($r), 1) = 1;
+ ok(select($rset, undef, undef, 5), 'child died before timeout');
+ is(waitpid($pid, 0), $pid, 'XCPU child process reaped');
+ isnt($?, 0, 'non-zero exit status');
+}
+
done_testing();
1;
--
EW
^ permalink raw reply related [relevance 6%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-03-12 4:00 7% [PATCH 00/13] support parsing cgitrc and spawning cgit Eric Wong
2019-03-12 4:00 6% ` [PATCH 04/13] spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE Eric Wong
2021-02-01 8:28 [PATCH 00/21] lei2mail worker segfault finally fixed Eric Wong
2021-02-01 8:28 6% ` [PATCH 21/21] doc: note optional BSD::Resource use Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).