* [PATCH 11/13] qspawn: wire up RLIMIT_* handling to limiters
2019-03-12 4:00 7% [PATCH 00/13] support parsing cgitrc and spawning cgit Eric Wong
@ 2019-03-12 4:00 5% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-03-12 4:00 UTC (permalink / raw)
To: meta
This allows users to configure RLIMIT_{CORE,CPU,DATA} using
our "limiter" config directive when spawning external processes.
---
Documentation/public-inbox-config.pod | 17 +++++++++++
lib/PublicInbox/Config.pm | 4 ++-
lib/PublicInbox/Qspawn.pm | 41 +++++++++++++++++++++++++--
lib/PublicInbox/Spawn.pm | 3 +-
4 files changed, 61 insertions(+), 4 deletions(-)
diff --git a/Documentation/public-inbox-config.pod b/Documentation/public-inbox-config.pod
index 9647e4a..dae6998 100644
--- a/Documentation/public-inbox-config.pod
+++ b/Documentation/public-inbox-config.pod
@@ -228,12 +228,29 @@ large inboxes, it makes sense to put large inboxes on a named
limiter with a low max value; while smaller inboxes can use
the default limiter.
+C<RLIMIT_*> keys may be set to enforce resource limits for
+a particular limiter.
+
=over 8
=item publicinboxlimiter.<name>.max
The maximum number of parallel processes for the given limiter.
+=item publicinboxlimiter.<name>.rlimitCore
+
+=item publicinboxlimiter.<name>.rlimitCPU
+
+=item publicinboxlimiter.<name>.rlimitData
+
+The maximum core size, CPU time, or data size processes run with the
+given limiter will use. This may be comma-separated to distinguish
+soft and hard limits. The word "INFINITY" is accepted as the
+RLIM_INFINITY constant (if supported by your OS).
+
+See L<setrlimit(2)> for more info on the behavior of RLIMIT_CORE,
+RLIMIT_CPU, and RLIMIT_DATA for you operating system.
+
=back
=head3 EXAMPLE WITH NAMED LIMITERS
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 15664c6..0f7485f 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -133,7 +133,9 @@ sub limiter {
$self->{-limiters}->{$name} ||= do {
require PublicInbox::Qspawn;
my $max = $self->{"publicinboxlimiter.$name.max"};
- PublicInbox::Qspawn::Limiter->new($max);
+ my $limiter = PublicInbox::Qspawn::Limiter->new($max);
+ $limiter->setup_rlimit($name, $self);
+ $limiter;
};
}
diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index 509a441..79cdae7 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -43,10 +43,18 @@ sub new ($$$;) {
sub _do_spawn {
my ($self, $cb) = @_;
my $err;
+ my ($cmd, $env, $opts) = @{$self->{args}};
+ my %opts = %{$opts || {}};
+ my $limiter = $self->{limiter};
+ foreach my $k (PublicInbox::Spawn::RLIMITS()) {
+ if (defined(my $rlimit = $limiter->{$k})) {
+ $opts{$k} = $rlimit;
+ }
+ }
- ($self->{rpipe}, $self->{pid}) = popen_rd(@{$self->{args}});
+ ($self->{rpipe}, $self->{pid}) = popen_rd($cmd, $env, \%opts);
if (defined $self->{pid}) {
- $self->{limiter}->{running}++;
+ $limiter->{running}++;
} else {
$self->{err} = $!;
}
@@ -251,9 +259,38 @@ sub new {
max => $max || 32,
running => 0,
run_queue => [],
+ # RLIMIT_CPU => undef,
+ # RLIMIT_DATA => undef,
+ # RLIMIT_CORE => undef,
}, $class;
}
+sub setup_rlimit {
+ my ($self, $name, $config) = @_;
+ foreach my $rlim (PublicInbox::Spawn::RLIMITS()) {
+ my $k = lc($rlim);
+ $k =~ tr/_//d;
+ $k = "publicinboxlimiter.$name.$k";
+ defined(my $v = $config->{$k}) or next;
+ my @rlimit = split(/\s*,\s*/, $v);
+ if (scalar(@rlimit) == 1) {
+ push @rlimit, $rlimit[0];
+ } elsif (scalar(@rlimit) != 2) {
+ warn "could not parse $k: $v\n";
+ }
+ eval { require BSD::Resource };
+ if ($@) {
+ warn "BSD::Resource missing for $rlim";
+ next;
+ }
+ foreach my $i (0..$#rlimit) {
+ next if $rlimit[$i] ne 'INFINITY';
+ $rlimit[$i] = BSD::Resource::RLIM_INFINITY();
+ }
+ $self->{$rlim} = \@rlimit;
+ }
+}
+
# captures everything into a buffer and executes a callback when done
package PublicInbox::Qspawn::Qx;
use strict;
diff --git a/lib/PublicInbox/Spawn.pm b/lib/PublicInbox/Spawn.pm
index 202cfca..fd98160 100644
--- a/lib/PublicInbox/Spawn.pm
+++ b/lib/PublicInbox/Spawn.pm
@@ -18,6 +18,7 @@ use Symbol qw(gensym);
use IO::Handle;
use PublicInbox::ProcessPipe;
our @EXPORT_OK = qw/which spawn popen_rd/;
+sub RLIMITS () { qw(RLIMIT_CPU RLIMIT_CORE RLIMIT_DATA) }
my $vfork_spawn = <<'VFORK_SPAWN';
#include <sys/types.h>
@@ -202,7 +203,7 @@ sub spawn ($;$$) {
my $err = $opts->{2} || 2;
my $rlim = [];
- foreach my $l (qw(RLIMIT_CPU RLIMIT_CORE RLIMIT_DATA)) {
+ foreach my $l (RLIMITS()) {
defined(my $v = $opts->{$l}) or next;
my ($soft, $hard);
if (ref($v)) {
--
EW
^ permalink raw reply related [relevance 5%]
* [PATCH 00/13] support parsing cgitrc and spawning cgit
@ 2019-03-12 4:00 7% Eric Wong
2019-03-12 4:00 5% ` [PATCH 11/13] qspawn: wire up RLIMIT_* handling to limiters Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2019-03-12 4:00 UTC (permalink / raw)
To: meta
Parsing an existing cgitrc reduces the amount of work some for
admins (who already maintain cgit instances) by allowing them to
skip the tedious setup of of setting up [coderepo "..."]
sections.
We currently do not support "scan-path", "project-list" or
macros in cgitrc processing, yet. So it's expected that
"repo.url" and "repo.path" be configured in the cgitrc for each
code repo. (I started using cgit in 2008 before cgit supported
path scanning, and never updated my setup :x)
Since cgit does not serve smart HTTP fetch/clone, we can
intercept requests for those and route such requests to
git-http-backend(1) using the same mechanisms we use for
serving inboxes.
Eric Wong (13):
git: add "commit_title" method
viewvcs: preliminary support for showing non-blobs
viewvcs: match 8000-byte lookup for git
spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE
support publicinbox.cgitrc directive
githttpbackend: move more psgi.input handling into subroutine
githttpbackend: check for other errors and relax CRLF check
spawn: support absolute paths
cgit: support running cgit as a standalone CGI
www: wire up cgit as a 404 handler if cgitrc is configured
qspawn: wire up RLIMIT_* handling to limiters
cgit: use a dedicated named limiter
spawn: require soft and hard entries in RLIMIT_* handling
Documentation/public-inbox-config.pod | 44 +++++++++++++-
MANIFEST | 2 +
examples/cgit.psgi | 29 +++++++++
lib/PublicInbox/Cgit.pm | 88 +++++++++++++++++++++++++++
lib/PublicInbox/Config.pm | 61 +++++++++++++++++--
lib/PublicInbox/Git.pm | 20 +++++-
lib/PublicInbox/GitHTTPBackend.pm | 29 ++++-----
lib/PublicInbox/Qspawn.pm | 41 ++++++++++++-
lib/PublicInbox/SolverGit.pm | 14 +++--
lib/PublicInbox/Spawn.pm | 40 ++++++++++--
lib/PublicInbox/SpawnPP.pm | 9 ++-
lib/PublicInbox/ViewVCS.pm | 34 ++++++++++-
lib/PublicInbox/WWW.pm | 20 +++++-
t/solver_git.t | 9 ++-
t/spawn.t | 19 ++++++
15 files changed, 416 insertions(+), 43 deletions(-)
create mode 100644 examples/cgit.psgi
create mode 100644 lib/PublicInbox/Cgit.pm
--
EW
^ permalink raw reply [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-03-12 4:00 7% [PATCH 00/13] support parsing cgitrc and spawning cgit Eric Wong
2019-03-12 4:00 5% ` [PATCH 11/13] qspawn: wire up RLIMIT_* handling to limiters Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).