user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 11/13] qspawn: wire up RLIMIT_* handling to limiters
  2019-03-12  4:00  7% [PATCH 00/13] support parsing cgitrc and spawning cgit Eric Wong
@ 2019-03-12  4:00  6% ` Eric Wong
  0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-03-12  4:00 UTC (permalink / raw)
  To: meta

This allows users to configure RLIMIT_{CORE,CPU,DATA} using
our "limiter" config directive when spawning external processes.
---
 Documentation/public-inbox-config.pod | 17 +++++++++++
 lib/PublicInbox/Config.pm             |  4 ++-
 lib/PublicInbox/Qspawn.pm             | 41 +++++++++++++++++++++++++--
 lib/PublicInbox/Spawn.pm              |  3 +-
 4 files changed, 61 insertions(+), 4 deletions(-)

diff --git a/Documentation/public-inbox-config.pod b/Documentation/public-inbox-config.pod
index 9647e4a..dae6998 100644
--- a/Documentation/public-inbox-config.pod
+++ b/Documentation/public-inbox-config.pod
@@ -228,12 +228,29 @@ large inboxes, it makes sense to put large inboxes on a named
 limiter with a low max value; while smaller inboxes can use
 the default limiter.
 
+C<RLIMIT_*> keys may be set to enforce resource limits for
+a particular limiter.
+
 =over 8
 
 =item publicinboxlimiter.<name>.max
 
 The maximum number of parallel processes for the given limiter.
 
+=item publicinboxlimiter.<name>.rlimitCore
+
+=item publicinboxlimiter.<name>.rlimitCPU
+
+=item publicinboxlimiter.<name>.rlimitData
+
+The maximum core size, CPU time, or data size processes run with the
+given limiter will use.  This may be comma-separated to distinguish
+soft and hard limits.  The word "INFINITY" is accepted as the
+RLIM_INFINITY constant (if supported by your OS).
+
+See L<setrlimit(2)> for more info on the behavior of RLIMIT_CORE,
+RLIMIT_CPU, and RLIMIT_DATA for you operating system.
+
 =back
 
 =head3 EXAMPLE WITH NAMED LIMITERS
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 15664c6..0f7485f 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -133,7 +133,9 @@ sub limiter {
 	$self->{-limiters}->{$name} ||= do {
 		require PublicInbox::Qspawn;
 		my $max = $self->{"publicinboxlimiter.$name.max"};
-		PublicInbox::Qspawn::Limiter->new($max);
+		my $limiter = PublicInbox::Qspawn::Limiter->new($max);
+		$limiter->setup_rlimit($name, $self);
+		$limiter;
 	};
 }
 
diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index 509a441..79cdae7 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -43,10 +43,18 @@ sub new ($$$;) {
 sub _do_spawn {
 	my ($self, $cb) = @_;
 	my $err;
+	my ($cmd, $env, $opts) = @{$self->{args}};
+	my %opts = %{$opts || {}};
+	my $limiter = $self->{limiter};
+	foreach my $k (PublicInbox::Spawn::RLIMITS()) {
+		if (defined(my $rlimit = $limiter->{$k})) {
+			$opts{$k} = $rlimit;
+		}
+	}
 
-	($self->{rpipe}, $self->{pid}) = popen_rd(@{$self->{args}});
+	($self->{rpipe}, $self->{pid}) = popen_rd($cmd, $env, \%opts);
 	if (defined $self->{pid}) {
-		$self->{limiter}->{running}++;
+		$limiter->{running}++;
 	} else {
 		$self->{err} = $!;
 	}
@@ -251,9 +259,38 @@ sub new {
 		max => $max || 32,
 		running => 0,
 		run_queue => [],
+		# RLIMIT_CPU => undef,
+		# RLIMIT_DATA => undef,
+		# RLIMIT_CORE => undef,
 	}, $class;
 }
 
+sub setup_rlimit {
+	my ($self, $name, $config) = @_;
+	foreach my $rlim (PublicInbox::Spawn::RLIMITS()) {
+		my $k = lc($rlim);
+		$k =~ tr/_//d;
+		$k = "publicinboxlimiter.$name.$k";
+		defined(my $v = $config->{$k}) or next;
+		my @rlimit = split(/\s*,\s*/, $v);
+		if (scalar(@rlimit) == 1) {
+			push @rlimit, $rlimit[0];
+		} elsif (scalar(@rlimit) != 2) {
+			warn "could not parse $k: $v\n";
+		}
+		eval { require BSD::Resource };
+		if ($@) {
+			warn "BSD::Resource missing for $rlim";
+			next;
+		}
+		foreach my $i (0..$#rlimit) {
+			next if $rlimit[$i] ne 'INFINITY';
+			$rlimit[$i] = BSD::Resource::RLIM_INFINITY();
+		}
+		$self->{$rlim} = \@rlimit;
+	}
+}
+
 # captures everything into a buffer and executes a callback when done
 package PublicInbox::Qspawn::Qx;
 use strict;
diff --git a/lib/PublicInbox/Spawn.pm b/lib/PublicInbox/Spawn.pm
index 202cfca..fd98160 100644
--- a/lib/PublicInbox/Spawn.pm
+++ b/lib/PublicInbox/Spawn.pm
@@ -18,6 +18,7 @@ use Symbol qw(gensym);
 use IO::Handle;
 use PublicInbox::ProcessPipe;
 our @EXPORT_OK = qw/which spawn popen_rd/;
+sub RLIMITS () { qw(RLIMIT_CPU RLIMIT_CORE RLIMIT_DATA) }
 
 my $vfork_spawn = <<'VFORK_SPAWN';
 #include <sys/types.h>
@@ -202,7 +203,7 @@ sub spawn ($;$$) {
 	my $err = $opts->{2} || 2;
 	my $rlim = [];
 
-	foreach my $l (qw(RLIMIT_CPU RLIMIT_CORE RLIMIT_DATA)) {
+	foreach my $l (RLIMITS()) {
 		defined(my $v = $opts->{$l}) or next;
 		my ($soft, $hard);
 		if (ref($v)) {
-- 
EW


^ permalink raw reply	[relevance 6%]

* [PATCH 00/13] support parsing cgitrc and spawning cgit
@ 2019-03-12  4:00  7% Eric Wong
  2019-03-12  4:00  6% ` [PATCH 11/13] qspawn: wire up RLIMIT_* handling to limiters Eric Wong
  0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2019-03-12  4:00 UTC (permalink / raw)
  To: meta

Parsing an existing cgitrc reduces the amount of work some for
admins (who already maintain cgit instances) by allowing them to
skip the tedious setup of of setting up [coderepo "..."]
sections.

We currently do not support "scan-path", "project-list" or
macros in cgitrc processing, yet.  So it's expected that
"repo.url" and "repo.path" be configured in the cgitrc for each
code repo.  (I started using cgit in 2008 before cgit supported
path scanning, and never updated my setup :x)

Since cgit does not serve smart HTTP fetch/clone, we can
intercept requests for those and route such requests to
git-http-backend(1) using the same mechanisms we use for
serving inboxes.

Eric Wong (13):
  git: add "commit_title" method
  viewvcs: preliminary support for showing non-blobs
  viewvcs: match 8000-byte lookup for git
  spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE
  support publicinbox.cgitrc directive
  githttpbackend: move more psgi.input handling into subroutine
  githttpbackend: check for other errors and relax CRLF check
  spawn: support absolute paths
  cgit: support running cgit as a standalone CGI
  www: wire up cgit as a 404 handler if cgitrc is configured
  qspawn: wire up RLIMIT_* handling to limiters
  cgit: use a dedicated named limiter
  spawn: require soft and hard entries in RLIMIT_* handling

 Documentation/public-inbox-config.pod | 44 +++++++++++++-
 MANIFEST                              |  2 +
 examples/cgit.psgi                    | 29 +++++++++
 lib/PublicInbox/Cgit.pm               | 88 +++++++++++++++++++++++++++
 lib/PublicInbox/Config.pm             | 61 +++++++++++++++++--
 lib/PublicInbox/Git.pm                | 20 +++++-
 lib/PublicInbox/GitHTTPBackend.pm     | 29 ++++-----
 lib/PublicInbox/Qspawn.pm             | 41 ++++++++++++-
 lib/PublicInbox/SolverGit.pm          | 14 +++--
 lib/PublicInbox/Spawn.pm              | 40 ++++++++++--
 lib/PublicInbox/SpawnPP.pm            |  9 ++-
 lib/PublicInbox/ViewVCS.pm            | 34 ++++++++++-
 lib/PublicInbox/WWW.pm                | 20 +++++-
 t/solver_git.t                        |  9 ++-
 t/spawn.t                             | 19 ++++++
 15 files changed, 416 insertions(+), 43 deletions(-)
 create mode 100644 examples/cgit.psgi
 create mode 100644 lib/PublicInbox/Cgit.pm

-- 
EW


^ permalink raw reply	[relevance 7%]

Results 1-2 of 2 | reverse | sort options + mbox downloads above
-- links below jump to the message on this page --
2019-03-12  4:00  7% [PATCH 00/13] support parsing cgitrc and spawning cgit Eric Wong
2019-03-12  4:00  6% ` [PATCH 11/13] qspawn: wire up RLIMIT_* handling to limiters Eric Wong

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).