* [PATCH 00/26] process management simplifications
@ 2023-10-25 0:29 7% Eric Wong
2023-10-25 0:29 4% ` [PATCH 01/26] limiter: split out from qspawn Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-10-25 0:29 UTC (permalink / raw)
To: meta
The convoluted HTTPD/Async code and is now gone and replaced
with the simpler InputPipe (originally developed for lei).
Fortunately, the s/psgi_return/psgi_yield/ change went more
smoothly than I expected.
cindex gets some simplifications, too; which will be helpful
since more work is required on that code. I didn't get a
chance to use the split out Limiter+Qspawn, though...
3/26 is a fairly large philosophy change to use temporary files
over pipes, but it's probably fine as it's simpler and more
straightforward data flow and only used for small outputs which
can fit in memory, so unlikely to hit disk.
9/26 is the major change which was enabled by a change made
4 years ago.
Eric Wong (26):
limiter: split out from qspawn
spawn: support synchronous run_qx
psgi_qx: use a temporary file rather than pipe
www_coderepo: capture uses a flattened list
qspawn: psgi_return allows list for callback args
qspawn: drop unused err arg for ->event_step
httpd/async: require IO arg
xt/check-run: call DS->Reset after all tests
qspawn: introduce new psgi_yield API
repo_atom: switch to psgi_yield
repo_snapshot: psgi_yield
viewvcs: psgi_yield
www_altid: switch to psgi_yield
cgit: switch to psgi_yield
www_coderepo: psgi_yield
drop psgi_return, httpd/async and GetlineBody
qspawn: use WwwStatic for fallbacks and error code
qspawn: simplify internal argument passing
cidx_log_p: don't bother with F_SETPIPE_SZ
cindex: avoid awaitpid for popen
cindex: use timer for inits
cindex: start using run_await to simplify code
cindex: use run_await to read extensions.objectFormat
cindex: drop XH_PID global
cindex: use run_await wrapper for git commands
cindex: use sysread for generating fingerprint
MANIFEST | 5 +-
lib/PublicInbox/Aspawn.pm | 34 +++
lib/PublicInbox/Cgit.pm | 2 +-
lib/PublicInbox/CidxLogP.pm | 3 +-
lib/PublicInbox/CodeSearchIdx.pm | 236 +++++++++----------
lib/PublicInbox/Config.pm | 4 +-
lib/PublicInbox/GetlineBody.pm | 46 ----
lib/PublicInbox/GetlineResponse.pm | 40 ++++
lib/PublicInbox/Git.pm | 6 +
lib/PublicInbox/GitHTTPBackend.pm | 17 +-
lib/PublicInbox/GzipFilter.pm | 5 +-
lib/PublicInbox/HTTP.pm | 8 +-
lib/PublicInbox/HTTPD.pm | 5 +-
lib/PublicInbox/HTTPD/Async.pm | 109 ---------
lib/PublicInbox/Inbox.pm | 4 +-
lib/PublicInbox/InputPipe.pm | 12 +-
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/Limiter.pm | 47 ++++
lib/PublicInbox/MailDiff.pm | 8 +-
lib/PublicInbox/Qspawn.pm | 349 +++++++++++------------------
lib/PublicInbox/RepoAtom.pm | 6 +-
lib/PublicInbox/RepoSnapshot.pm | 2 +-
lib/PublicInbox/SearchIdx.pm | 19 +-
lib/PublicInbox/SolverGit.pm | 10 +-
lib/PublicInbox/Spawn.pm | 69 ++++--
lib/PublicInbox/ViewVCS.pm | 7 +-
lib/PublicInbox/WwwAltId.pm | 6 +-
lib/PublicInbox/WwwCoderepo.pm | 12 +-
t/dir_idle.t | 1 -
t/fake_inotify.t | 2 -
t/httpd-corner.psgi | 14 +-
t/httpd-corner.t | 12 +-
t/qspawn.t | 3 +-
t/spawn.t | 13 +-
xt/check-run.t | 2 +
35 files changed, 515 insertions(+), 605 deletions(-)
create mode 100644 lib/PublicInbox/Aspawn.pm
delete mode 100644 lib/PublicInbox/GetlineBody.pm
create mode 100644 lib/PublicInbox/GetlineResponse.pm
delete mode 100644 lib/PublicInbox/HTTPD/Async.pm
create mode 100644 lib/PublicInbox/Limiter.pm
^ permalink raw reply [relevance 7%]
* [PATCH 01/26] limiter: split out from qspawn
2023-10-25 0:29 7% [PATCH 00/26] process management simplifications Eric Wong
@ 2023-10-25 0:29 4% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-10-25 0:29 UTC (permalink / raw)
To: meta
It's slightly better organized this way, especially since
`publicinboxLimiter' has its own user-facing config section
and knobs. I may use it in LeiMirror and CodeSearchIdx for
process management.
---
MANIFEST | 1 +
lib/PublicInbox/Config.pm | 4 +--
lib/PublicInbox/GitHTTPBackend.pm | 3 +-
lib/PublicInbox/Inbox.pm | 4 +--
lib/PublicInbox/Limiter.pm | 47 +++++++++++++++++++++++++++++++
lib/PublicInbox/MailDiff.pm | 1 +
lib/PublicInbox/Qspawn.pm | 47 ++-----------------------------
t/qspawn.t | 3 +-
8 files changed, 60 insertions(+), 50 deletions(-)
create mode 100644 lib/PublicInbox/Limiter.pm
diff --git a/MANIFEST b/MANIFEST
index 791d91a7..dcce801c 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -287,6 +287,7 @@ lib/PublicInbox/LeiUp.pm
lib/PublicInbox/LeiViewText.pm
lib/PublicInbox/LeiWatch.pm
lib/PublicInbox/LeiXSearch.pm
+lib/PublicInbox/Limiter.pm
lib/PublicInbox/Linkify.pm
lib/PublicInbox/Listener.pm
lib/PublicInbox/Lock.pm
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 15e0872e..d156b2d3 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -124,9 +124,9 @@ sub lookup_newsgroup {
sub limiter {
my ($self, $name) = @_;
$self->{-limiters}->{$name} //= do {
- require PublicInbox::Qspawn;
+ require PublicInbox::Limiter;
my $max = $self->{"publicinboxlimiter.$name.max"} || 1;
- my $limiter = PublicInbox::Qspawn::Limiter->new($max);
+ my $limiter = PublicInbox::Limiter->new($max);
$limiter->setup_rlimit($name, $self);
$limiter;
};
diff --git a/lib/PublicInbox/GitHTTPBackend.pm b/lib/PublicInbox/GitHTTPBackend.pm
index 74432429..d69f5f8b 100644
--- a/lib/PublicInbox/GitHTTPBackend.pm
+++ b/lib/PublicInbox/GitHTTPBackend.pm
@@ -9,13 +9,14 @@ use v5.10.1;
use Fcntl qw(:seek);
use IO::Handle; # ->flush
use HTTP::Date qw(time2str);
+use PublicInbox::Limiter;
use PublicInbox::Qspawn;
use PublicInbox::Tmpfile;
use PublicInbox::WwwStatic qw(r @NO_CACHE);
use Carp ();
# 32 is same as the git-daemon connection limit
-my $default_limiter = PublicInbox::Qspawn::Limiter->new(32);
+my $default_limiter = PublicInbox::Limiter->new(32);
# n.b. serving "description" and "cloneurl" should be innocuous enough to
# not cause problems. serving "config" might...
diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index 9afbb478..3dad7004 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -55,8 +55,8 @@ sub _set_limiter ($$$) {
my $val = $self->{$mkey} or return;
my $lim;
if ($val =~ /\A[0-9]+\z/) {
- require PublicInbox::Qspawn;
- $lim = PublicInbox::Qspawn::Limiter->new($val);
+ require PublicInbox::Limiter;
+ $lim = PublicInbox::Limiter->new($val);
} elsif ($val =~ /\A[a-z][a-z0-9]*\z/) {
$lim = $pi_cfg->limiter($val);
warn "$mkey limiter=$val not found\n" if !$lim;
diff --git a/lib/PublicInbox/Limiter.pm b/lib/PublicInbox/Limiter.pm
new file mode 100644
index 00000000..48a2b6a3
--- /dev/null
+++ b/lib/PublicInbox/Limiter.pm
@@ -0,0 +1,47 @@
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+package PublicInbox::Limiter;
+use v5.12;
+use PublicInbox::Spawn;
+
+sub new {
+ my ($class, $max) = @_;
+ bless {
+ # 32 is same as the git-daemon connection limit
+ max => $max || 32,
+ running => 0,
+ run_queue => [],
+ # RLIMIT_CPU => undef,
+ # RLIMIT_DATA => undef,
+ # RLIMIT_CORE => undef,
+ }, $class;
+}
+
+sub setup_rlimit {
+ my ($self, $name, $cfg) = @_;
+ for my $rlim (@PublicInbox::Spawn::RLIMITS) {
+ my $k = lc($rlim);
+ $k =~ tr/_//d;
+ $k = "publicinboxlimiter.$name.$k";
+ my $v = $cfg->{$k} // next;
+ my @rlimit = split(/\s*,\s*/, $v);
+ if (scalar(@rlimit) == 1) {
+ push @rlimit, $rlimit[0];
+ } elsif (scalar(@rlimit) != 2) {
+ warn "could not parse $k: $v\n";
+ }
+ eval { require BSD::Resource };
+ if ($@) {
+ warn "BSD::Resource missing for $rlim";
+ next;
+ }
+ for my $i (0..$#rlimit) {
+ next if $rlimit[$i] ne 'INFINITY';
+ $rlimit[$i] = BSD::Resource::RLIM_INFINITY();
+ }
+ $self->{$rlim} = \@rlimit;
+ }
+}
+
+1;
diff --git a/lib/PublicInbox/MailDiff.pm b/lib/PublicInbox/MailDiff.pm
index 994c7851..c3ce9365 100644
--- a/lib/PublicInbox/MailDiff.pm
+++ b/lib/PublicInbox/MailDiff.pm
@@ -8,6 +8,7 @@ use PublicInbox::MsgIter qw(msg_part_text);
use PublicInbox::ViewDiff qw(flush_diff);
use PublicInbox::GitAsyncCat;
use PublicInbox::ContentDigestDbg;
+use PublicInbox::Qspawn;
sub write_part { # Eml->each_part callback
my ($ary, $self) = @_;
diff --git a/lib/PublicInbox/Qspawn.pm b/lib/PublicInbox/Qspawn.pm
index 0e52617c..a4d78e49 100644
--- a/lib/PublicInbox/Qspawn.pm
+++ b/lib/PublicInbox/Qspawn.pm
@@ -29,6 +29,7 @@ use v5.12;
use PublicInbox::Spawn qw(popen_rd);
use PublicInbox::GzipFilter;
use Scalar::Util qw(blessed);
+use PublicInbox::Limiter;
# n.b.: we get EAGAIN with public-inbox-httpd, and EINTR on other PSGI servers
use Errno qw(EAGAIN EINTR);
@@ -183,7 +184,7 @@ sub psgi_qx {
$self->{qx_arg} = $qx_arg;
$self->{qx_fh} = $qx_fh;
$self->{qx_buf} = \$qx_buf;
- $limiter ||= $def_limiter ||= PublicInbox::Qspawn::Limiter->new(32);
+ $limiter ||= $def_limiter ||= PublicInbox::Limiter->new(32);
start($self, $limiter, \&psgi_qx_start);
}
@@ -317,7 +318,7 @@ sub psgi_return {
$self->{psgi_env} = $env;
$self->{hdr_buf} = \(my $hdr_buf = '');
$self->{parse_hdr} = [ $parse_hdr, $hdr_arg ];
- $limiter ||= $def_limiter ||= PublicInbox::Qspawn::Limiter->new(32);
+ $limiter ||= $def_limiter ||= PublicInbox::Limiter->new(32);
# the caller already captured the PSGI write callback from
# the PSGI server, so we can call ->start, here:
@@ -334,46 +335,4 @@ sub psgi_return {
}
}
-package PublicInbox::Qspawn::Limiter;
-use v5.12;
-
-sub new {
- my ($class, $max) = @_;
- bless {
- # 32 is same as the git-daemon connection limit
- max => $max || 32,
- running => 0,
- run_queue => [],
- # RLIMIT_CPU => undef,
- # RLIMIT_DATA => undef,
- # RLIMIT_CORE => undef,
- }, $class;
-}
-
-sub setup_rlimit {
- my ($self, $name, $cfg) = @_;
- foreach my $rlim (@PublicInbox::Spawn::RLIMITS) {
- my $k = lc($rlim);
- $k =~ tr/_//d;
- $k = "publicinboxlimiter.$name.$k";
- defined(my $v = $cfg->{$k}) or next;
- my @rlimit = split(/\s*,\s*/, $v);
- if (scalar(@rlimit) == 1) {
- push @rlimit, $rlimit[0];
- } elsif (scalar(@rlimit) != 2) {
- warn "could not parse $k: $v\n";
- }
- eval { require BSD::Resource };
- if ($@) {
- warn "BSD::Resource missing for $rlim";
- next;
- }
- foreach my $i (0..$#rlimit) {
- next if $rlimit[$i] ne 'INFINITY';
- $rlimit[$i] = BSD::Resource::RLIM_INFINITY();
- }
- $self->{$rlim} = \@rlimit;
- }
-}
-
1;
diff --git a/t/qspawn.t b/t/qspawn.t
index 224e20db..507f86a5 100644
--- a/t/qspawn.t
+++ b/t/qspawn.t
@@ -3,6 +3,7 @@
use v5.12;
use Test::More;
use_ok 'PublicInbox::Qspawn';
+use_ok 'PublicInbox::Limiter';
{
my $cmd = [qw(sh -c), 'echo >&2 err; echo out'];
@@ -23,7 +24,7 @@ sub finish_err ($) {
$qsp->{qsp_err} && ${$qsp->{qsp_err}};
}
-my $limiter = PublicInbox::Qspawn::Limiter->new(1);
+my $limiter = PublicInbox::Limiter->new(1);
{
my $x = PublicInbox::Qspawn->new([qw(true)]);
$x->{qsp_err} = \(my $err = '');
^ permalink raw reply related [relevance 4%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-10-25 0:29 7% [PATCH 00/26] process management simplifications Eric Wong
2023-10-25 0:29 4% ` [PATCH 01/26] limiter: split out from qspawn Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).