* [PATCH 1/7] ipc: support awaitpid in WQ workers
2023-04-05 11:26 6% [PATCH 0/7] cindex: fix FreeBSD freezes Eric Wong
@ 2023-04-05 11:26 7% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2023-04-05 11:26 UTC (permalink / raw)
To: meta
Using signalfd is necessary to get reliable signal wakeups w/o
polling on fixed intervals. This change will make it possible
to use awaitpid in cidx shard workers so they can perform prune
work while waiting on the initial output of `git log -p'.
---
lib/PublicInbox/IPC.pm | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 1f0e87ee..cca3dacb 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -268,12 +268,14 @@ sub sock_defined {
defined($wqw->{sock});
}
-sub wq_worker_loop ($$) {
- my ($self, $bcast2) = @_;
+sub wq_worker_loop ($$$) {
+ my ($self, $bcast2, $oldset) = @_;
my $wqw = PublicInbox::WQWorker->new($self, $self->{-wq_s2});
PublicInbox::WQWorker->new($self, $bcast2) if $bcast2;
local @PublicInbox::DS::post_loop_do = (\&sock_defined, $wqw);
- PublicInbox::DS::event_loop();
+ my $sig = delete($self->{wq_sig});
+ $sig->{CHLD} //= \&PublicInbox::DS::enqueue_reap;
+ PublicInbox::DS::event_loop($sig, $oldset);
PublicInbox::DS->Reset;
}
@@ -405,8 +407,7 @@ sub _wq_worker_start {
local @$self{keys %$fields} = values(%$fields);
my $on_destroy = $self->ipc_atfork_child;
local @SIG{keys %SIG} = values %SIG;
- PublicInbox::DS::sig_setmask($oldset);
- wq_worker_loop($self, $bcast2);
+ wq_worker_loop($self, $bcast2, $oldset);
};
warn "worker $self->{-wq_ident} PID:$$ died: $@" if $@;
undef $end; # trigger exit
^ permalink raw reply related [relevance 7%]
* [PATCH 0/7] cindex: fix FreeBSD freezes
@ 2023-04-05 11:26 6% Eric Wong
2023-04-05 11:26 7% ` [PATCH 1/7] ipc: support awaitpid in WQ workers Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2023-04-05 11:26 UTC (permalink / raw)
To: meta
PATCH 6/7 is the fix. It took me a while to realize how to fix it.
1-5 improve parallelism and reduce resource usage while limiting
the workaround to a single callsite in #6. And #7 is just a
cleanliness fix.
Eric Wong (7):
ipc: support awaitpid in WQ workers
cindex: do prune work while waiting for `git log -p'
cindex: share PktOp socket across prune workers
cindex: share PktOp across indexing workers
cindex: enter event loop once per run
cindex: workaround for FreeBSD missing SIGCHLD
cindex: reset DS internals on cidx_run completion
MANIFEST | 1 +
lib/PublicInbox/CidxLogP.pm | 29 ++++
lib/PublicInbox/CodeSearchIdx.pm | 290 ++++++++++++++++---------------
lib/PublicInbox/IPC.pm | 11 +-
4 files changed, 184 insertions(+), 147 deletions(-)
create mode 100644 lib/PublicInbox/CidxLogP.pm
^ permalink raw reply [relevance 6%]
* [PATCH 1/4] ipc: support awaitpid in WQ workers
2023-03-31 10:20 6% [PATCH 0/4] cindex: parallelism++, sockets-- Eric Wong
@ 2023-03-31 10:20 7% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2023-03-31 10:20 UTC (permalink / raw)
To: meta
Using signalfd is necessary to get reliable signal wakeups w/o
polling on fixed intervals. This change will make it possible
to use awaitpid in cidx shard workers so they can perform prune
work while waiting on the initial output of `git log -p'.
---
lib/PublicInbox/IPC.pm | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 1f0e87ee..cca3dacb 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -268,12 +268,14 @@ sub sock_defined {
defined($wqw->{sock});
}
-sub wq_worker_loop ($$) {
- my ($self, $bcast2) = @_;
+sub wq_worker_loop ($$$) {
+ my ($self, $bcast2, $oldset) = @_;
my $wqw = PublicInbox::WQWorker->new($self, $self->{-wq_s2});
PublicInbox::WQWorker->new($self, $bcast2) if $bcast2;
local @PublicInbox::DS::post_loop_do = (\&sock_defined, $wqw);
- PublicInbox::DS::event_loop();
+ my $sig = delete($self->{wq_sig});
+ $sig->{CHLD} //= \&PublicInbox::DS::enqueue_reap;
+ PublicInbox::DS::event_loop($sig, $oldset);
PublicInbox::DS->Reset;
}
@@ -405,8 +407,7 @@ sub _wq_worker_start {
local @$self{keys %$fields} = values(%$fields);
my $on_destroy = $self->ipc_atfork_child;
local @SIG{keys %SIG} = values %SIG;
- PublicInbox::DS::sig_setmask($oldset);
- wq_worker_loop($self, $bcast2);
+ wq_worker_loop($self, $bcast2, $oldset);
};
warn "worker $self->{-wq_ident} PID:$$ died: $@" if $@;
undef $end; # trigger exit
^ permalink raw reply related [relevance 7%]
* [PATCH 0/4] cindex: parallelism++, sockets--
@ 2023-03-31 10:20 6% Eric Wong
2023-03-31 10:20 7% ` [PATCH 1/4] ipc: support awaitpid in WQ workers Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2023-03-31 10:20 UTC (permalink / raw)
To: meta
I've noticed `git log -p ...' can take a while to generate
diffs, so being able to do prune work is helpful in the
meantime.
I've also noticed PktOp use was excessive so we now share
sockets across processes without an increase in complexity.
Eric Wong (4):
ipc: support awaitpid in WQ workers
cindex: do prune work while waiting for `git log -p'
cindex: share PktOp socket across prune workers
cindex: share PktOp across indexing workers
MANIFEST | 1 +
lib/PublicInbox/CidxLogP.pm | 29 +++++++
lib/PublicInbox/CodeSearchIdx.pm | 129 ++++++++++++++++++-------------
lib/PublicInbox/IPC.pm | 11 +--
4 files changed, 110 insertions(+), 60 deletions(-)
create mode 100644 lib/PublicInbox/CidxLogP.pm
^ permalink raw reply [relevance 6%]
Results 1-4 of 4 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-03-31 10:20 6% [PATCH 0/4] cindex: parallelism++, sockets-- Eric Wong
2023-03-31 10:20 7% ` [PATCH 1/4] ipc: support awaitpid in WQ workers Eric Wong
2023-04-05 11:26 6% [PATCH 0/7] cindex: fix FreeBSD freezes Eric Wong
2023-04-05 11:26 7% ` [PATCH 1/7] ipc: support awaitpid in WQ workers Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).