* [PATCH 05/26] lei: use spawn (vfork + execve) for lazy start
2020-12-18 12:09 5% [PATCH 00/26] lei: basic UI + IPC work Eric Wong
@ 2020-12-18 12:09 7% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2020-12-18 12:09 UTC (permalink / raw)
To: meta
This allows us to rely on FD_CLOEXEC being set on pipes
from prove(1), so forgetting `daemon-stop' won't cause
tests to hang.
Unfortunately, daemon tests will be slower with this.
---
lib/PublicInbox/LeiDaemon.pm | 12 +++++-------
script/lei | 14 ++++++++++----
2 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LeiDaemon.pm b/lib/PublicInbox/LeiDaemon.pm
index b4b1ac59..fd4d00d4 100644
--- a/lib/PublicInbox/LeiDaemon.pm
+++ b/lib/PublicInbox/LeiDaemon.pm
@@ -324,29 +324,27 @@ sub accept_dispatch { # Listener {post_accept} callback
sub noop {}
# lei(1) calls this when it can't connect
-sub lazy_start ($$) {
+sub lazy_start {
my ($path, $err) = @_;
if ($err == ECONNREFUSED) {
unlink($path) or die "unlink($path): $!";
} elsif ($err != ENOENT) {
die "connect($path): $!";
}
+ require IO::FDPass;
my $umask = umask(077) // die("umask(077): $!");
my $l = IO::Socket::UNIX->new(Local => $path,
Listen => 1024,
Type => SOCK_STREAM) or
$err = $!;
umask($umask) or die("umask(restore): $!");
- $l or return $err;
+ $l or return die "bind($path): $err";
my @st = stat($path) or die "stat($path): $!";
my $dev_ino_expect = pack('dd', $st[0], $st[1]); # dev+ino
pipe(my ($eof_r, $eof_w)) or die "pipe: $!";
my $oldset = PublicInbox::Sigfd::block_signals();
my $pid = fork // die "fork: $!";
- if ($pid) {
- PublicInbox::Sigfd::sig_setmask($oldset);
- return; # client will connect to $path
- }
+ return if $pid;
openlog($path, 'pid', 'user');
local $SIG{__DIE__} = sub {
syslog('crit', "@_");
@@ -360,7 +358,7 @@ sub lazy_start ($$) {
open STDERR, '>&STDIN' or die "redirect stderr failed: $!\n";
setsid();
$pid = fork // die "fork: $!";
- exit if $pid;
+ return if $pid;
$0 = "lei-daemon $path";
require PublicInbox::Listener;
require PublicInbox::EOFpipe;
diff --git a/script/lei b/script/lei
index 1b5af3a1..637c1951 100755
--- a/script/lei
+++ b/script/lei
@@ -21,13 +21,19 @@ if (eval { require IO::FDPass; 1 }) { # use daemon to reduce load time
};
my $sock = IO::Socket::UNIX->new(Peer => $path, Type => SOCK_STREAM);
unless ($sock) { # start the daemon if not started
- my $err = $!;
- require PublicInbox::LeiDaemon;
- $err = PublicInbox::LeiDaemon::lazy_start($path, $err);
+ my $err = $! + 0;
+ my $env = { PERL5LIB => join(':', @INC) };
+ my $cmd = [ $^X, qw[-MPublicInbox::LeiDaemon
+ -E PublicInbox::LeiDaemon::lazy_start(@ARGV)],
+ $path, $err ];
+ require PublicInbox::Spawn;
+ waitpid(PublicInbox::Spawn::spawn($cmd, $env), 0);
+ warn "lei-daemon exited with \$?=$?\n" if $?;
+
# try connecting again anyways, unlink+bind may be racy
$sock = IO::Socket::UNIX->new(Peer => $path,
Type => SOCK_STREAM) // die
- "connect($path): $! (bind($path): $err)";
+ "connect($path): $! (after attempted daemon start)";
}
my $pwd = $ENV{PWD};
my $cwd = cwd();
^ permalink raw reply related [relevance 7%]
* [PATCH 00/26] lei: basic UI + IPC work
@ 2020-12-18 12:09 5% Eric Wong
2020-12-18 12:09 7% ` [PATCH 05/26] lei: use spawn (vfork + execve) for lazy start Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2020-12-18 12:09 UTC (permalink / raw)
To: meta
Some work on the storage side, but MiscIdx still needs work to
handle existing publicinboxes, extinboxes (over HTTP(S)), and
other config things.
PATCH 22/26 - bash completion sorta works, but filename
completions get broken. Not sure why and help would be
greatly appreciated (along with help for other shells).
I don't know bash-specific stuff well at all, even; and
less about other non-POSIX shells.
Somewhat nice UI things (at least to my delirious sleep-deprived
state):
* -$DIGIT option parsing works (e.g. "git log -10"),
"kill -9"
* help-based CLI arg/prototype checking seems working
and hopefully cuts down on long-term maintenance work
while promoting UI consistency
* having IO::FDPass hides startup time, 20-30ms isn't
really noticeable for humans on interactive terminals,
but still not ideal for loops.
* lei.sh + "make symlink-install"
And some internal improvements:
* several simplifications to existing Search code,
->xdb_shards_flat will come in handy
* generic OnDestroy - long overdue
Eric Wong (26):
lei: FD-passing and IPC basics
lei: proposed command-listing and options
lei_store: local storage for Local Email Interface
tests: more common JSON module loading
lei: use spawn (vfork + execve) for lazy start
lei: refine help/option parsing, implement "init"
t/lei-oneshot: standalone oneshot (non-socket) test
lei: ensure we run a restrictive umask
lei: support `daemon-env' for modifying long-lived env
lei_store: simplify git_epoch_max, slightly
search: simplify initialization, add ->xdb_shards_flat
rename LeiDaemon package to PublicInbox::LEI
lei: support pass-through for `lei config'
lei: help: show actual paths being operated on
lei: rename $client => $self and bless
lei: micro-optimize startup time
lei_store: relax GIT_COMMITTER_IDENT check
lei_store: keyword extraction from mbox and Maildir
on_destroy: generic localized END
lei: restore default __DIE__ handler for event loop
lei: drop $SIG{__DIE__}, add oneshot fallbacks
lei: start working on bash completion
build: add lei.sh + "make symlink-install" target
lei: support for -$DIGIT and -$SIG CLI switches
lei: revise output routines
lei: extinbox: start implementing in config file
MANIFEST | 11 +
Makefile.PL | 11 +
contrib/completion/lei-completion.bash | 11 +
lei.sh | 7 +
lib/PublicInbox/Daemon.pm | 6 +-
lib/PublicInbox/ExtSearch.pm | 10 +-
lib/PublicInbox/ExtSearchIdx.pm | 35 +-
lib/PublicInbox/Import.pm | 4 +
lib/PublicInbox/LEI.pm | 776 +++++++++++++++++++++++++
lib/PublicInbox/LeiExtinbox.pm | 52 ++
lib/PublicInbox/LeiSearch.pm | 39 ++
lib/PublicInbox/LeiStore.pm | 227 ++++++++
lib/PublicInbox/ManifestJsGz.pm | 2 +-
lib/PublicInbox/OnDestroy.pm | 16 +
lib/PublicInbox/OverIdx.pm | 10 +
lib/PublicInbox/Search.pm | 65 +--
lib/PublicInbox/SearchIdx.pm | 62 +-
lib/PublicInbox/SearchIdxShard.pm | 33 ++
lib/PublicInbox/TestCommon.pm | 7 +-
lib/PublicInbox/V2Writable.pm | 10 +-
script/lei | 76 +++
t/extsearch.t | 3 +-
t/lei-oneshot.t | 25 +
t/lei.t | 306 ++++++++++
t/lei_store.t | 88 +++
t/on_destroy.t | 25 +
t/www_listing.t | 8 +-
27 files changed, 1843 insertions(+), 82 deletions(-)
create mode 100644 contrib/completion/lei-completion.bash
create mode 100755 lei.sh
create mode 100644 lib/PublicInbox/LEI.pm
create mode 100644 lib/PublicInbox/LeiExtinbox.pm
create mode 100644 lib/PublicInbox/LeiSearch.pm
create mode 100644 lib/PublicInbox/LeiStore.pm
create mode 100644 lib/PublicInbox/OnDestroy.pm
create mode 100755 script/lei
create mode 100644 t/lei-oneshot.t
create mode 100644 t/lei.t
create mode 100644 t/lei_store.t
create mode 100644 t/on_destroy.t
^ permalink raw reply [relevance 5%]
* [RFC 7/7] lei: use spawn (vfork + execve) for lazy start
2020-12-15 11:47 6% [PATCH/RFC 0/7] lei - Local Email Interface skeleton Eric Wong
@ 2020-12-15 11:47 7% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2020-12-15 11:47 UTC (permalink / raw)
To: meta
This allows us to rely on FD_CLOEXEC being set on pipes
from prove(1), so forgetting `daemon-stop' won't cause
tests to hang.
Unfortunately, daemon tests will be slower with this.
---
lib/PublicInbox/LeiDaemon.pm | 12 +++++-------
script/lei | 14 ++++++++++----
2 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LeiDaemon.pm b/lib/PublicInbox/LeiDaemon.pm
index 20ff0758..2f614ba4 100644
--- a/lib/PublicInbox/LeiDaemon.pm
+++ b/lib/PublicInbox/LeiDaemon.pm
@@ -335,29 +335,27 @@ sub accept_dispatch { # Listener {post_accept} callback
sub noop {}
# lei(1) calls this when it can't connect
-sub lazy_start ($$) {
+sub lazy_start {
my ($path, $err) = @_;
if ($err == ECONNREFUSED) {
unlink($path) or die "unlink($path): $!";
} elsif ($err != ENOENT) {
die "connect($path): $!";
}
+ require IO::FDPass;
my $umask = umask(077) // die("umask(077): $!");
my $l = IO::Socket::UNIX->new(Local => $path,
Listen => 1024,
Type => SOCK_STREAM) or
$err = $!;
umask($umask) or die("umask(restore): $!");
- $l or return $err;
+ $l or return die "bind($path): $err";
my @st = stat($path) or die "stat($path): $!";
my $dev_ino_expect = pack('dd', $st[0], $st[1]); # dev+ino
pipe(my ($eof_r, $eof_w)) or die "pipe: $!";
my $oldset = PublicInbox::Sigfd::block_signals();
my $pid = fork // die "fork: $!";
- if ($pid) {
- PublicInbox::Sigfd::sig_setmask($oldset);
- return; # client will connect to $path
- }
+ return if $pid;
openlog($path, 'pid', 'user');
local $SIG{__DIE__} = sub {
syslog('crit', "@_");
@@ -371,7 +369,7 @@ sub lazy_start ($$) {
open STDERR, '>&STDIN' or die "redirect stderr failed: $!\n";
setsid();
$pid = fork // die "fork: $!";
- exit if $pid;
+ return if $pid;
$0 = "lei-daemon $path";
require PublicInbox::Listener;
require PublicInbox::EOFpipe;
diff --git a/script/lei b/script/lei
index 1b5af3a1..637c1951 100755
--- a/script/lei
+++ b/script/lei
@@ -21,13 +21,19 @@ if (eval { require IO::FDPass; 1 }) { # use daemon to reduce load time
};
my $sock = IO::Socket::UNIX->new(Peer => $path, Type => SOCK_STREAM);
unless ($sock) { # start the daemon if not started
- my $err = $!;
- require PublicInbox::LeiDaemon;
- $err = PublicInbox::LeiDaemon::lazy_start($path, $err);
+ my $err = $! + 0;
+ my $env = { PERL5LIB => join(':', @INC) };
+ my $cmd = [ $^X, qw[-MPublicInbox::LeiDaemon
+ -E PublicInbox::LeiDaemon::lazy_start(@ARGV)],
+ $path, $err ];
+ require PublicInbox::Spawn;
+ waitpid(PublicInbox::Spawn::spawn($cmd, $env), 0);
+ warn "lei-daemon exited with \$?=$?\n" if $?;
+
# try connecting again anyways, unlink+bind may be racy
$sock = IO::Socket::UNIX->new(Peer => $path,
Type => SOCK_STREAM) // die
- "connect($path): $! (bind($path): $err)";
+ "connect($path): $! (after attempted daemon start)";
}
my $pwd = $ENV{PWD};
my $cwd = cwd();
^ permalink raw reply related [relevance 7%]
* [PATCH/RFC 0/7] lei - Local Email Interface skeleton
@ 2020-12-15 11:47 6% Eric Wong
2020-12-15 11:47 7% ` [RFC 7/7] lei: use spawn (vfork + execve) for lazy start Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2020-12-15 11:47 UTC (permalink / raw)
To: meta
patches 1 and 2 are boring cleanups.
The most important is 4/7 which features data structures
for a proposed command set. Hopefully the command-names
and 1-line descriptions are helpful.
Comments from (potential) users appreciated, especially about 4/7.
I decided to take care of patch 3/7 (FD-passing) early on
because startup latency sucks.
I never used notmuch, but this will feature saved searches (aka
"named queries"). Otherwise, the query subcommand will probably
operate like mairix and dump the results to a
Maildir/mbox/etc...
patch 5/7 - keywords (e.g. `seen', 'draft', ...) read/write
(but not query) support added.
And a couple more cleanups.
lei will have its own writable git storage on top of extindex,
but will be able to do read-only queries against extinbox
(publicinbox || extindex) sources.
Eric Wong (7):
daemon: support --daemonize without Net::Server::Daemonize
daemon: simplify fork() failure checks
lei: FD-passing and IPC basics
lei: proposed command-listing and options
lei_store: local storage for Local Email Interface
tests: more common JSON module loading
lei: use spawn (vfork + execve) for lazy start
MANIFEST | 6 +
lib/PublicInbox/Daemon.pm | 26 +-
lib/PublicInbox/ExtSearch.pm | 4 +-
lib/PublicInbox/ExtSearchIdx.pm | 35 ++-
lib/PublicInbox/Import.pm | 4 +
lib/PublicInbox/LeiDaemon.pm | 449 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiSearch.pm | 40 +++
lib/PublicInbox/LeiStore.pm | 197 +++++++++++++
lib/PublicInbox/ManifestJsGz.pm | 2 +-
lib/PublicInbox/OverIdx.pm | 10 +
lib/PublicInbox/SearchIdx.pm | 47 +++-
lib/PublicInbox/SearchIdxShard.pm | 33 +++
lib/PublicInbox/TestCommon.pm | 4 +
lib/PublicInbox/V2Writable.pm | 2 +-
script/lei | 64 +++++
t/extsearch.t | 3 +-
t/lei.t | 79 ++++++
t/lei_store.t | 74 +++++
t/www_listing.t | 8 +-
19 files changed, 1055 insertions(+), 32 deletions(-)
create mode 100644 lib/PublicInbox/LeiDaemon.pm
create mode 100644 lib/PublicInbox/LeiSearch.pm
create mode 100644 lib/PublicInbox/LeiStore.pm
create mode 100755 script/lei
create mode 100644 t/lei.t
create mode 100644 t/lei_store.t
^ permalink raw reply [relevance 6%]
Results 1-4 of 4 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-12-15 11:47 6% [PATCH/RFC 0/7] lei - Local Email Interface skeleton Eric Wong
2020-12-15 11:47 7% ` [RFC 7/7] lei: use spawn (vfork + execve) for lazy start Eric Wong
2020-12-18 12:09 5% [PATCH 00/26] lei: basic UI + IPC work Eric Wong
2020-12-18 12:09 7% ` [PATCH 05/26] lei: use spawn (vfork + execve) for lazy start Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).