| Date | Commit message (Collapse) |
|---|
|
We'll enable automatic cleanup when IPC classes go out-of-scope
to avoid leaving zombies around.
->wq_workers will be a useful convenience method to change
worker counts.
|
|
It is not used anywhere.
|
|
Increasing/decreasing workers count will be useful in
some situations.
|
|
We can just EOF the pipe, and instead rely on per-class
error handling to deal with uncommitted transactions and
what not.
|
|
This will allow any number of younger sibling processes to
communicate with older siblings directly without relying on a
mediator process. This is intended to be useful for
distributing search work across multiple workers without caring
which worker hits it (we only care about shard members).
And any request sent with this will be able to hit any worker
without locking on our part.
Unix stream sockets with a listener were also considered;
binding to a file on the FS may confuse users given there's
already a socket path for lei(1). Linux-only Abstract or
autobind sockets are rejected due to lack of portability.
SOCK_SEQPACKET via socketpair(2) was chosen since it's POSIX
2008 and available on FreeBSD 9+ in addition to Linux, and
doesn't require filesystem access.
|
|
We should not need an eval for warning with our code base.
Nowadays, dwaitpid() automatically does the right thing
regardless of whether we're in the event loop, so no eval
is needed there, either.
|
|
Similar to git->cat_async, this will let us deal with responses
asynchronously, as well as being able to mix synchronous and
asynchronous code transparently (though perhaps not optimally).
|
|
This lets us call dwaitpid long before a process exits
and not have to wait around for it.
This is advantageous for lei where we can run dwaitpid on the
pager as soon as we spawn it, instead of waiting for a client
socket to go away on DESTROY.
|
|
This fixes a performance regression in multi-process v2 indexing
due to the switch to PublicInbox::IPC. While Unix sockets are
fewer FDs to manage, pipes allow unprivileged processes to use
larger buffers (up to 1M) on out-of-the-box Linux instances.
A larger buffer via F_SETPIPE_SZ afforded by pipes was proven
valuable during v2 development in 2018 and continues to be
valuable when we get significant amounts of one-way traffic from
the producer parent to worker children.
Compression may be an option for systems without F_SETPIPE_SZ;
but it increases CPU usage with no memory bandwidth savings on
hosts where larger buffers are available.
|
|
It's nice to prove the new code works by swapping it into
the current V2Writable / SearchIdxShard packages. This is
only the first step for the core bits, and we'll be able
to delete more code in a subsequent patch.
|
|
Fix some comments and add some short summary descriptions to
hopefully make things easier-to-follow.
|
|
Using "make update-copyrights" after setting GNULIB_PATH in my
config.mak
|
|
shutdown(2) on a socket can be preferable if there's multiple
forked processes writing to a single worker and we really want
to shut things down ASAP.
It may also be good to provide an ipc_worker_exit method which
subclasses can override if needed for graceful shutdown. But we
won't need equivalents to atexit(3) since we can rely on DESTROY
handlers given this is Perl5.
|
|
Some testing will be needed to see if it's worth the code
and maintenance overhead, but it seems easy-enough to get
working.
|
|
I intend to use this with LeiStore when importing from multiple
slow sources at once (e.g. curl, IMAP, etc). This is because
over.sqlite3 can only have a single writer, and we'll have
several slow readers running in parallel.
Watch and SearchIdxShard should also be able to use this code
in the future, but this will be proven with LeiStore, first.
|