* [PATCH 07/16] lei: clamp internal worker processes to 4
2021-09-19 12:50 4% [PATCH 00/16] lei IPC overhaul, NNTP fixes Eric Wong
@ 2021-09-19 12:50 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-09-19 12:50 UTC (permalink / raw)
To: meta
"All" my CPUs is only 4, but it's probably ridiculous for
somebody with a 16-core system to have 16 processes for
accessing SQLite DBs.
We do the same thing in Pmdir for parallel Maildir access
(and V2Writable).
---
lib/PublicInbox/LeiImportKw.pm | 4 +++-
lib/PublicInbox/LeiNoteEvent.pm | 3 ++-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm
index 2863d17f..379101c2 100644
--- a/lib/PublicInbox/LeiImportKw.pm
+++ b/lib/PublicInbox/LeiImportKw.pm
@@ -11,7 +11,9 @@ use parent qw(PublicInbox::IPC);
sub new {
my ($cls, $lei) = @_;
my $self = bless { -wq_ident => 'lei import_kw worker' }, $cls;
- my ($op_c, $ops) = $lei->workers_start($self, $self->detect_nproc);
+ my $j = $self->detect_nproc // 4;
+ $j = 4 if $j > 4;
+ my ($op_c, $ops) = $lei->workers_start($self, $j);
$op_c->{ops} = $ops; # for PktOp->event_step
$self->{lei_sock} = $lei->{sock};
$lei->{ikw} = $self;
diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm
index 5f692e75..a0591a09 100644
--- a/lib/PublicInbox/LeiNoteEvent.pm
+++ b/lib/PublicInbox/LeiNoteEvent.pm
@@ -80,8 +80,9 @@ sub lei_note_event {
my $self = $cfg->{-lei_note_event} //= do {
my $wq = bless { lms => $lms }, __PACKAGE__;
# MUAs such as mutt can trigger massive rename() storms so
- # use all CPU power available:
+ # use some CPU, but don't overwhelm slower storage, either
my $jobs = $wq->detect_nproc // 1;
+ $jobs = 4 if $jobs > 4; # same default as V2Writable
my ($op_c, $ops) = $lei->workers_start($wq, $jobs);
$lei->wait_wq_events($op_c, $ops);
note_event_arm_done($lei);
^ permalink raw reply related [relevance 7%]
* [PATCH 00/16] lei IPC overhaul, NNTP fixes
@ 2021-09-19 12:50 4% Eric Wong
2021-09-19 12:50 7% ` [PATCH 07/16] lei: clamp internal worker processes to 4 Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-09-19 12:50 UTC (permalink / raw)
To: meta
11/16 is a bit worrying for saved search dedupe over HTTP(S),
and I can't seem to reproduce it reliably, either..
ls-mail-source and import use is far nicer, as it provides a
good avenue for doing partial fetches.
lei/store IPC got a massive overhaul, and the sto_done_request
simplification is nice. This will probably simplify automatic
export-kw support to IMAP folders.
I also noticed "lei config --edit" was wonky, so
I made it share code with "lei edit-search".
Starting to document config knobs, too.
Eric Wong (16):
ipc: wq_do: support synchronous waits and responses
ipc: allow disabling broadcast for wq_workers
lei/store: use SOCK_SEQPACKET rather than pipe
lei: simplify sto_done_request
lei_xsearch: drop Data::Dumper use
ipc: drop dynamic WQ process counts
lei: clamp internal worker processes to 4
lei ls-mail-source: use "high"/"low" for NNTP
lei ls-mail-source: pretty JSON support
net_reader: fix single NNTP article fetch, test ranges
xt: add fsck script over over.sqlite3
watch: use net_reader->mic_new wrapper for SOCKS+TLS
net_reader: no STARTTLS for IMAP localhost or onions
lei config --edit: use controlling terminal
net_reader: disallow imap.fetchBatchSize=0
doc: lei-config: document various knobs
Documentation/lei-config.pod | 91 +++++++++++++++++++-
MANIFEST | 2 +
lib/PublicInbox/IPC.pm | 117 +++++++++++---------------
lib/PublicInbox/LEI.pm | 32 +++----
lib/PublicInbox/LeiConfig.pm | 42 +++++++++
lib/PublicInbox/LeiEditSearch.pm | 60 +++++--------
lib/PublicInbox/LeiExternal.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 2 +-
lib/PublicInbox/LeiImportKw.pm | 6 +-
lib/PublicInbox/LeiIndex.pm | 2 +-
lib/PublicInbox/LeiInit.pm | 4 +-
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiLsMailSource.pm | 25 +++---
lib/PublicInbox/LeiNoteEvent.pm | 11 +--
lib/PublicInbox/LeiRefreshMailSync.pm | 2 +-
lib/PublicInbox/LeiRemote.pm | 4 +-
lib/PublicInbox/LeiRm.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 16 +---
lib/PublicInbox/LeiStore.pm | 22 ++---
lib/PublicInbox/LeiTag.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 22 ++---
lib/PublicInbox/LeiXSearch.pm | 9 +-
lib/PublicInbox/NetReader.pm | 39 +++++----
lib/PublicInbox/WQWorker.pm | 9 +-
lib/PublicInbox/Watch.pm | 3 +-
t/imapd-tls.t | 11 ++-
t/ipc.t | 19 ++---
t/lei-import-nntp.t | 26 ++++++
t/lei.t | 3 +
t/nntpd-tls.t | 8 ++
t/uri_nntps.t | 3 +
xt/over-fsck.perl | 44 ++++++++++
32 files changed, 403 insertions(+), 239 deletions(-)
create mode 100644 lib/PublicInbox/LeiConfig.pm
create mode 100644 xt/over-fsck.perl
^ permalink raw reply [relevance 4%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-09-19 12:50 4% [PATCH 00/16] lei IPC overhaul, NNTP fixes Eric Wong
2021-09-19 12:50 7% ` [PATCH 07/16] lei: clamp internal worker processes to 4 Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).