* [PATCH 2/5] lei_to_mail: limit workers for text, reply and v2 outputs
2021-10-30 8:11 5% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
@ 2021-10-30 8:11 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
"text" and "reply" outputs are intended for the pager, so
parallelizing them is a waste of resources.
v2 has shards, of course, so parallelizing writes to it
is also a waste since the deduplication work is a bit
more complex.
---
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 3d0e5b144d3a..352ee60131aa 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -37,7 +37,7 @@ sub _start_query { # used by "lei q" and "lei up"
$lms->lms_write_prepare->lms_pause; # just create
}
}
- $l2m and $l2m->{-wq_nr_workers} = $mj //
+ $l2m and $l2m->{-wq_nr_workers} //= $mj //
int($nproc * 0.75 + 0.5); # keep some CPU for git
# descending docid order is cheapest, MUA controls sorting order
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 83f58a29405b..90db30e53089 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -415,11 +415,13 @@ sub new {
require PublicInbox::LeiViewText;
$lei->{lvt} = PublicInbox::LeiViewText->new($lei, $fmt);
$self->{base_type} = 'text';
+ $self->{-wq_nr_workers} = 1; # for pager
@conflict = qw(mua save);
} elsif ($fmt eq 'v2') {
die "--dedupe=oid and v2 are incompatible\n" if
($lei->{opt}->{dedupe}//'') eq 'oid';
$self->{base_type} = 'v2';
+ $self->{-wq_nr_workers} = 1; # v2 has shards
$lei->{opt}->{save} = \1;
$dst = $lei->{ovv}->{dst} = $lei->abs_path($dst);
@conflict = qw(mua sort);
^ permalink raw reply related [relevance 7%]
* [PATCH 0/5] lei: fix various SIGPIPE problems
@ 2021-10-30 8:11 5% Eric Wong
2021-10-30 8:11 7% ` [PATCH 2/5] lei_to_mail: limit workers for text, reply and v2 outputs Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-10-30 8:11 UTC (permalink / raw)
To: meta
Most worrying was the the bug fixed in 4/5; but at least there
wasn't data loss involved...
While the bug fixed in 4/5 didn't cause data loss, it was
dumping core files and filling up my disk while polluting the
kernel log buffer.
Eric Wong (5):
lei: do not access {sock} after SIGPIPE
lei_to_mail: limit workers for text, reply and v2 outputs
lei_xsearch: quiet error message on SIG{PIPE,TERM}
lei_to_mail: avoid SEGV on worker exit via SIGTERM
doc: lei-security: add a note about core dumps
Documentation/lei-security.pod | 6 ++++++
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 10 +++++++++-
lib/PublicInbox/LeiXSearch.pm | 5 ++++-
t/lei-sigpipe.t | 7 +++++--
6 files changed, 26 insertions(+), 6 deletions(-)
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-10-30 8:11 5% [PATCH 0/5] lei: fix various SIGPIPE problems Eric Wong
2021-10-30 8:11 7% ` [PATCH 2/5] lei_to_mail: limit workers for text, reply and v2 outputs Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).