From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Cc: Ricardo Ribalda <ribalda@chromium.org>
Subject: [PATCH] lei q|up: limit default write --jobs for IMAP(S)
Date: Mon, 14 Nov 2022 08:07:02 +0000 [thread overview]
Message-ID: <20221114080702.M933003@dcvr> (raw)
In-Reply-To: <20220910201958.GA12212@dcvr>
Eric Wong <e@80x24.org> wrote:
> Thanks for confirming things work as intended. I think the
> default should be clamped, though... 15 seems a bit high for
> smaller IMAP servers *shrug*
--------8<-------
Subject: [PATCH] lei q|up: limit default write --jobs for IMAP(S)
IMAP(S) servers often limit per-user connections, so avoid
bumping into limits to improve the out-of-the-box experience.
4 seems like a conservative default, since we already chose
that number for remote HTTP(S) endpoints.
Link: https://public-inbox.org/meta/20220910201958.GA12212@dcvr/
---
/me having git-repack OOM due to excessive default pack.threads
reminded me of this issue :x
Documentation/lei-q.pod | 4 ++--
lib/PublicInbox/LeiQuery.pm | 7 +++++--
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 8134223e..d52c5b04 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -135,8 +135,8 @@ Set the number of query and write worker processes for parallelism.
C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per
remote (HTTP/HTTPS) host.
-C<WRITE_WORKERS> defaults to the number of CPUs available for Maildir,
-IMAP/IMAPS, and mbox* destinations.
+C<WRITE_WORKERS> defaults to 75% of the number of CPUs available for
+Maildir and mbox* destinations, but 4 per IMAP/IMAPS host.
Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows
one to only set C<WRITE_WORKERS>
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index df9c32b3..0f839236 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -39,8 +39,11 @@ sub _start_query { # used by "lei q" and "lei up"
$lms->lms_write_prepare->lms_pause; # just create
}
}
- $l2m and $l2m->{-wq_nr_workers} //= $mj //
- int($nproc * 0.75 + 0.5); # keep some CPU for git
+ $l2m and $l2m->{-wq_nr_workers} //= $mj // do {
+ # keep some CPU for git, and don't overload IMAP destinations
+ my $n = int($nproc * 0.75 + 0.5);
+ $self->{net} && $n > 4 ? 4 : $n;
+ };
# descending docid order is cheapest, MUA controls sorting order
$self->{mset_opt}->{relevance} //= -2 if $l2m || $opt->{threads};
prev parent reply other threads:[~2022-11-14 8:07 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-09 10:09 imap: "Can't use an undefined value as a subroutine reference" Ricardo Ribalda
2022-09-09 17:44 ` [PATCH] lei: add diagnostics for IMAP writer failures Eric Wong
2022-09-09 18:00 ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
2022-09-09 20:35 ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
2022-09-10 1:18 ` [PATCH v2] lei: bail out earlier on " Eric Wong
2022-09-10 19:34 ` Ricardo Ribalda
2022-09-10 19:50 ` Eric Wong
2022-09-10 19:53 ` Ricardo Ribalda
2022-09-10 20:19 ` Eric Wong
2022-11-14 8:07 ` Eric Wong [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221114080702.M933003@dcvr \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
--cc=ribalda@chromium.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).