From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 125CC1F910; Mon, 14 Nov 2022 08:07:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1668413223; bh=dzRYde0N6zGCJIMXJP3P2Ps7wN9weK5+3ODfRVpQzgA=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=fqYx79B4YLVG4ZLgYXGdX5q3RJLaZhvy+WXb+eJKrdi0S1NwHcRt9e9Mk53UcGeJW IQ3iU76Hc4TjHwzGfunwfFt+lje867IHG4xs3eRPQ+YVBoCHWjAq92LJjcL4Il91WO CablRe7Dd0/qbjdm63ChBjniAwHhAnYfEQDczksw= Date: Mon, 14 Nov 2022 08:07:02 +0000 From: Eric Wong To: meta@public-inbox.org Cc: Ricardo Ribalda Subject: [PATCH] lei q|up: limit default write --jobs for IMAP(S) Message-ID: <20221114080702.M933003@dcvr> References: <20220909174410.M560915@dcvr> <20220910011859.M68532@dcvr> <20220910195012.M987266@dcvr> <20220910201958.GA12212@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20220910201958.GA12212@dcvr> List-Id: Eric Wong wrote: > Thanks for confirming things work as intended. I think the > default should be clamped, though... 15 seems a bit high for > smaller IMAP servers *shrug* --------8<------- Subject: [PATCH] lei q|up: limit default write --jobs for IMAP(S) IMAP(S) servers often limit per-user connections, so avoid bumping into limits to improve the out-of-the-box experience. 4 seems like a conservative default, since we already chose that number for remote HTTP(S) endpoints. Link: https://public-inbox.org/meta/20220910201958.GA12212@dcvr/ --- /me having git-repack OOM due to excessive default pack.threads reminded me of this issue :x Documentation/lei-q.pod | 4 ++-- lib/PublicInbox/LeiQuery.pm | 7 +++++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod index 8134223e..d52c5b04 100644 --- a/Documentation/lei-q.pod +++ b/Documentation/lei-q.pod @@ -135,8 +135,8 @@ Set the number of query and write worker processes for parallelism. C defaults to the number of CPUs available, but 4 per remote (HTTP/HTTPS) host. -C defaults to the number of CPUs available for Maildir, -IMAP/IMAPS, and mbox* destinations. +C defaults to 75% of the number of CPUs available for +Maildir and mbox* destinations, but 4 per IMAP/IMAPS host. Omitting C but leaving the comma (C<,>) allows one to only set C diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm index df9c32b3..0f839236 100644 --- a/lib/PublicInbox/LeiQuery.pm +++ b/lib/PublicInbox/LeiQuery.pm @@ -39,8 +39,11 @@ sub _start_query { # used by "lei q" and "lei up" $lms->lms_write_prepare->lms_pause; # just create } } - $l2m and $l2m->{-wq_nr_workers} //= $mj // - int($nproc * 0.75 + 0.5); # keep some CPU for git + $l2m and $l2m->{-wq_nr_workers} //= $mj // do { + # keep some CPU for git, and don't overload IMAP destinations + my $n = int($nproc * 0.75 + 0.5); + $self->{net} && $n > 4 ? 4 : $n; + }; # descending docid order is cheapest, MUA controls sorting order $self->{mset_opt}->{relevance} //= -2 if $l2m || $opt->{threads};