From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 5F0ED1FC0B for ; Sun, 19 Sep 2021 12:50:36 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 07/16] lei: clamp internal worker processes to 4 Date: Sun, 19 Sep 2021 12:50:26 +0000 Message-Id: <20210919125035.6331-8-e@80x24.org> In-Reply-To: <20210919125035.6331-1-e@80x24.org> References: <20210919125035.6331-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: "All" my CPUs is only 4, but it's probably ridiculous for somebody with a 16-core system to have 16 processes for accessing SQLite DBs. We do the same thing in Pmdir for parallel Maildir access (and V2Writable). --- lib/PublicInbox/LeiImportKw.pm | 4 +++- lib/PublicInbox/LeiNoteEvent.pm | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/PublicInbox/LeiImportKw.pm b/lib/PublicInbox/LeiImportKw.pm index 2863d17f..379101c2 100644 --- a/lib/PublicInbox/LeiImportKw.pm +++ b/lib/PublicInbox/LeiImportKw.pm @@ -11,7 +11,9 @@ use parent qw(PublicInbox::IPC); sub new { my ($cls, $lei) = @_; my $self = bless { -wq_ident => 'lei import_kw worker' }, $cls; - my ($op_c, $ops) = $lei->workers_start($self, $self->detect_nproc); + my $j = $self->detect_nproc // 4; + $j = 4 if $j > 4; + my ($op_c, $ops) = $lei->workers_start($self, $j); $op_c->{ops} = $ops; # for PktOp->event_step $self->{lei_sock} = $lei->{sock}; $lei->{ikw} = $self; diff --git a/lib/PublicInbox/LeiNoteEvent.pm b/lib/PublicInbox/LeiNoteEvent.pm index 5f692e75..a0591a09 100644 --- a/lib/PublicInbox/LeiNoteEvent.pm +++ b/lib/PublicInbox/LeiNoteEvent.pm @@ -80,8 +80,9 @@ sub lei_note_event { my $self = $cfg->{-lei_note_event} //= do { my $wq = bless { lms => $lms }, __PACKAGE__; # MUAs such as mutt can trigger massive rename() storms so - # use all CPU power available: + # use some CPU, but don't overwhelm slower storage, either my $jobs = $wq->detect_nproc // 1; + $jobs = 4 if $jobs > 4; # same default as V2Writable my ($op_c, $ops) = $lei->workers_start($wq, $jobs); $lei->wait_wq_events($op_c, $ops); note_event_arm_done($lei);