From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 90E8A1FAEE for ; Thu, 22 Feb 2018 21:42:25 +0000 (UTC) From: "Eric Wong (Contractor, The Linux Foundation)" To: meta@public-inbox.org Subject: [PATCH 12/12] searchidxpart: increase pipe size for partitions Date: Thu, 22 Feb 2018 21:42:22 +0000 Message-Id: <20180222214222.1086-13-e@80x24.org> In-Reply-To: <20180222214222.1086-1-e@80x24.org> References: <20180222214222.1086-1-e@80x24.org> List-Id: We want to reduce the time in the main V2Writable process spends writing to the pipe, as the main process itself is the primary source of contention. While we're at it, always flush after writing to ensure the child sees it at once. (Grr... Perl doesn't use writev) --- lib/PublicInbox/SearchIdxPart.pm | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/SearchIdxPart.pm b/lib/PublicInbox/SearchIdxPart.pm index d5a3fd1..5582d67 100644 --- a/lib/PublicInbox/SearchIdxPart.pm +++ b/lib/PublicInbox/SearchIdxPart.pm @@ -20,6 +20,11 @@ sub new { } $v2writable = undef; close $w; + + # F_SETPIPE_SZ = 1031 on Linux; increasing the pipe size here + # speeds V2Writable batch imports across 8 cores by nearly 20% + fcntl($r, 1031, 1048576) if $^O eq 'linux'; + eval { partition_worker_loop($self, $r) }; die "worker $part died: $@\n" if $@; die "unexpected MM $self->{mm}" if $self->{mm}; @@ -63,8 +68,10 @@ sub partition_worker_loop ($$) { # called by V2Writable sub index_raw { my ($self, $len, $msgref, $artnum, $object_id) = @_; - print { $self->{w} } "$len $artnum $object_id\n", $$msgref or die + my $w = $self->{w}; + print $w "$len $artnum $object_id\n", $$msgref or die "failed to write partition $!\n"; + $w->flush or die "failed to flush: $!\n"; } 1; -- EW