From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=AWL,BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id C9CBF1F5AE for ; Fri, 31 Jul 2020 10:54:16 +0000 (UTC) Received: from [127.0.0.1] (helo=sfs-ml-4.v29.lw.sourceforge.com) by sfs-ml-4.v29.lw.sourceforge.com with esmtp (Exim 4.90_1) (envelope-from ) id 1k1Sfy-0001el-5N; Fri, 31 Jul 2020 10:54:10 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-4.v29.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k1Sfw-0001ed-Ne for sox-devel@lists.sourceforge.net; Fri, 31 Jul 2020 10:54:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:Content-Type:MIME-Version :Message-ID:In-Reply-To:Date:References:Subject:Cc:To:From:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=OzpmmamqpyC3CH8nQwlMoJcyK2cq3kIbr6AbiQuD+M0=; b=Sjk1EmSe49B7+/WX/w6tkRBnYe cXdRjhtHVjG0aoa2WuPL8rNBi8o3tQRlo2EeNromjDp9eksuti8mdP+um1iUb7A5Vup8UXVT4+ZUX 6UYrHbO3QtES1rUE2vhTN/vKNYSp1RCVqgdqaHsl5RyVTG7/38e9OUfB/9J9W4xP1Ysg=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-ID: In-Reply-To:Date:References:Subject:Cc:To:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=OzpmmamqpyC3CH8nQwlMoJcyK2cq3kIbr6AbiQuD+M0=; b=DKEeNGjCrwVg94CxR99+gDFDbM x8MQ+WD1LtLXmcwIyCxN8TzT6ilsLtRiepfqiHKAURGbkBYnpYHXGJhGweuNoS3LXm/XTASQYeJCg rdsQ85z+HjEc5kxd8p3C2yVKtrOsBGcemuoarKNHPtvqTq2kY0cFW67AbGPeGcC5+jw0=; Received: from unicorn.mansr.com ([81.2.72.234]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92.2) id 1k1Sfs-00FmrJ-JA for sox-devel@lists.sourceforge.net; Fri, 31 Jul 2020 10:54:08 +0000 Received: from raven.mansr.com (raven.mansr.com [IPv6:2001:8b0:ca0d:8d8e::3]) by unicorn.mansr.com (Postfix) with ESMTPS id 8E5AC15360; Fri, 31 Jul 2020 11:53:46 +0100 (BST) Received: by raven.mansr.com (Postfix, from userid 51770) id 835A821A6F2; Fri, 31 Jul 2020 11:53:46 +0100 (BST) From: =?iso-8859-1?Q?M=E5ns_Rullg=E5rd?= To: Eric Wong References: <20200731093801.23548-1-normalperson@yhbt.net> <20200731093801.23548-3-normalperson@yhbt.net> Date: Fri, 31 Jul 2020 11:53:46 +0100 In-Reply-To: <20200731093801.23548-3-normalperson@yhbt.net> (Eric Wong's message of "Fri, 31 Jul 2020 09:37:54 +0000") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-Headers-End: 1k1Sfs-00FmrJ-JA Subject: Re: [PATCH RESEND 2/9] speed up "|program" inputs on Linux 2.6.35+ X-BeenThere: sox-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: sox-devel@lists.sourceforge.net Cc: sox-devel@lists.sourceforge.net Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Errors-To: sox-devel-bounces@lists.sourceforge.net Eric Wong writes: > Linux 2.6.35+ allows pipe buffer resizing via fcntl(2). When > running multi-threaded SoX invocations with large buffers, the > default pipe size (64K) can be too small and become a bottleneck > for IPC. Increasing the pipe to the maximum allowed size > reduces the amount of stalls in data flow between processes (SoX > or otherwise). > > When using SOX_OPTS=3D"--multi-thread --buffer 131072" on a 4-core > system, a command like: > > sox -M "|sox $< -p $(lft_fx)" "|sox $< -p $(rgt_fx)" $@ > > ..can run significantly faster (10-80%) depending on the > processing chain, file sizes/formats and effects in use. > > Before this patch, using "--buffer 131072" could be hugely > detrimental to performance due to the pipe being only half > the size (64K) of the SoX buffer. > --- > src/formats.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 51 insertions(+) > > diff --git a/src/formats.c b/src/formats.c > index f3efe764..1da8489c 100644 > --- a/src/formats.c > +++ b/src/formats.c > @@ -18,6 +18,7 @@ > * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA > */ > > +#define _GNU_SOURCE > #include "sox_i.h" > > #include > @@ -37,6 +38,10 @@ > #include > #endif > > +#ifdef HAVE_UNISTD_H > +# include > +#endif > + > #define PIPE_AUTO_DETECT_SIZE 256 /* Only as much as we can rewind a pip= e */ > #define AUTO_DETECT_SIZE 4096 /* For seekable file, so no restrictio= n */ > > @@ -370,6 +375,50 @@ static int xfclose(FILE * file, lsx_io_type io_type) > fclose(file); > } > > +static void incr_pipe_size(FILE *f) > +{ > +/* > + * Linux 2.6.35 and later has the ability to expand the pipe buffer > + * Try to get it as big as possible to avoid stalls when SoX itself > + * is using big buffers > + */ > +#if defined(F_GETPIPE_SZ) && defined(F_SETPIPE_SZ) > + static long max_pipe_size; > + > + /* read the maximum size of the pipe the first time this is called */ > + if (max_pipe_size =3D=3D 0) { > + const char path[] =3D "/proc/sys/fs/pipe-max-size"; > + int fd =3D open(path, O_RDONLY); > + > + max_pipe_size =3D -1; > + if (fd >=3D 0) { > + char buf[80]; > + ssize_t r =3D read(fd, buf, sizeof(buf)); > + > + if (r > 0) { > + buf[r] =3D 0; In the (bizarre) event that read() actually reads 80 bytes, that will write outside the buffer. I'll fix it, just to be sure. > + max_pipe_size =3D strtol(buf, NULL, 10); > + > + /* guard against obviously wrong values on messed up systems */ > + if (max_pipe_size <=3D PIPE_BUF || max_pipe_size > INT_MAX) > + max_pipe_size =3D -1; > + } > + close(fd); > + } > + } > + > + if (max_pipe_size > PIPE_BUF) { > + int fd =3D fileno(f); > + > + if (fcntl(fd, F_SETPIPE_SZ, max_pipe_size) >=3D 0) > + lsx_debug("got pipe %ld bytes\n", max_pipe_size); > + else > + lsx_warn("couldn't set pipe size to %ld bytes: %s\n", > + max_pipe_size, strerror(errno)); > + } > +#endif /* do nothing for platforms without F_{GET,SET}PIPE_SZ */ > +} > + > static FILE * xfopen(char const * identifier, char const * mode, lsx_io_= type * io_type) > { > *io_type =3D lsx_io_file; > @@ -382,6 +431,7 @@ static FILE * xfopen(char const * identifier, char co= nst * mode, lsx_io_type * i > #endif > f =3D popen(identifier + 1, POPEN_MODE); > *io_type =3D lsx_io_pipe; > + incr_pipe_size(f); > #else > lsx_fail("this build of SoX cannot open pipes"); > #endif > @@ -394,6 +444,7 @@ static FILE * xfopen(char const * identifier, char co= nst * mode, lsx_io_type * i > char * command =3D lsx_malloc(strlen(command_format) + strlen(identi= fier)); > sprintf(command, command_format, identifier); > f =3D popen(command, POPEN_MODE); > + incr_pipe_size(f); > free(command); > *io_type =3D lsx_io_url; > #else -- = M=E5ns Rullg=E5rd _______________________________________________ SoX-devel mailing list SoX-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sox-devel