sox-devel@lists.sourceforge.net unofficial mirror
 help / color / mirror / Atom feed
From: "Måns Rullgård" <mans@mansr.com>
To: Eric Wong <normalperson@yhbt.net>
Cc: sox-devel@lists.sourceforge.net
Subject: Re: [PATCH RESEND 2/9] speed up "|program" inputs on Linux 2.6.35+
Date: Fri, 31 Jul 2020 11:53:46 +0100	[thread overview]
Message-ID: <yw1xo8nwymyt.fsf@mansr.com> (raw)
In-Reply-To: <20200731093801.23548-3-normalperson@yhbt.net> (Eric Wong's message of "Fri, 31 Jul 2020 09:37:54 +0000")

Eric Wong <normalperson@yhbt.net> writes:

> Linux 2.6.35+ allows pipe buffer resizing via fcntl(2).   When
> running multi-threaded SoX invocations with large buffers, the
> default pipe size (64K) can be too small and become a bottleneck
> for IPC.  Increasing the pipe to the maximum allowed size
> reduces the amount of stalls in data flow between processes (SoX
> or otherwise).
>
> When using SOX_OPTS="--multi-thread --buffer 131072" on a 4-core
> system, a command like:
>
> 	sox -M "|sox $< -p $(lft_fx)" "|sox $< -p $(rgt_fx)" $@
>
> ..can run significantly faster (10-80%) depending on the
> processing chain, file sizes/formats and effects in use.
>
> Before this patch, using "--buffer 131072" could be hugely
> detrimental to performance due to the pipe being only half
> the size (64K) of the SoX buffer.
> ---
>  src/formats.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 51 insertions(+)
>
> diff --git a/src/formats.c b/src/formats.c
> index f3efe764..1da8489c 100644
> --- a/src/formats.c
> +++ b/src/formats.c
> @@ -18,6 +18,7 @@
>   * Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
>   */
>
> +#define _GNU_SOURCE
>  #include "sox_i.h"
>
>  #include <assert.h>
> @@ -37,6 +38,10 @@
>    #include <magic.h>
>  #endif
>
> +#ifdef HAVE_UNISTD_H
> +#  include <unistd.h>
> +#endif
> +
>  #define PIPE_AUTO_DETECT_SIZE 256 /* Only as much as we can rewind a pipe */
>  #define AUTO_DETECT_SIZE 4096     /* For seekable file, so no restriction */
>
> @@ -370,6 +375,50 @@ static int xfclose(FILE * file, lsx_io_type io_type)
>      fclose(file);
>  }
>
> +static void incr_pipe_size(FILE *f)
> +{
> +/*
> + * Linux 2.6.35 and later has the ability to expand the pipe buffer
> + * Try to get it as big as possible to avoid stalls when SoX itself
> + * is using big buffers
> + */
> +#if defined(F_GETPIPE_SZ) && defined(F_SETPIPE_SZ)
> +  static long max_pipe_size;
> +
> +  /* read the maximum size of the pipe the first time this is called */
> +  if (max_pipe_size == 0) {
> +    const char path[] = "/proc/sys/fs/pipe-max-size";
> +    int fd = open(path, O_RDONLY);
> +
> +    max_pipe_size = -1;
> +    if (fd >= 0) {
> +      char buf[80];
> +      ssize_t r = read(fd, buf, sizeof(buf));
> +
> +      if (r > 0) {
> +        buf[r] = 0;

In the (bizarre) event that read() actually reads 80 bytes, that will
write outside the buffer.  I'll fix it, just to be sure.

> +        max_pipe_size = strtol(buf, NULL, 10);
> +
> +        /* guard against obviously wrong values on messed up systems */
> +        if (max_pipe_size <= PIPE_BUF || max_pipe_size > INT_MAX)
> +          max_pipe_size = -1;
> +      }
> +      close(fd);
> +    }
> +  }
> +
> +  if (max_pipe_size > PIPE_BUF) {
> +    int fd = fileno(f);
> +
> +    if (fcntl(fd, F_SETPIPE_SZ, max_pipe_size) >= 0)
> +      lsx_debug("got pipe %ld bytes\n", max_pipe_size);
> +    else
> +      lsx_warn("couldn't set pipe size to %ld bytes: %s\n",
> +               max_pipe_size, strerror(errno));
> +  }
> +#endif /* do nothing for platforms without F_{GET,SET}PIPE_SZ */
> +}
> +
>  static FILE * xfopen(char const * identifier, char const * mode, lsx_io_type * io_type)
>  {
>    *io_type = lsx_io_file;
> @@ -382,6 +431,7 @@ static FILE * xfopen(char const * identifier, char const * mode, lsx_io_type * i
>  #endif
>      f = popen(identifier + 1, POPEN_MODE);
>      *io_type = lsx_io_pipe;
> +    incr_pipe_size(f);
>  #else
>      lsx_fail("this build of SoX cannot open pipes");
>  #endif
> @@ -394,6 +444,7 @@ static FILE * xfopen(char const * identifier, char const * mode, lsx_io_type * i
>      char * command = lsx_malloc(strlen(command_format) + strlen(identifier));
>      sprintf(command, command_format, identifier);
>      f = popen(command, POPEN_MODE);
> +    incr_pipe_size(f);
>      free(command);
>      *io_type = lsx_io_url;
>  #else

-- 
Måns Rullgård


_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-devel

  reply	other threads:[~2020-07-31 10:54 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-31  9:37 [PATCH RESEND 0/9] some old accumulated patches Eric Wong
2020-07-31  9:37 ` [PATCH RESEND 1/9] use non-blocking stdin for interactive mode Eric Wong
2020-07-31  9:37 ` [PATCH RESEND 2/9] speed up "|program" inputs on Linux 2.6.35+ Eric Wong
2020-07-31 10:53   ` Måns Rullgård [this message]
2020-07-31  9:37 ` [PATCH RESEND 3/9] sox.1: fix section name Eric Wong
2020-07-31  9:37 ` [PATCH RESEND 4/9] sndio: handle 24-bit samples properly on OpenBSD Eric Wong
2020-07-31 10:59   ` Måns Rullgård
2020-07-31  9:37 ` [PATCH RESEND 5/9] Handle vorbis_analysis_headerout errors Eric Wong
2020-07-31  9:37 ` [PATCH RESEND 6/9] fix manpage warning: "table wider than line width" Eric Wong
2020-07-31  9:37 ` [PATCH RESEND 7/9] spectrogram: remove arbitrary limit on height of spectrogram Eric Wong
2020-07-31 14:34   ` Måns Rullgård
     [not found]     ` <CAL4-wQpF-qOD=BRVPhgZFC7fjvFDV-rQx1stvwY_xCTyj5uooA@mail.gmail.com>
2020-08-13 16:20       ` Måns Rullgård
2020-08-13 16:30         ` Pander via SoX-devel
2020-08-13 16:38           ` Måns Rullgård
2020-07-31  9:38 ` [PATCH RESEND 8/9] Add spectrogram -n flag to normalise the output to maximum brightness Eric Wong
2020-07-31  9:38 ` [PATCH RESEND 9/9] Added average power spectrum for stat -freq -a Eric Wong
2020-08-01 11:52   ` Måns Rullgård
2020-08-02  1:03     ` Eric Wong
2020-08-02 10:43       ` Måns Rullgård
2020-08-02 10:52     ` Måns Rullgård
2020-08-03 12:44       ` Pander via SoX-devel
2020-08-03 13:41         ` Måns Rullgård
2020-07-31 10:13 ` [PATCH RESEND 0/9] some old accumulated patches Måns Rullgård
2020-07-31 21:16   ` Eric Wong
2020-07-31 21:44     ` Måns Rullgård
2020-08-01 16:57 ` Måns Rullgård

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-list from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://lists.sourceforge.net/lists/listinfo/sox-devel

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=yw1xo8nwymyt.fsf@mansr.com \
    --to=sox-devel@lists.sourceforge.net \
    --cc=normalperson@yhbt.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).