new user trying to set up a processing chain

new user trying to set up a processing chain

[raymond grote]

Hi all, new user of sox here. I'm generally unaccustomed to command
line programs, although I can usually get by. So, pardon me if I'm
asking dumb questions  :).

Sox is really appealing to me especially for batch processing tasks.
Right now, I'm trying to think of the most efficient way to make two
copies of a sound, one being the original, and another being processed
in some way eg. pitch shifting. I would then like to mix these two
copies together and perform further processing on the result eg. add
reverb.

As someone who is generally good at putting pieces together, I can
work out how to do this in a messy way. Basically I would manually
mix/process each file. But I unfortunately struggle with efficiency
sometimes, so it would be quite tedious to do this to multiple files.
So, I'm primarily trying to work out how much I can do with one
instance of sox, and if multiple instances are required, where would
they split?
Thanks for any guidance you can provide!


_______________________________________________
Sox-users mailing list
Sox-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-users

Re: new user trying to set up a processing chain

[Jeremy Nicoll - ml sox users]

On 2020-11-16 10:04, raymond grote wrote:

> As someone who is generally good at putting pieces together, I can
> work out how to do this in a messy way. Basically I would manually
> mix/process each file. But I unfortunately struggle with efficiency
> sometimes, so it would be quite tedious to do this to multiple files.

> So, I'm primarily trying to work out how much I can do with one
> instance of sox, and if multiple instances are required, where would
> they split?

I think your question's a bit too vague to answer easily.

What OS are you using, and what scripting language (or shell?) are you
comfortable with?

Do you have an idea yet of what directories/folders you're planning to
keep files in?

Do you know how you plan to name them? (Eg, are you going to have base
leafnames and add words to imply what changes you've made, so that eg

   mysong.wav

might end up with modified versions named eg

   mysong (shifted 2khz).wav

   mysong (shifted 2khz) (reverb).wav

or could it be ...mysong\file001.wav        (the original)
            and ...mysong\filennn.wav        (the n'th variant)?


Do all the files you want to process have a similar naming style?


As far as sox goes, I am not personally in favour of trying to use 
chains
of commands, because although efficiency may be a problem, stuff can run
while I look for food, have a shower, sleep etc... and simplicity in the
scripts wins over for me.  (I'm a retired professional programmer, so my
"simple" might be more complex than yours. And my scripts are written in
ooREXX and run under Windows.)

It's also (I think) a great deal easier (if you realise later on that a
sequence of effects didn't do quite what you wanted) that you can go 
back
and redo (say) the third change taking the output from the second one.
You can't do that if you no longer have the intermediate files.

Also at the start when you are experimenting, I think you'd want to test
each stage separately.


-- 
Jeremy Nicoll - my opinions are my own


_______________________________________________
Sox-users mailing list
Sox-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-users

Re: new user trying to set up a processing chain

[raymond grote]

Hi Jeremy, You're correct, I didn't give enough information.
I'm using Windows 10. I'm not at a very proficient level with
scripting. I see the potential of it, but am not confident with a
scripting language yet. I'm assuming batch script would be fine for my
needs, but right now I'm just trying to learn some basics, so that I
have a better idea of what I'll be scripting when I get there.

I totally understand what you're saying about making sure the
processing is easy to follow and adjust, rather than trying to make it
run as efficiently as possible. I tend to agree with that mindset, but
I feel like there's a line between an easy-to-follow way, and a way
that's hacked together because you don't understand the basic tools
which make certain things a hundred times easier. I sometimes feel
like I'm on the wrong side of that, So I'm a bit paranoid that I'm
doing things in a stupid way haha

A simple example follows. Not sure if I'd actually use this, I was
just trying to think of a situation which illustrates my
uncertainties.
Step 1. I have a file called test.wav and I want to pitch shift it.
sox test.wav temp.wav pitch -500
Step 2. Then I should mix the pitch shift copy with the original, and
apply reverb to the mix.
sox -m test.wav temp.wav test_processed.wav reverb

File/directory structure should be simple. Basically, the original
file and the _processed version should be in the same folder, at least
for now. Temp.wav is not needed. It would be useful to hear
intermediate results, but assuming I don't need to check those, I
really don't need it.

So my question is twofold. Is it common practice to launch sox once
for some processing and then again for other stuff? I know you can
chain effects in one sox instance, but mixing is not an effect as I
understand, which is why I came to the conclusion that you needed to
use a separate instance for mixing. But for all I know, I'm missing
something. Secondly, is it a good idea to use a temporary output file
on the hard drive as I have done for intermediate processing steps?


On 11/16/20, Jeremy Nicoll - ml sox users
<jn.ml.sxu.88@wingsandbeaks.org.uk> wrote:
> On 2020-11-16 10:04, raymond grote wrote:
>
>> As someone who is generally good at putting pieces together, I can
>> work out how to do this in a messy way. Basically I would manually
>> mix/process each file. But I unfortunately struggle with efficiency
>> sometimes, so it would be quite tedious to do this to multiple files.
>
>> So, I'm primarily trying to work out how much I can do with one
>> instance of sox, and if multiple instances are required, where would
>> they split?
>
> I think your question's a bit too vague to answer easily.
>
> What OS are you using, and what scripting language (or shell?) are you
> comfortable with?
>
> Do you have an idea yet of what directories/folders you're planning to
> keep files in?
>
> Do you know how you plan to name them? (Eg, are you going to have base
> leafnames and add words to imply what changes you've made, so that eg
>
>    mysong.wav
>
> might end up with modified versions named eg
>
>    mysong (shifted 2khz).wav
>
>    mysong (shifted 2khz) (reverb).wav
>
> or could it be ...mysong\file001.wav        (the original)
>             and ...mysong\filennn.wav        (the n'th variant)?
>
>
> Do all the files you want to process have a similar naming style?
>
>
> As far as sox goes, I am not personally in favour of trying to use
> chains
> of commands, because although efficiency may be a problem, stuff can run
> while I look for food, have a shower, sleep etc... and simplicity in the
> scripts wins over for me.  (I'm a retired professional programmer, so my
> "simple" might be more complex than yours. And my scripts are written in
> ooREXX and run under Windows.)
>
> It's also (I think) a great deal easier (if you realise later on that a
> sequence of effects didn't do quite what you wanted) that you can go
> back
> and redo (say) the third change taking the output from the second one.
> You can't do that if you no longer have the intermediate files.
>
> Also at the start when you are experimenting, I think you'd want to test
> each stage separately.
>
>
> --
> Jeremy Nicoll - my opinions are my own
>
>
> _______________________________________________
> Sox-users mailing list
> Sox-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sox-users
>


_______________________________________________
Sox-users mailing list
Sox-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-users

Re: new user trying to set up a processing chain

[Jeremy Nicoll - ml sox users]

On 2020-11-17 03:07, raymond grote wrote:
> Hi Jeremy, You're correct, I didn't give enough information.
> I'm using Windows 10. I'm not at a very proficient level with
> scripting. I see the potential of it, but am not confident with a
> scripting language yet. I'm assuming batch script would be fine for my
> needs ...

If you know no other alternative (eg: python, perl etc) and don't
want to install anything else, basic .bat/.cmd scripts might be ok.
If they are only issuing sox commands it should be fine, but if eg
you also want to issue "lame" commands to make mp3 files, and set
tags and have to chop up and recreate modified filenames, you might
need something else.  VBS would probably be better than .bat/.cmd
and the cscript.exe/wscript.exe that runs those is part of Windows.

For bat/cmd (and vbs etc) there's excellent information on how to
use the available commands, at

  https://ss64.com/


> I totally understand what you're saying about making sure the
> processing is easy to follow and adjust, rather than trying to make it
> run as efficiently as possible. I tend to agree with that mindset, but
> I feel like there's a line between an easy-to-follow way, and a way
> that's hacked together because you don't understand the basic tools
> which make certain things a hundred times easier. I sometimes feel
> like I'm on the wrong side of that, So I'm a bit paranoid that I'm
> doing things in a stupid way haha

I think you're overthinking this.  Everyone has to learn, whether it's
better use of script language, or better use of tools like sox.

One of the reasons I do things in stages is that a script that (say)
applies a certain amount of gain to a set of files will be usable in
the next project that wants a different amount of gain applied to a
different set of files.

The same script can be adapted to do something slightly different, as
it already has all the logic for working through a set of files (and
all the error-checking that I wrap around the overall logic).


Time spent writing "the perfect script" is possibly wasted.  What you
need (IMO) is a set of tools that are easy to adapt to new 
circumstances.




> A simple example follows. Not sure if I'd actually use this, I was
> just trying to think of a situation which illustrates my
> uncertainties.

> Step 1. I have a file called test.wav and I want to pitch shift it.
> sox test.wav temp.wav pitch -500

> Step 2. Then I should mix the pitch shift copy with the original, and
> apply reverb to the mix.

> sox -m test.wav temp.wav test_processed.wav reverb

OK, but suppose you get into that habit, of always naming the output 
file
"temp.wav"?  One day you'll forget and find you overwrote the output 
from
another sox command that previously created the same file.


That's an argument for:

  - always using the sox command option that prevents any file from
    being overwritten; that's the "global option": --no-clobber

  - or having scripts always generate result filenames that should not
    already exist, verifying that they don't already exist (and to be
    careful, also using the sox don't-overwrite option)


Sox command parameters are made up as

   [gopts] [[fopts] infile]... [fopts] outfile [effect [effopt]]...

ie global options,  then a set of input files each possibly preceded by
options for how they are processed, then the output file (possibly
preceded by its options) followed by effects each of which is possibly
followed by its options.  (The "[" and "]" might be in the wrong place
but I knew what /I/ meant.)

I find it easier in scripts that build up commands to have code that 
looks
at each potential part of the overall command, maybe contributing 
options,
maybe not.

I also tend to name intermediate (or final) output files so once can see
at least something about the process that created them, so if eg one of
the steps in a script was to reverse a stereo image, the file leafname
might have eg "reversed" in it.





> File/directory structure should be simple. Basically, the original
> file and the _processed version should be in the same folder, at least
> for now. Temp.wav is not needed. It would be useful to hear
> intermediate results, but assuming I don't need to check those, I
> really don't need it.

For /pitch shift/ you might not need it, but for lots of other more
subjective effects you probably will need it.  For example if you
experiment with a dozen slight variations in the parameters for a
particular effect, you'll want all the output files to exist so you
can listen to them (or eg run stat or stats effects to check their
levels etc).  And then you'll not find names like temp, temp1, temp2
... much use for identifying which one's which.



> So my question is twofold. Is it common practice to launch sox once
> for some processing and then again for other stuff?

I don't it matters what's common practice or not.  What matters is
that you understand what your commands do and how they do them, and
are happy with that.

> I know you can chain effects in one sox instance ...

and also pipe stuff between successive calls of sox


If (say) your input files are from some laboratory process so there's
going to be tens of thousands of them and they're alll basically
similar, and you want to do the exact same thing to them all, then
it's maybe worth finding the most efficient way to do it.  But in
the initial stages while you experiment and refine what that process
is, I'd say you still want to make life easy for yourself.  You want
to be able to glance at a filename and have it tell you what's in
the file.

If on the other hand (like me) you deal with distinct sets of files
- recordings of concerts that then get split into separate files, and
have eg levels measured, gains applied, fades in & out applied, and
then mp3s at various quality settings made with appropriate tags)
then every project is different - the times, lengths, position of
fades, fade parameters, gain settings, format of the tags etc are
different from concert to concert.  I'll listen to parts of every
file along the way, so I /have/to create every one of them.


> Secondly, is it a good idea to use a temporary output file
> on the hard drive as I have done for intermediate processing steps?

I'd say yes, but if you really really don't want to, don't.

Going back to your example, suppose you find out later that the final
file has clipping in it?  How do you tell (assuming test.wav was ok)
where that problem came from if you've not got the intermediate files
to listen to?



-- 
Jeremy Nicoll - my opinions are my own


_______________________________________________
Sox-users mailing list
Sox-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-users

Re: new user trying to set up a processing chain

[raymond grote]

Thanks for the ss64.com resource, I will certainly look at that!

I don't worry as much about storing the results of intermediate steps
as I probably should. My habit at the moment is to check a few files I
think could potentially cause the most issues, and if they're fine,
make a batch process to do the rest if the task is the same on every
input. Once I get proficient enough, though, I can make the choice as
to whether I make these temp files or not as you say. Putting an
effort into naming them descriptively is good advice.

I'm not sure exactly what to think about clipping. So far as I know,
many editors work in 32 bit fp so clipping is far less likely to
happen, unless of course you save the file in 16 bit or something. So
I normalize before saving the file. If I understand the manual
correctly, in sox I could use -g before specifying the input file to
guard against clipping from the processing. I'll have to try it.

I've read about piping a number of times but didn't understand it, so
at this point I feel like I really need to learn it. When I last tried
to read up on it, I remember being confused, but that was a while ago,
and I don't think I tried very hard. Now that I'm looking into this
more seriously, I may have success. It's probably a simple thing once
you know it.

On 11/17/20, Jeremy Nicoll - ml sox users
<jn.ml.sxu.88@wingsandbeaks.org.uk> wrote:
> On 2020-11-17 03:07, raymond grote wrote:
>> Hi Jeremy, You're correct, I didn't give enough information.
>> I'm using Windows 10. I'm not at a very proficient level with
>> scripting. I see the potential of it, but am not confident with a
>> scripting language yet. I'm assuming batch script would be fine for my
>> needs ...
>
> If you know no other alternative (eg: python, perl etc) and don't
> want to install anything else, basic .bat/.cmd scripts might be ok.
> If they are only issuing sox commands it should be fine, but if eg
> you also want to issue "lame" commands to make mp3 files, and set
> tags and have to chop up and recreate modified filenames, you might
> need something else.  VBS would probably be better than .bat/.cmd
> and the cscript.exe/wscript.exe that runs those is part of Windows.
>
> For bat/cmd (and vbs etc) there's excellent information on how to
> use the available commands, at
>
>   https://ss64.com/
>
>
>> I totally understand what you're saying about making sure the
>> processing is easy to follow and adjust, rather than trying to make it
>> run as efficiently as possible. I tend to agree with that mindset, but
>> I feel like there's a line between an easy-to-follow way, and a way
>> that's hacked together because you don't understand the basic tools
>> which make certain things a hundred times easier. I sometimes feel
>> like I'm on the wrong side of that, So I'm a bit paranoid that I'm
>> doing things in a stupid way haha
>
> I think you're overthinking this.  Everyone has to learn, whether it's
> better use of script language, or better use of tools like sox.
>
> One of the reasons I do things in stages is that a script that (say)
> applies a certain amount of gain to a set of files will be usable in
> the next project that wants a different amount of gain applied to a
> different set of files.
>
> The same script can be adapted to do something slightly different, as
> it already has all the logic for working through a set of files (and
> all the error-checking that I wrap around the overall logic).
>
>
> Time spent writing "the perfect script" is possibly wasted.  What you
> need (IMO) is a set of tools that are easy to adapt to new
> circumstances.
>
>
>
>
>> A simple example follows. Not sure if I'd actually use this, I was
>> just trying to think of a situation which illustrates my
>> uncertainties.
>
>> Step 1. I have a file called test.wav and I want to pitch shift it.
>> sox test.wav temp.wav pitch -500
>
>> Step 2. Then I should mix the pitch shift copy with the original, and
>> apply reverb to the mix.
>
>> sox -m test.wav temp.wav test_processed.wav reverb
>
> OK, but suppose you get into that habit, of always naming the output
> file
> "temp.wav"?  One day you'll forget and find you overwrote the output
> from
> another sox command that previously created the same file.
>
>
> That's an argument for:
>
>   - always using the sox command option that prevents any file from
>     being overwritten; that's the "global option": --no-clobber
>
>   - or having scripts always generate result filenames that should not
>     already exist, verifying that they don't already exist (and to be
>     careful, also using the sox don't-overwrite option)
>
>
> Sox command parameters are made up as
>
>    [gopts] [[fopts] infile]... [fopts] outfile [effect [effopt]]...
>
> ie global options,  then a set of input files each possibly preceded by
> options for how they are processed, then the output file (possibly
> preceded by its options) followed by effects each of which is possibly
> followed by its options.  (The "[" and "]" might be in the wrong place
> but I knew what /I/ meant.)
>
> I find it easier in scripts that build up commands to have code that
> looks
> at each potential part of the overall command, maybe contributing
> options,
> maybe not.
>
> I also tend to name intermediate (or final) output files so once can see
> at least something about the process that created them, so if eg one of
> the steps in a script was to reverse a stereo image, the file leafname
> might have eg "reversed" in it.
>
>
>
>
>
>> File/directory structure should be simple. Basically, the original
>> file and the _processed version should be in the same folder, at least
>> for now. Temp.wav is not needed. It would be useful to hear
>> intermediate results, but assuming I don't need to check those, I
>> really don't need it.
>
> For /pitch shift/ you might not need it, but for lots of other more
> subjective effects you probably will need it.  For example if you
> experiment with a dozen slight variations in the parameters for a
> particular effect, you'll want all the output files to exist so you
> can listen to them (or eg run stat or stats effects to check their
> levels etc).  And then you'll not find names like temp, temp1, temp2
> ... much use for identifying which one's which.
>
>
>
>> So my question is twofold. Is it common practice to launch sox once
>> for some processing and then again for other stuff?
>
> I don't it matters what's common practice or not.  What matters is
> that you understand what your commands do and how they do them, and
> are happy with that.
>
>> I know you can chain effects in one sox instance ...
>
> and also pipe stuff between successive calls of sox
>
>
> If (say) your input files are from some laboratory process so there's
> going to be tens of thousands of them and they're alll basically
> similar, and you want to do the exact same thing to them all, then
> it's maybe worth finding the most efficient way to do it.  But in
> the initial stages while you experiment and refine what that process
> is, I'd say you still want to make life easy for yourself.  You want
> to be able to glance at a filename and have it tell you what's in
> the file.
>
> If on the other hand (like me) you deal with distinct sets of files
> - recordings of concerts that then get split into separate files, and
> have eg levels measured, gains applied, fades in & out applied, and
> then mp3s at various quality settings made with appropriate tags)
> then every project is different - the times, lengths, position of
> fades, fade parameters, gain settings, format of the tags etc are
> different from concert to concert.  I'll listen to parts of every
> file along the way, so I /have/to create every one of them.
>
>
>> Secondly, is it a good idea to use a temporary output file
>> on the hard drive as I have done for intermediate processing steps?
>
> I'd say yes, but if you really really don't want to, don't.
>
> Going back to your example, suppose you find out later that the final
> file has clipping in it?  How do you tell (assuming test.wav was ok)
> where that problem came from if you've not got the intermediate files
> to listen to?
>
>
>
> --
> Jeremy Nicoll - my opinions are my own
>
>
> _______________________________________________
> Sox-users mailing list
> Sox-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sox-users
>


_______________________________________________
Sox-users mailing list
Sox-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-users