Re: too many open files

[Eric Wong <normalperson@yhbt.net>]

"Peter P." <peterparker@fastmail.com> wrote:
> Dear Sox list,
> 
> I am stumbling over a (known) problem:
> 
> I am trying to concatenate many audio files with 
> 
> sox "*.wav" out.wav 
> 
> and after some time am getting the error message
> 
> sox FAIL formats: can’t open input file ‘whatever.wav’ : Too many open files
> 
> I have found other people's postings about the same issue, as eg
> http://sox.10957.n7.nabble.com/Sox-Too-many-open-files-concatenating-from-playlists-with-a-large-number-of-entries-td5117.html
> 
> The max number of files to be concatenated this way seems to vary in
> between operating systems. Or does anyone know exact numbers, or how to
> determine the maximum number on a given OS?

"ulimit -n" shows open files on *nix based systems.
You can change it (given appropriate permissions) by setting it
to a higher number (e.g. "ulimit -n 16384")

> Now I understand that sox is keeping them all opened when concatenating
> them, which might be necessary in other cases, as when trying to
> normalize them all to a common level. Sadly this seems to impose the
> described limit. Or is there another reason why all files have to be
> kept open?

I haven't checked, but I can't think of a reason off hand why
they must be kept open.  I suppose there could be some reasons
for wanting to seek around...

I'm too sleepy and barely awake to think straight now :<
Will check the code when I've more time.

> I tried to work around this by recursively concatenating each input file
> to a common (growing) output file using a simple bash until loop. The
> performance penalty of this is enormous, as the same output file has to
> be opened, read and written each time again.
> 
> I am curious what a possible workaround could be, other than dividing
> above task between multiple calls of sox, each with a reduced number of
> files.

Similar to what Erich said, maybe something like this works by
utilizing pipelines to avoid temporary files:

	FMT="-ts32 -c2 -b24 -r48000" # adjust format to match your data
	sox $FMT "|for i in *.wav; do sox \$i $FMT -; done" out.wav

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Sox-users mailing list
Sox-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-users