Adjusting write buffer for recording

Adjusting write buffer for recording

[Shane Blaser]

[-- Attachment #1.1: Type: text/plain, Size: 838 bytes --]

Hello I have an application that processes audio as rec writes the file.

rec is buffering the data before it writes and I get the updates about 2 or
3 seconds late.

I have played with the --buffer pram and I not been able to see a
difference.

I would like to tell rec to write the data as soon as it gets it or reduce
the file write buffer size.

I have silence detection on

rec -c 1 -r 8000 --buffer 4096 currentRecording.wav silence 1 0.0001 2.1% 1
15.0 2.6%


The writes look like this

  Size of currentRecording.wav = 24576  Size of currentRecording.wav = 36864
*Size* *of* *currentRecording.wav* *=* *45056*  *Size* *of* *
currentRecording.wav* *=* *77824*


The data I am looking for is split between the two last writes ....


Any ideas ?

I just pulled down the source any ideas what file handles writing the files?


Thanks

[-- Attachment #1.2: Type: text/html, Size: 3482 bytes --]

[-- Attachment #2: Type: text/plain, Size: 405 bytes --]

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-devel

Re: Adjusting write buffer for recording

[Jan Stary]

On Aug 09 14:05:32, shane.blaser@callfire.com wrote:
> Hello I have an application that processes audio as rec writes the file.

Does this mean that rec(1) writes into a pipe
and your application is reading that pipe?

> rec is buffering the data before it writes and I get
> the updates about 2 or 3 seconds late.

How do you know that? How exactly are you reading
what rec(1) is writing?

> I have played with the --buffer pram and I not been able to see a
> difference.
> 
> I would like to tell rec to write the data as soon as it gets it or reduce
> the file write buffer size.

That's what --buffer is supposed to do ...

> I have silence detection on
> rec -c 1 -r 8000 --buffer 4096 currentRecording.wav silence 1 0.0001 2.1% 1
> 15.0 2.6%
> 
> The writes look like this
> 
>   Size of currentRecording.wav = 24576  Size of currentRecording.wav = 36864
> *Size* *of* *currentRecording.wav* *=* *45056*  *Size* *of* *
> currentRecording.wav* *=* *77824*

The steps are indeed multiples of 4096.
Can you try with other --buffer to see
if the data becomes available to your application
in the chunks that rec(1) is writing?


------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

Re: Adjusting write buffer for recording

[Shane Blaser]

[-- Attachment #1.1: Type: text/plain, Size: 2183 bytes --]

Rec writes to a file I watch the file.

I listen to the sound coming in and watch the file size.  There is a period
of silence then when volume rises the file is written to.  But this
introduces a delay I can not afford

I have adjusted the buffer down to 128 ( below that I get constant
overrun)  I see no change in the file writing....
On Aug 11, 2013 10:10 AM, "Jan Stary" <hans@stare.cz> wrote:

> On Aug 09 14:05:32, shane.blaser@callfire.com wrote:
> > Hello I have an application that processes audio as rec writes the file.
>
> Does this mean that rec(1) writes into a pipe
> and your application is reading that pipe?
>
> > rec is buffering the data before it writes and I get
> > the updates about 2 or 3 seconds late.
>
> How do you know that? How exactly are you reading
> what rec(1) is writing?
>
> > I have played with the --buffer pram and I not been able to see a
> > difference.
> >
> > I would like to tell rec to write the data as soon as it gets it or
> reduce
> > the file write buffer size.
>
> That's what --buffer is supposed to do ...
>
> > I have silence detection on
> > rec -c 1 -r 8000 --buffer 4096 currentRecording.wav silence 1 0.0001
> 2.1% 1
> > 15.0 2.6%
> >
> > The writes look like this
> >
> >   Size of currentRecording.wav = 24576  Size of currentRecording.wav =
> 36864
> > *Size* *of* *currentRecording.wav* *=* *45056*  *Size* *of* *
> > currentRecording.wav* *=* *77824*
>
> The steps are indeed multiples of 4096.
> Can you try with other --buffer to see
> if the data becomes available to your application
> in the chunks that rec(1) is writing?
>
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> SoX-devel mailing list
> SoX-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sox-devel
>

[-- Attachment #1.2: Type: text/html, Size: 2965 bytes --]

[-- Attachment #2: Type: text/plain, Size: 405 bytes --]

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-devel

Re: Adjusting write buffer for recording

[Jan Stary]

On Aug 11 22:11:44, shane.blaser@callfire.com wrote:
> Rec writes to a file I watch the file.
> I listen to the sound coming in and watch the file size.

So you are calling stat(2) in a loop or what?
That itself would introduce a delay.

> There is a period of silence
> then when volume rises the file is written to.

Yes, that's what the 'silence' effect does.
That's what you want, right?

> But this introduces a delay I can not afford

Well, if there is a half hour of 'silence',
there will be a half hour pause in writing to the file of course.


------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

Re: Adjusting write buffer for recording

[Shane Blaser]

[-- Attachment #1.1: Type: text/plain, Size: 2003 bytes --]

Yes i am calling stat,

But I can see that the file size does not change for a few seconds and then
the data is flushed.

What I am doing is decoding caller id on a phone line:

So the flow goes like this:

1. rec listening with silence detection
2. Call comes in and recording starts.
3. Caller id is between 1st and second ring.
(ok this is were the issue comes in)
after the first ring I get the first part of the caller id signal written
to disk (call it 1/2 of the caller id signal). (listening to the audio I
know we have whole signal.)
But not until after the second ring do I get the audio for the rest of the
caller id written to disk.

Can I have rec write silence to the file after the recording starts?

Thanks for sticking with me on this

Shane






On Sun, Aug 11, 2013 at 11:09 PM, Jan Stary <hans@stare.cz> wrote:

> On Aug 11 22:11:44, shane.blaser@callfire.com wrote:
> > Rec writes to a file I watch the file.
> > I listen to the sound coming in and watch the file size.
>
> So you are calling stat(2) in a loop or what?
> That itself would introduce a delay.
>
> > There is a period of silence
> > then when volume rises the file is written to.
>
> Yes, that's what the 'silence' effect does.
> That's what you want, right?
>
> > But this introduces a delay I can not afford
>
> Well, if there is a half hour of 'silence',
> there will be a half hour pause in writing to the file of course.
>
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> SoX-devel mailing list
> SoX-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sox-devel
>

[-- Attachment #1.2: Type: text/html, Size: 3142 bytes --]

[-- Attachment #2: Type: text/plain, Size: 405 bytes --]

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-devel

Re: Adjusting write buffer for recording

[Jan Stary]

On Aug 12 07:31:24, shane.blaser@callfire.com wrote:
> Yes i am calling stat,

Wouldn't it be easire to just write to a pipe,
and have your application read from that pipe?

> But I can see that the file size does not change
> for a few seconds and then the data is flushed.
> 
> What I am doing is decoding caller id on a phone line:
> So the flow goes like this:

Please, in the future, _start_ with such top level description
of what you are trying to do. Make it easy for people to help you.

> 1. rec listening with silence detection
> 2. Call comes in and recording starts.
> 3. Caller id is between 1st and second ring.
> (ok this is were the issue comes in)
> after the first ring I get the first part of the caller id signal written
> to disk (call it 1/2 of the caller id signal).

Let me understand this: in the phone calls you are recording,
there is a piece of _audio_ signal, comming between the
first and second ring, which carries the caller's ID.
Is that right? Can you please elaborate? I never knew
there is something like that coming on a phone line.

> (listening to the audio I know we have whole signal.)

So this caller ID is an _audible_ signal,
present in the audio of the phone call?

> But not until after the second ring
> do I get the audio for the rest of the caller id written to disk.
> Can I have rec write silence to the file after the recording starts?

Of course: just rec(1) everything,
without using the 'silence' effect.

What you are doing now is the exact opposite:
you are telling rec(1) _not_ to write until
a certain treshold of 'silence' is surpassed.

I am too lazy now to look up your exact 'silence' parameters,
but could it be that the caller ID audio signal contains
enough 'silence' so that rec(1) pauses?

	Jan


------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

Re: Adjusting write buffer for recording

[Shane Blaser]

[-- Attachment #1.1: Type: text/plain, Size: 2875 bytes --]

comments inline


On Mon, Aug 12, 2013 at 9:03 AM, Jan Stary <hans@stare.cz> wrote:

> On Aug 12 07:31:24, shane.blaser@callfire.com wrote:
> > Yes i am calling stat,
>
> Wouldn't it be easire to just write to a pipe,
> and have your application read from that pipe?
>

Maybe not sure the file seems to work.


> > But I can see that the file size does not change
> > for a few seconds and then the data is flushed.
> >
> > What I am doing is decoding caller id on a phone line:
> > So the flow goes like this:
>
> Please, in the future, _start_ with such top level description
> of what you are trying to do. Make it easy for people to help you.
>

Sorry about that

>
> > 1. rec listening with silence detection
> > 2. Call comes in and recording starts.
> > 3. Caller id is between 1st and second ring.
> > (ok this is were the issue comes in)
> > after the first ring I get the first part of the caller id signal written
> > to disk (call it 1/2 of the caller id signal).
>
> Let me understand this: in the phone calls you are recording,
> there is a piece of _audio_ signal, comming between the
> first and second ring, which carries the caller's ID.
> Is that right? Can you please elaborate? I never knew
> there is something like that coming on a phone line.
>

yes it is audible and it sounds like a fax machine or modem ..

>
> > (listening to the audio I know we have whole signal.)
>
> So this caller ID is an _audible_ signal,
> present in the audio of the phone call?
>

Yes see above

>
> > But not until after the second ring
> > do I get the audio for the rest of the caller id written to disk.
> > Can I have rec write silence to the file after the recording starts?
>
> Of course: just rec(1) everything,
> without using the 'silence' effect.


> What you are doing now is the exact opposite:
> you are telling rec(1) _not_ to write until
> a certain treshold of 'silence' is surpassed.
>
> I am too lazy now to look up your exact 'silence' parameters,
> but could it be that the caller ID audio signal contains
> enough 'silence' so that rec(1) pauses?
>

After the caller id signal there is a silence until the next ring ...

Do you see any way for me to adjust the write buffers, if I can make them
smaller I think this issue will fall away ...?

>
>         Jan
>
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> SoX-devel mailing list
> SoX-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sox-devel
>

[-- Attachment #1.2: Type: text/html, Size: 4825 bytes --]

[-- Attachment #2: Type: text/plain, Size: 405 bytes --]

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-devel

Re: Adjusting write buffer for recording

[Jan Stary]

On Aug 12 09:16:32, shane.blaser@callfire.com wrote:
> > On Aug 12 07:31:24, shane.blaser@callfire.com wrote:
> > > Yes i am calling stat,
> >
> > Wouldn't it be easire to just write to a pipe,
> > and have your application read from that pipe?
> >
> Maybe not sure the file seems to work.

No doubt, but you have to call stat() in a loop
to find out whether the file has grown; with a pipe,
you would just be read()ing and getting the data (or not).

> > > 1. rec listening with silence detection
> > > 2. Call comes in and recording starts.
> > > 3. Caller id is between 1st and second ring.
> > > (ok this is were the issue comes in)
> > > after the first ring I get the first part of the caller id signal written
> > > to disk (call it 1/2 of the caller id signal).
> >
> > Let me understand this: in the phone calls you are recording,
> > there is a piece of _audio_ signal, comming between the
> > first and second ring, which carries the caller's ID.
> > Is that right? Can you please elaborate? I never knew
> > there is something like that coming on a phone line.
> >
> yes it is audible and it sounds like a fax machine or modem ..

> > > But not until after the second ring
> > > do I get the audio for the rest of the caller id written to disk.
> > > Can I have rec write silence to the file after the recording starts?
> >
> > Of course: just rec(1) everything,
> > without using the 'silence' effect.
> 
> > What you are doing now is the exact opposite:
> > you are telling rec(1) _not_ to write until
> > a certain treshold of 'silence' is surpassed.
> >
> > I am too lazy now to look up your exact 'silence' parameters,
> > but could it be that the caller ID audio signal contains
> > enough 'silence' so that rec(1) pauses?
> >
> After the caller id signal there is a silence until the next ring ...
> 
> Do you see any way for me to adjust the write buffers, if I can make them
> smaller I think this issue will fall away ...?

This has nothing to do with buffers.

You are running rec(1) with the silence effect.
That makes rec(1) pause when it sees the silence.
Just run rec(1) without the silence effect - that
way, rec will just write everything, without pausing
at the 'silent' moments.

The writing will of course be buffered, and you can play
with --buffer to make the continuous writing more smooth.
But that's not your problem now: rec waits with writing
because you are telling it to wait (for a non-silence).

	Jan


------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

Re: Adjusting write buffer for recording

[Shane Blaser]

[-- Attachment #1.1: Type: text/plain, Size: 3466 bytes --]

Thanks Jan,

Just to confirm there is no way to have rec not pause once it has started?

I could use silence detection to start the recording then record everything
until we see the set number of seconds of silence ?

Thanks again

Shane


On Mon, Aug 12, 2013 at 10:11 AM, Jan Stary <hans@stare.cz> wrote:

> On Aug 12 09:16:32, shane.blaser@callfire.com wrote:
> > > On Aug 12 07:31:24, shane.blaser@callfire.com wrote:
> > > > Yes i am calling stat,
> > >
> > > Wouldn't it be easire to just write to a pipe,
> > > and have your application read from that pipe?
> > >
> > Maybe not sure the file seems to work.
>
> No doubt, but you have to call stat() in a loop
> to find out whether the file has grown; with a pipe,
> you would just be read()ing and getting the data (or not).
>
> > > > 1. rec listening with silence detection
> > > > 2. Call comes in and recording starts.
> > > > 3. Caller id is between 1st and second ring.
> > > > (ok this is were the issue comes in)
> > > > after the first ring I get the first part of the caller id signal
> written
> > > > to disk (call it 1/2 of the caller id signal).
> > >
> > > Let me understand this: in the phone calls you are recording,
> > > there is a piece of _audio_ signal, comming between the
> > > first and second ring, which carries the caller's ID.
> > > Is that right? Can you please elaborate? I never knew
> > > there is something like that coming on a phone line.
> > >
> > yes it is audible and it sounds like a fax machine or modem ..
>
> > > > But not until after the second ring
> > > > do I get the audio for the rest of the caller id written to disk.
> > > > Can I have rec write silence to the file after the recording starts?
> > >
> > > Of course: just rec(1) everything,
> > > without using the 'silence' effect.
> >
> > > What you are doing now is the exact opposite:
> > > you are telling rec(1) _not_ to write until
> > > a certain treshold of 'silence' is surpassed.
> > >
> > > I am too lazy now to look up your exact 'silence' parameters,
> > > but could it be that the caller ID audio signal contains
> > > enough 'silence' so that rec(1) pauses?
> > >
> > After the caller id signal there is a silence until the next ring ...
> >
> > Do you see any way for me to adjust the write buffers, if I can make them
> > smaller I think this issue will fall away ...?
>
> This has nothing to do with buffers.
>
> You are running rec(1) with the silence effect.
> That makes rec(1) pause when it sees the silence.
> Just run rec(1) without the silence effect - that
> way, rec will just write everything, without pausing
> at the 'silent' moments.
>
> The writing will of course be buffered, and you can play
> with --buffer to make the continuous writing more smooth.
> But that's not your problem now: rec waits with writing
> because you are telling it to wait (for a non-silence).
>
>         Jan
>
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> SoX-devel mailing list
> SoX-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sox-devel
>

[-- Attachment #1.2: Type: text/html, Size: 4858 bytes --]

[-- Attachment #2: Type: text/plain, Size: 405 bytes --]

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-devel

Re: Adjusting write buffer for recording

[Jan Stary]

On Aug 12 10:28:51, shane.blaser@callfire.com wrote:
> Just to confirm there is no way to have rec not pause once it has started?

I repeat: call rec(1) without the silence effect.
That way, you record everything, continuously.

> I could use silence detection to start the recording then record everything
> until we see the set number of seconds of silence ?

Yes, that is also possible. But if you have the (unexplained) need
to process the audio signal in realtime, just let rec(1) do its writing,
and maybe postprocess the file later (trimming the silence).


------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk