Re: Examples of concurrent coproc usage?

[Carl Edquist <edquist@cs.wisc.edu>]

[-- Attachment #1: Type: text/plain, Size: 5088 bytes --]

On Sat, 13 Apr 2024, Chet Ramey wrote:

> The original intent was to allow the shell to drive a long-running 
> process that ran more-or-less in parallel with it. Look at 
> examples/scripts/bcalc for an example of that kind of use.

Thanks for mentioning this example.  As you understand, this model use 
case does not require closing the coproc fds when finished, because they 
will be closed implicitly when the shell exits.  (As bcalc itself admits.)

And if the coproc is left open for the lifetime of the shell, the 
alternate behavior of deferring the coproc deallocation (until both coproc 
fds are closed) would not require anything extra from the user.

The bcalc example does close both coproc fds though - both at the end, and 
whenever it resets.  And so in this example (which as you say, was the 
original intent), the user is already explicitly closing both coproc fds 
explicitly; so the alternate deferring behavior would not require anything 
extra from the user here either.

...

Yet another point brought to light by the bcalc example relates to the 
coproc pid variable.  The reset() function first closes the coproc pipe 
fds, then sleeps for a second to give the BC coproc some time to finish.

An alternative might be to 'wait' for the coproc to finish (likely faster 
than sleeping for a second).  But you have to make and use your 
$coproc_pid copy rather than $BC_PID directly, because 'wait $BC_PID' may 
happen before or after the coproc is reaped and BC_PID is unset.  (As the 
bcalc author seems to understand.)  So in general the coproc *_PID 
variable only seems usable for making a copy when starting the coproc.

The info page has the following:

> The process ID of the shell spawned to execute the coprocess is 
> available as the value of the variable 'NAME_PID'.  The 'wait' builtin 
> command may be used to wait for the coprocess to terminate.

But it seems to me that the copy is necessary, and it is never reliable to 
run 'wait $NAME_PID'.  Because any time the shell is in a position to wait 
for the coproc to finish, by that time it's going to be a race whether or 
not NAME_PID is still set.


So this is another example for me of why it would be handy if coproc 
deallocation were deferred until explicit user action (closing both coproc 
fds, or unsetting the coproc variable).  That way ${NAME[@]} and $NAME_PID 
could reliably be used directly without having to make copies.

Anyway, just food for thought if down the line you make a shell option for 
coproc deallocation behavior.


>> Now, if you built bash with multiple coproc support, I would have 
>> expected you could still rig this up, by doing the redirection work 
>> explicitly yourself.  Something like this:
>>
>>      coproc UP   { stdbuf -oL tr a-z A-Z; }
>>      coproc DOWN { stdbuf -oL tr A-Z a-z; }
>>
>>      # make user-managed backup copies of coproc fds
>>      exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
>>      exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}
>>
>>      coproc THREEWAY { tee /dev/fd/$up_w  /dev/fd/$down_w; }
>> 
>> 
>> But the above doesn't actually work, as it seems that the coproc shell 
>> (THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even 
>> the user-managed ones explicitly copied with exec.
>
> File descriptors the user saves with exec redirections beyond [0-2] are 
> set to close-on-exec. POSIX makes that behavior unspecified, but bash 
> has always done it.

Ah, ok, thanks.  I believe I found where this gets set in 
do_redirection_internal() in redir.c.  (Whew, a big function.)

As far as I can tell the close-on-exec state is "duplicated" rather than 
set unconditionally.  That is, the new fd in a redirection is only set 
close-on-exec if the source is.  (Good, because in general I rely on 
redirections to be made available to external commands.)  But apparently 
coproc marks its pipe fds close-on-exec, so there's no way to expose 
manual copies of these fds to external commands.

So, that explains the behavior I was seeing ...

It's just a bit too bad for anyone that actually wants to do more 
elaborate coproc interconnections with manual redirections, as they're 
limited to shell builtins.

...

I might pose a question to ponder about this though:

With the multi-coproc support code, is it still necessary to set the 
coproc pipe fds close-on-exec?  (If, perhaps, they're already getting 
explicitly closed in the right places.)

Because if the new coproc fds are _not_ set close-on-exec, in general that 
would allow the user to do manual redirections for external commands (eg 
tee(1) or paste(1)) to communicate with multiple coproc fds together.


> Shells don't offer any standard way to modify the state of that flag, 
> but there is the `fdflags' loadable builtin you can experiment with to 
> change close-on-exec.

Thanks for the tip.  It's nice to know there is a workaround to leave 
copies of the coproc fds open across exec; though for now I will probably 
continue setting up pipes in the shell by methods other than the coproc 
keyword.


Cheers,

Carl