Re: Examples of concurrent coproc usage?

[Zachary Santer <zsanter@gmail.com>]

On Sat, Apr 13, 2024 at 2:45 PM Chet Ramey <chet.ramey@case.edu> wrote:
>
> On 4/8/24 11:44 PM, Zachary Santer wrote:
>
> > The fact that the current implementation allows the coproc fds to get
> > into process substitutions is a little weird to me. A process
> > substitution, in combination with exec, is kind of the one other way
> > to communicate with background processes through fds without using
> > FIFOs. I still have to close the coproc fds there myself, right now.
>
> So are you advocating for the shell to close coproc file descriptors
> when forking children for command substitutions, process substitutions,
> and subshells, in addition to additional coprocs? Right now, it closes
> coproc file descriptors when forking subshells.

Yes. I couldn't come up with a way that letting the coproc fds into
command substitutions could cause a problem, in the same sense that
letting them into regular (  ) subshells doesn't seem like a problem.
That bit is at least good for my arbitrary standard of "consistency,"
though.

At least in my use case, trying to use the coproc file descriptors
directly in a pipeline forced the use of a process substitution,
because I needed the coproc fds accessible in the second segment of
what would've been a three-segment pipeline. (Obviously, I'm using
'shopt -s lastpipe' here.) I ultimately chose to do 'exec {fd}> >(
command )' and redirect from one command within the second segment
into ${fd} instead of ending the second segment with '> >( command );
wait "${?}"'. In the first case, you have all the same drawbacks as
allowing the coproc fds into a subshell forked with &. In the second
case, it's effectively the same as allowing the coproc fds into the
segments of a pipeline that become subshells. I guess that would be a
concern if the segment of the pipeline in the parent shell closes the
fds to the coproc while the pipeline is still executing. That seems
like an odd thing to do, but okay.

Now that I've got my own fds that I'm managing myself, I've turned
that bit of code into a plain, three-segment pipeline, at least for
now.

> > Consider the following situation: I've got different kinds of
> > background processes going on, and I've got fds exec'd from process
> > substitutions, fds from coprocs,
>
> If you have more than one coproc, you have to manage all this yourself
> already.

Not if we manage to convince you to turn MULTIPLE_COPROCS=1 on by
default. Or if someone builds bash that way for themselves.

On Sat, Apr 13, 2024 at 2:51 PM Chet Ramey <chet.ramey@case.edu> wrote:
>
> On 4/9/24 10:46 AM, Zachary Santer wrote:
>
> >> If you want two processes to communicate (really three), you might want
> >> to build with the multiple coproc support and use the shell as the
> >> arbiter.
> >
> > If you've written a script for other people than just yourself,
> > expecting all of them to build their own bash install with a
> > non-default preprocessor directive is pretty unreasonable.
>
> This all started because I wasn't comfortable with the amount of testing
> the multiple coprocs code had undergone. If we can get more people to
> test these features, there's a better chance of making it the default.
>
> > The part that I've been missing this whole time is that using exec
> > with the fds provided by the coproc keyword is actually a complete
> > solution for my use case, if I'm willing to close all the resultant
> > fds myself in background processes where I don't want them to go.
> > Which I am.
>
> Good deal.
>
> > Whether the coproc fds should be automatically kept out of most kinds
> > of subshells, like it is now; or out of more kinds than currently; is
> > kind of beside the point to me now.
>
> Sure, but it's the potential for deadlock that we're trying to reduce.

I hesitate to say to just set MULTIPLE_COPROCS=1 free and wait for
people to complain. I'm stunned at my luck in getting Carl Edquist's
attention directed at this. Hopefully there are other people who
aren't subscribed to this email list who are interested in using this
functionality, if it becomes more fully implemented.

> > But, having a builtin to ensure
> > the same behavior is applied to any arbitrary fd might be useful to
> > people, especially if those fds get removed from process substitutions
> > as well.
>
> What does this mean? What kind of builtin? And what `same behavior'?

Let's say it's called 'nosub', and takes fd arguments. It would make
the shell take responsibility for keeping those fds out of subshells.
Perhaps it could take a -u flag, to make it stop keeping the fd
arguments out of subshells. That would be a potential way to get bash
to quit closing coproc fds in subshells, as the user is declaring that
s/he is now responsible for those fds. Still a separate matter from
whether those fds get closed automatically in the parent shell at any
given point.

People who exec fds have to know to take responsibility for everywhere
they go, right now. Exec'ing fds like this is bound to be more common
than using coprocs, as is, I assume, exec'ing fds to process
substitutions. If there are benefits to keeping coproc fds out of
subshells like bash attempts to do, the same benefits would apply if
bash offers to take the burden of keeping other, user-specified fds
out of subshells.