From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=0.6 required=3.0 tests=BODY_8BITS,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 385141F44D for ; Mon, 15 Apr 2024 17:00:51 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (2048-bit key; secure) header.d=cs.wisc.edu header.i=@cs.wisc.edu header.a=rsa-sha256 header.s=csl-2018021300 header.b=Qtzlzpvd; dkim-atps=neutral Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 347AD3858427 for ; Mon, 15 Apr 2024 17:00:48 +0000 (GMT) Received: from smtpout2.cs.wisc.edu (smtpout2.cs.wisc.edu [128.105.6.54]) by sourceware.org (Postfix) with ESMTPS id 9CC603858D35 for ; Mon, 15 Apr 2024 17:00:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 9CC603858D35 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=cs.wisc.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cs.wisc.edu ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 9CC603858D35 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=128.105.6.54 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713200424; cv=none; b=bjE3gS/gD41qUAPsfnarINm3AMrs5oGnMSmeCbMJGCQWNDALHEqSYCMVEgbWmNuHvjt655rmeTM/Lo0a0pwyHn7KT/SWeFjgkCKlmbuz61Mk8/soeMRjaD+ujbpJl6k5VK3Gfy1bchxUzdLRGnpI38JTeVBrCKHym3ewpH8+yys= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713200424; c=relaxed/simple; bh=h5FBxt8xjuHfwSxTJUtNKP1BP5ADMIAkPVQz4OgFQXs=; h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version; b=qfH8Bc8eBFkMLvRO+uP3RIybxoISWQDHOjDwxEGNOI8d/dI0VCE+ulVaPdUeALGom5xH3Z8oFZP5D5yoKuv/6KWLXCznNt4bzscgwF9GjJ4R1X2K0mzfzkRlJbOu1Alrf+6SLqLiu3BkN3g5t7gx8U0coOmxrMT0Y53kfgf0CRU= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from alumni.cs.wisc.edu (alumni.cs.wisc.edu [128.105.2.11]) by flint.cs.wisc.edu (8.14.7/8.14.4) with ESMTP id 43FGxWjW019534; Mon, 15 Apr 2024 11:59:32 -0500 DKIM-Filter: OpenDKIM Filter v2.11.0 flint.cs.wisc.edu 43FGxWjW019534 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.wisc.edu; s=csl-2018021300; t=1713200373; bh=1ciibb0PmC8nlIbkr5buJEfulRNwsLBX5cO2/Vf8D8E=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=Qtzlzpvd54BWNEpCadL0AR04uhlUyGOgb3f0tl25+4QKo4DXmnIVUHUw0sxwlOmZZ me8rCWlEoeVAK9pqXRP5NP0GfkXIV3urHTeynZEfTIinLmTpVipn1JIpI6r9e8W6Go 1Iozc0U596uzJCfaBjgIUeF0dHUTCYM3HY2w1x4uTUcHM3c+AORbii7cesa/cjHdOo 0ObCpMCMiHZ92Tj3LNe0/JUvkhTpxAIHVKmLyRwUDKAAsrtWRhz+PoohD6KnfuPGnW +qW+FnfO9xyRqlapc6H7g0piFm3m3JdkLA/Bko24cJ57Xav48H2caOT5JLLHT2f0J5 pLy/Raqek5yBQ== Received: from localhost (localhost.localdomain [127.0.0.1]) by alumni.cs.wisc.edu (Postfix) with ESMTP id 8E82D1E0810; Mon, 15 Apr 2024 11:59:31 -0500 (CDT) Date: Mon, 15 Apr 2024 12:01:16 -0500 (CDT) From: Carl Edquist To: Chet Ramey cc: Martin D Kealey , Zachary Santer , bug-bash , libc-alpha@sourceware.org Subject: Re: Examples of concurrent coproc usage? In-Reply-To: <32bd9e76-24bc-4206-aa8a-8bcc817228b1@case.edu> Message-ID: References: <9831afe6-958a-fbd3-9434-05dd0c9b602a@draigBrady.com> <317fe0e2-8cf9-d4ac-ed56-e6ebcc2baa55@cs.wisc.edu> <8c490a55-598a-adf6-67c2-eb2a6099620a@cs.wisc.edu> <88a67f36-2a56-a838-f763-f55b3073bb50@lando.namek.net> <2791ad90-a871-474d-89dd-bc6b20cdd1f2@case.edu> <32bd9e76-24bc-4206-aa8a-8bcc817228b1@case.edu> MIME-Version: 1.0 Content-Type: multipart/mixed; BOUNDARY=-146376107530933654417131864423129 Content-ID: <89a10d2f-1ead-d2ed-7e87-6d930522c02c@cs.wisc.edu> X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---146376107530933654417131864423129 Content-Type: text/plain; CHARSET=utf-8; format=flowed Content-Transfer-Encoding: 8BIT Content-ID: <0fbbc271-d6ca-7d1a-5e5d-e801c0475c4a@cs.wisc.edu> On Sat, 13 Apr 2024, Chet Ramey wrote: > The original intent was to allow the shell to drive a long-running > process that ran more-or-less in parallel with it. Look at > examples/scripts/bcalc for an example of that kind of use. Thanks for mentioning this example. As you understand, this model use case does not require closing the coproc fds when finished, because they will be closed implicitly when the shell exits. (As bcalc itself admits.) And if the coproc is left open for the lifetime of the shell, the alternate behavior of deferring the coproc deallocation (until both coproc fds are closed) would not require anything extra from the user. The bcalc example does close both coproc fds though - both at the end, and whenever it resets. And so in this example (which as you say, was the original intent), the user is already explicitly closing both coproc fds explicitly; so the alternate deferring behavior would not require anything extra from the user here either. ... Yet another point brought to light by the bcalc example relates to the coproc pid variable. The reset() function first closes the coproc pipe fds, then sleeps for a second to give the BC coproc some time to finish. An alternative might be to 'wait' for the coproc to finish (likely faster than sleeping for a second). But you have to make and use your $coproc_pid copy rather than $BC_PID directly, because 'wait $BC_PID' may happen before or after the coproc is reaped and BC_PID is unset. (As the bcalc author seems to understand.) So in general the coproc *_PID variable only seems usable for making a copy when starting the coproc. The info page has the following: > The process ID of the shell spawned to execute the coprocess is > available as the value of the variable 'NAME_PID'. The 'wait' builtin > command may be used to wait for the coprocess to terminate. But it seems to me that the copy is necessary, and it is never reliable to run 'wait $NAME_PID'. Because any time the shell is in a position to wait for the coproc to finish, by that time it's going to be a race whether or not NAME_PID is still set. So this is another example for me of why it would be handy if coproc deallocation were deferred until explicit user action (closing both coproc fds, or unsetting the coproc variable). That way ${NAME[@]} and $NAME_PID could reliably be used directly without having to make copies. Anyway, just food for thought if down the line you make a shell option for coproc deallocation behavior. >> Now, if you built bash with multiple coproc support, I would have >> expected you could still rig this up, by doing the redirection work >> explicitly yourself. Something like this: >> >>     coproc UP   { stdbuf -oL tr a-z A-Z; } >>     coproc DOWN { stdbuf -oL tr A-Z a-z; } >> >>     # make user-managed backup copies of coproc fds >>     exec {up_r}<&${UP[0]} {up_w}>&${UP[1]} >>     exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]} >> >>     coproc THREEWAY { tee /dev/fd/$up_w  /dev/fd/$down_w; } >> >> >> But the above doesn't actually work, as it seems that the coproc shell >> (THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even >> the user-managed ones explicitly copied with exec. > > File descriptors the user saves with exec redirections beyond [0-2] are > set to close-on-exec. POSIX makes that behavior unspecified, but bash > has always done it. Ah, ok, thanks. I believe I found where this gets set in do_redirection_internal() in redir.c. (Whew, a big function.) As far as I can tell the close-on-exec state is "duplicated" rather than set unconditionally. That is, the new fd in a redirection is only set close-on-exec if the source is. (Good, because in general I rely on redirections to be made available to external commands.) But apparently coproc marks its pipe fds close-on-exec, so there's no way to expose manual copies of these fds to external commands. So, that explains the behavior I was seeing ... It's just a bit too bad for anyone that actually wants to do more elaborate coproc interconnections with manual redirections, as they're limited to shell builtins. ... I might pose a question to ponder about this though: With the multi-coproc support code, is it still necessary to set the coproc pipe fds close-on-exec? (If, perhaps, they're already getting explicitly closed in the right places.) Because if the new coproc fds are _not_ set close-on-exec, in general that would allow the user to do manual redirections for external commands (eg tee(1) or paste(1)) to communicate with multiple coproc fds together. > Shells don't offer any standard way to modify the state of that flag, > but there is the `fdflags' loadable builtin you can experiment with to > change close-on-exec. Thanks for the tip. It's nice to know there is a workaround to leave copies of the coproc fds open across exec; though for now I will probably continue setting up pipes in the shell by methods other than the coproc keyword. Cheers, Carl ---146376107530933654417131864423129--