From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 97DE31F44D for ; Tue, 9 Apr 2024 15:57:50 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (2048-bit key; secure) header.d=cs.wisc.edu header.i=@cs.wisc.edu header.a=rsa-sha256 header.s=csl-2018021300 header.b=0blRsn4W; dkim-atps=neutral Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DC83F3846411 for ; Tue, 9 Apr 2024 15:57:49 +0000 (GMT) Received: from smtpout2.cs.wisc.edu (smtpout2.cs.wisc.edu [128.105.6.54]) by sourceware.org (Postfix) with ESMTPS id C44C3385840B for ; Tue, 9 Apr 2024 15:57:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C44C3385840B Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=cs.wisc.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cs.wisc.edu ARC-Filter: OpenARC Filter v1.0.0 sourceware.org C44C3385840B Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=128.105.6.54 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1712678246; cv=none; b=nnlGsMYeBwoynbDrtSsWJJUCxE7O0S3NLQXc3W85Q2icbSriDVIaT10MQ5ax4f56hK1yrtCLEh1jf0LlUhz+leKAbbcnXljCHUFiPgdANuJCMVq17Yzos4tCKAWqjPju1Imh1CwOxnSQkRhLTRgdumXmtdjTYO4N4J8B9XjxGhw= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1712678246; c=relaxed/simple; bh=cDl9T2hAeXAtcVTLzMZ9BKlk3Aki/16XtmP+OV/iKI4=; h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version; b=DAluwQIajPQoElI4F/pcB4mLMm89jFbHdO3rQMsLyvS54an9/94ij2HlgQe5u2CqsSE+JV2pIdc/9b0wxOP8zclUjWi/R+1G0aDr7zKzxswvOd/5QMjTPLe4ccXiq9ivCgatV2+D5+LFJHVLAB+AEsStrlLenNvBZa7qNTwg9CU= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from alumni.cs.wisc.edu (alumni.cs.wisc.edu [128.105.2.11]) by flint.cs.wisc.edu (8.14.7/8.14.4) with ESMTP id 439Fv29p014408; Tue, 9 Apr 2024 10:57:03 -0500 DKIM-Filter: OpenDKIM Filter v2.11.0 flint.cs.wisc.edu 439Fv29p014408 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.wisc.edu; s=csl-2018021300; t=1712678225; bh=hp96wKlWAKsDxS4/spHQbJYyohV7wxkHaMLzyGqjLE0=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=0blRsn4WMYhOsuFuRQL14HMtBlz8iVanmca8D2s+UVlt/IcxtpPfTt0uh9Z5//5TX DQ5LHdabvdz/Hwa7bnuvMHMw/CL07Mz0oxWXB0Lk2+/4eYqtDoDIjuYCg4Pt4wAkJO UpShvbPKhTPtjGIhPIAxWLjfCycuxe4NxmnjiQh9u50XAvhNydDx4z7nl4zohN4v9E aFi9NuzIA6Qf9dH3CgrsnZ7co6V/zEP2NsTJfss+p1bFoJFLHpK/JAVUofIaa/fH1I xM7Ylqg/qCs3iVEpjncQdM0VHuuUavrcFk/dFDJZT6+HOyf/MBE9RyWcbGQcl760nJ 7EPJDh8U2C1sw== Received: from localhost (localhost.localdomain [127.0.0.1]) by alumni.cs.wisc.edu (Postfix) with ESMTP id 3A8C01E0396; Tue, 9 Apr 2024 10:57:01 -0500 (CDT) Date: Tue, 9 Apr 2024 10:58:41 -0500 (CDT) From: Carl Edquist To: Chet Ramey , Martin D Kealey cc: Zachary Santer , bug-bash , libc-alpha@sourceware.org Subject: Re: Examples of concurrent coproc usage? In-Reply-To: Message-ID: References: <9831afe6-958a-fbd3-9434-05dd0c9b602a@draigBrady.com> <317fe0e2-8cf9-d4ac-ed56-e6ebcc2baa55@cs.wisc.edu> <8c490a55-598a-adf6-67c2-eb2a6099620a@cs.wisc.edu> <88a67f36-2a56-a838-f763-f55b3073bb50@lando.namek.net> <2791ad90-a871-474d-89dd-bc6b20cdd1f2@case.edu> MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=us-ascii X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org On 4/4/24 7:23 PM, Martin D Kealey wrote: > I'm somewhat uneasy about having coprocs inaccessible to each other. I > can foresee reasonable cases where I'd want a coproc to utilize one or > more other coprocs. > > In particular, I can see cases where a coproc is written to by one > process, and read from by another. > > Can we at least have the auto-close behaviour be made optional, so that > it can be turned off when we want to do something more sophisticated? With support for multiple coprocs, auto-closing the fds to other coprocs when creating new ones is important in order to avoid deadlocks. But if you're willing to take on management of those coproc fds yourself, you can expose them to new coprocs by making your own copies with exec redirections. But this only "kind of" works, because for some reason bash seems to close all pipe fds for external commands in coprocs, even the ones that the user explicitly copies with exec redirections. (More on that in a bit.) On Mon, 8 Apr 2024, Chet Ramey wrote: > On 4/4/24 7:23 PM, Martin D Kealey wrote: >> I'm somewhat uneasy about having coprocs inaccessible to each other. I >> can foresee reasonable cases where I'd want a coproc to utilize one or >> more other coprocs. > > That's not the intended purpose, Just a bit of levity here - i can picture Doc from back to the future exclaiming, "Marty, it's perfect! You're just not thinking 4th dimensionally!" > so I don't think not fixing a bug to accommodate some future > hypothetical use case is a good idea. That's why there's a warning > message when you try to use more than one coproc -- the shell doesn't > keep track of more than one. > > If you want two processes to communicate (really three), you might want > to build with the multiple coproc support and use the shell as the > arbiter. For what it's worth, my experience is that coprocesses in bash (rigged up by means other than the coproc keyword) become very fun and interesting when you allow for the possibility of communication between coprocesses. (Most of my use cases for coprocesses fall under this category, actually.) The most basic commands for tying multiple coprocesses together are tee(1) and paste(1), for writing to or reading from multiple coprocesses at once. You can do this already with process substitutions like tee >(cmd1) >(cmd2) paste <(cmd3) <(cmd4) My claim here is that there are uses for this where these commands are all separate coprocesses; that is, you'd want to read the output from cmd1 and cmd2 separately, and provide input for cmd3 and cmd4 separately. (I'll try to send some examples in a later email.) Nevertheless it's still crucial to keep the shell's existing coprocess fds out of new coprocesses, otherwise you easily run yourself into deadlock. Now, if you built bash with multiple coproc support, I would have expected you could still rig this up, by doing the redirection work explicitly yourself. Something like this: coproc UP { stdbuf -oL tr a-z A-Z; } coproc DOWN { stdbuf -oL tr A-Z a-z; } # make user-managed backup copies of coproc fds exec {up_r}<&${UP[0]} {up_w}>&${UP[1]} exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]} coproc THREEWAY { tee /dev/fd/$up_w /dev/fd/$down_w; } But the above doesn't actually work, as it seems that the coproc shell (THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even the user-managed ones explicitly copied with exec. As a result, you get back errors like this: tee: /dev/fd/11: No such file or directory tee: /dev/fd/13: No such file or directory That's the case even if you do something more explicit like: coproc UP_AND_OUT { tee /dev/fd/99 99>&$up_w; } the '99>&$up_w' redirection succeeds, showing that the coproc does have access to its backup fd $up_w (*), but apparently the shell closes fd 99 (as well as $up_w) before exec'ing the tee command. Note the coproc shell only does this with pipes; it leaves other user managed fds like files or directories alone. I have no idea why that's the case, and i wonder whether it's intentional or an oversight. But anyway, i imagine that if one wants to use multi coproc support (which requires automatically closing the shell's coproc fds for new coprocs), and wants to set up multiple coprocs to communicate amongst themselves, then the way to go would be explicit redirections. (But again, this requires fixing this peculiar behavior where the coproc shell closes even the user managed copies of pipe fds before exec'ing external commands.) (*) to prove that the coproc shell does have access to $up_w, we can make a shell-only replacement for tee(1) : (actually works) fdtee () { local line fd while read -r line; do for fd; do printf '%s\n' "$line" >&$fd; done; done; } coproc UP { stdbuf -oL tr a-z A-Z; } coproc DOWN { stdbuf -oL tr A-Z a-z; } # make user-managed backup copies of coproc fds exec {up_r}<&${UP[0]} {up_w}>&${UP[1]} exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]} stdout=1 coproc THREEWAY { fdtee $stdout $up_w $down_w; } # save these too, for safe keeping exec {tee_r}<&${THREEWAY[0]} {tee_w}>&${THREEWAY[1]} Then: (actually works) $ echo 'Greetings!' >&$tee_w $ read -u $tee_r plain $ read -u $up_r upped $ read -u $down_r downed $ echo "[$plain] [$upped] [$downed]" [Greetings!] [GREETINGS!] [greetings!] This is a pretty trivial example just to demonstrate the concept. But once you have the freedom to play with it, you find more interesting, useful applications. Of course, for the above technique to be generally useful, external commands need access to these user-managed fds (copied with exec). (I have no idea why the coproc shell closes them.) The shell is crippled when limited to builtins. (I'll try to tidy up some working examples with my coprocess management library this week, for the curious.) Juicy thread hey? I can hardly keep up! :) Carl