From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id D3C761F44D for ; Sun, 14 Apr 2024 02:11:32 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=YqDDph1i; dkim-atps=neutral Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 99053385843B for ; Sun, 14 Apr 2024 02:11:27 +0000 (GMT) Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by sourceware.org (Postfix) with ESMTPS id 6208B3858C31 for ; Sun, 14 Apr 2024 02:10:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6208B3858C31 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 6208B3858C31 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::630 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713060615; cv=none; b=Kb9UqOYFrg9CclsOOFOe334lhqiY88Fyl4oj9eT+iroloh4OeM80hejWsFb2euj3kUW/KxOUk07bPN6X9+4xXANov/RP19ESkknGeNUVkQshDIohBzCRsO36pgN3QnVMrneqmXT7gcOOi9XA+Qa+qth6azEzyND5R9I9X+x0/+4= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713060615; c=relaxed/simple; bh=Zat/kD01finoepjkfmxR1leZU6V7qu/YSu+Ixe7jX44=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=P/SO8eNU0ubx5gO137VCY+W0Kx1mZJCwClh1AUoPxZLn3QZw5g4OEZ+cVntonDY7neflbOjNywO3oxMHRzKZcQfiAcBikqhtDNOwIchGqvzQwpDzmLdUkVjHazSwiePQCKyLFNtGMsC1lb0SW6KOm1lolXywC3OiZTCPjZWvR/4= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-a51969e780eso277325266b.3 for ; Sat, 13 Apr 2024 19:10:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713060601; x=1713665401; darn=sourceware.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ksH484fs+xclVZHfTAjlLTSH3tap2EbbE1bPPsgkqZo=; b=YqDDph1iXaMV4gzxoJqnyqCA97Pz8Hvi4n4Hmx4zOPxRBtp4u1ENfKN6LbiqznXY+y FOhU2YW24GvyNm8npccIznVzeEtCBRcFHmR4XagukKQ629c0q74e/AjrSFz4s2UW/fqR ywRYKzodCSzuR4iiydcaS111Pe416/D+NIOZChiVTm66wTs7yQJSRGxO1qR8a09k5QmU 5qzp6C9L4C1gSXhSlkNWyyKZE3a64BFUe6cy0od4Q4zAz3vvVnh6ejpuUhsp3h4Yh7jZ jklsK3JOSORevBxdYX4Q3VCD2qNSQsLlYrxFxVNoF5IcfHUfc4u8hxRQb2C5mPrqIa8W sDcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713060601; x=1713665401; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ksH484fs+xclVZHfTAjlLTSH3tap2EbbE1bPPsgkqZo=; b=ndT3cCXqI10X+oSvhjVENAcJobbAqnMtlk2EmkjnKOHaDx95IlxlJHg28ZGOgpUp6Y jcJq32+pAOEJkyY2PMvIw7XdoBfseQC+ohxpRJPj1HeMuWiKzts/yAVEAVN/BW6Lc0uz frzkGmTgJ2wCpa4OtnfpYgmTjBoZA3LPXS2vkPZSlRktG6FKCr8SMhTziAcibr+knwds DICVentIJSBBBSQEJA9LJfJb11klC2tFrkPMx9KR6sbKhS5bdk7VWMQjWoOjGNgBV7UL vZvTXC5AJPuZJwkw/ecMW9mLEMmnlblBPKgcPA6Wp894sfBPxX3f3b7xAqj8+bZt679E Pzbg== X-Forwarded-Encrypted: i=1; AJvYcCWzxVeF0qgTPXXYapL4pA9D83BXUV0NyfGeKTeS4I3qd4paQZnjEgaf/GTk0+PF6wuIvYFWRQFGMATg3wybTyCkWHvlxd+3569H X-Gm-Message-State: AOJu0YyeAcIKydcyep4XCllRKoaLbArMc4Lr/MQC480N6soqg3wsEeaY 8xCD3cHpHtzny5/rldiF2Cfhh+rHrzyplaoYJ66fbuTpS9/Mj4iDt5VC0Wmt9Vn1I8id8YeeMT+ kSEQ04+9Ot1A/wlxe7O6eKaNsboI= X-Google-Smtp-Source: AGHT+IE3Na/XbJDfsnqR1y/8kCoHz/AFcN9WEw+XsCe+KiN+YT52BYWvDzv4KFNLEE+J4yeO5K0o5I0/eUHB35oadHY= X-Received: by 2002:a17:907:983:b0:a52:19cb:fb48 with SMTP id bf3-20020a170907098300b00a5219cbfb48mr4626084ejc.77.1713060601103; Sat, 13 Apr 2024 19:10:01 -0700 (PDT) MIME-Version: 1.0 References: <9831afe6-958a-fbd3-9434-05dd0c9b602a@draigBrady.com> <317fe0e2-8cf9-d4ac-ed56-e6ebcc2baa55@cs.wisc.edu> <8c490a55-598a-adf6-67c2-eb2a6099620a@cs.wisc.edu> <88a67f36-2a56-a838-f763-f55b3073bb50@lando.namek.net> <2791ad90-a871-474d-89dd-bc6b20cdd1f2@case.edu> <08f67ddd-486e-4498-934d-c823be42dcab@case.edu> <5b9ebc45-d33d-4128-9ddb-574072edab01@case.edu> In-Reply-To: <5b9ebc45-d33d-4128-9ddb-574072edab01@case.edu> From: Zachary Santer Date: Sat, 13 Apr 2024 22:09:48 -0400 Message-ID: Subject: Re: Examples of concurrent coproc usage? To: chet.ramey@case.edu Cc: Martin D Kealey , Carl Edquist , bug-bash , libc-alpha@sourceware.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org On Sat, Apr 13, 2024 at 2:45=E2=80=AFPM Chet Ramey wr= ote: > > On 4/8/24 11:44 PM, Zachary Santer wrote: > > > The fact that the current implementation allows the coproc fds to get > > into process substitutions is a little weird to me. A process > > substitution, in combination with exec, is kind of the one other way > > to communicate with background processes through fds without using > > FIFOs. I still have to close the coproc fds there myself, right now. > > So are you advocating for the shell to close coproc file descriptors > when forking children for command substitutions, process substitutions, > and subshells, in addition to additional coprocs? Right now, it closes > coproc file descriptors when forking subshells. Yes. I couldn't come up with a way that letting the coproc fds into command substitutions could cause a problem, in the same sense that letting them into regular ( ) subshells doesn't seem like a problem. That bit is at least good for my arbitrary standard of "consistency," though. At least in my use case, trying to use the coproc file descriptors directly in a pipeline forced the use of a process substitution, because I needed the coproc fds accessible in the second segment of what would've been a three-segment pipeline. (Obviously, I'm using 'shopt -s lastpipe' here.) I ultimately chose to do 'exec {fd}> >( command )' and redirect from one command within the second segment into ${fd} instead of ending the second segment with '> >( command ); wait "${?}"'. In the first case, you have all the same drawbacks as allowing the coproc fds into a subshell forked with &. In the second case, it's effectively the same as allowing the coproc fds into the segments of a pipeline that become subshells. I guess that would be a concern if the segment of the pipeline in the parent shell closes the fds to the coproc while the pipeline is still executing. That seems like an odd thing to do, but okay. Now that I've got my own fds that I'm managing myself, I've turned that bit of code into a plain, three-segment pipeline, at least for now. > > Consider the following situation: I've got different kinds of > > background processes going on, and I've got fds exec'd from process > > substitutions, fds from coprocs, > > If you have more than one coproc, you have to manage all this yourself > already. Not if we manage to convince you to turn MULTIPLE_COPROCS=3D1 on by default. Or if someone builds bash that way for themselves. On Sat, Apr 13, 2024 at 2:51=E2=80=AFPM Chet Ramey wr= ote: > > On 4/9/24 10:46 AM, Zachary Santer wrote: > > >> If you want two processes to communicate (really three), you might wan= t > >> to build with the multiple coproc support and use the shell as the > >> arbiter. > > > > If you've written a script for other people than just yourself, > > expecting all of them to build their own bash install with a > > non-default preprocessor directive is pretty unreasonable. > > This all started because I wasn't comfortable with the amount of testing > the multiple coprocs code had undergone. If we can get more people to > test these features, there's a better chance of making it the default. > > > The part that I've been missing this whole time is that using exec > > with the fds provided by the coproc keyword is actually a complete > > solution for my use case, if I'm willing to close all the resultant > > fds myself in background processes where I don't want them to go. > > Which I am. > > Good deal. > > > Whether the coproc fds should be automatically kept out of most kinds > > of subshells, like it is now; or out of more kinds than currently; is > > kind of beside the point to me now. > > Sure, but it's the potential for deadlock that we're trying to reduce. I hesitate to say to just set MULTIPLE_COPROCS=3D1 free and wait for people to complain. I'm stunned at my luck in getting Carl Edquist's attention directed at this. Hopefully there are other people who aren't subscribed to this email list who are interested in using this functionality, if it becomes more fully implemented. > > But, having a builtin to ensure > > the same behavior is applied to any arbitrary fd might be useful to > > people, especially if those fds get removed from process substitutions > > as well. > > What does this mean? What kind of builtin? And what `same behavior'? Let's say it's called 'nosub', and takes fd arguments. It would make the shell take responsibility for keeping those fds out of subshells. Perhaps it could take a -u flag, to make it stop keeping the fd arguments out of subshells. That would be a potential way to get bash to quit closing coproc fds in subshells, as the user is declaring that s/he is now responsible for those fds. Still a separate matter from whether those fds get closed automatically in the parent shell at any given point. People who exec fds have to know to take responsibility for everywhere they go, right now. Exec'ing fds like this is bound to be more common than using coprocs, as is, I assume, exec'ing fds to process substitutions. If there are benefits to keeping coproc fds out of subshells like bash attempts to do, the same benefits would apply if bash offers to take the burden of keeping other, user-specified fds out of subshells.