From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 3CA3D1F44D for ; Sat, 20 Apr 2024 23:10:13 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (2048-bit key; secure) header.d=cs.wisc.edu header.i=@cs.wisc.edu header.a=rsa-sha256 header.s=csl-2018021300 header.b=0wlUtHlK; dkim-atps=neutral Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 543873858D3C for ; Sat, 20 Apr 2024 23:10:12 +0000 (GMT) Received: from smtpout2.cs.wisc.edu (smtpout2.cs.wisc.edu [128.105.6.54]) by sourceware.org (Postfix) with ESMTPS id 626E33858D34 for ; Sat, 20 Apr 2024 23:09:51 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 626E33858D34 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=cs.wisc.edu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=cs.wisc.edu ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 626E33858D34 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=128.105.6.54 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713654594; cv=none; b=m9dU4jEN0ArJAVTp0Hrjzj8TDu8pdQeR1tXspio5R0QnRPFhzVBF1QsveqMaCuUboel0zfaZ5dmmL7dV9k5M8dqQIirGihP94uV+3PfALOXSZgt+Fe0pTwYuDlDOQ4mSvVgA/GKaQgQdeqqpYKAhigkprkNMfqYXMTg3nxCiSUY= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1713654594; c=relaxed/simple; bh=lnAKsMdp5X+a8EetAbUnqCy9wuu91O61jTAfn4rmz20=; h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version; b=G/Dnx5AuMao+GzbAifIxKpPtMPb3cftCQA1VJxJIjIfvWCb9eAPe7jnmN9cialKs5PF6SxI15qYHHdyvvbbubvpgD/TIiQXQLOzpRHzxsUMXoxeO333T80oo88J7Dq2KnWTyPpnaT0ULjE/GCElDCFHkXlN3z6Ar9zV++Zd1XZ0= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from alumni.cs.wisc.edu (alumni.cs.wisc.edu [128.105.2.11]) by flint.cs.wisc.edu (8.14.7/8.14.4) with ESMTP id 43KN9iog011814; Sat, 20 Apr 2024 18:09:44 -0500 DKIM-Filter: OpenDKIM Filter v2.11.0 flint.cs.wisc.edu 43KN9iog011814 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.wisc.edu; s=csl-2018021300; t=1713654584; bh=ywGq7L6XHztKiZrfueQJzyCSaTva1i7RfSxK2ZbtH2U=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=0wlUtHlKY2fMnDhrkzOs4wqfHdSqhYaRXM0f+LjnMZ3eNPIRi0HG9DRM5lqPrg4xi zislul8DSOvA/B+W9vVK1uBEppfJEKS3DHGe/JT/pSt516suPkN5xjZbzWtfj9PZBx ZqB91rxxu1oh5dtoQkwn5UuyEc3wy5nf9XCKRd6whSt47GHGMMx+WTMOH6gUH2hv2h xw8DSwFiZyRvx1RNoBE737SFjGIkZeCtHf/CvHcZV95dlengesYwZWTYtDvmOZTda2 QHX+UFyqpnm5uMvhgkmtH9nb6ytmM6+napsypGfIT/trVDOPEwgA14PDBrbU2Kpu0C wn8T+CWKInCcA== Received: from localhost (localhost.localdomain [127.0.0.1]) by alumni.cs.wisc.edu (Postfix) with ESMTP id BEF421E08CC; Sat, 20 Apr 2024 18:09:43 -0500 (CDT) Date: Sat, 20 Apr 2024 18:11:31 -0500 (CDT) From: Carl Edquist To: Chet Ramey cc: Zachary Santer , bug-bash , libc-alpha@sourceware.org Subject: Re: Examples of concurrent coproc usage? In-Reply-To: <4625270d-c8f6-42d1-afa0-fafb7a33571e@case.edu> Message-ID: <0488843f-339a-f25e-a3d2-cb0afeec91d1@cs.wisc.edu> References: <9831afe6-958a-fbd3-9434-05dd0c9b602a@draigBrady.com> <317fe0e2-8cf9-d4ac-ed56-e6ebcc2baa55@cs.wisc.edu> <8c490a55-598a-adf6-67c2-eb2a6099620a@cs.wisc.edu> <88a67f36-2a56-a838-f763-f55b3073bb50@lando.namek.net> <2791ad90-a871-474d-89dd-bc6b20cdd1f2@case.edu> <86c3765e-e29d-48d5-b468-3f20b59916b2@case.edu> <6bcbd956-7296-7150-765f-63318a425d1b@cs.wisc.edu> <4625270d-c8f6-42d1-afa0-fafb7a33571e@case.edu> MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=us-ascii X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org On Tue, 16 Apr 2024, Chet Ramey wrote: > The bigger concern was how to synchronize between the processes, but > that's something that the script writer has to do on their own. Right. It can be tricky and depends entirely on what the user's up to. > My concern was always coproc fds leaking into other processes, > especially pipelines. If someone has a coproc now and is `messy' about > cleaning it up, I feel like there's the possibility of deadlock. I get where you're coming from with the concern. I would welcome being shown otherwise, but as far as I can tell, deadlock is a ghost of a concern once the coproc is dead. Maybe it helps to step through it ... - First, where does deadlock start? (In the context of pipes) I think the answer is: When there is a read or write attempted on a pipe that blocks (indefinitely). - What causes a read or a write on a pipe to block? A pipe read blocks when a corresponding write-end is open, but there is no data available to read. A pipe write blocks when a corresponding read-end is open, but the pipe is full. - Are the coproc's corresponding ends of the shell's pipe fds open? Well, not if the coproc is really dead. - Will a read or write ever be attempted? If the shell's stray coproc fds are left open, sure they will leak into pipelines too - but since they're forgotten, in theory no command will actually attempt to use them. - What if a command attempts to use these stray fds anyway, by mistake? If the coproc is really dead, then its side of the pipe fds will have been closed. Thus read/write attempts on the fds on the shell's side (either from the shell itself, or from commands / pipelines that the fds leaked into) WILL NOT BLOCK, and thus will not result in deadlock. (A read attempt will hit EOF, a write attempt will get SIGPIPE/EPIPE.) HOPEFULLY that is enough to put any reasonable fears of deadlock to bed - at least in terms of the shell's leaked fds leading to deadlock. - But what if the _coproc_ leaked its pipe fds before it died? At this point I think perhaps we get into what you called a "my arm hurts when I do this" situation. It kind of breaks the whole coproc model: if the stdin/stdout of a coproc are still open by one of the coproc's children, then I might say the coproc is not really dead. But anyway I want to be a good sport, for completeness. An existing use case that would lead to trouble would perhaps have to look something like this: The shell sends a quit command to a coproc, without closing the shell's coproc fds. The coproc has a child, then exits. The coproc (parent) is dead. The coproc's child has inherited the coproc's pipe fds. The script author _expects_ that the coproc parent will exit, and expects that this will trigger the old behavior, that the shell will automatically close its fds to the coproc parent. Thus the author _expects_ that the coproc exiting will, indirectly but automatically, cause any blocked reads/writes on stdin/stdout in the coproc's child to stop blocking. Thus the author _expects_ the coproc's child to promptly complete, even though its output _will not be consumable_ (because the author _expects_ that its stdout will be attached to a broken pipe). But [here's where the potential problem starts] with the new deferring behavior, the shell's coproc fds are not automatically closed, and thus the coproc's _child_ does not stop blocking, and thus the author's short-lived expectations for this coproc's useless child are dashed to the ground, while that child is left standing idle until the cows come home. (That is, until the shell exits.) It really seems like a contrived and senseless scenario, doesn't it? (Even to me!) [And an even more far-fetched scenario: a coproc transmits copies of its pipe fds to another process over a unix socket ancillary message (SCM_RIGHTS), instead of to a child by inheritance. The rest of the story is the same, and equally senseless.] > But I don't know how extensively they're used, or all the use cases, so > I'm not sure how likely it is. I've learned there are users who do > things with shell features I never imagined. (People wanting to use > coprocs without the shell as the arbiter, for instance. :-) ) Hehe... Well, yeah, once you gift-wrap yourself a friendly, reliable interface and have the freedom to play with it to your heart's content - you find some fun things to do with coprocesses. (Much like regular shell pipelines.) I get your meaning though - without knowing all the potential uses, it's hard to say with absolute certainty that no user will be negatively affected by a new improvement or bug fix. >> [This is a common model for using coprocs, by the way, where an >> auxiliary coprocess is left open for the lifetime of the shell session >> and never explicitly closed. When the shell session exits, the fds are >> closed implicitly by the OS, and the coprocess sees EOF and exits on >> its own.] > > That's one common model, yes. Another is that the shell process > explicitly sends a close or shutdown command to the coproc, so > termination is expected. Right, but here also (after sending a quit command) the conclusion is the same as my point just below - that if the user is expecting the coproc to terminate, and expecting the current behavior that as a result the coproc variable will go away automatically, then that variable is as good as forgotten to the user. >> If a user expects the coproc variable to go away automatically, that >> user won't be accessing a still-open fd from that variable for >> anything. > > I'm more concerned about a pipe with unread data that would potentially > cause problems. I suppose we just need more testing. If I understand you right, you are talking about a scenario like this: - a coproc writes to its output pipe - the coproc terminates - the shell leaves its fd for the read end of this pipe open - there is unread data left sitting in this pipe - [theoretical concern here] Is that right? I can't imagine this possibly leading to deadlock. Either (1) the user has forgotten about this pipe, and never attempts to read from it, or (2) the user attempts to read from this pipe, returning some or all of the data, and possibly hitting EOF, but in any case DOES NOT BLOCK. (I'm sorry if this is basically restating what I've already said earlier.) > That's more of a "my arm hurts when I do this" situation. If a script > opened 500 fds using exec redirection, resource exhaustion would be > their own responsibility. Ha, good! [I had a small fear that fd exhaustion might have been your actual concern.] >> Meanwhile, the bash man page does not specify the shell's behavior for >> when a coproc terminates, so you might say there's room for >> interpretation and the new deferring behavior would not break any >> promises. > > I could always enable it in the devel branch and see what happens with > the folks who use that. It would be three years after any release when > distros would put it into production anyway. Oh, fun :) >> But since you mention it, writing to a broken pipe is still >> semantically meaningful also. (I would even say valid.) In the >> typical case it's expected behavior for a process to get killed when it >> attempts this and shell pipeline programming is designed with this in >> mind. > > You'd be surprised at how often I get requests to put in an internal > SIGPIPE handler to avoid problems/shell termination with builtins > writing to closed pipes. Ah, well, I get it though. It _is_ a bit jarring to see your shell get blown away with something like this - $ exec 9> >(typo) $ ... $ echo >&9 # Boom! So it does not surprise me that you have some users puzzling over it. But FWIW I do think it is the most consistent & correct behavior. Plus, of course, the user can install their own shell handler code for that case, or downgrade the effect to a non-fatal error with $ trap '' SIGPIPE >> So even for write attempts, you introduce uncertain behavior by >> automatically closing the fds, when the normal, predictable, valid >> thing would be to die by SIGPIPE. > > Again, you might be surprised at how many people view that as a bug in > the shell. I'm not terribly surprised, since at first (before reasoning about it) the behavior is admittedly alarming. ("What happened to my terminal?!?!") But I'd argue the alternative is worse, because then it's an unpredictable race between SIGPIPE (which they're complaining about) and EBADF. > I think we're talking about our different interpretations of `invalid' > (EBADF as opposed to EPIPE/SIGPIPE). Right - just explaining; I think by now we are on the same page. > My original intention for the coprocs (and Korn's from whence they came) > was that the shell would be in the middle -- it's another way for the > shell to do IPC. And coprocesses are great for this, too! It's just that external commands in a sense are extensions of the shell. The arms and legs, you might say, for doing the heavy lifting. Carl