From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 95A061F44D for ; Tue, 12 Mar 2024 03:35:16 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=QwvR9Rji; dkim-atps=neutral Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id CCF4538582A6 for ; Tue, 12 Mar 2024 03:35:15 +0000 (GMT) Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) by sourceware.org (Postfix) with ESMTPS id 5C5073858CD1 for ; Tue, 12 Mar 2024 03:34:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 5C5073858CD1 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 5C5073858CD1 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::130 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710214496; cv=none; b=tl9CEltPmpWwr5ZBhTfGY+ABXtwpa3Ewgup8XyafFclxP1dD99MpsdtKo82MiiOmxLiEARmMmq4S5EmhHEsMVm1x7gDM1RQIwacBBNWz3PfewxbekScagataseMtq1oJqE+0bRqhT+ERdEuJN4MzdEy8HFXLaslSoa3hXnVQUSQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710214496; c=relaxed/simple; bh=hyfSJeylgPFVv2IQQq21/9j4hLviIj/ahEweF8oxyWY=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=JqyY/bBvwyaX/GvWu6e7T0BuF7rsIJN6qJgiq2Y9hadFjBFO//QrtuEN+weS6n+xzsyve+tlTGOPujEpawZKZKewaaSHjOKKDTUEcCGeNXfVWYzOZda5W7hZ0jaCY1NbG8bkcyrqJ3GQl/OAbHB0YcrhIFBUSSI1RXWlIKtTqhA= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-512ed314881so4469488e87.2 for ; Mon, 11 Mar 2024 20:34:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710214492; x=1710819292; darn=sourceware.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=vt7uqc1VpyGPUHmGJAeponKpXPCfGPqSNPgGRpgk6bA=; b=QwvR9RjiXHkrodC9K9dZl+GlUKL2ShybZrL3//Am3zYKef1e0foJeXNB3yq/99LIXC zMXwDQnFGWmS0FHY7ckHp+d3cvx+icBThujHLKZjvV1YTc9D4+wqkJCeRbnpXMjZW63g nyQYN11UMefbcU11b/bMML1i0E0LY/ABCDbtX7jQuv9oLij8DaaHFULVCyQhgDKtb8E6 2Na6mWlbVQUo3coU9i2FiqktOLaooA4QJruJ7Ev8mJbGgmIejvgdsz6RnpcSwovJl00v WhiZh/Skp34T/LG6Ks4FqXkgr4dW29ifWE1Vn66lORGO3NMDWHERvjZ02ydmNi0XvZzy kXRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710214492; x=1710819292; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vt7uqc1VpyGPUHmGJAeponKpXPCfGPqSNPgGRpgk6bA=; b=lUNt8q14EeHX9g6QVletF2yrlHND/Xzrh/VV9MitTj24o+XoPIBfd7GmM3NX9uvOn/ kEd+vVN/QMXKt1oS087dFjaVtFWAvwR6NRdfRlO65kVbpAYh3Ud2QrRmTvwdSvyWuoiw MOr3DhQFOKOizIhg/wdFCpNDveJO9pT6JVZjq5DJ8ErcWFsDjSBq/gMi8TuUelr7XuxN v+YpG4gc3HMrM0J0935BV+SRHgMUwsOTYbndnjhwNswHYG7Ay9R7lNOlG0RldnNVnkxo Hx+J20DeJCUtCHLWUjMkmsDPl7JuO4xGQ3ctDFMGmcuPKb23wp75BY3HIxEAiLJ84YGO 1zng== X-Gm-Message-State: AOJu0Yw3bjc0scwteJXxsI+Tyj+hp8kDCXnMu0SKjh0TvAsgdJrBYsZ9 3gDbZRrUPupDNA9TF4capyTn5q8mp+7FeHnZs6G8pX+FARuciPo6EiffKuSElWLkGcaN7K9e26D dqv7zpDlyOu/qHMylaNMdFssgF3k= X-Google-Smtp-Source: AGHT+IGAHt92QiRHyQ+SHBhmGD2NzBGChLpHStF0UV2uuBb2WQ5dJznbzPGXUrMnJF69ZQcn5/sER4x4/TOWiLk2Uqw= X-Received: by 2002:a05:6512:3456:b0:512:da79:91a with SMTP id j22-20020a056512345600b00512da79091amr4502045lfr.46.1710214492340; Mon, 11 Mar 2024 20:34:52 -0700 (PDT) MIME-Version: 1.0 References: <9831afe6-958a-fbd3-9434-05dd0c9b602a@draigBrady.com> <317fe0e2-8cf9-d4ac-ed56-e6ebcc2baa55@cs.wisc.edu> <8c490a55-598a-adf6-67c2-eb2a6099620a@cs.wisc.edu> In-Reply-To: <8c490a55-598a-adf6-67c2-eb2a6099620a@cs.wisc.edu> From: Zachary Santer Date: Mon, 11 Mar 2024 23:34:39 -0400 Message-ID: Subject: Re: RFE: enable buffering on null-terminated data To: Carl Edquist Cc: libc-alpha@sourceware.org, coreutils@gnu.org, p@draigbrady.com Content-Type: multipart/mixed; boundary="0000000000004ef1db06136e57e2" X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+e=80x24.org@sourceware.org --0000000000004ef1db06136e57e2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Mar 11, 2024 at 7:54=E2=80=AFAM Carl Edquist = wrote: > > (In my coprocess management library, I effectively run every coproc with > --output=3DL by default, by eval'ing the output of 'env -i stdbuf -oL env= ', > because most of the time for a coprocess, that's whats wanted/necessary.) Surrounded by 'set -a' and 'set +a', I guess? Now that's interesting. I just added that to a script I have that prints lines output by another command that it runs, generally a build script, to the command line, but updating the same line over and over again. I want to see if it updates more continuously like that. > ... Although, for your example coprocess use, where the shell both > produces the input for the coproc and consumes its output, you might be > able to simplify things by making the producer and consumer separate > processes. Then you could do a simpler 'producer | filter | consumer' > without having to worry about buffering at all. But if the producer and > consumer need to be in the same process (eg they share state and are > logically interdependent), then yeah that's where you need a coprocess fo= r > the filter. Yeah, there's really no way to break what I'm doing into a standard pipelin= e. > (Although given your time output, you might say the performance hit for > unbuffered is not that huge.) We see a somewhat bigger difference, at least proportionally, if we get bash more or less out of the way. See command-buffering, attached. Standard: real 0m0.202s user 0m0.280s sys 0m0.076s Line-buffered: real 0m0.497s user 0m0.374s sys 0m0.545s Unbuffered: real 0m0.648s user 0m0.544s sys 0m0.702s In coproc-buffering, unbuffered output was 21.7% slower than line-buffered output, whereas here it's 30.4% slower. Of course, using line-buffered or unbuffered output in this situation makes no sense. Where it might be useful in a pipeline is when an earlier command in a pipeline might only print things occasionally, and you want those things transformed and printed to the command line immediately. > So ... again in theory I also feel like a null-terminated buffering mode > for stdbuf(1) (and setbuf(3)) is kind of a missing feature. My assumption is that line-buffering through setbuf(3) was implemented for printing to the command line, so its availability to stdbuf(1) is just a useful side effect. In the BUGS section in the man page for stdbuf(1), we see: On GLIBC platforms, specifying a buffer size, i.e., using fully buffered mode will result in undefined operation. If I'm not mistaken, then buffer modes other than 0 and L don't actually work. Maybe I should count my blessings here. I don't know what's going on in the background that would explain glibc not supporting any of that, or stdbuf(1) implementing features that aren't supported on the vast majority of systems where it will be installed. > It may just > be that nobody has actually had a real need for it. (Yet?) I imagine if anybody has, they just set --output=3D0 and moved on. Bash scripts aren't the fastest thing in the world, anyway. --0000000000004ef1db06136e57e2 Content-Type: application/octet-stream; name=command-buffering Content-Disposition: attachment; filename=command-buffering Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_ltnq563m0 IyEvdXNyL2Jpbi9lbnYgYmFzaAoKc2V0IC1vIG5vdW5zZXQgLW8gbm9nbG9iICtvIGJyYWNlZXhw YW5kCnNob3B0IC1zIGxhc3RwaXBlCmV4cG9ydCBMQ19BTEw9J0MuVVRGLTgnCgp0YWJfc3BhY2Vz PTgKCnNlZF9leHByPSdzL1tbOmJsYW5rOl1dKyQvLycKCnRlc3Q9JCcgIFx0TGluZSB3aXRoIHRh YnNcdCB3aHk/XHQgICcKCnJlcGVhdD0iJHsxfSIKCmZvciAoKCBpID0gMDsgaSA8IHJlcGVhdDsg aSsrICkpOyBkbwogIHByaW50ZiAnJXNcbicgIiR7dGVzdH0iCmRvbmUgPiB0YWItaW5wdXQudHh0 CgpwcmludGYgJyVzJyAiU3RhbmRhcmQ6Igp0aW1lIHsKICBzZWQgLS1iaW5hcnkgLS1yZWdleHAt ZXh0ZW5kZWQgLS1leHByZXNzaW9uPSIke3NlZF9leHByfSIgPCB0YWItaW5wdXQudHh0IHwKICAg IGV4cGFuZCAtLXRhYnM9IiR7dGFiX3NwYWNlc30iID4gL2Rldi9udWxsCn0KCnByaW50ZiAnJXMn ICJMaW5lLWJ1ZmZlcmVkOiIKdGltZSB7CiAgc3RkYnVmIC0tb3V0cHV0PUwgLS0gXAogICAgICBz ZWQgLS1iaW5hcnkgLS1yZWdleHAtZXh0ZW5kZWQgLS1leHByZXNzaW9uPSIke3NlZF9leHByfSIg PCB0YWItaW5wdXQudHh0IHwKICAgIHN0ZGJ1ZiAtLW91dHB1dD1MIC0tIFwKICAgICAgICBleHBh bmQgLS10YWJzPSIke3RhYl9zcGFjZXN9IiA+IC9kZXYvbnVsbAp9CgpwcmludGYgJyVzJyAiVW5i dWZmZXJlZDoiCnRpbWUgewogIHN0ZGJ1ZiAtLW91dHB1dD0wIC0tIFwKICAgICAgc2VkIC0tYmlu YXJ5IC0tcmVnZXhwLWV4dGVuZGVkIC0tZXhwcmVzc2lvbj0iJHtzZWRfZXhwcn0iIDwgdGFiLWlu cHV0LnR4dCB8CiAgICBzdGRidWYgLS1vdXRwdXQ9MCAtLSBcCiAgICAgICAgZXhwYW5kIC0tdGFi cz0iJHt0YWJfc3BhY2VzfSIgPiAvZGV2L251bGwKfQo= --0000000000004ef1db06136e57e2--