From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on starla X-Spam-Level: X-Spam-Status: No, score=0.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_BL_SPAMCOP_NET,SPF_HELO_PASS, SPF_PASS autolearn=no autolearn_force=no version=3.4.6 Received: from nue.mailmanlists.eu (nue.mailmanlists.eu [IPv6:2a01:4f8:1c0c:6b10::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id A1E711F44D for ; Wed, 17 Apr 2024 04:25:31 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; secure) header.d=ml.ruby-lang.org header.i=@ml.ruby-lang.org header.a=rsa-sha256 header.s=mail header.b=pKZ3VGHe; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=jB46yNC/; dkim-atps=neutral Received: from nue.mailmanlists.eu (localhost [127.0.0.1]) by nue.mailmanlists.eu (Postfix) with ESMTP id 248D78432D; Wed, 17 Apr 2024 04:25:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ml.ruby-lang.org; s=mail; t=1713327923; bh=g66nJYw0M87w52+jq9zDiJK0TmEIgqR+A0DmxzKULwQ=; h=Date:References:To:Reply-To:Subject:List-Id:List-Archive: List-Help:List-Owner:List-Post:List-Subscribe:List-Unsubscribe: From:Cc:From; b=pKZ3VGHesCS7tWKjVTCHtBXH7RiH2Ki/6AfK7LUghomzo9W4d4ZZcD82qB8hHajV2 s5T0dwasEE1N74/BUcjLmF222TvmPN4CXbF7rY35R0c2Idfhnzl3MwjN5OI/NKwcyf UI4MJ3H8hoth0PTvA3TWbtvdNfvSoK9wNwUc2bF4= Received: from s.wrqvwxzv.outbound-mail.sendgrid.net (s.wrqvwxzv.outbound-mail.sendgrid.net [149.72.154.232]) by nue.mailmanlists.eu (Postfix) with ESMTPS id B494C842F3 for ; Wed, 17 Apr 2024 04:25:19 +0000 (UTC) Authentication-Results: nue.mailmanlists.eu; dkim=pass (2048-bit key; unprotected) header.d=ruby-lang.org header.i=@ruby-lang.org header.a=rsa-sha256 header.s=s1 header.b=jB46yNC/; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ruby-lang.org; h=from:references:subject:mime-version:content-type: content-transfer-encoding:list-id:to:cc:content-type:from:subject:to; s=s1; bh=Yq0H55a6RNH5oLBTw+z/sBc5dtb2o28Cqig82tL5GGs=; b=jB46yNC/H+jQJqgZneHFSmvOe3s/Ltn7KwIUxTSHF5cm/B3QySM71aREF+rohmxUWc8M C8JPASqL1NV860Fb6pAqGIlvPkOzDWHJKPMuKqftaQmwTi6L/akrRVPvYbWli5KI6Q/eKV Jmgnsla70HHQ7F41CJTzFtVN5HjsliedIRMypt6AksviI7N676kA1XtGTZbjUnxHd6oHoC qvnGk/DuMVV5AH1sPdhgh8hD/NGbCm35/2MFltCBJvCPVwQFwto+qhShfae1asQZMOaBZY 2Ot630n2ZydF3nxXlHx5KgqpHEHC7yTXyiouEq+u9fhPKpjC9IntuXiL/+CsoiiA== Received: by recvd-6b888cd74b-wnss2 with SMTP id recvd-6b888cd74b-wnss2-1-661F4F2D-9 2024-04-17 04:25:17.646357214 +0000 UTC m=+368622.447067908 Received: from herokuapp.com (unknown) by geopod-ismtpd-38 (SG) with ESMTP id cYHQAtALTGSF8qRQis3M9w for ; Wed, 17 Apr 2024 04:25:17.628 +0000 (UTC) Date: Wed, 17 Apr 2024 04:25:17 +0000 (UTC) Message-ID: References: Mime-Version: 1.0 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Feature X-Redmine-Issue-Id: 20215 X-Redmine-Issue-Author: ioquatix X-Redmine-Issue-Priority: Normal X-Redmine-Sender: ioquatix X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-Redmine-MailingListIntegration-Message-Ids: 94139 X-SG-EID: =?us-ascii?Q?u001=2E3QFJPY5gaRwyNXkncYONHM3OeFnb=2FgeOrLDgg3PYIKrq5rvk81iAipQ1q?= =?us-ascii?Q?cdmuT9o5PIWGjXtTDstodNmslT=2F6xfCuIofWR2K?= =?us-ascii?Q?w8msDgVaVLyRvpJRfJhyeq=2FfPW9ZqCW6aq2YMsj?= =?us-ascii?Q?w4EHSlS3b5BtZ6sYMsUX3fSY9SQCwtUmZBBmrt6?= =?us-ascii?Q?hEVPPwnzyd=2FHJqPj3YJ9syTrCCy4gAwTlVUB3lE?= =?us-ascii?Q?9JcPIWy28zjlmPSaIdSAz7M4dz4xn5Ahwa7kYrL?= =?us-ascii?Q?RAzEVVVukuV7GnSXlmmG03OisA=3D=3D?= To: ruby-core@ml.ruby-lang.org X-Entity-ID: u001.I8uzylDtAfgbeCOeLBYDww== Message-ID-Hash: 77MGMULEB6TRVS6XSST6KT6WSB5ESEGD X-Message-ID-Hash: 77MGMULEB6TRVS6XSST6KT6WSB5ESEGD X-MailFrom: bounces+313651-b711-ruby-core=ml.ruby-lang.org@em5188.ruby-lang.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list Reply-To: Ruby developers Subject: [ruby-core:117544] [Ruby master Feature#20215] Introduce `IO#readable?` List-Id: Ruby developers Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: "ioquatix (Samuel Williams) via ruby-core" Cc: "ioquatix (Samuel Williams)" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Issue #20215 has been updated by ioquatix (Samuel Williams). After considering the various use cases I have, I think the easiest to describe problem is knowing whether a connection is still "connected" or not, i.e. whether read will definitely fail or might succeed. I added a full working example of the problem here: . You can try different implementation of `IO#readable?` to see the behaviour. The example demonstrates HTTP/1 persistent connection handling, where the remote server may at any time disconnect. In `server.rb`, it has a 50% chance of disconnecting. `client.rb` makes 10 connections, and tries to use persistent connections. The key problem that I'm trying to address, is that there is no protocol-level mechanism to advertise that the remote server is closing the connection (in contrast, HTTP/2 has such a feature). So, what that means, is in the request loop, when we want to write the request, we want to ensure, with the best effort possible, that the connection is still alive and working. That is the purpose of `IO#readable?` in this context - whether there is a significantly good chance that writing an HTTP request will be successful. In practice, persistent connections may sit in a connection pool for minutes or hours, and thus when you come to write a request, there is no easy operation to check "Is this connection still working?". That is the purpose of `IO#readable?`. Specifically, before writing a request, we check if the connection is still readable. The logic for "Is the connection still readable?" depends on the situation and the underlying IO. As you know there are many different semantics for handling Sockets, Pipes, and so on, and we even provide our own blended semantics in `StringIO`. I'd like to introduce `IO#readable?`, `BasicSocket#readable?` based on `recv_nonblock` and `StringIO#readable?` which is similar to non-blocking `eof?`. In other words, in the case of sockets, `BasicSocket#readable?` is querying the operating system to find out if the TCP connection is still working (i.e. not closed explicitly). It's true that this can be a race condition, for example the TCP reset/shutdown could be delayed or received while writing the request. However, it's still better to prevent writing the request entirely if possible. That's because not all requests are idempotent e.g. POST requests for handling payments. It's much better to know ahead of time that the request will fail because the persistent connection has been shut down, than to find out half way through writing the non-idempotent request. Example output from the client: ``` > bundle exec client.rb Connected to # # Writing request... Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Client is not readable, closing... Reconnecting... Connected to # # Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Client is not readable, closing... Reconnecting... Connected to # # Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Client is not readable, closing... Reconnecting... Connected to # # Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Client is not readable, closing... Reconnecting... Connected to # # Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Client is not readable, closing... Reconnecting... Connected to # # Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Writing request... Client is not readable, closing... Reconnecting... Connected to # # Reading response... Got response: ["HTTP/1.1", 200, "OK", #, #] Hello World Closing client... Exiting. ``` Note that `Client is not readable, closing...` indicates that the client was closed before the request was written, which is the ideal case. ---------------------------------------- Feature #20215: Introduce `IO#readable?` https://bugs.ruby-lang.org/issues/20215#change-107937 * Author: ioquatix (Samuel Williams) * Status: Open ---------------------------------------- There are some cases where, as an optimisation, it's useful to know whether more data is potentially available. We already have `IO#eof?` but the problem with using `IO#eof?` is that it can block indefinitely for sockets. Therefore, code which uses `IO#eof?` to determine if there is potentially more data, may hang. ```ruby def make_request(path = "/") client = connect_remote_host # HTTP/1.0 request: client.write("GET #{path} HTTP/1.0\r\n\r\n") # Read response client.gets("\r\n") # => "HTTP/1.0 200 OK\r\n" # Assuming connection close, there are two things the server can do: # 1. peer.close # 2. peer.write(...); peer.close if client.eof? # <--- Can hang here! puts "Connection closed" # Avoid yielding as we know there definitely won't be any data. else puts "Connection open, data may be available..." # There might be data available, so yield. yield(client) end ensure client&.close end make_request do |client| puts client.read # <--- Prefer to wait here. end ``` The proposed `IO#readable?` is similar to `IO#eof?` but rather than blocking, would simply return false. The expectation is the user will subsequently call `read` which may then wait. The proposed implementation would look something like this: ```ruby class IO def readable? !self.closed? end end class BasicSocket # Is it likely that the socket is still connected? # May return false positive, but won't return false negative. def readable? return false unless super # If we can wait for the socket to become readable, we know that the socket may still be open. result = self.recv_nonblock(1, MSG_PEEK, exception: false) # No data was available - newer Ruby can return nil instead of empty string: return false if result.nil? # Either there was some data available, or we can wait to see if there is data avaialble. return !result.empty? || result == :wait_readable rescue Errno::ECONNRESET # This might be thrown by recv_nonblock. return false end end ``` For `IO` itself, when there is buffered data, `readable?` would also return true immediately, similar to `eof?`. This is not shown in the above implementation as I'm not sure if there is any Ruby method which exposes "there is buffered data". -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/