From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-2.7 required=3.0 tests=AWL,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 676001F463 for ; Sat, 28 Dec 2019 16:16:12 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 75B1C120AB4; Sun, 29 Dec 2019 01:15:55 +0900 (JST) Received: from xtrwkhkc.outbound-mail.sendgrid.net (xtrwkhkc.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id D3453120978 for ; Sun, 29 Dec 2019 01:15:52 +0900 (JST) Received: by filterdrecv-p3las1-5bf99c48d-2gb8k with SMTP id filterdrecv-p3las1-5bf99c48d-2gb8k-18-5E077FBE-42 2019-12-28 16:15:58.897786404 +0000 UTC m=+1006212.906196052 Received: from herokuapp.com (unknown [3.82.152.27]) by ismtpd0075p1iad2.sendgrid.net (SG) with ESMTP id RgDE7XS0SPqp1THfVY-Sxg for ; Sat, 28 Dec 2019 16:15:58.799 +0000 (UTC) Date: Sat, 28 Dec 2019 16:15:58 +0000 (UTC) From: eregontp@gmail.com Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 72214 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 16463 X-Redmine-Issue-Author: Eregon X-Redmine-Sender: Eregon X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?KippOI8ZHtTweq7XfQzW93937kJ4QNWwSBuHnaMEcr2uz74Vt0OM6FONEl36CM?= =?us-ascii?Q?bZo4eH9HEesYpRkk0bEag4Bj0vIgkpngeD+JfFX?= =?us-ascii?Q?uWYH+++4NFV2kmFMRKX=2FoGxZ6E+f=2FugaSX+TNiA?= =?us-ascii?Q?j=2FkBRAkyK4lSARY1oFv8NTJI0NBPnBo1OWBuQjH?= =?us-ascii?Q?x3QdQ222iflWHKFa4Fu0p+L71JpTD=2Fds6Uk+UFE?= =?us-ascii?Q?8Z2HmVihuVRAkwj3k=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 96555 Subject: [ruby-core:96555] [Ruby master Feature#16463] Fixing *args-delegation in Ruby 2.7: ruby2_keywords semantics by default in 2.7.1 X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #16463 has been updated by Eregon (Benoit Daloze). I should be more precise about "*args-delegation broke in Ruby 2.7 for keyword arguments.". For example, let's take this simple example from https://eregon.me/blog/2019/11/10/the-delegation-challenge-of-ruby27.html: ```ruby def target(*args, **kwargs) [args, kwargs] end def delegate(*args, &block) target(*args, &block) end target(1, b: 2) # => [[1], {b: 2}] in Ruby 2 & 3 delegate(1, b: 2) # => [[1], {b: 2}] in Ruby <= 2.6 # Ruby 2.7: # kwargs.rb:6: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call # kwargs.rb:1: warning: The called method `target' is defined here # => [[1], {:b=>2}] # Ruby 3.0-3.warn: [[1], {b: 2}] if using `ruby2_keywords` (explicitly or by default); Otherwise [[1, {:b=>2}], {}] (breaking) # Ruby 3.clean+: [[1, {:b=>2}], {}] in Ruby 3 ``` If we use ruby2_keywords semantics by default, it works fine until Ruby 3.warn, at which points it warns to change to `*args, **kwargs`-delegation, which works for Ruby 3.0+. Without `ruby2_keywords` (explicit or by default) it would break in 3.0. Without `ruby2_keywords` it returns the correct result but produces a confusing warning in Ruby 2.7. Based on this confusing warning in Ruby 2.7, people are likely to think they need to pass `**kwargs` explicitly like: ```ruby def delegate(*args, **kwargs, &block) target(*args, **kwargs, &block) end ``` Which would be the natural and intuitive thing to do (who can blame them?), and would actually work in Ruby 3.0+. At that point though, delegation actually breaks when delegating to a method not taking keyword arguments such as `def target(*args); args; end`. In Ruby 2.6, `**kwargs` inside `delegate` passes an extra positional `{}` if no keyword arguments are passed: `delegate()` returns `[{}]` while `target()` returns `[]`. In Ruby 2.7, `delegate({})` returns `[]`, dropping the positional `{}`, while a direct `target({})` call would correctly return `[{}]`. If we use ruby2_keywords semantics by default, we avoid this very complicated corner cases, __avoid these confusing warnings__, and we __only require people to change their delegation code when they can actually migrate__ (i.e., drop Ruby 2.x support) to use `*args, **kwargs`-delegation. If they want to support both Ruby 3.clean and Ruby 2.x (e.g., RSpec might) it's possible with a simple version check (like the first solution in the blog post). In a simplified way, `*args`-delegation would work just fine in Ruby 2.0 - 3.warn, and then change to the intuitive `*args, **kwargs` delegation in Ruby 3.clean. No confusing and full of weird corner cases state in between. ---------------------------------------- Feature #16463: Fixing *args-delegation in Ruby 2.7: ruby2_keywords semantics by default in 2.7.1 https://bugs.ruby-lang.org/issues/16463#change-83487 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- Ruby 2.7.0 is out. It aims to warn for every keyword argument change that will happen in Ruby 3.0. Most warnings are useful: adding `**`, etc is needed to not break code when migrating to 3.0. Ruby 2.7 also aims at remaining compatible with 2.6. However there is a big breaking change here: __`*args`-delegation broke in Ruby 2.7 for keyword arguments__. The workaround is adding `ruby2_keywords` to the method/block receiving the keywords arguments to delegate later on. But is it needed or useful at all to require everyone to add `ruby2_keywords` in many places of their codebase? And for rubyists to get major headaches as to why `*args`-delegation broke and instead has strange semantics in Ruby 2.7? Was it useful to break delegation in Ruby 2.7? I think not, and here I propose a solution to keep delegation in 2.7 compatible with 2.6 (just use `*args` as before). --- First I'll introduce some context. The end goal is to have [separation of positional and keyword arguments](https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/). However, this will not happen in 3.0, because as long as `ruby2_keyword` exist, the separation will only be partial. For example, `foo(*args)` should only pass positional arguments, never keyword arguments, but this can only be guaranteed once `ruby2_keyword` is removed. The plan to get there, as far as I heard and imagine it is: * In Ruby release 3.warn (around Ruby 2.7 EOL, maybe 3.3?), warn for every usage of `ruby2_keywords`, mentioning it should be replaced by `*args, **kwargs`-delegation (or `...`, but that's severely restricted currently: #16378). `*args, **kwargs`-delegation is only correct in Ruby 3.0+ so at that point Ruby 2.x support needs to be dropped, or a version check be used. * In Ruby release 3.clean (that is 3.(warn+1), maybe 3.4?), remove `ruby2_keywords`. At that point, the separation of positional and keyword arguments is finally achieved. `foo(*args)` will always mean "pass only positional arguments". Everytime keyword arguments are passed it will be explicit (`foo(**kwargs)` or `foo(key: value)`), no more magic and a clean separation. So no matter what, to get the clean separation we'll have to wait many (5?) years for Ruby 3.clean, and delegation code will need to change in 3.warn. But right now, we broke delegation in 2.7 and require to add `ruby2_keywords` (which means __changing twice delegation code__ in this period) for seemingly little to no benefit. --- My proposition is to simply use ruby2_keywords semantics for all methods and blocks in Ruby 2.7 (and until version 3.warn). This would be compatible with Ruby 2.6 and before. This means, no explicit `ruby2_keywords` anywhere, no need to change anything for delegation to work in Ruby 2.7, 3.0, ... until Ruby 3.clean. Importantly, it means __only change delegation code once__ and __Ruby 2.0 until Ruby.warn keep `*args`-delegation compatible and working__. The semantics of that are (same as if `ruby2_keywords` was applied to all methods): * When passing keyword arguments syntactically (using either `foo(**kwargs)` or `foo(key: value)`) to a method not accepting keyword arguments (e.g., `def m(*args)`), flag the keyword arguments Hash as "keyword arguments". * Whenever calling a method with a `*rest` argument and no keyword arguments (e.g., `foo(*args)`), if the last argument is flagged as "keyword arguments", pass them as keyword arguments. If the called method doesn't accept keyword arguments, pass the Hash as positional argument and keep the "keyword arguments" flag. That way, code like this just keeps working: ```ruby def target(*args, **kwargs) [args, kwargs] end def delegate(*args, &block) target(*args, &block) end target(1, b: 2) # => [[1], {b: 2}] in Ruby 2 & 3 delegate(1, b: 2) # => [[1], {b: 2}] in Ruby 2 & 3, no warning in 2.7 because {b: 2} is passed as keyword arguments to target ``` And also if `args` is stored somewhere or delegated multiple levels down. Do we lose anything by not marking delegation methods with `ruby2_keywords`? I think we lose nothing, and we gain a lot (compatibility and avoiding needless ugly changes). In Ruby 3.warn we can easily warn for every case that passes keyword arguments using `foo(*args)` and even have a debug mode telling where the Hash was flagged as a "keyword Hash". Thoughts? Should we fix delegation in Ruby 2.7 .. Ruby 3.warn so it works again and not needlessly break Ruby code? I believe YES! --- P.S.: I actually proposed this idea on the ruby-core Slack on 13th December, but got just one response from @jeremyevans0: > Me: If we applied `ruby2_keywords` automatically on all methods, would `*args`-delegation just keep working in 2.7 and later? I think the fundamental issue with kwargs changes is that we break *args by changing its meaning, in a way it no longer works to delegate "all arguments except block". Probably almost every method that takes `(*args)` and then call some methods with `*args` intents to pass positional and kwargs as-is, no matter the Ruby version. If we could save this pattern we'd make the transition much nicer. > Jeremy: I worked on a branch with `ruby2_keywords` behavior by default (for all methods taking `*args`, not just those that delegate `*args` inside the method: https://github.com/jeremyevans/ruby/tree/ruby2_keywords-by-default . I don't recommend that approach, as it is much more likely to result in a keyword-flag hashed being created to a method where the hash should be treated as positional. > Me: Does it matter if the Hash is flagged and passed to a method not taking kwargs? It would still be the same behavior, no? > Jeremy: You can end up with the hash being passed as keywords when you expect it to be passed as non-keywords. It's not safe in general unless you know the method will be used for argument delegation. Jeremy's concern is sometimes you might want `foo(*args)`, with `args[-1]` a Hash with a "keyword arguments" flag, to pass as positional to `def foo(*args, **kwargs)`. However, that seems extremely unlikely to me, and not worth breaking delegation in Ruby 2.7. To have the "keyword arguments" flag, the Hash must have been passed originally as keyword arguments. It sounds unlikely you would then want to pass it as positional to a method taking keyword arguments. If you do want that, it's always possible to do `foo(*args, **{})`, which also works in Ruby 2.6 (and before). -- https://bugs.ruby-lang.org/