From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id BE8DE1F66F for ; Tue, 17 Nov 2020 16:07:30 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 2E1BB120A88; Wed, 18 Nov 2020 01:06:47 +0900 (JST) Received: from o1678948x4.outbound-mail.sendgrid.net (o1678948x4.outbound-mail.sendgrid.net [167.89.48.4]) by neon.ruby-lang.org (Postfix) with ESMTPS id 54EDF120A87 for ; Wed, 18 Nov 2020 01:06:44 +0900 (JST) Received: by filterdrecv-p3iad2-5dc87598f5-8bxxp with SMTP id filterdrecv-p3iad2-5dc87598f5-8bxxp-19-5FB3F53C-1 2020-11-17 16:07:24.032342725 +0000 UTC m=+70596.442236721 Received: from herokuapp.com (unknown) by ismtpd0046p1mdw1.sendgrid.net (SG) with ESMTP id tirht_ZDTPmClLmGcWB7wQ for ; Tue, 17 Nov 2020 16:07:23.902 +0000 (UTC) Date: Tue, 17 Nov 2020 16:07:24 +0000 (UTC) From: merch-redmine@jeremyevans.net Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 76782 X-Redmine-Project: ruby-master X-Redmine-Issue-Tracker: Feature X-Redmine-Issue-Id: 17276 X-Redmine-Issue-Author: no6v X-Redmine-Sender: jeremyevans0 X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?RVE3t853K5scBhbmJHUzZTFFeVC=2FZSUmHZ0Dc+26wcEi2CTgsF1oz0wTSSxGGN?= =?us-ascii?Q?BIvXYZs2IrCjmSigb1n32SilHfjQplsyR7gDf6E?= =?us-ascii?Q?nHF=2F7gTUHgnZltweh0rjBljh8TSsT0vvP0W=2FnDS?= =?us-ascii?Q?kBzZ5fs0vV98VLjlvorrC54mQY2aua+CCPV4kkr?= =?us-ascii?Q?lz1ZR0ii=2FCAmjzMV96vGRu02Gde43Uf2gi+iB=2FA?= =?us-ascii?Q?Ii0Rh1IiF6cFbgxJY=3D?= To: ruby-core@ruby-lang.org X-Entity-ID: b/2+PoftWZ6GuOu3b0IycA== X-ML-Name: ruby-core X-Mail-Count: 100908 Subject: [ruby-core:100908] [Ruby master Feature#17276] Ripper stops tokenizing after keyword as a method parameter X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #17276 has been updated by jeremyevans0 (Jeremy Evans). Eregon (Benoit Daloze) wrote in #note-2: > jeremyevans0 (Jeremy Evans) wrote in #note-1: > > Maybe we could support keyword arguments in `Ripper.lex` and `Ripper.tokenize` to raise SyntaxError for errors? Here's a pull request for that approach: https://github.com/ruby/ruby/pull/3774 > > I agree it would be nice. > > Do you think the same would be possible for `Ripper.sexp/sexp_raw`? Yes, the same is possible with `Ripper.sexp/sexp_raw`. I've updated the pull request to handle those as well. ---------------------------------------- Feature #17276: Ripper stops tokenizing after keyword as a method parameter https://bugs.ruby-lang.org/issues/17276#change-88560 * Author: no6v (Nobuhiro IMAI) * Status: Open * Priority: Normal ---------------------------------------- Although these are obviously syntax errors at this time, the following codes cannot be tokenized correctly by `Ripper.tokenize`. ``` $ cat src.rb def req(true) end def opt(true=0) end def rest(*true) end def keyrest(**true) end def block(&true) end ->true{} ->true=0{} ->*true{} ->**true{} ->&true{} $ ruby -rripper -vlne 'p Ripper.tokenize($_)' src.rb ruby 3.0.0dev (2020-10-21T00:24:47Z master da25affdac) [x86_64-linux] ["def", " ", "req", "(", "true", ")"] ["def", " ", "opt", "(", "true", "=", "0", ")"] ["def", " ", "rest", "(", "*", "true", ")"] ["def", " ", "keyrest", "(", "**", "true", ")"] ["def", " ", "block", "(", "&", "true", ")"] ["->", "true", "{"] ["->", "true", "=", "0", "{"] ["->", "*", "true", "{"] ["->", "**", "true", "{"] ["->", "&", "true", "{"] ``` `end` and `}` are not shown in result. This seems to prevent `irb` from determining the continuity of the input. See: https://github.com/ruby/irb/issues/38 -- https://bugs.ruby-lang.org/