From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS4713 221.184.0.0/13 X-Spam-Status: No, score=-3.1 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, FORGED_GMAIL_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by dcvr.yhbt.net (Postfix) with ESMTP id 221F81F461 for ; Fri, 17 May 2019 12:56:51 +0000 (UTC) Received: from neon.ruby-lang.org (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id B228A120C36; Fri, 17 May 2019 21:56:43 +0900 (JST) Received: from o1678916x28.outbound-mail.sendgrid.net (o1678916x28.outbound-mail.sendgrid.net [167.89.16.28]) by neon.ruby-lang.org (Postfix) with ESMTPS id 5723E120C36 for ; Fri, 17 May 2019 21:56:40 +0900 (JST) Received: by filter0060p3las1.sendgrid.net with SMTP id filter0060p3las1-26204-5CDEAF88-1E 2019-05-17 12:56:40.482837988 +0000 UTC m=+137821.468877672 Received: from herokuapp.com (unknown [3.95.247.207]) by ismtpd0050p1iad1.sendgrid.net (SG) with ESMTP id avzXPAGDTg65Z9JsGUMeaw for ; Fri, 17 May 2019 12:56:40.438 +0000 (UTC) Date: Fri, 17 May 2019 12:56:40 +0000 (UTC) From: eregontp@gmail.com Message-ID: References: Mime-Version: 1.0 X-Redmine-MailingListIntegration-Message-Ids: 68179 X-Redmine-Project: ruby-trunk X-Redmine-Issue-Id: 14844 X-Redmine-Issue-Author: rmosolgo X-Redmine-Issue-Assignee: yui-knk X-Redmine-Sender: Eregon X-Mailer: Redmine X-Redmine-Host: bugs.ruby-lang.org X-Redmine-Site: Ruby Issue Tracking System X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-SG-EID: =?us-ascii?Q?KippOI8ZHtTweq7XfQzW93937kJ4QNWwSBuHnaMEcr1WH6yneesvaGB2TxcyFa?= =?us-ascii?Q?XWsvNU2JFf4rnzpZgu2299SOoQfR1U+fzQqte3R?= =?us-ascii?Q?98VDnZFGSCg0eKGHBISf3M5ZNkTQdzHmsez5MZd?= =?us-ascii?Q?nfXXj7u4KP8ePalBNIyLvCJkDUoH9eAFD7LtQe8?= =?us-ascii?Q?vFZTYto1Q8TRe4pQkHG=2FVmk0Diydxltc3pA=3D=3D?= To: ruby-core@ruby-lang.org X-ML-Name: ruby-core X-Mail-Count: 92696 Subject: [ruby-core:92696] [Ruby trunk Feature#14844] Future of RubyVM::AST? X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" Issue #14844 has been updated by Eregon (Benoit Daloze). @mame Thank you for the reply. Could you or @yui-knk propose a description to include in the documentation, summarizing what was said? Could you also give your opinion on accessing Node members by name (https://bugs.ruby-lang.org/issues/14844#note-13) ? > Ripper does not reproduce the details including parser-level optimization. What kind of details? Could you give an example? Things like OPCALL instead of CALL? Is that useful for any tool? I tried a simple expression to compare Ripper and RubyVM::AbstractSyntaxTree: ```ruby pry(main)> Ripper.sexp("def m(a) a * 2 end") => [:program, [[:def, [:@ident, "m", [1, 4]], [:paren, [:params, [[:@ident, "a", [1, 6]]], nil, nil, nil, nil, nil, nil]], [:bodystmt, [[:binary, [:var_ref, [:@ident, "a", [1, 9]]], :*, [:@int, "2", [1, 13]]]], nil, nil, nil]]]] pry(main)> RubyVM::AbstractSyntaxTree.parse("def m(a) a * 2 end") => (SCOPE@1:0-1:18 tbl: [] args: nil body: (DEFN@1:0-1:18 mid: :m body: (SCOPE@1:0-1:18 tbl: [:a] args: (ARGS@1:6-1:7 pre_num: 1 pre_init: nil opt: nil first_post: nil post_num: 0 post_init: nil rest: nil kw: nil kwrest: nil block: nil) body: (OPCALL@1:9-1:14 (LVAR@1:9-1:10 :a) :* (ARRAY@1:13-1:14 (LIT@1:13-1:14 2) nil))))) ``` Indeed, the RubyVM::AbstractSyntaxTree version seems easier to read (and access once we have `RubyVM::AST::Node#[:field_name]`). I think one of the main gains is node fields are named, while they are just a flat Array in `Ripper.sexp`. OTOH, things are far from perfectly clear (so I think "experimental/not for serious use" seems appropriate currently). For instance, one has to manually associate arguments given as e.g. a number for `pre_num` and their names in `tbl`. Optional arguments seem exposed more clearly, by having node under `ARGSnode[:opt]`, however the OPT_ARG look nested like a cons-list instead of being an Array which would be more intuitive. So if we compare a slightly more complex example with the `parser` gem, we see there are lots of opportunities to make RubyVM::AST easier to access/process/read/understand: ```ruby pry(main)> require 'parser/current' pry(main)> RubyVM::AbstractSyntaxTree.parse("def m(b,a,c=3,d=4) a * 2 end") => (SCOPE@1:0-1:28 tbl: [] args: nil body: (DEFN@1:0-1:28 mid: :m body: (SCOPE@1:0-1:28 tbl: [:b, :a, :c, :d] args: (ARGS@1:6-1:17 pre_num: 2 pre_init: nil opt: (OPT_ARG@1:10-1:17 (LASGN@1:10-1:13 :c (LIT@1:12-1:13 3)) (OPT_ARG@1:14-1:17 (LASGN@1:14-1:17 :d (LIT@1:16-1:17 4)) nil)) first_post: nil post_num: 0 post_init: nil rest: nil kw: nil kwrest: nil block: nil) body: (OPCALL@1:19-1:24 (LVAR@1:19-1:20 :a) :* (ARRAY@1:23-1:24 (LIT@1:23-1:24 2) nil))))) pry(main)> Parser::CurrentRuby.parse("def m(b,a,c=3,d=4) a * 2 end") => s(:def, :m, s(:args, s(:arg, :b), s(:arg, :a), s(:optarg, :c, s(:int, 3)), s(:optarg, :d, s(:int, 4))), s(:send, s(:lvar, :a), :*, s(:int, 2))) ``` I think it would be good to take inspiration from `parser` here, which makes it really convenient to access the AST and still seems to not lose any important information. In fact, in what cases the additional things in RubyVM::AST such as the SCOPE nodes would be useful beyond debugging the MRI parser? Would any tool be able to do anything with those that it could not without? I understand exposing the internal AST directly is the simplest implementation-wise. But I think it's quite sub-optimal to access, process and understand. Would it be better to expose an AST more similar, or even exactly the same, as the `parser` gem? ---------------------------------------- Feature #14844: Future of RubyVM::AST? https://bugs.ruby-lang.org/issues/14844#change-78054 * Author: rmosolgo (Robert Mosolgo) * Status: Open * Priority: Normal * Assignee: yui-knk (Kaneko Yuichiro) * Target version: ---------------------------------------- Hi! Thanks for all your great work on the Ruby language. I saw the new RubyVM::AST module in 2.6.0-preview2 and I quickly went to try it out. I'd love to have a well-documented, user-friendly way to parse and manipulate Ruby code using the Ruby standard library, so I'm pretty excited to try it out. (I've been trying to learn Ripper recently, too: https://ripper-preview.herokuapp.com/, https://rmosolgo.github.io/ripper_events/ .) Based on my exploration, I opened a small PR on GitHub with some documentation: https://github.com/ruby/ruby/pull/1888 I'm curious though, are there future plans for this module? For example, we might: - Add more details about each node (for example, we could expose the names of identifiers and operators through the node classes) - Document each node type I see there is a lot more information in the C structures that we could expose, and I'm interested to help out if it's valuable. What do you think? -- https://bugs.ruby-lang.org/