From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: poffice@blade.nagaokaut.ac.jp Delivered-To: poffice@blade.nagaokaut.ac.jp Received: from kankan.nagaokaut.ac.jp (kankan.nagaokaut.ac.jp [133.44.2.24]) by blade.nagaokaut.ac.jp (Postfix) with ESMTP id 738D419E005C for ; Fri, 18 Dec 2015 12:42:39 +0900 (JST) Received: from voscc.nagaokaut.ac.jp (voscc.nagaokaut.ac.jp [133.44.1.100]) by kankan.nagaokaut.ac.jp (Postfix) with ESMTP id 80FF8B5D8BB for ; Fri, 18 Dec 2015 13:14:46 +0900 (JST) Received: from neon.ruby-lang.org (neon.ruby-lang.org [221.186.184.75]) by voscc.nagaokaut.ac.jp (Postfix) with ESMTP id CAF9B18CC7CC for ; Fri, 18 Dec 2015 13:14:46 +0900 (JST) Received: from [221.186.184.76] (localhost [IPv6:::1]) by neon.ruby-lang.org (Postfix) with ESMTP id 57F0E120771; Fri, 18 Dec 2015 13:13:49 +0900 (JST) X-Original-To: ruby-core@ruby-lang.org Delivered-To: ruby-core@ruby-lang.org Received: from mail-io0-f181.google.com (mail-io0-f181.google.com [209.85.223.181]) by neon.ruby-lang.org (Postfix) with ESMTPS id 1FF711204EB for ; Fri, 18 Dec 2015 13:13:20 +0900 (JST) Received: by mail-io0-f181.google.com with SMTP id q126so77645539iof.2 for ; Thu, 17 Dec 2015 20:13:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:message-id:subject:mime-version:content-type; bh=JD5AJmUWlvVkPGAGpdAZ9tE6RqqV9oL3GAMnLzt9FUE=; b=mUxgeWQizgNKIQiaRyq+FQqRP5yD59cT8NiFoM3D8nn/nPTkJEBxn1zSl4w/s3YX9b GbpdBWvAR3POEMcbyjY/pYaHS32Bzuj3JJFVu/sci4Xgy6U4x2ukIXdWWWLzKESgaDpL BuYcAdsvBV9+JMZfzYwKQNEIOUhJ8rkee+tM/I1WfYQUlX1wqCZlvh5gp5Do5gwMi8dA BUuDKXqkIr4tenLiOL4iEMNwRLnXTpNz3DVxHaaYEVJcy8o77wgsLqMBlvX42uwXh2kw ku8gZbk0PHlt+fcF4facEk5R+q/wudGEKjdfiW5OWSywF6Gh8eH0UHAC5Y4sRNJ6wzm3 nMgg== X-Received: by 10.107.137.222 with SMTP id t91mr2561308ioi.172.1450411998921; Thu, 17 Dec 2015 20:13:18 -0800 (PST) Received: from Josephe-Jones (75-166-130-47.hlrn.qwest.net. [75.166.130.47]) by smtp.gmail.com with ESMTPSA id n14sm1937922igx.17.2015.12.17.20.13.17 for (version=TLSv1/SSLv3 cipher=OTHER); Thu, 17 Dec 2015 20:13:18 -0800 (PST) Date: Thu, 17 Dec 2015 21:13:17 -0700 From: Joseph Jones To: Ruby developers Cc: Message-ID: <6D750438-EE71-4850-9365-430F282B4D9E@gmail.com> X-Mailer: BoxerFree 6.0.4 (321) X-Boxer-Generated: true X-Boxer-IsLike: true MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="567387dd_1d545c4d_16c" X-ML-Name: ruby-core X-Mail-Count: 72339 Subject: [ruby-core:72339] [Ruby trunk - Feature #11788] [Open] New ISeq serialize binary format X-BeenThere: ruby-core@ruby-lang.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: Ruby developers List-Id: Ruby developers List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ruby-core-bounces@ruby-lang.org Sender: "ruby-core" --567387dd_1d545c4d_16c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Joseph Jones liked your message with Boxer. On December 8, 2015 at 03:56:= 15 MST, ko1=40atdot.net wrote:Issue =2311788 has been reported by Koichi = Sasada.----------------------------------------=46eature =2311788: New IS= eq serialize binary formathttps://bugs.ruby-lang.org/issues/11788* Author= : Koichi Sasada* Status: Open* Priority: Normal* Assignee: Koichi Sasada-= ---------------------------------------=23 AbstractI wrote a new RubyVM::= InstructionSequence (ISeq) object serializer and de-serializer binary for= mat.Matz had approved to introduce this feature to Ruby 2.3 as *experimen= tal* feature.So I'll commit them.There are two methods to serialize and d= e-serialize.* RubyVM::InstructionSequence=23to=5Fbinary=5Fformat returns = binary format data as String object.* RubyVM::InstructionSequence.from=5F= binary=5Fformat(data) de-serialize it.The goal of this project is to prov= ide =22machine dependent=22 binary file to achieve:* fast bootstrap time = for big applications* reduce memory consumption with several techniques=22= Machine dependent=22 means you can't migrate compiled binaries to other m= achines.They are not goals of this project:* packing scripts to one packa= ge* migrate obfuscate binary to other node to hide source codeTo achieve = such goals, we need to consider compatibility issues such as =60=5F=5F=46= ILE=5F=5F=60, =60=5F=5Fdir=5F=5F=60, =60DATA=60, and so on (for example, = consider about this code: =60Dir.glob(=46ile.join(=5F=5Fdir=5F=5F, '*.rb'= )=60).This proposal doesn't contain =22how to store compiled binaries=22.= =46or example, Rubinius makes *.rbc file automatically.However, Matz does= not like such automatic compilation.So that my proposal only show user s= torage class interface.People can try to make your own ISeq binary storag= e.=46or example,* making a compiled binary files automatically in same di= rectory of script files like Rubinius,* store compiled binaries in some D= B* make storage data structure in your own.I wrote several samples:* dbm:= use dbm* fs: =5Bdefault=5D use file system. locate compiled file in same= directory of script file like Rubinius. foo.rb.yarb will be created for = foo.rb.* fs2: use file system. locate compiled file in specified director= y.* nothing: do nothing.You can see my sample implementation:https://gith= ub.com/ko1/ruby/blob/iseq=5Fp1/sample/iseq=5Floader.rbThe key interface i= s =60RubyVM::InstructionSequence.load=5Fiseq(fname)=60.When MRI try to lo= ad any script named =60fname=60, then call this method with =60fname=60 i= f defined.The return value is an ISeq object, then MRI use this ISeq obje= ct instead of parsing/compiling =60fname=60 file.Note that this proposal = is =22experimental=22.These interfaces are only for experiments.=46or exa= mple, if we want to use several binary storage,this interface doesn't sup= port multiple storage (lack of extensibility).=23 Current statusThe curre= nt implementation is not matured because the binary size is very big beca= use pointer size consumes 32/64 bits.It is easy to reduce, but I remain t= his weak point.Now, one goal =22reduction of memory consumption=22 is not= achieved because no techniques are introduced to share/unload or somethi= ng.This is future work.=23 EvaluationSeveral evaluation results:=23=23 re= solv.rbTry to load resolv.rb 1,000 times (and remove Resolv class each ti= me).=60=60=60compile 12.360000 0.310000 12.670000 ( 13.413011)compile 12.= 120000 0.300000 12.420000 ( 13.195313)compile 12.230000 0.270000 12.50000= 0 ( 13.242140)eager loadload 3.750000 0.180000 3.930000 ( 3.918169)load 4= .000000 0.170000 4.170000 ( 4.178442)load 4.120000 0.200000 4.320000 ( 4.= 320233)lazy loadload 2.410000 0.090000 2.500000 ( 2.609716)load 2.280000 = 0.210000 2.490000 ( 2.518892)load 2.310000 0.110000 2.420000 ( 2.419687)=60= =60=603.25 times faster than normal compilation.If we use lazy loading te= chnique, it is 5.2 times faster.=23=23 fileutils.rbTry similar to resolv.= rb.=60=60=60 user system total realcompile 8.540000 0.130000 8.670000 ( 8= .703615)compile 8.540000 0.150000 8.690000 ( 8.693870)compile 8.430000 0.= 120000 8.550000 ( 8.547480)eager loadload 4.470000 0.150000 4.620000 ( 4.= 659934)load 4.500000 0.140000 4.640000 ( 4.640365)load 4.610000 0.100000 = 4.710000 ( 4.708825)lazy loadload 3.510000 0.140000 3.650000 ( 3.694146)l= oad 3.470000 0.130000 3.600000 ( 3.609040)load 3.550000 0.150000 3.700000= ( 3.831015)=60=60=60Only 1.8 times faster (eager) and 2.4 times faster (= lazy).This is because the initialization of =46ileUtils class run long ti= me.It uses module=5Feval(str) to add methods.=23=23 Simple rails applicat= ionrun =60time rails r ''=60 on simple Rails application (https://github.= com/ko1/tracer=5Fdemo=5Frails=5Fapp tracers are disabled).=60=60=60compil= e:real 0m2.049suser 0m1.601ssys 0m0.402seager:real 0m1.544suser 0m1.094ss= ys 0m0.422slazy:=24 time rails r ''real 0m1.536suser 0m1.112ssys 0m0.388s= =60=60=60Not so impressive result. It seems there are many initialization= code.-- https://bugs.ruby-lang.org/ --567387dd_1d545c4d_16c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Joseph Jones liked your message with Boxer.


= On December 8, 2015 at 03:56:15 MST, ko1=40atdot.net wrote:
Issue =2311788 has been reported by= Koichi Sasada.

----------------------------------------
= =46eature =2311788: New ISeq serialize binary format
https://bugs.ru= by-lang.org/issues/11788

* Author: Koichi Sasada
* Status= : Open
* Priority: Normal
* Assignee: Koichi Sasada
------= ----------------------------------
=23 Abstract

I wrote a= new RubyVM::InstructionSequence (ISeq) object serializer and de-serializ= er binary format.
Matz had approved to introduce this feature to Rub= y 2.3 as *experimental* feature.
So I'll commit them.

The= re are two methods to serialize and de-serialize.

* RubyVM::In= structionSequence=23to=5Fbinary=5Fformat returns binary format data as St= ring object.
* RubyVM::InstructionSequence.from=5Fbinary=5Fformat(da= ta) de-serialize it.

The goal of this project is to provide =22= machine dependent=22 binary file to achieve:

* fast bootstrap = time for big applications
* reduce memory consumption with several t= echniques

=22Machine dependent=22 means you can't migrate comp= iled binaries to other machines.

They are not goals of this pr= oject:

* packing scripts to one package
* migrate obfusca= te binary to other node to hide source code

To achieve such go= als, we need to consider compatibility issues such as =60=5F=5F=46ILE=5F=5F= =60, =60=5F=5Fdir=5F=5F=60, =60DATA=60, and so on (for example, consider = about this code: =60Dir.glob(=46ile.join(=5F=5Fdir=5F=5F, '*.rb')=60).
This proposal doesn't contain =22how to store compiled binaries=22= .
=46or example, Rubinius makes *.rbc file automatically.
Howev= er, Matz does not like such automatic compilation.

So that my = proposal only show user storage class interface.
People can try to m= ake your own ISeq binary storage.

=46or example,

* = making a compiled binary files automatically in same directory of script = files like Rubinius,
* store compiled binaries in some DB
* mak= e storage data structure in your own.

I wrote several samples:=

* dbm: use dbm
* fs: =5Bdefault=5D use file system. loca= te compiled file in same directory of script file like Rubinius. foo.rb.y= arb will be created for foo.rb.
* fs2: use file system. locate compi= led file in specified directory.
* nothing: do nothing.

Y= ou can see my sample implementation:
https://github.com/ko1/ruby/blo= b/iseq=5Fp1/sample/iseq=5Floader.rb

The key interface is =60Ru= byVM::InstructionSequence.load=5Fiseq(fname)=60.
When MRI try to loa= d any script named =60fname=60, then call this method with =60fname=60 if= defined.
The return value is an ISeq object, then MRI use this ISeq= object instead of parsing/compiling =60fname=60 file.

Note th= at this proposal is =22experimental=22.
These interfaces are only fo= r experiments.
=46or example, if we want to use several binary stora= ge,
this interface doesn't support multiple storage (lack of extensi= bility).

=23 Current status

The current implementat= ion is not matured because the binary size is very big because pointer si= ze consumes 32/64 bits.
It is easy to reduce, but I remain this weak= point.

Now, one goal =22reduction of memory consumption=22 is= not achieved because no techniques are introduced to share/unload or som= ething.
This is future work.

=23 Evaluation

Se= veral evaluation results:

=23=23 resolv.rb

Try to l= oad resolv.rb 1,000 times (and remove Resolv class each time).

=60=60=60
compile 12.360000 0.310000 12.670000 ( 13.413011)<= br />compile 12.120000 0.300000 12.420000 ( 13.195313)
compil= e 12.230000 0.270000 12.500000 ( 13.242140)

eager load<= br />load 3.750000 0.180000 3.930000 ( 3.918169)
load = 4.000000 0.170000 4.170000 ( 4.178442)
load 4.12= 0000 0.200000 4.320000 ( 4.320233)

lazy load
load = 2.410000 0.090000 2.500000 ( 2.609716)
load 2.280= 000 0.210000 2.490000 ( 2.518892)
load 2.310000 0.110= 000 2.420000 ( 2.419687)
=60=60=60

3.25 times faster t= han normal compilation.
If we use lazy loading technique, it is 5.2 = times faster.

=23=23 fileutils.rb

Try similar to re= solv.rb.

=60=60=60
user system = total real
compile 8.540000 0.130000 8.670000 ( 8.7= 03615)
compile 8.540000 0.150000 8.690000 ( 8.693870)
compile 8.430000 0.120000 8.550000 ( 8.547480)

eage= r load
load 4.470000 0.150000 4.620000 ( 4.659934)
load 4.500000 0.140000 4.640000 ( 4.640365)
load = 4.610000 0.100000 4.710000 ( 4.708825)

lazy load
= load 3.510000 0.140000 3.650000 ( 3.694146)
load = 3.470000 0.130000 3.600000 ( 3.609040)
load 3.550000 = 0.150000 3.700000 ( 3.831015)
=60=60=60

Only 1.8 tim= es faster (eager) and 2.4 times faster (lazy).
This is because the i= nitialization of =46ileUtils class run long time.
It uses module=5Fe= val(str) to add methods.

=23=23 Simple rails application
=
run =60time rails r ''=60 on simple Rails application (https://gith= ub.com/ko1/tracer=5Fdemo=5Frails=5Fapp tracers are disabled).

= =60=60=60
compile:
real 0m2.049s
user 0m1.601s
= sys 0m0.402s

eager:
real 0m1.544s
user 0m1= .094s
sys 0m0.422s

lazy:
=24 time rails r ''
real 0m1.536s
user 0m1.112s
sys 0m0.388s
=60=60= =60

Not so impressive result. It seems there are many initiali= zation code.




--
https://bugs.ruby-lang= .org/
--567387dd_1d545c4d_16c--