[ruby-core:87907] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[ruby-core:87907] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[chopraanmol1]

Issue #14908 has been reported by chopraanmol1 (Anmol Chopra).

----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes to trunk:
~~~
diff --git a/enumerator.c b/enumerator.c
index 050a9ce58f..c21d6f36f1 100644
--- a/enumerator.c
+++ b/enumerator.c
@@ -103,7 +103,7 @@
  */
 VALUE rb_cEnumerator;
 static VALUE rb_cLazy;
-static ID id_rewind, id_new, id_yield, id_to_enum;
+static ID id_rewind, id_new, id_yield, id_yield_push, id_to_enum;
 static ID id_next, id_result, id_lazy, id_receiver, id_arguments, id_memo, id_method, id_force;
 static VALUE sym_each, sym_cycle;
 
@@ -1265,9 +1265,14 @@ yielder_yield(VALUE obj, VALUE args)
 
 /* :nodoc: */
 static VALUE
-yielder_yield_push(VALUE obj, VALUE args)
+yielder_yield_push(VALUE obj, VALUE arg)
 {
-    yielder_yield(obj, args);
+    // VALUE args[1];
+    struct yielder *ptr = yielder_ptr(obj);
+    // args[0] = arg;
+
+    rb_proc_call_with_block(ptr->proc, 1, &arg, Qnil);
+
     return obj;
 }
 
@@ -1510,7 +1515,7 @@ lazy_init_yielder(VALUE val, VALUE m, int argc, VALUE *argv)
     }
 
     if (cont) {
- rb_funcall2(yielder, id_yield, 1, &(result->memo_value));
+ rb_funcall2(yielder, id_yield_push, 1, &(result->memo_value));
     }
     if (LAZY_MEMO_BREAK_P(result)) {
  rb_iter_break();
@@ -2448,7 +2453,7 @@ InitVM_Enumerator(void)
     rb_define_alloc_func(rb_cYielder, yielder_allocate);
     rb_define_method(rb_cYielder, "initialize", yielder_initialize, 0);
     rb_define_method(rb_cYielder, "yield", yielder_yield, -2);
-    rb_define_method(rb_cYielder, "<<", yielder_yield_push, -2);
+    rb_define_method(rb_cYielder, "<<", yielder_yield_push, 1);
 
     rb_provide("enumerator.so"); /* for backward compatibility */
 }
@@ -2459,6 +2464,7 @@ Init_Enumerator(void)
 {
     id_rewind = rb_intern("rewind");
     id_yield = rb_intern("yield");
+    id_yield_push = rb_intern("<<");
     id_new = rb_intern("new");
     id_next = rb_intern("next");
     id_result = rb_intern("result");
~~~

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:87914] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[shyouhei]

Issue #14908 has been updated by shyouhei (Shyouhei Urabe).


Understand the problem.  The proposed fix however involves spec change.  We need to discuss effects of it before applying this patch.

----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-72924

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes to trunk:
~~~
diff --git a/enumerator.c b/enumerator.c
index 050a9ce58f..c21d6f36f1 100644
--- a/enumerator.c
+++ b/enumerator.c
@@ -103,7 +103,7 @@
  */
 VALUE rb_cEnumerator;
 static VALUE rb_cLazy;
-static ID id_rewind, id_new, id_yield, id_to_enum;
+static ID id_rewind, id_new, id_yield, id_yield_push, id_to_enum;
 static ID id_next, id_result, id_lazy, id_receiver, id_arguments, id_memo, id_method, id_force;
 static VALUE sym_each, sym_cycle;
 
@@ -1265,9 +1265,14 @@ yielder_yield(VALUE obj, VALUE args)
 
 /* :nodoc: */
 static VALUE
-yielder_yield_push(VALUE obj, VALUE args)
+yielder_yield_push(VALUE obj, VALUE arg)
 {
-    yielder_yield(obj, args);
+    struct yielder *ptr = yielder_ptr(obj);
+
+    rb_proc_call_with_block(ptr->proc, 1, &arg, Qnil);
+
     return obj;
 }
 
@@ -1510,7 +1515,7 @@ lazy_init_yielder(VALUE val, VALUE m, int argc, VALUE *argv)
     }
 
     if (cont) {
- rb_funcall2(yielder, id_yield, 1, &(result->memo_value));
+ rb_funcall2(yielder, id_yield_push, 1, &(result->memo_value));
     }
     if (LAZY_MEMO_BREAK_P(result)) {
  rb_iter_break();
@@ -2448,7 +2453,7 @@ InitVM_Enumerator(void)
     rb_define_alloc_func(rb_cYielder, yielder_allocate);
     rb_define_method(rb_cYielder, "initialize", yielder_initialize, 0);
     rb_define_method(rb_cYielder, "yield", yielder_yield, -2);
-    rb_define_method(rb_cYielder, "<<", yielder_yield_push, -2);
+    rb_define_method(rb_cYielder, "<<", yielder_yield_push, 1);
 
     rb_provide("enumerator.so"); /* for backward compatibility */
 }
@@ -2459,6 +2464,7 @@ Init_Enumerator(void)
 {
     id_rewind = rb_intern("rewind");
     id_yield = rb_intern("yield");
+    id_yield_push = rb_intern("<<");
     id_new = rb_intern("new");
     id_next = rb_intern("next");
     id_result = rb_intern("result");
~~~

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:87918] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[chopraanmol1]

Issue #14908 has been updated by chopraanmol1 (Anmol Chopra).


shyouhei (Shyouhei Urabe) wrote:
> Understand the problem.  The proposed fix however involves spec change.  We need to discuss effects of it before applying this patch.

If this is a big breaking change than alternate will be to create a different method for Enumerator::Lazy's internal use. I'm also up for updating the patch to reflect that. But I think for future release it will make more sense to have Enumerator::Yielder#<< to have an arity of 1 if you consider syntax use case.

But for the backporting purpose, I'm more inclined to create a new method.

----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-72928

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes to trunk:
~~~
diff --git a/enumerator.c b/enumerator.c
index 050a9ce58f..c21d6f36f1 100644
--- a/enumerator.c
+++ b/enumerator.c
@@ -103,7 +103,7 @@
  */
 VALUE rb_cEnumerator;
 static VALUE rb_cLazy;
-static ID id_rewind, id_new, id_yield, id_to_enum;
+static ID id_rewind, id_new, id_yield, id_yield_push, id_to_enum;
 static ID id_next, id_result, id_lazy, id_receiver, id_arguments, id_memo, id_method, id_force;
 static VALUE sym_each, sym_cycle;
 
@@ -1265,9 +1265,14 @@ yielder_yield(VALUE obj, VALUE args)
 
 /* :nodoc: */
 static VALUE
-yielder_yield_push(VALUE obj, VALUE args)
+yielder_yield_push(VALUE obj, VALUE arg)
 {
-    yielder_yield(obj, args);
+    struct yielder *ptr = yielder_ptr(obj);
+
+    rb_proc_call_with_block(ptr->proc, 1, &arg, Qnil);
+
     return obj;
 }
 
@@ -1510,7 +1515,7 @@ lazy_init_yielder(VALUE val, VALUE m, int argc, VALUE *argv)
     }
 
     if (cont) {
- rb_funcall2(yielder, id_yield, 1, &(result->memo_value));
+ rb_funcall2(yielder, id_yield_push, 1, &(result->memo_value));
     }
     if (LAZY_MEMO_BREAK_P(result)) {
  rb_iter_break();
@@ -2448,7 +2453,7 @@ InitVM_Enumerator(void)
     rb_define_alloc_func(rb_cYielder, yielder_allocate);
     rb_define_method(rb_cYielder, "initialize", yielder_initialize, 0);
     rb_define_method(rb_cYielder, "yield", yielder_yield, -2);
-    rb_define_method(rb_cYielder, "<<", yielder_yield_push, -2);
+    rb_define_method(rb_cYielder, "<<", yielder_yield_push, 1);
 
     rb_provide("enumerator.so"); /* for backward compatibility */
 }
@@ -2459,6 +2464,7 @@ Init_Enumerator(void)
 {
     id_rewind = rb_intern("rewind");
     id_yield = rb_intern("yield");
+    id_yield_push = rb_intern("<<");
     id_new = rb_intern("new");
     id_next = rb_intern("next");
     id_result = rb_intern("result");
~~~

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:87924] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[eregontp]

Issue #14908 has been updated by Eregon (Benoit Daloze).


Changing Enumerator::Yielder#<< to have arity 1 seems fine to me, as I guess nobody calls << on an Enumerator::Yielder with more than 1 argument, isn't it?

----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-72934

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes to trunk:
~~~
diff --git a/enumerator.c b/enumerator.c
index 050a9ce58f..c21d6f36f1 100644
--- a/enumerator.c
+++ b/enumerator.c
@@ -103,7 +103,7 @@
  */
 VALUE rb_cEnumerator;
 static VALUE rb_cLazy;
-static ID id_rewind, id_new, id_yield, id_to_enum;
+static ID id_rewind, id_new, id_yield, id_yield_push, id_to_enum;
 static ID id_next, id_result, id_lazy, id_receiver, id_arguments, id_memo, id_method, id_force;
 static VALUE sym_each, sym_cycle;
 
@@ -1265,9 +1265,14 @@ yielder_yield(VALUE obj, VALUE args)
 
 /* :nodoc: */
 static VALUE
-yielder_yield_push(VALUE obj, VALUE args)
+yielder_yield_push(VALUE obj, VALUE arg)
 {
-    yielder_yield(obj, args);
+    struct yielder *ptr = yielder_ptr(obj);
+
+    rb_proc_call_with_block(ptr->proc, 1, &arg, Qnil);
+
     return obj;
 }
 
@@ -1510,7 +1515,7 @@ lazy_init_yielder(VALUE val, VALUE m, int argc, VALUE *argv)
     }
 
     if (cont) {
- rb_funcall2(yielder, id_yield, 1, &(result->memo_value));
+ rb_funcall2(yielder, id_yield_push, 1, &(result->memo_value));
     }
     if (LAZY_MEMO_BREAK_P(result)) {
  rb_iter_break();
@@ -2448,7 +2453,7 @@ InitVM_Enumerator(void)
     rb_define_alloc_func(rb_cYielder, yielder_allocate);
     rb_define_method(rb_cYielder, "initialize", yielder_initialize, 0);
     rb_define_method(rb_cYielder, "yield", yielder_yield, -2);
-    rb_define_method(rb_cYielder, "<<", yielder_yield_push, -2);
+    rb_define_method(rb_cYielder, "<<", yielder_yield_push, 1);
 
     rb_provide("enumerator.so"); /* for backward compatibility */
 }
@@ -2459,6 +2464,7 @@ Init_Enumerator(void)
 {
     id_rewind = rb_intern("rewind");
     id_yield = rb_intern("yield");
+    id_yield_push = rb_intern("<<");
     id_new = rb_intern("new");
     id_next = rb_intern("next");
     id_result = rb_intern("result");
~~~

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:87933] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[chopraanmol1]

Issue #14908 has been updated by chopraanmol1 (Anmol Chopra).


Eregon (Benoit Daloze) wrote:
> Changing Enumerator::Yielder#<< to have arity 1 seems fine to me, as I guess nobody calls << on an Enumerator::Yielder with more than 1 argument, isn't it?

Yes, that will be the general case. Exception:
~~~ ruby
.send(:<<,...)
~~~
~~~ ruby
.<<(...)
~~~



----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-72940

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes to trunk:
~~~
diff --git a/enumerator.c b/enumerator.c
index 050a9ce58f..c21d6f36f1 100644
--- a/enumerator.c
+++ b/enumerator.c
@@ -103,7 +103,7 @@
  */
 VALUE rb_cEnumerator;
 static VALUE rb_cLazy;
-static ID id_rewind, id_new, id_yield, id_to_enum;
+static ID id_rewind, id_new, id_yield, id_yield_push, id_to_enum;
 static ID id_next, id_result, id_lazy, id_receiver, id_arguments, id_memo, id_method, id_force;
 static VALUE sym_each, sym_cycle;
 
@@ -1265,9 +1265,14 @@ yielder_yield(VALUE obj, VALUE args)
 
 /* :nodoc: */
 static VALUE
-yielder_yield_push(VALUE obj, VALUE args)
+yielder_yield_push(VALUE obj, VALUE arg)
 {
-    yielder_yield(obj, args);
+    struct yielder *ptr = yielder_ptr(obj);
+
+    rb_proc_call_with_block(ptr->proc, 1, &arg, Qnil);
+
     return obj;
 }
 
@@ -1510,7 +1515,7 @@ lazy_init_yielder(VALUE val, VALUE m, int argc, VALUE *argv)
     }
 
     if (cont) {
- rb_funcall2(yielder, id_yield, 1, &(result->memo_value));
+ rb_funcall2(yielder, id_yield_push, 1, &(result->memo_value));
     }
     if (LAZY_MEMO_BREAK_P(result)) {
  rb_iter_break();
@@ -2448,7 +2453,7 @@ InitVM_Enumerator(void)
     rb_define_alloc_func(rb_cYielder, yielder_allocate);
     rb_define_method(rb_cYielder, "initialize", yielder_initialize, 0);
     rb_define_method(rb_cYielder, "yield", yielder_yield, -2);
-    rb_define_method(rb_cYielder, "<<", yielder_yield_push, -2);
+    rb_define_method(rb_cYielder, "<<", yielder_yield_push, 1);
 
     rb_provide("enumerator.so"); /* for backward compatibility */
 }
@@ -2459,6 +2464,7 @@ Init_Enumerator(void)
 {
     id_rewind = rb_intern("rewind");
     id_yield = rb_intern("yield");
+    id_yield_push = rb_intern("<<");
     id_new = rb_intern("new");
     id_next = rb_intern("next");
     id_result = rb_intern("result");
~~~

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:87936] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[ruby-core]

Issue #14908 has been updated by marcandre (Marc-Andre Lafortune).


Indeed, as long as `Yielder#yield` is kept with arity -1 (as in this patch), indeed I don't think that would be an "incompatibility" we should worry about.

----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-72943

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes to trunk:
~~~
diff --git a/enumerator.c b/enumerator.c
index 050a9ce58f..c21d6f36f1 100644
--- a/enumerator.c
+++ b/enumerator.c
@@ -103,7 +103,7 @@
  */
 VALUE rb_cEnumerator;
 static VALUE rb_cLazy;
-static ID id_rewind, id_new, id_yield, id_to_enum;
+static ID id_rewind, id_new, id_yield, id_yield_push, id_to_enum;
 static ID id_next, id_result, id_lazy, id_receiver, id_arguments, id_memo, id_method, id_force;
 static VALUE sym_each, sym_cycle;
 
@@ -1265,9 +1265,14 @@ yielder_yield(VALUE obj, VALUE args)
 
 /* :nodoc: */
 static VALUE
-yielder_yield_push(VALUE obj, VALUE args)
+yielder_yield_push(VALUE obj, VALUE arg)
 {
-    yielder_yield(obj, args);
+    struct yielder *ptr = yielder_ptr(obj);
+
+    rb_proc_call_with_block(ptr->proc, 1, &arg, Qnil);
+
     return obj;
 }
 
@@ -1510,7 +1515,7 @@ lazy_init_yielder(VALUE val, VALUE m, int argc, VALUE *argv)
     }
 
     if (cont) {
- rb_funcall2(yielder, id_yield, 1, &(result->memo_value));
+ rb_funcall2(yielder, id_yield_push, 1, &(result->memo_value));
     }
     if (LAZY_MEMO_BREAK_P(result)) {
  rb_iter_break();
@@ -2448,7 +2453,7 @@ InitVM_Enumerator(void)
     rb_define_alloc_func(rb_cYielder, yielder_allocate);
     rb_define_method(rb_cYielder, "initialize", yielder_initialize, 0);
     rb_define_method(rb_cYielder, "yield", yielder_yield, -2);
-    rb_define_method(rb_cYielder, "<<", yielder_yield_push, -2);
+    rb_define_method(rb_cYielder, "<<", yielder_yield_push, 1);
 
     rb_provide("enumerator.so"); /* for backward compatibility */
 }
@@ -2459,6 +2464,7 @@ Init_Enumerator(void)
 {
     id_rewind = rb_intern("rewind");
     id_yield = rb_intern("yield");
+    id_yield_push = rb_intern("<<");
     id_new = rb_intern("new");
     id_next = rb_intern("next");
     id_result = rb_intern("result");
~~~

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:88254] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[kelvinbhoss]

Issue #14908 has been updated by kelvinbhoss (Kelvin Hoss).


I must say that emulators are always good when it comes to get the same experience with limited sources. https://psncoders.com


----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-73276

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes to trunk(Pull Request for the same: https://github.com/ruby/ruby/pull/1912):
~~~
diff --git a/enumerator.c b/enumerator.c
index 050a9ce58f..c21d6f36f1 100644
--- a/enumerator.c
+++ b/enumerator.c
@@ -103,7 +103,7 @@
  */
 VALUE rb_cEnumerator;
 static VALUE rb_cLazy;
-static ID id_rewind, id_new, id_yield, id_to_enum;
+static ID id_rewind, id_new, id_yield, id_yield_push, id_to_enum;
 static ID id_next, id_result, id_lazy, id_receiver, id_arguments, id_memo, id_method, id_force;
 static VALUE sym_each, sym_cycle;
 
@@ -1265,9 +1265,14 @@ yielder_yield(VALUE obj, VALUE args)
 
 /* :nodoc: */
 static VALUE
-yielder_yield_push(VALUE obj, VALUE args)
+yielder_yield_push(VALUE obj, VALUE arg)
 {
-    yielder_yield(obj, args);
+    struct yielder *ptr = yielder_ptr(obj);
+
+    rb_proc_call_with_block(ptr->proc, 1, &arg, Qnil);
+
     return obj;
 }
 
@@ -1510,7 +1515,7 @@ lazy_init_yielder(VALUE val, VALUE m, int argc, VALUE *argv)
     }
 
     if (cont) {
- rb_funcall2(yielder, id_yield, 1, &(result->memo_value));
+ rb_funcall2(yielder, id_yield_push, 1, &(result->memo_value));
     }
     if (LAZY_MEMO_BREAK_P(result)) {
  rb_iter_break();
@@ -2448,7 +2453,7 @@ InitVM_Enumerator(void)
     rb_define_alloc_func(rb_cYielder, yielder_allocate);
     rb_define_method(rb_cYielder, "initialize", yielder_initialize, 0);
     rb_define_method(rb_cYielder, "yield", yielder_yield, -2);
-    rb_define_method(rb_cYielder, "<<", yielder_yield_push, -2);
+    rb_define_method(rb_cYielder, "<<", yielder_yield_push, 1);
 
     rb_provide("enumerator.so"); /* for backward compatibility */
 }
@@ -2459,6 +2464,7 @@ Init_Enumerator(void)
 {
     id_rewind = rb_intern("rewind");
     id_yield = rb_intern("yield");
+    id_yield_push = rb_intern("<<");
     id_new = rb_intern("new");
     id_next = rb_intern("next");
     id_result = rb_intern("result");
~~~

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:88986] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[knu]

Issue #14908 has been updated by knu (Akinori MUSHA).


I'm the original author of Enumerator & Yielder and I don't think I particularly intended to make `<<` accept many arguments.  `<<` was an alias of `yield`, and I just didn't bother to make a separate function.  So, I'd say go ahead.

----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-74015

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1 and use it internally for the lazy enum. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes:

**https://github.com/ruby/ruby/pull/1912**

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:89054] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[chopraanmol1]

Issue #14908 has been updated by chopraanmol1 (Anmol Chopra).


All the response of the proposed patch is positive till date. @shyouhei can we go ahead with this now?

----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-74079

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1 and use it internally for the lazy enum. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes:

**https://github.com/ruby/ruby/pull/1912**

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:89055] [Ruby trunk Bug#14908] Enumerator::Lazy creates unnecessary Array objects.

[shyouhei]

Issue #14908 has been updated by shyouhei (Shyouhei Urabe).


No objection.  I guess @nobu is trying to fix it.

----------------------------------------
Bug #14908: Enumerator::Lazy creates unnecessary Array objects.
https://bugs.ruby-lang.org/issues/14908#change-74080

* Author: chopraanmol1 (Anmol Chopra)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: ruby 2.6.0dev (2018-07-11 trunk 63949) [x86_64-linux]
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
Benchmark result on trunk:
~~~
                 user     system      total        real
Lazy:        0.120000   0.000000   0.120000 (  0.119958)
Normal:      0.056000   0.004000   0.060000 (  0.062848)
             2.142857   0.000000        NaN (  1.908698)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 122240 bytes (3033 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
    120480  Array
       880  Proc
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
      3012  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++

~~~

As you may observe an extra array is created for every final element. Enumerator::Yielder#yield method has arity of -1 which wrap every elements in array. The same goes for Enumerator::Yielder#<< method, I'm proposing to change arity for Enumerator::Yielder#<< to 1 from -1 and use it internally for the lazy enum. It will also make << method definition consistent with other classes(Array, String & etc).

I've applied the following set of changes:

**https://github.com/ruby/ruby/pull/1912**

which result in the following:

~~~
                 user     system      total        real
Lazy:        0.108000   0.000000   0.108000 (  0.108484)
Normal:      0.052000   0.008000   0.060000 (  0.062528)
             2.076923   0.000000        NaN (  1.734961)
++++++++++++++++++++++++++++++++++++++++++++++++++
Lazy:
Total allocated: 2240 bytes (33 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
       880  Proc
       480  Array
       384  Enumerator::Lazy
       264  Object
       168  Enumerator::Generator
        64  Enumerator::Yielder

allocated objects by class
--------------------------------------------------
        12  Array
        11  Proc
         3  Enumerator::Generator
         3  Enumerator::Lazy
         3  Object
         1  Enumerator::Yielder
++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++
Normal:
Total allocated: 72120 bytes (3 objects)
Total retained:  0 bytes (0 objects)

allocated memory by class
--------------------------------------------------
     72120  Array

allocated objects by class
--------------------------------------------------
         3  Array
++++++++++++++++++++++++++++++++++++++++++++++++++
~~~

This changes reduces the memory utilization and also by a tiny fraction improves performance for the lazy enumerator 


---Files--------------------------------
lazy_test.rb (1.26 KB)


-- 
https://bugs.ruby-lang.org/