From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from kankan.nagaokaut.ac.jp (kankan.nagaokaut.ac.jp [133.44.2.24]) by blade.nagaokaut.ac.jp (8.12.3/8.12.3/Debian-6.6) with ESMTP id j6DHHHgZ027779 for ; Thu, 14 Jul 2005 02:17:17 +0900 Received: from funfun.nagaokaut.ac.jp (funfun.nagaokaut.ac.jp [133.44.2.201]) by kankan.nagaokaut.ac.jp (Postfix) with ESMTP id EF7D458AC for ; Thu, 14 Jul 2005 02:17:22 +0900 (JST) Received: from localhost (localhost.nagaokaut.ac.jp [127.0.0.1]) by funfun.nagaokaut.ac.jp (Postfix) with ESMTP id D00ADF0484A for ; Thu, 14 Jul 2005 02:17:22 +0900 (JST) Received: from voscc.nagaokaut.ac.jp (voscc.nagaokaut.ac.jp [133.44.1.100]) by funfun.nagaokaut.ac.jp (Postfix) with ESMTP id 8D847F0487C for ; Thu, 14 Jul 2005 02:17:21 +0900 (JST) Received: from beryllium.ruby-lang.org (beryllium.ruby-lang.org [210.163.138.100]) by voscc.nagaokaut.ac.jp (Postfix) with ESMTP id 4BAC7630029 for ; Thu, 14 Jul 2005 02:17:21 +0900 (JST) Received: from localhost (beryllium.ruby-lang.org [127.0.0.1]) by beryllium.ruby-lang.org (Postfix) with ESMTP id 8B63F33D90; Thu, 14 Jul 2005 02:17:20 +0900 (JST) Received: from beryllium.ruby-lang.org ([127.0.0.1]) by localhost (beryllium.ruby-lang.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 04136-06; Thu, 14 Jul 2005 02:17:20 +0900 (JST) Received: from beryllium.ruby-lang.org (beryllium.ruby-lang.org [127.0.0.1]) by beryllium.ruby-lang.org (Postfix) with ESMTP id 7CF0A3398E; Thu, 14 Jul 2005 02:17:19 +0900 (JST) Received: from localhost (beryllium.ruby-lang.org [127.0.0.1]) by beryllium.ruby-lang.org (Postfix) with ESMTP id 5DF8533D90 for ; Thu, 14 Jul 2005 02:17:18 +0900 (JST) Received: from beryllium.ruby-lang.org ([127.0.0.1]) by localhost (beryllium.ruby-lang.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 04137-04 for ; Thu, 14 Jul 2005 02:17:18 +0900 (JST) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by beryllium.ruby-lang.org (Postfix) with SMTP id EC9753398E for ; Thu, 14 Jul 2005 02:17:16 +0900 (JST) Received: (qmail invoked by alias); 13 Jul 2005 17:17:13 -0000 Received: from p54B173C6.dip.t-dialin.net (EHLO [192.168.0.7]) [84.177.115.198] by mail.gmx.net (mp018) with SMTP; 13 Jul 2005 19:17:13 +0200 Delivered-To: ruby-core@ruby-lang.org Date: Thu, 14 Jul 2005 02:17:18 +0900 Posted: Wed, 13 Jul 2005 19:17:11 +0200 From: Stefan Kaes Reply-To: ruby-core@ruby-lang.org Subject: GC tweak To: ruby-core@ruby-lang.org Message-Id: <42D54C97.7050703@gmx.net> X-ML-Name: ruby-core X-Mail-Count: 05445 X-MLServer: fml [fml 4.0.3 release (20011202/4.0.3)]; post only (only members can post) X-ML-Info: If you have a question, send e-mail with the body "help" (without quotes) to the address ruby-core-ctl@ruby-lang.org; help= User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Original-To: ruby-core@ruby-lang.org X-Authenticated: #26035925 X-Accept-Language: en-us, en X-Y-GMX-Trusted: 0 Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------020403070203090704020204" Precedence: bulk List-Id: ruby-core.ruby-lang.org List-Software: fml [fml 4.0.3 release (20011202/4.0.3)] List-Post: List-Owner: List-Help: List-Unsubscribe: X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at ruby-lang.org X-Virus-Scanned: by AMaViS snapshot-20020531 --------------020403070203090704020204 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit I have found that the performance of current garbage collector implementation very much depends on the code size of the application. The behaviour is not linear in the code size. Sometimes adding more code will cause run GC less often, sometimes it will cause more collections, because the code's AST just fits inside the available heap. Especially for long running server applications, like RAILS, this behaviour leaves a lot to be desired. I have created a small patch that enables several GC parameters to be set using environment variables. Most importantly, the initial heap size can be specified (RUBY_HEAP_MIN_SLOTS) and GC statistics can be obtained by setting RUBY_GC_STATS to 1. By tuning the initial heap size and RUBY_GC_MALLOC_LIMIT, I have seen improvements ranging from 10 to 30%. I would really like to see something like this included in 1.8.3 or 1.9, if possible. The patch is farely trivial, and does not change any GC internals. Any comments? Cheers, Stefan Kaes --------------020403070203090704020204 Content-Type: text/plain; name="rubygc.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="rubygc.patch" --- gc.c.orig 2005-06-30 09:53:27.000000000 +0200 +++ gc.c 2005-07-03 16:59:41.815484680 +0200 @@ -21,6 +21,7 @@ #include #include #include +#include #ifdef HAVE_SYS_TIME_H #include @@ -118,6 +119,7 @@ if (malloc_increase > malloc_limit) { garbage_collect(); } + RUBY_CRITICAL(mem = malloc(size)); if (!mem) { garbage_collect(); @@ -308,7 +310,7 @@ static RVALUE *freelist = 0; static RVALUE *deferred_final_list = 0; -#define HEAPS_INCREMENT 10 +static int heaps_increment = 10; static struct heaps_slot { RVALUE *slot; int limit; @@ -316,13 +318,57 @@ static int heaps_length = 0; static int heaps_used = 0; -#define HEAP_MIN_SLOTS 10000 -static int heap_slots = HEAP_MIN_SLOTS; +static int heap_min_slots = 10000; +static int heap_slots = 10000; + +static int heap_free_min = 4096; -#define FREE_MIN 4096 +static long initial_malloc_limit = GC_MALLOC_LIMIT; + +static int gc_stats = 0; static RVALUE *himem, *lomem; +static void set_gc_parameters() +{ + char* min_slots_ptr = getenv("RUBY_HEAP_MIN_SLOTS"); + if (min_slots_ptr != NULL) { + int min_slots_i = atoi(min_slots_ptr); + if (min_slots_i > 0) { + heap_slots = min_slots_i; + heap_min_slots = min_slots_i; + } + } + char* free_min_ptr = getenv("RUBY_HEAP_FREE_MIN"); + if (free_min_ptr != NULL) { + int free_min_i = atoi(free_min_ptr); + if (free_min_i > 0) { + heap_free_min = free_min_i; + } + } + char* heap_incr_ptr = getenv("RUBY_HEAP_INCREMENT"); + if (heap_incr_ptr != NULL) { + int heap_incr_i = atoi(heap_incr_ptr); + if (heap_incr_i > 0) { + heaps_increment = heap_incr_i; + } + } + char* gc_stats_ptr = getenv("RUBY_GC_STATS"); + if (gc_stats_ptr != NULL) { + int gc_stats_i = atoi(gc_stats_ptr); + if (gc_stats_i > 0) { + gc_stats = gc_stats_i; + } + } + char* malloc_limit_ptr = getenv("RUBY_GC_MALLOC_LIMIT"); + if (malloc_limit_ptr != NULL) { + int malloc_limit_i = atol(malloc_limit_ptr); + if (malloc_limit_i > 0) { + initial_malloc_limit = malloc_limit_i; + } + } +} + static void add_heap() { @@ -333,7 +379,7 @@ struct heaps_slot *p; int length; - heaps_length += HEAPS_INCREMENT; + heaps_length += heaps_increment; length = heaps_length*sizeof(struct heaps_slot); RUBY_CRITICAL( if (heaps_used > 0) { @@ -350,10 +396,10 @@ RUBY_CRITICAL(p = heaps[heaps_used].slot = (RVALUE*)malloc(sizeof(RVALUE)*heap_slots)); heaps[heaps_used].limit = heap_slots; if (p == 0) { - if (heap_slots == HEAP_MIN_SLOTS) { + if (heap_slots == heap_min_slots) { rb_memerror(); } - heap_slots = HEAP_MIN_SLOTS; + heap_slots = heap_min_slots; continue; } break; @@ -1031,6 +1077,39 @@ } } +static char* obj_type(int tp) +{ + switch (tp) { + case T_NIL : return "NIL"; + case T_OBJECT : return "OBJECT"; + case T_CLASS : return "CLASS"; + case T_ICLASS : return "ICLASS"; + case T_MODULE : return "MODULE"; + case T_FLOAT : return "FLOAT"; + case T_STRING : return "STRING"; + case T_REGEXP : return "REGEXP"; + case T_ARRAY : return "ARRAY"; + case T_FIXNUM : return "FIXNUM"; + case T_HASH : return "HASH"; + case T_STRUCT : return "STRUCT"; + case T_BIGNUM : return "BIGNUM"; + case T_FILE : return "FILE"; + + case T_TRUE : return "TRUE"; + case T_FALSE : return "FALSE"; + case T_DATA : return "DATA"; + case T_MATCH : return "MATCH"; + case T_SYMBOL : return "SYMBOL"; + + case T_BLKTAG : return "BLKTAG"; + case T_UNDEF : return "UNDEF"; + case T_VARMAP : return "VARMAP"; + case T_SCOPE : return "SCOPE"; + case T_NODE : return "NODE"; + default: return "____"; + } +} + static void gc_sweep() { @@ -1039,6 +1118,14 @@ int i; unsigned long live = 0; + unsigned long really_freed = 0; + int free_counts[256]; + int live_counts[256]; + + if (gc_stats) { + for (i = 0 ; i< 256; i++) { free_counts[i] = live_counts[i] = 0; } + } + if (ruby_in_compile && ruby_parser_stack_on_heap()) { /* should not reclaim nodes during compilation if yacc's semantic stack is not allocated on machine stack */ @@ -1068,6 +1155,9 @@ if (!(p->as.basic.flags & FL_MARK)) { if (p->as.basic.flags) { obj_free((VALUE)p); + if (gc_stats) { + really_freed++; + } } if (need_call_final && FL_TEST(p, FL_FINALIZE)) { p->as.free.flags = FL_MARK; /* remain marked */ @@ -1075,6 +1165,12 @@ final_list = p; } else { + if (gc_stats) { + int obt = p->as.basic.flags & T_MASK; + if (obt) { + free_counts[obt]++; + } + } p->as.free.flags = 0; p->as.free.next = freelist; freelist = p; @@ -1088,10 +1184,13 @@ else { RBASIC(p)->flags &= ~FL_MARK; live++; + if (gc_stats) { + live_counts[RANY((VALUE)p)->as.basic.flags & T_MASK]++; + } } p++; } - if (n == heaps[i].limit && freed > FREE_MIN) { + if (n == heaps[i].limit && freed > heap_free_min) { RVALUE *pp; heaps[i].limit = 0; @@ -1106,14 +1205,28 @@ } if (malloc_increase > malloc_limit) { malloc_limit += (malloc_increase - malloc_limit) * (double)live / (live + freed); - if (malloc_limit < GC_MALLOC_LIMIT) malloc_limit = GC_MALLOC_LIMIT; + if (malloc_limit < initial_malloc_limit) malloc_limit = initial_malloc_limit; } malloc_increase = 0; - if (freed < FREE_MIN) { + if (freed < heap_free_min) { add_heap(); } during_gc = 0; + if (gc_stats) { + fprintf(stderr, "objects processed: %.7d\n", live+freed); + fprintf(stderr, "live objects : %.7d\n", live); + fprintf(stderr, "freelist objects : %.7d\n", freed - really_freed); + fprintf(stderr, "freed objects : %.7d\n", really_freed); + for(i=0; i<256; i++) { + if (free_counts[i]>0) { + fprintf(stderr, + "kept %.7d / freed %.7d objects of type %s\n", + live_counts[i], free_counts[i], obj_type(i)); + } + } + } + /* clear finalization list */ if (final_list) { deferred_final_list = final_list; @@ -1327,6 +1440,12 @@ if (during_gc) return; during_gc++; + struct timeval gctv1, gctv2; + if (gc_stats) { + gettimeofday(&gctv1, NULL); + fprintf(stderr, "Garbage collection started\n"); + } + init_mark_stack(); /* mark frame stack */ @@ -1409,6 +1528,16 @@ } } gc_sweep(); + + if (gc_stats) { + gettimeofday(&gctv2, NULL); + fprintf(stderr, "Garbage collection completed\n"); + gettimeofday(&gctv2, NULL); + fprintf(stderr, "GC time: %d msec\n", + ((gctv2.tv_sec * 1000000 + gctv2.tv_usec) + -(gctv1.tv_sec * 1000000 + gctv1.tv_usec))/1000); + } + } void @@ -1530,6 +1659,7 @@ if (!rb_gc_stack_start) { Init_stack(0); } + set_gc_parameters(); add_heap(); } --------------020403070203090704020204--