user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 10/11] ds: flatten $EXPMAP, delete entries on close
Date: Sun, 12 Jan 2020 21:17:55 +0000
Message-ID: <20200112211756.23100-11-e@yhbt.net> (raw)
In-Reply-To: <20200112211756.23100-1-e@yhbt.net>

We can reduce the amount of small arrayrefs in memory
by flattening $EXPMAP.  This forces us to properly clean
up references during deferred close handling, so NNTP
(and soon HTTP) connections no longer linger until expiry.
---
 lib/PublicInbox/DS.pm | 57 ++++++++++++++++++++++---------------------
 1 file changed, 29 insertions(+), 28 deletions(-)

diff --git a/lib/PublicInbox/DS.pm b/lib/PublicInbox/DS.pm
index fc3c82cc..c434464b 100644
--- a/lib/PublicInbox/DS.pm
+++ b/lib/PublicInbox/DS.pm
@@ -41,7 +41,7 @@ use Carp qw(confess carp);
 my $nextq; # queue for next_tick
 my $wait_pids; # list of [ pid, callback, callback_arg ]
 my $later_queue; # list of callbacks to run at some later interval
-my $EXPMAP; # fd -> [ idle_time, $self ]
+my $EXPMAP; # fd -> idle_time
 our $EXPTIME = 180; # 3 minutes
 my ($later_timer, $reap_timer, $exp_timer);
 my $ToClose; # sockets to close when event loop is done
@@ -259,9 +259,13 @@ sub PostEventLoop () {
 	# loop)
 	if (my $close_now = $ToClose) {
 		$ToClose = undef; # will be autovivified on push
+		@$close_now = map { fileno($_) } @$close_now;
+
+		# order matters, destroy expiry times, first:
+		delete @$EXPMAP{@$close_now};
+
 		# ->DESTROY methods may populate ToClose
-		delete($DescriptorMap{fileno($_)}) for @$close_now;
-		# let refcounting drop everything in $close_now at once
+		delete @DescriptorMap{@$close_now};
 	}
 
 	# by default we keep running, unless a postloop callback cancels it
@@ -642,37 +646,34 @@ sub later ($) {
 }
 
 sub expire_old () {
-    my $now = now();
-    my $exp = $EXPTIME;
-    my $old = $now - $exp;
-    my %new;
-    while (my ($fd, $v) = each %$EXPMAP) {
-        my ($idle_time, $ds_obj) = @$v;
-        if ($idle_time < $old) {
-            if (!$ds_obj->shutdn) {
-                $new{$fd} = $v;
-            }
-        } else {
-            $new{$fd} = $v;
-        }
-    }
-    $EXPMAP = \%new;
-    $exp_timer = scalar(keys %new) ? later(\&expire_old) : undef;
+	my $now = now();
+	my $exp = $EXPTIME;
+	my $old = $now - $exp;
+	my %new;
+	while (my ($fd, $idle_at) = each %$EXPMAP) {
+		if ($idle_at < $old) {
+			my $ds_obj = $DescriptorMap{$fd};
+			$new{$fd} = $idle_at if !$ds_obj->shutdn;
+		} else {
+			$new{$fd} = $idle_at;
+		}
+	}
+	$EXPMAP = \%new;
+	$exp_timer = scalar(keys %new) ? later(\&expire_old) : undef;
 }
 
 sub update_idle_time {
-    my ($self) = @_;
-    my $sock = $self->{sock} or return;
-    $EXPMAP->{fileno($sock)} = [ now(), $self ];
-    $exp_timer //= later(\&expire_old);
+	my ($self) = @_;
+	my $sock = $self->{sock} or return;
+	$EXPMAP->{fileno($sock)} = now();
+	$exp_timer //= later(\&expire_old);
 }
 
 sub not_idle_long {
-    my ($self, $now) = @_;
-    my $sock = $self->{sock} or return;
-    my $ary = $EXPMAP->{fileno($sock)} or return;
-    my $exp_at = $ary->[0] + $EXPTIME;
-    $exp_at > $now;
+	my ($self, $now) = @_;
+	my $sock = $self->{sock} or return;
+	my $idle_at = $EXPMAP->{fileno($sock)} or return;
+	($idle_at + $EXPTIME) > $now;
 }
 
 1;

  parent reply index

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-12 21:17 [PATCH 00/11] ds: various cleanups and fixes Eric Wong
2020-01-12 21:17 ` [PATCH 01/11] ds: new: avoid redundant check, make clobbering fatal Eric Wong
2020-01-12 21:17 ` [PATCH 02/11] ds: add_timer: rename from AddTimer, remove a parameter Eric Wong
2020-01-12 21:17 ` [PATCH 03/11] ds: add an in_loop() function for Inbox.pm use Eric Wong
2020-01-12 21:17 ` [PATCH 04/11] ds: remove Timer->cancel and Timer class+bless Eric Wong
2020-01-12 21:17 ` [PATCH 05/11] ds: guard ToClose against DESTROY side-effects Eric Wong
2020-01-12 21:17 ` [PATCH 06/11] ds|http|nntp: simplify {wbuf} population Eric Wong
2020-01-12 21:17 ` [PATCH 07/11] ds: rely on autovivification for nextq Eric Wong
2020-01-12 21:17 ` [PATCH 08/11] ds: rely on autovivication for waitpid bits Eric Wong
2020-01-12 21:17 ` [PATCH 09/11] ds: rely on autovivification for $later_queue Eric Wong
2020-01-12 21:17 ` Eric Wong [this message]
2020-01-12 21:17 ` [PATCH 11/11] sigfd: simplify loop and improve documentation Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200112211756.23100-11-e@yhbt.net \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git