user/dev discussion of public-inbox itself
 help / color / mirror / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 2/3] nntp: event_step: prepare for async git reads
Date: Tue, 16 Jun 2020 22:31:21 +0000
Message-ID: <20200616223122.7197-3-e@yhbt.net> (raw)
In-Reply-To: <20200616223122.7197-1-e@yhbt.net>

This matches PublicInbox::IMAP::event_step and will allow us to
handle blob retrievals from git asynchronously without falling
over on pipelined requests.
---
 lib/PublicInbox/NNTP.pm        | 39 ++++++++++++++++------------------
 lib/PublicInbox/NNTPdeflate.pm | 10 +++++++++
 2 files changed, 28 insertions(+), 21 deletions(-)

diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index be3bddc3f5d..80dd8614fe8 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -956,38 +956,35 @@ sub out ($$;@) {
 sub event_step {
 	my ($self) = @_;
 
-	return unless $self->flush_write && $self->{sock};
+	return unless $self->flush_write && $self->{sock} && !$self->{long_cb};
 
 	$self->update_idle_time;
 	# only read more requests if we've drained the write buffer,
 	# otherwise we can be buffering infinitely w/o backpressure
 
-	my $rbuf = $self->{rbuf} // (\(my $x = ''));
-	my $r = 1;
-
-	if (index($$rbuf, "\n") < 0) {
-		my $off = bytes::length($$rbuf);
-		$r = $self->do_read($rbuf, LINE_MAX, $off) or return;
-	}
-	while ($r > 0 && $$rbuf =~ s/\A[ \t]*([^\n]*?)\r?\n//) {
-		my $line = $1;
-		return $self->close if $line =~ /[[:cntrl:]]/s;
-		my $t0 = now();
-		my $fd = fileno($self->{sock});
-		$r = eval { process_line($self, $line) };
-		my $pending = $self->{wbuf} ? ' pending' : '';
-		out($self, "[$fd] %s - %0.6f$pending", $line, now() - $t0);
-	}
-
+	my $rbuf = $self->{rbuf} // \(my $x = '');
+	my $line = index($$rbuf, "\n");
+	while ($line < 0) {
+		return $self->close if length($$rbuf) >= LINE_MAX;
+		$self->do_read($rbuf, LINE_MAX, length($$rbuf)) or return;
+		$line = index($$rbuf, "\n");
+	}
+	$line = substr($$rbuf, 0, $line + 1, '');
+	$line =~ s/\r?\n\z//s;
+	return $self->close if $line =~ /[[:cntrl:]]/s;
+
+	my $t0 = now();
+	my $fd = fileno($self->{sock});
+	my $r = eval { process_line($self, $line) };
+	my $pending = $self->{wbuf} ? ' pending' : '';
+	out($self, "[$fd] %s - %0.6f$pending", $line, now() - $t0);
 	return $self->close if $r < 0;
-	my $len = bytes::length($$rbuf);
-	return $self->close if ($len >= LINE_MAX);
 	$self->rbuf_idle($rbuf);
 	$self->update_idle_time;
 
 	# maybe there's more pipelined data, or we'll have
 	# to register it for socket-readiness notifications
-	$self->requeue unless $self->{wbuf};
+	$self->requeue unless $pending;
 }
 
 # for graceful shutdown in PublicInbox::Daemon:
diff --git a/lib/PublicInbox/NNTPdeflate.pm b/lib/PublicInbox/NNTPdeflate.pm
index eb400c9c220..dec88aba3a5 100644
--- a/lib/PublicInbox/NNTPdeflate.pm
+++ b/lib/PublicInbox/NNTPdeflate.pm
@@ -71,6 +71,16 @@ sub do_read ($$$$) {
 	$doff = length($dbuf);
 	my $r = PublicInbox::DS::do_read($self, \$dbuf, $len, $doff) or return;
 
+	# Workaround inflate bug appending to OOK scalars:
+	# <https://rt.cpan.org/Ticket/Display.html?id=132734>
+	# We only have $off if the client is pipelining, and pipelining
+	# is where our substr() OOK optimization in event_step makes sense.
+	if ($off) {
+		my $copy = $$rbuf;
+		undef $$rbuf;
+		$$rbuf = $copy;
+	}
+
 	# assert(length($$rbuf) == $off) as far as NNTP.pm is concerned
 	# -ConsumeInput is true, so $dbuf is automatically emptied
 	my $err = $zin->inflate($dbuf, $rbuf);

  parent reply	other threads:[~2020-06-16 22:31 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-16 22:31 [PATCH 0/3] nntpd: updates from imapd Eric Wong
2020-06-16 22:31 ` [PATCH 1/3] daemon: use ->can to check for IO::Socket::SSL Eric Wong
2020-06-16 22:31 ` Eric Wong [this message]
2020-06-16 22:31 ` [PATCH 3/3] nntp: support slow blob retrievals Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200616223122.7197-3-e@yhbt.net \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ http://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git