user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 2/3] eml: speed up common LF-only emails
Date: Sat,  9 May 2020 08:27:37 +0000
Message-ID: <20200509082738.23602-3-e@yhbt.net> (raw)
In-Reply-To: <20200509082738.23602-1-e@yhbt.net>

Emails a *nix MTA are typically LF-only, so we don't need the
complexity of the RE engine when a simple index() works.  We
still need to ensure there's no "\r\n\r\n" before the first
"\n\n", but two calls to index() is still faster than a RE
match.

This gives a 2-5% speedup in some informal tests and saves ~30MB
when scanning a 30MB spam message on newer versions of Perl.
I'll have to diagnose why Perl wastes so much memory doing
RE matches on giant strings, though.
---
 lib/PublicInbox/Eml.pm | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/lib/PublicInbox/Eml.pm b/lib/PublicInbox/Eml.pm
index 80e7c1af..f022516c 100644
--- a/lib/PublicInbox/Eml.pm
+++ b/lib/PublicInbox/Eml.pm
@@ -71,10 +71,18 @@ sub re_memo ($) {
 # compatible with our uses of Email::MIME
 sub new {
 	my $ref = ref($_[1]) ? $_[1] : \(my $cpy = $_[1]);
-	if ($$ref =~ /\r?\n(\r?\n)/s) { # likely
-		# This can modify $$ref in-place and to avoid memcpy/memmove
-		# on a potentially large $$ref.  It does need to make a
-		# copy for $hdr, though.  Idea stolen from Email::Simple
+	# substr() can modify the first arg in-place and to avoid
+	# memcpy/memmove on a potentially large scalar.  It does need
+	# to make a copy for $hdr, though.  Idea stolen from Email::Simple.
+
+	# We also prefer index() on common LFLF emails since it's faster
+	# and re scan can bump RSS by length($$ref) on big strings
+	if (index($$ref, "\r\n") < 0 && (my $pos = index($$ref, "\n\n")) >= 0) {
+		# likely on *nix
+		my $hdr = substr($$ref, 0, $pos + 2, ''); # sv_chop on $$ref
+		chop($hdr); # lower SvCUR
+		bless { hdr => \$hdr, crlf => "\n", bdy => $ref }, __PACKAGE__;
+	} elsif ($$ref =~ /\r?\n(\r?\n)/s) {
 		my $hdr = substr($$ref, 0, $+[0], ''); # sv_chop on $$ref
 		substr($hdr, -(length($1))) = ''; # lower SvCUR
 		bless { hdr => \$hdr, crlf => $1, bdy => $ref }, __PACKAGE__;

  parent reply index

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-09  8:27 [PATCH 0/3] eml: some fixes and speedups Eric Wong
2020-05-09  8:27 ` [PATCH 1/3] eml: reduce RE captures and possible side effects Eric Wong
2020-05-09  8:27 ` Eric Wong [this message]
2020-05-09  8:27 ` [PATCH 3/3] emlcontentfoo: quiet warning on missing attributes Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200509082738.23602-3-e@yhbt.net \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git