[PATCH 0/3] eml: some fixes and speedups

[PATCH 0/3] eml: some fixes and speedups

[Eric Wong]

Nothing major and the 1/3 side effect fix never affected
any of our use cases.

Eric Wong (3):
  eml: reduce RE captures and possible side effects
  eml: speed up common LF-only emails
  emlcontentfoo: avoid undefined warning on missing attributes

 lib/PublicInbox/Eml.pm           | 26 +++++++++++++++++---------
 lib/PublicInbox/EmlContentFoo.pm |  2 +-
 2 files changed, 18 insertions(+), 10 deletions(-)

[PATCH 1/3] eml: reduce RE captures and possible side effects

[Eric Wong]

Since Perl 5.6, the `@-' (aka @LAST_MATCH_START) and `@+' (aka
@LAST_MATCH_END) arrays provides integer offsets for every match
as documented in perlvar(1), regardless of regexp modifiers.

We can avoid relying on $1 in the epilogue scan, entirely.

So use these instead of relying on m//g and pos(), since the `g'
modifier can be affected by m//g matches performed in other
places.

Unrelated, but while we're in the area: remove some unnecessary
use of (?:...), too.
---
 lib/PublicInbox/Eml.pm | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/PublicInbox/Eml.pm b/lib/PublicInbox/Eml.pm
index 4508bd84..80e7c1af 100644
--- a/lib/PublicInbox/Eml.pm
+++ b/lib/PublicInbox/Eml.pm
@@ -71,11 +71,11 @@ sub re_memo ($) {
 # compatible with our uses of Email::MIME
 sub new {
 	my $ref = ref($_[1]) ? $_[1] : \(my $cpy = $_[1]);
-	if ($$ref =~ /(?:\r?\n(\r?\n))/gs) { # likely
+	if ($$ref =~ /\r?\n(\r?\n)/s) { # likely
 		# This can modify $$ref in-place and to avoid memcpy/memmove
 		# on a potentially large $$ref.  It does need to make a
 		# copy for $hdr, though.  Idea stolen from Email::Simple
-		my $hdr = substr($$ref, 0, pos($$ref), ''); # sv_chop on $$ref
+		my $hdr = substr($$ref, 0, $+[0], ''); # sv_chop on $$ref
 		substr($hdr, -(length($1))) = ''; # lower SvCUR
 		bless { hdr => \$hdr, crlf => $1, bdy => $ref }, __PACKAGE__;
 	} elsif ($$ref =~ /^[a-z0-9-]+[ \t]*:/ims && $$ref =~ /(\r?\n)\z/s) {
@@ -90,8 +90,8 @@ sub new {
 sub new_sub {
 	my (undef, $ref) = @_;
 	# special case for messages like <85k5su9k59.fsf_-_@lola.goethe.zz>
-	$$ref =~ /\A(?:(\r?\n))/gs or goto &new;
-	my $hdr = substr($$ref, 0, pos($$ref), ''); # sv_chop on $$ref
+	$$ref =~ /\A(\r?\n)/s or goto &new;
+	my $hdr = substr($$ref, 0, $+[0], ''); # sv_chop on $$ref
 	bless { hdr => \$hdr, crlf => $1, bdy => $ref }, __PACKAGE__;
 }
 
@@ -132,8 +132,8 @@ sub mp_descend ($$) {
 	# *sigh* just the regexp match alone seems to bump RSS by
 	# length($$bdy) on a ~30M string:
 	my $epilogue_missing;
-	if ($$bdy =~ /((?:\r?\n)?^--$bnd--[ \t]*\r?$)/gsm) {
-		substr($$bdy, pos($$bdy) - length($1)) = '';
+	if ($$bdy =~ /(?:\r?\n)?^--$bnd--[ \t]*\r?$/sm) {
+		substr($$bdy, $-[0]) = '';
 	} else {
 		$epilogue_missing = 1;
 	}

[PATCH 2/3] eml: speed up common LF-only emails

[Eric Wong]

Emails a *nix MTA are typically LF-only, so we don't need the
complexity of the RE engine when a simple index() works.  We
still need to ensure there's no "\r\n\r\n" before the first
"\n\n", but two calls to index() is still faster than a RE
match.

This gives a 2-5% speedup in some informal tests and saves ~30MB
when scanning a 30MB spam message on newer versions of Perl.
I'll have to diagnose why Perl wastes so much memory doing
RE matches on giant strings, though.
---
 lib/PublicInbox/Eml.pm | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/lib/PublicInbox/Eml.pm b/lib/PublicInbox/Eml.pm
index 80e7c1af..f022516c 100644
--- a/lib/PublicInbox/Eml.pm
+++ b/lib/PublicInbox/Eml.pm
@@ -71,10 +71,18 @@ sub re_memo ($) {
 # compatible with our uses of Email::MIME
 sub new {
 	my $ref = ref($_[1]) ? $_[1] : \(my $cpy = $_[1]);
-	if ($$ref =~ /\r?\n(\r?\n)/s) { # likely
-		# This can modify $$ref in-place and to avoid memcpy/memmove
-		# on a potentially large $$ref.  It does need to make a
-		# copy for $hdr, though.  Idea stolen from Email::Simple
+	# substr() can modify the first arg in-place and to avoid
+	# memcpy/memmove on a potentially large scalar.  It does need
+	# to make a copy for $hdr, though.  Idea stolen from Email::Simple.
+
+	# We also prefer index() on common LFLF emails since it's faster
+	# and re scan can bump RSS by length($$ref) on big strings
+	if (index($$ref, "\r\n") < 0 && (my $pos = index($$ref, "\n\n")) >= 0) {
+		# likely on *nix
+		my $hdr = substr($$ref, 0, $pos + 2, ''); # sv_chop on $$ref
+		chop($hdr); # lower SvCUR
+		bless { hdr => \$hdr, crlf => "\n", bdy => $ref }, __PACKAGE__;
+	} elsif ($$ref =~ /\r?\n(\r?\n)/s) {
 		my $hdr = substr($$ref, 0, $+[0], ''); # sv_chop on $$ref
 		substr($hdr, -(length($1))) = ''; # lower SvCUR
 		bless { hdr => \$hdr, crlf => $1, bdy => $ref }, __PACKAGE__;

[PATCH 3/3] emlcontentfoo: quiet warning on missing attributes

[Eric Wong]

This bug was also present in Email::MIME::ContentType:
commit ae081fb576d8507efca4928116ad81efa756c723 (refs/pull/pull/9/head)
in https://github.com/rjbs/Email-MIME-ContentType.git

Our fix is shorter, but dependent on 5.10+ as our codebase
relies on Perl 5.10 features, anyways.
---
 lib/PublicInbox/EmlContentFoo.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/PublicInbox/EmlContentFoo.pm b/lib/PublicInbox/EmlContentFoo.pm
index 7472f8d2..e6005d29 100644
--- a/lib/PublicInbox/EmlContentFoo.pm
+++ b/lib/PublicInbox/EmlContentFoo.pm
@@ -190,7 +190,7 @@ sub _process_rfc2231 {
 	foreach (keys %{$attribs}) {
 		next unless $_ =~ m/^(.*)\*$/;
 		my $key = $1;
-		next unless $attribs->{$_} =~ m/^$re_exvalue$/;
+		next unless ($attribs->{$_} // '') =~ m/^$re_exvalue$/;
 		my ($charset, $value) = ($1, $2);
 		$value =~ s/%([0-9A-Fa-f]{2})/pack('C', hex($1))/eg;
 		if (length $charset) {