user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 2/2] lei_to_mail: optimize for MUAs
Date: Mon, 18 Jan 2021 04:30:32 -0600	[thread overview]
Message-ID: <20210118103032.1430-3-e@80x24.org> (raw)
In-Reply-To: <20210118103032.1430-1-e@80x24.org>

Instead of optimizing our own performance, this optimizes
our data to reduce work done by the MUA consumer.

Maildir and mbox destinations no longer support any notion of
the IMAP \Recent flag.  JMAP has no functioning \Recent
equivalent, and neither do we.

In practice, having MUAs (e.g. mutt) clear the \Recent flag when
committing changes to the mbox is expensive: it creates a
rename(2) storm with Maildir and overwrites the entire mbox.

For mboxcl2 (and mboxcl), we'll further optimize mutt behavior
by setting the Lines: header in addition to Content-Length.

With these changes, mutt exits instantaneously on mboxcl2,
mboxcl, and Maildirs generated by "lei q".
---
 lib/PublicInbox/LeiToMail.pm | 30 ++++++++++++++++++++++--------
 t/lei_to_mail.t              | 16 +++++++++++-----
 t/mbox_reader.t              |  2 ++
 3 files changed, 35 insertions(+), 13 deletions(-)

diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 17d48a90..1281aef4 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -35,7 +35,11 @@ my %kw2status = (
 sub _mbox_hdr_buf ($$$) {
 	my ($eml, $type, $kw) = @_;
 	$eml->header_set($_) for (qw(Lines Bytes Content-Length));
-	my %hdr; # set Status, X-Status
+
+	# Messages are always 'O' (non-\Recent in IMAP), it saves
+	# MUAs the trouble of rewriting the mbox if no other
+	# changes are made
+	my %hdr = (Status => [ 'O' ]); # set Status, X-Status
 	for my $k (@$kw) {
 		if (my $ent = $kw2status{$k}) {
 			push @{$hdr{$ent->[0]}}, $ent->[1];
@@ -93,6 +97,16 @@ sub eml2mboxo {
 	$buf;
 }
 
+sub _mboxcl_common ($$$) {
+	my ($buf, $bdy, $crlf) = @_;
+	# add Lines: so mutt won't have to add it on MUA close
+	my $lines = $$bdy =~ tr!\n!\n!;
+	$$buf .= 'Content-Length: '.length($$bdy).$crlf.
+		'Lines: '.$lines.$crlf.$crlf;
+	substr($$bdy, 0, 0, $$buf); # prepend header
+	$_[0] = $bdy;
+}
+
 # mboxcl still escapes "From " lines
 sub eml2mboxcl {
 	my ($eml, $kw) = @_;
@@ -100,9 +114,7 @@ sub eml2mboxcl {
 	my $crlf = $eml->{crlf};
 	if (my $bdy = delete $eml->{bdy}) {
 		$$bdy =~ s/^From />From /gm;
-		$$buf .= 'Content-Length: '.length($$bdy).$crlf.$crlf;
-		substr($$bdy, 0, 0, $$buf); # prepend header
-		$buf = $bdy;
+		_mboxcl_common($buf, $bdy, $crlf);
 	}
 	$$buf .= $crlf;
 	$buf;
@@ -114,9 +126,7 @@ sub eml2mboxcl2 {
 	my $buf = _mbox_hdr_buf($eml, 'mboxcl2', $kw);
 	my $crlf = $eml->{crlf};
 	if (my $bdy = delete $eml->{bdy}) {
-		$$buf .= 'Content-Length: '.length($$bdy).$crlf.$crlf;
-		substr($$bdy, 0, 0, $$buf); # prepend header
-		$buf = $bdy;
+		_mboxcl_common($buf, $bdy, $crlf);
 	}
 	$$buf .= $crlf;
 	$buf;
@@ -277,7 +287,11 @@ sub _buf2maildir {
 	} while (!sysopen($fh, $tmp, O_CREAT|O_EXCL|O_WRONLY) &&
 		$! == EEXIST && ($rand = int(rand 0x7fffffff).','));
 	if (print $fh $$buf and close($fh)) {
-		$dst .= $sfx eq '' ? 'new/' : 'cur/';
+		# ignore new/ and write only to cur/, otherwise MUAs
+		# with R/W access to the Maildir will end up doing
+		# a mass rename which can take a while with thousands
+		# of messages.
+		$dst .= 'cur/';
 		$rand = '';
 		do {
 			$final = $dst.$rand."oid=$oid:2,$sfx";
diff --git a/t/lei_to_mail.t b/t/lei_to_mail.t
index 083e0df4..e5ac8eac 100644
--- a/t/lei_to_mail.t
+++ b/t/lei_to_mail.t
@@ -25,7 +25,7 @@ for my $mbox (@MBOX) {
 	my $s = $cb->(PublicInbox::Eml->new($from), $kw);
 	is(substr($$s, -1, 1), "\n", "trailing LF in normal $mbox");
 	my $eml = PublicInbox::Eml->new($s);
-	is($eml->header('Status'), 'R', "Status: set by $m");
+	is($eml->header('Status'), 'OR', "Status: set by $m");
 	is($eml->header('X-Status'), 'AF', "X-Status: set by $m");
 	if ($mbox eq 'mboxcl2') {
 		like($eml->body_raw, qr/^From /, "From not escaped $m");
@@ -170,6 +170,12 @@ for my $zsfx (qw(gz bz2 xz)) { # XXX should we support zst, zz, lzo, lzma?
 	}
 }
 
+my $as_orig = sub {
+	my ($eml) = @_;
+	$eml->header_set('Status');
+	$eml->as_string;
+};
+
 unlink $fn or BAIL_OUT $!;
 if ('default deduplication uses content_hash') {
 	my $wcb = $wcb_get->('mboxo', $fn);
@@ -177,7 +183,7 @@ if ('default deduplication uses content_hash') {
 	undef $wcb; # undef to commit changes
 	my $cmp = '';
 	open my $fh, '<', $fn or BAIL_OUT $!;
-	PublicInbox::MboxReader->mboxo($fh, sub { $cmp .= shift->as_string });
+	PublicInbox::MboxReader->mboxo($fh, sub { $cmp .= $as_orig->(@_) });
 	is($cmp, $buf, 'only one message written');
 
 	local $lei->{opt} = { augment => 1 };
@@ -186,7 +192,7 @@ if ('default deduplication uses content_hash') {
 	undef $wcb; # undef to commit changes
 	open $fh, '<', $fn or BAIL_OUT $!;
 	my @x;
-	PublicInbox::MboxReader->mboxo($fh, sub { push @x, shift->as_string });
+	PublicInbox::MboxReader->mboxo($fh, sub { push @x, $as_orig->(@_) });
 	is(scalar(@x), 2, 'augmented mboxo');
 	is($x[0], $cmp, 'original message preserved');
 	is($x[1], $buf . "\nx\n", 'new message appended');
@@ -200,7 +206,7 @@ if ('default deduplication uses content_hash') {
 	undef $wcb; # commit
 	seek($tmp, 0, SEEK_SET) or BAIL_OUT $!;
 	my $cmp = '';
-	PublicInbox::MboxReader->mboxrd($tmp, sub { $cmp .= shift->as_string });
+	PublicInbox::MboxReader->mboxrd($tmp, sub { $cmp .= $as_orig->(@_) });
 	is($cmp, $buf, 'message written to stdout');
 }
 
@@ -213,7 +219,7 @@ SKIP: { # FIFO support
 	$wcb->(\(my $x = $buf), 'deadbeef', []);
 	undef $wcb; # commit
 	my $cmp = '';
-	PublicInbox::MboxReader->mboxo($cat, sub { $cmp .= shift->as_string });
+	PublicInbox::MboxReader->mboxo($cat, sub { $cmp .= $as_orig->(@_) });
 	is($cmp, $buf, 'message written to FIFO');
 }
 
diff --git a/t/mbox_reader.t b/t/mbox_reader.t
index 4ea2ae29..30a5e6e3 100644
--- a/t/mbox_reader.t
+++ b/t/mbox_reader.t
@@ -45,6 +45,8 @@ my $check_fmt = sub {
 	seek($fh, 0, SEEK_SET) or BAIL_OUT "seek: $!";
 	$reader->$fmt($fh, sub {
 		my ($eml) = @_;
+		$eml->header_set('Status');
+		$eml->header_set('Lines');
 		my $cur = shift @order;
 		my @cl = $eml->header_raw('Content-Length');
 		if ($fmt =~ /\Amboxcl/) {

      parent reply	other threads:[~2021-01-18 10:30 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-18 10:30 [PATCH 0/2] lei q: write faster, mutt does less work Eric Wong
2021-01-18 10:30 ` [PATCH 1/2] lei q: parallelize Maildir and mbox writing Eric Wong
2021-01-18 21:19   ` Eric Wong
2021-01-18 10:30 ` Eric Wong [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210118103032.1430-3-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    --subject='Re: [PATCH 2/2] lei_to_mail: optimize for MUAs' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).