[PATCH 0/2] lei doc: mail formats

[PATCH 0/2] lei doc: mail formats

[Eric Wong]

Eric Wong (2):
  doc: add lei-mail-formats(5) manpage
  doc: lei-overview: favor Maildir for mutt examples

 Documentation/lei-mail-formats.pod | 101 +++++++++++++++++++++++++++++
 Documentation/lei-overview.pod     |  11 ++--
 MANIFEST                           |   1 +
 Makefile.PL                        |   4 +-
 lib/PublicInbox/Watch.pm           |   2 +-
 5 files changed, 112 insertions(+), 7 deletions(-)
 create mode 100644 Documentation/lei-mail-formats.pod

[PATCH 1/2] doc: add lei-mail-formats(5) manpage

[Eric Wong]

While plenty of online documentation exists, it's good to have
a locally-available summary for users to look at offline.
Fix a URL in Watch.pm while we're at it, too.
---
 Documentation/lei-mail-formats.pod | 101 +++++++++++++++++++++++++++++
 MANIFEST                           |   1 +
 Makefile.PL                        |   4 +-
 lib/PublicInbox/Watch.pm           |   2 +-
 4 files changed, 106 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/lei-mail-formats.pod

diff --git a/Documentation/lei-mail-formats.pod b/Documentation/lei-mail-formats.pod
new file mode 100644
index 00000000..8accedb4
--- /dev/null
+++ b/Documentation/lei-mail-formats.pod
@@ -0,0 +1,101 @@
+=head1 NAME
+
+lei-mail-formats - description of mail formats supported by lei
+
+=head1 DESCRIPTION
+
+L<lei-q(1)> supports writing to several existing mail formats
+for interoperability with existing mail user agents (MUA);
+below is an overview of them to help users choose.
+
+=head1 Maildir
+
+The default output format when given a filesystem path, it supports
+parallel read-write access.  Performance is acceptable for smaller
+directories, but degrades as mailboxes get larger.  Speed and
+scalability are limited by kernel and filesystem performance
+due to the use of small files and large number of syscalls.
+
+See also: L<https://cr.yp.to/proto/maildir.html> and
+L<https://wiki2.dovecot.org/MailboxFormat/Maildir>
+
+=head1 Mbox family
+
+The mbox family consists of several incompatible formats.
+Locking for parallel access is supported, but may not be
+compatible across tools.  With compression (e.g. L<gzip(1)>),
+they require the least amount of space while offering good
+read-only performance.
+
+Keyword updates (C<Status:> and/or C<X-Status:> headers)
+generally require rewriting the entire mbox.
+
+See also:
+L<https://www.loc.gov/preservation/digital/formats/fdd/fdd000383.shtml>,
+L<mbox(5)>
+
+=head2 mboxo
+
+The traditional BSD format.  It quotes C<From > to C<E<gt>From >,
+but lines already beginning with C<E<gt>From > do not get quoted,
+thus automatic reversibility is not guaranteed.  MUAs which favor
+L</mboxcl> or L</mboxcl2> may convert these automatically to their
+preferred format.
+
+Truncation is undetectable unless compressed with gzip or similar.
+
+=head2 mboxrd
+
+An evolution of L</mboxo>, but quotes C<From > lines prefixed
+with any number of C<E<gt>> characters and is thus fully
+reversible.
+
+This format is emitted by L<PublicInbox::WWW(3pm)> with gzip.
+It is supported by L<git-am(1)> since git 2.10.
+
+As with uncompressed L</mboxo>, uncompressed mboxrd are vulnerable
+to undetectable truncation.
+
+It gracefully degrades to being treated as L</mboxo> by MUAs
+unaware of the format as excessive C<E<gt>From > quoting is
+recognizable to humans.
+
+=head2 mboxcl
+
+L</mboxo> with a C<Content-Length:> header, C<From > lines
+remain quoted to retain readability with L</mboxo> and L</mboxrd> MUAs.
+However, it is easy to corrupt these files when using tools
+which are not aware of C<Content-Length:> and write out updates
+as L</mboxo>.
+
+L<mutt(1)> will convert L</mboxo> and L</mboxrd> to mboxcl upon opening.
+
+See also: L<https://www.jwz.org/doc/content-length.html>
+
+=head2 mboxcl2
+
+Like L</mboxcl>, but without C<From > any quoting.  It is wholly
+incompatible with MUAs which only handle L</mboxo> and/or L</mboxrd>.
+This is format is generated by L<mutt(1)> when writing to a new
+mbox.
+
+=head1 MH
+
+Not yet supported, locking semantics (or lack thereof) appear to
+make it unsuitable for parallel access.
+
+=head1 IMAP
+
+Depending on the IMAP server software and configuration, IMAP
+servers may use any (or combination) of the aforementioned
+formats or a non-standard database backend.
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei(1)>, L<lei-q(1)>, L<lei-convert(1)>, L<lei-overview(7)>
diff --git a/MANIFEST b/MANIFEST
index f3cb0147..49d273fc 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -31,6 +31,7 @@ Documentation/lei-import.pod
 Documentation/lei-init.pod
 Documentation/lei-ls-external.pod
 Documentation/lei-ls-label.pod
+Documentation/lei-mail-formats.pod
 Documentation/lei-overview.pod
 Documentation/lei-p2q.pod
 Documentation/lei-q.pod
diff --git a/Makefile.PL b/Makefile.PL
index 27b49c53..feb89ec1 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -48,7 +48,9 @@ $v->{-m1} = [ map {
 	lei-forget-external lei-import lei-init lei-ls-external lei-ls-label
 	lei-tag lei-p2q lei-q)];
 $v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
-		public-inbox-v2-format public-inbox-extindex-format) ];
+		public-inbox-v2-format public-inbox-extindex-format
+		lei-mail-formats
+		) ];
 $v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
 		public-inbox-glossary) ];
 $v->{-m8} = [ qw(public-inbox-daemon) ];
diff --git a/lib/PublicInbox/Watch.pm b/lib/PublicInbox/Watch.pm
index 4fbc9640..05956cbb 100644
--- a/lib/PublicInbox/Watch.pm
+++ b/lib/PublicInbox/Watch.pm
@@ -2,7 +2,7 @@
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 #
 # ref: https://cr.yp.to/proto/maildir.html
-#	httsp://wiki2.dovecot.org/MailboxFormat/Maildir
+#	https://wiki2.dovecot.org/MailboxFormat/Maildir
 package PublicInbox::Watch;
 use strict;
 use v5.10.1;

[PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples

[Eric Wong]

mboxes are generally horrible for interactive read-write use due
to locking.  Describe our parallel behavior with mutt, since
writing mail can take a long while and being able to read
results as they're written is nice.

We'll also use a gzipped mboxrd for the import example, since
we can decompress gzipped mboxrds automatically, now.
---
 Documentation/lei-overview.pod | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index c1f952c9..70dbf2b5 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -23,9 +23,9 @@ to memoize messages from remotes.
 
 =over
 
-=item $ lei import mboxrd:t.mbox
+=item $ lei import mboxrd:t.mbox.gz
 
-Import the messages from an mbox into the local storage.
+Import the messages from a gzipped mboxrd into the local storage.
 
 =item $ lei blob 59ec517f9
 
@@ -82,10 +82,11 @@ Search for messages whose subject includes "lei" and "skeleton".
 Do the same, but also report unmatched messages that are in the same
 thread as a matched message.
 
-=item $ lei q -t -o mboxcl2:t.mbox --mua=mutt s:lei s:skeleton
+=item $ lei q -t -o mdir --mua=mutt s:lei s:skeleton
 
-Write mboxcl2-formatted results to t.mbox and enter mutt to view the
-file by invoking C<mutt -f %f>.
+Write results to a Maildir at "mdir".  Mutt will be invoked
+to open mfolder (C<mutt -f %f>) while results are being fetched
+and written.
 
 =item $ lei q kw:flagged L:next
 

Re: [PATCH 1/2] doc: add lei-mail-formats(5) manpage

[Kyle Meyer]

Eric Wong writes:

> While plenty of online documentation exists, it's good to have
> a locally-available summary for users to look at offline.

That's great.  Thanks!

(The other patch makes sense to me too.)