diff options
Diffstat (limited to 'Documentation')
95 files changed, 5884 insertions, 361 deletions
diff --git a/Documentation/.gitignore b/Documentation/.gitignore index 92510039..142bce32 100644 --- a/Documentation/.gitignore +++ b/Documentation/.gitignore @@ -1,3 +1,4 @@ +/lei*.txt /public-inbox-*.txt /public-inbox.cgi.txt /standards.txt diff --git a/Documentation/RelNotes/v1.6.1.eml b/Documentation/RelNotes/v1.6.1.eml new file mode 100644 index 00000000..49d2ade0 --- /dev/null +++ b/Documentation/RelNotes/v1.6.1.eml @@ -0,0 +1,57 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.6.1 +Date: Thu, 31 Dec 2020 23:45:56 +0000 +Message-ID: <20201231234556.public-inbox-1.6.1-rele@sed> +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline + +A small, bugfix release on top of 1.6.0 from September 2020. + +Bug fixes: + +* MIME header decoding no longer warns on undefined variables, + with Perl <5.28. Thanks to a bug report by Ali Alnubani. + https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/ + +* Fixed a message threading bug thanks to a report from Kyle Meyer. + "public-inbox-index --rethread --reindex" will be necessary + in case of certain messages arrive out-of-order. + Link: https://public-inbox.org/meta/87360nlc44.fsf@kyleam.com/ + +* WWW: per-inbox grokmirror manifests no longer return info + for all inboxes, only the root /manifest.js.gz includes all + inboxes. This regression appeared in 1.6. + +* public-inbox-mda matches List-Id headers insensitively, + matching public-inbox-watch behavior. Similarly, List-Id + is always indexed lower-cased for boolean matches to avoid + matching an incorrect term. + +* Newsgroup and Path NNTP headers are now emitted in conformance + with RFC 5536 3.1.[45]. Thanks to Andrey Melnikov for the report: + https://public-inbox.org/meta/CA+PODjpUN5Q4gBFQhAzUNuMasVEdmp9f=8Uo0Ej0mFumdSwi4w@mail.gmail.com/ + +* Inotify fixes for public-inbox-imapd users relying on SIGHUP + reloads and thousands of watches. + +* Read-only daemon fixes around TLS and Linux <4.5 systems + +Bugfixes with minor behavior changes: + +* The X-Status mbox header is now excluded from imports, + just like the Status: header has been for many years. + They have no place in public archives and can be privacy + concern for people sharing archives. + +* WWW prevents deep-linking to attachments to limit abuse + vectors. Noticed by Leah Neukirchen: + https://public-inbox.org/meta/87imagyap9.fsf@vuxu.org/ + +There are also several ocumentation fixes from Uwe Kleine-König +and Kyle Meyer. + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. diff --git a/Documentation/RelNotes/v1.7.0.eml b/Documentation/RelNotes/v1.7.0.eml new file mode 100644 index 00000000..45fd0fa6 --- /dev/null +++ b/Documentation/RelNotes/v1.7.0.eml @@ -0,0 +1,101 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.7.0 +Date: Thu, 04 Nov 2021 07:52:00 +0000 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Message-ID: <2021-11-04-public-inbox-1.7.0-finally-rele@sed> +Content-Disposition: inline + +Another big release focused on multi-inbox search and scalability. + +Special thanks to Konstantin Ryabitsev and Kyle Meyer for +numerous bug reports and documentation help. + +* general changes + + - config file parsing is 2x faster with 50K inboxes + + - deduplication ignores whitespace differences within address fields + + - "PRAGMA optimize" is now issued on commits for SQLite 3.18+ + +* public-inbox-extindex + + A new Xapian + SQLite index able to search across several inboxes. + This may be configured to replace per-inbox Xapian DBs, + (but not per-inbox SQLite indices) and speed up manifest.js.gz + generation. + + See public-inbox-extindex-format(5) and + public-inbox-extindex(1) manpages for more details. + + Using it with "--all" speeds up various multi-inbox operations in + PublicInbox::WWW, public-inbox-nntpd, and public-inbox-imapd. + +* read-only public-inbox-daemon (-httpd, -nntpd, -imapd): + + libgit2 may be used via Inline::C to avoid hitting system pipe + and process limits. See public-inbox-tuning(7) manpage + for more details. + +* various memory usage reductions and workarounds for leaks in + Encode <3.15, these mainly affect PublicInbox::WWW + +* public-inbox-nntpd + + - startup is 6x faster with 50K inboxes if using -extindex + +* PublicInbox::WWW + + - mboxrd search results are returned in reverse Xapian docid order, + so more recent results are more likely to show up first + + - d: and dt: search prefixes allow "approxidate" formats supported + by "git log --since=" + + - manifest.js.gz generation is ~25x faster with -extindex + + - minor navigation improvements in search results HTML page + +* lei - local email interface + + An experimental, subject-to-change, likely-to-eat-your-mail tool for + personal mail as well as interacting with public-inboxes on the local + filesystem or over HTTP(S). See lei(1), lei-overview(7), and other + lei-* manpages for details. This is only ready-to-use w.r.t. external + public-inbox instances, but mail synchronization for personal mail + remains clunky. + +* public-inbox-index + + - non-strict (Subject-based) threading supports non-ASCII characters, + reindexing is necessary for old messages with non-ASCII subjects. + + - --batch-size is now 8M on 64-bit systems for throughput improvements, + higher values are still advised for more powerful hardware. + +* public-inbox-watch + + - IMAP and NNTP code shared with lei, fixing an off-by-one error + in IMAP synchronization for single-message IMAP folders. + + - \Deleted and \Draft messages ignored for IMAP, as they are for + Maildir. + + - IMAP and NNTP connection establishment (including git-credential + prompts) ordering is now tied to config file order. + +Compatibility: + +* Rollbacks all the way to public-inbox 1.2.0 remain supported + +Internal changes + +* public-inbox-index switched to new internal IPC code shared + with lei + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v1.7.0.wip b/Documentation/RelNotes/v1.7.0.wip deleted file mode 100644 index a35ff227..00000000 --- a/Documentation/RelNotes/v1.7.0.wip +++ /dev/null @@ -1,16 +0,0 @@ -To: meta@public-inbox.org -Subject: [WIP] public-inbox 1.7.0 -MIME-Version: 1.0 -Content-Type: text/plain; charset=utf-8 -Content-Disposition: inline - -TODO: gcf2, detached indices, JMAP, ... - -Compatibility: - -* Rollbacks all the way to public-inbox 1.2.0 remain supported - -Please report bugs via plain-text mail to: meta@public-inbox.org - -See archives at https://public-inbox.org/meta/ for all history. -See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v1.8.0.eml b/Documentation/RelNotes/v1.8.0.eml new file mode 100644 index 00000000..d835f8d8 --- /dev/null +++ b/Documentation/RelNotes/v1.8.0.eml @@ -0,0 +1,47 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.8.0 +Date: Sat, 23 Apr 2022 08:22:59 +0000 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline +Message-ID: <2022-04-23-public-inbox-1.8.0-released@finally> + +A minor release focused on bugfixes and minor improvements. +Upgrades should happen transparently, but downgrading back to +1.7.0 will likely cause problems for lei users (and only lei +users). + +lei users may experience duplicate messages in Maildirs if attempting to +downgrade from 1.8.0 to 1.7.x. public-inbox-* tools are unaffected and +may downgrade freely. + +Bugfixes: + + Numerous test fixes thanks to NixOS developers. + + Long-running daemons are more robust in case of corrupt blobs + or crashes of git-cat-file processes + + PublicInbox::WWW: all CR are removed before LF, fixing display of + CR-CR-LF messages. + + Solver supports SHA-256 code repositories (inbox and lei store support + is still pending). + +Internal updates: + + Reduced dependencies on Inline::C for Linux users; Linux users may + now use lei with neither Inline::C nor Socket::MsgHdr installed. + +New features: + + The --dangerous flag is now supported in public-inbox-index and + public-inbox-extindex to use the Xapian::DB_DANGEROUS flag for initial + indexes. This may reduce SSD/HDD wear at the expense of disallowing + concurrency and data integrity in case of an unexpected shutdown. + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v1.9.0.eml b/Documentation/RelNotes/v1.9.0.eml new file mode 100644 index 00000000..08e16a66 --- /dev/null +++ b/Documentation/RelNotes/v1.9.0.eml @@ -0,0 +1,68 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.9.0 +Date: Sun, 21 Aug 2022 02:36:59 +0000 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline +Message-ID: <2022-08-21T023659Z-public-inbox-1.9.0-rele@sed> + +Upgrading: + + lei users need to "lei daemon-kill" after installation to load + new code. Normal daemons (read-only, and public-inbox-watch) + will also need restarts, of course, but there's no + backwards-incompatible data format changes so rolling back to + older versions is harmless. + +Major bugfixes: + + * lei no longer freezes from inotify/EVFILT_VNODE handling, + user interrupts (Ctrl-C), nor excessive errors/warnings + + * IMAP server fairness improved to avoid excessive blob prefetch + +New features: + + * POP3 server support added, use either public-inbox-pop3d or + the new public-inbox-netd superserver + + * public-inbox-netd superserver supporting any combination of HTTP, + IMAP, POP3, and NNTP services; simplifying management and allowing + more sharing of memory used for various data structures. + + * public-inbox-httpd and -netd support per-listener .psgi files + + * SIGHUP reloads TLS certs and keys in addition to config and .psgi files + + * "lei reindex" command for lei users to update personal index + in ~/.local/share/lei/store for search improvements below: + +Search improvements: + + These will require --reindex with public-inbox-index and/or + public-inbox-extindex for public inboxes. + + * patchid: prefix search support added to WWW and lei for + "git patch-id --stable" support + + * text inside base-85 binary patches is no longer indexed + to avoid false positives + + * for lei users, "lei reindex" now exists and is required + to take advantage of aforementioned indexing changes + +Performance improvements: + + * IMAP server startup is faster with many mailboxes when using + "public-inbox-extindex --all" + + * NNTP group listings are also faster with many inboxes when + using "public-inbox-extindex --all" + + * various small opcode and memory usage reductions + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v2.0.0.wip b/Documentation/RelNotes/v2.0.0.wip new file mode 100644 index 00000000..794d7956 --- /dev/null +++ b/Documentation/RelNotes/v2.0.0.wip @@ -0,0 +1,193 @@ +To: meta@public-inbox.org +Subject: [WIP] public-inbox 2.2.0 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline + +This release includes several new features and fixes; mostly +around improved integration between inboxes and coderepos for +solver. Portability and reliability is also improved, especially +in the internal process management of lei. + +public-inbox-cindex is a new command to index coderepos for +WWW search and perform automatic associations between +coderepos and inboxes. This makes solver vastly more useful +for the WWW UI as admins will no longer have to manually +associate coderepos with inboxes. + +public-inbox-clone gains the ability to mirror entire (or partial) +grokmirror-compatible manifests. + +Internal process and object management data structures are vastly +simplified throughout and error handling made more robust. + +git SHA-256 support remains a work-in-progress for inboxes and +extindex due to the need to interoperate with SHA-1 epochs. + +Upgrading: + + lei users need to "lei daemon-kill" after installation to load + new code. Normal daemons (read-only, and public-inbox-watch) + will also need restarts, of course, but there's no + backwards-incompatible data format changes so rolling back to + older versions is harmless. + +Compatibility: + + Uppercase newsgroup names were always broken with IMAP, POP3, and + -extindex. Uppercase names will now be lowercased by default and + warnings will be emitted. Conflicting newsgroup names (and `inboxdir' + entries if `newsgroup' isn't specified) will also generate warnings + since they break -extindex and the new -cindex (coderepo index). + +New users + hackers: + + The install/ directory includes tools to automate installation and + removal of dependencies for stripped-down or full setups. See + install/README for more details. + +treewide + + * support raw UTF-8 headers from SMTPUTF8 hosts + + * standardize on `#' prefix for stderr diagnostics (previously `I:') + + * SHA-256 coderepos are fully supported (but not inboxes, yet) + + * for daemons serving public traffic, MALLOC_MMAP_THRESHOLD_=131072 is + recommended to reduce fragmentation in glibc malloc, while jemalloc + (tested as an LD_PRELOAD) is another option. + +PublicInbox::WWW + + * support `+' in inbox names + + * support coderepo displays for systems without cgit + + * improve display of git tags, commits and trees in $INBOX/$OID/s/ endpoint + + * numerous memory usage reductions by avoiding Perl scratchpads + + * add #related anchor and search form to find related patches + based on blob OIDs (IOW, exposing `lei p2q' to the web) + + * fix footer in listing of >200 inboxes + + * support dumb HTTP clones of SHA-256 git repos + + * add /$INBOX/$MSGID/d/ endpoint to show diffs in reused Message-IDs + (`lei mail-diff' for the web) + + * support POST /$INBOX/$MSGID/?x=m&q= to limit mbox results to a thread + + * add topics_(new|active).(html|atom) endpoints + + * linkify peer public-inbox addresses in To/Cc headers + +public-inbox-watch: + + * watching MH folders is now supported + +lei + + * use http.proxy / http.<remote>.proxy from system-wide git-config if + unconfigured for lei + + * improve IMAP and NNTP error reporting + + * reduce default IMAP connections to avoid overloading servers + + * compatibility with SQLite <3.8.3 on CentOS 7.x + + * fix `lei q -tt' on locally indexed messages (still broken for remotes: + https://public-inbox.org/meta/20230226170931.M947721@dcvr/ ) + + * `lei import' now sets labels+keywords consistently on all + already imported messages + + * fix `lei up' on saved local queries which previously used -t/--threads + + * `lei convert' output to v2 public-inboxes is now idempotent + + * improved bash completion for labels (see contrib/completion) + + * support for reading (but not writing) MH folders + + * `lei index' accepts `+L:$LABEL' like `lei import' does + +solver (used by lei (rediff|blob), and PublicInbox::WWW) + + * handle copies in patches properly + + * no longer redundantly parallelized within each WWW process + +portability + + * SIGWINCH is handled properly on less common architectures and OSes + + * fix EINTR handling for kqueue users + + * various fixes for CentOS 7.x + + * fix excessive pipelining to `git cat-file' on systems with small + getdelim(3) buffers (mainly affects musl) + + * support Alpine Linux, Dragonfly, NetBSD and OpenBSD. This resulted + not only in bugfixes to our code, but also to Dragonfly and OpenBSD. + + * Inline::C||Socket::MsgHdr no longer required for SCM_RIGHTS + with sendmsg/recvmsg on supported *BSDs. + + * inotify support no longer requires Linux::Inotify2 XS package + for most architectures + +public-inbox-pop3d + + * support `limit=NUM' and `initial_limit=NUM' query parameters + in mailbox names to limit results + +public-inbox-nntpd + + * fix LISTGROUP with range (affects neomutt) + +public-inbox-clone / public-inbox-fetch / `lei add-external --mirror' + + * mtime of downloaded manifest preserved + +public-inbox-clone: + + * parallel mirroring of multiple inboxes/coderepos via manifest, + public-inbox-fetch is not used in this mode + + * new flags to support manifest mirroring include: + --dry-run, --inbox-config=, --project-list=, --prune, --purge, + --keep-going, --jobs, --include=, --exclude=, --objstore=, + --manifest=, --remote-manifest= + See public-inbox-clone(1) man page for more details. + +PublicInbox::SaPlugin::ListMirror + + * List-ID handling special-cased according to RFC 2919 rules + +Search improvements (lei and PublicInbox::WWW) + + * quoted text inside base-85 binary patches is no longer indexed + + * `public-inbox-cindex --join' prefers using Xapian's C++ API + directly to avoid Perl method dispatch overhead to get usable + performance associating ~300 inboxes with over 1K coderepos + (and vice versa). Users requiring such performance will need + a C++ compiler, pkg-config, and the Xapian development files + (see INSTALL). + + This C++ helper will be used more heavily in the future + to enable query parser customizations and other functionality + unavailable from the Xapian SWIG or XS bindings. + +Thanks to all the bug reporters and users who made this release +possible, and thanks for bearing with my anxiety over making releases. + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/common.perl b/Documentation/common.perl new file mode 100755 index 00000000..3a6617c4 --- /dev/null +++ b/Documentation/common.perl @@ -0,0 +1,69 @@ +#!perl -w +# Copyright (C) all contributors <meta@public-inbox.org> +# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> +use strict; +use Fcntl qw(SEEK_SET); +my $have_search = eval { require PublicInbox::Search; 1 }; +my $addr = 'meta@public-inbox.org'; +for my $pod (@ARGV) { + open my $fh, '+<', $pod or die "open($pod): $!"; + my $s = do { local $/; <$fh> } // die "read $!"; + my $orig = $s; + $s =~ s!^=head1 COPYRIGHT\n.+?^=head1([^\n]+)\n!=head1 COPYRIGHT + +Copyright all contributors L<mailto:$addr> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1$1 + !ms; + + $s =~ s!^=head1 CONTACT\n.+?^=head1([^\n]+)\n!=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:$addr> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1$1 + !ms; + $have_search and $s =~ s!^=for\scomment\n + ^AUTO-GENERATED-SEARCH-TERMS-BEGIN\n + .+? + ^=for\scomment\n + ^AUTO-GENERATED-SEARCH-TERMS-END\n + !search_terms()!emsx; + $s =~ s/[ \t]+$//sgm; + if ($s eq $orig) { + my $t = time; + utime($t, $t, $fh); + } else { + seek($fh, 0, SEEK_SET) or die "seek: $!"; + truncate($fh, 0) or die "truncate: $!"; + print $fh $s or die "print: $!"; + close $fh or die "close: $!"; + } +} + +sub search_terms { + my $help = eval('\@PublicInbox::Search::HELP'); + my $s = ''; + my $pad = 0; + my $i; + for ($i = 0; $i < @$help; $i += 2) { + my $pfx = $help->[$i]; + my $n = length($pfx); + $pad = $n if $n > $pad; + $s .= $pfx . "\0"; + $s .= $help->[$i + 1]; + $s .= "\f\n"; + } + $pad += 2; + my $padding = ' ' x ($pad + 4); + $s =~ s/^/$padding/gms; + $s =~ s/^$padding(\S+)\0/" $1".(' ' x ($pad - length($1)))/egms; + $s =~ s/\f\n/\n/gs; + $s =~ s/^ //gms; + substr($s, 0, 0, "=for comment\nAUTO-GENERATED-SEARCH-TERMS-BEGIN\n\n"); + $s .= "\n=for comment\nAUTO-GENERATED-SEARCH-TERMS-END\n"; +} diff --git a/Documentation/dc-dlvr-spam-flow.txt b/Documentation/dc-dlvr-spam-flow.txt index d151d272..6210fc7d 100644 --- a/Documentation/dc-dlvr-spam-flow.txt +++ b/Documentation/dc-dlvr-spam-flow.txt @@ -39,7 +39,7 @@ delivery path as well as removing the message from the git tree. * incron - run commands based on filesystem events: http://incron.aiken.cz/ -* sendmail / MTA - we use and recommend use postfix, which includes a +* sendmail / MTA - we use and recommend postfix, which includes a sendmail-compatible wrapper: http://www.postfix.org/ * spamc / spamd - SpamAssassin: http://spamassassin.apache.org/ diff --git a/Documentation/design_notes.txt b/Documentation/design_notes.txt index e871f4c8..95f02556 100644 --- a/Documentation/design_notes.txt +++ b/Documentation/design_notes.txt @@ -18,7 +18,7 @@ Use existing infrastructure * public-inbox can coexist with existing mailing lists, any subscriber to the existing mailing list can begin delivering messages to - public-inbox-mda(1) + public-inbox-mda(1) or public-inbox-watch(1) * public-inbox uses SMTP for posting. Posting a message to a public-inbox instance is no different than sending a message to any _open_ mailing @@ -52,13 +52,15 @@ Why email? There is no need to ask the NSA for backups of your mail archives :) * git, one of the most widely-used version control systems, includes many - tools for for email, including: git-format-patch(1), git-send-email(1), + tools for email, including: git-format-patch(1), git-send-email(1), git-am(1), git-imap-send(1). Furthermore, the development of git itself is based on the git mailing list: https://public-inbox.org/git/ - (or http://hjrcffqmbrq6wope.onion/git/ for Tor users) + (or + http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/git/ + for Tor users). * Email is already the de-facto form of communication in many Free Software - communities.. + communities. * Fallback/transition to private email and other lists, in case the public-inbox host becomes unavailable, users may still directly email @@ -70,16 +72,17 @@ Why git? * git is distributed and robust while being both fast and space-efficient with text data. NNTP was considered, but does not support delta-compression and places no guarantees on data/transport - integrity. However, a read-only NNTP gateway is implemented. + integrity. However, read-only IMAP and NNTP gateways are implemented. * As of 2016, git is widely used and known to nearly all Free Software developers. For non-developers it is packaged for all major GNU/Linux - and *BSD distributions. NNTP is not as widely-used nowadays. + and *BSD distributions. NNTP is not as widely used nowadays, and + most IMAP clients do not have good support for read-only mailboxes. Why perl 5? ----------- -* Perl 5 is widely available on modern *nix systems with good a history +* Perl 5 is widely available on modern *nix systems, with a good history of backwards and forward compatibility. * git and SpamAssassin both use it, so it should be one less thing for @@ -140,6 +143,8 @@ What sucks about public-inbox * some (mostly GUI) mail clients cannot set In-Reply-To headers properly without the original message. +* marketing - as it should: <https://public-inbox.org/marketing.txt> + Scalability notes ----------------- @@ -149,5 +154,5 @@ problems solved. Copyright --------- -Copyright 2013-2020 all contributors <meta@public-inbox.org> +Copyright all contributors <meta@public-inbox.org> License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/design_www.txt b/Documentation/design_www.txt index b1f916dd..a0003f99 100644 --- a/Documentation/design_www.txt +++ b/Documentation/design_www.txt @@ -7,7 +7,7 @@ URL and anchor naming /$INBOX/?r=$GIT_COMMIT -> HTML only /$INBOX/new.atom -> Atom feed -#### Optional, relies on Search::Xapian (or Xapian SWIG binding) +#### Optional, relies on Xapian /$INBOX/$MESSAGE_ID/t/ -> HTML content of thread (nested) /$INBOX/$MESSAGE_ID/T/ -> HTML content of thread (flat) anchors: @@ -102,7 +102,7 @@ We also set <title> to make window management easier. We favor <pre>-formatted text since public-inbox is intended as a place to share and discuss patches and code. Unfortunately, long paragraphs -tends to be less readable with fixed-width serif fonts which GUI +tend to be less readable with fixed-width serif fonts which GUI browsers default to. * No graphics, images, or icons at all. We tolerate, but do not @@ -122,12 +122,12 @@ browsers default to. avoided as they do not render well with some displays or user-chosen fonts. -* No JavaScript. JS is historically too buggy and insecure, and we will +* No JavaScript. JS is historically too buggy and insecure, and we will never expect our readers to do either of the following: - a) read and audit all our code for on every single page load - b) trust us and and run code without reading it + a) read and audit all our code on every single page load + b) trust us and run code without reading it -* We only use CSS for one reason: wrapping pre-formatted text +* We only use CSS for one reason: wrapping pre-formatted text. This is necessary because unfortunate GUI browsers tend to be prone to layout widening from unwrapped mailers. Do not expect CSS to be enabled, especially with scary things like: @@ -141,4 +141,4 @@ CSS classes (for user-supplied CSS) ----------------------------------- See examples in contrib/css/ and lib/PublicInbox/WwwText.pm -(or https://public-inbox.org/meta/_/text/color/ soon) +(or <https://public-inbox.org/meta/_/text/color/>) diff --git a/Documentation/extman.perl b/Documentation/extman.perl deleted file mode 100755 index a9a830c0..00000000 --- a/Documentation/extman.perl +++ /dev/null @@ -1,33 +0,0 @@ -#!/usr/bin/perl -w -# Copyright (C) 2019-2020 all contributors <meta@public-inbox.org> -# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> -# prints a manpage to stdout -use strict; -my $xapmsg = 'See https://xapian.org/ for more information on Xapian'; -my $usage = "$0 /path/to/manpage.SECTION.txt"; -my $manpage = shift or die $usage; -my $MAN = $ENV{MAN} || 'man'; -my @args; -$manpage = (split('/', $manpage))[-1]; -$manpage =~ s/\.txt\z//; -$manpage =~ s/\A\.//; # no leading dot (see Documentation/include.mk) -$manpage =~ s/\.(\d+.*)\z// and push @args, $1; # section -push @args, $manpage; - -# don't use UTF-8 characters which readers may not have fonts for -$ENV{LC_ALL} = $ENV{LANG} = 'C'; -$ENV{COLUMNS} = '76'; # same as pod2text default -$ENV{PAGER} = 'cat'; -my $cmd = join(' ', $MAN, @args); -system($MAN, @args) and die "$cmd failed: $!\n"; -$manpage =~ /\A(?:copydatabase|xapian-compact)\z/ and - print "\n\n", $xapmsg, "\n"; - -# touch -r $(man -w $section $manpage) output.txt -if (-f \*STDOUT) { - open(my $fh, '-|', $MAN, '-w', @args) or die "$MAN -w broken?: $!\n"; - chomp(my $path = <$fh>); - my @st = stat($path) or die "stat($path) failed: $!\n"; - # 9 - mtime - utime($st[9], $st[9], \*STDOUT) or die "utime(STDOUT) failed: $!\n"; -} diff --git a/Documentation/flow.ge b/Documentation/flow.ge index 0cc1c333..5ad92fec 100644 --- a/Documentation/flow.ge +++ b/Documentation/flow.ge @@ -1,9 +1,11 @@ # public-inbox data flow # # Note: choose either "delivery tools" OR "git mirroring tools" -# for a given inboxdir. Combining them for the SAME inboxdir -# will cause conflicts. Of course, different inboxdirs may -# choose different means of getting mail into them. +# for a given inboxdir. Using them simultaneously for the +# SAME inboxdir will cause conflicts. Of course, different +# inboxdirs may choose different means of getting mail into them. +# You may fork any inbox by starting with "git mirroring tools", +# and switching to "delivery tools". graph { flow: down } @@ -13,6 +15,8 @@ graph { flow: down } public-inbox-learn] -> [inboxdir] [git mirroring tools:\n + public-inbox-clone,\n + public-inbox-fetch,\n grok-pull,\n various scripts ] -- git (clone|fetch) &&\n @@ -20,9 +24,10 @@ graph { flow: down } [inboxdir] -> [read-only daemons:\n + public-inbox-netd\n public-inbox-httpd\n public-inbox-imapd\n public-inbox-nntpd] -# Copyright 2020 all contributors <meta@public-inbox.org> +# Copyright all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/flow.txt b/Documentation/flow.txt index 8225cc8f..ed2dd80b 100644 --- a/Documentation/flow.txt +++ b/Documentation/flow.txt @@ -1,9 +1,11 @@ # public-inbox data flow # # Note: choose either "delivery tools" OR "git mirroring tools" -# for a given inboxdir. Combining them for the SAME inboxdir -# will cause conflicts. Of course, different inboxdirs may -# choose different means of getting mail into them. +# for a given inboxdir. Using them simultaneously for the +# SAME inboxdir will cause conflicts. Of course, different +# inboxdirs may choose different means of getting mail into them. +# You may fork any inbox by starting with "git mirroring tools", +# and switching to "delivery tools". +--------------------+ | delivery tools: | @@ -15,8 +17,10 @@ | v +----------------------+ +--------------------+ -| git mirroring tools: | git (clone|fetch) && | | -| grok-pull, | public-inbox-index | inboxdir | +| git mirroring tools: | | | +| public-inbox-clone, | | | +| public-inbox-fetch, | git (clone|fetch) && | inboxdir | +| grok-pull, | public-inbox-index | | | various scripts | ----------------------> | | +----------------------+ +--------------------+ | @@ -24,10 +28,12 @@ v +--------------------+ | read-only daemons: | + | public-inbox-netd | | public-inbox-httpd | | public-inbox-imapd | | public-inbox-nntpd | +--------------------+ -# Copyright 2020 all contributors <meta@public-inbox.org> +# Copyright all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> +# This file was generated from flow.txt using Graph::Easy diff --git a/Documentation/hosted.txt b/Documentation/hosted.txt deleted file mode 100644 index 188ad254..00000000 --- a/Documentation/hosted.txt +++ /dev/null @@ -1,41 +0,0 @@ -unofficially hosted mirrors at public-inbox.org - -In addition to eating our own dogfood at <https://public-inbox.org/meta/>, -public-inbox.org hosts unofficial archives for several other projects -to further test our own software. - -These mirrors are NOT to be considered reliable or permanent. -Interested parties are strongly encouraged to host their own mirrors. - -The presence of these archives does not imply these projects endorse -public-inbox or public-inbox.org in any way. - -* https://public-inbox.org/bug-gnulib/ - bug-gnulib@gnu.org - Discussion for Gnulib portability/common source project - https://lists.gnu.org/mailman/listinfo/bug-gnulib - -* https://public-inbox.org/git/ - git@vger.kernel.org - Mailing list for the git version control system - http://vger.kernel.org/majordomo-info.html - -* https://public-inbox.org/libc-alpha/ - libc-alpha@sourceware.org - Mailing list for GNU C library development - https://www.gnu.org/software/libc/involved.html - -* https://public-inbox.org/rack-devel/ - rack-devel@googlegroups.com - Development list for the Ruby webserver interface - https://groups.google.com/group/rack-devel - -* https://public-inbox.org/sox-users/ - sox-users@lists.sourceforge.net - Users' list for the SoX sound processing tool - https://lists.sourceforge.net/lists/listinfo/sox-users - -* https://public-inbox.org/sox-devel/ - sox-devel@lists.sourceforge.net - Developers' list for the SoX sound processing tool - https://lists.sourceforge.net/lists/listinfo/sox-devel diff --git a/Documentation/include.mk b/Documentation/include.mk index 207983f0..86851376 100644 --- a/Documentation/include.mk +++ b/Documentation/include.mk @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2020 all contributors <meta@public-inbox.org> +# Copyright (C) all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> all:: @@ -7,23 +7,22 @@ RSYNC_DEST = public-inbox.org:/srv/public-inbox/ AWK = awk MAN = man +# part of `man-db' on Debian, not sure about other distros +LEXGROG = lexgrog + # this is "xml" on FreeBSD and maybe some other distros: XMLSTARLET = xmlstarlet # libgraph-easy-perl from Debian, Graph::Easy from CPAN GRAPH_EASY = graph-easy -# same as pod2text -COLUMNS = 76 - INSTALL = install PODMAN = pod2man PODMAN_OPTS = -v --stderr -d 1993-10-02 -c 'public-inbox user manual' PODMAN_OPTS += -r public-inbox.git podman = $(PODMAN) $(PODMAN_OPTS) -PODTEXT = pod2text -PODTEXT_OPTS = --stderr -podtext = $(PODTEXT) $(PODTEXT_OPTS) + +man2text = COLUMNS=80 MANWIDTH=80 TERM=dumb MANOPT='--nj --nh' man all:: man @@ -50,11 +49,14 @@ install-man: man doc_install :: install-man -check :: check-man -check_man = $(AWK) '{gsub(/\b./,"")}length>80{print;err=1}END{exit(err)}'\ - >&2 && >$@ +check : check-man +check_man = $(AWK) \ + '{gsub(/\b./,"")}$$0 !~ /\.onion/&&length>80{print;e=1}END{exit(e)}' \ + >&2 + +check-man : $(check_80) -check-man :: $(check_80) +check-lexgrog : $(check_lexgrog) all :: $(docs) @@ -67,20 +69,26 @@ Documentation/standards.txt : Documentation/standards.perl # flow.txt is checked into git since Graph::Easy isn't in many distros Documentation/flow.txt : Documentation/flow.ge - (sed -ne '1,/^$$/p' <Documentation/flow.ge; \ - $(GRAPH_EASY) Documentation/flow.ge || \ - cat Documentation/flow.txt; \ + +%.txt : %.ge + (sed -ne '1,/^$$/p' <$<; \ + $(GRAPH_EASY) $< || grep -v '^#' $@; \ echo; \ - sed -ne '/^# Copyright/,$$p' <Documentation/flow.ge \ + sed -ne '/^# Copyright/,$$p' <$< \ ) >$@+ - touch -r Documentation/flow.ge $@+ + echo >>$@+ \ + '# This file was generated from $(@F) using Graph::Easy' + touch -r $< $@+ mv $@+ $@ +Documentation/lei-q.pod : lib/PublicInbox/Search.pm Documentation/common.perl + $(PERL) -I lib -w Documentation/common.perl $@ + NEWS NEWS.atom NEWS.html : $(news_deps) $(PERL) -I lib -w Documentation/mknews.perl $@ $(RELEASES) # check for internal API changes: -check :: NEWS .NEWS.atom.check NEWS.html +check : NEWS .NEWS.atom.check NEWS.html .NEWS.atom.check: NEWS.atom $(XMLSTARLET) val NEWS.atom || \ @@ -98,18 +106,16 @@ doc: $(docs) gz-doc: $(gz_docs) -gz-xdoc: $(gz_xdocs) - rsync-doc: NEWS.atom.gz # /usr/share/doc/rsync/scripts/git-set-file-times{.gz} on Debian systems # It is also at: https://yhbt.net/git-set-file-times -git set-file-times $(docs) $(txt) - $(MAKE) gz-doc gz-xdoc - $(RSYNC) --chmod=Fugo=r -av $(rsync_docs) $(rsync_xdocs) $(RSYNC_DEST) + $(MAKE) gz-doc + $(RSYNC) --chmod=Fugo=r -av $(rsync_docs) $(RSYNC_DEST) clean-doc: $(RM_F) $(man1) $(man5) $(man7) $(man8) $(gz_docs) $(docs_html) \ - $(mantxt) $(rsync_xdocs) \ + $(mantxt) \ NEWS NEWS.atom NEWS.html Documentation/standards.txt \ Documentation/flow.html Documentation/flow.html.gz \ Documentation/flow.txt.gz diff --git a/Documentation/lei-add-external.pod b/Documentation/lei-add-external.pod new file mode 100644 index 00000000..2a131b55 --- /dev/null +++ b/Documentation/lei-add-external.pod @@ -0,0 +1,133 @@ +=head1 NAME + +lei-add-external - add inbox or external index + +=head1 SYNOPSIS + +lei add-external [OPTIONS] LOCATION + +=head1 DESCRIPTION + +Configure lei to search against an external (an inbox or external +index). When C<LOCATION> is an existing local path, it should point +to a directory that is a C<publicinbox.$NAME.inboxdir> or +C<extindex.$NAME.topdir> value in ~/.public-inbox/config. + +=head1 OPTIONS + +=for comment +TODO: mention curl options? + +=over + +=item --boost=NUMBER + +Set priority of a new or existing location. + +Default: 0 + +=item --mirror=URL + +Create C<LOCATION> by mirroring the public-inbox at C<URL>. +C<LOCATION> will have a Makefile with a C<make update> +target to update the external. + +=item --epoch=RANGE + +Restrict clones of L<public-inbox-v2-format(5)> inboxes to the +given range of epochs. The range may be a single non-negative +integer or a (possibly open-ended) C<LOW..HIGH> range of +non-negative integers. C<~> may be prefixed to either (or both) +integer values to represent the offset from the maximum possible +value. + +For example, C<--epoch=~0> alone clones only the latest epoch, +C<--epoch=~2..> clones the three latest epochs. + +Default: C<0..~0> or C<0..> or C<..~0> +(all epochs, all three examples are equivalent) + +=item -v + +=item --verbose + +Provide more feedback on stderr. + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head2 MIRRORING + +=over + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=item --inbox-version=NUM + +Force a remote public-inbox version (must be C<1> or C<2>). +This is auto-detected by default, and this option exists mainly +for testing. + +=back + +The following options are passed to L<public-inbox-init(1)>: + +=over + +=item -j JOBS, --jobs=JOBS + +=item -L LEVEL, --indexlevel=LEVEL + +=back + +The following options are passed to L<public-inbox-index(1)>: + +=over + +=item --batch-size=SIZE + +=item --compact + +=item -j JOBS, --jobs=JOBS + +=item --max-size=SIZE + +=item --sequential-shard + +=item --skip-docdata + +=back + +=head1 FILES + +The configuration for lei resides at C<$XDG_CONFIG_HOME/lei/config>. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-forget-external(1)>, L<lei-ls-external(1)>, L<lei-import(1)>, +L<public-inbox-index(1)>, L<public-inbox-extindex(1)>, +L<public-inbox-extindex-format(5)> diff --git a/Documentation/lei-add-watch.pod b/Documentation/lei-add-watch.pod new file mode 100644 index 00000000..1b48b43b --- /dev/null +++ b/Documentation/lei-add-watch.pod @@ -0,0 +1,37 @@ +=head1 NAME + +lei-add-watch - watch for new messages and flag changes + +=head1 SYNOPSIS + +lei add-watch [OPTIONS] LOCATION [LOCATION...] + +=head1 DESCRIPTION + +Tell lei to watch C<LOCATION> for new messages and flag changes. +Currently only Maildir locations are supported. + +WARNING: watches are not always reliable, occasional use +of L<lei-index(1)> and L<lei-refresh-mail-sync(1)> is recommended +if L<lei-daemon(8)> crashes or needs to be restarted. This will +be improved in the future. + +=for comment +TODO: Document --state? Believe valid values are pause, import-ro, + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-ls-watch(1)>, L<lei-rm-watch(1)> diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod new file mode 100644 index 00000000..558fc54c --- /dev/null +++ b/Documentation/lei-blob.pod @@ -0,0 +1,109 @@ +=head1 NAME + +lei-blob - display a git blob, reconstructing from mail if necessary + +=head1 SYNOPSIS + +lei blob [OPTIONS] OID + +=head1 DESCRIPTION + +Display a git blob. The blob may correspond to a message from the +local store, any local external, or blobs associated with a +project git repository (if run from a git (working) directory). +For blobs which do not exist, it will attempt to recreate the blob +using patch emails. + +=head1 OPTIONS + +=over + +=item --git-dir=DIR + +Specify an additional .git/ directory to scan. This option may be +given multiple times. + +Default: the output of C<git rev-parse --git-dir> + +=item --no-cwd + +Do not look in the git repository of the current working directory. + +=item --no-mail + +Do not look in mail storage for C<OID>. This is implied by +C<--oid-a>, C<--path-a>, and C<--path-b>. + +=item -A OID-A + +=item --oid-a=OID-A + +Provide pre-image object ID as a hint for reconstructing C<OID>. + +=item -a PATH-A + +=item --path-a=PATH-A + +Provide pre-image pathname as a hint for reconstructing C<OID>. + +=item -b PATH-B + +=item --path-b=PATH-B + +Provide post-image pathname as a hint for reconstructing C<OID>. + +=item -v + +=item --verbose + +Provide more feedback on stderr. + +=back + +The following options are also supported and are described in +L<lei-q(1)>. + +=over + +=item --remote + +Remote externals only get queried when the blob needs to be +reconstructed from patch emails. + +=item --no-local + +=item --no-external + +=item -I LOCATION, --include=LOCATION + +=item --exclude=LOCATION + +=item --only=LOCATION + +=item --no-import-remote + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +=item --proxy=PROTOCOL://HOST[:PORT] + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-external(1)>, L<lei-q(1)> diff --git a/Documentation/lei-config.pod b/Documentation/lei-config.pod new file mode 100644 index 00000000..699f45cb --- /dev/null +++ b/Documentation/lei-config.pod @@ -0,0 +1,136 @@ +=head1 NAME + +lei-config - git-config wrapper for lei configuration file + +=head1 SYNOPSIS + +lei config <name> [[<value>] [<value-pattern>]] + +lei config -l | --list + +lei config -e | --edit + +=head1 DESCRIPTION + +Call L<git-config(1)> with C<$XDG_CONFIG_HOME/lei/config> as the +configuration file. All C<OPTIONS> are passed through, but those that +override the configuration file are not permitted. + +All C<imap> and C<nntp> options may be specified per-host or +(if using git 2.26+) with wildcards: + + [imap "imap://*.onion"] + proxy = socks5h://127.0.0.1:9050 + + [nntp "nntp://example.com"] + proxy = socks5h://127.0.0.1:1080 + +=head2 VARIABLES + +=over 8 + +=item external.* + +Managed by L<lei-add-external(1)> and L<lei-forget-external(1)> + +=item imap.proxy + +=item nntp.proxy + +The C<socks5h://> proxy address. Older versions of SOCKS may +be supported if there is user demand. + +=item imap.starttls + +=item nntp.starttls + +Enable or disable STARTTLS on non-imaps:// and non-nntps:// +hosts. By default, STARTTLS is enabled if available unless +connecting to a Tor .onion or localhost. + +=item imap.compress + +=item nntp.compress + +Enable protocol-level compression. This may be incompatible +or broken with some servers. + +Note: L<Net::NNTP> compression support is pending: +L<https://rt.cpan.org/Ticket/Display.html?id=129967> + +=item imap.debug + +=item nntp.debug + +Enable debugging output of underlying IMAP and NNTP libraries, +currently L<Mail::IMAPClient> and L<Net::NNTP>, respectively. +If L<imap.proxy> or L<nntp.proxy> points to a SOCKS proxy, +debugging output for L<IO::Socket::Socks> will be enabled as +well. + +Disabling L<imap.compress> may be required for readability. + +=item imap.timeout + +=item nntp.timeout + +The read timeout for responses. + +Default: 600 seconds (IMAP); 120 seconds (NNTP) + +=item imap.fetchBatchSize + +Number of full messages to fetch at once. Larger values reduce +network round trips at the cost of higher memory use, especially +when retrieving large messages. + +Small responses for IMAP flags are fetched at 10000 times this value. + +Default: 1 + +=item color.SLOT + +C<quoted>, C<hdrdefault>, C<status>, C<attachment> color slots +are supported for the C<-f text> and C<-f reply> output formats +of L<lei-lcat(1)> and L<lei-q(1)>. + +Any per-project .git/config, and global ~/.gitconfig files +will also be parsed for diff coloring. git diff color slots +(C<color.diff.SLOT>) supported are C<new>, C<old>, C<meta>, +C<frag>, C<func>, and C<context>. + +=back + +=head1 OPTIONS + +Most L<git-config(1)> command-line switches are accepted by C<lei config> +as is. The most frequently used options are expected to be: + +=over 4 + +=item -e + +=item --edit + +Open an editor to edit the lei config file. + +=item -l + +=item --list + +List all variables set in config file, along with their values. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/lei-convert.pod b/Documentation/lei-convert.pod new file mode 100644 index 00000000..b3e29824 --- /dev/null +++ b/Documentation/lei-convert.pod @@ -0,0 +1,70 @@ +=head1 NAME + +lei-convert - one-time conversion from one mail format to another + +=head1 SYNOPSIS + +lei convert -o OUTPUT [OPTIONS] LOCATION + +lei convert -o OUTPUT [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Convert messages to another format. C<LOCATION> is a source of +messages: a directory (Maildir), a file (various mbox), or a URL +(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs +requiring authentication use L<git-credential(1)> to +fill in the username and password. + +For a regular file, the location must have a C<E<lt>formatE<gt>:> +prefix specifying one of the following formats: C<mboxrd>, +C<mboxcl2>, C<mboxcl>, or C<mboxo>. + +=head1 OPTIONS + +=over + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +Message input format. Unless messages are given on stdin, using a +format prefix with C<LOCATION> is preferred. + +=back + +The following options are also supported and are described in +L<lei-q(1)>. + +=over + +=item -o MFOLDER, --output=MFOLDER + +=item --lock METHOD + +=item --no-kw + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +=item --proxy=PROTOCOL://HOST[:PORT] + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-mail-formats(5)> diff --git a/Documentation/lei-daemon-kill.pod b/Documentation/lei-daemon-kill.pod new file mode 100644 index 00000000..50f75f4c --- /dev/null +++ b/Documentation/lei-daemon-kill.pod @@ -0,0 +1,51 @@ +=head1 NAME + +lei-daemon-kill - signal the lei-daemon + +=head1 SYNOPSIS + +lei daemon-kill [-SIGNAL | -s SIGNAL | --signal SIGNAL] + +=head1 DESCRIPTION + +Send a signal to the L<lei-daemon(8)>. C<SIGNAL> defaults to C<TERM>. + +This command should be run after updating the code of lei. + +=head1 SIGNALS + +=over 8 + +=item SIGTERM + +Send a graceful termination signal. L<lei-daemon(8)> will exit +when all currently running lei commands are done. The listen +socket will be released as soon as the signal is processed +so another L<lei-daemon(8)> process can take its place. + +=item SIGKILL + +Kills L<lei-daemon(8)> immediately. Some worker processes may +remain running for a short while. + +=back + +=for comment +SIGQUIT and SIGINT currently do what SIGTERM does, may change... + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-daemon-pid(1)>, L<lei-daemon(8)> diff --git a/Documentation/lei-daemon-pid.pod b/Documentation/lei-daemon-pid.pod new file mode 100644 index 00000000..8637324b --- /dev/null +++ b/Documentation/lei-daemon-pid.pod @@ -0,0 +1,28 @@ +=head1 NAME + +lei-daemon-pid - show the PID of the lei-daemon + +=head1 SYNOPSIS + +lei daemon-pid + +=head1 DESCRIPTION + +Show the PID of the lei-daemon. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-daemon-kill(1)> diff --git a/Documentation/lei-daemon.pod b/Documentation/lei-daemon.pod new file mode 100644 index 00000000..92bc9ac5 --- /dev/null +++ b/Documentation/lei-daemon.pod @@ -0,0 +1,61 @@ +=head1 NAME + +lei-daemon - technical information for local email interface daemon + +=head1 DESCRIPTION + +This documentation is a high-level overview for developers and +administrators interested in how lei works. + +lei-daemon is a background daemon which powers the L<lei(1)> +command-line tool. It may support virtual users and read-write +IMAP+JMAP APIs in the future. It is designed to optimize shell +completion by avoiding module loading costs, monitor Maildirs +(and in the near future, IMAP folders) for changes. + +=head2 worker processes + +Most commands cause lei-daemon to L<fork(2)> new worker +processes to isolate and parallelize work. lei-daemon is +significantly more aggressive than read-only +L<public-inbox-daemon(8)> processes with regards to resource use +since it's not designed to support C10K/C100K scenarios. + +=head2 file descriptor passing + +FD passing is used to reduce IPC costs for bulk I/O when +importing large mboxes from stdin and dumping large mboxes +to stdout. + +=head2 SOCK_SEQPACKET + +SOCK_SEQPACKET sockets are used for both communicating with +L<lei(1)> and to internal workers. SOCK_SEQPACKET guarantees +reliability (unlike SOCK_DGRAM), allows easy load distribution, +and saves developers the trouble of maintaining stream parsers. + +=head2 file monitoring + +Inotify or EVFILT_VNODE is used depending on the platform +to monitor Maildirs for changes and track keyword changes. + +The listen socket (default: C<$XDG_RUNTIME_DIR/lei/5.seq.sock>) +is also monitored, and the daemon will automatically shutdown +if it is unlinked. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-overview(7)>, L<lei-daemon-kill(1)>, L<lei-daemon-pid(1)> diff --git a/Documentation/lei-edit-search.pod b/Documentation/lei-edit-search.pod new file mode 100644 index 00000000..7f447ca2 --- /dev/null +++ b/Documentation/lei-edit-search.pod @@ -0,0 +1,30 @@ +=head1 NAME + +lei-edit-search - edit saved search + +=head1 SYNOPSIS + +lei edit-search [OPTIONS] OUTPUT + +=head1 DESCRIPTION + +Invoke C<git config --edit> to edit the saved search at C<OUTPUT>, +where C<OUTPUT> was supplied for argument of C<lei q -o OUTPUT ...> +A listing of outputs is available via C<lei ls-search>. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-ls-search(1)>, L<lei-forget-search(1)>, L<lei-up(1)>, L<lei-q(1)> diff --git a/Documentation/lei-export-kw.pod b/Documentation/lei-export-kw.pod new file mode 100644 index 00000000..4cb673ac --- /dev/null +++ b/Documentation/lei-export-kw.pod @@ -0,0 +1,55 @@ +=head1 NAME + +lei-export-kw - export keywords (flags) to Maildir and IMAP folders + +=head1 SYNOPSIS + +lei export-kw --all=[<remote|local>] + +lei export-kw MFOLDER [MFOLDER...] + +=head1 DESCRIPTION + +C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>, +C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs. +It only works for messages lei knows about (e.g. was used as a +C<lei q --output>, or imported via L<lei-import(1)>, or indexed +via L<lei-index(1)>). + +It does not delete, write, nor modify messages themselves; +it only sets metadata on Maildirs and IMAP folders. + +=head1 OPTIONS + +=over + +=item --all + +Export to all local Maildirs and remote IMAP folders + +=item --all=local + +Export all local Maildirs + +=item --all=remote + +Export all remote IMAP folders + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-refresh-mail-sync(1)>, L<lei-tag(1)> diff --git a/Documentation/lei-forget-external.pod b/Documentation/lei-forget-external.pod new file mode 100644 index 00000000..e0e3b8e1 --- /dev/null +++ b/Documentation/lei-forget-external.pod @@ -0,0 +1,42 @@ +=head1 NAME + +lei-forget-external - forget external locations + +=head1 SYNOPSIS + +lei forget-external [OPTIONS] LOCATION [LOCATION...] + +=head1 DESCRIPTION + +Forget the specified externals by removing their entries from +C<$XDG_CONFIG_HOME/lei/config>. This excludes the locations from +future search results. + +=head1 OPTIONS + +=over + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-add-external(1)>, L<lei-ls-external(1)> diff --git a/Documentation/lei-forget-mail-sync.pod b/Documentation/lei-forget-mail-sync.pod new file mode 100644 index 00000000..29ff5671 --- /dev/null +++ b/Documentation/lei-forget-mail-sync.pod @@ -0,0 +1,32 @@ +=head1 NAME + +lei-forget-mail-sync - forget sync information for a mail folder + +=head1 SYNOPSIS + +lei forget-mail-sync [OPTIONS] LOCATION [LOCATION...] + +=head1 DESCRIPTION + +Forget synchronization information for C<LOCATION>, an IMAP or Maildir +folder. Note that this won't delete any messages on the filesystem. +Users using L<lei-index(1)> without L<lei-import(1)> will be left +with dangling references in search results. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-ls-mail-sync(1)>, L<lei-index(1)> diff --git a/Documentation/lei-forget-search.pod b/Documentation/lei-forget-search.pod new file mode 100644 index 00000000..5ff526f1 --- /dev/null +++ b/Documentation/lei-forget-search.pod @@ -0,0 +1,43 @@ +=head1 NAME + +lei-forget-search - forget saved search + +=head1 SYNOPSIS + +lei forget-search [OPTIONS] OUTPUT + +=head1 DESCRIPTION + +Forget a saved search at C<OUTPUT>, +where C<OUTPUT> was supplied for argument of C<lei q -o OUTPUT ...> +A listing of outputs is available via C<lei ls-search>. + +=head1 OPTIONS + +=over + +=item --prune[=<local|remote>] + +C<--prune> will forget saved searches if the C<OUTPUT> no longer +exists. C<--prune=local> only prunes local mailboxes, +C<--prune=remote> only prunes remote mailboxes (currently +C<imap://> and C<imaps://>). + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-edit-search(1)>, L<lei-ls-search(1)>, L<lei-up(1)>, L<lei-q(1)> diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod new file mode 100644 index 00000000..31d6db13 --- /dev/null +++ b/Documentation/lei-import.pod @@ -0,0 +1,111 @@ +=head1 NAME + +lei-import - one-time import of messages into local store + +=head1 SYNOPSIS + +lei import [OPTIONS] LOCATION [LOCATION...] [+L:LABEL] + +lei import [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Import messages into the local storage of L<lei(1)> +(aka L<leiE<sol>store|lei-store-format(5)>). C<LOCATION> is a +source of messages: a directory (Maildir), a file, or a URL +(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring +authentication use L<git-credential(1)> to +fill in the username and password. + +For a regular file, the C<LOCATION> must have a C<E<lt>formatE<gt>:> +prefix specifying one of the following formats: C<mboxrd>, +C<mboxcl2>, C<mboxcl>, or C<mboxo>. + +=head1 OPTIONS + +=over + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +Message input format. Unless messages are given on stdin, using a +format prefix with C<LOCATION> is preferred. + +=item --stdin + +Read messages from stdin. + +=item --lock + +L<mbox(5)> locking method(s) to use: C<dotlock>, C<fcntl>, C<flock> or +C<none>. + +Default: fcntl,dotlock + +=item +L:LABEL + +Add the given C<LABEL> to all messages imported, where C<LABEL> +is an arbitrary user-defined value consisting of lowercase and digits. +See L<lei-tag(1)> for more info on labels. + +For example, specifying C<+L:inbox> applies the C<inbox> label +to all messages being imported. + +May be specified multiple times to apply multiple labels. + +Default: none + +=item +kw:KEYWORD + +Apply C<KEYWORD> to all messages being imported in addition +to any per-message keywords from the store (unless C<--no-kw> +is specified). See L<lei-tag(1)> for more info on keywords. + +May be specified multiple times to apply multiple keywords. + +Default: none + +=item --no-kw + +Don't import message keywords (or "flags" in IMAP terminology). + +=item --no-incremental + +Import already seen IMAP and NNTP articles. + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=item --proxy=PROTOCOL://HOST[:PORT] + +Use the specified proxy (e.g., C<socks5h://0:9050>). + +Consider L<imap.proxy> and L<nntp.proxy> which can be persistently +configured on a per-host basis in L<lei-config(1)>. + +=back + +See L<lei-config(1)> for various C<imap.*> and C<nntp.*> options. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-config(1)>, L<lei-index(1)>, L<lei-store-format(5)> diff --git a/Documentation/lei-index.pod b/Documentation/lei-index.pod new file mode 100644 index 00000000..f8ff6950 --- /dev/null +++ b/Documentation/lei-index.pod @@ -0,0 +1,64 @@ +=head1 NAME + +lei-index - index messages without importing them into lei/store + +=head1 SYNOPSIS + +lei index [OPTIONS] FOLDER + +=head1 DESCRIPTION + +Similar to L<lei-import(1)>, but does not store a copy of +messages into C<lei/store>. + +This command only makes sense for messages stored in Maildir +folders. Other folder types may be supported in the future +(they can all be indexed, but the message isn't automatically +retrieved by L<lei-q(1)> or L<lei-lcat(1)>). + +Combined with L<lei-q(1)>, C<lei index> allows Maildir users to +have similar functionality to L<mairix(1)> by not duplicating +messages into C<lei/store>. + +Occasional invocations of C<lei-refresh-mail-sync --all=local> +are recommended to keep indexed messages retrievable. + +=head1 OPTIONS + +=over + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +There is currently no need for this option. It will support C<mh>, +eventually. For now, the default (and only supported) format is +C<maildir>. When IMAP and NNTP support are fleshed out, those +formats will be inferred from their URLs. + +Default: C<maildir> + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-refresh-mail-sync(1)>, L<lei-store-format(5)>, L<lei-import(1)> diff --git a/Documentation/lei-init.pod b/Documentation/lei-init.pod new file mode 100644 index 00000000..4bfc3b9f --- /dev/null +++ b/Documentation/lei-init.pod @@ -0,0 +1,44 @@ +=head1 NAME + +lei-init - initialize storage + +=head1 SYNOPSIS + +lei init [OPTIONS] [DIRNAME] + +=head1 DESCRIPTION + +Initialize local writable storage for L<lei(1)>. If C<DIRNAME> is +unspecified, the storage is created at C<$XDG_DATA_HOME/lei/store>. +C<leistore.dir> in C<$XDG_CONFIG_HOME/lei/config> records this +location. + +=head1 OPTIONS + +=over + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-external(1)> diff --git a/Documentation/lei-inspect.pod b/Documentation/lei-inspect.pod new file mode 100644 index 00000000..82b9651a --- /dev/null +++ b/Documentation/lei-inspect.pod @@ -0,0 +1,57 @@ +=head1 NAME + +lei-inspect - general purpose inspector + +=head1 SYNOPSIS + +lei inspect [OPTIONS] ITEM [ITEM...] + +lei inspect [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +This is a diagnostic command that provides a general purpose inspector +of various things, including blobs, message IDs, Xapian document IDs, +and mail sync sources. + +=head1 OPTIONS + +=over + +=item -d DIR + +=item --dir=DIR + +An inboxdir, extindex topdir, or Xapian shard + +=item --pretty + +Pretty-print output. If stdout is opened to a tty, C<--pretty> is +enabled by default. + +=item - + +=item --stdin + +Read message from stdin. This is implicit if no arguments are given +and stdin is a pipe or regular file. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-mail-diff(1)> diff --git a/Documentation/lei-lcat.pod b/Documentation/lei-lcat.pod new file mode 100644 index 00000000..530b755e --- /dev/null +++ b/Documentation/lei-lcat.pod @@ -0,0 +1,98 @@ +=head1 NAME + +lei-lcat - display local copy of messages(s) + +=head1 SYNOPSIS + +lei lcat [OPTIONS] MSGID_OR_URL [MSGID_OR_URL...] + +lei lcat [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +lcat (local cat) is a wrapper around L<lei-q(1)> that displays local +messages by Message-ID. It is able to extract Message-IDs from URLs +as well as from common formats such as C<E<lt>$MSGIDE<gt>> and +C<id:$MSGID>. When reading from stdin, input that isn't understood is +discarded, so the caller doesn't have to bother extracting the +Message-ID or link from surrounding text (e.g., a "Link: $URL" line). + +=head1 OPTIONS + +The following options, described in L<lei-q(1)>, are supported. +One deviation from L<lei-q(1)> is the default output format is +C<-f text> when writing to stdout. + +=over + +=item --format=FORMAT + +=item -f FORMAT + +Most commonly C<text> (the default) or C<reply> to +display the message(s) in a format suitable for trimming +and sending as an email reply. + +=item --stdin + +=item - + +C<lei lcat> implicitly reads from stdin if it is a L<pipe(7)> +or regular file. This is handy for invoking C<lei lcat> from +inside an C<$EDITOR> session (assuming you use an C<$EDITOR> +which lets you pipe arbitrary lines to arbitrary commands). + +=item --[no-]remote + +=item --no-local + +=item --no-external + +=item --no-import-remote + +=item --torsocks=auto|no|yes, --no-torsocks + +=item --proxy=PROTOCOL://HOST[:PORT] + +=item -o MFOLDER, --output=MFOLDER + +=item -d STRATEGY, --dedupe=STRATEGY + +=item -t, --threads + +=item -s KEY, --sort=KEY + +=item -r, --reverse + +=item --offset=NUMBER + +=item -g, --globoff + +=item -a, --augment + +=item --lock=METHOD + +=item --alert=COMMAND + +=item --mua=COMMAND + +=item --no-color + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)>, L<lei-blob(1)> diff --git a/Documentation/lei-ls-external.pod b/Documentation/lei-ls-external.pod new file mode 100644 index 00000000..4c0c263d --- /dev/null +++ b/Documentation/lei-ls-external.pod @@ -0,0 +1,55 @@ +=head1 NAME + +lei-ls-external - list inbox and external index locations + +=head1 SYNOPSIS + +lei ls-external [OPTIONS] [FILTER] + +=head1 DESCRIPTION + +List configured externals. If C<FILTER> is given, restrict the output +to matching entries. + +=head1 OPTIONS + +=over + +=item -g + +=item --globoff + +Do not match C<FILTER> using C<*?> wildcards and C<[]> ranges. + +=item --local + +Limit operations to the local filesystem. + +=item --remote + +Limit operations to those requiring network access. + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-add-external(1)>, L<lei-forget-external(1)> diff --git a/Documentation/lei-ls-label.pod b/Documentation/lei-ls-label.pod new file mode 100644 index 00000000..8342705a --- /dev/null +++ b/Documentation/lei-ls-label.pod @@ -0,0 +1,48 @@ +=head1 NAME + +lei-ls-label - list labels + +=head1 SYNOPSIS + +lei ls-label [OPTIONS] + +=head1 DESCRIPTION + +List all known message labels ("mailboxes" in JMAP terminology). +This is handy for writing L<lei-import(1)> invocations. + +=head1 OPTIONS + +=over + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-external(1)> diff --git a/Documentation/lei-ls-mail-source.pod b/Documentation/lei-ls-mail-source.pod new file mode 100644 index 00000000..0e485923 --- /dev/null +++ b/Documentation/lei-ls-mail-source.pod @@ -0,0 +1,59 @@ +=head1 NAME + +lei-ls-mail-source - list IMAP or NNTP mail source folders + +=head1 SYNOPSIS + +lei ls-mail-source [OPTIONS] URL + +=head1 DESCRIPTION + +List information about the IMAP or NNTP mail source at C<URL>. +This command populates the cache used for Bash shell completion +and is handy for writing L<lei-import(1)> invocations. + +=head1 OPTIONS + +=over + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. + +=item -l + +Format output as JSON and include more information. + +=item --pretty + +Pretty-print JSON output. If stdout is opened to a tty, C<--pretty> +is enabled by default. + +=item --ascii + +Escape non-ASCII characters. + +=item --url + +Show full URL of newsgroup or IMAP folder. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-import(1)> diff --git a/Documentation/lei-ls-mail-sync.pod b/Documentation/lei-ls-mail-sync.pod new file mode 100644 index 00000000..883eeead --- /dev/null +++ b/Documentation/lei-ls-mail-sync.pod @@ -0,0 +1,55 @@ +=head1 NAME + +lei-ls-mail-sync - list mail sync folders + +=head1 SYNOPSIS + +lei mail-sync [OPTIONS] [FILTER] + +=head1 DESCRIPTION + +List mail sync folders. If C<FILTER> is given, restrict the output to +matching entries. + +=head1 OPTIONS + +=over + +=item -g + +=item --globoff + +Do not match C<FILTER> using C<*?> wildcards and C<[]> ranges. + +=item --local + +Limit operations to the local filesystem. + +=item --remote + +Limit operations to those requiring network access. + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-refresh-mail-sync(1)>, L<lei-export-kw(1)> diff --git a/Documentation/lei-ls-search.pod b/Documentation/lei-ls-search.pod new file mode 100644 index 00000000..367f4ad6 --- /dev/null +++ b/Documentation/lei-ls-search.pod @@ -0,0 +1,66 @@ +=head1 NAME + +lei-ls-search - list saved search queries + +=head1 SYNOPSIS + +lei ls-search [OPTIONS] [PREFIX] + +=head1 DESCRIPTION + +List saved search queries (generated from C<lei q -o OUTPUT>). +If C<PREFIX> is given, restrict the output +to entries that start with the specified value. + +=head1 OPTIONS + +=over + +=item -f FORMAT + +=item --format=FORMAT + +Display JSON output rather than default short output that includes +only the saved search location. Possible values are C<json>, +C<jsonl>, or C<concatjson>. + +=item --pretty + +Pretty-print C<json> or C<concatjson> output. If stdout is opened to +a tty and used as the C<--output> destination, C<--pretty> is enabled +by default. + +=item -l + +Long listing format (shortcut for C<--format=json>). + +=item --ascii + +Escape non-ASCII characters. + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. This option +is incompatible with C<--format>. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)>, L<lei-up(1)>, L<lei-edit-search(1)>, +L<lei-forget-search(1)> diff --git a/Documentation/lei-ls-watch.pod b/Documentation/lei-ls-watch.pod new file mode 100644 index 00000000..8063d34d --- /dev/null +++ b/Documentation/lei-ls-watch.pod @@ -0,0 +1,30 @@ +=head1 NAME + +lei-ls-watch - list active watches + +=head1 SYNOPSIS + +lei ls-watch + +=head1 DESCRIPTION + +List locations that lei is configured to watch. This command is +incomplete, mail-sync locations are implicitly watched. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-watch(1)>, L<lei-rm-watch(1)> diff --git a/Documentation/lei-mail-diff.pod b/Documentation/lei-mail-diff.pod new file mode 100644 index 00000000..96e49a8b --- /dev/null +++ b/Documentation/lei-mail-diff.pod @@ -0,0 +1,33 @@ +=head1 NAME + +lei-mail-diff - diff the contents of emails + +=head1 SYNOPSIS + + +lei mail-diff [OPTIONS] LOCATION + +lei mail-diff [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +This is a diagnostic command that's useful for finding deduplication +bugs. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-inspect(1)> diff --git a/Documentation/lei-mail-formats.pod b/Documentation/lei-mail-formats.pod new file mode 100644 index 00000000..618bada2 --- /dev/null +++ b/Documentation/lei-mail-formats.pod @@ -0,0 +1,138 @@ +=head1 NAME + +lei-mail-formats - description of mail formats supported by lei + +=head1 DESCRIPTION + +L<lei-q(1)> supports writing to several existing mail formats +for interoperability with existing mail user agents (MUA); +below is an overview of them to help users choose. + +=head1 Maildir + +The default output format when given a filesystem path, it supports +parallel read-write access. Performance is acceptable for smaller +directories, but degrades as mailboxes get larger. Speed and +scalability are limited by kernel and filesystem performance +due to the use of small files and large number of syscalls. + +See also: L<https://cr.yp.to/proto/maildir.html> and +L<https://wiki2.dovecot.org/MailboxFormat/Maildir> + +=head1 Mbox family + +The mbox family consists of several incompatible formats. +Locking for parallel access is supported, but may not be +compatible across tools. With compression (e.g. L<gzip(1)>), +they require the least amount of space while offering good +read-only performance. + +Keyword updates (C<Status:> and/or C<X-Status:> headers) +generally require rewriting the entire mbox. + +See also: +L<https://www.loc.gov/preservation/digital/formats/fdd/fdd000383.shtml>, +L<mbox(5)> + +=head2 mboxo + +The traditional BSD format. It quotes C<From > to C<E<gt>From >, +but lines already beginning with C<E<gt>From > do not get quoted, +thus automatic reversibility is not guaranteed. MUAs which favor +L</mboxcl> or L</mboxcl2> may convert these automatically to their +preferred format. + +Truncation is undetectable unless compressed with gzip or similar. + +=head2 mboxrd + +An evolution of L</mboxo>, but quotes C<From > lines prefixed +with any number of C<E<gt>> characters and is thus fully +reversible. + +This format is emitted by L<PublicInbox::WWW(3pm)> with gzip. +Since git 2.10, C<git am --patch-format=mboxrd> reads this +format. C<git log> and C<git format-patch --stdout> can also +generate this format with the C<--pretty=mboxrd> switch. + +As with uncompressed L</mboxo>, uncompressed mboxrd are vulnerable +to undetectable truncation. + +It gracefully degrades to being treated as L</mboxo> by MUAs +unaware of the format as excessive C<E<gt>From > quoting is +recognizable to humans. + +=head2 mboxcl + +L</mboxo> with a C<Content-Length:> header, C<From > lines +remain quoted to retain readability with L</mboxo> and L</mboxrd> MUAs. +However, it is easy to corrupt these files when using tools +which are not aware of C<Content-Length:> and write out updates +as L</mboxo>. + +L<mutt(1)> will convert L</mboxo> and L</mboxrd> to mboxcl upon opening. + +See also: L<https://www.jwz.org/doc/content-length.html> + +=head2 mboxcl2 + +Like L</mboxcl>, but without C<From > any quoting. It is wholly +incompatible with MUAs which only handle L</mboxo> and/or L</mboxrd>. +This is format is generated by L<mutt(1)> when writing to a new +mbox. + +=head1 MH + +Preliminary support for reads as of 2.0.0. Locking semantics differ +incompatibly amongst existing writers: Python and nmh appear +compatible with each other, while mutt appears racy and unsuitable +for parallel access due to rename(2) potentially clobbering the +C<.mh_sequences> file. More info about other clients is greatly +appreciated. + +Sequence numbers may be packed and reused by some writers, so lei +users may need to run L<lei-refresh-mail-sync(1)> if inotify|kevent +missed packing while L<lei-daemon(8)> wasn't running. + +lei is safe for reading mlmmj archives as MH since mlmmj neither +packs nor uses a .mh_sequences file to store state. + +=head1 MMDF + +Not yet supported, and it's unclear if current usage/support makes +it worth supporting. + +=head1 IMAP + +Depending on the IMAP server software and configuration, IMAP +servers may use any (or combination) of the aforementioned +formats or a non-standard database backend. Currently, lei +uses L<Mail::IMAPClient> which has acceptable performance +over low-latency links. Performance over high-latency links +is currently poor. + +=head1 eml + +A single raw message file. C<eml> is not an output format for lei, +but accepted by as an C<--input-format> (C<-F>) for read-only +commands such as L<lei-tag(1)> and L<lei-import(1)>. + +Since C<eml> is the suffix for the C<message/rfc822> MIME type +(according to the C<mime.types> file), lei will infer the type +based on the C<.eml> suffix if C<--input-format> is unspecified + +C<.patch>-suffixed files generated by L<git-format-patch(1)> +(without C<--stdout>) are C<eml> files with the addition of an +mbox C<From > header. L<lei(1)> removes C<From > lines to treat +them as C<eml> when reading these for compatibility with +C<git-am(1)> and similar tools. + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei(1)>, L<lei-q(1)>, L<lei-convert(1)>, L<lei-overview(7)> diff --git a/Documentation/lei-mail-sync-overview.pod b/Documentation/lei-mail-sync-overview.pod new file mode 100644 index 00000000..7ae7e887 --- /dev/null +++ b/Documentation/lei-mail-sync-overview.pod @@ -0,0 +1,53 @@ +=head1 NAME + +lei - an overview of lei mail synchronization + +=head1 DESCRIPTION + +L<lei(1)> provides several plumbing-level commands to synchronize +mail and keywords (flags) between lei/store and existing IMAP +and Maildir stores. Nothing documented in this manpage is required +for day-to-day use against externals. + +Mail and keyword synchronization is currently a clunky process. +Future work will be done to improve it and add IMAP IDLE support. + +=head1 TYPICAL WORKFLOW + + # import mail from a user's IMAP inbox and give it the "inbox" label: + lei import +L:inbox imaps://user@example.com/INBOX + + # dump "inbox" labeled files from the past week to a Maildir + lei q L:inbox rt:last.week.. -o /tmp/results + + # Open /tmp/results in your favorite mail agent. If inotify or kevent + # works, keyword changes (e.g. marking messages as `seen') are + # synchronized automatically. + + # If the inotify queue overflows, or if lei-daemon crashes, + # "lei index" will tell lei about keyword changes: + lei index /tmp/results + + # Optional: cleanup stale entries from mail_sync.sqlite3 + lei refresh-mail-sync /tmp/results + + # to export keyword changes back to IMAP + lei export-kw imaps://user@example.com/INBOX + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-import(1)>, L<lei-q(1)>, L<lei-index(1)>, +L<lei-refresh-mail-sync(1)>, L<lei-export-kw(1)> diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod new file mode 100644 index 00000000..e9a97d64 --- /dev/null +++ b/Documentation/lei-overview.pod @@ -0,0 +1,167 @@ +=head1 NAME + +lei - an overview of lei + +=head1 DESCRIPTION + +L<lei(1)> is a local email interface for public-inbox and personal mail. +This document provides some basic examples. + +=head1 LEI STORE + +lei has writable local storage based on L<public-inbox-v2-format(5)>. +Commands will automatically initialize the store behind the scenes if +needed, but you can call L<lei-init(1)> directly if you want to use a +store location other than the default C<$XDG_DATA_HOME/lei/store>. + +The L<lei-import(1)> command provides the primary interface for +importing messages into the local storage. In addition, other +commands, such as L<lei-q(1)> and L<lei-blob(1)>, use the local store +to memoize messages from remotes. + +=head2 EXAMPLES + +=over + +=item $ lei import mboxrd:t.mbox.gz + +Import the messages from a gzipped mboxrd into the local storage. + +=item $ lei blob 59ec517f9 + +Show message with the git blob OID of 59ec517f9. If a message with +that OID isn't found, check if the current git repository has the +blob, trying to reconstruct it from a message if needed. + +=item $ lei blob 59ec517f9 | lei tag -F eml +kw:flagged +L:next + +Set the "flagged" keyword and "next" label on the message with the +blob OID of 59ec517f9. + +=back + +=head1 EXTERNALS + +In addition to the above store, lei can make read-only queries to +"externals": inboxes and external indices. An external can be +registered by passing a URL or local path to L<lei-add-external(1)>. +For existing local paths, the external needs to be indexed with +L<public-inbox-index(1)> (in the case of a regular inbox) or +L<public-inbox-extindex(1)> (in the case of an external index). + +=head1 SYNCHRONIZATION + +lei currently has primitive mail synchronization abilities; +see L<lei-mail-sync-overview(7)> for more details. + +=head2 EXAMPLES + +=over + +=item $ lei add-external https://public-inbox.org/meta/ + +Add a remote external for public-inbox's inbox. + +=item $ lei add-external --mirror https://public-inbox.org/meta/ path + +Clone L<https://public-inbox.org/meta/> to C<path>, index it with +L<public-inbox-index(1)>, and add it as a local external. + +=back + +=head1 SEARCHING + +The L<lei-q(1)> command searches the local store and externals. The +search prefixes match those available via L<public-inbox-httpd(1)>. + +=head2 EXAMPLES + +=over + +=item $ lei q s:lei s:skeleton + +Search for messages whose subject includes "lei" and "skeleton". + +=item $ lei q -t s:lei s:skeleton + +Do the same, but also report unmatched messages that are in the same +thread as a matched message. + +=item $ lei q -t -o /tmp/mdir --mua=mutt s:lei s:skeleton + +Write results to a Maildir at "mdir". Mutt will be invoked +to open mfolder (C<mutt -f %f>) while results are being fetched +and written. + +=item $ lei q kw:flagged L:next + +Search for all flagged messages that also have a "next" label. + +=item $ lei p2q HEAD | lei q -tt -o /tmp/mdir + +Search for messages that have post-image git blob IDs that match those +of the current repository's HEAD commit, writing them to the Maildir +directory "mdir" and flagging the messages that were an exact match. + +=item $ git show -s HEAD | lei lcat + +Display a local message for the public-inbox link contained in a +commit message. + +=item $ lei q -f text m:MESSAGE-ID | lei rediff -U5 + +Feed a message containing a diff to L<lei-rediff(1)> to regenerate its +diff with five context lines. Unless C<--git-dir> is specified, this +requires the current working directory to be within the associated +code repository. + +=back + +=head1 PERFORMANCE NOTES + +L<Inline::C> is required on BSDs and can speed things up on Linux. + +lei runs as a background daemon to reduce startup costs and can +provide real-time L<kqueue(2)>/L<inotify(7)> Maildir monitoring. +L<IO::KQueue> (p5-IO-KQueue on FreeBSD) and L<Linux::Inotify2> +(liblinux-inotify2-perl and perl-Linux-Inotify2 in .deb and .rpm-based +distros, respectively) are recommended. + +L<Socket::MsgHdr> is optional (libsocket-msghdr-perl in Debian), +and further improves startup performance. Its effect is most felt +when using shell completion. + +=head1 BASH COMPLETION + +Preliminary Bash completion for lei is provided in +C<contrib/completion/>. Contributions adding support for other +shells, as well as improvements to the existing Bash completion, are +welcome. + +=head1 UPGRADING + +Since lei runs as a daemon, L<lei-daemon-kill(1)> is required to kill +the daemon so it can load new code. It will be restarted with the +next invocation of any lei command. + +=head1 CAVEATS + +IMAP and NNTP client performance is poor on high-latency connections. +It will hopefully be fixed in 2022. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-mail-sync-overview(7)> diff --git a/Documentation/lei-p2q.pod b/Documentation/lei-p2q.pod new file mode 100644 index 00000000..4bc5d25f --- /dev/null +++ b/Documentation/lei-p2q.pod @@ -0,0 +1,101 @@ +=head1 NAME + +lei-p2q - use a patch to generate a lei-q query + +=head1 SYNOPSIS + +lei p2q [OPTIONS] (FILE|COMMIT) + +lei p2q [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Given a patch, create a query that can be fed on stdin to L<lei-q(1)>. +This is useful for mapping the patch to associated messages of an +inbox. + +The patch can be provided on stdin or as a file. Alternatively, when +an argument is given that does not point to an existing file, it is +taken as a reference to a commit in the current git repository, and +L<git-format-patch(1)> is used to generate the patch. + +=head1 OPTIONS + +=over + +=item -w PREFIX[,PREFIX] + +=item --want=PREFIX[,PREFIX] + +Search prefixes to use. C<dfpost> (post-image git blob ID) and C<dfn> +(file names from the diff) are the most useful. Other available +values are C<dfa>, C<dfb>, C<dfctx>, C<dfhh>, and C<dfpre>. + +=for comment +TODO: Put a table of prefixes somewhere and reference that (at least +here and in lei-q)? + +Appending an integer to C<dfpost> or C<dfpre> indicates a minimum ID +length, and the generated query will be for that value up through the +default abbreviation length. For example, if the repository's +C<core.abbrev> is set to C<auto> and git calculates the default +abbreviation length as 7, C<dfpost6> will expand a post-image blob ID +of e7b4b32 (seven characters) into C<dfpost:e7b4b32 OR dfpost:e7b4b3>. + +This option may be given multiple times. + +Default: C<dfpost7> + +=item --stdin + +Read message from stdin. This is implicit if no arguments are given +and stdin is a pipe or regular file. + +=item --debug + +Dump output that shows the information collected for every prefix. +This information can be useful for seeing how a patch is processed, +but the format should not be considered stable. + +=item --uri + +URI escape output for interacting with HTTP(S) public-inbox instances. + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 EXAMPLES + + # to search for all threads which touch a given thread: + lei p2q $COMMIT_OID | lei q -t -o /tmp/results + + # to view results on a remote HTTP(S) public-inbox instance + $BROWSER https://example.com/pub-inbox/?q=$(lei p2q --uri $COMMIT_OID) + + # to view unapplied patches for a given $FILE from the past year: + echo \( rt:last.year.. AND dfn:$FILE \) AND NOT \( \ + $(git log -p --pretty=mboxrd --since=last.year $FILE | + lei p2q -F mboxrd ) + \) | lei q -o /tmp/unapplied + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)> diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod new file mode 100644 index 00000000..79156750 --- /dev/null +++ b/Documentation/lei-q.pod @@ -0,0 +1,354 @@ +=head1 NAME + +lei-q - search for messages matching terms + +=head1 SYNOPSIS + +lei q [OPTIONS] TERM [TERM...] + +lei q [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Search for messages across the lei/store and externals. + +=head1 OPTIONS + +=for comment +TODO: mention curl options? + +=over + +=item --stdin + +Read search terms from stdin. + +=item --no-save + +Do not save the search for L<lei-up(1)>. + +=item --output=MFOLDER + +=item -o MFOLDER + +=item --mfolder=MFOLDER + +Warning: this clobbers and overwrites the output destination unless +L</--augment> is specified. + +Destination for results (e.g., C</tmp/results-Maildir>, +C<imaps://user@mail.example.com/INBOX.test>, or +C<mboxcl2:/tmp/results-mboxcl2>). The prefix may be a supported protocol: +C<imap://> or C<imaps://>. URLs requiring +authentication use L<git-credential(1)> to +fill in the username and password. + +A prefix can specify the format of the output: C<maildir>, +C<mboxrd>, C<mboxcl2>, C<mboxcl>, C<mboxo>. For a description of +mail formats, see L<lei-mail-formats(5)>. + +C<v2:/path/to/inbox> may be used to create a new inbox of +L<public-inbox-v2-format(5)>. The new inbox will not be configured +in the L<public-inbox-config(5)> file. + +C<maildir> is the default for an existing directory or non-existing path. + +Default: C<-> (stdout) + +=item --format=FORMAT + +=item -f FORMAT + +Format of results to stdout. This option exists as a convenient +way to specify the format for the default stdout destination. +C<reply>, C<text>, C<json>, C<jsonl>, or C<concatjson> are all supported, +as are the various mbox variants described in L</--output>. + +When a format isn't specified, it's chosen based on the +L</--output> destination or prefix. C<json> is used for the +default destination (stdout). + +Using a C<format:> prefix with the C<--output> destination is +preferred when not writing to stdout. + +=item --no-color + +Disable color (for C<-f reply> and C<-f text>). + +=item --pretty + +Pretty-print C<json> or C<concatjson> output. If stdout is opened to +a tty and used as the C<--output> destination, C<--pretty> is enabled +by default. + +=item --mua=COMMAND + +A command to run on C<--output> Maildir or mbox (e.g., C<mutt -f %f>). +For a subset of MUAs known to accept a mailbox via C<-f>, COMMAND can +be abbreviated to the name of the program: C<mutt>, C<mailx>, C<mail>, +or C<neomutt>. + +=item --alert=COMMAND[,COMMAND...] + +Run C<COMMAND> after writing to output. C<:WINCH> indicates to send +C<SIGWINCH> to the C<--mua> process. C<:bell> indicates to print a +bell code. Any other value is interpreted as a command to execute as +is. + +This option may be given multiple times. + +Default: C<:WINCH,:bell> when C<--mua> is specified and C<--output> +doesn't point to stdout, nothing otherwise. + +=item --augment + +=item -a + +Augment output destination instead of clobbering it. + +=item --no-import-before + +Do not import messages before writing to an existing output destination. +Be certain you do not need existing data in your output before using +this, it permanently erases data unless C<--augment> is used. + +=item --threads + +=item -t + +Return all messages in the same thread as the actual match(es). + +Using this twice (C<-tt>) sets the C<flagged> (AKA "important") +on messages which were actual matches. This is useful to distinguish +messages which were direct hits from messages which were merely part +of the same thread. + +TODO: Warning: this flag may become persistent and saved in +lei/store unless an MUA unflags it! (Behavior undecided) + +Caveat: C<-tt> only works on locally-indexed messages at the +moment, and not on remote (HTTP(S)) endpoints. + +=item --thread-id=MSGID + +=item -T MSGID + +Only search messages in the same thread as the given Message-ID. + +For HTTP(S) externals, this only works on instances running +public-inbox 2.0+ (UNRELEASED). + +=item --jobs=QUERY_WORKERS[,WRITE_WORKERS] + +=item --jobs=,WRITE_WORKERS + +=item -j QUERY_WORKERS[,WRITE_WORKERS] + +=item -j ,WRITE_WORKERS + +Set the number of query and write worker processes for parallelism. + +C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per +remote (HTTP/HTTPS) host. + +C<WRITE_WORKERS> defaults to 75% of the number of CPUs available for +Maildir and mbox* destinations, but 4 per IMAP/IMAPS host. + +Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows +one to only set C<WRITE_WORKERS> + +=item --dedupe=STRATEGY + +=item -d STRATEGY + +Strategy for deduplicating messages: C<content>, C<oid>, C<mid>, or +C<none>. + +Default: C<content> + +=for comment +TODO: Provide description of strategies? + +=item --[no-]remote + +Whether to include results requiring network access. When local +externals are configured, C<--remote> must be explicitly passed to +enable reporting of results from remote externals. + +=item --no-local + +Limit operations to those requiring network access. + +=item --no-external + +Don't include results from externals. + +=item --include=LOCATION + +=item -I LOCATION + +Include specified external in search. This option may be given +multiple times. + +=item --exclude=LOCATION + +Exclude specified external from search. This option may be given +multiple times. + +=item --only=LOCATION + +=item -O LOCATION + +Use only the specified external for search. This option may be given +multiple times, in which case the search uses only the specified set. + +=item --globoff + +=item -g + +Do not match locations using C<*?> wildcards and C<[]> ranges. This +option applies to C<--include>, C<--exclude>, and C<--only>. + +=item --no-import-remote + +Disable the default behavior of memoizing remote messages into the +local store. + +=item --lock=METHOD + +L<mbox(5)> locking method(s) to use: C<dotlock>, C<fcntl>, C<flock> or +C<none>. + +Default: fcntl,dotlock + +=item --limit=NUMBER + +=item -NUMBER + +=item -n NUMBER + +Fuzzy-limit the number of matches per local external and lei/store. +Messages added by the L<--threads> switch do not count towards this +limit, and there is no limit on remote externals. + +Default: 10000 + +=item --offset=NUMBER + +Shift start of search results. + +Default: 0 + +=item --reverse + +=item -r + +Reverse the results. Note that this applies before C<--limit>. + +=item --sort=KEY + +=item -s KEY + +Order the results by KEY. Valid keys are C<received>, C<relevance>, +and C<docid>. + +Default: C<received> + +=item --verbose + +=item -v + +Provide more feedback on stderr. + +=item --quiet + +=item -q + +Suppress feedback messages. + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=item --proxy=PROTOCOL://HOST[:PORT] + +=back + +=head1 SEARCH TERMS + +C<lei q> supports the same search prefixes used by HTTP(S) public-inbox +instances: + +=for comment +AUTO-GENERATED-SEARCH-TERMS-BEGIN + + s: match within Subject e.g. s:"a quick brown fox" + d: match date-time range, git "approxidate" formats supported + Open-ended ranges such as `d:last.week..' and + `d:..2.days.ago' are supported + b: match within message body, including text attachments + nq: match non-quoted text within message body + q: match quoted text within message body + n: match filename of attachment(s) + t: match within the To header + c: match within the Cc header + f: match within the From header + a: match within the To, Cc, and From headers + tc: match within the To and Cc headers + l: match contents of the List-Id header + bs: match within the Subject and body + dfn: match filename from diff + dfa: match diff removed (-) lines + dfb: match diff added (+) lines + dfhh: match diff hunk header context (usually a function name) + dfctx: match diff context lines + dfpre: match pre-image git blob ID + dfpost: match post-image git blob ID + dfblob: match either pre or post-image git blob ID + patchid: match `git patch-id --stable' output + rt: match received time, like `d:' if sender's clock was correct + +=for comment +AUTO-GENERATED-SEARCH-TERMS-END + +Additional search prefixes which only affect the local lei/store: + + L: match the given label + kw: match the given keywords + +See L<lei-tag(1)> for more info on labels and keywords. + +Most prefixes are probabilistic, meaning they support stemming +and wildcards (C<*>). Ranges (such as C<d:>) and boolean prefixes +do not support stemming or wildcards. +The upstream Xapian query parser documentation fully explains +the query syntax: L<https://xapian.org/docs/queryparser.html> + +=head1 TIPS + +C<-f reply> is intended to aid in turning a cover letter +into a reply (since using C<git format-patch --in-reply-to=...> +is tedious). Results (including "From " lines) should be edited +and trimmed in your favorite C<$EDITOR> before sending. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-add-external(1)>, L<lei-lcat(1)>, L<lei-up(1)>, +L<Xapian::QueryParser Syntax|https://xapian.org/docs/queryparser.html> diff --git a/Documentation/lei-rediff.pod b/Documentation/lei-rediff.pod new file mode 100644 index 00000000..f18548d3 --- /dev/null +++ b/Documentation/lei-rediff.pod @@ -0,0 +1,126 @@ +=head1 NAME + +lei-rediff - regenerate a diff with different options + +=head1 SYNOPSIS + +lei rediff [OPTIONS] LOCATION [LOCATION...] + +lei rediff [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Read a message from C<LOCATION> or stdin and regenerate its diff with +the specified L<git-diff(1)> options. This is useful if you want to +change the display of the original patch (e.g., increasing context, +coloring moved lines differently, or using an external diff viewer). + +It relies on the contents of the .git directory of your current +project working tree. In other words, it works anywhere +L<git-am(1)> works. Otherwise, C<--git-dir=> may be specified +any number of times to add repositories to build blob data from. + +=head1 OPTIONS + +In addition to many L<git-diff(1)> options (e.g. C<-W>, C<-w>, +C<-U $LINES>) the following options are supported: + +=over + +=item --stdin + +Read message from stdin. This is implicit if no arguments are given +and stdin is a pipe or regular file. + +For users of text editors and pagers capable of piping its +buffer to arbitrary commands, it is useful to pipe a patch email +to C<lei rediff> before piping it to L<git-am(1)>. The output +of C<lei rediff> is compatible with C<git am> if its input was a +patch email. + +=item --drq[=COUNT] + +De-Re-Quote. De-quotes the input and re-quotes (the output). +Removes COUNT levels of C<E<gt> > email reply prefixes and +re-adds them upon regenerating the diff. + +This switch is intended as a convenience for running inside a +pipe-capable text editor when writing replies to a patch email. +Note: this may over-add C<E<gt> > prefixes if some input lines +are missing C<E<gt> > prefixes. + +COUNT is 1 if unspecified; in other words, C<--drq=1> and +C<--drq> are equivalent. + +It implies L</--quiet> unless L</--verbose> is specified +since text editors tend to combine stderr with stdout. + +=item --dequote-only[=COUNT] + +Like L</--drq>, but does not re-add quote prefixes to the output. + +This can be useful for feeding a hunk to L<git-apply(1)> +or L<patch(1)> while writing a reply or further processing +by another diff viewer. + +Unlike L</--drq>, it does NOT imply L</--quiet>. + +=item --git-dir=DIR + +Specify an additional .git/ directory to scan. This option may be +given multiple times. + +Default: the output of C<git rev-parse --git-dir> + +=item --no-cwd + +Do not look in the git repository of the current working directory. + +=item -q + +=item --quiet + +Suppress progress output. + +=item -v + +=item --verbose + +Provide more feedback on stderr. + +=back + +The options below, described in L<lei-q(1)>, are also supported. + +=over + +=item --[no-]remote + +=item --no-local + +=item --no-external + +=item --no-import-remote + +=item --torsocks=auto|no|yes, --no-torsocks + +=item --proxy=PROTOCOL://HOST[:PORT] + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)>, L<lei-blob(1)>, L<lei-p2q(1)> diff --git a/Documentation/lei-refresh-mail-sync.pod b/Documentation/lei-refresh-mail-sync.pod new file mode 100644 index 00000000..65150ae3 --- /dev/null +++ b/Documentation/lei-refresh-mail-sync.pod @@ -0,0 +1,57 @@ +=head1 NAME + +lei-refresh-mail-sync - refresh sync info with Maildir, IMAP + +=head1 SYNOPSIS + +lei refresh-mail-sync --all[=<remote|local>] + +lei refresh-mail-sync MFOLDER [MFOLDER...] + +=head1 DESCRIPTION + +C<lei refresh-mail-sync> is intended to keep old messages +indexed with L<lei-index(1)> retrievable if Maildir flags change +a filename. It will prune invalid entries for messages which no +longer exist in a Maildir. + +It is also useful for ensuring L<lei-export-kw(1)> can propagate +keyword (flag) changes to Maildirs and IMAP folders. + +It only needs read-only access to Maildirs and IMAP folders +and will not attempt to write to them at all. + +=head1 OPTIONS + +=over + +=item --all + +Refresh all local Maildirs and remote IMAP folders + +=item --all=local + +Refresh all local Maildirs + +=item --all=remote + +Refresh all remote IMAP folders + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-index(1)>, L<lei-export-kw(1)>, L<lei-ls-mail-sync(1)> diff --git a/Documentation/lei-reindex.pod b/Documentation/lei-reindex.pod new file mode 100644 index 00000000..3a5861c4 --- /dev/null +++ b/Documentation/lei-reindex.pod @@ -0,0 +1,47 @@ +=head1 NAME + +lei-reindex - reindex messages already in lei/store + +=head1 SYNOPSIS + +lei reindex [OPTIONS] + +=head1 DESCRIPTION + +Forces a re-index of all messages previously-indexed by L<lei-import(1)> +or L<lei-index(1)>. This can be used for in-place upgrades and bugfixes +while other processes are querying the store. Keep in mind this roughly +doubles the size of the already-large Xapian database. + +It does not re-index messages in externals, using the C<--reindex> +switch of L<public-inbox-index(1)> or L<public-inbox-extindex(1)> is +needed for that. + +=head1 OPTIONS + +=over + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-index(1)>, L<lei-import(1)> diff --git a/Documentation/lei-rm-watch.pod b/Documentation/lei-rm-watch.pod new file mode 100644 index 00000000..711d7dc4 --- /dev/null +++ b/Documentation/lei-rm-watch.pod @@ -0,0 +1,30 @@ +=head1 NAME + +lei-rm-watch - stop watching locations + +=head1 SYNOPSIS + +lei rm-watch [OPTIONS] LOCATION [LOCATION...] + +=head1 DESCRIPTION + +Tell lei to stop watching C<LOCATION> for new messages and flag +changes. Currently only Maildir locations are supported. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-watch(1)>, L<lei-ls-watch(1)> diff --git a/Documentation/lei-rm.pod b/Documentation/lei-rm.pod new file mode 100644 index 00000000..67bfb551 --- /dev/null +++ b/Documentation/lei-rm.pod @@ -0,0 +1,75 @@ +=head1 NAME + +lei-rm - unindex a message in lei/store + +=head1 SYNOPSIS + +lei rm [OPTIONS] (-|--stdin) + +lei rm [OPTIONS] LOCATION + +=head1 DESCRIPTION + +Removes message(s) and associated private metadata from lei/store +indices. It does not affect messages stored in externals, so it's +still possible to get "removed" messages from externals in L<lei-q> +search results. + +This does not remove the message from underlying git storage nor +does it remove messages from Maildir/mbox/IMAP/etc. sources. + +=head1 OPTIONS + +=over + +=item - + +=item --stdin + +Read input from standard input. This is the default if standard +input is a pipe or regular file and there are no arguments on +the command-line. + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +Message input format: C<eml>, C<mboxrd>, C<mboxcl2>, C<mboxcl>, or C<mboxo> +when reading from stdin or using one of the mbox variants. + +Not necessary when using an IMAP URL, NNTP URL or Maildir. + +Default: C<eml> when reading from stdin or if the file suffix +ends in C<.patch> or C<.eml>. + +=item --lock=METHOD + +L<mbox(5)> locking method(s) to use: C<dotlock>, C<fcntl>, C<flock> or +C<none>. + +Default: fcntl,dotlock + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-store-format(5)> diff --git a/Documentation/lei-security.pod b/Documentation/lei-security.pod new file mode 100644 index 00000000..e54cae90 --- /dev/null +++ b/Documentation/lei-security.pod @@ -0,0 +1,151 @@ +=head1 NAME + +lei - security information + +=head1 SYNOPSIS + +L<lei(1)> is intended for use with both publicly archived +and "private" mail in personal mailboxes. This document is +intended to give an overview of security implications and +lower^Wmanage user expectations. + +=head1 DESCRIPTION + +lei expects to be run as a regular user on a Unix-like system. +It expects a case-sensitive filesystem with standard Unix +permissions support. + +It does not use POSIX ACLs, extended attributes, nor any other +security-related functions which require non-standard Perl modules. + +There is preliminary support for "virtual users", but it is +incomplete and undocumented. + +=head1 INTERNAL FILES + +lei runs with a umask of 077 to prevent other users on the +system from accessing each other's mail. + +The git storage and Xapian databases are located at +C<$XDG_DATA_HOME/lei/store> (typically C<~/.local/share/lei/store>). +Any personal mail imported will reside here, so this should +be on an encrypted filesystem or block device. + +C<$XDG_RUNTIME_DIR/lei> (typically C</run/user/$UID/lei> or +C</tmp/lei-$UID>) contain the socket used to access the lei +daemon. It must only be accessible to the owner (mode 0700). + +C<$XDG_CACHE_HOME/lei> (typically C<~/.cache/lei>) will +contain IMAP and Maildir folder names which could leak sensitive +information as well as git repository names. + +C<$XDG_DATA_HOME/lei/saved-searches> (typically +C<~/.local/share/lei/saved-searches>) will contain aforementioned +folder names as well as (removable) search history. + +The configuration for lei resides at C<$XDG_CONFIG_HOME/lei/config> +(typically C<~/.config/lei/config>). It may contain sensitive pathnames +and hostnames in the config if a user chooses to configure them. + +lei itself will never write credentials to the +filesystem. However, L<git-credential(1)> may be +configured to do so. lei will only read C<~/.netrc> if +C<--netrc> is used (and it will never write to C<~/.netrc>). + +C<$XDG_CACHE_HOME/public-inbox> (typically C<~/.cache/public-inbox>) +can contain data and L<Inline::C>-built modules which can be +shared with public-facing L<public-inbox-daemon(8)> instances; +so no private data should be in "public-inbox" paths. + +=head1 EXTERNAL FILES + +Locations set by L<lei-add-external(1)> can be shared with +public-facing L<public-inbox-daemon(8)> processes. They may +reside on shared storage and may be made world-readable to +other users on the local system. + +=head1 CORE DUMPS + +In case any process crashes, a core dump may contain passwords or +contents of sensitive messages. Please report these so they can be +fixed (see L</CONTACT>). + +=head1 NETWORK ACCESS + +lei currently uses the L<curl(1)> and L<git(1)> executables in +C<$PATH> for HTTP and HTTPS network access. Interactive +authentication for HTTP and HTTPS is not yet supported since all +currently supported HTTP/HTTPS sources are L<PublicInbox::WWW> +instances. + +The L<Mail::IMAPClient> library is used for IMAP and IMAPS. +L<Net::NNTP> (standard library) is used for NNTP and NNTPS. + +L<Mail::IMAPClient> and L<Net::NNTP> will use L<IO::Socket::SSL> +for TLS if available. In turn, L<IO::Socket::SSL> uses the +widely installed OpenSSL library. + +STARTTLS will be attempted if advertised by the server +unless IMAPS or NNTPS are used. C<-c imap.starttls=0> +and C<-c nntp.startls=0> may be used to disable STARTTLS. + +L<IO::Socket::Socks> will be used if C<-c imap.proxy> or +C<-c nntp.proxy> point to a C<socks5h://$HOST:$PORT> +address (common for Tor). + +The C<--netrc> switch may be passed to curl and used for +NNTP/IMAP access (via L<Net::Netrc>). + +=head1 CREDENTIAL DATA + +lei uses L<git-credential(1)> to prompt users for IMAP and NNTP +usernames and passwords. These passwords are not encrypted in +memory and get transferred across processes via anonymous UNIX +sockets and pipes. They may be exposed via syscall tracing +tools (e.g. L<strace(1)>), kernel and hardware bugs/attacks. + +While credentials are not written to the filesystem by default, +it is possible for them to end up on disk if processes are +swapped out. Use of an encrypted swap partition is recommended. + +=head1 AUTHENTICATION METHODS + +LOGIN (username + password) is known to work over IMAP(S), +as does AUTH=ANONYMOUS (which is used by L<public-inbox-imapd(1)> +as part of our test suite). AUTHINFO may work for NNTP, but +is untested. Testers will be needed for other authentication +methods. + +=head1 DENIAL-OF-SERVICE VECTORS + +lei uses the same MIME parsing library as L<public-inbox-mda(1)> +with limits header sizes, parts, nesting and boundary limits +similar to those found in SpamAssassin and postfix. + +Email address parsing is handled by L<Email::Address::XS> if +available, but may fall back to regular expressions which favor +speed and predictable execution times over correctness. + +=head1 ENCRYPTED EMAILS + +Not yet supported, but it should eventually be possible to +configure decryption and indexing of encrypted messages and +attachments. When supported, decrypted terms will be stored +in Xapian DBs under C<$XDG_DATA_HOME/lei/store>. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-overview(7)>, L<lei(1)> diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod new file mode 100644 index 00000000..d4bb42d5 --- /dev/null +++ b/Documentation/lei-store-format.pod @@ -0,0 +1,98 @@ +% public-inbox developer manual + +=head1 NAME + +lei-store-format - lei/store format description + +=head1 DESCRIPTION + +C<lei/store> is a hybrid store based on L<public-inbox-extindex-format(5)> +("extindex") combined with L<public-inbox-v2-format(5)> ("v2") for blob +storage. While v2 is ideal for archiving a single public mailing list; +it was never intended for personal mail nor storing multiple +blobs of the "same" message. + +As with extindex, it can index disparate C<List-Id> headers +belonging to the "same" message with different git blob OIDs. +Unlike v2 and extindex, C<Message-ID> headers are NOT required; +allowing unsent draft messages to be stored and indexed. + +=head1 DIRECTORY LAYOUT + +Blob storage exists in the form of v2-style epochs. These epochs +are under the C<local/> directory (instead of C<git/>) to +prevent them from being accidentally treated as a v2 inbox. + +=head2 INDEX OVERVIEW AND DEFINITIONS + + $EPOCH - Integer starting with 0 based on time + $SCHEMA_VERSION - DB schema version (for Xapian) + $SHARD - Integer starting with 0 based on parallelism + + ~/.local/share/lei/store + - local/$EPOCH.git # normal bare git repositories + - mail_sync.sqlite3 # sync state IMAP, Maildir, NNTP + +Additionally, the following share the same roles they do in extindex: + + - ei.lock # lock file to protect global state + - ALL.git # empty, alternates for local/*.git + - ei$SCHEMA_VERSION/$SHARD # per-shard Xapian DB + - ei$SCHEMA_VERSION/over.sqlite3 # overview DB for WWW, IMAP + - ei$SCHEMA_VERSION/misc # misc Xapian DB + +=head2 XREF3 DEDUPLICATION + +Index deduplication follows extindex, see +L<public-inbox-extindex-format(5)/XREF3 DEDUPLICATION> for +more information. + +=head2 BLOB DEDUPLICATION + +The contents of C<local/*.git> repos is deduplicated by git blob +object IDs (currently SHA-1). This allows multiple copies of +cross-posted and personally Cc-ed messages to be stored with +different C<Received:>, C<X-Spam-Status:> and similar headers to +allow troubleshooting. + +=head2 VOLATILE METADATA + +Keywords and label information (as described in RFC 8621 for JMAP) +is stored in existing Xapian shards (C<ei$SCHEMA_VERSION/$SHARD>). +It is possible to search for messages matching labels and +keywords using C<L:> and C<kw:>, respectively. As with all data +stored in Xapian indices, volatile metadata is associated with +the Xapian document, thus it is shared across different blobs of +the "same" message. + +=head2 mail_sync.sqlite3 + +This SQLite database is maintained for bidirectional mapping of +git blobs to IMAP UIDs, Maildir file names, and NNTP article numbers. + +It is also used for retrieving messages from Maildirs indexed by +L<lei-index(1)>. + +=head1 IPC + +L<lei-daemon(8)> communicates with the C<lei/store> process using +L<unix(7)> C<SOCK_SEQPACKET> sockets. + +=head1 CAVEATS + +Reindexing and synchronization is not yet supported. + +=head1 THANKS + +Thanks to the Linux Foundation for sponsoring the development +and testing. + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-v2-format(5)>, L<public-inbox-extindex(5)> diff --git a/Documentation/lei-tag.pod b/Documentation/lei-tag.pod new file mode 100644 index 00000000..8cb9e736 --- /dev/null +++ b/Documentation/lei-tag.pod @@ -0,0 +1,81 @@ +=head1 NAME + +lei-tag - set/unset metadata on messages + +=head1 SYNOPSIS + +lei tag [OPTIONS] FILE [FILE...] METADATA [METADATA...] + +lei tag [OPTIONS] (-|--stdin) METADATA [METADATA...] + +=head1 DESCRIPTION + +Set or unset volatile metadata on messages. In JMAP terms, "volatile +metadata" includes "mailboxes" (analogous to a folder or label) and a +restricted set of "keywords". This supported keywords are the +combination of system keywords (seen, answered, flagged, and draft), +which map to Maildir flags and mbox Status/X-Status headers, as well +as reserved keywords (forwarded, phishing, junk, and notjunk). + +To add a label or keyword, prefix it with "+L:" and "+kw:", +respectively. To remove a label or keyword, use "-L:" or "-kw:". For +example, "+kw:flagged" would set the "flagged" keyword for the +specified messages, and "-L:INBOX" would remove the "INBOX" label. + +=head1 OPTIONS + +=over + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +Message input format: C<eml>, C<mboxrd>, C<mboxcl2>, C<mboxcl>, or +C<mboxo>. + +Default: C<eml> + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 LABELS + +Labels are user-defined values analogous to IMAP/JMAP mailbox +names. They must only contain lowercase characters, digits, and +a limited amount of punctuation (e.g. C<.>, C<->, C<@>). + +Messages may have multiple labels. + +=head1 KEYWORDS + +Keywords are "flags" in Maildir and IMAP terminology. +Common keywords include: C<seen>, C<answered>, C<flagged>, and +C<draft>, though C<forwarded>, C<phishing>, C<junk>, and C<notjunk> +are also supported. + +When writing to various mboxes, the common keywords will be +mapped to the C<Status> and C<X-Status> headers. + +Messages may have multiple keywords. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-add-external(1)> diff --git a/Documentation/lei-up.pod b/Documentation/lei-up.pod new file mode 100644 index 00000000..8c426942 --- /dev/null +++ b/Documentation/lei-up.pod @@ -0,0 +1,92 @@ +=head1 NAME + +lei-up - update a saved search + +=head1 SYNOPSIS + +lei up [OPTIONS] OUTPUT + +lei up [OPTIONS] --all[=<local|remote>] + +=head1 DESCRIPTION + +Update the saved search at C<OUTPUT> or all saved searches. + +=head1 OPTIONS + +=over + +=item --all[=<local|remote>] + +C<--all> updates all saved searches (listed in L<lei-ls-search(1)>). +C<--all=local> only updates local mailboxes, C<--all=remote> only +updates remote mailboxes (currently C<imap://> and C<imaps://>). + +=item --remote-fudge-time=INTERVAL + +Look for mail older than the time of the last successful query. +Using a small interval will reduce bandwidth use. A larger +interval reduces the likelihood of missing a result due to MTA +delays or downtime. + +The time(s) of the last successful queries are the C<lastresult> +values visible from L<lei-edit-search(1)>. + +Date formats understood by L<git-rev-parse(1)> may be used, +e.g., C<1.hour> or C<3.days>. + +Default: 2.days + +=item --no-external + +=item --no-local + +=item --no-remote + +These disable the use of all externals, local externals, or +remote externals respectively. They are useful during +temporary network or mount-point outages. + +Unlike C<lei q>, these switches override the original C<lei q --only> +options saved as C<lei.q.only>. + +The combination C<--all=remote --no-remote> is supported for +offline use in case a user is updating an IMAP folder on localhost. + +=item --exclude=LOCATION + +As with L<lei-q(1)>, but may also exclude externals originally +specified via C<lei q --only>. + +=item --lock=METHOD + +=item --alert=CMD + +=item --mua=CMD + +=item --jobs QUERY_WORKERS[,WRITE_WORKERS] + +C<--lock>, C<--alert>, C<--mua>, and C<--jobs> are all supported and +documented in L<lei-q(1)>. + +C<--mua> is incompatible with C<--all>. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)>, L<lei-ls-search(1)>, L<lei-edit-search(1)>, +L<lei-forget-search(1)> diff --git a/Documentation/lei.pod b/Documentation/lei.pod new file mode 100644 index 00000000..2b10f490 --- /dev/null +++ b/Documentation/lei.pod @@ -0,0 +1,154 @@ +=head1 NAME + +lei - local email interface + +=head1 SYNOPSIS + +lei [OPTIONS] COMMAND + +=head1 DESCRIPTION + +lei is a command-line tool for importing and searching email, +regardless of whether it is from a personal mailbox or a public-inbox. +lei supports a local, writable store built on top of +L<public-inbox-v2-format(5)> and L<public-inbox-extindex(1)>. +L<lei-q(1)> provides an interface for querying messages across the lei +store and read-only local and remote "externals" (inboxes and external +indices). + +Warning: lei is still in its early stages and may destroy mail. +Be sure to have backups of destinations lei writes to. + +Available in public-inbox 1.7.0+. + +=head1 OPTIONS + +=over + +=item -c NAME=VALUE + +Override configuration C<NAME> to C<VALUE>. + +=item -C DIR + +Change current working directory to the specified directory before +running the command. This option can be given before or after +C<COMMAND> and is accepted by all lei subcommands except +L<lei-daemon-kill(1)>. + +=back + +=head1 COMMANDS + +Subcommands for initializing and managing local, writable storage: + +=over + +=item * L<lei-init(1)> + +=item * L<lei-import(1)> + +=item * L<lei-tag(1)> + +=back + +The following subcommands can be used to manage and inspect external +locations: + +=over + +=item * L<lei-add-external(1)> + +=item * L<lei-forget-external(1)> + +=item * L<lei-ls-external(1)> + +=back + +Subcommands related to searching and inspecting messages from the lei +store and configured externals are + +=over + +=item * L<lei-blob(1)> + +=item * L<lei-config(1)> + +=item * L<lei-edit-search(1)> + +=item * L<lei-forget-search(1)> + +=item * L<lei-lcat(1)> + +=item * L<lei-ls-search(1)> + +=item * L<lei-p2q(1)> + +=item * L<lei-q(1)> + +=item * L<lei-rediff(1)> + +=item * L<lei-up(1)> + +=back + +Other subcommands include + +=over + +=item * L<lei-add-watch(1)> + +=item * L<lei-config(1)> + +=item * L<lei-convert(1)> + +=item * L<lei-daemon-kill(1)> + +=item * L<lei-daemon-pid(1)> + +=item * L<lei-forget-mail-sync(1)> + +=item * L<lei-mail-diff(1)> + +=item * L<lei-inspect(1)> + +=item * L<lei-ls-label(1)> + +=item * L<lei-ls-mail-source(1)> + +=item * L<lei-ls-mail-sync(1)> + +=item * L<lei-ls-watch(1)> + +=item * L<lei-rm-watch(1)> + +=back + +=head1 FILES + +By default, storage is located at C<$XDG_DATA_HOME/lei/store>. The +configuration for lei resides at C<$XDG_CONFIG_HOME/lei/config>. + +=head1 ERRORS + +Errors and dianostics for interactive commands are reported to +stderr. Some errors for background tasks are emitted via +L<syslog(3)> as L<lei-daemon(8)> for the top-level daemon, +and C<lei/store> for the L<lei-store-format(5)> worker. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-overview(7)>, L<lei-daemon(8)> diff --git a/Documentation/lei_design_notes.txt b/Documentation/lei_design_notes.txt new file mode 100644 index 00000000..b5180374 --- /dev/null +++ b/Documentation/lei_design_notes.txt @@ -0,0 +1,32 @@ +lei design notes +---------------- + +Daemon architecture +------------------- + +The use of a persistent daemon works around slow startup time of +Perl. This is especially important for built-in support for +shell completion. It attempts to support inotify and EVFILT_VNODE +background monitoring of Maildir keyword changes. + +If lei were reimplemented in a language with faster startup +time, the daemon architecture would likely remain since it also +lets us easily decouple the local storage from slow IMAP/NNTP +backends and allow us to serialize writes to git-fast-import, +SQLite, and Xapian across multiple processes. + +The coupling of IMAP and NNTP network latency to local storage +is a current weakness of public-inbox-watch. Therefore, -watch +will likely adopt the daemon architecture of lei in the future. + +Read/write vs read-only storage +------------------------------- + +public-inboxes are intended to be written and read by different +Unix users. Commonly, a single Unix user or group will write to +a public-inbox, but the inbox will be served by a user with +read-only permissions (e.g. "www-data" or "nobody"). + +lei/store is intended to be read and written by a single user, +thus we can rely on the Write-Ahead-Log journal of SQLite to +improve performance: <https://sqlite.org/wal.html> diff --git a/Documentation/marketing.txt b/Documentation/marketing.txt index 385e5172..8e4aa3b5 100644 --- a/Documentation/marketing.txt +++ b/Documentation/marketing.txt @@ -3,7 +3,9 @@ marketing guide for public-inbox TL; DR: Don't market this. If you must: don't be pushy and annoying about it. Slow down. -Please no superlatives, hype or BS. +Please no superlatives, hype or BS. Please keep all marketing +materials text-only to be accessible to those on slow networks +and ancient hardware. It's online and public, so it already markets itself. Being informative is not a bad thing, being insistent is. @@ -25,3 +27,12 @@ than the adoption of any software. Every time somebody recognizes and rejects various forms of lock-in and centralization is already a victory for us. + +Please keep in mind: + +* Perl 5 is not a well-liked language +* AGPL is not a well-liked license +* maintainer is a shy introvert + +Be sure to mention these things in any marketing materials +to avoid wasting time of people who hate Perl and/or AGPL. diff --git a/Documentation/mknews.perl b/Documentation/mknews.perl index 510a4e18..001ad310 100755 --- a/Documentation/mknews.perl +++ b/Documentation/mknews.perl @@ -1,5 +1,5 @@ #!/usr/bin/perl -w -# Copyright (C) 2019-2020 all contributors <meta@public-inbox.org> +# Copyright (C) all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> # Generates NEWS, NEWS.atom, and NEWS.html files using release emails # this uses unstable internal APIs of public-inbox, and this script @@ -43,9 +43,14 @@ if ($dst eq 'NEWS') { ); $ibx->{-primary_address} = $addr; my $ctx = { - -inbox => $ibx, + ibx => $ibx, -upfx => "$base_url/", -hr => 1, + zfh => $out, + env => { + HTTP_HOST => 'public-inbox.org', + 'psgi.url_scheme' => 'https', + }, }; if ($dst eq 'NEWS.html') { html_start($out, $ctx); @@ -119,10 +124,10 @@ sub html_start { } sub html_end { - print $out <<EOF or die; - git clone $PublicInbox::WwwStream::CODE_URL -</pre></body></html> -EOF + for (@$PublicInbox::WwwStream::CODE_URL) { + print $out " git clone $_\n" or die; + } + print $out "</pre></body></html>\n" or die; } sub atom_start { @@ -131,7 +136,7 @@ sub atom_start { # WwwAtomStream stats this dir for mtime my $astream = PublicInbox::WwwAtomStream->new($ctx); delete $astream->{emit_header}; - my $ibx = $ctx->{-inbox}; + my $ibx = $ctx->{ibx}; my $title = PublicInbox::WwwAtomStream::title_tag($ibx->description); my $updated = PublicInbox::WwwAtomStream::feed_updated($mtime); print $out <<EOF or die; diff --git a/Documentation/public-inbox-cindex.pod b/Documentation/public-inbox-cindex.pod new file mode 100644 index 00000000..fdc2b82d --- /dev/null +++ b/Documentation/public-inbox-cindex.pod @@ -0,0 +1,136 @@ +=head1 NAME + +public-inbox-cindex - create and update code repository search indices + +=head1 SYNOPSIS + +public-inbox-cindex [OPTIONS] -g GIT_DIR [-g GIT_DIR]... + +public-inbox-cindex [OPTIONS] --update + +=head1 DESCRIPTION + +public-inbox-cindex creates and updates the Xapian search index for +git code repository (C<coderepo>) search. It can also associate +(fuzzy join) coderepos with Xapian-indexed inboxes. It only indexes +commit messages and diffs as they would show up in an email. It +does not index the contents of blobs directly. + +Like inbox indices, coderepo indices can either be internal or external +to a coderepo. Either way, they're both created and updated through +public-inbox-cindex. + +Once the initial indices are created by public-inbox-cindex, +the L</--update> switch will incrementally update them. + +=head1 OPTIONS + +=over + +=item -d EXTDIR + +Use the given directory as an external index. External indices are +generally recommended to internal indices since they do not need +write access to any code repositories themselves. They are highly +recommended when many repositories share a common history or if +there is an M:N relationship between inboxes and coderepos. + +=item -j JOBS + +=item --jobs=JOBS + +Influences the number of Xapian indexing shards. + +If the repo has not been indexed or initialized, C<JOBS - 1> +shards will be created. + +Default: the number of existing Xapian shards + +=item --reindex + +Forces a re-index of all commits. This can be used for in-place +upgrades and bugfixes while read-only processes are utilizing the index. + +=item --update + +=item -u + +Incrementally index all previously-indexed coderepos. + +=item --prune + +Removes commits which are no longer accessible via git. +Use this after L<git-gc(1)> (or L<git-prune(1)>). + +=item --no-fsync + +=item --dangerous + +=item --max-size SIZE + +=item --batch-size SIZE + +These affect the coderepo index the same way they affect +inbox indices. See L<public-inbox-index(1)>. + +=back + +=head1 FILES + +For internal indices, the Xapian DB is stored in +C<$GIT_DIR/public-inbox-cindex>. + +External indices are stored wherever L</-d> EXTDIR points. + +=head1 CONFIGURATION + +=over 8 + +=item publicinbox.indexMaxSize + +=item publicinbox.indexBatchSize + +These configuration knobs affect the coderepo index the same way +they affect inbox indices. See L<public-inbox-index(1)>. + +=back + +=head1 ENVIRONMENT + +=over 8 + +=item PI_CONFIG + +Used to override the default "~/.public-inbox/config" value. + +=item XAPIAN_FLUSH_THRESHOLD + +The number of documents to update before committing changes to +disk. This variable is handled directly by Xapian, refer to +Xapian API documentation for more details. + +Use C<publicinbox.indexBatchSize> instead. + +=back + +=head1 UPGRADING + +Occasionally, public-inbox will update its schema version and +require a full reindex by running this command with L</--reindex>. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-index(1)> diff --git a/Documentation/public-inbox-clone.pod b/Documentation/public-inbox-clone.pod new file mode 100644 index 00000000..64ee3138 --- /dev/null +++ b/Documentation/public-inbox-clone.pod @@ -0,0 +1,287 @@ +=head1 NAME + +public-inbox-clone - "git clone --mirror" wrapper + +=head1 SYNOPSIS + +public-inbox-clone [OPTIONS] INBOX_URL [INBOX_DIR] + +public-inbox-clone [OPTIONS] ROOT_URL [DESTINATION] # public-inbox 2.0+ + +=head1 DESCRIPTION + +public-inbox-clone is a wrapper around C<git clone --mirror> for +making the initial clone of a remote HTTP(S) public-inbox. It +allows cloning multi-epoch v2 inboxes with a single command and +zero configuration. + +In public-inbox 2.0+, public-inbox-clone can create and maintain +a mirror of multiple inboxes or code repositories using manifest.js.gz +files like L<grok-pull(1)> from grokmirror. L<public-inbox-fetch(1)> is +NOT required when using this mode. + +It does not run L<public-inbox-init(1)> nor +L<public-inbox-index(1)>. Those commands must be run separately +if serving/searching the mirror is required. As-is, +public-inbox-clone is suitable for creating a git-only backup +without Xapian and SQLite indices. + +When cloning a single inbox, public-inbox-clone creates a Makefile +with handy targets to update the inbox once indexed. +This Makefile may be edited by the user; it will +not be rewritten by L<public-inbox-fetch(1)> unless it is removed +completely. + +public-inbox-clone does not use nor require any extra +configuration files (not even C<~/.public-inbox/config>), +but it can download snippets suitable for adding to any +L<public-inbox-config(5)> file. + +L<public-inbox-fetch(1)> may be used to keep a single C<INBOX_DIR> +up-to-date. + +For v2 inboxes, it will create a C<$INBOX_DIR/manifest.js.gz> +file to speed up subsequent L<public-inbox-fetch(1)>. + +=head1 OPTIONS + +=over + +=item --epoch=RANGE + +Restrict clones of L<public-inbox-v2-format(5)> inboxes to the +given range of epochs. The range may be a single non-negative +integer or a (possibly open-ended) C<LOW..HIGH> range of +non-negative integers. C<~> may be prefixed to either (or both) +integer values to represent the offset from the maximum possible +value. + +For example, C<--epoch=~0> alone clones only the latest epoch, +C<--epoch=~2..> clones the three latest epochs. + +Default: C<0..~0> or C<0..> or C<..~0> +(all epochs, all three examples are equivalent) + +=item -I PATTERN + +=item --include=PATTERN + +When cloning a top-level with multiple inboxes via manifest, +only clone inboxes and repositories matching a given wildcard pattern +(using C<*?> and C<[]> is supported). + +This is a new option in public-inbox 2.0+ + +=item --exclude=PATTERN + +When cloning a top-level with multiple inboxes via manifest, +ignore inboxes and repositories matching the given wildcard pattern. +Supports the same wildcards as L</--include> + +This is a new option in public-inbox 2.0+ + +=item --inbox-config=always|v2|v1|never + +Whether or not to retrieve the C<$INBOX/_/text/config/raw> HTTP(S) +endpoint when cloning. + +Since we can't deduce v1 inboxes from code repositories, setting this +to C<v2> or C<never> can allow faster clones of code repositories if +no v1 inboxes are present. + +Default: C<always> + +This is a new option in public-inbox 2.0+ + +=item --inbox-version=NUM + +Force a remote public-inbox version (must be C<1> or C<2>). +This is auto-detected by default, and this option exists mainly +for testing. + +This is a new option in public-inbox 2.0+ + +=item --objstore=DIR + +Enables space savings when the remote C<manifest.js.gz> +includes C<forkgroup> entries as generated by grokmirror 2.x. + +If C<DIR> does not start with C</>, C<./>, or C<../>, it is treated +as relative to the C<DESTINATION> directory. If only C<--objstore=> +is specified where C<DIR> is an empty string (C<"">), then C<objstore> +(C<$DESTINATION/objstore>) is the implied value of C<DIR>. + +This is a new option in public-inbox 2.0+ + +=item --manifest=FILE + +When incrementally updating an existing mirror, load the given +manifest (typically C<manifest.js.gz>) to speed up updates. + +By default, public-inbox writes the retrieved manifest to +C<$DESTINATION/manifest.js.gz>, this directive also +changes the destination to the specified C<FILE> + +If C<FILE> does not start with C</>, C<./>, or C<../>, it is treated +as relative to the C<DESTINATION> directory. If only C<--manifest=> +is specified where C<FILE> is an empty string (C<"">), then C<manifest.js.gz> +(C<$DESTINATION/manifest.js.gz>) is the implied value of C<FILE>. + +When updating manifests with many forks using the same objstore, +git 2.41+ is highly recommended for performance as we automatically +use the C<fetch.hideRefs> feature to speed up negotiation. + +C<--manifest=> is a new option in public-inbox 2.0+ + +=item --remote-manifest=URL|RELATIVE_PATH + +Use an alternate location for the remote manifest.js.gz file. +This may be specified as a full absolute URL (e.g +C<--remote-manifest=https://80x24.org/lore/pub/manifest.js.gz>), +or a pathname relative to the ROOT_URL (e.g +C<--remote-manifest=pub/manifest.js.gz> when ROOT_URL is +C<https://80x24.org/lore/> + +By default, C<ROOT_URL/manifest.js.gz> is used. + +This is a new option in public-inbox 2.0+ + +=item --project-list=FILE + +When cloning code repos from a manifest, generate a cgit-compatible +project list. + +If C<FILE> does not start with C</>, C<./>, or C<../>, it is treated +as relative to the C<DESTINATION> directory. If only C<--project-list=> +is specified where C<FILE> is an empty string (C<"">), then C<projects.list> +(C<$DESTINATION/projects.list>) is the implied value of C<FILE>. + +This is a new option in public-inbox 2.0+ + +=item --post-update-hook=COMMAND + +Hooks to run after a repository is cloned or updated, C<COMMAND> will +have the bare git repository destination given as its first and only +argument. + +For v2 inboxes, this operates on a per-epoch basis. + +May be specified multiple times to run multiple commands in the +order specified on the command-line. + +This is a new option in public-inbox 2.0+ + +=item -p + +=item --prune + +Pass the C<--prune> and C<--prune-tags> flags to L<git-fetch(1)> +calls on incremental clones. + +This is a new option in public-inbox 2.0+ + +=item --purge + +Deletes entire repos which no longer exist in the remote manifest, +or are filtered out by C<--include=> or C<--exclude=>. + +This is only useful when using C<--manifest> + +This is a new option in public-inbox 2.0+ + +=item --exit-code + +Exit with C<127> if no updates are done when relying on a manifest. +Updates include fingerprint mismatches in the manifest, new symlinks, +new repositories, and removed repositories from the L<--project-list> + +This is a new option in public-inbox 2.0+ + +=item -k + +=item --keep-going + +Continue as much as possible after an error. + +This is a new option in public-inbox 2.0+ + +=item -n + +=item --dry-run + +Show what would be done, without making any changes. + +This is a new option in public-inbox 2.0+ + +=item -q + +=item --quiet + +Quiets down progress messages, also passed to L<git-fetch(1)>. + +=item -v + +=item --verbose + +Increases verbosity, also passed to L<git-fetch(1)>. + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=item -j JOBS + +=item --jobs=JOBS + +The number of parallel processes to spawn at once for various network +operations using L<git(1)> and/or L<curl(1)>. + +=back + +=head1 EXAMPLES + +=for comment +Sticking to smaller projects in examples to minimize load on servers + +=over + +=item To mirror the most recent epochs of dwarves and LTTng inboxes: + + public-inbox-clone --epoch=~0 \ + --include='*lttng*' --include='*dwarves' \ + https://80x24.org/lore/ /path/to/inbox-mirror + +C<https://lore.kernel.org/> may be used instead of C<https://80x24.org/lore/> + +=item To mirror all code repos of the sparse project: + + public-inbox-clone --objstore= --project-list= --prune \ + --include='*sparse*' --inbox-config=never \ + --remote-manifest=https://80x24.org/lore/pub/manifest.js.gz \ + https://80x24.org/lore/ /path/to/code-mirror + +C<https://git.kernel.org/> may be used instead of C<https://80x24.org/lore/> +and the C<--remote-manifest> option can be omitted. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-fetch(1)>, L<public-inbox-init(1)>, L<public-inbox-index(1)> diff --git a/Documentation/public-inbox-compact.pod b/Documentation/public-inbox-compact.pod index 4e9b6d9f..d2b74c86 100644 --- a/Documentation/public-inbox-compact.pod +++ b/Documentation/public-inbox-compact.pod @@ -34,7 +34,11 @@ Compact all inboxes configured in ~/.public-inbox/config. This is an alternative to specifying individual inboxes directories on the command-line. -=item --blocksize / --no-full / --fuller +=item --blocksize + +=item --no-full + +=item --fuller These options are passed directly to L<xapian-compact(1)>. @@ -63,12 +67,12 @@ Default: 10000 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2018-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2018-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-config.pod b/Documentation/public-inbox-config.pod index 2d845f16..b4a1d94d 100644 --- a/Documentation/public-inbox-config.pod +++ b/Documentation/public-inbox-config.pod @@ -62,13 +62,22 @@ Default: none, optional =item publicinbox.<name>.newsgroup -The NNTP group name for use with L<public-inbox-nntpd(8)>. This +The NNTP group name for use with L<public-inbox-nntpd(1)>. This may be any newsgroup name with hierarchies delimited by C<.>. For example, the newsgroup for L<mailto:meta@public-inbox.org> is: C<inbox.comp.mail.public-inbox.meta> -Omitting this for the given inbox will prevent the group from -being read by L<public-inbox-nntpd(1)> +It also configures the folder hierarchy used by L<public-inbox-imapd(1)> +as well as L<public-inbox-pop3d(1)> + +Omitting this for a given inbox will prevent the inbox from +being served by L<public-inbox-nntpd(1)>, +L<public-inbox-imapd(1)>, and/or L<public-inbox-pop3d(1)> + +Newsgroup names should be all lowercase. Uppercase characters are +converted to lowercase for compatibility with IMAP, POP3, and our +L<public-inbox-extindex(1)> and L<public-inbox-cindex(1)> tools +starting with public-inbox 2.0+ (they were unusable before). Default: none, optional @@ -98,6 +107,12 @@ needs to match. Default: none +=item publicinbox.<name>.imapmirror + +This may be the full IMAP URL of an independently-run IMAP mirror. + +Default: none + =item publicinbox.<name>.nntpmirror This may be the full NNTP URL of an independently-run mirror. @@ -115,8 +130,8 @@ C<basic> only requires L<DBD::SQLite(3pm)> and provides all NNTP functionality along with thread-awareness in the WWW interface. -C<medium> requires L<Search::Xapian(3pm)> to provide full-text -term search functionality in the WWW UI. +C<medium> requires L<Xapian(3pm)> or L<Search::Xapian(3pm)> to provide +full-text term search functionality in the WWW UI. C<full> also includes positional information used by Xapian to allow for searching for phrases using quoted text. @@ -124,6 +139,14 @@ allow for searching for phrases using quoted text. Default: C<full> +=item publicinbox.<name>.boost + +Control indexing order for L<public-inbox-extindex(1)>, with ties +broken by config file order. This only affects indexing and does +not affect messages which are already indexed. + +Default: C<0> + =item publicinbox.<name>.indexSequentialShard See L<public-inbox-index(1)/publicInbox.indexSequentialShard> @@ -173,11 +196,28 @@ Default: :all The local path name of a CSS file for the PSGI web interface. May contain the attributes "media", "title" and "href" which match the associated attributes of the HTML <style> tag. -"href" may be specified to point to the URL of an remote CSS file +"href" may be specified to point to the URL of a remote CSS file and the path may be "/dev/null" or any empty file. Multiple files may be specified and will be included in the order specified. +=item publicinboxImport.dropUniqueUnsubscribe + +Drop C<List-Unsubscribe> headers if the message also includes +the C<List-Unsubscribe-Post: List-Unsubscribe=One-Click> header +to signal MUAs to support an instantaneous unsubscribe. This +is strongly recommended for users creating their own public +archives of mailing lists they subscribe to, otherwise any +archive reader can unsubscribe the archivist. + +This may break DKIM signatures if the C<List-Unsubscribe*> +headers are signed, but breaking DKIM signatures is the +lesser evil compared to allowing any reader to unsubscribe +the archivist. + +This affects L<public-inbox-mda(1)>, L<public-inbox-watch(1)>, +and L<public-inbox-learn(1)> + =item publicinboxmda.spamcheck This may be set to C<none> to disable the use of SpamAssassin @@ -195,17 +235,32 @@ See L<public-inbox-watch(1)> See L<public-inbox-watch(1)> -=item publicinbox.nntpserver +=item publicinbox.imapserver -Set this to point to the hostname of the L<public-inbox-nntpd(1)> -instance. This is used to advertise the existence of the NNTP +Set this to point to the hostname(s) of the L<public-inbox-imapd(1)> +instance. This is used to advertise the existence of the IMAP endpoint in the L<PublicInbox::WWW> HTML interface. -Multiple values are allowed for instances with multiple hostnames -or mirrors. +Default: none + +=item publicinbox.nntpserver + +Same as C<publicinbox.imapserver>, but for the hostname(s) of the +L<public-inbox-nntpd(1)> instance. + +Default: none + +=item publicinbox.pop3server + +Same as C<publicinbox.imapserver>, but for the hostname(s) of the +L<public-inbox-pop3d(1)> instance. Default: none +=item publicinbox.pop3state + +See L<public-inbox-pop3d(1)/publicinbox.pop3state> + =item publicinbox.<name>.feedmax The size of an Atom feed for the inbox. If specified more than @@ -218,7 +273,9 @@ Default: 25 A comma-delimited list of listings to hide the inbox from. -Valid values are currently C<www> and C<manifest>. +Valid values are currently C<www> and C<manifest> for non-C<404> +values of L</publicinbox.wwwListing> and L</publicinbox.grokManifest>, +respectively Default: none @@ -232,6 +289,10 @@ The URL of the cgit instance associated with the coderepo. Default: none +=item coderepo.snapshots + +See C<snapshots> in L<cgitrc(5)> + =item publicinbox.cgitrc A path to a L<cgitrc(5)> file. "repo.url" directives in the cgitrc @@ -254,18 +315,50 @@ Default: /var/www/htdocs/cgit/cgit.cgi or /usr/lib/cgit/cgit.cgi =item publicinbox.cgitdata A path to the data directory used by cgit for storing static files. -Typically guessed based the location of C<cgit.cgi> (from -C<publicinbox.cgitbin>, but may be overridden. +Typically guessed based on the location of C<cgit.cgi> (from +C<publicinbox.cgitbin>), but may be overridden. -Default: basename of C<publicinbox.cgitbin>, /var/www/htdocs/cgit/ +Default: dirname of C<publicinbox.cgitbin>, /var/www/htdocs/cgit/ or /usr/share/cgit/ +=item publicinbox.cgit + +Controls whether or not and how C<cgit> is used for serving coderepos. +New in public-inbox 2.0.0 (PENDING). + +=over 8 + +=item * first + +Try using C<cgit> as the first choice, this is the default. + +=item * fallback + +Fall back to using C<cgit> only if our native, inbox-aware +git code repository viewer doesn't recognize the URL. + +=begin comment +=for comment rewrite is not yet implemented +=item * rewrite + +Rewrite C<cgit> URLs for our native, inbox-aware code repository viewer. +This implies C<fallback> for URLs the native viewer does not recognize. + +=end comment + +=back + +Default: C<first> (C<cgit> will be used iff C<publicinbox.cgitrc> +is set and the C<cgit> binary exists). + =item publicinbox.mailEditor See L<public-inbox-edit(1)> =item publicinbox.indexMaxSize + =item publicinbox.indexBatchSize + =item publicinbox.indexSequentialShard See L<public-inbox-index(1)> @@ -287,14 +380,28 @@ L<Plack::App::Cascade(3pm)> =item * match=domain - Only show inboxes with URLs which belong to the domain of the HTTP request -=for TODO comment - -support showing cgit listing +=for comment +TODO support showing cgit listing =back Default: C<404> +=item publicinbox.nameIsUrl + +Treat the name of the public inbox as its unqualified URL when +using C<publicInbox.wwwListing=all>. That is, every +C<[publicinbox "foo"]> section implicitly sets C<publicinbox.foo.url=foo>. + +This is a convenient alternative to specifying +C<publicinbox.E<lt>nameE<gt>.url> for every single inbox if +your inbox URLs are domain-agnostic when using +C<publicInbox.wwwListing=all> + +Default: false + +New in public-inbox 2.0.0 (PENDING). + =item publicinbox.grokmanifest Controls the generation of a grokmirror-compatible gzipped JSON file @@ -323,6 +430,38 @@ always contain all git repos used by the inbox at C<$INBOX_URL> Default: C<match=domain> +=item publicinbox.<name>.obfuscate + +Whether to obfuscate email addresses in the L<PublicInbox::WWW> HTML +interface. + +Default: false + +=item publicinbox.noObfuscate + +A space-delimited list of well-known addresses and domains that should +not be obfuscated when C<publicinbox.$NAME.obfuscate> is true (e.g., +C<public@example.com> and C<@example.com>). This may be specified +more than once, in which case the values are merged. + +Default: none + +=item extindex.<name>.topdir + +The directory of an external index. See +L<public-inbox-extindex(1)> for more details. + +=item extindex.<name>.url + +Identical to L</publicinbox.E<lt>nameE<gt>.url>, but for +external indices + +=item extindex.<name>.coderepo + +Identical to L</publicinbox.E<lt>nameE<gt>.coderepo>, but for +external indices. Code repos may be freely associated with +any number of public inboxes and external indices. + =back =head2 NAMED LIMITER (PSGI) @@ -335,7 +474,7 @@ limiter with a low max value; while smaller inboxes can use the default limiter. C<RLIMIT_*> keys may be set to enforce resource limits for -a particular limiter. +a particular limiter (L<BSD::Resource(3pm)> is required). Default named-limiters are prefixed with "-". Currently, the "-cgit" named limiter is reserved for instances spawning @@ -407,12 +546,12 @@ Used to override the default "~/.public-inbox/config" value. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-convert.pod b/Documentation/public-inbox-convert.pod index a7958cf8..a2f8caf5 100644 --- a/Documentation/public-inbox-convert.pod +++ b/Documentation/public-inbox-convert.pod @@ -25,7 +25,9 @@ By default, public-inbox-convert creates a new index in the v2 inbox and indexes all existing messages, a lengthy operation for large inboxes. -=item -j JOBS, --jobs=JOBS +=item -j JOBS + +=item --jobs=JOBS Control the number of indexing jobs and Xapian shards of the v2 inbox. By default, this is the detected CPU count but capped @@ -86,12 +88,12 @@ Maildirs. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2013-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-daemon.pod b/Documentation/public-inbox-daemon.pod index 747c1452..092be667 100644 --- a/Documentation/public-inbox-daemon.pod +++ b/Documentation/public-inbox-daemon.pod @@ -4,15 +4,18 @@ public-inbox-daemon - common usage for public-inbox network daemons =head1 SYNOPSIS + public-inbox-netd public-inbox-httpd + public-inbox-imapd public-inbox-nntpd + public-inbox-pop3d =head1 DESCRIPTION This manual describes common options and behavior for public-inbox network daemons. Network daemons for public-inbox -provide read-only NNTP and HTTP access to public-inboxes. Write -access to a public-inbox will never be required to run these. +provide read-only IMAP, HTTP, NNTP and POP3 access to public-inboxes. +Write access to a public-inbox will never be required to run these. These daemons are implemented with a common core using non-blocking sockets and optimized for fairness; even with @@ -28,7 +31,9 @@ processes to take advantage of multiple CPUs. =over -=item -l, --listen ADDRESS +=item -l [PROTOCOL://]ADDRESS[?opt1=val1,opt2=val2] + +=item --listen [PROTOCOL://]ADDRESS[?opt1=val1,opt2=val2] This takes an absolute path to a Unix socket or HOST:PORT to listen on. For example, to listen to TCP connections on @@ -39,26 +44,44 @@ like L<nginx(8)> to use. May be specified multiple times to allow listening on multiple sockets. -This does not need to be specified at all if relying on -L<systemd.socket(5)> or similar +Unless per-listener options are used (required for +L<public-inbox-netd(1)>), this does not need to be specified at +all if relying on L<systemd.socket(5)> or similar, + +Per-listener options may be specified after C<?> as C<KEY=VALUE> +pairs delimited by C<,>. See L<public-inbox-netd(1)> for +documentation on the C<cert=>, C<key=>, C<env.NAME=VALUE>, +C<out=>, C<err=>, and C<psgi=> options available. Default: server-dependent unless socket activation is used with L<systemd(1)> or similar (see L<systemd.socket(5)>). -=item -1, --stdout PATH +=item -1 + +=item --stdout PATH Specify an appendable path to redirect stdout descriptor (1) to. Using this is preferable to setting up the redirect externally (e.g. E<gt>E<gt>/path/to/log in shell) since it allows SIGUSR1 to be handled (see L<SIGNALS/SIGNALS> below). -Default: /dev/null +C<out=> may also be specified on a per-listener basis. + +Default: /dev/null with C<--daemonize>, inherited otherwise -=item -2, --stderr PATH +=item -2 PATH + +=item --stderr PATH Like C<--stdout>, but for the stderr descriptor (2). -=item -W, --worker-processes +C<err=> may also be specified on a per-listener basis. + +Default: /dev/null with C<--daemonize>, inherited otherwise + +=item -W INTEGER + +=item --worker-processes INTEGER Set the number of worker processes. @@ -73,12 +96,81 @@ the master on crashes. Default: 1 +=item -X INTEGER + +=item --xapian-helpers INTEGER + +Enables the use of Xapian helper processes to handle expensive, +non-deterministic Xapian search queries asynchronously without +blocking simple requests. + +With positive values, there is an additional manager process +that can be signaled to control the number of Xapian helper workers. + +* C<-X0> one worker, no manager process +* C<-X1> one worker, one manager process +... +* C<-X8> eight workers, one manager process + +As with the public-facing public-inbox-* daemons, sending C<SIGTTIN> +or C<SIGTTOU> to the Xapian helper manager process will increment or +decrement the number of workers. + +Both Xapian helper workers and managers automatically respawn if they +crash or are explicitly killed, even with C<-X0>. + +A C++ compiler, L<pkg-config(1)>, and Xapian development files (e.g. +C<libxapian-dev> or C<xapian*-core-dev*>) are required to gain access to +some expensive queries and significant memory savings. + +Xapian helper workers are shared by all C<--worker-processes> of the +Perl daemon for additional memory savings. + +New in public-inbox 2.0.0. + +Default: undefined, search queries are handled synchronously + +=item --cert /path/to/cert + +The default TLS certificate for HTTPS, IMAPS, NNTPS, POP3S and/or STARTTLS +support if the C<cert> option is not given with C<--listen>. + +With this option, well-known TCP ports automatically get TLS or STARTTLS +support if using systemd-compatible socket activation. That is, ports +443, 563, 993, and 995 support HTTPS, NNTPS, IMAPS, and POP3S, +respectively; while ports 110, 119, and 143 support STARTTLS on POP3, +NNTP, and IMAP, respectively. + +=item --key /path/to/key + +The default TLS certificate key for the default C<--cert> or +per-listener C<cert=> option. The private key may be +concatenated into the cert file itself, in which case this +option is not needed. + +=item --multi-accept INTEGER + +By default, each worker accepts one connection at a time to maximize +fairness and minimize contention across multiple processes on a +shared listen socket. Accepting multiple connections at once may be +useful in constrained deployments with few, heavily loaded workers. +Negative values enables a worker to accept all available clients at +once, possibly starving others in the process. C<-1> behaves like +C<multi_accept yes> in nginx; while C<0> (the default) is +C<multi_accept no> in nginx. Positive values allow +fine-tuning without the runaway behavior of C<-1>. + +This may be specified on a per-listener basis via the C<multi-accept=> +per-listener directive (e.g. C<-l http://127.0.0.1?multi-accept=1>). + +Default: 0 + =back =head1 SIGNALS Most of our signal handling behavior is copied from L<nginx(8)> -and/or L<starman(1)>; so it is possible to reuse common scripts +and/or L<starman(1)>, so it is possible to reuse common scripts for managing them. =over 8 @@ -95,11 +187,11 @@ See L</UPGRADING> below. =item SIGHUP Reload config files associated with the process. -(FIXME: not tested for -httpd, yet) +(Note: broken for L<public-inbox-httpd(1)> only in E<lt>= 1.6) =item SIGTTIN -Increase the number of running workers processes by one. +Increase the number of running worker processes by one. =item SIGTTOU @@ -107,7 +199,7 @@ Decrease the number of running worker processes by one. =item SIGWINCH -Stop all running worker processes. SIGHUP or SIGTTIN +Stop all running worker processes. SIGHUP or SIGTTIN may be used to restart workers. =item SIGQUIT @@ -135,7 +227,7 @@ activation. See L<systemd.socket(5)> and L<sd_listen_fds(3)>. =item PERL_INLINE_DIRECTORY -Pointing this to point to a writable directory enables the use +Pointing this to a writable directory enables the use of L<Inline> and L<Inline::C> extensions which may provide platform-specific performance improvements. Currently, this enables the use of L<vfork(2)> which speeds up subprocess @@ -152,8 +244,8 @@ created by a user. See L<Inline> and L<Inline::C> for more details. There are two ways to upgrade a running process. Users of process management systems with socket activation -(L<systemd(1)> or similar) may rely on multiple instances For -systemd, this means using two (or more) '@' instances for each +(L<systemd(1)> or similar) may rely on multiple daemon instances. +For systemd, this means using two (or more) '@' instances for each service (e.g. C<SERVICENAME@INSTANCE>) as documented in L<systemd.unit(5)>. @@ -169,15 +261,16 @@ interrupted and lost. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> =head1 SEE ALSO -L<public-inbox-httpd(1)>, L<public-inbox-nntpd(1)> +L<public-inbox-httpd(1)>, L<public-inbox-imapd(1)>, +L<public-inbox-nntpd(1)>, L<public-inbox-pop3d(1)>, L<public-inbox-netd(1)> diff --git a/Documentation/public-inbox-edit.pod b/Documentation/public-inbox-edit.pod index 55d1c163..17f66c7c 100644 --- a/Documentation/public-inbox-edit.pod +++ b/Documentation/public-inbox-edit.pod @@ -109,12 +109,12 @@ to anyone using L<git(1)> to mirror the inbox being edited. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2019-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-extindex-format.pod b/Documentation/public-inbox-extindex-format.pod new file mode 100644 index 00000000..d47e2102 --- /dev/null +++ b/Documentation/public-inbox-extindex-format.pod @@ -0,0 +1,110 @@ +% public-inbox developer manual + +=head1 NAME + +public-inbox-extindex-format - external index format description + +=head1 DESCRIPTION + +The extindex is an index-only evolution of the per-inbox +SQLite and Xapian indices used by L<public-inbox-v2-format(5)> +and L<public-inbox-v1-format(5)>. It exists to facilitate +searches across multiple inboxes as well as to reduce index +space when messages are cross-posted to several existing +inboxes. + +It transparently indexes messages across any combination of v1 and v2 +inboxes and data about inboxes themselves. + +=head1 DIRECTORY LAYOUT + +While inspired by v2, there is no git blob storage nor +C<msgmap.sqlite3> DB. + +Instead, there is an C<ALL.git> (all caps) git repo which treats +every indexed v1 inbox or v2 epoch as a git alternate. + +As with v2 inboxes, it uses C<over.sqlite3> and Xapian "shards" +for WWW and IMAP use. Several exclusive new tables are added +to deal with L</XREF3 DEDUPLICATION> and metadata. + +Unlike v1 and v2 inboxes, it is NOT designed to map to a NNTP +newsgroup. Thus it lacks C<msgmap.sqlite3> to enforce the +unique Message-ID requirement of NNTP. + +=head2 INDEX OVERVIEW AND DEFINITIONS + + $SCHEMA_VERSION - DB schema version (for Xapian) + $SHARD - Integer starting with 0 based on parallelism + + foo/ # "foo" is the name of the index + - ei.lock # lock file to protect global state + - ALL.git # empty, alternates for inboxes + - ei$SCHEMA_VERSION/$SHARD # per-shard Xapian DB + - ei$SCHEMA_VERSION/over.sqlite3 # overview DB for WWW, IMAP + - ei$SCHEMA_VERSION/misc # misc Xapian DB + +File and directory names are intentionally different from +analogous v2 names to ensure extindex and v2 inboxes can +easily be distinguished from each other. + +=head2 XREF3 DEDUPLICATION + +Due to cross-posted messages being the norm in the large Linux kernel +development community and Xapian indices being the primary consumer of +storage, it makes sense to deduplicate indexing as much as possible. + +The internal storage format is based on the NNTP "Xref" tuple, +but with the addition of a third element: the git blob OID. +Thus the triple is expressed in string form as: + + $NEWSGROUP_NAME:$ARTICLE_NUM:$OID + +If no C<newsgroup> is configured for an inbox, the C<inboxdir> +of the inbox is used. + +This data is stored in the C<xref3> table of over.sqlite3. + +=head2 misc XAPIAN DB + +In addition to the numeric Xapian shards for indexing messages, +there is a new, in-development Xapian index for storing data +about inboxes themselves and other non-message data. This +index allows us to speed up operations involving hundreds or +thousands of inboxes. + +=head1 BENEFITS + +In addition to providing cross-inbox search capabilities, it can +also replace per-inbox Xapian shards (but not per-inbox +over.sqlite3). This allows reduction in disk space, open file +handles, and associated memory use. + +=head1 CAVEATS + +Relocating v1 and v2 inboxes on the filesystem will require +extindex to be garbage-collected and/or reindexed. + +Configuring and maintaining stable C<newsgroup> names before any +messages are indexed from every inbox can avoid expensive +reindexing and rely exclusively on GC. + +=head1 LOCKING + +L<flock(2)> locking exclusively locks the empty ei.lock file +for all non-atomic operations. + +=head1 THANKS + +Thanks to the Linux Foundation for sponsoring the development +and testing. + +=head1 COPYRIGHT + +Copyright 2020-2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-v2-format(5)> diff --git a/Documentation/public-inbox-extindex.pod b/Documentation/public-inbox-extindex.pod new file mode 100644 index 00000000..2db7d7e9 --- /dev/null +++ b/Documentation/public-inbox-extindex.pod @@ -0,0 +1,180 @@ +=head1 NAME + +public-inbox-extindex - create and update external search indices + +=head1 SYNOPSIS + +public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR... + +public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all + +=head1 DESCRIPTION + +public-inbox-extindex creates and updates an external search and +overview database used by the read-only public-inbox PSGI (HTTP), +NNTP, and IMAP interfaces. This requires either the +L<Xapian> SWIG bindings OR or L<Search::Xapian> XS bindings +along with L<DBD::SQLite> and L<DBI> Perl modules. + +=head1 OPTIONS + +=over + +=item -j JOBS + +=item --jobs=JOBS + +=item --no-fsync + +=item --dangerous + +=item --rethread + +=item --max-size SIZE + +=item --batch-size SIZE + +These switches behave as they do for L<public-inbox-index(1)> + +=item --all + +Index all C<publicinbox> entries in C<PI_CONFIG>. + +C<publicinbox> entries indexed by C<public-inbox-extindex> can +have full Xapian searching abilities with the per-C<publicinbox> +C<indexlevel> set to C<basic> and their respective Xapian +(C<xap15> or C<xapian15>) directories removed. For multiple +public-inboxes where cross-posting is common, this allows +significant space savings on Xapian indices. + +=item --dedupe=MSGID + +=item --dedupe + +Rerun deduplication on messages with the given Message-ID or +all messages if no Message-ID is specified. Deduplication rules may +change and evolve over time, especially if filters are involved. + +C<--dedupe=MSGID> may be specified multiple times to deduplicate +multiple Message-IDs. + +Use this if you see C<W: BUG? $MSGID not deduplicated properly> +warnings from WWW logs. + +=item --gc + +Perform garbage collection instead of indexing. Use this if +inboxes are removed from the extindex, a newsgroup name is +set or changed, or if messages are purged or removed from +some inboxes. + +=item --reindex + +Forces a re-index of all messages in the extindex. This can be +used for in-place upgrades and bugfixes while read-only server +processes are utilizing the index. Keep in mind this roughly +doubles the size of the already-large Xapian database. + +=item --fast + +Used with C<--reindex>, it will only look for new and stale +entries and not touch already-indexed messages. + +=item --no-multi-pack-index + +Disable writing a L<git-multi-pack-index(1)> file to save memory. +Normally, enabling multi-pack-index speeds up startup time of +subsequent L<git-cat-file(1)> processes by 3-4%, but generating +this file requires several GB of memory with large repos. + +Unlike the C<core.multiPackIndex> directive in git, it's still +possible to read existing multi-pack-index files if they are +created elsewhere. + +Available in public-inbox 2.0.0+ + +=back + +=head1 FILES + +L<public-inbox-extindex-format(5)> + +=head1 CONFIGURATION + +public-inbox-extindex does not write to the L<public-inbox-config(5)> +file, it must be entered manually. +The extindex name of C<all> is a special case which +corresponds to indexing C<--all> inboxes. An example for +C<--all> is as follows: + + [extindex "all"] + topdir = /path/to/extindex_dir + url = all + coderepo = foo + coderepo = bar + +Putting an C<extindex> entry in the config allows L<PublicInbox::WWW>. +You can have any number of C<extentry.$NAME> sections where C<$NAME> +is something other than C<all> to display a union of several inboxes. + +It is strongly recommended any public inboxes indexed by this command +have a stable C<publicinbox.$NAME.newsgroup> entry (regardless of +the presence of an NNTP or IMAP server). Otherwise, public-inbox-extindex +will use C<publicinbox.$NAME.inboxdir> as an internal key which can +cause needless reindexing and require L<--gc> if inboxes are relocated. + +See L<public-inbox-config(5)> for more details. + +=head1 ENVIRONMENT + +=over 8 + +=item PI_CONFIG + +Used to override the default "~/.public-inbox/config" value. + +=item XAPIAN_FLUSH_THRESHOLD + +The number of documents to update before committing changes to +disk. This environment is handled directly by Xapian, refer to +Xapian API documentation for more details. + +Setting C<XAPIAN_FLUSH_THRESHOLD> or +C<publicinbox.indexBatchSize> for a large C<--reindex> may cause +L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and +L<public-inbox-watch(1)> tasks to wait long and unpredictable +periods of time during C<--reindex>. + +Default: none, uses C<publicinbox.indexBatchSize> + +=back + +=head1 UPGRADING + +Occasionally, public-inbox will update its schema version and +require a full index by running this command. + +=head1 LOCKING + +It is safe to use C<--dedupe>, C<--gc> and C<--reindex> while +other processes are writing to covered inboxes or extindex. +The extindex locks will be released roughly every 10s to +allow L<public-inbox-mda(1)> and L<public-inbox-watch(1)> +processes to write to the extindex. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<Search::Xapian>, L<DBD::SQLite> diff --git a/Documentation/public-inbox-fetch.pod b/Documentation/public-inbox-fetch.pod new file mode 100644 index 00000000..1ff0df44 --- /dev/null +++ b/Documentation/public-inbox-fetch.pod @@ -0,0 +1,118 @@ +=head1 NAME + +public-inbox-fetch - "git fetch" wrapper for v2 inbox mirrors + +=head1 SYNOPSIS + +public-inbox-fetch [--exit-code] -C INBOX_DIR + +=head1 DESCRIPTION + +public-inbox-fetch updates git storage of public-inbox mirrors. +With v2 inboxes, it allows detection of new epochs and avoids +unnecessary traffic on old epochs. + +public-inbox-fetch does not use nor require any configuration +files of its own. + +It does not run L<public-inbox-index(1)>, making it suitable +for maintaining git-only backups. + +For v2 inboxes, it will maintain C<$INBOX_DIR/manifest.js.gz> +file to speed up future invocations. It always safe to remove +manifest.js.gz, it is merely an optimization and will be +restored on the next invocation. + +To prevent fetches on any v2 epoch, use L<chmod(1)> to remove +write permissions to the top-level of the epoch. For example, +to disable fetches on epoch 4: + + chmod a-w $INBOX_DIR/git/4.git + +If you wish to re-enable fetches to the epoch: + + chmod u+w $INBOX_DIR/git/4.git + +=head1 OPTIONS + +=over + +=item -q + +=item --quiet + +Quiets down progress messages, also passed to L<git-fetch(1)>. + +=item -T REMOTE + +=item --try-remote REMOTE + +Try a given remote name instead of C<origin> or C<_grokmirror>. +May be specified more than once. + +Default: C<origin>, C<_grokmirror> + +=item --exit-code + +Exit with C<127> if no updates are done. This can be used in +shell scripts to avoid invoking L<public-inbox-index(1)> when +there are no updates: + + public-inbox-fetch -q --exit-code && public-inbox-index + test $? -eq 0 || exit $? + +=item -p + +=item --prune + +Pass the C<--prune> and C<--prune-tags> flags to L<git-fetch(1)> calls. + +This is a new option in public-inbox 2.0+ + +=item -v + +=item --verbose + +Increases verbosity, also passed to L<git-fetch(1)>. + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=back + +=head1 EXIT CODES + +=over + +=item 127 + +no updates when L</--exit-code> is used above + +=back + +public-inbox-fetch will also exit with curl L<curl(1)/EXIT CODES> +as documented in the L<curl(1)> manpage (e.g. C<7> when curl cannot +reach a host). Likewise, L<git-fetch(1)> failures are also +propagated to the user. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-index(1)>, L<curl(1)> diff --git a/Documentation/public-inbox-glossary.pod b/Documentation/public-inbox-glossary.pod new file mode 100644 index 00000000..d88539c8 --- /dev/null +++ b/Documentation/public-inbox-glossary.pod @@ -0,0 +1,127 @@ +=head1 NAME + +public-inbox-glossary - glossary for public-inbox + +=head1 DESCRIPTION + +public-inbox combines several independently-developed protocols +and data formats with overlapping concepts. This document is +intended as a guide to identify and clarify overlapping concepts +with different names. + +This is mainly intended for hackers of public-inbox, but may be useful +for administrators of public-facing services and/or users building +tools. + +=head1 TERMS + +=over 8 + +=item IMAP UID, NNTP article number, on-disk Xapian docid + +A sequentially-assigned positive integer. These integers are per-inbox, +or per-extindex. This is the C<num> column of the C<over> table in +C<over.sqlite3> + +=item tid, THREADID + +A sequentially assigned positive integer. These integers are +per-inbox or per-extindex. In the future, this may be prefixed +with C<T> for JMAP (RFC 8621) and RFC 8474. This may not be +strictly compliant with RFC 8621 since inboxes and extindices +are considered independent entities from each other. + +This is the C<tid> column of the C<over> table in C<over.sqlite3> + +=item blob + +For email, this is the git blob object ID (SHA-(1|256)) of an +RFC-(822|2822|5322) email message. + +=item IMAP EMAILID, JMAP Email Id + +To be decided. This will likely be the git blob ID prefixed with C<g> +rather than the numeric UID to accommodate the same blob showing +up in both an extindex and inbox (or multiple extindices). + +=item newsgroup + +The name of the NNTP newsgroup, see L<public-inbox-config(5)>. + +=item IMAP (folder|mailbox) slice + +A 50K slice of a newsgroup to accommodate the limitations of IMAP +clients with L<public-inbox-imapd(1)>. This is the C<newsgroup> +name with a C<.$INTEGER_SUFFIX>, e.g. a newsgroup named C<inbox.test> +would have its first slice named C<inbox.test.0>, and second slice +named C<inbox.test.1> and so forth. + +If implemented, the RFC 8474 MAILBOXID of an IMAP slice will NOT have +the same Mailbox Id as the public-facing full JMAP mailbox. + +=item inbox name, public JMAP mailbox name + +The HTTP(S) name of the public-inbox +(C<publicinbox.E<lt>nameE<gt>.*>). JMAP will use this name +rather than the newsgroup name since public-facing JMAP will be +part of the PSGI code and not need a separate daemon like +L<public-inbox-nntpd(1)> or L<public-inbox-imapd(1)> + +=item epoch + +A git repository used for blob storage. See +L<public-inbox-v2-format(5)/GIT EPOCHS>. + +=item keywords, (IMAP|Maildir) flags, mbox Status + X-Status + +Private, per-message keywords or flags as described in RFC 8621 +section 10.4. These are conveyed in the C<Status:> and +C<X-Status:> headers for L<mbox(5)>, as system IMAP FLAGS +(RFC 3501 section 2.3.2), or Maildir info flags. + +L<public-inbox-watch(1)> ignores drafts and trashed (deleted) +messages. L<lei-import(1)> ignores trashed (deleted) messages, +but it imports drafts. + +=item labels, private JMAP mailboxes + +For L<lei(1)> users only. This will allow lei users to place +the same email into one or more virtual folders for +ease of filtering. This is NOT tied to public-inbox names, as +messages stored by lei may not be public. + +These are similar in spirit to arbitrary freeform "tags" +in mail software such as L<notmuch(1)> and non-system IMAP FLAGS. + +=item volatile metadata (VMD) + +For L<lei(1)> users only, this refers to the combination of +keywords and labels which are subject to frequent change +independently of immutable message content. + +=item IMAP INTERNALDATE, JMAP receivedAt, rt: search prefix + +The first valid timestamp value of Received: headers (top first). +If no Received: header exists, the Date: header is used, and the +current time if neither header(s) exist. When mirroring via +git, this is the git commit time. + +=item IMAP SENT*, JMAP sentAt, dt: and d: search prefixes + +The first valid timestamp value of the Date: header(s). +If no Date: header exists, the time from the Received: header is +used, and then the current time if neither header exists. +When mirroring via git, this is the git author time. + +=back + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-v2-format(5)>, L<public-inbox-v1-format(5)>, +L<public-inbox-extindex-format(5)>, L<gitglossary(7)> diff --git a/Documentation/public-inbox-httpd.pod b/Documentation/public-inbox-httpd.pod index f4e9945a..3ed48adc 100644 --- a/Documentation/public-inbox-httpd.pod +++ b/Documentation/public-inbox-httpd.pod @@ -20,16 +20,29 @@ L<Plack::Middleware::Head> This may point to a PSGI file for supporting generic PSGI apps. +=head1 ENVIRONMENT + +=over 8 + +=item GIT_HTTP_MAX_REQUEST_BUFFER + +Shared with L<git-http-backend(1)>, this governs the maximum upload +size of an HTTP request. + +Default: 10m + +=back + =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-imapd.pod b/Documentation/public-inbox-imapd.pod index a5c996b8..85bf3651 100644 --- a/Documentation/public-inbox-imapd.pod +++ b/Documentation/public-inbox-imapd.pod @@ -27,10 +27,12 @@ are supported and documented below. =over -=item -l, --listen PROTO://ADDRESS/?cert=/path/to/cert,key=/path/to/key +=item -l PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item --listen PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key In addition to the normal C<-l>/C<--listen> switch described in -L<public-inbox-daemon(8)>, the C<PROTO> prefix (e.g. C<imap://> or +L<public-inbox-daemon(8)>, the C<PROTOCOL> prefix (e.g. C<imap://> or C<imaps://>) may be specified to force a given protocol. For STARTTLS and IMAPS support, the C<cert> and C<key> may be specified @@ -52,7 +54,7 @@ automatically gets STARTTLS support. The default private TLS certificate key for optional STARTTLS and IMAPS support if the C<key> option is not given with C<--listen>. The private -key may concatenated into the path used by C<--cert>, in which case this +key may be concatenated into the path used by C<--cert>, in which case this option is not needed. =back @@ -75,13 +77,13 @@ The newsgroup name maps to an IMAP folder name. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/>, +The mail archives are hosted at L<https://public-inbox.org/meta/>, and L<nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta>, -L<nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta> +L<nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta> =head1 COPYRIGHT -Copyright 2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2020-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod index 0848e860..f1a2180a 100644 --- a/Documentation/public-inbox-index.pod +++ b/Documentation/public-inbox-index.pod @@ -13,8 +13,8 @@ public-inbox-index [OPTIONS] --all public-inbox-index creates and updates the search, overview and NNTP article number database used by the read-only public-inbox HTTP and NNTP interfaces. Currently, this requires -L<DBD::SQLite> and L<DBI> Perl modules. L<Search::Xapian> -is optional, only to support the PSGI search interface. +L<DBD::SQLite> and L<DBI> Perl modules. L<Xapian> (or L<Search::Xapian>) +are optional, only to support the PSGI search interface. Once the initial indices are created by public-inbox-index, L<public-inbox-mda(1)> and L<public-inbox-watch(1)> will @@ -34,7 +34,9 @@ normal search functionality. =over -=item --jobs=JOBS, -j +=item -j JOBS + +=item --jobs=JOBS Influences the number of Xapian indexing shards in a (L<public-inbox-v2-format(5)>) inbox. @@ -52,7 +54,9 @@ the overview and article number mapping). Default: the number of existing Xapian shards -=item --compact / -c +=item -c + +=item --compact Compacts the Xapian DBs after indexing. This is recommended when using C<--reindex> to avoid running out of disk space @@ -146,6 +150,13 @@ data to accumulate, resulting on latency spikes from writeback. Available in public-inbox 1.6.0+. +=item --dangerous + +Speed up initial index by using in-place updates and denying support for +concurrent readers. This is only effective with Xapian 1.4+. + +Available in public-inbox 1.8.0+ + =item --sequential-shard Sets or overrides L</publicinbox.indexSequentialShard> on a @@ -162,6 +173,42 @@ See L<public-inbox-init(1)/--skip-docdata> for description and caveats. Available in public-inbox 1.6.0+. +=item -E EXTINDEX + +=item --update-extindex=EXTINDEX + +Update the given external index (L<public-inbox-extindex-format(5)>. +Either the configured section name (e.g. C<all>) or a directory name +may be specified. + +Defaults to C<all> if C<[extindex "all"]> is configured, +otherwise no external indices are updated. + +May be specified multiple times in rare cases where multiple +external indices are configured. + +=item --no-update-extindex + +Do not update the C<all> external index by default. This negates +all uses of C<-E> / C<--update-extindex=> on the command-line. + +=item --no-multi-pack-index + +Disables writing the multi-pack-index when using L</--update-extindex>. +See L<public-inbox-extindex(1)/--no-multi-pack-index> for details. + +Available in public-inbox 2.0.0+ + +=item --since=DATESTRING + +=item --after=DATESTRING + +=item --until=DATESTRING + +=item --before=DATESTRING + +Passed directly to L<git-log(1)> to limit changes for C<--reindex> + =back =head1 FILES @@ -279,22 +326,22 @@ Default: none, uses C<publicinbox.indexBatchSize> =head1 UPGRADING -Occasionally, public-inbox will update it's schema version and +Occasionally, public-inbox will update its schema version and require a full index by running this command. =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> =head1 SEE ALSO -L<Search::Xapian>, L<DBD::SQLite> +L<Search::Xapian>, L<DBD::SQLite>, L<public-inbox-extindex-format(5)> diff --git a/Documentation/public-inbox-init.pod b/Documentation/public-inbox-init.pod index f1ec05de..85c6c9e8 100644 --- a/Documentation/public-inbox-init.pod +++ b/Documentation/public-inbox-init.pod @@ -20,7 +20,9 @@ may be specified for inboxes with multiple addresses. =over -=item -V, --version FORMAT_VERSION +=item -V FORMAT_VERSION + +=item --version FORMAT_VERSION Specify C<2> here to use the scalable L<public-inbox-v2-format(5)> if you have L<DBD::SQLite> installed. @@ -31,7 +33,9 @@ L<DBD::SQLite>. Default: C<1> -=item -L, --indexlevel <basic|medium|full> +=item -L <basic|medium|full> + +=item --indexlevel <basic|medium|full> Controls the indexing level for L<public-inbox-index(1)> @@ -39,7 +43,9 @@ See L<public-inbox-config(5)> for more information. Default: C<full> -=item --ng, --newsgroup NEWSGROUP +=item --ng NEWSGROUP + +=item --newsgroup NEWSGROUP The NNTP group name for use with L<public-inbox-nntpd(8)>. This may be any newsgroup name with hierarchies delimited by C<.>. @@ -54,6 +60,13 @@ Available in public-inbox 1.6.0+. Default: none. +=item -c KEY=VALUE + +Allow setting arbitrary configs as C<publicinbox.$NAME.$KEY>. +This is idempotent for the same C<VALUE>, but allows setting +multiple values for keys such as C<publicinbox.$NAME.url> and +C<publicinbox.$NAME.watch>. + =item --skip-artnum This option allows archivists to publish incomplete archives @@ -70,7 +83,9 @@ Available in public-inbox 1.6.0+. Default: unset, no NNTP article numbers are skipped -=item -S, --skip-epoch +=item -S + +=item --skip-epoch For C<-V2> (L<public-inbox-v2-format(5)>) inboxes only, this option allows archivists to publish incomplete archives with newer @@ -81,7 +96,9 @@ Available since public-inbox 1.2.0. Default: unset, no epochs are skipped -=item -j, --jobs=JOBS +=item -j JOBS + +=item --jobs=JOBS Control the number of Xapian index shards in a C<-V2> (L<public-inbox-v2-format(5)>) inbox. @@ -137,12 +154,12 @@ to a given inbox. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2019-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-learn.pod b/Documentation/public-inbox-learn.pod index 498c5092..b08e4bc8 100644 --- a/Documentation/public-inbox-learn.pod +++ b/Documentation/public-inbox-learn.pod @@ -54,7 +54,7 @@ This is similar to the C<spam> command above, but does not feed the message to L<spamc(1)> and only removes messages which match on any of the C<To:>, C<Cc:>, and C<List-ID:> headers. -The C<--all> option may be used match C<spam> semantics in removing +The C<--all> option may be used to match C<spam> semantics in removing the message from all configured inboxes. C<--all> is only available in public-inbox 1.6.0+. @@ -73,16 +73,35 @@ Default: ~/.public-inbox/config =back +=head1 CONFIGURATION + +These configuration knobs should be used in the +L<public-inbox-config(5)> file. + +=over 8 + +=item publicinboxImport.dropUniqueUnsubscribe + +=item publicinbox.<name>.address + +=item publicinbox.<name>.listid + +=item publicinboxmda.spamcheck + +See L<public-inbox-config(5)> for descriptions of these options + +=back + =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-mda.pod b/Documentation/public-inbox-mda.pod index a5e353e5..edc90287 100644 --- a/Documentation/public-inbox-mda.pod +++ b/Documentation/public-inbox-mda.pod @@ -68,17 +68,33 @@ Default: ~/.public-inbox/emergency/ =back +=head1 CONFIGURATION + +Various configuration knobs should be used in the +L<public-inbox-config(5)> file. + +=over 8 + +=item publicinboxImport.dropUniqueUnsubscribe + +=item publicinbox.<name>.address + +=item publicinbox.<name>.listid + +See L<public-inbox-config(5)> for descriptions of these options + +=back =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-netd.pod b/Documentation/public-inbox-netd.pod new file mode 100644 index 00000000..71425e3c --- /dev/null +++ b/Documentation/public-inbox-netd.pod @@ -0,0 +1,97 @@ +=head1 NAME + +public-inbox-netd - read-only network daemon for sharing public-inboxes + +=head1 SYNOPSIS + + public-inbox-netd [OPTIONS] + +=head1 DESCRIPTION + +public-inbox-netd provides a read-only multi-protocol +(HTTP/IMAP/NNTP/POP3) daemon for public-inbox. It uses options +and environment variables common to all +L<public-inbox-daemon(8)> implementations. + +The default configuration will never require write access +to the directory where the public-inbox is stored, so it +may be run as a different user than the user running +L<public-inbox-watch(1)>, L<public-inbox-mda(1)>, or +L<git-fetch(1)>. + +=head1 OPTIONS + +See common options in L<public-inbox-daemon(8)/OPTIONS>. + +=over + +=item -l PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item --listen PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item -l http://ADDRESS/?env.PI_CONFIG=/path/to/cfg,psgi=/path/to/app.psgi + +In addition to the normal C<-l>/C<--listen> switch described in +L<public-inbox-daemon(8)>, the protocol prefix (e.g. C<nntp://> or +C<nntps://>) may be specified to force a given protocol. + +Environment variable overrides in effect during loading and +reloading (SIGHUP) can be specified as C<env.NAME=VALUE> for +all protocols. + +HTTP(S) listeners may also specify C<psgi=> to use a different +C<.psgi> file for each listener. + +C<err=/path/to/errors.log> may be used to isolate error/debug output +for a particular listener away from C<--stderr>. + +Non-HTTP(S) listeners may also specify C<out=> for logging to +C<stdout>. HTTP(S) users are encouraged to configure +L<Plack::Middleware::AccessLog> or +L<Plack::Middleware::AccessLog::Timed>, instead. + +=item --cert /path/to/cert + +See L<public-inbox-daemon(1)>. + +=item --key /path/to/key + +See L<public-inbox-daemon(1)>. + +=back + +=head1 CONFIGURATION + +These configuration knobs should be used in the +L<public-inbox-config(5)>. + +=over 8 + +=item publicinbox.<name>.newsgroup + +=item publicinbox.nntpserver + +=item publicinbox.pop3state + +=back + +See L<public-inbox-config(5)> for documentation on them. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/>, and +L<nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta>, +L<nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<git(1)>, L<git-config(1)>, L<public-inbox-daemon(8)>, +L<public-inbox-config(5)> diff --git a/Documentation/public-inbox-nntpd.pod b/Documentation/public-inbox-nntpd.pod index 18f83c9c..59111f92 100644 --- a/Documentation/public-inbox-nntpd.pod +++ b/Documentation/public-inbox-nntpd.pod @@ -26,7 +26,9 @@ are supported and documented below. =over -=item -l, --listen PROTO://ADDRESS/?cert=/path/to/cert,key=/path/to/key +=item -l PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item --listen PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key In addition to the normal C<-l>/C<--listen> switch described in L<public-inbox-daemon(8)>, the protocol prefix (e.g. C<nntp://> or @@ -51,7 +53,7 @@ automatically gets STARTTLS support. The default private TLS certificate key for optional STARTTLS and NNTPS support if the C<key> option is not given with C<--listen>. The private -key may concatenated into the path used by C<--cert>, in which case this +key may be concatenated into the path used by C<--cert>, in which case this option is not needed. =back @@ -75,13 +77,13 @@ See L<public-inbox-config(5)> for documentation on them. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/>, +The mail archives are hosted at L<https://public-inbox.org/meta/>, and L<nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta>, -L<nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta> +L<nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2013-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-overview.pod b/Documentation/public-inbox-overview.pod index 44989e6e..35917ccc 100644 --- a/Documentation/public-inbox-overview.pod +++ b/Documentation/public-inbox-overview.pod @@ -48,7 +48,7 @@ that inbox. The instructions are roughly: # Optional but strongly recommended for hosting HTTP # (and required for NNTP) - # enable overview (requires DBD::SQLite) and, if Search::Xapian is + # enable overview (requires DBD::SQLite) and, if Xapian is # available, search: public-inbox-index INBOX_DIR @@ -119,11 +119,11 @@ L<public-inbox-httpd(1)> Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2016-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-pop3d.pod b/Documentation/public-inbox-pop3d.pod new file mode 100644 index 00000000..fb16fb96 --- /dev/null +++ b/Documentation/public-inbox-pop3d.pod @@ -0,0 +1,122 @@ +=head1 NAME + +public-inbox-pop3d - POP3 server for sharing public-inboxes + +=head1 SYNOPSIS + + public-inbox-pop3d [OPTIONS] + +=head1 DESCRIPTION + +public-inbox-pop3d provides a POP3 daemon for public-inbox. +It uses options and environment variables common to all +read-only L<public-inbox-daemon(8)> implementations, +but requires additional read-write storage to keep track +of deleted messages on a per-user basis. + +Like L<public-inbox-imapd(1)>, C<public-inbox-pop3d> will never +require write access to the directory where the public-inboxes +are stored. + +It is designed for anonymous access, thus the password is +always C<anonymous> (all lower-case). + +Usernames are of the format: + + C<$UUID@$NEWSGROUP_NAME> + +Where C<$UUID> is the output of the L<uuidgen(1)> command. Dash +(C<->) characters in UUIDs are ignored, and C<[A-F]> hex +characters are case-insensitive. Users should keep their UUIDs +private to prevent others from deleting unretrieved messages. +Users may switch to a new UUID at any time to retrieve +previously-retrieved messages. + +Historical slices of 50K messages are available +by suffixing the integer L<$SLICE>, where C<0> is the oldest. + + C<$UUID@$NEWSGROUP_NAME.$SLICE> + +It may be run as a different user than the user running +L<public-inbox-watch(1)>, L<public-inbox-mda(1)>, or +L<public-inbox-fetch(1)>. + +To save storage, L</publicinbox.pop3state> only stores +the highest-numbered deleted message + +=head1 OPTIONS + +See common options in L<public-inbox-daemon(8)/OPTIONS>. + +=over + +=item -l PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item --listen PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +In addition to the normal C<-l>/C<--listen> switch described in +L<public-inbox-daemon(8)>, the C<PROTOCOL> prefix (e.g. C<pop3://> or +C<pop3s://>) may be specified to force a given protocol. + +For STARTTLS and POP3S support, the C<cert> and C<key> may be specified +on a per-listener basis after a C<?> character and separated by C<,>. +These directives are per-directive, and it's possible to use a different +cert for every listener. + +=item --cert /path/to/cert + +The default TLS certificate for optional STARTTLS and POP3S support +if the C<cert> option is not given with C<--listen>. + +If using systemd-compatible socket activation and a TCP listener on port +995 is inherited, it is automatically POP3S when this option is given. +When a listener on port 110 is inherited and this option is given, it +automatically gets STARTTLS support. + +=item --key /path/to/key + +The default private TLS certificate key for optional STARTTLS and POP3S +support if the C<key> option is not given with C<--listen>. The private +key may be concatenated into the path used by C<--cert>, in which case this +option is not needed. + +=back + +=head1 CONFIGURATION + +Aside from C<publicinbox.pop3state>, C<public-inbox-pop3d> uses the +same configuration knobs as L<public-inbox-nntpd(1)>, +see L<public-inbox-nntpd(1)> and L<public-inbox-config(5)>. + +=over 8 + +=item publicInbox.pop3state + +A directory containing per-user/mailbox account information; +must be writable to the C<public-inbox-pop3d> process. + +=item publicInbox.<name>.newsgroup + +The newsgroup name maps to a POP3 folder name. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/>, and +L<nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta>, +L<nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<git(1)>, L<git-config(1)>, L<public-inbox-daemon(8)>, +L<public-inbox-config(5)>, L<public-inbox-nntpd(1)>, +L<uuidgen(1)> diff --git a/Documentation/public-inbox-purge.pod b/Documentation/public-inbox-purge.pod index a9479657..1223b577 100644 --- a/Documentation/public-inbox-purge.pod +++ b/Documentation/public-inbox-purge.pod @@ -31,7 +31,7 @@ leads to discontiguous git history. =item --all Purge the message in all inboxes configured in ~/.public-inbox/config. -This is an alternative to specifying individual inboxes directories +This is an alternative to specifying individual inbox directories on the command-line. =back @@ -69,12 +69,12 @@ to anyone using L<git(1)> to mirror the inbox being purged. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-tuning.pod b/Documentation/public-inbox-tuning.pod index f5a25676..892ee0f2 100644 --- a/Documentation/public-inbox-tuning.pod +++ b/Documentation/public-inbox-tuning.pod @@ -16,7 +16,7 @@ New inboxes: public-inbox-init -V2 =item 2 -Process spawning +Optional Inline::C use =item 3 @@ -34,6 +34,22 @@ Performance on solid state drives Read-only daemons +=item 7 + +Other OS tuning knobs + +=item 8 + +Scalability to many inboxes + +=item 9 + +public-inbox-cindex --join performance + +=item 10 + +public-inbox-clone with shared object stores + =back =head2 New inboxes: public-inbox-init -V2 @@ -42,7 +58,7 @@ If you're starting a new inbox (and not mirroring an existing one), the L<-V2|public-inbox-v2-format(5)> requires L<DBD::SQLite>, but is orders of magnitude more scalable than the original C<-V1> format. -=head2 Process spawning +=head2 Optional Inline::C use Our optional use of L<Inline::C> speeds up subprocess spawning from large daemon processes. @@ -52,9 +68,17 @@ environment variable to point to a writable directory, or create C<~/.cache/public-inbox/inline-c> for any user(s) running public-inbox processes. +If libgit2 development files are installed and L<Inline::C> +is enabled (described above), per-inbox C<git cat-file --batch> +processes are replaced with a single L<perl(1)> process running +C<PublicInbox::Gcf2::loop> in read-only daemons. libgit2 use +will be available in public-inbox 1.7.0+ + More (optional) L<Inline::C> use will be introduced in the future to lower memory use and improve scalability. +Note: L<Inline::C> is required for L<lei(1)>, but not public-inbox-* + =head2 Performance on rotational hard disk drives Random I/O performance is poor on rotational HDDs. Xapian indexing @@ -63,8 +87,8 @@ RAM. Attempts to parallelize random I/O on HDDs leads to pathological slowdowns as inboxes grow. While C<-V2> introduced Xapian shards as a parallelization -mechanism for SSDs; enabling C<publicInbox.indexSequentialShard> -repurposes sharding as mechanism to reduce the kernel page cache +mechanism for SSDs, enabling C<publicInbox.indexSequentialShard> +repurposes sharding as a mechanism to reduce the kernel page cache footprint when indexing on HDDs. Initializing a mirror with a high C<--jobs> count to create more @@ -92,7 +116,7 @@ indices on btrfs to achieve acceptable performance (even on SSD). Disabling copy-on-write also disables checksumming, thus C<raid1> (or higher) configurations may be corrupt after unsafe shutdowns. -Fortunately, these SQLite and Xapian indices are designed to +Fortunately, these SQLite and Xapian indices are designed to be recoverable from git if missing. Disabling CoW does not prevent all fragmentation. Large values @@ -109,7 +133,7 @@ C<btrfs filesystem defragment -fr $INBOX_DIR> may be necessary. Large filesystems benefit significantly from the C<space_cache=v2> mount option documented in L<btrfs(5)>. -Older, non-CoW filesystems are generally work well out-of-the-box +Older, non-CoW filesystems generally work well out of the box for our Xapian and SQLite indices. =head2 Performance on solid state drives @@ -136,9 +160,51 @@ C<LimitNOFILE=> in L<systemd.exec(5)>) may need to be raised to accommodate many concurrent clients. Transport Layer Security (IMAPS, NNTPS, or via STARTTLS) significantly -increases memory use of client sockets, sure to account for that in +increases memory use of client sockets, be sure to account for that in capacity planning. +Bursts of small object allocations late in process life contribute to +fragmentation of the heap due to arenas (slabs) used internally by Perl. +glibc malloc users should use C<MALLOC_MMAP_THRESHOLD_=131072> to reduce +fragmentation from the sliding mmap window. jemalloc (tested as an +LD_PRELOAD on GNU/Linux) also reduces fragmentation compared to an +unconfigured glibc malloc in long-lived processes. + +=head2 Other OS tuning knobs + +Linux users: the C<sys.vm.max_map_count> sysctl may need to be increased if +handling thousands of inboxes (with L<public-inbox-extindex(1)>) to avoid +out-of-memory errors from git. + +Other OSes may have similar tuning knobs (patches appreciated). + +=head2 Scalability to many inboxes + +L<public-inbox-extindex(1)> allows any number of public-inboxes +to share the same Xapian indices. + +git 2.33+ startup time is orders of magnitude faster and uses +less memory when dealing with thousands of alternates required +for thousands of inboxes with L<public-inbox-extindex(1)>. + +Frequent packing (via L<git-gc(1)>) both improves performance +and reduces the need to increase C<sys.vm.max_map_count>. + +=head2 public-inbox-cindex --join performance + +A C++ compiler and the Xapian development files makes C<--join> or +C<--join=aggressive> orders of magnitude faster in L<public-inbox-cindex(1)>. +On Debian-based systems this is C<libxapian-dev>. RPM-based distros have +these in C<xapian-core-devel> or C<xapian14-core-libs>. *BSDs typically +package development files together with runtime libraries, so the C<xapian> +or C<xapian-core> package will already have the development files. + +=head2 public-inbox-clone with shared object stores + +When mirroring manifests with many forks using the same objstore, +git 2.41+ is highly recommended for performance as we automatically +use the C<fetch.hideRefs> feature to speed up negotiation. + =head1 CONTACT Feedback encouraged via plain-text mail to L<mailto:meta@public-inbox.org> @@ -147,10 +213,10 @@ Information for *BSDs and non-traditional filesystems especially welcome. Our archives are hosted at L<https://public-inbox.org/meta/>, -L<http://hjrcffqmbrq6wope.onion/meta/>, and other places +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>, and other places =head1 COPYRIGHT -Copyright 2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-v1-format.pod b/Documentation/public-inbox-v1-format.pod index e5b1dd06..db223fd9 100644 --- a/Documentation/public-inbox-v1-format.pod +++ b/Documentation/public-inbox-v1-format.pod @@ -2,7 +2,7 @@ =head1 NAME -public-inbox v1 git repository and tree description (aka "ssoma") +public-inbox-v1-format - git repository and tree description (aka "ssoma") =head1 DESCRIPTION @@ -175,7 +175,7 @@ This is up to the administrators. =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2013-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-v2-format.pod b/Documentation/public-inbox-v2-format.pod index 86a9b8f2..de3b0bfd 100644 --- a/Documentation/public-inbox-v2-format.pod +++ b/Documentation/public-inbox-v2-format.pod @@ -2,7 +2,7 @@ =head1 NAME -public-inbox v2 format description +public-inbox-v2-format - structure of public inbox v2 archives =head1 DESCRIPTION @@ -30,7 +30,7 @@ databases for parallelism by "shards". - all.git # empty, alternates to $EPOCH.git - xap$SCHEMA_VERSION/$SHARD # per-shard Xapian DB - xap$SCHEMA_VERSION/over.sqlite3 # OVER-view DB for NNTP, threading - - msgmap.sqlite3 # same the v1 msgmap + - msgmap.sqlite3 # same as the v1 msgmap For blob lookups, the reader only needs to open the "all.git" repository with $GIT_DIR/objects/info/alternates which references @@ -89,7 +89,7 @@ After-the-fact invocations of L<public-inbox-index> will ignore messages written to 'd' after they are written to 'm'. Deltafication is not significantly improved over v1, but overall -storage for trees is made as as small as possible. Initial +storage for trees is made as small as possible. Initial statistics and benchmarks showing the benefits of this approach are documented at: @@ -97,7 +97,7 @@ L<https://public-inbox.org/meta/20180209205140.GA11047@dcvr/> =head2 XAPIAN SHARDS -Another second scalability problem in v1 was the inability to +Another scalability problem in v1 was the inability to utilize multiple CPU cores for Xapian indexing. This is addressed by using shards in Xapian to perform import indexing in parallel. @@ -138,7 +138,7 @@ OVER/XOVER commands). The overview DB maintains all the header information necessary to implement the NNTP OVER/XOVER commands and non-search -endpoints of of the PSGI UI. +endpoints of the PSGI UI. Xapian has become completely optional for v2 (as it is for v1), but SQLite remains required for v2. SQLite turns out to be powerful @@ -235,7 +235,7 @@ and testing of the v2 format. =head1 COPYRIGHT -Copyright 2018-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2018-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-watch.pod b/Documentation/public-inbox-watch.pod index 38686645..6e2142fe 100644 --- a/Documentation/public-inbox-watch.pod +++ b/Documentation/public-inbox-watch.pod @@ -41,16 +41,18 @@ importing them into public-inbox git repositories and indices. public-inbox-watch is useful in situations when a user wishes to mirror an existing mailing list, but has no access to run L<public-inbox-mda(1)> on a server. Unlike public-inbox-mda -which is invoked once per-message, public-inbox-watch is a +which is invoked once per message, public-inbox-watch is a persistent process, making it faster for after-the-fact imports of large Maildirs. Upon startup, it scans the mailbox for new messages to be imported while it was not running. -As of public-inbox 1.6.0, Maildirs, IMAP folders, and NNTP -newsgroups are supported. Previous versions of public-inbox -only supported Maildirs. +All versions of public-inbox-watch support Maildirs. public-inbox +1.6.0 added support for IMAP folders and NNTP newsgroups. +public-inbox 2.0 adds support for MH directories. There are no +plans to support the mbox family since new messages are expensive +to detect in large mboxes. public-inbox-watch should be run inside a L<screen(1)> session or as a L<systemd(1)> service. Errors are emitted to stderr. @@ -62,10 +64,14 @@ public-inbox-watch takes no command-line options. =head1 CONFIGURATION These configuration knobs should be used in the -L<public-inbox-config(5)> file +L<public-inbox-config(5)> file. =over 8 +=item publicinboxImport.dropUniqueUnsubscribe + +See L<public-inbox-config(5)/publicinboxImport.dropUniqueUnsubscribe> + =item publicinbox.<name>.watch A location to watch. public-inbox 1.5.0 and earlier only supported @@ -74,17 +80,24 @@ C<maildir:> paths: [publicinbox "test"] watch = maildir:/path/to/maildirs/.INBOX.test/ -public-inbox 1.6.0 supports C<nntp://>, C<nntps://>, +public-inbox 1.6.0+ supports C<nntp://>, C<nntps://>, C<imap://> and C<imaps://> URLs: watch = nntp://news.example.com/inbox.test.group watch = imaps://user@mail.example.com/INBOX.test +2.0+ supports MH: + + watch = mh:/path/to/MH/inbox.test + This may be specified multiple times to combine several mailboxes into a single public-inbox. URLs requiring authentication -will require L<netrc(5)> and/or L<git-credential(1)> to fill +will require L<netrc(5)> and/or L<git-credential(1)> (preferred) to fill in the username and password. +public-inbox 2.0+ also supports boolean C<false> to prevent the global +L</publicinboxwatch.watchspam> directive from writing to the inbox. + Default: none =item publicinbox.<name>.watchheader @@ -120,7 +133,7 @@ Messages without the (S)een flag are not considered for hiding. This hiding affects all configured public-inboxes in PI_CONFIG. As with C<publicinbox.$NAME.watch>, C<imap://> and C<imaps://> URLs -are supported in public-inbox 1.6.0+. +are supported in public-inbox 1.6.0+, and C<MH> in 2.0+. Default: none; only for L<public-inbox-watch(1)> users @@ -196,12 +209,12 @@ daemons. See L<public-inbox-daemon(8)>. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod index 1bc1b1df..e7c07ed3 100644 --- a/Documentation/public-inbox-xcpdb.pod +++ b/Documentation/public-inbox-xcpdb.pod @@ -30,7 +30,9 @@ Copy all inboxes configured in ~/.public-inbox/config. This is an alternative to specifying individual inboxes directories on the command-line. -=item -c, --compact +=item -c + +=item --compact In addition to performing the copy operation, run L<xapian-compact(1)> on each Xapian shard after copying but before finalizing it. @@ -42,7 +44,9 @@ preferable for gigantic inboxes where the coarse-grained lock currently required for L<public-inbox-compact(1)> can cause the compaction to take hours at-a-time. -=item --reshard=N / -R N +=item -R N + +=item --reshard=N Reshard the Xapian database on a L<v2|public-inbox-v2-format(5)> inbox to C<N> shards . Since L<xapian-compact(1)> is not suitable @@ -52,7 +56,11 @@ existing Xapian database(s) to any positive value of C<N>. This is useful in case the Xapian DB was created with too few or too many shards given the capabilities of the current hardware. -=item --blocksize / --no-full / --fuller +=item --blocksize + +=item --no-full + +=item --fuller These options are passed directly to L<xapian-compact(1)> when used with C<--compact>. @@ -124,12 +132,12 @@ computing resources. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2019-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox.cgi.pod b/Documentation/public-inbox.cgi.pod index 8fd6f3e7..58d59ba2 100644 --- a/Documentation/public-inbox.cgi.pod +++ b/Documentation/public-inbox.cgi.pod @@ -4,7 +4,7 @@ public-inbox.cgi - CGI wrapper for PublicInbox::WWW =head1 SYNOPSIS -You generally want to run public-inbox-httpd, instead +You generally want to run public-inbox-netd or public-inbox-httpd, instead =head1 DESCRIPTION @@ -12,23 +12,30 @@ public-inbox.cgi provides a CGI interface wrapper on top of the PSGI/Plack L<PublicInbox::WWW> module. It is only provided for compatibility reasons and NOT recommended. -CGI with Perl is slow due to code loading overhead and web servers lack -the scheduling fairness of L<public-inbox-httpd(1)> for handling git -clones and streaming large mbox downloads. +CGI with Perl is slow due to code loading overhead and +web servers lack the scheduling fairness of L<public-inbox-netd(1)> +and L<public-inbox-httpd(1)> for handling git clones and +streaming large mbox downloads. + +=head1 COMPATIBILITY NOTE + +When using the CGI with Apache, make sure to set AllowEncodedSlashes to On, as +public-inbox makes heavy use of encoded slashes. =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> =head1 SEE ALSO -L<public-inbox-httpd(1)>, L<PublicInbox::WWW>, L<public-inbox-daemon(8)>, +L<public-inbox-netd(1)>, L<public-inbox-httpd(1)>, +L<PublicInbox::WWW>, L<public-inbox-daemon(8)>, diff --git a/Documentation/reproducibility.txt b/Documentation/reproducibility.txt index 4e56ada4..3336de73 100644 --- a/Documentation/reproducibility.txt +++ b/Documentation/reproducibility.txt @@ -12,7 +12,7 @@ reproducible. Keeping all communications as email ensures the full history of the entire project can be mirrored by anyone with the resources to do so. Compact, low-complexity data requires -less resources to mirror, so sticking with plain-text +less resources to mirror, so sticking with plain text ensures more parties can mirror and potentially fork the project with all its data. @@ -26,4 +26,4 @@ If these things make power hungry project leaders and admins uncomfortable, good. That was the point. It's how checks and balances ought to work. -Comments, corrections, etc welcome: meta@public-inbox.org +Comments, corrections, etc. welcome: meta@public-inbox.org diff --git a/Documentation/standards.perl b/Documentation/standards.perl index 0ac6cc52..743cdee1 100755 --- a/Documentation/standards.perl +++ b/Documentation/standards.perl @@ -1,6 +1,6 @@ #!/usr/bin/perl -w -use strict; -# Copyright 2019-2020 all contributors <meta@public-inbox.org> +use v5.12; +# Copyright all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> print <<EOF; @@ -11,11 +11,11 @@ Non-exhaustive list of standards public-inbox software attempts or intends to implement. This list is intended to be a quick reference for hackers and users. -Given the goals of interoperability and accessibility; strict +Given the goals of interoperability and accessibility, strict conformance to standards is not always possible, but rather best-effort taking into account real-world cases. In particular, "obsolete" standards remain relevant as long as clients and -data exists. +data using them exist. IETF RFCs --------- @@ -25,6 +25,12 @@ EOF my $rfcs = [ 3977 => 'NNTP', 977 => 'NNTP (old)', + 1036 => 'Standard for Interchange of USENET Messages', + 5536 => 'Netnews Article Format', + 5537 => 'Netnews Architecture and Protocols', + 1738 => 'Uniform resource locators', + 5092 => 'IMAP URL scheme', + 5538 => 'NNTP URI schemes', 6048 => 'NNTP additions to LIST command (TODO)', 8054 => 'NNTP compression', 4642 => 'NNTP TLS', @@ -56,8 +62,17 @@ my $rfcs = [ # 8621 => JSON Meta Application Protocol (JMAP) for Mail # ... - # TODO: flesh this out + # examples/unsubscribe.milter and PublicInbox::Unsubscribe + 2369 => 'URLs as Meta-Syntax for Core Mail List Commands', + 8058 => 'Signaling One-Click Functionality for List Email Headers', + + 1081 => 'Post Office Protocol – Version 3', + 1939 => 'Post Office Protocol – Version 3 (STD 53)', + 2449 => 'POP3 extension mechanism', + 2595 => 'STARTTLS for IMAP and POP3', + 2384 => 'POP URL Scheme', + # TODO: flesh this out ]; my @rfc_urls = qw(tools.ietf.org/html/rfc%d @@ -92,6 +107,6 @@ Other relevant documentation Copyright --------- -Copyright (C) 2019-2020 all contributors <meta@public-inbox.org> +Copyright (C) all contributors <meta@public-inbox.org> License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> EOF diff --git a/Documentation/technical/data_structures.txt b/Documentation/technical/data_structures.txt index 8776a67b..11f78041 100644 --- a/Documentation/technical/data_structures.txt +++ b/Documentation/technical/data_structures.txt @@ -32,19 +32,19 @@ Per-message classes Common abbreviation: $mime, $eml Used by: PublicInbox::WWW, PublicInbox::SearchIdx - An representation of an entire email, multipart or not. + A representation of an entire email, multipart or not. An option to use libgmime or libmailutils may be supported in the future for performance and memory use. This can be a memory hog with big messages and giant attachments, so our PublicInbox::WWW interface only keeps - one object of this class in memory at-a-time. + one object of this class in memory at a time. In other words, this is the "meat" of the message, whereas $smsg (below) is just the "skeleton". Our PublicInbox::V2Writable class may have two objects of this - type in memory at-a-time for deduplication. + type in memory at a time for deduplication. In public-inbox 1.4 and earlier, Email::MIME and its subclass, PublicInbox::MIME were used. Despite still slurping, @@ -61,10 +61,10 @@ Per-message classes This is loaded from either the overview DB (over.sqlite3) or the Xapian DB (docdata.glass), though the Xapian docdata - is won't hold NNTP-only fields (Cc:/To:) + won't hold NNTP-only fields (Cc:/To:). There may be hundreds or thousands of these objects in memory - at-a-time, so fields are pruned if unneeded. + at a time, so fields are pruned if unneeded. * PublicInbox::SearchThread::Msg - subclass of Smsg Common abbreviation: $cont or $node @@ -75,9 +75,9 @@ Per-message classes Nowadays, this is a re-blessed $smsg with additional fields. As with $smsg objects, there may be hundreds or thousands - of these objects in memory at-a-time. + of these objects in memory at a time. - We also do not use a linked-list for storing children as JWZ + We also do not use a linked list for storing children as JWZ describes, but instead a Perl hashref for {children} which becomes an arrayref upon sorting. @@ -88,7 +88,7 @@ Per-inbox classes * PublicInbox::Inbox - represents a single public-inbox Common abbreviation: $ibx - Used everywhere + Used everywhere. This represents a "publicinbox" section in the config file, see public-inbox-config(5) for details. @@ -117,7 +117,7 @@ Per-inbox classes * PublicInbox::Search - Xapian read-only interface Common abbreviation: $srch, $ibx->search - Used everywhere if Search::Xapian (or Xapian.pm) is available. + Used everywhere if Xapian is available. Each indexed inbox has one of these, see public-inbox-v1-format(5) and public-inbox-v2-format(5) @@ -152,7 +152,7 @@ ad-hoc structures shared across packages This holds the PSGI $env as well as any internal variables used by various modules of PublicInbox::WWW. - As with the PSGI $env, there is one per-active WWW + As with the PSGI $env, there is one per active WWW request+response cycle. It does not exist for idle HTTP clients. @@ -174,8 +174,8 @@ daemon classes Common abbreviation: $http Used by: PublicInbox::DS, public-inbox-httpd - Unlike PublicInbox::NNTP, this class no knowledge of any of - the email or git-specific parts of public-inbox, only PSGI. + Unlike PublicInbox::NNTP, this class has no knowledge of any of + the email- or git-specific parts of public-inbox, only PSGI. However, it supports APIs and behaviors (e.g. streaming large responses) which PublicInbox::WWW may take advantage of. @@ -188,7 +188,7 @@ daemon classes This class calls non-blocking accept(2) or accept4(2) on a listen socket to create new PublicInbox::HTTP and - PublicInbox::HTTP instances. + PublicInbox::NNTP instances. * PublicInbox::HTTPD Common abbreviation: $httpd @@ -197,9 +197,9 @@ daemon classes wrappers around client sockets accepted from PublicInbox::Listener. - Since the SERVER_NAME and SERVER_PORT PSGI variables needs to be + Since the SERVER_NAME and SERVER_PORT PSGI variables need to be exposed for HTTP/1.0 requests when Host: headers are missing, - this is per-Listener socket. + this is per Listener socket. * PublicInbox::HTTPD::Async Common abbreviation: $async @@ -222,10 +222,8 @@ daemon classes given PublicInbox::Config which may be instantiated more than once in the future. -* PublicInbox::ParentPipe - - Per-worker process class to detect shutdown of master process. - This is not used if using -W0 to disable worker processes - in public-inbox-httpd or public-inbox-nntpd. +* PublicInbox::EOFpipe - This is a per-worker singleton. + Used throughout to trigger a callback when a pipe(7) is closed. + This is frequently used to portably detect process exit without + relying on a catch-all waitpid(-1, ...) call. diff --git a/Documentation/technical/ds.txt b/Documentation/technical/ds.txt index a0793ca2..afead2f1 100644 --- a/Documentation/technical/ds.txt +++ b/Documentation/technical/ds.txt @@ -1,9 +1,14 @@ PublicInbox::DS - event loop and async I/O base class -Our PublicInbox::DS event loop which powers public-inbox-nntpd -and public-inbox-httpd diverges significantly from the -unmaintained Danga::Socket package we forked from. In fact, -it's probably different from most other event loops out there. +Our PublicInbox::DS event loop which powers most of our long-lived +processes(*) diverges significantly from the unmaintained Danga::Socket +package we forked from. In fact, it's probably different from most +other event loops out there. + +Most notably, it uses one-shot, level-trigger, and edge-trigger mode +modes of kqueue|epoll depending on the situation. + +(*) public-inbox-netd,(-httpd,-imapd,-nntpd,-pop3d,-watch) + lei-daemon Most notably: @@ -14,7 +19,7 @@ Most notably: triggers a call. The lack of read/write callback distinction is driven by the - fact TLS libraries (e.g. OpenSSL via IO::Socket::SSL) may + fact that TLS libraries (e.g. OpenSSL via IO::Socket::SSL) may declare SSL_WANT_READ on SSL_write(), and SSL_WANT_READ on SSL_read(). So we end up having to let each user object decide whether it wants to make read or write calls depending on its @@ -30,7 +35,7 @@ Most notably: Reducing the user-supplied code down to a single callback allows subclasses to keep their logic self-contained. The combination of this change and one-shot wakeups (see below) for bidirectional - data flows make asynchronous code easier to reason about. + data flows makes asynchronous code easier to reason about. Other divergences: @@ -48,7 +53,7 @@ Other divergences: Augmented features: -* obj->write(CODEREF) passes the object itself to the CODEREF +* obj->write(CODEREF) passes the object itself to the CODEREF. Being able to enqueue subroutine calls is a powerful feature in Danga::Socket for keeping linear logic in an asynchronous environment. Unfortunately, each subroutine takes several kilobytes of memory. @@ -64,7 +69,7 @@ Augmented features: * ->requeue support. An optimization of the AddTimer(0, ...) idiom for immediately dispatching code at the next event loop iteration. public-inbox uses this for fairly generating large responses - iteratively (see PublicInbox::NNTP::long_response or git_async_cat + iteratively (see PublicInbox::NNTP::long_response or ibx_async_cat for blob retrievals). New features @@ -77,12 +82,11 @@ New features which (if any) events it's interested in for the next loop iteration. * Edge-triggering available via EPOLLET or EV_CLEAR. These reduce wakeups - for unidirectional classes (e.g. PublicInbox::Listener sockets, - and pipes via PublicInbox::HTTPD::Async). + for unidirectional classes when throughput is more important than fairness. * IO::Socket::SSL support (for NNTPS, STARTTLS+NNTP, HTTPS) -* dwaitpid (waitpid wrapper) support for reaping dead children +* awaitpid (waitpid wrapper) support for reaping dead children * reliable signal wakeups are supported via signalfd on Linux, EVFILT_SIGNAL on *BSDs via IO::KQueue. diff --git a/Documentation/technical/memory.txt b/Documentation/technical/memory.txt index bb1c92fd..039694c3 100644 --- a/Documentation/technical/memory.txt +++ b/Documentation/technical/memory.txt @@ -8,12 +8,12 @@ memory-efficient. We strive to keep processes small to improve locality, allow the kernel to cache more files, and to be a good neighbor to other processes running on the machine. Taking advantage of -automatic reference counting (ARC) in Perl allows us +automatic reference counting (ARC) in Perl allows us to deterministically release memory back to the heap. We start with a simple data model with few circular references. This both eases human understanding and reduces -the likelyhood of bugs. +the likelihood of bugs. Knowing the relative sizes and quantities of our data structures, we limit the scope of allocations as much as @@ -48,3 +48,9 @@ In the future, our internal data model will be further flattened and simplified to reduce the overhead imposed by small objects. Large allocations may also be avoided by optionally using Inline::C. + +Finally, the mwrap-perl LD_PRELOAD wrapper was ported to Perl 5 +and enhanced to provide live memory usage tracking on 64-bit systems +with minimal performance impact on production traffic: + + git clone https://80x24.org/mwrap-perl.git diff --git a/Documentation/technical/weird-stuff.txt b/Documentation/technical/weird-stuff.txt new file mode 100644 index 00000000..0c8d6891 --- /dev/null +++ b/Documentation/technical/weird-stuff.txt @@ -0,0 +1,22 @@ +There's a lot of weird code in public-inbox which may be daunting +to new hackers. + +* The event loop (PublicInbox::DS) is an evolution of a fairly standard + C10K event loop. See ds.txt in this directory for more. + +Things got weirder in 2021: + +* The lei command-line tool is backed by a daemon. This was done to + improve startup time for shell completion and manage git/SQLite/Xapian + single-writer during long, parallel imports. It may eventually become + a read-write IMAP/JMAP server. + +* SOCK_SEQPACKET is used extensively in lei, and will likely make its + way into more places, still. + +And even more so in 2022: + +* public-inbox-clone / PublicInbox::LeiMirror relies on ->DESTROY + for make-like dependency management while providing parallelism. + +More to come, lei will expose Maildirs via FUSE 3... diff --git a/Documentation/technical/whyperl.txt b/Documentation/technical/whyperl.txt index de6f912a..db1d9793 100644 --- a/Documentation/technical/whyperl.txt +++ b/Documentation/technical/whyperl.txt @@ -21,7 +21,7 @@ Good Things Perl 5 is installed on many, if not most GNU/Linux and BSD-based servers and workstations. It is likely the most - widely-installed programming environment that offers a + widely installed programming environment that offers a significant amount of POSIX functionality. Users won't have to waste bandwidth or space with giant toolchains or architecture-specific binaries. @@ -47,8 +47,8 @@ Good Things * Predictable performance - While Perl is neither fast or memory-efficient, its - performance and memory use are predictable and does not + While Perl is neither fast nor memory-efficient, its + performance and memory use are predictable and do not require GC tuning by the user. public-inbox is developed for (and mostly on) old @@ -56,7 +56,7 @@ Good Things late 1990s, and any cheap VPS today has more than enough RAM and CPU for handling plain-text email. - Low hardware requirements increases the reach of our software + Low hardware requirements increase the reach of our software to more users, improving centralization resistance. * Compatibility @@ -66,9 +66,11 @@ Good Things Perl had fewer breaking changes than Python or Ruby; we expect that trend to continue given the inertia of Perl 5. - Note: this document was written before the Perl 7 announcement. - We'll continue to monitor and adapt to the situation around - what distros are doing in regard to maintaining compatibility. + As of April 2021, the Perl Steering Committee has confirmed + Perl 7 will require `use v7.0' and existing code should + continue working unchanged: + https://nntp.perl.org/group/perl.perl5.porters/259789 + <CAMvkq_SyTKZD=1=mHXwyzVYYDQb8Go0N0TuE5ZATYe_M4BCm-g@mail.gmail.com> * Built for text processing @@ -84,7 +86,7 @@ Good Things There should be no need to rely on language-specific package managers such as cpan(1), those systems increase - the learning curve for users and systems administrators. + the learning curve for users and system administrators. * Compactness and terseness @@ -96,7 +98,7 @@ Good Things * Performance ceiling and escape hatch With optional Inline::C, we can be "as fast as C" in some - cases. Inline::C is widely-packaged by distros and it + cases. Inline::C is widely packaged by distros and it gives us an escape hatch for dealing with missing bindings or performance problems should they arise. Inline::C use (as opposed to XS) also preserves the software freedom and @@ -133,7 +135,7 @@ Bad Things (m//, substr(), index(), etc.) still require memory copies into userspace, negating a benefit of zero-copy. -* The XS/C API make it difficult to improve internals while +* The XS/C API makes it difficult to improve internals while preserving compatibility. * Lack of optional type checking. This may be a blessing in @@ -159,14 +161,14 @@ Red herrings to ignore when evaluating other runtimes ----------------------------------------------------- These don't discount a language or runtime from being -being used, they're just not interesting. +used, they're just not interesting. * Lightweight threading While lightweight threading implementations are - convenient, they tend to be significantly heavier than a + convenient, they tend to be significantly heavier than pure event-loop systems (or multi-threaded event-loop - systems) + systems). Lightweight threading implementations have stack overhead and growth typically measured in kilobytes. The userspace diff --git a/Documentation/txt2pre b/Documentation/txt2pre index 75d2d12b..b45c52e8 100755 --- a/Documentation/txt2pre +++ b/Documentation/txt2pre @@ -1,29 +1,74 @@ -#!/usr/bin/env perl -# Copyright (C) 2014-2020 all contributors <meta@public-inbox.org> +#!perl -w +# n.b. this is invoked via $(PERL) in makefiles +# Copyright (C) all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> # # Stupid script to make HTML from preformatted, utf-8 text versions, # only generating links for http(s). Markdown does too much # and requires indentation to output preformatted text. -use strict; -use warnings; +use v5.12; use PublicInbox::Linkify; use PublicInbox::Hval qw(ascii_html); -my %xurls; -for (qw[public-inbox.cgi(1) - public-inbox-compact(1) +my (%xurls, %lei); +for (qw[lei(1) + lei-add-external(1) + lei-add-watch(1) + lei-blob(1) + lei-config(1) + lei-convert(1) + lei-daemon(8) + lei-daemon-kill(1) + lei-daemon-pid(1) + lei-edit-search(1) + lei-export-kw(1) + lei-forget-external(1) + lei-forget-mail-sync(1) + lei-forget-search(1) + lei-import(1) + lei-index(1) + lei-init(1) + lei-inspect(1) + lei-lcat(1) + lei-ls-external(1) + lei-ls-label(1) + lei-ls-mail-source(1) + lei-ls-mail-sync(1) + lei-ls-search(1) + lei-ls-watch(1) + lei-mail-diff(1) + lei-mail-sync-overview(7) + lei-overview(7) + lei-p2q(1) + lei-q(1) + lei-rediff(1) + lei-refresh-mail-sync(1) + lei-reindex(1) + lei-rm(1) + lei-rm-watch(1) + lei-security(7) + lei-store-format(5) + lei-tag(1) + lei-up(1) + public-inbox.cgi(1) + public-inbox-cindex(1) + public-inbox-clone(1) public-inbox-config(5) public-inbox-convert(1) public-inbox-daemon(8) public-inbox-edit(1) + public-inbox-extindex(1) + public-inbox-fetch(1) + public-inbox-glossary(7) public-inbox-httpd(1) public-inbox-imapd(1) public-inbox-index(1) public-inbox-init(1) public-inbox-learn(1) public-inbox-mda(1) + public-inbox-netd(1) public-inbox-nntpd(1) public-inbox-overview(7) + public-inbox-pop3d(1) public-inbox-purge(1) public-inbox-v1-format(5) public-inbox-v2-format(5) @@ -33,16 +78,15 @@ for (qw[public-inbox.cgi(1) my ($n) = (/([\w\-\.]+)/); $xurls{$_} = "$n.html"; $xurls{$n} = "$n.html"; + /\Alei-(.+?)\(1\)\z/ and $xurls{"lei $1"} = "$n.html"; } -for (qw[copydatabase(1) xapian-compact(1)]) { - my ($n) = (/([\w\-\.]+)/); - $xurls{$_} = ".$n.1.html" -} +$xurls{'lei/store'} = 'lei-store-format.html'; -for (qw[make(1) flock(2) setrlimit(2) vfork(2) tmpfs(5)]) { +for (qw[make(1) flock(2) setrlimit(2) vfork(2) tmpfs(5) inotify(7) unix(7) + syslog(3)]) { my ($n, $s) = (/([\w\-]+)\((\d)\)/); - $xurls{$_} = "http://www.man7.org/linux/man-pages/man$s/$n.$s.html"; + $xurls{$_} = "https://www.man7.org/linux/man-pages/man$s/$n.$s.html"; } for (qw[git(1) @@ -51,6 +95,7 @@ for (qw[git(1) git-config(1) git-credential(1) git-daemon(1) + git-diff(1) git-fast-import(1) git-fetch(1) git-filter-branch(1) @@ -61,6 +106,7 @@ for (qw[git(1) git-init(1) git-send-email(1) gitrepository-layout(5) + gitglossary(7) ]) { my ($n) = (/([\w\-\.]+)/); $xurls{$_} = "https://kernel.org/pub/software/scm/git/docs/$n.html" @@ -76,6 +122,7 @@ for (qw[ $xurls{$_} = "https://www.freedesktop.org/software/systemd/man/$n.html"; } +# favor upstream docs if they exist, use manpages.debian.org if they don't $xurls{'netrc(5)'} = 'https://manpages.debian.org/stable/ftp/netrc.5.en.html'; $xurls{'mbsync(1)'} = 'https://manpages.debian.org/stable/isync/mbsync.1.en.html'; @@ -91,6 +138,26 @@ $xurls{'git-filter-repo(1)'} = 'https://github.com/newren/git-filter-repo'. $xurls{'ssoma(1)'} = 'https://ssoma.public-inbox.org/ssoma.txt'; $xurls{'cgitrc(5)'} = 'https://git.zx2c4.com/cgit/tree/cgitrc.5.txt'; $xurls{'prove(1)'} = 'https://perldoc.perl.org/prove.html'; +$xurls{'mbox(5)'} = 'https://manpages.debian.org/stable/mutt/mbox.5.en.html'; +$xurls{'mmdf(5)'} = 'https://manpages.debian.org/stable/mutt/mmdf.5.en.html'; +$xurls{'mutt(1)'} = 'https://manpages.debian.org/stable/mutt/mutt.1.en.html'; +$xurls{'torsocks(1)'} = + 'https://manpages.debian.org/stable/torsocks/torsocks.1.en.html'; +$xurls{'curl(1)'} = 'https://manpages.debian.org/stable/curl/curl.1.en.html'; +$xurls{'copydatabase(1)'} = + 'https://manpages.debian.org/stable/xapian-tools/copydatabase.1.en.html'; +$xurls{'xapian-compact(1)'} = + 'https://manpages.debian.org/stable/xapian-tools/xapian-compact.1.en.html'; +$xurls{'xapian-delve(1)'} = + 'https://manpages.debian.org/stable/xapian-tools/xapian-delve.1.en.html'; +$xurls{'gzip(1)'} = 'https://manpages.debian.org/stable/gzip/gzip.1.en.html'; +$xurls{'chmod(1)'} = + 'https://manpages.debian.org/stable/coreutils/chmod.1.en.html'; +$xurls{'kqueue(2)'} = + 'https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2'; +$xurls{'notmuch(1)'} = 'https://notmuchmail.org/manpages/notmuch-1/'; +$xurls{'mairix(1)'} = + 'https://manpages.debian.org/stable/mairix/mairix.1.en.html'; my $str = do { local $/; <STDIN> }; my ($title) = ($str =~ /\A([^\n]+)/); @@ -100,6 +167,9 @@ if ($str =~ /^NAME\n\s+([^\n]+)/sm) { if ($title =~ /([\w\.\-]+)/) { delete $xurls{$1}; } + if ($title =~ /\blei-([\w\-]+)\b/) { + delete $xurls{"lei $1"}; + } } $title = ascii_html($title); my $l = PublicInbox::Linkify->new; @@ -119,9 +189,3 @@ print '<html><head>', "<title>$title</title>", "</head><body><pre>", $str , '</pre></body></html>'; STDOUT->flush; - -# keep mtime on website consistent so clients can cache -if (-f STDIN && -f STDOUT) { - my @st = stat(STDIN); - utime($st[8], $st[9], \*STDOUT); -} |