diff options
Diffstat (limited to 'Documentation')
98 files changed, 7509 insertions, 387 deletions
diff --git a/Documentation/.gitignore b/Documentation/.gitignore index 92510039..142bce32 100644 --- a/Documentation/.gitignore +++ b/Documentation/.gitignore @@ -1,3 +1,4 @@ +/lei*.txt /public-inbox-*.txt /public-inbox.cgi.txt /standards.txt diff --git a/Documentation/RelNotes/v1.4.0.eml b/Documentation/RelNotes/v1.4.0.eml index 6b1bc86e..845895b5 100644 --- a/Documentation/RelNotes/v1.4.0.eml +++ b/Documentation/RelNotes/v1.4.0.eml @@ -1,10 +1,86 @@ +Date: Fri, 17 Apr 2020 08:48:59 +0000 From: Eric Wong <e@yhbt.net> To: meta@public-inbox.org -Subject: [WIP] public-inbox 1.4.0 +Subject: [ANNOUNCE] public-inbox 1.4.0 +Message-ID: <20200417084800.public-inbox-1.4.0-rele@sed> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline -TBD +This release focuses on reproducibility improvements and +bugfixes for corner-cases. Busy instances of PublicInbox::WWW +may also notice memory usage reductions. + +For rare messages lacking Date and/or Received headers, mirrors +now fall back to using the git author/commit times to reindex +them. This ensures search and filtering queries behave +identically on mirrors as they do on the original machine. + +"altid" SQLite dumps are now accessible to all over the WWW +interface via `POST /$INBOX/$ALTID.sql.gz'. + +Busy instances of PublicInbox::WWW (whether via +public-inbox-httpd or another PSGI server) may notice +significant memory usage reductions from the single message +"permalink" lifetime optimization. There also ongoing work to +improve memory lifetime management to reduce the potential for +memory fragmentation in daemons. + +* general changes: + + - `include.*' directives in the public-inbox-config(5) file + are now honored as documented in git-config(1), + thanks to Andreas Rottmann. + + - `+0000' is assumed for dates missing TZ offsets; + thanks to Leah Neukirchen for spotting this regression from + v1.2.0. + + - `<' and `>' characters are dropped to avoid errors in git + in addresses for git, thanks again to Leah for noticing + this long-standing bug. + +* PublicInbox::WWW: + + - memory reductions for message display and rendering + - code preload improved to reduce memory fragmentation + - remove redundant "a=" parameter in links to solver + - escape '&' in hrefs properly + - fix optional address obfuscation in search results + - `POST /$INBOX/$ALTID.sql.gz' endpoint to retrieve SQLite dumps + +* public-inbox-httpd + public-inbox-nntpd: + + - fix SIGUSR2 upgrade in worker-less instances (-W0) + +* public-inbox-httpd: + + - fix RFC 7230 conformance when Content-Length and "chunked" + are both specified + +* public-inbox-index: + + - reproduce original date and time stamps in mirrors for messages + lacking Date: and/or Received: headers + + - new `--compact' (or `-c') switch to perform the equivalent of + public-inbox-compact(1) after indexing each inbox + +* documentation: + + - add Documentation/technical/data_structures.txt for new hackers + +* scripts/import_vger_from_mbox: (not really a production-level script) + + - fix ">From" unescaping thanks to a bug report from Kyle Meyer + +Thanks to Andreas Rottmann, Leah Neukirchen and Kyle Meyer +for their contributions to this release. + +Release tarball available for download over HTTPS or Tor .onion: + +https://yhbt.net/public-inbox.git/snapshot/public-inbox-1.4.0.tar.gz +http://ou63pmih66umazou.onion/public-inbox.git/snapshot/public-inbox-1.4.0.tar.gz Please report bugs via plain-text mail to: meta@public-inbox.org diff --git a/Documentation/RelNotes/v1.5.0.eml b/Documentation/RelNotes/v1.5.0.eml new file mode 100644 index 00000000..d6036d26 --- /dev/null +++ b/Documentation/RelNotes/v1.5.0.eml @@ -0,0 +1,64 @@ +From: Eric Wong <e@yhbt.net> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.5.0 +Date: Sun, 10 May 2020 07:04:00 +0000 +Message-ID: <20200510.public-inbox-1.5.0-rele@sed> +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline + +This release introduces a new pure-Perl lazy email parser, +PublicInbox::Eml, which uses roughly 10% less memory and +is up to 2x faster than Email::MIME. This is a major +internal change + +Limits commonly enforced by MTAs are also enforced in the +new parser, as messages may bypass MTA transports. + +Email::MIME and other Email::* modules are no longer +dependencies nor used at all outside of maintainer validation +tests. + +* public-inbox-index + + - `--max-size=SIZE' CLI switch and `publicinbox.indexMaxSize' + config file option added to prevent indexing of overly + large messages. + + - List-Id headers are indexed in new messages, old messages + can be found after `--reindex'. + +* public-inbox-watch + + - multiple values of `publicinbox.<name>.watchheader' are + now supported, thanks to Kyle Meyer + + - List-Id headers are matched case-insensitively as specified + by RFC 2919 + +* PublicInbox::WWW + + - $INBOX_DIR/description and $INBOX_DIR/cloneurl are not + memoized if missing + + - improved display of threads, thanks to Kyle Meyer + + - search for List-Id is available via `l:' prefix if indexed + + - all encodings are preloaded at startup to reduce fragmentation + + - diffstat linkification and highlighting are stricter and + less likely to linkify tables in cover letters + + - fix hunk header links to solver which were off-by-one line, + thanks again to Kyle Meyer + +Release tarball available for download over HTTPS or Tor .onion: + +https://yhbt.net/public-inbox.git/snapshot/public-inbox-1.5.0.tar.gz +http://ou63pmih66umazou.onion/public-inbox.git/snapshot/public-inbox-1.5.0.tar.gz + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v1.6.0.eml b/Documentation/RelNotes/v1.6.0.eml new file mode 100644 index 00000000..3fe8beae --- /dev/null +++ b/Documentation/RelNotes/v1.6.0.eml @@ -0,0 +1,169 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.6.0 +Date: Wed, 16 Sep 2020 20:03:09 +0000 +Message-ID: <20200916200309.public-inbox-1.6.0-rele@sed> +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline + +A big release containing several performance optimizations, a +new anonymous IMAP server, and more. It represents an +incremental improvement over 1.5 in several areas with more to +come in 1.7. + +The read-only httpd and nntpd daemons no longer block the event +loop when retrieving blobs from git, making better use of SMP +systems while accomodating slow storage. + +Indexing can be now be tuned to give somewhat usable performance +on HDD storage, though we can't defy the laws of physics, either. + +* General changes: + + - ~/.cache/public-inbox/inline-c is automatically used for Inline::C + if it exists. PERL_INLINE_DIRECTORY in env remains supported + and prioritized to support `nobody'-type users without HOME. + + - msgmap.sqlite3 uses journal_mode=TRUNCATE, matching over.sqlite3 + behavior for a minor reduction in VFS traffic + + - public-inbox-tuning(7) - new manpage containing pointers to + various tuning options and tips for certain HW and OS setups. + + - Copy-on-write is disabled on BTRFS for new indices to avoid + fragmentation. See the new public-inbox-tuning(7) manpage. + + - message/{rfc822,news,global} attachments are decoded recursively + and indexed for search. Reindexing (see below) is required + to ensure these attachments are indexed in old messages. + + - inbox.lock (v2) and ssoma.lock (v1) files are written to + on message delivery (or spam removal) to wake up read-only + daemons via inotify or kqueue. + + - `--help' switch supported by command-line tools + +* Upgrading for new features in 1.6 + + The ordering of these steps is only necessary if you intend to + use some new features in 1.6. Future releases may have + different instructions (or be entirely transparent). + + 0. install (use your OS package manager, or "make install") + + 1. restart public-inbox-watch instances if you have any + + 2. Optional: remove Plack::Middleware::Deflater if you're using + a custom .psgi file for PublicInbox::WWW. This only saves + some memory and CPU cycles, and you may also skip this step + if you expect to roll back to 1.5.0 for any reason. + + Steps 3a and 3b may happen in any order, 3b is optional + and is only required to use new WWW and IMAP features. + + 3a. restart existing read-only daemons if you have them + (public-inbox-nntpd, public-inbox-httpd) + + 3b. run "public-inbox-index -c --reindex --rethread --all" + to reindex all configured inboxes + + 4. configure and start the new public-inbox-imapd. This + requires reindexing in 3b, but there's no obligation to + run an IMAP server, either. + +* public-inbox-index + + There are several new options to improve usability on slow, + rotational storage. + + - `--batch-size=BYTES' or publicinbox.indexBatchSize parameter + to reduce frequency of random writes on HDDs + + - `--sequential-shard' or publicInbox.sequentialShard parameter + to improve OS page cache utilization on HDDs. + + - `--no-fsync' when combined with Xapian 1.4+ can be used to + speed up indexing on SSDs and small (default) `--batch-size' + + - `--rethread' option to go with `--reindex' (use sparringly, + see manpage) + + - parallelize v2 updates by default, `--sequential-shard' and + `-j0' is (once again) allowed to disable parallelization + + - (re-)indexing parallelizes blob reads from git + + - `--all' may be specified to index all configured inboxes + +* public-inbox-learn + + - `rm' supports `--all' to remove from all configured inboxes + +* public-inbox-imapd + + - new read-only IMAP daemon similar to public-inbox-nntpd + `AUTH=ANONYMOUS' is supported, but any username and + password for clients without `AUTH=ANONYMOUS' support. + +* public-inbox-nntpd + + - blob reads from git are handled asynchronously + +* public-inbox-httpd + + - Plack::Middleware::Deflater is no longer loaded by default + when no .psgi file is specified; PublicInbox::WWW can rely + on gzip for buffering (see below) + +* PublicInbox::WWW + + - use consistent blank line around attachment links + + - Attachments in message/{rfc822,news,global} messages can be + individually downloaded. Downloading the entire message/rfc822 + file in full remains supported + + - $INBOX_DIR/description is treated as UTF-8 + + - HTML, Atom, and text/plain responses are gzipped without + relying on Plack::Middleware::Deflater + + - Multi-message endpoints (/t.mbox.gz, /T/, /t/, etc) are ~10% faster + when running under public-inbox-httpd with asynchronous blob + retrieval + + - mbox search results may now include all messages pertaining to that + thread. Needs `--reindex' mentioned above in + `Upgrading for new features in 1.6'. + + - fix mbox.gz search results downloads for lynx users + + - small navigation tweaks, more prominent mirroring instructions + +* public-inbox-watch + + - Linux::Inotify2 or IO::KQueue is used directly, + Filesys::Notify::Simple is no longer required + + - NNTP groups and IMAP mailboxes may be watched in addition + to Maildirs (lightly tested). + +* Ongoing internal changes + + - reduce event loop hogging for many-inbox support + + - use more Perl v5.10-isms, future-proof against Perl 8 + + - more consistent variable and field naming, improve internal + documentation and comments + + - start supporting >=40 char git identifiers for SHA-256 + + - test -httpd-specific code paths via Plack::Test::ExternalServer + in addition to generic PSGI paths. + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v1.6.1.eml b/Documentation/RelNotes/v1.6.1.eml new file mode 100644 index 00000000..49d2ade0 --- /dev/null +++ b/Documentation/RelNotes/v1.6.1.eml @@ -0,0 +1,57 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.6.1 +Date: Thu, 31 Dec 2020 23:45:56 +0000 +Message-ID: <20201231234556.public-inbox-1.6.1-rele@sed> +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline + +A small, bugfix release on top of 1.6.0 from September 2020. + +Bug fixes: + +* MIME header decoding no longer warns on undefined variables, + with Perl <5.28. Thanks to a bug report by Ali Alnubani. + https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/ + +* Fixed a message threading bug thanks to a report from Kyle Meyer. + "public-inbox-index --rethread --reindex" will be necessary + in case of certain messages arrive out-of-order. + Link: https://public-inbox.org/meta/87360nlc44.fsf@kyleam.com/ + +* WWW: per-inbox grokmirror manifests no longer return info + for all inboxes, only the root /manifest.js.gz includes all + inboxes. This regression appeared in 1.6. + +* public-inbox-mda matches List-Id headers insensitively, + matching public-inbox-watch behavior. Similarly, List-Id + is always indexed lower-cased for boolean matches to avoid + matching an incorrect term. + +* Newsgroup and Path NNTP headers are now emitted in conformance + with RFC 5536 3.1.[45]. Thanks to Andrey Melnikov for the report: + https://public-inbox.org/meta/CA+PODjpUN5Q4gBFQhAzUNuMasVEdmp9f=8Uo0Ej0mFumdSwi4w@mail.gmail.com/ + +* Inotify fixes for public-inbox-imapd users relying on SIGHUP + reloads and thousands of watches. + +* Read-only daemon fixes around TLS and Linux <4.5 systems + +Bugfixes with minor behavior changes: + +* The X-Status mbox header is now excluded from imports, + just like the Status: header has been for many years. + They have no place in public archives and can be privacy + concern for people sharing archives. + +* WWW prevents deep-linking to attachments to limit abuse + vectors. Noticed by Leah Neukirchen: + https://public-inbox.org/meta/87imagyap9.fsf@vuxu.org/ + +There are also several ocumentation fixes from Uwe Kleine-König +and Kyle Meyer. + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. diff --git a/Documentation/RelNotes/v1.7.0.eml b/Documentation/RelNotes/v1.7.0.eml new file mode 100644 index 00000000..45fd0fa6 --- /dev/null +++ b/Documentation/RelNotes/v1.7.0.eml @@ -0,0 +1,101 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.7.0 +Date: Thu, 04 Nov 2021 07:52:00 +0000 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Message-ID: <2021-11-04-public-inbox-1.7.0-finally-rele@sed> +Content-Disposition: inline + +Another big release focused on multi-inbox search and scalability. + +Special thanks to Konstantin Ryabitsev and Kyle Meyer for +numerous bug reports and documentation help. + +* general changes + + - config file parsing is 2x faster with 50K inboxes + + - deduplication ignores whitespace differences within address fields + + - "PRAGMA optimize" is now issued on commits for SQLite 3.18+ + +* public-inbox-extindex + + A new Xapian + SQLite index able to search across several inboxes. + This may be configured to replace per-inbox Xapian DBs, + (but not per-inbox SQLite indices) and speed up manifest.js.gz + generation. + + See public-inbox-extindex-format(5) and + public-inbox-extindex(1) manpages for more details. + + Using it with "--all" speeds up various multi-inbox operations in + PublicInbox::WWW, public-inbox-nntpd, and public-inbox-imapd. + +* read-only public-inbox-daemon (-httpd, -nntpd, -imapd): + + libgit2 may be used via Inline::C to avoid hitting system pipe + and process limits. See public-inbox-tuning(7) manpage + for more details. + +* various memory usage reductions and workarounds for leaks in + Encode <3.15, these mainly affect PublicInbox::WWW + +* public-inbox-nntpd + + - startup is 6x faster with 50K inboxes if using -extindex + +* PublicInbox::WWW + + - mboxrd search results are returned in reverse Xapian docid order, + so more recent results are more likely to show up first + + - d: and dt: search prefixes allow "approxidate" formats supported + by "git log --since=" + + - manifest.js.gz generation is ~25x faster with -extindex + + - minor navigation improvements in search results HTML page + +* lei - local email interface + + An experimental, subject-to-change, likely-to-eat-your-mail tool for + personal mail as well as interacting with public-inboxes on the local + filesystem or over HTTP(S). See lei(1), lei-overview(7), and other + lei-* manpages for details. This is only ready-to-use w.r.t. external + public-inbox instances, but mail synchronization for personal mail + remains clunky. + +* public-inbox-index + + - non-strict (Subject-based) threading supports non-ASCII characters, + reindexing is necessary for old messages with non-ASCII subjects. + + - --batch-size is now 8M on 64-bit systems for throughput improvements, + higher values are still advised for more powerful hardware. + +* public-inbox-watch + + - IMAP and NNTP code shared with lei, fixing an off-by-one error + in IMAP synchronization for single-message IMAP folders. + + - \Deleted and \Draft messages ignored for IMAP, as they are for + Maildir. + + - IMAP and NNTP connection establishment (including git-credential + prompts) ordering is now tied to config file order. + +Compatibility: + +* Rollbacks all the way to public-inbox 1.2.0 remain supported + +Internal changes + +* public-inbox-index switched to new internal IPC code shared + with lei + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v1.8.0.eml b/Documentation/RelNotes/v1.8.0.eml new file mode 100644 index 00000000..d835f8d8 --- /dev/null +++ b/Documentation/RelNotes/v1.8.0.eml @@ -0,0 +1,47 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.8.0 +Date: Sat, 23 Apr 2022 08:22:59 +0000 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline +Message-ID: <2022-04-23-public-inbox-1.8.0-released@finally> + +A minor release focused on bugfixes and minor improvements. +Upgrades should happen transparently, but downgrading back to +1.7.0 will likely cause problems for lei users (and only lei +users). + +lei users may experience duplicate messages in Maildirs if attempting to +downgrade from 1.8.0 to 1.7.x. public-inbox-* tools are unaffected and +may downgrade freely. + +Bugfixes: + + Numerous test fixes thanks to NixOS developers. + + Long-running daemons are more robust in case of corrupt blobs + or crashes of git-cat-file processes + + PublicInbox::WWW: all CR are removed before LF, fixing display of + CR-CR-LF messages. + + Solver supports SHA-256 code repositories (inbox and lei store support + is still pending). + +Internal updates: + + Reduced dependencies on Inline::C for Linux users; Linux users may + now use lei with neither Inline::C nor Socket::MsgHdr installed. + +New features: + + The --dangerous flag is now supported in public-inbox-index and + public-inbox-extindex to use the Xapian::DB_DANGEROUS flag for initial + indexes. This may reduce SSD/HDD wear at the expense of disallowing + concurrency and data integrity in case of an unexpected shutdown. + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v1.9.0.eml b/Documentation/RelNotes/v1.9.0.eml new file mode 100644 index 00000000..08e16a66 --- /dev/null +++ b/Documentation/RelNotes/v1.9.0.eml @@ -0,0 +1,68 @@ +From: Eric Wong <e@80x24.org> +To: meta@public-inbox.org +Subject: [ANNOUNCE] public-inbox 1.9.0 +Date: Sun, 21 Aug 2022 02:36:59 +0000 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline +Message-ID: <2022-08-21T023659Z-public-inbox-1.9.0-rele@sed> + +Upgrading: + + lei users need to "lei daemon-kill" after installation to load + new code. Normal daemons (read-only, and public-inbox-watch) + will also need restarts, of course, but there's no + backwards-incompatible data format changes so rolling back to + older versions is harmless. + +Major bugfixes: + + * lei no longer freezes from inotify/EVFILT_VNODE handling, + user interrupts (Ctrl-C), nor excessive errors/warnings + + * IMAP server fairness improved to avoid excessive blob prefetch + +New features: + + * POP3 server support added, use either public-inbox-pop3d or + the new public-inbox-netd superserver + + * public-inbox-netd superserver supporting any combination of HTTP, + IMAP, POP3, and NNTP services; simplifying management and allowing + more sharing of memory used for various data structures. + + * public-inbox-httpd and -netd support per-listener .psgi files + + * SIGHUP reloads TLS certs and keys in addition to config and .psgi files + + * "lei reindex" command for lei users to update personal index + in ~/.local/share/lei/store for search improvements below: + +Search improvements: + + These will require --reindex with public-inbox-index and/or + public-inbox-extindex for public inboxes. + + * patchid: prefix search support added to WWW and lei for + "git patch-id --stable" support + + * text inside base-85 binary patches is no longer indexed + to avoid false positives + + * for lei users, "lei reindex" now exists and is required + to take advantage of aforementioned indexing changes + +Performance improvements: + + * IMAP server startup is faster with many mailboxes when using + "public-inbox-extindex --all" + + * NNTP group listings are also faster with many inboxes when + using "public-inbox-extindex --all" + + * various small opcode and memory usage reductions + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/RelNotes/v2.0.0.wip b/Documentation/RelNotes/v2.0.0.wip new file mode 100644 index 00000000..f04d8144 --- /dev/null +++ b/Documentation/RelNotes/v2.0.0.wip @@ -0,0 +1,193 @@ +To: meta@public-inbox.org +Subject: [WIP] public-inbox 2.2.0 +MIME-Version: 1.0 +Content-Type: text/plain; charset=utf-8 +Content-Disposition: inline + +This release includes several new features and fixes; mostly +around improved integration between inboxes and coderepos for +solver. Portability and reliability is also improved, especially +in the internal process management of lei. + +public-inbox-cindex is a new command to index coderepos for +WWW search and perform automatic associations between +coderepos and inboxes. This makes solver vastly more useful +for the WWW UI as admins will no longer have to manually +associate coderepos with inboxes. + +public-inbox-clone gains the ability to mirror entire (or partial) +grokmirror-compatible manifests. + +Internal process and object management data structures are vastly +simplified throughout and error handling made more robust. + +git SHA-256 support remains a work-in-progress for inboxes and +extindex due to the need to interoperate with SHA-1 epochs. + +Upgrading: + + lei users need to "lei daemon-kill" after installation to load + new code. Normal daemons (read-only, and public-inbox-watch) + will also need restarts, of course, but there's no + backwards-incompatible data format changes so rolling back to + older versions is harmless. + +Compatibility: + + Uppercase newsgroup names were always broken with IMAP, POP3, and + -extindex. Uppercase names will now be lowercased by default and + warnings will be emitted. Conflicting newsgroup names (and `inboxdir' + entries if `newsgroup' isn't specified) will also generate warnings + since they break -extindex and the new -cindex (coderepo index). + +New users + hackers: + + The install/ directory includes tools to automate installation and + removal of dependencies for stripped-down or full setups. See + install/README for more details. + +treewide + + * support raw UTF-8 headers from SMTPUTF8 hosts + + * standardize on `#' prefix for stderr diagnostics (previously `I:') + + * SHA-256 coderepos are fully supported (but not inboxes, yet) + + * for daemons serving public traffic, MALLOC_MMAP_THRESHOLD_=131072 is + recommended to reduce fragmentation in glibc malloc, while jemalloc + (tested as an LD_PRELOAD) is another option (at least for 64-bit). + +PublicInbox::WWW + + * support `+' in inbox names + + * support coderepo displays for systems without cgit + + * improve display of git tags, commits and trees in $INBOX/$OID/s/ endpoint + + * numerous memory usage reductions by avoiding Perl scratchpads + + * add #related anchor and search form to find related patches + based on blob OIDs (IOW, exposing `lei p2q' to the web) + + * fix footer in listing of >200 inboxes + + * support dumb HTTP clones of SHA-256 git repos + + * add /$INBOX/$MSGID/d/ endpoint to show diffs in reused Message-IDs + (`lei mail-diff' for the web) + + * support POST /$INBOX/$MSGID/?x=m&q= to limit mbox results to a thread + + * add topics_(new|active).(html|atom) endpoints + + * linkify peer public-inbox addresses in To/Cc headers + +public-inbox-watch: + + * watching MH folders is now supported + +lei + + * use http.proxy / http.<remote>.proxy from system-wide git-config if + unconfigured for lei + + * improve IMAP and NNTP error reporting + + * reduce default IMAP connections to avoid overloading servers + + * compatibility with SQLite <3.8.3 on CentOS 7.x + + * fix `lei q -tt' on locally indexed messages (still broken for remotes: + https://public-inbox.org/meta/20230226170931.M947721@dcvr/ ) + + * `lei import' now sets labels+keywords consistently on all + already imported messages + + * fix `lei up' on saved local queries which previously used -t/--threads + + * `lei convert' output to v2 public-inboxes is now idempotent + + * improved bash completion for labels (see contrib/completion) + + * support for reading (but not writing) MH folders + + * `lei index' accepts `+L:$LABEL' like `lei import' does + +solver (used by lei (rediff|blob), and PublicInbox::WWW) + + * handle copies in patches properly + + * no longer redundantly parallelized within each WWW process + +portability + + * SIGWINCH is handled properly on less common architectures and OSes + + * fix EINTR handling for kqueue users + + * various fixes for CentOS 7.x + + * fix excessive pipelining to `git cat-file' on systems with small + getdelim(3) buffers (mainly affects musl) + + * support Alpine Linux, Dragonfly, NetBSD and OpenBSD. This resulted + not only in bugfixes to our code, but also to Dragonfly and OpenBSD. + + * Inline::C||Socket::MsgHdr no longer required for SCM_RIGHTS + with sendmsg/recvmsg on supported *BSDs. + + * inotify support no longer requires Linux::Inotify2 XS package + for most architectures + +public-inbox-pop3d + + * support `limit=NUM' and `initial_limit=NUM' query parameters + in mailbox names to limit results + +public-inbox-nntpd + + * fix LISTGROUP with range (affects neomutt) + +public-inbox-clone / public-inbox-fetch / `lei add-external --mirror' + + * mtime of downloaded manifest preserved + +public-inbox-clone: + + * parallel mirroring of multiple inboxes/coderepos via manifest, + public-inbox-fetch is not used in this mode + + * new flags to support manifest mirroring include: + --dry-run, --inbox-config=, --project-list=, --prune, --purge, + --keep-going, --jobs, --include=, --exclude=, --objstore=, + --manifest=, --remote-manifest= + See public-inbox-clone(1) man page for more details. + +PublicInbox::SaPlugin::ListMirror + + * List-ID handling special-cased according to RFC 2919 rules + +Search improvements (lei and PublicInbox::WWW) + + * quoted text inside base-85 binary patches is no longer indexed + + * `public-inbox-cindex --join' prefers using Xapian's C++ API + directly to avoid Perl method dispatch overhead to get usable + performance associating ~300 inboxes with over 1K coderepos + (and vice versa). Users requiring such performance will need + a C++ compiler, pkg-config, and the Xapian development files + (see INSTALL). + + This C++ helper will be used more heavily in the future + to enable query parser customizations and other functionality + unavailable from the Xapian SWIG or XS bindings. + +Thanks to all the bug reporters and users who made this release +possible, and thanks for bearing with my anxiety over making releases. + +Please report bugs via plain-text mail to: meta@public-inbox.org + +See archives at https://public-inbox.org/meta/ for all history. +See https://public-inbox.org/TODO for what the future holds. diff --git a/Documentation/clients.txt b/Documentation/clients.txt new file mode 100644 index 00000000..22b98c66 --- /dev/null +++ b/Documentation/clients.txt @@ -0,0 +1,41 @@ +clients and tools related to public-inbox +----------------------------------------- + +While public-inbox exposes NNTP and gzipped mboxrd over HTTP, +some public-inbox-specific/aware tools have sprung up. + +Below is a non-exhaustive list of them. Feel free to send +additions, corrections and discussions to meta@public-inbox.org +Discussions will be visible from our own public-inbox instance: +https://public-inbox.org/meta/ + +Disclaimer: public-inbox itself comes with no warranty or +guarantees; so don't treat any of these links as endorsements, +either. + +* l2md - Maildir and procmail importer using C + libgit2 + https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git + +* b4 - helper utility for patch-based workflows + https://git.kernel.org/pub/scm/utils/b4/b4.git + +* impibe - Perl script to import v1 or v2 to Maildir + https://leahneukirchen.org/dotfiles/bin/impibe + discussion: https://public-inbox.org/meta/87v9m0l8t1.fsf@vuxu.org/ + +* kernel.org helpers - various scripts used by *.kernel.org + https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git + +* grokmirror - git mirroring tool (not public-inbox-specific) + https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git + +* ssoma - v1 only, abandoned in favor of NNTP + https://80x24.org/ssoma.git + +* piem - Emacs tools for working with public-index (and b4) + https://git.kyleam.com/piem/about/ + +There's also a bunch of random scripts in the scripts/ directory +of our source tree at: + + git clone https://public-inbox.org/public-inbox.git diff --git a/Documentation/common.perl b/Documentation/common.perl new file mode 100755 index 00000000..3a6617c4 --- /dev/null +++ b/Documentation/common.perl @@ -0,0 +1,69 @@ +#!perl -w +# Copyright (C) all contributors <meta@public-inbox.org> +# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> +use strict; +use Fcntl qw(SEEK_SET); +my $have_search = eval { require PublicInbox::Search; 1 }; +my $addr = 'meta@public-inbox.org'; +for my $pod (@ARGV) { + open my $fh, '+<', $pod or die "open($pod): $!"; + my $s = do { local $/; <$fh> } // die "read $!"; + my $orig = $s; + $s =~ s!^=head1 COPYRIGHT\n.+?^=head1([^\n]+)\n!=head1 COPYRIGHT + +Copyright all contributors L<mailto:$addr> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1$1 + !ms; + + $s =~ s!^=head1 CONTACT\n.+?^=head1([^\n]+)\n!=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:$addr> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1$1 + !ms; + $have_search and $s =~ s!^=for\scomment\n + ^AUTO-GENERATED-SEARCH-TERMS-BEGIN\n + .+? + ^=for\scomment\n + ^AUTO-GENERATED-SEARCH-TERMS-END\n + !search_terms()!emsx; + $s =~ s/[ \t]+$//sgm; + if ($s eq $orig) { + my $t = time; + utime($t, $t, $fh); + } else { + seek($fh, 0, SEEK_SET) or die "seek: $!"; + truncate($fh, 0) or die "truncate: $!"; + print $fh $s or die "print: $!"; + close $fh or die "close: $!"; + } +} + +sub search_terms { + my $help = eval('\@PublicInbox::Search::HELP'); + my $s = ''; + my $pad = 0; + my $i; + for ($i = 0; $i < @$help; $i += 2) { + my $pfx = $help->[$i]; + my $n = length($pfx); + $pad = $n if $n > $pad; + $s .= $pfx . "\0"; + $s .= $help->[$i + 1]; + $s .= "\f\n"; + } + $pad += 2; + my $padding = ' ' x ($pad + 4); + $s =~ s/^/$padding/gms; + $s =~ s/^$padding(\S+)\0/" $1".(' ' x ($pad - length($1)))/egms; + $s =~ s/\f\n/\n/gs; + $s =~ s/^ //gms; + substr($s, 0, 0, "=for comment\nAUTO-GENERATED-SEARCH-TERMS-BEGIN\n\n"); + $s .= "\n=for comment\nAUTO-GENERATED-SEARCH-TERMS-END\n"; +} diff --git a/Documentation/dc-dlvr-spam-flow.txt b/Documentation/dc-dlvr-spam-flow.txt index d151d272..6210fc7d 100644 --- a/Documentation/dc-dlvr-spam-flow.txt +++ b/Documentation/dc-dlvr-spam-flow.txt @@ -39,7 +39,7 @@ delivery path as well as removing the message from the git tree. * incron - run commands based on filesystem events: http://incron.aiken.cz/ -* sendmail / MTA - we use and recommend use postfix, which includes a +* sendmail / MTA - we use and recommend postfix, which includes a sendmail-compatible wrapper: http://www.postfix.org/ * spamc / spamd - SpamAssassin: http://spamassassin.apache.org/ diff --git a/Documentation/design_notes.txt b/Documentation/design_notes.txt index e871f4c8..95f02556 100644 --- a/Documentation/design_notes.txt +++ b/Documentation/design_notes.txt @@ -18,7 +18,7 @@ Use existing infrastructure * public-inbox can coexist with existing mailing lists, any subscriber to the existing mailing list can begin delivering messages to - public-inbox-mda(1) + public-inbox-mda(1) or public-inbox-watch(1) * public-inbox uses SMTP for posting. Posting a message to a public-inbox instance is no different than sending a message to any _open_ mailing @@ -52,13 +52,15 @@ Why email? There is no need to ask the NSA for backups of your mail archives :) * git, one of the most widely-used version control systems, includes many - tools for for email, including: git-format-patch(1), git-send-email(1), + tools for email, including: git-format-patch(1), git-send-email(1), git-am(1), git-imap-send(1). Furthermore, the development of git itself is based on the git mailing list: https://public-inbox.org/git/ - (or http://hjrcffqmbrq6wope.onion/git/ for Tor users) + (or + http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/git/ + for Tor users). * Email is already the de-facto form of communication in many Free Software - communities.. + communities. * Fallback/transition to private email and other lists, in case the public-inbox host becomes unavailable, users may still directly email @@ -70,16 +72,17 @@ Why git? * git is distributed and robust while being both fast and space-efficient with text data. NNTP was considered, but does not support delta-compression and places no guarantees on data/transport - integrity. However, a read-only NNTP gateway is implemented. + integrity. However, read-only IMAP and NNTP gateways are implemented. * As of 2016, git is widely used and known to nearly all Free Software developers. For non-developers it is packaged for all major GNU/Linux - and *BSD distributions. NNTP is not as widely-used nowadays. + and *BSD distributions. NNTP is not as widely used nowadays, and + most IMAP clients do not have good support for read-only mailboxes. Why perl 5? ----------- -* Perl 5 is widely available on modern *nix systems with good a history +* Perl 5 is widely available on modern *nix systems, with a good history of backwards and forward compatibility. * git and SpamAssassin both use it, so it should be one less thing for @@ -140,6 +143,8 @@ What sucks about public-inbox * some (mostly GUI) mail clients cannot set In-Reply-To headers properly without the original message. +* marketing - as it should: <https://public-inbox.org/marketing.txt> + Scalability notes ----------------- @@ -149,5 +154,5 @@ problems solved. Copyright --------- -Copyright 2013-2020 all contributors <meta@public-inbox.org> +Copyright all contributors <meta@public-inbox.org> License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/design_www.txt b/Documentation/design_www.txt index f15a5562..a0003f99 100644 --- a/Documentation/design_www.txt +++ b/Documentation/design_www.txt @@ -7,7 +7,7 @@ URL and anchor naming /$INBOX/?r=$GIT_COMMIT -> HTML only /$INBOX/new.atom -> Atom feed -#### Optional, relies on Search::Xapian +#### Optional, relies on Xapian /$INBOX/$MESSAGE_ID/t/ -> HTML content of thread (nested) /$INBOX/$MESSAGE_ID/T/ -> HTML content of thread (flat) anchors: @@ -21,6 +21,16 @@ URL and anchor naming /$INBOX/$MESSAGE_ID/t.atom -> Atom feed for thread /$INBOX/$MESSAGE_ID/t.mbox.gz -> gzipped mbox of thread +/$INBOX/$GIT_OID/s/ -> "git show" (via "git apply") + This endpoint requires "coderepo" entries configured for + a given inbox. It can recreate ("solve") blobs from + patch emails using Xapian and git-apply(1). It can also + display non-blob content, but that remains a + work-in-progress. + +/$INBOX/$GIT_OID/s/$FILENAME -> "git show", raw output + As above, but shows the raw (usually text/plain) output. + ### Stable endpoints /$INBOX/$MESSAGE_ID/ -> HTML content anchors: @@ -67,6 +77,23 @@ installed to render uncommon characters. Plain text (raw message) endpoints display in the original encoding(s) of the original email. +Offline friendly +---------------- + +The "/t/", "/T/", "t.mbox.gz" endpoints are designed to be +useful for reading long threads for users with intermittent +connections or saved for offline viewing. + +Date displays are always absolute, not the "X hours ago" +pattern commonly seen because readers may be reading a +previously-saved or cached copy. + +HTML URLs end with '/' or "$FILENAME.html". The reason many +URLs end with the '/' character is so it can trivially be saved +to a directory via wget or similar tools as "index.html", making +it easy to mirror all files ending in ".html" using any static +web server. + Guidelines for using limited HTML --------------------------------- @@ -75,7 +102,7 @@ We also set <title> to make window management easier. We favor <pre>-formatted text since public-inbox is intended as a place to share and discuss patches and code. Unfortunately, long paragraphs -tends to be less readable with fixed-width serif fonts which GUI +tend to be less readable with fixed-width serif fonts which GUI browsers default to. * No graphics, images, or icons at all. We tolerate, but do not @@ -95,12 +122,12 @@ browsers default to. avoided as they do not render well with some displays or user-chosen fonts. -* No JavaScript. JS is historically too buggy and insecure, and we will +* No JavaScript. JS is historically too buggy and insecure, and we will never expect our readers to do either of the following: - a) read and audit all our code for on every single page load - b) trust us and and run code without reading it + a) read and audit all our code on every single page load + b) trust us and run code without reading it -* We only use CSS for one reason: wrapping pre-formatted text +* We only use CSS for one reason: wrapping pre-formatted text. This is necessary because unfortunate GUI browsers tend to be prone to layout widening from unwrapped mailers. Do not expect CSS to be enabled, especially with scary things like: @@ -114,4 +141,4 @@ CSS classes (for user-supplied CSS) ----------------------------------- See examples in contrib/css/ and lib/PublicInbox/WwwText.pm -(or https://public-inbox.org/meta/_/text/color/ soon) +(or <https://public-inbox.org/meta/_/text/color/>) diff --git a/Documentation/extman.perl b/Documentation/extman.perl deleted file mode 100755 index a9a830c0..00000000 --- a/Documentation/extman.perl +++ /dev/null @@ -1,33 +0,0 @@ -#!/usr/bin/perl -w -# Copyright (C) 2019-2020 all contributors <meta@public-inbox.org> -# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> -# prints a manpage to stdout -use strict; -my $xapmsg = 'See https://xapian.org/ for more information on Xapian'; -my $usage = "$0 /path/to/manpage.SECTION.txt"; -my $manpage = shift or die $usage; -my $MAN = $ENV{MAN} || 'man'; -my @args; -$manpage = (split('/', $manpage))[-1]; -$manpage =~ s/\.txt\z//; -$manpage =~ s/\A\.//; # no leading dot (see Documentation/include.mk) -$manpage =~ s/\.(\d+.*)\z// and push @args, $1; # section -push @args, $manpage; - -# don't use UTF-8 characters which readers may not have fonts for -$ENV{LC_ALL} = $ENV{LANG} = 'C'; -$ENV{COLUMNS} = '76'; # same as pod2text default -$ENV{PAGER} = 'cat'; -my $cmd = join(' ', $MAN, @args); -system($MAN, @args) and die "$cmd failed: $!\n"; -$manpage =~ /\A(?:copydatabase|xapian-compact)\z/ and - print "\n\n", $xapmsg, "\n"; - -# touch -r $(man -w $section $manpage) output.txt -if (-f \*STDOUT) { - open(my $fh, '-|', $MAN, '-w', @args) or die "$MAN -w broken?: $!\n"; - chomp(my $path = <$fh>); - my @st = stat($path) or die "stat($path) failed: $!\n"; - # 9 - mtime - utime($st[9], $st[9], \*STDOUT) or die "utime(STDOUT) failed: $!\n"; -} diff --git a/Documentation/flow.ge b/Documentation/flow.ge index 27f2bfcb..5ad92fec 100644 --- a/Documentation/flow.ge +++ b/Documentation/flow.ge @@ -1,9 +1,11 @@ # public-inbox data flow # # Note: choose either "delivery tools" OR "git mirroring tools" -# for a given inboxdir. Combining them for the SAME inboxdir -# will cause conflicts. Of course, different inboxdirs may -# choose different means of getting mail into them. +# for a given inboxdir. Using them simultaneously for the +# SAME inboxdir will cause conflicts. Of course, different +# inboxdirs may choose different means of getting mail into them. +# You may fork any inbox by starting with "git mirroring tools", +# and switching to "delivery tools". graph { flow: down } @@ -13,6 +15,8 @@ graph { flow: down } public-inbox-learn] -> [inboxdir] [git mirroring tools:\n + public-inbox-clone,\n + public-inbox-fetch,\n grok-pull,\n various scripts ] -- git (clone|fetch) &&\n @@ -20,8 +24,10 @@ graph { flow: down } [inboxdir] -> [read-only daemons:\n + public-inbox-netd\n public-inbox-httpd\n + public-inbox-imapd\n public-inbox-nntpd] -# Copyright 2020 all contributors <meta@public-inbox.org> +# Copyright all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/flow.txt b/Documentation/flow.txt index ac3459e4..ed2dd80b 100644 --- a/Documentation/flow.txt +++ b/Documentation/flow.txt @@ -1,9 +1,11 @@ # public-inbox data flow # # Note: choose either "delivery tools" OR "git mirroring tools" -# for a given inboxdir. Combining them for the SAME inboxdir -# will cause conflicts. Of course, different inboxdirs may -# choose different means of getting mail into them. +# for a given inboxdir. Using them simultaneously for the +# SAME inboxdir will cause conflicts. Of course, different +# inboxdirs may choose different means of getting mail into them. +# You may fork any inbox by starting with "git mirroring tools", +# and switching to "delivery tools". +--------------------+ | delivery tools: | @@ -15,8 +17,10 @@ | v +----------------------+ +--------------------+ -| git mirroring tools: | git (clone|fetch) && | | -| grok-pull, | public-inbox-index | inboxdir | +| git mirroring tools: | | | +| public-inbox-clone, | | | +| public-inbox-fetch, | git (clone|fetch) && | inboxdir | +| grok-pull, | public-inbox-index | | | various scripts | ----------------------> | | +----------------------+ +--------------------+ | @@ -24,9 +28,12 @@ v +--------------------+ | read-only daemons: | + | public-inbox-netd | | public-inbox-httpd | + | public-inbox-imapd | | public-inbox-nntpd | +--------------------+ -# Copyright 2020 all contributors <meta@public-inbox.org> +# Copyright all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> +# This file was generated from flow.txt using Graph::Easy diff --git a/Documentation/hosted.txt b/Documentation/hosted.txt deleted file mode 100644 index 188ad254..00000000 --- a/Documentation/hosted.txt +++ /dev/null @@ -1,41 +0,0 @@ -unofficially hosted mirrors at public-inbox.org - -In addition to eating our own dogfood at <https://public-inbox.org/meta/>, -public-inbox.org hosts unofficial archives for several other projects -to further test our own software. - -These mirrors are NOT to be considered reliable or permanent. -Interested parties are strongly encouraged to host their own mirrors. - -The presence of these archives does not imply these projects endorse -public-inbox or public-inbox.org in any way. - -* https://public-inbox.org/bug-gnulib/ - bug-gnulib@gnu.org - Discussion for Gnulib portability/common source project - https://lists.gnu.org/mailman/listinfo/bug-gnulib - -* https://public-inbox.org/git/ - git@vger.kernel.org - Mailing list for the git version control system - http://vger.kernel.org/majordomo-info.html - -* https://public-inbox.org/libc-alpha/ - libc-alpha@sourceware.org - Mailing list for GNU C library development - https://www.gnu.org/software/libc/involved.html - -* https://public-inbox.org/rack-devel/ - rack-devel@googlegroups.com - Development list for the Ruby webserver interface - https://groups.google.com/group/rack-devel - -* https://public-inbox.org/sox-users/ - sox-users@lists.sourceforge.net - Users' list for the SoX sound processing tool - https://lists.sourceforge.net/lists/listinfo/sox-users - -* https://public-inbox.org/sox-devel/ - sox-devel@lists.sourceforge.net - Developers' list for the SoX sound processing tool - https://lists.sourceforge.net/lists/listinfo/sox-devel diff --git a/Documentation/include.mk b/Documentation/include.mk index 207983f0..86851376 100644 --- a/Documentation/include.mk +++ b/Documentation/include.mk @@ -1,4 +1,4 @@ -# Copyright (C) 2013-2020 all contributors <meta@public-inbox.org> +# Copyright (C) all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> all:: @@ -7,23 +7,22 @@ RSYNC_DEST = public-inbox.org:/srv/public-inbox/ AWK = awk MAN = man +# part of `man-db' on Debian, not sure about other distros +LEXGROG = lexgrog + # this is "xml" on FreeBSD and maybe some other distros: XMLSTARLET = xmlstarlet # libgraph-easy-perl from Debian, Graph::Easy from CPAN GRAPH_EASY = graph-easy -# same as pod2text -COLUMNS = 76 - INSTALL = install PODMAN = pod2man PODMAN_OPTS = -v --stderr -d 1993-10-02 -c 'public-inbox user manual' PODMAN_OPTS += -r public-inbox.git podman = $(PODMAN) $(PODMAN_OPTS) -PODTEXT = pod2text -PODTEXT_OPTS = --stderr -podtext = $(PODTEXT) $(PODTEXT_OPTS) + +man2text = COLUMNS=80 MANWIDTH=80 TERM=dumb MANOPT='--nj --nh' man all:: man @@ -50,11 +49,14 @@ install-man: man doc_install :: install-man -check :: check-man -check_man = $(AWK) '{gsub(/\b./,"")}length>80{print;err=1}END{exit(err)}'\ - >&2 && >$@ +check : check-man +check_man = $(AWK) \ + '{gsub(/\b./,"")}$$0 !~ /\.onion/&&length>80{print;e=1}END{exit(e)}' \ + >&2 + +check-man : $(check_80) -check-man :: $(check_80) +check-lexgrog : $(check_lexgrog) all :: $(docs) @@ -67,20 +69,26 @@ Documentation/standards.txt : Documentation/standards.perl # flow.txt is checked into git since Graph::Easy isn't in many distros Documentation/flow.txt : Documentation/flow.ge - (sed -ne '1,/^$$/p' <Documentation/flow.ge; \ - $(GRAPH_EASY) Documentation/flow.ge || \ - cat Documentation/flow.txt; \ + +%.txt : %.ge + (sed -ne '1,/^$$/p' <$<; \ + $(GRAPH_EASY) $< || grep -v '^#' $@; \ echo; \ - sed -ne '/^# Copyright/,$$p' <Documentation/flow.ge \ + sed -ne '/^# Copyright/,$$p' <$< \ ) >$@+ - touch -r Documentation/flow.ge $@+ + echo >>$@+ \ + '# This file was generated from $(@F) using Graph::Easy' + touch -r $< $@+ mv $@+ $@ +Documentation/lei-q.pod : lib/PublicInbox/Search.pm Documentation/common.perl + $(PERL) -I lib -w Documentation/common.perl $@ + NEWS NEWS.atom NEWS.html : $(news_deps) $(PERL) -I lib -w Documentation/mknews.perl $@ $(RELEASES) # check for internal API changes: -check :: NEWS .NEWS.atom.check NEWS.html +check : NEWS .NEWS.atom.check NEWS.html .NEWS.atom.check: NEWS.atom $(XMLSTARLET) val NEWS.atom || \ @@ -98,18 +106,16 @@ doc: $(docs) gz-doc: $(gz_docs) -gz-xdoc: $(gz_xdocs) - rsync-doc: NEWS.atom.gz # /usr/share/doc/rsync/scripts/git-set-file-times{.gz} on Debian systems # It is also at: https://yhbt.net/git-set-file-times -git set-file-times $(docs) $(txt) - $(MAKE) gz-doc gz-xdoc - $(RSYNC) --chmod=Fugo=r -av $(rsync_docs) $(rsync_xdocs) $(RSYNC_DEST) + $(MAKE) gz-doc + $(RSYNC) --chmod=Fugo=r -av $(rsync_docs) $(RSYNC_DEST) clean-doc: $(RM_F) $(man1) $(man5) $(man7) $(man8) $(gz_docs) $(docs_html) \ - $(mantxt) $(rsync_xdocs) \ + $(mantxt) \ NEWS NEWS.atom NEWS.html Documentation/standards.txt \ Documentation/flow.html Documentation/flow.html.gz \ Documentation/flow.txt.gz diff --git a/Documentation/lei-add-external.pod b/Documentation/lei-add-external.pod new file mode 100644 index 00000000..2a131b55 --- /dev/null +++ b/Documentation/lei-add-external.pod @@ -0,0 +1,133 @@ +=head1 NAME + +lei-add-external - add inbox or external index + +=head1 SYNOPSIS + +lei add-external [OPTIONS] LOCATION + +=head1 DESCRIPTION + +Configure lei to search against an external (an inbox or external +index). When C<LOCATION> is an existing local path, it should point +to a directory that is a C<publicinbox.$NAME.inboxdir> or +C<extindex.$NAME.topdir> value in ~/.public-inbox/config. + +=head1 OPTIONS + +=for comment +TODO: mention curl options? + +=over + +=item --boost=NUMBER + +Set priority of a new or existing location. + +Default: 0 + +=item --mirror=URL + +Create C<LOCATION> by mirroring the public-inbox at C<URL>. +C<LOCATION> will have a Makefile with a C<make update> +target to update the external. + +=item --epoch=RANGE + +Restrict clones of L<public-inbox-v2-format(5)> inboxes to the +given range of epochs. The range may be a single non-negative +integer or a (possibly open-ended) C<LOW..HIGH> range of +non-negative integers. C<~> may be prefixed to either (or both) +integer values to represent the offset from the maximum possible +value. + +For example, C<--epoch=~0> alone clones only the latest epoch, +C<--epoch=~2..> clones the three latest epochs. + +Default: C<0..~0> or C<0..> or C<..~0> +(all epochs, all three examples are equivalent) + +=item -v + +=item --verbose + +Provide more feedback on stderr. + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head2 MIRRORING + +=over + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=item --inbox-version=NUM + +Force a remote public-inbox version (must be C<1> or C<2>). +This is auto-detected by default, and this option exists mainly +for testing. + +=back + +The following options are passed to L<public-inbox-init(1)>: + +=over + +=item -j JOBS, --jobs=JOBS + +=item -L LEVEL, --indexlevel=LEVEL + +=back + +The following options are passed to L<public-inbox-index(1)>: + +=over + +=item --batch-size=SIZE + +=item --compact + +=item -j JOBS, --jobs=JOBS + +=item --max-size=SIZE + +=item --sequential-shard + +=item --skip-docdata + +=back + +=head1 FILES + +The configuration for lei resides at C<$XDG_CONFIG_HOME/lei/config>. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-forget-external(1)>, L<lei-ls-external(1)>, L<lei-import(1)>, +L<public-inbox-index(1)>, L<public-inbox-extindex(1)>, +L<public-inbox-extindex-format(5)> diff --git a/Documentation/lei-add-watch.pod b/Documentation/lei-add-watch.pod new file mode 100644 index 00000000..1b48b43b --- /dev/null +++ b/Documentation/lei-add-watch.pod @@ -0,0 +1,37 @@ +=head1 NAME + +lei-add-watch - watch for new messages and flag changes + +=head1 SYNOPSIS + +lei add-watch [OPTIONS] LOCATION [LOCATION...] + +=head1 DESCRIPTION + +Tell lei to watch C<LOCATION> for new messages and flag changes. +Currently only Maildir locations are supported. + +WARNING: watches are not always reliable, occasional use +of L<lei-index(1)> and L<lei-refresh-mail-sync(1)> is recommended +if L<lei-daemon(8)> crashes or needs to be restarted. This will +be improved in the future. + +=for comment +TODO: Document --state? Believe valid values are pause, import-ro, + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-ls-watch(1)>, L<lei-rm-watch(1)> diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod new file mode 100644 index 00000000..558fc54c --- /dev/null +++ b/Documentation/lei-blob.pod @@ -0,0 +1,109 @@ +=head1 NAME + +lei-blob - display a git blob, reconstructing from mail if necessary + +=head1 SYNOPSIS + +lei blob [OPTIONS] OID + +=head1 DESCRIPTION + +Display a git blob. The blob may correspond to a message from the +local store, any local external, or blobs associated with a +project git repository (if run from a git (working) directory). +For blobs which do not exist, it will attempt to recreate the blob +using patch emails. + +=head1 OPTIONS + +=over + +=item --git-dir=DIR + +Specify an additional .git/ directory to scan. This option may be +given multiple times. + +Default: the output of C<git rev-parse --git-dir> + +=item --no-cwd + +Do not look in the git repository of the current working directory. + +=item --no-mail + +Do not look in mail storage for C<OID>. This is implied by +C<--oid-a>, C<--path-a>, and C<--path-b>. + +=item -A OID-A + +=item --oid-a=OID-A + +Provide pre-image object ID as a hint for reconstructing C<OID>. + +=item -a PATH-A + +=item --path-a=PATH-A + +Provide pre-image pathname as a hint for reconstructing C<OID>. + +=item -b PATH-B + +=item --path-b=PATH-B + +Provide post-image pathname as a hint for reconstructing C<OID>. + +=item -v + +=item --verbose + +Provide more feedback on stderr. + +=back + +The following options are also supported and are described in +L<lei-q(1)>. + +=over + +=item --remote + +Remote externals only get queried when the blob needs to be +reconstructed from patch emails. + +=item --no-local + +=item --no-external + +=item -I LOCATION, --include=LOCATION + +=item --exclude=LOCATION + +=item --only=LOCATION + +=item --no-import-remote + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +=item --proxy=PROTOCOL://HOST[:PORT] + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-external(1)>, L<lei-q(1)> diff --git a/Documentation/lei-config.pod b/Documentation/lei-config.pod new file mode 100644 index 00000000..699f45cb --- /dev/null +++ b/Documentation/lei-config.pod @@ -0,0 +1,136 @@ +=head1 NAME + +lei-config - git-config wrapper for lei configuration file + +=head1 SYNOPSIS + +lei config <name> [[<value>] [<value-pattern>]] + +lei config -l | --list + +lei config -e | --edit + +=head1 DESCRIPTION + +Call L<git-config(1)> with C<$XDG_CONFIG_HOME/lei/config> as the +configuration file. All C<OPTIONS> are passed through, but those that +override the configuration file are not permitted. + +All C<imap> and C<nntp> options may be specified per-host or +(if using git 2.26+) with wildcards: + + [imap "imap://*.onion"] + proxy = socks5h://127.0.0.1:9050 + + [nntp "nntp://example.com"] + proxy = socks5h://127.0.0.1:1080 + +=head2 VARIABLES + +=over 8 + +=item external.* + +Managed by L<lei-add-external(1)> and L<lei-forget-external(1)> + +=item imap.proxy + +=item nntp.proxy + +The C<socks5h://> proxy address. Older versions of SOCKS may +be supported if there is user demand. + +=item imap.starttls + +=item nntp.starttls + +Enable or disable STARTTLS on non-imaps:// and non-nntps:// +hosts. By default, STARTTLS is enabled if available unless +connecting to a Tor .onion or localhost. + +=item imap.compress + +=item nntp.compress + +Enable protocol-level compression. This may be incompatible +or broken with some servers. + +Note: L<Net::NNTP> compression support is pending: +L<https://rt.cpan.org/Ticket/Display.html?id=129967> + +=item imap.debug + +=item nntp.debug + +Enable debugging output of underlying IMAP and NNTP libraries, +currently L<Mail::IMAPClient> and L<Net::NNTP>, respectively. +If L<imap.proxy> or L<nntp.proxy> points to a SOCKS proxy, +debugging output for L<IO::Socket::Socks> will be enabled as +well. + +Disabling L<imap.compress> may be required for readability. + +=item imap.timeout + +=item nntp.timeout + +The read timeout for responses. + +Default: 600 seconds (IMAP); 120 seconds (NNTP) + +=item imap.fetchBatchSize + +Number of full messages to fetch at once. Larger values reduce +network round trips at the cost of higher memory use, especially +when retrieving large messages. + +Small responses for IMAP flags are fetched at 10000 times this value. + +Default: 1 + +=item color.SLOT + +C<quoted>, C<hdrdefault>, C<status>, C<attachment> color slots +are supported for the C<-f text> and C<-f reply> output formats +of L<lei-lcat(1)> and L<lei-q(1)>. + +Any per-project .git/config, and global ~/.gitconfig files +will also be parsed for diff coloring. git diff color slots +(C<color.diff.SLOT>) supported are C<new>, C<old>, C<meta>, +C<frag>, C<func>, and C<context>. + +=back + +=head1 OPTIONS + +Most L<git-config(1)> command-line switches are accepted by C<lei config> +as is. The most frequently used options are expected to be: + +=over 4 + +=item -e + +=item --edit + +Open an editor to edit the lei config file. + +=item -l + +=item --list + +List all variables set in config file, along with their values. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/lei-convert.pod b/Documentation/lei-convert.pod new file mode 100644 index 00000000..b3e29824 --- /dev/null +++ b/Documentation/lei-convert.pod @@ -0,0 +1,70 @@ +=head1 NAME + +lei-convert - one-time conversion from one mail format to another + +=head1 SYNOPSIS + +lei convert -o OUTPUT [OPTIONS] LOCATION + +lei convert -o OUTPUT [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Convert messages to another format. C<LOCATION> is a source of +messages: a directory (Maildir), a file (various mbox), or a URL +(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs +requiring authentication use L<git-credential(1)> to +fill in the username and password. + +For a regular file, the location must have a C<E<lt>formatE<gt>:> +prefix specifying one of the following formats: C<mboxrd>, +C<mboxcl2>, C<mboxcl>, or C<mboxo>. + +=head1 OPTIONS + +=over + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +Message input format. Unless messages are given on stdin, using a +format prefix with C<LOCATION> is preferred. + +=back + +The following options are also supported and are described in +L<lei-q(1)>. + +=over + +=item -o MFOLDER, --output=MFOLDER + +=item --lock METHOD + +=item --no-kw + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +=item --proxy=PROTOCOL://HOST[:PORT] + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-mail-formats(5)> diff --git a/Documentation/lei-daemon-kill.pod b/Documentation/lei-daemon-kill.pod new file mode 100644 index 00000000..50f75f4c --- /dev/null +++ b/Documentation/lei-daemon-kill.pod @@ -0,0 +1,51 @@ +=head1 NAME + +lei-daemon-kill - signal the lei-daemon + +=head1 SYNOPSIS + +lei daemon-kill [-SIGNAL | -s SIGNAL | --signal SIGNAL] + +=head1 DESCRIPTION + +Send a signal to the L<lei-daemon(8)>. C<SIGNAL> defaults to C<TERM>. + +This command should be run after updating the code of lei. + +=head1 SIGNALS + +=over 8 + +=item SIGTERM + +Send a graceful termination signal. L<lei-daemon(8)> will exit +when all currently running lei commands are done. The listen +socket will be released as soon as the signal is processed +so another L<lei-daemon(8)> process can take its place. + +=item SIGKILL + +Kills L<lei-daemon(8)> immediately. Some worker processes may +remain running for a short while. + +=back + +=for comment +SIGQUIT and SIGINT currently do what SIGTERM does, may change... + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-daemon-pid(1)>, L<lei-daemon(8)> diff --git a/Documentation/lei-daemon-pid.pod b/Documentation/lei-daemon-pid.pod new file mode 100644 index 00000000..8637324b --- /dev/null +++ b/Documentation/lei-daemon-pid.pod @@ -0,0 +1,28 @@ +=head1 NAME + +lei-daemon-pid - show the PID of the lei-daemon + +=head1 SYNOPSIS + +lei daemon-pid + +=head1 DESCRIPTION + +Show the PID of the lei-daemon. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-daemon-kill(1)> diff --git a/Documentation/lei-daemon.pod b/Documentation/lei-daemon.pod new file mode 100644 index 00000000..92bc9ac5 --- /dev/null +++ b/Documentation/lei-daemon.pod @@ -0,0 +1,61 @@ +=head1 NAME + +lei-daemon - technical information for local email interface daemon + +=head1 DESCRIPTION + +This documentation is a high-level overview for developers and +administrators interested in how lei works. + +lei-daemon is a background daemon which powers the L<lei(1)> +command-line tool. It may support virtual users and read-write +IMAP+JMAP APIs in the future. It is designed to optimize shell +completion by avoiding module loading costs, monitor Maildirs +(and in the near future, IMAP folders) for changes. + +=head2 worker processes + +Most commands cause lei-daemon to L<fork(2)> new worker +processes to isolate and parallelize work. lei-daemon is +significantly more aggressive than read-only +L<public-inbox-daemon(8)> processes with regards to resource use +since it's not designed to support C10K/C100K scenarios. + +=head2 file descriptor passing + +FD passing is used to reduce IPC costs for bulk I/O when +importing large mboxes from stdin and dumping large mboxes +to stdout. + +=head2 SOCK_SEQPACKET + +SOCK_SEQPACKET sockets are used for both communicating with +L<lei(1)> and to internal workers. SOCK_SEQPACKET guarantees +reliability (unlike SOCK_DGRAM), allows easy load distribution, +and saves developers the trouble of maintaining stream parsers. + +=head2 file monitoring + +Inotify or EVFILT_VNODE is used depending on the platform +to monitor Maildirs for changes and track keyword changes. + +The listen socket (default: C<$XDG_RUNTIME_DIR/lei/5.seq.sock>) +is also monitored, and the daemon will automatically shutdown +if it is unlinked. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-overview(7)>, L<lei-daemon-kill(1)>, L<lei-daemon-pid(1)> diff --git a/Documentation/lei-edit-search.pod b/Documentation/lei-edit-search.pod new file mode 100644 index 00000000..7f447ca2 --- /dev/null +++ b/Documentation/lei-edit-search.pod @@ -0,0 +1,30 @@ +=head1 NAME + +lei-edit-search - edit saved search + +=head1 SYNOPSIS + +lei edit-search [OPTIONS] OUTPUT + +=head1 DESCRIPTION + +Invoke C<git config --edit> to edit the saved search at C<OUTPUT>, +where C<OUTPUT> was supplied for argument of C<lei q -o OUTPUT ...> +A listing of outputs is available via C<lei ls-search>. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-ls-search(1)>, L<lei-forget-search(1)>, L<lei-up(1)>, L<lei-q(1)> diff --git a/Documentation/lei-export-kw.pod b/Documentation/lei-export-kw.pod new file mode 100644 index 00000000..4cb673ac --- /dev/null +++ b/Documentation/lei-export-kw.pod @@ -0,0 +1,55 @@ +=head1 NAME + +lei-export-kw - export keywords (flags) to Maildir and IMAP folders + +=head1 SYNOPSIS + +lei export-kw --all=[<remote|local>] + +lei export-kw MFOLDER [MFOLDER...] + +=head1 DESCRIPTION + +C<lei export-kw> propagates keywords (e.g. C<seen>, C<answered>, +C<flagged>, etc.) from lei/store to IMAP folders and/or Maildirs. +It only works for messages lei knows about (e.g. was used as a +C<lei q --output>, or imported via L<lei-import(1)>, or indexed +via L<lei-index(1)>). + +It does not delete, write, nor modify messages themselves; +it only sets metadata on Maildirs and IMAP folders. + +=head1 OPTIONS + +=over + +=item --all + +Export to all local Maildirs and remote IMAP folders + +=item --all=local + +Export all local Maildirs + +=item --all=remote + +Export all remote IMAP folders + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-refresh-mail-sync(1)>, L<lei-tag(1)> diff --git a/Documentation/lei-forget-external.pod b/Documentation/lei-forget-external.pod new file mode 100644 index 00000000..e0e3b8e1 --- /dev/null +++ b/Documentation/lei-forget-external.pod @@ -0,0 +1,42 @@ +=head1 NAME + +lei-forget-external - forget external locations + +=head1 SYNOPSIS + +lei forget-external [OPTIONS] LOCATION [LOCATION...] + +=head1 DESCRIPTION + +Forget the specified externals by removing their entries from +C<$XDG_CONFIG_HOME/lei/config>. This excludes the locations from +future search results. + +=head1 OPTIONS + +=over + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-add-external(1)>, L<lei-ls-external(1)> diff --git a/Documentation/lei-forget-mail-sync.pod b/Documentation/lei-forget-mail-sync.pod new file mode 100644 index 00000000..29ff5671 --- /dev/null +++ b/Documentation/lei-forget-mail-sync.pod @@ -0,0 +1,32 @@ +=head1 NAME + +lei-forget-mail-sync - forget sync information for a mail folder + +=head1 SYNOPSIS + +lei forget-mail-sync [OPTIONS] LOCATION [LOCATION...] + +=head1 DESCRIPTION + +Forget synchronization information for C<LOCATION>, an IMAP or Maildir +folder. Note that this won't delete any messages on the filesystem. +Users using L<lei-index(1)> without L<lei-import(1)> will be left +with dangling references in search results. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-ls-mail-sync(1)>, L<lei-index(1)> diff --git a/Documentation/lei-forget-search.pod b/Documentation/lei-forget-search.pod new file mode 100644 index 00000000..5ff526f1 --- /dev/null +++ b/Documentation/lei-forget-search.pod @@ -0,0 +1,43 @@ +=head1 NAME + +lei-forget-search - forget saved search + +=head1 SYNOPSIS + +lei forget-search [OPTIONS] OUTPUT + +=head1 DESCRIPTION + +Forget a saved search at C<OUTPUT>, +where C<OUTPUT> was supplied for argument of C<lei q -o OUTPUT ...> +A listing of outputs is available via C<lei ls-search>. + +=head1 OPTIONS + +=over + +=item --prune[=<local|remote>] + +C<--prune> will forget saved searches if the C<OUTPUT> no longer +exists. C<--prune=local> only prunes local mailboxes, +C<--prune=remote> only prunes remote mailboxes (currently +C<imap://> and C<imaps://>). + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-edit-search(1)>, L<lei-ls-search(1)>, L<lei-up(1)>, L<lei-q(1)> diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod new file mode 100644 index 00000000..31d6db13 --- /dev/null +++ b/Documentation/lei-import.pod @@ -0,0 +1,111 @@ +=head1 NAME + +lei-import - one-time import of messages into local store + +=head1 SYNOPSIS + +lei import [OPTIONS] LOCATION [LOCATION...] [+L:LABEL] + +lei import [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Import messages into the local storage of L<lei(1)> +(aka L<leiE<sol>store|lei-store-format(5)>). C<LOCATION> is a +source of messages: a directory (Maildir), a file, or a URL +(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring +authentication use L<git-credential(1)> to +fill in the username and password. + +For a regular file, the C<LOCATION> must have a C<E<lt>formatE<gt>:> +prefix specifying one of the following formats: C<mboxrd>, +C<mboxcl2>, C<mboxcl>, or C<mboxo>. + +=head1 OPTIONS + +=over + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +Message input format. Unless messages are given on stdin, using a +format prefix with C<LOCATION> is preferred. + +=item --stdin + +Read messages from stdin. + +=item --lock + +L<mbox(5)> locking method(s) to use: C<dotlock>, C<fcntl>, C<flock> or +C<none>. + +Default: fcntl,dotlock + +=item +L:LABEL + +Add the given C<LABEL> to all messages imported, where C<LABEL> +is an arbitrary user-defined value consisting of lowercase and digits. +See L<lei-tag(1)> for more info on labels. + +For example, specifying C<+L:inbox> applies the C<inbox> label +to all messages being imported. + +May be specified multiple times to apply multiple labels. + +Default: none + +=item +kw:KEYWORD + +Apply C<KEYWORD> to all messages being imported in addition +to any per-message keywords from the store (unless C<--no-kw> +is specified). See L<lei-tag(1)> for more info on keywords. + +May be specified multiple times to apply multiple keywords. + +Default: none + +=item --no-kw + +Don't import message keywords (or "flags" in IMAP terminology). + +=item --no-incremental + +Import already seen IMAP and NNTP articles. + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=item --proxy=PROTOCOL://HOST[:PORT] + +Use the specified proxy (e.g., C<socks5h://0:9050>). + +Consider L<imap.proxy> and L<nntp.proxy> which can be persistently +configured on a per-host basis in L<lei-config(1)>. + +=back + +See L<lei-config(1)> for various C<imap.*> and C<nntp.*> options. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-config(1)>, L<lei-index(1)>, L<lei-store-format(5)> diff --git a/Documentation/lei-index.pod b/Documentation/lei-index.pod new file mode 100644 index 00000000..f8ff6950 --- /dev/null +++ b/Documentation/lei-index.pod @@ -0,0 +1,64 @@ +=head1 NAME + +lei-index - index messages without importing them into lei/store + +=head1 SYNOPSIS + +lei index [OPTIONS] FOLDER + +=head1 DESCRIPTION + +Similar to L<lei-import(1)>, but does not store a copy of +messages into C<lei/store>. + +This command only makes sense for messages stored in Maildir +folders. Other folder types may be supported in the future +(they can all be indexed, but the message isn't automatically +retrieved by L<lei-q(1)> or L<lei-lcat(1)>). + +Combined with L<lei-q(1)>, C<lei index> allows Maildir users to +have similar functionality to L<mairix(1)> by not duplicating +messages into C<lei/store>. + +Occasional invocations of C<lei-refresh-mail-sync --all=local> +are recommended to keep indexed messages retrievable. + +=head1 OPTIONS + +=over + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +There is currently no need for this option. It will support C<mh>, +eventually. For now, the default (and only supported) format is +C<maildir>. When IMAP and NNTP support are fleshed out, those +formats will be inferred from their URLs. + +Default: C<maildir> + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-refresh-mail-sync(1)>, L<lei-store-format(5)>, L<lei-import(1)> diff --git a/Documentation/lei-init.pod b/Documentation/lei-init.pod new file mode 100644 index 00000000..4bfc3b9f --- /dev/null +++ b/Documentation/lei-init.pod @@ -0,0 +1,44 @@ +=head1 NAME + +lei-init - initialize storage + +=head1 SYNOPSIS + +lei init [OPTIONS] [DIRNAME] + +=head1 DESCRIPTION + +Initialize local writable storage for L<lei(1)>. If C<DIRNAME> is +unspecified, the storage is created at C<$XDG_DATA_HOME/lei/store>. +C<leistore.dir> in C<$XDG_CONFIG_HOME/lei/config> records this +location. + +=head1 OPTIONS + +=over + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-external(1)> diff --git a/Documentation/lei-inspect.pod b/Documentation/lei-inspect.pod new file mode 100644 index 00000000..82b9651a --- /dev/null +++ b/Documentation/lei-inspect.pod @@ -0,0 +1,57 @@ +=head1 NAME + +lei-inspect - general purpose inspector + +=head1 SYNOPSIS + +lei inspect [OPTIONS] ITEM [ITEM...] + +lei inspect [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +This is a diagnostic command that provides a general purpose inspector +of various things, including blobs, message IDs, Xapian document IDs, +and mail sync sources. + +=head1 OPTIONS + +=over + +=item -d DIR + +=item --dir=DIR + +An inboxdir, extindex topdir, or Xapian shard + +=item --pretty + +Pretty-print output. If stdout is opened to a tty, C<--pretty> is +enabled by default. + +=item - + +=item --stdin + +Read message from stdin. This is implicit if no arguments are given +and stdin is a pipe or regular file. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-mail-diff(1)> diff --git a/Documentation/lei-lcat.pod b/Documentation/lei-lcat.pod new file mode 100644 index 00000000..530b755e --- /dev/null +++ b/Documentation/lei-lcat.pod @@ -0,0 +1,98 @@ +=head1 NAME + +lei-lcat - display local copy of messages(s) + +=head1 SYNOPSIS + +lei lcat [OPTIONS] MSGID_OR_URL [MSGID_OR_URL...] + +lei lcat [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +lcat (local cat) is a wrapper around L<lei-q(1)> that displays local +messages by Message-ID. It is able to extract Message-IDs from URLs +as well as from common formats such as C<E<lt>$MSGIDE<gt>> and +C<id:$MSGID>. When reading from stdin, input that isn't understood is +discarded, so the caller doesn't have to bother extracting the +Message-ID or link from surrounding text (e.g., a "Link: $URL" line). + +=head1 OPTIONS + +The following options, described in L<lei-q(1)>, are supported. +One deviation from L<lei-q(1)> is the default output format is +C<-f text> when writing to stdout. + +=over + +=item --format=FORMAT + +=item -f FORMAT + +Most commonly C<text> (the default) or C<reply> to +display the message(s) in a format suitable for trimming +and sending as an email reply. + +=item --stdin + +=item - + +C<lei lcat> implicitly reads from stdin if it is a L<pipe(7)> +or regular file. This is handy for invoking C<lei lcat> from +inside an C<$EDITOR> session (assuming you use an C<$EDITOR> +which lets you pipe arbitrary lines to arbitrary commands). + +=item --[no-]remote + +=item --no-local + +=item --no-external + +=item --no-import-remote + +=item --torsocks=auto|no|yes, --no-torsocks + +=item --proxy=PROTOCOL://HOST[:PORT] + +=item -o MFOLDER, --output=MFOLDER + +=item -d STRATEGY, --dedupe=STRATEGY + +=item -t, --threads + +=item -s KEY, --sort=KEY + +=item -r, --reverse + +=item --offset=NUMBER + +=item -g, --globoff + +=item -a, --augment + +=item --lock=METHOD + +=item --alert=COMMAND + +=item --mua=COMMAND + +=item --no-color + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)>, L<lei-blob(1)> diff --git a/Documentation/lei-ls-external.pod b/Documentation/lei-ls-external.pod new file mode 100644 index 00000000..4c0c263d --- /dev/null +++ b/Documentation/lei-ls-external.pod @@ -0,0 +1,55 @@ +=head1 NAME + +lei-ls-external - list inbox and external index locations + +=head1 SYNOPSIS + +lei ls-external [OPTIONS] [FILTER] + +=head1 DESCRIPTION + +List configured externals. If C<FILTER> is given, restrict the output +to matching entries. + +=head1 OPTIONS + +=over + +=item -g + +=item --globoff + +Do not match C<FILTER> using C<*?> wildcards and C<[]> ranges. + +=item --local + +Limit operations to the local filesystem. + +=item --remote + +Limit operations to those requiring network access. + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-add-external(1)>, L<lei-forget-external(1)> diff --git a/Documentation/lei-ls-label.pod b/Documentation/lei-ls-label.pod new file mode 100644 index 00000000..8342705a --- /dev/null +++ b/Documentation/lei-ls-label.pod @@ -0,0 +1,48 @@ +=head1 NAME + +lei-ls-label - list labels + +=head1 SYNOPSIS + +lei ls-label [OPTIONS] + +=head1 DESCRIPTION + +List all known message labels ("mailboxes" in JMAP terminology). +This is handy for writing L<lei-import(1)> invocations. + +=head1 OPTIONS + +=over + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-external(1)> diff --git a/Documentation/lei-ls-mail-source.pod b/Documentation/lei-ls-mail-source.pod new file mode 100644 index 00000000..0e485923 --- /dev/null +++ b/Documentation/lei-ls-mail-source.pod @@ -0,0 +1,59 @@ +=head1 NAME + +lei-ls-mail-source - list IMAP or NNTP mail source folders + +=head1 SYNOPSIS + +lei ls-mail-source [OPTIONS] URL + +=head1 DESCRIPTION + +List information about the IMAP or NNTP mail source at C<URL>. +This command populates the cache used for Bash shell completion +and is handy for writing L<lei-import(1)> invocations. + +=head1 OPTIONS + +=over + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. + +=item -l + +Format output as JSON and include more information. + +=item --pretty + +Pretty-print JSON output. If stdout is opened to a tty, C<--pretty> +is enabled by default. + +=item --ascii + +Escape non-ASCII characters. + +=item --url + +Show full URL of newsgroup or IMAP folder. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-import(1)> diff --git a/Documentation/lei-ls-mail-sync.pod b/Documentation/lei-ls-mail-sync.pod new file mode 100644 index 00000000..883eeead --- /dev/null +++ b/Documentation/lei-ls-mail-sync.pod @@ -0,0 +1,55 @@ +=head1 NAME + +lei-ls-mail-sync - list mail sync folders + +=head1 SYNOPSIS + +lei mail-sync [OPTIONS] [FILTER] + +=head1 DESCRIPTION + +List mail sync folders. If C<FILTER> is given, restrict the output to +matching entries. + +=head1 OPTIONS + +=over + +=item -g + +=item --globoff + +Do not match C<FILTER> using C<*?> wildcards and C<[]> ranges. + +=item --local + +Limit operations to the local filesystem. + +=item --remote + +Limit operations to those requiring network access. + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-refresh-mail-sync(1)>, L<lei-export-kw(1)> diff --git a/Documentation/lei-ls-search.pod b/Documentation/lei-ls-search.pod new file mode 100644 index 00000000..367f4ad6 --- /dev/null +++ b/Documentation/lei-ls-search.pod @@ -0,0 +1,66 @@ +=head1 NAME + +lei-ls-search - list saved search queries + +=head1 SYNOPSIS + +lei ls-search [OPTIONS] [PREFIX] + +=head1 DESCRIPTION + +List saved search queries (generated from C<lei q -o OUTPUT>). +If C<PREFIX> is given, restrict the output +to entries that start with the specified value. + +=head1 OPTIONS + +=over + +=item -f FORMAT + +=item --format=FORMAT + +Display JSON output rather than default short output that includes +only the saved search location. Possible values are C<json>, +C<jsonl>, or C<concatjson>. + +=item --pretty + +Pretty-print C<json> or C<concatjson> output. If stdout is opened to +a tty and used as the C<--output> destination, C<--pretty> is enabled +by default. + +=item -l + +Long listing format (shortcut for C<--format=json>). + +=item --ascii + +Escape non-ASCII characters. + +=item -z + +=item -0 + +Use C<\0> (NUL) instead of newline (CR) to delimit lines. This option +is incompatible with C<--format>. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)>, L<lei-up(1)>, L<lei-edit-search(1)>, +L<lei-forget-search(1)> diff --git a/Documentation/lei-ls-watch.pod b/Documentation/lei-ls-watch.pod new file mode 100644 index 00000000..8063d34d --- /dev/null +++ b/Documentation/lei-ls-watch.pod @@ -0,0 +1,30 @@ +=head1 NAME + +lei-ls-watch - list active watches + +=head1 SYNOPSIS + +lei ls-watch + +=head1 DESCRIPTION + +List locations that lei is configured to watch. This command is +incomplete, mail-sync locations are implicitly watched. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-watch(1)>, L<lei-rm-watch(1)> diff --git a/Documentation/lei-mail-diff.pod b/Documentation/lei-mail-diff.pod new file mode 100644 index 00000000..96e49a8b --- /dev/null +++ b/Documentation/lei-mail-diff.pod @@ -0,0 +1,33 @@ +=head1 NAME + +lei-mail-diff - diff the contents of emails + +=head1 SYNOPSIS + + +lei mail-diff [OPTIONS] LOCATION + +lei mail-diff [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +This is a diagnostic command that's useful for finding deduplication +bugs. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-inspect(1)> diff --git a/Documentation/lei-mail-formats.pod b/Documentation/lei-mail-formats.pod new file mode 100644 index 00000000..618bada2 --- /dev/null +++ b/Documentation/lei-mail-formats.pod @@ -0,0 +1,138 @@ +=head1 NAME + +lei-mail-formats - description of mail formats supported by lei + +=head1 DESCRIPTION + +L<lei-q(1)> supports writing to several existing mail formats +for interoperability with existing mail user agents (MUA); +below is an overview of them to help users choose. + +=head1 Maildir + +The default output format when given a filesystem path, it supports +parallel read-write access. Performance is acceptable for smaller +directories, but degrades as mailboxes get larger. Speed and +scalability are limited by kernel and filesystem performance +due to the use of small files and large number of syscalls. + +See also: L<https://cr.yp.to/proto/maildir.html> and +L<https://wiki2.dovecot.org/MailboxFormat/Maildir> + +=head1 Mbox family + +The mbox family consists of several incompatible formats. +Locking for parallel access is supported, but may not be +compatible across tools. With compression (e.g. L<gzip(1)>), +they require the least amount of space while offering good +read-only performance. + +Keyword updates (C<Status:> and/or C<X-Status:> headers) +generally require rewriting the entire mbox. + +See also: +L<https://www.loc.gov/preservation/digital/formats/fdd/fdd000383.shtml>, +L<mbox(5)> + +=head2 mboxo + +The traditional BSD format. It quotes C<From > to C<E<gt>From >, +but lines already beginning with C<E<gt>From > do not get quoted, +thus automatic reversibility is not guaranteed. MUAs which favor +L</mboxcl> or L</mboxcl2> may convert these automatically to their +preferred format. + +Truncation is undetectable unless compressed with gzip or similar. + +=head2 mboxrd + +An evolution of L</mboxo>, but quotes C<From > lines prefixed +with any number of C<E<gt>> characters and is thus fully +reversible. + +This format is emitted by L<PublicInbox::WWW(3pm)> with gzip. +Since git 2.10, C<git am --patch-format=mboxrd> reads this +format. C<git log> and C<git format-patch --stdout> can also +generate this format with the C<--pretty=mboxrd> switch. + +As with uncompressed L</mboxo>, uncompressed mboxrd are vulnerable +to undetectable truncation. + +It gracefully degrades to being treated as L</mboxo> by MUAs +unaware of the format as excessive C<E<gt>From > quoting is +recognizable to humans. + +=head2 mboxcl + +L</mboxo> with a C<Content-Length:> header, C<From > lines +remain quoted to retain readability with L</mboxo> and L</mboxrd> MUAs. +However, it is easy to corrupt these files when using tools +which are not aware of C<Content-Length:> and write out updates +as L</mboxo>. + +L<mutt(1)> will convert L</mboxo> and L</mboxrd> to mboxcl upon opening. + +See also: L<https://www.jwz.org/doc/content-length.html> + +=head2 mboxcl2 + +Like L</mboxcl>, but without C<From > any quoting. It is wholly +incompatible with MUAs which only handle L</mboxo> and/or L</mboxrd>. +This is format is generated by L<mutt(1)> when writing to a new +mbox. + +=head1 MH + +Preliminary support for reads as of 2.0.0. Locking semantics differ +incompatibly amongst existing writers: Python and nmh appear +compatible with each other, while mutt appears racy and unsuitable +for parallel access due to rename(2) potentially clobbering the +C<.mh_sequences> file. More info about other clients is greatly +appreciated. + +Sequence numbers may be packed and reused by some writers, so lei +users may need to run L<lei-refresh-mail-sync(1)> if inotify|kevent +missed packing while L<lei-daemon(8)> wasn't running. + +lei is safe for reading mlmmj archives as MH since mlmmj neither +packs nor uses a .mh_sequences file to store state. + +=head1 MMDF + +Not yet supported, and it's unclear if current usage/support makes +it worth supporting. + +=head1 IMAP + +Depending on the IMAP server software and configuration, IMAP +servers may use any (or combination) of the aforementioned +formats or a non-standard database backend. Currently, lei +uses L<Mail::IMAPClient> which has acceptable performance +over low-latency links. Performance over high-latency links +is currently poor. + +=head1 eml + +A single raw message file. C<eml> is not an output format for lei, +but accepted by as an C<--input-format> (C<-F>) for read-only +commands such as L<lei-tag(1)> and L<lei-import(1)>. + +Since C<eml> is the suffix for the C<message/rfc822> MIME type +(according to the C<mime.types> file), lei will infer the type +based on the C<.eml> suffix if C<--input-format> is unspecified + +C<.patch>-suffixed files generated by L<git-format-patch(1)> +(without C<--stdout>) are C<eml> files with the addition of an +mbox C<From > header. L<lei(1)> removes C<From > lines to treat +them as C<eml> when reading these for compatibility with +C<git-am(1)> and similar tools. + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei(1)>, L<lei-q(1)>, L<lei-convert(1)>, L<lei-overview(7)> diff --git a/Documentation/lei-mail-sync-overview.pod b/Documentation/lei-mail-sync-overview.pod new file mode 100644 index 00000000..7ae7e887 --- /dev/null +++ b/Documentation/lei-mail-sync-overview.pod @@ -0,0 +1,53 @@ +=head1 NAME + +lei - an overview of lei mail synchronization + +=head1 DESCRIPTION + +L<lei(1)> provides several plumbing-level commands to synchronize +mail and keywords (flags) between lei/store and existing IMAP +and Maildir stores. Nothing documented in this manpage is required +for day-to-day use against externals. + +Mail and keyword synchronization is currently a clunky process. +Future work will be done to improve it and add IMAP IDLE support. + +=head1 TYPICAL WORKFLOW + + # import mail from a user's IMAP inbox and give it the "inbox" label: + lei import +L:inbox imaps://user@example.com/INBOX + + # dump "inbox" labeled files from the past week to a Maildir + lei q L:inbox rt:last.week.. -o /tmp/results + + # Open /tmp/results in your favorite mail agent. If inotify or kevent + # works, keyword changes (e.g. marking messages as `seen') are + # synchronized automatically. + + # If the inotify queue overflows, or if lei-daemon crashes, + # "lei index" will tell lei about keyword changes: + lei index /tmp/results + + # Optional: cleanup stale entries from mail_sync.sqlite3 + lei refresh-mail-sync /tmp/results + + # to export keyword changes back to IMAP + lei export-kw imaps://user@example.com/INBOX + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-import(1)>, L<lei-q(1)>, L<lei-index(1)>, +L<lei-refresh-mail-sync(1)>, L<lei-export-kw(1)> diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod new file mode 100644 index 00000000..e9a97d64 --- /dev/null +++ b/Documentation/lei-overview.pod @@ -0,0 +1,167 @@ +=head1 NAME + +lei - an overview of lei + +=head1 DESCRIPTION + +L<lei(1)> is a local email interface for public-inbox and personal mail. +This document provides some basic examples. + +=head1 LEI STORE + +lei has writable local storage based on L<public-inbox-v2-format(5)>. +Commands will automatically initialize the store behind the scenes if +needed, but you can call L<lei-init(1)> directly if you want to use a +store location other than the default C<$XDG_DATA_HOME/lei/store>. + +The L<lei-import(1)> command provides the primary interface for +importing messages into the local storage. In addition, other +commands, such as L<lei-q(1)> and L<lei-blob(1)>, use the local store +to memoize messages from remotes. + +=head2 EXAMPLES + +=over + +=item $ lei import mboxrd:t.mbox.gz + +Import the messages from a gzipped mboxrd into the local storage. + +=item $ lei blob 59ec517f9 + +Show message with the git blob OID of 59ec517f9. If a message with +that OID isn't found, check if the current git repository has the +blob, trying to reconstruct it from a message if needed. + +=item $ lei blob 59ec517f9 | lei tag -F eml +kw:flagged +L:next + +Set the "flagged" keyword and "next" label on the message with the +blob OID of 59ec517f9. + +=back + +=head1 EXTERNALS + +In addition to the above store, lei can make read-only queries to +"externals": inboxes and external indices. An external can be +registered by passing a URL or local path to L<lei-add-external(1)>. +For existing local paths, the external needs to be indexed with +L<public-inbox-index(1)> (in the case of a regular inbox) or +L<public-inbox-extindex(1)> (in the case of an external index). + +=head1 SYNCHRONIZATION + +lei currently has primitive mail synchronization abilities; +see L<lei-mail-sync-overview(7)> for more details. + +=head2 EXAMPLES + +=over + +=item $ lei add-external https://public-inbox.org/meta/ + +Add a remote external for public-inbox's inbox. + +=item $ lei add-external --mirror https://public-inbox.org/meta/ path + +Clone L<https://public-inbox.org/meta/> to C<path>, index it with +L<public-inbox-index(1)>, and add it as a local external. + +=back + +=head1 SEARCHING + +The L<lei-q(1)> command searches the local store and externals. The +search prefixes match those available via L<public-inbox-httpd(1)>. + +=head2 EXAMPLES + +=over + +=item $ lei q s:lei s:skeleton + +Search for messages whose subject includes "lei" and "skeleton". + +=item $ lei q -t s:lei s:skeleton + +Do the same, but also report unmatched messages that are in the same +thread as a matched message. + +=item $ lei q -t -o /tmp/mdir --mua=mutt s:lei s:skeleton + +Write results to a Maildir at "mdir". Mutt will be invoked +to open mfolder (C<mutt -f %f>) while results are being fetched +and written. + +=item $ lei q kw:flagged L:next + +Search for all flagged messages that also have a "next" label. + +=item $ lei p2q HEAD | lei q -tt -o /tmp/mdir + +Search for messages that have post-image git blob IDs that match those +of the current repository's HEAD commit, writing them to the Maildir +directory "mdir" and flagging the messages that were an exact match. + +=item $ git show -s HEAD | lei lcat + +Display a local message for the public-inbox link contained in a +commit message. + +=item $ lei q -f text m:MESSAGE-ID | lei rediff -U5 + +Feed a message containing a diff to L<lei-rediff(1)> to regenerate its +diff with five context lines. Unless C<--git-dir> is specified, this +requires the current working directory to be within the associated +code repository. + +=back + +=head1 PERFORMANCE NOTES + +L<Inline::C> is required on BSDs and can speed things up on Linux. + +lei runs as a background daemon to reduce startup costs and can +provide real-time L<kqueue(2)>/L<inotify(7)> Maildir monitoring. +L<IO::KQueue> (p5-IO-KQueue on FreeBSD) and L<Linux::Inotify2> +(liblinux-inotify2-perl and perl-Linux-Inotify2 in .deb and .rpm-based +distros, respectively) are recommended. + +L<Socket::MsgHdr> is optional (libsocket-msghdr-perl in Debian), +and further improves startup performance. Its effect is most felt +when using shell completion. + +=head1 BASH COMPLETION + +Preliminary Bash completion for lei is provided in +C<contrib/completion/>. Contributions adding support for other +shells, as well as improvements to the existing Bash completion, are +welcome. + +=head1 UPGRADING + +Since lei runs as a daemon, L<lei-daemon-kill(1)> is required to kill +the daemon so it can load new code. It will be restarted with the +next invocation of any lei command. + +=head1 CAVEATS + +IMAP and NNTP client performance is poor on high-latency connections. +It will hopefully be fixed in 2022. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-mail-sync-overview(7)> diff --git a/Documentation/lei-p2q.pod b/Documentation/lei-p2q.pod new file mode 100644 index 00000000..4bc5d25f --- /dev/null +++ b/Documentation/lei-p2q.pod @@ -0,0 +1,101 @@ +=head1 NAME + +lei-p2q - use a patch to generate a lei-q query + +=head1 SYNOPSIS + +lei p2q [OPTIONS] (FILE|COMMIT) + +lei p2q [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Given a patch, create a query that can be fed on stdin to L<lei-q(1)>. +This is useful for mapping the patch to associated messages of an +inbox. + +The patch can be provided on stdin or as a file. Alternatively, when +an argument is given that does not point to an existing file, it is +taken as a reference to a commit in the current git repository, and +L<git-format-patch(1)> is used to generate the patch. + +=head1 OPTIONS + +=over + +=item -w PREFIX[,PREFIX] + +=item --want=PREFIX[,PREFIX] + +Search prefixes to use. C<dfpost> (post-image git blob ID) and C<dfn> +(file names from the diff) are the most useful. Other available +values are C<dfa>, C<dfb>, C<dfctx>, C<dfhh>, and C<dfpre>. + +=for comment +TODO: Put a table of prefixes somewhere and reference that (at least +here and in lei-q)? + +Appending an integer to C<dfpost> or C<dfpre> indicates a minimum ID +length, and the generated query will be for that value up through the +default abbreviation length. For example, if the repository's +C<core.abbrev> is set to C<auto> and git calculates the default +abbreviation length as 7, C<dfpost6> will expand a post-image blob ID +of e7b4b32 (seven characters) into C<dfpost:e7b4b32 OR dfpost:e7b4b3>. + +This option may be given multiple times. + +Default: C<dfpost7> + +=item --stdin + +Read message from stdin. This is implicit if no arguments are given +and stdin is a pipe or regular file. + +=item --debug + +Dump output that shows the information collected for every prefix. +This information can be useful for seeing how a patch is processed, +but the format should not be considered stable. + +=item --uri + +URI escape output for interacting with HTTP(S) public-inbox instances. + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 EXAMPLES + + # to search for all threads which touch a given thread: + lei p2q $COMMIT_OID | lei q -t -o /tmp/results + + # to view results on a remote HTTP(S) public-inbox instance + $BROWSER https://example.com/pub-inbox/?q=$(lei p2q --uri $COMMIT_OID) + + # to view unapplied patches for a given $FILE from the past year: + echo \( rt:last.year.. AND dfn:$FILE \) AND NOT \( \ + $(git log -p --pretty=mboxrd --since=last.year $FILE | + lei p2q -F mboxrd ) + \) | lei q -o /tmp/unapplied + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)> diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod new file mode 100644 index 00000000..79156750 --- /dev/null +++ b/Documentation/lei-q.pod @@ -0,0 +1,354 @@ +=head1 NAME + +lei-q - search for messages matching terms + +=head1 SYNOPSIS + +lei q [OPTIONS] TERM [TERM...] + +lei q [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Search for messages across the lei/store and externals. + +=head1 OPTIONS + +=for comment +TODO: mention curl options? + +=over + +=item --stdin + +Read search terms from stdin. + +=item --no-save + +Do not save the search for L<lei-up(1)>. + +=item --output=MFOLDER + +=item -o MFOLDER + +=item --mfolder=MFOLDER + +Warning: this clobbers and overwrites the output destination unless +L</--augment> is specified. + +Destination for results (e.g., C</tmp/results-Maildir>, +C<imaps://user@mail.example.com/INBOX.test>, or +C<mboxcl2:/tmp/results-mboxcl2>). The prefix may be a supported protocol: +C<imap://> or C<imaps://>. URLs requiring +authentication use L<git-credential(1)> to +fill in the username and password. + +A prefix can specify the format of the output: C<maildir>, +C<mboxrd>, C<mboxcl2>, C<mboxcl>, C<mboxo>. For a description of +mail formats, see L<lei-mail-formats(5)>. + +C<v2:/path/to/inbox> may be used to create a new inbox of +L<public-inbox-v2-format(5)>. The new inbox will not be configured +in the L<public-inbox-config(5)> file. + +C<maildir> is the default for an existing directory or non-existing path. + +Default: C<-> (stdout) + +=item --format=FORMAT + +=item -f FORMAT + +Format of results to stdout. This option exists as a convenient +way to specify the format for the default stdout destination. +C<reply>, C<text>, C<json>, C<jsonl>, or C<concatjson> are all supported, +as are the various mbox variants described in L</--output>. + +When a format isn't specified, it's chosen based on the +L</--output> destination or prefix. C<json> is used for the +default destination (stdout). + +Using a C<format:> prefix with the C<--output> destination is +preferred when not writing to stdout. + +=item --no-color + +Disable color (for C<-f reply> and C<-f text>). + +=item --pretty + +Pretty-print C<json> or C<concatjson> output. If stdout is opened to +a tty and used as the C<--output> destination, C<--pretty> is enabled +by default. + +=item --mua=COMMAND + +A command to run on C<--output> Maildir or mbox (e.g., C<mutt -f %f>). +For a subset of MUAs known to accept a mailbox via C<-f>, COMMAND can +be abbreviated to the name of the program: C<mutt>, C<mailx>, C<mail>, +or C<neomutt>. + +=item --alert=COMMAND[,COMMAND...] + +Run C<COMMAND> after writing to output. C<:WINCH> indicates to send +C<SIGWINCH> to the C<--mua> process. C<:bell> indicates to print a +bell code. Any other value is interpreted as a command to execute as +is. + +This option may be given multiple times. + +Default: C<:WINCH,:bell> when C<--mua> is specified and C<--output> +doesn't point to stdout, nothing otherwise. + +=item --augment + +=item -a + +Augment output destination instead of clobbering it. + +=item --no-import-before + +Do not import messages before writing to an existing output destination. +Be certain you do not need existing data in your output before using +this, it permanently erases data unless C<--augment> is used. + +=item --threads + +=item -t + +Return all messages in the same thread as the actual match(es). + +Using this twice (C<-tt>) sets the C<flagged> (AKA "important") +on messages which were actual matches. This is useful to distinguish +messages which were direct hits from messages which were merely part +of the same thread. + +TODO: Warning: this flag may become persistent and saved in +lei/store unless an MUA unflags it! (Behavior undecided) + +Caveat: C<-tt> only works on locally-indexed messages at the +moment, and not on remote (HTTP(S)) endpoints. + +=item --thread-id=MSGID + +=item -T MSGID + +Only search messages in the same thread as the given Message-ID. + +For HTTP(S) externals, this only works on instances running +public-inbox 2.0+ (UNRELEASED). + +=item --jobs=QUERY_WORKERS[,WRITE_WORKERS] + +=item --jobs=,WRITE_WORKERS + +=item -j QUERY_WORKERS[,WRITE_WORKERS] + +=item -j ,WRITE_WORKERS + +Set the number of query and write worker processes for parallelism. + +C<QUERY_WORKERS> defaults to the number of CPUs available, but 4 per +remote (HTTP/HTTPS) host. + +C<WRITE_WORKERS> defaults to 75% of the number of CPUs available for +Maildir and mbox* destinations, but 4 per IMAP/IMAPS host. + +Omitting C<QUERY_WORKERS> but leaving the comma (C<,>) allows +one to only set C<WRITE_WORKERS> + +=item --dedupe=STRATEGY + +=item -d STRATEGY + +Strategy for deduplicating messages: C<content>, C<oid>, C<mid>, or +C<none>. + +Default: C<content> + +=for comment +TODO: Provide description of strategies? + +=item --[no-]remote + +Whether to include results requiring network access. When local +externals are configured, C<--remote> must be explicitly passed to +enable reporting of results from remote externals. + +=item --no-local + +Limit operations to those requiring network access. + +=item --no-external + +Don't include results from externals. + +=item --include=LOCATION + +=item -I LOCATION + +Include specified external in search. This option may be given +multiple times. + +=item --exclude=LOCATION + +Exclude specified external from search. This option may be given +multiple times. + +=item --only=LOCATION + +=item -O LOCATION + +Use only the specified external for search. This option may be given +multiple times, in which case the search uses only the specified set. + +=item --globoff + +=item -g + +Do not match locations using C<*?> wildcards and C<[]> ranges. This +option applies to C<--include>, C<--exclude>, and C<--only>. + +=item --no-import-remote + +Disable the default behavior of memoizing remote messages into the +local store. + +=item --lock=METHOD + +L<mbox(5)> locking method(s) to use: C<dotlock>, C<fcntl>, C<flock> or +C<none>. + +Default: fcntl,dotlock + +=item --limit=NUMBER + +=item -NUMBER + +=item -n NUMBER + +Fuzzy-limit the number of matches per local external and lei/store. +Messages added by the L<--threads> switch do not count towards this +limit, and there is no limit on remote externals. + +Default: 10000 + +=item --offset=NUMBER + +Shift start of search results. + +Default: 0 + +=item --reverse + +=item -r + +Reverse the results. Note that this applies before C<--limit>. + +=item --sort=KEY + +=item -s KEY + +Order the results by KEY. Valid keys are C<received>, C<relevance>, +and C<docid>. + +Default: C<received> + +=item --verbose + +=item -v + +Provide more feedback on stderr. + +=item --quiet + +=item -q + +Suppress feedback messages. + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=item --proxy=PROTOCOL://HOST[:PORT] + +=back + +=head1 SEARCH TERMS + +C<lei q> supports the same search prefixes used by HTTP(S) public-inbox +instances: + +=for comment +AUTO-GENERATED-SEARCH-TERMS-BEGIN + + s: match within Subject e.g. s:"a quick brown fox" + d: match date-time range, git "approxidate" formats supported + Open-ended ranges such as `d:last.week..' and + `d:..2.days.ago' are supported + b: match within message body, including text attachments + nq: match non-quoted text within message body + q: match quoted text within message body + n: match filename of attachment(s) + t: match within the To header + c: match within the Cc header + f: match within the From header + a: match within the To, Cc, and From headers + tc: match within the To and Cc headers + l: match contents of the List-Id header + bs: match within the Subject and body + dfn: match filename from diff + dfa: match diff removed (-) lines + dfb: match diff added (+) lines + dfhh: match diff hunk header context (usually a function name) + dfctx: match diff context lines + dfpre: match pre-image git blob ID + dfpost: match post-image git blob ID + dfblob: match either pre or post-image git blob ID + patchid: match `git patch-id --stable' output + rt: match received time, like `d:' if sender's clock was correct + +=for comment +AUTO-GENERATED-SEARCH-TERMS-END + +Additional search prefixes which only affect the local lei/store: + + L: match the given label + kw: match the given keywords + +See L<lei-tag(1)> for more info on labels and keywords. + +Most prefixes are probabilistic, meaning they support stemming +and wildcards (C<*>). Ranges (such as C<d:>) and boolean prefixes +do not support stemming or wildcards. +The upstream Xapian query parser documentation fully explains +the query syntax: L<https://xapian.org/docs/queryparser.html> + +=head1 TIPS + +C<-f reply> is intended to aid in turning a cover letter +into a reply (since using C<git format-patch --in-reply-to=...> +is tedious). Results (including "From " lines) should be edited +and trimmed in your favorite C<$EDITOR> before sending. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-add-external(1)>, L<lei-lcat(1)>, L<lei-up(1)>, +L<Xapian::QueryParser Syntax|https://xapian.org/docs/queryparser.html> diff --git a/Documentation/lei-rediff.pod b/Documentation/lei-rediff.pod new file mode 100644 index 00000000..f18548d3 --- /dev/null +++ b/Documentation/lei-rediff.pod @@ -0,0 +1,126 @@ +=head1 NAME + +lei-rediff - regenerate a diff with different options + +=head1 SYNOPSIS + +lei rediff [OPTIONS] LOCATION [LOCATION...] + +lei rediff [OPTIONS] (--stdin|-) + +=head1 DESCRIPTION + +Read a message from C<LOCATION> or stdin and regenerate its diff with +the specified L<git-diff(1)> options. This is useful if you want to +change the display of the original patch (e.g., increasing context, +coloring moved lines differently, or using an external diff viewer). + +It relies on the contents of the .git directory of your current +project working tree. In other words, it works anywhere +L<git-am(1)> works. Otherwise, C<--git-dir=> may be specified +any number of times to add repositories to build blob data from. + +=head1 OPTIONS + +In addition to many L<git-diff(1)> options (e.g. C<-W>, C<-w>, +C<-U $LINES>) the following options are supported: + +=over + +=item --stdin + +Read message from stdin. This is implicit if no arguments are given +and stdin is a pipe or regular file. + +For users of text editors and pagers capable of piping its +buffer to arbitrary commands, it is useful to pipe a patch email +to C<lei rediff> before piping it to L<git-am(1)>. The output +of C<lei rediff> is compatible with C<git am> if its input was a +patch email. + +=item --drq[=COUNT] + +De-Re-Quote. De-quotes the input and re-quotes (the output). +Removes COUNT levels of C<E<gt> > email reply prefixes and +re-adds them upon regenerating the diff. + +This switch is intended as a convenience for running inside a +pipe-capable text editor when writing replies to a patch email. +Note: this may over-add C<E<gt> > prefixes if some input lines +are missing C<E<gt> > prefixes. + +COUNT is 1 if unspecified; in other words, C<--drq=1> and +C<--drq> are equivalent. + +It implies L</--quiet> unless L</--verbose> is specified +since text editors tend to combine stderr with stdout. + +=item --dequote-only[=COUNT] + +Like L</--drq>, but does not re-add quote prefixes to the output. + +This can be useful for feeding a hunk to L<git-apply(1)> +or L<patch(1)> while writing a reply or further processing +by another diff viewer. + +Unlike L</--drq>, it does NOT imply L</--quiet>. + +=item --git-dir=DIR + +Specify an additional .git/ directory to scan. This option may be +given multiple times. + +Default: the output of C<git rev-parse --git-dir> + +=item --no-cwd + +Do not look in the git repository of the current working directory. + +=item -q + +=item --quiet + +Suppress progress output. + +=item -v + +=item --verbose + +Provide more feedback on stderr. + +=back + +The options below, described in L<lei-q(1)>, are also supported. + +=over + +=item --[no-]remote + +=item --no-local + +=item --no-external + +=item --no-import-remote + +=item --torsocks=auto|no|yes, --no-torsocks + +=item --proxy=PROTOCOL://HOST[:PORT] + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)>, L<lei-blob(1)>, L<lei-p2q(1)> diff --git a/Documentation/lei-refresh-mail-sync.pod b/Documentation/lei-refresh-mail-sync.pod new file mode 100644 index 00000000..65150ae3 --- /dev/null +++ b/Documentation/lei-refresh-mail-sync.pod @@ -0,0 +1,57 @@ +=head1 NAME + +lei-refresh-mail-sync - refresh sync info with Maildir, IMAP + +=head1 SYNOPSIS + +lei refresh-mail-sync --all[=<remote|local>] + +lei refresh-mail-sync MFOLDER [MFOLDER...] + +=head1 DESCRIPTION + +C<lei refresh-mail-sync> is intended to keep old messages +indexed with L<lei-index(1)> retrievable if Maildir flags change +a filename. It will prune invalid entries for messages which no +longer exist in a Maildir. + +It is also useful for ensuring L<lei-export-kw(1)> can propagate +keyword (flag) changes to Maildirs and IMAP folders. + +It only needs read-only access to Maildirs and IMAP folders +and will not attempt to write to them at all. + +=head1 OPTIONS + +=over + +=item --all + +Refresh all local Maildirs and remote IMAP folders + +=item --all=local + +Refresh all local Maildirs + +=item --all=remote + +Refresh all remote IMAP folders + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-index(1)>, L<lei-export-kw(1)>, L<lei-ls-mail-sync(1)> diff --git a/Documentation/lei-reindex.pod b/Documentation/lei-reindex.pod new file mode 100644 index 00000000..3a5861c4 --- /dev/null +++ b/Documentation/lei-reindex.pod @@ -0,0 +1,47 @@ +=head1 NAME + +lei-reindex - reindex messages already in lei/store + +=head1 SYNOPSIS + +lei reindex [OPTIONS] + +=head1 DESCRIPTION + +Forces a re-index of all messages previously-indexed by L<lei-import(1)> +or L<lei-index(1)>. This can be used for in-place upgrades and bugfixes +while other processes are querying the store. Keep in mind this roughly +doubles the size of the already-large Xapian database. + +It does not re-index messages in externals, using the C<--reindex> +switch of L<public-inbox-index(1)> or L<public-inbox-extindex(1)> is +needed for that. + +=head1 OPTIONS + +=over + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-index(1)>, L<lei-import(1)> diff --git a/Documentation/lei-rm-watch.pod b/Documentation/lei-rm-watch.pod new file mode 100644 index 00000000..711d7dc4 --- /dev/null +++ b/Documentation/lei-rm-watch.pod @@ -0,0 +1,30 @@ +=head1 NAME + +lei-rm-watch - stop watching locations + +=head1 SYNOPSIS + +lei rm-watch [OPTIONS] LOCATION [LOCATION...] + +=head1 DESCRIPTION + +Tell lei to stop watching C<LOCATION> for new messages and flag +changes. Currently only Maildir locations are supported. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + + +=head1 SEE ALSO + +L<lei-add-watch(1)>, L<lei-ls-watch(1)> diff --git a/Documentation/lei-rm.pod b/Documentation/lei-rm.pod new file mode 100644 index 00000000..67bfb551 --- /dev/null +++ b/Documentation/lei-rm.pod @@ -0,0 +1,75 @@ +=head1 NAME + +lei-rm - unindex a message in lei/store + +=head1 SYNOPSIS + +lei rm [OPTIONS] (-|--stdin) + +lei rm [OPTIONS] LOCATION + +=head1 DESCRIPTION + +Removes message(s) and associated private metadata from lei/store +indices. It does not affect messages stored in externals, so it's +still possible to get "removed" messages from externals in L<lei-q> +search results. + +This does not remove the message from underlying git storage nor +does it remove messages from Maildir/mbox/IMAP/etc. sources. + +=head1 OPTIONS + +=over + +=item - + +=item --stdin + +Read input from standard input. This is the default if standard +input is a pipe or regular file and there are no arguments on +the command-line. + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +Message input format: C<eml>, C<mboxrd>, C<mboxcl2>, C<mboxcl>, or C<mboxo> +when reading from stdin or using one of the mbox variants. + +Not necessary when using an IMAP URL, NNTP URL or Maildir. + +Default: C<eml> when reading from stdin or if the file suffix +ends in C<.patch> or C<.eml>. + +=item --lock=METHOD + +L<mbox(5)> locking method(s) to use: C<dotlock>, C<fcntl>, C<flock> or +C<none>. + +Default: fcntl,dotlock + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-store-format(5)> diff --git a/Documentation/lei-security.pod b/Documentation/lei-security.pod new file mode 100644 index 00000000..e54cae90 --- /dev/null +++ b/Documentation/lei-security.pod @@ -0,0 +1,151 @@ +=head1 NAME + +lei - security information + +=head1 SYNOPSIS + +L<lei(1)> is intended for use with both publicly archived +and "private" mail in personal mailboxes. This document is +intended to give an overview of security implications and +lower^Wmanage user expectations. + +=head1 DESCRIPTION + +lei expects to be run as a regular user on a Unix-like system. +It expects a case-sensitive filesystem with standard Unix +permissions support. + +It does not use POSIX ACLs, extended attributes, nor any other +security-related functions which require non-standard Perl modules. + +There is preliminary support for "virtual users", but it is +incomplete and undocumented. + +=head1 INTERNAL FILES + +lei runs with a umask of 077 to prevent other users on the +system from accessing each other's mail. + +The git storage and Xapian databases are located at +C<$XDG_DATA_HOME/lei/store> (typically C<~/.local/share/lei/store>). +Any personal mail imported will reside here, so this should +be on an encrypted filesystem or block device. + +C<$XDG_RUNTIME_DIR/lei> (typically C</run/user/$UID/lei> or +C</tmp/lei-$UID>) contain the socket used to access the lei +daemon. It must only be accessible to the owner (mode 0700). + +C<$XDG_CACHE_HOME/lei> (typically C<~/.cache/lei>) will +contain IMAP and Maildir folder names which could leak sensitive +information as well as git repository names. + +C<$XDG_DATA_HOME/lei/saved-searches> (typically +C<~/.local/share/lei/saved-searches>) will contain aforementioned +folder names as well as (removable) search history. + +The configuration for lei resides at C<$XDG_CONFIG_HOME/lei/config> +(typically C<~/.config/lei/config>). It may contain sensitive pathnames +and hostnames in the config if a user chooses to configure them. + +lei itself will never write credentials to the +filesystem. However, L<git-credential(1)> may be +configured to do so. lei will only read C<~/.netrc> if +C<--netrc> is used (and it will never write to C<~/.netrc>). + +C<$XDG_CACHE_HOME/public-inbox> (typically C<~/.cache/public-inbox>) +can contain data and L<Inline::C>-built modules which can be +shared with public-facing L<public-inbox-daemon(8)> instances; +so no private data should be in "public-inbox" paths. + +=head1 EXTERNAL FILES + +Locations set by L<lei-add-external(1)> can be shared with +public-facing L<public-inbox-daemon(8)> processes. They may +reside on shared storage and may be made world-readable to +other users on the local system. + +=head1 CORE DUMPS + +In case any process crashes, a core dump may contain passwords or +contents of sensitive messages. Please report these so they can be +fixed (see L</CONTACT>). + +=head1 NETWORK ACCESS + +lei currently uses the L<curl(1)> and L<git(1)> executables in +C<$PATH> for HTTP and HTTPS network access. Interactive +authentication for HTTP and HTTPS is not yet supported since all +currently supported HTTP/HTTPS sources are L<PublicInbox::WWW> +instances. + +The L<Mail::IMAPClient> library is used for IMAP and IMAPS. +L<Net::NNTP> (standard library) is used for NNTP and NNTPS. + +L<Mail::IMAPClient> and L<Net::NNTP> will use L<IO::Socket::SSL> +for TLS if available. In turn, L<IO::Socket::SSL> uses the +widely installed OpenSSL library. + +STARTTLS will be attempted if advertised by the server +unless IMAPS or NNTPS are used. C<-c imap.starttls=0> +and C<-c nntp.startls=0> may be used to disable STARTTLS. + +L<IO::Socket::Socks> will be used if C<-c imap.proxy> or +C<-c nntp.proxy> point to a C<socks5h://$HOST:$PORT> +address (common for Tor). + +The C<--netrc> switch may be passed to curl and used for +NNTP/IMAP access (via L<Net::Netrc>). + +=head1 CREDENTIAL DATA + +lei uses L<git-credential(1)> to prompt users for IMAP and NNTP +usernames and passwords. These passwords are not encrypted in +memory and get transferred across processes via anonymous UNIX +sockets and pipes. They may be exposed via syscall tracing +tools (e.g. L<strace(1)>), kernel and hardware bugs/attacks. + +While credentials are not written to the filesystem by default, +it is possible for them to end up on disk if processes are +swapped out. Use of an encrypted swap partition is recommended. + +=head1 AUTHENTICATION METHODS + +LOGIN (username + password) is known to work over IMAP(S), +as does AUTH=ANONYMOUS (which is used by L<public-inbox-imapd(1)> +as part of our test suite). AUTHINFO may work for NNTP, but +is untested. Testers will be needed for other authentication +methods. + +=head1 DENIAL-OF-SERVICE VECTORS + +lei uses the same MIME parsing library as L<public-inbox-mda(1)> +with limits header sizes, parts, nesting and boundary limits +similar to those found in SpamAssassin and postfix. + +Email address parsing is handled by L<Email::Address::XS> if +available, but may fall back to regular expressions which favor +speed and predictable execution times over correctness. + +=head1 ENCRYPTED EMAILS + +Not yet supported, but it should eventually be possible to +configure decryption and indexing of encrypted messages and +attachments. When supported, decrypted terms will be stored +in Xapian DBs under C<$XDG_DATA_HOME/lei/store>. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-overview(7)>, L<lei(1)> diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod new file mode 100644 index 00000000..d4bb42d5 --- /dev/null +++ b/Documentation/lei-store-format.pod @@ -0,0 +1,98 @@ +% public-inbox developer manual + +=head1 NAME + +lei-store-format - lei/store format description + +=head1 DESCRIPTION + +C<lei/store> is a hybrid store based on L<public-inbox-extindex-format(5)> +("extindex") combined with L<public-inbox-v2-format(5)> ("v2") for blob +storage. While v2 is ideal for archiving a single public mailing list; +it was never intended for personal mail nor storing multiple +blobs of the "same" message. + +As with extindex, it can index disparate C<List-Id> headers +belonging to the "same" message with different git blob OIDs. +Unlike v2 and extindex, C<Message-ID> headers are NOT required; +allowing unsent draft messages to be stored and indexed. + +=head1 DIRECTORY LAYOUT + +Blob storage exists in the form of v2-style epochs. These epochs +are under the C<local/> directory (instead of C<git/>) to +prevent them from being accidentally treated as a v2 inbox. + +=head2 INDEX OVERVIEW AND DEFINITIONS + + $EPOCH - Integer starting with 0 based on time + $SCHEMA_VERSION - DB schema version (for Xapian) + $SHARD - Integer starting with 0 based on parallelism + + ~/.local/share/lei/store + - local/$EPOCH.git # normal bare git repositories + - mail_sync.sqlite3 # sync state IMAP, Maildir, NNTP + +Additionally, the following share the same roles they do in extindex: + + - ei.lock # lock file to protect global state + - ALL.git # empty, alternates for local/*.git + - ei$SCHEMA_VERSION/$SHARD # per-shard Xapian DB + - ei$SCHEMA_VERSION/over.sqlite3 # overview DB for WWW, IMAP + - ei$SCHEMA_VERSION/misc # misc Xapian DB + +=head2 XREF3 DEDUPLICATION + +Index deduplication follows extindex, see +L<public-inbox-extindex-format(5)/XREF3 DEDUPLICATION> for +more information. + +=head2 BLOB DEDUPLICATION + +The contents of C<local/*.git> repos is deduplicated by git blob +object IDs (currently SHA-1). This allows multiple copies of +cross-posted and personally Cc-ed messages to be stored with +different C<Received:>, C<X-Spam-Status:> and similar headers to +allow troubleshooting. + +=head2 VOLATILE METADATA + +Keywords and label information (as described in RFC 8621 for JMAP) +is stored in existing Xapian shards (C<ei$SCHEMA_VERSION/$SHARD>). +It is possible to search for messages matching labels and +keywords using C<L:> and C<kw:>, respectively. As with all data +stored in Xapian indices, volatile metadata is associated with +the Xapian document, thus it is shared across different blobs of +the "same" message. + +=head2 mail_sync.sqlite3 + +This SQLite database is maintained for bidirectional mapping of +git blobs to IMAP UIDs, Maildir file names, and NNTP article numbers. + +It is also used for retrieving messages from Maildirs indexed by +L<lei-index(1)>. + +=head1 IPC + +L<lei-daemon(8)> communicates with the C<lei/store> process using +L<unix(7)> C<SOCK_SEQPACKET> sockets. + +=head1 CAVEATS + +Reindexing and synchronization is not yet supported. + +=head1 THANKS + +Thanks to the Linux Foundation for sponsoring the development +and testing. + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-v2-format(5)>, L<public-inbox-extindex(5)> diff --git a/Documentation/lei-tag.pod b/Documentation/lei-tag.pod new file mode 100644 index 00000000..8cb9e736 --- /dev/null +++ b/Documentation/lei-tag.pod @@ -0,0 +1,81 @@ +=head1 NAME + +lei-tag - set/unset metadata on messages + +=head1 SYNOPSIS + +lei tag [OPTIONS] FILE [FILE...] METADATA [METADATA...] + +lei tag [OPTIONS] (-|--stdin) METADATA [METADATA...] + +=head1 DESCRIPTION + +Set or unset volatile metadata on messages. In JMAP terms, "volatile +metadata" includes "mailboxes" (analogous to a folder or label) and a +restricted set of "keywords". This supported keywords are the +combination of system keywords (seen, answered, flagged, and draft), +which map to Maildir flags and mbox Status/X-Status headers, as well +as reserved keywords (forwarded, phishing, junk, and notjunk). + +To add a label or keyword, prefix it with "+L:" and "+kw:", +respectively. To remove a label or keyword, use "-L:" or "-kw:". For +example, "+kw:flagged" would set the "flagged" keyword for the +specified messages, and "-L:INBOX" would remove the "INBOX" label. + +=head1 OPTIONS + +=over + +=item -F MAIL_FORMAT + +=item --in-format=MAIL_FORMAT + +Message input format: C<eml>, C<mboxrd>, C<mboxcl2>, C<mboxcl>, or +C<mboxo>. + +Default: C<eml> + +=item -q + +=item --quiet + +Suppress feedback messages. + +=back + +=head1 LABELS + +Labels are user-defined values analogous to IMAP/JMAP mailbox +names. They must only contain lowercase characters, digits, and +a limited amount of punctuation (e.g. C<.>, C<->, C<@>). + +Messages may have multiple labels. + +=head1 KEYWORDS + +Keywords are "flags" in Maildir and IMAP terminology. +Common keywords include: C<seen>, C<answered>, C<flagged>, and +C<draft>, though C<forwarded>, C<phishing>, C<junk>, and C<notjunk> +are also supported. + +When writing to various mboxes, the common keywords will be +mapped to the C<Status> and C<X-Status> headers. + +Messages may have multiple keywords. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-add-external(1)> diff --git a/Documentation/lei-up.pod b/Documentation/lei-up.pod new file mode 100644 index 00000000..8c426942 --- /dev/null +++ b/Documentation/lei-up.pod @@ -0,0 +1,92 @@ +=head1 NAME + +lei-up - update a saved search + +=head1 SYNOPSIS + +lei up [OPTIONS] OUTPUT + +lei up [OPTIONS] --all[=<local|remote>] + +=head1 DESCRIPTION + +Update the saved search at C<OUTPUT> or all saved searches. + +=head1 OPTIONS + +=over + +=item --all[=<local|remote>] + +C<--all> updates all saved searches (listed in L<lei-ls-search(1)>). +C<--all=local> only updates local mailboxes, C<--all=remote> only +updates remote mailboxes (currently C<imap://> and C<imaps://>). + +=item --remote-fudge-time=INTERVAL + +Look for mail older than the time of the last successful query. +Using a small interval will reduce bandwidth use. A larger +interval reduces the likelihood of missing a result due to MTA +delays or downtime. + +The time(s) of the last successful queries are the C<lastresult> +values visible from L<lei-edit-search(1)>. + +Date formats understood by L<git-rev-parse(1)> may be used, +e.g., C<1.hour> or C<3.days>. + +Default: 2.days + +=item --no-external + +=item --no-local + +=item --no-remote + +These disable the use of all externals, local externals, or +remote externals respectively. They are useful during +temporary network or mount-point outages. + +Unlike C<lei q>, these switches override the original C<lei q --only> +options saved as C<lei.q.only>. + +The combination C<--all=remote --no-remote> is supported for +offline use in case a user is updating an IMAP folder on localhost. + +=item --exclude=LOCATION + +As with L<lei-q(1)>, but may also exclude externals originally +specified via C<lei q --only>. + +=item --lock=METHOD + +=item --alert=CMD + +=item --mua=CMD + +=item --jobs QUERY_WORKERS[,WRITE_WORKERS] + +C<--lock>, C<--alert>, C<--mua>, and C<--jobs> are all supported and +documented in L<lei-q(1)>. + +C<--mua> is incompatible with C<--all>. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> +and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-q(1)>, L<lei-ls-search(1)>, L<lei-edit-search(1)>, +L<lei-forget-search(1)> diff --git a/Documentation/lei.pod b/Documentation/lei.pod new file mode 100644 index 00000000..2b10f490 --- /dev/null +++ b/Documentation/lei.pod @@ -0,0 +1,154 @@ +=head1 NAME + +lei - local email interface + +=head1 SYNOPSIS + +lei [OPTIONS] COMMAND + +=head1 DESCRIPTION + +lei is a command-line tool for importing and searching email, +regardless of whether it is from a personal mailbox or a public-inbox. +lei supports a local, writable store built on top of +L<public-inbox-v2-format(5)> and L<public-inbox-extindex(1)>. +L<lei-q(1)> provides an interface for querying messages across the lei +store and read-only local and remote "externals" (inboxes and external +indices). + +Warning: lei is still in its early stages and may destroy mail. +Be sure to have backups of destinations lei writes to. + +Available in public-inbox 1.7.0+. + +=head1 OPTIONS + +=over + +=item -c NAME=VALUE + +Override configuration C<NAME> to C<VALUE>. + +=item -C DIR + +Change current working directory to the specified directory before +running the command. This option can be given before or after +C<COMMAND> and is accepted by all lei subcommands except +L<lei-daemon-kill(1)>. + +=back + +=head1 COMMANDS + +Subcommands for initializing and managing local, writable storage: + +=over + +=item * L<lei-init(1)> + +=item * L<lei-import(1)> + +=item * L<lei-tag(1)> + +=back + +The following subcommands can be used to manage and inspect external +locations: + +=over + +=item * L<lei-add-external(1)> + +=item * L<lei-forget-external(1)> + +=item * L<lei-ls-external(1)> + +=back + +Subcommands related to searching and inspecting messages from the lei +store and configured externals are + +=over + +=item * L<lei-blob(1)> + +=item * L<lei-config(1)> + +=item * L<lei-edit-search(1)> + +=item * L<lei-forget-search(1)> + +=item * L<lei-lcat(1)> + +=item * L<lei-ls-search(1)> + +=item * L<lei-p2q(1)> + +=item * L<lei-q(1)> + +=item * L<lei-rediff(1)> + +=item * L<lei-up(1)> + +=back + +Other subcommands include + +=over + +=item * L<lei-add-watch(1)> + +=item * L<lei-config(1)> + +=item * L<lei-convert(1)> + +=item * L<lei-daemon-kill(1)> + +=item * L<lei-daemon-pid(1)> + +=item * L<lei-forget-mail-sync(1)> + +=item * L<lei-mail-diff(1)> + +=item * L<lei-inspect(1)> + +=item * L<lei-ls-label(1)> + +=item * L<lei-ls-mail-source(1)> + +=item * L<lei-ls-mail-sync(1)> + +=item * L<lei-ls-watch(1)> + +=item * L<lei-rm-watch(1)> + +=back + +=head1 FILES + +By default, storage is located at C<$XDG_DATA_HOME/lei/store>. The +configuration for lei resides at C<$XDG_CONFIG_HOME/lei/config>. + +=head1 ERRORS + +Errors and dianostics for interactive commands are reported to +stderr. Some errors for background tasks are emitted via +L<syslog(3)> as L<lei-daemon(8)> for the top-level daemon, +and C<lei/store> for the L<lei-store-format(5)> worker. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<lei-overview(7)>, L<lei-daemon(8)> diff --git a/Documentation/lei_design_notes.txt b/Documentation/lei_design_notes.txt new file mode 100644 index 00000000..b5180374 --- /dev/null +++ b/Documentation/lei_design_notes.txt @@ -0,0 +1,32 @@ +lei design notes +---------------- + +Daemon architecture +------------------- + +The use of a persistent daemon works around slow startup time of +Perl. This is especially important for built-in support for +shell completion. It attempts to support inotify and EVFILT_VNODE +background monitoring of Maildir keyword changes. + +If lei were reimplemented in a language with faster startup +time, the daemon architecture would likely remain since it also +lets us easily decouple the local storage from slow IMAP/NNTP +backends and allow us to serialize writes to git-fast-import, +SQLite, and Xapian across multiple processes. + +The coupling of IMAP and NNTP network latency to local storage +is a current weakness of public-inbox-watch. Therefore, -watch +will likely adopt the daemon architecture of lei in the future. + +Read/write vs read-only storage +------------------------------- + +public-inboxes are intended to be written and read by different +Unix users. Commonly, a single Unix user or group will write to +a public-inbox, but the inbox will be served by a user with +read-only permissions (e.g. "www-data" or "nobody"). + +lei/store is intended to be read and written by a single user, +thus we can rely on the Write-Ahead-Log journal of SQLite to +improve performance: <https://sqlite.org/wal.html> diff --git a/Documentation/marketing.txt b/Documentation/marketing.txt index 385e5172..8e4aa3b5 100644 --- a/Documentation/marketing.txt +++ b/Documentation/marketing.txt @@ -3,7 +3,9 @@ marketing guide for public-inbox TL; DR: Don't market this. If you must: don't be pushy and annoying about it. Slow down. -Please no superlatives, hype or BS. +Please no superlatives, hype or BS. Please keep all marketing +materials text-only to be accessible to those on slow networks +and ancient hardware. It's online and public, so it already markets itself. Being informative is not a bad thing, being insistent is. @@ -25,3 +27,12 @@ than the adoption of any software. Every time somebody recognizes and rejects various forms of lock-in and centralization is already a victory for us. + +Please keep in mind: + +* Perl 5 is not a well-liked language +* AGPL is not a well-liked license +* maintainer is a shy introvert + +Be sure to mention these things in any marketing materials +to avoid wasting time of people who hate Perl and/or AGPL. diff --git a/Documentation/mknews.perl b/Documentation/mknews.perl index d803ca77..001ad310 100755 --- a/Documentation/mknews.perl +++ b/Documentation/mknews.perl @@ -1,12 +1,13 @@ #!/usr/bin/perl -w -# Copyright (C) 2019-2020 all contributors <meta@public-inbox.org> +# Copyright (C) all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> # Generates NEWS, NEWS.atom, and NEWS.html files using release emails # this uses unstable internal APIs of public-inbox, and this script # needs to be updated if they change. use strict; -use PublicInbox::MIME; +use PublicInbox::Eml; use PublicInbox::View; +use PublicInbox::Hval qw(fmt_ts); use PublicInbox::MsgTime qw(msg_datestamp); use PublicInbox::MID qw(mids mid_escape); END { $INC{'Plack/Util.pm'} and warn "$0 should not have loaded Plack::Util\n" } @@ -37,14 +38,19 @@ if ($dst eq 'NEWS') { my $ibx = My::MockObject->new( description => 'public-inbox releases', over => undef, - search => 1, # for WwwStream:_html_top + search => 1, # for WwwStream::html_top base_url => "$base_url/", ); $ibx->{-primary_address} = $addr; my $ctx = { - -inbox => $ibx, + ibx => $ibx, -upfx => "$base_url/", -hr => 1, + zfh => $out, + env => { + HTTP_HOST => 'public-inbox.org', + 'psgi.url_scheme' => 'https', + }, }; if ($dst eq 'NEWS.html') { html_start($out, $ctx); @@ -76,7 +82,7 @@ sub release2mime { my ($release, $mtime_ref) = @_; my $f = "$dir/$release.eml"; open(my $fh, '<', $f) or die "open($f): $!"; - my $mime = PublicInbox::MIME->new(do { local $/; <$fh> }); + my $mime = PublicInbox::Eml->new(\(do { local $/; <$fh> })); # Documentation/include.mk relies on mtimes of each .eml file # to trigger rebuild, so make sure we sync the mtime to the Date: # header in the .eml @@ -91,7 +97,7 @@ sub mime2txt { my $title = $mime->header('Subject'); $title =~ s/^\s*\[\w+\]\s*//g; # [ANNOUNCE] or [ANN] my $dtime = msg_datestamp($mime->header_obj); - $title .= ' - ' . PublicInbox::View::fmt_ts($dtime) . ' UTC'; + $title .= ' - ' . fmt_ts($dtime) . ' UTC'; print $out $title, "\n" or die; my $uline = '=' x length($title); print $out $uline, "\n\n" or die; @@ -102,24 +108,26 @@ sub mime2txt { } sub mime2html { - my ($out, $mime, $ctx) = @_; - my $smsg = bless { mime => $mime }, 'PublicInbox::SearchMsg'; - print $out PublicInbox::View::index_entry($smsg, $ctx, 1) or die; + my ($out, $eml, $ctx) = @_; + my $smsg = $ctx->{smsg} = bless {}, 'PublicInbox::Smsg'; + $smsg->populate($eml); + $ctx->{msgs} = [ 1 ]; # for <hr> in eml_entry + print $out PublicInbox::View::eml_entry($ctx, $eml) or die; } sub html_start { my ($out, $ctx) = @_; require PublicInbox::WwwStream; $ctx->{www} = My::MockObject->new(style => ''); - my $www_stream = PublicInbox::WwwStream->new($ctx); - print $out $www_stream->_html_top, '<pre>' or die; + my $www_stream = PublicInbox::WwwStream::init($ctx); + print $out $www_stream->html_top, '<pre>' or die; } sub html_end { - print $out <<EOF or die; - git clone $PublicInbox::WwwStream::CODE_URL -</pre></body></html> -EOF + for (@$PublicInbox::WwwStream::CODE_URL) { + print $out " git clone $_\n" or die; + } + print $out "</pre></body></html>\n" or die; } sub atom_start { @@ -127,10 +135,10 @@ sub atom_start { require PublicInbox::WwwAtomStream; # WwwAtomStream stats this dir for mtime my $astream = PublicInbox::WwwAtomStream->new($ctx); - delete $ctx->{emit_header}; - my $ibx = $ctx->{-inbox}; + delete $astream->{emit_header}; + my $ibx = $ctx->{ibx}; my $title = PublicInbox::WwwAtomStream::title_tag($ibx->description); - my $updated = PublicInbox::WwwAtomStream::feed_updated(gmtime($mtime)); + my $updated = PublicInbox::WwwAtomStream::feed_updated($mtime); print $out <<EOF or die; <?xml version="1.0" encoding="us-ascii"?> <feed @@ -146,9 +154,10 @@ EOF } sub mime2atom { - my ($out, $astream, $mime, $ctx) = @_; - my $smsg = bless { mime => $mime }, 'PublicInbox::SearchMsg'; - if (defined(my $str = $astream->feed_entry($smsg))) { + my ($out, $astream, $eml, $ctx) = @_; + my $smsg = bless {}, 'PublicInbox::Smsg'; + $smsg->populate($eml); + if (defined(my $str = $astream->feed_entry($smsg, $eml))) { print $out $str or die; } } diff --git a/Documentation/public-inbox-cindex.pod b/Documentation/public-inbox-cindex.pod new file mode 100644 index 00000000..fdc2b82d --- /dev/null +++ b/Documentation/public-inbox-cindex.pod @@ -0,0 +1,136 @@ +=head1 NAME + +public-inbox-cindex - create and update code repository search indices + +=head1 SYNOPSIS + +public-inbox-cindex [OPTIONS] -g GIT_DIR [-g GIT_DIR]... + +public-inbox-cindex [OPTIONS] --update + +=head1 DESCRIPTION + +public-inbox-cindex creates and updates the Xapian search index for +git code repository (C<coderepo>) search. It can also associate +(fuzzy join) coderepos with Xapian-indexed inboxes. It only indexes +commit messages and diffs as they would show up in an email. It +does not index the contents of blobs directly. + +Like inbox indices, coderepo indices can either be internal or external +to a coderepo. Either way, they're both created and updated through +public-inbox-cindex. + +Once the initial indices are created by public-inbox-cindex, +the L</--update> switch will incrementally update them. + +=head1 OPTIONS + +=over + +=item -d EXTDIR + +Use the given directory as an external index. External indices are +generally recommended to internal indices since they do not need +write access to any code repositories themselves. They are highly +recommended when many repositories share a common history or if +there is an M:N relationship between inboxes and coderepos. + +=item -j JOBS + +=item --jobs=JOBS + +Influences the number of Xapian indexing shards. + +If the repo has not been indexed or initialized, C<JOBS - 1> +shards will be created. + +Default: the number of existing Xapian shards + +=item --reindex + +Forces a re-index of all commits. This can be used for in-place +upgrades and bugfixes while read-only processes are utilizing the index. + +=item --update + +=item -u + +Incrementally index all previously-indexed coderepos. + +=item --prune + +Removes commits which are no longer accessible via git. +Use this after L<git-gc(1)> (or L<git-prune(1)>). + +=item --no-fsync + +=item --dangerous + +=item --max-size SIZE + +=item --batch-size SIZE + +These affect the coderepo index the same way they affect +inbox indices. See L<public-inbox-index(1)>. + +=back + +=head1 FILES + +For internal indices, the Xapian DB is stored in +C<$GIT_DIR/public-inbox-cindex>. + +External indices are stored wherever L</-d> EXTDIR points. + +=head1 CONFIGURATION + +=over 8 + +=item publicinbox.indexMaxSize + +=item publicinbox.indexBatchSize + +These configuration knobs affect the coderepo index the same way +they affect inbox indices. See L<public-inbox-index(1)>. + +=back + +=head1 ENVIRONMENT + +=over 8 + +=item PI_CONFIG + +Used to override the default "~/.public-inbox/config" value. + +=item XAPIAN_FLUSH_THRESHOLD + +The number of documents to update before committing changes to +disk. This variable is handled directly by Xapian, refer to +Xapian API documentation for more details. + +Use C<publicinbox.indexBatchSize> instead. + +=back + +=head1 UPGRADING + +Occasionally, public-inbox will update its schema version and +require a full reindex by running this command with L</--reindex>. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-index(1)> diff --git a/Documentation/public-inbox-clone.pod b/Documentation/public-inbox-clone.pod new file mode 100644 index 00000000..64ee3138 --- /dev/null +++ b/Documentation/public-inbox-clone.pod @@ -0,0 +1,287 @@ +=head1 NAME + +public-inbox-clone - "git clone --mirror" wrapper + +=head1 SYNOPSIS + +public-inbox-clone [OPTIONS] INBOX_URL [INBOX_DIR] + +public-inbox-clone [OPTIONS] ROOT_URL [DESTINATION] # public-inbox 2.0+ + +=head1 DESCRIPTION + +public-inbox-clone is a wrapper around C<git clone --mirror> for +making the initial clone of a remote HTTP(S) public-inbox. It +allows cloning multi-epoch v2 inboxes with a single command and +zero configuration. + +In public-inbox 2.0+, public-inbox-clone can create and maintain +a mirror of multiple inboxes or code repositories using manifest.js.gz +files like L<grok-pull(1)> from grokmirror. L<public-inbox-fetch(1)> is +NOT required when using this mode. + +It does not run L<public-inbox-init(1)> nor +L<public-inbox-index(1)>. Those commands must be run separately +if serving/searching the mirror is required. As-is, +public-inbox-clone is suitable for creating a git-only backup +without Xapian and SQLite indices. + +When cloning a single inbox, public-inbox-clone creates a Makefile +with handy targets to update the inbox once indexed. +This Makefile may be edited by the user; it will +not be rewritten by L<public-inbox-fetch(1)> unless it is removed +completely. + +public-inbox-clone does not use nor require any extra +configuration files (not even C<~/.public-inbox/config>), +but it can download snippets suitable for adding to any +L<public-inbox-config(5)> file. + +L<public-inbox-fetch(1)> may be used to keep a single C<INBOX_DIR> +up-to-date. + +For v2 inboxes, it will create a C<$INBOX_DIR/manifest.js.gz> +file to speed up subsequent L<public-inbox-fetch(1)>. + +=head1 OPTIONS + +=over + +=item --epoch=RANGE + +Restrict clones of L<public-inbox-v2-format(5)> inboxes to the +given range of epochs. The range may be a single non-negative +integer or a (possibly open-ended) C<LOW..HIGH> range of +non-negative integers. C<~> may be prefixed to either (or both) +integer values to represent the offset from the maximum possible +value. + +For example, C<--epoch=~0> alone clones only the latest epoch, +C<--epoch=~2..> clones the three latest epochs. + +Default: C<0..~0> or C<0..> or C<..~0> +(all epochs, all three examples are equivalent) + +=item -I PATTERN + +=item --include=PATTERN + +When cloning a top-level with multiple inboxes via manifest, +only clone inboxes and repositories matching a given wildcard pattern +(using C<*?> and C<[]> is supported). + +This is a new option in public-inbox 2.0+ + +=item --exclude=PATTERN + +When cloning a top-level with multiple inboxes via manifest, +ignore inboxes and repositories matching the given wildcard pattern. +Supports the same wildcards as L</--include> + +This is a new option in public-inbox 2.0+ + +=item --inbox-config=always|v2|v1|never + +Whether or not to retrieve the C<$INBOX/_/text/config/raw> HTTP(S) +endpoint when cloning. + +Since we can't deduce v1 inboxes from code repositories, setting this +to C<v2> or C<never> can allow faster clones of code repositories if +no v1 inboxes are present. + +Default: C<always> + +This is a new option in public-inbox 2.0+ + +=item --inbox-version=NUM + +Force a remote public-inbox version (must be C<1> or C<2>). +This is auto-detected by default, and this option exists mainly +for testing. + +This is a new option in public-inbox 2.0+ + +=item --objstore=DIR + +Enables space savings when the remote C<manifest.js.gz> +includes C<forkgroup> entries as generated by grokmirror 2.x. + +If C<DIR> does not start with C</>, C<./>, or C<../>, it is treated +as relative to the C<DESTINATION> directory. If only C<--objstore=> +is specified where C<DIR> is an empty string (C<"">), then C<objstore> +(C<$DESTINATION/objstore>) is the implied value of C<DIR>. + +This is a new option in public-inbox 2.0+ + +=item --manifest=FILE + +When incrementally updating an existing mirror, load the given +manifest (typically C<manifest.js.gz>) to speed up updates. + +By default, public-inbox writes the retrieved manifest to +C<$DESTINATION/manifest.js.gz>, this directive also +changes the destination to the specified C<FILE> + +If C<FILE> does not start with C</>, C<./>, or C<../>, it is treated +as relative to the C<DESTINATION> directory. If only C<--manifest=> +is specified where C<FILE> is an empty string (C<"">), then C<manifest.js.gz> +(C<$DESTINATION/manifest.js.gz>) is the implied value of C<FILE>. + +When updating manifests with many forks using the same objstore, +git 2.41+ is highly recommended for performance as we automatically +use the C<fetch.hideRefs> feature to speed up negotiation. + +C<--manifest=> is a new option in public-inbox 2.0+ + +=item --remote-manifest=URL|RELATIVE_PATH + +Use an alternate location for the remote manifest.js.gz file. +This may be specified as a full absolute URL (e.g +C<--remote-manifest=https://80x24.org/lore/pub/manifest.js.gz>), +or a pathname relative to the ROOT_URL (e.g +C<--remote-manifest=pub/manifest.js.gz> when ROOT_URL is +C<https://80x24.org/lore/> + +By default, C<ROOT_URL/manifest.js.gz> is used. + +This is a new option in public-inbox 2.0+ + +=item --project-list=FILE + +When cloning code repos from a manifest, generate a cgit-compatible +project list. + +If C<FILE> does not start with C</>, C<./>, or C<../>, it is treated +as relative to the C<DESTINATION> directory. If only C<--project-list=> +is specified where C<FILE> is an empty string (C<"">), then C<projects.list> +(C<$DESTINATION/projects.list>) is the implied value of C<FILE>. + +This is a new option in public-inbox 2.0+ + +=item --post-update-hook=COMMAND + +Hooks to run after a repository is cloned or updated, C<COMMAND> will +have the bare git repository destination given as its first and only +argument. + +For v2 inboxes, this operates on a per-epoch basis. + +May be specified multiple times to run multiple commands in the +order specified on the command-line. + +This is a new option in public-inbox 2.0+ + +=item -p + +=item --prune + +Pass the C<--prune> and C<--prune-tags> flags to L<git-fetch(1)> +calls on incremental clones. + +This is a new option in public-inbox 2.0+ + +=item --purge + +Deletes entire repos which no longer exist in the remote manifest, +or are filtered out by C<--include=> or C<--exclude=>. + +This is only useful when using C<--manifest> + +This is a new option in public-inbox 2.0+ + +=item --exit-code + +Exit with C<127> if no updates are done when relying on a manifest. +Updates include fingerprint mismatches in the manifest, new symlinks, +new repositories, and removed repositories from the L<--project-list> + +This is a new option in public-inbox 2.0+ + +=item -k + +=item --keep-going + +Continue as much as possible after an error. + +This is a new option in public-inbox 2.0+ + +=item -n + +=item --dry-run + +Show what would be done, without making any changes. + +This is a new option in public-inbox 2.0+ + +=item -q + +=item --quiet + +Quiets down progress messages, also passed to L<git-fetch(1)>. + +=item -v + +=item --verbose + +Increases verbosity, also passed to L<git-fetch(1)>. + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=item -j JOBS + +=item --jobs=JOBS + +The number of parallel processes to spawn at once for various network +operations using L<git(1)> and/or L<curl(1)>. + +=back + +=head1 EXAMPLES + +=for comment +Sticking to smaller projects in examples to minimize load on servers + +=over + +=item To mirror the most recent epochs of dwarves and LTTng inboxes: + + public-inbox-clone --epoch=~0 \ + --include='*lttng*' --include='*dwarves' \ + https://80x24.org/lore/ /path/to/inbox-mirror + +C<https://lore.kernel.org/> may be used instead of C<https://80x24.org/lore/> + +=item To mirror all code repos of the sparse project: + + public-inbox-clone --objstore= --project-list= --prune \ + --include='*sparse*' --inbox-config=never \ + --remote-manifest=https://80x24.org/lore/pub/manifest.js.gz \ + https://80x24.org/lore/ /path/to/code-mirror + +C<https://git.kernel.org/> may be used instead of C<https://80x24.org/lore/> +and the C<--remote-manifest> option can be omitted. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-fetch(1)>, L<public-inbox-init(1)>, L<public-inbox-index(1)> diff --git a/Documentation/public-inbox-compact.pod b/Documentation/public-inbox-compact.pod index d7a7dba0..d2b74c86 100644 --- a/Documentation/public-inbox-compact.pod +++ b/Documentation/public-inbox-compact.pod @@ -4,7 +4,9 @@ public-inbox-compact - compact Xapian DBs in an inbox =head1 SYNOPSIS - public-inbox-compact INBOX_DIR +public-inbox-compact INBOX_DIR + +public-inbox-compact --all =head1 DESCRIPTION @@ -17,11 +19,26 @@ It enforces the use of the C<--no-renumber> option of L<xapian-compact(1)> which is required to work with the rest of the public-inbox search code. +This command is rarely needed for active inboxes. + +Using the C<--compact> option of L<public-inbox-index(1)> +is recommended, instead, and only when doing a C<--reindex>. + =head1 OPTIONS =over -=item --blocksize / --no-full / --fuller +=item --all + +Compact all inboxes configured in ~/.public-inbox/config. +This is an alternative to specifying individual inboxes directories +on the command-line. + +=item --blocksize + +=item --no-full + +=item --fuller These options are passed directly to L<xapian-compact(1)>. @@ -50,12 +67,12 @@ Default: 10000 Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2018-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2018-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-config.pod b/Documentation/public-inbox-config.pod index 53926ef4..b4a1d94d 100644 --- a/Documentation/public-inbox-config.pod +++ b/Documentation/public-inbox-config.pod @@ -62,35 +62,32 @@ Default: none, optional =item publicinbox.<name>.newsgroup -The NNTP group name for use with L<public-inbox-nntpd(8)>. This -may be any newsgroup name with hierarchies delimited by '.'. +The NNTP group name for use with L<public-inbox-nntpd(1)>. This +may be any newsgroup name with hierarchies delimited by C<.>. For example, the newsgroup for L<mailto:meta@public-inbox.org> is: C<inbox.comp.mail.public-inbox.meta> -Omitting this for the given inbox will prevent the group from -being read by L<public-inbox-nntpd(1)> +It also configures the folder hierarchy used by L<public-inbox-imapd(1)> +as well as L<public-inbox-pop3d(1)> -Default: none, optional +Omitting this for a given inbox will prevent the inbox from +being served by L<public-inbox-nntpd(1)>, +L<public-inbox-imapd(1)>, and/or L<public-inbox-pop3d(1)> -=item publicinbox.<name>.watch +Newsgroup names should be all lowercase. Uppercase characters are +converted to lowercase for compatibility with IMAP, POP3, and our +L<public-inbox-extindex(1)> and L<public-inbox-cindex(1)> tools +starting with public-inbox 2.0+ (they were unusable before). -A location for L<public-inbox-watch(1)> to watch. Currently, -only C<maildir:> paths are supported: +Default: none, optional - [publicinbox "test"] - watch = maildir:/path/to/maildirs/.INBOX.test/ +=item publicinbox.<name>.watch -Default: none; only for L<public-inbox-watch(1)> users +See L<public-inbox-watch(1)> =item publicinbox.<name>.watchheader - [publicinbox "test"] - watchheader = List-Id:<test.example.com> - -If specified, L<public-inbox-watch(1)> will only process mail matching -the given header. Multiple values are not currently supported. - -Default: none; only for L<public-inbox-watch(1)> users +See L<public-inbox-watch(1)> =item publicinbox.<name>.listid @@ -99,7 +96,7 @@ angle brackets for L<public-inbox-mda(1)> deliveries and L<public-inbox-watch(1)>. For public-inbox-watch users, this is a shortcut for specifying -C<publicinbox.$NAME.watchheader=List-Id:<foo.example.com>> +C<publicinbox.$NAME.watchheader=List-Id:E<lt>foo.example.comE<gt>> For public-inbox-mda users, this may be used to avoid recipient matching via C<ORIGINAL_RECIPIENT> environment variable. @@ -110,6 +107,12 @@ needs to match. Default: none +=item publicinbox.<name>.imapmirror + +This may be the full IMAP URL of an independently-run IMAP mirror. + +Default: none + =item publicinbox.<name>.nntpmirror This may be the full NNTP URL of an independently-run mirror. @@ -127,8 +130,8 @@ C<basic> only requires L<DBD::SQLite(3pm)> and provides all NNTP functionality along with thread-awareness in the WWW interface. -C<medium> requires L<Search::Xapian(3pm)> to provide full-text -term search functionality in the WWW UI. +C<medium> requires L<Xapian(3pm)> or L<Search::Xapian(3pm)> to provide +full-text term search functionality in the WWW UI. C<full> also includes positional information used by Xapian to allow for searching for phrases using quoted text. @@ -136,6 +139,18 @@ allow for searching for phrases using quoted text. Default: C<full> +=item publicinbox.<name>.boost + +Control indexing order for L<public-inbox-extindex(1)>, with ties +broken by config file order. This only affects indexing and does +not affect messages which are already indexed. + +Default: C<0> + +=item publicinbox.<name>.indexSequentialShard + +See L<public-inbox-index(1)/publicInbox.indexSequentialShard> + =item publicinbox.<name>.httpbackendmax If a digit, the maximum number of parallel @@ -181,11 +196,28 @@ Default: :all The local path name of a CSS file for the PSGI web interface. May contain the attributes "media", "title" and "href" which match the associated attributes of the HTML <style> tag. -"href" may be specified to point to the URL of an remote CSS file +"href" may be specified to point to the URL of a remote CSS file and the path may be "/dev/null" or any empty file. Multiple files may be specified and will be included in the order specified. +=item publicinboxImport.dropUniqueUnsubscribe + +Drop C<List-Unsubscribe> headers if the message also includes +the C<List-Unsubscribe-Post: List-Unsubscribe=One-Click> header +to signal MUAs to support an instantaneous unsubscribe. This +is strongly recommended for users creating their own public +archives of mailing lists they subscribe to, otherwise any +archive reader can unsubscribe the archivist. + +This may break DKIM signatures if the C<List-Unsubscribe*> +headers are signed, but breaking DKIM signatures is the +lesser evil compared to allowing any reader to unsubscribe +the archivist. + +This affects L<public-inbox-mda(1)>, L<public-inbox-watch(1)>, +and L<public-inbox-learn(1)> + =item publicinboxmda.spamcheck This may be set to C<none> to disable the use of SpamAssassin @@ -197,38 +229,38 @@ Default: spamc =item publicinboxwatch.spamcheck -This may be set to C<spamc> to enable the use of SpamAssassin -L<spamc(1)> for filtering spam before it is imported into git -history. Other spam filtering backends may be supported in -the future. - -This requires L<public-inbox-watch(1)>, but affects all configured -public-inboxes in PI_CONFIG. - -Default: none +See L<public-inbox-watch(1)> =item publicinboxwatch.watchspam -A Maildir to watch for confirmed spam messages to appear in. -Messages which appear in this folder with the (S)een Maildir flag -will be hidden from all configured inboxes based on Message-ID -and content matching. +See L<public-inbox-watch(1)> -Messages without the (S)een Maildir flag are not considered for hiding. +=item publicinbox.imapserver -Default: none; only for L<public-inbox-watch(1)> users +Set this to point to the hostname(s) of the L<public-inbox-imapd(1)> +instance. This is used to advertise the existence of the IMAP +endpoint in the L<PublicInbox::WWW> HTML interface. + +Default: none =item publicinbox.nntpserver -Set this to point to the hostname of the L<public-inbox-nntpd(1)> -instance. This is used to advertise the existence of the NNTP -endpoint in the L<PublicInbox::WWW> HTML interface. +Same as C<publicinbox.imapserver>, but for the hostname(s) of the +L<public-inbox-nntpd(1)> instance. + +Default: none + +=item publicinbox.pop3server -Multiple values are allowed for instances with multiple hostnames -or mirrors. +Same as C<publicinbox.imapserver>, but for the hostname(s) of the +L<public-inbox-pop3d(1)> instance. Default: none +=item publicinbox.pop3state + +See L<public-inbox-pop3d(1)/publicinbox.pop3state> + =item publicinbox.<name>.feedmax The size of an Atom feed for the inbox. If specified more than @@ -241,7 +273,9 @@ Default: 25 A comma-delimited list of listings to hide the inbox from. -Valid values are currently C<www> and C<manifest>. +Valid values are currently C<www> and C<manifest> for non-C<404> +values of L</publicinbox.wwwListing> and L</publicinbox.grokManifest>, +respectively Default: none @@ -255,6 +289,10 @@ The URL of the cgit instance associated with the coderepo. Default: none +=item coderepo.snapshots + +See C<snapshots> in L<cgitrc(5)> + =item publicinbox.cgitrc A path to a L<cgitrc(5)> file. "repo.url" directives in the cgitrc @@ -277,16 +315,54 @@ Default: /var/www/htdocs/cgit/cgit.cgi or /usr/lib/cgit/cgit.cgi =item publicinbox.cgitdata A path to the data directory used by cgit for storing static files. -Typically guessed based the location of C<cgit.cgi> (from -C<publicinbox.cgitbin>, but may be overridden. +Typically guessed based on the location of C<cgit.cgi> (from +C<publicinbox.cgitbin>), but may be overridden. -Default: basename of C<publicinbox.cgitbin>, /var/www/htdocs/cgit/ +Default: dirname of C<publicinbox.cgitbin>, /var/www/htdocs/cgit/ or /usr/share/cgit/ +=item publicinbox.cgit + +Controls whether or not and how C<cgit> is used for serving coderepos. +New in public-inbox 2.0.0 (PENDING). + +=over 8 + +=item * first + +Try using C<cgit> as the first choice, this is the default. + +=item * fallback + +Fall back to using C<cgit> only if our native, inbox-aware +git code repository viewer doesn't recognize the URL. + +=begin comment +=for comment rewrite is not yet implemented +=item * rewrite + +Rewrite C<cgit> URLs for our native, inbox-aware code repository viewer. +This implies C<fallback> for URLs the native viewer does not recognize. + +=end comment + +=back + +Default: C<first> (C<cgit> will be used iff C<publicinbox.cgitrc> +is set and the C<cgit> binary exists). + =item publicinbox.mailEditor See L<public-inbox-edit(1)> +=item publicinbox.indexMaxSize + +=item publicinbox.indexBatchSize + +=item publicinbox.indexSequentialShard + +See L<public-inbox-index(1)> + =item publicinbox.wwwlisting Enable a HTML listing style when the root path of the URL '/' is accessed. @@ -304,14 +380,28 @@ L<Plack::App::Cascade(3pm)> =item * match=domain - Only show inboxes with URLs which belong to the domain of the HTTP request -=for TODO comment - -support showing cgit listing +=for comment +TODO support showing cgit listing =back Default: C<404> +=item publicinbox.nameIsUrl + +Treat the name of the public inbox as its unqualified URL when +using C<publicInbox.wwwListing=all>. That is, every +C<[publicinbox "foo"]> section implicitly sets C<publicinbox.foo.url=foo>. + +This is a convenient alternative to specifying +C<publicinbox.E<lt>nameE<gt>.url> for every single inbox if +your inbox URLs are domain-agnostic when using +C<publicInbox.wwwListing=all> + +Default: false + +New in public-inbox 2.0.0 (PENDING). + =item publicinbox.grokmanifest Controls the generation of a grokmirror-compatible gzipped JSON file @@ -340,6 +430,38 @@ always contain all git repos used by the inbox at C<$INBOX_URL> Default: C<match=domain> +=item publicinbox.<name>.obfuscate + +Whether to obfuscate email addresses in the L<PublicInbox::WWW> HTML +interface. + +Default: false + +=item publicinbox.noObfuscate + +A space-delimited list of well-known addresses and domains that should +not be obfuscated when C<publicinbox.$NAME.obfuscate> is true (e.g., +C<public@example.com> and C<@example.com>). This may be specified +more than once, in which case the values are merged. + +Default: none + +=item extindex.<name>.topdir + +The directory of an external index. See +L<public-inbox-extindex(1)> for more details. + +=item extindex.<name>.url + +Identical to L</publicinbox.E<lt>nameE<gt>.url>, but for +external indices + +=item extindex.<name>.coderepo + +Identical to L</publicinbox.E<lt>nameE<gt>.coderepo>, but for +external indices. Code repos may be freely associated with +any number of public inboxes and external indices. + =back =head2 NAMED LIMITER (PSGI) @@ -352,7 +474,7 @@ limiter with a low max value; while smaller inboxes can use the default limiter. C<RLIMIT_*> keys may be set to enforce resource limits for -a particular limiter. +a particular limiter (L<BSD::Resource(3pm)> is required). Default named-limiters are prefixed with "-". Currently, the "-cgit" named limiter is reserved for instances spawning @@ -424,12 +546,12 @@ Used to override the default "~/.public-inbox/config" value. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-convert.pod b/Documentation/public-inbox-convert.pod index fc5c4762..a2f8caf5 100644 --- a/Documentation/public-inbox-convert.pod +++ b/Documentation/public-inbox-convert.pod @@ -20,12 +20,14 @@ once they are satisfied with the conversion. =item --no-index -Disables Xapian and overview DB indexing on the new repository. -By default, public-inbox-convert creates a new index in the v2 -repository and indexes all existing messages, a lengthy +Disables Xapian and overview DB indexing on the new inbox. +By default, public-inbox-convert creates a new index in the +v2 inbox and indexes all existing messages, a lengthy operation for large inboxes. -=item -j JOBS, --jobs=JOBS +=item -j JOBS + +=item --jobs=JOBS Control the number of indexing jobs and Xapian shards of the v2 inbox. By default, this is the detected CPU count but capped @@ -33,6 +35,25 @@ at 4 due to various bottlenecks. The number of Xapian shards will be 1 less than the JOBS value, since there is a single process which distributes work to the Xapian shards. +=item -L LEVEL, --index-level=LEVEL + +=item -c, --compact + +=item -v, --verbose + +=item --no-fsync + +=item --sequential-shard + +=item --batch-size=BYTES + +=item --max-size=BYTES + +These options affect indexing. They have no effect if +L</--no-index> is specified + +See L<public-inbox-index(1)> for a description of these options. + =back =head1 ENVIRONMENT @@ -67,12 +88,12 @@ Maildirs. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2013-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-daemon.pod b/Documentation/public-inbox-daemon.pod index cb903df2..092be667 100644 --- a/Documentation/public-inbox-daemon.pod +++ b/Documentation/public-inbox-daemon.pod @@ -4,16 +4,18 @@ public-inbox-daemon - common usage for public-inbox network daemons =head1 SYNOPSIS + public-inbox-netd public-inbox-httpd + public-inbox-imapd public-inbox-nntpd + public-inbox-pop3d =head1 DESCRIPTION This manual describes common options and behavior for public-inbox network daemons. Network daemons for public-inbox -provide read-only NNTP and HTTP access to public-inboxes. Write -access to a public-inbox repository will never be required to -run these. +provide read-only IMAP, HTTP, NNTP and POP3 access to public-inboxes. +Write access to a public-inbox will never be required to run these. These daemons are implemented with a common core using non-blocking sockets and optimized for fairness; even with @@ -29,7 +31,9 @@ processes to take advantage of multiple CPUs. =over -=item -l, --listen ADDRESS +=item -l [PROTOCOL://]ADDRESS[?opt1=val1,opt2=val2] + +=item --listen [PROTOCOL://]ADDRESS[?opt1=val1,opt2=val2] This takes an absolute path to a Unix socket or HOST:PORT to listen on. For example, to listen to TCP connections on @@ -40,26 +44,44 @@ like L<nginx(8)> to use. May be specified multiple times to allow listening on multiple sockets. -This does not need to be specified at all if relying on -L<systemd.socket(5)> or similar +Unless per-listener options are used (required for +L<public-inbox-netd(1)>), this does not need to be specified at +all if relying on L<systemd.socket(5)> or similar, + +Per-listener options may be specified after C<?> as C<KEY=VALUE> +pairs delimited by C<,>. See L<public-inbox-netd(1)> for +documentation on the C<cert=>, C<key=>, C<env.NAME=VALUE>, +C<out=>, C<err=>, and C<psgi=> options available. Default: server-dependent unless socket activation is used with L<systemd(1)> or similar (see L<systemd.socket(5)>). -=item -1, --stdout PATH +=item -1 + +=item --stdout PATH Specify an appendable path to redirect stdout descriptor (1) to. Using this is preferable to setting up the redirect externally (e.g. E<gt>E<gt>/path/to/log in shell) since it allows SIGUSR1 to be handled (see L<SIGNALS/SIGNALS> below). -Default: /dev/null +C<out=> may also be specified on a per-listener basis. + +Default: /dev/null with C<--daemonize>, inherited otherwise -=item -2, --stderr PATH +=item -2 PATH + +=item --stderr PATH Like C<--stdout>, but for the stderr descriptor (2). -=item -W, --worker-processes +C<err=> may also be specified on a per-listener basis. + +Default: /dev/null with C<--daemonize>, inherited otherwise + +=item -W INTEGER + +=item --worker-processes INTEGER Set the number of worker processes. @@ -74,12 +96,81 @@ the master on crashes. Default: 1 +=item -X INTEGER + +=item --xapian-helpers INTEGER + +Enables the use of Xapian helper processes to handle expensive, +non-deterministic Xapian search queries asynchronously without +blocking simple requests. + +With positive values, there is an additional manager process +that can be signaled to control the number of Xapian helper workers. + +* C<-X0> one worker, no manager process +* C<-X1> one worker, one manager process +... +* C<-X8> eight workers, one manager process + +As with the public-facing public-inbox-* daemons, sending C<SIGTTIN> +or C<SIGTTOU> to the Xapian helper manager process will increment or +decrement the number of workers. + +Both Xapian helper workers and managers automatically respawn if they +crash or are explicitly killed, even with C<-X0>. + +A C++ compiler, L<pkg-config(1)>, and Xapian development files (e.g. +C<libxapian-dev> or C<xapian*-core-dev*>) are required to gain access to +some expensive queries and significant memory savings. + +Xapian helper workers are shared by all C<--worker-processes> of the +Perl daemon for additional memory savings. + +New in public-inbox 2.0.0. + +Default: undefined, search queries are handled synchronously + +=item --cert /path/to/cert + +The default TLS certificate for HTTPS, IMAPS, NNTPS, POP3S and/or STARTTLS +support if the C<cert> option is not given with C<--listen>. + +With this option, well-known TCP ports automatically get TLS or STARTTLS +support if using systemd-compatible socket activation. That is, ports +443, 563, 993, and 995 support HTTPS, NNTPS, IMAPS, and POP3S, +respectively; while ports 110, 119, and 143 support STARTTLS on POP3, +NNTP, and IMAP, respectively. + +=item --key /path/to/key + +The default TLS certificate key for the default C<--cert> or +per-listener C<cert=> option. The private key may be +concatenated into the cert file itself, in which case this +option is not needed. + +=item --multi-accept INTEGER + +By default, each worker accepts one connection at a time to maximize +fairness and minimize contention across multiple processes on a +shared listen socket. Accepting multiple connections at once may be +useful in constrained deployments with few, heavily loaded workers. +Negative values enables a worker to accept all available clients at +once, possibly starving others in the process. C<-1> behaves like +C<multi_accept yes> in nginx; while C<0> (the default) is +C<multi_accept no> in nginx. Positive values allow +fine-tuning without the runaway behavior of C<-1>. + +This may be specified on a per-listener basis via the C<multi-accept=> +per-listener directive (e.g. C<-l http://127.0.0.1?multi-accept=1>). + +Default: 0 + =back =head1 SIGNALS Most of our signal handling behavior is copied from L<nginx(8)> -and/or L<starman(1)>; so it is possible to reuse common scripts +and/or L<starman(1)>, so it is possible to reuse common scripts for managing them. =over 8 @@ -96,11 +187,11 @@ See L</UPGRADING> below. =item SIGHUP Reload config files associated with the process. -(FIXME: not tested for -httpd, yet) +(Note: broken for L<public-inbox-httpd(1)> only in E<lt>= 1.6) =item SIGTTIN -Increase the number of running workers processes by one. +Increase the number of running worker processes by one. =item SIGTTOU @@ -108,7 +199,7 @@ Decrease the number of running worker processes by one. =item SIGWINCH -Stop all running worker processes. SIGHUP or SIGTTIN +Stop all running worker processes. SIGHUP or SIGTTIN may be used to restart workers. =item SIGQUIT @@ -136,15 +227,15 @@ activation. See L<systemd.socket(5)> and L<sd_listen_fds(3)>. =item PERL_INLINE_DIRECTORY -Pointing this to point to a writable directory enables the use +Pointing this to a writable directory enables the use of L<Inline> and L<Inline::C> extensions which may provide platform-specific performance improvements. Currently, this enables the use of L<vfork(2)> which speeds up subprocess spawning with the Linux kernel. public-inbox will never enable L<Inline::C> automatically without -this environment variable set. See L<Inline> and L<Inline::C> -for more details. +this environment variable set or C<~/.cache/public-inbox/inline-c> +created by a user. See L<Inline> and L<Inline::C> for more details. =back @@ -153,8 +244,8 @@ for more details. There are two ways to upgrade a running process. Users of process management systems with socket activation -(L<systemd(1)> or similar) may rely on multiple instances For -systemd, this means using two (or more) '@' instances for each +(L<systemd(1)> or similar) may rely on multiple daemon instances. +For systemd, this means using two (or more) '@' instances for each service (e.g. C<SERVICENAME@INSTANCE>) as documented in L<systemd.unit(5)>. @@ -170,15 +261,16 @@ interrupted and lost. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> =head1 SEE ALSO -L<public-inbox-httpd(1)>, L<public-inbox-nntpd(1)> +L<public-inbox-httpd(1)>, L<public-inbox-imapd(1)>, +L<public-inbox-nntpd(1)>, L<public-inbox-pop3d(1)>, L<public-inbox-netd(1)> diff --git a/Documentation/public-inbox-edit.pod b/Documentation/public-inbox-edit.pod index c64b4da8..17f66c7c 100644 --- a/Documentation/public-inbox-edit.pod +++ b/Documentation/public-inbox-edit.pod @@ -1,6 +1,6 @@ =head1 NAME -public-inbox-edit - edit messages in a public inbox +public-inbox-edit - destructively edit messages in a public inbox =head1 SYNOPSIS @@ -91,16 +91,30 @@ See L<public-inbox-config(5)> Only L<v2|public-inbox-v2-format(5)> repositories are supported. +This is safe to run while normal inbox writing tools +(L<public-inbox-mda(1)>, L<public-inbox-watch(1)>, +L<public-inbox-learn(1)>) are active. + +Running this in parallel with L<public-inbox-xcpdb(1)> or +C<"public-inbox-index --reindex"> can lead to errors or +edited data remaining indexed. + +Incremental L<public-inbox-index(1)> (without C<--reindex>) +is fine. + +Keep in mind this is a last resort, as it will be disruptive +to anyone using L<git(1)> to mirror the inbox being edited. + =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2019-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-extindex-format.pod b/Documentation/public-inbox-extindex-format.pod new file mode 100644 index 00000000..d47e2102 --- /dev/null +++ b/Documentation/public-inbox-extindex-format.pod @@ -0,0 +1,110 @@ +% public-inbox developer manual + +=head1 NAME + +public-inbox-extindex-format - external index format description + +=head1 DESCRIPTION + +The extindex is an index-only evolution of the per-inbox +SQLite and Xapian indices used by L<public-inbox-v2-format(5)> +and L<public-inbox-v1-format(5)>. It exists to facilitate +searches across multiple inboxes as well as to reduce index +space when messages are cross-posted to several existing +inboxes. + +It transparently indexes messages across any combination of v1 and v2 +inboxes and data about inboxes themselves. + +=head1 DIRECTORY LAYOUT + +While inspired by v2, there is no git blob storage nor +C<msgmap.sqlite3> DB. + +Instead, there is an C<ALL.git> (all caps) git repo which treats +every indexed v1 inbox or v2 epoch as a git alternate. + +As with v2 inboxes, it uses C<over.sqlite3> and Xapian "shards" +for WWW and IMAP use. Several exclusive new tables are added +to deal with L</XREF3 DEDUPLICATION> and metadata. + +Unlike v1 and v2 inboxes, it is NOT designed to map to a NNTP +newsgroup. Thus it lacks C<msgmap.sqlite3> to enforce the +unique Message-ID requirement of NNTP. + +=head2 INDEX OVERVIEW AND DEFINITIONS + + $SCHEMA_VERSION - DB schema version (for Xapian) + $SHARD - Integer starting with 0 based on parallelism + + foo/ # "foo" is the name of the index + - ei.lock # lock file to protect global state + - ALL.git # empty, alternates for inboxes + - ei$SCHEMA_VERSION/$SHARD # per-shard Xapian DB + - ei$SCHEMA_VERSION/over.sqlite3 # overview DB for WWW, IMAP + - ei$SCHEMA_VERSION/misc # misc Xapian DB + +File and directory names are intentionally different from +analogous v2 names to ensure extindex and v2 inboxes can +easily be distinguished from each other. + +=head2 XREF3 DEDUPLICATION + +Due to cross-posted messages being the norm in the large Linux kernel +development community and Xapian indices being the primary consumer of +storage, it makes sense to deduplicate indexing as much as possible. + +The internal storage format is based on the NNTP "Xref" tuple, +but with the addition of a third element: the git blob OID. +Thus the triple is expressed in string form as: + + $NEWSGROUP_NAME:$ARTICLE_NUM:$OID + +If no C<newsgroup> is configured for an inbox, the C<inboxdir> +of the inbox is used. + +This data is stored in the C<xref3> table of over.sqlite3. + +=head2 misc XAPIAN DB + +In addition to the numeric Xapian shards for indexing messages, +there is a new, in-development Xapian index for storing data +about inboxes themselves and other non-message data. This +index allows us to speed up operations involving hundreds or +thousands of inboxes. + +=head1 BENEFITS + +In addition to providing cross-inbox search capabilities, it can +also replace per-inbox Xapian shards (but not per-inbox +over.sqlite3). This allows reduction in disk space, open file +handles, and associated memory use. + +=head1 CAVEATS + +Relocating v1 and v2 inboxes on the filesystem will require +extindex to be garbage-collected and/or reindexed. + +Configuring and maintaining stable C<newsgroup> names before any +messages are indexed from every inbox can avoid expensive +reindexing and rely exclusively on GC. + +=head1 LOCKING + +L<flock(2)> locking exclusively locks the empty ei.lock file +for all non-atomic operations. + +=head1 THANKS + +Thanks to the Linux Foundation for sponsoring the development +and testing. + +=head1 COPYRIGHT + +Copyright 2020-2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-v2-format(5)> diff --git a/Documentation/public-inbox-extindex.pod b/Documentation/public-inbox-extindex.pod new file mode 100644 index 00000000..2db7d7e9 --- /dev/null +++ b/Documentation/public-inbox-extindex.pod @@ -0,0 +1,180 @@ +=head1 NAME + +public-inbox-extindex - create and update external search indices + +=head1 SYNOPSIS + +public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR... + +public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all + +=head1 DESCRIPTION + +public-inbox-extindex creates and updates an external search and +overview database used by the read-only public-inbox PSGI (HTTP), +NNTP, and IMAP interfaces. This requires either the +L<Xapian> SWIG bindings OR or L<Search::Xapian> XS bindings +along with L<DBD::SQLite> and L<DBI> Perl modules. + +=head1 OPTIONS + +=over + +=item -j JOBS + +=item --jobs=JOBS + +=item --no-fsync + +=item --dangerous + +=item --rethread + +=item --max-size SIZE + +=item --batch-size SIZE + +These switches behave as they do for L<public-inbox-index(1)> + +=item --all + +Index all C<publicinbox> entries in C<PI_CONFIG>. + +C<publicinbox> entries indexed by C<public-inbox-extindex> can +have full Xapian searching abilities with the per-C<publicinbox> +C<indexlevel> set to C<basic> and their respective Xapian +(C<xap15> or C<xapian15>) directories removed. For multiple +public-inboxes where cross-posting is common, this allows +significant space savings on Xapian indices. + +=item --dedupe=MSGID + +=item --dedupe + +Rerun deduplication on messages with the given Message-ID or +all messages if no Message-ID is specified. Deduplication rules may +change and evolve over time, especially if filters are involved. + +C<--dedupe=MSGID> may be specified multiple times to deduplicate +multiple Message-IDs. + +Use this if you see C<W: BUG? $MSGID not deduplicated properly> +warnings from WWW logs. + +=item --gc + +Perform garbage collection instead of indexing. Use this if +inboxes are removed from the extindex, a newsgroup name is +set or changed, or if messages are purged or removed from +some inboxes. + +=item --reindex + +Forces a re-index of all messages in the extindex. This can be +used for in-place upgrades and bugfixes while read-only server +processes are utilizing the index. Keep in mind this roughly +doubles the size of the already-large Xapian database. + +=item --fast + +Used with C<--reindex>, it will only look for new and stale +entries and not touch already-indexed messages. + +=item --no-multi-pack-index + +Disable writing a L<git-multi-pack-index(1)> file to save memory. +Normally, enabling multi-pack-index speeds up startup time of +subsequent L<git-cat-file(1)> processes by 3-4%, but generating +this file requires several GB of memory with large repos. + +Unlike the C<core.multiPackIndex> directive in git, it's still +possible to read existing multi-pack-index files if they are +created elsewhere. + +Available in public-inbox 2.0.0+ + +=back + +=head1 FILES + +L<public-inbox-extindex-format(5)> + +=head1 CONFIGURATION + +public-inbox-extindex does not write to the L<public-inbox-config(5)> +file, it must be entered manually. +The extindex name of C<all> is a special case which +corresponds to indexing C<--all> inboxes. An example for +C<--all> is as follows: + + [extindex "all"] + topdir = /path/to/extindex_dir + url = all + coderepo = foo + coderepo = bar + +Putting an C<extindex> entry in the config allows L<PublicInbox::WWW>. +You can have any number of C<extentry.$NAME> sections where C<$NAME> +is something other than C<all> to display a union of several inboxes. + +It is strongly recommended any public inboxes indexed by this command +have a stable C<publicinbox.$NAME.newsgroup> entry (regardless of +the presence of an NNTP or IMAP server). Otherwise, public-inbox-extindex +will use C<publicinbox.$NAME.inboxdir> as an internal key which can +cause needless reindexing and require L<--gc> if inboxes are relocated. + +See L<public-inbox-config(5)> for more details. + +=head1 ENVIRONMENT + +=over 8 + +=item PI_CONFIG + +Used to override the default "~/.public-inbox/config" value. + +=item XAPIAN_FLUSH_THRESHOLD + +The number of documents to update before committing changes to +disk. This environment is handled directly by Xapian, refer to +Xapian API documentation for more details. + +Setting C<XAPIAN_FLUSH_THRESHOLD> or +C<publicinbox.indexBatchSize> for a large C<--reindex> may cause +L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and +L<public-inbox-watch(1)> tasks to wait long and unpredictable +periods of time during C<--reindex>. + +Default: none, uses C<publicinbox.indexBatchSize> + +=back + +=head1 UPGRADING + +Occasionally, public-inbox will update its schema version and +require a full index by running this command. + +=head1 LOCKING + +It is safe to use C<--dedupe>, C<--gc> and C<--reindex> while +other processes are writing to covered inboxes or extindex. +The extindex locks will be released roughly every 10s to +allow L<public-inbox-mda(1)> and L<public-inbox-watch(1)> +processes to write to the extindex. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<Search::Xapian>, L<DBD::SQLite> diff --git a/Documentation/public-inbox-fetch.pod b/Documentation/public-inbox-fetch.pod new file mode 100644 index 00000000..1ff0df44 --- /dev/null +++ b/Documentation/public-inbox-fetch.pod @@ -0,0 +1,118 @@ +=head1 NAME + +public-inbox-fetch - "git fetch" wrapper for v2 inbox mirrors + +=head1 SYNOPSIS + +public-inbox-fetch [--exit-code] -C INBOX_DIR + +=head1 DESCRIPTION + +public-inbox-fetch updates git storage of public-inbox mirrors. +With v2 inboxes, it allows detection of new epochs and avoids +unnecessary traffic on old epochs. + +public-inbox-fetch does not use nor require any configuration +files of its own. + +It does not run L<public-inbox-index(1)>, making it suitable +for maintaining git-only backups. + +For v2 inboxes, it will maintain C<$INBOX_DIR/manifest.js.gz> +file to speed up future invocations. It always safe to remove +manifest.js.gz, it is merely an optimization and will be +restored on the next invocation. + +To prevent fetches on any v2 epoch, use L<chmod(1)> to remove +write permissions to the top-level of the epoch. For example, +to disable fetches on epoch 4: + + chmod a-w $INBOX_DIR/git/4.git + +If you wish to re-enable fetches to the epoch: + + chmod u+w $INBOX_DIR/git/4.git + +=head1 OPTIONS + +=over + +=item -q + +=item --quiet + +Quiets down progress messages, also passed to L<git-fetch(1)>. + +=item -T REMOTE + +=item --try-remote REMOTE + +Try a given remote name instead of C<origin> or C<_grokmirror>. +May be specified more than once. + +Default: C<origin>, C<_grokmirror> + +=item --exit-code + +Exit with C<127> if no updates are done. This can be used in +shell scripts to avoid invoking L<public-inbox-index(1)> when +there are no updates: + + public-inbox-fetch -q --exit-code && public-inbox-index + test $? -eq 0 || exit $? + +=item -p + +=item --prune + +Pass the C<--prune> and C<--prune-tags> flags to L<git-fetch(1)> calls. + +This is a new option in public-inbox 2.0+ + +=item -v + +=item --verbose + +Increases verbosity, also passed to L<git-fetch(1)>. + +=item --torsocks=auto|no|yes + +=item --no-torsocks + +Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. + +Default: C<auto> + +=back + +=head1 EXIT CODES + +=over + +=item 127 + +no updates when L</--exit-code> is used above + +=back + +public-inbox-fetch will also exit with curl L<curl(1)/EXIT CODES> +as documented in the L<curl(1)> manpage (e.g. C<7> when curl cannot +reach a host). Likewise, L<git-fetch(1)> failures are also +propagated to the user. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-index(1)>, L<curl(1)> diff --git a/Documentation/public-inbox-glossary.pod b/Documentation/public-inbox-glossary.pod new file mode 100644 index 00000000..d88539c8 --- /dev/null +++ b/Documentation/public-inbox-glossary.pod @@ -0,0 +1,127 @@ +=head1 NAME + +public-inbox-glossary - glossary for public-inbox + +=head1 DESCRIPTION + +public-inbox combines several independently-developed protocols +and data formats with overlapping concepts. This document is +intended as a guide to identify and clarify overlapping concepts +with different names. + +This is mainly intended for hackers of public-inbox, but may be useful +for administrators of public-facing services and/or users building +tools. + +=head1 TERMS + +=over 8 + +=item IMAP UID, NNTP article number, on-disk Xapian docid + +A sequentially-assigned positive integer. These integers are per-inbox, +or per-extindex. This is the C<num> column of the C<over> table in +C<over.sqlite3> + +=item tid, THREADID + +A sequentially assigned positive integer. These integers are +per-inbox or per-extindex. In the future, this may be prefixed +with C<T> for JMAP (RFC 8621) and RFC 8474. This may not be +strictly compliant with RFC 8621 since inboxes and extindices +are considered independent entities from each other. + +This is the C<tid> column of the C<over> table in C<over.sqlite3> + +=item blob + +For email, this is the git blob object ID (SHA-(1|256)) of an +RFC-(822|2822|5322) email message. + +=item IMAP EMAILID, JMAP Email Id + +To be decided. This will likely be the git blob ID prefixed with C<g> +rather than the numeric UID to accommodate the same blob showing +up in both an extindex and inbox (or multiple extindices). + +=item newsgroup + +The name of the NNTP newsgroup, see L<public-inbox-config(5)>. + +=item IMAP (folder|mailbox) slice + +A 50K slice of a newsgroup to accommodate the limitations of IMAP +clients with L<public-inbox-imapd(1)>. This is the C<newsgroup> +name with a C<.$INTEGER_SUFFIX>, e.g. a newsgroup named C<inbox.test> +would have its first slice named C<inbox.test.0>, and second slice +named C<inbox.test.1> and so forth. + +If implemented, the RFC 8474 MAILBOXID of an IMAP slice will NOT have +the same Mailbox Id as the public-facing full JMAP mailbox. + +=item inbox name, public JMAP mailbox name + +The HTTP(S) name of the public-inbox +(C<publicinbox.E<lt>nameE<gt>.*>). JMAP will use this name +rather than the newsgroup name since public-facing JMAP will be +part of the PSGI code and not need a separate daemon like +L<public-inbox-nntpd(1)> or L<public-inbox-imapd(1)> + +=item epoch + +A git repository used for blob storage. See +L<public-inbox-v2-format(5)/GIT EPOCHS>. + +=item keywords, (IMAP|Maildir) flags, mbox Status + X-Status + +Private, per-message keywords or flags as described in RFC 8621 +section 10.4. These are conveyed in the C<Status:> and +C<X-Status:> headers for L<mbox(5)>, as system IMAP FLAGS +(RFC 3501 section 2.3.2), or Maildir info flags. + +L<public-inbox-watch(1)> ignores drafts and trashed (deleted) +messages. L<lei-import(1)> ignores trashed (deleted) messages, +but it imports drafts. + +=item labels, private JMAP mailboxes + +For L<lei(1)> users only. This will allow lei users to place +the same email into one or more virtual folders for +ease of filtering. This is NOT tied to public-inbox names, as +messages stored by lei may not be public. + +These are similar in spirit to arbitrary freeform "tags" +in mail software such as L<notmuch(1)> and non-system IMAP FLAGS. + +=item volatile metadata (VMD) + +For L<lei(1)> users only, this refers to the combination of +keywords and labels which are subject to frequent change +independently of immutable message content. + +=item IMAP INTERNALDATE, JMAP receivedAt, rt: search prefix + +The first valid timestamp value of Received: headers (top first). +If no Received: header exists, the Date: header is used, and the +current time if neither header(s) exist. When mirroring via +git, this is the git commit time. + +=item IMAP SENT*, JMAP sentAt, dt: and d: search prefixes + +The first valid timestamp value of the Date: header(s). +If no Date: header exists, the time from the Received: header is +used, and then the current time if neither header exists. +When mirroring via git, this is the git author time. + +=back + +=head1 COPYRIGHT + +Copyright 2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<public-inbox-v2-format(5)>, L<public-inbox-v1-format(5)>, +L<public-inbox-extindex-format(5)>, L<gitglossary(7)> diff --git a/Documentation/public-inbox-httpd.pod b/Documentation/public-inbox-httpd.pod index c1ebfd82..3ed48adc 100644 --- a/Documentation/public-inbox-httpd.pod +++ b/Documentation/public-inbox-httpd.pod @@ -4,7 +4,7 @@ public-inbox-httpd - PSGI server optimized for public-inbox =head1 SYNOPSIS -B<public-inbox-httpd> [OPTIONS] [/path/to/myapp.psgi] + public-inbox-httpd [OPTIONS] [/path/to/myapp.psgi] =head1 DESCRIPTION @@ -15,22 +15,34 @@ the PSGI file. If a PSGI file is not specified, L<PublicInbox::WWW> is loaded with a default middleware stack consisting of -L<Plack::Middleware::Deflater>, L<Plack::Middleware::ReverseProxy>, and L<Plack::Middleware::Head> This may point to a PSGI file for supporting generic PSGI apps. +=head1 ENVIRONMENT + +=over 8 + +=item GIT_HTTP_MAX_REQUEST_BUFFER + +Shared with L<git-http-backend(1)>, this governs the maximum upload +size of an HTTP request. + +Default: 10m + +=back + =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-imapd.pod b/Documentation/public-inbox-imapd.pod new file mode 100644 index 00000000..85bf3651 --- /dev/null +++ b/Documentation/public-inbox-imapd.pod @@ -0,0 +1,93 @@ +=head1 NAME + +public-inbox-imapd - IMAP server for sharing public-inboxes + +=head1 SYNOPSIS + + public-inbox-imapd [OPTIONS] + +=head1 DESCRIPTION + +public-inbox-imapd provides a read-only IMAP daemon for +public-inbox. It uses options and environment variables common +to all L<public-inbox-daemon(8)> implementations. + +Like L<public-inbox-nntpd(1)> and L<public-inbox-httpd(1)>, +C<public-inbox-imapd> will never require write access +to the directory where the public-inboxes are stored, so it +may be run as a different user than the user running +L<public-inbox-watch(1)>, L<public-inbox-mda(1)>, or +L<git-fetch(1)>. + +=head1 OPTIONS + +See common options in L<public-inbox-daemon(8)/OPTIONS>. +Additionally, IMAP-specific behavior for certain options +are supported and documented below. + +=over + +=item -l PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item --listen PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +In addition to the normal C<-l>/C<--listen> switch described in +L<public-inbox-daemon(8)>, the C<PROTOCOL> prefix (e.g. C<imap://> or +C<imaps://>) may be specified to force a given protocol. + +For STARTTLS and IMAPS support, the C<cert> and C<key> may be specified +on a per-listener basis after a C<?> character and separated by C<,>. +These directives are per-directive, and it's possible to use a different +cert for every listener. + +=item --cert /path/to/cert + +The default TLS certificate for optional STARTTLS and IMAPS support +if the C<cert> option is not given with C<--listen>. + +If using systemd-compatible socket activation and a TCP listener on port +993 is inherited, it is automatically IMAPS when this option is given. +When a listener on port 143 is inherited and this option is given, it +automatically gets STARTTLS support. + +=item --key /path/to/key + +The default private TLS certificate key for optional STARTTLS and IMAPS +support if the C<key> option is not given with C<--listen>. The private +key may be concatenated into the path used by C<--cert>, in which case this +option is not needed. + +=back + +=head1 CONFIGURATION + +C<public-inbox-imapd> uses the same configuration knobs +as L<public-inbox-nntpd(1)>, see L<public-inbox-nntpd(1)> +and L<public-inbox-config(5)>. + +=over 8 + +=item publicinbox.<name>.newsgroup + +The newsgroup name maps to an IMAP folder name. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/>, and +L<nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta>, +L<nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta> + +=head1 COPYRIGHT + +Copyright 2020-2021 all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<git(1)>, L<git-config(1)>, L<public-inbox-daemon(8)>, +L<public-inbox-config(5)>, L<public-inbox-nntpd(1)> diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod index 7c04f753..f1a2180a 100644 --- a/Documentation/public-inbox-index.pod +++ b/Documentation/public-inbox-index.pod @@ -4,15 +4,17 @@ public-inbox-index - create and update search indices =head1 SYNOPSIS -public-inbox-index [OPTIONS] INBOX_DIR +public-inbox-index [OPTIONS] INBOX_DIR... + +public-inbox-index [OPTIONS] --all =head1 DESCRIPTION public-inbox-index creates and updates the search, overview and NNTP article number database used by the read-only public-inbox HTTP and NNTP interfaces. Currently, this requires -L<DBD::SQLite> and L<DBI> Perl modules. L<Search::Xapian> -is optional, only to support the PSGI search interface. +L<DBD::SQLite> and L<DBI> Perl modules. L<Xapian> (or L<Search::Xapian>) +are optional, only to support the PSGI search interface. Once the initial indices are created by public-inbox-index, L<public-inbox-mda(1)> and L<public-inbox-watch(1)> will @@ -32,16 +34,79 @@ normal search functionality. =over +=item -j JOBS + +=item --jobs=JOBS + +Influences the number of Xapian indexing shards in a +(L<public-inbox-v2-format(5)>) inbox. + +See L<public-inbox-init(1)/--jobs> for a full description +of sharding. + +C<--jobs=0> is accepted as of public-inbox 1.6.0 +to disable parallel indexing regardless of the number of +pre-existing shards. + +If the inbox has not been indexed or initialized, C<JOBS - 1> +shards will be created (one job is always needed for indexing +the overview and article number mapping). + +Default: the number of existing Xapian shards + +=item -c + +=item --compact + +Compacts the Xapian DBs after indexing. This is recommended +when using C<--reindex> to avoid running out of disk space +while indexing multiple inboxes. + +While option takes a negligible amount of time compared to +C<--reindex>, it requires temporarily duplicating the entire +contents of the Xapian DB. + +This switch may be specified twice, in which case compaction +happens both before and after indexing to minimize the temporal +footprint of the (re)indexing operation. + +Available since public-inbox 1.4.0. + =item --reindex -Forces a search engine re-index of all messages in the -repository. This can be used for in-place upgrades while +Forces a re-index of all messages in the inbox. +This can be used for in-place upgrades and bugfixes while NNTP/HTTP server processes are utilizing the index. Keep in mind this roughly doubles the size of the already-large -Xapian database. Running L<public-inbox-compact(1)> -afterwards is recommended to release free space. +Xapian database. Using this with C<--compact> or running +L<public-inbox-compact(1)> afterwards is recommended to +release free space. + +public-inbox protects writes to various indices with +L<flock(2)>, so it is safe to reindex (and rethread) while +L<public-inbox-watch(1)>, L<public-inbox-mda(1)> or +L<public-inbox-learn(1)> run. This does not touch the NNTP article number database. +It does not affect threading unless C<--rethread> is +used. + +=item --all + +Index all inboxes configured in ~/.public-inbox/config. +This is an alternative to specifying individual inboxes directories +on the command-line. + +=item --rethread + +Regenerate internal THREADID and message thread associations +when reindexing. + +This fixes some bugs in older versions of public-inbox. While +it is possible to use this without C<--reindex>, it makes little +sense to do so. + +Available in public-inbox 1.6.0+. =item --prune @@ -50,15 +115,187 @@ is detected. This is intended to be used in mirrors after running L<public-inbox-edit(1)> or L<public-inbox-purge(1)> to ensure data is expunged from mirrors. +Available since public-inbox 1.2.0. + +=item --max-size SIZE + +Sets or overrides L</publicinbox.indexMaxSize> on a +per-invocation basis. See L</publicinbox.indexMaxSize> +below. + +Available since public-inbox 1.5.0. + +=item --batch-size SIZE + +Sets or overrides L</publicinbox.indexBatchSize> on a +per-invocation basis. See L</publicinbox.indexBatchSize> +below. + +When using rotational storage but abundant RAM, using a large +value (e.g. C<500m>) with C<--sequential-shard> can +significantly speed up and reduce fragmentation during the +initial index and full C<--reindex> invocations (but not +incremental updates). + +Available in public-inbox 1.6.0+. + +=item --no-fsync + +Disables L<fsync(2)> and L<fdatasync(2)> operations on SQLite +and Xapian. This is only effective with Xapian 1.4+. This is +primarily intended for systems with low RAM and the small +(default) C<--batch-size=1m>. Users of large C<--batch-size> +may even find disabling L<fdatasync(2)> causes too much dirty +data to accumulate, resulting on latency spikes from writeback. + +Available in public-inbox 1.6.0+. + +=item --dangerous + +Speed up initial index by using in-place updates and denying support for +concurrent readers. This is only effective with Xapian 1.4+. + +Available in public-inbox 1.8.0+ + +=item --sequential-shard + +Sets or overrides L</publicinbox.indexSequentialShard> on a +per-invocation basis. See L</publicinbox.indexSequentialShard> +below. + +Available in public-inbox 1.6.0+. + +=item --skip-docdata + +Stop storing document data in Xapian on an existing inbox. + +See L<public-inbox-init(1)/--skip-docdata> for description and caveats. + +Available in public-inbox 1.6.0+. + +=item -E EXTINDEX + +=item --update-extindex=EXTINDEX + +Update the given external index (L<public-inbox-extindex-format(5)>. +Either the configured section name (e.g. C<all>) or a directory name +may be specified. + +Defaults to C<all> if C<[extindex "all"]> is configured, +otherwise no external indices are updated. + +May be specified multiple times in rare cases where multiple +external indices are configured. + +=item --no-update-extindex + +Do not update the C<all> external index by default. This negates +all uses of C<-E> / C<--update-extindex=> on the command-line. + +=item --no-multi-pack-index + +Disables writing the multi-pack-index when using L</--update-extindex>. +See L<public-inbox-extindex(1)/--no-multi-pack-index> for details. + +Available in public-inbox 2.0.0+ + +=item --since=DATESTRING + +=item --after=DATESTRING + +=item --until=DATESTRING + +=item --before=DATESTRING + +Passed directly to L<git-log(1)> to limit changes for C<--reindex> + =back =head1 FILES -For v1 (ssoma) repositories described in L<public-inbox-v1-format>. +For v1 (ssoma) repositories described in L<public-inbox-v1-format(5)>. All public-inbox-specific files are contained within the C<$GIT_DIR/public-inbox/> directory. -v2 repositories are described in L<public-inbox-v2-format>. +v2 inboxes are described in L<public-inbox-v2-format(5)>. + +=head1 CONFIGURATION + +=over 8 + +=item publicinbox.indexMaxSize + +Prevents indexing of messages larger than the specified size +value. A single suffix modifier of C<k>, C<m> or C<g> is +supported, thus the value of C<1m> to prevents indexing of +messages larger than one megabyte. + +This is useful for avoiding memory exhaustion in mirrors +via git. It does not prevent L<public-inbox-mda(1)> or +L<public-inbox-watch(1)> from importing (and indexing) +a message. + +This option is only available in public-inbox 1.5 or later. + +Default: none + +=item publicinbox.indexBatchSize + +Flushes changes to the filesystem and releases locks after +indexing the given number of bytes. The default value of C<1m> +(one megabyte) is low to minimize memory use and reduce +contention with parallel invocations of L<public-inbox-mda(1)>, +L<public-inbox-learn(1)>, and L<public-inbox-watch(1)>. + +Increase this value on powerful systems to improve throughput at +the expense of memory use. The reduction of lock granularity +may not be noticeable on fast systems. With SSDs, values above +C<4m> have little benefit. + +For L<public-inbox-v2-format(5)> inboxes, this value is +multiplied by the number of Xapian shards. Thus a typical v2 +inbox with 3 shards will flush every 3 megabytes by default +unless parallelism is disabled via C<--sequential-shard> +or C<--jobs=0>. + +This influences memory usage of Xapian, but it is not exact. +The actual memory used by Xapian and Perl has been observed +in excess of 10x this value. + +This option is available in public-inbox 1.6 or later. +public-inbox 1.5 and earlier used the current default, C<1m>. + +Default: 1m (one megabyte) + +=item publicinbox.indexSequentialShard + +For L<public-inbox-v2-format(5)> inboxes, setting this to C<true> +allows indexing Xapian shards in multiple passes. This speeds up +indexing on rotational storage with high seek latency by allowing +individual shards to fit into the kernel page cache. + +Using a higher-than-normal number of C<--jobs> with +L<public-inbox-init(1)> may be required to ensure individual +shards are small enough to fit into cache. + +Warning: interrupting C<public-inbox-index(1)> while this option +is in use may leave the search indices out-of-date with respect +to SQLite databases. WWW and IMAP users may notice incomplete +search results, but it is otherwise non-fatal. Using C<--reindex> +will bring everything back up-to-date. + +Available in public-inbox 1.6.0+. + +This is ignored on L<public-inbox-v1-format(5)> inboxes. + +Default: false, shards are indexed in parallel + +=item publicinbox.<name>.indexSequentialShard + +Identical to L</publicinbox.indexSequentialShard>, +but only affect the inbox matching E<lt>nameE<gt>. + +=back =head1 ENVIRONMENT @@ -74,31 +311,37 @@ The number of documents to update before committing changes to disk. This environment is handled directly by Xapian, refer to Xapian API documentation for more details. -Default: our indexing code flushes every megabyte of mail seen -to keep memory usage low. Setting this environment variable to -any positive value will switch to a document count-based -threshold in Xapian. +For public-inbox 1.6 and later, use C<publicinbox.indexBatchSize> +instead. + +Setting C<XAPIAN_FLUSH_THRESHOLD> or +C<publicinbox.indexBatchSize> for a large C<--reindex> may cause +L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and +L<public-inbox-watch(1)> tasks to wait long and unpredictable +periods of time during C<--reindex>. + +Default: none, uses C<publicinbox.indexBatchSize> =back =head1 UPGRADING -Occasionally, public-inbox will update it's schema version and +Occasionally, public-inbox will update its schema version and require a full index by running this command. =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> =head1 SEE ALSO -L<Search::Xapian>, L<DBD::SQLite> +L<Search::Xapian>, L<DBD::SQLite>, L<public-inbox-extindex-format(5)> diff --git a/Documentation/public-inbox-init.pod b/Documentation/public-inbox-init.pod index 4744da96..85c6c9e8 100644 --- a/Documentation/public-inbox-init.pod +++ b/Documentation/public-inbox-init.pod @@ -20,7 +20,9 @@ may be specified for inboxes with multiple addresses. =over -=item -V, --version FORMAT_VERSION +=item -V FORMAT_VERSION + +=item --version FORMAT_VERSION Specify C<2> here to use the scalable L<public-inbox-v2-format(5)> if you have L<DBD::SQLite> installed. @@ -31,6 +33,8 @@ L<DBD::SQLite>. Default: C<1> +=item -L <basic|medium|full> + =item --indexlevel <basic|medium|full> Controls the indexing level for L<public-inbox-index(1)> @@ -39,15 +43,92 @@ See L<public-inbox-config(5)> for more information. Default: C<full> -=item -S, --skip-epoch +=item --ng NEWSGROUP + +=item --newsgroup NEWSGROUP + +The NNTP group name for use with L<public-inbox-nntpd(8)>. This +may be any newsgroup name with hierarchies delimited by C<.>. +For example, the newsgroup for L<mailto:meta@public-inbox.org> +is: C<inbox.comp.mail.public-inbox.meta> + +This may be set after-the-fact via C<publicinbox.$NAME.newsgroup> +in the configuration file. See L<public-inbox-config(5)> for more +info. + +Available in public-inbox 1.6.0+. + +Default: none. + +=item -c KEY=VALUE + +Allow setting arbitrary configs as C<publicinbox.$NAME.$KEY>. +This is idempotent for the same C<VALUE>, but allows setting +multiple values for keys such as C<publicinbox.$NAME.url> and +C<publicinbox.$NAME.watch>. + +=item --skip-artnum + +This option allows archivists to publish incomplete archives +with only new mail while allowing NNTP article numbers +to be reserved for yet-to-be-archived old mail. + +This is mainly intended for users of C<--skip-epoch> (documented below) +but may be of use to L<public-inbox-v1-format(5)> users. + +There is no automatic way to use reserved NNTP article numbers +when old mail is found, yet. + +Available in public-inbox 1.6.0+. + +Default: unset, no NNTP article numbers are skipped + +=item -S + +=item --skip-epoch For C<-V2> (L<public-inbox-v2-format(5)>) inboxes only, this option allows archivists to publish incomplete archives with newer mail while allowing "0.git" (or "1.git" and so on) epochs to be added-after-the-fact (without affecting "git clone" followers). +Available since public-inbox 1.2.0. + Default: unset, no epochs are skipped +=item -j JOBS + +=item --jobs=JOBS + +Control the number of Xapian index shards in a +C<-V2> (L<public-inbox-v2-format(5)>) inbox. + +It can be useful to use a single shard (C<-j1>) for inboxes on +high-latency storage (e.g. rotational HDD) unless the system has +enough RAM to cache 5-10x the size of the git repository. + +Another approach for HDDs is to use the +L<public-inbox-index(1)/publicInbox.indexSequentialShard> option +and many shards, so each shard may fit into the kernel page +cache. Unfortunately, excessive shards slows down read-only +query performance. + +For fast storage, it is generally not useful to specify higher +values than the default due to the top-level producer process +being a bottleneck. + +Default: the number of online CPUs, up to 4 (3 shard workers, 1 producer) + +=item --skip-docdata + +Do not store document data in Xapian, reducing Xapian storage +overhead by around 1.5%. + +Warning: this option prevents rollbacks to public-inbox 1.5.0 +and earlier. + +Available in public-inbox 1.6.0+. + =back =head1 ENVIRONMENT @@ -62,21 +143,23 @@ Used to override the default C<~/.public-inbox/config> value. =head1 LIMITATIONS -This tool predates NNTP support in public-inbox and is missing -C<newsgroup> and many of the options documented in -L<public-inbox-config(5)>. See L<public-inbox-config(5)> for all the -options which may be applied to a given inbox. +Some of the options documented in L<public-inbox-config(5)> +require editing the config file. Old versions lack the +C<--ng>/C<--newsgroup> parameter + +See L<public-inbox-config(5)> for all the options which may be applied +to a given inbox. =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2019-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-learn.pod b/Documentation/public-inbox-learn.pod index addcbcb5..b08e4bc8 100644 --- a/Documentation/public-inbox-learn.pod +++ b/Documentation/public-inbox-learn.pod @@ -4,7 +4,7 @@ public-inbox-learn - spam trainer and remover for public-inbox =head1 SYNOPSIS -B<public-inbox-learn> <spam|ham|rm> E<lt>MESSAGE + public-inbox-learn <spam|ham|rm> </path/to/RFC2822_message =head1 DESCRIPTION @@ -50,8 +50,13 @@ C<publicinboxmda.spamcheck> is C<none> in L<public-inbox-config(5)>. =item rm -This is identical to the C<spam> command above, but does -not feed the message to L<spamc(1)> +This is similar to the C<spam> command above, but does +not feed the message to L<spamc(1)> and only removes messages +which match on any of the C<To:>, C<Cc:>, and C<List-ID:> headers. + +The C<--all> option may be used to match C<spam> semantics in removing +the message from all configured inboxes. C<--all> is only +available in public-inbox 1.6.0+. =back @@ -68,16 +73,35 @@ Default: ~/.public-inbox/config =back +=head1 CONFIGURATION + +These configuration knobs should be used in the +L<public-inbox-config(5)> file. + +=over 8 + +=item publicinboxImport.dropUniqueUnsubscribe + +=item publicinbox.<name>.address + +=item publicinbox.<name>.listid + +=item publicinboxmda.spamcheck + +See L<public-inbox-config(5)> for descriptions of these options + +=back + =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-mda.pod b/Documentation/public-inbox-mda.pod index 99c9053d..edc90287 100644 --- a/Documentation/public-inbox-mda.pod +++ b/Documentation/public-inbox-mda.pod @@ -4,7 +4,7 @@ public-inbox-mda - mail delivery agent for public-inbox =head1 SYNOPSIS -B<public-inbox-mda> E<lt>MESSAGE + public-inbox-mda </path/to/RFC2822_message =head1 DESCRIPTION @@ -68,17 +68,33 @@ Default: ~/.public-inbox/emergency/ =back +=head1 CONFIGURATION + +Various configuration knobs should be used in the +L<public-inbox-config(5)> file. + +=over 8 + +=item publicinboxImport.dropUniqueUnsubscribe + +=item publicinbox.<name>.address + +=item publicinbox.<name>.listid + +See L<public-inbox-config(5)> for descriptions of these options + +=back =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-netd.pod b/Documentation/public-inbox-netd.pod new file mode 100644 index 00000000..71425e3c --- /dev/null +++ b/Documentation/public-inbox-netd.pod @@ -0,0 +1,97 @@ +=head1 NAME + +public-inbox-netd - read-only network daemon for sharing public-inboxes + +=head1 SYNOPSIS + + public-inbox-netd [OPTIONS] + +=head1 DESCRIPTION + +public-inbox-netd provides a read-only multi-protocol +(HTTP/IMAP/NNTP/POP3) daemon for public-inbox. It uses options +and environment variables common to all +L<public-inbox-daemon(8)> implementations. + +The default configuration will never require write access +to the directory where the public-inbox is stored, so it +may be run as a different user than the user running +L<public-inbox-watch(1)>, L<public-inbox-mda(1)>, or +L<git-fetch(1)>. + +=head1 OPTIONS + +See common options in L<public-inbox-daemon(8)/OPTIONS>. + +=over + +=item -l PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item --listen PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item -l http://ADDRESS/?env.PI_CONFIG=/path/to/cfg,psgi=/path/to/app.psgi + +In addition to the normal C<-l>/C<--listen> switch described in +L<public-inbox-daemon(8)>, the protocol prefix (e.g. C<nntp://> or +C<nntps://>) may be specified to force a given protocol. + +Environment variable overrides in effect during loading and +reloading (SIGHUP) can be specified as C<env.NAME=VALUE> for +all protocols. + +HTTP(S) listeners may also specify C<psgi=> to use a different +C<.psgi> file for each listener. + +C<err=/path/to/errors.log> may be used to isolate error/debug output +for a particular listener away from C<--stderr>. + +Non-HTTP(S) listeners may also specify C<out=> for logging to +C<stdout>. HTTP(S) users are encouraged to configure +L<Plack::Middleware::AccessLog> or +L<Plack::Middleware::AccessLog::Timed>, instead. + +=item --cert /path/to/cert + +See L<public-inbox-daemon(1)>. + +=item --key /path/to/key + +See L<public-inbox-daemon(1)>. + +=back + +=head1 CONFIGURATION + +These configuration knobs should be used in the +L<public-inbox-config(5)>. + +=over 8 + +=item publicinbox.<name>.newsgroup + +=item publicinbox.nntpserver + +=item publicinbox.pop3state + +=back + +See L<public-inbox-config(5)> for documentation on them. + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/>, and +L<nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta>, +L<nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<git(1)>, L<git-config(1)>, L<public-inbox-daemon(8)>, +L<public-inbox-config(5)> diff --git a/Documentation/public-inbox-nntpd.pod b/Documentation/public-inbox-nntpd.pod index 122fabea..59111f92 100644 --- a/Documentation/public-inbox-nntpd.pod +++ b/Documentation/public-inbox-nntpd.pod @@ -4,7 +4,7 @@ public-inbox-nntpd - NNTP server for sharing public-inbox =head1 SYNOPSIS -B<public-inbox-nntpd> [OPTIONS] + public-inbox-nntpd [OPTIONS] =head1 DESCRIPTION @@ -26,7 +26,9 @@ are supported and documented below. =over -=item -l, --listen PROTO://ADDRESS/?cert=/path/to/cert,key=/path/to/key +=item -l PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item --listen PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key In addition to the normal C<-l>/C<--listen> switch described in L<public-inbox-daemon(8)>, the protocol prefix (e.g. C<nntp://> or @@ -51,7 +53,7 @@ automatically gets STARTTLS support. The default private TLS certificate key for optional STARTTLS and NNTPS support if the C<key> option is not given with C<--listen>. The private -key may concatenated into the path used by C<--cert>, in which case this +key may be concatenated into the path used by C<--cert>, in which case this option is not needed. =back @@ -75,13 +77,13 @@ See L<public-inbox-config(5)> for documentation on them. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/>, +The mail archives are hosted at L<https://public-inbox.org/meta/>, and L<nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta>, -L<nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta> +L<nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta> =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2013-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-overview.pod b/Documentation/public-inbox-overview.pod index 583aea87..35917ccc 100644 --- a/Documentation/public-inbox-overview.pod +++ b/Documentation/public-inbox-overview.pod @@ -48,7 +48,7 @@ that inbox. The instructions are roughly: # Optional but strongly recommended for hosting HTTP # (and required for NNTP) - # enable overview (requires DBD::SQLite) and, if Search::Xapian is + # enable overview (requires DBD::SQLite) and, if Xapian is # available, search: public-inbox-index INBOX_DIR @@ -70,7 +70,7 @@ your mirror to other readers. Mirroring mailing lists may be done by any reader of a mailing list using L<public-inbox-watch(1)>. - # This will create a new git repository: + # This will create a new v2 inbox: public-inbox-init -V2 NAME INBOX_DIR MY_URL LIST_ADDRESS Then, see the L<public-inbox-watch(1)> manual for configuring @@ -109,8 +109,7 @@ See L<public-inbox-httpd(1)> and L<public-inbox-nntpd(1)> for more information on using these daemons. Hosting a public-inbox over HTTP or NNTP will never require -write access to any files in the git repository, including -the search indices or article number map database. +write access to any files in the inbox directory. Users familiar with PSGI and L<Plack> may also use L<PublicInbox::WWW> with the preferred server instead of @@ -120,11 +119,11 @@ L<public-inbox-httpd(1)> Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2016-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-pop3d.pod b/Documentation/public-inbox-pop3d.pod new file mode 100644 index 00000000..fb16fb96 --- /dev/null +++ b/Documentation/public-inbox-pop3d.pod @@ -0,0 +1,122 @@ +=head1 NAME + +public-inbox-pop3d - POP3 server for sharing public-inboxes + +=head1 SYNOPSIS + + public-inbox-pop3d [OPTIONS] + +=head1 DESCRIPTION + +public-inbox-pop3d provides a POP3 daemon for public-inbox. +It uses options and environment variables common to all +read-only L<public-inbox-daemon(8)> implementations, +but requires additional read-write storage to keep track +of deleted messages on a per-user basis. + +Like L<public-inbox-imapd(1)>, C<public-inbox-pop3d> will never +require write access to the directory where the public-inboxes +are stored. + +It is designed for anonymous access, thus the password is +always C<anonymous> (all lower-case). + +Usernames are of the format: + + C<$UUID@$NEWSGROUP_NAME> + +Where C<$UUID> is the output of the L<uuidgen(1)> command. Dash +(C<->) characters in UUIDs are ignored, and C<[A-F]> hex +characters are case-insensitive. Users should keep their UUIDs +private to prevent others from deleting unretrieved messages. +Users may switch to a new UUID at any time to retrieve +previously-retrieved messages. + +Historical slices of 50K messages are available +by suffixing the integer L<$SLICE>, where C<0> is the oldest. + + C<$UUID@$NEWSGROUP_NAME.$SLICE> + +It may be run as a different user than the user running +L<public-inbox-watch(1)>, L<public-inbox-mda(1)>, or +L<public-inbox-fetch(1)>. + +To save storage, L</publicinbox.pop3state> only stores +the highest-numbered deleted message + +=head1 OPTIONS + +See common options in L<public-inbox-daemon(8)/OPTIONS>. + +=over + +=item -l PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +=item --listen PROTOCOL://ADDRESS/?cert=/path/to/cert,key=/path/to/key + +In addition to the normal C<-l>/C<--listen> switch described in +L<public-inbox-daemon(8)>, the C<PROTOCOL> prefix (e.g. C<pop3://> or +C<pop3s://>) may be specified to force a given protocol. + +For STARTTLS and POP3S support, the C<cert> and C<key> may be specified +on a per-listener basis after a C<?> character and separated by C<,>. +These directives are per-directive, and it's possible to use a different +cert for every listener. + +=item --cert /path/to/cert + +The default TLS certificate for optional STARTTLS and POP3S support +if the C<cert> option is not given with C<--listen>. + +If using systemd-compatible socket activation and a TCP listener on port +995 is inherited, it is automatically POP3S when this option is given. +When a listener on port 110 is inherited and this option is given, it +automatically gets STARTTLS support. + +=item --key /path/to/key + +The default private TLS certificate key for optional STARTTLS and POP3S +support if the C<key> option is not given with C<--listen>. The private +key may be concatenated into the path used by C<--cert>, in which case this +option is not needed. + +=back + +=head1 CONFIGURATION + +Aside from C<publicinbox.pop3state>, C<public-inbox-pop3d> uses the +same configuration knobs as L<public-inbox-nntpd(1)>, +see L<public-inbox-nntpd(1)> and L<public-inbox-config(5)>. + +=over 8 + +=item publicInbox.pop3state + +A directory containing per-user/mailbox account information; +must be writable to the C<public-inbox-pop3d> process. + +=item publicInbox.<name>.newsgroup + +The newsgroup name maps to a POP3 folder name. + +=back + +=head1 CONTACT + +Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> + +The mail archives are hosted at L<https://public-inbox.org/meta/>, and +L<nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta>, +L<nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta> + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> + +=head1 SEE ALSO + +L<git(1)>, L<git-config(1)>, L<public-inbox-daemon(8)>, +L<public-inbox-config(5)>, L<public-inbox-nntpd(1)>, +L<uuidgen(1)> diff --git a/Documentation/public-inbox-purge.pod b/Documentation/public-inbox-purge.pod index 56a3f95a..1223b577 100644 --- a/Documentation/public-inbox-purge.pod +++ b/Documentation/public-inbox-purge.pod @@ -31,7 +31,7 @@ leads to discontiguous git history. =item --all Purge the message in all inboxes configured in ~/.public-inbox/config. -This is an alternative to specifying individual inboxes directories +This is an alternative to specifying individual inbox directories on the command-line. =back @@ -51,16 +51,30 @@ See L<public-inbox-config(5)> Only L<public-inbox-v2-format(5)> inboxes are supported. +This is safe to run while normal inbox writing tools +(L<public-inbox-mda(1)>, L<public-inbox-watch(1)>, +L<public-inbox-learn(1)>) are active. + +Running this in parallel with L<public-inbox-xcpdb(1)> or +C<"public-inbox-index --reindex"> can lead to errors or +purged data remaining indexed. + +Incremental L<public-inbox-index(1)> (without C<--reindex>) +is fine. + +Keep in mind this is a last resort, as it will be disruptive +to anyone using L<git(1)> to mirror the inbox being purged. + =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-tuning.pod b/Documentation/public-inbox-tuning.pod new file mode 100644 index 00000000..b56c2b10 --- /dev/null +++ b/Documentation/public-inbox-tuning.pod @@ -0,0 +1,223 @@ +=head1 NAME + +public-inbox-tuning - tuning public-inbox + +=head1 DESCRIPTION + +public-inbox intends to support a wide variety of hardware. While +we strive to provide the best out-of-the-box performance possible, +tuning knobs are an unfortunate necessity in some cases. + +=over 4 + +=item 1 + +New inboxes: public-inbox-init -V2 + +=item 2 + +Optional Inline::C use + +=item 3 + +Performance on rotational hard disk drives + +=item 4 + +Btrfs (and possibly other copy-on-write filesystems) + +=item 5 + +Performance on solid state drives + +=item 6 + +Read-only daemons + +=item 7 + +Other OS tuning knobs + +=item 8 + +Scalability to many inboxes + +=item 9 + +public-inbox-cindex --join performance + +=item 10 + +public-inbox-clone with shared object stores + +=back + +=head2 New inboxes: public-inbox-init -V2 + +If you're starting a new inbox (and not mirroring an existing one), +the L<-V2|public-inbox-v2-format(5)> requires L<DBD::SQLite>, but is +orders of magnitude more scalable than the original C<-V1> format. + +=head2 Optional Inline::C use + +Our optional use of L<Inline::C> speeds up subprocess spawning from +large daemon processes. + +To enable L<Inline::C>, either set the C<PERL_INLINE_DIRECTORY> +environment variable to point to a writable directory, or create +C<~/.cache/public-inbox/inline-c> for any user(s) running +public-inbox processes. + +If libgit2 development files are installed and L<Inline::C> +is enabled (described above), per-inbox C<git cat-file --batch> +processes are replaced with a single L<perl(1)> process running +C<PublicInbox::Gcf2::loop> in read-only daemons. libgit2 use +will be available in public-inbox 1.7.0+ + +More (optional) L<Inline::C> use will be introduced in the future +to lower memory use and improve scalability. + +Note: L<Inline::C> is required for L<lei(1)>, but not public-inbox-* + +=head2 Performance on rotational hard disk drives + +Random I/O performance is poor on rotational HDDs. Xapian indexing +performance degrades significantly as DBs grow larger than available +RAM. Attempts to parallelize random I/O on HDDs leads to pathological +slowdowns as inboxes grow. + +While C<-V2> introduced Xapian shards as a parallelization +mechanism for SSDs, enabling C<publicInbox.indexSequentialShard> +repurposes sharding as a mechanism to reduce the kernel page cache +footprint when indexing on HDDs. + +Initializing a mirror with a high C<--jobs> count to create more +shards (in C<-V2> inboxes) will keep each shard smaller and +reduce its kernel page cache footprint. Keep in mind excessive +sharding imposes a performance penalty for read-only queries. + +Users with large amounts of RAM are advised to set a large value +for C<publicinbox.indexBatchSize> as documented in +L<public-inbox-index(1)>. + +C<dm-crypt> users on Linux 4.0+ are advised to try the +C<--perf-same_cpu_crypt> C<--perf-submit_from_crypt_cpus> +switches of L<cryptsetup(8)> to reduce I/O contention from +kernel workqueue threads. + +=head2 Btrfs (and possibly other copy-on-write filesystems) + +L<btrfs(5)> performance degrades from fragmentation when using +large databases and random writes. The Xapian + SQLite indices +used by public-inbox are no exception to that. + +public-inbox 1.6.0+ disables copy-on-write (CoW) on Xapian and SQLite +indices on btrfs to achieve acceptable performance (even on SSD). +Disabling copy-on-write also disables checksumming, thus C<raid1> +(or higher) configurations may be corrupt after unsafe shutdowns. + +Fortunately, these SQLite and Xapian indices are designed to be +recoverable from git if missing. + +Disabling CoW does not prevent all fragmentation. Large values +of C<publicInbox.indexBatchSize> also limit fragmentation during +the initial index. + +Avoid snapshotting subvolumes containing Xapian and/or SQLite indices. +Snapshots use CoW despite our efforts to disable it, resulting +in fragmentation. + +L<filefrag(8)> can be used to monitor fragmentation, and +C<btrfs filesystem defragment -fr $INBOX_DIR> may be necessary. + +Large filesystems benefit significantly from the C<space_cache=v2> +mount option documented in L<btrfs(5)>. + +Older, non-CoW filesystems generally work well out of the box +for our Xapian and SQLite indices. + +=head2 Performance on solid state drives + +While SSD read performance is generally good, SSD write performance +degrades as the drive ages and/or gets full. Issuing C<TRIM> commands +via L<fstrim(8)> or similar is required to sustain write performance. + +Users of the Flash-Friendly File System +L<F2FS|https://en.wikipedia.org/wiki/F2FS> may benefit from +optimizations found in SQLite 3.21.0+. Benchmarks are greatly +appreciated. + +=head2 Read-only daemons + +L<public-inbox-httpd(1)>, L<public-inbox-imapd(1)>, and +L<public-inbox-nntpd(1)> are all designed for C10K (or higher) +levels of concurrency from a single process. SMP systems may +use C<--worker-processes=NUM> as documented in L<public-inbox-daemon(8)> +for parallelism. + +The open file descriptor limit (C<RLIMIT_NOFILE>, C<ulimit -n> in L<sh(1)>, +C<LimitNOFILE=> in L<systemd.exec(5)>) may need to be raised to +accommodate many concurrent clients. + +Transport Layer Security (IMAPS, NNTPS, or via STARTTLS) significantly +increases memory use of client sockets, be sure to account for that in +capacity planning. + +Bursts of small object allocations late in process life contribute to +fragmentation of the heap due to arenas (slabs) used internally by Perl. +glibc malloc users should use C<MALLOC_MMAP_THRESHOLD_=131072> to reduce +fragmentation from the sliding mmap window. On 64-bit systems, jemalloc +(tested as an LD_PRELOAD on GNU/Linux) reduces fragmentation at the +expense of VM space. 32-bit systems may be better off sticking with +glibc and MALLOC_MMAP_THRESHOLD_. + +=head2 Other OS tuning knobs + +Linux users: the C<sys.vm.max_map_count> sysctl may need to be increased if +handling thousands of inboxes (with L<public-inbox-extindex(1)>) to avoid +out-of-memory errors from git. + +Other OSes may have similar tuning knobs (patches appreciated). + +=head2 Scalability to many inboxes + +L<public-inbox-extindex(1)> allows any number of public-inboxes +to share the same Xapian indices. + +git 2.33+ startup time is orders of magnitude faster and uses +less memory when dealing with thousands of alternates required +for thousands of inboxes with L<public-inbox-extindex(1)>. + +Frequent packing (via L<git-gc(1)>) both improves performance +and reduces the need to increase C<sys.vm.max_map_count>. + +=head2 public-inbox-cindex --join performance + +A C++ compiler and the Xapian development files makes C<--join> or +C<--join=aggressive> orders of magnitude faster in L<public-inbox-cindex(1)>. +On Debian-based systems this is C<libxapian-dev>. RPM-based distros have +these in C<xapian-core-devel> or C<xapian14-core-libs>. *BSDs typically +package development files together with runtime libraries, so the C<xapian> +or C<xapian-core> package will already have the development files. + +=head2 public-inbox-clone with shared object stores + +When mirroring manifests with many forks using the same objstore, +git 2.41+ is highly recommended for performance as we automatically +use the C<fetch.hideRefs> feature to speed up negotiation. + +=head1 CONTACT + +Feedback encouraged via plain-text mail to L<mailto:meta@public-inbox.org> + +Information for *BSDs and non-traditional filesystems especially +welcome. + +Our archives are hosted at L<https://public-inbox.org/meta/>, +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>, and other places + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-v1-format.pod b/Documentation/public-inbox-v1-format.pod index e5b1dd06..db223fd9 100644 --- a/Documentation/public-inbox-v1-format.pod +++ b/Documentation/public-inbox-v1-format.pod @@ -2,7 +2,7 @@ =head1 NAME -public-inbox v1 git repository and tree description (aka "ssoma") +public-inbox-v1-format - git repository and tree description (aka "ssoma") =head1 DESCRIPTION @@ -175,7 +175,7 @@ This is up to the administrators. =head1 COPYRIGHT -Copyright 2013-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2013-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-v2-format.pod b/Documentation/public-inbox-v2-format.pod index 730f6633..de3b0bfd 100644 --- a/Documentation/public-inbox-v2-format.pod +++ b/Documentation/public-inbox-v2-format.pod @@ -2,7 +2,7 @@ =head1 NAME -public-inbox v2 repository description +public-inbox-v2-format - structure of public inbox v2 archives =head1 DESCRIPTION @@ -30,7 +30,7 @@ databases for parallelism by "shards". - all.git # empty, alternates to $EPOCH.git - xap$SCHEMA_VERSION/$SHARD # per-shard Xapian DB - xap$SCHEMA_VERSION/over.sqlite3 # OVER-view DB for NNTP, threading - - msgmap.sqlite3 # same the v1 msgmap + - msgmap.sqlite3 # same as the v1 msgmap For blob lookups, the reader only needs to open the "all.git" repository with $GIT_DIR/objects/info/alternates which references @@ -89,7 +89,7 @@ After-the-fact invocations of L<public-inbox-index> will ignore messages written to 'd' after they are written to 'm'. Deltafication is not significantly improved over v1, but overall -storage for trees is made as as small as possible. Initial +storage for trees is made as small as possible. Initial statistics and benchmarks showing the benefits of this approach are documented at: @@ -97,7 +97,7 @@ L<https://public-inbox.org/meta/20180209205140.GA11047@dcvr/> =head2 XAPIAN SHARDS -Another second scalability problem in v1 was the inability to +Another scalability problem in v1 was the inability to utilize multiple CPU cores for Xapian indexing. This is addressed by using shards in Xapian to perform import indexing in parallel. @@ -113,9 +113,14 @@ improved with high-quality and high-quantity solid-state storage. Issuing TRIM commands with L<fstrim(8)> was necessary to maintain consistent performance while developing this feature. -Rotational storage devices are NOT recommended for indexing of -large mail archives; but are fine for backup and usable for -small instances. +Rotational storage devices perform significantly worse than +solid state storage for indexing of large mail archives; but are +fine for backup and usable for small instances. + +As of public-inbox 1.6.0, the C<publicInbox.indexSequentialShard> +option of L<public-inbox-index(1)> may be used with a high shard +count to ensure individual shards fit into page cache when the entire +Xapian DB cannot. Our use of the L</OVERVIEW DB> requires Xapian document IDs to remain stable. Using L<public-inbox-compact(1)> and @@ -133,7 +138,7 @@ OVER/XOVER commands). The overview DB maintains all the header information necessary to implement the NNTP OVER/XOVER commands and non-search -endpoints of of the PSGI UI. +endpoints of the PSGI UI. Xapian has become completely optional for v2 (as it is for v1), but SQLite remains required for v2. SQLite turns out to be powerful @@ -159,7 +164,7 @@ top-level of the directory. =head1 OBJECT IDENTIFIERS -There are three distinct type of identifiers. content_id is the +There are three distinct type of identifiers. content_hash is the new one for v2 and should make message removal and deduplication easier. object_id and Message-ID are already known. @@ -179,11 +184,11 @@ The email header; duplicates allowed for archival purposes. This remains a searchable field in Xapian. Note: it's possible for emails to have multiple Message-ID headers (and L<git-send-email(1)> had that bug for a bit); so we take all of them into account. -In case of conflicts detected by content_id below, we generate a new -Message-ID based on content_id; if the generated Message-ID still +In case of conflicts detected by content_hash below, we generate a new +Message-ID based on content_hash; if the generated Message-ID still conflicts, a random one is generated. -=item content_id +=item content_hash A hash of relevant headers and raw body content for purging of unwanted content. This is not stored anywhere, @@ -193,7 +198,7 @@ For now, the relevant headers are: Subject, From, Date, References, In-Reply-To, To, Cc -Received, List-Id, and similar headers are NOT part of content_id as +Received, List-Id, and similar headers are NOT part of content_hash as they differ across lists and we will want removal to be able to cross lists. @@ -203,7 +208,7 @@ raw body risks being broken by list signatures; but we can use filters (e.g. PublicInbox::Filter::Vger) to clean the body for imports. -content_id is SHA-256 for now; but can be changed at any time +content_hash is SHA-256 for now; but can be changed at any time without making DB changes. =back @@ -226,11 +231,11 @@ no sense in a public archive. =head1 THANKS Thanks to the Linux Foundation for sponsoring the development -and testing of the v2 repository format. +and testing of the v2 format. =head1 COPYRIGHT -Copyright 2018-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2018-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-watch.pod b/Documentation/public-inbox-watch.pod index 87e4da49..6e2142fe 100644 --- a/Documentation/public-inbox-watch.pod +++ b/Documentation/public-inbox-watch.pod @@ -4,7 +4,7 @@ public-inbox-watch - mailbox watcher for public-inbox =head1 SYNOPSIS -B<public-inbox-watch> + public-inbox-watch In ~/.public-inbox/config: @@ -35,25 +35,24 @@ In ~/.public-inbox/config: =head1 DESCRIPTION -public-inbox-watch allows watching a mailbox (currently only -Maildir) for the arrival of new messages and automatically -importing them into a public-inbox (git) repository. +public-inbox-watch allows watching a mailbox or newsgroup +for the arrival of new messages and automatically +importing them into public-inbox git repositories and indices. public-inbox-watch is useful in situations when a user wishes to mirror an existing mailing list, but has no access to run L<public-inbox-mda(1)> on a server. Unlike public-inbox-mda -which is invoked once per-message, public-inbox-watch is a +which is invoked once per message, public-inbox-watch is a persistent process, making it faster for after-the-fact imports of large Maildirs. Upon startup, it scans the mailbox for new messages to be imported while it was not running. -Currently, only Maildirs are supported and the -L<Filesys::Notify::Simple> Perl module is required. - -For now, IMAP users should use tools such as L<mbsync(1)> -or L<offlineimap(1)> to bidirectionally sync their IMAP -folders to Maildirs for public-inbox-watch. +All versions of public-inbox-watch support Maildirs. public-inbox +1.6.0 added support for IMAP folders and NNTP newsgroups. +public-inbox 2.0 adds support for MH directories. There are no +plans to support the mbox family since new messages are expensive +to detect in large mboxes. public-inbox-watch should be run inside a L<screen(1)> session or as a L<systemd(1)> service. Errors are emitted to stderr. @@ -65,21 +64,109 @@ public-inbox-watch takes no command-line options. =head1 CONFIGURATION These configuration knobs should be used in the -L<public-inbox-config(5)> +L<public-inbox-config(5)> file. =over 8 +=item publicinboxImport.dropUniqueUnsubscribe + +See L<public-inbox-config(5)/publicinboxImport.dropUniqueUnsubscribe> + =item publicinbox.<name>.watch +A location to watch. public-inbox 1.5.0 and earlier only supported +C<maildir:> paths: + + [publicinbox "test"] + watch = maildir:/path/to/maildirs/.INBOX.test/ + +public-inbox 1.6.0+ supports C<nntp://>, C<nntps://>, +C<imap://> and C<imaps://> URLs: + + watch = nntp://news.example.com/inbox.test.group + watch = imaps://user@mail.example.com/INBOX.test + +2.0+ supports MH: + + watch = mh:/path/to/MH/inbox.test + +This may be specified multiple times to combine several mailboxes +into a single public-inbox. URLs requiring authentication +will require L<netrc(5)> and/or L<git-credential(1)> (preferred) to fill +in the username and password. + +public-inbox 2.0+ also supports boolean C<false> to prevent the global +L</publicinboxwatch.watchspam> directive from writing to the inbox. + +Default: none + =item publicinbox.<name>.watchheader + [publicinbox "test"] + watchheader = List-Id:<test.example.com> + +If specified, L<public-inbox-watch(1)> will only process mail +matching the given header. If specified multiple times in +public-inbox 1.5 or later, mail will be processed if it matches +any of the values. Only the last value was used in public-inbox +1.4 and earlier. + +Default: none + =item publicinboxwatch.spamcheck +This may be set to C<spamc> to enable the use of SpamAssassin +L<spamc(1)> for filtering spam before it is imported into git +history. Other spam filtering backends may be supported in +the future. + +Default: none + =item publicinboxwatch.watchspam -=back +A Maildir to watch for confirmed spam messages to appear in. +Messages which appear in this folder with the (S)een flag +will be hidden from all configured inboxes based on Message-ID +and content matching. + +Messages without the (S)een flag are not considered for hiding. +This hiding affects all configured public-inboxes in PI_CONFIG. + +As with C<publicinbox.$NAME.watch>, C<imap://> and C<imaps://> URLs +are supported in public-inbox 1.6.0+, and C<MH> in 2.0+. + +Default: none; only for L<public-inbox-watch(1)> users + +=item imap.Starttls / imap.$URL.Starttls + +Whether or not to use C<STARTTLS> on plain C<imap://> connections. + +May be specified for certain URLs via L<git-config(1)/--get-urlmatch> +in C<git(1)> 1.8.5+. + +Default: C<true> + +=item imap.Compress / imap.$URL.Compress -See L<public-inbox-config(5)> for documentation on them. +Whether or not to use the IMAP COMPRESS (RFC4978) extension to +save bandwidth. This is not supported by all IMAP servers and +some advertising this feature may not implement it correctly. + +May be specified only for certain URLs if L<git(1)> 1.8.5+ is +installed to use L<git-config(1)/--get-urlmatch> + +Default: C<false> + +=item nntp.Starttls / nntp.$URL.Starttls + +Whether or not to use C<STARTTLS> on plain C<nntp://> connections. + +May be specified for certain URLs via L<git-config(1)/--get-urlmatch> +in C<git(1)> 1.8.5+. + +Default: C<false> if the hostname is a Tor C<.onion>, C<true> otherwise + +=back =head1 SIGNALS @@ -94,6 +181,11 @@ Reload the config file (default: ~/.public-inbox/config) Rescan all watched mailboxes. This is done automatically after startup. +=item SIGQUIT / SIGTERM / SIGINT + +Gracefully shut down. In-flight messages will be stored +and indexed. + =back =head1 ENVIRONMENT @@ -117,12 +209,12 @@ daemons. See L<public-inbox-daemon(8)>. Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2016-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod index 149c8f78..e7c07ed3 100644 --- a/Documentation/public-inbox-xcpdb.pod +++ b/Documentation/public-inbox-xcpdb.pod @@ -4,20 +4,34 @@ public-inbox-xcpdb - upgrade Xapian DB formats =head1 SYNOPSIS - public-inbox-xcpdb [OPTIONS] INBOX_DIR +public-inbox-xcpdb [OPTIONS] INBOX_DIR + +public-inbox-xcpdb [OPTIONS] --all =head1 DESCRIPTION public-inbox-xcpdb is similar to L<copydatabase(1)> for upgrading to the latest database format supported by Xapian (e.g. "glass" or "honey"), but is designed to tolerate and -recover from Xapian database modifications from -L<public-inbox-watch(1)> or L<public-inbox-mda(1)>. +accept parallel Xapian database modifications from +L<public-inbox-watch(1)>, L<public-inbox-mda(1)>, +L<public-inbox-learn(1)>, and L<public-inbox-index(1)>. + +This command is rarely used, as Xapian DB formats rarely +change. =head1 OPTIONS =over +=item --all + +Copy all inboxes configured in ~/.public-inbox/config. +This is an alternative to specifying individual inboxes directories +on the command-line. + +=item -c + =item --compact In addition to performing the copy operation, run L<xapian-compact(1)> @@ -30,7 +44,9 @@ preferable for gigantic inboxes where the coarse-grained lock currently required for L<public-inbox-compact(1)> can cause the compaction to take hours at-a-time. -=item --reshard=N / -R N +=item -R N + +=item --reshard=N Reshard the Xapian database on a L<v2|public-inbox-v2-format(5)> inbox to C<N> shards . Since L<xapian-compact(1)> is not suitable @@ -40,11 +56,39 @@ existing Xapian database(s) to any positive value of C<N>. This is useful in case the Xapian DB was created with too few or too many shards given the capabilities of the current hardware. -=item --blocksize / --no-full / --fuller +=item --blocksize + +=item --no-full + +=item --fuller These options are passed directly to L<xapian-compact(1)> when used with C<--compact>. +=item --no-fsync + +Disable L<fsync(2)> and L<fdatasync(2)>. +See L<public-inbox-index(1)/--no-fsync> for caveats. + +Available in public-inbox 1.6.0+. + +=item --sequential-shard + +Copy each shard sequentially, ignoring C<--jobs>. This also +affects indexing done at the end of a run. + +=item --batch-size=BYTES + +=item --max-size=BYTES + +See L<public-inbox-index(1)> for a description of these options. + +These indexing options indexing at the end of a run. +C<public-inbox-xcpdb> may run in parallel with with +L<public-inbox-index(1)>, and C<public-inbox-xcpdb> needs to +reindex changes made to the old Xapian DBs by +L<public-inbox-index(1)> while it was running. + =back =head1 ENVIRONMENT @@ -74,16 +118,26 @@ used by public-inbox, NOT users upgrading public-inbox itself. In particular, it DOES NOT upgrade the schema used by the PSGI search interface (see L<public-inbox-index(1)>). +=head1 LIMITATIONS + +Do not use L<public-inbox-purge(1)> or L<public-inbox-edit(1)> +while this is running; old (purged or edited data) may show up. + +Normal invocations L<public-inbox-index(1)> can safely run +while this is running, too. However, reindexing via the +L<public-inbox-index(1)/--reindex> switch will be a waste of +computing resources. + =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright 2019-2021 all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> diff --git a/Documentation/public-inbox.cgi.pod b/Documentation/public-inbox.cgi.pod index 8fd6f3e7..58d59ba2 100644 --- a/Documentation/public-inbox.cgi.pod +++ b/Documentation/public-inbox.cgi.pod @@ -4,7 +4,7 @@ public-inbox.cgi - CGI wrapper for PublicInbox::WWW =head1 SYNOPSIS -You generally want to run public-inbox-httpd, instead +You generally want to run public-inbox-netd or public-inbox-httpd, instead =head1 DESCRIPTION @@ -12,23 +12,30 @@ public-inbox.cgi provides a CGI interface wrapper on top of the PSGI/Plack L<PublicInbox::WWW> module. It is only provided for compatibility reasons and NOT recommended. -CGI with Perl is slow due to code loading overhead and web servers lack -the scheduling fairness of L<public-inbox-httpd(1)> for handling git -clones and streaming large mbox downloads. +CGI with Perl is slow due to code loading overhead and +web servers lack the scheduling fairness of L<public-inbox-netd(1)> +and L<public-inbox-httpd(1)> for handling git clones and +streaming large mbox downloads. + +=head1 COMPATIBILITY NOTE + +When using the CGI with Apache, make sure to set AllowEncodedSlashes to On, as +public-inbox makes heavy use of encoded slashes. =head1 CONTACT Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org> -The mail archives are hosted at L<https://public-inbox.org/meta/> -and L<http://hjrcffqmbrq6wope.onion/meta/> +The mail archives are hosted at L<https://public-inbox.org/meta/> and +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2019-2020 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> =head1 SEE ALSO -L<public-inbox-httpd(1)>, L<PublicInbox::WWW>, L<public-inbox-daemon(8)>, +L<public-inbox-netd(1)>, L<public-inbox-httpd(1)>, +L<PublicInbox::WWW>, L<public-inbox-daemon(8)>, diff --git a/Documentation/reproducibility.txt b/Documentation/reproducibility.txt new file mode 100644 index 00000000..3336de73 --- /dev/null +++ b/Documentation/reproducibility.txt @@ -0,0 +1,29 @@ +reproducibility => forkability +------------------------------ + +The ability to fork a project is a checks and balances +system for free software projects. Reproducibility is key +to forkability since every mirror is potential fork. + +git makes the code history of projects fully reproducible. +public-inbox uses git to make the email history of projects +reproducible. + +Keeping all communications as email ensures the full history +of the entire project can be mirrored by anyone with the +resources to do so. Compact, low-complexity data requires +less resources to mirror, so sticking with plain text +ensures more parties can mirror and potentially fork the +project with all its data. + +Any private or irreproducible data is a barrier to forking. +These include mailing list subscriber information and +non-federated user identities. The "pull" subscriber model +of NNTP and Atom feeds combined with open-to-all posting +means there's no need for private data. + +If these things make power hungry project leaders and admins +uncomfortable, good. That was the point. It's how checks +and balances ought to work. + +Comments, corrections, etc. welcome: meta@public-inbox.org diff --git a/Documentation/standards.perl b/Documentation/standards.perl index 76c8c432..743cdee1 100755 --- a/Documentation/standards.perl +++ b/Documentation/standards.perl @@ -1,6 +1,6 @@ #!/usr/bin/perl -w -use strict; -# Copyright 2019-2020 all contributors <meta@public-inbox.org> +use v5.12; +# Copyright all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> print <<EOF; @@ -11,11 +11,11 @@ Non-exhaustive list of standards public-inbox software attempts or intends to implement. This list is intended to be a quick reference for hackers and users. -Given the goals of interoperability and accessibility; strict +Given the goals of interoperability and accessibility, strict conformance to standards is not always possible, but rather best-effort taking into account real-world cases. In particular, "obsolete" standards remain relevant as long as clients and -data exists. +data using them exist. IETF RFCs --------- @@ -25,6 +25,12 @@ EOF my $rfcs = [ 3977 => 'NNTP', 977 => 'NNTP (old)', + 1036 => 'Standard for Interchange of USENET Messages', + 5536 => 'Netnews Article Format', + 5537 => 'Netnews Architecture and Protocols', + 1738 => 'Uniform resource locators', + 5092 => 'IMAP URL scheme', + 5538 => 'NNTP URI schemes', 6048 => 'NNTP additions to LIST command (TODO)', 8054 => 'NNTP compression', 4642 => 'NNTP TLS', @@ -39,9 +45,34 @@ my $rfcs = [ 2616 => 'HTTP/1.1 (newer updates should apply, too)', 7230 => 'HTTP/1.1 message syntax and routing', 7231 => 'HTTP/1.1 semantics and content', - 2822 => 'Internet message format', - # TODO: flesh this out + 822 => 'Internet message format (1982)', + 2822 => 'Internet message format (2001)', + 5322 => 'Internet message format (2008)', + 3501 => 'IMAP4rev1', + 2177 => 'IMAP IDLE', + 2683 => 'IMAP4 Implementation Recommendations', + # 5032 = 'WITHIN search extension for IMAP', + 4978 => 'IMAP COMPRESS Extension', + # 5182 = 'IMAP Extension for Referencing the Last SEARCH Result', + # 5256 => 'IMAP SORT and THREAD extensions', + # 5738 => 'IMAP Support for UTF-8', + # 8474 => 'IMAP Extension for Object Identifiers', + + # 8620 => JSON Meta Application Protocol (JMAP) + # 8621 => JSON Meta Application Protocol (JMAP) for Mail + # ... + + # examples/unsubscribe.milter and PublicInbox::Unsubscribe + 2369 => 'URLs as Meta-Syntax for Core Mail List Commands', + 8058 => 'Signaling One-Click Functionality for List Email Headers', + + 1081 => 'Post Office Protocol – Version 3', + 1939 => 'Post Office Protocol – Version 3 (STD 53)', + 2449 => 'POP3 extension mechanism', + 2595 => 'STARTTLS for IMAP and POP3', + 2384 => 'POP URL Scheme', + # TODO: flesh this out ]; my @rfc_urls = qw(tools.ietf.org/html/rfc%d @@ -60,6 +91,10 @@ print <<'EOF' Other relevant documentation ---------------------------- +* IMAP capabilities registry and response codes: + https://www.iana.org/assignments/imap-capabilities + https://www.iana.org/assignments/imap-response-codes + * Documentation/technical/http-protocol.txt in git source code: https://public-inbox.org/git/9c5b6f0fac/s @@ -72,6 +107,6 @@ Other relevant documentation Copyright --------- -Copyright (C) 2019-2020 all contributors <meta@public-inbox.org> +Copyright (C) all contributors <meta@public-inbox.org> License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> EOF diff --git a/Documentation/technical/data_structures.txt b/Documentation/technical/data_structures.txt new file mode 100644 index 00000000..11f78041 --- /dev/null +++ b/Documentation/technical/data_structures.txt @@ -0,0 +1,229 @@ +Internal data structures of public-inbox + +This is a guide for hackers new to our code base. Do not +consider our internal data structures stable for external +consumers, this document should be updated when internals +change. I recommend reading this document from the source tree, +with the source code easily accessible if you need examples. + +This mainly documents in-memory data structures. If you're +interested in the stable on-filesystem formats, see the +public-inbox-config(5), public-inbox-v1-format(5) and +public-inbox-v2-format(5) manpages. + +Common abbreviations when used outside of their packages are +documented. `$self' is the common variable name when used +within their package. + +PublicInbox::Config +------------------- + +PublicInbox::Config is the root class which loads a +public-inbox-config file and instantiates PublicInbox::Inbox, +PublicInbox::WWW, PublicInbox::NNTPD, and other top-level +classes. + +Outside of tests, this is typically a singleton. + +Per-message classes +------------------- + +* PublicInbox::Eml - Email::MIME-like class + Common abbreviation: $mime, $eml + Used by: PublicInbox::WWW, PublicInbox::SearchIdx + + A representation of an entire email, multipart or not. + An option to use libgmime or libmailutils may be supported + in the future for performance and memory use. + + This can be a memory hog with big messages and giant + attachments, so our PublicInbox::WWW interface only keeps + one object of this class in memory at a time. + + In other words, this is the "meat" of the message, whereas + $smsg (below) is just the "skeleton". + + Our PublicInbox::V2Writable class may have two objects of this + type in memory at a time for deduplication. + + In public-inbox 1.4 and earlier, Email::MIME and its subclass, + PublicInbox::MIME were used. Despite still slurping, + PublicInbox::Eml is faster and uses less memory due to + lazy header parsing and lazy subpart instantiation with + shorter object lifetimes. + +* PublicInbox::Smsg - small message skeleton + Used by: PublicInbox::{NNTP,WWW,SearchIdx} + Common abbreviation: $smsg + + Represents headers shown in NNTP overview and PSGI message + summaries (thread skeleton). + + This is loaded from either the overview DB (over.sqlite3) or + the Xapian DB (docdata.glass), though the Xapian docdata + won't hold NNTP-only fields (Cc:/To:). + + There may be hundreds or thousands of these objects in memory + at a time, so fields are pruned if unneeded. + +* PublicInbox::SearchThread::Msg - subclass of Smsg + Common abbreviation: $cont or $node + Used by: PublicInbox::WWW + + The structure we use for a non-recursive[1] variant of + JWZ's algorithm: <https://www.jwz.org/doc/threading.html>. + Nowadays, this is a re-blessed $smsg with additional fields. + + As with $smsg objects, there may be hundreds or thousands + of these objects in memory at a time. + + We also do not use a linked list for storing children as JWZ + describes, but instead a Perl hashref for {children} which + becomes an arrayref upon sorting. + + [1] https://rt.cpan.org/Ticket/Display.html?id=116727 + +Per-inbox classes +----------------- + +* PublicInbox::Inbox - represents a single public-inbox + Common abbreviation: $ibx + Used everywhere. + + This represents a "publicinbox" section in the config + file, see public-inbox-config(5) for details. + +* PublicInbox::Git - represents a single git repository + Common abbreviation: $git, $ibx->git + Used everywhere. + + Each configured "publicinbox" or "coderepo" has one of these. + +* PublicInbox::Msgmap - msgmap.sqlite3 read-write interface + Common abbreviation: $mm, $ibx->mm + Used everywhere if SQLite is available. + + Each indexed inbox has one of these, see + public-inbox-v1-format(5) and public-inbox-v2-format(5) + manpages for details. + +* PublicInbox::Over - over.sqlite3 read-only interface + Common abbreviation: $over, $ibx->over + Used everywhere if SQLite is available. + + Each indexed inbox has one of these, see + public-inbox-v1-format(5) and public-inbox-v2-format(5) + manpages for details. + +* PublicInbox::Search - Xapian read-only interface + Common abbreviation: $srch, $ibx->search + Used everywhere if Xapian is available. + + Each indexed inbox has one of these, see + public-inbox-v1-format(5) and public-inbox-v2-format(5) + manpages for details. + +PublicInbox::WWW +---------------- + +The main PSGI web interface, uses several other packages to +form our web interface. + +PublicInbox::SolverGit +---------------------- + +This is instantiated from the $INBOX/$BLOB_OID/s/ WWW endpoint +and represents the stages and states for "solving" a blob by +searching for and applying patches. See the code and comments +in PublicInbox/SolverGit.pm + +PublicInbox::Qspawn +------------------- + +This is instantiated from various WWW endpoints and represents +the stages and states for running and managing subprocesses +in a way which won't exceed configured process limits defined +via "publicinboxlimiter.*" directives in public-inbox-config(5). + +ad-hoc structures shared across packages +---------------------------------------- + +* $ctx - PublicInbox::WWW app request context + This holds the PSGI $env as well as any internal variables + used by various modules of PublicInbox::WWW. + + As with the PSGI $env, there is one per active WWW + request+response cycle. It does not exist for idle HTTP + clients. + +daemon classes +-------------- + +* PublicInbox::NNTP - a NNTP client socket + Common abbreviation: $nntp + Used by: PublicInbox::DS, public-inbox-nntpd + + Unlike PublicInbox::HTTP, all of the NNTP client logic for + serving to NNTP clients is here, including what would be + in $ctx on the HTTP or WWW side. + + There may be thousands of these since we support thousands of + NNTP clients. + +* PublicInbox::HTTP - a HTTP client socket + Common abbreviation: $http + Used by: PublicInbox::DS, public-inbox-httpd + + Unlike PublicInbox::NNTP, this class has no knowledge of any of + the email- or git-specific parts of public-inbox, only PSGI. + However, it supports APIs and behaviors (e.g. streaming large + responses) which PublicInbox::WWW may take advantage of. + + There may be thousands of these since we support thousands of + HTTP clients. + +* PublicInbox::Listener - a SOCK_STREAM listen socket (TCP or Unix) + Used by: PublicInbox::DS, public-inbox-httpd, public-inbox-nntpd + Common abbreviation: @listeners in PublicInbox::Daemon + + This class calls non-blocking accept(2) or accept4(2) on a + listen socket to create new PublicInbox::HTTP and + PublicInbox::NNTP instances. + +* PublicInbox::HTTPD + Common abbreviation: $httpd + + Represents an HTTP daemon which creates PublicInbox::HTTP + wrappers around client sockets accepted from + PublicInbox::Listener. + + Since the SERVER_NAME and SERVER_PORT PSGI variables need to be + exposed for HTTP/1.0 requests when Host: headers are missing, + this is per Listener socket. + +* PublicInbox::HTTPD::Async + Common abbreviation: $async + + Used for implementing an asynchronous "push" interface for + slow, expensive responses which may require spawning + git-httpd-backend(1), git-apply(1) or other commands. + This will also be used for dealing with future asynchronous + operations such as HTTP reverse proxying and slow storage + retrieval operations. + +* PublicInbox::NNTPD + Common abbreviation: $nntpd + + Represents an NNTP daemon which creates PublicInbox::NNTP + wrappers around client sockets accepted from + PublicInbox::Listener. + + This is currently a singleton, but it is associated with a + given PublicInbox::Config which may be instantiated more than + once in the future. + +* PublicInbox::EOFpipe + + Used throughout to trigger a callback when a pipe(7) is closed. + This is frequently used to portably detect process exit without + relying on a catch-all waitpid(-1, ...) call. diff --git a/Documentation/technical/ds.txt b/Documentation/technical/ds.txt index cbd06cfb..afead2f1 100644 --- a/Documentation/technical/ds.txt +++ b/Documentation/technical/ds.txt @@ -1,9 +1,14 @@ PublicInbox::DS - event loop and async I/O base class -Our PublicInbox::DS event loop which powers public-inbox-nntpd -and public-inbox-httpd diverges significantly from the -unmaintained Danga::Socket package we forked from. In fact, -it's probably different from most other event loops out there. +Our PublicInbox::DS event loop which powers most of our long-lived +processes(*) diverges significantly from the unmaintained Danga::Socket +package we forked from. In fact, it's probably different from most +other event loops out there. + +Most notably, it uses one-shot, level-trigger, and edge-trigger mode +modes of kqueue|epoll depending on the situation. + +(*) public-inbox-netd,(-httpd,-imapd,-nntpd,-pop3d,-watch) + lei-daemon Most notably: @@ -14,7 +19,7 @@ Most notably: triggers a call. The lack of read/write callback distinction is driven by the - fact TLS libraries (e.g. OpenSSL via IO::Socket::SSL) may + fact that TLS libraries (e.g. OpenSSL via IO::Socket::SSL) may declare SSL_WANT_READ on SSL_write(), and SSL_WANT_READ on SSL_read(). So we end up having to let each user object decide whether it wants to make read or write calls depending on its @@ -30,7 +35,7 @@ Most notably: Reducing the user-supplied code down to a single callback allows subclasses to keep their logic self-contained. The combination of this change and one-shot wakeups (see below) for bidirectional - data flows make asynchronous code easier to reason about. + data flows makes asynchronous code easier to reason about. Other divergences: @@ -48,7 +53,7 @@ Other divergences: Augmented features: -* obj->write(CODEREF) passes the object itself to the CODEREF +* obj->write(CODEREF) passes the object itself to the CODEREF. Being able to enqueue subroutine calls is a powerful feature in Danga::Socket for keeping linear logic in an asynchronous environment. Unfortunately, each subroutine takes several kilobytes of memory. @@ -64,8 +69,8 @@ Augmented features: * ->requeue support. An optimization of the AddTimer(0, ...) idiom for immediately dispatching code at the next event loop iteration. public-inbox uses this for fairly generating large responses - iteratively (see PublicInbox::NNTP::long_response or the use of - ->getline callbacks for generating gigantic gzipped mboxes). + iteratively (see PublicInbox::NNTP::long_response or ibx_async_cat + for blob retrievals). New features @@ -77,12 +82,11 @@ New features which (if any) events it's interested in for the next loop iteration. * Edge-triggering available via EPOLLET or EV_CLEAR. These reduce wakeups - for unidirectional classes (e.g. PublicInbox::Listener sockets, - and pipes via PublicInbox::HTTPD::Async). + for unidirectional classes when throughput is more important than fairness. * IO::Socket::SSL support (for NNTPS, STARTTLS+NNTP, HTTPS) -* dwaitpid (waitpid wrapper) support for reaping dead children +* awaitpid (waitpid wrapper) support for reaping dead children * reliable signal wakeups are supported via signalfd on Linux, EVFILT_SIGNAL on *BSDs via IO::KQueue. diff --git a/Documentation/technical/memory.txt b/Documentation/technical/memory.txt new file mode 100644 index 00000000..039694c3 --- /dev/null +++ b/Documentation/technical/memory.txt @@ -0,0 +1,56 @@ +semi-automatic memory management in public-inbox +------------------------------------------------ + +The majority of public-inbox is implemented in Perl 5, a +language and interpreter not particularly known for being +memory-efficient. + +We strive to keep processes small to improve locality, allow +the kernel to cache more files, and to be a good neighbor to +other processes running on the machine. Taking advantage of +automatic reference counting (ARC) in Perl allows us to +deterministically release memory back to the heap. + +We start with a simple data model with few circular +references. This both eases human understanding and reduces +the likelihood of bugs. + +Knowing the relative sizes and quantities of our data +structures, we limit the scope of allocations as much as +possible and keep large allocations shortest-lived. This +minimizes both the cognitive overhead on humans in addition +to reducing memory pressure on the machine. + +Short-lived non-immortal closures (aka "anonymous subs") are +avoided in long-running daemons unless required for +compatibility with PSGI. Closures are memory-intensive and +may make allocation lifetimes less obvious to humans. They +are also the source of memory leaks in older versions of +Perl, including 5.16.3 found in enterprise distros. + +We also use Perl's `delete' and `undef' built-ins to drop +reference counts sooner than scope allows. These functions +are required to break the few reference cycles we have that +would otherwise lead to leaks. + +Of note, `undef' may be used in two ways: + +1. to free(3) the underlying buffer: + + undef $scalar; + +2. to reset a buffer but reduce realloc(3) on subsequent growth: + + $scalar = ""; # useful when repeated appending + $scalar = undef; # usually not needed + +In the future, our internal data model will be further +flattened and simplified to reduce the overhead imposed by +small objects. Large allocations may also be avoided by +optionally using Inline::C. + +Finally, the mwrap-perl LD_PRELOAD wrapper was ported to Perl 5 +and enhanced to provide live memory usage tracking on 64-bit systems +with minimal performance impact on production traffic: + + git clone https://80x24.org/mwrap-perl.git diff --git a/Documentation/technical/weird-stuff.txt b/Documentation/technical/weird-stuff.txt new file mode 100644 index 00000000..0c8d6891 --- /dev/null +++ b/Documentation/technical/weird-stuff.txt @@ -0,0 +1,22 @@ +There's a lot of weird code in public-inbox which may be daunting +to new hackers. + +* The event loop (PublicInbox::DS) is an evolution of a fairly standard + C10K event loop. See ds.txt in this directory for more. + +Things got weirder in 2021: + +* The lei command-line tool is backed by a daemon. This was done to + improve startup time for shell completion and manage git/SQLite/Xapian + single-writer during long, parallel imports. It may eventually become + a read-write IMAP/JMAP server. + +* SOCK_SEQPACKET is used extensively in lei, and will likely make its + way into more places, still. + +And even more so in 2022: + +* public-inbox-clone / PublicInbox::LeiMirror relies on ->DESTROY + for make-like dependency management while providing parallelism. + +More to come, lei will expose Maildirs via FUSE 3... diff --git a/Documentation/technical/whyperl.txt b/Documentation/technical/whyperl.txt new file mode 100644 index 00000000..db1d9793 --- /dev/null +++ b/Documentation/technical/whyperl.txt @@ -0,0 +1,177 @@ +why public-inbox is currently implemented in Perl 5 +--------------------------------------------------- + +While Perl has many detractors and there's a lot not to like +about Perl, we use it anyways because it offers benefits not +(yet) available from other languages. + +This document is somewhat inspired by https://sqlite.org/whyc.html + +Other languages and runtimes may eventually be a possibility +for us, and this document can serve as our requirements list +for possible replacements. + +As always, comments and corrections and additions welcome at +<meta@public-inbox.org>. We're not Perl experts, either. + +Good Things +----------- + +* Availability + + Perl 5 is installed on many, if not most GNU/Linux and + BSD-based servers and workstations. It is likely the most + widely installed programming environment that offers a + significant amount of POSIX functionality. Users won't + have to waste bandwidth or space with giant toolchains or + architecture-specific binaries. + + Furthermore, Perl documentation is typically installed + locally as manpages, allowing users to quickly refer + to documentation as needed. + +* Scripted, always editable by the end user + + Users cannot lose access to the source code. Code written + entirely in any scripting language automatically satisfies + the GPL-2.0, making it easier to satisfy the AGPL-3.0. + + Use of a scripting language improves auditability for + malicious changes. It also reduces storage and bandwidth + requirements for distributors, as the same scripts can be + shared across multiple OSes and architectures. + + Perl's availability and the low barrier to entry of + scripting ensures it's easy for users to exercise their + software freedom. + +* Predictable performance + + While Perl is neither fast nor memory-efficient, its + performance and memory use are predictable and do not + require GC tuning by the user. + + public-inbox is developed for (and mostly on) old + hardware. Perl was fast enough to power the web of the + late 1990s, and any cheap VPS today has more than enough + RAM and CPU for handling plain-text email. + + Low hardware requirements increase the reach of our software + to more users, improving centralization resistance. + +* Compatibility + + Unlike similarly powerful scripting languages, there is no + forced migration to a major new version. From 2000-2020, + Perl had fewer breaking changes than Python or Ruby; we + expect that trend to continue given the inertia of Perl 5. + + As of April 2021, the Perl Steering Committee has confirmed + Perl 7 will require `use v7.0' and existing code should + continue working unchanged: + https://nntp.perl.org/group/perl.perl5.porters/259789 + <CAMvkq_SyTKZD=1=mHXwyzVYYDQb8Go0N0TuE5ZATYe_M4BCm-g@mail.gmail.com> + +* Built for text processing + + Our focus is plain-text mail, and Perl has many built-ins + optimized for text processing. It also has good support + for UTF-8 and legacy encodings found in old mail archives. + +* Integration with distros and non-Perl libraries + + Perl modules and bindings to common libraries such as + SQLite and Xapian are already distributed by many + GNU/Linux distros and BSD ports. + + There should be no need to rely on language-specific + package managers such as cpan(1), those systems increase + the learning curve for users and system administrators. + +* Compactness and terseness + + Less code generally means less bugs. We try to avoid the + "line noise" stereotype of some Perl codebases, yet still + manage to write less code than one would with + non-scripting languages. + +* Performance ceiling and escape hatch + + With optional Inline::C, we can be "as fast as C" in some + cases. Inline::C is widely packaged by distros and it + gives us an escape hatch for dealing with missing bindings + or performance problems should they arise. Inline::C use + (as opposed to XS) also preserves the software freedom and + auditability benefits to all users. + + Unfortunately, most C toolchains are big; so Inline::C + will always be optional for users who cannot afford the + bandwidth or space. + + +Bad Things +---------- + +* Slow startup time. Tokenization, parsing, and compilation of + pure Perl is not cached. Inline::C does cache its results, + however. + + We work around slow startup times in tests by preloading + code, similar to how mod_perl works for CGI. + +* High space overhead and poor locality of small data + structures, including the optree. This may not be fixable + in Perl itself given compatibility requirements of the C API. + + These problems are exacerbated on modern 64-bit platforms, + though the Linux x32 ABI offers promise. + +* Lack of vectored I/O support (writev, sendmmsg, etc. syscalls) + and "newer" POSIX functions in general. APIs end up being + slurpy, favoring large buffers and memory copies for + concatenation rather than rope (aka "cord") structures. + +* While mmap(2) is available via PerlIO::mmap, string ops + (m//, substr(), index(), etc.) still require memory copies + into userspace, negating a benefit of zero-copy. + +* The XS/C API makes it difficult to improve internals while + preserving compatibility. + +* Lack of optional type checking. This may be a blessing in + disguise, though, as it encourages us to simplify our data + models and lowers cognitive overhead. + +* SMP support is mostly limited to fork(), since many + libraries (including much of the standard library) are not + thread-safe. Even with threads.pm, sharing data between + interpreters within the same process is inefficient due to + the lack of lock-free and wait-free data structures from + projects such as Userspace RCU. + +* Process spawning speed degrades as memory use increases. + We work around this optionally via Inline::C and vfork(2), + since Perl lacks an approximation of posix_spawn(3). + + We also use `undef' and `delete' ops to free large buffers + as soon as we're done using them to save memory. + + +Red herrings to ignore when evaluating other runtimes +----------------------------------------------------- + +These don't discount a language or runtime from being +used, they're just not interesting. + +* Lightweight threading + + While lightweight threading implementations are + convenient, they tend to be significantly heavier than + pure event-loop systems (or multi-threaded event-loop + systems). + + Lightweight threading implementations have stack overhead + and growth typically measured in kilobytes. The userspace + state overhead of event-based systems is an order of + magnitude less, and a sunk cost regardless of concurrency + model. diff --git a/Documentation/txt2pre b/Documentation/txt2pre index dcef4b6c..b45c52e8 100755 --- a/Documentation/txt2pre +++ b/Documentation/txt2pre @@ -1,28 +1,74 @@ -#!/usr/bin/env perl -# Copyright (C) 2014-2020 all contributors <meta@public-inbox.org> +#!perl -w +# n.b. this is invoked via $(PERL) in makefiles +# Copyright (C) all contributors <meta@public-inbox.org> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt> # # Stupid script to make HTML from preformatted, utf-8 text versions, # only generating links for http(s). Markdown does too much # and requires indentation to output preformatted text. -use strict; -use warnings; +use v5.12; use PublicInbox::Linkify; use PublicInbox::Hval qw(ascii_html); -my %xurls; -for (qw[public-inbox.cgi(1) - public-inbox-compact(1) +my (%xurls, %lei); +for (qw[lei(1) + lei-add-external(1) + lei-add-watch(1) + lei-blob(1) + lei-config(1) + lei-convert(1) + lei-daemon(8) + lei-daemon-kill(1) + lei-daemon-pid(1) + lei-edit-search(1) + lei-export-kw(1) + lei-forget-external(1) + lei-forget-mail-sync(1) + lei-forget-search(1) + lei-import(1) + lei-index(1) + lei-init(1) + lei-inspect(1) + lei-lcat(1) + lei-ls-external(1) + lei-ls-label(1) + lei-ls-mail-source(1) + lei-ls-mail-sync(1) + lei-ls-search(1) + lei-ls-watch(1) + lei-mail-diff(1) + lei-mail-sync-overview(7) + lei-overview(7) + lei-p2q(1) + lei-q(1) + lei-rediff(1) + lei-refresh-mail-sync(1) + lei-reindex(1) + lei-rm(1) + lei-rm-watch(1) + lei-security(7) + lei-store-format(5) + lei-tag(1) + lei-up(1) + public-inbox.cgi(1) + public-inbox-cindex(1) + public-inbox-clone(1) public-inbox-config(5) public-inbox-convert(1) public-inbox-daemon(8) public-inbox-edit(1) + public-inbox-extindex(1) + public-inbox-fetch(1) + public-inbox-glossary(7) public-inbox-httpd(1) + public-inbox-imapd(1) public-inbox-index(1) public-inbox-init(1) public-inbox-learn(1) public-inbox-mda(1) + public-inbox-netd(1) public-inbox-nntpd(1) public-inbox-overview(7) + public-inbox-pop3d(1) public-inbox-purge(1) public-inbox-v1-format(5) public-inbox-v2-format(5) @@ -32,22 +78,25 @@ for (qw[public-inbox.cgi(1) my ($n) = (/([\w\-\.]+)/); $xurls{$_} = "$n.html"; $xurls{$n} = "$n.html"; + /\Alei-(.+?)\(1\)\z/ and $xurls{"lei $1"} = "$n.html"; } -for (qw[copydatabase(1) xapian-compact(1)]) { - my ($n) = (/([\w\-\.]+)/); - $xurls{$_} = ".$n.1.html" -} +$xurls{'lei/store'} = 'lei-store-format.html'; -for (qw[flock(2) setrlimit(2) vfork(2)]) { +for (qw[make(1) flock(2) setrlimit(2) vfork(2) tmpfs(5) inotify(7) unix(7) + syslog(3)]) { my ($n, $s) = (/([\w\-]+)\((\d)\)/); - $xurls{$_} = "http://www.man7.org/linux/man-pages/man2/$n.$s.html"; + $xurls{$_} = "https://www.man7.org/linux/man-pages/man$s/$n.$s.html"; } for (qw[git(1) git-am(1) + git-apply(1) git-config(1) + git-credential(1) git-daemon(1) + git-diff(1) + git-fast-import(1) git-fetch(1) git-filter-branch(1) git-format-patch(1) @@ -57,6 +106,7 @@ for (qw[git(1) git-init(1) git-send-email(1) gitrepository-layout(5) + gitglossary(7) ]) { my ($n) = (/([\w\-\.]+)/); $xurls{$_} = "https://kernel.org/pub/software/scm/git/docs/$n.html" @@ -72,15 +122,42 @@ for (qw[ $xurls{$_} = "https://www.freedesktop.org/software/systemd/man/$n.html"; } +# favor upstream docs if they exist, use manpages.debian.org if they don't +$xurls{'netrc(5)'} = 'https://manpages.debian.org/stable/ftp/netrc.5.en.html'; +$xurls{'mbsync(1)'} = + 'https://manpages.debian.org/stable/isync/mbsync.1.en.html'; +$xurls{'offlineimap(1)'} = + 'https://manpages.debian.org/stable/offlineimap/offlineimap.1.en.html'; $xurls{'spamc(1)'} = 'https://spamassassin.apache.org/full/3.4.x/doc/spamc.html'; $xurls{'grok-pull'} = 'https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git' . '/tree/man/grok-pull.1.rst'; $xurls{'git-filter-repo(1)'} = 'https://github.com/newren/git-filter-repo'. - './blob/master/Documentation/git-filter-repo.txt'; + '/blob/master/Documentation/git-filter-repo.txt'; $xurls{'ssoma(1)'} = 'https://ssoma.public-inbox.org/ssoma.txt'; $xurls{'cgitrc(5)'} = 'https://git.zx2c4.com/cgit/tree/cgitrc.5.txt'; +$xurls{'prove(1)'} = 'https://perldoc.perl.org/prove.html'; +$xurls{'mbox(5)'} = 'https://manpages.debian.org/stable/mutt/mbox.5.en.html'; +$xurls{'mmdf(5)'} = 'https://manpages.debian.org/stable/mutt/mmdf.5.en.html'; +$xurls{'mutt(1)'} = 'https://manpages.debian.org/stable/mutt/mutt.1.en.html'; +$xurls{'torsocks(1)'} = + 'https://manpages.debian.org/stable/torsocks/torsocks.1.en.html'; +$xurls{'curl(1)'} = 'https://manpages.debian.org/stable/curl/curl.1.en.html'; +$xurls{'copydatabase(1)'} = + 'https://manpages.debian.org/stable/xapian-tools/copydatabase.1.en.html'; +$xurls{'xapian-compact(1)'} = + 'https://manpages.debian.org/stable/xapian-tools/xapian-compact.1.en.html'; +$xurls{'xapian-delve(1)'} = + 'https://manpages.debian.org/stable/xapian-tools/xapian-delve.1.en.html'; +$xurls{'gzip(1)'} = 'https://manpages.debian.org/stable/gzip/gzip.1.en.html'; +$xurls{'chmod(1)'} = + 'https://manpages.debian.org/stable/coreutils/chmod.1.en.html'; +$xurls{'kqueue(2)'} = + 'https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2'; +$xurls{'notmuch(1)'} = 'https://notmuchmail.org/manpages/notmuch-1/'; +$xurls{'mairix(1)'} = + 'https://manpages.debian.org/stable/mairix/mairix.1.en.html'; my $str = do { local $/; <STDIN> }; my ($title) = ($str =~ /\A([^\n]+)/); @@ -90,6 +167,9 @@ if ($str =~ /^NAME\n\s+([^\n]+)/sm) { if ($title =~ /([\w\.\-]+)/) { delete $xurls{$1}; } + if ($title =~ /\blei-([\w\-]+)\b/) { + delete $xurls{"lei $1"}; + } } $title = ascii_html($title); my $l = PublicInbox::Linkify->new; @@ -109,9 +189,3 @@ print '<html><head>', "<title>$title</title>", "</head><body><pre>", $str , '</pre></body></html>'; STDOUT->flush; - -# keep mtime on website consistent so clients can cache -if (-f STDIN && -f STDOUT) { - my @st = stat(STDIN); - utime($st[8], $st[9], \*STDOUT); -} |