about summary refs log tree commit homepage
path: root/Documentation
DateCommit message (Collapse)
2021-09-25doc: lei: manpages for export-kw and refresh-mail-sync
Something is better than nothing.
2021-09-25doc: lei-index: remove --stdin, reword -F
lei-index really only works for Maildir, at the moment.
2021-09-25doc: lei-overview: implicit stdin, correct Inline::C notes
Implicit stdin based on standard input being a pipe or regular file is here to stay, so save users the trouble of typing '-' or '--stdin'. Inline::C is required as of commit 1d6e1f9a6a66 (lei: require Socket::MsgHdr or Inline::C, drop oneshot, 2021-05-26); but Socket::MsgHdr still gives a noticeable improvement in bash completion speed. Also, spell-out "MESSAGE-ID" since "MID" is actually not a common abbreviation ("MSGID" is used by RFC 3977 and several other RFCs, I recall).
2021-09-25doc: lei blob+rediff+p2q: add notes about git directory
Try to clarify these commands are intended to be useful for git-using (usually software) projects (and not the bare git repos we use internally). We'll also document some commonly useful git-diff switches in the lei-rediff man page to highlight the usefulness of the command.
2021-09-24clone|--mirror: support --epoch=RANGE for partial clones
Partial (v2) clones should be useful addition for users wanting to conserve storage while having fast access to recent messages. Continuing work started in 876e74283ff3 (fetch: ignore non-writable epoch dirs, 2021-09-17), this creates bare, read-only epoch git repos. These git repos have the remotes pre-configured, but does not fetch any objects. The goal is to allow users to set the writable bit on a previously-skipped epoch and start fetching it. Shell completion support may not be necessary given how short the epoch ranges are, here. Cc: Luis Chamberlain <mcgrof@kernel.org> Link: https://public-inbox.org/meta/20210917002204.GA13112@dcvr/T/#u
2021-09-21lei q: improve --limit behavior and progress
Avoid slurping gigantic (e.g. 100000) result sets into a single response if a giant limit is specified, and instead use 10000 as a window for the mset with a given offset. We'll also warn and hint towards about the --limit= switch when the estimated result set is larger than the default limit.
2021-09-21doc: lei-security: section for WIP auth methods
Lots of stuff out there that becomes a pain to setup configuration for and test...
2021-09-19doc: lei-config: document various knobs
It's still a work-in-progress, but the basic debug knob comes in handy for new users; as does proxy support.
2021-09-19doc: tuning: note git 2.33+, move libgit2 into Inline::C section
git 2.33+ contains important optimizations for the thousands-of-inboxes case. And combine the Inline::C stuff with libgit2, since our use of libgit2 requires Inline::C.
2021-09-18lei up: automatically use dt: for remote externals
Since we can't use maxuid for remote externals, automatically maintaining the last time we got results and appending a dt: range to the query will prevent HTTP(S) responses from getting too big. We could be using "rt:", but no stable release of public-inbox supports it, yet, so we'll use dt:, instead. By default, there's a two day fudge factor to account for MTA downtime and delays; which is hopefully enough. The fudge factor may be changed per-invocation with the --remote-fudge-factor=INTERVAL option Since different externals can have different message transport routes, "lastresult" entries are stored on a per-external basis.
2021-09-18doc: lei-lcat: document --stdin behavior
This is another feature I've found immensely useful, but I also wonder if I'm the only one who uses it.
2021-09-17doc: add lei-security(7) manpage
It seems like a good idea to have a manpage where somebody can quickly look up and address their concerns as to what to put on encrypted device/filesystem. And I probably would've designed lei around make(1) for parallelization if I didn't have to keep credentials off the FS :P
2021-09-17fetch: ignore non-writable epoch dirs
This will eventually be useful for maintaing partial mirrors. Keeping inline with the original public-inbox-fetch philosophy, there are no additional config files to manage: the user merely needs to remove write permissions to an $N.git directory to prevent it from being updated. Re-enabling updates just requires restoring write permission.
2021-09-16doc: glossary: add a reference for "epoch"
It comes up often enough and we need to ensure it's not confused with IMAP "slices".
2021-09-16doc: lei-mail-formats: add "eml" and expand on git things
While "eml" is not an output format, it seems worthy to document, here, since users are likely to have experience with *.patch files from "git format-patch".
2021-09-16www: support publicinbox.imapserver
This allows PublicInbox::WWW hosts to advertise the existence of IMAP servers in addition to NNTP servers.
2021-09-15fetch: support --exit-code switch
As noted in the new manpage entry, this is useful for avoiding public-inbox-index invocations when there's nothing to update. We use 127 to match "grok-pull", and also because it doesn't conflict with any of the current curl(1) exit codes.
2021-09-14doc: update authentication notes for lei
~/.netrc isn't used by default any more, and I'm not sure it's worthwhile to document the --netrc switch since it's rare for non-FTP clients to support. Followup-to: 9d11ed460ce113dd ("lei: do not read ~/.netrc by default") Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2021-09-12new public-inbox-{clone,fetch} commands
Setting up and maintaining git-only mirrors of v2 inboxes is complex since multiple commands are required to clone and fetch into epochs. Unlike grokmirror, these commands do not require any configuration. Instead, they rely on existing git config files and work like "git clone --mirror" and "git fetch", respectively. Like grokmirror, they use manifest.js.gz, but only on a per-inbox basis so users won't have to clone every inbox of a large instance nor edit config files to include/exclude inboxes they're interested in.
2021-09-11lei q|lcat: support "-f reply" output format
When composing replies in "git format-patch" cover letters, I'd been relying on "lei q -f text ...", but that still requires several steps to make it suitable for composing a reply: * s/^/> / to quote the body * drop existing In-Reply-To+References * s/^Message-ID:/In-Reply-To:/; * add an attribute line ... "lei q -f reply" takes care of most of that and users will only have to trim "From " lines, unnecessary results and over-quoted text (and trimming is likely less error-prone than doing all the steps above manually). This should also be a good replacement for "git format-patch --in-reply-to=...", since copying long Message-IDs can be error-prone (and this lets you include quoted text in replies).
2021-09-10doc: lei-index manpage
It's a pretty incomplete command, so it's important to document its incompleteness.
2021-09-09lei-rm: add man page, support LeiInput args
-F/--in-format and --lock=TYPE(S) are easily supported by all classes using LeiInput.
2021-09-08doc: acknowledge the MMDF mailbox format
While I don't currently see a point in supporting MMDF, we'll still acknowledge it since mutt actually supports it. Expand a bit on MH while we're at it, since MH seems at least relevant.
2021-09-07doc: lei-*.pod: update to Tor v3 .onion address
We missed a few when new documentation came in, and there's no going back to v2 onions. Followup-to: 0b15dfc58ceaecdc ("treewide: update to v3 Tor onions")
2021-09-07lei up: support --all for IMAP folders
Since "lei up" is expected to be a heavily-used command, better support for IMAP seems like a reasonable idea. This is inefficient since we waste an IMAP(S) TCP connection since it dies when an auth-only LeiUp worker process dies, but it's better than not working at all, right now.
2021-09-01extindex: document --all and indexlevel=basic interaction
Cache is expensive, so help users save on storage by documenting this behavior.
2021-08-19lei q: make --save the default
Since "lei up" is more often useful than not and incurs neglible overhead; enable --save by default and allow --no-save to work. This also fixes a long-standing when overwriting --output destinations with saved searches: dedupe data from previous searches are reset and no longer influences the new (changed) search, so results no longer go missing if two sequential invocations of "lei q --save" point to the same --output.
2021-07-25doc: lei-{p2q,rediff}: note implicit --stdin
lei actually uses implicit --stdin everywhere, but I thing these patch-related commands are the most common use of them.
2021-07-22init: allow arbitrary key-values via -c KEY=VALUE
This won't blindly append identical key=values, but allows specifying multiple, different key=value pairs as long as the values are different.
2021-07-22extsearch: support publicinbox.*.boost parameter
This behaves identically the lei external "boost" parameter in prioritizing raw messages for extindex. Relying exclusively on the config file order doesn't work well for mirrors since it's impossible to guarantee config file ordering via grokmirror hooks. Config file ordering remains the default if boost is unconfigured, or in case of ties. Note: I chose the name "boost" rather than "priority" or "rank" since I always get confused by whether higher or lower numbers take precedence when it comes to kernel scheduling. "weight" is also a part of Xapian API terminology, which we currently do not expose to configuration (but may in the future).
2021-07-20httpd: fix SIGHUP by invalidating cache on reload
Since we require separate PublicInbox::HTTPD instances for each listen socket address (in order to support {SERVER_<NAME|PORT>} for PSGI env), the old cache needed to be invalidated on rare app refreshes. SIGHUP has always been broken in -httpd (but not -imapd or -nntpd) due to this cache. Update the daemon documentation and 5.10.1-ize some bits while we're in the area.
2021-06-24doc: tuning: add a note about Linux sys.vm.max_map_count
git tends to die when mmap(2) fails on this limit, so let users know about it. Perhaps git could gracefully fallback.
2021-06-24favor git(1) rather than libgit2 for ExtSearch
While both git and libgit2 take around 16 minutes to load 100K alternates there's already a proposed patch to make git faster: <https://lore.kernel.org/git/20210624005806.12079-1-e@80x24.org/> It's also easier to patch and install git locally since the git.git build system defaults to prefix=$HOME and dealing with dynamic linking with libgit2 is more difficult for end users relying on Inline::C. libgit2 remains in use for the non-ALL.git case, but maybe it's not necessary (libgit2 is significantly slower than git in Debian 10 due to SHA-1 collision checking).
2021-05-17doc lei: update manpages with new options
2021-05-17doc lei: add manpages for new commands
[ew: MANIFEST: s/lei-cat/lei-lcat/]
2021-05-17doc lei: add manpage for convert
Link: https://public-inbox.org/meta/20210429015738.GA30172@dcvr/
2021-05-17doc lei q: add missing value for --lock
2021-05-17doc lei q: fix a typo
2021-05-17doc lei: resort lei-tag entries
The command was renamed in 54da988cfb049ea2 (lei tag: rename from "lei mark", 2021-03-30). Relocate its entries in txt2pre and Makefile.PL to restore alphabetical sorting.
2021-05-17doc lei blob: point to lei-q for shared options
lei-blob shares a good number of options with lei-q. Refer to lei-q's manpage rather than repeating the descriptions. Also, add lei-q to lei-blob's "see also" section. Link: https://public-inbox.org/meta/YGFImEcWX1mCJJwv@dcvr/
2021-05-17doc: split option variants into separate items
e226f18934eb7291 modified the lei-q manpage so that each variant of an option gets a dedicated =item to make L</--xyz> look nicer and to follow the Perl core documentation. Do the same for the other manpages. Note that this still leaves the variants of an option grouped in one scenario: when a list of options without descriptions is presented as a pointer to another location. Splitting the variants in that case would make it harder for the reader to tell what the distinct options are.
2021-05-17doc lei blob: avoid combined description of separate options
The next commit will update the manpages to split each option's variants into separate items. This change won't mix well with --oid-a, --path-a, and --path-b. These different options all share a single description, and, if each form is on its own line, the link between the variants of each option would no longer be clear. Use a dedicated description for each option to avoid confusion.
2021-05-04doc: ignore onion URLs for 80-column check
This failure was also passing under FreeBSD make + /bin/sh; so we also avoid the '&&' chain is avoided and use '>$@' as a separate line in the Makefile.
2021-05-04treewide: update to v3 Tor onions
v2 onions are insecure, deprecated and going away. v3 names are unfortunately longer and more difficult to remember, but should be more resistant to attack than v2 ones.
2021-04-30lei import: avoid IMAPTracker, use LeiMailSync more
IMAPTracker has a UNIQUE constraint on the `url' column, which may cause compatibility and/or rollback problems in attempting to deal with UIDVALIDITY changes. Having multiple sources of truth leads to confusion and bugs, so relying on LeiMailSync exclusively ought to simplify things. Furthermore, since LeiMailSync is only written to by LeiStore, it is safer in that it won't mark a UID or article as imported until git-fast-import has seen it, and the SQLite commit always happens after "done\n" is sent to fast-import. This mostly reverts recent commits to IMAPTracker to support lei, those are: 1) commit 7632d8f7590daf70c65d4270e750c36552fa9389 ("net_reader: restart on first UID when UIDVALIDITY changes") 2) commit 311a5d37ad275cd75b1e64d87827c4d13fe4bfab ("imap_tracker: prepare for use with lei"). This means public-inbox-watch will not change between 1.6 and 1.7: -watch stops synching a folder when UIDVALIDITY changes.
2021-04-28doc: lei q: split --output and --format, note "text"
I guess --format makes sense for stdout, after all; and I'm enjoying "-f text" quite a bit, so far.
2021-04-28doc: lei q: split =item aliases onto separate lines
It makes L</--augment> look nicer without resorting to L<--augment|/-a, --augment> and similarly verbose nastiness. Having each option as a separate =item (with a blank line in between each =item) seems to be the preferred style used within Perl core documentation (I used perlrun.pod as an example), so we'll follow Perl core style, here. This needs to be done for other manpages, at some point...
2021-04-28doc: lei: use /tmp for search results pathnames
This drives the point home about results being volatile and discardable.
2021-04-24doc: lei_design_notes: add a bit on WAL usage
2021-04-22lei import|convert: drop --no-kw aliases
Supporting --no-keywords and --no-flags aliases is harmful if users end up assuming "keywords:" and "flags:" are valid search prefixes (they're not).