about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-01-01lei_to_mail: lazy-require LeiDedupe
LeiDedupe requires SQLite, so we may want to be able to test writing mail without DBI or SQLite down the line.
2021-01-01lei: implement various deduplication strategies
For writing mboxes and Maildirs, users may wish to use stricter or looser deduplication strategies. This gives them more control.
2021-01-01lei_to_mail: start --augment, dedupe, bz2 and xz
--augment will match the mairix(1) option of the same name to augment existing search results. We'll need to implement deduplication for a better user experience. mutt ships with compressed mbox support for bz2 and xz, at least, so we'll support those out-of-the-box.
2021-01-01mboxreader: new class for reading various mbox formats
This is only lightly-tested against stuff LeiToMail generates and will need real-world tests to validate.
2021-01-01lei_to_mail: start atomic and compressed mbox writing
We'll allow using multiple workers to write to a single mbox (which could be compressed). This is can be done safely with O_APPEND + syswrite for uncompressed files, and using a lock when piping to pigz/gzip/bzip2/xz.
2021-01-01sharedkv: split out index_values
In most cases, we won't need to index by value, so don't waste cycles or space on it.
2021-01-01sharedkv: fork()-friendly key-value store
This is intended for maintaining Maildir states, mbox message deduplication, but may be useful for other purposes...
2021-01-01lei_to_mail: initial implementation for writing mbox formats
No Maildir, support, yet, but it'll come.
2021-01-01revert "lei_store: use per-machine refname as git HEAD"
In retrospect, per-machine HEADs was a bad idea because users of removable storage would be thrown off when moving storage between different machines. This is only a partial revert, the Import::init_bare change to support alternate head names still exists because we may use it for other reasons.
2021-01-01lei_store: use per-machine refname as git HEAD
It may be helpful to identify the source of messages and perhaps avoid conflicting history. On the other hand, this may be a terrible idea for users who move portable storage (e.g. USB sticks) across computers...
2021-01-01import: respect init.defaultBranch
This matches git v2.28.0+ behavior in case users prefer a different name.
2021-01-01Merge remote-tracking branch 'origin/lei' into eidx
* origin/lei: (28 commits) lei: rename proposed "query" command to "q", add JSON output lei_xsearch: cross-(inbox|extindex) search lei: extinbox: start implementing in config file lei: revise output routines lei: support for -$DIGIT and -$SIG CLI switches build: add lei.sh + "make symlink-install" target lei: start working on bash completion lei: drop $SIG{__DIE__}, add oneshot fallbacks lei: restore default __DIE__ handler for event loop on_destroy: generic localized END lei_store: keyword extraction from mbox and Maildir lei_store: relax GIT_COMMITTER_IDENT check lei: micro-optimize startup time lei: rename $client => $self and bless lei: help: show actual paths being operated on lei: support pass-through for `lei config' rename LeiDaemon package to PublicInbox::LEI search: simplify initialization, add ->xdb_shards_flat lei_store: simplify git_epoch_max, slightly lei: support `daemon-env' for modifying long-lived env ...
2021-01-01Merge tag 'v1.6.1' into eidx
public-inbox 1.6.1 - minor bugfix release * tag 'v1.6.1': (31 commits) public-inbox 1.6.1 - minor bugfix release import: drop X-Status in addition to Status eml: fix undefined vars on <Perl 5.28 t/config: test --get-urlmatch for git <2.26 inboxidle: avoid needless syscalls on refresh inboxidle: clue users into resolving ENOSPC from inotify inbox: name variable for values loop iterator public-inbox-v[12]-format.pod: make lexgrog happy manifest.js.gz: fix per-inbox /$INBOX/manifest.js.gz Fix manpage section of perl module documentation t/psgi_v2: ignore warnings on missing P::M::ReverseProxy daemon: support --daemonize without Net::Server::Daemonize doc: v2-format: drop repeated word over: ensure old, merged {tid} is really gone wwwattach: prevent deep-linking via Referer match t/eml.t: workaround newer Email::MIME* behavior nntp: attempt RFC 5536 3.1.5-conformant Path: headers nntp: delimit Newsgroup: header with commas tls: epollbit: account for miscellaneous OpenSSL errors scripts/dupe-finder: restore $dbh variable ...
2020-12-31public-inbox 1.6.1 - minor bugfix release v1.6.1
2020-12-31Merge remote-tracking branch 'origin/master' into lorelei
* origin/master: (58 commits) ds: flatten + reuse @events, epoll_wait style fixes ds: simplify EventLoop implementation check defined return value for localized slurp errors import: check for git->qx errors, clearer return values git: qx: avoid extra "local" for scalar context case search: remove {mset} option for ->mset method search: remove pointless {relevance} setting miscsearch: take reopen from Search and use it extsearch: unconditionally reopen on access extindex: allow using --all without EXTINDEX_DIR extindex: add undocumented --no-scan switch extindex: enable autoflush on STDOUT/STDERR extindex: various --watch signal handling fixes extindex: --watch for inotify-based updates eml: fix undefined vars on <Perl 5.28 t/config: test --get-urlmatch for git <2.26 default to CORE::warn in $SIG{__WARN__} handlers inbox: name variable for values loop iterator inboxidle: avoid needless syscalls on refresh inboxidle: clue users into resolving ENOSPC from inotify ...
2020-12-31lei: rename proposed "query" command to "q", add JSON output
Using "query" as a verb may be confusing when we'll also refer to them as nouns with the "<ls|rm|mv>-query" sub commands. "query" is also many characters to type without tab-completion on what I expect to be one of the most commonly used sub-commands Furthermore, "q" is also the common query parameter name used by our PSGI interface, as is the case with several major web search engines; so there's an element of familiarity there. The name "search" was disregarded because "show" could be a commonly used lei sub-command, too, and typing "se" for tab-completion may be slow since two-handed typists on QWERTY keyboards won't be able to use alternating hands. "f" or "find" could be a possibility here, too; but we're currently using the term "forget" as a weaker version of "remove" or "rm", though "ignore" could be substituted for "forget", perhaps... Kyle Meyer noted the lack of (proposed) JSON output support so that's been added to the proposed UI.
2020-12-31lei_xsearch: cross-(inbox|extindex) search
While a single extindex combines multiple inboxes into a single search index, extindex still requires up-front indexing on items which can be searched. XSearch has no on-disk footprint itself and uses Xapian DBs of existing publicinbox and extindex ("extinbox") exclusively. XSearch still suffers from the multi-shard Xapian scalability problems which led to the creation of extindex, but I expect the number of shards to remain relatively low. I envision users hosting public-inbox instances on their workstations will only have two extindex combined by this, one read-only extindex for serving public archives, and one read-write extindex managed by LeiStore for private mail.
2020-12-31import: drop X-Status in addition to Status
It's actually supported by mutt, dovecot[1], and likely some other software to augment the Status: header. While dovecot doesn't expose X-Status to clients, mutt will write 'A' (answered) and 'F' to X-Status (but not T (draft)). So we'll drop it like we do Status since it's not suitable for public mail, but stick it in an @UNWANTED_HEADERS array will allow us to configure an override if needed. [1] https://doc.dovecot.org/configuration_manual/mail_location/mbox/
2020-12-28ds: flatten + reuse @events, epoll_wait style fixes
Consistently returning the equivalent of pollfd.revents in a portable manner was never worth the effort for us, as we use the same ->event_step callback regardless of POLLIN/POLLOUT/POLLHUP. Being a Perl, @events knows it size and we don't have to return a maximum index for the caller to iterate on. We can also avoid redundant integer coercion ("+0") since we ensure everything is an IV in other places. Finally, vec() is preferable to ("\0" x $size) for resizing buffers because it only needs to write the extended portion and not overwrite the entire buffer.
2020-12-28ds: simplify EventLoop implementation
More importantly, make it easier-to-find the sub by avoiding runtime manipulation of subroutine names. There's no point in avoiding a potential call to _InitPoller in EventLoop since entering EventLoop is rare. On the contrary, PublicInbox::DS->new is called often and this change to avoid entering _InitPoller there may have more benefits (which may still be unmeasurable).
2020-12-28check defined return value for localized slurp errors
Reading from regular files (even on STDIN) can fail when dealing with flakey storage.
2020-12-28import: check for git->qx errors, clearer return values
Those git commands can fail and git->qx will set $? when it fails. There's no need for the extra indirection of the @ret array, either. Improve git->qx coverage to check for $? while we're at it.
2020-12-28git: qx: avoid extra "local" for scalar context case
We can use the ternary operator to avoid an early return, here
2020-12-28search: remove {mset} option for ->mset method
The ->mset method always returns a Xapian mset nowadays, so naming a parameter {mset} is too confusing. As it does with MiscSearch, setting the {relevance} parameter to -1 now sorts by ascending docid order. -2 is now supported for descending docid order, too, since it may be useful for lei users.
2020-12-28search: remove pointless {relevance} setting
SearchView will set it to `undef', others will set the 'mset' option (for the ->mset method :P) to 2 which causes {relevance} to be ignored. And the 'mset' option is poorly named now that the message is named ->mset...
2020-12-28miscsearch: take reopen from Search and use it
As with ExtSearch, MiscSearch lacks a janky cleanup timer of PublicInbox::Inbox objects, leading to info about inboxes/newsgroups going stale. Fortunately, we don't use MiscSearch very heavily, yet. In the future, we may be able to detect new inboxes without having to SIGHUP or restart daemons using MiscSearch.
2020-12-28extsearch: unconditionally reopen on access
Since ExtSearch lacks the janky cleanup timer of PublicInbox::Inbox objects, its search results get stale. Reopen the Xapian DB on every ->search call for now, as reducing reopen calls doesn't seem worth the complexity. The Xapian::Database::reopen operation itself takes only ~50us on my old workstation with 3 shards totaling <200GB. Other parts of Xapian dominates the search time, so the reopen seems inconsequential with single-digit shard counts.
2020-12-27extindex: allow using --all without EXTINDEX_DIR
If "--all" is specified to index all inboxes, implicitly choose the configured [extindex "all"] external index since "--all" is incompatible with specifying inbox directories on the command-line.
2020-12-27extindex: add undocumented --no-scan switch
This makes diagnosing --watch problems easier when there's 50K inboxes by avoiding the lengthy scan (which is the reason --watch exists in the first place).
2020-12-27extindex: enable autoflush on STDOUT/STDERR
With --watch, the output may be redirected to a pipe or socket which Perl may decide to buffer. Ensure Perl doesn't buffer these outputs since they can provide real-time status updates in response to signals or FS activity.
2020-12-27extindex: various --watch signal handling fixes
We need to clobber the SIGUSR1 resync queue on SIGHUP to invalidate old inbox objects. Furthermore, the lengthy initial scan needs to ignore signals intended for the event loop to avoid unexpected behavior. Finally, add some progress output to inform users on the terminal to inform users' of progress.
2020-12-27extindex: --watch for inotify-based updates
This reuses existing InboxIdle infrastructure to update external indices based on per-inbox updates. This is an alternative to auto-updating external indices via the -index command and also works with existing uses of -mda and public-inbox-watch. Using inotify (or EVFILT_VNODE) allows watching thousands of inboxes without having to scan every single one at every invocation. This is especially beneficial in cases where an external index is not writable to the users writing to per-inbox indices.
2020-12-26eml: fix undefined vars on <Perl 5.28
Encode::MIME::Header::_decode_octets did not correctly default to Encode::FB_DEFAULT until Encode 2.93 (perl5.git commit 0c541dc5633a341cf44b818014b58e7f8be532e9). Provide the default again to work with older Perls. Reported-by: Ali Alnubani <alialnu@nvidia.com> Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/ Tested-by: Ali Alnubani <alialnu@nvidia.com> (cherry picked from commit 46bd595f57cc3d425754b0d0770c125616e74448)
2020-12-26t/config: test --get-urlmatch for git <2.26
While git 1.8.5 learned --get-urlmatch, git did not learn to match URLs against wildcards until 2.26. So only depend on 1.8.5 for this test since 2.26 is too new. Reported-by: Ali Alnubani <alialnu@nvidia.com> Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/ Tested-by: Ali Alnubani <alialnu@nvidia.com> (cherry picked from commit 382a2bdd54cfb6c28a935c2b9fe4f1b1c2f469c4)
2020-12-26inboxidle: avoid needless syscalls on refresh
We don't have to replace a bunch of existing watches with identical new ones. On Linux with Linux::Inotify2 installed, this avoids a storm of inotify_add_watch(2) and inotify_rm_watch(2) syscalls on SIGHUP with -imapd and "-extindex --watch" (cherry picked from commit 10bf54305da8422d9ece6b809996092c1c4b1786) Note: this seems to fix missed wakeups with many watches, so it's in the stable branch. Link: https://public-inbox.org/meta/20201226201115.GA30142@dcvr/
2020-12-26inboxidle: clue users into resolving ENOSPC from inotify
It may not be obvious to users a ENOSPC error is from hitting a (tunable) kernel-imposed limit on inotify watches, and not some storage device running out of space. Give them a hint here to reduce our own support burden. (cherry picked from commit 5e05c2eb58a450849f1826f3d02ed62b814b6617)
2020-12-26inbox: name variable for values loop iterator
->on_inbox_unlock callbacks could clobber $_, and this seems to fix a problem with -extindex --watch failing to index some inboxes after SIGHUP reload. (cherry picked from commit b5e960f50289434025f5904c8c1311e4c8a02b82)
2020-12-26public-inbox-v[12]-format.pod: make lexgrog happy
The Debian package linter (lintian) emits the following warning: W: bad-whatis-entry N: N: A manual page should start with a NAME section, which lists the N: program name and a brief description. The NAME section is used to N: generate a database that can be queried by commands like apropos and N: whatis. You are seeing this tag because lexgrog was unable to parse N: the NAME section. N: N: Manual pages for multiple programs, functions, or files should list N: each separated by a comma and a space, followed by \- and a common N: description. N: N: Listed items may not contain any spaces. A manual page for a two-level N: command such as fs listacl must look like fs_listacl so the list is N: read correctly. N: N: Refer to the lexgrog(1) manual page, the groff_man(7) manual page, and N: the groff_mdoc(7) manual page for details. N: N: Severity: warning N: N: Check: documentation/manual N: N: Renamed from: manpage-has-bad-whatis-entry N: for public-inbox-v1-format and public-inbox-v2-format. Adapt the descriptions to make lexgrog and so lintian happy. (cherry picked from commit 1350f5ab09f72c75ac2cd6c88f6a2b9e198fef55)
2020-12-26manifest.js.gz: fix per-inbox /$INBOX/manifest.js.gz
/$INBOX/manifest.js.gz should not attempt to match every inbox in the domain (or every inbox); that is for /manifest.js.gz (without a /$INBOX prefix). Fixes: f303b4add8ea1883 ("wwwlisting: avoid hogging event loop") (cherry picked from commit 3ce4c38119f13d419bb865a0aa9b66feff339308)
2020-12-26Fix manpage section of perl module documentation
On Debian (at least) perl documentation is supposed to be installed in section 3pm. With the build system hardcoding this to 3 instead this results in a warning by the Debian package linter: W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Git.3.gz:74 3 != 3pm W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Import.3.gz:74 3 != 3pm W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::SaPlugin::ListMirror.3.gz:74 3 != 3pm W: public-inbox: wrong-manual-section ... use --no-tag-display-limit to see all (or pipe to a file/program) So use "$(MAN3EXT)" instead of "3". (cherry picked from commit cd12cf22c4d161b7aec819f45628b983f0bd675d)
2020-12-26t/psgi_v2: ignore warnings on missing P::M::ReverseProxy
Plack::Test::ExternalServer doesn't depend on Plack::Middleware::ReverseProxy, so we need to account for some warnings in stderr if P::M::RP is missing. (cherry picked from commit d26c2837f479b41182946a6540aad95d34b2b594)
2020-12-26daemon: support --daemonize without Net::Server::Daemonize
We don't actually need Net::Server::Daemonize to support the --daemonize flag, since the daemonize() sub provided by N::S::D doesn't exactly do the things we want. (cherry picked from commit ab11caeb8ae114087636748df4c956d0e625b237)
2020-12-26doc: v2-format: drop repeated word
(cherry picked from commit 4df1904582c4d7bec64e05d1566cd48c8c2e888e)
2020-12-26over: ensure old, merged {tid} is really gone
We must use the result of link_refs() since it can trigger merge_threads() and invalidate $old_tid. In case merge_threads() isn't triggered, link_refs() will return $old_tid anyways. When rethreading and allocating new {tid}, we also must update the row where the now-expired {tid} came from to ensure only the new {tid} is seen when reindexing subsequent messages in history. Otherwise, every subsequently reindexed+rethreaded message could end up getting a new {tid}. Reported-by: Kyle Meyer <kyle@kyleam.com> Link: https://public-inbox.org/meta/87360nlc44.fsf@kyleam.com/ (cherry picked from commit 9356ec0cc5afc95a8fd398ddf898942ef0acdb74)
2020-12-26wwwattach: prevent deep-linking via Referer match
This prevents `<img src=' tags from being used to deep-link image attachments from HTML outside of the current host and reduces potential for abuse. Some browsers (e.g. Firefox) favor content detection and will display images irrespective of the Content-Type header being "application/octet-stream", and "Content-Disposition: attachment" doesn't stop them, either. Tested with dillo and Firefox. Reported-by: Leah Neukirchen <leah@vuxu.org> (cherry picked from commit 46cbc5a7a4ba917bd7154be3b6e6898420ff85d3)
2020-12-26t/eml.t: workaround newer Email::MIME* behavior
Recent (2020) versions of Email::MIME (and/or dependencies) have different behavior than historical versions which seem to be less DWIM and perhaps technically more correct. We'll retain historical behavior for now, since it doesn't seem to cause real problems and DWIM-ness is often required to make sense of historical mail. Tested on a FreeBSD 11.4 VM with the following packages: p5-Email-MIME-1.949 p5-Email-MIME-ContentType-1.024_1 p5-Email-MIME-Encodings-1.315_2 (cherry picked from commit 5c4874309c4dd7aa8b9acd6bb97bd512360ab08a)
2020-12-26nntp: attempt RFC 5536 3.1.5-conformant Path: headers
Perhaps some NNTP clients would be unhappy with the old value "y". So use a bit more bandwidth+space to use the server-name and historical "!not-for-mail" tail-entry to better conform to a published RFC. Reported-by: Andrey Melnikov <temnota.am@gmail.com> (cherry picked from commit 47559005ee678592808c351daa1c9f2682beb248)
2020-12-26nntp: delimit Newsgroup: header with commas
...instead of spaces. This is specified in RFC 5536 3.1.4. Include references to RFC 1036, 5536 and 5537 in our docs while we're at it. Reported-by: Andrey Melnikov <temnota.am@gmail.com> Link: https://public-inbox.org/meta/CA+PODjpUN5Q4gBFQhAzUNuMasVEdmp9f=8Uo0Ej0mFumdSwi4w@mail.gmail.com/ (cherry picked from commit 44227c2624e4f954943d632cd5335396351373be)
2020-12-26tls: epollbit: account for miscellaneous OpenSSL errors
Apparently they happen (triggered by my -imapd instance), so bail out by closing the underlying socket rather than stopping the event loop and daemon process. (cherry picked from commit c51c22c349529d9c377160abcc7961a6ca7b7d5c)
2020-12-26scripts/dupe-finder: restore $dbh variable
When dupe-finder was switched from ->search->{over_ro} to ->over, the database handle was dropped. Restore it because a spot downstream uses it. Fixes: 73e3a6ed6e95adc6 (use more idiomatic internal API for ->over access) (cherry picked from commit 0438ee2b1a5fbeb587ee62aa3e267ec09d281bd9)