about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2020-12-31import: drop X-Status in addition to Status
It's actually supported by mutt, dovecot[1], and likely some other software to augment the Status: header. While dovecot doesn't expose X-Status to clients, mutt will write 'A' (answered) and 'F' to X-Status (but not T (draft)). So we'll drop it like we do Status since it's not suitable for public mail, but stick it in an @UNWANTED_HEADERS array will allow us to configure an override if needed. [1] https://doc.dovecot.org/configuration_manual/mail_location/mbox/
2020-12-26eml: fix undefined vars on <Perl 5.28
Encode::MIME::Header::_decode_octets did not correctly default to Encode::FB_DEFAULT until Encode 2.93 (perl5.git commit 0c541dc5633a341cf44b818014b58e7f8be532e9). Provide the default again to work with older Perls. Reported-by: Ali Alnubani <alialnu@nvidia.com> Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/ Tested-by: Ali Alnubani <alialnu@nvidia.com> (cherry picked from commit 46bd595f57cc3d425754b0d0770c125616e74448)
2020-12-26inboxidle: avoid needless syscalls on refresh
We don't have to replace a bunch of existing watches with identical new ones. On Linux with Linux::Inotify2 installed, this avoids a storm of inotify_add_watch(2) and inotify_rm_watch(2) syscalls on SIGHUP with -imapd and "-extindex --watch" (cherry picked from commit 10bf54305da8422d9ece6b809996092c1c4b1786) Note: this seems to fix missed wakeups with many watches, so it's in the stable branch. Link: https://public-inbox.org/meta/20201226201115.GA30142@dcvr/
2020-12-26inboxidle: clue users into resolving ENOSPC from inotify
It may not be obvious to users a ENOSPC error is from hitting a (tunable) kernel-imposed limit on inotify watches, and not some storage device running out of space. Give them a hint here to reduce our own support burden. (cherry picked from commit 5e05c2eb58a450849f1826f3d02ed62b814b6617)
2020-12-26inbox: name variable for values loop iterator
->on_inbox_unlock callbacks could clobber $_, and this seems to fix a problem with -extindex --watch failing to index some inboxes after SIGHUP reload. (cherry picked from commit b5e960f50289434025f5904c8c1311e4c8a02b82)
2020-12-26manifest.js.gz: fix per-inbox /$INBOX/manifest.js.gz
/$INBOX/manifest.js.gz should not attempt to match every inbox in the domain (or every inbox); that is for /manifest.js.gz (without a /$INBOX prefix). Fixes: f303b4add8ea1883 ("wwwlisting: avoid hogging event loop") (cherry picked from commit 3ce4c38119f13d419bb865a0aa9b66feff339308)
2020-12-26daemon: support --daemonize without Net::Server::Daemonize
We don't actually need Net::Server::Daemonize to support the --daemonize flag, since the daemonize() sub provided by N::S::D doesn't exactly do the things we want. (cherry picked from commit ab11caeb8ae114087636748df4c956d0e625b237)
2020-12-26over: ensure old, merged {tid} is really gone
We must use the result of link_refs() since it can trigger merge_threads() and invalidate $old_tid. In case merge_threads() isn't triggered, link_refs() will return $old_tid anyways. When rethreading and allocating new {tid}, we also must update the row where the now-expired {tid} came from to ensure only the new {tid} is seen when reindexing subsequent messages in history. Otherwise, every subsequently reindexed+rethreaded message could end up getting a new {tid}. Reported-by: Kyle Meyer <kyle@kyleam.com> Link: https://public-inbox.org/meta/87360nlc44.fsf@kyleam.com/ (cherry picked from commit 9356ec0cc5afc95a8fd398ddf898942ef0acdb74)
2020-12-26wwwattach: prevent deep-linking via Referer match
This prevents `<img src=' tags from being used to deep-link image attachments from HTML outside of the current host and reduces potential for abuse. Some browsers (e.g. Firefox) favor content detection and will display images irrespective of the Content-Type header being "application/octet-stream", and "Content-Disposition: attachment" doesn't stop them, either. Tested with dillo and Firefox. Reported-by: Leah Neukirchen <leah@vuxu.org> (cherry picked from commit 46cbc5a7a4ba917bd7154be3b6e6898420ff85d3)
2020-12-26nntp: attempt RFC 5536 3.1.5-conformant Path: headers
Perhaps some NNTP clients would be unhappy with the old value "y". So use a bit more bandwidth+space to use the server-name and historical "!not-for-mail" tail-entry to better conform to a published RFC. Reported-by: Andrey Melnikov <temnota.am@gmail.com> (cherry picked from commit 47559005ee678592808c351daa1c9f2682beb248)
2020-12-26nntp: delimit Newsgroup: header with commas
...instead of spaces. This is specified in RFC 5536 3.1.4. Include references to RFC 1036, 5536 and 5537 in our docs while we're at it. Reported-by: Andrey Melnikov <temnota.am@gmail.com> Link: https://public-inbox.org/meta/CA+PODjpUN5Q4gBFQhAzUNuMasVEdmp9f=8Uo0Ej0mFumdSwi4w@mail.gmail.com/ (cherry picked from commit 44227c2624e4f954943d632cd5335396351373be)
2020-12-26tls: epollbit: account for miscellaneous OpenSSL errors
Apparently they happen (triggered by my -imapd instance), so bail out by closing the underlying socket rather than stopping the event loop and daemon process. (cherry picked from commit c51c22c349529d9c377160abcc7961a6ca7b7d5c)
2020-12-26searchidx: index lower-case List-Id value
We don't want a List-Id value being confused with a Xapian term prefix, here. Followup-to: 8b06cda3a3af3f0e ("mda: match List-Id insensitively") (cherry picked from commit 447d049b70c6723a0de8e28aa7db1a4130394c4b)
2020-12-26ds: add missing label for systems w/o EPOLLEXCLUSIVE
Oops :x (cherry picked from commit 28dec3a76687617a3a697058964dc0bd1351eb6b)
2020-12-26imap: avoid raising exception if client disconnects
This ought to save a few cycles if a client disconnects while in the middle of a (UID) FETCH. This avoids: Can't call method "git" on an undefined value at .../PublicInbox/IMAP.pm errors in stderr. (cherry picked from commit 34880bc83077eac5739deca69f66df7685965064)
2020-12-26idxstack: fix comment about file_char
It's `d' for deletes, not `a'. (cherry picked from commit 9dd93e4399c67abf74df6832181d442f81e0aa33)
2020-12-26mda: match List-Id insensitively
This follows -watch commit b70473ab8296d31ebb600adb4fa8fe0ac5935ca8 to match List-Id headers case-insensitively. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20200921180152.uyqluod7qxbwqubo@chatter.i7.local/ (cherry picked from commit 8b06cda3a3af3f0e88b599a01f03edb7fec4f2c4)
2020-12-26mid: drop repeated ';' in mid_escape() regular expression
(cherry picked from commit dc93c36eb62d36e649b9500b7f190687a3fbcffd)
2020-12-26config: warn on multiple values for some fields
Our code doesn't support multi-values for these, and having unexpected arrays leads to unexpected results (e.g. showing stuff like "ARRAY(0xDEADBEEFADD12E55)" in user interfaces). So warn and only use the last value (matching git-config(1) behavior without `--get-all'). (cherry picked from commit c8fe4673056571ed251c3a20cb0bbec586c60bd6)
2020-09-16git_async_cat: fix outdated comment
We replaced Danga::Socket with PublicInbox::DS roughly a year before GitAsyncCat was introduced into our git history.
2020-09-16wwwtext: link to public-inbox.org/meta archives
Since we're advertising our address at meta@public-inbox.org, we should advertise the archives, too.
2020-09-16wwwstream: link to cgit URLs for coderepo
Hopefully this reduces the ambiguity between code for the project(s) using public-inbox and the code for public-inbox itself.
2020-09-16treewide: relax allow >=40 chars for git OID
This will help with eventual git SHA-256 transitions.
2020-09-16mid: rename MID_MAX to ID_MAX
It's only used for HTML anchors which we will need indefinitely.
2020-09-15imap: quiet uninitialized variable warning on FETCH
This was triggered by blindly trying to FETCH an MSN (not "UID FETCH") on an empty dummy inbox. It's harmless, and probably triggered by a wayward client or misbehaving bot.
2020-09-14tests: consistently check for xapian-compact
We may need to test against development versions of Xapian, which may rely on setting `XAPIAN_COMPACT=xapian-compact-1.5'. Ensure it's possible to do that. And add a missing check in t/xcpdb-reshard.t, too.
2020-09-14sigfd: fix typos and scoping on systems w/o epoll+kqueue
Unfortunately, I'm not sure how easy catching these at compile-time, is. Prototypes do not seem to check these at compile time when crossing packages (not even with exported subroutines).
2020-09-12nntp: share more code between art_lookup callers
This prepares us for future changes to improve scalability to many inboxes.
2020-09-12treewide: avoid `goto &NAME' for tail recursion
While Perl implements tail recursion via `goto' which allows avoiding warnings on deep recursion. It doesn't (as of 5.28) optimize the speed of such dispatches, though it may reduce ephemeral memory usage. Make the code less alien to hackers coming from other languages by using normal subroutine dispatch. It's actually slightly faster in micro benchmarks due to the complexity of `goto &NAME'.
2020-09-10wwwstream: show init + index instructions for -V1, too
This should've always been there. I'm not sure how widely spread 1.0 and earlier releases were, but we'll keep documenting the version requirement.
2020-09-10solver: async blob retrieval for diff extraction
Like the rest of the WWW code, public-inbox-httpd now uses git_async_cat to retrieve blobs without blocking the event loop. This improves fairness when git blobs are on slow storage and allows us to take better advantage of SMP systems.
2020-09-10solver: break apart inbox blob retrieval
To avoid hogging the event loop in public-inbox-httpd when many candidate messages match, we'll separate the steps to ensure fairness on slow storage.
2020-09-10solver: check one git coderepo and inbox at a time
With public-inbox-httpd, this mitigates the effect of slow git blob storage with multiple coderepos configured for an inbox. It's still synchronous for now (and may need to remain that way for ->last_check_err), but no longer monopolizes the event loop when checking multiple coderepos. We don't yet support multi-inbox scanning, yet; but this also prepares us for a future where we do. We'll also support >=40 char blob OIDs in preparation for future git SHA-256 support, too.
2020-09-10wwwlisting: avoid hogging event loop
By using the just-introduced ConfigIter class. And make ManifestJsGz a subclass of it to reduce duplication.
2020-09-10extmsg: prevent cross-inbox matches from hogging event loop
With many inboxes, checking multiple SQLite repos will be slow and time-consuming, so ensure we can schedule it fairly between multiple inboxes.
2020-09-10config: split out iterator into separate object
We will need to allow simultaneous iterators on the same config object, since we'll need this for ExtMsg, NNTPD, WwwListing, NewsWWW, and other places.
2020-09-10config: flatten each_inbox and iterate_start args
In Perl, we can simplify callers by passing a single array all the way down the stack instead of a single array ref which needs to be expanded every call.
2020-09-10www: manifest.js.gz generation no longer hogs event loop
It's still as slow as before with hundreds/thousands of inboxes, but at least it's fair. Future changes will allow it to be cached and memoized with persistent HTTP servers.
2020-09-10use "\&" where possible when referring to subroutines
"*foo" is ambiguous in that it may refer to a bareword file handle; so we'll use it where we can without triggering warnings. PublicInbox::TestCommon::run_script_exit required dropping the prototype, however. We'll also future-proof by dropping "use warnings" in Cgit.pm and use the less-ambiguous "//=" in Inbox.pm while we're in the area.
2020-09-10solver: drop warnings, modernize use v5.10.1, use SEEK_SET
With Perl upstream preparing to deprecate things, we'll move towards only enabling warnings during development via shebang and stop enabling them via "use". We'll also favor "use v5.10.1" over the Perl 5.6-compatible "use 5.010_001", since our code base never worked on 5.6. Finally, were also importing SEEK_SET without using it, just use it for readability since we can't avoid loading Fcntl in other places and it'll get constant-folded, anyways.
2020-09-10wwwtext: config comment improvements
Use the full URL of the inbox being mirrored to reduce ambiguity (instead of just the inbox name). Using asymmetric quotes (e.g `foo') improves readability for me in that it's more obvious when a quote begins and ends. It also lights up fewer pixels and reduces visual noise compared to double-quotes. We'll also reflow the `mainrepo' vs `inboxdir' comment slightly to emphasize the word `instead'.
2020-09-10wwwtext: don't blindly quote "git clone" destination
Save screen space and light up fewer pixels to reduce visual noise.
2020-09-10wwwtext: describe the use of `coderepo' entries
The `solver' feature is not very obvious, give potential users a hint about it.
2020-09-10nntp: fix cross-newsgroup Message-ID lookups
We cannot blindly use the selected newsgroup for HEAD/ARTICLE/BODY requests using Message-ID, since those commands look across all newsgroups; not just the selected one (if any). So stuff a reference to the Inbox object into $smsg. We can reduce args passed into set_nntp_headers() and msg_hdr_write(), too. Fixes: 0e6ceff37fc38f28 ("nntp: support slow blob retrievals")
2020-09-09wwwstream: fix "Atom feed" link
Oops, I wanted to stop escaping double-quotes with `qq()' but used `q()' instead :x Fixes: 2f61828fcb727e51 ("www: make mirror instructions more prominent")
2020-09-09contrib/css: limit <a> coloring to links, only
We don't want <a> tags without href= attributes to be colored, since the `<a id=mirror>' tag in the HTML footer is intended as an anchor destination for `<a href=#mirror>' link at the top.
2020-09-09www: make mirror instructions more prominent
In order to fight the misconception that public-inboxes are centralized, anchor "#mirror" to the clone instructions and place an emphasis on "mirror", not just cloning. While we're at it, better describe multi-epoch -V2 inboxes, since some users do not seem to realize epochs consist of different data.
2020-09-03v2writable: reuse read-only shard counting code
We'll also fix the read-only code to ensure we notice missing Xapian shards, since gaps would throw off our expectation that Xapian document IDs and NNTP article numbers are interchangeable.
2020-09-03overidx: document column uses
This may be useful for keeping our heads on straight dealing with IMAP, NNTP, JMAP, etc.
2020-09-03wwwaltid: drop unused sqlite3_missing function
It's inlined into the main function, which we'll shorten slightly with the defined-or (`//') operator. Also noticed and fixed a mismatched HTML tag.