about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-10-26lei p2q: use LeiInput for multi-patch series
The LeiInput backend now allows p2q to work like any other command which reads .eml, .patch, mbox*, Maildir, IMAP, and NNTP input. Running "git format-patch --stdout -1 $COMMIT" remains supported. This is intended to allow lower memory use while parsing "git log --pretty=mboxrd -p" output. Previously, the entire output of "git log" would be slurped into memory at once. The intended use is to allow easy(-ish :P) searching for unapplied patches as documented in the new example in the manpage.
2021-10-26lei: add net getopt spec to various commands
All of these commands should support --proxy, at least, if not other curl options.
2021-10-26lei inspect: fix atfork hook
The misnamed sub wasn't firing, but was unlikely to be noticeable given the short lifetime of the process. Fixes: 1f887bd51d92b0d4 ("lei inspect: add atfork hook")
2021-10-26lei q: enable expensive Xapian flags
FLAG_PURE_NOT is too expensive for public-facing WWW use, but lei isn't public-facing. We'll also unconditionally enable phrase search on old "chert" DBs since lei doesn't need to worry about fairness across 10K users.
2021-10-26eml: keep body if no headers are found
This easily allows us to treat "git diff" output as header-less "messages" for commands such as "lei p2q".
2021-10-26doc: lei-store-format: mail sync section, update IPC
mail_sync.sqlite3 needs to be documented, and brings the IPC section up-to-date while we're in the area.
2021-10-26doc: tuning: additional notes for many inboxes
-extindex is the most important piece for dealing with many inboxes, so note it first. Also, frequent use of "git gc" is important for both loose object performance and reducing memory mappings.
2021-10-26lei p2q: document --uri, add examples
This is useful for users lacking in local storage. Also, referencing lei-add-external(1) seems to make less sense than referencing lei-q(1). We'll also start dropping years from the copyright statement to reduce future churn.
2021-10-26www: mirror: fix rendering of NNTP URLs
As of commit 738c4a65, the code for reporting NNTP information in _/text/mirror/ incorrectly uses ->imap_url rather than ->nntp_url. Fixes: 738c4a65719e6278 ("www: various help text updates")
2021-10-25t/index-git-times: support non-master default branch
2021-10-25lei_to_mail: write directly to mail_sync.sqlite3
No need to go through the lei/store process when we write mail_sync.sqlite3. This ought to reduce ENOBUFS errors (and the sleep workaround) on RAM-starved systems.
2021-10-25contrib/css/216light: add more contrast to foreground text
333 on dimmed displays doesn't show up well. I still find 000 foregrounds too harsh, though, but 003 is available. It seems dark enough to not cause problems while not being too harsh. 003 should be available on more displays, even, and could fit a 22-color "safest" color scheme.
2021-10-25www: $MSGID/raw: set charset in HTTP response
By using the charset specified in the message, web browsers are more likely to display the raw text properly for human readers. Inspired by a patch by Thomas Weißschuh: https://public-inbox.org/meta/20211024214337.161779-3-thomas@t-8ch.de/ Cc: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-25gzip_filter: delay async wcb call
This will let us modify the response header later to set a proper charset for Content-Type when displaying raw messages. Cc: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-24t/git: support non-master default branch
2021-10-24t/watch_maildir: support non-master default branch
2021-10-24viewvcs: die on tmpfile() errors
Just let Plack::Util::run_app catch the error and generate a 500 response for it.
2021-10-24git: avoid Perl5 internal scratchpad target cache
Creating a scalar ref directly off substr() seemed to be causing the underlying non-ref scalar to end up in Perl's scratchpad. Assign the substr result to a local variable seems sufficient to prevent multi-megabyte SVs from lingering indefinitely when a read-only daemon serves rare, oversized blobs.
2021-10-24thread: avoid Perl5 internal scratchpad target cache
The use of array-returning built-ins such as `grep' inside arrayref declarations appears to result in permanently allocated scratchpad space for caching according to my malloc inspector. Thread skeletons get discarded every response, but multiple skeletons can exist in memory at once, so do what we can to prevent long-lived allocations from being made, here. In other words, replacing constructs such as: my $foo = [ grep(...) ]; with: my @foo = grep(...); Seems to ensure the mortality of the underlying array.
2021-10-24listener: emit warnings on EPERM
In retrospect, warnings for EPERM on accept4(2) failure may help detect misconfigured firewalls, so start emitting warnings for EPERM. Fwiw, I've never known excessive EPERM warnings to be excessively noisy in other TCP services I've run over the years.
2021-10-24http: use a larger buffer for ->getline responses
64K matches the Linux pipe default, and matches what we use in httpd/async and qspawn. This should reduce syscalls used for serving git packs via dumb HTTP and any ->getline code paths used by other PSGI code. This appears to speed up HTML rendering by w3m when serving giant HTML responsees from the Devel::Mwrap::PSGI memory debugger.
2021-10-24shared_kv: remove cache_size attribute support
We're not using it, anywhere.
2021-10-24lei export-kw: skip read-only IMAP folders
Since we want to store IMAP flags asynchronously and not wait for results, we can't check for IMAP errors this way and end up wasting bandwidth on public-inbox-imapd. Now, we just check PERMANENTFLAGS up front to ensure a folder can handle IMAP flag storage before proceeding.
2021-10-24lei: always pass $lei to LeiAuth->op_merge
This will make future developments easier.
2021-10-23cmd_ipc4: retry sendmsg on ENOBUFS/ENOMEM/ETOOMANYREFS
I'm seeing ENOBUFS on a RAM-starved system, and slowing the sender down enough for the receiver to drain the buffers seems to work. ENOMEM and ETOOMANYREFS could be in the same boat as ENOBUFS. Watching for POLLOUT events via select/poll/epoll_wait doesn't seem to work, since the kernel can already sleep (or return EAGAIN) for cases where POLLOUT would work.
2021-10-23www: respect coderepo.*.url during cgit init
This is necessary for showing "found $OID in $CODEREPO_URL" in solver-generated pages ($INBOX_URL/$OID/s/).
2021-10-23config: remove *_url_format support for cgit
We're not using them, anywhere.
2021-10-23git: simplify local_nick, avoid "foo.git.git"
We need to use a non-greedy regexp to avoid capturing the ".git" suffix in the pathname before blindly appending our own.
2021-10-23t/v2index-late-dedupe: don't read user's ~/.public-inbox/config
Otherwise things can get noisy if bad entries exist in that file, because they do.
2021-10-23searchidx: v1: raise on msgmap init failure
Indexing any inboxes requires SQLite and msgmap, so don't hide exceptions if it fails.
2021-10-23doc: lei-forget-search: fix option name in --prune description
Fixes: 6f8e16a266b30819 ("lei forget-search: support --prune=<local|remote>")
2021-10-22lei forget-search: support --prune=<local|remote>
Instead of: lei forget-search $OUTPUT && rm -r $OUTPUT we'll also allow a user to do: rm -r $OUTPUT && lei forget-search --prune This gives users flexibility to choose whatever flow is most natural to them.
2021-10-22lei export-kw: completion returns all Maildir+IMAP
It's theoretically possible an AUTH=ANONYMOUS login could be writable and allowed to store flags for various people (e.g. within a private network).
2021-10-22lei export-kw: don't recreate deleted IMAP folders
In case an IMAP folder is deleted, just set an error and ignore it rather than creating an empty folder which we attempt to export keywords to for non-existent messages.
2021-10-22wwwatomstream: call gmtime with scalar
When the gmtime() calls were moved from feed_entry() and atom_header() into feed_updated() in c447bbbd, @_ rather than a scalar was passed to gmtime(). As a result, feed <updated> values end up as "1970-01-01T00:00:00Z". Switch back to using a scalar argument to restore the correct timestamps. Fixes: c447bbbddb4ac8e1 ("wwwatomstream: simplify feed_update callers")
2021-10-22lei: use RENAME_NOREPLACE on Linux 3.15+
One syscall is better than two for atomicity in Maildirs. This means there's no window where another process can see both the old and new file at the same time (link && unlink), nor a window where we might inadvertantly clobber an existing file if we were to do `stat && rename'.
2021-10-22lei_mail_sync: mv_src: use transaction, check UNIQUE
We need a transaction across two SQL statements so readers (which don't use flock) will see the result as atomic. This may help against some occasional test failures I'm seeing from t/lei-auto-watch.t and t/lei-watch.t, or make the problem more apparent.
2021-10-22lei: no Perl FileHandle for `undef' w/ ECONNRESET
Error reporting for recv_cmd4 methods is a bit wonky.
2021-10-22dir_idle: treat IN_MOVED_FROM as a gone event
Whether an MUA uses rename(2) or link(2)+unlink(2) combination should not matter to us. We should be able to handle both cases.
2021-10-22lei note-event: clear_src on ENOENT
When a file goes away, try to make sure we don't waste time trying to access or store it.
2021-10-22doc: lei-overview: add CAVEATS section
IMAP and NNTP client performance absolutely sucks compared to what the read-only daemons are capable of...
2021-10-22watch: remove redundant signal mask manipulation
The top-level daemon process already blocks all signals, so there's no reason to block them around fork() calls.
2021-10-22watch: check for {quit} before IDLE
This may make it less likely for watch-dependent tests to get stuck. Unfortunately, due to the synchronous API of Mail::IMAPClient, ->idle is still susceptible to missing signals.
2021-10-22lei_search: try harder to associate "lei index"-ed messages
Allow checking for keyword changes if we have an known OID, even if the blob isn't currently reachable.
2021-10-22lei note-event: wq_io_do => wq_do
No need to pass extra arrayref args, here.
2021-10-22lei note-event: drop unnecessary eval guard
We don't want to lose the failure message in case note-event fails.
2021-10-22lei/store: check for any unexpected process death
The lei/store process should only exit from EOF on the socket, so make sure we note any unintended signals
2021-10-22t/lei-p2q: extra diagnostics
I got one mysterious test failure here, once, and can't seem to reproduce it...
2021-10-22t/lei-import-maildir: rename fix (SR -> RS)
While it doesn't matter to us, the Maildir spec specifies characters are to be sorted in alphabetical order.
2021-10-22t/lei-{auto-watch,export-kw}: extra diagnostics on failure
Maybe these will help track down some failures and make diagnosing bugs easier. "lei export-kw" should also become optional, even, so allow disabling it easily in the test.