about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-10-31lei_input: disallow uppercase characters for labels
Xapian boolean terms rely on upper-case prefixes, so the terms themselves need to be all lowercase.
2021-10-31doc: add lei-mail-sync-overview manpage
Mostly illustrating how clunky the process is :p We'll also tweak some things in existing man pages around mail synchronization.
2021-10-30doc: lei-security: add a note about core dumps
Maybe we can avoid them if we stop having buggy code :P
2021-10-30lei_to_mail: avoid SEGV on worker exit via SIGTERM
->DESTROY ordering via "exit()" calls is tricky, and dedupe checks were causing problems. AFAIK, this only affects users who manually enable WAL on lei/store/ei*/over.sqlite3. Fortunately, there is no data corruption as a result even though "read-only" WAL requires write permissions.
2021-10-30lei_xsearch: quiet error message on SIG{PIPE,TERM}
SIGPIPE and SIGTERM are common and user-induced, so they're not worth warning on. Add the value of "$?", though, since it can help users notice other errors (e.g. SIGSEGV).
2021-10-30lei_to_mail: limit workers for text, reply and v2 outputs
"text" and "reply" outputs are intended for the pager, so parallelizing them is a waste of resources. v2 has shards, of course, so parallelizing writes to it is also a waste since the deduplication work is a bit more complex.
2021-10-30lei: do not access {sock} after SIGPIPE
It's possible for this to break out of the event loop if note_sigpipe fires via PktOp in the same iteration.
2021-10-28test_common: clear XDG_CACHE_HOME before lei tests
We don't want to read a users' $XDG_CACHE_HOME/lei/all_locals_ever.git during tests. Reported-by: Thomas Weißschuh <thomas@t-8ch.de> Tested-by: Thomas Weißschuh <thomas@t-8ch.de> Link: https://public-inbox.org/meta/f239abac-4aee-4573-a0d6-e533c7a32662@t-8ch.de/
2021-10-28lei rm: move generic input_maildir_cb to LeiInput parent class
It's not much of a savings, right now, but maybe it can be in the future. I wanted to eliminate the "lei convert" one, too, but convert needs to preserve keywords which isn't possible with the generic fallback, so new tests were written for convert, instead.
2021-10-28lei sucks: show nproc in CPU info
Some bugs are triggered with more CPUs, some with 1 CPU.
2021-10-28doc: lei-add-watch: add warning about unreliability
This needs work at some point in the future.
2021-10-28lei convert: remove redundant input_net_cb
Use the one provided by the LeiInput parent class.
2021-10-28doc: lei blob: wording fixups, describe --remote
There's no current way to retrieve blobs by OID directly from remote externals. Maybe the $INBOX_NAME/$OID/s/raw.eml endpoint could be overloaded for that.
2021-10-28doc: lei-convert: various updates and cleanups
Note that "-o OUTPUT" is required in the synopsis. Leave out "eml:" for now since it doesn't work as an output and I doubt anybody would use it as a prefix, and it's not really useful. --no-import-remote is also not accepted by convert, since it doesn't touch lei/store at all.
2021-10-28lei convert: use "--output" in failure message
The extra dashes should help users find the correct option more easily.
2021-10-28xt/net_writer_imap: test "lei convert" w/ IMAP source
I just did a double-take and nearly thought authentication was broken while reading LeiConvert.pm. Add a comment in LeiConvert.pm to clarify things, too.
2021-10-28lei add-watch: ensure folders are known to mail_sync.sqlite3
This prevents noisy errors in syslog when running t/lei-watch.t
2021-10-27lei q: fix remote import accounting
We need to update the {-nr_remote_eml} counter regardless of progress display being enabled since it's needed for saved searches. We'll also split out the {-imported} flag separately and only call LeiStore->done if a new message was imported. Note: this change is NOT expected to fix errors reported by Thomas in <ebf92218-1470-4602-b534-6dae59639dc6@t-8ch.de> Cc: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-27test_common: key test inboxes to init.defaultBranch
This lets users change their global init.defaultBranch config knob in ~/.gitconfig or similar without breaking tests. Reported-by: Thomas Weißschuh <thomas@t-8ch.de> Tested-by: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-27lei mail-diff: support more inputs, split newlines
Support --in-format like the rest of LeiInput users, and don't default to .eml if a per-input format was specified. In any case, I saved a bunch of messages from mutt which uses mboxcl2. We'll also split newlines for diff, since it's a pain to read diffs with escaped "\n" characters in them.
2021-10-26t/lei-watch: add diagnostics for failure
I just got a difficult-to-reproduce failure, here; so there's still some issues with the up-to-dateness of the inotify watcher.
2021-10-26lei_to_mail: only run lms_write_prepare for IMAP+Maildir
Mail synchronization in lei_to_mail only works for IMAP and Maildir; so don't waste time preparing mbox* writers for it.
2021-10-26input_pipe: account for undefined {sock}
It's possible for ->event_step to fire twice due to ->requeue with EPOLLET (but not EPOLLONESHOT). So account for that and avoid causing event loop errors as a result.
2021-10-26lei rm|tag: drop redundant mbox+net callbacks
These are supplied by the base LeiInput class
2021-10-26lei p2q: use LeiInput for multi-patch series
The LeiInput backend now allows p2q to work like any other command which reads .eml, .patch, mbox*, Maildir, IMAP, and NNTP input. Running "git format-patch --stdout -1 $COMMIT" remains supported. This is intended to allow lower memory use while parsing "git log --pretty=mboxrd -p" output. Previously, the entire output of "git log" would be slurped into memory at once. The intended use is to allow easy(-ish :P) searching for unapplied patches as documented in the new example in the manpage.
2021-10-26lei: add net getopt spec to various commands
All of these commands should support --proxy, at least, if not other curl options.
2021-10-26lei inspect: fix atfork hook
The misnamed sub wasn't firing, but was unlikely to be noticeable given the short lifetime of the process. Fixes: 1f887bd51d92b0d4 ("lei inspect: add atfork hook")
2021-10-26lei q: enable expensive Xapian flags
FLAG_PURE_NOT is too expensive for public-facing WWW use, but lei isn't public-facing. We'll also unconditionally enable phrase search on old "chert" DBs since lei doesn't need to worry about fairness across 10K users.
2021-10-26eml: keep body if no headers are found
This easily allows us to treat "git diff" output as header-less "messages" for commands such as "lei p2q".
2021-10-26doc: lei-store-format: mail sync section, update IPC
mail_sync.sqlite3 needs to be documented, and brings the IPC section up-to-date while we're in the area.
2021-10-26doc: tuning: additional notes for many inboxes
-extindex is the most important piece for dealing with many inboxes, so note it first. Also, frequent use of "git gc" is important for both loose object performance and reducing memory mappings.
2021-10-26lei p2q: document --uri, add examples
This is useful for users lacking in local storage. Also, referencing lei-add-external(1) seems to make less sense than referencing lei-q(1). We'll also start dropping years from the copyright statement to reduce future churn.
2021-10-26www: mirror: fix rendering of NNTP URLs
As of commit 738c4a65, the code for reporting NNTP information in _/text/mirror/ incorrectly uses ->imap_url rather than ->nntp_url. Fixes: 738c4a65719e6278 ("www: various help text updates")
2021-10-25t/index-git-times: support non-master default branch
2021-10-25lei_to_mail: write directly to mail_sync.sqlite3
No need to go through the lei/store process when we write mail_sync.sqlite3. This ought to reduce ENOBUFS errors (and the sleep workaround) on RAM-starved systems.
2021-10-25contrib/css/216light: add more contrast to foreground text
333 on dimmed displays doesn't show up well. I still find 000 foregrounds too harsh, though, but 003 is available. It seems dark enough to not cause problems while not being too harsh. 003 should be available on more displays, even, and could fit a 22-color "safest" color scheme.
2021-10-25www: $MSGID/raw: set charset in HTTP response
By using the charset specified in the message, web browsers are more likely to display the raw text properly for human readers. Inspired by a patch by Thomas Weißschuh: https://public-inbox.org/meta/20211024214337.161779-3-thomas@t-8ch.de/ Cc: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-25gzip_filter: delay async wcb call
This will let us modify the response header later to set a proper charset for Content-Type when displaying raw messages. Cc: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-24t/git: support non-master default branch
2021-10-24t/watch_maildir: support non-master default branch
2021-10-24viewvcs: die on tmpfile() errors
Just let Plack::Util::run_app catch the error and generate a 500 response for it.
2021-10-24git: avoid Perl5 internal scratchpad target cache
Creating a scalar ref directly off substr() seemed to be causing the underlying non-ref scalar to end up in Perl's scratchpad. Assign the substr result to a local variable seems sufficient to prevent multi-megabyte SVs from lingering indefinitely when a read-only daemon serves rare, oversized blobs.
2021-10-24thread: avoid Perl5 internal scratchpad target cache
The use of array-returning built-ins such as `grep' inside arrayref declarations appears to result in permanently allocated scratchpad space for caching according to my malloc inspector. Thread skeletons get discarded every response, but multiple skeletons can exist in memory at once, so do what we can to prevent long-lived allocations from being made, here. In other words, replacing constructs such as: my $foo = [ grep(...) ]; with: my @foo = grep(...); Seems to ensure the mortality of the underlying array.
2021-10-24listener: emit warnings on EPERM
In retrospect, warnings for EPERM on accept4(2) failure may help detect misconfigured firewalls, so start emitting warnings for EPERM. Fwiw, I've never known excessive EPERM warnings to be excessively noisy in other TCP services I've run over the years.
2021-10-24http: use a larger buffer for ->getline responses
64K matches the Linux pipe default, and matches what we use in httpd/async and qspawn. This should reduce syscalls used for serving git packs via dumb HTTP and any ->getline code paths used by other PSGI code. This appears to speed up HTML rendering by w3m when serving giant HTML responsees from the Devel::Mwrap::PSGI memory debugger.
2021-10-24shared_kv: remove cache_size attribute support
We're not using it, anywhere.
2021-10-24lei export-kw: skip read-only IMAP folders
Since we want to store IMAP flags asynchronously and not wait for results, we can't check for IMAP errors this way and end up wasting bandwidth on public-inbox-imapd. Now, we just check PERMANENTFLAGS up front to ensure a folder can handle IMAP flag storage before proceeding.
2021-10-24lei: always pass $lei to LeiAuth->op_merge
This will make future developments easier.
2021-10-23cmd_ipc4: retry sendmsg on ENOBUFS/ENOMEM/ETOOMANYREFS
I'm seeing ENOBUFS on a RAM-starved system, and slowing the sender down enough for the receiver to drain the buffers seems to work. ENOMEM and ETOOMANYREFS could be in the same boat as ENOBUFS. Watching for POLLOUT events via select/poll/epoll_wait doesn't seem to work, since the kernel can already sleep (or return EAGAIN) for cases where POLLOUT would work.
2021-10-23www: respect coderepo.*.url during cgit init
This is necessary for showing "found $OID in $CODEREPO_URL" in solver-generated pages ($INBOX_URL/$OID/s/).