about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-10-16t/lei*: set EDITOR for dumb terminals
Running tests over a non-interactive ssh session fails, otherwise.
2021-10-16doc: lei: add manpages for remaining commands
At this point all of the current lei commands, aside from -help and -sucks, should be covered.
2021-10-16doc: lei: restore alphabetical order to some listings
Most the lei-related entries in txt2pre and Makefile.PL are in alphabetical order. Reorder the few that aren't. While at it, reflow the Makefile.PL entries in preparation for the entries that will be added in the next commit.
2021-10-16extindex: avoid triggering a buggy unref
We can't attempt to unref messages beyond the highwater mark of an inbox. This bugfix was found by commit c485036d0b1ce7ed (extindex: guard against buggy unrefs, 2021-10-14), which actually did its intended job and guarded against a buggy unref.
2021-10-16httpd/async: switch to level-triggered epoll
We'll save ourselves some code here and let the kernel do more work, instead.
2021-10-16inbox + search: use 5.10.1 and do some golfing
Some yak-shaving while I try to track down other bugs...
2021-10-16lei_to_mail: quiet down abort messages
We don't need to flood the terminal with "W: $oid is (!= blob)\n" messages when somebody nukes a git cat-file process from under us.
2021-10-16lei_overview: die rather than lei->fail
This will make our code more flexible in case it gets used in non-lei things.
2021-10-16extindex: prune invalid alternate entries on --gc
Seeing the same warning over and over again gets annoying.
2021-10-16lei: more eval guards for die on failure
Relying on $lei->fail is unsustainable since there'll always be parts of our code and dependencies which can trigger die() and break the event loop.
2021-10-16lei: always keep cwd fd {3} for ->fchdir
The extra FD shouldn't cause noticeable overhead in short-lived workers, and it lets us simplify lei->rel2abs. Get rid of a 2-argument form of open() while we're at it, since it's been considered for warning+deprecation by Perl for safety reasons.
2021-10-16lei: golf PATH2CFG cleanup
More code means more bugs.
2021-10-16httpd: move pipeline logic into event_step
Most of the HTTP server code was written for Danga::Socket and not fully-transitioned to take advantage of PublicInbox::DS. This change brings it up-to-date with the style of pipeline handling used for -imapd and -nntpd.
2021-10-16imapd+nntpd: drop timer-based expiration
It's needlessly complex and O(n), so it doesn't scale well to a high number of clients nor is it easy-to-scale with the data structures available to us in pure Perl. In any case, I see no evidence of either -imapd nor -nntpd experiencing high connection loads on public-facing sites. -httpd has never had its own timer-based expiration, either. Fwiw, public-inbox.org itself has been running a public-facing HTTP/HTTPS server with no userspace idle client expiration for the past 8 years or with no ill effect. Clients can come and go as they wish, and SO_KEEPALIVE takes care of truly broken connections if they're gone for ~2 hours. Internet connections drop all time, so it should be harmless to drop connections w/o warning since both NNTP and IMAP protocols have well-defined semantics for determining if a message was truncated (as does HTTP/1.1+).
2021-10-16dir_idle: do not add watches in ->new
There's no savings in having two ways to add watches to an inotify nor kqueue descriptor.
2021-10-16smsg: add ->oidbin method
This makes some of our code less noisy by reducing the amount of pack('H*', ...) use.
2021-10-15lei q: guard query_done against die()
v2w->wq_do('done') may die on I/O errors, and likely other places. Just guard the entire block with an eval and ->fail as appropriate.
2021-10-15lei forget-search: support multiple args
I've been testing a lot of searches which I don't want to keep around, so make it easy to remove a bunch at once. We'll behave like rm(1) and keep going in the face of failure.
2021-10-15lei note-event: fix explicit flush reliability
We need to send the socket over to lei/store and wait for the kernel to drop the socket refcount down to zero before script/lei can exit. This is not a new bug and only caused very sporadic test failures. I only noticed it while simplifying IPC stuff.
2021-10-15lei + ipc: simplify process reaping
Simplify our APIs and force dwaitpid() to work in async mode for all lei workers. This avoids having lingering zombies for parallel searches if one worker finishes soon before another. The old distinction between "old" and "new" workers was needlessly complex, error-prone, and embarrasingly bad. We also never handled v2:// writers properly before on Ctrl-C/Ctrl-Z (SIGINT/SIGTSTP), so add them to @WQ_KEYS to ensure they get handled by $lei when appropropriate.
2021-10-15lei forget-search: fix for symlink-ed paths
If lei up and edit-search work on something, so should forget-search.
2021-10-15lei q: avoid kw lookup failure on remote mboxrd
When importing several sources in parallel via http(s) mboxrd, we need to be able to get keywords of uncommitted documents directly from shard workers. Otherwise, Xapian DocNotFound errors happen because the read-only LeiSearch won't see documents from uncomitted transactions. Keep in mind that it's possible the keywords can be changed on-the-fly even for uncommitted documents because of inotify watches from LeiNoteEvent.
2021-10-15www: various help text updates
`dt:' documentation is redundant with `d:' approxidate support; so drop `dt:' since mairix uses `d:'. We'll also document `rt:' since there are legit messages from senders with broken clocks. Reduce indentation level of help texts to be in 2-space increments to using too much horizontal space. We'll always place IMAP ahead of NNTP since it's alphabetical and there's likely more IMAP clients out there. Add "--ng NEWSGROUP" to -init instructions if configured. There's also some minor wording changes throughout.
2021-10-15lei up --all: send signals to workers, receive errors
The redispatch mechanism wasn't routing signals and messages between redispatched workers and script/lei properly. We now rely on PktOp to do bidirectional message forwarding and carefully avoiding circular references by using PktOp.
2021-10-15lei up: actually rely on DESTROY for --alllll
We need to use DESTROY here to ensure we wait for workers, too; not just the initial dispatch. Fixes: cafbd77b3c82167d ("lei up: avoid excessively parallel --all")
2021-10-15lei: TSTP affects all curl and related subprocesses
By relying more on pgroups for remaining remaining processes, this lets us pause all curl+tail subprocesses with a single kill(2) to avoid cluttering stderr. We won't bother pausing the pigz/gzip/bzip2/xz compressor process not cat-file processes, though, since those don't write to the terminal and they idle soon after the workers react to SIGSTOP. AutoReap is hoisted out from TestCommon.pm. CLONE_SKIP is gone since we won't be using Perl threads any time soon (they're discouraged by the maintainers of Perl).
2021-10-15git: cat-file --batch are their own pgrp
We want these long-lived processes to die naturally when their parent dies. Hopefully this improves graceful shutdown for -extindex because I'm interrupting a lot of reindexing...
2021-10-15git: ->fail invokes current callback
While we try to invoke all pending callbacks to force error handling, the current callback wasn't getting invoked on invoked on async_abort if my_read/my_readline failed.
2021-10-15git: async_err shows retried requests properly
We make $req a reference upon retrying, but "SCALAR(...)" in error messages isn't helpful, so dereference the scalar ref.
2021-10-15lei: use send() perlop for signals
This may save us a small bit of startup time since there's fewer args and opcodes should be smaller.
2021-10-15lei: give workers their own process group
This lets users Ctrl-Z from their terminal to pause an entire git-clone process hierarchy.
2021-10-14lei: -d (--dir) and -O (only) shortcuts
`-d' seems like a non-brainer for --dir with inspect. I find myself using `--only' a bit, too, and `-O' seems like a reasonable shortcut for it.
2021-10-14lei add-external --mirror: respect client umask
While lei is intended for non-public mail and runs umask(077) by default, externals are one area which can safely defer to the user's umask. Instead of sending it unconditionally with every command, only have lei-daemon request it when necessary.
2021-10-14clone+fetch: respect umask for all downloaded files
Since public inboxes are usually intended to be public, the File::Temp default permission of 0600 is wrong. Just respect the user's umask in this case as git-clone does. This doesn't work for "lei add-external --mirror", yet; but it will...
2021-10-14lei inspect: account for non-extindex inboxes
Inbox->xdb does not exist, but this code path was apparently never tested :x I noticed this on basic v2 inbox, but it could happen with any v1/v2 inbox. Move ->num2docid into Search so it's less awkward to use.
2021-10-14extindex: guard against buggy unrefs
I noticed some unref messages which shouldn't have been happening, but they were. Which is troubling. So add a guard around an unref path until we can get to the bottom of this.
2021-10-13fetch: support --try-remote/-T for alternate remote names
This allows -fetch to work out-of-the-box on using the grokmirror 2.x default of "_grokmirror".
2021-10-13t/nntpd-tls: change diag() to like() assertion
This test wasn't finished when I initially wrote it :x
2021-10-13t/git: avoid "once" warning for async_warn
No point in testing use_ok when we have no outside dependencies nor exports in this case.
2021-10-13t/lei-mirror: avoid reading ~/.public-inbox/config in test
Oops, we shouldn't attempt to read a users' actual HOME when running -index, since mine has a bunch of invalid entries in there.
2021-10-13eml: avoid Encode 2.87..3.12 leak
Encode::FB_CROAK leaks memory in old versions of Encode: <https://rt.cpan.org/Public/Bug/Display.html?id=139622> Since I expect there's still many users on old systems and old Perls, we can use "$SIG{__WARN__} = \&croak" here with Encode::FB_WARN to emulate Encode::FB_CROAK behavior.
2021-10-13t/www_listing: require opt-in for grokmirror tests
grokmirror 2.x seems to idle in several places for 5s at-a-time, causing t/www_listing.t to take longer than "make check-run" on a 4-core system when run without grokmirror. So make it optional but add some test knobs to allow tailing the log output so I can see what's going on.
2021-10-13test_common: hoist out tail_f sub
We'll be reusing this in more places. While we're at it, allow it to tail all run_script() users, including lei() in TestCommon.
2021-10-13xt/perf-msgview: drop unnecessary use_ok
require_mods covers it, and we're not testing Plack itself.
2021-10-13www: preload: load ExtSearch via ->ALL
This ought to give us more CoW savings and fragmentation avoidance in -httpd.
2021-10-13extindex: set {current_info} in eidxq processing
This gives context as to where warnings are coming from.
2021-10-13treewide: use warn() or carp() instead of env->{psgi.errors}
Large chunks of our codebase and 3rd-party dependencies do not use ->{psgi.errors}, so trying to standardize on it was a fruitless endeavor. Since warn() and carp() are standard mechanism within Perl, just use that instead and simplify a bunch of existing code.
2021-10-13lei: use standard warn() in more places
warn() is easier to augment with context information, and frankly unavoidable in the presence of 3rd-party libraries we don't control.
2021-10-13extindex: show OID on bad blob failure
AFAIK I've never hit these messages, but I might be glad if I ever do.
2021-10-13daemon: set $SIG{__WARN__} properly
Eml->warn_ignore_cb itself returns a callback, so creating a reference to it was wrong when assigning it to $SIG{__WARN__}; Fixes: 176cd51f9aa81b74 ("daemon: quiet down Eml-related warnings")