about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-11-04public-inbox 1.7.0 v1.7.0
2021-11-04doc: relnotes: a few more 1.7.0 related updates
Note "--all" for -extindex, and some minor wording fixes.
2021-11-04AUTHORS: clarify my title
Being an anti-centralization, anti-authority project; the traditional meaning of "Benevolent Dictator" never sat well with me. Benevolence is relative; and I've never been benevolent towards monopolist-types who try to consolidate power and influence. Power corrupts, after all. In any case, I'll never be more than a random idiot serving data which anybody can mirror and fork.
2021-11-04doc: design_notes: updates for "newer" things
public-inbox-imapd, public-inbox-watch, and marketing.txt all exist, now.
2021-11-04lei_curl: use http.proxy knob via URL match for curl
Using the --proxy on the command-line affects the entire lei invocation, and users searching HTTP(S) remotes and writing to an IMAP folder may want more fine-grained proxy use: lei q -o imap://no-proxy.example/foo -O https://need-proxy.example/bar ...
2021-11-04doc: txt2pre: linkify a add a few more well-known things
Maybe these will help folks less familiar with some of these things.
2021-11-04doc: switch to man(1) for pod => (text|html)
pod2text(1) will wrap long .onion URLs and cause resulting HTML to be linkified improperly.
2021-11-04doc: add more 3rd-party refs, use Debian manpages for xapian
curl, torsocks, and gitglossary manpages are all newly referenced, so make sure they're linkified properly in HTML. We'll be using Debian's manpages as an ad-free, Tor-accessible host for manpages as a fallback since hosting manpages for all 3rd-party projects we reference doesn't scale.
2021-11-03doc: relnotes: 1.7.0: move extindex, note search results change
extindex is a far more important feature than libgit2 support (which is actually underperforming and might go away). The search results page is also improved (IMHO), nowadays.
2021-11-03doc: -clone|lei add-external: add bit about the Makefile
It's pretty useful, I think.
2021-11-03doc: extindex: document current behavior + knobs
I'm not really sure if extindex writing to the config file is a good idea (since -index doesn't, as -init exists). Just document what it does and let the user handle it, since the config file shouldn't be daunting to new users.
2021-11-03doc: lei-q: document SEARCH TERMS prefixes
The new Documentation/common.perl file will be used for all manpages in the future.
2021-11-03doc: txt2pre: add references to newish manpages
2021-11-02lei <rediff|rm|tag>: stdin implies `-F eml'
These commands are usually run on a single message, so saving the user the trouble of typing `-F eml' on the command-line seems reasonable. I don't think commands like "index" and "import" will be too useful for single messages, though.
2021-11-02lei: simplify common LeiInput users with ->wq1_start
This method replaces a common pattern of starting workers, preparing internal auth ops, and asynchronous waiting of command completion. It also adds missing LeiAuth support to rediff and rm which rarely need auth.
2021-11-02lei mail-diff: do not default to 'eml'
In retrospect, this doesn't make sense, since it needs at least two messages to diff. So go about "normal" input rules and require users to specify the format.
2021-11-02t/lei-refresh-mail-sync: speed up test on FreeBSD 12
And improve reliability while we're at it. It seems closing a TCP listen socket on FreeBSD 12.2 doesn't cause connect()-ing clients to fail. This happens regardless of whether a socket is IPv4 or IPv6 This non-failure was causing tests to timeout slowly on the client side instead of failing immediately. We now fork a new process which does nothing but accept() + shutdown() to emulate a dead server. Reliability improves on all OSes since there's never a point in time when another process can bind the socket.
2021-11-02init: respect umask when creating description
I noticed a description for a new inbox had st_mode=0600.
2021-11-02mbox_reader: do not blindly pass --rsyncable to gzip
FreeBSD gzip does not support --rsyncable, though my VM usually has pigz installed.
2021-11-01treewide: kill problematic "$h->{k} //= do {" assignments
As stated in the previous change, conditional hash assignments which trigger other hash assignments seem problematic, at times. So replace: $h->{k} //= do { $h->{x} = ...; $val }; $h->{k} // do { $h->{x} = ...; $hk->{k} = $val }; "||=" is affected the same way, and some instances of "||=" are replaced with "//=" or "// do {", now.
2021-11-01idx_stack: avoid conditional hash assignment weirdness
I've been seeing the following error on occasion during "make check-run": $PWD/t/data-gen/reindex-time-range.v1-master index failed: Modification of a read-only value attempted at $DIR/lib/PublicInbox/SearchIdx.pm line 899, <$r> line 1. Perhaps this fixes it. In any case, a construct of: $h->{k} //= do { $h->{x} = ...; $val }; seems wrong and may cause Perl to error out depending on how hashes are randomized.
2021-11-01doc: lei-config: fix missing =back
2021-11-01doc: update release notes and INSTALL
This is what I can think of at the moment.
2021-10-31lei_input: disallow uppercase characters for labels
Xapian boolean terms rely on upper-case prefixes, so the terms themselves need to be all lowercase.
2021-10-31doc: add lei-mail-sync-overview manpage
Mostly illustrating how clunky the process is :p We'll also tweak some things in existing man pages around mail synchronization.
2021-10-30doc: lei-security: add a note about core dumps
Maybe we can avoid them if we stop having buggy code :P
2021-10-30lei_to_mail: avoid SEGV on worker exit via SIGTERM
->DESTROY ordering via "exit()" calls is tricky, and dedupe checks were causing problems. AFAIK, this only affects users who manually enable WAL on lei/store/ei*/over.sqlite3. Fortunately, there is no data corruption as a result even though "read-only" WAL requires write permissions.
2021-10-30lei_xsearch: quiet error message on SIG{PIPE,TERM}
SIGPIPE and SIGTERM are common and user-induced, so they're not worth warning on. Add the value of "$?", though, since it can help users notice other errors (e.g. SIGSEGV).
2021-10-30lei_to_mail: limit workers for text, reply and v2 outputs
"text" and "reply" outputs are intended for the pager, so parallelizing them is a waste of resources. v2 has shards, of course, so parallelizing writes to it is also a waste since the deduplication work is a bit more complex.
2021-10-30lei: do not access {sock} after SIGPIPE
It's possible for this to break out of the event loop if note_sigpipe fires via PktOp in the same iteration.
2021-10-28test_common: clear XDG_CACHE_HOME before lei tests
We don't want to read a users' $XDG_CACHE_HOME/lei/all_locals_ever.git during tests. Reported-by: Thomas Weißschuh <thomas@t-8ch.de> Tested-by: Thomas Weißschuh <thomas@t-8ch.de> Link: https://public-inbox.org/meta/f239abac-4aee-4573-a0d6-e533c7a32662@t-8ch.de/
2021-10-28lei rm: move generic input_maildir_cb to LeiInput parent class
It's not much of a savings, right now, but maybe it can be in the future. I wanted to eliminate the "lei convert" one, too, but convert needs to preserve keywords which isn't possible with the generic fallback, so new tests were written for convert, instead.
2021-10-28lei sucks: show nproc in CPU info
Some bugs are triggered with more CPUs, some with 1 CPU.
2021-10-28doc: lei-add-watch: add warning about unreliability
This needs work at some point in the future.
2021-10-28lei convert: remove redundant input_net_cb
Use the one provided by the LeiInput parent class.
2021-10-28doc: lei blob: wording fixups, describe --remote
There's no current way to retrieve blobs by OID directly from remote externals. Maybe the $INBOX_NAME/$OID/s/raw.eml endpoint could be overloaded for that.
2021-10-28doc: lei-convert: various updates and cleanups
Note that "-o OUTPUT" is required in the synopsis. Leave out "eml:" for now since it doesn't work as an output and I doubt anybody would use it as a prefix, and it's not really useful. --no-import-remote is also not accepted by convert, since it doesn't touch lei/store at all.
2021-10-28lei convert: use "--output" in failure message
The extra dashes should help users find the correct option more easily.
2021-10-28xt/net_writer_imap: test "lei convert" w/ IMAP source
I just did a double-take and nearly thought authentication was broken while reading LeiConvert.pm. Add a comment in LeiConvert.pm to clarify things, too.
2021-10-28lei add-watch: ensure folders are known to mail_sync.sqlite3
This prevents noisy errors in syslog when running t/lei-watch.t
2021-10-27lei q: fix remote import accounting
We need to update the {-nr_remote_eml} counter regardless of progress display being enabled since it's needed for saved searches. We'll also split out the {-imported} flag separately and only call LeiStore->done if a new message was imported. Note: this change is NOT expected to fix errors reported by Thomas in <ebf92218-1470-4602-b534-6dae59639dc6@t-8ch.de> Cc: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-27test_common: key test inboxes to init.defaultBranch
This lets users change their global init.defaultBranch config knob in ~/.gitconfig or similar without breaking tests. Reported-by: Thomas Weißschuh <thomas@t-8ch.de> Tested-by: Thomas Weißschuh <thomas@t-8ch.de>
2021-10-27lei mail-diff: support more inputs, split newlines
Support --in-format like the rest of LeiInput users, and don't default to .eml if a per-input format was specified. In any case, I saved a bunch of messages from mutt which uses mboxcl2. We'll also split newlines for diff, since it's a pain to read diffs with escaped "\n" characters in them.
2021-10-26t/lei-watch: add diagnostics for failure
I just got a difficult-to-reproduce failure, here; so there's still some issues with the up-to-dateness of the inotify watcher.
2021-10-26lei_to_mail: only run lms_write_prepare for IMAP+Maildir
Mail synchronization in lei_to_mail only works for IMAP and Maildir; so don't waste time preparing mbox* writers for it.
2021-10-26input_pipe: account for undefined {sock}
It's possible for ->event_step to fire twice due to ->requeue with EPOLLET (but not EPOLLONESHOT). So account for that and avoid causing event loop errors as a result.
2021-10-26lei rm|tag: drop redundant mbox+net callbacks
These are supplied by the base LeiInput class
2021-10-26lei p2q: use LeiInput for multi-patch series
The LeiInput backend now allows p2q to work like any other command which reads .eml, .patch, mbox*, Maildir, IMAP, and NNTP input. Running "git format-patch --stdout -1 $COMMIT" remains supported. This is intended to allow lower memory use while parsing "git log --pretty=mboxrd -p" output. Previously, the entire output of "git log" would be slurped into memory at once. The intended use is to allow easy(-ish :P) searching for unapplied patches as documented in the new example in the manpage.
2021-10-26lei: add net getopt spec to various commands
All of these commands should support --proxy, at least, if not other curl options.
2021-10-26lei inspect: fix atfork hook
The misnamed sub wasn't firing, but was unlikely to be noticeable given the short lifetime of the process. Fixes: 1f887bd51d92b0d4 ("lei inspect: add atfork hook")