about summary refs log tree commit homepage
DateCommit message (Collapse)
2020-12-26default to CORE::warn in $SIG{__WARN__} handlers
As with CORE::die and $SIG{__DIE__}, it turns out CORE::warn is safe to use inside $SIG{__WARN__} handlers without triggering infinite recursion. So fall back to reusing CORE::warn instead of creating a new sub.
2020-12-26inbox: name variable for values loop iterator
->on_inbox_unlock callbacks could clobber $_, and this seems to fix a problem with -extindex --watch failing to index some inboxes after SIGHUP reload.
2020-12-26inboxidle: avoid needless syscalls on refresh
We don't have to replace a bunch of existing watches with identical new ones. On Linux with Linux::Inotify2 installed, this avoids a storm of inotify_add_watch(2) and inotify_rm_watch(2) syscalls on SIGHUP with -imapd and "-extindex --watch"
2020-12-26inboxidle: clue users into resolving ENOSPC from inotify
It may not be obvious to users a ENOSPC error is from hitting a (tunable) kernel-imposed limit on inotify watches, and not some storage device running out of space. Give them a hint here to reduce our own support burden.
2020-12-26index: filter out indexlevel=basic from extindex
extindex users will likely want to use indexlevel=basic for per-inbox indices, however extindex itself doesn't support basic index level (yet?). Let's ensure we don't trip up extindex users who specify "-L basic" on the -index command-line.
2020-12-26v2writable: don't verify tip if reindexing
We only rely on git-rev-parse to resolve symbolic names ("HEAD") to a SHA-* git commit ID. We'll assume any git commit IDs we get from SQLite DBs are valid and let "git-log" fail if it isn't.
2020-12-26index: fix --no-fsync flag propagation to extindex
Negation in flag names are confusing, but trying to deviate from the DB_NO_SYNC name used by Xapian is also confusing.
2020-12-26index: do not attach inbox to extindex unless updated
We'll count the number of log changes (regardless of index or unindex) and only attach inboxes to ExtSearchIdx objects when they get new work. We'll also reduce lock bouncing and only update external indices after all per-inbox indexing is done. This also updates existing v2 indexing/unindexing callers to be more consistent and ensures unindex log entries update per-inbox last commit information.
2020-12-26extsearchidx: close DB handles after use if FD constrained
Most distros ship with low RLIMIT_NOFILE limits and surprises may lurk for admins who configure many inboxes. Keep FD usage under control to avoid EMFILE errors at inopportune times during reindex. From what I can tell, this is the only place where extindex can have unpredictable FD growth when there's thousands of inboxes, and it's in an extremely rare code path.
2020-12-26extsearchidx: delay SQLite availability checks
This will make attach_inbox faster for no-op calls. It also helps us avoid races in case msgmap or over.sqlite3 gets unlinked while -extindex is running.
2020-12-26index: disable --fast-noop on --reindex
These options make no sense when used together, just inform the user and move on since it's probably harmless to continue.
2020-12-26init: use the return value of rel2abs_collapsed
:x Fixes: 9fcce78e40b0a7c6 ("script/public-inbox-*: favor caller-provided pathnames")
2020-12-25index: support --fast-noop / -F switch
Note: I'm not sure if it's worth documenting and supporting this long-term. We can can avoid taking locks for invocations of "index --all" and rely on high-resolution ctime (struct timespec st_ctim) comparisons of msgmap.sqlite3 and the packed-refs + refs/heads directory of the newest epoch. This cuts public-inbox-index invocations with "--all --no-update-extindex -L basic" down from 0.92s to 0.31s. The change with "-L medium" or "-L full" and (default) non-zero jobs is even more drastic, reducing a 12-13s no-op invocation down to the same 0.31s
2020-12-25inboxwritable: delay umask_prepare calls
This simplifies all ->with_umask callers and opens the door for further optimizations to delay/elide process spawning.
2020-12-24index: update [extindex "all"] by default, support -E
In most cases, this ensures users will only have to opt-in to using -extindex once and won't have to issue extra commands to keep external indices up-to-date when using public-inbox-index. Since we support arbitrary numbers of external indices for ease-of-development, we'll support repeating "-E" ("--update-extindex=") in case users want to test changes in parallel.
2020-12-23config: config_fh_parse: micro-optimize harder
Instead of relying on split() and a regexp, we'll drop split() entirely and rely on index() + two substr() calls to operate on fixed strings. This brings PublicInbox::Config->new time down from 0.98s down to 0.84s.
2020-12-23config: config_fh_parse: micro-optimize
We can avoid a slow regexp capture and instead and rely on rindex + substr to extract the section from the config file. Then we use the defined-or-assignment (//=) operator combined with the documented return value of `push' to ensure @section_order is unique without repeating a hash lookup. Finally, we avoid short-lived variables inside the loop and declare them subroutine-wide to knock a teeny bit of allocation time. Combined, these optimizations bring the ~1.22s PublicInbox::Config->new time down to ~0.98s with 50K inboxes.
2020-12-23config: git_config_dump: pre-compile RE for split
It appears the Perl split() operator is not optimized for fixed strings at all. With this change, PublicInbox::Config->new (w/o ->fill_all) time is reduced from 1.81s to 1.22s on a config file with 50K inboxes.
2020-12-23config: _fill: inbox name extraction optimization
Using substr() instead of a string copy + s// substitution here reduces ->fill_all from 4.00s to 3.88s with 50K inboxes on my workstation.
2020-12-23extsearchidx: close SQLite handles after attaching
This is needed to prevent us from running out of FDs when indexing many inboxes. Perhaps checking these on attach_inbox is unnecessary and may be removed entirely down the line.
2020-12-23miscsearch: index UIDVALIDITY, use as startup cache
This brings -nntpd startup time down from ~35s to ~5s with 50K inboxes. Further improvements ought to be possible with deeper changes to MiscIdx, since -mda having to load every inbox seems unreasonable; but this general change is fairly unintrusive.
2020-12-23inboxwritable: _init_v1: set created_at ASAP
This ensures we have UIDVALIDITY to index earlier rather than later for v1 inboxes, matching v2 behavior.
2020-12-23inbox: git_epoch: correct false comment
The original comment hasn't been true since PublicInbox::Git->modified was changed to use cat_async blob responses. In any case, manifest.js.gz generation already cleans up per-epoch git processes used for ->modified.
2020-12-23xt: add create-many-inboxes helper test
I've been using something like this to mock out thousands of inboxes for testing.
2020-12-23miscsearch: load Xapian at initialization
We need Xapian bindings loaded before calling (Search::)Xapian::Database->new
2020-12-22wwwstream: show relative coderepo URLs correctly
Trying to link "foo.git" relative to the current URL usually does not provide correct results, so prefix it by going into the parent directory if an absolute (or protocol-relative) URL is not supplied.
2020-12-22admin: resolve inboxes to absolute paths for index
Some of my ancient v1-only scripts called public-inbox-index to operate on GIT_DIR: GIT_DIR=/path/to/foo.git public-inbox-index This change ensures they keep working, otherwise "." will be passed to the --git-dir= switch of git(1) because that's the default directory if no inboxes are specified on the command-line. Fixes: 9fcce78e40b0a7c6 ("script/public-inbox-*: favor caller-provided pathnames")
2020-12-22public-inbox-v[12]-format.pod: make lexgrog happy
The Debian package linter (lintian) emits the following warning: W: bad-whatis-entry N: N: A manual page should start with a NAME section, which lists the N: program name and a brief description. The NAME section is used to N: generate a database that can be queried by commands like apropos and N: whatis. You are seeing this tag because lexgrog was unable to parse N: the NAME section. N: N: Manual pages for multiple programs, functions, or files should list N: each separated by a comma and a space, followed by \- and a common N: description. N: N: Listed items may not contain any spaces. A manual page for a two-level N: command such as fs listacl must look like fs_listacl so the list is N: read correctly. N: N: Refer to the lexgrog(1) manual page, the groff_man(7) manual page, and N: the groff_mdoc(7) manual page for details. N: N: Severity: warning N: N: Check: documentation/manual N: N: Renamed from: manpage-has-bad-whatis-entry N: for public-inbox-v1-format and public-inbox-v2-format. Adapt the descriptions to make lexgrog and so lintian happy.
2020-12-22support multiple CODE_URLs
public-inbox.org will expire in a few years, so ensure Tor .onions can be known before then.
2020-12-21extsearch*: drop unnecessary path canonicalization
Unlike inboxdir, the canonical-ness of -extindex paths is not relevant at the moment, and may never be relevant at all. So don't mislead others into thinking these paths being canonicalized matters.
2020-12-21searchidx: rename get_val to int_val and return IV
Values can be strings in Xapian, although we currently use integer values exclusively. Give the wrapper a more appropriate name in case we start using string columns. For future-proofing, we'll now return `undef' on missing columns and coerce the return value to an IV (integer value) to save memory, as sortable_unserialise returns a PV (pointer value) scalar despite it existing to support numeric values.
2020-12-21use rel2abs_collapsed when loading Inbox objects
We need to canonicalize paths for inboxes which do not have a newsgroup defined, otherwise ->eidx_key matches can fail in unexpected ways.
2020-12-21isearch: use numeric sort for article numbers
Perl sort is alphabetical by default and Xapian uses numeric document IDs, so sort must be told explicitly to use numeric comparisons even if the scalars are integer values (IV) internally. And eliminate extra hash marks ("#") since they're probably too noisy if there are many IDs. Note: I haven't seen this warning message in syslog, yet :>
2020-12-21inbox: delay ->version detection
Our read-only code won't need to know the version until an inbox is accessed. This is a small step towards eliminating many stat() calls on read-only daemon startup.
2020-12-21manifest.js.gz: fix per-inbox /$INBOX/manifest.js.gz
/$INBOX/manifest.js.gz should not attempt to match every inbox in the domain (or every inbox); that is for /manifest.js.gz (without a /$INBOX prefix). Fixes: f303b4add8ea1883 ("wwwlisting: avoid hogging event loop")
2020-12-20config: eliminate unnecessary join call up front
We can rely on implicit join in string interpolation on die() iff needed. And just creating the arrayref up front to avoid an extra backslash seems nicer at the moment.
2020-12-20daemon: kill_workers: eliminate unnecessary loop
The `kill' perl op takes multiple PIDs, so there's no need to iterate through the %pids hash.
2020-12-20daemon: unconditionally call IO::Handle::blocking(0)
IO::Handle::blocking will always return the initial value from the F_GETFL op and it won't issue F_SETFL if a socket is already non-blocking.
2020-12-20daemon: lazy load Cwd only for --daemonize users
systemd users won't need it polluting the namespace; though other things are still likely to load it.
2020-12-20inboxidle: remove needless check for {inboxdir}
->each_inbox will never attempt to iterate an object without {inboxdir}, and simplify + short-circuit the corresponding code
2020-12-20script/public-inbox-*: favor caller-provided pathnames
We'll try to avoid calling Cwd::abs_path and use File::Spec->rel2abs instead, since abs_path will resolve symlinks the user specified on the command-line. Unfortunately, ->rel2abs still leaves "/.." and "/../" uncollapsed, so we still need to fall back to Cwd::abs_path in those cases. While we are at it, we'll also resolve inboxdir from deep inside v2 directories instead of misdetecting them as v1 bare git repos. In any case, stop matching directories by name and instead rely on the unique combination of st_dev + st_ino on stat() as we started doing in the extindex code.
2020-12-20wwwstream: linkify coderepo URLs
It seems like a good idea to get more cgit visibility.
2020-12-18extsearchidx: improve missing machine-id fallback
It's likely most GNU/Linux systems have /etc/machine-id these days, so anything missing it is likely a *BSD, most of which support and favor "sysctl -n kern.hostid". We'll also support "ghostid" since GNU utils are commonly prefixed with 'g' on non-GNU platforms. In any case, we'll suppress stderr from missing commands and fall back to hard coding an $OSNAME-based identifier as a last resort and hope the hostname is unique.
2020-12-18nntpd: skip inboxes w/o {newsgroup}
So we don't trigger an uninitialized variable warning :x
2020-12-18Fix manpage section of perl module documentation
On Debian (at least) perl documentation is supposed to be installed in section 3pm. With the build system hardcoding this to 3 instead this results in a warning by the Debian package linter: W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Git.3.gz:74 3 != 3pm W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Import.3.gz:74 3 != 3pm W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::SaPlugin::ListMirror.3.gz:74 3 != 3pm W: public-inbox: wrong-manual-section ... use --no-tag-display-limit to see all (or pipe to a file/program) So use "$(MAN3EXT)" instead of "3".
2020-12-18import: drop X-Status in addition to Status
It's actually supported by mutt, dovecot[1], and likely some other software to augment the Status: header. While dovecot doesn't expose X-Status to clients, mutt will write 'A' (answered) and 'F' to X-Status (but not T (draft)). So we'll drop it like we do Status since it's not suitable for public mail, but stick it in an @UNWANTED_HEADERS array will allow us to configure an override if needed. [1] https://doc.dovecot.org/configuration_manual/mail_location/mbox/
2020-12-17extsearchidx: no need to make InboxWritable
extindex treats v1/v2 public inboxes as read-only, so there's no need to scare people by using the InboxWritable package now that ->git_dir_n is gone and we can use ->max_git_epoch instead of ->git_dir_latest.
2020-12-17inboxwritable: drop git_dir_n sub
There's only one caller, unlikely to be any more, and should be harmless to open code.
2020-12-17inbox: simplify v2 epoch counting
Perl readdir detects list context and can return an array suitable for the grep op. From there, we can rely on substr to remove the ".git" suffix and integerize the value to save a few bytes before letting List::Util::max return the value. This is how we detect Xapian shards nowadays, too, and we'll also use defined-or (//) to simplify the return value there. We'll also simplify InboxWritable->git_dir_latest, remove some callers, and consider removing it entirely.
2020-12-17index: ignore some warnings, set {current_info} for v1
-index runs on data that's already frozen in git, so there's no point in warning users about it. While we're at it, set the {current_info} prefix for v1 as we do in v2 inboxes in case new problems show up.