about summary refs log tree commit homepage
path: root/lib/PublicInbox/V2Writable.pm
DateCommit message (Collapse)
2020-02-02v2writable: more ways to detect online CPU count
OpenBSD and FreeBSD support `getconf NPROCESSORS_ONLN` (no leading underscore). They may also have GNU nproc installed as "gnproc". We may also encounter Linux systems w/o GNU coreutils, but able to use `getconf _NPROCESSORS_ONLN` (with leading underscore).
2020-02-02v2writable: do not clobber {shards} or {parallel} if unset
The $jobs parameter in `public-inbox-convert' is passed to V2Writable->init_inbox as `undef' by default, causing parallelization to be disabled. Instead, leave the underlying {parallel} flag untouched if $shards is undef and do not clobber the default shard count. This allows us to take advantage of multicore systems when running public-inbox-convert with no command-line switches.
2020-02-02v2writable: nproc_shards: subtract 1 from given value
This is to be consistent with the `nproc(1)' code path. It also quiets down a warning from Admin when "-j $JOBS" is specified, since the master process (which distributes work to shards and handles OverIdx and Msgmap) is considered a job on its own.
2020-01-28v2writable: newest epochs go first in alternates
New epochs are the most likely to have loose objects. git won't be able to take advantage of pack indices and needs to scan every alternate for the loose object via open/openat syscalls. Those syscalls will add up some day when we've got hundreds or thousands of epochs.
2020-01-13use popen_rd for bidirectional pipes
popen_rd accepts arbitrary redirects, so we can reuse its code to setup the pipe end we want to read, saving each caller a few lines of code compared to calling pipe+spawn.
2020-01-11spawn (and thus popen_rd) die on failure
Most spawn and popen_rd callers die on failure to spawn, anyways, and some are missing checks entirely. This saves us a bunch of verbose error-checking code in callers. This also makes popen_rd more consistent, since it already dies on pipe creation failures.
2020-01-06treewide: "require" + "use" cleanup and docs
There's a bunch of leftover "require" and "use" statements we no longer need and can get rid of, along with some excessive imports via "use". IO::Handle usage isn't always obvious, so add comments describing why a package loads it. Along the same lines, document the tmpdir support as the reason we depend on File::Temp 0.19, even though every Perl 5.10.1+ user has it. While we're at it, favor "use" over "require", since it it gives us extra compile-time checking.
2019-12-30spawn: allow passing GLOB handles for redirects
We can save callers the trouble of {-hold} and {-dev_null} refs as well as the trouble of calling fileno().
2019-12-24search: support SWIG-generated Xapian.pm
Xapian upstream is slowly phasing out the XS-based Search::Xapian in favor of the SWIG-generated "Xapian" package. While Debian and both FreeBSD have Search::Xapian, OpenBSD only includes the "Xapian" binding. More information about the status of the "Xapian" Perl module here: https://trac.xapian.org/ticket/523
2019-11-29replace: quiet "git gc" invocation
Since we give users no indication or control of how "git gc" runs, showing its progress is confusing.
2019-11-04index: "git log" failures are fatal
While I've never seen "git log" fail on its own, it could happen one day and we should be prepared to abort indexing when it happens. Beef up tests for t/spawn.t to ensure close() behaves on popen_rd the way we expect it to.
2019-10-30inboxwritable: add assert_usable_dir sub
And use it for mda, since "0" could be a usable directory if somebody insists on using relative paths...
2019-10-22v2writable: use msgmap as multi_mid queue
Instead of storing Message-IDs in the Msgmap object, we can store the blob OID. For initial indexing of mirrors, this lets us preserve $sync->{regen} by storing the intended article number in the queue. On --reindex, the article number we store in Msgmap is ignored but only used for ordering purposes. This also allows us to avoid ENOMEM errors if somebody abuses our system by reusing Message-IDs; but we now risk ENOSPC instead (but systems tend to have more FS storage than RAM).
2019-10-22v2writable: move git->cleanup to the correct place
We need to stop the git process to avoid leaking FDs to Xapian if we recurse ->index_sync on reindex.
2019-10-21v2writable: reindex handles 3-headered monsters
And maybe 8-headered ones, too... I noticed --reindex failing on the linux-renesas-soc mirror due one 3-headed monster of a message having 3 sets of headers; while another normal message had a Message-ID that matched one of the 3 IDs of the 3-headed monster. We still try to do the majority of indexing backwards, but we defer indexing multi-Message-ID'd messages until the end to ensure we get all the "good" messages in before we process the multi-headered ones. Link: https://public-inbox.org/meta/20191016211415.GA6084@dcvr/
2019-10-21v2writable: improve "num_for" API and disambiguate
Make it obvious that we're not the Msgmap sub and return an array because it's less awkward than providing a modifiable ref to a function to write to.
2019-10-21v2writable: set unindexed article number
We'll actually use the keys of this hash in future commits.
2019-10-16config: support "inboxdir" in addition to "mainrepo"
"mainrepo" ws a bad name and artifact from the early days when I intended for there to be a "spamrepo" (now just the ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be especially confusing, since v2 needs at least two git repositories (epoch + all.git) to function and we shouldn't confuse users by having them point to a git repository for v2. Much of our documentation already references "INBOX_DIR" for command-line arguments, so use "inboxdir" as the git-config(1)-friendly variant for that. "mainrepo" remains supported indefinitely for compatibility. Users may need to revert to old versions, or may be referring to old documentation and must not be forced to change config files to account for this change. So if you're using "mainrepo" today, I do NOT recommend changing it right away because other bugs can lurk. Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
2019-09-27v2writable: only change $EPOCH.git/config on epoch creation
We don't need to make unnecesary writes to the git config file and wear out storage devices every time we run "public-inbox-index"
2019-09-09run update-copyrights from gnulib for 2019
2019-06-15comments: replace "partition" with "shard"
Now that the code matches Xapian terminology, ensure our comments match, too.
2019-06-14v2writable: use "epoch" consistently when referring to git repos
Be consistent with our own terminology and use "epoch" for [0-9]+\.git repos. The term "partition" is going away entirely.
2019-06-14v2writable: rename local vars to match Xapian terminology
2019-06-14v2writable: avoid "part" in internal subs and fields
We'll be using the term "shard" from now on to be consistent with Xapian terminology.
2019-06-14v2: rename SearchIdxPart => SearchIdxShard
Another step towards keeping our file and package names consistent with Xapian terminology.
2019-06-14v2writable: rename {partitions} field to {shards}
Our internal data structure should be consistent with Xapian terminology.
2019-06-14v2writable: count_partitions => count_shards
Another step towards becoming consistent with Xapian terminology
2019-06-14v2writable: update comments regarding xcpdb --reshard
Using compact to change shard count was abandoned during the v2 development phase.
2019-06-14v2writable: fix brainfart when capping NPROC_MAX_DEFAULT
Oops :x
2019-06-14Merge remote-tracking branch 'origin/reshard' into next
* origin/reshard: xcpdb: support resharding v2 repos xcpdb: use destination shard as progress prefix xapcmd: preserve indexlevel based on the destination v2writable: use a smaller default for Xapian partitions
2019-06-14v2writable: use a smaller default for Xapian partitions
Apparently 16 CPUs (probably HT) and SATA storage is common these days. Having excessive Xapian partitions leads to contention and excessive FD/space use. So set a smaller default but continue allowing user-specified values to bump this up.
2019-06-10v2writable: replace: kill git processes before reindexing
Xapian on Linux <3.15 has trouble with coprocesses since it used fork() for locking and would hold onto pipes used for git unnecessarily.
2019-06-09v2writable: implement ->replace call
Much of the existing purge code is repurposed to a general "replace" functionality. ->purge is simpler because it can just drop the information. Unlike ->purge, ->replace needs to edit existing git commits (in case of From: and Subject: headers) and reindex the modified message. We currently disallow editing of References:, In-Reply-To: and Message-ID headers because it can cause bad side effects with our threading (and our lack of rethreading support to deal with excessive matching from incorrect/invalid References).
2019-06-09import: switch to "replace_oids" interface for purge
Continuing the work by Eric Biederman in commit a118d58a402bd31b ("Import.pm: When purging replace a purged file with a zero length file"), we can use a generic OID replacement mechanism to implement purge.
2019-06-09v2writable: consolidate overview and indexing call
It's one ugly sub with lots of parameters, but it's better than calling a bunch of ugly subs with lots of parameters; as we'll be needing to call it again when reindexing for message replacements.
2019-06-04require ASCII digits for local FS items
In case some BOFH decides to randomly create directories using non-ASCII digits all over the place.
2019-05-30v2writable: short-circuit is_ancestor check on equality
We don't need to use git to check ancestry if object IDs match on a string comparison. This saves 100ms or so and brings down the ~0.5s no-op time on lore.kernel.org/lkml down to ~0.4s.
2019-05-30v2writable: avoid mm_tmp creation without regen
Creating mm_tmp is an expensive operation with large inboxes and can be avoided if there are no new messages to process. Since git-fetch(1) currently lacks an --exit-code option(*), mirrors will run `public-inbox-index' unconditionally after fetch, which is an expensive op if it needs to duplicate a large SQLite DB. This speeds up the mirror case of: git --git-dir=git/$EPOCH.git fetch && public-inbox-index This reduces the no-op `public-inbox-index' time from over 8s to ~0.5s on a (currently) 7-epoch clone of https://lore.kernel.org/lkml/ on my system. (*) WIP --exit-code for git-fetch: https://public-inbox.org/git/87ftphw7mv.fsf@evledraar.gmail.com/
2019-05-30v2writable: hoist out index_epoch sub
This will make future changes easier-to-follow.
2019-05-30v2writable: split off unindex_range mapping
It'll make it easier to detect if we have anything to unindex and run git-log on, at all.
2019-05-29v2writable: show progress updates for index_sync
We can show progress whenever we commit changes to the FS.
2019-05-29v2writable: move index_sync options to sync state
And use singular `opt' to be consistent with the common name of 'getopt'.
2019-05-29v2writable: use prototypes for internal subs
Hopefully this improves maintainability by allowing Perl to do some arg checking for us.
2019-05-29v2writable: localize unindex-range.$EPOCH to $sync state
We don't need to stuff that into $self (V2Writable) which can be longer-lived than a ->index_sync invocation.
2019-05-29v2writable: move {ranges} into $sync state
Yet another temporary variable with no use outside of index_sync.
2019-05-29v2writable: move {regen} into $sync state
regen is always enabled for index_sync nowadays (and has been for a while). Rename `index_prepare' to `sync_prepare' to show it's for ->index_sync; and not the online indexing we do for ->add.
2019-05-29v2writable: move {reindex} field to $sync state
reindexing info is not used outside of the index_sync code path.
2019-05-29v2writable: sync: move delete markers into $sync state
Another small step to reduce parameters passed to reindex_oid.
2019-05-29v2writable: introduce $sync state and put mm_tmp in it
A first step towards making the v2 index_sync code easier-to-follow. More fields to follow...
2019-05-27v2: fix reindex skipping NNTP article numbers
`public-inbox-index --reindex' could cause NNTP article number gaps to form when it also has to deal with new, never-before-seen commits in mirrors running off `git fetch'. Fix this by running two distinct invocations of ->index_sync; once to only reindex old commits, and a second time to index new commits. This does not appear to be a problem on v1 at the moment, but I'll need more time to analyze this.