Date | Commit message (Collapse) |
|
We'll continue supporting `--no-sync' even if its yet-to-make it
it into a release, but the term `sync' is overloaded in our
codebase which may be confusing to new hackers and users.
None of our our code nor dependencies issue the sync(2) syscall,
either, only fsync(2) and fdatasync(2).
|
|
This is useful for speeding up indexing runs when only Xapian
rules change but SQLite indexing doesn't change. This mostly
implies `--reindex', but does NOT pick up new messages (because
SQLite indexing needs to occur for that).
I'm leaving this undocumented in the manpage for now since it's
mainly to speed up development and testing. Users upgrading to
1.6.0 will be advised to `--reindex --rethread', anyways, due to
the threading improvements since 1.1.0-pre1.
It may make sense to document for 1.7+ when there's Xapian-only
indexing changes, though.
|
|
This gives better page cache utilization for Xapian indexing on
slow storage by improving locality for random I/O activity on
the Xapian DB.
Instead of doing a single-pass to index both SQLite and Xapian;
this indexes them separately. The first pass is identical to
indexlevel=basic: it indexes both over.sqlite3 and msgmap.sqlite3.
Subsequent passes only operate on a single Xapian shard for
documents belonging to that shard. Given enough shards, each
individual shard can be made small enough to fit into the kernel
page cache and avoid HDD seeks for read activity.
Doing rough tests with a busy system with a 7200 RPM HDD with ext4,
full indexing of LKML (9 epochs) goes from ~80 hours (-j0) to
~30 hours (-j8) with 16GB RAM with 7 shards configured and fsync(2)
disabled (--no-sync) and `--batch-size=10m'.
|
|
We need to drop old ghosts properly while inside the
transaction, otherwise it becomes a no-op. This isn't a big
deal, as it only results in a few dangling DB rows and a
small amount of wasted space.
|
|
We replaced Xtmpdir with File::Temp->newdir in
commit 2a3e3a0469f54f6a4f80bf04614e5ddd794a6c5e
("xapcmd: replace Xtmpdirs with File::Temp->newdir")
but forgot to remove the outdated comment.
|
|
We already "use" it starting with commit
cd8dd7b08fddc7c2b5f218c3fcaa5dca5f9ad945
("search: support SWIG-generated Xapian.pm"),
so there's no need to require it redundantly.
|
|
I find myself mindlessly adding "-c" to public-inbox-index,
and other users may do the same. Instead of erroring out,
we'll just silently ignore it, for now and allow
public-inbox-compact to work on SQLite-only inboxes.
We'll only check for xapian-compact if search exists, since
it won't be needed in case we support SQLite VACUUM.
|
|
Thanks to the GCC compile farm project, we can wire up syscalls
for sparc64 and set system-specific SFD_* constants properly.
I've FINALLY figured out how to use POSIX::SigSet to generate
a usable buffer for the syscall perlfunc. This is required
for endian-neutral behavior and relevant to sparc64, at least.
There's no need for signalfd-related stuff to be constants,
either. signalfd initialization is never a hot path and a stub
subroutine for constants uses several KB of memory in the
interpreter.
We'll drop the needless SEEK_CUR import while we're importing
O_NONBLOCK, too.
|
|
This is specified in RFC 3501 but was accidentally omitted :x
I probably got it confused with TEXT, so add a comment about
TEXT being "everything" in the message.
|
|
While we always generate YYYYMMDDhhmmss query parameters
ourselves, the regexps in paginate_recent allow YYYYMMDD-only
(no hhmmss) timestamps, so don't trigger Time::Local::timegm
warnings about empty numeric comparisons on empty strings when a
client starts making up their own URLs.
|
|
Since reindexing releases the DB handle every indexBatchSize bytes,
we need to ensure we keep the journal in-memory when reopening
the DB since this is throwaway data.
|
|
epoll_wait_mod8 places a dummy element into the [2] slot of the
nested array, which caused is_deeply to fail.
Tested on aarch64.
|
|
Email::Address::XS and PublicInbox::MsgTime both emit warnings
which are likely to trigger from spam messages. Since this can
be configured to remove spam, just filter out those warnings to
avoid cluttering up stderr with useless information.
|
|
We don't want ENV changes propagated to other tests
when using t/run.perl via "make check-run"
|
|
No need to waste resources when doing minimal work. With
PI_TEST_VERSION=2, this fixes a test failure where
Net::NNTP::DESTROY was getting called in the shard process.
We'll also get rid of an unnecessary use_ok under v2, too.
|
|
We used ->header_obj in the past as an optimization with
Email::MIME. That optimization is no longer necessary
with PublicInbox::Eml.
This doesn't make any functional difference even if we were to
go back to Email::MIME. However, it reduces the amount of code
we have and slightly reduces allocations with PublicInbox::Eml.
|
|
We can rely on the newer mids() sub directly and use faster
numeric comparisons for Msgmap unindexing in v1.
|
|
This is more accurate given we use PublicInbox::Eml instead
of Email::MIME/PublicInbox::MIME, nowadays.
|
|
We don't want `local $/' affecting Eml->new, and we can
use implicit returns which may be faster on older Perl.
|
|
The return value of art_lookup changed but this command wasn't
updated since it wasn't tested.
Fixes: 0e6ceff37fc38f28 ("nntp: support slow blob retrievals")
|
|
v?fork failures seems to be the cause of locks not getting
released in -watch. Ensure lock release doesn't get skipped
in ->done for both v1 and v2 inboxes. We also need to do
everything we can to ensure DB handles, pipes and processes
get released even in the face of failure.
While we're at it, make failures around `git update-server-info'
non-fatal, since smart HTTP seems more popular anyways.
v2 changes:
- spawn: show failing command
- ensure waitpid is synchronous for inotify events
- teardown all fast-import processes on exception,
not just the failing one
- beef up lock_release error handling
- release lock on fast-import spawn failure
|
|
Although the ->async_next method does not take $self as
a receiver, but rather a PublicInbox::HTTP object, we may
still retrieve it to be called with the HTTP object via
UNIVERSAL->can.
|
|
This ought to be useful for diagnosing bugs in -watch.
|
|
The temporary clone starts as large as the full msgmap
and deletes will write to it randomly. So ensure it
doesn't get fragmented and slower as time goes on.
|
|
The grep call in list_match_domain_i returns true for all inboxes,
even ones without a URL that matches the regular expression, because
the qr value passed to grep is not surrounded by slashes. Add them.
Fixes: 1988d730c0088e8b (config: support multi-value inbox.*.*url)
|
|
It's possible for ~/.public-inbox/ to not exist if PI_CONFIG
points to an alternate location. Only noticed from the previous
patch fixing t/init.t behavior.
|
|
We'll just set the documented PI_EMERGENCY env to
a writable location.
|
|
The query parser test pulls in all of the IMAP stuff, so it
has the same dependencies.
|
|
This gives an opportunity for users already suffering from CoW
fragmentation to at least get the Xapian DBs off CoW. Aside
from over.sqlite3 in v1, the SQLite DBs remain untouched; though
VACUUM support may come in the future.
|
|
SQLite and Xapian files are written randomly, thus they become
fragmented under btrfs with copy-on-write. This leads to
noticeable performance problems (and probably ENOSPC) as these
files get big.
lore/git (v2, <1GB) indexes around 20% faster with this on an
ancient SSD. lore/lkml seems to be taking forever and I'll
probably cancel it to save wear on my SSD.
Unfortunately, disabling CoW also means disabling checksumming
(and compression), so we'll be careful to only set the No_COW
attribute on regeneratable data. We want to keep CoW (and
checksums+compression) on git storage because current ref
storage is neither checksummed nor compressed, and git streams
pack output.
|
|
Otherwise, a user is more likely to remove the msgmap-XXXXXXXX
SQLite file from $TMPDIR and cause SQLite to error out.
|
|
This seems to speed up --reindex on smallish v2 inboxes by about
30% on both HDD and SSD. lore/git (~1GB) on an SSD even gives a
30% improvement with 3 shards. I'm only seeing a ~4% speedup on
LKML with a SATA SSD (which is difficult to repeat because it
takes around 4 hours).
Testing LKML on an HDD will take much more time...
|
|
I finally noticed descriptions weren't showing up in my mirrors :x
|
|
mbsync was not retrieving anything since it was looking for
"inbox" when we need to return "INBOX" as a special case
for IMAP.
Fixes: 8af34015e9aa94e5 (imap: LIST shows "INBOX" in all caps)
|
|
We can keep the git process more active by sending another
request to it while fetch_run_ops() is running. This
parallelization speeds up mutt's initial FETCH for headers by
around ~35%(!).
|
|
Tests for failures should not leave junk temporary files lying
around in a users' ~/.public-inbox/.
On a side note, I'm not sure if PI_DIR is or was ever
necessary. It's never been documented, so perhaps
using $HOME for this is better...
|
|
And -compact supports --jobs=0 like -index to disable parallel
execution. Running three xapian-compact processes in parallel
on a USB 2.0 HDD is pretty painful.
|
|
We still need to use SQL_BLOB to ensure existing versions of
public-inbox can read over.sqlite3 because they're still using
{sqlite_unicode}. This partially reverts commit
e9fc1290ead44e06d20ff58e0a6acb5306d4fbe2.
Fixes: e9fc1290ead44e06 ("over: unset sqlite_unicode attribute")
|
|
There's no reason for {unindexed} to persist beyond
an ->index_sync call.
|
|
Another closure gone, and we may be able to share more
code with v2 in upcoming commits.
|
|
This allows v1 indexing to run while the `cat-file --batch-check'
process is waiting on high-latency storage.
|
|
Another step in making v1 and v2 more similar.
|
|
This allows us to speed up indexing operations to SQLite
and Xapian.
Unfortunately, it doesn't affect operations using
`xapian-compact' and the compactor API, since that doesn't seem
to support Xapian::DB_NO_SYNC, yet.
|
|
We'll switch to using IdxStack here to ensure we get repeatable
results and ascending THREADIDs according to git chronology.
This means we'll need a two-pass reindex to index existing
messages before indexing new messages.
Since we no longer have a long-lived git-log process, we don't
have to worry about old Xapian referencing the git-log pipe
w/o FD_CLOEXEC, either.
|
|
The "xdb" prefix was inaccurate since it's used by
indexlevel=basic, which is Xapian-free. The '_' (underscore)
prefix was also wrong for a method which is called across
package boundaries.
|
|
This was a bug, but I'm not sure where it matters, yet, but it
may matter in the future.
|
|
|
|
Since normal per-epoch indexing no longer holds a "git log"
process open, we don't need to worry about not sharing the
pipe with forked shards when we restart the indexer.
While we're in the area, better describe what `unindex' does,
since it's a rarely-used but necessary code path.
|
|
We can reduce the number of parameters we pass around on stack
and make our read-write and read-only code paths more uniform.
|
|
Instead, storing {xdir} will allow us to avoid string
concatenation in the read-only path and save us a little
hash entry space.
|