about summary refs log tree commit homepage
path: root/Documentation/public-inbox-xcpdb.pod
DateCommit message (Collapse)
2021-05-17doc: split option variants into separate items
e226f18934eb7291 modified the lei-q manpage so that each variant of an option gets a dedicated =item to make L</--xyz> look nicer and to follow the Perl core documentation. Do the same for the other manpages. Note that this still leaves the variants of an option grouped in one scenario: when a list of options without descriptions is presented as a pointer to another location. Splitting the variants in that case would make it harder for the reader to tell what the distinct options are.
2021-05-04treewide: update to v3 Tor onions
v2 onions are insecure, deprecated and going away. v3 names are unfortunately longer and more difficult to remember, but should be more resistant to attack than v2 ones.
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2020-09-20doc: post-1.6 updates, start 1.7
I should've dropped "PENDING" notes before the 1.6 release; they're dropped now, and a note is added to remind my future self to drop them before 1.7.
2020-08-26doc: add some more tuning notes
I've learned a thing or three about btrfs in the past few weeks and remembered some old HDD things, too. The Xapian MultiDatabase problem will need to be addressed for 1.7...
2020-08-20doc: note -compact and -xcpdb are rarely used
Slowly improving the learning curve...
2020-08-14index|compact|xcpdb: support --all switch
For -index, this is a convenient way to quickly index all inboxes after a grok-pull. Might as well support it for rarely used commands like -compact and -xcpdb, too.
2020-08-13xcpdb: wire up new index options and --help
--sequential-shard also disables the copy parallelism (--jobs), so it can be useful for systems unable to handle parallel random I/O but still want many shards. There was a missing "use strict", too, which is fixed.
2020-08-10doc: add some notes around -xcpdb / -edit / -purge
These rarely-used commands have some caveats that needed expanding on.
2020-08-07index+xcpdb: rename `--no-sync' to `--no-fsync'
We'll continue supporting `--no-sync' even if its yet-to-make it it into a release, but the term `sync' is overloaded in our codebase which may be confusing to new hackers and users. None of our our code nor dependencies issue the sync(2) syscall, either, only fsync(2) and fdatasync(2).
2020-07-25index+xcpdb: support --no-sync flag
This allows us to speed up indexing operations to SQLite and Xapian. Unfortunately, it doesn't affect operations using `xapian-compact' and the compactor API, since that doesn't seem to support Xapian::DB_NO_SYNC, yet.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2020-02-06doc: txt2pre: auto-linkify manpage references
This can be more convenient for people browsing HTML docs remotely or locally.
2019-06-14doc: rename our Xapian "partitions" to "shards"
For consistency with Xapian documentation (in the "master" branch).
2019-06-14xcpdb: support resharding v2 repos
v2 repos are sometimes created on machines where CPU parallelization exceeds the capability of the storage devices. In that case, users may reshard the Xapian DB to any smaller, positive integer to avoid excessive overhead and contention when bottlenecked by slow storage. Resharding can also be used to increase shard count after hardware upgrades.
2019-05-24doc: add URLs for Xapian manpages
Since we go through the effort of hosting these manpages, link to them.
2019-05-24doc: xcpdb: add switch documentation
In particular, the '--compact' switch is really useful since it works without holding the inbox-wide lock for minutes at a time on giant inboxes (inboxes where copies can take dozens, if not hundreds of minutes).
2019-05-23doc: various updates to reflect current state
-index documentation avoid redundant v1 information and refers readers to apropriate v1/v2 manpages. Search::Xapian can also be optional, now, as only the PSGI search interface uses it. Favor "INBOX_DIR" where appropriate, since "REPO_DIR" can be confused for code repos which we also support. XAPIAN_FLUSH_THRESHOLD is documented for all relevant bulk commands.
2019-05-23doc: xcpdb: update to reflect the current state
It is no longer a wrapper around copydatabase(1), since copydatabase did not recover from DatabaseModifiedError.
2019-05-23xcpdb: new tool which wraps Xapian's copydatabase(1)
copydatabase(1) is an existing Xapian tool which is the recommended way to upgrade existing DBs to the latest Xapian database format (currently "glass" for stable/released versions). Our use of Xapian relies on preserving document IDs, so we'll wrap it like we do xapian-compact(1) and use the "--no-renumber" switch. I could not name the tool "public-inbox-copydatabase" since it would be ambiguous as to which DB it's actually copying. So, I abbreviated the suffix to "xcpdb" (Xapian CoPy DataBase), which I hope is acceptable and unambiguous.