diff options
Diffstat (limited to 'Documentation/public-inbox-tuning.pod')
| -rw-r--r-- | Documentation/public-inbox-tuning.pod | 222 |
1 files changed, 222 insertions, 0 deletions
diff --git a/Documentation/public-inbox-tuning.pod b/Documentation/public-inbox-tuning.pod new file mode 100644 index 00000000..892ee0f2 --- /dev/null +++ b/Documentation/public-inbox-tuning.pod @@ -0,0 +1,222 @@ +=head1 NAME + +public-inbox-tuning - tuning public-inbox + +=head1 DESCRIPTION + +public-inbox intends to support a wide variety of hardware. While +we strive to provide the best out-of-the-box performance possible, +tuning knobs are an unfortunate necessity in some cases. + +=over 4 + +=item 1 + +New inboxes: public-inbox-init -V2 + +=item 2 + +Optional Inline::C use + +=item 3 + +Performance on rotational hard disk drives + +=item 4 + +Btrfs (and possibly other copy-on-write filesystems) + +=item 5 + +Performance on solid state drives + +=item 6 + +Read-only daemons + +=item 7 + +Other OS tuning knobs + +=item 8 + +Scalability to many inboxes + +=item 9 + +public-inbox-cindex --join performance + +=item 10 + +public-inbox-clone with shared object stores + +=back + +=head2 New inboxes: public-inbox-init -V2 + +If you're starting a new inbox (and not mirroring an existing one), +the L<-V2|public-inbox-v2-format(5)> requires L<DBD::SQLite>, but is +orders of magnitude more scalable than the original C<-V1> format. + +=head2 Optional Inline::C use + +Our optional use of L<Inline::C> speeds up subprocess spawning from +large daemon processes. + +To enable L<Inline::C>, either set the C<PERL_INLINE_DIRECTORY> +environment variable to point to a writable directory, or create +C<~/.cache/public-inbox/inline-c> for any user(s) running +public-inbox processes. + +If libgit2 development files are installed and L<Inline::C> +is enabled (described above), per-inbox C<git cat-file --batch> +processes are replaced with a single L<perl(1)> process running +C<PublicInbox::Gcf2::loop> in read-only daemons. libgit2 use +will be available in public-inbox 1.7.0+ + +More (optional) L<Inline::C> use will be introduced in the future +to lower memory use and improve scalability. + +Note: L<Inline::C> is required for L<lei(1)>, but not public-inbox-* + +=head2 Performance on rotational hard disk drives + +Random I/O performance is poor on rotational HDDs. Xapian indexing +performance degrades significantly as DBs grow larger than available +RAM. Attempts to parallelize random I/O on HDDs leads to pathological +slowdowns as inboxes grow. + +While C<-V2> introduced Xapian shards as a parallelization +mechanism for SSDs, enabling C<publicInbox.indexSequentialShard> +repurposes sharding as a mechanism to reduce the kernel page cache +footprint when indexing on HDDs. + +Initializing a mirror with a high C<--jobs> count to create more +shards (in C<-V2> inboxes) will keep each shard smaller and +reduce its kernel page cache footprint. Keep in mind excessive +sharding imposes a performance penalty for read-only queries. + +Users with large amounts of RAM are advised to set a large value +for C<publicinbox.indexBatchSize> as documented in +L<public-inbox-index(1)>. + +C<dm-crypt> users on Linux 4.0+ are advised to try the +C<--perf-same_cpu_crypt> C<--perf-submit_from_crypt_cpus> +switches of L<cryptsetup(8)> to reduce I/O contention from +kernel workqueue threads. + +=head2 Btrfs (and possibly other copy-on-write filesystems) + +L<btrfs(5)> performance degrades from fragmentation when using +large databases and random writes. The Xapian + SQLite indices +used by public-inbox are no exception to that. + +public-inbox 1.6.0+ disables copy-on-write (CoW) on Xapian and SQLite +indices on btrfs to achieve acceptable performance (even on SSD). +Disabling copy-on-write also disables checksumming, thus C<raid1> +(or higher) configurations may be corrupt after unsafe shutdowns. + +Fortunately, these SQLite and Xapian indices are designed to be +recoverable from git if missing. + +Disabling CoW does not prevent all fragmentation. Large values +of C<publicInbox.indexBatchSize> also limit fragmentation during +the initial index. + +Avoid snapshotting subvolumes containing Xapian and/or SQLite indices. +Snapshots use CoW despite our efforts to disable it, resulting +in fragmentation. + +L<filefrag(8)> can be used to monitor fragmentation, and +C<btrfs filesystem defragment -fr $INBOX_DIR> may be necessary. + +Large filesystems benefit significantly from the C<space_cache=v2> +mount option documented in L<btrfs(5)>. + +Older, non-CoW filesystems generally work well out of the box +for our Xapian and SQLite indices. + +=head2 Performance on solid state drives + +While SSD read performance is generally good, SSD write performance +degrades as the drive ages and/or gets full. Issuing C<TRIM> commands +via L<fstrim(8)> or similar is required to sustain write performance. + +Users of the Flash-Friendly File System +L<F2FS|https://en.wikipedia.org/wiki/F2FS> may benefit from +optimizations found in SQLite 3.21.0+. Benchmarks are greatly +appreciated. + +=head2 Read-only daemons + +L<public-inbox-httpd(1)>, L<public-inbox-imapd(1)>, and +L<public-inbox-nntpd(1)> are all designed for C10K (or higher) +levels of concurrency from a single process. SMP systems may +use C<--worker-processes=NUM> as documented in L<public-inbox-daemon(8)> +for parallelism. + +The open file descriptor limit (C<RLIMIT_NOFILE>, C<ulimit -n> in L<sh(1)>, +C<LimitNOFILE=> in L<systemd.exec(5)>) may need to be raised to +accommodate many concurrent clients. + +Transport Layer Security (IMAPS, NNTPS, or via STARTTLS) significantly +increases memory use of client sockets, be sure to account for that in +capacity planning. + +Bursts of small object allocations late in process life contribute to +fragmentation of the heap due to arenas (slabs) used internally by Perl. +glibc malloc users should use C<MALLOC_MMAP_THRESHOLD_=131072> to reduce +fragmentation from the sliding mmap window. jemalloc (tested as an +LD_PRELOAD on GNU/Linux) also reduces fragmentation compared to an +unconfigured glibc malloc in long-lived processes. + +=head2 Other OS tuning knobs + +Linux users: the C<sys.vm.max_map_count> sysctl may need to be increased if +handling thousands of inboxes (with L<public-inbox-extindex(1)>) to avoid +out-of-memory errors from git. + +Other OSes may have similar tuning knobs (patches appreciated). + +=head2 Scalability to many inboxes + +L<public-inbox-extindex(1)> allows any number of public-inboxes +to share the same Xapian indices. + +git 2.33+ startup time is orders of magnitude faster and uses +less memory when dealing with thousands of alternates required +for thousands of inboxes with L<public-inbox-extindex(1)>. + +Frequent packing (via L<git-gc(1)>) both improves performance +and reduces the need to increase C<sys.vm.max_map_count>. + +=head2 public-inbox-cindex --join performance + +A C++ compiler and the Xapian development files makes C<--join> or +C<--join=aggressive> orders of magnitude faster in L<public-inbox-cindex(1)>. +On Debian-based systems this is C<libxapian-dev>. RPM-based distros have +these in C<xapian-core-devel> or C<xapian14-core-libs>. *BSDs typically +package development files together with runtime libraries, so the C<xapian> +or C<xapian-core> package will already have the development files. + +=head2 public-inbox-clone with shared object stores + +When mirroring manifests with many forks using the same objstore, +git 2.41+ is highly recommended for performance as we automatically +use the C<fetch.hideRefs> feature to speed up negotiation. + +=head1 CONTACT + +Feedback encouraged via plain-text mail to L<mailto:meta@public-inbox.org> + +Information for *BSDs and non-traditional filesystems especially +welcome. + +Our archives are hosted at L<https://public-inbox.org/meta/>, +L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>, and other places + +=head1 COPYRIGHT + +Copyright all contributors L<mailto:meta@public-inbox.org> + +License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> |