=head1 NAME

public-inbox-extindex - create and update external search indices

=head1 SYNOPSIS

public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...

public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all

=head1 DESCRIPTION

public-inbox-extindex creates and updates an external search and
overview database used by the read-only public-inbox PSGI (HTTP),
NNTP, and IMAP interfaces.  This requires either the
L<Search::Xapian> XS bindings OR the L<Xapian> SWIG bindings,
along with L<DBD::SQLite> and L<DBI> Perl modules.

=head1 OPTIONS

=over

=item -j JOBS

=item --jobs=JOBS

... TODO, see L<public-inbox-index(5)>

=item --all

Index all C<publicinbox> entries in C<PI_CONFIG>.

C<publicinbox> entries indexed by C<public-inbox-extindex> can
have full Xapian searching abilities with the per-C<publicinbox>
C<indexlevel> set to C<basic> and their respective Xapian
(C<xap15> or C<xapian15>) directories removed.  For multiple
public-inboxes where cross-posting is common, this allows
significant space savings on Xapian indices.

=item --gc

Perform garbage collection instead of indexing.  Use this if
inboxes are removed from the extindex, or if messages are
purged or removed from some inboxes.

=item --reindex

Forces a re-index of all messages in the extindex.  This can be
used for in-place upgrades and bugfixes while read-only server
processes are utilizing the index.  Keep in mind this roughly
doubles the size of the already-large Xapian database.

The extindex locks will be released roughly every 10s to
allow L<public-inbox-mda(1)> and L<public-inbox-watch(1)>
processes to write to the extindex.

=item --fast

Used with C<--reindex>, it will only look for new and stale
entries and not touch already-indexed messages.

=back

=head1 FILES

L<public-inbox-extindex-format(5)>

=head1 CONFIGURATION

public-inbox-extindex does not currently write to the
L<public-inbox-config(5)> file, configuration may be entered
manually.  The extindex name of C<all> is a special case which
corresponds to indexing C<--all> inboxes.  An example for
C<--all> is as follows:

	[extindex "all"]
		topdir = /path/to/extindex_dir
		url = all
		coderepo = foo
		coderepo = bar

See L<public-inbox-config(5)> for more details.

=head1 ENVIRONMENT

=over 8

=item PI_CONFIG

Used to override the default "~/.public-inbox/config" value.

=item XAPIAN_FLUSH_THRESHOLD

The number of documents to update before committing changes to
disk.  This environment is handled directly by Xapian, refer to
Xapian API documentation for more details.

Setting C<XAPIAN_FLUSH_THRESHOLD> or
C<publicinbox.indexBatchSize> for a large C<--reindex> may cause
L<public-inbox-mda(1)>, L<public-inbox-learn(1)> and
L<public-inbox-watch(1)> tasks to wait long and unpredictable
periods of time during C<--reindex>.

Default: none, uses C<publicinbox.indexBatchSize>

=back

=head1 UPGRADING

Occasionally, public-inbox will update it's schema version and
require a full index by running this command.

=head1 CONTACT

Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>

The mail archives are hosted at L<https://public-inbox.org/meta/> and
L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>

=head1 COPYRIGHT

Copyright 2021 all contributors L<mailto:meta@public-inbox.org>

License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>

=head1 SEE ALSO

L<Search::Xapian>, L<DBD::SQLite>