PUBLIC-INBOX-EXTINDEX(1)   public-inbox user manual   PUBLIC-INBOX-EXTINDEX(1)

NAME
       public-inbox-extindex - create and update external search indices

SYNOPSIS
       public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...

       public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all

DESCRIPTION
       public-inbox-extindex creates and updates an external search and
       overview database used by the read-only public-inbox PSGI (HTTP), NNTP,
       and IMAP interfaces.  This requires either the Xapian SWIG bindings OR
       or Search::Xapian XS bindings along with DBD::SQLite and DBI Perl
       modules.

OPTIONS
       -j JOBS
       --jobs=JOBS
       --no-fsync
       --defrag=MSGCOUNT
       --dangerous
       --rethread
       --max-size SIZE
       --batch-size SIZE
       --split-shards
       --split-at DOCCOUNT
       --multipass
       --wal
       --commit SECONDS
           These switches behave as they do for public-inbox-index(1)

       --all
           Index all "publicinbox" entries in "PI_CONFIG".

           "publicinbox" entries indexed by "public-inbox-extindex" can have
           full Xapian searching abilities with the per-"publicinbox"
           "indexlevel" set to "basic" and their respective Xapian ("xap15" or
           "xapian15") directories removed.  For multiple public-inboxes where
           cross-posting is common, this allows significant space savings on
           Xapian indices.

       --dedupe=MSGID
       --dedupe
           Rerun deduplication on messages with the given Message-ID or all
           messages if no Message-ID is specified.  Deduplication rules may
           change and evolve over time, especially if filters are involved.

           "--dedupe=MSGID" may be specified multiple times to deduplicate
           multiple Message-IDs.

           Use this if you see "W: BUG? $MSGID not deduplicated properly"
           warnings from WWW logs.

       --gc
           Perform garbage collection instead of indexing.  Use this if
           inboxes are removed from the extindex, a newsgroup name is set or
           changed, or if messages are purged or removed from some inboxes.

       --reindex
           Forces a re-index of all messages in the extindex.  This can be
           used for in-place upgrades and bugfixes while read-only server
           processes are utilizing the index.  Keep in mind this roughly
           doubles the size of the already-large Xapian database.

       --fast
           Used with "--reindex", it will only look for new and stale entries
           and not touch already-indexed messages.

       --no-multi-pack-index
           Disable writing a git-multi-pack-index(1) file to save memory.
           Normally, enabling multi-pack-index speeds up startup time of
           subsequent git-cat-file(1) processes by 3-4%, but generating this
           file requires several GB of memory with large repos.

           Unlike the "core.multiPackIndex" directive in git, it's still
           possible to read existing multi-pack-index files if they are
           created elsewhere.

           Available in public-inbox 2.0.0+

FILES
       public-inbox-extindex-format(5)

CONFIGURATION
       public-inbox-extindex does not write to the public-inbox-config(5)
       file, it must be entered manually.  The extindex name of "all" is a
       special case which corresponds to indexing "--all" inboxes.  An example
       for "--all" is as follows:

               [extindex "all"]
                       topdir = /path/to/extindex_dir
                       url = all
                       coderepo = foo
                       coderepo = bar

       Putting an "extindex" entry in the config allows PublicInbox::WWW.  You
       can have any number of "extindex.$NAME" sections where $NAME is
       something other than "all" to display a union of several inboxes.

       It is strongly recommended any public inboxes indexed by this command
       have a stable "publicinbox.$NAME.newsgroup" entry (regardless of the
       presence of an NNTP or IMAP server).  Otherwise, public-inbox-extindex
       will use "publicinbox.$NAME.inboxdir" as an internal key which can
       cause needless reindexing and require --gc if inboxes are relocated.

       Per-inbox "publicinbox.$NAME.altid" config is only supported as of
       public-inbox 2.0+.

       See public-inbox-config(5) for more details.

ENVIRONMENT
       PI_CONFIG
               Used to override the default "~/.public-inbox/config" value.

       XAPIAN_FLUSH_THRESHOLD
               The number of documents to update before committing changes to
               disk.  This environment is handled directly by Xapian, refer to
               Xapian API documentation for more details.

               Setting "XAPIAN_FLUSH_THRESHOLD" or
               "publicinbox.indexBatchSize" for a large "--reindex" may cause
               public-inbox-mda(1), public-inbox-learn(1) and
               public-inbox-watch(1) tasks to wait long and unpredictable
               periods of time during "--reindex".

               Default: none, uses "publicinbox.indexBatchSize"

UPGRADING
       Occasionally, public-inbox will update its schema version and require a
       full index by running this command.

LOCKING
       It is safe to use "--dedupe", "--gc" and "--reindex" while other
       processes are writing to covered inboxes or extindex.  The extindex
       locks will be released roughly every 10s to allow public-inbox-mda(1)
       and public-inbox-watch(1) processes to write to the extindex.

CONTACT
       Feedback welcome via plain-text mail to <mailto:meta@public-inbox.org>

       The mail archives are hosted at <https://public-inbox.org/meta/> and
       <http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>

COPYRIGHT
       Copyright all contributors <mailto:meta@public-inbox.org>

       License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>

SEE ALSO
       Search::Xapian, DBD::SQLite

public-inbox.git                  1993-10-02          PUBLIC-INBOX-EXTINDEX(1)