NAME
    public-inbox extindex format description

DESCRIPTION
    The extindex is an index-only evolution of the per-inbox SQLite and
    Xapian indices used by public-inbox-v2-format(5) and
    public-inbox-v1-format(5). It exists to facilitate searches across
    multiple inboxes as well as to reduce index space when messages are
    cross-posted to several existing inboxes.

    It transparently indexes messages across any combination of v1 and v2
    inboxes and data about inboxes themselves.

DIRECTORY LAYOUT
    While inspired by v2, there is no git blob storage nor "msgmap.sqlite3"
    DB.

    Instead, there is an "ALL.git" (all caps) git repo which treats every
    indexed v1 inbox or v2 epoch as a git alternate.

    As with v2 inboxes, it uses "over.sqlite3" and Xapian "shards" for WWW
    and IMAP use. Several exclusive new tables are added to deal with "XREF3
    DEDUPLICATION" and metadata.

    Unlike v1 and v2 inboxes, it is NOT designed to map to a NNTP newsgroup.
    Thus it lacks "msgmap.sqlite3" to enforce the unique Message-ID
    requirement of NNTP.

  INDEX OVERVIEW AND DEFINITIONS
      $SCHEMA_VERSION - DB schema version (for Xapian)
      $SHARD - Integer starting with 0 based on parallelism

      foo/                              # "foo" is the name of the index
      - ei.lock                         # lock file to protect global state
      - ALL.git                         # empty, alternates for inboxes
      - ei$SCHEMA_VERSION/$SHARD        # per-shard Xapian DB
      - ei$SCHEMA_VERSION/over.sqlite3  # overview DB for WWW, IMAP
      - ei$SCHEMA_VERSION/misc          # misc Xapian DB

    File and directory names are intentionally different from analogous v2
    names to ensure extindex and v2 inboxes can easily be distinguished from
    each other.

  XREF3 DEDUPLICATION
    Due to cross-posted messages being the norm in the large Linux kernel
    development community and Xapian indices being the primary consumer of
    storage, it makes sense to deduplicate indexing as much as possible.

    The internal storage format is based on the NNTP "Xref" tuple, but with
    the addition of a third element: the git blob OID. Thus the triple is
    expressed in string form as:

            $NEWSGROUP_NAME:$ARTICLE_NUM:$OID

    If no "newsgroup" is configured for an inbox, the "inboxdir" of the
    inbox is used.

    This data is stored in the "xref3" table of over.sqlite3.

  misc XAPIAN DB
    In addition to the numeric Xapian shards for indexing messages, there is
    a new, in-development Xapian index for storing data about inboxes
    themselves and other non-message data. This index allows us to speed up
    operations involving hundreds or thousands of inboxes.

BENEFITS
    In addition to providing cross-inbox search capabilities, it can also
    replace per-inbox Xapian shards (but not per-inbox over.sqlite3). This
    allows reduction in disk space, open file handles, and associated memory
    use.

CAVEATS
    Relocating v1 and v2 inboxes on the filesystem will require extindex to
    be garbage-collected and/or reindexed.

    Configuring and maintaining stable "newsgroup" names before any messages
    are indexed from every inbox can avoid expensive reindexing and rely
    exclusively on GC.

LOCKING
    flock(2) locking exclusively locks the empty ei.lock file for all
    non-atomic operations.

THANKS
    Thanks to the Linux Foundation for sponsoring the development and
    testing.

COPYRIGHT
    Copyright 2020-2021 all contributors <mailto:meta@public-inbox.org>

    License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>

SEE ALSO
    public-inbox-v2-format(5)