user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
blob 554346de1b7259f562368a5258a4bc225449f005 2943 bytes (raw)
name: Documentation/public-inbox-index.pod 	 # note: path name is non-authoritative(*)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
 
=head1 NAME

public-inbox-index - create and update search indices

=head1 SYNOPSIS

public-inbox-index GIT_DIR

=head1 DESCRIPTION

public-inbox-index creates and updates the search and NNTP
article number database used by the read-only public-inbox HTTP
and NNTP interfaces.  Currently, this requires L<Search::Xapian>
and L<DBD::SQlite> and L<DBI> Perl modules.

Once the initial indices are created by public-inbox-index,
L<public-inbox-mda(1)> and L<public-inbox-watch(1)> will
automatically maintain them.

Running this manually to update indices is only required if
relying on L<git-fetch(1)> to mirror an existing public-inbox.

Having a search and article number database is essential to
running the NNTP interface, and strongly recommended for the
HTTP interface as it provides thread grouping in addition
to normal search functionality.

=head1 FILES

All public-inbox-specific files are contained within the
C<$GIT_DIR/public-inbox/> directory.  All files are expected to
grow in size as more messages are archived, so using compaction
commands (e.g. L<xapian-compact(1)>) is not recommended unless
the list is no longer active.

=item $GIT_DIR/public-inbox/msgmap.sqlite3

The stable NNTP article number to Message-ID mapping is
stored in an SQLite3 database.

This is required for users of L<public-inbox-nntpd(1)>, but
users of the L<PublicInbox::WWW> interface will find it
useful for attempting recovery from copy-paste truncations of
URLs containing long Message-IDs.

Avoid removing this file and regenerating it; it may cause
existing NNTP readers to lose sync and miss (or duplicate)
messages.

This file is relatively small, and typically less than 5%
of the space of the mail stored in a packed git repository.

=item $GIT_DIR/public-inbox/xapian*

The database used by L<Search::Xapian>.  This directory name is
followed by a number indicating the index schema version this
installation of public-inbox uses.

These directories may be safely deleted or removed in full
while the NNTP and HTTP interfaces are no longer accessing
them.

In addition to providing a search interface for the HTTP
interface, the Xapian database is used to group and combine
related messages into threads.  For NNTP servers, it also
provides a cache of metadata and header information often
requested by NNTP clients.

This directory is large, often two to three times the size of
the objects stored in a packed git repository.

=head1 ENVIRONMENT

=over 8

=item PI_CONFIG

Used to override the default "~/.public-inbox/config" value.

=back

=head1 CONTACT

Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>

The mail archives are hosted at L<https://public-inbox.org/meta/>
and L<http://hjrcffqmbrq6wope.onion/meta/>

=head1 COPYRIGHT

Copyright 2016 all contributors L<mailto:meta@public-inbox.org>

License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>

=head1 SEE ALSO

L<Search::Xapian>, L<DBD::SQLite>

debug log:

solving 554346d ...
found 554346d in https://80x24.org/public-inbox.git

(*) Git path names are given by the tree(s) the blob belongs to.
    Blobs themselves have no identifier aside from the hash of its contents.^

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).