user/dev discussion of public-inbox itself
 help / color / Atom feed
acc903921e94e8a4a641fcc6c34b62a420e06872 blob 3465 bytes (raw)

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
 
=head1 NAME

public-inbox-index - create and update search indices

=head1 SYNOPSIS

public-inbox-index [OPTIONS] REPO_DIR

=head1 DESCRIPTION

public-inbox-index creates and updates the search and NNTP
article number database used by the read-only public-inbox HTTP
and NNTP interfaces.  Currently, this requires L<Search::Xapian>
and L<DBD::SQlite> and L<DBI> Perl modules.

Once the initial indices are created by public-inbox-index,
L<public-inbox-mda(1)> and L<public-inbox-watch(1)> will
automatically maintain them.

Running this manually to update indices is only required if
relying on L<git-fetch(1)> to mirror an existing public-inbox;
or if upgrading to a new version of public-inbox using
the C<--reindex> option.

Having a search and article number database is essential to
running the NNTP interface, and strongly recommended for the
HTTP interface as it provides thread grouping in addition
to normal search functionality.

=head1 OPTIONS

=over

=item --reindex

Forces a search engine re-index of all messages in the
repository.  This can be used for in-place upgrades while
NNTP/HTTP server processes are utilizing the index.  Keep in
mind this roughly doubles the size of the already-large
Xapian database.

This does not touch the NNTP article number database.

=back

=head1 FILES

All public-inbox-specific files are contained within the
C<$REPO_DIR/public-inbox/> directory.  All files are expected to
grow in size as more messages are archived, so using compaction
commands (e.g. L<xapian-compact(1)>) is not recommended unless
the list is no longer active.

=over

=item $REPO_DIR/public-inbox/msgmap.sqlite3

The stable NNTP article number to Message-ID mapping is
stored in an SQLite3 database.

This is required for users of L<public-inbox-nntpd(1)>, but
users of the L<PublicInbox::WWW> interface will find it
useful for attempting recovery from copy-paste truncations of
URLs containing long Message-IDs.

Avoid removing this file and regenerating it; it may cause
existing NNTP readers to lose sync and miss (or see duplicate)
messages.

This file is relatively small, and typically less than 5%
of the space of the mail stored in a packed git repository.

=item $REPO_DIR/public-inbox/xapian*

The database used by L<Search::Xapian>.  This directory name is
followed by a number indicating the index schema version this
installation of public-inbox uses.

These directories may be safely deleted or removed in full
while the NNTP and HTTP interfaces are no longer accessing
them.

In addition to providing a search interface for the HTTP
interface, the Xapian database is used to group and combine
related messages into threads.  For NNTP servers, it also
provides a cache of metadata and header information often
requested by NNTP clients.

This directory is large, often two to three times the size of
the objects stored in a packed git repository.  Using the
C<--reindex> option makes it larger, still.

=back

=head1 ENVIRONMENT

=over 8

=item PI_CONFIG

Used to override the default "~/.public-inbox/config" value.

=back

=head1 CONTACT

Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>

The mail archives are hosted at L<https://public-inbox.org/meta/>
and L<http://hjrcffqmbrq6wope.onion/meta/>

=head1 COPYRIGHT

Copyright 2016-2018 all contributors L<mailto:meta@public-inbox.org>

License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>

=head1 SEE ALSO

L<Search::Xapian>, L<DBD::SQLite>
debug log:

solving acc9039 ...
found acc9039 in https://80x24.org/public-inbox.git

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox