Date | Commit message (Collapse) |
|
This allows v1 tests to continue working on git 1.8.0 for
now. This allows git 2.1.4 packaged with Debian 8 ("jessie")
to run old tests, at least.
I suppose it's safe to drop Debian 7 ("wheezy") due to our
dependency on git 1.8.0 for "merge-base --is-ancestor".
Writing V2 repositories requires git 2.6 for "get-mark"
support, so mask out tests for older gits.
|
|
It looks like Net::Socket::IP comes with Perl 5.20 and
later; so we won't have to hassle users with another
package to install.
|
|
Hopefully this helps people familiarize themselves with
the source code.
|
|
We need to parse the MIME object in order to get the
datestamp for those sites.
Fixes: 7d02b9e64455 ("view: stop storing all MIME objects on large threads")
|
|
do_write must return 0 or 1.
|
|
While we try to discard the $smsg (SearchMsg) objects quickly,
they remain referenced via $node (SearchThread::Msg) objects,
which are stored forever in $ctx->{mapping} to cull redundant
words out of subjects in the thread skeleton.
This significantly cuts memory bloat with large search results
with '&x=t'. Now, the search results overhead of
SearchThread::Msg and linked objects are stable at around 350K
instead of ~7M per response in a rough test (there's more
savings to be had in the same areas).
Several hundred kilobytes is still huge and a large per-client
cost; but it's far better than MEGABYTES per-client.
|
|
I've hit /proc/sys/fs/pipe-user-pages-* limits on some systems.
So stop hogging resources on pipes which don't benefit from
giant sizes.
Some of these can use eventfd in the future to further reduce
resource use.
|
|
The new t/*filter_rubylang.t tests call -index immediately
after -init, which causes confusing messages to show up to
the end user.
Check the validity of the ref before calling "git-log".
|
|
Clearly the AltId stuff was never tested for v2. Ensure
this tricky filter (which reuses Msgmap to avoid introducing
new serial numbers) doesn't trigger deadlocks SQLite due
to opening a DB for writing multiple times.
I went through several iterations of this change before
going with this one, which is the least intrusive I could
fine.
|
|
|
|
Remove redundant slashes while we're at it.
|
|
Unused since commit 6c2caa791bd5fbf5c4edb1a4a2c1807e527348a7
("watchmaildir: support v2 repositories")
|
|
Not sure what I was smoking when I originally wrote this code.
cf. https://public-inbox.org/meta/874li887mp.fsf@vuxu.org/
|
|
There is no need for parallelism if we're not using Xapian.
|
|
Since "publicinbox" sections are analogous to git remotes, we
may use the same rules for naming git remotes to reduce
cognitive overhead.
Most notably, this allows '.' in the middle of inbox names,
(e.g. "foo.bar") as it's common for email addresses, too.
|
|
No need to reach into PublicInbox::Config internals and iterate
through the hashref by hand
|
|
Remove confusing documentation around ssoma now that we
have NNTP and downloadable mbox support.
Only lightly-checked for grammar and speling, and not yet
formatting. Edits, corrections and addendums expected :>
|
|
We can't run cleanup stuff without Danga::Socket.
|
|
GUI browsers have a tendency to use a larger (though sometimes
smaller) font than the rest of the page for some reason I could
not find...
So set everything to 100% to give uniformity to the page; which
benefits visually-challenged users who want to use gigantic
fonts for the entire page.
|
|
I've found two examples on https://lore.kernel.org/lkml/
where the messages declared themselves to be "multipart/mixed"
but were actually plain text:
<87llgalspt.fsf@free.fr>
<200308111450.h7BEoOu20077@mail.osdl.org>
With the mboxrd downloaded, mutt is able to view them without
difficulty.
Note: this change would require reindexing of Xapian to pick up
the changes. But it's only two ancient messages, the first was
resent by the original sender and the second is too old to be
relevant.
|
|
Unfortunately, long inbox names and URLs don't really display well
with my gigantic fonts...
|
|
Extracted from import_slrnspool, since some spools get converted
to mbox or what not.
|
|
This allows archivists to publish incomplete archives with newer
mail while allowing "0.git" (or "1.git" and so on) epochs to be
added-after-the-fact (without affecting "git clone" followers).
A reindex will be necessary for Xapian and SQLite to catch up
once the old epochs are added; but the reindexing code is also
capable of tolerating missing epochs.
|
|
This can be useful for configuring archives of lists which are
no longer active.
|
|
When a client starts pipelining requests to us which trigger
long responses, we need to keep socket readiness checks disabled
and only enable them when our socket rbuf is drained.
Failure to do this caused aborted clients with
"BUG: nested long response" when Danga::Socket calls event_read
for read-readiness after our "next_tick" sub fires in the
same event loop iteration.
Reported-by: Jonathan Corbet <corbet@lwn.net>
cf. https://public-inbox.org/meta/20181013124658.23b9f9d2@lwn.net/
|
|
Putting the Xref field into xover lines allows newsreaders to mark
cross-posted messages read when catching up a group. That, in turn,
massively improves the life of crazy people who try to follow dozens of
kernel lists, where emails are often heavily cross-posted.
|
|
RFC 5536 sec 3.2.14 says that the server-name in an Xref line is "which
news server generated the header field"; indeed, that is necessary for
newsreaders like gnus to handle references properly. So pick up the server
name from the config if available (the first name if there's more than
one), from the host name otherwise, and use it rather than the domain
name of the list server.
Tests have been adjusted to match the new behavior.
|
|
This ensures that the number of added files remains the same and thus
the article numbers derived from a repository will remain the same.
I think this is the last place in public-inbox that has to be tweaked to
guarantee the generated article number will remain the same in an public
inbox archive.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Otherwise, walking backwards through history could mean the root
message in a thread forgets its `tid' and it prevents messages
from being looked up by it.
This bug was hidden by the fact that `sid' matches were often
good enough to link threads together.
|
|
The "loose" (Subject:-based) thread matching yields too many
hits for some common subjects (e.g. "[GIT] Networking" on LKML)
and causes thread skeletons to not show the current messages.
Favor strict matches in the query and only add loose matches
if there's space.
While working on this, I noticed the backwards --reindex walk
breaks `tid' on v1 repositories, at least. That bug was hidden
by the Subject: match logic and not discovered until now. It
will be fixed separately.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
Incremental indexing fixes from Eric W. Biederman.
These prevents the highest message number in msgmap from
being reassigned after deletes in rare cases and ensures
messages are deleted from msgmap in v2.
* eb/index-incremental:
V2Writeable.pm: In unindex_oid delete the message from msgmap
V2Writeable.pm: Ensure that a found message number is in the msgmap
SearchIdx,V2Writeable: Update num_highwater on optimized deletes
t/v[12]reindex.t: Verify the num highwater is as expected
t/v[12]reindex.t Verify num_highwater
Msgmap.pm: Track the largest value of num ever assigned
SearchIdx.pm: Always assign numbers backwards during incremental indexing
t/v[12]reindex.t: Test incremental indexing works
t/v[12]reindex.t: Test that the resulting msgmap is as expected
t/v[12]reindex.t: Place expected second in Xapian tests
t/v2reindex.t: Isolate the test cases more
t/v1reindex.t: Isolate the test cases
Import.pm: Don't assume {in} and {out} always exist
|
|
Now that we track the num highwater mark it is safe to remove messages
from msgmap that have been previously allocated. Removing even the
highest numbered article will no longer cause new message numbers to
move backwards.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
The lookup to see if a num has already been assigned to a message
happens in a temporary copy of message map. It is possible that the
number has been removed from the current message map. The
unindex/reindex after a history rewrite triggered by a purge should be
one such case. Therefore add the number to the msgmap in case it is
not currently present.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
When performing an incremental index update with index_sync if a message is seen
to be both added and deleted update the num_highwater mark even though the
message is not otherwise indexed.
This ensures index_sync generates the same msgmap no matter which commit
it stops at during incremental syncs.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Today the only thing that prevents public-inbox not reusing the
message numbers of deleted messages is the sqlite autoincrement magic
and that only works part of the time. The new incremental indexing
test has revealed areas where today public-inbox does try to reuse
numbers of deleted messages.
Reusing the message numbers of existing messages is a problem because
if a client ever sees messages that are subsequently deleted the
client will not see the new messages with their old numbers.
In practice this is difficult to trigger because it requires the most
recently added message to be removed and have the removal show up in a
separate pull request. Still it can happen and it should be handled.
Instead of infering the highset number ever used by finding the maximum
number in the message map, track the largest number ever assigned directly.
Update Msgmap to track this value and update the indexers to use this
value.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Not sure what was going through my mind when I made my first
attempt at this, but we really want to make sure we index all
the text we display in the web view (and presumably anything a
reasonable mail client can display).
Followup-to: 0cf6196025d4e4880cd1ed859257ce21dd3cdcf6
("search: match the behavior of WWW for indexing text")
|
|
When walking messages newest to oldest, assigning the larger numbers
before smaller numbers ensures older messages get smaller numbers.
This leads to the possibility of a msgmap that can be regenerated when
needed.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
While working on one of the tests I did:
my $im = PublicInbox::V2Writable->new($ibx, 1);
my $im0 = $im->importer();
$im->add($mime);
Which resulted in a warning of the use of an undefined value from
atfork_child, and the test failing nastily. Inspection of the code
reveals this can happen anytime gfi_start has not been called.
So just fix atfork_child to skip closing file descriptors that have
not yet been setup.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
While playing with git fast export I discovered that mixing <> and
read would give inconsistent results. I tracked the issue down to
using sysread in ProcessPipe instead of plain read.
If it is desirable to use readline I can't see how using sysread
can work as readline to be efficient needs to use buffered I/O.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
This reuses some of the configuration from -watch, but remains
independent since some configurations will use -watch for some
inboxes and -mda for others.
The default remains "spamc" for -mda users so nothing changes
without explicit configuration.
Per-inbox configurations may also be supported in the future.
|
|
It's a convenient wrapper nowadays, so get rid of some legacy
code and minimize differences from the -watch code.
|
|
I've hit some case where probabilistic searches don't work when
using dfpre:/dfpost:/dfblob: search prefixes because stemming in
the query parser interferes.
In any case, our indexing code indexes longer/unabbreviated blob
names down to its 7 character abbreviation, so there should be
no need to do wildcard searches on git blob names.
|
|
For v1 repos, we don't need to write any metadata to Xapian
and changing from 'basic' to 'medium' or 'full' will work.
For v2, the metadata for indexing is stored in msgmap (because
the Xapian databases are partitioned for parallelism), so a
reindex is required.
|
|
Use ||= '' to ensure that if the From or Sender header is not present
the code sees an empty string and instead of undefined.
I had some email messages with a From field without an @ (because the
sender was local) and without a Sender which were causing errors when
imported. I think this was bad enough that the email messages were
failing to be imported.
Signed-off-by: Eric Biederamn <ebiederm@xmission.com>
|
|
Xapian documents and respect XAPIAN_FLUSH_THRESHOLD to define
the interval in documents to flush, so don't override it with
our own BATCH_BYTES. This is helpful for initial indexing for
those on slower storage but enough RAM.
It is unnecessary for -watch and frequent incremental indexing;
and it increases transaction times if -watch is playing "catch-up"
if it was stopped for a while.
The original BATCH_BYTES was tuned for a machine with little
memory as the default XAPIAN_FLUSH_THRESHOLD of 10000 documents
was causing swap storms. Using document counts also proved an
innaccurate estimator of RAM usage compared to the actual bytes
processed.
|
|
This adds a new inbox configuration option 'indexlevel' that can take
the values 'full', 'medium', and 'basic'.
When set to 'full' everything is indexed including the positions
of all terms.
When set to 'medium' everything except the positions of terms is
indexed.
When set to 'basic' terms and positions are not indexed. Just the
Overview database for NNTP is created. Which is still quite good and
allows searching for messages by Message-ID. But there are no indexes to support
searching inside the email messages themselves.
Update the reindex tests to exercise the full medium and basic code paths
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Create a new method add_xapian that holds all of the code to create
Xapian indexes. The creation of this method simpliy involved
idenitifying the relevant code and moving it from add_message.
A call is added to add_xapian from add_message to keep everything
working as it currently does. The new call is made conditional upon
index levels of 'full' and 'medium'. The index levels that index
positions and terms the two things public-inbox uses Xapian to index.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
About half the size of the Xapian search index turns out to be search
positions. The search positions are only used in a very narrow set of
queries. Make the search positions optional so people don't need to
pay the cost of queries they will never make.
This also makes public-inbox more approachable for light hacking as
generating all of the indexes is time consuming.
The way this is done is to add a method to SearchIdx called index_text
that wraps the call of the term generator method index_text. The new
index_text method takes care of calling both index_text and
increase_termpos (the two functions that are responsible for position
data).
Then index_users, index_diff_inc, index_old_diff_fn, index_diff,
index_body are made proper methods that calls the new index_text.
Callers of the new index_text are slightly simplified as they don't
need to call increase_termpos as well.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Decrement regen_down when visiting messages that appear in %D that we
know will later be deleted. This ensures consistent message numbers are
generated no matter which commit number is on top. Allowing deletes to
propagage separately from the messages they delete without causing
problems.
The v2 trees already do this and when the indexes are deleted and
rebuilt they maintain they commit numbers.
Add a v1 version of the v2reindex test to verify that reindexing is
working properly on v1 as well as v2.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
The normal behavior is to prevent the deleted messages from
being indexed in the first place. However, when fetching
incrementally via git; public-inbox-index needs to account for
deleted files which were created outside of the most recent
fetch/reindexing window.
Reported-by: Eric W. Biederman <ebiederm@xmission.com>
|