Date | Commit message (Collapse) |
|
This should probably use lower-level git plumbing, but until
then, consistently add a bunch of --no-* options to "git log"
to get more consistent output.
Noticed-by: Johannes Berg
https://public-inbox.org/meta/1538164205.14416.76.camel@sipsolutions.net/
|
|
This allows v1 tests to continue working on git 1.8.0 for
now. This allows git 2.1.4 packaged with Debian 8 ("jessie")
to run old tests, at least.
I suppose it's safe to drop Debian 7 ("wheezy") due to our
dependency on git 1.8.0 for "merge-base --is-ancestor".
Writing V2 repositories requires git 2.6 for "get-mark"
support, so mask out tests for older gits.
|
|
These fields are only necessary in NNTP and not even stored in
Xapian; so keeping them around for the PSGI web UI search
results wastes nearly 80K when loading large result sets.
|
|
We need to parse the MIME object in order to get the
datestamp for those sites.
Fixes: 7d02b9e64455 ("view: stop storing all MIME objects on large threads")
|
|
Tests should not write to the default ~/.public-inbox/emergency
|
|
The new t/*filter_rubylang.t tests call -index immediately
after -init, which causes confusing messages to show up to
the end user.
Check the validity of the ref before calling "git-log".
|
|
Clearly the AltId stuff was never tested for v2. Ensure
this tricky filter (which reuses Msgmap to avoid introducing
new serial numbers) doesn't trigger deadlocks SQLite due
to opening a DB for writing multiple times.
I went through several iterations of this change before
going with this one, which is the least intrusive I could
fine.
|
|
Remove redundant slashes while we're at it.
|
|
Not sure what I was smoking when I originally wrote this code.
cf. https://public-inbox.org/meta/874li887mp.fsf@vuxu.org/
|
|
Most of these test cases are in t/plack.t, already; and that
runs much faster. Just ensure the slashy corner case and search
stuff works. While we're at it, avoid using the
public-inbox-index command and just use the internal API to
index.
|
|
No point in implementing these slowly with the CGI wrapper
when PSGI is sufficient for testing.
|
|
No need to test this via CGI .cgi is a wrapper around
PSGI and PSGI tests are way faster.
|
|
It is redundant with what is in t/plack.t
|
|
t/plack.t already has the same test.
|
|
More of this test will be, we use PSGI nowadays; and
most of these tests can be ported over to use PSGI and
not fork+exec as much.
|
|
No need to waste cycles with this anymore.
|
|
libxapian30:amd64 1.4.9-1 on Debian sid seems to give an 8KB
position.glass database with "hello world" as the document
regardless of our indexlevel. Use the text of the AGPL-3.0 for
a more realisitic Xapian database size.
And perhaps tying our tests to the AGPL will make life more
difficult for would-be copyright violators :>
|
|
I've found two examples on https://lore.kernel.org/lkml/
where the messages declared themselves to be "multipart/mixed"
but were actually plain text:
<87llgalspt.fsf@free.fr>
<200308111450.h7BEoOu20077@mail.osdl.org>
With the mboxrd downloaded, mutt is able to view them without
difficulty.
Note: this change would require reindexing of Xapian to pick up
the changes. But it's only two ancient messages, the first was
resent by the original sender and the second is too old to be
relevant.
|
|
We can't skip tests after "use_ok"
|
|
No need to write our own loop when an assignment will do.
|
|
IPC::Run provides a nice simplification in several places; and
we already use it (optionally) on a lot of tests.
For the non-test code, we still rely on our vfork-capable
Inline::C stuff since real-world server processes can get large
enough to where vfork is an advantage. Maybe Perl5 can use
CLONE_VFORK somehow, one day:
https://rt.perl.org/Ticket/Display.html?id=128227
Ohg V'q engure cbeg choyvp-vaobk gb Ehol :C
|
|
This allows archivists to publish incomplete archives with newer
mail while allowing "0.git" (or "1.git" and so on) epochs to be
added-after-the-fact (without affecting "git clone" followers).
A reindex will be necessary for Xapian and SQLite to catch up
once the old epochs are added; but the reindexing code is also
capable of tolerating missing epochs.
|
|
This can be useful for configuring archives of lists which are
no longer active.
|
|
TEST_CHUNK hast not been relevant since 2016:
(commit bb38f0fcce73904e "http: chunk in the server, not middleware")
|
|
This performance test doesn't normally get run...
Fixes: dd7049951c052c54 ("Put the NNTP server name into Xref lines")
|
|
It is redundant to set default values in the public-inbox
config file. Lets not clutter up users' screens when they
view or edit the config file.
|
|
When a client starts pipelining requests to us which trigger
long responses, we need to keep socket readiness checks disabled
and only enable them when our socket rbuf is drained.
Failure to do this caused aborted clients with
"BUG: nested long response" when Danga::Socket calls event_read
for read-readiness after our "next_tick" sub fires in the
same event loop iteration.
Reported-by: Jonathan Corbet <corbet@lwn.net>
cf. https://public-inbox.org/meta/20181013124658.23b9f9d2@lwn.net/
|
|
Putting the Xref field into xover lines allows newsreaders to mark
cross-posted messages read when catching up a group. That, in turn,
massively improves the life of crazy people who try to follow dozens of
kernel lists, where emails are often heavily cross-posted.
|
|
RFC 5536 sec 3.2.14 says that the server-name in an Xref line is "which
news server generated the header field"; indeed, that is necessary for
newsreaders like gnus to handle references properly. So pick up the server
name from the config if available (the first name if there's more than
one), from the host name otherwise, and use it rather than the domain
name of the list server.
Tests have been adjusted to match the new behavior.
|
|
Otherwise, walking backwards through history could mean the root
message in a thread forgets its `tid' and it prevents messages
from being looked up by it.
This bug was hidden by the fact that `sid' matches were often
good enough to link threads together.
|
|
Instrument the tests to verify the highwater num highwater mark is
where it is expected.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Capture interesting commits of the test repository in mark variables.
Use those marks to build interesting scenarios where index_sync proceeds
as if those marks are the heads of the repositor. Use this capability to
test what happens when adds and deletes are mixed within a repository.
Be sad because things don't yet work as they should.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Deeply inspect the entire message map in the reindexing tests
as the actual message order is significant and can result
in surprises.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Place the expected value second in is and isnt tests because when
these tests fail they report the second value as the expected value.
A report saying got 0 expected 8 'no Xapian search results' can be confusing.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
While inspecting the tests I realized that because we have been
reusing variables there can be a memory between one test case and
another. Add scopes and local variables to prevent an unintended
memory between one test cases.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
While inspecting the tests I realized that because we have been
reusing variables there can be a memory between one test case and
another. Add scopes and local variables to prevent an unintended
memory between one test and another.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
This reuses some of the configuration from -watch, but remains
independent since some configurations will use -watch for some
inboxes and -mda for others.
The default remains "spamc" for -mda users so nothing changes
without explicit configuration.
Per-inbox configurations may also be supported in the future.
|
|
We must not clobber the original message string, as Email::MIME(*)
still needs it for iterating through parts in SearchIdx (but not
when handing it as a raw string to git-fast-import).
I've noticed message bodies (especially dfpre/dpost) were not
getting indexed when going through -mda (no problems with
-watch). This also did not affect v1 repos, since indexing is a
separate process for v1 and requires re-reading the data from
git.
(*) tested Email::MIME 1.937 on Debian stretch
|
|
It will help track down a bug which only seems to happen in v2 repos.
|
|
For v1 repos, we don't need to write any metadata to Xapian
and changing from 'basic' to 'medium' or 'full' will work.
For v2, the metadata for indexing is stored in msgmap (because
the Xapian databases are partitioned for parallelism), so a
reindex is required.
|
|
The correct field is underscore-less for consistency with
git-config naming conventions. While we're at it, beef up
the v2 tests with actual size checks, too.
I also noticed phrase searching still seems to work for
the limited test case, so I left it documented; but the
size checking verifies the space savings.
|
|
This adds a new inbox configuration option 'indexlevel' that can take
the values 'full', 'medium', and 'basic'.
When set to 'full' everything is indexed including the positions
of all terms.
When set to 'medium' everything except the positions of terms is
indexed.
When set to 'basic' terms and positions are not indexed. Just the
Overview database for NNTP is created. Which is still quite good and
allows searching for messages by Message-ID. But there are no indexes to support
searching inside the email messages themselves.
Update the reindex tests to exercise the full medium and basic code paths
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Previously if a minmax test failed it would say it was expecting the
incorrect value, which is confusing when looking into why the test
fails.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Now that some of the indexes are optionals these tests might fail
so teach them to fail more cleanly.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Decrement regen_down when visiting messages that appear in %D that we
know will later be deleted. This ensures consistent message numbers are
generated no matter which commit number is on top. Allowing deletes to
propagage separately from the messages they delete without causing
problems.
The v2 trees already do this and when the indexes are deleted and
rebuilt they maintain they commit numbers.
Add a v1 version of the v2reindex test to verify that reindexing is
working properly on v1 as well as v2.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
The normal behavior is to prevent the deleted messages from
being indexed in the first place. However, when fetching
incrementally via git; public-inbox-index needs to account for
deleted files which were created outside of the most recent
fetch/reindexing window.
Reported-by: Eric W. Biederman <ebiederm@xmission.com>
|
|
Recently I ran git --git-dir=lkml/git/1.git fsck
and it reported:
> warning in commit 299dbd50b6995c6debe2275f0df984ce697fb4cc: nulInCommit: NULL byte inthe commit object body
Which I found quite scary. Nulls in the wrong place have a bad tendency
to make programs misbehave.
It turns out someone had placed "=?iso-8859-1?q?=00?=" at the end of
their subject line. Which is the mime encoding for NULL. Email::Mime
had correctly decoded the header, and then public-inbox had simply
copied the contents of the header into the subject line of the git
commit.
To prevent that from causing problems replace nulls in such subject
lines with spaces.
Signed-off-by: Eric Biederman <ebiederm@xmission.com>
|