about summary refs log tree commit homepage
DateCommit message (Collapse)
2019-05-22ci: support CentOS-7
Tested on an amd64 chroot built with rinse 3.4
2019-05-22t/search*: require DBI and DBD::SQLite, too
None of the Search::Xapian-dependent stuff works without DBI and DBD::SQLite. There are no plans to support Xapian w/o DBD::SQLite since SQLite is more common and less resource-intensive than Xapian.
2019-05-22git: workaround old git-rev-parse(1) (--git-path)
git < 2.5.0 was missing --git-path support. This means any users relying on some rare environment variables will need git 2.5.0+
2019-05-22t/watch_filter_rubylang: disable v2 test for git < 2.6
This test was not disabled properly for ancient versions of git without get-mark support.
2019-05-22init: preserve permissions for git prior to 2.1.0
"git config" did not preserve permissions of the config file it modifies prior to git 2.1.0, so workaround that.
2019-05-21Merge remote-tracking branch 'origin/xap-optional' into master
* origin/xap-optional: admin: improve warnings and errors for missing modules searchidx: do not create empty Xapian partitions for basic lazy load Xapian and make it optional for v2 www: use Inbox->over where appropriate nntp: use Inbox->over directly inbox: add ->over method to ease access
2019-05-21searchidx: remove unused Compress::Zlib import
We only compress in OverIdx, now; since we no longer do overview stuff in Xapian (and Xapian compresses document data, anyways).
2019-05-17PublicInbox::Import::add: Consolidate subject handling
Consolidate subject handling in the add function to make it easier to read and understand. Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2019-05-16www: unescape '+' => ' ' before general URI unescape
This allows searching for terms with "+" in them properly.
2019-05-16Revert "view: perform highlighting for space-prefixed diffs"
This was buggy and was causing non-diff text to have extra leading spaces. The diff parsing code needs to be cleaned up, so this will be fixed, later. This reverts commit 1a67b91c1326efa372d1ec957e2494849d894f0b.
2019-05-16search: disable phrase searching, for now
There probably needs to be an option to enable this independently of indexlevel; but for now this is the safest option. And, as I discovered during the development of the indexlevel option, Xapian does a pretty good job of finding phrases without position data, anyways.
2019-05-16view: perform highlighting for space-prefixed diffs
"git format-patch --interdiff" and similar can prefix diffs with leading white space. Teach our diff parser to account for it and set appropriate CSS classes for them.
2019-05-15remove hard Devel::Peek dependency and lazy load for daemons
It's only useful for a corner case in long-running daemons when an admin decides to compact or vacuum a Xapian or SQLite DB. As a result, other scripts should run slightly faster. For instance, this saves about 80ms (2.710s => 2.630s) in t/mda.t on my remote workstation. While we're at it, make sure EvCleanup is properly require'd in Daemon.pm and HTTP.pm and document our use of Devel::Peek.
2019-05-15inbox: remove POSIX strftime import
We no longer need it since Inbox->recent hits the overview DB instead of Xapian
2019-05-15doc: add marketing guide
Preventative measures; since marketing is almost always annoying to me. And trying to avoid unintended consequences.
2019-05-15admin: improve warnings and errors for missing modules
Since we lazy-load Xapian now, some errors may become more cryptic or buried. Try to improve that by making Admin show better errors.
2019-05-15searchidx: do not create empty Xapian partitions for basic
No point in leaving a mess of empty directories when Xapian doesn't load.
2019-05-15lazy load Xapian and make it optional for v2
More tests work without Search::Xapian, now. Usability issues still need to be fixed
2019-05-15www: use Inbox->over where appropriate
We don't need to rely on Xapian search functionality for the majority of the WWW code, even. subject_normalized is moved to SearchMsg, where it (probably) makes more sense, anyways.
2019-05-15nntp: use Inbox->over directly
None of the NNTP code actually relies on Xapian, anymore.
2019-05-15inbox: add ->over method to ease access
One small step towards making installing Xapian optional for v2 and providing more WWW and NNTP functionality without it.
2019-05-14tests: get rid of unnecessary Cwd module use
We only need it for tests that chdir, and maybe for ENV{PATH} portability (dash seems fine, not sure about others). v2: revert change to solver_git.t for FreeBSD 11.2 and document
2019-05-14t/nntp.t: skip if Data::Dumper is missing
We can revisit this, later; but Data::Dumper requires a separate package for CentOS-7 users, at least.
2019-05-14t/config.t: remove Data::Dumper dependency
CentOS-7 needs the perl-Data-Dumper package, and the test is small enough to roll our own escaping, here.
2019-05-14tests: remove unnecessary loading of ::DS and Socket
PublicInbox::DS works for every platform we we care about, nowadays; so checking for it is a waste of time. Cleanup a few POSIX and Socket imports while we're in the area.
2019-05-14httpd: get rid of Deflater warning
Deflating responses may be done by the reverse proxy (e.g. varnish or nginx), so the warning for it could be invalid.
2019-05-14searchidx: fix incremental index with indexlevel=basic on v1
We were reindexing the full history every invocation of -index when Xapian was not used because we were incorrectly relying on 'last_commit' metadata stored in Xapian. Rewrite the indexing logic to be less confusing while we're at it, since we rely on `git merge-base --is-ancestor' nowadays. Furthermore, we need to handle message removals from the overview index correctly when Xapian is not in use. Co-authored-by: Eric W. Biederman <ebiederm@xmission.com>
2019-05-14v2writable: allow setting nproc via creat options
Avoiding reliance on environment variables is a bit cleaner for writing tests
2019-05-14v1writable: new wrapper which is closer to v2writable
Import initialization is a little strange from history, but we also can't change it too much because it's technically a public API which external code may rely on... And we may need to support v1 repos indefinitely. This should make it easier to write tests for both formats.
2019-05-09ci: include some scripts to make CI easier
This should make it easier to test a bunch of package installation profiles across whatever OS isolation one chooses (chroots, containers, jails, VMs).
2019-05-08doc/include.mk: remove git use and redundant declarations
And document that we still have GNU-isms in that include.mk Makefile (and may continue to do so). Finally, take advantage of GNU-isms to warn users to run "gmake" to build all manpages.
2019-05-08Makefile.PL: improve make(1) portability
The `shell' function appears missing, so we'll rely on Bourne shell expansioN, instead. Use "$?" instead of "$<" since the latter is only specified for inference and .DEFAULT rules, not target rules. Tested on FreeBSD make(1) and bmake(1) on Debian.
2019-05-08doc: give txt files proper titles
We can fix the redundant rule in include.mk which causes make(1) on FreeBSD to complain; but HTML docs will likely still require GNU make.
2019-05-08doc: use bullet list for wwwlisting options
Otherwise, pod2man complains about "=item 404" not starting with a letter and thinking it's part of a numbered list.
2019-05-08t/purge.t: fix unreferenced variable
2019-05-08INSTALL: note xapian-compact(1) tool
The Perl Xapian bindings do not support compacting, yet, so we rely on the xapian-compact tool.
2019-05-08INSTALL: depend on Encode instead of Encode::MIME::Header
Email::MIME uses Encode::MIME::Header and depends on that appropriately; however we depend on other parts of the Encode distribution, but that's bundled with Perl by upstream, anyways; and should place no additional burden on users.
2019-05-08INSTALL: update with FreeBSD pkg names and reorder
Fortunately, there is a pattern to most of these package names in all distros I've tested (and hopefully other BSDs have them, too). Then, reorder the INSTALL document to mention the top-level modules, first, so users can avoid needing to type extra dependencies. However, we still list some implicit dependencies in case the upstream package drops dependencies independently of us. Finally, Devel::Peek is not a dependency worth making optional since it's bundled by Perl upstream. Fedora/RH-based distros are the only one which turn it into a non-standard package when Perl5 is installed.
2019-05-08Merge remote-tracking branch 'origin/danga-bundle'
* origin/danga-bundle: DS: epoll: fix misordered EPOLL_CTL_DEL call DS: drop unused "_undef" sub syscall: drop readahead wrapper build: do not manify DS and Syscall pods DS: handle EINTR in IO::Poll path, too DS: workaround IO::Kqueue EINTR (mis-)handling DS: drop profiling support DS: remove unused fields and functions listener: use EPOLLEXCLUSIVE for listen sockets bundle Danga::Socket and Sys::Syscall
2019-05-08DS: epoll: fix misordered EPOLL_CTL_DEL call
Any operations on an fd after POSIX::close() are invalid, so epoll_ctl will fail. Worse off, in a multi-threaded Perl, the fd may be reused by another thread and EPOLL_CTL_DEL can hit the wrong file description as a result. cf. https://rt.cpan.org/Ticket/Display.html?id=129487
2019-05-08DS: drop unused "_undef" sub
No longer used since we removed the *_ip_string fields
2019-05-08syscall: drop readahead wrapper
No backwards compatibility to worry about for us; and fadvise is superior anyways.
2019-05-08build: do not manify DS and Syscall pods
We don't need to increase our install footprint with documentation from our internals (which will surely change).
2019-05-08DS: handle EINTR in IO::Poll path, too
IO::Poll::_poll returns -1, which is "true" to Perl. cf. https://rt.cpan.org/Ticket/Display.html?id=129484
2019-05-08nntp: avoid uninitialized variable from blank requests
We'll ignore blank lines from clients, since that's what innd seems to do.
2019-05-07wwwstream: do not load URI.pm
It's unneeded since commit e358bd7a3833f8c5 (2016-07-02) ("inbox: base_url method takes PSGI env hashref instead") So we only depend on URI::Escape from the "URI" CPAN distribution, at the moment.
2019-05-07spawn (Inline::C): fix off-by-one error
Noticed while testing on FreeBSD 11.2 amd64 with the optional Inline::C extension using clang 6.0.0. The end result on FreeBSD was spawning processes failed badly and things were immediately unusable with this enabled. av_len is a misleading API, and I failed to read the API comments in perl:/av.c which state: > Note that, unlike what the name implies, it returns > the highest index in the array, so to get the size of > the array you need to use "av_len(av) + 1". > This is unlike "sv_len", which returns what you would expect. If this bug affected anybody, it would've only affected users using both the optional Inline::C module AND set the PERL_INLINE_DIRECTORY environment variable. That said, I've never seen any evidence of it on Debian GNU/Linux + gcc on any x86 variant. That includes full 64-bit systems, a full 32-bit system, a 64-bit system with 32-bit userspace, across multiple gcc versions since 2016.
2019-05-06index: warn with info about the message as context
This can help users track down the source of warnings when presented with imperfect emails. While we're at it, make the __WARN__ callback in t/v2writable.t a no-op since we don't check for warnings, there.
2019-05-05Merge remote-tracking branch 'origin/wwwlisting'
* origin/wwwlisting: www: support listing of inboxes start depending on Perl 5.10.1+
2019-05-05DS: workaround IO::Kqueue EINTR (mis-)handling
IO::Kqueue seems unmaintained, so workaround a long-standing bug where it falls over on signals: https://rt.cpan.org/Ticket/Display.html?id=116615