about summary refs log tree commit homepage
path: root/MANIFEST
DateCommit message (Collapse)
2020-01-04xt/solver.t: real-world regression tests
There's a lot of test cases which we should probably make self-contained at some point, but right now it's easier to just mark them off in a maintainer test.
2020-01-03examples: add empty "lib" dir to placate plackup
This is necessary for Filesys::Notify::Simple 0.13 using Linux::Inotify2, since 0.13 started croaking on inotify_add_watch failures.
2020-01-02doc: release notes: set Date for 1.2.0, start 1.3.0
Seems like a lot's happened since 1.2, but it's mostly internal stuff...
2020-01-01wwwstatic: add directory listing + index.html support
It's now possible to use WwwStatic as a standalone PSGI app to serve static files and recreate the award-winning web design of https://public-inbox.org/ :>
2019-12-27githttpbackend: split out wwwstatic
Make it easier to share code between our GitHTTPBackend and Cgit packages, for now, and possibly other packages in the future. We can avoid inline_object and anonymous subs at the same time, reducing per-request memory overhead.
2019-12-19t/run.perl: to avoid repeated process spawning for *.t
Spawning a new Perl interpreter for every test case means Perl has to reparse and recompile every single file it needs, costing us performance and development time. Now that we've modified our code to avoid global state, we can preload everything we need. The new "check-run" test target is now 20-30% faster than the original "check" target.
2019-12-19tests: move t/common.perl to PublicInbox::TestCommon
We want to be able to use run_script with *.t files, so t/common.perl putting subs into the top-level "main" namespace won't work. Instead, make it a module which uses Exporter like other libraries.
2019-12-15address: use Email::Address::XS if available
Email::Address::XS is a dependency of modern versions of Email::MIME, so it's likely loaded and installed on newer systems, already; and capable of handling more corner-cases than our pure-Perl fallback. We still fallback to the imperfect-but-good-enough-in-practice pure-Perl code while avoiding the non-XS Email::Address (which was susceptible to DoS attacks (CVE-2015-7686)). We just need to keep "git fast-import" happy.
2019-12-14ds: move EvCleanup code into DS
EvCleanup only existed since Danga::Socket was a separate component, and cleanup code belongs with the event loop.
2019-12-12add msgtime_cmp maintainer test
Changes will be coming for MsgTime to stop depending on Date::Parse due to lack of package availability on OpenBSD and suboptimal performance on RFC822 dates.
2019-12-12git: async batch interface
This is a transitionary interface which does NOT require an event loop. It can be plugged into in current synchronous code without major surgery. It allows HTTP/1.1 pipelining-like functionality by taking advantage of predictable and well-specified POSIX pipe semantics by stuffing multiple git cat-file requests into the --batch pipe With xt/git_async_cmp.t and GIANT_GIT_DIR=git.git, the async interface is 10-25% faster than the synchronous interface since it can keep the "git cat-file" process busier. This is expected to improve performance on systems with slower storage (but multiple cores).
2019-11-27httpd|nntpd: avoid missed signal wakeups
Our attempt at using a self-pipe in signal handlers was ineffective, since pure Perl code execution is deferred and Perl doesn't use an internal self-pipe/eventfd. In retrospect, I actually prefer the simplicity of Perl in this regard... We can use sigprocmask() from Perl, so we can introduce signalfd(2) and EVFILT_SIGNAL support on Linux and *BSD-based systems, respectively. These OS primitives allow us to avoid a race where Perl checks for signals right before epoll_wait() or kevent() puts the process to sleep. The (few) systems nowadays without signalfd(2) or IO::KQueue will now see wakeups every second to avoid missed signals.
2019-11-27dskqxs: fix missing EV_DISPATCH define
Oops, IO::KQueue support was broken due to this missing constant. Add a new ds-kqxs.t test case to ensure we test the IO::KQueue path if IO::KQueue is available.
2019-11-24tests: move giant inbox/git dependent tests to xt/
xt/ is typically reserved for "eXtended tests" intended for the maintainers and not ordinary users. Since these require special configuration and do nothing by waste cycles during startup, they qualify.
2019-11-24tests: quiet down commit graph
Newer versions of git enable the commit graph by default. Since we blow away our temporary directories every test, generating graphis is a waste and clutters stderr with "Computing commit graph generation numbers" messages.
2019-11-16mbox: split mboxgz out into a separate file
It'll make using Compress::Raw::Zlib easier, since we can use that and import constants more easily.
2019-11-03public-inbox v1.2.0 v1.2.0
2019-11-03doc: add public-inbox.cgi(1) manpage
Yet another case of documenting things which should NOT be used :>
2019-11-02doc: add public-inbox-purge(1) manpage
Tools intended for end users need manpages, and doubly so to convince potential users NOT to use them :)
2019-10-31msgiter: do not assume UTF-8 if Email::MIME->body_str succeeds
ISO-2202-JP and other non-UTF-8 messages need to be displayed correctly. Fixes: 7d82a8bc04ce ('handle "multipart/mixed" messages which are not multipart')
2019-10-30doc: add public-inbox-learn(1) manpage
Tools intended for end users need manpages.
2019-10-09doc: PublicInbox::SaPlugin::ListMirror manpage
This is a plugin for SpamAssassin that happens to be quite useful in keeping spam off lists I mirror. Hopefully more people can find it useful now that it has a manpage.
2019-10-07examples: add grok-pull post_update_hook example
This requires the latest (to be in 1.2) -init changes for synchronization and has no dependencies on GNU or bash-isms so it should run on *BSD systems without GNU tools. It does attempt to use curl on <$INBOX_URL/_/text/config/raw>, but curl is fairly standard nowadays, and falls back to using an invalid address to initialize.
2019-10-07doc: generate NEWS, NEWS.atom, and NEWS.html
We'll use our Documentation/RelNotes directory and internal APIs to generate these files for website use (the website should be completely reproducible).
2019-10-05doc: add manpage for public-inbox-init(1)
This old command was lacking a manpage, so (finally) create one.
2019-09-18doc: add release notes directory
The v1.2.0 is a work-in-progress, while the others are copied out of our mail archives. Eventually, a NEWS file will be generated from these emails and distributed in the release tarball. There'll also be an Atom feed for the website reusing our feed generation code.
2019-09-14doc: update nntpd with NNTPS and STARTTLS examples
NNTPS and STARTTLS seems to be working for several months without incident on news.public-inbox.org, so consider it a success and maybe others can try using it. HTTPS technically works, too, but isn't documented at the moment since I can't recommend production deployments without varnish protecting it.
2019-09-14tmpfile: give temporary files meaningful names
Although we always unlink temporary files, give them a meaningful name so that we can we can still make sense of the pre-unlink name when using lsof(8) or similar tools on Linux.
2019-07-06nntp: support COMPRESS DEFLATE per RFC 8054
This is only tested so far with my patches to Net::NNTP at: https://rt.cpan.org/Ticket/Display.html?id=129967 Memory use in C10K situations is disappointing, but that's the nature of compression. gzip compression over HTTPS does have the advantage of not keeping zlib streams open when clients are idle, at the cost of worse compression.
2019-06-29http: support HTTPS (kinda)
It's barely any effort at all to support HTTPS now that we have NNTPS support and can share all the code for writing daemons. However, we still depend on Varnish to avoid hug-of-death situations, so supporting reverse-proxying will be required.
2019-06-24ds: reimplement IO::Poll support to look like epoll
At least the subset of epoll we use. EPOLLET might be difficult to emulate if we end up using it.
2019-06-24ds: split out IO::KQueue-specific code
We don't need to code multiple event loops or have branches in watch() if we can easily make the IO::KQueue-based interface look like our lower-level epoll_* API.
2019-06-24nntp: NNTPS and NNTP+STARTTLS working
It kinda, barely works, and I'm most happy I got it working without any modifications to the main NNTP::event_step callback thanks to the DS->write(CODE) support we inherited from Danga::Socket.
2019-06-14v2: rename SearchIdxPart => SearchIdxShard
Another step towards keeping our file and package names consistent with Xapian terminology.
2019-06-14Merge remote-tracking branch 'origin/reshard' into next
* origin/reshard: xcpdb: support resharding v2 repos xcpdb: use destination shard as progress prefix xapcmd: preserve indexlevel based on the destination v2writable: use a smaller default for Xapian partitions
2019-06-14Merge remote-tracking branch 'origin/manifest' into next
* origin/manifest: git: ensure ->modified returns an integer www: support $INBOX/git/$EPOCH.git for v2 cloning www: wire up /$INBOX/manifest.js.gz, too wwwlisting: generate grokmirror-compatible manifest.js.gz wwwlisting: allow hiding entries from manifest
2019-06-14xcpdb: support resharding v2 repos
v2 repos are sometimes created on machines where CPU parallelization exceeds the capability of the storage devices. In that case, users may reshard the Xapian DB to any smaller, positive integer to avoid excessive overhead and contention when bottlenecked by slow storage. Resharding can also be used to increase shard count after hardware upgrades.
2019-06-09edit: new tool to perform edits
This wrapper around V2Writable->replace provides a user-interface for editing messages as single-message mboxes (or the raw text via $EDITOR).
2019-06-09purge: start moving common options to AdminEdit module
Editing and purging are similar operations involving history rewrites, so there'll be common options and code between them.
2019-06-09v2writable: implement ->replace call
Much of the existing purge code is repurposed to a general "replace" functionality. ->purge is simpler because it can just drop the information. Unlike ->purge, ->replace needs to edit existing git commits (in case of From: and Subject: headers) and reindex the modified message. We currently disallow editing of References:, In-Reply-To: and Message-ID headers because it can cause bad side effects with our threading (and our lack of rethreading support to deal with excessive matching from incorrect/invalid References).
2019-06-09wwwlisting: generate grokmirror-compatible manifest.js.gz
Support on-demand generation of "/manifest.js.gz" for inboxes. By default, this matches inboxes with URLs matching the given request hostname by default. This makes it easier to create full mirrors of several inboxes without needing to configure static file serving. cf. https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git
2019-06-05scripts: add README to describe its purpose
Well, it could probably be moved to contrib...
2019-06-04t: avoid "subtest" for Perl 5.10.1 compatibility
The version of Test::More from Perl 5.10.1 did not support "subtest", and the earliest version which did is Perl 5.12.0 The good news is this gives me an excuse to parallelize the indexlevels-mirror test by splitting it into two. (it could be further split, even). Update t/nntpd. to use PI_TEST_VERSION consistently while we're at it.
2019-06-04MANIFEST: add examples/nginx_proxy
2019-06-01ds: fix and test for FD leaks with kqueue on ->Reset
Even though we currently don't use it repeatedly, ->Reset should close() kqueue FDs and not cause the process to run out of descriptors. Add a close-on-exec test while we're at it.
2019-05-24MANIFEST: add extman.perl
Oops :x
2019-05-23xcpdb: new tool which wraps Xapian's copydatabase(1)
copydatabase(1) is an existing Xapian tool which is the recommended way to upgrade existing DBs to the latest Xapian database format (currently "glass" for stable/released versions). Our use of Xapian relies on preserving document IDs, so we'll wrap it like we do xapian-compact(1) and use the "--no-renumber" switch. I could not name the tool "public-inbox-copydatabase" since it would be ambiguous as to which DB it's actually copying. So, I abbreviated the suffix to "xcpdb" (Xapian CoPy DataBase), which I hope is acceptable and unambiguous.
2019-05-23xapcmd: new module for wrapping Xapian commands
Port public-inbox-compact(1) over to using it, and we will need to wrap copydatabase(1) to ease glass migrations, too.
2019-05-23v1writable: retire in favor of InboxWritable
In retrospect, introducing V1Writable was unnecessary and InboxWritable->importer is in a better position to abstract away differences between v1 and v2 writers. So teach InboxWritable to initialize inboxes and get rid of V1Writable.
2019-05-15doc: add marketing guide
Preventative measures; since marketing is almost always annoying to me. And trying to avoid unintended consequences.