Date | Commit message (Collapse) |
|
It'll make using Compress::Raw::Zlib easier, since we
can use that and import constants more easily.
|
|
|
|
Yet another case of documenting things which should NOT be used :>
|
|
Tools intended for end users need manpages, and doubly so
to convince potential users NOT to use them :)
|
|
ISO-2202-JP and other non-UTF-8 messages need to be displayed
correctly.
Fixes: 7d82a8bc04ce ('handle "multipart/mixed" messages which are not multipart')
|
|
Tools intended for end users need manpages.
|
|
This is a plugin for SpamAssassin that happens to be quite
useful in keeping spam off lists I mirror. Hopefully more
people can find it useful now that it has a manpage.
|
|
This requires the latest (to be in 1.2) -init changes for
synchronization and has no dependencies on GNU or bash-isms
so it should run on *BSD systems without GNU tools.
It does attempt to use curl on <$INBOX_URL/_/text/config/raw>,
but curl is fairly standard nowadays, and falls back to using
an invalid address to initialize.
|
|
We'll use our Documentation/RelNotes directory and internal APIs
to generate these files for website use (the website should be
completely reproducible).
|
|
This old command was lacking a manpage, so (finally) create one.
|
|
The v1.2.0 is a work-in-progress, while the others are copied
out of our mail archives.
Eventually, a NEWS file will be generated from these emails and
distributed in the release tarball. There'll also be an Atom
feed for the website reusing our feed generation code.
|
|
NNTPS and STARTTLS seems to be working for several months
without incident on news.public-inbox.org, so consider it a
success and maybe others can try using it.
HTTPS technically works, too, but isn't documented at
the moment since I can't recommend production deployments
without varnish protecting it.
|
|
Although we always unlink temporary files, give them a
meaningful name so that we can we can still make sense
of the pre-unlink name when using lsof(8) or similar
tools on Linux.
|
|
This is only tested so far with my patches to Net::NNTP at:
https://rt.cpan.org/Ticket/Display.html?id=129967
Memory use in C10K situations is disappointing, but that's
the nature of compression.
gzip compression over HTTPS does have the advantage of not
keeping zlib streams open when clients are idle, at the
cost of worse compression.
|
|
It's barely any effort at all to support HTTPS now that we have
NNTPS support and can share all the code for writing daemons.
However, we still depend on Varnish to avoid hug-of-death
situations, so supporting reverse-proxying will be required.
|
|
At least the subset of epoll we use. EPOLLET might be
difficult to emulate if we end up using it.
|
|
We don't need to code multiple event loops or have branches in
watch() if we can easily make the IO::KQueue-based interface
look like our lower-level epoll_* API.
|
|
It kinda, barely works, and I'm most happy I got it working
without any modifications to the main NNTP::event_step callback
thanks to the DS->write(CODE) support we inherited from
Danga::Socket.
|
|
Another step towards keeping our file and package names
consistent with Xapian terminology.
|
|
* origin/reshard:
xcpdb: support resharding v2 repos
xcpdb: use destination shard as progress prefix
xapcmd: preserve indexlevel based on the destination
v2writable: use a smaller default for Xapian partitions
|
|
* origin/manifest:
git: ensure ->modified returns an integer
www: support $INBOX/git/$EPOCH.git for v2 cloning
www: wire up /$INBOX/manifest.js.gz, too
wwwlisting: generate grokmirror-compatible manifest.js.gz
wwwlisting: allow hiding entries from manifest
|
|
v2 repos are sometimes created on machines where CPU
parallelization exceeds the capability of the storage devices.
In that case, users may reshard the Xapian DB to any smaller,
positive integer to avoid excessive overhead and contention when
bottlenecked by slow storage.
Resharding can also be used to increase shard count after
hardware upgrades.
|
|
This wrapper around V2Writable->replace provides a user-interface
for editing messages as single-message mboxes (or the raw text
via $EDITOR).
|
|
Editing and purging are similar operations involving history
rewrites, so there'll be common options and code between them.
|
|
Much of the existing purge code is repurposed to a general
"replace" functionality.
->purge is simpler because it can just drop the information.
Unlike ->purge, ->replace needs to edit existing git commits (in
case of From: and Subject: headers) and reindex the modified
message.
We currently disallow editing of References:, In-Reply-To: and
Message-ID headers because it can cause bad side effects with
our threading (and our lack of rethreading support to deal with
excessive matching from incorrect/invalid References).
|
|
Support on-demand generation of "/manifest.js.gz" for inboxes.
By default, this matches inboxes with URLs matching the given
request hostname by default.
This makes it easier to create full mirrors of several inboxes
without needing to configure static file serving.
cf. https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git
|
|
Well, it could probably be moved to contrib...
|
|
The version of Test::More from Perl 5.10.1 did not support
"subtest", and the earliest version which did is Perl 5.12.0
The good news is this gives me an excuse to parallelize
the indexlevels-mirror test by splitting it into two.
(it could be further split, even).
Update t/nntpd. to use PI_TEST_VERSION consistently while
we're at it.
|
|
|
|
Even though we currently don't use it repeatedly, ->Reset
should close() kqueue FDs and not cause the process to run
out of descriptors.
Add a close-on-exec test while we're at it.
|
|
Oops :x
|
|
copydatabase(1) is an existing Xapian tool which is the
recommended way to upgrade existing DBs to the latest Xapian
database format (currently "glass" for stable/released
versions). Our use of Xapian relies on preserving document IDs,
so we'll wrap it like we do xapian-compact(1) and use the
"--no-renumber" switch.
I could not name the tool "public-inbox-copydatabase" since it
would be ambiguous as to which DB it's actually copying. So, I
abbreviated the suffix to "xcpdb" (Xapian CoPy DataBase), which
I hope is acceptable and unambiguous.
|
|
Port public-inbox-compact(1) over to using it, and we will need
to wrap copydatabase(1) to ease glass migrations, too.
|
|
In retrospect, introducing V1Writable was unnecessary and
InboxWritable->importer is in a better position to abstract
away differences between v1 and v2 writers.
So teach InboxWritable to initialize inboxes and get rid
of V1Writable.
|
|
Preventative measures; since marketing is almost always annoying
to me. And trying to avoid unintended consequences.
|
|
We were reindexing the full history every invocation of -index
when Xapian was not used because we were incorrectly relying on
'last_commit' metadata stored in Xapian.
Rewrite the indexing logic to be less confusing while we're
at it, since we rely on `git merge-base --is-ancestor' nowadays.
Furthermore, we need to handle message removals from the
overview index correctly when Xapian is not in use.
Co-authored-by: Eric W. Biederman <ebiederm@xmission.com>
|
|
Import initialization is a little strange from history, but we
also can't change it too much because it's technically a public
API which external code may rely on...
And we may need to support v1 repos indefinitely. This should
make it easier to write tests for both formats.
|
|
This should make it easier to test a bunch of package
installation profiles across whatever OS isolation
one chooses (chroots, containers, jails, VMs).
|
|
* origin/danga-bundle:
DS: epoll: fix misordered EPOLL_CTL_DEL call
DS: drop unused "_undef" sub
syscall: drop readahead wrapper
build: do not manify DS and Syscall pods
DS: handle EINTR in IO::Poll path, too
DS: workaround IO::Kqueue EINTR (mis-)handling
DS: drop profiling support
DS: remove unused fields and functions
listener: use EPOLLEXCLUSIVE for listen sockets
bundle Danga::Socket and Sys::Syscall
|
|
* origin/wwwlisting:
www: support listing of inboxes
start depending on Perl 5.10.1+
|
|
These modules are unmaintained upstream at the moment, but I'll
be able to help with the intended maintainer once/if CPAN
ownership is transferred. OTOH, we've been waiting for that
transfer for several years, now...
Changes I intend to make:
* EPOLLEXCLUSIVE for Linux
* remove unused fields wasting memory
* kqueue bugfixes e.g. https://rt.cpan.org/Ticket/Display.html?id=116615
* accept4 support
And some lower priority experiments:
* switch to EV_ONESHOT / EPOLLONESHOT (incompatible changes)
* nginx-style buffering to tmpfile instead of string array
* sendfile off tmpfile buffers
* io_uring maybe?
|
|
I'm using this as the cgit about-filter and source-filter
in https://80x24.org/public-inbox.git
|
|
Incomplete at the moment, but this ought to be a handy reference
for both implementers and users alike.
|
|
We will still return a 404 by default to '/' for compatibility
with users of Plack::App::Cascade or similar. Inboxes are
sorted by modification times to help users detect activity
(similar to the /$INBOX/ topic view).
New configuration options:
* publicinbox.wwwlisting - configure the listing type
* publicinbox.<name>.hide - hide a particular inbox from the listing
See changes to public-inbox-config.pod for full descriptions
of the new options.
Requested-by: Leah Neukirchen <leah@vuxu.org>
https://public-inbox.org/meta/871sdfzy80.fsf@gmail.com/
|
|
We depend on git-http-backend for smart HTTP clone support,
however; since cgit does not support smart clones natively.
WWW.pm will be able to cascade down to this as a 404 handler in
the future.
|
|
Fixes: 285b9b4d7de53b0d ("examples/newswww.psgi: demonstrate standalone NewsWWW usage")
|
|
This is the fallback for the normal WWW endpoint.
Adding this to the top-level seems to be alright, since lynx and
w3m both understand nntp://<HOSTNAME>/<Message-ID> anyways.
If newsgroup and inbox names conflict, then consider it the
fault of the original sender.
Since NewsWWW is intended to support buggy linkifiers in mail clients,
they can interpret nntp:// URLs as http://<HOSTNAME>/<Message-ID>
Inbox ordering from the config file is preserved since
commit cfa8ff7c256e20f3240aed5f98d155c019788e3b
("config: each_inbox iteration preserves config order"),
so admins can rely on that to configure how scanning
works.
Requested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
cf. https://public-inbox.org/meta/20190107190719.GE9442@pure.paranoia.local/
nntp://news.public-inbox.org/20190107190719.GE9442@pure.paranoia.local
|
|
* origin/purge:
implement public-inbox-purge tool
v2writable: read epoch on purge
v2writable: cleanup processes when done
v2writable: purge ignores non-existent git epoch directories
v2writable: ->purge returns undef on no-op
import: purge: reap fast-export process
hoist out resolve_repo_dir from -index
|
|
I'll probably expose the PSGI service for cgit;
but it could be useful to others as well.
|
|
Since we now support more CSS classes for coloring,
give this feature more visibility.
|