about summary refs log tree commit
AgeCommit message (Collapse)AuthorFilesLines
2019-05-09ci: include some scripts to make CI easier ci-WIPEric Wong5-0/+366
This should make it easier to test a bunch of package installation profiles across whatever OS isolation one chooses (chroots, containers, jails, VMs)
2019-05-08doc/include.mk: remove git use and redundant declarationsEric Wong1-7/+19
And document that we still have GNU-isms in that include.mk Makefile (and may continue to do so). Finally, take advantage of GNU-isms to warn users to run "gmake" to build all manpages.
2019-05-08Makefile.PL: improve make(1) portabilityEric Wong1-3/+3
The `shell' function appears missing, so we'll rely on Bourne shell expansioN, instead. Use "$?" instead of "$<" since the latter is only specified for inference and .DEFAULT rules, not target rules. Tested on FreeBSD make(1) and bmake(1) on Debian.
2019-05-08doc: give txt files proper titlesEric Wong4-4/+8
We can fix the redundant rule in include.mk which causes make(1) on FreeBSD to complain; but HTML docs will likely still require GNU make.
2019-05-08doc: use bullet list for wwwlisting optionsEric Wong1-10/+6
Otherwise, pod2man complains about "=item 404" not starting with a letter and thinking it's part of a numbered list.
2019-05-08t/purge.t: fix unreferenced variableEric Wong1-1/+1
2019-05-08INSTALL: note xapian-compact(1) toolEric Wong1-0/+5
The Perl Xapian bindings do not support compacting, yet, so we rely on the xapian-compact tool.
2019-05-08INSTALL: depend on Encode instead of Encode::MIME::HeaderEric Wong2-6/+8
Email::MIME uses Encode::MIME::Header and depends on that appropriately; however we depend on other parts of the Encode distribution, but that's bundled with Perl by upstream, anyways; and should place no additional burden on users.
2019-05-08INSTALL: update with FreeBSD pkg names and reorderEric Wong2-77/+128
Fortunately, there is a pattern to most of these package names in all distros I've tested (and hopefully other BSDs have them, too). Then, reorder the INSTALL document to mention the top-level modules, first, so users can avoid needing to type extra dependencies. However, we still list some implicit dependencies in case the upstream package drops dependencies independently of us. Finally, Devel::Peek is not a dependency worth making optional since it's bundled by Perl upstream. Fedora/RH-based distros are the only one which turn it into a non-standard package when Perl5 is installed.
2019-05-08Merge remote-tracking branch 'origin/danga-bundle'Eric Wong23-47/+1415
* origin/danga-bundle: DS: epoll: fix misordered EPOLL_CTL_DEL call DS: drop unused "_undef" sub syscall: drop readahead wrapper build: do not manify DS and Syscall pods DS: handle EINTR in IO::Poll path, too DS: workaround IO::Kqueue EINTR (mis-)handling DS: drop profiling support DS: remove unused fields and functions listener: use EPOLLEXCLUSIVE for listen sockets bundle Danga::Socket and Sys::Syscall
2019-05-08DS: epoll: fix misordered EPOLL_CTL_DEL callEric Wong1-1/+1
Any operations on an fd after POSIX::close() are invalid, so epoll_ctl will fail. Worse off, in a multi-threaded Perl, the fd may be reused by another thread and EPOLL_CTL_DEL can hit the wrong file description as a result. cf. https://rt.cpan.org/Ticket/Display.html?id=129487
2019-05-08DS: drop unused "_undef" subEric Wong1-7/+0
No longer used since we removed the *_ip_string fields
2019-05-08syscall: drop readahead wrapperEric Wong1-14/+0
No backwards compatibility to worry about for us; and fadvise is superior anyways.
2019-05-08build: do not manify DS and Syscall podsEric Wong1-0/+10
We don't need to increase our install footprint with documentation from our internals (which will surely change).
2019-05-08DS: handle EINTR in IO::Poll path, tooEric Wong1-1/+1
IO::Poll::_poll returns -1, which is "true" to Perl. cf. https://rt.cpan.org/Ticket/Display.html?id=129484
2019-05-08nntp: avoid uninitialized variable from blank requestsEric Wong1-1/+1
We'll ignore blank lines from clients, since that's what innd seems to do.
2019-05-07wwwstream: do not load URI.pmEric Wong1-1/+0
It's unneeded since commit e358bd7a3833f8c5 (2016-07-02) ("inbox: base_url method takes PSGI env hashref instead") So we only depend on URI::Escape from the "URI" CPAN distribution, at the moment.
2019-05-07spawn (Inline::C): fix off-by-one errorEric Wong1-19/+28
Noticed while testing on FreeBSD 11.2 amd64 with the optional Inline::C extension using clang 6.0.0. The end result on FreeBSD was spawning processes failed badly and things were immediately unusable with this enabled. av_len is a misleading API, and I failed to read the API comments in perl:/av.c which state: > Note that, unlike what the name implies, it returns > the highest index in the array, so to get the size of > the array you need to use "av_len(av) + 1". > This is unlike "sv_len", which returns what you would expect. If this bug affected anybody, it would've only affected users using both the optional Inline::C module AND set the PERL_INLINE_DIRECTORY environment variable. That said, I've never seen any evidence of it on Debian GNU/Linux + gcc on any x86 variant. That includes full 64-bit systems, a full 32-bit system, a 64-bit system with 32-bit userspace, across multiple gcc versions since 2016.
2019-05-06index: warn with info about the message as contextEric Wong4-3/+15
This can help users track down the source of warnings when presented with imperfect emails. While we're at it, make the __WARN__ callback in t/v2writable.t a no-op since we don't check for warnings, there.
2019-05-05Merge remote-tracking branch 'origin/wwwlisting'Eric Wong8-8/+173
* origin/wwwlisting: www: support listing of inboxes start depending on Perl 5.10.1+
2019-05-05DS: workaround IO::Kqueue EINTR (mis-)handlingEric Wong2-5/+9
IO::Kqueue seems unmaintained, so workaround a long-standing bug where it falls over on signals: https://rt.cpan.org/Ticket/Display.html?id=116615
2019-05-05t/search.t: fix permissions check on FreeBSDEric Wong1-1/+11
FreeBSD does not allow non-root users to set S_ISGID; so git skips this bit on FreeBSD and Debian/kFreeBSD platforms.
2019-05-05DS: drop profiling supportEric Wong1-94/+0
There's other ways to profile and we don't need to add runtime branches to do this.
2019-05-05DS: remove unused fields and functionsEric Wong1-199/+0
More will likely be dropped in the future, but drop the obvious ones we aren't using, for now; especially since some of them are set at ->new time and unavoidable. This saves 579 bytes per-client on my 64-bit Debian stable system as measured by Devel::Size::total_size from PublicInbox::HTTP::event_read. This adds up in C10K or C100K situations. Things we drop are: * corked - MSG_MORE requires fewer syscalls * read_push_back - tried to use it, ate CPU with slow clients * IP/port fields - accept() already returns what we care about
2019-05-05listener: use EPOLLEXCLUSIVE for listen socketsEric Wong4-12/+22
Since our listen sockets are non-blocking and we may run multiple httpd|nntpd processes; we need a way to avoid thundering herds when there are multiple httpd|nntpd worker processes. EPOLLEXCLUSIVE was added just for that in Linux 4.5
2019-05-04bundle Danga::Socket and Sys::SyscallEric Wong22-40/+1698
These modules are unmaintained upstream at the moment, but I'll be able to help with the intended maintainer once/if CPAN ownership is transferred. OTOH, we've been waiting for that transfer for several years, now... Changes I intend to make: * EPOLLEXCLUSIVE for Linux * remove unused fields wasting memory * kqueue bugfixes e.g. https://rt.cpan.org/Ticket/Display.html?id=116615 * accept4 support And some lower priority experiments: * switch to EV_ONESHOT / EPOLLONESHOT (incompatible changes) * nginx-style buffering to tmpfile instead of string array * sendfile off tmpfile buffers * io_uring maybe?
2019-05-04wwwstream: drop tor2web URLEric Wong1-5/+0
It was a MITM hazard and been killed off by the DoS subsystem last July: https://marc.info/?i=87d0vwwkbs.fsf@riseup.net
2019-04-29doc/standards: add a link to PSGI specsEric Wong1-0/+3
2019-04-27extmsg: escape ampersands in @EXT_URL arrayEric Wong1-2/+2
We already escape the user-provided Message-IDs (so there's no security problem AFAIK), but the URL templates which exist in our source code were not escaped properly. This quiets down tidy(1).
2019-04-26viewdiff: do not break out of DSTATE_CTX on /^$/Eric Wong1-1/+1
It seems a common case for mangled patches is editors or MUAs dropping trailing whitespace, and lines matching /^ $/ gets the space dropped to only match /^$/.
2019-04-25cgit: improve handling of cgit data pathEric Wong2-3/+22
Document `publicinbox.cgitdata' config directive, but allow it to be unspecified and/or missing for installations which do not wish to serve static data at all. For users installing cgit from source to their home directory, we can usually infer the cgit data path based on the cgit.cgi binary path, even.
2019-04-25examples/cgit-commit-filter.lua: some doc updatesEric Wong1-0/+9
It's been a while since I wrote this, and it needs to be kept up-to-date with some advances in our Perl code.
2019-04-25examples: cgit filter for use with WwwHighlightEric Wong2-0/+106
I'm using this as the cgit about-filter and source-filter in https://80x24.org/public-inbox.git
2019-04-24wwwhighlight: deal with no-op highlightEric Wong1-1/+6
Not all inputs are highlight-able, so reuse the original input and just linkify it if it can't be highlighted.
2019-04-24examples/cgit-commit-filter: remove unused variableEric Wong1-1/+0
2019-04-24wwwhighlight: do linkification, tooEric Wong1-0/+4
Otherwise, there's no reason to use this API over highlight(1). Maybe this can be an option in the future; but I'm struggling to find a reason to not do it by default.
2019-04-23view: avoid "1+ messages" in per-message footer of /t/ and /T/Eric Wong1-1/+2
Try to appear gramatically correct and state: "only message in thread" when there's only one known (to us) message in the thread.
2019-04-23doc: add standards referenceEric Wong4-1/+81
Incomplete at the moment, but this ought to be a handy reference for both implementers and users alike.
2019-04-19www: support listing of inboxesEric Wong7-6/+171
We will still return a 404 by default to '/' for compatibility with users of Plack::App::Cascade or similar. Inboxes are sorted by modification times to help users detect activity (similar to the /$INBOX/ topic view). New configuration options: * publicinbox.wwwlisting - configure the listing type * publicinbox.<name>.hide - hide a particular inbox from the listing See changes to public-inbox-config.pod for full descriptions of the new options. Requested-by: Leah Neukirchen <leah@vuxu.org> https://public-inbox.org/meta/871sdfzy80.fsf@gmail.com/
2019-04-19start depending on Perl 5.10.1+Eric Wong2-2/+2
I mainly want to start using the '//' (defined-or) operator to simplify code, and Perl 5.10.1 is roughly a decade old at this point. "given/when" would've be nice, but it's future is in doubt AFAIK. I also started using the 'parent' module in WwwHighlight, and 'autodie' in UserContent.pm, both of which were only distributed with Perl since 5.10.1; and testing with ancient versions/distros is time-consuming. Anyways, I think this a small-enough jump to not break any existing installations, given we already depend on fairly recent versions of git and Xapian. Maybe we can use more newish Perl features in the future...
2019-04-19t/hl_mod: workaround w3m not handling &apos;Eric Wong1-0/+4
This fixes a test failure on my Debian buster system. Bug report filed for w3m to handle "&apos;": https://bugs.debian.org/927409 and for "highlight" to favor "&#39;" in case other browsers fail: https://bugs.debian.org/927410
2019-04-18linkify: require parentheses pairs in URLsEric Wong2-2/+25
Dangling parentheses with trailing punctuation usually means the parentheses is not intended as part of the URL.
2019-04-18view: show "(no subject)" consistently in HTMLEric Wong2-0/+4
Empty subjects ("") and undefined Subjects: are now both displayed as "(no subject)" for now.
2019-04-18doc: config: fix braino/typo :xEric Wong1-1/+1
2019-04-18linkify: don't get confused by URLs in Perl code, at leastEric Wong2-4/+30
The URLs at the top of WwwStream.pm weren't getting linkified correctly.
2019-04-18inbox: add `modified' subEric Wong1-0/+12
For inboxes with SQLite enabled (all v2, and probably most v1); we can use the overview DB to get the timestamp of the latest message. It's faster than scanning git branches for commit times, but not always the same.
2019-04-18git: calculate modified time of repositoryEric Wong3-1/+24
This will be used for generating an HTML listing for v1 inboxes, at least. The logic for this follows that of grokmirror, and we may dynamically generate manifest.js.gz natively...
2019-04-18config: use '$ibx' instead of '$rv' to denote Inbox objectsEric Wong1-24/+24
Followup-to: 6e6f7999361925e4 ("cleanup: use '$ibx' consistently when referring to Inbox refs")
2019-04-17TODO: add an item for hooks supportEric Wong1-0/+5
and add a note for grokmirror support/integration, too
2019-04-16HACKING: stop advertising mailing list subscription infoEric Wong1-5/+0
It's still available, but no point in advertising something which increases the dependency on a centralized subscriber list. In fact, it's actively against the goal of promoting resistance against centralization.