about summary refs log tree commit homepage
DateCommit message (Collapse)
2016-05-22www: avoid warnings on bad offsets for Xapian
The offset argument must be an integer for Xapian, however users (or bots) type the darndest things. AFAIK this has no security implications besides triggering a warning (which could lead to out-of-space-errors)
2016-05-22git-http-backend: switch to async_pass
This simplifies the code somewhat; but it could probably still be made simpler. It will need to support command queueing for expensive commands so expensive processes can be queued up.
2016-05-22http: rework async_pass support
Unfortunately, the original design did not work because middleware can wrap the response body and make `async_pass' invisible to HTTP.pm
2016-05-22git-http-backend: simplify dumb serving
We can rely entirely on getline + close callbacks and be compatible with 100% of PSGI servers.
2016-05-22git-http-backend: remove process limit
We will figure out a different way to avoid overloading...
2016-05-22t/spawn.t: additional tests for popen_rd
We need to ensure $? is set properly for users.
2016-05-22http: pass reference to Danga::Socket::write
This can avoid an expensive copy for big strings.
2016-05-22http: fix typo: write_buf => write_buf_size
Otherwise, we get deep recursion as we keep calling recursively on giant responses
2016-05-22http: async getline supports push_back_read
Sometimes we need to read something to ensure it's a successful response.
2016-05-21http: support async_pass for Danga::Socket
This will allow us to minimize buffering after we wait (possibly a long time) for readability. This also greatly reduces the amount of Danga::Socket-specific knowledge we have in our PSGI code, making it easier for others to understand.
2016-05-21import: avoid needless git update-server-info
We don't need to update-server-info (or read-tree) if fast import was spawned for removals and no changes were made.
2016-05-21daemon: simplify forking
We shouldn't need sigprocmask unless we're running multiple native threads or using vfork, neither of which is the case, here.
2016-05-21localize $/ in more places to avoid potential problems
This hopefully makes the intent of the code clearer, too. The the HTTP use of the numeric reference for getline caused problems in Git.pm, already.
2016-05-21unsubscribe: prevent decrypt from showing random crap
Wow, I don't know crypto at all.
2016-05-21examples/unsubscribe-psgi@.service: disable worker processes
This unsubscribe PSGI endpoint should never incur enough load to justify using multiple worker processes. If it's unstable and crashes, systemd can automatically restart it.
2016-05-21unsubscribe: bad URL fixup
Fixup a comment about s/query string/PATH_INFO/ while we're at it, as pre-published versions of this used query strings before I determined it could be harder to copy+paste URLs with query parameters in them.
2016-05-21mbox: switch generation over to pull model
This allows us to easily provide gigantic inboxes with proper backpressure handling for slow clients. It also eliminates public-inbox-httpd and Danga::Socket-specific knowledge from this class, making it easier to follow for those used to generic PSGI applications.
2016-05-21http: reduce over-buffering for getline responses
By switching to a "pull"-based I/O model for reading application responses, we should be able to throttle buffering to slow clients more effectively and avoid wasting precious RAM. This will also allow us to more Danga::Socket-specific knowledge out of the PSGI application and keep it confined to PublicInbox::HTTP.
2016-05-20ssoma-replay: use TMPDIR for temporary path
Otherwise, tempfile() will use the current working directory, which may not be writable.
2016-05-20unsubscribe: get off mah lawn^H^H^Hist
While public-inbox is intended primarily for archival, SMTP list subscriptions are still in use in most places and users are likely to want a good unsubscribe mechanism. HTTP (or HTTPS) links in the List-Unsubscribe header are often preferable since some users may use an incorrect email address for mailto: links. Thus, it is useful to provide an example which generates an HTTPS link for users to click on. The default .psgi requires a POST confirmation (as destructive actions with GET are considered bad practice). However, the "confirm" parameter may be disabled for a true "one-click" unsubscribe. The generated URLs are hopefully short enough and both shell and highlighting-friendly to reduce copy+paste errors.
2016-05-19www: tighten up allowable filenames for attachments
Having a file start with '.' or '-' can be confusing and for users, so do not allow it.
2016-05-19view: reduce clutter for attachments w/o description
For attachments without a filename or description, reduce the amount of precious screen space required to display a link to it.
2016-05-19www: validate and check filenames in URLs
We shall ensure links continue working for this.
2016-05-19msg_iter: workaround broken Email::MIME versions
Email::MIME >= 1.923 and < 1.935 would drop too many newlines in attachments. This would lead to ugly text files without a proper trailing newline if using quoted-printable, 7bit, or 8bit. Attachments encoded with base64 were not affected. These versions of Email::MIME are widely available in Debian 8 (Jessie) and even Ubuntu LTS distros so we will need to support this workaround for a while.
2016-05-19www: support downloading attachments
This can be useful for lists where the convention is to attach (rather than inline) patches into the message body.
2016-05-19switch read-only uses of walk_parts to msg_iter
msg_iter lets us know the index of the attachment, allow us to make more sensible labels and in a future commit, hyperlinks to download attachments.
2016-05-19msg_iter: new internal API for iterating through MIME
Unlike Email::MIME::walk_parts, this is non-recursive and gives depth + index offset information about the part for creating links for later retrieval It is intended for read-only access and changes are not propagated to the parent; however future versions of it may clobber bodies or the original version as it iterates to reduce memory overhead. It is intended for making it easy to locate attachments within a message in the WWW view.
2016-05-19view: rely on Email::MIME::body_str for decoding
Or is it "encoding"? Gah, Perl character set handling confuses me no matter how many times I RTFM :< This contains placeholders for attachment downloading which will be in a future commit.
2016-05-19nntpd: avoid uninitialized warning
Oops, but at least it was mostly harmless, just ugly. Followup-to: 9bfe40e7a4ac 'nntp: use "newsgroup" instead of "name"''
2016-05-18nntpd: reject control characters entirely
There's no place for them in the commands and we don't take messages; potentially printing them into a log opened in a terminal is too dangerous. Hoist out read_til_dot in the test while we're at it.
2016-05-18tests: add check-www-inbox script
This can be useful for hammering a live HTTP server with requests to ensure it does not fall over under load.
2016-05-18view: avoid redirect to reply endpoint
Oops, but perhaps the "reply" endpoint should be embedded into the permalink message view itself to reduce URLs.
2016-05-18feed: inline feed entry generation
Remove unnecessary wrapper subroutines and constants which are only used once.
2016-05-17http: release resources when idle
This lets us release old git processes so unlinked packs (leftover from repacking) can be released. This may also be helpful for Xapian as indices get rebuilt for tuning. For SQLite (msgmap), the there may be no benefit besides reducing FD pressure. Followup changes will unify the Inbox and NewsGroup classes and allow better code-sharing between NNTP and HTTP classes (as well as the planned POP3 class).
2016-05-17view: escape Message-ID for "next" link
Oops, we need to escape Message-IDs since they can contain bad characters such as '%' in them. '@' actually seems fine and does not need to be escaped; however, but we've been doing it forever.
2016-05-16www: fix for running under mount paths
We try to avoid issues like these by using relative URLs in hrefs, but we can't avoid the problem with Location: for redirects and Atom feeds which are likely to be rehosted elsewhere. We also reorder some of the code to work around a weird issue on the psgi-plack mailing list: <20160516073750.GA11931@dcvr.yhbt.net> (Somewhere on https://groups.google.com/group/psgi-plack but it's probably not bookmarkable)
2016-05-16config: allow taking an existing reference
This should make creating test cases easier and faster.
2016-05-16declare Inbox object for reusability
From the beginning, we've avoided objects here in favor of faster startup time; but it may not be worth it since a persistent httpd/nntpd is faster and -mda isn't hit as often.
2016-05-16doc: sync ~/.spamassassin/user_prefs with my prod machine
This is what I'm running on public-inbox.org as of today.
2016-05-15mbox: support /$INBOX/all.mbox.gz endpoint
Allows easily downloading the entire archive without special tools. In any case, it's not yet advertised to via HTML until we can test it better. It'll also support range queries in the future to avoid wasting bandwidth.
2016-05-15mbox: consistent header order when decompressed
This should make validating the output easier when testing between different servers.
2016-05-15git-http-backend: set cache headers
Mostly stolen from git upstream, these should prevent any caches such as varnish or squid from acting improperly.
2016-05-14rename most instances of "list" to "inbox"
A public-inbox is NOT necessarily a mailing list, but it could serve as an input point for zero, one, or infinite mailing lists :D
2016-05-14nntp: use "newsgroup" instead of "name"
This reduces the cognitive overhead for mapping names of configuration values to internal field names of our classes. Further changes along these lines coming...
2016-05-14import ssoma-replay example script I've been using
Unfortunately, most users still prefer their mail delivered over SMTP; so we'll at least document mlmmj integration for now until we can popularize pull-based reading over POP3/NNTP/ssoma.
2016-05-14t/nntpd: test for wide characters and UTF-8 mangling
We'll need to test non-UTF-8 messages at some point, too. There are lots of legacy-encoded messages in old archives and I would not bet we behave sanely w.r.t. those.
2016-05-14t/nntpd: avoid fork+exec for search indexing
The Xapian search index is required for the NNTP server, so there's no point in calling system() for it like we do in other tests. This should speed up the test a small amount.
2016-05-14build: support eatmydata in "make check" target by default
This should help poor developers who still use rotating disks on cheap netbooks.
2016-05-13nntp: fixup "Wide character" warnings
We need Perl to believe everything we send is UTF-8, make it so, even if it may not be. Fixes: 265e79ff82ce 'Revert "nntp: proper UTF-8 support (hopefully?)"'
2016-05-13Revert "nntp: proper UTF-8 support (hopefully?)"
This reverts commit f81ad477cb013d05b9b11fa051a9ebc5983a5be6. The raw, undecoded body is probably what should be sent over the wire anyways for clients to deal with. We'll need this to avoid deprecation warnings with Perl 5.24+ since we use send()/recv()/sysread().