about summary refs log tree commit homepage
path: root/t
DateCommit message (Collapse)
2016-05-30git-http-backend: remove dependency on Plack::Request
Plack::Request is unnecessary overhead for this given the strictness of git-http-backend. Furthermore, having to make commit 311c2adc8c63 ("avoid Plack::Request parsing body") to avoid tempfiles should not have been necessary.
2016-05-29nntp: fix for missing articles/bodies/heads
Oops, we totally forgot to automate testing for this :x
2016-05-28remove redundant NewsGroup class
Most of its functionality is in the PublicInbox::Inbox class. While we're at it, we no longer auto-create newsgroup names based on the inbox name, since newsgroup names probably deserve some thought when it comes to hierarchy.
2016-05-28t/plack: ensure we can cascade on common endpoints
We don't serve things like robots.txt, favicon.ico, or .well-known/ endpoints ourselves, but ensure we can be used with Plack::App::Cascade for others.
2016-05-27config: fix NewsWWW fallback for newsgroups in HTTP URLs
Oops, added a test to prevent regressions while we're at it.
2016-05-25remove Email::Address dependency
git has stricter requirements for ident names (no '<>') which Email::Address allows. Even in 1.908, Email::Address also has an incomplete fix for CVE-2015-7686 with a DoS-able regexp for comments. Since we don't care for or need all the RFC compliance of Email::Address, avoiding it entirely may be preferable. Email::Address will still be installed as a requirement for Email::MIME, but it is only used by the Email::MIME::header_str_set which we do not use
2016-05-24git-http-backend: use qspawn to limit running processes
Having an excessive amount of git-pack-objects processes is dangerous to the health of the server. Queue up process spawning for long-running responses and serve them sequentially, instead.
2016-05-23http: chunk in the server, not middleware
Since PSGI does not require Transfer-Encoding: chunked or Content-Length, we cannot expect random apps we host to chunk their responses. Thus, to improve interoperability, chunk at the HTTP layer like other PSGI servers do. I'm chosing a more syscall-intensive method (via multiple send(...MSG_MORE) for now to reduce copy + packet overhead.
2016-05-23t/config.t: remove GIT_DIR usage in test
Followup-to: commit 24e0219f364ed402f9136227756e0f196dc651aa ("remove GIT_DIR env usage in favor of --git-dir")
2016-05-22t/spawn.t: additional tests for popen_rd
We need to ensure $? is set properly for users.
2016-05-21localize $/ in more places to avoid potential problems
This hopefully makes the intent of the code clearer, too. The the HTTP use of the numeric reference for getline caused problems in Git.pm, already.
2016-05-19www: tighten up allowable filenames for attachments
Having a file start with '.' or '-' can be confusing and for users, so do not allow it.
2016-05-19www: validate and check filenames in URLs
We shall ensure links continue working for this.
2016-05-19msg_iter: workaround broken Email::MIME versions
Email::MIME >= 1.923 and < 1.935 would drop too many newlines in attachments. This would lead to ugly text files without a proper trailing newline if using quoted-printable, 7bit, or 8bit. Attachments encoded with base64 were not affected. These versions of Email::MIME are widely available in Debian 8 (Jessie) and even Ubuntu LTS distros so we will need to support this workaround for a while.
2016-05-19switch read-only uses of walk_parts to msg_iter
msg_iter lets us know the index of the attachment, allow us to make more sensible labels and in a future commit, hyperlinks to download attachments.
2016-05-19msg_iter: new internal API for iterating through MIME
Unlike Email::MIME::walk_parts, this is non-recursive and gives depth + index offset information about the part for creating links for later retrieval It is intended for read-only access and changes are not propagated to the parent; however future versions of it may clobber bodies or the original version as it iterates to reduce memory overhead. It is intended for making it easy to locate attachments within a message in the WWW view.
2016-05-18nntpd: reject control characters entirely
There's no place for them in the commands and we don't take messages; potentially printing them into a log opened in a terminal is too dangerous. Hoist out read_til_dot in the test while we're at it.
2016-05-18tests: add check-www-inbox script
This can be useful for hammering a live HTTP server with requests to ensure it does not fall over under load.
2016-05-16www: fix for running under mount paths
We try to avoid issues like these by using relative URLs in hrefs, but we can't avoid the problem with Location: for redirects and Atom feeds which are likely to be rehosted elsewhere. We also reorder some of the code to work around a weird issue on the psgi-plack mailing list: <20160516073750.GA11931@dcvr.yhbt.net> (Somewhere on https://groups.google.com/group/psgi-plack but it's probably not bookmarkable)
2016-05-16declare Inbox object for reusability
From the beginning, we've avoided objects here in favor of faster startup time; but it may not be worth it since a persistent httpd/nntpd is faster and -mda isn't hit as often.
2016-05-14rename most instances of "list" to "inbox"
A public-inbox is NOT necessarily a mailing list, but it could serve as an input point for zero, one, or infinite mailing lists :D
2016-05-14t/nntpd: test for wide characters and UTF-8 mangling
We'll need to test non-UTF-8 messages at some point, too. There are lots of legacy-encoded messages in old archives and I would not bet we behave sanely w.r.t. those.
2016-05-14t/nntpd: avoid fork+exec for search indexing
The Xapian search index is required for the NNTP server, so there's no point in calling system() for it like we do in other tests. This should speed up the test a small amount.
2016-05-05t/view: note possibly invalid test...
Ugh, I really need to get off my ass to write automated tests for an Apache2 + mod_perl config.
2016-05-03git-http-backend: reduce memory use for clone/fetch
When serving large static files or large packs, we may call Danga::Socket::write directly to queue up callbacks to resume reading and defer firing them until the socket is writable. This prevents us from scheduling writes or buffering until we know the socket is writable and prevents needless buffering by Danga::Socket when faced with slow clients. For smart clones, this comes at the cost of throttling the output of "git pack-objects" to the speed of the client connection. This is probably not ideal, but is the behavior of the standard git-daemon, too; and is preferable to running the httpd out-of-memory. Buffering to the filesystem may be an option in the future...
2016-05-02t/*.t: reduce -mda calls
Process startup times are atrocious for fast tests and there's far too much setup involved. Rely on git-fast-import instead; but more work is needed in this area.
2016-05-02t/nntpd.t: stop hard coding message :bytes into test
It limits flexibility and makes it harder to switch to use PublicImport::Import.
2016-05-02nntp: append Archived-At and List-Archive headers
For readers using NNTP, we should do our best to advertise the clonable HTTP/HTTPS URLs and the message permalink URL for ease-of-referencing messages, since we don't want the NNTP server and it's sequential article numbers to be relied on.
2016-04-30daemon: graceful shutdown warning and limit removal
git clones may take longer than 30s, much longer... So prepare to wait almost indefinitely for sockets to timeout and document the second signal behavior for immediate shutdown. While we're at it, move parent death handling to a separate class to avoid Danga::Socket->AddOtherFds, since that does not allow proper handling the parent pipe being closed and would actually misterminate a worker prematurely. t/nntpd.t is update to illustrate the failure with workers enabled. We will work to keep memory usage low and let clients take their time without interrupting them.
2016-04-30searchmsg: ensure long subject lines are not broken
Noticed when using a long URL in the subject.
2016-04-29http: avoid corking on "Content-Length: 0" response
We must use a normal write instead of send(.., MSG_MORE) when writing responses of "Content-Length: 0" to avoid the corking effect MSG_MORE provides. We only want to cork headers if we will send a non-empty body. Fixes: c3eeaf664cf0 ("http: clarify intent for persistence") This needs a proper test.
2016-04-28githttpbackend: clamp to one smart HTTP request at-a-time
Server admins may not be able to afford to have too many git-pack-objects processes running at once. Since PSGI HTTP servers should already be configured to use multiple processes for other requests; limit concurrency of smart backends to one; and fall back to dumb responses if we're already generating a pack.
2016-04-28githttpbackend: fall back to dumb if smart HTTP is off
Using http.getanyfile still keeps the http-backend process alive, so it's better to break out of that process and handle serving entirely within the HTTP server.
2016-04-27import: document API for public consumption
This is probably trivial enough to be final?
2016-04-25t/feed.t: run properly without ssoma installed
While we're at it, update some references to ssoma in the Makefile.PL comment.
2016-04-25remove GIT_DIR env usage in favor of --git-dir
No need to maintain per-block environment state when we can localize it to per-command. We've had --git-dir= in git since 1.4.2 (2006-08-12) and already use it all over the place.
2016-04-25remove ssoma dependency
By converting to using ourt git-fast-import-based Import module. This should allow us to be more easily installed.
2016-04-15www: redirect /$MESSAGE_ID/f/ endpoints
Quote-folding was a major design mistake pre-1.0. Since this project is still in its infancy and unlikely to be in wide use at the moment, redirect the /f/ endpoints back to the plain message.
2016-04-14view: drop vestigial elements of quote folding
...And mark quotes as <span class="q"> since it barely costs us anything and allows users to choose colors themselves with custom, user-supplied CSS. Reduce allocations of the Linkify object, too.
2016-04-13www: stop generating /$MESSAGE_ID/f/ links
Quote-folding can be detrimental as it fails to hide the real problem of over-quoting. Over-quoting wastes bandwidth and space for all readers, not just WWW readers of the public-inbox. So hopefully removing quote-folding support from the WWW interface can shame those repliers into quoting only relevant portions of what they reply to.
2016-04-11import: initial module + test case
This will allow us to write fast importers for existing archives as well as eventually removing the ssoma dependency for performance and ease-of-installation.
2016-04-11git: add support for qx wrapper
This lets us one-line git commands easily like ``, but without having to remember --git-dir or escape arguments.
2016-04-09filter: preserve Mail-Followup-To and Mail-Reply-To
Allow users to do wacky things here if they really wish... It's bad practice, but at least allow other readers to mock users of these headers :P
2016-03-12http: use Plack::HTTPParser for HTTP parsing
This allows us to reduce installation dependencies while retaining performance as it favors HTTP::Parser::XS when it is installed and available. PLACK_HTTP_PARSER_PP may be set to 1 to force a pure Perl parser for testing.
2016-03-06http: reject excessively large HTTP request bodies
We cannot risk using all of a users' disk space buffering gigantic requests. Use the defaults git gives us since we primarily host git repositories.
2016-03-06http: reject excessive headers
HTTP::Parser::XS::PP does not reject excessively large headers like the XS version. Ensure we reject headers over 16K since public-inbox should never need such large request headers.
2016-03-05daemon: sockname detects listeners correctly
This means we can avoid false-positives when inheriting multiple Unix domain sockets.
2016-03-05daemon: document optional Net::Server dependency
Non-socket activation users will want to install Net::Server for daemonization, pid file writing, and user/group switching.
2016-03-05t/httpd-corner: avoid clobbering existing FDs after fork
Due to the deterministic way reference counting works, we do not want to drop references to existing FDs even if we no longer need the glob reference; the actual FD is all we can pass through on exec.
2016-03-05t/httpd-corner: additional callback test
Just to ensure we hit the code path independently of WWW code.