about summary refs log tree commit homepage
path: root/t
DateCommit message (Collapse)
2016-07-09www: add configurable limiters
Currently only for git-http-backend use, this allows limiting the number of spawned processes per-inbox or by group, if there are multiple large inboxes amidst a sea of small ones. For example, a "big" repo limiter could be used for big inboxes: which would be shared between multiple repos: [limiter "big"] max = 4 [publicinbox "git"] address = git@vger.kernel.org mainrepo = /path/to/git.git ; shared limiter with giant: httpbackendmax = big [publicinbox "giant"] address = giant@project.org mainrepo = /path/to/giant.git ; shared limiter with git: httpbackendmax = big ; This is a tiny inbox, use the default limiter with 32 slots: [publicinbox "meta"] address = meta@public-inbox.org mainrepo = /path/to/meta.git
2016-07-09qspawn: allow configurable limiters
And bump the default limit to 32 so we match git-daemon behavior. This shall allow us to configure different levels of concurrency for different repositories and prevent clones of giant repos from stalling service to small repos.
2016-07-07www: remove old footer generation code and normalize new.html
We now generate all of our HTML using WwwStream which forces us to have consistent headers and footers in the HTML itself. This also makes the search-capable vs search-less installs go to the new.html endpoint to maintain consistency (in case an admin decides to enable Xapian).
2016-07-07t/git-http-backend: check BSD::Resource availability
We should not fail tests when this is not available.
2016-07-06address: attempt to handle comments somewhat
They're uncommon, fortunately, but we make no attempt to handle nested comments (which would open us up to things like CVE-2015-7686) or use the comment in place of a missing name.
2016-07-02nntp: respect 3 minute idle time for shutdown
This avoids breaking clients on graceful shutdown since NNTP responses should usually be quick.
2016-07-02www: remove Plack::Request dependency entirely
Lighter and ever-so-slightly faster! Most importantly, this won't do non-obvious stuff behind our backs like trying to parse a POST request body for a query string param.
2016-07-02inbox: base_url method takes PSGI env hashref instead
This is lighter and we can work further towards eliminating our Plack::Request dependency entirely.
2016-07-01address: filter out domain from address-as-name idents
It seems common for address entries to end up as: "foo@example" <foo@example> Avoid needlessly displaying the domain name in that case.
2016-07-01t/watch_maildir: quiet down spam check warning
Probably better than bloating our own API with configurable warning streams and such...
2016-06-30www_stream: add response wrapper sub
This encapsulates an entire PSGI response array, hopefully making it easier to generate responses and avoid typos when setting the Content-Type.
2016-06-25view: safer and optional quoting for --in-reply-to arg
Angle brackets around the --in-reply-to= arg for git send-email has been optional since git v1.5.3.2, so strip them and make the command-line argument easier-to-type.
2016-06-25address: remove Address::from_name
Address::names is sufficient to handle what from_name did.
2016-06-25address: beef up the module with name list extaction
We may remove from_name in the future. ...And disallow quotes in email addresses. Technically I believe they're allowed, but they're definitely uncommon and unlikely to show up in legitimate mail.
2016-06-24watch_maildir: implement optional spam checking
Mailing lists I watch and mirror may not have the best spam filtering, and an extra layer should not hurt.
2016-06-24document Filesys::Notify::Simple dependency
And improve documentation for existing dependencies, too.
2016-06-24split out spamcheck/spamc to its own module.
This should hopefully make it easier to try other anti-spam systems (or none at all) in the future.
2016-06-20inbox: move field population logic to initializer
Inboxes are normally created by Config, but having the population logic in Inbox should make it easier to mock for testing.
2016-06-20feed: various object-orientation cleanups
Favor Inbox objects as our primary source of truth to simplify our code. This increases our coupling with PSGI to make it easier to write tests in the future. A lot of this code was originally designed to be usable standalone without PSGI or CGI at all; but that might increase development effort.
2016-06-19watch_maildir: tighten up path checks
Only mark seen messages as spam, otherwise it could be too aggressive and cause problems or over training. We wouldn't want a wayward FIFO ruining our day, either :)
2016-06-19watch_maildir: spam removal support
We can support spam removal by watching a special "spam" Maildir, too. We can run public-inbox-learn as a separate step, and that command will be improved to support auto-learning, too.
2016-06-18watch_maildir: add scan test
This should be portable despite the intended use of this directory being non-portable.
2016-06-18spawn: try to keep signals blocked in spawned child
While we only want to stop our daemons and gracefully destroy subprocesses, it is common for 'Ctrl-C' from a terminal to kill the entire pgroup. Killing an entire pgroup nukes subprocesses like git-upload-pack breaks graceful shutdown on long clones. Make a best effort to ensure git-upload-pack processes are not broken when somebody signals an entire process group. Followup-to: commit 37bf2db81bbbe114d7fc5a00e30d3d5a6fa74de5 ("doc: systemd examples should only kill one process")
2016-06-18view: introduce WwwStream interface
This will allow us to commonalize HTML generation in the future and is the start of moving existing HTML generation to a "pull" streaming model (from the existing "push" one). Using the getline/close pull model is superior to the existing $fh->write streaming as it allows us to throttle response generation based on backpressure from slow clients.
2016-06-17remove dependency on IPC::Run
We no longer depend on it for the core code, and tests are optional for users. Hopefully this makes this easier-to-install.
2016-06-15inbox: allow undef return value for base_url
It should be possible to serve the contents of a public-inbox over NNTP but not HTTP.
2016-06-15mda: hook up new filter functionality
This removes the Email::Filter dependency as well as the signature-breaking scrubber code. We now prefer to reject unacceptable messages and grudgingly (and blindly) mirror messages we're not the primary endpoint for.
2016-06-15emergency: implement new emergency Maildir delivery
This is transactional and hopefully safer in case we hit SIGSEGV or SIGKILL during processing, as the tmp/ copy will remain on the FS even if DESTROY/END handlers are not called.
2016-06-15filter: begin work on a new filter API
This filter API should be independent of Email::Filter and hopefully less intrusive to long running processes.
2016-06-15mda: precheck no longer depends on Email::Filter
Email::Filter doesn't offer any functionality we need, here; and our dependency on Email::Filter will gradually be removed since it (and Email::LocalDelivery) seem abandoned and we can have more-fine-grained control by rolling our own Maildir delivery which can work transactionally.
2016-06-15t/mda: use only Maildir for testing
Remove mbox tests since mbox is unreliable due to raciness and incompatible implementations. We will drop support for mbox emergency destinations, soon.
2016-06-15t/mda.t: remove senseless use of Email::Filter
Totally unnecessary...
2016-06-15t/feed.t: make IPC::Run usage optional
Since ssoma is optional, here, IPC::Run shall also be optional. (And it may be removed entirely in the future).
2016-06-14nntp: do not double-encode UTF-8 body
Or whatever the appropriate Perl terminology, is... And we will need to do something appropriate for other encodings, too. I still barely understand Perl Unicode despite attempting to understand the docs over the years..
2016-06-13view: msg_html uses getline body to reduce latency
We need to ensure we show the message body ASAP since the thread generation via Xapian could take a while and maybe even raise an exception or crash.
2016-06-07view: escape From name properly for title
Oops :x Add an additional test for live data for any unprintable characters, too, since this could be a dangerous source of HTML injection.
2016-06-05view: inline message reply into message view
This should reduce link following for replies and improve visibility. This should also reduce cache overhead/footprint for crawlers.
2016-05-30git-http-backend: remove dependency on Plack::Request
Plack::Request is unnecessary overhead for this given the strictness of git-http-backend. Furthermore, having to make commit 311c2adc8c63 ("avoid Plack::Request parsing body") to avoid tempfiles should not have been necessary.
2016-05-29nntp: fix for missing articles/bodies/heads
Oops, we totally forgot to automate testing for this :x
2016-05-28remove redundant NewsGroup class
Most of its functionality is in the PublicInbox::Inbox class. While we're at it, we no longer auto-create newsgroup names based on the inbox name, since newsgroup names probably deserve some thought when it comes to hierarchy.
2016-05-28t/plack: ensure we can cascade on common endpoints
We don't serve things like robots.txt, favicon.ico, or .well-known/ endpoints ourselves, but ensure we can be used with Plack::App::Cascade for others.
2016-05-27config: fix NewsWWW fallback for newsgroups in HTTP URLs
Oops, added a test to prevent regressions while we're at it.
2016-05-25remove Email::Address dependency
git has stricter requirements for ident names (no '<>') which Email::Address allows. Even in 1.908, Email::Address also has an incomplete fix for CVE-2015-7686 with a DoS-able regexp for comments. Since we don't care for or need all the RFC compliance of Email::Address, avoiding it entirely may be preferable. Email::Address will still be installed as a requirement for Email::MIME, but it is only used by the Email::MIME::header_str_set which we do not use
2016-05-24git-http-backend: use qspawn to limit running processes
Having an excessive amount of git-pack-objects processes is dangerous to the health of the server. Queue up process spawning for long-running responses and serve them sequentially, instead.
2016-05-23http: chunk in the server, not middleware
Since PSGI does not require Transfer-Encoding: chunked or Content-Length, we cannot expect random apps we host to chunk their responses. Thus, to improve interoperability, chunk at the HTTP layer like other PSGI servers do. I'm chosing a more syscall-intensive method (via multiple send(...MSG_MORE) for now to reduce copy + packet overhead.
2016-05-23t/config.t: remove GIT_DIR usage in test
Followup-to: commit 24e0219f364ed402f9136227756e0f196dc651aa ("remove GIT_DIR env usage in favor of --git-dir")
2016-05-22t/spawn.t: additional tests for popen_rd
We need to ensure $? is set properly for users.
2016-05-21localize $/ in more places to avoid potential problems
This hopefully makes the intent of the code clearer, too. The the HTTP use of the numeric reference for getline caused problems in Git.pm, already.
2016-05-19www: tighten up allowable filenames for attachments
Having a file start with '.' or '-' can be confusing and for users, so do not allow it.
2016-05-19www: validate and check filenames in URLs
We shall ensure links continue working for this.