about summary refs log tree commit homepage
path: root/lib/PublicInbox/Qspawn.pm
DateCommit message (Collapse)
2017-01-21qspawn: better annotate where $qx_cb is called
Hopefully this makes the code easier-to-follow for random readers. This requires a small amount of modification to our one caller, but this is a new, unstable API (as is nearly all of our code).
2017-01-18repobrowse: git snapshot waits for all commands asynchronously
This new asynchronous API, psgi_qx, will allow us to take advantage of non-blocking I/O from even small commands; as those may still need to wait for slow operations.
2017-01-17qspawn: better description
We'll probably use this in a lot of places...
2017-01-11async: improve and fix out-of-date comments
2017-01-11repobrowse: qspawn + streaming for git commit display
This prevents "git show" processes from monopolizing the system and allows us to better handle backpressure from gigantic commits.
2017-01-11qspawn: fix bad error reporting on errors
Oops :x
2017-01-08githttpbackend: use psgi_return shortcut
This drastically cuts down the amount of duplicate code we have in this branch.
2017-01-08Merge remote-tracking branch 'origin/master' into repobrowse
* origin/master: inbox: properly register cleanup timer for git processes search: remove subject_summary searchmsg: favor direct hash access over accessor methods remove incorrect comment about strftime + locales config: allow per-inbox nntpserver inbox: eliminate weaken usage entirely inbox: describe the full key name config: remove unused get() method config: always use namespaced "publicinboxlimiter" qspawn: prepare to support runtime reloading of Limiter http: remove weaken usage, reduce anonsub capture scope httpd/async: remove weaken usage http: fix spelling error watch: watchspam affects all configured inboxes doc: minor updates to design notes
2017-01-07qspawn: prepare to support runtime reloading of Limiter
We may allow the {max} value of a limiter to be changed in the future, so lets start accounting for it before we spawn followup processes.
2016-12-26repobrowse: port patch generation over to qspawn
And start generalizing the qspawn usage code for PSGI with psgi_return.
2016-07-09www: add configurable limiters
Currently only for git-http-backend use, this allows limiting the number of spawned processes per-inbox or by group, if there are multiple large inboxes amidst a sea of small ones. For example, a "big" repo limiter could be used for big inboxes: which would be shared between multiple repos: [limiter "big"] max = 4 [publicinbox "git"] address = git@vger.kernel.org mainrepo = /path/to/git.git ; shared limiter with giant: httpbackendmax = big [publicinbox "giant"] address = giant@project.org mainrepo = /path/to/giant.git ; shared limiter with git: httpbackendmax = big ; This is a tiny inbox, use the default limiter with 32 slots: [publicinbox "meta"] address = meta@public-inbox.org mainrepo = /path/to/meta.git
2016-07-09qspawn: allow configurable limiters
And bump the default limit to 32 so we match git-daemon behavior. This shall allow us to configure different levels of concurrency for different repositories and prevent clones of giant repos from stalling service to small repos.
2016-06-21spawn: improve error checking for fork failures
fork failures are unfortunately common when Xapian has gigabytes and gigabytes mmapped.
2016-05-24git-http-backend: use qspawn to limit running processes
Having an excessive amount of git-pack-objects processes is dangerous to the health of the server. Queue up process spawning for long-running responses and serve them sequentially, instead.