about summary refs log tree commit homepage
DateCommit message (Collapse)
2020-01-27viewdiff: add "b=" param with non-standard diff prefix
<20180228012207.GB251290@aiede.svl.corp.google.com> (posted to git@vger) uses "i" and "w" prefixes instead of the standard "a" and "b" prefixes, ensure we emit a "b=$FILENAME" param for the solver endpoint to improve search accuracy, syntax highlighting, and information density in the URL itself.
2020-01-27searchidx: don't assume "a/" and "b/" as prefixes
Some people use "--{src,dst}-prefix=", try to deal with those since git-apply can handle them when called by solver.
2020-01-27searchidx: skip filenames on "diff --git ..."
We already capture filenames on the lines beginning with "---" and "+++", so it's redundant work to capture filenames from "diff --git ..." lines.
2020-01-27linkify: move to_html over from ViewDiff
We use the same idiom in many places for doing two-step linkification and HTML escaping. Get rid of an outdated comment in flush_quote while we're at it.
2020-01-27linkify: compile $LINK_RE once
This gives a 3-4% performance improvement in xt/perf-msgview.t with a mirror of https://public-inbox.org/meta/
2020-01-27view: inline and eliminate msg_html
No need to keep the old sub around, anymore. Rename auxiliary subs to "msg_page_*" instead of the "html" version.
2020-01-27xt/perf-msgview: switch to multipart_text_as_html
It's a more widely-used (but still internal) API which will probably last longer than msg_html. It also reaches deeper into the stack and avoids the overhead of ->getline via PSGI, so it's faster and gives a more accurate measurement of lower-level parts.
2020-01-27tests: move the majority of t/view.t into t/plack.t
And some more into t/mid.t. PublicInbox::View::msg_html may change internally, so lets rely on the stable PSGI interface to test it, rather than a test which reaches deep into the internals.
2020-01-27init: use Import::run_die instead of system()
We already load PublicInbox::Import via PublicInbox::InboxWritable, so it's not an extra module to load. This can give us a slight speedup in tests.
2020-01-27t/plack.t: modernize and unindent
This test will be expanded, and we can take advantage of run_script to simplify our internal API use.
2020-01-27view: start performing buffering into {obuf}
Get rid of the confusingly named {rv} and {tip} fields and unify them into {obuf} for readability. {obuf} usage may be expanded to more areas in the future. This will eventually make it easier for us to experiment with alternative buffering schemes.
2020-01-27wwwstream: discard single-use $ctx fields after use
This should make it clear that we only use these elements once and can discard them. While we're in the area, avoid escaping '"' by using qq() instead of "" to quote strings requiring interpolation.
2020-01-27view: simplify duplicate Message-ID handling
It's an uncommon code path, no need to make it more complex than it needs to be by having extra sub parameters.
2020-01-27view: thread_skel: drop constant tpfx parameter
It hasn't changed in a few years. Now we can rely on constant folding to avoid extraneous ops to the $skel buffer.
2020-01-27view: reduce parameters for html_footer
Put more logic into html_footer and less in its only caller so we can control the buffering and string creation.
2020-01-27searchview: keep $noop sub private to the package
It'll always be used as a callback, so there's no point in giving it a name to be called non-anonymously. Making assigments to it is slightly faster since there's no need to repeatedly do a lookup by name.
2020-01-27view: improve readability around walk_thread
Pass \&coderefs explicitly to walk_thread, and add some prototypes + comments to describe what goes on.
2020-01-27www: use "skel" terminology consistently
This saves us a few comments and confusion. Yes, it's a destination so "dst" can be appropriate, but we may be using that term elsewhere.
2020-01-27www*stream: favor \&close instead of *close
Be explicit that we're making a code reference, and not a reference to a scalar, array, hash, or IO...
2020-01-27xapcmd: increase scope of lock
The old lock scope was only sufficient for protecting against concurrent modifications from the common -mda, -watch, or -learn writers. It was not sufficient for protecting against parallel -compact or -xcpdb invocations from eager admins. Most of the time this only leads to confusing and misleading warning messages, but parallel xcpdb --reshard could lead to errors.
2020-01-27search: {version} => {ibx_ver}
We don't confuse human readers with the Xapian schema version. We also want to make it obvious this is the version of the inbox we're indexing, these are Search or SearchIdx objects, not Inbox objects.
2020-01-27inbox: add ->version method
This allows us to simplify version checking by avoiding "//" or "||" operators sprinkled around.
2020-01-27switch to sysseek + sysread for serving static files
The "perlio" layer doesn't do read(2) syscalls over 8192 bytes at the moment, and binmode($fh, ':unix') leaks[1]. So use sysseek and sysread for now, since I can't see retaining compatibility with PerlIO::scalar being worth the trouble. [1] http://nntp.perl.org/group/perl.perl5.porters/256918
gmane still has a NNTP server, so update links to point to it. cf. https://lars.ingebrigtsen.no/2020/01/06/whatever-happened-to-news-gmane-org/
2020-01-25wwwstatic: wire up buffer bypass for -httpd
This prevents public-inbox-httpd from buffering ->getline results from a static file into another temporary file when writing to slow clients. Instead we inject the static file ref with offsets and length directly into the {wbuf} queue. It took me a while to decide to go this route, some rejected ideas: 1. Using Plack::Util::set_io_path and having PublicInbox::HTTP serve the result directly. This is compatible with what some other PSGI servers do using sendfile. However, neither Starman or Twiggy currently use sendfile for partial responses. 2. Parsing the Content-Range response header for offsets and lengths to use with set_io_path for partial responses. These rejected ideas required increasing the complexity of HTTP response writing in PublicInbox::HTTP in the common, non-static file cases. Instead, we made minor changes to the colder write buffering path of PublicInbox::DS and leave the hot paths untouched. We still support generic PSGI servers via ->getline. However, since we don't know the characteristics of other PSGI servers, we no longer do a 64K initial read in an attempt to negotiate a larger TCP window.
2020-01-25ds: tmpio: store offsets per-buffer
We want to be able to inject existing file handles + offsets and even lengths into this in the future, without going through the ->getline interface[1] We also switch to using a 64K buffer size since we can safely discard whatever got truncated on write and full writes can help negotiate a larger TCP window for high-latency, high-bandwidth links. While we're at it, make it obvious that we're using O_APPEND for our tmpfile() interface so we can seek freely for reading while the writer always prints to the end of the file. [1] the getline interface for serving static files may result in us buffering on-FS data into another temporary file, which is a waste.
2020-01-25wwwstatic: offload error handling to PSGI server
The PSGI server needs to account for ->getline failing due to disk failures or truncated files, anyways. So just die() ourselves and let the PSGI server log and drop the client.
2020-01-25http: eliminate short-lived cyclic ref for psgix.io
While there is no known actual leak due to reference cycles, here, eliminating a potential source of leaks is helpful.
2020-01-25spelling: favor `publicly' over `publically'
While both can be correct, the former seems more common, is shorter, and is also consistent with the spelling found in the AGPL-3.0 text.
2020-01-25website: omit technical/ and other subdirs
We don't need to clutter the website with unnecessary technical information. Anybody who reads the technical/ directory should be looking at our source code, anyways; and we also have cgit and gitweb mirrors.
2020-01-25doc: INSTALL describe required deps for released versions
1.3.0 isn't out, yet, and sometimes folks will rely on INSTALL on our website while installing older versions, so try to clarify that.
2020-01-25website: re-add top-level files
I noticed the TODO was out-of-date on the website, among some other things. This was broken in moving GNU-isms in the Makefile to Perl.
2020-01-25doc: avoid needless rebuilds of NEWS
Repeatedly rebuilding `NEWS' because the mtime of `NEWS' is synched to the latest release .eml is a bit annoying, but necessary to save bandwidth for the website. So we'll also update the mtime of the source .eml file when reading them. It's kinda gross to be setting mtimes of source .eml files in Documentation/RelNotes/, but I can't think of anything better at the moment...
2020-01-25mbox: handle empty subjects after dropping "Re:" prefix
We can't pass empty strings to `to_filename' without triggering warnings, and `to_filename' on an empty string makes no sense.
2020-01-24contentid: ignore duplicate References: headers
OverIdx::parse_references already skips duplicate References (which we use in SearchThread for rendering). So there's no reason for our content deduplication logic to care if a Message-Id in the Reference header is mentioned twice.
2020-01-24wwwstream: shorten cloneurl uniquification
Another place where List::Scalar::uniq doesn't make sense, but there's a small op reduction to be had anyways.
2020-01-24mid: shorten uniq_mids logic
We won't be able to use List::Util::uniq here, but we can still shorten our logic and make it more consistent with the rest of our code which does similar things.
2020-01-24inbox: simplify filtering for duplicate NNTP URLs
And add a note to remind ourselves to use List::Util::uniq when it becomes common.
2020-01-24nntp: simplify setting X-Alt-Message-ID
We can cut down on the number of operations required using "grep" instead of "foreach".
2020-01-24contentid: use map to generate %seen for Message-Ids
This use of map {} is a common idiom as we no longer consider the Message-ID as part of the digest.
2020-01-23hval: from_attr: move to unit test
We don't call from_attr anywhere outside of tests, so don't bloat normal processes with it.
2020-01-23hval: to_attr: support wide characters
We need to escape wide characters when making attribute names from filename-looking things in diffstats.
2020-01-19doc: some 1.3.0 release notes updates
2020-01-13sigfd: simplify loop and improve documentation
We can use the return value of sysread to bound our loop instead of repeatedly shortening the string. Furthermore add some comments which can be easily checked against the signalfd(2) manpage.
2020-01-13ds: flatten $EXPMAP, delete entries on close
We can reduce the amount of small arrayrefs in memory by flattening $EXPMAP. This forces us to properly clean up references during deferred close handling, so NNTP (and soon HTTP) connections no longer linger until expiry.
2020-01-13ds: rely on autovivification for $later_queue
No reason to have an empty arrayref lying around when not everybody needs it. Re-indent the later-related subs since we're changing a bunch of lines, anyways.
2020-01-13ds: rely on autovivication for waitpid bits
No need to create an arrayref until we need it, and fix up a comment while we're in the area. Some aesthetic changes while we're at it: - Rename $WaitPids to $wait_pids to make it clear this is unique to our implementation and not in Danga::Socket. - rewrite dwaitpid() to reduce indentation level
2020-01-13ds: rely on autovivification for nextq
Another place we can delay creating arrays until needed.
2020-01-13ds|http|nntp: simplify {wbuf} population
We can rely on autovification to turn `undef' value of {wbuf} into an arrayref. Furthermore, "push" returns the (new) size of the array since at least Perl 5.0 (I didn't look further back), so we can use that return value instead of calling "scalar" again.
2020-01-13ds: guard ToClose against DESTROY side-effects
This does not affect our current code, but theoretically a DESTROY callback could call PublicInbox::DS::close to enqueue elements into the ToClose array. So take a similar strategy as we do with other queues (e.g. $nextq) by swapping references to arrays, rather than operating on the array itself. Since close operations are relatively rare, we can rely on auto-vivification via "push" ops to create the array on an as-needed basis. Since we're in the area, clean up the PostLoopCallback invocation to use the ternary operator rather than a confusing (to me) combination of statements. Finally, add a prototype to strengthen compile-time checking, and move it in front of our only caller to make use of the prototype.