about summary refs log tree commit homepage
DateCommit message (Collapse)
2020-01-28v2writable: newest epochs go first in alternates
New epochs are the most likely to have loose objects. git won't be able to take advantage of pack indices and needs to scan every alternate for the loose object via open/openat syscalls. Those syscalls will add up some day when we've got hundreds or thousands of epochs.
2020-01-28INSTALL: fix Linux::Inotify2 package name
The "2" is important, since "Linux::Inotify" without the "2" is not available from Debian 9/10 or CentOS 7.x and seems unmaintained.
2020-01-28t/v2reindex.t: 5.10.1 glob compatibility
I'm not sure when `for (<"quoted string/glob/*">)' became supported, and maybe it was inadvertant, but it fails with Perl 5.10.1. Just use the glob() function to be explicit.
2020-01-28t/hl_mod: document IO::Handle for autoflush
We don't need IO::File for this test, but IO::Handle is needed for ->autoflush with Perl <5.14. Note: I haven't tested highlight.pm under 5.10.1 since it's a weird dependency which isn't easy to install w/o distro support.
2020-01-28avoid relying on IO::Handle/IO::File autoload
Perl 5.14+ gained the ability to autoload IO::File (and IO::Handle) on missing methods, so relying on this breaks under 5.10.1. There's no reason to load IO::File or IO::Handle when built-in perlops work fine and are even a hair faster.
2020-01-28daemon: provide TCP_DEFER_ACCEPT for Perl <5.14
Socket::TCP_DEFER_ACCEPT() did not appear in the Socket module distributed with Perl until 5.14, despite it being available since Linux 2.4.
2020-01-27viewdiff: rewrite and simplify
Instead of going line-by-line, use split() with a giant regexp to capture groups of contiguous lines. This offloads state management to the regexp itself and makes it FAR easier to keep track of <span> and </span> pairings. Performance seems roughly on par after this change for the meta@public-inbox archives. It seems a tiny bit faster for git@vger with xt/perf-msgview.t, likely due to the longer messages and larger contiguous groups of lines having the same prefix (or no prefix at all) and drastically reduces the number of subroutine calls and Perl ops executed.
2020-01-27viewdiff: use autovivification for long_path hash
No sense in wasting code to do something the interpreter already does for us.
2020-01-27viewdiff: add "b=" param when missing "diff --git" line
<2841d2de-32ad-eae8-6039-9251a40bb00e@tngtech.com> as posted to git@vger contained an otherwise valid diff without a "diff --git" line. Generate a "b=" parameter in that case using the "+++" line instead of the "diff --git" line. SearchIdx.pm no longer uses the "diff --git" line for filename information, either.
2020-01-27viewdiff: add "b=" param with non-standard diff prefix
<20180228012207.GB251290@aiede.svl.corp.google.com> (posted to git@vger) uses "i" and "w" prefixes instead of the standard "a" and "b" prefixes, ensure we emit a "b=$FILENAME" param for the solver endpoint to improve search accuracy, syntax highlighting, and information density in the URL itself.
2020-01-27searchidx: don't assume "a/" and "b/" as prefixes
Some people use "--{src,dst}-prefix=", try to deal with those since git-apply can handle them when called by solver.
2020-01-27searchidx: skip filenames on "diff --git ..."
We already capture filenames on the lines beginning with "---" and "+++", so it's redundant work to capture filenames from "diff --git ..." lines.
2020-01-27linkify: move to_html over from ViewDiff
We use the same idiom in many places for doing two-step linkification and HTML escaping. Get rid of an outdated comment in flush_quote while we're at it.
2020-01-27linkify: compile $LINK_RE once
This gives a 3-4% performance improvement in xt/perf-msgview.t with a mirror of https://public-inbox.org/meta/
2020-01-27view: inline and eliminate msg_html
No need to keep the old sub around, anymore. Rename auxiliary subs to "msg_page_*" instead of the "html" version.
2020-01-27xt/perf-msgview: switch to multipart_text_as_html
It's a more widely-used (but still internal) API which will probably last longer than msg_html. It also reaches deeper into the stack and avoids the overhead of ->getline via PSGI, so it's faster and gives a more accurate measurement of lower-level parts.
2020-01-27tests: move the majority of t/view.t into t/plack.t
And some more into t/mid.t. PublicInbox::View::msg_html may change internally, so lets rely on the stable PSGI interface to test it, rather than a test which reaches deep into the internals.
2020-01-27init: use Import::run_die instead of system()
We already load PublicInbox::Import via PublicInbox::InboxWritable, so it's not an extra module to load. This can give us a slight speedup in tests.
2020-01-27t/plack.t: modernize and unindent
This test will be expanded, and we can take advantage of run_script to simplify our internal API use.
2020-01-27view: start performing buffering into {obuf}
Get rid of the confusingly named {rv} and {tip} fields and unify them into {obuf} for readability. {obuf} usage may be expanded to more areas in the future. This will eventually make it easier for us to experiment with alternative buffering schemes.
2020-01-27wwwstream: discard single-use $ctx fields after use
This should make it clear that we only use these elements once and can discard them. While we're in the area, avoid escaping '"' by using qq() instead of "" to quote strings requiring interpolation.
2020-01-27view: simplify duplicate Message-ID handling
It's an uncommon code path, no need to make it more complex than it needs to be by having extra sub parameters.
2020-01-27view: thread_skel: drop constant tpfx parameter
It hasn't changed in a few years. Now we can rely on constant folding to avoid extraneous ops to the $skel buffer.
2020-01-27view: reduce parameters for html_footer
Put more logic into html_footer and less in its only caller so we can control the buffering and string creation.
2020-01-27searchview: keep $noop sub private to the package
It'll always be used as a callback, so there's no point in giving it a name to be called non-anonymously. Making assigments to it is slightly faster since there's no need to repeatedly do a lookup by name.
2020-01-27view: improve readability around walk_thread
Pass \&coderefs explicitly to walk_thread, and add some prototypes + comments to describe what goes on.
2020-01-27www: use "skel" terminology consistently
This saves us a few comments and confusion. Yes, it's a destination so "dst" can be appropriate, but we may be using that term elsewhere.
2020-01-27www*stream: favor \&close instead of *close
Be explicit that we're making a code reference, and not a reference to a scalar, array, hash, or IO...
2020-01-27xapcmd: increase scope of lock
The old lock scope was only sufficient for protecting against concurrent modifications from the common -mda, -watch, or -learn writers. It was not sufficient for protecting against parallel -compact or -xcpdb invocations from eager admins. Most of the time this only leads to confusing and misleading warning messages, but parallel xcpdb --reshard could lead to errors.
2020-01-27search: {version} => {ibx_ver}
We don't confuse human readers with the Xapian schema version. We also want to make it obvious this is the version of the inbox we're indexing, these are Search or SearchIdx objects, not Inbox objects.
2020-01-27inbox: add ->version method
This allows us to simplify version checking by avoiding "//" or "||" operators sprinkled around.
2020-01-27switch to sysseek + sysread for serving static files
The "perlio" layer doesn't do read(2) syscalls over 8192 bytes at the moment, and binmode($fh, ':unix') leaks[1]. So use sysseek and sysread for now, since I can't see retaining compatibility with PerlIO::scalar being worth the trouble. [1] http://nntp.perl.org/group/perl.perl5.porters/256918
2020-01-25s/news.gmane.org/news.gmane.io/
gmane still has a NNTP server, so update links to point to it. cf. https://lars.ingebrigtsen.no/2020/01/06/whatever-happened-to-news-gmane-org/
2020-01-25wwwstatic: wire up buffer bypass for -httpd
This prevents public-inbox-httpd from buffering ->getline results from a static file into another temporary file when writing to slow clients. Instead we inject the static file ref with offsets and length directly into the {wbuf} queue. It took me a while to decide to go this route, some rejected ideas: 1. Using Plack::Util::set_io_path and having PublicInbox::HTTP serve the result directly. This is compatible with what some other PSGI servers do using sendfile. However, neither Starman or Twiggy currently use sendfile for partial responses. 2. Parsing the Content-Range response header for offsets and lengths to use with set_io_path for partial responses. These rejected ideas required increasing the complexity of HTTP response writing in PublicInbox::HTTP in the common, non-static file cases. Instead, we made minor changes to the colder write buffering path of PublicInbox::DS and leave the hot paths untouched. We still support generic PSGI servers via ->getline. However, since we don't know the characteristics of other PSGI servers, we no longer do a 64K initial read in an attempt to negotiate a larger TCP window.
2020-01-25ds: tmpio: store offsets per-buffer
We want to be able to inject existing file handles + offsets and even lengths into this in the future, without going through the ->getline interface[1] We also switch to using a 64K buffer size since we can safely discard whatever got truncated on write and full writes can help negotiate a larger TCP window for high-latency, high-bandwidth links. While we're at it, make it obvious that we're using O_APPEND for our tmpfile() interface so we can seek freely for reading while the writer always prints to the end of the file. [1] the getline interface for serving static files may result in us buffering on-FS data into another temporary file, which is a waste.
2020-01-25wwwstatic: offload error handling to PSGI server
The PSGI server needs to account for ->getline failing due to disk failures or truncated files, anyways. So just die() ourselves and let the PSGI server log and drop the client.
2020-01-25http: eliminate short-lived cyclic ref for psgix.io
While there is no known actual leak due to reference cycles, here, eliminating a potential source of leaks is helpful.
2020-01-25spelling: favor `publicly' over `publically'
While both can be correct, the former seems more common, is shorter, and is also consistent with the spelling found in the AGPL-3.0 text.
2020-01-25website: omit technical/ and other subdirs
We don't need to clutter the website with unnecessary technical information. Anybody who reads the technical/ directory should be looking at our source code, anyways; and we also have cgit and gitweb mirrors.
2020-01-25doc: INSTALL describe required deps for released versions
1.3.0 isn't out, yet, and sometimes folks will rely on INSTALL on our website while installing older versions, so try to clarify that.
2020-01-25website: re-add top-level files
I noticed the TODO was out-of-date on the website, among some other things. This was broken in moving GNU-isms in the Makefile to Perl.
2020-01-25doc: avoid needless rebuilds of NEWS
Repeatedly rebuilding `NEWS' because the mtime of `NEWS' is synched to the latest release .eml is a bit annoying, but necessary to save bandwidth for the website. So we'll also update the mtime of the source .eml file when reading them. It's kinda gross to be setting mtimes of source .eml files in Documentation/RelNotes/, but I can't think of anything better at the moment...
2020-01-25mbox: handle empty subjects after dropping "Re:" prefix
We can't pass empty strings to `to_filename' without triggering warnings, and `to_filename' on an empty string makes no sense.
2020-01-24contentid: ignore duplicate References: headers
OverIdx::parse_references already skips duplicate References (which we use in SearchThread for rendering). So there's no reason for our content deduplication logic to care if a Message-Id in the Reference header is mentioned twice.
2020-01-24wwwstream: shorten cloneurl uniquification
Another place where List::Scalar::uniq doesn't make sense, but there's a small op reduction to be had anyways.
2020-01-24mid: shorten uniq_mids logic
We won't be able to use List::Util::uniq here, but we can still shorten our logic and make it more consistent with the rest of our code which does similar things.
2020-01-24inbox: simplify filtering for duplicate NNTP URLs
And add a note to remind ourselves to use List::Util::uniq when it becomes common.
2020-01-24nntp: simplify setting X-Alt-Message-ID
We can cut down on the number of operations required using "grep" instead of "foreach".
2020-01-24contentid: use map to generate %seen for Message-Ids
This use of map {} is a common idiom as we no longer consider the Message-ID as part of the digest.
2020-01-23hval: from_attr: move to unit test
We don't call from_attr anywhere outside of tests, so don't bloat normal processes with it.