about summary refs log tree commit homepage
path: root/lib/PublicInbox/SolverGit.pm
DateCommit message (Collapse)
2020-12-05isearch: emulate per-inbox search with ->ALL
Using "eidx_key:" boolean prefix to limit results to a given inbox, we can use ->ALL to emulate and replace per-Inbox xap15/[0-9] search indices. With this change, the presence of "extindex.all.topdir" in the $PI_CONFIG will cause the WWW code to use that extindex and ignore per-inbox Xapian DBs in xap15/[0-9]. Unfortunately IMAP search still requires old per-inbox indices, for now. Mapping extindex Xapian docids to per-Inbox UIDs and vice-versa is proving tricky. Fortunately, IMAP search is rarely used and optional. The RFCs don't specify expensive phrase search, either, so `indexlevel=medium' can be used in per-inbox Xapian indices to save space. For primarily WWW (and future JMAP) users; this should result in significant disk space, FD, and page cache footprint savings for large instances with many inboxes and many cross-posted messages.
2020-09-12treewide: avoid `goto &NAME' for tail recursion
While Perl implements tail recursion via `goto' which allows avoiding warnings on deep recursion. It doesn't (as of 5.28) optimize the speed of such dispatches, though it may reduce ephemeral memory usage. Make the code less alien to hackers coming from other languages by using normal subroutine dispatch. It's actually slightly faster in micro benchmarks due to the complexity of `goto &NAME'.
2020-09-10solver: async blob retrieval for diff extraction
Like the rest of the WWW code, public-inbox-httpd now uses git_async_cat to retrieve blobs without blocking the event loop. This improves fairness when git blobs are on slow storage and allows us to take better advantage of SMP systems.
2020-09-10solver: break apart inbox blob retrieval
To avoid hogging the event loop in public-inbox-httpd when many candidate messages match, we'll separate the steps to ensure fairness on slow storage.
2020-09-10solver: check one git coderepo and inbox at a time
With public-inbox-httpd, this mitigates the effect of slow git blob storage with multiple coderepos configured for an inbox. It's still synchronous for now (and may need to remain that way for ->last_check_err), but no longer monopolizes the event loop when checking multiple coderepos. We don't yet support multi-inbox scanning, yet; but this also prepares us for a future where we do. We'll also support >=40 char blob OIDs in preparation for future git SHA-256 support, too.
2020-09-10solver: drop warnings, modernize use v5.10.1, use SEEK_SET
With Perl upstream preparing to deprecate things, we'll move towards only enabling warnings during development via shebang and stop enabling them via "use". We'll also favor "use v5.10.1" over the Perl 5.6-compatible "use 5.010_001", since our code base never worked on 5.6. Finally, were also importing SEEK_SET without using it, just use it for readability since we can't avoid loading Fcntl in other places and it'll get constant-folded, anyways.
2020-09-03search: replace ->query with ->mset
Nearly all of the search uses in the production code rely on a Xapian mset iterator being returned (instead of an array of $smsg objects). So default to returning the mset and move the burden of smsg array conversion into the test cases.
2020-07-10viewvcs: stop checking unused "B" query parameter
The resulting OID ("oid_b") is a required arg and part of $env->{PATH_INFO}, instead; so it's never part of an optional query parameter.
2020-06-03inbox: introduce smsg_eml method
The goal of this is to eventually remove the $smsg->{mime} field which is easy-to-misuse and cause memory explosions which necessitated fixes like commit 7d02b9e64455831d ("view: stop storing all MIME objects on large threads").
2020-05-09msg_iter: make ->each_part method for PublicInbox::MIME
The reliance on Email::MIME->subparts is a tad inefficient with a work-in-progress module to replace Email::MIME. So move towards using ->each_part as a class-specific iterator which can take advantage of more class-specific optimizations in the yet-to-be-revealed PublicInbox::Eml and PublicInbox::Gmime classes. The msg_iter() sub remains for compatibility with existing 3rd-party scripts/modules which use our small public Perl API and Email::MIME.
2020-03-22rename PublicInbox::SearchMsg => PublicInbox::Smsg
Since the introduction of over.sqlite3, SearchMsg is not tied to our search functionality in any way, so stop confusing ourselves and future hackers by just calling it "PublicInbox::Smsg". Add a missing "use" in ExtMsg while we're at it.
2020-03-20www: avoid `state' usage to perform allocations up-front
We want WWW->preload to get as many immortal allocations done as possible, and the `state' feature from Perl 5.10 prevents that.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2020-02-01solver: join multiple URLs with "||"
It seems to make sense to the target audience that any of the URLs displayed could work.
2020-01-25spelling: favor `publicly' over `publically'
While both can be correct, the former seems more common, is shorter, and is also consistent with the spelling found in the AGPL-3.0 text.
2020-01-13solver: path_a may be undef from /dev/null
This avoids uninitialized variable warnings when viewing newly-created files.
2020-01-12www: discard multipart parent on iteration
We're often iterating through messages while writing to another buffer in our WWW interface, causing memory usage to multiply. Since we know we won't need to keep the MIME object around in some cases, and can tell msg_iter to clobber the on-stack variable while it operates on subparts of multipart messages. With xt/mem-msgview.t switched to multipart from the previous commit, this shows a 13 MB memory reduction on that test.
2020-01-06treewide: "require" + "use" cleanup and docs
There's a bunch of leftover "require" and "use" statements we no longer need and can get rid of, along with some excessive imports via "use". IO::Handle usage isn't always obvious, so add comments describing why a package loads it. Along the same lines, document the tmpdir support as the reason we depend on File::Temp 0.19, even though every Perl 5.10.1+ user has it. While we're at it, favor "use" over "require", since it it gives us extra compile-time checking.
2020-01-04solver: allow literal '\r' character in diff lines
While filenames are escaped, the actual diff contents may contain an unescaped "\r" carriage return byte not in front of the "\n" line feed. So just allow "\r" to appear in the middle of a line.
2020-01-04solver: minor cleanups to diff extraction
Initialize the $di hashref at use to make it more obvious it's a local variable. We can also use the :utf8 IO layer via open+print to save ourselves the trouble of converting the UTF-8 patch to an octet stream.
2020-01-04solver: do not enforce order on extended headers
This is needed to work with patches with many renames, such as what makes "git/eebf7a8/s/?b=t%2Ftest-lib.sh"
2020-01-03qspawn: use per-call quiet flag for solver
solver can spawn multiple processes per HTTP request, but "git apply" failures are needlessly noisy due to corrupt patches. We also don't want to silence "git ls-files" or "git update-index" errors using $env->{'qspawn.quiet'}, either, so this granularity is needed. Admins can check for 500 errors in access logs to detect (and reproduce) solver failures, anyways, so there's no need to log every time "git apply" rejects a corrupt patch.
2020-01-03solver: extract_diff: deal with missing "diff --git" line
Rewrite the patch extraction loop using a single regexp which accounts for missing "diff --git ..." lines and is capable of extracting pathnames off the "+++ b/foo" line. This fixes the solving of blob "96f1c7f" off <2841d2de-32ad-eae8-6039-9251a40bb00e@tngtech.com> in git@vger archives. v2: * Fix regressions in git@vger archives: - git/776fa90f7f/s/?b=contrib/git-jump/git-jump (fallback to "old mode" properly) - git/5cd8845/s/?b=submodule.c (no leading space in context) * use "state" in a Perl <5.28.0-compatible way
2020-01-03solver: try the next patch on apply failures
Sometimes a patch is corrupted and resent to create the same OID. We need to account for that case and actually move onto the next patch instead of blindly trying "git ls-files" to get nothing out of it.
2019-12-30spawn: support chdir via -C option
This simplifies our admin module a bit and allows solver to be used with v1 inboxes using git versions prior to v1.8.5 (but still >= git v1.8.0).
2019-12-30spawn: allow passing GLOB handles for redirects
We can save callers the trouble of {-hold} and {-dev_null} refs as well as the trouble of calling fileno().
2019-12-28solvergit: allow passing arg to user-supplied callback
This allows us to get rid of the requirement to capture on-stack variables with an anonymous sub, as illustrated with the update to viewvcs to take advantage of this. v2: fix error handling for missing OIDs
2019-12-26msg_iter: provide means to stop using anonymous subs
And remove the last anonymous sub in SolverGit itself.
2019-12-26qspawn: remove some anonymous subs for psgi_qx
By passing a user-supplied arg to $qx_cb, we can eliminate the callers' need to capture on-stack variables with a closure. This saves several kilobytes of memory allocation at the expense of some extra hash table lookups in user-supplied callbacks. It also reduces the risk of memory leaks by eliminating a common source of circular references.
2019-12-26httpd/async: support passing arg to callbacks
Another step towards removing anonymous subs to eliminate a possible source of memory leaks and high memory use.
2019-11-24check for File::Temp 0.19 for ->newdir method
This is distributed with Perl 5.10.1 and onwards, so it should not be an installation burden for any users. I'm planning to move away from tempdir() entirely and use File::Temp->newdir to remove dependencies on END{} blocks.
2019-11-14solvergit: use --unidiff-zero with git-apply(1)
I sometimes post context-free documentation patches generated with "-U0" to reduce size and bandwidth overhead when replacing URLs or updating copyright notices. git-apply(1) needs the --unidiff-zero switch to work properly with context-free patches. Given our search looks for blob OIDs, and we're never going to be running the code we regenerate, "--unidiff-zero" ought to be safe.
2019-10-31solvergit: deal with false-positive dfpost: results
When solving for blob 81c1164ae5 in https://public-inbox.org/git/, at least two messages get indexed with the dfpost result for that blob (after fixing MsgIter to decode all text/* parts): 1. https://public-inbox.org/git/b9fb52b8-8168-6bf0-9a72-1e6c44a281a5@oracle.com/ 2. https://public-inbox.org/git/56664222-6c29-09dc-ef78-7b380b113c4a@oracle.com/ However, only the first message contains a usable patch. So we must adjust SolverGit to account for multiple messages hitting the same "dfpost:" search result and attempt "git apply" on all results, not just the first. In the future, changes to SearchIdx.pm may rid us of invalid search results and speed up performance (at the expense of developer/indexing time); but we need to account for old search indices, first.
2019-09-14tmpfile: give temporary files meaningful names
Although we always unlink temporary files, give them a meaningful name so that we can we can still make sense of the pre-unlink name when using lsof(8) or similar tools on Linux.
2019-09-12solvergit: don't drop update-index stdin with qspawn
It's possible for Qspawn callers to be deferred, in which case we must ensure we don't cause the temporary file used for stdin to become unref-ed and closed. This can be a problem when we exceed the default Qspawn limiter of 32 concurrent processes for "git update-index".
2019-07-10solver: remove redundant spawn imports
We're using Qspawn, now
2019-06-04solver|viewdiff: restrict digit matches to ASCII
git would not generate non-ASCII digits to describe hunk offsets, so don't waste more time than necessary to make sense of non-ASCII digit chars for line offsets.
2019-04-04support publicinbox.cgitrc directive
We can save admins the trouble of declaring [coderepo "..."] sections in the public-inbox config by parsing the cgitrc directly. Macro expansion (e.g. $HTTP_HOST) expansion is not supported, yet; but may be in the future.
2019-04-04viewvcs: preliminary support for showing non-blobs
Eventually, we'll have special displays for various git objects (commit, tree, tag). But for now, we'll just use git-show to spew whatever comes from git.
2019-02-05solvergit: include the $oid_want tmpdir name
This can help admins diagnose problems with SolverGit, since qspawn logs the failed "git apply" command-line in stderr. (or it can waste admins' time because sometimes there's crap mail clients which mangle patches)
2019-01-31solvergit: allow shorter-than-necessary OIDs from user
We can rely on git to disambiguate, here; because sometimes shorter OIDs can be unambiguous even if we only resolved the longer one.
2019-01-31solvergit: allow searching on longer-than-needed OIDs
public-inbox can only index the abbreviated object_ids in emails, not the full or even longer-than-necessary object_ids. So retry failed object_ids if they're longer than 7 hex characters.
2019-01-30solvergit: don't confuse Xapian with ".." in filenames
Xapian will interpret ".." as ranges, even quoted phrases. So break up words on ".." since punctuation (AFAIK) is not searchable, anyways.
2019-01-30git: use "git rev-parse --git-path"
Using git worktrees was causing t/solver_git.t to fail on me.
2019-01-30solvergit: deal with alternative diff prefixes
At least, without extra directory levels, since git-diff supports --src-prefix and --dst-prefix, and /git/6aa8857a11/s/ uses it...
2019-01-30solvergit: extract mode from diff headers properly
grep() won't set $1, so use "=~", instead.
2019-01-30solvergit: avoid "Wide character" warnings
Just quiet Perl down, since we don't know or care about the encoding of the patch we hand off to git-apply.
2019-01-30solvergit: do not show full path names to "git apply"
"git apply" will warn about whitespace with the full path of the patch, which will expose the $TMPDIR environment to users over HTTP(S). This change breaks compatibility with git pre-1.8.5, again; but that was released in late-2013; so hopefully everybody is on newer versions.
2019-01-29solvergit: do not solve blobs twice
In some cases, a file may ping-pong between blob IDs in the same message when reverts occur. So break out of this early. This doesn't account for different abbreviations, but the limited variations of abbreviations should alleviate the problem.
2019-01-27solver: crank up max patches to 9999
Might as well, since the only constraint is filesystem space for temporary files for public-inbox-httpd users. -httpd can fairly share work across clients with our use of psgi_qx; and there's a recent patch series in git@vger with 64 patches in sequence.