about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2016-12-10search: retry document loading from Xapian
In addition to needing to retry enquire queries, we also need to protect document loading from the Xapian DB and retry on modification, as it seems to throw the same errors. Checking the $@ ref for Search::Xapian::DatabaseModifiedError is actually in the test suite for both the XS and SWIG Xapian bindings, so we should be good as far as forward/backwards compatibility.
2016-12-10search: always sort thread results in ascending time order
This makes life easier for the threading algorithm, as we can use the implied ordering of timestamps to avoid temporary ghosts and resulting container vivication. This would've also allowed us to hide the bug (in most cases) fixed by the patch titled "thread: last Reference always wins", in case that needs to be reverted due to infinite looping.
2016-12-10thread: last Reference always wins
Since we use SearchMsg from Xapian data, we can be assured we do not get self-referential {references} field. However, we may need to be more careful when checking has_descendent for loops, as blindly calling add_child could open us up to that possibility...
2016-12-10view: skip ghosts with no direct children
Otherwise, a malicious or broken client could populate the thread skeleton with invalid References. We only care about ghosts which messages correctly refer to, not totally bogus ones which may be the result of long line or token truncation + wrapping in MUA headers.
2016-12-10view: reduce indentation for skeleton generation
This should reduce the number of subroutine calls needed for the common case of real (non-ghost) messages as well as shortening code.
2016-12-10thread: fix comment describing its existence
Mail::Thread is UNavailable on many distros, meaning ordinary users will have to rely on CPAN, a Perl-specific packaging tool.
2016-12-10view: favor SearchMsg for In-Reply-To over Email::MIME
This should avoid warnings during thread skeleton generation if ever the Xapian database disagrees with View.pm about which is the proper direct parent of a message. We will treat the data in Xapian as the truth (if Xapian is available).
2016-12-10search: favor In-Reply-To over last References iff IRT exists
Some email clients set the References headers backwards, so trust the In-Reply-To header if (and only if) it exists and is parseable as direct parent of the current message. For affected repos, this will require reindexing (via "public-inbox-index --reindex"), but there will be no version bump for this bugfix.
2016-12-06linkify: implement Markdown link compatibility (again)
Although unescaped parentheses in URLs are technically allowed, they are uncommon. However, Markdown-like syntaxes are unfortunately common for URLs, so we might as well support them. This fixes parentheses detection at sentence endings, as seen in practice on emails.
2016-12-06Revert "linkify: implement Markdown link compatibility"
This reverts commit 130d0c4e33c5c73dc69e270fc698735d49e0f159.
2016-12-06linkify: implement Markdown link compatibility
Although unescaped parentheses in URLs are technically allowed, they are uncommon. However, Markdown-like syntaxes are unfortunately common for URLs, so we might as well support them.
2016-12-03atom: switch to getline/close for response bodies
This will let us stream larger Atom documents bodies without wasting too much memory and reduce the amount of round-trip requests needed to get necessary information. Hopefully clients are using streaming (SAX) parsers, too. This is the final transition in the core public-inbox code to allow migrating to a "pull"-based body streaming scheme which allows a HTTP server to respond appropriately to backpressure from slow clients.
2016-12-03wwwstream: improve documentation and variable naming
Hopefully this makes the code more readable for newbies.
2016-12-03searchview: fix <title> tag in Atom feed
This only affects the Atom feed for search results. "xmlstarlet val" failed to detect or warn about this, and I only noticed this bug while working on another patch.
2016-11-29note the source code is AGPL for cloning
This should be adequate warning for folks who may be uncomfortable or uncertain about even possessing AGPL source code due to employer agreements and such. Disclaimer: I remain completely in favor of AGPL and strong copyleft, and am more than willing to risk my own future on it. However, I refuse to even nudge people into downloading AGPL source code if it presents any legal risk to them.
2016-11-26avoid IO::File for anonymous temporary files
We do not need to import IO::File into the main programs since Perl 5.8+ supports literal "undef" for generating anonymous temporary file handles.
2016-11-26githttpbackend: error checking for input handling
This was sloppy code, all calls need to be checked for failure.
2016-11-22view: fix spaces in mailto: link
Some mail clients do not seem to handle '+' as a space in query parameters for the mail subject, use the more common '%20' for compatibility.
2016-10-16import: failed GC runs are non-fatal
We should not completely kill a process if "git gc --auto" errors out due to a warning or whatnot.
2016-10-14thread: reinstates stable ordering when ghosts are present
This reverts commit 3c9dd6619f825f0515e7e4afa1bd55c99c1a68d3 ("thread: fix sorting without topmost") and reinstates the "topmost" routine for sorting purposes.
2016-10-13thread: fix parent/child relationships
The ordering change in add_child is critical if $self == $parent as the {children} hash was lost before this change. has_descendent can be simplified by walking upwards from the child instead of downwards from the parent. This fixes threading regressions introduced in commit 30100c46326e2eac275e0af13116636701d2537e ("thread: use hash + array instead of hand-rolled linked list")
2016-10-13thread: reduce indentation level
This should reduce differences from the original Mail::Thread code and hopefully make things easier-to-follow.
2016-10-05thread: remove weaken dependency
We have to walk through all the messages after threading anyways to build the rootset, so we can just delete all the parent references at that point.
2016-10-05view: remove redundant children array in thread views
Each node has an entire arrayref of its children nowadays, so there's no need to waste time and memory creating another one.
2016-10-05thread: use hash + array instead of hand-rolled linked list
This starts to show noticeable performance improvements when attempting to thread over 400 messages; but the improvement may not be measurable with less. However, the resulting code is much shorter and (IMHO) much easier to understand.
2016-10-05thread: fix sorting without topmost
This bug was hidden, and we may not be able to efficiently implement a topmost subroutine with the hash-based (vs linked-list) based container for threading in the next commit.
2016-10-05thread: inline and remove recurse_down logic
We no longer recurse, and it's too hard to come up with a new name for a sub we will only use once.
2016-10-05thread: order_children no longer cares about depth
We never use the depth anywhere in this sub
2016-10-05thread: avoid incrementing undefined value
It is pointless to increment when setting a true value is simpler as there is no need to read before writing.
2016-10-05thread: remove iterate_down
Unnecessary subs and complexity. This was hiding the fact that $before is never used.
2016-10-05thread: simplify
Single use subroutines actually make the code more complex in this case, and there's never a {seen} field in $self.
2016-10-05thread: remove rootset accessor method
It doesn't buy us much and copying to a new array is slower; but probably not measurable in real-world use.
2016-10-05thread: remove Email::Abstract wrapping
This roughly doubles performance due to the reduction in object creation and abstraction layers.
2016-10-05inbox: deal with ghost smsg
smsg will be undef for ghost messages in a subsequent commit
2016-10-05thread: remove accessor usage in internals
This improves top-level index generation performance by 3-4%.
2016-10-05thread: pass array refs instead of entire arrays
Copying large arrays is expensive, so avoid it. This reduces /$INBOX/ time by around 1%.
2016-10-05thread: remove Mail::Thread dependency
Introduce our own SearchThread class for threading messages. This should allow us to specialize and optimize away objects in future commits.
2016-10-05view: remove "subject dummy" references
We will not care for inexact threading by subject or pruning.
2016-09-13help: document new search prefixes
Support (and document) 'a:' after all, as "mairix -h" uses it, so this should reduce the learning curve for mairix users.
2016-09-09nntp: cleanup: move use statements out of sub scope
This clarifies the code somewhat, and we don't care to lazy-load in NNTP.pm anyways since this is only used for a long-lived daemon.
2016-09-09search: index attachment filenames
And while we're at it, ensure searching inside displayable attachment bodies works.
2016-09-09search: match the behavior of WWW for indexing text
The basic rule is that if it is displayable via our WWW interface, it should be indexable text for Xapian search.
2016-09-09search: avoid mindlessly calling body_set
It's not worth entering a complex codepath in Email::MIME to save some (probably immeasurable amount of) memory, here. We've already stopped doing this in our WWW code a while back, too. If we really cared enough about it, we'd prioritize work on a streaming replacement for Email::MIME.
2016-09-09search: fix compatibility with Debian wheezy
Specifying the "d:" field only worked for NumberValueRangeProcessor in older versions of Xapian, such as the one in Debian wheezy (libsearch-xapian-perl=1.2.10.0-1) This slipped through since I rarely use wheezy, anymore, and perhaps nobody else does, either. Perhaps wheezy support may be dropped, soon. Unfortunately, this requires a schema version bump.
2016-09-09search: increase term positions for each quoted hunk
We pay a storage cost for storing positional information in Xapian, make good use of it by attempting to preserve it for (hopefully) better search results.
2016-09-09search: match quote detection behavior of view
This is stricter than the mutt quote_regexp default ("^([ \t]*[|>:}#])+" on Debian jessie), but matches what we have in View.pm. I prefer the stricter quote detection since it is less ambiguous and less likely to hide/obscure important details.
2016-09-09search: fix space regressions from recent changes
As of Xapian 1.0.4 (from 2007) is possible to use Search::Xapian::QueryParser::add_prefix multiple times with the same user field name but different term prefixes. This brings my current git@vger mirror from 6.5GB to 2.1GB (both sizes are after xapian-compact).
2016-09-09search: more granular message body searching
"bs:" and "b:" are adapted from mairix(1) We will also support searching explicitly for quoted vs non-quoted text via "q:" and "nq:" prefixes since sometimes readers will not care for quoted text. In the future, we will support parsing diffs (perhaps when repobrowse integration is complete). Note: this roughly doubles the size of the Xapian database due to the additional information; so this change may not be worth it.
2016-09-09search: drop longer subject: prefix for search
We only document the "s:" anyways. While the long name is more descriptive, the ambiguity makes agnostic caching (by Varnish or similar) slightly harder and longer URLs are more likely to be accidentally truncated when shared.
2016-09-09search: allow searching user fields (To/Cc/From)
Sometimes it can be useful to search based on who the message was sent to, sent by, or Cc:-ed. Of course, headers can be faked, but they usually are not... Anyways this mostly matches the behavior of mairix(1).