| Date | Commit message (Collapse) |
|---|
|
The "loose" (Subject:-based) thread matching yields too many
hits for some common subjects (e.g. "[GIT] Networking" on LKML)
and causes thread skeletons to not show the current messages.
Favor strict matches in the query and only add loose matches
if there's space.
While working on this, I noticed the backwards --reindex walk
breaks `tid' on v1 repositories, at least. That bug was hidden
by the Subject: match logic and not discovered until now. It
will be fixed separately.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
We no longer need to parse and dedupe References:
ourselves, PublicInbox::MID::references does it for us.
|
|
It is common to have large amounts of addresses Cc:-ed in large
mailing lists like LKML. Make them more readable by wrapping
after addresses. Unfortunately, line breaks inserted by the
MUA get lost when using the public Email::MIME API.
Subject and body lines remain unwrapped, as it's the author's
fault to have such long lines :P
|
|
The old loop did not help with code clarity with the various
conditional statements. It also hid a bug where we forgot to
(optionally) obfuscate email addresses in Subject: lines if
search was enabled.
|
|
Gigantic feeds probably make some clients unhappy,
clamp it to what it was in the past.
Fixes: b9534449ecce2c59 ("view: avoid offset during pagination")
|
|
Favor simpler internal APIs this time around, this cuts
a fair amount of code out and takes another step towards
removing Xapian as a dependency for v2 repos.
|
|
OFFSET in SQLite gets painful to deal with. Instead,
rely on timestamps (from Received:) for pagination.
This also sets us up for more precise Date searching
in case we want it.
|
|
In many cases, we do not care about the total number of
messages. It's a rather expensive operation in SQLite
(Xapian only provides an estimate).
For LKML, this brings top-level /$INBOX/ loading time from
~375ms to around 60ms on my system. Days ago, this operation
was taking 800-900ms(!) for me before introducing the SQLite
overview DB.
|
|
This is a smaller improvement than the landing /$INBOX/ page
because full message bodies are shown; but still saves around
100ms for my system with LKML.
|
|
It's no longer necessary to have this since load_expand
now populates $smsg->mid with the "preferred" Message-ID.
This saves around 10ms on the homepage for me.
|
|
This saves over 400ms on my system with the full LKML
with over 2.8 million messages.
|
|
We no longer need some of these old subroutines which
assumed a single Message-ID for each message.
|
|
Too many similar functions doing the same basic thing was
redundant and misleading, especially since Message-ID is
no longer treated as a truly unique identifier.
For displaying threads in the HTML, this makes it clear
that we favor the primary Message-ID mapped to an NNTP
article number if a message cannot be found.
|
|
By using the "primary" Message-ID in WwwAttach, we can avoid
conflicts in the links we use for downloading attachments.
|
|
We do not need to care about ghosts at multiple call sites; they
cannot have a {blob} field and we've stored the blob field in
Xapian since SCHEMA_VERSION=13.
|
|
Since we need to handle messages with multiple and duplicate
Message-ID headers, our thread skeleton display must account
for that.
Since we have a "preferred" Message-ID in case of conflicts,
use it as the UUID in an Atom feed so readers do not get
confused by conflicts.
|
|
This needs tests and further refinement, but current tests pass.
|
|
We want to rely on Date: to sort messages within individual
threads since it keeps messages from git-send-email(1) sorted.
However, since developers occasionally have the clock set
wrong on their machines, sort overall messages by the newest
date in a Received: header so the landing page isn't forever
polluted by messages from the future.
This also gives us determinism for commit times in most cases,
as we'll used the Received: timestamp there, as well.
|
|
The first Received: header is believable since it typically
hits the user's mail server and can be treated as relatively
trustworthy. We still show the Date: in per-message (permalink)
views, which may expose users for having incorrect Date:
headers, but all the ISO YYYY-MM-DD dates we display will
match what we see.
|
|
We haven't needed this since we integrated threading
and dropped Email::Abstract and Mail::Thread usage.
|
|
Using update-copyrights from gnulib
While we're at it, use the SPDX identifier for AGPL-3.0+ to
ease mechanical processing.
|
|
Sometimes, it can be desirable to jump directly to the "nested"
view when viewing a thread skeleton. This makes it possible.
While we're at it, shorten some of the text to ensure it still
fits in 80 columns.
|
|
We leave the mailto: link out when obfuscating address, so
do not stuff the "</pre>" closing tag into it. Instead,
keep the closing tag in the same context as the opening one,
making it easier to keep track of.
|
|
This makes the wording less confusing when showing archives
for lists where the convention is reply-to-list.
I still hate reply-to-list, but it's still better than no
archives or list at all.
|
|
It is usually pointless to replace a single word with a '"' character.
|
|
Perl 5.22 started warning about this.
|
|
We always do threading, so perhaps it's not a good name.
"Nested" is probably more appropriate and closer to what
people are used to seeing.
|
|
Since we attempt to fill in threads by Subject, our thread
skeletons can cross actual thread IDs, leading to the
possibility of false ghosts showing up in the skeleton.
Try to fill in the ghosts as well as possible by performing
a message lookup.
|
|
We should not blindly join References and In-Reply-To headers
as a single string, because some messages can have an open
angle brace '<' in References: without a corresponding '>'.
|
|
There is no need to show the same phrases over and over again
in thread skeletons, it adds to visual noise and makes things
more difficult to read.
|
|
We will also treat all known list addresses as non-obfuscated.
By setting publicinbox.noObfuscate in ~/.public-inbox/config,
this will allow users to disable address obfuscation on a
per-domain or per-address basis.
|
|
This is necessary to retain consistent spacing around bullet
points.
Fixes: 666844ae42b5b17f ("reply: handle address obfuscation :<")
|
|
We can show users a lightly-obfuscated Bourne shell command
for invoking "git send-email" for address obfuscation. However,
I'm not sure if the mailto: arg will work effectively since
URL encoding is probably too well-known to be effective.
|
|
This is lightly-tested and seems to work. I'm still
hesitant to support this, but the alternative of receiving death
threats for displaying unobfuscated addresses seems to
be not worth it.
|
|
This allows us to support centralized mailing lists (which suck,
but better than no mailing list at all).
|
|
We'll be adding more reply options for centralized mailing
lists. So split out the logic so it's easy-to-find.
Organizing code is hard :<
|
|
Otherwise funky filenames can cause HTML injection
vulnerabilities (hope you have JavaScript disabled!)
|
|
It seems possible for git-send-email(1) to generate repeated
repeated instances of References and In-Reply-To headers,
as evidenced in:
https://public-inbox.org/git/20161111124541.8216-17-vascomalmeida@sapo.pt/raw
This causes a mismatch between how our search indexer threads
and how our HTML view handles threading. In the future, View.pm
will use the smsg-parsed {references} field and avoid redoing
Email::MIME header parsing.
We will still need to figure out a way to deal with messages
with repeated Message-IDs, at some point, too.
|
|
This should fix problems with multipart messages where
text/plain parts lack a header.
cf. git clone --mirror https://github.com/rjbs/Email-MIME.git
refs/pull/28/head
In the future, we may still introduce as streaming
interface to reduce memory usage on large emails.
|
|
This results in over 1% speedup doing $MESSAGE_ID/T/ HTML
generation for a 368-message thread.
|
|
This results in a half percent speedup or so doing
$MESSAGE_ID/T/ HTML generation for a 368 message thread.
|
|
This allows a 3-4% speedup in $MESSAGE_ID/T/ page generation
speed for a 368+ message thread. It also more faithfully
preserves the message as intended; even if the it makes the
sender look like a space-wasting slob :P
|
|
And add a comment about it to remind our future selves.
|
|
This simplifies callers to prevent errors and avoids
needless object-orientation in favor of a single procedure
call to handle threading and ordering.
|
|
Instead, only preload the ->mid field for threading,
as we only need ->thread and ->path once in Search->get_thread
(but we will need the ->mid field repeatedly).
This more than doubles View->load_results performance on
according to thread-all on an inbox with over 300K messages.
|
|
In addition to needing to retry enquire queries, we also need
to protect document loading from the Xapian DB and retry on
modification, as it seems to throw the same errors.
Checking the $@ ref for Search::Xapian::DatabaseModifiedError
is actually in the test suite for both the XS and SWIG Xapian
bindings, so we should be good as far as forward/backwards
compatibility.
|
|
This makes life easier for the threading algorithm, as we can
use the implied ordering of timestamps to avoid temporary ghosts
and resulting container vivication.
This would've also allowed us to hide the bug (in most cases)
fixed by the patch titled "thread: last Reference always wins",
in case that needs to be reverted due to infinite looping.
|
|
This should reduce the number of subroutine calls needed
for the common case of real (non-ghost) messages as well
as shortening code.
|
|
This should avoid warnings during thread skeleton generation if
ever the Xapian database disagrees with View.pm about which is
the proper direct parent of a message. We will treat the data
in Xapian as the truth (if Xapian is available).
|
|
Some mail clients do not seem to handle '+' as a space in query
parameters for the mail subject, use the more common '%20' for
compatibility.
|