about summary refs log tree commit homepage
DateCommit message (Collapse)
2018-05-09public-inbox 1.1.0-pre1 v1.1.0-pre1
Pre-release for v2 repository support. Thanks to The Linux Foundation for supporting this work!
2018-05-02scripts/import_slrnspool: cleanup progress messages
Stop showing redundant slashes and stop showing progress for messages which do not exist.
2018-05-02scripts/import_slrnspool: support v2 repos
2018-05-01searchidx: preserve umask when starting/committing transactions
Xapian will replace files upon committing, so non-parallel V2Writable users need to have umask preserved this way.
2018-04-25thread: sort incoming messages by Date
Improve the display by finding any parent when we see out-of-order References. This prevents us from having two roots in the test case like Mail::Thread does.
2018-04-25thread: prevent hidden threads in /$INBOX/ landing page
In retrospect, the loop prevention done by our indexer is not always sufficient since it can have an improperly sorted or incomplete References headers. This bug was triggered multiple bracketed Message-IDs in an In-Reply-To: header (not References) where the Message-IDs were in non-chronological order when somebody tried to reply to different leafs of a thread with a single message. So we must check for descendents before blindly trying to use the last one. Fixes: c6a8fdf71e2c336f ("thread: last Reference always wins")
2018-04-24msgmap: add limit to response for NNTP
All callers in expect to iterate through results. This was causing unfairness when fetching large ranges via XHDR as rtin does :< Fixes: b8c41362f2a5c8fc "nntp: simplify the long_response API"
2018-04-23search: avoid repeated mbox results from search
Previous search queries already set sort order on the Enquire object, altering the ordering of results and was causing messages to be redundantly downloaded via POST /$INBOX/?q=$QUERY&x=m So stop caching the Search::Xapian::Enquire object since it wasn't providing any measurable performance improvement.
2018-04-23searchview: do not blindly append "l" parameter to URL
It's ugly and all of our other parameters are omitted when values are not the default.
2018-04-23TODO: add EPOLLEXCLUSIVE item
Threads are generally discouraged in Perl5, so I won't be using a dedicated blocking accept4() thread like I would in other languages.
2018-04-23view: drop redundant References: display code
We no longer need to parse and dedupe References: ourselves, PublicInbox::MID::references does it for us.
2018-04-23view: wrap To: and Cc: headers in HTML display
It is common to have large amounts of addresses Cc:-ed in large mailing lists like LKML. Make them more readable by wrapping after addresses. Unfortunately, line breaks inserted by the MUA get lost when using the public Email::MIME API. Subject and body lines remain unwrapped, as it's the author's fault to have such long lines :P
2018-04-23view: untangle loop when showing message headers
The old loop did not help with code clarity with the various conditional statements. It also hid a bug where we forgot to (optionally) obfuscate email addresses in Subject: lines if search was enabled.
2018-04-22extmsg: use Xapian only for partial matches
"LIKE" in SQLite (and other SQL implementations I've seen) is expensive with nearly 3 million messages in the archives. This caused some partial Message-ID lookups to take over 600ms on my workstation (~300ms on a faster Xeon). Cut that to below under 30ms on average on my workstation by relying exclusively on Xapian for partial Message-ID lookups as we have in the past. Unlike in the past when we tried using Xapian to match partial Message-IDs; we now optimize our indexing of Message-IDs to break apart "words" in Message-IDs for searching, yielding (hopefully) "good enough" accuracy for folks who get long URLs broken across lines when copy+pasting. We'll also drop the (in retrospect) pointless stripping of "/[tTf]" suffixes for the partial match, since anybody who hits that codepath would be hitting an invalid message ID. Finally, limit wildcard expansion to prevent easy DoS vectors on short terms. And blame Pine and alpine for generating Message-IDs with low-entropy prefixes :P
2018-04-20convert: copy description and git config from v1 repo
I noticed I lost a $GIT_DIR/description in a conversion, so we should preserve it. While we're at it, we ought to copy any config in the old repo to the new one. We will need to warn about cloneurl since it's unfortunately not an automatic process to update. Oh well..
2018-04-20searchidx: remove leftover debugging code
I was using this to trace the path of brian's message. Fixes: 017fed7bc4d33ac4 ("searchidx: regenerate and avoid article number gaps on full index")
2018-04-20searchidx: release lock again during v1 batch callback
Relaxing this lock during a v1 --reindex is important to keep messages showing up in -watch process in a timely manner. Looks like I deleted an extra line when doing the following for v2: s/xdb->commit_transaction/self->commit_txn_lazy/ Fixes: 35ff6bb106909b1c ("replace Xapian skeleton with SQLite overview DB")
2018-04-20disallow "\t" and "\n" in OVER headers
For Subject/To/Cc/From headers, we squeeze them to a space (' '). For Message-IDs (including References/In-Reply-To), '\t', '\n', '\r' are deleted since some MUAs might screw them up: https://public-inbox.org/git/656C30A1EFC89F6B2082D9B6@localhost/raw
2018-04-20import: cleanup git cat-file processes when ->done
This should reduce idle cat-file instances
2018-04-19filter/rubylang: do not set altid on spam training
I suppose it's a bug or inconsistency that altid is write-only and their deletions do not get reflected. But for now, we do not set it when training spam so there's no window where an invalid NNTP article number shows up. This should solve the problem where there's massive gaps in messages solved by spam training for ruby groups: https://public-inbox.org/meta/20180307093754.GA27748@dcvr/
2018-04-19TODO: a few more updates
Mainly, v2 stuff is done
2018-04-19fix tests to run without Xapian installed
We'll be ensuring we can run more of the HTTP and all of the NNTP interface with only SQLite (and not Xapian) installed in the future.
2018-04-18ensure SQLite and Xapian files respect core.sharedRepository
We can't have files with permissions inconsistent with what's in git objects.
2018-04-18Merge remote-tracking branch 'origin/master' into v2
* origin/master: nntp: allow and ignore empty commands mbox: do not barf on queries which return no results nntp: fix NEWNEWS command searchview: fix non-numeric comparison Allow specification of the number of search results to return githttpbackend: avoid infinite loop on generic PSGI servers http: fix modification of read-only value extmsg: use news.gmane.org for Message-ID lookups extmsg: rework partial MID matching to favor current inbox Update the installation instructions with Fedora package names nntp: do not drain rbuf if there is a command pending nntp: improve fairness during XOVER and similar commands searchidx: do not modify Xapian DB while iterating Don't use LIMIT in UPDATE statements
2018-04-18searchidx: revert default BATCH_BYTES to 1_000_000
This increases indexing time by around 10% but roughly halves memory usage of an -index process. We will probably make this tunable in the future for people with bigger/smaller machines.
2018-04-18nntp: allow and ignore empty commands
Somebody hitting "\n" into telnet shouldn't hold a client up indefinitely and prevent shutdown.
2018-04-18searchidx: increase term positions for all text terms
We do not want phrase searches to cross between independent fields (filenames/Message-ID vs bodies)
2018-04-18use %H consistently to disable abbreviations
We generally do not want git to waste time finding abbreviations and we do not want the possibility of them becoming ambiguous over time, either.
2018-04-18extmsg: remove expensive git path checks
Searching across different inboxes is expensive without SQLite (or Xapian) installed, so avoid doing expensive tree lookups in git. Since SQLite is required for Xapian support anyways, we won't need to check Xapian, either. Sites without SQLite installed will simply 404 if somebody requests a message which isn't in the current inbox.
2018-04-18searchidx: regenerate and avoid article number gaps on full index
Some messages to git@vger went missing from Msgmap from old bugs and became inaccessible via NNTP. Forcing NNTP article numbers when the overview DB came about made the problem more visible when reindexing old (v1) repositories as all removed spam messages took up AUTOINCREMENT numbers again before they were removed. Having large gaps in NNTP article numbers is not good since it throws off NNTP clients. This does NOT prevent NNTP clients from seeing some messages twice, but is better than having them miss several messages entirely. We also avoid depending on --reverse in git-log, as git requires storing an entire commit list in memory for --reverse, so it's cheaper to store only deleted blobs in the %D hash since they do not live long.
2018-04-18import: cat_blob drops leading 'From ' lines like Inbox
In case people were running old buggy versions from 2016... (and -convert should probably clean those up, eventually)
2018-04-18v2: improve deduplication checks
First off, decode text portions of messages since some archived mail I got was converted from quoted-printable or base-64 to 8bit by the original recipient. Attempting to merge them with my own archives (which had no conversion done) led to unnecessary duplicates showing up. Then, normalize CRLF line endings in text portions to LF. In the headers, we relax the content_id hashing to ignore quotes and lower-case domain names in To, Cc, and From headers since some mail processors will alter them. Finally, I've discovered Email::MIME->new($mime->as_string) does not always round-trip reliably, so we calculate the content_id twice on user-supplied messages.
2018-04-18v2: generate better Message-IDs for duplicates
While hunting duplicates, I noticed a leading '-' in some Message-IDs as a result of RFC4648 encoding. While '-' seems allowed by RFC5322 and URL-friendly (RFC4648), they are uncommon and make using Message-IDs as arguments for command-line tools more difficult. So prefix them with a datestamp to at least give readers some sense of the age. And shorten the "localhost" hostname to "z" to save space.
2018-04-18search: preserve References in Xapian smsg for x=t view
I'm not sure how useful this view is, but it exists for now.
2018-04-18v2writable: reduce partititions by one
git fast-import and the main V2Writable process combined takes about one CPU, so avoid having too many Xapian partitions which cause unnecessary I/O contention.
2018-04-18compact: do not merge v2 repos by default
--no-renumber does not allow merging, and merging is not ideal for reindexing, either.
2018-04-18v1: remove articles from overview DB
Otherwise articles show up again...
2018-04-18feed: respect feedmax, again
Gigantic feeds probably make some clients unhappy, clamp it to what it was in the past. Fixes: b9534449ecce2c59 ("view: avoid offset during pagination")
2018-04-07msgmap: speed up minmax with separate queries
This significantly improves the performance of the NNTP GROUP command with 2.7 million messages from over 250ms to 700us. SQLite is weird about this, but at least there's a way to optimize it.
2018-04-07store less data in the Xapian document
Since we only query the SQLite over DB for OVER/XOVER; do not need to waste space storing fields To/Cc/:bytes/:lines or the XNUM term. We only use From/Subject/References/Message-ID/:blob in various places of the PSGI code. For reindexing, we will take advantage of docid stability in "xapian-compact --no-renumber" to ensure duplicates do not show up in search results. Since the PSGI interface is the only consumer of Xapian at the moment, it has no need to search based on NNTP article number.
2018-04-07convert: support converting with altid defined
public-inbox-convert ought to be 100% lossless, now
2018-04-07index: allow specifying --jobs=0 to disable multiprocess
Not everybody needs multiprocess support.
2018-04-07v2writable: reduce barriers
Since we handle the overview info synchronously, we only need barriers in tests, now. We will use asynchronous checkpoints to sync less-important Xapian data. For data deduplication, this requires us to hoist out the cat-blob support in ::Import for reading uncommitted data in git.
2018-04-07over: remove forked subprocess
Since the overview stuff is a synchronization point anyways, move it into the main V2Writable process and allow us to drop a bunch of code. This is another step towards making Xapian optional for v2. In other words, the fan-out point is moved and the Xapian partitions no longer need to synchronize against each other: Before: /-------->\ /---------->\ v2writable -->+----parts----> over \---------->/ \-------->/ After: /----------> /-----------> v2writable --> over-->+----parts---> \-----------> \----------> Since the overview/threading logic needs to run on the same core that feeds git-fast-import, it's slower for small repos but is not noticeable in large imports where I/O wait in the partitions dominates.
2018-04-07over: avoid excessive SELECT
No need to read what we don't need into the Perl process. Fix some broken capitalization while we're at it.
2018-04-07psgi: ensure /$INBOX/$MESSAGE_ID/T/ endpoint is chronological
We only need to call get_thread beyond 1000 messages for fetching entire mboxes. It's probably too much for the HTML display otherwise.
2018-04-06ensure Xapian and SQLite are still optional for v1 tests
Xapian is size-intensive and SQLite is not strictly necessary for v1.
2018-04-06www: favor reading more from SQLite, and less from Xapian
Favor simpler internal APIs this time around, this cuts a fair amount of code out and takes another step towards removing Xapian as a dependency for v2 repos.
2018-04-06nntp: set Xref across multiple inboxes
Noted by Jonathan Corbet in https://lwn.net/Articles/748184/
2018-04-06altid: fix miscopied field name
Oops :x