about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2018-04-05searchmsg: remove unused `tid' and `path' methods
These internal attributes are not exposed and no longer used in our APIs.
2018-04-05v2writable: recount partitions after acquiring lock
The partition count can change if public-inbox-compact runs while public-inbox-watch or public-inbox-index is running.
2018-04-04v2writable: do not modify DBs while iterating for ->remove
Xapian may become unhappy if a DB is modified during iteration: nntp://news.gmane.org/20180228004400.GU12724@survex.com
2018-04-04v2: support incremental indexing + purge
This is important for people running mirrors via "git fetch", as they need to be kept up-to-date. Purging is also now supported in mirrors. The short-lived "--regenerate" option is gone and is now implicitly enabled as a result. It's still cheap when article number regeneration is unnecessary, as we track the range for each git repository.
2018-04-04import: rewrite less history during purge
We do not need to rewrite old commits unaffected by the object_id purge, only newer commits. This was a state management bug :x We will also return the new commit ID of rewritten history to aid in incremental indexing of mirrors for the next change.
2018-04-04searchidx: ensure duplicated Message-IDs can be linked together
This allows us to emulate the display of thread-aware MUAs when multiple messages share the same Message-ID. This also is a place where "public-inbox-index --reindex" is useful to fix existing messages and no schema version bump is necessary.
2018-04-03nntp: simplify the long_response API
We we worked around the default range/termination conditions of long_response in many cases to reduce calls to SQLite or Xapian. So continue that trend and become more like the PSGI API which doesn't force callers to specify an article range or work inside a loop.
2018-04-03msgmap: replace id_batch with ids_after
id_batch had a an overly complicated interface, replace it with id_batch which is simpler and takes advantage of selectcol_arrayref in DBI. This allows simplification of callers and the diffstat agrees with me.
2018-04-03mbox: remove remaining OFFSET usage in SQLite
We can use id_batch in the common case to speed up full mbox retrievals. Gigantic msets are still a problem, but will be fixed in future commits.
2018-04-03view: avoid offset during pagination
OFFSET in SQLite gets painful to deal with. Instead, rely on timestamps (from Received:) for pagination. This also sets us up for more precise Date searching in case we want it.
2018-04-03nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster
While SQLite is faster than Xapian for some queries we use, it sucks at handling OFFSET. Fortunately, we do not need offsets when retrieving sorted results and can bake it into the query. For inbox.comp.version-control.git (v1 Xapian), XOVER and XHDR are over 20x faster.
2018-04-02over: speedup get_thread by avoiding JOIN
JOIN operations on SQLite can be disasterously slow. This reduces per-message pages with the thread overview at the bottom of those pages from over 800ms to ~60ms. In comparison, the v1 code took around 70-80ms using Xapian on my machine.
2018-04-02www: rework query responses to avoid COUNT in SQLite
In many cases, we do not care about the total number of messages. It's a rather expensive operation in SQLite (Xapian only provides an estimate). For LKML, this brings top-level /$INBOX/ loading time from ~375ms to around 60ms on my system. Days ago, this operation was taking 800-900ms(!) for me before introducing the SQLite overview DB.
2018-04-02v2writable: simplify barrier vs checkpoints
searchidx_checkpoint was too convoluted and confusing. Since barrier is mostly the same thing; use that instead and add an fsync option for the overview DB.
2018-04-02replace Xapian skeleton with SQLite overview DB
This ought to provide better performance and scalability which is less dependent on inbox size. Xapian does not seem optimized for some queries used by the WWW homepage, Atom feeds, XOVER and NEWNEWS NNTP commands. This can actually make Xapian optional for NNTP usage, and allow more functionality to work without Xapian installed. Indexing performance was extremely bad at first, but DBI::Profile helped me optimize away problematic queries.
2018-04-01search: reduce columns stored in Xapian
We can store :bytes and :lines in doc_data since we never sort or search by them. We don't have much use for the Date: stamp at the moment, either.
2018-04-01truncate Message-IDs and References consistently
We need to stop ghost messages from generating longer Message-IDs than Xapian can handle with terms.
2018-04-01v2writable: fix parallel termination
I was too aggressively disabling parallelization to speed up the test suite and broke this :x Re-enable parallelization for the v2reindex test so we can catch it later.
2018-04-01v2: one file, really
We need to ensure there is only one file in the top-level tree at any commit so the "add; remove; add;" sequence on the same message is detected properly. Otherwise, git will not detect the second "add" unless a second message is added to history. Deletes are now stored in "d" (and not "D" or "_/D") at the top-level, now. There's no need to have a "_" to reduce churn as "m" and "d" should never co-exist. It's now lowercased to make it easier-to-distinguish from "D" in git-log output.
2018-03-30searchidx: correct warning for over-vivification
We will vivify multiple ghosts if a message has multiple Message-IDs.
2018-03-30v2: respect core.sharedRepository in git configs
Ensure -convert and -compact do not make repositories unreadable on live servers.
2018-03-30search: move permissions handling to InboxWritable
We'll be making sure V2Writable uses this.
2018-03-30msgtime: parse 3-digit years properly
Some folks had bad mail clients which generated 3-digit years around Y2K...
2018-03-30feed: optimize query for feeds, too
This is a smaller improvement than the landing /$INBOX/ page because full message bodies are shown; but still saves around 100ms for my system with LKML.
2018-03-30view: drop load_results
It's no longer necessary to have this since load_expand now populates $smsg->mid with the "preferred" Message-ID. This saves around 10ms on the homepage for me.
2018-03-30view: speed up homepage loading time with date clamp
This saves over 400ms on my system with the full LKML with over 2.8 million messages.
2018-03-30v2writable: go backwards through alternate Message-IDs
This is consistent with how we internally generate new Message-IDs to break conflicts and allows ->reindex to succeed while walking backwards through history
2018-03-30wwwstream: flesh out clone instructions for v2
Relying solely on git for v2 repos is probably not so useful, so add pointers to public-inbox-init/index commands.
2018-03-30v2writable: convert some fatal reindex errors to warnings
By supporting purge and allowing users to delete git partitions, we can open up ourselves to gaps and un-reindexible data. Let that be.
2018-03-30v2writable: allow gaps in git partitions
Somebody may only care about the most recent history, so allow -init and -index to operate quietly on missing partitions.
2018-03-30search: warn on reopens and die on total failure
-watch on a busy/giant Maildir caused too many Xapian errors while attempting to browse.
2018-03-29mda: support v2 inboxes
I mainly focus on -watch for mirroring busy mailing lists, but using -mda should remain an option.
2018-03-29v2writable: initializing an existing inbox is idempotent
And we do not want to start making confused repos if somebody leaves out "-V2" the second time around.
2018-03-29import: run_die supports redirects as spawn does
We'll be using it in more future tests and scripts.
2018-03-29search: retry_reopen on first_smsg_by_mid
This was causing errors while attempting to load messages via the WWW interface while mass-importing LKML. While we're at it, remove unnecessary eval from lookup_article.
2018-03-29view: get rid of some unnecessary imports
We no longer need some of these old subroutines which assumed a single Message-ID for each message.
2018-03-29www: cleanup expensive fallback for legacy URLs
Back in the day, we compressed long Message-IDs to SHA-1 hexdigests for the URL. This now redirects to a 301 in the hopes we can remove these checks some day to reduce overhead.
2018-03-29mbox: avoid extracting Message-ID for linkification
We can avoid a small amount of overhead and use the "preferred" Message-ID based on what is in the SearchMsg object.
2018-03-29v2writable: cleanup: get rid of unused fields
The layout of this structure ended up being a bit different and the read-only access is handled through the ::Inbox class, instead.
2018-03-29search: move find_doc_ids to searchidx
We do not need this subroutine for read-only use in Search.pm
2018-03-29search: get rid of most lookup_* subroutines
Too many similar functions doing the same basic thing was redundant and misleading, especially since Message-ID is no longer treated as a truly unique identifier. For displaying threads in the HTML, this makes it clear that we favor the primary Message-ID mapped to an NNTP article number if a message cannot be found.
2018-03-29search: cleanup uniqueness checking
The only Xapian term which should be unique is the NNTP article number; so we no longer need find_unique_doc_id.
2018-03-29v2writable: support purging messages from git entirely
Purging existing messages is fairly straightforward since we can take advantage of Xapian and lookup the git object_id with it. Unfortunately, purging an already "removed" message (which is no longer in Xapian) is not as easy and we'll need to expose ->purge_oids to purge by the git object_id (currently SHA-1). Furthermore, we expire reflogs and prune in hopes a dumb HTTP client won't get the object.
2018-03-29searchmsg: document why we store To: and Cc: for NNTP
Otherwise I would forget and be tempted to remove them.
2018-03-29www: fix attachment downloads for conflicted Message-IDs
By using the "primary" Message-ID in WwwAttach, we can avoid conflicts in the links we use for downloading attachments.
2018-03-29lookup by Message-ID favors the "primary" one
The Message-ID mapped to an NNTP article number is stronger, so we will favor that for attachment lookups.
2018-03-29v2writable: append, instead of prepending generated Message-ID
The original Message-ID is still the most important when discussing with other recipients who do not rely on a message flowing through public-inbox. So whatever Message-ID we use to deduplicate internally will be secondary and less important. All of our front-end v2 code is order-independent, so we won't let the message count against us, that way.
2018-03-29www: remove unnecessary ghost checks
We do not need to care about ghosts at multiple call sites; they cannot have a {blob} field and we've stored the blob field in Xapian since SCHEMA_VERSION=13.
2018-03-27www: support cloning individual v2 git partitions
This will require multiple client invocations, but should reduce load on the server and make it easier for readers to only clone the latest data. Unfortunately, supporting a cloneurl file for externally-hosted repos will be more difficult as we cannot easily know if the clones use v1 or v2 repositories, or how many git partitions they have.
2018-03-27githttpbackend: avoid infinite loop on generic PSGI servers
We must detect EOF when reading a POST body with standard PSGI servers. This does not affect deployments using the standard public-inbox-httpd; but most smaller inboxes should be able to get away using a generic PSGI server.