From: Eric Wong <firstname.lastname@example.org> To: email@example.com Subject: [PATCH 4/7] doc: update 1.7 release notes, tuning, TODO Date: Thu, 11 Mar 2021 02:45:35 -0800 [thread overview] Message-ID: <firstname.lastname@example.org> (raw) In-Reply-To: <email@example.com> Some stuff done, some stuff still needs doing. --- Documentation/RelNotes/v1.7.0.wip | 59 ++++++++++++++++++++++++++- Documentation/public-inbox-tuning.pod | 9 ++++ TODO | 16 +++----- 3 files changed, 72 insertions(+), 12 deletions(-) diff --git a/Documentation/RelNotes/v1.7.0.wip b/Documentation/RelNotes/v1.7.0.wip index a35ff227..f71f447f 100644 --- a/Documentation/RelNotes/v1.7.0.wip +++ b/Documentation/RelNotes/v1.7.0.wip @@ -4,12 +4,69 @@ MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline -TODO: gcf2, detached indices, JMAP, ... +Another big release focused on multi-inbox search and scalability. + +* general changes + + config file parsing is 2x faster with 50K inboxes + +* read-only public-inbox-daemon (-httpd, -nntpd, -imapd): + + libgit2 may be used via Inline::C to avoid hitting system pipe + and process limits. See public-inbox-tuning(7) manpage + for more details. + +* public-inbox-extindex + + A new Xapian + SQLite index able to search across several inboxes. + This may be configured to replace per-inbox Xapian DBs, + (but not per-inbox SQLite indices) and speed up manifest.js.gz + generation. + + See public-inbox-extindex-format(5) and + public-inbox-extindex(1) manpages for more details. + +* public-inbox-nntpd + + - startup is 6x faster with 50K inboxes if using -extindex + +* PublicInbox::WWW + + - mboxrd search results are returned in reverse Xapian docid order, + so more recent results are more likely to show up first + + - d: and dt: search prefixes allow "approxidate" formats supported + by "git log --since=" + + - manifest.js.gz generation is ~25x faster with -extindex + +* lei - local email interface + + An experimental, subject-to-change, likely-to-eat-your-mail tool for + personal mail as well as interacting with public-inboxes on the local + filesystem or over HTTP(S). See lei(1), lei-overview(7), and other + lei-* manpages for details. + +* public-inbox-watch + + - IMAP and NNTP code shared with lei, fixing an off-by-one error + in IMAP synchronization for single-message IMAP folders. + + - \Deleted and \Draft messages ignored for IMAP, as they are for + Maildir. + + - IMAP and NNTP connection establishment (including git-credential + prompts) ordering is now tied to config file order. Compatibility: * Rollbacks all the way to public-inbox 1.2.0 remain supported +Internal changes + +* public-inbox-index switched to new internal IPC code shared + with lei + Please report bugs via plain-text mail to: firstname.lastname@example.org See archives at https://public-inbox.org/meta/ for all history. diff --git a/Documentation/public-inbox-tuning.pod b/Documentation/public-inbox-tuning.pod index e9702416..b3a2b411 100644 --- a/Documentation/public-inbox-tuning.pod +++ b/Documentation/public-inbox-tuning.pod @@ -55,6 +55,15 @@ public-inbox processes. More (optional) L<Inline::C> use will be introduced in the future to lower memory use and improve scalability. +=head2 libgit2 usage via Inline::C + +If libgit2 development files are installed and L<Inline::C> +is enabled (described above), per-inbox C<git cat-file --batch> +processes are replaced with a single L<perl(1)> process running +C<PublicInbox::Gcf2::loop> in read-only daemons. + +Available as of public-inbox 1.7.0. + =head2 Performance on rotational hard disk drives Random I/O performance is poor on rotational HDDs. Xapian indexing diff --git a/TODO b/TODO index 53907efd..4993b02c 100644 --- a/TODO +++ b/TODO @@ -86,7 +86,10 @@ all need to be considered for everything we introduce) * more and better test cases (use git fast-import to speed up creation) -* large mbox/Maildir/MH/NNTP spool import (see PublicInbox::Import) +* large mbox/Maildir/MH/NNTP spool import (in lei, but not + for public-facing inboxes) + +* MH import support (read-only, at least) * Read-only WebDAV interface to the git repo so it can be mounted via davfs2 or fusedav to avoid full clones. @@ -133,18 +136,9 @@ all need to be considered for everything we introduce) - inotify-based manifest.js.gz updates - - process/FD reduction (needs to be slow-storage friendly) - ... -* command-line tool (similar to mairix/notmuch, but solver+git-aware) - -* consider removing doc_data from Xapian, redundant with over.sqlite3 - It's no longer read as of public-inbox 1.6.0, but still written for - compatibility. - -* share "git cat-file --batch" processes across inboxes to avoid - bumping into /proc/sys/fs/pipe-user-pages-* limits +* lei - see %CMD in lib/PublicInbox/LEI.pm * make "git cat-file --batch" detect unlinked packfiles so we don't have to restart processes (very long-term)
next prev parent reply other threads:[~2021-03-11 10:45 UTC|newest] Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-03-11 10:45 [PATCH 0/7] doc updates, fixups, and more Eric Wong 2021-03-11 10:45 ` [PATCH 1/7] doc: glossary: add information for dates and timestamps Eric Wong 2021-03-11 10:45 ` [PATCH 2/7] searchidx: remove smsg_from_doc Eric Wong 2021-03-11 10:45 ` [PATCH 3/7] lei_curl: note proposed master/client mode for curl Eric Wong 2021-03-11 10:45 ` Eric Wong [this message] 2021-03-11 10:45 ` [PATCH 5/7] imapclient: disable workaround for Mail::IMAPClient 3.43+ Eric Wong 2021-03-11 20:34 ` [SQUASH] " Eric Wong 2021-03-11 10:45 ` [PATCH 6/7] config: use '-f' key to store config file pathname Eric Wong 2021-03-11 10:45 ` [PATCH 7/7] v2writable: fix undocumented --xapian-only Eric Wong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://public-inbox.org/README * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [PATCH 4/7] doc: update 1.7 release notes, tuning, TODO' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://80x24.org/public-inbox.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).