To: meta@public-inbox.org
Subject: [WIP] public-inbox 2.2.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline

This release includes several new features and fixes; mostly
around improved integration between inboxes and coderepos for
solver.  Portability and reliability is also improved, especially
in the internal process management of lei.

public-inbox-cindex is a new command to index coderepos for
WWW search and perform automatic associations between
coderepos and inboxes.  This makes solver vastly more useful
for the WWW UI as admins will no longer have to manually
associate coderepos with inboxes.

public-inbox-clone gains the ability to mirror entire (or partial)
grokmirror-compatible manifests.

Internal process and object management data structures are vastly
simplified throughout and error handling made more robust.

git SHA-256 support remains a work-in-progress for inboxes and
extindex due to the need to interoperate with SHA-1 epochs.

Upgrading:

  lei users need to "lei daemon-kill" after installation to load
  new code.  Normal daemons (read-only, and public-inbox-watch)
  will also need restarts, of course, but there's no
  backwards-incompatible data format changes so rolling back to
  older versions is harmless.

Compatibility:

  Uppercase newsgroup names were always broken with IMAP, POP3, and
  -extindex.  Uppercase names will now be lowercased by default and
  warnings will be emitted.  Conflicting newsgroup names (and `inboxdir'
  entries if `newsgroup' isn't specified) will also generate warnings
  since they break -extindex and the new -cindex (coderepo index).

New users + hackers:

  The install/ directory includes tools to automate installation and
  removal of dependencies for stripped-down or full setups.  See
  install/README for more details.

treewide

  * support raw UTF-8 headers from SMTPUTF8 hosts

  * standardize on `#' prefix for stderr diagnostics (previously `I:')

  * SHA-256 coderepos are fully supported (but not inboxes, yet)

  * for daemons serving public traffic, MALLOC_MMAP_THRESHOLD_=131072 is
    recommended to reduce fragmentation in glibc malloc, while jemalloc
    (tested as an LD_PRELOAD) is another option.

PublicInbox::WWW

  * support `+' in inbox names

  * support coderepo displays for systems without cgit

  * improve display of git tags, commits and trees in $INBOX/$OID/s/ endpoint

  * numerous memory usage reductions by avoiding Perl scratchpads

  * add #related anchor and search form to find related patches
    based on blob OIDs (IOW, exposing `lei p2q' to the web)

  * fix footer in listing of >200 inboxes

  * support dumb HTTP clones of SHA-256 git repos

  * add /$INBOX/$MSGID/d/ endpoint to show diffs in reused Message-IDs
    (`lei mail-diff' for the web)

  * support POST /$INBOX/$MSGID/?x=m&q= to limit mbox results to a thread

  * add topics_(new|active).(html|atom) endpoints

  * linkify peer public-inbox addresses in To/Cc headers

public-inbox-watch:

  * watching MH folders is now supported

lei

  * use http.proxy / http.<remote>.proxy from system-wide git-config if
    unconfigured for lei

  * improve IMAP and NNTP error reporting

  * reduce default IMAP connections to avoid overloading servers

  * compatibility with SQLite <3.8.3 on CentOS 7.x

  * fix `lei q -tt' on locally indexed messages (still broken for remotes:
    https://public-inbox.org/meta/20230226170931.M947721@dcvr/ )

  * `lei import' now sets labels+keywords consistently on all
     already imported messages

  * fix `lei up' on saved local queries which previously used -t/--threads

  * `lei convert' output to v2 public-inboxes is now idempotent

  * improved bash completion for labels (see contrib/completion)

  * support for reading (but not writing) MH folders

  * `lei index' accepts `+L:$LABEL' like `lei import' does

solver (used by lei (rediff|blob), and PublicInbox::WWW)

  * handle copies in patches properly

  * no longer redundantly parallelized within each WWW process

portability

  * SIGWINCH is handled properly on less common architectures and OSes

  * fix EINTR handling for kqueue users

  * various fixes for CentOS 7.x

  * fix excessive pipelining to `git cat-file' on systems with small
    getdelim(3) buffers (mainly affects musl)

  * support Alpine Linux, Dragonfly, NetBSD and OpenBSD.  This resulted
    not only in bugfixes to our code, but also to Dragonfly and OpenBSD.

  * Inline::C||Socket::MsgHdr no longer required for SCM_RIGHTS
    with sendmsg/recvmsg on supported *BSDs.

  * inotify support no longer requires Linux::Inotify2 XS package
    for most architectures

public-inbox-pop3d

  * support `limit=NUM' and `initial_limit=NUM' query parameters
    in mailbox names to limit results

public-inbox-nntpd

  * fix LISTGROUP with range (affects neomutt)

public-inbox-clone / public-inbox-fetch / `lei add-external --mirror'

  * mtime of downloaded manifest preserved

public-inbox-clone:

  * parallel mirroring of multiple inboxes/coderepos via manifest,
    public-inbox-fetch is not used in this mode

  * new flags to support manifest mirroring include:
    --dry-run, --inbox-config=, --project-list=, --prune, --purge,
    --keep-going, --jobs, --include=, --exclude=, --objstore=,
    --manifest=, --remote-manifest=
    See public-inbox-clone(1) man page for more details.

PublicInbox::SaPlugin::ListMirror

  * List-ID handling special-cased according to RFC 2919 rules

Search improvements (lei and PublicInbox::WWW)

  * quoted text inside base-85 binary patches is no longer indexed

  * `public-inbox-cindex --join' prefers using Xapian's C++ API
    directly to avoid Perl method dispatch overhead to get usable
    performance associating ~300 inboxes with over 1K coderepos
    (and vice versa).  Users requiring such performance will need
    a C++ compiler, pkg-config, and the Xapian development files
    (see INSTALL).

    This C++ helper will be used more heavily in the future
    to enable query parser customizations and other functionality
    unavailable from the Xapian SWIG or XS bindings.

Thanks to all the bug reporters and users who made this release
possible, and thanks for bearing with my anxiety over making releases.

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.