public-inbox releases
help / color / mirror / Atom feed
* [ANNOUNCE] public-inbox 1.9.0
From: Eric Wong @ 2022-08-21  2:36 UTC (permalink / raw)
  To: meta

Upgrading:

  lei users need to "lei daemon-kill" after installation to load
  new code.  Normal daemons (read-only, and public-inbox-watch)
  will also need restarts, of course, but there's no
  backwards-incompatible data format changes so rolling back to
  older versions is harmless.

Major bugfixes:

  * lei no longer freezes from inotify/EVFILT_VNODE handling,
    user interrupts (Ctrl-C), nor excessive errors/warnings

  * IMAP server fairness improved to avoid excessive blob prefetch

New features:

  * POP3 server support added, use either public-inbox-pop3d or
    the new public-inbox-netd superserver

  * public-inbox-netd superserver supporting any combination of HTTP,
    IMAP, POP3, and NNTP services; simplifying management and allowing
    more sharing of memory used for various data structures.

  * public-inbox-httpd and -netd support per-listener .psgi files

  * SIGHUP reloads TLS certs and keys in addition to config and .psgi files

  * "lei reindex" command for lei users to update personal index
    in ~/.local/share/lei/store for search improvements below:

Search improvements:

  These will require --reindex with public-inbox-index and/or
  public-inbox-extindex for public inboxes.

  * patchid: prefix search support added to WWW and lei for
    "git patch-id --stable" support

  * text inside base-85 binary patches are no longer indexed
    to avoid false positives

  * for lei users, "lei reindex" now exists and is required
    to take advantage of aforementioned indexing changes

Performance improvements:

  * IMAP server startup is faster with many mailboxes when using
    "public-inbox-extindex --all"

  * NNTP group listings are also faster with many inboxes when
    using "public-inbox-extindex --all"

  * various small opcode and memory usage reductions

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.8.0
From: Eric Wong @ 2022-04-23  8:22 UTC (permalink / raw)
  To: meta

A minor release focused on bugfixes and minor improvements.
Upgrades should happen transparently, but downgrading back to
1.7.0 will likely cause problems for lei users (and only lei
users).

lei users may experience duplicate messages in Maildirs if attempting to
downgrade from 1.8.0 to 1.7.x.  public-inbox-* tools are unaffected and
may downgrade freely.

Bugfixes:

  Numerous test fixes thanks to NixOS developers.

  Long-running daemons are more robust in case of corrupt blobs
  or crashes of git-cat-file processes

  PublicInbox::WWW: all CR are removed before LF, fixing display of
  CR-CR-LF messages.

  Solver supports SHA-256 code repositories (inbox and lei store support
  is still pending).

Internal updates:

  Reduced dependencies on Inline::C for Linux users; Linux users may
  now use lei with neither Inline::C nor Socket::MsgHdr installed.

New features:

  The --dangerous flag is now supported in public-inbox-index and
  public-inbox-extindex to use the Xapian::DB_DANGEROUS flag for initial
  indexes.  This may reduce SSD/HDD wear at the expense of disallowing
  concurrency and data integrity in case of an unexpected shutdown.

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.7.0
From: Eric Wong @ 2021-11-04  7:52 UTC (permalink / raw)
  To: meta

Another big release focused on multi-inbox search and scalability.

Special thanks to Konstantin Ryabitsev and Kyle Meyer for
numerous bug reports and documentation help.

* general changes

  - config file parsing is 2x faster with 50K inboxes

  - deduplication ignores whitespace differences within address fields

  - "PRAGMA optimize" is now issued on commits for SQLite 3.18+

* public-inbox-extindex

  A new Xapian + SQLite index able to search across several inboxes.
  This may be configured to replace per-inbox Xapian DBs,
  (but not per-inbox SQLite indices) and speed up manifest.js.gz
  generation.

  See public-inbox-extindex-format(5) and
  public-inbox-extindex(1) manpages for more details.

  Using it with "--all" speeds up various multi-inbox operations in
  PublicInbox::WWW, public-inbox-nntpd, and public-inbox-imapd.

* read-only public-inbox-daemon (-httpd, -nntpd, -imapd):

  libgit2 may be used via Inline::C to avoid hitting system pipe
  and process limits.  See public-inbox-tuning(7) manpage
  for more details.

* various memory usage reductions and workarounds for leaks in
  Encode <3.15, these mainly affect PublicInbox::WWW

* public-inbox-nntpd

  - startup is 6x faster with 50K inboxes if using -extindex

* PublicInbox::WWW

  - mboxrd search results are returned in reverse Xapian docid order,
    so more recent results are more likely to show up first

  - d: and dt: search prefixes allow "approxidate" formats supported
    by "git log --since="

  - manifest.js.gz generation is ~25x faster with -extindex

  - minor navigation improvements in search results HTML page

* lei - local email interface

  An experimental, subject-to-change, likely-to-eat-your-mail tool for
  personal mail as well as interacting with public-inboxes on the local
  filesystem or over HTTP(S).  See lei(1), lei-overview(7), and other
  lei-* manpages for details.  This is only ready-to-use w.r.t. external
  public-inbox instances, but mail synchronization for personal mail
  remains clunky.

* public-inbox-index

  - non-strict (Subject-based) threading supports non-ASCII characters,
    reindexing is necessary for old messages with non-ASCII subjects.

  - --batch-size is now 8M on 64-bit systems for throughput improvements,
    higher values are still advised for more powerful hardware.

* public-inbox-watch

  - IMAP and NNTP code shared with lei, fixing an off-by-one error
    in IMAP synchronization for single-message IMAP folders.

  - \Deleted and \Draft messages ignored for IMAP, as they are for
    Maildir.

  - IMAP and NNTP connection establishment (including git-credential
    prompts) ordering is now tied to config file order.

Compatibility:

* Rollbacks all the way to public-inbox 1.2.0 remain supported

Internal changes

* public-inbox-index switched to new internal IPC code shared
  with lei

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.6.1
From: Eric Wong @ 2020-12-31 23:45 UTC (permalink / raw)
  To: meta

A small, bugfix release on top of 1.6.0 from September 2020.

Bug fixes:

* MIME header decoding no longer warns on undefined variables,
  with Perl <5.28.  Thanks to a bug report by Ali Alnubani.
  https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/

* Fixed a message threading bug thanks to a report from Kyle Meyer.
  "public-inbox-index --rethread --reindex" will be necessary
  in case of certain messages arrive out-of-order.
  Link: https://public-inbox.org/meta/87360nlc44.fsf@kyleam.com/

* WWW: per-inbox grokmirror manifests no longer return info
  for all inboxes, only the root /manifest.js.gz includes all
  inboxes.  This regression appeared in 1.6.

* public-inbox-mda matches List-Id headers insensitively,
  matching public-inbox-watch behavior.  Similarly, List-Id
  is always indexed lower-cased for boolean matches to avoid
  matching an incorrect term.

* Newsgroup and Path NNTP headers are now emitted in conformance
  with RFC 5536 3.1.[45].  Thanks to Andrey Melnikov for the report:
  https://public-inbox.org/meta/CA+PODjpUN5Q4gBFQhAzUNuMasVEdmp9f=8Uo0Ej0mFumdSwi4w@mail.gmail.com/

* Inotify fixes for public-inbox-imapd users relying on SIGHUP
  reloads and thousands of watches.

* Read-only daemon fixes around TLS and Linux <4.5 systems

Bugfixes with minor behavior changes:

* The X-Status mbox header is now excluded from imports,
  just like the Status: header has been for many years.
  They have no place in public archives and can be privacy
  concern for people sharing archives.

* WWW prevents deep-linking to attachments to limit abuse
  vectors.  Noticed by Leah Neukirchen:
  https://public-inbox.org/meta/87imagyap9.fsf@vuxu.org/

There are also several ocumentation fixes from Uwe Kleine-König
and Kyle Meyer.

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.6.0
From: Eric Wong @ 2020-09-16 20:03 UTC (permalink / raw)
  To: meta

A big release containing several performance optimizations, a
new anonymous IMAP server, and more.  It represents an
incremental improvement over 1.5 in several areas with more to
come in 1.7.

The read-only httpd and nntpd daemons no longer block the event
loop when retrieving blobs from git, making better use of SMP
systems while accomodating slow storage.

Indexing can be now be tuned to give somewhat usable performance
on HDD storage, though we can't defy the laws of physics, either.

* General changes:

  - ~/.cache/public-inbox/inline-c is automatically used for Inline::C
    if it exists.  PERL_INLINE_DIRECTORY in env remains supported
    and prioritized to support `nobody'-type users without HOME.

  - msgmap.sqlite3 uses journal_mode=TRUNCATE, matching over.sqlite3
    behavior for a minor reduction in VFS traffic

  - public-inbox-tuning(7) - new manpage containing pointers to
    various tuning options and tips for certain HW and OS setups.

  - Copy-on-write is disabled on BTRFS for new indices to avoid
    fragmentation.  See the new public-inbox-tuning(7) manpage.

  - message/{rfc822,news,global} attachments are decoded recursively
    and indexed for search.  Reindexing (see below) is required
    to ensure these attachments are indexed in old messages.

  - inbox.lock (v2) and ssoma.lock (v1) files are written to
    on message delivery (or spam removal) to wake up read-only
    daemons via inotify or kqueue.

  - `--help' switch supported by command-line tools

* Upgrading for new features in 1.6

  The ordering of these steps is only necessary if you intend to
  use some new features in 1.6.  Future releases may have
  different instructions (or be entirely transparent).

  0. install (use your OS package manager, or "make install")

  1. restart public-inbox-watch instances if you have any

  2. Optional: remove Plack::Middleware::Deflater if you're using
     a custom .psgi file for PublicInbox::WWW.  This only saves
     some memory and CPU cycles, and you may also skip this step
     if you expect to roll back to 1.5.0 for any reason.

  Steps 3a and 3b may happen in any order, 3b is optional
  and is only required to use new WWW and IMAP features.

  3a. restart existing read-only daemons if you have them
      (public-inbox-nntpd, public-inbox-httpd)

  3b. run "public-inbox-index -c --reindex --rethread --all"
      to reindex all configured inboxes

  4. configure and start the new public-inbox-imapd.  This
     requires reindexing in 3b, but there's no obligation to
     run an IMAP server, either.

* public-inbox-index

  There are several new options to improve usability on slow,
  rotational storage.

  - `--batch-size=BYTES' or publicinbox.indexBatchSize parameter
    to reduce frequency of random writes on HDDs

  - `--sequential-shard' or publicInbox.sequentialShard parameter
    to improve OS page cache utilization on HDDs.

  - `--no-fsync' when combined with Xapian 1.4+ can be used to
    speed up indexing on SSDs and small (default) `--batch-size'

  - `--rethread' option to go with `--reindex' (use sparringly,
    see manpage)

  - parallelize v2 updates by default, `--sequential-shard' and
    `-j0' is (once again) allowed to disable parallelization

  - (re-)indexing parallelizes blob reads from git

  - `--all' may be specified to index all configured inboxes

* public-inbox-learn

  - `rm' supports `--all' to remove from all configured inboxes

* public-inbox-imapd

  - new read-only IMAP daemon similar to public-inbox-nntpd
    `AUTH=ANONYMOUS' is supported, but any username and
    password for clients without `AUTH=ANONYMOUS' support.

* public-inbox-nntpd

  - blob reads from git are handled asynchronously

* public-inbox-httpd

  - Plack::Middleware::Deflater is no longer loaded by default
    when no .psgi file is specified; PublicInbox::WWW can rely
    on gzip for buffering (see below)

* PublicInbox::WWW

  - use consistent blank line around attachment links

  - Attachments in message/{rfc822,news,global} messages can be
    individually downloaded.  Downloading the entire message/rfc822
    file in full remains supported

  - $INBOX_DIR/description is treated as UTF-8

  - HTML, Atom, and text/plain responses are gzipped without
    relying on Plack::Middleware::Deflater

  - Multi-message endpoints (/t.mbox.gz, /T/, /t/, etc) are ~10% faster
    when running under public-inbox-httpd with asynchronous blob
    retrieval

  - mbox search results may now include all messages pertaining to that
    thread.  Needs `--reindex' mentioned above in
    `Upgrading for new features in 1.6'.

  - fix mbox.gz search results downloads for lynx users

  - small navigation tweaks, more prominent mirroring instructions

* public-inbox-watch

  - Linux::Inotify2 or IO::KQueue is used directly,
    Filesys::Notify::Simple is no longer required

  - NNTP groups and IMAP mailboxes may be watched in addition
    to Maildirs (lightly tested).

* Ongoing internal changes

  - reduce event loop hogging for many-inbox support

  - use more Perl v5.10-isms, future-proof against Perl 8

  - more consistent variable and field naming, improve internal
    documentation and comments

  - start supporting >=40 char git identifiers for SHA-256

  - test -httpd-specific code paths via Plack::Test::ExternalServer
    in addition to generic PSGI paths.

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.5.0
From: Eric Wong @ 2020-05-10  7:04 UTC (permalink / raw)
  To: meta

This release introduces a new pure-Perl lazy email parser,
PublicInbox::Eml, which uses roughly 10% less memory and
is up to 2x faster than Email::MIME.   This is a major
internal change

Limits commonly enforced by MTAs are also enforced in the
new parser, as messages may bypass MTA transports.

Email::MIME and other Email::* modules are no longer
dependencies nor used at all outside of maintainer validation
tests.

* public-inbox-index

  - `--max-size=SIZE' CLI switch and `publicinbox.indexMaxSize'
    config file option added to prevent indexing of overly
    large messages.

  - List-Id headers are indexed in new messages, old messages
    can be found after `--reindex'.

* public-inbox-watch

  - multiple values of `publicinbox.<name>.watchheader' are
    now supported, thanks to Kyle Meyer

  - List-Id headers are matched case-insensitively as specified
    by RFC 2919

* PublicInbox::WWW

  - $INBOX_DIR/description and $INBOX_DIR/cloneurl are not
    memoized if missing

  - improved display of threads, thanks to Kyle Meyer

  - search for List-Id is available via `l:' prefix if indexed

  - all encodings are preloaded at startup to reduce fragmentation

  - diffstat linkification and highlighting are stricter and
    less likely to linkify tables in cover letters

  - fix hunk header links to solver which were off-by-one line,
    thanks again to Kyle Meyer

Release tarball available for download over HTTPS or Tor .onion:

https://yhbt.net/public-inbox.git/snapshot/public-inbox-1.5.0.tar.gz
http://ou63pmih66umazou.onion/public-inbox.git/snapshot/public-inbox-1.5.0.tar.gz

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.4.0
From: Eric Wong @ 2020-04-17  8:48 UTC (permalink / raw)
  To: meta

This release focuses on reproducibility improvements and
bugfixes for corner-cases.  Busy instances of PublicInbox::WWW
may also notice memory usage reductions.

For rare messages lacking Date and/or Received headers, mirrors
now fall back to using the git author/commit times to reindex
them.  This ensures search and filtering queries behave
identically on mirrors as they do on the original machine.

"altid" SQLite dumps are now accessible to all over the WWW
interface via `POST /$INBOX/$ALTID.sql.gz'.

Busy instances of PublicInbox::WWW (whether via
public-inbox-httpd or another PSGI server) may notice
significant memory usage reductions from the single message
"permalink" lifetime optimization.  There also ongoing work to
improve memory lifetime management to reduce the potential for
memory fragmentation in daemons.

* general changes:

  - `include.*' directives in the public-inbox-config(5) file
    are now honored as documented in git-config(1),
    thanks to Andreas Rottmann.

  - `+0000' is assumed for dates missing TZ offsets;
    thanks to Leah Neukirchen for spotting this regression from
    v1.2.0.

  - `<' and `>' characters are dropped to avoid errors in git
    in addresses for git, thanks again to Leah for noticing
    this long-standing bug.

* PublicInbox::WWW:

  - memory reductions for message display and rendering
  - code preload improved to reduce memory fragmentation
  - remove redundant "a=" parameter in links to solver
  - escape '&' in hrefs properly
  - fix optional address obfuscation in search results
  - `POST /$INBOX/$ALTID.sql.gz' endpoint to retrieve SQLite dumps

* public-inbox-httpd + public-inbox-nntpd:

  - fix SIGUSR2 upgrade in worker-less instances (-W0)

* public-inbox-httpd:

  - fix RFC 7230 conformance when Content-Length and "chunked"
    are both specified

* public-inbox-index:

  - reproduce original date and time stamps in mirrors for messages
    lacking Date: and/or Received: headers

  - new `--compact' (or `-c') switch to perform the equivalent of
    public-inbox-compact(1) after indexing each inbox

* documentation:

  - add Documentation/technical/data_structures.txt for new hackers

* scripts/import_vger_from_mbox: (not really a production-level script)

  - fix ">From" unescaping thanks to a bug report from Kyle Meyer

Thanks to Andreas Rottmann, Leah Neukirchen and Kyle Meyer
for their contributions to this release.

Release tarball available for download over HTTPS or Tor .onion:

https://yhbt.net/public-inbox.git/snapshot/public-inbox-1.4.0.tar.gz
http://ou63pmih66umazou.onion/public-inbox.git/snapshot/public-inbox-1.4.0.tar.gz

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.3.0
From: Eric Wong @ 2020-02-10  5:52 UTC (permalink / raw)
  To: meta

Many internal improvements to improve the developer experience,
long-term maintainability, ease-of-installation and compatibility.
There are also several bugfixes.

Some of the internal improvements involve avoiding Perl startup
time in tests.  "make check" now runs about 50% faster than
before, and the new "make check-run" can be around 30% faster
than "make check" after being primed by "make check".

Most closures (anonymous subroutines) are purged from the
-nntpd, -httpd and WWW code paths to make checking for memory
leaks easier.

* documentation now builds on BSD make

* Date::Parse (TimeDate CPAN distribution) is now optional, allowing
  installation from OpenBSD systems via "pkg".

* the work-in-progress Xapian.pm SWIG bindings are now supported
  in addition to the traditional Search::Xapian XS bindings.
  Only the SWIG bindings are packaged for OpenBSD.

* Plack is optional for users who wish to avoid web-related components

* Filesys::Notify::Simple is optional for non-watch users
  (but Plack will still pull it in)

* improved internal error checking and reporting in numerous places

* fixed Perl 5.10.1 compatibility (tested with Devel::PatchPerl)

* IPC::Run and XML::Feed are no longer used in tests,
  though XML::TreePP becomes an optional test dependency.

* Email::Address::XS used if available (newer Email::MIME
  requires it), it should handle more corner cases.

* PublicInbox::WWW:
  - "nested" search results page now shows relevancy percentages
  - many solver bugs fixed
  - solver works on "-U0" patches using "git apply --unidiff-zero"
  - solver now compatible with git < v1.8.5 (but >= v1.8.0)
  - raw HTML no longer shown inline in multipart/alternative messages
    (v1.2.0 regression)
  - reduced memory usage for displaying multipart messages
  - static file responses support Last-Modified/If-Modified-Since
  - avoid trailing underlines in diffstat linkification
  - more consistent handling of messages without Subjects

* public-inbox-httpd / public-inbox-nntpd:
  - MSG_MORE used consistently in long responses
  - fixed IO::KQueue usage on *BSDs
  - listen sockets are closed immediately on graceful shutdown
  - missed signals avoided with signalfd or EVFILT_SIGNAL
  - Linux x32 ABI support

* public-inbox-nntpd:
  - Y2020 workaround for Time::Local

* public-inbox-watch:
  - avoid memory leak from cyclic reference on SIGHUP
  - fix documentation of publicinboxwatch.watchspam

* public-inbox-convert:
  - avoid article number jumps when converting indexed v1 inboxes

* public-inbox-compact / public-inbox-xcpdb:
  - concurrent invocations of -compact and -xcpdb commands,
    not just -mda, -watch, -learn, -purge

* examples/unsubscribe.milter:
  - support unique mailto: unsubscribe

Release tarball available for download at:

https://public-inbox.org/public-inbox.git/snapshot/public-inbox-1.3.0.tar.gz

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.2.0
From: Eric Wong @ 2019-11-03  3:12 UTC (permalink / raw)
  To: meta

* first non-pre/rc release with v2 format support for scalability.
  See public-inbox-v2-format(5) manpage for more details.

* new admin tools for v2 inboxes:
  - public-inbox-convert - converts v1 to v2 repo formats
  - public-inbox-compact - v2 convenience wrapper for xapian-compact(1)
  - public-inbox-purge - purges entire messages out of v2 history
  - public-inbox-edit - edits sensitive data out messages from v2 history
  - public-inbox-xcpdb - copydatabase(1) wrapper to upgrade Xapian formats
                         (e.g. from "chert" to "glass") and resharding
                         of v2 inboxes

* SQLite3 support decoupled from Xapian support, and Xapian DBs may be
  configured without phrase support to save space.  See "indexlevel" in
  public-inbox-config(5) manpage for more info.

* codebase now uses Perl 5.10.1+ features (e.g. "//")

* public-inbox-nntpd
  - support STARTTLS and NNTPS
  - support COMPRESS extension
  - fix several RFC3977 compliance bugs
  - improved interopability with picky clients such as leafnode
    and Alpine

* public-inbox-watch
  - support multiple spam training directories
  - support mapping multiple inboxes per Maildir
  - List-ID header support (see "listid" in public-inbox-config(5))

* public-inbox-mda
  - List-ID header support (see above)

* PublicInbox::WWW
  - grokmirror-compatible manifest.js.gz endpoint generation
  - user-configurable color support in $INBOX_URL/_/text/color/
  - BOFHs may set default colors via "publicinbox.css"
    (see public-inbox-config(5))
  - ability to map git code repositories and run cgit
    (see "coderepo" and "cgitrc" in public-inbox-config(5))
  - able to recreate blobs with coderepo associations and Xapian
  - search results may be reversed
  - reduce memory usage when rendering large threads
  - syntax highlighting for patches and blobs

* public-inbox-httpd / public-inbox-nntpd:
  - lower memory usage in C10K scenarios
  - buffers slow clients to filesystem (TMPDIR) instead of RAM
  - improved FreeBSD support
  - Danga::Socket is no longer a runtime dependency

* many documentation updates, new manpages for:
  - PublicInbox::SaPlugin::ListMirror
  - public-inbox-init
  - public-inbox-learn

* workaround memory leaks on Perl 5.16.3 (on CentOS/RHEL 7.x)

Thanks to Ali Alnubani, Alyssa Ross, Amitai Schleier, Dave Taht,
Dmitry Alexandrov, Eric W. Biederman, Jan Kiszka, Jonathan Corbet,
Kyle Meyer, Leah Neukirchen, Mateusz Łoskot, Nicolás Ojeda Bär,
SZEDER Gábor, Urs Janßen, Wang Kang, and edef for all their help,
bug reports, patches and suggestions.

Special thanks to Konstantin Ryabitsev and The Linux Foundation
for their sponsorship and support over the past two years.

https://public-inbox.org/releases/public-inbox-1.2.0.tar.gz

SHA256: dabc735a5cfe396f457ac721559de26ae38abbaaa74612eb786e9e2e1ca94269

  Chances are: You don't know me and never will.  Everybody else
  can verify the tarball and sign a reply saying they've
  verified it, instead.  The more who do this, the better, but
  don't trust the BOFH :P

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.1.0-pre1
From: Eric Wong @ 2018-05-09 20:23 UTC (permalink / raw)
  To: meta; +Cc: Konstantin Ryabitsev

Pre-release for v2 repository support.
Thanks to The Linux Foundation for supporting this work!

https://public-inbox.org/releases/public-inbox-1.1.0-pre1.tar.gz

SHA-256: d0023770a63ca109e6fe2c58b04c58987d4f81572ac69d18f95d6af0915fa009
(only intended to guard against accidental file corruption)

shortlog below:

Eric Wong (27):
      nntp: improve fairness during XOVER and similar commands
      nntp: do not drain rbuf if there is a command pending
      extmsg: use news.gmane.org for Message-ID lookups
      searchview: fix non-numeric comparison
      mbox: do not barf on queries which return no results
      nntp: allow and ignore empty commands
      ensure SQLite and Xapian files respect core.sharedRepository
      TODO: a few more updates
      filter/rubylang: do not set altid on spam training
      import: cleanup git cat-file processes when ->done
      disallow "\t" and "\n" in OVER headers
      searchidx: release lock again during v1 batch callback
      searchidx: remove leftover debugging code
      convert: copy description and git config from v1 repo
      view: untangle loop when showing message headers
      view: wrap To: and Cc: headers in HTML display
      view: drop redundant References: display code
      TODO: add EPOLLEXCLUSIVE item
      searchview: do not blindly append "l" parameter to URL
      search: avoid repeated mbox results from search
      msgmap: add limit to response for NNTP
      thread: prevent hidden threads in /$INBOX/ landing page
      thread: sort incoming messages by Date
      searchidx: preserve umask when starting/committing transactions
      scripts/import_slrnspool: support v2 repos
      scripts/import_slrnspool: cleanup progress messages
      public-inbox 1.1.0-pre1

Eric Wong (Contractor, The Linux Foundation) (239):
      AUTHORS: add The Linux Foundation
      watch_maildir: allow '-' in mail filename
      scripts/import_vger_from_mbox: relax From_ line match slightly
      import: stop writing legacy ssoma.index by default
      import: begin supporting this without ssoma.lock
      import: initial handling for v2
      t/import: test for last_object_id insertion
      content_id: add test case
      searchmsg: add mid_mime import for _extract_mid
      scripts/import_vger_from_mbox: support --dry-run option
      import: APIs to support v2 use
      search: free up 'Q' prefix for a real unique identifier
      searchidx: fix comment around next_thread_id
      address: extract more characters from email addresses
      import: pass "raw" dates to git-fast-import(1)
      scripts/import_vger_from_mbox: use v2 layout for import
      import: quiet down warnings from bogus From: lines
      import: allow the epoch (0s) as a valid time
      extmsg: fix broken Xapian MID lookup
      search: stop assuming Message-ID is unique
      www: stop assuming mainrepo == git_dir
      v2writable: initial cut for repo-rotation
      git: reload alternates file on missing blob
      v2: support Xapian + SQLite indexing
      import_vger_from_inbox: allow "-V" option
      import_vger_from_mbox: use PublicInbox::MIME and avoid clobbering
      v2: parallelize Xapian indexing
      v2writable: round-robin to partitions based on article number
      searchidxpart: increase pipe size for partitions
      v2writable: warn on duplicate Message-IDs
      searchidx: do not modify Xapian DB while iterating
      v2/ui: some hacky things to get the PSGI UI to show up
      v2/ui: retry DB reopens in a few more places
      v2writable: cleanup unused pipes in partitions
      searchidxpart: binmode
      use PublicInbox::MIME consistently
      searchidxpart: chomp line before splitting
      searchidx*: name child subprocesses
      searchidx: get rid of pointless index_blob wrapper
      view: remove X-PI-TS reference
      searchidxthread: load doc data for references
      searchidxpart: force integers into add_message
      search: reopen skeleton DB as well
      searchidx: index values in the threader
      search: use different Enquire object for skeleton queries
      rename SearchIdxThread to SearchIdxSkeleton
      v2writable: commit to skeleton via remote partitions
      searchidxskeleton: extra error checking
      searchidx: do not modify Xapian DB while iterating
      search: query_xover uses skeleton DB iff available
      v2/ui: get nntpd and init tests running on v2
      v2writable: delete ::Import obj when ->done
      search: remove informational "warning" message
      searchidx: add PID to error message when die-ing
      content_id: special treatment for Message-Id headers
      evcleanup: disable outside of daemon
      v2writable: deduplicate detection on add
      evcleanup: do not create event loop if nothing was registered
      mid: add `mids' and `references' methods for extraction
      content_id: use `mids' and `references' for MID extraction
      searchidx: use new `references' method for parsing References
      content_id: no need to be human-friendly
      v2writable: inject new Message-IDs on true duplicates
      search: revert to using 'Q' as a uniQue id per-Xapian conventions
      searchidx: support indexing multiple MIDs
      mid: be strict with References, but loose on Message-Id
      searchidx: avoid excessive XNQ indexing with diffs
      searchidxskeleton: add a note about locking
      v2writable: generated Message-ID goes first
      searchidx: use add_boolean_term for internal terms
      searchidx: add NNTP article number as a searchable term
      mid: truncate excessively long MIDs early
      nntp: use NNTP article numbers for lookups
      nntp: fix NEWNEWS command
      searchidx: store the primary MID in doc data for NNTP
      import: consolidate object info for v2 imports
      v2: avoid redundant/repeated configs for git partition repos
      INSTALL: document more optional dependencies
      search: favor skeleton DB for lookup_mail
      search: each_smsg_by_mid uses skeleton if available
      v2writable: remove unnecessary skeleton commit
      favor Received: date over Date: header globally
      import: fall back to Sender for extracting name and email
      scripts/import_vger_from_mbox: perform mboxrd or mboxo escaping
      v2writable: detect and use previous partition count
      extmsg: rework partial MID matching to favor current inbox
      extmsg: rework partial MID matching to favor current inbox
      content_id: use Sender header if From is not available
      v2writable: support "barrier" operation to avoid reforking
      use string ref for Email::Simple->new
      v2writable: remove unnecessary idx_init call
      searchidx: do not delete documents while iterating
      search: allow ->reopen to be chainable
      v2writable: implement remove correctly
      skeleton: barrier init requires a lock
      import: (v2) delete writes the blob into history in subdir
      import: (v2): write deletes to a separate '_' subdirectory
      import: implement barrier operation for v1 repos
      mid: mid_mime uses v2-compatible mids function
      watchmaildir: use content_digest to generate Message-Id
      import: force Message-ID generation for v1 here
      import: switch to URL-safe Base64 for Message-IDs
      v2writable: test for idempotent removals
      import: enable locking under v2
      index: s/GIT_DIR/REPO_DIR/
      Lock: new base class for writable lockers
      t/watch_maildir: note the reason for FIFO creation
      v2writable: ensure ->done is idempotent
      watchmaildir: support v2 repositories
      searchidxpart: s/barrier/remote_barrier/
      v2writable: allow disabling parallelization
      scripts/import_vger_from_mbox: filter out same headers as MDA
      v2writable: add DEBUG_DIFF env support
      v2writable: remove "resent" message for duplicate Message-IDs
      content_id: do not take Message-Id into account
      introduce InboxWritable class
      import: discard all the same headers as MDA
      InboxWritable: add mbox/maildir parsing + import logic
      use both Date: and Received: times
      msgmap: add tmp_clone to create an anonymous copy
      fix syntax warnings
      v2writable: support reindexing Xapian
      t/altid.t: extra tests for mid_set
      v2writable: add NNTP article number regeneration support
      v2writable: clarify header cleanups
      v2writable: DEBUG_DIFF respects $TMPDIR
      feed: $INBOX/new.atom endpoint supports v2 inboxes
      import: consolidate mid prepend logic, here
      www: $MESSAGE_ID/raw endpoint supports "duplicates"
      search: reopen DB if each_smsg_by_mid fails
      t/psgi_v2: minimal test for Atom feed and t.mbox.gz
      feed: fix new.html for v2
      view: permalink (per-message) view shows multiple messages
      searchidx: warn about vivifying multiple ghosts
      v2writable: warn on unseen deleted files
      www: get rid of unnecessary 'inbox' name reference
      searchview: remove unnecessary imports from MID module
      view: depend on SearchMsg for Message-ID
      http: fix modification of read-only value
      githttpbackend: avoid infinite loop on generic PSGI servers
      www: support cloning individual v2 git partitions
      http: fix modification of read-only value
      githttpbackend: avoid infinite loop on generic PSGI servers
      www: remove unnecessary ghost checks
      v2writable: append, instead of prepending generated Message-ID
      lookup by Message-ID favors the "primary" one
      www: fix attachment downloads for conflicted Message-IDs
      searchmsg: document why we store To: and Cc: for NNTP
      public-inbox-convert: tool for converting old to new inboxes
      v2writable: support purging messages from git entirely
      search: cleanup uniqueness checking
      search: get rid of most lookup_* subroutines
      search: move find_doc_ids to searchidx
      v2writable: cleanup: get rid of unused fields
      mbox: avoid extracting Message-ID for linkification
      www: cleanup expensive fallback for legacy URLs
      view: get rid of some unnecessary imports
      search: retry_reopen on first_smsg_by_mid
      import: run_die supports redirects as spawn does
      v2writable: initializing an existing inbox is idempotent
      public-inbox-compact: new tool for driving xapian-compact
      mda: support v2 inboxes
      search: warn on reopens and die on total failure
      v2writable: allow gaps in git partitions
      v2writable: convert some fatal reindex errors to warnings
      wwwstream: flesh out clone instructions for v2
      v2writable: go backwards through alternate Message-IDs
      view: speed up homepage loading time with date clamp
      view: drop load_results
      feed: optimize query for feeds, too
      msgtime: parse 3-digit years properly
      convert: avoid redundant "done\n" statement for fast-import
      search: move permissions handling to InboxWritable
      t/v2writable: use simplify permissions reading
      v2: respect core.sharedRepository in git configs
      searchidx: correct warning for over-vivification
      v2: one file, really
      v2writable: fix parallel termination
      truncate Message-IDs and References consistently
      scripts/import_vger_from_mbox: set address properly
      search: reduce columns stored in Xapian
      replace Xapian skeleton with SQLite overview DB
      v2writable: simplify barrier vs checkpoints
      t/over: test empty Subject: line matching
      www: rework query responses to avoid COUNT in SQLite
      over: speedup get_thread by avoiding JOIN
      nntp: fix NEWNEWS command
      t/thread-all.t: modernize test to support modern inboxes
      rename+rewrite test using Benchmark module
      nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster
      view: avoid offset during pagination
      mbox: remove remaining OFFSET usage in SQLite
      msgmap: replace id_batch with ids_after
      nntp: simplify the long_response API
      searchidx: ensure duplicated Message-IDs can be linked together
      init: s/GIT_DIR/REPO_DIR/ in usage
      import: rewrite less history during purge
      v2: support incremental indexing + purge
      v2writable: do not modify DBs while iterating for ->remove
      v2writable: recount partitions after acquiring lock
      searchmsg: remove unused `tid' and `path' methods
      search: remove unnecessary OP_AND of query
      mbox: do not sort search results
      searchview: minor cleanup
      support altid mechanism for v2
      compact: better handling of over.sqlite3* files
      v2writable: remove redundant remove from Over DB
      v2writable: allow tracking parallel versions
      v2writable: refer to git each repository as "epoch"
      over: use only supported and safe SQLite APIs
      search: index and allow searching by date-time
      altid: fix miscopied field name
      nntp: set Xref across multiple inboxes
      www: favor reading more from SQLite, and less from Xapian
      ensure Xapian and SQLite are still optional for v1 tests
      psgi: ensure /$INBOX/$MESSAGE_ID/T/ endpoint is chronological
      over: avoid excessive SELECT
      over: remove forked subprocess
      v2writable: reduce barriers
      index: allow specifying --jobs=0 to disable multiprocess
      convert: support converting with altid defined
      store less data in the Xapian document
      msgmap: speed up minmax with separate queries
      feed: respect feedmax, again
      v1: remove articles from overview DB
      compact: do not merge v2 repos by default
      v2writable: reduce partititions by one
      search: preserve References in Xapian smsg for x=t view
      v2: generate better Message-IDs for duplicates
      v2: improve deduplication checks
      import: cat_blob drops leading 'From ' lines like Inbox
      searchidx: regenerate and avoid article number gaps on full index
      extmsg: remove expensive git path checks
      use %H consistently to disable abbreviations
      searchidx: increase term positions for all text terms
      searchidx: revert default BATCH_BYTES to 1_000_000
      Merge remote-tracking branch 'origin/master' into v2
      fix tests to run without Xapian installed
      extmsg: use Xapian only for partial matches

Jonathan Corbet (3):
      Don't use LIMIT in UPDATE statements
      Update the installation instructions with Fedora package names
      Allow specification of the number of search results to return
-- 
git clone https://public-inbox.org/ public-inbox
(working on a homepage... sorta :)

^ permalink raw reply

* [ANNOUNCE] public-inbox 1.0.0
From: Eric Wong @ 2018-02-08  2:33 UTC (permalink / raw)
  To: meta

After some 3.5 odd years of working on this, I suppose now is
as good a time as any to tar this up and call it 1.0.0.

The TODO list is still very long and there'll be some new
development in coming weeks :>

So, here you have a release:

        https://public-inbox.org/releases/public-inbox-1.0.0.tar.gz

Checksums, mainly as a safeguard against accidental file corruption:

SHA-256 4a08569f3d99310f713bb32bec0aa4819d6b41871e0421ec4eec0657a5582216
	(in other words, don't trust me; instead read the code :>)

^ permalink raw reply

	git clone http://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/public-inbox.git
	git clone https://public-inbox.org/public-inbox.git