user/dev discussion of public-inbox itself
 help / color / mirror / Atom feed
* [ANNOUNCE] public-inbox 1.6.0
@ 2020-09-16 20:03 Eric Wong
  2020-09-19 20:01 ` Leah Neukirchen
  0 siblings, 1 reply; 4+ messages in thread
From: Eric Wong @ 2020-09-16 20:03 UTC (permalink / raw)
  To: meta

A big release containing several performance optimizations, a
new anonymous IMAP server, and more.  It represents an
incremental improvement over 1.5 in several areas with more to
come in 1.7.

The read-only httpd and nntpd daemons no longer block the event
loop when retrieving blobs from git, making better use of SMP
systems while accomodating slow storage.

Indexing can be now be tuned to give somewhat usable performance
on HDD storage, though we can't defy the laws of physics, either.

* General changes:

  - ~/.cache/public-inbox/inline-c is automatically used for Inline::C
    if it exists.  PERL_INLINE_DIRECTORY in env remains supported
    and prioritized to support `nobody'-type users without HOME.

  - msgmap.sqlite3 uses journal_mode=TRUNCATE, matching over.sqlite3
    behavior for a minor reduction in VFS traffic

  - public-inbox-tuning(7) - new manpage containing pointers to
    various tuning options and tips for certain HW and OS setups.

  - Copy-on-write is disabled on BTRFS for new indices to avoid
    fragmentation.  See the new public-inbox-tuning(7) manpage.

  - message/{rfc822,news,global} attachments are decoded recursively
    and indexed for search.  Reindexing (see below) is required
    to ensure these attachments are indexed in old messages.

  - inbox.lock (v2) and ssoma.lock (v1) files are written to
    on message delivery (or spam removal) to wake up read-only
    daemons via inotify or kqueue.

  - `--help' switch supported by command-line tools

* Upgrading for new features in 1.6

  The ordering of these steps is only necessary if you intend to
  use some new features in 1.6.  Future releases may have
  different instructions (or be entirely transparent).

  0. install (use your OS package manager, or "make install")

  1. restart public-inbox-watch instances if you have any

  2. Optional: remove Plack::Middleware::Deflater if you're using
     a custom .psgi file for PublicInbox::WWW.  This only saves
     some memory and CPU cycles, and you may also skip this step
     if you expect to roll back to 1.5.0 for any reason.

  Steps 3a and 3b may happen in any order, 3b is optional
  and is only required to use new WWW and IMAP features.

  3a. restart existing read-only daemons if you have them
      (public-inbox-nntpd, public-inbox-httpd)

  3b. run "public-inbox-index -c --reindex --rethread --all"
      to reindex all configured inboxes

  4. configure and start the new public-inbox-imapd.  This
     requires reindexing in 3b, but there's no obligation to
     run an IMAP server, either.

* public-inbox-index

  There are several new options to improve usability on slow,
  rotational storage.

  - `--batch-size=BYTES' or publicinbox.indexBatchSize parameter
    to reduce frequency of random writes on HDDs

  - `--sequential-shard' or publicInbox.sequentialShard parameter
    to improve OS page cache utilization on HDDs.

  - `--no-fsync' when combined with Xapian 1.4+ can be used to
    speed up indexing on SSDs and small (default) `--batch-size'

  - `--rethread' option to go with `--reindex' (use sparringly,
    see manpage)

  - parallelize v2 updates by default, `--sequential-shard' and
    `-j0' is (once again) allowed to disable parallelization

  - (re-)indexing parallelizes blob reads from git

  - `--all' may be specified to index all configured inboxes

* public-inbox-learn

  - `rm' supports `--all' to remove from all configured inboxes

* public-inbox-imapd

  - new read-only IMAP daemon similar to public-inbox-nntpd
    `AUTH=ANONYMOUS' is supported, but any username and
    password for clients without `AUTH=ANONYMOUS' support.

* public-inbox-nntpd

  - blob reads from git are handled asynchronously

* public-inbox-httpd

  - Plack::Middleware::Deflater is no longer loaded by default
    when no .psgi file is specified; PublicInbox::WWW can rely
    on gzip for buffering (see below)

* PublicInbox::WWW

  - use consistent blank line around attachment links

  - Attachments in message/{rfc822,news,global} messages can be
    individually downloaded.  Downloading the entire message/rfc822
    file in full remains supported

  - $INBOX_DIR/description is treated as UTF-8

  - HTML, Atom, and text/plain responses are gzipped without
    relying on Plack::Middleware::Deflater

  - Multi-message endpoints (/t.mbox.gz, /T/, /t/, etc) are ~10% faster
    when running under public-inbox-httpd with asynchronous blob
    retrieval

  - mbox search results may now include all messages pertaining to that
    thread.  Needs `--reindex' mentioned above in
    `Upgrading for new features in 1.6'.

  - fix mbox.gz search results downloads for lynx users

  - small navigation tweaks, more prominent mirroring instructions

* public-inbox-watch

  - Linux::Inotify2 or IO::KQueue is used directly,
    Filesys::Notify::Simple is no longer required

  - NNTP groups and IMAP mailboxes may be watched in addition
    to Maildirs (lightly tested).

* Ongoing internal changes

  - reduce event loop hogging for many-inbox support

  - use more Perl v5.10-isms, future-proof against Perl 8

  - more consistent variable and field naming, improve internal
    documentation and comments

  - start supporting >=40 char git identifiers for SHA-256

  - test -httpd-specific code paths via Plack::Test::ExternalServer
    in addition to generic PSGI paths.

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [ANNOUNCE] public-inbox 1.6.0
  2020-09-16 20:03 [ANNOUNCE] public-inbox 1.6.0 Eric Wong
@ 2020-09-19 20:01 ` Leah Neukirchen
  2020-09-19 21:17   ` Eric Wong
  0 siblings, 1 reply; 4+ messages in thread
From: Leah Neukirchen @ 2020-09-19 20:01 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Hi,

thanks for the release!

Eric Wong <e@80x24.org> writes:

> * Upgrading for new features in 1.6
>
>   The ordering of these steps is only necessary if you intend to
>   use some new features in 1.6.  Future releases may have
>   different instructions (or be entirely transparent).
>
>   0. install (use your OS package manager, or "make install")
>
>   1. restart public-inbox-watch instances if you have any
>
>   2. Optional: remove Plack::Middleware::Deflater if you're using
>      a custom .psgi file for PublicInbox::WWW.  This only saves
>      some memory and CPU cycles, and you may also skip this step
>      if you expect to roll back to 1.5.0 for any reason.
>
>   Steps 3a and 3b may happen in any order, 3b is optional
>   and is only required to use new WWW and IMAP features.
>
>   3a. restart existing read-only daemons if you have them
>       (public-inbox-nntpd, public-inbox-httpd)
>
>   3b. run "public-inbox-index -c --reindex --rethread --all"
>       to reindex all configured inboxes
>
>   4. configure and start the new public-inbox-imapd.  This
>      requires reindexing in 3b, but there's no obligation to
>      run an IMAP server, either.

I did all these steps in this order, NNTP works fine but IMAP shows
all folders as empty.  Any ideas how to debug this?

thx,
-- 
Leah Neukirchen  <leah@vuxu.org>  https://leahneukirchen.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [ANNOUNCE] public-inbox 1.6.0
  2020-09-19 20:01 ` Leah Neukirchen
@ 2020-09-19 21:17   ` Eric Wong
  2020-09-19 21:24     ` Leah Neukirchen
  0 siblings, 1 reply; 4+ messages in thread
From: Eric Wong @ 2020-09-19 21:17 UTC (permalink / raw)
  To: Leah Neukirchen; +Cc: meta

Leah Neukirchen <leah@vuxu.org> wrote:
> Hi,
> 
> thanks for the release!

You're welcome!

> > * Upgrading for new features in 1.6

<snip>

> I did all these steps in this order, NNTP works fine but IMAP shows
> all folders as empty.  Any ideas how to debug this?

Any chance you're hitting "$NEWSGROUP" and not "$NEWSGROUP.0"?
(or ".1", ".2" ...)?

I had to split the mailboxes into slices (".0"-".$N") to deal
with client-side limitations; so "$NEWSGROUP" is just an empty
folder with sub-folders containing messages.

Otherwise, stdout should have a trace of commands issued by the
client, and strace/truss/tcpdump/etc should show more...

It's been a while since I've really touched this part of the
IMAP code (and it's been a long year :x), but I don't think I
ever saw something like this while working on it.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [ANNOUNCE] public-inbox 1.6.0
  2020-09-19 21:17   ` Eric Wong
@ 2020-09-19 21:24     ` Leah Neukirchen
  0 siblings, 0 replies; 4+ messages in thread
From: Leah Neukirchen @ 2020-09-19 21:24 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong <e@80x24.org> writes:

> Leah Neukirchen <leah@vuxu.org> wrote:
>> Hi,
>> 
>> thanks for the release!
>
> You're welcome!
>
>> > * Upgrading for new features in 1.6
>
> <snip>
>
>> I did all these steps in this order, NNTP works fine but IMAP shows
>> all folders as empty.  Any ideas how to debug this?
>
> Any chance you're hitting "$NEWSGROUP" and not "$NEWSGROUP.0"?
> (or ".1", ".2" ...)?
>
> I had to split the mailboxes into slices (".0"-".$N") to deal
> with client-side limitations; so "$NEWSGROUP" is just an empty
> folder with sub-folders containing messages.

Of course, that was it.  Works fine now. :)

-- 
Leah Neukirchen  <leah@vuxu.org>  https://leahneukirchen.org

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2020-09-19 21:24 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-16 20:03 [ANNOUNCE] public-inbox 1.6.0 Eric Wong
2020-09-19 20:01 ` Leah Neukirchen
2020-09-19 21:17   ` Eric Wong
2020-09-19 21:24     ` Leah Neukirchen

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ http://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git