user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <>
To: Alyssa Ross <>
Subject: Re: public-inbox-init with minimal info
Date: Sat, 5 Oct 2019 05:14:34 +0000	[thread overview]
Message-ID: <20191005051434.GA23947@dcvr> (raw)
In-Reply-To: <>

Alyssa Ross <> wrote:
> >> Since the V2 initialization isn't encapsulated in one easy command, I'm
> >> wondering what the best way to accomplish initialization without writing
> >> a config file or asking for unnecessary information is.  I could just
> >> run public-inbox-init with PI_CONFIG=/dev/null, but then it's still not
> >> clear to me what information about the mailbox the script requires to be
> >> able to initialize the mailbox.  Looking at the code, I see that at
> >> least the primary address is passed to PublicInbox::Inbox, but I'm not
> >> sure what that would actually be used for inside the inbox.
> >
> > That address is used for making commits using the
> > public-inbox-{learn,edit,purge} commands, and also matching
> > with public-inbox-mda for delivery.
> I'm guessing those read from the config file, though?  I'm trying to
> figure out what configuration is stored in the inbox directory as
> opposed to the config file.

Yes, they all read from the config file because they need to
operate on inboxes.   public-inbox-{edit,purge,index,xcpdb,compact}
can also operate on inboxes outside of the config file, but -mda
and -learn needs the config.

There isn't any config stored in the inbox directories
themselves, although there's some metadata in SQLite for
incremental indexing and indexlevel for Xapian.

> > You want to provide that inbox during the post-install?
> >
> > I'm not sure if I understand what's going on (but it's been a
> > long day :x).  I'm not sure if providing an inbox on package
> > installation is necessary or even a good thing...
> >
> > Using git itself as an analogy: I wouldn't expect installing git
> > to auto-generate an empty git repo for me.  So same with
> > public-inbox, and stuff like cgit/gitweb...
> I agree -- creating an inbox on package installation would be a terrible
> idea, and is not what I'm proposing to do.

Good to know :>

> Some background: in the Nix ecosystem, we have a package repository,
> Nixpkgs.  These packages are fairly typical except for unusual paths
> (containing checksums of the "inputs" -- think dependencies -- of the
> package) and the functional language they're written in.  We also have
> NixOS, which is a GNU/Linux distribution, that uses those packages but
> also adds the concept of "modules", written in the same language as the
> packages.  These modules do things like configuring the users on your
> system, setting up config files, etc.  The idea being that you can
> generate a whole Linux system from pure (in the functional programming
> sense) configuration, reproducibly, and have the system stuff be read
> only at runtime to the maximum extent possible.  I'm working on both a
> package _and_ a module for public-inbox.  The package will do exactly
> what you'd expect a package to do, but the module will let you express
> global and per-inbox configuration in the Nix language, generate a
> read-only public-inbox config file and systemd units from those, and, at
> boot time or configuration change time, initialize the inboxes defined
> by the user.

Thanks for that clarification, NixOS modules are a new concept
to me.

I'm not sure how the public-inbox config file can/should remain
read-only.  It's analogous to a config file for cgit or gitweb,
so maybe modules for those can offer some inspiration...

> As an example, Tor in NixOS looks like this:
>     { ... }:
>     {
>       services.tor.enable = true;
>       services.tor.hiddenServices = [
>         {
>           name = "public-inbox";
>           map = [ { port = 80; destination = 8000; } ];
>           version = 3;
>         };
>       ];
>     };
> This will generate a static tor.service and tor config file -- we do as
> much as possible staticly and purely because then we know that if a
> configuration is rolled back, we can remove the service etc.  For state,
> however, like the hidden service private key (in this case -- I could
> have used a static one here if I'd wanted), it will be generated either
> at boot time or in Tor's ExecStartPre.  This is the same mechanism I
> would use to run public-inbox-init.

So that means a new tor process is spawned for every hidden
service?  (instead of one tor process running multiple
services).  It works, but it's not memory-efficient (but
could be more secure if tor has bugs).

Like gitweb and cgit, public-inbox-{httpd,nntpd} usually expects
to serve multiple inboxes off one instance.  It's possible to
start an independent -httpd/-nntpd instance for every inbox,
each with it's own config, but that's not efficient at all.

Both the WWW and NNTP code are able to scan Message-IDs across
multiple inboxes in the same process in case of cross-posts
between related inboxes.  At some point, I think it'll be useful
to define groups/relationships between inboxes for the WWW UI.

Likewise, public-inbox-watch would be expected to watch multiple
Maildirs and write to multiple inboxes, if needed.  Maybe
modules for mail downloading/notification tools such as
offlineimap/mbsync(isync) could serve as inspiration, there.

I tried to keep most of the daemon-specific config knobs in the
command-line, so it goes into .service files, which can be
read-only (well, unless somebody wants to change worker
processes via "-W $NUM").

But yeah, packaging services/modules for different systems and
use-cases is hard, everybody seems to do it differently and
want different things.  So I'm really not sure how packaging
a module would work, it seems like that's something each user
would want to manage on their own.

  reply	other threads:[~2019-10-05  5:14 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-03 11:16 public-inbox-init with minimal info Alyssa Ross
2019-10-04  2:45 ` Eric Wong
2019-10-04 11:18   ` Alyssa Ross
2019-10-05  5:14     ` Eric Wong [this message]
2019-10-05 13:05       ` Alyssa Ross
2019-10-05 19:58         ` Eric Wong
2019-10-06  9:52           ` Alyssa Ross
2019-10-06 12:01             ` Eric Wong
2019-10-07 20:52               ` Alyssa Ross
2019-10-08  7:11                 ` Eric Wong
2019-10-09 12:09                   ` Alyssa Ross
2019-10-10  8:19                     ` Eric Wong
2019-10-16 10:04                       ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191005051434.GA23947@dcvr \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).