From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 453751F4BD; Sat, 5 Oct 2019 05:14:34 +0000 (UTC) Date: Sat, 5 Oct 2019 05:14:34 +0000 From: Eric Wong To: Alyssa Ross Cc: meta@public-inbox.org Subject: Re: public-inbox-init with minimal info Message-ID: <20191005051434.GA23947@dcvr> References: <87bluyhyfi.fsf@alyssa.is> <20191004024559.GA24741@dcvr> <878sq0iwt2.fsf@alyssa.is> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <878sq0iwt2.fsf@alyssa.is> List-Id: Alyssa Ross wrote: > >> Since the V2 initialization isn't encapsulated in one easy command, I'm > >> wondering what the best way to accomplish initialization without writing > >> a config file or asking for unnecessary information is. I could just > >> run public-inbox-init with PI_CONFIG=/dev/null, but then it's still not > >> clear to me what information about the mailbox the script requires to be > >> able to initialize the mailbox. Looking at the code, I see that at > >> least the primary address is passed to PublicInbox::Inbox, but I'm not > >> sure what that would actually be used for inside the inbox. > > > > That address is used for making commits using the > > public-inbox-{learn,edit,purge} commands, and also matching > > with public-inbox-mda for delivery. > > I'm guessing those read from the config file, though? I'm trying to > figure out what configuration is stored in the inbox directory as > opposed to the config file. Yes, they all read from the config file because they need to operate on inboxes. public-inbox-{edit,purge,index,xcpdb,compact} can also operate on inboxes outside of the config file, but -mda and -learn needs the config. There isn't any config stored in the inbox directories themselves, although there's some metadata in SQLite for incremental indexing and indexlevel for Xapian. > > You want to provide that inbox during the post-install? > > > > I'm not sure if I understand what's going on (but it's been a > > long day :x). I'm not sure if providing an inbox on package > > installation is necessary or even a good thing... > > > > Using git itself as an analogy: I wouldn't expect installing git > > to auto-generate an empty git repo for me. So same with > > public-inbox, and stuff like cgit/gitweb... > > I agree -- creating an inbox on package installation would be a terrible > idea, and is not what I'm proposing to do. Good to know :> > Some background: in the Nix ecosystem, we have a package repository, > Nixpkgs. These packages are fairly typical except for unusual paths > (containing checksums of the "inputs" -- think dependencies -- of the > package) and the functional language they're written in. We also have > NixOS, which is a GNU/Linux distribution, that uses those packages but > also adds the concept of "modules", written in the same language as the > packages. These modules do things like configuring the users on your > system, setting up config files, etc. The idea being that you can > generate a whole Linux system from pure (in the functional programming > sense) configuration, reproducibly, and have the system stuff be read > only at runtime to the maximum extent possible. I'm working on both a > package _and_ a module for public-inbox. The package will do exactly > what you'd expect a package to do, but the module will let you express > global and per-inbox configuration in the Nix language, generate a > read-only public-inbox config file and systemd units from those, and, at > boot time or configuration change time, initialize the inboxes defined > by the user. Thanks for that clarification, NixOS modules are a new concept to me. I'm not sure how the public-inbox config file can/should remain read-only. It's analogous to a config file for cgit or gitweb, so maybe modules for those can offer some inspiration... > As an example, Tor in NixOS looks like this: > > { ... }: > > { > services.tor.enable = true; > services.tor.hiddenServices = [ > { > name = "public-inbox"; > map = [ { port = 80; destination = 8000; } ]; > version = 3; > }; > ]; > }; > > This will generate a static tor.service and tor config file -- we do as > much as possible staticly and purely because then we know that if a > configuration is rolled back, we can remove the service etc. For state, > however, like the hidden service private key (in this case -- I could > have used a static one here if I'd wanted), it will be generated either > at boot time or in Tor's ExecStartPre. This is the same mechanism I > would use to run public-inbox-init. So that means a new tor process is spawned for every hidden service? (instead of one tor process running multiple services). It works, but it's not memory-efficient (but could be more secure if tor has bugs). Like gitweb and cgit, public-inbox-{httpd,nntpd} usually expects to serve multiple inboxes off one instance. It's possible to start an independent -httpd/-nntpd instance for every inbox, each with it's own config, but that's not efficient at all. Both the WWW and NNTP code are able to scan Message-IDs across multiple inboxes in the same process in case of cross-posts between related inboxes. At some point, I think it'll be useful to define groups/relationships between inboxes for the WWW UI. Likewise, public-inbox-watch would be expected to watch multiple Maildirs and write to multiple inboxes, if needed. Maybe modules for mail downloading/notification tools such as offlineimap/mbsync(isync) could serve as inspiration, there. I tried to keep most of the daemon-specific config knobs in the command-line, so it goes into .service files, which can be read-only (well, unless somebody wants to change worker processes via "-W $NUM"). But yeah, packaging services/modules for different systems and use-cases is hard, everybody seems to do it differently and want different things. So I'm really not sure how packaging a module would work, it seems like that's something each user would want to manage on their own.