up and running, integrated with exim4 mta

up and running, integrated with exim4 mta

[lkcl]

hi eric we have things running, hooray, i thought you might appreciate
it is a little different
http://inbox.libre-riscv.org/libre-riscv-dev/new.html

http://bugs.libre-riscv.org/show_bug.cgi?id=181

here are attachments for *direct* integration into exim4. rather than
store in Maildir we have a use subscribed to the mailman2 list, where
in the exim4 configs we specifically look out for that user and
run a pipe to public-inbox-mda

a section to disable spam and also adding the listid to the config is
critical otherwise public-inbox-mda fails silently.

regarding the httpd it was a nuisance to set up as there was no error
reporting.  i tracked down in the exammples how to add 3 lines to do
logging, this really should be there by default at least commented out.

also being able to specify the URL for git repos would be handy because
we already run gitweb and git-daemon, the URL is different ane would
need source hacking to specify an alternative.

with the exim4 integration we do not need the cron job. also i set p
a sysvinit darmon 
argh no editing using console mail program sorry
set up a sysvinit daemon to start httpd, no god no way i let systemd run.
will send that example based on start-stop-daemon later.

very much aporeciated even the existence of pbi as it is one of the last
pieces which allows our project to be entirely offline developed via
git backed storage.

l.

Re: up and running, integrated with exim4 mta

[Eric Wong]

lkcl <lkcl@libre-riscv.org> wrote:
> hi eric we have things running, hooray, i thought you might appreciate
> it is a little different
> http://inbox.libre-riscv.org/libre-riscv-dev/new.html

Good to know!  Btw, if you have DBD::SQLite (and optionally,
Search::Xapian), you can run `public-inbox-index $INBOX_DIR'
to get message threading and search enabled.

> http://bugs.libre-riscv.org/show_bug.cgi?id=181
> 
> here are attachments for *direct* integration into exim4. rather than
> store in Maildir we have a use subscribed to the mailman2 list, where
> in the exim4 configs we specifically look out for that user and
> run a pipe to public-inbox-mda

Cool, hope that helps other folks using exim4.

> a section to disable spam and also adding the listid to the config is
> critical otherwise public-inbox-mda fails silently.

There's also '--no-precheck' on the command-line to disable
spam filtering, but yes, listid is useful.

> regarding the httpd it was a nuisance to set up as there was no error
> reporting.  i tracked down in the exammples how to add 3 lines to do
> logging, this really should be there by default at least commented out.

Huh?  Do you mean AccessLog middleware or something else?
Errors should be logged to stderr.

AccessLog isn't enabled by default since either folks don't care
and don't want to waste space; or they're very picky about what
format their logs are in and any default format wouldn't work for
them.

> also being able to specify the URL for git repos would be handy because
> we already run gitweb and git-daemon, the URL is different ane would
> need source hacking to specify an alternative.

Huh?  Do you mean $INBOX_DIR/cloneurl for the inbox?  That's
gitweb-compatible for v1 inboxes, at least.

Or do you mean for coderepos? There's `coderepo.<nick>.cgitUrl'
(see public-inbox-config(5)) right now, but perhaps
`coderepo.<nick>.gitwebUrl' should also be supported.

> with the exim4 integration we do not need the cron job. also i set p
> a sysvinit darmon 
> argh no editing using console mail program sorry
> set up a sysvinit daemon to start httpd, no god no way i let systemd run.
> will send that example based on start-stop-daemon later.
> 
> very much aporeciated even the existence of pbi as it is one of the last
> pieces which allows our project to be entirely offline developed via
> git backed storage.

Thanks :>

Re: up and running, integrated with exim4 mta

[lkcl]

On Thu, Mar 19, 2020 at 3:06 AM Eric Wong <e@yhbt.net> wrote:
>
> lkcl <lkcl@libre-riscv.org> wrote:
> > hi eric we have things running, hooray, i thought you might appreciate
> > it is a little different
> > http://inbox.libre-riscv.org/libre-riscv-dev/new.html
>
> Good to know!  Btw, if you have DBD::SQLite (and optionally,
> Search::Xapian), you can run `public-inbox-index $INBOX_DIR'
> to get message threading and search enabled.

ah excellent and that just kicks it in, and mda picks it up
(continuously) from there.  excellent, that looks better:
http://inbox.libre-riscv.org/libre-riscv-dev

> > http://bugs.libre-riscv.org/show_bug.cgi?id=181
> >
> > here are attachments for *direct* integration into exim4. rather than
> > store in Maildir we have a use subscribed to the mailman2 list, where
> > in the exim4 configs we specifically look out for that user and
> > run a pipe to public-inbox-mda
>
> Cool, hope that helps other folks using exim4.

it would do just as well without mailman2, by just manually naming the
"list" as the incoming email in the exim4 configs.  we chose that
incoming email to be a *subscriber* email of the mailman2 list, so
mailman2 gets it first and *then* passes on.

actually i think for convenience (and the archives here) i'll
cut/paste the relevant section:

exim4 - public-inbox setup
 setup exim4 as usual in Debian 10
  use split conf
 setup mailman as usual in Debian 10

exim4: routers - please note order of routers is important
 otherwise mail would reach one and not other router
890_local_pi
900_exim4-config_local_user

conf.d/router/890_local_pi
---
# Router for public-inbox

public_inbox:
    debug_print = "R: public_inbox for $local_part@$domain"
    driver = accept
#   require_files = PI_HOME/check.pck
    domains = test.local
    check_local_user
    local_parts = test
    transport = public_inbox_transport
---

conf.d/transport/45_local_pi
---
# Public-inbox transport

public_inbox_transport:
    debug_print = "T: publix_inbox for $local_part@$domain"
    driver = pipe
    command = /usr/local/bin/public-inbox-mda
    user = $local_part
    group = $local_part
    delivery_date_add
    envelope_to_add
    return_path_add
    umask = 066
---

i set that umask to 066 because (A) the archives are public and (B) if
you don't, various other programs can't see the result (gitweb for
example).

"local_parts=test" is what needs replacing with
"the_subscriber_or_list" before the @.
"domains = test.local" is what you replace with the list domain.

there's a way to get multiple email addresses "listened" to (multiple
subscribers), however if using the mailman2 setup, and the same
"subscriber" is subscribed to all the lists then all mailman messages
would come to the one email address.

if however you wanted to do away with mailman2, and use exim4 as *the*
front for diverting to public-inbox, and you want multiple separate
addresses, they'd go into "local_parts = ...." using exim4
colon-separated lists, i forget exactly how that's done, it's in the
bugreport above.

> > a section to disable spam and also adding the listid to the config is
> > critical otherwise public-inbox-mda fails silently.
>
> There's also '--no-precheck' on the command-line to disable
> spam filtering, but yes, listid is useful.

veera couldn't get it working without

> > regarding the httpd it was a nuisance to set up as there was no error
> > reporting.  i tracked down in the exammples how to add 3 lines to do
> > logging, this really should be there by default at least commented out.
>
> Huh?  Do you mean AccessLog middleware or something else?

yes AccessLog::Timed.  it was in the cgi script (commented-out), i was
having a hell of a job working out the difference between "is this
failing at nginx proxy_pass, is this failing at fcgid, is this failing
at public-inbox-httpd, is the url correct on nginx proxy_pass" and
because of the complete lack of access log indication that anything
was going through, let alone what url was being passed from nginx
(when i finally got that running), it was much more challenging than
it should have been.

> Errors should be logged to stderr.
>
> AccessLog isn't enabled by default since either folks don't care
> and don't want to waste space;

that's what logrotate is for :)  and, also, i have fail2ban monitoring
log files, with a rather draconian "zero tolerance" policy for anyone
trying to access anything with ".php" let alone
"wordpressadminlogin.php".  you run a server, so you know how mental
things get.

> or they're very picky about what
> format their logs are in and any default format wouldn't work for
> them.

it's better than a kick in the teeth.  commented-out - in
public-inbox-httpd - would at least give them the hint.  ideally it
should be command-line-enabled (and the format specifier allowed to be
overridden, that's how it works in nginx / apache2).

> > also being able to specify the URL for git repos would be handy because
> > we already run gitweb and git-daemon, the URL is different ane would
> > need source hacking to specify an alternative.
>
> Huh?  Do you mean $INBOX_DIR/cloneurl for the inbox?  That's
> gitweb-compatible for v1 inboxes, at least.

yes the clone url.  it... works!  however i have one via gitweb as well.

> Or do you mean for coderepos? There's `coderepo.<nick>.cgitUrl'
> (see public-inbox-config(5)) right now, but perhaps
> `coderepo.<nick>.gitwebUrl' should also be supported.

https://git.libre-riscv.org/?p=libre-riscv-dev.git;a=summary

i'd like that URL (or its git clone equivalent for gitweb) to be in
the list of mirrors.  i looked up those two config options, they don't
seem to exist (not exactly by those names).

l.

Re: up and running, integrated with exim4 mta

[Eric Wong]

lkcl <lkcl@libre-riscv.org> wrote:
> On Thu, Mar 19, 2020 at 3:06 AM Eric Wong <e@yhbt.net> wrote:
 
> > > a section to disable spam and also adding the listid to the config is
> > > critical otherwise public-inbox-mda fails silently.
> >
> > There's also '--no-precheck' on the command-line to disable
> > spam filtering, but yes, listid is useful.
> 
> veera couldn't get it working without

Oh, nevermind :>  Well, --no-precheck is a different set of
checks, actually (it's in the -mda manpage).

> > > regarding the httpd it was a nuisance to set up as there was no error
> > > reporting.  i tracked down in the exammples how to add 3 lines to do
> > > logging, this really should be there by default at least commented out.
> >
> > Huh?  Do you mean AccessLog middleware or something else?
> 
> yes AccessLog::Timed.  it was in the cgi script (commented-out), i was
> having a hell of a job working out the difference between "is this
> failing at nginx proxy_pass, is this failing at fcgid, is this failing
> at public-inbox-httpd, is the url correct on nginx proxy_pass" and
> because of the complete lack of access log indication that anything
> was going through, let alone what url was being passed from nginx
> (when i finally got that running), it was much more challenging than
> it should have been.
> 
> > Errors should be logged to stderr.
> >
> > AccessLog isn't enabled by default since either folks don't care
> > and don't want to waste space;
> 
> that's what logrotate is for :)  and, also, i have fail2ban monitoring
> log files, with a rather draconian "zero tolerance" policy for anyone
> trying to access anything with ".php" let alone
> "wordpressadminlogin.php".  you run a server, so you know how mental
> things get.

I discourage fail2ban for read-only stuff over HTTP(S) or
NNTP(S) entirely.  It's too easy to end up with collateral
damage with shared IPs coming from public WiFi, Tor exits,
etc...

I don't want to burden people with setting up logrotate or
anything else, either.  I'm actually considering making Plack an
optional requirement for the -httpd and WWW stuff since it's a
lot to install and we don't use much of it.

> > or they're very picky about what
> > format their logs are in and any default format wouldn't work for
> > them.
> 
> it's better than a kick in the teeth.  commented-out - in
> public-inbox-httpd - would at least give them the hint.  ideally it
> should be command-line-enabled (and the format specifier allowed to be
> overridden, that's how it works in nginx / apache2).

Since public-inbox-httpd is a normal PSGI server(*), it accepts
.psgi files.  I've just sent a patch for examples to hopefully
make them more obvious:

	https://public-inbox.org/meta/20200319235550.10401-1-e@yhbt.net/

I'm avoiding multiple ways of enabling the same options.
Having command-line functionality overlap with what .psgi
files provide would increase the support and documentation
burden.

> > > also being able to specify the URL for git repos would be handy because
> > > we already run gitweb and git-daemon, the URL is different ane would
> > > need source hacking to specify an alternative.
> >
> > Huh?  Do you mean $INBOX_DIR/cloneurl for the inbox?  That's
> > gitweb-compatible for v1 inboxes, at least.
> 
> yes the clone url.  it... works!  however i have one via gitweb as well.

Viewing git repos used for v1 or v2 inboxes would via gitweb or
cgit would be kinda pointless.  v1 inboxes could get really
expensive for gitweb as trees grow, too (that's why v2 was
created).

> > Or do you mean for coderepos? There's `coderepo.<nick>.cgitUrl'
> > (see public-inbox-config(5)) right now, but perhaps
> > `coderepo.<nick>.gitwebUrl' should also be supported.
> 
> https://git.libre-riscv.org/?p=libre-riscv-dev.git;a=summary
> 
> i'd like that URL (or its git clone equivalent for gitweb) to be in
> the list of mirrors.  i looked up those two config options, they don't
> seem to exist (not exactly by those names).

Oh, "coderepo" stuff is only for code and totally optional.

I've also started calling inbox-specific repos "inboxdir"
instead of "mainrepo" a few months back, since v2 format inboxes
are made up of multiple git repos.

Maybe the example config snippet for git@vger.kernel.org
archives showing the relationship between the git.git source
code mirror and inboxdir hopefully gives a better idea:

	https://public-inbox.org/git/_/text/config/raw