Re: Mirroring mailing lists directly

[Eric Wong <e@80x24.org>]

Mateusz Loskot <mateusz@loskot.net> wrote:
> All the details you gave about configs will be useful, I'm sure.
> It's just that I'm not there yet. I'm still missing some
> aspects of the bigger picture of mirroring a mailing list.
> 
> I've realised, I'm missing an outline of the overall procedure:
> 
> 0. Install public-inbox and its dependencies
> 1. Find mailing list to mirror e.g. public@list.org
> 2. Get a new e-mail address e.g. mirror@user.org
> 3. Subscribe to list@host.org with mirror@user.org

I typically use my normal address and not a list-specific one.

> 4. Set up (to run manually or daemon) offlineimap to
>    sync from mirror@user.org to local Maildir

You need to initialize the inbox repo:

  public-inbox-init -V2 NAME /path/to/inbox HTTP_URL list@host.org

(I still need to write the manpage for -init :x)

> 5. Set up public-inbox-watch
> 6. Set up public-inbox-httpd to publish via HTTP

Correct.

> 7. Set up public-inbox-index to enable search

Running -index separately is not necessary if you used "-V2" for
public-inbox-init right now.  V2 repos are significantly more
scalable when you have hundreds of thousands of messages, but
require SQLite and Xapian.

(*) Xapian dependency can be removed, though

For future updates to public-inbox code itself, you may need to
run public-inbox-index if the Xapian schema changes
incompatibly.

> 8. Set up git daemon to allow `git clone` access to mirrored archives.

public-inbox-httpd already supports smart HTTP clone.
git-daemon is only necessary for git://, which seems to be
falling out of favor given the popularity of HTTP/HTTPS.

> Is this plan correct, complete or am I missing anything?

Looks close to me.

I also suggest running public-inbox-nntpd in addition to -httpd
for NNTP users.  It shares a common core with -nntpd and I have
plans for a combined server to minimize memory use.

> Do I need to bother with public-inbox-watch's bidirectional sync?

There is no bidirectional sync in -watch.  Perhaps you mean
offlineimap?


You may also want to prevent your Maildir and IMAP folder from
growing too large.  You can setup a cronjob to remove old mails
from the Maildir; which causes offlineimap bidirectional sync to
remove the old messages from IMAP, too.

The following example removes mails older than 7 days:

   cd /path/to/Maildir &&
   find new cur -ctime +7 -type f -print0 | xargs -0 rm -f

> The docs of the public-inbox toolset are clear and they seem complete.
> As explained above, I am missing a basic "where do I start" to
> create mirror, especially for someone who is not a sysadmin,
> like myself :)

Yes, public-inbox-overview.pod probably needs to be updated.

> Once I get to the 4. and later points of the plan above,
> I will get back to your other suggestions on the implementation
> details.

Alright, please do :)