user/dev discussion of public-inbox itself
 help / color / mirror / Atom feed
From: Konstantin Ryabitsev <>
Subject: Subscribing to public-inbox lists using grokmirror + procmail
Date: Wed, 7 Oct 2020 17:42:16 -0400
Message-ID: <20201007214216.ztns57g22rowlccu@chatter.i7.local> (raw)

Hi, all:

I needed a way to pipe public-inbox straight into 
without having to manage yet another list subscription with postfix pipe 
integration. Then I realized that it's generally useful as a way to 
deliver straight from public-inbox archives into local inboxes -- 
similar to how l2md does it, but with some extra perks:

- we use manifest.js.gz provided by public-inbox to poll the remote 
  manifest instead of running "git remote update" in each mirrored list
- we automatically recognize when remote epochs roll over
- if configured with `shallow=yes`, we can automatically re-shallow 
  local git copies after each successful delivery, so they don't take up 
  any space

You can read the details here:

it's also available below.

Hope this is useful to others!


# Subscribing to lore lists with grokmirror

You can have mailing lists delivered right into your inbox
straight from the git archive (in fact, this will work for any public-inbox
server, not just for It's efficient and (optionally)
preserves a full copy of entire list archives on your system -- should you
wish to keep them.

Note: this requires grokmirror-2.0.2+, as earlier versions do not come with
the grok-pi-piper utility.

## Installing grokmirror-2.0

Easiest is to install from pip:

    pip install --user grokmirror~=2.0.2

You may have grokmirror available from your distro packages, too, but make
sure it's version 2.0.2 or above.

## Installing procmail

Procmail should be available with your distribution, so install it like any
other package.

## Configuring procmail

Procmail configuration can be a book in itself, but if you just want to have
messages delivered into your inbox, all you have to do is create a
`~/.procmailrc` with the following contents:


    # Don't deliver duplicates sent to multiple lists
    :0 Wh: .msgid.lock
    | formail -D 8192 .msgid.cache

If your mailbox is not in `~/Mail`, then you should adjust the above

## Configuring grokmirror

Create a `~/.config/lore.conf` with the following contents. We'll use three
lists as examples: git, linux-hardening, and linux-doc, but you'll obviously
want to use the lists you care about. You can see which lists are available
from, or the exact git repositories on

    toplevel = ~/.local/share/grokmirror/lore
    log = ${toplevel}/grokmirror.log

    site =
    manifest =

    post_update_hook = ~/.local/bin/grok-pi-piper -c ~/.config/pi-piper.conf
    refresh = 300
    include = /git/*

The above assumes that you installed grokmirror with `pip install --user`. Now
make the toplevel directory:

    $ mkdir -p ~/.local/share/grokmirror/lore

## Configuring pi-piper

The last step is to create `~/.config/pi-piper.conf`:

    pipe = /usr/bin/procmail
    shallow = yes

The important bit here is `shallow = yes`. Public-inbox stores every mail
message as a separate commit, so once a message is piped to procmail and
delivered, we usually don't care about keeping a copy of that commit any more.
If you set `shallow = yes`, pi-piper will prune all but the last successfully
processed commit out of your local git copy by turning those repos into
shallow git repositories. This helps to greatly save disk space, especially
for large archives.

If you *do* want to keep full archives, then don't set `shallow`. You can
change your mind at any time by running `git fetch _grokmirror master
--unshallow` in each underlying git repository (you can find them in

You can also specify this option per list:

    pipe = /usr/bin/procmail

    shallow = yes

## Running grok-pull

You can now run grok-pull to get the initial repo copies. Note, that during
the first run grokmirror will perform full clones even if you specified
`shallow = yes` in the pi-piper config, so it may take some time for large
archives like those for the git list. However, once the pi-piper hook runs,
they will be repacked to almost nothing. Future versions of grokmirror may
become smarter about this and perform shallow clones from the beginning.

During the initial pi-piper run, there will be no mail delivered, as it will
just perform initial setup and make a note where the HEAD is pointing. If you
run grok-pull again, two things may happen:

1. There will be no changes and grok-pull will exit right away
2. If there are changes, they will be fetched and the hook will deliver them
   to procmail (and to your inbox)

## Running in the background

You can run grok-pull in the background, where it will check for updates as
frequently as the `refresh` setting says (300 seconds in the example above).
You can either background it "the old way":

    grok-pull -o -c ~/.config/lore.conf &

Or the new way, using a systemd user service:

    $ cat .config/systemd/user/grok-pull@.service

    Description=Grok-pull service for %I

    ExecStart=%h/.local/bin/grok-pull -o -c %h/.config/%i.conf


    $ systemctl --user enable grok-pull@lore
    $ systemctl --user start grok-pull@lore

If you make changes to `~/.config/lore.conf`, for example to add new lists,
you will need to restart the service:

    $ systemctl --user restart grok-pull@lore

## Combining with mbsync

You can totally combine this with mbsync and deliver into the same local
inbox. As a perk, any messages injected from grokmirror will be uploaded to
your remote imap mailbox. See this post from mcgrof about configuring mbsync:


## Troubles

Email if you have any trouble getting the above to
work. The grok-pi-piper utility is fairly new, so it's entirely possible that
it's full of bugs.

                 reply	other threads:[~2020-10-07 21:42 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201007214216.ztns57g22rowlccu@chatter.i7.local \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ \
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
 note: .onion URLs require Tor:

code repositories for the project(s) associated with this inbox:

AGPL code for this site: git clone