Re: make menuconfig interface for lei / grok-pull

[Eric Wong <e@80x24.org>]

Luis Chamberlain <mcgrof@kernel.org> wrote:
> Greetings,
> 
> I unsubscribed to most mailing lists thanks to public-inbox. Thanks
> to public-inbox my inbox went on quite a bit of diet.
> 
> Now, the way I work though is... I like to hop onto different systems
> every now and then, and not rely on one central place for everything.
> This is because when I've done that before I get outages / network
> outages / etc.
> 
> So I often have to start from scratch.

Same here, this goes for project code repos, as well...

At the same time, I don't want too much synchronization since
the systems I hop on to vary hugely in storage capacity.

<snip replied parts>

> lei q --only=linux-fsdevel <search-term>
> 
> but its not clear if a date is supported as a search term.

All the search prefixes documented in
https://public-inbox.org/meta/_/text/help/ work for lei.

dt: (date-time), d: (date (mairix-compatible) have been there a while,
rt: (Received-time) is new.

They gained support for git approxidate recently, so "rt:last.week.."
gives you all the messages that were received from last week til now.
Anything w/o spaces which git understands should work.

> Is my intended use case better with grok-pull or is there
> hopes I can end up using lei for all what I have intentions for?

I think introducing support for "lei update-external [--all]"
could be useful combined with my previous note about
add-external.

> My vision is, kernel deveoper would 'make menuconfig' and select
> a few mailing lists they want to import, then when one is selected
> new options appear so you can pick the starting year from where
> you want the import to occur for. Then for each mailing list there
> is also a refresh thing, so that we can grok-pull differently
> for each mailing list, which will be handled by systemd timers
> and a service for each. The reason for differnt times for refresh
> is updating linux-kernel is not a priority to me, but linux-fsdevel
> and linux-block are, so I want linux-fsdevel and linux-block to
> refresh once every 10 minutes, while linux-kernel can update... 
> I don't know once every 8 hours. Anyway, this should all be
> configurable. And my hope is that developers would not have to
> know what's going on behind the scenes. All they need to know
> is that their ~Mail/linux/ directory will eventually be stuffed
> with Maildir directories for the mailing lists they enabled and
> they'll be updated as often as specified in their configuration.

Interesting...  I suppose it could work.  I don't know much
about curses or interactive UIs, but "make menuconfig" isn't
a bad UI :)  I've been halfway envisioning something else
that would use curses (more on that some other time).

Does lore.kernel.org run public-inbox-imapd?
Well, at least nntp works for lore:

   # populate ~/.cache/lei/uri_folder/.sqlite3
   lei ls-mail-source nntp://nntp.lore.kernel.org

   # if you're using bash:
   . contrib/completion/lei-completion.bash

   lei import <TAB><TAB>

lei import will download the entire newsgroup, which may get
huge, unfortunately.  It should support ranges (e.g.
nntp://example.com/inbox.foo.example/900-1000) but the URI
package doesn't support it, yet...

public-inbox-imapd IMAP mailbox slices are limited to 50K
messages since typical MUAs can't handle much more, so it's
easier to import just the most recent .[0-9]+ slices.

If you have Tor (on port 9050), libio-socket-socks (IO::Socket::Socks)

  # long Tor v3 URL :<
  url=imap://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion

  # configure IMAP to use Tor on port 9050:
  lei config imap.$url.proxy socks5h://0:9050

  # git 2.26+ can use a wildcard for all .onions:
  # lei config 'imap.imap://*.onion.proxy' socks5h://0:9050

  # I also suggest enabling compression for public-inbox-imapd
  # (some IMAP servers can't handle it, our -imapd does):
  lei config imap.$url.compress 1

  # Same as above with NNTP:
  lei ls-mail-source $url
  lei import <TAB-COMPLETE>

For public-inbox IMAP folders, make sure the \.[0-9]+ is at the
end of the mailbox to get the slice you want (there isn't
recursive import, yet).

Right now, "lei import" can be run periodically.  The plan is to
support IMAP IDLE and auto-imports.  For read/write IMAP
folders, there'd be auto keyword (FLAG) synchronization
(currently manual: lei export-kw imaps://...)
It mostly works for Maildirs at the moment...

...Which reminds me, I need to add an imapserver config variable
so the WWW interface can advertise IMAP servers like it does
NNTP servers