user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: meta@public-inbox.org
Subject: Re: make menuconfig interface for lei / grok-pull
Date: Wed, 15 Sep 2021 23:06:05 +0000	[thread overview]
Message-ID: <20210915230605.GA22173@dcvr> (raw)
In-Reply-To: <YUJm8F5E174ZSGgC@bombadil.infradead.org>

Luis Chamberlain <mcgrof@kernel.org> wrote:
> Greetings,
> 
> I unsubscribed to most mailing lists thanks to public-inbox. Thanks
> to public-inbox my inbox went on quite a bit of diet.
> 
> Now, the way I work though is... I like to hop onto different systems
> every now and then, and not rely on one central place for everything.
> This is because when I've done that before I get outages / network
> outages / etc.
> 
> So I often have to start from scratch.

Same here, this goes for project code repos, as well...

At the same time, I don't want too much synchronization since
the systems I hop on to vary hugely in storage capacity.

<snip replied parts>

> lei q --only=linux-fsdevel <search-term>
> 
> but its not clear if a date is supported as a search term.

All the search prefixes documented in
https://public-inbox.org/meta/_/text/help/ work for lei.

dt: (date-time), d: (date (mairix-compatible) have been there a while,
rt: (Received-time) is new.

They gained support for git approxidate recently, so "rt:last.week.."
gives you all the messages that were received from last week til now.
Anything w/o spaces which git understands should work.

> Is my intended use case better with grok-pull or is there
> hopes I can end up using lei for all what I have intentions for?

I think introducing support for "lei update-external [--all]"
could be useful combined with my previous note about
add-external.

> My vision is, kernel deveoper would 'make menuconfig' and select
> a few mailing lists they want to import, then when one is selected
> new options appear so you can pick the starting year from where
> you want the import to occur for. Then for each mailing list there
> is also a refresh thing, so that we can grok-pull differently
> for each mailing list, which will be handled by systemd timers
> and a service for each. The reason for differnt times for refresh
> is updating linux-kernel is not a priority to me, but linux-fsdevel
> and linux-block are, so I want linux-fsdevel and linux-block to
> refresh once every 10 minutes, while linux-kernel can update... 
> I don't know once every 8 hours. Anyway, this should all be
> configurable. And my hope is that developers would not have to
> know what's going on behind the scenes. All they need to know
> is that their ~Mail/linux/ directory will eventually be stuffed
> with Maildir directories for the mailing lists they enabled and
> they'll be updated as often as specified in their configuration.

Interesting...  I suppose it could work.  I don't know much
about curses or interactive UIs, but "make menuconfig" isn't
a bad UI :)  I've been halfway envisioning something else
that would use curses (more on that some other time).

Does lore.kernel.org run public-inbox-imapd?
Well, at least nntp works for lore:

   # populate ~/.cache/lei/uri_folder/.sqlite3
   lei ls-mail-source nntp://nntp.lore.kernel.org

   # if you're using bash:
   . contrib/completion/lei-completion.bash

   lei import <TAB><TAB>

lei import will download the entire newsgroup, which may get
huge, unfortunately.  It should support ranges (e.g.
nntp://example.com/inbox.foo.example/900-1000) but the URI
package doesn't support it, yet...

public-inbox-imapd IMAP mailbox slices are limited to 50K
messages since typical MUAs can't handle much more, so it's
easier to import just the most recent .[0-9]+ slices.

If you have Tor (on port 9050), libio-socket-socks (IO::Socket::Socks)

  # long Tor v3 URL :<
  url=imap://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion

  # configure IMAP to use Tor on port 9050:
  lei config imap.$url.proxy socks5h://0:9050

  # git 2.26+ can use a wildcard for all .onions:
  # lei config 'imap.imap://*.onion.proxy' socks5h://0:9050

  # I also suggest enabling compression for public-inbox-imapd
  # (some IMAP servers can't handle it, our -imapd does):
  lei config imap.$url.compress 1

  # Same as above with NNTP:
  lei ls-mail-source $url
  lei import <TAB-COMPLETE>

For public-inbox IMAP folders, make sure the \.[0-9]+ is at the
end of the mailbox to get the slice you want (there isn't
recursive import, yet).

Right now, "lei import" can be run periodically.  The plan is to
support IMAP IDLE and auto-imports.  For read/write IMAP
folders, there'd be auto keyword (FLAG) synchronization
(currently manual: lei export-kw imaps://...)
It mostly works for Maildirs at the moment...

...Which reminds me, I need to add an imapserver config variable
so the WWW interface can advertise IMAP servers like it does
NNTP servers

  parent reply	other threads:[~2021-09-15 23:06 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-15 21:34 Luis Chamberlain
2021-09-15 21:44 ` Eric Wong
2021-09-15 21:57   ` Luis Chamberlain
2021-09-15 23:12     ` Eric Wong
2021-09-15 22:36   ` Kyle Meyer
2021-09-15 23:31     ` Eric Wong
2021-09-16  2:59   ` Eric Wong
2021-09-15 23:06 ` Eric Wong [this message]
2021-09-16 17:43   ` Konstantin Ryabitsev
2021-09-16 20:04     ` IMAP vs NNTP [was: make menuconfig interface for lei / grok-pull] Eric Wong
2021-09-19 21:21   ` make menuconfig interface for lei / grok-pull Eric Wong
2021-09-16 17:38 ` Konstantin Ryabitsev
2021-09-16 23:54   ` Luis Chamberlain
2021-09-16 21:09 ` lei import on epochs [was: make menuconfig interface for lei / grok-pull] Eric Wong
2021-09-16 23:53   ` Luis Chamberlain
2021-09-17  0:22     ` Eric Wong
2021-09-17  0:48       ` Luis Chamberlain

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210915230605.GA22173@dcvr \
    --to=e@80x24.org \
    --cc=mcgrof@kernel.org \
    --cc=meta@public-inbox.org \
    --subject='Re: make menuconfig interface for lei / grok-pull' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).