From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 21EC81F8C8; Wed, 15 Sep 2021 23:06:05 +0000 (UTC) Date: Wed, 15 Sep 2021 23:06:05 +0000 From: Eric Wong To: Luis Chamberlain Cc: meta@public-inbox.org Subject: Re: make menuconfig interface for lei / grok-pull Message-ID: <20210915230605.GA22173@dcvr> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: List-Id: Luis Chamberlain wrote: > Greetings, > > I unsubscribed to most mailing lists thanks to public-inbox. Thanks > to public-inbox my inbox went on quite a bit of diet. > > Now, the way I work though is... I like to hop onto different systems > every now and then, and not rely on one central place for everything. > This is because when I've done that before I get outages / network > outages / etc. > > So I often have to start from scratch. Same here, this goes for project code repos, as well... At the same time, I don't want too much synchronization since the systems I hop on to vary hugely in storage capacity. > lei q --only=linux-fsdevel > > but its not clear if a date is supported as a search term. All the search prefixes documented in https://public-inbox.org/meta/_/text/help/ work for lei. dt: (date-time), d: (date (mairix-compatible) have been there a while, rt: (Received-time) is new. They gained support for git approxidate recently, so "rt:last.week.." gives you all the messages that were received from last week til now. Anything w/o spaces which git understands should work. > Is my intended use case better with grok-pull or is there > hopes I can end up using lei for all what I have intentions for? I think introducing support for "lei update-external [--all]" could be useful combined with my previous note about add-external. > My vision is, kernel deveoper would 'make menuconfig' and select > a few mailing lists they want to import, then when one is selected > new options appear so you can pick the starting year from where > you want the import to occur for. Then for each mailing list there > is also a refresh thing, so that we can grok-pull differently > for each mailing list, which will be handled by systemd timers > and a service for each. The reason for differnt times for refresh > is updating linux-kernel is not a priority to me, but linux-fsdevel > and linux-block are, so I want linux-fsdevel and linux-block to > refresh once every 10 minutes, while linux-kernel can update... > I don't know once every 8 hours. Anyway, this should all be > configurable. And my hope is that developers would not have to > know what's going on behind the scenes. All they need to know > is that their ~Mail/linux/ directory will eventually be stuffed > with Maildir directories for the mailing lists they enabled and > they'll be updated as often as specified in their configuration. Interesting... I suppose it could work. I don't know much about curses or interactive UIs, but "make menuconfig" isn't a bad UI :) I've been halfway envisioning something else that would use curses (more on that some other time). Does lore.kernel.org run public-inbox-imapd? Well, at least nntp works for lore: # populate ~/.cache/lei/uri_folder/.sqlite3 lei ls-mail-source nntp://nntp.lore.kernel.org # if you're using bash: . contrib/completion/lei-completion.bash lei import lei import will download the entire newsgroup, which may get huge, unfortunately. It should support ranges (e.g. nntp://example.com/inbox.foo.example/900-1000) but the URI package doesn't support it, yet... public-inbox-imapd IMAP mailbox slices are limited to 50K messages since typical MUAs can't handle much more, so it's easier to import just the most recent .[0-9]+ slices. If you have Tor (on port 9050), libio-socket-socks (IO::Socket::Socks) # long Tor v3 URL :< url=imap://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion # configure IMAP to use Tor on port 9050: lei config imap.$url.proxy socks5h://0:9050 # git 2.26+ can use a wildcard for all .onions: # lei config 'imap.imap://*.onion.proxy' socks5h://0:9050 # I also suggest enabling compression for public-inbox-imapd # (some IMAP servers can't handle it, our -imapd does): lei config imap.$url.compress 1 # Same as above with NNTP: lei ls-mail-source $url lei import For public-inbox IMAP folders, make sure the \.[0-9]+ is at the end of the mailbox to get the slice you want (there isn't recursive import, yet). Right now, "lei import" can be run periodically. The plan is to support IMAP IDLE and auto-imports. For read/write IMAP folders, there'd be auto keyword (FLAG) synchronization (currently manual: lei export-kw imaps://...) It mostly works for Maildirs at the moment... ...Which reminds me, I need to add an imapserver config variable so the WWW interface can advertise IMAP servers like it does NNTP servers