Re: publicinbox watch path globbing

["Robin H. Johnson" <robbat2@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 3424 bytes --]

On Mon, Nov 20, 2023 at 12:10:01AM +0000, Eric Wong wrote:
> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> > Hi!
> > 
> > Writing to see about work in converting Gentoo's (now-broken) other
> > archives web interface over into using public-inbox instead.
> > 
> > This is the first of a few questions/bumps along the way.
> > 
> > For historical reasons on the scaling side, the archive maildirs are
> > stored by date:
> > watch = maildir:$REDACTED/$LISTNAME/.200001/
> > watch = maildir:$REDACTED/$LISTNAME/.200102/
> > watch = maildir:$REDACTED/$LISTNAME/.YYYYMM/
> > watch = maildir:$REDACTED/$LISTNAME/.202311/
> > etc.
> > (over time, directories are moved to stable read-only storage)
> 
> Is there any reason to expect new messages to appear the /.2000??/
> and other old directories?
> 
> IOW, if somebody with a broken clock sends a message from a past
> year/month in the Date: header, does it end up in an old bucket
> or the current one?
The date is based on arrival time at the archive ingest.

For some of the very old lists, we do have a list of message-ids that we
know existed but aren't captured in the archive, and those mails have
been added to the old locations if they are ever found (maybe once a
year).

> 
> If your old buckets are frozen, lei in public-inbox.git should be
> able to start them off with:
> 
> 	for d in $REDACTED/$LISTNAME/.??????
> 	do
> 		lei convert -o v2:/path/to/inbox-$LISTNAME maildir:$d
> 	done
> 	lei daemon-kill # optional, stops lei-daemon when done
> 
> And then you'd only have to watch the latest maildir.
Any concerns during the month rollover period?
E.g. making sure the 202310 & 202311 are both watched right as time
increments from October to November, because the archive ingest is
likely to write to 202311, but it's possible that public-inbox is still
run for the last few new messages in 202310 yet?

> > While I could generate the config file, I'm wondering about better
> > solution, to allow globbing the path.
> 
> I wanted to have recursive watches at some point but never got
> around to it.  So I guess something like this could work recursively:
> 	watchglob = maildir:$REDACTED/$LISTNAME/**
> 
> > I tried to locate a single place in the codebase where this would be
> > applied, but it's not clear enough to me if there's a single place that
> > it can easily modified.
> 
> The `new' sub in lib/PublicInbox/Watch.pm sets up maildirs/imap/nntp
> 
> The glob2re function is better nowadays in public-inbox.git,
> and the mdre regexp will probably needs to be updated when it sees
> a new maildir...
Thanks. I'd want to explicitly scope the glob to the dates.
The spam processing has been to move spam to .spam.YYYYMM.

> > If there's a consistent place, I think the cleanest syntax that doesn't
> > break existing consumers would be something like this:
> > [publicinbox "$LISTNAME"]
> > watch = maildirglob:$REDACTED/$LISTNAME/.19????/
> > watch = maildirglob:$REDACTED/$LISTNAME/.20????/
> 
> I think `watchglob = maildir:...' is preferable since I don't
> want maildirglob: to be confused as a type.
Agreed, I see concerns there.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]