lei stores mails in wrong folder

lei stores mails in wrong folder

[Leon Romanovsky]

Hi,

I have a strange behavior for newly installed lei ("dnf install lei" on FC41).
All new messages are stored in "cur" Maildir folder and not in "new" as I would expect.
Current behavior causes to weird situation where all messages are getting "O"  (old) tag.

➜  ~ lei q -I https://lore.kernel.org/all/ -o ~/mail/lore/rdma --threads --dedupe=mid '(l:linux-rdma.vger.kernel.org) AND rt:1.week.ago..' 
# /home/leonro/.config/lei/config created
# /home/leonro/.local/share/lei/store 15/15
# /usr/bin/curl -gSf -s -d '' https://lore.kernel.org/all/?x=m&t=1&q=(l%3Alinux-rdma.vger.kernel.org)+AND+rt%3A1740421535..
# https://lore.kernel.org/all/ 186/?
# https://lore.kernel.org/all/ 404/?
# https://lore.kernel.org/all/ 571/571
# 488 written to /home/leonro/mail/lore/rdma/ (586 matches, 155 duplicates)
➜  ~ ls ~/mail/lore/rdma/new 
➜  ~ ls ~/mail/lore/rdma/cur 
00c34ae956857960504f10d4a2d401fdc2d7b4c9:2,       450d677dfc827f61e3d0f807ae9e9950a6c91f1a:2,      
....
➜  ~ lei edit-search ~/mail/lore/rdma
    1 ; to refresh with new results, run: lei up /home/leonro/mail/lore/rdma
    2 ; `maxuid' and `lastresult' lines are maintained by "lei up" for optimization
    3 [lei]
    4         q = (l:linux-rdma.vger.kernel.org) AND rt:1.week.ago..
    5 [lei "q"]
    6         include = https://lore.kernel.org/all/
    7         external = 1
    8         local = 1
    9         remote = 1
   10         threads = 1
   11         dedupe = mid
   12         output = maildir:/home/leonro/mail/lore/rdma
   13 [external "/home/leonro/.local/share/lei/store"]
   14         maxuid = 109652
   15 [external "https://lore.kernel.org/all/"]
   16         lastresult = 1741026335

Thanks

Re: lei stores mails in wrong folder

[Eric Wong]

Leon Romanovsky <leon@kernel.org> wrote:
> Hi,
> 
> I have a strange behavior for newly installed lei ("dnf install lei" on FC41).
> All new messages are stored in "cur" Maildir folder and not in "new" as I would expect.
> Current behavior causes to weird situation where all messages are getting "O"  (old) tag.

Right, lei used JMAP for its internal indexing design, and
JMAP doesn't have a new (aka `\Recent' in IMAP) vs old
distinction that IMAP, Maildir or mbox* have.   JMAP has unseen
vs seen, but not new|\Recent vs old.

I also seem to recall common rename(2) storms overflowing the
inotify queue as another reason to avoid adding \Recent keyword
support.

Given JMAP doesn't seem all that popular after a few years, I
could be convinced to support a `recent' keyword in lei.
However it's more effort, another source of potential bugs,
higher likelyhood of IN_Q_OVERFLOW and excess metadata traffic
for something which (IMHO) doesn't seem all that useful.

As with all keywords, `recent' would also be tied logically to
the message itself across all folders; not a particular copy of
the message in a particular folder.  I suspect that may be at
least of of the reasons `recent' was omitted from JMAP since
JMAP (and lei) treats keywords as metadata associated with all
copies of a particular message.

Synchronizing keywords like `seen' and `answered' across
multiple Maildir copies is still likely buggy, even :x

Re: lei stores mails in wrong folder

[Eric Wong]

Leon Romanovsky <leon@kernel.org> wrote:
> On Tue, Mar 04, 2025 at 06:43:37AM +0000, Eric Wong wrote:
> > Leon Romanovsky <leon@kernel.org> wrote:
> > > Hi,
> > > 
> > > I have a strange behavior for newly installed lei ("dnf install lei" on FC41).
> > > All new messages are stored in "cur" Maildir folder and not in "new" as I would expect.
> > > Current behavior causes to weird situation where all messages are getting "O"  (old) tag.
> > 
> > Right, lei used JMAP for its internal indexing design, and
> > JMAP doesn't have a new (aka `\Recent' in IMAP) vs old
> > distinction that IMAP, Maildir or mbox* have.   JMAP has unseen
> > vs seen, but not new|\Recent vs old.
> 
> I asked my colleague, who are using lei on Ubuntu, and for him, lei stores
> messages correctly, in "new" folder.

Ah, that's for augmenting existing searches (via `lei up').

https://80x24.org/public-inbox.git/2d4ac1c886908bb1d79592913a2a2037a1a024e7/s/
2d4ac1c8 (lei <q|up>: set \Recent on non-empty mbox and Maildir, 2021-05-23)

> Any suggestion how to set lei to store in "new" anyway?

So I suppose you can use --augment with any non-empty Maildir
target.

> Maybe I need to push to IMAPand fetch from there with mbsync?

Yeah, setting \Recent is forced by IMAP and mutt uses a
non-standard "Old" flag as mentioned in the message of the
aforementioned commit.

> I'm attaching picture of how it looks in mutt now.

No images, please.  Your message ended up in my spam folder
because of it.  Everything in userspace CLI and daemons can be
reported and debugged completely in text w/o relying on complex
graphics support.  I'm also not willing to deal with images
for mental health and various safety reasons.

Thanks.

Re: lei stores mails in wrong folder

[Leon Romanovsky]

On Fri, Mar 14, 2025, at 11:43, Eric Wong wrote:
> Leon Romanovsky <leon@kernel.org> wrote:
>> On Tue, Mar 04, 2025 at 06:43:37AM +0000, Eric Wong wrote:
>> > Leon Romanovsky <leon@kernel.org> wrote:
>> > > Hi,
>> > > 
>> > > I have a strange behavior for newly installed lei ("dnf install lei" on FC41).
>> > > All new messages are stored in "cur" Maildir folder and not in "new" as I would expect.
>> > > Current behavior causes to weird situation where all messages are getting "O"  (old) tag.
>> > 
>> > Right, lei used JMAP for its internal indexing design, and
>> > JMAP doesn't have a new (aka `\Recent' in IMAP) vs old
>> > distinction that IMAP, Maildir or mbox* have.   JMAP has unseen
>> > vs seen, but not new|\Recent vs old.
>> 
>> I asked my colleague, who are using lei on Ubuntu, and for him, lei stores
>> messages correctly, in "new" folder.
>
> Ah, that's for augmenting existing searches (via `lei up').

So I tried it now after waiting for a couple of hours and run "lei up" that worked and put all
new mails in correct folder. So it looks like the mystery solved.

Thanks a lot.

>
> https://80x24.org/public-inbox.git/2d4ac1c886908bb1d79592913a2a2037a1a024e7/s/
> 2d4ac1c8 (lei <q|up>: set \Recent on non-empty mbox and Maildir, 2021-05-23)
>
>> Any suggestion how to set lei to store in "new" anyway?
>
> So I suppose you can use --augment with any non-empty Maildir
> target.
>
>> Maybe I need to push to IMAPand fetch from there with mbsync?
>
> Yeah, setting \Recent is forced by IMAP and mutt uses a
> non-standard "Old" flag as mentioned in the message of the
> aforementioned commit.
>
>> I'm attaching picture of how it looks in mutt now.
>
> No images, please.  Your message ended up in my spam folder
> because of it.  Everything in userspace CLI and daemons can be
> reported and debugged completely in text w/o relying on complex
> graphics support.  I'm also not willing to deal with images
> for mental health and various safety reasons.

Sure, I wanted to show how it looks in mutt. 
"A picture is worth a thousand words"

>
> Thanks.