user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 4/4] lei q: -tt marks direct hits as "flagged"
  2021-02-25 10:11  6% [PATCH 0/4] lei: fleshing out some existing features Eric Wong
@ 2021-02-25 10:11  4% ` Eric Wong
  2021-02-26  3:38  7%   ` Kyle Meyer
  0 siblings, 1 reply; 6+ results
From: Eric Wong @ 2021-02-25 10:11 UTC (permalink / raw)
  To: meta

This can be used to quickly distinguish messages which were
direct hits when doing thread expansion vs messages that
were merely part of the same thread.

This is NOT mairix-derived behavior, but I occasionally found
it useful when looking at results in an MUA to know whether
a message was a direct hit or not.

This makes "-t" consistent with non-"-t" cases as far as keyword
reading goes.
---
 Documentation/lei-q.pod       |  8 ++++++
 MANIFEST                      |  1 +
 lib/PublicInbox/LEI.pm        |  4 +--
 lib/PublicInbox/LeiXSearch.pm | 21 +++++++++++++---
 t/lei-q-thread.t              | 47 +++++++++++++++++++++++++++++++++++
 5 files changed, 75 insertions(+), 6 deletions(-)
 create mode 100644 t/lei-q-thread.t

diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 75fdc613..0959beac 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -79,6 +79,14 @@ Augment output destination instead of clobbering it.
 
 Return all messages in the same thread as the actual match(es).
 
+Using this twice (C<-tt>) sets the C<flagged> (AKA "important")
+on messages which were actual messages.  This is useful to distinguish
+messages which were direct hits from messages which were merely part
+of the same thread.
+
+TODO: Warning: this flag may become persistent and saved in
+lei/store unless an MUA unflags it!  (Behavior undecided)
+
 =item -d STRATEGY, --dedupe=STRATEGY
 
 Strategy for deduplicating messages: C<content>, C<oid>, C<mid>, or
diff --git a/MANIFEST b/MANIFEST
index adbd108f..9cf33d48 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -373,6 +373,7 @@ t/lei-import-nntp.t
 t/lei-import.t
 t/lei-mirror.t
 t/lei-q-remote-import.t
+t/lei-q-thread.t
 t/lei.t
 t/lei_dedupe.t
 t/lei_external.t
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8eb96e78..8825fa43 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -109,7 +109,7 @@ sub index_opt {
 # command => [ positional_args, 1-line description, Getopt::Long option spec ]
 our %CMD = ( # sorted in order of importance/use:
 'q' => [ '--stdin|SEARCH_TERMS...', 'search for messages matching terms', qw(
-	save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t augment|a
+	save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+ augment|a
 	sort|s=s reverse|r offset=i remote! local! external! pretty
 	include|I=s@ exclude=s@ only=s@ jobs|j=s globoff|g stdin|
 	import-remote!
@@ -233,7 +233,7 @@ my %OPTDESC = (
 'dedupe|d=s' => ['STRATEGY|content|oid|mid|none',
 		'deduplication strategy'],
 'show	threads|t' => 'display entire thread a message belongs to',
-'q	threads|t' =>
+'q	threads|t+' =>
 	'return all messages in the same threads as the actual match(es)',
 'alert=s@' => ['CMD,:WINCH,:bell,<any command>',
 	'run command(s) or perform ops when done writing to output ' .
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2d399653..eb015978 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -66,6 +66,13 @@ sub remotes { @{$_[0]->{remotes} // []} }
 # called by PublicInbox::Search::xdb
 sub xdb_shards_flat { @{$_[0]->{shards_flat} // []} }
 
+sub mitem_kw ($$;$) {
+	my ($smsg, $mitem, $flagged) = @_;
+	my $kw = xap_terms('K', $mitem->get_document);
+	$kw->{flagged} = 1 if $flagged;
+	$smsg->{kw} = [ sort keys %$kw ];
+}
+
 # like over->get_art
 sub smsg_for {
 	my ($self, $mitem) = @_;
@@ -76,10 +83,7 @@ sub smsg_for {
 	my $num = int(($docid - 1) / $nshard) + 1;
 	my $ibx = $self->{shard2ibx}->[$shard];
 	my $smsg = $ibx->over->get_art($num);
-	if (ref($ibx->can('msg_keywords'))) {
-		my $kw = xap_terms('K', $mitem->get_document);
-		$smsg->{kw} = [ sort keys %$kw ];
-	}
+	mitem_kw($smsg, $mitem) if $ibx->can('msg_keywords');
 	$smsg->{docid} = $docid;
 	$smsg;
 }
@@ -143,6 +147,8 @@ sub query_thread_mset { # for --threads
 	my $mo = { %{$lei->{mset_opt}} };
 	my $mset;
 	my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $ibxish);
+	my $can_kw = !!$ibxish->can('msg_keywords');
+	my $fl = $lei->{opt}->{threads} > 1;
 	do {
 		$mset = $srch->mset($mo->{qstr}, $mo);
 		mset_progress($lei, $desc, $mset->size,
@@ -156,6 +162,13 @@ sub query_thread_mset { # for --threads
 				my $smsg = $over->get_art($n) or next;
 				wait_startq($lei);
 				my $mitem = delete $n2item{$smsg->{num}};
+				if ($mitem) {
+					if ($can_kw) {
+						mitem_kw($smsg, $mitem, $fl);
+					} else {
+						$smsg->{kw} = [ 'flagged' ];
+					}
+				}
 				$each_smsg->($smsg, $mitem);
 			}
 			@{$ctx->{xids}} = ();
diff --git a/t/lei-q-thread.t b/t/lei-q-thread.t
new file mode 100644
index 00000000..66db28a9
--- /dev/null
+++ b/t/lei-q-thread.t
@@ -0,0 +1,47 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10.1; use PublicInbox::TestCommon;
+require_git 2.6;
+require_mods(qw(json DBD::SQLite Search::Xapian));
+use PublicInbox::LeiToMail;
+my ($ro_home, $cfg_path) = setup_public_inboxes;
+test_lei(sub {
+	my $eml = eml_load('t/utf8.eml');
+	my $buf = PublicInbox::LeiToMail::eml2mboxrd($eml, { kw => ['seen'] });
+	lei_ok([qw(import -F mboxrd -)], undef, { 0 => $buf, %$lei_opt });
+
+	lei_ok qw(q -t m:testmessage@example.com);
+	my $res = json_utf8->decode($lei_out);
+	is_deeply($res->[0]->{kw}, [ 'seen' ], 'q -t sets keywords');
+
+	$eml = eml_load('t/utf8.eml');
+	$eml->header_set('References', $eml->header('Message-ID'));
+	$eml->header_set('Message-ID', '<a-reply@miss>');
+	$buf = PublicInbox::LeiToMail::eml2mboxrd($eml, { kw => ['draft'] });
+	lei_ok([qw(import -F mboxrd -)], undef, { 0 => $buf, %$lei_opt });
+
+	lei_ok qw(q -t m:testmessage@example.com);
+	$res = json_utf8->decode($lei_out);
+	is(scalar(@$res), 3, 'got 2 results');
+	pop @$res;
+	my %m = map { $_->{'m'} => $_ } @$res;
+	is_deeply($m{'<testmessage@example.com>'}->{kw}, ['seen'],
+		'flag set in direct hit');
+	'TODO' or is_deeply($m{'<a-reply@miss>'}->{kw}, ['draft'],
+		'flag set in thread hit');
+
+	lei_ok qw(q -t -t m:testmessage@example.com);
+	$res = json_utf8->decode($lei_out);
+	is(scalar(@$res), 3, 'got 2 results with -t -t');
+	pop @$res;
+	%m = map { $_->{'m'} => $_ } @$res;
+	is_deeply($m{'<testmessage@example.com>'}->{kw}, ['flagged', 'seen'],
+		'flagged set in direct hit');
+	'TODO' or is_deeply($m{'<testmessage@example.com>'}->{kw}, ['draft'],
+		'flagged set in direct hit');
+	lei_ok qw(q -t -t m:testmessage@example.com --only), "$ro_home/t2";
+	$res = json_utf8->decode($lei_out);
+	is_deeply($res->[0]->{kw}, [ 'flagged' ], 'flagged set on external');
+});
+done_testing;

^ permalink raw reply related	[relevance 4%]

* [PATCH 0/4] lei: fleshing out some existing features
@ 2021-02-25 10:11  6% Eric Wong
  2021-02-25 10:11  4% ` [PATCH 4/4] lei q: -tt marks direct hits as "flagged" Eric Wong
  0 siblings, 1 reply; 6+ results
From: Eric Wong @ 2021-02-25 10:11 UTC (permalink / raw)
  To: meta

Managed to get more stuff done while still pondering keyword
 storage with read-only externals(*)

1/4 fleshes out convert, which should be feature-complete as far
as currently supported inputs and outputs (no MH, JMAP, POP3,
MMDF, yet)

2/4 represents a major incompatibility in replacing --format/-f with
--in-format/-F in "lei import" for consistency with "lei convert".
Anyways, this is pre-release software and I discouraged "-f" anyways;
so hopefully nobody's scripts are broken :x

4/4 is another one of the things I've found myself wanting
for a while (it wasn't in mairix).

(*) https://public-inbox.org/meta/20210224204950.GA2076@dcvr/

Eric Wong (4):
  lei convert: support IMAP output and "-F eml" inputs
  lei import: use --in-format/-F for consistency
  test_common: io_modes: always support read/write
  lei q: -tt marks direct hits as "flagged"

 Documentation/lei-import.pod  |  2 +-
 Documentation/lei-q.pod       |  8 ++++++
 MANIFEST                      |  1 +
 lib/PublicInbox/LEI.pm        | 12 ++++-----
 lib/PublicInbox/LeiConvert.pm | 51 ++++++++++++++++++++++-------------
 lib/PublicInbox/LeiImport.pm  |  8 +++---
 lib/PublicInbox/LeiXSearch.pm | 21 ++++++++++++---
 lib/PublicInbox/NetWriter.pm  |  3 ++-
 lib/PublicInbox/TestCommon.pm |  4 +--
 t/lei-convert.t               | 15 +++++++++++
 t/lei-import.t                | 12 ++++-----
 t/lei-q-thread.t              | 47 ++++++++++++++++++++++++++++++++
 t/lei_to_mail.t               |  2 +-
 xt/net_writer-imap.t          |  4 +++
 14 files changed, 146 insertions(+), 44 deletions(-)
 create mode 100644 t/lei-q-thread.t


^ permalink raw reply	[relevance 6%]

* [PATCH] lei q: one -t shouldn't set `flagged' on external mail
@ 2021-03-05  4:03  6% Eric Wong
  0 siblings, 0 replies; 6+ results
From: Eric Wong @ 2021-03-05  4:03 UTC (permalink / raw)
  To: meta

We only want to set `flagged' if a user requests it via
a two '-t' switches.

Fixes: 232f8e376fe2856c ("lei q: -tt marks direct hits as flagged")
---
 lib/PublicInbox/LeiXSearch.pm | 6 +++---
 t/lei-q-thread.t              | 8 ++++++--
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 3270b420..f2c8c02e 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -148,7 +148,7 @@ sub query_thread_mset { # for --threads
 	my $mset;
 	my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $ibxish);
 	my $can_kw = !!$ibxish->can('msg_keywords');
-	my $fl = $lei->{opt}->{threads} > 1;
+	my $fl = $lei->{opt}->{threads} > 1 ? [ 'flagged' ] : undef;
 	do {
 		$mset = $srch->mset($mo->{qstr}, $mo);
 		mset_progress($lei, $desc, $mset->size,
@@ -165,8 +165,8 @@ sub query_thread_mset { # for --threads
 				if ($mitem) {
 					if ($can_kw) {
 						mitem_kw($smsg, $mitem, $fl);
-					} else {
-						$smsg->{kw} = [ 'flagged' ];
+					} elsif ($fl) {
+						$smsg->{kw} = $fl;
 					}
 				}
 				$each_smsg->($smsg, $mitem);
diff --git a/t/lei-q-thread.t b/t/lei-q-thread.t
index 0ddf47a6..28c639f5 100644
--- a/t/lei-q-thread.t
+++ b/t/lei-q-thread.t
@@ -41,8 +41,12 @@ test_lei(sub {
 		'flagged set in direct hit');
 	'TODO' or is_deeply($m{'<testmessage@example.com>'}->{kw}, ['draft'],
 		'flagged set in direct hit');
-	lei_ok qw(q -t -t m:testmessage@example.com --only), "$ro_home/t2";
+	lei_ok qw(q -tt m:testmessage@example.com --only), "$ro_home/t2";
 	$res = json_utf8->decode($lei_out);
-	is_deeply($res->[0]->{kw}, [ 'flagged' ], 'flagged set on external');
+	is_deeply($res->[0]->{kw}, [ 'flagged' ],
+		'flagged set on external with -tt');
+	lei_ok qw(q -t m:testmessage@example.com --only), "$ro_home/t2";
+	$res = json_utf8->decode($lei_out);
+	ok(!exists($res->[0]->{kw}), 'flagged not set on external with 1 -t');
 });
 done_testing;

^ permalink raw reply related	[relevance 6%]

* Re: [PATCH 4/4] lei q: -tt marks direct hits as "flagged"
  2021-02-25 10:11  4% ` [PATCH 4/4] lei q: -tt marks direct hits as "flagged" Eric Wong
@ 2021-02-26  3:38  7%   ` Kyle Meyer
  2021-02-26  4:13  7%     ` Eric Wong
  0 siblings, 1 reply; 6+ results
From: Kyle Meyer @ 2021-02-26  3:38 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong writes:

> This can be used to quickly distinguish messages which were
> direct hits when doing thread expansion vs messages that
> were merely part of the same thread.

Ah, that's very useful.

> +Using this twice (C<-tt>) sets the C<flagged> (AKA "important")
> +on messages which were actual messages.  This is useful to distinguish
> +messages which were direct hits from messages which were merely part
> +of the same thread.
> +
> +TODO: Warning: this flag may become persistent and saved in
> +lei/store unless an MUA unflags it!  (Behavior undecided)

Oy, I understand even less than I thought I did.  How does the
information about what the MUA unflags get back into the store?  Is
there an implicit additional step (`lei import ...')?

^ permalink raw reply	[relevance 7%]

* Re: [PATCH 4/4] lei q: -tt marks direct hits as "flagged"
  2021-02-26  3:38  7%   ` Kyle Meyer
@ 2021-02-26  4:13  7%     ` Eric Wong
  2021-02-26  4:38  7%       ` Kyle Meyer
  0 siblings, 1 reply; 6+ results
From: Eric Wong @ 2021-02-26  4:13 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: meta

Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> > +TODO: Warning: this flag may become persistent and saved in
> > +lei/store unless an MUA unflags it!  (Behavior undecided)
> 
> Oy, I understand even less than I thought I did.  How does the
> information about what the MUA unflags get back into the store?  Is
> there an implicit additional step (`lei import ...')?

lei will watch (via inotify/EVFILT_VNODE) mail stores it knows
about for flag updates.  At least that's the plan...

Also, when overwriting an existing output, I think it would be
wise to do an implicit import of any messages that aren't
already in lei/store or an external.  That would save users
from accidentally trashing their data.

^ permalink raw reply	[relevance 7%]

* Re: [PATCH 4/4] lei q: -tt marks direct hits as "flagged"
  2021-02-26  4:13  7%     ` Eric Wong
@ 2021-02-26  4:38  7%       ` Kyle Meyer
  0 siblings, 0 replies; 6+ results
From: Kyle Meyer @ 2021-02-26  4:38 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong writes:

> Kyle Meyer <kyle@kyleam.com> wrote:

>> Oy, I understand even less than I thought I did.  How does the
>> information about what the MUA unflags get back into the store?  Is
>> there an implicit additional step (`lei import ...')?
>
> lei will watch (via inotify/EVFILT_VNODE) mail stores it knows
> about for flag updates.  At least that's the plan...
>
> Also, when overwriting an existing output, I think it would be
> wise to do an implicit import of any messages that aren't
> already in lei/store or an external.  That would save users
> from accidentally trashing their data.

Makes sense.  Thanks for the details (especially if you're repeating
yourself).

^ permalink raw reply	[relevance 7%]

Results 1-6 of 6 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-02-25 10:11  6% [PATCH 0/4] lei: fleshing out some existing features Eric Wong
2021-02-25 10:11  4% ` [PATCH 4/4] lei q: -tt marks direct hits as "flagged" Eric Wong
2021-02-26  3:38  7%   ` Kyle Meyer
2021-02-26  4:13  7%     ` Eric Wong
2021-02-26  4:38  7%       ` Kyle Meyer
2021-03-05  4:03  6% [PATCH] lei q: one -t shouldn't set `flagged' on external mail Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).