user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* angle brackets in "m:" and "refs:" in "lei q" JSON
@ 2021-03-04 18:43 Eric Wong
  2021-03-06 18:26 ` Kyle Meyer
  0 siblings, 1 reply; 3+ messages in thread
From: Eric Wong @ 2021-03-04 18:43 UTC (permalink / raw)
  To: meta

I'm thinking these shouldn't include angle brackets:

  "m": "<20210228122528.18552-2-e@80x24.org>",
  "refs": ["<20210228122528.18552-1-e@80x24.org>"],

Using angle brackets on the command-line requires quoting to
disambiguate against redirects, so it's a pain.  Leaving the
brackets in still works because of how Xapian's query parser
works, not because of anything we do on our end.

Since the actual headers are "Message-ID" and "References", (and
not "m" or "refs"), I think it's clear that we don't have to
match the raw mail contents exactly.  We RFC 2047 decode
"f|t|c|s" fields anyways instead of showing the raw values,
so more precedence for leaving out <>.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: angle brackets in "m:" and "refs:" in "lei q" JSON
  2021-03-04 18:43 angle brackets in "m:" and "refs:" in "lei q" JSON Eric Wong
@ 2021-03-06 18:26 ` Kyle Meyer
  2021-03-08  8:08   ` [PATCH] lei q: remove angle brackets around Message-IDs Eric Wong
  0 siblings, 1 reply; 3+ messages in thread
From: Kyle Meyer @ 2021-03-06 18:26 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong writes:

> I'm thinking these shouldn't include angle brackets:
>
>   "m": "<20210228122528.18552-2-e@80x24.org>",
>   "refs": ["<20210228122528.18552-1-e@80x24.org>"],
>
> Using angle brackets on the command-line requires quoting to
> disambiguate against redirects, so it's a pain.  Leaving the
> brackets in still works because of how Xapian's query parser
> works, not because of anything we do on our end.

I think it'd be nice to drop the brackets from a noise perspective too.

Also, does m: work with brackets?  Trying it out with a recent message
ID:

  $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
    '20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
  {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}

  $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
    m:'20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
  {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}

  $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
    '<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
  {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}

  $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
    m:'<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
  # no results

> Since the actual headers are "Message-ID" and "References", (and
> not "m" or "refs"), I think it's clear that we don't have to
> match the raw mail contents exactly.  We RFC 2047 decode
> "f|t|c|s" fields anyways instead of showing the raw values,
> so more precedence for leaving out <>.

Fwiw I don't think leaving out the brackets would be a source of
confusion.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH] lei q: remove angle brackets around Message-IDs
  2021-03-06 18:26 ` Kyle Meyer
@ 2021-03-08  8:08   ` Eric Wong
  0 siblings, 0 replies; 3+ messages in thread
From: Eric Wong @ 2021-03-08  8:08 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: meta

Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> 
> > I'm thinking these shouldn't include angle brackets:
> >
> >   "m": "<20210228122528.18552-2-e@80x24.org>",
> >   "refs": ["<20210228122528.18552-1-e@80x24.org>"],
> >
> > Using angle brackets on the command-line requires quoting to
> > disambiguate against redirects, so it's a pain.  Leaving the
> > brackets in still works because of how Xapian's query parser
> > works, not because of anything we do on our end.
> 
> I think it'd be nice to drop the brackets from a noise perspective too.

Yes, we don't include them in name email address pairs, either.

> Also, does m: work with brackets?  Trying it out with a recent message
> ID:
> 
>   $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
>     '20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
>   {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
> 
>   $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
>     m:'20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
>   {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
> 
>   $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
>     '<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
>   {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
> 
>   $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
>     m:'<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
>   # no results

Odd, I'm not sure about that one...  It's probably something
the Xapian query parser is doing internally and nothing on
our end...

> > Since the actual headers are "Message-ID" and "References", (and
> > not "m" or "refs"), I think it's clear that we don't have to
> > match the raw mail contents exactly.  We RFC 2047 decode
> > "f|t|c|s" fields anyways instead of showing the raw values,
> > so more precedence for leaving out <>.
> 
> Fwiw I don't think leaving out the brackets would be a source of
> confusion.

Agreed.  And I'm now wondering if we should start indexing
"References:" to be a searchable header with the "refs:" prefix
(but also wary about increasing disk space usage as a result...)

In any case, this denoises the output a bit:
------------8<----------
Subject: [PATCH] lei q: remove angle brackets around Message-IDs

They're unnecessary visual noise, and angle brackets don't
always work as intended when going through Xapian's query
parser.

Since we already use "m:" and "refs:" instead of the actual
header names, it should be obvious we're at liberty to
abbreviate such things

Link: https://public-inbox.org/meta/20210304184348.GA19350@dcvr/
---
 lib/PublicInbox/LeiOverview.pm | 5 ++---
 t/lei-externals.t              | 2 +-
 t/lei-q-thread.t               | 8 ++++----
 xt/net_writer-imap.t           | 2 +-
 4 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 4db1d8c8..01556273 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -141,17 +141,16 @@ sub _unbless_smsg {
 	$smsg->{dt} = _iso8601(delete $smsg->{ds}); # JMAP UTCDate
 	$smsg->{pct} = get_pct($mitem) if $mitem;
 	if (my $r = delete $smsg->{references}) {
-		$smsg->{refs} = [ map { "<$_>" } ($r =~ m/$MID_EXTRACT/go) ];
+		$smsg->{refs} = [ map { $_ } ($r =~ m/$MID_EXTRACT/go) ];
 	}
 	if (my $m = delete($smsg->{mid})) {
-		$smsg->{'m'} = "<$m>";
+		$smsg->{'m'} = $m;
 	}
 	for my $f (qw(from to cc)) {
 		my $v = delete $smsg->{$f} or next;
 		$smsg->{substr($f, 0, 1)} = pairs($v);
 	}
 	$smsg->{'s'} = delete $smsg->{subject};
-	# can we be bothered to parse From/To/Cc into arrays?
 	scalar { %$smsg }; # unbless
 }
 
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 29667640..2a92d101 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -25,7 +25,7 @@ SKIP: {
 	lei_ok(@cmd, \"query $url");
 	is($lei_err, '', "no errors on $url");
 	my $res = json_utf8->decode($lei_out);
-	is($res->[0]->{'m'}, "<$mid>", "got expected mid from $url") or
+	is($res->[0]->{'m'}, $mid, "got expected mid from $url") or
 		skip 'further remote tests', 1;
 	lei_ok(@cmd, 'd:..20101002', \'no results, no error');
 	is($lei_err, '', 'no output on 404, matching local FS behavior');
diff --git a/t/lei-q-thread.t b/t/lei-q-thread.t
index 28c639f5..e24fb2cb 100644
--- a/t/lei-q-thread.t
+++ b/t/lei-q-thread.t
@@ -27,9 +27,9 @@ test_lei(sub {
 	is(scalar(@$res), 3, 'got 2 results');
 	pop @$res;
 	my %m = map { $_->{'m'} => $_ } @$res;
-	is_deeply($m{'<testmessage@example.com>'}->{kw}, ['seen'],
+	is_deeply($m{'testmessage@example.com'}->{kw}, ['seen'],
 		'flag set in direct hit');
-	'TODO' or is_deeply($m{'<a-reply@miss>'}->{kw}, ['draft'],
+	'TODO' or is_deeply($m{'a-reply@miss'}->{kw}, ['draft'],
 		'flag set in thread hit');
 
 	lei_ok qw(q -t -t m:testmessage@example.com);
@@ -37,9 +37,9 @@ test_lei(sub {
 	is(scalar(@$res), 3, 'got 2 results with -t -t');
 	pop @$res;
 	%m = map { $_->{'m'} => $_ } @$res;
-	is_deeply($m{'<testmessage@example.com>'}->{kw}, ['flagged', 'seen'],
+	is_deeply($m{'testmessage@example.com'}->{kw}, ['flagged', 'seen'],
 		'flagged set in direct hit');
-	'TODO' or is_deeply($m{'<testmessage@example.com>'}->{kw}, ['draft'],
+	'TODO' or is_deeply($m{'testmessage@example.com'}->{kw}, ['draft'],
 		'flagged set in direct hit');
 	lei_ok qw(q -tt m:testmessage@example.com --only), "$ro_home/t2";
 	$res = json_utf8->decode($lei_out);
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index c24fa993..3631d932 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -168,7 +168,7 @@ test_lei(sub {
 	is_deeply($empty, [], 'clobbered folder');
 	lei_ok qw(q -o /dev/stdout m:testmessage@example.com --no-external);
 	$res = json_utf8->decode($lei_out)->[0];
-	is_deeply([@$res{qw(m kw)}], ['<testmessage@example.com>', ['seen']],
+	is_deeply([@$res{qw(m kw)}], ['testmessage@example.com', ['seen']],
 		'kw set');
 });
 

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-03-08  8:08 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-04 18:43 angle brackets in "m:" and "refs:" in "lei q" JSON Eric Wong
2021-03-06 18:26 ` Kyle Meyer
2021-03-08  8:08   ` [PATCH] lei q: remove angle brackets around Message-IDs Eric Wong

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.mail.public-inbox.meta
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.mail.public-inbox.meta
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git