user/dev discussion of public-inbox itself
 help / color / mirror / Atom feed
* JSON field names in terminal/pager output
@ 2021-01-05  9:56 Eric Wong
  2021-01-06  0:00 ` Kyle Meyer
  0 siblings, 1 reply; 3+ messages in thread
From: Eric Wong @ 2021-01-05  9:56 UTC (permalink / raw)
  To: meta

Are "f", "s", "t", "c" acceptable field names to show in JSON
output?  (instead of from/subject/to/cc)

The single-chars have been used in the search queries for as
long as we've had search.  And I stole that UI bit from
mairix(1), so there's prior art.

Anyways, the current JSON output looks something like the
following, comments inline

[{
  "blob": "d21717dae7e18dbc9efcd5ca57fe2ee92747bc06",
  "docid": 41946,
  "dt": "2021-01-05T09:24:49Z",

  # dt: is not an exact match for current WWW behavior which
  # needs YYYYMMDDHHMMSS (all digits).  Getting Xapian to parse
  # dates from Perl (w/o custom C++) isn't possible, yet.
  # dt: is the date header, "UTCDate" in JMAP.

  "f": "Eric Wong <e@80x24.org>",

  # "from": might be more obvious, but seeing it thousands of
  # times every message might be too much

  "m": "<20210105092449.GA22853@dcvr>",

  # mid/msgid/message-id

  "rcvd": "2021-01-05T09:24:49Z",
  # IMAP calls this INTERNALDATE, JMAP calls it "receivedAt"

  "references": [
    "<20210105090437.22801-1-e@80x24.org>",
    "<20210105090437.22801-5-e@80x24.org>"
  ],

  # we don't actually support searching on rereference, yet;
  # not sure if it's needed since we already do thread # expansion

  "relevance": 13,

  # relevance is ->get_percent from Xapian

  "s": "JSON pretty-printing [was: [4/4] ... (and maybe lei)]",
  "t": "meta@public-inbox.org"

  # Subject and To: headers
}, {
  # Another message, we do "}, {" to save vertical white space
  rather than "},\n{"

  "blob": "0ee1d7d9ec9b29c1d8f103033ed06e9e48e6ebfe",
  "docid": 41930,

  # side note: it's probably not worth displaying docid for
  # ephemeral search indices like LeiXSearch

  "dt": "2021-01-05T09:04:36Z",
  "f": "Eric Wong <e@80x24.org>",
  "m": "<20210105090437.22801-4-e@80x24.org>",
  "rcvd": "2021-01-05T09:04:38Z",
  "references": [
    "<20210105090437.22801-1-e@80x24.org>"
  ],
  "relevance": 36,
  "s": "[PATCH 3/4] lei: use client env as-is, drop daemon-env command",
  "t": "meta@public-inbox.org"
}, {


...

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: JSON field names in terminal/pager output
  2021-01-05  9:56 JSON field names in terminal/pager output Eric Wong
@ 2021-01-06  0:00 ` Kyle Meyer
  2021-01-06 10:27   ` Eric Wong
  0 siblings, 1 reply; 3+ messages in thread
From: Kyle Meyer @ 2021-01-06  0:00 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong writes:

> Are "f", "s", "t", "c" acceptable field names to show in JSON
> output?  (instead of from/subject/to/cc)

In my view they are, and, as you mention next, I like that they align
with the search prefixes.

> Anyways, the current JSON output looks something like the
> following, comments inline
>
> [{
[...]

Nice.

>   "references": [
>     "<20210105090437.22801-1-e@80x24.org>",
>     "<20210105090437.22801-5-e@80x24.org>"
>   ],
>
>   # we don't actually support searching on rereference, yet;
>   # not sure if it's needed since we already do thread # expansion

Is "thread # expansion" the "num -> tid -> nums" mapping done by Over's
expand_thread()?  And, from the CLI, that will be triggered by
`thread|t', right?

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: JSON field names in terminal/pager output
  2021-01-06  0:00 ` Kyle Meyer
@ 2021-01-06 10:27   ` Eric Wong
  0 siblings, 0 replies; 3+ messages in thread
From: Eric Wong @ 2021-01-06 10:27 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: meta

Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> 
> > Are "f", "s", "t", "c" acceptable field names to show in JSON
> > output?  (instead of from/subject/to/cc)
> 
> In my view they are, and, as you mention next, I like that they align
> with the search prefixes.
> 
> > Anyways, the current JSON output looks something like the
> > following, comments inline
> >
> > [{
> [...]
> 
> Nice.

Btw, I remember there's several JSON streaming formats.  I think
"JSONL" aka "NDJSON" aka "LDJSON" is the best for interopability
with tools like jq(1) and will be the default format when stdout
is to a pipe or regular file.

"Concatenated JSON" may be nice for pretty-printing

<RS>-delimited JSON is gross, but there's an RFC behind it...

cf.  https://en.wikipedia.org/wiki/JSON_streaming

> 
> >   "references": [
> >     "<20210105090437.22801-1-e@80x24.org>",
> >     "<20210105090437.22801-5-e@80x24.org>"
> >   ],
> >
> >   # we don't actually support searching on rereference, yet;
> >   # not sure if it's needed since we already do thread # expansion
> 
> Is "thread # expansion" the "num -> tid -> nums" mapping done by Over's
> expand_thread()?  And, from the CLI, that will be triggered by
> `thread|t', right?

Yup.  It works well for single sources (and even a single
extindex), but not when combinining multiple inboxes/extindices
since THREADID ({tid}) is per-inbox/extindex.  So extra dedupe
work needs to be done...

So searching across externals needs to be parallelized.  I'm
thinking SOCK_SEQPACKET Unix sockets is ideal for local
externals and hope it's usable in the modern *BSDs.

And SIGPIPE on pager exit needs to get delivered in a timely
fashion, too... (yes, little things like that really bug me :x)

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-01-06 10:27 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-05  9:56 JSON field names in terminal/pager output Eric Wong
2021-01-06  0:00 ` Kyle Meyer
2021-01-06 10:27   ` Eric Wong

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git