* lei q: importing messages when specifying '-f *json*'?
@ 2021-12-30 23:04 5% Kyle Meyer
0 siblings, 0 replies; 2+ results
From: Kyle Meyer @ 2021-12-30 23:04 UTC (permalink / raw)
To: meta; +Cc: piem
The lei-q interface I'm working on for piem [1] consumes JSON output to
display search results. From there, an individual message can be shown.
If the query is against a remote external, that involves two curl calls
to the remote, one for the search and one for the display.
I'd like to import hits from the first step into the local store so that
I can avoid the second curl call. However, I haven't been able to
figure out a way to do this when requesting JSON output.
For example [2], say I don't have meta's 20211124154539.350522-1-e@80x24.org
locally:
$ lei daemon-kill
$ export HOME=$(mktemp -d "${TMPDIR:-/tmp}"/pi-testing-XXXXXXX)
$ lei q m:20211124154539.350522-1-e@80x24.org
# /tmp/pi-testing-3JaSz2K/.config/lei/config created
[null]
If I search for that message against https://public-inbox.org/meta/ and
request JSON output
$ lei q -I https://public-inbox.org/meta/ -f ldjson \
m:20211124154539.350522-1-e@80x24.org
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
then no message ends up in the local store:
$ lei q m:20211124154539.350522-1-e@80x24.org
[null]
In contrast, if I request mboxrd output
$ lei q -I https://public-inbox.org/meta/ -f mboxrd \
m:20211124154539.350522-1-e@80x24.org >/dev/null
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
the message is imported to the local store:
$ lei q -f ldjson m:20211124154539.350522-1-e@80x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
$ git -C $HOME/.local/share/lei/store/local/0.git/ log --oneline
ca12a1b (HEAD -> master) [PATCH] eliminate some unused subs
I was hoping that --import-remote might do the trick, but that doesn't
seem to be the case:
$ lei daemon-kill
$ export HOME=$(mktemp -d "${TMPDIR:-/tmp}"/pi-testing-XXXXXXX)
$ lei q --import-remote -I https://public-inbox.org/meta/ \
-f ldjson m:20211124154539.350522-1-e@80x24.org
# /tmp/pi-testing-Ny7KDcB/.config/lei/config created
# /usr/bin/curl -Sf -s -d '' https://public-inbox.org/meta/?x=m&q=m%3A20211124154539.350522-1-e%4080x24.org
{"blob":"a8754283bd9e985d6e1156215071be59aa2b5a53",...}
$ lei q m:20211124154539.350522-1-e@80x24.org
[null]
Should --import-remote trigger an import in the case above?
[1] https://git.kyleam.com/piem/tree/piem-lei.el
[2] These are with public-inbox's current master (07cd8973baf).
^ permalink raw reply [relevance 5%]
* [PATCH] eliminate some unused subs
@ 2021-11-24 15:45 7% Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-11-24 15:45 UTC (permalink / raw)
To: meta
->newsgroup_matches was never used, and ->shard_over_check
was dropped in 89193578d21f (extindex: --gc checkpoints, 2021-10-06).
---
lib/PublicInbox/MiscSearch.pm | 35 -------------------------------
lib/PublicInbox/SearchIdxShard.pm | 10 ---------
2 files changed, 45 deletions(-)
diff --git a/lib/PublicInbox/MiscSearch.pm b/lib/PublicInbox/MiscSearch.pm
index 6b575b0da083..c6d2a062a668 100644
--- a/lib/PublicInbox/MiscSearch.pm
+++ b/lib/PublicInbox/MiscSearch.pm
@@ -81,41 +81,6 @@ sub mset {
retry_reopen($self, \&misc_enquire_once, $qr, $opt);
}
-sub ibx_matches_once { # retry_reopen callback
- my ($self, $qr, $by_newsgroup) = @_;
- # double in case no newsgroups are configured:
- my $limit = scalar(keys %$by_newsgroup) * 2;
- my $opt = { limit => $limit, offset => 0, relevance => -1 };
- my $ret = {}; # newsgroup => $ibx of matches
- while (1) {
- my $mset = misc_enquire_once($self, $qr, $opt);
- for my $mi ($mset->items) {
- my ($eidx_key) = xap_terms('Q', $mi->get_document);
- if (defined($eidx_key)) {
- if (my $ibx = $by_newsgroup->{$eidx_key}) {
- $ret->{$eidx_key} = $ibx;
- }
- } else {
- warn <<EOF;
-W: docid=${\$mi->get_docid} has no `Q' (eidx_key) term
-EOF
- }
- }
- my $nr = $mset->size;
- return $ret if $nr < $limit;
- $opt->{offset} += $nr;
- }
-}
-
-# returns a newsgroup => PublicInbox::Inbox mapping
-sub newsgroup_matches {
- my ($self, $qs, $pi_cfg) = @_;
- my $qp = $self->{qp} //= mi_qp_new($self);
- $qs .= ' type:inbox';
- my $qr = $qp->parse_query($qs, $PublicInbox::Search::QP_FLAGS);
- retry_reopen($self, \&ibx_matches_once, $qr, $pi_cfg->{-by_newsgroup});
-}
-
sub ibx_data_once {
my ($self, $ibx) = @_;
my $xdb = $self->{xdb};
diff --git a/lib/PublicInbox/SearchIdxShard.pm b/lib/PublicInbox/SearchIdxShard.pm
index 8635f5858a8a..000abd9413b5 100644
--- a/lib/PublicInbox/SearchIdxShard.pm
+++ b/lib/PublicInbox/SearchIdxShard.pm
@@ -75,14 +75,4 @@ sub shard_close {
$self->ipc_worker_stop;
}
-sub shard_over_check {
- my ($self, $over) = @_;
- if ($self->{-ipc_req} && $over->{dbh}) {
- # can't send DB handles over IPC, and use read-only to avoid
- # create_tables lock conflict:
- $over = PublicInbox::Over->new($over->{dbh}->sqlite_db_filename)
- }
- $self->ipc_do('over_check', $over);
-}
-
1;
^ permalink raw reply related [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-11-24 15:45 7% [PATCH] eliminate some unused subs Eric Wong
2021-12-30 23:04 5% lei q: importing messages when specifying '-f *json*'? Kyle Meyer
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).