* [PATCH 08/14] lei q: reinstate smsg dedupe
2021-01-14 7:06 7% [PATCH 00/14] lei: another pile of changes Eric Wong
@ 2021-01-14 7:06 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-01-14 7:06 UTC (permalink / raw)
To: meta
Now that dedupe is serialization and fork-safe, we can
wire it back up in our query results paths.
---
lib/PublicInbox/LeiQuery.pm | 5 ++---
lib/PublicInbox/LeiXSearch.pm | 8 ++++++--
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 1a3e1193..69d2f9a6 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -26,14 +26,13 @@ sub lei_q {
my $sto = $self->_lei_store(1);
my $cfg = $self->_lei_cfg(1);
my $opt = $self->{opt};
- require PublicInbox::LeiDedupe;
- my $dd = PublicInbox::LeiDedupe->new($self);
# --local is enabled by default
# src: LeiXSearch || LeiSearch || Inbox
my @srcs;
require PublicInbox::LeiXSearch;
require PublicInbox::LeiOverview;
+ require PublicInbox::LeiDedupe;
my $lxs = PublicInbox::LeiXSearch->new;
# --external is enabled by default, but allow --no-external
@@ -49,8 +48,8 @@ sub lei_q {
unshift(@srcs, $sto->search) if $opt->{'local'};
# no forking workers after this
- require PublicInbox::LeiOverview;
$self->{ovv} = PublicInbox::LeiOverview->new($self);
+ $self->{dd} = PublicInbox::LeiDedupe->new($self);
my %mset_opt = map { $_ => $opt->{$_} } qw(thread limit offset);
$mset_opt{asc} = $opt->{'reverse'} ? 1 : 0;
$mset_opt{qstr} = join(' ', map {;
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 68889e81..80e7a7f7 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -103,6 +103,8 @@ sub query_thread_mset { # for --thread
my $mo = { %{$lei->{mset_opt}} };
my $mset;
my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
+ my $dd = $lei->{dd};
+ $dd->prepare_dedupe;
do {
$mset = $srch->mset($mo->{qstr}, $mo);
my $ids = $srch->mset_to_artnums($mset, $mo);
@@ -112,7 +114,7 @@ sub query_thread_mset { # for --thread
while ($over->expand_thread($ctx)) {
for my $n (@{$ctx->{xids}}) {
my $smsg = $over->get_art($n) or next;
- # next if $dd->is_smsg_dup($smsg); TODO
+ next if $dd->is_smsg_dup($smsg);
my $mitem = delete $n2item{$smsg->{num}};
$each_smsg->($smsg, $mitem);
# $self->out($buf .= $ORS);
@@ -132,11 +134,13 @@ sub query_mset { # non-parallel for non-"--thread" users
my $mset;
$self->attach_external($_) for @$srcs;
my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
+ my $dd = $lei->{dd};
+ $dd->prepare_dedupe;
do {
$mset = $self->mset($mo->{qstr}, $mo);
for my $it ($mset->items) {
my $smsg = smsg_for($self, $it) or next;
- # next if $dd->is_smsg_dup($smsg);
+ next if $dd->is_smsg_dup($smsg);
$each_smsg->($smsg, $it);
# $self->out($buf .= $ORS) if defined $buf;
#$emit_cb->($smsg);
^ permalink raw reply related [relevance 6%]
* [PATCH 00/14] lei: another pile of changes
@ 2021-01-14 7:06 7% Eric Wong
2021-01-14 7:06 6% ` [PATCH 08/14] lei q: reinstate smsg dedupe Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-01-14 7:06 UTC (permalink / raw)
To: meta
PATCH 2/14 took forever to figure out; turns out I was hunting
an old bug in Perl :x (and led to PATCH 3/14, too)
We could probably go farther on 5/14 and eliminate the
need for @TO_CLOSE_ATFORK_CHILD completely, but my brain
was fried from 2/14 :x.
The "ts:" => "rt:" change is technically user-visible,
but "ts:" was never publicly documented so I doubt it
affects anybody. "rt:" (received time) may be documented
in the future.
Eric Wong (14):
cmd_ipc: support + test EINTR + EAGAIN, no FDs
lei: test SIGPIPE, stop xsearch workers on client abort
daemon+watch: fix localization of %SIG for non-signalfd users
lei: do not unlink socket path at exit
lei: reduce live FD references in wq child
lei: rely on localized $current_lei for warnings
lei_dedupe+shared_kv: ensure round-tripping serialization
lei q: reinstate smsg dedupe
search: rename "ts:" prefix to "rt:"
lei_overview: rename "references" to "refs"
lei: q: lock stdout on overview output
leixsearch: remove some commented out code
lei: remove temporary var on open
lei: pass FD to CWD via cmsg, use fchdir on server
MANIFEST | 2 +
lib/PublicInbox/CmdIPC4.pm | 6 +-
lib/PublicInbox/Daemon.pm | 4 +-
lib/PublicInbox/IMAPsearchqp.pm | 6 +-
lib/PublicInbox/IPC.pm | 45 +++-----
lib/PublicInbox/LEI.pm | 182 +++++++++++++++++---------------
lib/PublicInbox/LeiDedupe.pm | 29 ++---
lib/PublicInbox/LeiOverview.pm | 43 +++++++-
lib/PublicInbox/LeiQuery.pm | 27 ++---
lib/PublicInbox/LeiXSearch.pm | 60 +++++++----
lib/PublicInbox/Lock.pm | 2 +-
lib/PublicInbox/Search.pm | 2 +-
lib/PublicInbox/SharedKV.pm | 12 ++-
lib/PublicInbox/Spawn.pm | 13 ++-
script/lei | 88 +++++++++------
script/public-inbox-watch | 2 +-
t/cmd_ipc.t | 32 ++++++
t/imap_searchqp.t | 6 +-
t/lei.t | 33 +-----
t/lei_dedupe.t | 13 +++
t/lei_overview.t | 33 ++++++
xt/lei-sigpipe.t | 32 ++++++
22 files changed, 417 insertions(+), 255 deletions(-)
create mode 100644 t/lei_overview.t
create mode 100644 xt/lei-sigpipe.t
^ permalink raw reply [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-01-14 7:06 7% [PATCH 00/14] lei: another pile of changes Eric Wong
2021-01-14 7:06 6% ` [PATCH 08/14] lei q: reinstate smsg dedupe Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).