* does "lei init" even need to exist?
@ 2021-03-25 8:32 71% Eric Wong
2021-03-26 1:15 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-25 8:32 UTC (permalink / raw)
To: meta
All the commands just initialize the config and store on an
as-needed basis, so I'm starting to think it's an unnecessary
thing I mindlessly copied from git...
I'd honestly forgotten the command exists until now.
^ permalink raw reply [relevance 71%]
* Re: does "lei init" even need to exist?
2021-03-25 8:32 71% does "lei init" even need to exist? Eric Wong
@ 2021-03-26 1:15 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-26 1:15 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> All the commands just initialize the config and store on an
> as-needed basis, so I'm starting to think it's an unnecessary
> thing I mindlessly copied from git...
If somebody wants to use some place other than $XDG_DATA_HOME/lei/store,
I guess `lei init /other/location' provides an easy way to do that. I'm
not really sure there's much value in that, though.
^ permalink raw reply [relevance 71%]
* Re: is "lei mark" a good name?
@ 2021-03-26 1:54 71% ` Kyle Meyer
2021-03-26 9:48 68% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-03-26 1:54 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> It can set/unset volatile metadata for any number of messages.
>
> "Volatile metadata" being "labels" (aka "mailboxes" in
> JMAP-speak) and "keywords" (seen|flagged|answered|...),
> (aka "flags" in IMAP/Maildir-speak).
>
> "lei mark +kw:seen" # makes sense
> "lei mark +L:some-folder-name" # might sound odd...
>
>
> AFAIK, notmuch uses "notmuch tag" which combines both labels
> and keywords into one thing: "tags". But I'm also not a
> notmuch user...
(I am, though I still might be wrong.) Notmuch allows arbitrary tags to
be associated with message IDs. It has some automatic tags like
"attachment", "encrypted", and "signed" that it adds when indexing. By
default, it also syncs some maildir flags to its tags (e.g., "F ->
flagged", "R -> replied", "No S -> unread").
As far as I understand, tags are never connected up to the maildir
folder/location, though Notmuch does support a separate "folder:..."
search term.
So, I guess in JMAP terms, Notmuch tags would be "mailboxes" (a subset
of which are synced with maildir flags).
> Would "lei tag" be better?
Neither one really jumps out to me as better. "mark" sounds fine to me,
and I don't find "tag" any more or less odd for the +L case above.
> Anything else?
Nothing comes to mind.
^ permalink raw reply [relevance 71%]
* [PATCH 0/3] lei labels support
@ 2021-03-26 4:29 71% Eric Wong
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-03-26 4:29 UTC (permalink / raw)
To: meta
Unfortunately, it's limited to imported messages and
searching for labeled external messages doesn't work,
yet. I'm not sure how to best go about this...
Eric Wong (3):
lei_xsearch: wait for kw updates for non-threaded case, too
lei: _lei_store: use default even if unconfigured
lei: add some labels support
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 5 +--
lib/PublicInbox/LeiLsLabel.pm | 17 ++++++++++
lib/PublicInbox/LeiMark.pm | 6 ++--
lib/PublicInbox/LeiOverview.pm | 4 +--
lib/PublicInbox/LeiSearch.pm | 37 ++++++++++++++++++---
lib/PublicInbox/LeiStore.pm | 59 +++++++++++++++++++++++++---------
lib/PublicInbox/LeiXSearch.pm | 17 ++++++----
lib/PublicInbox/Search.pm | 6 ++--
lib/PublicInbox/SearchIdx.pm | 2 +-
t/lei-mark.t | 46 +++++++++++++++++++++++---
11 files changed, 160 insertions(+), 40 deletions(-)
create mode 100644 lib/PublicInbox/LeiLsLabel.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/3] lei: _lei_store: use default even if unconfigured
2021-03-26 4:29 71% [PATCH 0/3] lei labels support Eric Wong
@ 2021-03-26 4:29 71% ` Eric Wong
2021-03-26 5:01 71% ` [SQUASH 4/3] lei: account for unconfigured leistore.dir Eric Wong
2021-03-26 4:29 30% ` [PATCH 3/3] lei: add some labels support Eric Wong
2021-03-26 10:31 69% ` labels for externals [was: lei labels support] Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-26 4:29 UTC (permalink / raw)
To: meta
Perhaps leistore.dir doesn't need to have a config file
entry if we're using the default location.
---
lib/PublicInbox/LEI.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d534f1d0..b42ba0ae 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -719,8 +719,8 @@ sub _lei_store ($;$) {
my $cfg = _lei_cfg($self, $creat);
$cfg->{-lei_store} //= do {
require PublicInbox::LeiStore;
- my $dir = $cfg->{'leistore.dir'};
- $dir //= $creat ? store_path($self) : return;
+ my $dir = $cfg->{'leistore.dir'} // store_path($self);
+ return unless $creat || -d $dir;
PublicInbox::LeiStore->new($dir, { creat => $creat });
};
}
^ permalink raw reply related [relevance 71%]
* [PATCH 3/3] lei: add some labels support
2021-03-26 4:29 71% [PATCH 0/3] lei labels support Eric Wong
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
@ 2021-03-26 4:29 30% ` Eric Wong
2021-03-26 10:31 69% ` labels for externals [was: lei labels support] Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 4:29 UTC (permalink / raw)
To: meta
"lei q" now displays labels in JSON output, "lei mark"
can add or remove labels for any messages.
"lei ls-label" is supported, too.
Unfortunately, "lei q" won't hande "kw:" or "L:" for
external messages, they must be imported, first.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 1 +
lib/PublicInbox/LeiLsLabel.pm | 17 ++++++++++
lib/PublicInbox/LeiMark.pm | 6 ++--
lib/PublicInbox/LeiOverview.pm | 4 +--
lib/PublicInbox/LeiSearch.pm | 37 ++++++++++++++++++---
lib/PublicInbox/LeiStore.pm | 59 +++++++++++++++++++++++++---------
lib/PublicInbox/LeiXSearch.pm | 13 +++++---
lib/PublicInbox/Search.pm | 6 ++--
lib/PublicInbox/SearchIdx.pm | 2 +-
t/lei-mark.t | 46 +++++++++++++++++++++++---
11 files changed, 156 insertions(+), 36 deletions(-)
create mode 100644 lib/PublicInbox/LeiLsLabel.pm
diff --git a/MANIFEST b/MANIFEST
index 87e4b616..6b2b33ac 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -188,6 +188,7 @@ lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInput.pm
+lib/PublicInbox/LeiLsLabel.pm
lib/PublicInbox/LeiMark.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index b42ba0ae..fab2af90 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -145,6 +145,7 @@ our %CMD = ( # sorted in order of importance/use:
PublicInbox::LeiQuery::curl_opt() ],
'ls-external' => [ '[FILTER]', 'list publicinbox|extindex locations',
qw(format|f=s z|0 globoff|g invert-match|v local remote), @c_opt ],
+'ls-label' => [ '', 'list labels', qw(z|0 stats:s), @c_opt ],
'forget-external' => [ 'LOCATION...|--prune',
'exclude further results from a publicinbox|extindex',
qw(prune), @c_opt ],
diff --git a/lib/PublicInbox/LeiLsLabel.pm b/lib/PublicInbox/LeiLsLabel.pm
new file mode 100644
index 00000000..474224d4
--- /dev/null
+++ b/lib/PublicInbox/LeiLsLabel.pm
@@ -0,0 +1,17 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei ls-label" command
+package PublicInbox::LeiLsLabel;
+use strict;
+use v5.10.1;
+
+sub lei_ls_label { # the "lei ls-label" method
+ my ($lei, @argv) = @_;
+ # TODO: document stats/counts (expensive)
+ my @L = eval { $lei->_lei_store->search->all_terms('L') };
+ my $ORS = $lei->{opt}->{z} ? "\0" : "\n";
+ $lei->out(map { $_.$ORS } @L);
+}
+
+1;
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 9d77f4b4..7a2ccf77 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -60,7 +60,7 @@ sub vmd_mod_extract {
sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
my ($self, $eml) = @_;
if (my $xoids = $self->{lei}->{ale}->xoids_for($eml)) {
- $self->{lei}->{sto}->ipc_do('update_xvmd', $xoids,
+ $self->{lei}->{sto}->ipc_do('update_xvmd', $xoids, $eml,
$self->{vmd_mod});
} else {
++$self->{missing};
@@ -168,7 +168,9 @@ sub _complete_mark_common ($) {
# FIXME: same problems as _complete_forget_external and similar
sub _complete_mark {
my ($self, @argv) = @_;
- my @all = map { ("+kw:$_", "-kw:$_") } @KW;
+ my @L = eval { $self->_lei_store->search->all_terms('L') };
+ my @all = ((map { ("+kw:$_", "-kw:$_") } @KW),
+ (map { ("+L:$_", "-L:$_") } @L));
return @all if !@argv;
my ($cur, $re) = _complete_mark_common(\@argv);
map {
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 1ce2a098..b4d81328 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -227,7 +227,7 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
sub { # DIY prettiness :P
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
- $lse->xsmsg_vmd($smsg);
+ $lse->xsmsg_vmd($smsg, $smsg->{L} ? undef : 1);
$smsg = _unbless_smsg($smsg, $mitem);
$buf .= "{\n";
$buf .= join(",\n", map {
@@ -251,7 +251,7 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
sub {
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
- $lse->xsmsg_vmd($smsg);
+ $lse->xsmsg_vmd($smsg, $smsg->{L} ? undef : 1);
$buf .= $json->encode(_unbless_smsg(@_)) . $ORS;
return if length($buf) < 65536;
my $lk = $self->lock_for_scope;
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index bbb00661..07d570ec 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -27,18 +27,25 @@ sub msg_keywords {
wantarray ? sort(keys(%$kw)) : $kw;
}
+# lookup keywords+labels for external messages
sub xsmsg_vmd {
- my ($self, $smsg) = @_;
+ my ($self, $smsg, $want_label) = @_;
return if $smsg->{kw};
my $xdb = $self->xdb; # set {nshard};
- my %kw;
+ my (%kw, %L, $doc, $x);
$kw{flagged} = 1 if delete($smsg->{lei_q_tt_flagged});
my @num = $self->over->blob_exists($smsg->{blob});
for my $num (@num) { # there should only be one...
- my $kw = xap_terms('K', $xdb, num2docid($self, $num));
- %kw = (%kw, %$kw);
+ $doc = $xdb->get_document(num2docid($self, $num));
+ $x = xap_terms('K', $doc);
+ %kw = (%kw, %$x);
+ if ($want_label) { # JSON/JMAP only
+ $x = xap_terms('L', $doc);
+ %L = (%L, %$x);
+ }
}
$smsg->{kw} = [ sort keys %kw ] if scalar(keys(%kw));
+ $smsg->{L} = [ sort keys %L ] if scalar(keys(%L));
}
# when a message has no Message-IDs at all, this is needed for
@@ -100,4 +107,26 @@ sub kw_changed {
join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
}
+sub all_terms {
+ my ($self, $pfx) = @_;
+ my $xdb = $self->xdb;
+ my $cur = $xdb->allterms_begin($pfx);
+ my $end = $xdb->allterms_end($pfx);
+ my %ret;
+ for (; $cur != $end; $cur++) {
+ my $tn = $cur->get_termname;
+ index($tn, $pfx) == 0 and
+ $ret{substr($tn, length($pfx))} = undef;
+ }
+ wantarray ? (sort keys %ret) : \%ret;
+}
+
+sub qparse_new {
+ my ($self) = @_;
+ my $qp = $self->SUPER::qparse_new; # PublicInbox::Search
+ $qp->add_boolean_prefix('kw', 'K');
+ $qp->add_boolean_prefix('L', 'L');
+ $qp
+}
+
1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 1311ad46..b76af4d3 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -228,8 +228,30 @@ sub set_eml {
set_eml_vmd($self, $eml, $vmd);
}
+sub _external_only ($$$) {
+ my ($self, $xoids, $eml) = @_;
+ my $eidx = $self->{priv_eidx};
+ my $oidx = $eidx->{oidx} // die 'BUG: {oidx} missing';
+ my $smsg = bless { blob => '' }, 'PublicInbox::Smsg';
+ $smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
+ # save space for an externals-only message
+ my $hdr = $eml->header_obj;
+ $smsg->populate($hdr); # sets lines == 0
+ $smsg->{bytes} = 0;
+ delete @$smsg{qw(From Subject)};
+ $smsg->{to} = $smsg->{cc} = $smsg->{from} = '';
+ $oidx->add_overview($hdr, $smsg); # subject+references for threading
+ $smsg->{subject} = '';
+ for my $oid (keys %$xoids) {
+ $oidx->add_xref3($smsg->{num}, -1, $oid, '.');
+ }
+ my $idx = $eidx->idx_shard($smsg->{num});
+ $idx->index_eml(PublicInbox::Eml->new("\n\n"), $smsg);
+ ($smsg, $idx);
+}
+
sub update_xvmd {
- my ($self, $xoids, $vmd_mod) = @_;
+ my ($self, $xoids, $eml, $vmd_mod) = @_;
my $eidx = eidx_init($self);
my $oidx = $eidx->{oidx};
my %seen;
@@ -242,7 +264,25 @@ sub update_xvmd {
my $idx = $eidx->idx_shard($docid);
$idx->ipc_do('update_vmd', $docid, $vmd_mod);
}
+ delete $xoids->{$oid};
}
+ return unless scalar(keys(%$xoids));
+
+ # see if it was indexed, but with different OID(s)
+ if (my @docids = _docids_for($self, $eml)) {
+ for my $docid (@docids) {
+ next if $seen{$docid};
+ for my $oid (keys %$xoids) {
+ $oidx->add_xref3($docid, -1, $oid, '.');
+ }
+ my $idx = $eidx->idx_shard($docid);
+ $idx->ipc_do('update_vmd', $docid, $vmd_mod);
+ }
+ return;
+ }
+ # totally unseen
+ my ($smsg, $idx) = _external_only($self, $xoids, $eml);
+ $idx->ipc_do('update_vmd', $smsg->{num}, $vmd_mod);
}
# set or update keywords for external message, called via ipc_do
@@ -270,6 +310,7 @@ sub set_xvmd {
# see if it was indexed, but with different OID(s)
if (my @docids = _docids_for($self, $eml)) {
for my $docid (@docids) {
+ next if $seen{$docid};
for my $oid (keys %$xoids) {
$oidx->add_xref3($docid, -1, $oid, '.');
}
@@ -279,21 +320,7 @@ sub set_xvmd {
return;
}
# totally unseen
- my $smsg = bless { blob => '' }, 'PublicInbox::Smsg';
- $smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
- # save space for an externals-only message
- my $hdr = $eml->header_obj;
- $smsg->populate($hdr); # sets lines == 0
- $smsg->{bytes} = 0;
- delete @$smsg{qw(From Subject)};
- $smsg->{to} = $smsg->{cc} = $smsg->{from} = '';
- $oidx->add_overview($hdr, $smsg); # subject+references for threading
- $smsg->{subject} = '';
- for my $oid (keys %$xoids) {
- $oidx->add_xref3($smsg->{num}, -1, $oid, '.');
- }
- my $idx = $eidx->idx_shard($smsg->{num});
- $idx->index_eml(PublicInbox::Eml->new("\n\n"), $smsg);
+ my ($smsg, $idx) = _external_only($self, $xoids, $eml);
$idx->ipc_do('add_vmd', $smsg->{num}, $vmd);
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 386c4eba..f64b2c62 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -69,11 +69,13 @@ sub xdb_shards_flat { @{$_[0]->{shards_flat} // []} }
sub mitem_kw ($$;$) {
my ($smsg, $mitem, $flagged) = @_;
- my $kw = xap_terms('K', $mitem->get_document);
+ my $kw = xap_terms('K', my $doc = $mitem->get_document);
$kw->{flagged} = 1 if $flagged;
- # we keep the empty array here to prevent expensive work in
+ # we keep the empty {kw} array here to prevent expensive work in
# ->xsmsg_vmd, _unbless_smsg will clobber it iff it's empty
$smsg->{kw} = [ sort keys %$kw ];
+ my $L = xap_terms('L', $doc);
+ $smsg->{L} = [ sort keys %$L ] if scalar(keys %$L);
}
# like over->get_art
@@ -86,8 +88,10 @@ sub smsg_for {
my $num = int(($docid - 1) / $nshard) + 1;
my $ibx = $self->{shard2ibx}->[$shard];
my $smsg = $ibx->over->get_art($num);
- return if $smsg->{bytes} == 0;
- mitem_kw($smsg, $mitem) if $ibx->can('msg_keywords');
+ return if $smsg->{bytes} == 0; # external message
+ if ($ibx->can('msg_keywords')) {
+ mitem_kw($smsg, $mitem);
+ }
$smsg;
}
@@ -170,6 +174,7 @@ sub query_thread_mset { # for --threads
if ($can_kw) {
mitem_kw($smsg, $mitem, $fl);
} elsif ($fl) {
+ # call ->xsmsg_vmd, later
$smsg->{lei_q_tt_flagged} = 1;
}
}
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index c7d52daf..ab04d430 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -370,7 +370,7 @@ sub query_approxidate {
sub mset {
my ($self, $query_string, $opts) = @_;
$opts ||= {};
- my $qp = $self->{qp} //= qparse_new($self);
+ my $qp = $self->{qp} //= $self->qparse_new;
my $query = $qp->parse_query($query_string, $self->{qp_flags});
_do_enquire($self, $query, $opts);
}
@@ -463,7 +463,7 @@ sub mset_to_smsg {
sub stemmer { $X{Stem}->new($LANG) }
# read-only
-sub qparse_new ($) {
+sub qparse_new {
my ($self) = @_;
my $xdb = xdb($self);
@@ -516,7 +516,7 @@ EOF
sub help {
my ($self) = @_;
- $self->{qp} //= qparse_new($self); # parse altids
+ $self->{qp} //= $self->qparse_new; # parse altids
my @ret = @HELP;
if (my $user_pfx = $self->{-user_pfx}) {
push @ret, @$user_pfx;
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 7d46489c..ca1f3588 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -35,7 +35,7 @@ use constant DEBUG => !!$ENV{DEBUG};
my $xapianlevels = qr/\A(?:full|medium)\z/;
my $hex = '[a-f0-9]';
my $OID = $hex .'{40,}';
-my @VMD_MAP = (kw => 'K', label => 'L');
+my @VMD_MAP = (kw => 'K', L => 'L');
our $INDEXLEVELS = qr/\A(?:full|medium|basic)\z/;
sub new {
diff --git a/t/lei-mark.t b/t/lei-mark.t
index 76995589..23f5002e 100644
--- a/t/lei-mark.t
+++ b/t/lei-mark.t
@@ -4,22 +4,32 @@
use strict; use v5.10.1; use PublicInbox::TestCommon;
require_git 2.6;
require_mods(qw(json DBD::SQLite Search::Xapian));
+my ($ro_home, $cfg_path) = setup_public_inboxes;
my $check_kw = sub {
my ($exp, %opt) = @_;
+ my $args = $opt{args} // [];
my $mid = $opt{mid} // 'testmessage@example.com';
- lei_ok('q', "m:$mid");
+ lei_ok('q', "m:$mid", @$args);
my $res = json_utf8->decode($lei_out);
is($res->[1], undef, 'only got one result');
my $msg = $opt{msg} ? " $opt{msg}" : '';
($exp ? is_deeply($res->[0]->{kw}, $exp, "got @$exp$msg")
: is($res->[0]->{kw}, undef, "got undef$msg")) or
diag explain($res);
+ if (exists $opt{L}) {
+ $exp = $opt{L};
+ ($exp ? is_deeply($res->[0]->{L}, $exp, "got @$exp$msg")
+ : is($res->[0]->{L}, undef, "got undef$msg")) or
+ diag explain($res);
+ }
};
test_lei(sub {
+ lei_ok(qw(ls-label)); is($lei_out, '', 'no labels, yet');
lei_ok(qw(import -F eml t/utf8.eml));
- lei_ok(qw(mark -F eml t/utf8.eml +kw:flagged));
- $check_kw->(['flagged']);
+ lei_ok(qw(mark -F eml t/utf8.eml +kw:flagged +L:urgent));
+ $check_kw->(['flagged'], L => ['urgent']);
+ lei_ok(qw(ls-label)); is($lei_out, "urgent\n", 'label found');
ok(!lei(qw(mark -F eml t/utf8.eml +kw:seeen)), 'bad kw rejected');
like($lei_err, qr/`seeen' is not one of/, 'got helpful error');
ok(!lei(qw(mark -F eml t/utf8.eml +k:seen)), 'bad prefix rejected');
@@ -41,7 +51,35 @@ test_lei(sub {
$check_kw->(['answered'], msg => 'Maildir Status ignored');
open my $in, '<', 't/utf8.eml' or BAIL_OUT $!;
- lei_ok([qw(mark -F eml - +kw:seen)], undef, { %$lei_opt, 0 => $in });
+ lei_ok([qw(mark -F eml - +kw:seen +L:nope)],
+ undef, { %$lei_opt, 0 => $in });
$check_kw->(['answered', 'seen'], msg => 'stdin works');
+ lei_ok(qw(q L:urgent));
+ my $res = json_utf8->decode($lei_out);
+ is($res->[0]->{'m'}, 'testmessage@example.com', 'L: query works');
+ lei_ok(qw(q kw:seen));
+ my $r2 = json_utf8->decode($lei_out);
+ is_deeply($r2, $res, 'kw: query works, too') or
+ diag explain([$r2, $res]);
+
+ lei_ok(qw(_complete lei mark));
+ my %c = map { $_ => 1 } split(/\s+/, $lei_out);
+ ok($c{'+L:urgent'} && $c{'-L:urgent'} &&
+ $c{'+L:nope'} && $c{'-L:nope'}, 'completed with labels');
+
+ my $mid = 'qp@example.com';
+ lei_ok qw(q -f mboxrd --only), "$ro_home/t2", "mid:$mid";
+ $in = $lei_out;
+ lei_ok [qw(mark -F mboxrd --stdin +kw:seen +L:qp)],
+ undef, { %$lei_opt, 0 => \$in };
+ $check_kw->(['seen'], L => ['qp'], mid => $mid,
+ args => [ '--only', "$ro_home/t2" ],
+ msg => 'external-only message');
+ lei_ok(qw(ls-label));
+ is($lei_out, "nope\nqp\nurgent\n", 'ls-label shows qp');
+
+ if (0) { # TODO label+kw search w/ externals
+ lei_ok(qw(q L:qp), "mid:$mid", '--only', "$ro_home/t2");
+ }
});
done_testing;
^ permalink raw reply related [relevance 30%]
* [SQUASH 4/3] lei: account for unconfigured leistore.dir
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
@ 2021-03-26 5:01 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 5:01 UTC (permalink / raw)
To: meta
We just use the default directory if it's not in the
config.
---
lib/PublicInbox/LEI.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fab2af90..6a5c32b3 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -702,8 +702,9 @@ sub _lei_cfg ($;$) {
bless $cfg, 'PublicInbox::Config';
$cfg->{-st} = $cur_st;
$cfg->{'-f'} = $f;
- if ($sto && File::Spec->canonpath($sto_dir) eq
- File::Spec->canonpath($cfg->{'leistore.dir'})) {
+ if ($sto && File::Spec->canonpath($sto_dir // store_path($self))
+ eq File::Spec->canonpath($cfg->{'leistore.dir'} //
+ store_path($self))) {
$cfg->{-lei_store} = $sto;
}
if (scalar(keys %PATH2CFG) > 5) {
^ permalink raw reply related [relevance 71%]
* Re: is "lei mark" a good name?
2021-03-26 1:54 71% ` Kyle Meyer
@ 2021-03-26 9:48 68% ` Eric Wong
2021-03-28 2:21 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-26 9:48 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
>
> > It can set/unset volatile metadata for any number of messages.
> >
> > "Volatile metadata" being "labels" (aka "mailboxes" in
> > JMAP-speak) and "keywords" (seen|flagged|answered|...),
> > (aka "flags" in IMAP/Maildir-speak).
> >
> > "lei mark +kw:seen" # makes sense
>
> > "lei mark +L:some-folder-name" # might sound odd...
> >
> >
> > AFAIK, notmuch uses "notmuch tag" which combines both labels
> > and keywords into one thing: "tags". But I'm also not a
> > notmuch user...
>
> (I am, though I still might be wrong.) Notmuch allows arbitrary tags to
> be associated with message IDs. It has some automatic tags like
> "attachment", "encrypted", and "signed" that it adds when indexing.
Hmm..., JMAP has a "hasAttachment" property to search on.
> By default, it also syncs some maildir flags to its tags (e.g., "F ->
> flagged", "R -> replied", "No S -> unread").
>
> As far as I understand, tags are never connected up to the maildir
> folder/location, though Notmuch does support a separate "folder:..."
> search term.
I've been thinking about storing source location data, too.
Having the source of an email in lei/store would make it easier
and faster to export keywords back to IMAP/Maildir.
> So, I guess in JMAP terms, Notmuch tags would be "mailboxes" (a subset
> of which are synced with maildir flags).
The arbitrary nm tags (and "attachment|signed|encrypted) would be
"mailboxes", yes. But the Maildir flags are JMAP keywords:
# LeiToMail.pm:
my %kw2char = ( # Maildir characters
draft => 'D',
flagged => 'F',
answered => 'R',
seen => 'S'
);
> > Would "lei tag" be better?
>
> Neither one really jumps out to me as better. "mark" sounds fine to me,
> and I don't find "tag" any more or less odd for the +L case above.
OK. I'm still on the fence...
On one hand, "tag" is probably a more familiar term to users of
existing software. Not just notmuch, but also other software
for music metadata, software (debtags), etc...
On the other hand, lei will support "lei show" to wrap "git show"
with SolverGit support. Thus I'm worried "lei tag" might
be misconstrued as a wrapper for "git tag"...
But now that I think about it more; "lei show" probably won't
treat git tree/tag/commit objects any differently than "git show".
Perhaps "lei show" can be named "lei blob", instead.
Originally, I intended "lei show" to do something with commit
objects; but we already have "lei p2q" which lends itself
better to composability.
^ permalink raw reply [relevance 68%]
* [PATCH 0/4] lei minor things
@ 2021-03-26 9:51 71% Eric Wong
2021-03-26 9:51 69% ` [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
Been putting off 3/4 off for a bit, still thinking about
labels a bit...
Eric Wong (4):
lei q: skip lei/store->write_prepare for JSON outputs
lei: do not blindly commit to lei/store on close
lei: support /dev/fd/[0-2] inputs and outputs in daemon
lei mark: disallow '!' in labels
lib/PublicInbox/LEI.pm | 26 ++++++++++----------------
lib/PublicInbox/LeiConvert.pm | 3 ++-
lib/PublicInbox/LeiInput.pm | 5 ++++-
lib/PublicInbox/LeiMark.pm | 2 +-
lib/PublicInbox/LeiOverview.pm | 17 +++++++++--------
lib/PublicInbox/LeiP2q.pm | 7 +++++--
lib/PublicInbox/LeiQuery.pm | 16 ++++++++--------
lib/PublicInbox/LeiToMail.pm | 28 ++++++++++++++++------------
lib/PublicInbox/LeiXSearch.pm | 2 ++
t/lei-convert.t | 2 +-
t/lei-import.t | 2 +-
11 files changed, 59 insertions(+), 51 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 2/4] lei: do not blindly commit to lei/store on close
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
2021-03-26 9:51 69% ` [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs Eric Wong
@ 2021-03-26 9:51 90% ` Eric Wong
2021-03-26 9:51 42% ` [PATCH 3/4] lei: support /dev/fd/[0-2] inputs and outputs in daemon Eric Wong
2021-03-26 9:51 71% ` [PATCH 4/4] lei mark: disallow '!' in labels Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
It may hide errors/bugs, instead do it explicitly for each
worker that writes to it. For lei_xsearch, it will be better
to close before spawning the MUA for future use since we may
need it again once the user starts changing keywords.
---
lib/PublicInbox/LEI.pm | 3 ---
lib/PublicInbox/LeiXSearch.pm | 1 +
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 6a5c32b3..59715633 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1030,9 +1030,6 @@ sub dclose {
}
}
close(delete $self->{1}) if $self->{1}; # may reap_compress
- if (my $sto = delete $self->{sto}) {
- $sto->ipc_do('done');
- }
$self->close if $self->{-event_init_done}; # PublicInbox::DS::close
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 6410e0ea..b41daffe 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -319,6 +319,7 @@ sub query_done { # EOF callback for main daemon
if (my $lxs = delete $lei->{lxs}) {
$lxs->wq_wait_old(\&xsearch_done_wait, $lei);
}
+ my $wait = $lei->{sto} ? $lei->{sto}->ipc_do('done') : undef;
$lei->{ovv}->ovv_end($lei);
if ($l2m) { # close() calls LeiToMail reap_compress
if (my $out = delete $lei->{old_1}) {
^ permalink raw reply related [relevance 90%]
* [PATCH 4/4] lei mark: disallow '!' in labels
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
` (2 preceding siblings ...)
2021-03-26 9:51 42% ` [PATCH 3/4] lei: support /dev/fd/[0-2] inputs and outputs in daemon Eric Wong
@ 2021-03-26 9:51 71% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
'!' could wreak havoc if exposed to a shell like bash. It seems
like a rare character for use in file/directory/mailbox names.
---
lib/PublicInbox/LeiMark.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 7a2ccf77..6d236411 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -23,7 +23,7 @@ my %ERR = (
my ($label) = @_;
length($label) >= $L_MAX and
return "`$label' too long (must be <= $L_MAX)";
- $label =~ m{\A[a-z0-9_][a-z0-9_\-\./\@\!,]*[a-z0-9]\z} ?
+ $label =~ m{\A[a-z0-9_][a-z0-9_\-\./\@,]*[a-z0-9]\z} ?
undef : "`$label' is invalid";
},
kw => sub {
^ permalink raw reply related [relevance 71%]
* [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
@ 2021-03-26 9:51 69% ` Eric Wong
2021-03-26 9:51 90% ` [PATCH 2/4] lei: do not blindly commit to lei/store on close Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
JSON outputs won't write to lei/store at all, so there's
no point in forking the store worker if it's not already
running.
LeiSearch object ($lse) is also fork-safe until it opens a
persistent FD for Xapian/SQLite so we can unconditionally
carry it across fork.
---
lib/PublicInbox/LeiOverview.pm | 4 ++--
lib/PublicInbox/LeiQuery.pm | 16 ++++++++--------
lib/PublicInbox/LeiToMail.pm | 16 ++++++++--------
lib/PublicInbox/LeiXSearch.pm | 1 +
4 files changed, 19 insertions(+), 18 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index b4d81328..96bfff24 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -223,7 +223,7 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
}
} elsif ($self->{fmt} =~ /\A(concat)?json\z/ && $lei->{opt}->{pretty}) {
my $EOR = ($1//'') eq 'concat' ? "\n}" : "\n},";
- my $lse = $lei->{sto}->search;
+ my $lse = $lei->{lse};
sub { # DIY prettiness :P
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
@@ -247,7 +247,7 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
}
} elsif ($json) {
my $ORS = $self->{fmt} eq 'json' ? ",\n" : "\n"; # JSONL
- my $lse = $lei->{sto}->search;
+ my $lse = $lei->{lse};
sub {
my ($smsg, $mitem) = @_;
return if $dedupe->is_smsg_dup($smsg);
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 84996e7e..65aa9e87 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -25,8 +25,7 @@ sub qstr_add { # PublicInbox::InputPipe::consume callback for --stdin
my ($self) = @_; # $_[1] = $rbuf
if (defined($_[1])) {
$_[1] eq '' and return eval {
- my $lse = delete $self->{lse};
- $lse->query_approxidate($lse->git,
+ $self->{lse}->query_approxidate($self->{lse}->git,
$self->{mset_opt}->{qstr});
_start_query($self);
};
@@ -50,11 +49,7 @@ sub lei_q {
# --local is enabled by default unless --only is used
# we'll allow "--only $LOCATION --local"
my $sto = $self->_lei_store(1);
- if (($opt->{'import-remote'} //= 1) |
- (($opt->{'import-before'} //= \1) ? 1 : 0)) {
- $sto->write_prepare($self);
- }
- my $lse = $sto->search;
+ my $lse = $self->{lse} = $sto->search;
if ($opt->{'local'} //= scalar(@only) ? 0 : 1) {
$lxs->prepare_external($lse);
}
@@ -103,6 +98,12 @@ sub lei_q {
return $self->fail("`$mj' writer jobs must be >= 1");
}
PublicInbox::LeiOverview->new($self) or return;
+ if ($self->{l2m} && ($opt->{'import-remote'} //= 1) |
+ # we use \1 (a ref) to distinguish between
+ # user-supplied and default value
+ (($opt->{'import-before'} //= \1) ? 1 : 0)) {
+ $sto->write_prepare($self);
+ }
$self->{l2m} and $self->{l2m}->{-wq_nr_workers} = $mj // do {
$mj = POSIX::lround($nproc * 3 / 4); # keep some CPU for git
$mj <= 0 ? 1 : $mj;
@@ -131,7 +132,6 @@ sub lei_q {
no query allowed on command-line with --stdin
require PublicInbox::InputPipe;
- $self->{lse} = $lse; # for query_approxidate
PublicInbox::InputPipe::consume($self->{0}, \&qstr_add, $self);
return;
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 1be15707..f71f74cc 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -263,7 +263,7 @@ sub _mbox_write_cb ($$) {
my $atomic_append = !defined($ovv->{lock_path});
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe;
- my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
+ my $lse = $lei->{lse}; # may be undef
sub { # for git_to_mail
my ($buf, $smsg, $eml) = @_;
$eml //= PublicInbox::Eml->new($buf);
@@ -352,7 +352,7 @@ sub _maildir_write_cb ($$) {
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe if $dedupe;
my $dst = $lei->{ovv}->{dst};
- my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
+ my $lse = $lei->{lse}; # may be undef
sub { # for git_to_mail
my ($buf, $smsg, $eml) = @_;
$dst // return $lei->fail; # dst may be undef-ed in last run
@@ -373,7 +373,7 @@ sub _imap_write_cb ($$) {
my $imap_append = $lei->{net}->can('imap_append');
my $mic = $lei->{net}->mic_get($self->{uri});
my $folder = $self->{uri}->mailbox;
- my $lse = $lei->{sto} ? $lei->{sto}->search : undef;
+ my $lse = $lei->{lse}; # may be undef
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$mic // return $lei->fail; # dst may be undef-ed in last run
@@ -449,7 +449,7 @@ sub _pre_augment_maildir {
sub _do_augment_maildir {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
- my $lse = $lei->{sto}->search if $lei->{opt}->{'import-before'};
+ my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
my ($mod, $shard) = @{$self->{shard_info} // []};
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
@@ -481,7 +481,7 @@ sub _imap_augment_or_delete { # PublicInbox::NetReader::imap_each cb
sub _do_augment_imap {
my ($self, $lei) = @_;
my $net = $lei->{net};
- my $lse = $lei->{sto}->search if $lei->{opt}->{'import-before'};
+ my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
if ($dedupe && $dedupe->prepare_dedupe) {
@@ -523,9 +523,9 @@ sub _pre_augment_mbox {
die "seek($dst): $!\n";
}
if (!$self->{seekable}) {
- my $ia = $lei->{opt}->{'import-before'};
+ my $imp_before = $lei->{opt}->{'import-before'};
die "--import-before specified but $dst is not seekable\n"
- if $ia && !ref($ia);
+ if $imp_before && !ref($imp_before);
die "--augment specified but $dst is not seekable\n" if
$lei->{opt}->{augment};
}
@@ -562,7 +562,7 @@ sub _do_augment_mbox {
$dedupe->prepare_dedupe if $dedupe;
}
if ($opt->{'import-before'}) { # the default
- my $lse = $lei->{sto}->search;
+ my $lse = $lei->{lse};
PublicInbox::MboxReader->$fmt($rd, \&_mbox_augment_kw_maybe,
$lei, $lse, $opt->{augment});
if (!$opt->{augment} and !truncate($out, 0)) {
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index f64b2c62..6410e0ea 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -430,6 +430,7 @@ sub do_query {
$lei->{1}->autoflush(1);
$lei->start_pager if delete $lei->{need_pager};
$lei->{ovv}->ovv_begin($lei);
+ die 'BUG: xdb|over open' if $lei->{lse}->{xdb} || $lei->{lse}->{over};
if ($l2m) {
$l2m->pre_augment($lei);
if ($lei->{opt}->{augment} && delete $lei->{early_mua}) {
^ permalink raw reply related [relevance 69%]
* [PATCH 3/4] lei: support /dev/fd/[0-2] inputs and outputs in daemon
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
2021-03-26 9:51 69% ` [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs Eric Wong
2021-03-26 9:51 90% ` [PATCH 2/4] lei: do not blindly commit to lei/store on close Eric Wong
@ 2021-03-26 9:51 42% ` Eric Wong
2021-03-26 9:51 71% ` [PATCH 4/4] lei mark: disallow '!' in labels Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 9:51 UTC (permalink / raw)
To: meta
Since lei-daemon won't have the same FDs as the client, we
need to special-case thse mappings and won't be able to open
arbitrary, non-standard FDs.
We also won't attempt to support /proc/self/fd/[0-2] since
that's a Linux-ism. /dev/fd/[0-2] and /dev/std{in,out,err}
are portable to FreeBSD, at least. mawk(1) also supports
/dev/std{out,err}, as does gawk(1) (which supports everything
we can support, and arbitrary /dev/fd/$FD).
---
lib/PublicInbox/LEI.pm | 23 ++++++++++-------------
lib/PublicInbox/LeiConvert.pm | 3 ++-
lib/PublicInbox/LeiInput.pm | 5 ++++-
lib/PublicInbox/LeiOverview.pm | 13 +++++++------
lib/PublicInbox/LeiP2q.pm | 7 +++++--
lib/PublicInbox/LeiToMail.pm | 12 ++++++++----
t/lei-convert.t | 2 +-
t/lei-import.t | 2 +-
8 files changed, 38 insertions(+), 29 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 59715633..eb3ad9e2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -464,7 +464,6 @@ sub _lei_atfork_child {
}
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
- delete $self->{0};
}
for (delete @$self{qw(3 old_1 au_done)}) {
close($_) if defined($_);
@@ -929,19 +928,17 @@ sub poke_mua { # forces terminal MUAs to wake up and hopefully notice new mail
}
my %path_to_fd = ('/dev/stdin' => 0, '/dev/stdout' => 1, '/dev/stderr' => 2);
-$path_to_fd{"/dev/fd/$_"} = $path_to_fd{"/proc/self/fd/$_"} for (0..2);
-sub fopen {
- my ($self, $mode, $path) = @_;
- rel2abs($self, $path);
+$path_to_fd{"/dev/fd/$_"} = $_ for (0..2);
+
+# this also normalizes the path
+sub path_to_fd {
+ my ($self, $path) = @_;
+ $path = rel2abs($self, $path);
$path =~ tr!/!/!s;
- if (defined(my $fd = $path_to_fd{$path})) {
- return $self->{$fd};
- }
- if ($path =~ m!\A/(?:dev|proc/self)/fd/[0-9]+\z!) {
- return fail($self, "cannot open $path from daemon");
- }
- open my $fh, $mode, $path or return;
- $fh;
+ $path_to_fd{$path} // (
+ ($path =~ m!\A/(?:dev|proc/self)/fd/[0-9]+\z!) ?
+ fail($self, "cannot open $path from daemon") : -1
+ );
}
# caller needs to "-t $self->{1}" to check if tty
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 0cc65108..083ecc33 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -50,7 +50,8 @@ sub lei_convert { # the main "lei convert" method
my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format');
$lei->{l2m} or return
$lei->fail("output not specified or is not a mail destination");
- $lei->{opt}->{augment} = 1 unless $ovv->{dst} eq '/dev/stdout';
+ my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
+ $lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
my $op = $lei->workers_start($self, 'lei_convert', 1);
$lei->{cnv} = $self;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index b059ecda..eed0eed7 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -67,7 +67,10 @@ sub input_path_url {
return;
}
$input =~ s!\A([a-z0-9]+):!!i and $ifmt = lc($1);
- if (-f $input) {
+ my $devfd = $lei->path_to_fd($input) // return;
+ if ($devfd >= 0) {
+ $self->input_fh($ifmt, $lei->{$devfd}, $input, @args);
+ } elsif (-f $input) {
my $m = $lei->{opt}->{'lock'} // ($ifmt eq 'eml' ? ['none'] :
PublicInbox::MboxLock->defaults);
my $mbl = PublicInbox::MboxLock->acq($input, 0, $m);
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 96bfff24..8e26cba4 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -71,8 +71,9 @@ sub new {
--$ofmt_key=$fmt and --output=$ofmt conflict
}
- $fmt //= 'json' if $dst eq '/dev/stdout';
- $fmt //= detect_fmt($lei, $dst) or return;
+
+ my $devfd = $lei->path_to_fd($dst) // return;
+ $fmt //= $devfd >= 0 ? 'json' : (detect_fmt($lei, $dst) or return);
if (index($dst, '://') < 0) { # not a URL, so assume path
$dst = File::Spec->canonpath($dst);
@@ -84,11 +85,11 @@ sub new {
if ($fmt =~ /\A($JSONL|(?:concat)?json)\z/) {
$json = $self->{json} = ref(PublicInbox::Config->json);
}
- if ($dst eq '/dev/stdout') {
- my $isatty = $lei->{need_pager} = -t $lei->{1};
+ if ($devfd >= 0) {
+ my $isatty = $lei->{need_pager} = -t $lei->{$devfd};
$opt->{pretty} //= $isatty;
if (!$isatty && -f _) {
- my $fl = fcntl($lei->{1}, F_GETFL, 0) //
+ my $fl = fcntl($lei->{$devfd}, F_GETFL, 0) //
return $lei->fail("fcntl(stdout): $!");
ovv_out_lk_init($self) unless ($fl & O_APPEND);
} else {
@@ -101,7 +102,7 @@ sub new {
$lei->{dedupe} //= PublicInbox::LeiDedupe->new($lei);
} else {
# default to the cheapest sort since MUA usually resorts
- $opt->{'sort'} //= 'docid' if $dst ne '/dev/stdout';
+ $opt->{'sort'} //= 'docid' if $devfd < 0;
$lei->{l2m} = eval { PublicInbox::LeiToMail->new($lei) };
return $lei->fail($@) if $@;
if ($opt->{mua} && $lei->{l2m}->lock_free) {
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index fda055fe..25f63a10 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -107,8 +107,11 @@ sub do_p2q { # via wq_do
my $in = $self->{0};
unless ($in) {
my $input = $self->{input};
- if (-e $input) {
- $in = $lei->fopen('<', $input) or
+ my $devfd = $lei->path_to_fd($input) // return;
+ if ($devfd >= 0) {
+ $in = $lei->{$devfd};
+ } elsif (-e $input) {
+ open($in, '<', $input) or
return $lei->fail("open < $input: $!");
} else {
my @cmd = (qw(git format-patch --stdout -1), $input);
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index f71f74cc..88468c34 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -503,10 +503,12 @@ sub _do_augment_imap {
sub _pre_augment_mbox {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
- my $out = $lei->{1};
- if ($dst ne '/dev/stdout') {
+ my $out;
+ my $devfd = $lei->path_to_fd($dst) // die "bad $dst";
+ if ($devfd >= 0) {
+ $out = $lei->{$devfd};
+ } else { # normal-looking path
if (-p $dst) {
- $out = undef;
open $out, '>', $dst or die "open($dst): $!";
} elsif (-f _ || !-e _) {
require PublicInbox::MboxLock;
@@ -514,12 +516,14 @@ sub _pre_augment_mbox {
PublicInbox::MboxLock->defaults;
$self->{mbl} = PublicInbox::MboxLock->acq($dst, 1, $m);
$out = $self->{mbl}->{fh};
+ } else {
+ die "$dst is not a file or FIFO\n";
}
$lei->{old_1} = $lei->{1}; # keep for spawning MUA
}
# Perl does SEEK_END even with O_APPEND :<
$self->{seekable} = seek($out, 0, SEEK_SET);
- if (!$self->{seekable} && $! != ESPIPE && $dst ne '/dev/stdout') {
+ if (!$self->{seekable} && $! != ESPIPE && !defined($devfd)) {
die "seek($dst): $!\n";
}
if (!$self->{seekable}) {
diff --git a/t/lei-convert.t b/t/lei-convert.t
index e147715d..9b430d8e 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -87,7 +87,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
my $exp = do { local $/; <$fh> };
is($out, $exp, 'stdin => stdout');
- lei_ok qw(convert -F eml -o mboxcl2:/dev/stdout t/plack-qp.eml);
+ lei_ok qw(convert -F eml -o mboxcl2:/dev/fd/1 t/plack-qp.eml);
open $fh, '<', \$lei_out or BAIL_OUT;
@bar = ();
PublicInbox::MboxReader->mboxcl2($fh, sub {
diff --git a/t/lei-import.t b/t/lei-import.t
index a697d756..fa40ad01 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -69,7 +69,7 @@ is($res->[0]->{kw}, undef, 'no keywords set');
$eml->header_set('Message-ID', '<k@y>');
$in = 'From k@y Fri Oct 2 00:00:00 1993'."\n".$eml->as_string;
-lei_ok([qw(import -F mboxrd -)], undef, { %$lei_opt, 0 => \$in },
+lei_ok([qw(import -F mboxrd /dev/fd/0)], undef, { %$lei_opt, 0 => \$in },
\'import single file with --kw (default) from stdin');
lei(qw(q m:k@y));
$res = json_utf8->decode($lei_out);
^ permalink raw reply related [relevance 42%]
* labels for externals [was: lei labels support]
2021-03-26 4:29 71% [PATCH 0/3] lei labels support Eric Wong
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
2021-03-26 4:29 30% ` [PATCH 3/3] lei: add some labels support Eric Wong
@ 2021-03-26 10:31 69% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-26 10:31 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> Unfortunately, it's limited to imported messages and
> searching for labeled external messages doesn't work,
> yet. I'm not sure how to best go about this...
I'm thinking adding a label to an external-only message
should cause it to be imported into lei/store.
Related to "lei: per-message keywords and externals" at
https://public-inbox.org/meta/20210224204950.GA2076@dcvr/
in that it's technically the same problem, but not semantically.
The key difference between keywords and labels is labels require
explicit, concious effort on a user's part to "lei mark" a
message. Even within an MUA, copying/moving a message to a new
mailbox requires explicit user action.
Keyword updates are largely transparent from the MUA, with
"seen" is the most common keyword. Default MUA behavior is to
set the "seen" keyword on any read message. "seen" keyword
updates will be even more transparent when inotify/IMAP IDLE
support is added to lei.
So again, I think users should get an automatic import when they
label. Maybe implementing indexlevel=medium on a per-message level
could be used to save space.
git storage shouldn't be too expensive, and alternates can be
used safely unless the external was is subject to
-purge/edit-prone
^ permalink raw reply [relevance 69%]
* [PATCH 0/4] lei blob (formerly known as "lei show")
@ 2021-03-27 11:45 90% Eric Wong
2021-03-27 11:45 90% ` [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm Eric Wong
2021-03-27 11:45 38% ` [PATCH 4/4] lei blob: aka "git-show-harder" for blobs Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-03-27 11:45 UTC (permalink / raw)
To: meta
It's still a "git show" with blob reconstruction (aka "solver")
I'm not sure if lei <add|ls|forget>-coderepo is necessary,
since this supports multiple --git-dir args (and uses
the current working directory).
Maybe submodule directories could be scanned...
Eric Wong (4):
lei_ale: do not create store unnecessarily
lei help: move "lei help" into LeiHelp.pm
lei_query: hoist out lxs_prepare
lei blob: aka "git-show-harder" for blobs
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 26 ++++----
lib/PublicInbox/LeiALE.pm | 3 +-
lib/PublicInbox/LeiBlob.pm | 119 ++++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiHelp.pm | 3 +
lib/PublicInbox/LeiQuery.pm | 38 +++++++-----
t/lei-import.t | 2 +
t/solver_git.t | 14 ++++-
8 files changed, 173 insertions(+), 33 deletions(-)
create mode 100644 lib/PublicInbox/LeiBlob.pm
^ permalink raw reply [relevance 90%]
* [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm
2021-03-27 11:45 90% [PATCH 0/4] lei blob (formerly known as "lei show") Eric Wong
@ 2021-03-27 11:45 90% ` Eric Wong
2021-03-27 11:45 38% ` [PATCH 4/4] lei blob: aka "git-show-harder" for blobs Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-27 11:45 UTC (permalink / raw)
To: meta
We need to load LeiHelp.pm anyways if somebody calls "lei help",
so save a few kB RAM for users who don't need help.
---
lib/PublicInbox/LEI.pm | 2 --
lib/PublicInbox/LeiHelp.pm | 3 +++
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index eb3ad9e2..e680f5f0 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -787,8 +787,6 @@ sub lei_daemon_kill {
kill($sig, $$) or fail($self, "kill($sig, $$): $!");
}
-sub lei_help { _help($_[0]) }
-
# Shell completion helper. Used by lei-completion.bash and hopefully
# other shells. Try to do as much here as possible to avoid redundancy
# and improve maintainability.
diff --git a/lib/PublicInbox/LeiHelp.pm b/lib/PublicInbox/LeiHelp.pm
index be31c2a8..9c1b30a1 100644
--- a/lib/PublicInbox/LeiHelp.pm
+++ b/lib/PublicInbox/LeiHelp.pm
@@ -97,4 +97,7 @@ EOF
undef;
}
+# the "lei help" command
+sub lei_help { $_[0]->_help }
+
1;
^ permalink raw reply related [relevance 90%]
* [PATCH 4/4] lei blob: aka "git-show-harder" for blobs
2021-03-27 11:45 90% [PATCH 0/4] lei blob (formerly known as "lei show") Eric Wong
2021-03-27 11:45 90% ` [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm Eric Wong
@ 2021-03-27 11:45 38% ` Eric Wong
2021-03-27 20:20 71% ` [SQUASH] lei blob: use absolute path Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-03-27 11:45 UTC (permalink / raw)
To: meta
This implements blob reconstruction via SolverGit,
emulating the functionality of /$INBOX/$OID/s/ endpoint
in PublicInbox::WWW.
It uses the current working tree as a coderepo, and
accepts any number of --git-dir=$PATH args.
Remote externals are not yet supported.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 24 ++++----
lib/PublicInbox/LeiBlob.pm | 119 +++++++++++++++++++++++++++++++++++++
t/lei-import.t | 2 +
t/solver_git.t | 14 ++++-
5 files changed, 146 insertions(+), 14 deletions(-)
create mode 100644 lib/PublicInbox/LeiBlob.pm
diff --git a/MANIFEST b/MANIFEST
index 6b2b33ac..64b3626f 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -181,6 +181,7 @@ lib/PublicInbox/KQNotify.pm
lib/PublicInbox/LEI.pm
lib/PublicInbox/LeiALE.pm
lib/PublicInbox/LeiAuth.pm
+lib/PublicInbox/LeiBlob.pm
lib/PublicInbox/LeiConvert.pm
lib/PublicInbox/LeiCurl.pm
lib/PublicInbox/LeiDedupe.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index e680f5f0..478912cd 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -120,6 +120,9 @@ sub index_opt {
}
my @c_opt = qw(c=s@ C=s@ quiet|q);
+my @lxs_opt = (qw(remote! local! external! include|I=s@ exclude=s@ only=s@
+ import-remote! no-torsocks torsocks=s),
+ PublicInbox::LeiQuery::curl_opt());
# we generate shell completion + help using %CMD and %OPTDESC,
# see lei__complete() and PublicInbox::LeiHelp
@@ -127,16 +130,15 @@ my @c_opt = qw(c=s@ C=s@ quiet|q);
our %CMD = ( # sorted in order of importance/use:
'q' => [ '--stdin|SEARCH_TERMS...', 'search for messages matching terms',
'stdin|', # /|\z/ must be first for lone dash
+ @lxs_opt,
qw(save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+
- sort|s=s reverse|r offset=i remote! local! external! pretty
- include|I=s@ exclude=s@ only=s@ jobs|j=s globoff|g augment|a
- import-remote! import-before! lock=s@ rsyncable
- alert=s@ mua=s no-torsocks torsocks=s verbose|v+), @c_opt,
- PublicInbox::LeiQuery::curl_opt(), opt_dash('limit|n=i', '[0-9]+') ],
+ sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
+ import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
+ opt_dash('limit|n=i', '[0-9]+') ],
-'show' => [ 'MID|OID', 'show a given object (Message-ID or object ID)',
- qw(type=s solve! format|f=s dedupe|d=s threads|t remote local!
- verbose|v+), @c_opt, pass_through('git show') ],
+'blob' => [ 'OID', 'display a git blob object, solving if necessary',
+ qw(git-dir=s@ cwd! verbose|v+ oid-a|A=s path-a|a=s path-b|b=s),
+ @lxs_opt, @c_opt ],
'add-external' => [ 'LOCATION',
'add/set priority of a publicinbox|extindex for extra matches',
@@ -350,7 +352,7 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q mark); # internal workers
+my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q mark sol); # internal workers
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
sub x_it ($$) {
@@ -726,10 +728,6 @@ sub _lei_store ($;$) {
};
}
-sub lei_show {
- my ($self, @argv) = @_;
-}
-
sub _config {
my ($self, @argv) = @_;
my %env = (%{$self->{env}}, GIT_CONFIG => undef);
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
new file mode 100644
index 00000000..a50255aa
--- /dev/null
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -0,0 +1,119 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei blob $OID" command
+package PublicInbox::LeiBlob;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::IPC);
+use PublicInbox::Spawn qw(spawn popen_rd);
+use PublicInbox::DS;
+use PublicInbox::Eml;
+
+sub sol_done_wait { # dwaitpid callback
+ my ($arg, $pid) = @_;
+ my (undef, $lei) = @$arg;
+ $lei->child_error($?) if $?;
+ $lei->dclose;
+}
+
+sub sol_done { # EOF callback for main daemon
+ my ($lei) = @_;
+ my $sol = delete $lei->{sol} or return;
+ $sol->wq_wait_old(\&sol_done_wait, $lei);
+}
+
+sub get_git_dir ($) {
+ my ($d) = @_;
+ return $d if -d "$d/objects" && -d "$d/refs" && -e "$d/HEAD";
+
+ my $cmd = [ qw(git rev-parse --git-dir) ];
+ my ($r, $pid) = popen_rd($cmd, {GIT_DIR => undef}, { '-C' => $d });
+ chomp(my $gd = do { local $/; <$r> });
+ waitpid($pid, 0) == $pid or die "BUG: waitpid @$cmd ($!)";
+ $? == 0 ? $gd : undef;
+}
+
+sub solver_user_cb { # called by solver when done
+ my ($res, $self) = @_;
+ my $lei = $self->{lei};
+ my $log_buf = delete $lei->{'log_buf'};
+ $$log_buf =~ s/^/# /sgm;
+ ref($res) eq 'ARRAY' or return $lei->fail($$log_buf);
+ $lei->qerr($$log_buf);
+ my ($git, $oid, $type, $size, $di) = @$res;
+ my $gd = $git->{git_dir};
+
+ # don't try to support all the git-show(1) options for non-blob,
+ # this is just a convenience:
+ $type ne 'blob' and
+ $lei->err("# $oid is a $type of $size bytes in:\n#\t$gd");
+
+ my $cmd = [ 'git', "--git-dir=$gd", 'show', $oid ];
+ my $rdr = { 1 => $lei->{1}, 2 => $lei->{2} };
+ waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ $lei->child_error($?) if $?;
+}
+
+sub do_solve_blob { # via wq_do
+ my ($self) = @_;
+ my $lei = $self->{lei};
+ my $git_dirs = $lei->{opt}->{'git-dir'};
+ my $hints = {};
+ for my $x (qw(oid-a path-a path-b)) {
+ my $v = $lei->{opt}->{$x} // next;
+ $x =~ tr/-/_/;
+ $hints->{$x} = $v;
+ }
+ open my $log, '+>', \(my $log_buf = '') or die "PerlIO::scalar: $!";
+ $lei->{log_buf} = \$log_buf;
+ my $git = $lei->ale->git;
+ my $solver = bless {
+ gits => [ map { PublicInbox::Git->new($_) } @$git_dirs ],
+ user_cb => \&solver_user_cb,
+ uarg => $self,
+ # -cur_di, -qsp, -msg => temporary fields for Qspawn callbacks
+ inboxes => [ $self->{lxs}->locals ],
+ }, 'PublicInbox::SolverGit';
+ $lei->{env}->{'psgi.errors'} = $lei->{2}; # ugh...
+ local $PublicInbox::DS::in_loop = 0; # waitpid synchronously
+ $solver->solve($lei->{env}, $log, $self->{oid_b}, $hints);
+}
+
+sub lei_blob {
+ my ($lei, $blob) = @_;
+ $lei->start_pager if -t $lei->{1};
+
+ # first, see if it's a blob returned by "lei q" JSON output:
+ my $rdr = { 1 => $lei->{1} };
+ open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
+ 'cat-file', 'blob', $blob ];
+ waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ return if $? == 0;
+
+ # maybe it's a non-email (code) blob from a coderepo
+ my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
+ if ($lei->{opt}->{'cwd'} //= 1) {
+ my $cgd = get_git_dir('.');
+ unshift(@$git_dirs, $cgd) if defined $cgd;
+ }
+ my $lxs = $lei->lxs_prepare or return;
+ require PublicInbox::SolverGit;
+ my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
+ my $op = $lei->workers_start($self, 'lei_solve', 1,
+ { '' => [ \&sol_done, $lei ] });
+ $lei->{sol} = $self;
+ $self->wq_io_do('do_solve_blob', []);
+ $self->wq_close(1);
+ while ($op && $op->{sock}) { $op->event_step }
+}
+
+sub ipc_atfork_child {
+ my ($self) = @_;
+ $self->{lei}->_lei_atfork_child;
+ $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
+ $self->SUPER::ipc_atfork_child;
+}
+
+1;
diff --git a/t/lei-import.t b/t/lei-import.t
index fa40ad01..33ce490d 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -54,6 +54,8 @@ is($res->[0]->{'m'}, 'x@y', 'got expected message');
is($res->[0]->{kw}, undef, 'Status ignored for eml');
lei_ok(qw(q -f mboxrd m:x@y));
unlike($lei_out, qr/^Status:/, 'no Status: in imported message');
+lei_ok('blob', $res->[0]->{blob});
+is($lei_out, "From: a\@b\nMessage-ID: <x\@y>\n", 'got blob back');
$eml->header_set('Message-ID', '<v@y>');
diff --git a/t/solver_git.t b/t/solver_git.t
index 99ffb9e3..22714ae5 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -6,6 +6,7 @@ use v5.10.1;
use PublicInbox::TestCommon;
use Cwd qw(abs_path);
require_git(2.6);
+use Digest::SHA qw(sha1_hex);
use PublicInbox::Spawn qw(popen_rd);
require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
@@ -27,6 +28,18 @@ my $ibx = create_inbox 'v2', version => 2,
};
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
+my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
+
+test_lei({tmpdir => $tmpdir}, sub {
+ lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
+ is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
+ $expect, 'blob contents output');
+
+ # fallbacks
+ lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
+ lei_ok('blob', $v1_0_0_tag_short, '-I', $ibx->{inboxdir});
+});
+
my $git = PublicInbox::Git->new($git_dir);
$ibx->{-repo_objs} = [ $git ];
my $res;
@@ -38,7 +51,6 @@ $solver->solve($psgi_env, $log, '69df7d5', {});
ok($res, 'solved a blob!');
my $wt_git = $res->[0];
is(ref($wt_git), 'PublicInbox::Git', 'got a git object for the blob');
-my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
is($res->[1], $expect, 'resolved blob to unabbreviated identifier');
is($res->[2], 'blob', 'type specified');
is($res->[3], 4405, 'size returned');
^ permalink raw reply related [relevance 38%]
* [SQUASH] lei blob: use absolute path
2021-03-27 11:45 38% ` [PATCH 4/4] lei blob: aka "git-show-harder" for blobs Eric Wong
@ 2021-03-27 20:20 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-27 20:20 UTC (permalink / raw)
To: meta
I found this bug because I /wasn't/ in an alternate git worktree :x
---
lib/PublicInbox/LeiBlob.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index a50255aa..2facbad3 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -69,7 +69,9 @@ sub do_solve_blob { # via wq_do
$lei->{log_buf} = \$log_buf;
my $git = $lei->ale->git;
my $solver = bless {
- gits => [ map { PublicInbox::Git->new($_) } @$git_dirs ],
+ gits => [ map {
+ PublicInbox::Git->new($lei->rel2abs($_))
+ } @$git_dirs ],
user_cb => \&solver_user_cb,
uarg => $self,
# -cur_di, -qsp, -msg => temporary fields for Qspawn callbacks
^ permalink raw reply related [relevance 71%]
* [PATCH] lei mark: relax label requirements
@ 2021-03-27 23:22 69% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-27 23:22 UTC (permalink / raw)
To: meta
It seems safe to use ALLCAPS labels like "INBOX" with Xapian.
We'll also allow single-character labels.
---
lib/PublicInbox/LeiMark.pm | 2 +-
t/lei-mark.t | 6 ++++++
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 6d236411..34846b84 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -23,7 +23,7 @@ my %ERR = (
my ($label) = @_;
length($label) >= $L_MAX and
return "`$label' too long (must be <= $L_MAX)";
- $label =~ m{\A[a-z0-9_][a-z0-9_\-\./\@,]*[a-z0-9]\z} ?
+ $label =~ m{\A[a-z0-9_](?:[a-z0-9_\-\./\@,]*[a-z0-9])?\z}i ?
undef : "`$label' is invalid";
},
kw => sub {
diff --git a/t/lei-mark.t b/t/lei-mark.t
index 23f5002e..7855839e 100644
--- a/t/lei-mark.t
+++ b/t/lei-mark.t
@@ -78,6 +78,12 @@ test_lei(sub {
lei_ok(qw(ls-label));
is($lei_out, "nope\nqp\nurgent\n", 'ls-label shows qp');
+ lei_ok qw(mark -F eml t/utf8.eml +L:INBOX +L:x); diag $lei_err;
+ lei_ok qw(q m:testmessage@example.com);
+ $check_kw->([qw(answered seen)], L => [qw(INBOX nope urgent x)]);
+ lei_ok(qw(ls-label));
+ is($lei_out, "INBOX\nnope\nqp\nurgent\nx\n", 'ls-label shows qp');
+
if (0) { # TODO label+kw search w/ externals
lei_ok(qw(q L:qp), "mid:$mid", '--only', "$ro_home/t2");
}
^ permalink raw reply related [relevance 69%]
* Re: is "lei mark" a good name?
2021-03-26 9:48 68% ` Eric Wong
@ 2021-03-28 2:21 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 2:21 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Eric Wong <e@80x24.org> wrote:
> Kyle Meyer <kyle@kyleam.com> wrote:
> > Eric Wong writes:
> >
> > > It can set/unset volatile metadata for any number of messages.
> > >
> > > "Volatile metadata" being "labels" (aka "mailboxes" in
> > > JMAP-speak) and "keywords" (seen|flagged|answered|...),
> > > (aka "flags" in IMAP/Maildir-speak).
> > >
> > > "lei mark +kw:seen" # makes sense
> >
> > > "lei mark +L:some-folder-name" # might sound odd...
> > >
> > >
> > > AFAIK, notmuch uses "notmuch tag" which combines both labels
> > > and keywords into one thing: "tags". But I'm also not a
> > > notmuch user...
> >
> > (I am, though I still might be wrong.) Notmuch allows arbitrary tags to
> > be associated with message IDs. It has some automatic tags like
> > "attachment", "encrypted", and "signed" that it adds when indexing.
>
> Hmm..., JMAP has a "hasAttachment" property to search on.
>
> > By default, it also syncs some maildir flags to its tags (e.g., "F ->
> > flagged", "R -> replied", "No S -> unread").
> >
> > As far as I understand, tags are never connected up to the maildir
> > folder/location, though Notmuch does support a separate "folder:..."
> > search term.
>
> I've been thinking about storing source location data, too.
> Having the source of an email in lei/store would make it easier
> and faster to export keywords back to IMAP/Maildir.
>
> > So, I guess in JMAP terms, Notmuch tags would be "mailboxes" (a subset
> > of which are synced with maildir flags).
>
> The arbitrary nm tags (and "attachment|signed|encrypted) would be
> "mailboxes", yes. But the Maildir flags are JMAP keywords:
>
> # LeiToMail.pm:
> my %kw2char = ( # Maildir characters
> draft => 'D',
> flagged => 'F',
> answered => 'R',
> seen => 'S'
> );
>
> > > Would "lei tag" be better?
> >
> > Neither one really jumps out to me as better. "mark" sounds fine to me,
> > and I don't find "tag" any more or less odd for the +L case above.
>
> OK. I'm still on the fence...
>
> On one hand, "tag" is probably a more familiar term to users of
> existing software. Not just notmuch, but also other software
> for music metadata, software (debtags), etc...
>
> On the other hand, lei will support "lei show" to wrap "git show"
> with SolverGit support. Thus I'm worried "lei tag" might
> be misconstrued as a wrapper for "git tag"...
>
> But now that I think about it more; "lei show" probably won't
> treat git tree/tag/commit objects any differently than "git show".
> Perhaps "lei show" can be named "lei blob", instead.
OK, so I've gone ahead with "lei blob" so far...
"git show" has a plethora of options affecting diff generation
which would be redundant to have in lei. "lei blob" can focus
on reconstructing and showing blobs. The name also corresponds
to the JSON output field of "lei q".
I like "lei tag", more than "lei mark", since I think "tag" is
more commonly used for attaching metadata labels to stuff.
"lei tag" might still give people the idea that it's for
operating on git tags (because "lei blob" operates on git blobs);
but I don't think it's a huge risk since our code base does
nothing with git tags
^ permalink raw reply [relevance 71%]
* [PATCH 00/12] lei blob and some yak-shaving
@ 2021-03-28 9:01 67% Eric Wong
2021-03-28 9:01 43% ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
` (9 more replies)
0 siblings, 10 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
"lei blob" manages to work with HTTPS and HTTP .onion endpoints.
Eric Wong (12):
lei: simplify PktOp callers
lei init: split out into separate file
lei blob: dclose if already failed
lei blob: support --no-mail switch
lei blob: fail early if no git dirs
lei blob: some extra tests
lei help: show "NAME=VALUE" properly for -c
lei blob: flesh out help text
t/lei_store: ensure LeiSearch responds to ->isrch
lei blob: add remote external support
lei: drop coderepo placeholders, submodule TODO
treewide: shorten temporary filename
MANIFEST | 2 +
lib/PublicInbox/LEI.pm | 67 +++++++---------------------
lib/PublicInbox/LeiBlob.pm | 47 +++++++++++++-------
lib/PublicInbox/LeiConvert.pm | 4 +-
lib/PublicInbox/LeiHelp.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 4 +-
lib/PublicInbox/LeiInit.pm | 41 +++++++++++++++++
lib/PublicInbox/LeiMark.pm | 4 +-
lib/PublicInbox/LeiMirror.pm | 4 +-
lib/PublicInbox/LeiOverview.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 4 +-
lib/PublicInbox/LeiRemote.pm | 81 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiXSearch.pm | 8 ++--
lib/PublicInbox/Msgmap.pm | 2 +-
lib/PublicInbox/PktOp.pm | 20 ++++++---
lib/PublicInbox/SolverGit.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 2 +-
lib/PublicInbox/V2Writable.pm | 6 +--
lib/PublicInbox/Xapcmd.pm | 9 ++--
script/public-inbox-edit | 2 +-
script/public-inbox-init | 2 +-
scripts/ssoma-replay | 2 +-
t/check-www-inbox.perl | 2 +-
t/inbox.t | 2 +-
t/lei_store.t | 1 +
t/nodatacow.t | 2 +-
t/solver_git.t | 40 +++++++++++++++--
27 files changed, 254 insertions(+), 110 deletions(-)
create mode 100644 lib/PublicInbox/LeiInit.pm
create mode 100644 lib/PublicInbox/LeiRemote.pm
^ permalink raw reply [relevance 67%]
* [PATCH 03/12] lei blob: dclose if already failed
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 43% ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
2021-03-28 9:01 58% ` [PATCH 02/12] lei init: split out into separate file Eric Wong
@ 2021-03-28 9:01 71% ` Eric Wong
2021-03-28 9:01 57% ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
` (6 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
We must close the socket to trigger pager exit if blob
reconstruction fails. Not sure how to test this in the
test suite...
---
lib/PublicInbox/LeiBlob.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 97747220..9b4c4f30 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -19,7 +19,7 @@ sub sol_done_wait { # dwaitpid callback
sub sol_done { # EOF callback for main daemon
my ($lei) = @_;
- my $sol = delete $lei->{sol} or return;
+ my $sol = delete $lei->{sol} // return $lei->dclose; # already failed
$sol->wq_wait_old(\&sol_done_wait, $lei);
}
^ permalink raw reply related [relevance 71%]
* [PATCH 05/12] lei blob: fail early if no git dirs
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (3 preceding siblings ...)
2021-03-28 9:01 57% ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
@ 2021-03-28 9:01 71% ` Eric Wong
2021-03-28 9:01 67% ` [PATCH 06/12] lei blob: some extra tests Eric Wong
` (4 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This avoids triggering a "BUG:" message in the solver code.
---
lib/PublicInbox/LeiBlob.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 4bd86253..f44d8af1 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -103,6 +103,7 @@ sub lei_blob {
my $cgd = get_git_dir('.');
unshift(@$git_dirs, $cgd) if defined $cgd;
}
+ return $lei->fail('no --git-dir to try') unless @$git_dirs;
my $lxs = $lei->lxs_prepare or return;
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
^ permalink raw reply related [relevance 71%]
* [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (5 preceding siblings ...)
2021-03-28 9:01 67% ` [PATCH 06/12] lei blob: some extra tests Eric Wong
@ 2021-03-28 9:01 71% ` Eric Wong
2021-03-28 9:01 69% ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
` (2 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
---
lib/PublicInbox/LeiHelp.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiHelp.pm b/lib/PublicInbox/LeiHelp.pm
index 9c1b30a1..fa0e7866 100644
--- a/lib/PublicInbox/LeiHelp.pm
+++ b/lib/PublicInbox/LeiHelp.pm
@@ -49,7 +49,7 @@ sub call {
length($_) > 1 ? push(@l, "--$_") : push(@s, "-$_");
}
if (!scalar(@vals)) { # no args 'threads|t'
- } elsif ($arg_vals =~ s/\A([A-Z_]+)\b//) { # "NAME"
+ } elsif ($arg_vals =~ s/\A([A-Z_=]+)\b//) { # "NAME"
$vals[1] = $1;
} else {
$vals[1] = uc(substr($l[0], 2)); # "--type" => "TYPE"
^ permalink raw reply related [relevance 71%]
* [PATCH 08/12] lei blob: flesh out help text
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (6 preceding siblings ...)
2021-03-28 9:01 71% ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
@ 2021-03-28 9:01 69% ` Eric Wong
2021-03-28 9:01 44% ` [PATCH 10/12] lei blob: add remote external support Eric Wong
2021-03-28 9:01 66% ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This means "lei blob" gets shell completion, too.
---
lib/PublicInbox/LEI.pm | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index bad7fad9..a4f4e58c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -249,7 +249,12 @@ my %OPTDESC = (
"and\xa0'[]'\x{a0}ranges",
'verbose|v+' => 'be more verbose',
'external!' => 'do not use externals',
-'solve!' => 'do not attempt to reconstruct blobs from emails',
+'mail!' => 'do not look in mail storage for OID',
+'cwd!' => 'do not look in git repo of current working directory',
+'oid-a|A=s' => 'pre-image OID',
+'path-a|a=s' => 'pre-image pathname associated with OID',
+'path-b|b=s' => 'post-image pathname associated with OID',
+'git-dir=s@' => 'additional git repository to scan',
'torsocks=s' => ['VAL|auto|no|yes',
'whether or not to wrap git and curl commands with torsocks'],
'no-torsocks' => 'alias for --torsocks=no',
@@ -786,7 +791,7 @@ sub lei__complete {
if (s/[:=].+\z//) { # req/optional args, e.g output|o=i
} elsif (s/\+\z//) { # verbose|v+
} elsif (s/!\z//) {
- # negation: solve! => no-solve|solve
+ # negation: mail! => no-mail|mail
s/([\w\-]+)/$1|no-$1/g
}
map {
^ permalink raw reply related [relevance 69%]
* [PATCH 06/12] lei blob: some extra tests
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (4 preceding siblings ...)
2021-03-28 9:01 71% ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
@ 2021-03-28 9:01 67% ` Eric Wong
2021-03-28 9:01 71% ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
` (3 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Most of it already gets tested since most of the logic is in
SolverGit, but make sure it's all wired up properly to lei.
---
t/solver_git.t | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/t/solver_git.t b/t/solver_git.t
index 7bf3ba21..6d4b93c7 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -29,6 +29,7 @@ my $ibx = create_inbox 'v2', version => 2,
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
+my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
test_lei({tmpdir => $tmpdir}, sub {
lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
@@ -37,6 +38,25 @@ test_lei({tmpdir => $tmpdir}, sub {
my $prev = $lei_out;
lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
is($lei_out, $prev, '--no-mail works');
+ ok(!lei(qw(blob -I), $ibx->{inboxdir}, $non_existent),
+ 'non-existent blob fails');
+ SKIP: {
+ skip '/.git exists', 1 if -e '/.git';
+ require PublicInbox::OnDestroy;
+ opendir my $dh, '.' or xbail "opendir: $!";
+ my $end = PublicInbox::OnDestroy->new($$, sub {
+ chdir $dh or xbail "chdir: $!";
+ });
+ lei_ok(qw(-C / blob 69df7d5 -I), $ibx->{inboxdir},
+ "--git-dir=$git_dir");
+ is($lei_out, $prev, '--git-dir works');
+
+ ok(!lei(qw(-C / blob --no-cwd 69df7d5 -I), $ibx->{inboxdir}),
+ '--no-cwd works');
+
+ ok(!lei(qw(-C / blob -I), $ibx->{inboxdir}, $non_existent),
+ 'non-existent blob fails');
+ }
# fallbacks
lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
@@ -163,7 +183,6 @@ EOF
close $cfgfh or die;
my $cfg = PublicInbox::Config->new($cfgpath);
my $www = PublicInbox::WWW->new($cfg);
- my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
my $client = sub {
my ($cb) = @_;
my $mid = '20190401081523.16213-1-BOFH@YHBT.net';
^ permalink raw reply related [relevance 67%]
* [PATCH 02/12] lei init: split out into separate file
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 43% ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
@ 2021-03-28 9:01 58% ` Eric Wong
2021-03-28 9:01 71% ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
` (7 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This is a rarely-needed command, so keep it separate file
so it's easier-to-find and maybe saves a bit of RAM.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 32 -----------------------------
lib/PublicInbox/LeiInit.pm | 41 ++++++++++++++++++++++++++++++++++++++
3 files changed, 42 insertions(+), 32 deletions(-)
create mode 100644 lib/PublicInbox/LeiInit.pm
diff --git a/MANIFEST b/MANIFEST
index 64b3626f..9048b900 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -188,6 +188,7 @@ lib/PublicInbox/LeiDedupe.pm
lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
+lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiLsLabel.pm
lib/PublicInbox/LeiMark.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9cacb142..fdb0bbcf 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -744,38 +744,6 @@ sub lei_config {
x_it($self, $?) if $?;
}
-sub lei_init {
- my ($self, $dir) = @_;
- my $cfg = _lei_cfg($self, 1);
- my $cur = $cfg->{'leistore.dir'};
- $dir //= store_path($self);
- $dir = rel2abs($self, $dir);
- my @cur = stat($cur) if defined($cur);
- $cur = File::Spec->canonpath($cur // $dir);
- my @dir = stat($dir);
- my $exists = "# leistore.dir=$cur already initialized" if @dir;
- if (@cur) {
- if ($cur eq $dir) {
- _lei_store($self, 1)->done;
- return qerr($self, $exists);
- }
-
- # some folks like symlinks and bind mounts :P
- if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
- lei_config($self, 'leistore.dir', $dir);
- _lei_store($self, 1)->done;
- return qerr($self, "$exists (as $cur)");
- }
- return fail($self, <<"");
-E: leistore.dir=$cur already initialized and it is not $dir
-
- }
- lei_config($self, 'leistore.dir', $dir);
- _lei_store($self, 1)->done;
- $exists //= "# leistore.dir=$dir newly initialized";
- return qerr($self, $exists);
-}
-
sub lei_daemon_pid { puts shift, $$ }
sub lei_daemon_kill {
diff --git a/lib/PublicInbox/LeiInit.pm b/lib/PublicInbox/LeiInit.pm
new file mode 100644
index 00000000..c6c0c01b
--- /dev/null
+++ b/lib/PublicInbox/LeiInit.pm
@@ -0,0 +1,41 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# for the "lei init" command, not sure if it's even needed...
+package PublicInbox::LeiInit;
+use v5.10.1;
+use File::Spec;
+
+sub lei_init {
+ my ($self, $dir) = @_;
+ my $cfg = $self->_lei_cfg(1);
+ my $cur = $cfg->{'leistore.dir'};
+ $dir //= $self->store_path;
+ $dir = $self->rel2abs($dir);
+ my @cur = stat($cur) if defined($cur);
+ $cur = File::Spec->canonpath($cur // $dir);
+ my @dir = stat($dir);
+ my $exists = "# leistore.dir=$cur already initialized" if @dir;
+ if (@cur) {
+ if ($cur eq $dir) {
+ $self->_lei_store(1)->done;
+ return $self->qerr($exists);
+ }
+
+ # some folks like symlinks and bind mounts :P
+ if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
+ $self->lei_config('leistore.dir', $dir);
+ $self->_lei_store(1)->done;
+ return $self->qerr("$exists (as $cur)");
+ }
+ return $self->fail(<<"");
+E: leistore.dir=$cur already initialized and it is not $dir
+
+ }
+ $self->lei_config('leistore.dir', $dir);
+ $self->_lei_store(1)->done;
+ $exists //= "# leistore.dir=$dir newly initialized";
+ $self->qerr($exists);
+}
+
+1;
^ permalink raw reply related [relevance 58%]
* [PATCH 04/12] lei blob: support --no-mail switch
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (2 preceding siblings ...)
2021-03-28 9:01 71% ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
@ 2021-03-28 9:01 57% ` Eric Wong
2021-03-28 9:01 71% ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
` (5 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
It's possible for a abbreviated OID to be resolved unambiguously
to an email before we attempt to look at externals via xsearch;
so provide a way for a user to force searching coderepos.
If hints (--oid-a, --path-a, --path-b) are present, we'll
assume --no-mail by default, otherwise we'll assume the
user wants to look through mail for a matching blob.
---
lib/PublicInbox/LEI.pm | 4 ++--
lib/PublicInbox/LeiBlob.pm | 23 +++++++++++++----------
t/solver_git.t | 3 +++
3 files changed, 18 insertions(+), 12 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fdb0bbcf..bad7fad9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -136,8 +136,8 @@ our %CMD = ( # sorted in order of importance/use:
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
-'blob' => [ 'OID', 'display a git blob object, solving if necessary',
- qw(git-dir=s@ cwd! verbose|v+ oid-a|A=s path-a|a=s path-b|b=s),
+'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
+ qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
@lxs_opt, @c_opt ],
'add-external' => [ 'LOCATION',
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 9b4c4f30..4bd86253 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -8,7 +8,6 @@ use v5.10.1;
use parent qw(PublicInbox::IPC);
use PublicInbox::Spawn qw(spawn popen_rd);
use PublicInbox::DS;
-use PublicInbox::Eml;
sub sol_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
@@ -85,18 +84,22 @@ sub do_solve_blob { # via wq_do
sub lei_blob {
my ($lei, $blob) = @_;
$lei->start_pager if -t $lei->{1};
+ my $opt = $lei->{opt};
+ my $has_hints = grep(defined, @$opt{qw(oid-a path-a path-b)});
- # first, see if it's a blob returned by "lei q" JSON output:
- my $rdr = { 1 => $lei->{1} };
- open $rdr->{2}, '>', '/dev/null' or die "open: $!";
- my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
- 'cat-file', 'blob', $blob ];
- waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
- return if $? == 0;
+ # first, see if it's a blob returned by "lei q" JSON output:k
+ if ($opt->{mail} // ($has_hints ? 0 : 1)) {
+ my $rdr = { 1 => $lei->{1} };
+ open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
+ 'cat-file', 'blob', $blob ];
+ waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ return if $? == 0;
+ }
# maybe it's a non-email (code) blob from a coderepo
- my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
- if ($lei->{opt}->{'cwd'} //= 1) {
+ my $git_dirs = $opt->{'git-dir'} //= [];
+ if ($opt->{'cwd'} // 1) {
my $cgd = get_git_dir('.');
unshift(@$git_dirs, $cgd) if defined $cgd;
}
diff --git a/t/solver_git.t b/t/solver_git.t
index 22714ae5..7bf3ba21 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -34,6 +34,9 @@ test_lei({tmpdir => $tmpdir}, sub {
lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
$expect, 'blob contents output');
+ my $prev = $lei_out;
+ lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
+ is($lei_out, $prev, '--no-mail works');
# fallbacks
lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
^ permalink raw reply related [relevance 57%]
* [PATCH 01/12] lei: simplify PktOp callers
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
@ 2021-03-28 9:01 43% ` Eric Wong
2021-03-28 9:01 58% ` [PATCH 02/12] lei init: split out into separate file Eric Wong
` (8 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Provide a consistent ->op_wait_event method instead of
forcing callers to loop (or not) at each callsite.
This also avoid a leak possibility by avoiding circular
references.
---
lib/PublicInbox/LEI.pm | 11 +++++------
lib/PublicInbox/LeiBlob.pm | 4 ++--
lib/PublicInbox/LeiConvert.pm | 4 ++--
lib/PublicInbox/LeiImport.pm | 4 ++--
lib/PublicInbox/LeiMark.pm | 4 ++--
lib/PublicInbox/LeiMirror.pm | 4 ++--
lib/PublicInbox/LeiP2q.pm | 4 ++--
lib/PublicInbox/LeiXSearch.pm | 8 +++-----
lib/PublicInbox/PktOp.pm | 20 +++++++++++++++-----
9 files changed, 35 insertions(+), 28 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 478912cd..9cacb142 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -494,11 +494,11 @@ sub _delete_pkt_op { # OnDestroy callback to prevent leaks on die
}
sub pkt_op_pair {
- my ($self, $ops) = @_;
+ my ($self) = @_;
require PublicInbox::OnDestroy;
require PublicInbox::PktOp;
my $end = PublicInbox::OnDestroy->new($$, \&_delete_pkt_op, $self);
- @$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair($ops);
+ @$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair;
$end;
}
@@ -512,14 +512,13 @@ sub workers_start {
($ops ? %$ops : ()),
};
$ops->{''} //= [ \&dclose, $lei ];
- my $end = $lei->pkt_op_pair($ops);
+ my $end = $lei->pkt_op_pair;
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
delete $lei->{pkt_op_p};
- my $op = delete $lei->{pkt_op_c};
+ my $op_c = delete $lei->{pkt_op_c};
@$end = ();
$lei->event_step_init;
- # oneshot needs $op, daemon-mode uses DS->EventLoop to handle $op
- $lei->{oneshot} ? $op : undef;
+ ($op_c, $ops);
}
sub _help {
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 2facbad3..97747220 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -103,12 +103,12 @@ sub lei_blob {
my $lxs = $lei->lxs_prepare or return;
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
- my $op = $lei->workers_start($self, 'lei_solve', 1,
+ my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
{ '' => [ \&sol_done, $lei ] });
$lei->{sol} = $self;
$self->wq_io_do('do_solve_blob', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 083ecc33..5d0adb14 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -53,11 +53,11 @@ sub lei_convert { # the main "lei convert" method
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
- my $op = $lei->workers_start($self, 'lei_convert', 1);
+ my ($op_c, $ops) = $lei->workers_start($self, 'lei_convert', 1);
$lei->{cnv} = $self;
$self->wq_io_do('do_convert', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 7c5b7d09..803b5cda 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -76,11 +76,11 @@ sub lei_import { # the main "lei import" method
my $ops = { '' => [ \&import_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{-wq_nr_workers} = $j // 1; # locked
- my $op = $lei->workers_start($self, 'lei_import', undef, $ops);
+ my ($op_c, undef) = $lei->workers_start($self, 'lei_import', $j, $ops);
$lei->{imp} = $self;
$self->wq_io_do('input_stdin', []) if $self->{0};
net_merge_complete($self) unless $lei->{auth};
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
no warnings 'once';
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 34846b84..6e611318 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -116,11 +116,11 @@ sub lei_mark { # the "lei mark" method
my $ops = { '' => [ \&mark_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- my $op = $lei->workers_start($self, 'lei_mark', 1, $ops);
+ my ($op_c, undef) = $lei->workers_start($self, 'lei_mark', 1, $ops);
$lei->{mark} = $self;
$self->wq_io_do('input_stdin', []) if $self->{0};
net_merge_complete($self) unless $lei->{auth};
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub note_missing {
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index c83386c6..89574d28 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -282,13 +282,13 @@ sub start {
require PublicInbox::Inbox;
require PublicInbox::Admin;
require PublicInbox::InboxWritable;
- my $op = $lei->workers_start($self, 'lei_mirror', 1, {
+ my ($op, $ops) = $lei->workers_start($self, 'lei_mirror', 1, {
'' => [ \&mirror_done, $lei ]
});
$lei->{mrr} = $self;
$self->wq_io_do('do_mirror', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 25f63a10..a8a3dd2c 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -185,11 +185,11 @@ sub lei_p2q { # the "lei patch-to-query" entry point
} else {
$self->{input} = $input;
}
- my $op = $lei->workers_start($self, 'lei_p2q', 1);
+ my ($op, $ops) = $lei->workers_start($self, 'lei_p2q', 1);
$lei->{p2q} = $self;
$self->wq_io_do('do_p2q', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index b41daffe..1a194f1c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -427,7 +427,7 @@ sub do_query {
'incr_start_query' => [ \&incr_start_query, $self, $l2m ],
};
$lei->{auth}->op_merge($ops, $l2m) if $l2m && $lei->{auth};
- my $end = $lei->pkt_op_pair($ops);
+ my $end = $lei->pkt_op_pair;
$lei->{1}->autoflush(1);
$lei->start_pager if delete $lei->{need_pager};
$lei->{ovv}->ovv_begin($lei);
@@ -445,7 +445,7 @@ sub do_query {
}
$self->wq_workers_start('lei_xsearch', undef,
$lei->oldset, { lei => $lei });
- my $op = delete $lei->{pkt_op_c};
+ my $op_c = delete $lei->{pkt_op_c};
delete $lei->{pkt_op_p};
@$end = ();
$self->{threads} = $lei->{opt}->{threads};
@@ -455,9 +455,7 @@ sub do_query {
start_query($self);
}
$lei->event_step_init; # wait for shutdowns
- if ($lei->{oneshot}) {
- while ($op->{sock}) { $op->event_step }
- }
+ $op_c->op_wait_event($ops);
}
sub add_uri {
diff --git a/lib/PublicInbox/PktOp.pm b/lib/PublicInbox/PktOp.pm
index 5d8e78ea..c3221735 100644
--- a/lib/PublicInbox/PktOp.pm
+++ b/lib/PublicInbox/PktOp.pm
@@ -16,21 +16,23 @@ use PublicInbox::IPC qw(ipc_freeze ipc_thaw);
our @EXPORT_OK = qw(pkt_do);
sub new {
- my ($cls, $r, $ops) = @_;
- my $self = bless { sock => $r, ops => $ops }, $cls;
+ my ($cls, $r) = @_;
+ my $self = bless { sock => $r }, $cls;
if ($PublicInbox::DS::in_loop) { # iff using DS->EventLoop
$r->blocking(0);
$self->SUPER::new($r, EPOLLIN|EPOLLET);
+ } else {
+ $self->{blocking} = 1;
}
$self;
}
# returns a blessed object as the consumer, and a GLOB/IO for the producer
sub pair {
- my ($cls, $ops) = @_;
+ my ($cls) = @_;
my ($c, $p);
socketpair($c, $p, AF_UNIX, SOCK_SEQPACKET, 0) or die "socketpair: $!";
- (new($cls, $c, $ops), $p);
+ (new($cls, $c), $p);
}
sub pkt_do { # for the producer to trigger event_step in consumer
@@ -41,7 +43,7 @@ sub pkt_do { # for the producer to trigger event_step in consumer
sub close {
my ($self) = @_;
my $c = $self->{sock} or return;
- $c->blocking ? delete($self->{sock}) : $self->SUPER::close;
+ $self->{blocking} ? delete($self->{sock}) : $self->SUPER::close;
}
sub event_step {
@@ -73,4 +75,12 @@ sub event_step {
}
}
+# call this when we're ready to wait on events,
+# returns immediately if non-blocking
+sub op_wait_event {
+ my ($self, $ops) = @_;
+ $self->{ops} = $ops;
+ while ($self->{blocking} && $self->{sock}) { event_step($self) }
+}
+
1;
^ permalink raw reply related [relevance 43%]
* [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (8 preceding siblings ...)
2021-03-28 9:01 44% ` [PATCH 10/12] lei blob: add remote external support Eric Wong
@ 2021-03-28 9:01 66% ` Eric Wong
2021-03-28 9:31 71% ` Eric Wong
9 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
"lei blob" supports --git-dir and -C, and checks if the
current directory has a git directory associated with it.
It will likely support submodules in the future.
I'm inclined to believe declaring coderepos in a command-line
tool is needless clutter and users will rarely want to search
for blobs across different projects when on the command-line.
---
lib/PublicInbox/LEI.pm | 9 ---------
lib/PublicInbox/LeiBlob.pm | 1 +
2 files changed, 1 insertion(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a94941a9..8a07a4c8 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -172,15 +172,6 @@ our %CMD = ( # sorted in order of importance/use:
'remove imported messages from IMAP, Maildirs, and MH',
qw(exact! all jobs:i indexed), @c_opt ],
-# code repos are used for `show' to solve blobs from patch mails
-'add-coderepo' => [ 'DIRNAME', 'add or set priority of a git code repo',
- qw(boost=i), @c_opt ],
-'ls-coderepo' => [ '[FILTER_TERMS...]',
- 'list known code repos', qw(format|f=s z), @c_opt ],
-'forget-coderepo' => [ 'DIRNAME',
- 'stop using repo to solve blobs from patches',
- qw(prune), @c_opt ],
-
'add-watch' => [ 'LOCATION', 'watch for new messages and flag changes',
qw(import! kw|keywords|flags! interval=s recursive|r
exclude=s include=s), @c_opt ],
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 8e610efd..91098a90 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -2,6 +2,7 @@
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# "lei blob $OID" command
+# TODO: this doesn't scan submodules, but maybe it should
package PublicInbox::LeiBlob;
use strict;
use v5.10.1;
^ permalink raw reply related [relevance 66%]
* [PATCH 10/12] lei blob: add remote external support
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (7 preceding siblings ...)
2021-03-28 9:01 69% ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
@ 2021-03-28 9:01 44% ` Eric Wong
2021-03-28 9:01 66% ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
9 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Introduce a new LeiRemote wrapper to provide an internal API
which SolverGit expects. This lets us use HTTP/HTTPS endpoints
to reconstruct blobs off patches as we would with local
endpoints, just more slowly...
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiBlob.pm | 16 +++++--
lib/PublicInbox/LeiRemote.pm | 81 ++++++++++++++++++++++++++++++++++++
t/solver_git.t | 16 ++++++-
5 files changed, 110 insertions(+), 6 deletions(-)
create mode 100644 lib/PublicInbox/LeiRemote.pm
diff --git a/MANIFEST b/MANIFEST
index 9048b900..913ce55c 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -196,6 +196,7 @@ lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
lib/PublicInbox/LeiQuery.pm
+lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
lib/PublicInbox/LeiToMail.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a4f4e58c..a94941a9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -121,7 +121,7 @@ sub index_opt {
my @c_opt = qw(c=s@ C=s@ quiet|q);
my @lxs_opt = (qw(remote! local! external! include|I=s@ exclude=s@ only=s@
- import-remote! no-torsocks torsocks=s),
+ import-remote! no-torsocks torsocks=s),
PublicInbox::LeiQuery::curl_opt());
# we generate shell completion + help using %CMD and %OPTDESC,
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index f44d8af1..8e610efd 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -6,7 +6,7 @@ package PublicInbox::LeiBlob;
use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC);
-use PublicInbox::Spawn qw(spawn popen_rd);
+use PublicInbox::Spawn qw(spawn popen_rd which);
use PublicInbox::DS;
sub sol_done_wait { # dwaitpid callback
@@ -66,7 +66,10 @@ sub do_solve_blob { # via wq_do
}
open my $log, '+>', \(my $log_buf = '') or die "PerlIO::scalar: $!";
$lei->{log_buf} = \$log_buf;
- my $git = $lei->ale->git;
+ my $git = $lei->{ale}->git;
+ my @rmt = map {
+ PublicInbox::LeiRemote->new($lei, $_)
+ } $self->{lxs}->remotes;
my $solver = bless {
gits => [ map {
PublicInbox::Git->new($lei->rel2abs($_))
@@ -74,7 +77,7 @@ sub do_solve_blob { # via wq_do
user_cb => \&solver_user_cb,
uarg => $self,
# -cur_di, -qsp, -msg => temporary fields for Qspawn callbacks
- inboxes => [ $self->{lxs}->locals ],
+ inboxes => [ $self->{lxs}->locals, @rmt ],
}, 'PublicInbox::SolverGit';
$lei->{env}->{'psgi.errors'} = $lei->{2}; # ugh...
local $PublicInbox::DS::in_loop = 0; # waitpid synchronously
@@ -105,8 +108,15 @@ sub lei_blob {
}
return $lei->fail('no --git-dir to try') unless @$git_dirs;
my $lxs = $lei->lxs_prepare or return;
+ if ($lxs->remotes) {
+ require PublicInbox::LeiRemote;
+ $lei->{curl} //= which('curl') or return
+ $lei->fail('curl needed for', $lxs->remotes);
+ $lei->_lei_store(1)->write_prepare($lei);
+ }
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
+ $lei->ale;
my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
{ '' => [ \&sol_done, $lei ] });
$lei->{sol} = $self;
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
new file mode 100644
index 00000000..399fc936
--- /dev/null
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -0,0 +1,81 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Make remote externals HTTP(S) inboxes behave like
+# PublicInbox::Inbox and PublicInbox::Search/ExtSearch.
+# This exists solely for SolverGit. It is a high-latency a
+# synchronous API that is not at all fast.
+package PublicInbox::LeiRemote;
+use v5.10.1;
+use strict;
+use IO::Uncompress::Gunzip;
+use PublicInbox::OnDestroy;
+use PublicInbox::MboxReader;
+use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::LeiCurl;
+use PublicInbox::ContentHash qw(git_sha);
+
+sub new {
+ my ($cls, $lei, $uri) = @_;
+ bless { uri => $uri, lei => $lei }, $cls;
+}
+
+sub isrch { $_[0] } # SolverGit expcets this
+
+sub _each_mboxrd_eml { # callback for MboxReader->mboxrd
+ my ($eml, $self) = @_;
+ my $lei = $self->{lei};
+ my $xoids = $lei->{ale}->xoids_for($eml, 1);
+ if ($lei->{sto} && !$xoids) { # memoize locally
+ $lei->{sto}->ipc_do('add_eml', $eml);
+ }
+ my $smsg = bless {}, 'PublicInbox::Smsg';
+ $smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+ : git_sha(1, $eml)->hexdigest;
+ $smsg->populate($eml);
+ $smsg->{mid} //= '(none)';
+ push @{$self->{smsg}}, $smsg;
+}
+
+sub mset {
+ my ($self, $qstr, undef) = @_; # $opt ($_[2]) ignored
+ my $lei = $self->{lei};
+ my $curl = PublicInbox::LeiCurl->new($lei, $lei->{curl});
+ push @$curl, '-s', '-d', '';
+ my $uri = $self->{uri}->clone;
+ $uri->query_form(q => $qstr, x => 'm', r => 1); # r=1: relevance
+ my $cmd = $curl->for_uri($self->{lei}, $uri);
+ $self->{lei}->qerr("# $cmd");
+ my $rdr = { 2 => $lei->{2}, pgid => 0 };
+ my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
+ my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
+ $self->{smsg} = [];
+ $fh = IO::Uncompress::Gunzip->new($fh);
+ PublicInbox::MboxReader->mboxrd($fh, \&_each_mboxrd_eml, $self);
+ my $err = waitpid($pid, 0) == $pid ? undef
+ : "BUG: waitpid($cmd): $!";
+ @$reap = (); # cancel OnDestroy
+ my $wait = $self->{lei}->{sto}->ipc_do('done');
+ die $err if $err;
+ $self; # we are the mset (and $ibx, and $self)
+}
+
+sub size { scalar @{$_[0]->{smsg}} } # size of previous results
+
+sub mset_to_smsg {
+ my ($self, $ibx, $mset) = @_; # all 3 are $self
+ wantarray ? ($self->size, @{$self->{smsg}}) : $self->{smsg};
+}
+
+sub base_url { "$_[0]->{uri}" }
+
+sub smsg_eml {
+ my ($self, $smsg) = @_;
+ if (my $bref = $self->{lei}->ale->git->cat_file($smsg->{blob})) {
+ return PublicInbox::Eml->new($bref);
+ }
+ $self->{lei}->err("E: $self->{uri} $smsg->{blob} gone <$smsg->{mid}>");
+ undef;
+}
+
+1;
diff --git a/t/solver_git.t b/t/solver_git.t
index 6d4b93c7..2d803d47 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -7,7 +7,7 @@ use PublicInbox::TestCommon;
use Cwd qw(abs_path);
require_git(2.6);
use Digest::SHA qw(sha1_hex);
-use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::Spawn qw(popen_rd which);
require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
$? == 0 or plan skip_all => "$0 must be run from a git working tree";
@@ -227,8 +227,20 @@ EOF
my $cmd = [ qw(-httpd -W0), "--stdout=$out", "--stderr=$err" ];
my $td = start_script($cmd, $env, { 3 => $sock });
my ($h, $p) = tcp_host_port($sock);
- local $ENV{PLACK_TEST_EXTERNALSERVER_URI} = "http://$h:$p";
+ my $url = "http://$h:$p";
+ local $ENV{PLACK_TEST_EXTERNALSERVER_URI} = $url;
Plack::Test::ExternalServer::test_psgi(client => $client);
+ skip 'no curl', 1 unless which('curl');
+
+ mkdir "$tmpdir/ext" // xbail "mkdir $!";
+ test_lei({tmpdir => "$tmpdir/ext"}, sub {
+ my $rurl = "$url/$name";
+ lei_ok(qw(blob --no-mail 69df7d5 -I), $rurl);
+ is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
+ $expect, 'blob contents output');
+ ok(!lei(qw(blob -I), $rurl, $non_existent),
+ 'non-existent blob fails');
+ });
}
}
^ permalink raw reply related [relevance 44%]
* Re: [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO
2021-03-28 9:01 66% ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
@ 2021-03-28 9:31 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-28 9:31 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> "lei blob" supports --git-dir and -C, and checks if the
> current directory has a git directory associated with it.
> It will likely support submodules in the future.
>
> I'm inclined to believe declaring coderepos in a command-line
> tool is needless clutter and users will rarely want to search
> for blobs across different projects when on the command-line.
Fwiw, we may we end up supporting read-write JMAP AND expose
blob reconstruction as a vendor-specific JMAP function.
But that's not happening for 1.7...
^ permalink raw reply [relevance 71%]
* [PATCH 0/8] doc: lei manpages, round 4
@ 2021-03-29 3:11 69% Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 1/8] doc lei-q: fix typo in -tt description Kyle Meyer
` (8 more replies)
0 siblings, 9 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
This series updates the lei manpages, continuing from
<20210227180328.28057-1-kyle@kyleam.com>. It covers changes up to the
current tip of master (c9ff20cbef45d32e..80f4192574065106).
[1/8] doc lei-q: fix typo in -tt description
[2/8] doc lei: note --stdin shortcut in synopses
[3/8] doc lei: drop an unnecessary to-do comment
[4/8] doc lei: don't render most to-do comments
[5/8] doc lei: update manpages with new options
[6/8] doc lei: add manpages for new commands
[7/8] doc lei overview: note that lei-init is usually unnecessary
[8/8] doc lei overview: better explain routes into local store
Documentation/lei-add-external.pod | 1 +
Documentation/lei-blob.pod | 109 +++++++++++++++++++++++++++++
Documentation/lei-import.pod | 2 +-
Documentation/lei-ls-label.pod | 43 ++++++++++++
Documentation/lei-mark.pod | 58 +++++++++++++++
Documentation/lei-overview.pod | 32 ++++++++-
Documentation/lei-p2q.pod | 79 +++++++++++++++++++++
Documentation/lei-q.pod | 13 +++-
Documentation/lei.pod | 14 +++-
Documentation/txt2pre | 4 ++
MANIFEST | 4 ++
Makefile.PL | 5 +-
12 files changed, 355 insertions(+), 9 deletions(-)
create mode 100644 Documentation/lei-blob.pod
create mode 100644 Documentation/lei-ls-label.pod
create mode 100644 Documentation/lei-mark.pod
create mode 100644 Documentation/lei-p2q.pod
base-commit: 80f4192574065106ae72a7a73ee0f02ebd86708a
--
2.31.0
^ permalink raw reply [relevance 69%]
* [PATCH 1/8] doc lei-q: fix typo in -tt description
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 2/8] doc lei: note --stdin shortcut in synopses Kyle Meyer
` (7 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-q.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index e878157d..446a82c6 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -84,7 +84,7 @@ Augment output destination instead of clobbering it.
Return all messages in the same thread as the actual match(es).
Using this twice (C<-tt>) sets the C<flagged> (AKA "important")
-on messages which were actual messages. This is useful to distinguish
+on messages which were actual matches. This is useful to distinguish
messages which were direct hits from messages which were merely part
of the same thread.
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 2/8] doc lei: note --stdin shortcut in synopses
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 1/8] doc lei-q: fix typo in -tt description Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 3/8] doc lei: drop an unnecessary to-do comment Kyle Meyer
` (6 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-import.pod | 2 +-
Documentation/lei-q.pod | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index 7f4e3745..acc4f776 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -6,7 +6,7 @@ lei-import - one-time import of messages into local store
lei import [OPTIONS] LOCATION [LOCATION...]
-lei import [OPTIONS] --stdin
+lei import [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 446a82c6..cb1227fb 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -6,7 +6,7 @@ lei-q - search for messages matching terms
lei q [OPTIONS] TERM [TERM...]
-lei q [OPTIONS] --stdin
+lei q [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 3/8] doc lei: drop an unnecessary to-do comment
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 1/8] doc lei-q: fix typo in -tt description Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 2/8] doc lei: note --stdin shortcut in synopses Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:11 68% ` [PATCH 4/8] doc lei: don't render most to-do comments Kyle Meyer
` (5 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
When a new command is implemented, it is probably clear that it should
be added to lei.pod, but either way, having a to-do comment in lei.pod
isn't likely to help.
---
Documentation/lei.pod | 2 --
1 file changed, 2 deletions(-)
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index e1502122..36abe513 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -66,8 +66,6 @@ store and configured externals are
=back
-TODO: Add lei-show (and perhaps others) once implemented.
-
Other subcommands include
=over
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 5/8] doc lei: update manpages with new options
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (3 preceding siblings ...)
2021-03-29 3:11 68% ` [PATCH 4/8] doc lei: don't render most to-do comments Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:11 30% ` [PATCH 6/8] doc lei: add manpages for new commands Kyle Meyer
` (3 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-q.pod | 5 +++++
Documentation/lei.pod | 4 ++++
2 files changed, 9 insertions(+)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 6f67c277..787c51bf 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -82,6 +82,11 @@ doesn't point to stdout, nothing otherwise.
Augment output destination instead of clobbering it.
+=item --no-import-before
+
+Do not importing keywords before writing to an existing output
+destination.
+
=item -t, --threads
Return all messages in the same thread as the actual match(es).
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 36abe513..0b81b9f3 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -23,6 +23,10 @@ Available in public-inbox 1.7.0+.
=over
+=item -c NAME=VALUE
+
+Override configuration C<NAME> to C<VALUE>.
+
=item -C DIR
Change current working directory to the specified directory before
--
2.31.0
^ permalink raw reply related [relevance 71%]
* [PATCH 4/8] doc lei: don't render most to-do comments
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (2 preceding siblings ...)
2021-03-29 3:11 71% ` [PATCH 3/8] doc lei: drop an unnecessary to-do comment Kyle Meyer
@ 2021-03-29 3:11 68% ` Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 5/8] doc lei: update manpages with new options Kyle Meyer
` (4 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
The lei manpages have a number of to-dos, but with the exception of
the lei-q's -tt warning, none of them seem worth displaying to the
reader (and some might not be worth addressing at all).
---
Documentation/lei-add-external.pod | 1 +
Documentation/lei-q.pod | 4 ++++
2 files changed, 5 insertions(+)
diff --git a/Documentation/lei-add-external.pod b/Documentation/lei-add-external.pod
index 3bc0ba83..47158146 100644
--- a/Documentation/lei-add-external.pod
+++ b/Documentation/lei-add-external.pod
@@ -15,6 +15,7 @@ C<extindex.<name>.topdir> value in ~/.public-inbox/config.
=head1 OPTIONS
+=for comment
TODO: mention curl options?
=over
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index cb1227fb..6f67c277 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -12,10 +12,12 @@ lei q [OPTIONS] (--stdin|-)
Search for messages across the lei store and externals.
+=for comment
TODO: Give common prefixes, or at least a description/reference.
=head1 OPTIONS
+=for comment
TODO: mention curl options?
=over
@@ -40,6 +42,7 @@ the destination. C<json> is used for the default destination
(stdout), and C<maildir> is used for an existing directory or
non-existing path.
+=for comment
TODO: Provide description of formats?
Default: -
@@ -98,6 +101,7 @@ C<none>.
Default: C<content>
+=for comment
TODO: Provide description of strategies?
=item --[no-]remote
--
2.31.0
^ permalink raw reply related [relevance 68%]
* [PATCH 6/8] doc lei: add manpages for new commands
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (4 preceding siblings ...)
2021-03-29 3:11 71% ` [PATCH 5/8] doc lei: update manpages with new options Kyle Meyer
@ 2021-03-29 3:11 30% ` Kyle Meyer
2021-03-29 3:25 71% ` Eric Wong
2021-03-29 3:11 90% ` [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary Kyle Meyer
` (2 subsequent siblings)
8 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-blob.pod | 109 +++++++++++++++++++++++++++++++++
Documentation/lei-ls-label.pod | 43 +++++++++++++
Documentation/lei-mark.pod | 58 ++++++++++++++++++
Documentation/lei-overview.pod | 21 +++++++
Documentation/lei-p2q.pod | 79 ++++++++++++++++++++++++
Documentation/lei.pod | 8 +++
Documentation/txt2pre | 4 ++
MANIFEST | 4 ++
Makefile.PL | 5 +-
9 files changed, 329 insertions(+), 2 deletions(-)
create mode 100644 Documentation/lei-blob.pod
create mode 100644 Documentation/lei-ls-label.pod
create mode 100644 Documentation/lei-mark.pod
create mode 100644 Documentation/lei-p2q.pod
diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod
new file mode 100644
index 00000000..ecdd1e99
--- /dev/null
+++ b/Documentation/lei-blob.pod
@@ -0,0 +1,109 @@
+=head1 NAME
+
+lei-blob - display a git blob, reconstructing from mail if necessary
+
+=head1 SYNOPSIS
+
+lei blob [OPTIONS] OID
+
+=head1 DESCRIPTION
+
+Display a git blob. The blob may correspond to a message from the
+local store, an existing blob in the current repository, or a
+not-yet-created blob in the current repository that can be
+reconstructed from a message.
+
+=head1 OPTIONS
+
+=over
+
+=item --git-dir=DIR
+
+Specify an additional .git/ directory to scan. This option may be
+given multiple times.
+
+=item --no-cwd
+
+Do not look in the git repository of the current working directory.
+
+=item --no-mail
+
+Do not look in mail storage for C<OID>. This is implied by
+C<--oid-a>, C<--path-a>, and C<--path-b>.
+
+=item -A OID-A, --oid-a=OID-A
+
+=item -a PATH-A, --path-a=PATH-A
+
+=item -b PATH-B, --path-b=PATH-B
+
+Provide pre-image object ID, pre-image pathname, or post-image
+pathname as a hint for reconstructing C<OID>.
+
+=for comment
+TODO: The below options are shared with lei-q. Any good approaches to
+not repeating the text?
+
+=item --[no-]remote
+
+Whether to include results requiring network access. When local
+externals are configured, C<--remote> must be explicitly passed to
+enable reporting of results from remote externals.
+
+=item --no-local
+
+Limit operations to those requiring network access.
+
+=item --no-external
+
+Don't include results from externals.
+
+=item -I LOCATION, --include=LOCATION
+
+Include specified external in search. This option may be given
+multiple times.
+
+=item --exclude=LOCATION
+
+Exclude specified external from search. This option may be given
+multiple times.
+
+=item --only=LOCATION
+
+Use only the specified external for search. This option may be given
+multiple times, in which case the search uses only the specified set.
+
+=item --no-import-remote
+
+Disable the default behavior of memoizing remote messages into the
+local store.
+
+=item -v, --verbose
+
+Provide more feedback on stderr.
+
+=item --torsocks=auto|no|yes, --no-torsocks
+
+Whether to wrap L<git(1)> and L<curl(1)> commands with torsocks.
+
+Default: C<auto>
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-external(1)>
diff --git a/Documentation/lei-ls-label.pod b/Documentation/lei-ls-label.pod
new file mode 100644
index 00000000..0b4e8769
--- /dev/null
+++ b/Documentation/lei-ls-label.pod
@@ -0,0 +1,43 @@
+=head1 NAME
+
+lei-ls-label - list labels
+
+=head1 SYNOPSIS
+
+lei ls-label [OPTIONS]
+
+=head1 DESCRIPTION
+
+List all known message labels ("mailboxes" in JMAP terminology).
+
+=head1 OPTIONS
+
+=over
+
+=item -z, -0
+
+Use C<\0> (NUL) instead of newline (CR) to delimit lines.
+
+=item -q, --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-external(1)>
diff --git a/Documentation/lei-mark.pod b/Documentation/lei-mark.pod
new file mode 100644
index 00000000..8ef1dce2
--- /dev/null
+++ b/Documentation/lei-mark.pod
@@ -0,0 +1,58 @@
+=head1 NAME
+
+lei-mark - set/unset metadata on messages
+
+=head1 SYNOPSIS
+
+lei mark [OPTIONS] FILE [FILE...] METADATA [METADATA...]
+
+lei mark [OPTIONS] (-|--stdin) METADATA [METADATA...]
+
+=head1 DESCRIPTION
+
+Set or unset volatile metadata on messages. In JMAP terms, "volatile
+metadata" includes "mailboxes" (analogous to a folder or label) and a
+restricted set of "keywords". This supported keywords are the
+combination of system keywords (seen, answered, flagged, and draft),
+which map to Maildir flags and mbox Status/X-Status headers, as well
+as reserved keywords (forwarded, phishing, junk, and notjunk).
+
+To add a label or keyword, prefix it with "+L:" and "+kw:",
+respectively. To remove a label or keyword, use "-L:" or "-kw:". For
+example, "+kw:flagged" would set the "flagged" keyword for the
+specified messages, and "-L:INBOX" would remove the "INBOX" label.
+
+=head1 OPTIONS
+
+=over
+
+=item -F MAIL_FORMAT, --in-format=MAIL_FORMAT
+
+Message input format: C<eml>, C<mboxrd>, C<mboxcl2>, C<mboxcl>, or
+C<mboxo>.
+
+Default: C<eml>
+
+=item -q, --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-external(1)>
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index c3379caa..7c7337ab 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -20,6 +20,17 @@ L<public-inbox-v2-format(5)>.
Import the messages from an mbox into the local storage.
+=item $ lei blob 59ec517f9
+
+Show message with the git blob OID of 59ec517f9. If a message with
+that OID isn't found, check if the current git repository has the
+blob, trying to reconstruct it from a message if needed.
+
+=item $ lei blob 59ec517f9 | lei mark - -F eml +kw:flagged +L:next
+
+Set the "flagged" keyword and "next" label on the message with the
+blob OID of 59ec517f9.
+
=back
=head1 EXTERNALS
@@ -69,6 +80,16 @@ thread as a matched message.
Write mboxcl2-formatted results to t.mbox and enter mutt to view the
file by invoking C<mutt -f %f>.
+=item $ lei q kw:flagged L:next
+
+Search for all flagged messages that also have a "next" label.
+
+=item $ lei p2q HEAD | lei q --stdin -tt -o mdir
+
+Search for messages that have post-image git blob IDs that match those
+of the current repository's HEAD commit, writing them to the Maildir
+directory "mdir" and flagging the messages that were an exact match.
+
=back
=head1 PERFORMANCE NOTES
diff --git a/Documentation/lei-p2q.pod b/Documentation/lei-p2q.pod
new file mode 100644
index 00000000..cc342bd5
--- /dev/null
+++ b/Documentation/lei-p2q.pod
@@ -0,0 +1,79 @@
+=head1 NAME
+
+lei-p2q - use a patch to generate a lei-q query
+
+=head1 SYNOPSIS
+
+lei p2q [OPTIONS] (FILE|COMMIT)
+
+lei p2q [OPTIONS] (--stdin|-)
+
+=head1 DESCRIPTION
+
+Given a patch, create a query that can be fed on stdin to L<lei-q(1)>.
+This is useful for mapping the patch to associated messages of an
+inbox.
+
+The patch can be provided on stdin or as a file. Alternatively, when
+an argument is given that does not point to an existing file, it is
+taken as a reference to a commit in the current repository, and
+L<git-format-patch(1)> is used to generate the patch.
+
+=head1 OPTIONS
+
+=over
+
+=item -w PREFIX[,PREFIX], --want=PREFIX[,PREFIX]
+
+Search prefixes to use. C<dfpost> (post-image git blob ID) and C<dfn>
+(file names from the diff) are the most useful. Other available
+values are C<dfa>, C<dfb>, C<dfctx>, C<dfhh>, and C<dfpre>.
+
+=for comment
+TODO: Put a table of prefixes somewhere and reference that (at least
+here and in lei-q)?
+
+Appending an integer to C<dfpost> or C<dfpre> indicates a minimum ID
+length, and the generated query will be for that value up through the
+default abbreviation length. For example, if the repository's
+C<core.abbrev> is set to C<auto> and git calculates the default
+abbreviation length as 7, C<dfpost6> will expand a post-image blob ID
+of e7b4b32 (seven characters) into C<dfpost:e7b4b32 OR dfpost:e7b4b3>.
+
+This option may be given multiple times.
+
+Default: C<dfpost7>
+
+=item --stdin
+
+Read patch from stdin.
+
+=item --debug
+
+Dump output that shows the information collected for every prefix.
+This information can be useful for seeing how a patch is processed,
+but the format should not be considered stable.
+
+=item -q, --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/>
+and L<http://hjrcffqmbrq6wope.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+
+=head1 SEE ALSO
+
+L<lei-add-external(1)>
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 0b81b9f3..6c8cfc3d 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -46,6 +46,8 @@ Subcommands for initializing and managing local, writable storage:
=item * L<lei-import(1)>
+=item * L<lei-mark(1)>
+
=back
The following subcommands can be used to manage and inspect external
@@ -66,6 +68,10 @@ store and configured externals are
=over
+=item * L<lei-blob(1)>
+
+=item * L<lei-p2q(1)>
+
=item * L<lei-q(1)>
=back
@@ -80,6 +86,8 @@ Other subcommands include
=item * L<lei-daemon-pid(1)>
+=item * L<lei-ls-label(1)>
+
=back
=head1 FILES
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 244dc50c..bfffdef1 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -12,6 +12,7 @@ use PublicInbox::Hval qw(ascii_html);
my %xurls;
for (qw[lei(1)
lei-add-external(1)
+ lei-blob(1)
lei-config(1)
lei-daemon-kill(1)
lei-daemon-pid(1)
@@ -19,7 +20,10 @@ for (qw[lei(1)
lei-import(1)
lei-init(1)
lei-ls-external(1)
+ lei-ls-label(1)
+ lei-mark(1)
lei-overview(7)
+ lei-p2q(1)
lei-q(1)
public-inbox.cgi(1)
public-inbox-compact(1)
diff --git a/MANIFEST b/MANIFEST
index 913ce55c..3d521a64 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -22,6 +22,7 @@ Documentation/flow.txt
Documentation/hosted.txt
Documentation/include.mk
Documentation/lei-add-external.pod
+Documentation/lei-blob.pod
Documentation/lei-config.pod
Documentation/lei-daemon-kill.pod
Documentation/lei-daemon-pid.pod
@@ -29,7 +30,10 @@ Documentation/lei-forget-external.pod
Documentation/lei-import.pod
Documentation/lei-init.pod
Documentation/lei-ls-external.pod
+Documentation/lei-ls-label.pod
+Documentation/lei-mark.pod
Documentation/lei-overview.pod
+Documentation/lei-p2q.pod
Documentation/lei-q.pod
Documentation/lei.pod
Documentation/marketing.txt
diff --git a/Makefile.PL b/Makefile.PL
index 8165e601..cdb67214 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -44,8 +44,9 @@ $v->{-m1} = [ map {
}
} @EXE_FILES,
qw(
- lei-add-external lei-config lei-daemon-kill lei-daemon-pid
- lei-forget-external lei-import lei-init lei-ls-external lei-q)];
+ lei-add-external lei-blob lei-config lei-daemon-kill lei-daemon-pid
+ lei-forget-external lei-import lei-init lei-ls-external lei-ls-label
+ lei-mark lei-p2q lei-q)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format) ];
$v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
--
2.31.0
^ permalink raw reply related [relevance 30%]
* [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (5 preceding siblings ...)
2021-03-29 3:11 30% ` [PATCH 6/8] doc lei: add manpages for new commands Kyle Meyer
@ 2021-03-29 3:11 90% ` Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 8/8] doc lei overview: better explain routes into local store Kyle Meyer
2021-03-29 3:18 71% ` [PATCH 0/8] doc: lei manpages, round 4 Eric Wong
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
cf. https://public-inbox.org/meta/20210325083207.GA30551@dcvr
---
Documentation/lei-overview.pod | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 7c7337ab..55e84254 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -9,8 +9,10 @@ provides some basic examples.
=head1 LEI STORE
-L<lei-init(1)> initializes writable local storage based on
-L<public-inbox-v2-format(5)>.
+lei has writable local storage based on L<public-inbox-v2-format(5)>.
+Commands will automatically initialize the store behind the scenes if
+needed, but you can call L<lei-init(1)> directly if you want to use a
+store location other than the default C<$XDG_DATA_HOME/lei/store>.
=head2 EXAMPLES
--
2.31.0
^ permalink raw reply related [relevance 90%]
* [PATCH 8/8] doc lei overview: better explain routes into local store
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (6 preceding siblings ...)
2021-03-29 3:11 90% ` [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary Kyle Meyer
@ 2021-03-29 3:11 71% ` Kyle Meyer
2021-03-29 3:18 71% ` [PATCH 0/8] doc: lei manpages, round 4 Eric Wong
8 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:11 UTC (permalink / raw)
To: meta
---
Documentation/lei-overview.pod | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 55e84254..f74a228a 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -14,6 +14,11 @@ Commands will automatically initialize the store behind the scenes if
needed, but you can call L<lei-init(1)> directly if you want to use a
store location other than the default C<$XDG_DATA_HOME/lei/store>.
+The L<lei-import(1)> command provides the primary interface for
+importing messages into the local storage. In addition, other
+commands, such as L<lei-q(1)> and L<lei-blob(1)>, use the local store
+to memoize messages from remotes.
+
=head2 EXAMPLES
=over
--
2.31.0
^ permalink raw reply related [relevance 71%]
* Re: [PATCH 0/8] doc: lei manpages, round 4
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
` (7 preceding siblings ...)
2021-03-29 3:11 71% ` [PATCH 8/8] doc lei overview: better explain routes into local store Kyle Meyer
@ 2021-03-29 3:18 71% ` Eric Wong
8 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 3:18 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Thanks, pushed as commit 165eabbffd15866f581ca374cb37fbf3c07989a5
^ permalink raw reply [relevance 71%]
* Re: [PATCH 6/8] doc lei: add manpages for new commands
2021-03-29 3:11 30% ` [PATCH 6/8] doc lei: add manpages for new commands Kyle Meyer
@ 2021-03-29 3:25 71% ` Eric Wong
2021-03-29 3:35 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-29 3:25 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> +=for comment
> +TODO: The below options are shared with lei-q. Any good approaches to
> +not repeating the text?
I don't know about natively in POD. Perhaps just a list of
switches and a pointer to lei-q(1)?
Or maybe we generate some .pod files via cat + sh
cat lei-blob.pod.in lei-q-common.pod.in >lei-blob.pod
I don't know if depending on cpp or some other widely-installed
preprocessor is worth it.
> +=for comment
> +TODO: Put a table of prefixes somewhere and reference that (at least
> +here and in lei-q)?
Yes, probably.
^ permalink raw reply [relevance 71%]
* Re: [PATCH 6/8] doc lei: add manpages for new commands
2021-03-29 3:25 71% ` Eric Wong
@ 2021-03-29 3:35 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-29 3:35 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Kyle Meyer <kyle@kyleam.com> wrote:
>> +=for comment
>
>> +TODO: The below options are shared with lei-q. Any good approaches to
>> +not repeating the text?
>
> I don't know about natively in POD. Perhaps just a list of
> switches and a pointer to lei-q(1)?
Yeah, this sounds like the way to go to me. I'll make a note to do it
for the next round.
Thanks.
^ permalink raw reply [relevance 71%]
* [PATCH 0/3] lei input improvements
@ 2021-03-29 7:08 71% Eric Wong
2021-03-29 7:08 70% ` [PATCH 2/3] lei: use IO::Uncompress::Gunzip MultiStream Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-03-29 7:08 UTC (permalink / raw)
To: meta
These affect the convert, import, mark sub-commands.
Eric Wong (3):
lei_input: avoid special case sub for --stdin
lei: use IO::Uncompress::Gunzip MultiStream
lei_input: treat ".eml" and ".patch" suffix as "eml"
lib/PublicInbox/LeiConvert.pm | 1 -
lib/PublicInbox/LeiImport.pm | 1 -
lib/PublicInbox/LeiInput.pm | 33 +++++++++++++++++++--------------
lib/PublicInbox/LeiMark.pm | 1 -
lib/PublicInbox/LeiRemote.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 2 +-
t/lei-import.t | 2 +-
t/lei-mark.t | 4 ++--
8 files changed, 24 insertions(+), 22 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 2/3] lei: use IO::Uncompress::Gunzip MultiStream
2021-03-29 7:08 71% [PATCH 0/3] lei input improvements Eric Wong
@ 2021-03-29 7:08 70% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 7:08 UTC (permalink / raw)
To: meta
This is compatible with default gunzip(1) behavior and
future-proofs us against potential changes in PublicInbox::WWW
to save memory on public-inbox-httpd instances.
---
lib/PublicInbox/LeiRemote.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
index 399fc936..945d9990 100644
--- a/lib/PublicInbox/LeiRemote.pm
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -50,7 +50,7 @@ sub mset {
my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
$self->{smsg} = [];
- $fh = IO::Uncompress::Gunzip->new($fh);
+ $fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&_each_mboxrd_eml, $self);
my $err = waitpid($pid, 0) == $pid ? undef
: "BUG: waitpid($cmd): $!";
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 1a194f1c..f3b8cc25 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -272,7 +272,7 @@ sub query_remote_mboxrd {
$lei->qerr("# $cmd");
my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
$reap_curl = PublicInbox::OnDestroy->new($sigint_reap, $pid);
- $fh = IO::Uncompress::Gunzip->new($fh);
+ $fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
PublicInbox::MboxReader->mboxrd($fh, \&each_remote_eml, $self,
$lei, $each_smsg);
my $err = waitpid($pid, 0) == $pid ? undef
^ permalink raw reply related [relevance 70%]
* [PATCH 1/4] doc: lei q: drop NNTP from --output description
@ 2021-03-29 8:04 71% ` Eric Wong
2021-03-29 8:04 71% ` [PATCH 2/4] doc: lei q: add warning for --output clobbering Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 8:04 UTC (permalink / raw)
To: meta
We only support NNTP as inputs for convert, import, and
mark|tag. I'm not sure if supporting NNTP output is worth
it, nor do we have a good way to test it.
---
Documentation/lei-q.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 787c51bf..8668533a 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -31,7 +31,7 @@ Read search terms from stdin.
Destination for results (e.g., C<path/to/Maildir>,
C<imaps://user@mail.example.com/INBOX.test>, or
C<mboxcl2:path/to/mbox>). The prefix may be a supported protocol:
-C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>. URLs requiring
+C<imap://> or C<imaps://>. URLs requiring
authentication must use L<netrc(5)> and/or L<git-credential(1)> to
fill in the username and password.
^ permalink raw reply related [relevance 71%]
* [PATCH 2/4] doc: lei q: add warning for --output clobbering
2021-03-29 8:04 71% ` [PATCH 1/4] doc: lei q: drop NNTP from --output description Eric Wong
@ 2021-03-29 8:04 71% ` Eric Wong
2021-03-29 8:04 71% ` [PATCH 3/4] doc: lei q: clarify default output as stdout Eric Wong
2021-03-29 8:04 67% ` [PATCH 4/4] doc: lei: update description, add warnings Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 8:04 UTC (permalink / raw)
To: meta
The behavior matching mairix still frightens me a bit when it
comes to supporting new users. On the other hand, I've rarely
ever used --augment with mairix, so I still think the current
(dangerous) behavior makes sense in the context of search results.
---
Documentation/lei-q.pod | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 8668533a..8d5053cd 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -28,6 +28,9 @@ Read search terms from stdin.
=item -o MFOLDER, --output=MFOLDER, --mfolder=MFOLDER
+Warning: this clobbers and overwrites the output destination unless
+L</-a, --augment> is specified.
+
Destination for results (e.g., C<path/to/Maildir>,
C<imaps://user@mail.example.com/INBOX.test>, or
C<mboxcl2:path/to/mbox>). The prefix may be a supported protocol:
^ permalink raw reply related [relevance 71%]
* [PATCH 3/4] doc: lei q: clarify default output as stdout
2021-03-29 8:04 71% ` [PATCH 1/4] doc: lei q: drop NNTP from --output description Eric Wong
2021-03-29 8:04 71% ` [PATCH 2/4] doc: lei q: add warning for --output clobbering Eric Wong
@ 2021-03-29 8:04 71% ` Eric Wong
2021-03-29 8:04 67% ` [PATCH 4/4] doc: lei: update description, add warnings Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 8:04 UTC (permalink / raw)
To: meta
Seeing a plain "-" may be confusing, especially when we also
support it for --stdin. Use the C<> POD directive to denote it
as code, too.
---
Documentation/lei-q.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 8d5053cd..a84fc440 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -48,7 +48,7 @@ non-existing path.
=for comment
TODO: Provide description of formats?
-Default: -
+Default: C<-> (stdout)
=item -f FORMAT, --format=FORMAT
^ permalink raw reply related [relevance 71%]
* [PATCH 4/4] doc: lei: update description, add warnings
` (2 preceding siblings ...)
2021-03-29 8:04 71% ` [PATCH 3/4] doc: lei q: clarify default output as stdout Eric Wong
@ 2021-03-29 8:04 67% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 8:04 UTC (permalink / raw)
To: meta
Describing the design details and architecture doesn't seem
appropriate for a section 1 manpage.
We'll also add a warning about it being in the early stages.
---
Documentation/lei.pod | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 6c8cfc3d..248e5931 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -1,6 +1,6 @@
=head1 NAME
-lei - local email interface for public-inbox
+lei - local email interface
=head1 SYNOPSIS
@@ -8,15 +8,17 @@ lei [OPTIONS] COMMAND
=head1 DESCRIPTION
-Unlike the C10K-oriented L<public-inbox-daemon(8)>, lei is designed
-exclusively to handle trusted local clients with read/write access to
-the file system, using as many system resources as the local user has
-access to. lei supports a local, writable store built on top of
+lei is a command-line tool for importing and searching email,
+regardless of whether it is from a personal mailbox or a public-inbox.
+lei supports a local, writable store built on top of
L<public-inbox-v2-format(5)> and L<public-inbox-extindex(1)>.
L<lei-q(1)> provides an interface for querying messages across the lei
store and read-only local and remote "externals" (inboxes and external
indices).
+Warning: lei is still in its early stages and may destroy mail.
+Be sure to have backups of destinations lei writes to.
+
Available in public-inbox 1.7.0+.
=head1 OPTIONS
^ permalink raw reply related [relevance 67%]
* [PATCH] lei blob: cleanup solver tmpdir on failure
@ 2021-03-29 17:47 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-29 17:47 UTC (permalink / raw)
To: meta
$lei->fail sends SIGTERM which prevents the File::Temp::Dir in
$solver->{tmp} from being cleaned up, so use $lei->child_error
instead.
---
lib/PublicInbox/LeiBlob.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 91098a90..6195b368 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -39,7 +39,7 @@ sub solver_user_cb { # called by solver when done
my $lei = $self->{lei};
my $log_buf = delete $lei->{'log_buf'};
$$log_buf =~ s/^/# /sgm;
- ref($res) eq 'ARRAY' or return $lei->fail($$log_buf);
+ ref($res) eq 'ARRAY' or return $lei->child_error(1 << 8, $$log_buf);
$lei->qerr($$log_buf);
my ($git, $oid, $type, $size, $di) = @$res;
my $gd = $git->{git_dir};
^ permalink raw reply related [relevance 71%]
* [PATCH] lei q: avoid redundant default setting for sort with l2m
@ 2021-03-30 9:10 70% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-30 9:10 UTC (permalink / raw)
To: meta
No point in munging user-supplied $lei->{opt} when %mset_opt
exists. We'll be depending on docid being in descending order
for saved search support.
---
lib/PublicInbox/LeiOverview.pm | 2 --
lib/PublicInbox/LeiQuery.pm | 4 ++--
2 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 68f6c792..cdd9ee04 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -101,8 +101,6 @@ sub new {
if ($json) {
$lei->{dedupe} //= PublicInbox::LeiDedupe->new($lei);
} else {
- # default to the cheapest sort since MUA usually resorts
- $opt->{'sort'} //= 'docid' if $devfd < 0;
$lei->{l2m} = eval { PublicInbox::LeiToMail->new($lei) };
return $lei->fail($@) if $@;
if ($opt->{mua} && $lei->{l2m}->lock_free) {
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 5376c7f8..3a437bf0 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -128,8 +128,8 @@ sub lei_q {
die "unrecognized --sort=$sort\n";
}
}
- # descending docid order
- $mset_opt{relevance} //= -2 if $opt->{threads};
+ # descending docid order is cheapest, MUA controls sorting order
+ $mset_opt{relevance} //= -2 if $self->{l2m} || $opt->{threads};
$self->{mset_opt} = \%mset_opt;
if ($opt->{stdin}) {
^ permalink raw reply related [relevance 70%]
* [PATCH] lei tag: rename from "lei mark"
@ 2021-03-30 9:39 51% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-30 9:39 UTC (permalink / raw)
To: meta
I've decided "tag" is a better verb since it seems more
widely-used term for associating metadata with data.
Not only is it analogous to the "notmuch tag" command, but
also makes sense when compared to tooling for manipulating
metadata for non-mail data (e.g. audio metadata tags).
There's even a Wikipedia entry for it:
https://en.wikipedia.org/wiki/Tag_(metadata)
whereas "mark" is used in the description, but has no
entry of its own with regards to metadata.
---
Documentation/lei-overview.pod | 2 +-
Documentation/{lei-mark.pod => lei-tag.pod} | 6 ++---
Documentation/lei.pod | 2 +-
Documentation/txt2pre | 2 +-
MANIFEST | 6 ++---
Makefile.PL | 2 +-
lib/PublicInbox/LEI.pm | 6 ++---
lib/PublicInbox/{LeiMark.pm => LeiTag.pm} | 26 ++++++++++-----------
t/{lei-mark.t => lei-tag.t} | 22 ++++++++---------
9 files changed, 37 insertions(+), 37 deletions(-)
rename Documentation/{lei-mark.pod => lei-tag.pod} (89%)
rename lib/PublicInbox/{LeiMark.pm => LeiTag.pm} (90%)
rename t/{lei-mark.t => lei-tag.t} (83%)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index f74a228a..c1f952c9 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -33,7 +33,7 @@ Show message with the git blob OID of 59ec517f9. If a message with
that OID isn't found, check if the current git repository has the
blob, trying to reconstruct it from a message if needed.
-=item $ lei blob 59ec517f9 | lei mark - -F eml +kw:flagged +L:next
+=item $ lei blob 59ec517f9 | lei tag - -F eml +kw:flagged +L:next
Set the "flagged" keyword and "next" label on the message with the
blob OID of 59ec517f9.
diff --git a/Documentation/lei-mark.pod b/Documentation/lei-tag.pod
similarity index 89%
rename from Documentation/lei-mark.pod
rename to Documentation/lei-tag.pod
index 8ef1dce2..a07738d7 100644
--- a/Documentation/lei-mark.pod
+++ b/Documentation/lei-tag.pod
@@ -1,12 +1,12 @@
=head1 NAME
-lei-mark - set/unset metadata on messages
+lei-tag - set/unset metadata on messages
=head1 SYNOPSIS
-lei mark [OPTIONS] FILE [FILE...] METADATA [METADATA...]
+lei tag [OPTIONS] FILE [FILE...] METADATA [METADATA...]
-lei mark [OPTIONS] (-|--stdin) METADATA [METADATA...]
+lei tag [OPTIONS] (-|--stdin) METADATA [METADATA...]
=head1 DESCRIPTION
diff --git a/Documentation/lei.pod b/Documentation/lei.pod
index 248e5931..805e5a75 100644
--- a/Documentation/lei.pod
+++ b/Documentation/lei.pod
@@ -48,7 +48,7 @@ Subcommands for initializing and managing local, writable storage:
=item * L<lei-import(1)>
-=item * L<lei-mark(1)>
+=item * L<lei-tag(1)>
=back
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index bfffdef1..7b9d7853 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -21,7 +21,7 @@ for (qw[lei(1)
lei-init(1)
lei-ls-external(1)
lei-ls-label(1)
- lei-mark(1)
+ lei-tag(1)
lei-overview(7)
lei-p2q(1)
lei-q(1)
diff --git a/MANIFEST b/MANIFEST
index 3d521a64..f3cb0147 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -31,10 +31,10 @@ Documentation/lei-import.pod
Documentation/lei-init.pod
Documentation/lei-ls-external.pod
Documentation/lei-ls-label.pod
-Documentation/lei-mark.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
Documentation/lei-q.pod
+Documentation/lei-tag.pod
Documentation/lei.pod
Documentation/marketing.txt
Documentation/mknews.perl
@@ -195,7 +195,6 @@ lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiLsLabel.pm
-lib/PublicInbox/LeiMark.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
@@ -203,6 +202,7 @@ lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
+lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
lib/PublicInbox/LeiXSearch.pm
lib/PublicInbox/Linkify.pm
@@ -386,12 +386,12 @@ t/lei-import-imap.t
t/lei-import-maildir.t
t/lei-import-nntp.t
t/lei-import.t
-t/lei-mark.t
t/lei-mirror.t
t/lei-p2q.t
t/lei-q-kw.t
t/lei-q-remote-import.t
t/lei-q-thread.t
+t/lei-tag.t
t/lei.t
t/lei_dedupe.t
t/lei_external.t
diff --git a/Makefile.PL b/Makefile.PL
index cdb67214..27b49c53 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -46,7 +46,7 @@ $v->{-m1} = [ map {
qw(
lei-add-external lei-blob lei-config lei-daemon-kill lei-daemon-pid
lei-forget-external lei-import lei-init lei-ls-external lei-ls-label
- lei-mark lei-p2q lei-q)];
+ lei-tag lei-p2q lei-q)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format) ];
$v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8a07a4c8..69d48bd1 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -160,8 +160,8 @@ our %CMD = ( # sorted in order of importance/use:
'plonk' => [ '--threads|--from=IDENT',
'exclude mail matching From: or threads from non-Message-ID searches',
qw(stdin| threads|t from|f=s mid=s oid=s), @c_opt ],
-'mark' => [ 'KEYWORDS...',
- 'set/unset keywords on message(s)',
+'tag' => [ 'KEYWORDS...',
+ 'set/unset keywords and/or labels on message(s)',
qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@), @c_opt,
pass_through('-kw:foo for delete') ],
'forget' => [ '[--stdin|--oid=OID|--by-mid=MID]',
@@ -348,7 +348,7 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q mark sol); # internal workers
+my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol); # internal workers
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
sub x_it ($$) {
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiTag.pm
similarity index 90%
rename from lib/PublicInbox/LeiMark.pm
rename to lib/PublicInbox/LeiTag.pm
index b187d6e7..56ac25fa 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -1,8 +1,8 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# handles "lei mark" command
-package PublicInbox::LeiMark;
+# handles "lei tag" command
+package PublicInbox::LeiTag;
use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
@@ -69,19 +69,19 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub input_mbox_cb { input_eml_cb($_[1], $_[0]) }
-sub mark_done_wait { # dwaitpid callback
+sub tag_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
- my ($mark, $lei) = @$arg;
- $lei->child_error($?, 'non-fatal errors during mark') if $?;
+ my ($tag, $lei) = @$arg;
+ $lei->child_error($?, 'non-fatal errors during tag') if $?;
my $sto = delete $lei->{sto};
my $wait = $sto->ipc_do('done') if $sto; # PublicInbox::LeiStore::done
$lei->dclose;
}
-sub mark_done { # EOF callback for main daemon
+sub tag_done { # EOF callback for main daemon
my ($lei) = @_;
- my $mark = delete $lei->{mark} or return;
- $mark->wq_wait_old(\&mark_done_wait, $lei);
+ my $tag = delete $lei->{tag} or return;
+ $tag->wq_wait_old(\&tag_done_wait, $lei);
}
sub net_merge_complete { # callback used by LeiAuth
@@ -102,7 +102,7 @@ sub input_net_cb { # imap_each, nntp_each cb
input_eml_cb($self, $eml);
}
-sub lei_mark { # the "lei mark" method
+sub lei_tag { # the "lei tag" method
my ($lei, @argv) = @_;
my $sto = $lei->_lei_store(1);
$sto->write_prepare($lei);
@@ -113,11 +113,11 @@ sub lei_mark { # the "lei mark" method
$self->prepare_inputs($lei, \@argv) or return;
grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
return $lei->fail('no keywords or labels specified');
- my $ops = { '' => [ \&mark_done, $lei ] };
+ my $ops = { '' => [ \&tag_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- my ($op_c, undef) = $lei->workers_start($self, 'lei_mark', 1, $ops);
- $lei->{mark} = $self;
+ my ($op_c, undef) = $lei->workers_start($self, 'lei_tag', 1, $ops);
+ $lei->{tag} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
}
@@ -165,7 +165,7 @@ sub _complete_mark_common ($) {
}
# FIXME: same problems as _complete_forget_external and similar
-sub _complete_mark {
+sub _complete_tag {
my ($self, @argv) = @_;
my @L = eval { $self->_lei_store->search->all_terms('L') };
my @all = ((map { ("+kw:$_", "-kw:$_") } @KW),
diff --git a/t/lei-mark.t b/t/lei-tag.t
similarity index 83%
rename from t/lei-mark.t
rename to t/lei-tag.t
index 98652c85..5cb6d9ce 100644
--- a/t/lei-mark.t
+++ b/t/lei-tag.t
@@ -27,13 +27,13 @@ my $check_kw = sub {
test_lei(sub {
lei_ok(qw(ls-label)); is($lei_out, '', 'no labels, yet');
lei_ok(qw(import t/utf8.eml));
- lei_ok(qw(mark t/utf8.eml +kw:flagged +L:urgent));
+ lei_ok(qw(tag t/utf8.eml +kw:flagged +L:urgent));
$check_kw->(['flagged'], L => ['urgent']);
lei_ok(qw(ls-label)); is($lei_out, "urgent\n", 'label found');
- ok(!lei(qw(mark -F eml t/utf8.eml +kw:seeen)), 'bad kw rejected');
+ ok(!lei(qw(tag -F eml t/utf8.eml +kw:seeen)), 'bad kw rejected');
like($lei_err, qr/`seeen' is not one of/, 'got helpful error');
- ok(!lei(qw(mark -F eml t/utf8.eml +k:seen)), 'bad prefix rejected');
- ok(!lei(qw(mark -F eml t/utf8.eml)), 'no keywords');
+ ok(!lei(qw(tag -F eml t/utf8.eml +k:seen)), 'bad prefix rejected');
+ ok(!lei(qw(tag -F eml t/utf8.eml)), 'no keywords');
my $mb = "$ENV{HOME}/mb";
my $md = "$ENV{HOME}/md";
lei_ok(qw(q m:testmessage@example.com -o), "mboxrd:$mb");
@@ -43,15 +43,15 @@ test_lei(sub {
scalar(@fn) == 1 or xbail $lei_err, 'no mail', \@fn;
rename($fn[0], "$fn[0]S") or BAIL_OUT "rename $!";
$check_kw->(['flagged'], msg => 'after bad request');
- lei_ok(qw(mark -F eml t/utf8.eml -kw:flagged));
+ lei_ok(qw(tag -F eml t/utf8.eml -kw:flagged));
$check_kw->(undef, msg => 'keyword cleared');
- lei_ok(qw(mark -F mboxrd +kw:seen), $mb);
+ lei_ok(qw(tag -F mboxrd +kw:seen), $mb);
$check_kw->(['seen'], msg => 'mbox Status ignored');
- lei_ok(qw(mark -kw:seen +kw:answered), $md);
+ lei_ok(qw(tag -kw:seen +kw:answered), $md);
$check_kw->(['answered'], msg => 'Maildir Status ignored');
open my $in, '<', 't/utf8.eml' or BAIL_OUT $!;
- lei_ok([qw(mark -F eml - +kw:seen +L:nope)],
+ lei_ok([qw(tag -F eml - +kw:seen +L:nope)],
undef, { %$lei_opt, 0 => $in });
$check_kw->(['answered', 'seen'], msg => 'stdin works');
lei_ok(qw(q L:urgent));
@@ -62,7 +62,7 @@ test_lei(sub {
is_deeply($r2, $res, 'kw: query works, too') or
diag explain([$r2, $res]);
- lei_ok(qw(_complete lei mark));
+ lei_ok(qw(_complete lei tag));
my %c = map { $_ => 1 } split(/\s+/, $lei_out);
ok($c{'+L:urgent'} && $c{'-L:urgent'} &&
$c{'+L:nope'} && $c{'-L:nope'}, 'completed with labels');
@@ -70,7 +70,7 @@ test_lei(sub {
my $mid = 'qp@example.com';
lei_ok qw(q -f mboxrd --only), "$ro_home/t2", "mid:$mid";
$in = $lei_out;
- lei_ok [qw(mark -F mboxrd --stdin +kw:seen +L:qp)],
+ lei_ok [qw(tag -F mboxrd --stdin +kw:seen +L:qp)],
undef, { %$lei_opt, 0 => \$in };
$check_kw->(['seen'], L => ['qp'], mid => $mid,
args => [ '--only', "$ro_home/t2" ],
@@ -78,7 +78,7 @@ test_lei(sub {
lei_ok(qw(ls-label));
is($lei_out, "nope\nqp\nurgent\n", 'ls-label shows qp');
- lei_ok qw(mark -F eml t/utf8.eml +L:INBOX +L:x); diag $lei_err;
+ lei_ok qw(tag -F eml t/utf8.eml +L:INBOX +L:x); diag $lei_err;
lei_ok qw(q m:testmessage@example.com);
$check_kw->([qw(answered seen)], L => [qw(INBOX nope urgent x)]);
lei_ok(qw(ls-label));
^ permalink raw reply related [relevance 51%]
* [PATCH] lei: fix IMAP auth failure handling
@ 2021-03-30 22:29 62% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-30 22:29 UTC (permalink / raw)
To: meta
We must use the $ops hashref returned by lei->workers_start,
since it's modified to include extra handlers for auth failures
and whatnot.
Fixes: 954581b8e575966a ("lei: simplify PktOp callers")
---
lib/PublicInbox/LeiImport.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 2 +-
xt/lei-auth-fail.t | 15 +++++++--------
3 files changed, 9 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 227a2a21..dbf655b6 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -76,7 +76,7 @@ sub lei_import { # the main "lei import" method
my $ops = { '' => [ \&import_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{-wq_nr_workers} = $j // 1; # locked
- my ($op_c, undef) = $lei->workers_start($self, 'lei_import', $j, $ops);
+ (my $op_c, $ops) = $lei->workers_start($self, 'lei_import', $j, $ops);
$lei->{imp} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 56ac25fa..8b012b16 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -116,7 +116,7 @@ sub lei_tag { # the "lei tag" method
my $ops = { '' => [ \&tag_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- my ($op_c, undef) = $lei->workers_start($self, 'lei_tag', 1, $ops);
+ (my $op_c, $ops) = $lei->workers_start($self, 'lei_tag', 1, $ops);
$lei->{tag} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
diff --git a/xt/lei-auth-fail.t b/xt/lei-auth-fail.t
index 78f8466d..e352aab3 100644
--- a/xt/lei-auth-fail.t
+++ b/xt/lei-auth-fail.t
@@ -9,13 +9,12 @@ require_mods(qw(Mail::IMAPClient));
my $imap_fail = $ENV{TEST_LEI_IMAP_FAIL_URL} //
'imaps://AzureDiamond:Hunter2@public-inbox.org:994/INBOX';
test_lei(sub {
- ok(!lei(qw(convert -o mboxrd:/dev/stdout), $imap_fail),
- 'IMAP auth failure on convert');
- like($lei_err, qr!\bE:.*?imaps://.*?!sm, 'error shown');
- unlike($lei_err, qr!Hunter2!s, 'password not shown');
- is($lei_out, '', 'nothing output');
- ok(!lei(qw(import), $imap_fail), 'IMAP auth failure on import');
- like($lei_err, qr!\bE:.*?imaps://.*?!sm, 'error shown');
- unlike($lei_err, qr!Hunter2!s, 'password not shown');
+ for my $pfx ([qw(convert -o mboxrd:/dev/stdout)], ['import'],
+ [qw(tag +L:INBOX)]) {
+ ok(!lei(@$pfx, $imap_fail), "IMAP auth failure on @$pfx");
+ like($lei_err, qr!\bE:.*?imaps://.*?!sm, 'error shown');
+ unlike($lei_err, qr!Hunter2!s, 'password not shown');
+ is($lei_out, '', 'nothing output');
+ }
});
done_testing;
^ permalink raw reply related [relevance 62%]
* [PATCH 0/2] lei doc: mail formats
@ 2021-03-31 0:41 71% Eric Wong
2021-03-31 0:41 46% ` [PATCH 1/2] doc: add lei-mail-formats(5) manpage Eric Wong
2021-03-31 0:41 66% ` [PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-03-31 0:41 UTC (permalink / raw)
To: meta
Eric Wong (2):
doc: add lei-mail-formats(5) manpage
doc: lei-overview: favor Maildir for mutt examples
Documentation/lei-mail-formats.pod | 101 +++++++++++++++++++++++++++++
Documentation/lei-overview.pod | 11 ++--
MANIFEST | 1 +
Makefile.PL | 4 +-
lib/PublicInbox/Watch.pm | 2 +-
5 files changed, 112 insertions(+), 7 deletions(-)
create mode 100644 Documentation/lei-mail-formats.pod
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples
2021-03-31 0:41 71% [PATCH 0/2] lei doc: mail formats Eric Wong
2021-03-31 0:41 46% ` [PATCH 1/2] doc: add lei-mail-formats(5) manpage Eric Wong
@ 2021-03-31 0:41 66% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-03-31 0:41 UTC (permalink / raw)
To: meta
mboxes are generally horrible for interactive read-write use due
to locking. Describe our parallel behavior with mutt, since
writing mail can take a long while and being able to read
results as they're written is nice.
We'll also use a gzipped mboxrd for the import example, since
we can decompress gzipped mboxrds automatically, now.
---
Documentation/lei-overview.pod | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index c1f952c9..70dbf2b5 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -23,9 +23,9 @@ to memoize messages from remotes.
=over
-=item $ lei import mboxrd:t.mbox
+=item $ lei import mboxrd:t.mbox.gz
-Import the messages from an mbox into the local storage.
+Import the messages from a gzipped mboxrd into the local storage.
=item $ lei blob 59ec517f9
@@ -82,10 +82,11 @@ Search for messages whose subject includes "lei" and "skeleton".
Do the same, but also report unmatched messages that are in the same
thread as a matched message.
-=item $ lei q -t -o mboxcl2:t.mbox --mua=mutt s:lei s:skeleton
+=item $ lei q -t -o mdir --mua=mutt s:lei s:skeleton
-Write mboxcl2-formatted results to t.mbox and enter mutt to view the
-file by invoking C<mutt -f %f>.
+Write results to a Maildir at "mdir". Mutt will be invoked
+to open mfolder (C<mutt -f %f>) while results are being fetched
+and written.
=item $ lei q kw:flagged L:next
^ permalink raw reply related [relevance 66%]
* [PATCH 1/2] doc: add lei-mail-formats(5) manpage
2021-03-31 0:41 71% [PATCH 0/2] lei doc: mail formats Eric Wong
@ 2021-03-31 0:41 46% ` Eric Wong
2021-03-31 3:15 71% ` Kyle Meyer
2021-03-31 0:41 66% ` [PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-03-31 0:41 UTC (permalink / raw)
To: meta
While plenty of online documentation exists, it's good to have
a locally-available summary for users to look at offline.
Fix a URL in Watch.pm while we're at it, too.
---
Documentation/lei-mail-formats.pod | 101 +++++++++++++++++++++++++++++
MANIFEST | 1 +
Makefile.PL | 4 +-
lib/PublicInbox/Watch.pm | 2 +-
4 files changed, 106 insertions(+), 2 deletions(-)
create mode 100644 Documentation/lei-mail-formats.pod
diff --git a/Documentation/lei-mail-formats.pod b/Documentation/lei-mail-formats.pod
new file mode 100644
index 00000000..8accedb4
--- /dev/null
+++ b/Documentation/lei-mail-formats.pod
@@ -0,0 +1,101 @@
+=head1 NAME
+
+lei-mail-formats - description of mail formats supported by lei
+
+=head1 DESCRIPTION
+
+L<lei-q(1)> supports writing to several existing mail formats
+for interoperability with existing mail user agents (MUA);
+below is an overview of them to help users choose.
+
+=head1 Maildir
+
+The default output format when given a filesystem path, it supports
+parallel read-write access. Performance is acceptable for smaller
+directories, but degrades as mailboxes get larger. Speed and
+scalability are limited by kernel and filesystem performance
+due to the use of small files and large number of syscalls.
+
+See also: L<https://cr.yp.to/proto/maildir.html> and
+L<https://wiki2.dovecot.org/MailboxFormat/Maildir>
+
+=head1 Mbox family
+
+The mbox family consists of several incompatible formats.
+Locking for parallel access is supported, but may not be
+compatible across tools. With compression (e.g. L<gzip(1)>),
+they require the least amount of space while offering good
+read-only performance.
+
+Keyword updates (C<Status:> and/or C<X-Status:> headers)
+generally require rewriting the entire mbox.
+
+See also:
+L<https://www.loc.gov/preservation/digital/formats/fdd/fdd000383.shtml>,
+L<mbox(5)>
+
+=head2 mboxo
+
+The traditional BSD format. It quotes C<From > to C<E<gt>From >,
+but lines already beginning with C<E<gt>From > do not get quoted,
+thus automatic reversibility is not guaranteed. MUAs which favor
+L</mboxcl> or L</mboxcl2> may convert these automatically to their
+preferred format.
+
+Truncation is undetectable unless compressed with gzip or similar.
+
+=head2 mboxrd
+
+An evolution of L</mboxo>, but quotes C<From > lines prefixed
+with any number of C<E<gt>> characters and is thus fully
+reversible.
+
+This format is emitted by L<PublicInbox::WWW(3pm)> with gzip.
+It is supported by L<git-am(1)> since git 2.10.
+
+As with uncompressed L</mboxo>, uncompressed mboxrd are vulnerable
+to undetectable truncation.
+
+It gracefully degrades to being treated as L</mboxo> by MUAs
+unaware of the format as excessive C<E<gt>From > quoting is
+recognizable to humans.
+
+=head2 mboxcl
+
+L</mboxo> with a C<Content-Length:> header, C<From > lines
+remain quoted to retain readability with L</mboxo> and L</mboxrd> MUAs.
+However, it is easy to corrupt these files when using tools
+which are not aware of C<Content-Length:> and write out updates
+as L</mboxo>.
+
+L<mutt(1)> will convert L</mboxo> and L</mboxrd> to mboxcl upon opening.
+
+See also: L<https://www.jwz.org/doc/content-length.html>
+
+=head2 mboxcl2
+
+Like L</mboxcl>, but without C<From > any quoting. It is wholly
+incompatible with MUAs which only handle L</mboxo> and/or L</mboxrd>.
+This is format is generated by L<mutt(1)> when writing to a new
+mbox.
+
+=head1 MH
+
+Not yet supported, locking semantics (or lack thereof) appear to
+make it unsuitable for parallel access.
+
+=head1 IMAP
+
+Depending on the IMAP server software and configuration, IMAP
+servers may use any (or combination) of the aforementioned
+formats or a non-standard database backend.
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei(1)>, L<lei-q(1)>, L<lei-convert(1)>, L<lei-overview(7)>
diff --git a/MANIFEST b/MANIFEST
index f3cb0147..49d273fc 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -31,6 +31,7 @@ Documentation/lei-import.pod
Documentation/lei-init.pod
Documentation/lei-ls-external.pod
Documentation/lei-ls-label.pod
+Documentation/lei-mail-formats.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
Documentation/lei-q.pod
diff --git a/Makefile.PL b/Makefile.PL
index 27b49c53..feb89ec1 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -48,7 +48,9 @@ $v->{-m1} = [ map {
lei-forget-external lei-import lei-init lei-ls-external lei-ls-label
lei-tag lei-p2q lei-q)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
- public-inbox-v2-format public-inbox-extindex-format) ];
+ public-inbox-v2-format public-inbox-extindex-format
+ lei-mail-formats
+ ) ];
$v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
public-inbox-glossary) ];
$v->{-m8} = [ qw(public-inbox-daemon) ];
diff --git a/lib/PublicInbox/Watch.pm b/lib/PublicInbox/Watch.pm
index 4fbc9640..05956cbb 100644
--- a/lib/PublicInbox/Watch.pm
+++ b/lib/PublicInbox/Watch.pm
@@ -2,7 +2,7 @@
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
#
# ref: https://cr.yp.to/proto/maildir.html
-# httsp://wiki2.dovecot.org/MailboxFormat/Maildir
+# https://wiki2.dovecot.org/MailboxFormat/Maildir
package PublicInbox::Watch;
use strict;
use v5.10.1;
^ permalink raw reply related [relevance 46%]
* [PATCH] lei blob: "--mail" disables solver, use --include/only
@ 2021-03-31 1:53 54% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-31 1:53 UTC (permalink / raw)
To: meta
Assume a user specifying --mail doesn't want to spend cycles
reconstructing a blob from a code repo. Also, don't require
users to use add-external or a previous -I or --only to ready an
external for use with ale.git.
---
lib/PublicInbox/LeiBlob.pm | 18 +++++++++++++++---
t/solver_git.t | 21 +++++++++++++++------
2 files changed, 30 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 6195b368..ed0754a3 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -90,15 +90,25 @@ sub lei_blob {
$lei->start_pager if -t $lei->{1};
my $opt = $lei->{opt};
my $has_hints = grep(defined, @$opt{qw(oid-a path-a path-b)});
+ my $lxs;
# first, see if it's a blob returned by "lei q" JSON output:k
if ($opt->{mail} // ($has_hints ? 0 : 1)) {
+ if (grep(defined, @$opt{qw(include only)})) {
+ $lxs = $lei->lxs_prepare;
+ $lei->ale->refresh_externals($lxs);
+ }
my $rdr = { 1 => $lei->{1} };
- open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ if ($opt->{mail}) {
+ $rdr->{2} = $lei->{2};
+ } else {
+ open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ }
my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
'cat-file', 'blob', $blob ];
waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
return if $? == 0;
+ return $lei->child_error($?) if $opt->{mail};
}
# maybe it's a non-email (code) blob from a coderepo
@@ -108,7 +118,10 @@ sub lei_blob {
unshift(@$git_dirs, $cgd) if defined $cgd;
}
return $lei->fail('no --git-dir to try') unless @$git_dirs;
- my $lxs = $lei->lxs_prepare or return;
+ unless ($lxs) {
+ $lxs = $lei->lxs_prepare or return;
+ $lei->ale->refresh_externals($lxs);
+ }
if ($lxs->remotes) {
require PublicInbox::LeiRemote;
$lei->{curl} //= which('curl') or return
@@ -117,7 +130,6 @@ sub lei_blob {
}
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
- $lei->ale;
my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
{ '' => [ \&sol_done, $lei ] });
$lei->{sol} = $self;
diff --git a/t/solver_git.t b/t/solver_git.t
index 2d803d47..8acf907f 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -6,7 +6,8 @@ use v5.10.1;
use PublicInbox::TestCommon;
use Cwd qw(abs_path);
require_git(2.6);
-use Digest::SHA qw(sha1_hex);
+use PublicInbox::ContentHash qw(git_sha);
+use PublicInbox::Eml;
use PublicInbox::Spawn qw(popen_rd which);
require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
@@ -17,14 +18,15 @@ chomp $git_dir;
$git_dir = abs_path($git_dir);
use_ok "PublicInbox::$_" for (qw(Inbox V2Writable Git SolverGit WWW));
+my $patch2 = eml_load 't/solve/0002-rename-with-modifications.patch';
+my $patch2_oid = git_sha(1, $patch2)->hexdigest;
my ($tmpdir, $for_destroy) = tmpdir();
my $ibx = create_inbox 'v2', version => 2,
indexlevel => 'medium', sub {
my ($im) = @_;
$im->add(eml_load 't/solve/0001-simple-mod.patch') or BAIL_OUT;
- $im->add(eml_load 't/solve/0002-rename-with-modifications.patch') or
- BAIL_OUT;
+ $im->add($patch2) or BAIL_OUT;
};
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
@@ -32,8 +34,14 @@ my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
test_lei({tmpdir => $tmpdir}, sub {
+ lei_ok('blob', '--mail', $patch2_oid, '-I', $ibx->{inboxdir},
+ \'--mail works for existing oid');
+ is($lei_out, $patch2->as_string, 'blob matches');
+ ok(!lei('blob', '--mail', '69df7d5', '-I', $ibx->{inboxdir}),
+ "--mail won't run solver");
+
lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
- is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
+ is(git_sha(1, PublicInbox::Eml->new($lei_out))->hexdigest,
$expect, 'blob contents output');
my $prev = $lei_out;
lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
@@ -236,8 +244,9 @@ EOF
test_lei({tmpdir => "$tmpdir/ext"}, sub {
my $rurl = "$url/$name";
lei_ok(qw(blob --no-mail 69df7d5 -I), $rurl);
- is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
- $expect, 'blob contents output');
+ my $eml = PublicInbox::Eml->new($lei_out);
+ is(git_sha(1, $eml)->hexdigest, $expect,
+ 'blob contents output');
ok(!lei(qw(blob -I), $rurl, $non_existent),
'non-existent blob fails');
});
^ permalink raw reply related [relevance 54%]
* Re: [PATCH 1/2] doc: add lei-mail-formats(5) manpage
2021-03-31 0:41 46% ` [PATCH 1/2] doc: add lei-mail-formats(5) manpage Eric Wong
@ 2021-03-31 3:15 71% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-03-31 3:15 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> While plenty of online documentation exists, it's good to have
> a locally-available summary for users to look at offline.
That's great. Thanks!
(The other patch makes sense to me too.)
^ permalink raw reply [relevance 71%]
* [PATCH] script/lei: background ourselves on MUA/pager exec
@ 2021-03-31 23:29 62% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-03-31 23:29 UTC (permalink / raw)
To: meta
This ought to gives the MUA or pager exclusive access to the
controlling terminal. The downside is we can only exec the
pager or MUA once per invocation, but I can't imagine a valid
case for running those things multiple times, either.
Note: I'm no expert when it comes to terminal control matters,
but this allows Ctrl-Z-ed mutt instance to come back and is
a nice code reduction, as well.
---
script/lei | 37 +++++++++++++++----------------------
1 file changed, 15 insertions(+), 22 deletions(-)
diff --git a/script/lei b/script/lei
index cb605e2e..bea8dcde 100755
--- a/script/lei
+++ b/script/lei
@@ -14,36 +14,31 @@ my $send_cmd = PublicInbox::CmdIPC4->can('send_cmd4') // do {
PublicInbox::Spawn->can('send_cmd4');
};
-my %pids;
-my $sigchld = sub {
- my $flags = scalar(@_) ? POSIX::WNOHANG() : 0;
- for my $pid (keys %pids) {
- delete($pids{$pid}) if waitpid($pid, $flags) == $pid;
- }
-};
-
+my @orig_pid;
my $exec_cmd = sub {
my ($fds, $argc, @argv) = @_;
+ die "BUG: already exec-ed\n" if @orig_pid;
+ @orig_pid = ($$);
+ require POSIX; # WNOHANG
my @old = (*STDIN{IO}, *STDOUT{IO}, *STDERR{IO});
my @rdr;
for my $fd (@$fds) {
open(my $tmpfh, '+<&=', $fd) or die "open +<&=$fd: $!";
push @rdr, shift(@old), $tmpfh;
}
- require POSIX; # WNOHANG
- $SIG{CHLD} = $sigchld;
my $pid = fork // die "fork: $!";
if ($pid == 0) {
- my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
- while (my ($old_io, $tmpfh) = splice(@rdr, 0, 2)) {
- open $old_io, '+<&', $tmpfh or die "open +<&=: $!";
- }
- %ENV = (%ENV, %env);
- exec(@argv);
- warn "exec: @argv: $!\n";
- POSIX::_exit(1);
+ POSIX::setsid() > 0 or die "setsid: $!";
+ return; # continue $recv_cmd in background
+ }
+ my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
+ while (my ($old_io, $tmpfh) = splice(@rdr, 0, 2)) {
+ open $old_io, '+<&', $tmpfh or die "open +<&=: $!";
}
- $pids{$pid} = 1;
+ @ENV{keys %env} = values %env;
+ exec(@argv);
+ warn "exec: @argv: $!\n";
+ POSIX::_exit(1);
};
if ($send_cmd && eval {
@@ -100,18 +95,16 @@ Falling back to (slow) one-shot mode
if ($buf =~ /\Aexec (.+)\z/) {
$exec_cmd->(\@fds, split(/\0/, $1));
} elsif ($buf eq '-WINCH') {
- kill($buf, $$); # for MUA
+ kill($buf, @orig_pid); # for MUA
} elsif ($buf =~ /\Ax_it ([0-9]+)\z/) {
$x_it_code = $1 + 0;
last;
} elsif ($buf =~ /\Achild_error ([0-9]+)\z/) {
$x_it_code = $1 + 0;
} else {
- $sigchld->();
die $buf;
}
}
- $sigchld->();
if (my $sig = ($x_it_code & 127)) {
kill $sig, $$;
sleep(1) while 1;
^ permalink raw reply related [relevance 62%]
* [PATCH 0/2] lei sucks
@ 2021-04-01 9:32 71% Eric Wong
2021-04-01 9:32 55% ` [PATCH 2/2] lei sucks: sub-command to aid bug reporting Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-01 9:32 UTC (permalink / raw)
To: meta
Inspired by the mutt slogan.
Eric Wong (2):
build: generate PublicInbox.pm with $VERSION
lei sucks: sub-command to aid bug reporting
.gitignore | 1 +
MANIFEST | 2 ++
Makefile.PL | 9 +++--
lib/PublicInbox/LeiSucks.pm | 71 +++++++++++++++++++++++++++++++++++++
t/config.t | 7 ++--
t/lei.t | 1 +
version-gen.perl | 29 +++++++++++++++
7 files changed, 115 insertions(+), 5 deletions(-)
create mode 100644 lib/PublicInbox/LeiSucks.pm
create mode 100644 version-gen.perl
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei sucks: sub-command to aid bug reporting
2021-04-01 9:32 71% [PATCH 0/2] lei sucks Eric Wong
@ 2021-04-01 9:32 55% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-01 9:32 UTC (permalink / raw)
To: meta
It's a bit of an Easter egg, though it's not possible to hide those
in Free Software... Anyways, it doesn't cost us an entry in %CMD
of LEI.pm and anybody frustrated enough with lei just might type
"lei sucks" on the command-line :>
---
MANIFEST | 1 +
lib/PublicInbox/LeiSucks.pm | 71 +++++++++++++++++++++++++++++++++++++
t/lei.t | 1 +
3 files changed, 73 insertions(+)
create mode 100644 lib/PublicInbox/LeiSucks.pm
diff --git a/MANIFEST b/MANIFEST
index 5e3b4aec..64293bb6 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -203,6 +203,7 @@ lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
+lib/PublicInbox/LeiSucks.pm
lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
lib/PublicInbox/LeiXSearch.pm
diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
new file mode 100644
index 00000000..d364a856
--- /dev/null
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -0,0 +1,71 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Undocumented hidden command somebody might discover if they're
+# frustrated and need to report a bug. There's no manpage and
+# it won't show up in tab completions or help.
+package PublicInbox::LeiSucks;
+use strict;
+use v5.10.1;
+use Digest::SHA ();
+use Config;
+use POSIX ();
+use PublicInbox::Config;
+use PublicInbox::Search;
+
+sub lei_sucks {
+ my ($lei, @argv) = @_;
+ $lei->start_pager if -t $lei->{1};
+ my ($os, undef, $rel, undef, $mac)= POSIX::uname();
+ if ($mac eq 'x86_64' && $Config{ptrsize} == 4) {
+ $mac = $Config{cppsymbols} =~ /\b__ILP32__=1\b/ ? 'x32' : 'i386'
+ }
+ eval { require PublicInbox };
+ my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';
+ my $daemon = $lei->{oneshot} ? 'oneshot' : 'daemon';
+ my @out = ("lei $pi_ver mode=$daemon\n",
+ "perl $Config{version} / $os $rel / $mac ".
+ "ptrsize=$Config{ptrsize}\n");
+ chomp(my $gv = `git --version` || "git missing");
+ $gv =~ s/ version / /;
+ my $json = ref(PublicInbox::Config->json);
+ $json .= ' ' . eval('$'.$json.'::VERSION') if $json;
+ $json ||= '(no JSON)';
+ push @out, "$gv / $json\n";
+ if (eval { require PublicInbox::Over }) {
+ push @out, 'SQLite '.
+ (eval('$DBD::SQLite::sqlite_version') // '(undef)') .
+ ', DBI '.(eval('$DBI::VERSION') // '(undef)') .
+ ', DBD::SQLite '.
+ (eval('$DBD::SQLite::VERSION') // '(undef)')."\n";
+ } else {
+ push @out, "Unable to load DBI / DBD::SQLite: $@\n";
+ }
+ if (PublicInbox::Search::load_xapian()) {
+ push @out, 'Xapian '.
+ join('.', map {
+ $PublicInbox::Search::Xap->can($_)->();
+ } qw(major_version minor_version revision)) .
+ ", bindings: $PublicInbox::Search::Xap";
+ my $xs_ver = eval '$'."$PublicInbox::Search::Xap".'::VERSION';
+ push @out, $xs_ver ? " $xs_ver\n" : " SWIG\n";
+ } else {
+ push @out, "Xapian not available: $@\n";
+ }
+ my $dig = Digest::SHA->new(1);
+ push @out, "public-inbox blob OIDs of loaded features:\n";
+ for my $m (grep(m{^PublicInbox/}, sort keys %INC)) {
+ my $f = $INC{$m};
+ $dig->add('blob '.(-s $f)."\0");
+ $dig->addfile($f);
+ push @out, ' '.$dig->hexdigest.' '.$m."\n";
+ }
+ push @out, <<'EOM';
+Let us know how it sucks! Please include the above and any other
+relevant information when sending plain-text mail to us at:
+meta@public-inbox.org -- archives: https://public-inbox.org/meta/
+EOM
+ $lei->out(@out);
+}
+
+1;
diff --git a/t/lei.t b/t/lei.t
index 0cf97866..2be9b4e8 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -154,6 +154,7 @@ my $test_fail = sub {
"error noted with q $fl");
}
}
+ lei_ok('sucks', \'yes, but hopefully less every day');
SKIP: {
skip 'no curl', 3 unless which('curl');
lei(qw(q --only http://127.0.0.1:99999/bogus/ t:m));
^ permalink raw reply related [relevance 55%]
* [PATCH] lei: maildir: handle "forwarded" keyword as "P"
@ 2021-04-01 10:12 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-01 10:12 UTC (permalink / raw)
To: meta
mbox and IMAP seem to have no way of describing this keyword.
but Maildir does with the "P" flagged (for "passed").
---
lib/PublicInbox/LeiToMail.pm | 3 ++-
lib/PublicInbox/MdirReader.pm | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 4c33c752..da633da4 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -27,8 +27,9 @@ $PublicInbox::GitAsyncCat::GCF2C = 0;
my %kw2char = ( # Maildir characters
draft => 'D',
flagged => 'F',
+ forwarded => 'P', # passed
answered => 'R',
- seen => 'S'
+ seen => 'S',
);
my %kw2status = (
diff --git a/lib/PublicInbox/MdirReader.pm b/lib/PublicInbox/MdirReader.pm
index 30e6f8ad..1685e4d8 100644
--- a/lib/PublicInbox/MdirReader.pm
+++ b/lib/PublicInbox/MdirReader.pm
@@ -37,7 +37,8 @@ sub maildir_each_file ($$;@) {
}
}
-my %c2kw = ('D' => 'draft', F => 'flagged', R => 'answered', S => 'seen');
+my %c2kw = ('D' => 'draft', F => 'flagged', P => 'forwarded',
+ R => 'answered', S => 'seen');
sub maildir_each_eml ($$;@) {
my ($dir, $cb, @arg) = @_;
^ permalink raw reply related [relevance 71%]
* [PATCH 2/5] lei q: reduce lei/store work for kw changes to stored mail
@ 2021-04-01 12:10 84% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-01 12:10 UTC (permalink / raw)
To: meta
We can tweak lse->kw_changed to return docids and reduce IPC
traffic and reduce work the lei/store worker needs to do.
---
lib/PublicInbox/LeiSearch.pm | 9 +++++----
lib/PublicInbox/LeiStore.pm | 8 ++++----
lib/PublicInbox/LeiToMail.pm | 6 +++---
3 files changed, 12 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 07d570ec..69ba8303 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -100,10 +100,11 @@ sub xoids_for {
# returns true if $eml is indexed by lei/store and keywords don't match
sub kw_changed {
- my ($self, $eml, $new_kw_sorted) = @_;
- my $xoids = xoids_for($self, $eml, 1) // return;
- my ($num) = values %$xoids;
- my @cur_kw = msg_keywords($self, $num);
+ my ($self, $eml, $new_kw_sorted, $docids) = @_;
+ my $xoids = xoids_for($self, $eml) // return;
+ $docids //= [];
+ @$docids = sort { $a <=> $b } values %$xoids;
+ my @cur_kw = msg_keywords($self, $docids->[0]);
join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
}
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index b76af4d3..48ab1d76 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -131,13 +131,13 @@ sub _docids_for ($$) {
}
sub set_eml_vmd {
- my ($self, $eml, $vmd) = @_;
+ my ($self, $eml, $vmd, $docids) = @_;
my $eidx = eidx_init($self);
- my @docids = _docids_for($self, $eml);
- for my $docid (@docids) {
+ $docids //= [ _docids_for($self, $eml) ];
+ for my $docid (@$docids) {
$eidx->idx_shard($docid)->ipc_do('set_vmd', $docid, $vmd);
}
- \@docids;
+ $docids;
}
sub add_eml_vmd {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index da633da4..0364d8ef 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -228,10 +228,10 @@ sub _mbox_write_cb ($$) {
sub update_kw_maybe ($$$$) {
my ($lei, $lse, $eml, $kw) = @_;
return unless $lse;
- my $lse_oids = $lse->kw_changed($eml, $kw);
+ my $c = $lse->kw_changed($eml, $kw, my $docids = []);
my $vmd = { kw => $kw };
- if ($lse_oids) { # already in lei/store
- $lei->{sto}->ipc_do('set_eml', $eml, $vmd);
+ if (scalar @$docids) { # already in lei/store
+ $lei->{sto}->ipc_do('set_eml_vmd', undef, $vmd, $docids) if $c;
} elsif (my $xoids = $lei->{ale}->xoids_for($eml)) {
# it's in an external, only set kw, here
$lei->{sto}->ipc_do('set_xvmd', $xoids, $eml, $vmd);
^ permalink raw reply related [relevance 84%]
* [PATCH] lei: fix git-credential handling
@ 2021-04-02 9:42 57% Eric Wong
2021-04-05 4:17 71% ` Kyle Meyer
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-02 9:42 UTC (permalink / raw)
To: meta
I completely forgot about git-credential prompting when
making lei background the client process for MUA.
Now it backgrounds itself only for the MUA when no FDs are
passed, since the MUA is the final command run. Otherwise, it
relies on FD passing as before.
Fixes: c790a75439f3a1db ("script/lei: background ourselves on MUA/pager exec")
---
lib/PublicInbox/LEI.pm | 8 +++++++-
script/lei | 46 ++++++++++++++++++++++++++++--------------
2 files changed, 38 insertions(+), 16 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 69d48bd1..f9361c68 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -839,7 +839,13 @@ sub start_mua {
if (my $sock = $self->{sock}) { # lei(1) client process runs it
send($sock, exec_buf(\@cmd, {}), MSG_EOR);
} elsif ($self->{oneshot}) {
- $self->{"pid.$self.$$"}->{spawn(\@cmd)} = \@cmd;
+ my $pid = fork // die "fork: $!";
+ if ($pid > 0) { # original process
+ exec(@cmd);
+ warn "exec @cmd: $!\n";
+ POSIX::_exit(1);
+ }
+ POSIX::setsid() > 0 or die "setsid: $!";
}
if ($self->{lxs} && $self->{au_done}) { # kick wait_startq
syswrite($self->{au_done}, 'q' x ($self->{lxs}->{jobs} // 0));
diff --git a/script/lei b/script/lei
index bea8dcde..78a7dab9 100755
--- a/script/lei
+++ b/script/lei
@@ -14,31 +14,45 @@ my $send_cmd = PublicInbox::CmdIPC4->can('send_cmd4') // do {
PublicInbox::Spawn->can('send_cmd4');
};
-my @orig_pid;
+my %pids;
+my $sigchld = sub {
+ my $flags = scalar(@_) ? POSIX::WNOHANG() : 0;
+ for my $pid (keys %pids) {
+ delete($pids{$pid}) if waitpid($pid, $flags) == $pid;
+ }
+};
+my @parent;
my $exec_cmd = sub {
my ($fds, $argc, @argv) = @_;
- die "BUG: already exec-ed\n" if @orig_pid;
- @orig_pid = ($$);
- require POSIX; # WNOHANG
+ my $parent = $$;
+ require POSIX;
my @old = (*STDIN{IO}, *STDOUT{IO}, *STDERR{IO});
my @rdr;
for my $fd (@$fds) {
- open(my $tmpfh, '+<&=', $fd) or die "open +<&=$fd: $!";
- push @rdr, shift(@old), $tmpfh;
+ open(my $newfh, '+<&=', $fd) or die "open +<&=$fd: $!";
+ push @rdr, shift(@old), $newfh;
}
+ my $do_exec = sub {
+ my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
+ @ENV{keys %env} = values %env;
+ exec(@argv);
+ warn "exec: @argv: $!\n";
+ POSIX::_exit(1);
+ };
+ $SIG{CHLD} = $sigchld;
my $pid = fork // die "fork: $!";
if ($pid == 0) {
+ while (my ($io, $newfh) = splice(@rdr, 0, 2)) {
+ open $io, '+<&', $newfh or die "open +<&=: $!";
+ }
+ $do_exec->() if scalar(@$fds); # git-credential, pager
+
+ # parent backgrounds on MUA
POSIX::setsid() > 0 or die "setsid: $!";
+ @parent = ($parent);
return; # continue $recv_cmd in background
}
- my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
- while (my ($old_io, $tmpfh) = splice(@rdr, 0, 2)) {
- open $old_io, '+<&', $tmpfh or die "open +<&=: $!";
- }
- @ENV{keys %env} = values %env;
- exec(@argv);
- warn "exec: @argv: $!\n";
- POSIX::_exit(1);
+ $do_exec->() if !scalar(@$fds); # MUA reuses all FDs
};
if ($send_cmd && eval {
@@ -95,16 +109,18 @@ Falling back to (slow) one-shot mode
if ($buf =~ /\Aexec (.+)\z/) {
$exec_cmd->(\@fds, split(/\0/, $1));
} elsif ($buf eq '-WINCH') {
- kill($buf, @orig_pid); # for MUA
+ kill($buf, @parent); # for MUA
} elsif ($buf =~ /\Ax_it ([0-9]+)\z/) {
$x_it_code = $1 + 0;
last;
} elsif ($buf =~ /\Achild_error ([0-9]+)\z/) {
$x_it_code = $1 + 0;
} else {
+ $sigchld->();
die $buf;
}
}
+ $sigchld->();
if (my $sig = ($x_it_code & 127)) {
kill $sig, $$;
sleep(1) while 1;
^ permalink raw reply related [relevance 57%]
* [PATCH 0/2] lei MUA UX fixes
@ 2021-04-03 1:37 71% Eric Wong
2021-04-03 1:37 71% ` [PATCH 1/2] lei q: don't show remote progress if MUA is running Eric Wong
2021-04-03 1:37 71% ` [PATCH 2/2] lei: allow progress to non-TTY after MUA spawn Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-03 1:37 UTC (permalink / raw)
To: meta
More stuff around auth coming...
Eric Wong (2):
lei q: don't show remote progress if MUA is running
lei: allow progress to non-TTY after MUA spawn
lib/PublicInbox/LEI.pm | 1 +
lib/PublicInbox/LeiXSearch.pm | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei q: don't show remote progress if MUA is running
2021-04-03 1:37 71% [PATCH 0/2] lei MUA UX fixes Eric Wong
@ 2021-04-03 1:37 71% ` Eric Wong
2021-04-03 1:37 71% ` [PATCH 2/2] lei: allow progress to non-TTY after MUA spawn Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 1:37 UTC (permalink / raw)
To: meta
Remote results can safely use the same mset progress reporting
as local results, despite not knowing the size of the result
set. We're assuming terminal MUAs, for now.
---
lib/PublicInbox/LeiXSearch.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index f3b8cc25..2b23e8e9 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -232,7 +232,7 @@ sub each_remote_eml { # callback for MboxReader->mboxrd
if ($now > $next) {
$lei->{-next_progress} = $now + 1;
my $nr = $lei->{-nr_remote_eml};
- $lei->err("# $lei->{-current_url} $nr/?");
+ mset_progress($lei, $lei->{-current_url}, $nr, '?');
}
}
$each_smsg->($smsg, undef, $eml);
^ permalink raw reply related [relevance 71%]
* [PATCH 2/2] lei: allow progress to non-TTY after MUA spawn
2021-04-03 1:37 71% [PATCH 0/2] lei MUA UX fixes Eric Wong
2021-04-03 1:37 71% ` [PATCH 1/2] lei q: don't show remote progress if MUA is running Eric Wong
@ 2021-04-03 1:37 71% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 1:37 UTC (permalink / raw)
To: meta
Sometimes I want to save debug info to a file or pipe even when
spawning an MUA.
---
lib/PublicInbox/LEI.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f9361c68..96a27102 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -850,6 +850,7 @@ sub start_mua {
if ($self->{lxs} && $self->{au_done}) { # kick wait_startq
syswrite($self->{au_done}, 'q' x ($self->{lxs}->{jobs} // 0));
}
+ return unless -t $self->{2}; # XXX how to determine non-TUI MUAs?
$self->{opt}->{quiet} = 1;
delete $self->{-progress};
delete $self->{opt}->{verbose};
^ permalink raw reply related [relevance 71%]
* [PATCH 0/6] lei auth-related fixes
@ 2021-04-03 2:24 71% Eric Wong
2021-04-03 2:24 57% ` [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
Auth failure handling should be more complete, but it's still
horrifically slow to test because the dovecot instance I run has
rate-limits for auth failures.
Perhaps public-inbox-imapd should learn to reject certain
username + password combos, since the -imapd instance on
public-inbox.org sees a fair amount of automated traffic
looking to steal info off leaked credentials.
3/6 is me still learning to use APIs I invent :x
Eric Wong (6):
URInntps: add URI 5.08 release note
lei q: ensure wq workers shutdown on IMAP auth failures
lei tag: fix tagging of IMAP inputs
lei_auth: rename {net_merge} to {net_merge_continue}
net_reader: fix read-only "lei convert" auth failures
xt/lei-auth-fail: test more failure cases
lib/PublicInbox/LEI.pm | 24 ++++++++----------------
lib/PublicInbox/LeiAuth.pm | 17 ++++++++++-------
lib/PublicInbox/LeiTag.pm | 6 +++++-
lib/PublicInbox/NetReader.pm | 5 +++--
lib/PublicInbox/URInntps.pm | 1 +
t/lei-import-imap.t | 1 +
xt/lei-auth-fail.t | 11 +++++++----
7 files changed, 35 insertions(+), 30 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 3/6] lei tag: fix tagging of IMAP inputs
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
2021-04-03 2:24 57% ` [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures Eric Wong
@ 2021-04-03 2:24 68% ` Eric Wong
2021-04-03 2:24 71% ` [PATCH 5/6] net_reader: fix read-only "lei convert" auth failures Eric Wong
2021-04-03 2:24 69% ` [PATCH 6/6] xt/lei-auth-fail: test more failure cases Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
We need net_merge_all and to lock the number of worker jobs.
Parallel inputs are not supported, yet (is it needed?, I don't
expect this to be used for multiple files very often...).
---
lib/PublicInbox/LeiTag.pm | 6 +++++-
t/lei-import-imap.t | 1 +
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index d572a84a..06313139 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -112,7 +112,8 @@ sub lei_tag { # the "lei tag" method
my $ops = { '' => [ \&tag_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- (my $op_c, $ops) = $lei->workers_start($self, 'lei_tag', 1, $ops);
+ my $j = $self->{-wq_nr_workers} = 1; # locked for now
+ (my $op_c, $ops) = $lei->workers_start($self, 'lei_tag', $j, $ops);
$lei->{tag} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
@@ -175,4 +176,7 @@ sub _complete_tag {
} grep(/$re\Q$cur/, @all);
}
+no warnings 'once'; # the following works even when LeiAuth is lazy-loaded
+*net_merge_all = \&PublicInbox::LeiAuth::net_merge_all;
+
1;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index fd38037a..7e4d44b9 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -23,5 +23,6 @@ test_lei({ tmpdir => $tmpdir }, sub {
my %r;
for (@$out) { $r{ref($_)}++ }
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
+ lei_ok([qw(tag +kw:seen), "imap://$host_port/t.v2.0"], undef, undef);
});
done_testing;
^ permalink raw reply related [relevance 68%]
* [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
@ 2021-04-03 2:24 57% ` Eric Wong
2021-04-03 2:24 68% ` [PATCH 3/6] lei tag: fix tagging of IMAP inputs Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
Leaving workers running on after auth failures is bad and messy,
cleanup our process management to have consistent worker
teardowns. Improve error reporting, too, instead of letting
Mail::IMAPClient->exists fail due to undef.
---
lib/PublicInbox/LEI.pm | 24 ++++++++----------------
lib/PublicInbox/LeiAuth.pm | 15 +++++++++------
lib/PublicInbox/NetReader.pm | 2 +-
3 files changed, 18 insertions(+), 23 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f9361c68..cdb4b347 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -350,6 +350,11 @@ my %CONFIG_KEYS = (
my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol); # internal workers
+sub _drop_wq {
+ my ($self) = @_;
+ for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) { $wq->DESTROY }
+}
+
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
sub x_it ($$) {
my ($self, $code) = @_;
@@ -360,10 +365,7 @@ sub x_it ($$) {
send($s, "x_it $code", MSG_EOR);
} elsif ($self->{oneshot}) {
# don't want to end up using $? from child processes
- for my $f (@WQ_KEYS) {
- my $wq = delete $self->{$f} or next;
- $wq->DESTROY;
- }
+ _drop_wq($self);
# cleanup anything that has tempfiles or open file handles
%PATH2CFG = ();
delete @$self{qw(ovv dedupe sto cfg)};
@@ -392,11 +394,8 @@ sub qerr ($;@) { $_[0]->{opt}->{quiet} or err(shift, @_) }
sub fail_handler ($;$$) {
my ($lei, $code, $io) = @_;
- for my $f (@WQ_KEYS) {
- my $wq = delete $lei->{$f} or next;
- $wq->wq_wait_old(undef, $lei) if $wq->wq_kill_old; # lei-daemon
- }
close($io) if $io; # needed to avoid warnings on SIGPIPE
+ _drop_wq($lei);
x_it($lei, $code // (1 << 8));
}
@@ -983,14 +982,7 @@ sub accept_dispatch { # Listener {post_accept} callback
sub dclose {
my ($self) = @_;
delete $self->{-progress};
- for my $f (@WQ_KEYS) {
- my $wq = delete $self->{$f} or next;
- if ($wq->wq_kill) {
- $wq->wq_close(0, undef, $self);
- } elsif ($wq->wq_kill_old) {
- $wq->wq_wait_old(undef, $self);
- }
- }
+ _drop_wq($self);
close(delete $self->{1}) if $self->{1}; # may reap_compress
$self->close if $self->{-event_init_done}; # PublicInbox::DS::close
}
diff --git a/lib/PublicInbox/LeiAuth.pm b/lib/PublicInbox/LeiAuth.pm
index 927fe550..48deca93 100644
--- a/lib/PublicInbox/LeiAuth.pm
+++ b/lib/PublicInbox/LeiAuth.pm
@@ -13,12 +13,15 @@ sub do_auth_atfork { # used by IPC WQ workers
return if $wq->{-wq_worker_nr} != 0;
my $lei = $wq->{lei};
my $net = $lei->{net};
- my $mics = $net->imap_common_init($lei);
- my $nn = $net->nntp_common_init($lei);
- pkt_do($lei->{pkt_op_p}, 'net_merge', $net) or
- die "pkt_do net_merge: $!";
- $net->{mics_cached} = $mics if $mics;
- $net->{nn_cached} = $nn if $nn;
+ eval {
+ my $mics = $net->imap_common_init($lei);
+ my $nn = $net->nntp_common_init($lei);
+ pkt_do($lei->{pkt_op_p}, 'net_merge', $net) or
+ die "pkt_do net_merge: $!";
+ $net->{mics_cached} = $mics if $mics;
+ $net->{nn_cached} = $nn if $nn;
+ };
+ $lei->fail($@) if $@;
}
sub net_merge_done1 { # bump merge-count in top-level lei-daemon
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 6a52b479..c269d841 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -267,7 +267,7 @@ sub imap_common_init ($;$) {
$mics->{$sec} //= mic_for($self, "$sec/", $mic_args, $lei);
next unless $self->isa('PublicInbox::NetWriter');
my $dst = $uri->mailbox // next;
- my $mic = $mics->{$sec};
+ my $mic = $mics->{$sec} // die "Unable to continue\n";
next if $mic->exists($dst); # already exists
$mic->create($dst) or die "CREATE $dst failed <$uri>: $@";
}
^ permalink raw reply related [relevance 57%]
* [PATCH 5/6] net_reader: fix read-only "lei convert" auth failures
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
2021-04-03 2:24 57% ` [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures Eric Wong
2021-04-03 2:24 68% ` [PATCH 3/6] lei tag: fix tagging of IMAP inputs Eric Wong
@ 2021-04-03 2:24 71% ` Eric Wong
2021-04-03 2:24 69% ` [PATCH 6/6] xt/lei-auth-fail: test more failure cases Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
"convert" is actually a bit more complicated than "lei import"
since it may need auth for either input or output.
---
lib/PublicInbox/NetReader.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c269d841..821e5d7f 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -264,10 +264,11 @@ sub imap_common_init ($;$) {
my $mics = {}; # schema://authority => IMAPClient obj
for my $uri (@{$self->{imap_order}}) {
my $sec = uri_section($uri);
- $mics->{$sec} //= mic_for($self, "$sec/", $mic_args, $lei);
+ my $mic = $mics->{$sec} //=
+ mic_for($self, "$sec/", $mic_args, $lei) //
+ die "Unable to continue\n";
next unless $self->isa('PublicInbox::NetWriter');
my $dst = $uri->mailbox // next;
- my $mic = $mics->{$sec} // die "Unable to continue\n";
next if $mic->exists($dst); # already exists
$mic->create($dst) or die "CREATE $dst failed <$uri>: $@";
}
^ permalink raw reply related [relevance 71%]
* [PATCH 6/6] xt/lei-auth-fail: test more failure cases
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
` (2 preceding siblings ...)
2021-04-03 2:24 71% ` [PATCH 5/6] net_reader: fix read-only "lei convert" auth failures Eric Wong
@ 2021-04-03 2:24 69% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 2:24 UTC (permalink / raw)
To: meta
Because failures are often overlooked, unfortunately.
---
xt/lei-auth-fail.t | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/xt/lei-auth-fail.t b/xt/lei-auth-fail.t
index e352aab3..06cb8533 100644
--- a/xt/lei-auth-fail.t
+++ b/xt/lei-auth-fail.t
@@ -2,17 +2,20 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
-require_mods(qw(Mail::IMAPClient));
+require_mods(qw(Mail::IMAPClient lei));
# TODO: mock IMAP server which fails at authentication so we don't
# have to make external connections to test this:
my $imap_fail = $ENV{TEST_LEI_IMAP_FAIL_URL} //
'imaps://AzureDiamond:Hunter2@public-inbox.org:994/INBOX';
+my ($ro_home, $cfg_path) = setup_public_inboxes;
test_lei(sub {
- for my $pfx ([qw(convert -o mboxrd:/dev/stdout)], ['import'],
- [qw(tag +L:INBOX)]) {
+ for my $pfx ([qw(q z:0.. --only), "$ro_home/t1", '-o'],
+ [qw(convert -o mboxrd:/dev/stdout)],
+ [qw(convert t/utf8.eml -o), $imap_fail],
+ ['import'], [qw(tag +L:INBOX)]) {
ok(!lei(@$pfx, $imap_fail), "IMAP auth failure on @$pfx");
- like($lei_err, qr!\bE:.*?imaps://.*?!sm, 'error shown');
+ like($lei_err, qr!\bE:.*?imaps?://.*?!sm, 'error shown');
unlike($lei_err, qr!Hunter2!s, 'password not shown');
is($lei_out, '', 'nothing output');
}
^ permalink raw reply related [relevance 69%]
* [PATCH 0/5] lei/store: fix epoch roll, better errors
@ 2021-04-03 10:48 71% Eric Wong
2021-04-03 10:48 71% ` [PATCH 3/5] lei tag: note message mismatches on failure Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-04-03 10:48 UTC (permalink / raw)
To: meta
Some things I noticed importing my personal IMAP mailboxes
Eric Wong (5):
lei_store: update alternates on new epoch
test_common: lei_ok: improve diagnostics
lei tag: note message mismatches on failure
lei: improve handling of Message-ID-less draft messages
lei/store: (more) synchronous non-fatal error output
MANIFEST | 1 +
lib/PublicInbox/Import.pm | 6 +--
lib/PublicInbox/LeiSearch.pm | 2 +-
lib/PublicInbox/LeiStore.pm | 74 ++++++++++++++++++++++++++++------
lib/PublicInbox/LeiStoreErr.pm | 30 ++++++++++++++
lib/PublicInbox/LeiTag.pm | 3 +-
lib/PublicInbox/OverIdx.pm | 6 ++-
lib/PublicInbox/Smsg.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 3 +-
t/lei-import.t | 21 ++++++++++
10 files changed, 127 insertions(+), 21 deletions(-)
create mode 100644 lib/PublicInbox/LeiStoreErr.pm
^ permalink raw reply [relevance 71%]
* [PATCH 3/5] lei tag: note message mismatches on failure
2021-04-03 10:48 71% [PATCH 0/5] lei/store: fix epoch roll, better errors Eric Wong
@ 2021-04-03 10:48 71% ` Eric Wong
2021-04-03 10:48 55% ` [PATCH 4/5] lei: improve handling of Message-ID-less draft messages Eric Wong
2021-04-03 10:48 43% ` [PATCH 5/5] lei/store: (more) synchronous non-fatal error output Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 10:48 UTC (permalink / raw)
To: meta
Just exiting with a failure code and no error message is
confusing :x
---
lib/PublicInbox/LeiTag.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 06313139..c7a21c87 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -121,7 +121,8 @@ sub lei_tag { # the "lei tag" method
sub note_missing {
my ($self) = @_;
- $self->{lei}->child_error(1 << 8) if $self->{missing};
+ my $n = $self->{missing} or return;
+ $self->{lei}->child_error(1 << 8, "$n missed messages");
}
sub ipc_atfork_child {
^ permalink raw reply related [relevance 71%]
* [PATCH 4/5] lei: improve handling of Message-ID-less draft messages
2021-04-03 10:48 71% [PATCH 0/5] lei/store: fix epoch roll, better errors Eric Wong
2021-04-03 10:48 71% ` [PATCH 3/5] lei tag: note message mismatches on failure Eric Wong
@ 2021-04-03 10:48 55% ` Eric Wong
2021-04-03 10:48 43% ` [PATCH 5/5] lei/store: (more) synchronous non-fatal error output Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 10:48 UTC (permalink / raw)
To: meta
We need a stable fallback time for digest2mid in the presence
of messages without Received/Date headers. Furthermore, we
must avoid using uninitialized smsg->{mid} when parsing
References for draft replies.
---
lib/PublicInbox/Import.pm | 6 +++---
lib/PublicInbox/LeiSearch.pm | 2 +-
lib/PublicInbox/OverIdx.pm | 6 ++++--
lib/PublicInbox/Smsg.pm | 2 +-
t/lei-import.t | 21 +++++++++++++++++++++
5 files changed, 30 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index 34738279..46f57e27 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -510,8 +510,8 @@ sub atfork_child {
}
}
-sub digest2mid ($$) {
- my ($dig, $hdr) = @_;
+sub digest2mid ($$;$) {
+ my ($dig, $hdr, $fallback_time) = @_;
my $b64 = $dig->clone->b64digest;
# Make our own URLs nicer:
# See "Base 64 Encoding with URL and Filename Safe Alphabet" in RFC4648
@@ -520,7 +520,7 @@ sub digest2mid ($$) {
# Add a date prefix to prevent a leading '-' in case that trips
# up some tools (e.g. if a Message-ID were a expected as a
# command-line arg)
- my $dt = msg_datestamp($hdr);
+ my $dt = msg_datestamp($hdr, $fallback_time);
$dt = POSIX::strftime('%Y%m%d%H%M%S', gmtime($dt));
"$dt.$b64" . '@z';
}
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 69ba8303..148aa185 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -58,7 +58,7 @@ sub content_key ($) {
qw(Message-ID X-Alt-Message-ID Resent-Message-ID));
unless (@$mids) {
$eml->{-lei_fake_mid} = $mids->[0] =
- PublicInbox::Import::digest2mid($dig, $eml);
+ PublicInbox::Import::digest2mid($dig, $eml, 0);
}
($chash, $mids);
}
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index e1cd31b9..66dec099 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -264,8 +264,10 @@ sub add_overview {
$smsg->{lines} = $eml->body_raw =~ tr!\n!\n!;
my $mids = mids_for_index($eml);
my $refs = $smsg->parse_references($eml, $mids);
- $mids->[0] //= $smsg->{mid} //= $eml->{-lei_fake_mid};
- $smsg->{mid} //= '';
+ $mids->[0] //= do {
+ $smsg->{mid} //= '';
+ $eml->{-lei_fake_mid};
+ };
my $subj = $smsg->{subject};
my $xpath;
if ($subj ne '') {
diff --git a/lib/PublicInbox/Smsg.pm b/lib/PublicInbox/Smsg.pm
index b4cc2ecb..da8ce590 100644
--- a/lib/PublicInbox/Smsg.pm
+++ b/lib/PublicInbox/Smsg.pm
@@ -76,7 +76,7 @@ sub parse_references ($$$) {
return $refs if scalar(@$refs) == 0;
# prevent circular references here:
- my %seen = ( $smsg->{mid} => 1 );
+ my %seen = ( ($smsg->{mid} // '') => 1 );
my @keep;
foreach my $ref (@$refs) {
if (length($ref) > PublicInbox::MID::MAX_MID_SIZE) {
diff --git a/t/lei-import.t b/t/lei-import.t
index 99289748..9bb4e1fa 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -79,6 +79,27 @@ is($res->[1], undef, 'only one result');
is($res->[0]->{'m'}, 'k@y', 'got expected message');
is_deeply($res->[0]->{kw}, ['seen'], "`seen' keywords set");
+# no From, Sender, or Message-ID
+$eml_str = <<'EOM';
+Subject: draft message with no sender
+References: <y@y>
+
+No use for a name
+EOM
+lei_ok([qw(import -F eml -)], undef, { %$lei_opt, 0 => \$eml_str });
+lei_ok(['q', 's:draft message with no sender']);
+my $draft_a = json_utf8->decode($lei_out);
+ok(!exists $draft_a->[0]->{'m'}, 'no fake mid stored or exposed');
+lei_ok([qw(tag -F eml - +kw:draft)], undef, { %$lei_opt, 0 => \$eml_str });
+lei_ok(['q', 's:draft message with no sender']);
+my $draft_b = json_utf8->decode($lei_out);
+my $kw = delete $draft_b->[0]->{kw};
+is_deeply($kw, ['draft'], 'draft kw set');
+is_deeply($draft_a, $draft_b, 'fake Message-ID lookup') or
+ diag explain($draft_a, $draft_b);
+lei_ok('blob', '--mail', $draft_b->[0]->{blob});
+is($lei_out, $eml_str, 'draft retrieved by blob');
+
# see t/lei_to_mail.t for "import -F mbox*"
});
done_testing;
^ permalink raw reply related [relevance 55%]
* [PATCH 5/5] lei/store: (more) synchronous non-fatal error output
2021-04-03 10:48 71% [PATCH 0/5] lei/store: fix epoch roll, better errors Eric Wong
2021-04-03 10:48 71% ` [PATCH 3/5] lei tag: note message mismatches on failure Eric Wong
2021-04-03 10:48 55% ` [PATCH 4/5] lei: improve handling of Message-ID-less draft messages Eric Wong
@ 2021-04-03 10:48 43% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-03 10:48 UTC (permalink / raw)
To: meta
Since every command that writes to lei/store calls ->done
to commit its output, we can rely on that to return a
pathname for a readable file with errors in it.
Errors can still get crossed up if multiple lei commands
are writing to the store at once, but reduces the delay
in seeing them and ensures it won't get seen when somebody
is attempting to use shell completion.
---
MANIFEST | 1 +
lib/PublicInbox/LeiStore.pm | 69 ++++++++++++++++++++++++++++------
lib/PublicInbox/LeiStoreErr.pm | 30 +++++++++++++++
3 files changed, 89 insertions(+), 11 deletions(-)
create mode 100644 lib/PublicInbox/LeiStoreErr.pm
diff --git a/MANIFEST b/MANIFEST
index 64293bb6..b663c2a2 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -203,6 +203,7 @@ lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
+lib/PublicInbox/LeiStoreErr.pm
lib/PublicInbox/LeiSucks.pm
lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 87082638..094e1555 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -3,9 +3,14 @@
#
# Local storage (cache/memo) for lei(1), suitable for personal/private
# mail iff on encrypted device/FS. Based on v2, but only deduplicates
-# based on git OID.
+# git storage based on git OID (index deduplication is done in ContentHash)
#
# for xref3, the following are constant: $eidx_key = '.', $xnum = -1
+#
+# We rely on the synchronous IPC API for this in lei-daemon and
+# multiple lei clients to write to it at once. This allows the
+# lei/store IPC process to be decoupled from network latency in
+# lei WQ workers.
package PublicInbox::LeiStore;
use strict;
use v5.10.1;
@@ -19,7 +24,10 @@ use PublicInbox::ContentHash qw(content_hash);
use PublicInbox::MID qw(mids);
use PublicInbox::LeiSearch;
use PublicInbox::MDA;
+use PublicInbox::Spawn qw(spawn);
use List::Util qw(max);
+use File::Temp ();
+use POSIX ();
sub new {
my (undef, $dir, $opt) = @_;
@@ -102,18 +110,27 @@ sub search {
PublicInbox::LeiSearch->new($_[0]->{priv_eidx}->{topdir});
}
+# follows the stderr file
+sub _tail_err {
+ my ($self) = @_;
+ print { $self->{-err_wr} } readline($self->{-tmp_err});
+}
+
sub eidx_init {
my ($self) = @_;
my $eidx = $self->{priv_eidx};
+ my $tl = wantarray && $self->{-err_wr} ?
+ PublicInbox::OnDestroy->new($$, \&_tail_err, $self) :
+ undef;
$eidx->idx_init({-private => 1});
- $eidx;
+ wantarray ? ($eidx, $tl) : $eidx;
}
sub _docids_for ($$) {
my ($self, $eml) = @_;
my %docids;
+ my $eidx = $self->{priv_eidx};
my ($chash, $mids) = PublicInbox::LeiSearch::content_key($eml);
- my $eidx = eidx_init($self);
my $oidx = $eidx->{oidx};
my $im = $self->{im};
for my $mid (@$mids) {
@@ -137,7 +154,7 @@ sub _docids_for ($$) {
sub set_eml_vmd {
my ($self, $eml, $vmd, $docids) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
$docids //= [ _docids_for($self, $eml) ];
for my $docid (@$docids) {
$eidx->idx_shard($docid)->ipc_do('set_vmd', $docid, $vmd);
@@ -147,7 +164,7 @@ sub set_eml_vmd {
sub add_eml_vmd {
my ($self, $eml, $vmd) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
my @docids = _docids_for($self, $eml);
for my $docid (@docids) {
$eidx->idx_shard($docid)->ipc_do('add_vmd', $docid, $vmd);
@@ -157,7 +174,7 @@ sub add_eml_vmd {
sub remove_eml_vmd {
my ($self, $eml, $vmd) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
my @docids = _docids_for($self, $eml);
for my $docid (@docids) {
$eidx->idx_shard($docid)->ipc_do('remove_vmd', $docid, $vmd);
@@ -168,7 +185,7 @@ sub remove_eml_vmd {
sub add_eml {
my ($self, $eml, $vmd, $xoids) = @_;
my $im = $self->importer; # may create new epoch
- my $eidx = eidx_init($self); # writes ALL.git/objects/info/alternates
+ my ($eidx, $tl) = eidx_init($self); # updates/writes alternates file
my $oidx = $eidx->{oidx}; # PublicInbox::Import::add checks this
my $smsg = bless { -oidx => $oidx }, 'PublicInbox::Smsg';
$im->add($eml, undef, $smsg) or return; # duplicate returns undef
@@ -257,7 +274,7 @@ sub _external_only ($$$) {
sub update_xvmd {
my ($self, $xoids, $eml, $vmd_mod) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
my $oidx = $eidx->{oidx};
my %seen;
for my $oid (keys %$xoids) {
@@ -294,7 +311,7 @@ sub update_xvmd {
sub set_xvmd {
my ($self, $xoids, $eml, $vmd) = @_;
- my $eidx = eidx_init($self);
+ my ($eidx, $tl) = eidx_init($self);
my $oidx = $eidx->{oidx};
my %seen;
@@ -329,6 +346,21 @@ sub checkpoint {
$self->{priv_eidx}->checkpoint($wait);
}
+sub xchg_stderr {
+ my ($self) = @_;
+ _tail_err($self) if $self->{-err_wr};
+ my $dir = $self->{priv_eidx}->{topdir};
+ return unless -e $dir;
+ my $old = delete $self->{-tmp_err};
+ my $pfx = POSIX::strftime('%Y%m%d%H%M%S', gmtime(time));
+ my $err = File::Temp->new(TEMPLATE => "$pfx.$$.lei_storeXXXX",
+ SUFFIX => '.err', DIR => $dir);
+ open STDERR, '>>', $err->filename or die "dup2: $!";
+ STDERR->autoflush(1); # shared with shard subprocesses
+ $self->{-tmp_err} = $err; # separate file description for RO access
+ undef;
+}
+
sub done {
my ($self) = @_;
my $err = '';
@@ -339,7 +371,8 @@ sub done {
warn $err;
}
}
- $self->{priv_eidx}->done;
+ $self->{priv_eidx}->done; # V2Writable::done
+ xchg_stderr($self);
die $err if $err;
}
@@ -347,6 +380,11 @@ sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
$lei->_lei_atfork_child(1) if $lei;
+ xchg_stderr($self);
+ if (my $err = delete($self->{err_pipe})) {
+ close $err->[0];
+ $self->{-err_wr} = $err->[1];
+ }
$SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
}
@@ -357,11 +395,20 @@ sub write_prepare {
my $d = $lei->store_path;
$self->ipc_lock_init("$d/ipc.lock");
substr($d, -length('/lei/store'), 10, '');
+ my $err_pipe;
+ unless ($lei->{oneshot}) {
+ pipe(my ($r, $w)) or die "pipe: $!";
+ $err_pipe = [ $r, $w ];
+ }
# Mail we import into lei are private, so headers filtered out
# by -mda for public mail are not appropriate
local @PublicInbox::MDA::BAD_HEADERS = ();
$self->ipc_worker_spawn("lei/store $d", $lei->oldset,
- { lei => $lei });
+ { lei => $lei, err_pipe => $err_pipe });
+ if ($err_pipe) {
+ require PublicInbox::LeiStoreErr;
+ PublicInbox::LeiStoreErr->new($err_pipe->[0], $lei);
+ }
}
$lei->{sto} = $self;
}
diff --git a/lib/PublicInbox/LeiStoreErr.pm b/lib/PublicInbox/LeiStoreErr.pm
new file mode 100644
index 00000000..68ce96d6
--- /dev/null
+++ b/lib/PublicInbox/LeiStoreErr.pm
@@ -0,0 +1,30 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# forwards stderr from lei/store process to any lei clients using
+# the same store
+package PublicInbox::LeiStoreErr;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::DS);
+use PublicInbox::Syscall qw(EPOLLIN EPOLLONESHOT);
+
+sub new {
+ my ($cls, $rd, $lei) = @_;
+ my $self = bless { sock => $rd, store_path => $lei->store_path }, $cls;
+ $self->SUPER::new($rd, EPOLLIN | EPOLLONESHOT);
+}
+
+sub event_step {
+ my ($self) = @_;
+ $self->do_read(\(my $rbuf), 4096) or return;
+ my $cb;
+ for my $lei (values %PublicInbox::DS::DescriptorMap) {
+ $cb = $lei->can('store_path') // next;
+ next if $cb->($lei) ne $self->{store_path};
+ my $err = $lei->{2} // next;
+ print $err $rbuf;
+ }
+}
+
+1;
^ permalink raw reply related [relevance 43%]
* Re: [PATCH] lei: fix git-credential handling
2021-04-02 9:42 57% [PATCH] lei: fix git-credential handling Eric Wong
@ 2021-04-05 4:17 71% ` Kyle Meyer
2021-04-05 8:36 71% ` [PATCH] script/lei: waitpid for git-credential and pager Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-04-05 4:17 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> I completely forgot about git-credential prompting when
> making lei background the client process for MUA.
>
> Now it backgrounds itself only for the MUA when no FDs are
> passed, since the MUA is the final command run. Otherwise, it
> relies on FD passing as before.
This seems to break the pager functionality. For example, if I do
lei q -I https://public-inbox.org/meta/ s:blob
I end up in a misbehaving pager (/bin/less, on my system). Navigation
keys like 'j' and 'k' are instead passed through, and enter returns to
the prompt.
^ permalink raw reply [relevance 71%]
* [PATCH] script/lei: waitpid for git-credential and pager
2021-04-05 4:17 71% ` Kyle Meyer
@ 2021-04-05 8:36 71% ` Eric Wong
2021-04-05 8:59 71% ` [RFC] script/lei: don't setsid on MUA spawn Eric Wong
2021-04-05 21:26 71% ` [PATCH] script/lei: waitpid for git-credential and pager Kyle Meyer
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-05 8:36 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> This seems to break the pager functionality. For example, if I do
>
> lei q -I https://public-inbox.org/meta/ s:blob
>
> I end up in a misbehaving pager (/bin/less, on my system). Navigation
> keys like 'j' and 'k' are instead passed through, and enter returns to
> the prompt.
Odd, I'm not getting the exact same behavior, but I noticed
it's not getting reaped. Does this fix things for you?
-----8<-----
Subject: [PATCH] script/lei: waitpid for git-credential and pager
We need to ensure we reap things we spawn.
---
script/lei | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/script/lei b/script/lei
index 78a7dab9..76217ab9 100755
--- a/script/lei
+++ b/script/lei
@@ -52,7 +52,11 @@ my $exec_cmd = sub {
@parent = ($parent);
return; # continue $recv_cmd in background
}
- $do_exec->() if !scalar(@$fds); # MUA reuses all FDs
+ if (scalar(@$fds)) {
+ $pids{$pid} = undef;
+ } else {
+ $do_exec->(); # MUA reuses all FDs
+ }
};
if ($send_cmd && eval {
^ permalink raw reply related [relevance 71%]
* [RFC] script/lei: don't setsid on MUA spawn
2021-04-05 8:36 71% ` [PATCH] script/lei: waitpid for git-credential and pager Eric Wong
@ 2021-04-05 8:59 71% ` Eric Wong
2021-04-05 21:26 71% ` [PATCH] script/lei: waitpid for git-credential and pager Kyle Meyer
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-05 8:59 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Btw, the setsid call doesn't seem necessary, either, the MUA
seems to work fine without it...
------8<------
Subject: [PATCH] script/lei: don't setsid on MUA spawn
---
script/lei | 1 -
1 file changed, 1 deletion(-)
diff --git a/script/lei b/script/lei
index 76217ab9..cd535afb 100755
--- a/script/lei
+++ b/script/lei
@@ -48,7 +48,6 @@ my $exec_cmd = sub {
$do_exec->() if scalar(@$fds); # git-credential, pager
# parent backgrounds on MUA
- POSIX::setsid() > 0 or die "setsid: $!";
@parent = ($parent);
return; # continue $recv_cmd in background
}
^ permalink raw reply related [relevance 71%]
* [PATCH 3/5] lei: maildir: move shard support to MdirReader
@ 2021-04-05 10:27 43% ` Eric Wong
2021-04-05 10:27 60% ` [PATCH 5/5] lei q: fix auth IMAP --output with remote mboxrd Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-05 10:27 UTC (permalink / raw)
To: meta
We'll eventually want lei_input users like "lei import" and
"lei tag" to support parallel reads.
---
lib/PublicInbox/InboxWritable.pm | 4 ++--
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 29 +++++++++--------------------
lib/PublicInbox/MdirReader.pm | 25 +++++++++++++++++++++----
t/lei-convert.t | 2 +-
t/lei_to_mail.t | 8 ++++----
6 files changed, 38 insertions(+), 32 deletions(-)
diff --git a/lib/PublicInbox/InboxWritable.pm b/lib/PublicInbox/InboxWritable.pm
index eeebc485..45d8cdc7 100644
--- a/lib/PublicInbox/InboxWritable.pm
+++ b/lib/PublicInbox/InboxWritable.pm
@@ -154,8 +154,8 @@ sub import_maildir {
my $im = $self->importer(1);
my @self = $self->filter($im) ? ($self) : ();
require PublicInbox::MdirReader;
- PublicInbox::MdirReader::maildir_each_file(\&_each_maildir_fn,
- $im, @self);
+ PublicInbox::MdirReader->new->maildir_each_file(\&_each_maildir_fn,
+ $im, @self);
$im->done;
}
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 40d71f9e..e416d3ed 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -88,7 +88,7 @@ sub input_path_url {
return $lei->fail(<<EOM) if $ifmt && $ifmt ne 'maildir';
$input appears to a be a maildir, not $ifmt
EOM
- PublicInbox::MdirReader::maildir_each_eml($input,
+ PublicInbox::MdirReader->new->maildir_each_eml($input,
$self->can('input_maildir_cb'),
$self, @args);
} else {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 76a11b0e..2e736070 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -14,7 +14,6 @@ use PublicInbox::PktOp qw(pkt_do);
use Symbol qw(gensym);
use IO::Handle; # ->autoflush
use Fcntl qw(SEEK_SET SEEK_END O_CREAT O_EXCL O_WRONLY);
-use Digest::SHA qw(sha256_hex);
my %kw2char = ( # Maildir characters
draft => 'D',
@@ -234,17 +233,9 @@ sub update_kw_maybe ($$$$) {
}
}
-sub _augment_or_unlink { # maildir_each_eml cb
- my ($f, $kw, $eml, $lei, $lse, $mod, $shard, $unlink) = @_;
- if ($mod) {
- # can't get dirent.d_ino w/ pure Perl readdir, so we extract
- # the OID if it looks like one instead of doing stat(2)
- my $hex = $f =~ m!\b([a-f0-9]{40,})[^/]*\z! ?
- $1 : sha256_hex($f);
- my $recno = hex(substr($hex, 0, 8));
- return if ($recno % $mod) != $shard;
- update_kw_maybe($lei, $lse, $eml, $kw);
- }
+sub _md_update { # maildir_each_eml cb
+ my ($f, $kw, $eml, $lei, $lse, $unlink) = @_;
+ update_kw_maybe($lei, $lse, $eml, $kw);
$unlink ? unlink($f) : _augment($eml, $lei);
}
@@ -392,21 +383,19 @@ sub _do_augment_maildir {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
- my ($mod, $shard) = @{$self->{shard_info} // []};
+ my $mdr = PublicInbox::MdirReader->new;
if ($lei->{opt}->{augment}) {
my $dedupe = $lei->{dedupe};
if ($dedupe && $dedupe->prepare_dedupe) {
- PublicInbox::MdirReader::maildir_each_eml($dst,
- \&_augment_or_unlink,
- $lei, $lse, $mod, $shard);
+ $mdr->{shard_info} = $self->{shard_info};
+ $mdr->maildir_each_eml($dst, \&_md_update, $lei, $lse);
$dedupe->pause_dedupe;
}
} elsif ($lse) {
- PublicInbox::MdirReader::maildir_each_eml($dst,
- \&_augment_or_unlink,
- $lei, $lse, $mod, $shard, 1);
+ $mdr->{shard_info} = $self->{shard_info};
+ $mdr->maildir_each_eml($dst, \&_md_update, $lei, $lse, 1);
} else {# clobber existing Maildir
- PublicInbox::MdirReader::maildir_each_file($dst, \&_unlink);
+ $mdr->maildir_each_file($dst, \&_unlink);
}
}
diff --git a/lib/PublicInbox/MdirReader.pm b/lib/PublicInbox/MdirReader.pm
index 1685e4d8..b49c8ceb 100644
--- a/lib/PublicInbox/MdirReader.pm
+++ b/lib/PublicInbox/MdirReader.pm
@@ -8,6 +8,7 @@ package PublicInbox::MdirReader;
use strict;
use v5.10.1;
use PublicInbox::InboxWritable qw(eml_from_path);
+use Digest::SHA qw(sha256_hex);
# returns Maildir flags from a basename ('' for no flags, undef for invalid)
sub maildir_basename_flags {
@@ -24,14 +25,25 @@ sub maildir_path_flags {
$i >= 0 ? maildir_basename_flags(substr($f, $i + 1)) : undef;
}
-sub maildir_each_file ($$;@) {
- my ($dir, $cb, @arg) = @_;
+sub shard_ok ($$$) {
+ my ($bn, $mod, $shard) = @_;
+ # can't get dirent.d_ino w/ pure Perl readdir, so we extract
+ # the OID if it looks like one instead of doing stat(2)
+ my $hex = $bn =~ m!\A([a-f0-9]{40,})! ? $1 : sha256_hex($bn);
+ my $recno = hex(substr($hex, 0, 8));
+ ($recno % $mod) == $shard;
+}
+
+sub maildir_each_file {
+ my ($self, $dir, $cb, @arg) = @_;
$dir .= '/' unless substr($dir, -1) eq '/';
+ my ($mod, $shard) = @{$self->{shard_info} // []};
for my $d (qw(new/ cur/)) {
my $pfx = $dir.$d;
opendir my $dh, $pfx or next;
while (defined(my $bn = readdir($dh))) {
maildir_basename_flags($bn) // next;
+ next if defined($mod) && !shard_ok($bn, $mod, $shard);
$cb->($pfx.$bn, @arg);
}
}
@@ -40,15 +52,17 @@ sub maildir_each_file ($$;@) {
my %c2kw = ('D' => 'draft', F => 'flagged', P => 'forwarded',
R => 'answered', S => 'seen');
-sub maildir_each_eml ($$;@) {
- my ($dir, $cb, @arg) = @_;
+sub maildir_each_eml {
+ my ($self, $dir, $cb, @arg) = @_;
$dir .= '/' unless substr($dir, -1) eq '/';
+ my ($mod, $shard) = @{$self->{shard_info} // []};
my $pfx = $dir . 'new/';
if (opendir(my $dh, $pfx)) {
while (defined(my $bn = readdir($dh))) {
next if substr($bn, 0, 1) eq '.';
my @f = split(/:/, $bn, -1);
next if scalar(@f) != 1;
+ next if defined($mod) && !shard_ok($bn, $mod, $shard);
my $f = $pfx.$bn;
my $eml = eml_from_path($f) or next;
$cb->($f, [], $eml, @arg);
@@ -59,6 +73,7 @@ sub maildir_each_eml ($$;@) {
while (defined(my $bn = readdir($dh))) {
my $fl = maildir_basename_flags($bn) // next;
next if index($fl, 'T') >= 0;
+ next if defined($mod) && !shard_ok($bn, $mod, $shard);
my $f = $pfx.$bn;
my $eml = eml_from_path($f) or next;
my @kw = sort(map { $c2kw{$_} // () } split(//, $fl));
@@ -66,4 +81,6 @@ sub maildir_each_eml ($$;@) {
}
}
+sub new { bless {}, __PACKAGE__ }
+
1;
diff --git a/t/lei-convert.t b/t/lei-convert.t
index dc53b82c..0ea860c8 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -57,7 +57,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok('convert', '-o', "$d/md", "mboxrd:$d/foo.mboxrd");
ok(-d "$d/md", 'Maildir created');
my @md;
- PublicInbox::MdirReader::maildir_each_eml("$d/md", sub {
+ PublicInbox::MdirReader->new->maildir_each_eml("$d/md", sub {
push @md, $_[2];
});
is(scalar(@md), scalar(@mboxrd), 'got expected emails in Maildir') or
diff --git a/t/lei_to_mail.t b/t/lei_to_mail.t
index 75314add..51357257 100644
--- a/t/lei_to_mail.t
+++ b/t/lei_to_mail.t
@@ -253,7 +253,7 @@ SKIP: { # FIFO support
}
{ # Maildir support
- my $each_file = PublicInbox::MdirReader->can('maildir_each_file');
+ my $mdr = PublicInbox::MdirReader->new;
my $md = "$tmpdir/maildir/";
my $wcb = $wcb_get->('maildir', $md);
is(ref($wcb), 'CODE', 'got Maildir callback');
@@ -261,7 +261,7 @@ SKIP: { # FIFO support
$wcb->(\(my $x = $buf), $b4dc0ffee);
my @f;
- $each_file->($md, sub { push @f, shift });
+ $mdr->maildir_each_file($md, sub { push @f, shift });
open my $fh, $f[0] or BAIL_OUT $!;
is(do { local $/; <$fh> }, $buf, 'wrote to Maildir');
@@ -270,7 +270,7 @@ SKIP: { # FIFO support
$wcb->(\($x = $buf."\nx\n"), $deadcafe);
my @x = ();
- $each_file->($md, sub { push @x, shift });
+ $mdr->maildir_each_file($md, sub { push @x, shift });
is(scalar(@x), 1, 'wrote one new file');
ok(!-f $f[0], 'old file clobbered');
open $fh, $x[0] or BAIL_OUT $!;
@@ -281,7 +281,7 @@ SKIP: { # FIFO support
$wcb->(\($x = $buf."\ny\n"), $deadcafe);
$wcb->(\($x = $buf."\ny\n"), $b4dc0ffee); # skipped by dedupe
@f = ();
- $each_file->($md, sub { push @f, shift });
+ $mdr->maildir_each_file($md, sub { push @f, shift });
is(scalar grep(/\A\Q$x[0]\E\z/, @f), 1, 'old file still there');
my @new = grep(!/\A\Q$x[0]\E\z/, @f);
is(scalar @new, 1, '1 new file written (b4dc0ffee skipped)');
^ permalink raw reply related [relevance 43%]
* [PATCH 5/5] lei q: fix auth IMAP --output with remote mboxrd
2021-04-05 10:27 43% ` [PATCH 3/5] lei: maildir: move shard support to MdirReader Eric Wong
@ 2021-04-05 10:27 60% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-05 10:27 UTC (permalink / raw)
To: meta
IMAP authentication info is only shared amongst lei2mail workers,
so we must ensure all IMAP writes go through lei2mail workers
even if we don't have to access the mail through git.
This allows us to decouple the latency of the remote mboxrd from
the latency of the IMAP --output at the expense of extra IPC
overhead within our own processes.
---
lib/PublicInbox/LeiOverview.pm | 14 ++++----------
lib/PublicInbox/LeiToMail.pm | 3 ++-
lib/PublicInbox/LeiXSearch.pm | 4 ++--
3 files changed, 8 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index cdd9ee04..bfb8b143 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -195,7 +195,7 @@ sub _json_pretty {
}
sub ovv_each_smsg_cb { # runs in wq worker usually
- my ($self, $lei, $ibxish) = @_;
+ my ($self, $lei) = @_;
my ($json, $dedupe);
if (my $pkg = $self->{json}) {
$json = $pkg->new;
@@ -208,17 +208,11 @@ sub ovv_each_smsg_cb { # runs in wq worker usually
$dedupe->prepare_dedupe;
}
$lei->{ovv_buf} = \(my $buf = '') if !$l2m;
- if ($l2m && !$ibxish) { # remote https?:// mboxrd
- my $wcb = $l2m->write_cb($lei);
- sub {
- my ($smsg, undef, $eml) = @_; # no mitem in $_[1]
- $wcb->(undef, $smsg, $eml);
- };
- } elsif ($l2m && $l2m->{-wq_s1}) {
+ if ($l2m) {
sub {
- my ($smsg, $mitem) = @_;
+ my ($smsg, $mitem, $eml) = @_;
$smsg->{pct} = get_pct($mitem) if $mitem;
- $l2m->wq_io_do('write_mail', [], $smsg);
+ $l2m->wq_io_do('write_mail', [], $smsg, $eml);
}
} elsif ($self->{fmt} =~ /\A(concat)?json\z/ && $lei->{opt}->{pretty}) {
my $EOR = ($1//'') eq 'concat' ? "\n}" : "\n},";
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 9411313b..70164e40 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -583,7 +583,8 @@ sub poke_dst {
}
sub write_mail { # via ->wq_io_do
- my ($self, $smsg) = @_;
+ my ($self, $smsg, $eml) = @_;
+ return $self->{wcb}->(undef, $smsg, $eml) if $eml;
$self->{lei}->{ale}->git->cat_async($smsg->{blob}, \&git_to_mail,
[$self->{wcb}, $smsg]);
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2b23e8e9..692d5e54 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -153,7 +153,7 @@ sub query_thread_mset { # for --threads
return warn("$desc not indexed by Xapian\n") unless ($srch && $over);
my $mo = { %{$lei->{mset_opt}} };
my $mset;
- my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $ibxish);
+ my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
my $can_kw = !!$ibxish->can('msg_keywords');
my $fl = $lei->{opt}->{threads} > 1 ? 1 : undef;
do {
@@ -196,7 +196,7 @@ sub query_mset { # non-parallel for non-"--threads" users
for my $loc (locals($self)) {
attach_external($self, $loc);
}
- my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei, $self);
+ my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
do {
$mset = $self->mset($mo->{qstr}, $mo);
mset_progress($lei, 'xsearch', $mset->size,
^ permalink raw reply related [relevance 60%]
* Re: [PATCH] script/lei: waitpid for git-credential and pager
2021-04-05 8:36 71% ` [PATCH] script/lei: waitpid for git-credential and pager Eric Wong
2021-04-05 8:59 71% ` [RFC] script/lei: don't setsid on MUA spawn Eric Wong
@ 2021-04-05 21:26 71% ` Kyle Meyer
1 sibling, 0 replies; 200+ results
From: Kyle Meyer @ 2021-04-05 21:26 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Odd, I'm not getting the exact same behavior, but I noticed
> it's not getting reaped. Does this fix things for you?
That seems to do the trick. Thank you!
^ permalink raw reply [relevance 71%]
* [PATCH] lei blob: quiet "git rev-parse --git-dir" stderr w/o --cwd
@ 2021-04-12 17:32 68% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-12 17:32 UTC (permalink / raw)
To: meta
This seemed to be causing occasional "make check-run" failures
with errors bleeding into other tests.
---
lib/PublicInbox/LeiBlob.pm | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index ed0754a3..ad885306 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -23,12 +23,18 @@ sub sol_done { # EOF callback for main daemon
$sol->wq_wait_old(\&sol_done_wait, $lei);
}
-sub get_git_dir ($) {
- my ($d) = @_;
+sub get_git_dir ($$) {
+ my ($lei, $d) = @_;
return $d if -d "$d/objects" && -d "$d/refs" && -e "$d/HEAD";
my $cmd = [ qw(git rev-parse --git-dir) ];
- my ($r, $pid) = popen_rd($cmd, {GIT_DIR => undef}, { '-C' => $d });
+ my $opt = { '-C' => $d };
+ if (defined($lei->{opt}->{cwd})) { # --cwd used, report errors
+ $opt->{2} = $lei->{2};
+ } else { # implicit --cwd, quiet errors
+ open $opt->{2}, '>', '/dev/null' or die "open /dev/null: $!";
+ }
+ my ($r, $pid) = popen_rd($cmd, {GIT_DIR => undef}, $opt);
chomp(my $gd = do { local $/; <$r> });
waitpid($pid, 0) == $pid or die "BUG: waitpid @$cmd ($!)";
$? == 0 ? $gd : undef;
@@ -114,7 +120,7 @@ sub lei_blob {
# maybe it's a non-email (code) blob from a coderepo
my $git_dirs = $opt->{'git-dir'} //= [];
if ($opt->{'cwd'} // 1) {
- my $cgd = get_git_dir('.');
+ my $cgd = get_git_dir($lei, '.');
unshift(@$git_dirs, $cgd) if defined $cgd;
}
return $lei->fail('no --git-dir to try') unless @$git_dirs;
^ permalink raw reply related [relevance 68%]
* [PATCH 0/5] "lei q --save" + "lei up"
@ 2021-04-13 10:54 90% Eric Wong
2021-04-13 10:54 32% ` [PATCH 4/5] lei q: start wiring up saved search Eric Wong
2021-04-13 10:54 75% ` [PATCH 5/5] lei: add "lei up" to complement "lei q --save" Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-13 10:54 UTC (permalink / raw)
To: meta
"--save" may become the default when writing to a
pathname or URL. "lei up" will be used to update
the results of "--save".
This only supports local externals at the moment,
remote externals won't be able to avoid excess
traffic easily.
Usability improvements are coming...
Eric Wong (5):
lei_xsearch: use per-external queries when not sorting
lei_dedupe: adjust to prepare for saved searches
lei_query: rearrange internals to capture query early
lei q: start wiring up saved search
lei: add "lei up" to complement "lei q --save"
MANIFEST | 4 +
lib/PublicInbox/LEI.pm | 6 +-
lib/PublicInbox/LeiDedupe.pm | 16 ++--
lib/PublicInbox/LeiQuery.pm | 59 +++++++------
lib/PublicInbox/LeiSavedSearch.pm | 142 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiToMail.pm | 18 ++--
lib/PublicInbox/LeiUp.pm | 46 ++++++++++
lib/PublicInbox/LeiXSearch.pm | 94 +++++++++++++-------
t/lei-q-save.t | 25 ++++++
t/lei.t | 2 +-
t/lei_dedupe.t | 11 ++-
t/lei_saved_search.t | 10 +++
12 files changed, 356 insertions(+), 77 deletions(-)
create mode 100644 lib/PublicInbox/LeiSavedSearch.pm
create mode 100644 lib/PublicInbox/LeiUp.pm
create mode 100644 t/lei-q-save.t
create mode 100644 t/lei_saved_search.t
^ permalink raw reply [relevance 90%]
* [PATCH 4/5] lei q: start wiring up saved search
2021-04-13 10:54 90% [PATCH 0/5] "lei q --save" + "lei up" Eric Wong
@ 2021-04-13 10:54 32% ` Eric Wong
2021-04-13 11:25 71% ` Eric Wong
2021-04-13 10:54 75% ` [PATCH 5/5] lei: add "lei up" to complement "lei q --save" Eric Wong
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-04-13 10:54 UTC (permalink / raw)
To: meta
This will have a over.sqlite3 for content-based deduplication.
It may exhibit ibxish methods, so serving a read-only (or even
R/W) IMAP or instance or displaying HTML isn't outside the realm
of possibility.
---
MANIFEST | 3 +
lib/PublicInbox/LEI.pm | 4 +-
lib/PublicInbox/LeiQuery.pm | 2 +
lib/PublicInbox/LeiSavedSearch.pm | 134 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiToMail.pm | 10 ++-
lib/PublicInbox/LeiXSearch.pm | 31 ++++++-
t/lei-q-save.t | 12 +++
t/lei.t | 2 +-
t/lei_saved_search.t | 10 +++
9 files changed, 199 insertions(+), 9 deletions(-)
create mode 100644 lib/PublicInbox/LeiSavedSearch.pm
create mode 100644 t/lei-q-save.t
create mode 100644 t/lei_saved_search.t
diff --git a/MANIFEST b/MANIFEST
index 12247ad2..20615abc 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -201,6 +201,7 @@ lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
lib/PublicInbox/LeiQuery.pm
lib/PublicInbox/LeiRemote.pm
+lib/PublicInbox/LeiSavedSearch.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
lib/PublicInbox/LeiStoreErr.pm
@@ -393,12 +394,14 @@ t/lei-mirror.t
t/lei-p2q.t
t/lei-q-kw.t
t/lei-q-remote-import.t
+t/lei-q-save.t
t/lei-q-thread.t
t/lei-tag.t
t/lei.t
t/lei_dedupe.t
t/lei_external.t
t/lei_overview.t
+t/lei_saved_search.t
t/lei_store.t
t/lei_to_mail.t
t/lei_xsearch.t
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 475af8f0..7292d0f2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -131,7 +131,7 @@ our %CMD = ( # sorted in order of importance/use:
'q' => [ '--stdin|SEARCH_TERMS...', 'search for messages matching terms',
'stdin|', # /|\z/ must be first for lone dash
@lxs_opt,
- qw(save-as=s output|mfolder|o=s format|f=s dedupe|d=s threads|t+
+ qw(save output|mfolder|o=s format|f=s dedupe|d=s threads|t+
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
@@ -249,7 +249,7 @@ my %OPTDESC = (
'torsocks=s' => ['VAL|auto|no|yes',
'whether or not to wrap git and curl commands with torsocks'],
'no-torsocks' => 'alias for --torsocks=no',
-'save-as=s' => ['NAME', 'save a search terms by given name'],
+'save' => "save a search for `lei up'",
'import-remote!' => 'do not memoize remote messages into local store',
'type=s' => [ 'any|mid|git', 'disambiguate type' ],
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 224eba69..8bca1020 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -53,6 +53,7 @@ sub qstr_add { # PublicInbox::InputPipe::consume callback for --stdin
my ($self) = @_; # $_[1] = $rbuf
if (defined($_[1])) {
$_[1] eq '' and return eval {
+ $self->{mset_opt}->{q_raw} = $self->{mset_opt}->{qstr};
$self->{lse}->query_approxidate($self->{lse}->git,
$self->{mset_opt}->{qstr});
_start_query($self);
@@ -142,6 +143,7 @@ no query allowed on command-line with --stdin
PublicInbox::InputPipe::consume($self->{0}, \&qstr_add, $self);
return;
}
+ $mset_opt{q_raw} = \@argv;
$mset_opt{qstr} =
$self->{lse}->query_argv_to_string($self->{lse}->git, \@argv);
_start_query($self);
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
new file mode 100644
index 00000000..ab9f393b
--- /dev/null
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -0,0 +1,134 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# pretends to be like LeiDedupe and also PublicInbox::Inbox
+package PublicInbox::LeiSavedSearch;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::Lock);
+use PublicInbox::OverIdx;
+use PublicInbox::LeiSearch;
+use PublicInbox::Config;
+use PublicInbox::Spawn qw(run_die);
+use PublicInbox::ContentHash qw(content_hash git_sha);
+use PublicInbox::Eml;
+use PublicInbox::Hval qw(to_filename);
+
+sub new {
+ my ($cls, $lei, $dir) = @_;
+ my $self = bless { ale => $lei->ale, -cfg => {} }, $cls;
+ if (defined $dir) { # updating existing saved search
+ my $f = $self->{'-f'} = "$dir/lei.saved-search";
+ -f $f && -r _ or
+ return $lei->fail("$f non-existent or unreadable");
+ $self->{-cfg} = PublicInbox::Config::git_config_dump($f);
+ my $q = $lei->{mset_opt}->{q_raw} = $self->{-cfg}->{'lei.q'} //
+ return $lei->fail("lei.q unset in $f");
+ my $lse = $lei->{lse} // die 'BUG: {lse} missing';
+ $lei->{mset_opt}->{qstr} = ref($q) ?
+ $lse->query_argv_to_string($lse->git, $q) :
+ $lse->query_approxidate($lse->git, $q);
+ } else { # new saved search
+ my $saved_dir = $lei->store_path . '/../saved-searches/';
+ my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
+ push @name, to_filename($lei->{mset_opt}->{qstr});
+ $dir = $saved_dir . join('-', @name);
+ require File::Path;
+ File::Path::make_path($dir); # raises on error
+ $self->{'-f'} = "$dir/lei.saved-search";
+ my $q = $lei->{mset_opt}->{q_raw};
+ if (ref $q) {
+ cfg_set($self, '--add', 'lei.q', $_) for @$q;
+ } else {
+ cfg_set($self, 'lei.q', $q);
+ }
+ }
+ bless $self->{-cfg}, 'PublicInbox::Config';
+ $self->{lock_path} = "$self->{-f}.flock";
+ $self->{-ovf} = "$dir/over.sqlite3";
+ $self;
+}
+
+sub description { $_[0]->{qstr} } # for WWW
+
+sub cfg_set {
+ my ($self, @args) = @_;
+ my $lk = $self->lock_for_scope; # git-config doesn't wait
+ run_die([qw(git config -f), $self->{'-f'}, @args]);
+}
+
+# drop-in for LeiDedupe API
+sub is_dup {
+ my ($self, $eml, $smsg) = @_;
+ my $oidx = $self->{oidx} // die 'BUG: no {oidx}';
+ my $blob = $smsg ? $smsg->{blob} : undef;
+ return 1 if $blob && $oidx->blob_exists($blob);
+ my $lk = $self->lock_for_scope_fast;
+ if (my $xoids = PublicInbox::LeiSearch::xoids_for($self, $eml, 1)) {
+ for my $docid (values %$xoids) {
+ $oidx->add_xref3($docid, -1, $blob, '.');
+ }
+ $oidx->commit_lazy;
+ 1;
+ } else {
+ # n.b. above xoids_for fills out eml->{-lei_fake_mid} if needed
+ unless ($smsg) {
+ $smsg = bless {}, 'PublicInbox::Smsg';
+ $smsg->{bytes} = 0;
+ $smsg->populate($eml);
+ }
+ $oidx->begin_lazy;
+ $smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
+ $smsg->{blob} //= git_sha(1, $eml)->hexdigest;
+ $oidx->add_overview($eml, $smsg);
+ $oidx->add_xref3($smsg->{num}, -1, $smsg->{blob}, '.');
+ $oidx->commit_lazy;
+ undef;
+ }
+}
+
+sub prepare_dedupe {
+ my ($self) = @_;
+ $self->{oidx} //= do {
+ my $creat = !-f $self->{-ovf};
+ my $lk = $self->lock_for_scope; # git-config doesn't wait
+ my $oidx = PublicInbox::OverIdx->new($self->{-ovf});
+ $oidx->{-no_fsync} = 1;
+ $oidx->dbh;
+ if ($creat) {
+ $oidx->{dbh}->do('PRAGMA journal_mode = WAL');
+ $oidx->eidx_prep; # for xref3
+ }
+ $oidx
+ };
+}
+
+sub over { $_[0]->{oidx} } # for xoids_for
+
+sub git { $_[0]->{ale}->git }
+
+sub pause_dedupe {
+ my ($self) = @_;
+ $self->{ale}->git->cleanup;
+ my $oidx = delete($self->{oidx}) // return;
+ $oidx->commit_lazy;
+}
+
+sub mm { undef }
+
+sub altid_map { {} }
+
+sub cloneurl { [] }
+no warnings 'once';
+*nntp_url = \&cloneurl;
+*base_url = \&PublicInbox::Inbox::base_url;
+*smsg_eml = \&PublicInbox::Inbox::smsg_eml;
+*smsg_by_mid = \&PublicInbox::Inbox::smsg_by_mid;
+*msg_by_mid = \&PublicInbox::Inbox::msg_by_mid;
+*modified = \&PublicInbox::Inbox::modified;
+*recent = \&PublicInbox::Inbox::recent;
+*max_git_epoch = *nntp_usable = *msg_by_path = \&mm; # undef
+*isrch = *search = \&mm; # TODO
+*DESTROY = \&pause_dedupe;
+
+1;
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 7adbffe7..bd2b714a 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -9,7 +9,6 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Eml;
use PublicInbox::ProcessPipe;
use PublicInbox::Spawn qw(spawn);
-use PublicInbox::LeiDedupe;
use PublicInbox::PktOp qw(pkt_do);
use Symbol qw(gensym);
use IO::Handle; # ->autoflush
@@ -350,7 +349,11 @@ sub new {
die "bad mail --format=$fmt\n";
}
$self->{dst} = $dst;
- $lei->{dedupe} = PublicInbox::LeiDedupe->new($lei);
+ my $dd_cls = 'PublicInbox::'.
+ ($lei->{opt}->{save} ? 'LeiSavedSearch' : 'LeiDedupe');
+ eval "require $dd_cls";
+ die "$dd_cls: $@" if $@;
+ $lei->{dedupe} = $dd_cls->new($lei);
$self;
}
@@ -368,6 +371,7 @@ sub _pre_augment_maildir {
sub _do_augment_maildir {
my ($self, $lei) = @_;
+ return if defined($lei->{opt}->{save});
my $dst = $lei->{ovv}->{dst};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
my $mdr = PublicInbox::MdirReader->new;
@@ -398,6 +402,7 @@ sub _imap_augment_or_delete { # PublicInbox::NetReader::imap_each cb
sub _do_augment_imap {
my ($self, $lei) = @_;
+ return if defined($lei->{opt}->{save});
my $net = $lei->{net};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
if ($lei->{opt}->{augment}) {
@@ -468,6 +473,7 @@ sub _do_augment_mbox {
my ($self, $lei) = @_;
return unless $self->{seekable};
my $opt = $lei->{opt};
+ return if defined($opt->{save});
my $out = $lei->{1};
my ($fmt, $dst) = @{$lei->{ovv}}{qw(fmt dst)};
return unless -s $out;
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 9d367977..7c540c1c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -149,22 +149,38 @@ sub query_one_mset { # for --threads and l2m w/o sort
local $0 = "$0 query_one_mset";
my $lei = $self->{lei};
my ($srch, $over) = ($ibxish->search, $ibxish->over);
- my $desc = $ibxish->{inboxdir} // $ibxish->{topdir};
- return warn("$desc not indexed by Xapian\n") unless ($srch && $over);
- my $mo = { %{$lei->{mset_opt}} };
+ my $dir = $ibxish->{inboxdir} // $ibxish->{topdir};
+ return warn("$dir not indexed by Xapian\n") unless ($srch && $over);
+ my $mo = { %{$lei->{mset_opt}} }; # copy
my $mset;
my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
my $can_kw = !!$ibxish->can('msg_keywords');
my $threads = $lei->{opt}->{threads} // 0;
my $fl = $threads > 1 ? 1 : undef;
+ my $lss = $lei->{dedupe};
+ $lss = undef unless $lss && $lss->can('cfg_set'); # saved search
+ my $maxk = "external.$dir.maxnum";
+ my $stop_at = $lss ? $lss->{-cfg}->{$maxk} : undef;
+ if (defined $stop_at) {
+ die "$maxk=$stop_at has multiple values" if ref $stop_at;
+ my @e;
+ local $SIG{__WARN__} = sub { push @e, @_ };
+ $stop_at += 0;
+ return warn("$maxk=$stop_at: @e") if @e;
+ }
+ my $first_ids;
do {
$mset = $srch->mset($mo->{qstr}, $mo);
- mset_progress($lei, $desc, $mset->size,
+ mset_progress($lei, $dir, $mset->size,
$mset->get_matches_estimated);
wait_startq($lei); # wait for keyword updates
my $ids = $srch->mset_to_artnums($mset, $mo);
+ @$ids = grep { $_ > $stop_at } @$ids if defined($stop_at);
my $i = 0;
if ($threads) {
+ # copy $ids if $lss since over->expand_thread
+ # shifts @{$ctx->{ids}}
+ $first_ids = [ @$ids ] if $lss;
my $ctx = { ids => $ids };
my %n2item = map { ($ids->[$i++], $_) } $mset->items;
while ($over->expand_thread($ctx)) {
@@ -183,6 +199,7 @@ sub query_one_mset { # for --threads and l2m w/o sort
@{$ctx->{xids}} = ();
}
} else {
+ $first_ids = $ids;
my @items = $mset->items;
for my $n (@$ids) {
my $mitem = $items[$i++];
@@ -193,6 +210,12 @@ sub query_one_mset { # for --threads and l2m w/o sort
}
}
} while (_mset_more($mset, $mo));
+ if ($lss && scalar(@$first_ids)) {
+ undef $stop_at;
+ my $max = $first_ids->[0];
+ $lss->cfg_set($maxk, $max);
+ undef $lss;
+ }
undef $each_smsg; # may commit
$lei->{ovv}->ovv_atexit_child($lei);
}
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
new file mode 100644
index 00000000..56f7cb37
--- /dev/null
+++ b/t/lei-q-save.t
@@ -0,0 +1,12 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10.1; use PublicInbox::TestCommon;
+test_lei(sub {
+ my $home = $ENV{HOME};
+ lei_ok qw(import t/plack-qp.eml);
+ lei_ok qw(q --save z:0..), '-o', "$home/md/";
+ my @s = glob("$home/.local/share/lei/saved-searches/md-*");
+ is(scalar(@s), 1, 'got one saved search');
+});
+done_testing;
diff --git a/t/lei.t b/t/lei.t
index 2be9b4e8..6ade2f18 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -114,7 +114,7 @@ my $test_completion = sub {
%out = map { $_ => 1 } split(/\s+/s, $lei_out);
for my $sw (qw(-f --format -o --output --mfolder --augment -a
--mua --no-local --local --verbose -v
- --save-as --no-remote --remote --torsocks
+ --save --no-remote --remote --torsocks
--reverse -r )) {
ok($out{$sw}, "$sw offered as `lei q' completion");
}
diff --git a/t/lei_saved_search.t b/t/lei_saved_search.t
new file mode 100644
index 00000000..6d26cd2b
--- /dev/null
+++ b/t/lei_saved_search.t
@@ -0,0 +1,10 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use v5.10.1;
+use PublicInbox::TestCommon;
+require_mods(qw(DBD::SQLite));
+use_ok 'PublicInbox::LeiSavedSearch';
+
+done_testing;
^ permalink raw reply related [relevance 32%]
* [PATCH 5/5] lei: add "lei up" to complement "lei q --save"
2021-04-13 10:54 90% [PATCH 0/5] "lei q --save" + "lei up" Eric Wong
2021-04-13 10:54 32% ` [PATCH 4/5] lei q: start wiring up saved search Eric Wong
@ 2021-04-13 10:54 75% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-13 10:54 UTC (permalink / raw)
To: meta
The command isn't finalized, yet, but it's intended to update
an existing saved search.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 24 ++++++++++------
lib/PublicInbox/LeiToMail.pm | 12 ++++----
lib/PublicInbox/LeiUp.pm | 46 +++++++++++++++++++++++++++++++
t/lei-q-save.t | 17 ++++++++++--
7 files changed, 88 insertions(+), 16 deletions(-)
create mode 100644 lib/PublicInbox/LeiUp.pm
diff --git a/MANIFEST b/MANIFEST
index 20615abc..1b7d16ee 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -208,6 +208,7 @@ lib/PublicInbox/LeiStoreErr.pm
lib/PublicInbox/LeiSucks.pm
lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
+lib/PublicInbox/LeiUp.pm
lib/PublicInbox/LeiXSearch.pm
lib/PublicInbox/Linkify.pm
lib/PublicInbox/Listener.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 7292d0f2..4b87c104 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -135,6 +135,8 @@ our %CMD = ( # sorted in order of importance/use:
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
+'up' => [ 'SEARCH_TERMS...', 'update saved search',
+ qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+), @c_opt ],
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 8bca1020..7456f7f9 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -12,7 +12,7 @@ sub prep_ext { # externals_each callback
$lxs->prepare_external($loc) unless $exclude->{$loc};
}
-sub _start_query {
+sub _start_query { # used by "lei q" and "lei up"
my ($self) = @_;
PublicInbox::LeiOverview->new($self) or return;
my $opt = $self->{opt};
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index ab9f393b..815008fd 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -17,18 +17,12 @@ use PublicInbox::Hval qw(to_filename);
sub new {
my ($cls, $lei, $dir) = @_;
my $self = bless { ale => $lei->ale, -cfg => {} }, $cls;
- if (defined $dir) { # updating existing saved search
+ if (defined $dir) { # updating existing saved search via "lei up"
my $f = $self->{'-f'} = "$dir/lei.saved-search";
-f $f && -r _ or
return $lei->fail("$f non-existent or unreadable");
$self->{-cfg} = PublicInbox::Config::git_config_dump($f);
- my $q = $lei->{mset_opt}->{q_raw} = $self->{-cfg}->{'lei.q'} //
- return $lei->fail("lei.q unset in $f");
- my $lse = $lei->{lse} // die 'BUG: {lse} missing';
- $lei->{mset_opt}->{qstr} = ref($q) ?
- $lse->query_argv_to_string($lse->git, $q) :
- $lse->query_approxidate($lse->git, $q);
- } else { # new saved search
+ } else { # new saved search "lei q --save"
my $saved_dir = $lei->store_path . '/../saved-searches/';
my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
push @name, to_filename($lei->{mset_opt}->{qstr});
@@ -42,6 +36,20 @@ sub new {
} else {
cfg_set($self, 'lei.q', $q);
}
+ my $fmt = $lei->{opt}->{'format'};
+ cfg_set($self, 'lei.q.format', $fmt) if defined $fmt;
+ cfg_set($self, 'lei.q.output', $lei->{opt}->{output});
+ for my $k (qw(only include exclude)) {
+ my $ary = $lei->{opt}->{$k} // next;
+ for my $x (@$ary) {
+ cfg_set($self, '--add', "lei.q.$k", $x);
+ }
+ }
+ for my $k (qw(external local remote import-remote
+ import-before threads)) {
+ my $val = $lei->{opt}->{$k} // next;
+ cfg_set($self, "lei.q.$k", $val);
+ }
}
bless $self->{-cfg}, 'PublicInbox::Config';
$self->{lock_path} = "$self->{-f}.flock";
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index bd2b714a..4ebaf8f3 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -349,11 +349,13 @@ sub new {
die "bad mail --format=$fmt\n";
}
$self->{dst} = $dst;
- my $dd_cls = 'PublicInbox::'.
- ($lei->{opt}->{save} ? 'LeiSavedSearch' : 'LeiDedupe');
- eval "require $dd_cls";
- die "$dd_cls: $@" if $@;
- $lei->{dedupe} = $dd_cls->new($lei);
+ $lei->{dedupe} = $lei->{lss} // do {
+ my $dd_cls = 'PublicInbox::'.
+ ($lei->{opt}->{save} ? 'LeiSavedSearch' : 'LeiDedupe');
+ eval "require $dd_cls";
+ die "$dd_cls: $@" if $@;
+ $dd_cls->new($lei);
+ };
$self;
}
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
new file mode 100644
index 00000000..386a7566
--- /dev/null
+++ b/lib/PublicInbox/LeiUp.pm
@@ -0,0 +1,46 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei up" - updates the result of "lei q --save"
+package PublicInbox::LeiUp;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use PublicInbox::LeiOverview;
+
+sub lei_up {
+ my ($lei, $dir) = @_;
+ $lei->{lse} = $lei->_lei_store(1)->search;
+ my $lss = PublicInbox::LeiSavedSearch->new($lei, $dir) or return;
+ my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
+ $mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
+ my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
+ return $lei->fail("lei.q unset in $lss->{-f}");
+ my $lse = $lei->{lse} // die 'BUG: {lse} missing';
+ if (ref($q)) {
+ $mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
+ } else {
+ $lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q);
+ }
+ $lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
+ return $lei->fail("lei.q.output unset in $lss->{-f}");
+ $lei->{opt}->{'format'} //= $lss->{-cfg}->{'lei.q.format'}; # optional
+
+ my $to_avref = $lss->{-cfg}->can('_array');
+ for my $k (qw(only include exclude)) {
+ my $v = $lss->{-cfg}->{"lei.q.$k"} // next;
+ $lei->{opt}->{$k} = $to_avref->($v);
+ }
+ for my $k (qw(external local remote
+ import-remote import-before threads)) {
+ my $v = $lss->{-cfg}->{"lei.q.$k"} // next;
+ $lei->{opt}->{$k} = $v;
+ }
+ $lei->{lss} = $lss; # for LeiOverview->new
+ my $lxs = $lei->lxs_prepare or return;
+ $lei->ale->refresh_externals($lxs);
+ $lei->{opt}->{save} = 1;
+ $lei->_start_query;
+}
+
+1;
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 56f7cb37..a6d579cf 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -2,11 +2,24 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
+my $doc1 = eml_load('t/plack-qp.eml');
+my $doc2 = eml_load('t/utf8.eml');
test_lei(sub {
my $home = $ENV{HOME};
- lei_ok qw(import t/plack-qp.eml);
- lei_ok qw(q --save z:0..), '-o', "$home/md/";
+ lei_ok qw(import -q t/plack-qp.eml);
+ lei_ok qw(q -q --save z:0..), '-o', "$home/md/";
+ my %before = map { $_ => 1 } glob("$home/md/cur/*");
+ is_deeply(eml_load((keys %before)[0]), $doc1, 'doc1 matches');
+
my @s = glob("$home/.local/share/lei/saved-searches/md-*");
is(scalar(@s), 1, 'got one saved search');
+
+ # ensure "lei up" works, since it compliments "lei q --save"
+ lei_ok qw(import t/utf8.eml);
+ lei_ok qw(up), $s[0];
+ my %after = map { $_ => 1 } glob("$home/md/cur/*");
+ is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
+ is(scalar(keys %after), 1, 'one new message added');
+ is_deeply(eml_load((keys %after)[0]), $doc2, 'doc2 matches');
});
done_testing;
^ permalink raw reply related [relevance 75%]
* Re: [PATCH 4/5] lei q: start wiring up saved search
2021-04-13 10:54 32% ` [PATCH 4/5] lei q: start wiring up saved search Eric Wong
@ 2021-04-13 11:25 71% ` Eric Wong
2021-04-13 19:13 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-13 11:25 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> --- a/lib/PublicInbox/LeiXSearch.pm
> +++ b/lib/PublicInbox/LeiXSearch.pm
> @@ -149,22 +149,38 @@ sub query_one_mset { # for --threads and l2m w/o sort
> local $0 = "$0 query_one_mset";
> my $lei = $self->{lei};
> my ($srch, $over) = ($ibxish->search, $ibxish->over);
> - my $desc = $ibxish->{inboxdir} // $ibxish->{topdir};
> - return warn("$desc not indexed by Xapian\n") unless ($srch && $over);
> - my $mo = { %{$lei->{mset_opt}} };
> + my $dir = $ibxish->{inboxdir} // $ibxish->{topdir};
> + return warn("$dir not indexed by Xapian\n") unless ($srch && $over);
> + my $mo = { %{$lei->{mset_opt}} }; # copy
> my $mset;
> my $each_smsg = $lei->{ovv}->ovv_each_smsg_cb($lei);
> my $can_kw = !!$ibxish->can('msg_keywords');
> my $threads = $lei->{opt}->{threads} // 0;
> my $fl = $threads > 1 ? 1 : undef;
> + my $lss = $lei->{dedupe};
> + $lss = undef unless $lss && $lss->can('cfg_set'); # saved search
> + my $maxk = "external.$dir.maxnum";
Eh, that should probably be "maxuid" since "num" is more
strongly associated with "NNTP article number". extindex may
only be easily exposed via IMAP and HTTP (since we Message-IDs
may conflict with different list trailers).
^ permalink raw reply [relevance 71%]
* Re: [PATCH 4/5] lei q: start wiring up saved search
2021-04-13 11:25 71% ` Eric Wong
@ 2021-04-13 19:13 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-13 19:13 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> Eh, that should probably be "maxuid" since "num" is more
> strongly associated with "NNTP article number". extindex may
> only be easily exposed via IMAP and HTTP (since we Message-IDs
> may conflict with different list trailers).
Pushed with the following squashed in:
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 7c540c1c..e1538391 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -159,7 +159,7 @@ sub query_one_mset { # for --threads and l2m w/o sort
my $fl = $threads > 1 ? 1 : undef;
my $lss = $lei->{dedupe};
$lss = undef unless $lss && $lss->can('cfg_set'); # saved search
- my $maxk = "external.$dir.maxnum";
+ my $maxk = "external.$dir.maxuid";
my $stop_at = $lss ? $lss->{-cfg}->{$maxk} : undef;
if (defined $stop_at) {
die "$maxk=$stop_at has multiple values" if ref $stop_at;
^ permalink raw reply related [relevance 71%]
* [PATCH 0/9] lei saved search usability improvements
@ 2021-04-16 23:10 71% Eric Wong
2021-04-16 23:10 53% ` [PATCH 1/9] lei q: --save preserves relative time queries Eric Wong
` (6 more replies)
0 siblings, 7 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
Found a few bugfixes along the way, but after thinking it over,
I think "lei up /path/to/maildir/or/mbox/or/IMAP-URI" makes the
most sense.
Eric Wong (9):
lei q: --save preserves relative time queries
lei: expose share_path as a method
lei: saved searches keyed only by path/URL and format
lei_to_mail: cast to URIimap object early
test_common: handle '-C' (chdir) spawn option properly
lei: fix rel2abs
lei up: support output destination as arg
lei q --save: avoid lei.q.format
lei q --save: clobber config file on repeats
lib/PublicInbox/Config.pm | 9 ++++
lib/PublicInbox/LEI.pm | 19 +++++----
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 71 ++++++++++++++++++++++++-------
lib/PublicInbox/LeiToMail.pm | 12 +++---
lib/PublicInbox/LeiUp.pm | 5 +--
lib/PublicInbox/Reply.pm | 10 +----
lib/PublicInbox/TestCommon.pm | 7 +++
t/lei-q-save.t | 36 ++++++++++++++--
9 files changed, 126 insertions(+), 45 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 8/9] lei q --save: avoid lei.q.format
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
` (4 preceding siblings ...)
2021-04-16 23:10 50% ` [PATCH 7/9] lei up: support output destination as arg Eric Wong
@ 2021-04-16 23:10 90% ` Eric Wong
2021-04-16 23:10 61% ` [PATCH 9/9] lei q --save: clobber config file on repeats Eric Wong
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
It is redundant since we stuff everything into the lei.q.output
config key.
---
lib/PublicInbox/LeiSavedSearch.pm | 2 --
lib/PublicInbox/LeiUp.pm | 1 -
2 files changed, 3 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 93b1b23a..a8bf470b 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -51,8 +51,6 @@ sub new {
} else {
cfg_set($self, 'lei.q', $q);
}
- my $fmt = $lei->{opt}->{'format'};
- cfg_set($self, 'lei.q.format', $fmt) if defined $fmt;
$dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
cfg_set($self, 'lei.q.output', $dst);
for my $k (qw(only include exclude)) {
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 7ddb1dd0..9fe4901b 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -24,7 +24,6 @@ sub lei_up {
}
$lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
return $lei->fail("lei.q.output unset in $lss->{-f}");
- $lei->{opt}->{'format'} //= $lss->{-cfg}->{'lei.q.format'}; # optional
my $to_avref = $lss->{-cfg}->can('_array');
for my $k (qw(only include exclude)) {
^ permalink raw reply related [relevance 90%]
* [PATCH 6/9] lei: fix rel2abs
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
` (2 preceding siblings ...)
2021-04-16 23:10 69% ` [PATCH 3/9] lei: saved searches keyed only by path/URL and format Eric Wong
@ 2021-04-16 23:10 71% ` Eric Wong
2021-04-16 23:10 50% ` [PATCH 7/9] lei up: support output destination as arg Eric Wong
` (2 subsequent siblings)
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
We don't want pathnames with "GLOB(0xADD12355)" in them.
---
lib/PublicInbox/LEI.pm | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 52b588a2..ebd0f154 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -68,18 +68,19 @@ sub rel2abs ($$) {
my ($self, $p) = @_;
return $p if index($p, '/') == 0; # already absolute
my $pwd = $self->{env}->{PWD};
+ my $cwd;
if (defined $pwd) {
- my $cwd = $self->{3} // getcwd() // die "getcwd(PWD=$pwd): $!";
+ my $xcwd = $self->{3} //
+ ($cwd = getcwd() // die "getcwd(PWD=$pwd): $!");
if (my @st_pwd = stat($pwd)) {
- my @st_cwd = stat($cwd) or die "stat($cwd): $!";
+ my @st_cwd = stat($xcwd) or die "stat($xcwd): $!";
"@st_pwd[1,0]" eq "@st_cwd[1,0]" or
- $self->{env}->{PWD} = $pwd = $cwd;
+ $self->{env}->{PWD} = $pwd = undef;
} else { # PWD was invalid
- delete $self->{env}->{PWD};
- undef $pwd;
+ $self->{env}->{PWD} = $pwd = undef;
}
}
- $pwd //= $self->{env}->{PWD} = getcwd() // die "getcwd(PWD=$pwd): $!";
+ $pwd //= $self->{env}->{PWD} = $cwd // getcwd() // die "getcwd: $!";
File::Spec->rel2abs($p, $pwd);
}
^ permalink raw reply related [relevance 71%]
* [PATCH 2/9] lei: expose share_path as a method
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
2021-04-16 23:10 53% ` [PATCH 1/9] lei q: --save preserves relative time queries Eric Wong
@ 2021-04-16 23:10 69% ` Eric Wong
2021-04-16 23:10 69% ` [PATCH 3/9] lei: saved searches keyed only by path/URL and format Eric Wong
` (4 subsequent siblings)
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
Since saved-searches aren't a part of lei/store, nor
could it be considered cache data... (or can it? it
is discardable, after all).
---
lib/PublicInbox/LEI.pm | 6 ++++--
lib/PublicInbox/LeiSavedSearch.pm | 2 +-
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 4b87c104..52b588a2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -83,13 +83,15 @@ sub rel2abs ($$) {
File::Spec->rel2abs($p, $pwd);
}
-sub store_path ($) {
+sub share_path ($) { # $HOME/.local/share/lei/$FOO
my ($self) = @_;
rel2abs($self, ($self->{env}->{XDG_DATA_HOME} //
($self->{env}->{HOME} // '/nonexistent').'/.local/share')
- .'/lei/store');
+ .'/lei');
}
+sub store_path ($) { share_path($_[0]) . '/store' }
+
sub _config_path ($) {
my ($self) = @_;
rel2abs($self, ($self->{env}->{XDG_CONFIG_HOME} //
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index e79cf76a..fe8301d6 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -23,7 +23,7 @@ sub new {
return $lei->fail("$f non-existent or unreadable");
$self->{-cfg} = PublicInbox::Config::git_config_dump($f);
} else { # new saved search "lei q --save"
- my $saved_dir = $lei->store_path . '/../saved-searches/';
+ my $saved_dir = $lei->share_path . '/saved-searches/';
my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
my $q_raw_str = ref($q) ? "@$q" : $q;
^ permalink raw reply related [relevance 69%]
* [PATCH 3/9] lei: saved searches keyed only by path/URL and format
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
2021-04-16 23:10 53% ` [PATCH 1/9] lei q: --save preserves relative time queries Eric Wong
2021-04-16 23:10 69% ` [PATCH 2/9] lei: expose share_path as a method Eric Wong
@ 2021-04-16 23:10 69% ` Eric Wong
2021-04-16 23:10 71% ` [PATCH 6/9] lei: fix rel2abs Eric Wong
` (3 subsequent siblings)
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
We want users to be able to edit and refine the query over
time while using the same output destination.
---
lib/PublicInbox/LeiSavedSearch.pm | 13 ++++++-------
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index fe8301d6..ebc63091 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -10,9 +10,8 @@ use PublicInbox::OverIdx;
use PublicInbox::LeiSearch;
use PublicInbox::Config;
use PublicInbox::Spawn qw(run_die);
-use PublicInbox::ContentHash qw(content_hash git_sha);
-use PublicInbox::Eml;
-use PublicInbox::Hval qw(to_filename);
+use PublicInbox::ContentHash qw(git_sha);
+use Digest::SHA qw(sha256_hex);
sub new {
my ($cls, $lei, $dir) = @_;
@@ -24,11 +23,11 @@ sub new {
$self->{-cfg} = PublicInbox::Config::git_config_dump($f);
} else { # new saved search "lei q --save"
my $saved_dir = $lei->share_path . '/saved-searches/';
- my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
+ my (@n) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
- my $q_raw_str = ref($q) ? "@$q" : $q;
- push @name, to_filename($q_raw_str);
- $dir = $saved_dir . join('-', @name);
+ push @n, sha256_hex("$lei->{ovv}->{fmt}\0$lei->{ovv}->{dst}");
+
+ $dir = $saved_dir . join('-', @n);
require File::Path;
File::Path::make_path($dir); # raises on error
$self->{'-f'} = "$dir/lei.saved-search";
^ permalink raw reply related [relevance 69%]
* [PATCH 9/9] lei q --save: clobber config file on repeats
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
` (5 preceding siblings ...)
2021-04-16 23:10 90% ` [PATCH 8/9] lei q --save: avoid lei.q.format Eric Wong
@ 2021-04-16 23:10 61% ` Eric Wong
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
A user may wish to clobber/refine existing search parameters
by issuing "lei q --save" again. Support that by overwriting
the lei.saved-search state file entirely.
We continue to preserve over.sqlite3 for deduplication purposes.
This way, we don't get something redundant like:
[lei]
q = term1
q = term2
q = term1
q = term2
q = term3
...whenever a user wants to refine their search. Instead,
we'll just have:
[lei]
q = term1
q = term2
q = term3
On the second go.
---
lib/PublicInbox/Config.pm | 9 +++++++++
lib/PublicInbox/LeiSavedSearch.pm | 10 +++++++++-
lib/PublicInbox/Reply.pm | 10 ++--------
3 files changed, 20 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 26ac298e..603dad98 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -559,4 +559,13 @@ sub json {
};
}
+sub squote_maybe ($) {
+ my ($val) = @_;
+ if ($val =~ m{([^\w@\./,\%\+\-])}) {
+ $val =~ s/(['!])/'\\$1'/g; # '!' for csh
+ return "'$val'";
+ }
+ $val;
+}
+
1;
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index a8bf470b..932b2aa4 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -13,6 +13,8 @@ use PublicInbox::Spawn qw(run_die);
use PublicInbox::ContentHash qw(git_sha);
use Digest::SHA qw(sha256_hex);
+*squote_maybe = \&PublicInbox::Config::squote_maybe;
+
sub lss_dir_for ($$) {
my ($lei, $dstref) = @_;
my @n;
@@ -44,7 +46,13 @@ sub new {
require File::Path;
File::Path::make_path($dir); # raises on error
$self->{-cfg} = {};
- $self->{'-f'} = "$dir/lei.saved-search";
+ my $f = $self->{'-f'} = "$dir/lei.saved-search";
+ open my $fh, '>', $f or return $lei->fail("open $f: $!");
+ my $sq_dst = squote_maybe($dst);
+ print $fh <<EOM or return $lei->fail("print $f: $!");
+; to refresh with new results, run: lei up $sq_dst
+EOM
+ close $fh or return $lei->fail("close $f: $!");
my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
if (ref $q) {
cfg_set($self, '--add', 'lei.q', $_) for @$q;
diff --git a/lib/PublicInbox/Reply.pm b/lib/PublicInbox/Reply.pm
index 2a1066d2..79dd46a7 100644
--- a/lib/PublicInbox/Reply.pm
+++ b/lib/PublicInbox/Reply.pm
@@ -9,15 +9,9 @@ use URI::Escape qw/uri_escape_utf8/;
use PublicInbox::Hval qw(ascii_html obfuscate_addrs mid_href);
use PublicInbox::Address;
use PublicInbox::MID qw(mid_clean);
+use PublicInbox::Config;
-sub squote_maybe ($) {
- my ($val) = @_;
- if ($val =~ m{([^\w@\./,\%\+\-])}) {
- $val =~ s/(['!])/'\\$1'/g; # '!' for csh
- return "'$val'";
- }
- $val;
-}
+*squote_maybe = \&PublicInbox::Config::squote_maybe;
sub add_addrs {
my ($to, $cc, @addrs) = @_;
^ permalink raw reply related [relevance 61%]
* [PATCH 1/9] lei q: --save preserves relative time queries
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
@ 2021-04-16 23:10 53% ` Eric Wong
2021-04-16 23:10 69% ` [PATCH 2/9] lei: expose share_path as a method Eric Wong
` (5 subsequent siblings)
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
Somebody may want a saved search which consistently asks for
messages within a rolling time period window. In other words,
we want to support using "lei q --save dt:last.week.." and keeps
the "dt:last.week.." relative to whenever "lei up" is run. This
ensures relative date-time specifications get used in the future
rather than converting into an absolute date-time from the
initial "lei q" invocation.
---
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 5 +++--
t/lei-q-save.t | 25 +++++++++++++++++++++----
3 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 7456f7f9..7ddba4cf 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -143,7 +143,7 @@ no query allowed on command-line with --stdin
PublicInbox::InputPipe::consume($self->{0}, \&qstr_add, $self);
return;
}
- $mset_opt{q_raw} = \@argv;
+ $mset_opt{q_raw} = [ @argv ]; # copy
$mset_opt{qstr} =
$self->{lse}->query_argv_to_string($self->{lse}->git, \@argv);
_start_query($self);
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 815008fd..e79cf76a 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -25,12 +25,13 @@ sub new {
} else { # new saved search "lei q --save"
my $saved_dir = $lei->store_path . '/../saved-searches/';
my (@name) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
- push @name, to_filename($lei->{mset_opt}->{qstr});
+ my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
+ my $q_raw_str = ref($q) ? "@$q" : $q;
+ push @name, to_filename($q_raw_str);
$dir = $saved_dir . join('-', @name);
require File::Path;
File::Path::make_path($dir); # raises on error
$self->{'-f'} = "$dir/lei.saved-search";
- my $q = $lei->{mset_opt}->{q_raw};
if (ref $q) {
cfg_set($self, '--add', 'lei.q', $_) for @$q;
} else {
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index a6d579cf..6cfac20b 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -2,24 +2,41 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
use strict; use v5.10.1; use PublicInbox::TestCommon;
+use PublicInbox::Smsg;
my $doc1 = eml_load('t/plack-qp.eml');
+$doc1->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 5)}));
my $doc2 = eml_load('t/utf8.eml');
+$doc2->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
+
test_lei(sub {
my $home = $ENV{HOME};
- lei_ok qw(import -q t/plack-qp.eml);
- lei_ok qw(q -q --save z:0..), '-o', "$home/md/";
+ my $in = $doc1->as_string;
+ lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
+ lei_ok qw(q -q --save z:0.. d:last.week..), '-o', "$home/md/";
my %before = map { $_ => 1 } glob("$home/md/cur/*");
is_deeply(eml_load((keys %before)[0]), $doc1, 'doc1 matches');
my @s = glob("$home/.local/share/lei/saved-searches/md-*");
is(scalar(@s), 1, 'got one saved search');
+ my $cfg = PublicInbox::Config->new("$s[0]/lei.saved-search");
+ is_deeply($cfg->{'lei.q'}, ['z:0..', 'd:last.week..'],
+ 'store relative time, not parsed (absolute) timestamp');
# ensure "lei up" works, since it compliments "lei q --save"
- lei_ok qw(import t/utf8.eml);
- lei_ok qw(up), $s[0];
+ $in = $doc2->as_string;
+ lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
+ lei_ok qw(up -q), $s[0];
my %after = map { $_ => 1 } glob("$home/md/cur/*");
is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
is(scalar(keys %after), 1, 'one new message added');
is_deeply(eml_load((keys %after)[0]), $doc2, 'doc2 matches');
+
+ # check stdin
+ lei_ok [qw(q --save - -o), "mboxcl2:mbcl2" ],
+ undef, { -C => $home, %$lei_opt, 0 => \'d:last.week..'};
+ @s = glob("$home/.local/share/lei/saved-searches/mbcl2-*");
+ $cfg = PublicInbox::Config->new("$s[0]/lei.saved-search");
+ is_deeply $cfg->{'lei.q'}, 'd:last.week..',
+ 'q --stdin stores relative time';
});
done_testing;
^ permalink raw reply related [relevance 53%]
* [PATCH 7/9] lei up: support output destination as arg
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
` (3 preceding siblings ...)
2021-04-16 23:10 71% ` [PATCH 6/9] lei: fix rel2abs Eric Wong
@ 2021-04-16 23:10 50% ` Eric Wong
2021-04-16 23:10 90% ` [PATCH 8/9] lei q --save: avoid lei.q.format Eric Wong
2021-04-16 23:10 61% ` [PATCH 9/9] lei q --save: clobber config file on repeats Eric Wong
6 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-16 23:10 UTC (permalink / raw)
To: meta
Specifying a directory in ~/.local/share/lei/saved-searches/
is painful, so support (and start encouraging) the use of
the output.
---
lib/PublicInbox/LeiSavedSearch.pm | 55 ++++++++++++++++++++++++-------
lib/PublicInbox/LeiUp.pm | 4 +--
t/lei-q-save.t | 11 +++++++
3 files changed, 57 insertions(+), 13 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index ebc63091..93b1b23a 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -13,24 +13,39 @@ use PublicInbox::Spawn qw(run_die);
use PublicInbox::ContentHash qw(git_sha);
use Digest::SHA qw(sha256_hex);
+sub lss_dir_for ($$) {
+ my ($lei, $dstref) = @_;
+ my @n;
+ if ($$dstref =~ m,\Aimaps?://,i) { # already canonicalized
+ require PublicInbox::URIimap;
+ my $uri = PublicInbox::URIimap->new($$dstref)->canonical;
+ $$dstref = $$uri;
+ @n = ($uri->mailbox);
+ } else { # basename
+ @n = ($$dstref =~ m{([\w\-\.]+)/*\z});
+ $$dstref = $lei->rel2abs($$dstref);
+ }
+ push @n, sha256_hex($$dstref);
+ $lei->share_path . '/saved-searches/' . join('-', @n);
+}
+
sub new {
my ($cls, $lei, $dir) = @_;
- my $self = bless { ale => $lei->ale, -cfg => {} }, $cls;
+ my $self = bless { ale => $lei->ale }, $cls;
if (defined $dir) { # updating existing saved search via "lei up"
- my $f = $self->{'-f'} = "$dir/lei.saved-search";
- -f $f && -r _ or
+ my $f = "$dir/lei.saved-search";
+ ((-f $f && -r _) || output2lssdir($self, $lei, \$dir, \$f)) or
return $lei->fail("$f non-existent or unreadable");
- $self->{-cfg} = PublicInbox::Config::git_config_dump($f);
+ $self->{-cfg} //= PublicInbox::Config::git_config_dump($f);
+ $self->{'-f'} = $f;
} else { # new saved search "lei q --save"
- my $saved_dir = $lei->share_path . '/saved-searches/';
- my (@n) = ($lei->{ovv}->{dst} =~ m{([\w\-\.]+)/*\z});
- my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
- push @n, sha256_hex("$lei->{ovv}->{fmt}\0$lei->{ovv}->{dst}");
-
- $dir = $saved_dir . join('-', @n);
+ my $dst = $lei->{ovv}->{dst};
+ $dir = lss_dir_for($lei, \$dst);
require File::Path;
File::Path::make_path($dir); # raises on error
+ $self->{-cfg} = {};
$self->{'-f'} = "$dir/lei.saved-search";
+ my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
if (ref $q) {
cfg_set($self, '--add', 'lei.q', $_) for @$q;
} else {
@@ -38,7 +53,8 @@ sub new {
}
my $fmt = $lei->{opt}->{'format'};
cfg_set($self, 'lei.q.format', $fmt) if defined $fmt;
- cfg_set($self, 'lei.q.output', $lei->{opt}->{output});
+ $dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
+ cfg_set($self, 'lei.q.output', $dst);
for my $k (qw(only include exclude)) {
my $ary = $lei->{opt}->{$k} // next;
for my $x (@$ary) {
@@ -127,6 +143,23 @@ sub mm { undef }
sub altid_map { {} }
sub cloneurl { [] }
+
+# find existing directory containing a `lei.saved-search' file based on
+# $dir_ref which is an output
+sub output2lssdir {
+ my ($self, $lei, $dir_ref, $fn_ref) = @_;
+ my $dst = $$dir_ref; # imap://$MAILBOX, /path/to/maildir, /path/to/mbox
+ my $dir = lss_dir_for($lei, \$dst);
+ my $f = "$dir/lei.saved-search";
+ if (-f $f && -r _) {
+ $self->{-cfg} = PublicInbox::Config::git_config_dump($f);
+ $$dir_ref = $dir;
+ $$fn_ref = $f;
+ return 1;
+ }
+ undef;
+}
+
no warnings 'once';
*nntp_url = \&cloneurl;
*base_url = \&PublicInbox::Inbox::base_url;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 386a7566..7ddb1dd0 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -9,9 +9,9 @@ use PublicInbox::LeiSavedSearch;
use PublicInbox::LeiOverview;
sub lei_up {
- my ($lei, $dir) = @_;
+ my ($lei, $out) = @_;
$lei->{lse} = $lei->_lei_store(1)->search;
- my $lss = PublicInbox::LeiSavedSearch->new($lei, $dir) or return;
+ my $lss = PublicInbox::LeiSavedSearch->new($lei, $out) or return;
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 6cfac20b..d43f508b 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -7,6 +7,8 @@ my $doc1 = eml_load('t/plack-qp.eml');
$doc1->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 5)}));
my $doc2 = eml_load('t/utf8.eml');
$doc2->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
+my $doc3 = eml_load('t/msg_iter-order.eml');
+$doc3->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
test_lei(sub {
my $home = $ENV{HOME};
@@ -38,5 +40,14 @@ test_lei(sub {
$cfg = PublicInbox::Config->new("$s[0]/lei.saved-search");
is_deeply $cfg->{'lei.q'}, 'd:last.week..',
'q --stdin stores relative time';
+ my $size = -s "$home/mbcl2";
+ ok(defined($size) && $size > 0, 'results written');
+ lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
+ is(-s "$home/mbcl2", $size, 'size unchanged on noop up');
+
+ $in = $doc3->as_string;
+ lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
+ lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
+ ok(-s "$home/mbcl2" > $size, 'size increased after up');
});
done_testing;
^ permalink raw reply related [relevance 50%]
* [PATCH] lei up: fix canonicalization of Maildirs
@ 2021-04-17 10:24 62% Eric Wong
2021-04-17 19:00 63% ` [PATCH 2/] lei up: further improve Maildir canonicalization Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-17 10:24 UTC (permalink / raw)
To: meta
We always represent --output destination directories with a
trailing slash to disambiguate directories from mbox filenames.
Therefore, we must use the trailing slash when mapping the
destination beck from the lei/saved-search/* directory.
"lei up" now relies exclusively on the users --output pathname
or URL for updates. This ought to be less confusing since
pathnames in ~/.local/store/lei/saved-searches aren't ideal.
---
lib/PublicInbox/LeiSavedSearch.pm | 13 ++++++++-----
t/lei-q-save.t | 4 +++-
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 932b2aa4..0f632d93 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -26,18 +26,21 @@ sub lss_dir_for ($$) {
} else { # basename
@n = ($$dstref =~ m{([\w\-\.]+)/*\z});
$$dstref = $lei->rel2abs($$dstref);
+ $$dstref .= '/' if -d $$dstref;
}
push @n, sha256_hex($$dstref);
$lei->share_path . '/saved-searches/' . join('-', @n);
}
sub new {
- my ($cls, $lei, $dir) = @_;
+ my ($cls, $lei, $dst) = @_;
my $self = bless { ale => $lei->ale }, $cls;
- if (defined $dir) { # updating existing saved search via "lei up"
- my $f = "$dir/lei.saved-search";
- ((-f $f && -r _) || output2lssdir($self, $lei, \$dir, \$f)) or
- return $lei->fail("$f non-existent or unreadable");
+ my $dir;
+ if (defined $dst) { # updating existing saved search via "lei up"
+ my $f;
+ $dir = $dst;
+ output2lssdir($self, $lei, \$dir, \$f) or
+ return $lei->fail("--save was not used with $dst");
$self->{-cfg} //= PublicInbox::Config::git_config_dump($f);
$self->{'-f'} = $f;
} else { # new saved search "lei q --save"
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index d43f508b..6389825f 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -27,7 +27,9 @@ test_lei(sub {
# ensure "lei up" works, since it compliments "lei q --save"
$in = $doc2->as_string;
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
- lei_ok qw(up -q), $s[0];
+ opendir my $dh, '.' or xbail "opendir .: $!";
+ lei_ok qw(up -q md -C), $home;
+ chdir($dh) or xbail "fchdir . $!";
my %after = map { $_ => 1 } glob("$home/md/cur/*");
is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
is(scalar(keys %after), 1, 'one new message added');
^ permalink raw reply related [relevance 62%]
* lei q: --stdin confuses --mua
@ 2021-04-17 15:53 71% Kyle Meyer
2021-04-17 16:04 71% ` Kyle Meyer
2021-04-17 19:00 60% ` [PATCH] lei q: fix MUA spawn after reading query from stdin Eric Wong
0 siblings, 2 replies; 200+ results
From: Kyle Meyer @ 2021-04-17 15:53 UTC (permalink / raw)
To: meta
I used `lei p2p' to track down an associated thread recently. It worked
nicely, though I noticed that I wasn't able to jump straight into mutt
via --mua.
Here's an example using public-inbox.git. Feeding the p2q output on
stdin, I'm told that no recipients were specified, and the command exits
rather than dropping me into mutt's interface.
$ lei p2q 8ab43c1c27c725a8ef9307f5dba3e565169d48ca | \
lei q -q -tt - -o mboxcl2:/tmp/t.mbox --mua=mutt
No recipients were specified.
I'm not a mutt user, but I think that's because mutt sees that stdin
isn't attached to a tty. I haven't tried anything on my end yet, but
perhaps there's a clean way to make --stdin and --mua [*] work together.
Thoughts?
[*] The only other --mua value I checked was 'mail' (and that shows a
similar issue), but I'm guessing other MUAs don't not work well
with --stdin either.
^ permalink raw reply [relevance 71%]
* Re: lei q: --stdin confuses --mua
2021-04-17 15:53 71% lei q: --stdin confuses --mua Kyle Meyer
@ 2021-04-17 16:04 71% ` Kyle Meyer
2021-04-17 19:00 60% ` [PATCH] lei q: fix MUA spawn after reading query from stdin Eric Wong
1 sibling, 0 replies; 200+ results
From: Kyle Meyer @ 2021-04-17 16:04 UTC (permalink / raw)
To: meta
Kyle Meyer writes:
> [*] The only other --mua value I checked was 'mail' (and that shows a
> similar issue), but I'm guessing other MUAs don't not work well
guh, that's a confusing typo: s/don't not/don't/
> with --stdin either.
^ permalink raw reply [relevance 71%]
* [PATCH] lei q: fix MUA spawn after reading query from stdin
2021-04-17 15:53 71% lei q: --stdin confuses --mua Kyle Meyer
2021-04-17 16:04 71% ` Kyle Meyer
@ 2021-04-17 19:00 60% ` Eric Wong
2021-04-17 20:13 70% ` Kyle Meyer
1 sibling, 1 reply; 200+ results
From: Eric Wong @ 2021-04-17 19:00 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> I'm not a mutt user, but I think that's because mutt sees that stdin
> isn't attached to a tty. I haven't tried anything on my end yet, but
> perhaps there's a clean way to make --stdin and --mua [*] work together.
Thanks for the report. Yes, that's correct, fortunately we can
reasonably expect stdout to be a terminal and the patch below
should fix it.
> [*] The only other --mua value I checked was 'mail' (and that shows a
> similar issue), but I'm guessing other MUAs don't not work well
> with --stdin either.
Btw, since you seem to be a gnus user; does/can gnus work
via --mua= ?
----------8<---------
Subject: [PATCH] lei q: fix MUA spawn after reading query from stdin
Since "lei q" may read queries from stdin, we must reconnect a
known terminal before spawning terminal MUAs. Attempt to use
stdout as stdin for this purpose, since terminal MUAs tend to
expect stdout to be a terminal.
Reported-By: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87v98klxg3.fsf@kyleam.com/
---
lib/PublicInbox/LEI.pm | 10 ++++++++--
script/lei | 12 ++++++------
2 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index ebd0f154..f223b3de 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -840,11 +840,17 @@ sub start_mua {
@cmd = map { $_ eq '%f' ? ($replaced = $mfolder) : $_ } @cmd;
}
push @cmd, $mfolder unless defined($replaced);
- if (my $sock = $self->{sock}) { # lei(1) client process runs it
- send($sock, exec_buf(\@cmd, {}), MSG_EOR);
+ if ($self->{sock}) { # lei(1) client process runs it
+ # restore terminal: echo $query | lei q -stdin --mua=...
+ my $io = [];
+ $io->[0] = $self->{1} if $self->{opt}->{stdin} && -t $self->{1};
+ send_exec_cmd($self, $io, \@cmd, {});
} elsif ($self->{oneshot}) {
my $pid = fork // die "fork: $!";
if ($pid > 0) { # original process
+ if ($self->{opt}->{stdin} && -t STDOUT) {
+ open STDIN, '+<&', \*STDOUT or die "dup2: $!";
+ }
exec(@cmd);
warn "exec @cmd: $!\n";
POSIX::_exit(1);
diff --git a/script/lei b/script/lei
index 76217ab9..56e9d299 100755
--- a/script/lei
+++ b/script/lei
@@ -33,6 +33,9 @@ my $exec_cmd = sub {
push @rdr, shift(@old), $newfh;
}
my $do_exec = sub {
+ while (my ($io, $newfh) = splice(@rdr, 0, 2)) {
+ open $io, '+<&', $newfh or die "open +<&=: $!";
+ }
my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
@ENV{keys %env} = values %env;
exec(@argv);
@@ -42,20 +45,17 @@ my $exec_cmd = sub {
$SIG{CHLD} = $sigchld;
my $pid = fork // die "fork: $!";
if ($pid == 0) {
- while (my ($io, $newfh) = splice(@rdr, 0, 2)) {
- open $io, '+<&', $newfh or die "open +<&=: $!";
- }
- $do_exec->() if scalar(@$fds); # git-credential, pager
+ $do_exec->() if $fds->[1]; # git-credential, pager
# parent backgrounds on MUA
POSIX::setsid() > 0 or die "setsid: $!";
@parent = ($parent);
return; # continue $recv_cmd in background
}
- if (scalar(@$fds)) {
+ if ($fds->[1]) {
$pids{$pid} = undef;
} else {
- $do_exec->(); # MUA reuses all FDs
+ $do_exec->(); # MUA reuses stdout
}
};
^ permalink raw reply related [relevance 60%]
* [PATCH 2/] lei up: further improve Maildir canonicalization
2021-04-17 10:24 62% [PATCH] lei up: fix canonicalization of Maildirs Eric Wong
@ 2021-04-17 19:00 63% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-17 19:00 UTC (permalink / raw)
To: meta
We want to be able to use "lei up ." when inside a Maildir.
We'll also relax Maildir/mbox basenames to be any non-'/'
character after converting relative paths to absolute. The
old restriction on allowed characters was unnecessary and made
it impossible to reliably map "." when used as the sole argument
for "lei up".
---
lib/PublicInbox/LeiSavedSearch.pm | 6 ++++--
t/lei-q-save.t | 4 ++++
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 0f632d93..e44779ee 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -24,9 +24,10 @@ sub lss_dir_for ($$) {
$$dstref = $$uri;
@n = ($uri->mailbox);
} else { # basename
- @n = ($$dstref =~ m{([\w\-\.]+)/*\z});
$$dstref = $lei->rel2abs($$dstref);
$$dstref .= '/' if -d $$dstref;
+ $$dstref =~ tr!/!/!s;
+ @n = ($$dstref =~ m{([^/]+)/*\z});
}
push @n, sha256_hex($$dstref);
$lei->share_path . '/saved-searches/' . join('-', @n);
@@ -40,7 +41,8 @@ sub new {
my $f;
$dir = $dst;
output2lssdir($self, $lei, \$dir, \$f) or
- return $lei->fail("--save was not used with $dst");
+ return $lei->fail("--save was not used with $dst cwd=".
+ $lei->rel2abs('.'));
$self->{-cfg} //= PublicInbox::Config::git_config_dump($f);
$self->{'-f'} = $f;
} else { # new saved search "lei q --save"
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 6389825f..a8eda41e 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -29,6 +29,8 @@ test_lei(sub {
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
opendir my $dh, '.' or xbail "opendir .: $!";
lei_ok qw(up -q md -C), $home;
+ lei_ok qw(up -q . -C), "$home/md";
+ lei_ok qw(up -q), "/$home/md";
chdir($dh) or xbail "fchdir . $!";
my %after = map { $_ => 1 } glob("$home/md/cur/*");
is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
@@ -51,5 +53,7 @@ test_lei(sub {
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
ok(-s "$home/mbcl2" > $size, 'size increased after up');
+
+ ok(!lei(qw(up -q), $home), 'up fails w/o --save');
});
done_testing;
^ permalink raw reply related [relevance 63%]
* Re: [PATCH] lei q: fix MUA spawn after reading query from stdin
2021-04-17 19:00 60% ` [PATCH] lei q: fix MUA spawn after reading query from stdin Eric Wong
@ 2021-04-17 20:13 70% ` Kyle Meyer
0 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-04-17 20:13 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> Kyle Meyer <kyle@kyleam.com> wrote:
>> I'm not a mutt user, but I think that's because mutt sees that stdin
>> isn't attached to a tty. I haven't tried anything on my end yet, but
>> perhaps there's a clean way to make --stdin and --mua [*] work together.
>
> Thanks for the report. Yes, that's correct, fortunately we can
> reasonably expect stdout to be a terminal and the patch below
> should fix it.
Confirmed. Thanks for the quick fix.
>> [*] The only other --mua value I checked was 'mail' (and that shows a
>> similar issue), but I'm guessing other MUAs don't not work well
>> with --stdin either.
>
> Btw, since you seem to be a gnus user;
I'm a lightweight Gnus user :) I just use gnus to read NNTP, and use
Notmuch for mail. But...
> does/can gnus work via --mua= ?
... I don't see a good way to do this, no. A more natural approach
would be calling lei from Emacs.
I've been thinking a good amount about Emacs integration with lei in the
context of piem (<https://git.kyleam.com/piem/>). My initial focus will
be something closer to Notmuch's Emacs interface. I'm pretty excited
about the it, as it will be the main way I interact with lei (but sadly
it'll be at least a few weeks before I have the free time to begin any
work on it).
The above interface will probably reduce my use of Gnus to lists that
don't have public-inbox archives, but I still might try to add some Gnus
integration eventually. Gnus has a mairix backend, so that'd be the
first placed I'd study for approaching integration.
^ permalink raw reply [relevance 70%]
* [PATCH] lei ls-search: command to list saved searches
@ 2021-04-18 8:40 36% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-18 8:40 UTC (permalink / raw)
To: meta
Going forward, we'll probably support JSON for all the "ls-*"
subcommands. This also provides the basis for "lei up" shell
completion.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 10 +--
lib/PublicInbox/LeiExternal.pm | 11 +--
lib/PublicInbox/LeiLsSearch.pm | 109 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiSavedSearch.pm | 37 ++++++++--
lib/PublicInbox/LeiUp.pm | 6 ++
t/lei-q-save.t | 12 ++++
7 files changed, 173 insertions(+), 13 deletions(-)
create mode 100644 lib/PublicInbox/LeiLsSearch.pm
diff --git a/MANIFEST b/MANIFEST
index 1b7d16ee..f35c514c 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -196,6 +196,7 @@ lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiLsLabel.pm
+lib/PublicInbox/LeiLsSearch.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f223b3de..56640be1 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -157,8 +157,8 @@ our %CMD = ( # sorted in order of importance/use:
'exclude further results from a publicinbox|extindex',
qw(prune), @c_opt ],
-'ls-query' => [ '[FILTER...]', 'list saved search queries',
- qw(name-only format|f=s), @c_opt ],
+'ls-search' => [ '[PREFIX]', 'list saved search queries',
+ qw(format|f=s pretty l ascii z|0), @c_opt ],
'rm-query' => [ 'QUERY_NAME', 'remove a saved search', @c_opt ],
'mv-query' => [ qw(OLD_NAME NEW_NAME), 'rename a saved search', @c_opt ],
@@ -312,7 +312,9 @@ my %OPTDESC = (
'jobs|j=i add-external' => 'set parallelism when indexing after --mirror',
'in-format|F=s' => $stdin_formats,
-'format|f=s ls-query' => $ls_format,
+'format|f=s ls-search' => ['OUT|json|jsonl|concatjson',
+ 'listing output format' ],
+'l ls-search' => 'long listing format',
'format|f=s ls-external' => $ls_format,
'limit|n=i@' => ['NUM', 'limit on number of matches (default: 10000)' ],
@@ -353,7 +355,7 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol); # internal workers
+my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol lsss); # internal workers
sub _drop_wq {
my ($self) = @_;
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 5e8dc71a..b0ebe947 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -215,8 +215,8 @@ sub lei_forget_external {
}
}
-sub _complete_url_common ($) {
- my ($argv) = @_;
+sub complete_url_common {
+ my $argv = $_[-1];
# Workaround bash word-splitting URLs to ['https', ':', '//' ...]
# Maybe there's a better way to go about this in
# contrib/completion/lei-completion.bash
@@ -228,7 +228,8 @@ sub _complete_url_common ($) {
push @x, $cur;
$cur = '';
}
- while (@x > 2 && $x[0] !~ /\Ahttps?\z/ && $x[1] ne ':') {
+ while (@x > 2 && $x[0] !~ /\A(?:http|nntp|imap)s?\z/i &&
+ $x[1] ne ':') {
shift @x;
}
if (@x >= 2) { # qw(https : hostname : 443) or qw(http :)
@@ -245,7 +246,7 @@ sub _complete_url_common ($) {
sub _complete_forget_external {
my ($self, @argv) = @_;
my $cfg = $self->_lei_cfg;
- my ($cur, $re) = _complete_url_common(\@argv);
+ my ($cur, $re) = complete_url_common(\@argv);
# FIXME: bash completion off "http:" or "https:" when the last
# character is a colon doesn't work properly even if we're
# returning "//$HTTP_HOST/$PATH_INFO/", not sure why, could
@@ -261,7 +262,7 @@ sub _complete_forget_external {
sub _complete_add_external { # for bash, this relies on "compopt -o nospace"
my ($self, @argv) = @_;
my $cfg = $self->_lei_cfg;
- my ($cur, $re) = _complete_url_common(\@argv);
+ my ($cur, $re) = complete_url_common(\@argv);
require URI;
map {
my $u = URI->new(substr($_, length('external.')));
diff --git a/lib/PublicInbox/LeiLsSearch.pm b/lib/PublicInbox/LeiLsSearch.pm
new file mode 100644
index 00000000..2aa457c0
--- /dev/null
+++ b/lib/PublicInbox/LeiLsSearch.pm
@@ -0,0 +1,109 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei ls-search" to display results saved via "lei q --save"
+package PublicInbox::LeiLsSearch;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use parent qw(PublicInbox::IPC);
+
+sub do_ls_search_long {
+ my ($self, $pfx) = @_;
+ # TODO: share common JSON output code with LeiOverview
+ my $json = $self->{json}->new->utf8->canonical;
+ my $lei = $self->{lei};
+ $json->ascii(1) if $lei->{opt}->{ascii};
+ my $fmt = $lei->{opt}->{'format'};
+ $lei->{1}->autoflush(0);
+ my $ORS = "\n";
+ my $pretty = $lei->{opt}->{pretty};
+ my $EOR; # TODO: compact pretty like "lei q"
+ if ($fmt =~ /\A(concat)?json\z/ && $pretty) {
+ $EOR = ($1//'') eq 'concat' ? "\n}" : "\n},";
+ }
+ if ($fmt eq 'json') {
+ $lei->out('[');
+ $ORS = ",\n";
+ }
+ my @x = sort(grep(/\A\Q$pfx/, PublicInbox::LeiSavedSearch::list($lei)));
+ while (my $x = shift @x) {
+ $ORS = '' if !scalar(@x);
+ my $lss = PublicInbox::LeiSavedSearch->new($lei, $x) or next;
+ my $cfg = $lss->{-cfg};
+ my $ent = {
+ q => $cfg->get_all('lei.q'),
+ output => $cfg->{'lei.q.output'},
+ };
+ for my $k ($lss->ARRAY_FIELDS) {
+ my $ary = $cfg->get_all("lei.q.$k") // next;
+ $ent->{$k} = $ary;
+ }
+ for my $k ($lss->BOOL_FIELDS) {
+ my $val = $cfg->{"lei.q.$k"} // next;
+ $ent->{$k} = $val;
+ }
+ if (defined $EOR) { # pretty, but compact
+ $EOR = "\n}" if !scalar(@x);
+ my $buf = "{\n";
+ $buf .= join(",\n", map {;
+ my $f = $_;
+ if (my $v = $ent->{$f}) {
+ $v = $json->encode([$v]);
+ qq{ "$f": }.substr($v, 1, -1);
+ } else {
+ ();
+ }
+ # key order by importance
+ } (qw(output q), $lss->ARRAY_FIELDS,
+ $lss->BOOL_FIELDS) );
+ $lei->out($buf .= $EOR);
+ } else {
+ $lei->out($json->encode($ent), $ORS);
+ }
+ }
+ if ($fmt eq 'json') {
+ $lei->out("]\n");
+ } elsif ($fmt eq 'concatjson') {
+ $lei->out("\n");
+ }
+}
+
+sub bg_worker ($$$) {
+ my ($lei, $pfx, $json) = @_;
+ my $self = bless { -wq_nr_workers => 1, json => $json }, __PACKAGE__;
+ my ($op_c, $ops) = $lei->workers_start($self, 'ls-search', 1);
+ $lei->{lsss} = $self;
+ $self->wq_io_do('do_ls_search_long', [], $pfx);
+ $self->wq_close(1);
+ $op_c->op_wait_event($ops);
+}
+
+sub lei_ls_search {
+ my ($lei, $pfx) = @_;
+ my $fmt = $lei->{opt}->{'format'} // '';
+ if ($lei->{opt}->{l}) {
+ $lei->{opt}->{'format'} //= $fmt = 'json';
+ }
+ my $json;
+ my $tty = -t $lei->{1};
+ $lei->start_pager if $tty;
+ if ($fmt =~ /\A(ldjson|ndjson|jsonl|(?:concat)?json)\z/) {
+ $lei->{opt}->{pretty} //= $tty;
+ $json = ref(PublicInbox::Config->json);
+ } elsif ($fmt ne '') {
+ return $lei->fail("unknown format: $fmt");
+ }
+ my $ORS = "\n";
+ if ($lei->{opt}->{z}) {
+ return $lei->fail('-z and --format do not mix') if $json;
+ $ORS = "\0";
+ }
+ $pfx //= '';
+ return bg_worker($lei, $pfx, $json) if $json;
+ for (sort(grep(/\A\Q$pfx/, PublicInbox::LeiSavedSearch::list($lei)))) {
+ $lei->out($_, $ORS);
+ }
+}
+
+1;
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 3076d14c..d67622c9 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -21,6 +21,11 @@ sub cquote_val ($) { # cf. git-config(1)
$val;
}
+sub ARRAY_FIELDS () { qw(only include exclude) }
+sub BOOL_FIELDS () {
+ qw(external local remote import-remote import-before threads)
+}
+
sub lss_dir_for ($$) {
my ($lei, $dstref) = @_;
my @n;
@@ -39,6 +44,31 @@ sub lss_dir_for ($$) {
$lei->share_path . '/saved-searches/' . join('-', @n);
}
+sub list {
+ my ($lei, $pfx) = @_;
+ my $lss_dir = $lei->share_path.'/saved-searches/';
+ return () unless -d $lss_dir;
+ # TODO: persist the cache? Use another format?
+ my $f = $lei->cache_dir."/saved-tmp.$$.".time.'.config';
+ open my $fh, '>', $f or die "open $f: $!";
+ print $fh "[include]\n";
+ for my $p (glob("$lss_dir/*/lei.saved-search")) {
+ print $fh "\tpath = ", cquote_val($p), "\n";
+ }
+ close $fh or die "close $f: $!";
+ my $cfg = PublicInbox::Config::git_config_dump($f);
+ unlink($f);
+ bless $cfg, 'PublicInbox::Config';
+ my $out = $cfg->get_all('lei.q.output') or return ();
+ map {;
+ if (s!\A(?:maildir|mh|mbox.+|mmdf):!!i) {
+ -e $_ ? $_ : (); # TODO auto-prune somewhere?
+ } else { # IMAP, maybe JMAP
+ $_;
+ }
+ } @$out
+}
+
sub new {
my ($cls, $lei, $dst) = @_;
my $self = bless { ale => $lei->ale }, $cls;
@@ -74,16 +104,15 @@ $q
[lei "q"]
output = $dst
EOM
- for my $k (qw(only include exclude)) {
+ for my $k (ARRAY_FIELDS) {
my $ary = $lei->{opt}->{$k} // next;
for my $x (@$ary) {
print $fh "\t$k = ".cquote_val($x)."\n";
}
}
- for my $k (qw(external local remote import-remote
- import-before threads)) {
+ for my $k (BOOL_FIELDS) {
my $val = $lei->{opt}->{$k} // next;
- print $fh "\t$k = ".cquote_val($val)."\n";
+ print $fh "\t$k = ".($val ? 1 : 0)."\n";
}
close($fh) or return $lei->fail("close $f: $!");
}
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 9fe4901b..73286ea2 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -42,4 +42,10 @@ sub lei_up {
$lei->_start_query;
}
+sub _complete_up {
+ my ($lei, @argv) = @_;
+ my ($cur, $re) = $lei->complete_url_common(\@argv);
+ grep(/\A$re\Q$cur/, PublicInbox::LeiSavedSearch::list($lei));
+}
+
1;
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index a8eda41e..761814b4 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -55,5 +55,17 @@ test_lei(sub {
ok(-s "$home/mbcl2" > $size, 'size increased after up');
ok(!lei(qw(up -q), $home), 'up fails w/o --save');
+
+ lei_ok qw(ls-search); my @d = split(/\n/, $lei_out);
+ lei_ok qw(ls-search -z); my @z = split(/\0/, $lei_out);
+ is_deeply(\@d, \@z, '-z output matches non-z');
+ is_deeply(\@d, [ "$home/mbcl2", "$home/md/" ],
+ 'ls-search output alphabetically sorted');
+ lei_ok qw(ls-search -l);
+ my $json = PublicInbox::Config->json->decode($lei_out);
+ ok($json && $json->[0]->{output}, 'JSON has output');
+ lei_ok qw(_complete lei up);
+ like($lei_out, qr!^\Q$home/mbcl2\E$!sm, 'complete got mbcl2 output');
+ like($lei_out, qr!^\Q$home/md/\E$!sm, 'complete got maildir output');
});
done_testing;
^ permalink raw reply related [relevance 36%]
* [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
2021-04-19 8:52 68% ` [PATCH 1/6] lei: support unlinked/missing saved searches Eric Wong
2021-04-19 8:52 59% ` [PATCH 2/6] lei q: implement import-before default for --save Eric Wong
@ 2021-04-19 8:52 73% ` Eric Wong
2021-04-19 8:52 65% ` [PATCH 6/6] lei q: --save and --augment may be combined Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
They're more different than alike, and having two separate
methods seems less confusing to me.
---
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 79 ++++++++++++++++---------------
lib/PublicInbox/LeiUp.pm | 2 +-
3 files changed, 44 insertions(+), 39 deletions(-)
diff --git a/lib/PublicInbox/LeiLsSearch.pm b/lib/PublicInbox/LeiLsSearch.pm
index 2aa457c0..9ac4870f 100644
--- a/lib/PublicInbox/LeiLsSearch.pm
+++ b/lib/PublicInbox/LeiLsSearch.pm
@@ -29,7 +29,7 @@ sub do_ls_search_long {
my @x = sort(grep(/\A\Q$pfx/, PublicInbox::LeiSavedSearch::list($lei)));
while (my $x = shift @x) {
$ORS = '' if !scalar(@x);
- my $lss = PublicInbox::LeiSavedSearch->new($lei, $x) or next;
+ my $lss = PublicInbox::LeiSavedSearch->up($lei, $x) or next;
my $cfg = $lss->{-cfg};
my $ent = {
q => $cfg->get_all('lei.q'),
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index d3a32d36..948e4954 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -65,52 +65,57 @@ sub list {
} @$out
}
-sub new {
+sub up { # updating existing saved search via "lei up"
my ($cls, $lei, $dst) = @_;
+ my $f;
my $self = bless { ale => $lei->ale }, $cls;
- my $dir;
- if (defined $dst) { # updating existing saved search via "lei up"
- my $f;
- $dir = $dst;
- output2lssdir($self, $lei, \$dir, \$f) or
- return $lei->fail("--save was not used with $dst cwd=".
- $lei->rel2abs('.'));
- $self->{'-f'} = $f;
- } else { # new saved search "lei q --save"
- $dst = $lei->{ovv}->{dst};
- $dir = lss_dir_for($lei, \$dst);
- require File::Path;
- File::Path::make_path($dir); # raises on error
- $self->{-cfg} = {};
- my $f = $self->{'-f'} = "$dir/lei.saved-search";
- open my $fh, '>', $f or return $lei->fail("open $f: $!");
- my $sq_dst = PublicInbox::Config::squote_maybe($dst);
- my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
- if (ref $q) {
- $q = join("\n", map { "\tq = ".cquote_val($_) } @$q);
- } else {
- $q = "\tq = ".cquote_val($q);
- }
- $dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
- print $fh <<EOM;
+ my $dir = $dst;
+ output2lssdir($self, $lei, \$dir, \$f) or
+ return $lei->fail("--save was not used with $dst cwd=".
+ $lei->rel2abs('.'));
+ $self->{-cfg} = PublicInbox::Config->git_config_dump($f);
+ $self->{-ovf} = "$dir/over.sqlite3";
+ $self->{'-f'} = $f;
+ $self->{lock_path} = "$self->{-f}.flock";
+ $self;
+}
+
+sub new { # new saved search "lei q --save"
+ my ($cls, $lei) = @_;
+ my $self = bless { ale => $lei->ale }, $cls;
+ my $dst = $lei->{ovv}->{dst};
+ my $dir = lss_dir_for($lei, \$dst);
+ require File::Path;
+ File::Path::make_path($dir); # raises on error
+ $self->{-cfg} = {};
+ my $f = $self->{'-f'} = "$dir/lei.saved-search";
+ open my $fh, '>', $f or return $lei->fail("open $f: $!");
+ my $sq_dst = PublicInbox::Config::squote_maybe($dst);
+ my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
+ if (ref $q) {
+ $q = join("\n", map { "\tq = ".cquote_val($_) } @$q);
+ } else {
+ $q = "\tq = ".cquote_val($q);
+ }
+ $dst = "$lei->{ovv}->{fmt}:$dst" if $dst !~ m!\Aimaps?://!i;
+ print $fh <<EOM;
; to refresh with new results, run: lei up $sq_dst
[lei]
-$q
+ $q
[lei "q"]
output = $dst
EOM
- for my $k (ARRAY_FIELDS) {
- my $ary = $lei->{opt}->{$k} // next;
- for my $x (@$ary) {
- print $fh "\t$k = ".cquote_val($x)."\n";
- }
- }
- for my $k (BOOL_FIELDS) {
- my $val = $lei->{opt}->{$k} // next;
- print $fh "\t$k = ".($val ? 1 : 0)."\n";
+ for my $k (ARRAY_FIELDS) {
+ my $ary = $lei->{opt}->{$k} // next;
+ for my $x (@$ary) {
+ print $fh "\t$k = ".cquote_val($x)."\n";
}
- close($fh) or return $lei->fail("close $f: $!");
}
+ for my $k (BOOL_FIELDS) {
+ my $val = $lei->{opt}->{$k} // next;
+ print $fh "\t$k = ".($val ? 1 : 0)."\n";
+ }
+ close($fh) or return $lei->fail("close $f: $!");
$self->{lock_path} = "$self->{-f}.flock";
$self->{-ovf} = "$dir/over.sqlite3";
$self;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index e4cbc825..23c5c606 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -11,7 +11,7 @@ use PublicInbox::LeiOverview;
sub lei_up {
my ($lei, $out) = @_;
$lei->{lse} = $lei->_lei_store(1)->search;
- my $lss = PublicInbox::LeiSavedSearch->new($lei, $out) or return;
+ my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
^ permalink raw reply related [relevance 73%]
* [PATCH 0/6] lei saved search improvements
@ 2021-04-19 8:52 71% Eric Wong
2021-04-19 8:52 68% ` [PATCH 1/6] lei: support unlinked/missing saved searches Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
I think the "lei q --save" and "lei up" commands are in a good
state. There's no way to edit/forget saved searches, yet,
so I guess that's next...
Eric Wong (6):
lei: support unlinked/missing saved searches
lei q: implement import-before default for --save
lei_saved_search: avoid needless var shadowing
config: git_config_dump blesses
lei_saved_search: split "lei q --save" and "lei up" init paths
lei q: --save and --augment may be combined
lib/PublicInbox/Config.pm | 10 ++--
lib/PublicInbox/LEI.pm | 3 +-
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiSavedSearch.pm | 94 +++++++++++++++----------------
lib/PublicInbox/LeiToMail.pm | 6 +-
lib/PublicInbox/LeiUp.pm | 4 +-
t/lei-q-save.t | 41 ++++++++++++++
8 files changed, 101 insertions(+), 61 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/6] lei: support unlinked/missing saved searches
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
@ 2021-04-19 8:52 68% ` Eric Wong
2021-04-19 8:52 59% ` [PATCH 2/6] lei q: implement import-before default for --save Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
It's conceivable a user will want to erase all previous
results but still rerun/refresh a search to get new results.
We probably won't support prune functionality, here, and
instead require explicit removal of saved searches.
---
lib/PublicInbox/LeiSavedSearch.pm | 7 ++-----
t/lei-q-save.t | 7 +++++++
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index d67622c9..94920a4e 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -61,11 +61,8 @@ sub list {
bless $cfg, 'PublicInbox::Config';
my $out = $cfg->get_all('lei.q.output') or return ();
map {;
- if (s!\A(?:maildir|mh|mbox.+|mmdf):!!i) {
- -e $_ ? $_ : (); # TODO auto-prune somewhere?
- } else { # IMAP, maybe JMAP
- $_;
- }
+ s!\A(?:maildir|mh|mbox.+|mmdf):!!i;
+ $_;
} @$out
}
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 761814b4..4e6ed642 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -67,5 +67,12 @@ test_lei(sub {
lei_ok qw(_complete lei up);
like($lei_out, qr!^\Q$home/mbcl2\E$!sm, 'complete got mbcl2 output');
like($lei_out, qr!^\Q$home/md/\E$!sm, 'complete got maildir output');
+
+ unlink("$home/mbcl2") or xbail "unlink $!";
+ lei_ok qw(_complete lei up);
+ like($lei_out, qr!^\Q$home/mbcl2\E$!sm,
+ 'mbcl2 output shown despite unlink');
+ lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
+ ok(-f "$home/mbcl2" && -s _ == 0, 'up recreates on missing output');
});
done_testing;
^ permalink raw reply related [relevance 68%]
* [PATCH 6/6] lei q: --save and --augment may be combined
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
` (2 preceding siblings ...)
2021-04-19 8:52 73% ` [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths Eric Wong
@ 2021-04-19 8:52 65% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
This necessitated fixing pause_dedupe to release the handle
used by ->lock_for_scope_fast, but otherwise no changes to
the LeiToMail package.
---
lib/PublicInbox/LeiSavedSearch.pm | 1 +
t/lei-q-save.t | 13 +++++++++++++
2 files changed, 14 insertions(+)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 948e4954..cd9effce 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -182,6 +182,7 @@ sub git { $_[0]->{ale}->git }
sub pause_dedupe {
my ($self) = @_;
$self->{ale}->git->cleanup;
+ my $lockfh = delete $self->{lockfh}; # from lock_for_scope_fast;
my $oidx = delete($self->{oidx}) // return;
$oidx->commit_lazy;
}
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 5bc8fb74..c0c74581 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -84,6 +84,7 @@ test_lei(sub {
lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
ok(-f "$home/mbcl2" && -s _ == 0, 'up recreates on missing output');
+ # no --augment
open my $mb, '>', "$home/mbrd" or xbail "open $!";
print $mb $pre_existing;
close $mb or xbail "close: $!";
@@ -92,8 +93,20 @@ test_lei(sub {
open $mb, '<', "$home/mbrd" or xbail "open $!";
is_deeply([grep(/pre-existing/, <$mb>)], [],
'pre-existing messsage gone w/o augment');
+ close $mb;
lei_ok(qw(q m:import-before@example.com));
is(json_utf8->decode($lei_out)->[0]->{'s'},
'pre-existing', '--save imported before clobbering');
+
+ # --augment
+ open $mb, '>', "$home/mbrd-aug" or xbail "open $!";
+ print $mb $pre_existing;
+ close $mb or xbail "close: $!";
+ lei_ok(qw(q -a --save -o mboxrd:mbrd-aug m:qp@example.com -C), $home);
+ chdir($dh) or xbail "fchdir . $!";
+ open $mb, '<', "$home/mbrd-aug" or xbail "open $!";
+ $mb = do { local $/; <$mb> };
+ like($mb, qr/pre-existing/, 'pre-existing message preserved w/ -a');
+ like($mb, qr/<qp\@example\.com>/, 'new result written w/ -a');
});
done_testing;
^ permalink raw reply related [relevance 65%]
* [PATCH 2/6] lei q: implement import-before default for --save
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
2021-04-19 8:52 68% ` [PATCH 1/6] lei: support unlinked/missing saved searches Eric Wong
@ 2021-04-19 8:52 59% ` Eric Wong
2021-04-19 8:52 73% ` [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths Eric Wong
2021-04-19 8:52 65% ` [PATCH 6/6] lei q: --save and --augment may be combined Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 8:52 UTC (permalink / raw)
To: meta
This makes "lei q --save" as safe as "lei q" to prevent against
accidental data loss when clobbering an existing output,
---
lib/PublicInbox/LeiToMail.pm | 6 +++---
lib/PublicInbox/LeiUp.pm | 2 +-
t/lei-q-save.t | 21 +++++++++++++++++++++
3 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index daa8084b..46a82a4b 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -375,7 +375,7 @@ sub _pre_augment_maildir {
sub _do_augment_maildir {
my ($self, $lei) = @_;
- return if defined($lei->{opt}->{save});
+ return if ($lei->{opt}->{save} // 0) < 0;
my $dst = $lei->{ovv}->{dst};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
my $mdr = PublicInbox::MdirReader->new;
@@ -406,7 +406,7 @@ sub _imap_augment_or_delete { # PublicInbox::NetReader::imap_each cb
sub _do_augment_imap {
my ($self, $lei) = @_;
- return if defined($lei->{opt}->{save});
+ return if ($lei->{opt}->{save} // 0) < 0;
my $net = $lei->{net};
my $lse = $lei->{opt}->{'import-before'} ? $lei->{lse} : undef;
if ($lei->{opt}->{augment}) {
@@ -477,7 +477,7 @@ sub _do_augment_mbox {
my ($self, $lei) = @_;
return unless $self->{seekable};
my $opt = $lei->{opt};
- return if defined($opt->{save});
+ return if ($opt->{save} // 0) < 0;
my $out = $lei->{1};
my ($fmt, $dst) = @{$lei->{ovv}}{qw(fmt dst)};
return unless -s $out;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 73286ea2..e4cbc825 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -38,7 +38,7 @@ sub lei_up {
$lei->{lss} = $lss; # for LeiOverview->new
my $lxs = $lei->lxs_prepare or return;
$lei->ale->refresh_externals($lxs);
- $lei->{opt}->{save} = 1;
+ $lei->{opt}->{save} = -1;
$lei->_start_query;
}
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 4e6ed642..5bc8fb74 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -10,6 +10,15 @@ $doc2->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
my $doc3 = eml_load('t/msg_iter-order.eml');
$doc3->header_set('Date', PublicInbox::Smsg::date({ds => time - (86400 * 4)}));
+my $pre_existing = <<'EOF';
+From x Mon Sep 17 00:00:00 2001
+Message-ID: <import-before@example.com>
+Subject: pre-existing
+Date: Sat, 02 Oct 2010 00:00:00 +0000
+
+blah
+EOF
+
test_lei(sub {
my $home = $ENV{HOME};
my $in = $doc1->as_string;
@@ -74,5 +83,17 @@ test_lei(sub {
'mbcl2 output shown despite unlink');
lei_ok([qw(up mbcl2)], undef, { -C => $home, %$lei_opt });
ok(-f "$home/mbcl2" && -s _ == 0, 'up recreates on missing output');
+
+ open my $mb, '>', "$home/mbrd" or xbail "open $!";
+ print $mb $pre_existing;
+ close $mb or xbail "close: $!";
+ lei_ok(qw(q --save -o mboxrd:mbrd m:qp@example.com -C), $home);
+ chdir($dh) or xbail "fchdir . $!";
+ open $mb, '<', "$home/mbrd" or xbail "open $!";
+ is_deeply([grep(/pre-existing/, <$mb>)], [],
+ 'pre-existing messsage gone w/o augment');
+ lei_ok(qw(q m:import-before@example.com));
+ is(json_utf8->decode($lei_out)->[0]->{'s'},
+ 'pre-existing', '--save imported before clobbering');
});
done_testing;
^ permalink raw reply related [relevance 59%]
* [PATCH 0/4] "lei up --all=local" support
@ 2021-04-19 23:48 71% Eric Wong
2021-04-19 23:48 71% ` [PATCH 1/4] lei up: fix help output and ARGV handling Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-04-19 23:48 UTC (permalink / raw)
To: meta
Eventually, "--all" and "--all=remote" will be supported,
but "--all=local" was relatively easy and will probably be
the most popular for latency-sensitive users.
Eric Wong (4):
lei up: fix help output and ARGV handling
config: favor ->get_all when possible
lei up: more error checking for config loading
lei up: support --all=local
lib/PublicInbox/LEI.pm | 7 +++-
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiUp.pm | 75 +++++++++++++++++++++++++++++++------
lib/PublicInbox/Watch.pm | 3 +-
t/lei-q-save.t | 2 +
5 files changed, 72 insertions(+), 17 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/4] lei up: fix help output and ARGV handling
2021-04-19 23:48 71% [PATCH 0/4] "lei up --all=local" support Eric Wong
@ 2021-04-19 23:48 71% ` Eric Wong
2021-04-19 23:49 65% ` [PATCH 3/4] lei up: more error checking for config loading Eric Wong
2021-04-19 23:49 49% ` [PATCH 4/4] lei up: support --all=local Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 23:48 UTC (permalink / raw)
To: meta
We don't support changing search terms once "lei q --save" is
used.
---
lib/PublicInbox/LEI.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index f641f0d9..5ee02f64 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -138,8 +138,8 @@ our %CMD = ( # sorted in order of importance/use:
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
-'up' => [ 'SEARCH_TERMS...', 'update saved search',
- qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+), @c_opt ],
+'up' => [ 'OUTPUT', 'update saved search',
+ qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all), @c_opt ],
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
^ permalink raw reply related [relevance 71%]
* [PATCH 3/4] lei up: more error checking for config loading
2021-04-19 23:48 71% [PATCH 0/4] "lei up --all=local" support Eric Wong
2021-04-19 23:48 71% ` [PATCH 1/4] lei up: fix help output and ARGV handling Eric Wong
@ 2021-04-19 23:49 65% ` Eric Wong
2021-04-19 23:49 49% ` [PATCH 4/4] lei up: support --all=local Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 23:49 UTC (permalink / raw)
To: meta
We'll support editing the saved search config file, so user
errors may happen and we need to throw sensible errors in that
case.
---
lib/PublicInbox/LeiUp.pm | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 63a7f996..e80ccf57 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -14,24 +14,27 @@ sub lei_up {
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
+ my $f = $lss->{'-f'};
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
- return $lei->fail("lei.q unset in $lss->{-f}");
+ return $lei->fail("lei.q unset in $f");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
if (ref($q)) {
$mset_opt->{qstr} = $lse->query_argv_to_string($lse->git, $q);
} else {
$lse->query_approxidate($lse->git, $mset_opt->{qstr} = $q);
}
- $lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
- return $lei->fail("lei.q.output unset in $lss->{-f}");
-
+ my $o = $lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
+ return $lei->fail("lei.q.output unset in $f");
+ ref($o) and return $lei->fail("multiple values of lei.q.output in $f");
for my $k (qw(only include exclude)) {
my $v = $lss->{-cfg}->get_all("lei.q.$k") // next;
$lei->{opt}->{$k} = $v;
}
for my $k (qw(external local remote
import-remote import-before threads)) {
- my $v = $lss->{-cfg}->{"lei.q.$k"} // next;
+ my $c = "lei.q.$k";
+ my $v = $lss->{-cfg}->{$c} // next;
+ ref($v) and return $lei->fail("multiple values of $c in $f");
$lei->{opt}->{$k} = $v;
}
$lei->{lss} = $lss; # for LeiOverview->new
^ permalink raw reply related [relevance 65%]
* [PATCH 4/4] lei up: support --all=local
2021-04-19 23:48 71% [PATCH 0/4] "lei up --all=local" support Eric Wong
2021-04-19 23:48 71% ` [PATCH 1/4] lei up: fix help output and ARGV handling Eric Wong
2021-04-19 23:49 65% ` [PATCH 3/4] lei up: more error checking for config loading Eric Wong
@ 2021-04-19 23:49 49% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-19 23:49 UTC (permalink / raw)
To: meta
Users may wish to update several saved searches at once. We can
support parallel updates in lei-daemon so users won't have to do
it themselves via xargs or similar.
Supporting IMAP outputs would be significantly more involved
since we'd have to pre-authenticate for every single IMAP
output before entering the redispatch loop.
---
lib/PublicInbox/LEI.pm | 7 +++--
lib/PublicInbox/LeiQuery.pm | 2 +-
lib/PublicInbox/LeiUp.pm | 59 +++++++++++++++++++++++++++++++++----
t/lei-q-save.t | 2 ++
4 files changed, 62 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 5ee02f64..c0385eb5 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -138,8 +138,9 @@ our %CMD = ( # sorted in order of importance/use:
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
-'up' => [ 'OUTPUT', 'update saved search',
- qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all), @c_opt ],
+
+'up' => [ 'OUTPUT|--all', 'update saved search',
+ qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all:s), @c_opt ],
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
@@ -331,6 +332,8 @@ my %OPTDESC = (
'remote' => 'limit operations to those requiring network access',
'remote!' => 'prevent operations requiring network access',
+'all:s up' => ['local', 'update all (local) saved searches' ],
+
'mid=s' => 'specify the Message-ID of a message',
'oid=s' => 'specify the git object ID of a message',
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 385ba0a9..4099b26c 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -14,6 +14,7 @@ sub prep_ext { # externals_each callback
sub _start_query { # used by "lei q" and "lei up"
my ($self) = @_;
+ require PublicInbox::LeiOverview;
PublicInbox::LeiOverview->new($self) or return;
my $opt = $self->{opt};
my ($xj, $mj) = split(/,/, $opt->{jobs} // '');
@@ -117,7 +118,6 @@ sub lxs_prepare {
# the main "lei q SEARCH_TERMS" method
sub lei_q {
my ($self, @argv) = @_;
- require PublicInbox::LeiOverview;
PublicInbox::Config->json; # preload before forking
my $lxs = lxs_prepare($self) or return;
$self->ale->refresh_externals($lxs);
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index e80ccf57..0fb9698b 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -6,15 +6,14 @@ package PublicInbox::LeiUp;
use strict;
use v5.10.1;
use PublicInbox::LeiSavedSearch;
-use PublicInbox::LeiOverview;
+use parent qw(PublicInbox::IPC);
-sub lei_up {
+sub up1 ($$) {
my ($lei, $out) = @_;
- $lei->{lse} = $lei->_lei_store(1)->search;
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
+ my $f = $lss->{'-f'};
my $mset_opt = $lei->{mset_opt} = { relevance => -2 };
$mset_opt->{limit} = $lei->{opt}->{limit} // 10000;
- my $f = $lss->{'-f'};
my $q = $mset_opt->{q_raw} = $lss->{-cfg}->{'lei.q'} //
return $lei->fail("lei.q unset in $f");
my $lse = $lei->{lse} // die 'BUG: {lse} missing';
@@ -40,10 +39,60 @@ sub lei_up {
$lei->{lss} = $lss; # for LeiOverview->new
my $lxs = $lei->lxs_prepare or return;
$lei->ale->refresh_externals($lxs);
- $lei->{opt}->{save} = -1;
$lei->_start_query;
}
+sub up1_redispatch {
+ my ($lei, $out, $op_p) = @_;
+ my $l = bless { %$lei }, ref($lei);
+ $l->{opt} = { %{$l->{opt}} };
+ delete $l->{sock};
+ $l->{''} = $op_p; # daemon only
+ eval {
+ $l->qerr("# updating $out");
+ up1($l, $out);
+ $l->qerr("# $out done");
+ };
+ $l->err($@) if $@;
+}
+
+sub lei_up {
+ my ($lei, $out) = @_;
+ $lei->{lse} = $lei->_lei_store(1)->search;
+ my $opt = $lei->{opt};
+ $opt->{save} = -1;
+ if (defined $opt->{all}) {
+ length($opt->{mua}//'') and return
+ $lei->fail('--all and --mua= are incompatible');
+
+ # supporting IMAP outputs is more involved due to
+ # git-credential prompts. TODO: add this in 1.8
+ $opt->{all} eq 'local' or return
+ $lei->fail('only --all=local works at the moment');
+ my @all = PublicInbox::LeiSavedSearch::list($lei);
+ my @local = grep(!m!\Aimaps?://!i, @all);
+ $lei->_lei_store->write_prepare($lei); # share early
+ if ($lei->{oneshot}) { # synchronous
+ up1_redispatch($lei, $_) for @local;
+ } else {
+ # daemon mode, re-dispatch into our event loop w/o
+ # creating an extra fork-level
+ require PublicInbox::DS;
+ require PublicInbox::PktOp;
+ my ($op_c, $op_p) = PublicInbox::PktOp->pair;
+ for my $o (@local) {
+ PublicInbox::DS::requeue(sub {
+ up1_redispatch($lei, $o, $op_p);
+ });
+ }
+ $lei->event_step_init;
+ $op_c->{ops} = { '' => [$lei->can('dclose'), $lei] };
+ }
+ } else {
+ up1($lei, $out);
+ }
+}
+
sub _complete_up {
my ($lei, @argv) = @_;
my ($cur, $re) = $lei->complete_url_common(\@argv);
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index c0c74581..58342171 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -108,5 +108,7 @@ test_lei(sub {
$mb = do { local $/; <$mb> };
like($mb, qr/pre-existing/, 'pre-existing message preserved w/ -a');
like($mb, qr/<qp\@example\.com>/, 'new result written w/ -a');
+
+ lei_ok(qw(up --all=local));
});
done_testing;
^ permalink raw reply related [relevance 49%]
* [PATCH 0/2] lei {edit,forget}-search
@ 2021-04-20 7:16 71% Eric Wong
2021-04-20 7:16 57% ` [PATCH 1/2] lei forget-search: new command to forget saved searches Eric Wong
2021-04-20 7:16 64% ` [PATCH 2/2] lei edit-search: command to tweak search parameters Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-20 7:16 UTC (permalink / raw)
To: meta
Some convenience commands, since I expect users will want to
tweak and experiment with things.
I don't think "mv-search" (formerly "mv-query") is necessary
since it's easy enough to use "lei convert" (testing anything
requiring authentication is not fun nor easily automated, atm).
Eric Wong (2):
lei forget-search: new command to forget saved searches
lei edit-search: command to tweak search parameters
MANIFEST | 2 ++
lib/PublicInbox/LEI.pm | 6 ++++--
lib/PublicInbox/LeiEditSearch.pm | 25 +++++++++++++++++++++++
lib/PublicInbox/LeiForgetSearch.pm | 32 ++++++++++++++++++++++++++++++
t/lei-q-save.t | 11 ++++++++++
5 files changed, 74 insertions(+), 2 deletions(-)
create mode 100644 lib/PublicInbox/LeiEditSearch.pm
create mode 100644 lib/PublicInbox/LeiForgetSearch.pm
^ permalink raw reply [relevance 71%]
* [PATCH 2/2] lei edit-search: command to tweak search parameters
2021-04-20 7:16 71% [PATCH 0/2] lei {edit,forget}-search Eric Wong
2021-04-20 7:16 57% ` [PATCH 1/2] lei forget-search: new command to forget saved searches Eric Wong
@ 2021-04-20 7:16 64% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-20 7:16 UTC (permalink / raw)
To: meta
This may be useful for users to tweak search parameters.
This command is also the reason lei.saved-search is a git-config
file rather than JSON.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiEditSearch.pm | 25 +++++++++++++++++++++++++
3 files changed, 28 insertions(+)
create mode 100644 lib/PublicInbox/LeiEditSearch.pm
diff --git a/MANIFEST b/MANIFEST
index d4055af4..197da2c0 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -190,6 +190,7 @@ lib/PublicInbox/LeiBlob.pm
lib/PublicInbox/LeiConvert.pm
lib/PublicInbox/LeiCurl.pm
lib/PublicInbox/LeiDedupe.pm
+lib/PublicInbox/LeiEditSearch.pm
lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiForgetSearch.pm
lib/PublicInbox/LeiHelp.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index c3c79631..8fa89944 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -162,6 +162,8 @@ our %CMD = ( # sorted in order of importance/use:
qw(format|f=s pretty l ascii z|0), @c_opt ],
'forget-search' => [ 'OUTPUT', 'forget a saved search',
qw(verbose|v+), @c_opt ],
+'edit-search' => [ 'OUTPUT', "edit saved search via `git config --edit'",
+ @c_opt ],
'plonk' => [ '--threads|--from=IDENT',
'exclude mail matching From: or threads from non-Message-ID searches',
diff --git a/lib/PublicInbox/LeiEditSearch.pm b/lib/PublicInbox/LeiEditSearch.pm
new file mode 100644
index 00000000..fb36fdcd
--- /dev/null
+++ b/lib/PublicInbox/LeiEditSearch.pm
@@ -0,0 +1,25 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei edit-search" edit a saved search following "lei q --save"
+package PublicInbox::LeiEditSearch;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use PublicInbox::LeiUp;
+
+sub lei_edit_search {
+ my ($lei, $out) = @_;
+ my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
+ my @cmd = (qw(git config --edit -f), $lss->{'-f'});
+ $lei->qerr("# spawning @cmd");
+ if ($lei->{oneshot}) {
+ exec(@cmd) or die "exec @cmd: $!\n";
+ } else {
+ $lei->send_exec_cmd([], \@cmd, {});
+ }
+}
+
+*_complete_edit_search = \&PublicInbox::LeiUp::_complete_up;
+
+1;
^ permalink raw reply related [relevance 64%]
* [PATCH 1/2] lei forget-search: new command to forget saved searches
2021-04-20 7:16 71% [PATCH 0/2] lei {edit,forget}-search Eric Wong
@ 2021-04-20 7:16 57% ` Eric Wong
2021-04-20 7:16 64% ` [PATCH 2/2] lei edit-search: command to tweak search parameters Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-20 7:16 UTC (permalink / raw)
To: meta
Readers may lose interest in subscription topics. This lets
them avoid clutter by forgetting a saved search.
This does not and will not destroy the contents of an --output
mailbox. In other words, this is similar to unsubscribing
from an Atom/RSS feed or NNTP group.
I've also decided we won't support 'mv-search', since it'll
probably be rarely used and "lei convert" can be used, instead.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 4 ++--
lib/PublicInbox/LeiForgetSearch.pm | 32 ++++++++++++++++++++++++++++++
t/lei-q-save.t | 11 ++++++++++
4 files changed, 46 insertions(+), 2 deletions(-)
create mode 100644 lib/PublicInbox/LeiForgetSearch.pm
diff --git a/MANIFEST b/MANIFEST
index f35c514c..d4055af4 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -191,6 +191,7 @@ lib/PublicInbox/LeiConvert.pm
lib/PublicInbox/LeiCurl.pm
lib/PublicInbox/LeiDedupe.pm
lib/PublicInbox/LeiExternal.pm
+lib/PublicInbox/LeiForgetSearch.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInit.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index c0385eb5..c3c79631 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -160,8 +160,8 @@ our %CMD = ( # sorted in order of importance/use:
'ls-search' => [ '[PREFIX]', 'list saved search queries',
qw(format|f=s pretty l ascii z|0), @c_opt ],
-'rm-query' => [ 'QUERY_NAME', 'remove a saved search', @c_opt ],
-'mv-query' => [ qw(OLD_NAME NEW_NAME), 'rename a saved search', @c_opt ],
+'forget-search' => [ 'OUTPUT', 'forget a saved search',
+ qw(verbose|v+), @c_opt ],
'plonk' => [ '--threads|--from=IDENT',
'exclude mail matching From: or threads from non-Message-ID searches',
diff --git a/lib/PublicInbox/LeiForgetSearch.pm b/lib/PublicInbox/LeiForgetSearch.pm
new file mode 100644
index 00000000..b5fe5fb1
--- /dev/null
+++ b/lib/PublicInbox/LeiForgetSearch.pm
@@ -0,0 +1,32 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei forget-search" forget/remove a saved search "lei q --save"
+package PublicInbox::LeiForgetSearch;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiSavedSearch;
+use PublicInbox::LeiUp;
+use File::Path ();
+use SelectSaver;
+
+sub lei_forget_search {
+ my ($lei, $out) = @_;
+ my $d = PublicInbox::LeiSavedSearch::lss_dir_for($lei, \$out);
+ if (-e $d) {
+ my $save;
+ my $opt = { safe => 1 };
+ if ($lei->{opt}->{verbose}) {
+ $opt->{verbose} = 1;
+ $save = SelectSaver->new($lei->{2});
+ }
+ File::Path::remove_tree($d, $opt);
+ } else {
+ $lei->fail("--save was not used with $out cwd=".
+ $lei->rel2abs('.'));
+ }
+}
+
+*_complete_forget_search = \&PublicInbox::LeiUp::_complete_up;
+
+1;
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 58342171..5a2f7fff 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -110,5 +110,16 @@ test_lei(sub {
like($mb, qr/<qp\@example\.com>/, 'new result written w/ -a');
lei_ok(qw(up --all=local));
+
+ ok(!lei(qw(forget-search), "$home/bogus"), 'bogus forget');
+ lei_ok qw(_complete lei forget-search);
+ like($lei_out, qr/mbrd-aug/, 'forget-search completion');
+ lei_ok(qw(forget-search -v), "$home/mbrd-aug");
+ is($lei_out, '', 'no output');
+ like($lei_err, qr/\bmbrd-aug\b/, '-v (verbose) reported unlinks');
+ lei_ok qw(_complete lei forget-search);
+ unlike($lei_out, qr/mbrd-aug/,
+ 'forget-search completion cleared after forget');
+ ok(!lei('up', "$home/mbrd-aug"), 'lei up fails after forget');
});
done_testing;
^ permalink raw reply related [relevance 57%]
* [PATCH] lei-sigpipe: update and move test from xt => t
@ 2021-04-20 9:17 52% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-20 9:17 UTC (permalink / raw)
To: meta
We have "lei import" and better test infrastructure for lei,
now, so we can more easily test SIGPIPE without relying on
an already-configured instance.
---
MANIFEST | 2 +-
t/lei-sigpipe.t | 43 +++++++++++++++++++++++++++++++++++
xt/lei-sigpipe.t | 58 ------------------------------------------------
3 files changed, 44 insertions(+), 59 deletions(-)
create mode 100644 t/lei-sigpipe.t
delete mode 100644 xt/lei-sigpipe.t
diff --git a/MANIFEST b/MANIFEST
index f35c514c..f4a55687 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -398,6 +398,7 @@ t/lei-q-kw.t
t/lei-q-remote-import.t
t/lei-q-save.t
t/lei-q-thread.t
+t/lei-sigpipe.t
t/lei-tag.t
t/lei.t
t/lei_dedupe.t
@@ -501,7 +502,6 @@ xt/httpd-async-stream.t
xt/imapd-mbsync-oimap.t
xt/imapd-validate.t
xt/lei-auth-fail.t
-xt/lei-sigpipe.t
xt/mem-imapd-tls.t
xt/mem-msgview.t
xt/msgtime_cmp.t
diff --git a/t/lei-sigpipe.t b/t/lei-sigpipe.t
new file mode 100644
index 00000000..f84d6d22
--- /dev/null
+++ b/t/lei-sigpipe.t
@@ -0,0 +1,43 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict;
+use v5.10.1;
+use PublicInbox::TestCommon;
+use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
+test_lei(sub {
+ my $f = "$ENV{HOME}/big.eml";
+ my $imported;
+ for my $out ([], [qw(-f mboxcl2)]) {
+ pipe(my ($r, $w)) or BAIL_OUT $!;
+ my $size = 65536;
+ if ($^O eq 'linux' && fcntl($w, 1031, 4096)) {
+ $size = 4096;
+ }
+ unless (-f $f) {
+ open my $fh, '>', $f or xbail "open $f: $!";
+ print $fh <<'EOM' or xbail;
+From: big@example.com
+Message-ID: <big@example.com>
+EOM
+ print $fh 'Subject:';
+ print $fh (' '.('x' x 72)."\n") x (($size / 73) + 1);
+ print $fh "\nbody\n";
+ close $fh or xbail "close: $!";
+ }
+
+ lei_ok(qw(import), $f) if $imported++ == 0;
+ open my $errfh, '>>', "$ENV{HOME}/stderr.log" or xbail $!;
+ my $opt = { run_mode => 0, 2 => $errfh, 1 => $w };
+ my $cmd = [qw(lei q -q -t), @$out, 'z:1..'];
+ my $tp = start_script($cmd, undef, $opt);
+ close $w;
+ is(sysread($r, my $buf, 1), 1, 'read one byte');
+ close $r; # trigger SIGPIPE
+ $tp->join;
+ ok(WIFSIGNALED($?), "signaled @$out");
+ is(WTERMSIG($?), SIGPIPE, "got SIGPIPE @$out");
+ }
+});
+
+done_testing;
diff --git a/xt/lei-sigpipe.t b/xt/lei-sigpipe.t
deleted file mode 100644
index 44020bad..00000000
--- a/xt/lei-sigpipe.t
+++ /dev/null
@@ -1,58 +0,0 @@
-#!perl -w
-# Copyright (C) 2021 all contributors <meta@public-inbox.org>
-# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-use strict;
-use v5.10.1;
-use Test::More;
-use PublicInbox::TestCommon;
-use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
-require_mods(qw(json DBD::SQLite Search::Xapian));
-# XXX this needs an already configured lei instance with many messages
-
-my $do_test = sub {
- my $env = shift // {};
- for my $out ([], [qw(-f mboxcl2)]) {
- pipe(my ($r, $w)) or BAIL_OUT $!;
- open my $err, '+>', undef or BAIL_OUT $!;
- my $opt = { run_mode => 0, 1 => $w, 2 => $err };
- my $cmd = [qw(lei q -q -t), @$out, 'z:1..'];
- my $tp = start_script($cmd, $env, $opt);
- close $w;
- sysread($r, my $buf, 1);
- close $r; # trigger SIGPIPE
- $tp->join;
- ok(WIFSIGNALED($?), "signaled @$out");
- is(WTERMSIG($?), SIGPIPE, "got SIGPIPE @$out");
- seek($err, 0, 0);
- my @err = grep(!m{mkdir /dev/null\b}, <$err>);
- is_deeply(\@err, [], "no errors @$out");
- }
-};
-
-my ($tmp, $for_destroy) = tmpdir();
-my $pid;
-my $opt = { run_mode => 0, 1 => \(my $out = '') };
-if (run_script([qw(lei daemon-pid)], undef, $opt)) {
- chomp($pid = $out);
- mkdir "$tmp/d" or BAIL_OUT $!;
- local $ENV{TMPDIR} = "$tmp/d";
- $do_test->();
- $out = '';
- ok(run_script([qw(lei daemon-pid)], undef, $opt), 'daemon-pid again');
- chomp($out);
- is($out, $pid, 'daemon-pid unchanged');
- ok(kill(0, $pid), 'daemon still running');
- $out = '';
-}
-{
- mkdir "$tmp/1" or BAIL_OUT $!;
- local $ENV{TMPDIR} = "$tmp/1";
- $do_test->({XDG_RUNTIME_DIR => '/dev/null'});
- is(unlink(glob("$tmp/1/*")), 0, 'nothing left over w/ oneshot');
-}
-
-# the one-shot test should be slow enough that the daemon has cleaned
-# up in the background:
-is_deeply([glob("$tmp/d/*")], [], 'nothing left over with daemon');
-
-done_testing;
^ permalink raw reply related [relevance 52%]
* t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set
@ 2021-04-20 20:33 71% Konstantin Ryabitsev
2021-04-20 20:38 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-04-20 20:33 UTC (permalink / raw)
To: meta
While playing with libgit2/Gcf2, I've discovered that t/lei-daemon.t will fail
when PERL_INLINE_DIRECTORY is set:
t/lei-daemon.t ............... 18/?
# Failed test 'connect error noted'
# at t/lei-daemon.t line 80.
# ''
# doesn't match '(?^:\bconnect\()'
# Looks like you failed 1 test of 34.
t/lei-daemon.t ............... Dubious, test returned 1 (wstat 256, 0x100)
I'll be happy to help troubleshoot things, since I don't know how easy it
would be to reproduce this.
-K
^ permalink raw reply [relevance 71%]
* Re: t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set
2021-04-20 20:33 71% t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set Konstantin Ryabitsev
@ 2021-04-20 20:38 71% ` Eric Wong
2021-04-20 21:37 68% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-20 20:38 UTC (permalink / raw)
To: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> While playing with libgit2/Gcf2, I've discovered that t/lei-daemon.t will fail
> when PERL_INLINE_DIRECTORY is set:
>
> t/lei-daemon.t ............... 18/?
> # Failed test 'connect error noted'
> # at t/lei-daemon.t line 80.
> # ''
> # doesn't match '(?^:\bconnect\()'
> # Looks like you failed 1 test of 34.
> t/lei-daemon.t ............... Dubious, test returned 1 (wstat 256, 0x100)
>
> I'll be happy to help troubleshoot things, since I don't know how easy it
> would be to reproduce this.
I can't seem to reproduce it on my end with CentOS 7 and
libgit2-devel 0.26.8. Try setting TEST_LEI_ERR_LOUD=1
^ permalink raw reply [relevance 71%]
* Re: t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set
2021-04-20 20:38 71% ` Eric Wong
@ 2021-04-20 21:37 68% ` Konstantin Ryabitsev
2021-04-20 22:06 71% ` [PATCH] t/lei-daemon: skip inaccessible socket test as root Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-04-20 21:37 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Tue, Apr 20, 2021 at 08:38:54PM +0000, Eric Wong wrote:
> > t/lei-daemon.t ............... 18/?
> > # Failed test 'connect error noted'
> > # at t/lei-daemon.t line 80.
> > # ''
> > # doesn't match '(?^:\bconnect\()'
> > # Looks like you failed 1 test of 34.
> > t/lei-daemon.t ............... Dubious, test returned 1 (wstat 256, 0x100)
> >
> > I'll be happy to help troubleshoot things, since I don't know how easy it
> > would be to reproduce this.
>
> I can't seem to reproduce it on my end with CentOS 7 and
> libgit2-devel 0.26.8. Try setting TEST_LEI_ERR_LOUD=1
While poking around, I've discovered that it only fails when "make test" runs
as root (don't judge -- this is in a throwaway lab VM):
t/lei-daemon.t ............... 1/? # lei_err=/tmp/pi-lei-daemon-6667-ojwj/lei-daemon/xdg_run/lei/errors.log from previous run
# phail
# Failed test 'connect error noted'
# at t/lei-daemon.t line 80.
# ''
# doesn't match '(?^:\bconnect\()'
# Looks like you failed 1 test of 34.
t/lei-daemon.t ............... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/34 subtests
When I run as a non-root user, it passes:
t/lei-daemon.t ............... 1/? # lei_err=/tmp/pi-lei-daemon-11879-P3hC/lei-daemon/xdg_run/lei/errors.log from previous run
# phail
# lei_err=connect(/tmp/pi-lei-daemon-11879-P3hC/lei-daemon/xdg_run/lei/5.seq.sock): Permission denied at /home/mricon/public-inbox/blib/lib/PublicInbox/LEI.pm line 1073.
# lei-daemon could not start, exited with $?=3328
# connect(/tmp/pi-lei-daemon-11879-P3hC/lei-daemon/xdg_run/lei/5.seq.sock): Permission denied (after attempted daemon start)
# Falling back to (slow) one-shot mode
t/lei-daemon.t ............... ok
Hope that helps.
-K
^ permalink raw reply [relevance 68%]
* [PATCH] t/lei-daemon: skip inaccessible socket test as root
2021-04-20 21:37 68% ` Konstantin Ryabitsev
@ 2021-04-20 22:06 71% ` Eric Wong
2021-04-21 15:03 71% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-20 22:06 UTC (permalink / raw)
To: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> While poking around, I've discovered that it only fails when "make test" runs
> as root (don't judge -- this is in a throwaway lab VM):
> Hope that helps.
Yes :)
----------8<--------
Subject: [PATCH] t/lei-daemon: skip inaccessible socket test as root
"chmod 0000" on a Unix socket can't stop root from connecting to it;
so just skip the test for rare cases when testing as root.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210420213712.qfpftr2r543cqg7l@nitro.local/
---
t/lei-daemon.t | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/t/lei-daemon.t b/t/lei-daemon.t
index 35e059b9..84e2791d 100644
--- a/t/lei-daemon.t
+++ b/t/lei-daemon.t
@@ -74,7 +74,8 @@ test_lei({ daemon_only => 1 }, sub {
chomp $lei_out;
is($lei_out, $new_pid, 'PID unchanged after -0/-CHLD');
- if ('socket inaccessible') {
+ SKIP: { # socket inaccessible
+ skip "cannot test connect EPERM as root", 3 if $> == 0;
chmod 0000, $sock or BAIL_OUT "chmod 0000: $!";
lei_ok('help', \'connect fail, one-shot fallback works');
like($lei_err, qr/\bconnect\(/, 'connect error noted');
^ permalink raw reply related [relevance 71%]
* [PATCH] doc: add lei_design_notes.txt and lei-store-format(5)
@ 2021-04-21 10:03 46% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-21 10:03 UTC (permalink / raw)
To: meta
lei itself is a somewhat weird design, but lei/store is
a fairly normal hybrid of extindex with v2-style storage.
---
Documentation/lei-store-format.pod | 91 ++++++++++++++++++++++++++++++
Documentation/lei_design_notes.txt | 20 +++++++
MANIFEST | 2 +
Makefile.PL | 2 +-
4 files changed, 114 insertions(+), 1 deletion(-)
create mode 100644 Documentation/lei-store-format.pod
create mode 100644 Documentation/lei_design_notes.txt
diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod
new file mode 100644
index 00000000..a42c770e
--- /dev/null
+++ b/Documentation/lei-store-format.pod
@@ -0,0 +1,91 @@
+% public-inbox developer manual
+
+=head1 NAME
+
+lei-store-format - lei/store format description
+
+=head1 DESCRIPTION
+
+C<lei/store> is a hybrid store based on L<public-inbox-extindex-format(5)>
+("extindex") combined with L<public-inbox-v2-format(5)> ("v2") for blob
+storage. While v2 is ideal for archiving a single public mailing list;
+it was never intended for personal mail nor storing multiple
+blobs of the "same" message.
+
+As with extindex, it can index disparate C<List-Id> headers
+belonging to the "same" message with different git blob OIDs.
+Unlike v2 and extindex, C<Message-ID> headers are NOT required;
+allowing unsent draft messages to be stored and indexed.
+
+=head1 DIRECTORY LAYOUT
+
+Blob storage exists in the form of v2-style epochs. These epochs
+are under the C<local/> directory (instead of C<git/>) to
+prevent them from being accidentally treated as a v2 inbox.
+
+=head2 INDEX OVERVIEW AND DEFINITIONS
+
+ $EPOCH - Integer starting with 0 based on time
+ $SCHEMA_VERSION - DB schema version (for Xapian)
+ $SHARD - Integer starting with 0 based on parallelism
+
+ ~/.local/share/lei/store
+ - ipc.lock # lock file for internal lei IPC
+ - local/$EPOCH.git # normal bare git repositories
+
+Additionally, the following share the same roles they do in extindex:
+
+ - ei.lock # lock file to protect global state
+ - ALL.git # empty, alternates for local/*.git
+ - ei$SCHEMA_VERSION/$SHARD # per-shard Xapian DB
+ - ei$SCHEMA_VERSION/over.sqlite3 # overview DB for WWW, IMAP
+ - ei$SCHEMA_VERSION/misc # misc Xapian DB
+
+=head2 XREF3 DEDUPLICATION
+
+Index deduplication follows extindex, see
+L<public-inbox-extindex-format(5)/XREF3 DEDUPLICATION> for
+more information.
+
+=head2 BLOB DEDUPLICATION
+
+The contents of C<local/*.git> repos is deduplicated by git blob
+object IDs (currently SHA-1). This allows multiple copies of
+cross-posted and personally Cc-ed messages to be stored with
+different C<Received:>, C<X-Spam-Status:> and similar headers to
+allow troubleshooting.
+
+=head2 VOLATILE METADATA
+
+Keywords and label information (as described in RFC 8621 for JMAP)
+is stored in existing Xapian shards (C<ei$SCHEMA_VERSION/$SHARD>).
+It is possible to search for messages matching labels and
+keywords using C<L:> and C<kw:>, respectively. As with all data
+stored in Xapian indices, volatile metadata is associated with
+the Xapian document, thus it is shared across different blobs of
+the "same" message.
+
+=head1 IPC
+
+When L<lei(1)> is run in daemon mode, L<flock(2)> is used on
+C<ipc.lock> is used to serialize writes to C<lei/store> across
+multiple internal lei workers while minimizing commits.
+
+=head1 CAVEATS
+
+Reindexing and synchronization is not yet supported.
+
+=head1 THANKS
+
+Thanks to the Linux Foundation for sponsoring the development
+and testing.
+
+=head1 COPYRIGHT
+
+Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<http://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<public-inbox-v2-format(5)>, L<public-inbox-extindex(5)>
diff --git a/Documentation/lei_design_notes.txt b/Documentation/lei_design_notes.txt
new file mode 100644
index 00000000..a5606c05
--- /dev/null
+++ b/Documentation/lei_design_notes.txt
@@ -0,0 +1,20 @@
+lei design notes
+----------------
+
+Daemon architecture
+-------------------
+
+The use of a persistent daemon works around slow startup time of
+Perl. This is especially important for built-in support for
+shell completion. It will eventually support inotify and EVFILT_VNODE
+background monitoring of Maildir keyword changes.
+
+If lei were reimplemented in a language with faster startup
+time, the daemon architecture would likely remain since it also
+lets us easily decouple the local storage from slow IMAP/NNTP
+backends and allow us to serialize writes to git-fast-import,
+SQLite, and Xapian across multiple processes.
+
+The coupling of IMAP and NNTP network latency to local storage
+is a current weakness of public-inbox-watch. Therefore, -watch
+will likely adopt the daemon architecture of lei in the future.
diff --git a/MANIFEST b/MANIFEST
index 1a1d72a6..e0f9c35b 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -35,8 +35,10 @@ Documentation/lei-mail-formats.pod
Documentation/lei-overview.pod
Documentation/lei-p2q.pod
Documentation/lei-q.pod
+Documentation/lei-store-format.pod
Documentation/lei-tag.pod
Documentation/lei.pod
+Documentation/lei_design_notes.txt
Documentation/marketing.txt
Documentation/mknews.perl
Documentation/public-inbox-compact.pod
diff --git a/Makefile.PL b/Makefile.PL
index 129b082d..85b18e7d 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -50,7 +50,7 @@ $v->{-m1} = [ map {
lei-tag lei-p2q lei-q)];
$v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format
public-inbox-v2-format public-inbox-extindex-format
- lei-mail-formats
+ lei-mail-formats lei-store-format
) ];
$v->{-m7} = [ qw(lei-overview public-inbox-overview public-inbox-tuning
public-inbox-glossary) ];
^ permalink raw reply related [relevance 46%]
* Re: [PATCH] t/lei-daemon: skip inaccessible socket test as root
2021-04-20 22:06 71% ` [PATCH] t/lei-daemon: skip inaccessible socket test as root Eric Wong
@ 2021-04-21 15:03 71% ` Konstantin Ryabitsev
0 siblings, 0 replies; 200+ results
From: Konstantin Ryabitsev @ 2021-04-21 15:03 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Tue, Apr 20, 2021 at 10:06:08PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > While poking around, I've discovered that it only fails when "make test" runs
> > as root (don't judge -- this is in a throwaway lab VM):
>
> > Hope that helps.
>
> Yes :)
I can confirm that the error is gone now. Thanks!
-K
^ permalink raw reply [relevance 71%]
* [PATCH] lei: share common *done_wait callbacks
@ 2021-04-21 18:36 61% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-21 18:36 UTC (permalink / raw)
To: meta
Code is the enemy, and there's no need to duplicate things, here.
There may be further opportunities along these lines to further
deduplicate things...
---
lib/PublicInbox/LEI.pm | 7 +++++++
lib/PublicInbox/LeiBlob.pm | 9 +--------
lib/PublicInbox/LeiImport.pm | 9 +--------
lib/PublicInbox/LeiTag.pm | 9 +--------
4 files changed, 10 insertions(+), 24 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8fa89944..2e1aa246 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -1211,4 +1211,11 @@ sub DESTROY {
$? = $err if $err; # preserve ->fail or ->x_it code
}
+sub wq_done_wait { # dwaitpid callback
+ my ($arg, $pid) = @_;
+ my ($wq, $lei, $e) = @$arg;
+ $? and $lei->child_error($?, $e ? "$e errors during $lei->{cmd}" : ());
+ $lei->dclose;
+}
+
1;
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index ad885306..e4cd4cca 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -10,17 +10,10 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Spawn qw(spawn popen_rd which);
use PublicInbox::DS;
-sub sol_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my (undef, $lei) = @$arg;
- $lei->child_error($?) if $?;
- $lei->dclose;
-}
-
sub sol_done { # EOF callback for main daemon
my ($lei) = @_;
my $sol = delete $lei->{sol} // return $lei->dclose; # already failed
- $sol->wq_wait_old(\&sol_done_wait, $lei);
+ $sol->wq_wait_old($lei->can('wq_done_wait'), $lei);
}
sub get_git_dir ($$) {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index d33143ef..16271603 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -35,17 +35,10 @@ sub input_net_cb { # imap_each, nntp_each cb
input_eml_cb($self, $eml, $self->{-import_kw} ? { kw => $kw } : undef);
}
-sub import_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my ($imp, $lei) = @$arg;
- $lei->child_error($?, 'non-fatal errors during import') if $?;
- $lei->dclose;
-}
-
sub import_done { # EOF callback for main daemon
my ($lei) = @_;
my $imp = delete $lei->{imp} // return $lei->fail('BUG: {imp} gone');
- $imp->wq_wait_old(\&import_done_wait, $lei);
+ $imp->wq_wait_old($lei->can('wq_done_wait'), $lei, 'non-fatal');
}
sub net_merge_complete { # callback used by LeiAuth
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 1dfc841d..f019202f 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -71,17 +71,10 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub input_mbox_cb { input_eml_cb($_[1], $_[0]) }
-sub tag_done_wait { # dwaitpid callback
- my ($arg, $pid) = @_;
- my ($tag, $lei) = @$arg;
- $lei->child_error($?, 'non-fatal errors during tag') if $?;
- $lei->dclose;
-}
-
sub tag_done { # EOF callback for main daemon
my ($lei) = @_;
my $tag = delete $lei->{tag} or return;
- $tag->wq_wait_old(\&tag_done_wait, $lei);
+ $tag->wq_wait_old($lei->can('wq_done_wait'), $lei, 'non-fatal');
}
sub net_merge_complete { # callback used by LeiAuth
^ permalink raw reply related [relevance 61%]
* [PATCH] lei: flesh out `forwarded' kw support for Maildir and IMAP
@ 2021-04-21 23:50 50% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-21 23:50 UTC (permalink / raw)
To: meta
Maildir and IMAP can both handle `forwarded'. Ensure we don't
lose `forwarded' when reading from stores which do not support
it, but ensure we can set it when reading from IMAP and Maildir
stores.
---
lib/PublicInbox/LeiSearch.pm | 12 ++++++++++--
lib/PublicInbox/LeiToMail.pm | 2 +-
lib/PublicInbox/NetReader.pm | 1 +
t/lei-q-kw.t | 31 +++++++++++++++++++++++++++++++
xt/net_writer-imap.t | 14 +++++++++++---
5 files changed, 54 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index 082176e7..ff615d89 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -103,8 +103,16 @@ sub kw_changed {
my $xoids = xoids_for($self, $eml) // return;
$docids //= [];
@$docids = sort { $a <=> $b } values %$xoids;
- my @cur_kw = msg_keywords($self, $docids->[0]);
- join("\0", @$new_kw_sorted) eq join("\0", @cur_kw) ? 0 : 1;
+ my $cur_kw = msg_keywords($self, $docids->[0]);
+
+ # RFC 5550 sec 5.9 on the $Forwarded keyword states:
+ # "Once set, the flag SHOULD NOT be cleared"
+ if (exists($cur_kw->{forwarded}) &&
+ !grep(/\Aforwarded\z/, @$new_kw_sorted)) {
+ delete $cur_kw->{forwarded};
+ }
+ $cur_kw = join("\0", sort keys %$cur_kw);
+ join("\0", @$new_kw_sorted) eq $cur_kw ? 0 : 1;
}
sub all_terms {
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 46a82a4b..0fa0bd9a 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -38,7 +38,7 @@ sub _mbox_hdr_buf ($$$) {
if (my $ent = $kw2status{$k}) {
push @{$hdr{$ent->[0]}}, $ent->[1];
} else { # X-Label?
- warn "TODO: keyword `$k' not supported for mbox\n";
+ warn "# keyword `$k' not supported for mbox\n";
}
}
# Messages are always 'O' (non-\Recent in IMAP), it saves
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 821e5d7f..0ef66fd8 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -8,6 +8,7 @@ use v5.10.1;
use parent qw(Exporter PublicInbox::IPC);
use PublicInbox::Eml;
our %IMAPflags2kw = map {; "\\\u$_" => $_ } qw(seen answered flagged draft);
+$IMAPflags2kw{'$Forwarded'} = 'forwarded'; # RFC 5550
our @EXPORT = qw(uri_section imap_uri nntp_uri);
diff --git a/t/lei-q-kw.t b/t/lei-q-kw.t
index c17411fb..c00a0a43 100644
--- a/t/lei-q-kw.t
+++ b/t/lei-q-kw.t
@@ -205,5 +205,36 @@ open $fh, '<', \$lei_out or BAIL_OUT $!;
PublicInbox::MboxReader->mboxrd($fh, sub { push @another, shift });
is($another[0]->header('Status'), 'RO', 'seen kw set');
+# forwarded
+{
+ local $ENV{DBG} = 1;
+ $o = "$ENV{HOME}/forwarded";
+ lei_ok(qw(q -o), $o, "m:$m");
+ my @p = glob("$o/cur/*");
+ scalar(@p) == 1 or xbail('multiple when 1 expected', \@p);
+ my $passed = $p[0];
+ $passed =~ s/,S\z/,PS/ or xbail "failed to replace $passed";
+ rename($p[0], $passed) or xbail "rename $!";
+ lei_ok(qw(q -o), $o, 'm:bogus', \'clobber maildir');
+ is_deeply([glob("$o/cur/*")], [], 'old results clobbered');
+ lei_ok(qw(q -o), $o, "m:$m");
+ @p = glob("$o/cur/*");
+ scalar(@p) == 1 or xbail('multiple when 1 expected', \@p);
+ like($p[0], qr/,PS/, 'passed (Forwarded) flag kept');
+ lei_ok(qw(q -o), "mboxrd:$o.mboxrd", "m:$m");
+ open $fh, '<', "$o.mboxrd" or xbail $!;
+ my @res;
+ PublicInbox::MboxReader->mboxrd($fh, sub { push @res, shift });
+ scalar(@res) == 1 or xbail('multiple when 1 expected', \@res);
+ is($res[0]->header('Status'), 'RO', 'seen kw set');
+ is($res[0]->header('X-Status'), undef, 'no X-Status');
+
+ lei_ok(qw(q -o), "mboxrd:$o.mboxrd", 'bogus-for-import-before');
+ lei_ok(qw(q -o), $o, "m:$m");
+ @p = glob("$o/cur/*");
+ scalar(@p) == 1 or xbail('multiple when 1 expected', \@p);
+ like($p[0], qr/,PS/, 'passed (Forwarded) flag still kept');
+}
+
}); # test_lei
done_testing;
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index 11a10e74..007de35e 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -173,17 +173,18 @@ test_lei(sub {
is_deeply([@$res{qw(m kw)}], ['testmessage@example.com', ['seen']],
'kw set');
+ # prepare messages for watch
$mic = $nwr->mic_for_folder($folder_uri);
- for my $kw (qw(Deleted Seen Answered Draft)) {
+ for my $kw (qw(Deleted Seen Answered Draft forwarded)) {
my $buf = <<EOM;
From: x\@example.com
Message-ID: <$kw\@test.example.com>
EOM
- $mic->append_string($folder_uri->mailbox, $buf, "\\$kw")
+ my $f = $kw eq 'forwarded' ? '$Forwarded' : "\\$kw";
+ $mic->append_string($folder_uri->mailbox, $buf, $f)
or BAIL_OUT "append $kw $@";
}
- # $mic->expunge or BAIL_OUT "expunge: $@";
$mic->disconnect;
my $inboxdir = "$ENV{HOME}/wtest";
@@ -214,6 +215,13 @@ EOM
'-watch ignored \\Deleted');
ok(!defined($mm->num_for('Draft@test.example.com')),
'-watch ignored \\Draft');
+ ok(defined($mm->num_for('forwarded@test.example.com')),
+ '-watch takes forwarded message');
+ undef $w; # done with watch
+ lei_ok qw(import), $$folder_uri;
+ lei_ok qw(q m:forwarded@test.example.com);
+ is_deeply(json_utf8->decode($lei_out)->[0]->{kw}, ['forwarded'],
+ 'forwarded kw imported from IMAP');
});
undef $cleanup; # remove temporary folder
^ permalink raw reply related [relevance 50%]
* [PATCH 0/3] lei import: network sync things
@ 2021-04-22 9:08 71% Eric Wong
2021-04-22 9:08 70% ` [PATCH 1/3] imap_tracker: prepare for use with lei Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:08 UTC (permalink / raw)
To: meta
Once keyword synchronization exists; I'll be able to
retire my personal uses of offlineimap and mbsync...
Eric Wong (3):
imap_tracker: prepare for use with lei
lei import: --incremental default for NNTP and IMAP
lei import|convert: drop --no-kw aliases
Documentation/lei-import.pod | 2 +-
Documentation/lei-store-format.pod | 1 +
lib/PublicInbox/IMAPTracker.pm | 22 ++++++++++++++--------
lib/PublicInbox/LEI.pm | 9 +++++----
lib/PublicInbox/LeiImport.pm | 5 +++++
lib/PublicInbox/NetReader.pm | 13 +++++++++----
t/lei-import-imap.t | 3 +++
t/lei-import-nntp.t | 3 +++
t/lei.t | 3 +--
9 files changed, 42 insertions(+), 19 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/3] imap_tracker: prepare for use with lei
2021-04-22 9:08 71% [PATCH 0/3] lei import: network sync things Eric Wong
@ 2021-04-22 9:08 70% ` Eric Wong
2021-04-22 9:08 51% ` [PATCH 2/3] lei import: --incremental default for NNTP and IMAP Eric Wong
2021-04-22 9:08 59% ` [PATCH 3/3] lei import|convert: drop --no-kw aliases Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:08 UTC (permalink / raw)
To: meta
We'll support nodatacow as we do in other SQLite DBs
---
lib/PublicInbox/IMAPTracker.pm | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
index 6d4fb227..bcf7af2e 100644
--- a/lib/PublicInbox/IMAPTracker.pm
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -62,21 +62,27 @@ VALUES (?, ?, ?)
}
sub new {
- my ($class, $url) = @_;
+ my ($class, $url, $dbname) = @_;
- # original name for compatibility with old setups:
- my $dbname = PublicInbox::Config->config_dir() . "/imap.sqlite3";
+ unless (defined($dbname)) {
+ # original name for compatibility with old setups:
+ $dbname = PublicInbox::Config->config_dir() . '/imap.sqlite3';
- # use the new XDG-compliant name for new setups:
- if (!-f $dbname) {
- $dbname = ($ENV{XDG_DATA_HOME} //
- (($ENV{HOME} // '/nonexistent').'/.local/share')) .
- '/public-inbox/imap.sqlite3';
+ # use the new XDG-compliant name for new setups:
+ if (!-f $dbname) {
+ $dbname = ($ENV{XDG_DATA_HOME} //
+ (($ENV{HOME} // '/nonexistent').
+ '/.local/share')) .
+ '/public-inbox/imap.sqlite3';
+ }
}
if (!-f $dbname) {
require File::Path;
require File::Basename;
+ require PublicInbox::Spawn;
File::Path::mkpath(File::Basename::dirname($dbname));
+ open my $fh, '+>>', $dbname or die "failed to open $dbname: $!";
+ PublicInbox::Spawn::nodatacow_fd(fileno($fh));
}
my $self = bless { lock_path => "$dbname.lock", url => $url }, $class;
$self->lock_acquire;
^ permalink raw reply related [relevance 70%]
* [PATCH 3/3] lei import|convert: drop --no-kw aliases
2021-04-22 9:08 71% [PATCH 0/3] lei import: network sync things Eric Wong
2021-04-22 9:08 70% ` [PATCH 1/3] imap_tracker: prepare for use with lei Eric Wong
2021-04-22 9:08 51% ` [PATCH 2/3] lei import: --incremental default for NNTP and IMAP Eric Wong
@ 2021-04-22 9:08 59% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:08 UTC (permalink / raw)
To: meta
Supporting --no-keywords and --no-flags aliases is harmful
if users end up assuming "keywords:" and "flags:" are valid
search prefixes (they're not).
---
Documentation/lei-import.pod | 2 +-
lib/PublicInbox/LEI.pm | 9 ++++-----
t/lei.t | 3 +--
3 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/Documentation/lei-import.pod b/Documentation/lei-import.pod
index acc4f776..7d70191d 100644
--- a/Documentation/lei-import.pod
+++ b/Documentation/lei-import.pod
@@ -40,7 +40,7 @@ C<none>.
Default: fcntl,dotlock
-=item --no-kw, --no-keywords, --no-flags
+=item --no-kw
Don't import message keywords (or "flags" in IMAP terminology).
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d9e644eb..9f49fc03 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -181,7 +181,7 @@ our %CMD = ( # sorted in order of importance/use:
qw(exact! all jobs:i indexed), @c_opt ],
'add-watch' => [ 'LOCATION', 'watch for new messages and flag changes',
- qw(import! kw|keywords|flags! interval=s recursive|r
+ qw(import! kw! interval=s recursive|r
exclude=s include=s), @c_opt ],
'ls-watch' => [ '[FILTER...]', 'list active watches with numbers and status',
qw(format|f=s z), @c_opt ],
@@ -193,12 +193,11 @@ our %CMD = ( # sorted in order of importance/use:
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw|keywords|flags! verbose|v+
- incremental!), @c_opt ],
+ lock=s@ in-format|F=s kw! verbose|v+ incremental!), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
- lock=s@ kw|keywords|flags!), @c_opt ],
+ lock=s@ kw!), @c_opt ],
'p2q' => [ 'FILE|COMMIT_OID|--stdin',
"use a patch to generate a query for `lei q --stdin'",
qw(stdin| want|w=s@ uri debug), @c_opt ],
@@ -350,7 +349,7 @@ my %OPTDESC = (
'by-mid|mid:s' => [ 'MID', 'match only by Message-ID, ignoring contents' ],
-'kw|keywords|flags!' => 'disable/enable importing flags',
+'kw!' => 'disable/enable importing keywords (aka "flags")',
# xargs, env, use "-0", git(1) uses "-z". We support z|0 everywhere
'z|0' => 'use NUL \\0 instead of newline (CR) to delimit lines',
diff --git a/t/lei.t b/t/lei.t
index 6ade2f18..6d276050 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -131,8 +131,7 @@ my $test_completion = sub {
}
lei_ok(qw(_complete lei import));
%out = map { $_ => 1 } split(/\s+/s, $lei_out);
- for my $sw (qw(--flags --no-flags --no-kw --kw --no-keywords
- --keywords)) {
+ for my $sw (qw(--no-kw --kw)) {
ok($out{$sw}, "$sw offered as `lei import' completion");
}
};
^ permalink raw reply related [relevance 59%]
* [PATCH 2/3] lei import: --incremental default for NNTP and IMAP
2021-04-22 9:08 71% [PATCH 0/3] lei import: network sync things Eric Wong
2021-04-22 9:08 70% ` [PATCH 1/3] imap_tracker: prepare for use with lei Eric Wong
@ 2021-04-22 9:08 51% ` Eric Wong
2021-04-22 9:08 59% ` [PATCH 3/3] lei import|convert: drop --no-kw aliases Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:08 UTC (permalink / raw)
To: meta
No point in burning through bandwidth to import stuff we already
saw. All this logic is shared with -watch but uses a different
pathname for lei since it's tied to lei/store (and not a
public-inbox).
---
Documentation/lei-store-format.pod | 1 +
lib/PublicInbox/LEI.pm | 4 +++-
lib/PublicInbox/LeiImport.pm | 5 +++++
lib/PublicInbox/NetReader.pm | 13 +++++++++----
t/lei-import-imap.t | 3 +++
t/lei-import-nntp.t | 3 +++
6 files changed, 24 insertions(+), 5 deletions(-)
diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod
index a42c770e..3e1ddc65 100644
--- a/Documentation/lei-store-format.pod
+++ b/Documentation/lei-store-format.pod
@@ -32,6 +32,7 @@ prevent them from being accidentally treated as a v2 inbox.
~/.local/share/lei/store
- ipc.lock # lock file for internal lei IPC
- local/$EPOCH.git # normal bare git repositories
+ - net_last.sqlite3 # import state for IMAP & NNTP
Additionally, the following share the same roles they do in extindex:
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 2e1aa246..d9e644eb 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -193,7 +193,8 @@ our %CMD = ( # sorted in order of importance/use:
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw|keywords|flags! verbose|v+), @c_opt ],
+ lock=s@ in-format|F=s kw|keywords|flags! verbose|v+
+ incremental!), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
@@ -244,6 +245,7 @@ my %OPTDESC = (
'lock=s@' => [ 'METHOD|dotlock|fcntl|flock|none',
'mbox(5) locking method(s) to use (default: fcntl,dotlock)' ],
+'incremental! import' => 'import already seen IMAP and NNTP articles',
'globoff|g' => "do not match locations using '*?' wildcards ".
"and\xa0'[]'\x{a0}ranges",
'verbose|v+' => 'be more verbose',
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 16271603..accf08f5 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -58,6 +58,11 @@ sub lei_import { # the main "lei import" method
my $j = $lei->{opt}->{jobs} // scalar(@{$self->{inputs}}) || 1;
if (my $net = $lei->{net}) {
# $j = $net->net_concurrency($j); TODO
+ if ($lei->{opt}->{incremental} // 1) {
+ $net->{incremental} = 1;
+ $net->{itrk_fn} = $lei->store_path .
+ '/net_last.sqlite3';
+ }
} else {
my $nproc = $self->detect_nproc;
$j = $nproc if $j > $nproc;
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 0ef66fd8..c7b43f01 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -373,6 +373,13 @@ sub run_commit_cb ($) {
$cb->(@args);
}
+sub _itrk ($$) {
+ my ($self, $uri) = @_;
+ return unless $self->{incremental};
+ # itrk_fn is set by lei
+ PublicInbox::IMAPTracker->new($$uri, $self->{itrk_fn});
+}
+
sub _imap_fetch_all ($$$) {
my ($self, $mic, $uri) = @_;
my $sec = uri_section($uri);
@@ -389,8 +396,7 @@ sub _imap_fetch_all ($$$) {
return "E: $uri cannot get UIDVALIDITY";
$r_uidnext //= $mic->uidnext($mbx) //
return "E: $uri cannot get UIDNEXT";
- my $itrk = $self->{incremental} ?
- PublicInbox::IMAPTracker->new($$uri) : 0;
+ my $itrk = _itrk($self, $uri);
my ($l_uidval, $l_uid) = $itrk ? $itrk->get_last : ();
$l_uidval //= $r_uidval; # first time
$l_uid //= 0;
@@ -543,8 +549,7 @@ sub _nntp_fetch_all ($$$) {
# IMAPTracker is also used for tracking NNTP, UID == article number
# LIST.ACTIVE can get the equivalent of UIDVALIDITY, but that's
# expensive. So we assume newsgroups don't change:
- my $itrk = $self->{incremental} ?
- PublicInbox::IMAPTracker->new($$uri) : 0;
+ my $itrk = _itrk($self, $uri);
my (undef, $l_art) = $itrk ? $itrk->get_last : ();
# allow users to specify articles to refetch
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 7e4d44b9..490ea9be 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -24,5 +24,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
for (@$out) { $r{ref($_)}++ }
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
lei_ok([qw(tag +kw:seen), "imap://$host_port/t.v2.0"], undef, undef);
+
+ my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
+ ok(-s $f, 'net tracked for redundant imports');
});
done_testing;
diff --git a/t/lei-import-nntp.t b/t/lei-import-nntp.t
index 1fc6dbad..d795a86a 100644
--- a/t/lei-import-nntp.t
+++ b/t/lei-import-nntp.t
@@ -26,5 +26,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
my %r;
for (@$out) { $r{ref($_)}++ }
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
+
+ my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
+ ok(-s $f, 'net tracked for redundant imports');
});
done_testing;
^ permalink raw reply related [relevance 51%]
* [PATCH] lei: XDG_RUNTIME_DIR=/dev/null disables daemon mode
@ 2021-04-22 9:44 63% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-22 9:44 UTC (permalink / raw)
To: meta
We'll support this mode of operation for now to quiet down
testing of oneshot mode where the daemon doesn't persist.
---
lib/PublicInbox/TestCommon.pm | 9 ++-------
script/lei | 3 ++-
t/lei-externals.t | 2 +-
3 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 67fe6336..b5d0b9f8 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -468,7 +468,7 @@ sub have_xapian_compact () {
PublicInbox::Spawn::which($ENV{XAPIAN_COMPACT} || 'xapian-compact');
}
-our ($err_skip, $lei_opt, $lei_out, $lei_err);
+our ($lei_opt, $lei_out, $lei_err);
# favor lei() or lei_ok() over $lei for new code
sub lei (@) {
my ($cmd, $env, $xopt) = @_;
@@ -478,8 +478,6 @@ sub lei (@) {
$cmd = [ grep { defined && !ref } @_ ];
}
my $res = run_script(['lei', @$cmd], $env, $xopt // $lei_opt);
- $err_skip and
- $lei_err = join('', grep(!/$err_skip/, split(/^/m, $lei_err)));
if ($lei_err ne '') {
if ($lei_err =~ /Use of uninitialized/ ||
$lei_err =~ m!\bArgument .*? isn't numeric in !) {
@@ -570,10 +568,7 @@ EOM
my $home = "$tmpdir/lei-oneshot";
mkdir($home, 0700) or BAIL_OUT "mkdir: $!";
local $ENV{HOME} = $home;
- # force sun_path[108] overflow:
- my $xrd = "$home/1shot-test".('.sun_path' x 108);
- local $err_skip = qr!\Q$xrd!; # for lei() filtering
- local $ENV{XDG_RUNTIME_DIR} = $xrd;
+ local $ENV{XDG_RUNTIME_DIR} = '/dev/null';
$cb->();
}
if ($daemon_pid) {
diff --git a/script/lei b/script/lei
index 56e9d299..db302422 100755
--- a/script/lei
+++ b/script/lei
@@ -62,6 +62,7 @@ my $exec_cmd = sub {
if ($send_cmd && eval {
my $path = do {
my $runtime_dir = ($ENV{XDG_RUNTIME_DIR} // '') . '/lei';
+ die \0 if $runtime_dir eq '/dev/null/lei'; # oneshot forced
if ($runtime_dir eq '/lei') {
require File::Spec;
$runtime_dir = File::Spec->tmpdir."/lei-$<";
@@ -131,7 +132,7 @@ Falling back to (slow) one-shot mode
}
exit($x_it_code >> 8);
} else { # for systems lacking Socket::MsgHdr or Inline::C
- warn $@ if $@;
+ warn $@ if $@ && !ref($@);
require PublicInbox::LEI;
PublicInbox::LEI::oneshot(__PACKAGE__);
}
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 488bf5ad..2291dd99 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -49,7 +49,7 @@ SKIP: {
ok(WIFSIGNALED($?), "signaled @$out");
is(WTERMSIG($?), SIGPIPE, "got SIGPIPE @$out");
seek($err, 0, 0);
- my @err = grep(!m{mkdir .*sun_path\b}, <$err>);
+ my @err = <$err>;
is_deeply(\@err, [], "no errors @$out");
}
if (-d $ENV{XDG_RUNTIME_DIR} && -w _) {
^ permalink raw reply related [relevance 63%]
* [PATCH 0/2] "lei up" surprise reduction fixes
@ 2021-04-23 1:45 71% Eric Wong
2021-04-23 1:45 50% ` [PATCH 1/2] lei: saved searches support --dedupe=<mid|oid> Eric Wong
2021-04-23 1:45 55% ` [PATCH 2/2] lei up: support symlinked pathnames Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-23 1:45 UTC (permalink / raw)
To: meta
Eric Wong (2):
lei: saved searches support --dedupe=<mid|oid>
lei up: support symlinked pathnames
lib/PublicInbox/LeiSavedSearch.pm | 67 ++++++++++++++++++++++++++-----
lib/PublicInbox/LeiUp.pm | 4 ++
t/lei-q-save.t | 43 ++++++++++++++++++++
3 files changed, 104 insertions(+), 10 deletions(-)
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei: saved searches support --dedupe=<mid|oid>
2021-04-23 1:45 71% [PATCH 0/2] "lei up" surprise reduction fixes Eric Wong
@ 2021-04-23 1:45 50% ` Eric Wong
2021-04-23 1:45 55% ` [PATCH 2/2] lei up: support symlinked pathnames Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-23 1:45 UTC (permalink / raw)
To: meta
This is less surprising in case users are used to using --dedupe=
without --save.
---
lib/PublicInbox/LeiSavedSearch.pm | 27 ++++++++++++++++++++--
lib/PublicInbox/LeiUp.pm | 4 ++++
t/lei-q-save.t | 37 +++++++++++++++++++++++++++++++
3 files changed, 66 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index cd9effce..ed217cf2 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -11,6 +11,7 @@ use PublicInbox::LeiSearch;
use PublicInbox::Config;
use PublicInbox::Spawn qw(run_die);
use PublicInbox::ContentHash qw(git_sha);
+use PublicInbox::MID qw(mids_for_index);
use Digest::SHA qw(sha256_hex);
# move this to PublicInbox::Config if other things use it:
@@ -65,6 +66,14 @@ sub list {
} @$out
}
+sub translate_dedupe ($$$) {
+ my ($self, $lei, $dd) = @_;
+ $dd //= 'content';
+ return 1 if $dd eq 'content'; # the default
+ return $self->{"-dedupe_$dd"} = 1 if ($dd eq 'oid' || $dd eq 'mid');
+ $lei->fail("--dedupe=$dd unsupported with --save");
+}
+
sub up { # updating existing saved search via "lei up"
my ($cls, $lei, $dst) = @_;
my $f;
@@ -89,6 +98,8 @@ sub new { # new saved search "lei q --save"
File::Path::make_path($dir); # raises on error
$self->{-cfg} = {};
my $f = $self->{'-f'} = "$dir/lei.saved-search";
+ my $dd = $lei->{opt}->{dedupe};
+ translate_dedupe($self, $lei, $dd) or return;
open my $fh, '>', $f or return $lei->fail("open $f: $!");
my $sq_dst = PublicInbox::Config::squote_maybe($dst);
my $q = $lei->{mset_opt}->{q_raw} // die 'BUG: {q_raw} missing';
@@ -105,6 +116,7 @@ sub new { # new saved search "lei q --save"
[lei "q"]
output = $dst
EOM
+ print $fh "\tdedupe = $dd\n" if $dd;
for my $k (ARRAY_FIELDS) {
my $ary = $lei->{opt}->{$k} // next;
for my $x (@$ary) {
@@ -134,14 +146,25 @@ sub is_dup {
my ($self, $eml, $smsg) = @_;
my $oidx = $self->{oidx} // die 'BUG: no {oidx}';
my $blob = $smsg ? $smsg->{blob} : undef;
- return 1 if $blob && $oidx->blob_exists($blob);
my $lk = $self->lock_for_scope_fast;
+ return 1 if $blob && $oidx->blob_exists($blob);
+ if ($self->{-dedupe_mid}) {
+ for my $mid (@{mids_for_index($eml)}) {
+ my ($id, $prv);
+ return 1 if $oidx->next_by_mid($mid, \$id, \$prv);
+ }
+ }
if (my $xoids = PublicInbox::LeiSearch::xoids_for($self, $eml, 1)) {
for my $docid (values %$xoids) {
$oidx->add_xref3($docid, -1, $blob, '.');
}
$oidx->commit_lazy;
- 1;
+ if ($self->{-dedupe_oid}) {
+ $smsg->{blob} //= git_sha(1, $eml)->hexdigest;
+ exists $xoids->{$smsg->{blob}} ? 1 : undef;
+ } else {
+ 1;
+ }
} else {
# n.b. above xoids_for fills out eml->{-lei_fake_mid} if needed
unless ($smsg) {
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index 0fb9698b..f4ff070b 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -25,6 +25,10 @@ sub up1 ($$) {
my $o = $lei->{opt}->{output} = $lss->{-cfg}->{'lei.q.output'} //
return $lei->fail("lei.q.output unset in $f");
ref($o) and return $lei->fail("multiple values of lei.q.output in $f");
+ if (defined(my $dd = $lss->{-cfg}->{'lei.q.dedupe'})) {
+ $lss->translate_dedupe($lei, $dd) or return;
+ $lei->{opt}->{dedupe} = $dd;
+ }
for my $k (qw(only include exclude)) {
my $v = $lss->{-cfg}->get_all("lei.q.$k") // next;
$lei->{opt}->{$k} = $v;
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 5a2f7fff..26ea5cb8 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -121,5 +121,42 @@ test_lei(sub {
unlike($lei_out, qr/mbrd-aug/,
'forget-search completion cleared after forget');
ok(!lei('up', "$home/mbrd-aug"), 'lei up fails after forget');
+
+ # dedupe=mid
+ my $o = "$home/dd-mid";
+ $in = $doc2->as_string . "\n-------\nappended list sig\n";
+ lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
+ lei_ok(qw(q --dedupe=mid --save m:testmessage@example.com -o), $o);
+ my @m = glob("$o/cur/*");
+ is(scalar(@m), 1, '--dedupe=mid w/ --save');
+ $in = $doc2->as_string . "\n-------\nanother list sig\n";
+ lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
+ lei_ok 'up', $o;
+ is_deeply([glob("$o/cur/*")], \@m, 'lei up dedupe=mid works');
+
+ for my $dd (qw(content)) {
+ $o = "$home/dd-$dd";
+ lei_ok(qw(q --save m:testmessage@example.com -o), $o,
+ "--dedupe=$dd");
+ @m = glob("$o/cur/*");
+ is(scalar(@m), 3, 'all 3 matches with dedupe='.$dd);
+ }
+
+ # dedupe=oid
+ $o = "$home/dd-oid";
+ my $ibx = create_inbox 'ibx', indexlevel => 'medium',
+ tmpdir => "$home/v1", sub {};
+ lei_ok(qw(q --save --dedupe=oid m:qp@example.com -o), $o,
+ '-I', $ibx->{inboxdir});
+ @m = glob("$o/cur/*");
+ is(scalar(@m), 1, 'got first result');
+
+ my $im = $ibx->importer(0);
+ my $diff = "X-Insignificant-Header: x\n".$doc1->as_string;
+ $im->add(PublicInbox::Eml->new($diff));
+ $im->done;
+ lei_ok('up', $o);
+ @m = glob("$o/cur/*");
+ is(scalar(@m), 2, 'got 2nd result due to different OID');
});
done_testing;
^ permalink raw reply related [relevance 50%]
* [PATCH 2/2] lei up: support symlinked pathnames
2021-04-23 1:45 71% [PATCH 0/2] "lei up" surprise reduction fixes Eric Wong
2021-04-23 1:45 50% ` [PATCH 1/2] lei: saved searches support --dedupe=<mid|oid> Eric Wong
@ 2021-04-23 1:45 55% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-23 1:45 UTC (permalink / raw)
To: meta
On my default FreeBSD 11.x system, "/home" is a symlink to
"/usr/home", which causes "lei up" path resolution to fail when
I use outputs in $HOME. Fall back to a slow path of globbing
and matching pathnames based on st_ino+st_dev.
---
lib/PublicInbox/LeiSavedSearch.pm | 40 ++++++++++++++++++++++++-------
t/lei-q-save.t | 6 +++++
2 files changed, 38 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index ed217cf2..af864a50 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -13,6 +13,7 @@ use PublicInbox::Spawn qw(run_die);
use PublicInbox::ContentHash qw(git_sha);
use PublicInbox::MID qw(mids_for_index);
use Digest::SHA qw(sha256_hex);
+my $LOCAL_PFX = qr!\A(?:maildir|mh|mbox.+|mmdf):!i; # TODO: put in LeiToMail?
# move this to PublicInbox::Config if other things use it:
my %cquote = ("\n" => '\\n', "\t" => '\\t', "\b" => '\\b');
@@ -27,27 +28,50 @@ sub BOOL_FIELDS () {
qw(external local remote import-remote import-before threads)
}
-sub lss_dir_for ($$) {
- my ($lei, $dstref) = @_;
+sub lss_dir_for ($$;$) {
+ my ($lei, $dstref, $on_fs) = @_;
my @n;
if ($$dstref =~ m,\Aimaps?://,i) { # already canonicalized
require PublicInbox::URIimap;
my $uri = PublicInbox::URIimap->new($$dstref)->canonical;
$$dstref = $$uri;
@n = ($uri->mailbox);
- } else { # basename
+ } else {
+ # can't use Cwd::abs_path since dirname($$dstref) may not exist
$$dstref = $lei->rel2abs($$dstref);
+ # Maildirs have trailing '/' internally
$$dstref .= '/' if -d $$dstref;
$$dstref =~ tr!/!/!s;
- @n = ($$dstref =~ m{([^/]+)/*\z});
+ @n = ($$dstref =~ m{([^/]+)/*\z}); # basename
}
push @n, sha256_hex($$dstref);
- $lei->share_path . '/saved-searches/' . join('-', @n);
+ my $lss_dir = $lei->share_path . '/saved-searches/';
+ my $d = $lss_dir . join('-', @n);
+
+ # fall-back to looking up by st_ino + st_dev in case we're in
+ # a symlinked or bind-mounted path
+ if ($on_fs && !-d $d && -e $$dstref) {
+ my @cur = stat(_);
+ my $want = pack('dd', @cur[1,0]); # st_ino + st_dev
+ my ($c, $o, @st);
+ for my $g ("$n[0]-*", '*') {
+ my @maybe = glob("$lss_dir$g/lei.saved-search");
+ for my $f (@maybe) {
+ $c = PublicInbox::Config->git_config_dump($f);
+ $o = $c->{'lei.q.output'} // next;
+ $o =~ s!$LOCAL_PFX!! or next;
+ @st = stat($o) or next;
+ next if pack('dd', @st[1,0]) ne $want;
+ $f =~ m!\A(.+?)/[^/]+\z! and return $1;
+ }
+ }
+ }
+ $d;
}
sub list {
my ($lei, $pfx) = @_;
- my $lss_dir = $lei->share_path.'/saved-searches/';
+ my $lss_dir = $lei->share_path.'/saved-searches';
return () unless -d $lss_dir;
# TODO: persist the cache? Use another format?
my $f = $lei->cache_dir."/saved-tmp.$$.".time.'.config';
@@ -61,7 +85,7 @@ sub list {
unlink($f);
my $out = $cfg->get_all('lei.q.output') or return ();
map {;
- s!\A(?:maildir|mh|mbox.+|mmdf):!!i;
+ s!$LOCAL_PFX!!;
$_;
} @$out
}
@@ -221,7 +245,7 @@ sub cloneurl { [] }
sub output2lssdir {
my ($self, $lei, $dir_ref, $fn_ref) = @_;
my $dst = $$dir_ref; # imap://$MAILBOX, /path/to/maildir, /path/to/mbox
- my $dir = lss_dir_for($lei, \$dst);
+ my $dir = lss_dir_for($lei, \$dst, 1);
my $f = "$dir/lei.saved-search";
if (-f $f && -r _) {
$self->{-cfg} = PublicInbox::Config->git_config_dump($f);
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 26ea5cb8..170f7ce5 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -158,5 +158,11 @@ test_lei(sub {
lei_ok('up', $o);
@m = glob("$o/cur/*");
is(scalar(@m), 2, 'got 2nd result due to different OID');
+
+ SKIP: {
+ symlink($o, "$home/ln -s") or
+ skip "symlinks not supported in $home?: $!", 1;
+ lei_ok('up', "$home/ln -s");
+ };
});
done_testing;
^ permalink raw reply related [relevance 55%]
* [PATCH] lei import: support adding keywords and labels on import
@ 2021-04-23 11:22 40% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-23 11:22 UTC (permalink / raw)
To: meta
This saves some work and makes it easier to set volatile
metadata on a message at import time.
---
lib/PublicInbox/LeiImport.pm | 7 +++++
lib/PublicInbox/LeiInput.pm | 54 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiTag.pm | 56 ++----------------------------------
lib/PublicInbox/SearchIdx.pm | 41 +++++++++++++++-----------
t/lei-import.t | 10 +++++++
5 files changed, 98 insertions(+), 70 deletions(-)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index accf08f5..e3c756e8 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -12,6 +12,10 @@ use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
my ($self, $eml, $vmd) = @_;
my $xoids = $self->{lei}->{ale}->xoids_for($eml);
+ if (my $all_vmd = $self->{all_vmd}) {
+ $vmd //= {};
+ @$vmd{keys %$all_vmd} = values %$all_vmd;
+ }
$self->{lei}->{sto}->ipc_do('set_eml', $eml, $vmd, $xoids);
}
@@ -53,6 +57,9 @@ sub lei_import { # the main "lei import" method
$sto->write_prepare($lei);
my $self = bless {}, __PACKAGE__;
$self->{-import_kw} = $lei->{opt}->{kw} // 1;
+ my $vmd_mod = $self->vmd_mod_extract(\@inputs);
+ return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
+ $self->{all_vmd} = $vmd_mod if scalar keys %$vmd_mod;
$self->prepare_inputs($lei, \@inputs) or return;
$lei->ale; # initialize for workers to read
my $j = $lei->{opt}->{jobs} // scalar(@{$self->{inputs}}) || 1;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index e416d3ed..de60a076 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -7,6 +7,38 @@ use strict;
use v5.10.1;
use PublicInbox::DS;
+# JMAP RFC 8621 4.1.1
+# https://www.iana.org/assignments/imap-jmap-keywords/imap-jmap-keywords.xhtml
+our @KW = (qw(seen answered flagged draft), # widely-compatible
+ qw(forwarded), # IMAP + Maildir
+ qw(phishing junk notjunk)); # rarely supported
+
+# note: RFC 8621 states "Users may add arbitrary keywords to an Email",
+# but is it good idea? Stick to the system and reserved ones, for now.
+# The widely-compatible ones map to IMAP system flags, Maildir flags
+# and mbox Status/X-Status headers.
+my %KW = map { $_ => 1 } @KW;
+my $L_MAX = 244; # Xapian term limit - length('L')
+
+# RFC 8621, sec 2 (Mailboxes) a "label" for us is a JMAP Mailbox "name"
+# "Servers MAY reject names that violate server policy"
+my %ERR = (
+ L => sub {
+ my ($label) = @_;
+ length($label) >= $L_MAX and
+ return "`$label' too long (must be <= $L_MAX)";
+ $label =~ m{\A[a-z0-9_](?:[a-z0-9_\-\./\@,]*[a-z0-9])?\z}i ?
+ undef : "`$label' is invalid";
+ },
+ kw => sub {
+ my ($kw) = @_;
+ $KW{$kw} ? undef : <<EOM;
+`$kw' is not one of: `seen', `flagged', `answered', `draft'
+`junk', `notjunk', `phishing' or `forwarded'
+EOM
+ }
+);
+
sub check_input_format ($;$) {
my ($lei, $files) = @_;
my $opt_key = 'in-format';
@@ -183,4 +215,26 @@ sub input_only_atfork_child {
undef;
}
+# like Getopt::Long, but for +kw:FOO and -kw:FOO to prepare
+# for update_xvmd -> update_vmd
+sub vmd_mod_extract {
+ my $argv = $_[-1];
+ my $vmd_mod = {};
+ my @new_argv;
+ for my $x (@$argv) {
+ if ($x =~ /\A(\+|\-)(kw|L):(.+)\z/) {
+ my ($op, $pfx, $val) = ($1, $2, $3);
+ if (my $err = $ERR{$pfx}->($val)) {
+ push @{$vmd_mod->{err}}, $err;
+ } else { # set "+kw", "+L", "-L", "-kw"
+ push @{$vmd_mod->{$op.$pfx}}, $val;
+ }
+ } else {
+ push @new_argv, $x;
+ }
+ }
+ @$argv = @new_argv;
+ $vmd_mod;
+}
+
1;
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index f019202f..f5791947 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -7,58 +7,6 @@ use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
-# JMAP RFC 8621 4.1.1
-# https://www.iana.org/assignments/imap-jmap-keywords/imap-jmap-keywords.xhtml
-my @KW = (qw(seen answered flagged draft), # widely-compatible
- qw(forwarded phishing junk notjunk)); # rarely supported
-# note: RFC 8621 states "Users may add arbitrary keywords to an Email",
-# but is it good idea? Stick to the system and reserved ones, for now.
-# The widely-compatible ones map to IMAP system flags, Maildir flags
-# and mbox Status/X-Status headers.
-my %KW = map { $_ => 1 } @KW;
-my $L_MAX = 244; # Xapian term limit - length('L')
-
-# RFC 8621, sec 2 (Mailboxes) a "label" for us is a JMAP Mailbox "name"
-# "Servers MAY reject names that violate server policy"
-my %ERR = (
- L => sub {
- my ($label) = @_;
- length($label) >= $L_MAX and
- return "`$label' too long (must be <= $L_MAX)";
- $label =~ m{\A[a-z0-9_](?:[a-z0-9_\-\./\@,]*[a-z0-9])?\z}i ?
- undef : "`$label' is invalid";
- },
- kw => sub {
- my ($kw) = @_;
- $KW{$kw} ? undef : <<EOM;
-`$kw' is not one of: `seen', `flagged', `answered', `draft'
-`junk', `notjunk', `phishing' or `forwarded'
-EOM
- }
-);
-
-# like Getopt::Long, but for +kw:FOO and -kw:FOO to prepare
-# for update_xvmd -> update_vmd
-sub vmd_mod_extract {
- my $argv = $_[-1];
- my $vmd_mod = {};
- my @new_argv;
- for my $x (@$argv) {
- if ($x =~ /\A(\+|\-)(kw|L):(.+)\z/) {
- my ($op, $pfx, $val) = ($1, $2, $3);
- if (my $err = $ERR{$pfx}->($val)) {
- push @{$vmd_mod->{err}}, $err;
- } else { # set "+kw", "+L", "-L", "-kw"
- push @{$vmd_mod->{$op.$pfx}}, $val;
- }
- } else {
- push @new_argv, $x;
- }
- }
- @$argv = @new_argv;
- $vmd_mod;
-}
-
sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
my ($self, $eml) = @_;
if (my $xoids = $self->{lei}->{ale}->xoids_for($eml)) {
@@ -99,7 +47,7 @@ sub lei_tag { # the "lei tag" method
$sto->write_prepare($lei);
my $self = bless { missing => 0 }, __PACKAGE__;
$lei->ale; # refresh and prepare
- my $vmd_mod = vmd_mod_extract(\@argv);
+ my $vmd_mod = $self->vmd_mod_extract(\@argv);
return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
$self->prepare_inputs($lei, \@argv) or return;
grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
@@ -161,7 +109,7 @@ sub _complete_mark_common ($) {
sub _complete_tag {
my ($self, @argv) = @_;
my @L = eval { $self->_lei_store->search->all_terms('L') };
- my @all = ((map { ("+kw:$_", "-kw:$_") } @KW),
+ my @all = ((map { ("+kw:$_", "-kw:$_") } @PublicInbox::LeiInput::KW),
(map { ("+L:$_", "-L:$_") } @L));
return @all if !@argv;
my ($cur, $re) = _complete_mark_common(\@argv);
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index ca1f3588..f066cc92 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -567,16 +567,39 @@ sub set_vmd {
$self->{xdb}->replace_document($docid, $doc);
}
+sub apply_vmd_mod ($$) {
+ my ($doc, $vmd_mod) = @_;
+ my $updated = 0;
+ my @x = @VMD_MAP;
+ while (my ($field, $pfx) = splice(@x, 0, 2)) {
+ # field: "label" or "kw"
+ for my $val (@{$vmd_mod->{"-$field"} // []}) {
+ eval {
+ $doc->remove_term($pfx . $val);
+ ++$updated;
+ };
+ }
+ for my $val (@{$vmd_mod->{"+$field"} // []}) {
+ $doc->add_boolean_term($pfx . $val);
+ ++$updated;
+ }
+ }
+ $updated;
+}
+
sub add_vmd {
my ($self, $docid, $vmd) = @_;
begin_txn_lazy($self);
my $doc = _get_doc($self, $docid) or return;
my @x = @VMD_MAP;
+ my $updated = 0;
while (my ($field, $pfx) = splice(@x, 0, 2)) {
my $add = $vmd->{$field} // next;
$doc->add_boolean_term($pfx . $_) for @$add;
+ $updated += scalar(@$add);
}
- $self->{xdb}->replace_document($docid, $doc);
+ $updated += apply_vmd_mod($doc, $vmd);
+ $self->{xdb}->replace_document($docid, $doc) if $updated;
}
sub remove_vmd {
@@ -601,21 +624,7 @@ sub update_vmd {
my ($self, $docid, $vmd_mod) = @_;
begin_txn_lazy($self);
my $doc = _get_doc($self, $docid) or return;
- my $updated = 0;
- my @x = @VMD_MAP;
- while (my ($field, $pfx) = splice(@x, 0, 2)) {
- # field: "label" or "kw"
- for my $val (@{$vmd_mod->{"-$field"} // []}) {
- eval {
- $doc->remove_term($pfx . $val);
- ++$updated;
- };
- }
- for my $val (@{$vmd_mod->{"+$field"} // []}) {
- $doc->add_boolean_term($pfx . $val);
- ++$updated;
- }
- }
+ my $updated = apply_vmd_mod($doc, $vmd_mod);
$self->{xdb}->replace_document($docid, $doc) if $updated;
$updated;
}
diff --git a/t/lei-import.t b/t/lei-import.t
index 8635df5a..6e9a853c 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -101,6 +101,16 @@ is_deeply($draft_a, $draft_b, 'fake Message-ID lookup') or
lei_ok('blob', '--mail', $draft_b->[0]->{blob});
is($lei_out, $eml_str, 'draft retrieved by blob');
+
+$eml_str = "Message-ID: <inbox\@example.com>\nSubject: label-this\n\n";
+lei_ok([qw(import -F eml - +kw:seen +L:inbox)],
+ undef, { %$lei_opt, 0 => \$eml_str });
+lei_ok(qw(q m:inbox@example.com));
+$res = json_utf8->decode($lei_out);
+is_deeply($res->[0]->{kw}, ['seen'], 'keyword set');
+is_deeply($res->[0]->{L}, ['inbox'], 'label set');
+
+
# see t/lei_to_mail.t for "import -F mbox*"
});
done_testing;
^ permalink raw reply related [relevance 40%]
* [PATCH 0/7] lei sync preparations, "lei inspect"
@ 2021-04-24 9:28 89% Eric Wong
2021-04-24 9:28 66% ` [PATCH 2/7] t/lei_to_mail: split "lei import" test $HOME directory Eric Wong
2021-04-24 9:28 27% ` [PATCH 7/7] lei import: keep sync info for Maildir and IMAP folders Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-04-24 9:28 UTC (permalink / raw)
To: meta
"lei import" now tracks IMAP and Maildir source information in
preparation for propagating keywords back to IMAP|Maildir.
"lei inspect" is a long-overdue debug/diagnostic thing.
Eric Wong (7):
lei_input: drop outdated comment w.r.t. compression
t/lei_to_mail: split "lei import" test $HOME directory
URIimap: support ->uidvalidity and ->iuid
net_reader: imap_each: add UIDVALIDITY to URL arg
doc: lei_design_notes: add a bit on WAL usage
lei_mail_sync: for bidirectional keyword sync
lei import: keep sync info for Maildir and IMAP folders
Documentation/lei_design_notes.txt | 12 ++
MANIFEST | 3 +
lib/PublicInbox/LEI.pm | 16 ++-
lib/PublicInbox/LeiImport.pm | 22 ++-
lib/PublicInbox/LeiInput.pm | 42 +++++-
lib/PublicInbox/LeiInspect.pm | 96 +++++++++++++
lib/PublicInbox/LeiMailSync.pm | 211 +++++++++++++++++++++++++++++
lib/PublicInbox/LeiSearch.pm | 7 +
lib/PublicInbox/LeiStore.pm | 20 ++-
lib/PublicInbox/NetReader.pm | 11 +-
lib/PublicInbox/TestCommon.pm | 2 +
lib/PublicInbox/URIimap.pm | 38 +++++-
t/lei-import-imap.t | 27 +++-
t/lei-import-maildir.t | 21 +++
t/lei_mail_sync.t | 68 ++++++++++
t/lei_to_mail.t | 8 +-
t/net_reader-imap.t | 4 +-
t/uri_imap.t | 32 ++++-
18 files changed, 612 insertions(+), 28 deletions(-)
create mode 100644 lib/PublicInbox/LeiInspect.pm
create mode 100644 lib/PublicInbox/LeiMailSync.pm
create mode 100644 t/lei_mail_sync.t
^ permalink raw reply [relevance 89%]
* [PATCH 2/7] t/lei_to_mail: split "lei import" test $HOME directory
2021-04-24 9:28 89% [PATCH 0/7] lei sync preparations, "lei inspect" Eric Wong
@ 2021-04-24 9:28 66% ` Eric Wong
2021-04-24 9:28 27% ` [PATCH 7/7] lei import: keep sync info for Maildir and IMAP folders Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-24 9:28 UTC (permalink / raw)
To: meta
"lei import" behavior will may change w.r.t. keyword
handling. Use separate $HOME between different test_lei
to ensure isolation between the tests.
---
lib/PublicInbox/TestCommon.pm | 2 ++
t/lei_to_mail.t | 8 ++++----
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index b5d0b9f8..49cecacd 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -517,6 +517,7 @@ SKIP: {
require_git(2.6, 1) or skip('git 2.6+ required for lei test', 2);
require_mods(qw(json DBD::SQLite Search::Xapian), 2);
require PublicInbox::Config;
+ require File::Path;
local %ENV = %ENV;
delete $ENV{XDG_DATA_HOME};
delete $ENV{XDG_CONFIG_HOME};
@@ -534,6 +535,7 @@ EOM
$lei_opt = { 1 => \$lei_out, 2 => \$lei_err };
my ($daemon_pid, $for_destroy, $daemon_xrd);
my $tmpdir = $test_opt->{tmpdir};
+ File::Path::mkpath($tmpdir) if (defined $tmpdir && !-d $tmpdir);
($tmpdir, $for_destroy) = tmpdir unless $tmpdir;
state $persist_xrd = $ENV{TEST_LEI_DAEMON_PERSIST_DIR};
SKIP: {
diff --git a/t/lei_to_mail.t b/t/lei_to_mail.t
index 51357257..32532a98 100644
--- a/t/lei_to_mail.t
+++ b/t/lei_to_mail.t
@@ -129,9 +129,9 @@ my $orig = do {
$raw;
};
-test_lei(sub {
- ok(lei(qw(import -F), $mbox, $fn), 'imported mbox');
- ok(lei(qw(q s:x)), 'lei q works') or diag $lei_err;
+test_lei({tmpdir => "$tmpdir/using -F"}, sub {
+ lei_ok(qw(import -F), $mbox, $fn, \'imported mbox');
+ lei_ok(qw(q s:x), \'lei q works') or diag $lei_err;
my $res = json_utf8->decode($lei_out);
my $x = $res->[0];
is($x->{'s'}, 'x', 'subject imported') or diag $lei_out;
@@ -139,7 +139,7 @@ test_lei(sub {
is($res->[1], undef, 'only one result');
});
-test_lei(sub {
+test_lei({tmpdir => "$tmpdir/using TYPE: prefix"}, sub {
lei_ok('import', "$mbox:$fn", \'imported mbox:/path') or diag $lei_err;
lei_ok(qw(q s:x), \'lei q works') or diag $lei_err;
my $res = json_utf8->decode($lei_out);
^ permalink raw reply related [relevance 66%]
* [PATCH 7/7] lei import: keep sync info for Maildir and IMAP folders
2021-04-24 9:28 89% [PATCH 0/7] lei sync preparations, "lei inspect" Eric Wong
2021-04-24 9:28 66% ` [PATCH 2/7] t/lei_to_mail: split "lei import" test $HOME directory Eric Wong
@ 2021-04-24 9:28 27% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-04-24 9:28 UTC (permalink / raw)
To: meta
We aren't using it, yet, but the plan is to be able to use
this information to propagate keyword changes back to IMAP
and Maildir folders using some to-be-implemented command.
"lei inspect" is a half-baked new command to make testing this
change easier. It will be updated to support more SQLite+Xapian
introspection duties in the future, including public-inbox
things independent of lei.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 16 ++++--
lib/PublicInbox/LeiImport.pm | 22 ++++++--
lib/PublicInbox/LeiInput.pm | 41 +++++++++++++--
lib/PublicInbox/LeiInspect.pm | 96 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiSearch.pm | 7 +++
lib/PublicInbox/LeiStore.pm | 20 +++++++-
t/lei-import-imap.t | 27 +++++++++-
t/lei-import-maildir.t | 21 ++++++++
9 files changed, 238 insertions(+), 13 deletions(-)
create mode 100644 lib/PublicInbox/LeiInspect.pm
diff --git a/MANIFEST b/MANIFEST
index abaf54b0..79d393c5 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -199,6 +199,7 @@ lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
+lib/PublicInbox/LeiInspect.pm
lib/PublicInbox/LeiLsLabel.pm
lib/PublicInbox/LeiLsSearch.pm
lib/PublicInbox/LeiMailSync.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9f49fc03..39278de6 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -64,9 +64,13 @@ sub opt_dash ($$) {
($spec, '<>' => $cb, $GLP_PASS) # for Getopt::Long
}
-sub rel2abs ($$) {
+# rel2abs preserves symlinks in parent, unlike abs_path
+sub rel2abs {
my ($self, $p) = @_;
- return $p if index($p, '/') == 0; # already absolute
+ if (index($p, '/') == 0) { # already absolute
+ $p =~ tr!/!/!s; # squeeze redundant slashes
+ return $p;
+ }
my $pwd = $self->{env}->{PWD};
my $cwd;
if (defined $pwd) {
@@ -84,6 +88,9 @@ sub rel2abs ($$) {
File::Spec->rel2abs($p, $pwd);
}
+# abs_path resolves symlinks in parent iff all parents exist
+sub abs_path { Cwd::abs_path($_[1]) // rel2abs(@_) }
+
sub share_path ($) { # $HOME/.local/share/lei/$FOO
my ($self) = @_;
rel2abs($self, ($self->{env}->{XDG_DATA_HOME} //
@@ -193,7 +200,7 @@ our %CMD = ( # sorted in order of importance/use:
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw! verbose|v+ incremental!), @c_opt ],
+ lock=s@ in-format|F=s kw! verbose|v+ incremental! sync!), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
@@ -205,6 +212,9 @@ our %CMD = ( # sorted in order of importance/use:
'git-config(1) wrapper for '._config_path($_[0]);
}, qw(config-file|system|global|file|f=s), # for conflict detection
qw(c=s@ C=s@), pass_through('git config') ],
+'inspect' => [ 'ITEMS...', 'inspect lei/store and/or local external',
+ qw(pretty ascii dir=s), @c_opt ],
+
'init' => [ '[DIRNAME]', sub {
"initialize storage, default: ".store_path($_[0]);
}, @c_opt ],
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index e3c756e8..daaa6753 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -13,7 +13,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
my ($self, $eml, $vmd) = @_;
my $xoids = $self->{lei}->{ale}->xoids_for($eml);
if (my $all_vmd = $self->{all_vmd}) {
- $vmd //= {};
@$vmd{keys %$all_vmd} = values %$all_vmd;
}
$self->{lei}->{sto}->ipc_do('set_eml', $eml, $vmd, $xoids);
@@ -31,11 +30,26 @@ sub input_mbox_cb { # MboxReader callback
sub input_maildir_cb { # maildir_each_eml cb
my ($f, $kw, $eml, $self) = @_;
- input_eml_cb($self, $eml, $self->{-import_kw} ? { kw => $kw } : undef);
+ my $vmd = $self->{-import_kw} ? { kw => $kw } : undef;
+ if ($self->{-mail_sync}) {
+ if ($f =~ m!\A(.+?)/(?:new|cur)/([^/]+)\z!) { # ugh...
+ $vmd->{sync_info} = [ "maildir:$1", \(my $n = $2) ];
+ } else {
+ warn "E: $f was not from a Maildir?\n";
+ }
+ }
+ input_eml_cb($self, $eml, $vmd);
}
-sub input_net_cb { # imap_each, nntp_each cb
+sub input_imap_cb { # imap_each
my ($url, $uid, $kw, $eml, $self) = @_;
+ my $vmd = $self->{-import_kw} ? { kw => $kw } : undef;
+ $vmd->{sync_info} = [ $url, $uid ] if $self->{-mail_sync};
+ input_eml_cb($self, $eml, $vmd);
+}
+
+sub input_nntp_cb { # nntp_each
+ my ($url, $num, $kw, $eml, $self) = @_;
input_eml_cb($self, $eml, $self->{-import_kw} ? { kw => $kw } : undef);
}
@@ -61,6 +75,8 @@ sub lei_import { # the main "lei import" method
return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
$self->{all_vmd} = $vmd_mod if scalar keys %$vmd_mod;
$self->prepare_inputs($lei, \@inputs) or return;
+ $self->{-mail_sync} = $lei->{opt}->{sync} // 1;
+
$lei->ale; # initialize for workers to read
my $j = $lei->{opt}->{jobs} // scalar(@{$self->{inputs}}) || 1;
if (my $net = $lei->{net}) {
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 0114f5ee..d11d23d4 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -83,11 +83,13 @@ sub input_path_url {
my $ifmt = lc($lei->{opt}->{'in-format'} // '');
# TODO auto-detect?
if ($input =~ m!\Aimaps?://!i) {
- $lei->{net}->imap_each($input, $self->can('input_net_cb'),
+ $lei->{net}->imap_each($input, $self->can('input_imap_cb') //
+ $self->can('input_net_cb'),
$self, @args);
return;
} elsif ($input =~ m!\A(?:nntps?|s?news)://!i) {
- $lei->{net}->nntp_each($input, $self->can('input_net_cb'),
+ $lei->{net}->nntp_each($input, $self->can('input_nntp_cb') //
+ $self->can('input_net_cb'),
$self, @args);
return;
}
@@ -130,11 +132,13 @@ EOM
sub prepare_inputs { # returns undef on error
my ($self, $lei, $inputs) = @_;
my $in_fmt = $lei->{opt}->{'in-format'};
+ my $sync = $lei->{opt}->{sync} ? {} : undef; # using LeiMailSync
if ($lei->{opt}->{stdin}) {
@$inputs and return
$lei->fail("--stdin and @$inputs do not mix");
check_input_format($lei) or return;
push @$inputs, '/dev/stdin';
+ push @{$sync->{no}}, '/dev/stdin' if $sync;
}
my $net = $lei->{net}; # NetWriter may be created by l2m
my (@f, @d);
@@ -145,6 +149,13 @@ sub prepare_inputs { # returns undef on error
require PublicInbox::NetReader;
$net //= PublicInbox::NetReader->new;
$net->add_url($input);
+ if ($sync) {
+ if ($input =~ m!\Aimaps?://!) {
+ push @{$sync->{ok}}, $input;
+ } else {
+ push @{$sync->{no}}, $input;
+ }
+ }
} elsif ($input_path =~ s/\A([a-z0-9]+)://is) {
my $ifmt = lc $1;
if (($in_fmt // $ifmt) ne $ifmt) {
@@ -152,6 +163,13 @@ sub prepare_inputs { # returns undef on error
--in-format=$in_fmt and `$ifmt:' conflict
}
+ if ($sync) {
+ if ($ifmt =~ /\A(?:maildir|mh)\z/i) {
+ push @{$sync->{ok}}, $input;
+ } else {
+ push @{$sync->{no}}, $input;
+ }
+ }
my $devfd = $lei->path_to_fd($input_path) // return;
if ($devfd >= 0 || (-f $input_path || -p _)) {
require PublicInbox::MboxLock;
@@ -162,6 +180,7 @@ sub prepare_inputs { # returns undef on error
require PublicInbox::MdirReader;
$ifmt eq 'maildir' or return
$lei->fail("$ifmt not supported");
+ $input = $lei->abs_path($input) if $sync;
} else {
return $lei->fail("Unable to handle $input");
}
@@ -170,12 +189,18 @@ sub prepare_inputs { # returns undef on error
$input is `eml', not --in-format=$in_fmt
require PublicInbox::Eml;
+ push @{$sync->{no}}, $input if $sync;
} else {
my $devfd = $lei->path_to_fd($input) // return;
if ($devfd >= 0 || -f $input || -p _) {
- push @f, $input
+ push @{$sync->{no}}, $input if $sync;
+ push @f, $input;
} elsif (-d $input) {
- push @d, $input
+ if ($sync) {
+ $input = $lei->abs_path($input);
+ push @{$sync->{ok}}, $input;
+ }
+ push @d, $input;
} else {
return $lei->fail("Unable to handle $input")
}
@@ -185,6 +210,14 @@ $input is `eml', not --in-format=$in_fmt
if (@d) { # TODO: check for MH vs Maildir, here
require PublicInbox::MdirReader;
}
+ if ($sync && $sync->{no}) {
+ return $lei->fail(<<"") if !$sync->{ok};
+--sync specified but no inputs support it
+
+ # non-fatal if some inputs support support sync
+ $lei->err("# --sync will only be used for @{$sync->{ok}}");
+ $lei->err("# --sync is not supported for: @{$sync->{no}}");
+ }
if ($net) {
if (my $err = $net->errors) {
return $lei->fail($err);
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
new file mode 100644
index 00000000..6cfc8083
--- /dev/null
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -0,0 +1,96 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# "lei inspect" general purpose inspector for stuff in SQLite and
+# Xapian. Will eventually be useful with plain public-inboxes,
+# not just lei/store. This is totally half-baked at the moment
+# but useful for testing.
+package PublicInbox::LeiInspect;
+use strict;
+use v5.10.1;
+use PublicInbox::Config;
+
+sub inspect_blob ($$) {
+ my ($lei, $oidhex) = @_;
+ my $ent = {};
+ if (my $lse = $lei->{lse}) {
+ my @docids = $lse ? $lse->over->blob_exists($oidhex) : ();
+ $ent->{'lei/store'} = \@docids if @docids;
+ my $lms = $lse->lms;
+ if (my $loc = $lms ? $lms->locations_for($oidhex) : undef) {
+ $ent->{sync} = $loc;
+ }
+ }
+ $ent;
+}
+
+sub inspect_sync_folder ($$) {
+ my ($lei, $folder) = @_;
+ my $ent = {};
+ my $lse = $lei->{lse} or return $ent;
+ my $lms = $lse->lms or return $ent;
+ my @folders;
+ if ($folder =~ m!\Aimaps?://!i) {
+ require PublicInbox::URIimap;
+ my $uri = PublicInbox::URIimap->new($folder)->canonical;
+ if (defined($uri->uidvalidity)) {
+ $folders[0] = $$uri;
+ } else {
+ my @maybe = $lms->folders($$uri);
+ @folders = grep {
+ my $u = PublicInbox::URIimap->new($_);
+ $uri->uidvalidity($u->uidvalidity);
+ $$uri eq $$u;
+ } @maybe;
+ }
+ } elsif ($folder =~ m!\A(maildir|mh):(.+)!i) {
+ my $type = $1;
+ $folders[0] = "$type:".$lei->abs_path($2);
+ } elsif (-d $folder) {
+ $folders[0] = 'maildir:'.$lei->abs_path($folder);
+ } else {
+ $lei->fail("$folder not understood");
+ }
+ $lei->qerr("# no folders match $folder (non-fatal)") if !@folders;
+ for my $f (@folders) {
+ $ent->{$f} = $lms->location_stats($f); # may be undef
+ }
+ $ent
+}
+
+sub inspect1 ($$$) {
+ my ($lei, $item, $more) = @_;
+ my $ent;
+ if ($item =~ /\Ablob:(.+)/) {
+ $ent = inspect_blob($lei, $1);
+ } elsif ($item =~ m!\Aimaps?://!i ||
+ $item =~ m!\A(?:maildir|mh):!i || -d $item) {
+ $ent = inspect_sync_folder($lei, $item);
+ } else { # TODO: more things
+ return $lei->fail("$item not understood");
+ }
+ $lei->out($lei->{json}->encode($ent));
+ $lei->out(',') if $more;
+ 1;
+}
+
+sub lei_inspect {
+ my ($lei, @argv) = @_;
+ $lei->{1}->autoflush(0);
+ my $multi = scalar(@argv) > 1;
+ $lei->out('[') if $multi;
+ $lei->{json} = ref(PublicInbox::Config::json())->new->utf8->canonical;
+ $lei->{lse} = ($lei->{opt}->{external} // 1) ? do {
+ my $sto = $lei->_lei_store;
+ $sto ? $sto->search : undef;
+ } : undef;
+ if ($lei->{opt}->{pretty} || -t $lei->{1}) {
+ $lei->{json}->pretty(1)->indent(2);
+ }
+ while (defined(my $x = shift @argv)) {
+ inspect1($lei, $x, scalar(@argv)) or return;
+ }
+ $lei->out(']') if $multi;
+}
+
+1;
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index ff615d89..cd28a700 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -137,4 +137,11 @@ sub qparse_new {
$qp
}
+sub lms {
+ my ($self) = @_;
+ require PublicInbox::LeiMailSync;
+ my $f = "$self->{topdir}/mail_sync.sqlite3";
+ -f $f ? PublicInbox::LeiMailSync->new($f) : undef;
+}
+
1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index f8371abf..1cf7ffc1 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -190,13 +190,28 @@ sub remove_eml_vmd {
\@docids;
}
+sub set_sync_info ($$$) {
+ my ($self, $oidhex, $sync_info) = @_;
+ ($self->{lms} //= do {
+ require PublicInbox::LeiMailSync;
+ my $f = "$self->{priv_eidx}->{topdir}/mail_sync.sqlite3";
+ my $lms = PublicInbox::LeiMailSync->new($f);
+ $lms->lms_begin;
+ $lms;
+ })->set_src($oidhex, @$sync_info);
+}
+
sub add_eml {
my ($self, $eml, $vmd, $xoids) = @_;
my $im = $self->importer; # may create new epoch
my ($eidx, $tl) = eidx_init($self); # updates/writes alternates file
my $oidx = $eidx->{oidx}; # PublicInbox::Import::add checks this
my $smsg = bless { -oidx => $oidx }, 'PublicInbox::Smsg';
- $im->add($eml, undef, $smsg) or return; # duplicate returns undef
+ my $im_mark = $im->add($eml, undef, $smsg);
+ if ($vmd && $vmd->{sync_info}) {
+ set_sync_info($self, $smsg->{blob}, $vmd->{sync_info});
+ }
+ $im_mark or return; # duplicate blob returns undef
local $self->{current_info} = $smsg->{blob};
my $vivify_xvmd = delete($smsg->{-vivify_xvmd}) // []; # exact matches
@@ -379,6 +394,9 @@ sub done {
warn $err;
}
}
+ if (my $lms = delete $self->{lms}) {
+ $lms->lms_commit;
+ }
$self->{priv_eidx}->done; # V2Writable::done
xchg_stderr($self);
die $err if $err;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 490ea9be..4a3bd6d8 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -12,10 +12,28 @@ my $td = start_script($cmd, $env, { 3 => $sock }) or BAIL_OUT("-imapd: $?");
my $host_port = tcp_host_port($sock);
undef $sock;
test_lei({ tmpdir => $tmpdir }, sub {
+ my $url = "imap://$host_port/t.v2.0";
+
lei_ok(qw(q z:1..));
my $out = json_utf8->decode($lei_out);
is_deeply($out, [ undef ], 'nothing imported, yet');
- lei_ok('import', "imap://$host_port/t.v2.0");
+
+ lei_ok('inspect', $url);
+ is_deeply(json_utf8->decode($lei_out), {}, 'no inspect stats, yet');
+
+ lei_ok('import', $url);
+
+ lei_ok('inspect', $url);
+ my $inspect = json_utf8->decode($lei_out);
+ my @k = keys %$inspect;
+ is(scalar(@k), 1, 'one URL resolved');
+ like($k[0], qr!\A\Q$url\E;UIDVALIDITY=\d+\z!, 'inspect URL matches');
+ my $stats = $inspect->{$k[0]};
+ is_deeply([ sort keys %$stats ],
+ [ qw(uid.count uid.max uid.min) ], 'keys match');
+ ok($stats->{'uid.min'} < $stats->{'uid.max'}, 'min < max');
+ ok($stats->{'uid.count'} > 0, 'count > 0');
+
lei_ok(qw(q z:1..));
$out = json_utf8->decode($lei_out);
ok(scalar(@$out) > 1, 'got imported messages');
@@ -23,9 +41,14 @@ test_lei({ tmpdir => $tmpdir }, sub {
my %r;
for (@$out) { $r{ref($_)}++ }
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
- lei_ok([qw(tag +kw:seen), "imap://$host_port/t.v2.0"], undef, undef);
+ lei_ok([qw(tag +kw:seen), $url], undef, undef);
my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
ok(-s $f, 'net tracked for redundant imports');
+ lei_ok('inspect', "blob:$out->[5]->{blob}");
+ my $x = json_utf8->decode($lei_out);
+ is(ref($x->{'lei/store'}), 'ARRAY', 'lei/store in inspect');
+ is(ref($x->{sync}), 'HASH', 'sync in inspect');
+ is(ref($x->{sync}->{$k[0]}), 'ARRAY', 'UID arrays in inspect');
});
done_testing;
diff --git a/t/lei-import-maildir.t b/t/lei-import-maildir.t
index 6706b014..3e3d9188 100644
--- a/t/lei-import-maildir.t
+++ b/t/lei-import-maildir.t
@@ -12,6 +12,21 @@ test_lei(sub {
BAIL_OUT "symlink $md $!";
lei_ok(qw(import), $md, \'import Maildir');
my $imp_err = $lei_err;
+
+ my %i;
+ lei_ok('inspect', $md); $i{no_type} = $lei_out;
+ lei_ok('inspect', "maildir:$md"), $i{with_type} = $lei_out;
+ lei_ok(['inspect', $md], undef, { -C => $ENV{HOME}, %$lei_opt });
+ $i{rel_no_type} = $lei_out;
+ lei_ok(['inspect', "maildir:$md"], undef,
+ { -C => $ENV{HOME}, %$lei_opt });
+ $i{rel_with_type} = $lei_out;
+ my %v = map { $_ => 1 } values %i;
+ is(scalar(keys %v), 1, 'inspect handles relative and absolute paths');
+ my $inspect = json_utf8->decode([ keys %v ]->[0]);
+ is_deeply($inspect, {"maildir:$md" => { 'name.count' => 1 }},
+ 'inspect maildir: path had expected output');
+
lei_ok(qw(q s:boolean));
my $res = json_utf8->decode($lei_out);
like($res->[0]->{'s'}, qr/use boolean/, 'got expected result')
@@ -19,6 +34,12 @@ test_lei(sub {
is_deeply($res->[0]->{kw}, ['seen'], 'keyword set');
is($res->[1], undef, 'only got one result');
+ lei_ok('inspect', "blob:$res->[0]->{blob}");
+ $inspect = json_utf8->decode($lei_out);
+ is(ref(delete $inspect->{"lei/store"}), 'ARRAY', 'lei/store IDs');
+ is_deeply($inspect, { sync => { "maildir:$md" => [ 'x:2,S' ] } },
+ 'maildir sync info as expected');
+
lei_ok(qw(import), $md, \'import Maildir again');
$imp_err = $lei_err;
lei_ok(qw(q -d none s:boolean), \'lei q w/o dedupe');
^ permalink raw reply related [relevance 27%]
* [PATCH] lei p2q: exit with failure if format-patch fails
@ 2021-04-26 8:43 66% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-26 8:43 UTC (permalink / raw)
To: meta
Merely redirecting the failure message from git to our stderr is
insufficient.
---
lib/PublicInbox/LeiP2q.pm | 7 +++++--
t/lei-p2q.t | 4 +++-
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index a8a3dd2c..cb2309c7 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -105,6 +105,7 @@ sub do_p2q { # via wq_do
}
my $smsg = bless {}, 'PublicInbox::Smsg';
my $in = $self->{0};
+ my @cmd;
unless ($in) {
my $input = $self->{input};
my $devfd = $lei->path_to_fd($input) // return;
@@ -114,11 +115,13 @@ sub do_p2q { # via wq_do
open($in, '<', $input) or
return $lei->fail("open < $input: $!");
} else {
- my @cmd = (qw(git format-patch --stdout -1), $input);
+ @cmd = (qw(git format-patch --stdout -1), $input);
$in = popen_rd(\@cmd, undef, { 2 => $lei->{2} });
}
};
- my $eml = PublicInbox::Eml->new(\(do { local $/; <$in> }));
+ my $str = do { local $/; <$in> };
+ @cmd && !close($in) and return $lei->fail("E: @cmd failed: $?");
+ my $eml = PublicInbox::Eml->new(\$str);
$lei->{diff_want} = +{ map { $_ => 1 } @want };
$smsg->populate($eml);
while (my ($pfx, $fields) = each %pfx2smsg) {
diff --git a/t/lei-p2q.t b/t/lei-p2q.t
index 87cf9fa7..be2d437c 100644
--- a/t/lei-p2q.t
+++ b/t/lei-p2q.t
@@ -6,9 +6,11 @@ require_git 2.6;
require_mods(qw(json DBD::SQLite Search::Xapian));
test_lei(sub {
+ ok(!lei(qw(p2q this-better-cause-format-patch-to-fail)),
+ 'p2q fails on bogus arg');
lei_ok(qw(p2q -w dfpost t/data/0001.patch));
is($lei_out, "dfpost:6e006fd73b1d\n", 'pathname');
- open my $fh, '+<', 't/data/0001.patch';
+ open my $fh, '+<', 't/data/0001.patch' or xbail "open: $!";
lei_ok([qw(p2q -w dfpost -)], undef, { %$lei_opt, 0 => $fh });
is($lei_out, "dfpost:6e006fd73b1d\n", '--stdin');
^ permalink raw reply related [relevance 66%]
* lei-managed pseudo mailing lists
@ 2021-04-26 16:44 70% Konstantin Ryabitsev
2021-04-26 17:37 70% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-04-26 16:44 UTC (permalink / raw)
To: meta
Hello:
One of the services I think would be interesting to provide is ability for
people to subscribe to "curated saved searches". For example, a kernel
subsystem maintainer can define a set of query parameters (a thread mentions
these files/functions/terms, etc), and allow others to follow this saved
search either by defining it as a remote source for their own lei command, or
by subscribing to it as they would to any regular mailing list.
The latter is specifically something I think would be of interest to kernel
folks, so I envision that we'd have something like the following:
- a maintainer publishes a configuration file we can pass to lei
- our backend lei process uses all of lore.kernel.org sources to create and
continuously update a new public-inbox repository with matching search
results
- we set up a mlmmj list that doesn't receive any direct mail but is only fed
from saved search results; people can subscribe/unsubscribe as they would
with any other mlmmj list
Any particular reason this wouldn't work?
-K
^ permalink raw reply [relevance 70%]
* Re: lei-managed pseudo mailing lists
2021-04-26 16:44 70% lei-managed pseudo mailing lists Konstantin Ryabitsev
@ 2021-04-26 17:37 70% ` Eric Wong
2021-04-26 18:20 65% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-26 17:37 UTC (permalink / raw)
To: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
>
> One of the services I think would be interesting to provide is ability for
> people to subscribe to "curated saved searches". For example, a kernel
> subsystem maintainer can define a set of query parameters (a thread mentions
> these files/functions/terms, etc), and allow others to follow this saved
> search either by defining it as a remote source for their own lei command, or
> by subscribing to it as they would to any regular mailing list.
>
> The latter is specifically something I think would be of interest to kernel
> folks, so I envision that we'd have something like the following:
>
> - a maintainer publishes a configuration file we can pass to lei
The command-line might be enough, the pathname of the current
state/config file is a bit tricky and tied to its output.
I suppose "lei import-search" can be a command, though...
> - our backend lei process uses all of lore.kernel.org sources to create and
> continuously update a new public-inbox repository with matching search
> results
There's already some accomodations for that in LeiSavedSearch
which can present itself as a PublicInbox::Inbox-ish object to
PublicInbox::WWW (untested).
Searching an within LSS isn't implemented, yet, but I think it's
doable w/o extra Xapian storage.
However, git object storage isn't duplicated, which is nice for
local use (instaweb-like), but supporting clone/fetch isn't as
natural...
Perhaps supporting a v2 inbox as an lei q output destination
is in order:
lei q --output v2publicinbox:/path/to/v2 --shared SEARCH_TERMS
--shared would be "git clone --shared", the new v2 inbox can
use ~/.cache/lei/all_locals_ever.git/ as an alternate and not
duplicate space for blobs.
> - we set up a mlmmj list that doesn't receive any direct mail but is only fed
> from saved search results; people can subscribe/unsubscribe as they would
> with any other mlmmj list
>
> Any particular reason this wouldn't work?
Nope :) As long as all the data formats can interoperate
(mostly RFC5322/2822). "lei convert" is nice, too :)
^ permalink raw reply [relevance 70%]
* Re: lei-managed pseudo mailing lists
2021-04-26 17:37 70% ` Eric Wong
@ 2021-04-26 18:20 65% ` Konstantin Ryabitsev
2021-04-26 18:47 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-04-26 18:20 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Mon, Apr 26, 2021 at 05:37:26PM +0000, Eric Wong wrote:
> > The latter is specifically something I think would be of interest to kernel
> > folks, so I envision that we'd have something like the following:
> >
> > - a maintainer publishes a configuration file we can pass to lei
>
> The command-line might be enough, the pathname of the current
> state/config file is a bit tricky and tied to its output.
> I suppose "lei import-search" can be a command, though...
Excellent, excellent. How well does it deal with the situation when the search
parameters change?
> > - our backend lei process uses all of lore.kernel.org sources to create and
> > continuously update a new public-inbox repository with matching search
> > results
>
> There's already some accomodations for that in LeiSavedSearch
> which can present itself as a PublicInbox::Inbox-ish object to
> PublicInbox::WWW (untested).
>
> Searching an within LSS isn't implemented, yet, but I think it's
> doable w/o extra Xapian storage.
>
> However, git object storage isn't duplicated, which is nice for
> local use (instaweb-like), but supporting clone/fetch isn't as
> natural...
I'm thinking we need the ability to make it a real clonable repository --
perhaps without its own xapian index? Actual git repositories aren't large,
especially if they are only used for direct git operations. Disk space is
cheap, it's the IO that's expensive. :)
If these are real clonable repositories, then it would be easy for people to
set up replication for just the curated content people want.
> Perhaps supporting a v2 inbox as an lei q output destination
> is in order:
>
> lei q --output v2publicinbox:/path/to/v2 --shared SEARCH_TERMS
>
> --shared would be "git clone --shared", the new v2 inbox can
> use ~/.cache/lei/all_locals_ever.git/ as an alternate and not
> duplicate space for blobs.
Not really worried about deduping blobs, but I'm wondering how to make it work
well when search parameters change (see above). E.g.:
1. we create the repo with one set of parameters
2. maintainer then broadens it up to include something else
3. maintainer then decides that it's now *way* too much and narrows it down again
We don't really want step 2 to lead to a permanent ballooning of the
repository, so perhaps all query changes should force-append a dt: with the
open-ended datetime of the change? Or do you already have a way to deal with
this situation?
> > - we set up a mlmmj list that doesn't receive any direct mail but is only fed
> > from saved search results; people can subscribe/unsubscribe as they would
> > with any other mlmmj list
> >
> > Any particular reason this wouldn't work?
>
> Nope :) As long as all the data formats can interoperate
> (mostly RFC5322/2822). "lei convert" is nice, too :)
Great! I believe this will help untangle the current situation with "where
should I send this kernel patch".
I want "just send it to linux-kernel@vger.kernel.org" to be a valid option
again. Participating subsystems can then define what patches they want to see
by setting up pseudo-lists and letting participating reviewers/maintainers
subscribe to them via their preferred mail delivery mechanism.
-K
^ permalink raw reply [relevance 65%]
* Re: lei-managed pseudo mailing lists
2021-04-26 18:20 65% ` Konstantin Ryabitsev
@ 2021-04-26 18:47 71% ` Eric Wong
2021-04-26 19:46 68% ` Konstantin Ryabitsev
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-26 18:47 UTC (permalink / raw)
To: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Mon, Apr 26, 2021 at 05:37:26PM +0000, Eric Wong wrote:
> > > The latter is specifically something I think would be of interest to kernel
> > > folks, so I envision that we'd have something like the following:
> > >
> > > - a maintainer publishes a configuration file we can pass to lei
> >
> > The command-line might be enough, the pathname of the current
> > state/config file is a bit tricky and tied to its output.
> > I suppose "lei import-search" can be a command, though...
>
> Excellent, excellent. How well does it deal with the situation when the search
> parameters change?
"lei edit-search" can be used to zero the maxuid parameters;
and normal v2 deduplication will prevent duplicates from showing up.
It's not automatic, though; though that probably seems like a good
idea to keep manual, anyways, given the step 2. below.
> > > - our backend lei process uses all of lore.kernel.org sources to create and
> > > continuously update a new public-inbox repository with matching search
> > > results
> >
> > There's already some accomodations for that in LeiSavedSearch
> > which can present itself as a PublicInbox::Inbox-ish object to
> > PublicInbox::WWW (untested).
> >
> > Searching an within LSS isn't implemented, yet, but I think it's
> > doable w/o extra Xapian storage.
> >
> > However, git object storage isn't duplicated, which is nice for
> > local use (instaweb-like), but supporting clone/fetch isn't as
> > natural...
>
> I'm thinking we need the ability to make it a real clonable repository --
> perhaps without its own xapian index? Actual git repositories aren't large,
> especially if they are only used for direct git operations. Disk space is
> cheap, it's the IO that's expensive. :)
True, though cache overheads hurt a bit. I also wonder if lei
can increase traffic to public-inbox-<imapd|nntpd> to reduce
the need/use of "git clone".
> If these are real clonable repositories, then it would be easy for people to
> set up replication for just the curated content people want.
Understood. Using --output v2publicinbox:... w/o --shared is
totally doable.
> > Perhaps supporting a v2 inbox as an lei q output destination
> > is in order:
> >
> > lei q --output v2publicinbox:/path/to/v2 --shared SEARCH_TERMS
> >
> > --shared would be "git clone --shared", the new v2 inbox can
> > use ~/.cache/lei/all_locals_ever.git/ as an alternate and not
> > duplicate space for blobs.
>
> Not really worried about deduping blobs, but I'm wondering how to make it work
> well when search parameters change (see above). E.g.:
>
> 1. we create the repo with one set of parameters
> 2. maintainer then broadens it up to include something else
> 3. maintainer then decides that it's now *way* too much and narrows it down again
>
> We don't really want step 2 to lead to a permanent ballooning of the
> repository, so perhaps all query changes should force-append a dt: with the
> open-ended datetime of the change? Or do you already have a way to deal with
> this situation?
The aforementioned maxuid prevents stuff that's too old from
being seen. Otherwise, there's always "public-inbox-learn rm".
> > > - we set up a mlmmj list that doesn't receive any direct mail but is only fed
> > > from saved search results; people can subscribe/unsubscribe as they would
> > > with any other mlmmj list
> > >
> > > Any particular reason this wouldn't work?
> >
> > Nope :) As long as all the data formats can interoperate
> > (mostly RFC5322/2822). "lei convert" is nice, too :)
>
> Great! I believe this will help untangle the current situation with "where
> should I send this kernel patch".
>
> I want "just send it to linux-kernel@vger.kernel.org" to be a valid option
> again. Participating subsystems can then define what patches they want to see
> by setting up pseudo-lists and letting participating reviewers/maintainers
> subscribe to them via their preferred mail delivery mechanism.
Yup, that seems easiest for new contributors.
^ permalink raw reply [relevance 71%]
* Re: lei-managed pseudo mailing lists
2021-04-26 18:47 71% ` Eric Wong
@ 2021-04-26 19:46 68% ` Konstantin Ryabitsev
2021-04-26 20:34 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Konstantin Ryabitsev @ 2021-04-26 19:46 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
On Mon, Apr 26, 2021 at 06:47:17PM +0000, Eric Wong wrote:
> > I'm thinking we need the ability to make it a real clonable repository --
> > perhaps without its own xapian index? Actual git repositories aren't large,
> > especially if they are only used for direct git operations. Disk space is
> > cheap, it's the IO that's expensive. :)
>
> True, though cache overheads hurt a bit. I also wonder if lei
> can increase traffic to public-inbox-<imapd|nntpd> to reduce
> the need/use of "git clone".
>
> > If these are real clonable repositories, then it would be easy for people to
> > set up replication for just the curated content people want.
>
> Understood. Using --output v2publicinbox:... w/o --shared is
> totally doable.
I'm just worried that if we overuse the alternates, then we may find ourselves
in a situation where when we repack the "every blob" shared repository, we'll
end up with a pack that isn't really optimized to be used by any of the
member repos. So, in a situation where a clone is performed, git-upload-pack
will have to spend a lot of cycles navigating through the monstrous parent
pack just to build and re-compress the small subset of objects it needs to
send.
Git has ways of dealing with this by allowing to set things like pack islands,
but it's finicky and requires that each child repo is defined as refs in the
parent repo. We deal with this in grokmirror, but it's messy and requires
properly tracking child repo additions/removals/etc.
I think it may be one of those cases where wasting disk space on duplicate
objects is worth the CPU cycle savings.
> > Not really worried about deduping blobs, but I'm wondering how to make it work
> > well when search parameters change (see above). E.g.:
> >
> > 1. we create the repo with one set of parameters
> > 2. maintainer then broadens it up to include something else
> > 3. maintainer then decides that it's now *way* too much and narrows it down again
> >
> > We don't really want step 2 to lead to a permanent ballooning of the
> > repository, so perhaps all query changes should force-append a dt: with the
> > open-ended datetime of the change? Or do you already have a way to deal with
> > this situation?
>
> The aforementioned maxuid prevents stuff that's too old from
> being seen. Otherwise, there's always "public-inbox-learn rm".
How would it handle the situation where we import a new list into lore with a
10-year-long archive of messages?
-K
^ permalink raw reply [relevance 68%]
* Re: lei-managed pseudo mailing lists
2021-04-26 19:46 68% ` Konstantin Ryabitsev
@ 2021-04-26 20:34 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-26 20:34 UTC (permalink / raw)
To: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> I'm just worried that if we overuse the alternates, then we may find ourselves
> in a situation where when we repack the "every blob" shared repository, we'll
> end up with a pack that isn't really optimized to be used by any of the
> member repos. So, in a situation where a clone is performed, git-upload-pack
> will have to spend a lot of cycles navigating through the monstrous parent
> pack just to build and re-compress the small subset of objects it needs to
> send.
>
> Git has ways of dealing with this by allowing to set things like pack islands,
> but it's finicky and requires that each child repo is defined as refs in the
> parent repo. We deal with this in grokmirror, but it's messy and requires
> properly tracking child repo additions/removals/etc.
At least for personal use, I've been meaning to look into
automatically managing islands.
> I think it may be one of those cases where wasting disk space on duplicate
> objects is worth the CPU cycle savings.
Agreed for serving public inboxes.
> On Mon, Apr 26, 2021 at 06:47:17PM +0000, Eric Wong wrote:
> > The aforementioned maxuid prevents stuff that's too old from
> > being seen. Otherwise, there's always "public-inbox-learn rm".
>
> How would it handle the situation where we import a new list into lore with a
> 10-year-long archive of messages?
maxuid is either per-inbox or per-extindex.
If the search is going off of inboxes via --only, then it would
not see the new inbox at all. If it's on an extindex like
"all", then yes, the newly-imported historical messages would
show up.
So using "rt:" (Received time) is helpful in the [extindex "all"] case
Also, the approxidate parsing is done every time with "lei up",
so you can have a rolling window with "rt:last.week.." as a
search parameter.
^ permalink raw reply [relevance 71%]
* [PATCH 0/5] lei lcat - local cat (not lolcat :P)
@ 2021-04-27 11:07 64% Eric Wong
2021-04-27 11:07 56% ` [PATCH 1/5] lei: add "ls-sync" command for listing sync folders Eric Wong
` (4 more replies)
0 siblings, 5 replies; 200+ results
From: Eric Wong @ 2021-04-27 11:07 UTC (permalink / raw)
To: meta
"lei lcat" is a convenience command to extract Message-IDs
from URLs and <$MSGID> or "id:$MSGID" args (or stdin) and
attempt to display them as text.
--format=text is now the default for lcat, and an option for
"lei q" for stdout users. It decodes base64 and QP just like
the WWW interface. It also supports ANSI terminal colors and
loads the diff ones from the users' existing git config.
It's actually my first time using Term::ANSIColor, even though
it's bundled with Perl since 5.6.
I got sidetracked on the sync stuff, but "ls-sync" exists, now.
I'm not sure how sync would work, especially since I want to
avoid reconnecting for imports...
Eric Wong (5):
lei: add "ls-sync" command for listing sync folders
lei blob: support retrieving attachments via $OID:$IDX
lei: standardize on _lei_wq_eof callback for workers
lei lcat: extract Message-IDs from URLs and show them
lei q + lcat: support --format=text output
MANIFEST | 5 +
lib/PublicInbox/Hval.pm | 2 +-
lib/PublicInbox/LEI.pm | 12 +-
lib/PublicInbox/LeiBlob.pm | 37 ++++-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiExternal.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 6 +-
lib/PublicInbox/LeiLcat.pm | 125 +++++++++++++++++
lib/PublicInbox/LeiLsSync.pm | 29 ++++
lib/PublicInbox/LeiMirror.pm | 6 +-
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 8 +-
lib/PublicInbox/LeiToMail.pm | 63 ++++++++-
lib/PublicInbox/LeiViewText.pm | 237 +++++++++++++++++++++++++++++++++
lib/PublicInbox/ViewDiff.pm | 4 +-
t/lei-import-imap.t | 8 ++
t/lei-import-maildir.t | 3 +
t/lei-lcat.t | 16 +++
t/lei_lcat.t | 44 ++++++
19 files changed, 584 insertions(+), 27 deletions(-)
create mode 100644 lib/PublicInbox/LeiLcat.pm
create mode 100644 lib/PublicInbox/LeiLsSync.pm
create mode 100644 lib/PublicInbox/LeiViewText.pm
create mode 100644 t/lei-lcat.t
create mode 100644 t/lei_lcat.t
^ permalink raw reply [relevance 64%]
* [PATCH 2/5] lei blob: support retrieving attachments via $OID:$IDX
2021-04-27 11:07 64% [PATCH 0/5] lei lcat - local cat (not lolcat :P) Eric Wong
2021-04-27 11:07 56% ` [PATCH 1/5] lei: add "ls-sync" command for listing sync folders Eric Wong
@ 2021-04-27 11:07 61% ` Eric Wong
2021-04-27 11:07 49% ` [PATCH 3/5] lei: standardize on _lei_wq_eof callback for workers Eric Wong
` (2 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-27 11:07 UTC (permalink / raw)
To: meta
We'll be supporting some sort of text view for pager or
piping to an $EDITOR buffer.
---
lib/PublicInbox/LeiBlob.pm | 32 ++++++++++++++++++++++++++++++--
t/lei-import-imap.t | 6 ++++++
2 files changed, 36 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index e4cd4cca..4e52c8a5 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -84,12 +84,25 @@ sub do_solve_blob { # via wq_do
$solver->solve($lei->{env}, $log, $self->{oid_b}, $hints);
}
+sub cat_attach_i { # Eml->each_part callback
+ my ($part, $depth, $idx) = @{$_[0]};
+ my $lei = $_[1];
+ my $want = $lei->{-attach_idx} // return;
+ return if $idx ne $want; # [0-9]+(?:\.[0-9]+)+
+ delete $lei->{-attach_idx};
+ $lei->out($part->body);
+}
+
sub lei_blob {
my ($lei, $blob) = @_;
$lei->start_pager if -t $lei->{1};
my $opt = $lei->{opt};
my $has_hints = grep(defined, @$opt{qw(oid-a path-a path-b)});
my $lxs;
+ if ($blob =~ s/:([0-9\.]+)\z//) {
+ $lei->{-attach_idx} = $1;
+ $opt->{mail} = 1;
+ }
# first, see if it's a blob returned by "lei q" JSON output:k
if ($opt->{mail} // ($has_hints ? 0 : 1)) {
@@ -97,7 +110,7 @@ sub lei_blob {
$lxs = $lei->lxs_prepare;
$lei->ale->refresh_externals($lxs);
}
- my $rdr = { 1 => $lei->{1} };
+ my $rdr = {};
if ($opt->{mail}) {
$rdr->{2} = $lei->{2};
} else {
@@ -105,7 +118,22 @@ sub lei_blob {
}
my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
'cat-file', 'blob', $blob ];
- waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ if (defined $lei->{-attach_idx}) {
+ my $fh = popen_rd($cmd, $lei->{env}, $rdr);
+ require PublicInbox::Eml;
+ my $str = do { local $/; <$fh> };
+ if (close $fh) {
+ my $eml = PublicInbox::Eml->new(\$str);
+ $eml->each_part(\&cat_attach_i, $lei, 1);
+ my $idx = delete $lei->{-attach_idx};
+ defined($idx) and return $lei->fail(<<EOM);
+E: attachment $idx not found in $blob
+EOM
+ }
+ } else {
+ $rdr->{1} = $lei->{1};
+ waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ }
return if $? == 0;
return $lei->child_error($?) if $opt->{mail};
}
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 376a8b48..cf1fa49d 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -52,5 +52,11 @@ test_lei({ tmpdir => $tmpdir }, sub {
is(ref($x->{'lei/store'}), 'ARRAY', 'lei/store in inspect');
is(ref($x->{sync}), 'HASH', 'sync in inspect');
is(ref($x->{sync}->{$k[0]}), 'ARRAY', 'UID arrays in inspect');
+
+ my $psgi_attach = 'cfa3622cbeffc9bd6b0fc66c4d60d420ba74f60d';
+ lei_ok('blob', $psgi_attach);
+ like($lei_out, qr!^Content-Type: multipart/mixed;!sm, 'got full blob');
+ lei_ok('blob', "$psgi_attach:2");
+ is($lei_out, "b64\xde\xad\xbe\xef\n", 'got attachment');
});
done_testing;
^ permalink raw reply related [relevance 61%]
* [PATCH 1/5] lei: add "ls-sync" command for listing sync folders
2021-04-27 11:07 64% [PATCH 0/5] lei lcat - local cat (not lolcat :P) Eric Wong
@ 2021-04-27 11:07 56% ` Eric Wong
2021-04-27 11:07 61% ` [PATCH 2/5] lei blob: support retrieving attachments via $OID:$IDX Eric Wong
` (3 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-27 11:07 UTC (permalink / raw)
To: meta
This will be useful, later.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 ++
lib/PublicInbox/LeiExternal.pm | 2 +-
lib/PublicInbox/LeiLsSync.pm | 29 +++++++++++++++++++++++++++++
t/lei-import-imap.t | 2 ++
t/lei-import-maildir.t | 3 +++
6 files changed, 38 insertions(+), 1 deletion(-)
create mode 100644 lib/PublicInbox/LeiLsSync.pm
diff --git a/MANIFEST b/MANIFEST
index ce824fcf..d4e7d66f 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -202,6 +202,7 @@ lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiInspect.pm
lib/PublicInbox/LeiLsLabel.pm
lib/PublicInbox/LeiLsSearch.pm
+lib/PublicInbox/LeiLsSync.pm
lib/PublicInbox/LeiMailSync.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 39278de6..c170572b 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -161,6 +161,8 @@ our %CMD = ( # sorted in order of importance/use:
'ls-external' => [ '[FILTER]', 'list publicinbox|extindex locations',
qw(format|f=s z|0 globoff|g invert-match|v local remote), @c_opt ],
'ls-label' => [ '', 'list labels', qw(z|0 stats:s), @c_opt ],
+'ls-sync' => [ '', 'list sync folders',
+ qw(z|0 z|0 globoff|g invert-match|v local remote), @c_opt ],
'forget-external' => [ 'LOCATION...|--prune',
'exclude further results from a publicinbox|extindex',
qw(prune), @c_opt ],
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index b0ebe947..3858085e 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -50,7 +50,7 @@ my %re_map = ( '*' => '[^/]*?', '?' => '[^/]',
'[' => '[', ']' => ']', ',' => ',' );
sub glob2re {
- my ($re) = @_;
+ my $re = $_[-1];
my $p = '';
my $in_bracket = 0;
my $qm = 0;
diff --git a/lib/PublicInbox/LeiLsSync.pm b/lib/PublicInbox/LeiLsSync.pm
new file mode 100644
index 00000000..71f111a9
--- /dev/null
+++ b/lib/PublicInbox/LeiLsSync.pm
@@ -0,0 +1,29 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# front-end for the "lei ls-sync" sub-command
+package PublicInbox::LeiLsSync;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiMailSync;
+
+sub lei_ls_sync {
+ my ($lei, $filter) = @_;
+ my $sto = $lei->_lei_store or return;
+ my $lms = $sto->search->lms or return;
+ my $opt = $lei->{opt};
+ my $re;
+ $re = defined($filter) ? qr/\Q$filter\E/ : qr/./ if $opt->{globoff};
+ $re //= $lei->glob2re($filter // '*');
+ my @f = $lms->folders;
+ @f = $opt->{'invert-match'} ? grep(!/$re/, @f) : grep(/$re/, @f);
+ if ($opt->{'local'} && !$opt->{remote}) {
+ @f = grep(!m!\A[a-z\+]+://!i, @f);
+ } elsif ($opt->{remote} && !$opt->{'local'}) {
+ @f = grep(m!\A[a-z\+]+://!i, @f);
+ }
+ my $ORS = $opt->{z} ? "\0" : "\n";
+ $lei->out(join($ORS, @f, ''));
+}
+
+1;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 4a3bd6d8..376a8b48 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -22,6 +22,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
is_deeply(json_utf8->decode($lei_out), {}, 'no inspect stats, yet');
lei_ok('import', $url);
+ lei_ok 'ls-sync';
+ like($lei_out, qr!\A\Q$url\E;UIDVALIDITY=\d+\n\z!, 'ls-sync');
lei_ok('inspect', $url);
my $inspect = json_utf8->decode($lei_out);
diff --git a/t/lei-import-maildir.t b/t/lei-import-maildir.t
index 3e3d9188..808e1a73 100644
--- a/t/lei-import-maildir.t
+++ b/t/lei-import-maildir.t
@@ -40,6 +40,9 @@ test_lei(sub {
is_deeply($inspect, { sync => { "maildir:$md" => [ 'x:2,S' ] } },
'maildir sync info as expected');
+ lei_ok qw(ls-sync);
+ is($lei_out, "maildir:$md\n", 'ls-sync as expected');
+
lei_ok(qw(import), $md, \'import Maildir again');
$imp_err = $lei_err;
lei_ok(qw(q -d none s:boolean), \'lei q w/o dedupe');
^ permalink raw reply related [relevance 56%]
* [PATCH 3/5] lei: standardize on _lei_wq_eof callback for workers
2021-04-27 11:07 64% [PATCH 0/5] lei lcat - local cat (not lolcat :P) Eric Wong
2021-04-27 11:07 56% ` [PATCH 1/5] lei: add "ls-sync" command for listing sync folders Eric Wong
2021-04-27 11:07 61% ` [PATCH 2/5] lei blob: support retrieving attachments via $OID:$IDX Eric Wong
@ 2021-04-27 11:07 49% ` Eric Wong
2021-04-27 11:07 38% ` [PATCH 4/5] lei lcat: extract Message-IDs from URLs and show them Eric Wong
2021-04-27 11:07 32% ` [PATCH 5/5] lei q + lcat: support --format=text output Eric Wong
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-27 11:07 UTC (permalink / raw)
To: meta
Simplify our internals a little bit.
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiBlob.pm | 5 ++---
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 6 +++---
lib/PublicInbox/LeiMirror.pm | 6 ++----
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 8 ++++----
7 files changed, 14 insertions(+), 17 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index c170572b..effc905a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -531,7 +531,7 @@ sub workers_start {
'child_error' => [ \&child_error, $lei ],
($ops ? %$ops : ()),
};
- $ops->{''} //= [ \&dclose, $lei ];
+ $ops->{''} //= [ $wq->can('_lei_wq_eof') || \&dclose, $lei ];
my $end = $lei->pkt_op_pair;
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
delete $lei->{pkt_op_p};
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 4e52c8a5..0b96bd04 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -10,7 +10,7 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Spawn qw(spawn popen_rd which);
use PublicInbox::DS;
-sub sol_done { # EOF callback for main daemon
+sub _lei_wq_eof { # EOF callback for main daemon
my ($lei) = @_;
my $sol = delete $lei->{sol} // return $lei->dclose; # already failed
$sol->wq_wait_old($lei->can('wq_done_wait'), $lei);
@@ -157,8 +157,7 @@ EOM
}
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
- my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
- { '' => [ \&sol_done, $lei ] });
+ my ($op_c, $ops) = $lei->workers_start($self, 'lei-blob', 1);
$lei->{sol} = $self;
$self->wq_io_do('do_solve_blob', []);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 0ce49ea9..0c324169 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -52,7 +52,7 @@ sub lei_convert { # the main "lei convert" method
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
- my ($op_c, $ops) = $lei->workers_start($self, 'lei_convert', 1);
+ my ($op_c, $ops) = $lei->workers_start($self, 'lei-convert', 1);
$lei->{cnv} = $self;
$self->wq_io_do('process_inputs', []);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index daaa6753..e0d899cc 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -53,7 +53,7 @@ sub input_nntp_cb { # nntp_each
input_eml_cb($self, $eml, $self->{-import_kw} ? { kw => $kw } : undef);
}
-sub import_done { # EOF callback for main daemon
+sub _lei_wq_eof { # EOF callback for main daemon
my ($lei) = @_;
my $imp = delete $lei->{imp} // return $lei->fail('BUG: {imp} gone');
$imp->wq_wait_old($lei->can('wq_done_wait'), $lei, 'non-fatal');
@@ -90,10 +90,10 @@ sub lei_import { # the main "lei import" method
my $nproc = $self->detect_nproc;
$j = $nproc if $j > $nproc;
}
- my $ops = { '' => [ \&import_done, $lei ] };
+ my $ops = {};
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{-wq_nr_workers} = $j // 1; # locked
- (my $op_c, $ops) = $lei->workers_start($self, 'lei_import', $j, $ops);
+ (my $op_c, $ops) = $lei->workers_start($self, 'lei-import', $j, $ops);
$lei->{imp} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index 15adb71b..50ab4c85 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -24,7 +24,7 @@ sub do_finish_mirror { # dwaitpid callback
$lei->dclose;
}
-sub mirror_done { # EOF callback for main daemon
+sub _lei_wq_eof { # EOF callback for main daemon
my ($lei) = @_;
my $mrr = delete $lei->{mrr} or return;
$mrr->wq_wait_old(\&do_finish_mirror, $lei);
@@ -282,9 +282,7 @@ sub start {
require PublicInbox::Inbox;
require PublicInbox::Admin;
require PublicInbox::InboxWritable;
- my ($op, $ops) = $lei->workers_start($self, 'lei_mirror', 1, {
- '' => [ \&mirror_done, $lei ]
- });
+ my ($op, $ops) = $lei->workers_start($self, 'lei_mirror', 1);
$lei->{mrr} = $self;
$self->wq_io_do('do_mirror', []);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index cb2309c7..3248afd7 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -188,7 +188,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
} else {
$self->{input} = $input;
}
- my ($op, $ops) = $lei->workers_start($self, 'lei_p2q', 1);
+ my ($op, $ops) = $lei->workers_start($self, 'lei-p2q', 1);
$lei->{p2q} = $self;
$self->wq_io_do('do_p2q', []);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index f5791947..3cda2eca 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -19,9 +19,9 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub input_mbox_cb { input_eml_cb($_[1], $_[0]) }
-sub tag_done { # EOF callback for main daemon
+sub _lei_wq_eof { # EOF callback for main daemon
my ($lei) = @_;
- my $tag = delete $lei->{tag} or return;
+ my $tag = delete $lei->{tag} // return $lei->dclose;
$tag->wq_wait_old($lei->can('wq_done_wait'), $lei, 'non-fatal');
}
@@ -52,11 +52,11 @@ sub lei_tag { # the "lei tag" method
$self->prepare_inputs($lei, \@argv) or return;
grep(defined, @$vmd_mod{qw(+kw +L -L -kw)}) or
return $lei->fail('no keywords or labels specified');
- my $ops = { '' => [ \&tag_done, $lei ] };
+ my $ops = {};
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
my $j = $self->{-wq_nr_workers} = 1; # locked for now
- (my $op_c, $ops) = $lei->workers_start($self, 'lei_tag', $j, $ops);
+ (my $op_c, $ops) = $lei->workers_start($self, 'lei-tag', $j, $ops);
$lei->{tag} = $self;
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
^ permalink raw reply related [relevance 49%]
* [PATCH 4/5] lei lcat: extract Message-IDs from URLs and show them
2021-04-27 11:07 64% [PATCH 0/5] lei lcat - local cat (not lolcat :P) Eric Wong
` (2 preceding siblings ...)
2021-04-27 11:07 49% ` [PATCH 3/5] lei: standardize on _lei_wq_eof callback for workers Eric Wong
@ 2021-04-27 11:07 38% ` Eric Wong
2021-04-27 11:07 32% ` [PATCH 5/5] lei q + lcat: support --format=text output Eric Wong
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-27 11:07 UTC (permalink / raw)
To: meta
It's a wrapper around "lei q" which extracts Message-IDs
from URLs, "<$MSGID>", "id:$MSGID" and attempts to display the
local version of the message.
Its main purpose is to extract Message-IDs out of
commonly-understood URLs to save users bandwidth and time
by displaying the message locally. When reading from stdin,
it will discard things it doesn't understand, so you can just
pipe an entire "Link: $URL" line to it and it'll attempt to
pluck the Message-ID out of the URL.
---
MANIFEST | 3 +
lib/PublicInbox/LEI.pm | 8 +++
lib/PublicInbox/LeiLcat.pm | 125 +++++++++++++++++++++++++++++++++++++
t/lei-lcat.t | 16 +++++
t/lei_lcat.t | 44 +++++++++++++
5 files changed, 196 insertions(+)
create mode 100644 lib/PublicInbox/LeiLcat.pm
create mode 100644 t/lei-lcat.t
create mode 100644 t/lei_lcat.t
diff --git a/MANIFEST b/MANIFEST
index d4e7d66f..d3b46f8b 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -200,6 +200,7 @@ lib/PublicInbox/LeiImport.pm
lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiInspect.pm
+lib/PublicInbox/LeiLcat.pm
lib/PublicInbox/LeiLsLabel.pm
lib/PublicInbox/LeiLsSearch.pm
lib/PublicInbox/LeiLsSync.pm
@@ -400,6 +401,7 @@ t/lei-import-imap.t
t/lei-import-maildir.t
t/lei-import-nntp.t
t/lei-import.t
+t/lei-lcat.t
t/lei-mirror.t
t/lei-p2q.t
t/lei-q-kw.t
@@ -411,6 +413,7 @@ t/lei-tag.t
t/lei.t
t/lei_dedupe.t
t/lei_external.t
+t/lei_lcat.t
t/lei_mail_sync.t
t/lei_overview.t
t/lei_saved_search.t
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index effc905a..ef72758c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -149,6 +149,14 @@ our %CMD = ( # sorted in order of importance/use:
'up' => [ 'OUTPUT|--all', 'update saved search',
qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all:s), @c_opt ],
+'lcat' => [ '--stdin|MSGID_OR_URL..', 'display local copy of message(s)',
+ 'stdin|', # /|\z/ must be first for lone dash
+ # some of these options are ridiculous for lcat
+ @lxs_opt, qw(output|mfolder|o=s format|f=s dedupe|d=s threads|t+
+ sort|s=s reverse|r offset=i jobs|j=s globoff|g augment|a
+ import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
+ opt_dash('limit|n=i', '[0-9]+') ],
+
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
@lxs_opt, @c_opt ],
diff --git a/lib/PublicInbox/LeiLcat.pm b/lib/PublicInbox/LeiLcat.pm
new file mode 100644
index 00000000..f10452be
--- /dev/null
+++ b/lib/PublicInbox/LeiLcat.pm
@@ -0,0 +1,125 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# lcat: local cat, display a local message by Message-ID or blob,
+# extracting from URL necessary
+# "lei lcat <URL|SPEC>"
+package PublicInbox::LeiLcat;
+use strict;
+use v5.10.1;
+use PublicInbox::LeiViewText;
+use URI::Escape qw(uri_unescape);
+use URI;
+use PublicInbox::MID qw($MID_EXTRACT);
+
+sub lcat_redispatch {
+ my ($lei, $out, $op_p) = @_;
+ my $l = bless { %$lei }, ref($lei);
+ delete $l->{sock};
+ $l->{''} = $op_p; # daemon only
+ eval {
+ $l->qerr("# updating $out");
+ up1($l, $out);
+ $l->qerr("# $out done");
+ };
+ $l->err($@) if $@;
+}
+
+sub extract_1 ($$) {
+ my ($lei, $x) = @_;
+ if ($x =~ m!\b([a-z]+?://\S+)!i) {
+ my $u = $1;
+ $u =~ s/[\>\]\)\,\.\;]+\z//;
+ $u = URI->new($u);
+ my $p = $u->path;
+ my $term;
+ if ($p =~ m!([^/]+\@[^/]+)!) { # common msgid pattern
+ $term = 'mid:'.uri_unescape($1);
+
+ # is it a URL which returns the full thread?
+ if ($u->scheme =~ /\Ahttps?/i &&
+ $p =~ m!/(?:T/?|t/?|t\.mbox\.gz|t\.atom)\b!) {
+
+ $lei->{mset_opt}->{threads} = 1;
+ }
+ } elsif ($u->scheme =~ /\Ahttps?/i &&
+ # some msgids don't have '@', see if it looks like
+ # a public-inbox URL:
+ $p =~ m!/([^/]+)/(raw|t/?|T/?|
+ t\.mbox\.gz|t\.atom)\z!x) {
+ $lei->{mset_opt}->{threads} = 1 if $2 && $2 ne 'raw';
+ $term = 'mid:'.uri_unescape($1);
+ }
+ $term;
+ } elsif ($x =~ $MID_EXTRACT) { # <$MSGID>
+ "mid:$1";
+ } elsif ($x =~ /\b((?:m|mid):\S+)/) { # our own prefixes (and mairix)
+ $1;
+ } elsif ($x =~ /\bid:(\S+)/) { # notmuch convention
+ "mid:$1";
+ } else {
+ undef;
+ }
+}
+
+sub extract_all {
+ my ($lei, @argv) = @_;
+ my $strict = !$lei->{opt}->{stdin};
+ my @q;
+ for my $x (@argv) {
+ if (my $term = extract_1($lei,$x)) {
+ push @q, $term;
+ } elsif ($strict) {
+ return $lei->fail(<<"");
+could not extract Message-ID from $x
+
+ }
+ }
+ @q ? join(' OR ', @q) : $lei->fail("no Message-ID in: @argv");
+}
+
+sub _stdin { # PublicInbox::InputPipe::consume callback for --stdin
+ my ($lei) = @_; # $_[1] = $rbuf
+ if (defined($_[1])) {
+ $_[1] eq '' and return eval {
+ if (my $dfd = $lei->{3}) {
+ chdir($dfd) or return $lei->fail("fchdir: $!");
+ }
+ my @argv = split(/\s+/, $lei->{mset_opt}->{qstr});
+ $lei->{mset_opt}->{qstr} = extract_all($lei, @argv)
+ or return;
+ $lei->_start_query;
+ };
+ $lei->{mset_opt}->{qstr} .= $_[1];
+ } else {
+ $lei->fail("error reading stdin: $!");
+ }
+}
+
+sub lei_lcat {
+ my ($lei, @argv) = @_;
+ my $lxs = $lei->lxs_prepare or return;
+ $lei->ale->refresh_externals($lxs);
+ my $sto = $lei->_lei_store(1);
+ $lei->{lse} = $sto->search;
+ my $opt = $lei->{opt};
+ my %mset_opt = map { $_ => $opt->{$_} } qw(threads limit offset);
+ $mset_opt{asc} = $opt->{'reverse'} ? 1 : 0;
+ $mset_opt{limit} //= 10000;
+ $opt->{sort} //= 'relevance';
+ $mset_opt{relevance} = 1;
+ $lei->{mset_opt} = \%mset_opt;
+ $opt->{'format'} //= 'mboxrd' unless defined($opt->{output});
+ if ($lei->{opt}->{stdin}) {
+ return $lei->fail(<<'') if @argv;
+no args allowed on command-line with --stdin
+
+ require PublicInbox::InputPipe;
+ PublicInbox::InputPipe::consume($lei->{0}, \&_stdin, $lei);
+ return;
+ }
+ $lei->{mset_opt}->{qstr} = extract_all($lei, @argv) or return;
+ $lei->_start_query;
+}
+
+1;
diff --git a/t/lei-lcat.t b/t/lei-lcat.t
new file mode 100644
index 00000000..e5f00706
--- /dev/null
+++ b/t/lei-lcat.t
@@ -0,0 +1,16 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10.1; use PublicInbox::TestCommon;
+require_mods(qw(lei));
+
+test_lei(sub {
+ my $in = "\nMessage-id: <qp\@example.com>\n";
+ lei_ok([qw(lcat --stdin)], undef, { 0 => \$in, %$lei_opt });
+ unlike($lei_out, qr/\S/, 'nothing, yet');
+ lei_ok('import', 't/plack-qp.eml');
+ lei_ok([qw(lcat --stdin)], undef, { 0 => \$in, %$lei_opt });
+ like($lei_out, qr/qp\@example\.com/, 'got a result');
+});
+
+done_testing;
diff --git a/t/lei_lcat.t b/t/lei_lcat.t
new file mode 100644
index 00000000..536abdea
--- /dev/null
+++ b/t/lei_lcat.t
@@ -0,0 +1,44 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+# unit test for "lei lcat" internals, see t/lei-lcat.t for functional test
+use strict;
+use v5.10.1;
+use Test::More;
+use_ok 'PublicInbox::LeiLcat';
+my $cb = \&PublicInbox::LeiLcat::extract_1;
+my $ck = sub {
+ my ($txt, $exp, $t) = @_;
+ my $lei = {};
+ is($cb->($lei, $txt), $exp, $txt);
+ ($t ? is_deeply($lei, { mset_opt => { threads => 1 } }, "-t $exp")
+ : is_deeply($lei, {}, "no -t for $exp")) or diag explain($lei);
+};
+
+for my $txt (qw(https://example.com/inbox/foo@bar/
+ https://example.com/inbox/foo@bar
+ https://example.com/inbox/foo@bar/raw
+ id:foo@bar
+ mid:foo@bar
+ <foo@bar>
+ <https://example.com/inbox/foo@bar>
+ <https://example.com/inbox/foo@bar/raw>
+ <https://example.com/inbox/foo@bar/>
+ <nntp://example.com/foo@bar>)) {
+ $ck->($txt, 'mid:foo@bar');
+}
+
+for my $txt (qw(https://example.com/inbox/foo@bar/T/
+ https://example.com/inbox/foo@bar/t/
+ https://example.com/inbox/foo@bar/t.mbox.gz
+ <https://example.com/inbox/foo@bar/t.atom>
+ <https://example.com/inbox/foo@bar/t/>)) {
+ $ck->($txt, 'mid:foo@bar', '-t');
+}
+
+$ck->('https://example.com/x/foobar/T/', 'mid:foobar', '-t');
+$ck->('https://example.com/x/foobar/raw', 'mid:foobar');
+is($cb->(my $lei = {}, 'asdf'), undef, 'no Message-ID');
+is($cb->($lei = {}, 'm:x'), 'm:x', 'bare m: accepted');
+
+done_testing;
^ permalink raw reply related [relevance 38%]
* [PATCH 5/5] lei q + lcat: support --format=text output
2021-04-27 11:07 64% [PATCH 0/5] lei lcat - local cat (not lolcat :P) Eric Wong
` (3 preceding siblings ...)
2021-04-27 11:07 38% ` [PATCH 4/5] lei lcat: extract Message-IDs from URLs and show them Eric Wong
@ 2021-04-27 11:07 32% ` Eric Wong
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-27 11:07 UTC (permalink / raw)
To: meta
This is mainly for "lei lcat" where it's the default,
but I find it useful anyways compared to the JSON view.
Colors are loaded from ~/.config/lei/config, and fall back
to using diff colors from a normal git config
(e.g. ~/.gitconfig).
---
MANIFEST | 1 +
lib/PublicInbox/Hval.pm | 2 +-
lib/PublicInbox/LeiLcat.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 63 ++++++++-
lib/PublicInbox/LeiViewText.pm | 237 +++++++++++++++++++++++++++++++++
lib/PublicInbox/ViewDiff.pm | 4 +-
6 files changed, 301 insertions(+), 8 deletions(-)
create mode 100644 lib/PublicInbox/LeiViewText.pm
diff --git a/MANIFEST b/MANIFEST
index d3b46f8b..5933ddf4 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -218,6 +218,7 @@ lib/PublicInbox/LeiSucks.pm
lib/PublicInbox/LeiTag.pm
lib/PublicInbox/LeiToMail.pm
lib/PublicInbox/LeiUp.pm
+lib/PublicInbox/LeiViewText.pm
lib/PublicInbox/LeiXSearch.pm
lib/PublicInbox/Linkify.pm
lib/PublicInbox/Listener.pm
diff --git a/lib/PublicInbox/Hval.pm b/lib/PublicInbox/Hval.pm
index eab4738e..00b3c8b4 100644
--- a/lib/PublicInbox/Hval.pm
+++ b/lib/PublicInbox/Hval.pm
@@ -34,7 +34,7 @@ my %escape_sequence = (
"\x7f" => '\\x7f', # DEL
);
-my %xhtml_map = (
+our %xhtml_map = (
'"' => '"',
'&' => '&',
"'" => ''',
diff --git a/lib/PublicInbox/LeiLcat.pm b/lib/PublicInbox/LeiLcat.pm
index f10452be..87729acf 100644
--- a/lib/PublicInbox/LeiLcat.pm
+++ b/lib/PublicInbox/LeiLcat.pm
@@ -109,7 +109,7 @@ sub lei_lcat {
$opt->{sort} //= 'relevance';
$mset_opt{relevance} = 1;
$lei->{mset_opt} = \%mset_opt;
- $opt->{'format'} //= 'mboxrd' unless defined($opt->{output});
+ $opt->{'format'} //= 'text' unless defined($opt->{output});
if ($lei->{opt}->{stdin}) {
return $lei->fail(<<'') if @argv;
no args allowed on command-line with --stdin
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 8b2f82dc..fa3af710 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -309,6 +309,26 @@ sub _imap_write_cb ($$) {
}
}
+sub _text_write_cb ($$) {
+ my ($self, $lei) = @_;
+ my $dedupe = $lei->{dedupe};
+ $dedupe->prepare_dedupe if $dedupe;
+ my $lvt = $lei->{lvt};
+ my $ovv = $lei->{ovv};
+ $lei->{1} // die "no stdout ($ovv->{dst})"; # redirected earlier
+ $lei->{1}->autoflush(1);
+ binmode $lei->{1}, ':utf8';
+ my $lse = $lei->{lse}; # may be undef
+ sub { # for git_to_mail
+ my ($bref, $smsg, $eml) = @_;
+ $lse->xsmsg_vmd($smsg) if $lse;
+ $eml //= PublicInbox::Eml->new($bref); # copy bref
+ return if $dedupe && $dedupe->is_dup($eml, $smsg);
+ my $lk = $ovv->lock_for_scope;
+ $lei->out(${$lvt->eml_to_text($smsg, $eml)}, "\n");
+ }
+}
+
sub write_cb { # returns a callback for git_to_mail
my ($self, $lei) = @_;
# _mbox_write_cb, _maildir_write_cb or _imap_write_cb
@@ -329,8 +349,6 @@ sub new {
$lei->{ovv}->{dst} = $dst .= '/' if substr($dst, -1) ne '/';
} elsif (substr($fmt, 0, 4) eq 'mbox') {
require PublicInbox::MboxReader;
- (-d $dst || (-e _ && !-w _)) and die
- "$dst exists and is not a writable file\n";
$self->can("eml2$fmt") or die "bad mbox format: $fmt\n";
$self->{base_type} = 'mbox';
} elsif ($fmt =~ /\Aimaps?\z/) { # TODO .onion support
@@ -347,9 +365,23 @@ sub new {
$dst = $lei->{ovv}->{dst} = $$uri; # canonicalized
$lei->{net} = $net;
$self->{base_type} = 'imap';
+ } elsif ($fmt eq 'text') {
+ require PublicInbox::LeiViewText;
+ $lei->{lvt} = PublicInbox::LeiViewText->new($lei);
+ $self->{base_type} = 'text';
} else {
die "bad mail --format=$fmt\n";
}
+ if ($self->{base_type} =~ /\A(?:text|mbox)\z/) {
+ (-d $dst || (-e _ && !-w _)) and die
+ "$dst exists and is not a writable file\n";
+ }
+ if ($self->{base_type} eq 'text') {
+ my @err = map {
+ defined($lei->{opt}->{$_}) ? "--$_" : ();
+ } (qw(mua save));
+ die "@err incompatible with $fmt\n" if @err;
+ }
$self->{dst} = $dst;
$lei->{dedupe} = $lei->{lss} // do {
my $dd_cls = 'PublicInbox::'.
@@ -429,6 +461,29 @@ sub _do_augment_imap {
}
}
+sub _pre_augment_text {
+ my ($self, $lei) = @_;
+ my $dst = $lei->{ovv}->{dst};
+ my $out;
+ my $devfd = $lei->path_to_fd($dst) // die "bad $dst";
+ if ($devfd >= 0) {
+ $out = $lei->{$devfd};
+ } else { # normal-looking path
+ if (-p $dst) {
+ open $out, '>', $dst or die "open($dst): $!";
+ } elsif (-f _ || !-e _) {
+ # text allows augment, HTML/Atom won't
+ my $mode = $lei->{opt}->{augment} ? '>>' : '>';
+ open $out, $mode, $dst or die "open($mode, $dst): $!";
+ } else {
+ die "$dst is not a file or FIFO\n";
+ }
+ }
+ $lei->{ovv}->ovv_out_lk_init if !$lei->{ovv}->{lock_path};
+ $lei->{1} = $out;
+ undef;
+}
+
sub _pre_augment_mbox {
my ($self, $lei) = @_;
my $dst = $lei->{ovv}->{dst};
@@ -523,8 +578,8 @@ sub pre_augment { # fast (1 disk seek), runs in same process as post_augment
sub do_augment { # slow, runs in wq worker
my ($self, $lei) = @_;
# _do_augment_maildir, _do_augment_mbox, or _do_augment_imap
- my $m = "_do_augment_$self->{base_type}";
- $self->$m($lei);
+ my $m = $self->can("_do_augment_$self->{base_type}") or return;
+ $m->($self, $lei);
}
# fast (spawn compressor or mkdir), runs in same process as pre_augment
diff --git a/lib/PublicInbox/LeiViewText.pm b/lib/PublicInbox/LeiViewText.pm
new file mode 100644
index 00000000..6f5fca49
--- /dev/null
+++ b/lib/PublicInbox/LeiViewText.pm
@@ -0,0 +1,237 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# PublicInbox::Eml to (optionally colorized) text coverter for terminals
+# the non-HTML counterpart to PublicInbox::View
+package PublicInbox::LeiViewText;
+use strict;
+use v5.10.1;
+use PublicInbox::MsgIter qw(msg_part_text);
+use PublicInbox::ContentHash qw(git_sha);
+use PublicInbox::MID qw(references);
+use PublicInbox::View;
+use PublicInbox::Hval;
+use PublicInbox::ViewDiff;
+use PublicInbox::Spawn qw(popen_rd);
+use Term::ANSIColor;
+
+sub _xs {
+ # xhtml_map works since we don't search for HTML ([&<>'"])
+ $_[0] =~ s/([\x7f\x00-\x1f])/$PublicInbox::Hval::xhtml_map{$1}/sge;
+}
+
+my %DEFAULT_COLOR = (
+ # mutt names, loaded from ~/.config/lei/config
+ quoted => 'blue',
+ hdrdefault => 'cyan',
+ status => 'bright_cyan', # smsg stuff
+
+ # git names and defaults, falls back to ~/.gitconfig
+ new => 'green',
+ old => 'red',
+ meta => 'bold',
+ frag => 'cyan',
+ func => undef,
+ context => undef,
+);
+
+sub my_colored {
+ my ($self, $slot) = @_; # $_[2] = buffer
+ my $val = $self->{"color.$slot"} //=
+ $self->{-leicfg}->{"color.$slot"} //
+ $self->{-gitcfg}->{"color.diff.$slot"} //
+ $self->{-gitcfg}->{"diff.color.$slot"} //
+ $DEFAULT_COLOR{$slot};
+ $val = $val->[-1] if ref($val) eq 'ARRAY';
+ if (defined $val) {
+ # git doesn't use "_", Term::ANSIColor does
+ $val =~ s/\Abright([^_])/bright_$1/i;
+ ${$self->{obuf}} .= Term::ANSIColor::colored($_[2], lc $val);
+ } else {
+ ${$self->{obuf}} .= $_[2];
+ }
+}
+
+sub uncolored { ${$_[0]->{obuf}} .= $_[2] }
+
+sub new {
+ my ($cls, $lei) = @_;
+ my $self = bless { %{$lei->{opt}}, -colored => \&uncolored }, $cls;
+ return $self unless $self->{color} || -t $lei->{1};
+ my $cmd = [ qw(git config -z --includes -l) ];
+ my ($r, $pid) = popen_rd($cmd, undef, { 2 => $lei->{2} });
+ my $cfg = PublicInbox::Config::config_fh_parse($r, "\0", "\n");
+ waitpid($pid, 0);
+ if ($?) {
+ $lei->err("# git-config failed, no color (non-fatal)");
+ return $self;
+ }
+ $self->{-colored} = \&my_colored;
+ $self->{-gitcfg} = $cfg;
+ $self->{-leicfg} = $lei->{cfg};
+ $self;
+}
+
+sub hdr_buf ($$) {
+ my ($self, $eml) = @_;
+ my $hbuf = '';
+ for my $f (qw(From To Cc)) {
+ for my $v ($eml->header($f)) {
+ next if $v !~ /\S/;
+ PublicInbox::View::fold_addresses($v);
+ _xs($v);
+ $hbuf .= "$f: $v\n";
+ }
+ }
+ for my $f (qw(Subject Date Newsgroups Message-ID X-Message-ID)) {
+ for my $v ($eml->header($f)) {
+ _xs($v);
+ $hbuf .= "$f: $v\n";
+ }
+ }
+ if (my @irt = $eml->header_raw('In-Reply-To')) {
+ for my $v (@irt) {
+ _xs($v);
+ $hbuf .= "In-Reply-To: $v\n";
+ }
+ } else {
+ my $refs = references($eml);
+ if (defined(my $irt = pop @$refs)) {
+ _xs($irt);
+ $hbuf .= "In-Reply-To: <$irt>\n";
+ }
+ if (@$refs) {
+ my $max = $self->{-max_cols};
+ $hbuf .= 'References: ' .
+ join("\n\t", map { '<'._xs($_).'>' } @$refs) .
+ ">\n";
+ }
+ }
+ $self->{-colored}->($self, 'hdrdefault', $hbuf .= "\n");
+}
+
+sub attach_note ($$$$;$) {
+ my ($self, $ct, $p, $fn, $err) = @_;
+ my ($part, $depth, $idx) = @$p;
+ my $obuf = $self->{obuf};
+ my $nl = $idx eq '1' ? '' : "\n"; # like join("\n", ...)
+ $$obuf .= <<EOF if $err;
+[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
+EOF
+ my $blob = $self->{-smsg}->{blob} // '';
+ $blob .= ':' if $blob ne '';
+ $$obuf .= "[-- Attachment $blob$idx ";
+ _xs($ct);
+ my $size = length($part->body);
+ my $ts = "Type: $ct, Size: $size bytes";
+ my $d = $part->header('Content-Description') // $fn // '';
+ _xs($d);
+ $$obuf .= $d eq '' ? "$ts --]\n" : "$d --]\n[-- $ts --]\n";
+ hdr_buf($self, $part) if $part->{is_submsg};
+}
+
+sub flush_text_diff ($$) {
+ my ($self, $cur) = @_;
+ my @top = split($PublicInbox::ViewDiff::EXTRACT_DIFFS, $$cur);
+ undef $$cur; # free memory
+ my $dctx;
+ my $obuf = $self->{obuf};
+ my $colored = $self->{-colored};
+ while (defined(my $x = shift @top)) {
+ if (scalar(@top) >= 4 &&
+ $top[1] =~ $PublicInbox::ViewDiff::IS_OID &&
+ $top[0] =~ $PublicInbox::ViewDiff::IS_OID) {
+ splice(@top, 0, 4);
+ $dctx = 1;
+ $colored->($self, 'meta', $x);
+ } elsif ($dctx) {
+ # Quiet "Complex regular subexpression recursion limit"
+ # warning. Perl will truncate matches upon hitting
+ # that limit, giving us more (and shorter) scalars than
+ # would be ideal, but otherwise it's harmless.
+ #
+ # We could replace the `+' metacharacter with `{1,100}'
+ # to limit the matches ourselves to 100, but we can
+ # let Perl do it for us, quietly.
+ no warnings 'regexp';
+
+ for my $s (split(/((?:(?:^\+[^\n]*\n)+)|
+ (?:(?:^-[^\n]*\n)+)|
+ (?:^@@ [^\n]+\n))/xsm, $x)) {
+ if (!defined($dctx)) {
+ ${$self->{obuf}} .= $s;
+ } elsif ($s =~ s/\A(@@ \S+ \S+ @@\s*)//) {
+ $colored->($self, 'frag', $1);
+ $colored->($self, 'func', $s);
+ } elsif ($s =~ /\A\+/) {
+ $colored->($self, 'new', $s);
+ } elsif ($s =~ /\A-- $/sm) { # email sig starts
+ $dctx = undef;
+ ${$self->{obuf}} .= $s;
+ } elsif ($s =~ /\A-/) {
+ $colored->($self, 'old', $s);
+ } else {
+ $colored->($self, 'context', $s);
+ }
+ }
+ } else {
+ ${$self->{obuf}} .= $x;
+ }
+ }
+}
+
+sub add_text_buf { # callback for Eml->each_part
+ my ($p, $self) = @_;
+ my ($part, $depth, $idx) = @$p;
+ my $ct = $part->content_type || 'text/plain';
+ my $fn = $part->filename;
+ my ($s, $err) = msg_part_text($part, $ct);
+ return attach_note($self, $ct, $p, $fn) unless defined $s;
+ hdr_buf($self, $part) if $part->{is_submsg};
+ $s =~ s/\r\n/\n/sg;
+ _xs($s);
+ $s .= "\n" unless substr($s, -1, 1) eq "\n";
+ my $diff = ($s =~ /^--- [^\n]+\n\+{3} [^\n]+\n@@ /ms);
+ my @sections = PublicInbox::MsgIter::split_quotes($s);
+ undef $s; # free memory
+ if (defined($fn) || ($depth > 0 && !$part->{is_submsg}) || $err) {
+ # badly-encoded message with $err? tell the world about it!
+ attach_note($self, $ct, $p, $fn, $err);
+ ${$self->{obuf}} .= "\n";
+ }
+ my $colored = $self->{-colored};
+ for my $cur (@sections) {
+ if ($cur =~ /\A>/) {
+ $colored->($self, 'quoted', $cur);
+ } elsif ($diff) {
+ flush_text_diff($self, \$cur);
+ } else {
+ ${$self->{obuf}} .= $cur;
+ }
+ undef $cur; # free memory
+ }
+}
+
+# returns an arrayref suitable for $lei->out or print
+sub eml_to_text {
+ my ($self, $smsg, $eml) = @_;
+ local $Term::ANSIColor::EACHLINE = "\n";
+ $self->{obuf} = \(my $obuf = '');
+ $self->{-smsg} = $smsg;
+ $self->{-max_cols} = ($self->{columns} //= 80) - 8; # for header wrap
+ my @h = ();
+ for my $f (qw(blob pct)) {
+ push @h, "$f:$smsg->{$f}" if defined $smsg->{$f};
+ }
+ @h = ("# @h\n") if @h;
+ for my $f (qw(kw L)) {
+ my $v = $smsg->{$f} or next;
+ push @h, "# $f:".join(',', @$v)."\n" if @$v;
+ }
+ $self->{-colored}->($self, 'status', join('', @h));
+ hdr_buf($self, $eml);
+ $eml->each_part(\&add_text_buf, $self, 1);
+ delete $self->{obuf};
+}
+
+1;
diff --git a/lib/PublicInbox/ViewDiff.pm b/lib/PublicInbox/ViewDiff.pm
index 8fe7261f..e9a7bf69 100644
--- a/lib/PublicInbox/ViewDiff.pm
+++ b/lib/PublicInbox/ViewDiff.pm
@@ -30,7 +30,7 @@ my $DIFFSTAT_COMMENT =
my $NULL_TO_BLOB = qr/^(index $OID_NULL\.\.)($OID_BLOB)\b/ms;
my $BLOB_TO_NULL = qr/^index ($OID_BLOB)(\.\.$OID_NULL)\b/ms;
my $BLOB_TO_BLOB = qr/^index ($OID_BLOB)\.\.($OID_BLOB)/ms;
-my $EXTRACT_DIFFS = qr/(
+our $EXTRACT_DIFFS = qr/(
(?: # begin header stuff, don't capture filenames, here,
# but instead wait for the --- and +++ lines.
(?:^diff\x20--git\x20$FN\x20$FN$LF)
@@ -41,7 +41,7 @@ my $EXTRACT_DIFFS = qr/(
^index\x20($OID_BLOB)\.\.($OID_BLOB)$ANY*$LF
^---\x20($FN)$LF
^\+{3}\x20($FN)$LF)/msx;
-my $IS_OID = qr/\A$OID_BLOB\z/s;
+our $IS_OID = qr/\A$OID_BLOB\z/s;
# link to line numbers in blobs
sub diff_hunk ($$$$) {
^ permalink raw reply related [relevance 32%]
* [PATCH 0/3] doc: lei updates around lei-q
@ 2021-04-28 4:51 90% Eric Wong
2021-04-28 4:51 64% ` [PATCH 1/3] doc: lei: use /tmp for search results pathnames Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-04-28 4:51 UTC (permalink / raw)
To: meta
1/3 is probably a game changer and will help anybody supporting
this project sleep easier at night :)
Eric Wong (3):
doc: lei: use /tmp for search results pathnames
doc: lei q: split =item aliases onto separate lines
doc: lei q: split --output and --format, note "text"
Documentation/lei-overview.pod | 4 +-
Documentation/lei-q.pod | 88 +++++++++++++++++++++++-----------
2 files changed, 63 insertions(+), 29 deletions(-)
^ permalink raw reply [relevance 90%]
* [PATCH 1/3] doc: lei: use /tmp for search results pathnames
2021-04-28 4:51 90% [PATCH 0/3] doc: lei updates around lei-q Eric Wong
@ 2021-04-28 4:51 64% ` Eric Wong
2021-04-28 4:51 54% ` [PATCH 2/3] doc: lei q: split =item aliases onto separate lines Eric Wong
2021-04-28 4:51 63% ` [PATCH 3/3] doc: lei q: split --output and --format, note "text" Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 4:51 UTC (permalink / raw)
To: meta
This drives the point home about results being volatile
and discardable.
---
Documentation/lei-overview.pod | 4 ++--
Documentation/lei-q.pod | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-overview.pod b/Documentation/lei-overview.pod
index 70dbf2b5..6b5fa721 100644
--- a/Documentation/lei-overview.pod
+++ b/Documentation/lei-overview.pod
@@ -82,7 +82,7 @@ Search for messages whose subject includes "lei" and "skeleton".
Do the same, but also report unmatched messages that are in the same
thread as a matched message.
-=item $ lei q -t -o mdir --mua=mutt s:lei s:skeleton
+=item $ lei q -t -o /tmp/mdir --mua=mutt s:lei s:skeleton
Write results to a Maildir at "mdir". Mutt will be invoked
to open mfolder (C<mutt -f %f>) while results are being fetched
@@ -92,7 +92,7 @@ and written.
Search for all flagged messages that also have a "next" label.
-=item $ lei p2q HEAD | lei q --stdin -tt -o mdir
+=item $ lei p2q HEAD | lei q --stdin -tt -o /tmp/mdir
Search for messages that have post-image git blob IDs that match those
of the current repository's HEAD commit, writing them to the Maildir
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index a84fc440..b938746a 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -31,9 +31,9 @@ Read search terms from stdin.
Warning: this clobbers and overwrites the output destination unless
L</-a, --augment> is specified.
-Destination for results (e.g., C<path/to/Maildir>,
+Destination for results (e.g., C</tmp/results-Maildir>,
C<imaps://user@mail.example.com/INBOX.test>, or
-C<mboxcl2:path/to/mbox>). The prefix may be a supported protocol:
+C<mboxcl2:/tmp/results-mboxcl2>). The prefix may be a supported protocol:
C<imap://> or C<imaps://>. URLs requiring
authentication must use L<netrc(5)> and/or L<git-credential(1)> to
fill in the username and password.
^ permalink raw reply related [relevance 64%]
* [PATCH 3/3] doc: lei q: split --output and --format, note "text"
2021-04-28 4:51 90% [PATCH 0/3] doc: lei updates around lei-q Eric Wong
2021-04-28 4:51 64% ` [PATCH 1/3] doc: lei: use /tmp for search results pathnames Eric Wong
2021-04-28 4:51 54% ` [PATCH 2/3] doc: lei q: split =item aliases onto separate lines Eric Wong
@ 2021-04-28 4:51 63% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 4:51 UTC (permalink / raw)
To: meta
I guess --format makes sense for stdout, after all;
and I'm enjoying "-f text" quite a bit, so far.
---
Documentation/lei-q.pod | 26 +++++++++++++++-----------
1 file changed, 15 insertions(+), 11 deletions(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index 2b9936b8..bf7a5f70 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -42,15 +42,11 @@ C<imap://> or C<imaps://>. URLs requiring
authentication must use L<netrc(5)> and/or L<git-credential(1)> to
fill in the username and password.
-The prefix can instead specify the format of the output: C<maildir>,
-C<mboxrd>, C<mboxcl2>, C<mboxcl>, C<mboxo>, C<json>, C<jsonl>, or
-C<concatjson>. When a format isn't specified, it's chosen based on
-the destination. C<json> is used for the default destination
-(stdout), and C<maildir> is used for an existing directory or
-non-existing path.
+A prefix can specify the format of the output: C<maildir>,
+C<mboxrd>, C<mboxcl2>, C<mboxcl>, C<mboxo>. For a description of
+mail formats, see L<lei-mail-formats(5)>.
-=for comment
-TODO: Provide description of formats?
+C<maildir> is the default for an existing directory or non-existing path.
Default: C<-> (stdout)
@@ -58,9 +54,17 @@ Default: C<-> (stdout)
=item -f FORMAT
-Format of results. This option exists as a convenient way to specify
-the format for the default stdout destination. Using a C<format:>
-prefix with the C<--output> destination is preferred otherwise.
+Format of results to stdout. This option exists as a convenient
+way to specify the format for the default stdout destination.
+C<text>, C<json>, C<jsonl>, or C<concatjson> are all supported,
+as are the various mbox variants described in L</--output>.
+
+When a format isn't specified, it's chosen based on the
+L</--output> destination or prefix. C<json> is used for the
+default destination (stdout).
+
+Using a C<format:> prefix with the C<--output> destination is
+preferred when not writing to stdout.
=item --pretty
^ permalink raw reply related [relevance 63%]
* [PATCH 2/3] doc: lei q: split =item aliases onto separate lines
2021-04-28 4:51 90% [PATCH 0/3] doc: lei updates around lei-q Eric Wong
2021-04-28 4:51 64% ` [PATCH 1/3] doc: lei: use /tmp for search results pathnames Eric Wong
@ 2021-04-28 4:51 54% ` Eric Wong
2021-04-29 1:39 71% ` Kyle Meyer
2021-04-28 4:51 63% ` [PATCH 3/3] doc: lei q: split --output and --format, note "text" Eric Wong
2 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-04-28 4:51 UTC (permalink / raw)
To: meta
It makes L</--augment> look nicer without resorting to
L<--augment|/-a, --augment> and similarly verbose nastiness.
Having each option as a separate =item (with a blank line in
between each =item) seems to be the preferred style used within
Perl core documentation (I used perlrun.pod as an example),
so we'll follow Perl core style, here.
This needs to be done for other manpages, at some point...
---
Documentation/lei-q.pod | 58 +++++++++++++++++++++++++++++++----------
1 file changed, 44 insertions(+), 14 deletions(-)
diff --git a/Documentation/lei-q.pod b/Documentation/lei-q.pod
index b938746a..2b9936b8 100644
--- a/Documentation/lei-q.pod
+++ b/Documentation/lei-q.pod
@@ -26,10 +26,14 @@ TODO: mention curl options?
Read search terms from stdin.
-=item -o MFOLDER, --output=MFOLDER, --mfolder=MFOLDER
+=item --output=MFOLDER
+
+=item -o MFOLDER
+
+=item --mfolder=MFOLDER
Warning: this clobbers and overwrites the output destination unless
-L</-a, --augment> is specified.
+L</--augment> is specified.
Destination for results (e.g., C</tmp/results-Maildir>,
C<imaps://user@mail.example.com/INBOX.test>, or
@@ -50,7 +54,9 @@ TODO: Provide description of formats?
Default: C<-> (stdout)
-=item -f FORMAT, --format=FORMAT
+=item --format=FORMAT
+
+=item -f FORMAT
Format of results. This option exists as a convenient way to specify
the format for the default stdout destination. Using a C<format:>
@@ -81,7 +87,9 @@ This option may be given multiple times.
Default: C<:WINCH,:bell> when C<--mua> is specified and C<--output>
doesn't point to stdout, nothing otherwise.
-=item -a, --augment
+=item --augment
+
+=item -a
Augment output destination instead of clobbering it.
@@ -90,7 +98,9 @@ Augment output destination instead of clobbering it.
Do not importing keywords before writing to an existing output
destination.
-=item -t, --threads
+=item --threads
+
+=item -t
Return all messages in the same thread as the actual match(es).
@@ -102,7 +112,9 @@ of the same thread.
TODO: Warning: this flag may become persistent and saved in
lei/store unless an MUA unflags it! (Behavior undecided)
-=item -d STRATEGY, --dedupe=STRATEGY
+=item --dedupe=STRATEGY
+
+=item -d STRATEGY
Strategy for deduplicating messages: C<content>, C<oid>, C<mid>, or
C<none>.
@@ -126,7 +138,9 @@ Limit operations to those requiring network access.
Don't include results from externals.
-=item -I LOCATION, --include=LOCATION
+=item --include=LOCATION
+
+=item -I LOCATION
Include specified external in search. This option may be given
multiple times.
@@ -141,7 +155,9 @@ multiple times.
Use only the specified external for search. This option may be given
multiple times, in which case the search uses only the specified set.
-=item -g, --globoff
+=item --globoff
+
+=item -g
Do not match locations using C<*?> wildcards and C<[]> ranges. This
option applies to C<--include>, C<--exclude>, and C<--only>.
@@ -158,7 +174,11 @@ C<none>.
Default: fcntl,dotlock
-=item -NUMBER, -n NUMBER, --limit=NUMBER
+=item --limit=NUMBER
+
+=item -NUMBER
+
+=item -n NUMBER
Limit the number of matches.
@@ -170,26 +190,36 @@ Shift start of search results.
Default: 0
-=item -r, --reverse
+=item --reverse
+
+=item -r
Reverse the results. Note that this applies before C<--limit>.
-=item -s KEY, --sort=KEY
+=item --sort=KEY
+
+=item -s KEY
Order the results by KEY. Valid keys are C<received>, C<relevance>,
and C<docid>.
Default: C<received>
-=item -v, --verbose
+=item --verbose
+
+=item -v
Provide more feedback on stderr.
-=item -q, --quiet
+=item --quiet
+
+=item -q
Suppress feedback messages.
-=item --torsocks=auto|no|yes, --no-torsocks
+=item --torsocks=auto|no|yes
+
+=item --no-torsocks
Whether to wrap L<git(1)> and L<curl(1)> commands with torsocks.
^ permalink raw reply related [relevance 54%]
* [PATCH 01/11] t/lei-p2q: add diagnostics
2021-04-28 7:51 68% [PATCH 00/11] lei: misc fixes, more lcat color support Eric Wong
@ 2021-04-28 7:51 71% ` Eric Wong
2021-04-28 7:51 52% ` [PATCH 02/11] tests: restore CWD with "lei -C" and run_script Eric Wong
` (4 subsequent siblings)
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 7:51 UTC (permalink / raw)
To: meta
Maybe this helps fix occasional problems in daemon mode,
but I haven't seen anything on failure, yet...
---
t/lei-p2q.t | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/t/lei-p2q.t b/t/lei-p2q.t
index be2d437c..f8b073cf 100644
--- a/t/lei-p2q.t
+++ b/t/lei-p2q.t
@@ -9,10 +9,10 @@ test_lei(sub {
ok(!lei(qw(p2q this-better-cause-format-patch-to-fail)),
'p2q fails on bogus arg');
lei_ok(qw(p2q -w dfpost t/data/0001.patch));
- is($lei_out, "dfpost:6e006fd73b1d\n", 'pathname');
+ is($lei_out, "dfpost:6e006fd73b1d\n", 'pathname') or diag $lei_err;
open my $fh, '+<', 't/data/0001.patch' or xbail "open: $!";
lei_ok([qw(p2q -w dfpost -)], undef, { %$lei_opt, 0 => $fh });
- is($lei_out, "dfpost:6e006fd73b1d\n", '--stdin');
+ is($lei_out, "dfpost:6e006fd73b1d\n", '--stdin') or diag $lei_err;
lei_ok(qw(p2q --uri t/data/0001.patch -w), 'dfpost,dfn');
is($lei_out, "dfpost%3A6e006fd73b1d+".
^ permalink raw reply related [relevance 71%]
* [PATCH 05/11] lei-daemon: note FD count mismatch to client
2021-04-28 7:51 68% [PATCH 00/11] lei: misc fixes, more lcat color support Eric Wong
2021-04-28 7:51 71% ` [PATCH 01/11] t/lei-p2q: add diagnostics Eric Wong
2021-04-28 7:51 52% ` [PATCH 02/11] tests: restore CWD with "lei -C" and run_script Eric Wong
@ 2021-04-28 7:51 71% ` Eric Wong
2021-04-28 7:52 59% ` [PATCH 07/11] lei: quiet down Eml-related warnings consistently Eric Wong
` (2 subsequent siblings)
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 7:51 UTC (permalink / raw)
To: meta
This should help in some error diagnostics
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index ef72758c..a949ae3e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -998,7 +998,7 @@ sub accept_dispatch { # Listener {post_accept} callback
open($self->{$i++}, '+<&=', $fd) and next;
send($sock, "open(+<&=$fd) (FD=$i): $!", MSG_EOR);
}
- return if scalar(@fds) != 4;
+ $i == 4 or return send($sock, 'not enough FDs='.($i-1), MSG_EOR)
}
$self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
# $ENV_STR = join('', map { "\0$_=$ENV{$_}" } keys %ENV);
^ permalink raw reply related [relevance 71%]
* [PATCH 00/11] lei: misc fixes, more lcat color support
@ 2021-04-28 7:51 68% Eric Wong
2021-04-28 7:51 71% ` [PATCH 01/11] t/lei-p2q: add diagnostics Eric Wong
` (5 more replies)
0 siblings, 6 replies; 200+ results
From: Eric Wong @ 2021-04-28 7:51 UTC (permalink / raw)
To: meta
I'm seeing some odd test failures in "make check-run"
(but not "make check") that I haven't diagnosed, yet.
So there's some diagnostic changes and some golfing
to make internals more consistent.
Eric Wong (11):
t/lei-p2q: add diagnostics
tests: restore CWD with "lei -C" and run_script
view_diff: minor coding style fixes
lei_p2q: add _lei_wq_eof callback
lei-daemon: note FD count mismatch to client
t/run.perl: add (GNU) tail and strace support
lei: quiet down Eml-related warnings consistently
lei: simple WQ workers use {wq1} field
lei_view_text: improve attachment display
lei_view_text: translate background colors from git
lei (lcat|q): support --no-color and --color
lib/PublicInbox/LEI.pm | 30 ++++++++++++++++++++---------
lib/PublicInbox/LeiBlob.pm | 9 +--------
lib/PublicInbox/LeiConvert.pm | 3 +--
lib/PublicInbox/LeiImport.pm | 10 +++-------
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 4 ++--
lib/PublicInbox/LeiP2q.pm | 6 ++----
lib/PublicInbox/LeiStore.pm | 1 -
lib/PublicInbox/LeiTag.pm | 9 ++-------
lib/PublicInbox/LeiViewText.pm | 35 +++++++++++++++++++++++-----------
lib/PublicInbox/LeiXSearch.pm | 1 -
lib/PublicInbox/TestCommon.pm | 12 ++++++++----
lib/PublicInbox/ViewDiff.pm | 9 ++++-----
script/lei | 4 ++--
t/lei-externals.t | 5 -----
t/lei-p2q.t | 4 ++--
t/lei-q-save.t | 4 ----
t/run.perl | 7 +++++++
t/solver_git.t | 5 -----
19 files changed, 80 insertions(+), 80 deletions(-)
^ permalink raw reply [relevance 68%]
* [PATCH 07/11] lei: quiet down Eml-related warnings consistently
2021-04-28 7:51 68% [PATCH 00/11] lei: misc fixes, more lcat color support Eric Wong
` (2 preceding siblings ...)
2021-04-28 7:51 71% ` [PATCH 05/11] lei-daemon: note FD count mismatch to client Eric Wong
@ 2021-04-28 7:52 59% ` Eric Wong
2021-04-28 7:52 44% ` [PATCH 08/11] lei: simple WQ workers use {wq1} field Eric Wong
2021-04-28 7:52 60% ` [PATCH 11/11] lei (lcat|q): support --no-color and --color Eric Wong
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 7:52 UTC (permalink / raw)
To: meta
"lei import" is probably the only place where it users
might care about warnings.
---
lib/PublicInbox/LEI.pm | 3 +++
lib/PublicInbox/LeiBlob.pm | 1 -
lib/PublicInbox/LeiConvert.pm | 1 -
lib/PublicInbox/LeiImport.pm | 1 +
lib/PublicInbox/LeiP2q.pm | 4 +---
lib/PublicInbox/LeiStore.pm | 1 -
lib/PublicInbox/LeiXSearch.pm | 1 -
7 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a949ae3e..cfbf12f0 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -23,6 +23,7 @@ use PublicInbox::Sigfd;
use PublicInbox::DS qw(now dwaitpid);
use PublicInbox::Spawn qw(spawn popen_rd);
use PublicInbox::Lock;
+use PublicInbox::Eml;
use Time::HiRes qw(stat); # ctime comparisons for config cache
use File::Path qw(mkpath);
use File::Spec;
@@ -509,6 +510,8 @@ sub _lei_atfork_child {
%PATH2CFG = ();
undef $errors_log;
$quit = \&CORE::exit;
+ $self->{-eml_noisy} or # only "lei import" sets this atm
+ $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$current_lei = $persist ? undef : $self; # for SIG{__WARN__}
}
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 0b96bd04..ff079e65 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -167,7 +167,6 @@ EOM
sub ipc_atfork_child {
my ($self) = @_;
$self->{lei}->_lei_atfork_child;
- $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
}
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 0c324169..14bed901 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -68,7 +68,6 @@ sub ipc_atfork_child {
$net->{mics_cached} = $net->imap_common_init($lei);
$net->{nn_cached} = $net->nntp_common_init($lei);
}
- $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$l2m->pre_augment($lei);
$l2m->do_augment($lei);
$l2m->post_augment($lei);
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index e0d899cc..f2a0c95a 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -93,6 +93,7 @@ sub lei_import { # the main "lei import" method
my $ops = {};
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{-wq_nr_workers} = $j // 1; # locked
+ $lei->{-eml_noisy} = 1;
(my $op_c, $ops) = $lei->workers_start($self, 'lei-import', $j, $ops);
$lei->{imp} = $self;
net_merge_complete($self) unless $lei->{auth};
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 07357e32..deb31974 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -197,9 +197,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
sub ipc_atfork_child {
my ($self) = @_;
- my $lei = $self->{lei};
- $lei->_lei_atfork_child;
- $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
+ $self->{lei}->_lei_atfork_child;
$self->SUPER::ipc_atfork_child;
}
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 1cf7ffc1..fcc9224d 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -411,7 +411,6 @@ sub ipc_atfork_child {
close $err->[0];
$self->{-err_wr} = $err->[1];
}
- $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
}
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 018b60f9..b3fd79d0 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -443,7 +443,6 @@ sub incr_start_query { # called whenever an l2m shard starts do_post_auth
sub ipc_atfork_child {
my ($self) = @_;
$self->{lei}->_lei_atfork_child;
- $SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
}
^ permalink raw reply related [relevance 59%]
* [PATCH 02/11] tests: restore CWD with "lei -C" and run_script
2021-04-28 7:51 68% [PATCH 00/11] lei: misc fixes, more lcat color support Eric Wong
2021-04-28 7:51 71% ` [PATCH 01/11] t/lei-p2q: add diagnostics Eric Wong
@ 2021-04-28 7:51 52% ` Eric Wong
2021-04-28 7:51 71% ` [PATCH 05/11] lei-daemon: note FD count mismatch to client Eric Wong
` (3 subsequent siblings)
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 7:51 UTC (permalink / raw)
To: meta
This simplifies test_lei users in t/*.t
---
lib/PublicInbox/TestCommon.pm | 12 ++++++++----
t/lei-externals.t | 5 -----
t/lei-q-save.t | 4 ----
t/solver_git.t | 5 -----
4 files changed, 8 insertions(+), 18 deletions(-)
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 09256852..460c9da0 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -12,6 +12,7 @@ use IO::Socket::INET;
use File::Spec;
our @EXPORT;
my $lei_loud = $ENV{TEST_LEI_ERR_LOUD};
+our ($lei_opt, $lei_out, $lei_err, $lei_cwdfh);
BEGIN {
@EXPORT = qw(tmpdir tcp_server tcp_connect require_git require_mods
run_script start_script key2sub xsys xsys_e xqx eml_load tick
@@ -306,14 +307,16 @@ sub run_script ($;$$) {
local %SIG = %SIG;
local $0 = join(' ', @$cmd);
my $orig_io = _prepare_redirects($fhref);
- my $cwdfh;
+ my $cwdfh = $lei_cwdfh;
if (my $d = $opt->{'-C'}) {
- opendir $cwdfh, '.' or die "opendir .: $!";
+ unless ($cwdfh) {
+ opendir $cwdfh, '.' or die "opendir .: $!";
+ }
chdir $d or die "chdir $d: $!";
}
_run_sub($sub, $key, \@argv);
eval { PublicInbox::Inbox::cleanup_task() };
- die "chdir(restore): $!" if $cwdfh && !chdir($cwdfh);
+ die "fchdir(restore): $!" if $cwdfh && !chdir($cwdfh);
_undo_redirects($orig_io);
select STDOUT;
}
@@ -469,7 +472,6 @@ sub have_xapian_compact () {
PublicInbox::Spawn::which($ENV{XAPIAN_COMPACT} || 'xapian-compact');
}
-our ($lei_opt, $lei_out, $lei_err);
# favor lei() or lei_ok() over $lei for new code
sub lei (@) {
my ($cmd, $env, $xopt) = @_;
@@ -515,6 +517,8 @@ sub test_lei {
SKIP: {
my ($cb) = pop @_;
my $test_opt = shift // {};
+ local $lei_cwdfh;
+ opendir $lei_cwdfh, '.' or xbail "opendir .: $!";
require_git(2.6, 1) or skip('git 2.6+ required for lei test', 2);
require_mods(qw(json DBD::SQLite Search::Xapian), 2);
require PublicInbox::Config;
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 2291dd99..16241e02 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -4,7 +4,6 @@
use strict; use v5.10.1; use PublicInbox::TestCommon;
use Fcntl qw(SEEK_SET);
use PublicInbox::Spawn qw(which);
-use PublicInbox::OnDestroy;
require_git 2.6;
require_mods(qw(json DBD::SQLite Search::Xapian));
use POSIX qw(WTERMSIG WIFSIGNALED SIGPIPE);
@@ -266,10 +265,6 @@ test_lei(sub {
{
skip 'TEST_LEI_DAEMON_PERSIST_DIR in use', 1 if
$ENV{TEST_LEI_DAEMON_PERSIST_DIR};
- opendir my $dh, '.' or BAIL_OUT "opendir(.) $!";
- my $od = PublicInbox::OnDestroy->new($$, sub {
- chdir $dh or BAIL_OUT "chdir: $!"
- });
my @q = qw(q -o mboxcl2:rel.mboxcl2 bye);
lei_ok('-C', $home, @q);
is(unlink("$home/rel.mboxcl2"), 1, '-C works before q');
diff --git a/t/lei-q-save.t b/t/lei-q-save.t
index 170f7ce5..9f65e4a2 100644
--- a/t/lei-q-save.t
+++ b/t/lei-q-save.t
@@ -36,11 +36,9 @@ test_lei(sub {
# ensure "lei up" works, since it compliments "lei q --save"
$in = $doc2->as_string;
lei_ok [qw(import -q -F eml -)], undef, { 0 => \$in, %$lei_opt };
- opendir my $dh, '.' or xbail "opendir .: $!";
lei_ok qw(up -q md -C), $home;
lei_ok qw(up -q . -C), "$home/md";
lei_ok qw(up -q), "/$home/md";
- chdir($dh) or xbail "fchdir . $!";
my %after = map { $_ => 1 } glob("$home/md/cur/*");
is(delete $after{(keys(%before))[0]}, 1, 'original message kept');
is(scalar(keys %after), 1, 'one new message added');
@@ -89,7 +87,6 @@ test_lei(sub {
print $mb $pre_existing;
close $mb or xbail "close: $!";
lei_ok(qw(q --save -o mboxrd:mbrd m:qp@example.com -C), $home);
- chdir($dh) or xbail "fchdir . $!";
open $mb, '<', "$home/mbrd" or xbail "open $!";
is_deeply([grep(/pre-existing/, <$mb>)], [],
'pre-existing messsage gone w/o augment');
@@ -103,7 +100,6 @@ test_lei(sub {
print $mb $pre_existing;
close $mb or xbail "close: $!";
lei_ok(qw(q -a --save -o mboxrd:mbrd-aug m:qp@example.com -C), $home);
- chdir($dh) or xbail "fchdir . $!";
open $mb, '<', "$home/mbrd-aug" or xbail "open $!";
$mb = do { local $/; <$mb> };
like($mb, qr/pre-existing/, 'pre-existing message preserved w/ -a');
diff --git a/t/solver_git.t b/t/solver_git.t
index 8acf907f..6875e26b 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -50,11 +50,6 @@ test_lei({tmpdir => $tmpdir}, sub {
'non-existent blob fails');
SKIP: {
skip '/.git exists', 1 if -e '/.git';
- require PublicInbox::OnDestroy;
- opendir my $dh, '.' or xbail "opendir: $!";
- my $end = PublicInbox::OnDestroy->new($$, sub {
- chdir $dh or xbail "chdir: $!";
- });
lei_ok(qw(-C / blob 69df7d5 -I), $ibx->{inboxdir},
"--git-dir=$git_dir");
is($lei_out, $prev, '--git-dir works');
^ permalink raw reply related [relevance 52%]
* [PATCH 11/11] lei (lcat|q): support --no-color and --color
2021-04-28 7:51 68% [PATCH 00/11] lei: misc fixes, more lcat color support Eric Wong
` (4 preceding siblings ...)
2021-04-28 7:52 44% ` [PATCH 08/11] lei: simple WQ workers use {wq1} field Eric Wong
@ 2021-04-28 7:52 60% ` Eric Wong
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 7:52 UTC (permalink / raw)
To: meta
This should be familiar to git users who wish to force color
when writing to pipes or disable color.
---
lib/PublicInbox/LEI.pm | 9 +++++----
lib/PublicInbox/LeiViewText.pm | 2 +-
2 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 403f9ed8..7ffcf163 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -144,8 +144,8 @@ our %CMD = ( # sorted in order of importance/use:
@lxs_opt,
qw(save output|mfolder|o=s format|f=s dedupe|d=s threads|t+
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
- import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
- opt_dash('limit|n=i', '[0-9]+') ],
+ import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+
+ color!), @c_opt, opt_dash('limit|n=i', '[0-9]+') ],
'up' => [ 'OUTPUT|--all', 'update saved search',
qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all:s), @c_opt ],
@@ -155,8 +155,8 @@ our %CMD = ( # sorted in order of importance/use:
# some of these options are ridiculous for lcat
@lxs_opt, qw(output|mfolder|o=s format|f=s dedupe|d=s threads|t+
sort|s=s reverse|r offset=i jobs|j=s globoff|g augment|a
- import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
- opt_dash('limit|n=i', '[0-9]+') ],
+ import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+
+ color!), @c_opt, opt_dash('limit|n=i', '[0-9]+') ],
'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
@@ -268,6 +268,7 @@ my %OPTDESC = (
'incremental! import' => 'import already seen IMAP and NNTP articles',
'globoff|g' => "do not match locations using '*?' wildcards ".
"and\xa0'[]'\x{a0}ranges",
+'color!' => 'disable color (for --format=text)',
'verbose|v+' => 'be more verbose',
'external!' => 'do not use externals',
'mail!' => 'do not look in mail storage for OID',
diff --git a/lib/PublicInbox/LeiViewText.pm b/lib/PublicInbox/LeiViewText.pm
index e0d62c0d..d0f8b7f4 100644
--- a/lib/PublicInbox/LeiViewText.pm
+++ b/lib/PublicInbox/LeiViewText.pm
@@ -69,7 +69,7 @@ sub uncolored { ${$_[0]->{obuf}} .= $_[2] }
sub new {
my ($cls, $lei) = @_;
my $self = bless { %{$lei->{opt}}, -colored => \&uncolored }, $cls;
- return $self unless $self->{color} || -t $lei->{1};
+ return $self unless $self->{color} //= -t $lei->{1};
my $cmd = [ qw(git config -z --includes -l) ];
my ($r, $pid) = popen_rd($cmd, undef, { 2 => $lei->{2} });
my $cfg = PublicInbox::Config::config_fh_parse($r, "\0", "\n");
^ permalink raw reply related [relevance 60%]
* [PATCH 08/11] lei: simple WQ workers use {wq1} field
2021-04-28 7:51 68% [PATCH 00/11] lei: misc fixes, more lcat color support Eric Wong
` (3 preceding siblings ...)
2021-04-28 7:52 59% ` [PATCH 07/11] lei: quiet down Eml-related warnings consistently Eric Wong
@ 2021-04-28 7:52 44% ` Eric Wong
2021-04-28 7:52 60% ` [PATCH 11/11] lei (lcat|q): support --no-color and --color Eric Wong
5 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 7:52 UTC (permalink / raw)
To: meta
This lets us share more code and reduces cognitive overhead when
it comes to picking names (because {lsss} was ridiculous).
We'll need to ensure the first error set in lei is the actual
error we exit with, otherwise things can get confusing and
errors may get lost.
---
lib/PublicInbox/LEI.pm | 16 ++++++++++++----
lib/PublicInbox/LeiBlob.pm | 8 +-------
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 9 ++-------
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 4 ++--
lib/PublicInbox/LeiP2q.pm | 8 +-------
lib/PublicInbox/LeiTag.pm | 9 ++-------
script/lei | 4 ++--
9 files changed, 24 insertions(+), 38 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index cfbf12f0..403f9ed8 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -382,7 +382,7 @@ my %CONFIG_KEYS = (
'leistore.dir' => 'top-level storage location',
);
-my @WQ_KEYS = qw(lxs l2m imp mrr cnv p2q tag sol lsss); # internal workers
+my @WQ_KEYS = qw(lxs l2m wq1); # internal workers
sub _drop_wq {
my ($self) = @_;
@@ -542,7 +542,7 @@ sub workers_start {
'child_error' => [ \&child_error, $lei ],
($ops ? %$ops : ()),
};
- $ops->{''} //= [ $wq->can('_lei_wq_eof') || \&dclose, $lei ];
+ $ops->{''} //= [ $wq->can('_lei_wq_eof') || \&wq_eof, $lei ];
my $end = $lei->pkt_op_pair;
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
delete $lei->{pkt_op_p};
@@ -1237,9 +1237,17 @@ sub DESTROY {
sub wq_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
- my ($wq, $lei, $e) = @$arg;
- $? and $lei->child_error($?, $e ? "$e errors during $lei->{cmd}" : ());
+ my ($wq, $lei) = @$arg;
+ my $err_type = $lei->{-err_type};
+ $? and $lei->child_error($?,
+ $err_type ? "$err_type errors during $lei->{cmd}" : ());
$lei->dclose;
}
+sub wq_eof { # EOF callback for main daemon
+ my ($lei) = @_;
+ my $wq1 = delete $lei->{wq1} // return $lei->fail; # already failed
+ $wq1->wq_wait_old(\&wq_done_wait, $lei);
+}
+
1;
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index ff079e65..0a957358 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -10,12 +10,6 @@ use parent qw(PublicInbox::IPC);
use PublicInbox::Spawn qw(spawn popen_rd which);
use PublicInbox::DS;
-sub _lei_wq_eof { # EOF callback for main daemon
- my ($lei) = @_;
- my $sol = delete $lei->{sol} // return $lei->dclose; # already failed
- $sol->wq_wait_old($lei->can('wq_done_wait'), $lei);
-}
-
sub get_git_dir ($$) {
my ($lei, $d) = @_;
return $d if -d "$d/objects" && -d "$d/refs" && -e "$d/HEAD";
@@ -158,7 +152,7 @@ EOM
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
my ($op_c, $ops) = $lei->workers_start($self, 'lei-blob', 1);
- $lei->{sol} = $self;
+ $lei->{wq1} = $self;
$self->wq_io_do('do_solve_blob', []);
$self->wq_close(1);
$op_c->op_wait_event($ops);
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 14bed901..cefcaf65 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -53,7 +53,7 @@ sub lei_convert { # the main "lei convert" method
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
my ($op_c, $ops) = $lei->workers_start($self, 'lei-convert', 1);
- $lei->{cnv} = $self;
+ $lei->{wq1} = $self;
$self->wq_io_do('process_inputs', []);
$self->wq_close(1);
$op_c->op_wait_event($ops);
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index f2a0c95a..26127ece 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -53,12 +53,6 @@ sub input_nntp_cb { # nntp_each
input_eml_cb($self, $eml, $self->{-import_kw} ? { kw => $kw } : undef);
}
-sub _lei_wq_eof { # EOF callback for main daemon
- my ($lei) = @_;
- my $imp = delete $lei->{imp} // return $lei->fail('BUG: {imp} gone');
- $imp->wq_wait_old($lei->can('wq_done_wait'), $lei, 'non-fatal');
-}
-
sub net_merge_complete { # callback used by LeiAuth
my ($self) = @_;
$self->wq_io_do('process_inputs');
@@ -95,7 +89,8 @@ sub lei_import { # the main "lei import" method
$self->{-wq_nr_workers} = $j // 1; # locked
$lei->{-eml_noisy} = 1;
(my $op_c, $ops) = $lei->workers_start($self, 'lei-import', $j, $ops);
- $lei->{imp} = $self;
+ $lei->{wq1} = $self;
+ $lei->{-err_type} = 'non-fatal';
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
}
diff --git a/lib/PublicInbox/LeiLsSearch.pm b/lib/PublicInbox/LeiLsSearch.pm
index 9ac4870f..a00e78fc 100644
--- a/lib/PublicInbox/LeiLsSearch.pm
+++ b/lib/PublicInbox/LeiLsSearch.pm
@@ -73,7 +73,7 @@ sub bg_worker ($$$) {
my ($lei, $pfx, $json) = @_;
my $self = bless { -wq_nr_workers => 1, json => $json }, __PACKAGE__;
my ($op_c, $ops) = $lei->workers_start($self, 'ls-search', 1);
- $lei->{lsss} = $self;
+ $lei->{wq1} = $self;
$self->wq_io_do('do_ls_search_long', [], $pfx);
$self->wq_close(1);
$op_c->op_wait_event($ops);
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index 50ab4c85..db97b98c 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -26,7 +26,7 @@ sub do_finish_mirror { # dwaitpid callback
sub _lei_wq_eof { # EOF callback for main daemon
my ($lei) = @_;
- my $mrr = delete $lei->{mrr} or return;
+ my $mrr = delete $lei->{wq1} or return $lei->fail;
$mrr->wq_wait_old(\&do_finish_mirror, $lei);
}
@@ -283,7 +283,7 @@ sub start {
require PublicInbox::Admin;
require PublicInbox::InboxWritable;
my ($op, $ops) = $lei->workers_start($self, 'lei_mirror', 1);
- $lei->{mrr} = $self;
+ $lei->{wq1} = $self;
$self->wq_io_do('do_mirror', []);
$self->wq_close(1);
$op->op_wait_event($ops);
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index deb31974..b4893489 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -189,7 +189,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
$self->{input} = $input;
}
my ($op, $ops) = $lei->workers_start($self, 'lei-p2q', 1);
- $lei->{p2q} = $self;
+ $lei->{wq1} = $self;
$self->wq_io_do('do_p2q', []);
$self->wq_close(1);
$op->op_wait_event($ops);
@@ -201,10 +201,4 @@ sub ipc_atfork_child {
$self->SUPER::ipc_atfork_child;
}
-sub _lei_wq_eof { # EOF callback for main daemon
- my ($lei) = @_;
- my $p2q = delete $lei->{p2q} // return $lei->dclose;
- $p2q->wq_wait_old($lei->can('wq_done_wait'), $lei);
-}
-
1;
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 3cda2eca..989a6954 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -19,12 +19,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub input_mbox_cb { input_eml_cb($_[1], $_[0]) }
-sub _lei_wq_eof { # EOF callback for main daemon
- my ($lei) = @_;
- my $tag = delete $lei->{tag} // return $lei->dclose;
- $tag->wq_wait_old($lei->can('wq_done_wait'), $lei, 'non-fatal');
-}
-
sub net_merge_complete { # callback used by LeiAuth
my ($self) = @_;
$self->wq_io_do('process_inputs');
@@ -57,7 +51,8 @@ sub lei_tag { # the "lei tag" method
$self->{vmd_mod} = $vmd_mod;
my $j = $self->{-wq_nr_workers} = 1; # locked for now
(my $op_c, $ops) = $lei->workers_start($self, 'lei-tag', $j, $ops);
- $lei->{tag} = $self;
+ $lei->{wq1} = $self;
+ $lei->{-err_type} = 'non-fatal';
net_merge_complete($self) unless $lei->{auth};
$op_c->op_wait_event($ops);
}
diff --git a/script/lei b/script/lei
index db302422..90a93839 100755
--- a/script/lei
+++ b/script/lei
@@ -116,10 +116,10 @@ Falling back to (slow) one-shot mode
} elsif ($buf eq '-WINCH') {
kill($buf, @parent); # for MUA
} elsif ($buf =~ /\Ax_it ([0-9]+)\z/) {
- $x_it_code = $1 + 0;
+ $x_it_code ||= $1 + 0;
last;
} elsif ($buf =~ /\Achild_error ([0-9]+)\z/) {
- $x_it_code = $1 + 0;
+ $x_it_code ||= $1 + 0;
} else {
$sigchld->();
die $buf;
^ permalink raw reply related [relevance 44%]
* [PATCH 2/2] lei: avoid close(STD{IN,OUT,ERR}) in oneshot mode
@ 2021-04-28 19:37 63% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-28 19:37 UTC (permalink / raw)
To: meta
This seems to fix the occasional "make check-run" failures I've
been chasing.
Some parts of our code assumes we can close($lei->{1})
and similar, which causes IO::Handle::autoflush to behave
badly when STDOUT is the "select"-ed FH of the Perl process.
Since oneshot mode is (hopefully) the uncommon case, we'll
just accept the cost of extra FDs and minimize differences
between lei in oneshot vs daemon mode.
---
lib/PublicInbox/LEI.pm | 18 ++++--------------
t/lei.t | 1 +
2 files changed, 5 insertions(+), 14 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 7ffcf163..1ea7c9ca 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -487,19 +487,14 @@ sub _lei_atfork_child {
# we need to explicitly close things which are on stack
if ($persist) {
chdir '/' or die "chdir(/): $!";
- my @io = delete @$self{qw(0 1 2 sock)};
- unless ($self->{oneshot}) {
- close($_) for @io;
- }
+ close($_) for (grep(defined, delete @$self{qw(0 1 2 sock)}));
if (my $cfg = $self->{cfg}) {
delete $cfg->{-lei_store};
}
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
}
- for (delete @$self{qw(3 old_1 au_done)}) {
- close($_) if defined($_);
- }
+ close($_) for (grep(defined, delete @$self{qw(3 old_1 au_done)}));
if (my $op_c = delete $self->{pkt_op_c}) {
close(delete $op_c->{sock});
}
@@ -1213,13 +1208,8 @@ sub oneshot {
local $quit = $exit if $exit;
local %PATH2CFG;
umask(077) // die("umask(077): $!");
- my $self = bless {
- oneshot => 1,
- 0 => *STDIN{GLOB},
- 1 => *STDOUT{GLOB},
- 2 => *STDERR{GLOB},
- env => \%ENV
- }, __PACKAGE__;
+ my $self = bless { oneshot => 1, env => \%ENV }, __PACKAGE__;
+ for (0..2) { open($self->{$_}, '+<&=', $_) or die "open fd=$_: $!" }
dispatch($self, @ARGV);
x_it($self, $self->{child_error}) if $self->{child_error};
}
diff --git a/t/lei.t b/t/lei.t
index 6d276050..8211c01d 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -154,6 +154,7 @@ my $test_fail = sub {
}
}
lei_ok('sucks', \'yes, but hopefully less every day');
+ like($lei_out, qr/loaded features/, 'loaded features shown');
SKIP: {
skip 'no curl', 3 unless which('curl');
lei(qw(q --only http://127.0.0.1:99999/bogus/ t:m));
^ permalink raw reply related [relevance 63%]
* Re: [PATCH 2/3] doc: lei q: split =item aliases onto separate lines
2021-04-28 4:51 54% ` [PATCH 2/3] doc: lei q: split =item aliases onto separate lines Eric Wong
@ 2021-04-29 1:39 71% ` Kyle Meyer
2021-04-29 1:57 71% ` Eric Wong
0 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-04-29 1:39 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> It makes L</--augment> look nicer without resorting to
> L<--augment|/-a, --augment> and similarly verbose nastiness.
>
> Having each option as a separate =item (with a blank line in
> between each =item) seems to be the preferred style used within
> Perl core documentation (I used perlrun.pod as an example),
> so we'll follow Perl core style, here.
Okay, thanks providing a rationale for using separate lines. I went
with the single line style based on some combination of a subjective
visual preference and looking at other manpages (presumably git's).
> This needs to be done for other manpages, at some point...
I should be able to get to another round of lei doc updates this
weekend.
^ permalink raw reply [relevance 71%]
* Re: [PATCH 2/3] doc: lei q: split =item aliases onto separate lines
2021-04-29 1:39 71% ` Kyle Meyer
@ 2021-04-29 1:57 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-29 1:57 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
>
> > It makes L</--augment> look nicer without resorting to
> > L<--augment|/-a, --augment> and similarly verbose nastiness.
> >
> > Having each option as a separate =item (with a blank line in
> > between each =item) seems to be the preferred style used within
> > Perl core documentation (I used perlrun.pod as an example),
> > so we'll follow Perl core style, here.
>
> Okay, thanks providing a rationale for using separate lines. I went
> with the single line style based on some combination of a subjective
> visual preference and looking at other manpages (presumably git's).
Yeah, also the existing public-inbox-* manpages were using
commas or slash; probably because I based them on git manpages :x
> > This needs to be done for other manpages, at some point...
>
> I should be able to get to another round of lei doc updates this
> weekend.
Thanks in advance. Also, in response to
https://public-inbox.org/meta/20210227180328.28057-1-kyle@kyleam.com/
w.r.t. lei-convert; I think it's helpful to document
since the WWW interface provides gzipped mboxrd and
IMAP|Maildir are probably the most commonly used.
^ permalink raw reply [relevance 71%]
* [PATCH 0/4] some lei synchronization work
@ 2021-04-29 9:46 69% Eric Wong
2021-04-29 9:46 33% ` [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more Eric Wong
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Eric Wong @ 2021-04-29 9:46 UTC (permalink / raw)
To: meta
2/4 is a fairly big change to recent-ish behavior; but ought to
help us avoid problems by avoiding mismatched sources of truth.
-watch once again halts on UIDVALIDITY changes (matching <=1.6
behavior).
Tor .onion URLs for NNTP and IMAP also aren't supported unless
the entire lei-daemon process is running under torsocks.
Net::NNTP doesn't seem to have provisions for using
user-supplied classes like IO::Socket::SOCKS, either.
Mail::IMAPClient supports ->Socket and ->RawSocket, at least...
Eric Wong (4):
content_hash: git_sha: allow unblessed SCALAR refs
lei import: avoid IMAPTracker, use LeiMailSync more
lei import: support UIDVALIDITY in IMAP URL
lei import: support shell completion of known folders
Documentation/lei-store-format.pod | 2 +-
contrib/completion/lei-completion.bash | 1 +
lib/PublicInbox/ContentHash.pm | 6 +--
lib/PublicInbox/IMAPTracker.pm | 35 +++++---------
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiExternal.pm | 22 +++++----
lib/PublicInbox/LeiImport.pm | 20 ++++----
lib/PublicInbox/LeiInput.pm | 20 +++++---
lib/PublicInbox/LeiMailSync.pm | 2 +-
lib/PublicInbox/LeiUp.pm | 4 +-
lib/PublicInbox/LeiViewText.pm | 1 -
lib/PublicInbox/NetReader.pm | 67 +++++++++++++++++---------
lib/PublicInbox/URIimap.pm | 2 +
t/lei-import-imap.t | 8 ++-
t/lei-import-nntp.t | 4 +-
t/solver_git.t | 7 +--
16 files changed, 112 insertions(+), 91 deletions(-)
^ permalink raw reply [relevance 69%]
* [PATCH 3/4] lei import: support UIDVALIDITY in IMAP URL
2021-04-29 9:46 69% [PATCH 0/4] some lei synchronization work Eric Wong
2021-04-29 9:46 33% ` [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more Eric Wong
@ 2021-04-29 9:46 61% ` Eric Wong
2021-04-29 9:46 57% ` [PATCH 4/4] lei import: support shell completion of known folders Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-29 9:46 UTC (permalink / raw)
To: meta
Specifying a UIDVALIDITY value allows the user to enforce
a strict match and force failure. This necessitated changes
to NetReader to allow die() and make error reporting more
suitable for CLI usage rather than daemonized usage of -watch.
---
lib/PublicInbox/LeiInput.pm | 10 +++++++++-
lib/PublicInbox/NetReader.pm | 7 +++++++
t/lei-import-imap.t | 4 ++++
3 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index ce675f40..277ad88d 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -293,6 +293,7 @@ $input is `eml', not --in-format=$in_fmt
$lei->err("# --sync is not supported for: @{$sync->{no}}");
}
if ($net) {
+ $net->{-can_die} = 1;
if (my $err = $net->errors) {
return $lei->fail($err);
}
@@ -306,10 +307,17 @@ $input is `eml', not --in-format=$in_fmt
sub process_inputs {
my ($self) = @_;
+ my $err;
for my $input (@{$self->{inputs}}) {
- $self->input_path_url($input);
+ eval { $self->input_path_url($input) };
+ next unless $@;
+ $err = "$input: $@";
+ last;
}
+ # always commit first, even on error partial work is acceptable for
+ # lei <import|tag|convert>
my $wait = $self->{lei}->{sto}->ipc_do('done') if $self->{lei}->{sto};
+ $self->{lei}->fail($err) if $err;
}
sub input_only_atfork_child {
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 81d25ead..3fc37b10 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -407,6 +407,11 @@ sub _imap_fetch_all ($$$) {
return "E: $orig_uri cannot get UIDVALIDITY";
$r_uidnext //= $mic->uidnext($mbx) //
return "E: $orig_uri cannot get UIDNEXT";
+ my $expect = $orig_uri->uidvalidity // $r_uidval;
+ return <<EOF if $expect != $r_uidval;
+E: $orig_uri UIDVALIDITY mismatch (got $r_uidval)
+EOF
+
my $uri = $orig_uri->clone;
my ($itrk, $l_uid, $l_uidval) = _itrk_last($self, $uri, $r_uidval);
return <<EOF if $l_uidval != $r_uidval;
@@ -520,6 +525,7 @@ sub imap_each {
} else {
$err = "E: <$uri> not connected: $!";
}
+ die $err if $err && $self->{-can_die};
warn $err if $err;
$mic;
}
@@ -620,6 +626,7 @@ sub nntp_each {
} else {
$err = "E: <$uri> not connected: $!";
}
+ die $err if $err && $self->{-can_die};
warn $err if $err;
$nn;
}
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 611328b4..c977c68e 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -24,6 +24,10 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok('import', $url);
lei_ok 'ls-sync';
like($lei_out, qr!\A\Q$url\E;UIDVALIDITY=\d+\n\z!, 'ls-sync');
+ chomp(my $u = $lei_out);
+ lei_ok('import', $u, \'UIDVALIDITY match in URL');
+ $u =~ s/;UIDVALIDITY=(\d+)\s*/;UIDVALIDITY=9$1/s;
+ ok(!lei('import', $u), 'UIDVALIDITY mismatch in URL rejected');
lei_ok('inspect', $url);
my $inspect = json_utf8->decode($lei_out);
^ permalink raw reply related [relevance 61%]
* [PATCH 4/4] lei import: support shell completion of known folders
2021-04-29 9:46 69% [PATCH 0/4] some lei synchronization work Eric Wong
2021-04-29 9:46 33% ` [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more Eric Wong
2021-04-29 9:46 61% ` [PATCH 3/4] lei import: support UIDVALIDITY in IMAP URL Eric Wong
@ 2021-04-29 9:46 57% ` Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-29 9:46 UTC (permalink / raw)
To: meta
This also fixes completion of "lei up" for IMAP folders.
---
contrib/completion/lei-completion.bash | 1 +
lib/PublicInbox/LeiExternal.pm | 22 ++++++++++++----------
lib/PublicInbox/LeiImport.pm | 8 ++++++++
lib/PublicInbox/LeiUp.pm | 4 ++--
4 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/contrib/completion/lei-completion.bash b/contrib/completion/lei-completion.bash
index 2c28d44a..5c137e68 100644
--- a/contrib/completion/lei-completion.bash
+++ b/contrib/completion/lei-completion.bash
@@ -9,6 +9,7 @@ _lei() {
*':'* | *'='* | '//'*) compopt -o nospace ;;
*) compopt +o nospace ;; # the default
esac
+ wordlist="${wordlist//;/\\\\;}" # escape ';' for ';UIDVALIDITY' and such
COMPREPLY=($(compgen -W "$wordlist" -- "${COMP_WORDS[COMP_CWORD]}"))
return 0
}
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 3858085e..6fd3efef 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -215,7 +215,8 @@ sub lei_forget_external {
}
}
-sub complete_url_common {
+# returns an anonymous sub which returns an array of potential results
+sub complete_url_prepare {
my $argv = $_[-1];
# Workaround bash word-splitting URLs to ['https', ':', '//' ...]
# Maybe there's a better way to go about this in
@@ -239,37 +240,38 @@ sub complete_url_common {
}
$re = quotemeta($re);
}
- ($cur, $re);
+ my $match_cb = sub {
+ # only return the part specified on the CLI
+ # don't duplicate if already 100% completed
+ $_[0] =~ /\A$re(\Q$cur\E.*)/ ? ($cur eq $1 ? () : $1) : ()
+ };
+ wantarray ? ($re, $cur, $match_cb) : $match_cb;
}
# shell completion helper called by lei__complete
sub _complete_forget_external {
my ($self, @argv) = @_;
my $cfg = $self->_lei_cfg;
- my ($cur, $re) = complete_url_common(\@argv);
+ my ($cur, $re, $match_cb) = complete_url_prepare(\@argv);
# FIXME: bash completion off "http:" or "https:" when the last
# character is a colon doesn't work properly even if we're
# returning "//$HTTP_HOST/$PATH_INFO/", not sure why, could
# be a bash issue.
map {
- my $x = substr($_, length('external.'));
- # only return the part specified on the CLI
- # don't duplicate if already 100% completed
- $x =~ /\A$re(\Q$cur\E.*)/ ? ($cur eq $1 ? () : $1) : ();
+ $match_cb->(substr($_, length('external.')));
} grep(/\Aexternal\.$re\Q$cur/, @{$cfg->{-section_order}});
}
sub _complete_add_external { # for bash, this relies on "compopt -o nospace"
my ($self, @argv) = @_;
my $cfg = $self->_lei_cfg;
- my ($cur, $re) = complete_url_common(\@argv);
+ my $match_cb = complete_url_prepare(\@argv);
require URI;
map {
my $u = URI->new(substr($_, length('external.')));
my ($base) = ($u->path =~ m!((?:/?.*)?/)[^/]+/?\z!);
$u->path($base);
- $u = $u->as_string;
- $u =~ /\A$re(\Q$cur\E.*)/ ? ($cur eq $1 ? () : $1) : ();
+ $match_cb->($u->as_string);
} grep(m!\Aexternal\.https?://!, @{$cfg->{-section_order}});
}
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 277f4f95..def121ab 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -89,6 +89,14 @@ sub lei_import { # the main "lei import" method
$op_c->op_wait_event($ops);
}
+sub _complete_import {
+ my ($lei, @argv) = @_;
+ my $sto = $lei->_lei_store or return;
+ my $lms = $sto->search->lms or return;
+ my $match_cb = $lei->complete_url_prepare(\@argv);
+ map { $match_cb->($_) } $lms->folders;
+}
+
no warnings 'once';
*ipc_atfork_child = \&PublicInbox::LeiInput::input_only_atfork_child;
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index f4ff070b..4399c4fb 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -99,8 +99,8 @@ sub lei_up {
sub _complete_up {
my ($lei, @argv) = @_;
- my ($cur, $re) = $lei->complete_url_common(\@argv);
- grep(/\A$re\Q$cur/, PublicInbox::LeiSavedSearch::list($lei));
+ my $match_cb = $lei->complete_url_prepare(\@argv);
+ map { $match_cb->($_) } PublicInbox::LeiSavedSearch::list($lei);
}
1;
^ permalink raw reply related [relevance 57%]
* [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more
2021-04-29 9:46 69% [PATCH 0/4] some lei synchronization work Eric Wong
@ 2021-04-29 9:46 33% ` Eric Wong
2021-04-29 9:46 61% ` [PATCH 3/4] lei import: support UIDVALIDITY in IMAP URL Eric Wong
2021-04-29 9:46 57% ` [PATCH 4/4] lei import: support shell completion of known folders Eric Wong
2 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-29 9:46 UTC (permalink / raw)
To: meta
IMAPTracker has a UNIQUE constraint on the `url' column,
which may cause compatibility and/or rollback problems
in attempting to deal with UIDVALIDITY changes.
Having multiple sources of truth leads to confusion and bugs,
so relying on LeiMailSync exclusively ought to simplify things.
Furthermore, since LeiMailSync is only written to by LeiStore,
it is safer in that it won't mark a UID or article as imported
until git-fast-import has seen it, and the SQLite commit always
happens after "done\n" is sent to fast-import.
This mostly reverts recent commits to IMAPTracker to support
lei, those are:
1) commit 7632d8f7590daf70c65d4270e750c36552fa9389
("net_reader: restart on first UID when UIDVALIDITY changes")
2) commit 311a5d37ad275cd75b1e64d87827c4d13fe4bfab
("imap_tracker: prepare for use with lei").
This means public-inbox-watch will not change between 1.6 and
1.7: -watch stops synching a folder when UIDVALIDITY changes.
---
Documentation/lei-store-format.pod | 2 +-
lib/PublicInbox/IMAPTracker.pm | 35 ++++++-----------
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 12 ++----
lib/PublicInbox/LeiInput.pm | 10 ++---
lib/PublicInbox/LeiMailSync.pm | 2 +-
lib/PublicInbox/NetReader.pm | 60 ++++++++++++++++++------------
lib/PublicInbox/URIimap.pm | 2 +
t/lei-import-imap.t | 4 +-
t/lei-import-nntp.t | 4 +-
10 files changed, 64 insertions(+), 69 deletions(-)
diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod
index 3e1ddc65..71aa72cb 100644
--- a/Documentation/lei-store-format.pod
+++ b/Documentation/lei-store-format.pod
@@ -32,7 +32,7 @@ prevent them from being accidentally treated as a v2 inbox.
~/.local/share/lei/store
- ipc.lock # lock file for internal lei IPC
- local/$EPOCH.git # normal bare git repositories
- - net_last.sqlite3 # import state for IMAP & NNTP
+ - mail_sync.sqlite3 # sync state IMAP, Maildir, NNTP
Additionally, the following share the same roles they do in extindex:
diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
index fe813582..5eb33cf7 100644
--- a/lib/PublicInbox/IMAPTracker.pm
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -39,20 +39,12 @@ sub dbh_new ($) {
$dbh;
}
-sub get_last ($;$) {
- my ($self, $validity) = @_;
- my $sth;
- if (defined $validity) {
- $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
-SELECT uid_validity, uid FROM imap_last WHERE url = ? AND uid_validity = ?
-
- $sth->execute($self->{url}, $validity);
- } else {
- $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+sub get_last ($) {
+ my ($self) = @_;
+ my $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
SELECT uid_validity, uid FROM imap_last WHERE url = ?
- $sth->execute($self->{url});
- }
+ $sth->execute($self->{url});
$sth->fetchrow_array;
}
@@ -70,19 +62,16 @@ VALUES (?, ?, ?)
}
sub new {
- my ($class, $url, $dbname) = @_;
+ my ($class, $url) = @_;
- unless (defined($dbname)) {
- # original name for compatibility with old setups:
- $dbname = PublicInbox::Config->config_dir() . '/imap.sqlite3';
+ # original name for compatibility with old setups:
+ my $dbname = PublicInbox::Config->config_dir() . '/imap.sqlite3';
- # use the new XDG-compliant name for new setups:
- if (!-f $dbname) {
- $dbname = ($ENV{XDG_DATA_HOME} //
- (($ENV{HOME} // '/nonexistent').
- '/.local/share')) .
- '/public-inbox/imap.sqlite3';
- }
+ # use the new XDG-compliant name for new setups:
+ if (!-f $dbname) {
+ $dbname = ($ENV{XDG_DATA_HOME} //
+ (($ENV{HOME} // '/nonexistent').'/.local/share')) .
+ '/public-inbox/imap.sqlite3';
}
if (!-f $dbname) {
require File::Path;
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 1ea7c9ca..52ce8ec2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -211,7 +211,7 @@ our %CMD = ( # sorted in order of importance/use:
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw! verbose|v+ incremental! sync!), @c_opt ],
+ lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 26127ece..277f4f95 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -41,18 +41,13 @@ sub input_maildir_cb { # maildir_each_eml cb
input_eml_cb($self, $eml, $vmd);
}
-sub input_imap_cb { # imap_each
+sub input_net_cb { # imap_each / nntp_each
my ($url, $uid, $kw, $eml, $self) = @_;
my $vmd = $self->{-import_kw} ? { kw => $kw } : undef;
$vmd->{sync_info} = [ $url, $uid ] if $self->{-mail_sync};
input_eml_cb($self, $eml, $vmd);
}
-sub input_nntp_cb { # nntp_each
- my ($url, $num, $kw, $eml, $self) = @_;
- input_eml_cb($self, $eml, $self->{-import_kw} ? { kw => $kw } : undef);
-}
-
sub net_merge_complete { # callback used by LeiAuth
my ($self) = @_;
$self->wq_io_do('process_inputs');
@@ -69,7 +64,7 @@ sub lei_import { # the main "lei import" method
return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
$self->{all_vmd} = $vmd_mod if scalar keys %$vmd_mod;
$self->prepare_inputs($lei, \@inputs) or return;
- $self->{-mail_sync} = $lei->{opt}->{sync} // 1;
+ $self->{-mail_sync} = $lei->{opt}->{'mail-sync'} // 1;
$lei->ale; # initialize for workers to read
my $j = $lei->{opt}->{jobs} // scalar(@{$self->{inputs}}) || 1;
@@ -77,8 +72,7 @@ sub lei_import { # the main "lei import" method
# $j = $net->net_concurrency($j); TODO
if ($lei->{opt}->{incremental} // 1) {
$net->{incremental} = 1;
- $net->{itrk_fn} = $lei->store_path .
- '/net_last.sqlite3';
+ $net->{-lms_ro} = $lei->_lei_store->search->lms // 0;
}
} else {
my $nproc = $self->detect_nproc;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 785e607d..ce675f40 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -110,14 +110,12 @@ sub input_path_url {
my $ifmt = lc($lei->{opt}->{'in-format'} // '');
# TODO auto-detect?
if ($input =~ m!\Aimaps?://!i) {
- $lei->{net}->imap_each($input, $self->can('input_imap_cb') //
- $self->can('input_net_cb'),
- $self, @args);
+ $lei->{net}->imap_each($input, $self->can('input_net_cb'),
+ $self, @args);
return;
} elsif ($input =~ m!\A(?:nntps?|s?news)://!i) {
- $lei->{net}->nntp_each($input, $self->can('input_nntp_cb') //
- $self->can('input_net_cb'),
- $self, @args);
+ $lei->{net}->nntp_each($input, $self->can('input_net_cb'),
+ $self, @args);
return;
} elsif ($input =~ m!\Ahttps?://!i) {
handle_http_input($self, $input, @args);
diff --git a/lib/PublicInbox/LeiMailSync.pm b/lib/PublicInbox/LeiMailSync.pm
index 52f26d69..2ce189fa 100644
--- a/lib/PublicInbox/LeiMailSync.pm
+++ b/lib/PublicInbox/LeiMailSync.pm
@@ -143,7 +143,7 @@ sub each_src {
}
sub location_stats {
- my ($self, $folder, $cb, @args) = @_;
+ my ($self, $folder) = @_;
my $dbh = $self->{dbh} //= dbh_new($self);
my $fid;
my $ret = {};
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 5978752f..81d25ead 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -235,7 +235,7 @@ sub imap_common_init ($;$) {
$self->{quiet} = 1 if $lei && $lei->{opt}->{quiet};
eval { require PublicInbox::IMAPClient } or
die "Mail::IMAPClient is required for IMAP:\n$@\n";
- eval { require PublicInbox::IMAPTracker } or
+ ($lei || eval { require PublicInbox::IMAPTracker }) or
die "DBD::SQLite is required for IMAP\n:$@\n";
require PublicInbox::URIimap;
my $cfg = $self->{pi_cfg} // $lei->_lei_cfg;
@@ -283,7 +283,7 @@ sub nntp_common_init ($;$) {
$self->{quiet} = 1 if $lei && $lei->{opt}->{quiet};
eval { require Net::NNTP } or
die "Net::NNTP is required for NNTP:\n$@\n";
- eval { require PublicInbox::IMAPTracker } or
+ ($lei || eval { require PublicInbox::IMAPTracker }) or
die "DBD::SQLite is required for NNTP\n:$@\n";
my $cfg = $self->{pi_cfg} // $lei->_lei_cfg;
my $nn_args = {}; # scheme://authority => Net::NNTP->new arg
@@ -373,17 +373,28 @@ sub run_commit_cb ($) {
$cb->(@args);
}
-sub _itrk ($$) {
- my ($self, $uri) = @_;
- return unless $self->{incremental};
- # itrk_fn is set by lei
- PublicInbox::IMAPTracker->new($$uri, $self->{itrk_fn});
+sub _itrk_last ($$;$) {
+ my ($self, $uri, $r_uidval) = @_;
+ return (undef, undef, $r_uidval) unless $self->{incremental};
+ my ($itrk, $l_uid, $l_uidval);
+ if (defined(my $lms = $self->{-lms_ro})) { # LeiMailSync or 0
+ $uri->uidvalidity($r_uidval) if defined $r_uidval;
+ my $x;
+ $l_uid = ($lms && ($x = $lms->location_stats($$uri))) ?
+ $x->{'uid.max'} : undef;
+ # itrk remains undef, lei/store worker writes to
+ # mail_sync.sqlite3
+ } else {
+ $itrk = PublicInbox::IMAPTracker->new($$uri);
+ ($l_uidval, $l_uid) = $itrk->get_last($$uri);
+ }
+ ($itrk, $l_uid, $l_uidval //= $r_uidval);
}
sub _imap_fetch_all ($$$) {
- my ($self, $mic, $uri) = @_;
- my $sec = uri_section($uri);
- my $mbx = $uri->mailbox;
+ my ($self, $mic, $orig_uri) = @_;
+ my $sec = uri_section($orig_uri);
+ my $mbx = $orig_uri->mailbox;
$mic->Clear(1); # trim results history
$mic->examine($mbx) or return "E: EXAMINE $mbx ($sec) failed: $!";
my ($r_uidval, $r_uidnext);
@@ -393,20 +404,22 @@ sub _imap_fetch_all ($$$) {
last if $r_uidval && $r_uidnext;
}
$r_uidval //= $mic->uidvalidity($mbx) //
- return "E: $uri cannot get UIDVALIDITY";
+ return "E: $orig_uri cannot get UIDVALIDITY";
$r_uidnext //= $mic->uidnext($mbx) //
- return "E: $uri cannot get UIDNEXT";
- my $url = ref($uri)->new($$uri);
- $url->uidvalidity($r_uidval);
- $url = $$url;
- my $itrk = _itrk($self, $uri);
- my $l_uid;
- $l_uid = $itrk->get_last($r_uidval) if $itrk;
+ return "E: $orig_uri cannot get UIDNEXT";
+ my $uri = $orig_uri->clone;
+ my ($itrk, $l_uid, $l_uidval) = _itrk_last($self, $uri, $r_uidval);
+ return <<EOF if $l_uidval != $r_uidval;
+E: $uri UIDVALIDITY mismatch
+E: local=$l_uidval != remote=$r_uidval
+EOF
+ $uri->uidvalidity($r_uidval);
$l_uid //= 0;
my $r_uid = $r_uidnext - 1;
- if ($l_uid > $r_uid) {
- return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n";
- }
+ return <<EOF if $l_uid > $r_uid;
+E: $uri local UID exceeds remote ($l_uid > $r_uid)
+E: $uri strangely, UIDVALIDLITY matches ($l_uidval)
+EOF
return if $l_uid >= $r_uid; # nothing to do
$l_uid ||= 1;
my ($mod, $shard) = @{$self->{shard_info} // []};
@@ -458,7 +471,7 @@ sub _imap_fetch_all ($$$) {
# messages get deleted, so holes appear
my $per_uid = delete $r->{$uid} // next;
my $raw = delete($per_uid->{$key}) // next;
- _imap_do_msg($self, $url, $uid, \$raw,
+ _imap_do_msg($self, $$uri, $uid, \$raw,
$per_uid->{FLAGS});
$last_uid = $uid;
last if $self->{quit};
@@ -547,8 +560,7 @@ sub _nntp_fetch_all ($$$) {
# IMAPTracker is also used for tracking NNTP, UID == article number
# LIST.ACTIVE can get the equivalent of UIDVALIDITY, but that's
# expensive. So we assume newsgroups don't change:
- my $itrk = _itrk($self, $uri);
- my (undef, $l_art) = $itrk ? $itrk->get_last : ();
+ my ($itrk, $l_art) = _itrk_last($self, $uri);
# allow users to specify articles to refetch
# cf. https://tools.ietf.org/id/draft-gilman-news-url-01.txt
diff --git a/lib/PublicInbox/URIimap.pm b/lib/PublicInbox/URIimap.pm
index df9f5fd9..f6244137 100644
--- a/lib/PublicInbox/URIimap.pm
+++ b/lib/PublicInbox/URIimap.pm
@@ -144,4 +144,6 @@ sub scheme {
sub as_string { ${$_[0]} }
+sub clone { ref($_[0])->new(as_string($_[0])) }
+
1;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index cf1fa49d..611328b4 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -45,8 +45,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
lei_ok([qw(tag +kw:seen), $url], undef, undef);
- my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
- ok(-s $f, 'net tracked for redundant imports');
+ my $f = "$ENV{HOME}/.local/share/lei/store/mail_sync.sqlite3";
+ ok(-s $f, 'mail_sync tracked for redundant imports');
lei_ok('inspect', "blob:$out->[5]->{blob}");
my $x = json_utf8->decode($lei_out);
is(ref($x->{'lei/store'}), 'ARRAY', 'lei/store in inspect');
diff --git a/t/lei-import-nntp.t b/t/lei-import-nntp.t
index d795a86a..12bb002a 100644
--- a/t/lei-import-nntp.t
+++ b/t/lei-import-nntp.t
@@ -27,7 +27,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
for (@$out) { $r{ref($_)}++ }
is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
- my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
- ok(-s $f, 'net tracked for redundant imports');
+ my $f = "$ENV{HOME}/.local/share/lei/store/mail_sync.sqlite3";
+ ok(-s $f, 'mail_sync exists tracked for redundant imports');
});
done_testing;
^ permalink raw reply related [relevance 33%]
* [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate
2021-04-30 9:24 67% [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
@ 2021-04-30 9:24 71% ` Eric Wong
2021-04-30 9:24 71% ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
It's helpful for us to distinguish x86 kernels from x86_64
kernels when using an x86 userspace. OSes are dropping i386
support and only support i486 and newer, so "x86" is a more
appropriate description for that platform than "i386".
---
lib/PublicInbox/LeiSucks.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
index d364a856..2ce64d62 100644
--- a/lib/PublicInbox/LeiSucks.pm
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -18,7 +18,8 @@ sub lei_sucks {
$lei->start_pager if -t $lei->{1};
my ($os, undef, $rel, undef, $mac)= POSIX::uname();
if ($mac eq 'x86_64' && $Config{ptrsize} == 4) {
- $mac = $Config{cppsymbols} =~ /\b__ILP32__=1\b/ ? 'x32' : 'i386'
+ $mac .= $Config{cppsymbols} =~ /\b__ILP32__=1\b/ ?
+ ',u=x32' : ',u=x86';
}
eval { require PublicInbox };
my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';
^ permalink raw reply related [relevance 71%]
* [PATCH 3/8] lei: kill old PIDs when dropping
2021-04-30 9:24 67% [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
2021-04-30 9:24 71% ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
@ 2021-04-30 9:24 71% ` Eric Wong
2021-04-30 9:24 70% ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
2021-04-30 9:24 45% ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
This ensures hitting Ctrl-C on a long-running "lei convert" or
similar will stop the WQ worker, even after we've closed
the WQ socketpair in the daemon.
---
lib/PublicInbox/LEI.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 52ce8ec2..3468094f 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -387,7 +387,14 @@ my @WQ_KEYS = qw(lxs l2m wq1); # internal workers
sub _drop_wq {
my ($self) = @_;
- for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) { $wq->DESTROY }
+ for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) {
+ if ($wq->wq_kill) {
+ $wq->wq_close(0, undef, $self);
+ } elsif ($wq->wq_kill_old) {
+ $wq->wq_wait_old(undef, $self);
+ }
+ $wq->DESTROY;
+ }
}
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
^ permalink raw reply related [relevance 71%]
* [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes
@ 2021-04-30 9:24 67% Eric Wong
2021-04-30 9:24 71% ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
Attempting to use torsocks(1) for NNTP or IMAP could get tricky.
Fortunately, IO::Socket::Socks is packaged for on CentOS 7,
FreeBSD, and Debian, so it seems to be a reasonable way to
support NNTP and IMAP Tor onions.
--proxy= (shared with curl) is supported for one-off
command-line use, but imap.proxy and nntp.proxy are both
supported along with URL-matching variants with git 1.8.5 (or
git 2.26 for wildcard URL matching).
Only socks5h:// proxies are supported (the default with
IO::Socket::Socks), which is what Tor uses. I doubt its worth
the effort (and potential for DNS request leaks) to support
prior versions of SOCKS in 2021.
Eric Wong (8):
lei sucks: preserve utsname.machine, add "x86" where appropriate
lei_curl: improve correctness of LD_PRELOAD check
lei: kill old PIDs when dropping
lei: ensure autoflush(1) is on STDERR
net_reader: {nn,mic}_for: use prototypes for internal subs
lei: IMAP .onion support via --proxy=s switch
net_reader: Net::NNTP --proxy=socks5h:// support
net_reader: support (imap|nntp).proxy in config file
MANIFEST | 2 +
lib/PublicInbox/Config.pm | 1 +
lib/PublicInbox/LEI.pm | 24 ++++++++---
lib/PublicInbox/LeiCurl.pm | 2 +-
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiSucks.pm | 3 +-
lib/PublicInbox/LeiToMail.pm | 4 +-
lib/PublicInbox/NetNNTPSocks.pm | 33 +++++++++++++++
lib/PublicInbox/NetReader.pm | 72 +++++++++++++++++++++++++++------
xt/net_nntp_socks.t | 22 ++++++++++
10 files changed, 141 insertions(+), 24 deletions(-)
create mode 100644 lib/PublicInbox/NetNNTPSocks.pm
create mode 100644 xt/net_nntp_socks.t
^ permalink raw reply [relevance 67%]
* [PATCH 4/8] lei: ensure autoflush(1) is on STDERR
2021-04-30 9:24 67% [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
2021-04-30 9:24 71% ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
2021-04-30 9:24 71% ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
@ 2021-04-30 9:24 70% ` Eric Wong
2021-04-30 9:24 45% ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
This fixes error reporting for oneshot tests in xt/lei-auth-failure.t
---
lib/PublicInbox/LEI.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 3468094f..6a82d497 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -500,6 +500,7 @@ sub _lei_atfork_child {
}
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
+ STDERR->autoflush(1);
}
close($_) for (grep(defined, delete @$self{qw(3 old_1 au_done)}));
if (my $op_c = delete $self->{pkt_op_c}) {
@@ -676,6 +677,7 @@ sub lazy_cb ($$$) {
sub dispatch {
my ($self, $cmd, @argv) = @_;
local $current_lei = $self; # for __WARN__
+ $self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
dump_and_clear_log("from previous run\n");
return _help($self, 'no command given') unless defined($cmd);
# do not support Getopt bundling for this
@@ -1006,7 +1008,6 @@ sub accept_dispatch { # Listener {post_accept} callback
}
$i == 4 or return send($sock, 'not enough FDs='.($i-1), MSG_EOR)
}
- $self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
# $ENV_STR = join('', map { "\0$_=$ENV{$_}" } keys %ENV);
# $buf = "$argc\0".join("\0", @ARGV).$ENV_STR."\0\0";
substr($buf, -2, 2, '') eq "\0\0" or # s/\0\0\z//
^ permalink raw reply related [relevance 70%]
* [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch
2021-04-30 9:24 67% [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
` (2 preceding siblings ...)
2021-04-30 9:24 70% ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
@ 2021-04-30 9:24 45% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
Mail::IMAPClient provides the ability to pass a pre-connected
Socket to it. We can rely on this functionality to use
IO::Socket::Socks in place whatever socket class
Mail::IMAPClient chooses to use.
The --proxy=s is shared with curl(1), though we only support
socks5h:// at the moment. Is there any need for SOCKS4 or SOCKS5
without name resolution? Tor .onions require socks5h:// for
name resolution and to prevent data leakage.
---
lib/PublicInbox/LEI.pm | 12 ++++++++----
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 4 ++--
lib/PublicInbox/NetReader.pm | 31 ++++++++++++++++++++++++++++---
4 files changed, 39 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 6a82d497..bb67fc0b 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -188,7 +188,8 @@ our %CMD = ( # sorted in order of importance/use:
qw(stdin| threads|t from|f=s mid=s oid=s), @c_opt ],
'tag' => [ 'KEYWORDS...',
'set/unset keywords and/or labels on message(s)',
- qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@), @c_opt,
+ qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@),
+ qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt,
pass_through('-kw:foo for delete') ],
'forget' => [ '[--stdin|--oid=OID|--by-mid=MID]',
"exclude message(s) on stdin from `q' search results",
@@ -211,11 +212,12 @@ our %CMD = ( # sorted in order of importance/use:
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!), @c_opt ],
+ lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!),
+ qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
- qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
- lock=s@ kw!), @c_opt ],
+ qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s lock=s@ kw!),
+ qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt ],
'p2q' => [ 'FILE|COMMIT_OID|--stdin',
"use a patch to generate a query for `lei q --stdin'",
qw(stdin| want|w=s@ uri debug), @c_opt ],
@@ -277,6 +279,8 @@ my %OPTDESC = (
'path-a|a=s' => 'pre-image pathname associated with OID',
'path-b|b=s' => 'post-image pathname associated with OID',
'git-dir=s@' => 'additional git repository to scan',
+'proxy=s' => [ 'PROTO://HOST[:PORT]', # shared with curl(1)
+ "proxy for (e.g. `socks5h://0:9050')" ],
'torsocks=s' => ['VAL|auto|no|yes',
'whether or not to wrap git and curl commands with torsocks'],
'no-torsocks' => 'alias for --torsocks=no',
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 277ad88d..86f300c3 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -294,7 +294,7 @@ $input is `eml', not --in-format=$in_fmt
}
if ($net) {
$net->{-can_die} = 1;
- if (my $err = $net->errors) {
+ if (my $err = $net->errors($lei)) {
return $lei->fail($err);
}
$net->{quiet} = $lei->{opt}->{quiet};
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index fa3af710..eda4701c 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -351,14 +351,14 @@ sub new {
require PublicInbox::MboxReader;
$self->can("eml2$fmt") or die "bad mbox format: $fmt\n";
$self->{base_type} = 'mbox';
- } elsif ($fmt =~ /\Aimaps?\z/) { # TODO .onion support
+ } elsif ($fmt =~ /\Aimaps?\z/) {
require PublicInbox::NetWriter;
require PublicInbox::URIimap;
my $net = PublicInbox::NetWriter->new;
$net->{quiet} = $lei->{opt}->{quiet};
my $uri = PublicInbox::URIimap->new($dst)->canonical;
$net->add_url($uri);
- my $err = $net->errors;
+ my $err = $net->errors($lei);
return $lei->fail($err) if $err;
$uri->mailbox or return $lei->fail("No mailbox: $dst");
$self->{uri} = $uri;
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index b9365c05..ac23e701 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -7,6 +7,7 @@ use strict;
use v5.10.1;
use parent qw(Exporter PublicInbox::IPC);
use PublicInbox::Eml;
+use PublicInbox::Config;
our %IMAPflags2kw = map {; "\\\u$_" => $_ } qw(seen answered flagged draft);
$IMAPflags2kw{'$Forwarded'} = 'forwarded'; # RFC 5550
@@ -51,7 +52,16 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
%$common, # may set Starttls, Compress, Debug ....
};
require PublicInbox::IMAPClient;
- my $mic = PublicInbox::IMAPClient->new(%$mic_arg) or
+ my %socks;
+ if ($lei && $lei->{socks5h}) {
+ my %opt = %{$lei->{socks5h}};
+ $opt{ConnectAddr} = delete $mic_arg->{Server};
+ $opt{ConnectPort} = delete $mic_arg->{Port};
+ $socks{Socket} = IO::Socket::Socks->new(%opt) or die
+ "E: <$url> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
+ $self->{mic_socks5h} = \%opt;
+ }
+ my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks) or
die "E: <$url> new: $@\n";
# default to using STARTTLS if it's available, but allow
@@ -331,7 +341,7 @@ sub add_url {
}
sub errors {
- my ($self) = @_;
+ my ($self, $lei) = @_;
if (my $u = $self->{unsupported_url}) {
return "Unsupported URL(s): @$u";
}
@@ -343,6 +353,16 @@ sub errors {
eval { require Net::NNTP } or
die "Net::NNTP is required for NNTP:\n$@\n";
}
+ if ($lei && (($lei->{opt}->{proxy}//'') =~ m!\Asocks5h://
+ (?: \[ ([^\]]+) \] | ([^:/]+) )
+ (?::([0-9]+))?/?(?:,|\z)!ix)) {
+ my ($h, $p) = ($1 // $2, $3 + 0);
+ $h = '127.0.0.1' if $h eq '0';
+ eval { require IO::Socket::Socks } or die <<EOM;
+IO::Socket::Socks missing for socks5h://$h:$p
+EOM
+ $lei->{socks5h} = { ProxyAddr => $h, ProxyPort => $p };
+ }
undef;
}
@@ -507,7 +527,12 @@ sub mic_get {
$mic_arg->{Authcallback} = $self->can($cb_name);
}
}
- my $mic = PublicInbox::IMAPClient->new(%$mic_arg);
+ my %socks;
+ if (my $s5h = $self->{mic_socks5h}) {
+ $socks{Socket} = IO::Socket::Socks->new(%$s5h) or die
+ "E: <$$uri> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
+ }
+ my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks);
$cached //= {}; # invalid placeholder if no cache enabled
$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
}
^ permalink raw reply related [relevance 45%]
* [PATCH 5/5] lei edit-search: support relocating lei.q.output
2021-05-01 6:21 70% [PATCH 0/5] lei: more UI/UX tweaks Eric Wong
` (3 preceding siblings ...)
2021-05-01 6:21 56% ` [PATCH 4/5] lei: rename ls-sync to ls-mail-sync Eric Wong
@ 2021-05-01 6:21 75% ` Eric Wong
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-01 6:21 UTC (permalink / raw)
To: meta
The contents of the old lei.q.output will not be removed,
but will be converted into the new one.
---
lib/PublicInbox/LeiConvert.pm | 5 ++-
lib/PublicInbox/LeiEditSearch.pm | 15 +++++++--
lib/PublicInbox/LeiSavedSearch.pm | 51 +++++++++++++++++++++++++++++++
script/lei | 7 +++++
4 files changed, 74 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index cefcaf65..5b27ec2d 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -37,8 +37,11 @@ sub process_inputs { # via wq_do
my ($self) = @_;
local $PublicInbox::DS::in_loop = 0; # force synchronous dwaitpid
$self->SUPER::process_inputs;
- delete $self->{lei}->{1};
+ my $lei = $self->{lei};
+ delete $lei->{1};
delete $self->{wcb}; # commit
+ my $nr = delete($lei->{-nr_write}) // 0;
+ $lei->err("# converted $nr messages") if $lei->{opt}->{verbose};
}
sub lei_convert { # the main "lei convert" method
diff --git a/lib/PublicInbox/LeiEditSearch.pm b/lib/PublicInbox/LeiEditSearch.pm
index fb36fdcd..30ac65bd 100644
--- a/lib/PublicInbox/LeiEditSearch.pm
+++ b/lib/PublicInbox/LeiEditSearch.pm
@@ -13,10 +13,19 @@ sub lei_edit_search {
my $lss = PublicInbox::LeiSavedSearch->up($lei, $out) or return;
my @cmd = (qw(git config --edit -f), $lss->{'-f'});
$lei->qerr("# spawning @cmd");
+ $lss->edit_begin($lei);
if ($lei->{oneshot}) {
- exec(@cmd) or die "exec @cmd: $!\n";
- } else {
- $lei->send_exec_cmd([], \@cmd, {});
+ require PublicInbox::Spawn;
+ waitpid(PublicInbox::Spawn::spawn(\@cmd), 0);
+ # non-fatal, editor could fail after successful write
+ $lei->child_error($?) if $?;
+ $lss->edit_done($lei);
+ } else { # run in script/lei foreground
+ require PublicInbox::PktOp;
+ my ($op_c, $op_p) = PublicInbox::PktOp->pair;
+ # $op_p will EOF when $EDITOR is done
+ $op_c->{ops} = { '' => [$lss->can('edit_done'), $lss, $lei] };
+ $lei->send_exec_cmd([ @$lei{qw(0 1 2)}, $op_p ], \@cmd, {});
}
}
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 79125214..8177c98e 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -259,6 +259,57 @@ sub output2lssdir {
undef;
}
+sub edit_begin {
+ my ($self, $lei) = @_;
+ if (ref($self->{-cfg}->{'lei.q.output'})) {
+ delete $self->{-cfg}->{'lei.q.output'}; # invalid
+ $lei->err(<<EOM);
+$self->{-f} has multiple values of lei.q.output
+please remove redundant ones
+EOM
+ }
+ $lei->{-lss_for_edit} = $self;
+}
+
+sub edit_done {
+ my ($self, $lei) = @_;
+ my $cfg = PublicInbox::Config->git_config_dump($self->{'-f'});
+ my $new_out = $cfg->{'lei.q.output'} // '';
+ return $lei->fail(<<EOM) if ref $new_out;
+$self->{-f} has multiple values of lei.q.output
+please edit again
+EOM
+ return $lei->fail(<<EOM) if $new_out eq '';
+$self->{-f} needs lei.q.output
+please edit again
+EOM
+ my $old_out = $self->{-cfg}->{'lei.q.output'} // '';
+ return if $old_out eq $new_out;
+ my $old_path = $old_out;
+ my $new_path = $new_out;
+ s!$LOCAL_PFX!! for ($old_path, $new_path);
+ my $dir_old = lss_dir_for($lei, \$old_path, 1);
+ my $dir_new = lss_dir_for($lei, \$new_path);
+ return if $dir_new eq $dir_old; # no change, likely
+ return $lei->fail(<<EOM) if -e $dir_new;
+lei.q.output changed from `$old_out' to `$new_out'
+However, $dir_new exists
+EOM
+ # start the conversion asynchronously
+ my $old_sq = PublicInbox::Config::squote_maybe($old_out);
+ my $new_sq = PublicInbox::Config::squote_maybe($new_out);
+ $lei->puts("lei.q.output changed from $old_sq to $new_sq");
+ $lei->qerr("# lei convert $old_sq -o $new_sq");
+ my $v = !$lei->{opt}->{quiet};
+ $lei->{opt} = { output => $new_out, verbose => $v };
+ require PublicInbox::LeiConvert;
+ PublicInbox::LeiConvert::lei_convert($lei, $old_out);
+
+ $lei->fail(<<EOM) if -e $dir_old && !rename($dir_old, $dir_new);
+E: rename($dir_old, $dir_new) error: $!
+EOM
+}
+
no warnings 'once';
*nntp_url = \&cloneurl;
*base_url = \&PublicInbox::Inbox::base_url;
diff --git a/script/lei b/script/lei
index 90a93839..bec6b001 100755
--- a/script/lei
+++ b/script/lei
@@ -33,8 +33,15 @@ my $exec_cmd = sub {
push @rdr, shift(@old), $newfh;
}
my $do_exec = sub {
+ my @non_std; # ex. $op_p from lei_edit_search
while (my ($io, $newfh) = splice(@rdr, 0, 2)) {
+ my $old_io = !!$io;
open $io, '+<&', $newfh or die "open +<&=: $!";
+ push @non_std, $io unless $old_io;
+ }
+ if (@non_std) {
+ require Fcntl;
+ fcntl($_, Fcntl::F_SETFD(), 0) for @non_std;
}
my %env = map { split(/=/, $_, 2) } splice(@argv, $argc);
@ENV{keys %env} = values %env;
^ permalink raw reply related [relevance 75%]
* [PATCH 3/5] lei_saved_search: fix excess indent for first lei.q entry
2021-05-01 6:21 70% [PATCH 0/5] lei: more UI/UX tweaks Eric Wong
2021-05-01 6:21 61% ` [PATCH 1/5] xt/lei-onion-convert: test for NNTP+IMAP onions Eric Wong
2021-05-01 6:21 55% ` [PATCH 2/5] lei <q|up>: distinguish between mset and l2m counts Eric Wong
@ 2021-05-01 6:21 71% ` Eric Wong
2021-05-01 6:21 56% ` [PATCH 4/5] lei: rename ls-sync to ls-mail-sync Eric Wong
2021-05-01 6:21 75% ` [PATCH 5/5] lei edit-search: support relocating lei.q.output Eric Wong
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-01 6:21 UTC (permalink / raw)
To: meta
This was harmless, but ugly and possibly confusing to
users who run "lei edit-search".
---
lib/PublicInbox/LeiSavedSearch.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 682a43e8..79125214 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -137,7 +137,7 @@ sub new { # new saved search "lei q --save"
print $fh <<EOM;
; to refresh with new results, run: lei up $sq_dst
[lei]
- $q
+$q
[lei "q"]
output = $dst
EOM
^ permalink raw reply related [relevance 71%]
* [PATCH 0/5] lei: more UI/UX tweaks
@ 2021-05-01 6:21 70% Eric Wong
2021-05-01 6:21 61% ` [PATCH 1/5] xt/lei-onion-convert: test for NNTP+IMAP onions Eric Wong
` (4 more replies)
0 siblings, 5 replies; 200+ results
From: Eric Wong @ 2021-05-01 6:21 UTC (permalink / raw)
To: meta
4/5 is an incompatible change of an incomplete feature
2/5 is something I noticed from using "lei up --all=local"
Editing searches is less surprising, as lei.q.output updates
and auto-conversion are now supported (since there's no
"lei mv-search").
Eric Wong (5):
xt/lei-onion-convert: test for NNTP+IMAP onions
lei <q|up>: distinguish between mset and l2m counts
lei_saved_search: fix excess indent for first lei.q entry
lei: rename ls-sync to ls-mail-sync
lei edit-search: support relocating lei.q.output
MANIFEST | 3 +-
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 5 +-
lib/PublicInbox/LeiEditSearch.pm | 15 ++++-
.../{LeiLsSync.pm => LeiLsMailSync.pm} | 4 +-
lib/PublicInbox/LeiSavedSearch.pm | 53 +++++++++++++++-
lib/PublicInbox/LeiToMail.pm | 36 ++++++-----
lib/PublicInbox/LeiXSearch.pm | 18 ++++--
script/lei | 7 +++
t/lei-import-imap.t | 4 +-
t/lei-import-maildir.t | 4 +-
xt/lei-onion-convert.t | 61 +++++++++++++++++++
12 files changed, 180 insertions(+), 32 deletions(-)
rename lib/PublicInbox/{LeiLsSync.pm => LeiLsMailSync.pm} (93%)
create mode 100644 xt/lei-onion-convert.t
^ permalink raw reply [relevance 70%]
* [PATCH 1/5] xt/lei-onion-convert: test for NNTP+IMAP onions
2021-05-01 6:21 70% [PATCH 0/5] lei: more UI/UX tweaks Eric Wong
@ 2021-05-01 6:21 61% ` Eric Wong
2021-05-01 6:21 55% ` [PATCH 2/5] lei <q|up>: distinguish between mset and l2m counts Eric Wong
` (3 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-01 6:21 UTC (permalink / raw)
To: meta
These tests require a running Tor instance (defaulting to
127.0.0.1:9050) and Internet connectivity, but otherwise
work pretty well.
---
MANIFEST | 1 +
xt/lei-onion-convert.t | 61 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 62 insertions(+)
create mode 100644 xt/lei-onion-convert.t
diff --git a/MANIFEST b/MANIFEST
index bc2ad671..82f25735 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -516,6 +516,7 @@ xt/httpd-async-stream.t
xt/imapd-mbsync-oimap.t
xt/imapd-validate.t
xt/lei-auth-fail.t
+xt/lei-onion-convert.t
xt/mem-imapd-tls.t
xt/mem-msgview.t
xt/msgtime_cmp.t
diff --git a/xt/lei-onion-convert.t b/xt/lei-onion-convert.t
new file mode 100644
index 00000000..d38b4b16
--- /dev/null
+++ b/xt/lei-onion-convert.t
@@ -0,0 +1,61 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10; use PublicInbox::TestCommon;
+use PublicInbox::MboxReader;
+my $test_tor = $ENV{TEST_TOR};
+plan skip_all => "TEST_TOR unset" unless $test_tor;
+unless ($test_tor =~ m!\Asocks5h://!i) {
+ my $default = 'socks5h://127.0.0.1:9050';
+ diag "using $default (set TEST_TOR=socks5h://ADDR:PORT to override)";
+ $test_tor = $default;
+}
+my $onion = $ENV{TEST_ONION_HOST} //'ou63pmih66umazou.onion';
+my $ng = 'inbox.comp.mail.public-inbox.meta';
+my $nntp_url = $ENV{TEST_NNTP_ONION_URL} // "nntp://$onion/$ng";
+my $imap_url = $ENV{TEST_IMAP_ONION_URL} // "imap://$onion/$ng.0";
+my @cnv = qw(lei convert -o mboxrd:/dev/stdout);
+my @proxy_cli = ("--proxy=$test_tor");
+my $proxy_cfg = "proxy=$test_tor";
+test_lei(sub {
+ my $run = {};
+ for my $args ([$nntp_url, @proxy_cli], [$imap_url, @proxy_cli],
+ [ $nntp_url, '-c', "nntp.$proxy_cfg" ],
+ [ $imap_url, '-c', "imap.$proxy_cfg" ]) {
+ pipe(my ($r, $w)) or xbail "pipe: $!";
+ my $cmd = [@cnv, @$args];
+ my $td = start_script($cmd, undef, { 1 => $w, run_mode => 0 });
+ $args->[0] =~ s!\A(.+?://).*!$1...!;
+ my $key = "@$args";
+ ok($td, "$key running");
+ $run->{$key} = { td => $td, r => $r };
+ }
+ while (my ($key, $x) = each %$run) {
+ my ($td, $r) = delete(@$x{qw(td r)});
+ eval {
+ PublicInbox::MboxReader->mboxrd($r, sub {
+ my ($eml) = @_;
+ if ($key =~ m!\Anntps?://!i) {
+ for (qw(Xref Newsgroups Path)) {
+ $eml->header_set($_);
+ }
+ }
+ push @{$x->{eml}}, $eml;
+ close $r;
+ $td->kill('-INT');
+ die "$key done\n";
+ });
+ };
+ chomp(my $done = $@);
+ like($done, qr/\Q$key\E done/, $done);
+ $td->join;
+ }
+ my @keys = keys %$run;
+ my $first_key = shift @keys;
+ for my $key (@keys) {
+ is_deeply($run->{$key}, $run->{$first_key},
+ "`$key' matches `$first_key'");
+ }
+});
+
+done_testing;
^ permalink raw reply related [relevance 61%]
* [PATCH 4/5] lei: rename ls-sync to ls-mail-sync
2021-05-01 6:21 70% [PATCH 0/5] lei: more UI/UX tweaks Eric Wong
` (2 preceding siblings ...)
2021-05-01 6:21 71% ` [PATCH 3/5] lei_saved_search: fix excess indent for first lei.q entry Eric Wong
@ 2021-05-01 6:21 56% ` Eric Wong
2021-05-01 19:29 59% ` [PATCH 6/5] lei import: fix --mail-sync handling in LeiInput Eric Wong
2021-05-01 6:21 75% ` [PATCH 5/5] lei edit-search: support relocating lei.q.output Eric Wong
4 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-05-01 6:21 UTC (permalink / raw)
To: meta
This allows tab-completion for "ls-search" to work with fewer
characters ("ls-s<TAB>" instead of "ls-se<TAB>"), and I expect
"ls-search" to be used more frequently than "ls-mail-sync".
This also matches the --mail-sync switch of "lei import"
---
MANIFEST | 2 +-
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/{LeiLsSync.pm => LeiLsMailSync.pm} | 4 ++--
t/lei-import-imap.t | 4 ++--
t/lei-import-maildir.t | 4 ++--
5 files changed, 8 insertions(+), 8 deletions(-)
rename lib/PublicInbox/{LeiLsSync.pm => LeiLsMailSync.pm} (93%)
diff --git a/MANIFEST b/MANIFEST
index 82f25735..b7e55793 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -202,8 +202,8 @@ lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiInspect.pm
lib/PublicInbox/LeiLcat.pm
lib/PublicInbox/LeiLsLabel.pm
+lib/PublicInbox/LeiLsMailSync.pm
lib/PublicInbox/LeiLsSearch.pm
-lib/PublicInbox/LeiLsSync.pm
lib/PublicInbox/LeiMailSync.pm
lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index bb67fc0b..5d701d5e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -170,7 +170,7 @@ our %CMD = ( # sorted in order of importance/use:
'ls-external' => [ '[FILTER]', 'list publicinbox|extindex locations',
qw(format|f=s z|0 globoff|g invert-match|v local remote), @c_opt ],
'ls-label' => [ '', 'list labels', qw(z|0 stats:s), @c_opt ],
-'ls-sync' => [ '', 'list sync folders',
+'ls-mail-sync' => [ '', 'list mail sync folders',
qw(z|0 z|0 globoff|g invert-match|v local remote), @c_opt ],
'forget-external' => [ 'LOCATION...|--prune',
'exclude further results from a publicinbox|extindex',
diff --git a/lib/PublicInbox/LeiLsSync.pm b/lib/PublicInbox/LeiLsMailSync.pm
similarity index 93%
rename from lib/PublicInbox/LeiLsSync.pm
rename to lib/PublicInbox/LeiLsMailSync.pm
index 71f111a9..2b3d326d 100644
--- a/lib/PublicInbox/LeiLsSync.pm
+++ b/lib/PublicInbox/LeiLsMailSync.pm
@@ -2,12 +2,12 @@
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# front-end for the "lei ls-sync" sub-command
-package PublicInbox::LeiLsSync;
+package PublicInbox::LeiLsMailSync;
use strict;
use v5.10.1;
use PublicInbox::LeiMailSync;
-sub lei_ls_sync {
+sub lei_ls_mail_sync {
my ($lei, $filter) = @_;
my $sto = $lei->_lei_store or return;
my $lms = $sto->search->lms or return;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index c977c68e..3a1fff4c 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -22,8 +22,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
is_deeply(json_utf8->decode($lei_out), {}, 'no inspect stats, yet');
lei_ok('import', $url);
- lei_ok 'ls-sync';
- like($lei_out, qr!\A\Q$url\E;UIDVALIDITY=\d+\n\z!, 'ls-sync');
+ lei_ok 'ls-mail-sync';
+ like($lei_out, qr!\A\Q$url\E;UIDVALIDITY=\d+\n\z!, 'ls-mail-sync');
chomp(my $u = $lei_out);
lei_ok('import', $u, \'UIDVALIDITY match in URL');
$u =~ s/;UIDVALIDITY=(\d+)\s*/;UIDVALIDITY=9$1/s;
diff --git a/t/lei-import-maildir.t b/t/lei-import-maildir.t
index 808e1a73..02fe43e1 100644
--- a/t/lei-import-maildir.t
+++ b/t/lei-import-maildir.t
@@ -40,8 +40,8 @@ test_lei(sub {
is_deeply($inspect, { sync => { "maildir:$md" => [ 'x:2,S' ] } },
'maildir sync info as expected');
- lei_ok qw(ls-sync);
- is($lei_out, "maildir:$md\n", 'ls-sync as expected');
+ lei_ok qw(ls-mail-sync);
+ is($lei_out, "maildir:$md\n", 'ls-mail-sync as expected');
lei_ok(qw(import), $md, \'import Maildir again');
$imp_err = $lei_err;
^ permalink raw reply related [relevance 56%]
* [PATCH 2/5] lei <q|up>: distinguish between mset and l2m counts
2021-05-01 6:21 70% [PATCH 0/5] lei: more UI/UX tweaks Eric Wong
2021-05-01 6:21 61% ` [PATCH 1/5] xt/lei-onion-convert: test for NNTP+IMAP onions Eric Wong
@ 2021-05-01 6:21 55% ` Eric Wong
2021-05-01 6:21 71% ` [PATCH 3/5] lei_saved_search: fix excess indent for first lei.q entry Eric Wong
` (2 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-01 6:21 UTC (permalink / raw)
To: meta
The number of messages we write to --output is usually different
than the mset count due to deduplication from combining multiple
sources.
This change makes the stderr output of "lei up --all=local" way
more useful IMHO.
---
lib/PublicInbox/LeiToMail.pm | 36 ++++++++++++++++++++---------------
lib/PublicInbox/LeiXSearch.pm | 18 ++++++++++++++----
2 files changed, 35 insertions(+), 19 deletions(-)
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index eda4701c..a546ab42 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -196,9 +196,13 @@ sub _mbox_write_cb ($$) {
return if $dedupe->is_dup($eml, $smsg);
$lse->xsmsg_vmd($smsg) if $lse;
$buf = $eml2mbox->($eml, $smsg);
- return atomic_append($lei, $buf) if $atomic_append;
- my $lk = $ovv->lock_for_scope;
- $lei->out($$buf);
+ if ($atomic_append) {
+ atomic_append($lei, $buf);
+ } else {
+ my $lk = $ovv->lock_for_scope;
+ $lei->out($$buf);
+ }
+ ++$lei->{-nr_write};
}
}
@@ -273,15 +277,13 @@ sub _maildir_write_cb ($$) {
my $dst = $lei->{ovv}->{dst};
my $lse = $lei->{lse}; # may be undef
sub { # for git_to_mail
- my ($buf, $smsg, $eml) = @_;
+ my ($bref, $smsg, $eml) = @_;
$dst // return $lei->fail; # dst may be undef-ed in last run
- $buf //= \($eml->as_string);
+ return if $dedupe && $dedupe->is_dup($eml //
+ PublicInbox::Eml->new($$bref));
$lse->xsmsg_vmd($smsg) if $lse;
- return _buf2maildir($dst, $buf, $smsg) if !$dedupe;
- $eml //= PublicInbox::Eml->new($$buf); # copy buf
- return if $dedupe->is_dup($eml, $smsg);
- undef $eml;
- _buf2maildir($dst, $buf, $smsg);
+ _buf2maildir($dst, $bref // \($eml->as_string), $smsg);
+ ++$lei->{-nr_write};
}
}
@@ -296,16 +298,15 @@ sub _imap_write_cb ($$) {
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$mic // return $lei->fail; # mic may be undef-ed in last run
- if ($dedupe) {
- $eml //= PublicInbox::Eml->new($$bref); # copy bref
- return if $dedupe->is_dup($eml, $smsg);
- }
+ return if $dedupe && $dedupe->is_dup($eml //
+ PublicInbox::Eml->new($$bref));
$lse->xsmsg_vmd($smsg) if $lse;
eval { $imap_append->($mic, $folder, $bref, $smsg, $eml) };
if (my $err = $@) {
undef $mic;
die $err;
}
+ ++$lei->{-nr_write};
}
}
@@ -659,7 +660,12 @@ sub write_mail { # via ->wq_io_do
sub wq_atexit_child {
my ($self) = @_;
delete $self->{wcb};
- $self->{lei}->{ale}->git->async_wait_all;
+ my $lei = $self->{lei};
+ $lei->{ale}->git->async_wait_all;
+ my $nr = delete($lei->{-nr_write}) or return;
+ return if $lei->{early_mua} || !$lei->{-progress} || !$lei->{pkt_op_p};
+ require PublicInbox::PktOp;
+ PublicInbox::PktOp::pkt_do($lei->{pkt_op_p}, 'l2m_progress', $nr);
}
# called in top-level lei-daemon when LeiAuth is done
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index b3fd79d0..e04af0dc 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -144,6 +144,11 @@ sub mset_progress {
}
}
+sub l2m_progress {
+ my ($lei, $nr) = @_;
+ $lei->{-nr_write} += $nr;
+}
+
sub query_one_mset { # for --threads and l2m w/o sort
my ($self, $ibxish) = @_;
local $0 = "$0 query_one_mset";
@@ -354,7 +359,7 @@ sub query_done { # EOF callback for main daemon
}
my $wait = $lei->{sto} ? $lei->{sto}->ipc_do('done') : undef;
$lei->{ovv}->ovv_end($lei);
- my @out;
+ my (@out, $start_mua);
if ($l2m) { # close() calls LeiToMail reap_compress
@out = (" in $lei->{ovv}->{dst}");
if (my $out = delete $lei->{old_1}) {
@@ -370,11 +375,15 @@ Error closing $lei->{ovv}->{dst}: $!
$lei->poke_mua;
} else { # mbox users
delete $l2m->{mbl}; # drop dotlock
- $lei->start_mua;
+ $start_mua = 1;
}
}
- $lei->{-progress} and
- $lei->err('# ', $lei->{-mset_total} // 0, " matches", @out);
+ if ($lei->{-progress}) {
+ $lei->qerr('# ', $lei->{-mset_total} // 0, " matches", @out);
+ my $nr = $lei->{-nr_write} // 0;
+ $lei->qerr("# $nr written to $lei->{ovv}->{dst}") if $l2m;
+ }
+ $lei->start_mua if $start_mua;
$lei->dclose;
}
@@ -456,6 +465,7 @@ sub do_query {
'+' => [ \&incr_post_augment, $lei ],
'' => [ \&query_done, $lei ],
'mset_progress' => [ \&mset_progress, $lei ],
+ 'l2m_progress' => [ \&l2m_progress, $lei ],
'x_it' => [ $lei->can('x_it'), $lei ],
'child_error' => [ $lei->can('child_error'), $lei ],
'incr_start_query' => [ \&incr_start_query, $self, $l2m ],
^ permalink raw reply related [relevance 55%]
* [PATCH 6/5] lei import: fix --mail-sync handling in LeiInput
2021-05-01 6:21 56% ` [PATCH 4/5] lei: rename ls-sync to ls-mail-sync Eric Wong
@ 2021-05-01 19:29 59% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-01 19:29 UTC (permalink / raw)
To: meta
"lei inspect" also shows "mail-sync" as a field name
---
lib/PublicInbox/LeiInput.pm | 8 ++++----
lib/PublicInbox/LeiInspect.pm | 2 +-
t/lei-import-imap.t | 4 ++--
t/lei-import-maildir.t | 2 +-
4 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 86f300c3..9bcc86e1 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -204,7 +204,7 @@ sub prepare_http_input ($$$) {
sub prepare_inputs { # returns undef on error
my ($self, $lei, $inputs) = @_;
my $in_fmt = $lei->{opt}->{'in-format'};
- my $sync = $lei->{opt}->{sync} ? {} : undef; # using LeiMailSync
+ my $sync = $lei->{opt}->{'mail-sync'} ? {} : undef; # using LeiMailSync
if ($lei->{opt}->{stdin}) {
@$inputs and return
$lei->fail("--stdin and @$inputs do not mix");
@@ -286,11 +286,11 @@ $input is `eml', not --in-format=$in_fmt
}
if ($sync && $sync->{no}) {
return $lei->fail(<<"") if !$sync->{ok};
---sync specified but no inputs support it
+--mail-sync specified but no inputs support it
# non-fatal if some inputs support support sync
- $lei->err("# --sync will only be used for @{$sync->{ok}}");
- $lei->err("# --sync is not supported for: @{$sync->{no}}");
+ $lei->err("# --mail-sync will only be used for @{$sync->{ok}}");
+ $lei->err("# --mail-sync is not supported for: @{$sync->{no}}");
}
if ($net) {
$net->{-can_die} = 1;
diff --git a/lib/PublicInbox/LeiInspect.pm b/lib/PublicInbox/LeiInspect.pm
index 6cfc8083..714d2526 100644
--- a/lib/PublicInbox/LeiInspect.pm
+++ b/lib/PublicInbox/LeiInspect.pm
@@ -18,7 +18,7 @@ sub inspect_blob ($$) {
$ent->{'lei/store'} = \@docids if @docids;
my $lms = $lse->lms;
if (my $loc = $lms ? $lms->locations_for($oidhex) : undef) {
- $ent->{sync} = $loc;
+ $ent->{'mail-sync'} = $loc;
}
}
$ent;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 3a1fff4c..fd15ef4f 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -54,8 +54,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok('inspect', "blob:$out->[5]->{blob}");
my $x = json_utf8->decode($lei_out);
is(ref($x->{'lei/store'}), 'ARRAY', 'lei/store in inspect');
- is(ref($x->{sync}), 'HASH', 'sync in inspect');
- is(ref($x->{sync}->{$k[0]}), 'ARRAY', 'UID arrays in inspect');
+ is(ref($x->{'mail-sync'}), 'HASH', 'sync in inspect');
+ is(ref($x->{'mail-sync'}->{$k[0]}), 'ARRAY', 'UID arrays in inspect');
my $psgi_attach = 'cfa3622cbeffc9bd6b0fc66c4d60d420ba74f60d';
lei_ok('blob', $psgi_attach);
diff --git a/t/lei-import-maildir.t b/t/lei-import-maildir.t
index 02fe43e1..b85d3026 100644
--- a/t/lei-import-maildir.t
+++ b/t/lei-import-maildir.t
@@ -37,7 +37,7 @@ test_lei(sub {
lei_ok('inspect', "blob:$res->[0]->{blob}");
$inspect = json_utf8->decode($lei_out);
is(ref(delete $inspect->{"lei/store"}), 'ARRAY', 'lei/store IDs');
- is_deeply($inspect, { sync => { "maildir:$md" => [ 'x:2,S' ] } },
+ is_deeply($inspect, { 'mail-sync' => { "maildir:$md" => [ 'x:2,S' ] } },
'maildir sync info as expected');
lei_ok qw(ls-mail-sync);
^ permalink raw reply related [relevance 59%]
* [PATCH 1/6] lei <q|up>: combine written/results into one line
2021-05-02 6:05 70% [PATCH 0/6] lei: more steps towards kw sync Eric Wong
@ 2021-05-02 6:05 70% ` Eric Wong
2021-05-02 6:05 66% ` [PATCH 2/6] lei_input: common net_merge_all_done for lei <import|tag> Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-02 6:05 UTC (permalink / raw)
To: meta
Having multiple lines of output mean they can be interleaved in
daemon mode. Put stats into one line to reduce screen
real-estate size and improve readability.
---
lib/PublicInbox/LeiXSearch.pm | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index d212a732..21b15025 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -359,9 +359,8 @@ sub query_done { # EOF callback for main daemon
}
my $wait = $lei->{sto} ? $lei->{sto}->ipc_do('done') : undef;
$lei->{ovv}->ovv_end($lei);
- my (@out, $start_mua);
+ my $start_mua;
if ($l2m) { # close() calls LeiToMail reap_compress
- @out = (" in $lei->{ovv}->{dst}");
if (my $out = delete $lei->{old_1}) {
if (my $mbout = $lei->{1}) {
close($mbout) or return $lei->fail(<<"");
@@ -379,9 +378,11 @@ Error closing $lei->{ovv}->{dst}: $!
}
}
if ($lei->{-progress}) {
- $lei->qerr('# ', $lei->{-mset_total} // 0, " matches", @out);
+ my $tot = $lei->{-mset_total} // 0;
my $nr = $lei->{-nr_write} // 0;
- $lei->qerr("# $nr written to $lei->{ovv}->{dst}") if $l2m;
+ $lei->qerr($l2m ?
+ "# $nr written to $lei->{ovv}->{dst} ($tot matches)" :
+ "# $tot matches");
}
$lei->start_mua if $start_mua;
$lei->dclose;
^ permalink raw reply related [relevance 70%]
* [PATCH 0/6] lei: more steps towards kw sync
@ 2021-05-02 6:05 70% Eric Wong
2021-05-02 6:05 70% ` [PATCH 1/6] lei <q|up>: combine written/results into one line Eric Wong
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Eric Wong @ 2021-05-02 6:05 UTC (permalink / raw)
To: meta
6/6 is a fairly significant change, but I think it's
worth doing by default to make things less surprising
for eventual users of "lei export-kw" (I'm not sure if
that's even a good name).
Eric Wong (6):
lei <q|up>: combine written/results into one line
lei_input: common net_merge_all_done for lei <import|tag>
lei_input: reject --mail-sync if using HTTP(S) for now
lei: simplify workers_start API
net_writer: use "FLAGS.SILENT" to set keywords
lei <q|up>: writes to Maildirs and IMAP use mail-sync
lib/PublicInbox/LEI.pm | 5 +++--
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 9 ++-------
lib/PublicInbox/LeiInput.pm | 9 +++++++++
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 7 +++----
lib/PublicInbox/LeiStore.pm | 8 ++++----
lib/PublicInbox/LeiTag.pm | 9 ++-------
lib/PublicInbox/LeiToMail.pm | 24 +++++++++++++++++-------
lib/PublicInbox/LeiXSearch.pm | 9 +++++----
lib/PublicInbox/NetWriter.pm | 14 ++++++--------
t/lei-import-http.t | 5 +++++
xt/net_writer-imap.t | 9 +++++++--
16 files changed, 68 insertions(+), 50 deletions(-)
^ permalink raw reply [relevance 70%]
* [PATCH 2/6] lei_input: common net_merge_all_done for lei <import|tag>
2021-05-02 6:05 70% [PATCH 0/6] lei: more steps towards kw sync Eric Wong
2021-05-02 6:05 70% ` [PATCH 1/6] lei <q|up>: combine written/results into one line Eric Wong
@ 2021-05-02 6:05 66% ` Eric Wong
2021-05-02 6:05 53% ` [PATCH 4/6] lei: simplify workers_start API Eric Wong
2021-05-02 6:05 49% ` [PATCH 6/6] lei <q|up>: writes to Maildirs and IMAP use mail-sync Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-02 6:05 UTC (permalink / raw)
To: meta
I suspect there'll be more lei_input-only things in the future.
---
lib/PublicInbox/LeiImport.pm | 7 +------
lib/PublicInbox/LeiInput.pm | 7 +++++++
lib/PublicInbox/LeiTag.pm | 7 +------
3 files changed, 9 insertions(+), 12 deletions(-)
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 575cf125..394138b4 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -48,12 +48,6 @@ sub input_net_cb { # imap_each / nntp_each
input_eml_cb($self, $eml, $vmd);
}
-sub net_merge_all_done { # callback used by LeiAuth
- my ($self) = @_;
- $self->wq_io_do('process_inputs');
- $self->wq_close(1);
-}
-
sub lei_import { # the main "lei import" method
my ($lei, @inputs) = @_;
my $sto = $lei->_lei_store(1);
@@ -99,6 +93,7 @@ sub _complete_import {
no warnings 'once';
*ipc_atfork_child = \&PublicInbox::LeiInput::input_only_atfork_child;
+*net_merge_all_done = \&PublicInbox::LeiInput::input_only_net_merge_all_done;
# the following works even when LeiAuth is lazy-loaded
*net_merge_all = \&PublicInbox::LeiAuth::net_merge_all;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 9bcc86e1..917f682b 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -329,6 +329,13 @@ sub input_only_atfork_child {
undef;
}
+# alias this as "net_merge_all_done" to use as an LeiAuth callback
+sub input_only_net_merge_all_done {
+ my ($self) = @_;
+ $self->wq_io_do('process_inputs');
+ $self->wq_close(1);
+}
+
# like Getopt::Long, but for +kw:FOO and -kw:FOO to prepare
# for update_xvmd -> update_vmd
sub vmd_mod_extract {
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 2170e3f2..6025c93e 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -19,12 +19,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
sub input_mbox_cb { input_eml_cb($_[1], $_[0]) }
-sub net_merge_all_done { # callback used by LeiAuth
- my ($self) = @_;
- $self->wq_io_do('process_inputs');
- $self->wq_close(1);
-}
-
sub input_maildir_cb { # maildir_each_eml cb
my ($f, $kw, $eml, $self) = @_;
input_eml_cb($self, $eml);
@@ -117,5 +111,6 @@ sub _complete_tag {
no warnings 'once'; # the following works even when LeiAuth is lazy-loaded
*net_merge_all = \&PublicInbox::LeiAuth::net_merge_all;
+*net_merge_all_done = \&PublicInbox::LeiInput::input_only_net_merge_all_done;
1;
^ permalink raw reply related [relevance 66%]
* [PATCH 4/6] lei: simplify workers_start API
2021-05-02 6:05 70% [PATCH 0/6] lei: more steps towards kw sync Eric Wong
2021-05-02 6:05 70% ` [PATCH 1/6] lei <q|up>: combine written/results into one line Eric Wong
2021-05-02 6:05 66% ` [PATCH 2/6] lei_input: common net_merge_all_done for lei <import|tag> Eric Wong
@ 2021-05-02 6:05 53% ` Eric Wong
2021-05-02 6:05 49% ` [PATCH 6/6] lei <q|up>: writes to Maildirs and IMAP use mail-sync Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-02 6:05 UTC (permalink / raw)
To: meta
In most cases, we just name the worker process based
on the command. The only change is for LeiMirror
vs "lei add-external --mirror", but I doubt it matters.
---
lib/PublicInbox/LEI.pm | 3 ++-
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 2 +-
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 2 +-
8 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 5d701d5e..d5c6bd52 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -542,7 +542,7 @@ sub pkt_op_pair {
}
sub workers_start {
- my ($lei, $wq, $ident, $jobs, $ops) = @_;
+ my ($lei, $wq, $jobs, $ops) = @_;
$ops = {
'!' => [ \&fail_handler, $lei ],
'|' => [ \&sigpipe_handler, $lei ],
@@ -552,6 +552,7 @@ sub workers_start {
};
$ops->{''} //= [ $wq->can('_lei_wq_eof') || \&wq_eof, $lei ];
my $end = $lei->pkt_op_pair;
+ my $ident = $wq->{-wq_ident} // "lei-$lei->{cmd} worker";
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
delete $lei->{pkt_op_p};
my $op_c = delete $lei->{pkt_op_c};
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 0a957358..710430a2 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -151,7 +151,7 @@ EOM
}
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
- my ($op_c, $ops) = $lei->workers_start($self, 'lei-blob', 1);
+ my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_solve_blob', []);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 5b27ec2d..395a80f8 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -55,7 +55,7 @@ sub lei_convert { # the main "lei convert" method
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
- my ($op_c, $ops) = $lei->workers_start($self, 'lei-convert', 1);
+ my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('process_inputs', []);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 394138b4..6a57df47 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -76,7 +76,7 @@ sub lei_import { # the main "lei import" method
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{-wq_nr_workers} = $j // 1; # locked
$lei->{-eml_noisy} = 1;
- (my $op_c, $ops) = $lei->workers_start($self, 'lei-import', $j, $ops);
+ (my $op_c, $ops) = $lei->workers_start($self, $j, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
net_merge_all_done($self) unless $lei->{auth};
diff --git a/lib/PublicInbox/LeiLsSearch.pm b/lib/PublicInbox/LeiLsSearch.pm
index a00e78fc..6cea6ae8 100644
--- a/lib/PublicInbox/LeiLsSearch.pm
+++ b/lib/PublicInbox/LeiLsSearch.pm
@@ -72,7 +72,7 @@ sub do_ls_search_long {
sub bg_worker ($$$) {
my ($lei, $pfx, $json) = @_;
my $self = bless { -wq_nr_workers => 1, json => $json }, __PACKAGE__;
- my ($op_c, $ops) = $lei->workers_start($self, 'ls-search', 1);
+ my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_ls_search_long', [], $pfx);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index db97b98c..a37e1d5c 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -282,7 +282,7 @@ sub start {
require PublicInbox::Inbox;
require PublicInbox::Admin;
require PublicInbox::InboxWritable;
- my ($op, $ops) = $lei->workers_start($self, 'lei_mirror', 1);
+ my ($op, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_mirror', []);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index b4893489..f381a31c 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -188,7 +188,7 @@ sub lei_p2q { # the "lei patch-to-query" entry point
} else {
$self->{input} = $input;
}
- my ($op, $ops) = $lei->workers_start($self, 'lei-p2q', 1);
+ my ($op, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('do_p2q', []);
$self->wq_close(1);
diff --git a/lib/PublicInbox/LeiTag.pm b/lib/PublicInbox/LeiTag.pm
index 6025c93e..c650e886 100644
--- a/lib/PublicInbox/LeiTag.pm
+++ b/lib/PublicInbox/LeiTag.pm
@@ -44,7 +44,7 @@ sub lei_tag { # the "lei tag" method
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
my $j = $self->{-wq_nr_workers} = 1; # locked for now
- (my $op_c, $ops) = $lei->workers_start($self, 'lei-tag', $j, $ops);
+ (my $op_c, $ops) = $lei->workers_start($self, $j, $ops);
$lei->{wq1} = $self;
$lei->{-err_type} = 'non-fatal';
net_merge_all_done($self) unless $lei->{auth};
^ permalink raw reply related [relevance 53%]
* [PATCH 6/6] lei <q|up>: writes to Maildirs and IMAP use mail-sync
2021-05-02 6:05 70% [PATCH 0/6] lei: more steps towards kw sync Eric Wong
` (2 preceding siblings ...)
2021-05-02 6:05 53% ` [PATCH 4/6] lei: simplify workers_start API Eric Wong
@ 2021-05-02 6:05 49% ` Eric Wong
3 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-02 6:05 UTC (permalink / raw)
To: meta
This will allow keyword updates from other folders to propagate
to folders where search results may be duplicated.
---
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiQuery.pm | 7 +++----
lib/PublicInbox/LeiStore.pm | 8 ++++----
lib/PublicInbox/LeiToMail.pm | 24 +++++++++++++++++-------
4 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index d5c6bd52..599cfab2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -145,7 +145,7 @@ our %CMD = ( # sorted in order of importance/use:
qw(save output|mfolder|o=s format|f=s dedupe|d=s threads|t+
sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+
- color!), @c_opt, opt_dash('limit|n=i', '[0-9]+') ],
+ color! mail-sync!), @c_opt, opt_dash('limit|n=i', '[0-9]+') ],
'up' => [ 'OUTPUT|--all', 'update saved search',
qw(jobs|j=s lock=s@ alert=s@ mua=s verbose|v+ all:s), @c_opt ],
diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index efe328cc..1999a534 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -29,10 +29,9 @@ sub _start_query { # used by "lei q" and "lei up"
return $self->fail("`$mj' writer jobs must be >= 1");
}
my $l2m = $self->{l2m};
- if ($l2m && ($opt->{'import-remote'} //= 1) |
- # we use \1 (a ref) to distinguish between
- # user-supplied and default value
- (($opt->{'import-before'} //= \1) ? 1 : 0)) {
+ # we use \1 (a ref) to distinguish between default vs. user-supplied
+ if ($l2m && grep { $opt->{$_} //= \1 } (qw(mail-sync import-remote
+ import-before))) {
$self->_lei_store(1)->write_prepare($self);
}
$l2m and $l2m->{-wq_nr_workers} = $mj // do {
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 8af740fd..29362b2e 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -193,15 +193,15 @@ sub remove_eml_vmd {
\@docids;
}
-sub set_sync_info ($$$) {
- my ($self, $oidhex, $sync_info) = @_;
+sub set_sync_info {
+ my ($self, $oidhex, $folder, $id) = @_;
($self->{lms} //= do {
require PublicInbox::LeiMailSync;
my $f = "$self->{priv_eidx}->{topdir}/mail_sync.sqlite3";
my $lms = PublicInbox::LeiMailSync->new($f);
$lms->lms_begin;
$lms;
- })->set_src($oidhex, @$sync_info);
+ })->set_src($oidhex, $folder, $id);
}
sub add_eml {
@@ -212,7 +212,7 @@ sub add_eml {
my $smsg = bless { -oidx => $oidx }, 'PublicInbox::Smsg';
my $im_mark = $im->add($eml, undef, $smsg);
if ($vmd && $vmd->{sync_info}) {
- set_sync_info($self, $smsg->{blob}, $vmd->{sync_info});
+ set_sync_info($self, $smsg->{blob}, @{$vmd->{sync_info}});
}
$im_mark or return; # duplicate blob returns undef
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index ab4de378..71acf952 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -242,7 +242,7 @@ sub _buf2maildir {
my $kw = $smsg->{kw} // [];
my $sfx = join('', sort(map { $kw2char{$_} // () } @$kw));
my $rand = ''; # chosen by die roll :P
- my ($tmp, $fh, $final, $ok);
+ my ($tmp, $fh, $base, $ok);
my $common = $smsg->{blob} // _rand;
if (defined(my $pct = $smsg->{pct})) { $common .= "=$pct" }
do {
@@ -257,11 +257,12 @@ sub _buf2maildir {
$dst .= 'cur/';
$rand = '';
do {
- $final = $dst.$rand.$common.':2,'.$sfx;
- } while (!($ok = link($tmp, $final)) && $!{EEXIST} &&
+ $base = $rand.$common.':2,'.$sfx
+ } while (!($ok = link($tmp, $dst.$base)) && $!{EEXIST} &&
($rand = _rand.','));
- die "link($tmp, $final): $!" unless $ok;
+ die "link($tmp, $dst$base): $!" unless $ok;
unlink($tmp) or warn "W: failed to unlink $tmp: $!\n";
+ \$base;
} else {
my $err = "Error writing $smsg->{blob} to $dst: $!\n";
$_[0] = undef; # clobber dst
@@ -276,13 +277,16 @@ sub _maildir_write_cb ($$) {
$dedupe->prepare_dedupe if $dedupe;
my $dst = $lei->{ovv}->{dst};
my $lse = $lei->{lse}; # may be undef
+ my $sto = $lei->{opt}->{'mail-sync'} ? $lei->{sto} : undef;
+ my $out = $sto ? 'maildir:'.$lei->rel2abs($dst) : undef;
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$dst // return $lei->fail; # dst may be undef-ed in last run
return if $dedupe && $dedupe->is_dup($eml //
PublicInbox::Eml->new($$bref));
$lse->xsmsg_vmd($smsg) if $lse;
- _buf2maildir($dst, $bref // \($eml->as_string), $smsg);
+ my $n = _buf2maildir($dst, $bref // \($eml->as_string), $smsg);
+ $sto->ipc_do('set_sync_info', $smsg->{blob}, $out, $n) if $sto;
++$lei->{-nr_write};
}
}
@@ -291,21 +295,27 @@ sub _imap_write_cb ($$) {
my ($self, $lei) = @_;
my $dedupe = $lei->{dedupe};
$dedupe->prepare_dedupe if $dedupe;
- my $imap_append = $lei->{net}->can('imap_append');
+ my $append = $lei->{net}->can('imap_append');
my $mic = $lei->{net}->mic_get($self->{uri});
my $folder = $self->{uri}->mailbox;
my $lse = $lei->{lse}; # may be undef
+ my $sto = $lei->{opt}->{'mail-sync'} ? $lei->{sto} : undef;
+ my $out = $lei->{ovv}->{dst};
sub { # for git_to_mail
my ($bref, $smsg, $eml) = @_;
$mic // return $lei->fail; # mic may be undef-ed in last run
return if $dedupe && $dedupe->is_dup($eml //
PublicInbox::Eml->new($$bref));
$lse->xsmsg_vmd($smsg) if $lse;
- eval { $imap_append->($mic, $folder, $bref, $smsg, $eml) };
+ my $uid = eval { $append->($mic, $folder, $bref, $smsg, $eml) };
if (my $err = $@) {
undef $mic;
die $err;
}
+ # imap_append returns UID if IMAP server has UIDPLUS extension
+ ($sto && $uid =~ /\A[0-9]+\z/) and
+ $sto->ipc_do('set_sync_info',
+ $smsg->{blob}, $out, $uid + 0);
++$lei->{-nr_write};
}
}
^ permalink raw reply related [relevance 49%]
* yes [Re: should lei attempt to index mail outside of git?]
@ 2021-05-02 6:12 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-02 6:12 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> Currently, every mail lei indexes has a git blob associated with it.
It still will. But we have PublicInbox::ContentHash::git_sha
> I understand some folks might want to keep using their existing
> storage and not have a redundant, expensive-to-erase copy of the
> mail in git; but just want an indexing-only solution like mairix.
At least for Maildir, IMAP, and NNTP where random access is
reliable and fast, it should be doable.
> So, is this a feature worth implementing?
Since we have ~/.local/share/lei/store/mail_sync.sqlite3
nowadays, I think it's possible to make LeiToMail fallback
to retrieving from Maildir, IMAP, NNTP; at least. mbox
can be a pain since message offsets can change.
^ permalink raw reply [relevance 71%]
* [PATCH] lei up: fix dedupe with remote externals on Maildir + IMAP
@ 2021-05-03 20:57 57% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-03 20:57 UTC (permalink / raw)
To: meta
LeiToMail Maildir and IMAP write callbacks need to account for
the caller-supplied smsg. We'll also make better use of the
user-supplied smsg object by ensuring blob deduplication happens
ASAP.
Fixes: e76683309ca4f254 ("lei <q|up>: distinguish between mset and l2m counts")
---
lib/PublicInbox/LeiSavedSearch.pm | 15 ++++++++-------
lib/PublicInbox/LeiToMail.pm | 6 ++++--
t/lei-q-remote-import.t | 6 ++++++
3 files changed, 18 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LeiSavedSearch.pm b/lib/PublicInbox/LeiSavedSearch.pm
index 8177c98e..92ced28b 100644
--- a/lib/PublicInbox/LeiSavedSearch.pm
+++ b/lib/PublicInbox/LeiSavedSearch.pm
@@ -170,23 +170,24 @@ sub cfg_set { # called by LeiXSearch
sub is_dup {
my ($self, $eml, $smsg) = @_;
my $oidx = $self->{oidx} // die 'BUG: no {oidx}';
- my $blob = $smsg ? $smsg->{blob} : undef;
- my $lk = $self->lock_for_scope_fast;
- return 1 if $blob && $oidx->blob_exists($blob);
+ my $lk;
if ($self->{-dedupe_mid}) {
+ $lk //= $self->lock_for_scope_fast;
for my $mid (@{mids_for_index($eml)}) {
my ($id, $prv);
return 1 if $oidx->next_by_mid($mid, \$id, \$prv);
}
}
+ my $blob = $smsg ? $smsg->{blob} : git_sha(1, $eml)->hexdigest;
+ $lk //= $self->lock_for_scope_fast;
+ return 1 if $oidx->blob_exists($blob);
if (my $xoids = PublicInbox::LeiSearch::xoids_for($self, $eml, 1)) {
for my $docid (values %$xoids) {
$oidx->add_xref3($docid, -1, $blob, '.');
}
$oidx->commit_lazy;
if ($self->{-dedupe_oid}) {
- $smsg->{blob} //= git_sha(1, $eml)->hexdigest;
- exists $xoids->{$smsg->{blob}} ? 1 : undef;
+ exists $xoids->{$blob} ? 1 : undef;
} else {
1;
}
@@ -197,11 +198,11 @@ sub is_dup {
$smsg->{bytes} = 0;
$smsg->populate($eml);
}
+ $smsg->{blob} //= $blob;
$oidx->begin_lazy;
$smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
- $smsg->{blob} //= git_sha(1, $eml)->hexdigest;
$oidx->add_overview($eml, $smsg);
- $oidx->add_xref3($smsg->{num}, -1, $smsg->{blob}, '.');
+ $oidx->add_xref3($smsg->{num}, -1, $blob, '.');
$oidx->commit_lazy;
undef;
}
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 71acf952..64061788 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -283,7 +283,8 @@ sub _maildir_write_cb ($$) {
my ($bref, $smsg, $eml) = @_;
$dst // return $lei->fail; # dst may be undef-ed in last run
return if $dedupe && $dedupe->is_dup($eml //
- PublicInbox::Eml->new($$bref));
+ PublicInbox::Eml->new($$bref),
+ $smsg);
$lse->xsmsg_vmd($smsg) if $lse;
my $n = _buf2maildir($dst, $bref // \($eml->as_string), $smsg);
$sto->ipc_do('set_sync_info', $smsg->{blob}, $out, $n) if $sto;
@@ -305,7 +306,8 @@ sub _imap_write_cb ($$) {
my ($bref, $smsg, $eml) = @_;
$mic // return $lei->fail; # mic may be undef-ed in last run
return if $dedupe && $dedupe->is_dup($eml //
- PublicInbox::Eml->new($$bref));
+ PublicInbox::Eml->new($$bref),
+ $smsg);
$lse->xsmsg_vmd($smsg) if $lse;
my $uid = eval { $append->($mic, $folder, $bref, $smsg, $eml) };
if (my $err = $@) {
diff --git a/t/lei-q-remote-import.t b/t/lei-q-remote-import.t
index 32c5172b..80067061 100644
--- a/t/lei-q-remote-import.t
+++ b/t/lei-q-remote-import.t
@@ -91,5 +91,11 @@ EOF
lei_ok(qw(q -o mboxrd:/dev/stdout m:never-before-seen@example.com));
like($lei_out, qr/seen\@example\.com>\nStatus: RO\n\nwhatever/sm,
'--import-before imported totally unseen message');
+
+ lei_ok(qw(q --save z:0.. -o), "$ENV{HOME}/md", '--only', $url);
+ my @f = glob("$ENV{HOME}/md/*/*");
+ lei_ok('up', "$ENV{HOME}/md");
+ is_deeply(\@f, [ glob("$ENV{HOME}/md/*/*") ],
+ 'lei up remote dedupe works on maildir');
});
done_testing;
^ permalink raw reply related [relevance 57%]
* [PATCH 1/5] lei ls-mail-sync: update reference to ls-sync
@ 2021-05-04 4:45 71% ` Kyle Meyer
2021-05-04 4:45 71% ` [PATCH 2/5] lei ls-mail-sync: drop repeated -z/0 option Kyle Meyer
` (3 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-05-04 4:45 UTC (permalink / raw)
To: meta
ls-sync was renamed to ls-mail-sync in cb0e9d42b799c748. Update a
stale reference to the old name.
---
lib/PublicInbox/LeiLsMailSync.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiLsMailSync.pm b/lib/PublicInbox/LeiLsMailSync.pm
index 2b3d326d..532ea9b5 100644
--- a/lib/PublicInbox/LeiLsMailSync.pm
+++ b/lib/PublicInbox/LeiLsMailSync.pm
@@ -1,7 +1,7 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# front-end for the "lei ls-sync" sub-command
+# front-end for the "lei ls-mail-sync" sub-command
package PublicInbox::LeiLsMailSync;
use strict;
use v5.10.1;
--
2.31.1
^ permalink raw reply related [relevance 71%]
* [PATCH 2/5] lei ls-mail-sync: drop repeated -z/0 option
2021-05-04 4:45 71% ` [PATCH 1/5] lei ls-mail-sync: update reference to ls-sync Kyle Meyer
@ 2021-05-04 4:45 71% ` Kyle Meyer
2021-05-04 4:45 71% ` [PATCH 3/5] lei ls-mail-sync: accept a filter Kyle Meyer
` (2 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-05-04 4:45 UTC (permalink / raw)
To: meta
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 599cfab2..e9c1675a 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -171,7 +171,7 @@ our %CMD = ( # sorted in order of importance/use:
qw(format|f=s z|0 globoff|g invert-match|v local remote), @c_opt ],
'ls-label' => [ '', 'list labels', qw(z|0 stats:s), @c_opt ],
'ls-mail-sync' => [ '', 'list mail sync folders',
- qw(z|0 z|0 globoff|g invert-match|v local remote), @c_opt ],
+ qw(z|0 globoff|g invert-match|v local remote), @c_opt ],
'forget-external' => [ 'LOCATION...|--prune',
'exclude further results from a publicinbox|extindex',
qw(prune), @c_opt ],
--
2.31.1
^ permalink raw reply related [relevance 71%]
* [PATCH 3/5] lei ls-mail-sync: accept a filter
2021-05-04 4:45 71% ` [PATCH 1/5] lei ls-mail-sync: update reference to ls-sync Kyle Meyer
2021-05-04 4:45 71% ` [PATCH 2/5] lei ls-mail-sync: drop repeated -z/0 option Kyle Meyer
@ 2021-05-04 4:45 71% ` Kyle Meyer
2021-05-04 4:45 68% ` [PATCH 4/5] lei ls-mail-sync: fix handling of non-wildcard filters Kyle Meyer
2021-05-04 4:45 71% ` [PATCH 5/5] lei: add help output for --invert match Kyle Meyer
4 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-05-04 4:45 UTC (permalink / raw)
To: meta
lei_ls_mail_sync() is written to accept a filter, and ls-mail-sync has
related command-line options (--globoff, --invert-match), but a
positional argument isn't actually accepted. Add it.
---
lib/PublicInbox/LEI.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index e9c1675a..7be68121 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -170,7 +170,7 @@ our %CMD = ( # sorted in order of importance/use:
'ls-external' => [ '[FILTER]', 'list publicinbox|extindex locations',
qw(format|f=s z|0 globoff|g invert-match|v local remote), @c_opt ],
'ls-label' => [ '', 'list labels', qw(z|0 stats:s), @c_opt ],
-'ls-mail-sync' => [ '', 'list mail sync folders',
+'ls-mail-sync' => [ '[FILTER]', 'list mail sync folders',
qw(z|0 globoff|g invert-match|v local remote), @c_opt ],
'forget-external' => [ 'LOCATION...|--prune',
'exclude further results from a publicinbox|extindex',
--
2.31.1
^ permalink raw reply related [relevance 71%]
* [PATCH 5/5] lei: add help output for --invert match
` (3 preceding siblings ...)
2021-05-04 4:45 68% ` [PATCH 4/5] lei ls-mail-sync: fix handling of non-wildcard filters Kyle Meyer
@ 2021-05-04 4:45 71% ` Kyle Meyer
4 siblings, 0 replies; 200+ results
From: Kyle Meyer @ 2021-05-04 4:45 UTC (permalink / raw)
To: meta
ls-external and ls-mail-sync accept an --invert-match option. Show it
in the --help output.
---
lib/PublicInbox/LEI.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 7be68121..c5fdfeb8 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -270,6 +270,7 @@ my %OPTDESC = (
'incremental! import' => 'import already seen IMAP and NNTP articles',
'globoff|g' => "do not match locations using '*?' wildcards ".
"and\xa0'[]'\x{a0}ranges",
+'invert-match|v' => 'select non-matching lines',
'color!' => 'disable color (for --format=text)',
'verbose|v+' => 'be more verbose',
'external!' => 'do not use externals',
--
2.31.1
^ permalink raw reply related [relevance 71%]
* [PATCH 4/5] lei ls-mail-sync: fix handling of non-wildcard filters
` (2 preceding siblings ...)
2021-05-04 4:45 71% ` [PATCH 3/5] lei ls-mail-sync: accept a filter Kyle Meyer
@ 2021-05-04 4:45 68% ` Kyle Meyer
2021-05-04 5:14 67% ` Eric Wong
2021-05-04 4:45 71% ` [PATCH 5/5] lei: add help output for --invert match Kyle Meyer
4 siblings, 1 reply; 200+ results
From: Kyle Meyer @ 2021-05-04 4:45 UTC (permalink / raw)
To: meta
If lei_ls_mail_sync() is given a filter without any wildcards and
--globoff is unspecified, glob2re() will return undef, resulting in
the final regular expression being undefined. Add a fallback value.
---
I'm not sure if this is the cleanest approach; repeating
qr/\Q$filter\E/ seems ugly. I considered something closer to what
lei_ls_external() does, but decided against it because it leads to
--globoff with no filter showing no output, which I think is
surprising (even if passing --globoff with no filter doesn't make
any sense).
This probably deserves a test, but I'm out of time for tonight. If
the change looks okay, I'm happy to look into adding a test
tomorrow.
lib/PublicInbox/LeiLsMailSync.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiLsMailSync.pm b/lib/PublicInbox/LeiLsMailSync.pm
index 532ea9b5..06e25a63 100644
--- a/lib/PublicInbox/LeiLsMailSync.pm
+++ b/lib/PublicInbox/LeiLsMailSync.pm
@@ -14,7 +14,7 @@ sub lei_ls_mail_sync {
my $opt = $lei->{opt};
my $re;
$re = defined($filter) ? qr/\Q$filter\E/ : qr/./ if $opt->{globoff};
- $re //= $lei->glob2re($filter // '*');
+ $re //= $lei->glob2re($filter // '*') // qr/\Q$filter\E/;;
my @f = $lms->folders;
@f = $opt->{'invert-match'} ? grep(!/$re/, @f) : grep(/$re/, @f);
if ($opt->{'local'} && !$opt->{remote}) {
--
2.31.1
^ permalink raw reply related [relevance 68%]
* Re: [PATCH 4/5] lei ls-mail-sync: fix handling of non-wildcard filters
2021-05-04 4:45 68% ` [PATCH 4/5] lei ls-mail-sync: fix handling of non-wildcard filters Kyle Meyer
@ 2021-05-04 5:14 67% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-04 5:14 UTC (permalink / raw)
To: Kyle Meyer; +Cc: meta
Kyle Meyer <kyle@kyleam.com> wrote:
> If lei_ls_mail_sync() is given a filter without any wildcards and
> --globoff is unspecified, glob2re() will return undef, resulting in
> the final regular expression being undefined. Add a fallback value.
> ---
>
> I'm not sure if this is the cleanest approach; repeating
> qr/\Q$filter\E/ seems ugly. I considered something closer to what
> lei_ls_external() does, but decided against it because it leads to
> --globoff with no filter showing no output, which I think is
> surprising (even if passing --globoff with no filter doesn't make
> any sense).
Thanks for bringing this up.
My code there was an insomnia-generated disaster :x
Mixing a ternary statement with a trailing "if" was just gross.
Below is a shorter, less-redundant rewrite based on your fix.
> This probably deserves a test, but I'm out of time for tonight. If
> the change looks okay, I'm happy to look into adding a test
> tomorrow.
Yes, but probably no rush. I still need to get mail-sync
working... (sidetracked into working on "lei index" :x)
----------8<------------------------
Subject: [PATCH] lei ls-mail-sync: fix handling of non-wildcard filters
If lei_ls_mail_sync() is given a filter without any wildcards
and --globoff is unspecified, glob2re() will return undef,
resulting in the final regular expression being undefined.
Always use a fallback value when there's no RE.
Based-on-patch-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/20210504044559.12941-5-kyle@kyleam.com/
---
lib/PublicInbox/LeiLsMailSync.pm | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/LeiLsMailSync.pm b/lib/PublicInbox/LeiLsMailSync.pm
index 532ea9b5..505c0b3f 100644
--- a/lib/PublicInbox/LeiLsMailSync.pm
+++ b/lib/PublicInbox/LeiLsMailSync.pm
@@ -12,9 +12,8 @@ sub lei_ls_mail_sync {
my $sto = $lei->_lei_store or return;
my $lms = $sto->search->lms or return;
my $opt = $lei->{opt};
- my $re;
- $re = defined($filter) ? qr/\Q$filter\E/ : qr/./ if $opt->{globoff};
- $re //= $lei->glob2re($filter // '*');
+ my $re = $opt->{globoff} ? undef : $lei->glob2re($filter // '*');
+ $re //= qr/\Q$filter\E/;
my @f = $lms->folders;
@f = $opt->{'invert-match'} ? grep(!/$re/, @f) : grep(/$re/, @f);
if ($opt->{'local'} && !$opt->{remote}) {
^ permalink raw reply related [relevance 67%]
* [PATCH] lei: fix mail_sync.qlite3 folder names for NNTP
@ 2021-05-04 5:24 66% Eric Wong
2021-05-04 5:26 71% ` s/qlite3/s&/ [was: [PATCH] lei: fix ... folder names for NNTP] Eric Wong
0 siblings, 1 reply; 200+ results
From: Eric Wong @ 2021-05-04 5:24 UTC (permalink / raw)
To: meta
We should not have "SCALAR(XXXXXXX)" showing up in SQLite DBs
because we passed a SCALAR ref instead of a non-ref SCALAR.
---
lib/PublicInbox/NetReader.pm | 3 ++-
t/lei-import-nntp.t | 7 ++++---
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 64910fe1..fd0d1682 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -628,6 +628,7 @@ sub _nntp_fetch_all ($$$) {
warn "# $uri fetching ARTICLE $beg..$end\n";
}
my $n = $self->{max_batch};
+ my $url = $$uri;
for ($beg..$end) {
last if $self->{quit};
$art = $_;
@@ -650,7 +651,7 @@ sub _nntp_fetch_all ($$$) {
$raw = join('', @$raw);
$raw =~ s/\r\n/\n/sg;
my ($eml_cb, @args) = @{$self->{eml_each}};
- $eml_cb->($uri, $art, $kw, PublicInbox::Eml->new(\$raw), @args);
+ $eml_cb->($url, $art, $kw, PublicInbox::Eml->new(\$raw), @args);
$last_art = $art;
}
run_commit_cb($self);
diff --git a/t/lei-import-nntp.t b/t/lei-import-nntp.t
index 12bb002a..662da309 100644
--- a/t/lei-import-nntp.t
+++ b/t/lei-import-nntp.t
@@ -16,10 +16,9 @@ test_lei({ tmpdir => $tmpdir }, sub {
lei_ok(qw(q z:1..));
my $out = json_utf8->decode($lei_out);
is_deeply($out, [ undef ], 'nothing imported, yet');
- lei_ok('import', "nntp://$host_port/t.v2");
- diag $lei_err;
+ my $url = "nntp://$host_port/t.v2";
+ lei_ok('import', $url);
lei_ok(qw(q z:1..));
- diag $lei_err;
$out = json_utf8->decode($lei_out);
ok(scalar(@$out) > 1, 'got imported messages');
is(pop @$out, undef, 'trailing JSON null element was null');
@@ -29,5 +28,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
my $f = "$ENV{HOME}/.local/share/lei/store/mail_sync.sqlite3";
ok(-s $f, 'mail_sync exists tracked for redundant imports');
+ lei_ok 'ls-mail-sync';
+ like($lei_out, qr!\A\Q$url\E\n\z!, 'ls-mail-sync output as-expected');
});
done_testing;
^ permalink raw reply related [relevance 66%]
* s/qlite3/s&/ [was: [PATCH] lei: fix ... folder names for NNTP]
2021-05-04 5:24 66% [PATCH] lei: fix mail_sync.qlite3 folder names for NNTP Eric Wong
@ 2021-05-04 5:26 71% ` Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-04 5:26 UTC (permalink / raw)
To: meta
ill ix he itle efore ushing :x
^ permalink raw reply [relevance 71%]
* [PATCH] lei index: new command to index mail w/o git storage
@ 2021-05-04 9:49 26% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-04 9:49 UTC (permalink / raw)
To: meta
Since completely purging blobs from git is slow, users may wish
to index messages in Maildirs (and eventually other local
storage) without storing data in git.
Much code from LeiImport and LeiInput is reused, and a new dummy
FakeImport class supplies a non-storing $im->add and minimize
changes to LeiStore.
The tricky part of this command is to support "lei import"
after a message has gone through "lei index". Relying on
$smsg->{bytes} == 0 (as we do for external-only vmd storage)
does not work here, since it would break searching for "z:"
byte-ranges when not using externals.
This eventually required PublicInbox::Import::add to use a
SharedKV to keep track of imported blobs and prevent
duplication.
---
MANIFEST | 3 ++
lib/PublicInbox/FakeImport.pm | 23 ++++++++++++++
lib/PublicInbox/Import.pm | 26 +++++++--------
lib/PublicInbox/LeiBlob.pm | 38 +++++++++++++++-------
lib/PublicInbox/LeiImport.pm | 15 ++++++---
lib/PublicInbox/LeiIndex.pm | 48 ++++++++++++++++++++++++++++
lib/PublicInbox/LeiInput.pm | 30 ++++++------------
lib/PublicInbox/LeiMailSync.pm | 27 ++++++++++++++++
lib/PublicInbox/LeiStore.pm | 10 +++++-
lib/PublicInbox/LeiToMail.pm | 10 +++++-
lib/PublicInbox/OverIdx.pm | 18 +++++++++++
t/lei-index.t | 58 ++++++++++++++++++++++++++++++++++
12 files changed, 253 insertions(+), 53 deletions(-)
create mode 100644 lib/PublicInbox/FakeImport.pm
create mode 100644 lib/PublicInbox/LeiIndex.pm
create mode 100644 t/lei-index.t
diff --git a/MANIFEST b/MANIFEST
index e23297fa..42729b9c 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -148,6 +148,7 @@ lib/PublicInbox/EmlContentFoo.pm
lib/PublicInbox/ExtMsg.pm
lib/PublicInbox/ExtSearch.pm
lib/PublicInbox/ExtSearchIdx.pm
+lib/PublicInbox/FakeImport.pm
lib/PublicInbox/FakeInotify.pm
lib/PublicInbox/Feed.pm
lib/PublicInbox/Filter/Base.pm
@@ -198,6 +199,7 @@ lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiForgetSearch.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
+lib/PublicInbox/LeiIndex.pm
lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiInspect.pm
@@ -404,6 +406,7 @@ t/lei-import-imap.t
t/lei-import-maildir.t
t/lei-import-nntp.t
t/lei-import.t
+t/lei-index.t
t/lei-lcat.t
t/lei-mirror.t
t/lei-p2q.t
diff --git a/lib/PublicInbox/FakeImport.pm b/lib/PublicInbox/FakeImport.pm
new file mode 100644
index 00000000..dea25cbe
--- /dev/null
+++ b/lib/PublicInbox/FakeImport.pm
@@ -0,0 +1,23 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# pretend to do PublicInbox::Import::add for "lei index"
+package PublicInbox::FakeImport;
+use strict;
+use PublicInbox::ContentHash qw(git_sha);
+
+sub new { bless { bytes_added => 0 }, __PACKAGE__ }
+
+sub add {
+ my ($self, $eml, $check_cb, $smsg) = @_;
+ $smsg->populate($eml);
+ my $raw = $eml->as_string;
+ $smsg->{blob} = git_sha(1, \$raw)->hexdigest;
+ $smsg->set_bytes($raw, length($raw));
+ if (my $oidx = delete $smsg->{-oidx}) { # used by LeiStore
+ $oidx->vivify_xvmd($smsg) or return;
+ }
+ 1;
+}
+
+1;
diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index 3adf9dec..362cdc47 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -413,19 +413,19 @@ sub add {
$smsg->{blob} = $self->get_mark(":$blob");
$smsg->set_bytes($raw_email, $n);
if (my $oidx = delete $smsg->{-oidx}) { # used by LeiStore
- my @docids = $oidx->blob_exists($smsg->{blob});
- my @vivify_xvmd;
- for my $id (@docids) {
- if (my $cur = $oidx->get_art($id)) {
- # already imported if bytes > 0
- return if $cur->{bytes} > 0;
- push @vivify_xvmd, $id;
- } else {
- warn "W: $smsg->{blob} ",
- "#$id gone (bug?)\n";
- }
- }
- $smsg->{-vivify_xvmd} = \@vivify_xvmd;
+ my $eidx_git = delete $smsg->{-eidx_git};
+
+ # we need this sharedkv to dedupe blobs added in the
+ # same fast-import transaction
+ my $u = $self->{uniq_skv} //= do {
+ require PublicInbox::SharedKV;
+ my $x = PublicInbox::SharedKV->new;
+ $x->dbh;
+ $x;
+ };
+ return if !$u->set_maybe(pack('H*', $smsg->{blob}), 1);
+ return if (!$oidx->vivify_xvmd($smsg) &&
+ $eidx_git->check($smsg->{blob}));
}
}
my $ref = $self->{ref};
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 710430a2..8de86565 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -87,6 +87,16 @@ sub cat_attach_i { # Eml->each_part callback
$lei->out($part->body);
}
+sub extract_attach ($$$) {
+ my ($lei, $blob, $bref) = @_;
+ my $eml = PublicInbox::Eml->new($bref);
+ $eml->each_part(\&cat_attach_i, $lei, 1);
+ my $idx = delete $lei->{-attach_idx};
+ defined($idx) and return $lei->fail(<<EOM);
+E: attachment $idx not found in $blob
+EOM
+}
+
sub lei_blob {
my ($lei, $blob) = @_;
$lei->start_pager if -t $lei->{1};
@@ -106,7 +116,7 @@ sub lei_blob {
}
my $rdr = {};
if ($opt->{mail}) {
- $rdr->{2} = $lei->{2};
+ open $rdr->{2}, '+>', undef or die "open: $!";
} else {
open $rdr->{2}, '>', '/dev/null' or die "open: $!";
}
@@ -115,21 +125,25 @@ sub lei_blob {
if (defined $lei->{-attach_idx}) {
my $fh = popen_rd($cmd, $lei->{env}, $rdr);
require PublicInbox::Eml;
- my $str = do { local $/; <$fh> };
- if (close $fh) {
- my $eml = PublicInbox::Eml->new(\$str);
- $eml->each_part(\&cat_attach_i, $lei, 1);
- my $idx = delete $lei->{-attach_idx};
- defined($idx) and return $lei->fail(<<EOM);
-E: attachment $idx not found in $blob
-EOM
- }
+ my $buf = do { local $/; <$fh> };
+ return extract_attach($lei, $blob, \$buf) if close($fh);
} else {
$rdr->{1} = $lei->{1};
waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
}
- return if $? == 0;
- return $lei->child_error($?) if $opt->{mail};
+ my $ce = $?;
+ return if $ce == 0;
+ my $sto = $lei->_lei_store;
+ my $lms = $sto ? $sto->search->lms : undef;
+ if (my $bref = $lms ? $lms->local_blob($blob, 1) : undef) {
+ defined($lei->{-attach_idx}) and
+ return extract_attach($lei, $blob, $bref);
+ return $lei->out($$bref);
+ } elsif ($opt->{mail}) {
+ my $eh = $rdr->{2};
+ seek($eh, 0, 0);
+ return $lei->child_error($ce, do { local $/; <$eh> });
+ } # else: fall through to solver below
}
# maybe it's a non-email (code) blob from a coderepo
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 6a57df47..55925cc5 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -38,21 +38,20 @@ sub input_maildir_cb { # maildir_each_eml cb
warn "E: $f was not from a Maildir?\n";
}
}
- input_eml_cb($self, $eml, $vmd);
+ $self->input_eml_cb($eml, $vmd);
}
sub input_net_cb { # imap_each / nntp_each
my ($url, $uid, $kw, $eml, $self) = @_;
my $vmd = $self->{-import_kw} ? { kw => $kw } : undef;
$vmd->{sync_info} = [ $url, $uid ] if $self->{-mail_sync};
- input_eml_cb($self, $eml, $vmd);
+ $self->input_eml_cb($eml, $vmd);
}
-sub lei_import { # the main "lei import" method
- my ($lei, @inputs) = @_;
+sub do_import_index ($$@) {
+ my ($self, $lei, @inputs) = @_;
my $sto = $lei->_lei_store(1);
$sto->write_prepare($lei);
- my $self = bless {}, __PACKAGE__;
$self->{-import_kw} = $lei->{opt}->{kw} // 1;
my $vmd_mod = $self->vmd_mod_extract(\@inputs);
return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
@@ -83,6 +82,12 @@ sub lei_import { # the main "lei import" method
$op_c->op_wait_event($ops);
}
+sub lei_import { # the main "lei import" method
+ my ($lei, @inputs) = @_;
+ my $self = bless {}, __PACKAGE__;
+ do_import_index($self, $lei, @inputs);
+}
+
sub _complete_import {
my ($lei, @argv) = @_;
my $sto = $lei->_lei_store or return;
diff --git a/lib/PublicInbox/LeiIndex.pm b/lib/PublicInbox/LeiIndex.pm
new file mode 100644
index 00000000..cc3e83e7
--- /dev/null
+++ b/lib/PublicInbox/LeiIndex.pm
@@ -0,0 +1,48 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# front-end for the "lei index" sub-command, this is similar to
+# "lei import" but doesn't put a git blob into ~/.local/share/lei/store
+package PublicInbox::LeiIndex;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
+use PublicInbox::LeiImport;
+
+# /^input_/ subs are used by (or override) PublicInbox::LeiInput superclass
+sub input_eml_cb { # used by input_maildir_cb and input_net_cb
+ my ($self, $eml, $vmd) = @_;
+ my $xoids = $self->{lei}->{ale}->xoids_for($eml);
+ if (my $all_vmd = $self->{all_vmd}) {
+ @$vmd{keys %$all_vmd} = values %$all_vmd;
+ }
+ $self->{lei}->{sto}->ipc_do('index_eml_only', $eml, $vmd, $xoids);
+}
+
+sub input_fh { # overrides PublicInbox::LeiInput::input_fh
+ my ($self, $ifmt, $fh, $input, @args) = @_;
+ $self->{lei}->child_error(1<<8, <<EOM);
+$input ($ifmt) not yet supported, try `lei import'
+EOM
+}
+
+sub lei_index {
+ my ($lei, @argv) = @_;
+ $lei->{opt}->{'mail-sync'} = 1;
+ my $self = bless {}, __PACKAGE__;
+ PublicInbox::LeiImport::do_import_index($self, $lei, @argv);
+}
+
+no warnings 'once';
+no strict 'refs';
+for my $m (qw(input_maildir_cb input_net_cb)) {
+ *$m = PublicInbox::LeiImport->can($m);
+}
+
+*_complete_import = \&PublicInbox::LeiImport::_complete_import;
+*ipc_atfork_child = \&PublicInbox::LeiInput::input_only_atfork_child;
+*net_merge_all_done = \&PublicInbox::LeiInput::input_only_net_merge_all_done;
+
+# the following works even when LeiAuth is lazy-loaded
+*net_merge_all = \&PublicInbox::LeiAuth::net_merge_all;
+1;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 85caac35..46eea111 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -1,7 +1,7 @@
# Copyright (C) 2021 all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# parent class for LeiImport, LeiConvert
+# parent class for LeiImport, LeiConvert, LeiIndex
package PublicInbox::LeiInput;
use strict;
use v5.10.1;
@@ -93,11 +93,7 @@ sub handle_http_input ($$@) {
my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
grep(/\A--compressed\z/, @$curl) or
$fh = IO::Uncompress::Gunzip->new($fh, MultiStream => 1);
- eval {
- PublicInbox::MboxReader->mboxrd($fh,
- $self->can('input_mbox_cb'),
- $self, @args);
- };
+ eval { $self->input_fh('mboxrd', $fh, $url, @args) };
my $err = $@;
waitpid($pid, 0);
$? || $err and
@@ -221,14 +217,8 @@ sub prepare_inputs { # returns undef on error
require PublicInbox::NetReader;
$net //= PublicInbox::NetReader->new;
$net->add_url($input);
- if ($sync) {
- if ($input =~ m!\Aimaps?://!) {
- push @{$sync->{ok}}, $input;
- } else {
- push @{$sync->{no}}, $input;
- }
- }
- } elsif ($input_path =~ m!\Ahttps?://!i) {
+ push @{$sync->{ok}}, $input if $sync;
+ } elsif ($input_path =~ m!\Ahttps?://!i) { # mboxrd.gz
# TODO: how would we detect r/w JMAP?
push @{$sync->{no}}, $input if $sync;
prepare_http_input($self, $lei, $input_path) or return;
@@ -239,12 +229,10 @@ sub prepare_inputs { # returns undef on error
--in-format=$in_fmt and `$ifmt:' conflict
}
- if ($sync) {
- if ($ifmt =~ /\A(?:maildir|mh)\z/i) {
- push @{$sync->{ok}}, $input;
- } else {
- push @{$sync->{no}}, $input;
- }
+ if ($ifmt =~ /\A(?:maildir|mh)\z/i) {
+ push @{$sync->{ok}}, $input if $sync;
+ } else {
+ push @{$sync->{no}}, $input if $sync;
}
my $devfd = $lei->path_to_fd($input_path) // return;
if ($devfd >= 0 || (-f $input_path || -p _)) {
@@ -260,7 +248,7 @@ sub prepare_inputs { # returns undef on error
} else {
return $lei->fail("Unable to handle $input");
}
- } elsif ($input =~ /\.(eml|patch)\z/i && -f $input) {
+ } elsif ($input =~ /\.(?:eml|patch)\z/i && -f $input) {
lc($in_fmt//'eml') eq 'eml' or return $lei->fail(<<"");
$input is `eml', not --in-format=$in_fmt
diff --git a/lib/PublicInbox/LeiMailSync.pm b/lib/PublicInbox/LeiMailSync.pm
index 2ce189fa..2e74e433 100644
--- a/lib/PublicInbox/LeiMailSync.pm
+++ b/lib/PublicInbox/LeiMailSync.pm
@@ -6,6 +6,7 @@ package PublicInbox::LeiMailSync;
use strict;
use v5.10.1;
use DBI;
+use PublicInbox::ContentHash qw(git_sha);
sub dbh_new {
my ($self, $rw) = @_;
@@ -208,4 +209,30 @@ sub folders {
map { $_->[0] } @{$dbh->selectall_arrayref($sql, undef, @pfx)};
}
+sub local_blob {
+ my ($self, $oidhex, $vrfy) = @_;
+ my $dbh = $self->{dbh} //= dbh_new($self);
+ my $b2n = $dbh->prepare(<<'');
+SELECT f.loc,b.name FROM blob2name b
+LEFT JOIN folders f ON b.fid = f.fid
+WHERE b.oidbin = ?
+
+ $b2n->execute(pack('H*', $oidhex));
+ while (my ($d, $n) = $b2n->fetchrow_array) {
+ substr($d, 0, length('maildir:')) = '';
+ my $f = "$d/" . ($n =~ /:2,[a-zA-Z]*\z/ ? "cur/$n" : "new/$n");
+ open my $fh, '<', $f or next;
+ if (-s $fh) {
+ local $/;
+ my $raw = <$fh>;
+ if ($vrfy && git_sha(1, \$raw)->hexdigest ne $oidhex) {
+ warn "$f changed $oidhex\n";
+ next;
+ }
+ return \$raw;
+ }
+ }
+ undef;
+}
+
1;
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 29362b2e..a7a0ebef 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -206,10 +206,11 @@ sub set_sync_info {
sub add_eml {
my ($self, $eml, $vmd, $xoids) = @_;
- my $im = $self->importer; # may create new epoch
+ my $im = $self->{-fake_im} // $self->importer; # may create new epoch
my ($eidx, $tl) = eidx_init($self);
my $oidx = $eidx->{oidx}; # PublicInbox::Import::add checks this
my $smsg = bless { -oidx => $oidx }, 'PublicInbox::Smsg';
+ $smsg->{-eidx_git} = $eidx->git if !$self->{-fake_im};
my $im_mark = $im->add($eml, undef, $smsg);
if ($vmd && $vmd->{sync_info}) {
set_sync_info($self, $smsg->{blob}, @{$vmd->{sync_info}});
@@ -276,6 +277,13 @@ sub set_eml {
set_eml_vmd($self, $eml, $vmd);
}
+sub index_eml_only {
+ my ($self, $eml, $vmd, $xoids) = @_;
+ require PublicInbox::FakeImport;
+ local $self->{-fake_im} = PublicInbox::FakeImport->new;
+ set_eml($self, $eml, $vmd, $xoids);
+}
+
sub _external_only ($$$) {
my ($self, $xoids, $eml) = @_;
my $eidx = $self->{priv_eidx};
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index 64061788..da3a95d2 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -137,9 +137,15 @@ sub eml2mboxcl2 {
sub git_to_mail { # git->cat_async callback
my ($bref, $oid, $type, $size, $arg) = @_;
+ my ($write_cb, $smsg) = @$arg;
+ if ($type eq 'missing' && $smsg->{-lms_ro}) {
+ if ($bref = $smsg->{-lms_ro}->local_blob($oid, 1)) {
+ $type = 'blob';
+ $size = length($$bref);
+ }
+ }
return warn("W: $oid is $type (!= blob)\n") if $type ne 'blob';
return warn("E: $oid is empty\n") unless $size;
- my ($write_cb, $smsg) = @$arg;
die "BUG: expected=$smsg->{blob} got=$oid" if $smsg->{blob} ne $oid;
$write_cb->($bref, $smsg);
}
@@ -644,6 +650,7 @@ sub ipc_atfork_child {
my ($self) = @_;
my $lei = $self->{lei};
$lei->_lei_atfork_child;
+ $self->{-lms_ro} = $lei->{lse}->lms if $lei->{lse};
$lei->{auth}->do_auth_atfork($self) if $lei->{auth};
$SIG{__WARN__} = PublicInbox::Eml::warn_ignore_cb();
$self->SUPER::ipc_atfork_child;
@@ -665,6 +672,7 @@ sub poke_dst {
sub write_mail { # via ->wq_io_do
my ($self, $smsg, $eml) = @_;
return $self->{wcb}->(undef, $smsg, $eml) if $eml;
+ $smsg->{-lms_ro} = $self->{-lms_ro};
$self->{lei}->{ale}->git->cat_async($smsg->{blob}, \&git_to_mail,
[$self->{wcb}, $smsg]);
}
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index 66dec099..5f96a5b0 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -670,4 +670,22 @@ DELETE FROM eidxq WHERE docid = ?
}
+# returns true if we're vivifying a message for lei/store that was
+# previously external-metadata only
+sub vivify_xvmd {
+ my ($self, $smsg) = @_;
+ my @docids = $self->blob_exists($smsg->{blob});
+ my @vivify_xvmd;
+ for my $id (@docids) {
+ if (my $cur = $self->get_art($id)) {
+ # already indexed if bytes > 0
+ return if $cur->{bytes} > 0;
+ push @vivify_xvmd, $id;
+ } else {
+ warn "W: $smsg->{blob} #$id gone (bug?)\n";
+ }
+ }
+ $smsg->{-vivify_xvmd} = \@vivify_xvmd;
+}
+
1;
diff --git a/t/lei-index.t b/t/lei-index.t
new file mode 100644
index 00000000..3382d42b
--- /dev/null
+++ b/t/lei-index.t
@@ -0,0 +1,58 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use strict; use v5.10.1; use PublicInbox::TestCommon;
+use File::Spec;
+require_mods(qw(lei -nntpd));
+my ($ro_home, $cfg_path) = setup_public_inboxes;
+my ($tmpdir, $for_destroy) = tmpdir;
+my $env = { PI_CONFIG => $cfg_path };
+
+my $sock = tcp_server;
+my $cmd = [ '-nntpd', '-W0', "--stdout=$tmpdir/n1", "--stderr=$tmpdir/n2" ];
+my $nntpd = start_script($cmd, $env, { 3 => $sock }) or BAIL_OUT("-nntpd $?");
+my $nntp_host_port = tcp_host_port($sock);
+
+$sock = tcp_server;
+$cmd = [ '-imapd', '-W0', "--stdout=$tmpdir/i1", "--stderr=$tmpdir/i2" ];
+my $imapd = start_script($cmd, $env, { 3 => $sock }) or BAIL_OUT("-imapd $?");
+my $imap_host_port = tcp_host_port($sock);
+undef $sock;
+for ('', qw(cur new)) {
+ mkdir "$tmpdir/md/$_" or xbail "mkdir: $!";
+}
+symlink(File::Spec->rel2abs('t/plack-qp.eml'), "$tmpdir/md/cur/x:2,");
+my $expect = do {
+ open my $fh, '<', 't/plack-qp.eml' or xbail $!;
+ local $/;
+ <$fh>;
+};
+test_lei({ tmpdir => $tmpdir }, sub {
+ my $store_path = "$ENV{HOME}/.local/share/lei/store/";
+
+ lei_ok('index', "$tmpdir/md");
+ lei_ok(qw(q mid:qp@example.com));
+ my $res_a = json_utf8->decode($lei_out);
+ my $blob = $res_a->[0]->{'blob'};
+ like($blob, qr/\A[0-9a-f]{40,}\z/, 'got blob from qp@example');
+ lei_ok('blob', $blob);
+ is($lei_out, $expect, 'got expected blob via Maildir');
+ lei_ok(qw(q mid:qp@example.com -f text));
+ like($lei_out, qr/^hi = bye/sm, 'lei2mail fallback');
+
+ my $all_obj = ['git', "--git-dir=$store_path/ALL.git",
+ qw(cat-file --batch-check --batch-all-objects)];
+ is_deeply([xqx($all_obj)], [], 'no git objects');
+ lei_ok('import', 't/plack-qp.eml');
+ ok(grep(/\A$blob blob /, my @objs = xqx($all_obj)),
+ 'imported blob');
+ lei_ok(qw(q z:0.. --dedupe=none));
+ my $res_b = json_utf8->decode($lei_out);
+ is_deeply($res_b, $res_a, 'no extra DB entries');
+
+ lei_ok('index', "nntp://$nntp_host_port/t.v2");
+ lei_ok('index', "imap://$imap_host_port/t.v2.0");
+ is_deeply([xqx($all_obj)], \@objs, 'no new objects from NNTP+IMAP');
+});
+
+done_testing;
^ permalink raw reply related [relevance 26%]
* [PATCH 2/2] lei blob: support "lei index"-ed mail
2021-05-05 10:46 71% [PATCH 0/2] lei rediff + solver-related fix Eric Wong
2021-05-05 10:46 30% ` [PATCH 1/2] lei rediff: regenerate diffs from stdin Eric Wong
@ 2021-05-05 10:46 87% ` Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-05-05 10:46 UTC (permalink / raw)
To: meta
Normal git retrieval don't work for Maildir blobs indexed using
"lei index". Fortunately, this oddness is limited to the
LeiStore class and we can override smsg_eml with a fallback
to read blobs from Maildirs.
---
lib/PublicInbox/LeiSearch.pm | 10 ++++++++++
t/solver_git.t | 10 ++++++++++
2 files changed, 20 insertions(+)
diff --git a/lib/PublicInbox/LeiSearch.pm b/lib/PublicInbox/LeiSearch.pm
index cd28a700..c2b12146 100644
--- a/lib/PublicInbox/LeiSearch.pm
+++ b/lib/PublicInbox/LeiSearch.pm
@@ -144,4 +144,14 @@ sub lms {
-f $f ? PublicInbox::LeiMailSync->new($f) : undef;
}
+# allow SolverGit->resolve_patch to work with "lei index"
+sub smsg_eml {
+ my ($self, $smsg) = @_;
+ PublicInbox::Inbox::smsg_eml($self, $smsg) // do {
+ my $lms = lms($self);
+ my $bref = $lms ? $lms->local_blob($smsg->{blob}, 1) : undef;
+ $bref ? PublicInbox::Eml->new($bref) : undef;
+ };
+}
+
1;
diff --git a/t/solver_git.t b/t/solver_git.t
index e566efb3..44cbbfdb 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -27,6 +27,11 @@ my $ibx = create_inbox 'v2', version => 2,
$im->add(eml_load 't/solve/0001-simple-mod.patch') or BAIL_OUT;
$im->add($patch2) or BAIL_OUT;
};
+my $md = "$tmpdir/md";
+File::Path::mkpath([map { $md.$_ } (qw(/ /cur /new /tmp))]);
+symlink(abs_path('t/solve/0001-simple-mod.patch'), "$md/cur/foo:2,") or
+ xbail "symlink: $!";
+
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
@@ -70,6 +75,11 @@ test_lei({tmpdir => "$tmpdir/rediff"}, sub {
'got more context with -U9');
});
+test_lei({tmpdir => "$tmpdir/index-eml-only"}, sub {
+ lei_ok(qw(index), $md);
+ lei_ok(qw(blob 69df7d5)); # hits LeiSearch->smsg_eml -> lms->local_blob
+});
+
my $git = PublicInbox::Git->new($git_dir);
$ibx->{-repo_objs} = [ $git ];
my $res;
^ permalink raw reply related [relevance 87%]
* [PATCH 0/2] lei rediff + solver-related fix
@ 2021-05-05 10:46 71% Eric Wong
2021-05-05 10:46 30% ` [PATCH 1/2] lei rediff: regenerate diffs from stdin Eric Wong
2021-05-05 10:46 87% ` [PATCH 2/2] lei blob: support "lei index"-ed mail Eric Wong
0 siblings, 2 replies; 200+ results
From: Eric Wong @ 2021-05-05 10:46 UTC (permalink / raw)
To: meta
"rediff" is something I've wanted in the WWW UI for several
years, however:
1) the amount of git-diff options is staggering and
I find HTML <form> elements difficult-to-use
2) git-diff can be monumentally expensive, on top of
being difficult/impossible to cache
..so I couldn't figure out a good way to support it
On a local system, we can feed "git diff" command-line options
and not worry about 10K users trying to regenerate diffs at
once.
Eric Wong (2):
lei rediff: regenerate diffs from stdin
lei blob: support "lei index"-ed mail
MANIFEST | 1 +
lib/PublicInbox/Inbox.pm | 2 +-
lib/PublicInbox/LEI.pm | 22 ++++
lib/PublicInbox/LeiInput.pm | 6 +
lib/PublicInbox/LeiRediff.pm | 245 +++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiSearch.pm | 10 ++
t/solver_git.t | 18 ++-
7 files changed, 302 insertions(+), 2 deletions(-)
create mode 100644 lib/PublicInbox/LeiRediff.pm
^ permalink raw reply [relevance 71%]
* [PATCH 1/2] lei rediff: regenerate diffs from stdin
2021-05-05 10:46 71% [PATCH 0/2] lei rediff + solver-related fix Eric Wong
@ 2021-05-05 10:46 30% ` Eric Wong
2021-05-05 10:46 87% ` [PATCH 2/2] lei blob: support "lei index"-ed mail Eric Wong
1 sibling, 0 replies; 200+ results
From: Eric Wong @ 2021-05-05 10:46 UTC (permalink / raw)
To: meta
Sometimes a mailed patch is generated with non-ideal output,
(lacking context, noisy whitespace changes, etc.), or a user
wants to use the same external diff viewer they've configured
git to use.
Since we have SolverGit to regenerate arbitrary blobs from
patches; this new command allows us to regenerate a diff with
different options using the blobs SolverGit gives us.
The amount of git-diff(1) options is mind numbing, so it's
likely I missed some favorites or botched the getopt spec
translation.
This also fixes Inbox::base_url to check psgi.url_scheme
before attempting to generate URLs and avoid uninitialized
variable warnings. Oddly, the "lei blob" tests did not
trigger these uninitialized warnings.
Note: this will automatically import+index the message(s)
it's regenerating, because solver relies on being able
to lookup pre/postimage OIDs and read blobs.
---
MANIFEST | 1 +
lib/PublicInbox/Inbox.pm | 2 +-
lib/PublicInbox/LEI.pm | 22 ++++
lib/PublicInbox/LeiInput.pm | 6 +
lib/PublicInbox/LeiRediff.pm | 245 +++++++++++++++++++++++++++++++++++
t/solver_git.t | 8 +-
6 files changed, 282 insertions(+), 2 deletions(-)
create mode 100644 lib/PublicInbox/LeiRediff.pm
diff --git a/MANIFEST b/MANIFEST
index b40147b0..7be07aa5 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -211,6 +211,7 @@ lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
lib/PublicInbox/LeiQuery.pm
+lib/PublicInbox/LeiRediff.pm
lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSavedSearch.pm
lib/PublicInbox/LeiSearch.pm
diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index da7ea75f..b94ffdb0 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -241,7 +241,7 @@ sub cloneurl {
sub base_url {
my ($self, $env) = @_; # env - PSGI env
- if ($env) {
+ if ($env && $env->{'psgi.url_scheme'}) {
my $url = PublicInbox::Git::host_prefix_url($env, '');
# for mount in Plack::Builder
$url .= '/' if $url !~ m!/\z!;
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index c5fdfeb8..9dbbeba9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -135,6 +135,23 @@ my @lxs_opt = (qw(remote! local! external! include|I=s@ exclude=s@ only=s@
import-remote! no-torsocks torsocks=s),
PublicInbox::LeiQuery::curl_opt());
+# we don't support -C as an alias for --find-copies since it's already
+# used for chdir
+our @diff_opt = qw(unified|U=i output-indicator-new=s output-indicator-old=s
+ output-indicator-context=s indent-heuristic!
+ minimal patience histogram anchored=s@ diff-algorithm=s
+ color-moved=s color-moved-ws=s no-color-moved no-color-moved-ws
+ word-diff:s word-diff-regex=s color-words:s no-renames
+ rename-empty! check ws-error-highlight=s full-index binary
+ abbrev:i break-rewrites|B:s find-renames|M:s find-copies:s
+ find-copies-harder irreversible-delete|D l=i diff-filter=s
+ S=s G=s find-object=s pickaxe-all pickaxe-regex O=s R
+ relative:s text|a ignore-cr-at-eol ignore-space-at-eol
+ ignore-space-change|b ignore-all-space|w ignore-blank-lines
+ inter-hunk-context=i function-context|W exit-code ext-diff
+ no-ext-diff textconv! src-prefix=s dst-prefix=s no-prefix
+ line-prefix=s);
+
# we generate shell completion + help using %CMD and %OPTDESC,
# see lei__complete() and PublicInbox::LeiHelp
# command => [ positional_args, 1-line description, Getopt::Long option spec ]
@@ -162,6 +179,11 @@ our %CMD = ( # sorted in order of importance/use:
qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
@lxs_opt, @c_opt ],
+'rediff' => [ '[--stdin|LOCATION...]',
+ 'regenerate a diff with different options',
+ qw(git-dir=s@ cwd! verbose|v+ color:s no-color),
+ @diff_opt, @lxs_opt, @c_opt ],
+
'add-external' => [ 'LOCATION',
'add/set priority of a publicinbox|extindex for extra matches',
qw(boost=i mirror=s no-torsocks torsocks=s inbox-version=i
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 46eea111..87083564 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -69,6 +69,12 @@ error reading $name: $!
# but no Content-Length or "From " escaping.
# "git format-patch" also generates such files by default.
$buf =~ s/\A[\r\n]*From [^\r\n]*\r?\n//s;
+
+ # a user may feed just a body: git diff | lei rediff -U9
+ if ($self->{-force_eml}) {
+ my $eml = PublicInbox::Eml->new($buf);
+ substr($buf, 0, 0) = "\n\n" if !$eml->{bdy};
+ }
$self->input_eml_cb(PublicInbox::Eml->new(\$buf), @args);
} else {
# prepare_inputs already validated $ifmt
diff --git a/lib/PublicInbox/LeiRediff.pm b/lib/PublicInbox/LeiRediff.pm
new file mode 100644
index 00000000..6c734bef
--- /dev/null
+++ b/lib/PublicInbox/LeiRediff.pm
@@ -0,0 +1,245 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# The "lei rediff" sub-command, regenerates diffs with new options
+package PublicInbox::LeiRediff;
+use strict;
+use v5.10.1;
+use parent qw(PublicInbox::IPC PublicInbox::LeiInput);
+use File::Temp 0.19 (); # 0.19 for ->newdir
+use PublicInbox::Spawn qw(spawn which);
+use PublicInbox::MsgIter qw(msg_part_text);
+use PublicInbox::ViewDiff;
+use PublicInbox::LeiBlob;
+use PublicInbox::Git qw(git_quote git_unquote);
+use PublicInbox::Import;
+use PublicInbox::LEI;
+use PublicInbox::SolverGit;
+
+sub rediff_user_cb { # called by solver when done
+ my ($res, $self) = @_;
+ my $lei = $self->{lei};
+ my $log_buf = delete $lei->{log_buf};
+ $$log_buf =~ s/^/# /sgm;
+ ref($res) eq 'ARRAY' or return $lei->child_error(1 << 8, $$log_buf);
+ $lei->qerr($$log_buf);
+ my ($git, $oid, $type, $size, $di) = @$res;
+ my $oid_want = delete $self->{cur_oid_want};
+
+ # don't try to support all the git-show(1) options for non-blob,
+ # this is just a convenience:
+ $type ne 'blob' and return $lei->err(<<EOF);
+# $oid is a $type of $size bytes in:
+# $git->{git_dir} (wanted: $oid_want)
+EOF
+ $self->{blob}->{$oid_want} = $oid;
+ push @{$self->{gits}}, $git if $git->{-tmp};
+}
+
+# returns a full blob for oid_want
+sub solve_1 ($$$) {
+ my ($self, $oid_want, $hints) = @_;
+ return if $oid_want =~ /\A0+\z/;
+ $self->{cur_oid_want} = $oid_want;
+ my $solver = bless {
+ gits => $self->{gits},
+ user_cb => \&rediff_user_cb,
+ uarg => $self,
+ inboxes => [ $self->{lxs}->locals, @{$self->{rmt}} ],
+ }, 'PublicInbox::SolverGit';
+ open my $log, '+>', \(my $log_buf = '') or die "PerlIO::scalar: $!";
+ $self->{lei}->{log_buf} = \$log_buf;
+ local $PublicInbox::DS::in_loop = 0; # waitpid synchronously
+ $solver->solve($self->{lei}->{env}, $log, $oid_want, $hints);
+ $self->{blob}->{$oid_want}; # full OID
+}
+
+sub diff_ctxq ($$) {
+ my ($self, $ctxq) = @_;
+ return unless $ctxq;
+ my $blob = $self->{blob};
+ my $ta = <<'EOM';
+reset refs/heads/A
+commit refs/heads/A
+author <a@s> 0 +0000
+committer <c@s> 0 +0000
+data 0
+EOM
+ my $tb = $ta;
+ $tb =~ tr!A!B!;
+ my $lei = $self->{lei};
+ my $wait = delete($self->{-do_done}) ? $lei->{sto}->ipc_do('done') : 0;
+ while (my ($oid_a, $oid_b, $pa, $pb) = splice(@$ctxq, 0, 4)) {
+ my $xa = $blob->{$oid_a} //= solve_1($self, $oid_a,
+ { path_b => $pa });
+ my $xb = $blob->{$oid_b} //= solve_1($self, $oid_b, {
+ oid_a => $oid_a,
+ path_a => $pa,
+ path_b => $pb
+ });
+ $ta .= "M 100644 $xa ".git_quote($pa)."\n" if $xa;
+ $tb .= "M 100644 $xb ".git_quote($pb)."\n" if $xb;
+ }
+ my $rw = $self->{gits}->[-1]; # has all known alternates
+ if (!$rw->{-tmp}) {
+ my $d = "$self->{rdtmp}/for_tree.git";
+ -d $d or PublicInbox::Import::init_bare($d);
+ my $f = "$d/objects/info/alternates"; # always overwrite
+ open my $fh, '>', $f or die "open $f: $!";
+ for my $git (@{$self->{gits}}) {
+ print $fh $git->git_path('objects'),"\n";
+ }
+ close $fh or die "close $f: $!";
+ $rw = PublicInbox::Git->new($d);
+ }
+ pipe(my ($r, $w)) or die "pipe: $!";
+ my $pid = spawn(['git', "--git-dir=$rw->{git_dir}",
+ qw(fast-import --quiet --done --date-format=raw)],
+ $lei->{env}, { 2 => $lei->{2}, 0 => $r });
+ close $r or die "close r fast-import: $!";
+ print $w $ta, "\n", $tb, "\ndone\n" or die "print fast-import: $!";
+ close $w or die "close w fast-import: $!";
+ waitpid($pid, 0);
+ die "fast-import failed: \$?=$?" if $?;
+
+ my @cmd = qw(diff);
+ my $opt = $lei->{opt};
+ push @cmd, '--'.($opt->{color} && !$opt->{'no-color'} ? '' : 'no-').
+ 'color';
+ for my $o (@PublicInbox::LEI::diff_opt) {
+ $o =~ s/\|([a-z0-9])\b//i; # remove single char short option
+ my $c = $1;
+ if ($o =~ s/=[is]@\z//) {
+ my $v = $opt->{$o} or next;
+ push @cmd, map { $c ? "-$c$_" : "--$o=$_" } @$v;
+ } elsif ($o =~ s/=[is]\z//) {
+ my $v = $opt->{$o} // next;
+ push @cmd, $c ? "-$c$v" : "--$o=$v";
+ } elsif ($o =~ s/:[is]\z//) {
+ my $v = $opt->{$o} // next;
+ push @cmd, $c ? "-$c$v" :
+ ($v eq '' ? "--$o" : "--$o=$v");
+ } elsif ($o =~ s/!\z//) {
+ my $v = $opt->{$o} // next;
+ push @cmd, $v ? "--$o" : "--no-$o";
+ } elsif ($opt->{$o}) {
+ push @cmd, $c ? "-$c" : "--$o";
+ }
+ }
+ $lei->qerr("# git @cmd");
+ push @cmd, qw(A B);
+ unshift @cmd, 'git', "--git-dir=$rw->{git_dir}";
+ $pid = spawn(\@cmd, $lei->{env}, { 2 => $lei->{2}, 1 => $lei->{1} });
+ waitpid($pid, 0);
+ $lei->child_error($?) if $?; # for git diff --exit-code
+}
+
+sub extract_oids { # Eml each_part callback
+ my ($ary, $self) = @_;
+ my ($p, undef, $idx) = @$ary;
+ $self->{lei}->out($p->header_obj->as_string, "\n");
+ my ($s, undef) = msg_part_text($p, $p->content_type || 'text/plain');
+ defined $s or return;
+ my @top = split($PublicInbox::ViewDiff::EXTRACT_DIFFS, $s);
+ undef $s;
+ my $blobs = $self->{blobs}; # blobs to resolve
+ my $ctxq;
+ while (defined(my $x = shift @top)) {
+ if (scalar(@top) >= 4 &&
+ $top[1] =~ $PublicInbox::ViewDiff::IS_OID &&
+ $top[0] =~ $PublicInbox::ViewDiff::IS_OID) {
+ my ($oid_a, $oid_b, $pa, $pb) = splice(@top, 0, 4);
+ $pa eq '/dev/null' or
+ $pa = (split(m'/', git_unquote($pa), 2))[1];
+ $pb eq '/dev/null' or
+ $pb = (split(m'/', git_unquote($pb), 2))[1];
+ $blobs->{$oid_a} //= undef;
+ $blobs->{$oid_b} //= undef;
+ push @$ctxq, $oid_a, $oid_b, $pa, $pb;
+ } elsif ($ctxq) {
+ my @out;
+ for (split(/^/sm, $x)) {
+ if (/\A-- \r?\n/s) { # email sig starts
+ push @out, $_;
+ $ctxq = diff_ctxq($self, $ctxq);
+ } elsif ($ctxq && (/\A[\+\- ]/ || /\A@@ / ||
+ # allow totally blank lines w/o leading
+ # SP, git-apply does:
+ /\A\r?\n/s)) {
+ next;
+ } else {
+ push @out, $_;
+ }
+ }
+ $self->{lei}->out(@out) if @out;
+ } else {
+ $ctxq = diff_ctxq($self, $ctxq);
+ $self->{lei}->out($x);
+ }
+ }
+ $ctxq = diff_ctxq($self, $ctxq);
+}
+
+sub input_eml_cb { # callback for all emails
+ my ($self, $eml) = @_;
+ $self->{lei}->{sto}->ipc_do('add_eml', $eml);
+ $self->{-do_done} = 1;
+ $eml->each_part(\&extract_oids, $self, 1);
+}
+
+sub lei_rediff {
+ my ($lei, @inputs) = @_;
+ $lei->_lei_store(1)->write_prepare($lei);
+ $lei->{opt}->{stdin} = 1 if !@inputs;
+ $lei->{opt}->{'in-format'} //= 'eml';
+ # maybe it's a non-email (code) blob from a coderepo
+ my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
+ if ($lei->{opt}->{cwd} // 1) {
+ my $cgd = PublicInbox::LeiBlob::get_git_dir($lei, '.');
+ unshift(@$git_dirs, $cgd) if defined $cgd;
+ }
+ return $lei->fail('no --git-dir to try') unless @$git_dirs;
+ my $lxs = $lei->lxs_prepare;
+ if ($lxs->remotes) {
+ require PublicInbox::LeiRemote;
+ $lei->{curl} //= which('curl') or return
+ $lei->fail('curl needed for', $lxs->remotes);
+ }
+ $lei->ale->refresh_externals($lxs);
+ my $self = bless {
+ -force_eml => 1, # for LeiInput->input_fh
+ lxs => $lxs,
+ }, __PACKAGE__;
+ $self->prepare_inputs($lei, \@inputs) or return;
+ my $isatty = -t $lei->{1};
+ $lei->{opt}->{color} //= $isatty;
+ $lei->start_pager if $isatty;
+ my ($op_c, $ops) = $lei->workers_start($self, 1);
+ $lei->{wq1} = $self;
+ net_merge_all_done($self) unless $lei->{auth};
+ $op_c->op_wait_event($ops);
+}
+
+sub ipc_atfork_child {
+ my ($self) = @_;
+ PublicInbox::LeiInput::input_only_atfork_child(@_);
+ my $lei = $self->{lei};
+ $lei->{1}->autoflush(1);
+ binmode $lei->{1}, ':utf8';
+ $self->{blobs} = {}; # oidhex => filename
+ $self->{rdtmp} = File::Temp->newdir('lei-rediff-XXXX', TMPDIR => 1);
+ $self->{rmt} = [ map {
+ PublicInbox::LeiRemote->new($lei, $_)
+ } $self->{lxs}->remotes ];
+ $self->{gits} = [ map {
+ PublicInbox::Git->new($lei->rel2abs($_))
+ } @{$self->{lei}->{opt}->{'git-dir'}} ];
+ $lei->{env}->{'psgi.errors'} = $lei->{2}; # ugh...
+ $lei->{env}->{TMPDIR} = $self->{rdtmp}->dirname;
+ undef;
+}
+
+no warnings 'once';
+*net_merge_all_done = \&PublicInbox::LeiInput::input_only_net_merge_all_done;
+*net_merge_all = \&PublicInbox::LeiAuth::net_merge_all;
+1;
diff --git a/t/solver_git.t b/t/solver_git.t
index 75387b2a..e566efb3 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -32,7 +32,7 @@ my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
-test_lei({tmpdir => $tmpdir}, sub {
+test_lei({tmpdir => "$tmpdir/blob"}, sub {
lei_ok('blob', '--mail', $patch2_oid, '-I', $ibx->{inboxdir},
\'--mail works for existing oid');
is($lei_out, $patch2->as_string, 'blob matches');
@@ -64,6 +64,12 @@ test_lei({tmpdir => $tmpdir}, sub {
lei_ok('blob', $v1_0_0_tag_short, '-I', $ibx->{inboxdir});
});
+test_lei({tmpdir => "$tmpdir/rediff"}, sub {
+ lei_ok(qw(rediff -q -U9 t/solve/0001-simple-mod.patch));
+ like($lei_out, qr!^\Q+++\E b/TODO\n@@ -103,9 \+103,11 @@!sm,
+ 'got more context with -U9');
+});
+
my $git = PublicInbox::Git->new($git_dir);
$ibx->{-repo_objs} = [ $git ];
my $res;
^ permalink raw reply related [relevance 30%]
* [PATCH 0/3] lei rediff fixes
@ 2021-05-05 17:49 71% Eric Wong
0 siblings, 0 replies; 200+ results
From: Eric Wong @ 2021-05-05 17:49 UTC (permalink / raw)
To: meta
1/3 was necessary anyways, and also provided a test case for
2/3. 3/3 makes things less surprising, and thankfully wasn't
difficult at all.
Eric Wong (3):
script/public-inbox-extindex: chmod +x
lei rediff: capture and regenerate file modes
lei rediff: do not automatically store patches/mails
MANIFEST | 1 +
lib/PublicInbox/LeiRediff.pm | 28 ++++++++++++++++++++--------
script/public-inbox-extindex | 1 -
t/solve/bare.patch | 8 ++++++++
t/solver_git.t | 16 ++++++++++++++++
5 files changed, 45 insertions(+), 9 deletions(-)
mode change 100644 => 100755 script/public-inbox-extindex
create mode 100644 t/solve/bare.patch
^ permalink raw reply [relevance 71%]
Results 401-600 of ~1312 next (newer) | prev (older) | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-03-03 3:53 should lei attempt to index mail outside of git? Eric Wong
2021-05-02 6:12 71% ` yes [Re: should lei attempt to index mail outside of git?] Eric Wong
2021-03-25 5:22 is "lei mark" a good name? Eric Wong
2021-03-26 1:54 71% ` Kyle Meyer
2021-03-26 9:48 68% ` Eric Wong
2021-03-28 2:21 71% ` Eric Wong
2021-03-25 8:32 71% does "lei init" even need to exist? Eric Wong
2021-03-26 1:15 71% ` Kyle Meyer
2021-03-26 4:29 71% [PATCH 0/3] lei labels support Eric Wong
2021-03-26 4:29 71% ` [PATCH 2/3] lei: _lei_store: use default even if unconfigured Eric Wong
2021-03-26 5:01 71% ` [SQUASH 4/3] lei: account for unconfigured leistore.dir Eric Wong
2021-03-26 4:29 30% ` [PATCH 3/3] lei: add some labels support Eric Wong
2021-03-26 10:31 69% ` labels for externals [was: lei labels support] Eric Wong
2021-03-26 9:51 71% [PATCH 0/4] lei minor things Eric Wong
2021-03-26 9:51 69% ` [PATCH 1/4] lei q: skip lei/store->write_prepare for JSON outputs Eric Wong
2021-03-26 9:51 90% ` [PATCH 2/4] lei: do not blindly commit to lei/store on close Eric Wong
2021-03-26 9:51 42% ` [PATCH 3/4] lei: support /dev/fd/[0-2] inputs and outputs in daemon Eric Wong
2021-03-26 9:51 71% ` [PATCH 4/4] lei mark: disallow '!' in labels Eric Wong
2021-03-27 11:45 90% [PATCH 0/4] lei blob (formerly known as "lei show") Eric Wong
2021-03-27 11:45 90% ` [PATCH 2/4] lei help: move "lei help" into LeiHelp.pm Eric Wong
2021-03-27 11:45 38% ` [PATCH 4/4] lei blob: aka "git-show-harder" for blobs Eric Wong
2021-03-27 20:20 71% ` [SQUASH] lei blob: use absolute path Eric Wong
2021-03-27 23:22 69% [PATCH] lei mark: relax label requirements Eric Wong
2021-03-28 9:01 67% [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 43% ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
2021-03-28 9:01 58% ` [PATCH 02/12] lei init: split out into separate file Eric Wong
2021-03-28 9:01 71% ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
2021-03-28 9:01 57% ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
2021-03-28 9:01 71% ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
2021-03-28 9:01 67% ` [PATCH 06/12] lei blob: some extra tests Eric Wong
2021-03-28 9:01 71% ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
2021-03-28 9:01 69% ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
2021-03-28 9:01 44% ` [PATCH 10/12] lei blob: add remote external support Eric Wong
2021-03-28 9:01 66% ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
2021-03-28 9:31 71% ` Eric Wong
2021-03-29 3:11 69% [PATCH 0/8] doc: lei manpages, round 4 Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 1/8] doc lei-q: fix typo in -tt description Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 2/8] doc lei: note --stdin shortcut in synopses Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 3/8] doc lei: drop an unnecessary to-do comment Kyle Meyer
2021-03-29 3:11 68% ` [PATCH 4/8] doc lei: don't render most to-do comments Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 5/8] doc lei: update manpages with new options Kyle Meyer
2021-03-29 3:11 30% ` [PATCH 6/8] doc lei: add manpages for new commands Kyle Meyer
2021-03-29 3:25 71% ` Eric Wong
2021-03-29 3:35 71% ` Kyle Meyer
2021-03-29 3:11 90% ` [PATCH 7/8] doc lei overview: note that lei-init is usually unnecessary Kyle Meyer
2021-03-29 3:11 71% ` [PATCH 8/8] doc lei overview: better explain routes into local store Kyle Meyer
2021-03-29 3:18 71% ` [PATCH 0/8] doc: lei manpages, round 4 Eric Wong
2021-03-29 7:08 71% [PATCH 0/3] lei input improvements Eric Wong
2021-03-29 7:08 70% ` [PATCH 2/3] lei: use IO::Uncompress::Gunzip MultiStream Eric Wong
2021-03-29 8:04 [PATCH 0/4] doc: some clarifications and warnings Eric Wong
2021-03-29 8:04 71% ` [PATCH 1/4] doc: lei q: drop NNTP from --output description Eric Wong
2021-03-29 8:04 71% ` [PATCH 2/4] doc: lei q: add warning for --output clobbering Eric Wong
2021-03-29 8:04 71% ` [PATCH 3/4] doc: lei q: clarify default output as stdout Eric Wong
2021-03-29 8:04 67% ` [PATCH 4/4] doc: lei: update description, add warnings Eric Wong
2021-03-29 17:47 71% [PATCH] lei blob: cleanup solver tmpdir on failure Eric Wong
2021-03-30 9:10 70% [PATCH] lei q: avoid redundant default setting for sort with l2m Eric Wong
2021-03-30 9:39 51% [PATCH] lei tag: rename from "lei mark" Eric Wong
2021-03-30 22:29 62% [PATCH] lei: fix IMAP auth failure handling Eric Wong
2021-03-31 0:41 71% [PATCH 0/2] lei doc: mail formats Eric Wong
2021-03-31 0:41 46% ` [PATCH 1/2] doc: add lei-mail-formats(5) manpage Eric Wong
2021-03-31 3:15 71% ` Kyle Meyer
2021-03-31 0:41 66% ` [PATCH 2/2] doc: lei-overview: favor Maildir for mutt examples Eric Wong
2021-03-31 1:53 54% [PATCH] lei blob: "--mail" disables solver, use --include/only Eric Wong
2021-03-31 23:29 62% [PATCH] script/lei: background ourselves on MUA/pager exec Eric Wong
2021-04-01 9:32 71% [PATCH 0/2] lei sucks Eric Wong
2021-04-01 9:32 55% ` [PATCH 2/2] lei sucks: sub-command to aid bug reporting Eric Wong
2021-04-01 10:12 71% [PATCH] lei: maildir: handle "forwarded" keyword as "P" Eric Wong
2021-04-01 12:10 [PATCH 0/5] quieter and less noisy Eric Wong
2021-04-01 12:10 84% ` [PATCH 2/5] lei q: reduce lei/store work for kw changes to stored mail Eric Wong
2021-04-02 9:42 57% [PATCH] lei: fix git-credential handling Eric Wong
2021-04-05 4:17 71% ` Kyle Meyer
2021-04-05 8:36 71% ` [PATCH] script/lei: waitpid for git-credential and pager Eric Wong
2021-04-05 8:59 71% ` [RFC] script/lei: don't setsid on MUA spawn Eric Wong
2021-04-05 21:26 71% ` [PATCH] script/lei: waitpid for git-credential and pager Kyle Meyer
2021-04-03 1:37 71% [PATCH 0/2] lei MUA UX fixes Eric Wong
2021-04-03 1:37 71% ` [PATCH 1/2] lei q: don't show remote progress if MUA is running Eric Wong
2021-04-03 1:37 71% ` [PATCH 2/2] lei: allow progress to non-TTY after MUA spawn Eric Wong
2021-04-03 2:24 71% [PATCH 0/6] lei auth-related fixes Eric Wong
2021-04-03 2:24 57% ` [PATCH 2/6] lei q: ensure wq workers shutdown on IMAP auth failures Eric Wong
2021-04-03 2:24 68% ` [PATCH 3/6] lei tag: fix tagging of IMAP inputs Eric Wong
2021-04-03 2:24 71% ` [PATCH 5/6] net_reader: fix read-only "lei convert" auth failures Eric Wong
2021-04-03 2:24 69% ` [PATCH 6/6] xt/lei-auth-fail: test more failure cases Eric Wong
2021-04-03 10:48 71% [PATCH 0/5] lei/store: fix epoch roll, better errors Eric Wong
2021-04-03 10:48 71% ` [PATCH 3/5] lei tag: note message mismatches on failure Eric Wong
2021-04-03 10:48 55% ` [PATCH 4/5] lei: improve handling of Message-ID-less draft messages Eric Wong
2021-04-03 10:48 43% ` [PATCH 5/5] lei/store: (more) synchronous non-fatal error output Eric Wong
2021-04-05 10:27 [PATCH 0/5] lei_to_mail fixes Eric Wong
2021-04-05 10:27 43% ` [PATCH 3/5] lei: maildir: move shard support to MdirReader Eric Wong
2021-04-05 10:27 60% ` [PATCH 5/5] lei q: fix auth IMAP --output with remote mboxrd Eric Wong
2021-04-12 17:32 68% [PATCH] lei blob: quiet "git rev-parse --git-dir" stderr w/o --cwd Eric Wong
2021-04-13 10:54 90% [PATCH 0/5] "lei q --save" + "lei up" Eric Wong
2021-04-13 10:54 32% ` [PATCH 4/5] lei q: start wiring up saved search Eric Wong
2021-04-13 11:25 71% ` Eric Wong
2021-04-13 19:13 71% ` Eric Wong
2021-04-13 10:54 75% ` [PATCH 5/5] lei: add "lei up" to complement "lei q --save" Eric Wong
2021-04-16 23:10 71% [PATCH 0/9] lei saved search usability improvements Eric Wong
2021-04-16 23:10 53% ` [PATCH 1/9] lei q: --save preserves relative time queries Eric Wong
2021-04-16 23:10 69% ` [PATCH 2/9] lei: expose share_path as a method Eric Wong
2021-04-16 23:10 69% ` [PATCH 3/9] lei: saved searches keyed only by path/URL and format Eric Wong
2021-04-16 23:10 71% ` [PATCH 6/9] lei: fix rel2abs Eric Wong
2021-04-16 23:10 50% ` [PATCH 7/9] lei up: support output destination as arg Eric Wong
2021-04-16 23:10 90% ` [PATCH 8/9] lei q --save: avoid lei.q.format Eric Wong
2021-04-16 23:10 61% ` [PATCH 9/9] lei q --save: clobber config file on repeats Eric Wong
2021-04-17 10:24 62% [PATCH] lei up: fix canonicalization of Maildirs Eric Wong
2021-04-17 19:00 63% ` [PATCH 2/] lei up: further improve Maildir canonicalization Eric Wong
2021-04-17 15:53 71% lei q: --stdin confuses --mua Kyle Meyer
2021-04-17 16:04 71% ` Kyle Meyer
2021-04-17 19:00 60% ` [PATCH] lei q: fix MUA spawn after reading query from stdin Eric Wong
2021-04-17 20:13 70% ` Kyle Meyer
2021-04-18 8:40 36% [PATCH] lei ls-search: command to list saved searches Eric Wong
2021-04-19 8:52 71% [PATCH 0/6] lei saved search improvements Eric Wong
2021-04-19 8:52 68% ` [PATCH 1/6] lei: support unlinked/missing saved searches Eric Wong
2021-04-19 8:52 59% ` [PATCH 2/6] lei q: implement import-before default for --save Eric Wong
2021-04-19 8:52 73% ` [PATCH 5/6] lei_saved_search: split "lei q --save" and "lei up" init paths Eric Wong
2021-04-19 8:52 65% ` [PATCH 6/6] lei q: --save and --augment may be combined Eric Wong
2021-04-19 23:48 71% [PATCH 0/4] "lei up --all=local" support Eric Wong
2021-04-19 23:48 71% ` [PATCH 1/4] lei up: fix help output and ARGV handling Eric Wong
2021-04-19 23:49 65% ` [PATCH 3/4] lei up: more error checking for config loading Eric Wong
2021-04-19 23:49 49% ` [PATCH 4/4] lei up: support --all=local Eric Wong
2021-04-20 7:16 71% [PATCH 0/2] lei {edit,forget}-search Eric Wong
2021-04-20 7:16 57% ` [PATCH 1/2] lei forget-search: new command to forget saved searches Eric Wong
2021-04-20 7:16 64% ` [PATCH 2/2] lei edit-search: command to tweak search parameters Eric Wong
2021-04-20 9:17 52% [PATCH] lei-sigpipe: update and move test from xt => t Eric Wong
2021-04-20 20:33 71% t/lei-daemon.t failure when PERL_INLINE_DIRECTORY is set Konstantin Ryabitsev
2021-04-20 20:38 71% ` Eric Wong
2021-04-20 21:37 68% ` Konstantin Ryabitsev
2021-04-20 22:06 71% ` [PATCH] t/lei-daemon: skip inaccessible socket test as root Eric Wong
2021-04-21 15:03 71% ` Konstantin Ryabitsev
2021-04-21 10:03 46% [PATCH] doc: add lei_design_notes.txt and lei-store-format(5) Eric Wong
2021-04-21 18:36 61% [PATCH] lei: share common *done_wait callbacks Eric Wong
2021-04-21 23:50 50% [PATCH] lei: flesh out `forwarded' kw support for Maildir and IMAP Eric Wong
2021-04-22 9:08 71% [PATCH 0/3] lei import: network sync things Eric Wong
2021-04-22 9:08 70% ` [PATCH 1/3] imap_tracker: prepare for use with lei Eric Wong
2021-04-22 9:08 51% ` [PATCH 2/3] lei import: --incremental default for NNTP and IMAP Eric Wong
2021-04-22 9:08 59% ` [PATCH 3/3] lei import|convert: drop --no-kw aliases Eric Wong
2021-04-22 9:44 63% [PATCH] lei: XDG_RUNTIME_DIR=/dev/null disables daemon mode Eric Wong
2021-04-23 1:45 71% [PATCH 0/2] "lei up" surprise reduction fixes Eric Wong
2021-04-23 1:45 50% ` [PATCH 1/2] lei: saved searches support --dedupe=<mid|oid> Eric Wong
2021-04-23 1:45 55% ` [PATCH 2/2] lei up: support symlinked pathnames Eric Wong
2021-04-23 11:22 40% [PATCH] lei import: support adding keywords and labels on import Eric Wong
2021-04-24 9:28 89% [PATCH 0/7] lei sync preparations, "lei inspect" Eric Wong
2021-04-24 9:28 66% ` [PATCH 2/7] t/lei_to_mail: split "lei import" test $HOME directory Eric Wong
2021-04-24 9:28 27% ` [PATCH 7/7] lei import: keep sync info for Maildir and IMAP folders Eric Wong
2021-04-26 8:43 66% [PATCH] lei p2q: exit with failure if format-patch fails Eric Wong
2021-04-26 16:44 70% lei-managed pseudo mailing lists Konstantin Ryabitsev
2021-04-26 17:37 70% ` Eric Wong
2021-04-26 18:20 65% ` Konstantin Ryabitsev
2021-04-26 18:47 71% ` Eric Wong
2021-04-26 19:46 68% ` Konstantin Ryabitsev
2021-04-26 20:34 71% ` Eric Wong
2021-04-27 11:07 64% [PATCH 0/5] lei lcat - local cat (not lolcat :P) Eric Wong
2021-04-27 11:07 56% ` [PATCH 1/5] lei: add "ls-sync" command for listing sync folders Eric Wong
2021-04-27 11:07 61% ` [PATCH 2/5] lei blob: support retrieving attachments via $OID:$IDX Eric Wong
2021-04-27 11:07 49% ` [PATCH 3/5] lei: standardize on _lei_wq_eof callback for workers Eric Wong
2021-04-27 11:07 38% ` [PATCH 4/5] lei lcat: extract Message-IDs from URLs and show them Eric Wong
2021-04-27 11:07 32% ` [PATCH 5/5] lei q + lcat: support --format=text output Eric Wong
2021-04-28 4:51 90% [PATCH 0/3] doc: lei updates around lei-q Eric Wong
2021-04-28 4:51 64% ` [PATCH 1/3] doc: lei: use /tmp for search results pathnames Eric Wong
2021-04-28 4:51 54% ` [PATCH 2/3] doc: lei q: split =item aliases onto separate lines Eric Wong
2021-04-29 1:39 71% ` Kyle Meyer
2021-04-29 1:57 71% ` Eric Wong
2021-04-28 4:51 63% ` [PATCH 3/3] doc: lei q: split --output and --format, note "text" Eric Wong
2021-04-28 7:51 68% [PATCH 00/11] lei: misc fixes, more lcat color support Eric Wong
2021-04-28 7:51 71% ` [PATCH 01/11] t/lei-p2q: add diagnostics Eric Wong
2021-04-28 7:51 52% ` [PATCH 02/11] tests: restore CWD with "lei -C" and run_script Eric Wong
2021-04-28 7:51 71% ` [PATCH 05/11] lei-daemon: note FD count mismatch to client Eric Wong
2021-04-28 7:52 59% ` [PATCH 07/11] lei: quiet down Eml-related warnings consistently Eric Wong
2021-04-28 7:52 44% ` [PATCH 08/11] lei: simple WQ workers use {wq1} field Eric Wong
2021-04-28 7:52 60% ` [PATCH 11/11] lei (lcat|q): support --no-color and --color Eric Wong
2021-04-28 19:37 [PATCH 0/2] "make check-run" fixed Eric Wong
2021-04-28 19:37 63% ` [PATCH 2/2] lei: avoid close(STD{IN,OUT,ERR}) in oneshot mode Eric Wong
2021-04-29 9:46 69% [PATCH 0/4] some lei synchronization work Eric Wong
2021-04-29 9:46 33% ` [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more Eric Wong
2021-04-29 9:46 61% ` [PATCH 3/4] lei import: support UIDVALIDITY in IMAP URL Eric Wong
2021-04-29 9:46 57% ` [PATCH 4/4] lei import: support shell completion of known folders Eric Wong
2021-04-30 9:24 67% [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
2021-04-30 9:24 71% ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
2021-04-30 9:24 71% ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
2021-04-30 9:24 70% ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
2021-04-30 9:24 45% ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
2021-05-01 6:21 70% [PATCH 0/5] lei: more UI/UX tweaks Eric Wong
2021-05-01 6:21 61% ` [PATCH 1/5] xt/lei-onion-convert: test for NNTP+IMAP onions Eric Wong
2021-05-01 6:21 55% ` [PATCH 2/5] lei <q|up>: distinguish between mset and l2m counts Eric Wong
2021-05-01 6:21 71% ` [PATCH 3/5] lei_saved_search: fix excess indent for first lei.q entry Eric Wong
2021-05-01 6:21 56% ` [PATCH 4/5] lei: rename ls-sync to ls-mail-sync Eric Wong
2021-05-01 19:29 59% ` [PATCH 6/5] lei import: fix --mail-sync handling in LeiInput Eric Wong
2021-05-01 6:21 75% ` [PATCH 5/5] lei edit-search: support relocating lei.q.output Eric Wong
2021-05-02 6:05 70% [PATCH 0/6] lei: more steps towards kw sync Eric Wong
2021-05-02 6:05 70% ` [PATCH 1/6] lei <q|up>: combine written/results into one line Eric Wong
2021-05-02 6:05 66% ` [PATCH 2/6] lei_input: common net_merge_all_done for lei <import|tag> Eric Wong
2021-05-02 6:05 53% ` [PATCH 4/6] lei: simplify workers_start API Eric Wong
2021-05-02 6:05 49% ` [PATCH 6/6] lei <q|up>: writes to Maildirs and IMAP use mail-sync Eric Wong
2021-05-03 20:57 57% [PATCH] lei up: fix dedupe with remote externals on Maildir + IMAP Eric Wong
2021-05-04 4:45 [PATCH 0/5] Minor ls-mail-sync fixes Kyle Meyer
2021-05-04 4:45 71% ` [PATCH 1/5] lei ls-mail-sync: update reference to ls-sync Kyle Meyer
2021-05-04 4:45 71% ` [PATCH 2/5] lei ls-mail-sync: drop repeated -z/0 option Kyle Meyer
2021-05-04 4:45 71% ` [PATCH 3/5] lei ls-mail-sync: accept a filter Kyle Meyer
2021-05-04 4:45 68% ` [PATCH 4/5] lei ls-mail-sync: fix handling of non-wildcard filters Kyle Meyer
2021-05-04 5:14 67% ` Eric Wong
2021-05-04 4:45 71% ` [PATCH 5/5] lei: add help output for --invert match Kyle Meyer
2021-05-04 5:24 66% [PATCH] lei: fix mail_sync.qlite3 folder names for NNTP Eric Wong
2021-05-04 5:26 71% ` s/qlite3/s&/ [was: [PATCH] lei: fix ... folder names for NNTP] Eric Wong
2021-05-04 9:49 26% [PATCH] lei index: new command to index mail w/o git storage Eric Wong
2021-05-05 10:46 71% [PATCH 0/2] lei rediff + solver-related fix Eric Wong
2021-05-05 10:46 30% ` [PATCH 1/2] lei rediff: regenerate diffs from stdin Eric Wong
2021-05-05 10:46 87% ` [PATCH 2/2] lei blob: support "lei index"-ed mail Eric Wong
2021-05-05 17:49 71% [PATCH 0/3] lei rediff fixes Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).