user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* [PATCH 00/12] lei blob and some yak-shaving
@ 2021-03-28  9:01 Eric Wong
  2021-03-28  9:01 ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
                   ` (11 more replies)
  0 siblings, 12 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

"lei blob" manages to work with HTTPS and HTTP .onion endpoints.

Eric Wong (12):
  lei: simplify PktOp callers
  lei init: split out into separate file
  lei blob: dclose if already failed
  lei blob: support --no-mail switch
  lei blob: fail early if no git dirs
  lei blob: some extra tests
  lei help: show "NAME=VALUE" properly for -c
  lei blob: flesh out help text
  t/lei_store: ensure LeiSearch responds to ->isrch
  lei blob: add remote external support
  lei: drop coderepo placeholders, submodule TODO
  treewide: shorten temporary filename

 MANIFEST                       |  2 +
 lib/PublicInbox/LEI.pm         | 67 +++++++---------------------
 lib/PublicInbox/LeiBlob.pm     | 47 +++++++++++++-------
 lib/PublicInbox/LeiConvert.pm  |  4 +-
 lib/PublicInbox/LeiHelp.pm     |  2 +-
 lib/PublicInbox/LeiImport.pm   |  4 +-
 lib/PublicInbox/LeiInit.pm     | 41 +++++++++++++++++
 lib/PublicInbox/LeiMark.pm     |  4 +-
 lib/PublicInbox/LeiMirror.pm   |  4 +-
 lib/PublicInbox/LeiOverview.pm |  2 +-
 lib/PublicInbox/LeiP2q.pm      |  4 +-
 lib/PublicInbox/LeiRemote.pm   | 81 ++++++++++++++++++++++++++++++++++
 lib/PublicInbox/LeiXSearch.pm  |  8 ++--
 lib/PublicInbox/Msgmap.pm      |  2 +-
 lib/PublicInbox/PktOp.pm       | 20 ++++++---
 lib/PublicInbox/SolverGit.pm   |  2 +-
 lib/PublicInbox/TestCommon.pm  |  2 +-
 lib/PublicInbox/V2Writable.pm  |  6 +--
 lib/PublicInbox/Xapcmd.pm      |  9 ++--
 script/public-inbox-edit       |  2 +-
 script/public-inbox-init       |  2 +-
 scripts/ssoma-replay           |  2 +-
 t/check-www-inbox.perl         |  2 +-
 t/inbox.t                      |  2 +-
 t/lei_store.t                  |  1 +
 t/nodatacow.t                  |  2 +-
 t/solver_git.t                 | 40 +++++++++++++++--
 27 files changed, 254 insertions(+), 110 deletions(-)
 create mode 100644 lib/PublicInbox/LeiInit.pm
 create mode 100644 lib/PublicInbox/LeiRemote.pm

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 01/12] lei: simplify PktOp callers
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 02/12] lei init: split out into separate file Eric Wong
                   ` (10 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

Provide a consistent ->op_wait_event method instead of
forcing callers to loop (or not) at each callsite.
This also avoid a leak possibility by avoiding circular
references.
---
 lib/PublicInbox/LEI.pm        | 11 +++++------
 lib/PublicInbox/LeiBlob.pm    |  4 ++--
 lib/PublicInbox/LeiConvert.pm |  4 ++--
 lib/PublicInbox/LeiImport.pm  |  4 ++--
 lib/PublicInbox/LeiMark.pm    |  4 ++--
 lib/PublicInbox/LeiMirror.pm  |  4 ++--
 lib/PublicInbox/LeiP2q.pm     |  4 ++--
 lib/PublicInbox/LeiXSearch.pm |  8 +++-----
 lib/PublicInbox/PktOp.pm      | 20 +++++++++++++++-----
 9 files changed, 35 insertions(+), 28 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 478912cd..9cacb142 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -494,11 +494,11 @@ sub _delete_pkt_op { # OnDestroy callback to prevent leaks on die
 }
 
 sub pkt_op_pair {
-	my ($self, $ops) = @_;
+	my ($self) = @_;
 	require PublicInbox::OnDestroy;
 	require PublicInbox::PktOp;
 	my $end = PublicInbox::OnDestroy->new($$, \&_delete_pkt_op, $self);
-	@$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair($ops);
+	@$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair;
 	$end;
 }
 
@@ -512,14 +512,13 @@ sub workers_start {
 		($ops ? %$ops : ()),
 	};
 	$ops->{''} //= [ \&dclose, $lei ];
-	my $end = $lei->pkt_op_pair($ops);
+	my $end = $lei->pkt_op_pair;
 	$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
 	delete $lei->{pkt_op_p};
-	my $op = delete $lei->{pkt_op_c};
+	my $op_c = delete $lei->{pkt_op_c};
 	@$end = ();
 	$lei->event_step_init;
-	# oneshot needs $op, daemon-mode uses DS->EventLoop to handle $op
-	$lei->{oneshot} ? $op : undef;
+	($op_c, $ops);
 }
 
 sub _help {
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 2facbad3..97747220 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -103,12 +103,12 @@ sub lei_blob {
 	my $lxs = $lei->lxs_prepare or return;
 	require PublicInbox::SolverGit;
 	my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
-	my $op = $lei->workers_start($self, 'lei_solve', 1,
+	my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
 		{ '' => [ \&sol_done, $lei ] });
 	$lei->{sol} = $self;
 	$self->wq_io_do('do_solve_blob', []);
 	$self->wq_close(1);
-	while ($op && $op->{sock}) { $op->event_step }
+	$op_c->op_wait_event($ops);
 }
 
 sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 083ecc33..5d0adb14 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -53,11 +53,11 @@ sub lei_convert { # the main "lei convert" method
 	my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
 	$lei->{opt}->{augment} = 1 if $devfd < 0;
 	$self->prepare_inputs($lei, \@inputs) or return;
-	my $op = $lei->workers_start($self, 'lei_convert', 1);
+	my ($op_c, $ops) = $lei->workers_start($self, 'lei_convert', 1);
 	$lei->{cnv} = $self;
 	$self->wq_io_do('do_convert', []);
 	$self->wq_close(1);
-	while ($op && $op->{sock}) { $op->event_step }
+	$op_c->op_wait_event($ops);
 }
 
 sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 7c5b7d09..803b5cda 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -76,11 +76,11 @@ sub lei_import { # the main "lei import" method
 	my $ops = { '' => [ \&import_done, $lei ] };
 	$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
 	$self->{-wq_nr_workers} = $j // 1; # locked
-	my $op = $lei->workers_start($self, 'lei_import', undef, $ops);
+	my ($op_c, undef) = $lei->workers_start($self, 'lei_import', $j, $ops);
 	$lei->{imp} = $self;
 	$self->wq_io_do('input_stdin', []) if $self->{0};
 	net_merge_complete($self) unless $lei->{auth};
-	while ($op && $op->{sock}) { $op->event_step }
+	$op_c->op_wait_event($ops);
 }
 
 no warnings 'once';
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 34846b84..6e611318 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -116,11 +116,11 @@ sub lei_mark { # the "lei mark" method
 	my $ops = { '' => [ \&mark_done, $lei ] };
 	$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
 	$self->{vmd_mod} = $vmd_mod;
-	my $op = $lei->workers_start($self, 'lei_mark', 1, $ops);
+	my ($op_c, undef) = $lei->workers_start($self, 'lei_mark', 1, $ops);
 	$lei->{mark} = $self;
 	$self->wq_io_do('input_stdin', []) if $self->{0};
 	net_merge_complete($self) unless $lei->{auth};
-	while ($op && $op->{sock}) { $op->event_step }
+	$op_c->op_wait_event($ops);
 }
 
 sub note_missing {
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index c83386c6..89574d28 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -282,13 +282,13 @@ sub start {
 	require PublicInbox::Inbox;
 	require PublicInbox::Admin;
 	require PublicInbox::InboxWritable;
-	my $op = $lei->workers_start($self, 'lei_mirror', 1, {
+	my ($op, $ops) = $lei->workers_start($self, 'lei_mirror', 1, {
 		'' => [ \&mirror_done, $lei ]
 	});
 	$lei->{mrr} = $self;
 	$self->wq_io_do('do_mirror', []);
 	$self->wq_close(1);
-	while ($op && $op->{sock}) { $op->event_step }
+	$op->op_wait_event($ops);
 }
 
 sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 25f63a10..a8a3dd2c 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -185,11 +185,11 @@ sub lei_p2q { # the "lei patch-to-query" entry point
 	} else {
 		$self->{input} = $input;
 	}
-	my $op = $lei->workers_start($self, 'lei_p2q', 1);
+	my ($op, $ops) = $lei->workers_start($self, 'lei_p2q', 1);
 	$lei->{p2q} = $self;
 	$self->wq_io_do('do_p2q', []);
 	$self->wq_close(1);
-	while ($op && $op->{sock}) { $op->event_step }
+	$op->op_wait_event($ops);
 }
 
 sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index b41daffe..1a194f1c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -427,7 +427,7 @@ sub do_query {
 		'incr_start_query' => [ \&incr_start_query, $self, $l2m ],
 	};
 	$lei->{auth}->op_merge($ops, $l2m) if $l2m && $lei->{auth};
-	my $end = $lei->pkt_op_pair($ops);
+	my $end = $lei->pkt_op_pair;
 	$lei->{1}->autoflush(1);
 	$lei->start_pager if delete $lei->{need_pager};
 	$lei->{ovv}->ovv_begin($lei);
@@ -445,7 +445,7 @@ sub do_query {
 	}
 	$self->wq_workers_start('lei_xsearch', undef,
 				$lei->oldset, { lei => $lei });
-	my $op = delete $lei->{pkt_op_c};
+	my $op_c = delete $lei->{pkt_op_c};
 	delete $lei->{pkt_op_p};
 	@$end = ();
 	$self->{threads} = $lei->{opt}->{threads};
@@ -455,9 +455,7 @@ sub do_query {
 		start_query($self);
 	}
 	$lei->event_step_init; # wait for shutdowns
-	if ($lei->{oneshot}) {
-		while ($op->{sock}) { $op->event_step }
-	}
+	$op_c->op_wait_event($ops);
 }
 
 sub add_uri {
diff --git a/lib/PublicInbox/PktOp.pm b/lib/PublicInbox/PktOp.pm
index 5d8e78ea..c3221735 100644
--- a/lib/PublicInbox/PktOp.pm
+++ b/lib/PublicInbox/PktOp.pm
@@ -16,21 +16,23 @@ use PublicInbox::IPC qw(ipc_freeze ipc_thaw);
 our @EXPORT_OK = qw(pkt_do);
 
 sub new {
-	my ($cls, $r, $ops) = @_;
-	my $self = bless { sock => $r, ops => $ops }, $cls;
+	my ($cls, $r) = @_;
+	my $self = bless { sock => $r }, $cls;
 	if ($PublicInbox::DS::in_loop) { # iff using DS->EventLoop
 		$r->blocking(0);
 		$self->SUPER::new($r, EPOLLIN|EPOLLET);
+	} else {
+		$self->{blocking} = 1;
 	}
 	$self;
 }
 
 # returns a blessed object as the consumer, and a GLOB/IO for the producer
 sub pair {
-	my ($cls, $ops) = @_;
+	my ($cls) = @_;
 	my ($c, $p);
 	socketpair($c, $p, AF_UNIX, SOCK_SEQPACKET, 0) or die "socketpair: $!";
-	(new($cls, $c, $ops), $p);
+	(new($cls, $c), $p);
 }
 
 sub pkt_do { # for the producer to trigger event_step in consumer
@@ -41,7 +43,7 @@ sub pkt_do { # for the producer to trigger event_step in consumer
 sub close {
 	my ($self) = @_;
 	my $c = $self->{sock} or return;
-	$c->blocking ? delete($self->{sock}) : $self->SUPER::close;
+	$self->{blocking} ? delete($self->{sock}) : $self->SUPER::close;
 }
 
 sub event_step {
@@ -73,4 +75,12 @@ sub event_step {
 	}
 }
 
+# call this when we're ready to wait on events,
+# returns immediately if non-blocking
+sub op_wait_event {
+	my ($self, $ops) = @_;
+	$self->{ops} = $ops;
+	while ($self->{blocking} && $self->{sock}) { event_step($self) }
+}
+
 1;

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 02/12] lei init: split out into separate file
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
  2021-03-28  9:01 ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
                   ` (9 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

This is a rarely-needed command, so keep it separate file
so it's easier-to-find and maybe saves a bit of RAM.
---
 MANIFEST                   |  1 +
 lib/PublicInbox/LEI.pm     | 32 -----------------------------
 lib/PublicInbox/LeiInit.pm | 41 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 42 insertions(+), 32 deletions(-)
 create mode 100644 lib/PublicInbox/LeiInit.pm

diff --git a/MANIFEST b/MANIFEST
index 64b3626f..9048b900 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -188,6 +188,7 @@ lib/PublicInbox/LeiDedupe.pm
 lib/PublicInbox/LeiExternal.pm
 lib/PublicInbox/LeiHelp.pm
 lib/PublicInbox/LeiImport.pm
+lib/PublicInbox/LeiInit.pm
 lib/PublicInbox/LeiInput.pm
 lib/PublicInbox/LeiLsLabel.pm
 lib/PublicInbox/LeiMark.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9cacb142..fdb0bbcf 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -744,38 +744,6 @@ sub lei_config {
 	x_it($self, $?) if $?;
 }
 
-sub lei_init {
-	my ($self, $dir) = @_;
-	my $cfg = _lei_cfg($self, 1);
-	my $cur = $cfg->{'leistore.dir'};
-	$dir //= store_path($self);
-	$dir = rel2abs($self, $dir);
-	my @cur = stat($cur) if defined($cur);
-	$cur = File::Spec->canonpath($cur // $dir);
-	my @dir = stat($dir);
-	my $exists = "# leistore.dir=$cur already initialized" if @dir;
-	if (@cur) {
-		if ($cur eq $dir) {
-			_lei_store($self, 1)->done;
-			return qerr($self, $exists);
-		}
-
-		# some folks like symlinks and bind mounts :P
-		if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
-			lei_config($self, 'leistore.dir', $dir);
-			_lei_store($self, 1)->done;
-			return qerr($self, "$exists (as $cur)");
-		}
-		return fail($self, <<"");
-E: leistore.dir=$cur already initialized and it is not $dir
-
-	}
-	lei_config($self, 'leistore.dir', $dir);
-	_lei_store($self, 1)->done;
-	$exists //= "# leistore.dir=$dir newly initialized";
-	return qerr($self, $exists);
-}
-
 sub lei_daemon_pid { puts shift, $$ }
 
 sub lei_daemon_kill {
diff --git a/lib/PublicInbox/LeiInit.pm b/lib/PublicInbox/LeiInit.pm
new file mode 100644
index 00000000..c6c0c01b
--- /dev/null
+++ b/lib/PublicInbox/LeiInit.pm
@@ -0,0 +1,41 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# for the "lei init" command, not sure if it's even needed...
+package PublicInbox::LeiInit;
+use v5.10.1;
+use File::Spec;
+
+sub lei_init {
+	my ($self, $dir) = @_;
+	my $cfg = $self->_lei_cfg(1);
+	my $cur = $cfg->{'leistore.dir'};
+	$dir //= $self->store_path;
+	$dir = $self->rel2abs($dir);
+	my @cur = stat($cur) if defined($cur);
+	$cur = File::Spec->canonpath($cur // $dir);
+	my @dir = stat($dir);
+	my $exists = "# leistore.dir=$cur already initialized" if @dir;
+	if (@cur) {
+		if ($cur eq $dir) {
+			$self->_lei_store(1)->done;
+			return $self->qerr($exists);
+		}
+
+		# some folks like symlinks and bind mounts :P
+		if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
+			$self->lei_config('leistore.dir', $dir);
+			$self->_lei_store(1)->done;
+			return $self->qerr("$exists (as $cur)");
+		}
+		return $self->fail(<<"");
+E: leistore.dir=$cur already initialized and it is not $dir
+
+	}
+	$self->lei_config('leistore.dir', $dir);
+	$self->_lei_store(1)->done;
+	$exists //= "# leistore.dir=$dir newly initialized";
+	$self->qerr($exists);
+}
+
+1;

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 03/12] lei blob: dclose if already failed
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
  2021-03-28  9:01 ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
  2021-03-28  9:01 ` [PATCH 02/12] lei init: split out into separate file Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
                   ` (8 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

We must close the socket to trigger pager exit if blob
reconstruction fails.  Not sure how to test this in the
test suite...
---
 lib/PublicInbox/LeiBlob.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 97747220..9b4c4f30 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -19,7 +19,7 @@ sub sol_done_wait { # dwaitpid callback
 
 sub sol_done { # EOF callback for main daemon
 	my ($lei) = @_;
-	my $sol = delete $lei->{sol} or return;
+	my $sol = delete $lei->{sol} // return $lei->dclose; # already failed
 	$sol->wq_wait_old(\&sol_done_wait, $lei);
 }
 

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 04/12] lei blob: support --no-mail switch
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (2 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
                   ` (7 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

It's possible for a abbreviated OID to be resolved unambiguously
to an email before we attempt to look at externals via xsearch;
so provide a way for a user to force searching coderepos.

If hints (--oid-a, --path-a, --path-b) are present, we'll
assume --no-mail by default, otherwise we'll assume the
user wants to look through mail for a matching blob.
---
 lib/PublicInbox/LEI.pm     |  4 ++--
 lib/PublicInbox/LeiBlob.pm | 23 +++++++++++++----------
 t/solver_git.t             |  3 +++
 3 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fdb0bbcf..bad7fad9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -136,8 +136,8 @@ our %CMD = ( # sorted in order of importance/use:
 	import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
 	opt_dash('limit|n=i', '[0-9]+') ],
 
-'blob' => [ 'OID', 'display a git blob object, solving if necessary',
-	qw(git-dir=s@ cwd! verbose|v+ oid-a|A=s path-a|a=s path-b|b=s),
+'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
+	qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
 	@lxs_opt, @c_opt ],
 
 'add-external' => [ 'LOCATION',
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 9b4c4f30..4bd86253 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -8,7 +8,6 @@ use v5.10.1;
 use parent qw(PublicInbox::IPC);
 use PublicInbox::Spawn qw(spawn popen_rd);
 use PublicInbox::DS;
-use PublicInbox::Eml;
 
 sub sol_done_wait { # dwaitpid callback
 	my ($arg, $pid) = @_;
@@ -85,18 +84,22 @@ sub do_solve_blob { # via wq_do
 sub lei_blob {
 	my ($lei, $blob) = @_;
 	$lei->start_pager if -t $lei->{1};
+	my $opt = $lei->{opt};
+	my $has_hints = grep(defined, @$opt{qw(oid-a path-a path-b)});
 
-	# first, see if it's a blob returned by "lei q" JSON output:
-	my $rdr = { 1 => $lei->{1} };
-	open $rdr->{2}, '>', '/dev/null' or die "open: $!";
-	my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
-			'cat-file', 'blob', $blob ];
-	waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
-	return if $? == 0;
+	# first, see if it's a blob returned by "lei q" JSON output:k
+	if ($opt->{mail} // ($has_hints ? 0 : 1)) {
+		my $rdr = { 1 => $lei->{1} };
+		open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+		my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
+				'cat-file', 'blob', $blob ];
+		waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+		return if $? == 0;
+	}
 
 	# maybe it's a non-email (code) blob from a coderepo
-	my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
-	if ($lei->{opt}->{'cwd'} //= 1) {
+	my $git_dirs = $opt->{'git-dir'} //= [];
+	if ($opt->{'cwd'} // 1) {
 		my $cgd = get_git_dir('.');
 		unshift(@$git_dirs, $cgd) if defined $cgd;
 	}
diff --git a/t/solver_git.t b/t/solver_git.t
index 22714ae5..7bf3ba21 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -34,6 +34,9 @@ test_lei({tmpdir => $tmpdir}, sub {
 	lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
 	is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
 		$expect, 'blob contents output');
+	my $prev = $lei_out;
+	lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
+	is($lei_out, $prev, '--no-mail works');
 
 	# fallbacks
 	lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 05/12] lei blob: fail early if no git dirs
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (3 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 06/12] lei blob: some extra tests Eric Wong
                   ` (6 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

This avoids triggering a "BUG:" message in the solver code.
---
 lib/PublicInbox/LeiBlob.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 4bd86253..f44d8af1 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -103,6 +103,7 @@ sub lei_blob {
 		my $cgd = get_git_dir('.');
 		unshift(@$git_dirs, $cgd) if defined $cgd;
 	}
+	return $lei->fail('no --git-dir to try') unless @$git_dirs;
 	my $lxs = $lei->lxs_prepare or return;
 	require PublicInbox::SolverGit;
 	my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 06/12] lei blob: some extra tests
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (4 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
                   ` (5 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

Most of it already gets tested since most of the logic is in
SolverGit, but make sure it's all wired up properly to lei.
---
 t/solver_git.t | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/t/solver_git.t b/t/solver_git.t
index 7bf3ba21..6d4b93c7 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -29,6 +29,7 @@ my $ibx = create_inbox 'v2', version => 2,
 my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
 my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
 my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
+my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
 
 test_lei({tmpdir => $tmpdir}, sub {
 	lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
@@ -37,6 +38,25 @@ test_lei({tmpdir => $tmpdir}, sub {
 	my $prev = $lei_out;
 	lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
 	is($lei_out, $prev, '--no-mail works');
+	ok(!lei(qw(blob -I), $ibx->{inboxdir}, $non_existent),
+			'non-existent blob fails');
+	SKIP: {
+		skip '/.git exists', 1 if -e '/.git';
+		require PublicInbox::OnDestroy;
+		opendir my $dh, '.' or xbail "opendir: $!";
+		my $end = PublicInbox::OnDestroy->new($$, sub {
+			chdir $dh or xbail "chdir: $!";
+		});
+		lei_ok(qw(-C / blob 69df7d5 -I), $ibx->{inboxdir},
+			"--git-dir=$git_dir");
+		is($lei_out, $prev, '--git-dir works');
+
+		ok(!lei(qw(-C / blob --no-cwd 69df7d5 -I), $ibx->{inboxdir}),
+			'--no-cwd works');
+
+		ok(!lei(qw(-C / blob -I), $ibx->{inboxdir}, $non_existent),
+			'non-existent blob fails');
+	}
 
 	# fallbacks
 	lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
@@ -163,7 +183,6 @@ EOF
 	close $cfgfh or die;
 	my $cfg = PublicInbox::Config->new($cfgpath);
 	my $www = PublicInbox::WWW->new($cfg);
-	my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
 	my $client = sub {
 		my ($cb) = @_;
 		my $mid = '20190401081523.16213-1-BOFH@YHBT.net';

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (5 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 06/12] lei blob: some extra tests Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
                   ` (4 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

---
 lib/PublicInbox/LeiHelp.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LeiHelp.pm b/lib/PublicInbox/LeiHelp.pm
index 9c1b30a1..fa0e7866 100644
--- a/lib/PublicInbox/LeiHelp.pm
+++ b/lib/PublicInbox/LeiHelp.pm
@@ -49,7 +49,7 @@ sub call {
 			length($_) > 1 ? push(@l, "--$_") : push(@s, "-$_");
 		}
 		if (!scalar(@vals)) { # no args 'threads|t'
-		} elsif ($arg_vals =~ s/\A([A-Z_]+)\b//) { # "NAME"
+		} elsif ($arg_vals =~ s/\A([A-Z_=]+)\b//) { # "NAME"
 			$vals[1] = $1;
 		} else {
 			$vals[1] = uc(substr($l[0], 2)); # "--type" => "TYPE"

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 08/12] lei blob: flesh out help text
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (6 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 09/12] t/lei_store: ensure LeiSearch responds to ->isrch Eric Wong
                   ` (3 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

This means "lei blob" gets shell completion, too.
---
 lib/PublicInbox/LEI.pm | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index bad7fad9..a4f4e58c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -249,7 +249,12 @@ my %OPTDESC = (
 		"and\xa0'[]'\x{a0}ranges",
 'verbose|v+' => 'be more verbose',
 'external!' => 'do not use externals',
-'solve!' => 'do not attempt to reconstruct blobs from emails',
+'mail!' => 'do not look in mail storage for OID',
+'cwd!' => 'do not look in git repo of current working directory',
+'oid-a|A=s' => 'pre-image OID',
+'path-a|a=s' => 'pre-image pathname associated with OID',
+'path-b|b=s' => 'post-image pathname associated with OID',
+'git-dir=s@' => 'additional git repository to scan',
 'torsocks=s' => ['VAL|auto|no|yes',
 		'whether or not to wrap git and curl commands with torsocks'],
 'no-torsocks' => 'alias for --torsocks=no',
@@ -786,7 +791,7 @@ sub lei__complete {
 			if (s/[:=].+\z//) { # req/optional args, e.g output|o=i
 			} elsif (s/\+\z//) { # verbose|v+
 			} elsif (s/!\z//) {
-				# negation: solve! => no-solve|solve
+				# negation: mail! => no-mail|mail
 				s/([\w\-]+)/$1|no-$1/g
 			}
 			map {

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 09/12] t/lei_store: ensure LeiSearch responds to ->isrch
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (7 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 10/12] lei blob: add remote external support Eric Wong
                   ` (2 subsequent siblings)
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

This is needed for SolverGit, at least, and maybe other
bits we share with PSGI in lei.
---
 t/lei_store.t | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/lei_store.t b/t/lei_store.t
index 024ff527..db94f6da 100644
--- a/t/lei_store.t
+++ b/t/lei_store.t
@@ -23,6 +23,7 @@ is($sto->add_eml($eml), undef, 'idempotent');
 $sto->done;
 {
 	my $es = $sto->search;
+	ok($es->can('isrch'), ref($es). ' can ->isrch (for SolverGit)');
 	my $msgs = $es->over->query_xover(0, 1000);
 	is(scalar(@$msgs), 1, 'one message');
 	is($msgs->[0]->{blob}, $smsg->{blob}, 'blob matches');

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 10/12] lei blob: add remote external support
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (8 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 09/12] t/lei_store: ensure LeiSearch responds to ->isrch Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:01 ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
  2021-03-28  9:01 ` [PATCH 12/12] treewide: shorten temporary filename Eric Wong
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

Introduce a new LeiRemote wrapper to provide an internal API
which SolverGit expects.  This lets us use HTTP/HTTPS endpoints
to reconstruct blobs off patches as we would with local
endpoints, just more slowly...
---
 MANIFEST                     |  1 +
 lib/PublicInbox/LEI.pm       |  2 +-
 lib/PublicInbox/LeiBlob.pm   | 16 +++++--
 lib/PublicInbox/LeiRemote.pm | 81 ++++++++++++++++++++++++++++++++++++
 t/solver_git.t               | 16 ++++++-
 5 files changed, 110 insertions(+), 6 deletions(-)
 create mode 100644 lib/PublicInbox/LeiRemote.pm

diff --git a/MANIFEST b/MANIFEST
index 9048b900..913ce55c 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -196,6 +196,7 @@ lib/PublicInbox/LeiMirror.pm
 lib/PublicInbox/LeiOverview.pm
 lib/PublicInbox/LeiP2q.pm
 lib/PublicInbox/LeiQuery.pm
+lib/PublicInbox/LeiRemote.pm
 lib/PublicInbox/LeiSearch.pm
 lib/PublicInbox/LeiStore.pm
 lib/PublicInbox/LeiToMail.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a4f4e58c..a94941a9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -121,7 +121,7 @@ sub index_opt {
 
 my @c_opt = qw(c=s@ C=s@ quiet|q);
 my @lxs_opt = (qw(remote! local! external! include|I=s@ exclude=s@ only=s@
-	import-remote!  no-torsocks torsocks=s),
+	import-remote! no-torsocks torsocks=s),
 	PublicInbox::LeiQuery::curl_opt());
 
 # we generate shell completion + help using %CMD and %OPTDESC,
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index f44d8af1..8e610efd 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -6,7 +6,7 @@ package PublicInbox::LeiBlob;
 use strict;
 use v5.10.1;
 use parent qw(PublicInbox::IPC);
-use PublicInbox::Spawn qw(spawn popen_rd);
+use PublicInbox::Spawn qw(spawn popen_rd which);
 use PublicInbox::DS;
 
 sub sol_done_wait { # dwaitpid callback
@@ -66,7 +66,10 @@ sub do_solve_blob { # via wq_do
 	}
 	open my $log, '+>', \(my $log_buf = '') or die "PerlIO::scalar: $!";
 	$lei->{log_buf} = \$log_buf;
-	my $git = $lei->ale->git;
+	my $git = $lei->{ale}->git;
+	my @rmt = map {
+		PublicInbox::LeiRemote->new($lei, $_)
+	} $self->{lxs}->remotes;
 	my $solver = bless {
 		gits => [ map {
 				PublicInbox::Git->new($lei->rel2abs($_))
@@ -74,7 +77,7 @@ sub do_solve_blob { # via wq_do
 		user_cb => \&solver_user_cb,
 		uarg => $self,
 		# -cur_di, -qsp, -msg => temporary fields for Qspawn callbacks
-		inboxes => [ $self->{lxs}->locals ],
+		inboxes => [ $self->{lxs}->locals, @rmt ],
 	}, 'PublicInbox::SolverGit';
 	$lei->{env}->{'psgi.errors'} = $lei->{2}; # ugh...
 	local $PublicInbox::DS::in_loop = 0; # waitpid synchronously
@@ -105,8 +108,15 @@ sub lei_blob {
 	}
 	return $lei->fail('no --git-dir to try') unless @$git_dirs;
 	my $lxs = $lei->lxs_prepare or return;
+	if ($lxs->remotes) {
+		require PublicInbox::LeiRemote;
+		$lei->{curl} //= which('curl') or return
+			$lei->fail('curl needed for', $lxs->remotes);
+		$lei->_lei_store(1)->write_prepare($lei);
+	}
 	require PublicInbox::SolverGit;
 	my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
+	$lei->ale;
 	my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
 		{ '' => [ \&sol_done, $lei ] });
 	$lei->{sol} = $self;
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
new file mode 100644
index 00000000..399fc936
--- /dev/null
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -0,0 +1,81 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Make remote externals HTTP(S) inboxes behave like
+# PublicInbox::Inbox and PublicInbox::Search/ExtSearch.
+# This exists solely for SolverGit.  It is a high-latency a
+# synchronous API that is not at all fast.
+package PublicInbox::LeiRemote;
+use v5.10.1;
+use strict;
+use IO::Uncompress::Gunzip;
+use PublicInbox::OnDestroy;
+use PublicInbox::MboxReader;
+use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::LeiCurl;
+use PublicInbox::ContentHash qw(git_sha);
+
+sub new {
+	my ($cls, $lei, $uri) = @_;
+	bless { uri => $uri, lei => $lei }, $cls;
+}
+
+sub isrch { $_[0] } # SolverGit expcets this
+
+sub _each_mboxrd_eml { # callback for MboxReader->mboxrd
+	my ($eml, $self) = @_;
+	my $lei = $self->{lei};
+	my $xoids = $lei->{ale}->xoids_for($eml, 1);
+	if ($lei->{sto} && !$xoids) { # memoize locally
+		$lei->{sto}->ipc_do('add_eml', $eml);
+	}
+	my $smsg = bless {}, 'PublicInbox::Smsg';
+	$smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+				: git_sha(1, $eml)->hexdigest;
+	$smsg->populate($eml);
+	$smsg->{mid} //= '(none)';
+	push @{$self->{smsg}}, $smsg;
+}
+
+sub mset {
+	my ($self, $qstr, undef) = @_; # $opt ($_[2]) ignored
+	my $lei = $self->{lei};
+	my $curl = PublicInbox::LeiCurl->new($lei, $lei->{curl});
+	push @$curl, '-s', '-d', '';
+	my $uri = $self->{uri}->clone;
+	$uri->query_form(q => $qstr, x => 'm', r => 1); # r=1: relevance
+	my $cmd = $curl->for_uri($self->{lei}, $uri);
+	$self->{lei}->qerr("# $cmd");
+	my $rdr = { 2 => $lei->{2}, pgid => 0 };
+	my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
+	my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
+	$self->{smsg} = [];
+	$fh = IO::Uncompress::Gunzip->new($fh);
+	PublicInbox::MboxReader->mboxrd($fh, \&_each_mboxrd_eml, $self);
+	my $err = waitpid($pid, 0) == $pid ? undef
+					: "BUG: waitpid($cmd): $!";
+	@$reap = (); # cancel OnDestroy
+	my $wait = $self->{lei}->{sto}->ipc_do('done');
+	die $err if $err;
+	$self; # we are the mset (and $ibx, and $self)
+}
+
+sub size { scalar @{$_[0]->{smsg}} } # size of previous results
+
+sub mset_to_smsg {
+	my ($self, $ibx, $mset) = @_; # all 3 are $self
+	wantarray ? ($self->size, @{$self->{smsg}}) : $self->{smsg};
+}
+
+sub base_url { "$_[0]->{uri}" }
+
+sub smsg_eml {
+	my ($self, $smsg) = @_;
+	if (my $bref = $self->{lei}->ale->git->cat_file($smsg->{blob})) {
+		return PublicInbox::Eml->new($bref);
+	}
+	$self->{lei}->err("E: $self->{uri} $smsg->{blob} gone <$smsg->{mid}>");
+	undef;
+}
+
+1;
diff --git a/t/solver_git.t b/t/solver_git.t
index 6d4b93c7..2d803d47 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -7,7 +7,7 @@ use PublicInbox::TestCommon;
 use Cwd qw(abs_path);
 require_git(2.6);
 use Digest::SHA qw(sha1_hex);
-use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::Spawn qw(popen_rd which);
 require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
 my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
 $? == 0 or plan skip_all => "$0 must be run from a git working tree";
@@ -227,8 +227,20 @@ EOF
 		my $cmd = [ qw(-httpd -W0), "--stdout=$out", "--stderr=$err" ];
 		my $td = start_script($cmd, $env, { 3 => $sock });
 		my ($h, $p) = tcp_host_port($sock);
-		local $ENV{PLACK_TEST_EXTERNALSERVER_URI} = "http://$h:$p";
+		my $url = "http://$h:$p";
+		local $ENV{PLACK_TEST_EXTERNALSERVER_URI} = $url;
 		Plack::Test::ExternalServer::test_psgi(client => $client);
+		skip 'no curl', 1 unless which('curl');
+
+		mkdir "$tmpdir/ext" // xbail "mkdir $!";
+		test_lei({tmpdir => "$tmpdir/ext"}, sub {
+			my $rurl = "$url/$name";
+			lei_ok(qw(blob --no-mail 69df7d5 -I), $rurl);
+			is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
+				$expect, 'blob contents output');
+			ok(!lei(qw(blob -I), $rurl, $non_existent),
+					'non-existent blob fails');
+		});
 	}
 }
 

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (9 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 10/12] lei blob: add remote external support Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  2021-03-28  9:31   ` Eric Wong
  2021-03-28  9:01 ` [PATCH 12/12] treewide: shorten temporary filename Eric Wong
  11 siblings, 1 reply; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

"lei blob" supports --git-dir and -C, and checks if the
current directory has a git directory associated with it.
It will likely support submodules in the future.

I'm inclined to believe declaring coderepos in a command-line
tool is needless clutter and users will rarely want to search
for blobs across different projects when on the command-line.
---
 lib/PublicInbox/LEI.pm     | 9 ---------
 lib/PublicInbox/LeiBlob.pm | 1 +
 2 files changed, 1 insertion(+), 9 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a94941a9..8a07a4c8 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -172,15 +172,6 @@ our %CMD = ( # sorted in order of importance/use:
 	'remove imported messages from IMAP, Maildirs, and MH',
 	qw(exact! all jobs:i indexed), @c_opt ],
 
-# code repos are used for `show' to solve blobs from patch mails
-'add-coderepo' => [ 'DIRNAME', 'add or set priority of a git code repo',
-	qw(boost=i), @c_opt ],
-'ls-coderepo' => [ '[FILTER_TERMS...]',
-		'list known code repos', qw(format|f=s z), @c_opt ],
-'forget-coderepo' => [ 'DIRNAME',
-	'stop using repo to solve blobs from patches',
-	qw(prune), @c_opt ],
-
 'add-watch' => [ 'LOCATION', 'watch for new messages and flag changes',
 	qw(import! kw|keywords|flags! interval=s recursive|r
 	exclude=s include=s), @c_opt ],
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 8e610efd..91098a90 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -2,6 +2,7 @@
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 
 # "lei blob $OID" command
+# TODO: this doesn't scan submodules, but maybe it should
 package PublicInbox::LeiBlob;
 use strict;
 use v5.10.1;

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 12/12] treewide: shorten temporary filename
  2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
                   ` (10 preceding siblings ...)
  2021-03-28  9:01 ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
@ 2021-03-28  9:01 ` Eric Wong
  11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:01 UTC (permalink / raw)
  To: meta

File::Temp only requires four 'X' characters (unlike mkstemp(3),
which requires six).  So only so only give it 4 to avoid an
80-column violation and maybe save metadata space on FSes.
---
 lib/PublicInbox/LeiOverview.pm | 2 +-
 lib/PublicInbox/Msgmap.pm      | 2 +-
 lib/PublicInbox/SolverGit.pm   | 2 +-
 lib/PublicInbox/TestCommon.pm  | 2 +-
 lib/PublicInbox/V2Writable.pm  | 6 +++---
 lib/PublicInbox/Xapcmd.pm      | 9 ++++-----
 script/public-inbox-edit       | 2 +-
 script/public-inbox-init       | 2 +-
 scripts/ssoma-replay           | 2 +-
 t/check-www-inbox.perl         | 2 +-
 t/inbox.t                      | 2 +-
 t/nodatacow.t                  | 2 +-
 12 files changed, 17 insertions(+), 18 deletions(-)

diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 8e26cba4..68f6c792 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -26,7 +26,7 @@ sub _iso8601 ($) { strftime('%Y-%m-%dT%H:%M:%SZ', gmtime($_[0])) }
 # we open this in the parent process before ->wq_io_do handoff
 sub ovv_out_lk_init ($) {
 	my ($self) = @_;
-	my $tmp = File::Temp->new("lei-ovv.dst.$$.lock-XXXXXX",
+	my $tmp = File::Temp->new("lei-ovv.dst.$$.lock-XXXX",
 					TMPDIR => 1, UNLINK => 0);
 	$self->{"lk_id.$self.$$"} = $self->{lock_path} = $tmp->filename;
 }
diff --git a/lib/PublicInbox/Msgmap.pm b/lib/PublicInbox/Msgmap.pm
index 826c4b30..16a9a476 100644
--- a/lib/PublicInbox/Msgmap.pm
+++ b/lib/PublicInbox/Msgmap.pm
@@ -46,7 +46,7 @@ sub new_file {
 sub tmp_clone {
 	my ($self, $dir) = @_;
 	require File::Temp;
-	my $tmp = "mm_tmp-$$-XXXXXX";
+	my $tmp = "mm_tmp-$$-XXXX";
 	my ($fh, $fn) = File::Temp::tempfile($tmp, EXLOCK => 0, DIR => $dir);
 	PublicInbox::Spawn::nodatacow_fd(fileno($fh));
 	$self->{dbh}->sqlite_backup_to_file($fn);
diff --git a/lib/PublicInbox/SolverGit.pm b/lib/PublicInbox/SolverGit.pm
index 1d70975e..92106e75 100644
--- a/lib/PublicInbox/SolverGit.pm
+++ b/lib/PublicInbox/SolverGit.pm
@@ -681,7 +681,7 @@ sub solve ($$$$$) {
 	$self->{todo} = [ { %$hints, oid_b => $oid_want } ];
 	$self->{patches} = []; # [ $di, $di, ... ]
 	$self->{found} = {}; # { abbr => [ ::Git, oid, type, size, $di ] }
-	$self->{tmp} = File::Temp->newdir("solver.$oid_want-XXXXXXXX", TMPDIR => 1);
+	$self->{tmp} = File::Temp->newdir("solver.$oid_want-XXXX", TMPDIR => 1);
 
 	dbg($self, "solving $oid_want ...");
 	if (my $async = $env->{'pi-httpd.async'}) {
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index d36a63aa..d6e090dd 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -40,7 +40,7 @@ sub tmpdir (;$) {
 	unless (defined $base) {
 		($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
 	}
-	my $tmpdir = File::Temp->newdir("pi-$base-$$-XXXXXX", TMPDIR => 1);
+	my $tmpdir = File::Temp->newdir("pi-$base-$$-XXXX", TMPDIR => 1);
 	($tmpdir->dirname, $tmpdir);
 }
 
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index 4130a472..0461257f 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -660,7 +660,7 @@ sub done {
 
 sub write_alternates ($$$) {
 	my ($info_dir, $mode, $out) = @_;
-	my $fh = File::Temp->new(TEMPLATE => 'alt-XXXXXXXX', DIR => $info_dir);
+	my $fh = File::Temp->new(TEMPLATE => 'alt-XXXX', DIR => $info_dir);
 	my $tmp = $fh->filename;
 	print $fh @$out or die "print $tmp: $!\n";
 	chmod($mode, $fh) or die "fchmod $tmp: $!\n";
@@ -772,11 +772,11 @@ sub import_init {
 sub diff ($$$) {
 	my ($mid, $cur, $new) = @_;
 
-	my $ah = File::Temp->new(TEMPLATE => 'email-cur-XXXXXXXX', TMPDIR => 1);
+	my $ah = File::Temp->new(TEMPLATE => 'email-cur-XXXX', TMPDIR => 1);
 	print $ah $cur->as_string or die "print: $!";
 	$ah->flush or die "flush: $!";
 	PublicInbox::Import::drop_unwanted_headers($new);
-	my $bh = File::Temp->new(TEMPLATE => 'email-new-XXXXXXXX', TMPDIR => 1);
+	my $bh = File::Temp->new(TEMPLATE => 'email-new-XXXX', TMPDIR => 1);
 	print $bh $new->as_string or die "print: $!";
 	$bh->flush or die "flush: $!";
 	my $cmd = [ qw(diff -u), $ah->filename, $bh->filename ];
diff --git a/lib/PublicInbox/Xapcmd.pm b/lib/PublicInbox/Xapcmd.pm
index e2d67f6a..9791f02c 100644
--- a/lib/PublicInbox/Xapcmd.pm
+++ b/lib/PublicInbox/Xapcmd.pm
@@ -192,7 +192,7 @@ sub prepare_run {
 		my $dir = dirname($old);
 		same_fs_or_die($dir, $old);
 		my $v = PublicInbox::Search::SCHEMA_VERSION();
-		my $wip = File::Temp->newdir("xapian$v-XXXXXXXX", DIR => $dir);
+		my $wip = File::Temp->newdir("xapian$v-XXXX", DIR => $dir);
 		$tmp->{$old} = $wip;
 		nodatacow_dir($wip->dirname);
 		push @queue, [ $old, $wip ];
@@ -220,8 +220,7 @@ sub prepare_run {
 			$src = [ map { "$old/$_" } @old_shards ];
 		}
 		foreach my $dn (0..$max_shard) {
-			my $tmpl = "$dn-XXXXXXXX";
-			my $wip = File::Temp->newdir($tmpl, DIR => $old);
+			my $wip = File::Temp->newdir("$dn-XXXX", DIR => $old);
 			same_fs_or_die($old, $wip->dirname);
 			my $cur = "$old/$dn";
 			push @queue, [ $src // $cur , $wip ];
@@ -291,7 +290,7 @@ sub cpdb_retryable ($$) {
 }
 
 sub progress_pfx ($) {
-	my ($wip) = @_; # tempdir v2: ([0-9])+-XXXXXXXX
+	my ($wip) = @_; # tempdir v2: ([0-9])+-XXXX
 	my @p = split('/', $wip);
 
 	# return "xap15/0" for v2, or "xapian15" for v1:
@@ -418,7 +417,7 @@ sub cpdb ($$) { # cb_spawn callback
 	if ($opt->{compact}) {
 		my $dir = dirname($new);
 		same_fs_or_die($dir, $new);
-		$ft = File::Temp->newdir("$new.compact-XXXXXX", DIR => $dir);
+		$ft = File::Temp->newdir("$new.compact-XXXX", DIR => $dir);
 		setup_signals();
 		$tmp = $ft->dirname;
 		nodatacow_dir($tmp);
diff --git a/script/public-inbox-edit b/script/public-inbox-edit
index 1c6c4e4a..9498038b 100755
--- a/script/public-inbox-edit
+++ b/script/public-inbox-edit
@@ -133,7 +133,7 @@ $mids
 }
 
 my %tmpopt = (
-	TEMPLATE => 'public-inbox-edit-XXXXXX',
+	TEMPLATE => 'public-inbox-edit-XXXX',
 	TMPDIR => 1,
 	SUFFIX => $opt->{raw} ? '.eml' : '.mbox',
 );
diff --git a/script/public-inbox-init b/script/public-inbox-init
index e93cab73..335eb476 100755
--- a/script/public-inbox-init
+++ b/script/public-inbox-init
@@ -79,7 +79,7 @@ PublicInbox::Lock::lock_acquire($lock_obj);
 
 # git-config will operate on this (and rename on success):
 require File::Temp;
-my $fh = File::Temp->new(TEMPLATE => 'pi-init-XXXXXXXX', DIR => $dir);
+my $fh = File::Temp->new(TEMPLATE => 'pi-init-XXXX', DIR => $dir);
 
 # Now, we grab another lock to use git-config(1) locking, so it won't
 # wait on the lock, unlike some of our internal flock()-based locks.
diff --git a/scripts/ssoma-replay b/scripts/ssoma-replay
index cfb0fbd9..70d0081d 100755
--- a/scripts/ssoma-replay
+++ b/scripts/ssoma-replay
@@ -29,7 +29,7 @@ use strict;
 use Email::Simple;
 use URI::Escape qw/uri_escape_utf8/;
 use File::Temp qw/tempfile/;
-my ($fh, $filename) = tempfile('ssoma-replay-XXXXXXXX', TMPDIR => 1);
+my ($fh, $filename) = tempfile('ssoma-replay-XXXX', TMPDIR => 1);
 my $msg = Email::Simple->new(do { local $/; <STDIN> });
 select $fh;
 
diff --git a/t/check-www-inbox.perl b/t/check-www-inbox.perl
index eee8adc2..033b90d1 100644
--- a/t/check-www-inbox.perl
+++ b/t/check-www-inbox.perl
@@ -91,7 +91,7 @@ foreach my $p (1..$nproc) {
 	}
 }
 
-my ($fh, $tmp) = tempfile('www-check-XXXXXXXX',
+my ($fh, $tmp) = tempfile('www-check-XXXX',
 			SUFFIX => '.gdbm', UNLINK => 1, TMPDIR => 1);
 my $gdbm = tie my %seen, 'GDBM_File', $tmp, &GDBM_WRCREAT, 0600;
 defined $gdbm or die "gdbm open failed: $!\n";
diff --git a/t/inbox.t b/t/inbox.t
index b7239e6d..0580cd23 100644
--- a/t/inbox.t
+++ b/t/inbox.t
@@ -13,7 +13,7 @@ is($x->base_url, 'http://example.com/test/', 'added trailing slash');
 $x = PublicInbox::Inbox->new({});
 
 is($x->base_url, undef, 'undef base_url allowed');
-my $tmpdir = File::Temp->newdir('pi-inbox-XXXXXX', TMPDIR => 1);
+my $tmpdir = File::Temp->newdir('pi-inbox-XXXX', TMPDIR => 1);
 $x->{inboxdir} = $tmpdir->dirname;
 is_deeply($x->cloneurl, [], 'no cloneurls');
 is($x->description, '($INBOX_DIR/description missing)', 'default description');
diff --git a/t/nodatacow.t b/t/nodatacow.t
index 72860d43..9b67c521 100644
--- a/t/nodatacow.t
+++ b/t/nodatacow.t
@@ -15,7 +15,7 @@ SKIP: {
 	skip 'BTRFS_TESTDIR not defined', $nr unless defined $dir;
 	skip 'chattr(1) not installed', $nr unless which('chattr');
 	my $lsattr = which('lsattr') or skip 'lsattr(1) not installed', $nr;
-	my $tmp = File::Temp->newdir('nodatacow-XXXXX', DIR => $dir);
+	my $tmp = File::Temp->newdir('nodatacow-XXXX', DIR => $dir);
 	my $dn = $tmp->dirname;
 
 	my $name = "$dn/pp.f";

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO
  2021-03-28  9:01 ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
@ 2021-03-28  9:31   ` Eric Wong
  0 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28  9:31 UTC (permalink / raw)
  To: meta

Eric Wong <e@80x24.org> wrote:
> "lei blob" supports --git-dir and -C, and checks if the
> current directory has a git directory associated with it.
> It will likely support submodules in the future.
> 
> I'm inclined to believe declaring coderepos in a command-line
> tool is needless clutter and users will rarely want to search
> for blobs across different projects when on the command-line.

Fwiw, we may we end up supporting read-write JMAP AND expose
blob reconstruction as a vendor-specific JMAP function.
But that's not happening for 1.7...

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2021-03-28  9:31 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-28  9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28  9:01 ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
2021-03-28  9:01 ` [PATCH 02/12] lei init: split out into separate file Eric Wong
2021-03-28  9:01 ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
2021-03-28  9:01 ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
2021-03-28  9:01 ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
2021-03-28  9:01 ` [PATCH 06/12] lei blob: some extra tests Eric Wong
2021-03-28  9:01 ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
2021-03-28  9:01 ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
2021-03-28  9:01 ` [PATCH 09/12] t/lei_store: ensure LeiSearch responds to ->isrch Eric Wong
2021-03-28  9:01 ` [PATCH 10/12] lei blob: add remote external support Eric Wong
2021-03-28  9:01 ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
2021-03-28  9:31   ` Eric Wong
2021-03-28  9:01 ` [PATCH 12/12] treewide: shorten temporary filename Eric Wong

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ http://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git