user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* [PATCH 0/8] lei: docs and cleanups
@ 2021-10-28 11:14 Eric Wong
  2021-10-28 11:14 ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
  To: meta

Expect more on the way.  As always some bugs and fixes were
found while attempting to write docs :x

Eric Wong (8):
  xt/net_writer_imap: test "lei convert" w/ IMAP source
  lei convert: use "--output" in failure message
  doc: lei-convert: various updates and cleanups
  doc: lei blob: wording fixups, describe --remote
  lei convert: remove redundant input_net_cb
  doc: lei-add-watch: add warning about unreliability
  lei sucks: show nproc in CPU info
  lei rm: move generic input_maildir_cb to LeiInput parent class

 Documentation/lei-add-watch.pod |  8 ++++++--
 Documentation/lei-blob.pod      | 12 ++++++++----
 Documentation/lei-convert.pod   | 21 +++++++++------------
 lib/PublicInbox/LeiConvert.pm   |  8 ++------
 lib/PublicInbox/LeiInput.pm     |  5 +++++
 lib/PublicInbox/LeiRm.pm        |  5 -----
 lib/PublicInbox/LeiSucks.pm     |  4 +++-
 t/lei-convert.t                 | 10 ++++++++++
 xt/net_writer-imap.t            |  7 +++++++
 9 files changed, 50 insertions(+), 30 deletions(-)


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source
  2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
  2021-10-28 11:14 ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
  To: meta

I just did a double-take and nearly thought authentication
was broken while reading LeiConvert.pm.  Add a comment in
LeiConvert.pm to clarify things, too.
---
 lib/PublicInbox/LeiConvert.pm | 1 +
 xt/net_writer-imap.t          | 7 +++++++
 2 files changed, 8 insertions(+)

diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 68fc7c0ba2b0..424eab8e22e2 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -55,6 +55,7 @@ sub lei_convert { # the main "lei convert" method
 	my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
 	$lei->{opt}->{augment} = 1 if $devfd < 0;
 	$self->prepare_inputs($lei, \@inputs) or return;
+	# n.b. {net} {auth} is handled by l2m worker
 	my ($op_c, $ops) = $lei->workers_start($self, 1);
 	$lei->{wq1} = $self;
 	$self->wq_io_do('process_inputs', []);
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index afa4bcc3e881..333e0e3b7ee0 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -137,6 +137,13 @@ test_lei(sub {
 	is_deeply($res->[0]->[1], $plack_qp_eml,
 			'lei q wrote expected result');
 
+	my $mdir = "$ENV{HOME}/t.mdir";
+	lei_ok 'convert', $folder_url, '-o', $mdir;
+	my @mdfiles = glob("$mdir/*/*");
+	is(scalar(@mdfiles), 1, '1 message from IMAP => Maildir conversion');
+	is_deeply(eml_load($mdfiles[0]), $plack_qp_eml,
+		'conversion from IMAP to Maildir');
+
 	lei_ok qw(q f:matz -a -o), $folder_url;
 	$nwr->imap_each($folder_uri, $imap_slurp_all, my $aug = []);
 	is(scalar(@$aug), 2, '2 results after augment') or diag explain($aug);

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 2/8] lei convert: use "--output" in failure message
  2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
  2021-10-28 11:14 ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
  2021-10-28 11:14 ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
  To: meta

The extra dashes should help users find the correct option
more easily.
---
 lib/PublicInbox/LeiConvert.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 424eab8e22e2..623599aef883 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -51,7 +51,7 @@ sub lei_convert { # the main "lei convert" method
 	my $self = bless {}, __PACKAGE__;
 	my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format');
 	$lei->{l2m} or return
-		$lei->fail("output not specified or is not a mail destination");
+		$lei->fail('--output unspecified or is not a mail destination');
 	my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
 	$lei->{opt}->{augment} = 1 if $devfd < 0;
 	$self->prepare_inputs($lei, \@inputs) or return;

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 3/8] doc: lei-convert: various updates and cleanups
  2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
  2021-10-28 11:14 ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
  2021-10-28 11:14 ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
  2021-10-28 11:14 ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
  To: meta

Note that "-o OUTPUT" is required in the synopsis.

Leave out "eml:" for now since it doesn't work as an output and
I doubt anybody would use it as a prefix, and it's not really
useful.

--no-import-remote is also not accepted by convert, since it
doesn't touch lei/store at all.
---
 Documentation/lei-convert.pod | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)

diff --git a/Documentation/lei-convert.pod b/Documentation/lei-convert.pod
index 750ba54ffa1a..c113db18b1e4 100644
--- a/Documentation/lei-convert.pod
+++ b/Documentation/lei-convert.pod
@@ -1,23 +1,23 @@
 =head1 NAME
 
-lei-convert - one-time conversion from URL or filesystem to another format
+lei-convert - one-time conversion from one mail format to another
 
 =head1 SYNOPSIS
 
-lei convert [OPTIONS] LOCATION
+lei convert -o OUTPUT [OPTIONS] LOCATION
 
-lei convert [OPTIONS] (--stdin|-)
+lei convert -o OUTPUT [OPTIONS] (--stdin|-)
 
 =head1 DESCRIPTION
 
 Convert messages to another format.  C<LOCATION> is a source of
-messages: a directory (Maildir), a file, or a URL (C<imap://>,
-C<imaps://>, C<nntp://>, or C<nntps://>).  URLs requiring
-authentication use L<git-credential(1)> to
+messages: a directory (Maildir), a file (various mbox), or a URL
+(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>).  URLs
+requiring authentication use L<git-credential(1)> to
 fill in the username and password.
 
 For a regular file, the location must have a C<E<lt>formatE<gt>:>
-prefix specifying one of the following formats: C<eml>, C<mboxrd>,
+prefix specifying one of the following formats: C<mboxrd>,
 C<mboxcl2>, C<mboxcl>, or C<mboxo>.
 
 =head1 OPTIONS
@@ -44,8 +44,6 @@ L<lei-q(1)>.
 
 =item --no-kw
 
-=item --no-import-remote
-
 =item --torsocks=auto|no|yes
 
 =item --no-torsocks
@@ -63,11 +61,10 @@ and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta
 
 =head1 COPYRIGHT
 
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
-
 =head1 SEE ALSO
 
-L<lei-add-external(1)>
+L<lei-mail-formats(5)>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 4/8] doc: lei blob: wording fixups, describe --remote
  2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
                   ` (2 preceding siblings ...)
  2021-10-28 11:14 ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
  2021-10-28 11:14 ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
  To: meta

There's no current way to retrieve blobs by OID directly
from remote externals.  Maybe the $INBOX_NAME/$OID/s/raw.eml
endpoint could be overloaded for that.
---
 Documentation/lei-blob.pod | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod
index 429d206e358e..e401bb4711a6 100644
--- a/Documentation/lei-blob.pod
+++ b/Documentation/lei-blob.pod
@@ -9,9 +9,10 @@ lei blob [OPTIONS] OID
 =head1 DESCRIPTION
 
 Display a git blob.  The blob may correspond to a message from the
-local store, an existing blob in the current repository, or a
-not-yet-created blob in the current git project repository (if any)
-that can be reconstructed from a message.
+local store, any local external, or blobs associated with a
+project git repository (if run from a git (working) directory).
+For blobs which do not exist, it will attempt to recreate the blob
+using patch emails.
 
 =head1 OPTIONS
 
@@ -64,7 +65,10 @@ L<lei-q(1)>.
 
 =over
 
-=item --[no-]remote
+=item --remote
+
+Remote externals only get queried when the blob needs to be
+reconstructed from patch emails.
 
 =item --no-local
 

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 5/8] lei convert: remove redundant input_net_cb
  2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
                   ` (3 preceding siblings ...)
  2021-10-28 11:14 ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
  2021-10-28 11:14 ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
  To: meta

Use the one provided by the LeiInput parent class.
---
 lib/PublicInbox/LeiConvert.pm | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 623599aef883..906f30268344 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -23,11 +23,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
 	$self->{wcb}->(undef, {}, $eml);
 }
 
-sub input_net_cb { # callback for ->imap_each, ->nntp_each
-	my (undef, undef, $kw, $eml, $self) = @_; # @_[0,1]: url + uid ignored
-	$self->{wcb}->(undef, { kw => $kw }, $eml);
-}
-
 sub input_maildir_cb {
 	my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
 	$self->{wcb}->(undef, { kw => $kw }, $eml);

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 6/8] doc: lei-add-watch: add warning about unreliability
  2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
                   ` (4 preceding siblings ...)
  2021-10-28 11:14 ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
  2021-10-28 11:15 ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
  2021-10-28 11:15 ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
  To: meta

This needs work at some point in the future.
---
 Documentation/lei-add-watch.pod | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Documentation/lei-add-watch.pod b/Documentation/lei-add-watch.pod
index 609846181578..1b48b43b589c 100644
--- a/Documentation/lei-add-watch.pod
+++ b/Documentation/lei-add-watch.pod
@@ -11,6 +11,11 @@ lei add-watch [OPTIONS] LOCATION [LOCATION...]
 Tell lei to watch C<LOCATION> for new messages and flag changes.
 Currently only Maildir locations are supported.
 
+WARNING: watches are not always reliable, occasional use
+of L<lei-index(1)> and L<lei-refresh-mail-sync(1)> is recommended
+if L<lei-daemon(8)> crashes or needs to be restarted.  This will
+be improved in the future.
+
 =for comment
 TODO: Document --state?  Believe valid values are pause, import-ro,
 
@@ -23,11 +28,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
 
 =head1 COPYRIGHT
 
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
 
 License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
-
 =head1 SEE ALSO
 
 L<lei-ls-watch(1)>, L<lei-rm-watch(1)>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 7/8] lei sucks: show nproc in CPU info
  2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
                   ` (5 preceding siblings ...)
  2021-10-28 11:14 ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
@ 2021-10-28 11:15 ` Eric Wong
  2021-10-28 11:15 ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:15 UTC (permalink / raw)
  To: meta

Some bugs are triggered with more CPUs, some with 1 CPU.
---
 lib/PublicInbox/LeiSucks.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
index e832f95e0736..8e866fc96655 100644
--- a/lib/PublicInbox/LeiSucks.pm
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -11,6 +11,7 @@ use Digest::SHA ();
 use Config;
 use POSIX ();
 use PublicInbox::Config;
+use PublicInbox::IPC;
 
 sub lei_sucks {
 	my ($lei, @argv) = @_;
@@ -22,9 +23,10 @@ sub lei_sucks {
 	}
 	eval { require PublicInbox };
 	my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';
+	my $nproc = PublicInbox::IPC::detect_nproc() // '?';
 	my @out = ("lei $pi_ver\n",
 		"perl $Config{version} / $os $rel / $mac ".
-		"ptrsize=$Config{ptrsize}\n");
+		"ptrsize=$Config{ptrsize} nproc=$nproc\n");
 	chomp(my $gv = `git --version` || "git missing");
 	$gv =~ s/ version / /;
 	my $json = ref(PublicInbox::Config->json);

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class
  2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
                   ` (6 preceding siblings ...)
  2021-10-28 11:15 ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
@ 2021-10-28 11:15 ` Eric Wong
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:15 UTC (permalink / raw)
  To: meta

It's not much of a savings, right now, but maybe it can be in the
future.  I wanted to eliminate the "lei convert" one, too, but
convert needs to preserve keywords which isn't possible with the
generic fallback, so new tests were written for convert, instead.
---
 lib/PublicInbox/LeiInput.pm |  5 +++++
 lib/PublicInbox/LeiRm.pm    |  5 -----
 t/lei-convert.t             | 10 ++++++++++
 3 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 540681e3ff6b..84fc579dc480 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -64,6 +64,11 @@ sub input_mbox_cb { # base MboxReader callback
 	$self->input_eml_cb($eml);
 }
 
+sub input_maildir_cb {
+	my ($fn, $kw, $eml, $self) = @_;
+	$self->input_eml_cb($eml);
+}
+
 sub input_net_cb { # imap_each, nntp_each cb
 	my ($url, $uid, $kw, $eml, $self) = @_;
 	$self->input_eml_cb($eml);
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index 524c178e3b47..cc1abbff66cb 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -13,11 +13,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
 	$self->{lei}->{sto}->wq_do('remove_eml', $eml);
 }
 
-sub input_maildir_cb {
-	my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
-	input_eml_cb($self, $eml);
-}
-
 sub lei_rm {
 	my ($lei, @inputs) = @_;
 	$lei->_lei_store(1)->write_prepare($lei);
diff --git a/t/lei-convert.t b/t/lei-convert.t
index 0ea860c82189..e1849ff796cd 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -115,5 +115,15 @@ test_lei({ tmpdir => $tmpdir }, sub {
 	@bar = ();
 	PublicInbox::MboxReader->mboxrd($fh, sub { push @bar, shift });
 	is_deeply(\@bar, [ $qp_eml ], 'readed gzipped mboxrd');
+
+	# Status => Maildir flag => Status round trip
+	$lei_out =~ s/^Status: O/Status: RO/sm or xbail "`seen' Status";
+	$rdr = { 0 => \($in = $lei_out), %$lei_opt };
+	lei_ok([qw(convert -F mboxrd -o), "$d/md2"], undef, $rdr);
+	@md = glob("$d/md2/*/*");
+	is(scalar(@md), 1, 'one message');
+	like($md[0], qr/:2,S\z/, "`seen' flag set in Maildir");
+	lei_ok(qw(convert -o mboxrd:/dev/stdout), "$d/md2");
+	like($lei_out, qr/^Status: RO/sm, "`seen' flag preserved");
 });
 done_testing;

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-10-28 11:15 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
2021-10-28 11:14 ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
2021-10-28 11:14 ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
2021-10-28 11:14 ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
2021-10-28 11:14 ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
2021-10-28 11:14 ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
2021-10-28 11:15 ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
2021-10-28 11:15 ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).