* [PATCH/RFC 0/7] lei - Local Email Interface skeleton
@ 2020-12-15 11:47 5% Eric Wong
2020-12-15 11:47 7% ` [RFC 6/7] tests: more common JSON module loading Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2020-12-15 11:47 UTC (permalink / raw)
To: meta
patches 1 and 2 are boring cleanups.
The most important is 4/7 which features data structures
for a proposed command set. Hopefully the command-names
and 1-line descriptions are helpful.
Comments from (potential) users appreciated, especially about 4/7.
I decided to take care of patch 3/7 (FD-passing) early on
because startup latency sucks.
I never used notmuch, but this will feature saved searches (aka
"named queries"). Otherwise, the query subcommand will probably
operate like mairix and dump the results to a
Maildir/mbox/etc...
patch 5/7 - keywords (e.g. `seen', 'draft', ...) read/write
(but not query) support added.
And a couple more cleanups.
lei will have its own writable git storage on top of extindex,
but will be able to do read-only queries against extinbox
(publicinbox || extindex) sources.
Eric Wong (7):
daemon: support --daemonize without Net::Server::Daemonize
daemon: simplify fork() failure checks
lei: FD-passing and IPC basics
lei: proposed command-listing and options
lei_store: local storage for Local Email Interface
tests: more common JSON module loading
lei: use spawn (vfork + execve) for lazy start
MANIFEST | 6 +
lib/PublicInbox/Daemon.pm | 26 +-
lib/PublicInbox/ExtSearch.pm | 4 +-
lib/PublicInbox/ExtSearchIdx.pm | 35 ++-
lib/PublicInbox/Import.pm | 4 +
lib/PublicInbox/LeiDaemon.pm | 449 ++++++++++++++++++++++++++++++
lib/PublicInbox/LeiSearch.pm | 40 +++
lib/PublicInbox/LeiStore.pm | 197 +++++++++++++
lib/PublicInbox/ManifestJsGz.pm | 2 +-
lib/PublicInbox/OverIdx.pm | 10 +
lib/PublicInbox/SearchIdx.pm | 47 +++-
lib/PublicInbox/SearchIdxShard.pm | 33 +++
lib/PublicInbox/TestCommon.pm | 4 +
lib/PublicInbox/V2Writable.pm | 2 +-
script/lei | 64 +++++
t/extsearch.t | 3 +-
t/lei.t | 79 ++++++
t/lei_store.t | 74 +++++
t/www_listing.t | 8 +-
19 files changed, 1055 insertions(+), 32 deletions(-)
create mode 100644 lib/PublicInbox/LeiDaemon.pm
create mode 100644 lib/PublicInbox/LeiSearch.pm
create mode 100644 lib/PublicInbox/LeiStore.pm
create mode 100755 script/lei
create mode 100644 t/lei.t
create mode 100644 t/lei_store.t
^ permalink raw reply [relevance 5%]
* [RFC 6/7] tests: more common JSON module loading
2020-12-15 11:47 5% [PATCH/RFC 0/7] lei - Local Email Interface skeleton Eric Wong
@ 2020-12-15 11:47 7% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2020-12-15 11:47 UTC (permalink / raw)
To: meta
We'll probably be using JSON more in the future, so make
it easier to require in tests
---
lib/PublicInbox/ManifestJsGz.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 4 ++++
t/extsearch.t | 3 +--
t/lei.t | 3 +--
t/www_listing.t | 8 +++-----
5 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/ManifestJsGz.pm b/lib/PublicInbox/ManifestJsGz.pm
index 6d5b57ee..33df020a 100644
--- a/lib/PublicInbox/ManifestJsGz.pm
+++ b/lib/PublicInbox/ManifestJsGz.pm
@@ -11,7 +11,7 @@ use PublicInbox::Config;
use IO::Compress::Gzip qw(gzip);
use HTTP::Date qw(time2str);
-our $json = PublicInbox::Config::json();
+my $json = PublicInbox::Config::json();
# called by WwwListing
sub url_regexp {
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 299b9c6a..2116575b 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -75,6 +75,10 @@ sub require_mods {
my $maybe = pop @mods if $mods[-1] =~ /\A[0-9]+\z/;
my @need;
while (my $mod = shift(@mods)) {
+ if ($mod eq 'json') {
+ $mod = 'Cpanel::JSON::XS||JSON::MaybeXS||'.
+ 'JSON||JSON::PP'
+ }
if ($mod eq 'Search::Xapian') {
if (eval { require PublicInbox::Search } &&
PublicInbox::Search::load_xapian()) {
diff --git a/t/extsearch.t b/t/extsearch.t
index fb31b0ab..ffbc10e2 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -8,9 +8,8 @@ use PublicInbox::Config;
use PublicInbox::Search;
use PublicInbox::InboxWritable;
use Fcntl qw(:seek);
-my $json = PublicInbox::Config::json() or plan skip_all => 'JSON missing';
require_git(2.6);
-require_mods(qw(DBD::SQLite Search::Xapian));
+require_mods(qw(json DBD::SQLite Search::Xapian));
use_ok 'PublicInbox::ExtSearch';
use_ok 'PublicInbox::ExtSearchIdx';
use_ok 'PublicInbox::OverIdx';
diff --git a/t/lei.t b/t/lei.t
index feee9270..02f21322 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -6,8 +6,7 @@ use v5.10.1;
use Test::More;
use PublicInbox::TestCommon;
use PublicInbox::Config;
-my $json = PublicInbox::Config::json() or plan skip_all => 'JSON missing';
-require_mods(qw(DBD::SQLite Search::Xapian));
+require_mods(qw(json DBD::SQLite Search::Xapian));
my ($home, $for_destroy) = tmpdir();
my $opt = { 1 => \(my $out = ''), 2 => \(my $err = '') };
diff --git a/t/www_listing.t b/t/www_listing.t
index 63613371..94c1e5bb 100644
--- a/t/www_listing.t
+++ b/t/www_listing.t
@@ -7,14 +7,12 @@ use Test::More;
use PublicInbox::Spawn qw(which);
use PublicInbox::TestCommon;
use PublicInbox::Import;
-require_mods(qw(URI::Escape Plack::Builder Digest::SHA
+require_mods(qw(json URI::Escape Plack::Builder Digest::SHA
IO::Compress::Gzip IO::Uncompress::Gunzip HTTP::Tiny));
require PublicInbox::WwwListing;
require PublicInbox::ManifestJsGz;
-my $json = do {
- no warnings 'once';
- $PublicInbox::ManifestJsGz::json;
-} or plan skip_all => "JSON module missing";
+use PublicInbox::Config;
+my $json = PublicInbox::Config::json();
use_ok 'PublicInbox::Git';
^ permalink raw reply related [relevance 7%]
* [PATCH 00/26] lei: basic UI + IPC work
@ 2020-12-18 12:09 4% Eric Wong
2020-12-18 12:09 7% ` [PATCH 04/26] tests: more common JSON module loading Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2020-12-18 12:09 UTC (permalink / raw)
To: meta
Some work on the storage side, but MiscIdx still needs work to
handle existing publicinboxes, extinboxes (over HTTP(S)), and
other config things.
PATCH 22/26 - bash completion sorta works, but filename
completions get broken. Not sure why and help would be
greatly appreciated (along with help for other shells).
I don't know bash-specific stuff well at all, even; and
less about other non-POSIX shells.
Somewhat nice UI things (at least to my delirious sleep-deprived
state):
* -$DIGIT option parsing works (e.g. "git log -10"),
"kill -9"
* help-based CLI arg/prototype checking seems working
and hopefully cuts down on long-term maintenance work
while promoting UI consistency
* having IO::FDPass hides startup time, 20-30ms isn't
really noticeable for humans on interactive terminals,
but still not ideal for loops.
* lei.sh + "make symlink-install"
And some internal improvements:
* several simplifications to existing Search code,
->xdb_shards_flat will come in handy
* generic OnDestroy - long overdue
Eric Wong (26):
lei: FD-passing and IPC basics
lei: proposed command-listing and options
lei_store: local storage for Local Email Interface
tests: more common JSON module loading
lei: use spawn (vfork + execve) for lazy start
lei: refine help/option parsing, implement "init"
t/lei-oneshot: standalone oneshot (non-socket) test
lei: ensure we run a restrictive umask
lei: support `daemon-env' for modifying long-lived env
lei_store: simplify git_epoch_max, slightly
search: simplify initialization, add ->xdb_shards_flat
rename LeiDaemon package to PublicInbox::LEI
lei: support pass-through for `lei config'
lei: help: show actual paths being operated on
lei: rename $client => $self and bless
lei: micro-optimize startup time
lei_store: relax GIT_COMMITTER_IDENT check
lei_store: keyword extraction from mbox and Maildir
on_destroy: generic localized END
lei: restore default __DIE__ handler for event loop
lei: drop $SIG{__DIE__}, add oneshot fallbacks
lei: start working on bash completion
build: add lei.sh + "make symlink-install" target
lei: support for -$DIGIT and -$SIG CLI switches
lei: revise output routines
lei: extinbox: start implementing in config file
MANIFEST | 11 +
Makefile.PL | 11 +
contrib/completion/lei-completion.bash | 11 +
lei.sh | 7 +
lib/PublicInbox/Daemon.pm | 6 +-
lib/PublicInbox/ExtSearch.pm | 10 +-
lib/PublicInbox/ExtSearchIdx.pm | 35 +-
lib/PublicInbox/Import.pm | 4 +
lib/PublicInbox/LEI.pm | 776 +++++++++++++++++++++++++
lib/PublicInbox/LeiExtinbox.pm | 52 ++
lib/PublicInbox/LeiSearch.pm | 39 ++
lib/PublicInbox/LeiStore.pm | 227 ++++++++
lib/PublicInbox/ManifestJsGz.pm | 2 +-
lib/PublicInbox/OnDestroy.pm | 16 +
lib/PublicInbox/OverIdx.pm | 10 +
lib/PublicInbox/Search.pm | 65 +--
lib/PublicInbox/SearchIdx.pm | 62 +-
lib/PublicInbox/SearchIdxShard.pm | 33 ++
lib/PublicInbox/TestCommon.pm | 7 +-
lib/PublicInbox/V2Writable.pm | 10 +-
script/lei | 76 +++
t/extsearch.t | 3 +-
t/lei-oneshot.t | 25 +
t/lei.t | 306 ++++++++++
t/lei_store.t | 88 +++
t/on_destroy.t | 25 +
t/www_listing.t | 8 +-
27 files changed, 1843 insertions(+), 82 deletions(-)
create mode 100644 contrib/completion/lei-completion.bash
create mode 100755 lei.sh
create mode 100644 lib/PublicInbox/LEI.pm
create mode 100644 lib/PublicInbox/LeiExtinbox.pm
create mode 100644 lib/PublicInbox/LeiSearch.pm
create mode 100644 lib/PublicInbox/LeiStore.pm
create mode 100644 lib/PublicInbox/OnDestroy.pm
create mode 100755 script/lei
create mode 100644 t/lei-oneshot.t
create mode 100644 t/lei.t
create mode 100644 t/lei_store.t
create mode 100644 t/on_destroy.t
^ permalink raw reply [relevance 4%]
* [PATCH 04/26] tests: more common JSON module loading
2020-12-18 12:09 4% [PATCH 00/26] lei: basic UI + IPC work Eric Wong
@ 2020-12-18 12:09 7% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2020-12-18 12:09 UTC (permalink / raw)
To: meta
We'll probably be using JSON more in the future, so make
it easier to require in tests
---
lib/PublicInbox/ManifestJsGz.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 4 ++++
t/extsearch.t | 3 +--
t/lei.t | 3 +--
t/www_listing.t | 8 +++-----
5 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/ManifestJsGz.pm b/lib/PublicInbox/ManifestJsGz.pm
index 6d5b57ee..33df020a 100644
--- a/lib/PublicInbox/ManifestJsGz.pm
+++ b/lib/PublicInbox/ManifestJsGz.pm
@@ -11,7 +11,7 @@ use PublicInbox::Config;
use IO::Compress::Gzip qw(gzip);
use HTTP::Date qw(time2str);
-our $json = PublicInbox::Config::json();
+my $json = PublicInbox::Config::json();
# called by WwwListing
sub url_regexp {
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 299b9c6a..2116575b 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -75,6 +75,10 @@ sub require_mods {
my $maybe = pop @mods if $mods[-1] =~ /\A[0-9]+\z/;
my @need;
while (my $mod = shift(@mods)) {
+ if ($mod eq 'json') {
+ $mod = 'Cpanel::JSON::XS||JSON::MaybeXS||'.
+ 'JSON||JSON::PP'
+ }
if ($mod eq 'Search::Xapian') {
if (eval { require PublicInbox::Search } &&
PublicInbox::Search::load_xapian()) {
diff --git a/t/extsearch.t b/t/extsearch.t
index fb31b0ab..ffbc10e2 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -8,9 +8,8 @@ use PublicInbox::Config;
use PublicInbox::Search;
use PublicInbox::InboxWritable;
use Fcntl qw(:seek);
-my $json = PublicInbox::Config::json() or plan skip_all => 'JSON missing';
require_git(2.6);
-require_mods(qw(DBD::SQLite Search::Xapian));
+require_mods(qw(json DBD::SQLite Search::Xapian));
use_ok 'PublicInbox::ExtSearch';
use_ok 'PublicInbox::ExtSearchIdx';
use_ok 'PublicInbox::OverIdx';
diff --git a/t/lei.t b/t/lei.t
index feee9270..02f21322 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -6,8 +6,7 @@ use v5.10.1;
use Test::More;
use PublicInbox::TestCommon;
use PublicInbox::Config;
-my $json = PublicInbox::Config::json() or plan skip_all => 'JSON missing';
-require_mods(qw(DBD::SQLite Search::Xapian));
+require_mods(qw(json DBD::SQLite Search::Xapian));
my ($home, $for_destroy) = tmpdir();
my $opt = { 1 => \(my $out = ''), 2 => \(my $err = '') };
diff --git a/t/www_listing.t b/t/www_listing.t
index 63613371..94c1e5bb 100644
--- a/t/www_listing.t
+++ b/t/www_listing.t
@@ -7,14 +7,12 @@ use Test::More;
use PublicInbox::Spawn qw(which);
use PublicInbox::TestCommon;
use PublicInbox::Import;
-require_mods(qw(URI::Escape Plack::Builder Digest::SHA
+require_mods(qw(json URI::Escape Plack::Builder Digest::SHA
IO::Compress::Gzip IO::Uncompress::Gunzip HTTP::Tiny));
require PublicInbox::WwwListing;
require PublicInbox::ManifestJsGz;
-my $json = do {
- no warnings 'once';
- $PublicInbox::ManifestJsGz::json;
-} or plan skip_all => "JSON module missing";
+use PublicInbox::Config;
+my $json = PublicInbox::Config::json();
use_ok 'PublicInbox::Git';
^ permalink raw reply related [relevance 7%]
Results 1-4 of 4 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-12-15 11:47 5% [PATCH/RFC 0/7] lei - Local Email Interface skeleton Eric Wong
2020-12-15 11:47 7% ` [RFC 6/7] tests: more common JSON module loading Eric Wong
2020-12-18 12:09 4% [PATCH 00/26] lei: basic UI + IPC work Eric Wong
2020-12-18 12:09 7% ` [PATCH 04/26] tests: more common JSON module loading Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).