user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH/RFC 0/7] lei - Local Email Interface skeleton
@ 2020-12-15 11:47  5% Eric Wong
  2020-12-15 11:47  7% ` [RFC 6/7] tests: more common JSON module loading Eric Wong
  0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2020-12-15 11:47 UTC (permalink / raw)
  To: meta

patches 1 and 2 are boring cleanups.

The most important is 4/7 which features data structures
for a proposed command set.  Hopefully the command-names
and 1-line descriptions are helpful.

Comments from (potential) users appreciated, especially about 4/7.


I decided to take care of patch 3/7 (FD-passing) early on
because startup latency sucks.

I never used notmuch, but this will feature saved searches (aka
"named queries").  Otherwise, the query subcommand will probably
operate like mairix and dump the results to a
Maildir/mbox/etc...

patch 5/7 - keywords (e.g. `seen', 'draft', ...) read/write
(but not query) support added.

And a couple more cleanups.

lei will have its own writable git storage on top of extindex,
but will be able to do read-only queries against extinbox
(publicinbox || extindex) sources.

Eric Wong (7):
  daemon: support --daemonize without Net::Server::Daemonize
  daemon: simplify fork() failure checks
  lei: FD-passing and IPC basics
  lei: proposed command-listing and options
  lei_store: local storage for Local Email Interface
  tests: more common JSON module loading
  lei: use spawn (vfork + execve) for lazy start

 MANIFEST                          |   6 +
 lib/PublicInbox/Daemon.pm         |  26 +-
 lib/PublicInbox/ExtSearch.pm      |   4 +-
 lib/PublicInbox/ExtSearchIdx.pm   |  35 ++-
 lib/PublicInbox/Import.pm         |   4 +
 lib/PublicInbox/LeiDaemon.pm      | 449 ++++++++++++++++++++++++++++++
 lib/PublicInbox/LeiSearch.pm      |  40 +++
 lib/PublicInbox/LeiStore.pm       | 197 +++++++++++++
 lib/PublicInbox/ManifestJsGz.pm   |   2 +-
 lib/PublicInbox/OverIdx.pm        |  10 +
 lib/PublicInbox/SearchIdx.pm      |  47 +++-
 lib/PublicInbox/SearchIdxShard.pm |  33 +++
 lib/PublicInbox/TestCommon.pm     |   4 +
 lib/PublicInbox/V2Writable.pm     |   2 +-
 script/lei                        |  64 +++++
 t/extsearch.t                     |   3 +-
 t/lei.t                           |  79 ++++++
 t/lei_store.t                     |  74 +++++
 t/www_listing.t                   |   8 +-
 19 files changed, 1055 insertions(+), 32 deletions(-)
 create mode 100644 lib/PublicInbox/LeiDaemon.pm
 create mode 100644 lib/PublicInbox/LeiSearch.pm
 create mode 100644 lib/PublicInbox/LeiStore.pm
 create mode 100755 script/lei
 create mode 100644 t/lei.t
 create mode 100644 t/lei_store.t


^ permalink raw reply	[relevance 5%]

* [RFC 6/7] tests: more common JSON module loading
  2020-12-15 11:47  5% [PATCH/RFC 0/7] lei - Local Email Interface skeleton Eric Wong
@ 2020-12-15 11:47  7% ` Eric Wong
  0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2020-12-15 11:47 UTC (permalink / raw)
  To: meta

We'll probably be using JSON more in the future, so make
it easier to require in tests
---
 lib/PublicInbox/ManifestJsGz.pm | 2 +-
 lib/PublicInbox/TestCommon.pm   | 4 ++++
 t/extsearch.t                   | 3 +--
 t/lei.t                         | 3 +--
 t/www_listing.t                 | 8 +++-----
 5 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/lib/PublicInbox/ManifestJsGz.pm b/lib/PublicInbox/ManifestJsGz.pm
index 6d5b57ee..33df020a 100644
--- a/lib/PublicInbox/ManifestJsGz.pm
+++ b/lib/PublicInbox/ManifestJsGz.pm
@@ -11,7 +11,7 @@ use PublicInbox::Config;
 use IO::Compress::Gzip qw(gzip);
 use HTTP::Date qw(time2str);
 
-our $json = PublicInbox::Config::json();
+my $json = PublicInbox::Config::json();
 
 # called by WwwListing
 sub url_regexp {
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 299b9c6a..2116575b 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -75,6 +75,10 @@ sub require_mods {
 	my $maybe = pop @mods if $mods[-1] =~ /\A[0-9]+\z/;
 	my @need;
 	while (my $mod = shift(@mods)) {
+		if ($mod eq 'json') {
+			$mod = 'Cpanel::JSON::XS||JSON::MaybeXS||'.
+				'JSON||JSON::PP'
+		}
 		if ($mod eq 'Search::Xapian') {
 			if (eval { require PublicInbox::Search } &&
 				PublicInbox::Search::load_xapian()) {
diff --git a/t/extsearch.t b/t/extsearch.t
index fb31b0ab..ffbc10e2 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -8,9 +8,8 @@ use PublicInbox::Config;
 use PublicInbox::Search;
 use PublicInbox::InboxWritable;
 use Fcntl qw(:seek);
-my $json = PublicInbox::Config::json() or plan skip_all => 'JSON missing';
 require_git(2.6);
-require_mods(qw(DBD::SQLite Search::Xapian));
+require_mods(qw(json DBD::SQLite Search::Xapian));
 use_ok 'PublicInbox::ExtSearch';
 use_ok 'PublicInbox::ExtSearchIdx';
 use_ok 'PublicInbox::OverIdx';
diff --git a/t/lei.t b/t/lei.t
index feee9270..02f21322 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -6,8 +6,7 @@ use v5.10.1;
 use Test::More;
 use PublicInbox::TestCommon;
 use PublicInbox::Config;
-my $json = PublicInbox::Config::json() or plan skip_all => 'JSON missing';
-require_mods(qw(DBD::SQLite Search::Xapian));
+require_mods(qw(json DBD::SQLite Search::Xapian));
 my ($home, $for_destroy) = tmpdir();
 my $opt = { 1 => \(my $out = ''), 2 => \(my $err = '') };
 
diff --git a/t/www_listing.t b/t/www_listing.t
index 63613371..94c1e5bb 100644
--- a/t/www_listing.t
+++ b/t/www_listing.t
@@ -7,14 +7,12 @@ use Test::More;
 use PublicInbox::Spawn qw(which);
 use PublicInbox::TestCommon;
 use PublicInbox::Import;
-require_mods(qw(URI::Escape Plack::Builder Digest::SHA
+require_mods(qw(json URI::Escape Plack::Builder Digest::SHA
 		IO::Compress::Gzip IO::Uncompress::Gunzip HTTP::Tiny));
 require PublicInbox::WwwListing;
 require PublicInbox::ManifestJsGz;
-my $json = do {
-	no warnings 'once';
-	$PublicInbox::ManifestJsGz::json;
-} or plan skip_all => "JSON module missing";
+use PublicInbox::Config;
+my $json = PublicInbox::Config::json();
 
 use_ok 'PublicInbox::Git';
 

^ permalink raw reply related	[relevance 7%]

* [PATCH 00/26] lei: basic UI + IPC work
@ 2020-12-18 12:09  4% Eric Wong
  2020-12-18 12:09  7% ` [PATCH 04/26] tests: more common JSON module loading Eric Wong
  0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2020-12-18 12:09 UTC (permalink / raw)
  To: meta

Some work on the storage side, but MiscIdx still needs work to
handle existing publicinboxes, extinboxes (over HTTP(S)), and
other config things.

PATCH 22/26 - bash completion sorta works, but filename
completions get broken.  Not sure why and help would be
greatly appreciated (along with help for other shells).
I don't know bash-specific stuff well at all, even; and
less about other non-POSIX shells.

Somewhat nice UI things (at least to my delirious sleep-deprived
state):

* -$DIGIT option parsing works (e.g. "git log -10"),
  "kill -9"

* help-based CLI arg/prototype checking seems working
  and hopefully cuts down on long-term maintenance work
  while promoting UI consistency

* having IO::FDPass hides startup time, 20-30ms isn't
  really noticeable for humans on interactive terminals,
  but still not ideal for loops.

* lei.sh + "make symlink-install"

And some internal improvements:

* several simplifications to existing Search code,
  ->xdb_shards_flat will come in handy

* generic OnDestroy - long overdue

Eric Wong (26):
  lei: FD-passing and IPC basics
  lei: proposed command-listing and options
  lei_store: local storage for Local Email Interface
  tests: more common JSON module loading
  lei: use spawn (vfork + execve) for lazy start
  lei: refine help/option parsing, implement "init"
  t/lei-oneshot: standalone oneshot (non-socket) test
  lei: ensure we run a restrictive umask
  lei: support `daemon-env' for modifying long-lived env
  lei_store: simplify git_epoch_max, slightly
  search: simplify initialization, add ->xdb_shards_flat
  rename LeiDaemon package to PublicInbox::LEI
  lei: support pass-through for `lei config'
  lei: help: show actual paths being operated on
  lei: rename $client => $self and bless
  lei: micro-optimize startup time
  lei_store: relax GIT_COMMITTER_IDENT check
  lei_store: keyword extraction from mbox and Maildir
  on_destroy: generic localized END
  lei: restore default __DIE__ handler for event loop
  lei: drop $SIG{__DIE__}, add oneshot fallbacks
  lei: start working on bash completion
  build: add lei.sh + "make symlink-install" target
  lei: support for -$DIGIT and -$SIG CLI switches
  lei: revise output routines
  lei: extinbox: start implementing in config file

 MANIFEST                               |  11 +
 Makefile.PL                            |  11 +
 contrib/completion/lei-completion.bash |  11 +
 lei.sh                                 |   7 +
 lib/PublicInbox/Daemon.pm              |   6 +-
 lib/PublicInbox/ExtSearch.pm           |  10 +-
 lib/PublicInbox/ExtSearchIdx.pm        |  35 +-
 lib/PublicInbox/Import.pm              |   4 +
 lib/PublicInbox/LEI.pm                 | 776 +++++++++++++++++++++++++
 lib/PublicInbox/LeiExtinbox.pm         |  52 ++
 lib/PublicInbox/LeiSearch.pm           |  39 ++
 lib/PublicInbox/LeiStore.pm            | 227 ++++++++
 lib/PublicInbox/ManifestJsGz.pm        |   2 +-
 lib/PublicInbox/OnDestroy.pm           |  16 +
 lib/PublicInbox/OverIdx.pm             |  10 +
 lib/PublicInbox/Search.pm              |  65 +--
 lib/PublicInbox/SearchIdx.pm           |  62 +-
 lib/PublicInbox/SearchIdxShard.pm      |  33 ++
 lib/PublicInbox/TestCommon.pm          |   7 +-
 lib/PublicInbox/V2Writable.pm          |  10 +-
 script/lei                             |  76 +++
 t/extsearch.t                          |   3 +-
 t/lei-oneshot.t                        |  25 +
 t/lei.t                                | 306 ++++++++++
 t/lei_store.t                          |  88 +++
 t/on_destroy.t                         |  25 +
 t/www_listing.t                        |   8 +-
 27 files changed, 1843 insertions(+), 82 deletions(-)
 create mode 100644 contrib/completion/lei-completion.bash
 create mode 100755 lei.sh
 create mode 100644 lib/PublicInbox/LEI.pm
 create mode 100644 lib/PublicInbox/LeiExtinbox.pm
 create mode 100644 lib/PublicInbox/LeiSearch.pm
 create mode 100644 lib/PublicInbox/LeiStore.pm
 create mode 100644 lib/PublicInbox/OnDestroy.pm
 create mode 100755 script/lei
 create mode 100644 t/lei-oneshot.t
 create mode 100644 t/lei.t
 create mode 100644 t/lei_store.t
 create mode 100644 t/on_destroy.t

^ permalink raw reply	[relevance 4%]

* [PATCH 04/26] tests: more common JSON module loading
  2020-12-18 12:09  4% [PATCH 00/26] lei: basic UI + IPC work Eric Wong
@ 2020-12-18 12:09  7% ` Eric Wong
  0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2020-12-18 12:09 UTC (permalink / raw)
  To: meta

We'll probably be using JSON more in the future, so make
it easier to require in tests
---
 lib/PublicInbox/ManifestJsGz.pm | 2 +-
 lib/PublicInbox/TestCommon.pm   | 4 ++++
 t/extsearch.t                   | 3 +--
 t/lei.t                         | 3 +--
 t/www_listing.t                 | 8 +++-----
 5 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/lib/PublicInbox/ManifestJsGz.pm b/lib/PublicInbox/ManifestJsGz.pm
index 6d5b57ee..33df020a 100644
--- a/lib/PublicInbox/ManifestJsGz.pm
+++ b/lib/PublicInbox/ManifestJsGz.pm
@@ -11,7 +11,7 @@ use PublicInbox::Config;
 use IO::Compress::Gzip qw(gzip);
 use HTTP::Date qw(time2str);
 
-our $json = PublicInbox::Config::json();
+my $json = PublicInbox::Config::json();
 
 # called by WwwListing
 sub url_regexp {
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index 299b9c6a..2116575b 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -75,6 +75,10 @@ sub require_mods {
 	my $maybe = pop @mods if $mods[-1] =~ /\A[0-9]+\z/;
 	my @need;
 	while (my $mod = shift(@mods)) {
+		if ($mod eq 'json') {
+			$mod = 'Cpanel::JSON::XS||JSON::MaybeXS||'.
+				'JSON||JSON::PP'
+		}
 		if ($mod eq 'Search::Xapian') {
 			if (eval { require PublicInbox::Search } &&
 				PublicInbox::Search::load_xapian()) {
diff --git a/t/extsearch.t b/t/extsearch.t
index fb31b0ab..ffbc10e2 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -8,9 +8,8 @@ use PublicInbox::Config;
 use PublicInbox::Search;
 use PublicInbox::InboxWritable;
 use Fcntl qw(:seek);
-my $json = PublicInbox::Config::json() or plan skip_all => 'JSON missing';
 require_git(2.6);
-require_mods(qw(DBD::SQLite Search::Xapian));
+require_mods(qw(json DBD::SQLite Search::Xapian));
 use_ok 'PublicInbox::ExtSearch';
 use_ok 'PublicInbox::ExtSearchIdx';
 use_ok 'PublicInbox::OverIdx';
diff --git a/t/lei.t b/t/lei.t
index feee9270..02f21322 100644
--- a/t/lei.t
+++ b/t/lei.t
@@ -6,8 +6,7 @@ use v5.10.1;
 use Test::More;
 use PublicInbox::TestCommon;
 use PublicInbox::Config;
-my $json = PublicInbox::Config::json() or plan skip_all => 'JSON missing';
-require_mods(qw(DBD::SQLite Search::Xapian));
+require_mods(qw(json DBD::SQLite Search::Xapian));
 my ($home, $for_destroy) = tmpdir();
 my $opt = { 1 => \(my $out = ''), 2 => \(my $err = '') };
 
diff --git a/t/www_listing.t b/t/www_listing.t
index 63613371..94c1e5bb 100644
--- a/t/www_listing.t
+++ b/t/www_listing.t
@@ -7,14 +7,12 @@ use Test::More;
 use PublicInbox::Spawn qw(which);
 use PublicInbox::TestCommon;
 use PublicInbox::Import;
-require_mods(qw(URI::Escape Plack::Builder Digest::SHA
+require_mods(qw(json URI::Escape Plack::Builder Digest::SHA
 		IO::Compress::Gzip IO::Uncompress::Gunzip HTTP::Tiny));
 require PublicInbox::WwwListing;
 require PublicInbox::ManifestJsGz;
-my $json = do {
-	no warnings 'once';
-	$PublicInbox::ManifestJsGz::json;
-} or plan skip_all => "JSON module missing";
+use PublicInbox::Config;
+my $json = PublicInbox::Config::json();
 
 use_ok 'PublicInbox::Git';
 

^ permalink raw reply related	[relevance 7%]

Results 1-4 of 4 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-12-15 11:47  5% [PATCH/RFC 0/7] lei - Local Email Interface skeleton Eric Wong
2020-12-15 11:47  7% ` [RFC 6/7] tests: more common JSON module loading Eric Wong
2020-12-18 12:09  4% [PATCH 00/26] lei: basic UI + IPC work Eric Wong
2020-12-18 12:09  7% ` [PATCH 04/26] tests: more common JSON module loading Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).