* [PATCH] t/filter_rubylang.t: avoid warning for non-word prefix
@ 2020-03-29 19:32 5% Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2020-03-29 19:32 UTC (permalink / raw)
To: meta
The "-" was never supported by Xapian in the prefix, but
it could still be used to make documentation and URLs more
readable in certain cases.
Fixes: 7909c5f7439777e3 ("altid: warn about non-word prefixes")
---
t/filter_rubylang.t | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/t/filter_rubylang.t b/t/filter_rubylang.t
index 62b3affe..8215b6f1 100644
--- a/t/filter_rubylang.t
+++ b/t/filter_rubylang.t
@@ -27,7 +27,12 @@ SKIP: {
use_ok 'PublicInbox::Inbox';
my ($git_dir, $for_destroy) = tmpdir();
is(mkdir("$git_dir/public-inbox"), 1, "created public-inbox dir");
- my $altid = [ "serial:ruby-core:file=msgmap.sqlite3" ];
+ my $altid = [
+ # 'serial:ruby-core:file=msgmap.sqlite3' can be used here
+ # for documentation purposes, but Xapian ignores everything
+ # up to and including the '-'
+ 'serial:core:file=msgmap.sqlite3'
+ ];
my $ibx = PublicInbox::Inbox->new({ inboxdir => $git_dir,
altid => $altid });
$f = PublicInbox::Filter::RubyLang->new(-inbox => $ibx);
^ permalink raw reply related [relevance 5%]
* [PATCH 10/11] altid: warn about non-word prefixes
2020-03-21 2:03 5% [PATCH 00/11] www: export SQLite altid dumps Eric Wong
@ 2020-03-21 2:03 7% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2020-03-21 2:03 UTC (permalink / raw)
To: meta
We only support searching on prefixes matching /\A\w+\z/ because
Xapian requires ':' to delimit the prefix and splits on spaces
without quotes.
I've also verified Xapian supports multibyte UTF-8 characters,
underscores, and bare numbers as search prefixes, so there's
no need to restrict it beyond what Perl's UTF-8 aware \w
character class offers.
---
lib/PublicInbox/AltId.pm | 2 +-
lib/PublicInbox/Search.pm | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/AltId.pm b/lib/PublicInbox/AltId.pm
index 8ce70e46..3be6c73c 100644
--- a/lib/PublicInbox/AltId.pm
+++ b/lib/PublicInbox/AltId.pm
@@ -22,7 +22,7 @@ sub new {
my ($class, $ibx, $spec, $writable) = @_;
my ($type, $prefix, $query) = split(/:/, $spec, 3);
$type eq 'serial' or die "non-serial not supported, yet\n";
-
+ $prefix =~ /\A\w+\z/ or warn "non-word prefix not searchable\n";
my %params = map {
my ($k, $v) = split(/=/, uri_unescape($_), 2);
$v = '' unless defined $v;
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index 372dc5a7..00dddc6b 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -316,6 +316,8 @@ sub qp {
my $user_pfx = $self->{-user_pfx} = [];
for (@$altid) {
# $_ = 'serial:gmane:/path/to/gmane.msgmap.sqlite3'
+ # note: Xapian supports multibyte UTF-8, /^[0-9]+$/,
+ # and '_' with prefixes matching \w+
/\Aserial:(\w+):/ or next;
my $pfx = $1;
push @$user_pfx, "$pfx:", <<EOF;
^ permalink raw reply related [relevance 7%]
* [PATCH 00/11] www: export SQLite altid dumps
@ 2020-03-21 2:03 5% Eric Wong
2020-03-21 2:03 7% ` [PATCH 10/11] altid: warn about non-word prefixes Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2020-03-21 2:03 UTC (permalink / raw)
To: meta
To improve reproducibility in mirrors, altid dumps can be
exported via "POST /$INBOX_URL/$prefix.sql.gz". $prefix is
something like "gmane" (though the search prefix is "gmane:"
with a colon).
Eric Wong (11):
qspawn: reinstate filter support, add gzip filter
gzipfilter: lazy allocate the deflate context
wwwstream: introduce oneshot API to avoid ->getline
extmsg: use WwwResponse::oneshot
wwwstream: oneshot sets content-length
mbox: need_gzip uses WwwStream::oneshot
qspawn: handle ENOENT (and other errors on exec)
search: clobber -user_pfx on query parser initialization
wwwtext: show thread endpoints info w/ indexlevel=basic
altid: warn about non-word prefixes
www: add endpoint to retrieve altid dumps
MANIFEST | 4 ++
lib/PublicInbox/AltId.pm | 3 +-
lib/PublicInbox/ExtMsg.pm | 4 +-
lib/PublicInbox/GetlineBody.pm | 21 ++++----
lib/PublicInbox/GzipFilter.pm | 59 +++++++++++++++++++++
lib/PublicInbox/Mbox.pm | 16 +++---
lib/PublicInbox/Qspawn.pm | 66 ++++++++++++++----------
lib/PublicInbox/Search.pm | 4 +-
lib/PublicInbox/ViewVCS.pm | 8 +--
lib/PublicInbox/WWW.pm | 14 ++++-
lib/PublicInbox/WwwAltId.pm | 94 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/WwwStream.pm | 29 +++++++++--
lib/PublicInbox/WwwText.pm | 10 +++-
t/gzip_filter.t | 37 +++++++++++++
t/httpd-corner.psgi | 16 ++++++
t/httpd-corner.t | 48 +++++++++++++++++
t/www_altid.t | 83 ++++++++++++++++++++++++++++++
17 files changed, 452 insertions(+), 64 deletions(-)
create mode 100644 lib/PublicInbox/GzipFilter.pm
create mode 100644 lib/PublicInbox/WwwAltId.pm
create mode 100644 t/gzip_filter.t
create mode 100644 t/www_altid.t
^ permalink raw reply [relevance 5%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-03-21 2:03 5% [PATCH 00/11] www: export SQLite altid dumps Eric Wong
2020-03-21 2:03 7% ` [PATCH 10/11] altid: warn about non-word prefixes Eric Wong
2020-03-29 19:32 5% [PATCH] t/filter_rubylang.t: avoid warning for non-word prefix Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).