* [PATCH 00/13] lei approxidate, startup fix, --alert
@ 2021-02-08 9:05 7% Eric Wong
2021-02-08 9:05 7% ` [PATCH 12/13] search: use one git-rev-parse process for all dates Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-02-08 9:05 UTC (permalink / raw)
To: meta
I've redone and squashed some changes into PATCH 1/13 which
was posted yesterday.
3/13 (SIGWINCH) is rebase necessary after 1/13,
4/13 (--alert=CMD) is a generalized take on 3/13.
12/13 is...
Eric Wong (13):
lei q: improve remote mboxrd UX + MUA
lei_xsearch: quiet Eml warnings from remote mboxrds
lei q: SIGWINCH process group with the terminal
lei q: support --alert=CMD for early MUA users
tests: favor IPv6
ds: improve add_timer usability
lei: start_pager: drop COLUMNS default
lei: avoid racing on unlink + bind + listen
lei: drop BSD::Resource usage
git: implement date_parse method
lei q: use git approxidate with d:, dt: and rt: ranges
search: use one git-rev-parse process for all dates
spawnpp: raise exception on E2BIG errors
lib/PublicInbox/DS.pm | 10 ++--
lib/PublicInbox/ExtSearchIdx.pm | 5 +-
lib/PublicInbox/FakeInotify.pm | 4 +-
lib/PublicInbox/Git.pm | 10 +++-
lib/PublicInbox/IPC.pm | 8 +--
lib/PublicInbox/LEI.pm | 100 ++++++++++++++++++++++----------
lib/PublicInbox/LeiCurl.pm | 11 +++-
lib/PublicInbox/LeiMirror.pm | 5 +-
lib/PublicInbox/LeiOverview.pm | 6 +-
lib/PublicInbox/LeiQuery.pm | 12 ++--
lib/PublicInbox/LeiToMail.pm | 24 ++++----
lib/PublicInbox/LeiXSearch.pm | 97 ++++++++++++++++++++-----------
lib/PublicInbox/Search.pm | 86 +++++++++++++++++++++++++++
lib/PublicInbox/SpawnPP.pm | 23 ++++++--
lib/PublicInbox/TestCommon.pm | 30 ++++++++--
lib/PublicInbox/Watch.pm | 19 +++---
script/lei | 16 ++---
t/extsearch.t | 2 +-
t/git.t | 17 +++++-
t/httpd-corner.psgi | 2 +-
t/httpd-corner.t | 12 ++--
t/httpd-https.t | 2 +-
t/httpd-unix.t | 7 +--
t/httpd.t | 8 +--
t/imapd-tls.t | 4 +-
t/imapd.t | 8 +--
t/lei-mirror.t | 2 +-
t/nntpd-tls.t | 4 +-
t/nntpd.t | 11 ++--
t/psgi_attach.t | 2 +-
t/psgi_v2.t | 2 +-
t/search.t | 51 ++++++++++++++++
t/solver_git.t | 2 +-
t/v2mirror.t | 3 +-
t/v2writable.t | 3 +-
t/www_altid.t | 2 +-
t/www_listing.t | 3 +-
xt/git-http-backend.t | 4 +-
xt/httpd-async-stream.t | 2 +-
xt/imapd-mbsync-oimap.t | 4 +-
xt/imapd-validate.t | 4 +-
xt/mem-imapd-tls.t | 2 +-
xt/nntpd-validate.t | 3 +-
xt/perf-nntpd.t | 16 ++---
xt/solver.t | 3 +-
45 files changed, 441 insertions(+), 210 deletions(-)
^ permalink raw reply [relevance 7%]
* [PATCH 12/13] search: use one git-rev-parse process for all dates
2021-02-08 9:05 7% [PATCH 00/13] lei approxidate, startup fix, --alert Eric Wong
@ 2021-02-08 9:05 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-02-08 9:05 UTC (permalink / raw)
To: meta
This is necessary to avoid slowdowns with pathological cases
with many dates in the query, since each rev-parse invocation
takes ~5ms.
This is immeasurably slower with one open-ended range, but
already faster with any closed range featuring two dates which
require parsing via git.
---
lib/PublicInbox/Git.pm | 6 ++--
lib/PublicInbox/Search.pm | 63 +++++++++++++++++++++++++--------------
t/git.t | 16 +++++-----
3 files changed, 54 insertions(+), 31 deletions(-)
diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index 9207962b..ac7ff267 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -377,8 +377,10 @@ sub qx {
}
sub date_parse {
- my $d = $_[0]->qx('rev-parse', "--since=$_[1]");
- substr($d, length('--max-age='), -1)
+ my $self = shift;
+ map {
+ substr($_, length('--max-age='), -1)
+ } $self->qx('rev-parse', map { "--since=$_" } @_);
}
# check_async and cat_async may trigger the other, so ensure they're
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index f42d70e3..aa737d63 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -265,8 +265,10 @@ sub reopen {
# and neither the SWIG nor XS bindings allow us to use custom code
# to parse dates (and libgit2 doesn't expose git__date_parse, either,
# so we're running git-rev-parse(1)).
-sub date_range {
- my ($git, $pfx, $range) = @_;
+# This replaces things we need to send to $git->date_parse with
+# "\0".$strftime_format.['+'|$idx]."\0" placeholders
+sub date_parse_prepare {
+ my ($to_parse, $pfx, $range) = @_;
# are we inside a parenthesized statement?
my $end = $range =~ s/([\)\s]*)\z// ? $1 : '';
my @r = split(/\.\./, $range, 2);
@@ -275,55 +277,72 @@ sub date_range {
# n.b. git doesn't do YYYYMMDD w/o '-', it needs YYYY-MM-DD
if ($pfx eq 'd') {
if (!defined($r[1])) {
- $r[0] =~ s/\A([0-9]{4})([0-9]{2})([0-9]{2})\z/$1-$2-$3/;
- $r[0] = $git->date_parse($r[0]);
- $r[1] = $r[0] + 86400;
- for my $x (@r) {
- $x = strftime('%Y%m%d', gmtime($x));
+ if ($r[0] =~ /\A([0-9]{4})([0-9]{2})([0-9]{2})\z/) {
+ push @$to_parse, "$1-$2-$3";
+ # we could've handled as-is, but we need
+ # to parse anyways for "d+" below
+ } else {
+ push @$to_parse, $r[0];
}
+ $r[0] = "\0%Y%m%d$#$to_parse\0";
+ $r[1] = "\0%Y%m%d+\0";
} else {
for my $x (@r) {
next if $x eq '' || $x =~ /\A[0-9]{8}\z/;
- $x = strftime('%Y%m%d',
- gmtime($git->date_parse($x)));
+ push @$to_parse, $x;
+ $x = "\0%Y%m%d$#$to_parse\0";
}
}
} elsif ($pfx eq 'dt') {
if (!defined($r[1])) { # git needs gaps and not /\d{14}/
- $r[0] =~ s/\A([0-9]{4})([0-9]{2})([0-9]{2})
- ([0-9]{2})([0-9]{2})([0-9]{2})\z
- /$1-$2-$3 $4:$5:$6/x;
- $r[0] = $git->date_parse($r[0]);
- $r[1] = $r[0] + 86400;
- for my $x (@r) {
- $x = strftime('%Y%m%d%H%M%S', gmtime($x));
+ if ($r[0] =~ /\A([0-9]{4})([0-9]{2})([0-9]{2})
+ ([0-9]{2})([0-9]{2})([0-9]{2})\z/x) {
+ push @$to_parse, "$1-$2-$3 $4:$5:$6";
+ } else {
+ push @$to_parse, $r[0];
}
+ $r[0] = "\0%Y%m%d%H%M%S$#$to_parse\0";
+ $r[1] = "\0%Y%m%d%H%M%S+\0";
} else {
for my $x (@r) {
next if $x eq '' || $x =~ /\A[0-9]{14}\z/;
- $x = strftime('%Y%m%d%H%M%S',
- gmtime($git->date_parse($x)));
+ push @$to_parse, $x;
+ $x = "\0%Y%m%d%H%M%S$#$to_parse\0";
}
}
} else { # "rt", let git interpret "YYYY", deal with Y10K later :P
for my $x (@r) {
next if $x eq '' || $x =~ /\A[0-9]{5,}\z/;
- $x = $git->date_parse($x);
+ push @$to_parse, $x;
+ $x = "\0%s$#$to_parse\0";
}
- $r[1] //= $r[0] + 86400;
+ $r[1] //= "\0%s+\0";
}
"$pfx:".join('..', @r).$end;
}
+# n.b. argv never has NUL, though we'll need to filter it out
+# if this $argv isn't from a command execution
sub query_argv_to_string {
my (undef, $git, $argv) = @_;
- join(' ', map {;
- if (s!\b(d|rt|dt):(.+)\z!date_range($git, $1, $2)!sge) {
+ my $to_parse;
+ my $tmp = join(' ', map {;
+ if (s!\b(d|rt|dt):([[:print:]]+)\z!date_parse_prepare(
+ $to_parse //= [], $1, $2)!sge) {
$_;
} else {
/\s/ ? (s/\A(\w+:)// ? qq{$1"$_"} : qq{"$_}) : $_
}
} @$argv);
+ # git-rev-parse can handle any number of args up to system
+ # limits (around (4096*32) bytes on Linux).
+ if ($to_parse) {
+ my @r = $git->date_parse(@$to_parse);
+ my $i;
+ $tmp =~ s/\0(%[%YmdHMSs]+)([0-9\+]+)\0/strftime($1,
+ gmtime($2 eq '+' ? ($r[$i]+86400) : $r[$i=$2+0]))/sge;
+ }
+ $tmp
}
# read-only
diff --git a/t/git.t b/t/git.t
index 7b950d88..4a45bbaf 100644
--- a/t/git.t
+++ b/t/git.t
@@ -20,14 +20,16 @@ use_ok 'PublicInbox::Git';
}
{
my $git = PublicInbox::Git->new($dir);
- my $s = $git->date_parse('1970-01-01T00:00:00Z');
- is($s, 0, 'parsed epoch');
+ my @s = $git->date_parse('1970-01-01T00:00:00Z');
+ is($s[0], 0, 'parsed epoch');
local $ENV{TZ} = 'UTC';
- $s = $git->date_parse('1993-10-02 01:02:09');
- is(strftime('%Y-%m-%dT%H:%M:%SZ', gmtime($s)), '1993-10-02T01:02:09Z',
- 'round trips');
- $s = $git->date_parse('1993-10-02');
- is(strftime('%Y-%m-%d', gmtime($s)), '1993-10-02',
+ @s = $git->date_parse('1993-10-02 01:02:09', '2010-10-02 01:03:04');
+ is(strftime('%Y-%m-%dT%H:%M:%SZ', gmtime($s[0])),
+ '1993-10-02T01:02:09Z', 'round trips');
+ is(strftime('%Y-%m-%dT%H:%M:%SZ', gmtime($s[1])),
+ '2010-10-02T01:03:04Z', '2nd arg round trips');
+ @s = $git->date_parse('1993-10-02');
+ is(strftime('%Y-%m-%d', gmtime($s[0])), '1993-10-02',
'round trips date-only');
}
^ permalink raw reply related [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-02-08 9:05 7% [PATCH 00/13] lei approxidate, startup fix, --alert Eric Wong
2021-02-08 9:05 7% ` [PATCH 12/13] search: use one git-rev-parse process for all dates Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).