* Re: [PATCH 2/3] search: support "patchid:" prefix (git patch-id --stable)
2022-06-20 19:27 5% ` [PATCH 2/3] search: support "patchid:" prefix (git patch-id --stable) Eric Wong
@ 2022-06-20 20:01 7% ` Kyle Meyer
0 siblings, 0 replies; 3+ results
From: Kyle Meyer @ 2022-06-20 20:01 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
Eric Wong writes:
> This allows easy searching via patch-id from a git commit.
Neat. I played around a bit with this locally, and it seems to be
working nicely.
> diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
> index 523003b3..6f9fdde1 100644
> --- a/lib/PublicInbox/Search.pm
> +++ b/lib/PublicInbox/Search.pm
> @@ -1,4 +1,4 @@
> -# Copyright (C) 2015-2021 all contributors <meta@public-inbox.org>
> +# Copyright (C) all contributors <meta@public-inbox.org>
> # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
> # based on notmuch, but with no concept of folders, files or flags
> #
> @@ -118,9 +118,10 @@ my %bool_pfx_external = (
> dfpre => 'XDFPRE',
> dfpost => 'XDFPOST',
> dfblob => 'XDFPRE XDFPOST',
> + patchid => 'XDFID',
> );
>
> -my $non_quoted_body = 'XNQ XDFN XDFA XDFB XDFHH XDFCTX XDFPRE XDFPOST';
> +my $non_quoted_body = 'XNQ XDFN XDFA XDFB XDFHH XDFCTX XDFPRE XDFPOST XDFID';
> my %prob_prefix = (
> # for mairix compatibility
> s => 'S',
Should an entry for this be added to Search.pm's @HELP?
^ permalink raw reply [relevance 7%]
* [PATCH 0/3] search indexing improvements
@ 2022-06-20 19:27 6% Eric Wong
2022-06-20 19:27 5% ` [PATCH 2/3] search: support "patchid:" prefix (git patch-id --stable) Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2022-06-20 19:27 UTC (permalink / raw)
To: meta
Still stuck on POP3 account manglement, but here's some easy-ish
indexing changes for public-inbox-* tools. These require a full
reindex with either public-inbox-index or public-inbox-extindex,
but old/new indexes should be fully compatible and should be
doable hot:
public-inbox-index --no-fsync --reindex /path/to/v1-or-v2
public-inbox-extindex --no-fsync --reindex --all /path/to/eidx
Will probably take 2 days or so on my own machine.
Note: lei doesn't support reindexing, yet, but will, soon...
Eric Wong (3):
searchidx: use regexp as first arg for `split' op
search: support "patchid:" prefix (git patch-id --stable)
search: do not index base-85 binary patches
MANIFEST | 1 +
TODO | 5 ---
lib/PublicInbox/Search.pm | 5 ++-
lib/PublicInbox/SearchIdx.pm | 75 ++++++++++++++++++++++++++----------
t/data/binary.patch | 20 ++++++++++
t/extsearch.t | 7 +++-
t/search.t | 15 ++++++++
t/v2mda.t | 10 ++++-
8 files changed, 108 insertions(+), 30 deletions(-)
create mode 100644 t/data/binary.patch
^ permalink raw reply [relevance 6%]
* [PATCH 2/3] search: support "patchid:" prefix (git patch-id --stable)
2022-06-20 19:27 6% [PATCH 0/3] search indexing improvements Eric Wong
@ 2022-06-20 19:27 5% ` Eric Wong
2022-06-20 20:01 7% ` Kyle Meyer
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2022-06-20 19:27 UTC (permalink / raw)
To: meta
This allows easy searching via patch-id from a git commit.
Currently, abbreviations are not supported, and it seems
needless to support them since AFAIK (git) doesn't generate
nor resolve abbreviated patch-ids anywhere.
---
TODO | 3 ---
lib/PublicInbox/Search.pm | 5 +++--
lib/PublicInbox/SearchIdx.pm | 15 +++++++++++++++
t/extsearch.t | 7 ++++++-
t/v2mda.t | 10 ++++++++--
5 files changed, 32 insertions(+), 8 deletions(-)
diff --git a/TODO b/TODO
index 5be4b5e3..43eee063 100644
--- a/TODO
+++ b/TODO
@@ -137,9 +137,6 @@ all need to be considered for everything we introduce)
* make "git cat-file --batch" detect unlinked packfiles so we don't
have to restart processes (very long-term)
-* support searching based on `git-patch-id --stable` to improve
- bidirectional mapping of commits <=> emails
-
* linter to check validity of config file
* linter option and WWW endpoint to graph relationships and flows
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index 523003b3..6f9fdde1 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2015-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# based on notmuch, but with no concept of folders, files or flags
#
@@ -118,9 +118,10 @@ my %bool_pfx_external = (
dfpre => 'XDFPRE',
dfpost => 'XDFPOST',
dfblob => 'XDFPRE XDFPOST',
+ patchid => 'XDFID',
);
-my $non_quoted_body = 'XNQ XDFN XDFA XDFB XDFHH XDFCTX XDFPRE XDFPOST';
+my $non_quoted_body = 'XNQ XDFN XDFA XDFB XDFHH XDFCTX XDFPRE XDFPOST XDFID';
my %prob_prefix = (
# for mairix compatibility
s => 'S',
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 50e26050..53ec23a5 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -18,6 +18,7 @@ use PublicInbox::MsgIter;
use PublicInbox::IdxStack;
use Carp qw(croak carp);
use POSIX qw(strftime);
+use Fcntl qw(SEEK_SET);
use Time::Local qw(timegm);
use PublicInbox::OverIdx;
use PublicInbox::Spawn qw(spawn);
@@ -349,6 +350,20 @@ sub index_xapian { # msg_iter callback
defined $s or return;
$_[0]->[0] = $part = undef; # free memory
+ if ($s =~ /^(?:diff|---|\+\+\+) /ms) {
+ open(my $fh, '+>:utf8', undef) or die "open: $!";
+ open(my $eh, '+>', undef) or die "open: $!";
+ $fh->autoflush(1);
+ print $fh $s or die "print: $!";
+ sysseek($fh, 0, SEEK_SET) or die "sysseek: $!";
+ my $id = ($self->{ibx} // $self->{eidx})->git->qx(
+ [qw(patch-id --stable)],
+ {}, { 0 => $fh, 2 => $eh });
+ $id =~ /\A([a-f0-9]{40,})/ and $doc->add_term('XDFID'.$1);
+ seek($eh, 0, SEEK_SET) or die "seek: $!";
+ while (<$eh>) { warn $_ }
+ }
+
# split off quoted and unquoted blocks:
my @sections = PublicInbox::MsgIter::split_quotes($s);
undef $s; # free memory
diff --git a/t/extsearch.t b/t/extsearch.t
index 09cbdabe..2d7375d6 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -314,7 +314,12 @@ if ('reindex catches missed messages') {
is($new->{subject}, $eml->header('Subject'), 'new message added');
$es->{xdb}->reopen;
- my $mset = $es->mset("mid:$new->{mid}");
+ # git patch-id --stable <t/data/0001.patch | awk '{print $1}'
+ my $patchid = '91ee6b761fc7f47cad9f2b09b10489f313eb5b71';
+ my $mset = $es->search->mset("patchid:$patchid");
+ is($mset->size, 1, 'patchid search works');
+
+ $mset = $es->mset("mid:$new->{mid}");
is($mset->size, 1, 'previously unseen, now indexed in Xapian');
ok($im->remove($eml), 'remove new message from v2 inbox');
diff --git a/t/v2mda.t b/t/v2mda.t
index 3dfc569e..8f2f335d 100644
--- a/t/v2mda.t
+++ b/t/v2mda.t
@@ -1,7 +1,8 @@
-# Copyright (C) 2018-2021 all contributors <meta@public-inbox.org>
+#!perl -w
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use v5.10.1;
use strict;
-use warnings;
use Test::More;
use Fcntl qw(SEEK_SET);
use Cwd;
@@ -88,6 +89,11 @@ is($eml->as_string, $mime->as_string, 'injected message');
$pre = $ibx->search->mset_to_smsg($ibx, $pre);
$post = $ibx->search->mset_to_smsg($ibx, $post);
is($post->[0]->{blob}, $pre->[0]->{blob}, 'same message in both cases');
+
+ # git patch-id --stable <t/data/0001.patch | awk '{print $1}'
+ my $patchid = '91ee6b761fc7f47cad9f2b09b10489f313eb5b71';
+ my $mset = $ibx->search->mset("patchid:$patchid");
+ is($mset->size, 1, 'patchid search works');
}
done_testing();
^ permalink raw reply related [relevance 5%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2022-06-20 19:27 6% [PATCH 0/3] search indexing improvements Eric Wong
2022-06-20 19:27 5% ` [PATCH 2/3] search: support "patchid:" prefix (git patch-id --stable) Eric Wong
2022-06-20 20:01 7% ` Kyle Meyer
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).