user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [RFC 01/11] search: implement index_sync to fixup indexer
  2015-08-16  8:37  7% [RFC 0/11] work-in-progress search branch updated Eric Wong
@ 2015-08-16  8:37  7% ` Eric Wong
  0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2015-08-16  8:37 UTC (permalink / raw)
  To: meta; +Cc: Eric Wong

We need to make the indexer executable and installable
while we're at it.
---
 Makefile.PL               |  3 ++-
 lib/PublicInbox/Search.pm | 39 ++++++++++++++++++++++++++++++++++++++-
 public-inbox-index        |  0
 3 files changed, 40 insertions(+), 2 deletions(-)
 mode change 100644 => 100755 public-inbox-index

diff --git a/Makefile.PL b/Makefile.PL
index 1ee1089..f302b7c 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -9,7 +9,8 @@ WriteMakefile(
 	AUTHOR => 'Eric Wong <normalperson@yhbt.net>',
 	ABSTRACT => 'public-inbox server infrastructure',
 	EXE_FILES => [qw/public-inbox-mda public-inbox.cgi
-			public-inbox-learn public-inbox-init/],
+			public-inbox-learn public-inbox-init
+			public-inbox-index/],
 	PREREQ_PM => {
 		# note: we use ssoma(1) and spamc(1), NOT the Perl modules
 		# We also depend on git through ssoma.
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index fe4984e..15bb9f6 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -226,7 +226,6 @@ sub remove_message {
 	} else {
 		$db->commit_transaction;
 	}
-	$db->commit;
 	$doc_id;
 }
 
@@ -536,4 +535,42 @@ sub enquire {
 	$self->{enquire} ||= Search::Xapian::Enquire->new($self->{xdb});
 }
 
+# indexes all unindexed messages
+sub index_sync {
+	my ($self, $git) = @_;
+	my $db = $self->{xdb};
+	my $latest = $db->get_metadata('last_commit');
+	my $range = length $latest ? "$latest..HEAD" : 'HEAD';
+	$latest = undef;
+
+	my $hex = '[a-f0-9]';
+	my $h40 = $hex .'{40}';
+	my $addmsg = qr!^:000000 100644 \S+ ($h40) A\t${hex}{2}/${hex}{38}$!;
+	my $delmsg = qr!^:100644 000000 ($h40) \S+ D\t${hex}{2}/${hex}{38}$!;
+
+	# get indexed messages
+	my @cmd = ('git', "--git-dir=$git->{git_dir}", "log",
+		    qw/--reverse --no-notes --no-color --raw -r --no-abbrev/,
+		    $range);
+
+	my $pid = open(my $log, '-|', @cmd) or
+		die('open` '.join(' ', @cmd) . " pipe failed: $!\n");
+	my $last;
+	while (my $line = <$log>) {
+		if ($line =~ /$addmsg/o) {
+			$self->index_blob($git, $1);
+		} elsif ($line =~ /$delmsg/o) {
+			$self->unindex_blob($git, $1);
+		} elsif ($line =~ /^commit ($h40)/o) {
+			my $commit = $1;
+			if (defined $latest) {
+				$db->set_metadata('last_commit', $latest)
+			}
+			$latest = $commit;
+		}
+	}
+	close $log;
+	$db->set_metadata('last_commit', $latest) if defined $latest;
+}
+
 1;
diff --git a/public-inbox-index b/public-inbox-index
old mode 100644
new mode 100755
-- 
EW


^ permalink raw reply related	[relevance 7%]

* [RFC 0/11] work-in-progress search branch updated
@ 2015-08-16  8:37  7% Eric Wong
  2015-08-16  8:37  7% ` [RFC 01/11] search: implement index_sync to fixup indexer Eric Wong
  0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2015-08-16  8:37 UTC (permalink / raw)
  To: meta

The following changes since commit 2981d6929f400630e575bcc8fc1851bc531e73e9:

  view: consistent ordering of Cc: addresses (2015-08-12 22:41:42 +0000)

are available in the git repository at:

  git://80x24.org/public-inbox search

for you to fetch changes up to eb5f82b20944d780ac3b2ff9a926c023da9468fd:

  implement /s/$SUBJECT_PATH.html lookups (2015-08-16 08:19:53 +0000)

----------------------------------------------------------------
Eric Wong (11):
      initial search backend implementation
      search: implement index_sync to fixup indexer
      extract redundant Message-ID handling code
      search: make search results more OO
      view: display replies in per-message view
      thread: common sorting code
      view: reply threading adjustment
      view: hoist out index_walk function
      www: /t/$MESSAGE_ID.html for threads
      search: remove unnecessary xpfx export
      implement /s/$SUBJECT_PATH.html lookups

 Makefile.PL                  |   3 +-
 lib/PublicInbox/Feed.pm      |  25 +-
 lib/PublicInbox/Hval.pm      |  13 +-
 lib/PublicInbox/MID.pm       |  27 +++
 lib/PublicInbox/Search.pm    | 549 ++++++++++++++++++++++++++++++++++++++++++
 lib/PublicInbox/SearchMsg.pm | 204 ++++++++++++++++
 lib/PublicInbox/Thread.pm    |  14 ++
 lib/PublicInbox/View.pm      | 389 +++++++++++++++++++++++-------
 lib/PublicInbox/WWW.pm       |  87 ++++++-
 public-inbox-index           |  29 +++
 public-inbox.cgi             |   1 +
 t/search.t                   | 239 ++++++++++++++++++
 12 files changed, 1456 insertions(+), 124 deletions(-)
 create mode 100644 lib/PublicInbox/MID.pm
 create mode 100644 lib/PublicInbox/Search.pm
 create mode 100644 lib/PublicInbox/SearchMsg.pm
 create mode 100755 public-inbox-index
 create mode 100644 t/search.t



^ permalink raw reply	[relevance 7%]

Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2015-08-16  8:37  7% [RFC 0/11] work-in-progress search branch updated Eric Wong
2015-08-16  8:37  7% ` [RFC 01/11] search: implement index_sync to fixup indexer Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).