* [v2] one file to rule them all?
@ 2018-02-09 20:51 5% Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2018-02-09 20:51 UTC (permalink / raw)
To: meta
[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]
Since 95acd5901491e4f333f5d2bbeed6fb5e6b53e07c
("searchmsg: add git object ID to doc_data")
the need for having file stored in trees is reduced
since Xapian stores the git object_id and asks git to
retrieve it without doing tree lookups.
So, as long as git knows an object exists, it should be no
problem to just continually replace a single blob at the top
level.
Testing with git@vger history (https://public-inbox.org/git/ at
10066bdacf246bf885f7a11d06a5a87946d7af73 <20180208172153.GA30760@tor.lan>
by Torsten Bögershausen <tboegi@web.de> @ Thu Feb 8 18:21:53 2018 +0100)
For the 2-2-36 and 2-2-36 trees I took into account naming the
last 16-bytes since that's what git.git uses for
sorting/grouping for packing (pack_name_hash in pack-objects.h)
2-38 914M (baseline)
2-2-2-34 849M
2-2-36 832M
1-file 839M
2-2-2-34 has the most trees, so it's not great in terms of
space. 2-2-36 optimizes deltas better than the 1-file route;
but not significantly so.
It seems optimizing for deltafication isn't worth the effort...
Timing "git rev-list --objects --all |wc -l" reveals much bigger
differences. Timings are a bit fuzzy since (AFAIK) this is a
shared system, but it's not really a contest:
2-38 ~5 minutes
2-2-2-34 ~30 seconds
2-2-36 ~30 seconds
1-file ~5 seconds
Smaller trees are way faster :)
The downside of this change is squashing history will no longer
be possible; but it won't be needed for efficiency reasons.
In other words, git scales infinitely well to deep history
depths, but not to breadth of large trees[1].
Marking spam and handling message removals might be a little
trickier as chronology will have to be taken into account...
(will post more on this, later)
I also considered storing messages in the commit object itself
but that would be tougher to reconcile if rewriting git history
is necessary for legal reasons (DMCA).
[1] - we currently process history with --reverse to walk
in chronological order to ease processing of message
removals; but --reverse is has an O(n) cost associated
with it so we should avoid it. The thread association
logic should be robust enough to be time-independent.
[-- Attachment #2: 1file-convert.perl --]
[-- Type: text/plain, Size: 889 bytes --]
#!/usr/bin/perl -w
# Copyright 2018 The Linux Foundation
# License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use warnings;
use Email::MIME;
use Digest::MD5 qw(md5_hex);
$| = 0;
my $h = '[0-9a-f]';
my $state = '';
my $blob;
my $suff; # 16 bytes for git hashing
while (<STDIN>) {
if ($_ eq "blob\n") {
$state = 'blob';
} elsif (/^commit /) {
$state = 'commit';
} elsif ($state eq 'commit') {
if (m{^(M 100644 :\d+) ${h}{2}/${h}{38}}o) {
my ($pfx) = ($1);
print "$pfx msg\n";
next;
}
if (/^data (\d+)/) {
print $_;
my $len = $1;
if ($len) {
my $tmp;
read(STDIN, $tmp, $len) or die "read: $!\n";
print $tmp;
}
next;
}
} elsif ($state eq 'blob') {
if (/^data (\d+)/) {
my $len = $1;
print $_;
next unless $len;
read(STDIN, $blob, $len) or die "read: $!\n";
print $blob;
next;
}
}
print $_;
}
[-- Attachment #3: 2-2-36-convert.perl --]
[-- Type: text/plain, Size: 1141 bytes --]
#!/usr/bin/perl -w
# Copyright 2018 The Linux Foundation
# License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use warnings;
use Email::MIME;
use Digest::MD5 qw(md5_hex);
$| = 0;
my $h = '[0-9a-f]';
my $state = '';
my $blob;
my $suff; # 16 bytes for git hashing
while (<STDIN>) {
if ($_ eq "blob\n") {
$state = 'blob';
} elsif (/^commit /) {
$state = 'commit';
} elsif ($state eq 'commit') {
if (m{^(M 100644 :\d+) (${h}{2})/(${h}{2})(${h}{36})}o) {
my ($pfx, $x2, $x4, $x36) = ($1, $2, $3, $4);
print "$pfx $x2/$x4/$x36.$suff\n";
next;
}
if (/^data (\d+)/) {
print $_;
my $len = $1;
if ($len) {
my $tmp;
read(STDIN, $tmp, $len) or die "read: $!\n";
print $tmp;
}
next;
}
} elsif ($state eq 'blob') {
if (/^data (\d+)/) {
my $len = $1;
print $_;
next unless $len;
read(STDIN, $blob, $len) or die "read: $!\n";
print $blob;
my $mime = Email::MIME->new($blob);
$suff = $mime->header('Subject');
utf8::encode($suff);
# git uses the last 16 bytes for deltas
$suff = substr(md5_hex(substr($suff, -16)), -16);
next;
}
}
print $_;
}
[-- Attachment #4: 2-2-2-34-convert.perl --]
[-- Type: text/plain, Size: 1163 bytes --]
#!/usr/bin/perl -w
# Copyright 2018 The Linux Foundation
# License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use warnings;
use Email::MIME;
use Digest::MD5 qw(md5_hex);
$| = 0;
my $h = '[0-9a-f]';
my $state = '';
my $blob;
my $suff; # 16 bytes for git hashing
while (<STDIN>) {
if ($_ eq "blob\n") {
$state = 'blob';
} elsif (/^commit /) {
$state = 'commit';
} elsif ($state eq 'commit') {
if (m{^(M 100644 :\d+) (${h}{2})/(${h}{2})(${h}{2})(${h}{34})}o) {
my ($pfx, $x2, $x4, $x6, $x34) = ($1, $2, $3, $4, $5);
print "$pfx $x2/$x4/$x6/$x34.$suff\n";
next;
}
if (/^data (\d+)/) {
print $_;
my $len = $1;
if ($len) {
my $tmp;
read(STDIN, $tmp, $len) or die "read: $!\n";
print $tmp;
}
next;
}
} elsif ($state eq 'blob') {
if (/^data (\d+)/) {
my $len = $1;
print $_;
next unless $len;
read(STDIN, $blob, $len) or die "read: $!\n";
print $blob;
my $mime = Email::MIME->new($blob);
$suff = $mime->header('Subject');
utf8::encode($suff);
# git uses the last 16 bytes for deltas
$suff = substr(md5_hex(substr($suff, -16)), -16);
next;
}
}
print $_;
}
^ permalink raw reply [relevance 5%]
* [PATCH] searchmsg: add git object ID to doc_data
@ 2016-08-05 1:03 7% Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2016-08-05 1:03 UTC (permalink / raw)
To: meta
Doing git tree lookups based on the SHA-1 of the Message-ID
is expensive as trees get larger, instead, use the SHA-1
object ID directly. This drastically reduces the amount
of time spent in the "git cat-file --batch" process for
fetching the /$INBOX/all.mbox.gz endpoint on the ~800MB
git@vger.kernel.org mirror
This retains backwards compatibility and allows existing
indices to be transparently upgraded without performance
degradation.
---
lib/PublicInbox/Inbox.pm | 12 ++++++++++++
lib/PublicInbox/Mbox.pm | 2 +-
lib/PublicInbox/SearchIdx.pm | 24 +++++++++++++-----------
lib/PublicInbox/SearchMsg.pm | 20 ++++++++++++++++----
4 files changed, 42 insertions(+), 16 deletions(-)
diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index 4fbbb52..e552cd4 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -162,6 +162,18 @@ sub msg_by_path ($$;$) {
$str;
}
+sub msg_by_smsg ($$;$) {
+ my ($self, $smsg, $ref) = @_;
+
+ # backwards compat to fallback to msg_by_mid
+ # TODO: remove if we bump SCHEMA_VERSION in Search.pm:
+ defined(my $blob = $smsg->blob) or return msg_by_mid($self, $smsg->mid);
+
+ my $str = git($self)->cat_file($blob, $ref);
+ $$str =~ s/\A[\r\n]*From [^\r\n]*\r?\n//s if $str;
+ $str;
+}
+
sub path_check {
my ($self, $path) = @_;
git($self)->check('HEAD:'.$path);
diff --git a/lib/PublicInbox/Mbox.pm b/lib/PublicInbox/Mbox.pm
index d2c0954..1e3de5b 100644
--- a/lib/PublicInbox/Mbox.pm
+++ b/lib/PublicInbox/Mbox.pm
@@ -129,7 +129,7 @@ sub getline {
my $gz = $self->{gz};
do {
while (defined(my $smsg = shift @{$self->{msgs}})) {
- my $msg = eval { $ibx->msg_by_mid($smsg->mid) } or next;
+ my $msg = eval { $ibx->msg_by_smsg($smsg) } or next;
$msg = Email::Simple->new($msg);
$gz->write(PublicInbox::Mbox::msg_str($ctx, $msg));
my $bref = $self->{buf};
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index f0a3687..f8249c5 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -91,7 +91,7 @@ sub add_val {
}
sub add_message {
- my ($self, $mime, $bytes, $num) = @_; # mime = Email::MIME object
+ my ($self, $mime, $bytes, $num, $blob) = @_; # mime = Email::MIME object
my $db = $self->{xdb};
my ($doc_id, $old_tid);
@@ -170,7 +170,7 @@ sub add_message {
});
link_message($self, $smsg, $old_tid);
- $doc->set_data($smsg->to_doc_data);
+ $doc->set_data($smsg->to_doc_data($blob));
if (defined $doc_id) {
$db->replace_document($doc_id, $doc);
} else {
@@ -279,8 +279,8 @@ sub link_message {
}
sub index_blob {
- my ($self, $git, $mime, $bytes, $num) = @_;
- $self->add_message($mime, $bytes, $num);
+ my ($self, $git, $mime, $bytes, $num, $blob) = @_;
+ $self->add_message($mime, $bytes, $num, $blob);
}
sub unindex_blob {
@@ -300,9 +300,9 @@ sub unindex_mm {
}
sub index_mm2 {
- my ($self, $git, $mime, $bytes) = @_;
+ my ($self, $git, $mime, $bytes, $blob) = @_;
my $num = $self->{mm}->num_for(mid_clean(mid_mime($mime)));
- index_blob($self, $git, $mime, $bytes, $num);
+ index_blob($self, $git, $mime, $bytes, $num, $blob);
}
sub unindex_mm2 {
@@ -312,9 +312,9 @@ sub unindex_mm2 {
}
sub index_both {
- my ($self, $git, $mime, $bytes) = @_;
+ my ($self, $git, $mime, $bytes, $blob) = @_;
my $num = index_mm($self, $git, $mime);
- index_blob($self, $git, $mime, $bytes, $num);
+ index_blob($self, $git, $mime, $bytes, $num, $blob);
}
sub unindex_both {
@@ -355,10 +355,12 @@ sub rlog {
my $line;
while (defined($line = <$log>)) {
if ($line =~ /$addmsg/o) {
- my $mime = do_cat_mail($git, $1, \$bytes) or next;
- $add_cb->($self, $git, $mime, $bytes);
+ my $blob = $1;
+ my $mime = do_cat_mail($git, $blob, \$bytes) or next;
+ $add_cb->($self, $git, $mime, $bytes, $blob);
} elsif ($line =~ /$delmsg/o) {
- my $mime = do_cat_mail($git, $1) or next;
+ my $blob = $1;
+ my $mime = do_cat_mail($git, $blob) or next;
$del_cb->($self, $git, $mime);
} elsif ($line =~ /^commit ($h40)/o) {
if (defined $max && --$max <= 0) {
diff --git a/lib/PublicInbox/SearchMsg.pm b/lib/PublicInbox/SearchMsg.pm
index 4b0b645..9d873c4 100644
--- a/lib/PublicInbox/SearchMsg.pm
+++ b/lib/PublicInbox/SearchMsg.pm
@@ -38,7 +38,7 @@ sub load_doc {
my $data = $doc->get_data or return;
my $ts = get_val($doc, &PublicInbox::Search::TS);
utf8::decode($data);
- my ($subj, $from, $refs, $to, $cc) = split(/\n/, $data);
+ my ($subj, $from, $refs, $to, $cc, $blob) = split(/\n/, $data);
bless {
doc => $doc,
subject => $subj,
@@ -47,6 +47,7 @@ sub load_doc {
references => $refs,
to => $to,
cc => $cc,
+ blob => $blob,
}, $class;
}
@@ -105,9 +106,11 @@ sub ts {
}
sub to_doc_data {
- my ($self) = @_;
- join("\n", $self->subject, $self->from, $self->references,
- $self->to, $self->cc);
+ my ($self, $blob) = @_;
+ my @rows = ($self->subject, $self->from, $self->references,
+ $self->to, $self->cc);
+ push @rows, $blob if defined $blob;
+ join("\n", @rows);
}
sub references {
@@ -185,6 +188,15 @@ sub mid ($;$) {
sub _extract_mid { mid_clean(mid_mime($_[0]->mime)) }
+sub blob {
+ my ($self, $x40) = @_;
+ if (defined $x40) {
+ $self->{blob} = $x40;
+ } else {
+ $self->{blob};
+ }
+}
+
sub mime {
my ($self, $mime) = @_;
if (defined $mime) {
--
EW
^ permalink raw reply related [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2016-08-05 1:03 7% [PATCH] searchmsg: add git object ID to doc_data Eric Wong
2018-02-09 20:51 5% [v2] one file to rule them all? Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).