From: Eric Wong <e@80x24.org> To: meta@public-inbox.org Subject: [v2] one file to rule them all? Date: Fri, 9 Feb 2018 20:51:40 +0000 Message-ID: <20180209205140.GA11047@dcvr> (raw) [-- Attachment #1: Type: text/plain, Size: 2218 bytes --] Since 95acd5901491e4f333f5d2bbeed6fb5e6b53e07c ("searchmsg: add git object ID to doc_data") the need for having file stored in trees is reduced since Xapian stores the git object_id and asks git to retrieve it without doing tree lookups. So, as long as git knows an object exists, it should be no problem to just continually replace a single blob at the top level. Testing with git@vger history (https://public-inbox.org/git/ at 10066bdacf246bf885f7a11d06a5a87946d7af73 <20180208172153.GA30760@tor.lan> by Torsten Bögershausen <tboegi@web.de> @ Thu Feb 8 18:21:53 2018 +0100) For the 2-2-36 and 2-2-36 trees I took into account naming the last 16-bytes since that's what git.git uses for sorting/grouping for packing (pack_name_hash in pack-objects.h) 2-38 914M (baseline) 2-2-2-34 849M 2-2-36 832M 1-file 839M 2-2-2-34 has the most trees, so it's not great in terms of space. 2-2-36 optimizes deltas better than the 1-file route; but not significantly so. It seems optimizing for deltafication isn't worth the effort... Timing "git rev-list --objects --all |wc -l" reveals much bigger differences. Timings are a bit fuzzy since (AFAIK) this is a shared system, but it's not really a contest: 2-38 ~5 minutes 2-2-2-34 ~30 seconds 2-2-36 ~30 seconds 1-file ~5 seconds Smaller trees are way faster :) The downside of this change is squashing history will no longer be possible; but it won't be needed for efficiency reasons. In other words, git scales infinitely well to deep history depths, but not to breadth of large trees[1]. Marking spam and handling message removals might be a little trickier as chronology will have to be taken into account... (will post more on this, later) I also considered storing messages in the commit object itself but that would be tougher to reconcile if rewriting git history is necessary for legal reasons (DMCA). [1] - we currently process history with --reverse to walk in chronological order to ease processing of message removals; but --reverse is has an O(n) cost associated with it so we should avoid it. The thread association logic should be robust enough to be time-independent. [-- Attachment #2: 1file-convert.perl --] [-- Type: text/plain, Size: 889 bytes --] #!/usr/bin/perl -w # Copyright 2018 The Linux Foundation # License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt> use strict; use warnings; use Email::MIME; use Digest::MD5 qw(md5_hex); $| = 0; my $h = '[0-9a-f]'; my $state = ''; my $blob; my $suff; # 16 bytes for git hashing while (<STDIN>) { if ($_ eq "blob\n") { $state = 'blob'; } elsif (/^commit /) { $state = 'commit'; } elsif ($state eq 'commit') { if (m{^(M 100644 :\d+) ${h}{2}/${h}{38}}o) { my ($pfx) = ($1); print "$pfx msg\n"; next; } if (/^data (\d+)/) { print $_; my $len = $1; if ($len) { my $tmp; read(STDIN, $tmp, $len) or die "read: $!\n"; print $tmp; } next; } } elsif ($state eq 'blob') { if (/^data (\d+)/) { my $len = $1; print $_; next unless $len; read(STDIN, $blob, $len) or die "read: $!\n"; print $blob; next; } } print $_; } [-- Attachment #3: 2-2-36-convert.perl --] [-- Type: text/plain, Size: 1141 bytes --] #!/usr/bin/perl -w # Copyright 2018 The Linux Foundation # License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt> use strict; use warnings; use Email::MIME; use Digest::MD5 qw(md5_hex); $| = 0; my $h = '[0-9a-f]'; my $state = ''; my $blob; my $suff; # 16 bytes for git hashing while (<STDIN>) { if ($_ eq "blob\n") { $state = 'blob'; } elsif (/^commit /) { $state = 'commit'; } elsif ($state eq 'commit') { if (m{^(M 100644 :\d+) (${h}{2})/(${h}{2})(${h}{36})}o) { my ($pfx, $x2, $x4, $x36) = ($1, $2, $3, $4); print "$pfx $x2/$x4/$x36.$suff\n"; next; } if (/^data (\d+)/) { print $_; my $len = $1; if ($len) { my $tmp; read(STDIN, $tmp, $len) or die "read: $!\n"; print $tmp; } next; } } elsif ($state eq 'blob') { if (/^data (\d+)/) { my $len = $1; print $_; next unless $len; read(STDIN, $blob, $len) or die "read: $!\n"; print $blob; my $mime = Email::MIME->new($blob); $suff = $mime->header('Subject'); utf8::encode($suff); # git uses the last 16 bytes for deltas $suff = substr(md5_hex(substr($suff, -16)), -16); next; } } print $_; } [-- Attachment #4: 2-2-2-34-convert.perl --] [-- Type: text/plain, Size: 1163 bytes --] #!/usr/bin/perl -w # Copyright 2018 The Linux Foundation # License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt> use strict; use warnings; use Email::MIME; use Digest::MD5 qw(md5_hex); $| = 0; my $h = '[0-9a-f]'; my $state = ''; my $blob; my $suff; # 16 bytes for git hashing while (<STDIN>) { if ($_ eq "blob\n") { $state = 'blob'; } elsif (/^commit /) { $state = 'commit'; } elsif ($state eq 'commit') { if (m{^(M 100644 :\d+) (${h}{2})/(${h}{2})(${h}{2})(${h}{34})}o) { my ($pfx, $x2, $x4, $x6, $x34) = ($1, $2, $3, $4, $5); print "$pfx $x2/$x4/$x6/$x34.$suff\n"; next; } if (/^data (\d+)/) { print $_; my $len = $1; if ($len) { my $tmp; read(STDIN, $tmp, $len) or die "read: $!\n"; print $tmp; } next; } } elsif ($state eq 'blob') { if (/^data (\d+)/) { my $len = $1; print $_; next unless $len; read(STDIN, $blob, $len) or die "read: $!\n"; print $blob; my $mime = Email::MIME->new($blob); $suff = $mime->header('Subject'); utf8::encode($suff); # git uses the last 16 bytes for deltas $suff = substr(md5_hex(substr($suff, -16)), -16); next; } } print $_; }
next reply other threads:[~2018-02-09 20:51 UTC|newest] Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-02-09 20:51 Eric Wong [this message] 2018-02-15 10:55 ` Eric Wong 2018-02-15 11:08 ` [WIP 0/17] initial v2 work based on one-file tree Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 01/17] AUTHORS: add The Linux Foundation Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 02/17] watch_maildir: allow '-' in mail filename Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 03/17] scripts/import_vger_from_mbox: relax From_ line match slightly Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 04/17] import: stop writing legacy ssoma.index by default Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 05/17] import: begin supporting this without ssoma.lock Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 06/17] import: initial handling for v2 Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 07/17] t/import: test for last_object_id insertion Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 08/17] content_id: add test case Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 09/17] searchmsg: add mid_mime import for _extract_mid Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 10/17] scripts/import_vger_from_mbox: support --dry-run option Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 11/17] import: APIs to support v2 use Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 12/17] search: free up 'Q' prefix for a real unique identifier Eric Wong (Contractor, The Linux Foundation) 2018-02-22 21:08 ` Eric Wong 2018-02-15 11:08 ` [WIP 13/17] searchidx: fix comment around next_thread_id Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 14/17] address: extract more characters from email addresses Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 15/17] import: pass "raw" dates to git-fast-import(1) Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 16/17] scripts/import_vger_from_mbox: use v2 layout for import Eric Wong (Contractor, The Linux Foundation) 2018-02-15 11:08 ` [WIP 17/17] import: quiet down warnings from bogus From: lines Eric Wong (Contractor, The Linux Foundation)
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://public-inbox.org/README * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20180209205140.GA11047@dcvr \ --to=e@80x24.org \ --cc=meta@public-inbox.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
user/dev discussion of public-inbox itself This inbox may be cloned and mirrored by anyone: git clone --mirror https://public-inbox.org/meta git clone --mirror http://czquwvybam4bgbro.onion/meta git clone --mirror http://hjrcffqmbrq6wope.onion/meta git clone --mirror http://ou63pmih66umazou.onion/meta # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \ meta@public-inbox.org public-inbox-index meta Example config snippet for mirrors. Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta nntp://news.gmane.io/gmane.mail.public-inbox.general note: .onion URLs require Tor: https://www.torproject.org/ code repositories for the project(s) associated with this inbox: https://80x24.org/public-inbox.git AGPL code for this site: git clone https://public-inbox.org/public-inbox.git