user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* [v2] one file to rule them all?
@ 2018-02-09 20:51 Eric Wong
  2018-02-15 10:55 ` Eric Wong
  0 siblings, 1 reply; 21+ messages in thread
From: Eric Wong @ 2018-02-09 20:51 UTC (permalink / raw)
  To: meta

[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]

Since 95acd5901491e4f333f5d2bbeed6fb5e6b53e07c
("searchmsg: add git object ID to doc_data")
the need for having file stored in trees is reduced
since Xapian stores the git object_id and asks git to
retrieve it without doing tree lookups.

So, as long as git knows an object exists, it should be no
problem to just continually replace a single blob at the top
level.

Testing with git@vger history (https://public-inbox.org/git/ at
10066bdacf246bf885f7a11d06a5a87946d7af73 <20180208172153.GA30760@tor.lan>
 by Torsten Bögershausen <tboegi@web.de> @ Thu Feb 8 18:21:53 2018 +0100)


For the 2-2-36 and 2-2-36 trees I took into account naming the
last 16-bytes since that's what git.git uses for
sorting/grouping for packing (pack_name_hash in pack-objects.h)


2-38         914M (baseline)
2-2-2-34     849M
2-2-36       832M
1-file       839M

2-2-2-34 has the most trees, so it's not great in terms of
space. 2-2-36 optimizes deltas better than the 1-file route;
but not significantly so.

It seems optimizing for deltafication isn't worth the effort...


Timing "git rev-list --objects --all |wc -l" reveals much bigger
differences.  Timings are a bit fuzzy since (AFAIK) this is a
shared system, but it's not really a contest:

2-38         ~5 minutes
2-2-2-34     ~30 seconds
2-2-36       ~30 seconds
1-file       ~5 seconds

Smaller trees are way faster :)

The downside of this change is squashing history will no longer
be possible; but it won't be needed for efficiency reasons.

In other words, git scales infinitely well to deep history
depths, but not to breadth of large trees[1].


Marking spam and handling message removals might be a little
trickier as chronology will have to be taken into account...
(will post more on this, later)

I also considered storing messages in the commit object itself
but that would be tougher to reconcile if rewriting git history
is necessary for legal reasons (DMCA).



[1] - we currently process history with --reverse to walk
      in chronological order to ease processing of message
      removals; but --reverse is has an O(n) cost associated
      with it so we should avoid it.  The thread association
      logic should be robust enough to be time-independent.

[-- Attachment #2: 1file-convert.perl --]
[-- Type: text/plain, Size: 889 bytes --]

#!/usr/bin/perl -w
# Copyright 2018 The Linux Foundation
# License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use warnings;
use Email::MIME;
use Digest::MD5 qw(md5_hex);

$| = 0;
my $h = '[0-9a-f]';
my $state = '';
my $blob;
my $suff; # 16 bytes for git hashing
while (<STDIN>) {
	if ($_ eq "blob\n") {
		$state = 'blob';
	} elsif (/^commit /) {
		$state = 'commit';
	} elsif ($state eq 'commit') {
		if (m{^(M 100644 :\d+) ${h}{2}/${h}{38}}o) {
			my ($pfx) = ($1);
			print "$pfx msg\n";
			next;
		}
		if (/^data (\d+)/) {
			print $_;
			my $len = $1;
			if ($len) {
				my $tmp;
				read(STDIN, $tmp, $len) or die "read: $!\n";
				print $tmp;
			}
			next;
		}
	} elsif ($state eq 'blob') {
		if (/^data (\d+)/) {
			my $len = $1;
			print $_;
			next unless $len;

			read(STDIN, $blob, $len) or die "read: $!\n";
			print $blob;
			next;
		}
	}
	print $_;
}

[-- Attachment #3: 2-2-36-convert.perl --]
[-- Type: text/plain, Size: 1141 bytes --]

#!/usr/bin/perl -w
# Copyright 2018 The Linux Foundation
# License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use warnings;
use Email::MIME;
use Digest::MD5 qw(md5_hex);

$| = 0;
my $h = '[0-9a-f]';
my $state = '';
my $blob;
my $suff; # 16 bytes for git hashing
while (<STDIN>) {
	if ($_ eq "blob\n") {
		$state = 'blob';
	} elsif (/^commit /) {
		$state = 'commit';
	} elsif ($state eq 'commit') {
		if (m{^(M 100644 :\d+) (${h}{2})/(${h}{2})(${h}{36})}o) {
			my ($pfx, $x2, $x4, $x36) = ($1, $2, $3, $4);
			print "$pfx $x2/$x4/$x36.$suff\n";
			next;
		}
		if (/^data (\d+)/) {
			print $_;
			my $len = $1;
			if ($len) {
				my $tmp;
				read(STDIN, $tmp, $len) or die "read: $!\n";
				print $tmp;
			}
			next;
		}
	} elsif ($state eq 'blob') {
		if (/^data (\d+)/) {
			my $len = $1;
			print $_;
			next unless $len;

			read(STDIN, $blob, $len) or die "read: $!\n";
			print $blob;
			my $mime = Email::MIME->new($blob);
			$suff = $mime->header('Subject');
			utf8::encode($suff);
			# git uses the last 16 bytes for deltas
			$suff = substr(md5_hex(substr($suff, -16)), -16);
			next;
		}
	}
	print $_;
}

[-- Attachment #4: 2-2-2-34-convert.perl --]
[-- Type: text/plain, Size: 1163 bytes --]

#!/usr/bin/perl -w
# Copyright 2018 The Linux Foundation
# License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>
use strict;
use warnings;
use Email::MIME;
use Digest::MD5 qw(md5_hex);

$| = 0;
my $h = '[0-9a-f]';
my $state = '';
my $blob;
my $suff; # 16 bytes for git hashing
while (<STDIN>) {
	if ($_ eq "blob\n") {
		$state = 'blob';
	} elsif (/^commit /) {
		$state = 'commit';
	} elsif ($state eq 'commit') {
		if (m{^(M 100644 :\d+) (${h}{2})/(${h}{2})(${h}{2})(${h}{34})}o) {
			my ($pfx, $x2, $x4, $x6, $x34) = ($1, $2, $3, $4, $5);
			print "$pfx $x2/$x4/$x6/$x34.$suff\n";
			next;
		}
		if (/^data (\d+)/) {
			print $_;
			my $len = $1;
			if ($len) {
				my $tmp;
				read(STDIN, $tmp, $len) or die "read: $!\n";
				print $tmp;
			}
			next;
		}
	} elsif ($state eq 'blob') {
		if (/^data (\d+)/) {
			my $len = $1;
			print $_;
			next unless $len;

			read(STDIN, $blob, $len) or die "read: $!\n";
			print $blob;
			my $mime = Email::MIME->new($blob);
			$suff = $mime->header('Subject');
			utf8::encode($suff);
			# git uses the last 16 bytes for deltas
			$suff = substr(md5_hex(substr($suff, -16)), -16);
			next;
		}
	}
	print $_;
}

^ permalink raw reply	[flat|nested] 21+ messages in thread
end of thread, other threads:[~2018-02-22 21:08 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-09 20:51 [v2] one file to rule them all? Eric Wong
2018-02-15 10:55 ` Eric Wong
2018-02-15 11:08   ` [WIP 0/17] initial v2 work based on one-file tree Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 01/17] AUTHORS: add The Linux Foundation Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 02/17] watch_maildir: allow '-' in mail filename Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 03/17] scripts/import_vger_from_mbox: relax From_ line match slightly Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 04/17] import: stop writing legacy ssoma.index by default Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 05/17] import: begin supporting this without ssoma.lock Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 06/17] import: initial handling for v2 Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 07/17] t/import: test for last_object_id insertion Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 08/17] content_id: add test case Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 09/17] searchmsg: add mid_mime import for _extract_mid Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 10/17] scripts/import_vger_from_mbox: support --dry-run option Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 11/17] import: APIs to support v2 use Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 12/17] search: free up 'Q' prefix for a real unique identifier Eric Wong (Contractor, The Linux Foundation)
2018-02-22 21:08       ` Eric Wong
2018-02-15 11:08     ` [WIP 13/17] searchidx: fix comment around next_thread_id Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 14/17] address: extract more characters from email addresses Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 15/17] import: pass "raw" dates to git-fast-import(1) Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 16/17] scripts/import_vger_from_mbox: use v2 layout for import Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 17/17] import: quiet down warnings from bogus From: lines Eric Wong (Contractor, The Linux Foundation)

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).