From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id DCACC1F404; Thu, 15 Mar 2018 20:14:20 +0000 (UTC) Date: Thu, 15 Mar 2018 20:14:20 +0000 From: Eric Wong To: Stefan Monnier Cc: meta@public-inbox.org Subject: Re: internal format Message-ID: <20180315201420.GA30804@whir> References: <20180305020754.GA11496@dcvr> <20180315164012.GA20246@whir> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: List-Id: Stefan Monnier wrote: > > v1 or v2? Some of the reasoning for v2 was here: > > https://public-inbox.org/meta/20180209205140.GA11047@dcvr/ > > IIUC, the issues you consider important are: > > - Size > - Time to perform "git rev-list --objects --all" > - Flexibility, e.g. to be able to remove messages. > > For size your benchmarks seem to indicate that as long as it's kept > inside Git, the choice of format doesn't actually affect it > significantly (and this matches my expectations). > Tho I guess it's probably possible to improve on it with enough efforts > (e.g. storing attachments separately, or splitting large messages into > chunks, e.g. like `bup` does), but I doubt it's worth the effort > (especially if you assume that the mailing-list imposes a limit on > message size). Right, I decided splitting big messages wasn't worth the complexity and we leave it up to the (usually reasonable) mail server. > For timing, I'm curious why you only consider > "git rev-list --objects --all". Which operation does this corresponds > to in public-inbox and is that really the only one that is > performance-sensitive? That traverses the object graph (same walk used for repacking where bitmaps don't help). I got it from Peff https://public-inbox.org/git/20160805092805.w3nwv2l6jkbuwlzf@sigill.intra.peff.net/ That's the main thing we can control with repository layout. Large packs are generally a problem with git, so v2 partitions repositories at roughly 1G. > > As for git itself: reliability, ease-of-replication, storage > > efficiency. > > Yes, that part I totally understand (same reason I used Git in BuGit > https://gitlab.com/monnier/bugit). Part of my question was related to > the fact that in BuGit I store the messages in the commit-object rather > than in files (which trivially gives me conflict-free merges as well as > "discussion threads") so I was wondering if it would make sense in the > case of public-inbox to keep the email messages in the commit objects > rather than in files, but since I don't really know which operations are > frequent/important I really have no idea. I thought about storing messages in the commit object, but that would break our current use of Xapian if history rewrites are required for legal reasons. > One thing that strikes me is that you don't seem to use its > "decentralization": IIUC public-inbox always assumes one of the > repositories is the "master" and others are mirrors (or mirrors of > mirrors), so you get efficient "fast-forward" updates, but you > don't do "merges". Right, git merges require the use of pre-established communications channels (e.g. email) to coordinate. I don't believe merging and keeping an authoritative history/order makes sense with public-inbox (more on this later). What's important to decentralization is the "root" can change easily (change of URLs / archival addresses) and all the messages eventually end up replicatable. I consider ease-of-replication and efficiency the building blocks of decentralization. Beyond that, I believe encouraging "pull" via NNTP and discouraging "push" via SMTP with mlmmj/mailman/etc. can eventually lend itself to entirely forkable communities. > This probably means that keeping the email messages in commit objects > wouldn't bring any benefits. > > Also this means that public-inbox could freely rewrite history, for > example (which you'll need to really expunge messages) and just use > "forced updates" in mirrors. We currently store blob SHA-1s in Xapian to avoid tree lookups in git. Having a history rewrite can break an entire chain of unrelated messages if we store commit SHA-1 in Xapian instead of blobs. > Now I'm left wondering what it would mean for something like > public-inbox to support merging. I consider it a waste of effort to maintain an authoritive commit history when archiving mail. There's too many variables when it comes to mail servers and headers and no guarantees on message ordering. Among other things, the last (top) Received: header will surely differ if multiple people start archiving a list independently of each other. The email messages are what's important, so replaying an mbox/Maildir into an importer will get the data that matters (and deduplication checks will avoid redundant mails).