Re: internal format

[Eric Wong <e@80x24.org>]

Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
> >> For timing, I'm curious why you only consider
> >> "git rev-list --objects --all".  Which operation does this corresponds
> >> to in public-inbox and is that really the only one that is
> >> performance-sensitive?
> > That traverses the object graph (same walk used for repacking
> > where bitmaps don't help).
> 
> Yes, I understand what it does in Git, but I wonder why a full traversal
> of the graph is the only/main operation you care about.
> 
> Hmm... I guess your other operations are:
> - lookup by message-id (which is made efficient because you index files
>   by the message-id).
> - everything else is done by keeping another index (from NNTP article
>   number to message-id (or to blob?)), as in the case of Xapian.
> 
> Actually, if you directly index the blobs, you don't really need to
> index your file by message-id (you could keep the index from message-id
> to blobs external, just as is done for Xapian, right?).

Right, storing blob OIDs in Xapian means tree lookups are irrelevant
to read performance.  Since we can rely on Xapian for v2, we can
fix the graph traversal problem by simplifying the trees and
speed up writes by having smaller trees.

The only remaining performance pain point is the overall size of
repos (which we work around by partitioning).