Re: internal format

[Stefan Monnier <monnier@IRO.UMontreal.CA>]

>> For timing, I'm curious why you only consider
>> "git rev-list --objects --all".  Which operation does this corresponds
>> to in public-inbox and is that really the only one that is
>> performance-sensitive?
> That traverses the object graph (same walk used for repacking
> where bitmaps don't help).

Yes, I understand what it does in Git, but I wonder why a full traversal
of the graph is the only/main operation you care about.

Hmm... I guess your other operations are:
- lookup by message-id (which is made efficient because you index files
  by the message-id).
- everything else is done by keeping another index (from NNTP article
  number to message-id (or to blob?)), as in the case of Xapian.

Actually, if you directly index the blobs, you don't really need to
index your file by message-id (you could keep the index from message-id
to blobs external, just as is done for Xapian, right?).

> We currently store blob SHA-1s in Xapian to avoid tree lookups
> in git.  Having a history rewrite can break an entire chain of
> unrelated messages if we store commit SHA-1 in Xapian instead of
> blobs.

Ah, indeed, keeping them as files means that the file's own SHA won't
change when you rewrite history so it makes it much easier to rewrite
history if you rely on this (also probably a lot more efficient within
Git).

>> Now I'm left wondering what it would mean for something like
>> public-inbox to support merging.
> I consider it a waste of effort to maintain an authoritive
> commit history when archiving mail.

Indeed, as long as we're left wondering what good it would do to be able
to merge, we're left with its downsides.


        Stefan