Since 95acd5901491e4f333f5d2bbeed6fb5e6b53e07c
("searchmsg: add git object ID to doc_data")
the need for having file stored in trees is reduced
since Xapian stores the git object_id and asks git to
retrieve it without doing tree lookups.

So, as long as git knows an object exists, it should be no
problem to just continually replace a single blob at the top
level.

Testing with git@vger history (https://public-inbox.org/git/ at
10066bdacf246bf885f7a11d06a5a87946d7af73 <20180208172153.GA30760@tor.lan>
 by Torsten Bögershausen <tboegi@web.de> @ Thu Feb 8 18:21:53 2018 +0100)


For the 2-2-36 and 2-2-36 trees I took into account naming the
last 16-bytes since that's what git.git uses for
sorting/grouping for packing (pack_name_hash in pack-objects.h)


2-38         914M (baseline)
2-2-2-34     849M
2-2-36       832M
1-file       839M

2-2-2-34 has the most trees, so it's not great in terms of
space. 2-2-36 optimizes deltas better than the 1-file route;
but not significantly so.

It seems optimizing for deltafication isn't worth the effort...


Timing "git rev-list --objects --all |wc -l" reveals much bigger
differences.  Timings are a bit fuzzy since (AFAIK) this is a
shared system, but it's not really a contest:

2-38         ~5 minutes
2-2-2-34     ~30 seconds
2-2-36       ~30 seconds
1-file       ~5 seconds

Smaller trees are way faster :)

The downside of this change is squashing history will no longer
be possible; but it won't be needed for efficiency reasons.

In other words, git scales infinitely well to deep history
depths, but not to breadth of large trees[1].


Marking spam and handling message removals might be a little
trickier as chronology will have to be taken into account...
(will post more on this, later)

I also considered storing messages in the commit object itself
but that would be tougher to reconcile if rewriting git history
is necessary for legal reasons (DMCA).



[1] - we currently process history with --reverse to walk
      in chronological order to ease processing of message
      removals; but --reverse is has an O(n) cost associated
      with it so we should avoid it.  The thread association
      logic should be robust enough to be time-independent.