Re: Git-only operation mode

[ebiederm@xmission.com (Eric W. Biederman)]

Eric Wong <e@80x24.org> writes:

> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
>> On Wed, Sep 25, 2019 at 07:45:03PM +0000, Eric Wong wrote:
>> > > Is there a way to run just the archiver component of public-inbox --
>> > > just
>> > > writing to git repos without any of the indexing/frontend bits? One of the
>> > > idle conversations I had with vger.kernel.org folks was to see if we can
>> > > shift the source of truth archive generation to happen at their end. We
>> > > would then clone repositories from them and provide the frontend/search bits
>> > > on lore.kernel.org. From my cursory looking, it would seem that the
>> > > watch/delivery tools always expect to be taking care of xapian/indexing, but
>> > > I think being able to decouple git bits from search/frontend bits would be a
>> > > useful mode or operation.
>> > 
>> > v1 was git-only (that led to scalability problems from big trees).
>> > v2 needs SQLite to do dedupe with indexlevel=basic, but not Xapian,
>> > anymore.  We could get rid of dedupe for v2, but I'm not sure it's
>> > worth it...
>> 
>> Needing sqlite is not a big deal -- compared to the size of the repos,
>> that's reasonably small (e.g. all of lkml git trees are 8.2GB, while
>> msgmap.sqlite3 is 600MB).
>
> Right, it'll also need xap15/over.sqlite* but that's still not too
> big.

For linux-kernel my copy looks to be about 2.4G while the git repos
run 9.1G.

>> Is there an easy way to exclude xapian indexes from being generated during
>> watch/mda runs then?
>
> public-inbox-init --indexlevel=basic <usual args>
>
> Or setting publicinbox.$INBOX_NAME.indexlevel=basic in the
> config file after-the-fact.  You should also be able to remove
> any non-SQLite files from xap15 after-the-fact, if you already
> generated them, too (but I haven't tested that).
>
> I started working on a public-inbox-init manpage the other day,
> still need to finish that...
>
>> A follow-up to that -- is running "public-inbox-index" on the repository
>> after it's been updated enough to update the xapian db? It would be easy to
>> do so as part of the grok-pull post-update hook.
>
> Yes, on a fresh clone.  You'll need to change indexlevel to
> medium or full if it was setup using basic.
>
> I haven't figured out how to use a grok-pull post-update hook to
> run index on my clone of erol, since there's multiple epochs
> per-inbox to deal with.

I have a perl script I use.

Which boils down to:

	git remote update
        public-inbox-index

Which is enough get things up to date.

The tricky bit when the you have a archive like linux-kernel that uses
multiple git repos.

Given that except in the case of bugs article numbers are stable it
should be completely possible do this.

The nasty case is when someone rebases the git history.  I have been
meaning to report this after tracking it down.  To the best of my
knowledge public-inbox-inbox throws out all of the history that was
rebased.  Which can be expensive.   For me it meant I had to drop from
indexlevel=full to indexlevel=basic on linux-kernel.  Because my laptop
machine could not handle the reindexing of all of those messages.

Given that the message numbers remain stable in an event like that it
should be possible to optimize and only reindex things if the blob in
git for a particular message number has changed.  Maybe we already try
and even that is too expensive.  I haven't re-read that code since I
noticed the problem.

Eric