Re: Searching via git grep?

[ebiederm@xmission.com (Eric W. Biederman)]

Eric Wong <e@80x24.org> writes:

> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> Have you considered searching public inboxes via git grep?
>
> Not yet...
>
>> For a big server lore.kernel.org with a lot of searches and a lot of
>> clients it might not make sense.  But for home use where searches are
>> rare and the indexes can not be kept in ram, but the mailbox might fit
>> git grep sounds attractive?
>> 
>> I performed a preliminary test and just running git grep manually and
>> I was search all of the git mailling list archive pretty much
>> immediately.
>> 
>> For v1 it is just 'git grep <regexp> HEAD'
>> For v2 it is 'git --rev-list --all | xargs git grep <regexp>'
>> 
>> If this sounds reasonable to you I will take a look at what it takes to
>> wire that up over the next while.
>
> Having something like this on a potentially public-facing web UI
> seems like a liability support-wise(*).
>
> However, I'd be open to having this as a command-line tool.
> Maybe in the scripts/ directory for one-off scripts...
> If I were building a personal mail tool, I could use
> scripts/dupe-finder as a starting point.

My current goal is to make it pleasant to read linux-kernel and possibly
other large archives on my personal machine.  Right now the git
trees for linux-kernel are aboug 6.8G.  Small enough to fit in RAM.

The Xapian indexes are about 63G.  Not small enough to fit in ram.
They are also not fast to update when I pull in a new batch of messages
from linux-kernel.

So I am looking at using git grep as a stand-in for the Xapian indexes
when indexlevel eq 'basic'.

Given my personal ratio of searches to indexing I think I will save
time in doing that.  I don't have it all wired up yet to know if it will
work well, but I suspect it will.

Is it only the web interface where the advanced search functionality is
available?

> (*) I would also caution against having personal mail accessible
> over http://localhost/ on any port without a password; as
> there's attacks on browsers which could hit them.

Good point.  While I may get there that is not my primary focus.

I have for example emails archived from public mailling lists
but because I was the author the mail machine stripped the domain from
from.  This was all about 20 years ago.

Eric