Re: Searching via git grep?

[ebiederm@xmission.com (Eric W. Biederman)]

ebiederm@xmission.com (Eric W. Biederman) writes:

> Eric Wong <e@80x24.org> writes:
>
>> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>>> My current goal is to make it pleasant to read linux-kernel and possibly
>>> other large archives on my personal machine.  Right now the git
>>> trees for linux-kernel are aboug 6.8G.  Small enough to fit in RAM.
>>> 
>>> The Xapian indexes are about 63G.  Not small enough to fit in ram.
>>> They are also not fast to update when I pull in a new batch of messages
>>> from linux-kernel.
>>
>> Interesting, how long does it take to do an incremental index
>> medium/full for you?  Setting XAPIAN_FLUSH_THRESHOLD after my
>> patch yesterday should help noticeably, especially if you're on
>> HDD.
>
> For a small sample less than a days worth of lkml messages
> I get:
>
> $ git --git-dir git/6.git/ fetch
> Enter passphrase for key '/home/eric/.ssh/id_rsa': 
> Fetching origin
>> From https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/lkml/6
>    35280da650..0a97acb7e7  master     -> master
> remote: Counting objects: 1791, done.
> remote: Compressing objects: 100% (1085/1085), done.
> remote: Total 1791 (delta 109), reused 1791 (delta 109)
> Receiving objects: 100% (1791/1791), 1.94 MiB | 1.98 MiB/s, done.
> Resolving deltas: 100% (109/109), done.
>> From git:/public-inbox/vger.kernel.org/linux-kernel/6
>    35280da65057..0a97acb7e709  master     -> master
>
> $ time public-inbox-index 
> real    2m1.482s
> user    0m26.084s
> sys     0m20.792s
>
> I am not on a HDD.  I will play with XAPIAN_FLUSH_THRESHOLD next time
> and see if things get better.  Initially building the Xapian index was
> extremely painful, with swapping and took over a day.
>
> Subjectively searcing all of 6.git feels faster than those 2 minutes.
> If for no other reason than I get some of the results back immediately.

XAPIAN_FLUSH_THRESHOLD seems to help.

$ git --git-dir git/6.git/ fetch
Enter passphrase for key '/home/eric/.ssh/id_rsa': 
Fetching origin
From https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/lkml/6
   0a97acb7e7..61d959b624  master     -> master
remote: Counting objects: 2562, done.
remote: Compressing objects: 100% (1384/1384), done.
remote: Total 2562 (delta 324), reused 2562 (delta 324)
Receiving objects: 100% (2562/2562), 2.45 MiB | 4.03 MiB/s, done.
Resolving deltas: 100% (324/324), done.
From git:/public-inbox/vger.kernel.org/linux-kernel/6
   0a97acb7e709..61d959b62473  master     -> master

$ (export XAPIAN_FLUSH_THRESHOLD=4000000000; time public-inbox-index )
Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$in_r> line 121.
Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$in_r> line 121.
Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$in_r> line 121.
Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$in_r> line 121.
Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$r> line 41.
Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$r> line 41.
Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$r> line 41.
Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$r> line 41.

real    0m58.239s
user    0m15.820s
sys     0m11.088s

It looks like it cut a minute off running with a slighlty larger pool of
objects.

Eric