* Searching via git grep? @ 2018-07-19 20:47 Eric W. Biederman 2018-07-19 21:12 ` Eric Wong 0 siblings, 1 reply; 6+ messages in thread From: Eric W. Biederman @ 2018-07-19 20:47 UTC (permalink / raw) To: Eric Wong; +Cc: meta Have you considered searching public inboxes via git grep? For a big server lore.kernel.org with a lot of searches and a lot of clients it might not make sense. But for home use where searches are rare and the indexes can not be kept in ram, but the mailbox might fit git grep sounds attractive? I performed a preliminary test and just running git grep manually and I was search all of the git mailling list archive pretty much immediately. For v1 it is just 'git grep <regexp> HEAD' For v2 it is 'git --rev-list --all | xargs git grep <regexp>' If this sounds reasonable to you I will take a look at what it takes to wire that up over the next while. Eric ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Searching via git grep? 2018-07-19 20:47 Searching via git grep? Eric W. Biederman @ 2018-07-19 21:12 ` Eric Wong 2018-07-19 22:27 ` Eric W. Biederman 0 siblings, 1 reply; 6+ messages in thread From: Eric Wong @ 2018-07-19 21:12 UTC (permalink / raw) To: Eric W. Biederman; +Cc: meta "Eric W. Biederman" <ebiederm@xmission.com> wrote: > Have you considered searching public inboxes via git grep? Not yet... > For a big server lore.kernel.org with a lot of searches and a lot of > clients it might not make sense. But for home use where searches are > rare and the indexes can not be kept in ram, but the mailbox might fit > git grep sounds attractive? > > I performed a preliminary test and just running git grep manually and > I was search all of the git mailling list archive pretty much > immediately. > > For v1 it is just 'git grep <regexp> HEAD' > For v2 it is 'git --rev-list --all | xargs git grep <regexp>' > > If this sounds reasonable to you I will take a look at what it takes to > wire that up over the next while. Having something like this on a potentially public-facing web UI seems like a liability support-wise(*). However, I'd be open to having this as a command-line tool. Maybe in the scripts/ directory for one-off scripts... If I were building a personal mail tool, I could use scripts/dupe-finder as a starting point. (*) I would also caution against having personal mail accessible over http://localhost/ on any port without a password; as there's attacks on browsers which could hit them. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Searching via git grep? 2018-07-19 21:12 ` Eric Wong @ 2018-07-19 22:27 ` Eric W. Biederman 2018-07-20 6:11 ` Eric Wong 0 siblings, 1 reply; 6+ messages in thread From: Eric W. Biederman @ 2018-07-19 22:27 UTC (permalink / raw) To: Eric Wong; +Cc: meta Eric Wong <e@80x24.org> writes: > "Eric W. Biederman" <ebiederm@xmission.com> wrote: >> Have you considered searching public inboxes via git grep? > > Not yet... > >> For a big server lore.kernel.org with a lot of searches and a lot of >> clients it might not make sense. But for home use where searches are >> rare and the indexes can not be kept in ram, but the mailbox might fit >> git grep sounds attractive? >> >> I performed a preliminary test and just running git grep manually and >> I was search all of the git mailling list archive pretty much >> immediately. >> >> For v1 it is just 'git grep <regexp> HEAD' >> For v2 it is 'git --rev-list --all | xargs git grep <regexp>' >> >> If this sounds reasonable to you I will take a look at what it takes to >> wire that up over the next while. > > Having something like this on a potentially public-facing web UI > seems like a liability support-wise(*). > > However, I'd be open to having this as a command-line tool. > Maybe in the scripts/ directory for one-off scripts... > If I were building a personal mail tool, I could use > scripts/dupe-finder as a starting point. My current goal is to make it pleasant to read linux-kernel and possibly other large archives on my personal machine. Right now the git trees for linux-kernel are aboug 6.8G. Small enough to fit in RAM. The Xapian indexes are about 63G. Not small enough to fit in ram. They are also not fast to update when I pull in a new batch of messages from linux-kernel. So I am looking at using git grep as a stand-in for the Xapian indexes when indexlevel eq 'basic'. Given my personal ratio of searches to indexing I think I will save time in doing that. I don't have it all wired up yet to know if it will work well, but I suspect it will. Is it only the web interface where the advanced search functionality is available? > (*) I would also caution against having personal mail accessible > over http://localhost/ on any port without a password; as > there's attacks on browsers which could hit them. Good point. While I may get there that is not my primary focus. I have for example emails archived from public mailling lists but because I was the author the mail machine stripped the domain from from. This was all about 20 years ago. Eric ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Searching via git grep? 2018-07-19 22:27 ` Eric W. Biederman @ 2018-07-20 6:11 ` Eric Wong 2018-07-20 12:37 ` Eric W. Biederman 0 siblings, 1 reply; 6+ messages in thread From: Eric Wong @ 2018-07-20 6:11 UTC (permalink / raw) To: Eric W. Biederman; +Cc: meta "Eric W. Biederman" <ebiederm@xmission.com> wrote: > My current goal is to make it pleasant to read linux-kernel and possibly > other large archives on my personal machine. Right now the git > trees for linux-kernel are aboug 6.8G. Small enough to fit in RAM. > > The Xapian indexes are about 63G. Not small enough to fit in ram. > They are also not fast to update when I pull in a new batch of messages > from linux-kernel. Interesting, how long does it take to do an incremental index medium/full for you? Setting XAPIAN_FLUSH_THRESHOLD after my patch yesterday should help noticeably, especially if you're on HDD. > So I am looking at using git grep as a stand-in for the Xapian indexes > when indexlevel eq 'basic'. > > Given my personal ratio of searches to indexing I think I will save > time in doing that. I don't have it all wired up yet to know if it will > work well, but I suspect it will. Totally understandable, and yes, if you can fit the LKML repos into RAM it should be usable enough for a single user. "git grep" also has the advantage of being able to use regexps, which isn't possible with Xapian at the moment. > Is it only the web interface where the advanced search functionality is > available? Yes. I don't think there's a good way to implement search for NNTP on the server side... IMAP has specs for implementing search; but I don't know how much overlap there is with what our web UI currently offers. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Searching via git grep? 2018-07-20 6:11 ` Eric Wong @ 2018-07-20 12:37 ` Eric W. Biederman 2018-07-20 23:56 ` Eric W. Biederman 0 siblings, 1 reply; 6+ messages in thread From: Eric W. Biederman @ 2018-07-20 12:37 UTC (permalink / raw) To: Eric Wong; +Cc: meta Eric Wong <e@80x24.org> writes: > "Eric W. Biederman" <ebiederm@xmission.com> wrote: >> My current goal is to make it pleasant to read linux-kernel and possibly >> other large archives on my personal machine. Right now the git >> trees for linux-kernel are aboug 6.8G. Small enough to fit in RAM. >> >> The Xapian indexes are about 63G. Not small enough to fit in ram. >> They are also not fast to update when I pull in a new batch of messages >> from linux-kernel. > > Interesting, how long does it take to do an incremental index > medium/full for you? Setting XAPIAN_FLUSH_THRESHOLD after my > patch yesterday should help noticeably, especially if you're on > HDD. For a small sample less than a days worth of lkml messages I get: $ git --git-dir git/6.git/ fetch Enter passphrase for key '/home/eric/.ssh/id_rsa': Fetching origin From https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/lkml/6 35280da650..0a97acb7e7 master -> master remote: Counting objects: 1791, done. remote: Compressing objects: 100% (1085/1085), done. remote: Total 1791 (delta 109), reused 1791 (delta 109) Receiving objects: 100% (1791/1791), 1.94 MiB | 1.98 MiB/s, done. Resolving deltas: 100% (109/109), done. From git:/public-inbox/vger.kernel.org/linux-kernel/6 35280da65057..0a97acb7e709 master -> master $ time public-inbox-index real 2m1.482s user 0m26.084s sys 0m20.792s I am not on a HDD. I will play with XAPIAN_FLUSH_THRESHOLD next time and see if things get better. Initially building the Xapian index was extremely painful, with swapping and took over a day. Subjectively searcing all of 6.git feels faster than those 2 minutes. If for no other reason than I get some of the results back immediately. >> So I am looking at using git grep as a stand-in for the Xapian indexes >> when indexlevel eq 'basic'. >> >> Given my personal ratio of searches to indexing I think I will save >> time in doing that. I don't have it all wired up yet to know if it will >> work well, but I suspect it will. > > Totally understandable, and yes, if you can fit the LKML repos > into RAM it should be usable enough for a single user. > > "git grep" also has the advantage of being able to use regexps, > which isn't possible with Xapian at the moment. My only concern with "git grep" for v2 is how do I get it to exclude messages that have been deleted. >> Is it only the web interface where the advanced search functionality is >> available? > > Yes. I don't think there's a good way to implement search for > NNTP on the server side... IMAP has specs for implementing > search; but I don't know how much overlap there is with what > our web UI currently offers. I skimmed the IMAP rfcs earlier and the search sounds very close to what Xapian makes available. Roughly terms and quoted terms (aka terms with positions). If the IMAP interface is sensible it might be worth doing the work to extend NNTP to provide a search interface modeled on it. Eric ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Searching via git grep? 2018-07-20 12:37 ` Eric W. Biederman @ 2018-07-20 23:56 ` Eric W. Biederman 0 siblings, 0 replies; 6+ messages in thread From: Eric W. Biederman @ 2018-07-20 23:56 UTC (permalink / raw) To: Eric Wong; +Cc: meta ebiederm@xmission.com (Eric W. Biederman) writes: > Eric Wong <e@80x24.org> writes: > >> "Eric W. Biederman" <ebiederm@xmission.com> wrote: >>> My current goal is to make it pleasant to read linux-kernel and possibly >>> other large archives on my personal machine. Right now the git >>> trees for linux-kernel are aboug 6.8G. Small enough to fit in RAM. >>> >>> The Xapian indexes are about 63G. Not small enough to fit in ram. >>> They are also not fast to update when I pull in a new batch of messages >>> from linux-kernel. >> >> Interesting, how long does it take to do an incremental index >> medium/full for you? Setting XAPIAN_FLUSH_THRESHOLD after my >> patch yesterday should help noticeably, especially if you're on >> HDD. > > For a small sample less than a days worth of lkml messages > I get: > > $ git --git-dir git/6.git/ fetch > Enter passphrase for key '/home/eric/.ssh/id_rsa': > Fetching origin >> From https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/lkml/6 > 35280da650..0a97acb7e7 master -> master > remote: Counting objects: 1791, done. > remote: Compressing objects: 100% (1085/1085), done. > remote: Total 1791 (delta 109), reused 1791 (delta 109) > Receiving objects: 100% (1791/1791), 1.94 MiB | 1.98 MiB/s, done. > Resolving deltas: 100% (109/109), done. >> From git:/public-inbox/vger.kernel.org/linux-kernel/6 > 35280da65057..0a97acb7e709 master -> master > > $ time public-inbox-index > real 2m1.482s > user 0m26.084s > sys 0m20.792s > > I am not on a HDD. I will play with XAPIAN_FLUSH_THRESHOLD next time > and see if things get better. Initially building the Xapian index was > extremely painful, with swapping and took over a day. > > Subjectively searcing all of 6.git feels faster than those 2 minutes. > If for no other reason than I get some of the results back immediately. XAPIAN_FLUSH_THRESHOLD seems to help. $ git --git-dir git/6.git/ fetch Enter passphrase for key '/home/eric/.ssh/id_rsa': Fetching origin From https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/lkml/6 0a97acb7e7..61d959b624 master -> master remote: Counting objects: 2562, done. remote: Compressing objects: 100% (1384/1384), done. remote: Total 2562 (delta 324), reused 2562 (delta 324) Receiving objects: 100% (2562/2562), 2.45 MiB | 4.03 MiB/s, done. Resolving deltas: 100% (324/324), done. From git:/public-inbox/vger.kernel.org/linux-kernel/6 0a97acb7e709..61d959b62473 master -> master $ (export XAPIAN_FLUSH_THRESHOLD=4000000000; time public-inbox-index ) Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$in_r> line 121. Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$in_r> line 121. Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$in_r> line 121. Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$in_r> line 121. Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$r> line 41. Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$r> line 41. Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$r> line 41. Use of uninitialized value in lc at /usr/share/perl5/Email/Simple/Header.pm line 181, <$r> line 41. real 0m58.239s user 0m15.820s sys 0m11.088s It looks like it cut a minute off running with a slighlty larger pool of objects. Eric ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-07-20 23:56 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-07-19 20:47 Searching via git grep? Eric W. Biederman 2018-07-19 21:12 ` Eric Wong 2018-07-19 22:27 ` Eric W. Biederman 2018-07-20 6:11 ` Eric Wong 2018-07-20 12:37 ` Eric W. Biederman 2018-07-20 23:56 ` Eric W. Biederman
Code repositories for project(s) associated with this public inbox https://80x24.org/public-inbox.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).