From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: Re: Setup pointers for extindex and /all
Date: Fri, 23 Apr 2021 20:29:16 +0000 [thread overview]
Message-ID: <20210423202916.GA23488@dcvr> (raw)
In-Reply-To: <20210423201300.qdvaaxacp2f6rfy7@nitro.local>
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Eric:
>
> I'm working on the new incarnation of lore.kernel.org (that will run on
> multiple frontends as opposed to the centralized version we have now) -- I
> hope to have everything ready to go by the time 1.7.x rolls out. I wonder if
> you can give some pointers for extindex and /all, specifically:
>
> - what needs to go into the config file to enable the feature? I poked at the
> source, and it appears that this will do it?:
> [extindex "all"]
> topdir = /path/to/where/xap15/will/live
Yes, I would run `public-inbox-extindex' to create the extindex
before altering the config file.
It's probably safe to use --no-fsync by default and some larger
value of --batch-size= since you have beefy machines. The
initial index of lore took over 30 hours for me IIRC; but
that's on an ancient machine with a SATA2 SSD.
I'm not 100% sure how configuration should work since there's no
"public-inbox-extindex-init" command (and I'm not sure if it's
necessary).
> - when "public-inbox-index" runs in grok-post script, will it automatically
> update the "all" extindex without any additional flags, or do we need to
> pass something special to it?
Yes. However I've been favoring using --no-update-extindex with
plain -index, and then doing "public-inbox-extindex --all /path/to/extindex"
after all -index are done to reduce transactions.
> - when handling message-id based lookups, will the "all" extindex be used
> automatically, if found?
Yes, it should be.
> Thanks in advance!
No problem. Please let us know if anything doesn't work as it
should. The configuration could be smoother, I think...
I think the on-disk format is sufficiently stable at this point,
but not yet 100%...
prev parent reply other threads:[~2021-04-23 20:29 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-23 20:13 Setup pointers for extindex and /all Konstantin Ryabitsev
2021-04-23 20:29 ` Eric Wong [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210423202916.GA23488@dcvr \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).