From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 6F38F1F5AE; Fri, 23 Apr 2021 20:29:16 +0000 (UTC) Date: Fri, 23 Apr 2021 20:29:16 +0000 From: Eric Wong To: meta@public-inbox.org Subject: Re: Setup pointers for extindex and /all Message-ID: <20210423202916.GA23488@dcvr> References: <20210423201300.qdvaaxacp2f6rfy7@nitro.local> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20210423201300.qdvaaxacp2f6rfy7@nitro.local> List-Id: Konstantin Ryabitsev wrote: > Eric: > > I'm working on the new incarnation of lore.kernel.org (that will run on > multiple frontends as opposed to the centralized version we have now) -- I > hope to have everything ready to go by the time 1.7.x rolls out. I wonder if > you can give some pointers for extindex and /all, specifically: > > - what needs to go into the config file to enable the feature? I poked at the > source, and it appears that this will do it?: > [extindex "all"] > topdir = /path/to/where/xap15/will/live Yes, I would run `public-inbox-extindex' to create the extindex before altering the config file. It's probably safe to use --no-fsync by default and some larger value of --batch-size= since you have beefy machines. The initial index of lore took over 30 hours for me IIRC; but that's on an ancient machine with a SATA2 SSD. I'm not 100% sure how configuration should work since there's no "public-inbox-extindex-init" command (and I'm not sure if it's necessary). > - when "public-inbox-index" runs in grok-post script, will it automatically > update the "all" extindex without any additional flags, or do we need to > pass something special to it? Yes. However I've been favoring using --no-update-extindex with plain -index, and then doing "public-inbox-extindex --all /path/to/extindex" after all -index are done to reduce transactions. > - when handling message-id based lookups, will the "all" extindex be used > automatically, if found? Yes, it should be. > Thanks in advance! No problem. Please let us know if anything doesn't work as it should. The configuration could be smoother, I think... I think the on-disk format is sufficiently stable at this point, but not yet 100%...