From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.3 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 7D5681F464 for ; Wed, 25 Sep 2019 19:58:41 +0000 (UTC) Received: by mail-qt1-x831.google.com with SMTP id c21so811248qtj.12 for ; Wed, 25 Sep 2019 12:58:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Nit53Ymzpp8eBtiKJNXVIe6KdIqhbYXOK23UJt94Nk8=; b=GGiw/Kimm/4QCCgY62oNg4Aa6JyofAfeJ9wuTFj+8BJKi4lm0AsrYETMT0sD6YzwRy FaNGJMxv5/mIpbRhI5y8CCFi+3YLawC7e117tUTZ8MB/NQOSJQUTryeFGziUwz0xX+f3 lzT69MgAlEt/eb1qJ2OB9sDJGZFzfoLGeFsHc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=Nit53Ymzpp8eBtiKJNXVIe6KdIqhbYXOK23UJt94Nk8=; b=j2roV9v7EJCcGaGwYDS8WPmih3Ao8Z1Df4x1IkadtK5xiJ3rTXkB2eoTM9e3OLBNgv V5eY0OODIL67iHSuf+cCK18r1/WzJbvmjtQVHcDUWIVFXDrhSTL2DUPBCf/1XkPFOUKB ri9vM70POYmtOyTQuVMfNN7QrFg6ZhSGqUQL55JOTHDKttq9Yzg8oeZ7mCiur/hAwdXq apUxmxObiBccMoMh6Q1OyXs6EHz98QlFYOL0VhcQ7BhyYEVYkDZId9doIB4WIq1fqhFl YPTTRw8x/Kde6rNKeHPbr7G6F3J8JtaAvXZkiElPVBKRNtc2rM+BIsAhQ3+tTEFm2//P cbyg== X-Gm-Message-State: APjAAAXJ8jZpjkxsDf/o83T3snqFLFhksutXeLOwTxC7ZGRo1pFVp3ro RY4DVxK0i/bHHwigEnbG3OJZrA== X-Google-Smtp-Source: APXvYqzbM+x45Bzr0A/IQisYCi5zmwgRQnWxi7vq7g/GRi5hSzcHVYMmiAssZ58lPgUnkFN7mM5uTg== X-Received: by 2002:aed:3576:: with SMTP id b51mr115886qte.378.1569441520116; Wed, 25 Sep 2019 12:58:40 -0700 (PDT) Received: from chatter.i7.local (192-0-228-88.cpe.teksavvy.com. [192.0.228.88]) by smtp.gmail.com with ESMTPSA id l48sm426946qtb.50.2019.09.25.12.58.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2019 12:58:39 -0700 (PDT) Date: Wed, 25 Sep 2019 15:58:38 -0400 From: Konstantin Ryabitsev To: Eric Wong Cc: meta@public-inbox.org Subject: Re: Git-only operation mode Message-ID: <20190925195838.GB4628@chatter.i7.local> References: <20190925182431.GA4628@chatter.i7.local> <20190925194503.GA21501@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline In-Reply-To: <20190925194503.GA21501@dcvr> User-Agent: Mutt/1.12.1 (2019-06-15) List-Id: On Wed, Sep 25, 2019 at 07:45:03PM +0000, Eric Wong wrote: >> Is there a way to run just the archiver component of public-inbox -- >> just >> writing to git repos without any of the indexing/frontend bits? One of the >> idle conversations I had with vger.kernel.org folks was to see if we can >> shift the source of truth archive generation to happen at their end. We >> would then clone repositories from them and provide the frontend/search bits >> on lore.kernel.org. From my cursory looking, it would seem that the >> watch/delivery tools always expect to be taking care of xapian/indexing, but >> I think being able to decouple git bits from search/frontend bits would be a >> useful mode or operation. > >v1 was git-only (that led to scalability problems from big trees). >v2 needs SQLite to do dedupe with indexlevel=basic, but not Xapian, >anymore. We could get rid of dedupe for v2, but I'm not sure it's >worth it... Needing sqlite is not a big deal -- compared to the size of the repos, that's reasonably small (e.g. all of lkml git trees are 8.2GB, while msgmap.sqlite3 is 600MB). Is there an easy way to exclude xapian indexes from being generated during watch/mda runs then? A follow-up to that -- is running "public-inbox-index" on the repository after it's been updated enough to update the xapian db? It would be easy to do so as part of the grok-pull post-update hook. Best, -K