From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.1 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 5586B1F453 for ; Mon, 5 Nov 2018 13:20:26 +0000 (UTC) Received: by mail-qk1-x72f.google.com with SMTP id o125so14534942qkf.3 for ; Mon, 05 Nov 2018 05:20:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=n6+WEKfXbFM1mC5eOvRAzxjKj9p0ML+SgiJCyfAY/zc=; b=cRw3/K/teQxrU0qEOVDTRNrUySf2bhFX+SOVHteKbuNlfaTKZ3dUl+6Ko8MXqCw0jL 7MekitEOQSZzT5hTWJIUX7Pl/iYnfRi41kHftgyiv5EjHvRTjGaHSvk6xQ2VQGvQ8Xpv EAl2wKkZXz+5fcrESSL0D20Boj7+8c7vnYSz4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=n6+WEKfXbFM1mC5eOvRAzxjKj9p0ML+SgiJCyfAY/zc=; b=YVPzunUjvSa6yAGVLu6DgfLCfHuR9IzUWgDo7P6eWWw+CQcAoLL3AXydp0yo0IX6WE i4jPYzoQ5A9Xb+x6siaPZTlcijH2veidxijTeoH495Gw0xhfmKZPz+r/EQArvwxU5tDg CoBR+A19u/zPa5aKyV11G4agazn00IHuKfXDoO1UlxZUQvhD0lCczgGFzxnI526RHsFH rcY5WPT7QlgJ9jpAYej2HmIZRXaLw6GcrzIjacCzlh3j0QFdJZgF9l9mM8yTfZMWxQxU g1lOXMPsqfS5H0MjpR75Pdc7WXWOkqv6lXUaXkio3hQxB4Mr1fTKu750uQoh2cG/M4qh oXeg== X-Gm-Message-State: AGRZ1gKU0uOZT40ZXD8QiI1b5M/pZmL4HF4zrLi/bHKaA3/HtjnqR6Jp rp1CSEoVYKp244jGu6QWWhyj5eU+pmOeoe6I X-Google-Smtp-Source: AJdET5fTjLGe8GYpZivRuiWS7PUowQIua5o5OURWbJKTH6KfI8JH85ZsZNkDVBjf/tXl6vuRfPzoUQ== X-Received: by 2002:a37:5e86:: with SMTP id s128-v6mr20798692qkb.69.1541424025140; Mon, 05 Nov 2018 05:20:25 -0800 (PST) Received: from pure.paranoia.local ([185.232.22.59]) by smtp.gmail.com with ESMTPSA id l33-v6sm21328332qte.24.2018.11.05.05.20.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 05 Nov 2018 05:20:23 -0800 (PST) Date: Mon, 5 Nov 2018 08:20:21 -0500 From: Konstantin Ryabitsev To: Eric Wong Cc: Yaron Scheffer , meta@public-inbox.org Subject: Re: Corrupted lkml repo? Message-ID: <20181105132021.GA1285@pure.paranoia.local> References: <20181030153806.GB9025@pure.paranoia.local> <20181102203658.otux6kwxsr3wh37y@dcvr> <20181104035017.rmlr3sf7wybwuvo6@untitled> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20181104035017.rmlr3sf7wybwuvo6@untitled> User-Agent: Mutt/1.10.1 (2018-07-13) List-Id: On Sun, Nov 04, 2018 at 03:50:17AM +0000, Eric Wong wrote: > > Maybe, but If "git clone" stresses the server to the point of breakage > > as you say, something should be done on the server side. I've been > > using the kernel.org repo as Konstantin suggested. > > Based on Konstantin's message; it didn't seem like the server > itself was stressed, just something on Amazon's side. Right, it's a bit of an unfixable stupidity on the AWS side of things that manifests itself when cloning very large repositories (search for "git clone aws connection draining" -- and keep in mind that we've tried all of those workarounds). The only fix for this is to switch from using AWS ALB (http-level proxy) to an NLB (tcp-level proxy), but then we lose some niceties like certificate management. We'll probably just end up setting cloneurl to direct people to git.kernel.org clone locations instead. > > Perhaps lower the recommended maximum shard size? I didn't hit > > this issue with the most recent shard, which is about 50% smaller. > > That's another option; but migrating/converting existing repos > to it would be a flag day and break existing mirrors. > ~1G seemed like a reasonable starting point It's not really the size, but how long the client takes fetching the repository objects, which is why the problem only manifests itself to a subset of people. Best, -K