* Corrupted lkml repo?
@ 2018-10-30 3:55 Yaron Scheffer
2018-10-30 15:38 ` Konstantin Ryabitsev
0 siblings, 1 reply; 6+ messages in thread
From: Yaron Scheffer @ 2018-10-30 3:55 UTC (permalink / raw)
To: meta@public-inbox.org
Several attempts spread over several days to git clone
https://lore.kernel.org/lkml/0
The process fails with the error "server hung up
unexpectedly", at right about the 100% mark.
Is anyone else seeing this?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Corrupted lkml repo?
2018-10-30 3:55 Corrupted lkml repo? Yaron Scheffer
@ 2018-10-30 15:38 ` Konstantin Ryabitsev
2018-11-02 20:36 ` Eric Wong
0 siblings, 1 reply; 6+ messages in thread
From: Konstantin Ryabitsev @ 2018-10-30 15:38 UTC (permalink / raw)
To: Yaron Scheffer; +Cc: meta@public-inbox.org
On Tue, Oct 30, 2018 at 03:55:23AM +0000, Yaron Scheffer wrote:
>
> Several attempts spread over several days to git clone
>
> https://lore.kernel.org/lkml/0
>
> The process fails with the error "server hung up
> unexpectedly", at right about the 100% mark.
>
> Is anyone else seeing this?
You're hitting a timeout on the AWS side of things that we can't extend.
Please clone from our mirror instead:
https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/lkml/
Regards,
-K
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Corrupted lkml repo?
2018-10-30 15:38 ` Konstantin Ryabitsev
@ 2018-11-02 20:36 ` Eric Wong
2018-11-03 8:34 ` Yaron Scheffer
0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2018-11-02 20:36 UTC (permalink / raw)
To: Konstantin Ryabitsev, Yaron Scheffer; +Cc: meta
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Tue, Oct 30, 2018 at 03:55:23AM +0000, Yaron Scheffer wrote:
> >
> > Several attempts spread over several days to git clone
> >
> > https://lore.kernel.org/lkml/0
> >
> > The process fails with the error "server hung up
> > unexpectedly", at right about the 100% mark.
> >
> > Is anyone else seeing this?
>
> You're hitting a timeout on the AWS side of things that we can't extend.
> Please clone from our mirror instead:
Konstantin: Is that time-to-first-byte, overall transfer time
or something to do with POST requests being treated differently?
Yaron: does disabling smart HTTP to rely on only GET transfers
improve things? Smart HTTP is great for bandwidth reduction but
uses more CPU/memory on the server side, and perhaps
recommending clients do that for big repos is a good thing
anyways for the initial clone.
GIT_SMART_HTTP=0 git clone --mirror ...
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Corrupted lkml repo?
2018-11-02 20:36 ` Eric Wong
@ 2018-11-03 8:34 ` Yaron Scheffer
2018-11-04 3:50 ` Eric Wong
0 siblings, 1 reply; 6+ messages in thread
From: Yaron Scheffer @ 2018-11-03 8:34 UTC (permalink / raw)
To: Eric Wong; +Cc: Konstantin Ryabitsev, meta@public-inbox.org
Eric Wong wrote:
> Yaron: does disabling smart HTTP to rely on only GET transfers
> improve things? Smart HTTP is great for bandwidth reduction but
> uses more CPU/memory on the server side, and perhaps
> recommending clients do that for big repos is a good thing
> anyways for the initial clone.
>
> GIT_SMART_HTTP=0 git clone --mirror ...
Maybe, but If "git clone" stresses the server to the point of breakage
as you say, something should be done on the server side. I've been
using the kernel.org repo as Konstantin suggested.
Perhaps lower the recommended maximum shard size? I didn't hit
this issue with the most recent shard, which is about 50% smaller.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Corrupted lkml repo?
2018-11-03 8:34 ` Yaron Scheffer
@ 2018-11-04 3:50 ` Eric Wong
2018-11-05 13:20 ` Konstantin Ryabitsev
0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2018-11-04 3:50 UTC (permalink / raw)
To: Yaron Scheffer; +Cc: Konstantin Ryabitsev, meta
Yaron Scheffer <yscheffer@protonmail.com> wrote:
> Eric Wong wrote:
>
> > Yaron: does disabling smart HTTP to rely on only GET transfers
> > improve things? Smart HTTP is great for bandwidth reduction but
> > uses more CPU/memory on the server side, and perhaps
> > recommending clients do that for big repos is a good thing
> > anyways for the initial clone.
> >
> > GIT_SMART_HTTP=0 git clone --mirror ...
>
> Maybe, but If "git clone" stresses the server to the point of breakage
> as you say, something should be done on the server side. I've been
> using the kernel.org repo as Konstantin suggested.
Based on Konstantin's message; it didn't seem like the server
itself was stressed, just something on Amazon's side.
> Perhaps lower the recommended maximum shard size? I didn't hit
> this issue with the most recent shard, which is about 50% smaller.
That's another option; but migrating/converting existing repos
to it would be a flag day and break existing mirrors.
~1G seemed like a reasonable starting point
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Corrupted lkml repo?
2018-11-04 3:50 ` Eric Wong
@ 2018-11-05 13:20 ` Konstantin Ryabitsev
0 siblings, 0 replies; 6+ messages in thread
From: Konstantin Ryabitsev @ 2018-11-05 13:20 UTC (permalink / raw)
To: Eric Wong; +Cc: Yaron Scheffer, meta
On Sun, Nov 04, 2018 at 03:50:17AM +0000, Eric Wong wrote:
> > Maybe, but If "git clone" stresses the server to the point of breakage
> > as you say, something should be done on the server side. I've been
> > using the kernel.org repo as Konstantin suggested.
>
> Based on Konstantin's message; it didn't seem like the server
> itself was stressed, just something on Amazon's side.
Right, it's a bit of an unfixable stupidity on the AWS side of things
that manifests itself when cloning very large repositories (search for
"git clone aws connection draining" -- and keep in mind that we've tried
all of those workarounds).
The only fix for this is to switch from using AWS ALB (http-level proxy)
to an NLB (tcp-level proxy), but then we lose some niceties like
certificate management.
We'll probably just end up setting cloneurl to direct people to
git.kernel.org clone locations instead.
> > Perhaps lower the recommended maximum shard size? I didn't hit
> > this issue with the most recent shard, which is about 50% smaller.
>
> That's another option; but migrating/converting existing repos
> to it would be a flag day and break existing mirrors.
> ~1G seemed like a reasonable starting point
It's not really the size, but how long the client takes fetching the
repository objects, which is why the problem only manifests itself to a
subset of people.
Best,
-K
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-11-05 13:20 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-30 3:55 Corrupted lkml repo? Yaron Scheffer
2018-10-30 15:38 ` Konstantin Ryabitsev
2018-11-02 20:36 ` Eric Wong
2018-11-03 8:34 ` Yaron Scheffer
2018-11-04 3:50 ` Eric Wong
2018-11-05 13:20 ` Konstantin Ryabitsev
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).