user/dev discussion of public-inbox itself
 help / Atom feed
* Corrupted lkml repo?
@ 2018-10-30  3:55 Yaron Scheffer
  2018-10-30 15:38 ` Konstantin Ryabitsev
  0 siblings, 1 reply; 6+ messages in thread
From: Yaron Scheffer @ 2018-10-30  3:55 UTC (permalink / raw)
  To: meta


Several attempts spread over several days to git clone

https://lore.kernel.org/lkml/0

The process fails with the error "server hung up
unexpectedly", at right about the 100% mark.

Is anyone else seeing this?

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Corrupted lkml repo?
  2018-10-30  3:55 Corrupted lkml repo? Yaron Scheffer
@ 2018-10-30 15:38 ` Konstantin Ryabitsev
  2018-11-02 20:36   ` Eric Wong
  0 siblings, 1 reply; 6+ messages in thread
From: Konstantin Ryabitsev @ 2018-10-30 15:38 UTC (permalink / raw)
  To: Yaron Scheffer; +Cc: meta

On Tue, Oct 30, 2018 at 03:55:23AM +0000, Yaron Scheffer wrote:
> 
> Several attempts spread over several days to git clone
> 
> https://lore.kernel.org/lkml/0
> 
> The process fails with the error "server hung up
> unexpectedly", at right about the 100% mark.
> 
> Is anyone else seeing this?

You're hitting a timeout on the AWS side of things that we can't extend.
Please clone from our mirror instead:

https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/lkml/

Regards,
-K

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Corrupted lkml repo?
  2018-10-30 15:38 ` Konstantin Ryabitsev
@ 2018-11-02 20:36   ` Eric Wong
  2018-11-03  8:34     ` Yaron Scheffer
  0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2018-11-02 20:36 UTC (permalink / raw)
  To: Konstantin Ryabitsev, Yaron Scheffer; +Cc: meta

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Tue, Oct 30, 2018 at 03:55:23AM +0000, Yaron Scheffer wrote:
> > 
> > Several attempts spread over several days to git clone
> > 
> > https://lore.kernel.org/lkml/0
> > 
> > The process fails with the error "server hung up
> > unexpectedly", at right about the 100% mark.
> > 
> > Is anyone else seeing this?
> 
> You're hitting a timeout on the AWS side of things that we can't extend.
> Please clone from our mirror instead:

Konstantin: Is that time-to-first-byte, overall transfer time
or something to do with POST requests being treated differently?

Yaron: does disabling smart HTTP to rely on only GET transfers
improve things?  Smart HTTP is great for bandwidth reduction but
uses more CPU/memory on the server side, and perhaps
recommending clients do that for big repos is a good thing
anyways for the initial clone.

GIT_SMART_HTTP=0 git clone --mirror ...

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Corrupted lkml repo?
  2018-11-02 20:36   ` Eric Wong
@ 2018-11-03  8:34     ` Yaron Scheffer
  2018-11-04  3:50       ` Eric Wong
  0 siblings, 1 reply; 6+ messages in thread
From: Yaron Scheffer @ 2018-11-03  8:34 UTC (permalink / raw)
  To: Eric Wong; +Cc: Konstantin Ryabitsev, meta

Eric Wong wrote:

> Yaron: does disabling smart HTTP to rely on only GET transfers
> improve things? Smart HTTP is great for bandwidth reduction but
> uses more CPU/memory on the server side, and perhaps
> recommending clients do that for big repos is a good thing
> anyways for the initial clone.
>
> GIT_SMART_HTTP=0 git clone --mirror ...

Maybe, but If "git clone" stresses the server to the point of breakage
as you say, something should be done on the server side. I've been
using the kernel.org repo as Konstantin suggested.

Perhaps lower the recommended maximum shard size? I didn't hit
this issue with the most recent shard, which is about 50% smaller.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Corrupted lkml repo?
  2018-11-03  8:34     ` Yaron Scheffer
@ 2018-11-04  3:50       ` Eric Wong
  2018-11-05 13:20         ` Konstantin Ryabitsev
  0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2018-11-04  3:50 UTC (permalink / raw)
  To: Yaron Scheffer; +Cc: Konstantin Ryabitsev, meta

Yaron Scheffer <yscheffer@protonmail.com> wrote:
> Eric Wong wrote:
> 
> > Yaron: does disabling smart HTTP to rely on only GET transfers
> > improve things? Smart HTTP is great for bandwidth reduction but
> > uses more CPU/memory on the server side, and perhaps
> > recommending clients do that for big repos is a good thing
> > anyways for the initial clone.
> >
> > GIT_SMART_HTTP=0 git clone --mirror ...
> 
> Maybe, but If "git clone" stresses the server to the point of breakage
> as you say, something should be done on the server side. I've been
> using the kernel.org repo as Konstantin suggested.

Based on Konstantin's message; it didn't seem like the server
itself was stressed, just something on Amazon's side.

> Perhaps lower the recommended maximum shard size? I didn't hit
> this issue with the most recent shard, which is about 50% smaller.

That's another option; but migrating/converting existing repos
to it would be a flag day and break existing mirrors.
~1G seemed like a reasonable starting point

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Corrupted lkml repo?
  2018-11-04  3:50       ` Eric Wong
@ 2018-11-05 13:20         ` Konstantin Ryabitsev
  0 siblings, 0 replies; 6+ messages in thread
From: Konstantin Ryabitsev @ 2018-11-05 13:20 UTC (permalink / raw)
  To: Eric Wong; +Cc: Yaron Scheffer, meta

On Sun, Nov 04, 2018 at 03:50:17AM +0000, Eric Wong wrote:
> > Maybe, but If "git clone" stresses the server to the point of breakage
> > as you say, something should be done on the server side. I've been
> > using the kernel.org repo as Konstantin suggested.
> 
> Based on Konstantin's message; it didn't seem like the server
> itself was stressed, just something on Amazon's side.

Right, it's a bit of an unfixable stupidity on the AWS side of things
that manifests itself when cloning very large repositories (search for
"git clone aws connection draining" -- and keep in mind that we've tried
all of those workarounds). 

The only fix for this is to switch from using AWS ALB (http-level proxy)
to an NLB (tcp-level proxy), but then we lose some niceties like
certificate management.

We'll probably just end up setting cloneurl to direct people to
git.kernel.org clone locations instead.

> > Perhaps lower the recommended maximum shard size? I didn't hit
> > this issue with the most recent shard, which is about 50% smaller.
> 
> That's another option; but migrating/converting existing repos
> to it would be a flag day and break existing mirrors.
> ~1G seemed like a reasonable starting point

It's not really the size, but how long the client takes fetching the
repository objects, which is why the problem only manifests itself to a
subset of people.

Best,
-K

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-30  3:55 Corrupted lkml repo? Yaron Scheffer
2018-10-30 15:38 ` Konstantin Ryabitsev
2018-11-02 20:36   ` Eric Wong
2018-11-03  8:34     ` Yaron Scheffer
2018-11-04  3:50       ` Eric Wong
2018-11-05 13:20         ` Konstantin Ryabitsev

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox