Partial fetch?

Partial fetch?

[Dāvis Mosāns]

I'm trying to fetch a remote repository over https but sadly it
timeouts too soon.

$ git fetch -v upstream
POST git-upload-pack (gzip 1148 to 641 bytes)
POST git-upload-pack (gzip 1148 to 644 bytes)
POST git-upload-pack (gzip 1948 to 1038 bytes)
POST git-upload-pack (gzip 3548 to 1845 bytes)
POST git-upload-pack (gzip 6748 to 3431 bytes)
POST git-upload-pack (gzip 13148 to 6659 bytes)
POST git-upload-pack (gzip 25948 to 13084 bytes)
POST git-upload-pack (gzip 51548 to 25997 bytes)
POST git-upload-pack (gzip 102748 to 51375 bytes)
POST git-upload-pack (gzip 205148 to 101973 bytes)
POST git-upload-pack (gzip 409948 to 203200 bytes)
POST git-upload-pack (gzip 82248 to 41185 bytes)
POST git-upload-pack (gzip 90448 to 45257 bytes)
POST git-upload-pack (gzip 99448 to 49719 bytes)
POST git-upload-pack (gzip 109348 to 54623 bytes)
POST git-upload-pack (gzip 120248 to 59937 bytes)
POST git-upload-pack (gzip 132248 to 65839 bytes)
POST git-upload-pack (gzip 145448 to 72450 bytes)
POST git-upload-pack (gzip 159948 to 79682 bytes)
POST git-upload-pack (gzip 175898 to 87436 bytes)
POST git-upload-pack (gzip 193448 to 96171 bytes)
POST git-upload-pack (gzip 212748 to 105783 bytes)
POST git-upload-pack (gzip 233998 to 116207 bytes)
POST git-upload-pack (gzip 257398 to 127793 bytes)
POST git-upload-pack (gzip 283098 to 140511 bytes)
POST git-upload-pack (gzip 311348 to 154447 bytes)
POST git-upload-pack (gzip 342448 to 169815 bytes)
POST git-upload-pack (gzip 376648 to 186850 bytes)
POST git-upload-pack (gzip 414298 to 205408 bytes)
POST git-upload-pack (gzip 455698 to 225879 bytes)
POST git-upload-pack (gzip 501248 to 248389 bytes)
POST git-upload-pack (gzip 551298 to 273131 bytes)
POST git-upload-pack (gzip 606398 to 300334 bytes)
POST git-upload-pack (gzip 666998 to 330414 bytes)
POST git-upload-pack (gzip 733698 to 363387 bytes)
POST git-upload-pack (gzip 807048 to 399647 bytes)
POST git-upload-pack (gzip 689453 to 341449 bytes)
error: RPC failed; HTTP 504 curl 22 The requested URL returned error:
504 Gateway Time-out
fatal: The remote end hung up unexpectedly

Is there some way to fetch partially by smaller chunks and then repeat
that again till everything is fetched?

Thanks!

Re: Partial fetch?

[Jeff King]

On Mon, Nov 28, 2016 at 10:34:51PM +0200, Dāvis Mosāns wrote:

> I'm trying to fetch a remote repository over https but sadly it
> timeouts too soon.
> 
> $ git fetch -v upstream
> POST git-upload-pack (gzip 1148 to 641 bytes)
> POST git-upload-pack (gzip 1148 to 644 bytes)
> [...]
> Is there some way to fetch partially by smaller chunks and then repeat
> that again till everything is fetched?

Not an easy one. The series of POSTs is an indication that the fetch
negotiation is going on for a long time, which probably means you have a
lot of commits in your local repository that aren't in the remote, or
vice versa.

Here are the things I might try:

  - git v2.10.2 has commit 06b3d386e (fetch-pack: do not reset in_vain
    on non-novel acks, 2016-09-23), which may help with this.

  - HTTP, because the server is stateless, performs less well than other
    protocols. If you can fetch over ssh or git://, it will probably
    just work.

  - If this is a one-time thing to fetch unrelated history from another
    repository, you can "clone --mirror" instead of fetching,
    then fetch from the mirror locally. Subsequent fetches should be
    fast.

If you do try v2.10.2 and it improves things, I'd be interested to hear
about it as a data point.

-Peff

Re: Partial fetch?

[Dāvis Mosāns]

2016-11-29 23:55 GMT+02:00 Jeff King <peff@peff.net>:
> On Mon, Nov 28, 2016 at 10:34:51PM +0200, Dāvis Mosāns wrote:
>
>> I'm trying to fetch a remote repository over https but sadly it
>> timeouts too soon.
>>
>> $ git fetch -v upstream
>> POST git-upload-pack (gzip 1148 to 641 bytes)
>> POST git-upload-pack (gzip 1148 to 644 bytes)
>> [...]
>> Is there some way to fetch partially by smaller chunks and then repeat
>> that again till everything is fetched?
>
> Not an easy one. The series of POSTs is an indication that the fetch
> negotiation is going on for a long time, which probably means you have a
> lot of commits in your local repository that aren't in the remote, or
> vice versa.
>
> Here are the things I might try:
>
>   - git v2.10.2 has commit 06b3d386e (fetch-pack: do not reset in_vain
>     on non-novel acks, 2016-09-23), which may help with this.
>

That output and this is already with git v2.10.2

>   - HTTP, because the server is stateless, performs less well than other
>     protocols. If you can fetch over ssh or git://, it will probably
>     just work.
>

It's only available under https://git.replicant.us/replicant/frameworks_base.git

>   - If this is a one-time thing to fetch unrelated history from another
>     repository, you can "clone --mirror" instead of fetching,
>     then fetch from the mirror locally. Subsequent fetches should be
>     fast.
>

Looks like something is wrong with their server/setup since currently
even clone doesn't work for me.

 Cloning into bare repository 'frameworks_base.git'...
remote: Counting objects: 739930, done.
remote: Compressing objects: 100% (196567/196567), done.
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

But someone have made a mirror for it and there clone works only still
same issue with fetch.
Once I will have this cloned I will try local fetch.

Thanks!