git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
* Argument list too long when fetching many missing objects from partial clone
@ 2021-03-01 13:18 Bagas Sanjaya
  2021-03-01 19:21 ` Jeff Hostetler
  2021-03-01 19:22 ` Bryan Turner
  0 siblings, 2 replies; 5+ messages in thread
From: Bagas Sanjaya @ 2021-03-01 13:18 UTC (permalink / raw)
  To: git

Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

I'm poking around with partial clone. On my local server on my computer
I have full clone of Git (git.git) repo.

I do partial clone from that server as remote (which is actually on the same
computer):

    git clone https://my-local-server.git/myself/git.git --filter=blob:none

Inside the partial clone, I tried to convert it to full clone. First, I gather
list of missing objects:

    git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list

Then I fetched those:

    git fetch origin $(cat .git/missing.list)

What did you expect to happen? (Expected behavior)

All missing objects fetched successfully

What happened instead? (Actual behavior)

Git returned:

     -bash: /opt/git/bin/git: Argument list too long

What's different between what you expected and what actually happened?

(nothing)

Anything else you want to add:

I think the problem lies on how to feed list of 110K+ objects to `git fetch`.

Note: Both the local server and the computer use Git 2.30.1

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.

[System Info]
git version:
git version 2.30.1
cpu: x86_64
built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
compiler info: gnuc: 9.3
libc info: glibc: 2.31
$SHELL (typically, interactive shell): /bin/bash

[Enabled Hooks]
(none)

-- 
An old man doll... just what I always wanted! - Clara

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Argument list too long when fetching many missing objects from partial clone
  2021-03-01 13:18 Argument list too long when fetching many missing objects from partial clone Bagas Sanjaya
@ 2021-03-01 19:21 ` Jeff Hostetler
  2021-03-01 19:22 ` Bryan Turner
  1 sibling, 0 replies; 5+ messages in thread
From: Jeff Hostetler @ 2021-03-01 19:21 UTC (permalink / raw)
  To: Bagas Sanjaya, git



On 3/1/21 8:18 AM, Bagas Sanjaya wrote:
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
> 
> What did you do before the bug happened? (Steps to reproduce your issue)
> 
> I'm poking around with partial clone. On my local server on my computer
> I have full clone of Git (git.git) repo.
> 
> I do partial clone from that server as remote (which is actually on the 
> same
> computer):
> 
>     git clone https://my-local-server.git/myself/git.git --filter=blob:none
> 
> Inside the partial clone, I tried to convert it to full clone. First, I 
> gather
> list of missing objects:
> 
>     git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > 
> .git/missing.list
> 
> Then I fetched those:
> 
>     git fetch origin $(cat .git/missing.list)
> 

That command will build a very very long command line and cause the
shell to complain.  Instead, try passing the list of objects via stdin.

     $ cat .git/missing.list | git fetch --stdin origin



> What did you expect to happen? (Expected behavior)
> 
> All missing objects fetched successfully
> 
> What happened instead? (Actual behavior)
> 
> Git returned:
> 
>      -bash: /opt/git/bin/git: Argument list too long
> 
> What's different between what you expected and what actually happened?
> 
> (nothing)
> 
> Anything else you want to add:
> 
> I think the problem lies on how to feed list of 110K+ objects to `git 
> fetch`.
> 
> Note: Both the local server and the computer use Git 2.30.1
> 
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
> 
> [System Info]
> git version:
> git version 2.30.1
> cpu: x86_64
> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 
> 12:56:19 WIB 2021 x86_64
> compiler info: gnuc: 9.3
> libc info: glibc: 2.31
> $SHELL (typically, interactive shell): /bin/bash
> 
> [Enabled Hooks]
> (none)
> 

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Argument list too long when fetching many missing objects from partial clone
  2021-03-01 13:18 Argument list too long when fetching many missing objects from partial clone Bagas Sanjaya
  2021-03-01 19:21 ` Jeff Hostetler
@ 2021-03-01 19:22 ` Bryan Turner
  2021-03-02  6:55   ` Bagas Sanjaya
  1 sibling, 1 reply; 5+ messages in thread
From: Bryan Turner @ 2021-03-01 19:22 UTC (permalink / raw)
  To: Bagas Sanjaya; +Cc: Git Users

On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
>
> I'm poking around with partial clone. On my local server on my computer
> I have full clone of Git (git.git) repo.
>
> I do partial clone from that server as remote (which is actually on the same
> computer):
>
>     git clone https://my-local-server.git/myself/git.git --filter=blob:none
>
> Inside the partial clone, I tried to convert it to full clone. First, I gather
> list of missing objects:
>
>     git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
>
> Then I fetched those:
>
>     git fetch origin $(cat .git/missing.list)

Since you're on Git 2.30, instead of trying to place all the object
IDs on the `git fetch` command line, have you tried adding `--stdin`
and having it read from your `missing.list` directly? Something like
`git fetch --stdin origin <.git/missing.list` might do what you need.

>
> What did you expect to happen? (Expected behavior)
>
> All missing objects fetched successfully
>
> What happened instead? (Actual behavior)
>
> Git returned:
>
>      -bash: /opt/git/bin/git: Argument list too long
>
> What's different between what you expected and what actually happened?
>
> (nothing)
>
> Anything else you want to add:
>
> I think the problem lies on how to feed list of 110K+ objects to `git fetch`.
>
> Note: Both the local server and the computer use Git 2.30.1
>
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
>
> [System Info]
> git version:
> git version 2.30.1
> cpu: x86_64
> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
> compiler info: gnuc: 9.3
> libc info: glibc: 2.31
> $SHELL (typically, interactive shell): /bin/bash
>
> [Enabled Hooks]
> (none)
>
> --
> An old man doll... just what I always wanted! - Clara

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Argument list too long when fetching many missing objects from partial clone
  2021-03-01 19:22 ` Bryan Turner
@ 2021-03-02  6:55   ` Bagas Sanjaya
  2021-03-03  0:42     ` brian m. carlson
  0 siblings, 1 reply; 5+ messages in thread
From: Bagas Sanjaya @ 2021-03-02  6:55 UTC (permalink / raw)
  To: Bryan Turner; +Cc: bagasdotme, git

After trying your suggestion, I now get:

error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
fatal: unable to write request to remote: Broken pipe

On 02/03/21 02.22, Bryan Turner wrote:
> On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>>
>> Thank you for filling out a Git bug report!
>> Please answer the following questions to help us understand your issue.
>>
>> What did you do before the bug happened? (Steps to reproduce your issue)
>>
>> I'm poking around with partial clone. On my local server on my computer
>> I have full clone of Git (git.git) repo.
>>
>> I do partial clone from that server as remote (which is actually on the same
>> computer):
>>
>>      git clone https://my-local-server.git/myself/git.git --filter=blob:none
>>
>> Inside the partial clone, I tried to convert it to full clone. First, I gather
>> list of missing objects:
>>
>>      git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
>>
>> Then I fetched those:
>>
>>      git fetch origin $(cat .git/missing.list)
> 
> Since you're on Git 2.30, instead of trying to place all the object
> IDs on the `git fetch` command line, have you tried adding `--stdin`
> and having it read from your `missing.list` directly? Something like
> `git fetch --stdin origin <.git/missing.list` might do what you need.
> 
>>
>> What did you expect to happen? (Expected behavior)
>>
>> All missing objects fetched successfully
>>
>> What happened instead? (Actual behavior)
>>
>> Git returned:
>>
>>       -bash: /opt/git/bin/git: Argument list too long
>>
>> What's different between what you expected and what actually happened?
>>
>> (nothing)
>>
>> Anything else you want to add:
>>
>> I think the problem lies on how to feed list of 110K+ objects to `git fetch`.
>>
>> Note: Both the local server and the computer use Git 2.30.1
>>
>> Please review the rest of the bug report below.
>> You can delete any lines you don't wish to share.
>>
>> [System Info]
>> git version:
>> git version 2.30.1
>> cpu: x86_64
>> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
>> sizeof-long: 8
>> sizeof-size_t: 8
>> shell-path: /bin/sh
>> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
>> compiler info: gnuc: 9.3
>> libc info: glibc: 2.31
>> $SHELL (typically, interactive shell): /bin/bash
>>
>> [Enabled Hooks]
>> (none)
>>
>> --
>> An old man doll... just what I always wanted! - Clara

-- 
An old man doll... just what I always wanted! - Clara

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Argument list too long when fetching many missing objects from partial clone
  2021-03-02  6:55   ` Bagas Sanjaya
@ 2021-03-03  0:42     ` brian m. carlson
  0 siblings, 0 replies; 5+ messages in thread
From: brian m. carlson @ 2021-03-03  0:42 UTC (permalink / raw)
  To: Bagas Sanjaya; +Cc: Bryan Turner, git

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

[Note: we generally prefer inline responses.]

On 2021-03-02 at 06:55:27, Bagas Sanjaya wrote:
> After trying your suggestion, I now get:
> 
> error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
> fatal: unable to write request to remote: Broken pipe

Hmmm, this means that your local Git server is unhappy with the size of
the request.  That means your request is probably very large, and it's
likely that it is kindly asking you not to DoS it.

> On 02/03/21 02.22, Bryan Turner wrote:
> > On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> > > Inside the partial clone, I tried to convert it to full clone. First, I gather
> > > list of missing objects:
> > > 
> > >      git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
> > > 
> > > Then I fetched those:
> > > 
> > >      git fetch origin $(cat .git/missing.list)
> > 
> > Since you're on Git 2.30, instead of trying to place all the object
> > IDs on the `git fetch` command line, have you tried adding `--stdin`
> > and having it read from your `missing.list` directly? Something like
> > `git fetch --stdin origin <.git/missing.list` might do what you need.

You may want to try this:

  xargs git fetch origin < .git/missing.list

That will batch the requests to fit on the command line, which will
probably also have the pleasant side effect that it makes each
individual request smaller to avoid the 413.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-03-03 14:21 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-01 13:18 Argument list too long when fetching many missing objects from partial clone Bagas Sanjaya
2021-03-01 19:21 ` Jeff Hostetler
2021-03-01 19:22 ` Bryan Turner
2021-03-02  6:55   ` Bagas Sanjaya
2021-03-03  0:42     ` brian m. carlson

Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).