* Argument list too long when fetching many missing objects from partial clone
@ 2021-03-01 13:18 Bagas Sanjaya
2021-03-01 19:21 ` Jeff Hostetler
2021-03-01 19:22 ` Bryan Turner
0 siblings, 2 replies; 5+ messages in thread
From: Bagas Sanjaya @ 2021-03-01 13:18 UTC (permalink / raw)
To: git
Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.
What did you do before the bug happened? (Steps to reproduce your issue)
I'm poking around with partial clone. On my local server on my computer
I have full clone of Git (git.git) repo.
I do partial clone from that server as remote (which is actually on the same
computer):
git clone https://my-local-server.git/myself/git.git --filter=blob:none
Inside the partial clone, I tried to convert it to full clone. First, I gather
list of missing objects:
git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
Then I fetched those:
git fetch origin $(cat .git/missing.list)
What did you expect to happen? (Expected behavior)
All missing objects fetched successfully
What happened instead? (Actual behavior)
Git returned:
-bash: /opt/git/bin/git: Argument list too long
What's different between what you expected and what actually happened?
(nothing)
Anything else you want to add:
I think the problem lies on how to feed list of 110K+ objects to `git fetch`.
Note: Both the local server and the computer use Git 2.30.1
Please review the rest of the bug report below.
You can delete any lines you don't wish to share.
[System Info]
git version:
git version 2.30.1
cpu: x86_64
built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
compiler info: gnuc: 9.3
libc info: glibc: 2.31
$SHELL (typically, interactive shell): /bin/bash
[Enabled Hooks]
(none)
--
An old man doll... just what I always wanted! - Clara
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Argument list too long when fetching many missing objects from partial clone
2021-03-01 13:18 Argument list too long when fetching many missing objects from partial clone Bagas Sanjaya
@ 2021-03-01 19:21 ` Jeff Hostetler
2021-03-01 19:22 ` Bryan Turner
1 sibling, 0 replies; 5+ messages in thread
From: Jeff Hostetler @ 2021-03-01 19:21 UTC (permalink / raw)
To: Bagas Sanjaya, git
On 3/1/21 8:18 AM, Bagas Sanjaya wrote:
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
>
> I'm poking around with partial clone. On my local server on my computer
> I have full clone of Git (git.git) repo.
>
> I do partial clone from that server as remote (which is actually on the
> same
> computer):
>
> git clone https://my-local-server.git/myself/git.git --filter=blob:none
>
> Inside the partial clone, I tried to convert it to full clone. First, I
> gather
> list of missing objects:
>
> git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' >
> .git/missing.list
>
> Then I fetched those:
>
> git fetch origin $(cat .git/missing.list)
>
That command will build a very very long command line and cause the
shell to complain. Instead, try passing the list of objects via stdin.
$ cat .git/missing.list | git fetch --stdin origin
> What did you expect to happen? (Expected behavior)
>
> All missing objects fetched successfully
>
> What happened instead? (Actual behavior)
>
> Git returned:
>
> -bash: /opt/git/bin/git: Argument list too long
>
> What's different between what you expected and what actually happened?
>
> (nothing)
>
> Anything else you want to add:
>
> I think the problem lies on how to feed list of 110K+ objects to `git
> fetch`.
>
> Note: Both the local server and the computer use Git 2.30.1
>
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
>
> [System Info]
> git version:
> git version 2.30.1
> cpu: x86_64
> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29
> 12:56:19 WIB 2021 x86_64
> compiler info: gnuc: 9.3
> libc info: glibc: 2.31
> $SHELL (typically, interactive shell): /bin/bash
>
> [Enabled Hooks]
> (none)
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Argument list too long when fetching many missing objects from partial clone
2021-03-01 13:18 Argument list too long when fetching many missing objects from partial clone Bagas Sanjaya
2021-03-01 19:21 ` Jeff Hostetler
@ 2021-03-01 19:22 ` Bryan Turner
2021-03-02 6:55 ` Bagas Sanjaya
1 sibling, 1 reply; 5+ messages in thread
From: Bryan Turner @ 2021-03-01 19:22 UTC (permalink / raw)
To: Bagas Sanjaya; +Cc: Git Users
On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
>
> I'm poking around with partial clone. On my local server on my computer
> I have full clone of Git (git.git) repo.
>
> I do partial clone from that server as remote (which is actually on the same
> computer):
>
> git clone https://my-local-server.git/myself/git.git --filter=blob:none
>
> Inside the partial clone, I tried to convert it to full clone. First, I gather
> list of missing objects:
>
> git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
>
> Then I fetched those:
>
> git fetch origin $(cat .git/missing.list)
Since you're on Git 2.30, instead of trying to place all the object
IDs on the `git fetch` command line, have you tried adding `--stdin`
and having it read from your `missing.list` directly? Something like
`git fetch --stdin origin <.git/missing.list` might do what you need.
>
> What did you expect to happen? (Expected behavior)
>
> All missing objects fetched successfully
>
> What happened instead? (Actual behavior)
>
> Git returned:
>
> -bash: /opt/git/bin/git: Argument list too long
>
> What's different between what you expected and what actually happened?
>
> (nothing)
>
> Anything else you want to add:
>
> I think the problem lies on how to feed list of 110K+ objects to `git fetch`.
>
> Note: Both the local server and the computer use Git 2.30.1
>
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
>
> [System Info]
> git version:
> git version 2.30.1
> cpu: x86_64
> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
> compiler info: gnuc: 9.3
> libc info: glibc: 2.31
> $SHELL (typically, interactive shell): /bin/bash
>
> [Enabled Hooks]
> (none)
>
> --
> An old man doll... just what I always wanted! - Clara
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Argument list too long when fetching many missing objects from partial clone
2021-03-01 19:22 ` Bryan Turner
@ 2021-03-02 6:55 ` Bagas Sanjaya
2021-03-03 0:42 ` brian m. carlson
0 siblings, 1 reply; 5+ messages in thread
From: Bagas Sanjaya @ 2021-03-02 6:55 UTC (permalink / raw)
To: Bryan Turner; +Cc: bagasdotme, git
After trying your suggestion, I now get:
error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
fatal: unable to write request to remote: Broken pipe
On 02/03/21 02.22, Bryan Turner wrote:
> On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>>
>> Thank you for filling out a Git bug report!
>> Please answer the following questions to help us understand your issue.
>>
>> What did you do before the bug happened? (Steps to reproduce your issue)
>>
>> I'm poking around with partial clone. On my local server on my computer
>> I have full clone of Git (git.git) repo.
>>
>> I do partial clone from that server as remote (which is actually on the same
>> computer):
>>
>> git clone https://my-local-server.git/myself/git.git --filter=blob:none
>>
>> Inside the partial clone, I tried to convert it to full clone. First, I gather
>> list of missing objects:
>>
>> git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
>>
>> Then I fetched those:
>>
>> git fetch origin $(cat .git/missing.list)
>
> Since you're on Git 2.30, instead of trying to place all the object
> IDs on the `git fetch` command line, have you tried adding `--stdin`
> and having it read from your `missing.list` directly? Something like
> `git fetch --stdin origin <.git/missing.list` might do what you need.
>
>>
>> What did you expect to happen? (Expected behavior)
>>
>> All missing objects fetched successfully
>>
>> What happened instead? (Actual behavior)
>>
>> Git returned:
>>
>> -bash: /opt/git/bin/git: Argument list too long
>>
>> What's different between what you expected and what actually happened?
>>
>> (nothing)
>>
>> Anything else you want to add:
>>
>> I think the problem lies on how to feed list of 110K+ objects to `git fetch`.
>>
>> Note: Both the local server and the computer use Git 2.30.1
>>
>> Please review the rest of the bug report below.
>> You can delete any lines you don't wish to share.
>>
>> [System Info]
>> git version:
>> git version 2.30.1
>> cpu: x86_64
>> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
>> sizeof-long: 8
>> sizeof-size_t: 8
>> shell-path: /bin/sh
>> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
>> compiler info: gnuc: 9.3
>> libc info: glibc: 2.31
>> $SHELL (typically, interactive shell): /bin/bash
>>
>> [Enabled Hooks]
>> (none)
>>
>> --
>> An old man doll... just what I always wanted! - Clara
--
An old man doll... just what I always wanted! - Clara
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Argument list too long when fetching many missing objects from partial clone
2021-03-02 6:55 ` Bagas Sanjaya
@ 2021-03-03 0:42 ` brian m. carlson
0 siblings, 0 replies; 5+ messages in thread
From: brian m. carlson @ 2021-03-03 0:42 UTC (permalink / raw)
To: Bagas Sanjaya; +Cc: Bryan Turner, git
[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]
[Note: we generally prefer inline responses.]
On 2021-03-02 at 06:55:27, Bagas Sanjaya wrote:
> After trying your suggestion, I now get:
>
> error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
> fatal: unable to write request to remote: Broken pipe
Hmmm, this means that your local Git server is unhappy with the size of
the request. That means your request is probably very large, and it's
likely that it is kindly asking you not to DoS it.
> On 02/03/21 02.22, Bryan Turner wrote:
> > On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> > > Inside the partial clone, I tried to convert it to full clone. First, I gather
> > > list of missing objects:
> > >
> > > git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
> > >
> > > Then I fetched those:
> > >
> > > git fetch origin $(cat .git/missing.list)
> >
> > Since you're on Git 2.30, instead of trying to place all the object
> > IDs on the `git fetch` command line, have you tried adding `--stdin`
> > and having it read from your `missing.list` directly? Something like
> > `git fetch --stdin origin <.git/missing.list` might do what you need.
You may want to try this:
xargs git fetch origin < .git/missing.list
That will batch the requests to fit on the command line, which will
probably also have the pleasant side effect that it makes each
individual request smaller to avoid the 413.
--
brian m. carlson (he/him or they/them)
Houston, Texas, US
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-03-03 14:21 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-01 13:18 Argument list too long when fetching many missing objects from partial clone Bagas Sanjaya
2021-03-01 19:21 ` Jeff Hostetler
2021-03-01 19:22 ` Bryan Turner
2021-03-02 6:55 ` Bagas Sanjaya
2021-03-03 0:42 ` brian m. carlson
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).