git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / Atom feed
* Fetching 24 Linux commits = 1.2 GiB
@ 2020-04-15  8:01 Jiri Slaby
  2020-04-15  8:11 ` Andreas Schwab
                   ` (2 more replies)
  0 siblings, 3 replies; 20+ messages in thread
From: Jiri Slaby @ 2020-04-15  8:01 UTC (permalink / raw)
  To: git

Hi,

I was at 8f3d9f354286 of:
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

I did git remote update today and it fetched:
Receiving objects: 100% (7330823/7330823), 1.20 GiB
It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.

One colleague of mine fetched 1324 commits:
Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
   7e63420847ae..8632e9b5645b  master     -> origin/master

Another colleague fetched the same what I and:
  Receiving objects: 100% (7330823/7330823), 1.20 GiB
too.

I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.

Is that a bug? What info should I provide?

thanks,
-- 
js
suse labs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15  8:01 Fetching 24 Linux commits = 1.2 GiB Jiri Slaby
@ 2020-04-15  8:11 ` Andreas Schwab
  2020-04-15  8:16   ` Jiri Slaby
  2020-04-15 10:52 ` Kevin Daudt
  2020-04-15 13:56 ` Konstantin Ryabitsev
  2 siblings, 1 reply; 20+ messages in thread
From: Andreas Schwab @ 2020-04-15  8:11 UTC (permalink / raw)
  To: Jiri Slaby; +Cc: git

On Apr 15 2020, Jiri Slaby wrote:

> I was at 8f3d9f354286 of:
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>
> I did git remote update today and it fetched:
> Receiving objects: 100% (7330823/7330823), 1.20 GiB
> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.

What's your git version?  I just did exactly the same update with git
2.26.1, and it only fetched 144 objects:

Receiving objects: 100% (144/144), 50.50 KiB | 1.87 MiB/s, done.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15  8:11 ` Andreas Schwab
@ 2020-04-15  8:16   ` Jiri Slaby
  2020-04-15  8:23     ` Andreas Schwab
  0 siblings, 1 reply; 20+ messages in thread
From: Jiri Slaby @ 2020-04-15  8:16 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: git

On 15. 04. 20, 10:11, Andreas Schwab wrote:
> On Apr 15 2020, Jiri Slaby wrote:
> 
>> I was at 8f3d9f354286 of:
>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>
>> I did git remote update today and it fetched:
>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
> 
> What's your git version?  I just did exactly the same update with git
> 2.26.1, and it only fetched 144 objects:
> 
> Receiving objects: 100% (144/144), 50.50 KiB | 1.87 MiB/s, done.

$ git --version
git version 2.26.1

The same as you have.

-- 
js
suse labs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15  8:16   ` Jiri Slaby
@ 2020-04-15  8:23     ` Andreas Schwab
  2020-04-15  8:27       ` Jiri Slaby
  0 siblings, 1 reply; 20+ messages in thread
From: Andreas Schwab @ 2020-04-15  8:23 UTC (permalink / raw)
  To: Jiri Slaby; +Cc: git

On Apr 15 2020, Jiri Slaby wrote:

> On 15. 04. 20, 10:11, Andreas Schwab wrote:
>> On Apr 15 2020, Jiri Slaby wrote:
>> 
>>> I was at 8f3d9f354286 of:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>
>>> I did git remote update today and it fetched:
>>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
>> 
>> What's your git version?  I just did exactly the same update with git
>> 2.26.1, and it only fetched 144 objects:
>> 
>> Receiving objects: 100% (144/144), 50.50 KiB | 1.87 MiB/s, done.
>
> $ git --version
> git version 2.26.1

Was this the first time you used git >= 2.26.0?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15  8:23     ` Andreas Schwab
@ 2020-04-15  8:27       ` Jiri Slaby
  2020-04-15  8:34         ` Andreas Schwab
  0 siblings, 1 reply; 20+ messages in thread
From: Jiri Slaby @ 2020-04-15  8:27 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: git

On 15. 04. 20, 10:23, Andreas Schwab wrote:
> On Apr 15 2020, Jiri Slaby wrote:
> 
>> On 15. 04. 20, 10:11, Andreas Schwab wrote:
>>> On Apr 15 2020, Jiri Slaby wrote:
>>>
>>>> I was at 8f3d9f354286 of:
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>>
>>>> I did git remote update today and it fetched:
>>>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>>>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
>>>
>>> What's your git version?  I just did exactly the same update with git
>>> 2.26.1, and it only fetched 144 objects:
>>>
>>> Receiving objects: 100% (144/144), 50.50 KiB | 1.87 MiB/s, done.
>>
>> $ git --version
>> git version 2.26.1
> 
> Was this the first time you used git >= 2.26.0?

I doubt that:
$ grep git-core /var/log/zypp/history|sed 's@x86_64.*@@'
2020-03-23 16:14:24|install|git-core|2.25.2-467.2|
2020-03-24 17:19:37|install|git-core|2.26.0-468.1|
2020-03-29 09:38:34|install|git-core|2.26.0-471.1|
2020-04-06 06:34:36|install|git-core|2.26.0-473.1|
2020-04-15 08:21:37|install|git-core|2.26.1-474.1|

This is the first time I used 2.26.1, though.

thanks,
-- 
js
suse labs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15  8:27       ` Jiri Slaby
@ 2020-04-15  8:34         ` Andreas Schwab
  0 siblings, 0 replies; 20+ messages in thread
From: Andreas Schwab @ 2020-04-15  8:34 UTC (permalink / raw)
  To: Jiri Slaby; +Cc: git

On Apr 15 2020, Jiri Slaby wrote:

> This is the first time I used 2.26.1, though.

Same for me.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15  8:01 Fetching 24 Linux commits = 1.2 GiB Jiri Slaby
  2020-04-15  8:11 ` Andreas Schwab
@ 2020-04-15 10:52 ` Kevin Daudt
  2020-04-15 13:56 ` Konstantin Ryabitsev
  2 siblings, 0 replies; 20+ messages in thread
From: Kevin Daudt @ 2020-04-15 10:52 UTC (permalink / raw)
  To: Jiri Slaby; +Cc: git

On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
> Hi,
> 
> I was at 8f3d9f354286 of:
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 
> I did git remote update today and it fetched:
> Receiving objects: 100% (7330823/7330823), 1.20 GiB
> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
> 
> One colleague of mine fetched 1324 commits:
> Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
> Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
>    7e63420847ae..8632e9b5645b  master     -> origin/master
> 
> Another colleague fetched the same what I and:
>   Receiving objects: 100% (7330823/7330823), 1.20 GiB
> too.
> 
> I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
> 
> Is that a bug? What info should I provide?
> 
> thanks,
> -- 
> js
> suse labs

Not as big as you report, but I recall a user on IRC (guardian) was
wondering as well why a packfile was 240MB while they claimed that the
objects were committed months ago.



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15  8:01 Fetching 24 Linux commits = 1.2 GiB Jiri Slaby
  2020-04-15  8:11 ` Andreas Schwab
  2020-04-15 10:52 ` Kevin Daudt
@ 2020-04-15 13:56 ` Konstantin Ryabitsev
  2020-04-15 15:08   ` Junio C Hamano
  2020-04-16  6:31   ` Jiri Slaby
  2 siblings, 2 replies; 20+ messages in thread
From: Konstantin Ryabitsev @ 2020-04-15 13:56 UTC (permalink / raw)
  To: Jiri Slaby; +Cc: git

On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
> Hi,
> 
> I was at 8f3d9f354286 of:
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 
> I did git remote update today and it fetched:
> Receiving objects: 100% (7330823/7330823), 1.20 GiB
> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
> 
> One colleague of mine fetched 1324 commits:
> Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
> Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
>    7e63420847ae..8632e9b5645b  master     -> origin/master
> 
> Another colleague fetched the same what I and:
>   Receiving objects: 100% (7330823/7330823), 1.20 GiB
> too.
> 
> I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
> 
> Is that a bug? What info should I provide?

I've helped sfr troubleshoot the same issue last week -- it's most 
likely due to 2.26 turning on protocol version=2 by default.  
Unfortunately, reproducing this has been tricky, so if you can reliably 
make this happen again, then providing a full copy of your local tree as 
well as the remote you're trying to fetch may greatly help narrow it 
down.

With sfr (for whom fetching 1.2G from .au is a bit of a big deal), we 
solved it by forcing protocol.version=1.

-K

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15 13:56 ` Konstantin Ryabitsev
@ 2020-04-15 15:08   ` Junio C Hamano
  2020-04-15 15:16     ` Jeff King
                       ` (2 more replies)
  2020-04-16  6:31   ` Jiri Slaby
  1 sibling, 3 replies; 20+ messages in thread
From: Junio C Hamano @ 2020-04-15 15:08 UTC (permalink / raw)
  To: Jonathan Nieder, Jonathan Tan; +Cc: git

Do these (and I think we saw other reports) make us rethink the
status of protocol v2 as the default?  Are all of these fallouts 
we saw so far easy-to-fix bugs, or are there more fundamental issues
in the v2 protocol design?

Thanks.

Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:

> On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
>> Hi,
>> 
>> I was at 8f3d9f354286 of:
>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>> 
>> I did git remote update today and it fetched:
>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
>> 
>> One colleague of mine fetched 1324 commits:
>> Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
>> Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
>> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
>>    7e63420847ae..8632e9b5645b  master     -> origin/master
>> 
>> Another colleague fetched the same what I and:
>>   Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> too.
>> 
>> I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
>> 
>> Is that a bug? What info should I provide?
>
> I've helped sfr troubleshoot the same issue last week -- it's most 
> likely due to 2.26 turning on protocol version=2 by default.  
> Unfortunately, reproducing this has been tricky, so if you can reliably 
> make this happen again, then providing a full copy of your local tree as 
> well as the remote you're trying to fetch may greatly help narrow it 
> down.
>
> With sfr (for whom fetching 1.2G from .au is a bit of a big deal), we 
> solved it by forcing protocol.version=1.
>
> -K

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15 15:08   ` Junio C Hamano
@ 2020-04-15 15:16     ` Jeff King
  2020-04-15 18:52     ` Jonathan Nieder
  2020-05-20  8:53     ` Andreas Schwab
  2 siblings, 0 replies; 20+ messages in thread
From: Jeff King @ 2020-04-15 15:16 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Jonathan Nieder, Jonathan Tan, git

On Wed, Apr 15, 2020 at 08:08:17AM -0700, Junio C Hamano wrote:

> Do these (and I think we saw other reports) make us rethink the
> status of protocol v2 as the default?  Are all of these fallouts 
> we saw so far easy-to-fix bugs, or are there more fundamental issues
> in the v2 protocol design?

I don't think we know yet.

I agree with Konstantin that the v2 switch is the likely culprit for
these issues, but without having been able to reproduce, I don't think
we know exactly what the problem is yet. It could be a protocol design
issue, or it could be a minor implementation bug.

Note that there is one other issue that's turned up, that I discussed
here:

  https://lore.kernel.org/git/20200328154936.GA1217052@coredump.intra.peff.net/

That's more fundamental to the v2 design, but:

  - it only happens when one side drops the connection, so it's not
    impacting normal operation (it does turn an error case into a hang,
    though, which can be rather annoying)

  - it's not in the network protocol itself, but rather the protocol
    between Git and the remote helper. So we could solve it purely as a
    client-side fix.

-Peff

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15 15:08   ` Junio C Hamano
  2020-04-15 15:16     ` Jeff King
@ 2020-04-15 18:52     ` Jonathan Nieder
  2020-05-20  8:53     ` Andreas Schwab
  2 siblings, 0 replies; 20+ messages in thread
From: Jonathan Nieder @ 2020-04-15 18:52 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Jonathan Tan, git, Jeff King, Konstantin Ryabitsev

Hi,

>> On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:

>>> I was at 8f3d9f354286 of:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>
>>> I did git remote update today and it fetched:
>>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.

I suspect this is due to negotiation differing in protocol v2: in
protocols that do not maintain server side state, we have to record
previously matched "have"s at each round and the number of additional
"have"s sent on top leads the server to have insufficient information
about what the client has.  In other words, I suspect that with
https:// in protocol v0 you would experience the same thing.

Does

	git config --global fetch.negotiationStrategy skipping

help?

Thanks,
Jonathan

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15 13:56 ` Konstantin Ryabitsev
  2020-04-15 15:08   ` Junio C Hamano
@ 2020-04-16  6:31   ` Jiri Slaby
  2020-04-21 10:02     ` Martin Wilck
  1 sibling, 1 reply; 20+ messages in thread
From: Jiri Slaby @ 2020-04-16  6:31 UTC (permalink / raw)
  To: git

On 15. 04. 20, 15:56, Konstantin Ryabitsev wrote:
> On Wed, Apr 15, 2020 at 10:01:46AM +0200, Jiri Slaby wrote:
>> Hi,
>>
>> I was at 8f3d9f354286 of:
>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>
>> I did git remote update today and it fetched:
>> Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> It updated master: 8f3d9f354286..8632e9b5645b, that is 24 small commits.
>>
>> One colleague of mine fetched 1324 commits:
>> Receiving objects: 100% (6820/6820), 4.21 MiB | 6.70 MiB/s, done.
>> Resolving deltas: 100% (5114/5114), completed with 1035 local objects.
>> From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
>>    7e63420847ae..8632e9b5645b  master     -> origin/master
>>
>> Another colleague fetched the same what I and:
>>   Receiving objects: 100% (7330823/7330823), 1.20 GiB
>> too.
>>
>> I did git gc --prune && git prune now and I am at 1.7G back from 3.5 G.
>>
>> Is that a bug? What info should I provide?
> 
> I've helped sfr troubleshoot the same issue last week -- it's most 
> likely due to 2.26 turning on protocol version=2 by default.  
> Unfortunately, reproducing this has been tricky, so if you can reliably 
> make this happen again, then providing a full copy of your local tree as 
> well as the remote you're trying to fetch may greatly help narrow it 
> down.

I tried hard, but cannot reproduce. I noticed a difference between
2.25.1 and 2.25.1+protocol.version=2, though:

$ git config protocol.version 1 # the default in 2.25
$ git fetch
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
8f3d9f354286745c751374f5f1fcafee6b3f3136
error: Server does not allow request for unadvertised object
8f3d9f354286745c751374f5f1fcafee6b3f3136
$ git config protocol.version 2
$ git fetch
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
8f3d9f354286745c751374f5f1fcafee6b3f3136
remote: Enumerating objects: 1433262, done.
...

Doing fetch v5.7-rc1 (which is 8f3d9f3 above) with proto 1 works. So the
server obviously advertises different set of objects with proto 1 and 2.

thanks,
-- 
js
suse labs

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-16  6:31   ` Jiri Slaby
@ 2020-04-21 10:02     ` Martin Wilck
  0 siblings, 0 replies; 20+ messages in thread
From: Martin Wilck @ 2020-04-21 10:02 UTC (permalink / raw)
  To: Jiri Slaby, git; +Cc: konstantin

On Thu, 2020-04-16 at 08:31 +0200, Jiri Slaby wrote:
> On 15. 04. 20, 15:56, Konstantin Ryabitsev wrote:
> > 
> I tried hard, but cannot reproduce. I noticed a difference between
> 2.25.1 and 2.25.1+protocol.version=2, though:
> 
> $ git config protocol.version 1 # the default in 2.25
> $ git fetch
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 8f3d9f354286745c751374f5f1fcafee6b3f3136
> error: Server does not allow request for unadvertised object
> 8f3d9f354286745c751374f5f1fcafee6b3f3136
> $ git config protocol.version 2
> $ git fetch
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 8f3d9f354286745c751374f5f1fcafee6b3f3136
> remote: Enumerating objects: 1433262, done.
> ...
> 
> Doing fetch v5.7-rc1 (which is 8f3d9f3 above) with proto 1 works. So
> the
> server obviously advertises different set of objects with proto 1 and
> 2.

I just had the same issue with several kernel repos. The affected repo 
is a bare clone of Linus' kernel repo, plus "stable" and various
maintainer repos as additional remotes.

Interestingly, a single "git fetch --dry-run" with protol version 1
"fixed" the issue. I hadn't expected that to happen, so unfortunately I
hadn't made a  backup of the repo before. As this is a 50GB bare repo,
I'd appreciate instructions on how exactly to archive/upload it for
debugging if this should happen again.

mwilck@apollon:linux.git[BARE:master]> git fetch --dry-run -v net
Looking up git.kernel.org ... done.
Connecting to git.kernel.org (port 9418) ... 136.144.49.103 done.
remote: Enumerating objects: 7331255, done.
remote: Counting objects: 100% (7331255/7331255), done.
remote: Compressing objects: 100% (1113751/1113751), done.
^C
mwilck@apollon:linux.git[BARE:master]> git config --get
protocol.version
mwilck@apollon:linux.git[BARE:master]> git config protocol.version 1
mwilck@apollon:linux.git[BARE:master]> git fetch --dry-run -v net
Looking up git.kernel.org ... done.
Connecting to git.kernel.org (port 9418) ... 136.144.49.103 done.
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 53 (delta 42), reused 0 (delta 0)
Unpacking objects: 100% (53/53), 8.06 KiB | 5.00 KiB/s, done.
^C

(this fixed the issue despite "--dry-run" and myself interrupting the
process).

mwilck@apollon:linux.git[BARE:master]> git config protocol.version 2
mwilck@apollon:linux.git[BARE:master]> git fetch --dry-run -v net
Looking up git.kernel.org ... done.
Connecting to git.kernel.org (port 9418) ... 136.144.49.103 done.
From git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
   9bacd25..a460fc5      master       -> net/master
 = [up to date]          v2.6.11      -> v2.6.11
...

Regards
Martin



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-04-15 15:08   ` Junio C Hamano
  2020-04-15 15:16     ` Jeff King
  2020-04-15 18:52     ` Jonathan Nieder
@ 2020-05-20  8:53     ` Andreas Schwab
  2020-05-20  8:57       ` Andreas Schwab
                         ` (2 more replies)
  2 siblings, 3 replies; 20+ messages in thread
From: Andreas Schwab @ 2020-05-20  8:53 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Jonathan Nieder, Jonathan Tan, git

On Apr 15 2020, Junio C Hamano wrote:

> Do these (and I think we saw other reports) make us rethink the
> status of protocol v2 as the default?  Are all of these fallouts 
> we saw so far easy-to-fix bugs, or are there more fundamental issues
> in the v2 protocol design?

I'm now seeing the issue myself, and can provide a backup of the
offending repository.

$ git count-objects -v
count: 17
size: 76
in-pack: 387240
packs: 37
size-pack: 203738
prune-packable: 0
garbage: 0
size-garbage: 0
alternate: /home/andreas/src/linux/git/torvalds/linux.git/objects
alternate: /home/andreas/src/linux/git/stable/linux-stable.git/objects
$ GIT_TRACE=1 git fetch
10:40:32.829450 git.c:439               trace: built-in: git fetch
10:40:33.133448 run-command.c:663       trace: run_command: unset GIT_DIR GIT_IMPLICIT_WORK_TREE GIT_PREFIX; git --git-dir=/daten/src/linux/git/torvalds/linux.git for-each-ref '--format=%(objectname)'
10:40:33.135756 git.c:439               trace: built-in: git for-each-ref '--format=%(objectname)'
10:40:33.143936 run-command.c:663       trace: run_command: unset GIT_DIR GIT_IMPLICIT_WORK_TREE GIT_PREFIX; git --git-dir=/daten/src/linux/git/stable/linux-stable.git for-each-ref '--format=%(objectname)'
10:40:33.146087 git.c:439               trace: built-in: git for-each-ref '--format=%(objectname)'
remote: Enumerating objects: 30796, done.
remote: Counting objects: 100% (30796/30796), done.
remote: Compressing objects: 100% (6965/6965), done.
10:40:40.102198 run-command.c:663       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 12342 on igel.home' --pack_header=2,7350969
10:40:40.104872 git.c:439               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 12342 on igel.home' --pack_header=2,7350969
^Cceiving objects:   0% (3092/7350969), 1.32 MiB | 891.00 KiB/s  

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-05-20  8:53     ` Andreas Schwab
@ 2020-05-20  8:57       ` Andreas Schwab
  2020-05-20 10:05       ` Michal Suchánek
  2020-05-20 19:40       ` Jeff King
  2 siblings, 0 replies; 20+ messages in thread
From: Andreas Schwab @ 2020-05-20  8:57 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Jonathan Nieder, Jonathan Tan, git

On Mai 20 2020, Andreas Schwab wrote:

> On Apr 15 2020, Junio C Hamano wrote:
>
>> Do these (and I think we saw other reports) make us rethink the
>> status of protocol v2 as the default?  Are all of these fallouts 
>> we saw so far easy-to-fix bugs, or are there more fundamental issues
>> in the v2 protocol design?
>
> I'm now seeing the issue myself, and can provide a backup of the
> offending repository.
>
> $ git count-objects -v
> count: 17
> size: 76
> in-pack: 387240
> packs: 37
> size-pack: 203738
> prune-packable: 0
> garbage: 0
> size-garbage: 0
> alternate: /home/andreas/src/linux/git/torvalds/linux.git/objects
> alternate: /home/andreas/src/linux/git/stable/linux-stable.git/objects
> $ GIT_TRACE=1 git fetch
> 10:40:32.829450 git.c:439               trace: built-in: git fetch
> 10:40:33.133448 run-command.c:663       trace: run_command: unset GIT_DIR GIT_IMPLICIT_WORK_TREE GIT_PREFIX; git --git-dir=/daten/src/linux/git/torvalds/linux.git for-each-ref '--format=%(objectname)'
> 10:40:33.135756 git.c:439               trace: built-in: git for-each-ref '--format=%(objectname)'
> 10:40:33.143936 run-command.c:663       trace: run_command: unset GIT_DIR GIT_IMPLICIT_WORK_TREE GIT_PREFIX; git --git-dir=/daten/src/linux/git/stable/linux-stable.git for-each-ref '--format=%(objectname)'
> 10:40:33.146087 git.c:439               trace: built-in: git for-each-ref '--format=%(objectname)'
> remote: Enumerating objects: 30796, done.
> remote: Counting objects: 100% (30796/30796), done.
> remote: Compressing objects: 100% (6965/6965), done.
> 10:40:40.102198 run-command.c:663       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 12342 on igel.home' --pack_header=2,7350969
> 10:40:40.104872 git.c:439               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 12342 on igel.home' --pack_header=2,7350969
> ^Cceiving objects:   0% (3092/7350969), 1.32 MiB | 891.00 KiB/s  

$ git -c protocol.version=1 fetch
remote: Enumerating objects: 13226, done.
remote: Counting objects: 100% (10881/10881), done.
remote: Compressing objects: 100% (2322/2322), done.
remote: Total 10276 (delta 7860), reused 10215 (delta 7818), pack-reused 0
Receiving objects: 100% (10276/10276), 4.13 MiB | 615.00 KiB/s, done.
Resolving deltas: 100% (7860/7860), completed with 192 local objects.
From git://git.kernel.org/pub/scm/linux/kernel/git/geert/linux-m68k
 + c7630f6386a3...4684d38e4d47 m68k-queue -> m68k-queue  (forced update)
   39d91d3c3b47..e886fc082483  master     -> master

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-05-20  8:53     ` Andreas Schwab
  2020-05-20  8:57       ` Andreas Schwab
@ 2020-05-20 10:05       ` Michal Suchánek
  2020-05-20 10:34         ` Andreas Schwab
  2020-05-20 19:40       ` Jeff King
  2 siblings, 1 reply; 20+ messages in thread
From: Michal Suchánek @ 2020-05-20 10:05 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: Junio C Hamano, Jonathan Nieder, Jonathan Tan, git

On Wed, May 20, 2020 at 10:53:36AM +0200, Andreas Schwab wrote:
> On Apr 15 2020, Junio C Hamano wrote:
> 
> > Do these (and I think we saw other reports) make us rethink the
> > status of protocol v2 as the default?  Are all of these fallouts 
> > we saw so far easy-to-fix bugs, or are there more fundamental issues
> > in the v2 protocol design?
> 
> I'm now seeing the issue myself, and can provide a backup of the
> offending repository.
What git version? Should be fixed in git 2.26.2 in SUSE and git 2.26.3
upstream.

Thanks

Michal

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-05-20 10:05       ` Michal Suchánek
@ 2020-05-20 10:34         ` Andreas Schwab
  0 siblings, 0 replies; 20+ messages in thread
From: Andreas Schwab @ 2020-05-20 10:34 UTC (permalink / raw)
  To: Michal Suchánek; +Cc: Junio C Hamano, Jonathan Nieder, Jonathan Tan, git

On Mai 20 2020, Michal Suchánek wrote:

> What git version?

2.26.2

> git 2.26.3

I don't see that anywhere.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-05-20  8:53     ` Andreas Schwab
  2020-05-20  8:57       ` Andreas Schwab
  2020-05-20 10:05       ` Michal Suchánek
@ 2020-05-20 19:40       ` Jeff King
  2020-05-25 18:22         ` Lukas Wunner
  2 siblings, 1 reply; 20+ messages in thread
From: Jeff King @ 2020-05-20 19:40 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: Junio C Hamano, Jonathan Nieder, Jonathan Tan, git

On Wed, May 20, 2020 at 10:53:36AM +0200, Andreas Schwab wrote:

> On Apr 15 2020, Junio C Hamano wrote:
> 
> > Do these (and I think we saw other reports) make us rethink the
> > status of protocol v2 as the default?  Are all of these fallouts 
> > we saw so far easy-to-fix bugs, or are there more fundamental issues
> > in the v2 protocol design?
> 
> I'm now seeing the issue myself, and can provide a backup of the
> offending repository.

The "too big fetch" issue has since been fixed in "master", as well as
reverting the switch to the v2 protocol (which I think is just
belt-and-suspenders; AFAIK there are no known issues after the fix).
Both will be in v2.27. I don't see anything on "maint", but they _could_
be part of an eventual v2.26.3.

The fix was merged in 0b07eecf6e (Merge branch 'jt/v2-fetch-nego-fix',
2020-05-01) for reference.

-Peff

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-05-20 19:40       ` Jeff King
@ 2020-05-25 18:22         ` Lukas Wunner
  2020-05-25 18:37           ` Junio C Hamano
  0 siblings, 1 reply; 20+ messages in thread
From: Lukas Wunner @ 2020-05-25 18:22 UTC (permalink / raw)
  To: Jeff King
  Cc: Andreas Schwab, Junio C Hamano, Jonathan Nieder, Jonathan Tan, git

On Wed, May 20, 2020 at 03:40:19PM -0400, Jeff King wrote:
> The "too big fetch" issue has since been fixed in "master", as well as
> reverting the switch to the v2 protocol (which I think is just
> belt-and-suspenders; AFAIK there are no known issues after the fix).
> Both will be in v2.27. I don't see anything on "maint", but they _could_
> be part of an eventual v2.26.3.
> 
> The fix was merged in 0b07eecf6e (Merge branch 'jt/v2-fetch-nego-fix',
> 2020-05-01) for reference.

Please consider cutting a v2.26.3 release with this fix at your
earliest convenience.  The waste of bandwidth is mind-boggling.
(> 1 GByte whenever fetching from a kernel remote.)

Thanks,

Lukas

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: Fetching 24 Linux commits = 1.2 GiB
  2020-05-25 18:22         ` Lukas Wunner
@ 2020-05-25 18:37           ` Junio C Hamano
  0 siblings, 0 replies; 20+ messages in thread
From: Junio C Hamano @ 2020-05-25 18:37 UTC (permalink / raw)
  To: Lukas Wunner
  Cc: Jeff King, Andreas Schwab, Jonathan Nieder, Jonathan Tan, git

Lukas Wunner <lukas@wunner.de> writes:

> On Wed, May 20, 2020 at 03:40:19PM -0400, Jeff King wrote:
>> The "too big fetch" issue has since been fixed in "master", as well as
>> reverting the switch to the v2 protocol (which I think is just
>> belt-and-suspenders; AFAIK there are no known issues after the fix).
>> Both will be in v2.27. I don't see anything on "maint", but they _could_
>> be part of an eventual v2.26.3.
>> 
>> The fix was merged in 0b07eecf6e (Merge branch 'jt/v2-fetch-nego-fix',
>> 2020-05-01) for reference.
>
> Please consider cutting a v2.26.3 release with this fix at your
> earliest convenience.  The waste of bandwidth is mind-boggling.
> (> 1 GByte whenever fetching from a kernel remote.)

In the meantime, v2.27.0 rc2 will be out tomorrow.  Please consider
helping us polish it by testing that version.

Thanks.

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, back to index

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-15  8:01 Fetching 24 Linux commits = 1.2 GiB Jiri Slaby
2020-04-15  8:11 ` Andreas Schwab
2020-04-15  8:16   ` Jiri Slaby
2020-04-15  8:23     ` Andreas Schwab
2020-04-15  8:27       ` Jiri Slaby
2020-04-15  8:34         ` Andreas Schwab
2020-04-15 10:52 ` Kevin Daudt
2020-04-15 13:56 ` Konstantin Ryabitsev
2020-04-15 15:08   ` Junio C Hamano
2020-04-15 15:16     ` Jeff King
2020-04-15 18:52     ` Jonathan Nieder
2020-05-20  8:53     ` Andreas Schwab
2020-05-20  8:57       ` Andreas Schwab
2020-05-20 10:05       ` Michal Suchánek
2020-05-20 10:34         ` Andreas Schwab
2020-05-20 19:40       ` Jeff King
2020-05-25 18:22         ` Lukas Wunner
2020-05-25 18:37           ` Junio C Hamano
2020-04-16  6:31   ` Jiri Slaby
2020-04-21 10:02     ` Martin Wilck

git@vger.kernel.org list mirror (unofficial, one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git