protocol v2: More data transmitted between client and server since v2.20.0

protocol v2: More data transmitted between client and server since v2.20.0

[Kaartic Sivaraam]

Hi all,

I recently came across a blog[1] about how protocol v2 speeds up the
transfer between the client and the server. It states that the amount
of data transmitted between the client and the server is less when
using the protocol v2.

To demonstrate the difference between v0 and v2 two commands were
mentioned in the blog. The results hold true for me when I test using
Git v2.18.1. The commands given there worked as mentioned in the blog
until Git v2.18.1. But when I run the same commands using Git v2.19.2
or the one I built from source in the 'next' branch (at aa96b0ce6b).
I'm not sure why this is the case. Some observations below:

Protocol v0
===========
# Git version: 2.18.1
$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_0_git_2.18.1_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13	refs/heads/master

$ wc -l git_protocol_0_git_2.18.1_stderr.txt
36848 git_protocol_0_git_2.18.1_stderr.txt


# Git version: 2.19.0
$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_0_git_2.19.0_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13	refs/heads/master

$ wc -l git_protocol_0_git_2.19.0_stderr.txt 
36848 git_protocol_0_git_2.19.0_stderr.txt


# Git version: 2.20.0
$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_0_git_2.20.0_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13	refs/heads/master

$ wc -l git_protocol_0_git_2.20.0_stderr.txt 
36848 git_protocol_0_git_2.20.0_stderr.txt


# Git version: 2.20.1.495.gaa96b0ce6b
$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_0_git_2.20.1-source_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13	refs/heads/master

$ wc -l git_protocol_0_git_2.20.1-source_stderr.txt 
36848 git_protocol_0_git_2.20.1-source_stderr.txt 


Protocol v2
===========
# Git version: 2.18.1
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_2_git_2.18.1_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13	refs/heads/master

$ wc -l git_protocol_2_git_2.18.1_stderr.txt
22 git_protocol_2_git_2.18.1_stderr.txt


# Git version: 2.19.0
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_2_git_2.19.0_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13	refs/heads/master

$ wc -l
22 git_protocol_2_git_2.19.0_stderr.txt


# Git version: 2.20.0
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_2_git_2.20.0_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13	refs/heads/master
04845fdeae75ba5de7c93992a5d55663edf647e0	refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master
04845fdeae75ba5de7c93992a5d55663edf647e0	refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master
d4d4ebadfb373518013382560b1f505eb6217f13	refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master

$ wc -l git_protocol_2_git_2.20.0_stderr.txt 
160971 git_protocol_2_git_2.20.0_stderr.txt


# Git version: 2.20.1.495.gaa96b0ce6b
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote git@gitlab.com:gitlab-org/gitlab-ce.git master 2>git_protocol_2_git_2.20.1-source_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13	refs/heads/master
04845fdeae75ba5de7c93992a5d55663edf647e0	refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master
04845fdeae75ba5de7c93992a5d55663edf647e0	refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master
d4d4ebadfb373518013382560b1f505eb6217f13	refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master

$ wc -l git_protocol_2_git_2.20.1-source_stderr.txt
160971 git_protocol_2_git_2.20.1-source_stderr.txt


Also, note that there are more refs printed when using v2 which is
spooky too. I thought of running a bisect to find the actual culprit
but wanted to know if this was *actually* an issue or not? Any
thoughts?


References
==========
[1]: 
https://about.gitlab.com/2018/12/10/git-protocol-v2-enabled-for-ssh-on-gitlab-dot-com/

--
Sivaraam

Re: protocol v2: More data transmitted between client and server since v2.20.0

[Kaartic Sivaraam]

Oops. Small correction in the message.


On Mon, 2019-01-21 at 03:00 +0530, Kaartic Sivaraam wrote:
> .. The commands given there worked as mentioned in the blog
> until Git v2.18.1. But when I run the same commands using Git v2.19.2

That's Git v2.20.0 not Git v2.19.2.


> or the one I built from source in the 'next' branch (at aa96b0ce6b).
> I'm not sure why this is the case. Some observations below:
> 


That's it. Sorry for the confusion :-(

--
Sivaraam

protocol v2: More data transmitted between client and server since v2.20.0

[Kaartic Sivaraam]

Hi all,

I recently came across a blog[1] about how protocol v2 speeds up the
transfer between the client and the server. It states that the amount
of data transmitted between the client and the server is less when
using the protocol v2.

To demonstrate the difference between v0 and v2 two commands were
mentioned in the blog. The results hold true for me when I test using
Git v2.18.1. The commands given there worked as mentioned in the blog
until Git v2.18.1. But when I run the same commands using Git v2.20.0
or the one I built from source in the 'next' branch (at aa96b0ce6b).
I'm not sure why this is the case. Some observations below:

Protocol v0
===========
# Git version: 2.18.1
$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote
git@gitlab.com:gitlab-org/gitlab-ce.git master
2>git_protocol_0_git_2.18.1_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master

$ wc -l git_protocol_0_git_2.18.1_stderr.txt
36848 git_protocol_0_git_2.18.1_stderr.txt


# Git version: 2.19.0
$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote
git@gitlab.com:gitlab-org/gitlab-ce.git master
2>git_protocol_0_git_2.19.0_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master

$ wc -l git_protocol_0_git_2.19.0_stderr.txt
36848 git_protocol_0_git_2.19.0_stderr.txt


# Git version: 2.20.0
$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote
git@gitlab.com:gitlab-org/gitlab-ce.git master
2>git_protocol_0_git_2.20.0_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master

$ wc -l git_protocol_0_git_2.20.0_stderr.txt
36848 git_protocol_0_git_2.20.0_stderr.txt


# Git version: 2.20.1.495.gaa96b0ce6b
$ GIT_TRACE_PACKET=1 git -c protocol.version=0 ls-remote
git@gitlab.com:gitlab-org/gitlab-ce.git master
2>git_protocol_0_git_2.20.1-source_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master

$ wc -l git_protocol_0_git_2.20.1-source_stderr.txt
36848 git_protocol_0_git_2.20.1-source_stderr.txt


Protocol v2
===========
# Git version: 2.18.1
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote
git@gitlab.com:gitlab-org/gitlab-ce.git master
2>git_protocol_2_git_2.18.1_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master

$ wc -l git_protocol_2_git_2.18.1_stderr.txt
22 git_protocol_2_git_2.18.1_stderr.txt


# Git version: 2.19.0
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote
git@gitlab.com:gitlab-org/gitlab-ce.git master
2>git_protocol_2_git_2.19.0_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master

$ wc -l
22 git_protocol_2_git_2.19.0_stderr.txt


# Git version: 2.20.0
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote
git@gitlab.com:gitlab-org/gitlab-ce.git master
2>git_protocol_2_git_2.20.0_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master
04845fdeae75ba5de7c93992a5d55663edf647e0
refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master
04845fdeae75ba5de7c93992a5d55663edf647e0
refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master
d4d4ebadfb373518013382560b1f505eb6217f13
refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master

$ wc -l git_protocol_2_git_2.20.0_stderr.txt
160971 git_protocol_2_git_2.20.0_stderr.txt


# Git version: 2.20.1.495.gaa96b0ce6b
$ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote
git@gitlab.com:gitlab-org/gitlab-ce.git master
2>git_protocol_2_git_2.20.1-source_stderr.txt
d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master
04845fdeae75ba5de7c93992a5d55663edf647e0
refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master
04845fdeae75ba5de7c93992a5d55663edf647e0
refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master
d4d4ebadfb373518013382560b1f505eb6217f13
refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master

$ wc -l git_protocol_2_git_2.20.1-source_stderr.txt
160971 git_protocol_2_git_2.20.1-source_stderr.txt


Also, note that there are more refs printed when using v2 which is
spooky too. I thought of running a bisect to find the actual culprit
but wanted to know if this was *actually* an issue or not? Any
thoughts?


References
==========
[1]: https://about.gitlab.com/2018/12/10/git-protocol-v2-enabled-for-ssh-on-gitlab-dot-com/

--
Sivaraam

P.S.: I previously sent this email but it doesn't seemed to have reached the
mailing list (as verified by checking the archives) for some unknown reason.
So, I've re-sent this email. Sorry if this was redundant.

Re: protocol v2: More data transmitted between client and server since v2.20.0

[Jeff King]

On Mon, Jan 21, 2019 at 11:45:20AM +0530, Kaartic Sivaraam wrote:

> I recently came across a blog[1] about how protocol v2 speeds up the
> transfer between the client and the server. It states that the amount
> of data transmitted between the client and the server is less when
> using the protocol v2.

Yes, but...

> # Git version: 2.19.0
> $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote
> git@gitlab.com:gitlab-org/gitlab-ce.git master
> 2>git_protocol_2_git_2.19.0_stderr.txt
> d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master
> 
> $ wc -l
> 22 git_protocol_2_git_2.19.0_stderr.txt
> 
> 
> # Git version: 2.20.0
> $ GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote
> git@gitlab.com:gitlab-org/gitlab-ce.git master
> 2>git_protocol_2_git_2.20.0_stderr.txt
> d4d4ebadfb373518013382560b1f505eb6217f13        refs/heads/master
> 04845fdeae75ba5de7c93992a5d55663edf647e0
> refs/remotes/remote_mirror_630f74462b3b08a952486da866d5e702/master
> 04845fdeae75ba5de7c93992a5d55663edf647e0
> refs/remotes/remote_mirror_655ad545056a2ad17e7ebc5461a986e4/master
> d4d4ebadfb373518013382560b1f505eb6217f13
> refs/remotes/remote_mirror_d612bbe5bee4fbc624df371bc7caa759/master
> 
> $ wc -l git_protocol_2_git_2.20.0_stderr.txt
> 160971 git_protocol_2_git_2.20.0_stderr.txt

v2.19, etc, were buggy. The rules for ls-remote's pattern matching do
not permit us to use ref prefixes (because the documentation specifies
that it matches "master" at the end of _any_ ref, not just in the usual
spots). But we did anyway. The fix is in 631f0f8c4b (ls-remote: do not
send ref prefixes for patterns, 2018-10-31).

That should explain the extra refs, as well (they all match "master",
too, and it was wrong that v2.19 did not show them).

It's rather unfortunate, since otherwise ls-remote makes for a nice test
of the advertisement code. ;) You can see some difference with
"ls-remote --heads", as explained in 6a139cdd74 (ls-remote: pass
heads/tags prefixes to transport, 2018-10-31).

The best test is to do a noop git-fetch. E.g.:

  # just make sure we have all the objects. You still see the benefit
  # without it, but if you really want to count bytes, it makes sure
  # you're comparing apples to apples.
  git fetch

  # this has a big useless ref advertisement
  GIT_TRACE_PACKET=$PWD/v0 git -c protocol.version=0 fetch origin master

  # and this is much smaller. Unfortunately it's not just a single line
  # (for "master"), because tag-following requires that the other side
  # tell us about tags, too.
  GIT_TRACE_PACKET=$PWD/v2 git -c protocol.version=2 fetch origin master

  # this one is really tiny
  GIT_TRACE_PACKET=$PWD/v2-notags git -c protocol.version=2 fetch --no-tags origin master

With my origin as https://github.com/gitster/git (which has more heads
than the regular git.git), I get:

  $ wc v[02]* | head -n -1
  2441  17098 340884 v0
   733   5828 130263 v2
    22    141   2037 v2-notags

-Peff

Re: protocol v2: More data transmitted between client and server since v2.20.0

[Kaartic.Sivaraam]

Hi Peff,

Thanks for sharing the details about that spooky behaviour! Also, thanks 
for suggesting an alternative way to show the advantage of using 
protocol v2. I'll refer your post in that blog's comment to let them know.


Thanks,
Sivaraam