Bug: Git: Clone: University Network: No Output on Terminal

Bug: Git: Clone: University Network: No Output on Terminal

[Manish Devgan]

Hello,

I am a university student from India and I was trying to clone the
xfwm4 git repository from the git-url :
https://git.xfce.org/xfce/xfwm4
I was trying to clone this repository over my university network and
it didn't work. I worked my way around by using my mobile
wireless-hotspot for the same and was able to do it just fine.
The next day I tried doing the exact same thing since I couldn't get
an answer from our university if any such firewall was blocking me
from cloning it. But this time instead of failing to reach the url the
command ran with the following output:

gabru-md@bhaalu:~/Desktop$ git clone https://git.xfce.org/xfce/xfwm4
Cloning into 'xfwm4'...

after waiting for a few minutes the command didn't fail but it didn't
show any output/stats like it does when I try to clone a repository
from Github which is definitely not blocked on the university network.
for eg.

gabru-md@bhaalu:~/Desktop$ git clone https://github.com/gabru-md/stl-cpp
Cloning into 'stl-cpp'...
remote: Enumerating objects: 186, done.
remote: Counting objects: 100% (186/186), done.
remote: Compressing objects: 100% (128/128), done.
remote: Total 186 (delta 92), reused 146 (delta 56), pack-reused 0
Receiving objects: 100% (186/186), 105.58 KiB | 256.00 KiB/s, done.
Resolving deltas: 100% (92/92), done.

Furthermore the interesting thing is that I expected that my attempt
at cloning the xfwm4 repository might have failed but it surprisingly
did not. The command did clone the xfwm4 repository for me but it
didn't print out any stats/output during that time.

After searching on DuckDuckGo I saw that adding flags `--verbose
--progress` will allow me to achieve the desired functionality. I
tried doing the same and the output is given below.

gabru-md@bhaalu:~/Desktop/test$ git clone --progress --verbose
https://git.xfce.org/xfce/xfwm4
Cloning into 'xfwm4'...
gabru-md@bhaalu:~/Desktop/test$ ls
xfwm4
gabru-md@bhaalu:~/Desktop/test$ cd xfwm4/
gabru-md@bhaalu:~/Desktop/test/xfwm4$ cd icons/
gabru-md@bhaalu:~/Desktop/test/xfwm4/icons$ ls
22x22  48x48  Makefile.am  scalable

It is evident that the cloning worked perfectly fine but there didn't
seem to be any output on the terminal. At first I was convinced that
it was network error because it couldn't reach the target url but now
it doesn't seem to be a network problem.

I am not quite sure if this unfamiliar behavior while cloning a
repository is a bug or not hence I thought of reporting it. Also,
Apologies if this is not a bug but something which is known already.

I am currently running git version 2.17.1 on my Xubuntu.


Thanks and Regards
Manish Devgan
manish.nsit8@gmail.com

Re: Bug: Git: Clone: University Network: No Output on Terminal

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2841 bytes --]

On 2020-02-17 at 08:02:44, Manish Devgan wrote:
> Hello,
> 
> I am a university student from India and I was trying to clone the
> xfwm4 git repository from the git-url :
> https://git.xfce.org/xfce/xfwm4
> I was trying to clone this repository over my university network and
> it didn't work. I worked my way around by using my mobile
> wireless-hotspot for the same and was able to do it just fine.
> The next day I tried doing the exact same thing since I couldn't get
> an answer from our university if any such firewall was blocking me
> from cloning it. But this time instead of failing to reach the url the
> command ran with the following output:
> 
> gabru-md@bhaalu:~/Desktop$ git clone https://git.xfce.org/xfce/xfwm4
> Cloning into 'xfwm4'...
> 
> after waiting for a few minutes the command didn't fail but it didn't
> show any output/stats like it does when I try to clone a repository
> from Github which is definitely not blocked on the university network.
> for eg.

So it looks like git.xfce.org is using the old dumb HTTP protocol, which
basically serves static content.  Git has to look up the references,
attempt to download each object or pack, and then determine which ones
have the objects it needs.  The server provides no assistance in that
regard.

> gabru-md@bhaalu:~/Desktop$ git clone https://github.com/gabru-md/stl-cpp
> Cloning into 'stl-cpp'...
> remote: Enumerating objects: 186, done.
> remote: Counting objects: 100% (186/186), done.
> remote: Compressing objects: 100% (128/128), done.
> remote: Total 186 (delta 92), reused 146 (delta 56), pack-reused 0
> Receiving objects: 100% (186/186), 105.58 KiB | 256.00 KiB/s, done.
> Resolving deltas: 100% (92/92), done.

Here, GitHub uses the smart HTTP protocol.  The client fetches the
references and then reports what data it has.  The server actively
computes the differences and sends a single pack containing only the
required objects.  In addition, newer versions of the smart HTTP
protocol contain a sideband where the server can report status, like the
"remote:" message you see above.  The status messages on the client side
come from unpacking and indexing the single pack received.

So this is working by design, although it's confusing for users.  The
dumb HTTP protocol doesn't know how much stuff it needs to fetch—it
could be a single object or pack, or it could be tens of thousands of
items—so it's hard to provide any progress report on the client side.
Additionally, because the server isn't actively producing a pack and is
just serving static files, there's no way for it to provide status
output at all.

Hopefully this answers your question, although I agree the answer is a
little unsatisfying.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: Bug: Git: Clone: University Network: No Output on Terminal

[René Scharfe]

Am 17.02.20 um 09:02 schrieb Manish Devgan:
> Hello,
>
> I am a university student from India and I was trying to clone the
> xfwm4 git repository from the git-url :
> https://git.xfce.org/xfce/xfwm4
> I was trying to clone this repository over my university network and
> it didn't work. I worked my way around by using my mobile
> wireless-hotspot for the same and was able to do it just fine.
> The next day I tried doing the exact same thing since I couldn't get
> an answer from our university if any such firewall was blocking me
> from cloning it. But this time instead of failing to reach the url the
> command ran with the following output:
>
> gabru-md@bhaalu:~/Desktop$ git clone https://git.xfce.org/xfce/xfwm4
> Cloning into 'xfwm4'...
>
> after waiting for a few minutes the command didn't fail but it didn't
> show any output/stats like it does when I try to clone a repository
> from Github which is definitely not blocked on the university network.
> for eg.

That doesn't sound ideal, but it also doesn't necessarily have to be a
bug in the sense that git clone broke at some point.  It could be caused
by the server side as well.

What you see here is the default output of the so-called dumb HTTP
transport.  AFAUI it doesn't show any progress (yet?).  You can get some
kind of progress messages by dialing up the verbosity way up with
repeated -v or --verbose options.  E.g.:

   $ git clone -vvv https://git.xfce.org/xfce/xfwm4
   Cloning into 'xfwm4'...
   got 9a2556cfeedcc1b73cea36e5045073ecdf9fd39a
   walk 9a2556cfeedcc1b73cea36e5045073ecdf9fd39a
   Getting alternates list for https://git.xfce.org/xfce/xfwm4
   Getting pack list for https://git.xfce.org/xfce/xfwm4
   got 72b002b0f028fe610ba22035453647001e886969
   Getting index for pack 3a53ad6629b13fd830436a47c59b86e29b565d88
   got 600cb96c7eed28529fc8d5d0383d58070cb30957
   Getting index for pack 9ef7bd53f4441a79be936132e1c4cde62f6fd9a8
   Getting index for pack cef42fa75f4badc8e7ece5fee5ff8068dc8ae51b
   [...]

> gabru-md@bhaalu:~/Desktop$ git clone https://github.com/gabru-md/stl-cpp
> Cloning into 'stl-cpp'...
> remote: Enumerating objects: 186, done.
> remote: Counting objects: 100% (186/186), done.
> remote: Compressing objects: 100% (128/128), done.
> remote: Total 186 (delta 92), reused 146 (delta 56), pack-reused 0
> Receiving objects: 100% (186/186), 105.58 KiB | 256.00 KiB/s, done.
> Resolving deltas: 100% (92/92), done.

This is the so-called smart HTTP transport at work, which does support
progress output.

You could use the git protocol to clone xfwm4:

   $ git clone git://git.xfce.org/xfce/xfwm4
   Cloning into 'xfwm4'...
   remote: Counting objects: 66501, done.
   remote: Compressing objects: 100% (16901/16901), done.
   [...]

> Furthermore the interesting thing is that I expected that my attempt
> at cloning the xfwm4 repository might have failed but it surprisingly
> did not. The command did clone the xfwm4 repository for me but it
> didn't print out any stats/output during that time.
>
> After searching on DuckDuckGo I saw that adding flags `--verbose
> --progress` will allow me to achieve the desired functionality. I
> tried doing the same and the output is given below.
>
> gabru-md@bhaalu:~/Desktop/test$ git clone --progress --verbose
> https://git.xfce.org/xfce/xfwm4
> Cloning into 'xfwm4'...
> gabru-md@bhaalu:~/Desktop/test$ ls
> xfwm4
> gabru-md@bhaalu:~/Desktop/test$ cd xfwm4/
> gabru-md@bhaalu:~/Desktop/test/xfwm4$ cd icons/
> gabru-md@bhaalu:~/Desktop/test/xfwm4/icons$ ls
> 22x22  48x48  Makefile.am  scalable
>
> It is evident that the cloning worked perfectly fine but there didn't
> seem to be any output on the terminal. At first I was convinced that
> it was network error because it couldn't reach the target url but now
> it doesn't seem to be a network problem.
>
> I am not quite sure if this unfamiliar behavior while cloning a
> repository is a bug or not hence I thought of reporting it. Also,
> Apologies if this is not a bug but something which is known already.
>
> I am currently running git version 2.17.1 on my Xubuntu.

Git 2.25.1 shows the same behavior.

I guess the reason the dumb HTTP transport doesn't show any progress is
that there is no way to know how much data it needs to download, so it
can't calculate a percentage of work done.  Perhaps showing a number
that is incremented with each GET request or a spinning wheel or so
could reassure users that git clone is still working?

René

Re: Bug: Git: Clone: University Network: No Output on Terminal

[Manish Devgan]

On Tue, Feb 18, 2020 at 1:08 AM René Scharfe <l.s.r@web.de> wrote:
>
> Am 17.02.20 um 09:02 schrieb Manish Devgan:
> > Hello,
> >
> > I am a university student from India and I was trying to clone the
> > xfwm4 git repository from the git-url :
> > https://git.xfce.org/xfce/xfwm4
> > I was trying to clone this repository over my university network and
> > it didn't work. I worked my way around by using my mobile
> > wireless-hotspot for the same and was able to do it just fine.
> > The next day I tried doing the exact same thing since I couldn't get
> > an answer from our university if any such firewall was blocking me
> > from cloning it. But this time instead of failing to reach the url the
> > command ran with the following output:
> >
> > gabru-md@bhaalu:~/Desktop$ git clone https://git.xfce.org/xfce/xfwm4
> > Cloning into 'xfwm4'...
> >
> > after waiting for a few minutes the command didn't fail but it didn't
> > show any output/stats like it does when I try to clone a repository
> > from Github which is definitely not blocked on the university network.
> > for eg.
>
> That doesn't sound ideal, but it also doesn't necessarily have to be a
> bug in the sense that git clone broke at some point.  It could be caused
> by the server side as well.
>
> What you see here is the default output of the so-called dumb HTTP
> transport.  AFAUI it doesn't show any progress (yet?).  You can get some
> kind of progress messages by dialing up the verbosity way up with
> repeated -v or --verbose options.  E.g.:

Thanks I will make sure to use the -vvv option next time.

>
>    $ git clone -vvv https://git.xfce.org/xfce/xfwm4
>    Cloning into 'xfwm4'...
>    got 9a2556cfeedcc1b73cea36e5045073ecdf9fd39a
>    walk 9a2556cfeedcc1b73cea36e5045073ecdf9fd39a
>    Getting alternates list for https://git.xfce.org/xfce/xfwm4
>    Getting pack list for https://git.xfce.org/xfce/xfwm4
>    got 72b002b0f028fe610ba22035453647001e886969
>    Getting index for pack 3a53ad6629b13fd830436a47c59b86e29b565d88
>    got 600cb96c7eed28529fc8d5d0383d58070cb30957
>    Getting index for pack 9ef7bd53f4441a79be936132e1c4cde62f6fd9a8
>    Getting index for pack cef42fa75f4badc8e7ece5fee5ff8068dc8ae51b
>    [...]

I tried it this way and it worked just like you've mentioned.

>
> > gabru-md@bhaalu:~/Desktop$ git clone https://github.com/gabru-md/stl-cpp
> > Cloning into 'stl-cpp'...
> > remote: Enumerating objects: 186, done.
> > remote: Counting objects: 100% (186/186), done.
> > remote: Compressing objects: 100% (128/128), done.
> > remote: Total 186 (delta 92), reused 146 (delta 56), pack-reused 0
> > Receiving objects: 100% (186/186), 105.58 KiB | 256.00 KiB/s, done.
> > Resolving deltas: 100% (92/92), done.
>
> This is the so-called smart HTTP transport at work, which does support
> progress output.
>
> You could use the git protocol to clone xfwm4:
>
>    $ git clone git://git.xfce.org/xfce/xfwm4
>    Cloning into 'xfwm4'...
>    remote: Counting objects: 66501, done.
>    remote: Compressing objects: 100% (16901/16901), done.
>    [...]

The problem is that my university has blocked git protocol and hence I
cannot access it to clone the repository.
Moreover cloning and pushing over ssh are also blocked which is really
bad. The only way to get work done is to use HTTP. :(

>
> > Furthermore the interesting thing is that I expected that my attempt
> > at cloning the xfwm4 repository might have failed but it surprisingly
> > did not. The command did clone the xfwm4 repository for me but it
> > didn't print out any stats/output during that time.
> >
> > After searching on DuckDuckGo I saw that adding flags `--verbose
> > --progress` will allow me to achieve the desired functionality. I
> > tried doing the same and the output is given below.
> >
> > gabru-md@bhaalu:~/Desktop/test$ git clone --progress --verbose
> > https://git.xfce.org/xfce/xfwm4
> > Cloning into 'xfwm4'...
> > gabru-md@bhaalu:~/Desktop/test$ ls
> > xfwm4
> > gabru-md@bhaalu:~/Desktop/test$ cd xfwm4/
> > gabru-md@bhaalu:~/Desktop/test/xfwm4$ cd icons/
> > gabru-md@bhaalu:~/Desktop/test/xfwm4/icons$ ls
> > 22x22  48x48  Makefile.am  scalable
> >
> > It is evident that the cloning worked perfectly fine but there didn't
> > seem to be any output on the terminal. At first I was convinced that
> > it was network error because it couldn't reach the target url but now
> > it doesn't seem to be a network problem.
> >
> > I am not quite sure if this unfamiliar behavior while cloning a
> > repository is a bug or not hence I thought of reporting it. Also,
> > Apologies if this is not a bug but something which is known already.
> >
> > I am currently running git version 2.17.1 on my Xubuntu.
>
> Git 2.25.1 shows the same behavior.
>
> I guess the reason the dumb HTTP transport doesn't show any progress is
> that there is no way to know how much data it needs to download, so it
> can't calculate a percentage of work done.  Perhaps showing a number
> that is incremented with each GET request or a spinning wheel or so
> could reassure users that git clone is still working?
>
> René
>

Yes. Having some sort of assurance that the clone command has not
failed would definitely be great.
The -vvv option puts out a great deal of stuff on my terminal which is
a sort of assurance that the command is working but it seems to print
huge loads of text which definitely does not make any sense to me at
least. I'd be more happy to see as you mentioned, a number or perhaps
a spinning wheel
which denotes that the command is working.

Thanks & Regards
Manish Devgan

Re: Bug: Git: Clone: University Network: No Output on Terminal

[René Scharfe]

Am 18.02.20 um 12:25 schrieb Manish Devgan:
> The -vvv option puts out a great deal of stuff on my terminal which is
> a sort of assurance that the command is working but it seems to print
> huge loads of text which definitely does not make any sense to me at
> least. I'd be more happy to see as you mentioned, a number or perhaps
> a spinning wheel
> which denotes that the command is working.

How about something like this?

-- >8 --
Subject: [PATCH] remote-curl: show progress for fetches over dumb HTTP

Fetching over dumb HTTP transport doesn't show any progress, even with
the option --progress.  If the connection is slow or there is a lot of
data to get then this can take a long time while the user is left to
wonder if git got stuck.

We don't know the number of objects to fetch at the outset, but we can
count the ones we got.  Show an open-ended progress indicator based on
that number if the user asked for it.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 remote-curl.c |  1 +
 walker.c      | 13 ++++++++++++-
 walker.h      |  1 +
 3 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/remote-curl.c b/remote-curl.c
index 8eb96152f5..e4cd321844 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -1026,6 +1026,7 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)

 	walker = get_http_walker(url.buf);
 	walker->get_verbosely = options.verbosity >= 3;
+	walker->get_progress = options.progress;
 	walker->get_recover = 0;
 	ret = walker_fetch(walker, nr_heads, targets, NULL, NULL);
 	walker_free(walker);
diff --git a/walker.c b/walker.c
index bb010f7a2b..4984bf8b3d 100644
--- a/walker.c
+++ b/walker.c
@@ -8,6 +8,7 @@
 #include "tag.h"
 #include "blob.h"
 #include "refs.h"
+#include "progress.h"

 static struct object_id current_commit_oid;

@@ -162,6 +163,11 @@ static int process(struct walker *walker, struct object *obj)
 static int loop(struct walker *walker)
 {
 	struct object_list *elem;
+	struct progress *progress = NULL;
+	uint64_t nr = 0;
+
+	if (walker->get_progress)
+		progress = start_delayed_progress(_("Fetching objects"), 0);

 	while (process_queue) {
 		struct object *obj = process_queue->item;
@@ -176,15 +182,20 @@ static int loop(struct walker *walker)
 		 */
 		if (! (obj->flags & TO_SCAN)) {
 			if (walker->fetch(walker, obj->oid.hash)) {
+				stop_progress(&progress);
 				report_missing(obj);
 				return -1;
 			}
 		}
 		if (!obj->type)
 			parse_object(the_repository, &obj->oid);
-		if (process_object(walker, obj))
+		if (process_object(walker, obj)) {
+			stop_progress(&progress);
 			return -1;
+		}
+		display_progress(progress, ++nr);
 	}
+	stop_progress(&progress);
 	return 0;
 }

diff --git a/walker.h b/walker.h
index 6d8ae00e5b..d40b016bab 100644
--- a/walker.h
+++ b/walker.h
@@ -10,6 +10,7 @@ struct walker {
 	int (*fetch)(struct walker *, unsigned char *sha1);
 	void (*cleanup)(struct walker *);
 	int get_verbosely;
+	int get_progress;
 	int get_recover;

 	int corrupt_object_found;
--
2.25.1

Re: Bug: Git: Clone: University Network: No Output on Terminal

[Manish Devgan]

On Wed, Feb 19, 2020 at 12:54 AM René Scharfe <l.s.r@web.de> wrote:

> How about something like this?
>
> -- >8 --
> Subject: [PATCH] remote-curl: show progress for fetches over dumb HTTP
>
> Fetching over dumb HTTP transport doesn't show any progress, even with
> the option --progress.  If the connection is slow or there is a lot of
> data to get then this can take a long time while the user is left to
> wonder if git got stuck.
>
> We don't know the number of objects to fetch at the outset, but we can
> count the ones we got.  Show an open-ended progress indicator based on
> that number if the user asked for it.
>
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  remote-curl.c |  1 +
>  walker.c      | 13 ++++++++++++-
>  walker.h      |  1 +
>  3 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/remote-curl.c b/remote-curl.c
> index 8eb96152f5..e4cd321844 100644
> --- a/remote-curl.c
> +++ b/remote-curl.c
> @@ -1026,6 +1026,7 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)
>
>         walker = get_http_walker(url.buf);
>         walker->get_verbosely = options.verbosity >= 3;
> +       walker->get_progress = options.progress;
>         walker->get_recover = 0;
>         ret = walker_fetch(walker, nr_heads, targets, NULL, NULL);
>         walker_free(walker);
> diff --git a/walker.c b/walker.c
> index bb010f7a2b..4984bf8b3d 100644
> --- a/walker.c
> +++ b/walker.c
> @@ -8,6 +8,7 @@
>  #include "tag.h"
>  #include "blob.h"
>  #include "refs.h"
> +#include "progress.h"
>
>  static struct object_id current_commit_oid;
>
> @@ -162,6 +163,11 @@ static int process(struct walker *walker, struct object *obj)
>  static int loop(struct walker *walker)
>  {
>         struct object_list *elem;
> +       struct progress *progress = NULL;
> +       uint64_t nr = 0;
> +
> +       if (walker->get_progress)
> +               progress = start_delayed_progress(_("Fetching objects"), 0);
>
>         while (process_queue) {
>                 struct object *obj = process_queue->item;
> @@ -176,15 +182,20 @@ static int loop(struct walker *walker)
>                  */
>                 if (! (obj->flags & TO_SCAN)) {
>                         if (walker->fetch(walker, obj->oid.hash)) {
> +                               stop_progress(&progress);
>                                 report_missing(obj);
>                                 return -1;
>                         }
>                 }
>                 if (!obj->type)
>                         parse_object(the_repository, &obj->oid);
> -               if (process_object(walker, obj))
> +               if (process_object(walker, obj)) {
> +                       stop_progress(&progress);
>                         return -1;
> +               }
> +               display_progress(progress, ++nr);
>         }
> +       stop_progress(&progress);
>         return 0;
>  }
>
> diff --git a/walker.h b/walker.h
> index 6d8ae00e5b..d40b016bab 100644
> --- a/walker.h
> +++ b/walker.h
> @@ -10,6 +10,7 @@ struct walker {
>         int (*fetch)(struct walker *, unsigned char *sha1);
>         void (*cleanup)(struct walker *);
>         int get_verbosely;
> +       int get_progress;
>         int get_recover;
>
>         int corrupt_object_found;
> --
> 2.25.1

This is great. But perhaps I would want the output to come
irrespective of --progress argument in case of dumb_http.

Thanks & Regards
Manish Devgan

Re: Bug: Git: Clone: University Network: No Output on Terminal

[René Scharfe]

Am 20.02.20 um 11:20 schrieb Manish Devgan:
> This is great. But perhaps I would want the output to come
> irrespective of --progress argument in case of dumb_http.

It's on by default if you use git on a terminal (as opposed to calling
it from a script).

René