ctrl-z ignored by git; creates blobs from non-existent repos

ctrl-z ignored by git; creates blobs from non-existent repos

[Crls]

Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent 
repos

Steps to reproduce 1- git clone github 
whateverrepo/whatevernonexistentrepo or 1- git clone gitlab 
whateverrepo/whatevernonexistentrepo 2= Git prompts for a username

3- Press Ctrl-Z to stop *git* from running either on the virtual 
console/tty *git* automatically creates blobs with directories and 
disregards

^Z altogether It goes ahead anyway and creates pleases. It looks as if 
^Z is not enough.

Expected: The same issue does not happen with other non-existent repos 
e.g., git clone git.zx2c4/ it returns the message of fatal repo not found

   According to console_codes (4) CAN (0x18, ^X) and SUB (0x1A, ^Z) 
abort escape sequences should be clear enough


git version 2.36.3

Re: ctrl-z ignored by git; creates blobs from non-existent repos

[Theodore Ts'o]

On Fri, Jan 13, 2023 at 05:01:01PM -0500, Crls wrote:
> Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent
> repos
> 
> Steps to reproduce 1- git clone github whateverrepo/whatevernonexistentrepo
> or 1- git clone gitlab whateverrepo/whatevernonexistentrepo 2= Git prompts
> for a username

% git clone github whateverrepo/whatevernonexistentrepo
fatal: repository 'github' does not exist

I think what you meant was:

% git clone https://github.com/whateverrepo/whatevernonexistentrepo
Cloning into 'whatevernonexistentrepo'...
Username for 'https://github.com': 

> 3- Press Ctrl-Z to stop *git* from running either on the virtual console/tty
> *git* automatically creates blobs with directories and disregards

So it's not that Control-Z is being ignored.  It's that by the time
you see the prompt for "Username for 'https://github.com': ", the
directories already exist.  Try looking at
whatevernonexistentrepo/.git as soon as the prompt shows up.  You'll
see that the .git directory has been greated.

Now, when you type ^Z, the git processes are stopped --- but the
objects are created already.

Username for 'https://github.com': ^Z
[1]+  Stopped                 git clone https://github.com/whateverrepo/whatevernonexistentrepo
% ps aux | grep git
tytso       5097  0.0  0.0   9736  4480 pts/0    T    10:41   0:00 git clone https://github.com/wha
tytso       5098  0.0  0.0   9736  3992 pts/0    T    10:41   0:00 /usr/lib/git-core/git remote-htt
tytso       5099  0.0  0.1 102332 16104 pts/0    T    10:41   0:00 /usr/lib/git-core/git-remote-htt
tytso       5140  0.0  0.0   6332  2072 pts/0    S+   10:43   0:00 grep git


The 'T' means that the processes are stopped.

> Expected: The same issue does not happen with other non-existent repos e.g.,
> git clone git.zx2c4/ it returns the message of fatal repo not found

So what's going on is that github.com is not returning a non-existent
repo error; it's prompting for a username/password, as _if_ the
repository exists.  That's presumably to prevent disclosing
information as to whether or not a private repository exists or not.

Once the authentication fails, git will remove the partially created
repro, so it's really not a problem in practice.

Cheers,

						- Ted

Re: ctrl-z ignored by git; creates blobs from non-existent repos

[Jeff King]

On Sun, Jan 15, 2023 at 10:45:19AM -0500, Theodore Ts'o wrote:

> > Expected: The same issue does not happen with other non-existent repos e.g.,
> > git clone git.zx2c4/ it returns the message of fatal repo not found
> 
> So what's going on is that github.com is not returning a non-existent
> repo error; it's prompting for a username/password, as _if_ the
> repository exists.  That's presumably to prevent disclosing
> information as to whether or not a private repository exists or not.

I can confirm that this is exactly the reason.

-Peff

Re: ctrl-z ignored by git; creates blobs from non-existent repos

[Crls]

On Sun, Jan 15, 2023 at 10:45:19AM -0500, Theodore Ts'o wrote:
> On Fri, Jan 13, 2023 at 05:01:01PM -0500, Crls wrote:
> > Ctrl-Z is ignored by git; Git-clone injects blobs even with non-existent
> > repos
> > 
> > Steps to reproduce 1- git clone github whateverrepo/whatevernonexistentrepo
> > or 1- git clone gitlab whateverrepo/whatevernonexistentrepo 2= Git prompts
> > for a username
> 
> % git clone github whateverrepo/whatevernonexistentrepo
> fatal: repository 'github' does not exist
> 
> I think what you meant was:
> 
> % git clone https://github.com/whateverrepo/whatevernonexistentrepo
> Cloning into 'whatevernonexistentrepo'...
> Username for 'https://github.com': 
> 
Yes. That's what I meant… thank you. I was having a problem with the formatting while sending the message to the mailing list

Content-Policy reject msg: The message contains HTML subpart, therefore we consider it SPAM or Outlook Virus. TEXT/PLAIN is accepted.! BF:; S229631AbjAMVJQ (in reply to end of DATA command)


> > 3- Press Ctrl-Z to stop *git* from running either on the virtual console/tty
> > *git* automatically creates blobs with directories and disregards
> 
> So it's not that Control-Z is being ignored.  It's that by the time
> you see the prompt for "Username for 'https://github.com': ", the
> directories already exist.  Try looking at
> whatevernonexistentrepo/.git as soon as the prompt shows up.  You'll
> see that the .git directory has been greated.
> 
> Now, when you type ^Z, the git processes are stopped --- but the
> objects are created already.

The directories exist not because ^Z is used, but because by the time git-clone prompts for a username, git is already set on what to do next. Correct? in other words, the process is shoved down my throat. Or the user's throat in this case. Or going by another analogy, it certainly sounds as if I meant: «Git, please git-clone such and such repo, but let me fix just a typo on the repo name before submitting it, pretty please» and then Git replies: «too late for that chick-a-doodle» and there it goes. It injects blobs all over (well, not all over but on the dir specified).


> 
> Username for 'https://github.com': ^Z
> [1]+  Stopped                 git clone https://github.com/whateverrepo/whatevernonexistentrepo
> % ps aux | grep git
> tytso       5097  0.0  0.0   9736  4480 pts/0    T    10:41   0:00 git clone https://github.com/wha
> tytso       5098  0.0  0.0   9736  3992 pts/0    T    10:41   0:00 /usr/lib/git-core/git remote-htt
> tytso       5099  0.0  0.1 102332 16104 pts/0    T    10:41   0:00 /usr/lib/git-core/git-remote-htt
> tytso       5140  0.0  0.0   6332  2072 pts/0    S+   10:43   0:00 grep git
> 
> 
> The 'T' means that the processes are stopped.
> 
> > Expected: The same issue does not happen with other non-existent repos e.g.,
> > git clone git.zx2c4/ it returns the message of fatal repo not found
> 
> So what's going on is that github.com is not returning a non-existent
> repo error; it's prompting for a username/password, as _if_ the
> repository exists.  That's presumably to prevent disclosing
> information as to whether or not a private repository exists or not.

I agree with you there. Coincidentally speaking, why does a username warrants a prompt from git, is simply beyond me. I mean, that is certainly the more far-fetched reasoning of implementation I've read in a long long time.

Can you git-clone a user? What about the user's settings? What about the remainder its gpg tokens and so forth? In other words, if a user's repo is not found, why even prompting for a username? The latter, that is, the user's repo, is redundant,  when the prompt is clearly asking for a username, and not a repo.


> 
> Once the authentication fails, git will remove the partially created
> repro, so it's really not a problem in practice.
> 
> Cheers,
> 
> 						- Ted

Not true. Preventing the disclosure of information has nothing to do with the issue here. If anything seems clear to me, is that prompting for a username, does indeed disclose usernames, private, public and whatnot from either github or gitlab.

And Technically speaking, yes, I agree with you in that if ^Z main operation is to suspend the process, there's not much further ado. Right?

p.s

I consulted this issue with Thomas Dickey, before sending an email through this interface, and he acknowledges that the operation from git occurs before. Thus confirming some of your statements, and also… that by the time git exits and it's done with,  in whatever dir of my choosing, ^Z couldn't do anything else about it. But I beg to differ. While ^Z may not do anything about it, then git should instead. Or so I think.

running stty -a returns:

swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W;

take care Ted

Carlos


-- 
Seems a computer engineer, a systems analyst, and a programmer were
driving down a mountain when the brakes gave out.  They screamed down the
mountain, gaining speed, but finally managed to grind to a halt, more by
luck than anything else, just inches from a thousand foot drop to jagged
rocks.  They all got out of the car:
        The computer engineer said, "I think I can fix it."
        The systems analyst said, "No, no, I think we should take it
into town and have a specialist look at it."
        The programmer said, "OK, but first I think we should get back
in and see if it does it again."

Re: ctrl-z ignored by git; creates blobs from non-existent repos

[Jeff King]

On Sun, Jan 15, 2023 at 05:47:02PM -0500, Crls wrote:

> > Now, when you type ^Z, the git processes are stopped --- but the
> > objects are created already.
> 
> The directories exist not because ^Z is used, but because by the time
> git-clone prompts for a username, git is already set on what to do
> next. Correct? in other words, the process is shoved down my throat.
> Or the user's throat in this case. Or going by another analogy, it
> certainly sounds as if I meant: «Git, please git-clone such and such
> repo, but let me fix just a typo on the repo name before submitting
> it, pretty please» and then Git replies: «too late for that
> chick-a-doodle» and there it goes. It injects blobs all over (well,
> not all over but on the dir specified).

I don't know what you mean by "blobs" here, since we fail to download
any Git objects at all, let alone blobs. But yes, Git creates the local
repository, and then tries to fetch into it. So the directory is created
before it even contacts the remote server at all, and you will see the
same behavior whether the repository exists or not. And then if an error
occurs, it will rolls back by deleting the newly-created repository.

It _could_ be possible to contact the server first, and only when things
looked successful-ish, to create the local repository. But:

  1. The clone command is simply not written that way, and converting it
     now would be tricky. The clone command's view of the world is that
     it makes a new repository, sets up config, etc, then fetches into
     it.

  2. It would not fix all cases anyway. At some point we have to say
     "this looks close enough to success to create the directory", but
     things can still go wrong after that.

Now if you have a problem with the rollback, there might be a bug there.
But it sounds like you are simply stopping the process and not letting
it finish. It should roll back even if it receives a signal death, but
^Z is stopping the job and putting it in limbo. If it hurts, don't do it
(or use "fg" or "bg" to let it finish).

> > So what's going on is that github.com is not returning a non-existent
> > repo error; it's prompting for a username/password, as _if_ the
> > repository exists.  That's presumably to prevent disclosing
> > information as to whether or not a private repository exists or not.
> 
> I agree with you there. Coincidentally speaking, why does a username
> warrants a prompt from git, is simply beyond me. I mean, that is
> certainly the more far-fetched reasoning of implementation I've read
> in a long long time.
> 
> Can you git-clone a user? What about the user's settings? What about
> the remainder its gpg tokens and so forth? In other words, if a user's
> repo is not found, why even prompting for a username? The latter, that
> is, the user's repo, is redundant,  when the prompt is clearly asking
> for a username, and not a repo.

Huh? GitHub cannot tell you whether you have access to a repository (or,
for privacy reasons of the owner of the hypothetical repository, whether
a repository of that name might exist) unless you authenticate. So it
returns an HTTP 401. Your authentication in this case requires a
username and password. Git asks for the username first, then the
password.

Is there something else you think GitHub should be returning there? Or
something else you think Git should do with an HTTP 401?

-Peff

Re: ctrl-z ignored by git; creates blobs from non-existent repos

[Crls]

On Sun, Jan 15, 2023 at 06:33:09PM -0500, Jeff King wrote:
> On Sun, Jan 15, 2023 at 05:47:02PM -0500, Crls wrote:
> 
> > > Now, when you type ^Z, the git processes are stopped --- but the
> > > objects are created already.
> > 
> > The directories exist not because ^Z is used, but because by the time
> > git-clone prompts for a username, git is already set on what to do
> > next. Correct? in other words, the process is shoved down my throat.
> > Or the user's throat in this case. Or going by another analogy, it
> > certainly sounds as if I meant: «Git, please git-clone such and such
> > repo, but let me fix just a typo on the repo name before submitting
> > it, pretty please» and then Git replies: «too late for that
> > chick-a-doodle» and there it goes. It injects blobs all over (well,
> > not all over but on the dir specified).
> 
> I don't know what you mean by "blobs" here, since we fail to download
> any Git objects at all, let alone blobs. But yes, Git creates the local
> repository, and then tries to fetch into it. So the directory is created
> before it even contacts the remote server at all, and you will see the
> same behavior whether the repository exists or not. And then if an error
> occurs, it will rolls back by deleting the newly-created repository.
> 
> It _could_ be possible to contact the server first, and only when things
> looked successful-ish, to create the local repository. But:
> 
>   1. The clone command is simply not written that way, and converting it
>      now would be tricky. The clone command's view of the world is that
>      it makes a new repository, sets up config, etc, then fetches into
>      it.
> 
>   2. It would not fix all cases anyway. At some point we have to say
>      "this looks close enough to success to create the directory", but
>      things can still go wrong after that.
> 


What I meant by blobs is in the following:

Ending up with a list of directories and subdirectories, is not unusual, is actually very common really with every git-clone while fetching a non-existent repo with a git clone operation either through github or gitlab. First it comes with two (2) git, or one git instance first, then another  and finally one (1) git-remote-helper instance.

ps -a | grep 'git' can confirm it

You'd be able to see 

19159 pts/9    00:00:00 git
19160 pts/9    00:00:00 git
19161 pts/9    00:00:00 git-remote-http <defunct>
19188 pts/9    00:00:00 git
19189 pts/9    00:00:00 git
19190 pts/9    00:00:00 git-remote-http <defunct>


Obviously, its <defunct processs > part was only made possible just right after and invoking a `kill -9 process id`

And if you wanted to find out the leftovers behind through a `du -h <reponame>` by either by git/git-remote-helper (it doesn't really matter at this point which one out of the two may have been the culprit here):

8.0K    /.git/info
4.0K    /.git/objects/info
4.0K    /.git/objects/pack
12K     /.git/objects
64K     /.git/hooks
4.0K    /.git/refs/tags
4.0K    /.git/refs/heads
12K     /.git/refs
4.0K    /.git/branches
116K    /.git
120K    /

These  are the default values. Any curl operation through git with a non-existent repo will get those values.  

Now. We can all proceed trying to sugar coat these leftovers behind an explanation of whatever git-remote-http is actually doing or for that matter, as I said before and without being redundant here, the git-clone processs itself, but it doesn't realyl matter, the end results are the same. Are there leftovers? You can bet on it. It may not happen — and borowwing someone else's phrasing, wording, — «instantaneous», for it's actually quite clever really, to the detriment of an end-user. 

I've read tibbits from gitremote-helpers(7) quote: When Git encounters a URL of the form <transport>://<address>, where <transport> is a protocol that it cannot handle natively, it automatically invokes git remote-<transport> with the full URL as thesecond argument. End quote.  But it's unsettling. The whole issue. The gitremote-helper is not helping, no pun intended. Luckily for these use cases ^C and ^\ seem to be functioning. 


> Now if you have a problem with the rollback, there might be a bug there.
> But it sounds like you are simply stopping the process and not letting
> it finish. It should roll back even if it receives a signal death, but
> ^Z is stopping the job and putting it in limbo. If it hurts, don't do it
> (or use "fg" or "bg" to let it finish).
> 
> > > So what's going on is that github.com is not returning a non-existent
> > > repo error; it's prompting for a username/password, as _if_ the
> > > repository exists.  That's presumably to prevent disclosing
> > > information as to whether or not a private repository exists or not.
> > 
> > I agree with you there. Coincidentally speaking, why does a username
> > warrants a prompt from git, is simply beyond me. I mean, that is
> > certainly the more far-fetched reasoning of implementation I've read
> > in a long long time.
> > 
> > Can you git-clone a user? What about the user's settings? What about
> > the remainder its gpg tokens and so forth? In other words, if a user's
> > repo is not found, why even prompting for a username? The latter, that
> > is, the user's repo, is redundant,  when the prompt is clearly asking
> > for a username, and not a repo.
> 
> Huh? GitHub cannot tell you whether you have access to a repository (or,
> for privacy reasons of the owner of the hypothetical repository, whether
> a repository of that name might exist) unless you authenticate. So it
> returns an HTTP 401. Your authentication in this case requires a
> username and password. Git asks for the username first, then the
> password.
> 
> Is there something else you think GitHub should be returning there? Or
> something else you think Git should do with an HTTP 401?
> 
> -Peff

-- 
Don't hit the keys so hard, it hurts.

Re: ctrl-z ignored by git; creates blobs from non-existent repos

[Junio C Hamano]

Crls <kaploceh@gmail.com> writes:

> ... Coincidentally speaking, why does a username warrants a prompt
> from git, is simply beyond me. I mean, that is certainly the more
> far-fetched reasoning of implementation I've read in a long long
> time.
>
> Can you git-clone a user? What about the user's settings? What
> about the remainder its gpg tokens and so forth? In other words,
> if a user's repo is not found, why even prompting for a username?
> The latter, that is, the user's repo, is redundant, when the
> prompt is clearly asking for a username, and not a repo.

When you "git clone", you'd give a repository path to the server.
If the repository is not open to the general anonymous public, then
the server needs to check who you are (by asking username) and
verify that you are who you claim to be (by asking password).

Here two things you need to pay attention to.

 - A user can be the
   owner of more than one repositories, and

 - a repository can be accessed by users other than its owner.

So even after the repository is known by the server, the server
still needs to ask you who you are.

Imagine that there are many projects hosted at the same site, the
repository path is named after the codename of the project, and the
project codename is need-to-know secret.

If the server side reacted differently between an attempt to clone
existing repositories and missing ones, an attacker can try

	git clone https://site.example.com/projects/$X    

with many X's and observe the behaviour of the server.  If the
server is known to respond with "no such repository" for a missing
one, while responding with "please identify you" for an existing
one, you can easily tell if a word $X is a project codename, that is
supposed to be secret.

> Preventing the disclosure of information has nothing to do with
> the issue here. If anything seems clear to me, is that prompting
> for a username, does indeed disclose usernames, private, public
> and whatnot from either github or gitlab.

When you need to identify yourself to GitHub or GitLab, you'd give
your username and password.  You know that GitHub or GitLab have the
username so it is not secret to them.  Otherwise they wouldn't be
able to even recognise you.

So I am not sure how it "seems clear" that asking for the username
is a problem.  The observed behaviour to ask for the username even
for a missing repository is all about avoiding to disclose one bit
of information: whether a repository exists at the given URL.