* Fwd: Git credentials not working [not found] <a1ceb967-6020-6074-f504-c684242c79ab@gmail.com> @ 2018-10-03 14:06 ` Dimitri Kopriwa 2018-10-03 16:03 ` Christian Couder 2018-10-03 17:11 ` Fwd: " Jeff King 0 siblings, 2 replies; 10+ messages in thread From: Dimitri Kopriwa @ 2018-10-03 14:06 UTC (permalink / raw) To: git-security, git Dear Git list, I have tried to used git credentials within Gitlab-CI runners. I have 4 instance of GitLab and discovered a weird bug with Git credentials when use within a CI process. Please note before all that the time spend allowed me multiple time to check that my credentials are valid for the repository. And calling git fetch --tags with the full remote url that include the credentials always succeeded. Tested with Git 2.11, 2.19 Git credentials in ~/.git-credentials and ~/.config/git/credentials are being removed by git upon reading. This happen randomly accross my CI runner, and change that make them work on not related. { Error: Command failed: git fetch --tags https://git.example.com/example/some-project.git 18:25:52.554903 git.c:415 trace: built-in: git fetch --tags https://git.example.com/example/some-project.git 18:25:52.555234 run-command.c:637 trace: run_command: GIT_DIR=.git git-remote-https https://git.example.com/example/some-project.git https://git.example.com/example/some-project.git 18:25:52.692741 run-command.c:637 trace: run_command: 'git credential-store get' 18:25:52.697314 git.c:659 trace: exec: git-credential-store get 18:25:52.697372 run-command.c:637 trace: run_command: git-credential-store get 18:25:52.936024 run-command.c:637 trace: run_command: 'git credential-store erase' 18:25:52.940307 git.c:659 trace: exec: git-credential-store erase 18:25:52.940365 run-command.c:637 trace: run_command: git-credential-store erase remote: HTTP Basic: Access denied fatal: Authentication failed for 'https://git.example.com/example/some-project.git/' See the full question here: https://stackoverflow.com/questions/52614467/why-does-git-credential-store-call-git-credential-erase-and-make-my-credential-f Can you please help me found why is git credential-store erase called ? Best regards, ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git credentials not working 2018-10-03 14:06 ` Fwd: Git credentials not working Dimitri Kopriwa @ 2018-10-03 16:03 ` Christian Couder 2018-10-03 17:29 ` Dimitri Kopriwa 2018-10-03 17:11 ` Fwd: " Jeff King 1 sibling, 1 reply; 10+ messages in thread From: Christian Couder @ 2018-10-03 16:03 UTC (permalink / raw) To: d.kopriwa; +Cc: git (removing git-security from CC) On Wed, Oct 3, 2018 at 4:09 PM Dimitri Kopriwa <d.kopriwa@gmail.com> wrote: > Git credentials in ~/.git-credentials and ~/.config/git/credentials are > being removed by git upon reading. https://git-scm.com/docs/git-credential says: "If the action is reject, git-credential will send the description to any configured credential helpers, which may erase any stored credential matching the description." So maybe this is expected. Another possibility is that your .gitlab-ci.yml might launch scripts writing into those files, like the before_script.sh script that is described on: https://stackoverflow.com/questions/50553049/is-it-possible-to-do-a-git-push-within-a-gitlab-ci-without-ssh Could you also check which credential helper and which options are used? For example with commands like: $ git config -l --show-origin | grep -i cred $ git config -l --show-origin | grep -i http $ git config -l --show-origin | grep -i askpass $ env | grep -i askpass ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git credentials not working 2018-10-03 16:03 ` Christian Couder @ 2018-10-03 17:29 ` Dimitri Kopriwa 0 siblings, 0 replies; 10+ messages in thread From: Dimitri Kopriwa @ 2018-10-03 17:29 UTC (permalink / raw) To: Christian Couder; +Cc: git On 10/3/18 11:03 PM, Christian Couder wrote: > (removing git-security from CC) > > On Wed, Oct 3, 2018 at 4:09 PM Dimitri Kopriwa <d.kopriwa@gmail.com> wrote: > >> Git credentials in ~/.git-credentials and ~/.config/git/credentials are >> being removed by git upon reading. > https://git-scm.com/docs/git-credential says: > > "If the action is reject, git-credential will send the description to > any configured credential helpers, which may erase any stored > credential matching the description." > > So maybe this is expected. I am using this script to create my credential file, how am I supposed to do in a non tty environment? Is there a prefered way? > > Another possibility is that your .gitlab-ci.yml might launch scripts > writing into those files, like the before_script.sh script that is > described on: > > https://stackoverflow.com/questions/50553049/is-it-possible-to-do-a-git-push-within-a-gitlab-ci-without-ssh > > Could you also check which credential helper and which options are > used? For example with commands like: > > $ git config -l --show-origin | grep -i cred > $ git config -l --show-origin | grep -i http > $ git config -l --show-origin | grep -i askpass > $ env | grep -i askpass * branch HEAD -> FETCH_HEAD 17:15:36.175966 run-command.c:637 trace: run_command: git gc --auto 17:15:36.177688 git.c:415 trace: built-in: git gc --auto ^[[32;1m$ git config -l --show-origin | grep -i cred^[[0;m 17:15:36.180191 git.c:415 trace: built-in: git config -l --show-origin file:/root/.gitconfig credential.helper=store file:.git/config credential.helper=store ^[[32;1m$ git config -l --show-origin | grep -i http^[[0;m 17:15:36.182768 git.c:415 trace: built-in: git config -l --show-origin file:.git/config remote.origin.url=https://git.example.com/example/sample-project.git ^[[32;1m$ git config -l --show-origin | grep -i askpass || echo nothing to do^[[0;m 17:15:36.185306 git.c:415 trace: built-in: git config -l --show-origin nothing to do ^[[32;1m$ env | grep -i askpass || echo nothing to do^[[0;m nothing to do ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Fwd: Git credentials not working 2018-10-03 14:06 ` Fwd: Git credentials not working Dimitri Kopriwa 2018-10-03 16:03 ` Christian Couder @ 2018-10-03 17:11 ` Jeff King 2018-10-03 18:12 ` Dimitri Kopriwa 1 sibling, 1 reply; 10+ messages in thread From: Jeff King @ 2018-10-03 17:11 UTC (permalink / raw) To: Dimitri Kopriwa; +Cc: git-security, git On Wed, Oct 03, 2018 at 09:06:38PM +0700, Dimitri Kopriwa wrote: > 18:25:52.940307 git.c:659 trace: exec: git-credential-store erase > 18:25:52.940365 run-command.c:637 trace: run_command: git-credential-store erase > remote: HTTP Basic: Access denied > fatal: Authentication failed for > 'https://git.example.com/example/some-project.git/' > [...] > > Can you please help me found why is git credential-store erase called ? This is expected. We tried to use a credential that was rejected by the server, so we told all of the helpers it was invalid. You can try running GIT_TRACE_CURL=1 to see the HTTP conversation. There will be an HTTP 401 with the authentication failure, though it may not tell you anything more useful than that. git-credential-store is meant to be used interactively, to insert and erase credentials as they're grabbed from the terminal. It sounds more like you want to just have a stored credential that you try to use. You could do that with a custom helper. E.g., something like this in your ~/.gitconfig: [credential "https://example.com"] helper = "!f() { test $1 = get && echo password=$(cat /path/with/password); }; f" -Peff ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Fwd: Git credentials not working 2018-10-03 17:11 ` Fwd: " Jeff King @ 2018-10-03 18:12 ` Dimitri Kopriwa 2018-10-03 18:24 ` Jeff King 0 siblings, 1 reply; 10+ messages in thread From: Dimitri Kopriwa @ 2018-10-03 18:12 UTC (permalink / raw) To: Jeff King; +Cc: git Thanks for your reply. I have activated GIT_TRACE_CURL=1 and I can see that the request is failing 401. I can't see which token is used and using what header ? The log say: 17:50:26.414654 http.c:657 => Send header: Authorization: Basic <redacted> I have retested the token locally and it work when used in the url or using `Private-Token: <token>` as stated in the Gitlab documentation https://docs.gitlab.com/ee/api/README.html#personal-access-tokens Peff, what would be the appropriate way to input my git credential in a 100% success way in a CI? Is this good: git credential approve <<EOF protocol=https host=example.com username=bob password=secr3t OEF ? I would use the custom helper after I can understand how to properly use the git credential store in a CI environment. The fact that I am using a generated file is simply because this is what the documentation told me to do. I did not found anywhere in the doc how I should create that file in a non tty terminal. Thanks again for your help. On 10/4/18 12:11 AM, Jeff King wrote: > On Wed, Oct 03, 2018 at 09:06:38PM +0700, Dimitri Kopriwa wrote: > >> 18:25:52.940307 git.c:659 trace: exec: git-credential-store erase >> 18:25:52.940365 run-command.c:637 trace: run_command: git-credential-store erase >> remote: HTTP Basic: Access denied >> fatal: Authentication failed for >> 'https://git.example.com/example/some-project.git/' >> [...] >> >> Can you please help me found why is git credential-store erase called ? > This is expected. We tried to use a credential that was rejected by the > server, so we told all of the helpers it was invalid. You can try > running GIT_TRACE_CURL=1 to see the HTTP conversation. There will be an > HTTP 401 with the authentication failure, though it may not tell you > anything more useful than that. > > git-credential-store is meant to be used interactively, to insert and > erase credentials as they're grabbed from the terminal. > > It sounds more like you want to just have a stored credential that you > try to use. You could do that with a custom helper. E.g., something like > this in your ~/.gitconfig: > > [credential "https://example.com"] > helper = "!f() { test $1 = get && echo password=$(cat /path/with/password); }; f" > > -Peff ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Fwd: Git credentials not working 2018-10-03 18:12 ` Dimitri Kopriwa @ 2018-10-03 18:24 ` Jeff King 2018-10-03 19:34 ` Dimitri Kopriwa 0 siblings, 1 reply; 10+ messages in thread From: Jeff King @ 2018-10-03 18:24 UTC (permalink / raw) To: Dimitri Kopriwa; +Cc: git On Thu, Oct 04, 2018 at 01:12:11AM +0700, Dimitri Kopriwa wrote: > Thanks for your reply. I have activated GIT_TRACE_CURL=1 and I can see that > the request is failing 401. > > I can't see which token is used and using what header ? > > The log say: > > 17:50:26.414654 http.c:657 => Send header: Authorization: Basic <redacted> Yeah, we redact the auth information so people don't accidentally share it publicly. If you use the older GIT_CURL_VERBOSE=1, it will include the credential (I think it may be base64 encoded, though, so you'll have to decipher it). > I have retested the token locally and it work when used in the url or using > `Private-Token: <token>` as stated in the Gitlab documentation > https://docs.gitlab.com/ee/api/README.html#personal-access-tokens I don't think Git will ever send your token in either of those ways. It will always some as an Authorization header. > Peff, what would be the appropriate way to input my git credential in a 100% > success way in a CI? I don't know the details of what GitLab would want, but... > Is this good: > > git credential approve <<EOF > protocol=https > host=example.com > username=bob > password=secr3t > OEF Yes, that would work to preload a token into any configured helpers. -Peff ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Fwd: Git credentials not working 2018-10-03 18:24 ` Jeff King @ 2018-10-03 19:34 ` Dimitri Kopriwa 2018-10-03 20:03 ` Bryan Turner 2018-10-03 20:43 ` Jeff King 0 siblings, 2 replies; 10+ messages in thread From: Dimitri Kopriwa @ 2018-10-03 19:34 UTC (permalink / raw) To: Jeff King; +Cc: git I have replaced the way I fill the git credentials store, I have verify ~/.git-credentials and information are there, the ~/.gitconfig look fine too. I still have 401 error when reading from that file. This is the paste log : https://paste.gnome.org/pmntlkdw0 Now that I use git approve, I dont think that I need a custom helper. Any idea why I still can't log in using git-credential? Thanks in advance, On 10/4/18 1:24 AM, Jeff King wrote: > On Thu, Oct 04, 2018 at 01:12:11AM +0700, Dimitri Kopriwa wrote: > >> Thanks for your reply. I have activated GIT_TRACE_CURL=1 and I can see that >> the request is failing 401. >> >> I can't see which token is used and using what header ? >> >> The log say: >> >> 17:50:26.414654 http.c:657 => Send header: Authorization: Basic <redacted> > Yeah, we redact the auth information so people don't accidentally share > it publicly. If you use the older GIT_CURL_VERBOSE=1, it will include > the credential (I think it may be base64 encoded, though, so you'll have > to decipher it). > >> I have retested the token locally and it work when used in the url or using >> `Private-Token: <token>` as stated in the Gitlab documentation >> https://docs.gitlab.com/ee/api/README.html#personal-access-tokens > I don't think Git will ever send your token in either of those ways. It > will always some as an Authorization header. > >> Peff, what would be the appropriate way to input my git credential in a 100% >> success way in a CI? > I don't know the details of what GitLab would want, but... > >> Is this good: >> >> git credential approve <<EOF >> protocol=https >> host=example.com >> username=bob >> password=secr3t >> OEF > Yes, that would work to preload a token into any configured helpers. > > -Peff ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Fwd: Git credentials not working 2018-10-03 19:34 ` Dimitri Kopriwa @ 2018-10-03 20:03 ` Bryan Turner 2018-10-03 20:43 ` Jeff King 1 sibling, 0 replies; 10+ messages in thread From: Bryan Turner @ 2018-10-03 20:03 UTC (permalink / raw) To: d.kopriwa; +Cc: Jeff King, Git Users On Wed, Oct 3, 2018 at 12:34 PM Dimitri Kopriwa <d.kopriwa@gmail.com> wrote: > > I have replaced the way I fill the git credentials store, I have verify > ~/.git-credentials and information are there, the ~/.gitconfig look fine > too. > > I still have 401 error when reading from that file. > > This is the paste log : https://paste.gnome.org/pmntlkdw0 > > Now that I use git approve, I dont think that I need a custom helper. > > Any idea why I still can't log in using git-credential? I'm pretty sure Peff touched on this in his reply. When it works, you're either sending a "Private-Token" header or including it in the URL, but, as Peff said, Git will never do either of those things. It sends an "Authorization" header, and, based on their documentation, it doesn't appear Gitlab accepts access tokens in that header. It looks like you're either going to need to include it in the URL (like what happens earlier in the posted trace), or adjust your git config with a "http.extraHeader" set to "Private-Token: <token>" to include the "Private-Token" header (or you could pass it on the command line, like `git -c http.extraHeader="Private-Token: <token>" clone ...`. Hope this helps! Bryan > > Thanks in advance, > > On 10/4/18 1:24 AM, Jeff King wrote: > > On Thu, Oct 04, 2018 at 01:12:11AM +0700, Dimitri Kopriwa wrote: > > > >> Thanks for your reply. I have activated GIT_TRACE_CURL=1 and I can see that > >> the request is failing 401. > >> > >> I can't see which token is used and using what header ? > >> > >> The log say: > >> > >> 17:50:26.414654 http.c:657 => Send header: Authorization: Basic <redacted> > > Yeah, we redact the auth information so people don't accidentally share > > it publicly. If you use the older GIT_CURL_VERBOSE=1, it will include > > the credential (I think it may be base64 encoded, though, so you'll have > > to decipher it). > > > >> I have retested the token locally and it work when used in the url or using > >> `Private-Token: <token>` as stated in the Gitlab documentation > >> https://docs.gitlab.com/ee/api/README.html#personal-access-tokens > > I don't think Git will ever send your token in either of those ways. It > > will always some as an Authorization header. > > > >> Peff, what would be the appropriate way to input my git credential in a 100% > >> success way in a CI? > > I don't know the details of what GitLab would want, but... > > > >> Is this good: > >> > >> git credential approve <<EOF > >> protocol=https > >> host=example.com > >> username=bob > >> password=secr3t > >> OEF > > Yes, that would work to preload a token into any configured helpers. > > > > -Peff ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Fwd: Git credentials not working 2018-10-03 19:34 ` Dimitri Kopriwa 2018-10-03 20:03 ` Bryan Turner @ 2018-10-03 20:43 ` Jeff King 2018-10-03 21:35 ` Dimitri Kopriwa 1 sibling, 1 reply; 10+ messages in thread From: Jeff King @ 2018-10-03 20:43 UTC (permalink / raw) To: Dimitri Kopriwa; +Cc: git On Thu, Oct 04, 2018 at 02:34:17AM +0700, Dimitri Kopriwa wrote: > I have replaced the way I fill the git credentials store, I have verify > ~/.git-credentials and information are there, the ~/.gitconfig look fine > too. > > I still have 401 error when reading from that file. > > This is the paste log : https://paste.gnome.org/pmntlkdw0 > > Now that I use git approve, I dont think that I need a custom helper. > > Any idea why I still can't log in using git-credential? Looking at your pastebin, it looks like the server sometimes takes it and sometimes not. E.g., piping the log through: egrep '(Send|Recv) header:' | perl -lpe 's/^.*?(=>|<=) //' I see: Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1 Send header: User-Agent: git/2.19.0 ... Recv header: HTTP/1.1 401 Unauthorized Recv header: WWW-Authenticate: Basic realm="GitLab" ... Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1 Send header: Authorization: Basic <redacted> Send header: User-Agent: git/2.19.0 ... Recv header: HTTP/1.1 200 OK So that works. But then later we get: Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1 Send header: User-Agent: git/2.19.0 ... Recv header: HTTP/1.1 401 Unauthorized Recv header: WWW-Authenticate: Basic realm="GitLab" ... Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1 Send header: Authorization: Basic <redacted> Send header: User-Agent: git/2.19.0 ... Recv header: HTTP/1.1 401 Unauthorized And then that causes credential-store to delete the non-working entry, after which all of them must fail (because you have no working credential, and presumably no terminal to prompt the user). I have no idea why the same request would sometimes be allowed and sometimes not. It's possible the <redacted> data is different in those two times, but I don't know why that would be. It's also possible you're hitting different load-balancing servers that behave differently. -Peff ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Fwd: Git credentials not working 2018-10-03 20:43 ` Jeff King @ 2018-10-03 21:35 ` Dimitri Kopriwa 0 siblings, 0 replies; 10+ messages in thread From: Dimitri Kopriwa @ 2018-10-03 21:35 UTC (permalink / raw) To: Jeff King; +Cc: git Thanks everyone. All your answers helped. I found out that the issue was not related to git. I am using semantic-release to perform a release, apparently git-credentials is not working with semantic-release. I did also setup the double authentication and every fix applied on git-credentials were simply useless. Read more here : https://github.com/semantic-release/semantic-release/issues/941#issuecomment-426691824 Thanks a lot for your help and git is the best software ever made thanks! Dimitri Kopriwa On 10/4/18 3:43 AM, Jeff King wrote: > On Thu, Oct 04, 2018 at 02:34:17AM +0700, Dimitri Kopriwa wrote: > >> I have replaced the way I fill the git credentials store, I have verify >> ~/.git-credentials and information are there, the ~/.gitconfig look fine >> too. >> >> I still have 401 error when reading from that file. >> >> This is the paste log : https://paste.gnome.org/pmntlkdw0 >> >> Now that I use git approve, I dont think that I need a custom helper. >> >> Any idea why I still can't log in using git-credential? > Looking at your pastebin, it looks like the server sometimes takes it > and sometimes not. E.g., piping the log through: > > egrep '(Send|Recv) header:' | > perl -lpe 's/^.*?(=>|<=) //' > > I see: > > Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1 > Send header: User-Agent: git/2.19.0 > ... > Recv header: HTTP/1.1 401 Unauthorized > Recv header: WWW-Authenticate: Basic realm="GitLab" > ... > Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1 > Send header: Authorization: Basic <redacted> > Send header: User-Agent: git/2.19.0 > ... > Recv header: HTTP/1.1 200 OK > > So that works. But then later we get: > > Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1 > Send header: User-Agent: git/2.19.0 > ... > Recv header: HTTP/1.1 401 Unauthorized > Recv header: WWW-Authenticate: Basic realm="GitLab" > ... > Send header: GET /example-keys/sample-project.git/info/refs?service=git-upload-pack HTTP/1.1 > Send header: Authorization: Basic <redacted> > Send header: User-Agent: git/2.19.0 > ... > Recv header: HTTP/1.1 401 Unauthorized > > And then that causes credential-store to delete the non-working entry, > after which all of them must fail (because you have no working > credential, and presumably no terminal to prompt the user). > > I have no idea why the same request would sometimes be allowed and > sometimes not. It's possible the <redacted> data is different in those > two times, but I don't know why that would be. It's also possible you're > hitting different load-balancing servers that behave differently. > > -Peff ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-10-03 21:35 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <a1ceb967-6020-6074-f504-c684242c79ab@gmail.com> 2018-10-03 14:06 ` Fwd: Git credentials not working Dimitri Kopriwa 2018-10-03 16:03 ` Christian Couder 2018-10-03 17:29 ` Dimitri Kopriwa 2018-10-03 17:11 ` Fwd: " Jeff King 2018-10-03 18:12 ` Dimitri Kopriwa 2018-10-03 18:24 ` Jeff King 2018-10-03 19:34 ` Dimitri Kopriwa 2018-10-03 20:03 ` Bryan Turner 2018-10-03 20:43 ` Jeff King 2018-10-03 21:35 ` Dimitri Kopriwa
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).