[PATCH 00/13] Support for arbitrary schemes in credentials

["brian m. carlson" <sandals@crustytoothpaste.net>]

Right now, HTTP authentication in Git is mostly limited to approaches
that require a username and password or are Kerberos (GSSAPI).  In
addition, we effectively require that libcurl (or, for other software,
such as Git LFS, using the credential helper, that HTTP library) knows
how to implement the authentication scheme.

However, this poses two sets of problems.  First, some sites, such as
Azure DevOps, want to use Bearer authentication, which we don't support.
This is implemented using `http.extraHeader`, which is not a secure way
to store credentials, since our credential helper protocol does not
support this functionality.

In addition, other tools using the credential helper protocol do not
support the variety of authentication mechanisms that Git does.
Specifically, making NTLM function in a useful way on Windows is
nontrivial and requires extensive integration and testing with C code,
and because of this difficulty and the fact that NTLM uses cryptography
known to be insecure since 1995, there is often little interest in
implementing this support outside of libcurl. However, it would be
helpful if people who want to use it can still use it.

This series introduces new functionality to the credential helper
protocol that allows helpers to produce credentials for arbitrary HTTP
authentication schemes using the `authtype` and `credential`[0] fields.
This allows a suitable credential helper to send Bearer credentials or
any other standard or custom authentication scheme.  (It may be able to
be extended to other functionality in the future, such as
git-send-email, to implement custom SASL functionality, and due care has
been taken to make the protocol adequately flexible for that purpose.)

In addition, the protocol is also expanded to include per-helper state
and multi-legged authentication (the former is effectively required for
the latter).  The per-helper state can be useful to help credential
helpers identify where the credential is stored, or any other
information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
NTLM) require two rounds of authentication, the multi-legged
authentication support along with per-helper state allows the helper to
support these authentication methods without Git or other clients having
to be aware of how they work.  (This would also be useful for SASL, as
mentioned above.)

This series introduces a capability mechanism to announce this
functionality, which allows a helper to provide a username and password
on older versions of Git while supporting more advanced functionality on
newer versions.  (This is especially important on Azure DevOps, where
NTLM uses a username and password but Basic or Bearer can use a personal
access token.)  It is also designed such that extremely simple
credential helpers, such as the shell one-liner in the Git FAQ that
reads from the environment, don't accidentally claim to support
functionality they don't offer.

In addition, there is documentation for the expanded protocol, although
none of the built-in helpers have been updated (that will be a future
series for those for which it's possible).

My personal interest here is getting credentials out of config files
with `http.extraHeader` (which a future series will produce a warning
for) and also allowing Git LFS to support Digest and NTLM with a
suitable credential helper.  Git LFS used to support NTLM using custom
code (because the Go standard library does not), but it was found to be
broken in lots of ways on Windows, and nobody with a Windows system
wanted to fix it or support it, so we removed it.  However, there are
still some people who do want to use it, so allowing them to use a
custom credential helper they maintain themselves seems like the best
way forward.  Despite the advantages of this series for Azure DevOps, I
have no personal or professional stake in their product; my only
interest is the general one in whether their users can securely store
credentials.  I believe the changes here are of general advantage to the
Git userbase in a variety of ways such that the goal of this series
should be uncontroversial.

Feedback on any portion of this series is of course welcome.

[0] A name different from `password` was explicitly chosen to avoid
confusion from less capable protocol helpers so that they don't
accidentally send invalid data.  This does have the downside that
credential helpers must learn a new field to not log, but that should be
generally easy to fix in most cases.

brian m. carlson (13):
  credential: add an authtype field
  remote-curl: reset headers on new request
  http: use new headers for each object request
  credential: add a field for pre-encoded credentials
  credential: gate new fields on capability
  docs: indicate new credential protocol fields
  http: add support for authtype and credential
  credential: add an argument to keep state
  credential: enable state capability
  docs: set a limit on credential line length
  t5563: refactor for multi-stage authentication
  strvec: implement swapping two strvecs
  credential: add support for multistage credential rounds

 Documentation/git-credential.txt   |  59 +++++-
 builtin/credential-cache--daemon.c |   2 +-
 builtin/credential-store.c         |   2 +-
 builtin/credential.c               |   7 +-
 credential.c                       | 114 ++++++++++-
 credential.h                       |  69 ++++++-
 http.c                             | 128 +++++++-----
 http.h                             |   5 +
 imap-send.c                        |   2 +-
 remote-curl.c                      |  14 +-
 strvec.c                           |   7 +
 strvec.h                           |   5 +
 t/lib-httpd/nph-custom-auth.sh     |  17 +-
 t/t0300-credentials.sh             | 136 ++++++++++++-
 t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
 15 files changed, 760 insertions(+), 115 deletions(-)