[PATCH] http: enable keepalive on TCP sockets

[PATCH] http: enable keepalive on TCP sockets

[Eric Wong]

This is a follow up to commit e47a8583a20256851e7fc882233e3bd5bf33dc6e
(enable SO_KEEPALIVE for connected TCP sockets).

Sockets may never receive notification of some link errors,
causing "git fetch" or similar processes to hang forever.
Enabling keepalive messages allows hung processes to error out
after a few minutes/hours depending on the keepalive settings of
the system.

I noticed this problem with some non-interactive cronjobs getting
hung when talking to HTTP servers.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 http.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/http.c b/http.c
index f3e1439..5834c9b 100644
--- a/http.c
+++ b/http.c
@@ -260,6 +260,24 @@ static int has_cert_password(void)
 	return 1;
 }
 
+/* curl 7.25.0 has CURLOPT_TCP_KEEPALIVE, too, but we support older curl */
+static int sockopt_callback(void *client, curl_socket_t fd, curlsocktype type)
+{
+	int ka = 1;
+	int rc;
+	socklen_t len = (socklen_t)sizeof(ka);
+
+	if (type != CURLSOCKTYPE_IPCXN)
+		return 0;
+
+	rc = setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, (void *)&ka, len);
+	if (rc < 0)
+		warning("unable to set SO_KEEPALIVE on socket %s",
+			strerror(errno));
+
+	return 0; /* CURL_SOCKOPT_OK only exists since curl 7.21.5 */
+}
+
 static CURL *get_curl_handle(void)
 {
 	CURL *result = curl_easy_init();
@@ -332,6 +350,10 @@ static CURL *get_curl_handle(void)
 		curl_easy_setopt(result, CURLOPT_PROXYAUTH, CURLAUTH_ANY);
 	}
 
+#if LIBCURL_VERSION_NUM >= 0x071000
+	curl_easy_setopt(result, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);
+#endif
+
 	return result;
 }
 
-- 
Eric Wong

Re: [PATCH] http: enable keepalive on TCP sockets

[Daniel Stenberg]

On Sat, 12 Oct 2013, Eric Wong wrote:

> This is a follow up to commit e47a8583a20256851e7fc882233e3bd5bf33dc6e 
> (enable SO_KEEPALIVE for connected TCP sockets).

Just keep in mind that TCP keep-alive is enabled in awkwardly many different 
ways on different systems and this patch only supports one of them. Feel free 
to take inspiration from libcurl's source code for doing this. See:

   https://github.com/bagder/curl/blob/master/lib/connect.c#L108

-- 

  / daniel.haxx.se

Re: [PATCH] http: enable keepalive on TCP sockets

[Eric Wong]

Daniel Stenberg <daniel@haxx.se> wrote:
> On Sat, 12 Oct 2013, Eric Wong wrote:
> 
> >This is a follow up to commit
> >e47a8583a20256851e7fc882233e3bd5bf33dc6e (enable SO_KEEPALIVE for
> >connected TCP sockets).
> 
> Just keep in mind that TCP keep-alive is enabled in awkwardly many
> different ways on different systems and this patch only supports one
> of them. Feel free to take inspiration from libcurl's source code
> for doing this. See:
> 
>   https://github.com/bagder/curl/blob/master/lib/connect.c#L108

Thanks.  I think the Linux-specific TCP_KEEP* knobs are overkill for git.
(since this is mainly for non-interactive users, I went at least a day
 before realizing the process was stuck on my machine).
I cannot comment on the knobs for other OSes.

Re: [PATCH] http: enable keepalive on TCP sockets

[Jeff King]

On Mon, Oct 14, 2013 at 05:27:39AM +0000, Eric Wong wrote:

> Daniel Stenberg <daniel@haxx.se> wrote:
> > On Sat, 12 Oct 2013, Eric Wong wrote:
> > 
> > >This is a follow up to commit
> > >e47a8583a20256851e7fc882233e3bd5bf33dc6e (enable SO_KEEPALIVE for
> > >connected TCP sockets).
> > 
> > Just keep in mind that TCP keep-alive is enabled in awkwardly many
> > different ways on different systems and this patch only supports one
> > of them. Feel free to take inspiration from libcurl's source code
> > for doing this. See:
> > 
> >   https://github.com/bagder/curl/blob/master/lib/connect.c#L108
> 
> Thanks.  I think the Linux-specific TCP_KEEP* knobs are overkill for git.
> (since this is mainly for non-interactive users, I went at least a day
>  before realizing the process was stuck on my machine).
> I cannot comment on the knobs for other OSes.

I don't think we should get into having a big compatibility layer that
just reproduces what is in curl.

But is there any reason not to use CURLOPT_TCP_KEEPALIVE when it is
available, falling back to CURLOPT_SOCKOPTFUNCTION, and then finally to
nothing? That lets people on modern curl benefit from curl's more
portable code, without punishing people on older versions.

I.e., something like:

---
diff --git a/http.c b/http.c
index 5834c9b..e221efb 100644
--- a/http.c
+++ b/http.c
@@ -254,36 +254,54 @@ static int sockopt_callback(void *client, curl_socket_t fd, curlsocktype type)
 	if (!cert_auth.password) {
 		cert_auth.protocol = xstrdup("cert");
 		cert_auth.username = xstrdup("");
 		cert_auth.path = xstrdup(ssl_cert);
 		credential_fill(&cert_auth);
 	}
 	return 1;
 }
 
-/* curl 7.25.0 has CURLOPT_TCP_KEEPALIVE, too, but we support older curl */
+#if LIBCURL_VERSION_NUM >= 0x071900
+static void set_curl_keepalive(CURL *c)
+{
+	curl_easy_setopt(c, CURLOPT_TCP_KEEPALIVE, 1);
+}
+
+#elif LIBCURL_VERSION_NUM >= 0x071000
 static int sockopt_callback(void *client, curl_socket_t fd, curlsocktype type)
 {
 	int ka = 1;
 	int rc;
 	socklen_t len = (socklen_t)sizeof(ka);
 
 	if (type != CURLSOCKTYPE_IPCXN)
 		return 0;
 
 	rc = setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, (void *)&ka, len);
 	if (rc < 0)
 		warning("unable to set SO_KEEPALIVE on socket %s",
 			strerror(errno));
 
 	return 0; /* CURL_SOCKOPT_OK only exists since curl 7.21.5 */
 }
 
+static void set_curl_keepalive(CURL *c)
+{
+	curl_easy_setopt(c, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);
+}
+
+#else
+static void set_curl_keepalive(CURL *c)
+{
+	/* not supported on older curl versions */
+}
+#endif
+
 static CURL *get_curl_handle(void)
 {
 	CURL *result = curl_easy_init();
 
 	if (!curl_ssl_verify) {
 		curl_easy_setopt(result, CURLOPT_SSL_VERIFYPEER, 0);
 		curl_easy_setopt(result, CURLOPT_SSL_VERIFYHOST, 0);
 	} else {
 		/* Verify authenticity of the peer's certificate */
@@ -344,21 +362,19 @@ static CURL *get_curl_handle(void)
 	if (curl_ssl_try)
 		curl_easy_setopt(result, CURLOPT_USE_SSL, CURLUSESSL_TRY);
 #endif
 
 	if (curl_http_proxy) {
 		curl_easy_setopt(result, CURLOPT_PROXY, curl_http_proxy);
 		curl_easy_setopt(result, CURLOPT_PROXYAUTH, CURLAUTH_ANY);
 	}
 
-#if LIBCURL_VERSION_NUM >= 0x071000
-	curl_easy_setopt(result, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);
-#endif
+	set_curl_keepalive(result);
 
 	return result;
 }
 
 static void set_from_env(const char **var, const char *envname)
 {
 	const char *val = getenv(envname);
 	if (val)
 		*var = val;

Re: [PATCH] http: enable keepalive on TCP sockets

[Eric Wong]

Jeff King <peff@peff.net> wrote:
> On Mon, Oct 14, 2013 at 05:27:39AM +0000, Eric Wong wrote:
> > Daniel Stenberg <daniel@haxx.se> wrote:
> > > On Sat, 12 Oct 2013, Eric Wong wrote:
> > > 
> > > >This is a follow up to commit
> > > >e47a8583a20256851e7fc882233e3bd5bf33dc6e (enable SO_KEEPALIVE for
> > > >connected TCP sockets).
> > > 
> > > Just keep in mind that TCP keep-alive is enabled in awkwardly many
> > > different ways on different systems and this patch only supports one
> > > of them. Feel free to take inspiration from libcurl's source code
> > > for doing this. See:
> > > 
> > >   https://github.com/bagder/curl/blob/master/lib/connect.c#L108
> > 
> > Thanks.  I think the Linux-specific TCP_KEEP* knobs are overkill for git.
> > (since this is mainly for non-interactive users, I went at least a day
> >  before realizing the process was stuck on my machine).
> > I cannot comment on the knobs for other OSes.
> 
> I don't think we should get into having a big compatibility layer that
> just reproduces what is in curl.
> 
> But is there any reason not to use CURLOPT_TCP_KEEPALIVE when it is
> available, falling back to CURLOPT_SOCKOPTFUNCTION, and then finally to
> nothing? That lets people on modern curl benefit from curl's more
> portable code, without punishing people on older versions.

I wanted it to work as older curl first (since I noticed this
on an old server).  But your patch on top of mine looks reasonable,
thanks.

[PATCH] http: use curl's tcp keepalive if available

[Jeff King]

On Mon, Oct 14, 2013 at 11:38:39PM +0000, Eric Wong wrote:

> I wanted it to work as older curl first (since I noticed this
> on an old server).  But your patch on top of mine looks reasonable,
> thanks.

Makes sense. Here it is with a real commit message (on top of the
ew/keepalive topic).

-- >8 --
Subject: http: use curl's tcp keepalive if available

Commit a15d069 taught git to use curl's SOCKOPTFUNCTION hook
to turn on TCP keepalives. However, modern versions of curl
have a TCP_KEEPALIVE option, which can do this for us. As an
added bonus, the curl code knows how to turn on keepalive
for a much wider variety of platforms. The only downside to
using this option is that not everybody has a new enough curl.
Let's split our keepalive options into three conditionals:

  1. With curl 7.25.0 and newer, we rely on curl to do it
     right.

  2. With older curl that still knows SOCKOPTFUNCTION, we
     use the code from a15d069.

  3. Otherwise, we are out of luck, and the call is a no-op.

Signed-off-by: Jeff King <peff@peff.net>
---
Given the #ifdefs in curl's keepalive code, I suspect we may see build
problems for people in case 2 on some systems, with or without my patch.
I think this patch is a strict improvement, though; if they have a new
enough curl, they will not even look at the case 2 code. And if they do
not, our previous options were:

  a. Add platform-specific code for them.
  
  b. Tell them they are out of luck, and add an #ifdef to push them into
     case 3.

Now we have an extra option:

  c. Tell them to upgrade curl. :)

 http.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)

diff --git a/http.c b/http.c
index a2c1819..6359526 100644
--- a/http.c
+++ b/http.c
@@ -233,7 +233,13 @@ static int has_cert_password(void)
 		return 0;
 }
 
-/* curl 7.25.0 has CURLOPT_TCP_KEEPALIVE, too, but we support older curl */
+#if LIBCURL_VERSION_NUM >= 0x071900
+static void set_curl_keepalive(CURL *c)
+{
+	curl_easy_setopt(c, CURLOPT_TCP_KEEPALIVE, 1);
+}
+
+#elif LIBCURL_VERSION_NUM >= 0x071000
 static int sockopt_callback(void *client, curl_socket_t fd, curlsocktype type)
 {
 	int ka = 1;
@@ -251,6 +257,18 @@ static int sockopt_callback(void *client, curl_socket_t fd, curlsocktype type)
 	return 0; /* CURL_SOCKOPT_OK only exists since curl 7.21.5 */
 }
 
+static void set_curl_keepalive(CURL *c)
+{
+	curl_easy_setopt(c, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);
+}
+
+#else
+static void set_curl_keepalive(CURL *c)
+{
+	/* not supported on older curl versions */
+}
+#endif
+
 static CURL *get_curl_handle(void)
 {
 	CURL *result = curl_easy_init();
@@ -316,9 +334,7 @@ static CURL *get_curl_handle(void)
 	if (curl_http_proxy)
 		curl_easy_setopt(result, CURLOPT_PROXY, curl_http_proxy);
 
-#if LIBCURL_VERSION_NUM >= 0x071000
-	curl_easy_setopt(result, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);
-#endif
+	set_curl_keepalive(result);
 
 	return result;
 }
-- 
1.8.4.1.4.gf327177

Re: [PATCH] http: use curl's tcp keepalive if available

[Eric Wong]

Jeff King <peff@peff.net> wrote:
> On Mon, Oct 14, 2013 at 11:38:39PM +0000, Eric Wong wrote:
> 
> > I wanted it to work as older curl first (since I noticed this
> > on an old server).  But your patch on top of mine looks reasonable,
> > thanks.
> 
> Makes sense. Here it is with a real commit message (on top of the
> ew/keepalive topic).
> 
> -- >8 --
> Subject: http: use curl's tcp keepalive if available
> 
> Commit a15d069 taught git to use curl's SOCKOPTFUNCTION hook
> to turn on TCP keepalives. However, modern versions of curl
> have a TCP_KEEPALIVE option, which can do this for us. As an
> added bonus, the curl code knows how to turn on keepalive
> for a much wider variety of platforms. The only downside to
> using this option is that not everybody has a new enough curl.
> Let's split our keepalive options into three conditionals:
> 
>   1. With curl 7.25.0 and newer, we rely on curl to do it
>      right.
> 
>   2. With older curl that still knows SOCKOPTFUNCTION, we
>      use the code from a15d069.
> 
>   3. Otherwise, we are out of luck, and the call is a no-op.
> 
> Signed-off-by: Jeff King <peff@peff.net>

Thanks.
Tested-by: Eric Wong <normalperson@yhbt.net>
on curl 7.21.0 and 7.26.0, confirmed via strace:

	setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0

I can also confirm 7.26.0 adds:

	setsockopt(fd, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
	setsockopt(fd, SOL_TCP, TCP_KEEPINTVL, [60], 4) = 0

to the strace on Linux.

> ---
> Given the #ifdefs in curl's keepalive code, I suspect we may see build
> problems for people in case 2 on some systems, with or without my patch.
> I think this patch is a strict improvement, though; if they have a new
> enough curl, they will not even look at the case 2 code. And if they do
> not, our previous options were:
> 
>   a. Add platform-specific code for them.
>   
>   b. Tell them they are out of luck, and add an #ifdef to push them into
>      case 3.

Case 2 works fine on my Debian Squeeze system.

> Now we have an extra option:
> 
>   c. Tell them to upgrade curl. :)

Yes, I keep forgetting I still have a Debian Squeeze machine :x

Re: [PATCH] http: use curl's tcp keepalive if available

[Jeff King]

On Tue, Oct 15, 2013 at 04:58:14AM +0000, Eric Wong wrote:

> > Subject: http: use curl's tcp keepalive if available
> [...]
> Tested-by: Eric Wong <normalperson@yhbt.net>
> on curl 7.21.0 and 7.26.0, confirmed via strace:
> 
> 	setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
> 
> I can also confirm 7.26.0 adds:
> 
> 	setsockopt(fd, SOL_TCP, TCP_KEEPIDLE, [60], 4) = 0
> 	setsockopt(fd, SOL_TCP, TCP_KEEPINTVL, [60], 4) = 0
> 
> to the strace on Linux.

Thanks, I didn't actually do any testing myself.

> Case 2 works fine on my Debian Squeeze system.
> 
> > Now we have an extra option:
> > 
> >   c. Tell them to upgrade curl. :)
> 
> Yes, I keep forgetting I still have a Debian Squeeze machine :x

I am more concerned with Windows, which may not compile your original
patch at all.

-Peff

Re: [PATCH] http: use curl's tcp keepalive if available

[Eric Wong]

Jeff King <peff@peff.net> wrote:
> I am more concerned with Windows, which may not compile your original
> patch at all.

The code I introduced in e47a8583a20256851e7fc882233e3bd5bf33dc6e
("enable SO_KEEPALIVE for connected TCP sockets" Dec 2011)
didn't trigger the addition of any new #ifdef guards.  I think we're
fine (but I'll let others confirm it).