git@vger.kernel.org mailing list mirror (one of many)
 help / Atom feed
* [PATCH 00/26] protocol version 2
@ 2018-01-03  0:18 Brandon Williams
  2018-01-03  0:18 ` [PATCH 01/26] pkt-line: introduce packet_read_with_status Brandon Williams
                   ` (26 more replies)
  0 siblings, 27 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

The following patches extend what I sent out as an WIP
(https://public-inbox.org/git/20171204235823.63299-1-bmwill@google.com/) and
implement protocol version 2.

Some changes from that series are as follows:
 * Lots of various cleanup on the ls-refs and fetch command code, both server
   and client.
 * Fetch command now supports a stateless-rpc mode which enables communicating
   with a half-duplex connection.
 * Introduce a new remote-helper command 'connect-half-duplex' which is
   implemented by remote-curl (the http remote-helper).  This allows for a
   client to establish a half-duplex connection and use remote-curl as a proxy
   to wrap requests in http before sending them to the remote end and
   unwrapping the responses and sending them back to the client's stdin.
 * The transport code is refactored for ls-remote, fetch, and push to provide a
   list of ref-patterns (based on the refspec being used) when requesting refs
   from the remote end.  This allows the ls-refs code to send this list of
   patterns so the remote end and filter the refs it sends back.

This series effectively implements protocol version 2 for listing a remotes
refs (ls-remote) as well as for fetch for the builtin transports (ssh, git,
file) and for the http/https transports.  Push is not implemented yet and
doesn't need to be implemented at the same time as fetch since the
receive-pack code can default to using protocol v0 when v2 is requested by the
client.

Any feedback is appreciated! Thanks!
	Brandon

Brandon Williams (26):
  pkt-line: introduce packet_read_with_status
  pkt-line: introduce struct packet_reader
  pkt-line: add delim packet support
  upload-pack: convert to a builtin
  upload-pack: factor out processing lines
  transport: use get_refs_via_connect to get refs
  connect: convert get_remote_heads to use struct packet_reader
  connect: discover protocol version outside of get_remote_heads
  transport: store protocol version
  protocol: introduce enum protocol_version value protocol_v2
  serve: introduce git-serve
  ls-refs: introduce ls-refs server command
  connect: request remote refs using v2
  transport: convert get_refs_list to take a list of ref patterns
  transport: convert transport_get_remote_refs to take a list of ref
    patterns
  ls-remote: pass ref patterns when requesting a remote's refs
  fetch: pass ref patterns when fetching
  push: pass ref patterns when pushing
  upload-pack: introduce fetch server command
  fetch-pack: perform a fetch using v2
  transport-helper: remove name parameter
  transport-helper: refactor process_connect_service
  transport-helper: introduce connect-half-duplex
  pkt-line: add packet_buf_write_len function
  remote-curl: create copy of the service name
  remote-curl: implement connect-half-duplex command

 .gitignore                              |   1 +
 Documentation/technical/protocol-v2.txt | 131 ++++++++++
 Makefile                                |   6 +-
 builtin.h                               |   2 +
 builtin/clone.c                         |   2 +-
 builtin/fetch-pack.c                    |  21 +-
 builtin/fetch.c                         |  14 +-
 builtin/ls-remote.c                     |   7 +-
 builtin/receive-pack.c                  |   6 +
 builtin/remote.c                        |   2 +-
 builtin/send-pack.c                     |  20 +-
 builtin/serve.c                         |  30 +++
 connect.c                               | 226 +++++++++++++-----
 connect.h                               |   3 +
 fetch-pack.c                            | 267 ++++++++++++++++++++-
 fetch-pack.h                            |   4 +-
 git.c                                   |   2 +
 ls-refs.c                               |  97 ++++++++
 ls-refs.h                               |   9 +
 pkt-line.c                              | 147 +++++++++++-
 pkt-line.h                              |  76 ++++++
 protocol.c                              |   2 +
 protocol.h                              |   1 +
 remote-curl.c                           | 209 +++++++++++++++-
 remote.h                                |   9 +-
 serve.c                                 | 243 +++++++++++++++++++
 serve.h                                 |  15 ++
 t/t5701-protocol-v2.sh                  | 117 +++++++++
 transport-helper.c                      |  84 ++++---
 transport-internal.h                    |   4 +-
 transport.c                             | 119 ++++++---
 transport.h                             |   9 +-
 upload-pack.c                           | 412 ++++++++++++++++++++++++++++----
 upload-pack.h                           |   9 +
 34 files changed, 2108 insertions(+), 198 deletions(-)
 create mode 100644 Documentation/technical/protocol-v2.txt
 create mode 100644 builtin/serve.c
 create mode 100644 ls-refs.c
 create mode 100644 ls-refs.h
 create mode 100644 serve.c
 create mode 100644 serve.h
 create mode 100755 t/t5701-protocol-v2.sh
 create mode 100644 upload-pack.h

-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 01/26] pkt-line: introduce packet_read_with_status
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03 19:27   ` Stefan Beller
  2018-01-09 18:04   ` Jonathan Tan
  2018-01-03  0:18 ` [PATCH 02/26] pkt-line: introduce struct packet_reader Brandon Williams
                   ` (25 subsequent siblings)
  26 siblings, 2 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

The current pkt-line API encodes the status of a pkt-line read in the
length of the read content.  An error is indicated with '-1', a flush
with '0' (which can be confusing since a return value of '0' can also
indicate an empty pkt-line), and a positive integer for the length of
the read content otherwise.  This doesn't leave much room for allowing
the addition of additional special packets in the future.

To solve this introduce 'packet_read_with_status()' which reads a packet
and returns the status of the read encoded as an 'enum packet_status'
type.  This allows for easily identifying between special and normal
packets as well as errors.  It also enables easily adding a new special
packet in the future.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 pkt-line.c | 55 ++++++++++++++++++++++++++++++++++++++++++-------------
 pkt-line.h | 15 +++++++++++++++
 2 files changed, 57 insertions(+), 13 deletions(-)

diff --git a/pkt-line.c b/pkt-line.c
index 2827ca772..8d7cd389f 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -280,28 +280,33 @@ static int packet_length(const char *linelen)
 	return (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);
 }
 
-int packet_read(int fd, char **src_buf, size_t *src_len,
-		char *buffer, unsigned size, int options)
+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
+						char *buffer, unsigned size, int *pktlen,
+						int options)
 {
-	int len, ret;
+	int len;
 	char linelen[4];
 
-	ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);
-	if (ret < 0)
-		return ret;
+	if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)
+		return PACKET_READ_EOF;
+
 	len = packet_length(linelen);
 	if (len < 0)
 		die("protocol error: bad line length character: %.4s", linelen);
-	if (!len) {
+
+	if (len == 0) {
 		packet_trace("0000", 4, 0);
-		return 0;
+		return PACKET_READ_FLUSH;
+	} else if (len >= 1 && len <= 3) {
+		die("protocol error: bad line length character: %.4s", linelen);
 	}
+
 	len -= 4;
-	if (len >= size)
+	if ((len < 0) || ((unsigned)len >= size))
 		die("protocol error: bad line length %d", len);
-	ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);
-	if (ret < 0)
-		return ret;
+
+	if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)
+		return PACKET_READ_EOF;
 
 	if ((options & PACKET_READ_CHOMP_NEWLINE) &&
 	    len && buffer[len-1] == '\n')
@@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,
 
 	buffer[len] = 0;
 	packet_trace(buffer, len, 0);
-	return len;
+	*pktlen = len;
+	return PACKET_READ_NORMAL;
+}
+
+int packet_read(int fd, char **src_buffer, size_t *src_len,
+		char *buffer, unsigned size, int options)
+{
+	enum packet_read_status status;
+	int pktlen;
+
+	status = packet_read_with_status(fd, src_buffer, src_len,
+					 buffer, size, &pktlen,
+					 options);
+	switch (status) {
+	case PACKET_READ_EOF:
+		pktlen = -1;
+		break;
+	case PACKET_READ_NORMAL:
+		break;
+	case PACKET_READ_FLUSH:
+		pktlen = 0;
+		break;
+	}
+
+	return pktlen;
 }
 
 static char *packet_read_line_generic(int fd,
diff --git a/pkt-line.h b/pkt-line.h
index 3dad583e2..06c468927 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);
 int packet_read(int fd, char **src_buffer, size_t *src_len, char
 		*buffer, unsigned size, int options);
 
+/*
+ * Read a packetized line into a buffer like the 'packet_read()' function but
+ * returns an 'enum packet_read_status' which indicates the status of the read.
+ * The number of bytes read will be assigined to *pktlen if the status of the
+ * read was 'PACKET_READ_NORMAL'.
+ */
+enum packet_read_status {
+	PACKET_READ_EOF = -1,
+	PACKET_READ_NORMAL,
+	PACKET_READ_FLUSH,
+};
+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
+						char *buffer, unsigned size, int *pktlen,
+						int options);
+
 /*
  * Convenience wrapper for packet_read that is not gentle, and sets the
  * CHOMP_NEWLINE option. The return value is NULL for a flush packet,
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 02/26] pkt-line: introduce struct packet_reader
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
  2018-01-03  0:18 ` [PATCH 01/26] pkt-line: introduce packet_read_with_status Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-09 18:08   ` Jonathan Tan
  2018-01-03  0:18 ` [PATCH 03/26] pkt-line: add delim packet support Brandon Williams
                   ` (24 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Sometimes it is advantageous to be able to peek the next packet line
without consuming it (e.g. to be able to determine the protocol version
a server is speaking).  In order to do that introduce 'struct
packet_reader' which is an abstraction around the normal packet reading
logic.  This enables a caller to be able to peek a single line at a time
using 'packet_reader_peek()' and having a caller consume a line by
calling 'packet_reader_read()'.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 pkt-line.h | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 116 insertions(+)

diff --git a/pkt-line.c b/pkt-line.c
index 8d7cd389f..98c2d7d68 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -406,3 +406,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)
 	}
 	return sb_out->len - orig_len;
 }
+
+/* Packet Reader Functions */
+void packet_reader_init(struct packet_reader *reader, int fd,
+			char *src_buffer, size_t src_len,
+			int options)
+{
+	memset(reader, 0, sizeof(*reader));
+
+	reader->fd = fd;
+	reader->src_buffer = src_buffer;
+	reader->src_len = src_len;
+	reader->buffer = packet_buffer;
+	reader->buffer_size = sizeof(packet_buffer);
+	reader->options = options;
+}
+
+enum packet_read_status packet_reader_read(struct packet_reader *reader)
+{
+	if (reader->line_peeked) {
+		reader->line_peeked = 0;
+		return reader->status;
+	}
+
+	reader->status = packet_read_with_status(reader->fd,
+						 &reader->src_buffer,
+						 &reader->src_len,
+						 reader->buffer,
+						 reader->buffer_size,
+						 &reader->pktlen,
+						 reader->options);
+
+	switch (reader->status) {
+	case PACKET_READ_EOF:
+		reader->pktlen = -1;
+		reader->line = NULL;
+		break;
+	case PACKET_READ_NORMAL:
+		reader->line = reader->buffer;
+		break;
+	case PACKET_READ_FLUSH:
+		reader->pktlen = 0;
+		reader->line = NULL;
+		break;
+	}
+
+	return reader->status;
+}
+
+enum packet_read_status packet_reader_peek(struct packet_reader *reader)
+{
+	/* Only allow peeking a single line */
+	if (reader->line_peeked)
+		return reader->status;
+
+	/* Peek a line by reading it and setting peeked flag */
+	packet_reader_read(reader);
+	reader->line_peeked = 1;
+	return reader->status;
+}
diff --git a/pkt-line.h b/pkt-line.h
index 06c468927..c446e886a 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -111,6 +111,63 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
  */
 ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
 
+struct packet_reader {
+	/* source file descriptor */
+	int fd;
+
+	/* source buffer and its size */
+	char *src_buffer;
+	size_t src_len;
+
+	/* buffer that pkt-lines are read into and its size */
+	char *buffer;
+	unsigned buffer_size;
+
+	/* options to be used during reads */
+	int options;
+
+	/* status of the last read */
+	enum packet_read_status status;
+
+	/* length of data read during the last read */
+	int pktlen;
+
+	/* the last line read */
+	const char *line;
+
+	/* indicates if a line has been peeked */
+	int line_peeked;
+};
+
+/*
+ * Initialize a 'struct packet_reader' object which is an
+ * abstraction around the 'packet_read_with_status()' function.
+ */
+extern void packet_reader_init(struct packet_reader *reader, int fd,
+			       char *src_buffer, size_t src_len,
+			       int options);
+
+/*
+ * Perform a packet read and return the status of the read.
+ * The values of 'pktlen' and 'line' are updated based on the status of the
+ * read as follows:
+ *
+ * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL
+ * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read
+ *		       'line' is set to point at the read line
+ * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL
+ */
+extern enum packet_read_status packet_reader_read(struct packet_reader *reader);
+
+/*
+ * Peek the next packet line without consuming it and return the status.
+ * The next call to 'packet_reader_read()' will perform a read of the same line
+ * that was peeked, consuming the line.
+ *
+ * Only a single line can be peeked at a time.
+ */
+extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);
+
 #define DEFAULT_PACKET_MAX 1000
 #define LARGE_PACKET_MAX 65520
 #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 03/26] pkt-line: add delim packet support
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
  2018-01-03  0:18 ` [PATCH 01/26] pkt-line: introduce packet_read_with_status Brandon Williams
  2018-01-03  0:18 ` [PATCH 02/26] pkt-line: introduce struct packet_reader Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 04/26] upload-pack: convert to a builtin Brandon Williams
                   ` (23 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

One of the design goals of protocol-v2 is to improve the semantics of
flush packets.  Currently in protocol-v1, flush packets are used both to
indicate a break in a list of packet lines as well as an indication that
one side has finished speaking.  This makes it particularly difficult
to implement proxies as a proxy would need to completely understand git
protocol instead of simply looking for a flush packet.

To do this, introduce the special deliminator packet '0001'.  A delim
packet can then be used as a deliminator between lists of packet lines
while flush packets can be reserved to indicate the end of a response.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 pkt-line.c | 19 ++++++++++++++++++-
 pkt-line.h |  3 +++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/pkt-line.c b/pkt-line.c
index 98c2d7d68..3159cbe10 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -91,6 +91,12 @@ void packet_flush(int fd)
 	write_or_die(fd, "0000", 4);
 }
 
+void packet_delim(int fd)
+{
+	packet_trace("0001", 4, 1);
+	write_or_die(fd, "0001", 4);
+}
+
 int packet_flush_gently(int fd)
 {
 	packet_trace("0000", 4, 1);
@@ -105,6 +111,12 @@ void packet_buf_flush(struct strbuf *buf)
 	strbuf_add(buf, "0000", 4);
 }
 
+void packet_buf_delim(struct strbuf *buf)
+{
+	packet_trace("0001", 4, 1);
+	strbuf_add(buf, "0001", 4);
+}
+
 static void set_packet_header(char *buf, const int size)
 {
 	static char hexchar[] = "0123456789abcdef";
@@ -297,7 +309,10 @@ enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_
 	if (len == 0) {
 		packet_trace("0000", 4, 0);
 		return PACKET_READ_FLUSH;
-	} else if (len >= 1 && len <= 3) {
+	} else if (len == 1) {
+		packet_trace("0001", 4, 0);
+		return PACKET_READ_DELIM;
+	} else if (len >= 2 && len <= 3) {
 		die("protocol error: bad line length character: %.4s", linelen);
 	}
 
@@ -333,6 +348,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len,
 		break;
 	case PACKET_READ_NORMAL:
 		break;
+	case PACKET_READ_DELIM:
 	case PACKET_READ_FLUSH:
 		pktlen = 0;
 		break;
@@ -445,6 +461,7 @@ enum packet_read_status packet_reader_read(struct packet_reader *reader)
 	case PACKET_READ_NORMAL:
 		reader->line = reader->buffer;
 		break;
+	case PACKET_READ_DELIM:
 	case PACKET_READ_FLUSH:
 		reader->pktlen = 0;
 		reader->line = NULL;
diff --git a/pkt-line.h b/pkt-line.h
index c446e886a..97b6dd1c7 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -20,8 +20,10 @@
  * side can't, we stay with pure read/write interfaces.
  */
 void packet_flush(int fd);
+void packet_delim(int fd);
 void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
 void packet_buf_flush(struct strbuf *buf);
+void packet_buf_delim(struct strbuf *buf);
 void packet_write(int fd_out, const char *buf, size_t size);
 void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
 int packet_flush_gently(int fd);
@@ -75,6 +77,7 @@ enum packet_read_status {
 	PACKET_READ_EOF = -1,
 	PACKET_READ_NORMAL,
 	PACKET_READ_FLUSH,
+	PACKET_READ_DELIM,
 };
 enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
 						char *buffer, unsigned size, int *pktlen,
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 04/26] upload-pack: convert to a builtin
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (2 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 03/26] pkt-line: add delim packet support Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03 20:33   ` Stefan Beller
  2018-01-03  0:18 ` [PATCH 05/26] upload-pack: factor out processing lines Brandon Williams
                   ` (22 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

In order to allow for code sharing with the server-side of fetch in
protocol-v2 convert upload-pack to be a builtin.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Makefile      | 3 ++-
 builtin.h     | 1 +
 git.c         | 1 +
 upload-pack.c | 2 +-
 4 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 2a81ae22e..e0740b452 100644
--- a/Makefile
+++ b/Makefile
@@ -636,7 +636,6 @@ PROGRAM_OBJS += imap-send.o
 PROGRAM_OBJS += sh-i18n--envsubst.o
 PROGRAM_OBJS += shell.o
 PROGRAM_OBJS += show-index.o
-PROGRAM_OBJS += upload-pack.o
 PROGRAM_OBJS += remote-testsvn.o
 
 # Binary suffix, set to .exe for Windows builds
@@ -701,6 +700,7 @@ BUILT_INS += git-merge-subtree$X
 BUILT_INS += git-show$X
 BUILT_INS += git-stage$X
 BUILT_INS += git-status$X
+BUILT_INS += git-upload-pack$X
 BUILT_INS += git-whatchanged$X
 
 # what 'all' will build and 'install' will install in gitexecdir,
@@ -904,6 +904,7 @@ LIB_OBJS += tree-diff.o
 LIB_OBJS += tree.o
 LIB_OBJS += tree-walk.o
 LIB_OBJS += unpack-trees.o
+LIB_OBJS += upload-pack.o
 LIB_OBJS += url.o
 LIB_OBJS += urlmatch.o
 LIB_OBJS += usage.o
diff --git a/builtin.h b/builtin.h
index 42378f3aa..f332a1257 100644
--- a/builtin.h
+++ b/builtin.h
@@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);
 extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);
 extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);
 extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);
+extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);
 extern int cmd_var(int argc, const char **argv, const char *prefix);
 extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);
 extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);
diff --git a/git.c b/git.c
index c870b9719..f71073dc8 100644
--- a/git.c
+++ b/git.c
@@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {
 	{ "update-server-info", cmd_update_server_info, RUN_SETUP },
 	{ "upload-archive", cmd_upload_archive },
 	{ "upload-archive--writer", cmd_upload_archive_writer },
+	{ "upload-pack", cmd_upload_pack },
 	{ "var", cmd_var, RUN_SETUP_GENTLY },
 	{ "verify-commit", cmd_verify_commit, RUN_SETUP },
 	{ "verify-pack", cmd_verify_pack },
diff --git a/upload-pack.c b/upload-pack.c
index d5de18127..20acaa49d 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -1032,7 +1032,7 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
 	return parse_hide_refs_config(var, value, "uploadpack");
 }
 
-int cmd_main(int argc, const char **argv)
+int cmd_upload_pack(int argc, const char **argv, const char *prefix)
 {
 	const char *dir;
 	int strict = 0;
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 05/26] upload-pack: factor out processing lines
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (3 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 04/26] upload-pack: convert to a builtin Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03 20:38   ` Stefan Beller
  2018-01-03  0:18 ` [PATCH 06/26] transport: use get_refs_via_connect to get refs Brandon Williams
                   ` (21 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Factor out the logic for processing shallow, deepen, deepen_since, and
deepen_not lines into their own functions to simplify the
'receive_needs()' function in addition to making it easier to reuse some
of this logic when implementing protocol_v2.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 74 insertions(+), 39 deletions(-)

diff --git a/upload-pack.c b/upload-pack.c
index 20acaa49d..9a507ae53 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -731,6 +731,75 @@ static void deepen_by_rev_list(int ac, const char **av,
 	packet_flush(1);
 }
 
+static int process_shallow(const char *line, struct object_array *shallows)
+{
+	const char *arg;
+	if (skip_prefix(line, "shallow ", &arg)) {
+		struct object_id oid;
+		struct object *object;
+		if (get_oid_hex(arg, &oid))
+			die("invalid shallow line: %s", line);
+		object = parse_object(&oid);
+		if (!object)
+			return 1;
+		if (object->type != OBJ_COMMIT)
+			die("invalid shallow object %s", oid_to_hex(&oid));
+		if (!(object->flags & CLIENT_SHALLOW)) {
+			object->flags |= CLIENT_SHALLOW;
+			add_object_array(object, NULL, shallows);
+		}
+		return 1;
+	}
+
+	return 0;
+}
+
+static int process_deepen(const char *line, int *depth)
+{
+	const char *arg;
+	if (skip_prefix(line, "deepen ", &arg)) {
+		char *end = NULL;
+		*depth = strtol(arg, &end, 0);
+		if (!end || *end || depth <= 0)
+			die("Invalid deepen: %s", line);
+		return 1;
+	}
+
+	return 0;
+}
+
+static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)
+{
+	const char *arg;
+	if (skip_prefix(line, "deepen-since ", &arg)) {
+		char *end = NULL;
+		*deepen_since = parse_timestamp(arg, &end, 0);
+		if (!end || *end || !deepen_since ||
+		    /* revisions.c's max_age -1 is special */
+		    *deepen_since == -1)
+			die("Invalid deepen-since: %s", line);
+		*deepen_rev_list = 1;
+		return 1;
+	}
+	return 0;
+}
+
+static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)
+{
+	const char *arg;
+	if (skip_prefix(line, "deepen-not ", &arg)) {
+		char *ref = NULL;
+		struct object_id oid;
+		if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
+			die("git upload-pack: ambiguous deepen-not: %s", line);
+		string_list_append(deepen_not, ref);
+		free(ref);
+		*deepen_rev_list = 1;
+		return 1;
+	}
+	return 0;
+}
+
 static void receive_needs(void)
 {
 	struct object_array shallows = OBJECT_ARRAY_INIT;
@@ -752,49 +821,15 @@ static void receive_needs(void)
 		if (!line)
 			break;
 
-		if (skip_prefix(line, "shallow ", &arg)) {
-			struct object_id oid;
-			struct object *object;
-			if (get_oid_hex(arg, &oid))
-				die("invalid shallow line: %s", line);
-			object = parse_object(&oid);
-			if (!object)
-				continue;
-			if (object->type != OBJ_COMMIT)
-				die("invalid shallow object %s", oid_to_hex(&oid));
-			if (!(object->flags & CLIENT_SHALLOW)) {
-				object->flags |= CLIENT_SHALLOW;
-				add_object_array(object, NULL, &shallows);
-			}
+		if (process_shallow(line, &shallows))
 			continue;
-		}
-		if (skip_prefix(line, "deepen ", &arg)) {
-			char *end = NULL;
-			depth = strtol(arg, &end, 0);
-			if (!end || *end || depth <= 0)
-				die("Invalid deepen: %s", line);
+		if (process_deepen(line, &depth))
 			continue;
-		}
-		if (skip_prefix(line, "deepen-since ", &arg)) {
-			char *end = NULL;
-			deepen_since = parse_timestamp(arg, &end, 0);
-			if (!end || *end || !deepen_since ||
-			    /* revisions.c's max_age -1 is special */
-			    deepen_since == -1)
-				die("Invalid deepen-since: %s", line);
-			deepen_rev_list = 1;
+		if (process_deepen_since(line, &deepen_since, &deepen_rev_list))
 			continue;
-		}
-		if (skip_prefix(line, "deepen-not ", &arg)) {
-			char *ref = NULL;
-			struct object_id oid;
-			if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
-				die("git upload-pack: ambiguous deepen-not: %s", line);
-			string_list_append(&deepen_not, ref);
-			free(ref);
-			deepen_rev_list = 1;
+		if (process_deepen_not(line, &deepen_not, &deepen_rev_list))
 			continue;
-		}
+
 		if (!skip_prefix(line, "want ", &arg) ||
 		    get_oid_hex(arg, &oid_buf))
 			die("git upload-pack: protocol error, "
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 06/26] transport: use get_refs_via_connect to get refs
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (4 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 05/26] upload-pack: factor out processing lines Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03 21:20   ` Stefan Beller
  2018-01-03  0:18 ` [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader Brandon Williams
                   ` (20 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Remove code duplication and use the existing 'get_refs_via_connect()'
function to retrieve a remote's heads in 'fetch_refs_via_pack()' and
'git_transport_push()'.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 transport.c | 18 ++++--------------
 1 file changed, 4 insertions(+), 14 deletions(-)

diff --git a/transport.c b/transport.c
index fc802260f..8e8779096 100644
--- a/transport.c
+++ b/transport.c
@@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,
 	args.cloning = transport->cloning;
 	args.update_shallow = data->options.update_shallow;
 
-	if (!data->got_remote_heads) {
-		connect_setup(transport, 0);
-		get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
-				 NULL, &data->shallow);
-		data->got_remote_heads = 1;
-	}
+	if (!data->got_remote_heads)
+		refs_tmp = get_refs_via_connect(transport, 0);
 
 	refs = fetch_pack(&args, data->fd, data->conn,
 			  refs_tmp ? refs_tmp : transport->remote_refs,
@@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 	struct send_pack_args args;
 	int ret;
 
-	if (!data->got_remote_heads) {
-		struct ref *tmp_refs;
-		connect_setup(transport, 1);
-
-		get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,
-				 NULL, &data->shallow);
-		data->got_remote_heads = 1;
-	}
+	if (!data->got_remote_heads)
+		get_refs_via_connect(transport, 1);
 
 	memset(&args, 0, sizeof(args));
 	args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (5 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 06/26] transport: use get_refs_via_connect to get refs Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-09 18:27   ` Jonathan Tan
  2018-01-03  0:18 ` [PATCH 08/26] connect: discover protocol version outside of get_remote_heads Brandon Williams
                   ` (19 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

In order to allow for better control flow when protocol_v2 is introduced
convert 'get_remote_heads()' to use 'struct packet_reader' to read
packet lines.  This enables a client to be able to peek the first line
of a server's response (without consuming it) in order to determine the
protocol version its speaking and then passing control to the
appropriate handler.

This is needed because the initial response from a server speaking
protocol_v0 includes the first ref, while subsequent protocol versions
respond with a version line.  We want to be able to read this first line
without consuming the first ref sent in the protocol_v0 case so that the
protocol version the server is speaking can be determined outside of
'get_remote_heads()' in a future patch.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 connect.c | 127 +++++++++++++++++++++++++++++++++++---------------------------
 1 file changed, 72 insertions(+), 55 deletions(-)

diff --git a/connect.c b/connect.c
index c3a014c5b..03bbb74e4 100644
--- a/connect.c
+++ b/connect.c
@@ -48,6 +48,12 @@ int check_ref_type(const struct ref *ref, int flags)
 
 static void die_initial_contact(int unexpected)
 {
+	/*
+	 * A hang-up after seeing some response from the other end
+	 * means that it is unexpected, as we know the other end is
+	 * willing to talk to us.  A hang-up before seeing any
+	 * response does not necessarily mean an ACL problem, though.
+	 */
 	if (unexpected)
 		die(_("The remote end hung up upon initial contact"));
 	else
@@ -56,6 +62,41 @@ static void die_initial_contact(int unexpected)
 		      "and the repository exists."));
 }
 
+static enum protocol_version discover_version(struct packet_reader *reader)
+{
+	enum protocol_version version = protocol_unknown_version;
+
+	/*
+	 * Peek the first line of the server's response to
+	 * determine the protocol version the server is speaking.
+	 */
+	switch (packet_reader_peek(reader)) {
+	case PACKET_READ_EOF:
+		die_initial_contact(0);
+	case PACKET_READ_FLUSH:
+	case PACKET_READ_DELIM:
+		version = protocol_v0;
+		break;
+	case PACKET_READ_NORMAL:
+		version = determine_protocol_version_client(reader->line);
+		break;
+	}
+
+	/* Maybe process capabilities here, at least for v2 */
+	switch (version) {
+	case protocol_v1:
+		/* Read the peeked version line */
+		packet_reader_read(reader);
+		break;
+	case protocol_v0:
+		break;
+	case protocol_unknown_version:
+		die("unknown protocol version: '%s'\n", reader->line);
+	}
+
+	return version;
+}
+
 static void parse_one_symref_info(struct string_list *symref, const char *val, int len)
 {
 	char *sym, *target;
@@ -109,44 +150,10 @@ static void annotate_refs_with_symref_info(struct ref *ref)
 	string_list_clear(&symref, 0);
 }
 
-/*
- * Read one line of a server's ref advertisement into packet_buffer.
- */
-static int read_remote_ref(int in, char **src_buf, size_t *src_len,
-			   int *responded)
-{
-	int len = packet_read(in, src_buf, src_len,
-			      packet_buffer, sizeof(packet_buffer),
-			      PACKET_READ_GENTLE_ON_EOF |
-			      PACKET_READ_CHOMP_NEWLINE);
-	const char *arg;
-	if (len < 0)
-		die_initial_contact(*responded);
-	if (len > 4 && skip_prefix(packet_buffer, "ERR ", &arg))
-		die("remote error: %s", arg);
-
-	*responded = 1;
-
-	return len;
-}
-
-#define EXPECTING_PROTOCOL_VERSION 0
-#define EXPECTING_FIRST_REF 1
-#define EXPECTING_REF 2
-#define EXPECTING_SHALLOW 3
-
-/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */
-static int process_protocol_version(void)
-{
-	switch (determine_protocol_version_client(packet_buffer)) {
-	case protocol_v1:
-		return 1;
-	case protocol_v0:
-		return 0;
-	default:
-		die("server is speaking an unknown protocol");
-	}
-}
+#define EXPECTING_FIRST_REF 0
+#define EXPECTING_REF 1
+#define EXPECTING_SHALLOW 2
+#define EXPECTING_DONE 3
 
 static void process_capabilities(int *len)
 {
@@ -230,28 +237,36 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 			      struct oid_array *shallow_points)
 {
 	struct ref **orig_list = list;
+	int len = 0;
+	int state = EXPECTING_FIRST_REF;
+	struct packet_reader reader;
+	const char *arg;
 
-	/*
-	 * A hang-up after seeing some response from the other end
-	 * means that it is unexpected, as we know the other end is
-	 * willing to talk to us.  A hang-up before seeing any
-	 * response does not necessarily mean an ACL problem, though.
-	 */
-	int responded = 0;
-	int len;
-	int state = EXPECTING_PROTOCOL_VERSION;
+	packet_reader_init(&reader, in, src_buf, src_len,
+			   PACKET_READ_CHOMP_NEWLINE |
+			   PACKET_READ_GENTLE_ON_EOF);
+
+	discover_version(&reader);
 
 	*list = NULL;
 
-	while ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {
+	while (state != EXPECTING_DONE) {
+		switch (packet_reader_read(&reader)) {
+		case PACKET_READ_EOF:
+			die_initial_contact(1);
+		case PACKET_READ_NORMAL:
+			len = reader.pktlen;
+			if (len > 4 && skip_prefix(packet_buffer, "ERR ", &arg))
+				die("remote error: %s", arg);
+			break;
+		case PACKET_READ_FLUSH:
+			state = EXPECTING_DONE;
+			break;
+		case PACKET_READ_DELIM:
+			die("invalid packet\n");
+		}
+
 		switch (state) {
-		case EXPECTING_PROTOCOL_VERSION:
-			if (process_protocol_version()) {
-				state = EXPECTING_FIRST_REF;
-				break;
-			}
-			state = EXPECTING_FIRST_REF;
-			/* fallthrough */
 		case EXPECTING_FIRST_REF:
 			process_capabilities(&len);
 			if (process_dummy_ref()) {
@@ -269,6 +284,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 			if (process_shallow(len, shallow_points))
 				break;
 			die("protocol error: unexpected '%s'", packet_buffer);
+		case EXPECTING_DONE:
+			break;
 		default:
 			die("unexpected state %d", state);
 		}
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 08/26] connect: discover protocol version outside of get_remote_heads
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (6 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 09/26] transport: store protocol version Brandon Williams
                   ` (18 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

In order to prepare for the addition of protocol_v2 push the protocol
version discovery outside of 'get_remote_heads()'.  This will allow for
keeping the logic for processing the reference advertisement for
protocol_v1 and protocol_v0 separate from the logic for protocol_v2.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/fetch-pack.c | 16 +++++++++++++++-
 builtin/send-pack.c  | 17 +++++++++++++++--
 connect.c            | 15 ++++-----------
 connect.h            |  3 +++
 remote-curl.c        | 20 ++++++++++++++++++--
 remote.h             |  5 +++--
 transport.c          | 24 +++++++++++++++++++-----
 7 files changed, 77 insertions(+), 23 deletions(-)

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 366b9d13f..85d4faf76 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -4,6 +4,7 @@
 #include "remote.h"
 #include "connect.h"
 #include "sha1-array.h"
+#include "protocol.h"
 
 static const char fetch_pack_usage[] =
 "git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] "
@@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 	struct fetch_pack_args args;
 	struct oid_array shallow = OID_ARRAY_INIT;
 	struct string_list deepen_not = STRING_LIST_INIT_DUP;
+	struct packet_reader reader;
 
 	packet_trace_identity("fetch-pack");
 
@@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 		if (!conn)
 			return args.diag_url ? 0 : 1;
 	}
-	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
+
+	packet_reader_init(&reader, fd[0], NULL, 0,
+			   PACKET_READ_CHOMP_NEWLINE |
+			   PACKET_READ_GENTLE_ON_EOF);
+
+	switch (discover_version(&reader)) {
+	case protocol_v1:
+	case protocol_v0:
+		get_remote_heads(&reader, &ref, 0, NULL, &shallow);
+		break;
+	case protocol_unknown_version:
+		BUG("unknown protocol version");
+	}
 
 	ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,
 			 &shallow, pack_lockfile_ptr);
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index fc4f0bb5f..83cb125a6 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -14,6 +14,7 @@
 #include "sha1-array.h"
 #include "gpg-interface.h"
 #include "gettext.h"
+#include "protocol.h"
 
 static const char * const send_pack_usage[] = {
 	N_("git send-pack [--all | --mirror] [--dry-run] [--force] "
@@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int progress = -1;
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
+	struct packet_reader reader;
 
 	struct option options[] = {
 		OPT__VERBOSITY(&verbose),
@@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 			args.verbose ? CONNECT_VERBOSE : 0);
 	}
 
-	get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,
-			 &extra_have, &shallow);
+	packet_reader_init(&reader, fd[0], NULL, 0,
+			   PACKET_READ_CHOMP_NEWLINE |
+			   PACKET_READ_GENTLE_ON_EOF);
+
+	switch (discover_version(&reader)) {
+	case protocol_v1:
+	case protocol_v0:
+		get_remote_heads(&reader, &remote_refs, REF_NORMAL,
+				 &extra_have, &shallow);
+		break;
+	case protocol_unknown_version:
+		BUG("unknown protocol version");
+	}
 
 	transport_verify_remote_names(nr_refspecs, refspecs);
 
diff --git a/connect.c b/connect.c
index 03bbb74e4..1787b0212 100644
--- a/connect.c
+++ b/connect.c
@@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)
 		      "and the repository exists."));
 }
 
-static enum protocol_version discover_version(struct packet_reader *reader)
+enum protocol_version discover_version(struct packet_reader *reader)
 {
 	enum protocol_version version = protocol_unknown_version;
 
@@ -231,7 +231,7 @@ static int process_shallow(int len, struct oid_array *shallow_points)
 /*
  * Read all the refs from the other end
  */
-struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
+struct ref **get_remote_heads(struct packet_reader *reader,
 			      struct ref **list, unsigned int flags,
 			      struct oid_array *extra_have,
 			      struct oid_array *shallow_points)
@@ -239,23 +239,16 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 	struct ref **orig_list = list;
 	int len = 0;
 	int state = EXPECTING_FIRST_REF;
-	struct packet_reader reader;
 	const char *arg;
 
-	packet_reader_init(&reader, in, src_buf, src_len,
-			   PACKET_READ_CHOMP_NEWLINE |
-			   PACKET_READ_GENTLE_ON_EOF);
-
-	discover_version(&reader);
-
 	*list = NULL;
 
 	while (state != EXPECTING_DONE) {
-		switch (packet_reader_read(&reader)) {
+		switch (packet_reader_read(reader)) {
 		case PACKET_READ_EOF:
 			die_initial_contact(1);
 		case PACKET_READ_NORMAL:
-			len = reader.pktlen;
+			len = reader->pktlen;
 			if (len > 4 && skip_prefix(packet_buffer, "ERR ", &arg))
 				die("remote error: %s", arg);
 			break;
diff --git a/connect.h b/connect.h
index 01f14cdf3..cdb8979dc 100644
--- a/connect.h
+++ b/connect.h
@@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);
 extern const char *server_feature_value(const char *feature, int *len_ret);
 extern int url_is_local_not_ssh(const char *url);
 
+struct packet_reader;
+extern enum protocol_version discover_version(struct packet_reader *reader);
+
 #endif
diff --git a/remote-curl.c b/remote-curl.c
index 0053b0954..9f6d07683 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "config.h"
 #include "remote.h"
+#include "connect.h"
 #include "strbuf.h"
 #include "walker.h"
 #include "http.h"
@@ -13,6 +14,7 @@
 #include "credential.h"
 #include "sha1-array.h"
 #include "send-pack.h"
+#include "protocol.h"
 
 static struct remote *remote;
 /* always ends with a trailing slash */
@@ -176,8 +178,22 @@ static struct discovery *last_discovery;
 static struct ref *parse_git_refs(struct discovery *heads, int for_push)
 {
 	struct ref *list = NULL;
-	get_remote_heads(-1, heads->buf, heads->len, &list,
-			 for_push ? REF_NORMAL : 0, NULL, &heads->shallow);
+	struct packet_reader reader;
+
+	packet_reader_init(&reader, -1, heads->buf, heads->len,
+			   PACKET_READ_CHOMP_NEWLINE |
+			   PACKET_READ_GENTLE_ON_EOF);
+
+	switch (discover_version(&reader)) {
+	case protocol_v1:
+	case protocol_v0:
+		get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
+				 NULL, &heads->shallow);
+		break;
+	case protocol_unknown_version:
+		BUG("unknown protocol version");
+	}
+
 	return list;
 }
 
diff --git a/remote.h b/remote.h
index 1f6611be2..2016461df 100644
--- a/remote.h
+++ b/remote.h
@@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);
 void free_refs(struct ref *ref);
 
 struct oid_array;
-extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
+struct packet_reader;
+extern struct ref **get_remote_heads(struct packet_reader *reader,
 				     struct ref **list, unsigned int flags,
 				     struct oid_array *extra_have,
-				     struct oid_array *shallow);
+				     struct oid_array *shallow_points);
 
 int resolve_remote_symref(struct ref *ref, struct ref *list);
 int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);
diff --git a/transport.c b/transport.c
index 8e8779096..63c3dbab9 100644
--- a/transport.c
+++ b/transport.c
@@ -18,6 +18,7 @@
 #include "sha1-array.h"
 #include "sigchain.h"
 #include "transport-internal.h"
+#include "protocol.h"
 
 static void set_upstreams(struct transport *transport, struct ref *refs,
 	int pretend)
@@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)
 static struct ref *get_refs_via_connect(struct transport *transport, int for_push)
 {
 	struct git_transport_data *data = transport->data;
-	struct ref *refs;
+	struct ref *refs = NULL;
+	struct packet_reader reader;
 
 	connect_setup(transport, for_push);
-	get_remote_heads(data->fd[0], NULL, 0, &refs,
-			 for_push ? REF_NORMAL : 0,
-			 &data->extra_have,
-			 &data->shallow);
+
+	packet_reader_init(&reader, data->fd[0], NULL, 0,
+			   PACKET_READ_CHOMP_NEWLINE |
+			   PACKET_READ_GENTLE_ON_EOF);
+
+	switch (discover_version(&reader)) {
+	case protocol_v1:
+	case protocol_v0:
+		get_remote_heads(&reader, &refs,
+				 for_push ? REF_NORMAL : 0,
+				 &data->extra_have,
+				 &data->shallow);
+		break;
+	case protocol_unknown_version:
+		BUG("unknown protocol version");
+	}
 	data->got_remote_heads = 1;
 
 	return refs;
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 09/26] transport: store protocol version
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (7 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 08/26] connect: discover protocol version outside of get_remote_heads Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-09 18:41   ` Jonathan Tan
  2018-01-03  0:18 ` [PATCH 10/26] protocol: introduce enum protocol_version value protocol_v2 Brandon Williams
                   ` (17 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Once protocol_v2 is introduced requesting a fetch or a push will need to
be handled differently depending on the protocol version.  Store the
protocol version the server is speaking in 'struct git_transport_data'
and use it to determine what to do in the case of a fetch or a push.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 transport.c | 35 ++++++++++++++++++++++++++---------
 1 file changed, 26 insertions(+), 9 deletions(-)

diff --git a/transport.c b/transport.c
index 63c3dbab9..2378dcb38 100644
--- a/transport.c
+++ b/transport.c
@@ -118,6 +118,7 @@ struct git_transport_data {
 	struct child_process *conn;
 	int fd[2];
 	unsigned got_remote_heads : 1;
+	enum protocol_version version;
 	struct oid_array extra_have;
 	struct oid_array shallow;
 };
@@ -200,7 +201,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 			   PACKET_READ_CHOMP_NEWLINE |
 			   PACKET_READ_GENTLE_ON_EOF);
 
-	switch (discover_version(&reader)) {
+	data->version = discover_version(&reader);
+	switch (data->version) {
 	case protocol_v1:
 	case protocol_v0:
 		get_remote_heads(&reader, &refs,
@@ -221,7 +223,7 @@ static int fetch_refs_via_pack(struct transport *transport,
 {
 	int ret = 0;
 	struct git_transport_data *data = transport->data;
-	struct ref *refs;
+	struct ref *refs = NULL;
 	char *dest = xstrdup(transport->url);
 	struct fetch_pack_args args;
 	struct ref *refs_tmp = NULL;
@@ -247,10 +249,18 @@ static int fetch_refs_via_pack(struct transport *transport,
 	if (!data->got_remote_heads)
 		refs_tmp = get_refs_via_connect(transport, 0);
 
-	refs = fetch_pack(&args, data->fd, data->conn,
-			  refs_tmp ? refs_tmp : transport->remote_refs,
-			  dest, to_fetch, nr_heads, &data->shallow,
-			  &transport->pack_lockfile);
+	switch (data->version) {
+	case protocol_v1:
+	case protocol_v0:
+		refs = fetch_pack(&args, data->fd, data->conn,
+				  refs_tmp ? refs_tmp : transport->remote_refs,
+				  dest, to_fetch, nr_heads, &data->shallow,
+				  &transport->pack_lockfile);
+		break;
+	case protocol_unknown_version:
+		BUG("unknown protocol version");
+	}
+
 	close(data->fd[0]);
 	close(data->fd[1]);
 	if (finish_connect(data->conn))
@@ -549,7 +559,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 {
 	struct git_transport_data *data = transport->data;
 	struct send_pack_args args;
-	int ret;
+	int ret = 0;
 
 	if (!data->got_remote_heads)
 		get_refs_via_connect(transport, 1);
@@ -574,8 +584,15 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 	else
 		args.push_cert = SEND_PACK_PUSH_CERT_NEVER;
 
-	ret = send_pack(&args, data->fd, data->conn, remote_refs,
-			&data->extra_have);
+	switch (data->version) {
+	case protocol_v1:
+	case protocol_v0:
+		ret = send_pack(&args, data->fd, data->conn, remote_refs,
+				&data->extra_have);
+		break;
+	case protocol_unknown_version:
+		BUG("unknown protocol version");
+	}
 
 	close(data->fd[1]);
 	close(data->fd[0]);
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 10/26] protocol: introduce enum protocol_version value protocol_v2
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (8 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 09/26] transport: store protocol version Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 11/26] serve: introduce git-serve Brandon Williams
                   ` (16 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Introduce protocol_v2, a new value for 'enum protocol_version'.
Subsequent patches will fill in the implementation of protocol_v2.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/fetch-pack.c   | 3 +++
 builtin/receive-pack.c | 6 ++++++
 builtin/send-pack.c    | 3 +++
 connect.c              | 3 +++
 protocol.c             | 2 ++
 protocol.h             | 1 +
 remote-curl.c          | 3 +++
 transport.c            | 9 +++++++++
 upload-pack.c          | 6 ++++++
 9 files changed, 36 insertions(+)

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 85d4faf76..f492e8abd 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 			   PACKET_READ_GENTLE_ON_EOF);
 
 	switch (discover_version(&reader)) {
+	case protocol_v2:
+		die("support for protocol v2 not implemented yet");
+		break;
 	case protocol_v1:
 	case protocol_v0:
 		get_remote_heads(&reader, &ref, 0, NULL, &shallow);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b7ce7c7f5..3656e94fd 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		unpack_limit = receive_unpack_limit;
 
 	switch (determine_protocol_version_server()) {
+	case protocol_v2:
+		/*
+		 * push support for protocol v2 has not been implemented yet,
+		 * so ignore the request to use v2 and fallback to using v0.
+		 */
+		break;
 	case protocol_v1:
 		/*
 		 * v1 is just the original protocol with a version string,
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 83cb125a6..b5427f75e 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 			   PACKET_READ_GENTLE_ON_EOF);
 
 	switch (discover_version(&reader)) {
+	case protocol_v2:
+		die("support for protocol v2 not implemented yet");
+		break;
 	case protocol_v1:
 	case protocol_v0:
 		get_remote_heads(&reader, &remote_refs, REF_NORMAL,
diff --git a/connect.c b/connect.c
index 1787b0212..caa539b75 100644
--- a/connect.c
+++ b/connect.c
@@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)
 
 	/* Maybe process capabilities here, at least for v2 */
 	switch (version) {
+	case protocol_v2:
+		die("support for protocol v2 not implemented yet");
+		break;
 	case protocol_v1:
 		/* Read the peeked version line */
 		packet_reader_read(reader);
diff --git a/protocol.c b/protocol.c
index 43012b7eb..5e636785d 100644
--- a/protocol.c
+++ b/protocol.c
@@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)
 		return protocol_v0;
 	else if (!strcmp(value, "1"))
 		return protocol_v1;
+	else if (!strcmp(value, "2"))
+		return protocol_v2;
 	else
 		return protocol_unknown_version;
 }
diff --git a/protocol.h b/protocol.h
index 1b2bc94a8..2ad35e433 100644
--- a/protocol.h
+++ b/protocol.h
@@ -5,6 +5,7 @@ enum protocol_version {
 	protocol_unknown_version = -1,
 	protocol_v0 = 0,
 	protocol_v1 = 1,
+	protocol_v2 = 2,
 };
 
 /*
diff --git a/remote-curl.c b/remote-curl.c
index 9f6d07683..dae8a4a48 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
 			   PACKET_READ_GENTLE_ON_EOF);
 
 	switch (discover_version(&reader)) {
+	case protocol_v2:
+		die("support for protocol v2 not implemented yet");
+		break;
 	case protocol_v1:
 	case protocol_v0:
 		get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
diff --git a/transport.c b/transport.c
index 2378dcb38..83d9dd1df 100644
--- a/transport.c
+++ b/transport.c
@@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 
 	data->version = discover_version(&reader);
 	switch (data->version) {
+	case protocol_v2:
+		die("support for protocol v2 not implemented yet");
+		break;
 	case protocol_v1:
 	case protocol_v0:
 		get_remote_heads(&reader, &refs,
@@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,
 		refs_tmp = get_refs_via_connect(transport, 0);
 
 	switch (data->version) {
+	case protocol_v2:
+		die("support for protocol v2 not implemented yet");
+		break;
 	case protocol_v1:
 	case protocol_v0:
 		refs = fetch_pack(&args, data->fd, data->conn,
@@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 		args.push_cert = SEND_PACK_PUSH_CERT_NEVER;
 
 	switch (data->version) {
+	case protocol_v2:
+		die("support for protocol v2 not implemented yet");
+		break;
 	case protocol_v1:
 	case protocol_v0:
 		ret = send_pack(&args, data->fd, data->conn, remote_refs,
diff --git a/upload-pack.c b/upload-pack.c
index 9a507ae53..2bc888fc1 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -1104,6 +1104,12 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
 	git_config(upload_pack_config, NULL);
 
 	switch (determine_protocol_version_server()) {
+	case protocol_v2:
+		/*
+		 * fetch support for protocol v2 has not been implemented yet,
+		 * so ignore the request to use v2 and fallback to using v0.
+		 */
+		break;
 	case protocol_v1:
 		/*
 		 * v1 is just the original protocol with a version string,
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 11/26] serve: introduce git-serve
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (9 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 10/26] protocol: introduce enum protocol_version value protocol_v2 Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-09 20:24   ` Jonathan Tan
  2018-01-03  0:18 ` [PATCH 12/26] ls-refs: introduce ls-refs server command Brandon Williams
                   ` (15 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Introduce git-serve, the base server for protocol version 2.

Protocol version 2 is intended to be a replacement for Git's current
wire protocol.  The intention is that it will be a simpler, less
wasteful protocol which can evolve over time.

Protocol version 2 improves upon version 1 by eliminating the initial
ref advertisement.  In its place a server will export a list of
capabilities and commands which it supports in a capability
advertisement.  A client can then request that a particular command be
executed by providing a number of capabilities and command specific
parameters.  At the completion of a command, a client can request that
another command be executed or can terminate the connection by sending a
flush packet.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 .gitignore                              |   1 +
 Documentation/technical/protocol-v2.txt |  91 ++++++++++++
 Makefile                                |   2 +
 builtin.h                               |   1 +
 builtin/serve.c                         |  30 ++++
 git.c                                   |   1 +
 serve.c                                 | 239 ++++++++++++++++++++++++++++++++
 serve.h                                 |  15 ++
 8 files changed, 380 insertions(+)
 create mode 100644 Documentation/technical/protocol-v2.txt
 create mode 100644 builtin/serve.c
 create mode 100644 serve.c
 create mode 100644 serve.h

diff --git a/.gitignore b/.gitignore
index 833ef3b0b..2d0450c26 100644
--- a/.gitignore
+++ b/.gitignore
@@ -140,6 +140,7 @@
 /git-rm
 /git-send-email
 /git-send-pack
+/git-serve
 /git-sh-i18n
 /git-sh-i18n--envsubst
 /git-sh-setup
diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
new file mode 100644
index 000000000..b87ba3816
--- /dev/null
+++ b/Documentation/technical/protocol-v2.txt
@@ -0,0 +1,91 @@
+ Git Wire Protocol, Version 2
+==============================
+
+This document presents a specification for a version 2 of Git's wire
+protocol.  Protocol v2 will improve upon v1 in the following ways:
+
+  * Instead of multiple service names, multiple commands will be
+    supported by a single service.
+  * Easily extendable as capabilities are moved into their own section
+    of the protocol, no longer being hidden behind a NUL byte and
+    limited by the size of a pkt-line (as there will be a single
+    capability per pkt-line).
+  * Separate out other information hidden behind NUL bytes (e.g. agent
+    string as a capability and symrefs can be requested using 'ls-refs')
+  * Reference advertisement will be omitted unless explicitly requested
+  * ls-refs command to explicitly request some refs
+
+ Detailed Design
+=================
+
+A client can request to speak protocol v2 by sending `version=2` in the
+side-channel `GIT_PROTOCOL` in the initial request to the server.
+
+In protocol v2 communication is command oriented.  When first contacting a
+server a list of capabilities will advertised.  Some of these capabilities
+will be commands which a client can request be executed.  Once a command
+has completed, a client can reuse the connection and request that other
+commands be executed.
+
+ Special Packets
+-----------------
+
+In protocol v2 these special packets will have the following semantics:
+
+  * '0000' Flush Packet (flush-pkt) - indicates the end of a message
+  * '0001' Delimiter Packet (delim-pkt) - separates sections of a message
+
+ Capability Advertisement
+--------------------------
+
+A server which decides to communicate (based on a request from a client)
+using protocol version 2, notifies the client by sending a version string
+in its initial response followed by an advertisement of its capabilities.
+Each capability is a key with an optional value.  Clients must ignore all
+unknown keys.  Semantics of unknown values are left to the definition of
+each key.  Some capabilities will describe commands which can be requested
+to be executed by the client.
+
+    capability-advertisement = protocol-version
+			       capability-list
+			       flush-pkt
+
+    protocol-version = PKT-LINE("version 2" LF)
+    capability-list = *capability
+    capability = PKT-LINE(key[=value] LF)
+
+    key = 1*CHAR
+    value = 1*CHAR
+    CHAR = 1*(ALPHA / DIGIT / "-" / "_")
+
+A client then responds to select the command it wants with any particular
+capabilities or arguments.  There is then an optional section where the
+client can provide any command specific parameters or queries.
+
+    command-request = command
+		      capability-list
+		      (command-args)
+		      flush-pkt
+    command = PKT-LINE("command=" key LF)
+    command-args = delim-pkt
+		   *arg
+    arg = 1*CHAR
+
+The server will then check to ensure that the client's request is
+comprised of a valid command as well as valid capabilities which were
+advertised.  If the request is valid the server will then execute the
+command.
+
+A particular command can last for as many rounds as are required to
+complete the service (multiple for negotiation during fetch or no
+additional trips in the case of ls-refs).
+
+When finished a client should send an empty request of just a flush-pkt to
+terminate the connection.
+
+ Commands in v2
+~~~~~~~~~~~~~~~~
+
+Commands are the core actions that a client wants to perform (fetch, push,
+etc).  Each command will be provided with a list capabilities and
+arguments as requested by a client.
diff --git a/Makefile b/Makefile
index e0740b452..5f3b5fe8b 100644
--- a/Makefile
+++ b/Makefile
@@ -876,6 +876,7 @@ LIB_OBJS += revision.o
 LIB_OBJS += run-command.o
 LIB_OBJS += send-pack.o
 LIB_OBJS += sequencer.o
+LIB_OBJS += serve.o
 LIB_OBJS += server-info.o
 LIB_OBJS += setup.o
 LIB_OBJS += sha1-array.o
@@ -1009,6 +1010,7 @@ BUILTIN_OBJS += builtin/rev-parse.o
 BUILTIN_OBJS += builtin/revert.o
 BUILTIN_OBJS += builtin/rm.o
 BUILTIN_OBJS += builtin/send-pack.o
+BUILTIN_OBJS += builtin/serve.o
 BUILTIN_OBJS += builtin/shortlog.o
 BUILTIN_OBJS += builtin/show-branch.o
 BUILTIN_OBJS += builtin/show-ref.o
diff --git a/builtin.h b/builtin.h
index f332a1257..3f3fdfc28 100644
--- a/builtin.h
+++ b/builtin.h
@@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);
 extern int cmd_revert(int argc, const char **argv, const char *prefix);
 extern int cmd_rm(int argc, const char **argv, const char *prefix);
 extern int cmd_send_pack(int argc, const char **argv, const char *prefix);
+extern int cmd_serve(int argc, const char **argv, const char *prefix);
 extern int cmd_shortlog(int argc, const char **argv, const char *prefix);
 extern int cmd_show(int argc, const char **argv, const char *prefix);
 extern int cmd_show_branch(int argc, const char **argv, const char *prefix);
diff --git a/builtin/serve.c b/builtin/serve.c
new file mode 100644
index 000000000..bb726786a
--- /dev/null
+++ b/builtin/serve.c
@@ -0,0 +1,30 @@
+#include "cache.h"
+#include "builtin.h"
+#include "parse-options.h"
+#include "serve.h"
+
+static char const * const grep_usage[] = {
+	N_("git serve [<options>]"),
+	NULL
+};
+
+int cmd_serve(int argc, const char **argv, const char *prefix)
+{
+	struct serve_options opts = SERVE_OPTIONS_INIT;
+
+	struct option options[] = {
+		OPT_BOOL(0, "stateless-rpc", &opts.stateless_rpc,
+			 N_("quit after a single request/response exchange")),
+		OPT_BOOL(0, "advertise-capabilities", &opts.advertise_capabilities,
+			 N_("exit immediately after advertising capabilities")),
+		OPT_END()
+	};
+
+	/* ignore all unknown cmdline switches for now */
+	argc = parse_options(argc, argv, prefix, options, grep_usage,
+			     PARSE_OPT_KEEP_DASHDASH |
+			     PARSE_OPT_KEEP_UNKNOWN);
+	serve(&opts);
+
+	return 0;
+}
diff --git a/git.c b/git.c
index f71073dc8..f85d682b6 100644
--- a/git.c
+++ b/git.c
@@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {
 	{ "revert", cmd_revert, RUN_SETUP | NEED_WORK_TREE },
 	{ "rm", cmd_rm, RUN_SETUP },
 	{ "send-pack", cmd_send_pack, RUN_SETUP },
+	{ "serve", cmd_serve, RUN_SETUP },
 	{ "shortlog", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },
 	{ "show", cmd_show, RUN_SETUP },
 	{ "show-branch", cmd_show_branch, RUN_SETUP },
diff --git a/serve.c b/serve.c
new file mode 100644
index 000000000..da8127775
--- /dev/null
+++ b/serve.c
@@ -0,0 +1,239 @@
+#include "cache.h"
+#include "repository.h"
+#include "config.h"
+#include "pkt-line.h"
+#include "version.h"
+#include "argv-array.h"
+#include "serve.h"
+
+static int always_advertise(struct repository *r,
+			    struct strbuf *value)
+{
+	return 1;
+}
+
+static int agent_advertise(struct repository *r,
+			   struct strbuf *value)
+{
+	if (value)
+		strbuf_addstr(value, git_user_agent_sanitized());
+	return 1;
+}
+
+struct protocol_capability {
+	const char *name; /* capability name */
+
+	/*
+	 * Function queried to see if a capability should be advertised.
+	 * Optionally a value can be specified by adding it to 'value'.
+	 */
+	int (*advertise)(struct repository *r, struct strbuf *value);
+
+	/*
+	 * Function called when a client requests the capability as a command.
+	 * The command request will be provided to the function via 'keys', the
+	 * capabilities requested, and 'args', the command specific parameters.
+	 *
+	 * This field should be NULL for capabilities which are not commands.
+	 */
+	int (*command)(struct repository *r,
+		       struct argv_array *keys,
+		       struct argv_array *args);
+};
+
+static struct protocol_capability capabilities[] = {
+	{ "agent", agent_advertise, NULL },
+	{ "stateless-rpc", always_advertise, NULL },
+};
+
+static void advertise_capabilities(void)
+{
+	struct strbuf capability = STRBUF_INIT;
+	struct strbuf value = STRBUF_INIT;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
+		struct protocol_capability *c = &capabilities[i];
+
+		if (c->advertise(the_repository, &value)) {
+			strbuf_addstr(&capability, c->name);
+
+			if (value.len) {
+				strbuf_addch(&capability, '=');
+				strbuf_addbuf(&capability, &value);
+			}
+
+			strbuf_addch(&capability, '\n');
+			packet_write(1, capability.buf, capability.len);
+		}
+
+		strbuf_reset(&capability);
+		strbuf_reset(&value);
+	}
+
+	packet_flush(1);
+	strbuf_release(&capability);
+	strbuf_release(&value);
+}
+
+static struct protocol_capability *get_capability(const char *key)
+{
+	int i;
+
+	if (!key)
+		return NULL;
+
+	for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
+		struct protocol_capability *c = &capabilities[i];
+		const char *out;
+		if (skip_prefix(key, c->name, &out) && (!*out || *out == '='))
+			return c;
+	}
+
+	return NULL;
+}
+
+static int is_valid_capability(const char *key)
+{
+	const struct protocol_capability *c = get_capability(key);
+
+	return c && c->advertise(the_repository, NULL);
+}
+
+static int is_command(const char *key, struct protocol_capability **command)
+{
+	const char *out;
+
+	if (skip_prefix(key, "command=", &out)) {
+		struct protocol_capability *cmd = get_capability(out);
+
+		if (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)
+			die("invalid cmd '%s'", out);
+		if (*command)
+			die("command already requested");
+
+		*command = cmd;
+		return 1;
+	}
+
+	return 0;
+}
+
+int has_capability(const struct argv_array *keys, const char *capability,
+		   const char **value)
+{
+	int i;
+	for (i = 0; i < keys->argc; i++) {
+		const char *out;
+		if (skip_prefix(keys->argv[i], capability, &out) &&
+		    (!*out || *out == '=')) {
+			if (value) {
+				if (*out == '=')
+					out++;
+				*value = out;
+			}
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+#define PROCESS_REQUEST_KEYS 0
+#define PROCESS_REQUEST_ARGS 1
+#define PROCESS_REQUEST_DONE 2
+
+static int process_request(void)
+{
+	int state = PROCESS_REQUEST_KEYS;
+	struct packet_reader reader;
+	struct argv_array keys = ARGV_ARRAY_INIT;
+	struct argv_array args = ARGV_ARRAY_INIT;
+	struct protocol_capability *command = NULL;
+
+	packet_reader_init(&reader, 0, NULL, 0,
+			   PACKET_READ_CHOMP_NEWLINE);
+
+	while (state != PROCESS_REQUEST_DONE) {
+		switch (packet_reader_read(&reader)) {
+		case PACKET_READ_EOF:
+			BUG("Should have already died when seeing EOF");
+		case PACKET_READ_NORMAL:
+			break;
+		case PACKET_READ_FLUSH:
+			state = PROCESS_REQUEST_DONE;
+			continue;
+		case PACKET_READ_DELIM:
+			if (state != PROCESS_REQUEST_KEYS)
+				die("protocol error");
+			state = PROCESS_REQUEST_ARGS;
+			/*
+			 * maybe include a check to make sure that a
+			 * command/capabilities were given.
+			 */
+			continue;
+		}
+
+		switch (state) {
+		case PROCESS_REQUEST_KEYS:
+			/* collect request; a sequence of keys and values */
+			if (is_command(reader.line, &command) ||
+			    is_valid_capability(reader.line))
+				argv_array_push(&keys, reader.line);
+			break;
+		case PROCESS_REQUEST_ARGS:
+			/* collect arguments for the requested command */
+			argv_array_push(&args, reader.line);
+			break;
+		case PROCESS_REQUEST_DONE:
+			continue;
+		default:
+			BUG("invalid state");
+		}
+	}
+
+	/*
+	 * If no command and no keys were given then the client wanted to
+	 * terminate the connection.
+	 */
+	if (!keys.argc && !args.argc)
+		return 1;
+
+	if (!command)
+		die("no command requested");
+
+	command->command(the_repository, &keys, &args);
+
+	argv_array_clear(&keys);
+	argv_array_clear(&args);
+	return 0;
+}
+
+/* Main serve loop for protocol version 2 */
+void serve(struct serve_options *options)
+{
+	if (options->advertise_capabilities || !options->stateless_rpc) {
+		/* serve by default supports v2 */
+		packet_write_fmt(1, "version 2\n");
+
+		advertise_capabilities();
+		/*
+		 * If only the list of capabilities was requested exit
+		 * immediately after advertising capabilities
+		 */
+		if (options->advertise_capabilities)
+			return;
+	}
+
+	/*
+	 * If stateless-rpc was requested then exit after
+	 * a single request/response exchange
+	 */
+	if (options->stateless_rpc) {
+		process_request();
+	} else {
+		for (;;)
+			if (process_request())
+				break;
+	}
+}
diff --git a/serve.h b/serve.h
new file mode 100644
index 000000000..fe65ba9f4
--- /dev/null
+++ b/serve.h
@@ -0,0 +1,15 @@
+#ifndef SERVE_H
+#define SERVE_H
+
+struct argv_array;
+extern int has_capability(const struct argv_array *keys, const char *capability,
+			  const char **value);
+
+struct serve_options {
+	unsigned advertise_capabilities;
+	unsigned stateless_rpc;
+};
+#define SERVE_OPTIONS_INIT { 0 }
+extern void serve(struct serve_options *options);
+
+#endif /* SERVE_H */
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 12/26] ls-refs: introduce ls-refs server command
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (10 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 11/26] serve: introduce git-serve Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-04  0:17   ` Stefan Beller
  2018-01-09 20:50   ` Jonathan Tan
  2018-01-03  0:18 ` [PATCH 13/26] connect: request remote refs using v2 Brandon Williams
                   ` (14 subsequent siblings)
  26 siblings, 2 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Introduce the ls-refs server command.  In protocol v2, the ls-refs
command is used to request the ref advertisement from the server.  Since
it is a command which can be requested (as opposed to mandatory in v1),
a client can sent a number of parameters in its request to limit the ref
advertisement based on provided ref-patterns.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/technical/protocol-v2.txt | 26 +++++++++
 Makefile                                |  1 +
 ls-refs.c                               | 97 +++++++++++++++++++++++++++++++++
 ls-refs.h                               |  9 +++
 serve.c                                 |  2 +
 5 files changed, 135 insertions(+)
 create mode 100644 ls-refs.c
 create mode 100644 ls-refs.h

diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
index b87ba3816..5f4d0e719 100644
--- a/Documentation/technical/protocol-v2.txt
+++ b/Documentation/technical/protocol-v2.txt
@@ -89,3 +89,29 @@ terminate the connection.
 Commands are the core actions that a client wants to perform (fetch, push,
 etc).  Each command will be provided with a list capabilities and
 arguments as requested by a client.
+
+ Ls-refs
+---------
+
+Ls-refs is the command used to request a reference advertisement in v2.
+Unlike the current reference advertisement, ls-refs takes in parameters
+which can be used to limit the refs sent from the server.
+
+Ls-ref takes in the following parameters wraped in packet-lines:
+
+  symrefs: In addition to the object pointed by it, show the underlying
+	   ref pointed by it when showing a symbolic ref.
+  peel: Show peeled tags.
+  ref-pattern <pattern>: When specified, only references matching the
+			 given patterns are displayed.
+
+The output of ls-refs is as follows:
+
+    output = *ref
+	     flush-pkt
+    ref = PKT-LINE((tip | peeled) LF)
+    tip = obj-id SP refname (SP symref-target)
+    peeled = obj-id SP refname "^{}"
+
+    symref = PKT-LINE("symref" SP symbolic-ref SP resolved-ref LF)
+    shallow = PKT-LINE("shallow" SP obj-id LF)
diff --git a/Makefile b/Makefile
index 5f3b5fe8b..152a73bec 100644
--- a/Makefile
+++ b/Makefile
@@ -820,6 +820,7 @@ LIB_OBJS += list-objects-filter-options.o
 LIB_OBJS += ll-merge.o
 LIB_OBJS += lockfile.o
 LIB_OBJS += log-tree.o
+LIB_OBJS += ls-refs.o
 LIB_OBJS += mailinfo.o
 LIB_OBJS += mailmap.o
 LIB_OBJS += match-trees.o
diff --git a/ls-refs.c b/ls-refs.c
new file mode 100644
index 000000000..ac4904a40
--- /dev/null
+++ b/ls-refs.c
@@ -0,0 +1,97 @@
+#include "cache.h"
+#include "repository.h"
+#include "refs.h"
+#include "remote.h"
+#include "argv-array.h"
+#include "ls-refs.h"
+#include "pkt-line.h"
+
+struct ls_refs_data {
+	unsigned peel;
+	unsigned symrefs;
+	struct argv_array patterns;
+};
+
+/*
+ * Check if one of the patterns matches the tail part of the ref.
+ * If no patterns were provided, all refs match.
+ */
+static int ref_match(const struct argv_array *patterns, const char *refname)
+{
+	char *pathbuf;
+	int i;
+
+	if (!patterns->argc)
+		return 1; /* no restriction */
+
+	pathbuf = xstrfmt("/%s", refname);
+	for (i = 0; i < patterns->argc; i++) {
+		if (!wildmatch(patterns->argv[i], pathbuf, 0)) {
+			free(pathbuf);
+			return 1;
+		}
+	}
+	free(pathbuf);
+	return 0;
+}
+
+static int send_ref(const char *refname, const struct object_id *oid,
+		    int flag, void *cb_data)
+{
+	struct ls_refs_data *data = cb_data;
+	const char *refname_nons = strip_namespace(refname);
+	struct strbuf refline = STRBUF_INIT;
+
+	if (!ref_match(&data->patterns, refname))
+		return 0;
+
+	strbuf_addf(&refline, "%s %s", oid_to_hex(oid), refname_nons);
+	if (data->symrefs && flag & REF_ISSYMREF) {
+		struct object_id unused;
+		const char *symref_target = resolve_ref_unsafe(refname, 0,
+							       &unused,
+							       &flag);
+
+		if (!symref_target)
+			die("'%s' is a symref but it is not?", refname);
+
+		strbuf_addf(&refline, " %s", symref_target);
+	}
+
+	strbuf_addch(&refline, '\n');
+
+	packet_write(1, refline.buf, refline.len);
+	if (data->peel) {
+		struct object_id peeled;
+		if (!peel_ref(refname, &peeled))
+			packet_write_fmt(1, "%s %s^{}\n", oid_to_hex(&peeled),
+					 refname_nons);
+	}
+
+	strbuf_release(&refline);
+	return 0;
+}
+
+int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)
+{
+	int i;
+	struct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };
+
+	for (i = 0; i < args->argc; i++) {
+		const char *arg = args->argv[i];
+		const char *out;
+
+		if (!strcmp("peel", arg))
+			data.peel = 1;
+		else if (!strcmp("symrefs", arg))
+			data.symrefs = 1;
+		else if (skip_prefix(arg, "ref-pattern ", &out))
+			argv_array_pushf(&data.patterns, "*/%s", out);
+	}
+
+	head_ref_namespaced(send_ref, &data);
+	for_each_namespaced_ref(send_ref, &data);
+	packet_flush(1);
+	argv_array_clear(&data.patterns);
+	return 0;
+}
diff --git a/ls-refs.h b/ls-refs.h
new file mode 100644
index 000000000..9e4c57bfe
--- /dev/null
+++ b/ls-refs.h
@@ -0,0 +1,9 @@
+#ifndef LS_REFS_H
+#define LS_REFS_H
+
+struct repository;
+struct argv_array;
+extern int ls_refs(struct repository *r, struct argv_array *keys,
+		   struct argv_array *args);
+
+#endif /* LS_REFS_H */
diff --git a/serve.c b/serve.c
index da8127775..88d548410 100644
--- a/serve.c
+++ b/serve.c
@@ -4,6 +4,7 @@
 #include "pkt-line.h"
 #include "version.h"
 #include "argv-array.h"
+#include "ls-refs.h"
 #include "serve.h"
 
 static int always_advertise(struct repository *r,
@@ -44,6 +45,7 @@ struct protocol_capability {
 static struct protocol_capability capabilities[] = {
 	{ "agent", agent_advertise, NULL },
 	{ "stateless-rpc", always_advertise, NULL },
+	{ "ls-refs", always_advertise, ls_refs },
 };
 
 static void advertise_capabilities(void)
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 13/26] connect: request remote refs using v2
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (11 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 12/26] ls-refs: introduce ls-refs server command Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-09 22:24   ` Jonathan Tan
  2018-01-03  0:18 ` [PATCH 14/26] transport: convert get_refs_list to take a list of ref patterns Brandon Williams
                   ` (13 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Teach the client to be able to request a remote's refs using protocol
v2.  This is done by having a client issue a 'ls-refs' request to a v2
server.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 connect.c              | 101 ++++++++++++++++++++++++++++++++++++++++++++++++-
 remote.h               |   4 ++
 t/t5701-protocol-v2.sh |  28 ++++++++++++++
 transport.c            |   2 +-
 upload-pack.c          |   9 +++--
 5 files changed, 138 insertions(+), 6 deletions(-)
 create mode 100755 t/t5701-protocol-v2.sh

diff --git a/connect.c b/connect.c
index caa539b75..9badd403f 100644
--- a/connect.c
+++ b/connect.c
@@ -12,9 +12,11 @@
 #include "sha1-array.h"
 #include "transport.h"
 #include "strbuf.h"
+#include "version.h"
 #include "protocol.h"
 
 static char *server_capabilities;
+static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;
 static const char *parse_feature_value(const char *, const char *, int *);
 
 static int check_ref(const char *name, unsigned int flags)
@@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)
 		      "and the repository exists."));
 }
 
+static int server_supports_v2(const char *c, int die_on_error)
+{
+	int i;
+
+	for (i = 0; i < server_capabilities_v2.argc; i++) {
+		const char *out;
+		if (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&
+		    (!*out || *out == '='))
+			return 1;
+	}
+
+	if (die_on_error)
+		die("server doesn't support '%s'", c);
+
+	return 0;
+}
+
+static void process_capabilities_v2(struct packet_reader *reader)
+{
+	while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
+		argv_array_push(&server_capabilities_v2, reader->line);
+	}
+
+	if (reader->status != PACKET_READ_FLUSH)
+		die("protocol error");
+}
+
 enum protocol_version discover_version(struct packet_reader *reader)
 {
 	enum protocol_version version = protocol_unknown_version;
@@ -85,7 +114,7 @@ enum protocol_version discover_version(struct packet_reader *reader)
 	/* Maybe process capabilities here, at least for v2 */
 	switch (version) {
 	case protocol_v2:
-		die("support for protocol v2 not implemented yet");
+		process_capabilities_v2(reader);
 		break;
 	case protocol_v1:
 		/* Read the peeked version line */
@@ -292,6 +321,76 @@ struct ref **get_remote_heads(struct packet_reader *reader,
 	return list;
 }
 
+static int process_ref_v2(const char *line, struct ref ***list)
+{
+	int ret = 1;
+	int i = 0;
+	struct object_id old_oid;
+	struct ref *ref;
+	struct string_list line_sections = STRING_LIST_INIT_DUP;
+
+	if (string_list_split(&line_sections, line, ' ', -1) < 2) {
+		ret = 0;
+		goto out;
+	}
+
+	if (get_oid_hex(line_sections.items[i++].string, &old_oid)) {
+		ret = 0;
+		goto out;
+	}
+
+	ref = alloc_ref(line_sections.items[i++].string);
+
+	if (i < line_sections.nr)
+		ref->symref = xstrdup(line_sections.items[i++].string);
+
+	oidcpy(&ref->old_oid, &old_oid);
+	**list = ref;
+	*list = &ref->next;
+
+out:
+	string_list_clear(&line_sections, 0);
+	return ret;
+}
+
+struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
+			     struct ref **list, int for_push,
+			     const struct argv_array *ref_patterns)
+{
+	int i;
+	*list = NULL;
+
+	/* Check that the server supports the ls-refs command */
+	/* Issue request for ls-refs */
+	if (server_supports_v2("ls-refs", 1))
+		packet_write_fmt(fd_out, "command=ls-refs\n");
+
+	if (server_supports_v2("agent", 0))
+	    packet_write_fmt(fd_out, "agent=%s", git_user_agent_sanitized());
+
+	packet_delim(fd_out);
+	/* When pushing we don't want to request the peeled tags */
+	if (!for_push)
+		packet_write_fmt(fd_out, "peel\n");
+	packet_write_fmt(fd_out, "symrefs\n");
+	for (i = 0; ref_patterns && i < ref_patterns->argc; i++) {
+		packet_write_fmt(fd_out, "ref-pattern %s\n",
+				 ref_patterns->argv[i]);
+	}
+	packet_flush(fd_out);
+
+	/* Process response from server */
+	while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
+		if (!process_ref_v2(reader->line, &list))
+			die("invalid ls-refs response: %s", reader->line);
+	}
+
+	if (reader->status != PACKET_READ_FLUSH)
+		die("protocol error");
+
+	return list;
+}
+
 static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)
 {
 	int len;
diff --git a/remote.h b/remote.h
index 2016461df..21d0c776c 100644
--- a/remote.h
+++ b/remote.h
@@ -151,10 +151,14 @@ void free_refs(struct ref *ref);
 
 struct oid_array;
 struct packet_reader;
+struct argv_array;
 extern struct ref **get_remote_heads(struct packet_reader *reader,
 				     struct ref **list, unsigned int flags,
 				     struct oid_array *extra_have,
 				     struct oid_array *shallow_points);
+extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
+				    struct ref **list, int for_push,
+				    const struct argv_array *ref_patterns);
 
 int resolve_remote_symref(struct ref *ref, struct ref *list);
 int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);
diff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh
new file mode 100755
index 000000000..4bf4d61ac
--- /dev/null
+++ b/t/t5701-protocol-v2.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+
+test_description='test git wire-protocol version 2'
+
+TEST_NO_CREATE_REPO=1
+
+. ./test-lib.sh
+
+# Test protocol v2 with 'file://' transport
+#
+test_expect_success 'create repo to be served by file:// transport' '
+	git init file_parent &&
+	test_commit -C file_parent one
+'
+
+test_expect_success 'list refs with file:// using protocol v2' '
+	GIT_TRACE_PACKET=1 git -c protocol.version=2 \
+		ls-remote --symref "file://$(pwd)/file_parent" >actual 2>log &&
+
+	# Server responded using protocol v2
+	cat log &&
+	grep "git< version 2" log &&
+
+	git ls-remote --symref "file://$(pwd)/file_parent" >expect &&
+	test_cmp actual expect
+'
+
+test_done
diff --git a/transport.c b/transport.c
index 83d9dd1df..ffc6b2614 100644
--- a/transport.c
+++ b/transport.c
@@ -204,7 +204,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 	data->version = discover_version(&reader);
 	switch (data->version) {
 	case protocol_v2:
-		die("support for protocol v2 not implemented yet");
+		get_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);
 		break;
 	case protocol_v1:
 	case protocol_v0:
diff --git a/upload-pack.c b/upload-pack.c
index 2bc888fc1..2ca60d27c 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -19,6 +19,7 @@
 #include "argv-array.h"
 #include "prio-queue.h"
 #include "protocol.h"
+#include "serve.h"
 
 static const char * const upload_pack_usage[] = {
 	N_("git upload-pack [<options>] <dir>"),
@@ -1071,6 +1072,7 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
 {
 	const char *dir;
 	int strict = 0;
+	struct serve_options opts = SERVE_OPTIONS_INIT;
 	struct option options[] = {
 		OPT_BOOL(0, "stateless-rpc", &stateless_rpc,
 			 N_("quit after a single request/response exchange")),
@@ -1105,10 +1107,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
 
 	switch (determine_protocol_version_server()) {
 	case protocol_v2:
-		/*
-		 * fetch support for protocol v2 has not been implemented yet,
-		 * so ignore the request to use v2 and fallback to using v0.
-		 */
+		opts.advertise_capabilities = advertise_refs;
+		opts.stateless_rpc = stateless_rpc;
+		serve(&opts);
 		break;
 	case protocol_v1:
 		/*
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 14/26] transport: convert get_refs_list to take a list of ref patterns
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (12 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 13/26] connect: request remote refs using v2 Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 15/26] transport: convert transport_get_remote_refs " Brandon Williams
                   ` (12 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Convert the 'struct transport' virtual function 'get_refs_list()' to
optionally take an argv_array of ref patterns.  When communicating with
a server using protocol v2 these ref patterns can be sent when
requesting a listing of their refs allowing the server to filter the
refs it sends based on the sent patterns.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 transport-helper.c   |  5 +++--
 transport-internal.h |  4 +++-
 transport.c          | 16 +++++++++-------
 3 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/transport-helper.c b/transport-helper.c
index 508015023..4c334b5ee 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -1026,7 +1026,8 @@ static int has_attribute(const char *attrs, const char *attr) {
 	}
 }
 
-static struct ref *get_refs_list(struct transport *transport, int for_push)
+static struct ref *get_refs_list(struct transport *transport, int for_push,
+				 const struct argv_array *ref_patterns)
 {
 	struct helper_data *data = transport->data;
 	struct child_process *helper;
@@ -1039,7 +1040,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)
 
 	if (process_connect(transport, for_push)) {
 		do_take_over(transport);
-		return transport->vtable->get_refs_list(transport, for_push);
+		return transport->vtable->get_refs_list(transport, for_push, ref_patterns);
 	}
 
 	if (data->push && for_push)
diff --git a/transport-internal.h b/transport-internal.h
index 3c1a29d72..a67657ce3 100644
--- a/transport-internal.h
+++ b/transport-internal.h
@@ -3,6 +3,7 @@
 
 struct ref;
 struct transport;
+struct argv_array;
 
 struct transport_vtable {
 	/**
@@ -21,7 +22,8 @@ struct transport_vtable {
 	 * the ref without a huge amount of effort, it should store it
 	 * in the ref's old_sha1 field; otherwise it should be all 0.
 	 **/
-	struct ref *(*get_refs_list)(struct transport *transport, int for_push);
+	struct ref *(*get_refs_list)(struct transport *transport, int for_push,
+				     const struct argv_array *ref_patterns);
 
 	/**
 	 * Fetch the objects for the given refs. Note that this gets
diff --git a/transport.c b/transport.c
index ffc6b2614..c54a44630 100644
--- a/transport.c
+++ b/transport.c
@@ -72,7 +72,7 @@ struct bundle_transport_data {
 	struct bundle_header header;
 };
 
-static struct ref *get_refs_from_bundle(struct transport *transport, int for_push)
+static struct ref *get_refs_from_bundle(struct transport *transport, int for_push, const struct argv_array *ref_patterns)
 {
 	struct bundle_transport_data *data = transport->data;
 	struct ref *result = NULL;
@@ -189,7 +189,8 @@ static int connect_setup(struct transport *transport, int for_push)
 	return 0;
 }
 
-static struct ref *get_refs_via_connect(struct transport *transport, int for_push)
+static struct ref *get_refs_via_connect(struct transport *transport, int for_push,
+					const struct argv_array *ref_patterns)
 {
 	struct git_transport_data *data = transport->data;
 	struct ref *refs = NULL;
@@ -204,7 +205,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 	data->version = discover_version(&reader);
 	switch (data->version) {
 	case protocol_v2:
-		get_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);
+		get_remote_refs(data->fd[1], &reader, &refs, for_push,
+				ref_patterns);
 		break;
 	case protocol_v1:
 	case protocol_v0:
@@ -250,7 +252,7 @@ static int fetch_refs_via_pack(struct transport *transport,
 	args.update_shallow = data->options.update_shallow;
 
 	if (!data->got_remote_heads)
-		refs_tmp = get_refs_via_connect(transport, 0);
+		refs_tmp = get_refs_via_connect(transport, 0, NULL);
 
 	switch (data->version) {
 	case protocol_v2:
@@ -568,7 +570,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
 	int ret = 0;
 
 	if (!data->got_remote_heads)
-		get_refs_via_connect(transport, 1);
+		get_refs_via_connect(transport, 1, NULL);
 
 	memset(&args, 0, sizeof(args));
 	args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
@@ -1028,7 +1030,7 @@ int transport_push(struct transport *transport,
 		if (check_push_refs(local_refs, refspec_nr, refspec) < 0)
 			return -1;
 
-		remote_refs = transport->vtable->get_refs_list(transport, 1);
+		remote_refs = transport->vtable->get_refs_list(transport, 1, NULL);
 
 		if (flags & TRANSPORT_PUSH_ALL)
 			match_flags |= MATCH_REFS_ALL;
@@ -1137,7 +1139,7 @@ int transport_push(struct transport *transport,
 const struct ref *transport_get_remote_refs(struct transport *transport)
 {
 	if (!transport->got_remote_refs) {
-		transport->remote_refs = transport->vtable->get_refs_list(transport, 0);
+		transport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);
 		transport->got_remote_refs = 1;
 	}
 
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 15/26] transport: convert transport_get_remote_refs to take a list of ref patterns
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (13 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 14/26] transport: convert get_refs_list to take a list of ref patterns Brandon Williams
@ 2018-01-03  0:18 ` " Brandon Williams
  2018-01-03  0:18 ` [PATCH 16/26] ls-remote: pass ref patterns when requesting a remote's refs Brandon Williams
                   ` (11 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Convert 'transport_get_remote_refs()' to optionally take a list of ref
patterns.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/clone.c     | 2 +-
 builtin/fetch.c     | 4 ++--
 builtin/ls-remote.c | 2 +-
 builtin/remote.c    | 2 +-
 transport.c         | 7 +++++--
 transport.h         | 3 ++-
 6 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index 2da71db10..4db3079ac 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1104,7 +1104,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	if (transport->smart_options && !deepen)
 		transport->smart_options->check_self_contained_and_connected = 1;
 
-	refs = transport_get_remote_refs(transport);
+	refs = transport_get_remote_refs(transport, NULL);
 
 	if (refs) {
 		mapped_refs = wanted_peer_refs(refs, refspec);
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 7bbcd26fa..850382f55 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -250,7 +250,7 @@ static void find_non_local_tags(struct transport *transport,
 	struct string_list_item *item = NULL;
 
 	for_each_ref(add_existing, &existing_refs);
-	for (ref = transport_get_remote_refs(transport); ref; ref = ref->next) {
+	for (ref = transport_get_remote_refs(transport, NULL); ref; ref = ref->next) {
 		if (!starts_with(ref->name, "refs/tags/"))
 			continue;
 
@@ -336,7 +336,7 @@ static struct ref *get_ref_map(struct transport *transport,
 	/* opportunistically-updated references: */
 	struct ref *orefs = NULL, **oref_tail = &orefs;
 
-	const struct ref *remote_refs = transport_get_remote_refs(transport);
+	const struct ref *remote_refs = transport_get_remote_refs(transport, NULL);
 
 	if (refspec_count) {
 		struct refspec *fetch_refspec;
diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c
index c4be98ab9..c6e9847c5 100644
--- a/builtin/ls-remote.c
+++ b/builtin/ls-remote.c
@@ -96,7 +96,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 	if (uploadpack != NULL)
 		transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);
 
-	ref = transport_get_remote_refs(transport);
+	ref = transport_get_remote_refs(transport, NULL);
 	if (transport_disconnect(transport))
 		return 1;
 
diff --git a/builtin/remote.c b/builtin/remote.c
index d95bf904c..d0b6ff6e2 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -862,7 +862,7 @@ static int get_remote_ref_states(const char *name,
 	if (query) {
 		transport = transport_get(states->remote, states->remote->url_nr > 0 ?
 			states->remote->url[0] : NULL);
-		remote_refs = transport_get_remote_refs(transport);
+		remote_refs = transport_get_remote_refs(transport, NULL);
 		transport_disconnect(transport);
 
 		states->queried = 1;
diff --git a/transport.c b/transport.c
index c54a44630..dfc603b36 100644
--- a/transport.c
+++ b/transport.c
@@ -1136,10 +1136,13 @@ int transport_push(struct transport *transport,
 	return 1;
 }
 
-const struct ref *transport_get_remote_refs(struct transport *transport)
+const struct ref *transport_get_remote_refs(struct transport *transport,
+					    const struct argv_array *ref_patterns)
 {
 	if (!transport->got_remote_refs) {
-		transport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);
+		transport->remote_refs =
+			transport->vtable->get_refs_list(transport, 0,
+							 ref_patterns);
 		transport->got_remote_refs = 1;
 	}
 
diff --git a/transport.h b/transport.h
index 731c78b67..4b656f315 100644
--- a/transport.h
+++ b/transport.h
@@ -178,7 +178,8 @@ int transport_push(struct transport *connection,
 		   int refspec_nr, const char **refspec, int flags,
 		   unsigned int * reject_reasons);
 
-const struct ref *transport_get_remote_refs(struct transport *transport);
+const struct ref *transport_get_remote_refs(struct transport *transport,
+					    const struct argv_array *ref_patterns);
 
 int transport_fetch_refs(struct transport *transport, struct ref *refs);
 void transport_unlock_pack(struct transport *transport);
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 16/26] ls-remote: pass ref patterns when requesting a remote's refs
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (14 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 15/26] transport: convert transport_get_remote_refs " Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 17/26] fetch: pass ref patterns when fetching Brandon Williams
                   ` (10 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Construct an argv_array of the ref patterns supplied via the command
line and pass them to 'transport_get_remote_refs()' to be used when
communicating protocol v2 so that the server can limit the ref
advertisement based on the supplied patterns.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/ls-remote.c    | 7 +++++--
 t/t5701-protocol-v2.sh | 8 ++++++++
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c
index c6e9847c5..caf1051f3 100644
--- a/builtin/ls-remote.c
+++ b/builtin/ls-remote.c
@@ -43,6 +43,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 	int show_symref_target = 0;
 	const char *uploadpack = NULL;
 	const char **pattern = NULL;
+	struct argv_array ref_patterns = ARGV_ARRAY_INIT;
 
 	struct remote *remote;
 	struct transport *transport;
@@ -74,8 +75,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 	if (argc > 1) {
 		int i;
 		pattern = xcalloc(argc, sizeof(const char *));
-		for (i = 1; i < argc; i++)
+		for (i = 1; i < argc; i++) {
 			pattern[i - 1] = xstrfmt("*/%s", argv[i]);
+			argv_array_push(&ref_patterns, argv[i]);
+		}
 	}
 
 	remote = remote_get(dest);
@@ -96,7 +99,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 	if (uploadpack != NULL)
 		transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);
 
-	ref = transport_get_remote_refs(transport, NULL);
+	ref = transport_get_remote_refs(transport, &ref_patterns);
 	if (transport_disconnect(transport))
 		return 1;
 
diff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh
index 4bf4d61ac..7d8aeb766 100755
--- a/t/t5701-protocol-v2.sh
+++ b/t/t5701-protocol-v2.sh
@@ -25,4 +25,12 @@ test_expect_success 'list refs with file:// using protocol v2' '
 	test_cmp actual expect
 '
 
+test_expect_success 'ref advertisment is filtered with ls-remote using protocol v2' '
+	GIT_TRACE_PACKET=1 git -c protocol.version=2 \
+		ls-remote "file://$(pwd)/file_parent" master 2>log &&
+
+	grep "ref-pattern master" log &&
+	! grep "refs/tags/" log
+'
+
 test_done
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 17/26] fetch: pass ref patterns when fetching
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (15 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 16/26] ls-remote: pass ref patterns when requesting a remote's refs Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 18/26] push: pass ref patterns when pushing Brandon Williams
                   ` (9 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Construct a list of ref patterns to be passed to
'transport_get_remote_refs()' from the refspec to be used during the
fetch.  This list of ref patterns will be used to allow the server to
filter the ref advertisement when communicating using protocol v2.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/fetch.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 850382f55..8128450bf 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -332,11 +332,21 @@ static struct ref *get_ref_map(struct transport *transport,
 	struct ref *rm;
 	struct ref *ref_map = NULL;
 	struct ref **tail = &ref_map;
+	struct argv_array ref_patterns = ARGV_ARRAY_INIT;
 
 	/* opportunistically-updated references: */
 	struct ref *orefs = NULL, **oref_tail = &orefs;
 
-	const struct ref *remote_refs = transport_get_remote_refs(transport, NULL);
+	const struct ref *remote_refs;
+
+	for (i = 0; i < refspec_count; i++) {
+		if (!refspecs[i].exact_sha1)
+			argv_array_push(&ref_patterns, refspecs[i].src);
+	}
+
+	remote_refs = transport_get_remote_refs(transport, &ref_patterns);
+
+	argv_array_clear(&ref_patterns);
 
 	if (refspec_count) {
 		struct refspec *fetch_refspec;
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 18/26] push: pass ref patterns when pushing
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (16 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 17/26] fetch: pass ref patterns when fetching Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 19/26] upload-pack: introduce fetch server command Brandon Williams
                   ` (8 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Construct a list of ref patterns to be passed to 'get_refs_list()' from
the refspec to be used during the push.  This list of ref patterns will
be used to allow the server to filter the ref advertisement when
communicating using protocol v2.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 transport.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/transport.c b/transport.c
index dfc603b36..6ea3905e3 100644
--- a/transport.c
+++ b/transport.c
@@ -1026,11 +1026,26 @@ int transport_push(struct transport *transport,
 		int porcelain = flags & TRANSPORT_PUSH_PORCELAIN;
 		int pretend = flags & TRANSPORT_PUSH_DRY_RUN;
 		int push_ret, ret, err;
+		struct refspec *tmp_rs;
+		struct argv_array ref_patterns = ARGV_ARRAY_INIT;
+		int i;
 
 		if (check_push_refs(local_refs, refspec_nr, refspec) < 0)
 			return -1;
 
-		remote_refs = transport->vtable->get_refs_list(transport, 1, NULL);
+		tmp_rs = parse_push_refspec(refspec_nr, refspec);
+		for (i = 0; i < refspec_nr; i++) {
+			if (tmp_rs[i].dst)
+				argv_array_push(&ref_patterns, tmp_rs[i].dst);
+			else if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)
+				argv_array_push(&ref_patterns, tmp_rs[i].src);
+		}
+
+		remote_refs = transport->vtable->get_refs_list(transport, 1,
+							       &ref_patterns);
+
+		argv_array_clear(&ref_patterns);
+		free_refspec(refspec_nr, tmp_rs);
 
 		if (flags & TRANSPORT_PUSH_ALL)
 			match_flags |= MATCH_REFS_ALL;
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 19/26] upload-pack: introduce fetch server command
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (17 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 18/26] push: pass ref patterns when pushing Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-04  1:07   ` Stefan Beller
  2018-01-03  0:18 ` [PATCH 20/26] fetch-pack: perform a fetch using v2 Brandon Williams
                   ` (7 subsequent siblings)
  26 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Introduce the 'fetch' server command.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/technical/protocol-v2.txt |  14 ++
 serve.c                                 |   2 +
 upload-pack.c                           | 290 ++++++++++++++++++++++++++++++++
 upload-pack.h                           |   9 +
 4 files changed, 315 insertions(+)
 create mode 100644 upload-pack.h

diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
index 5f4d0e719..2a8e2f226 100644
--- a/Documentation/technical/protocol-v2.txt
+++ b/Documentation/technical/protocol-v2.txt
@@ -115,3 +115,17 @@ The output of ls-refs is as follows:
 
     symref = PKT-LINE("symref" SP symbolic-ref SP resolved-ref LF)
     shallow = PKT-LINE("shallow" SP obj-id LF)
+
+ Fetch
+-------
+
+Fetch will need to be a modified version of the v1 fetch protocol.  Some
+potential areas for improvement are: Ref-in-want, CDN offloading,
+Fetch-options.
+
+Since we'll have an 'ls-ref' service we can eliminate the need of fetch
+to perform a ref-advertisement, instead a client can run the 'ls-refs'
+service first, in order to find out what refs the server has, and then
+request those refs directly using the fetch service.
+
+//TODO Flesh out the design
diff --git a/serve.c b/serve.c
index 88d548410..ca3bb7190 100644
--- a/serve.c
+++ b/serve.c
@@ -6,6 +6,7 @@
 #include "argv-array.h"
 #include "ls-refs.h"
 #include "serve.h"
+#include "upload-pack.h"
 
 static int always_advertise(struct repository *r,
 			    struct strbuf *value)
@@ -46,6 +47,7 @@ static struct protocol_capability capabilities[] = {
 	{ "agent", agent_advertise, NULL },
 	{ "stateless-rpc", always_advertise, NULL },
 	{ "ls-refs", always_advertise, ls_refs },
+	{ "fetch", always_advertise, upload_pack_v2 },
 };
 
 static void advertise_capabilities(void)
diff --git a/upload-pack.c b/upload-pack.c
index 2ca60d27c..c41f6f528 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -20,6 +20,7 @@
 #include "prio-queue.h"
 #include "protocol.h"
 #include "serve.h"
+#include "upload-pack.h"
 
 static const char * const upload_pack_usage[] = {
 	N_("git upload-pack [<options>] <dir>"),
@@ -1040,6 +1041,295 @@ static void upload_pack(void)
 	}
 }
 
+struct upload_pack_data {
+	struct object_array wants;
+	struct oid_array haves;
+
+	unsigned stateless_rpc : 1;
+
+	unsigned use_thin_pack : 1;
+	unsigned use_ofs_delta : 1;
+	unsigned no_progress : 1;
+	unsigned use_include_tag : 1;
+	unsigned done : 1;
+};
+
+#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }
+
+static void upload_pack_data_clear(struct upload_pack_data *data)
+{
+	object_array_clear(&data->wants);
+	oid_array_clear(&data->haves);
+}
+
+static int parse_want(const char *line)
+{
+	const char *arg;
+	if (skip_prefix(line, "want ", &arg)) {
+		struct object_id oid;
+		struct object *o;
+
+		if (get_oid_hex(arg, &oid))
+			die("git upload-pack: protocol error, "
+			    "expected to get oid, not '%s'", line);
+
+		o = parse_object(&oid);
+		if (!o) {
+			packet_write_fmt(1,
+					 "ERR upload-pack: not our ref %s",
+					 oid_to_hex(&oid));
+			die("git upload-pack: not our ref %s",
+			    oid_to_hex(&oid));
+		}
+
+		if (!(o->flags & WANTED)) {
+			o->flags |= WANTED;
+			add_object_array(o, NULL, &want_obj);
+		}
+
+		return 1;
+	}
+
+	return 0;
+}
+
+static int parse_have(const char *line, struct oid_array *haves)
+{
+	const char *arg;
+	if (skip_prefix(line, "have ", &arg)) {
+		struct object_id oid;
+
+		if (get_oid_hex(arg, &oid))
+			die("git upload-pack: expected SHA1 object, got '%s'", arg);
+		oid_array_append(haves, &oid);
+		return 1;
+	}
+
+	return 0;
+}
+
+static void process_args(struct argv_array *args, struct upload_pack_data *data)
+{
+	int i;
+
+	for (i = 0; i < args->argc; i++) {
+		const char *arg = args->argv[i];
+
+		/* process want */
+		if (parse_want(arg))
+			continue;
+		/* process have line */
+		if (parse_have(arg, &data->haves))
+			continue;
+
+		/* process args like thin-pack */
+		if (!strcmp(arg, "thin-pack")) {
+			use_thin_pack = 1;
+			continue;
+		}
+		if (!strcmp(arg, "ofs-delta")) {
+			use_ofs_delta = 1;
+			continue;
+		}
+		if (!strcmp(arg, "no-progress")) {
+			no_progress = 1;
+			continue;
+		}
+		if (!strcmp(arg, "include-tag")) {
+			use_include_tag = 1;
+			continue;
+		}
+		if (!strcmp(arg, "done")) {
+			data->done = 1;
+			continue;
+		}
+
+		/* ignore unknown lines maybe? */
+		die("unexpect line: '%s'", arg);
+	}
+}
+
+static void read_haves(struct upload_pack_data *data)
+{
+	struct packet_reader reader;
+	packet_reader_init(&reader, 0, NULL, 0,
+			   PACKET_READ_CHOMP_NEWLINE);
+
+	while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {
+
+		if (parse_have(reader.line, &data->haves))
+			continue;
+		if (!strcmp(reader.line, "done")) {
+			data->done = 1;
+			continue;
+		}
+	}
+	if (reader.status != PACKET_READ_FLUSH)
+		die("ERROR");
+}
+
+static int process_haves(struct oid_array *haves, struct oid_array *common)
+{
+	int i;
+
+	/* Process haves */
+	for (i = 0; i < haves->nr; i++) {
+		const struct object_id *oid = &haves->oid[i];
+		struct object *o;
+		int we_knew_they_have = 0;
+
+		if (!has_object_file(oid))
+			continue;
+
+		oid_array_append(common, oid);
+
+		o = parse_object(oid);
+		if (!o)
+			die("oops (%s)", oid_to_hex(oid));
+		if (o->type == OBJ_COMMIT) {
+			struct commit_list *parents;
+			struct commit *commit = (struct commit *)o;
+			if (o->flags & THEY_HAVE)
+				we_knew_they_have = 1;
+			else
+				o->flags |= THEY_HAVE;
+			if (!oldest_have || (commit->date < oldest_have))
+				oldest_have = commit->date;
+			for (parents = commit->parents;
+			     parents;
+			     parents = parents->next)
+				parents->item->object.flags |= THEY_HAVE;
+		}
+		if (!we_knew_they_have)
+			add_object_array(o, NULL, &have_obj);
+	}
+
+	return 0;
+}
+
+static int send_acks(struct oid_array *acks, struct strbuf *response)
+{
+	int i;
+	/* Send Acks */
+	if (!acks->nr)
+		packet_buf_write(response, "NAK\n");
+
+	for (i = 0; i < acks->nr; i++) {
+		packet_buf_write(response, "ACK %s common\n",
+				 oid_to_hex(&acks->oid[i]));
+	}
+
+	if (ok_to_give_up()) {
+		/* Send Ready */
+		packet_buf_write(response, "ACK %s ready\n",
+				 oid_to_hex(&acks->oid[i-1]));
+		return 1;
+	}
+
+	return 0;
+}
+
+static int process_haves_and_send_acks(struct upload_pack_data *data)
+{
+	struct oid_array common = OID_ARRAY_INIT;
+	struct strbuf response = STRBUF_INIT;
+	int ret = 0;
+
+	process_haves(&data->haves, &common);
+	if (data->done) {
+		ret = 1;
+	} else if (send_acks(&common, &response)) {
+		packet_buf_delim(&response);
+		ret = 1;
+	} else {
+		/* Add Flush */
+		packet_buf_flush(&response);
+		ret = 0;
+	}
+
+	/* Send response */
+	write_or_die(1, response.buf, response.len);
+	strbuf_release(&response);
+
+	oid_array_clear(&data->haves);
+	oid_array_clear(&common);
+	return ret;
+}
+
+#define FETCH_PROCESS_ARGS 0
+#define FETCH_READ_HAVES 1
+#define FETCH_SEND_ACKS 2
+#define FETCH_SEND_PACK 3
+#define FETCH_DONE 4
+
+int upload_pack_v2(struct repository *r, struct argv_array *keys,
+		   struct argv_array *args)
+{
+	int state = FETCH_PROCESS_ARGS;
+	struct upload_pack_data data = UPLOAD_PACK_DATA_INIT;
+	const char *out;
+	use_sideband = LARGE_PACKET_MAX;
+
+	/* Check if cmd is being run as a stateless-rpc */
+	if (has_capability(keys, "stateless-rpc", &out))
+		if (!strcmp(out, "true"))
+			data.stateless_rpc = 1;
+
+	while (state != FETCH_DONE) {
+		switch (state) {
+		case FETCH_PROCESS_ARGS:
+			process_args(args, &data);
+
+			if (!want_obj.nr) {
+				/*
+				 * Request didn't contain any 'want' lines,
+				 * guess they didn't want anything.
+				 */
+				state = FETCH_DONE;
+			} else if (data.haves.nr) {
+				/*
+				 * Request had 'have' lines, so lets ACK them.
+				 */
+				state = FETCH_SEND_ACKS;
+			} else {
+				/*
+				 * Request had 'want's but no 'have's so we can
+				 * immedietly go to construct and send a pack.
+				 */
+				state = FETCH_SEND_PACK;
+			}
+			break;
+		case FETCH_READ_HAVES:
+			read_haves(&data);
+			state = FETCH_SEND_ACKS;
+			break;
+		case FETCH_SEND_ACKS:
+			if (process_haves_and_send_acks(&data))
+				state = FETCH_SEND_PACK;
+			else if (data.stateless_rpc)
+				/*
+				 * Request was made via stateless-rpc and a
+				 * packfile isn't ready to be created and sent.
+				 */
+				state = FETCH_DONE;
+			else
+				state = FETCH_READ_HAVES;
+			break;
+		case FETCH_SEND_PACK:
+			create_pack_file();
+			state = FETCH_DONE;
+			break;
+		case FETCH_DONE:
+			break;
+		default:
+			BUG("invalid state");
+		}
+	}
+
+	upload_pack_data_clear(&data);
+	return 0;
+}
+
 static int upload_pack_config(const char *var, const char *value, void *unused)
 {
 	if (!strcmp("uploadpack.allowtipsha1inwant", var)) {
diff --git a/upload-pack.h b/upload-pack.h
new file mode 100644
index 000000000..54c429563
--- /dev/null
+++ b/upload-pack.h
@@ -0,0 +1,9 @@
+#ifndef UPLOAD_PACK_H
+#define UPLOAD_PACK_H
+
+struct repository;
+struct argv_array;
+extern int upload_pack_v2(struct repository *r, struct argv_array *keys,
+			  struct argv_array *args);
+
+#endif /* UPLOAD_PACK_H */
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 20/26] fetch-pack: perform a fetch using v2
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (18 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 19/26] upload-pack: introduce fetch server command Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-04  1:23   ` Stefan Beller
  2018-01-10  0:05   ` Jonathan Tan
  2018-01-03  0:18 ` [PATCH 21/26] transport-helper: remove name parameter Brandon Williams
                   ` (6 subsequent siblings)
  26 siblings, 2 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

When communicating with a v2 server, perform a fetch by requesting the
'fetch' command.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/fetch-pack.c   |   2 +-
 fetch-pack.c           | 267 ++++++++++++++++++++++++++++++++++++++++++++++++-
 fetch-pack.h           |   4 +-
 t/t5701-protocol-v2.sh |  40 ++++++++
 transport.c            |   8 +-
 5 files changed, 314 insertions(+), 7 deletions(-)

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index f492e8abd..867dd3cc7 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -213,7 +213,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 	}
 
 	ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,
-			 &shallow, pack_lockfile_ptr);
+			 &shallow, pack_lockfile_ptr, protocol_v0);
 	if (pack_lockfile) {
 		printf("lock %s\n", pack_lockfile);
 		fflush(stdout);
diff --git a/fetch-pack.c b/fetch-pack.c
index 9f6b07ad9..c26fdc539 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -1008,6 +1008,262 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 	return ref;
 }
 
+static void add_wants(const struct ref *wants, struct strbuf *req_buf)
+{
+	for ( ; wants ; wants = wants->next) {
+		const struct object_id *remote = &wants->old_oid;
+		const char *remote_hex;
+		struct object *o;
+
+		/*
+		 * If that object is complete (i.e. it is an ancestor of a
+		 * local ref), we tell them we have it but do not have to
+		 * tell them about its ancestors, which they already know
+		 * about.
+		 *
+		 * We use lookup_object here because we are only
+		 * interested in the case we *know* the object is
+		 * reachable and we have already scanned it.
+		 */
+		if (((o = lookup_object(remote->hash)) != NULL) &&
+		    (o->flags & COMPLETE)) {
+			continue;
+		}
+
+		remote_hex = oid_to_hex(remote);
+		packet_buf_write(req_buf, "want %s\n", remote_hex);
+	}
+}
+
+static int add_haves(struct strbuf *req_buf, int *in_vain)
+{
+	int ret = 0;
+	int haves_added = 0;
+	const struct object_id *oid;
+
+	while ((oid = get_rev())) {
+		packet_buf_write(req_buf, "have %s\n", oid_to_hex(oid));
+		if (++haves_added >= INITIAL_FLUSH)
+			break;
+	};
+
+	*in_vain += haves_added;
+	if (!haves_added || *in_vain >= MAX_IN_VAIN) {
+		/* Send Done */
+		packet_buf_write(req_buf, "done\n");
+		ret = 1;
+	}
+
+	return ret;
+}
+
+static int send_haves(int fd_out, int *in_vain)
+{
+	int ret = 0;
+	struct strbuf req_buf = STRBUF_INIT;
+
+	ret = add_haves(&req_buf, in_vain);
+
+	/* Send request */
+	packet_buf_flush(&req_buf);
+	write_or_die(fd_out, req_buf.buf, req_buf.len);
+
+	strbuf_release(&req_buf);
+	return ret;
+}
+
+static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,
+			      const struct ref *wants, struct oidset *common,
+			      int *in_vain)
+{
+	int ret = 0;
+	struct strbuf req_buf = STRBUF_INIT;
+
+	packet_buf_write(&req_buf, "command=fetch");
+	packet_buf_write(&req_buf, "agent=%s", git_user_agent_sanitized());
+	if (args->stateless_rpc)
+		packet_buf_write(&req_buf, "stateless-rpc=true");
+
+	packet_buf_delim(&req_buf);
+	if (args->use_thin_pack)
+		packet_buf_write(&req_buf, "thin-pack");
+	if (args->no_progress)
+		packet_buf_write(&req_buf, "no-progress");
+	if (args->include_tag)
+		packet_buf_write(&req_buf, "include-tag");
+	if (prefer_ofs_delta)
+		packet_buf_write(&req_buf, "ofs-delta");
+
+	/* add wants */
+	add_wants(wants, &req_buf);
+
+	/*
+	 * If we are running stateless-rpc we need to add all the common
+	 * commits we've found in previous rounds
+	 */
+	if (args->stateless_rpc) {
+		struct oidset_iter iter;
+		const struct object_id *oid;
+		oidset_iter_init(common, &iter);
+
+		while ((oid = oidset_iter_next(&iter))) {
+			packet_buf_write(&req_buf, "have %s\n", oid_to_hex(oid));
+		}
+	}
+
+	/* Add initial haves */
+	ret = add_haves(&req_buf, in_vain);
+
+	/* Send request */
+	packet_buf_flush(&req_buf);
+	write_or_die(fd_out, req_buf.buf, req_buf.len);
+
+	strbuf_release(&req_buf);
+	return ret;
+}
+
+static enum ack_type process_ack(const char *line, struct object_id *oid)
+{
+	const char *arg;
+
+	if (!strcmp(line, "NAK"))
+		return NAK;
+	if (skip_prefix(line, "ACK ", &arg)) {
+		if (!parse_oid_hex(arg, oid, &arg)) {
+			if (strstr(arg, "continue"))
+				return ACK_continue;
+			if (strstr(arg, "common"))
+				return ACK_common;
+			if (strstr(arg, "ready"))
+				return ACK_ready;
+			return ACK;
+		}
+	}
+	if (skip_prefix(line, "ERR ", &arg))
+		die(_("remote error: %s"), arg);
+	die(_("git fetch-pack: expected ACK/NAK, got '%s'"), line);
+}
+
+static int process_acks(struct packet_reader *reader, struct oidset *common)
+{
+	int got_ready = 0;
+	int got_common = 0;
+	while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
+		struct object_id oid;
+		struct commit *commit;
+		enum ack_type ack = process_ack(reader->line, &oid);
+
+		switch (ack) {
+		case ACK_ready:
+			clear_prio_queue(&rev_list);
+			got_ready = 1;
+			/* fallthrough */
+		case ACK_common:
+			oidset_insert(common, &oid);
+			commit = lookup_commit(&oid);
+			mark_common(commit, 0, 1);
+			got_common = 1;
+			break;
+		case NAK:
+			break;
+		case ACK:
+		case ACK_continue:
+			die("ACK/ACK_continue not supported");
+		}
+	}
+
+	if (reader->status != PACKET_READ_FLUSH &&
+	    reader->status != PACKET_READ_DELIM)
+		die("Error during processing acks: %d", reader->status);
+
+	/* return 0 if no common, 1 if there are common, or 2 if ready */
+	return got_ready + got_common;
+}
+
+#define FETCH_CHECK_LOCAL 0
+#define FETCH_SEND_REQUEST 1
+#define FETCH_PROCESS_ACKS 2
+#define FETCH_SEND_HAVES 3
+#define FETCH_GET_PACK 4
+#define FETCH_DONE 5
+
+static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
+				    int fd[2],
+				    const struct ref *orig_ref,
+				    struct ref **sought, int nr_sought,
+				    char **pack_lockfile)
+{
+	struct ref *ref = copy_ref_list(orig_ref);
+	int state = FETCH_CHECK_LOCAL;
+	struct oidset common = OIDSET_INIT;
+	struct packet_reader reader;
+	int in_vain = 0;
+	packet_reader_init(&reader, fd[0], NULL, 0,
+			   PACKET_READ_CHOMP_NEWLINE);
+
+	while (state != FETCH_DONE) {
+		switch (state) {
+		case FETCH_CHECK_LOCAL:
+			sort_ref_list(&ref, ref_compare_name);
+			QSORT(sought, nr_sought, cmp_ref_by_name);
+
+			/* v2 supports these by default */
+			allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;
+			use_sideband = 2;
+
+			/* Filter 'ref' by 'sought' and those that aren't local */
+			if (everything_local(args, &ref, sought, nr_sought))
+				state = FETCH_DONE;
+			else
+				state = FETCH_SEND_REQUEST;
+			break;
+		case FETCH_SEND_REQUEST:
+			if (send_fetch_request(fd[1], args, ref, &common, &in_vain))
+				state = FETCH_GET_PACK;
+			else
+				state = FETCH_PROCESS_ACKS;
+			break;
+		case FETCH_PROCESS_ACKS:
+			/* Process ACKs/NAKs */
+			switch (process_acks(&reader, &common)) {
+			case 2:
+				state = FETCH_GET_PACK;
+				break;
+			case 1:
+				in_vain = 0;
+				/* fallthrough */
+			default:
+				if (args->stateless_rpc)
+					state = FETCH_SEND_REQUEST;
+				else
+					state = FETCH_SEND_HAVES;
+				break;
+			}
+			break;
+		case FETCH_SEND_HAVES:
+			if (send_haves(fd[1], &in_vain))
+				state = FETCH_GET_PACK;
+			else
+				state = FETCH_PROCESS_ACKS;
+			break;
+		case FETCH_GET_PACK:
+			/* get the pack */
+			if (get_pack(args, fd, pack_lockfile))
+				die(_("git fetch-pack: fetch failed."));
+
+			state = FETCH_DONE;
+			break;
+		case FETCH_DONE:
+			break;
+		default:
+			die("invalid state");
+		}
+	}
+
+	oidset_clear(&common);
+	return ref;
+}
+
 static void fetch_pack_config(void)
 {
 	git_config_get_int("fetch.unpacklimit", &fetch_unpack_limit);
@@ -1153,7 +1409,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		       const char *dest,
 		       struct ref **sought, int nr_sought,
 		       struct oid_array *shallow,
-		       char **pack_lockfile)
+		       char **pack_lockfile,
+		       enum protocol_version version)
 {
 	struct ref *ref_cpy;
 	struct shallow_info si;
@@ -1167,8 +1424,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		die(_("no matching remote head"));
 	}
 	prepare_shallow_info(&si, shallow);
-	ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
-				&si, pack_lockfile);
+	if (version == protocol_v2)
+		ref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,
+					   pack_lockfile);
+	else
+		ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
+					&si, pack_lockfile);
 	reprepare_packed_git();
 	update_shallow(args, sought, nr_sought, &si);
 	clear_shallow_info(&si);
diff --git a/fetch-pack.h b/fetch-pack.h
index b6aeb43a8..7afca7305 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -3,6 +3,7 @@
 
 #include "string-list.h"
 #include "run-command.h"
+#include "protocol.h"
 
 struct oid_array;
 
@@ -43,7 +44,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 		       struct ref **sought,
 		       int nr_sought,
 		       struct oid_array *shallow,
-		       char **pack_lockfile);
+		       char **pack_lockfile,
+		       enum protocol_version version);
 
 /*
  * Print an appropriate error message for each sought ref that wasn't
diff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh
index 7d8aeb766..3e411e178 100755
--- a/t/t5701-protocol-v2.sh
+++ b/t/t5701-protocol-v2.sh
@@ -33,4 +33,44 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol
 	! grep "refs/tags/" log
 '
 
+test_expect_success 'clone with file:// using protocol v2' '
+	GIT_TRACE_PACKET=1 git -c protocol.version=2 \
+		clone "file://$(pwd)/file_parent" file_child 2>log &&
+
+	git -C file_child log -1 --format=%s >actual &&
+	git -C file_parent log -1 --format=%s >expect &&
+	test_cmp expect actual &&
+
+	# Server responded using protocol v1
+	grep "clone< version 2" log
+'
+
+test_expect_success 'fetch with file:// using protocol v2' '
+	test_commit -C file_parent two &&
+
+	GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \
+		fetch origin 2>log &&
+
+	git -C file_child log -1 --format=%s origin/master >actual &&
+	git -C file_parent log -1 --format=%s >expect &&
+	test_cmp expect actual &&
+
+	# Server responded using protocol v1
+	grep "fetch< version 2" log
+'
+
+test_expect_success 'ref advertisment is filtered during fetch using protocol v2' '
+	test_commit -C file_parent three &&
+
+	GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \
+		fetch origin master 2>log &&
+
+	git -C file_child log -1 --format=%s origin/master >actual &&
+	git -C file_parent log -1 --format=%s >expect &&
+	test_cmp expect actual &&
+
+	grep "ref-pattern master" log &&
+	! grep "refs/tags/" log
+'
+
 test_done
diff --git a/transport.c b/transport.c
index 6ea3905e3..4fdbd9adc 100644
--- a/transport.c
+++ b/transport.c
@@ -256,14 +256,18 @@ static int fetch_refs_via_pack(struct transport *transport,
 
 	switch (data->version) {
 	case protocol_v2:
-		die("support for protocol v2 not implemented yet");
+		refs = fetch_pack(&args, data->fd, data->conn,
+				  refs_tmp ? refs_tmp : transport->remote_refs,
+				  dest, to_fetch, nr_heads, &data->shallow,
+				  &transport->pack_lockfile, data->version);
+		packet_flush(data->fd[1]);
 		break;
 	case protocol_v1:
 	case protocol_v0:
 		refs = fetch_pack(&args, data->fd, data->conn,
 				  refs_tmp ? refs_tmp : transport->remote_refs,
 				  dest, to_fetch, nr_heads, &data->shallow,
-				  &transport->pack_lockfile);
+				  &transport->pack_lockfile, data->version);
 		break;
 	case protocol_unknown_version:
 		BUG("unknown protocol version");
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 21/26] transport-helper: remove name parameter
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (19 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 20/26] fetch-pack: perform a fetch using v2 Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 22/26] transport-helper: refactor process_connect_service Brandon Williams
                   ` (5 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Commit 266f1fdfa (transport-helper: be quiet on read errors from
helpers, 2013-06-21) removed a call to 'die()' which printed the name of
the remote helper passed in to the 'recvline_fh()' function using the
'name' parameter.  Once the call to 'die()' was removed the parameter
was no longer necessary but wasn't removed.  Clean up 'recvline_fh()'
parameter list by removing the 'name' parameter.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 transport-helper.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/transport-helper.c b/transport-helper.c
index 4c334b5ee..d72155768 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -49,7 +49,7 @@ static void sendline(struct helper_data *helper, struct strbuf *buffer)
 		die_errno("Full write to remote helper failed");
 }
 
-static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)
+static int recvline_fh(FILE *helper, struct strbuf *buffer)
 {
 	strbuf_reset(buffer);
 	if (debug)
@@ -67,7 +67,7 @@ static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)
 
 static int recvline(struct helper_data *helper, struct strbuf *buffer)
 {
-	return recvline_fh(helper->out, buffer, helper->name);
+	return recvline_fh(helper->out, buffer);
 }
 
 static void write_constant(int fd, const char *str)
@@ -586,7 +586,7 @@ static int process_connect_service(struct transport *transport,
 		goto exit;
 
 	sendline(data, &cmdbuf);
-	if (recvline_fh(input, &cmdbuf, name))
+	if (recvline_fh(input, &cmdbuf))
 		exit(128);
 
 	if (!strcmp(cmdbuf.buf, "")) {
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 22/26] transport-helper: refactor process_connect_service
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (20 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 21/26] transport-helper: remove name parameter Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 23/26] transport-helper: introduce connect-half-duplex Brandon Williams
                   ` (4 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

A future patch will need to take advantage of the logic which runs and
processes the response of the connect command on a remote helper so
factor out this logic from 'process_connect_service()' and place it into
a helper function 'run_connect()'.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 transport-helper.c | 67 +++++++++++++++++++++++++++++++-----------------------
 1 file changed, 38 insertions(+), 29 deletions(-)

diff --git a/transport-helper.c b/transport-helper.c
index d72155768..c032a2a87 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -545,14 +545,13 @@ static int fetch_with_import(struct transport *transport,
 	return 0;
 }
 
-static int process_connect_service(struct transport *transport,
-				   const char *name, const char *exec)
+static int run_connect(struct transport *transport, struct strbuf *cmdbuf)
 {
 	struct helper_data *data = transport->data;
-	struct strbuf cmdbuf = STRBUF_INIT;
-	struct child_process *helper;
-	int r, duped, ret = 0;
+	int ret = 0;
+	int duped;
 	FILE *input;
+	struct child_process *helper;
 
 	helper = get_helper(transport);
 
@@ -568,44 +567,54 @@ static int process_connect_service(struct transport *transport,
 	input = xfdopen(duped, "r");
 	setvbuf(input, NULL, _IONBF, 0);
 
+	sendline(data, cmdbuf);
+	if (recvline_fh(input, cmdbuf))
+		exit(128);
+
+	if (!strcmp(cmdbuf->buf, "")) {
+		data->no_disconnect_req = 1;
+		if (debug)
+			fprintf(stderr, "Debug: Smart transport connection "
+				"ready.\n");
+		ret = 1;
+	} else if (!strcmp(cmdbuf->buf, "fallback")) {
+		if (debug)
+			fprintf(stderr, "Debug: Falling back to dumb "
+				"transport.\n");
+	} else {
+		die("Unknown response to connect: %s",
+			cmdbuf->buf);
+	}
+
+	fclose(input);
+	return ret;
+}
+
+static int process_connect_service(struct transport *transport,
+				   const char *name, const char *exec)
+{
+	struct helper_data *data = transport->data;
+	struct strbuf cmdbuf = STRBUF_INIT;
+	int ret = 0;
+
 	/*
 	 * Handle --upload-pack and friends. This is fire and forget...
 	 * just warn if it fails.
 	 */
 	if (strcmp(name, exec)) {
-		r = set_helper_option(transport, "servpath", exec);
+		int r = set_helper_option(transport, "servpath", exec);
 		if (r > 0)
 			warning("Setting remote service path not supported by protocol.");
 		else if (r < 0)
 			warning("Invalid remote service path.");
 	}
 
-	if (data->connect)
+	if (data->connect) {
 		strbuf_addf(&cmdbuf, "connect %s\n", name);
-	else
-		goto exit;
-
-	sendline(data, &cmdbuf);
-	if (recvline_fh(input, &cmdbuf))
-		exit(128);
-
-	if (!strcmp(cmdbuf.buf, "")) {
-		data->no_disconnect_req = 1;
-		if (debug)
-			fprintf(stderr, "Debug: Smart transport connection "
-				"ready.\n");
-		ret = 1;
-	} else if (!strcmp(cmdbuf.buf, "fallback")) {
-		if (debug)
-			fprintf(stderr, "Debug: Falling back to dumb "
-				"transport.\n");
-	} else
-		die("Unknown response to connect: %s",
-			cmdbuf.buf);
+		ret = run_connect(transport, &cmdbuf);
+	}
 
-exit:
 	strbuf_release(&cmdbuf);
-	fclose(input);
 	return ret;
 }
 
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 23/26] transport-helper: introduce connect-half-duplex
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (21 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 22/26] transport-helper: refactor process_connect_service Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 24/26] pkt-line: add packet_buf_write_len function Brandon Williams
                   ` (3 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Introduce the transport-helper capability 'connect-half-duplex'.  This
capability indicates that the transport-helper can be requested to run
the 'connect-half-duplex' command which should attempt to make a
half-duplex connection with a remote end.  Once established, the
half-duplex connection can be used by the git client to communicate with
the remote end natively in a stateles-rpc manner as supported by
protocol v2.

If a half-duplex connection cannot be established then the remote-helper
will respond in the same manner as the 'connect' command indicating that
the client should fallback to using the dumb remote-helper commands.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 transport-helper.c | 8 ++++++++
 transport.c        | 1 +
 transport.h        | 6 ++++++
 3 files changed, 15 insertions(+)

diff --git a/transport-helper.c b/transport-helper.c
index c032a2a87..d037609bc 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -26,6 +26,7 @@ struct helper_data {
 		option : 1,
 		push : 1,
 		connect : 1,
+		connect_half_duplex : 1,
 		signed_tags : 1,
 		check_connectivity : 1,
 		no_disconnect_req : 1,
@@ -188,6 +189,8 @@ static struct child_process *get_helper(struct transport *transport)
 			refspecs[refspec_nr++] = xstrdup(arg);
 		} else if (!strcmp(capname, "connect")) {
 			data->connect = 1;
+		} else if (!strcmp(capname, "connect-half-duplex")) {
+			data->connect_half_duplex = 1;
 		} else if (!strcmp(capname, "signed-tags")) {
 			data->signed_tags = 1;
 		} else if (skip_prefix(capname, "export-marks ", &arg)) {
@@ -612,6 +615,11 @@ static int process_connect_service(struct transport *transport,
 	if (data->connect) {
 		strbuf_addf(&cmdbuf, "connect %s\n", name);
 		ret = run_connect(transport, &cmdbuf);
+	} else if (data->connect_half_duplex) {
+		strbuf_addf(&cmdbuf, "connect-half-duplex %s\n", name);
+		ret = run_connect(transport, &cmdbuf);
+		if (ret)
+			transport->stateless_rpc = 1;
 	}
 
 	strbuf_release(&cmdbuf);
diff --git a/transport.c b/transport.c
index 4fdbd9adc..aafb8fbb4 100644
--- a/transport.c
+++ b/transport.c
@@ -250,6 +250,7 @@ static int fetch_refs_via_pack(struct transport *transport,
 		data->options.check_self_contained_and_connected;
 	args.cloning = transport->cloning;
 	args.update_shallow = data->options.update_shallow;
+	args.stateless_rpc = transport->stateless_rpc;
 
 	if (!data->got_remote_heads)
 		refs_tmp = get_refs_via_connect(transport, 0, NULL);
diff --git a/transport.h b/transport.h
index 4b656f315..9eac809ee 100644
--- a/transport.h
+++ b/transport.h
@@ -55,6 +55,12 @@ struct transport {
 	 */
 	unsigned cloning : 1;
 
+	/*
+	 * Indicates that the transport is connected via a half-duplex
+	 * connection and should operate in stateless-rpc mode.
+	 */
+	unsigned stateless_rpc : 1;
+
 	/*
 	 * These strings will be passed to the {pre, post}-receive hook,
 	 * on the remote side, if both sides support the push options capability.
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 24/26] pkt-line: add packet_buf_write_len function
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (22 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 23/26] transport-helper: introduce connect-half-duplex Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 25/26] remote-curl: create copy of the service name Brandon Williams
                   ` (2 subsequent siblings)
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Add the 'packet_buf_write_len()' function which allows for writing an
arbitrary length buffer into a 'struct strbuf' and formatting it in
packet-line format.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 pkt-line.c | 16 ++++++++++++++++
 pkt-line.h |  1 +
 2 files changed, 17 insertions(+)

diff --git a/pkt-line.c b/pkt-line.c
index 3159cbe10..e9968b7df 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -215,6 +215,22 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)
 	va_end(args);
 }
 
+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len)
+{
+	size_t orig_len, n;
+
+	orig_len = buf->len;
+	strbuf_addstr(buf, "0000");
+	strbuf_add(buf, data, len);
+	n = buf->len - orig_len;
+
+	if (n > LARGE_PACKET_MAX)
+		die("protocol error: impossibly long line");
+
+	set_packet_header(&buf->buf[orig_len], n);
+	packet_trace(buf->buf + orig_len + 4, n - 4, 1);
+}
+
 int write_packetized_from_fd(int fd_in, int fd_out)
 {
 	static char buf[LARGE_PACKET_DATA_MAX];
diff --git a/pkt-line.h b/pkt-line.h
index 97b6dd1c7..d411fcb30 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -26,6 +26,7 @@ void packet_buf_flush(struct strbuf *buf);
 void packet_buf_delim(struct strbuf *buf);
 void packet_write(int fd_out, const char *buf, size_t size);
 void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len);
 int packet_flush_gently(int fd);
 int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
 int write_packetized_from_fd(int fd_in, int fd_out);
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 25/26] remote-curl: create copy of the service name
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (23 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 24/26] pkt-line: add packet_buf_write_len function Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-03  0:18 ` [PATCH 26/26] remote-curl: implement connect-half-duplex command Brandon Williams
  2018-01-09 17:55 ` [PATCH 00/26] protocol version 2 Jonathan Tan
  26 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Make a copy of the service name being requested instead of relying on
the buffer pointed to by the passed in 'const char *' to remain
unchanged.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 remote-curl.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/remote-curl.c b/remote-curl.c
index dae8a4a48..4086aa733 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -165,7 +165,7 @@ static int set_option(const char *name, const char *value)
 }
 
 struct discovery {
-	const char *service;
+	char *service;
 	char *buf_alloc;
 	char *buf;
 	size_t len;
@@ -257,6 +257,7 @@ static void free_discovery(struct discovery *d)
 		free(d->shallow.oid);
 		free(d->buf_alloc);
 		free_refs(d->refs);
+		free(d->service);
 		free(d);
 	}
 }
@@ -343,7 +344,7 @@ static struct discovery *discover_refs(const char *service, int for_push)
 		warning(_("redirecting to %s"), url.buf);
 
 	last= xcalloc(1, sizeof(*last_discovery));
-	last->service = service;
+	last->service = xstrdup(service);
 	last->buf_alloc = strbuf_detach(&buffer, &last->len);
 	last->buf = last->buf_alloc;
 
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* [PATCH 26/26] remote-curl: implement connect-half-duplex command
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (24 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 25/26] remote-curl: create copy of the service name Brandon Williams
@ 2018-01-03  0:18 ` Brandon Williams
  2018-01-10  0:10   ` Jonathan Tan
  2018-01-10 17:57   ` Jonathan Tan
  2018-01-09 17:55 ` [PATCH 00/26] protocol version 2 Jonathan Tan
  26 siblings, 2 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03  0:18 UTC (permalink / raw)
  To: git; +Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams

Teach remote-curl the 'connect-half-duplex' command which is used to
establish a half-duplex connection with servers which support protocol
version 2.  This allows remote-curl to act as a proxy, allowing the git
client to communicate natively with a remote end, simply using
remote-curl as a pass through to convert requests to http.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 remote-curl.c          | 185 ++++++++++++++++++++++++++++++++++++++++++++++++-
 t/t5701-protocol-v2.sh |  41 +++++++++++
 2 files changed, 224 insertions(+), 2 deletions(-)

diff --git a/remote-curl.c b/remote-curl.c
index 4086aa733..b63b06398 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -171,6 +171,7 @@ struct discovery {
 	size_t len;
 	struct ref *refs;
 	struct oid_array shallow;
+	enum protocol_version version;
 	unsigned proto_git : 1;
 };
 static struct discovery *last_discovery;
@@ -184,9 +185,13 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
 			   PACKET_READ_CHOMP_NEWLINE |
 			   PACKET_READ_GENTLE_ON_EOF);
 
-	switch (discover_version(&reader)) {
+	heads->version = discover_version(&reader);
+	switch (heads->version) {
 	case protocol_v2:
-		die("support for protocol v2 not implemented yet");
+		/*
+		 * Do nothing.  Client should run 'connect-half-duplex' and
+		 * request the refs themselves.
+		 */
 		break;
 	case protocol_v1:
 	case protocol_v0:
@@ -1047,6 +1052,178 @@ static void parse_push(struct strbuf *buf)
 	free(specs);
 }
 
+struct proxy_state {
+	char *service_name;
+	char *service_url;
+	char *hdr_content_type;
+	char *hdr_accept;
+	struct strbuf request_buffer;
+	int in;
+	int out;
+	struct packet_reader reader;
+	size_t pos;
+	int seen_flush;
+};
+
+static void proxy_state_init(struct proxy_state *p, const char *service_name)
+{
+	struct strbuf buf = STRBUF_INIT;
+
+	memset(p, 0, sizeof(*p));
+	p->service_name = xstrdup(service_name);
+
+	p->in = 0;
+	p->out = 1;
+	strbuf_init(&p->request_buffer, 0);
+
+	strbuf_addf(&buf, "%s%s", url.buf, p->service_name);
+	p->service_url = strbuf_detach(&buf, NULL);
+
+	strbuf_addf(&buf, "Content-Type: application/x-%s-request", p->service_name);
+	p->hdr_content_type = strbuf_detach(&buf, NULL);
+
+	strbuf_addf(&buf, "Accept: application/x-%s-result", p->service_name);
+	p->hdr_accept = strbuf_detach(&buf, NULL);
+
+	packet_reader_init(&p->reader, p->in, NULL, 0,
+			   PACKET_READ_GENTLE_ON_EOF);
+}
+
+static void proxy_state_clear(struct proxy_state *p)
+{
+	free(p->service_name);
+	free(p->service_url);
+	free(p->hdr_content_type);
+	free(p->hdr_accept);
+	strbuf_release(&p->request_buffer);
+}
+
+static size_t proxy_in(void *ptr, size_t eltsize,
+		       size_t nmemb, void *buffer_)
+{
+	size_t max = eltsize * nmemb;
+	struct proxy_state *p = buffer_;
+	size_t avail = p->request_buffer.len - p->pos;
+
+	if (!avail) {
+		if (p->seen_flush) {
+			p->seen_flush = 0;
+			return 0;
+		}
+
+		strbuf_reset(&p->request_buffer);
+		switch (packet_reader_read(&p->reader)) {
+		case PACKET_READ_EOF:
+			die("error reading request from parent process");
+		case PACKET_READ_NORMAL:
+			packet_buf_write_len(&p->request_buffer, p->reader.line,
+					     p->reader.pktlen);
+			break;
+		case PACKET_READ_DELIM:
+			packet_buf_delim(&p->request_buffer);
+			break;
+		case PACKET_READ_FLUSH:
+			packet_buf_flush(&p->request_buffer);
+			p->seen_flush = 1;
+			break;
+		}
+		p->pos = 0;
+		avail = p->request_buffer.len;
+	}
+
+	if (max < avail)
+		avail = max;
+	memcpy(ptr, p->request_buffer.buf + p->pos, avail);
+	p->pos += avail;
+	return avail;
+}
+static size_t proxy_out(char *ptr, size_t eltsize,
+			size_t nmemb, void *buffer_)
+{
+	size_t size = eltsize * nmemb;
+	struct proxy_state *p = buffer_;
+
+	write_or_die(p->out, ptr, size);
+	return size;
+}
+
+static int proxy_post(struct proxy_state *p)
+{
+	struct active_request_slot *slot;
+	struct curl_slist *headers = http_copy_default_headers();
+	int err;
+
+	headers = curl_slist_append(headers, p->hdr_content_type);
+	headers = curl_slist_append(headers, p->hdr_accept);
+	headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
+
+	slot = get_active_slot();
+
+	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
+	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
+	curl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);
+	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
+
+	/* Setup function to read request from client */
+	curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);
+	curl_easy_setopt(slot->curl, CURLOPT_READDATA, p);
+
+	/* Setup function to write server response to client */
+	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);
+	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);
+
+	err = run_slot(slot, NULL);
+
+	if (err != HTTP_OK)
+		err = -1;
+
+	curl_slist_free_all(headers);
+	return err;
+}
+
+static int connect_half_duplex(const char *service_name)
+{
+	struct discovery *discover;
+	struct proxy_state p;
+
+	/*
+	 * Run the info/refs request and see if the server supports protocol
+	 * v2.  If and only if the server supports v2 can we successfully
+	 * establish a half-duplex connection, otherwise we need to tell the
+	 * client to fallback to using other transport helper functions to
+	 * complete their request.
+	 */
+	discover = discover_refs(service_name, 0);
+	if (discover->version != protocol_v2) {
+		printf("fallback\n");
+		fflush(stdout);
+		return -1;
+	} else {
+		/* Half-Duplex Connection established */
+		printf("\n");
+		fflush(stdout);
+	}
+
+	proxy_state_init(&p, service_name);
+
+	/*
+	 * Dump the capability listing that we got from the server earlier
+	 * during the info/refs request.
+	 */
+	write_or_die(p.out, discover->buf, discover->len);
+
+	/* Peek the next packet line.  Until we see EOF keep sending POSTs */
+	while (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {
+		if (proxy_post(&p)) {
+			/* We would have an err here */
+			break;
+		}
+	}
+
+	proxy_state_clear(&p);
+	return 0;
+}
+
 int cmd_main(int argc, const char **argv)
 {
 	struct strbuf buf = STRBUF_INIT;
@@ -1115,12 +1292,16 @@ int cmd_main(int argc, const char **argv)
 			fflush(stdout);
 
 		} else if (!strcmp(buf.buf, "capabilities")) {
+			printf("connect-half-duplex\n");
 			printf("fetch\n");
 			printf("option\n");
 			printf("push\n");
 			printf("check-connectivity\n");
 			printf("\n");
 			fflush(stdout);
+		} else if (skip_prefix(buf.buf, "connect-half-duplex ", &arg)) {
+			if (!connect_half_duplex(arg))
+				break;
 		} else {
 			error("remote-curl: unknown command '%s' from git", buf.buf);
 			return 1;
diff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh
index 3e411e178..ada69ac09 100755
--- a/t/t5701-protocol-v2.sh
+++ b/t/t5701-protocol-v2.sh
@@ -73,4 +73,45 @@ test_expect_success 'ref advertisment is filtered during fetch using protocol v2
 	! grep "refs/tags/" log
 '
 
+# Test protocol v2 with 'http://' transport
+#
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
+
+test_expect_success 'create repo to be served by http:// transport' '
+	git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+	git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config http.receivepack true &&
+	test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" one
+'
+
+test_expect_success 'clone with http:// using protocol v2' '
+	GIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=2 \
+		clone "$HTTPD_URL/smart/http_parent" http_child 2>log &&
+
+	git -C http_child log -1 --format=%s >actual &&
+	git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" log -1 --format=%s >expect &&
+	test_cmp expect actual &&
+
+	# Client requested to use protocol v2
+	grep "Git-Protocol: version=2" log &&
+	# Server responded using protocol v2
+	grep "git< version 2" log
+'
+
+test_expect_success 'fetch with http:// using protocol v2' '
+	test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" two &&
+
+	GIT_TRACE_PACKET=1 git -C http_child -c protocol.version=2 \
+		fetch 2>log &&
+
+	git -C http_child log -1 --format=%s origin/master >actual &&
+	git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" log -1 --format=%s >expect &&
+	test_cmp expect actual &&
+
+	# Server responded using protocol v2
+	grep "git< version 2" log
+'
+
+stop_httpd
+
 test_done
-- 
2.15.1.620.gb9897f4670-goog


^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 01/26] pkt-line: introduce packet_read_with_status
  2018-01-03  0:18 ` [PATCH 01/26] pkt-line: introduce packet_read_with_status Brandon Williams
@ 2018-01-03 19:27   ` Stefan Beller
  2018-01-05 23:41     ` Brandon Williams
  2018-01-09 18:04   ` Jonathan Tan
  1 sibling, 1 reply; 59+ messages in thread
From: Stefan Beller @ 2018-01-03 19:27 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> The current pkt-line API encodes the status of a pkt-line read in the
> length of the read content.  An error is indicated with '-1', a flush
> with '0' (which can be confusing since a return value of '0' can also
> indicate an empty pkt-line), and a positive integer for the length of
> the read content otherwise.  This doesn't leave much room for allowing
> the addition of additional special packets in the future.
>
> To solve this introduce 'packet_read_with_status()' which reads a packet
> and returns the status of the read encoded as an 'enum packet_status'
> type.  This allows for easily identifying between special and normal
> packets as well as errors.  It also enables easily adding a new special
> packet in the future.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  pkt-line.c | 55 ++++++++++++++++++++++++++++++++++++++++++-------------
>  pkt-line.h | 15 +++++++++++++++
>  2 files changed, 57 insertions(+), 13 deletions(-)
>
> diff --git a/pkt-line.c b/pkt-line.c
> index 2827ca772..8d7cd389f 100644
> --- a/pkt-line.c
> +++ b/pkt-line.c
> @@ -280,28 +280,33 @@ static int packet_length(const char *linelen)
>         return (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);
>  }
>
> -int packet_read(int fd, char **src_buf, size_t *src_len,
> -               char *buffer, unsigned size, int options)
> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
> +                                               char *buffer, unsigned size, int *pktlen,
> +                                               int options)
>  {
> -       int len, ret;
> +       int len;
>         char linelen[4];
>
> -       ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);
> -       if (ret < 0)
> -               return ret;
> +       if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)
> +               return PACKET_READ_EOF;
> +
>         len = packet_length(linelen);
>         if (len < 0)
>                 die("protocol error: bad line length character: %.4s", linelen);
> -       if (!len) {
> +
> +       if (len == 0) {
>                 packet_trace("0000", 4, 0);
> -               return 0;
> +               return PACKET_READ_FLUSH;
> +       } else if (len >= 1 && len <= 3) {
> +               die("protocol error: bad line length character: %.4s", linelen);

I wonder how much libified code we want here already, maybe we could
have PACKET_READ_ERROR as a return value here instead of die()ing.
There could also be an option that tells this code to die on error, this reminds
me of the repository discovery as well as the refs code, both of which have
this pattern.

Currently this series is only upgrading commands that use the network
anyway, so I guess die()ing in an ls-remote or fetch is no big deal,
but it could
be interesting to keep going once we have more of the partial clone
stuff working
(e.g. remote assisted log/blame would want to gracefully fall back instead of
die()ing without any useful output, I would think.)

>         }
> +
>         len -= 4;
> -       if (len >= size)
> +       if ((len < 0) || ((unsigned)len >= size))
>                 die("protocol error: bad line length %d", len);
> -       ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);
> -       if (ret < 0)
> -               return ret;
> +
> +       if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)
> +               return PACKET_READ_EOF;
>
>         if ((options & PACKET_READ_CHOMP_NEWLINE) &&
>             len && buffer[len-1] == '\n')
> @@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,
>
>         buffer[len] = 0;
>         packet_trace(buffer, len, 0);
> -       return len;
> +       *pktlen = len;
> +       return PACKET_READ_NORMAL;
> +}
> +
> +int packet_read(int fd, char **src_buffer, size_t *src_len,
> +               char *buffer, unsigned size, int options)
> +{
> +       enum packet_read_status status;
> +       int pktlen;
> +
> +       status = packet_read_with_status(fd, src_buffer, src_len,
> +                                        buffer, size, &pktlen,
> +                                        options);
> +       switch (status) {
> +       case PACKET_READ_EOF:
> +               pktlen = -1;
> +               break;
> +       case PACKET_READ_NORMAL:
> +               break;
> +       case PACKET_READ_FLUSH:
> +               pktlen = 0;
> +               break;
> +       }
> +
> +       return pktlen;
>  }
>
>  static char *packet_read_line_generic(int fd,
> diff --git a/pkt-line.h b/pkt-line.h
> index 3dad583e2..06c468927 100644
> --- a/pkt-line.h
> +++ b/pkt-line.h
> @@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);
>  int packet_read(int fd, char **src_buffer, size_t *src_len, char
>                 *buffer, unsigned size, int options);
>
> +/*
> + * Read a packetized line into a buffer like the 'packet_read()' function but
> + * returns an 'enum packet_read_status' which indicates the status of the read.
> + * The number of bytes read will be assigined to *pktlen if the status of the
> + * read was 'PACKET_READ_NORMAL'.
> + */
> +enum packet_read_status {
> +       PACKET_READ_EOF = -1,
> +       PACKET_READ_NORMAL,
> +       PACKET_READ_FLUSH,
> +};
> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
> +                                               char *buffer, unsigned size, int *pktlen,
> +                                               int options);
> +
>  /*
>   * Convenience wrapper for packet_read that is not gentle, and sets the
>   * CHOMP_NEWLINE option. The return value is NULL for a flush packet,
> --
> 2.15.1.620.gb9897f4670-goog
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 04/26] upload-pack: convert to a builtin
  2018-01-03  0:18 ` [PATCH 04/26] upload-pack: convert to a builtin Brandon Williams
@ 2018-01-03 20:33   ` Stefan Beller
  2018-01-03 20:39     ` Brandon Williams
  0 siblings, 1 reply; 59+ messages in thread
From: Stefan Beller @ 2018-01-03 20:33 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> In order to allow for code sharing with the server-side of fetch in
> protocol-v2 convert upload-pack to be a builtin.

What is the security aspect of this patch?

By making upload-pack builtin, it gains additional abilities,
such as answers to '-h' or '--help' (which would start a pager).
Is there an easy way to sooth my concerns? (best put into the
commit message)

Thanks,
Stefan

>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  Makefile      | 3 ++-
>  builtin.h     | 1 +
>  git.c         | 1 +
>  upload-pack.c | 2 +-
>  4 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 2a81ae22e..e0740b452 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -636,7 +636,6 @@ PROGRAM_OBJS += imap-send.o
>  PROGRAM_OBJS += sh-i18n--envsubst.o
>  PROGRAM_OBJS += shell.o
>  PROGRAM_OBJS += show-index.o
> -PROGRAM_OBJS += upload-pack.o
>  PROGRAM_OBJS += remote-testsvn.o
>
>  # Binary suffix, set to .exe for Windows builds
> @@ -701,6 +700,7 @@ BUILT_INS += git-merge-subtree$X
>  BUILT_INS += git-show$X
>  BUILT_INS += git-stage$X
>  BUILT_INS += git-status$X
> +BUILT_INS += git-upload-pack$X
>  BUILT_INS += git-whatchanged$X
>
>  # what 'all' will build and 'install' will install in gitexecdir,
> @@ -904,6 +904,7 @@ LIB_OBJS += tree-diff.o
>  LIB_OBJS += tree.o
>  LIB_OBJS += tree-walk.o
>  LIB_OBJS += unpack-trees.o
> +LIB_OBJS += upload-pack.o
>  LIB_OBJS += url.o
>  LIB_OBJS += urlmatch.o
>  LIB_OBJS += usage.o
> diff --git a/builtin.h b/builtin.h
> index 42378f3aa..f332a1257 100644
> --- a/builtin.h
> +++ b/builtin.h
> @@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);
>  extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);
>  extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);
>  extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);
> +extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);
>  extern int cmd_var(int argc, const char **argv, const char *prefix);
>  extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);
>  extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);
> diff --git a/git.c b/git.c
> index c870b9719..f71073dc8 100644
> --- a/git.c
> +++ b/git.c
> @@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {
>         { "update-server-info", cmd_update_server_info, RUN_SETUP },
>         { "upload-archive", cmd_upload_archive },
>         { "upload-archive--writer", cmd_upload_archive_writer },
> +       { "upload-pack", cmd_upload_pack },
>         { "var", cmd_var, RUN_SETUP_GENTLY },
>         { "verify-commit", cmd_verify_commit, RUN_SETUP },
>         { "verify-pack", cmd_verify_pack },
> diff --git a/upload-pack.c b/upload-pack.c
> index d5de18127..20acaa49d 100644
> --- a/upload-pack.c
> +++ b/upload-pack.c
> @@ -1032,7 +1032,7 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
>         return parse_hide_refs_config(var, value, "uploadpack");
>  }
>
> -int cmd_main(int argc, const char **argv)
> +int cmd_upload_pack(int argc, const char **argv, const char *prefix)
>  {
>         const char *dir;
>         int strict = 0;
> --
> 2.15.1.620.gb9897f4670-goog
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 05/26] upload-pack: factor out processing lines
  2018-01-03  0:18 ` [PATCH 05/26] upload-pack: factor out processing lines Brandon Williams
@ 2018-01-03 20:38   ` Stefan Beller
  0 siblings, 0 replies; 59+ messages in thread
From: Stefan Beller @ 2018-01-03 20:38 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> Factor out the logic for processing shallow, deepen, deepen_since, and
> deepen_not lines into their own functions to simplify the
> 'receive_needs()' function in addition to making it easier to reuse some
> of this logic when implementing protocol_v2.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------
>  1 file changed, 74 insertions(+), 39 deletions(-)
>
> diff --git a/upload-pack.c b/upload-pack.c
> index 20acaa49d..9a507ae53 100644
> --- a/upload-pack.c
> +++ b/upload-pack.c
> @@ -731,6 +731,75 @@ static void deepen_by_rev_list(int ac, const char **av,
>         packet_flush(1);
>  }
>
> +static int process_shallow(const char *line, struct object_array *shallows)
> +{
> +       const char *arg;
> +       if (skip_prefix(line, "shallow ", &arg)) {

stylistic nit:

    You could invert the condition in each of the process_* functions
    to just have

        if (!skip_prefix...))
            return 0

        /* less indented code goes here */

        return 1;

    That way we have less indentation as well as easier code.
    (The reader doesn't need to keep in mind what the else
    part is about; it is a rather local decision to bail out instead
    of having the return at the end of the function.)


> +               struct object_id oid;
> +               struct object *object;
> +               if (get_oid_hex(arg, &oid))
> +                       die("invalid shallow line: %s", line);
> +               object = parse_object(&oid);
> +               if (!object)
> +                       return 1;
> +               if (object->type != OBJ_COMMIT)
> +                       die("invalid shallow object %s", oid_to_hex(&oid));
> +               if (!(object->flags & CLIENT_SHALLOW)) {
> +                       object->flags |= CLIENT_SHALLOW;
> +                       add_object_array(object, NULL, shallows);
> +               }
> +               return 1;
> +       }
> +
> +       return 0;
> +}
> +
> +static int process_deepen(const char *line, int *depth)
> +{
> +       const char *arg;
> +       if (skip_prefix(line, "deepen ", &arg)) {
> +               char *end = NULL;
> +               *depth = strtol(arg, &end, 0);
> +               if (!end || *end || depth <= 0)
> +                       die("Invalid deepen: %s", line);
> +               return 1;
> +       }
> +
> +       return 0;
> +}
> +
> +static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)
> +{
> +       const char *arg;
> +       if (skip_prefix(line, "deepen-since ", &arg)) {
> +               char *end = NULL;
> +               *deepen_since = parse_timestamp(arg, &end, 0);
> +               if (!end || *end || !deepen_since ||
> +                   /* revisions.c's max_age -1 is special */
> +                   *deepen_since == -1)
> +                       die("Invalid deepen-since: %s", line);
> +               *deepen_rev_list = 1;
> +               return 1;
> +       }
> +       return 0;
> +}
> +
> +static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)
> +{
> +       const char *arg;
> +       if (skip_prefix(line, "deepen-not ", &arg)) {
> +               char *ref = NULL;
> +               struct object_id oid;
> +               if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
> +                       die("git upload-pack: ambiguous deepen-not: %s", line);
> +               string_list_append(deepen_not, ref);
> +               free(ref);
> +               *deepen_rev_list = 1;
> +               return 1;
> +       }
> +       return 0;
> +}
> +
>  static void receive_needs(void)
>  {
>         struct object_array shallows = OBJECT_ARRAY_INIT;
> @@ -752,49 +821,15 @@ static void receive_needs(void)
>                 if (!line)
>                         break;
>
> -               if (skip_prefix(line, "shallow ", &arg)) {
> -                       struct object_id oid;
> -                       struct object *object;
> -                       if (get_oid_hex(arg, &oid))
> -                               die("invalid shallow line: %s", line);
> -                       object = parse_object(&oid);
> -                       if (!object)
> -                               continue;
> -                       if (object->type != OBJ_COMMIT)
> -                               die("invalid shallow object %s", oid_to_hex(&oid));
> -                       if (!(object->flags & CLIENT_SHALLOW)) {
> -                               object->flags |= CLIENT_SHALLOW;
> -                               add_object_array(object, NULL, &shallows);
> -                       }
> +               if (process_shallow(line, &shallows))
>                         continue;
> -               }
> -               if (skip_prefix(line, "deepen ", &arg)) {
> -                       char *end = NULL;
> -                       depth = strtol(arg, &end, 0);
> -                       if (!end || *end || depth <= 0)
> -                               die("Invalid deepen: %s", line);
> +               if (process_deepen(line, &depth))
>                         continue;
> -               }
> -               if (skip_prefix(line, "deepen-since ", &arg)) {
> -                       char *end = NULL;
> -                       deepen_since = parse_timestamp(arg, &end, 0);
> -                       if (!end || *end || !deepen_since ||
> -                           /* revisions.c's max_age -1 is special */
> -                           deepen_since == -1)
> -                               die("Invalid deepen-since: %s", line);
> -                       deepen_rev_list = 1;
> +               if (process_deepen_since(line, &deepen_since, &deepen_rev_list))
>                         continue;
> -               }
> -               if (skip_prefix(line, "deepen-not ", &arg)) {
> -                       char *ref = NULL;
> -                       struct object_id oid;
> -                       if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
> -                               die("git upload-pack: ambiguous deepen-not: %s", line);
> -                       string_list_append(&deepen_not, ref);
> -                       free(ref);
> -                       deepen_rev_list = 1;
> +               if (process_deepen_not(line, &deepen_not, &deepen_rev_list))
>                         continue;
> -               }
> +
>                 if (!skip_prefix(line, "want ", &arg) ||
>                     get_oid_hex(arg, &oid_buf))
>                         die("git upload-pack: protocol error, "
> --
> 2.15.1.620.gb9897f4670-goog
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 04/26] upload-pack: convert to a builtin
  2018-01-03 20:33   ` Stefan Beller
@ 2018-01-03 20:39     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-03 20:39 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On 01/03, Stefan Beller wrote:
> On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> > In order to allow for code sharing with the server-side of fetch in
> > protocol-v2 convert upload-pack to be a builtin.
> 
> What is the security aspect of this patch?
> 
> By making upload-pack builtin, it gains additional abilities,
> such as answers to '-h' or '--help' (which would start a pager).
> Is there an easy way to sooth my concerns? (best put into the
> commit message)

receive-pack is already a builtin, so theres that.

> 
> Thanks,
> Stefan
> 

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 06/26] transport: use get_refs_via_connect to get refs
  2018-01-03  0:18 ` [PATCH 06/26] transport: use get_refs_via_connect to get refs Brandon Williams
@ 2018-01-03 21:20   ` Stefan Beller
  0 siblings, 0 replies; 59+ messages in thread
From: Stefan Beller @ 2018-01-03 21:20 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> Remove code duplication and use the existing 'get_refs_via_connect()'
> function to retrieve a remote's heads in 'fetch_refs_via_pack()' and
> 'git_transport_push()'.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>

Reviewed-by: Stefan Beller <sbeller@google.com>

> ---
>  transport.c | 18 ++++--------------
>  1 file changed, 4 insertions(+), 14 deletions(-)
>
> diff --git a/transport.c b/transport.c
> index fc802260f..8e8779096 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,
>         args.cloning = transport->cloning;
>         args.update_shallow = data->options.update_shallow;
>
> -       if (!data->got_remote_heads) {
> -               connect_setup(transport, 0);
> -               get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
> -                                NULL, &data->shallow);
> -               data->got_remote_heads = 1;
> -       }
> +       if (!data->got_remote_heads)
> +               refs_tmp = get_refs_via_connect(transport, 0);
>
>         refs = fetch_pack(&args, data->fd, data->conn,
>                           refs_tmp ? refs_tmp : transport->remote_refs,
> @@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
>         struct send_pack_args args;
>         int ret;
>
> -       if (!data->got_remote_heads) {
> -               struct ref *tmp_refs;
> -               connect_setup(transport, 1);
> -
> -               get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,
> -                                NULL, &data->shallow);
> -               data->got_remote_heads = 1;
> -       }
> +       if (!data->got_remote_heads)
> +               get_refs_via_connect(transport, 1);
>
>         memset(&args, 0, sizeof(args));
>         args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
> --
> 2.15.1.620.gb9897f4670-goog
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 12/26] ls-refs: introduce ls-refs server command
  2018-01-03  0:18 ` [PATCH 12/26] ls-refs: introduce ls-refs server command Brandon Williams
@ 2018-01-04  0:17   ` Stefan Beller
  2018-01-05 23:49     ` Brandon Williams
  2018-01-09 20:50   ` Jonathan Tan
  1 sibling, 1 reply; 59+ messages in thread
From: Stefan Beller @ 2018-01-04  0:17 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> Introduce the ls-refs server command.  In protocol v2, the ls-refs
> command is used to request the ref advertisement from the server.  Since
> it is a command which can be requested (as opposed to mandatory in v1),
> a client can sent a number of parameters in its request to limit the ref
> advertisement based on provided ref-patterns.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  Documentation/technical/protocol-v2.txt | 26 +++++++++
>  Makefile                                |  1 +
>  ls-refs.c                               | 97 +++++++++++++++++++++++++++++++++
>  ls-refs.h                               |  9 +++

Maybe consider putting any served command into a sub directory?

For example the code in builtin/ has laxer rules w.r.t. die()ing
as it is a user facing command, whereas some devs want to see
code at the root of the repo to not die() at all as the eventual goal
is to have a library there.
All this code is on the remote side, which also has different traits than
the code at the root of the git.git repo; non-localisation comes to mind,
but there might be other aspects as well (security?).


>  serve.c                                 |  2 +
>  5 files changed, 135 insertions(+)
>  create mode 100644 ls-refs.c
>  create mode 100644 ls-refs.h
>
> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
> index b87ba3816..5f4d0e719 100644
> --- a/Documentation/technical/protocol-v2.txt
> +++ b/Documentation/technical/protocol-v2.txt
> @@ -89,3 +89,29 @@ terminate the connection.
>  Commands are the core actions that a client wants to perform (fetch, push,
>  etc).  Each command will be provided with a list capabilities and
>  arguments as requested by a client.
> +
> + Ls-refs

So is it ls-refs or Ls-refs or is any capitalization valid?

> +---------
> +
> +Ls-refs is the command used to request a reference advertisement in v2.
> +Unlike the current reference advertisement, ls-refs takes in parameters
> +which can be used to limit the refs sent from the server.
> +
> +Ls-ref takes in the following parameters wraped in packet-lines:
> +
> +  symrefs: In addition to the object pointed by it, show the underlying
> +          ref pointed by it when showing a symbolic ref.
> +  peel: Show peeled tags.
> +  ref-pattern <pattern>: When specified, only references matching the
> +                        given patterns are displayed.

What kind of pattern matching is allowed here?
strictly prefix only, or globbing, regexes?
Is there a given grammar to follow? Maybe a link to the git
glossary is or somewhere else might be fine.

Seeing that we do wildmatch() down there (as opposed to regexes),
I wonder if it provides an entry for a denial of service attack, by crafting
a pattern that is very expensive for the server to compute but cheap to
ask for from a client. (c.f. 94da9193a6 (grep: add support for PCRE v2,
2017-06-01, but that is regexes!)

> +The output of ls-refs is as follows:
> +
> +    output = *ref
> +            flush-pkt
> +    ref = PKT-LINE((tip | peeled) LF)
> +    tip = obj-id SP refname (SP symref-target)
> +    peeled = obj-id SP refname "^{}"
> +
> +    symref = PKT-LINE("symref" SP symbolic-ref SP resolved-ref LF)
> +    shallow = PKT-LINE("shallow" SP obj-id LF)
> diff --git a/Makefile b/Makefile
> index 5f3b5fe8b..152a73bec 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -820,6 +820,7 @@ LIB_OBJS += list-objects-filter-options.o
>  LIB_OBJS += ll-merge.o
>  LIB_OBJS += lockfile.o
>  LIB_OBJS += log-tree.o
> +LIB_OBJS += ls-refs.o
>  LIB_OBJS += mailinfo.o
>  LIB_OBJS += mailmap.o
>  LIB_OBJS += match-trees.o
> diff --git a/ls-refs.c b/ls-refs.c
> new file mode 100644
> index 000000000..ac4904a40
> --- /dev/null
> +++ b/ls-refs.c
> @@ -0,0 +1,97 @@
> +#include "cache.h"
> +#include "repository.h"
> +#include "refs.h"
> +#include "remote.h"
> +#include "argv-array.h"
> +#include "ls-refs.h"
> +#include "pkt-line.h"
> +
> +struct ls_refs_data {
> +       unsigned peel;
> +       unsigned symrefs;
> +       struct argv_array patterns;
> +};
> +
> +/*
> + * Check if one of the patterns matches the tail part of the ref.
> + * If no patterns were provided, all refs match.
> + */
> +static int ref_match(const struct argv_array *patterns, const char *refname)
> +{
> +       char *pathbuf;
> +       int i;
> +
> +       if (!patterns->argc)
> +               return 1; /* no restriction */
> +
> +       pathbuf = xstrfmt("/%s", refname);
> +       for (i = 0; i < patterns->argc; i++) {
> +               if (!wildmatch(patterns->argv[i], pathbuf, 0)) {
> +                       free(pathbuf);
> +                       return 1;
> +               }
> +       }
> +       free(pathbuf);
> +       return 0;
> +}
> +
> +static int send_ref(const char *refname, const struct object_id *oid,
> +                   int flag, void *cb_data)
> +{
> +       struct ls_refs_data *data = cb_data;
> +       const char *refname_nons = strip_namespace(refname);
> +       struct strbuf refline = STRBUF_INIT;
> +
> +       if (!ref_match(&data->patterns, refname))
> +               return 0;
> +
> +       strbuf_addf(&refline, "%s %s", oid_to_hex(oid), refname_nons);
> +       if (data->symrefs && flag & REF_ISSYMREF) {
> +               struct object_id unused;
> +               const char *symref_target = resolve_ref_unsafe(refname, 0,
> +                                                              &unused,
> +                                                              &flag);
> +
> +               if (!symref_target)
> +                       die("'%s' is a symref but it is not?", refname);
> +
> +               strbuf_addf(&refline, " %s", symref_target);
> +       }
> +
> +       strbuf_addch(&refline, '\n');
> +
> +       packet_write(1, refline.buf, refline.len);
> +       if (data->peel) {
> +               struct object_id peeled;
> +               if (!peel_ref(refname, &peeled))
> +                       packet_write_fmt(1, "%s %s^{}\n", oid_to_hex(&peeled),
> +                                        refname_nons);
> +       }
> +
> +       strbuf_release(&refline);
> +       return 0;
> +}
> +
> +int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)
> +{
> +       int i;
> +       struct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };
> +
> +       for (i = 0; i < args->argc; i++) {
> +               const char *arg = args->argv[i];
> +               const char *out;
> +
> +               if (!strcmp("peel", arg))
> +                       data.peel = 1;
> +               else if (!strcmp("symrefs", arg))
> +                       data.symrefs = 1;
> +               else if (skip_prefix(arg, "ref-pattern ", &out))
> +                       argv_array_pushf(&data.patterns, "*/%s", out);
> +       }
> +
> +       head_ref_namespaced(send_ref, &data);
> +       for_each_namespaced_ref(send_ref, &data);
> +       packet_flush(1);
> +       argv_array_clear(&data.patterns);
> +       return 0;
> +}
> diff --git a/ls-refs.h b/ls-refs.h
> new file mode 100644
> index 000000000..9e4c57bfe
> --- /dev/null
> +++ b/ls-refs.h
> @@ -0,0 +1,9 @@
> +#ifndef LS_REFS_H
> +#define LS_REFS_H
> +
> +struct repository;
> +struct argv_array;
> +extern int ls_refs(struct repository *r, struct argv_array *keys,
> +                  struct argv_array *args);
> +
> +#endif /* LS_REFS_H */
> diff --git a/serve.c b/serve.c
> index da8127775..88d548410 100644
> --- a/serve.c
> +++ b/serve.c
> @@ -4,6 +4,7 @@
>  #include "pkt-line.h"
>  #include "version.h"
>  #include "argv-array.h"
> +#include "ls-refs.h"
>  #include "serve.h"
>
>  static int always_advertise(struct repository *r,
> @@ -44,6 +45,7 @@ struct protocol_capability {
>  static struct protocol_capability capabilities[] = {
>         { "agent", agent_advertise, NULL },
>         { "stateless-rpc", always_advertise, NULL },
> +       { "ls-refs", always_advertise, ls_refs },
>  };
>
>  static void advertise_capabilities(void)
> --
> 2.15.1.620.gb9897f4670-goog
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 19/26] upload-pack: introduce fetch server command
  2018-01-03  0:18 ` [PATCH 19/26] upload-pack: introduce fetch server command Brandon Williams
@ 2018-01-04  1:07   ` Stefan Beller
  0 siblings, 0 replies; 59+ messages in thread
From: Stefan Beller @ 2018-01-04  1:07 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> Introduce the 'fetch' server command.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  Documentation/technical/protocol-v2.txt |  14 ++
>  serve.c                                 |   2 +
>  upload-pack.c                           | 290 ++++++++++++++++++++++++++++++++
>  upload-pack.h                           |   9 +
>  4 files changed, 315 insertions(+)
>  create mode 100644 upload-pack.h
>
> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
> index 5f4d0e719..2a8e2f226 100644
> --- a/Documentation/technical/protocol-v2.txt
> +++ b/Documentation/technical/protocol-v2.txt
> @@ -115,3 +115,17 @@ The output of ls-refs is as follows:
>
>      symref = PKT-LINE("symref" SP symbolic-ref SP resolved-ref LF)
>      shallow = PKT-LINE("shallow" SP obj-id LF)
> +
> + Fetch
> +-------
> +
> +Fetch will need to be a modified version of the v1 fetch protocol.  Some
> +potential areas for improvement are: Ref-in-want, CDN offloading,
> +Fetch-options.
> +
> +Since we'll have an 'ls-ref' service we can eliminate the need of fetch
> +to perform a ref-advertisement, instead a client can run the 'ls-refs'
> +service first, in order to find out what refs the server has, and then
> +request those refs directly using the fetch service.
> +
> +//TODO Flesh out the design

TODO: actually do it. ;)

a couple notes from the discussion in office:
* Could we split fetch into multiple phases
  (negotiation + getting the pack)
* negotiation could be reused in forced push to
  minimize the pack to be sent
* negotiation in a half duplex is actually better
  called 'discovery', which discovers about the set
  of objects available on the remote side.
  (the opposite would be reveal, or 'ask-for-discovery', which
  is could be used for a symmetric design of fetch and push)


> diff --git a/serve.c b/serve.c
> index 88d548410..ca3bb7190 100644
> --- a/serve.c
> +++ b/serve.c
> @@ -6,6 +6,7 @@
>  #include "argv-array.h"
>  #include "ls-refs.h"
>  #include "serve.h"
> +#include "upload-pack.h"
>
>  static int always_advertise(struct repository *r,
>                             struct strbuf *value)
> @@ -46,6 +47,7 @@ static struct protocol_capability capabilities[] = {
>         { "agent", agent_advertise, NULL },
>         { "stateless-rpc", always_advertise, NULL },
>         { "ls-refs", always_advertise, ls_refs },
> +       { "fetch", always_advertise, upload_pack_v2 },
>  };
>
>  static void advertise_capabilities(void)
> diff --git a/upload-pack.c b/upload-pack.c
> index 2ca60d27c..c41f6f528 100644
> --- a/upload-pack.c
> +++ b/upload-pack.c
> @@ -20,6 +20,7 @@
>  #include "prio-queue.h"
>  #include "protocol.h"
>  #include "serve.h"
> +#include "upload-pack.h"
>
>  static const char * const upload_pack_usage[] = {
>         N_("git upload-pack [<options>] <dir>"),
> @@ -1040,6 +1041,295 @@ static void upload_pack(void)
>         }
>  }
>
> +struct upload_pack_data {
> +       struct object_array wants;
> +       struct oid_array haves;
> +
> +       unsigned stateless_rpc : 1;
> +
> +       unsigned use_thin_pack : 1;
> +       unsigned use_ofs_delta : 1;
> +       unsigned no_progress : 1;
> +       unsigned use_include_tag : 1;
> +       unsigned done : 1;
> +};
> +
> +#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }
> +
> +static void upload_pack_data_clear(struct upload_pack_data *data)
> +{
> +       object_array_clear(&data->wants);
> +       oid_array_clear(&data->haves);
> +}
> +
> +static int parse_want(const char *line)
> +{
> +       const char *arg;
> +       if (skip_prefix(line, "want ", &arg)) {
> +               struct object_id oid;
> +               struct object *o;
> +
> +               if (get_oid_hex(arg, &oid))
> +                       die("git upload-pack: protocol error, "
> +                           "expected to get oid, not '%s'", line);
> +
> +               o = parse_object(&oid);
> +               if (!o) {
> +                       packet_write_fmt(1,
> +                                        "ERR upload-pack: not our ref %s",
> +                                        oid_to_hex(&oid));
> +                       die("git upload-pack: not our ref %s",
> +                           oid_to_hex(&oid));
> +               }
> +
> +               if (!(o->flags & WANTED)) {
> +                       o->flags |= WANTED;
> +                       add_object_array(o, NULL, &want_obj);
> +               }
> +
> +               return 1;
> +       }
> +
> +       return 0;
> +}
> +
> +static int parse_have(const char *line, struct oid_array *haves)
> +{
> +       const char *arg;
> +       if (skip_prefix(line, "have ", &arg)) {
> +               struct object_id oid;
> +
> +               if (get_oid_hex(arg, &oid))
> +                       die("git upload-pack: expected SHA1 object, got '%s'", arg);
> +               oid_array_append(haves, &oid);
> +               return 1;
> +       }

The same stylistic nit as in an earlier patch: Maybe
shortcircuit by inverting the condition?



> +
> +       return 0;
> +}
> +
> +static void process_args(struct argv_array *args, struct upload_pack_data *data)
> +{
> +       int i;
> +
> +       for (i = 0; i < args->argc; i++) {
> +               const char *arg = args->argv[i];
> +
> +               /* process want */
> +               if (parse_want(arg))
> +                       continue;
> +               /* process have line */
> +               if (parse_have(arg, &data->haves))
> +                       continue;
> +
> +               /* process args like thin-pack */
> +               if (!strcmp(arg, "thin-pack")) {
> +                       use_thin_pack = 1;
> +                       continue;
> +               }
> +               if (!strcmp(arg, "ofs-delta")) {
> +                       use_ofs_delta = 1;
> +                       continue;
> +               }
> +               if (!strcmp(arg, "no-progress")) {
> +                       no_progress = 1;
> +                       continue;
> +               }
> +               if (!strcmp(arg, "include-tag")) {
> +                       use_include_tag = 1;
> +                       continue;
> +               }
> +               if (!strcmp(arg, "done")) {
> +                       data->done = 1;
> +                       continue;
> +               }
> +
> +               /* ignore unknown lines maybe? */
> +               die("unexpect line: '%s'", arg);
> +       }
> +}
> +
> +static void read_haves(struct upload_pack_data *data)
> +{
> +       struct packet_reader reader;
> +       packet_reader_init(&reader, 0, NULL, 0,
> +                          PACKET_READ_CHOMP_NEWLINE);
> +
> +       while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {
> +
> +               if (parse_have(reader.line, &data->haves))
> +                       continue;
> +               if (!strcmp(reader.line, "done")) {
> +                       data->done = 1;
> +                       continue;
> +               }
> +       }
> +       if (reader.status != PACKET_READ_FLUSH)
> +               die("ERROR");
> +}
> +
> +static int process_haves(struct oid_array *haves, struct oid_array *common)
> +{
> +       int i;
> +
> +       /* Process haves */
> +       for (i = 0; i < haves->nr; i++) {
> +               const struct object_id *oid = &haves->oid[i];
> +               struct object *o;
> +               int we_knew_they_have = 0;
> +
> +               if (!has_object_file(oid))
> +                       continue;
> +
> +               oid_array_append(common, oid);
> +
> +               o = parse_object(oid);
> +               if (!o)
> +                       die("oops (%s)", oid_to_hex(oid));
> +               if (o->type == OBJ_COMMIT) {
> +                       struct commit_list *parents;
> +                       struct commit *commit = (struct commit *)o;
> +                       if (o->flags & THEY_HAVE)
> +                               we_knew_they_have = 1;
> +                       else
> +                               o->flags |= THEY_HAVE;
> +                       if (!oldest_have || (commit->date < oldest_have))
> +                               oldest_have = commit->date;
> +                       for (parents = commit->parents;
> +                            parents;
> +                            parents = parents->next)
> +                               parents->item->object.flags |= THEY_HAVE;
> +               }
> +               if (!we_knew_they_have)
> +                       add_object_array(o, NULL, &have_obj);
> +       }
> +
> +       return 0;
> +}
> +
> +static int send_acks(struct oid_array *acks, struct strbuf *response)
> +{
> +       int i;
> +       /* Send Acks */
> +       if (!acks->nr)
> +               packet_buf_write(response, "NAK\n");
> +
> +       for (i = 0; i < acks->nr; i++) {
> +               packet_buf_write(response, "ACK %s common\n",
> +                                oid_to_hex(&acks->oid[i]));
> +       }
> +
> +       if (ok_to_give_up()) {
> +               /* Send Ready */
> +               packet_buf_write(response, "ACK %s ready\n",
> +                                oid_to_hex(&acks->oid[i-1]));
> +               return 1;
> +       }
> +
> +       return 0;
> +}
> +
> +static int process_haves_and_send_acks(struct upload_pack_data *data)
> +{
> +       struct oid_array common = OID_ARRAY_INIT;
> +       struct strbuf response = STRBUF_INIT;
> +       int ret = 0;
> +
> +       process_haves(&data->haves, &common);
> +       if (data->done) {
> +               ret = 1;
> +       } else if (send_acks(&common, &response)) {
> +               packet_buf_delim(&response);
> +               ret = 1;
> +       } else {
> +               /* Add Flush */
> +               packet_buf_flush(&response);
> +               ret = 0;
> +       }
> +
> +       /* Send response */
> +       write_or_die(1, response.buf, response.len);
> +       strbuf_release(&response);
> +
> +       oid_array_clear(&data->haves);
> +       oid_array_clear(&common);
> +       return ret;
> +}
> +
> +#define FETCH_PROCESS_ARGS 0
> +#define FETCH_READ_HAVES 1
> +#define FETCH_SEND_ACKS 2
> +#define FETCH_SEND_PACK 3
> +#define FETCH_DONE 4
> +
> +int upload_pack_v2(struct repository *r, struct argv_array *keys,
> +                  struct argv_array *args)
> +{
> +       int state = FETCH_PROCESS_ARGS;
> +       struct upload_pack_data data = UPLOAD_PACK_DATA_INIT;
> +       const char *out;
> +       use_sideband = LARGE_PACKET_MAX;
> +
> +       /* Check if cmd is being run as a stateless-rpc */
> +       if (has_capability(keys, "stateless-rpc", &out))
> +               if (!strcmp(out, "true"))
> +                       data.stateless_rpc = 1;
> +
> +       while (state != FETCH_DONE) {
> +               switch (state) {
> +               case FETCH_PROCESS_ARGS:
> +                       process_args(args, &data);
> +
> +                       if (!want_obj.nr) {
> +                               /*
> +                                * Request didn't contain any 'want' lines,
> +                                * guess they didn't want anything.
> +                                */
> +                               state = FETCH_DONE;
> +                       } else if (data.haves.nr) {
> +                               /*
> +                                * Request had 'have' lines, so lets ACK them.
> +                                */
> +                               state = FETCH_SEND_ACKS;
> +                       } else {
> +                               /*
> +                                * Request had 'want's but no 'have's so we can
> +                                * immedietly go to construct and send a pack.
> +                                */
> +                               state = FETCH_SEND_PACK;
> +                       }
> +                       break;
> +               case FETCH_READ_HAVES:
> +                       read_haves(&data);
> +                       state = FETCH_SEND_ACKS;
> +                       break;
> +               case FETCH_SEND_ACKS:
> +                       if (process_haves_and_send_acks(&data))
> +                               state = FETCH_SEND_PACK;
> +                       else if (data.stateless_rpc)
> +                               /*
> +                                * Request was made via stateless-rpc and a
> +                                * packfile isn't ready to be created and sent.
> +                                */
> +                               state = FETCH_DONE;
> +                       else
> +                               state = FETCH_READ_HAVES;
> +                       break;
> +               case FETCH_SEND_PACK:
> +                       create_pack_file();
> +                       state = FETCH_DONE;
> +                       break;
> +               case FETCH_DONE:
> +                       break;
> +               default:
> +                       BUG("invalid state");
> +               }
> +       }
> +
> +       upload_pack_data_clear(&data);
> +       return 0;
> +}
> +
>  static int upload_pack_config(const char *var, const char *value, void *unused)
>  {
>         if (!strcmp("uploadpack.allowtipsha1inwant", var)) {
> diff --git a/upload-pack.h b/upload-pack.h
> new file mode 100644
> index 000000000..54c429563
> --- /dev/null
> +++ b/upload-pack.h
> @@ -0,0 +1,9 @@
> +#ifndef UPLOAD_PACK_H
> +#define UPLOAD_PACK_H
> +
> +struct repository;
> +struct argv_array;
> +extern int upload_pack_v2(struct repository *r, struct argv_array *keys,
> +                         struct argv_array *args);
> +
> +#endif /* UPLOAD_PACK_H */
> --
> 2.15.1.620.gb9897f4670-goog
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 20/26] fetch-pack: perform a fetch using v2
  2018-01-03  0:18 ` [PATCH 20/26] fetch-pack: perform a fetch using v2 Brandon Williams
@ 2018-01-04  1:23   ` Stefan Beller
  2018-01-05 23:55     ` Brandon Williams
  2018-01-10  0:05   ` Jonathan Tan
  1 sibling, 1 reply; 59+ messages in thread
From: Stefan Beller @ 2018-01-04  1:23 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> When communicating with a v2 server, perform a fetch by requesting the
> 'fetch' command.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  builtin/fetch-pack.c   |   2 +-
>  fetch-pack.c           | 267 ++++++++++++++++++++++++++++++++++++++++++++++++-
>  fetch-pack.h           |   4 +-
>  t/t5701-protocol-v2.sh |  40 ++++++++
>  transport.c            |   8 +-
>  5 files changed, 314 insertions(+), 7 deletions(-)
>
> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
> index f492e8abd..867dd3cc7 100644
> --- a/builtin/fetch-pack.c
> +++ b/builtin/fetch-pack.c
> @@ -213,7 +213,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
>         }
>
>         ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,
> -                        &shallow, pack_lockfile_ptr);
> +                        &shallow, pack_lockfile_ptr, protocol_v0);
>         if (pack_lockfile) {
>                 printf("lock %s\n", pack_lockfile);
>                 fflush(stdout);
> diff --git a/fetch-pack.c b/fetch-pack.c
> index 9f6b07ad9..c26fdc539 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -1008,6 +1008,262 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
>         return ref;
>  }
>
> +static void add_wants(const struct ref *wants, struct strbuf *req_buf)
> +{
> +       for ( ; wants ; wants = wants->next) {
> +               const struct object_id *remote = &wants->old_oid;
> +               const char *remote_hex;
> +               struct object *o;
> +
> +               /*
> +                * If that object is complete (i.e. it is an ancestor of a
> +                * local ref), we tell them we have it but do not have to
> +                * tell them about its ancestors, which they already know
> +                * about.
> +                *
> +                * We use lookup_object here because we are only
> +                * interested in the case we *know* the object is
> +                * reachable and we have already scanned it.
> +                */
> +               if (((o = lookup_object(remote->hash)) != NULL) &&
> +                   (o->flags & COMPLETE)) {
> +                       continue;
> +               }
> +
> +               remote_hex = oid_to_hex(remote);
> +               packet_buf_write(req_buf, "want %s\n", remote_hex);
> +       }
> +}
> +
> +static int add_haves(struct strbuf *req_buf, int *in_vain)
> +{
> +       int ret = 0;
> +       int haves_added = 0;
> +       const struct object_id *oid;
> +
> +       while ((oid = get_rev())) {
> +               packet_buf_write(req_buf, "have %s\n", oid_to_hex(oid));
> +               if (++haves_added >= INITIAL_FLUSH)
> +                       break;
> +       };
> +
> +       *in_vain += haves_added;
> +       if (!haves_added || *in_vain >= MAX_IN_VAIN) {
> +               /* Send Done */
> +               packet_buf_write(req_buf, "done\n");
> +               ret = 1;
> +       }
> +
> +       return ret;
> +}
> +
> +static int send_haves(int fd_out, int *in_vain)
> +{
> +       int ret = 0;
> +       struct strbuf req_buf = STRBUF_INIT;
> +
> +       ret = add_haves(&req_buf, in_vain);
> +
> +       /* Send request */
> +       packet_buf_flush(&req_buf);
> +       write_or_die(fd_out, req_buf.buf, req_buf.len);
> +
> +       strbuf_release(&req_buf);
> +       return ret;
> +}
> +
> +static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,
> +                             const struct ref *wants, struct oidset *common,
> +                             int *in_vain)
> +{
> +       int ret = 0;
> +       struct strbuf req_buf = STRBUF_INIT;
> +
> +       packet_buf_write(&req_buf, "command=fetch");
> +       packet_buf_write(&req_buf, "agent=%s", git_user_agent_sanitized());
> +       if (args->stateless_rpc)
> +               packet_buf_write(&req_buf, "stateless-rpc=true");
> +
> +       packet_buf_delim(&req_buf);
> +       if (args->use_thin_pack)
> +               packet_buf_write(&req_buf, "thin-pack");
> +       if (args->no_progress)
> +               packet_buf_write(&req_buf, "no-progress");
> +       if (args->include_tag)
> +               packet_buf_write(&req_buf, "include-tag");
> +       if (prefer_ofs_delta)
> +               packet_buf_write(&req_buf, "ofs-delta");
> +
> +       /* add wants */
> +       add_wants(wants, &req_buf);
> +
> +       /*
> +        * If we are running stateless-rpc we need to add all the common
> +        * commits we've found in previous rounds
> +        */
> +       if (args->stateless_rpc) {
> +               struct oidset_iter iter;
> +               const struct object_id *oid;
> +               oidset_iter_init(common, &iter);
> +
> +               while ((oid = oidset_iter_next(&iter))) {
> +                       packet_buf_write(&req_buf, "have %s\n", oid_to_hex(oid));
> +               }
> +       }
> +
> +       /* Add initial haves */
> +       ret = add_haves(&req_buf, in_vain);
> +
> +       /* Send request */
> +       packet_buf_flush(&req_buf);
> +       write_or_die(fd_out, req_buf.buf, req_buf.len);
> +
> +       strbuf_release(&req_buf);
> +       return ret;
> +}
> +
> +static enum ack_type process_ack(const char *line, struct object_id *oid)
> +{
> +       const char *arg;
> +
> +       if (!strcmp(line, "NAK"))
> +               return NAK;
> +       if (skip_prefix(line, "ACK ", &arg)) {
> +               if (!parse_oid_hex(arg, oid, &arg)) {
> +                       if (strstr(arg, "continue"))
> +                               return ACK_continue;
> +                       if (strstr(arg, "common"))
> +                               return ACK_common;
> +                       if (strstr(arg, "ready"))
> +                               return ACK_ready;
> +                       return ACK;
> +               }
> +       }
> +       if (skip_prefix(line, "ERR ", &arg))
> +               die(_("remote error: %s"), arg);
> +       die(_("git fetch-pack: expected ACK/NAK, got '%s'"), line);
> +}
> +
> +static int process_acks(struct packet_reader *reader, struct oidset *common)
> +{
> +       int got_ready = 0;
> +       int got_common = 0;
> +       while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
> +               struct object_id oid;
> +               struct commit *commit;
> +               enum ack_type ack = process_ack(reader->line, &oid);
> +
> +               switch (ack) {
> +               case ACK_ready:
> +                       clear_prio_queue(&rev_list);
> +                       got_ready = 1;
> +                       /* fallthrough */
> +               case ACK_common:
> +                       oidset_insert(common, &oid);
> +                       commit = lookup_commit(&oid);
> +                       mark_common(commit, 0, 1);
> +                       got_common = 1;
> +                       break;
> +               case NAK:
> +                       break;
> +               case ACK:
> +               case ACK_continue:
> +                       die("ACK/ACK_continue not supported");
> +               }
> +       }
> +
> +       if (reader->status != PACKET_READ_FLUSH &&
> +           reader->status != PACKET_READ_DELIM)
> +               die("Error during processing acks: %d", reader->status);
> +
> +       /* return 0 if no common, 1 if there are common, or 2 if ready */
> +       return got_ready + got_common;
> +}
> +
> +#define FETCH_CHECK_LOCAL 0
> +#define FETCH_SEND_REQUEST 1
> +#define FETCH_PROCESS_ACKS 2
> +#define FETCH_SEND_HAVES 3
> +#define FETCH_GET_PACK 4
> +#define FETCH_DONE 5
> +
> +static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
> +                                   int fd[2],
> +                                   const struct ref *orig_ref,
> +                                   struct ref **sought, int nr_sought,
> +                                   char **pack_lockfile)
> +{
> +       struct ref *ref = copy_ref_list(orig_ref);
> +       int state = FETCH_CHECK_LOCAL;

Is there any reason to use #defines over an enum here?

> +       struct oidset common = OIDSET_INIT;
> +       struct packet_reader reader;
> +       int in_vain = 0;
> +       packet_reader_init(&reader, fd[0], NULL, 0,
> +                          PACKET_READ_CHOMP_NEWLINE);
> +
> +       while (state != FETCH_DONE) {
> +               switch (state) {
> +               case FETCH_CHECK_LOCAL:
> +                       sort_ref_list(&ref, ref_compare_name);
> +                       QSORT(sought, nr_sought, cmp_ref_by_name);
> +
> +                       /* v2 supports these by default */
> +                       allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;
> +                       use_sideband = 2;
> +
> +                       /* Filter 'ref' by 'sought' and those that aren't local */
> +                       if (everything_local(args, &ref, sought, nr_sought))
> +                               state = FETCH_DONE;
> +                       else
> +                               state = FETCH_SEND_REQUEST;
> +                       break;
> +               case FETCH_SEND_REQUEST:
> +                       if (send_fetch_request(fd[1], args, ref, &common, &in_vain))
> +                               state = FETCH_GET_PACK;
> +                       else
> +                               state = FETCH_PROCESS_ACKS;
> +                       break;
> +               case FETCH_PROCESS_ACKS:
> +                       /* Process ACKs/NAKs */
> +                       switch (process_acks(&reader, &common)) {
> +                       case 2:
> +                               state = FETCH_GET_PACK;
> +                               break;
> +                       case 1:
> +                               in_vain = 0;
> +                               /* fallthrough */
> +                       default:
> +                               if (args->stateless_rpc)
> +                                       state = FETCH_SEND_REQUEST;
> +                               else
> +                                       state = FETCH_SEND_HAVES;
> +                               break;
> +                       }
> +                       break;
> +               case FETCH_SEND_HAVES:
> +                       if (send_haves(fd[1], &in_vain))
> +                               state = FETCH_GET_PACK;
> +                       else
> +                               state = FETCH_PROCESS_ACKS;
> +                       break;
> +               case FETCH_GET_PACK:
> +                       /* get the pack */
> +                       if (get_pack(args, fd, pack_lockfile))
> +                               die(_("git fetch-pack: fetch failed."));
> +
> +                       state = FETCH_DONE;
> +                       break;
> +               case FETCH_DONE:
> +                       break;
> +               default:
> +                       die("invalid state");
> +               }
> +       }
> +
> +       oidset_clear(&common);
> +       return ref;
> +}
> +
>  static void fetch_pack_config(void)
>  {
>         git_config_get_int("fetch.unpacklimit", &fetch_unpack_limit);
> @@ -1153,7 +1409,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
>                        const char *dest,
>                        struct ref **sought, int nr_sought,
>                        struct oid_array *shallow,
> -                      char **pack_lockfile)
> +                      char **pack_lockfile,
> +                      enum protocol_version version)
>  {
>         struct ref *ref_cpy;
>         struct shallow_info si;
> @@ -1167,8 +1424,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
>                 die(_("no matching remote head"));
>         }
>         prepare_shallow_info(&si, shallow);
> -       ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
> -                               &si, pack_lockfile);
> +       if (version == protocol_v2)
> +               ref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,
> +                                          pack_lockfile);
> +       else
> +               ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
> +                                       &si, pack_lockfile);
>         reprepare_packed_git();
>         update_shallow(args, sought, nr_sought, &si);
>         clear_shallow_info(&si);
> diff --git a/fetch-pack.h b/fetch-pack.h
> index b6aeb43a8..7afca7305 100644
> --- a/fetch-pack.h
> +++ b/fetch-pack.h
> @@ -3,6 +3,7 @@
>
>  #include "string-list.h"
>  #include "run-command.h"
> +#include "protocol.h"
>
>  struct oid_array;
>
> @@ -43,7 +44,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
>                        struct ref **sought,
>                        int nr_sought,
>                        struct oid_array *shallow,
> -                      char **pack_lockfile);
> +                      char **pack_lockfile,
> +                      enum protocol_version version);
>
>  /*
>   * Print an appropriate error message for each sought ref that wasn't
> diff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh
> index 7d8aeb766..3e411e178 100755
> --- a/t/t5701-protocol-v2.sh
> +++ b/t/t5701-protocol-v2.sh
> @@ -33,4 +33,44 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol
>         ! grep "refs/tags/" log
>  '
>
> +test_expect_success 'clone with file:// using protocol v2' '
> +       GIT_TRACE_PACKET=1 git -c protocol.version=2 \
> +               clone "file://$(pwd)/file_parent" file_child 2>log &&
> +
> +       git -C file_child log -1 --format=%s >actual &&
> +       git -C file_parent log -1 --format=%s >expect &&
> +       test_cmp expect actual &&
> +
> +       # Server responded using protocol v1
> +       grep "clone< version 2" log
> +'
> +
> +test_expect_success 'fetch with file:// using protocol v2' '
> +       test_commit -C file_parent two &&
> +
> +       GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \
> +               fetch origin 2>log &&
> +
> +       git -C file_child log -1 --format=%s origin/master >actual &&
> +       git -C file_parent log -1 --format=%s >expect &&
> +       test_cmp expect actual &&
> +
> +       # Server responded using protocol v1
> +       grep "fetch< version 2" log
> +'
> +
> +test_expect_success 'ref advertisment is filtered during fetch using protocol v2' '
> +       test_commit -C file_parent three &&
> +
> +       GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \
> +               fetch origin master 2>log &&
> +
> +       git -C file_child log -1 --format=%s origin/master >actual &&
> +       git -C file_parent log -1 --format=%s >expect &&
> +       test_cmp expect actual &&
> +
> +       grep "ref-pattern master" log &&
> +       ! grep "refs/tags/" log
> +'
> +
>  test_done
> diff --git a/transport.c b/transport.c
> index 6ea3905e3..4fdbd9adc 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -256,14 +256,18 @@ static int fetch_refs_via_pack(struct transport *transport,
>
>         switch (data->version) {
>         case protocol_v2:
> -               die("support for protocol v2 not implemented yet");
> +               refs = fetch_pack(&args, data->fd, data->conn,
> +                                 refs_tmp ? refs_tmp : transport->remote_refs,
> +                                 dest, to_fetch, nr_heads, &data->shallow,
> +                                 &transport->pack_lockfile, data->version);
> +               packet_flush(data->fd[1]);
>                 break;
>         case protocol_v1:
>         case protocol_v0:
>                 refs = fetch_pack(&args, data->fd, data->conn,
>                                   refs_tmp ? refs_tmp : transport->remote_refs,
>                                   dest, to_fetch, nr_heads, &data->shallow,
> -                                 &transport->pack_lockfile);
> +                                 &transport->pack_lockfile, data->version);
>                 break;
>         case protocol_unknown_version:
>                 BUG("unknown protocol version");
> --
> 2.15.1.620.gb9897f4670-goog
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 01/26] pkt-line: introduce packet_read_with_status
  2018-01-03 19:27   ` Stefan Beller
@ 2018-01-05 23:41     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-05 23:41 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On 01/03, Stefan Beller wrote:
> On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> > The current pkt-line API encodes the status of a pkt-line read in the
> > length of the read content.  An error is indicated with '-1', a flush
> > with '0' (which can be confusing since a return value of '0' can also
> > indicate an empty pkt-line), and a positive integer for the length of
> > the read content otherwise.  This doesn't leave much room for allowing
> > the addition of additional special packets in the future.
> >
> > To solve this introduce 'packet_read_with_status()' which reads a packet
> > and returns the status of the read encoded as an 'enum packet_status'
> > type.  This allows for easily identifying between special and normal
> > packets as well as errors.  It also enables easily adding a new special
> > packet in the future.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> >  pkt-line.c | 55 ++++++++++++++++++++++++++++++++++++++++++-------------
> >  pkt-line.h | 15 +++++++++++++++
> >  2 files changed, 57 insertions(+), 13 deletions(-)
> >
> > diff --git a/pkt-line.c b/pkt-line.c
> > index 2827ca772..8d7cd389f 100644
> > --- a/pkt-line.c
> > +++ b/pkt-line.c
> > @@ -280,28 +280,33 @@ static int packet_length(const char *linelen)
> >         return (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);
> >  }
> >
> > -int packet_read(int fd, char **src_buf, size_t *src_len,
> > -               char *buffer, unsigned size, int options)
> > +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
> > +                                               char *buffer, unsigned size, int *pktlen,
> > +                                               int options)
> >  {
> > -       int len, ret;
> > +       int len;
> >         char linelen[4];
> >
> > -       ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);
> > -       if (ret < 0)
> > -               return ret;
> > +       if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)
> > +               return PACKET_READ_EOF;
> > +
> >         len = packet_length(linelen);
> >         if (len < 0)
> >                 die("protocol error: bad line length character: %.4s", linelen);
> > -       if (!len) {
> > +
> > +       if (len == 0) {
> >                 packet_trace("0000", 4, 0);
> > -               return 0;
> > +               return PACKET_READ_FLUSH;
> > +       } else if (len >= 1 && len <= 3) {
> > +               die("protocol error: bad line length character: %.4s", linelen);
> 
> I wonder how much libified code we want here already, maybe we could
> have PACKET_READ_ERROR as a return value here instead of die()ing.
> There could also be an option that tells this code to die on error, this reminds
> me of the repository discovery as well as the refs code, both of which have
> this pattern.
> 
> Currently this series is only upgrading commands that use the network
> anyway, so I guess die()ing in an ls-remote or fetch is no big deal,
> but it could
> be interesting to keep going once we have more of the partial clone
> stuff working
> (e.g. remote assisted log/blame would want to gracefully fall back instead of
> die()ing without any useful output, I would think.)

These are all things we could do, but the current code just dies and it
may be more hassle right now to change all the uses of packet_read to
handle errors gracefully.  But its definitely something we can do in the
future.

> 
> >         }
> > +
> >         len -= 4;
> > -       if (len >= size)
> > +       if ((len < 0) || ((unsigned)len >= size))
> >                 die("protocol error: bad line length %d", len);
> > -       ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);
> > -       if (ret < 0)
> > -               return ret;
> > +
> > +       if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)
> > +               return PACKET_READ_EOF;
> >
> >         if ((options & PACKET_READ_CHOMP_NEWLINE) &&
> >             len && buffer[len-1] == '\n')
> > @@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,
> >
> >         buffer[len] = 0;
> >         packet_trace(buffer, len, 0);
> > -       return len;
> > +       *pktlen = len;
> > +       return PACKET_READ_NORMAL;
> > +}
> > +
> > +int packet_read(int fd, char **src_buffer, size_t *src_len,
> > +               char *buffer, unsigned size, int options)
> > +{
> > +       enum packet_read_status status;
> > +       int pktlen;
> > +
> > +       status = packet_read_with_status(fd, src_buffer, src_len,
> > +                                        buffer, size, &pktlen,
> > +                                        options);
> > +       switch (status) {
> > +       case PACKET_READ_EOF:
> > +               pktlen = -1;
> > +               break;
> > +       case PACKET_READ_NORMAL:
> > +               break;
> > +       case PACKET_READ_FLUSH:
> > +               pktlen = 0;
> > +               break;
> > +       }
> > +
> > +       return pktlen;
> >  }
> >
> >  static char *packet_read_line_generic(int fd,
> > diff --git a/pkt-line.h b/pkt-line.h
> > index 3dad583e2..06c468927 100644
> > --- a/pkt-line.h
> > +++ b/pkt-line.h
> > @@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);
> >  int packet_read(int fd, char **src_buffer, size_t *src_len, char
> >                 *buffer, unsigned size, int options);
> >
> > +/*
> > + * Read a packetized line into a buffer like the 'packet_read()' function but
> > + * returns an 'enum packet_read_status' which indicates the status of the read.
> > + * The number of bytes read will be assigined to *pktlen if the status of the
> > + * read was 'PACKET_READ_NORMAL'.
> > + */
> > +enum packet_read_status {
> > +       PACKET_READ_EOF = -1,
> > +       PACKET_READ_NORMAL,
> > +       PACKET_READ_FLUSH,
> > +};
> > +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
> > +                                               char *buffer, unsigned size, int *pktlen,
> > +                                               int options);
> > +
> >  /*
> >   * Convenience wrapper for packet_read that is not gentle, and sets the
> >   * CHOMP_NEWLINE option. The return value is NULL for a flush packet,
> > --
> > 2.15.1.620.gb9897f4670-goog
> >

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 12/26] ls-refs: introduce ls-refs server command
  2018-01-04  0:17   ` Stefan Beller
@ 2018-01-05 23:49     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-05 23:49 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On 01/03, Stefan Beller wrote:
> On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> > Introduce the ls-refs server command.  In protocol v2, the ls-refs
> > command is used to request the ref advertisement from the server.  Since
> > it is a command which can be requested (as opposed to mandatory in v1),
> > a client can sent a number of parameters in its request to limit the ref
> > advertisement based on provided ref-patterns.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> >  Documentation/technical/protocol-v2.txt | 26 +++++++++
> >  Makefile                                |  1 +
> >  ls-refs.c                               | 97 +++++++++++++++++++++++++++++++++
> >  ls-refs.h                               |  9 +++
> 
> Maybe consider putting any served command into a sub directory?
> 
> For example the code in builtin/ has laxer rules w.r.t. die()ing
> as it is a user facing command, whereas some devs want to see
> code at the root of the repo to not die() at all as the eventual goal
> is to have a library there.
> All this code is on the remote side, which also has different traits than
> the code at the root of the git.git repo; non-localisation comes to mind,
> but there might be other aspects as well (security?).

Well if we were to do this then we should move upload-pack and
receive-pack into this same "server code" directory.

> 
> 
> >  serve.c                                 |  2 +
> >  5 files changed, 135 insertions(+)
> >  create mode 100644 ls-refs.c
> >  create mode 100644 ls-refs.h
> >
> > diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
> > index b87ba3816..5f4d0e719 100644
> > --- a/Documentation/technical/protocol-v2.txt
> > +++ b/Documentation/technical/protocol-v2.txt
> > @@ -89,3 +89,29 @@ terminate the connection.
> >  Commands are the core actions that a client wants to perform (fetch, push,
> >  etc).  Each command will be provided with a list capabilities and
> >  arguments as requested by a client.
> > +
> > + Ls-refs
> 
> So is it ls-refs or Ls-refs or is any capitalization valid?

"ls-refs"  I'll make sure to change this.
> 
> > +---------
> > +
> > +Ls-refs is the command used to request a reference advertisement in v2.
> > +Unlike the current reference advertisement, ls-refs takes in parameters
> > +which can be used to limit the refs sent from the server.
> > +
> > +Ls-ref takes in the following parameters wraped in packet-lines:
> > +
> > +  symrefs: In addition to the object pointed by it, show the underlying
> > +          ref pointed by it when showing a symbolic ref.
> > +  peel: Show peeled tags.
> > +  ref-pattern <pattern>: When specified, only references matching the
> > +                        given patterns are displayed.
> 
> What kind of pattern matching is allowed here?
> strictly prefix only, or globbing, regexes?
> Is there a given grammar to follow? Maybe a link to the git
> glossary is or somewhere else might be fine.
> 
> Seeing that we do wildmatch() down there (as opposed to regexes),
> I wonder if it provides an entry for a denial of service attack, by crafting
> a pattern that is very expensive for the server to compute but cheap to
> ask for from a client. (c.f. 94da9193a6 (grep: add support for PCRE v2,
> 2017-06-01, but that is regexes!)
> 
> > +The output of ls-refs is as follows:
> > +
> > +    output = *ref
> > +            flush-pkt
> > +    ref = PKT-LINE((tip | peeled) LF)
> > +    tip = obj-id SP refname (SP symref-target)
> > +    peeled = obj-id SP refname "^{}"
> > +
> > +    symref = PKT-LINE("symref" SP symbolic-ref SP resolved-ref LF)
> > +    shallow = PKT-LINE("shallow" SP obj-id LF)
> > diff --git a/Makefile b/Makefile
> > index 5f3b5fe8b..152a73bec 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -820,6 +820,7 @@ LIB_OBJS += list-objects-filter-options.o
> >  LIB_OBJS += ll-merge.o
> >  LIB_OBJS += lockfile.o
> >  LIB_OBJS += log-tree.o
> > +LIB_OBJS += ls-refs.o
> >  LIB_OBJS += mailinfo.o
> >  LIB_OBJS += mailmap.o
> >  LIB_OBJS += match-trees.o
> > diff --git a/ls-refs.c b/ls-refs.c
> > new file mode 100644
> > index 000000000..ac4904a40
> > --- /dev/null
> > +++ b/ls-refs.c
> > @@ -0,0 +1,97 @@
> > +#include "cache.h"
> > +#include "repository.h"
> > +#include "refs.h"
> > +#include "remote.h"
> > +#include "argv-array.h"
> > +#include "ls-refs.h"
> > +#include "pkt-line.h"
> > +
> > +struct ls_refs_data {
> > +       unsigned peel;
> > +       unsigned symrefs;
> > +       struct argv_array patterns;
> > +};
> > +
> > +/*
> > + * Check if one of the patterns matches the tail part of the ref.
> > + * If no patterns were provided, all refs match.
> > + */
> > +static int ref_match(const struct argv_array *patterns, const char *refname)
> > +{
> > +       char *pathbuf;
> > +       int i;
> > +
> > +       if (!patterns->argc)
> > +               return 1; /* no restriction */
> > +
> > +       pathbuf = xstrfmt("/%s", refname);
> > +       for (i = 0; i < patterns->argc; i++) {
> > +               if (!wildmatch(patterns->argv[i], pathbuf, 0)) {
> > +                       free(pathbuf);
> > +                       return 1;
> > +               }
> > +       }
> > +       free(pathbuf);
> > +       return 0;
> > +}
> > +
> > +static int send_ref(const char *refname, const struct object_id *oid,
> > +                   int flag, void *cb_data)
> > +{
> > +       struct ls_refs_data *data = cb_data;
> > +       const char *refname_nons = strip_namespace(refname);
> > +       struct strbuf refline = STRBUF_INIT;
> > +
> > +       if (!ref_match(&data->patterns, refname))
> > +               return 0;
> > +
> > +       strbuf_addf(&refline, "%s %s", oid_to_hex(oid), refname_nons);
> > +       if (data->symrefs && flag & REF_ISSYMREF) {
> > +               struct object_id unused;
> > +               const char *symref_target = resolve_ref_unsafe(refname, 0,
> > +                                                              &unused,
> > +                                                              &flag);
> > +
> > +               if (!symref_target)
> > +                       die("'%s' is a symref but it is not?", refname);
> > +
> > +               strbuf_addf(&refline, " %s", symref_target);
> > +       }
> > +
> > +       strbuf_addch(&refline, '\n');
> > +
> > +       packet_write(1, refline.buf, refline.len);
> > +       if (data->peel) {
> > +               struct object_id peeled;
> > +               if (!peel_ref(refname, &peeled))
> > +                       packet_write_fmt(1, "%s %s^{}\n", oid_to_hex(&peeled),
> > +                                        refname_nons);
> > +       }
> > +
> > +       strbuf_release(&refline);
> > +       return 0;
> > +}
> > +
> > +int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)
> > +{
> > +       int i;
> > +       struct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };
> > +
> > +       for (i = 0; i < args->argc; i++) {
> > +               const char *arg = args->argv[i];
> > +               const char *out;
> > +
> > +               if (!strcmp("peel", arg))
> > +                       data.peel = 1;
> > +               else if (!strcmp("symrefs", arg))
> > +                       data.symrefs = 1;
> > +               else if (skip_prefix(arg, "ref-pattern ", &out))
> > +                       argv_array_pushf(&data.patterns, "*/%s", out);
> > +       }
> > +
> > +       head_ref_namespaced(send_ref, &data);
> > +       for_each_namespaced_ref(send_ref, &data);
> > +       packet_flush(1);
> > +       argv_array_clear(&data.patterns);
> > +       return 0;
> > +}
> > diff --git a/ls-refs.h b/ls-refs.h
> > new file mode 100644
> > index 000000000..9e4c57bfe
> > --- /dev/null
> > +++ b/ls-refs.h
> > @@ -0,0 +1,9 @@
> > +#ifndef LS_REFS_H
> > +#define LS_REFS_H
> > +
> > +struct repository;
> > +struct argv_array;
> > +extern int ls_refs(struct repository *r, struct argv_array *keys,
> > +                  struct argv_array *args);
> > +
> > +#endif /* LS_REFS_H */
> > diff --git a/serve.c b/serve.c
> > index da8127775..88d548410 100644
> > --- a/serve.c
> > +++ b/serve.c
> > @@ -4,6 +4,7 @@
> >  #include "pkt-line.h"
> >  #include "version.h"
> >  #include "argv-array.h"
> > +#include "ls-refs.h"
> >  #include "serve.h"
> >
> >  static int always_advertise(struct repository *r,
> > @@ -44,6 +45,7 @@ struct protocol_capability {
> >  static struct protocol_capability capabilities[] = {
> >         { "agent", agent_advertise, NULL },
> >         { "stateless-rpc", always_advertise, NULL },
> > +       { "ls-refs", always_advertise, ls_refs },
> >  };
> >
> >  static void advertise_capabilities(void)
> > --
> > 2.15.1.620.gb9897f4670-goog
> >

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 20/26] fetch-pack: perform a fetch using v2
  2018-01-04  1:23   ` Stefan Beller
@ 2018-01-05 23:55     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-05 23:55 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee, Jonathan Nieder

On 01/03, Stefan Beller wrote:
> On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams <bmwill@google.com> wrote:
> > +
> > +#define FETCH_CHECK_LOCAL 0
> > +#define FETCH_SEND_REQUEST 1
> > +#define FETCH_PROCESS_ACKS 2
> > +#define FETCH_SEND_HAVES 3
> > +#define FETCH_GET_PACK 4
> > +#define FETCH_DONE 5
> > +
> > +static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
> > +                                   int fd[2],
> > +                                   const struct ref *orig_ref,
> > +                                   struct ref **sought, int nr_sought,
> > +                                   char **pack_lockfile)
> > +{
> > +       struct ref *ref = copy_ref_list(orig_ref);
> > +       int state = FETCH_CHECK_LOCAL;
> 
> Is there any reason to use #defines over an enum here?
> 

No, it would probably be better to use an enum, that would also get rid
of the default case of the switch statement.

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 00/26] protocol version 2
  2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
                   ` (25 preceding siblings ...)
  2018-01-03  0:18 ` [PATCH 26/26] remote-curl: implement connect-half-duplex command Brandon Williams
@ 2018-01-09 17:55 ` Jonathan Tan
  2018-01-11  0:23   ` Brandon Williams
  26 siblings, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 17:55 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:02 -0800
Brandon Williams <bmwill@google.com> wrote:

> The following patches extend what I sent out as an WIP
> (https://public-inbox.org/git/20171204235823.63299-1-bmwill@google.com/) and
> implement protocol version 2.

Summarizing (for myself) the rationale for protocol version 2:

The existing protocol has a few pain points: (a) limit on the length of
the capability line (the capability line can be used to include
additional parameters in a backwards-compatible way), (b) difficulty in
creating proxies because of inconsistent flush semantics, and (c) the
need to implement clients twice - once for HTTP and once for
connect-supporting transports. To which we can add another: (d) if we
want to support something entirely new (for example, a server-side "git
log"), we will need a new protocol anyway.

The new functionality introduced in this patch set is probably best done
using a new protocol. If it were done using the existing protocol (by
adding a parameter in the capabilities line), we would still run into
(a) and (c), so we might as well introduce the new protocol now.

Some of the above points are repeats from my previous e-mail:
https://public-inbox.org/git/20171110121347.1f7c184c543622b60164e9fb@google.com/

> Some changes from that series are as follows:
>  * Lots of various cleanup on the ls-refs and fetch command code, both server
>    and client.
>  * Fetch command now supports a stateless-rpc mode which enables communicating
>    with a half-duplex connection.

Good to hear about fetch support.

>  * Introduce a new remote-helper command 'connect-half-duplex' which is
>    implemented by remote-curl (the http remote-helper).  This allows for a
>    client to establish a half-duplex connection and use remote-curl as a proxy
>    to wrap requests in http before sending them to the remote end and
>    unwrapping the responses and sending them back to the client's stdin.

I'm not sure about the "half-duplex" name - it is half-duplex in that
each side must terminate their communications with a flush, but not
half-duplex in that request-response pairs can overlap each other (e.g.
during negotation during fetch - there is an optimization in which the
client tries to keep two requests pending at a time). I think that the
idea we want to communicate is that requests and responses are always
packetized, stateless, and always happen as a pair.

I wonder if "stateless-connect" is a better keyword - it makes sense to
me (once described) that "stateless" implies that the client sends
everything the server needs at once (thus, in a packet), the server
sends everything the client needs back at once (thus, in a packet), and
then the client must not assume any state-storing on the part of the
server or transport.

>  * The transport code is refactored for ls-remote, fetch, and push to provide a
>    list of ref-patterns (based on the refspec being used) when requesting refs
>    from the remote end.  This allows the ls-refs code to send this list of
>    patterns so the remote end and filter the refs it sends back.

Briefly looking at the implementation, the client seems to incur an
extra roundtrip when using ls-remote (and others) with a v2-supporting
server. I initially didn't like this, but upon further reflection, this
is probably fine for now. The client can be upgraded later, and I think
that clients will eventually want to query git-serve directly for
"ls-refs" first, and then fall back to v0 for ancient servers, instead
of checking git-upload-pack first (as in this patch set) - so, the
support for "ls-refs" here won't be carried forward merely for backwards
compatibility, but will eventually be actively used.

As for the decision to use a new endpoint "git-serve" instead of reusing
"git-upload-pack" (or "git-receive-pack"), reusing the existing one
might allow some sort of optimization later in which the first
"git-upload-pack" query immediately returns with the v2 answer (instead
of redirecting the client to "git-serve"), but this probably doesn't
matter in practice (as I stated above, I think that eventually clients
will query git-serve first).

> This series effectively implements protocol version 2 for listing a remotes
> refs (ls-remote) as well as for fetch for the builtin transports (ssh, git,
> file) and for the http/https transports.  Push is not implemented yet and
> doesn't need to be implemented at the same time as fetch since the
> receive-pack code can default to using protocol v0 when v2 is requested by the
> client.

Agreed - push can be done later.

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 01/26] pkt-line: introduce packet_read_with_status
  2018-01-03  0:18 ` [PATCH 01/26] pkt-line: introduce packet_read_with_status Brandon Williams
  2018-01-03 19:27   ` Stefan Beller
@ 2018-01-09 18:04   ` Jonathan Tan
  2018-01-09 19:28     ` Brandon Williams
  1 sibling, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 18:04 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:03 -0800
Brandon Williams <bmwill@google.com> wrote:

> -int packet_read(int fd, char **src_buf, size_t *src_len,
> -		char *buffer, unsigned size, int options)
> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
> +						char *buffer, unsigned size, int *pktlen,
> +						int options)
>  {
> -	int len, ret;
> +	int len;
>  	char linelen[4];
>  
> -	ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);
> -	if (ret < 0)
> -		return ret;
> +	if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)
> +		return PACKET_READ_EOF;
> +
>  	len = packet_length(linelen);
>  	if (len < 0)
>  		die("protocol error: bad line length character: %.4s", linelen);
> -	if (!len) {
> +
> +	if (len == 0) {

This change (replacing "!len" with "len == 0") is unnecessary, I think.

>  		packet_trace("0000", 4, 0);
> -		return 0;
> +		return PACKET_READ_FLUSH;
> +	} else if (len >= 1 && len <= 3) {
> +		die("protocol error: bad line length character: %.4s", linelen);
>  	}

This seems to be more of a "bad line length" than a "bad line length
character".

Also, some of the checks are redundant. Above, it is probably better to
delete "len >= 1", and optionally write "len < 4" instead of "len <= 3"
(to emphasize that the subtraction of 4 below does not result in a
negative value).

> +
>  	len -= 4;
> -	if (len >= size)
> +	if ((len < 0) || ((unsigned)len >= size))
>  		die("protocol error: bad line length %d", len);

The "len < 0" check is redundant.

> -	ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);
> -	if (ret < 0)
> -		return ret;
> +
> +	if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)
> +		return PACKET_READ_EOF;
>  
>  	if ((options & PACKET_READ_CHOMP_NEWLINE) &&
>  	    len && buffer[len-1] == '\n')

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 02/26] pkt-line: introduce struct packet_reader
  2018-01-03  0:18 ` [PATCH 02/26] pkt-line: introduce struct packet_reader Brandon Williams
@ 2018-01-09 18:08   ` Jonathan Tan
  2018-01-09 19:19     ` Brandon Williams
  0 siblings, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 18:08 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:04 -0800
Brandon Williams <bmwill@google.com> wrote:

> diff --git a/pkt-line.h b/pkt-line.h
> index 06c468927..c446e886a 100644
> --- a/pkt-line.h
> +++ b/pkt-line.h
> @@ -111,6 +111,63 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
>   */
>  ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
>  
> +struct packet_reader {
> +	/* source file descriptor */
> +	int fd;
> +
> +	/* source buffer and its size */
> +	char *src_buffer;
> +	size_t src_len;
> +
> +	/* buffer that pkt-lines are read into and its size */
> +	char *buffer;
> +	unsigned buffer_size;

Is the intention to support different buffers in the future?

[snip]

> +/*
> + * Peek the next packet line without consuming it and return the status.
> + * The next call to 'packet_reader_read()' will perform a read of the same line
> + * that was peeked, consuming the line.
> + *
> + * Only a single line can be peeked at a time.

It is logical to me that if you peeked at a line, and then peeked at it
again, you will get the same line - I would phrase this not as a
restriction ("only a single line") but just as a statement of fact (e.g.
"Peeking at the same line multiple times without an intervening
packet_reader_read will return the same result").

> + */
> +extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);
> +
>  #define DEFAULT_PACKET_MAX 1000
>  #define LARGE_PACKET_MAX 65520
>  #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader
  2018-01-03  0:18 ` [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader Brandon Williams
@ 2018-01-09 18:27   ` Jonathan Tan
  2018-01-09 19:09     ` Brandon Williams
  0 siblings, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 18:27 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:09 -0800
Brandon Williams <bmwill@google.com> wrote:

> -	while ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {
> +	while (state != EXPECTING_DONE) {
> +		switch (packet_reader_read(&reader)) {
> +		case PACKET_READ_EOF:
> +			die_initial_contact(1);
> +		case PACKET_READ_NORMAL:
> +			len = reader.pktlen;
> +			if (len > 4 && skip_prefix(packet_buffer, "ERR ", &arg))

This should be a field in reader, not the global packet_buffer, I think.

Also, I did a search of usages of packet_buffer, and there are just a
few of them - it might be worthwhile to eliminate it, and have each
component using it allocate its own buffer. But this can be done in a
separate patch set.

> @@ -269,6 +284,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
>  			if (process_shallow(len, shallow_points))
>  				break;
>  			die("protocol error: unexpected '%s'", packet_buffer);

Here too.

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 09/26] transport: store protocol version
  2018-01-03  0:18 ` [PATCH 09/26] transport: store protocol version Brandon Williams
@ 2018-01-09 18:41   ` Jonathan Tan
  2018-01-09 19:15     ` Brandon Williams
  0 siblings, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 18:41 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:11 -0800
Brandon Williams <bmwill@google.com> wrote:

> diff --git a/transport.c b/transport.c
> index 63c3dbab9..2378dcb38 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -118,6 +118,7 @@ struct git_transport_data {
>  	struct child_process *conn;
>  	int fd[2];
>  	unsigned got_remote_heads : 1;
> +	enum protocol_version version;

Should this be initialized to protocol_unknown_version? Right now, as
far as I can tell, it is zero-initialized, which means protocol_v0.

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader
  2018-01-09 18:27   ` Jonathan Tan
@ 2018-01-09 19:09     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-09 19:09 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/09, Jonathan Tan wrote:
> On Tue,  2 Jan 2018 16:18:09 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> > -	while ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {
> > +	while (state != EXPECTING_DONE) {
> > +		switch (packet_reader_read(&reader)) {
> > +		case PACKET_READ_EOF:
> > +			die_initial_contact(1);
> > +		case PACKET_READ_NORMAL:
> > +			len = reader.pktlen;
> > +			if (len > 4 && skip_prefix(packet_buffer, "ERR ", &arg))
> 
> This should be a field in reader, not the global packet_buffer, I think.

Thanks for catching that.

> 
> Also, I did a search of usages of packet_buffer, and there are just a
> few of them - it might be worthwhile to eliminate it, and have each
> component using it allocate its own buffer. But this can be done in a
> separate patch set.

I'll go through and eliminate the references to packet_buffer by passing
in the buffer explicitly.

> 
> > @@ -269,6 +284,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> >  			if (process_shallow(len, shallow_points))
> >  				break;
> >  			die("protocol error: unexpected '%s'", packet_buffer);
> 
> Here too.

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 09/26] transport: store protocol version
  2018-01-09 18:41   ` Jonathan Tan
@ 2018-01-09 19:15     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-09 19:15 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/09, Jonathan Tan wrote:
> On Tue,  2 Jan 2018 16:18:11 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> > diff --git a/transport.c b/transport.c
> > index 63c3dbab9..2378dcb38 100644
> > --- a/transport.c
> > +++ b/transport.c
> > @@ -118,6 +118,7 @@ struct git_transport_data {
> >  	struct child_process *conn;
> >  	int fd[2];
> >  	unsigned got_remote_heads : 1;
> > +	enum protocol_version version;
> 
> Should this be initialized to protocol_unknown_version? Right now, as
> far as I can tell, it is zero-initialized, which means protocol_v0.

I don't think it matters as the value isn't used until after the
version has already been discovered.

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 02/26] pkt-line: introduce struct packet_reader
  2018-01-09 18:08   ` Jonathan Tan
@ 2018-01-09 19:19     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-09 19:19 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/09, Jonathan Tan wrote:
> On Tue,  2 Jan 2018 16:18:04 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> > diff --git a/pkt-line.h b/pkt-line.h
> > index 06c468927..c446e886a 100644
> > --- a/pkt-line.h
> > +++ b/pkt-line.h
> > @@ -111,6 +111,63 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
> >   */
> >  ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
> >  
> > +struct packet_reader {
> > +	/* source file descriptor */
> > +	int fd;
> > +
> > +	/* source buffer and its size */
> > +	char *src_buffer;
> > +	size_t src_len;
> > +
> > +	/* buffer that pkt-lines are read into and its size */
> > +	char *buffer;
> > +	unsigned buffer_size;
> 
> Is the intention to support different buffers in the future?

Potentially at some point.

> 
> [snip]
> 
> > +/*
> > + * Peek the next packet line without consuming it and return the status.
> > + * The next call to 'packet_reader_read()' will perform a read of the same line
> > + * that was peeked, consuming the line.
> > + *
> > + * Only a single line can be peeked at a time.
> 
> It is logical to me that if you peeked at a line, and then peeked at it
> again, you will get the same line - I would phrase this not as a
> restriction ("only a single line") but just as a statement of fact (e.g.
> "Peeking at the same line multiple times without an intervening
> packet_reader_read will return the same result").

Fair enough, i'll change the wording.

> 
> > + */
> > +extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);
> > +
> >  #define DEFAULT_PACKET_MAX 1000
> >  #define LARGE_PACKET_MAX 65520
> >  #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 01/26] pkt-line: introduce packet_read_with_status
  2018-01-09 18:04   ` Jonathan Tan
@ 2018-01-09 19:28     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-09 19:28 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/09, Jonathan Tan wrote:
> On Tue,  2 Jan 2018 16:18:03 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> > -int packet_read(int fd, char **src_buf, size_t *src_len,
> > -		char *buffer, unsigned size, int options)
> > +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
> > +						char *buffer, unsigned size, int *pktlen,
> > +						int options)
> >  {
> > -	int len, ret;
> > +	int len;
> >  	char linelen[4];
> >  
> > -	ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);
> > -	if (ret < 0)
> > -		return ret;
> > +	if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)
> > +		return PACKET_READ_EOF;
> > +
> >  	len = packet_length(linelen);
> >  	if (len < 0)
> >  		die("protocol error: bad line length character: %.4s", linelen);
> > -	if (!len) {
> > +
> > +	if (len == 0) {
> 
> This change (replacing "!len" with "len == 0") is unnecessary, I think.
> 
> >  		packet_trace("0000", 4, 0);
> > -		return 0;
> > +		return PACKET_READ_FLUSH;
> > +	} else if (len >= 1 && len <= 3) {
> > +		die("protocol error: bad line length character: %.4s", linelen);
> >  	}
> 
> This seems to be more of a "bad line length" than a "bad line length
> character".

I'll make these changes, though I do think this needs to stay as a "bad
line length character" as the len could be neg which is an indication of
parsing the linelen character failed.

> 
> Also, some of the checks are redundant. Above, it is probably better to
> delete "len >= 1", and optionally write "len < 4" instead of "len <= 3"
> (to emphasize that the subtraction of 4 below does not result in a
> negative value).
> 
> > +
> >  	len -= 4;
> > -	if (len >= size)
> > +	if ((len < 0) || ((unsigned)len >= size))
> >  		die("protocol error: bad line length %d", len);
> 
> The "len < 0" check is redundant.
> 
> > -	ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);
> > -	if (ret < 0)
> > -		return ret;
> > +
> > +	if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)
> > +		return PACKET_READ_EOF;
> >  
> >  	if ((options & PACKET_READ_CHOMP_NEWLINE) &&
> >  	    len && buffer[len-1] == '\n')

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 11/26] serve: introduce git-serve
  2018-01-03  0:18 ` [PATCH 11/26] serve: introduce git-serve Brandon Williams
@ 2018-01-09 20:24   ` Jonathan Tan
  2018-01-09 22:16     ` Brandon Williams
  0 siblings, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 20:24 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:13 -0800
Brandon Williams <bmwill@google.com> wrote:

> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
> new file mode 100644
> index 000000000..b87ba3816
> --- /dev/null
> +++ b/Documentation/technical/protocol-v2.txt

I'll review the documentation later, once there is some consensus that
the overall design is OK. (Or maybe there already is consensus?)

> diff --git a/builtin/serve.c b/builtin/serve.c
> new file mode 100644
> index 000000000..bb726786a
> --- /dev/null
> +++ b/builtin/serve.c
> @@ -0,0 +1,30 @@
> +#include "cache.h"
> +#include "builtin.h"
> +#include "parse-options.h"
> +#include "serve.h"
> +
> +static char const * const grep_usage[] = {

Should be serve_usage.

> diff --git a/serve.c b/serve.c
> new file mode 100644
> index 000000000..da8127775
> --- /dev/null
> +++ b/serve.c

[snip]

> +struct protocol_capability {
> +	const char *name; /* capability name */

Maybe document as:

  The name of the capability. The server uses this name when advertising
  this capability, and the client uses this name to invoke the command
  corresponding to this capability.

> +	/*
> +	 * Function queried to see if a capability should be advertised.
> +	 * Optionally a value can be specified by adding it to 'value'.
> +	 */
> +	int (*advertise)(struct repository *r, struct strbuf *value);

Document what happens when value is appended to. For example:

  ... If value is appended to, the server will advertise this capability
  as <name>=<value> instead of <name>.

> +	/*
> +	 * Function called when a client requests the capability as a command.
> +	 * The command request will be provided to the function via 'keys', the
> +	 * capabilities requested, and 'args', the command specific parameters.
> +	 *
> +	 * This field should be NULL for capabilities which are not commands.
> +	 */
> +	int (*command)(struct repository *r,
> +		       struct argv_array *keys,
> +		       struct argv_array *args);

Looking at the code below, I see that the command is not executed unless
advertise returns true - this means that a command cannot be both
supported and unadvertised. Would this be too restrictive? For example,
this would disallow a gradual across-multiple-servers rollout in which
we allow but not advertise a capability, and then after some time,
advertise the capability.

If we change this, then the value parameter of advertise can be
mandatory instead of optional.

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 12/26] ls-refs: introduce ls-refs server command
  2018-01-03  0:18 ` [PATCH 12/26] ls-refs: introduce ls-refs server command Brandon Williams
  2018-01-04  0:17   ` Stefan Beller
@ 2018-01-09 20:50   ` Jonathan Tan
  2018-01-16 19:23     ` Brandon Williams
  1 sibling, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 20:50 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:14 -0800
Brandon Williams <bmwill@google.com> wrote:

> +  symrefs: In addition to the object pointed by it, show the underlying
> +	   ref pointed by it when showing a symbolic ref.
> +  peel: Show peeled tags.
> +  ref-pattern <pattern>: When specified, only references matching the
> +			 given patterns are displayed.

I notice "symrefs" being tested in patch 13 and "ref-pattern" being
tested in patch 16. Is it possible to make a test for "peel" as well?
(Or is it being tested somewhere I didn't notice?)

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 11/26] serve: introduce git-serve
  2018-01-09 20:24   ` Jonathan Tan
@ 2018-01-09 22:16     ` Brandon Williams
  2018-01-09 22:28       ` Jonathan Tan
  0 siblings, 1 reply; 59+ messages in thread
From: Brandon Williams @ 2018-01-09 22:16 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/09, Jonathan Tan wrote:
> On Tue,  2 Jan 2018 16:18:13 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> > diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
> > new file mode 100644
> > index 000000000..b87ba3816
> > --- /dev/null
> > +++ b/Documentation/technical/protocol-v2.txt
> 
> I'll review the documentation later, once there is some consensus that
> the overall design is OK. (Or maybe there already is consensus?)
> 
> > diff --git a/builtin/serve.c b/builtin/serve.c
> > new file mode 100644
> > index 000000000..bb726786a
> > --- /dev/null
> > +++ b/builtin/serve.c
> > @@ -0,0 +1,30 @@
> > +#include "cache.h"
> > +#include "builtin.h"
> > +#include "parse-options.h"
> > +#include "serve.h"
> > +
> > +static char const * const grep_usage[] = {
> 
> Should be serve_usage.
> 
> > diff --git a/serve.c b/serve.c
> > new file mode 100644
> > index 000000000..da8127775
> > --- /dev/null
> > +++ b/serve.c
> 
> [snip]
> 
> > +struct protocol_capability {
> > +	const char *name; /* capability name */
> 
> Maybe document as:
> 
>   The name of the capability. The server uses this name when advertising
>   this capability, and the client uses this name to invoke the command
>   corresponding to this capability.
> 
> > +	/*
> > +	 * Function queried to see if a capability should be advertised.
> > +	 * Optionally a value can be specified by adding it to 'value'.
> > +	 */
> > +	int (*advertise)(struct repository *r, struct strbuf *value);
> 
> Document what happens when value is appended to. For example:
> 
>   ... If value is appended to, the server will advertise this capability
>   as <name>=<value> instead of <name>.
> 

All good documentation changes.

> > +	/*
> > +	 * Function called when a client requests the capability as a command.
> > +	 * The command request will be provided to the function via 'keys', the
> > +	 * capabilities requested, and 'args', the command specific parameters.
> > +	 *
> > +	 * This field should be NULL for capabilities which are not commands.
> > +	 */
> > +	int (*command)(struct repository *r,
> > +		       struct argv_array *keys,
> > +		       struct argv_array *args);
> 
> Looking at the code below, I see that the command is not executed unless
> advertise returns true - this means that a command cannot be both
> supported and unadvertised. Would this be too restrictive? For example,
> this would disallow a gradual across-multiple-servers rollout in which
> we allow but not advertise a capability, and then after some time,
> advertise the capability.

One way to change this would be to just add another function to the
struct which is called to check if the command is allowed, instead of
relying on the same function to do that for both advertise and
allow...though I don't see a big win for allowing a command but not
advertising it.

> 
> If we change this, then the value parameter of advertise can be
> mandatory instead of optional.

I don't see how this fixes the issue you bring up.

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 13/26] connect: request remote refs using v2
  2018-01-03  0:18 ` [PATCH 13/26] connect: request remote refs using v2 Brandon Williams
@ 2018-01-09 22:24   ` Jonathan Tan
  0 siblings, 0 replies; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 22:24 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:15 -0800
Brandon Williams <bmwill@google.com> wrote:

> diff --git a/connect.c b/connect.c
> index caa539b75..9badd403f 100644
> --- a/connect.c
> +++ b/connect.c
> @@ -12,9 +12,11 @@
>  #include "sha1-array.h"
>  #include "transport.h"
>  #include "strbuf.h"
> +#include "version.h"
>  #include "protocol.h"
>  
>  static char *server_capabilities;
> +static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;
>  static const char *parse_feature_value(const char *, const char *, int *);
>  
>  static int check_ref(const char *name, unsigned int flags)
> @@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)
>  		      "and the repository exists."));
>  }
>  
> +static int server_supports_v2(const char *c, int die_on_error)

Document what "c" means.

[snip]

> +static void process_capabilities_v2(struct packet_reader *reader)
> +{
> +	while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
> +		argv_array_push(&server_capabilities_v2, reader->line);
> +	}

No need for braces on single-line blocks.

> +static int process_ref_v2(const char *line, struct ref ***list)

The "list" is the tail of a linked list, so maybe name it "tail"
instead.

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 11/26] serve: introduce git-serve
  2018-01-09 22:16     ` Brandon Williams
@ 2018-01-09 22:28       ` Jonathan Tan
  2018-01-09 22:34         ` Brandon Williams
  0 siblings, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-09 22:28 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue, 9 Jan 2018 14:16:42 -0800
Brandon Williams <bmwill@google.com> wrote:

> All good documentation changes.

Thanks!

> > > +	/*
> > > +	 * Function called when a client requests the capability as a command.
> > > +	 * The command request will be provided to the function via 'keys', the
> > > +	 * capabilities requested, and 'args', the command specific parameters.
> > > +	 *
> > > +	 * This field should be NULL for capabilities which are not commands.
> > > +	 */
> > > +	int (*command)(struct repository *r,
> > > +		       struct argv_array *keys,
> > > +		       struct argv_array *args);
> > 
> > Looking at the code below, I see that the command is not executed unless
> > advertise returns true - this means that a command cannot be both
> > supported and unadvertised. Would this be too restrictive? For example,
> > this would disallow a gradual across-multiple-servers rollout in which
> > we allow but not advertise a capability, and then after some time,
> > advertise the capability.
> 
> One way to change this would be to just add another function to the
> struct which is called to check if the command is allowed, instead of
> relying on the same function to do that for both advertise and
> allow...though I don't see a big win for allowing a command but not
> advertising it.

My rationale for allowing a command but not advertising it is in the
paragraph above (that you quoted), but if that is insufficient
rationale, then I agree that we don't need to do this.

> > If we change this, then the value parameter of advertise can be
> > mandatory instead of optional.
> 
> I don't see how this fixes the issue you bring up.

This is a consequence, not a fix - if we were to do as I suggested, then
we no longer need to invoke advertise to check whether something is
advertised except when we are advertising them, in which case "value"
never needs to be NULL.

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 11/26] serve: introduce git-serve
  2018-01-09 22:28       ` Jonathan Tan
@ 2018-01-09 22:34         ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-09 22:34 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/09, Jonathan Tan wrote:
> On Tue, 9 Jan 2018 14:16:42 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> > All good documentation changes.
> 
> Thanks!
> 
> > > > +	/*
> > > > +	 * Function called when a client requests the capability as a command.
> > > > +	 * The command request will be provided to the function via 'keys', the
> > > > +	 * capabilities requested, and 'args', the command specific parameters.
> > > > +	 *
> > > > +	 * This field should be NULL for capabilities which are not commands.
> > > > +	 */
> > > > +	int (*command)(struct repository *r,
> > > > +		       struct argv_array *keys,
> > > > +		       struct argv_array *args);
> > > 
> > > Looking at the code below, I see that the command is not executed unless
> > > advertise returns true - this means that a command cannot be both
> > > supported and unadvertised. Would this be too restrictive? For example,
> > > this would disallow a gradual across-multiple-servers rollout in which
> > > we allow but not advertise a capability, and then after some time,
> > > advertise the capability.
> > 
> > One way to change this would be to just add another function to the
> > struct which is called to check if the command is allowed, instead of
> > relying on the same function to do that for both advertise and
> > allow...though I don't see a big win for allowing a command but not
> > advertising it.
> 
> My rationale for allowing a command but not advertising it is in the
> paragraph above (that you quoted), but if that is insufficient
> rationale, then I agree that we don't need to do this.

I have no issues with adding that functionality, i don't really feel
that strongly one way or another.  Just seemed like additional work for
not much gain right now, key being right now.  It very well may be worth
it for the use case you specified.  If so I can definitely make the
change.

> 
> > > If we change this, then the value parameter of advertise can be
> > > mandatory instead of optional.
> > 
> > I don't see how this fixes the issue you bring up.
> 
> This is a consequence, not a fix - if we were to do as I suggested, then
> we no longer need to invoke advertise to check whether something is
> advertised except when we are advertising them, in which case "value"
> never needs to be NULL.

Oh I understand what you are trying to explain, yes you're right.

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 20/26] fetch-pack: perform a fetch using v2
  2018-01-03  0:18 ` [PATCH 20/26] fetch-pack: perform a fetch using v2 Brandon Williams
  2018-01-04  1:23   ` Stefan Beller
@ 2018-01-10  0:05   ` Jonathan Tan
  1 sibling, 0 replies; 59+ messages in thread
From: Jonathan Tan @ 2018-01-10  0:05 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:22 -0800
Brandon Williams <bmwill@google.com> wrote:

> +static enum ack_type process_ack(const char *line, struct object_id *oid)
> +{
> +	const char *arg;
> +
> +	if (!strcmp(line, "NAK"))
> +		return NAK;
> +	if (skip_prefix(line, "ACK ", &arg)) {
> +		if (!parse_oid_hex(arg, oid, &arg)) {
> +			if (strstr(arg, "continue"))
> +				return ACK_continue;

This function seems to be only used for v2, so I don't think we need to
parse "continue".

Also, maybe describe the plan for supporting functionality not supported
yet (e.g. server-side declaration of shallows and client-side "deepen").

It may be possible to delay support for server-side shallows on the
server (that is, only implement support for it in the client) since the
server can just declare that it doesn't support protocol v2 when serving
such repos (although it might just be easier to implement server-side
support in this case).

For "deepen", we need support for it both on the client and the server
now unless we plan to declare a "deepen" capability in the future (then,
as of these patches, clients that require "deepen" will use protocol v1;
when a new server declares "deepen", old clients will ignore it and keep
the status quo, and new clients can then use "deepen").

There may be others that I've missed.

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 26/26] remote-curl: implement connect-half-duplex command
  2018-01-03  0:18 ` [PATCH 26/26] remote-curl: implement connect-half-duplex command Brandon Williams
@ 2018-01-10  0:10   ` Jonathan Tan
  2018-01-10 17:57   ` Jonathan Tan
  1 sibling, 0 replies; 59+ messages in thread
From: Jonathan Tan @ 2018-01-10  0:10 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:28 -0800
Brandon Williams <bmwill@google.com> wrote:

> Teach remote-curl the 'connect-half-duplex' command which is used to
> establish a half-duplex connection with servers which support protocol
> version 2.  This allows remote-curl to act as a proxy, allowing the git
> client to communicate natively with a remote end, simply using
> remote-curl as a pass through to convert requests to http.
> 
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  remote-curl.c          | 185 ++++++++++++++++++++++++++++++++++++++++++++++++-
>  t/t5701-protocol-v2.sh |  41 +++++++++++
>  2 files changed, 224 insertions(+), 2 deletions(-)

I didn't look at the usage of the curl API in detail, but overall this
looks good. I'm pleasantly surprised that it didn't take so many lines
of code as I expected.

Overall everything looks good, except for the points that I have brought
up in my other e-mails.

> diff --git a/remote-curl.c b/remote-curl.c
> index 4086aa733..b63b06398 100644
> --- a/remote-curl.c
> +++ b/remote-curl.c

[snip]

> +struct proxy_state {
> +	char *service_name;
> +	char *service_url;
> +	char *hdr_content_type;
> +	char *hdr_accept;

Maybe document that the above 3 fields (service_url to hdr_accept) are
cached because we need to pass them to curl_easy_setopt() for every
request.

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 26/26] remote-curl: implement connect-half-duplex command
  2018-01-03  0:18 ` [PATCH 26/26] remote-curl: implement connect-half-duplex command Brandon Williams
  2018-01-10  0:10   ` Jonathan Tan
@ 2018-01-10 17:57   ` Jonathan Tan
  2018-01-11  1:09     ` Brandon Williams
  1 sibling, 1 reply; 59+ messages in thread
From: Jonathan Tan @ 2018-01-10 17:57 UTC (permalink / raw)
  To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On Tue,  2 Jan 2018 16:18:28 -0800
Brandon Williams <bmwill@google.com> wrote:

> +static size_t proxy_in(void *ptr, size_t eltsize,
> +		       size_t nmemb, void *buffer_)

OK, I managed to look at the Curl stuff in more detail.

I know that these parameter names are what remote_curl.c has been using
for its callbacks, but I find them confusing (in particular, some Curl
documentation rightly refer to the 1st parameter as a buffer, and the
4th parameter is actually userdata). Also, according to the Curl
documentation, the type of the first parameter is "char *". Could we
change the type of the first parameter to "char *", and the name of the
fourth parameter either to "proxy_state_" or "userdata"?

> +{
> +	size_t max = eltsize * nmemb;
> +	struct proxy_state *p = buffer_;
> +	size_t avail = p->request_buffer.len - p->pos;
> +
> +	if (!avail) {
> +		if (p->seen_flush) {
> +			p->seen_flush = 0;
> +			return 0;
> +		}
> +
> +		strbuf_reset(&p->request_buffer);
> +		switch (packet_reader_read(&p->reader)) {
> +		case PACKET_READ_EOF:
> +			die("error reading request from parent process");

This should say "BUG:", I think. I'm not sure what the best way of
explaining it is, but basically connect_half_duplex is supposed to
ensure (by peeking) that there is no EOF when proxy_in() is called.

> +		case PACKET_READ_NORMAL:
> +			packet_buf_write_len(&p->request_buffer, p->reader.line,
> +					     p->reader.pktlen);
> +			break;
> +		case PACKET_READ_DELIM:
> +			packet_buf_delim(&p->request_buffer);
> +			break;
> +		case PACKET_READ_FLUSH:
> +			packet_buf_flush(&p->request_buffer);
> +			p->seen_flush = 1;
> +			break;
> +		}
> +		p->pos = 0;
> +		avail = p->request_buffer.len;
> +	}
> +
> +	if (max < avail)
> +		avail = max;
> +	memcpy(ptr, p->request_buffer.buf + p->pos, avail);
> +	p->pos += avail;
> +	return avail;

Thanks, this looks correct. I wish that the Curl API had a way for us to
say "here are 4 more bytes, and that is all" instead of us having to
make a note (p->seen_flush) to remember to return 0 on the next call,
but that's the way it is.

> +}
> +static size_t proxy_out(char *ptr, size_t eltsize,
> +			size_t nmemb, void *buffer_)

Add a blank line before proxy_out. Also, same comment as proxy_in()
about the function signature.

> +{
> +	size_t size = eltsize * nmemb;
> +	struct proxy_state *p = buffer_;
> +
> +	write_or_die(p->out, ptr, size);
> +	return size;
> +}
> +
> +static int proxy_post(struct proxy_state *p)
> +{
> +	struct active_request_slot *slot;
> +	struct curl_slist *headers = http_copy_default_headers();
> +	int err;
> +
> +	headers = curl_slist_append(headers, p->hdr_content_type);
> +	headers = curl_slist_append(headers, p->hdr_accept);
> +	headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
> +
> +	slot = get_active_slot();
> +
> +	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
> +	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
> +	curl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);
> +	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);

I looked at the Curl documentation for CURLOPT_HTTPHEADER and
curl_easy_setopt doesn't consume the argument here (in fact, it asks us
to keep "headers" around), so it might be possible to just generate the
headers once in proxy_state_init().

> +
> +	/* Setup function to read request from client */
> +	curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);
> +	curl_easy_setopt(slot->curl, CURLOPT_READDATA, p);
> +
> +	/* Setup function to write server response to client */
> +	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);
> +	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);
> +
> +	err = run_slot(slot, NULL);
> +
> +	if (err != HTTP_OK)
> +		err = -1;

This seems to mean that we cannot have two requests in flight at the
same time even while there is no response (from the fact that we have a
HTTP status code after returning from run_slot()).

I thought that git fetch over HTTP uses the two-requests-in-flight
optimization that it also does over other protocols like SSH, but I see
that that code path (fetch_git() in remote-curl.c) also uses run_slot()
indirectly, so maybe my assumption is wrong. Anyway, this is outside the
scope of this patch.

> +
> +	curl_slist_free_all(headers);
> +	return err;
> +}
> +
> +static int connect_half_duplex(const char *service_name)
> +{
> +	struct discovery *discover;
> +	struct proxy_state p;
> +
> +	/*
> +	 * Run the info/refs request and see if the server supports protocol
> +	 * v2.  If and only if the server supports v2 can we successfully
> +	 * establish a half-duplex connection, otherwise we need to tell the
> +	 * client to fallback to using other transport helper functions to
> +	 * complete their request.
> +	 */
> +	discover = discover_refs(service_name, 0);
> +	if (discover->version != protocol_v2) {
> +		printf("fallback\n");
> +		fflush(stdout);
> +		return -1;
> +	} else {
> +		/* Half-Duplex Connection established */
> +		printf("\n");
> +		fflush(stdout);
> +	}
> +
> +	proxy_state_init(&p, service_name);
> +
> +	/*
> +	 * Dump the capability listing that we got from the server earlier
> +	 * during the info/refs request.
> +	 */
> +	write_or_die(p.out, discover->buf, discover->len);
> +
> +	/* Peek the next packet line.  Until we see EOF keep sending POSTs */
> +	while (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {
> +		if (proxy_post(&p)) {
> +			/* We would have an err here */

Probably better to comment "Error message already printed by
proxy_post".

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 00/26] protocol version 2
  2018-01-09 17:55 ` [PATCH 00/26] protocol version 2 Jonathan Tan
@ 2018-01-11  0:23   ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-11  0:23 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/09, Jonathan Tan wrote:
> On Tue,  2 Jan 2018 16:18:02 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> >  * Introduce a new remote-helper command 'connect-half-duplex' which is
> >    implemented by remote-curl (the http remote-helper).  This allows for a
> >    client to establish a half-duplex connection and use remote-curl as a proxy
> >    to wrap requests in http before sending them to the remote end and
> >    unwrapping the responses and sending them back to the client's stdin.
> 
> I'm not sure about the "half-duplex" name - it is half-duplex in that
> each side must terminate their communications with a flush, but not
> half-duplex in that request-response pairs can overlap each other (e.g.
> during negotation during fetch - there is an optimization in which the
> client tries to keep two requests pending at a time). I think that the
> idea we want to communicate is that requests and responses are always
> packetized, stateless, and always happen as a pair.
> 
> I wonder if "stateless-connect" is a better keyword - it makes sense to
> me (once described) that "stateless" implies that the client sends
> everything the server needs at once (thus, in a packet), the server
> sends everything the client needs back at once (thus, in a packet), and
> then the client must not assume any state-storing on the part of the
> server or transport.

I like that name much better, I think I'll change it to use
'stateless-connect'.  Thanks :)


-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 26/26] remote-curl: implement connect-half-duplex command
  2018-01-10 17:57   ` Jonathan Tan
@ 2018-01-11  1:09     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-11  1:09 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/10, Jonathan Tan wrote:
> On Tue,  2 Jan 2018 16:18:28 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> > +static size_t proxy_in(void *ptr, size_t eltsize,
> > +		       size_t nmemb, void *buffer_)
> 
> OK, I managed to look at the Curl stuff in more detail.
> 
> I know that these parameter names are what remote_curl.c has been using
> for its callbacks, but I find them confusing (in particular, some Curl
> documentation rightly refer to the 1st parameter as a buffer, and the
> 4th parameter is actually userdata). Also, according to the Curl
> documentation, the type of the first parameter is "char *". Could we
> change the type of the first parameter to "char *", and the name of the
> fourth parameter either to "proxy_state_" or "userdata"?

Sounds good, I'll make the change.

> 
> > +{
> > +	size_t max = eltsize * nmemb;
> > +	struct proxy_state *p = buffer_;
> > +	size_t avail = p->request_buffer.len - p->pos;
> > +
> > +	if (!avail) {
> > +		if (p->seen_flush) {
> > +			p->seen_flush = 0;
> > +			return 0;
> > +		}
> > +
> > +		strbuf_reset(&p->request_buffer);
> > +		switch (packet_reader_read(&p->reader)) {
> > +		case PACKET_READ_EOF:
> > +			die("error reading request from parent process");
> 
> This should say "BUG:", I think. I'm not sure what the best way of
> explaining it is, but basically connect_half_duplex is supposed to
> ensure (by peeking) that there is no EOF when proxy_in() is called.

This wouldn't necessarily be a bug if the parent dies early for some
reason though right?

> 
> > +		case PACKET_READ_NORMAL:
> > +			packet_buf_write_len(&p->request_buffer, p->reader.line,
> > +					     p->reader.pktlen);
> > +			break;
> > +		case PACKET_READ_DELIM:
> > +			packet_buf_delim(&p->request_buffer);
> > +			break;
> > +		case PACKET_READ_FLUSH:
> > +			packet_buf_flush(&p->request_buffer);
> > +			p->seen_flush = 1;
> > +			break;
> > +		}
> > +		p->pos = 0;
> > +		avail = p->request_buffer.len;
> > +	}
> > +
> > +	if (max < avail)
> > +		avail = max;
> > +	memcpy(ptr, p->request_buffer.buf + p->pos, avail);
> > +	p->pos += avail;
> > +	return avail;
> 
> Thanks, this looks correct. I wish that the Curl API had a way for us to
> say "here are 4 more bytes, and that is all" instead of us having to
> make a note (p->seen_flush) to remember to return 0 on the next call,
> but that's the way it is.
> 
> > +}
> > +static size_t proxy_out(char *ptr, size_t eltsize,
> > +			size_t nmemb, void *buffer_)
> 
> Add a blank line before proxy_out. Also, same comment as proxy_in()
> about the function signature.

I'll change this function too.

> 
> > +{
> > +	size_t size = eltsize * nmemb;
> > +	struct proxy_state *p = buffer_;
> > +
> > +	write_or_die(p->out, ptr, size);
> > +	return size;
> > +}
> > +
> > +static int proxy_post(struct proxy_state *p)
> > +{
> > +	struct active_request_slot *slot;
> > +	struct curl_slist *headers = http_copy_default_headers();
> > +	int err;
> > +
> > +	headers = curl_slist_append(headers, p->hdr_content_type);
> > +	headers = curl_slist_append(headers, p->hdr_accept);
> > +	headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
> > +
> > +	slot = get_active_slot();
> > +
> > +	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
> > +	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
> > +	curl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);
> > +	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
> 
> I looked at the Curl documentation for CURLOPT_HTTPHEADER and
> curl_easy_setopt doesn't consume the argument here (in fact, it asks us
> to keep "headers" around), so it might be possible to just generate the
> headers once in proxy_state_init().

Yeah I'll go ahead and do that, it'll make the post function a bit
cleaner too.

> 
> > +
> > +	/* Setup function to read request from client */
> > +	curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);
> > +	curl_easy_setopt(slot->curl, CURLOPT_READDATA, p);
> > +
> > +	/* Setup function to write server response to client */
> > +	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);
> > +	curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);
> > +
> > +	err = run_slot(slot, NULL);
> > +
> > +	if (err != HTTP_OK)
> > +		err = -1;
> 
> This seems to mean that we cannot have two requests in flight at the
> same time even while there is no response (from the fact that we have a
> HTTP status code after returning from run_slot()).
> 
> I thought that git fetch over HTTP uses the two-requests-in-flight
> optimization that it also does over other protocols like SSH, but I see
> that that code path (fetch_git() in remote-curl.c) also uses run_slot()
> indirectly, so maybe my assumption is wrong. Anyway, this is outside the
> scope of this patch.
> 
> > +
> > +	curl_slist_free_all(headers);
> > +	return err;
> > +}
> > +
> > +static int connect_half_duplex(const char *service_name)
> > +{
> > +	struct discovery *discover;
> > +	struct proxy_state p;
> > +
> > +	/*
> > +	 * Run the info/refs request and see if the server supports protocol
> > +	 * v2.  If and only if the server supports v2 can we successfully
> > +	 * establish a half-duplex connection, otherwise we need to tell the
> > +	 * client to fallback to using other transport helper functions to
> > +	 * complete their request.
> > +	 */
> > +	discover = discover_refs(service_name, 0);
> > +	if (discover->version != protocol_v2) {
> > +		printf("fallback\n");
> > +		fflush(stdout);
> > +		return -1;
> > +	} else {
> > +		/* Half-Duplex Connection established */
> > +		printf("\n");
> > +		fflush(stdout);
> > +	}
> > +
> > +	proxy_state_init(&p, service_name);
> > +
> > +	/*
> > +	 * Dump the capability listing that we got from the server earlier
> > +	 * during the info/refs request.
> > +	 */
> > +	write_or_die(p.out, discover->buf, discover->len);
> > +
> > +	/* Peek the next packet line.  Until we see EOF keep sending POSTs */
> > +	while (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {
> > +		if (proxy_post(&p)) {
> > +			/* We would have an err here */
> 
> Probably better to comment "Error message already printed by
> proxy_post".

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [PATCH 12/26] ls-refs: introduce ls-refs server command
  2018-01-09 20:50   ` Jonathan Tan
@ 2018-01-16 19:23     ` Brandon Williams
  0 siblings, 0 replies; 59+ messages in thread
From: Brandon Williams @ 2018-01-16 19:23 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, sbeller, gitster, peff, philipoakley, stolee, jrnieder

On 01/09, Jonathan Tan wrote:
> On Tue,  2 Jan 2018 16:18:14 -0800
> Brandon Williams <bmwill@google.com> wrote:
> 
> > +  symrefs: In addition to the object pointed by it, show the underlying
> > +	   ref pointed by it when showing a symbolic ref.
> > +  peel: Show peeled tags.
> > +  ref-pattern <pattern>: When specified, only references matching the
> > +			 given patterns are displayed.
> 
> I notice "symrefs" being tested in patch 13 and "ref-pattern" being
> tested in patch 16. Is it possible to make a test for "peel" as well?
> (Or is it being tested somewhere I didn't notice?)

Really good suggestion.  I'll introduce unit tests for both the
git-serve cmdline as well as for more simple server commands (ls-refs).

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 59+ messages in thread

end of thread, back to index

Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-03  0:18 [PATCH 00/26] protocol version 2 Brandon Williams
2018-01-03  0:18 ` [PATCH 01/26] pkt-line: introduce packet_read_with_status Brandon Williams
2018-01-03 19:27   ` Stefan Beller
2018-01-05 23:41     ` Brandon Williams
2018-01-09 18:04   ` Jonathan Tan
2018-01-09 19:28     ` Brandon Williams
2018-01-03  0:18 ` [PATCH 02/26] pkt-line: introduce struct packet_reader Brandon Williams
2018-01-09 18:08   ` Jonathan Tan
2018-01-09 19:19     ` Brandon Williams
2018-01-03  0:18 ` [PATCH 03/26] pkt-line: add delim packet support Brandon Williams
2018-01-03  0:18 ` [PATCH 04/26] upload-pack: convert to a builtin Brandon Williams
2018-01-03 20:33   ` Stefan Beller
2018-01-03 20:39     ` Brandon Williams
2018-01-03  0:18 ` [PATCH 05/26] upload-pack: factor out processing lines Brandon Williams
2018-01-03 20:38   ` Stefan Beller
2018-01-03  0:18 ` [PATCH 06/26] transport: use get_refs_via_connect to get refs Brandon Williams
2018-01-03 21:20   ` Stefan Beller
2018-01-03  0:18 ` [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader Brandon Williams
2018-01-09 18:27   ` Jonathan Tan
2018-01-09 19:09     ` Brandon Williams
2018-01-03  0:18 ` [PATCH 08/26] connect: discover protocol version outside of get_remote_heads Brandon Williams
2018-01-03  0:18 ` [PATCH 09/26] transport: store protocol version Brandon Williams
2018-01-09 18:41   ` Jonathan Tan
2018-01-09 19:15     ` Brandon Williams
2018-01-03  0:18 ` [PATCH 10/26] protocol: introduce enum protocol_version value protocol_v2 Brandon Williams
2018-01-03  0:18 ` [PATCH 11/26] serve: introduce git-serve Brandon Williams
2018-01-09 20:24   ` Jonathan Tan
2018-01-09 22:16     ` Brandon Williams
2018-01-09 22:28       ` Jonathan Tan
2018-01-09 22:34         ` Brandon Williams
2018-01-03  0:18 ` [PATCH 12/26] ls-refs: introduce ls-refs server command Brandon Williams
2018-01-04  0:17   ` Stefan Beller
2018-01-05 23:49     ` Brandon Williams
2018-01-09 20:50   ` Jonathan Tan
2018-01-16 19:23     ` Brandon Williams
2018-01-03  0:18 ` [PATCH 13/26] connect: request remote refs using v2 Brandon Williams
2018-01-09 22:24   ` Jonathan Tan
2018-01-03  0:18 ` [PATCH 14/26] transport: convert get_refs_list to take a list of ref patterns Brandon Williams
2018-01-03  0:18 ` [PATCH 15/26] transport: convert transport_get_remote_refs " Brandon Williams
2018-01-03  0:18 ` [PATCH 16/26] ls-remote: pass ref patterns when requesting a remote's refs Brandon Williams
2018-01-03  0:18 ` [PATCH 17/26] fetch: pass ref patterns when fetching Brandon Williams
2018-01-03  0:18 ` [PATCH 18/26] push: pass ref patterns when pushing Brandon Williams
2018-01-03  0:18 ` [PATCH 19/26] upload-pack: introduce fetch server command Brandon Williams
2018-01-04  1:07   ` Stefan Beller
2018-01-03  0:18 ` [PATCH 20/26] fetch-pack: perform a fetch using v2 Brandon Williams
2018-01-04  1:23   ` Stefan Beller
2018-01-05 23:55     ` Brandon Williams
2018-01-10  0:05   ` Jonathan Tan
2018-01-03  0:18 ` [PATCH 21/26] transport-helper: remove name parameter Brandon Williams
2018-01-03  0:18 ` [PATCH 22/26] transport-helper: refactor process_connect_service Brandon Williams
2018-01-03  0:18 ` [PATCH 23/26] transport-helper: introduce connect-half-duplex Brandon Williams
2018-01-03  0:18 ` [PATCH 24/26] pkt-line: add packet_buf_write_len function Brandon Williams
2018-01-03  0:18 ` [PATCH 25/26] remote-curl: create copy of the service name Brandon Williams
2018-01-03  0:18 ` [PATCH 26/26] remote-curl: implement connect-half-duplex command Brandon Williams
2018-01-10  0:10   ` Jonathan Tan
2018-01-10 17:57   ` Jonathan Tan
2018-01-11  1:09     ` Brandon Williams
2018-01-09 17:55 ` [PATCH 00/26] protocol version 2 Jonathan Tan
2018-01-11  0:23   ` Brandon Williams

git@vger.kernel.org mailing list mirror (one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox