* [PATCH v2 01/27] pkt-line: introduce packet_read_with_status
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 02/27] pkt-line: introduce struct packet_reader Brandon Williams
` (28 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
The current pkt-line API encodes the status of a pkt-line read in the
length of the read content. An error is indicated with '-1', a flush
with '0' (which can be confusing since a return value of '0' can also
indicate an empty pkt-line), and a positive integer for the length of
the read content otherwise. This doesn't leave much room for allowing
the addition of additional special packets in the future.
To solve this introduce 'packet_read_with_status()' which reads a packet
and returns the status of the read encoded as an 'enum packet_status'
type. This allows for easily identifying between special and normal
packets as well as errors. It also enables easily adding a new special
packet in the future.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
pkt-line.c | 57 +++++++++++++++++++++++++++++++++++++++++++--------------
pkt-line.h | 15 +++++++++++++++
2 files changed, 58 insertions(+), 14 deletions(-)
diff --git a/pkt-line.c b/pkt-line.c
index 2827ca772..af0d2430f 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -280,28 +280,33 @@ static int packet_length(const char *linelen)
return (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);
}
-int packet_read(int fd, char **src_buf, size_t *src_len,
- char *buffer, unsigned size, int options)
+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
+ char *buffer, unsigned size, int *pktlen,
+ int options)
{
- int len, ret;
+ int len;
char linelen[4];
- ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);
- if (ret < 0)
- return ret;
+ if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)
+ return PACKET_READ_EOF;
+
len = packet_length(linelen);
- if (len < 0)
+
+ if (len < 0) {
die("protocol error: bad line length character: %.4s", linelen);
- if (!len) {
+ } else if (!len) {
packet_trace("0000", 4, 0);
- return 0;
+ return PACKET_READ_FLUSH;
+ } else if (len < 4) {
+ die("protocol error: bad line length %d", len);
}
+
len -= 4;
- if (len >= size)
+ if ((unsigned)len >= size)
die("protocol error: bad line length %d", len);
- ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);
- if (ret < 0)
- return ret;
+
+ if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)
+ return PACKET_READ_EOF;
if ((options & PACKET_READ_CHOMP_NEWLINE) &&
len && buffer[len-1] == '\n')
@@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,
buffer[len] = 0;
packet_trace(buffer, len, 0);
- return len;
+ *pktlen = len;
+ return PACKET_READ_NORMAL;
+}
+
+int packet_read(int fd, char **src_buffer, size_t *src_len,
+ char *buffer, unsigned size, int options)
+{
+ enum packet_read_status status;
+ int pktlen;
+
+ status = packet_read_with_status(fd, src_buffer, src_len,
+ buffer, size, &pktlen,
+ options);
+ switch (status) {
+ case PACKET_READ_EOF:
+ pktlen = -1;
+ break;
+ case PACKET_READ_NORMAL:
+ break;
+ case PACKET_READ_FLUSH:
+ pktlen = 0;
+ break;
+ }
+
+ return pktlen;
}
static char *packet_read_line_generic(int fd,
diff --git a/pkt-line.h b/pkt-line.h
index 3dad583e2..06c468927 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);
int packet_read(int fd, char **src_buffer, size_t *src_len, char
*buffer, unsigned size, int options);
+/*
+ * Read a packetized line into a buffer like the 'packet_read()' function but
+ * returns an 'enum packet_read_status' which indicates the status of the read.
+ * The number of bytes read will be assigined to *pktlen if the status of the
+ * read was 'PACKET_READ_NORMAL'.
+ */
+enum packet_read_status {
+ PACKET_READ_EOF = -1,
+ PACKET_READ_NORMAL,
+ PACKET_READ_FLUSH,
+};
+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
+ char *buffer, unsigned size, int *pktlen,
+ int options);
+
/*
* Convenience wrapper for packet_read that is not gentle, and sets the
* CHOMP_NEWLINE option. The return value is NULL for a flush packet,
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 02/27] pkt-line: introduce struct packet_reader
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
2018-01-25 23:58 ` [PATCH v2 01/27] pkt-line: introduce packet_read_with_status Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 03/27] pkt-line: add delim packet support Brandon Williams
` (27 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Sometimes it is advantageous to be able to peek the next packet line
without consuming it (e.g. to be able to determine the protocol version
a server is speaking). In order to do that introduce 'struct
packet_reader' which is an abstraction around the normal packet reading
logic. This enables a caller to be able to peek a single line at a time
using 'packet_reader_peek()' and having a caller consume a line by
calling 'packet_reader_read()'.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
pkt-line.h | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 117 insertions(+)
diff --git a/pkt-line.c b/pkt-line.c
index af0d2430f..4fc9ad4b0 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -406,3 +406,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)
}
return sb_out->len - orig_len;
}
+
+/* Packet Reader Functions */
+void packet_reader_init(struct packet_reader *reader, int fd,
+ char *src_buffer, size_t src_len,
+ int options)
+{
+ memset(reader, 0, sizeof(*reader));
+
+ reader->fd = fd;
+ reader->src_buffer = src_buffer;
+ reader->src_len = src_len;
+ reader->buffer = packet_buffer;
+ reader->buffer_size = sizeof(packet_buffer);
+ reader->options = options;
+}
+
+enum packet_read_status packet_reader_read(struct packet_reader *reader)
+{
+ if (reader->line_peeked) {
+ reader->line_peeked = 0;
+ return reader->status;
+ }
+
+ reader->status = packet_read_with_status(reader->fd,
+ &reader->src_buffer,
+ &reader->src_len,
+ reader->buffer,
+ reader->buffer_size,
+ &reader->pktlen,
+ reader->options);
+
+ switch (reader->status) {
+ case PACKET_READ_EOF:
+ reader->pktlen = -1;
+ reader->line = NULL;
+ break;
+ case PACKET_READ_NORMAL:
+ reader->line = reader->buffer;
+ break;
+ case PACKET_READ_FLUSH:
+ reader->pktlen = 0;
+ reader->line = NULL;
+ break;
+ }
+
+ return reader->status;
+}
+
+enum packet_read_status packet_reader_peek(struct packet_reader *reader)
+{
+ /* Only allow peeking a single line */
+ if (reader->line_peeked)
+ return reader->status;
+
+ /* Peek a line by reading it and setting peeked flag */
+ packet_reader_read(reader);
+ reader->line_peeked = 1;
+ return reader->status;
+}
diff --git a/pkt-line.h b/pkt-line.h
index 06c468927..7d9f0e537 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
*/
ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
+struct packet_reader {
+ /* source file descriptor */
+ int fd;
+
+ /* source buffer and its size */
+ char *src_buffer;
+ size_t src_len;
+
+ /* buffer that pkt-lines are read into and its size */
+ char *buffer;
+ unsigned buffer_size;
+
+ /* options to be used during reads */
+ int options;
+
+ /* status of the last read */
+ enum packet_read_status status;
+
+ /* length of data read during the last read */
+ int pktlen;
+
+ /* the last line read */
+ const char *line;
+
+ /* indicates if a line has been peeked */
+ int line_peeked;
+};
+
+/*
+ * Initialize a 'struct packet_reader' object which is an
+ * abstraction around the 'packet_read_with_status()' function.
+ */
+extern void packet_reader_init(struct packet_reader *reader, int fd,
+ char *src_buffer, size_t src_len,
+ int options);
+
+/*
+ * Perform a packet read and return the status of the read.
+ * The values of 'pktlen' and 'line' are updated based on the status of the
+ * read as follows:
+ *
+ * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL
+ * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read
+ * 'line' is set to point at the read line
+ * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL
+ */
+extern enum packet_read_status packet_reader_read(struct packet_reader *reader);
+
+/*
+ * Peek the next packet line without consuming it and return the status.
+ * The next call to 'packet_reader_read()' will perform a read of the same line
+ * that was peeked, consuming the line.
+ *
+ * Peeking multiple times without calling 'packet_reader_read()' will return
+ * the same result.
+ */
+extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);
+
#define DEFAULT_PACKET_MAX 1000
#define LARGE_PACKET_MAX 65520
#define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 03/27] pkt-line: add delim packet support
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
2018-01-25 23:58 ` [PATCH v2 01/27] pkt-line: introduce packet_read_with_status Brandon Williams
2018-01-25 23:58 ` [PATCH v2 02/27] pkt-line: introduce struct packet_reader Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 04/27] upload-pack: convert to a builtin Brandon Williams
` (26 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
One of the design goals of protocol-v2 is to improve the semantics of
flush packets. Currently in protocol-v1, flush packets are used both to
indicate a break in a list of packet lines as well as an indication that
one side has finished speaking. This makes it particularly difficult
to implement proxies as a proxy would need to completely understand git
protocol instead of simply looking for a flush packet.
To do this, introduce the special deliminator packet '0001'. A delim
packet can then be used as a deliminator between lists of packet lines
while flush packets can be reserved to indicate the end of a response.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
pkt-line.c | 17 +++++++++++++++++
pkt-line.h | 3 +++
2 files changed, 20 insertions(+)
diff --git a/pkt-line.c b/pkt-line.c
index 4fc9ad4b0..726e109ca 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -91,6 +91,12 @@ void packet_flush(int fd)
write_or_die(fd, "0000", 4);
}
+void packet_delim(int fd)
+{
+ packet_trace("0001", 4, 1);
+ write_or_die(fd, "0001", 4);
+}
+
int packet_flush_gently(int fd)
{
packet_trace("0000", 4, 1);
@@ -105,6 +111,12 @@ void packet_buf_flush(struct strbuf *buf)
strbuf_add(buf, "0000", 4);
}
+void packet_buf_delim(struct strbuf *buf)
+{
+ packet_trace("0001", 4, 1);
+ strbuf_add(buf, "0001", 4);
+}
+
static void set_packet_header(char *buf, const int size)
{
static char hexchar[] = "0123456789abcdef";
@@ -297,6 +309,9 @@ enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_
} else if (!len) {
packet_trace("0000", 4, 0);
return PACKET_READ_FLUSH;
+ } else if (len == 1) {
+ packet_trace("0001", 4, 0);
+ return PACKET_READ_DELIM;
} else if (len < 4) {
die("protocol error: bad line length %d", len);
}
@@ -333,6 +348,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len,
break;
case PACKET_READ_NORMAL:
break;
+ case PACKET_READ_DELIM:
case PACKET_READ_FLUSH:
pktlen = 0;
break;
@@ -445,6 +461,7 @@ enum packet_read_status packet_reader_read(struct packet_reader *reader)
case PACKET_READ_NORMAL:
reader->line = reader->buffer;
break;
+ case PACKET_READ_DELIM:
case PACKET_READ_FLUSH:
reader->pktlen = 0;
reader->line = NULL;
diff --git a/pkt-line.h b/pkt-line.h
index 7d9f0e537..16fe8bdbf 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -20,8 +20,10 @@
* side can't, we stay with pure read/write interfaces.
*/
void packet_flush(int fd);
+void packet_delim(int fd);
void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
void packet_buf_flush(struct strbuf *buf);
+void packet_buf_delim(struct strbuf *buf);
void packet_write(int fd_out, const char *buf, size_t size);
void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
int packet_flush_gently(int fd);
@@ -75,6 +77,7 @@ enum packet_read_status {
PACKET_READ_EOF = -1,
PACKET_READ_NORMAL,
PACKET_READ_FLUSH,
+ PACKET_READ_DELIM,
};
enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
char *buffer, unsigned size, int *pktlen,
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 04/27] upload-pack: convert to a builtin
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (2 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 03/27] pkt-line: add delim packet support Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 05/27] upload-pack: factor out processing lines Brandon Williams
` (25 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
In order to allow for code sharing with the server-side of fetch in
protocol-v2 convert upload-pack to be a builtin.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
Makefile | 3 +-
builtin.h | 1 +
builtin/upload-pack.c | 67 +++++++++++++++++++++++++++++++
git.c | 1 +
upload-pack.c | 107 ++++++++++++--------------------------------------
upload-pack.h | 13 ++++++
6 files changed, 109 insertions(+), 83 deletions(-)
create mode 100644 builtin/upload-pack.c
create mode 100644 upload-pack.h
diff --git a/Makefile b/Makefile
index 1a9b23b67..b7ccc05fa 100644
--- a/Makefile
+++ b/Makefile
@@ -639,7 +639,6 @@ PROGRAM_OBJS += imap-send.o
PROGRAM_OBJS += sh-i18n--envsubst.o
PROGRAM_OBJS += shell.o
PROGRAM_OBJS += show-index.o
-PROGRAM_OBJS += upload-pack.o
PROGRAM_OBJS += remote-testsvn.o
# Binary suffix, set to .exe for Windows builds
@@ -909,6 +908,7 @@ LIB_OBJS += tree-diff.o
LIB_OBJS += tree.o
LIB_OBJS += tree-walk.o
LIB_OBJS += unpack-trees.o
+LIB_OBJS += upload-pack.o
LIB_OBJS += url.o
LIB_OBJS += urlmatch.o
LIB_OBJS += usage.o
@@ -1026,6 +1026,7 @@ BUILTIN_OBJS += builtin/update-index.o
BUILTIN_OBJS += builtin/update-ref.o
BUILTIN_OBJS += builtin/update-server-info.o
BUILTIN_OBJS += builtin/upload-archive.o
+BUILTIN_OBJS += builtin/upload-pack.o
BUILTIN_OBJS += builtin/var.o
BUILTIN_OBJS += builtin/verify-commit.o
BUILTIN_OBJS += builtin/verify-pack.o
diff --git a/builtin.h b/builtin.h
index 42378f3aa..f332a1257 100644
--- a/builtin.h
+++ b/builtin.h
@@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);
extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);
extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);
extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);
+extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);
extern int cmd_var(int argc, const char **argv, const char *prefix);
extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);
extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);
diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c
new file mode 100644
index 000000000..2cb5cb35b
--- /dev/null
+++ b/builtin/upload-pack.c
@@ -0,0 +1,67 @@
+#include "cache.h"
+#include "builtin.h"
+#include "exec_cmd.h"
+#include "pkt-line.h"
+#include "parse-options.h"
+#include "protocol.h"
+#include "upload-pack.h"
+
+static const char * const upload_pack_usage[] = {
+ N_("git upload-pack [<options>] <dir>"),
+ NULL
+};
+
+int cmd_upload_pack(int argc, const char **argv, const char *prefix)
+{
+ const char *dir;
+ int strict = 0;
+ struct upload_pack_options opts = { 0 };
+ struct option options[] = {
+ OPT_BOOL(0, "stateless-rpc", &opts.stateless_rpc,
+ N_("quit after a single request/response exchange")),
+ OPT_BOOL(0, "advertise-refs", &opts.advertise_refs,
+ N_("exit immediately after initial ref advertisement")),
+ OPT_BOOL(0, "strict", &strict,
+ N_("do not try <directory>/.git/ if <directory> is no Git directory")),
+ OPT_INTEGER(0, "timeout", &opts.timeout,
+ N_("interrupt transfer after <n> seconds of inactivity")),
+ OPT_END()
+ };
+
+ packet_trace_identity("upload-pack");
+ check_replace_refs = 0;
+
+ argc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);
+
+ if (argc != 1)
+ usage_with_options(upload_pack_usage, options);
+
+ if (opts.timeout)
+ opts.daemon_mode = 1;
+
+ setup_path();
+
+ dir = argv[0];
+
+ if (!enter_repo(dir, strict))
+ die("'%s' does not appear to be a git repository", dir);
+
+ switch (determine_protocol_version_server()) {
+ case protocol_v1:
+ /*
+ * v1 is just the original protocol with a version string,
+ * so just fall through after writing the version string.
+ */
+ if (opts.advertise_refs || !opts.stateless_rpc)
+ packet_write_fmt(1, "version 1\n");
+
+ /* fallthrough */
+ case protocol_v0:
+ upload_pack(&opts);
+ break;
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
+
+ return 0;
+}
diff --git a/git.c b/git.c
index c870b9719..f71073dc8 100644
--- a/git.c
+++ b/git.c
@@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {
{ "update-server-info", cmd_update_server_info, RUN_SETUP },
{ "upload-archive", cmd_upload_archive },
{ "upload-archive--writer", cmd_upload_archive_writer },
+ { "upload-pack", cmd_upload_pack },
{ "var", cmd_var, RUN_SETUP_GENTLY },
{ "verify-commit", cmd_verify_commit, RUN_SETUP },
{ "verify-pack", cmd_verify_pack },
diff --git a/upload-pack.c b/upload-pack.c
index d5de18127..2ad73a98b 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -6,7 +6,6 @@
#include "tag.h"
#include "object.h"
#include "commit.h"
-#include "exec_cmd.h"
#include "diff.h"
#include "revision.h"
#include "list-objects.h"
@@ -15,15 +14,10 @@
#include "sigchain.h"
#include "version.h"
#include "string-list.h"
-#include "parse-options.h"
#include "argv-array.h"
#include "prio-queue.h"
#include "protocol.h"
-
-static const char * const upload_pack_usage[] = {
- N_("git upload-pack [<options>] <dir>"),
- NULL
-};
+#include "upload-pack.h"
/* Remember to update object flag allocation in object.h */
#define THEY_HAVE (1u << 11)
@@ -61,7 +55,6 @@ static int keepalive = 5;
* otherwise maximum packet size (up to 65520 bytes).
*/
static int use_sideband;
-static int advertise_refs;
static int stateless_rpc;
static const char *pack_objects_hook;
@@ -977,33 +970,6 @@ static int find_symref(const char *refname, const struct object_id *oid,
return 0;
}
-static void upload_pack(void)
-{
- struct string_list symref = STRING_LIST_INIT_DUP;
-
- head_ref_namespaced(find_symref, &symref);
-
- if (advertise_refs || !stateless_rpc) {
- reset_timeout();
- head_ref_namespaced(send_ref, &symref);
- for_each_namespaced_ref(send_ref, &symref);
- advertise_shallow_grafts(1);
- packet_flush(1);
- } else {
- head_ref_namespaced(check_ref, NULL);
- for_each_namespaced_ref(check_ref, NULL);
- }
- string_list_clear(&symref, 1);
- if (advertise_refs)
- return;
-
- receive_needs();
- if (want_obj.nr) {
- get_common_commits();
- create_pack_file();
- }
-}
-
static int upload_pack_config(const char *var, const char *value, void *unused)
{
if (!strcmp("uploadpack.allowtipsha1inwant", var)) {
@@ -1032,58 +998,35 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
return parse_hide_refs_config(var, value, "uploadpack");
}
-int cmd_main(int argc, const char **argv)
+void upload_pack(struct upload_pack_options *options)
{
- const char *dir;
- int strict = 0;
- struct option options[] = {
- OPT_BOOL(0, "stateless-rpc", &stateless_rpc,
- N_("quit after a single request/response exchange")),
- OPT_BOOL(0, "advertise-refs", &advertise_refs,
- N_("exit immediately after initial ref advertisement")),
- OPT_BOOL(0, "strict", &strict,
- N_("do not try <directory>/.git/ if <directory> is no Git directory")),
- OPT_INTEGER(0, "timeout", &timeout,
- N_("interrupt transfer after <n> seconds of inactivity")),
- OPT_END()
- };
-
- packet_trace_identity("upload-pack");
- check_replace_refs = 0;
-
- argc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);
-
- if (argc != 1)
- usage_with_options(upload_pack_usage, options);
-
- if (timeout)
- daemon_mode = 1;
-
- setup_path();
-
- dir = argv[0];
+ struct string_list symref = STRING_LIST_INIT_DUP;
- if (!enter_repo(dir, strict))
- die("'%s' does not appear to be a git repository", dir);
+ stateless_rpc = options->stateless_rpc;
+ timeout = options->timeout;
+ daemon_mode = options->daemon_mode;
git_config(upload_pack_config, NULL);
- switch (determine_protocol_version_server()) {
- case protocol_v1:
- /*
- * v1 is just the original protocol with a version string,
- * so just fall through after writing the version string.
- */
- if (advertise_refs || !stateless_rpc)
- packet_write_fmt(1, "version 1\n");
-
- /* fallthrough */
- case protocol_v0:
- upload_pack();
- break;
- case protocol_unknown_version:
- BUG("unknown protocol version");
+ head_ref_namespaced(find_symref, &symref);
+
+ if (options->advertise_refs || !stateless_rpc) {
+ reset_timeout();
+ head_ref_namespaced(send_ref, &symref);
+ for_each_namespaced_ref(send_ref, &symref);
+ advertise_shallow_grafts(1);
+ packet_flush(1);
+ } else {
+ head_ref_namespaced(check_ref, NULL);
+ for_each_namespaced_ref(check_ref, NULL);
}
+ string_list_clear(&symref, 1);
+ if (options->advertise_refs)
+ return;
- return 0;
+ receive_needs();
+ if (want_obj.nr) {
+ get_common_commits();
+ create_pack_file();
+ }
}
diff --git a/upload-pack.h b/upload-pack.h
new file mode 100644
index 000000000..a71e4dc7e
--- /dev/null
+++ b/upload-pack.h
@@ -0,0 +1,13 @@
+#ifndef UPLOAD_PACK_H
+#define UPLOAD_PACK_H
+
+struct upload_pack_options {
+ int stateless_rpc;
+ int advertise_refs;
+ unsigned int timeout;
+ int daemon_mode;
+};
+
+void upload_pack(struct upload_pack_options *options);
+
+#endif /* UPLOAD_PACK_H */
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 05/27] upload-pack: factor out processing lines
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (3 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 04/27] upload-pack: convert to a builtin Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-26 20:12 ` Stefan Beller
2018-01-25 23:58 ` [PATCH v2 06/27] transport: use get_refs_via_connect to get refs Brandon Williams
` (24 subsequent siblings)
29 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Factor out the logic for processing shallow, deepen, deepen_since, and
deepen_not lines into their own functions to simplify the
'receive_needs()' function in addition to making it easier to reuse some
of this logic when implementing protocol_v2.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------
1 file changed, 74 insertions(+), 39 deletions(-)
diff --git a/upload-pack.c b/upload-pack.c
index 2ad73a98b..42d83d5b1 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,
packet_flush(1);
}
+static int process_shallow(const char *line, struct object_array *shallows)
+{
+ const char *arg;
+ if (skip_prefix(line, "shallow ", &arg)) {
+ struct object_id oid;
+ struct object *object;
+ if (get_oid_hex(arg, &oid))
+ die("invalid shallow line: %s", line);
+ object = parse_object(&oid);
+ if (!object)
+ return 1;
+ if (object->type != OBJ_COMMIT)
+ die("invalid shallow object %s", oid_to_hex(&oid));
+ if (!(object->flags & CLIENT_SHALLOW)) {
+ object->flags |= CLIENT_SHALLOW;
+ add_object_array(object, NULL, shallows);
+ }
+ return 1;
+ }
+
+ return 0;
+}
+
+static int process_deepen(const char *line, int *depth)
+{
+ const char *arg;
+ if (skip_prefix(line, "deepen ", &arg)) {
+ char *end = NULL;
+ *depth = (int) strtol(arg, &end, 0);
+ if (!end || *end || *depth <= 0)
+ die("Invalid deepen: %s", line);
+ return 1;
+ }
+
+ return 0;
+}
+
+static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)
+{
+ const char *arg;
+ if (skip_prefix(line, "deepen-since ", &arg)) {
+ char *end = NULL;
+ *deepen_since = parse_timestamp(arg, &end, 0);
+ if (!end || *end || !deepen_since ||
+ /* revisions.c's max_age -1 is special */
+ *deepen_since == -1)
+ die("Invalid deepen-since: %s", line);
+ *deepen_rev_list = 1;
+ return 1;
+ }
+ return 0;
+}
+
+static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)
+{
+ const char *arg;
+ if (skip_prefix(line, "deepen-not ", &arg)) {
+ char *ref = NULL;
+ struct object_id oid;
+ if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
+ die("git upload-pack: ambiguous deepen-not: %s", line);
+ string_list_append(deepen_not, ref);
+ free(ref);
+ *deepen_rev_list = 1;
+ return 1;
+ }
+ return 0;
+}
+
static void receive_needs(void)
{
struct object_array shallows = OBJECT_ARRAY_INIT;
@@ -745,49 +814,15 @@ static void receive_needs(void)
if (!line)
break;
- if (skip_prefix(line, "shallow ", &arg)) {
- struct object_id oid;
- struct object *object;
- if (get_oid_hex(arg, &oid))
- die("invalid shallow line: %s", line);
- object = parse_object(&oid);
- if (!object)
- continue;
- if (object->type != OBJ_COMMIT)
- die("invalid shallow object %s", oid_to_hex(&oid));
- if (!(object->flags & CLIENT_SHALLOW)) {
- object->flags |= CLIENT_SHALLOW;
- add_object_array(object, NULL, &shallows);
- }
+ if (process_shallow(line, &shallows))
continue;
- }
- if (skip_prefix(line, "deepen ", &arg)) {
- char *end = NULL;
- depth = strtol(arg, &end, 0);
- if (!end || *end || depth <= 0)
- die("Invalid deepen: %s", line);
+ if (process_deepen(line, &depth))
continue;
- }
- if (skip_prefix(line, "deepen-since ", &arg)) {
- char *end = NULL;
- deepen_since = parse_timestamp(arg, &end, 0);
- if (!end || *end || !deepen_since ||
- /* revisions.c's max_age -1 is special */
- deepen_since == -1)
- die("Invalid deepen-since: %s", line);
- deepen_rev_list = 1;
+ if (process_deepen_since(line, &deepen_since, &deepen_rev_list))
continue;
- }
- if (skip_prefix(line, "deepen-not ", &arg)) {
- char *ref = NULL;
- struct object_id oid;
- if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
- die("git upload-pack: ambiguous deepen-not: %s", line);
- string_list_append(&deepen_not, ref);
- free(ref);
- deepen_rev_list = 1;
+ if (process_deepen_not(line, &deepen_not, &deepen_rev_list))
continue;
- }
+
if (!skip_prefix(line, "want ", &arg) ||
get_oid_hex(arg, &oid_buf))
die("git upload-pack: protocol error, "
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v2 05/27] upload-pack: factor out processing lines
2018-01-25 23:58 ` [PATCH v2 05/27] upload-pack: factor out processing lines Brandon Williams
@ 2018-01-26 20:12 ` Stefan Beller
2018-01-26 21:33 ` Brandon Williams
0 siblings, 1 reply; 362+ messages in thread
From: Stefan Beller @ 2018-01-26 20:12 UTC (permalink / raw)
To: Brandon Williams
Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee,
Jonathan Nieder
On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams <bmwill@google.com> wrote:
> Factor out the logic for processing shallow, deepen, deepen_since, and
> deepen_not lines into their own functions to simplify the
> 'receive_needs()' function in addition to making it easier to reuse some
> of this logic when implementing protocol_v2.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
> upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------
> 1 file changed, 74 insertions(+), 39 deletions(-)
>
> diff --git a/upload-pack.c b/upload-pack.c
> index 2ad73a98b..42d83d5b1 100644
> --- a/upload-pack.c
> +++ b/upload-pack.c
> @@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,
> packet_flush(1);
> }
>
> +static int process_shallow(const char *line, struct object_array *shallows)
> +{
> + const char *arg;
> + if (skip_prefix(line, "shallow ", &arg)) {
stylistic nit:
You could invert the condition in each of the process_* functions
to just have
if (!skip_prefix...))
return 0
/* less indented code goes here */
return 1;
That way we have less indentation as well as easier code.
(The reader doesn't need to keep in mind what the else
part is about; it is a rather local decision to bail out instead
of having the return at the end of the function.)
> + struct object_id oid;
> + struct object *object;
> + if (get_oid_hex(arg, &oid))
> + die("invalid shallow line: %s", line);
> + object = parse_object(&oid);
> + if (!object)
> + return 1;
> + if (object->type != OBJ_COMMIT)
> + die("invalid shallow object %s", oid_to_hex(&oid));
> + if (!(object->flags & CLIENT_SHALLOW)) {
> + object->flags |= CLIENT_SHALLOW;
> + add_object_array(object, NULL, shallows);
> + }
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> +static int process_deepen(const char *line, int *depth)
> +{
> + const char *arg;
> + if (skip_prefix(line, "deepen ", &arg)) {
> + char *end = NULL;
> + *depth = (int) strtol(arg, &end, 0);
> + if (!end || *end || *depth <= 0)
> + die("Invalid deepen: %s", line);
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> +static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)
> +{
> + const char *arg;
> + if (skip_prefix(line, "deepen-since ", &arg)) {
> + char *end = NULL;
> + *deepen_since = parse_timestamp(arg, &end, 0);
> + if (!end || *end || !deepen_since ||
> + /* revisions.c's max_age -1 is special */
> + *deepen_since == -1)
> + die("Invalid deepen-since: %s", line);
> + *deepen_rev_list = 1;
> + return 1;
> + }
> + return 0;
> +}
> +
> +static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)
> +{
> + const char *arg;
> + if (skip_prefix(line, "deepen-not ", &arg)) {
> + char *ref = NULL;
> + struct object_id oid;
> + if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
> + die("git upload-pack: ambiguous deepen-not: %s", line);
> + string_list_append(deepen_not, ref);
> + free(ref);
> + *deepen_rev_list = 1;
> + return 1;
> + }
> + return 0;
> +}
> +
> static void receive_needs(void)
> {
> struct object_array shallows = OBJECT_ARRAY_INIT;
> @@ -745,49 +814,15 @@ static void receive_needs(void)
> if (!line)
> break;
>
> - if (skip_prefix(line, "shallow ", &arg)) {
> - struct object_id oid;
> - struct object *object;
> - if (get_oid_hex(arg, &oid))
> - die("invalid shallow line: %s", line);
> - object = parse_object(&oid);
> - if (!object)
> - continue;
> - if (object->type != OBJ_COMMIT)
> - die("invalid shallow object %s", oid_to_hex(&oid));
> - if (!(object->flags & CLIENT_SHALLOW)) {
> - object->flags |= CLIENT_SHALLOW;
> - add_object_array(object, NULL, &shallows);
> - }
> + if (process_shallow(line, &shallows))
> continue;
> - }
> - if (skip_prefix(line, "deepen ", &arg)) {
> - char *end = NULL;
> - depth = strtol(arg, &end, 0);
> - if (!end || *end || depth <= 0)
> - die("Invalid deepen: %s", line);
> + if (process_deepen(line, &depth))
> continue;
> - }
> - if (skip_prefix(line, "deepen-since ", &arg)) {
> - char *end = NULL;
> - deepen_since = parse_timestamp(arg, &end, 0);
> - if (!end || *end || !deepen_since ||
> - /* revisions.c's max_age -1 is special */
> - deepen_since == -1)
> - die("Invalid deepen-since: %s", line);
> - deepen_rev_list = 1;
> + if (process_deepen_since(line, &deepen_since, &deepen_rev_list))
> continue;
> - }
> - if (skip_prefix(line, "deepen-not ", &arg)) {
> - char *ref = NULL;
> - struct object_id oid;
> - if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
> - die("git upload-pack: ambiguous deepen-not: %s", line);
> - string_list_append(&deepen_not, ref);
> - free(ref);
> - deepen_rev_list = 1;
> + if (process_deepen_not(line, &deepen_not, &deepen_rev_list))
> continue;
> - }
> +
> if (!skip_prefix(line, "want ", &arg) ||
> get_oid_hex(arg, &oid_buf))
> die("git upload-pack: protocol error, "
> --
> 2.16.0.rc1.238.g530d649a79-goog
>
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 05/27] upload-pack: factor out processing lines
2018-01-26 20:12 ` Stefan Beller
@ 2018-01-26 21:33 ` Brandon Williams
2018-01-31 14:08 ` Derrick Stolee
0 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-01-26 21:33 UTC (permalink / raw)
To: Stefan Beller
Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee,
Jonathan Nieder
On 01/26, Stefan Beller wrote:
> On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams <bmwill@google.com> wrote:
> > Factor out the logic for processing shallow, deepen, deepen_since, and
> > deepen_not lines into their own functions to simplify the
> > 'receive_needs()' function in addition to making it easier to reuse some
> > of this logic when implementing protocol_v2.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> > upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------
> > 1 file changed, 74 insertions(+), 39 deletions(-)
> >
> > diff --git a/upload-pack.c b/upload-pack.c
> > index 2ad73a98b..42d83d5b1 100644
> > --- a/upload-pack.c
> > +++ b/upload-pack.c
> > @@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,
> > packet_flush(1);
> > }
> >
> > +static int process_shallow(const char *line, struct object_array *shallows)
> > +{
> > + const char *arg;
> > + if (skip_prefix(line, "shallow ", &arg)) {
>
> stylistic nit:
>
> You could invert the condition in each of the process_* functions
> to just have
>
> if (!skip_prefix...))
> return 0
>
> /* less indented code goes here */
>
> return 1;
>
> That way we have less indentation as well as easier code.
> (The reader doesn't need to keep in mind what the else
> part is about; it is a rather local decision to bail out instead
> of having the return at the end of the function.)
I was trying to move the existing code into helper functions so
rewriting them in transit may make it less reviewable?
>
> > + struct object_id oid;
> > + struct object *object;
> > + if (get_oid_hex(arg, &oid))
> > + die("invalid shallow line: %s", line);
> > + object = parse_object(&oid);
> > + if (!object)
> > + return 1;
> > + if (object->type != OBJ_COMMIT)
> > + die("invalid shallow object %s", oid_to_hex(&oid));
> > + if (!(object->flags & CLIENT_SHALLOW)) {
> > + object->flags |= CLIENT_SHALLOW;
> > + add_object_array(object, NULL, shallows);
> > + }
> > + return 1;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int process_deepen(const char *line, int *depth)
> > +{
> > + const char *arg;
> > + if (skip_prefix(line, "deepen ", &arg)) {
> > + char *end = NULL;
> > + *depth = (int) strtol(arg, &end, 0);
> > + if (!end || *end || *depth <= 0)
> > + die("Invalid deepen: %s", line);
> > + return 1;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)
> > +{
> > + const char *arg;
> > + if (skip_prefix(line, "deepen-since ", &arg)) {
> > + char *end = NULL;
> > + *deepen_since = parse_timestamp(arg, &end, 0);
> > + if (!end || *end || !deepen_since ||
> > + /* revisions.c's max_age -1 is special */
> > + *deepen_since == -1)
> > + die("Invalid deepen-since: %s", line);
> > + *deepen_rev_list = 1;
> > + return 1;
> > + }
> > + return 0;
> > +}
> > +
> > +static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)
> > +{
> > + const char *arg;
> > + if (skip_prefix(line, "deepen-not ", &arg)) {
> > + char *ref = NULL;
> > + struct object_id oid;
> > + if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
> > + die("git upload-pack: ambiguous deepen-not: %s", line);
> > + string_list_append(deepen_not, ref);
> > + free(ref);
> > + *deepen_rev_list = 1;
> > + return 1;
> > + }
> > + return 0;
> > +}
> > +
> > static void receive_needs(void)
> > {
> > struct object_array shallows = OBJECT_ARRAY_INIT;
> > @@ -745,49 +814,15 @@ static void receive_needs(void)
> > if (!line)
> > break;
> >
> > - if (skip_prefix(line, "shallow ", &arg)) {
> > - struct object_id oid;
> > - struct object *object;
> > - if (get_oid_hex(arg, &oid))
> > - die("invalid shallow line: %s", line);
> > - object = parse_object(&oid);
> > - if (!object)
> > - continue;
> > - if (object->type != OBJ_COMMIT)
> > - die("invalid shallow object %s", oid_to_hex(&oid));
> > - if (!(object->flags & CLIENT_SHALLOW)) {
> > - object->flags |= CLIENT_SHALLOW;
> > - add_object_array(object, NULL, &shallows);
> > - }
> > + if (process_shallow(line, &shallows))
> > continue;
> > - }
> > - if (skip_prefix(line, "deepen ", &arg)) {
> > - char *end = NULL;
> > - depth = strtol(arg, &end, 0);
> > - if (!end || *end || depth <= 0)
> > - die("Invalid deepen: %s", line);
> > + if (process_deepen(line, &depth))
> > continue;
> > - }
> > - if (skip_prefix(line, "deepen-since ", &arg)) {
> > - char *end = NULL;
> > - deepen_since = parse_timestamp(arg, &end, 0);
> > - if (!end || *end || !deepen_since ||
> > - /* revisions.c's max_age -1 is special */
> > - deepen_since == -1)
> > - die("Invalid deepen-since: %s", line);
> > - deepen_rev_list = 1;
> > + if (process_deepen_since(line, &deepen_since, &deepen_rev_list))
> > continue;
> > - }
> > - if (skip_prefix(line, "deepen-not ", &arg)) {
> > - char *ref = NULL;
> > - struct object_id oid;
> > - if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
> > - die("git upload-pack: ambiguous deepen-not: %s", line);
> > - string_list_append(&deepen_not, ref);
> > - free(ref);
> > - deepen_rev_list = 1;
> > + if (process_deepen_not(line, &deepen_not, &deepen_rev_list))
> > continue;
> > - }
> > +
> > if (!skip_prefix(line, "want ", &arg) ||
> > get_oid_hex(arg, &oid_buf))
> > die("git upload-pack: protocol error, "
> > --
> > 2.16.0.rc1.238.g530d649a79-goog
> >
--
Brandon Williams
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 05/27] upload-pack: factor out processing lines
2018-01-26 21:33 ` Brandon Williams
@ 2018-01-31 14:08 ` Derrick Stolee
0 siblings, 0 replies; 362+ messages in thread
From: Derrick Stolee @ 2018-01-31 14:08 UTC (permalink / raw)
To: Brandon Williams, Stefan Beller
Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Jonathan Nieder
On 1/26/2018 4:33 PM, Brandon Williams wrote:
> On 01/26, Stefan Beller wrote:
>> On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams <bmwill@google.com> wrote:
>>> Factor out the logic for processing shallow, deepen, deepen_since, and
>>> deepen_not lines into their own functions to simplify the
>>> 'receive_needs()' function in addition to making it easier to reuse some
>>> of this logic when implementing protocol_v2.
>>>
>>> Signed-off-by: Brandon Williams <bmwill@google.com>
>>> ---
>>> upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------
>>> 1 file changed, 74 insertions(+), 39 deletions(-)
>>>
>>> diff --git a/upload-pack.c b/upload-pack.c
>>> index 2ad73a98b..42d83d5b1 100644
>>> --- a/upload-pack.c
>>> +++ b/upload-pack.c
>>> @@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,
>>> packet_flush(1);
>>> }
>>>
>>> +static int process_shallow(const char *line, struct object_array *shallows)
>>> +{
>>> + const char *arg;
>>> + if (skip_prefix(line, "shallow ", &arg)) {
>> stylistic nit:
>>
>> You could invert the condition in each of the process_* functions
>> to just have
>>
>> if (!skip_prefix...))
>> return 0
>>
>> /* less indented code goes here */
>>
>> return 1;
>>
>> That way we have less indentation as well as easier code.
>> (The reader doesn't need to keep in mind what the else
>> part is about; it is a rather local decision to bail out instead
>> of having the return at the end of the function.)
> I was trying to move the existing code into helper functions so
> rewriting them in transit may make it less reviewable?
I think the way you kept to the existing code as much as possible is
good and easier to review. Perhaps a style pass after the patch lands is
good for #leftoverbits.
>>
>>> + struct object_id oid;
>>> + struct object *object;
>>> + if (get_oid_hex(arg, &oid))
>>> + die("invalid shallow line: %s", line);
>>> + object = parse_object(&oid);
>>> + if (!object)
>>> + return 1;
>>> + if (object->type != OBJ_COMMIT)
>>> + die("invalid shallow object %s", oid_to_hex(&oid));
>>> + if (!(object->flags & CLIENT_SHALLOW)) {
>>> + object->flags |= CLIENT_SHALLOW;
>>> + add_object_array(object, NULL, shallows);
>>> + }
>>> + return 1;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static int process_deepen(const char *line, int *depth)
>>> +{
>>> + const char *arg;
>>> + if (skip_prefix(line, "deepen ", &arg)) {
>>> + char *end = NULL;
>>> + *depth = (int) strtol(arg, &end, 0);
nit: space between (int) and strtol?
>>> + if (!end || *end || *depth <= 0)
>>> + die("Invalid deepen: %s", line);
>>> + return 1;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)
>>> +{
>>> + const char *arg;
>>> + if (skip_prefix(line, "deepen-since ", &arg)) {
>>> + char *end = NULL;
>>> + *deepen_since = parse_timestamp(arg, &end, 0);
>>> + if (!end || *end || !deepen_since ||
>>> + /* revisions.c's max_age -1 is special */
>>> + *deepen_since == -1)
>>> + die("Invalid deepen-since: %s", line);
>>> + *deepen_rev_list = 1;
>>> + return 1;
>>> + }
>>> + return 0;
>>> +}
>>> +
>>> +static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)
>>> +{
>>> + const char *arg;
>>> + if (skip_prefix(line, "deepen-not ", &arg)) {
>>> + char *ref = NULL;
>>> + struct object_id oid;
>>> + if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
>>> + die("git upload-pack: ambiguous deepen-not: %s", line);
>>> + string_list_append(deepen_not, ref);
>>> + free(ref);
>>> + *deepen_rev_list = 1;
>>> + return 1;
>>> + }
>>> + return 0;
>>> +}
>>> +
>>> static void receive_needs(void)
>>> {
>>> struct object_array shallows = OBJECT_ARRAY_INIT;
>>> @@ -745,49 +814,15 @@ static void receive_needs(void)
>>> if (!line)
>>> break;
>>>
>>> - if (skip_prefix(line, "shallow ", &arg)) {
>>> - struct object_id oid;
>>> - struct object *object;
>>> - if (get_oid_hex(arg, &oid))
>>> - die("invalid shallow line: %s", line);
>>> - object = parse_object(&oid);
>>> - if (!object)
>>> - continue;
>>> - if (object->type != OBJ_COMMIT)
>>> - die("invalid shallow object %s", oid_to_hex(&oid));
>>> - if (!(object->flags & CLIENT_SHALLOW)) {
>>> - object->flags |= CLIENT_SHALLOW;
>>> - add_object_array(object, NULL, &shallows);
>>> - }
>>> + if (process_shallow(line, &shallows))
>>> continue;
>>> - }
>>> - if (skip_prefix(line, "deepen ", &arg)) {
>>> - char *end = NULL;
>>> - depth = strtol(arg, &end, 0);
>>> - if (!end || *end || depth <= 0)
>>> - die("Invalid deepen: %s", line);
>>> + if (process_deepen(line, &depth))
>>> continue;
>>> - }
>>> - if (skip_prefix(line, "deepen-since ", &arg)) {
>>> - char *end = NULL;
>>> - deepen_since = parse_timestamp(arg, &end, 0);
>>> - if (!end || *end || !deepen_since ||
>>> - /* revisions.c's max_age -1 is special */
>>> - deepen_since == -1)
>>> - die("Invalid deepen-since: %s", line);
>>> - deepen_rev_list = 1;
>>> + if (process_deepen_since(line, &deepen_since, &deepen_rev_list))
>>> continue;
>>> - }
>>> - if (skip_prefix(line, "deepen-not ", &arg)) {
>>> - char *ref = NULL;
>>> - struct object_id oid;
>>> - if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)
>>> - die("git upload-pack: ambiguous deepen-not: %s", line);
>>> - string_list_append(&deepen_not, ref);
>>> - free(ref);
>>> - deepen_rev_list = 1;
>>> + if (process_deepen_not(line, &deepen_not, &deepen_rev_list))
>>> continue;
>>> - }
>>> +
>>> if (!skip_prefix(line, "want ", &arg) ||
>>> get_oid_hex(arg, &oid_buf))
>>> die("git upload-pack: protocol error, "
>>> --
>>> 2.16.0.rc1.238.g530d649a79-goog
>>>
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v2 06/27] transport: use get_refs_via_connect to get refs
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (4 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 05/27] upload-pack: factor out processing lines Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 07/27] connect: convert get_remote_heads to use struct packet_reader Brandon Williams
` (23 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Remove code duplication and use the existing 'get_refs_via_connect()'
function to retrieve a remote's heads in 'fetch_refs_via_pack()' and
'git_transport_push()'.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
transport.c | 18 ++++--------------
1 file changed, 4 insertions(+), 14 deletions(-)
diff --git a/transport.c b/transport.c
index fc802260f..8e8779096 100644
--- a/transport.c
+++ b/transport.c
@@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,
args.cloning = transport->cloning;
args.update_shallow = data->options.update_shallow;
- if (!data->got_remote_heads) {
- connect_setup(transport, 0);
- get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
- NULL, &data->shallow);
- data->got_remote_heads = 1;
- }
+ if (!data->got_remote_heads)
+ refs_tmp = get_refs_via_connect(transport, 0);
refs = fetch_pack(&args, data->fd, data->conn,
refs_tmp ? refs_tmp : transport->remote_refs,
@@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
struct send_pack_args args;
int ret;
- if (!data->got_remote_heads) {
- struct ref *tmp_refs;
- connect_setup(transport, 1);
-
- get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,
- NULL, &data->shallow);
- data->got_remote_heads = 1;
- }
+ if (!data->got_remote_heads)
+ get_refs_via_connect(transport, 1);
memset(&args, 0, sizeof(args));
args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 07/27] connect: convert get_remote_heads to use struct packet_reader
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (5 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 06/27] transport: use get_refs_via_connect to get refs Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads Brandon Williams
` (22 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
In order to allow for better control flow when protocol_v2 is introduced
convert 'get_remote_heads()' to use 'struct packet_reader' to read
packet lines. This enables a client to be able to peek the first line
of a server's response (without consuming it) in order to determine the
protocol version its speaking and then passing control to the
appropriate handler.
This is needed because the initial response from a server speaking
protocol_v0 includes the first ref, while subsequent protocol versions
respond with a version line. We want to be able to read this first line
without consuming the first ref sent in the protocol_v0 case so that the
protocol version the server is speaking can be determined outside of
'get_remote_heads()' in a future patch.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
connect.c | 174 ++++++++++++++++++++++++++++++++++----------------------------
1 file changed, 96 insertions(+), 78 deletions(-)
diff --git a/connect.c b/connect.c
index c3a014c5b..00e90075c 100644
--- a/connect.c
+++ b/connect.c
@@ -48,6 +48,12 @@ int check_ref_type(const struct ref *ref, int flags)
static void die_initial_contact(int unexpected)
{
+ /*
+ * A hang-up after seeing some response from the other end
+ * means that it is unexpected, as we know the other end is
+ * willing to talk to us. A hang-up before seeing any
+ * response does not necessarily mean an ACL problem, though.
+ */
if (unexpected)
die(_("The remote end hung up upon initial contact"));
else
@@ -56,6 +62,41 @@ static void die_initial_contact(int unexpected)
"and the repository exists."));
}
+static enum protocol_version discover_version(struct packet_reader *reader)
+{
+ enum protocol_version version = protocol_unknown_version;
+
+ /*
+ * Peek the first line of the server's response to
+ * determine the protocol version the server is speaking.
+ */
+ switch (packet_reader_peek(reader)) {
+ case PACKET_READ_EOF:
+ die_initial_contact(0);
+ case PACKET_READ_FLUSH:
+ case PACKET_READ_DELIM:
+ version = protocol_v0;
+ break;
+ case PACKET_READ_NORMAL:
+ version = determine_protocol_version_client(reader->line);
+ break;
+ }
+
+ /* Maybe process capabilities here, at least for v2 */
+ switch (version) {
+ case protocol_v1:
+ /* Read the peeked version line */
+ packet_reader_read(reader);
+ break;
+ case protocol_v0:
+ break;
+ case protocol_unknown_version:
+ die("unknown protocol version: '%s'\n", reader->line);
+ }
+
+ return version;
+}
+
static void parse_one_symref_info(struct string_list *symref, const char *val, int len)
{
char *sym, *target;
@@ -109,60 +150,21 @@ static void annotate_refs_with_symref_info(struct ref *ref)
string_list_clear(&symref, 0);
}
-/*
- * Read one line of a server's ref advertisement into packet_buffer.
- */
-static int read_remote_ref(int in, char **src_buf, size_t *src_len,
- int *responded)
+static void process_capabilities(const char *line, int *len)
{
- int len = packet_read(in, src_buf, src_len,
- packet_buffer, sizeof(packet_buffer),
- PACKET_READ_GENTLE_ON_EOF |
- PACKET_READ_CHOMP_NEWLINE);
- const char *arg;
- if (len < 0)
- die_initial_contact(*responded);
- if (len > 4 && skip_prefix(packet_buffer, "ERR ", &arg))
- die("remote error: %s", arg);
-
- *responded = 1;
-
- return len;
-}
-
-#define EXPECTING_PROTOCOL_VERSION 0
-#define EXPECTING_FIRST_REF 1
-#define EXPECTING_REF 2
-#define EXPECTING_SHALLOW 3
-
-/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */
-static int process_protocol_version(void)
-{
- switch (determine_protocol_version_client(packet_buffer)) {
- case protocol_v1:
- return 1;
- case protocol_v0:
- return 0;
- default:
- die("server is speaking an unknown protocol");
- }
-}
-
-static void process_capabilities(int *len)
-{
- int nul_location = strlen(packet_buffer);
+ int nul_location = strlen(line);
if (nul_location == *len)
return;
- server_capabilities = xstrdup(packet_buffer + nul_location + 1);
+ server_capabilities = xstrdup(line + nul_location + 1);
*len = nul_location;
}
-static int process_dummy_ref(void)
+static int process_dummy_ref(const char *line)
{
struct object_id oid;
const char *name;
- if (parse_oid_hex(packet_buffer, &oid, &name))
+ if (parse_oid_hex(line, &oid, &name))
return 0;
if (*name != ' ')
return 0;
@@ -171,20 +173,20 @@ static int process_dummy_ref(void)
return !oidcmp(&null_oid, &oid) && !strcmp(name, "capabilities^{}");
}
-static void check_no_capabilities(int len)
+static void check_no_capabilities(const char *line, int len)
{
- if (strlen(packet_buffer) != len)
+ if (strlen(line) != len)
warning("Ignoring capabilities after first line '%s'",
- packet_buffer + strlen(packet_buffer));
+ line + strlen(line));
}
-static int process_ref(int len, struct ref ***list, unsigned int flags,
- struct oid_array *extra_have)
+static int process_ref(const char *line, int len, struct ref ***list,
+ unsigned int flags, struct oid_array *extra_have)
{
struct object_id old_oid;
const char *name;
- if (parse_oid_hex(packet_buffer, &old_oid, &name))
+ if (parse_oid_hex(line, &old_oid, &name))
return 0;
if (*name != ' ')
return 0;
@@ -200,16 +202,17 @@ static int process_ref(int len, struct ref ***list, unsigned int flags,
**list = ref;
*list = &ref->next;
}
- check_no_capabilities(len);
+ check_no_capabilities(line, len);
return 1;
}
-static int process_shallow(int len, struct oid_array *shallow_points)
+static int process_shallow(const char *line, int len,
+ struct oid_array *shallow_points)
{
const char *arg;
struct object_id old_oid;
- if (!skip_prefix(packet_buffer, "shallow ", &arg))
+ if (!skip_prefix(line, "shallow ", &arg))
return 0;
if (get_oid_hex(arg, &old_oid))
@@ -217,10 +220,17 @@ static int process_shallow(int len, struct oid_array *shallow_points)
if (!shallow_points)
die("repository on the other end cannot be shallow");
oid_array_append(shallow_points, &old_oid);
- check_no_capabilities(len);
+ check_no_capabilities(line, len);
return 1;
}
+enum get_remote_heads_state {
+ EXPECTING_FIRST_REF = 0,
+ EXPECTING_REF,
+ EXPECTING_SHALLOW,
+ EXPECTING_DONE,
+};
+
/*
* Read all the refs from the other end
*/
@@ -230,47 +240,55 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
struct oid_array *shallow_points)
{
struct ref **orig_list = list;
+ int len = 0;
+ enum get_remote_heads_state state = EXPECTING_FIRST_REF;
+ struct packet_reader reader;
+ const char *arg;
- /*
- * A hang-up after seeing some response from the other end
- * means that it is unexpected, as we know the other end is
- * willing to talk to us. A hang-up before seeing any
- * response does not necessarily mean an ACL problem, though.
- */
- int responded = 0;
- int len;
- int state = EXPECTING_PROTOCOL_VERSION;
+ packet_reader_init(&reader, in, src_buf, src_len,
+ PACKET_READ_CHOMP_NEWLINE |
+ PACKET_READ_GENTLE_ON_EOF);
+
+ discover_version(&reader);
*list = NULL;
- while ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {
+ while (state != EXPECTING_DONE) {
+ switch (packet_reader_read(&reader)) {
+ case PACKET_READ_EOF:
+ die_initial_contact(1);
+ case PACKET_READ_NORMAL:
+ len = reader.pktlen;
+ if (len > 4 && skip_prefix(reader.line, "ERR ", &arg))
+ die("remote error: %s", arg);
+ break;
+ case PACKET_READ_FLUSH:
+ state = EXPECTING_DONE;
+ break;
+ case PACKET_READ_DELIM:
+ die("invalid packet");
+ }
+
switch (state) {
- case EXPECTING_PROTOCOL_VERSION:
- if (process_protocol_version()) {
- state = EXPECTING_FIRST_REF;
- break;
- }
- state = EXPECTING_FIRST_REF;
- /* fallthrough */
case EXPECTING_FIRST_REF:
- process_capabilities(&len);
- if (process_dummy_ref()) {
+ process_capabilities(reader.line, &len);
+ if (process_dummy_ref(reader.line)) {
state = EXPECTING_SHALLOW;
break;
}
state = EXPECTING_REF;
/* fallthrough */
case EXPECTING_REF:
- if (process_ref(len, &list, flags, extra_have))
+ if (process_ref(reader.line, len, &list, flags, extra_have))
break;
state = EXPECTING_SHALLOW;
/* fallthrough */
case EXPECTING_SHALLOW:
- if (process_shallow(len, shallow_points))
+ if (process_shallow(reader.line, len, shallow_points))
break;
- die("protocol error: unexpected '%s'", packet_buffer);
- default:
- die("unexpected state %d", state);
+ die("protocol error: unexpected '%s'", reader.line);
+ case EXPECTING_DONE:
+ break;
}
}
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (6 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 07/27] connect: convert get_remote_heads to use struct packet_reader Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-31 14:40 ` Derrick Stolee
2018-01-25 23:58 ` [PATCH v2 09/27] transport: store protocol version Brandon Williams
` (21 subsequent siblings)
29 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
In order to prepare for the addition of protocol_v2 push the protocol
version discovery outside of 'get_remote_heads()'. This will allow for
keeping the logic for processing the reference advertisement for
protocol_v1 and protocol_v0 separate from the logic for protocol_v2.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
builtin/fetch-pack.c | 16 +++++++++++++++-
builtin/send-pack.c | 17 +++++++++++++++--
connect.c | 27 ++++++++++-----------------
connect.h | 3 +++
remote-curl.c | 20 ++++++++++++++++++--
remote.h | 5 +++--
transport.c | 24 +++++++++++++++++++-----
7 files changed, 83 insertions(+), 29 deletions(-)
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 366b9d13f..85d4faf76 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -4,6 +4,7 @@
#include "remote.h"
#include "connect.h"
#include "sha1-array.h"
+#include "protocol.h"
static const char fetch_pack_usage[] =
"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] "
@@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
struct fetch_pack_args args;
struct oid_array shallow = OID_ARRAY_INIT;
struct string_list deepen_not = STRING_LIST_INIT_DUP;
+ struct packet_reader reader;
packet_trace_identity("fetch-pack");
@@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
if (!conn)
return args.diag_url ? 0 : 1;
}
- get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
+
+ packet_reader_init(&reader, fd[0], NULL, 0,
+ PACKET_READ_CHOMP_NEWLINE |
+ PACKET_READ_GENTLE_ON_EOF);
+
+ switch (discover_version(&reader)) {
+ case protocol_v1:
+ case protocol_v0:
+ get_remote_heads(&reader, &ref, 0, NULL, &shallow);
+ break;
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,
&shallow, pack_lockfile_ptr);
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index fc4f0bb5f..83cb125a6 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -14,6 +14,7 @@
#include "sha1-array.h"
#include "gpg-interface.h"
#include "gettext.h"
+#include "protocol.h"
static const char * const send_pack_usage[] = {
N_("git send-pack [--all | --mirror] [--dry-run] [--force] "
@@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
int progress = -1;
int from_stdin = 0;
struct push_cas_option cas = {0};
+ struct packet_reader reader;
struct option options[] = {
OPT__VERBOSITY(&verbose),
@@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
args.verbose ? CONNECT_VERBOSE : 0);
}
- get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,
- &extra_have, &shallow);
+ packet_reader_init(&reader, fd[0], NULL, 0,
+ PACKET_READ_CHOMP_NEWLINE |
+ PACKET_READ_GENTLE_ON_EOF);
+
+ switch (discover_version(&reader)) {
+ case protocol_v1:
+ case protocol_v0:
+ get_remote_heads(&reader, &remote_refs, REF_NORMAL,
+ &extra_have, &shallow);
+ break;
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
transport_verify_remote_names(nr_refspecs, refspecs);
diff --git a/connect.c b/connect.c
index 00e90075c..db3c9d24c 100644
--- a/connect.c
+++ b/connect.c
@@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)
"and the repository exists."));
}
-static enum protocol_version discover_version(struct packet_reader *reader)
+enum protocol_version discover_version(struct packet_reader *reader)
{
enum protocol_version version = protocol_unknown_version;
@@ -234,7 +234,7 @@ enum get_remote_heads_state {
/*
* Read all the refs from the other end
*/
-struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
+struct ref **get_remote_heads(struct packet_reader *reader,
struct ref **list, unsigned int flags,
struct oid_array *extra_have,
struct oid_array *shallow_points)
@@ -242,24 +242,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
struct ref **orig_list = list;
int len = 0;
enum get_remote_heads_state state = EXPECTING_FIRST_REF;
- struct packet_reader reader;
const char *arg;
- packet_reader_init(&reader, in, src_buf, src_len,
- PACKET_READ_CHOMP_NEWLINE |
- PACKET_READ_GENTLE_ON_EOF);
-
- discover_version(&reader);
-
*list = NULL;
while (state != EXPECTING_DONE) {
- switch (packet_reader_read(&reader)) {
+ switch (packet_reader_read(reader)) {
case PACKET_READ_EOF:
die_initial_contact(1);
case PACKET_READ_NORMAL:
- len = reader.pktlen;
- if (len > 4 && skip_prefix(reader.line, "ERR ", &arg))
+ len = reader->pktlen;
+ if (len > 4 && skip_prefix(reader->line, "ERR ", &arg))
die("remote error: %s", arg);
break;
case PACKET_READ_FLUSH:
@@ -271,22 +264,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
switch (state) {
case EXPECTING_FIRST_REF:
- process_capabilities(reader.line, &len);
- if (process_dummy_ref(reader.line)) {
+ process_capabilities(reader->line, &len);
+ if (process_dummy_ref(reader->line)) {
state = EXPECTING_SHALLOW;
break;
}
state = EXPECTING_REF;
/* fallthrough */
case EXPECTING_REF:
- if (process_ref(reader.line, len, &list, flags, extra_have))
+ if (process_ref(reader->line, len, &list, flags, extra_have))
break;
state = EXPECTING_SHALLOW;
/* fallthrough */
case EXPECTING_SHALLOW:
- if (process_shallow(reader.line, len, shallow_points))
+ if (process_shallow(reader->line, len, shallow_points))
break;
- die("protocol error: unexpected '%s'", reader.line);
+ die("protocol error: unexpected '%s'", reader->line);
case EXPECTING_DONE:
break;
}
diff --git a/connect.h b/connect.h
index 01f14cdf3..cdb8979dc 100644
--- a/connect.h
+++ b/connect.h
@@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);
extern const char *server_feature_value(const char *feature, int *len_ret);
extern int url_is_local_not_ssh(const char *url);
+struct packet_reader;
+extern enum protocol_version discover_version(struct packet_reader *reader);
+
#endif
diff --git a/remote-curl.c b/remote-curl.c
index 0053b0954..9f6d07683 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -1,6 +1,7 @@
#include "cache.h"
#include "config.h"
#include "remote.h"
+#include "connect.h"
#include "strbuf.h"
#include "walker.h"
#include "http.h"
@@ -13,6 +14,7 @@
#include "credential.h"
#include "sha1-array.h"
#include "send-pack.h"
+#include "protocol.h"
static struct remote *remote;
/* always ends with a trailing slash */
@@ -176,8 +178,22 @@ static struct discovery *last_discovery;
static struct ref *parse_git_refs(struct discovery *heads, int for_push)
{
struct ref *list = NULL;
- get_remote_heads(-1, heads->buf, heads->len, &list,
- for_push ? REF_NORMAL : 0, NULL, &heads->shallow);
+ struct packet_reader reader;
+
+ packet_reader_init(&reader, -1, heads->buf, heads->len,
+ PACKET_READ_CHOMP_NEWLINE |
+ PACKET_READ_GENTLE_ON_EOF);
+
+ switch (discover_version(&reader)) {
+ case protocol_v1:
+ case protocol_v0:
+ get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
+ NULL, &heads->shallow);
+ break;
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
+
return list;
}
diff --git a/remote.h b/remote.h
index 1f6611be2..2016461df 100644
--- a/remote.h
+++ b/remote.h
@@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);
void free_refs(struct ref *ref);
struct oid_array;
-extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
+struct packet_reader;
+extern struct ref **get_remote_heads(struct packet_reader *reader,
struct ref **list, unsigned int flags,
struct oid_array *extra_have,
- struct oid_array *shallow);
+ struct oid_array *shallow_points);
int resolve_remote_symref(struct ref *ref, struct ref *list);
int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);
diff --git a/transport.c b/transport.c
index 8e8779096..63c3dbab9 100644
--- a/transport.c
+++ b/transport.c
@@ -18,6 +18,7 @@
#include "sha1-array.h"
#include "sigchain.h"
#include "transport-internal.h"
+#include "protocol.h"
static void set_upstreams(struct transport *transport, struct ref *refs,
int pretend)
@@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)
static struct ref *get_refs_via_connect(struct transport *transport, int for_push)
{
struct git_transport_data *data = transport->data;
- struct ref *refs;
+ struct ref *refs = NULL;
+ struct packet_reader reader;
connect_setup(transport, for_push);
- get_remote_heads(data->fd[0], NULL, 0, &refs,
- for_push ? REF_NORMAL : 0,
- &data->extra_have,
- &data->shallow);
+
+ packet_reader_init(&reader, data->fd[0], NULL, 0,
+ PACKET_READ_CHOMP_NEWLINE |
+ PACKET_READ_GENTLE_ON_EOF);
+
+ switch (discover_version(&reader)) {
+ case protocol_v1:
+ case protocol_v0:
+ get_remote_heads(&reader, &refs,
+ for_push ? REF_NORMAL : 0,
+ &data->extra_have,
+ &data->shallow);
+ break;
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
data->got_remote_heads = 1;
return refs;
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads
2018-01-25 23:58 ` [PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads Brandon Williams
@ 2018-01-31 14:40 ` Derrick Stolee
2018-02-01 17:57 ` Brandon Williams
0 siblings, 1 reply; 362+ messages in thread
From: Derrick Stolee @ 2018-01-31 14:40 UTC (permalink / raw)
To: Brandon Williams, git; +Cc: sbeller, gitster, peff, philipoakley, jrnieder
On 1/25/2018 6:58 PM, Brandon Williams wrote:
> In order to prepare for the addition of protocol_v2 push the protocol
> version discovery outside of 'get_remote_heads()'. This will allow for
> keeping the logic for processing the reference advertisement for
> protocol_v1 and protocol_v0 separate from the logic for protocol_v2.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
> builtin/fetch-pack.c | 16 +++++++++++++++-
> builtin/send-pack.c | 17 +++++++++++++++--
> connect.c | 27 ++++++++++-----------------
> connect.h | 3 +++
> remote-curl.c | 20 ++++++++++++++++++--
> remote.h | 5 +++--
> transport.c | 24 +++++++++++++++++++-----
> 7 files changed, 83 insertions(+), 29 deletions(-)
>
> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
> index 366b9d13f..85d4faf76 100644
> --- a/builtin/fetch-pack.c
> +++ b/builtin/fetch-pack.c
> @@ -4,6 +4,7 @@
> #include "remote.h"
> #include "connect.h"
> #include "sha1-array.h"
> +#include "protocol.h"
>
> static const char fetch_pack_usage[] =
> "git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] "
> @@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
> struct fetch_pack_args args;
> struct oid_array shallow = OID_ARRAY_INIT;
> struct string_list deepen_not = STRING_LIST_INIT_DUP;
> + struct packet_reader reader;
>
> packet_trace_identity("fetch-pack");
>
> @@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
> if (!conn)
> return args.diag_url ? 0 : 1;
> }
> - get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
> +
> + packet_reader_init(&reader, fd[0], NULL, 0,
> + PACKET_READ_CHOMP_NEWLINE |
> + PACKET_READ_GENTLE_ON_EOF);
> +
> + switch (discover_version(&reader)) {
> + case protocol_v1:
> + case protocol_v0:
> + get_remote_heads(&reader, &ref, 0, NULL, &shallow);
> + break;
> + case protocol_unknown_version:
> + BUG("unknown protocol version");
Is this really a BUG in the client, or a bug/incompatibility in the server?
Perhaps I'm misunderstanding, but it looks like discover_version() will
die() on an unknown version (the die() is in
protocol.c:determine_protocol_version_client()). So maybe that's why
this is a BUG()?
If there is something to change here, this BUG() appears three more times.
> + }
>
> ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,
> &shallow, pack_lockfile_ptr);
> diff --git a/builtin/send-pack.c b/builtin/send-pack.c
> index fc4f0bb5f..83cb125a6 100644
> --- a/builtin/send-pack.c
> +++ b/builtin/send-pack.c
> @@ -14,6 +14,7 @@
> #include "sha1-array.h"
> #include "gpg-interface.h"
> #include "gettext.h"
> +#include "protocol.h"
>
> static const char * const send_pack_usage[] = {
> N_("git send-pack [--all | --mirror] [--dry-run] [--force] "
> @@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
> int progress = -1;
> int from_stdin = 0;
> struct push_cas_option cas = {0};
> + struct packet_reader reader;
>
> struct option options[] = {
> OPT__VERBOSITY(&verbose),
> @@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
> args.verbose ? CONNECT_VERBOSE : 0);
> }
>
> - get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,
> - &extra_have, &shallow);
> + packet_reader_init(&reader, fd[0], NULL, 0,
> + PACKET_READ_CHOMP_NEWLINE |
> + PACKET_READ_GENTLE_ON_EOF);
> +
> + switch (discover_version(&reader)) {
> + case protocol_v1:
> + case protocol_v0:
> + get_remote_heads(&reader, &remote_refs, REF_NORMAL,
> + &extra_have, &shallow);
> + break;
> + case protocol_unknown_version:
> + BUG("unknown protocol version");
> + }
>
> transport_verify_remote_names(nr_refspecs, refspecs);
>
> diff --git a/connect.c b/connect.c
> index 00e90075c..db3c9d24c 100644
> --- a/connect.c
> +++ b/connect.c
> @@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)
> "and the repository exists."));
> }
>
> -static enum protocol_version discover_version(struct packet_reader *reader)
> +enum protocol_version discover_version(struct packet_reader *reader)
> {
> enum protocol_version version = protocol_unknown_version;
>
> @@ -234,7 +234,7 @@ enum get_remote_heads_state {
> /*
> * Read all the refs from the other end
> */
> -struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> +struct ref **get_remote_heads(struct packet_reader *reader,
> struct ref **list, unsigned int flags,
> struct oid_array *extra_have,
> struct oid_array *shallow_points)
> @@ -242,24 +242,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> struct ref **orig_list = list;
> int len = 0;
> enum get_remote_heads_state state = EXPECTING_FIRST_REF;
> - struct packet_reader reader;
> const char *arg;
>
> - packet_reader_init(&reader, in, src_buf, src_len,
> - PACKET_READ_CHOMP_NEWLINE |
> - PACKET_READ_GENTLE_ON_EOF);
> -
> - discover_version(&reader);
> -
> *list = NULL;
>
> while (state != EXPECTING_DONE) {
> - switch (packet_reader_read(&reader)) {
> + switch (packet_reader_read(reader)) {
> case PACKET_READ_EOF:
> die_initial_contact(1);
> case PACKET_READ_NORMAL:
> - len = reader.pktlen;
> - if (len > 4 && skip_prefix(reader.line, "ERR ", &arg))
> + len = reader->pktlen;
> + if (len > 4 && skip_prefix(reader->line, "ERR ", &arg))
> die("remote error: %s", arg);
> break;
> case PACKET_READ_FLUSH:
> @@ -271,22 +264,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
>
> switch (state) {
> case EXPECTING_FIRST_REF:
> - process_capabilities(reader.line, &len);
> - if (process_dummy_ref(reader.line)) {
> + process_capabilities(reader->line, &len);
> + if (process_dummy_ref(reader->line)) {
> state = EXPECTING_SHALLOW;
> break;
> }
> state = EXPECTING_REF;
> /* fallthrough */
> case EXPECTING_REF:
> - if (process_ref(reader.line, len, &list, flags, extra_have))
> + if (process_ref(reader->line, len, &list, flags, extra_have))
> break;
> state = EXPECTING_SHALLOW;
> /* fallthrough */
> case EXPECTING_SHALLOW:
> - if (process_shallow(reader.line, len, shallow_points))
> + if (process_shallow(reader->line, len, shallow_points))
> break;
> - die("protocol error: unexpected '%s'", reader.line);
> + die("protocol error: unexpected '%s'", reader->line);
> case EXPECTING_DONE:
> break;
> }
> diff --git a/connect.h b/connect.h
> index 01f14cdf3..cdb8979dc 100644
> --- a/connect.h
> +++ b/connect.h
> @@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);
> extern const char *server_feature_value(const char *feature, int *len_ret);
> extern int url_is_local_not_ssh(const char *url);
>
> +struct packet_reader;
> +extern enum protocol_version discover_version(struct packet_reader *reader);
> +
> #endif
> diff --git a/remote-curl.c b/remote-curl.c
> index 0053b0954..9f6d07683 100644
> --- a/remote-curl.c
> +++ b/remote-curl.c
> @@ -1,6 +1,7 @@
> #include "cache.h"
> #include "config.h"
> #include "remote.h"
> +#include "connect.h"
> #include "strbuf.h"
> #include "walker.h"
> #include "http.h"
> @@ -13,6 +14,7 @@
> #include "credential.h"
> #include "sha1-array.h"
> #include "send-pack.h"
> +#include "protocol.h"
>
> static struct remote *remote;
> /* always ends with a trailing slash */
> @@ -176,8 +178,22 @@ static struct discovery *last_discovery;
> static struct ref *parse_git_refs(struct discovery *heads, int for_push)
> {
> struct ref *list = NULL;
> - get_remote_heads(-1, heads->buf, heads->len, &list,
> - for_push ? REF_NORMAL : 0, NULL, &heads->shallow);
> + struct packet_reader reader;
> +
> + packet_reader_init(&reader, -1, heads->buf, heads->len,
> + PACKET_READ_CHOMP_NEWLINE |
> + PACKET_READ_GENTLE_ON_EOF);
> +
> + switch (discover_version(&reader)) {
> + case protocol_v1:
> + case protocol_v0:
> + get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
> + NULL, &heads->shallow);
> + break;
> + case protocol_unknown_version:
> + BUG("unknown protocol version");
> + }
> +
> return list;
> }
>
> diff --git a/remote.h b/remote.h
> index 1f6611be2..2016461df 100644
> --- a/remote.h
> +++ b/remote.h
> @@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);
> void free_refs(struct ref *ref);
>
> struct oid_array;
> -extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> +struct packet_reader;
> +extern struct ref **get_remote_heads(struct packet_reader *reader,
> struct ref **list, unsigned int flags,
> struct oid_array *extra_have,
> - struct oid_array *shallow);
> + struct oid_array *shallow_points);
>
> int resolve_remote_symref(struct ref *ref, struct ref *list);
> int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);
> diff --git a/transport.c b/transport.c
> index 8e8779096..63c3dbab9 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -18,6 +18,7 @@
> #include "sha1-array.h"
> #include "sigchain.h"
> #include "transport-internal.h"
> +#include "protocol.h"
>
> static void set_upstreams(struct transport *transport, struct ref *refs,
> int pretend)
> @@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)
> static struct ref *get_refs_via_connect(struct transport *transport, int for_push)
> {
> struct git_transport_data *data = transport->data;
> - struct ref *refs;
> + struct ref *refs = NULL;
> + struct packet_reader reader;
>
> connect_setup(transport, for_push);
> - get_remote_heads(data->fd[0], NULL, 0, &refs,
> - for_push ? REF_NORMAL : 0,
> - &data->extra_have,
> - &data->shallow);
> +
> + packet_reader_init(&reader, data->fd[0], NULL, 0,
> + PACKET_READ_CHOMP_NEWLINE |
> + PACKET_READ_GENTLE_ON_EOF);
> +
> + switch (discover_version(&reader)) {
> + case protocol_v1:
> + case protocol_v0:
> + get_remote_heads(&reader, &refs,
> + for_push ? REF_NORMAL : 0,
> + &data->extra_have,
> + &data->shallow);
> + break;
> + case protocol_unknown_version:
> + BUG("unknown protocol version");
> + }
> data->got_remote_heads = 1;
>
> return refs;
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads
2018-01-31 14:40 ` Derrick Stolee
@ 2018-02-01 17:57 ` Brandon Williams
0 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-02-01 17:57 UTC (permalink / raw)
To: Derrick Stolee; +Cc: git, sbeller, gitster, peff, philipoakley, jrnieder
On 01/31, Derrick Stolee wrote:
> On 1/25/2018 6:58 PM, Brandon Williams wrote:
> > In order to prepare for the addition of protocol_v2 push the protocol
> > version discovery outside of 'get_remote_heads()'. This will allow for
> > keeping the logic for processing the reference advertisement for
> > protocol_v1 and protocol_v0 separate from the logic for protocol_v2.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> > builtin/fetch-pack.c | 16 +++++++++++++++-
> > builtin/send-pack.c | 17 +++++++++++++++--
> > connect.c | 27 ++++++++++-----------------
> > connect.h | 3 +++
> > remote-curl.c | 20 ++++++++++++++++++--
> > remote.h | 5 +++--
> > transport.c | 24 +++++++++++++++++++-----
> > 7 files changed, 83 insertions(+), 29 deletions(-)
> >
> > diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
> > index 366b9d13f..85d4faf76 100644
> > --- a/builtin/fetch-pack.c
> > +++ b/builtin/fetch-pack.c
> > @@ -4,6 +4,7 @@
> > #include "remote.h"
> > #include "connect.h"
> > #include "sha1-array.h"
> > +#include "protocol.h"
> > static const char fetch_pack_usage[] =
> > "git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] "
> > @@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
> > struct fetch_pack_args args;
> > struct oid_array shallow = OID_ARRAY_INIT;
> > struct string_list deepen_not = STRING_LIST_INIT_DUP;
> > + struct packet_reader reader;
> > packet_trace_identity("fetch-pack");
> > @@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
> > if (!conn)
> > return args.diag_url ? 0 : 1;
> > }
> > - get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
> > +
> > + packet_reader_init(&reader, fd[0], NULL, 0,
> > + PACKET_READ_CHOMP_NEWLINE |
> > + PACKET_READ_GENTLE_ON_EOF);
> > +
> > + switch (discover_version(&reader)) {
> > + case protocol_v1:
> > + case protocol_v0:
> > + get_remote_heads(&reader, &ref, 0, NULL, &shallow);
> > + break;
> > + case protocol_unknown_version:
> > + BUG("unknown protocol version");
>
> Is this really a BUG in the client, or a bug/incompatibility in the server?
>
> Perhaps I'm misunderstanding, but it looks like discover_version() will
> die() on an unknown version (the die() is in
> protocol.c:determine_protocol_version_client()). So maybe that's why this is
> a BUG()?
>
> If there is something to change here, this BUG() appears three more times.
Yes, I have it labeled as a BUG because discover_version can't return an
unknown protocol version. If the server actually returns an unknown
protocol version then it should be handled in
protocol.c:determine_protocol_version_client() as you mentioned.
>
> > + }
> > ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,
> > &shallow, pack_lockfile_ptr);
> > diff --git a/builtin/send-pack.c b/builtin/send-pack.c
> > index fc4f0bb5f..83cb125a6 100644
> > --- a/builtin/send-pack.c
> > +++ b/builtin/send-pack.c
> > @@ -14,6 +14,7 @@
> > #include "sha1-array.h"
> > #include "gpg-interface.h"
> > #include "gettext.h"
> > +#include "protocol.h"
> > static const char * const send_pack_usage[] = {
> > N_("git send-pack [--all | --mirror] [--dry-run] [--force] "
> > @@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
> > int progress = -1;
> > int from_stdin = 0;
> > struct push_cas_option cas = {0};
> > + struct packet_reader reader;
> > struct option options[] = {
> > OPT__VERBOSITY(&verbose),
> > @@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
> > args.verbose ? CONNECT_VERBOSE : 0);
> > }
> > - get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,
> > - &extra_have, &shallow);
> > + packet_reader_init(&reader, fd[0], NULL, 0,
> > + PACKET_READ_CHOMP_NEWLINE |
> > + PACKET_READ_GENTLE_ON_EOF);
> > +
> > + switch (discover_version(&reader)) {
> > + case protocol_v1:
> > + case protocol_v0:
> > + get_remote_heads(&reader, &remote_refs, REF_NORMAL,
> > + &extra_have, &shallow);
> > + break;
> > + case protocol_unknown_version:
> > + BUG("unknown protocol version");
> > + }
> > transport_verify_remote_names(nr_refspecs, refspecs);
> > diff --git a/connect.c b/connect.c
> > index 00e90075c..db3c9d24c 100644
> > --- a/connect.c
> > +++ b/connect.c
> > @@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)
> > "and the repository exists."));
> > }
> > -static enum protocol_version discover_version(struct packet_reader *reader)
> > +enum protocol_version discover_version(struct packet_reader *reader)
> > {
> > enum protocol_version version = protocol_unknown_version;
> > @@ -234,7 +234,7 @@ enum get_remote_heads_state {
> > /*
> > * Read all the refs from the other end
> > */
> > -struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> > +struct ref **get_remote_heads(struct packet_reader *reader,
> > struct ref **list, unsigned int flags,
> > struct oid_array *extra_have,
> > struct oid_array *shallow_points)
> > @@ -242,24 +242,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> > struct ref **orig_list = list;
> > int len = 0;
> > enum get_remote_heads_state state = EXPECTING_FIRST_REF;
> > - struct packet_reader reader;
> > const char *arg;
> > - packet_reader_init(&reader, in, src_buf, src_len,
> > - PACKET_READ_CHOMP_NEWLINE |
> > - PACKET_READ_GENTLE_ON_EOF);
> > -
> > - discover_version(&reader);
> > -
> > *list = NULL;
> > while (state != EXPECTING_DONE) {
> > - switch (packet_reader_read(&reader)) {
> > + switch (packet_reader_read(reader)) {
> > case PACKET_READ_EOF:
> > die_initial_contact(1);
> > case PACKET_READ_NORMAL:
> > - len = reader.pktlen;
> > - if (len > 4 && skip_prefix(reader.line, "ERR ", &arg))
> > + len = reader->pktlen;
> > + if (len > 4 && skip_prefix(reader->line, "ERR ", &arg))
> > die("remote error: %s", arg);
> > break;
> > case PACKET_READ_FLUSH:
> > @@ -271,22 +264,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> > switch (state) {
> > case EXPECTING_FIRST_REF:
> > - process_capabilities(reader.line, &len);
> > - if (process_dummy_ref(reader.line)) {
> > + process_capabilities(reader->line, &len);
> > + if (process_dummy_ref(reader->line)) {
> > state = EXPECTING_SHALLOW;
> > break;
> > }
> > state = EXPECTING_REF;
> > /* fallthrough */
> > case EXPECTING_REF:
> > - if (process_ref(reader.line, len, &list, flags, extra_have))
> > + if (process_ref(reader->line, len, &list, flags, extra_have))
> > break;
> > state = EXPECTING_SHALLOW;
> > /* fallthrough */
> > case EXPECTING_SHALLOW:
> > - if (process_shallow(reader.line, len, shallow_points))
> > + if (process_shallow(reader->line, len, shallow_points))
> > break;
> > - die("protocol error: unexpected '%s'", reader.line);
> > + die("protocol error: unexpected '%s'", reader->line);
> > case EXPECTING_DONE:
> > break;
> > }
> > diff --git a/connect.h b/connect.h
> > index 01f14cdf3..cdb8979dc 100644
> > --- a/connect.h
> > +++ b/connect.h
> > @@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);
> > extern const char *server_feature_value(const char *feature, int *len_ret);
> > extern int url_is_local_not_ssh(const char *url);
> > +struct packet_reader;
> > +extern enum protocol_version discover_version(struct packet_reader *reader);
> > +
> > #endif
> > diff --git a/remote-curl.c b/remote-curl.c
> > index 0053b0954..9f6d07683 100644
> > --- a/remote-curl.c
> > +++ b/remote-curl.c
> > @@ -1,6 +1,7 @@
> > #include "cache.h"
> > #include "config.h"
> > #include "remote.h"
> > +#include "connect.h"
> > #include "strbuf.h"
> > #include "walker.h"
> > #include "http.h"
> > @@ -13,6 +14,7 @@
> > #include "credential.h"
> > #include "sha1-array.h"
> > #include "send-pack.h"
> > +#include "protocol.h"
> > static struct remote *remote;
> > /* always ends with a trailing slash */
> > @@ -176,8 +178,22 @@ static struct discovery *last_discovery;
> > static struct ref *parse_git_refs(struct discovery *heads, int for_push)
> > {
> > struct ref *list = NULL;
> > - get_remote_heads(-1, heads->buf, heads->len, &list,
> > - for_push ? REF_NORMAL : 0, NULL, &heads->shallow);
> > + struct packet_reader reader;
> > +
> > + packet_reader_init(&reader, -1, heads->buf, heads->len,
> > + PACKET_READ_CHOMP_NEWLINE |
> > + PACKET_READ_GENTLE_ON_EOF);
> > +
> > + switch (discover_version(&reader)) {
> > + case protocol_v1:
> > + case protocol_v0:
> > + get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
> > + NULL, &heads->shallow);
> > + break;
> > + case protocol_unknown_version:
> > + BUG("unknown protocol version");
> > + }
> > +
> > return list;
> > }
> > diff --git a/remote.h b/remote.h
> > index 1f6611be2..2016461df 100644
> > --- a/remote.h
> > +++ b/remote.h
> > @@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);
> > void free_refs(struct ref *ref);
> > struct oid_array;
> > -extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> > +struct packet_reader;
> > +extern struct ref **get_remote_heads(struct packet_reader *reader,
> > struct ref **list, unsigned int flags,
> > struct oid_array *extra_have,
> > - struct oid_array *shallow);
> > + struct oid_array *shallow_points);
> > int resolve_remote_symref(struct ref *ref, struct ref *list);
> > int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);
> > diff --git a/transport.c b/transport.c
> > index 8e8779096..63c3dbab9 100644
> > --- a/transport.c
> > +++ b/transport.c
> > @@ -18,6 +18,7 @@
> > #include "sha1-array.h"
> > #include "sigchain.h"
> > #include "transport-internal.h"
> > +#include "protocol.h"
> > static void set_upstreams(struct transport *transport, struct ref *refs,
> > int pretend)
> > @@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)
> > static struct ref *get_refs_via_connect(struct transport *transport, int for_push)
> > {
> > struct git_transport_data *data = transport->data;
> > - struct ref *refs;
> > + struct ref *refs = NULL;
> > + struct packet_reader reader;
> > connect_setup(transport, for_push);
> > - get_remote_heads(data->fd[0], NULL, 0, &refs,
> > - for_push ? REF_NORMAL : 0,
> > - &data->extra_have,
> > - &data->shallow);
> > +
> > + packet_reader_init(&reader, data->fd[0], NULL, 0,
> > + PACKET_READ_CHOMP_NEWLINE |
> > + PACKET_READ_GENTLE_ON_EOF);
> > +
> > + switch (discover_version(&reader)) {
> > + case protocol_v1:
> > + case protocol_v0:
> > + get_remote_heads(&reader, &refs,
> > + for_push ? REF_NORMAL : 0,
> > + &data->extra_have,
> > + &data->shallow);
> > + break;
> > + case protocol_unknown_version:
> > + BUG("unknown protocol version");
> > + }
> > data->got_remote_heads = 1;
> > return refs;
>
--
Brandon Williams
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v2 09/27] transport: store protocol version
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (7 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-31 14:45 ` Derrick Stolee
2018-01-25 23:58 ` [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2 Brandon Williams
` (20 subsequent siblings)
29 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Once protocol_v2 is introduced requesting a fetch or a push will need to
be handled differently depending on the protocol version. Store the
protocol version the server is speaking in 'struct git_transport_data'
and use it to determine what to do in the case of a fetch or a push.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
transport.c | 35 ++++++++++++++++++++++++++---------
1 file changed, 26 insertions(+), 9 deletions(-)
diff --git a/transport.c b/transport.c
index 63c3dbab9..2378dcb38 100644
--- a/transport.c
+++ b/transport.c
@@ -118,6 +118,7 @@ struct git_transport_data {
struct child_process *conn;
int fd[2];
unsigned got_remote_heads : 1;
+ enum protocol_version version;
struct oid_array extra_have;
struct oid_array shallow;
};
@@ -200,7 +201,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
PACKET_READ_CHOMP_NEWLINE |
PACKET_READ_GENTLE_ON_EOF);
- switch (discover_version(&reader)) {
+ data->version = discover_version(&reader);
+ switch (data->version) {
case protocol_v1:
case protocol_v0:
get_remote_heads(&reader, &refs,
@@ -221,7 +223,7 @@ static int fetch_refs_via_pack(struct transport *transport,
{
int ret = 0;
struct git_transport_data *data = transport->data;
- struct ref *refs;
+ struct ref *refs = NULL;
char *dest = xstrdup(transport->url);
struct fetch_pack_args args;
struct ref *refs_tmp = NULL;
@@ -247,10 +249,18 @@ static int fetch_refs_via_pack(struct transport *transport,
if (!data->got_remote_heads)
refs_tmp = get_refs_via_connect(transport, 0);
- refs = fetch_pack(&args, data->fd, data->conn,
- refs_tmp ? refs_tmp : transport->remote_refs,
- dest, to_fetch, nr_heads, &data->shallow,
- &transport->pack_lockfile);
+ switch (data->version) {
+ case protocol_v1:
+ case protocol_v0:
+ refs = fetch_pack(&args, data->fd, data->conn,
+ refs_tmp ? refs_tmp : transport->remote_refs,
+ dest, to_fetch, nr_heads, &data->shallow,
+ &transport->pack_lockfile);
+ break;
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
+
close(data->fd[0]);
close(data->fd[1]);
if (finish_connect(data->conn))
@@ -549,7 +559,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
{
struct git_transport_data *data = transport->data;
struct send_pack_args args;
- int ret;
+ int ret = 0;
if (!data->got_remote_heads)
get_refs_via_connect(transport, 1);
@@ -574,8 +584,15 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
else
args.push_cert = SEND_PACK_PUSH_CERT_NEVER;
- ret = send_pack(&args, data->fd, data->conn, remote_refs,
- &data->extra_have);
+ switch (data->version) {
+ case protocol_v1:
+ case protocol_v0:
+ ret = send_pack(&args, data->fd, data->conn, remote_refs,
+ &data->extra_have);
+ break;
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
close(data->fd[1]);
close(data->fd[0]);
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v2 09/27] transport: store protocol version
2018-01-25 23:58 ` [PATCH v2 09/27] transport: store protocol version Brandon Williams
@ 2018-01-31 14:45 ` Derrick Stolee
0 siblings, 0 replies; 362+ messages in thread
From: Derrick Stolee @ 2018-01-31 14:45 UTC (permalink / raw)
To: Brandon Williams, git; +Cc: sbeller, gitster, peff, philipoakley, jrnieder
On 1/25/2018 6:58 PM, Brandon Williams wrote:
> + switch (data->version) {
> + case protocol_v1:
> + case protocol_v0:
> + refs = fetch_pack(&args, data->fd, data->conn,
> + refs_tmp ? refs_tmp : transport->remote_refs,
> + dest, to_fetch, nr_heads, &data->shallow,
> + &transport->pack_lockfile);
> + break;
> + case protocol_unknown_version:
> + BUG("unknown protocol version");
> + }
After seeing this pattern a few times, I think it would be good to
convert it to a macro that calls a statement for protocol_v1/v0 (and
later calls a different one for protocol_v2). It would at minimum reduce
the code clones surrounding this handling of unknown_version, and we
could have one place that is clear this BUG() is due to an unexpected
response from discover_version().
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (8 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 09/27] transport: store protocol version Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-31 14:54 ` Derrick Stolee
2018-01-25 23:58 ` [PATCH v2 11/27] test-pkt-line: introduce a packet-line test helper Brandon Williams
` (19 subsequent siblings)
29 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Introduce protocol_v2, a new value for 'enum protocol_version'.
Subsequent patches will fill in the implementation of protocol_v2.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
builtin/fetch-pack.c | 3 +++
builtin/receive-pack.c | 6 ++++++
builtin/send-pack.c | 3 +++
builtin/upload-pack.c | 7 +++++++
connect.c | 3 +++
protocol.c | 2 ++
protocol.h | 1 +
remote-curl.c | 3 +++
transport.c | 9 +++++++++
9 files changed, 37 insertions(+)
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 85d4faf76..f492e8abd 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
PACKET_READ_GENTLE_ON_EOF);
switch (discover_version(&reader)) {
+ case protocol_v2:
+ die("support for protocol v2 not implemented yet");
+ break;
case protocol_v1:
case protocol_v0:
get_remote_heads(&reader, &ref, 0, NULL, &shallow);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b7ce7c7f5..3656e94fd 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
unpack_limit = receive_unpack_limit;
switch (determine_protocol_version_server()) {
+ case protocol_v2:
+ /*
+ * push support for protocol v2 has not been implemented yet,
+ * so ignore the request to use v2 and fallback to using v0.
+ */
+ break;
case protocol_v1:
/*
* v1 is just the original protocol with a version string,
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 83cb125a6..b5427f75e 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
PACKET_READ_GENTLE_ON_EOF);
switch (discover_version(&reader)) {
+ case protocol_v2:
+ die("support for protocol v2 not implemented yet");
+ break;
case protocol_v1:
case protocol_v0:
get_remote_heads(&reader, &remote_refs, REF_NORMAL,
diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c
index 2cb5cb35b..8d53e9794 100644
--- a/builtin/upload-pack.c
+++ b/builtin/upload-pack.c
@@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
die("'%s' does not appear to be a git repository", dir);
switch (determine_protocol_version_server()) {
+ case protocol_v2:
+ /*
+ * fetch support for protocol v2 has not been implemented yet,
+ * so ignore the request to use v2 and fallback to using v0.
+ */
+ upload_pack(&opts);
+ break;
case protocol_v1:
/*
* v1 is just the original protocol with a version string,
diff --git a/connect.c b/connect.c
index db3c9d24c..f2157a821 100644
--- a/connect.c
+++ b/connect.c
@@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)
/* Maybe process capabilities here, at least for v2 */
switch (version) {
+ case protocol_v2:
+ die("support for protocol v2 not implemented yet");
+ break;
case protocol_v1:
/* Read the peeked version line */
packet_reader_read(reader);
diff --git a/protocol.c b/protocol.c
index 43012b7eb..5e636785d 100644
--- a/protocol.c
+++ b/protocol.c
@@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)
return protocol_v0;
else if (!strcmp(value, "1"))
return protocol_v1;
+ else if (!strcmp(value, "2"))
+ return protocol_v2;
else
return protocol_unknown_version;
}
diff --git a/protocol.h b/protocol.h
index 1b2bc94a8..2ad35e433 100644
--- a/protocol.h
+++ b/protocol.h
@@ -5,6 +5,7 @@ enum protocol_version {
protocol_unknown_version = -1,
protocol_v0 = 0,
protocol_v1 = 1,
+ protocol_v2 = 2,
};
/*
diff --git a/remote-curl.c b/remote-curl.c
index 9f6d07683..dae8a4a48 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
PACKET_READ_GENTLE_ON_EOF);
switch (discover_version(&reader)) {
+ case protocol_v2:
+ die("support for protocol v2 not implemented yet");
+ break;
case protocol_v1:
case protocol_v0:
get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
diff --git a/transport.c b/transport.c
index 2378dcb38..83d9dd1df 100644
--- a/transport.c
+++ b/transport.c
@@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
data->version = discover_version(&reader);
switch (data->version) {
+ case protocol_v2:
+ die("support for protocol v2 not implemented yet");
+ break;
case protocol_v1:
case protocol_v0:
get_remote_heads(&reader, &refs,
@@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,
refs_tmp = get_refs_via_connect(transport, 0);
switch (data->version) {
+ case protocol_v2:
+ die("support for protocol v2 not implemented yet");
+ break;
case protocol_v1:
case protocol_v0:
refs = fetch_pack(&args, data->fd, data->conn,
@@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
args.push_cert = SEND_PACK_PUSH_CERT_NEVER;
switch (data->version) {
+ case protocol_v2:
+ die("support for protocol v2 not implemented yet");
+ break;
case protocol_v1:
case protocol_v0:
ret = send_pack(&args, data->fd, data->conn, remote_refs,
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2
2018-01-25 23:58 ` [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2 Brandon Williams
@ 2018-01-31 14:54 ` Derrick Stolee
2018-02-02 22:44 ` Brandon Williams
0 siblings, 1 reply; 362+ messages in thread
From: Derrick Stolee @ 2018-01-31 14:54 UTC (permalink / raw)
To: Brandon Williams, git; +Cc: sbeller, gitster, peff, philipoakley, jrnieder
On 1/25/2018 6:58 PM, Brandon Williams wrote:
> Introduce protocol_v2, a new value for 'enum protocol_version'.
> Subsequent patches will fill in the implementation of protocol_v2.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
> builtin/fetch-pack.c | 3 +++
> builtin/receive-pack.c | 6 ++++++
> builtin/send-pack.c | 3 +++
> builtin/upload-pack.c | 7 +++++++
> connect.c | 3 +++
> protocol.c | 2 ++
> protocol.h | 1 +
> remote-curl.c | 3 +++
> transport.c | 9 +++++++++
> 9 files changed, 37 insertions(+)
>
> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
> index 85d4faf76..f492e8abd 100644
> --- a/builtin/fetch-pack.c
> +++ b/builtin/fetch-pack.c
> @@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
> PACKET_READ_GENTLE_ON_EOF);
>
> switch (discover_version(&reader)) {
> + case protocol_v2:
> + die("support for protocol v2 not implemented yet");
> + break;
> case protocol_v1:
> case protocol_v0:
> get_remote_heads(&reader, &ref, 0, NULL, &shallow);
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index b7ce7c7f5..3656e94fd 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
> unpack_limit = receive_unpack_limit;
>
> switch (determine_protocol_version_server()) {
> + case protocol_v2:
> + /*
> + * push support for protocol v2 has not been implemented yet,
> + * so ignore the request to use v2 and fallback to using v0.
> + */
> + break;
> case protocol_v1:
> /*
> * v1 is just the original protocol with a version string,
> diff --git a/builtin/send-pack.c b/builtin/send-pack.c
> index 83cb125a6..b5427f75e 100644
> --- a/builtin/send-pack.c
> +++ b/builtin/send-pack.c
> @@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
> PACKET_READ_GENTLE_ON_EOF);
>
> switch (discover_version(&reader)) {
> + case protocol_v2:
> + die("support for protocol v2 not implemented yet");
> + break;
> case protocol_v1:
> case protocol_v0:
> get_remote_heads(&reader, &remote_refs, REF_NORMAL,
> diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c
> index 2cb5cb35b..8d53e9794 100644
> --- a/builtin/upload-pack.c
> +++ b/builtin/upload-pack.c
> @@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
> die("'%s' does not appear to be a git repository", dir);
>
> switch (determine_protocol_version_server()) {
> + case protocol_v2:
> + /*
> + * fetch support for protocol v2 has not been implemented yet,
> + * so ignore the request to use v2 and fallback to using v0.
> + */
> + upload_pack(&opts);
> + break;
> case protocol_v1:
> /*
> * v1 is just the original protocol with a version string,
> diff --git a/connect.c b/connect.c
> index db3c9d24c..f2157a821 100644
> --- a/connect.c
> +++ b/connect.c
> @@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)
>
> /* Maybe process capabilities here, at least for v2 */
> switch (version) {
> + case protocol_v2:
> + die("support for protocol v2 not implemented yet");
> + break;
> case protocol_v1:
> /* Read the peeked version line */
> packet_reader_read(reader);
> diff --git a/protocol.c b/protocol.c
> index 43012b7eb..5e636785d 100644
> --- a/protocol.c
> +++ b/protocol.c
> @@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)
> return protocol_v0;
> else if (!strcmp(value, "1"))
> return protocol_v1;
> + else if (!strcmp(value, "2"))
> + return protocol_v2;
> else
> return protocol_unknown_version;
> }
> diff --git a/protocol.h b/protocol.h
> index 1b2bc94a8..2ad35e433 100644
> --- a/protocol.h
> +++ b/protocol.h
> @@ -5,6 +5,7 @@ enum protocol_version {
> protocol_unknown_version = -1,
> protocol_v0 = 0,
> protocol_v1 = 1,
> + protocol_v2 = 2,
> };
>
> /*
> diff --git a/remote-curl.c b/remote-curl.c
> index 9f6d07683..dae8a4a48 100644
> --- a/remote-curl.c
> +++ b/remote-curl.c
> @@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
> PACKET_READ_GENTLE_ON_EOF);
>
> switch (discover_version(&reader)) {
> + case protocol_v2:
> + die("support for protocol v2 not implemented yet");
> + break;
> case protocol_v1:
> case protocol_v0:
> get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
> diff --git a/transport.c b/transport.c
> index 2378dcb38..83d9dd1df 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
>
> data->version = discover_version(&reader);
> switch (data->version) {
> + case protocol_v2:
> + die("support for protocol v2 not implemented yet");
> + break;
> case protocol_v1:
> case protocol_v0:
> get_remote_heads(&reader, &refs,
> @@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,
> refs_tmp = get_refs_via_connect(transport, 0);
>
> switch (data->version) {
> + case protocol_v2:
> + die("support for protocol v2 not implemented yet");
> + break;
> case protocol_v1:
> case protocol_v0:
> refs = fetch_pack(&args, data->fd, data->conn,
> @@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
> args.push_cert = SEND_PACK_PUSH_CERT_NEVER;
>
> switch (data->version) {
> + case protocol_v2:
> + die("support for protocol v2 not implemented yet");
> + break;
> case protocol_v1:
> case protocol_v0:
> ret = send_pack(&args, data->fd, data->conn, remote_refs,
With a macro approach to version selection, this change becomes simpler
in some ways and harder in others.
It is simpler in that we can have the macro from the previous commits
just fall back to version 0 behavior.
It is harder in that this commit would need one of two options:
1. A macro that performs an arbitrary statement when given v2, which
would be the die() for these actions not in v2.
2. A macro that clearly states v2 is not supported and calls die() on v2.
Here is my simple, untested attempt at a union of these options:
#define ON_PROTOCOL_VERSION(version,v0,v2) switch(version) {\
case protocol_v2:\
(v2);\
break;\
case protocol_v1:\
case protocol_v0:\
(v0);\
break;\
case protocol_unknown_version:\
BUG("unknown protocol version");\
}
#define ON_PROTOCOL_VERSION_V0_FALLBACK(version,v0) switch(version) {\
case protocol_v2:\
case protocol_v1:\
case protocol_v0:\
(v0);\
break;\
case protocol_unknown_version:\
BUG("unknown protocol version");\
}
#define ON_PROTOCOL_VERSION_V0_ONLY(version,v0) \
ON_PROTOCOL_VERSION(version,v0,\
BUG("support for protocol v2 not implemented yet"))
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2
2018-01-31 14:54 ` Derrick Stolee
@ 2018-02-02 22:44 ` Brandon Williams
2018-02-05 14:14 ` Derrick Stolee
0 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-02-02 22:44 UTC (permalink / raw)
To: Derrick Stolee; +Cc: git, sbeller, gitster, peff, philipoakley, jrnieder
On 01/31, Derrick Stolee wrote:
> On 1/25/2018 6:58 PM, Brandon Williams wrote:
> > Introduce protocol_v2, a new value for 'enum protocol_version'.
> > Subsequent patches will fill in the implementation of protocol_v2.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> > builtin/fetch-pack.c | 3 +++
> > builtin/receive-pack.c | 6 ++++++
> > builtin/send-pack.c | 3 +++
> > builtin/upload-pack.c | 7 +++++++
> > connect.c | 3 +++
> > protocol.c | 2 ++
> > protocol.h | 1 +
> > remote-curl.c | 3 +++
> > transport.c | 9 +++++++++
> > 9 files changed, 37 insertions(+)
> >
> > diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
> > index 85d4faf76..f492e8abd 100644
> > --- a/builtin/fetch-pack.c
> > +++ b/builtin/fetch-pack.c
> > @@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
> > PACKET_READ_GENTLE_ON_EOF);
> > switch (discover_version(&reader)) {
> > + case protocol_v2:
> > + die("support for protocol v2 not implemented yet");
> > + break;
> > case protocol_v1:
> > case protocol_v0:
> > get_remote_heads(&reader, &ref, 0, NULL, &shallow);
> > diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> > index b7ce7c7f5..3656e94fd 100644
> > --- a/builtin/receive-pack.c
> > +++ b/builtin/receive-pack.c
> > @@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
> > unpack_limit = receive_unpack_limit;
> > switch (determine_protocol_version_server()) {
> > + case protocol_v2:
> > + /*
> > + * push support for protocol v2 has not been implemented yet,
> > + * so ignore the request to use v2 and fallback to using v0.
> > + */
> > + break;
> > case protocol_v1:
> > /*
> > * v1 is just the original protocol with a version string,
> > diff --git a/builtin/send-pack.c b/builtin/send-pack.c
> > index 83cb125a6..b5427f75e 100644
> > --- a/builtin/send-pack.c
> > +++ b/builtin/send-pack.c
> > @@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
> > PACKET_READ_GENTLE_ON_EOF);
> > switch (discover_version(&reader)) {
> > + case protocol_v2:
> > + die("support for protocol v2 not implemented yet");
> > + break;
> > case protocol_v1:
> > case protocol_v0:
> > get_remote_heads(&reader, &remote_refs, REF_NORMAL,
> > diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c
> > index 2cb5cb35b..8d53e9794 100644
> > --- a/builtin/upload-pack.c
> > +++ b/builtin/upload-pack.c
> > @@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
> > die("'%s' does not appear to be a git repository", dir);
> > switch (determine_protocol_version_server()) {
> > + case protocol_v2:
> > + /*
> > + * fetch support for protocol v2 has not been implemented yet,
> > + * so ignore the request to use v2 and fallback to using v0.
> > + */
> > + upload_pack(&opts);
> > + break;
> > case protocol_v1:
> > /*
> > * v1 is just the original protocol with a version string,
> > diff --git a/connect.c b/connect.c
> > index db3c9d24c..f2157a821 100644
> > --- a/connect.c
> > +++ b/connect.c
> > @@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)
> > /* Maybe process capabilities here, at least for v2 */
> > switch (version) {
> > + case protocol_v2:
> > + die("support for protocol v2 not implemented yet");
> > + break;
> > case protocol_v1:
> > /* Read the peeked version line */
> > packet_reader_read(reader);
> > diff --git a/protocol.c b/protocol.c
> > index 43012b7eb..5e636785d 100644
> > --- a/protocol.c
> > +++ b/protocol.c
> > @@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)
> > return protocol_v0;
> > else if (!strcmp(value, "1"))
> > return protocol_v1;
> > + else if (!strcmp(value, "2"))
> > + return protocol_v2;
> > else
> > return protocol_unknown_version;
> > }
> > diff --git a/protocol.h b/protocol.h
> > index 1b2bc94a8..2ad35e433 100644
> > --- a/protocol.h
> > +++ b/protocol.h
> > @@ -5,6 +5,7 @@ enum protocol_version {
> > protocol_unknown_version = -1,
> > protocol_v0 = 0,
> > protocol_v1 = 1,
> > + protocol_v2 = 2,
> > };
> > /*
> > diff --git a/remote-curl.c b/remote-curl.c
> > index 9f6d07683..dae8a4a48 100644
> > --- a/remote-curl.c
> > +++ b/remote-curl.c
> > @@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
> > PACKET_READ_GENTLE_ON_EOF);
> > switch (discover_version(&reader)) {
> > + case protocol_v2:
> > + die("support for protocol v2 not implemented yet");
> > + break;
> > case protocol_v1:
> > case protocol_v0:
> > get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
> > diff --git a/transport.c b/transport.c
> > index 2378dcb38..83d9dd1df 100644
> > --- a/transport.c
> > +++ b/transport.c
> > @@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
> > data->version = discover_version(&reader);
> > switch (data->version) {
> > + case protocol_v2:
> > + die("support for protocol v2 not implemented yet");
> > + break;
> > case protocol_v1:
> > case protocol_v0:
> > get_remote_heads(&reader, &refs,
> > @@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,
> > refs_tmp = get_refs_via_connect(transport, 0);
> > switch (data->version) {
> > + case protocol_v2:
> > + die("support for protocol v2 not implemented yet");
> > + break;
> > case protocol_v1:
> > case protocol_v0:
> > refs = fetch_pack(&args, data->fd, data->conn,
> > @@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
> > args.push_cert = SEND_PACK_PUSH_CERT_NEVER;
> > switch (data->version) {
> > + case protocol_v2:
> > + die("support for protocol v2 not implemented yet");
> > + break;
> > case protocol_v1:
> > case protocol_v0:
> > ret = send_pack(&args, data->fd, data->conn, remote_refs,
>
> With a macro approach to version selection, this change becomes simpler in
> some ways and harder in others.
>
> It is simpler in that we can have the macro from the previous commits just
> fall back to version 0 behavior.
>
> It is harder in that this commit would need one of two options:
>
> 1. A macro that performs an arbitrary statement when given v2, which would
> be the die() for these actions not in v2.
> 2. A macro that clearly states v2 is not supported and calls die() on v2.
>
> Here is my simple, untested attempt at a union of these options:
>
> #define ON_PROTOCOL_VERSION(version,v0,v2) switch(version) {\
> case protocol_v2:\
> (v2);\
> break;\
> case protocol_v1:\
> case protocol_v0:\
> (v0);\
> break;\
> case protocol_unknown_version:\
> BUG("unknown protocol version");\
> }
> #define ON_PROTOCOL_VERSION_V0_FALLBACK(version,v0) switch(version) {\
> case protocol_v2:\
> case protocol_v1:\
> case protocol_v0:\
> (v0);\
> break;\
> case protocol_unknown_version:\
> BUG("unknown protocol version");\
> }
> #define ON_PROTOCOL_VERSION_V0_ONLY(version,v0) \
> ON_PROTOCOL_VERSION(version,v0,\
> BUG("support for protocol v2 not implemented yet"))
While I understand wanting to isolate the switch statement code, I think
that creating such a macro would make reading the code much more
difficult (and a pain to get right). Really I don't want to try my hand
at crafting such a macro :D
--
Brandon Williams
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2
2018-02-02 22:44 ` Brandon Williams
@ 2018-02-05 14:14 ` Derrick Stolee
0 siblings, 0 replies; 362+ messages in thread
From: Derrick Stolee @ 2018-02-05 14:14 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, gitster, peff, philipoakley, jrnieder
On 2/2/2018 5:44 PM, Brandon Williams wrote:
> On 01/31, Derrick Stolee wrote:
>> On 1/25/2018 6:58 PM, Brandon Williams wrote:
>>> Introduce protocol_v2, a new value for 'enum protocol_version'.
>>> Subsequent patches will fill in the implementation of protocol_v2.
>>>
>>> Signed-off-by: Brandon Williams <bmwill@google.com>
>>> ---
>>> builtin/fetch-pack.c | 3 +++
>>> builtin/receive-pack.c | 6 ++++++
>>> builtin/send-pack.c | 3 +++
>>> builtin/upload-pack.c | 7 +++++++
>>> connect.c | 3 +++
>>> protocol.c | 2 ++
>>> protocol.h | 1 +
>>> remote-curl.c | 3 +++
>>> transport.c | 9 +++++++++
>>> 9 files changed, 37 insertions(+)
>>>
>>> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
>>> index 85d4faf76..f492e8abd 100644
>>> --- a/builtin/fetch-pack.c
>>> +++ b/builtin/fetch-pack.c
>>> @@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
>>> PACKET_READ_GENTLE_ON_EOF);
>>> switch (discover_version(&reader)) {
>>> + case protocol_v2:
>>> + die("support for protocol v2 not implemented yet");
>>> + break;
>>> case protocol_v1:
>>> case protocol_v0:
>>> get_remote_heads(&reader, &ref, 0, NULL, &shallow);
>>> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
>>> index b7ce7c7f5..3656e94fd 100644
>>> --- a/builtin/receive-pack.c
>>> +++ b/builtin/receive-pack.c
>>> @@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
>>> unpack_limit = receive_unpack_limit;
>>> switch (determine_protocol_version_server()) {
>>> + case protocol_v2:
>>> + /*
>>> + * push support for protocol v2 has not been implemented yet,
>>> + * so ignore the request to use v2 and fallback to using v0.
>>> + */
>>> + break;
>>> case protocol_v1:
>>> /*
>>> * v1 is just the original protocol with a version string,
>>> diff --git a/builtin/send-pack.c b/builtin/send-pack.c
>>> index 83cb125a6..b5427f75e 100644
>>> --- a/builtin/send-pack.c
>>> +++ b/builtin/send-pack.c
>>> @@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
>>> PACKET_READ_GENTLE_ON_EOF);
>>> switch (discover_version(&reader)) {
>>> + case protocol_v2:
>>> + die("support for protocol v2 not implemented yet");
>>> + break;
>>> case protocol_v1:
>>> case protocol_v0:
>>> get_remote_heads(&reader, &remote_refs, REF_NORMAL,
>>> diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c
>>> index 2cb5cb35b..8d53e9794 100644
>>> --- a/builtin/upload-pack.c
>>> +++ b/builtin/upload-pack.c
>>> @@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
>>> die("'%s' does not appear to be a git repository", dir);
>>> switch (determine_protocol_version_server()) {
>>> + case protocol_v2:
>>> + /*
>>> + * fetch support for protocol v2 has not been implemented yet,
>>> + * so ignore the request to use v2 and fallback to using v0.
>>> + */
>>> + upload_pack(&opts);
>>> + break;
>>> case protocol_v1:
>>> /*
>>> * v1 is just the original protocol with a version string,
>>> diff --git a/connect.c b/connect.c
>>> index db3c9d24c..f2157a821 100644
>>> --- a/connect.c
>>> +++ b/connect.c
>>> @@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)
>>> /* Maybe process capabilities here, at least for v2 */
>>> switch (version) {
>>> + case protocol_v2:
>>> + die("support for protocol v2 not implemented yet");
>>> + break;
>>> case protocol_v1:
>>> /* Read the peeked version line */
>>> packet_reader_read(reader);
>>> diff --git a/protocol.c b/protocol.c
>>> index 43012b7eb..5e636785d 100644
>>> --- a/protocol.c
>>> +++ b/protocol.c
>>> @@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)
>>> return protocol_v0;
>>> else if (!strcmp(value, "1"))
>>> return protocol_v1;
>>> + else if (!strcmp(value, "2"))
>>> + return protocol_v2;
>>> else
>>> return protocol_unknown_version;
>>> }
>>> diff --git a/protocol.h b/protocol.h
>>> index 1b2bc94a8..2ad35e433 100644
>>> --- a/protocol.h
>>> +++ b/protocol.h
>>> @@ -5,6 +5,7 @@ enum protocol_version {
>>> protocol_unknown_version = -1,
>>> protocol_v0 = 0,
>>> protocol_v1 = 1,
>>> + protocol_v2 = 2,
>>> };
>>> /*
>>> diff --git a/remote-curl.c b/remote-curl.c
>>> index 9f6d07683..dae8a4a48 100644
>>> --- a/remote-curl.c
>>> +++ b/remote-curl.c
>>> @@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
>>> PACKET_READ_GENTLE_ON_EOF);
>>> switch (discover_version(&reader)) {
>>> + case protocol_v2:
>>> + die("support for protocol v2 not implemented yet");
>>> + break;
>>> case protocol_v1:
>>> case protocol_v0:
>>> get_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,
>>> diff --git a/transport.c b/transport.c
>>> index 2378dcb38..83d9dd1df 100644
>>> --- a/transport.c
>>> +++ b/transport.c
>>> @@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
>>> data->version = discover_version(&reader);
>>> switch (data->version) {
>>> + case protocol_v2:
>>> + die("support for protocol v2 not implemented yet");
>>> + break;
>>> case protocol_v1:
>>> case protocol_v0:
>>> get_remote_heads(&reader, &refs,
>>> @@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,
>>> refs_tmp = get_refs_via_connect(transport, 0);
>>> switch (data->version) {
>>> + case protocol_v2:
>>> + die("support for protocol v2 not implemented yet");
>>> + break;
>>> case protocol_v1:
>>> case protocol_v0:
>>> refs = fetch_pack(&args, data->fd, data->conn,
>>> @@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
>>> args.push_cert = SEND_PACK_PUSH_CERT_NEVER;
>>> switch (data->version) {
>>> + case protocol_v2:
>>> + die("support for protocol v2 not implemented yet");
>>> + break;
>>> case protocol_v1:
>>> case protocol_v0:
>>> ret = send_pack(&args, data->fd, data->conn, remote_refs,
>> With a macro approach to version selection, this change becomes simpler in
>> some ways and harder in others.
>>
>> It is simpler in that we can have the macro from the previous commits just
>> fall back to version 0 behavior.
>>
>> It is harder in that this commit would need one of two options:
>>
>> 1. A macro that performs an arbitrary statement when given v2, which would
>> be the die() for these actions not in v2.
>> 2. A macro that clearly states v2 is not supported and calls die() on v2.
>>
>> Here is my simple, untested attempt at a union of these options:
>>
>> #define ON_PROTOCOL_VERSION(version,v0,v2) switch(version) {\
>> case protocol_v2:\
>> (v2);\
>> break;\
>> case protocol_v1:\
>> case protocol_v0:\
>> (v0);\
>> break;\
>> case protocol_unknown_version:\
>> BUG("unknown protocol version");\
>> }
>> #define ON_PROTOCOL_VERSION_V0_FALLBACK(version,v0) switch(version) {\
>> case protocol_v2:\
>> case protocol_v1:\
>> case protocol_v0:\
>> (v0);\
>> break;\
>> case protocol_unknown_version:\
>> BUG("unknown protocol version");\
>> }
>> #define ON_PROTOCOL_VERSION_V0_ONLY(version,v0) \
>> ON_PROTOCOL_VERSION(version,v0,\
>> BUG("support for protocol v2 not implemented yet"))
>
> While I understand wanting to isolate the switch statement code, I think
> that creating such a macro would make reading the code much more
> difficult (and a pain to get right). Really I don't want to try my hand
> at crafting such a macro :D
>
Sounds good. You're right that the macro approach is more likely to be
used incorrectly.
-Stolee
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v2 11/27] test-pkt-line: introduce a packet-line test helper
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (9 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2 Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 12/27] serve: introduce git-serve Brandon Williams
` (18 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Introduce a packet-line test helper which can either pack or unpack an
input stream into packet-lines and writes out the result to stdout.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
Makefile | 1 +
t/helper/test-pkt-line.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 63 insertions(+)
create mode 100644 t/helper/test-pkt-line.c
diff --git a/Makefile b/Makefile
index b7ccc05fa..3b849c060 100644
--- a/Makefile
+++ b/Makefile
@@ -669,6 +669,7 @@ TEST_PROGRAMS_NEED_X += test-mktemp
TEST_PROGRAMS_NEED_X += test-online-cpus
TEST_PROGRAMS_NEED_X += test-parse-options
TEST_PROGRAMS_NEED_X += test-path-utils
+TEST_PROGRAMS_NEED_X += test-pkt-line
TEST_PROGRAMS_NEED_X += test-prio-queue
TEST_PROGRAMS_NEED_X += test-read-cache
TEST_PROGRAMS_NEED_X += test-write-cache
diff --git a/t/helper/test-pkt-line.c b/t/helper/test-pkt-line.c
new file mode 100644
index 000000000..5df32b4cb
--- /dev/null
+++ b/t/helper/test-pkt-line.c
@@ -0,0 +1,62 @@
+#include "pkt-line.h"
+
+static void pack_line(const char *line)
+{
+ if (!strcmp(line, "0000") || !strcmp(line, "0000\n"))
+ packet_flush(1);
+ else if (!strcmp(line, "0001") || !strcmp(line, "0001\n"))
+ packet_delim(1);
+ else
+ packet_write_fmt(1, "%s", line);
+}
+
+static void pack(int argc, const char **argv)
+{
+ if (argc) { /* read from argv */
+ int i;
+ for (i = 0; i < argc; i++)
+ pack_line(argv[i]);
+ } else { /* read from stdin */
+ char line[LARGE_PACKET_MAX];
+ while (fgets(line, sizeof(line), stdin)) {
+ pack_line(line);
+ }
+ }
+}
+
+static void unpack(void)
+{
+ struct packet_reader reader;
+ packet_reader_init(&reader, 0, NULL, 0,
+ PACKET_READ_GENTLE_ON_EOF |
+ PACKET_READ_CHOMP_NEWLINE);
+
+ while (packet_reader_read(&reader) != PACKET_READ_EOF) {
+ switch (reader.status) {
+ case PACKET_READ_EOF:
+ break;
+ case PACKET_READ_NORMAL:
+ printf("%s\n", reader.line);
+ break;
+ case PACKET_READ_FLUSH:
+ printf("0000\n");
+ break;
+ case PACKET_READ_DELIM:
+ printf("0001\n");
+ break;
+ }
+ }
+}
+
+int cmd_main(int argc, const char **argv)
+{
+ if (argc < 2)
+ die("too few arguments");
+
+ if (!strcmp(argv[1], "pack"))
+ pack(argc - 2, argv + 2);
+ else if (!strcmp(argv[1], "unpack"))
+ unpack();
+
+ return 0;
+}
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 12/27] serve: introduce git-serve
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (10 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 11/27] test-pkt-line: introduce a packet-line test helper Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-26 10:39 ` Duy Nguyen
2018-01-31 15:39 ` Derrick Stolee
2018-01-25 23:58 ` [PATCH v2 13/27] ls-refs: introduce ls-refs server command Brandon Williams
` (17 subsequent siblings)
29 siblings, 2 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Introduce git-serve, the base server for protocol version 2.
Protocol version 2 is intended to be a replacement for Git's current
wire protocol. The intention is that it will be a simpler, less
wasteful protocol which can evolve over time.
Protocol version 2 improves upon version 1 by eliminating the initial
ref advertisement. In its place a server will export a list of
capabilities and commands which it supports in a capability
advertisement. A client can then request that a particular command be
executed by providing a number of capabilities and command specific
parameters. At the completion of a command, a client can request that
another command be executed or can terminate the connection by sending a
flush packet.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
.gitignore | 1 +
Documentation/technical/protocol-v2.txt | 117 +++++++++++++++
Makefile | 2 +
builtin.h | 1 +
builtin/serve.c | 30 ++++
git.c | 1 +
serve.c | 249 ++++++++++++++++++++++++++++++++
serve.h | 15 ++
t/t5701-git-serve.sh | 56 +++++++
9 files changed, 472 insertions(+)
create mode 100644 Documentation/technical/protocol-v2.txt
create mode 100644 builtin/serve.c
create mode 100644 serve.c
create mode 100644 serve.h
create mode 100755 t/t5701-git-serve.sh
diff --git a/.gitignore b/.gitignore
index 833ef3b0b..2d0450c26 100644
--- a/.gitignore
+++ b/.gitignore
@@ -140,6 +140,7 @@
/git-rm
/git-send-email
/git-send-pack
+/git-serve
/git-sh-i18n
/git-sh-i18n--envsubst
/git-sh-setup
diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
new file mode 100644
index 000000000..7f619a76c
--- /dev/null
+++ b/Documentation/technical/protocol-v2.txt
@@ -0,0 +1,117 @@
+ Git Wire Protocol, Version 2
+==============================
+
+This document presents a specification for a version 2 of Git's wire
+protocol. Protocol v2 will improve upon v1 in the following ways:
+
+ * Instead of multiple service names, multiple commands will be
+ supported by a single service.
+ * Easily extendable as capabilities are moved into their own section
+ of the protocol, no longer being hidden behind a NUL byte and
+ limited by the size of a pkt-line (as there will be a single
+ capability per pkt-line).
+ * Separate out other information hidden behind NUL bytes (e.g. agent
+ string as a capability and symrefs can be requested using 'ls-refs')
+ * Reference advertisement will be omitted unless explicitly requested
+ * ls-refs command to explicitly request some refs
+
+ Detailed Design
+=================
+
+A client can request to speak protocol v2 by sending `version=2` in the
+side-channel `GIT_PROTOCOL` in the initial request to the server.
+
+In protocol v2 communication is command oriented. When first contacting a
+server a list of capabilities will advertised. Some of these capabilities
+will be commands which a client can request be executed. Once a command
+has completed, a client can reuse the connection and request that other
+commands be executed.
+
+ Special Packets
+-----------------
+
+In protocol v2 these special packets will have the following semantics:
+
+ * '0000' Flush Packet (flush-pkt) - indicates the end of a message
+ * '0001' Delimiter Packet (delim-pkt) - separates sections of a message
+
+ Capability Advertisement
+--------------------------
+
+A server which decides to communicate (based on a request from a client)
+using protocol version 2, notifies the client by sending a version string
+in its initial response followed by an advertisement of its capabilities.
+Each capability is a key with an optional value. Clients must ignore all
+unknown keys. Semantics of unknown values are left to the definition of
+each key. Some capabilities will describe commands which can be requested
+to be executed by the client.
+
+ capability-advertisement = protocol-version
+ capability-list
+ flush-pkt
+
+ protocol-version = PKT-LINE("version 2" LF)
+ capability-list = *capability
+ capability = PKT-LINE(key[=value] LF)
+
+ key = 1*CHAR
+ value = 1*CHAR
+ CHAR = 1*(ALPHA / DIGIT / "-" / "_")
+
+A client then responds to select the command it wants with any particular
+capabilities or arguments. There is then an optional section where the
+client can provide any command specific parameters or queries.
+
+ command-request = command
+ capability-list
+ (command-args)
+ flush-pkt
+ command = PKT-LINE("command=" key LF)
+ command-args = delim-pkt
+ *arg
+ arg = 1*CHAR
+
+The server will then check to ensure that the client's request is
+comprised of a valid command as well as valid capabilities which were
+advertised. If the request is valid the server will then execute the
+command.
+
+When a command has finished a client can either request that another
+command be executed or can terminate the connection by sending an empty
+request consisting of just a flush-pkt.
+
+ Capabilities
+~~~~~~~~~~~~~~
+
+There are two different types of capabilities: normal capabilities,
+which can be used to to convey information or alter the behavior of a
+request, and command capabilities, which are the core actions that a
+client wants to perform (fetch, push, etc).
+
+ agent
+-------
+
+The server can advertise the `agent` capability with a value `X` (in the
+form `agent=X`) to notify the client that the server is running version
+`X`. The client may optionally send its own agent string by including
+the `agent` capability with a value `Y` (in the form `agent=Y`) in its
+request to the server (but it MUST NOT do so if the server did not
+advertise the agent capability). The `X` and `Y` strings may contain any
+printable ASCII characters except space (i.e., the byte range 32 < x <
+127), and are typically of the form "package/version" (e.g.,
+"git/1.8.3.1"). The agent strings are purely informative for statistics
+and debugging purposes, and MUST NOT be used to programmatically assume
+the presence or absence of particular features.
+
+ stateless-rpc
+---------------
+
+If advertised, the `stateless-rpc` capability indicates that the server
+supports running commands in a stateless-rpc mode, which means that a
+command lasts for only a single request-response round.
+
+Normally a command can last for as many rounds as are required to
+complete it (multiple for negotiation during fetch or no additional
+trips in the case of ls-refs). If the client sends the `stateless-rpc`
+capability with a value of `true` (in the form `stateless-rpc=true`)
+then the invoked command must only last a single round.
diff --git a/Makefile b/Makefile
index 3b849c060..18c255428 100644
--- a/Makefile
+++ b/Makefile
@@ -881,6 +881,7 @@ LIB_OBJS += revision.o
LIB_OBJS += run-command.o
LIB_OBJS += send-pack.o
LIB_OBJS += sequencer.o
+LIB_OBJS += serve.o
LIB_OBJS += server-info.o
LIB_OBJS += setup.o
LIB_OBJS += sha1-array.o
@@ -1014,6 +1015,7 @@ BUILTIN_OBJS += builtin/rev-parse.o
BUILTIN_OBJS += builtin/revert.o
BUILTIN_OBJS += builtin/rm.o
BUILTIN_OBJS += builtin/send-pack.o
+BUILTIN_OBJS += builtin/serve.o
BUILTIN_OBJS += builtin/shortlog.o
BUILTIN_OBJS += builtin/show-branch.o
BUILTIN_OBJS += builtin/show-ref.o
diff --git a/builtin.h b/builtin.h
index f332a1257..3f3fdfc28 100644
--- a/builtin.h
+++ b/builtin.h
@@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);
extern int cmd_revert(int argc, const char **argv, const char *prefix);
extern int cmd_rm(int argc, const char **argv, const char *prefix);
extern int cmd_send_pack(int argc, const char **argv, const char *prefix);
+extern int cmd_serve(int argc, const char **argv, const char *prefix);
extern int cmd_shortlog(int argc, const char **argv, const char *prefix);
extern int cmd_show(int argc, const char **argv, const char *prefix);
extern int cmd_show_branch(int argc, const char **argv, const char *prefix);
diff --git a/builtin/serve.c b/builtin/serve.c
new file mode 100644
index 000000000..d3fd240bb
--- /dev/null
+++ b/builtin/serve.c
@@ -0,0 +1,30 @@
+#include "cache.h"
+#include "builtin.h"
+#include "parse-options.h"
+#include "serve.h"
+
+static char const * const serve_usage[] = {
+ N_("git serve [<options>]"),
+ NULL
+};
+
+int cmd_serve(int argc, const char **argv, const char *prefix)
+{
+ struct serve_options opts = SERVE_OPTIONS_INIT;
+
+ struct option options[] = {
+ OPT_BOOL(0, "stateless-rpc", &opts.stateless_rpc,
+ N_("quit after a single request/response exchange")),
+ OPT_BOOL(0, "advertise-capabilities", &opts.advertise_capabilities,
+ N_("exit immediately after advertising capabilities")),
+ OPT_END()
+ };
+
+ /* ignore all unknown cmdline switches for now */
+ argc = parse_options(argc, argv, prefix, options, serve_usage,
+ PARSE_OPT_KEEP_DASHDASH |
+ PARSE_OPT_KEEP_UNKNOWN);
+ serve(&opts);
+
+ return 0;
+}
diff --git a/git.c b/git.c
index f71073dc8..f85d682b6 100644
--- a/git.c
+++ b/git.c
@@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {
{ "revert", cmd_revert, RUN_SETUP | NEED_WORK_TREE },
{ "rm", cmd_rm, RUN_SETUP },
{ "send-pack", cmd_send_pack, RUN_SETUP },
+ { "serve", cmd_serve, RUN_SETUP },
{ "shortlog", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },
{ "show", cmd_show, RUN_SETUP },
{ "show-branch", cmd_show_branch, RUN_SETUP },
diff --git a/serve.c b/serve.c
new file mode 100644
index 000000000..90e3defe8
--- /dev/null
+++ b/serve.c
@@ -0,0 +1,249 @@
+#include "cache.h"
+#include "repository.h"
+#include "config.h"
+#include "pkt-line.h"
+#include "version.h"
+#include "argv-array.h"
+#include "serve.h"
+
+static int always_advertise(struct repository *r,
+ struct strbuf *value)
+{
+ return 1;
+}
+
+static int agent_advertise(struct repository *r,
+ struct strbuf *value)
+{
+ if (value)
+ strbuf_addstr(value, git_user_agent_sanitized());
+ return 1;
+}
+
+struct protocol_capability {
+ /*
+ * The name of the capability. The server uses this name when
+ * advertising this capability, and the client uses this name to
+ * specify this capability.
+ */
+ const char *name;
+
+ /*
+ * Function queried to see if a capability should be advertised.
+ * Optionally a value can be specified by adding it to 'value'.
+ * If a value is added to 'value', the server will advertise this
+ * capability as "<name>=<value>" instead of "<name>".
+ */
+ int (*advertise)(struct repository *r, struct strbuf *value);
+
+ /*
+ * Function called when a client requests the capability as a command.
+ * The command request will be provided to the function via 'keys', the
+ * capabilities requested, and 'args', the command specific parameters.
+ *
+ * This field should be NULL for capabilities which are not commands.
+ */
+ int (*command)(struct repository *r,
+ struct argv_array *keys,
+ struct argv_array *args);
+};
+
+static struct protocol_capability capabilities[] = {
+ { "agent", agent_advertise, NULL },
+ { "stateless-rpc", always_advertise, NULL },
+};
+
+static void advertise_capabilities(void)
+{
+ struct strbuf capability = STRBUF_INIT;
+ struct strbuf value = STRBUF_INIT;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
+ struct protocol_capability *c = &capabilities[i];
+
+ if (c->advertise(the_repository, &value)) {
+ strbuf_addstr(&capability, c->name);
+
+ if (value.len) {
+ strbuf_addch(&capability, '=');
+ strbuf_addbuf(&capability, &value);
+ }
+
+ strbuf_addch(&capability, '\n');
+ packet_write(1, capability.buf, capability.len);
+ }
+
+ strbuf_reset(&capability);
+ strbuf_reset(&value);
+ }
+
+ packet_flush(1);
+ strbuf_release(&capability);
+ strbuf_release(&value);
+}
+
+static struct protocol_capability *get_capability(const char *key)
+{
+ int i;
+
+ if (!key)
+ return NULL;
+
+ for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
+ struct protocol_capability *c = &capabilities[i];
+ const char *out;
+ if (skip_prefix(key, c->name, &out) && (!*out || *out == '='))
+ return c;
+ }
+
+ return NULL;
+}
+
+static int is_valid_capability(const char *key)
+{
+ const struct protocol_capability *c = get_capability(key);
+
+ return c && c->advertise(the_repository, NULL);
+}
+
+static int is_command(const char *key, struct protocol_capability **command)
+{
+ const char *out;
+
+ if (skip_prefix(key, "command=", &out)) {
+ struct protocol_capability *cmd = get_capability(out);
+
+ if (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)
+ die("invalid command '%s'", out);
+ if (*command)
+ die("command already requested");
+
+ *command = cmd;
+ return 1;
+ }
+
+ return 0;
+}
+
+int has_capability(const struct argv_array *keys, const char *capability,
+ const char **value)
+{
+ int i;
+ for (i = 0; i < keys->argc; i++) {
+ const char *out;
+ if (skip_prefix(keys->argv[i], capability, &out) &&
+ (!*out || *out == '=')) {
+ if (value) {
+ if (*out == '=')
+ out++;
+ *value = out;
+ }
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+enum request_state {
+ PROCESS_REQUEST_KEYS = 0,
+ PROCESS_REQUEST_ARGS,
+ PROCESS_REQUEST_DONE,
+};
+
+static int process_request(void)
+{
+ enum request_state state = PROCESS_REQUEST_KEYS;
+ char *buffer = packet_buffer;
+ unsigned buffer_size = sizeof(packet_buffer);
+ int pktlen;
+ struct argv_array keys = ARGV_ARRAY_INIT;
+ struct argv_array args = ARGV_ARRAY_INIT;
+ struct protocol_capability *command = NULL;
+
+ while (state != PROCESS_REQUEST_DONE) {
+ switch (packet_read_with_status(0, NULL, NULL, buffer,
+ buffer_size, &pktlen,
+ PACKET_READ_CHOMP_NEWLINE)) {
+ case PACKET_READ_EOF:
+ BUG("Should have already died when seeing EOF");
+ case PACKET_READ_NORMAL:
+ break;
+ case PACKET_READ_FLUSH:
+ state = PROCESS_REQUEST_DONE;
+ continue;
+ case PACKET_READ_DELIM:
+ if (state != PROCESS_REQUEST_KEYS)
+ die("protocol error");
+ state = PROCESS_REQUEST_ARGS;
+ /*
+ * maybe include a check to make sure that a
+ * command/capabilities were given.
+ */
+ continue;
+ }
+
+ switch (state) {
+ case PROCESS_REQUEST_KEYS:
+ /* collect request; a sequence of keys and values */
+ if (is_command(buffer, &command) ||
+ is_valid_capability(buffer))
+ argv_array_push(&keys, buffer);
+ else
+ die("unknown capability '%s'", buffer);
+ break;
+ case PROCESS_REQUEST_ARGS:
+ /* collect arguments for the requested command */
+ argv_array_push(&args, buffer);
+ break;
+ case PROCESS_REQUEST_DONE:
+ continue;
+ }
+ }
+
+ /*
+ * If no command and no keys were given then the client wanted to
+ * terminate the connection.
+ */
+ if (!keys.argc && !args.argc)
+ return 1;
+
+ if (!command)
+ die("no command requested");
+
+ command->command(the_repository, &keys, &args);
+
+ argv_array_clear(&keys);
+ argv_array_clear(&args);
+ return 0;
+}
+
+/* Main serve loop for protocol version 2 */
+void serve(struct serve_options *options)
+{
+ if (options->advertise_capabilities || !options->stateless_rpc) {
+ /* serve by default supports v2 */
+ packet_write_fmt(1, "version 2\n");
+
+ advertise_capabilities();
+ /*
+ * If only the list of capabilities was requested exit
+ * immediately after advertising capabilities
+ */
+ if (options->advertise_capabilities)
+ return;
+ }
+
+ /*
+ * If stateless-rpc was requested then exit after
+ * a single request/response exchange
+ */
+ if (options->stateless_rpc) {
+ process_request();
+ } else {
+ for (;;)
+ if (process_request())
+ break;
+ }
+}
diff --git a/serve.h b/serve.h
new file mode 100644
index 000000000..fe65ba9f4
--- /dev/null
+++ b/serve.h
@@ -0,0 +1,15 @@
+#ifndef SERVE_H
+#define SERVE_H
+
+struct argv_array;
+extern int has_capability(const struct argv_array *keys, const char *capability,
+ const char **value);
+
+struct serve_options {
+ unsigned advertise_capabilities;
+ unsigned stateless_rpc;
+};
+#define SERVE_OPTIONS_INIT { 0 }
+extern void serve(struct serve_options *options);
+
+#endif /* SERVE_H */
diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh
new file mode 100755
index 000000000..b5cc049e5
--- /dev/null
+++ b/t/t5701-git-serve.sh
@@ -0,0 +1,56 @@
+#!/bin/sh
+
+test_description='test git-serve and server commands'
+
+. ./test-lib.sh
+
+test_expect_success 'test capability advertisement' '
+ cat >expect <<-EOF &&
+ version 2
+ agent=git/$(git version | cut -d" " -f3)
+ stateless-rpc
+ 0000
+ EOF
+
+ git serve --advertise-capabilities >out &&
+ test-pkt-line unpack <out >actual &&
+ test_cmp actual expect
+'
+
+test_expect_success 'stateless-rpc flag does not list capabilities' '
+ test-pkt-line pack >in <<-EOF &&
+ 0000
+ EOF
+ git serve --stateless-rpc >out <in &&
+ test_must_be_empty out
+'
+
+test_expect_success 'request invalid capability' '
+ test-pkt-line pack >in <<-EOF &&
+ foobar
+ 0000
+ EOF
+ test_must_fail git serve --stateless-rpc 2>err <in &&
+ test_i18ngrep "unknown capability" err
+'
+
+test_expect_success 'request with no command' '
+ test-pkt-line pack >in <<-EOF &&
+ agent=git/test
+ 0000
+ EOF
+ test_must_fail git serve --stateless-rpc 2>err <in &&
+ test_i18ngrep "no command requested" err
+'
+
+test_expect_success 'request invalid command' '
+ test-pkt-line pack >in <<-EOF &&
+ command=foo
+ agent=git/test
+ 0000
+ EOF
+ test_must_fail git serve --stateless-rpc 2>err <in &&
+ test_i18ngrep "invalid command" err
+'
+
+test_done
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v2 12/27] serve: introduce git-serve
2018-01-25 23:58 ` [PATCH v2 12/27] serve: introduce git-serve Brandon Williams
@ 2018-01-26 10:39 ` Duy Nguyen
2018-02-27 5:46 ` Jonathan Nieder
2018-01-31 15:39 ` Derrick Stolee
1 sibling, 1 reply; 362+ messages in thread
From: Duy Nguyen @ 2018-01-26 10:39 UTC (permalink / raw)
To: Brandon Williams
Cc: Git Mailing List, Stefan Beller, Junio C Hamano, Jeff King,
Philip Oakley, stolee, Jonathan Nieder
On Fri, Jan 26, 2018 at 6:58 AM, Brandon Williams <bmwill@google.com> wrote:
> + Detailed Design
> +=================
> +
> +A client can request to speak protocol v2 by sending `version=2` in the
> +side-channel `GIT_PROTOCOL` in the initial request to the server.
> +
> +In protocol v2 communication is command oriented. When first contacting a
> +server a list of capabilities will advertised. Some of these capabilities
s/will advertised/will be advertised/
> + Capability Advertisement
> +--------------------------
> +
> +A server which decides to communicate (based on a request from a client)
> +using protocol version 2, notifies the client by sending a version string
> +in its initial response followed by an advertisement of its capabilities.
> +Each capability is a key with an optional value. Clients must ignore all
> +unknown keys.
With have a convention in $GIT_DIR/index file format that's probably a
good thing to follow here: lowercase keys are optional, such unknown
keys can (and must) be ignored. Uppercase keys are mandatory. If a
client can't understand one of those keys, abort. This gives the
server a way to "select" clients and introduce incompatible changes if
we ever have to.
> Semantics of unknown values are left to the definition of
> +each key. Some capabilities will describe commands which can be requested
> +to be executed by the client.
> +
> + capability-advertisement = protocol-version
> + capability-list
> + flush-pkt
> +
> + protocol-version = PKT-LINE("version 2" LF)
> + capability-list = *capability
> + capability = PKT-LINE(key[=value] LF)
> +
> + key = 1*CHAR
> + value = 1*CHAR
> + CHAR = 1*(ALPHA / DIGIT / "-" / "_")
Is this a bit too restricted for "value"? Something like "." (e.g.
version) or "@" (I wonder if anybody will add an capability that
contains an email address). Unless there's a good reason to limit it,
should we just go full ascii (without control codes)?
> +A client then responds to select the command it wants with any particular
> +capabilities or arguments. There is then an optional section where the
> +client can provide any command specific parameters or queries.
> +
> + command-request = command
> + capability-list
> + (command-args)
> + flush-pkt
> + command = PKT-LINE("command=" key LF)
> + command-args = delim-pkt
> + *arg
> + arg = 1*CHAR
> +
> +The server will then check to ensure that the client's request is
> +comprised of a valid command as well as valid capabilities which were
> +advertised. If the request is valid the server will then execute the
> +command.
What happens when the request is not valid? Or..
> +When a command has finished
How does the client know a command has finished? Is it up to each
command design?
More or less related it bugs me that I have a translated git client,
but I still receive remote error messages in English. It's a hard
problem, but I'm hoping that we won't need to change the core protocol
to support that someday. Although we could make rule now that side
channel message could be sent in "printf"-like form, where the client
can translate the format string and substitutes placeholders with real
values afterward...
> a client can either request that another
> +command be executed or can terminate the connection by sending an empty
> +request consisting of just a flush-pkt.
> +
> + Capabilities
> +~~~~~~~~~~~~~~
> +
> +There are two different types of capabilities: normal capabilities,
> +which can be used to to convey information or alter the behavior of a
> +request, and command capabilities, which are the core actions that a
> +client wants to perform (fetch, push, etc).
> +
> + agent
> +-------
> +
> +The server can advertise the `agent` capability with a value `X` (in the
> +form `agent=X`) to notify the client that the server is running version
> +`X`. The client may optionally send its own agent string by including
> +the `agent` capability with a value `Y` (in the form `agent=Y`) in its
> +request to the server (but it MUST NOT do so if the server did not
> +advertise the agent capability). The `X` and `Y` strings may contain any
> +printable ASCII characters except space (i.e., the byte range 32 < x <
> +127), and are typically of the form "package/version" (e.g.,
> +"git/1.8.3.1"). The agent strings are purely informative for statistics
> +and debugging purposes, and MUST NOT be used to programmatically assume
> +the presence or absence of particular features.
> +
> + stateless-rpc
> +---------------
> +
> +If advertised, the `stateless-rpc` capability indicates that the server
> +supports running commands in a stateless-rpc mode, which means that a
> +command lasts for only a single request-response round.
> +
> +Normally a command can last for as many rounds as are required to
> +complete it (multiple for negotiation during fetch or no additional
> +trips in the case of ls-refs). If the client sends the `stateless-rpc`
> +capability with a value of `true` (in the form `stateless-rpc=true`)
> +then the invoked command must only last a single round.
Speaking of stateless-rpc, I remember last time this topic was brought
up, there was some discussion to kind of optimize it for http as well,
to fit the "client sends request, server responds data" model and
avoid too many round trips (ideally everything happens in one round
trip). Does it evolve to anything real? All the cool stuff happened
while I was away, sorry if this was discussed and settled.
--
Duy
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 12/27] serve: introduce git-serve
2018-01-26 10:39 ` Duy Nguyen
@ 2018-02-27 5:46 ` Jonathan Nieder
0 siblings, 0 replies; 362+ messages in thread
From: Jonathan Nieder @ 2018-02-27 5:46 UTC (permalink / raw)
To: Duy Nguyen
Cc: Brandon Williams, Git Mailing List, Stefan Beller,
Junio C Hamano, Jeff King, Philip Oakley, stolee
Hi Duy,
Duy Nguyen wrote:
> On Fri, Jan 26, 2018 at 6:58 AM, Brandon Williams <bmwill@google.com> wrote:
>> + stateless-rpc
>> +---------------
>> +
>> +If advertised, the `stateless-rpc` capability indicates that the server
>> +supports running commands in a stateless-rpc mode, which means that a
>> +command lasts for only a single request-response round.
>> +
>> +Normally a command can last for as many rounds as are required to
>> +complete it (multiple for negotiation during fetch or no additional
>> +trips in the case of ls-refs). If the client sends the `stateless-rpc`
>> +capability with a value of `true` (in the form `stateless-rpc=true`)
>> +then the invoked command must only last a single round.
>
> Speaking of stateless-rpc, I remember last time this topic was brought
> up, there was some discussion to kind of optimize it for http as well,
> to fit the "client sends request, server responds data" model and
> avoid too many round trips (ideally everything happens in one round
> trip). Does it evolve to anything real? All the cool stuff happened
> while I was away, sorry if this was discussed and settled.
We have a few different ideas for improving negotiation. They were
speculative enough that we didn't want to make them part of the
baseline protocol v2. Feel free to poke me in a new thread. :)
Some teasers:
- allow both client and server to suggest commits in negotiation,
instead of just the client?
- send a bloom filter for the peer to filter their suggestions
against?
- send other basic information like maximum generation number or
maximum commit date?
- exponential backoff in negotiation instead of linear walking?
prioritizing ref tips? Imitating the bitmap selection algorithm?
- at the "end" of negotiation, sending a graph data structure instead
of a pack, to allow an extra round trip to produce a truly minimal
pack?
Those are some initial ideas, but it's also likely that someone can
come up with some other experiments to try, too. (E.g. we've looked
at various papers on set reconciliation, but they don't make enough
use of the graph structure to help much.)
Thanks,
Jonathan
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v2 12/27] serve: introduce git-serve
2018-01-25 23:58 ` [PATCH v2 12/27] serve: introduce git-serve Brandon Williams
2018-01-26 10:39 ` Duy Nguyen
@ 2018-01-31 15:39 ` Derrick Stolee
1 sibling, 0 replies; 362+ messages in thread
From: Derrick Stolee @ 2018-01-31 15:39 UTC (permalink / raw)
To: Brandon Williams, git; +Cc: sbeller, gitster, peff, philipoakley, jrnieder
On 1/25/2018 6:58 PM, Brandon Williams wrote:
> Introduce git-serve, the base server for protocol version 2.
>
> Protocol version 2 is intended to be a replacement for Git's current
> wire protocol. The intention is that it will be a simpler, less
> wasteful protocol which can evolve over time.
>
> Protocol version 2 improves upon version 1 by eliminating the initial
> ref advertisement. In its place a server will export a list of
> capabilities and commands which it supports in a capability
> advertisement. A client can then request that a particular command be
> executed by providing a number of capabilities and command specific
> parameters. At the completion of a command, a client can request that
> another command be executed or can terminate the connection by sending a
> flush packet.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
> .gitignore | 1 +
> Documentation/technical/protocol-v2.txt | 117 +++++++++++++++
> Makefile | 2 +
> builtin.h | 1 +
> builtin/serve.c | 30 ++++
> git.c | 1 +
> serve.c | 249 ++++++++++++++++++++++++++++++++
> serve.h | 15 ++
> t/t5701-git-serve.sh | 56 +++++++
> 9 files changed, 472 insertions(+)
> create mode 100644 Documentation/technical/protocol-v2.txt
> create mode 100644 builtin/serve.c
> create mode 100644 serve.c
> create mode 100644 serve.h
> create mode 100755 t/t5701-git-serve.sh
>
> diff --git a/.gitignore b/.gitignore
> index 833ef3b0b..2d0450c26 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -140,6 +140,7 @@
> /git-rm
> /git-send-email
> /git-send-pack
> +/git-serve
> /git-sh-i18n
> /git-sh-i18n--envsubst
> /git-sh-setup
> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
> new file mode 100644
> index 000000000..7f619a76c
> --- /dev/null
> +++ b/Documentation/technical/protocol-v2.txt
> @@ -0,0 +1,117 @@
> + Git Wire Protocol, Version 2
> +==============================
> +
> +This document presents a specification for a version 2 of Git's wire
> +protocol. Protocol v2 will improve upon v1 in the following ways:
> +
> + * Instead of multiple service names, multiple commands will be
> + supported by a single service.
As someone unfamiliar with the old protocol code, this statement is
underselling the architectural significance of your change. The new
model allows a single service to handle all different wire protocols
(git://, ssh://, https://) while being agnostic to the command-specific
logic. It also hides the protocol negotiation away from these consumers.
The ease with which you are adding new commands in later commits really
demonstrates the value of this patch. To make that point here, you would
almost need to document the old model to show how it was difficult to
use and extend. Perhaps this document will not need expanding since the
code speaks for itself.
I just wanted to state for the record that the new architecture is a big
improvement and will make more commands much easier to implement.
> + * Easily extendable as capabilities are moved into their own section
> + of the protocol, no longer being hidden behind a NUL byte and
> + limited by the size of a pkt-line (as there will be a single
> + capability per pkt-line).
> + * Separate out other information hidden behind NUL bytes (e.g. agent
> + string as a capability and symrefs can be requested using 'ls-refs')
> + * Reference advertisement will be omitted unless explicitly requested
> + * ls-refs command to explicitly request some refs
> +
nit: some bullets have full stops (.) and others do not.
> + Detailed Design
> +=================
> +
> +A client can request to speak protocol v2 by sending `version=2` in the
> +side-channel `GIT_PROTOCOL` in the initial request to the server.
> +
> +In protocol v2 communication is command oriented. When first contacting a
> +server a list of capabilities will advertised. Some of these capabilities
> +will be commands which a client can request be executed. Once a command
> +has completed, a client can reuse the connection and request that other
> +commands be executed.
> +
> + Special Packets
> +-----------------
> +
> +In protocol v2 these special packets will have the following semantics:
> +
> + * '0000' Flush Packet (flush-pkt) - indicates the end of a message
> + * '0001' Delimiter Packet (delim-pkt) - separates sections of a message
> +
> + Capability Advertisement
> +--------------------------
> +
> +A server which decides to communicate (based on a request from a client)
> +using protocol version 2, notifies the client by sending a version string
> +in its initial response followed by an advertisement of its capabilities.
> +Each capability is a key with an optional value. Clients must ignore all
> +unknown keys. Semantics of unknown values are left to the definition of
> +each key. Some capabilities will describe commands which can be requested
> +to be executed by the client.
> +
> + capability-advertisement = protocol-version
> + capability-list
> + flush-pkt
> +
> + protocol-version = PKT-LINE("version 2" LF)
> + capability-list = *capability
> + capability = PKT-LINE(key[=value] LF)
> +
> + key = 1*CHAR
> + value = 1*CHAR
> + CHAR = 1*(ALPHA / DIGIT / "-" / "_")
> +
> +A client then responds to select the command it wants with any particular
> +capabilities or arguments. There is then an optional section where the
> +client can provide any command specific parameters or queries.
> +
> + command-request = command
> + capability-list
> + (command-args)
> + flush-pkt
> + command = PKT-LINE("command=" key LF)
> + command-args = delim-pkt
> + *arg
> + arg = 1*CHAR
> +
> +The server will then check to ensure that the client's request is
> +comprised of a valid command as well as valid capabilities which were
> +advertised. If the request is valid the server will then execute the
> +command.
> +
> +When a command has finished a client can either request that another
> +command be executed or can terminate the connection by sending an empty
> +request consisting of just a flush-pkt.
> +
> + Capabilities
> +~~~~~~~~~~~~~~
> +
> +There are two different types of capabilities: normal capabilities,
> +which can be used to to convey information or alter the behavior of a
> +request, and command capabilities, which are the core actions that a
> +client wants to perform (fetch, push, etc).
> +
> + agent
> +-------
> +
> +The server can advertise the `agent` capability with a value `X` (in the
> +form `agent=X`) to notify the client that the server is running version
> +`X`. The client may optionally send its own agent string by including
> +the `agent` capability with a value `Y` (in the form `agent=Y`) in its
> +request to the server (but it MUST NOT do so if the server did not
> +advertise the agent capability). The `X` and `Y` strings may contain any
> +printable ASCII characters except space (i.e., the byte range 32 < x <
> +127), and are typically of the form "package/version" (e.g.,
> +"git/1.8.3.1"). The agent strings are purely informative for statistics
> +and debugging purposes, and MUST NOT be used to programmatically assume
> +the presence or absence of particular features.
> +
> + stateless-rpc
> +---------------
> +
> +If advertised, the `stateless-rpc` capability indicates that the server
> +supports running commands in a stateless-rpc mode, which means that a
> +command lasts for only a single request-response round.
> +
> +Normally a command can last for as many rounds as are required to
> +complete it (multiple for negotiation during fetch or no additional
> +trips in the case of ls-refs). If the client sends the `stateless-rpc`
> +capability with a value of `true` (in the form `stateless-rpc=true`)
> +then the invoked command must only last a single round.
> diff --git a/Makefile b/Makefile
> index 3b849c060..18c255428 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -881,6 +881,7 @@ LIB_OBJS += revision.o
> LIB_OBJS += run-command.o
> LIB_OBJS += send-pack.o
> LIB_OBJS += sequencer.o
> +LIB_OBJS += serve.o
> LIB_OBJS += server-info.o
> LIB_OBJS += setup.o
> LIB_OBJS += sha1-array.o
> @@ -1014,6 +1015,7 @@ BUILTIN_OBJS += builtin/rev-parse.o
> BUILTIN_OBJS += builtin/revert.o
> BUILTIN_OBJS += builtin/rm.o
> BUILTIN_OBJS += builtin/send-pack.o
> +BUILTIN_OBJS += builtin/serve.o
> BUILTIN_OBJS += builtin/shortlog.o
> BUILTIN_OBJS += builtin/show-branch.o
> BUILTIN_OBJS += builtin/show-ref.o
> diff --git a/builtin.h b/builtin.h
> index f332a1257..3f3fdfc28 100644
> --- a/builtin.h
> +++ b/builtin.h
> @@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);
> extern int cmd_revert(int argc, const char **argv, const char *prefix);
> extern int cmd_rm(int argc, const char **argv, const char *prefix);
> extern int cmd_send_pack(int argc, const char **argv, const char *prefix);
> +extern int cmd_serve(int argc, const char **argv, const char *prefix);
> extern int cmd_shortlog(int argc, const char **argv, const char *prefix);
> extern int cmd_show(int argc, const char **argv, const char *prefix);
> extern int cmd_show_branch(int argc, const char **argv, const char *prefix);
> diff --git a/builtin/serve.c b/builtin/serve.c
> new file mode 100644
> index 000000000..d3fd240bb
> --- /dev/null
> +++ b/builtin/serve.c
> @@ -0,0 +1,30 @@
> +#include "cache.h"
> +#include "builtin.h"
> +#include "parse-options.h"
> +#include "serve.h"
> +
> +static char const * const serve_usage[] = {
> + N_("git serve [<options>]"),
> + NULL
> +};
> +
> +int cmd_serve(int argc, const char **argv, const char *prefix)
> +{
> + struct serve_options opts = SERVE_OPTIONS_INIT;
> +
> + struct option options[] = {
> + OPT_BOOL(0, "stateless-rpc", &opts.stateless_rpc,
> + N_("quit after a single request/response exchange")),
> + OPT_BOOL(0, "advertise-capabilities", &opts.advertise_capabilities,
> + N_("exit immediately after advertising capabilities")),
> + OPT_END()
> + };
> +
> + /* ignore all unknown cmdline switches for now */
> + argc = parse_options(argc, argv, prefix, options, serve_usage,
> + PARSE_OPT_KEEP_DASHDASH |
> + PARSE_OPT_KEEP_UNKNOWN);
> + serve(&opts);
> +
> + return 0;
> +}
> diff --git a/git.c b/git.c
> index f71073dc8..f85d682b6 100644
> --- a/git.c
> +++ b/git.c
> @@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {
> { "revert", cmd_revert, RUN_SETUP | NEED_WORK_TREE },
> { "rm", cmd_rm, RUN_SETUP },
> { "send-pack", cmd_send_pack, RUN_SETUP },
> + { "serve", cmd_serve, RUN_SETUP },
> { "shortlog", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },
> { "show", cmd_show, RUN_SETUP },
> { "show-branch", cmd_show_branch, RUN_SETUP },
> diff --git a/serve.c b/serve.c
> new file mode 100644
> index 000000000..90e3defe8
> --- /dev/null
> +++ b/serve.c
> @@ -0,0 +1,249 @@
> +#include "cache.h"
> +#include "repository.h"
> +#include "config.h"
> +#include "pkt-line.h"
> +#include "version.h"
> +#include "argv-array.h"
> +#include "serve.h"
> +
> +static int always_advertise(struct repository *r,
> + struct strbuf *value)
> +{
> + return 1;
> +}
> +
> +static int agent_advertise(struct repository *r,
> + struct strbuf *value)
> +{
> + if (value)
> + strbuf_addstr(value, git_user_agent_sanitized());
> + return 1;
> +}
> +
> +struct protocol_capability {
> + /*
> + * The name of the capability. The server uses this name when
> + * advertising this capability, and the client uses this name to
> + * specify this capability.
> + */
> + const char *name;
> +
> + /*
> + * Function queried to see if a capability should be advertised.
> + * Optionally a value can be specified by adding it to 'value'.
> + * If a value is added to 'value', the server will advertise this
> + * capability as "<name>=<value>" instead of "<name>".
> + */
> + int (*advertise)(struct repository *r, struct strbuf *value);
> +
> + /*
> + * Function called when a client requests the capability as a command.
> + * The command request will be provided to the function via 'keys', the
> + * capabilities requested, and 'args', the command specific parameters.
> + *
> + * This field should be NULL for capabilities which are not commands.
> + */
> + int (*command)(struct repository *r,
> + struct argv_array *keys,
> + struct argv_array *args);
> +};
> +
> +static struct protocol_capability capabilities[] = {
> + { "agent", agent_advertise, NULL },
> + { "stateless-rpc", always_advertise, NULL },
> +};
> +
> +static void advertise_capabilities(void)
> +{
> + struct strbuf capability = STRBUF_INIT;
> + struct strbuf value = STRBUF_INIT;
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
> + struct protocol_capability *c = &capabilities[i];
> +
> + if (c->advertise(the_repository, &value)) {
> + strbuf_addstr(&capability, c->name);
> +
> + if (value.len) {
> + strbuf_addch(&capability, '=');
> + strbuf_addbuf(&capability, &value);
> + }
> +
> + strbuf_addch(&capability, '\n');
> + packet_write(1, capability.buf, capability.len);
> + }
> +
> + strbuf_reset(&capability);
> + strbuf_reset(&value);
> + }
> +
> + packet_flush(1);
> + strbuf_release(&capability);
> + strbuf_release(&value);
> +}
> +
> +static struct protocol_capability *get_capability(const char *key)
> +{
> + int i;
> +
> + if (!key)
> + return NULL;
> +
> + for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
> + struct protocol_capability *c = &capabilities[i];
> + const char *out;
> + if (skip_prefix(key, c->name, &out) && (!*out || *out == '='))
> + return c;
> + }
> +
> + return NULL;
> +}
> +
> +static int is_valid_capability(const char *key)
> +{
> + const struct protocol_capability *c = get_capability(key);
> +
> + return c && c->advertise(the_repository, NULL);
> +}
> +
> +static int is_command(const char *key, struct protocol_capability **command)
> +{
> + const char *out;
> +
> + if (skip_prefix(key, "command=", &out)) {
> + struct protocol_capability *cmd = get_capability(out);
> +
> + if (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)
> + die("invalid command '%s'", out);
> + if (*command)
> + die("command already requested");
> +
> + *command = cmd;
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> +int has_capability(const struct argv_array *keys, const char *capability,
> + const char **value)
> +{
> + int i;
> + for (i = 0; i < keys->argc; i++) {
> + const char *out;
> + if (skip_prefix(keys->argv[i], capability, &out) &&
> + (!*out || *out == '=')) {
> + if (value) {
> + if (*out == '=')
> + out++;
> + *value = out;
> + }
> + return 1;
> + }
> + }
> +
> + return 0;
> +}
> +
> +enum request_state {
> + PROCESS_REQUEST_KEYS = 0,
> + PROCESS_REQUEST_ARGS,
> + PROCESS_REQUEST_DONE,
> +};
> +
> +static int process_request(void)
> +{
> + enum request_state state = PROCESS_REQUEST_KEYS;
> + char *buffer = packet_buffer;
> + unsigned buffer_size = sizeof(packet_buffer);
> + int pktlen;
> + struct argv_array keys = ARGV_ARRAY_INIT;
> + struct argv_array args = ARGV_ARRAY_INIT;
> + struct protocol_capability *command = NULL;
> +
> + while (state != PROCESS_REQUEST_DONE) {
> + switch (packet_read_with_status(0, NULL, NULL, buffer,
> + buffer_size, &pktlen,
> + PACKET_READ_CHOMP_NEWLINE)) {
> + case PACKET_READ_EOF:
> + BUG("Should have already died when seeing EOF");
> + case PACKET_READ_NORMAL:
> + break;
> + case PACKET_READ_FLUSH:
> + state = PROCESS_REQUEST_DONE;
> + continue;
> + case PACKET_READ_DELIM:
> + if (state != PROCESS_REQUEST_KEYS)
> + die("protocol error");
> + state = PROCESS_REQUEST_ARGS;
> + /*
> + * maybe include a check to make sure that a
> + * command/capabilities were given.
> + */
> + continue;
> + }
> +
> + switch (state) {
> + case PROCESS_REQUEST_KEYS:
> + /* collect request; a sequence of keys and values */
> + if (is_command(buffer, &command) ||
> + is_valid_capability(buffer))
> + argv_array_push(&keys, buffer);
> + else
> + die("unknown capability '%s'", buffer);
> + break;
> + case PROCESS_REQUEST_ARGS:
> + /* collect arguments for the requested command */
> + argv_array_push(&args, buffer);
> + break;
> + case PROCESS_REQUEST_DONE:
> + continue;
> + }
> + }
> +
> + /*
> + * If no command and no keys were given then the client wanted to
> + * terminate the connection.
> + */
> + if (!keys.argc && !args.argc)
> + return 1;
> +
> + if (!command)
> + die("no command requested");
> +
> + command->command(the_repository, &keys, &args);
> +
> + argv_array_clear(&keys);
> + argv_array_clear(&args);
> + return 0;
> +}
> +
> +/* Main serve loop for protocol version 2 */
> +void serve(struct serve_options *options)
> +{
> + if (options->advertise_capabilities || !options->stateless_rpc) {
> + /* serve by default supports v2 */
> + packet_write_fmt(1, "version 2\n");
> +
> + advertise_capabilities();
> + /*
> + * If only the list of capabilities was requested exit
> + * immediately after advertising capabilities
> + */
> + if (options->advertise_capabilities)
> + return;
> + }
> +
> + /*
> + * If stateless-rpc was requested then exit after
> + * a single request/response exchange
> + */
> + if (options->stateless_rpc) {
> + process_request();
> + } else {
> + for (;;)
> + if (process_request())
> + break;
> + }
> +}
> diff --git a/serve.h b/serve.h
> new file mode 100644
> index 000000000..fe65ba9f4
> --- /dev/null
> +++ b/serve.h
> @@ -0,0 +1,15 @@
> +#ifndef SERVE_H
> +#define SERVE_H
> +
> +struct argv_array;
> +extern int has_capability(const struct argv_array *keys, const char *capability,
> + const char **value);
> +
> +struct serve_options {
> + unsigned advertise_capabilities;
> + unsigned stateless_rpc;
> +};
> +#define SERVE_OPTIONS_INIT { 0 }
> +extern void serve(struct serve_options *options);
> +
> +#endif /* SERVE_H */
> diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh
> new file mode 100755
> index 000000000..b5cc049e5
> --- /dev/null
> +++ b/t/t5701-git-serve.sh
> @@ -0,0 +1,56 @@
> +#!/bin/sh
> +
> +test_description='test git-serve and server commands'
> +
> +. ./test-lib.sh
> +
> +test_expect_success 'test capability advertisement' '
> + cat >expect <<-EOF &&
> + version 2
> + agent=git/$(git version | cut -d" " -f3)
> + stateless-rpc
> + 0000
> + EOF
> +
> + git serve --advertise-capabilities >out &&
> + test-pkt-line unpack <out >actual &&
> + test_cmp actual expect
> +'
> +
> +test_expect_success 'stateless-rpc flag does not list capabilities' '
> + test-pkt-line pack >in <<-EOF &&
> + 0000
> + EOF
> + git serve --stateless-rpc >out <in &&
> + test_must_be_empty out
> +'
> +
> +test_expect_success 'request invalid capability' '
> + test-pkt-line pack >in <<-EOF &&
> + foobar
> + 0000
> + EOF
> + test_must_fail git serve --stateless-rpc 2>err <in &&
> + test_i18ngrep "unknown capability" err
> +'
> +
> +test_expect_success 'request with no command' '
> + test-pkt-line pack >in <<-EOF &&
> + agent=git/test
> + 0000
> + EOF
> + test_must_fail git serve --stateless-rpc 2>err <in &&
> + test_i18ngrep "no command requested" err
> +'
> +
> +test_expect_success 'request invalid command' '
> + test-pkt-line pack >in <<-EOF &&
> + command=foo
> + agent=git/test
> + 0000
> + EOF
> + test_must_fail git serve --stateless-rpc 2>err <in &&
> + test_i18ngrep "invalid command" err
> +'
> +
> +test_done
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v2 13/27] ls-refs: introduce ls-refs server command
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (11 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 12/27] serve: introduce git-serve Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-26 22:20 ` Stefan Beller
2018-01-25 23:58 ` [PATCH v2 14/27] connect: request remote refs using v2 Brandon Williams
` (16 subsequent siblings)
29 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Introduce the ls-refs server command. In protocol v2, the ls-refs
command is used to request the ref advertisement from the server. Since
it is a command which can be requested (as opposed to mandatory in v1),
a client can sent a number of parameters in its request to limit the ref
advertisement based on provided ref-patterns.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
Documentation/technical/protocol-v2.txt | 32 +++++++++
Makefile | 1 +
ls-refs.c | 96 ++++++++++++++++++++++++++
ls-refs.h | 9 +++
serve.c | 2 +
t/t5701-git-serve.sh | 115 ++++++++++++++++++++++++++++++++
6 files changed, 255 insertions(+)
create mode 100644 ls-refs.c
create mode 100644 ls-refs.h
diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
index 7f619a76c..4683d41ac 100644
--- a/Documentation/technical/protocol-v2.txt
+++ b/Documentation/technical/protocol-v2.txt
@@ -115,3 +115,35 @@ complete it (multiple for negotiation during fetch or no additional
trips in the case of ls-refs). If the client sends the `stateless-rpc`
capability with a value of `true` (in the form `stateless-rpc=true`)
then the invoked command must only last a single round.
+
+ ls-refs
+---------
+
+`ls-refs` is the command used to request a reference advertisement in v2.
+Unlike the current reference advertisement, ls-refs takes in parameters
+which can be used to limit the refs sent from the server.
+
+Additional features not supported in the base command will be advertised
+as the value of the command in the capability advertisement in the form
+of a space separated list of features, e.g. "<command>=<feature 1>
+<feature 2>".
+
+ls-refs takes in the following parameters wrapped in packet-lines:
+
+ symrefs
+ In addition to the object pointed by it, show the underlying ref
+ pointed by it when showing a symbolic ref.
+ peel
+ Show peeled tags.
+ ref-pattern <pattern>
+ When specified, only references matching the one of the provided
+ patterns are displayed.
+
+The output of ls-refs is as follows:
+
+ output = *ref
+ flush-pkt
+ ref = PKT-LINE(obj-id SP refname *(SP ref-attribute) LF)
+ ref-attribute = (symref | peeled)
+ symref = "symref-target:" symref-target
+ peeled = "peeled:" obj-id
diff --git a/Makefile b/Makefile
index 18c255428..e50927cfb 100644
--- a/Makefile
+++ b/Makefile
@@ -825,6 +825,7 @@ LIB_OBJS += list-objects-filter-options.o
LIB_OBJS += ll-merge.o
LIB_OBJS += lockfile.o
LIB_OBJS += log-tree.o
+LIB_OBJS += ls-refs.o
LIB_OBJS += mailinfo.o
LIB_OBJS += mailmap.o
LIB_OBJS += match-trees.o
diff --git a/ls-refs.c b/ls-refs.c
new file mode 100644
index 000000000..70682b4f7
--- /dev/null
+++ b/ls-refs.c
@@ -0,0 +1,96 @@
+#include "cache.h"
+#include "repository.h"
+#include "refs.h"
+#include "remote.h"
+#include "argv-array.h"
+#include "ls-refs.h"
+#include "pkt-line.h"
+
+struct ls_refs_data {
+ unsigned peel;
+ unsigned symrefs;
+ struct argv_array patterns;
+};
+
+/*
+ * Check if one of the patterns matches the tail part of the ref.
+ * If no patterns were provided, all refs match.
+ */
+static int ref_match(const struct argv_array *patterns, const char *refname)
+{
+ char *pathbuf;
+ int i;
+
+ if (!patterns->argc)
+ return 1; /* no restriction */
+
+ pathbuf = xstrfmt("/%s", refname);
+ for (i = 0; i < patterns->argc; i++) {
+ if (!wildmatch(patterns->argv[i], pathbuf, 0)) {
+ free(pathbuf);
+ return 1;
+ }
+ }
+ free(pathbuf);
+ return 0;
+}
+
+static int send_ref(const char *refname, const struct object_id *oid,
+ int flag, void *cb_data)
+{
+ struct ls_refs_data *data = cb_data;
+ const char *refname_nons = strip_namespace(refname);
+ struct strbuf refline = STRBUF_INIT;
+
+ if (!ref_match(&data->patterns, refname))
+ return 0;
+
+ strbuf_addf(&refline, "%s %s", oid_to_hex(oid), refname_nons);
+ if (data->symrefs && flag & REF_ISSYMREF) {
+ struct object_id unused;
+ const char *symref_target = resolve_ref_unsafe(refname, 0,
+ &unused,
+ &flag);
+
+ if (!symref_target)
+ die("'%s' is a symref but it is not?", refname);
+
+ strbuf_addf(&refline, " symref-target:%s", symref_target);
+ }
+
+ if (data->peel) {
+ struct object_id peeled;
+ if (!peel_ref(refname, &peeled))
+ strbuf_addf(&refline, " peeled:%s", oid_to_hex(&peeled));
+ }
+
+ strbuf_addch(&refline, '\n');
+ packet_write(1, refline.buf, refline.len);
+
+ strbuf_release(&refline);
+ return 0;
+}
+
+int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)
+{
+ int i;
+ struct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };
+
+ for (i = 0; i < args->argc; i++) {
+ const char *arg = args->argv[i];
+ const char *out;
+
+ if (!strcmp("peel", arg))
+ data.peel = 1;
+ else if (!strcmp("symrefs", arg))
+ data.symrefs = 1;
+ else if (skip_prefix(arg, "ref-pattern ", &out))
+ argv_array_pushf(&data.patterns, "*/%s", out);
+ }
+
+ head_ref_namespaced(send_ref, &data);
+ for_each_namespaced_ref(send_ref, &data);
+ packet_flush(1);
+ argv_array_clear(&data.patterns);
+ return 0;
+}
diff --git a/ls-refs.h b/ls-refs.h
new file mode 100644
index 000000000..9e4c57bfe
--- /dev/null
+++ b/ls-refs.h
@@ -0,0 +1,9 @@
+#ifndef LS_REFS_H
+#define LS_REFS_H
+
+struct repository;
+struct argv_array;
+extern int ls_refs(struct repository *r, struct argv_array *keys,
+ struct argv_array *args);
+
+#endif /* LS_REFS_H */
diff --git a/serve.c b/serve.c
index 90e3defe8..2f404154a 100644
--- a/serve.c
+++ b/serve.c
@@ -4,6 +4,7 @@
#include "pkt-line.h"
#include "version.h"
#include "argv-array.h"
+#include "ls-refs.h"
#include "serve.h"
static int always_advertise(struct repository *r,
@@ -51,6 +52,7 @@ struct protocol_capability {
static struct protocol_capability capabilities[] = {
{ "agent", agent_advertise, NULL },
{ "stateless-rpc", always_advertise, NULL },
+ { "ls-refs", always_advertise, ls_refs },
};
static void advertise_capabilities(void)
diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh
index b5cc049e5..debdc1b8d 100755
--- a/t/t5701-git-serve.sh
+++ b/t/t5701-git-serve.sh
@@ -9,6 +9,7 @@ test_expect_success 'test capability advertisement' '
version 2
agent=git/$(git version | cut -d" " -f3)
stateless-rpc
+ ls-refs
0000
EOF
@@ -53,4 +54,118 @@ test_expect_success 'request invalid command' '
test_i18ngrep "invalid command" err
'
+# Test the basics of ls-refs
+#
+test_expect_success 'setup some refs and tags' '
+ test_commit one &&
+ git branch dev master &&
+ test_commit two &&
+ git symbolic-ref refs/heads/release refs/heads/master &&
+ git tag -a -m "annotated tag" annotated-tag
+'
+
+test_expect_success 'basics of ls-refs' '
+ test-pkt-line pack >in <<-EOF &&
+ command=ls-refs
+ 0000
+ EOF
+
+ cat >expect <<-EOF &&
+ $(git rev-parse HEAD) HEAD
+ $(git rev-parse refs/heads/dev) refs/heads/dev
+ $(git rev-parse refs/heads/master) refs/heads/master
+ $(git rev-parse refs/heads/release) refs/heads/release
+ $(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag
+ $(git rev-parse refs/tags/one) refs/tags/one
+ $(git rev-parse refs/tags/two) refs/tags/two
+ 0000
+ EOF
+
+ git serve --stateless-rpc <in >out &&
+ test-pkt-line unpack <out >actual &&
+ test_cmp actual expect
+'
+
+test_expect_success 'basic ref-patterns' '
+ test-pkt-line pack >in <<-EOF &&
+ command=ls-refs
+ 0001
+ ref-pattern master
+ ref-pattern one
+ 0000
+ EOF
+
+ cat >expect <<-EOF &&
+ $(git rev-parse refs/heads/master) refs/heads/master
+ $(git rev-parse refs/tags/one) refs/tags/one
+ 0000
+ EOF
+
+ git serve --stateless-rpc <in >out &&
+ test-pkt-line unpack <out >actual &&
+ test_cmp actual expect
+'
+
+test_expect_success 'wildcard ref-patterns' '
+ test-pkt-line pack >in <<-EOF &&
+ command=ls-refs
+ 0001
+ ref-pattern refs/heads/*
+ 0000
+ EOF
+
+ cat >expect <<-EOF &&
+ $(git rev-parse refs/heads/dev) refs/heads/dev
+ $(git rev-parse refs/heads/master) refs/heads/master
+ $(git rev-parse refs/heads/release) refs/heads/release
+ 0000
+ EOF
+
+ git serve --stateless-rpc <in >out &&
+ test-pkt-line unpack <out >actual &&
+ test_cmp actual expect
+'
+
+test_expect_success 'peel parameter' '
+ test-pkt-line pack >in <<-EOF &&
+ command=ls-refs
+ 0001
+ peel
+ ref-pattern refs/tags/*
+ 0000
+ EOF
+
+ cat >expect <<-EOF &&
+ $(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag peeled:$(git rev-parse refs/tags/annotated-tag^{})
+ $(git rev-parse refs/tags/one) refs/tags/one
+ $(git rev-parse refs/tags/two) refs/tags/two
+ 0000
+ EOF
+
+ git serve --stateless-rpc <in >out &&
+ test-pkt-line unpack <out >actual &&
+ test_cmp actual expect
+'
+
+test_expect_success 'symrefs parameter' '
+ test-pkt-line pack >in <<-EOF &&
+ command=ls-refs
+ 0001
+ symrefs
+ ref-pattern refs/heads/*
+ 0000
+ EOF
+
+ cat >expect <<-EOF &&
+ $(git rev-parse refs/heads/dev) refs/heads/dev
+ $(git rev-parse refs/heads/master) refs/heads/master
+ $(git rev-parse refs/heads/release) refs/heads/release symref-target:refs/heads/master
+ 0000
+ EOF
+
+ git serve --stateless-rpc <in >out &&
+ test-pkt-line unpack <out >actual &&
+ test_cmp actual expect
+'
+
test_done
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v2 13/27] ls-refs: introduce ls-refs server command
2018-01-25 23:58 ` [PATCH v2 13/27] ls-refs: introduce ls-refs server command Brandon Williams
@ 2018-01-26 22:20 ` Stefan Beller
2018-02-02 22:31 ` Brandon Williams
0 siblings, 1 reply; 362+ messages in thread
From: Stefan Beller @ 2018-01-26 22:20 UTC (permalink / raw)
To: Brandon Williams
Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee,
Jonathan Nieder
On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams <bmwill@google.com> wrote:
> +ls-refs takes in the following parameters wrapped in packet-lines:
> +
> + symrefs
> + In addition to the object pointed by it, show the underlying ref
> + pointed by it when showing a symbolic ref.
> + peel
> + Show peeled tags.
Would it make sense to default these two to on, and rather have
optional no-symrefs and no-peel ?
That would save bandwidth in the default case, I would think.
> + cat >expect <<-EOF &&
> + $(git rev-parse HEAD) HEAD
> + $(git rev-parse refs/heads/dev) refs/heads/dev
> + $(git rev-parse refs/heads/master) refs/heads/master
> + $(git rev-parse refs/heads/release) refs/heads/release
> + $(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag
> + $(git rev-parse refs/tags/one) refs/tags/one
> + $(git rev-parse refs/tags/two) refs/tags/two
Invoking rev-parse quite a few times? I think the test suite is a
trade off between readability ("what we expect the test to do and test")
and speed (specifically on Windows forking is expensive);
I tried to come up with a more concise way to create this expectation
using git-rev-parse, but did not find a good way to do so.
However maybe
git for-each-ref --format='%(*objectname) %(*refname)' >expect
might help in reproducing the expected message? The downside
of this would be to have to closely guard which refs are there though.
I guess the '--pattern' could help there as it may be the same pattern
as the input to the ls-refs. This might be too abstract for a test though.
I dunno.
Stefan
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 13/27] ls-refs: introduce ls-refs server command
2018-01-26 22:20 ` Stefan Beller
@ 2018-02-02 22:31 ` Brandon Williams
0 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-02-02 22:31 UTC (permalink / raw)
To: Stefan Beller
Cc: git, Junio C Hamano, Jeff King, Philip Oakley, Derrick Stolee,
Jonathan Nieder
On 01/26, Stefan Beller wrote:
> On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams <bmwill@google.com> wrote:
>
> > +ls-refs takes in the following parameters wrapped in packet-lines:
> > +
> > + symrefs
> > + In addition to the object pointed by it, show the underlying ref
> > + pointed by it when showing a symbolic ref.
> > + peel
> > + Show peeled tags.
>
> Would it make sense to default these two to on, and rather have
> optional no-symrefs and no-peel ?
>
> That would save bandwidth in the default case, I would think.
Maybe? That would save sending those strings for each request
--
Brandon Williams
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v2 14/27] connect: request remote refs using v2
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (12 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 13/27] ls-refs: introduce ls-refs server command Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-31 15:22 ` Derrick Stolee
2018-01-25 23:58 ` [PATCH v2 15/27] transport: convert get_refs_list to take a list of ref patterns Brandon Williams
` (15 subsequent siblings)
29 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Teach the client to be able to request a remote's refs using protocol
v2. This is done by having a client issue a 'ls-refs' request to a v2
server.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
builtin/upload-pack.c | 10 ++--
connect.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++-
remote.h | 4 ++
t/t5702-protocol-v2.sh | 28 +++++++++++
transport.c | 2 +-
5 files changed, 160 insertions(+), 7 deletions(-)
create mode 100755 t/t5702-protocol-v2.sh
diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c
index 8d53e9794..a757df8da 100644
--- a/builtin/upload-pack.c
+++ b/builtin/upload-pack.c
@@ -5,6 +5,7 @@
#include "parse-options.h"
#include "protocol.h"
#include "upload-pack.h"
+#include "serve.h"
static const char * const upload_pack_usage[] = {
N_("git upload-pack [<options>] <dir>"),
@@ -16,6 +17,7 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
const char *dir;
int strict = 0;
struct upload_pack_options opts = { 0 };
+ struct serve_options serve_opts = SERVE_OPTIONS_INIT;
struct option options[] = {
OPT_BOOL(0, "stateless-rpc", &opts.stateless_rpc,
N_("quit after a single request/response exchange")),
@@ -48,11 +50,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
switch (determine_protocol_version_server()) {
case protocol_v2:
- /*
- * fetch support for protocol v2 has not been implemented yet,
- * so ignore the request to use v2 and fallback to using v0.
- */
- upload_pack(&opts);
+ serve_opts.advertise_capabilities = opts.advertise_refs;
+ serve_opts.stateless_rpc = opts.stateless_rpc;
+ serve(&serve_opts);
break;
case protocol_v1:
/*
diff --git a/connect.c b/connect.c
index f2157a821..3c653b65b 100644
--- a/connect.c
+++ b/connect.c
@@ -12,9 +12,11 @@
#include "sha1-array.h"
#include "transport.h"
#include "strbuf.h"
+#include "version.h"
#include "protocol.h"
static char *server_capabilities;
+static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;
static const char *parse_feature_value(const char *, const char *, int *);
static int check_ref(const char *name, unsigned int flags)
@@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)
"and the repository exists."));
}
+/* Checks if the server supports the capability 'c' */
+static int server_supports_v2(const char *c, int die_on_error)
+{
+ int i;
+
+ for (i = 0; i < server_capabilities_v2.argc; i++) {
+ const char *out;
+ if (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&
+ (!*out || *out == '='))
+ return 1;
+ }
+
+ if (die_on_error)
+ die("server doesn't support '%s'", c);
+
+ return 0;
+}
+
+static void process_capabilities_v2(struct packet_reader *reader)
+{
+ while (packet_reader_read(reader) == PACKET_READ_NORMAL)
+ argv_array_push(&server_capabilities_v2, reader->line);
+
+ if (reader->status != PACKET_READ_FLUSH)
+ die("protocol error");
+}
+
enum protocol_version discover_version(struct packet_reader *reader)
{
enum protocol_version version = protocol_unknown_version;
@@ -85,7 +114,7 @@ enum protocol_version discover_version(struct packet_reader *reader)
/* Maybe process capabilities here, at least for v2 */
switch (version) {
case protocol_v2:
- die("support for protocol v2 not implemented yet");
+ process_capabilities_v2(reader);
break;
case protocol_v1:
/* Read the peeked version line */
@@ -293,6 +322,98 @@ struct ref **get_remote_heads(struct packet_reader *reader,
return list;
}
+static int process_ref_v2(const char *line, struct ref ***list)
+{
+ int ret = 1;
+ int i = 0;
+ struct object_id old_oid;
+ struct ref *ref;
+ struct string_list line_sections = STRING_LIST_INIT_DUP;
+
+ if (string_list_split(&line_sections, line, ' ', -1) < 2) {
+ ret = 0;
+ goto out;
+ }
+
+ if (get_oid_hex(line_sections.items[i++].string, &old_oid)) {
+ ret = 0;
+ goto out;
+ }
+
+ ref = alloc_ref(line_sections.items[i++].string);
+
+ oidcpy(&ref->old_oid, &old_oid);
+ **list = ref;
+ *list = &ref->next;
+
+ for (; i < line_sections.nr; i++) {
+ const char *arg = line_sections.items[i].string;
+ if (skip_prefix(arg, "symref-target:", &arg))
+ ref->symref = xstrdup(arg);
+
+ if (skip_prefix(arg, "peeled:", &arg)) {
+ struct object_id peeled_oid;
+ char *peeled_name;
+ struct ref *peeled;
+ if (get_oid_hex(arg, &peeled_oid)) {
+ ret = 0;
+ goto out;
+ }
+
+ peeled_name = xstrfmt("%s^{}", ref->name);
+ peeled = alloc_ref(peeled_name);
+
+ oidcpy(&peeled->old_oid, &peeled_oid);
+ **list = peeled;
+ *list = &peeled->next;
+
+ free(peeled_name);
+ }
+ }
+
+out:
+ string_list_clear(&line_sections, 0);
+ return ret;
+}
+
+struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
+ struct ref **list, int for_push,
+ const struct argv_array *ref_patterns)
+{
+ int i;
+ *list = NULL;
+
+ /* Check that the server supports the ls-refs command */
+ /* Issue request for ls-refs */
+ if (server_supports_v2("ls-refs", 1))
+ packet_write_fmt(fd_out, "command=ls-refs\n");
+
+ if (server_supports_v2("agent", 0))
+ packet_write_fmt(fd_out, "agent=%s", git_user_agent_sanitized());
+
+ packet_delim(fd_out);
+ /* When pushing we don't want to request the peeled tags */
+ if (!for_push)
+ packet_write_fmt(fd_out, "peel\n");
+ packet_write_fmt(fd_out, "symrefs\n");
+ for (i = 0; ref_patterns && i < ref_patterns->argc; i++) {
+ packet_write_fmt(fd_out, "ref-pattern %s\n",
+ ref_patterns->argv[i]);
+ }
+ packet_flush(fd_out);
+
+ /* Process response from server */
+ while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
+ if (!process_ref_v2(reader->line, &list))
+ die("invalid ls-refs response: %s", reader->line);
+ }
+
+ if (reader->status != PACKET_READ_FLUSH)
+ die("protocol error");
+
+ return list;
+}
+
static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)
{
int len;
diff --git a/remote.h b/remote.h
index 2016461df..21d0c776c 100644
--- a/remote.h
+++ b/remote.h
@@ -151,10 +151,14 @@ void free_refs(struct ref *ref);
struct oid_array;
struct packet_reader;
+struct argv_array;
extern struct ref **get_remote_heads(struct packet_reader *reader,
struct ref **list, unsigned int flags,
struct oid_array *extra_have,
struct oid_array *shallow_points);
+extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
+ struct ref **list, int for_push,
+ const struct argv_array *ref_patterns);
int resolve_remote_symref(struct ref *ref, struct ref *list);
int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);
diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
new file mode 100755
index 000000000..4bf4d61ac
--- /dev/null
+++ b/t/t5702-protocol-v2.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+
+test_description='test git wire-protocol version 2'
+
+TEST_NO_CREATE_REPO=1
+
+. ./test-lib.sh
+
+# Test protocol v2 with 'file://' transport
+#
+test_expect_success 'create repo to be served by file:// transport' '
+ git init file_parent &&
+ test_commit -C file_parent one
+'
+
+test_expect_success 'list refs with file:// using protocol v2' '
+ GIT_TRACE_PACKET=1 git -c protocol.version=2 \
+ ls-remote --symref "file://$(pwd)/file_parent" >actual 2>log &&
+
+ # Server responded using protocol v2
+ cat log &&
+ grep "git< version 2" log &&
+
+ git ls-remote --symref "file://$(pwd)/file_parent" >expect &&
+ test_cmp actual expect
+'
+
+test_done
diff --git a/transport.c b/transport.c
index 83d9dd1df..ffc6b2614 100644
--- a/transport.c
+++ b/transport.c
@@ -204,7 +204,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
data->version = discover_version(&reader);
switch (data->version) {
case protocol_v2:
- die("support for protocol v2 not implemented yet");
+ get_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);
break;
case protocol_v1:
case protocol_v0:
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v2 14/27] connect: request remote refs using v2
2018-01-25 23:58 ` [PATCH v2 14/27] connect: request remote refs using v2 Brandon Williams
@ 2018-01-31 15:22 ` Derrick Stolee
2018-01-31 20:10 ` Eric Sunshine
0 siblings, 1 reply; 362+ messages in thread
From: Derrick Stolee @ 2018-01-31 15:22 UTC (permalink / raw)
To: Brandon Williams, git; +Cc: sbeller, gitster, peff, philipoakley, jrnieder
On 1/25/2018 6:58 PM, Brandon Williams wrote:
> Teach the client to be able to request a remote's refs using protocol
> v2. This is done by having a client issue a 'ls-refs' request to a v2
> server.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
> builtin/upload-pack.c | 10 ++--
> connect.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++-
> remote.h | 4 ++
> t/t5702-protocol-v2.sh | 28 +++++++++++
> transport.c | 2 +-
> 5 files changed, 160 insertions(+), 7 deletions(-)
> create mode 100755 t/t5702-protocol-v2.sh
>
> diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c
> index 8d53e9794..a757df8da 100644
> --- a/builtin/upload-pack.c
> +++ b/builtin/upload-pack.c
> @@ -5,6 +5,7 @@
> #include "parse-options.h"
> #include "protocol.h"
> #include "upload-pack.h"
> +#include "serve.h"
>
> static const char * const upload_pack_usage[] = {
> N_("git upload-pack [<options>] <dir>"),
> @@ -16,6 +17,7 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
> const char *dir;
> int strict = 0;
> struct upload_pack_options opts = { 0 };
> + struct serve_options serve_opts = SERVE_OPTIONS_INIT;
> struct option options[] = {
> OPT_BOOL(0, "stateless-rpc", &opts.stateless_rpc,
> N_("quit after a single request/response exchange")),
> @@ -48,11 +50,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)
>
> switch (determine_protocol_version_server()) {
> case protocol_v2:
> - /*
> - * fetch support for protocol v2 has not been implemented yet,
> - * so ignore the request to use v2 and fallback to using v0.
> - */
> - upload_pack(&opts);
> + serve_opts.advertise_capabilities = opts.advertise_refs;
> + serve_opts.stateless_rpc = opts.stateless_rpc;
> + serve(&serve_opts);
> break;
> case protocol_v1:
> /*
> diff --git a/connect.c b/connect.c
> index f2157a821..3c653b65b 100644
> --- a/connect.c
> +++ b/connect.c
> @@ -12,9 +12,11 @@
> #include "sha1-array.h"
> #include "transport.h"
> #include "strbuf.h"
> +#include "version.h"
> #include "protocol.h"
>
> static char *server_capabilities;
> +static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;
> static const char *parse_feature_value(const char *, const char *, int *);
>
> static int check_ref(const char *name, unsigned int flags)
> @@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)
> "and the repository exists."));
> }
>
> +/* Checks if the server supports the capability 'c' */
> +static int server_supports_v2(const char *c, int die_on_error)
> +{
> + int i;
> +
> + for (i = 0; i < server_capabilities_v2.argc; i++) {
> + const char *out;
> + if (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&
> + (!*out || *out == '='))
> + return 1;
> + }
> +
> + if (die_on_error)
> + die("server doesn't support '%s'", c);
> +
> + return 0;
> +}
> +
> +static void process_capabilities_v2(struct packet_reader *reader)
> +{
> + while (packet_reader_read(reader) == PACKET_READ_NORMAL)
> + argv_array_push(&server_capabilities_v2, reader->line);
> +
> + if (reader->status != PACKET_READ_FLUSH)
> + die("protocol error");
> +}
> +
> enum protocol_version discover_version(struct packet_reader *reader)
> {
> enum protocol_version version = protocol_unknown_version;
> @@ -85,7 +114,7 @@ enum protocol_version discover_version(struct packet_reader *reader)
> /* Maybe process capabilities here, at least for v2 */
> switch (version) {
> case protocol_v2:
> - die("support for protocol v2 not implemented yet");
> + process_capabilities_v2(reader);
> break;
> case protocol_v1:
> /* Read the peeked version line */
> @@ -293,6 +322,98 @@ struct ref **get_remote_heads(struct packet_reader *reader,
> return list;
> }
>
> +static int process_ref_v2(const char *line, struct ref ***list)
> +{
> + int ret = 1;
> + int i = 0;
nit: you set 'i' here, but first use it in a for loop with blank
initializer. Perhaps keep the first assignment closer to the first use?
> + struct object_id old_oid;
> + struct ref *ref;
> + struct string_list line_sections = STRING_LIST_INIT_DUP;
> +
> + if (string_list_split(&line_sections, line, ' ', -1) < 2) {
> + ret = 0;
> + goto out;
> + }
> +
> + if (get_oid_hex(line_sections.items[i++].string, &old_oid)) {
> + ret = 0;
> + goto out;
> + }
> +
> + ref = alloc_ref(line_sections.items[i++].string);
> +
> + oidcpy(&ref->old_oid, &old_oid);
> + **list = ref;
> + *list = &ref->next;
> +
> + for (; i < line_sections.nr; i++) {
> + const char *arg = line_sections.items[i].string;
> + if (skip_prefix(arg, "symref-target:", &arg))
> + ref->symref = xstrdup(arg);
> +
> + if (skip_prefix(arg, "peeled:", &arg)) {
> + struct object_id peeled_oid;
> + char *peeled_name;
> + struct ref *peeled;
> + if (get_oid_hex(arg, &peeled_oid)) {
> + ret = 0;
> + goto out;
> + }
> +
> + peeled_name = xstrfmt("%s^{}", ref->name);
> + peeled = alloc_ref(peeled_name);
> +
> + oidcpy(&peeled->old_oid, &peeled_oid);
> + **list = peeled;
> + *list = &peeled->next;
> +
> + free(peeled_name);
> + }
> + }
> +
> +out:
> + string_list_clear(&line_sections, 0);
> + return ret;
> +}
> +
> +struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
> + struct ref **list, int for_push,
> + const struct argv_array *ref_patterns)
> +{
> + int i;
> + *list = NULL;
> +
> + /* Check that the server supports the ls-refs command */
> + /* Issue request for ls-refs */
> + if (server_supports_v2("ls-refs", 1))
> + packet_write_fmt(fd_out, "command=ls-refs\n");
> +
> + if (server_supports_v2("agent", 0))
> + packet_write_fmt(fd_out, "agent=%s", git_user_agent_sanitized());
> +
> + packet_delim(fd_out);
> + /* When pushing we don't want to request the peeled tags */
> + if (!for_push)
> + packet_write_fmt(fd_out, "peel\n");
> + packet_write_fmt(fd_out, "symrefs\n");
> + for (i = 0; ref_patterns && i < ref_patterns->argc; i++) {
> + packet_write_fmt(fd_out, "ref-pattern %s\n",
> + ref_patterns->argv[i]);
> + }
> + packet_flush(fd_out);
> +
> + /* Process response from server */
> + while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
> + if (!process_ref_v2(reader->line, &list))
> + die("invalid ls-refs response: %s", reader->line);
> + }
> +
> + if (reader->status != PACKET_READ_FLUSH)
> + die("protocol error");
> +
> + return list;
> +}
> +
> static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)
> {
> int len;
> diff --git a/remote.h b/remote.h
> index 2016461df..21d0c776c 100644
> --- a/remote.h
> +++ b/remote.h
> @@ -151,10 +151,14 @@ void free_refs(struct ref *ref);
>
> struct oid_array;
> struct packet_reader;
> +struct argv_array;
> extern struct ref **get_remote_heads(struct packet_reader *reader,
> struct ref **list, unsigned int flags,
> struct oid_array *extra_have,
> struct oid_array *shallow_points);
> +extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
> + struct ref **list, int for_push,
> + const struct argv_array *ref_patterns);
>
> int resolve_remote_symref(struct ref *ref, struct ref *list);
> int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);
> diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
> new file mode 100755
> index 000000000..4bf4d61ac
> --- /dev/null
> +++ b/t/t5702-protocol-v2.sh
> @@ -0,0 +1,28 @@
> +#!/bin/sh
> +
> +test_description='test git wire-protocol version 2'
> +
> +TEST_NO_CREATE_REPO=1
> +
> +. ./test-lib.sh
> +
> +# Test protocol v2 with 'file://' transport
> +#
> +test_expect_success 'create repo to be served by file:// transport' '
> + git init file_parent &&
> + test_commit -C file_parent one
> +'
> +
> +test_expect_success 'list refs with file:// using protocol v2' '
> + GIT_TRACE_PACKET=1 git -c protocol.version=2 \
> + ls-remote --symref "file://$(pwd)/file_parent" >actual 2>log &&
> +
> + # Server responded using protocol v2
> + cat log &&
> + grep "git< version 2" log &&
> +
> + git ls-remote --symref "file://$(pwd)/file_parent" >expect &&
> + test_cmp actual expect
> +'
> +
> +test_done
> diff --git a/transport.c b/transport.c
> index 83d9dd1df..ffc6b2614 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -204,7 +204,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
> data->version = discover_version(&reader);
> switch (data->version) {
> case protocol_v2:
> - die("support for protocol v2 not implemented yet");
> + get_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);
> break;
> case protocol_v1:
> case protocol_v0:
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 14/27] connect: request remote refs using v2
2018-01-31 15:22 ` Derrick Stolee
@ 2018-01-31 20:10 ` Eric Sunshine
2018-01-31 22:14 ` Derrick Stolee
0 siblings, 1 reply; 362+ messages in thread
From: Eric Sunshine @ 2018-01-31 20:10 UTC (permalink / raw)
To: Derrick Stolee
Cc: Brandon Williams, Git List, Stefan Beller, Junio C Hamano,
Jeff King, Philip Oakley, Jonathan Nieder
On Wed, Jan 31, 2018 at 10:22 AM, Derrick Stolee <stolee@gmail.com> wrote:
> On 1/25/2018 6:58 PM, Brandon Williams wrote:
>> +static int process_ref_v2(const char *line, struct ref ***list)
>> +{
>> + int ret = 1;
>> + int i = 0;
>
> nit: you set 'i' here, but first use it in a for loop with blank
> initializer. Perhaps keep the first assignment closer to the first use?
Hmm, I see 'i' being incremented a couple times before the loop...
>> + if (string_list_split(&line_sections, line, ' ', -1) < 2) {
>> + ret = 0;
>> + goto out;
>> + }
>> +
>> + if (get_oid_hex(line_sections.items[i++].string, &old_oid)) {
here...
>> + ret = 0;
>> + goto out;
>> + }
>> +
>> + ref = alloc_ref(line_sections.items[i++].string);
and here...
>> +
>> + oidcpy(&ref->old_oid, &old_oid);
>> + **list = ref;
>> + *list = &ref->next;
>> +
>> + for (; i < line_sections.nr; i++) {
then it is used in the loop.
>> + const char *arg = line_sections.items[i].string;
>> + if (skip_prefix(arg, "symref-target:", &arg))
>> + ref->symref = xstrdup(arg);
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 14/27] connect: request remote refs using v2
2018-01-31 20:10 ` Eric Sunshine
@ 2018-01-31 22:14 ` Derrick Stolee
0 siblings, 0 replies; 362+ messages in thread
From: Derrick Stolee @ 2018-01-31 22:14 UTC (permalink / raw)
To: Eric Sunshine
Cc: Brandon Williams, Git List, Stefan Beller, Junio C Hamano,
Jeff King, Philip Oakley, Jonathan Nieder
On 1/31/2018 3:10 PM, Eric Sunshine wrote:
> On Wed, Jan 31, 2018 at 10:22 AM, Derrick Stolee <stolee@gmail.com> wrote:
>> On 1/25/2018 6:58 PM, Brandon Williams wrote:
>>> +static int process_ref_v2(const char *line, struct ref ***list)
>>> +{
>>> + int ret = 1;
>>> + int i = 0;
>> nit: you set 'i' here, but first use it in a for loop with blank
>> initializer. Perhaps keep the first assignment closer to the first use?
> Hmm, I see 'i' being incremented a couple times before the loop...
>
>>> + if (string_list_split(&line_sections, line, ' ', -1) < 2) {
>>> + ret = 0;
>>> + goto out;
>>> + }
>>> +
>>> + if (get_oid_hex(line_sections.items[i++].string, &old_oid)) {
> here...
>
>>> + ret = 0;
>>> + goto out;
>>> + }
>>> +
>>> + ref = alloc_ref(line_sections.items[i++].string);
> and here...
>
>>> +
>>> + oidcpy(&ref->old_oid, &old_oid);
>>> + **list = ref;
>>> + *list = &ref->next;
>>> +
>>> + for (; i < line_sections.nr; i++) {
> then it is used in the loop.
>
>>> + const char *arg = line_sections.items[i].string;
>>> + if (skip_prefix(arg, "symref-target:", &arg))
>>> + ref->symref = xstrdup(arg);
Thanks! Sorry I missed this.
-Stolee
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v2 15/27] transport: convert get_refs_list to take a list of ref patterns
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (13 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 14/27] connect: request remote refs using v2 Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 16/27] transport: convert transport_get_remote_refs " Brandon Williams
` (14 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Convert the 'struct transport' virtual function 'get_refs_list()' to
optionally take an argv_array of ref patterns. When communicating with
a server using protocol v2 these ref patterns can be sent when
requesting a listing of their refs allowing the server to filter the
refs it sends based on the sent patterns.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
transport-helper.c | 5 +++--
transport-internal.h | 4 +++-
transport.c | 16 +++++++++-------
3 files changed, 15 insertions(+), 10 deletions(-)
diff --git a/transport-helper.c b/transport-helper.c
index 508015023..4c334b5ee 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -1026,7 +1026,8 @@ static int has_attribute(const char *attrs, const char *attr) {
}
}
-static struct ref *get_refs_list(struct transport *transport, int for_push)
+static struct ref *get_refs_list(struct transport *transport, int for_push,
+ const struct argv_array *ref_patterns)
{
struct helper_data *data = transport->data;
struct child_process *helper;
@@ -1039,7 +1040,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)
if (process_connect(transport, for_push)) {
do_take_over(transport);
- return transport->vtable->get_refs_list(transport, for_push);
+ return transport->vtable->get_refs_list(transport, for_push, ref_patterns);
}
if (data->push && for_push)
diff --git a/transport-internal.h b/transport-internal.h
index 3c1a29d72..a67657ce3 100644
--- a/transport-internal.h
+++ b/transport-internal.h
@@ -3,6 +3,7 @@
struct ref;
struct transport;
+struct argv_array;
struct transport_vtable {
/**
@@ -21,7 +22,8 @@ struct transport_vtable {
* the ref without a huge amount of effort, it should store it
* in the ref's old_sha1 field; otherwise it should be all 0.
**/
- struct ref *(*get_refs_list)(struct transport *transport, int for_push);
+ struct ref *(*get_refs_list)(struct transport *transport, int for_push,
+ const struct argv_array *ref_patterns);
/**
* Fetch the objects for the given refs. Note that this gets
diff --git a/transport.c b/transport.c
index ffc6b2614..c54a44630 100644
--- a/transport.c
+++ b/transport.c
@@ -72,7 +72,7 @@ struct bundle_transport_data {
struct bundle_header header;
};
-static struct ref *get_refs_from_bundle(struct transport *transport, int for_push)
+static struct ref *get_refs_from_bundle(struct transport *transport, int for_push, const struct argv_array *ref_patterns)
{
struct bundle_transport_data *data = transport->data;
struct ref *result = NULL;
@@ -189,7 +189,8 @@ static int connect_setup(struct transport *transport, int for_push)
return 0;
}
-static struct ref *get_refs_via_connect(struct transport *transport, int for_push)
+static struct ref *get_refs_via_connect(struct transport *transport, int for_push,
+ const struct argv_array *ref_patterns)
{
struct git_transport_data *data = transport->data;
struct ref *refs = NULL;
@@ -204,7 +205,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
data->version = discover_version(&reader);
switch (data->version) {
case protocol_v2:
- get_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);
+ get_remote_refs(data->fd[1], &reader, &refs, for_push,
+ ref_patterns);
break;
case protocol_v1:
case protocol_v0:
@@ -250,7 +252,7 @@ static int fetch_refs_via_pack(struct transport *transport,
args.update_shallow = data->options.update_shallow;
if (!data->got_remote_heads)
- refs_tmp = get_refs_via_connect(transport, 0);
+ refs_tmp = get_refs_via_connect(transport, 0, NULL);
switch (data->version) {
case protocol_v2:
@@ -568,7 +570,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
int ret = 0;
if (!data->got_remote_heads)
- get_refs_via_connect(transport, 1);
+ get_refs_via_connect(transport, 1, NULL);
memset(&args, 0, sizeof(args));
args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
@@ -1028,7 +1030,7 @@ int transport_push(struct transport *transport,
if (check_push_refs(local_refs, refspec_nr, refspec) < 0)
return -1;
- remote_refs = transport->vtable->get_refs_list(transport, 1);
+ remote_refs = transport->vtable->get_refs_list(transport, 1, NULL);
if (flags & TRANSPORT_PUSH_ALL)
match_flags |= MATCH_REFS_ALL;
@@ -1137,7 +1139,7 @@ int transport_push(struct transport *transport,
const struct ref *transport_get_remote_refs(struct transport *transport)
{
if (!transport->got_remote_refs) {
- transport->remote_refs = transport->vtable->get_refs_list(transport, 0);
+ transport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);
transport->got_remote_refs = 1;
}
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 16/27] transport: convert transport_get_remote_refs to take a list of ref patterns
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (14 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 15/27] transport: convert get_refs_list to take a list of ref patterns Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 17/27] ls-remote: pass ref patterns when requesting a remote's refs Brandon Williams
` (13 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Convert 'transport_get_remote_refs()' to optionally take a list of ref
patterns.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
builtin/clone.c | 2 +-
builtin/fetch.c | 4 ++--
builtin/ls-remote.c | 2 +-
builtin/remote.c | 2 +-
transport.c | 7 +++++--
transport.h | 3 ++-
6 files changed, 12 insertions(+), 8 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index 284651797..6e77d993f 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1121,7 +1121,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
if (transport->smart_options && !deepen)
transport->smart_options->check_self_contained_and_connected = 1;
- refs = transport_get_remote_refs(transport);
+ refs = transport_get_remote_refs(transport, NULL);
if (refs) {
mapped_refs = wanted_peer_refs(refs, refspec);
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 7bbcd26fa..850382f55 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -250,7 +250,7 @@ static void find_non_local_tags(struct transport *transport,
struct string_list_item *item = NULL;
for_each_ref(add_existing, &existing_refs);
- for (ref = transport_get_remote_refs(transport); ref; ref = ref->next) {
+ for (ref = transport_get_remote_refs(transport, NULL); ref; ref = ref->next) {
if (!starts_with(ref->name, "refs/tags/"))
continue;
@@ -336,7 +336,7 @@ static struct ref *get_ref_map(struct transport *transport,
/* opportunistically-updated references: */
struct ref *orefs = NULL, **oref_tail = &orefs;
- const struct ref *remote_refs = transport_get_remote_refs(transport);
+ const struct ref *remote_refs = transport_get_remote_refs(transport, NULL);
if (refspec_count) {
struct refspec *fetch_refspec;
diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c
index c4be98ab9..c6e9847c5 100644
--- a/builtin/ls-remote.c
+++ b/builtin/ls-remote.c
@@ -96,7 +96,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
if (uploadpack != NULL)
transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);
- ref = transport_get_remote_refs(transport);
+ ref = transport_get_remote_refs(transport, NULL);
if (transport_disconnect(transport))
return 1;
diff --git a/builtin/remote.c b/builtin/remote.c
index d95bf904c..d0b6ff6e2 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -862,7 +862,7 @@ static int get_remote_ref_states(const char *name,
if (query) {
transport = transport_get(states->remote, states->remote->url_nr > 0 ?
states->remote->url[0] : NULL);
- remote_refs = transport_get_remote_refs(transport);
+ remote_refs = transport_get_remote_refs(transport, NULL);
transport_disconnect(transport);
states->queried = 1;
diff --git a/transport.c b/transport.c
index c54a44630..dfc603b36 100644
--- a/transport.c
+++ b/transport.c
@@ -1136,10 +1136,13 @@ int transport_push(struct transport *transport,
return 1;
}
-const struct ref *transport_get_remote_refs(struct transport *transport)
+const struct ref *transport_get_remote_refs(struct transport *transport,
+ const struct argv_array *ref_patterns)
{
if (!transport->got_remote_refs) {
- transport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);
+ transport->remote_refs =
+ transport->vtable->get_refs_list(transport, 0,
+ ref_patterns);
transport->got_remote_refs = 1;
}
diff --git a/transport.h b/transport.h
index 731c78b67..4b656f315 100644
--- a/transport.h
+++ b/transport.h
@@ -178,7 +178,8 @@ int transport_push(struct transport *connection,
int refspec_nr, const char **refspec, int flags,
unsigned int * reject_reasons);
-const struct ref *transport_get_remote_refs(struct transport *transport);
+const struct ref *transport_get_remote_refs(struct transport *transport,
+ const struct argv_array *ref_patterns);
int transport_fetch_refs(struct transport *transport, struct ref *refs);
void transport_unlock_pack(struct transport *transport);
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 17/27] ls-remote: pass ref patterns when requesting a remote's refs
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (15 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 16/27] transport: convert transport_get_remote_refs " Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 18/27] fetch: pass ref patterns when fetching Brandon Williams
` (12 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Construct an argv_array of the ref patterns supplied via the command
line and pass them to 'transport_get_remote_refs()' to be used when
communicating protocol v2 so that the server can limit the ref
advertisement based on the supplied patterns.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
builtin/ls-remote.c | 7 +++++--
t/t5702-protocol-v2.sh | 8 ++++++++
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c
index c6e9847c5..caf1051f3 100644
--- a/builtin/ls-remote.c
+++ b/builtin/ls-remote.c
@@ -43,6 +43,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
int show_symref_target = 0;
const char *uploadpack = NULL;
const char **pattern = NULL;
+ struct argv_array ref_patterns = ARGV_ARRAY_INIT;
struct remote *remote;
struct transport *transport;
@@ -74,8 +75,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
if (argc > 1) {
int i;
pattern = xcalloc(argc, sizeof(const char *));
- for (i = 1; i < argc; i++)
+ for (i = 1; i < argc; i++) {
pattern[i - 1] = xstrfmt("*/%s", argv[i]);
+ argv_array_push(&ref_patterns, argv[i]);
+ }
}
remote = remote_get(dest);
@@ -96,7 +99,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
if (uploadpack != NULL)
transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);
- ref = transport_get_remote_refs(transport, NULL);
+ ref = transport_get_remote_refs(transport, &ref_patterns);
if (transport_disconnect(transport))
return 1;
diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 4bf4d61ac..7d8aeb766 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -25,4 +25,12 @@ test_expect_success 'list refs with file:// using protocol v2' '
test_cmp actual expect
'
+test_expect_success 'ref advertisment is filtered with ls-remote using protocol v2' '
+ GIT_TRACE_PACKET=1 git -c protocol.version=2 \
+ ls-remote "file://$(pwd)/file_parent" master 2>log &&
+
+ grep "ref-pattern master" log &&
+ ! grep "refs/tags/" log
+'
+
test_done
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 18/27] fetch: pass ref patterns when fetching
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (16 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 17/27] ls-remote: pass ref patterns when requesting a remote's refs Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 19/27] push: pass ref patterns when pushing Brandon Williams
` (11 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Construct a list of ref patterns to be passed to
'transport_get_remote_refs()' from the refspec to be used during the
fetch. This list of ref patterns will be used to allow the server to
filter the ref advertisement when communicating using protocol v2.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
builtin/fetch.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 850382f55..8128450bf 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -332,11 +332,21 @@ static struct ref *get_ref_map(struct transport *transport,
struct ref *rm;
struct ref *ref_map = NULL;
struct ref **tail = &ref_map;
+ struct argv_array ref_patterns = ARGV_ARRAY_INIT;
/* opportunistically-updated references: */
struct ref *orefs = NULL, **oref_tail = &orefs;
- const struct ref *remote_refs = transport_get_remote_refs(transport, NULL);
+ const struct ref *remote_refs;
+
+ for (i = 0; i < refspec_count; i++) {
+ if (!refspecs[i].exact_sha1)
+ argv_array_push(&ref_patterns, refspecs[i].src);
+ }
+
+ remote_refs = transport_get_remote_refs(transport, &ref_patterns);
+
+ argv_array_clear(&ref_patterns);
if (refspec_count) {
struct refspec *fetch_refspec;
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 19/27] push: pass ref patterns when pushing
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (17 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 18/27] fetch: pass ref patterns when fetching Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 20/27] upload-pack: introduce fetch server command Brandon Williams
` (10 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Construct a list of ref patterns to be passed to 'get_refs_list()' from
the refspec to be used during the push. This list of ref patterns will
be used to allow the server to filter the ref advertisement when
communicating using protocol v2.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
transport.c | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/transport.c b/transport.c
index dfc603b36..6ea3905e3 100644
--- a/transport.c
+++ b/transport.c
@@ -1026,11 +1026,26 @@ int transport_push(struct transport *transport,
int porcelain = flags & TRANSPORT_PUSH_PORCELAIN;
int pretend = flags & TRANSPORT_PUSH_DRY_RUN;
int push_ret, ret, err;
+ struct refspec *tmp_rs;
+ struct argv_array ref_patterns = ARGV_ARRAY_INIT;
+ int i;
if (check_push_refs(local_refs, refspec_nr, refspec) < 0)
return -1;
- remote_refs = transport->vtable->get_refs_list(transport, 1, NULL);
+ tmp_rs = parse_push_refspec(refspec_nr, refspec);
+ for (i = 0; i < refspec_nr; i++) {
+ if (tmp_rs[i].dst)
+ argv_array_push(&ref_patterns, tmp_rs[i].dst);
+ else if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)
+ argv_array_push(&ref_patterns, tmp_rs[i].src);
+ }
+
+ remote_refs = transport->vtable->get_refs_list(transport, 1,
+ &ref_patterns);
+
+ argv_array_clear(&ref_patterns);
+ free_refspec(refspec_nr, tmp_rs);
if (flags & TRANSPORT_PUSH_ALL)
match_flags |= MATCH_REFS_ALL;
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 20/27] upload-pack: introduce fetch server command
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (18 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 19/27] push: pass ref patterns when pushing Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 21/27] fetch-pack: perform a fetch using v2 Brandon Williams
` (9 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Introduce the 'fetch' server command.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
Documentation/technical/protocol-v2.txt | 121 +++++++++++++
serve.c | 2 +
t/t5701-git-serve.sh | 1 +
upload-pack.c | 293 ++++++++++++++++++++++++++++++++
upload-pack.h | 5 +
5 files changed, 422 insertions(+)
diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt
index 4683d41ac..ca09a2cfe 100644
--- a/Documentation/technical/protocol-v2.txt
+++ b/Documentation/technical/protocol-v2.txt
@@ -147,3 +147,124 @@ The output of ls-refs is as follows:
ref-attribute = (symref | peeled)
symref = "symref-target:" symref-target
peeled = "peeled:" obj-id
+
+ fetch
+-------
+
+`fetch` is the command used to fetch a packfile in v2. It can be looked
+at as a modified version of the v1 fetch where the ref-advertisement is
+stripped out (since the `ls-refs` command fills that role) and the
+message format is tweaked to eliminate redundancies and permit easy
+addition of future extensions.
+
+Additional features not supported in the base command will be advertised
+as the value of the command in the capability advertisement in the form
+of a space separated list of features, e.g. "<command>=<feature 1>
+<feature 2>".
+
+A `fetch` request can take the following parameters wrapped in
+packet-lines:
+
+ want <oid>
+ Indicates to the server an object which the client wants to
+ retrieve.
+
+ have <oid>
+ Indicates to the server an object which the client has locally.
+ This allows the server to make a packfile which only contains
+ the objects that the client needs. Multiple 'have' lines can be
+ supplied.
+
+ done
+ Indicates to the server that negotiation should terminate (or
+ not even begin if performing a clone) and that the server should
+ use the information supplied in the request to construct the
+ packfile.
+
+ thin-pack
+ Request that a thin pack be sent, which is a pack with deltas
+ which reference base objects not contained within the pack (but
+ are known to exist at the receiving end). This can reduce the
+ network traffic significantly, but it requires the receiving end
+ to know how to "thicken" these packs by adding the missing bases
+ to the pack.
+
+ no-progress
+ Request that progress information that would normally be sent on
+ side-band channel 2, during the packfile transfer, should not be
+ sent. However, the side-band channel 3 is still used for error
+ responses.
+
+ include-tag
+ Request that annotated tags should be sent if the objects they
+ point to are being sent.
+
+ ofs-delta
+ Indicate that the client understands PACKv2 with delta referring
+ to its base by position in pack rather than by an oid. That is,
+ they can read OBJ_OFS_DELTA (ake type 6) in a packfile.
+
+The response of `fetch` is broken into a number of sections separated by
+delimiter packets (0001), with each section beginning with its section
+header.
+
+ output = *section
+ section = (acknowledgments | packfile)
+ (flush-pkt | delim-pkt)
+
+ acknowledgments = PKT-LINE("acknowledgments" LF)
+ *(ready | nak | ack)
+ ready = PKT-LINE("ready" LF)
+ nak = PKT-LINE("NAK" LF)
+ ack = PKT-LINE("ACK" SP obj-id LF)
+
+ packfile = PKT-LINE("packfile" LF)
+ [PACKFILE]
+
+----
+ acknowledgments section
+ * Always begins with the section header "acknowledgments"
+
+ * The server will respond with "NAK" if none of the object ids sent
+ as have lines were common.
+
+ * The server will respond with "ACK obj-id" for all of the
+ object ids sent as have lines which are common.
+
+ * A response cannot have both "ACK" lines as well as a "NAK"
+ line.
+
+ * The server will respond with a "ready" line indicating that
+ the server has found an acceptable common base and is ready to
+ make and send a packfile (which will be found in the packfile
+ section of the same response)
+
+ * If the client determines that it is finished with negotiations
+ by sending a "done" line, the acknowledgments sections can be
+ omitted from the server's response as an optimization.
+
+ * If the server has found a suitable cut point and has decided
+ to send a "ready" line, then the server can decide to (as an
+ optimization) omit any "ACK" lines it would have sent during
+ its response. This is because the server will have already
+ determined the objects it plans to send to the client and no
+ further negotiation is needed.
+
+----
+ packfile section
+ * Always begins with the section header "packfile"
+
+ * The transmission of the packfile begins immediately after the
+ section header
+
+ * The data transfer of the packfile is always multiplexed, using
+ the same semantics of the 'side-band-64k' capability from
+ protocol version 1. This means that each packet, during the
+ packfile data stream, is made up of a leading 4-byte pkt-line
+ length (typical of the pkt-line format), followed by a 1-byte
+ stream code, followed by the actual data.
+
+ The stream code can be one of:
+ 1 - pack data
+ 2 - progress messages
+ 3 - fatal error message just before stream aborts
diff --git a/serve.c b/serve.c
index 2f404154a..e0235e2bc 100644
--- a/serve.c
+++ b/serve.c
@@ -6,6 +6,7 @@
#include "argv-array.h"
#include "ls-refs.h"
#include "serve.h"
+#include "upload-pack.h"
static int always_advertise(struct repository *r,
struct strbuf *value)
@@ -53,6 +54,7 @@ static struct protocol_capability capabilities[] = {
{ "agent", agent_advertise, NULL },
{ "stateless-rpc", always_advertise, NULL },
{ "ls-refs", always_advertise, ls_refs },
+ { "fetch", always_advertise, upload_pack_v2 },
};
static void advertise_capabilities(void)
diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh
index debdc1b8d..e3bc08667 100755
--- a/t/t5701-git-serve.sh
+++ b/t/t5701-git-serve.sh
@@ -10,6 +10,7 @@ test_expect_success 'test capability advertisement' '
agent=git/$(git version | cut -d" " -f3)
stateless-rpc
ls-refs
+ fetch
0000
EOF
diff --git a/upload-pack.c b/upload-pack.c
index 42d83d5b1..f7944ffdc 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -18,6 +18,7 @@
#include "prio-queue.h"
#include "protocol.h"
#include "upload-pack.h"
+#include "serve.h"
/* Remember to update object flag allocation in object.h */
#define THEY_HAVE (1u << 11)
@@ -1065,3 +1066,295 @@ void upload_pack(struct upload_pack_options *options)
create_pack_file();
}
}
+
+struct upload_pack_data {
+ struct object_array wants;
+ struct oid_array haves;
+
+ unsigned stateless_rpc : 1;
+
+ unsigned use_thin_pack : 1;
+ unsigned use_ofs_delta : 1;
+ unsigned no_progress : 1;
+ unsigned use_include_tag : 1;
+ unsigned done : 1;
+};
+
+#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }
+
+static void upload_pack_data_clear(struct upload_pack_data *data)
+{
+ object_array_clear(&data->wants);
+ oid_array_clear(&data->haves);
+}
+
+static int parse_want(const char *line)
+{
+ const char *arg;
+ if (skip_prefix(line, "want ", &arg)) {
+ struct object_id oid;
+ struct object *o;
+
+ if (get_oid_hex(arg, &oid))
+ die("git upload-pack: protocol error, "
+ "expected to get oid, not '%s'", line);
+
+ o = parse_object(&oid);
+ if (!o) {
+ packet_write_fmt(1,
+ "ERR upload-pack: not our ref %s",
+ oid_to_hex(&oid));
+ die("git upload-pack: not our ref %s",
+ oid_to_hex(&oid));
+ }
+
+ if (!(o->flags & WANTED)) {
+ o->flags |= WANTED;
+ add_object_array(o, NULL, &want_obj);
+ }
+
+ return 1;
+ }
+
+ return 0;
+}
+
+static int parse_have(const char *line, struct oid_array *haves)
+{
+ const char *arg;
+ if (skip_prefix(line, "have ", &arg)) {
+ struct object_id oid;
+
+ if (get_oid_hex(arg, &oid))
+ die("git upload-pack: expected SHA1 object, got '%s'", arg);
+ oid_array_append(haves, &oid);
+ return 1;
+ }
+
+ return 0;
+}
+
+static void process_args(struct argv_array *args, struct upload_pack_data *data)
+{
+ int i;
+
+ for (i = 0; i < args->argc; i++) {
+ const char *arg = args->argv[i];
+
+ /* process want */
+ if (parse_want(arg))
+ continue;
+ /* process have line */
+ if (parse_have(arg, &data->haves))
+ continue;
+
+ /* process args like thin-pack */
+ if (!strcmp(arg, "thin-pack")) {
+ use_thin_pack = 1;
+ continue;
+ }
+ if (!strcmp(arg, "ofs-delta")) {
+ use_ofs_delta = 1;
+ continue;
+ }
+ if (!strcmp(arg, "no-progress")) {
+ no_progress = 1;
+ continue;
+ }
+ if (!strcmp(arg, "include-tag")) {
+ use_include_tag = 1;
+ continue;
+ }
+ if (!strcmp(arg, "done")) {
+ data->done = 1;
+ continue;
+ }
+
+ /* ignore unknown lines maybe? */
+ die("unexpect line: '%s'", arg);
+ }
+}
+
+static void read_haves(struct upload_pack_data *data)
+{
+ struct packet_reader reader;
+ packet_reader_init(&reader, 0, NULL, 0,
+ PACKET_READ_CHOMP_NEWLINE);
+
+ while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {
+
+ if (parse_have(reader.line, &data->haves))
+ continue;
+ if (!strcmp(reader.line, "done")) {
+ data->done = 1;
+ continue;
+ }
+ }
+ if (reader.status != PACKET_READ_FLUSH)
+ die("ERROR");
+}
+
+static int process_haves(struct oid_array *haves, struct oid_array *common)
+{
+ int i;
+
+ /* Process haves */
+ for (i = 0; i < haves->nr; i++) {
+ const struct object_id *oid = &haves->oid[i];
+ struct object *o;
+ int we_knew_they_have = 0;
+
+ if (!has_object_file(oid))
+ continue;
+
+ oid_array_append(common, oid);
+
+ o = parse_object(oid);
+ if (!o)
+ die("oops (%s)", oid_to_hex(oid));
+ if (o->type == OBJ_COMMIT) {
+ struct commit_list *parents;
+ struct commit *commit = (struct commit *)o;
+ if (o->flags & THEY_HAVE)
+ we_knew_they_have = 1;
+ else
+ o->flags |= THEY_HAVE;
+ if (!oldest_have || (commit->date < oldest_have))
+ oldest_have = commit->date;
+ for (parents = commit->parents;
+ parents;
+ parents = parents->next)
+ parents->item->object.flags |= THEY_HAVE;
+ }
+ if (!we_knew_they_have)
+ add_object_array(o, NULL, &have_obj);
+ }
+
+ return 0;
+}
+
+static int send_acks(struct oid_array *acks, struct strbuf *response)
+{
+ int i;
+
+ packet_buf_write(response, "acknowledgments\n");
+
+ /* Send Acks */
+ if (!acks->nr)
+ packet_buf_write(response, "NAK\n");
+
+ for (i = 0; i < acks->nr; i++) {
+ packet_buf_write(response, "ACK %s\n",
+ oid_to_hex(&acks->oid[i]));
+ }
+
+ if (ok_to_give_up()) {
+ /* Send Ready */
+ packet_buf_write(response, "ready\n");
+ return 1;
+ }
+
+ return 0;
+}
+
+static int process_haves_and_send_acks(struct upload_pack_data *data)
+{
+ struct oid_array common = OID_ARRAY_INIT;
+ struct strbuf response = STRBUF_INIT;
+ int ret = 0;
+
+ process_haves(&data->haves, &common);
+ if (data->done) {
+ ret = 1;
+ } else if (send_acks(&common, &response)) {
+ packet_buf_delim(&response);
+ ret = 1;
+ } else {
+ /* Add Flush */
+ packet_buf_flush(&response);
+ ret = 0;
+ }
+
+ /* Send response */
+ write_or_die(1, response.buf, response.len);
+ strbuf_release(&response);
+
+ oid_array_clear(&data->haves);
+ oid_array_clear(&common);
+ return ret;
+}
+
+enum fetch_state {
+ FETCH_PROCESS_ARGS = 0,
+ FETCH_READ_HAVES,
+ FETCH_SEND_ACKS,
+ FETCH_SEND_PACK,
+ FETCH_DONE,
+};
+
+int upload_pack_v2(struct repository *r, struct argv_array *keys,
+ struct argv_array *args)
+{
+ enum fetch_state state = FETCH_PROCESS_ARGS;
+ struct upload_pack_data data = UPLOAD_PACK_DATA_INIT;
+ const char *out;
+ use_sideband = LARGE_PACKET_MAX;
+
+ /* Check if cmd is being run as a stateless-rpc */
+ if (has_capability(keys, "stateless-rpc", &out))
+ if (!strcmp(out, "true"))
+ data.stateless_rpc = 1;
+
+ while (state != FETCH_DONE) {
+ switch (state) {
+ case FETCH_PROCESS_ARGS:
+ process_args(args, &data);
+
+ if (!want_obj.nr) {
+ /*
+ * Request didn't contain any 'want' lines,
+ * guess they didn't want anything.
+ */
+ state = FETCH_DONE;
+ } else if (data.haves.nr) {
+ /*
+ * Request had 'have' lines, so lets ACK them.
+ */
+ state = FETCH_SEND_ACKS;
+ } else {
+ /*
+ * Request had 'want's but no 'have's so we can
+ * immedietly go to construct and send a pack.
+ */
+ state = FETCH_SEND_PACK;
+ }
+ break;
+ case FETCH_READ_HAVES:
+ read_haves(&data);
+ state = FETCH_SEND_ACKS;
+ break;
+ case FETCH_SEND_ACKS:
+ if (process_haves_and_send_acks(&data))
+ state = FETCH_SEND_PACK;
+ else if (data.stateless_rpc)
+ /*
+ * Request was made via stateless-rpc and a
+ * packfile isn't ready to be created and sent.
+ */
+ state = FETCH_DONE;
+ else
+ state = FETCH_READ_HAVES;
+ break;
+ case FETCH_SEND_PACK:
+ packet_write_fmt(1, "packfile\n");
+ create_pack_file();
+ state = FETCH_DONE;
+ break;
+ case FETCH_DONE:
+ continue;
+ }
+ }
+
+ upload_pack_data_clear(&data);
+ return 0;
+}
diff --git a/upload-pack.h b/upload-pack.h
index a71e4dc7e..6b7890238 100644
--- a/upload-pack.h
+++ b/upload-pack.h
@@ -10,4 +10,9 @@ struct upload_pack_options {
void upload_pack(struct upload_pack_options *options);
+struct repository;
+struct argv_array;
+extern int upload_pack_v2(struct repository *r, struct argv_array *keys,
+ struct argv_array *args);
+
#endif /* UPLOAD_PACK_H */
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 21/27] fetch-pack: perform a fetch using v2
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (19 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 20/27] upload-pack: introduce fetch server command Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 22/27] transport-helper: remove name parameter Brandon Williams
` (8 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
When communicating with a v2 server, perform a fetch by requesting the
'fetch' command.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
builtin/fetch-pack.c | 2 +-
fetch-pack.c | 277 ++++++++++++++++++++++++++++++++++++++++++++++++-
fetch-pack.h | 4 +-
t/t5702-protocol-v2.sh | 40 +++++++
transport.c | 8 +-
5 files changed, 324 insertions(+), 7 deletions(-)
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index f492e8abd..867dd3cc7 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -213,7 +213,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
}
ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,
- &shallow, pack_lockfile_ptr);
+ &shallow, pack_lockfile_ptr, protocol_v0);
if (pack_lockfile) {
printf("lock %s\n", pack_lockfile);
fflush(stdout);
diff --git a/fetch-pack.c b/fetch-pack.c
index 9f6b07ad9..17927ae99 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -1008,6 +1008,272 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
return ref;
}
+static void add_wants(const struct ref *wants, struct strbuf *req_buf)
+{
+ for ( ; wants ; wants = wants->next) {
+ const struct object_id *remote = &wants->old_oid;
+ const char *remote_hex;
+ struct object *o;
+
+ /*
+ * If that object is complete (i.e. it is an ancestor of a
+ * local ref), we tell them we have it but do not have to
+ * tell them about its ancestors, which they already know
+ * about.
+ *
+ * We use lookup_object here because we are only
+ * interested in the case we *know* the object is
+ * reachable and we have already scanned it.
+ */
+ if (((o = lookup_object(remote->hash)) != NULL) &&
+ (o->flags & COMPLETE)) {
+ continue;
+ }
+
+ remote_hex = oid_to_hex(remote);
+ packet_buf_write(req_buf, "want %s\n", remote_hex);
+ }
+}
+
+static int add_haves(struct strbuf *req_buf, int *in_vain)
+{
+ int ret = 0;
+ int haves_added = 0;
+ const struct object_id *oid;
+
+ while ((oid = get_rev())) {
+ packet_buf_write(req_buf, "have %s\n", oid_to_hex(oid));
+ if (++haves_added >= INITIAL_FLUSH)
+ break;
+ };
+
+ *in_vain += haves_added;
+ if (!haves_added || *in_vain >= MAX_IN_VAIN) {
+ /* Send Done */
+ packet_buf_write(req_buf, "done\n");
+ ret = 1;
+ }
+
+ return ret;
+}
+
+static int send_haves(int fd_out, int *in_vain)
+{
+ int ret = 0;
+ struct strbuf req_buf = STRBUF_INIT;
+
+ ret = add_haves(&req_buf, in_vain);
+
+ /* Send request */
+ packet_buf_flush(&req_buf);
+ write_or_die(fd_out, req_buf.buf, req_buf.len);
+
+ strbuf_release(&req_buf);
+ return ret;
+}
+
+static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,
+ const struct ref *wants, struct oidset *common,
+ int *in_vain)
+{
+ int ret = 0;
+ struct strbuf req_buf = STRBUF_INIT;
+
+ packet_buf_write(&req_buf, "command=fetch");
+ packet_buf_write(&req_buf, "agent=%s", git_user_agent_sanitized());
+ if (args->stateless_rpc)
+ packet_buf_write(&req_buf, "stateless-rpc=true");
+
+ packet_buf_delim(&req_buf);
+ if (args->use_thin_pack)
+ packet_buf_write(&req_buf, "thin-pack");
+ if (args->no_progress)
+ packet_buf_write(&req_buf, "no-progress");
+ if (args->include_tag)
+ packet_buf_write(&req_buf, "include-tag");
+ if (prefer_ofs_delta)
+ packet_buf_write(&req_buf, "ofs-delta");
+
+ /* add wants */
+ add_wants(wants, &req_buf);
+
+ /*
+ * If we are running stateless-rpc we need to add all the common
+ * commits we've found in previous rounds
+ */
+ if (args->stateless_rpc) {
+ struct oidset_iter iter;
+ const struct object_id *oid;
+ oidset_iter_init(common, &iter);
+
+ while ((oid = oidset_iter_next(&iter))) {
+ packet_buf_write(&req_buf, "have %s\n", oid_to_hex(oid));
+ }
+ }
+
+ /* Add initial haves */
+ ret = add_haves(&req_buf, in_vain);
+
+ /* Send request */
+ packet_buf_flush(&req_buf);
+ write_or_die(fd_out, req_buf.buf, req_buf.len);
+
+ strbuf_release(&req_buf);
+ return ret;
+}
+
+/*
+ * Processes a section header in a server's response and checks if it matches
+ * `section`. If the value of `peek` is 1, the header line will be peeked (and
+ * not consumed); if 0, the line will be consumed and the function will die if
+ * the section header doesn't match what was expected.
+ */
+static int process_section_header(struct packet_reader *reader,
+ const char *section, int peek)
+{
+ int ret;
+
+ if (packet_reader_peek(reader) != PACKET_READ_NORMAL)
+ die("error reading packet");
+
+ ret = !strcmp(reader->line, section);
+
+ if (!peek) {
+ if (!ret)
+ die("expected '%s', received '%s'",
+ section, reader->line);
+ packet_reader_read(reader);
+ }
+
+ return ret;
+}
+
+static int process_acks(struct packet_reader *reader, struct oidset *common)
+{
+ /* received */
+ int received_ready = 0;
+ int received_ack = 0;
+
+ process_section_header(reader, "acknowledgments", 0);
+ while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
+ const char *arg;
+
+ if (!strcmp(reader->line, "NAK"))
+ continue;
+
+ if (skip_prefix(reader->line, "ACK ", &arg)) {
+ struct object_id oid;
+ if (!get_oid_hex(arg, &oid)) {
+ struct commit *commit;
+ oidset_insert(common, &oid);
+ commit = lookup_commit(&oid);
+ mark_common(commit, 0, 1);
+ }
+ continue;
+ }
+
+ if (!strcmp(reader->line, "ready")) {
+ clear_prio_queue(&rev_list);
+ received_ready = 1;
+ continue;
+ }
+
+ die(_("git fetch-pack: expected ACK/NAK, got '%s'"), reader->line);
+ }
+
+ if (reader->status != PACKET_READ_FLUSH &&
+ reader->status != PACKET_READ_DELIM)
+ die("Error during processing acks: %d", reader->status);
+
+ /* return 0 if no common, 1 if there are common, or 2 if ready */
+ return received_ready ? 2 : (received_ack ? 1 : 0);
+}
+
+enum fetch_state {
+ FETCH_CHECK_LOCAL = 0,
+ FETCH_SEND_REQUEST,
+ FETCH_PROCESS_ACKS,
+ FETCH_SEND_HAVES,
+ FETCH_GET_PACK,
+ FETCH_DONE,
+};
+
+static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
+ int fd[2],
+ const struct ref *orig_ref,
+ struct ref **sought, int nr_sought,
+ char **pack_lockfile)
+{
+ struct ref *ref = copy_ref_list(orig_ref);
+ enum fetch_state state = FETCH_CHECK_LOCAL;
+ struct oidset common = OIDSET_INIT;
+ struct packet_reader reader;
+ int in_vain = 0;
+ packet_reader_init(&reader, fd[0], NULL, 0,
+ PACKET_READ_CHOMP_NEWLINE);
+
+ while (state != FETCH_DONE) {
+ switch (state) {
+ case FETCH_CHECK_LOCAL:
+ sort_ref_list(&ref, ref_compare_name);
+ QSORT(sought, nr_sought, cmp_ref_by_name);
+
+ /* v2 supports these by default */
+ allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;
+ use_sideband = 2;
+
+ /* Filter 'ref' by 'sought' and those that aren't local */
+ if (everything_local(args, &ref, sought, nr_sought))
+ state = FETCH_DONE;
+ else
+ state = FETCH_SEND_REQUEST;
+ break;
+ case FETCH_SEND_REQUEST:
+ if (send_fetch_request(fd[1], args, ref, &common, &in_vain))
+ state = FETCH_GET_PACK;
+ else
+ state = FETCH_PROCESS_ACKS;
+ break;
+ case FETCH_PROCESS_ACKS:
+ /* Process ACKs/NAKs */
+ switch (process_acks(&reader, &common)) {
+ case 2:
+ state = FETCH_GET_PACK;
+ break;
+ case 1:
+ in_vain = 0;
+ /* fallthrough */
+ default:
+ if (args->stateless_rpc)
+ state = FETCH_SEND_REQUEST;
+ else
+ state = FETCH_SEND_HAVES;
+ break;
+ }
+ break;
+ case FETCH_SEND_HAVES:
+ if (send_haves(fd[1], &in_vain))
+ state = FETCH_GET_PACK;
+ else
+ state = FETCH_PROCESS_ACKS;
+ break;
+ case FETCH_GET_PACK:
+ /* get the pack */
+ process_section_header(&reader, "packfile", 0);
+ if (get_pack(args, fd, pack_lockfile))
+ die(_("git fetch-pack: fetch failed."));
+
+ state = FETCH_DONE;
+ break;
+ case FETCH_DONE:
+ continue;
+ }
+ }
+
+ oidset_clear(&common);
+ return ref;
+}
+
static void fetch_pack_config(void)
{
git_config_get_int("fetch.unpacklimit", &fetch_unpack_limit);
@@ -1153,7 +1419,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
const char *dest,
struct ref **sought, int nr_sought,
struct oid_array *shallow,
- char **pack_lockfile)
+ char **pack_lockfile,
+ enum protocol_version version)
{
struct ref *ref_cpy;
struct shallow_info si;
@@ -1167,8 +1434,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
die(_("no matching remote head"));
}
prepare_shallow_info(&si, shallow);
- ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
- &si, pack_lockfile);
+ if (version == protocol_v2)
+ ref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,
+ pack_lockfile);
+ else
+ ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
+ &si, pack_lockfile);
reprepare_packed_git();
update_shallow(args, sought, nr_sought, &si);
clear_shallow_info(&si);
diff --git a/fetch-pack.h b/fetch-pack.h
index b6aeb43a8..7afca7305 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -3,6 +3,7 @@
#include "string-list.h"
#include "run-command.h"
+#include "protocol.h"
struct oid_array;
@@ -43,7 +44,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
struct ref **sought,
int nr_sought,
struct oid_array *shallow,
- char **pack_lockfile);
+ char **pack_lockfile,
+ enum protocol_version version);
/*
* Print an appropriate error message for each sought ref that wasn't
diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 7d8aeb766..3e411e178 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -33,4 +33,44 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol
! grep "refs/tags/" log
'
+test_expect_success 'clone with file:// using protocol v2' '
+ GIT_TRACE_PACKET=1 git -c protocol.version=2 \
+ clone "file://$(pwd)/file_parent" file_child 2>log &&
+
+ git -C file_child log -1 --format=%s >actual &&
+ git -C file_parent log -1 --format=%s >expect &&
+ test_cmp expect actual &&
+
+ # Server responded using protocol v1
+ grep "clone< version 2" log
+'
+
+test_expect_success 'fetch with file:// using protocol v2' '
+ test_commit -C file_parent two &&
+
+ GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \
+ fetch origin 2>log &&
+
+ git -C file_child log -1 --format=%s origin/master >actual &&
+ git -C file_parent log -1 --format=%s >expect &&
+ test_cmp expect actual &&
+
+ # Server responded using protocol v1
+ grep "fetch< version 2" log
+'
+
+test_expect_success 'ref advertisment is filtered during fetch using protocol v2' '
+ test_commit -C file_parent three &&
+
+ GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \
+ fetch origin master 2>log &&
+
+ git -C file_child log -1 --format=%s origin/master >actual &&
+ git -C file_parent log -1 --format=%s >expect &&
+ test_cmp expect actual &&
+
+ grep "ref-pattern master" log &&
+ ! grep "refs/tags/" log
+'
+
test_done
diff --git a/transport.c b/transport.c
index 6ea3905e3..4fdbd9adc 100644
--- a/transport.c
+++ b/transport.c
@@ -256,14 +256,18 @@ static int fetch_refs_via_pack(struct transport *transport,
switch (data->version) {
case protocol_v2:
- die("support for protocol v2 not implemented yet");
+ refs = fetch_pack(&args, data->fd, data->conn,
+ refs_tmp ? refs_tmp : transport->remote_refs,
+ dest, to_fetch, nr_heads, &data->shallow,
+ &transport->pack_lockfile, data->version);
+ packet_flush(data->fd[1]);
break;
case protocol_v1:
case protocol_v0:
refs = fetch_pack(&args, data->fd, data->conn,
refs_tmp ? refs_tmp : transport->remote_refs,
dest, to_fetch, nr_heads, &data->shallow,
- &transport->pack_lockfile);
+ &transport->pack_lockfile, data->version);
break;
case protocol_unknown_version:
BUG("unknown protocol version");
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 22/27] transport-helper: remove name parameter
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (20 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 21/27] fetch-pack: perform a fetch using v2 Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 23/27] transport-helper: refactor process_connect_service Brandon Williams
` (7 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Commit 266f1fdfa (transport-helper: be quiet on read errors from
helpers, 2013-06-21) removed a call to 'die()' which printed the name of
the remote helper passed in to the 'recvline_fh()' function using the
'name' parameter. Once the call to 'die()' was removed the parameter
was no longer necessary but wasn't removed. Clean up 'recvline_fh()'
parameter list by removing the 'name' parameter.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
transport-helper.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/transport-helper.c b/transport-helper.c
index 4c334b5ee..d72155768 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -49,7 +49,7 @@ static void sendline(struct helper_data *helper, struct strbuf *buffer)
die_errno("Full write to remote helper failed");
}
-static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)
+static int recvline_fh(FILE *helper, struct strbuf *buffer)
{
strbuf_reset(buffer);
if (debug)
@@ -67,7 +67,7 @@ static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)
static int recvline(struct helper_data *helper, struct strbuf *buffer)
{
- return recvline_fh(helper->out, buffer, helper->name);
+ return recvline_fh(helper->out, buffer);
}
static void write_constant(int fd, const char *str)
@@ -586,7 +586,7 @@ static int process_connect_service(struct transport *transport,
goto exit;
sendline(data, &cmdbuf);
- if (recvline_fh(input, &cmdbuf, name))
+ if (recvline_fh(input, &cmdbuf))
exit(128);
if (!strcmp(cmdbuf.buf, "")) {
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 23/27] transport-helper: refactor process_connect_service
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (21 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 22/27] transport-helper: remove name parameter Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 24/27] transport-helper: introduce stateless-connect Brandon Williams
` (6 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
A future patch will need to take advantage of the logic which runs and
processes the response of the connect command on a remote helper so
factor out this logic from 'process_connect_service()' and place it into
a helper function 'run_connect()'.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
transport-helper.c | 67 +++++++++++++++++++++++++++++++-----------------------
1 file changed, 38 insertions(+), 29 deletions(-)
diff --git a/transport-helper.c b/transport-helper.c
index d72155768..c032a2a87 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -545,14 +545,13 @@ static int fetch_with_import(struct transport *transport,
return 0;
}
-static int process_connect_service(struct transport *transport,
- const char *name, const char *exec)
+static int run_connect(struct transport *transport, struct strbuf *cmdbuf)
{
struct helper_data *data = transport->data;
- struct strbuf cmdbuf = STRBUF_INIT;
- struct child_process *helper;
- int r, duped, ret = 0;
+ int ret = 0;
+ int duped;
FILE *input;
+ struct child_process *helper;
helper = get_helper(transport);
@@ -568,44 +567,54 @@ static int process_connect_service(struct transport *transport,
input = xfdopen(duped, "r");
setvbuf(input, NULL, _IONBF, 0);
+ sendline(data, cmdbuf);
+ if (recvline_fh(input, cmdbuf))
+ exit(128);
+
+ if (!strcmp(cmdbuf->buf, "")) {
+ data->no_disconnect_req = 1;
+ if (debug)
+ fprintf(stderr, "Debug: Smart transport connection "
+ "ready.\n");
+ ret = 1;
+ } else if (!strcmp(cmdbuf->buf, "fallback")) {
+ if (debug)
+ fprintf(stderr, "Debug: Falling back to dumb "
+ "transport.\n");
+ } else {
+ die("Unknown response to connect: %s",
+ cmdbuf->buf);
+ }
+
+ fclose(input);
+ return ret;
+}
+
+static int process_connect_service(struct transport *transport,
+ const char *name, const char *exec)
+{
+ struct helper_data *data = transport->data;
+ struct strbuf cmdbuf = STRBUF_INIT;
+ int ret = 0;
+
/*
* Handle --upload-pack and friends. This is fire and forget...
* just warn if it fails.
*/
if (strcmp(name, exec)) {
- r = set_helper_option(transport, "servpath", exec);
+ int r = set_helper_option(transport, "servpath", exec);
if (r > 0)
warning("Setting remote service path not supported by protocol.");
else if (r < 0)
warning("Invalid remote service path.");
}
- if (data->connect)
+ if (data->connect) {
strbuf_addf(&cmdbuf, "connect %s\n", name);
- else
- goto exit;
-
- sendline(data, &cmdbuf);
- if (recvline_fh(input, &cmdbuf))
- exit(128);
-
- if (!strcmp(cmdbuf.buf, "")) {
- data->no_disconnect_req = 1;
- if (debug)
- fprintf(stderr, "Debug: Smart transport connection "
- "ready.\n");
- ret = 1;
- } else if (!strcmp(cmdbuf.buf, "fallback")) {
- if (debug)
- fprintf(stderr, "Debug: Falling back to dumb "
- "transport.\n");
- } else
- die("Unknown response to connect: %s",
- cmdbuf.buf);
+ ret = run_connect(transport, &cmdbuf);
+ }
-exit:
strbuf_release(&cmdbuf);
- fclose(input);
return ret;
}
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 24/27] transport-helper: introduce stateless-connect
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (22 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 23/27] transport-helper: refactor process_connect_service Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 25/27] pkt-line: add packet_buf_write_len function Brandon Williams
` (5 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Introduce the transport-helper capability 'stateless-connect'. This
capability indicates that the transport-helper can be requested to run
the 'stateless-connect' command which should attempt to make a
stateless connection with a remote end. Once established, the
connection can be used by the git client to communicate with
the remote end natively in a stateless-rpc manner as supported by
protocol v2. This means that the client must send everything the server
needs in a single request as the client must not assume any
state-storing on the part of the server or transport.
If a stateless connection cannot be established then the remote-helper
will respond in the same manner as the 'connect' command indicating that
the client should fallback to using the dumb remote-helper commands.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
transport-helper.c | 8 ++++++++
transport.c | 1 +
transport.h | 6 ++++++
3 files changed, 15 insertions(+)
diff --git a/transport-helper.c b/transport-helper.c
index c032a2a87..82eb57c4a 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -26,6 +26,7 @@ struct helper_data {
option : 1,
push : 1,
connect : 1,
+ stateless_connect : 1,
signed_tags : 1,
check_connectivity : 1,
no_disconnect_req : 1,
@@ -188,6 +189,8 @@ static struct child_process *get_helper(struct transport *transport)
refspecs[refspec_nr++] = xstrdup(arg);
} else if (!strcmp(capname, "connect")) {
data->connect = 1;
+ } else if (!strcmp(capname, "stateless-connect")) {
+ data->stateless_connect = 1;
} else if (!strcmp(capname, "signed-tags")) {
data->signed_tags = 1;
} else if (skip_prefix(capname, "export-marks ", &arg)) {
@@ -612,6 +615,11 @@ static int process_connect_service(struct transport *transport,
if (data->connect) {
strbuf_addf(&cmdbuf, "connect %s\n", name);
ret = run_connect(transport, &cmdbuf);
+ } else if (data->stateless_connect) {
+ strbuf_addf(&cmdbuf, "stateless-connect %s\n", name);
+ ret = run_connect(transport, &cmdbuf);
+ if (ret)
+ transport->stateless_rpc = 1;
}
strbuf_release(&cmdbuf);
diff --git a/transport.c b/transport.c
index 4fdbd9adc..aafb8fbb4 100644
--- a/transport.c
+++ b/transport.c
@@ -250,6 +250,7 @@ static int fetch_refs_via_pack(struct transport *transport,
data->options.check_self_contained_and_connected;
args.cloning = transport->cloning;
args.update_shallow = data->options.update_shallow;
+ args.stateless_rpc = transport->stateless_rpc;
if (!data->got_remote_heads)
refs_tmp = get_refs_via_connect(transport, 0, NULL);
diff --git a/transport.h b/transport.h
index 4b656f315..9eac809ee 100644
--- a/transport.h
+++ b/transport.h
@@ -55,6 +55,12 @@ struct transport {
*/
unsigned cloning : 1;
+ /*
+ * Indicates that the transport is connected via a half-duplex
+ * connection and should operate in stateless-rpc mode.
+ */
+ unsigned stateless_rpc : 1;
+
/*
* These strings will be passed to the {pre, post}-receive hook,
* on the remote side, if both sides support the push options capability.
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 25/27] pkt-line: add packet_buf_write_len function
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (23 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 24/27] transport-helper: introduce stateless-connect Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 26/27] remote-curl: create copy of the service name Brandon Williams
` (4 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Add the 'packet_buf_write_len()' function which allows for writing an
arbitrary length buffer into a 'struct strbuf' and formatting it in
packet-line format.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
pkt-line.c | 16 ++++++++++++++++
pkt-line.h | 1 +
2 files changed, 17 insertions(+)
diff --git a/pkt-line.c b/pkt-line.c
index 726e109ca..5a8a17ecc 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -215,6 +215,22 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)
va_end(args);
}
+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len)
+{
+ size_t orig_len, n;
+
+ orig_len = buf->len;
+ strbuf_addstr(buf, "0000");
+ strbuf_add(buf, data, len);
+ n = buf->len - orig_len;
+
+ if (n > LARGE_PACKET_MAX)
+ die("protocol error: impossibly long line");
+
+ set_packet_header(&buf->buf[orig_len], n);
+ packet_trace(buf->buf + orig_len + 4, n - 4, 1);
+}
+
int write_packetized_from_fd(int fd_in, int fd_out)
{
static char buf[LARGE_PACKET_DATA_MAX];
diff --git a/pkt-line.h b/pkt-line.h
index 16fe8bdbf..63724d4bf 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -26,6 +26,7 @@ void packet_buf_flush(struct strbuf *buf);
void packet_buf_delim(struct strbuf *buf);
void packet_write(int fd_out, const char *buf, size_t size);
void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len);
int packet_flush_gently(int fd);
int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
int write_packetized_from_fd(int fd_in, int fd_out);
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 26/27] remote-curl: create copy of the service name
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (24 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 25/27] pkt-line: add packet_buf_write_len function Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-25 23:58 ` [PATCH v2 27/27] remote-curl: implement stateless-connect command Brandon Williams
` (3 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Make a copy of the service name being requested instead of relying on
the buffer pointed to by the passed in 'const char *' to remain
unchanged.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
remote-curl.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index dae8a4a48..4086aa733 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -165,7 +165,7 @@ static int set_option(const char *name, const char *value)
}
struct discovery {
- const char *service;
+ char *service;
char *buf_alloc;
char *buf;
size_t len;
@@ -257,6 +257,7 @@ static void free_discovery(struct discovery *d)
free(d->shallow.oid);
free(d->buf_alloc);
free_refs(d->refs);
+ free(d->service);
free(d);
}
}
@@ -343,7 +344,7 @@ static struct discovery *discover_refs(const char *service, int for_push)
warning(_("redirecting to %s"), url.buf);
last= xcalloc(1, sizeof(*last_discovery));
- last->service = service;
+ last->service = xstrdup(service);
last->buf_alloc = strbuf_detach(&buffer, &last->len);
last->buf = last->buf_alloc;
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* [PATCH v2 27/27] remote-curl: implement stateless-connect command
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (25 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 26/27] remote-curl: create copy of the service name Brandon Williams
@ 2018-01-25 23:58 ` Brandon Williams
2018-01-31 16:00 ` [PATCH v2 00/27] protocol version 2 Derrick Stolee
` (2 subsequent siblings)
29 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-01-25 23:58 UTC (permalink / raw)
To: git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder, Brandon Williams
Teach remote-curl the 'stateless-connect' command which is used to
establish a stateless connection with servers which support protocol
version 2. This allows remote-curl to act as a proxy, allowing the git
client to communicate natively with a remote end, simply using
remote-curl as a pass through to convert requests to http.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
remote-curl.c | 185 ++++++++++++++++++++++++++++++++++++++++++++++++-
t/t5702-protocol-v2.sh | 41 +++++++++++
2 files changed, 224 insertions(+), 2 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index 4086aa733..a17c7e228 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -171,6 +171,7 @@ struct discovery {
size_t len;
struct ref *refs;
struct oid_array shallow;
+ enum protocol_version version;
unsigned proto_git : 1;
};
static struct discovery *last_discovery;
@@ -184,9 +185,13 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
PACKET_READ_CHOMP_NEWLINE |
PACKET_READ_GENTLE_ON_EOF);
- switch (discover_version(&reader)) {
+ heads->version = discover_version(&reader);
+ switch (heads->version) {
case protocol_v2:
- die("support for protocol v2 not implemented yet");
+ /*
+ * Do nothing. Client should run 'stateless-connect' and
+ * request the refs themselves.
+ */
break;
case protocol_v1:
case protocol_v0:
@@ -1047,6 +1052,178 @@ static void parse_push(struct strbuf *buf)
free(specs);
}
+struct proxy_state {
+ char *service_name;
+ char *service_url;
+ struct curl_slist *headers;
+ struct strbuf request_buffer;
+ int in;
+ int out;
+ struct packet_reader reader;
+ size_t pos;
+ int seen_flush;
+};
+
+static void proxy_state_init(struct proxy_state *p, const char *service_name)
+{
+ struct strbuf buf = STRBUF_INIT;
+
+ memset(p, 0, sizeof(*p));
+ p->service_name = xstrdup(service_name);
+
+ p->in = 0;
+ p->out = 1;
+ strbuf_init(&p->request_buffer, 0);
+
+ strbuf_addf(&buf, "%s%s", url.buf, p->service_name);
+ p->service_url = strbuf_detach(&buf, NULL);
+
+ p->headers = http_copy_default_headers();
+
+ strbuf_addf(&buf, "Content-Type: application/x-%s-request", p->service_name);
+ p->headers = curl_slist_append(p->headers, buf.buf);
+ strbuf_reset(&buf);
+
+ strbuf_addf(&buf, "Accept: application/x-%s-result", p->service_name);
+ p->headers = curl_slist_append(p->headers, buf.buf);
+
+ p->headers = curl_slist_append(p->headers, "Transfer-Encoding: chunked");
+
+ packet_reader_init(&p->reader, p->in, NULL, 0,
+ PACKET_READ_GENTLE_ON_EOF);
+
+ strbuf_release(&buf);
+}
+
+static void proxy_state_clear(struct proxy_state *p)
+{
+ free(p->service_name);
+ free(p->service_url);
+ curl_slist_free_all(p->headers);
+ strbuf_release(&p->request_buffer);
+}
+
+static size_t proxy_in(char *buffer, size_t eltsize,
+ size_t nmemb, void *userdata)
+{
+ size_t max = eltsize * nmemb;
+ struct proxy_state *p = userdata;
+ size_t avail = p->request_buffer.len - p->pos;
+
+ if (!avail) {
+ if (p->seen_flush) {
+ p->seen_flush = 0;
+ return 0;
+ }
+
+ strbuf_reset(&p->request_buffer);
+ switch (packet_reader_read(&p->reader)) {
+ case PACKET_READ_EOF:
+ die("unexpected EOF when reading from parent process");
+ case PACKET_READ_NORMAL:
+ packet_buf_write_len(&p->request_buffer, p->reader.line,
+ p->reader.pktlen);
+ break;
+ case PACKET_READ_DELIM:
+ packet_buf_delim(&p->request_buffer);
+ break;
+ case PACKET_READ_FLUSH:
+ packet_buf_flush(&p->request_buffer);
+ p->seen_flush = 1;
+ break;
+ }
+ p->pos = 0;
+ avail = p->request_buffer.len;
+ }
+
+ if (max < avail)
+ avail = max;
+ memcpy(buffer, p->request_buffer.buf + p->pos, avail);
+ p->pos += avail;
+ return avail;
+}
+
+static size_t proxy_out(char *buffer, size_t eltsize,
+ size_t nmemb, void *userdata)
+{
+ size_t size = eltsize * nmemb;
+ struct proxy_state *p = userdata;
+
+ write_or_die(p->out, buffer, size);
+ return size;
+}
+
+static int proxy_post(struct proxy_state *p)
+{
+ struct active_request_slot *slot;
+ int err;
+
+ slot = get_active_slot();
+
+ curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
+ curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, p->headers);
+
+ /* Setup function to read request from client */
+ curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);
+ curl_easy_setopt(slot->curl, CURLOPT_READDATA, p);
+
+ /* Setup function to write server response to client */
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);
+
+ err = run_slot(slot, NULL);
+
+ if (err != HTTP_OK)
+ err = -1;
+
+ return err;
+}
+
+static int stateless_connect(const char *service_name)
+{
+ struct discovery *discover;
+ struct proxy_state p;
+
+ /*
+ * Run the info/refs request and see if the server supports protocol
+ * v2. If and only if the server supports v2 can we successfully
+ * establish a stateless connection, otherwise we need to tell the
+ * client to fallback to using other transport helper functions to
+ * complete their request.
+ */
+ discover = discover_refs(service_name, 0);
+ if (discover->version != protocol_v2) {
+ printf("fallback\n");
+ fflush(stdout);
+ return -1;
+ } else {
+ /* Stateless Connection established */
+ printf("\n");
+ fflush(stdout);
+ }
+
+ proxy_state_init(&p, service_name);
+
+ /*
+ * Dump the capability listing that we got from the server earlier
+ * during the info/refs request.
+ */
+ write_or_die(p.out, discover->buf, discover->len);
+
+ /* Peek the next packet line. Until we see EOF keep sending POSTs */
+ while (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {
+ if (proxy_post(&p)) {
+ /* We would have an err here */
+ break;
+ }
+ }
+
+ proxy_state_clear(&p);
+ return 0;
+}
+
int cmd_main(int argc, const char **argv)
{
struct strbuf buf = STRBUF_INIT;
@@ -1115,12 +1292,16 @@ int cmd_main(int argc, const char **argv)
fflush(stdout);
} else if (!strcmp(buf.buf, "capabilities")) {
+ printf("stateless-connect\n");
printf("fetch\n");
printf("option\n");
printf("push\n");
printf("check-connectivity\n");
printf("\n");
fflush(stdout);
+ } else if (skip_prefix(buf.buf, "stateless-connect ", &arg)) {
+ if (!stateless_connect(arg))
+ break;
} else {
error("remote-curl: unknown command '%s' from git", buf.buf);
return 1;
diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 3e411e178..ada69ac09 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -73,4 +73,45 @@ test_expect_success 'ref advertisment is filtered during fetch using protocol v2
! grep "refs/tags/" log
'
+# Test protocol v2 with 'http://' transport
+#
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
+
+test_expect_success 'create repo to be served by http:// transport' '
+ git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+ git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config http.receivepack true &&
+ test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" one
+'
+
+test_expect_success 'clone with http:// using protocol v2' '
+ GIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=2 \
+ clone "$HTTPD_URL/smart/http_parent" http_child 2>log &&
+
+ git -C http_child log -1 --format=%s >actual &&
+ git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" log -1 --format=%s >expect &&
+ test_cmp expect actual &&
+
+ # Client requested to use protocol v2
+ grep "Git-Protocol: version=2" log &&
+ # Server responded using protocol v2
+ grep "git< version 2" log
+'
+
+test_expect_success 'fetch with http:// using protocol v2' '
+ test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" two &&
+
+ GIT_TRACE_PACKET=1 git -C http_child -c protocol.version=2 \
+ fetch 2>log &&
+
+ git -C http_child log -1 --format=%s origin/master >actual &&
+ git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" log -1 --format=%s >expect &&
+ test_cmp expect actual &&
+
+ # Server responded using protocol v2
+ grep "git< version 2" log
+'
+
+stop_httpd
+
test_done
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v2 00/27] protocol version 2
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (26 preceding siblings ...)
2018-01-25 23:58 ` [PATCH v2 27/27] remote-curl: implement stateless-connect command Brandon Williams
@ 2018-01-31 16:00 ` Derrick Stolee
2018-02-07 0:58 ` Brandon Williams
2018-02-01 19:40 ` Jeff Hostetler
2018-02-07 1:12 ` [PATCH v3 00/35] " Brandon Williams
29 siblings, 1 reply; 362+ messages in thread
From: Derrick Stolee @ 2018-01-31 16:00 UTC (permalink / raw)
To: Brandon Williams, git; +Cc: sbeller, gitster, peff, philipoakley, jrnieder
Sorry for chiming in with mostly nitpicks so late since sending this
version. Mostly, I tried to read it to see if I could understand the
scope of the patch and how this code worked before. It looks very
polished, so I the nits were the best I could do.
On 1/25/2018 6:58 PM, Brandon Williams wrote:
> Changes in v2:
> * Added documentation for fetch
> * changes #defines for state variables to be enums
> * couple code changes to pkt-line functions and documentation
> * Added unit tests for the git-serve binary as well as for ls-refs
I'm a fan of more unit-level testing, and I think that will be more
important as we go on with these multiple configuration options.
> Areas for improvement
> * Push isn't implemented, right now this is ok because if v2 is requested the
> server can just default to v0. Before this can be merged we may want to
> change how the client request a new protocol, and not allow for sending
> "version=2" when pushing even though the user has it configured. Or maybe
> its fine to just have an older client who doesn't understand how to push
> (and request v2) to die if the server tries to speak v2 at it.
>
> Fixing this essentially would just require piping through a bit more
> information to the function which ultimately runs connect (for both builtins
> and remote-curl)
Definitely save push for a later patch. Getting 'fetch' online did
require 'ls-refs' at the same time. Future reviews will be easier when
adding one command at a time.
>
> * I want to make sure that the docs are well written before this gets merged
> so I'm hoping that someone can do a through review on the docs themselves to
> make sure they are clear.
I made a comment in the docs about the architectural changes. While I
think a discussion on that topic would be valuable, I'm not sure that's
the point of the document (i.e. documenting what v2 does versus selling
the value of the patch). I thought the docs were clear for how the
commands work.
> * Right now there is a capability 'stateless-rpc' which essentially makes sure
> that a server command completes after a single round (this is to make sure
> http works cleanly). After talking with some folks it may make more sense
> to just have v2 be stateless in nature so that all commands terminate after
> a single round trip. This makes things a bit easier if a server wants to
> have ssh just be a proxy for http.
>
> One potential thing would be to flip this so that by default the protocol is
> stateless and if a server/command has a state-full mode that can be
> implemented as a capability at a later point. Thoughts?
At minimum, all commands should be designed with a "stateless first"
philosophy since a large number of users communicate via HTTP[S] and any
decisions that make stateless communication painful should be rejected.
> * Shallow repositories and shallow clones aren't supported yet. I'm working
> on it and it can be either added to v2 by default if people think it needs
> to be in there from the start, or we can add it as a capability at a later
> point.
I'm happy to say the following:
1. Shallow repositories should not be used for servers, since they
cannot service all requests.
2. Since v2 has easy capability features, I'm happy to leave shallow for
later. We will want to verify that a shallow clone command reverts to v1.
I fetched bw/protocol-v2 with tip 13c70148, built, set
'protocol.version=2' in the config, and tested fetches against GitHub
and VSTS just as a compatibility test. Everything worked just fine.
Is there an easy way to test the existing test suite for clone and fetch
using protocol v2 to make sure there are no regressions with
protocol.version=2 in the config?
Thanks,
-Stolee
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v2 00/27] protocol version 2
2018-01-31 16:00 ` [PATCH v2 00/27] protocol version 2 Derrick Stolee
@ 2018-02-07 0:58 ` Brandon Williams
0 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-02-07 0:58 UTC (permalink / raw)
To: Derrick Stolee; +Cc: git, sbeller, gitster, peff, philipoakley, jrnieder
On 01/31, Derrick Stolee wrote:
> Sorry for chiming in with mostly nitpicks so late since sending this
> version. Mostly, I tried to read it to see if I could understand the scope
> of the patch and how this code worked before. It looks very polished, so I
> the nits were the best I could do.
>
> On 1/25/2018 6:58 PM, Brandon Williams wrote:
> > Changes in v2:
> > * Added documentation for fetch
> > * changes #defines for state variables to be enums
> > * couple code changes to pkt-line functions and documentation
> > * Added unit tests for the git-serve binary as well as for ls-refs
>
> I'm a fan of more unit-level testing, and I think that will be more
> important as we go on with these multiple configuration options.
>
> > Areas for improvement
> > * Push isn't implemented, right now this is ok because if v2 is requested the
> > server can just default to v0. Before this can be merged we may want to
> > change how the client request a new protocol, and not allow for sending
> > "version=2" when pushing even though the user has it configured. Or maybe
> > its fine to just have an older client who doesn't understand how to push
> > (and request v2) to die if the server tries to speak v2 at it.
> >
> > Fixing this essentially would just require piping through a bit more
> > information to the function which ultimately runs connect (for both builtins
> > and remote-curl)
>
> Definitely save push for a later patch. Getting 'fetch' online did require
> 'ls-refs' at the same time. Future reviews will be easier when adding one
> command at a time.
>
> >
> > * I want to make sure that the docs are well written before this gets merged
> > so I'm hoping that someone can do a through review on the docs themselves to
> > make sure they are clear.
>
> I made a comment in the docs about the architectural changes. While I think
> a discussion on that topic would be valuable, I'm not sure that's the point
> of the document (i.e. documenting what v2 does versus selling the value of
> the patch). I thought the docs were clear for how the commands work.
>
> > * Right now there is a capability 'stateless-rpc' which essentially makes sure
> > that a server command completes after a single round (this is to make sure
> > http works cleanly). After talking with some folks it may make more sense
> > to just have v2 be stateless in nature so that all commands terminate after
> > a single round trip. This makes things a bit easier if a server wants to
> > have ssh just be a proxy for http.
> >
> > One potential thing would be to flip this so that by default the protocol is
> > stateless and if a server/command has a state-full mode that can be
> > implemented as a capability at a later point. Thoughts?
>
> At minimum, all commands should be designed with a "stateless first"
> philosophy since a large number of users communicate via HTTP[S] and any
> decisions that make stateless communication painful should be rejected.
I agree with this and my next version will run with this philosophy in
mind (v2 will be stateless by default).
>
> > * Shallow repositories and shallow clones aren't supported yet. I'm working
> > on it and it can be either added to v2 by default if people think it needs
> > to be in there from the start, or we can add it as a capability at a later
> > point.
>
> I'm happy to say the following:
>
> 1. Shallow repositories should not be used for servers, since they cannot
> service all requests.
>
> 2. Since v2 has easy capability features, I'm happy to leave shallow for
> later. We will want to verify that a shallow clone command reverts to v1.
>
>
> I fetched bw/protocol-v2 with tip 13c70148, built, set 'protocol.version=2'
> in the config, and tested fetches against GitHub and VSTS just as a
> compatibility test. Everything worked just fine.
>
> Is there an easy way to test the existing test suite for clone and fetch
> using protocol v2 to make sure there are no regressions with
> protocol.version=2 in the config?
Yes there already exist interop tests for testing the addition of
requesting a new protocol at //t/interop/i5700-protocol-transition.sh
>
> Thanks,
> -Stolee
--
Brandon Williams
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v2 00/27] protocol version 2
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (27 preceding siblings ...)
2018-01-31 16:00 ` [PATCH v2 00/27] protocol version 2 Derrick Stolee
@ 2018-02-01 19:40 ` Jeff Hostetler
2018-02-07 1:12 ` [PATCH v3 00/35] " Brandon Williams
29 siblings, 0 replies; 362+ messages in thread
From: Jeff Hostetler @ 2018-02-01 19:40 UTC (permalink / raw)
To: Brandon Williams, git
Cc: sbeller, gitster, peff, philipoakley, stolee, jrnieder
On 1/25/2018 6:58 PM, Brandon Williams wrote:
> Changes in v2:
> * Added documentation for fetch
> * changes #defines for state variables to be enums
> * couple code changes to pkt-line functions and documentation
> * Added unit tests for the git-serve binary as well as for ls-refs
[...]
This looks really nice. I'm eager to get this in so we can do some
additional commands to help make partial clone more efficient.
Thanks,
Jeff
^ permalink raw reply [flat|nested] 362+ messages in thread* [PATCH v3 00/35] protocol version 2
2018-01-25 23:58 ` [PATCH v2 00/27] " Brandon Williams
` (28 preceding siblings ...)
2018-02-01 19:40 ` Jeff Hostetler
@ 2018-02-07 1:12 ` Brandon Williams
2018-02-07 1:12 ` [PATCH v3 01/35] pkt-line: introduce packet_read_with_status Brandon Williams
` (37 more replies)
29 siblings, 38 replies; 362+ messages in thread
From: Brandon Williams @ 2018-02-07 1:12 UTC (permalink / raw)
To: git
Cc: sbeller, peff, gitster, jrnieder, stolee, git, pclouds, Brandon Williams
Changes in v3:
* There were some comments about how the protocol should be designed
stateless first. I've made this change and instead of having to
supply the `stateless-rpc=true` capability to force stateless
behavior, the protocol just requires all commands to be stateless.
* Added some patches towards the end of the series to force the client
to not request to use protocol v2 when pushing (even if configured to
use v2). This is to ease the roll-out process of a push command in
protocol v2. This way when servers gain the ability to accept
pushing in v2 (and they start responding using v2 when requests are
sent to the git-receive-pack endpoint) that clients who still don't
understand how to push using v2 won't request to use v2 and then die
when they recognize that the server does indeed know how to accept a
push under v2.
* I implemented the `shallow` feature for fetch. This feature
encapsulates the existing functionality of all the shallow/deepen
capabilities in v0. So now a server can process shallow requests.
* Various other small tweaks that I can't remember :)
After all of that I think the series is in a pretty good state, baring
any more critical reviewing feedback.
Thanks!
Brandon Williams (35):
pkt-line: introduce packet_read_with_status
pkt-line: introduce struct packet_reader
pkt-line: add delim packet support
upload-pack: convert to a builtin
upload-pack: factor out processing lines
transport: use get_refs_via_connect to get refs
connect: convert get_remote_heads to use struct packet_reader
connect: discover protocol version outside of get_remote_heads
transport: store protocol version
protocol: introduce enum protocol_version value protocol_v2
test-pkt-line: introduce a packet-line test helper
serve: introduce git-serve
ls-refs: introduce ls-refs server command
connect: request remote refs using v2
transport: convert get_refs_list to take a list of ref patterns
transport: convert transport_get_remote_refs to take a list of ref
patterns
ls-remote: pass ref patterns when requesting a remote's refs
fetch: pass ref patterns when fetching
push: pass ref patterns when pushing
upload-pack: introduce fetch server command
fetch-pack: perform a fetch using v2
upload-pack: support shallow requests
fetch-pack: support shallow requests
connect: refactor git_connect to only get the protocol version once
connect: don't request v2 when pushing
transport-helper: remove name parameter
transport-helper: refactor process_connect_service
transport-helper: introduce stateless-connect
pkt-line: add packet_buf_write_len function
remote-curl: create copy of the service name
remote-curl: store the protocol version the server responded with
http: allow providing extra headers for http requests
http: don't always add Git-Protocol header
remote-curl: implement stateless-connect command
remote-curl: don't request v2 when pushing
.gitignore | 1 +
Documentation/technical/protocol-v2.txt | 338 +++++++++++++++++
Makefile | 7 +-
builtin.h | 2 +
builtin/clone.c | 2 +-
builtin/fetch-pack.c | 21 +-
builtin/fetch.c | 14 +-
builtin/ls-remote.c | 7 +-
builtin/receive-pack.c | 6 +
builtin/remote.c | 2 +-
builtin/send-pack.c | 20 +-
builtin/serve.c | 30 ++
builtin/upload-pack.c | 74 ++++
connect.c | 352 +++++++++++++-----
connect.h | 7 +
fetch-pack.c | 319 +++++++++++++++-
fetch-pack.h | 4 +-
git.c | 2 +
http.c | 25 +-
http.h | 2 +
ls-refs.c | 96 +++++
ls-refs.h | 9 +
pkt-line.c | 149 +++++++-
pkt-line.h | 77 ++++
protocol.c | 2 +
protocol.h | 1 +
remote-curl.c | 257 ++++++++++++-
remote.h | 9 +-
serve.c | 260 +++++++++++++
serve.h | 15 +
t/helper/test-pkt-line.c | 64 ++++
t/t5701-git-serve.sh | 176 +++++++++
t/t5702-protocol-v2.sh | 239 ++++++++++++
transport-helper.c | 84 +++--
transport-internal.h | 4 +-
transport.c | 116 ++++--
transport.h | 9 +-
upload-pack.c | 625 ++++++++++++++++++++++++--------
upload-pack.h | 21 ++
39 files changed, 3088 insertions(+), 360 deletions(-)
create mode 100644 Documentation/technical/protocol-v2.txt
create mode 100644 builtin/serve.c
create mode 100644 builtin/upload-pack.c
create mode 100644 ls-refs.c
create mode 100644 ls-refs.h
create mode 100644 serve.c
create mode 100644 serve.h
create mode 100644 t/helper/test-pkt-line.c
create mode 100755 t/t5701-git-serve.sh
create mode 100755 t/t5702-protocol-v2.sh
create mode 100644 upload-pack.h
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply [flat|nested] 362+ messages in thread* [PATCH v3 01/35] pkt-line: introduce packet_read_with_status
2018-02-07 1:12 ` [PATCH v3 00/35] " Brandon Williams
@ 2018-02-07 1:12 ` Brandon Williams
2018-02-13 0:25 ` Jonathan Nieder
2018-02-07 1:12 ` [PATCH v3 02/35] pkt-line: introduce struct packet_reader Brandon Williams
` (36 subsequent siblings)
37 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-02-07 1:12 UTC (permalink / raw)
To: git
Cc: sbeller, peff, gitster, jrnieder, stolee, git, pclouds, Brandon Williams
The current pkt-line API encodes the status of a pkt-line read in the
length of the read content. An error is indicated with '-1', a flush
with '0' (which can be confusing since a return value of '0' can also
indicate an empty pkt-line), and a positive integer for the length of
the read content otherwise. This doesn't leave much room for allowing
the addition of additional special packets in the future.
To solve this introduce 'packet_read_with_status()' which reads a packet
and returns the status of the read encoded as an 'enum packet_status'
type. This allows for easily identifying between special and normal
packets as well as errors. It also enables easily adding a new special
packet in the future.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
pkt-line.c | 57 +++++++++++++++++++++++++++++++++++++++++++--------------
pkt-line.h | 15 +++++++++++++++
2 files changed, 58 insertions(+), 14 deletions(-)
diff --git a/pkt-line.c b/pkt-line.c
index 2827ca772..af0d2430f 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -280,28 +280,33 @@ static int packet_length(const char *linelen)
return (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);
}
-int packet_read(int fd, char **src_buf, size_t *src_len,
- char *buffer, unsigned size, int options)
+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
+ char *buffer, unsigned size, int *pktlen,
+ int options)
{
- int len, ret;
+ int len;
char linelen[4];
- ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);
- if (ret < 0)
- return ret;
+ if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)
+ return PACKET_READ_EOF;
+
len = packet_length(linelen);
- if (len < 0)
+
+ if (len < 0) {
die("protocol error: bad line length character: %.4s", linelen);
- if (!len) {
+ } else if (!len) {
packet_trace("0000", 4, 0);
- return 0;
+ return PACKET_READ_FLUSH;
+ } else if (len < 4) {
+ die("protocol error: bad line length %d", len);
}
+
len -= 4;
- if (len >= size)
+ if ((unsigned)len >= size)
die("protocol error: bad line length %d", len);
- ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);
- if (ret < 0)
- return ret;
+
+ if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)
+ return PACKET_READ_EOF;
if ((options & PACKET_READ_CHOMP_NEWLINE) &&
len && buffer[len-1] == '\n')
@@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,
buffer[len] = 0;
packet_trace(buffer, len, 0);
- return len;
+ *pktlen = len;
+ return PACKET_READ_NORMAL;
+}
+
+int packet_read(int fd, char **src_buffer, size_t *src_len,
+ char *buffer, unsigned size, int options)
+{
+ enum packet_read_status status;
+ int pktlen;
+
+ status = packet_read_with_status(fd, src_buffer, src_len,
+ buffer, size, &pktlen,
+ options);
+ switch (status) {
+ case PACKET_READ_EOF:
+ pktlen = -1;
+ break;
+ case PACKET_READ_NORMAL:
+ break;
+ case PACKET_READ_FLUSH:
+ pktlen = 0;
+ break;
+ }
+
+ return pktlen;
}
static char *packet_read_line_generic(int fd,
diff --git a/pkt-line.h b/pkt-line.h
index 3dad583e2..06c468927 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);
int packet_read(int fd, char **src_buffer, size_t *src_len, char
*buffer, unsigned size, int options);
+/*
+ * Read a packetized line into a buffer like the 'packet_read()' function but
+ * returns an 'enum packet_read_status' which indicates the status of the read.
+ * The number of bytes read will be assigined to *pktlen if the status of the
+ * read was 'PACKET_READ_NORMAL'.
+ */
+enum packet_read_status {
+ PACKET_READ_EOF = -1,
+ PACKET_READ_NORMAL,
+ PACKET_READ_FLUSH,
+};
+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
+ char *buffer, unsigned size, int *pktlen,
+ int options);
+
/*
* Convenience wrapper for packet_read that is not gentle, and sets the
* CHOMP_NEWLINE option. The return value is NULL for a flush packet,
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v3 01/35] pkt-line: introduce packet_read_with_status
2018-02-07 1:12 ` [PATCH v3 01/35] pkt-line: introduce packet_read_with_status Brandon Williams
@ 2018-02-13 0:25 ` Jonathan Nieder
0 siblings, 0 replies; 362+ messages in thread
From: Jonathan Nieder @ 2018-02-13 0:25 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, peff, gitster, stolee, git, pclouds
Hi,
Brandon Williams wrote:
> The current pkt-line API encodes the status of a pkt-line read in the
> length of the read content. An error is indicated with '-1', a flush
> with '0' (which can be confusing since a return value of '0' can also
> indicate an empty pkt-line), and a positive integer for the length of
> the read content otherwise. This doesn't leave much room for allowing
> the addition of additional special packets in the future.
>
> To solve this introduce 'packet_read_with_status()' which reads a packet
> and returns the status of the read encoded as an 'enum packet_status'
> type. This allows for easily identifying between special and normal
> packets as well as errors. It also enables easily adding a new special
> packet in the future.
Makes sense, thanks. Using an enum return value is less opaque, too.
[...]
> --- a/pkt-line.c
> +++ b/pkt-line.c
> @@ -280,28 +280,33 @@ static int packet_length(const char *linelen)
> return (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);
> }
>
> -int packet_read(int fd, char **src_buf, size_t *src_len,
> - char *buffer, unsigned size, int options)
> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
> + char *buffer, unsigned size, int *pktlen,
> + int options)
This function definition straddles two worlds: it is line-wrapped as
though there are a limited number of columns, but it goes far past 80
columns.
Can "make style" or a similar tool take care of rewrapping it?
> {
> - int len, ret;
> + int len;
> char linelen[4];
>
> - ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);
> - if (ret < 0)
> - return ret;
> + if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)
> + return PACKET_READ_EOF;
> +
EOF is indeed the only error that get_packet_data can return.
Could be worth a doc comment on get_packet_data to make that clearer.
It's not too important since it's static, though.
> len = packet_length(linelen);
> - if (len < 0)
> +
> + if (len < 0) {
> die("protocol error: bad line length character: %.4s", linelen);
> - if (!len) {
> + } else if (!len) {
> packet_trace("0000", 4, 0);
> - return 0;
> + return PACKET_READ_FLUSH;
The advertised change. Makes sense.
[...]
> - if (len >= size)
> + if ((unsigned)len >= size)
> die("protocol error: bad line length %d", len);
The comparison is safe since we just checked that len >= 0.
Is there some static analysis that can make this kind of operation
easier?
[...]
> @@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,
>
> buffer[len] = 0;
> packet_trace(buffer, len, 0);
> - return len;
> + *pktlen = len;
> + return PACKET_READ_NORMAL;
> +}
> +
> +int packet_read(int fd, char **src_buffer, size_t *src_len,
> + char *buffer, unsigned size, int options)
> +{
> + enum packet_read_status status;
> + int pktlen;
> +
> + status = packet_read_with_status(fd, src_buffer, src_len,
> + buffer, size, &pktlen,
> + options);
> + switch (status) {
> + case PACKET_READ_EOF:
> + pktlen = -1;
> + break;
> + case PACKET_READ_NORMAL:
> + break;
> + case PACKET_READ_FLUSH:
> + pktlen = 0;
> + break;
> + }
> +
> + return pktlen;
nit: can simplify by avoiding the status temporary:
int pktlen;
switch (packet_read_with_status(...)) {
case PACKET_READ_EOF:
return -1;
case PACKET_READ_FLUSH:
return 0;
case PACKET_READ_NORMAL:
return pktlen;
}
As a bonus, that lets static analyzers check that the cases are
exhaustive. (On the other hand, C doesn't guarantee that an enum can
only have the values listed as enumerators. Did we end up figuring
out a way to handle that, beyond always including a 'default: BUG()'?)
> --- a/pkt-line.h
> +++ b/pkt-line.h
> @@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);
> int packet_read(int fd, char **src_buffer, size_t *src_len, char
> *buffer, unsigned size, int options);
>
> +/*
> + * Read a packetized line into a buffer like the 'packet_read()' function but
> + * returns an 'enum packet_read_status' which indicates the status of the read.
> + * The number of bytes read will be assigined to *pktlen if the status of the
> + * read was 'PACKET_READ_NORMAL'.
> + */
> +enum packet_read_status {
> + PACKET_READ_EOF = -1,
> + PACKET_READ_NORMAL,
> + PACKET_READ_FLUSH,
> +};
nit: do any callers treat the return value as a number? It would be
less magical if the numbering were left to the compiler (0, 1, 2).
Thanks,
Jonathan
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v3 02/35] pkt-line: introduce struct packet_reader
2018-02-07 1:12 ` [PATCH v3 00/35] " Brandon Williams
2018-02-07 1:12 ` [PATCH v3 01/35] pkt-line: introduce packet_read_with_status Brandon Williams
@ 2018-02-07 1:12 ` Brandon Williams
2018-02-13 0:49 ` Jonathan Nieder
2018-02-27 5:57 ` Jonathan Nieder
2018-02-07 1:12 ` [PATCH v3 03/35] pkt-line: add delim packet support Brandon Williams
` (35 subsequent siblings)
37 siblings, 2 replies; 362+ messages in thread
From: Brandon Williams @ 2018-02-07 1:12 UTC (permalink / raw)
To: git
Cc: sbeller, peff, gitster, jrnieder, stolee, git, pclouds, Brandon Williams
Sometimes it is advantageous to be able to peek the next packet line
without consuming it (e.g. to be able to determine the protocol version
a server is speaking). In order to do that introduce 'struct
packet_reader' which is an abstraction around the normal packet reading
logic. This enables a caller to be able to peek a single line at a time
using 'packet_reader_peek()' and having a caller consume a line by
calling 'packet_reader_read()'.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
pkt-line.h | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 117 insertions(+)
diff --git a/pkt-line.c b/pkt-line.c
index af0d2430f..4fc9ad4b0 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -406,3 +406,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)
}
return sb_out->len - orig_len;
}
+
+/* Packet Reader Functions */
+void packet_reader_init(struct packet_reader *reader, int fd,
+ char *src_buffer, size_t src_len,
+ int options)
+{
+ memset(reader, 0, sizeof(*reader));
+
+ reader->fd = fd;
+ reader->src_buffer = src_buffer;
+ reader->src_len = src_len;
+ reader->buffer = packet_buffer;
+ reader->buffer_size = sizeof(packet_buffer);
+ reader->options = options;
+}
+
+enum packet_read_status packet_reader_read(struct packet_reader *reader)
+{
+ if (reader->line_peeked) {
+ reader->line_peeked = 0;
+ return reader->status;
+ }
+
+ reader->status = packet_read_with_status(reader->fd,
+ &reader->src_buffer,
+ &reader->src_len,
+ reader->buffer,
+ reader->buffer_size,
+ &reader->pktlen,
+ reader->options);
+
+ switch (reader->status) {
+ case PACKET_READ_EOF:
+ reader->pktlen = -1;
+ reader->line = NULL;
+ break;
+ case PACKET_READ_NORMAL:
+ reader->line = reader->buffer;
+ break;
+ case PACKET_READ_FLUSH:
+ reader->pktlen = 0;
+ reader->line = NULL;
+ break;
+ }
+
+ return reader->status;
+}
+
+enum packet_read_status packet_reader_peek(struct packet_reader *reader)
+{
+ /* Only allow peeking a single line */
+ if (reader->line_peeked)
+ return reader->status;
+
+ /* Peek a line by reading it and setting peeked flag */
+ packet_reader_read(reader);
+ reader->line_peeked = 1;
+ return reader->status;
+}
diff --git a/pkt-line.h b/pkt-line.h
index 06c468927..7d9f0e537 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
*/
ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
+struct packet_reader {
+ /* source file descriptor */
+ int fd;
+
+ /* source buffer and its size */
+ char *src_buffer;
+ size_t src_len;
+
+ /* buffer that pkt-lines are read into and its size */
+ char *buffer;
+ unsigned buffer_size;
+
+ /* options to be used during reads */
+ int options;
+
+ /* status of the last read */
+ enum packet_read_status status;
+
+ /* length of data read during the last read */
+ int pktlen;
+
+ /* the last line read */
+ const char *line;
+
+ /* indicates if a line has been peeked */
+ int line_peeked;
+};
+
+/*
+ * Initialize a 'struct packet_reader' object which is an
+ * abstraction around the 'packet_read_with_status()' function.
+ */
+extern void packet_reader_init(struct packet_reader *reader, int fd,
+ char *src_buffer, size_t src_len,
+ int options);
+
+/*
+ * Perform a packet read and return the status of the read.
+ * The values of 'pktlen' and 'line' are updated based on the status of the
+ * read as follows:
+ *
+ * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL
+ * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read
+ * 'line' is set to point at the read line
+ * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL
+ */
+extern enum packet_read_status packet_reader_read(struct packet_reader *reader);
+
+/*
+ * Peek the next packet line without consuming it and return the status.
+ * The next call to 'packet_reader_read()' will perform a read of the same line
+ * that was peeked, consuming the line.
+ *
+ * Peeking multiple times without calling 'packet_reader_read()' will return
+ * the same result.
+ */
+extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);
+
#define DEFAULT_PACKET_MAX 1000
#define LARGE_PACKET_MAX 65520
#define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader
2018-02-07 1:12 ` [PATCH v3 02/35] pkt-line: introduce struct packet_reader Brandon Williams
@ 2018-02-13 0:49 ` Jonathan Nieder
2018-02-27 18:14 ` Brandon Williams
2018-02-27 5:57 ` Jonathan Nieder
1 sibling, 1 reply; 362+ messages in thread
From: Jonathan Nieder @ 2018-02-13 0:49 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, peff, gitster, stolee, git, pclouds
Hi,
Brandon Williams wrote:
> Subject: pkt-line: introduce struct packet_reader
nit: this subject line doesn't describe what the purpose/intent behind
the patch is. Maybe something like
pkt-line: allow peeking at a packet line without consuming it
would make it clearer.
> Sometimes it is advantageous to be able to peek the next packet line
> without consuming it (e.g. to be able to determine the protocol version
> a server is speaking). In order to do that introduce 'struct
> packet_reader' which is an abstraction around the normal packet reading
> logic. This enables a caller to be able to peek a single line at a time
> using 'packet_reader_peek()' and having a caller consume a line by
> calling 'packet_reader_read()'.
Makes sense. The packet_reader owns a buffer to support the peek
operation and make buffer reuse a little easier.
[...]
> --- a/pkt-line.h
> +++ b/pkt-line.h
> @@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
> */
> ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
>
> +struct packet_reader {
> + /* source file descriptor */
> + int fd;
> +
> + /* source buffer and its size */
> + char *src_buffer;
> + size_t src_len;
Can or should this be a strbuf?
> +
> + /* buffer that pkt-lines are read into and its size */
> + char *buffer;
> + unsigned buffer_size;
Likewise.
> +
> + /* options to be used during reads */
> + int options;
What option values are possible?
> +
> + /* status of the last read */
> + enum packet_read_status status;
This reminds me of FILE*'s status value, ferror, etc. I'm mildly
nervous about it --- it encourages a style of error handling where you
ignore errors from an individual operation and hope the recorded
status later has the most relevant error.
I think it is being used to support peek --- you need to record the
error to reply it. Is that right? I wonder if it would make sense to
structure as
struct packet_read_result last_line_read;
};
struct packet_read_result {
enum packet_read_status status;
const char *line;
int len;
};
What you have here also seems fine. I think what would help most
for readability is if the comment mentioned the purpose --- e.g.
/* status of the last read, to support peeking */
Or if the contract were tied to the purpose:
/* status of the last read, only valid if line_peeked is true */
[...]
> +/*
> + * Initialize a 'struct packet_reader' object which is an
> + * abstraction around the 'packet_read_with_status()' function.
> + */
> +extern void packet_reader_init(struct packet_reader *reader, int fd,
> + char *src_buffer, size_t src_len,
> + int options);
This comment doesn't describe how I should use the function. Is the
intent to point the reader to packet_read_with_status for more details
about the arguments?
Can src_buffer be a const char *?
[...]
> +/*
> + * Perform a packet read and return the status of the read.
nit: s/Perform a packet read/Read one pkt-line/
> + * The values of 'pktlen' and 'line' are updated based on the status of the
> + * read as follows:
> + *
> + * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL
> + * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read
> + * 'line' is set to point at the read line
> + * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL
> + */
> +extern enum packet_read_status packet_reader_read(struct packet_reader *reader);
This is reasonable. As described above an alternative would be
possible to have a separate packet_read_result output parameter but
the interface described here looks pretty easy/pleasant to use.
> +
> +/*
> + * Peek the next packet line without consuming it and return the status.
nit: s/Peek/Peek at/, or s/Peek/Read/
> + * The next call to 'packet_reader_read()' will perform a read of the same line
> + * that was peeked, consuming the line.
nit: s/peeked/peeked at/
> + *
> + * Peeking multiple times without calling 'packet_reader_read()' will return
> + * the same result.
> + */
> +extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);
Nice.
[...]
> --- a/pkt-line.c
> +++ b/pkt-line.c
> @@ -406,3 +406,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)
> }
> return sb_out->len - orig_len;
> }
> +
> +/* Packet Reader Functions */
> +void packet_reader_init(struct packet_reader *reader, int fd,
> + char *src_buffer, size_t src_len,
> + int options)
This comment looks like it's attached to packet_reader_init, but it's
meant to be attached to the whole collection. It's possible that this
title-above-multiple-functions won't be maintained, but that's okay.
> +{
> + memset(reader, 0, sizeof(*reader));
> +
> + reader->fd = fd;
> + reader->src_buffer = src_buffer;
> + reader->src_len = src_len;
> + reader->buffer = packet_buffer;
> + reader->buffer_size = sizeof(packet_buffer);
Looks like this is very non-reentrant. Can the doc comment warn about
that? Or even better, can it be made reentrant by owning its own
strbuf?
> + reader->options = options;
> +}
> +
> +enum packet_read_status packet_reader_read(struct packet_reader *reader)
> +{
> + if (reader->line_peeked) {
> + reader->line_peeked = 0;
> + return reader->status;
> + }
Nice.
> +
> + reader->status = packet_read_with_status(reader->fd,
> + &reader->src_buffer,
> + &reader->src_len,
> + reader->buffer,
> + reader->buffer_size,
> + &reader->pktlen,
> + reader->options);
> +
> + switch (reader->status) {
> + case PACKET_READ_EOF:
> + reader->pktlen = -1;
> + reader->line = NULL;
> + break;
> + case PACKET_READ_NORMAL:
> + reader->line = reader->buffer;
> + break;
> + case PACKET_READ_FLUSH:
> + reader->pktlen = 0;
> + reader->line = NULL;
> + break;
> + }
> +
> + return reader->status;
> +}
> +
> +enum packet_read_status packet_reader_peek(struct packet_reader *reader)
> +{
> + /* Only allow peeking a single line */
nit: s/peeking at/
> + if (reader->line_peeked)
> + return reader->status;
> +
> + /* Peek a line by reading it and setting peeked flag */
nit: s/Peek/Peek at/
> + packet_reader_read(reader);
> + reader->line_peeked = 1;
> + return reader->status;
> +}
Thanks for a pleasant read.
Jonathan
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader
2018-02-13 0:49 ` Jonathan Nieder
@ 2018-02-27 18:14 ` Brandon Williams
2018-02-27 19:20 ` Jonathan Nieder
0 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-02-27 18:14 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: git, sbeller, peff, gitster, stolee, git, pclouds
On 02/12, Jonathan Nieder wrote:
> [...]
> > --- a/pkt-line.h
> > +++ b/pkt-line.h
> > @@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
> > */
> > ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
> >
> > +struct packet_reader {
> > + /* source file descriptor */
> > + int fd;
> > +
> > + /* source buffer and its size */
> > + char *src_buffer;
> > + size_t src_len;
>
> Can or should this be a strbuf?
>
> > +
> > + /* buffer that pkt-lines are read into and its size */
> > + char *buffer;
> > + unsigned buffer_size;
>
> Likewise.
>
This struct is setup to be a drop in replacement for the existing
read_packet() family of functions. Because of this I tried to make the
interface as similar as possible to make it easy to convert to using it
as well as having no need to clean anything up (because the struct is
really just a wrapper and doesn't own anything).
--
Brandon Williams
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader
2018-02-27 18:14 ` Brandon Williams
@ 2018-02-27 19:20 ` Jonathan Nieder
0 siblings, 0 replies; 362+ messages in thread
From: Jonathan Nieder @ 2018-02-27 19:20 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, peff, gitster, stolee, git, pclouds
Brandon Williams wrote:
> On 02/12, Jonathan Nieder wrote:
>>> --- a/pkt-line.h
>>> +++ b/pkt-line.h
>>> @@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);
>>> */
>>> ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);
>>>
>>> +struct packet_reader {
>>> + /* source file descriptor */
>>> + int fd;
>>> +
>>> + /* source buffer and its size */
>>> + char *src_buffer;
>>> + size_t src_len;
>>
>> Can or should this be a strbuf?
>>
>>> +
>>> + /* buffer that pkt-lines are read into and its size */
>>> + char *buffer;
>>> + unsigned buffer_size;
>>
>> Likewise.
>
> This struct is setup to be a drop in replacement for the existing
> read_packet() family of functions. Because of this I tried to make the
> interface as similar as possible to make it easy to convert to using it
> as well as having no need to clean anything up (because the struct is
> really just a wrapper and doesn't own anything).
Sorry, I don't completely follow. Are you saying some callers play
with the buffer, or are you saying you haven't checked? (If the
latter, that's perfectly fine; I'm just trying to understand the API.)
Either way, can you add some comments about ownership / who is allowed
to write to it / etc to make it easier to clean up later?
Thanks,
Jonathan
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader
2018-02-07 1:12 ` [PATCH v3 02/35] pkt-line: introduce struct packet_reader Brandon Williams
2018-02-13 0:49 ` Jonathan Nieder
@ 2018-02-27 5:57 ` Jonathan Nieder
2018-02-27 6:12 ` Jonathan Nieder
1 sibling, 1 reply; 362+ messages in thread
From: Jonathan Nieder @ 2018-02-27 5:57 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, peff, gitster, stolee, git, pclouds
Hi,
Brandon Williams wrote:
> Sometimes it is advantageous to be able to peek the next packet line
> without consuming it (e.g. to be able to determine the protocol version
> a server is speaking). In order to do that introduce 'struct
> packet_reader' which is an abstraction around the normal packet reading
> logic. This enables a caller to be able to peek a single line at a time
> using 'packet_reader_peek()' and having a caller consume a line by
> calling 'packet_reader_read()'.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
> pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> pkt-line.h | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 117 insertions(+)
I like it!
The questions and nits from
https://public-inbox.org/git/20180213004937.GB42272@aiede.svl.corp.google.com/
still apply. In particular, the ownership of the buffers inside the
'struct packet_reader' is still unclear; could the packet_reader create
its own (strbuf) buffers so that the contract around them (who is allowed
to write to them; who is responsible for freeing them) is more obvious?
Thanks,
Jonathan
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader
2018-02-27 5:57 ` Jonathan Nieder
@ 2018-02-27 6:12 ` Jonathan Nieder
0 siblings, 0 replies; 362+ messages in thread
From: Jonathan Nieder @ 2018-02-27 6:12 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, peff, gitster, stolee, git, pclouds
Jonathan Nieder wrote:
> Brandon Williams wrote:
>> Sometimes it is advantageous to be able to peek the next packet line
>> without consuming it (e.g. to be able to determine the protocol version
>> a server is speaking). In order to do that introduce 'struct
>> packet_reader' which is an abstraction around the normal packet reading
>> logic. This enables a caller to be able to peek a single line at a time
>> using 'packet_reader_peek()' and having a caller consume a line by
>> calling 'packet_reader_read()'.
>>
>> Signed-off-by: Brandon Williams <bmwill@google.com>
>> ---
>> pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> pkt-line.h | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 2 files changed, 117 insertions(+)
>
> I like it!
>
> The questions and nits from
> https://public-inbox.org/git/20180213004937.GB42272@aiede.svl.corp.google.com/
> still apply. In particular, the ownership of the buffers inside the
> 'struct packet_reader' is still unclear; could the packet_reader create
> its own (strbuf) buffers so that the contract around them (who is allowed
> to write to them; who is responsible for freeing them) is more obvious?
Just to be clear: I sent that review after you sent this patch, so
there should not have been any reason for me to expect the q's and
nits to magically not apply. ;-)
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v3 03/35] pkt-line: add delim packet support
2018-02-07 1:12 ` [PATCH v3 00/35] " Brandon Williams
2018-02-07 1:12 ` [PATCH v3 01/35] pkt-line: introduce packet_read_with_status Brandon Williams
2018-02-07 1:12 ` [PATCH v3 02/35] pkt-line: introduce struct packet_reader Brandon Williams
@ 2018-02-07 1:12 ` Brandon Williams
2018-02-22 19:13 ` Stefan Beller
2018-02-07 1:12 ` [PATCH v3 04/35] upload-pack: convert to a builtin Brandon Williams
` (34 subsequent siblings)
37 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-02-07 1:12 UTC (permalink / raw)
To: git
Cc: sbeller, peff, gitster, jrnieder, stolee, git, pclouds, Brandon Williams
One of the design goals of protocol-v2 is to improve the semantics of
flush packets. Currently in protocol-v1, flush packets are used both to
indicate a break in a list of packet lines as well as an indication that
one side has finished speaking. This makes it particularly difficult
to implement proxies as a proxy would need to completely understand git
protocol instead of simply looking for a flush packet.
To do this, introduce the special deliminator packet '0001'. A delim
packet can then be used as a deliminator between lists of packet lines
while flush packets can be reserved to indicate the end of a response.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
pkt-line.c | 17 +++++++++++++++++
pkt-line.h | 3 +++
2 files changed, 20 insertions(+)
diff --git a/pkt-line.c b/pkt-line.c
index 4fc9ad4b0..726e109ca 100644
--- a/pkt-line.c
+++ b/pkt-line.c
@@ -91,6 +91,12 @@ void packet_flush(int fd)
write_or_die(fd, "0000", 4);
}
+void packet_delim(int fd)
+{
+ packet_trace("0001", 4, 1);
+ write_or_die(fd, "0001", 4);
+}
+
int packet_flush_gently(int fd)
{
packet_trace("0000", 4, 1);
@@ -105,6 +111,12 @@ void packet_buf_flush(struct strbuf *buf)
strbuf_add(buf, "0000", 4);
}
+void packet_buf_delim(struct strbuf *buf)
+{
+ packet_trace("0001", 4, 1);
+ strbuf_add(buf, "0001", 4);
+}
+
static void set_packet_header(char *buf, const int size)
{
static char hexchar[] = "0123456789abcdef";
@@ -297,6 +309,9 @@ enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_
} else if (!len) {
packet_trace("0000", 4, 0);
return PACKET_READ_FLUSH;
+ } else if (len == 1) {
+ packet_trace("0001", 4, 0);
+ return PACKET_READ_DELIM;
} else if (len < 4) {
die("protocol error: bad line length %d", len);
}
@@ -333,6 +348,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len,
break;
case PACKET_READ_NORMAL:
break;
+ case PACKET_READ_DELIM:
case PACKET_READ_FLUSH:
pktlen = 0;
break;
@@ -445,6 +461,7 @@ enum packet_read_status packet_reader_read(struct packet_reader *reader)
case PACKET_READ_NORMAL:
reader->line = reader->buffer;
break;
+ case PACKET_READ_DELIM:
case PACKET_READ_FLUSH:
reader->pktlen = 0;
reader->line = NULL;
diff --git a/pkt-line.h b/pkt-line.h
index 7d9f0e537..16fe8bdbf 100644
--- a/pkt-line.h
+++ b/pkt-line.h
@@ -20,8 +20,10 @@
* side can't, we stay with pure read/write interfaces.
*/
void packet_flush(int fd);
+void packet_delim(int fd);
void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
void packet_buf_flush(struct strbuf *buf);
+void packet_buf_delim(struct strbuf *buf);
void packet_write(int fd_out, const char *buf, size_t size);
void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));
int packet_flush_gently(int fd);
@@ -75,6 +77,7 @@ enum packet_read_status {
PACKET_READ_EOF = -1,
PACKET_READ_NORMAL,
PACKET_READ_FLUSH,
+ PACKET_READ_DELIM,
};
enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,
char *buffer, unsigned size, int *pktlen,
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v3 03/35] pkt-line: add delim packet support
2018-02-07 1:12 ` [PATCH v3 03/35] pkt-line: add delim packet support Brandon Williams
@ 2018-02-22 19:13 ` Stefan Beller
2018-02-22 19:37 ` Brandon Williams
0 siblings, 1 reply; 362+ messages in thread
From: Stefan Beller @ 2018-02-22 19:13 UTC (permalink / raw)
To: Brandon Williams
Cc: git, Jeff King, Junio C Hamano, Jonathan Nieder, Derrick Stolee,
Jeff Hostetler, Duy Nguyen
On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams <bmwill@google.com> wrote:
> One of the design goals of protocol-v2 is to improve the semantics of
> flush packets. Currently in protocol-v1, flush packets are used both to
> indicate a break in a list of packet lines as well as an indication that
> one side has finished speaking. This makes it particularly difficult
> to implement proxies as a proxy would need to completely understand git
> protocol instead of simply looking for a flush packet.
>
> To do this, introduce the special deliminator packet '0001'. A delim
> packet can then be used as a deliminator between lists of packet lines
> while flush packets can be reserved to indicate the end of a response.
Please mention where this can be found in the documentation.
(Defer to later patch?)
As the commit message states, this is only to be used for v2,
in v0 it is still an illegal pkt.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
The code is
Reviewed-by: Stefan Beller <sbeller@google.com>
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 03/35] pkt-line: add delim packet support
2018-02-22 19:13 ` Stefan Beller
@ 2018-02-22 19:37 ` Brandon Williams
0 siblings, 0 replies; 362+ messages in thread
From: Brandon Williams @ 2018-02-22 19:37 UTC (permalink / raw)
To: Stefan Beller
Cc: git, Jeff King, Junio C Hamano, Jonathan Nieder, Derrick Stolee,
Jeff Hostetler, Duy Nguyen
On 02/22, Stefan Beller wrote:
> On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams <bmwill@google.com> wrote:
> > One of the design goals of protocol-v2 is to improve the semantics of
> > flush packets. Currently in protocol-v1, flush packets are used both to
> > indicate a break in a list of packet lines as well as an indication that
> > one side has finished speaking. This makes it particularly difficult
> > to implement proxies as a proxy would need to completely understand git
> > protocol instead of simply looking for a flush packet.
> >
> > To do this, introduce the special deliminator packet '0001'. A delim
> > packet can then be used as a deliminator between lists of packet lines
> > while flush packets can be reserved to indicate the end of a response.
>
> Please mention where this can be found in the documentation.
> (Defer to later patch?)
Yeah the documentation does get added in a future patch, I'll make a
comment to that effect.
> As the commit message states, this is only to be used for v2,
> in v0 it is still an illegal pkt.
>
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
>
> The code is
> Reviewed-by: Stefan Beller <sbeller@google.com>
--
Brandon Williams
^ permalink raw reply [flat|nested] 362+ messages in thread
* [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-07 1:12 ` [PATCH v3 00/35] " Brandon Williams
` (2 preceding siblings ...)
2018-02-07 1:12 ` [PATCH v3 03/35] pkt-line: add delim packet support Brandon Williams
@ 2018-02-07 1:12 ` Brandon Williams
2018-02-21 21:44 ` Jonathan Tan
2018-02-07 1:12 ` [PATCH v3 05/35] upload-pack: factor out processing lines Brandon Williams
` (33 subsequent siblings)
37 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-02-07 1:12 UTC (permalink / raw)
To: git
Cc: sbeller, peff, gitster, jrnieder, stolee, git, pclouds, Brandon Williams
In order to allow for code sharing with the server-side of fetch in
protocol-v2 convert upload-pack to be a builtin.
Signed-off-by: Brandon Williams <bmwill@google.com>
---
Makefile | 3 +-
builtin.h | 1 +
builtin/upload-pack.c | 67 +++++++++++++++++++++++++++++++
git.c | 1 +
upload-pack.c | 107 ++++++++++++--------------------------------------
upload-pack.h | 13 ++++++
6 files changed, 109 insertions(+), 83 deletions(-)
create mode 100644 builtin/upload-pack.c
create mode 100644 upload-pack.h
diff --git a/Makefile b/Makefile
index 1a9b23b67..b7ccc05fa 100644
--- a/Makefile
+++ b/Makefile
@@ -639,7 +639,6 @@ PROGRAM_OBJS += imap-send.o
PROGRAM_OBJS += sh-i18n--envsubst.o
PROGRAM_OBJS += shell.o
PROGRAM_OBJS += show-index.o
-PROGRAM_OBJS += upload-pack.o
PROGRAM_OBJS += remote-testsvn.o
# Binary suffix, set to .exe for Windows builds
@@ -909,6 +908,7 @@ LIB_OBJS += tree-diff.o
LIB_OBJS += tree.o
LIB_OBJS += tree-walk.o
LIB_OBJS += unpack-trees.o
+LIB_OBJS += upload-pack.o
LIB_OBJS += url.o
LIB_OBJS += urlmatch.o
LIB_OBJS += usage.o
@@ -1026,6 +1026,7 @@ BUILTIN_OBJS += builtin/update-index.o
BUILTIN_OBJS += builtin/update-ref.o
BUILTIN_OBJS += builtin/update-server-info.o
BUILTIN_OBJS += builtin/upload-archive.o
+BUILTIN_OBJS += builtin/upload-pack.o
BUILTIN_OBJS += builtin/var.o
BUILTIN_OBJS += builtin/verify-commit.o
BUILTIN_OBJS += builtin/verify-pack.o
diff --git a/builtin.h b/builtin.h
index 42378f3aa..f332a1257 100644
--- a/builtin.h
+++ b/builtin.h
@@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);
extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);
extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);
extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);
+extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);
extern int cmd_var(int argc, const char **argv, const char *prefix);
extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);
extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);
diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c
new file mode 100644
index 000000000..2cb5cb35b
--- /dev/null
+++ b/builtin/upload-pack.c
@@ -0,0 +1,67 @@
+#include "cache.h"
+#include "builtin.h"
+#include "exec_cmd.h"
+#include "pkt-line.h"
+#include "parse-options.h"
+#include "protocol.h"
+#include "upload-pack.h"
+
+static const char * const upload_pack_usage[] = {
+ N_("git upload-pack [<options>] <dir>"),
+ NULL
+};
+
+int cmd_upload_pack(int argc, const char **argv, const char *prefix)
+{
+ const char *dir;
+ int strict = 0;
+ struct upload_pack_options opts = { 0 };
+ struct option options[] = {
+ OPT_BOOL(0, "stateless-rpc", &opts.stateless_rpc,
+ N_("quit after a single request/response exchange")),
+ OPT_BOOL(0, "advertise-refs", &opts.advertise_refs,
+ N_("exit immediately after initial ref advertisement")),
+ OPT_BOOL(0, "strict", &strict,
+ N_("do not try <directory>/.git/ if <directory> is no Git directory")),
+ OPT_INTEGER(0, "timeout", &opts.timeout,
+ N_("interrupt transfer after <n> seconds of inactivity")),
+ OPT_END()
+ };
+
+ packet_trace_identity("upload-pack");
+ check_replace_refs = 0;
+
+ argc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);
+
+ if (argc != 1)
+ usage_with_options(upload_pack_usage, options);
+
+ if (opts.timeout)
+ opts.daemon_mode = 1;
+
+ setup_path();
+
+ dir = argv[0];
+
+ if (!enter_repo(dir, strict))
+ die("'%s' does not appear to be a git repository", dir);
+
+ switch (determine_protocol_version_server()) {
+ case protocol_v1:
+ /*
+ * v1 is just the original protocol with a version string,
+ * so just fall through after writing the version string.
+ */
+ if (opts.advertise_refs || !opts.stateless_rpc)
+ packet_write_fmt(1, "version 1\n");
+
+ /* fallthrough */
+ case protocol_v0:
+ upload_pack(&opts);
+ break;
+ case protocol_unknown_version:
+ BUG("unknown protocol version");
+ }
+
+ return 0;
+}
diff --git a/git.c b/git.c
index c870b9719..f71073dc8 100644
--- a/git.c
+++ b/git.c
@@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {
{ "update-server-info", cmd_update_server_info, RUN_SETUP },
{ "upload-archive", cmd_upload_archive },
{ "upload-archive--writer", cmd_upload_archive_writer },
+ { "upload-pack", cmd_upload_pack },
{ "var", cmd_var, RUN_SETUP_GENTLY },
{ "verify-commit", cmd_verify_commit, RUN_SETUP },
{ "verify-pack", cmd_verify_pack },
diff --git a/upload-pack.c b/upload-pack.c
index d5de18127..2ad73a98b 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -6,7 +6,6 @@
#include "tag.h"
#include "object.h"
#include "commit.h"
-#include "exec_cmd.h"
#include "diff.h"
#include "revision.h"
#include "list-objects.h"
@@ -15,15 +14,10 @@
#include "sigchain.h"
#include "version.h"
#include "string-list.h"
-#include "parse-options.h"
#include "argv-array.h"
#include "prio-queue.h"
#include "protocol.h"
-
-static const char * const upload_pack_usage[] = {
- N_("git upload-pack [<options>] <dir>"),
- NULL
-};
+#include "upload-pack.h"
/* Remember to update object flag allocation in object.h */
#define THEY_HAVE (1u << 11)
@@ -61,7 +55,6 @@ static int keepalive = 5;
* otherwise maximum packet size (up to 65520 bytes).
*/
static int use_sideband;
-static int advertise_refs;
static int stateless_rpc;
static const char *pack_objects_hook;
@@ -977,33 +970,6 @@ static int find_symref(const char *refname, const struct object_id *oid,
return 0;
}
-static void upload_pack(void)
-{
- struct string_list symref = STRING_LIST_INIT_DUP;
-
- head_ref_namespaced(find_symref, &symref);
-
- if (advertise_refs || !stateless_rpc) {
- reset_timeout();
- head_ref_namespaced(send_ref, &symref);
- for_each_namespaced_ref(send_ref, &symref);
- advertise_shallow_grafts(1);
- packet_flush(1);
- } else {
- head_ref_namespaced(check_ref, NULL);
- for_each_namespaced_ref(check_ref, NULL);
- }
- string_list_clear(&symref, 1);
- if (advertise_refs)
- return;
-
- receive_needs();
- if (want_obj.nr) {
- get_common_commits();
- create_pack_file();
- }
-}
-
static int upload_pack_config(const char *var, const char *value, void *unused)
{
if (!strcmp("uploadpack.allowtipsha1inwant", var)) {
@@ -1032,58 +998,35 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
return parse_hide_refs_config(var, value, "uploadpack");
}
-int cmd_main(int argc, const char **argv)
+void upload_pack(struct upload_pack_options *options)
{
- const char *dir;
- int strict = 0;
- struct option options[] = {
- OPT_BOOL(0, "stateless-rpc", &stateless_rpc,
- N_("quit after a single request/response exchange")),
- OPT_BOOL(0, "advertise-refs", &advertise_refs,
- N_("exit immediately after initial ref advertisement")),
- OPT_BOOL(0, "strict", &strict,
- N_("do not try <directory>/.git/ if <directory> is no Git directory")),
- OPT_INTEGER(0, "timeout", &timeout,
- N_("interrupt transfer after <n> seconds of inactivity")),
- OPT_END()
- };
-
- packet_trace_identity("upload-pack");
- check_replace_refs = 0;
-
- argc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);
-
- if (argc != 1)
- usage_with_options(upload_pack_usage, options);
-
- if (timeout)
- daemon_mode = 1;
-
- setup_path();
-
- dir = argv[0];
+ struct string_list symref = STRING_LIST_INIT_DUP;
- if (!enter_repo(dir, strict))
- die("'%s' does not appear to be a git repository", dir);
+ stateless_rpc = options->stateless_rpc;
+ timeout = options->timeout;
+ daemon_mode = options->daemon_mode;
git_config(upload_pack_config, NULL);
- switch (determine_protocol_version_server()) {
- case protocol_v1:
- /*
- * v1 is just the original protocol with a version string,
- * so just fall through after writing the version string.
- */
- if (advertise_refs || !stateless_rpc)
- packet_write_fmt(1, "version 1\n");
-
- /* fallthrough */
- case protocol_v0:
- upload_pack();
- break;
- case protocol_unknown_version:
- BUG("unknown protocol version");
+ head_ref_namespaced(find_symref, &symref);
+
+ if (options->advertise_refs || !stateless_rpc) {
+ reset_timeout();
+ head_ref_namespaced(send_ref, &symref);
+ for_each_namespaced_ref(send_ref, &symref);
+ advertise_shallow_grafts(1);
+ packet_flush(1);
+ } else {
+ head_ref_namespaced(check_ref, NULL);
+ for_each_namespaced_ref(check_ref, NULL);
}
+ string_list_clear(&symref, 1);
+ if (options->advertise_refs)
+ return;
- return 0;
+ receive_needs();
+ if (want_obj.nr) {
+ get_common_commits();
+ create_pack_file();
+ }
}
diff --git a/upload-pack.h b/upload-pack.h
new file mode 100644
index 000000000..a71e4dc7e
--- /dev/null
+++ b/upload-pack.h
@@ -0,0 +1,13 @@
+#ifndef UPLOAD_PACK_H
+#define UPLOAD_PACK_H
+
+struct upload_pack_options {
+ int stateless_rpc;
+ int advertise_refs;
+ unsigned int timeout;
+ int daemon_mode;
+};
+
+void upload_pack(struct upload_pack_options *options);
+
+#endif /* UPLOAD_PACK_H */
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 362+ messages in thread* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-07 1:12 ` [PATCH v3 04/35] upload-pack: convert to a builtin Brandon Williams
@ 2018-02-21 21:44 ` Jonathan Tan
2018-02-22 9:58 ` Jeff King
0 siblings, 1 reply; 362+ messages in thread
From: Jonathan Tan @ 2018-02-21 21:44 UTC (permalink / raw)
To: Brandon Williams
Cc: git, sbeller, peff, gitster, jrnieder, stolee, git, pclouds
On Tue, 6 Feb 2018 17:12:41 -0800
Brandon Williams <bmwill@google.com> wrote:
> In order to allow for code sharing with the server-side of fetch in
> protocol-v2 convert upload-pack to be a builtin.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
As Stefan mentioned in [1], also mention in the commit message that this
means that the "git-upload-pack" invocation gains additional
capabilities (for example, invoking a pager for --help).
Having said that, the main purpose of this patch seems to be to libify
upload-pack, and the move to builtin is just a way of putting the
program somewhere - we could have easily renamed upload-pack.c and
created a new upload-pack.c containing the main(), preserving the
non-builtin-ness of upload-pack, while still gaining the benefits of
libifying upload-pack.
If the community does want to make upload-pack a builtin, I would write
the commit message this way:
upload-pack: libify
Libify upload-pack. The main() function is moved to
builtin/upload-pack.c, thus making upload-pack a builtin. Note that
this means that "git-upload-pack" gains functionality such as the
ability to invoke a pager when passed "--help".
And if not:
upload-pack: libify
Libify upload-pack by moving most of the functionality in
upload-pack.c into a file upload-pack-lib.c (or some other name),
to be used in subsequent patches.
[1] https://public-inbox.org/git/CAGZ79kb2=uU0_K8wr27gNdNX-T+P+7gVdgc5EBdYc3zBobsR8w@mail.gmail.com/
> -static void upload_pack(void)
> -{
> - struct string_list symref = STRING_LIST_INIT_DUP;
> -
> - head_ref_namespaced(find_symref, &symref);
> -
> - if (advertise_refs || !stateless_rpc) {
> - reset_timeout();
> - head_ref_namespaced(send_ref, &symref);
> - for_each_namespaced_ref(send_ref, &symref);
> - advertise_shallow_grafts(1);
> - packet_flush(1);
> - } else {
> - head_ref_namespaced(check_ref, NULL);
> - for_each_namespaced_ref(check_ref, NULL);
> - }
> - string_list_clear(&symref, 1);
> - if (advertise_refs)
> - return;
> -
> - receive_needs();
> - if (want_obj.nr) {
> - get_common_commits();
> - create_pack_file();
> - }
> -}
I see that this function had to be moved to the bottom because it now
also needs to make use of functions like upload_pack_config() - that's
fine.
> +struct upload_pack_options {
> + int stateless_rpc;
> + int advertise_refs;
> + unsigned int timeout;
> + int daemon_mode;
> +};
I would have expected "unsigned stateless_rpc : 1" etc., but I see that
this makes it easier to use with OPT_BOOL (which needs us to pass it a
pointer-to-int).
As for what existing code does, files like fetch-pack and diff use
"unsigned : 1", but they also process arguments without OPT_, so I don't
think they are relevant.
I think that we should decide if we're going to prefer "unsigned : 1" or
"int" for flags in new code. Personally, I prefer "unsigned : 1"
(despite the slight inconvenience in that argument parsers will need to
declare their own temporary "int" and then assign its contents to the
options struct) because of the stronger type, but I'm OK either way.
Whatever the decision, I don't think it needs to block the review of
this patch set.
^ permalink raw reply [flat|nested] 362+ messages in thread* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-21 21:44 ` Jonathan Tan
@ 2018-02-22 9:58 ` Jeff King
2018-02-22 18:07 ` Brandon Williams
0 siblings, 1 reply; 362+ messages in thread
From: Jeff King @ 2018-02-22 9:58 UTC (permalink / raw)
To: Jonathan Tan
Cc: Brandon Williams, git, sbeller, gitster, jrnieder, stolee, git, pclouds
On Wed, Feb 21, 2018 at 01:44:22PM -0800, Jonathan Tan wrote:
> On Tue, 6 Feb 2018 17:12:41 -0800
> Brandon Williams <bmwill@google.com> wrote:
>
> > In order to allow for code sharing with the server-side of fetch in
> > protocol-v2 convert upload-pack to be a builtin.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
>
> As Stefan mentioned in [1], also mention in the commit message that this
> means that the "git-upload-pack" invocation gains additional
> capabilities (for example, invoking a pager for --help).
And possibly respecting pager.upload-pack, which would violate our rule
that it is safe to run upload-pack in untrusted repositories.
(This actually doesn't work right now because pager.* is broken for
builtins that don't specify RUN_SETUP; but I think with the fixes last
year to the config code, we can now drop that restriction).
Obviously we can work around this with an extra RUN_NO_PAGER_CONFIG
flag. But I think it points to a general danger in making upload-pack a
builtin. I'm not sure what other features it would want to avoid (or
what might grow in the future).
> Having said that, the main purpose of this patch seems to be to libify
> upload-pack, and the move to builtin is just a way of putting the
> program somewhere - we could have easily renamed upload-pack.c and
> created a new upload-pack.c containing the main(), preserving the
> non-builtin-ness of upload-pack, while still gaining the benefits of
> libifying upload-pack.
Yeah, this seems like a better route to me.
-Peff
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-22 9:58 ` Jeff King
@ 2018-02-22 18:07 ` Brandon Williams
2018-02-22 18:14 ` Jeff King
0 siblings, 1 reply; 362+ messages in thread
From: Brandon Williams @ 2018-02-22 18:07 UTC (permalink / raw)
To: Jeff King
Cc: Jonathan Tan, git, sbeller, gitster, jrnieder, stolee, git, pclouds
On 02/22, Jeff King wrote:
> On Wed, Feb 21, 2018 at 01:44:22PM -0800, Jonathan Tan wrote:
>
> > On Tue, 6 Feb 2018 17:12:41 -0800
> > Brandon Williams <bmwill@google.com> wrote:
> >
> > > In order to allow for code sharing with the server-side of fetch in
> > > protocol-v2 convert upload-pack to be a builtin.
> > >
> > > Signed-off-by: Brandon Williams <bmwill@google.com>
> >
> > As Stefan mentioned in [1], also mention in the commit message that this
> > means that the "git-upload-pack" invocation gains additional
> > capabilities (for example, invoking a pager for --help).
>
> And possibly respecting pager.upload-pack, which would violate our rule
> that it is safe to run upload-pack in untrusted repositories.
And this isn't an issue with receive-pack because this same guarantee
doesn't exist?
>
> (This actually doesn't work right now because pager.* is broken for
> builtins that don't specify RUN_SETUP; but I think with the fixes last
> year to the config code, we can now drop that restriction).
>
> Obviously we can work around this with an extra RUN_NO_PAGER_CONFIG
> flag. But I think it points to a general danger in making upload-pack a
> builtin. I'm not sure what other features it would want to avoid (or
> what might grow in the future).
>
> > Having said that, the main purpose of this patch seems to be to libify
> > upload-pack, and the move to builtin is just a way of putting the
> > program somewhere - we could have easily renamed upload-pack.c and
> > created a new upload-pack.c containing the main(), preserving the
> > non-builtin-ness of upload-pack, while still gaining the benefits of
> > libifying upload-pack.
>
> Yeah, this seems like a better route to me.
>
> -Peff
--
Brandon Williams
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-22 18:07 ` Brandon Williams
@ 2018-02-22 18:14 ` Jeff King
2018-02-22 19:38 ` Jonathan Nieder
0 siblings, 1 reply; 362+ messages in thread
From: Jeff King @ 2018-02-22 18:14 UTC (permalink / raw)
To: Brandon Williams
Cc: Jonathan Tan, git, sbeller, gitster, jrnieder, stolee, git, pclouds
On Thu, Feb 22, 2018 at 10:07:15AM -0800, Brandon Williams wrote:
> On 02/22, Jeff King wrote:
> > On Wed, Feb 21, 2018 at 01:44:22PM -0800, Jonathan Tan wrote:
> >
> > > On Tue, 6 Feb 2018 17:12:41 -0800
> > > Brandon Williams <bmwill@google.com> wrote:
> > >
> > > > In order to allow for code sharing with the server-side of fetch in
> > > > protocol-v2 convert upload-pack to be a builtin.
> > > >
> > > > Signed-off-by: Brandon Williams <bmwill@google.com>
> > >
> > > As Stefan mentioned in [1], also mention in the commit message that this
> > > means that the "git-upload-pack" invocation gains additional
> > > capabilities (for example, invoking a pager for --help).
> >
> > And possibly respecting pager.upload-pack, which would violate our rule
> > that it is safe to run upload-pack in untrusted repositories.
>
> And this isn't an issue with receive-pack because this same guarantee
> doesn't exist?
Yes, exactly (which is confusing and weird, yes, but that's how it is).
-Peff
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-22 18:14 ` Jeff King
@ 2018-02-22 19:38 ` Jonathan Nieder
2018-02-22 20:19 ` Jeff King
0 siblings, 1 reply; 362+ messages in thread
From: Jonathan Nieder @ 2018-02-22 19:38 UTC (permalink / raw)
To: Jeff King
Cc: Brandon Williams, Jonathan Tan, git, sbeller, gitster, stolee,
git, pclouds
Hi,
Jeff King wrote:
> On Thu, Feb 22, 2018 at 10:07:15AM -0800, Brandon Williams wrote:
>> On 02/22, Jeff King wrote:
>>> On Wed, Feb 21, 2018 at 01:44:22PM -0800, Jonathan Tan wrote:
>>>> On Tue, 6 Feb 2018 17:12:41 -0800
>>>> Brandon Williams <bmwill@google.com> wrote:
>>>>> In order to allow for code sharing with the server-side of fetch in
>>>>> protocol-v2 convert upload-pack to be a builtin.
[...]
>>>> As Stefan mentioned in [1], also mention in the commit message that this
>>>> means that the "git-upload-pack" invocation gains additional
>>>> capabilities (for example, invoking a pager for --help).
>>>
>>> And possibly respecting pager.upload-pack, which would violate our rule
>>> that it is safe to run upload-pack in untrusted repositories.
>>
>> And this isn't an issue with receive-pack because this same guarantee
>> doesn't exist?
>
> Yes, exactly (which is confusing and weird, yes, but that's how it is).
To be clear, which of the following are you (most) worried about?
1. being invoked with --help and spawning a pager
2. receiving and acting on options between 'git' and 'upload-pack'
3. repository discovery
4. pager config
5. alias discovery
6. increased code surface / unknown threats
For (1), "--help" has to be the first argument. "git daemon" passes
--strict so it doesn't happen there. "git http-backend" passes
--stateless-rpc so it doesn't happen there. "git shell" sanitizes
input to avoid it happening there.
A custom setup could provide their own entry point that doesn't do
such sanitization. I think that in some sense it's out of our hands,
but it would be nice to harden as described upthread.
For (2), I am having trouble imagining a setup where it would happen.
upload-pack doesn't have the RUN_SETUP or RUN_SETUP_GENTLY flag set,
so (3) doesn't apply.
Although in most setups the user does not control the config files on
a server, item (4) looks like a real issue worth solving. I think we
should introduce a flag to skip looking for pager config. We could
use it for receive-pack, too.
Builtins are handled before aliases, so (5) doesn't apply.
(6) is a real issue: it is why "git shell" is not a builtin, for
example. But I would rather that we use appropriate sanitization
before upload-pack is invoked than live in fear. git upload-pack is
sufficiently complicated that I don't think the git.c wrapper
increases the complexity by a significant amount.
That leaves me with a personal answer of only being worried about (4)
and not the rest. What do you think? Is one of the other items I
listed above worrisome, or is there another item I missed?
Thanks,
Jonathan
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-22 19:38 ` Jonathan Nieder
@ 2018-02-22 20:19 ` Jeff King
2018-02-22 20:21 ` Jeff King
2018-02-22 21:24 ` Jonathan Nieder
0 siblings, 2 replies; 362+ messages in thread
From: Jeff King @ 2018-02-22 20:19 UTC (permalink / raw)
To: Jonathan Nieder
Cc: Brandon Williams, Jonathan Tan, git, sbeller, gitster, stolee,
git, pclouds
On Thu, Feb 22, 2018 at 11:38:14AM -0800, Jonathan Nieder wrote:
> >>> And possibly respecting pager.upload-pack, which would violate our rule
> >>> that it is safe to run upload-pack in untrusted repositories.
> >>
> >> And this isn't an issue with receive-pack because this same guarantee
> >> doesn't exist?
> >
> > Yes, exactly (which is confusing and weird, yes, but that's how it is).
>
> To be clear, which of the following are you (most) worried about?
>
> 1. being invoked with --help and spawning a pager
> 2. receiving and acting on options between 'git' and 'upload-pack'
> 3. repository discovery
> 4. pager config
> 5. alias discovery
> 6. increased code surface / unknown threats
My immediate concern is (4). But my greater concern is that people who
work on git.c should not have to worry about accidentally violating this
principle when they add a new feature or config option.
In other words, it seems like an accident waiting to happen. I'd be more
amenable to it if there was some compelling reason for it to be a
builtin, but I don't see one listed in the commit message. I see only
"let's make it easier to share the code", which AFAICT is equally served
by just lib-ifying the code and calling it from the standalone
upload-pack.c.
> Although in most setups the user does not control the config files on
> a server, item (4) looks like a real issue worth solving. I think we
> should introduce a flag to skip looking for pager config. We could
> use it for receive-pack, too.
There's not much point for receive-pack. It respects hooks, so any
security ship has already sailed there.
-Peff
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-22 20:19 ` Jeff King
@ 2018-02-22 20:21 ` Jeff King
2018-02-22 21:26 ` Jonathan Nieder
2018-02-23 21:09 ` Brandon Williams
2018-02-22 21:24 ` Jonathan Nieder
1 sibling, 2 replies; 362+ messages in thread
From: Jeff King @ 2018-02-22 20:21 UTC (permalink / raw)
To: Jonathan Nieder
Cc: Brandon Williams, Jonathan Tan, git, sbeller, gitster, stolee,
git, pclouds
On Thu, Feb 22, 2018 at 03:19:40PM -0500, Jeff King wrote:
> > To be clear, which of the following are you (most) worried about?
> >
> > 1. being invoked with --help and spawning a pager
> > 2. receiving and acting on options between 'git' and 'upload-pack'
> > 3. repository discovery
> > 4. pager config
> > 5. alias discovery
> > 6. increased code surface / unknown threats
>
> My immediate concern is (4). But my greater concern is that people who
> work on git.c should not have to worry about accidentally violating this
> principle when they add a new feature or config option.
>
> In other words, it seems like an accident waiting to happen. I'd be more
> amenable to it if there was some compelling reason for it to be a
> builtin, but I don't see one listed in the commit message. I see only
> "let's make it easier to share the code", which AFAICT is equally served
> by just lib-ifying the code and calling it from the standalone
> upload-pack.c.
By the way, any decision here would presumably need to be extended to
git-serve, etc. The current property is that it's safe to fetch from an
untrusted repository, even over ssh. If we're keeping that for protocol
v1, we'd want it to apply to protocol v2, as well.
-Peff
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-22 20:21 ` Jeff King
@ 2018-02-22 21:26 ` Jonathan Nieder
2018-02-22 21:44 ` Jeff King
2018-02-23 21:09 ` Brandon Williams
1 sibling, 1 reply; 362+ messages in thread
From: Jonathan Nieder @ 2018-02-22 21:26 UTC (permalink / raw)
To: Jeff King
Cc: Brandon Williams, Jonathan Tan, git, sbeller, gitster, stolee,
git, pclouds
Jeff King wrote:
> The current property is that it's safe to fetch from an
> untrusted repository, even over ssh. If we're keeping that for protocol
> v1, we'd want it to apply to protocol v2, as well.
Ah, this is what I had been missing (the non-ssh case).
I see your point. I think we need to fix the pager config issue and add
some clarifying documentation to git.c so that people know what to look
out for.
Keep in mind that git upload-archive (a read-only command, just like
git upload-pack) also already has the same issues.
Thanks,
Jonathan
^ permalink raw reply [flat|nested] 362+ messages in thread
* Re: [PATCH v3 04/35] upload-pack: convert to a builtin
2018-02-22 21:26 ` Jonathan Nieder
@ 2018-02-22 21:44 ` Jeff King
2018-03-12 22:43 ` Jonathan Nieder
0 siblings, 1 reply; 362+ messages in thread
From: Jeff King @ 2018-02-22 21:44 UTC (permalink / raw)
To: Jonathan Nieder
Cc: Brandon Williams, Jonathan Tan, git, sbeller, gitster, stolee,
git, pclouds
On Thu, Feb 22, 2018 at 01:26:34PM -0800, Jonathan Nieder wrote:
> Keep in mind that git upload-archive (a read-only command, just like
> git upload-pack) also already has the same issues.
Yuck. I don't think we've ever made a historical promise about that. But
then, I don't think the promise about upload-pack has ever really been
documented, except in mailing list discussions.
-Peff
^ permalink raw reply [flat|