Initial support for cloning submodules

Initial support for cloning submodules

[Sven Verdoolaege]

From: Sven Verdoolaege <skimo@kotnet.org>

This patch series implements a mechanism for cloning submodules.
Each submodule is specified by a 'submodule.<submodule>.url'
configuration option, e.g.,

bash-3.00$ ./git-config --remote=http://www.liacs.nl/~sverdool/isa.git --get-regexp 'submodule\..*\.url' 
submodule.cloog.url /home/sverdool/public_html/cloog.git
submodule.cloog.url http://www.liacs.nl/~sverdool/cloog.git

git-clone will use the first url that works.
E.g., a

git clone --submodules ssh://liacs/~/public_html/isa.git

(which only works for me), will use the first url, while a

git clone --submodules http://www.liacs.nl/~sverdool/isa.git

will use the second.

The submodules are currently not checked out.

skimo

Initial support for cloning submodules

[Sven Verdoolaege]

From: Sven Verdoolaege <skimo@kotnet.org>

This patch series implements a mechanism for cloning submodules.
Each submodule is specified by a 'submodule.<submodule>.url'
configuration option, e.g.,

bash-3.00$ ./git-config --remote=http://www.liacs.nl/~sverdool/isa.git --get-regexp 'submodule\..*\.url' 
submodule.cloog.url /home/sverdool/public_html/cloog.git
submodule.cloog.url http://www.liacs.nl/~sverdool/cloog.git

git-clone will use the first url that works.
E.g., a

git clone --submodules ssh://liacs/~/public_html/isa.git

(which only works for me), will use the first url, while a

git clone --submodules http://www.liacs.nl/~sverdool/isa.git

will use the second.

The submodules are currently not checked out.

skimo

[PATCH 1/5] Add dump-config

[Sven Verdoolaege]

From: Sven Verdoolaege <skimo@kotnet.org>

This command dumps the config of a repository and will be used
to read config options from a remote site.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 .gitignore                        |    1 +
 Documentation/cmd-list.perl       |    1 +
 Documentation/git-dump-config.txt |   37 +++++++++++++++++++++++++++++++++++++
 Makefile                          |    1 +
 daemon.c                          |    7 +++++++
 dump-config.c                     |   29 +++++++++++++++++++++++++++++
 6 files changed, 76 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/git-dump-config.txt
 create mode 100644 dump-config.c

diff --git a/.gitignore b/.gitignore
index 4dc0c39..d4e5492 100644
--- a/.gitignore
+++ b/.gitignore
@@ -38,6 +38,7 @@ git-diff-files
 git-diff-index
 git-diff-tree
 git-describe
+git-dump-config
 git-fast-import
 git-fetch
 git-fetch--tool
diff --git a/Documentation/cmd-list.perl b/Documentation/cmd-list.perl
index 443802a..fa04615 100755
--- a/Documentation/cmd-list.perl
+++ b/Documentation/cmd-list.perl
@@ -103,6 +103,7 @@ git-diff-files                          plumbinginterrogators
 git-diff-index                          plumbinginterrogators
 git-diff                                mainporcelain
 git-diff-tree                           plumbinginterrogators
+git-dump-config                         synchelpers
 git-fast-import				ancillarymanipulators
 git-fetch                               mainporcelain
 git-fetch-pack                          synchingrepositories
diff --git a/Documentation/git-dump-config.txt b/Documentation/git-dump-config.txt
new file mode 100644
index 0000000..370781c
--- /dev/null
+++ b/Documentation/git-dump-config.txt
@@ -0,0 +1,37 @@
+git-dump-config(1)
+====================
+
+NAME
+----
+git-dump-config - Dump config options
+
+
+SYNOPSIS
+--------
+'git-dump-config' <directory>
+
+DESCRIPTION
+-----------
+Invoked by 'git-config --remote' and dumps the config file to the
+other end over the git protocol.
+
+This command is usually not invoked directly by the end user.  The UI
+for the protocol is on the 'git-config' side, where it is used to get
+options from a remote repository.
+
+OPTIONS
+-------
+<directory>::
+	The repository to get the config options from.
+
+Author
+------
+Written by Sven Verdoolaege.
+
+Documentation
+--------------
+Documentation by Sven Verdoolaege.
+
+GIT
+---
+Part of the gitlink:git[7] suite
diff --git a/Makefile b/Makefile
index e0a1308..0185386 100644
--- a/Makefile
+++ b/Makefile
@@ -232,6 +232,7 @@ PROGRAMS = \
 	git-fast-import$X \
 	git-merge-base$X \
 	git-daemon$X \
+	git-dump-config$X \
 	git-merge-index$X git-mktag$X git-mktree$X git-patch-id$X \
 	git-peek-remote$X git-receive-pack$X \
 	git-send-pack$X git-shell$X \
diff --git a/daemon.c b/daemon.c
index e74ecac..3e5ebf3 100644
--- a/daemon.c
+++ b/daemon.c
@@ -378,10 +378,17 @@ static int receive_pack(void)
 	return -1;
 }
 
+static int dump_config(void)
+{
+	execl_git_cmd("dump-config", ".", NULL);
+	return -1;
+}
+
 static struct daemon_service daemon_service[] = {
 	{ "upload-archive", "uploadarch", upload_archive, 0, 1 },
 	{ "upload-pack", "uploadpack", upload_pack, 1, 1 },
 	{ "receive-pack", "receivepack", receive_pack, 0, 1 },
+	{ "dump-config", "dumpconfig", dump_config, 0, 1 },
 };
 
 static void enable_service(const char *name, int ena) {
diff --git a/dump-config.c b/dump-config.c
new file mode 100644
index 0000000..355920d
--- /dev/null
+++ b/dump-config.c
@@ -0,0 +1,29 @@
+#include "git-compat-util.h"
+#include "cache.h"
+#include "pkt-line.h"
+
+static const char dump_config_usage[] = "git-dump-config <dir>";
+
+static int dump_config(const char *var, const char *value)
+{
+	packet_write(1, "%s", var);
+	packet_write(1, "%s", value);
+	return 0;
+}
+
+int main(int argc, char **argv)
+{
+	char *dir;
+
+	if (argc != 2)
+		usage(dump_config_usage);
+
+	dir = argv[1];
+	if (!enter_repo(dir, 0))
+		die("'%s': unable to chdir or not a git archive", dir);
+
+	git_config(dump_config);
+	packet_flush(1);
+
+	return 0;
+}
-- 
1.5.2.rc1.25.g889f-dirty

[PATCH 2/5] git-config: add --remote option for reading config from remote repo

[Sven Verdoolaege]

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 Documentation/git-config.txt |   33 +++++++++++++++++++++---------
 builtin-config.c             |   44 ++++++++++++++++++++++++++++++++---------
 cache.h                      |    1 +
 config.c                     |   26 ++++++++++++++++++++++++
 4 files changed, 84 insertions(+), 20 deletions(-)

diff --git a/Documentation/git-config.txt b/Documentation/git-config.txt
index 280ef20..76398ab 100644
--- a/Documentation/git-config.txt
+++ b/Documentation/git-config.txt
@@ -9,16 +9,25 @@ git-config - Get and set repository or global options
 SYNOPSIS
 --------
 [verse]
-'git-config' [--system | --global] [type] name [value [value_regex]]
-'git-config' [--system | --global] [type] --add name value
-'git-config' [--system | --global] [type] --replace-all name [value [value_regex]]
-'git-config' [--system | --global] [type] --get name [value_regex]
-'git-config' [--system | --global] [type] --get-all name [value_regex]
-'git-config' [--system | --global] [type] --unset name [value_regex]
-'git-config' [--system | --global] [type] --unset-all name [value_regex]
-'git-config' [--system | --global] [type] --rename-section old_name new_name
-'git-config' [--system | --global] [type] --remove-section name
-'git-config' [--system | --global] -l | --list
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] name [value [value_regex]]
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] --add name value
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] --replace-all name [value [value_regex]]
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] --get name [value_regex]
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] --get-all name [value_regex]
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] --unset name [value_regex]
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] --unset-all name [value_regex]
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] --rename-section old_name new_name
+'git-config' [--system | --global | --remote=[<host>:]<directory ]
+	     [type] --remove-section name
+'git-config' [--system | --global | --remote=[<host>:]<directory ] -l | --list
 
 DESCRIPTION
 -----------
@@ -80,6 +89,10 @@ OPTIONS
 	Use system-wide $(prefix)/etc/gitconfig rather than the repository
 	.git/config.
 
+--remote=[<host>:]<directory
+	Use remote config instead of the repository .git/config.
+	Only available for reading options.
+
 --remove-section::
 	Remove the given section from the configuration file.
 
diff --git a/builtin-config.c b/builtin-config.c
index b2515f7..3a1e86c 100644
--- a/builtin-config.c
+++ b/builtin-config.c
@@ -2,8 +2,10 @@
 #include "cache.h"
 
 static const char git_config_set_usage[] =
-"git-config [ --global | --system ] [ --bool | --int ] [--get | --get-all | --get-regexp | --replace-all | --add | --unset | --unset-all] name [value [value_regex]] | --rename-section old_name new_name | --remove-section name | --list";
+"git-config [ --global | --system | --remote=[<host>:]<directory ] "
+"[ --bool | --int ] [--get | --get-all | --get-regexp | --replace-all | --add | --unset | --unset-all] name [value [value_regex]] | --rename-section old_name new_name | --remove-section name | --list";
 
+static char *dest;
 static char *key;
 static regex_t *key_regexp;
 static regex_t *regexp;
@@ -104,15 +106,19 @@ static int get_value(const char* key_, const char* regex_)
 		}
 	}
 
-	if (do_all && system_wide)
-		git_config_from_file(show_config, system_wide);
-	if (do_all && global)
-		git_config_from_file(show_config, global);
-	git_config_from_file(show_config, local);
-	if (!do_all && !seen && global)
-		git_config_from_file(show_config, global);
-	if (!do_all && !seen && system_wide)
-		git_config_from_file(show_config, system_wide);
+	if (dest)
+		git_config_from_remote(show_config, dest);
+	else {
+		if (do_all && system_wide)
+			git_config_from_file(show_config, system_wide);
+		if (do_all && global)
+			git_config_from_file(show_config, global);
+		git_config_from_file(show_config, local);
+		if (!do_all && !seen && global)
+			git_config_from_file(show_config, global);
+		if (!do_all && !seen && system_wide)
+			git_config_from_file(show_config, system_wide);
+	}
 
 	free(key);
 	if (regexp) {
@@ -155,8 +161,14 @@ int cmd_config(int argc, const char **argv, const char *prefix)
 		}
 		else if (!strcmp(argv[1], "--system"))
 			setenv("GIT_CONFIG", ETC_GITCONFIG, 1);
+		else if (!prefixcmp(argv[1], "--remote="))
+			dest = xstrdup(argv[1]+9);
 		else if (!strcmp(argv[1], "--rename-section")) {
 			int ret;
+			if (dest) {
+				fprintf(stderr, "Cannot rename on remote\n");
+				return 1;
+			}
 			if (argc != 4)
 				usage(git_config_set_usage);
 			ret = git_config_rename_section(argv[2], argv[3]);
@@ -170,6 +182,10 @@ int cmd_config(int argc, const char **argv, const char *prefix)
 		}
 		else if (!strcmp(argv[1], "--remove-section")) {
 			int ret;
+			if (dest) {
+				fprintf(stderr, "Cannot remove on remote\n");
+				return 1;
+			}
 			if (argc != 3)
 				usage(git_config_set_usage);
 			ret = git_config_rename_section(argv[2], NULL);
@@ -191,6 +207,10 @@ int cmd_config(int argc, const char **argv, const char *prefix)
 	case 2:
 		return get_value(argv[1], NULL);
 	case 3:
+		if (dest && prefixcmp(argv[1], "--get")) {
+			fprintf(stderr, "Cannot (un)set on remote\n");
+			return 1;
+		}
 		if (!strcmp(argv[1], "--unset"))
 			return git_config_set(argv[2], NULL);
 		else if (!strcmp(argv[1], "--unset-all"))
@@ -209,6 +229,10 @@ int cmd_config(int argc, const char **argv, const char *prefix)
 
 			return git_config_set(argv[1], argv[2]);
 	case 4:
+		if (dest && prefixcmp(argv[1], "--get")) {
+			fprintf(stderr, "Cannot (un)set on remote\n");
+			return 1;
+		}
 		if (!strcmp(argv[1], "--unset"))
 			return git_config_set_multivar(argv[2], NULL, argv[3], 0);
 		else if (!strcmp(argv[1], "--unset-all"))
diff --git a/cache.h b/cache.h
index 8e76152..e8c7791 100644
--- a/cache.h
+++ b/cache.h
@@ -499,6 +499,7 @@ extern int update_server_info(int);
 typedef int (*config_fn_t)(const char *, const char *);
 extern int git_default_config(const char *, const char *);
 extern int git_config_from_file(config_fn_t fn, const char *);
+extern int git_config_from_remote(config_fn_t fn, char *dest);
 extern int git_config(config_fn_t fn);
 extern int git_config_int(const char *, const char *);
 extern int git_config_bool(const char *, const char *);
diff --git a/config.c b/config.c
index 70d1055..0da74e0 100644
--- a/config.c
+++ b/config.c
@@ -6,9 +6,12 @@
  *
  */
 #include "cache.h"
+#include "pkt-line.h"
 
 #define MAXNAME (256)
 
+static const char *dumpconfig = "git-dump-config";
+
 static FILE *config_file;
 static const char *config_file_name;
 static int config_linenr;
@@ -392,6 +395,29 @@ int git_config_from_file(config_fn_t fn, const char *filename)
 	return ret;
 }
 
+int git_config_from_remote(config_fn_t fn, char *dest)
+{
+	int ret;
+	int fd[2];
+	pid_t pid;
+	static char var[MAXNAME];
+	static char value[1024];
+
+	pid = git_connect(fd, dest, dumpconfig);
+	if (pid < 0)
+		return 1;
+	ret = 0;
+	while (packet_read_line(fd[0], var, sizeof(var))) {
+		if (!packet_read_line(fd[0], value, sizeof(value)))
+			die("Missing value");
+		fn(var, value);
+	}
+	close(fd[0]);
+	close(fd[1]);
+	ret |= finish_connect(pid);
+	return !!ret;
+}
+
 int git_config(config_fn_t fn)
 {
 	int ret = 0;
-- 
1.5.2.rc1.25.g889f-dirty

[PATCH 3/5] http.h: make fill_active_slots a function pointer

[Sven Verdoolaege]

From: Sven Verdoolaege <skimo@kotnet.org>

This allows us to use the methods provided by http.c
from within libgit, in particular config.c.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 http-fetch.c |    5 ++++-
 http-push.c  |    5 ++++-
 http.h       |    2 +-
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/http-fetch.c b/http-fetch.c
index 09baedc..53fb2a9 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -317,7 +317,7 @@ static void release_object_request(struct object_request *obj_req)
 }
 
 #ifdef USE_CURL_MULTI
-void fill_active_slots(void)
+static void fetch_fill_active_slots(void)
 {
 	struct object_request *obj_req = object_queue_head;
 	struct active_request_slot *slot = active_queue_head;
@@ -1031,6 +1031,9 @@ int main(int argc, const char **argv)
 	}
 	url = argv[arg];
 
+#ifdef USE_CURL_MULTI
+	fill_active_slots = fetch_fill_active_slots;
+#endif
 	http_init();
 
 	no_pragma_header = curl_slist_append(no_pragma_header, "Pragma:");
diff --git a/http-push.c b/http-push.c
index e3f7675..d4c850b 100644
--- a/http-push.c
+++ b/http-push.c
@@ -794,7 +794,7 @@ static void finish_request(struct transfer_request *request)
 }
 
 #ifdef USE_CURL_MULTI
-void fill_active_slots(void)
+static void push_fill_active_slots(void)
 {
 	struct transfer_request *request = request_queue_head;
 	struct transfer_request *next;
@@ -2355,6 +2355,9 @@ int main(int argc, char **argv)
 
 	memset(remote_dir_exists, -1, 256);
 
+#ifdef USE_CURL_MULTI
+	fill_active_slots = push_fill_active_slots;
+#endif
 	http_init();
 
 	no_pragma_header = curl_slist_append(no_pragma_header, "Pragma:");
diff --git a/http.h b/http.h
index 69b6b66..7a41cde 100644
--- a/http.h
+++ b/http.h
@@ -69,7 +69,7 @@ extern void finish_all_active_slots(void);
 extern void release_active_slot(struct active_request_slot *slot);
 
 #ifdef USE_CURL_MULTI
-extern void fill_active_slots(void);
+extern void (*fill_active_slots)(void);
 extern void step_active_slots(void);
 #endif
 
-- 
1.5.2.rc1.25.g889f-dirty

[PATCH 4/5] git-config: read remote config files over HTTP

[Sven Verdoolaege]

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 Makefile           |    6 +++++-
 config.c           |   14 ++++++++++++++
 http.c             |   10 ++++++++--
 http_config.h      |    1 +
 http_config_curl.c |   49 +++++++++++++++++++++++++++++++++++++++++++++++++
 http_config_none.c |    6 ++++++
 6 files changed, 83 insertions(+), 3 deletions(-)
 create mode 100644 http_config.h
 create mode 100644 http_config_curl.c
 create mode 100644 http_config_none.c

diff --git a/Makefile b/Makefile
index 0185386..b782111 100644
--- a/Makefile
+++ b/Makefile
@@ -311,7 +311,7 @@ LIB_OBJS = \
 	write_or_die.o trace.o list-objects.o grep.o match-trees.o \
 	alloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \
 	color.o wt-status.o archive-zip.o archive-tar.o shallow.o utf8.o \
-	convert.o attr.o decorate.o progress.o mailmap.o
+	convert.o attr.o decorate.o progress.o mailmap.o $(HTTP_CONFIG_OBJ)
 
 BUILTIN_OBJS = \
 	builtin-add.o \
@@ -518,6 +518,10 @@ ifndef NO_CURL
 	ifndef NO_EXPAT
 		EXPAT_LIBEXPAT = -lexpat
 	endif
+	HTTP_CONFIG_OBJ = http_config_curl.o http.o
+	EXTLIBS += $(CURL_LIBCURL)
+else
+	HTTP_CONFIG_OBJ = http_config_none.o
 endif
 
 ifndef NO_OPENSSL
diff --git a/config.c b/config.c
index 0da74e0..36e3b97 100644
--- a/config.c
+++ b/config.c
@@ -7,6 +7,7 @@
  */
 #include "cache.h"
 #include "pkt-line.h"
+#include "http_config.h"
 
 #define MAXNAME (256)
 
@@ -395,6 +396,16 @@ int git_config_from_file(config_fn_t fn, const char *filename)
 	return ret;
 }
 
+static int config_from_http(config_fn_t fn, char *dest)
+{
+	static char *config_temp = "config.temp";
+	if (git_http_fetch_config(dest, config_temp))
+		return 1;
+	git_config_from_file(fn, config_temp);
+	unlink(config_temp);
+	return 0;
+}
+
 int git_config_from_remote(config_fn_t fn, char *dest)
 {
 	int ret;
@@ -403,6 +414,9 @@ int git_config_from_remote(config_fn_t fn, char *dest)
 	static char var[MAXNAME];
 	static char value[1024];
 
+	if (!prefixcmp(dest, "http://"))
+		return config_from_http(fn, dest);
+
 	pid = git_connect(fd, dest, dumpconfig);
 	if (pid < 0)
 		return 1;
diff --git a/http.c b/http.c
index ae27e0c..3e1ccce 100644
--- a/http.c
+++ b/http.c
@@ -25,6 +25,10 @@ long curl_low_speed_limit = -1;
 long curl_low_speed_time = -1;
 int curl_ftp_no_epsv = 0;
 
+#ifdef USE_CURL_MULTI
+void (*fill_active_slots)(void) = NULL;
+#endif
+
 struct curl_slist *pragma_header;
 
 struct active_request_slot *active_queue_head = NULL;
@@ -394,7 +398,8 @@ void step_active_slots(void)
 	} while (curlm_result == CURLM_CALL_MULTI_PERFORM);
 	if (num_transfers < active_requests) {
 		process_curl_messages();
-		fill_active_slots();
+		if (fill_active_slots)
+			fill_active_slots();
 	}
 }
 #endif
@@ -459,7 +464,8 @@ void release_active_slot(struct active_request_slot *slot)
 		slot->curl = NULL;
 	}
 #ifdef USE_CURL_MULTI
-	fill_active_slots();
+	if (fill_active_slots)
+		fill_active_slots();
 #endif
 }
 
diff --git a/http_config.h b/http_config.h
new file mode 100644
index 0000000..0fddf98
--- /dev/null
+++ b/http_config.h
@@ -0,0 +1 @@
+int git_http_fetch_config(const char *repo, const char *config_file);
diff --git a/http_config_curl.c b/http_config_curl.c
new file mode 100644
index 0000000..3047ea2
--- /dev/null
+++ b/http_config_curl.c
@@ -0,0 +1,49 @@
+#include "http_config.h"
+#include "http.h"
+
+int git_http_fetch_config(const char *repo, const char *config)
+{
+	char url[PATH_MAX];
+	int len = strlen(repo);
+
+	FILE *configfile;
+	struct active_request_slot *slot;
+	struct slot_results results;
+
+	strcpy(url, repo);
+	while (len > 0 && url[len-1] == '/')
+		--len;
+	snprintf(url+len, sizeof(url)-len, "/config");
+
+	configfile = fopen(config, "w");
+	if (!configfile)
+		return error("Unable to open local file %s for config",
+			     config);
+
+	http_init();
+
+	slot = get_active_slot();
+	slot->results = &results;
+	curl_easy_setopt(slot->curl, CURLOPT_FILE, configfile);
+	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
+	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+	slot->local = configfile;
+
+	if (start_active_slot(slot)) {
+		run_active_slot(slot);
+		if (results.curl_result != CURLE_OK) {
+			fclose(configfile);
+			return error("Unable to get config %s\n%s", url,
+				     curl_errorstr);
+		}
+	} else {
+		fclose(configfile);
+		return error("Unable to start request");
+	}
+
+	http_cleanup();
+
+	fclose(configfile);
+
+	return 0;
+}
diff --git a/http_config_none.c b/http_config_none.c
new file mode 100644
index 0000000..303160b
--- /dev/null
+++ b/http_config_none.c
@@ -0,0 +1,6 @@
+#include "http_config.h"
+
+int git_http_fetch_config(const char *repo, const char *config_file)
+{
+	return error("Reading http config files not supported");
+}
-- 
1.5.2.rc1.25.g889f-dirty

[PATCH 5/5] git-clone: add --submodules for cloning submodules

[Sven Verdoolaege]

From: Sven Verdoolaege <skimo@kotnet.org>

When the --submodules option is specified, git-clone will search
for submodule.<submodule>.url options in the remote configuration
and clone each submodule using the first url that it can use from
the local site.

The submodules are currently not checked out.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 Documentation/config.txt    |    3 ++
 Documentation/git-clone.txt |    6 +++-
 git-clone.sh                |   68 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 75 insertions(+), 2 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 24f9655..92747d8 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -597,6 +597,9 @@ showbranch.default::
 	The default set of branches for gitlink:git-show-branch[1].
 	See gitlink:git-show-branch[1].
 
+submodule.<submodule>.url
+	The URL of a submodule.  See gitlink:git-clone[1].
+
 tar.umask::
 	By default, gitlink:git-tar-tree[1] sets file and directories modes
 	to 0666 or 0777. While this is both useful and acceptable for projects
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 6d32c49..b112a6a 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -11,7 +11,7 @@ SYNOPSIS
 [verse]
 'git-clone' [--template=<template_directory>] [-l [-s]] [-q] [-n] [--bare]
 	  [-o <name>] [-u <upload-pack>] [--reference <repository>]
-	  [--depth <depth>] <repository> [<directory>]
+	  [--depth <depth>] [--submodules] <repository> [<directory>]
 
 DESCRIPTION
 -----------
@@ -105,6 +105,10 @@ OPTIONS
 	with a long history, and would want to send in a fixes
 	as patches.
 
+--submodules::
+	Clone submodules specified in (remote) configuration parameters
+	submodule.<submodule>.url.
+
 <repository>::
 	The (possibly remote) repository to clone from.  It can
 	be any URL git-fetch supports.
diff --git a/git-clone.sh b/git-clone.sh
index cad5c0c..3a9b09c 100755
--- a/git-clone.sh
+++ b/git-clone.sh
@@ -14,7 +14,7 @@ die() {
 }
 
 usage() {
-	die "Usage: $0 [--template=<template_directory>] [--reference <reference-repo>] [--bare] [-l [-s]] [-q] [-u <upload-pack>] [--origin <name>] [--depth <n>] [-n] <repo> [<dir>]"
+	die "Usage: $0 [--template=<template_directory>] [--reference <reference-repo>] [--bare] [-l [-s]] [-q] [-u <upload-pack>] [--origin <name>] [--depth <n>] [-n] [--submodules] <repo> [<dir>]"
 }
 
 get_repo_base() {
@@ -67,6 +67,60 @@ Perhaps git-update-server-info needs to be run there?"
 	rm -f "$GIT_DIR/REMOTE_HEAD"
 }
 
+local_URL() {
+	# tranforms a "URL" on the remote to a URL that works on the local machine
+	# $1 - remote, $2 - URL on remote
+	case "$1" in
+	https://*|http://*|ftp://*)
+		case "$2" in
+		https://*|http://*|ftp://*)
+			echo $2
+		esac
+		;;
+	ssh://*)
+		case "$2" in
+		https://*|http://*|ftp://*)
+			echo $2
+			;;
+		/*)
+			echo $(echo $1 | sed -e 's/\(ssh:\/\/[^\/]*\)\/.*/\1/')$2
+		esac
+		;;
+	/*)
+		echo $2
+		;;
+	*)
+		case "$2" in
+		https://*|http://*|ftp://*)
+			echo $2
+		esac
+	esac
+}
+
+clone_submodules () {
+	# $1 - remote
+	previous=
+	git-config --remote=$1 --get-regexp 'submodule\..*\.url' | \
+	sed -e 's/^submodule\.//;s/\.url / /' |
+	while read submodule URL
+	do
+		echo "$submodule $URL"
+		if test "$submodule" = "$previous"
+		then
+			continue;
+		fi
+		URL=$(local_URL "$1" "$URL")
+		echo "$submodule $URL"
+		if test -z "$URL"
+		then
+			continue;
+		fi
+		git-clone -n "$URL" "$submodule"
+		git-config "submodule.$submodule.url" "$URL"
+		previous="$submodule"
+	done
+}
+
 quiet=
 local=no
 use_local=no
@@ -81,6 +135,7 @@ origin_override=
 use_separate_remote=t
 depth=
 no_progress=
+submodules=
 test -t 1 || no_progress=--no-progress
 while
 	case "$#,$1" in
@@ -131,6 +186,8 @@ while
 	*,--depth)
 		shift
 		depth="--depth=$1";;
+	*,--su|*,--sub|*,--subm|*,--submo|*,--submod|*,--submodu|*,--submodul|\
+	*,--submodule|*,--submodules) submodules=yes ;;
 	*,-*) usage ;;
 	*) break ;;
 	esac
@@ -149,6 +206,10 @@ then
 	then
 		die '--bare and --origin $origin options are incompatible.'
 	fi
+	if test yes = "$submodules"
+	then
+		die '--bare and --submodules origin options are incompatible.'
+	fi
 	no_checkout=yes
 	use_separate_remote=
 fi
@@ -394,6 +455,11 @@ then
 		git-config branch."$head_points_at".merge "refs/heads/$head_points_at"
 	esac
 
+	if test yes = "$submodules"
+	then
+		clone_submodules "$repo"
+	fi
+
 	case "$no_checkout" in
 	'')
 		test "z$quiet" = z -a "z$no_progress" = z && v=-v || v=
-- 
1.5.2.rc1.25.g889f-dirty

Re: [PATCH 2/5] git-config: add --remote option for reading config from remote repo

[Frank Lichtenheld]

Some comments on the documentation:

On Fri, May 04, 2007 at 12:56:40PM +0200, Sven Verdoolaege wrote:
> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] name [value [value_regex]]

maybe something more like

'git-config' [ scope ]

where you mention later that scope can be --system | --global | --remote
would be more readable...

> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] --add name value
> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] --replace-all name [value [value_regex]]
> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] --get name [value_regex]
> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] --get-all name [value_regex]
> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] --unset name [value_regex]
> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] --unset-all name [value_regex]
> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] --rename-section old_name new_name
> +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> +	     [type] --remove-section name
> +'git-config' [--system | --global | --remote=[<host>:]<directory ] -l | --list
>  
>  DESCRIPTION
>  -----------
> @@ -80,6 +89,10 @@ OPTIONS
>  	Use system-wide $(prefix)/etc/gitconfig rather than the repository
>  	.git/config.
>  
> +--remote=[<host>:]<directory
> +	Use remote config instead of the repository .git/config.
> +	Only available for reading options.

Why did you add it to all options in the SYNOPSYS then?

Gruesse,
-- 
Frank Lichtenheld <frank@lichtenheld.de>
www: http://www.djpig.de/

Re: [PATCH 2/5] git-config: add --remote option for reading config from remote repo

[Sven Verdoolaege]

On Fri, May 04, 2007 at 11:03:40PM +0200, Frank Lichtenheld wrote:
> 
> Some comments on the documentation:
> 
> On Fri, May 04, 2007 at 12:56:40PM +0200, Sven Verdoolaege wrote:
> > +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> > +	     [type] name [value [value_regex]]
> 
> maybe something more like
> 
> 'git-config' [ scope ]

Sounds reasonable, although I'd probably say [<scope>].

> > +--remote=[<host>:]<directory
> > +	Use remote config instead of the repository .git/config.
> > +	Only available for reading options.
> 
> Why did you add it to all options in the SYNOPSYS then?

Because I wasn't thinking.

skimo

Re: [PATCH 2/5] git-config: add --remote option for reading config from remote repo

[Frank Lichtenheld]

On Fri, May 04, 2007 at 11:10:05PM +0200, Sven Verdoolaege wrote:
> On Fri, May 04, 2007 at 11:03:40PM +0200, Frank Lichtenheld wrote:
> > 
> > Some comments on the documentation:
> > 
> > On Fri, May 04, 2007 at 12:56:40PM +0200, Sven Verdoolaege wrote:
> > > +'git-config' [--system | --global | --remote=[<host>:]<directory ]
> > > +	     [type] name [value [value_regex]]
> > 
> > maybe something more like
> > 
> > 'git-config' [ scope ]
> 
> Sounds reasonable, although I'd probably say [<scope>].

Note that this is inconsistent with the rest of the documentation
(e.g. it says [type], not [<type>]). 

> > > +--remote=[<host>:]<directory
> > > +	Use remote config instead of the repository .git/config.
> > > +	Only available for reading options.
> > 
> > Why did you add it to all options in the SYNOPSYS then?
> 
> Because I wasn't thinking.

Another small error: You seem to be missing a '>' after each occourence of 
'<directory'.

Gruesse,
-- 
Frank Lichtenheld <frank@lichtenheld.de>
www: http://www.djpig.de/

Re: Initial support for cloning submodules

[Junio C Hamano]

The plumbing part looks very good, although I sensed a slight
slop toward the end (will comment on individual patches later).

I do not like the Porcelain part very much, though.  I do not
think we would want to add anything new to git-clone.  We should
lose as much code from git-clone that is common with git-fetch
as we can first, and add new features to git-fetch, with
possibly passthru options added to git-clone as needed (e.g. a
new --submodule option).

If you --submodule cloned a remote repository when it had two
submodules, and then later the remote adds another submodule,
you would need to have a way to fetch that can discover the
presense of the new submodule and add it for you, and at that
point, having the code that knows much about submodules in clone
would not help you much.

I suspect that a possible interaction between git-fetch and
git-clone would go like this:

 (1) "git-clone [--submodules]" would perform a normal clone,
     having most of its work done by git-remote and git-fetch;

 (2) when "--submodules" is given to "git-clone", it passes it
     through to underlying "git-fetch";

 (3) "git-fetch --submodules", after finishing what it would do
     without "--submodules" option, would inspect the fetched
     tree (or the index derived from it), find the tree entries
     with mode 160000 (i.e. submodule graft points), and _then_
     uses the pathnames of these tree entries to consult the
     config mechanism to see which URL(s) can be used to
     retrieve them, probably only for new submodules.  Your new
     "config --remote" mechanism may be one good way to prime
     the configuration from the originating site.

We could probably extend the upload-pack protocol to send the
same information as you are using submodules.*.url for, instead
of adding a yet another protocol and program pairs you used for
"git config --remote".  Having a generic program and protocol to
dump the whole configuration file is certainly simpler, easier
to debug, and easier to repurpose, it makes me somewhat worried
about security implications (if it is open to http then worrying
about it is not very useful, though).

Re: Initial support for cloning submodules

[Sven Verdoolaege]

On Fri, May 04, 2007 at 03:52:15PM -0700, Junio C Hamano wrote:
> I do not like the Porcelain part very much, though.  I do not
> think we would want to add anything new to git-clone.  We should
> lose as much code from git-clone that is common with git-fetch
> as we can first, and add new features to git-fetch, with
> possibly passthru options added to git-clone as needed (e.g. a
> new --submodule option).

So what would you want to keep in git-clone ?

> If you --submodule cloned a remote repository when it had two
> submodules, and then later the remote adds another submodule,
> you would need to have a way to fetch that can discover the
> presense of the new submodule and add it for you, and at that
> point, having the code that knows much about submodules in clone
> would not help you much.

True.

>  (3) "git-fetch --submodules", after finishing what it would do
>      without "--submodules" option, would inspect the fetched
>      tree (or the index derived from it), find the tree entries
>      with mode 160000 (i.e. submodule graft points), and _then_
>      uses the pathnames of these tree entries to consult the
>      config mechanism to see which URL(s) can be used to
>      retrieve them, probably only for new submodules.

Would git-fetch then call git-clone for these new submodules?

> Having a generic program and protocol to
> dump the whole configuration file is certainly simpler, easier
> to debug, and easier to repurpose, it makes me somewhat worried
> about security implications (if it is open to http then worrying
> about it is not very useful, though).

We could easily have dump-config only dump a predefined "known safe"
set of config options, although that would mean you have to upgrade
the server side each time you add a new dumpable config option.
Or we could do the preselection only when called from git-daemon.

skimo

Re: Initial support for cloning submodules

[Junio C Hamano]

Sven Verdoolaege <skimo@kotnet.org> writes:

> On Fri, May 04, 2007 at 03:52:15PM -0700, Junio C Hamano wrote:
>> I do not like the Porcelain part very much, though.  I do not
>> think we would want to add anything new to git-clone.  We should
>> lose as much code from git-clone that is common with git-fetch
>> as we can first, and add new features to git-fetch, with
>> possibly passthru options added to git-clone as needed (e.g. a
>> new --submodule option).
>
> So what would you want to keep in git-clone ?

 - Figuring out the name of the new directory we create (think
   "git clone git://repo.or.cz/git.git" -- it does "mkdir git &&
   cd git" for you before doing the other things).

 - Run "git init" there, obviously.

 - Probably run "git ls-remote" to figure out which branch HEAD
   should point at; you would definitely want to add a mechanism
   to pass the ls-remote result to the "git fetch" you are going
   to call next, as it is the first thing "git fetch" usually
   does -- we would want to reuse it.

 - Recently suggested good addition is to have "--track $branch"
   option to "git clone" to point remotes/origin/HEAD to
   something other than what the remote's HEAD actually points
   at (the discussion was primarily between Carl Worth and
   Linus; see archive).

 - Run "git remote add origin" with the given URL to set up the
   standard "separate remotes" tracking structure.

 - Run "git fetch" for initial fetch ("git remote add -f" could
   do that as well).  You would probably need to pass --tags to
   this "git fetch" to mimick what "git clone" does today.

 - Run initial checkout if asked.

>>  (3) "git-fetch --submodules", after finishing what it would do
>>      without "--submodules" option, would inspect the fetched
>>      tree (or the index derived from it), find the tree entries
>>      with mode 160000 (i.e. submodule graft points), and _then_
>>      uses the pathnames of these tree entries to consult the
>>      config mechanism to see which URL(s) can be used to
>>      retrieve them, probably only for new submodules.
>
> Would git-fetch then call git-clone for these new submodules?

Most likely yes but that is just my gut feeling -- I readily
admit I haven't thought it through.

Re: Initial support for cloning submodules

[Alon Ziv]

On Sat, 2007-05-05 at 10:14 +0200, Sven Verdoolaege wrote:
> We could easily have dump-config only dump a predefined "known safe"
> set of config options, although that would mean you have to upgrade
> the server side each time you add a new dumpable config option.
> Or we could do the preselection only when called from git-daemon.
> 

Or we could have the set of dumpable options itself in the config file;
a bit more cumbersome, but leaves all flexibility (including the
shoot-yourself-in-the-foot kind) in the hands of the users :)

	-az

Re: Initial support for cloning submodules

[Sven Verdoolaege]

On Fri, May 04, 2007 at 03:52:15PM -0700, Junio C Hamano wrote:
>  (3) "git-fetch --submodules", after finishing what it would do
>      without "--submodules" option, would inspect the fetched
>      tree (or the index derived from it), find the tree entries
>      with mode 160000 (i.e. submodule graft points), and _then_
>      uses the pathnames of these tree entries to consult the
>      config mechanism to see which URL(s) can be used to
>      retrieve them, probably only for new submodules.

I've gone for cloning all available submodules on the remote,
even if they are not used in the HEAD.
A submodule may have been removed already and IMHO, you wouldn't
want to clone a submodule at the time you reset to an intermediate
commit.

skimo