* [PATCH 0/7] First class shallow clone
@ 2013-07-17 12:47 Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 1/7] transport.h: remove send_pack prototype, already defined in send-pack.h Nguyễn Thái Ngọc Duy
` (7 more replies)
0 siblings, 8 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-17 12:47 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
This is probably the first attempt to treat shallow clones just like
ordinary ones. Which means you can push or fetch/clone between any two
repos, regardless of their shallow status. There are two purposes
behind this:
- to make local/shallow clone <-> (complete) upstream repo workflow
smoother, the note about shallow clone limitation in git-clone.txt
can be removed
- to make it possible for upstream to provide a lightweight repo that
others can use. For example, big repos with lots of activities can
be split into new base repo that only contains the work of maximum one
year and a complete repo mostly for archive.
This is a naive approach. I might overlook something again, which is
why I publish it early to get more eyes on it.
The idea is simple: in shallow case, we provide the pack _and_
.git/shallow file to the other end. The other end will setup extra
grafting to make sure the updated repo is "complete". More in
individual patch messages.
There might be issues with generating optimum pack for transfer when
both ends are shallow.. There's also an interesting issue, whether we
can take advantage of commit bitmaps in shallow clones if they are
more widely used.. We also might need a config key to protect a repo
from becoming shallow by a fetch or push, if the repo is to be a
backup one..
Nguyễn Thái Ngọc Duy (7):
transport.h: remove send_pack prototype, already defined in send-pack.h
{receive,upload}-pack: advertise shallow graft information
connect.c: teach get_remote_heads to parse "shallow" lines
Move setup_alternate_shallow and write_shallow_commits to shallow.c
fetch-pack: support fetching from a shallow repository
{send,receive}-pack: support pushing from a shallow clone
send-pack: support pushing to a shallow clone
Documentation/technical/pack-protocol.txt | 7 +-
builtin/fetch-pack.c | 6 +-
builtin/receive-pack.c | 54 ++++++++++++----
builtin/send-pack.c | 7 +-
cache.h | 1 +
commit.h | 8 +++
connect.c | 12 +++-
fetch-pack.c | 78 +++++++---------------
fetch-pack.h | 1 +
remote-curl.c | 2 +-
send-pack.c | 41 ++++++++++--
send-pack.h | 4 +-
shallow.c | 103 ++++++++++++++++++++++++++++++
t/t5536-fetch-shallow.sh (new +x) | 75 ++++++++++++++++++++++
t/t5537-push-shallow.sh (new +x) | 85 ++++++++++++++++++++++++
transport.c | 14 ++--
transport.h | 6 --
upload-pack.c | 3 +-
18 files changed, 414 insertions(+), 93 deletions(-)
create mode 100755 t/t5536-fetch-shallow.sh
create mode 100755 t/t5537-push-shallow.sh
--
1.8.2.83.gc99314b
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH 1/7] transport.h: remove send_pack prototype, already defined in send-pack.h
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
@ 2013-07-17 12:47 ` Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 2/7] {receive,upload}-pack: advertise shallow graft information Nguyễn Thái Ngọc Duy
` (6 subsequent siblings)
7 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-17 12:47 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
transport.h | 6 ------
1 file changed, 6 deletions(-)
diff --git a/transport.h b/transport.h
index ea70ea7..3178dc9 100644
--- a/transport.h
+++ b/transport.h
@@ -182,10 +182,4 @@ void transport_print_push_status(const char *dest, struct ref *refs,
typedef void alternate_ref_fn(const struct ref *, void *);
extern void for_each_alternate_ref(alternate_ref_fn, void *);
-
-struct send_pack_args;
-extern int send_pack(struct send_pack_args *args,
- int fd[], struct child_process *conn,
- struct ref *remote_refs,
- struct extra_have_objects *extra_have);
#endif
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH 2/7] {receive,upload}-pack: advertise shallow graft information
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 1/7] transport.h: remove send_pack prototype, already defined in send-pack.h Nguyễn Thái Ngọc Duy
@ 2013-07-17 12:47 ` Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 3/7] connect.c: teach get_remote_heads to parse "shallow" lines Nguyễn Thái Ngọc Duy
` (5 subsequent siblings)
7 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-17 12:47 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
If either receive-pack or upload-pack is called on a shallow
repository, shallow graft points will be sent after the ref
advertisement (but before the packet flush).
This breaks the protocol for all clients trying to push to a shallow
repo, or fetch from one. Which is basically the same end result as
today's "is_repository_shallow() && die()" in receive-pack and
upload-pack. New clients will be made aware of shallow upstream and
can make use of this information.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/technical/pack-protocol.txt | 3 +++
builtin/receive-pack.c | 5 ++---
commit.h | 1 +
shallow.c | 16 ++++++++++++++++
upload-pack.c | 3 +--
5 files changed, 23 insertions(+), 5 deletions(-)
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index b898e97..eb8edd1 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag.
----
advertised-refs = (no-refs / list-of-refs)
+ *shallow
flush-pkt
no-refs = PKT-LINE(zero-id SP "capabilities^{}"
@@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag.
other-tip = obj-id SP refname LF
other-peeled = obj-id SP refname "^{}" LF
+ shallow = PKT-LINE("shallow" SP obj-id)
+
capability-list = capability *(SP capability)
capability = 1*(LC_ALPHA / DIGIT / "-" / "_")
LC_ALPHA = %x61-7A
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index e3eb5fc..2d8e19b 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -176,6 +176,8 @@ static void write_head_info(void)
if (!sent_capabilities)
show_ref("capabilities^{}", null_sha1);
+ advertise_shallow_grafts(1);
+
/* EOF */
packet_flush(1);
}
@@ -990,9 +992,6 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
if (!enter_repo(dir, 0))
die("'%s' does not appear to be a git repository", dir);
- if (is_repository_shallow())
- die("attempt to push into a shallow repository");
-
git_config(receive_pack_config, NULL);
if (0 <= transfer_unpack_limit)
diff --git a/commit.h b/commit.h
index 4d452dc..e0688c3 100644
--- a/commit.h
+++ b/commit.h
@@ -187,6 +187,7 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
int depth, int shallow_flag, int not_shallow_flag);
extern void check_shallow_file_for_update(void);
extern void set_alternate_shallow_file(const char *path);
+extern void advertise_shallow_grafts(int);
int is_descendant_of(struct commit *, struct commit_list *);
int in_merge_bases(struct commit *, struct commit *);
diff --git a/shallow.c b/shallow.c
index cbe2526..ccdfefc 100644
--- a/shallow.c
+++ b/shallow.c
@@ -1,6 +1,7 @@
#include "cache.h"
#include "commit.h"
#include "tag.h"
+#include "pkt-line.h"
static int is_shallow = -1;
static struct stat shallow_stat;
@@ -146,3 +147,18 @@ void check_shallow_file_for_update(void)
)
die("shallow file was changed during fetch");
}
+
+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)
+{
+ int fd = *(int*)cb;
+ if (graft->nr_parent == -1)
+ packet_write(fd, "shallow %s\n", sha1_to_hex(graft->sha1));
+ return 0;
+}
+
+void advertise_shallow_grafts(int fd)
+{
+ if (!is_repository_shallow())
+ return;
+ for_each_commit_graft(advertise_shallow_grafts_cb, &fd);
+}
diff --git a/upload-pack.c b/upload-pack.c
index 127e59a..6cefe43 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -766,6 +766,7 @@ static void upload_pack(void)
reset_timeout();
head_ref_namespaced(send_ref, NULL);
for_each_namespaced_ref(send_ref, NULL);
+ advertise_shallow_grafts(1);
packet_flush(1);
} else {
head_ref_namespaced(mark_our_ref, NULL);
@@ -837,8 +838,6 @@ int main(int argc, char **argv)
if (!enter_repo(dir, strict))
die("'%s' does not appear to be a git repository", dir);
- if (is_repository_shallow())
- die("attempt to fetch/clone from a shallow repository");
git_config(upload_pack_config, NULL);
upload_pack();
return 0;
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH 3/7] connect.c: teach get_remote_heads to parse "shallow" lines
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 1/7] transport.h: remove send_pack prototype, already defined in send-pack.h Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 2/7] {receive,upload}-pack: advertise shallow graft information Nguyễn Thái Ngọc Duy
@ 2013-07-17 12:47 ` Nguyễn Thái Ngọc Duy
2013-07-20 3:27 ` Junio C Hamano
2013-07-17 12:47 ` [PATCH 4/7] Move setup_alternate_shallow and write_shallow_commits to shallow.c Nguyễn Thái Ngọc Duy
` (4 subsequent siblings)
7 siblings, 1 reply; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-17 12:47 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
No callers pass a non-empty pointer as shallow_points at this
stage. As a result, all clients still refuse to talk to shallow
repository on the other end.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/fetch-pack.c | 2 +-
builtin/send-pack.c | 2 +-
cache.h | 1 +
connect.c | 12 +++++++++++-
remote-curl.c | 2 +-
transport.c | 7 ++++---
6 files changed, 19 insertions(+), 7 deletions(-)
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index aba4465..080e599 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -144,7 +144,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
args.verbose ? CONNECT_VERBOSE : 0);
}
- get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);
+ get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
ref = fetch_pack(&args, fd, conn, ref, dest,
sought, nr_sought, pack_lockfile_ptr);
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 152c4ea..f0ad0ce 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -207,7 +207,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
memset(&extra_have, 0, sizeof(extra_have));
- get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);
+ get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);
transport_verify_remote_names(nr_refspecs, refspecs);
diff --git a/cache.h b/cache.h
index dd0fb33..7665e03 100644
--- a/cache.h
+++ b/cache.h
@@ -1091,6 +1091,7 @@ struct extra_have_objects {
};
extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
struct ref **list, unsigned int flags,
+ struct extra_have_objects *,
struct extra_have_objects *);
extern int server_supports(const char *feature);
extern int parse_feature_request(const char *features, const char *feature);
diff --git a/connect.c b/connect.c
index a0783d4..1cba424 100644
--- a/connect.c
+++ b/connect.c
@@ -64,7 +64,8 @@ static void die_initial_contact(int got_at_least_one_head)
*/
struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
struct ref **list, unsigned int flags,
- struct extra_have_objects *extra_have)
+ struct extra_have_objects *extra_have,
+ struct extra_have_objects *shallow_points)
{
int got_at_least_one_head = 0;
@@ -89,6 +90,15 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
if (len > 4 && !prefixcmp(buffer, "ERR "))
die("remote error: %s", buffer + 4);
+ if (len == 48 && !prefixcmp(buffer, "shallow ")) {
+ if (get_sha1_hex(buffer + 8, old_sha1))
+ die("protocol error: expected shallow sha, got '%s'", buffer + 8);
+ if (!shallow_points)
+ die("repository on the other end cannot be shallow");
+ add_extra_have(shallow_points, old_sha1);
+ continue;
+ }
+
if (len < 42 || get_sha1_hex(buffer, old_sha1) || buffer[40] != ' ')
die("protocol error: expected sha/ref, got '%s'", buffer);
name = buffer + 41;
diff --git a/remote-curl.c b/remote-curl.c
index 5b3ce9e..c329bd3 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -86,7 +86,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
{
struct ref *list = NULL;
get_remote_heads(-1, heads->buf, heads->len, &list,
- for_push ? REF_NORMAL : 0, NULL);
+ for_push ? REF_NORMAL : 0, NULL, NULL);
return list;
}
diff --git a/transport.c b/transport.c
index e15db98..10a8cb8 100644
--- a/transport.c
+++ b/transport.c
@@ -509,7 +509,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
connect_setup(transport, for_push, 0);
get_remote_heads(data->fd[0], NULL, 0, &refs,
- for_push ? REF_NORMAL : 0, &data->extra_have);
+ for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
data->got_remote_heads = 1;
return refs;
@@ -539,7 +539,8 @@ static int fetch_refs_via_pack(struct transport *transport,
if (!data->got_remote_heads) {
connect_setup(transport, 0, 0);
- get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0, NULL);
+ get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
+ NULL, NULL);
data->got_remote_heads = 1;
}
@@ -799,7 +800,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
struct ref *tmp_refs;
connect_setup(transport, 1, 0);
- get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL);
+ get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
data->got_remote_heads = 1;
}
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH 4/7] Move setup_alternate_shallow and write_shallow_commits to shallow.c
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
` (2 preceding siblings ...)
2013-07-17 12:47 ` [PATCH 3/7] connect.c: teach get_remote_heads to parse "shallow" lines Nguyễn Thái Ngọc Duy
@ 2013-07-17 12:47 ` Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 5/7] fetch-pack: support fetching from a shallow repository Nguyễn Thái Ngọc Duy
` (3 subsequent siblings)
7 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-17 12:47 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
commit.h | 3 +++
fetch-pack.c | 53 +----------------------------------------------------
shallow.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 57 insertions(+), 52 deletions(-)
diff --git a/commit.h b/commit.h
index e0688c3..678fa20 100644
--- a/commit.h
+++ b/commit.h
@@ -188,6 +188,9 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
extern void check_shallow_file_for_update(void);
extern void set_alternate_shallow_file(const char *path);
extern void advertise_shallow_grafts(int);
+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);
+extern void setup_alternate_shallow(struct lock_file *shallow_lock,
+ const char **alternate_shallow_file);
int is_descendant_of(struct commit *, struct commit_list *);
int in_merge_bases(struct commit *, struct commit *);
diff --git a/fetch-pack.c b/fetch-pack.c
index abe5ffb..dc71a2b 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -185,36 +185,6 @@ static void consume_shallow_list(struct fetch_pack_args *args, int fd)
}
}
-struct write_shallow_data {
- struct strbuf *out;
- int use_pack_protocol;
- int count;
-};
-
-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
-{
- struct write_shallow_data *data = cb_data;
- const char *hex = sha1_to_hex(graft->sha1);
- data->count++;
- if (data->use_pack_protocol)
- packet_buf_write(data->out, "shallow %s", hex);
- else {
- strbuf_addstr(data->out, hex);
- strbuf_addch(data->out, '\n');
- }
- return 0;
-}
-
-static int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
-{
- struct write_shallow_data data;
- data.out = out;
- data.use_pack_protocol = use_pack_protocol;
- data.count = 0;
- for_each_commit_graft(write_one_shallow, &data);
- return data.count;
-}
-
static enum ack_type get_ack(int fd, unsigned char *result_sha1)
{
int len;
@@ -795,27 +765,6 @@ static int cmp_ref_by_name(const void *a_, const void *b_)
return strcmp(a->name, b->name);
}
-static void setup_alternate_shallow(void)
-{
- struct strbuf sb = STRBUF_INIT;
- int fd;
-
- check_shallow_file_for_update();
- fd = hold_lock_file_for_update(&shallow_lock, git_path("shallow"),
- LOCK_DIE_ON_ERROR);
- if (write_shallow_commits(&sb, 0)) {
- if (write_in_full(fd, sb.buf, sb.len) != sb.len)
- die_errno("failed to write to %s", shallow_lock.filename);
- alternate_shallow_file = shallow_lock.filename;
- } else
- /*
- * is_repository_shallow() sees empty string as "no
- * shallow file".
- */
- alternate_shallow_file = "";
- strbuf_release(&sb);
-}
-
static struct ref *do_fetch_pack(struct fetch_pack_args *args,
int fd[2],
const struct ref *orig_ref,
@@ -896,7 +845,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
if (args->stateless_rpc)
packet_flush(fd[1]);
if (args->depth > 0)
- setup_alternate_shallow();
+ setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
if (get_pack(args, fd, pack_lockfile))
die("git fetch-pack: fetch failed.");
diff --git a/shallow.c b/shallow.c
index ccdfefc..ee9edd4 100644
--- a/shallow.c
+++ b/shallow.c
@@ -162,3 +162,56 @@ void advertise_shallow_grafts(int fd)
return;
for_each_commit_graft(advertise_shallow_grafts_cb, &fd);
}
+
+struct write_shallow_data {
+ struct strbuf *out;
+ int use_pack_protocol;
+ int count;
+};
+
+static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
+{
+ struct write_shallow_data *data = cb_data;
+ const char *hex = sha1_to_hex(graft->sha1);
+ data->count++;
+ if (data->use_pack_protocol)
+ packet_buf_write(data->out, "shallow %s", hex);
+ else {
+ strbuf_addstr(data->out, hex);
+ strbuf_addch(data->out, '\n');
+ }
+ return 0;
+}
+
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
+{
+ struct write_shallow_data data;
+ data.out = out;
+ data.use_pack_protocol = use_pack_protocol;
+ data.count = 0;
+ for_each_commit_graft(write_one_shallow, &data);
+ return data.count;
+}
+
+void setup_alternate_shallow(struct lock_file *shallow_lock,
+ const char **alternate_shallow_file)
+{
+ struct strbuf sb = STRBUF_INIT;
+ int fd;
+
+ check_shallow_file_for_update();
+ fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
+ LOCK_DIE_ON_ERROR);
+ if (write_shallow_commits(&sb, 0)) {
+ if (write_in_full(fd, sb.buf, sb.len) != sb.len)
+ die_errno("failed to write to %s",
+ shallow_lock->filename);
+ *alternate_shallow_file = shallow_lock->filename;
+ } else
+ /*
+ * is_repository_shallow() sees empty string as "no
+ * shallow file".
+ */
+ *alternate_shallow_file = "";
+ strbuf_release(&sb);
+}
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH 5/7] fetch-pack: support fetching from a shallow repository
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
` (3 preceding siblings ...)
2013-07-17 12:47 ` [PATCH 4/7] Move setup_alternate_shallow and write_shallow_commits to shallow.c Nguyễn Thái Ngọc Duy
@ 2013-07-17 12:47 ` Nguyễn Thái Ngọc Duy
2013-07-20 3:17 ` Junio C Hamano
2013-07-17 12:47 ` [PATCH 6/7] {send,receive}-pack: support pushing from a shallow clone Nguyễn Thái Ngọc Duy
` (2 subsequent siblings)
7 siblings, 1 reply; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-17 12:47 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
upload-pack already advertises all shallow grafts if server repository
is shallow. This information can be used to add more grafts to the
client if the server sends commit chains down to its graft points.
If the server is shallow, before we receive the pack, we setup a
temporary shallow file that contains both local graft points and the
server's. This stops index-pack from going beyond server's graft
points.
Only server graft points that do not have corresponding SHA-1s in
local repo are added to the temp shallow file because we don't want to
accidentally cut the client history because the server's is
shorter. The client cutting can only happen when --depth is requested.
After index-pack finishes successfully, we write the temporary shallow
down with one exception: unused graft points provided by the server
are removed. We don't want those lying around and suddenly become
active.
Note that in the "shallow -> shallow" case, the server might not have
enough information to find common roots to create an optimum pack. It
might send complete commit chains down to the graft points as a
result. I don't think we can improve this, unless upload-pack somehow
has access to a full repository.
"shallow -> shallow" case only makes sense when the upstream provides
a stable shallow repo (e.g. make a cut every year or so and ask devs
to all move to the new base). If the cloned repos are all based on a
stable (shallow) upstream, the above problem is unlikely to happen.
A side effect of this change is we can now clone from a shallow
repository. And a full repository may automatically become shallow if
you fetch from a shallow repository.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/fetch-pack.c | 6 ++--
commit.h | 8 +++--
fetch-pack.c | 27 +++++++++++---
fetch-pack.h | 1 +
shallow.c | 46 ++++++++++++++++++++----
t/t5536-fetch-shallow.sh (new +x) | 75 +++++++++++++++++++++++++++++++++++++++
transport.c | 8 +++--
7 files changed, 153 insertions(+), 18 deletions(-)
create mode 100755 t/t5536-fetch-shallow.sh
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 080e599..b89d753 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
char **pack_lockfile_ptr = NULL;
struct child_process *conn;
struct fetch_pack_args args;
+ struct extra_have_objects shallow;
packet_trace_identity("fetch-pack");
@@ -144,10 +145,11 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
args.verbose ? CONNECT_VERBOSE : 0);
}
- get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
+ memset(&shallow, 0, sizeof(shallow));
+ get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
ref = fetch_pack(&args, fd, conn, ref, dest,
- sought, nr_sought, pack_lockfile_ptr);
+ sought, nr_sought, &shallow, pack_lockfile_ptr);
if (pack_lockfile) {
printf("lock %s\n", pack_lockfile);
fflush(stdout);
diff --git a/commit.h b/commit.h
index 678fa20..7faf0e4 100644
--- a/commit.h
+++ b/commit.h
@@ -188,9 +188,13 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
extern void check_shallow_file_for_update(void);
extern void set_alternate_shallow_file(const char *path);
extern void advertise_shallow_grafts(int);
-extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);
+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+ struct extra_have_objects *extra,
+ int remove_unused_grafts);
extern void setup_alternate_shallow(struct lock_file *shallow_lock,
- const char **alternate_shallow_file);
+ const char **alternate_shallow_file,
+ struct extra_have_objects *extra,
+ int rewrite);
int is_descendant_of(struct commit *, struct commit_list *);
int in_merge_bases(struct commit *, struct commit *);
diff --git a/fetch-pack.c b/fetch-pack.c
index dc71a2b..68b95a5 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args *args,
}
if (is_repository_shallow())
- write_shallow_commits(&req_buf, 1);
+ write_shallow_commits(&req_buf, 1, NULL, 0);
if (args->depth > 0)
packet_buf_write(&req_buf, "deepen %d", args->depth);
packet_buf_flush(&req_buf);
@@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
int fd[2],
const struct ref *orig_ref,
struct ref **sought, int nr_sought,
+ struct extra_have_objects *shallow,
char **pack_lockfile)
{
struct ref *ref = copy_ref_list(orig_ref);
@@ -844,8 +845,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
if (args->stateless_rpc)
packet_flush(fd[1]);
- if (args->depth > 0)
- setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
+ if (args->depth > 0 || shallow->nr)
+ setup_alternate_shallow(&shallow_lock, &alternate_shallow_file,
+ shallow, 0);
if (get_pack(args, fd, pack_lockfile))
die("git fetch-pack: fetch failed.");
@@ -922,6 +924,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
const struct ref *ref,
const char *dest,
struct ref **sought, int nr_sought,
+ struct extra_have_objects *shallow,
char **pack_lockfile)
{
struct ref *ref_cpy;
@@ -934,14 +937,28 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
packet_flush(fd[1]);
die("no matching remote head");
}
- ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);
+ ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
+ shallow, pack_lockfile);
if (alternate_shallow_file) {
if (*alternate_shallow_file == '\0') { /* --unshallow */
unlink_or_warn(git_path("shallow"));
rollback_lock_file(&shallow_lock);
- } else
+ } else {
+ /*
+ * The server is a shallow clone and it sends
+ * us all of its shallow grafts. Some may be
+ * needed if the server sends objects down to
+ * the bottom. Remove all unused grafts.
+ */
+ if (shallow->nr) {
+ reprepare_packed_git();
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ shallow, 1);
+ }
commit_lock_file(&shallow_lock);
+ }
}
reprepare_packed_git();
diff --git a/fetch-pack.h b/fetch-pack.h
index 40f08ba..c55bf1c 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
const char *dest,
struct ref **sought,
int nr_sought,
+ struct extra_have_objects *shallow,
char **pack_lockfile);
#endif
diff --git a/shallow.c b/shallow.c
index ee9edd4..d909d95 100644
--- a/shallow.c
+++ b/shallow.c
@@ -183,26 +183,60 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
return 0;
}
-int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+ struct extra_have_objects *extra,
+ int remove_unused_grafts)
{
struct write_shallow_data data;
+ int i;
data.out = out;
data.use_pack_protocol = use_pack_protocol;
data.count = 0;
for_each_commit_graft(write_one_shallow, &data);
+ if (!extra)
+ return data.count;
+
+ for (i = 0; i < extra->nr; i++) {
+ if (!remove_unused_grafts && has_sha1_file(extra->array[i]))
+ /*
+ * The server may have even shorter history
+ * than the client (e.g. the client has
+ * A-B-C-D but the server only has C-D). We do
+ * NOT want to cut client's history down to
+ * C-D simply because the server is set up
+ * so. If we don't have "C" and the server
+ * sends C-D to us, then we set up a graft at
+ * "C".
+ */
+ continue;
+ if (remove_unused_grafts && !has_sha1_file(extra->array[i]))
+ continue;
+ strbuf_addstr(out, sha1_to_hex(extra->array[i]));
+ strbuf_addch(out, '\n');
+ data.count++;
+ }
return data.count;
}
void setup_alternate_shallow(struct lock_file *shallow_lock,
- const char **alternate_shallow_file)
+ const char **alternate_shallow_file,
+ struct extra_have_objects *extra,
+ int rewrite)
{
struct strbuf sb = STRBUF_INIT;
int fd;
- check_shallow_file_for_update();
- fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
- LOCK_DIE_ON_ERROR);
- if (write_shallow_commits(&sb, 0)) {
+ if (!rewrite)
+ fd = hold_lock_file_for_update(shallow_lock,
+ git_path("shallow"),
+ LOCK_DIE_ON_ERROR);
+ else
+ fd = shallow_lock->fd;
+ if (write_shallow_commits(&sb, 0, extra, rewrite)) {
+ if (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||
+ ftruncate(fd, 0) == -1))
+ die_errno("unable to truncate the new shallow file %s",
+ shallow_lock->filename);
if (write_in_full(fd, sb.buf, sb.len) != sb.len)
die_errno("failed to write to %s",
shallow_lock->filename);
diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
new file mode 100755
index 0000000..599928f
--- /dev/null
+++ b/t/t5536-fetch-shallow.sh
@@ -0,0 +1,75 @@
+#!/bin/sh
+
+test_description='fetch/clone from a shallow clone'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+ test_commit 1 &&
+ test_commit 2 &&
+ test_commit 3 &&
+ test_commit 4
+'
+
+test_expect_success 'setup shallow clone' '
+ git clone --no-local --depth=2 .git shallow &&
+ git --git-dir=shallow/.git log --format=%s >actual &&
+ cat <<EOF >expect &&
+4
+3
+EOF
+ test_cmp expect actual
+'
+
+test_expect_success 'clone from shallow clone' '
+ git clone --no-local shallow shallow2 &&
+ (
+ cd shallow2 &&
+ git fsck &&
+ git log --format=%s >actual &&
+ cat <<EOF >expect &&
+4
+3
+EOF
+ test_cmp expect actual
+ )
+'
+
+test_expect_success 'fetch from shallow clone' '
+ (
+ cd shallow &&
+ test_commit 5
+ ) &&
+ (
+ cd shallow2 &&
+ git fetch &&
+ git fsck &&
+ git log --format=%s origin/master >actual &&
+ cat <<EOF >expect &&
+5
+4
+3
+EOF
+ test_cmp expect actual
+ )
+'
+
+test_expect_success 'fetch --depth from shallow clone' '
+ (
+ cd shallow &&
+ test_commit 6
+ ) &&
+ (
+ cd shallow2 &&
+ git fetch --depth=2 &&
+ git fsck &&
+ git log --format=%s origin/master >actual &&
+ cat <<EOF >expect &&
+6
+5
+EOF
+ test_cmp expect actual
+ )
+'
+
+test_done
diff --git a/transport.c b/transport.c
index 10a8cb8..17fef16 100644
--- a/transport.c
+++ b/transport.c
@@ -453,6 +453,7 @@ struct git_transport_data {
int fd[2];
unsigned got_remote_heads : 1;
struct extra_have_objects extra_have;
+ struct extra_have_objects shallow;
};
static int set_git_option(struct git_transport_options *opts,
@@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
connect_setup(transport, for_push, 0);
get_remote_heads(data->fd[0], NULL, 0, &refs,
- for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
+ for_push ? REF_NORMAL : 0,
+ &data->extra_have, &data->shallow);
data->got_remote_heads = 1;
return refs;
@@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport *transport,
if (!data->got_remote_heads) {
connect_setup(transport, 0, 0);
get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
- NULL, NULL);
+ NULL, &data->shallow);
data->got_remote_heads = 1;
}
refs = fetch_pack(&args, data->fd, data->conn,
refs_tmp ? refs_tmp : transport->remote_refs,
- dest, to_fetch, nr_heads,
+ dest, to_fetch, nr_heads, &data->shallow,
&transport->pack_lockfile);
close(data->fd[0]);
close(data->fd[1]);
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH 6/7] {send,receive}-pack: support pushing from a shallow clone
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
` (4 preceding siblings ...)
2013-07-17 12:47 ` [PATCH 5/7] fetch-pack: support fetching from a shallow repository Nguyễn Thái Ngọc Duy
@ 2013-07-17 12:47 ` Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 7/7] send-pack: support pushing to " Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
7 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-17 12:47 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
Pushing from a shallow clone using today's send-pack and receive-pack
may work, if the transferred pack does not ends up at any graft
points. If it does, recent receive-pack that does connectivity check
will reject the push. If receive-pack is old, the upstream repo
becomes corrupt.
The pack protocol is updated and send-pack now sends all shallow
grafts before it sends the commands, if the repo is shallow. This
protocol extension will break current receive-pack, which is intended,
mostly to stop corrupting the upstream repo.
The receiver end, the newreceive-pack, does something similar to
fetch-pack: it creates a temporary shallow file with grafts from
send-pack, then receives the pack, and finally writes the refined
shallow file down.
shadow file is not cleaned up after deleting (or force updating) a ref
if that ref is the only way to reach the graft points. The reason is
once we delete graft points, we can't recover. That may make reflog
entries on server useless. Leave that for the administrators to decide
when to clean up shadow file (maybe at repack/gc time).
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/technical/pack-protocol.txt | 4 +-
builtin/receive-pack.c | 49 ++++++++++++++++++----
send-pack.c | 3 ++
t/t5537-push-shallow.sh (new +x) | 67 +++++++++++++++++++++++++++++++
4 files changed, 114 insertions(+), 9 deletions(-)
create mode 100755 t/t5537-push-shallow.sh
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index eb8edd1..c73b62f 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -464,7 +464,9 @@ contain all the objects that the server will need to complete the new
references.
----
- update-request = command-list [pack-file]
+ update-request = *shallow command-list [pack-file]
+
+ shallow = PKT-LINE("shallow" SP obj-id)
command-list = PKT-LINE(command NUL capability-list LF)
*PKT-LINE(command LF)
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 2d8e19b..0537e26 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -41,6 +41,10 @@ static int auto_gc = 1;
static const char *head_name;
static void *head_name_to_free;
static int sent_capabilities;
+static int shallow_changed;
+static const char* alternate_shallow_file;
+static struct lock_file shallow_lock;
+static struct extra_have_objects shallow;
static enum deny_action parse_deny_action(const char *var, const char *value)
{
@@ -751,6 +755,13 @@ static void execute_commands(struct command *commands, const char *unpacker_erro
}
}
+static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)
+{
+ ALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);
+ hashcpy(&(extra->array[extra->nr][0]), sha1);
+ extra->nr++;
+}
+
static struct command *read_head_info(void)
{
struct command *commands = NULL;
@@ -765,6 +776,17 @@ static struct command *read_head_info(void)
line = packet_read_line(0, &len);
if (!line)
break;
+
+ if (len == 48 && !prefixcmp(line, "shallow ")) {
+ if (get_sha1_hex(line + 8, old_sha1))
+ die("protocol error: expected shallow sha, got '%s'", line + 8);
+ if (!has_sha1_file(old_sha1)) {
+ add_extra_have(&shallow, old_sha1);
+ shallow_changed = 1;
+ }
+ continue;
+ }
+
if (len < 83 ||
line[40] != ' ' ||
line[81] != ' ' ||
@@ -827,6 +849,12 @@ static const char *unpack(int err_fd)
? transfer_fsck_objects
: 0);
+ if (shallow_changed)
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ &shallow, 0);
+
+
hdr_err = parse_pack_header(&hdr);
if (hdr_err) {
if (err_fd > 0)
@@ -854,9 +882,8 @@ static const char *unpack(int err_fd)
child.err = err_fd;
child.git_cmd = 1;
code = run_command(&child);
- if (!code)
- return NULL;
- return "unpack-objects abnormal exit";
+ if (code)
+ return "unpack-objects abnormal exit";
} else {
const char *keeper[7];
int s, status, i = 0;
@@ -887,12 +914,18 @@ static const char *unpack(int err_fd)
pack_lockfile = index_pack_lockfile(ip.out);
close(ip.out);
status = finish_command(&ip);
- if (!status) {
- reprepare_packed_git();
- return NULL;
- }
- return "index-pack abnormal exit";
+ if (status)
+ return "index-pack abnormal exit";
+ reprepare_packed_git();
+ }
+
+ if (shallow_changed) {
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ &shallow, 1);
+ commit_lock_file(&shallow_lock);
}
+ return NULL;
}
static const char *unpack_with_sideband(void)
diff --git a/send-pack.c b/send-pack.c
index 7d172ef..81d4b1c 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -212,6 +212,9 @@ int send_pack(struct send_pack_args *args,
return 0;
}
+ if (!args->dry_run && !args->stateless_rpc)
+ advertise_shallow_grafts(out);
+
/*
* Finally, tell the other end!
*/
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
new file mode 100755
index 0000000..30fdbc8
--- /dev/null
+++ b/t/t5537-push-shallow.sh
@@ -0,0 +1,67 @@
+#!/bin/sh
+
+test_description='push from/to a shallow clone'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+ test_commit 1 &&
+ test_commit 2 &&
+ test_commit 3 &&
+ test_commit 4
+'
+
+test_expect_success 'setup shallow clone' '
+ git clone --no-local --depth=2 .git shallow &&
+ git --git-dir=shallow/.git log --format=%s >actual &&
+ cat <<EOF >expect &&
+4
+3
+EOF
+ test_cmp expect actual
+'
+
+test_expect_success 'push from shallow clone' '
+ (
+ cd shallow &&
+ test_commit 5 &&
+ git push ../.git +master:refs/remotes/shallow/master
+ ) &&
+ git log --format=%s shallow/master >actual &&
+ git fsck &&
+ cat <<EOF >expect &&
+5
+4
+3
+2
+1
+EOF
+ test_cmp expect actual
+'
+
+test_expect_success 'push from shallow clone, with grafted roots' '
+ git init shallow2 &&
+ (
+ cd shallow2 &&
+ test_commit a &&
+ test_commit b &&
+ test_commit c &&
+ git rev-parse b > .git/shallow &&
+ git log --format=%s >actual &&
+ cat <<EOF >expect &&
+c
+b
+EOF
+ test_cmp expect actual &&
+ git push ../.git +master:refs/remotes/shallow2/master
+ ) &&
+ git log --format=%s shallow2/master >actual &&
+ git fsck &&
+ cat <<EOF >expect &&
+c
+b
+EOF
+ test_cmp expect actual
+'
+
+test_done
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH 7/7] send-pack: support pushing to a shallow clone
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
` (5 preceding siblings ...)
2013-07-17 12:47 ` [PATCH 6/7] {send,receive}-pack: support pushing from a shallow clone Nguyễn Thái Ngọc Duy
@ 2013-07-17 12:47 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
7 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-17 12:47 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
When send-pack receives "shallow" lines from receive-pack, it knows
the other end does not have a complete commit chains. It restrict
itself to the commits that are not cut out by either end to make sure
the result pack is usuable by receive-pack.
The same technique here, using setup_alternate_shallow() and
--shallow-file, might simplify similar code in upload-pack.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/send-pack.c | 7 +++++--
send-pack.c | 38 +++++++++++++++++++++++++++++++++-----
send-pack.h | 4 +++-
t/t5537-push-shallow.sh | 18 ++++++++++++++++++
transport.c | 5 +++--
5 files changed, 62 insertions(+), 10 deletions(-)
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index f0ad0ce..b177120 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -94,6 +94,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
int fd[2];
struct child_process *conn;
struct extra_have_objects extra_have;
+ struct extra_have_objects shallow;
struct ref *remote_refs, *local_refs;
int ret;
int helper_status = 0;
@@ -206,8 +207,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
}
memset(&extra_have, 0, sizeof(extra_have));
+ memset(&shallow, 0, sizeof(shallow));
- get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);
+ get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,
+ &extra_have, &shallow);
transport_verify_remote_names(nr_refspecs, refspecs);
@@ -227,7 +230,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
set_ref_status_for_push(remote_refs, args.send_mirror,
args.force_update);
- ret = send_pack(&args, fd, conn, remote_refs, &extra_have);
+ ret = send_pack(&args, fd, conn, remote_refs, &extra_have, &shallow);
if (helper_status)
print_helper_status(remote_refs);
diff --git a/send-pack.c b/send-pack.c
index 81d4b1c..08de681 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -27,14 +27,19 @@ static int feed_object(const unsigned char *sha1, int fd, int negative)
/*
* Make a pack stream and spit it out into file descriptor fd
*/
-static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *extra, struct send_pack_args *args)
+static int pack_objects(int fd, struct ref *refs,
+ struct extra_have_objects *extra,
+ struct extra_have_objects *extra_shallow,
+ struct send_pack_args *args)
{
/*
* The child becomes pack-objects --revs; we feed
* the revision parameters to it via its stdin and
* let its stdout go back to the other end.
*/
- const char *argv[] = {
+ const char *av[] = {
+ "--shallow-file",
+ NULL,
"pack-objects",
"--all-progress-implied",
"--revs",
@@ -45,10 +50,27 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
NULL,
NULL,
};
+ const char **argv;
struct child_process po;
+ static struct lock_file shallow_lock;
+ const char *alternate_shallow_file = NULL;
int i;
- i = 4;
+ if (extra_shallow->nr) {
+ memset(&shallow_lock, 0, sizeof(shallow_lock));
+ /* just to load up .git/shallow if exists */
+ is_repository_shallow();
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ extra_shallow,
+ 0);
+ av[1] = alternate_shallow_file;
+ argv = av;
+ i = 6;
+ } else {
+ argv = &av[2];
+ i = 4;
+ }
if (args->use_thin_pack)
argv[i++] = "--thin";
if (args->use_ofs_delta)
@@ -100,6 +122,10 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
if (finish_command(&po))
return -1;
+
+ if (extra_shallow->nr)
+ rollback_lock_file(&shallow_lock);
+
return 0;
}
@@ -176,7 +202,8 @@ static int sideband_demux(int in, int out, void *data)
int send_pack(struct send_pack_args *args,
int fd[], struct child_process *conn,
struct ref *remote_refs,
- struct extra_have_objects *extra_have)
+ struct extra_have_objects *extra_have,
+ struct extra_have_objects *extra_shallow)
{
int in = fd[0];
int out = fd[1];
@@ -294,7 +321,8 @@ int send_pack(struct send_pack_args *args,
}
if (new_refs && cmds_sent) {
- if (pack_objects(out, remote_refs, extra_have, args) < 0) {
+ if (pack_objects(out, remote_refs, extra_have, extra_shallow,
+ args) < 0) {
for (ref = remote_refs; ref; ref = ref->next)
ref->status = REF_STATUS_NONE;
if (args->stateless_rpc)
diff --git a/send-pack.h b/send-pack.h
index 05d7ab1..1c8a343 100644
--- a/send-pack.h
+++ b/send-pack.h
@@ -16,6 +16,8 @@ struct send_pack_args {
int send_pack(struct send_pack_args *args,
int fd[], struct child_process *conn,
- struct ref *remote_refs, struct extra_have_objects *extra_have);
+ struct ref *remote_refs,
+ struct extra_have_objects *extra_have,
+ struct extra_have_objects *extra_shallow);
#endif
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index 30fdbc8..76d3d51 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -64,4 +64,22 @@ EOF
test_cmp expect actual
'
+test_expect_success 'push from shallow to shallow' '
+ (
+ cd shallow &&
+ git push ../shallow2/.git +master:refs/remotes/shallow/master
+ ) &&
+ (
+ cd shallow2 &&
+ git log --format=%s shallow/master >actual &&
+ git fsck &&
+ cat <<EOF >expect &&
+5
+4
+3
+EOF
+ test_cmp expect actual
+ )
+'
+
test_done
diff --git a/transport.c b/transport.c
index 17fef16..d592e8d 100644
--- a/transport.c
+++ b/transport.c
@@ -802,7 +802,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
struct ref *tmp_refs;
connect_setup(transport, 1, 0);
- get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
+ get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,
+ NULL, &data->shallow);
data->got_remote_heads = 1;
}
@@ -817,7 +818,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
args.porcelain = !!(flags & TRANSPORT_PUSH_PORCELAIN);
ret = send_pack(&args, data->fd, data->conn, remote_refs,
- &data->extra_have);
+ &data->extra_have, &data->shallow);
close(data->fd[1]);
close(data->fd[0]);
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* Re: [PATCH 5/7] fetch-pack: support fetching from a shallow repository
2013-07-17 12:47 ` [PATCH 5/7] fetch-pack: support fetching from a shallow repository Nguyễn Thái Ngọc Duy
@ 2013-07-20 3:17 ` Junio C Hamano
0 siblings, 0 replies; 44+ messages in thread
From: Junio C Hamano @ 2013-07-20 3:17 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> Only server graft points that do not have corresponding SHA-1s in
> local repo are added to the temp shallow file because we don't want to
> accidentally cut the client history because the server's is
> shorter. The client cutting can only happen when --depth is requested.
This "have corresponding SHA-1" is iffy from connectivity point of view,
isn't it? That is,...
> + for (i = 0; i < extra->nr; i++) {
> + if (!remove_unused_grafts && has_sha1_file(extra->array[i]))
...shouldn't this require much more than "has_sha1_file()", like
"has it, and we have everything behind it with respect to the
shallow graft points we started with" or something?
Otherwise an isolated island that was left behind by an earlier
aborted commit walker fetch may mislead us to believe that we have
the complete history behind this commit object.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH 3/7] connect.c: teach get_remote_heads to parse "shallow" lines
2013-07-17 12:47 ` [PATCH 3/7] connect.c: teach get_remote_heads to parse "shallow" lines Nguyễn Thái Ngọc Duy
@ 2013-07-20 3:27 ` Junio C Hamano
0 siblings, 0 replies; 44+ messages in thread
From: Junio C Hamano @ 2013-07-20 3:27 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy; +Cc: git
Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:
> diff --git a/cache.h b/cache.h
> index dd0fb33..7665e03 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1091,6 +1091,7 @@ struct extra_have_objects {
> };
> extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
> struct ref **list, unsigned int flags,
> + struct extra_have_objects *,
> struct extra_have_objects *);
When it is the sole parameter of a specific type (i.e. not "char *",
int, size_t, etc. but an application specific structure like "struct
extra_have_objects *"), it is perfectly fine (and even preferrable)
to omit the parameter name from the declaration, as it is clear what
the parameter is and means.
But when you add another of the same type, you need to give both of
them a descriptive name (e.g. the ones you use in the definition).
Otherwise, somebody who wants to write a caller cannot tell which
"struct extra_have_objects *" parameter is the extra and which one
is the shallow graft points list.
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 00/16] First class shallow clone
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
` (6 preceding siblings ...)
2013-07-17 12:47 ` [PATCH 7/7] send-pack: support pushing to " Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:57 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 01/16] send-pack: forbid pushing from a shallow repository Nguyễn Thái Ngọc Duy
` (16 more replies)
7 siblings, 17 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:57 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
v2 includes:
- fix Junio comments, especially the one that may lead to incomplete
commit islands.
- fix send-pack setting up temporary shallow file, but never passes
it to index-pack/unpack-objects (also fix the tests to catch this)
- support smart http
- add core.noshallow for repos that wish to be always complete
- fix locally cloning a shallow repository
- make upload-pack pass --shallow-file to pack-objects in order to
remove duplicate object counting code just for shallow case.
Nguyễn Thái Ngọc Duy (16):
send-pack: forbid pushing from a shallow repository
{receive,upload}-pack: advertise shallow graft information
connect.c: teach get_remote_heads to parse "shallow" lines
Move setup_alternate_shallow and write_shallow_commits to shallow.c
fetch-pack: support fetching from a shallow repository
{send,receive}-pack: support pushing from a shallow clone
send-pack: support pushing to a shallow clone
upload-pack: let pack-objects do the object counting in shallow case
pack-protocol.txt: a bit about smart http
Add document for command arguments for supporting smart http
{fetch,upload}-pack: support fetching from a shallow clone via smart http
receive-pack: support pushing to a shallow clone via http
send-pack: support pushing from a shallow clone via http
git-clone.txt: remove shallow clone limitations
config: add core.noshallow to prevent turning a repo into a shallow one
clone: use git protocol for cloning shallow repo locally
Documentation/config.txt | 5 +
Documentation/git-clone.txt | 7 +-
Documentation/git-fetch-pack.txt | 11 +-
Documentation/git-receive-pack.txt | 16 ++-
Documentation/git-send-pack.txt | 9 +-
Documentation/git-upload-pack.txt | 13 ++-
Documentation/technical/pack-protocol.txt | 76 ++++++++++++-
builtin/clone.c | 14 ++-
builtin/fetch-pack.c | 6 +-
builtin/receive-pack.c | 76 +++++++++++--
builtin/send-pack.c | 7 +-
cache.h | 4 +-
commit.h | 27 +++++
config.c | 5 +
connect.c | 12 +-
environment.c | 1 +
fetch-pack.c | 90 ++++++---------
fetch-pack.h | 1 +
remote-curl.c | 4 +-
send-pack.c | 57 +++++++++-
send-pack.h | 4 +-
shallow.c | 147 +++++++++++++++++++++++++
t/t5530-upload-pack-error.sh | 3 -
t/t5536-fetch-shallow.sh (new +x) | 141 ++++++++++++++++++++++++
t/t5537-push-shallow.sh (new +x) | 176 ++++++++++++++++++++++++++++++
t/t5601-clone.sh | 7 ++
transport.c | 14 ++-
upload-pack.c | 132 ++++++----------------
28 files changed, 858 insertions(+), 207 deletions(-)
create mode 100755 t/t5536-fetch-shallow.sh
create mode 100755 t/t5537-push-shallow.sh
--
1.8.2.83.gc99314b
^ permalink raw reply [flat|nested] 44+ messages in thread
* [PATCH v2 01/16] send-pack: forbid pushing from a shallow repository
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:57 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 02/16] {receive,upload}-pack: advertise shallow graft information Nguyễn Thái Ngọc Duy
` (15 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:57 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
send-pack can send a pack with loose ends to the server. receive-pack
before 6d4bb38 (fetch: verify we have everything we need before
updating our ref - 2011-09-01) does not detect this and keeps the pack
anyway, which corrupts the repository, at least from fsck point of
view.
send-pack will learn to safely push from a shallow repository later.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/send-pack.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 152c4ea..d4cbf64 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -184,6 +184,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
(send_all && args.send_mirror))
usage(send_pack_usage);
+ if (is_repository_shallow())
+ die("attempt to push from a shallow repository");
+
if (remote_name) {
remote = remote_get(remote_name);
if (!remote_has_url(remote, dest)) {
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 02/16] {receive,upload}-pack: advertise shallow graft information
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 01/16] send-pack: forbid pushing from a shallow repository Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:57 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 03/16] connect.c: teach get_remote_heads to parse "shallow" lines Nguyễn Thái Ngọc Duy
` (14 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:57 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
If either receive-pack or upload-pack is called on a shallow
repository, shallow graft points will be sent after the ref
advertisement (but before the packet flush), so that the client has
the full "shape" of the server's commit graph.
This breaks the protocol for all clients trying to push to a shallow
repo, or fetch from one. Which is basically the same end result as
today's "is_repository_shallow() && die()" in receive-pack and
upload-pack. New clients will be made aware of shallow upstream and
can make use of this information.
Smart HTTP is not affected by this patch. Shallow support on
smart-http comes later separately.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/technical/pack-protocol.txt | 3 +++
builtin/receive-pack.c | 4 +++-
commit.h | 1 +
shallow.c | 16 ++++++++++++++++
upload-pack.c | 6 ++++--
5 files changed, 27 insertions(+), 3 deletions(-)
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index b898e97..eb8edd1 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag.
----
advertised-refs = (no-refs / list-of-refs)
+ *shallow
flush-pkt
no-refs = PKT-LINE(zero-id SP "capabilities^{}"
@@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag.
other-tip = obj-id SP refname LF
other-peeled = obj-id SP refname "^{}" LF
+ shallow = PKT-LINE("shallow" SP obj-id)
+
capability-list = capability *(SP capability)
capability = 1*(LC_ALPHA / DIGIT / "-" / "_")
LC_ALPHA = %x61-7A
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index e3eb5fc..6ffe526 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -176,6 +176,8 @@ static void write_head_info(void)
if (!sent_capabilities)
show_ref("capabilities^{}", null_sha1);
+ advertise_shallow_grafts(1);
+
/* EOF */
packet_flush(1);
}
@@ -990,7 +992,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
if (!enter_repo(dir, 0))
die("'%s' does not appear to be a git repository", dir);
- if (is_repository_shallow())
+ if (is_repository_shallow() && stateless_rpc)
die("attempt to push into a shallow repository");
git_config(receive_pack_config, NULL);
diff --git a/commit.h b/commit.h
index 4d452dc..e0688c3 100644
--- a/commit.h
+++ b/commit.h
@@ -187,6 +187,7 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
int depth, int shallow_flag, int not_shallow_flag);
extern void check_shallow_file_for_update(void);
extern void set_alternate_shallow_file(const char *path);
+extern void advertise_shallow_grafts(int);
int is_descendant_of(struct commit *, struct commit_list *);
int in_merge_bases(struct commit *, struct commit *);
diff --git a/shallow.c b/shallow.c
index cbe2526..ccdfefc 100644
--- a/shallow.c
+++ b/shallow.c
@@ -1,6 +1,7 @@
#include "cache.h"
#include "commit.h"
#include "tag.h"
+#include "pkt-line.h"
static int is_shallow = -1;
static struct stat shallow_stat;
@@ -146,3 +147,18 @@ void check_shallow_file_for_update(void)
)
die("shallow file was changed during fetch");
}
+
+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)
+{
+ int fd = *(int*)cb;
+ if (graft->nr_parent == -1)
+ packet_write(fd, "shallow %s\n", sha1_to_hex(graft->sha1));
+ return 0;
+}
+
+void advertise_shallow_grafts(int fd)
+{
+ if (!is_repository_shallow())
+ return;
+ for_each_commit_graft(advertise_shallow_grafts_cb, &fd);
+}
diff --git a/upload-pack.c b/upload-pack.c
index 127e59a..c377a3e 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -766,6 +766,7 @@ static void upload_pack(void)
reset_timeout();
head_ref_namespaced(send_ref, NULL);
for_each_namespaced_ref(send_ref, NULL);
+ advertise_shallow_grafts(1);
packet_flush(1);
} else {
head_ref_namespaced(mark_our_ref, NULL);
@@ -837,8 +838,9 @@ int main(int argc, char **argv)
if (!enter_repo(dir, strict))
die("'%s' does not appear to be a git repository", dir);
- if (is_repository_shallow())
- die("attempt to fetch/clone from a shallow repository");
+ if (is_repository_shallow() && stateless_rpc)
+ die("attempt to push into a shallow repository");
+
git_config(upload_pack_config, NULL);
upload_pack();
return 0;
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 03/16] connect.c: teach get_remote_heads to parse "shallow" lines
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 01/16] send-pack: forbid pushing from a shallow repository Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 02/16] {receive,upload}-pack: advertise shallow graft information Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:57 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 04/16] Move setup_alternate_shallow and write_shallow_commits to shallow.c Nguyễn Thái Ngọc Duy
` (13 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:57 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
No callers pass a non-empty pointer as shallow_points at this
stage. As a result, all clients still refuse to talk to shallow
repository on the other end.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/fetch-pack.c | 2 +-
builtin/send-pack.c | 2 +-
cache.h | 3 ++-
connect.c | 12 +++++++++++-
remote-curl.c | 2 +-
transport.c | 7 ++++---
6 files changed, 20 insertions(+), 8 deletions(-)
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index aba4465..080e599 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -144,7 +144,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
args.verbose ? CONNECT_VERBOSE : 0);
}
- get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);
+ get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
ref = fetch_pack(&args, fd, conn, ref, dest,
sought, nr_sought, pack_lockfile_ptr);
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index d4cbf64..4be5931 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -210,7 +210,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
memset(&extra_have, 0, sizeof(extra_have));
- get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);
+ get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);
transport_verify_remote_names(nr_refspecs, refspecs);
diff --git a/cache.h b/cache.h
index dd0fb33..7f17228 100644
--- a/cache.h
+++ b/cache.h
@@ -1091,7 +1091,8 @@ struct extra_have_objects {
};
extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
struct ref **list, unsigned int flags,
- struct extra_have_objects *);
+ struct extra_have_objects *have,
+ struct extra_have_objects *shallow);
extern int server_supports(const char *feature);
extern int parse_feature_request(const char *features, const char *feature);
extern const char *server_feature_value(const char *feature, int *len_ret);
diff --git a/connect.c b/connect.c
index a0783d4..1cba424 100644
--- a/connect.c
+++ b/connect.c
@@ -64,7 +64,8 @@ static void die_initial_contact(int got_at_least_one_head)
*/
struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
struct ref **list, unsigned int flags,
- struct extra_have_objects *extra_have)
+ struct extra_have_objects *extra_have,
+ struct extra_have_objects *shallow_points)
{
int got_at_least_one_head = 0;
@@ -89,6 +90,15 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
if (len > 4 && !prefixcmp(buffer, "ERR "))
die("remote error: %s", buffer + 4);
+ if (len == 48 && !prefixcmp(buffer, "shallow ")) {
+ if (get_sha1_hex(buffer + 8, old_sha1))
+ die("protocol error: expected shallow sha, got '%s'", buffer + 8);
+ if (!shallow_points)
+ die("repository on the other end cannot be shallow");
+ add_extra_have(shallow_points, old_sha1);
+ continue;
+ }
+
if (len < 42 || get_sha1_hex(buffer, old_sha1) || buffer[40] != ' ')
die("protocol error: expected sha/ref, got '%s'", buffer);
name = buffer + 41;
diff --git a/remote-curl.c b/remote-curl.c
index 5b3ce9e..c329bd3 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -86,7 +86,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
{
struct ref *list = NULL;
get_remote_heads(-1, heads->buf, heads->len, &list,
- for_push ? REF_NORMAL : 0, NULL);
+ for_push ? REF_NORMAL : 0, NULL, NULL);
return list;
}
diff --git a/transport.c b/transport.c
index e15db98..10a8cb8 100644
--- a/transport.c
+++ b/transport.c
@@ -509,7 +509,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
connect_setup(transport, for_push, 0);
get_remote_heads(data->fd[0], NULL, 0, &refs,
- for_push ? REF_NORMAL : 0, &data->extra_have);
+ for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
data->got_remote_heads = 1;
return refs;
@@ -539,7 +539,8 @@ static int fetch_refs_via_pack(struct transport *transport,
if (!data->got_remote_heads) {
connect_setup(transport, 0, 0);
- get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0, NULL);
+ get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
+ NULL, NULL);
data->got_remote_heads = 1;
}
@@ -799,7 +800,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
struct ref *tmp_refs;
connect_setup(transport, 1, 0);
- get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL);
+ get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
data->got_remote_heads = 1;
}
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 04/16] Move setup_alternate_shallow and write_shallow_commits to shallow.c
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (2 preceding siblings ...)
2013-07-20 9:57 ` [PATCH v2 03/16] connect.c: teach get_remote_heads to parse "shallow" lines Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:57 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository Nguyễn Thái Ngọc Duy
` (12 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:57 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
commit.h | 3 +++
fetch-pack.c | 53 +----------------------------------------------------
shallow.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 57 insertions(+), 52 deletions(-)
diff --git a/commit.h b/commit.h
index e0688c3..678fa20 100644
--- a/commit.h
+++ b/commit.h
@@ -188,6 +188,9 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
extern void check_shallow_file_for_update(void);
extern void set_alternate_shallow_file(const char *path);
extern void advertise_shallow_grafts(int);
+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);
+extern void setup_alternate_shallow(struct lock_file *shallow_lock,
+ const char **alternate_shallow_file);
int is_descendant_of(struct commit *, struct commit_list *);
int in_merge_bases(struct commit *, struct commit *);
diff --git a/fetch-pack.c b/fetch-pack.c
index abe5ffb..dc71a2b 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -185,36 +185,6 @@ static void consume_shallow_list(struct fetch_pack_args *args, int fd)
}
}
-struct write_shallow_data {
- struct strbuf *out;
- int use_pack_protocol;
- int count;
-};
-
-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
-{
- struct write_shallow_data *data = cb_data;
- const char *hex = sha1_to_hex(graft->sha1);
- data->count++;
- if (data->use_pack_protocol)
- packet_buf_write(data->out, "shallow %s", hex);
- else {
- strbuf_addstr(data->out, hex);
- strbuf_addch(data->out, '\n');
- }
- return 0;
-}
-
-static int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
-{
- struct write_shallow_data data;
- data.out = out;
- data.use_pack_protocol = use_pack_protocol;
- data.count = 0;
- for_each_commit_graft(write_one_shallow, &data);
- return data.count;
-}
-
static enum ack_type get_ack(int fd, unsigned char *result_sha1)
{
int len;
@@ -795,27 +765,6 @@ static int cmp_ref_by_name(const void *a_, const void *b_)
return strcmp(a->name, b->name);
}
-static void setup_alternate_shallow(void)
-{
- struct strbuf sb = STRBUF_INIT;
- int fd;
-
- check_shallow_file_for_update();
- fd = hold_lock_file_for_update(&shallow_lock, git_path("shallow"),
- LOCK_DIE_ON_ERROR);
- if (write_shallow_commits(&sb, 0)) {
- if (write_in_full(fd, sb.buf, sb.len) != sb.len)
- die_errno("failed to write to %s", shallow_lock.filename);
- alternate_shallow_file = shallow_lock.filename;
- } else
- /*
- * is_repository_shallow() sees empty string as "no
- * shallow file".
- */
- alternate_shallow_file = "";
- strbuf_release(&sb);
-}
-
static struct ref *do_fetch_pack(struct fetch_pack_args *args,
int fd[2],
const struct ref *orig_ref,
@@ -896,7 +845,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
if (args->stateless_rpc)
packet_flush(fd[1]);
if (args->depth > 0)
- setup_alternate_shallow();
+ setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
if (get_pack(args, fd, pack_lockfile))
die("git fetch-pack: fetch failed.");
diff --git a/shallow.c b/shallow.c
index ccdfefc..ee9edd4 100644
--- a/shallow.c
+++ b/shallow.c
@@ -162,3 +162,56 @@ void advertise_shallow_grafts(int fd)
return;
for_each_commit_graft(advertise_shallow_grafts_cb, &fd);
}
+
+struct write_shallow_data {
+ struct strbuf *out;
+ int use_pack_protocol;
+ int count;
+};
+
+static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
+{
+ struct write_shallow_data *data = cb_data;
+ const char *hex = sha1_to_hex(graft->sha1);
+ data->count++;
+ if (data->use_pack_protocol)
+ packet_buf_write(data->out, "shallow %s", hex);
+ else {
+ strbuf_addstr(data->out, hex);
+ strbuf_addch(data->out, '\n');
+ }
+ return 0;
+}
+
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
+{
+ struct write_shallow_data data;
+ data.out = out;
+ data.use_pack_protocol = use_pack_protocol;
+ data.count = 0;
+ for_each_commit_graft(write_one_shallow, &data);
+ return data.count;
+}
+
+void setup_alternate_shallow(struct lock_file *shallow_lock,
+ const char **alternate_shallow_file)
+{
+ struct strbuf sb = STRBUF_INIT;
+ int fd;
+
+ check_shallow_file_for_update();
+ fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
+ LOCK_DIE_ON_ERROR);
+ if (write_shallow_commits(&sb, 0)) {
+ if (write_in_full(fd, sb.buf, sb.len) != sb.len)
+ die_errno("failed to write to %s",
+ shallow_lock->filename);
+ *alternate_shallow_file = shallow_lock->filename;
+ } else
+ /*
+ * is_repository_shallow() sees empty string as "no
+ * shallow file".
+ */
+ *alternate_shallow_file = "";
+ strbuf_release(&sb);
+}
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (3 preceding siblings ...)
2013-07-20 9:57 ` [PATCH v2 04/16] Move setup_alternate_shallow and write_shallow_commits to shallow.c Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:57 ` Nguyễn Thái Ngọc Duy
2013-07-22 19:10 ` Philip Oakley
2013-07-20 9:58 ` [PATCH v2 06/16] {send,receive}-pack: support pushing from a shallow clone Nguyễn Thái Ngọc Duy
` (11 subsequent siblings)
16 siblings, 1 reply; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:57 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
upload-pack already advertises all shallow grafts if server repository
is shallow. This information can be used to add more grafts to the
client if the server sends commit chains down to its graft points.
If the server is shallow, before we receive the pack, we setup a
temporary shallow file that contains both local graft points and the
server's. This stops index-pack from going beyond server's graft
points.
Only server graft points that do not have corresponding SHA-1s in
local repo are added to the temp shallow file because we don't want to
accidentally cut the client history because the server's is
shorter. The client cutting can only happen when --depth is requested.
After index-pack finishes successfully, we write the temporary shallow
down with one exception: unused graft points provided by the server
are removed. We don't want those lying around and suddenly become
active.
Note that in the "shallow -> shallow" case, the server might not have
enough information to find common roots to create an optimum pack. It
might send complete commit chains down to the graft points as a
result. I don't think we can improve this, unless upload-pack somehow
has access to a full repository.
"shallow -> shallow" case only makes sense when the upstream provides
a stable shallow repo (e.g. make a cut every year or so and ask devs
to all move to the new base). If the cloned repos are all based on a
stable (shallow) upstream, the above problem is unlikely to happen.
A side effect of this change is we can now clone from a shallow
repository. And a full repository may automatically become shallow if
you fetch from a shallow repository.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/fetch-pack.c | 7 ++-
commit.h | 27 +++++++++-
fetch-pack.c | 30 +++++++++--
fetch-pack.h | 1 +
shallow.c | 90 +++++++++++++++++++++++++++++---
t/t5536-fetch-shallow.sh (new +x) | 105 ++++++++++++++++++++++++++++++++++++++
transport.c | 10 ++--
7 files changed, 252 insertions(+), 18 deletions(-)
create mode 100755 t/t5536-fetch-shallow.sh
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 080e599..f6a6d76 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
char **pack_lockfile_ptr = NULL;
struct child_process *conn;
struct fetch_pack_args args;
+ struct extra_have_objects shallow;
packet_trace_identity("fetch-pack");
@@ -144,10 +145,12 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
args.verbose ? CONNECT_VERBOSE : 0);
}
- get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
+ memset(&shallow, 0, sizeof(shallow));
+ get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,
+ args.stateless_rpc ? NULL : &shallow);
ref = fetch_pack(&args, fd, conn, ref, dest,
- sought, nr_sought, pack_lockfile_ptr);
+ sought, nr_sought, &shallow, pack_lockfile_ptr);
if (pack_lockfile) {
printf("lock %s\n", pack_lockfile);
fflush(stdout);
diff --git a/commit.h b/commit.h
index 678fa20..97fd54e 100644
--- a/commit.h
+++ b/commit.h
@@ -188,9 +188,32 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,
extern void check_shallow_file_for_update(void);
extern void set_alternate_shallow_file(const char *path);
extern void advertise_shallow_grafts(int);
-extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);
+
+/*
+ * Ignore graft points that cut out existing "good" parts of our
+ * history. Graft points pointing to incomplete commit islands are
+ * _not_ ignore.
+ */
+#define WRITE_SHALLOW_NO_CUT 1
+/*
+ * Remove graft points that are not attached to any SHA-1 in current
+ * history.
+ */
+#define WRITE_SHALLOW_NO_UNUSED 2
+/*
+ * Not really write_shallow_commits() flag, this lets
+ * setup_alternate_shallow know this is the "final" setup before
+ * committing to .git/shallow
+ */
+#define WRITE_SHALLOW_REWRITE 4
+
+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+ struct extra_have_objects *extra,
+ unsigned int flags);
extern void setup_alternate_shallow(struct lock_file *shallow_lock,
- const char **alternate_shallow_file);
+ const char **alternate_shallow_file,
+ struct extra_have_objects *extra,
+ unsigned int flags);
int is_descendant_of(struct commit *, struct commit_list *);
int in_merge_bases(struct commit *, struct commit *);
diff --git a/fetch-pack.c b/fetch-pack.c
index dc71a2b..f337526 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args *args,
}
if (is_repository_shallow())
- write_shallow_commits(&req_buf, 1);
+ write_shallow_commits(&req_buf, 1, NULL, WRITE_SHALLOW_NO_CUT);
if (args->depth > 0)
packet_buf_write(&req_buf, "deepen %d", args->depth);
packet_buf_flush(&req_buf);
@@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
int fd[2],
const struct ref *orig_ref,
struct ref **sought, int nr_sought,
+ struct extra_have_objects *shallow,
char **pack_lockfile)
{
struct ref *ref = copy_ref_list(orig_ref);
@@ -844,8 +845,10 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
if (args->stateless_rpc)
packet_flush(fd[1]);
- if (args->depth > 0)
- setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
+ if (args->depth > 0 || shallow->nr)
+ setup_alternate_shallow(&shallow_lock, &alternate_shallow_file,
+ shallow,
+ WRITE_SHALLOW_NO_CUT);
if (get_pack(args, fd, pack_lockfile))
die("git fetch-pack: fetch failed.");
@@ -922,6 +925,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
const struct ref *ref,
const char *dest,
struct ref **sought, int nr_sought,
+ struct extra_have_objects *shallow,
char **pack_lockfile)
{
struct ref *ref_cpy;
@@ -934,14 +938,30 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
packet_flush(fd[1]);
die("no matching remote head");
}
- ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);
+ ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
+ shallow, pack_lockfile);
if (alternate_shallow_file) {
if (*alternate_shallow_file == '\0') { /* --unshallow */
unlink_or_warn(git_path("shallow"));
rollback_lock_file(&shallow_lock);
- } else
+ } else {
+ /*
+ * The server is a shallow clone and it sends
+ * us all of its shallow grafts. Some may be
+ * needed if the server sends objects down to
+ * the bottom. Remove all unused grafts.
+ */
+ if (shallow->nr) {
+ reprepare_packed_git();
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ shallow,
+ WRITE_SHALLOW_NO_CUT |
+ WRITE_SHALLOW_REWRITE);
+ }
commit_lock_file(&shallow_lock);
+ }
}
reprepare_packed_git();
diff --git a/fetch-pack.h b/fetch-pack.h
index 40f08ba..c55bf1c 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
const char *dest,
struct ref **sought,
int nr_sought,
+ struct extra_have_objects *shallow,
char **pack_lockfile);
#endif
diff --git a/shallow.c b/shallow.c
index ee9edd4..337f676 100644
--- a/shallow.c
+++ b/shallow.c
@@ -2,6 +2,7 @@
#include "commit.h"
#include "tag.h"
#include "pkt-line.h"
+#include "connected.h"
static int is_shallow = -1;
static struct stat shallow_stat;
@@ -183,26 +184,103 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)
return 0;
}
-int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
+static int sha1_iterate(void *cb, unsigned char *sha1)
+{
+ if (is_null_sha1(cb))
+ return 1;
+ hashcpy(sha1, cb);
+ hashclr(cb);
+ return 0;
+}
+
+struct connectivity_cache_item {
+ unsigned char sha1[20];
+ int good;
+};
+
+static int good_connectivity_from(const unsigned char *sha1)
+{
+ static struct connectivity_cache_item *cache;
+ static int nr, alloc;
+ int i;
+ unsigned char sha1_cb[20];
+
+ for (i = 0; i < nr; i++)
+ if (!hashcmp(cache[i].sha1, sha1))
+ return cache[i].good;
+
+ ALLOC_GROW(cache, nr + 1, alloc);
+ hashcpy(sha1_cb, sha1);
+ hashcpy(cache[nr].sha1, sha1);
+ cache[nr].good = !check_everything_connected(sha1_iterate, 1, sha1_cb);
+ nr++;
+ return cache[nr - 1].good;
+}
+
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+ struct extra_have_objects *extra,
+ unsigned int flags)
{
struct write_shallow_data data;
+ int i;
data.out = out;
data.use_pack_protocol = use_pack_protocol;
data.count = 0;
for_each_commit_graft(write_one_shallow, &data);
+ if (!extra)
+ return data.count;
+
+ for (i = 0; i < extra->nr; i++) {
+ if ((flags & WRITE_SHALLOW_NO_CUT) &&
+ good_connectivity_from(extra->array[i]))
+ /*
+ * The server may have even shorter history
+ * than the client (e.g. the client has
+ * A-B-C-D but the server only has C-D). We do
+ * NOT want to cut client's history down to
+ * C-D simply because the server is set up
+ * so. If we don't have "C" and the server
+ * sends C-D to us, then we set up a graft at
+ * "C".
+ *
+ * If the client has (broken)-B-C-D, then "C"
+ * _is_ accepted. The -B-C-D is an incomplete
+ * island that normally has no connectivity to
+ * the rest of the repo.
+ */
+ continue;
+ if ((flags & WRITE_SHALLOW_NO_UNUSED) &&
+ !has_sha1_file(extra->array[i]))
+ continue;
+ strbuf_addstr(out, sha1_to_hex(extra->array[i]));
+ strbuf_addch(out, '\n');
+ data.count++;
+ }
return data.count;
}
void setup_alternate_shallow(struct lock_file *shallow_lock,
- const char **alternate_shallow_file)
+ const char **alternate_shallow_file,
+ struct extra_have_objects *extra,
+ unsigned int flags)
{
struct strbuf sb = STRBUF_INIT;
+ int rewrite = flags & WRITE_SHALLOW_REWRITE;
int fd;
- check_shallow_file_for_update();
- fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
- LOCK_DIE_ON_ERROR);
- if (write_shallow_commits(&sb, 0)) {
+ if (!rewrite)
+ fd = hold_lock_file_for_update(shallow_lock,
+ git_path("shallow"),
+ LOCK_DIE_ON_ERROR);
+ else {
+ flags |= WRITE_SHALLOW_NO_UNUSED;
+ fd = shallow_lock->fd;
+ }
+ if (write_shallow_commits(&sb, 0, extra, flags)) {
+ if (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||
+ ftruncate(fd, 0) == -1))
+ die_errno("unable to truncate the new shallow file %s",
+ shallow_lock->filename);
if (write_in_full(fd, sb.buf, sb.len) != sb.len)
die_errno("failed to write to %s",
shallow_lock->filename);
diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
new file mode 100755
index 0000000..15a8208
--- /dev/null
+++ b/t/t5536-fetch-shallow.sh
@@ -0,0 +1,105 @@
+#!/bin/sh
+
+test_description='fetch/clone from a shallow clone'
+
+. ./test-lib.sh
+
+commit() {
+ echo "$1" >tracked &&
+ git add tracked &&
+ git commit -m "$1"
+}
+
+test_expect_success 'setup' '
+ commit 1 &&
+ commit 2 &&
+ commit 3 &&
+ commit 4 &&
+ git config --global transfer.fsckObjects true
+'
+
+test_expect_success 'setup shallow clone' '
+ git clone --no-local --depth=2 .git shallow &&
+ git --git-dir=shallow/.git log --format=%s >actual &&
+ cat <<EOF >expect &&
+4
+3
+EOF
+ test_cmp expect actual
+'
+
+test_expect_success 'clone from shallow clone' '
+ git clone --no-local shallow shallow2 &&
+ (
+ cd shallow2 &&
+ git fsck &&
+ git log --format=%s >actual &&
+ cat <<EOF >expect &&
+4
+3
+EOF
+ test_cmp expect actual
+ )
+'
+
+test_expect_success 'fetch from shallow clone' '
+ (
+ cd shallow &&
+ commit 5
+ ) &&
+ (
+ cd shallow2 &&
+ git fetch &&
+ git fsck &&
+ git log --format=%s origin/master >actual &&
+ cat <<EOF >expect &&
+5
+4
+3
+EOF
+ test_cmp expect actual
+ )
+'
+
+test_expect_success 'fetch --depth from shallow clone' '
+ (
+ cd shallow &&
+ commit 6
+ ) &&
+ (
+ cd shallow2 &&
+ git fetch --depth=2 &&
+ git fsck &&
+ git log --format=%s origin/master >actual &&
+ cat <<EOF >expect &&
+6
+5
+EOF
+ test_cmp expect actual
+ )
+'
+
+test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '
+ # the blob "1" is available in .git but hidden by the
+ # shallow2/.git/shallow and it should be resent
+ ! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null &&
+ echo 1 > 1.t &&
+ git add 1.t &&
+ git commit -m add-1-back &&
+ (
+ cd shallow2 &&
+ git fetch ../.git +refs/heads/master:refs/remotes/top/master &&
+ git fsck &&
+ git log --format=%s top/master >actual &&
+ cat <<EOF >expect &&
+add-1-back
+4
+3
+EOF
+ test_cmp expect actual
+ ) &&
+ git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null
+
+'
+
+test_done
diff --git a/transport.c b/transport.c
index 10a8cb8..f167ca5 100644
--- a/transport.c
+++ b/transport.c
@@ -453,6 +453,7 @@ struct git_transport_data {
int fd[2];
unsigned got_remote_heads : 1;
struct extra_have_objects extra_have;
+ struct extra_have_objects shallow;
};
static int set_git_option(struct git_transport_options *opts,
@@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
connect_setup(transport, for_push, 0);
get_remote_heads(data->fd[0], NULL, 0, &refs,
- for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
+ for_push ? REF_NORMAL : 0,
+ &data->extra_have, &data->shallow);
data->got_remote_heads = 1;
return refs;
@@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport *transport,
if (!data->got_remote_heads) {
connect_setup(transport, 0, 0);
get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
- NULL, NULL);
+ NULL, &data->shallow);
data->got_remote_heads = 1;
}
refs = fetch_pack(&args, data->fd, data->conn,
refs_tmp ? refs_tmp : transport->remote_refs,
- dest, to_fetch, nr_heads,
+ dest, to_fetch, nr_heads, &data->shallow,
&transport->pack_lockfile);
close(data->fd[0]);
close(data->fd[1]);
@@ -803,6 +805,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
data->got_remote_heads = 1;
}
+ if (data->shallow.nr)
+ die("pushing to a shallow repository is not supported");
memset(&args, 0, sizeof(args));
args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 06/16] {send,receive}-pack: support pushing from a shallow clone
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (4 preceding siblings ...)
2013-07-20 9:57 ` [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 07/16] send-pack: support pushing to " Nguyễn Thái Ngọc Duy
` (10 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Pushing from a shallow clone using today's send-pack and receive-pack
may work, if the transferred pack does not ends up at any graft
points. If it does, recent receive-pack that does connectivity check
will reject the push. If receive-pack is old, the upstream repo
becomes corrupt.
The pack protocol is updated and send-pack now sends all shallow
grafts before it sends the commands, if the repo is shallow. This
protocol extension will break current receive-pack, which is intended,
mostly to stop corrupting the upstream repo.
The receiver end, the newreceive-pack, does something similar to
fetch-pack: it creates a temporary shallow file with grafts from
send-pack, then receives the pack, and finally writes the refined
shallow file down.
shadow file is not cleaned up after deleting (or force updating) a ref
if that ref is the only way to reach the graft points. The reason is
once we delete graft points, we can't recover. That may make reflog
entries on server useless. Leave that for the administrators to decide
when to clean up shadow file (maybe at repack/gc time).
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/technical/pack-protocol.txt | 4 +-
builtin/receive-pack.c | 66 ++++++++++++++++++++++-----
builtin/send-pack.c | 2 +-
send-pack.c | 3 ++
t/t5537-push-shallow.sh (new +x) | 74 +++++++++++++++++++++++++++++++
5 files changed, 137 insertions(+), 12 deletions(-)
create mode 100755 t/t5537-push-shallow.sh
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index eb8edd1..c73b62f 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -464,7 +464,9 @@ contain all the objects that the server will need to complete the new
references.
----
- update-request = command-list [pack-file]
+ update-request = *shallow command-list [pack-file]
+
+ shallow = PKT-LINE("shallow" SP obj-id)
command-list = PKT-LINE(command NUL capability-list LF)
*PKT-LINE(command LF)
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 6ffe526..cbb2025 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -41,6 +41,10 @@ static int auto_gc = 1;
static const char *head_name;
static void *head_name_to_free;
static int sent_capabilities;
+static int shallow_changed;
+static const char* alternate_shallow_file;
+static struct lock_file shallow_lock;
+static struct extra_have_objects shallow;
static enum deny_action parse_deny_action(const char *var, const char *value)
{
@@ -751,6 +755,13 @@ static void execute_commands(struct command *commands, const char *unpacker_erro
}
}
+static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)
+{
+ ALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);
+ hashcpy(&(extra->array[extra->nr][0]), sha1);
+ extra->nr++;
+}
+
static struct command *read_head_info(void)
{
struct command *commands = NULL;
@@ -765,6 +776,17 @@ static struct command *read_head_info(void)
line = packet_read_line(0, &len);
if (!line)
break;
+
+ if (len == 48 && !prefixcmp(line, "shallow ")) {
+ if (get_sha1_hex(line + 8, old_sha1))
+ die("protocol error: expected shallow sha, got '%s'", line + 8);
+ if (!has_sha1_file(old_sha1)) {
+ add_extra_have(&shallow, old_sha1);
+ shallow_changed = 1;
+ }
+ continue;
+ }
+
if (len < 83 ||
line[40] != ' ' ||
line[81] != ' ' ||
@@ -827,6 +849,13 @@ static const char *unpack(int err_fd)
? transfer_fsck_objects
: 0);
+ if (shallow_changed)
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ &shallow,
+ WRITE_SHALLOW_NO_CUT);
+
+
hdr_err = parse_pack_header(&hdr);
if (hdr_err) {
if (err_fd > 0)
@@ -840,7 +869,12 @@ static const char *unpack(int err_fd)
if (ntohl(hdr.hdr_entries) < unpack_limit) {
int code, i = 0;
struct child_process child;
- const char *unpacker[5];
+ const char *unpacker[7];
+ if (alternate_shallow_file &&
+ *alternate_shallow_file) {
+ unpacker[i++] = "--shallow-file";
+ unpacker[i++] = alternate_shallow_file;
+ }
unpacker[i++] = "unpack-objects";
if (quiet)
unpacker[i++] = "-q";
@@ -854,15 +888,19 @@ static const char *unpack(int err_fd)
child.err = err_fd;
child.git_cmd = 1;
code = run_command(&child);
- if (!code)
- return NULL;
- return "unpack-objects abnormal exit";
+ if (code)
+ return "unpack-objects abnormal exit";
} else {
- const char *keeper[7];
+ const char *keeper[9];
int s, status, i = 0;
char keep_arg[256];
struct child_process ip;
+ if (alternate_shallow_file &&
+ *alternate_shallow_file) {
+ keeper[i++] = "--shallow-file";
+ keeper[i++] = alternate_shallow_file;
+ }
s = sprintf(keep_arg, "--keep=receive-pack %"PRIuMAX" on ", (uintmax_t) getpid());
if (gethostname(keep_arg + s, sizeof(keep_arg) - s))
strcpy(keep_arg + s, "localhost");
@@ -887,12 +925,20 @@ static const char *unpack(int err_fd)
pack_lockfile = index_pack_lockfile(ip.out);
close(ip.out);
status = finish_command(&ip);
- if (!status) {
- reprepare_packed_git();
- return NULL;
- }
- return "index-pack abnormal exit";
+ if (status)
+ return "index-pack abnormal exit";
+ reprepare_packed_git();
+ }
+
+ if (shallow_changed) {
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ &shallow,
+ WRITE_SHALLOW_NO_CUT |
+ WRITE_SHALLOW_REWRITE);
+ commit_lock_file(&shallow_lock);
}
+ return NULL;
}
static const char *unpack_with_sideband(void)
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 4be5931..7dd254f 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -184,7 +184,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
(send_all && args.send_mirror))
usage(send_pack_usage);
- if (is_repository_shallow())
+ if (is_repository_shallow() && args.stateless_rpc)
die("attempt to push from a shallow repository");
if (remote_name) {
diff --git a/send-pack.c b/send-pack.c
index 7d172ef..dcd3345 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -212,6 +212,9 @@ int send_pack(struct send_pack_args *args,
return 0;
}
+ if (!args->dry_run)
+ advertise_shallow_grafts(out);
+
/*
* Finally, tell the other end!
*/
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
new file mode 100755
index 0000000..37ad69b
--- /dev/null
+++ b/t/t5537-push-shallow.sh
@@ -0,0 +1,74 @@
+#!/bin/sh
+
+test_description='push from/to a shallow clone'
+
+. ./test-lib.sh
+
+commit() {
+ echo "$1" >tracked &&
+ git add tracked &&
+ git commit -m "$1"
+}
+
+test_expect_success 'setup' '
+ git config --global transfer.fsckObjects true &&
+ commit 1 &&
+ commit 2 &&
+ commit 3 &&
+ commit 4 &&
+ (
+ git init full-abc &&
+ cd full-abc &&
+ commit a &&
+ commit b &&
+ commit c
+ ) &&
+ git clone --no-local --depth=2 .git shallow &&
+ git --git-dir=shallow/.git log --format=%s >actual &&
+ cat <<EOF >expect &&
+4
+3
+EOF
+ test_cmp expect actual &&
+ git clone --no-local --depth=2 full-abc/.git shallow2 &&
+ git --git-dir=shallow2/.git log --format=%s >actual &&
+ cat <<EOF >expect &&
+c
+b
+EOF
+ test_cmp expect actual
+'
+
+test_expect_success 'push from shallow clone' '
+ (
+ cd shallow &&
+ commit 5 &&
+ git push ../.git +master:refs/remotes/shallow/master
+ ) &&
+ git log --format=%s shallow/master >actual &&
+ git fsck &&
+ cat <<EOF >expect &&
+5
+4
+3
+2
+1
+EOF
+ test_cmp expect actual
+'
+
+test_expect_success 'push from shallow clone, with grafted roots' '
+ (
+ cd shallow2 &&
+ git push ../.git +master:refs/remotes/shallow2/master
+ ) &&
+ git log --format=%s shallow2/master >actual &&
+ git fsck &&
+ cat <<EOF >expect &&
+c
+b
+EOF
+ test_cmp expect actual
+'
+
+test_done
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 07/16] send-pack: support pushing to a shallow clone
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (5 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 06/16] {send,receive}-pack: support pushing from a shallow clone Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 08/16] upload-pack: let pack-objects do the object counting in shallow case Nguyễn Thái Ngọc Duy
` (9 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
When send-pack receives "shallow" lines from receive-pack, it knows
the other end does not have a complete commit chains. It restrict
itself to the commits that are not cut out by either end to make sure
the result pack is usuable by receive-pack.
The same technique here, using setup_alternate_shallow() and
--shallow-file, might simplify similar code in upload-pack.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/send-pack.c | 7 +++++--
send-pack.c | 37 ++++++++++++++++++++++++++++++++-----
send-pack.h | 4 +++-
t/t5537-push-shallow.sh | 36 ++++++++++++++++++++++++++++++++++++
transport.c | 7 +++----
5 files changed, 79 insertions(+), 12 deletions(-)
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 7dd254f..14a73ba 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -94,6 +94,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
int fd[2];
struct child_process *conn;
struct extra_have_objects extra_have;
+ struct extra_have_objects shallow;
struct ref *remote_refs, *local_refs;
int ret;
int helper_status = 0;
@@ -209,8 +210,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
}
memset(&extra_have, 0, sizeof(extra_have));
+ memset(&shallow, 0, sizeof(shallow));
- get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);
+ get_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,
+ &extra_have, &shallow);
transport_verify_remote_names(nr_refspecs, refspecs);
@@ -230,7 +233,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
set_ref_status_for_push(remote_refs, args.send_mirror,
args.force_update);
- ret = send_pack(&args, fd, conn, remote_refs, &extra_have);
+ ret = send_pack(&args, fd, conn, remote_refs, &extra_have, &shallow);
if (helper_status)
print_helper_status(remote_refs);
diff --git a/send-pack.c b/send-pack.c
index dcd3345..1b828b0 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -27,14 +27,19 @@ static int feed_object(const unsigned char *sha1, int fd, int negative)
/*
* Make a pack stream and spit it out into file descriptor fd
*/
-static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *extra, struct send_pack_args *args)
+static int pack_objects(int fd, struct ref *refs,
+ struct extra_have_objects *extra,
+ struct extra_have_objects *extra_shallow,
+ struct send_pack_args *args)
{
/*
* The child becomes pack-objects --revs; we feed
* the revision parameters to it via its stdin and
* let its stdout go back to the other end.
*/
- const char *argv[] = {
+ const char *av[] = {
+ "--shallow-file",
+ NULL,
"pack-objects",
"--all-progress-implied",
"--revs",
@@ -45,10 +50,26 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
NULL,
NULL,
};
+ const char **argv;
struct child_process po;
+ static struct lock_file shallow_lock;
+ const char *alternate_shallow_file = NULL;
int i;
- i = 4;
+ if (extra_shallow->nr) {
+ memset(&shallow_lock, 0, sizeof(shallow_lock));
+ /* just to load up .git/shallow if exists */
+ is_repository_shallow();
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ extra_shallow, 0);
+ av[1] = alternate_shallow_file;
+ argv = av;
+ i = 6;
+ } else {
+ argv = &av[2];
+ i = 4;
+ }
if (args->use_thin_pack)
argv[i++] = "--thin";
if (args->use_ofs_delta)
@@ -100,6 +121,10 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext
if (finish_command(&po))
return -1;
+
+ if (extra_shallow->nr)
+ rollback_lock_file(&shallow_lock);
+
return 0;
}
@@ -176,7 +201,8 @@ static int sideband_demux(int in, int out, void *data)
int send_pack(struct send_pack_args *args,
int fd[], struct child_process *conn,
struct ref *remote_refs,
- struct extra_have_objects *extra_have)
+ struct extra_have_objects *extra_have,
+ struct extra_have_objects *extra_shallow)
{
int in = fd[0];
int out = fd[1];
@@ -294,7 +320,8 @@ int send_pack(struct send_pack_args *args,
}
if (new_refs && cmds_sent) {
- if (pack_objects(out, remote_refs, extra_have, args) < 0) {
+ if (pack_objects(out, remote_refs, extra_have, extra_shallow,
+ args) < 0) {
for (ref = remote_refs; ref; ref = ref->next)
ref->status = REF_STATUS_NONE;
if (args->stateless_rpc)
diff --git a/send-pack.h b/send-pack.h
index 05d7ab1..1c8a343 100644
--- a/send-pack.h
+++ b/send-pack.h
@@ -16,6 +16,8 @@ struct send_pack_args {
int send_pack(struct send_pack_args *args,
int fd[], struct child_process *conn,
- struct ref *remote_refs, struct extra_have_objects *extra_have);
+ struct ref *remote_refs,
+ struct extra_have_objects *extra_have,
+ struct extra_have_objects *extra_shallow);
#endif
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index 37ad69b..beb85ce 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -71,4 +71,40 @@ EOF
test_cmp expect actual
'
+test_expect_success 'push from shallow to shallow' '
+ (
+ cd shallow &&
+ git push ../shallow2/.git +master:refs/remotes/shallow/master
+ ) &&
+ (
+ cd shallow2 &&
+ git log --format=%s shallow/master >actual &&
+ git fsck &&
+ cat <<EOF >expect &&
+5
+4
+3
+EOF
+ test_cmp expect actual
+ )
+'
+
+test_expect_success 'push from full to shallow' '
+ ! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` &&
+ commit 1 &&
+ git push shallow2/.git +master:refs/remotes/top/master &&
+ (
+ cd shallow2 &&
+ git log --format=%s top/master >actual &&
+ git fsck &&
+ cat <<EOF >expect &&
+1
+4
+3
+EOF
+ test_cmp expect actual &&
+ git cat-file blob `echo 1|git hash-object --stdin` >/dev/null
+ )
+'
+
test_done
diff --git a/transport.c b/transport.c
index f167ca5..d592e8d 100644
--- a/transport.c
+++ b/transport.c
@@ -802,11 +802,10 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
struct ref *tmp_refs;
connect_setup(transport, 1, 0);
- get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);
+ get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,
+ NULL, &data->shallow);
data->got_remote_heads = 1;
}
- if (data->shallow.nr)
- die("pushing to a shallow repository is not supported");
memset(&args, 0, sizeof(args));
args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
@@ -819,7 +818,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
args.porcelain = !!(flags & TRANSPORT_PUSH_PORCELAIN);
ret = send_pack(&args, data->fd, data->conn, remote_refs,
- &data->extra_have);
+ &data->extra_have, &data->shallow);
close(data->fd[1]);
close(data->fd[0]);
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 08/16] upload-pack: let pack-objects do the object counting in shallow case
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (6 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 07/16] send-pack: support pushing to " Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 09/16] pack-protocol.txt: a bit about smart http Nguyễn Thái Ngọc Duy
` (8 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Remove the duplicate object counting code in upload-pack, dump out all
register_shallow()'d SHA-1 into a temporary shallow file and feed it
to pack-objects. The end result is the same, except with less code,
and fewer bytes sending over pipe to pack-objects.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
t/t5530-upload-pack-error.sh | 3 -
upload-pack.c | 128 ++++++++++---------------------------------
2 files changed, 30 insertions(+), 101 deletions(-)
diff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh
index c983d36..3932e79 100755
--- a/t/t5530-upload-pack-error.sh
+++ b/t/t5530-upload-pack-error.sh
@@ -54,9 +54,6 @@ test_expect_success 'upload-pack fails due to error in rev-list' '
printf "0032want %s\n0034shallow %s00000009done\n0000" \
$(git rev-parse HEAD) $(git rev-parse HEAD^) >input &&
test_must_fail git upload-pack . <input >/dev/null 2>output.err &&
- # pack-objects survived
- grep "Total.*, reused" output.err &&
- # but there was an error, which must have been in rev-list
grep "bad tree object" output.err
'
diff --git a/upload-pack.c b/upload-pack.c
index c377a3e..c3e68ae 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -35,7 +35,6 @@ static int no_done;
static int use_thin_pack, use_ofs_delta, use_include_tag;
static int no_progress, daemon_mode;
static int allow_tip_sha1_in_want;
-static int shallow_nr;
static struct object_array have_obj;
static struct object_array want_obj;
static struct object_array extra_edge_obj;
@@ -68,87 +67,32 @@ static ssize_t send_client_data(int fd, const char *data, ssize_t sz)
return sz;
}
-static FILE *pack_pipe = NULL;
-static void show_commit(struct commit *commit, void *data)
-{
- if (commit->object.flags & BOUNDARY)
- fputc('-', pack_pipe);
- if (fputs(sha1_to_hex(commit->object.sha1), pack_pipe) < 0)
- die("broken output pipe");
- fputc('\n', pack_pipe);
- fflush(pack_pipe);
- free(commit->buffer);
- commit->buffer = NULL;
-}
-
-static void show_object(struct object *obj,
- const struct name_path *path, const char *component,
- void *cb_data)
-{
- show_object_with_name(pack_pipe, obj, path, component);
-}
-
-static void show_edge(struct commit *commit)
-{
- fprintf(pack_pipe, "-%s\n", sha1_to_hex(commit->object.sha1));
-}
-
-static int do_rev_list(int in, int out, void *user_data)
-{
- int i;
- struct rev_info revs;
-
- pack_pipe = xfdopen(out, "w");
- init_revisions(&revs, NULL);
- revs.tag_objects = 1;
- revs.tree_objects = 1;
- revs.blob_objects = 1;
- if (use_thin_pack)
- revs.edge_hint = 1;
-
- for (i = 0; i < want_obj.nr; i++) {
- struct object *o = want_obj.objects[i].item;
- /* why??? */
- o->flags &= ~UNINTERESTING;
- add_pending_object(&revs, o, NULL);
- }
- for (i = 0; i < have_obj.nr; i++) {
- struct object *o = have_obj.objects[i].item;
- o->flags |= UNINTERESTING;
- add_pending_object(&revs, o, NULL);
- }
- setup_revisions(0, NULL, &revs, NULL);
- if (prepare_revision_walk(&revs))
- die("revision walk setup failed");
- mark_edges_uninteresting(revs.commits, &revs, show_edge);
- if (use_thin_pack)
- for (i = 0; i < extra_edge_obj.nr; i++)
- fprintf(pack_pipe, "-%s\n", sha1_to_hex(
- extra_edge_obj.objects[i].item->sha1));
- traverse_commit_list(&revs, show_commit, show_object, NULL);
- fflush(pack_pipe);
- fclose(pack_pipe);
- return 0;
-}
-
static void create_pack_file(void)
{
- struct async rev_list;
struct child_process pack_objects;
char data[8193], progress[128];
char abort_msg[] = "aborting due to possible repository "
"corruption on the remote side.";
int buffered = -1;
ssize_t sz;
- const char *argv[10];
- int arg = 0;
+ const char *argv[12];
+ int i, arg = 0;
+ FILE *pipe_fd;
+ static struct lock_file shallow_lock;
+ const char *alternate_shallow_file = NULL;
+
+ setup_alternate_shallow(&shallow_lock,
+ &alternate_shallow_file,
+ NULL, 0);
+ if (*alternate_shallow_file) {
+ argv[arg++] = "--shallow-file";
+ argv[arg++] = alternate_shallow_file;
+ }
argv[arg++] = "pack-objects";
- if (!shallow_nr) {
- argv[arg++] = "--revs";
- if (use_thin_pack)
- argv[arg++] = "--thin";
- }
+ argv[arg++] = "--revs";
+ if (use_thin_pack)
+ argv[arg++] = "--thin";
argv[arg++] = "--stdout";
if (!no_progress)
@@ -169,28 +113,18 @@ static void create_pack_file(void)
if (start_command(&pack_objects))
die("git upload-pack: unable to fork git-pack-objects");
- if (shallow_nr) {
- memset(&rev_list, 0, sizeof(rev_list));
- rev_list.proc = do_rev_list;
- rev_list.out = pack_objects.in;
- if (start_async(&rev_list))
- die("git upload-pack: unable to fork git-rev-list");
- }
- else {
- FILE *pipe_fd = xfdopen(pack_objects.in, "w");
- int i;
+ pipe_fd = xfdopen(pack_objects.in, "w");
- for (i = 0; i < want_obj.nr; i++)
- fprintf(pipe_fd, "%s\n",
- sha1_to_hex(want_obj.objects[i].item->sha1));
- fprintf(pipe_fd, "--not\n");
- for (i = 0; i < have_obj.nr; i++)
- fprintf(pipe_fd, "%s\n",
- sha1_to_hex(have_obj.objects[i].item->sha1));
- fprintf(pipe_fd, "\n");
- fflush(pipe_fd);
- fclose(pipe_fd);
- }
+ for (i = 0; i < want_obj.nr; i++)
+ fprintf(pipe_fd, "%s\n",
+ sha1_to_hex(want_obj.objects[i].item->sha1));
+ fprintf(pipe_fd, "--not\n");
+ for (i = 0; i < have_obj.nr; i++)
+ fprintf(pipe_fd, "%s\n",
+ sha1_to_hex(have_obj.objects[i].item->sha1));
+ fprintf(pipe_fd, "\n");
+ fflush(pipe_fd);
+ fclose(pipe_fd);
/* We read from pack_objects.err to capture stderr output for
@@ -290,8 +224,9 @@ static void create_pack_file(void)
error("git upload-pack: git-pack-objects died with error.");
goto fail;
}
- if (shallow_nr && finish_async(&rev_list))
- goto fail; /* error was already reported */
+
+ if (*alternate_shallow_file)
+ rollback_lock_file(&shallow_lock);
/* flush the data */
if (0 <= buffered) {
@@ -575,7 +510,6 @@ static void receive_needs(void)
int depth = 0;
int has_non_tip = 0;
- shallow_nr = 0;
for (;;) {
struct object *o;
const char *features;
@@ -679,7 +613,6 @@ static void receive_needs(void)
packet_write(1, "shallow %s",
sha1_to_hex(object->sha1));
register_shallow(object->sha1);
- shallow_nr++;
}
result = result->next;
}
@@ -715,7 +648,6 @@ static void receive_needs(void)
register_shallow(shallows.objects[i].item->sha1);
}
- shallow_nr += shallows.nr;
free(shallows.objects);
}
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 09/16] pack-protocol.txt: a bit about smart http
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (7 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 08/16] upload-pack: let pack-objects do the object counting in shallow case Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 10/16] Add document for command arguments for supporting " Nguyễn Thái Ngọc Duy
` (7 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/technical/pack-protocol.txt | 66 +++++++++++++++++++++++++++++++
1 file changed, 66 insertions(+)
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index c73b62f..a1672bc 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -551,3 +551,69 @@ An example client/server communication might look like this:
S: 0018ok refs/heads/debug\n
S: 002ang refs/heads/master non-fast-forward\n
----
+
+Smart HTTP Transport
+--------------------
+
+Smart HTTP protocol is basically "git protocol on top of http". The
+base protocol is modified slightly to fit HTTP processing model: no
+bidirectional full-duplex connections, the program may read the
+request, write a response and must exit.
+
+Reference Discovery
+-------------------
+
+The server end always sends the list of references in both push and
+fetch cases. This ref list is retrieved by the client's sending HTTP
+GET request to a smart http url ending with
+"/info/refs?service=<service>" where <service> could be either
+git-upload-pack or git-receive-pack for fetching or pushing
+respectively. The output is in pkt-line format.
+
+----
+ advertised-refs = service
+ flush-pkt
+ (no-refs / list-of-refs)
+ flush-pkt
+
+ service = PKT-LINE("# service=" service-name)
+ service-name = ("git-upload-pack" / "git-receive-pack")
+
+ no-refs = PKT-LINE(zero-id SP "capabilities^{}"
+ NUL capability-list LF)
+
+ list-of-refs = first-ref *other-ref
+ first-ref = PKT-LINE(obj-id SP refname
+ NUL capability-list LF)
+
+ other-ref = PKT-LINE(other-tip / other-peeled)
+ other-tip = obj-id SP refname LF
+ other-peeled = obj-id SP refname "^{}" LF
+
+ capability-list = capability *(SP capability)
+ capability = 1*(LC_ALPHA / DIGIT / "-" / "_")
+ LC_ALPHA = %x61-7A
+----
+
+Packfile Negotiation
+--------------------
+
+For fetching, packet negotiation is via a series of HTTP POST requests
+to an url ending with "/git-upload-pack" with the content in pkt-line
+format. git-upload-pack's response consists of a "service" line like
+in Reference Discovery followed by normal git-upload-pack packet
+lines. Capability multi_ack_detailed is required by Smart HTTP.
+
+Common objects that are discovered are appended onto the request as
+have lines and are sent again on the next request. This allows the
+remote side to reinitialize its in-memory list of common objects
+during the next request and the remote does not need to maintain the
+negotiation state.
+
+Reference Update Request
+------------------------
+
+For pushing, a HTTP POST request is sent to an url ending with
+"/git-receive-pack" with the content in pkt-line format.
+git-receive-pack's response consists of a "service" line like in
+Reference Discovery followed by normal git-receive-pack packet lines.
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 10/16] Add document for command arguments for supporting smart http
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (8 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 09/16] pack-protocol.txt: a bit about smart http Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 11/16] {fetch,upload}-pack: support fetching from a shallow clone via " Nguyễn Thái Ngọc Duy
` (6 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/git-fetch-pack.txt | 11 +++++++----
Documentation/git-receive-pack.txt | 16 +++++++++++++++-
Documentation/git-send-pack.txt | 9 ++++++++-
Documentation/git-upload-pack.txt | 13 ++++++++++++-
4 files changed, 42 insertions(+), 7 deletions(-)
diff --git a/Documentation/git-fetch-pack.txt b/Documentation/git-fetch-pack.txt
index 1e71754..85a9437 100644
--- a/Documentation/git-fetch-pack.txt
+++ b/Documentation/git-fetch-pack.txt
@@ -9,10 +9,7 @@ git-fetch-pack - Receive missing objects from another repository
SYNOPSIS
--------
[verse]
-'git fetch-pack' [--all] [--quiet|-q] [--keep|-k] [--thin] [--include-tag]
- [--upload-pack=<git-upload-pack>]
- [--depth=<n>] [--no-progress]
- [-v] [<host>:]<directory> [<refs>...]
+'git fetch-pack' [options] [<host>:]<directory> [<refs>...]
DESCRIPTION
-----------
@@ -90,6 +87,12 @@ be in a separate packet, and the list must end with a flush packet.
--no-progress::
Do not show the progress.
+--stateless-rpc::
+ Smart HTTP mode.
+
+--lock-pack::
+ Issue "lock" command to the remote helper via stdout.
+
-v::
Run verbosely.
diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt
index b1f7dc6..b56d2eb 100644
--- a/Documentation/git-receive-pack.txt
+++ b/Documentation/git-receive-pack.txt
@@ -9,7 +9,7 @@ git-receive-pack - Receive what is pushed into the repository
SYNOPSIS
--------
[verse]
-'git-receive-pack' <directory>
+'git-receive-pack' [options] <directory>
DESCRIPTION
-----------
@@ -35,6 +35,20 @@ are not fast-forwards.
OPTIONS
-------
+--stateless-rpc::
+ git-receive-pack performs only a single read-write cycle with
+ stdin and stdout to fit with the HTTP POST request processing
+ model where a program may read the request, write a response,
+ and must exit.
+
+--advertise-refs::
+ Only the initial ref advertisement is output then exits
+ immediately to fit with the HTTP GET request model, where no
+ request content is received but a response must be produced.
+
+--quiet::
+ Make unpack-objects at the receive-pack end quiet.
+
<directory>::
The repository to sync into.
diff --git a/Documentation/git-send-pack.txt b/Documentation/git-send-pack.txt
index dc3a568..a88e7e0 100644
--- a/Documentation/git-send-pack.txt
+++ b/Documentation/git-send-pack.txt
@@ -9,7 +9,7 @@ git-send-pack - Push objects over Git protocol to another repository
SYNOPSIS
--------
[verse]
-'git send-pack' [--all] [--dry-run] [--force] [--receive-pack=<git-receive-pack>] [--verbose] [--thin] [<host>:]<directory> [<ref>...]
+'git send-pack' [options] [<host>:]<directory> [<ref>...]
DESCRIPTION
-----------
@@ -52,6 +52,13 @@ OPTIONS
Send a "thin" pack, which records objects in deltified form based
on objects not included in the pack to reduce network traffic.
+--stateless-rpc::
+ Smart HTTP mode.
+
+--helper-status:
+ Issue status commands (e.g. "ok" or "error") to the remote
+ help via stdout.
+
<host>::
A remote host to house the repository. When this
part is specified, 'git-receive-pack' is invoked via
diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt
index 0abc806..98d73cc 100644
--- a/Documentation/git-upload-pack.txt
+++ b/Documentation/git-upload-pack.txt
@@ -9,7 +9,7 @@ git-upload-pack - Send objects packed back to git-fetch-pack
SYNOPSIS
--------
[verse]
-'git-upload-pack' [--strict] [--timeout=<n>] <directory>
+'git-upload-pack' [options] <directory>
DESCRIPTION
-----------
@@ -31,6 +31,17 @@ OPTIONS
--timeout=<n>::
Interrupt transfer after <n> seconds of inactivity.
+--stateless-rpc::
+ git-upload-pack performs only a single read-write cycle with
+ stdin and stdout to fit with the HTTP POST request processing
+ model where a program may read the request, write a response,
+ and must exit.
+
+--advertise-refs::
+ Only the initial ref advertisement is output then exits
+ immediately to fit with the HTTP GET request model, where no
+ request content is received but a response must be produced.
+
<directory>::
The repository to sync from.
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 11/16] {fetch,upload}-pack: support fetching from a shallow clone via smart http
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (9 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 10/16] Add document for command arguments for supporting " Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 12/16] receive-pack: support pushing to a shallow clone via http Nguyễn Thái Ngọc Duy
` (5 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/technical/pack-protocol.txt | 3 +++
builtin/fetch-pack.c | 3 +--
remote-curl.c | 4 +++-
t/t5536-fetch-shallow.sh | 27 +++++++++++++++++++++++++++
upload-pack.c | 2 --
5 files changed, 34 insertions(+), 5 deletions(-)
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index a1672bc..5013652 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -574,6 +574,7 @@ respectively. The output is in pkt-line format.
advertised-refs = service
flush-pkt
(no-refs / list-of-refs)
+ *shallow
flush-pkt
service = PKT-LINE("# service=" service-name)
@@ -590,6 +591,8 @@ respectively. The output is in pkt-line format.
other-tip = obj-id SP refname LF
other-peeled = obj-id SP refname "^{}" LF
+ shallow = PKT-LINE("shallow" SP obj-id)
+
capability-list = capability *(SP capability)
capability = 1*(LC_ALPHA / DIGIT / "-" / "_")
LC_ALPHA = %x61-7A
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index f6a6d76..b89d753 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -146,8 +146,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
}
memset(&shallow, 0, sizeof(shallow));
- get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,
- args.stateless_rpc ? NULL : &shallow);
+ get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
ref = fetch_pack(&args, fd, conn, ref, dest,
sought, nr_sought, &shallow, pack_lockfile_ptr);
diff --git a/remote-curl.c b/remote-curl.c
index c329bd3..de2cc8a 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -78,6 +78,7 @@ struct discovery {
char *buf;
size_t len;
struct ref *refs;
+ struct extra_have_objects shallow;
unsigned proto_git : 1;
};
static struct discovery *last_discovery;
@@ -86,7 +87,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)
{
struct ref *list = NULL;
get_remote_heads(-1, heads->buf, heads->len, &list,
- for_push ? REF_NORMAL : 0, NULL, NULL);
+ for_push ? REF_NORMAL : 0, NULL, &heads->shallow);
return list;
}
@@ -146,6 +147,7 @@ static void free_discovery(struct discovery *d)
if (d) {
if (d == last_discovery)
last_discovery = NULL;
+ free(d->shallow.array);
free(d->buf_alloc);
free_refs(d->refs);
free(d);
diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
index 15a8208..6ea6347 100755
--- a/t/t5536-fetch-shallow.sh
+++ b/t/t5536-fetch-shallow.sh
@@ -102,4 +102,31 @@ EOF
'
+if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
+ say 'skipping remaining tests, git built without http support'
+ test_done
+fi
+
+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5536'}
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
+
+test_expect_success 'clone http repository' '
+ git clone --bare --no-local shallow "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ git clone --quiet $HTTPD_URL/smart/repo.git clone &&
+ (
+ cd clone &&
+ git fsck &&
+ git log --format=%s origin/master >actual &&
+ cat <<EOF >expect &&
+6
+5
+4
+3
+EOF
+ test_cmp expect actual
+ )
+'
+
+stop_httpd
test_done
diff --git a/upload-pack.c b/upload-pack.c
index c3e68ae..263ae08 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -770,8 +770,6 @@ int main(int argc, char **argv)
if (!enter_repo(dir, strict))
die("'%s' does not appear to be a git repository", dir);
- if (is_repository_shallow() && stateless_rpc)
- die("attempt to push into a shallow repository");
git_config(upload_pack_config, NULL);
upload_pack();
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 12/16] receive-pack: support pushing to a shallow clone via http
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (10 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 11/16] {fetch,upload}-pack: support fetching from a shallow clone via " Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 13/16] send-pack: support pushing from " Nguyễn Thái Ngọc Duy
` (4 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/receive-pack.c | 3 ---
t/t5537-push-shallow.sh | 35 +++++++++++++++++++++++++++++++++++
2 files changed, 35 insertions(+), 3 deletions(-)
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index cbb2025..54bf6b2 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1038,9 +1038,6 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
if (!enter_repo(dir, 0))
die("'%s' does not appear to be a git repository", dir);
- if (is_repository_shallow() && stateless_rpc)
- die("attempt to push into a shallow repository");
-
git_config(receive_pack_config, NULL);
if (0 <= transfer_unpack_limit)
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index beb85ce..02d9198 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -16,6 +16,7 @@ test_expect_success 'setup' '
commit 2 &&
commit 3 &&
commit 4 &&
+ git clone . full &&
(
git init full-abc &&
cd full-abc &&
@@ -107,4 +108,38 @@ EOF
)
'
+if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
+ say 'skipping remaining tests, git built without http support'
+ test_done
+fi
+
+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5537'}
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
+
+test_expect_success 'push to shallow repo via http' '
+ git clone --bare --no-local shallow "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ (
+ cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ git config http.receivepack true
+ ) &&
+ (
+ cd full &&
+ commit 9 &&
+ git push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master
+ ) &&
+ (
+ cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ git fsck &&
+ git log --format=%s top/master >actual &&
+ cat <<EOF >expect &&
+9
+4
+3
+EOF
+ test_cmp expect actual
+ )
+'
+
+stop_httpd
test_done
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 13/16] send-pack: support pushing from a shallow clone via http
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (11 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 12/16] receive-pack: support pushing to a shallow clone via http Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 14/16] git-clone.txt: remove shallow clone limitations Nguyễn Thái Ngọc Duy
` (3 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/send-pack.c | 3 ---
send-pack.c | 19 +++++++++++++++++--
t/t5537-push-shallow.sh | 25 +++++++++++++++++++++++++
3 files changed, 42 insertions(+), 5 deletions(-)
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 14a73ba..b177120 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -185,9 +185,6 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
(send_all && args.send_mirror))
usage(send_pack_usage);
- if (is_repository_shallow() && args.stateless_rpc)
- die("attempt to push from a shallow repository");
-
if (remote_name) {
remote = remote_get(remote_name);
if (!remote_has_url(remote, dest)) {
diff --git a/send-pack.c b/send-pack.c
index 1b828b0..520c838 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -198,6 +198,21 @@ static int sideband_demux(int in, int out, void *data)
return ret;
}
+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)
+{
+ struct strbuf *sb = cb;
+ if (graft->nr_parent == -1)
+ packet_buf_write(sb, "shallow %s\n", sha1_to_hex(graft->sha1));
+ return 0;
+}
+
+void advertise_shallow_grafts_buf(struct strbuf *sb)
+{
+ if (!is_repository_shallow())
+ return;
+ for_each_commit_graft(advertise_shallow_grafts_cb, sb);
+}
+
int send_pack(struct send_pack_args *args,
int fd[], struct child_process *conn,
struct ref *remote_refs,
@@ -239,7 +254,7 @@ int send_pack(struct send_pack_args *args,
}
if (!args->dry_run)
- advertise_shallow_grafts(out);
+ advertise_shallow_grafts_buf(&req_buf);
/*
* Finally, tell the other end!
@@ -299,7 +314,7 @@ int send_pack(struct send_pack_args *args,
}
if (args->stateless_rpc) {
- if (!args->dry_run && cmds_sent) {
+ if (!args->dry_run && (cmds_sent || is_repository_shallow())) {
packet_buf_flush(&req_buf);
send_sideband(out, -1, req_buf.buf, req_buf.len, LARGE_PACKET_MAX);
}
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index 02d9198..8bea496 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -141,5 +141,30 @@ EOF
)
'
+test_expect_success 'push from shallow repo via http' '
+ mv "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" shallow-upstream.git &&
+ git clone --bare --no-local full "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ (
+ cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ git config http.receivepack true
+ ) &&
+ commit 10 &&
+ git push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master &&
+ (
+ cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ git fsck &&
+ git log --format=%s top/master >actual &&
+ cat <<EOF >expect &&
+10
+1
+4
+3
+2
+1
+EOF
+ test_cmp expect actual
+ )
+'
+
stop_httpd
test_done
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 14/16] git-clone.txt: remove shallow clone limitations
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (12 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 13/16] send-pack: support pushing from " Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one Nguyễn Thái Ngọc Duy
` (2 subsequent siblings)
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Now that git supports push/pull from/to a shallow clone, these
limitations are not true anymore.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/git-clone.txt | 7 +------
1 file changed, 1 insertion(+), 6 deletions(-)
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 450f158..4987857 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -181,12 +181,7 @@ objects from the source repository into a pack in the cloned repository.
--depth <depth>::
Create a 'shallow' clone with a history truncated to the
- specified number of revisions. A shallow repository has a
- number of limitations (you cannot clone or fetch from
- it, nor push from nor into it), but is adequate if you
- are only interested in the recent history of a large project
- with a long history, and would want to send in fixes
- as patches.
+ specified number of revisions.
--[no-]single-branch::
Clone only the history leading to the tip of a single branch,
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (13 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 14/16] git-clone.txt: remove shallow clone limitations Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-22 19:23 ` Philip Oakley
2013-07-20 9:58 ` [PATCH v2 16/16] clone: use git protocol for cloning shallow repo locally Nguyễn Thái Ngọc Duy
2013-07-22 23:41 ` [PATCH v2 00/16] First class shallow clone Philip Oakley
16 siblings, 1 reply; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Documentation/config.txt | 5 +++++
builtin/receive-pack.c | 9 ++++++++-
cache.h | 1 +
config.c | 5 +++++
environment.c | 1 +
fetch-pack.c | 9 ++++++++-
t/t5536-fetch-shallow.sh | 9 +++++++++
t/t5537-push-shallow.sh | 6 ++++++
8 files changed, 43 insertions(+), 2 deletions(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 81856dd..e811180 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -655,6 +655,11 @@ core.abbrev::
for abbreviated object names to stay unique for sufficiently long
time.
+core.noshallow::
+ If true, reject any pushes or fetches that may turn the
+ repository into a shallow one. This setting is ignored if the
+ repository is already shallow.
+
add.ignore-errors::
add.ignoreErrors::
Tells 'git add' to continue adding files when some files cannot be
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 54bf6b2..95ea481 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -936,7 +936,14 @@ static const char *unpack(int err_fd)
&shallow,
WRITE_SHALLOW_NO_CUT |
WRITE_SHALLOW_REWRITE);
- commit_lock_file(&shallow_lock);
+ if (*alternate_shallow_file == '\0') {
+ unlink_or_warn(git_path("shallow"));
+ rollback_lock_file(&shallow_lock);
+ } else {
+ if (!is_repository_shallow() && cannot_be_shallow)
+ die("not allowed to turn this repository into a shallow one");
+ commit_lock_file(&shallow_lock);
+ }
}
return NULL;
}
diff --git a/cache.h b/cache.h
index 7f17228..3a52b08 100644
--- a/cache.h
+++ b/cache.h
@@ -592,6 +592,7 @@ extern int fsync_object_files;
extern int core_preload_index;
extern int core_apply_sparse_checkout;
extern int precomposed_unicode;
+extern int cannot_be_shallow;
/*
* The character that begins a commented line in user-editable file
diff --git a/config.c b/config.c
index d04e815..31f5a57 100644
--- a/config.c
+++ b/config.c
@@ -784,6 +784,11 @@ static int git_default_core_config(const char *var, const char *value)
return 0;
}
+ if (!strcmp(var, "core.noshallow")) {
+ cannot_be_shallow = git_config_bool(var, value);
+ return 0;
+ }
+
/* Add other config variables here and to Documentation/config.txt. */
return 0;
}
diff --git a/environment.c b/environment.c
index 0cb67b2..14c8005 100644
--- a/environment.c
+++ b/environment.c
@@ -61,6 +61,7 @@ int merge_log_config = -1;
int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
struct startup_info *startup_info;
unsigned long pack_size_limit_cfg;
+int cannot_be_shallow;
/*
* The character that begins a commented line in user-editable file
diff --git a/fetch-pack.c b/fetch-pack.c
index f337526..40e7aa2 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -960,7 +960,14 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
WRITE_SHALLOW_NO_CUT |
WRITE_SHALLOW_REWRITE);
}
- commit_lock_file(&shallow_lock);
+ if (*alternate_shallow_file == '\0') {
+ unlink_or_warn(git_path("shallow"));
+ rollback_lock_file(&shallow_lock);
+ } else {
+ if (!is_repository_shallow() && cannot_be_shallow)
+ die("not allowed to turn this repository into a shallow one");
+ commit_lock_file(&shallow_lock);
+ }
}
}
diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
index 6ea6347..b7f89b1 100755
--- a/t/t5536-fetch-shallow.sh
+++ b/t/t5536-fetch-shallow.sh
@@ -102,6 +102,15 @@ EOF
'
+test_expect_success 'core.noshallow' '
+ git init clean &&
+ (
+ cd clean &&
+ git config core.noshallow true &&
+ test_must_fail git fetch ../shallow/.git
+ )
+'
+
if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
say 'skipping remaining tests, git built without http support'
test_done
diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
index 8bea496..0edd51f 100755
--- a/t/t5537-push-shallow.sh
+++ b/t/t5537-push-shallow.sh
@@ -108,6 +108,12 @@ EOF
)
'
+test_expect_success 'core.noshallow' '
+ git init clean &&
+ git --git-dir=clean/.git config core.noshallow true &&
+ test_must_fail git --git-dir=shallow/.git push clean master:refs/remotes/shallow/master
+'
+
if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
say 'skipping remaining tests, git built without http support'
test_done
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* [PATCH v2 16/16] clone: use git protocol for cloning shallow repo locally
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (14 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one Nguyễn Thái Ngọc Duy
@ 2013-07-20 9:58 ` Nguyễn Thái Ngọc Duy
2013-07-22 23:41 ` [PATCH v2 00/16] First class shallow clone Philip Oakley
16 siblings, 0 replies; 44+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2013-07-20 9:58 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
clone_local() does not handle $SRC/shallow. It could be made so, but
it's simpler to use fetch-pack/upload-pack instead.
This used by be caught by the check in upload-pack, which is triggered
by transport_get_remote_refs(), even in local clone case. The check is
now gone and check_everything_connected() should catch the result
incomplete repo. But check_everything_connected() will soon be skipped
in local clone case, opening a door to corrupt repo. This patch should
close that door.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/clone.c | 14 ++++++++++++--
t/t5601-clone.sh | 7 +++++++
2 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index 17f57cd..09e383e 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -789,8 +789,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
else
repo = repo_name;
is_local = option_local != 0 && path && !is_bundle;
- if (is_local && option_depth)
- warning(_("--depth is ignored in local clones; use file:// instead."));
+ if (is_local) {
+ struct strbuf sb = STRBUF_INIT;
+ if (option_depth)
+ warning(_("--depth is ignored in local clones; use file:// instead."));
+ strbuf_addf(&sb, "%s/shallow", path);
+ if (!access(sb.buf, F_OK)) {
+ if (option_local > 0)
+ warning(_("source repository is shallow, ignoring --local"));
+ is_local = 0;
+ }
+ strbuf_release(&sb);
+ }
if (option_local > 0 && !is_local)
warning(_("--local is ignored"));
diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index 0629149..35a2a3d 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -285,4 +285,11 @@ test_expect_success NOT_MINGW,NOT_CYGWIN 'clone local path foo:bar' '
git clone "./foo:bar" foobar
'
+test_expect_success 'shallow clone locally' '
+ git clone --depth=1 --no-local src ssrrcc &&
+ git clone ssrrcc ddsstt &&
+ test_cmp ssrrcc/.git/shallow ddsstt/.git/shallow &&
+ ( cd ddsstt && git fsck )
+'
+
test_done
--
1.8.2.83.gc99314b
^ permalink raw reply related [flat|nested] 44+ messages in thread
* Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository
2013-07-20 9:57 ` [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository Nguyễn Thái Ngọc Duy
@ 2013-07-22 19:10 ` Philip Oakley
2013-07-23 2:06 ` Duy Nguyen
0 siblings, 1 reply; 44+ messages in thread
From: Philip Oakley @ 2013-07-22 19:10 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy, git; +Cc: Junio C Hamano
From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
> upload-pack already advertises all shallow grafts if server repository
> is shallow. This information can be used to add more grafts to the
> client if the server sends commit chains down to its graft points.
>
> If the server is shallow, before we receive the pack, we setup a
> temporary shallow file that contains both local graft points and the
> server's. This stops index-pack from going beyond server's graft
> points.
>
> Only server graft points that do not have corresponding SHA-1s in
> local repo are added to the temp shallow file because we don't want to
> accidentally cut the client history because the server's is
> shorter. The client cutting can only happen when --depth is requested.
>
> After index-pack finishes successfully, we write the temporary shallow
> down with one exception: unused graft points provided by the server
> are removed. We don't want those lying around and suddenly become
> active.
>
> Note that in the "shallow -> shallow" case, the server might not have
> enough information to find common roots to create an optimum pack. It
> might send complete commit chains down to the graft points as a
> result. I don't think we can improve this, unless upload-pack somehow
> has access to a full repository.
>
> "shallow -> shallow" case only makes sense when the upstream provides
> a stable shallow repo (e.g. make a cut every year or so and ask devs
> to all move to the new base). If the cloned repos are all based on a
> stable (shallow) upstream, the above problem is unlikely to happen.
>
> A side effect of this change is we can now clone from a shallow
> repository.
> And a full repository may automatically become shallow if
> you fetch from a shallow repository.
This "sounds" risky: Perhaps add a note about '[PATCH v2 15/16] config:
add core.noshallow to prevent turning a repo into a shallow one' which
will become available later in the series. The later core.noshallow may
be a hostage to fortune.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
> builtin/fetch-pack.c | 7 ++-
> commit.h | 27 +++++++++-
> fetch-pack.c | 30 +++++++++--
> fetch-pack.h | 1 +
> shallow.c | 90
> +++++++++++++++++++++++++++++---
> t/t5536-fetch-shallow.sh (new +x) | 105
> ++++++++++++++++++++++++++++++++++++++
> transport.c | 10 ++--
> 7 files changed, 252 insertions(+), 18 deletions(-)
> create mode 100755 t/t5536-fetch-shallow.sh
>
> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
> index 080e599..f6a6d76 100644
> --- a/builtin/fetch-pack.c
> +++ b/builtin/fetch-pack.c
> @@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv,
> const char *prefix)
> char **pack_lockfile_ptr = NULL;
> struct child_process *conn;
> struct fetch_pack_args args;
> + struct extra_have_objects shallow;
>
> packet_trace_identity("fetch-pack");
>
> @@ -144,10 +145,12 @@ int cmd_fetch_pack(int argc, const char **argv,
> const char *prefix)
> args.verbose ? CONNECT_VERBOSE : 0);
> }
>
> - get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);
> + memset(&shallow, 0, sizeof(shallow));
> + get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,
> + args.stateless_rpc ? NULL : &shallow);
>
> ref = fetch_pack(&args, fd, conn, ref, dest,
> - sought, nr_sought, pack_lockfile_ptr);
> + sought, nr_sought, &shallow, pack_lockfile_ptr);
> if (pack_lockfile) {
> printf("lock %s\n", pack_lockfile);
> fflush(stdout);
> diff --git a/commit.h b/commit.h
> index 678fa20..97fd54e 100644
> --- a/commit.h
> +++ b/commit.h
> @@ -188,9 +188,32 @@ extern struct commit_list
> *get_shallow_commits(struct object_array *heads,
> extern void check_shallow_file_for_update(void);
> extern void set_alternate_shallow_file(const char *path);
> extern void advertise_shallow_grafts(int);
> -extern int write_shallow_commits(struct strbuf *out, int
> use_pack_protocol);
> +
> +/*
> + * Ignore graft points that cut out existing "good" parts of our
> + * history. Graft points pointing to incomplete commit islands are
> + * _not_ ignore.
> + */
> +#define WRITE_SHALLOW_NO_CUT 1
> +/*
> + * Remove graft points that are not attached to any SHA-1 in current
> + * history.
> + */
> +#define WRITE_SHALLOW_NO_UNUSED 2
> +/*
> + * Not really write_shallow_commits() flag, this lets
> + * setup_alternate_shallow know this is the "final" setup before
> + * committing to .git/shallow
> + */
> +#define WRITE_SHALLOW_REWRITE 4
> +
> +extern int write_shallow_commits(struct strbuf *out, int
> use_pack_protocol,
> + struct extra_have_objects *extra,
> + unsigned int flags);
> extern void setup_alternate_shallow(struct lock_file *shallow_lock,
> - const char **alternate_shallow_file);
> + const char **alternate_shallow_file,
> + struct extra_have_objects *extra,
> + unsigned int flags);
>
> int is_descendant_of(struct commit *, struct commit_list *);
> int in_merge_bases(struct commit *, struct commit *);
> diff --git a/fetch-pack.c b/fetch-pack.c
> index dc71a2b..f337526 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args
> *args,
> }
>
> if (is_repository_shallow())
> - write_shallow_commits(&req_buf, 1);
> + write_shallow_commits(&req_buf, 1, NULL, WRITE_SHALLOW_NO_CUT);
> if (args->depth > 0)
> packet_buf_write(&req_buf, "deepen %d", args->depth);
> packet_buf_flush(&req_buf);
> @@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct
> fetch_pack_args *args,
> int fd[2],
> const struct ref *orig_ref,
> struct ref **sought, int nr_sought,
> + struct extra_have_objects *shallow,
> char **pack_lockfile)
> {
> struct ref *ref = copy_ref_list(orig_ref);
> @@ -844,8 +845,10 @@ static struct ref *do_fetch_pack(struct
> fetch_pack_args *args,
>
> if (args->stateless_rpc)
> packet_flush(fd[1]);
> - if (args->depth > 0)
> - setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);
> + if (args->depth > 0 || shallow->nr)
> + setup_alternate_shallow(&shallow_lock, &alternate_shallow_file,
> + shallow,
> + WRITE_SHALLOW_NO_CUT);
> if (get_pack(args, fd, pack_lockfile))
> die("git fetch-pack: fetch failed.");
>
> @@ -922,6 +925,7 @@ struct ref *fetch_pack(struct fetch_pack_args
> *args,
> const struct ref *ref,
> const char *dest,
> struct ref **sought, int nr_sought,
> + struct extra_have_objects *shallow,
> char **pack_lockfile)
> {
> struct ref *ref_cpy;
> @@ -934,14 +938,30 @@ struct ref *fetch_pack(struct fetch_pack_args
> *args,
> packet_flush(fd[1]);
> die("no matching remote head");
> }
> - ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
> pack_lockfile);
> + ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
> + shallow, pack_lockfile);
>
> if (alternate_shallow_file) {
> if (*alternate_shallow_file == '\0') { /* --unshallow */
> unlink_or_warn(git_path("shallow"));
> rollback_lock_file(&shallow_lock);
> - } else
> + } else {
> + /*
> + * The server is a shallow clone and it sends
> + * us all of its shallow grafts. Some may be
> + * needed if the server sends objects down to
> + * the bottom. Remove all unused grafts.
> + */
> + if (shallow->nr) {
> + reprepare_packed_git();
> + setup_alternate_shallow(&shallow_lock,
> + &alternate_shallow_file,
> + shallow,
> + WRITE_SHALLOW_NO_CUT |
> + WRITE_SHALLOW_REWRITE);
> + }
> commit_lock_file(&shallow_lock);
> + }
> }
>
> reprepare_packed_git();
> diff --git a/fetch-pack.h b/fetch-pack.h
> index 40f08ba..c55bf1c 100644
> --- a/fetch-pack.h
> +++ b/fetch-pack.h
> @@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
> const char *dest,
> struct ref **sought,
> int nr_sought,
> + struct extra_have_objects *shallow,
> char **pack_lockfile);
>
> #endif
> diff --git a/shallow.c b/shallow.c
> index ee9edd4..337f676 100644
> --- a/shallow.c
> +++ b/shallow.c
> @@ -2,6 +2,7 @@
> #include "commit.h"
> #include "tag.h"
> #include "pkt-line.h"
> +#include "connected.h"
>
> static int is_shallow = -1;
> static struct stat shallow_stat;
> @@ -183,26 +184,103 @@ static int write_one_shallow(const struct
> commit_graft *graft, void *cb_data)
> return 0;
> }
>
> -int write_shallow_commits(struct strbuf *out, int use_pack_protocol)
> +static int sha1_iterate(void *cb, unsigned char *sha1)
> +{
> + if (is_null_sha1(cb))
> + return 1;
> + hashcpy(sha1, cb);
> + hashclr(cb);
> + return 0;
> +}
> +
> +struct connectivity_cache_item {
> + unsigned char sha1[20];
> + int good;
> +};
> +
> +static int good_connectivity_from(const unsigned char *sha1)
> +{
> + static struct connectivity_cache_item *cache;
> + static int nr, alloc;
> + int i;
> + unsigned char sha1_cb[20];
> +
> + for (i = 0; i < nr; i++)
> + if (!hashcmp(cache[i].sha1, sha1))
> + return cache[i].good;
> +
> + ALLOC_GROW(cache, nr + 1, alloc);
> + hashcpy(sha1_cb, sha1);
> + hashcpy(cache[nr].sha1, sha1);
> + cache[nr].good = !check_everything_connected(sha1_iterate, 1,
> sha1_cb);
> + nr++;
> + return cache[nr - 1].good;
> +}
> +
> +int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
> + struct extra_have_objects *extra,
> + unsigned int flags)
> {
> struct write_shallow_data data;
> + int i;
> data.out = out;
> data.use_pack_protocol = use_pack_protocol;
> data.count = 0;
> for_each_commit_graft(write_one_shallow, &data);
> + if (!extra)
> + return data.count;
> +
> + for (i = 0; i < extra->nr; i++) {
> + if ((flags & WRITE_SHALLOW_NO_CUT) &&
> + good_connectivity_from(extra->array[i]))
> + /*
> + * The server may have even shorter history
> + * than the client (e.g. the client has
> + * A-B-C-D but the server only has C-D). We do
> + * NOT want to cut client's history down to
> + * C-D simply because the server is set up
> + * so. If we don't have "C" and the server
> + * sends C-D to us, then we set up a graft at
> + * "C".
> + *
> + * If the client has (broken)-B-C-D, then "C"
> + * _is_ accepted. The -B-C-D is an incomplete
> + * island that normally has no connectivity to
> + * the rest of the repo.
> + */
> + continue;
> + if ((flags & WRITE_SHALLOW_NO_UNUSED) &&
> + !has_sha1_file(extra->array[i]))
> + continue;
> + strbuf_addstr(out, sha1_to_hex(extra->array[i]));
> + strbuf_addch(out, '\n');
> + data.count++;
> + }
> return data.count;
> }
>
> void setup_alternate_shallow(struct lock_file *shallow_lock,
> - const char **alternate_shallow_file)
> + const char **alternate_shallow_file,
> + struct extra_have_objects *extra,
> + unsigned int flags)
> {
> struct strbuf sb = STRBUF_INIT;
> + int rewrite = flags & WRITE_SHALLOW_REWRITE;
> int fd;
>
> - check_shallow_file_for_update();
> - fd = hold_lock_file_for_update(shallow_lock, git_path("shallow"),
> - LOCK_DIE_ON_ERROR);
> - if (write_shallow_commits(&sb, 0)) {
> + if (!rewrite)
> + fd = hold_lock_file_for_update(shallow_lock,
> + git_path("shallow"),
> + LOCK_DIE_ON_ERROR);
> + else {
> + flags |= WRITE_SHALLOW_NO_UNUSED;
> + fd = shallow_lock->fd;
> + }
> + if (write_shallow_commits(&sb, 0, extra, flags)) {
> + if (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||
> + ftruncate(fd, 0) == -1))
> + die_errno("unable to truncate the new shallow file %s",
> + shallow_lock->filename);
> if (write_in_full(fd, sb.buf, sb.len) != sb.len)
> die_errno("failed to write to %s",
> shallow_lock->filename);
> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
> new file mode 100755
> index 0000000..15a8208
> --- /dev/null
> +++ b/t/t5536-fetch-shallow.sh
> @@ -0,0 +1,105 @@
> +#!/bin/sh
> +
> +test_description='fetch/clone from a shallow clone'
> +
> +. ./test-lib.sh
> +
> +commit() {
> + echo "$1" >tracked &&
> + git add tracked &&
> + git commit -m "$1"
> +}
> +
> +test_expect_success 'setup' '
> + commit 1 &&
> + commit 2 &&
> + commit 3 &&
> + commit 4 &&
> + git config --global transfer.fsckObjects true
> +'
> +
> +test_expect_success 'setup shallow clone' '
> + git clone --no-local --depth=2 .git shallow &&
> + git --git-dir=shallow/.git log --format=%s >actual &&
> + cat <<EOF >expect &&
> +4
> +3
> +EOF
> + test_cmp expect actual
> +'
> +
> +test_expect_success 'clone from shallow clone' '
> + git clone --no-local shallow shallow2 &&
> + (
> + cd shallow2 &&
> + git fsck &&
> + git log --format=%s >actual &&
> + cat <<EOF >expect &&
> +4
> +3
> +EOF
> + test_cmp expect actual
> + )
> +'
> +
> +test_expect_success 'fetch from shallow clone' '
> + (
> + cd shallow &&
> + commit 5
> + ) &&
> + (
> + cd shallow2 &&
> + git fetch &&
> + git fsck &&
> + git log --format=%s origin/master >actual &&
> + cat <<EOF >expect &&
> +5
> +4
> +3
> +EOF
> + test_cmp expect actual
> + )
> +'
> +
> +test_expect_success 'fetch --depth from shallow clone' '
> + (
> + cd shallow &&
> + commit 6
> + ) &&
> + (
> + cd shallow2 &&
> + git fetch --depth=2 &&
> + git fsck &&
> + git log --format=%s origin/master >actual &&
> + cat <<EOF >expect &&
> +6
> +5
> +EOF
> + test_cmp expect actual
> + )
> +'
> +
> +test_expect_success 'fetch something upstream has but hidden by
> clients shallow boundaries' '
> + # the blob "1" is available in .git but hidden by the
> + # shallow2/.git/shallow and it should be resent
> + ! git --git-dir=shallow2/.git cat-file blob `echo 1|git
> hash-object --stdin` >/dev/null &&
> + echo 1 > 1.t &&
> + git add 1.t &&
> + git commit -m add-1-back &&
> + (
> + cd shallow2 &&
> + git fetch ../.git +refs/heads/master:refs/remotes/top/master &&
> + git fsck &&
> + git log --format=%s top/master >actual &&
> + cat <<EOF >expect &&
> +add-1-back
> +4
> +3
> +EOF
> + test_cmp expect actual
> + ) &&
> + git --git-dir=shallow2/.git cat-file blob `echo 1|git
> hash-object --stdin` >/dev/null
> +
> +'
> +
> +test_done
> diff --git a/transport.c b/transport.c
> index 10a8cb8..f167ca5 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -453,6 +453,7 @@ struct git_transport_data {
> int fd[2];
> unsigned got_remote_heads : 1;
> struct extra_have_objects extra_have;
> + struct extra_have_objects shallow;
> };
>
> static int set_git_option(struct git_transport_options *opts,
> @@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct
> transport *transport, int for_pus
>
> connect_setup(transport, for_push, 0);
> get_remote_heads(data->fd[0], NULL, 0, &refs,
> - for_push ? REF_NORMAL : 0, &data->extra_have, NULL);
> + for_push ? REF_NORMAL : 0,
> + &data->extra_have, &data->shallow);
> data->got_remote_heads = 1;
>
> return refs;
> @@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport
> *transport,
> if (!data->got_remote_heads) {
> connect_setup(transport, 0, 0);
> get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,
> - NULL, NULL);
> + NULL, &data->shallow);
> data->got_remote_heads = 1;
> }
>
> refs = fetch_pack(&args, data->fd, data->conn,
> refs_tmp ? refs_tmp : transport->remote_refs,
> - dest, to_fetch, nr_heads,
> + dest, to_fetch, nr_heads, &data->shallow,
> &transport->pack_lockfile);
> close(data->fd[0]);
> close(data->fd[1]);
> @@ -803,6 +805,8 @@ static int git_transport_push(struct transport
> *transport, struct ref *remote_re
> get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL,
> NULL);
> data->got_remote_heads = 1;
> }
> + if (data->shallow.nr)
> + die("pushing to a shallow repository is not supported");
>
> memset(&args, 0, sizeof(args));
> args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
> --
> 1.8.2.83.gc99314b
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date:
> 07/19/13
>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one
2013-07-20 9:58 ` [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one Nguyễn Thái Ngọc Duy
@ 2013-07-22 19:23 ` Philip Oakley
2013-07-23 1:28 ` Duy Nguyen
0 siblings, 1 reply; 44+ messages in thread
From: Philip Oakley @ 2013-07-22 19:23 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy, git; +Cc: Junio C Hamano
From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
Subject: [PATCH v2 15/16] config: add core.noshallow to prevent turning
a repo into a shallow one
Surely this should be the default now that it is possible to corrupt a
golden repo by pushing/fetching a shallow repository to it and it then
becomes shallow, and all the followers become shallow as well, with
consequent problems (IIUC) [PATCH v2 05/16].
It would be just as easy to change the config to core.allowshallow which
then must be enabled by the user, and can be mentioned in the shallow
clone option's documentation.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
> Documentation/config.txt | 5 +++++
> builtin/receive-pack.c | 9 ++++++++-
> cache.h | 1 +
> config.c | 5 +++++
> environment.c | 1 +
> fetch-pack.c | 9 ++++++++-
> t/t5536-fetch-shallow.sh | 9 +++++++++
> t/t5537-push-shallow.sh | 6 ++++++
> 8 files changed, 43 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 81856dd..e811180 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -655,6 +655,11 @@ core.abbrev::
> for abbreviated object names to stay unique for sufficiently long
> time.
>
> +core.noshallow::
> + If true, reject any pushes or fetches that may turn the
> + repository into a shallow one. This setting is ignored if the
> + repository is already shallow.
> +
> add.ignore-errors::
> add.ignoreErrors::
> Tells 'git add' to continue adding files when some files cannot be
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index 54bf6b2..95ea481 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -936,7 +936,14 @@ static const char *unpack(int err_fd)
> &shallow,
> WRITE_SHALLOW_NO_CUT |
> WRITE_SHALLOW_REWRITE);
> - commit_lock_file(&shallow_lock);
> + if (*alternate_shallow_file == '\0') {
> + unlink_or_warn(git_path("shallow"));
> + rollback_lock_file(&shallow_lock);
> + } else {
> + if (!is_repository_shallow() && cannot_be_shallow)
> + die("not allowed to turn this repository into a shallow one");
> + commit_lock_file(&shallow_lock);
> + }
> }
> return NULL;
> }
> diff --git a/cache.h b/cache.h
> index 7f17228..3a52b08 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -592,6 +592,7 @@ extern int fsync_object_files;
> extern int core_preload_index;
> extern int core_apply_sparse_checkout;
> extern int precomposed_unicode;
> +extern int cannot_be_shallow;
>
> /*
> * The character that begins a commented line in user-editable file
> diff --git a/config.c b/config.c
> index d04e815..31f5a57 100644
> --- a/config.c
> +++ b/config.c
> @@ -784,6 +784,11 @@ static int git_default_core_config(const char
> *var, const char *value)
> return 0;
> }
>
> + if (!strcmp(var, "core.noshallow")) {
> + cannot_be_shallow = git_config_bool(var, value);
> + return 0;
> + }
> +
> /* Add other config variables here and to Documentation/config.txt.
> */
> return 0;
> }
> diff --git a/environment.c b/environment.c
> index 0cb67b2..14c8005 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -61,6 +61,7 @@ int merge_log_config = -1;
> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition()
> */
> struct startup_info *startup_info;
> unsigned long pack_size_limit_cfg;
> +int cannot_be_shallow;
>
> /*
> * The character that begins a commented line in user-editable file
> diff --git a/fetch-pack.c b/fetch-pack.c
> index f337526..40e7aa2 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -960,7 +960,14 @@ struct ref *fetch_pack(struct fetch_pack_args
> *args,
> WRITE_SHALLOW_NO_CUT |
> WRITE_SHALLOW_REWRITE);
> }
> - commit_lock_file(&shallow_lock);
> + if (*alternate_shallow_file == '\0') {
> + unlink_or_warn(git_path("shallow"));
> + rollback_lock_file(&shallow_lock);
> + } else {
> + if (!is_repository_shallow() && cannot_be_shallow)
> + die("not allowed to turn this repository into a shallow one");
> + commit_lock_file(&shallow_lock);
> + }
> }
> }
>
> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh
> index 6ea6347..b7f89b1 100755
> --- a/t/t5536-fetch-shallow.sh
> +++ b/t/t5536-fetch-shallow.sh
> @@ -102,6 +102,15 @@ EOF
>
> '
>
> +test_expect_success 'core.noshallow' '
> + git init clean &&
> + (
> + cd clean &&
> + git config core.noshallow true &&
> + test_must_fail git fetch ../shallow/.git
> + )
> +'
> +
> if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
> say 'skipping remaining tests, git built without http support'
> test_done
> diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh
> index 8bea496..0edd51f 100755
> --- a/t/t5537-push-shallow.sh
> +++ b/t/t5537-push-shallow.sh
> @@ -108,6 +108,12 @@ EOF
> )
> '
>
> +test_expect_success 'core.noshallow' '
> + git init clean &&
> + git --git-dir=clean/.git config core.noshallow true &&
> + test_must_fail git --git-dir=shallow/.git push clean
> master:refs/remotes/shallow/master
> +'
> +
> if test -n "$NO_CURL" -o -z "$GIT_TEST_HTTPD"; then
> say 'skipping remaining tests, git built without http support'
> test_done
> --
> 1.8.2.83.gc99314b
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date:
> 07/19/13
>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
` (15 preceding siblings ...)
2013-07-20 9:58 ` [PATCH v2 16/16] clone: use git protocol for cloning shallow repo locally Nguyễn Thái Ngọc Duy
@ 2013-07-22 23:41 ` Philip Oakley
2013-07-23 1:20 ` Duy Nguyen
16 siblings, 1 reply; 44+ messages in thread
From: Philip Oakley @ 2013-07-22 23:41 UTC (permalink / raw)
To: Nguyễn Thái Ngọc Duy, git; +Cc: Junio C Hamano
From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
Subject: [PATCH v2 00/16] First class shallow clone
It's nice to see that shallow can be a first class clone.
Thinking outside the box, does this infrastructure offer the opportunity
to maybe add a date based depth option that would establish the shallow
watermark based on date rather than count. (e.g. the "deepen" SP depth
could have an alternate with a leading 'T' to indicate a time limit
ratherv than revision count - I'm expecting such a format would be an
error for existing servers).
My other thought was this style of cut limit list may also allow a big
file limit to do a similar process of listing objects (e.g. blobs) that
are size-shallow in the repo, though it maybe a long list on some repos,
or with a small size limit.
Philip
> v2 includes:
>
> - fix Junio comments, especially the one that may lead to incomplete
> commit islands.
> - fix send-pack setting up temporary shallow file, but never passes
> it to index-pack/unpack-objects (also fix the tests to catch this)
> - support smart http
> - add core.noshallow for repos that wish to be always complete
> - fix locally cloning a shallow repository
> - make upload-pack pass --shallow-file to pack-objects in order to
> remove duplicate object counting code just for shallow case.
>
> Nguyễn Thái Ngọc Duy (16):
> send-pack: forbid pushing from a shallow repository
> {receive,upload}-pack: advertise shallow graft information
> connect.c: teach get_remote_heads to parse "shallow" lines
> Move setup_alternate_shallow and write_shallow_commits to shallow.c
> fetch-pack: support fetching from a shallow repository
> {send,receive}-pack: support pushing from a shallow clone
> send-pack: support pushing to a shallow clone
> upload-pack: let pack-objects do the object counting in shallow case
> pack-protocol.txt: a bit about smart http
> Add document for command arguments for supporting smart http
> {fetch,upload}-pack: support fetching from a shallow clone via smart
> http
> receive-pack: support pushing to a shallow clone via http
> send-pack: support pushing from a shallow clone via http
> git-clone.txt: remove shallow clone limitations
> config: add core.noshallow to prevent turning a repo into a shallow
> one
> clone: use git protocol for cloning shallow repo locally
>
> Documentation/config.txt | 5 +
> Documentation/git-clone.txt | 7 +-
> Documentation/git-fetch-pack.txt | 11 +-
> Documentation/git-receive-pack.txt | 16 ++-
> Documentation/git-send-pack.txt | 9 +-
> Documentation/git-upload-pack.txt | 13 ++-
> Documentation/technical/pack-protocol.txt | 76 ++++++++++++-
> builtin/clone.c | 14 ++-
> builtin/fetch-pack.c | 6 +-
> builtin/receive-pack.c | 76 +++++++++++--
> builtin/send-pack.c | 7 +-
> cache.h | 4 +-
> commit.h | 27 +++++
> config.c | 5 +
> connect.c | 12 +-
> environment.c | 1 +
> fetch-pack.c | 90 ++++++---------
> fetch-pack.h | 1 +
> remote-curl.c | 4 +-
> send-pack.c | 57 +++++++++-
> send-pack.h | 4 +-
> shallow.c | 147
> +++++++++++++++++++++++++
> t/t5530-upload-pack-error.sh | 3 -
> t/t5536-fetch-shallow.sh (new +x) | 141
> ++++++++++++++++++++++++
> t/t5537-push-shallow.sh (new +x) | 176
> ++++++++++++++++++++++++++++++
> t/t5601-clone.sh | 7 ++
> transport.c | 14 ++-
> upload-pack.c | 132 ++++++----------------
> 28 files changed, 858 insertions(+), 207 deletions(-)
> create mode 100755 t/t5536-fetch-shallow.sh
> create mode 100755 t/t5537-push-shallow.sh
>
> --
> 1.8.2.83.gc99314b
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date:
> 07/19/13
>
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-22 23:41 ` [PATCH v2 00/16] First class shallow clone Philip Oakley
@ 2013-07-23 1:20 ` Duy Nguyen
2013-07-23 4:08 ` Junio C Hamano
2013-07-23 22:33 ` Philip Oakley
0 siblings, 2 replies; 44+ messages in thread
From: Duy Nguyen @ 2013-07-23 1:20 UTC (permalink / raw)
To: Philip Oakley; +Cc: Git Mailing List, Junio C Hamano
On Tue, Jul 23, 2013 at 6:41 AM, Philip Oakley <philipoakley@iee.org> wrote:
> From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
> Subject: [PATCH v2 00/16] First class shallow clone
>
> It's nice to see that shallow can be a first class clone.
>
> Thinking outside the box, does this infrastructure offer the opportunity to
> maybe add a date based depth option that would establish the shallow
> watermark based on date rather than count. (e.g. the "deepen" SP depth could
I've been carefully avoiding the deepen issues because, as you see,
it's complicated. But no, this series does not enable or disable new
deeepen mechanisms. They can always be added as protocol extensions.
Still thinking if it's worth exposing a (restricted form of) rev-list
to the protocol..
> have an alternate with a leading 'T' to indicate a time limit ratherv than
> revision count - I'm expecting such a format would be an error for existing
> servers).
>
> My other thought was this style of cut limit list may also allow a big file
> limit to do a similar process of listing objects (e.g. blobs) that are
> size-shallow in the repo, though it maybe a long list on some repos, or with
> a small size limit.
This one, on the other hand, changes the "shape" of the repo (now with
holes) and might need to go through the same process we do with this
series. Maybe we should prepare for it now. Do you have a use case for
size-based filtering? What can we do with a repo with some arbitrary
blobs missing? Another form of this is narrow clone, where we cut by
paths, not by blob size. Narrow clone sounds more useful to me because
it's easier to control what we leave out.
--
Duy
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one
2013-07-22 19:23 ` Philip Oakley
@ 2013-07-23 1:28 ` Duy Nguyen
2013-07-23 4:06 ` Junio C Hamano
2013-07-23 19:44 ` Philip Oakley
0 siblings, 2 replies; 44+ messages in thread
From: Duy Nguyen @ 2013-07-23 1:28 UTC (permalink / raw)
To: Philip Oakley; +Cc: Git Mailing List, Junio C Hamano
On Tue, Jul 23, 2013 at 2:23 AM, Philip Oakley <philipoakley@iee.org> wrote:
> From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
> Subject: [PATCH v2 15/16] config: add core.noshallow to prevent turning a
> repo into a shallow one
>
> Surely this should be the default now that it is possible to corrupt a
> golden repo by pushing/fetching a shallow repository to it and it then
> becomes shallow, and all the followers become shallow as well, with
> consequent problems (IIUC) [PATCH v2 05/16].
>
> It would be just as easy to change the config to core.allowshallow which
> then must be enabled by the user, and can be mentioned in the shallow clone
> option's documentation.
Clarification, it's not really "corrupt". If you have full history
from a ref "A", fetching from another shallow clone does not touch the
history of ref A at all (that is if you do _not_ specify --depth). It
may add a a shallow ref B, which is the reason the whole repo becomes
shallow. The same goes for push. This is not implemented, but I'm
thinking of adding "clean .git/shallow" to git repack -ad. Then if you
delete ref B and repack -ad, the repo could become full again.
But yeah, maybe defaulting to no shallow is better. Will do so in the
reroll unless someone objects.
--
Duy
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository
2013-07-22 19:10 ` Philip Oakley
@ 2013-07-23 2:06 ` Duy Nguyen
2013-07-23 14:39 ` Duy Nguyen
0 siblings, 1 reply; 44+ messages in thread
From: Duy Nguyen @ 2013-07-23 2:06 UTC (permalink / raw)
To: Philip Oakley; +Cc: Git Mailing List, Junio C Hamano
On Tue, Jul 23, 2013 at 2:10 AM, Philip Oakley <philipoakley@iee.org> wrote:
>> And a full repository may automatically become shallow if
>> you fetch from a shallow repository.
>
>
> This "sounds" risky: Perhaps add a note about '[PATCH v2 15/16] config: add
> core.noshallow to prevent turning a repo into a shallow one' which will
> become available later in the series. The later core.noshallow may be a
> hostage to fortune.
core.noshallow is one way. Another is reject new shallow history (i.e.
no additions to .git/shallow) unless the user explicitly asks so
either via --depth or a new option --shallow. This does not mean that
fetching from a shallow clone always fails without either of those
options. The shallow clone may provide updates to an existing ref and
_not_ introduce new shallow grafts and that would be successful.
core.noshallow (or an opposite alternative) is probably the only way
for push.
--
Duy
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one
2013-07-23 1:28 ` Duy Nguyen
@ 2013-07-23 4:06 ` Junio C Hamano
2013-07-23 19:44 ` Philip Oakley
1 sibling, 0 replies; 44+ messages in thread
From: Junio C Hamano @ 2013-07-23 4:06 UTC (permalink / raw)
To: Duy Nguyen; +Cc: Philip Oakley, Git Mailing List
Duy Nguyen <pclouds@gmail.com> writes:
> But yeah, maybe defaulting to no shallow is better. Will do so in the
> reroll unless someone objects.
No objections from me ;-).
Thanks.
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-23 1:20 ` Duy Nguyen
@ 2013-07-23 4:08 ` Junio C Hamano
2013-07-23 5:01 ` Duy Nguyen
2013-07-23 22:33 ` Philip Oakley
1 sibling, 1 reply; 44+ messages in thread
From: Junio C Hamano @ 2013-07-23 4:08 UTC (permalink / raw)
To: Duy Nguyen; +Cc: Philip Oakley, Git Mailing List
Duy Nguyen <pclouds@gmail.com> writes:
> This one, on the other hand, changes the "shape" of the repo (now with
> holes) and might need to go through the same process we do with this
> series. Maybe we should prepare for it now. Do you have a use case for
> size-based filtering? What can we do with a repo with some arbitrary
> blobs missing? Another form of this is narrow clone, where we cut by
> paths, not by blob size. Narrow clone sounds more useful to me because
> it's easier to control what we leave out.
I was about to say "Hear, hear", but then stopped with a question to
myself: why are these "some people do not want them" paths in the
same repository in the first place?
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-23 4:08 ` Junio C Hamano
@ 2013-07-23 5:01 ` Duy Nguyen
0 siblings, 0 replies; 44+ messages in thread
From: Duy Nguyen @ 2013-07-23 5:01 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Philip Oakley, Git Mailing List
On Tue, Jul 23, 2013 at 11:08 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Duy Nguyen <pclouds@gmail.com> writes:
>
>> This one, on the other hand, changes the "shape" of the repo (now with
>> holes) and might need to go through the same process we do with this
>> series. Maybe we should prepare for it now. Do you have a use case for
>> size-based filtering? What can we do with a repo with some arbitrary
>> blobs missing? Another form of this is narrow clone, where we cut by
>> paths, not by blob size. Narrow clone sounds more useful to me because
>> it's easier to control what we leave out.
>
> I was about to say "Hear, hear", but then stopped with a question to
> myself: why are these "some people do not want them" paths in the
> same repository in the first place?
I think there are situations that splitting repos is not the best
choice but I can't think of any. There's one case though that such
"some people" exist: when they migrate from another version control to
git and do not want to change the directory layout (because it used to
work ok, because of the cost of updating toolchain...)
--
Duy
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository
2013-07-23 2:06 ` Duy Nguyen
@ 2013-07-23 14:39 ` Duy Nguyen
0 siblings, 0 replies; 44+ messages in thread
From: Duy Nguyen @ 2013-07-23 14:39 UTC (permalink / raw)
To: Philip Oakley; +Cc: Git Mailing List, Junio C Hamano
On Tue, Jul 23, 2013 at 9:06 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> Another is reject new shallow history (i.e.
> no additions to .git/shallow) unless the user explicitly asks so
> either via --depth or a new option --shallow. This does not mean that
> fetching from a shallow clone always fails without either of those
> options.
The above was at pack level when I wrote that, although I think, from
the user perspective, rejecting at ref level makes more sense. That is
if a fetch request returns one ref update with incremental updates and
one with new shallow history, instead of rejecting the whole request,
we reject the second ref update and accept the first one.
--
Duy
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one
2013-07-23 1:28 ` Duy Nguyen
2013-07-23 4:06 ` Junio C Hamano
@ 2013-07-23 19:44 ` Philip Oakley
1 sibling, 0 replies; 44+ messages in thread
From: Philip Oakley @ 2013-07-23 19:44 UTC (permalink / raw)
To: Duy Nguyen; +Cc: Git Mailing List, Junio C Hamano
From: "Duy Nguyen" <pclouds@gmail.com>
Sent: Tuesday, July 23, 2013 2:28 AM
> On Tue, Jul 23, 2013 at 2:23 AM, Philip Oakley <philipoakley@iee.org>
> wrote:
>> From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
>> Subject: [PATCH v2 15/16] config: add core.noshallow to prevent
>> turning a
>> repo into a shallow one
>>
>> Surely this should be the default now that it is possible to corrupt
>> a
>> golden repo by pushing/fetching a shallow repository to it and it
>> then
>> becomes shallow, and all the followers become shallow as well, with
>> consequent problems (IIUC) [PATCH v2 05/16].
>>
>> It would be just as easy to change the config to core.allowshallow
>> which
>> then must be enabled by the user, and can be mentioned in the shallow
>> clone
>> option's documentation.
>
> Clarification, it's not really "corrupt". If you have full history
> from a ref "A", fetching from another shallow clone does not touch the
> history of ref A at all
> (that is if you do _not_ specify --depth).
I hadn't appreciated this conditional.
> It
> may add a a shallow ref B, which is the reason the whole repo becomes
> shallow.
I had read the initial commit message (in 05/16) as implying that it was
possible to fool someone into pulling a shallow repo and that would make
their repo just as shallow (that's without a --depth argument to the
command). Had that been the case then it would have been possible to
loose some data from deep in the DAG. Glad to hear I was mistaken.
Perhaps if your comment above is included in the commit message to
ensure that clarification is there.
>The same goes for push. This is not implemented, but I'm
> thinking of adding "clean .git/shallow" to git repack -ad. Then if you
> delete ref B and repack -ad, the repo could become full again.
>
> But yeah, maybe defaulting to no shallow is better. Will do so in the
> reroll unless someone objects.
> --
> Duy
> --
Philip
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-23 1:20 ` Duy Nguyen
2013-07-23 4:08 ` Junio C Hamano
@ 2013-07-23 22:33 ` Philip Oakley
2013-07-24 1:57 ` Duy Nguyen
1 sibling, 1 reply; 44+ messages in thread
From: Philip Oakley @ 2013-07-23 22:33 UTC (permalink / raw)
To: Duy Nguyen; +Cc: Git Mailing List, Junio C Hamano
From: "Duy Nguyen" <pclouds@gmail.com>
Sent: Tuesday, July 23, 2013 2:20 AM
> On Tue, Jul 23, 2013 at 6:41 AM, Philip Oakley <philipoakley@iee.org>
> wrote:
>> From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
>> Subject: [PATCH v2 00/16] First class shallow clone
>>
>> It's nice to see that shallow can be a first class clone.
>>
>> Thinking outside the box, does this infrastructure offer the
>> opportunity to
>> maybe add a date based depth option that would establish the shallow
>> watermark based on date rather than count. (e.g. the "deepen" SP
>> depth could
>
> I've been carefully avoiding the deepen issues because, as you see,
> it's complicated. But no, this series does not enable or disable new
> deeepen mechanisms. They can always be added as protocol extensions.
> Still thinking if it's worth exposing a (restricted form of) rev-list
> to the protocol..
Interesting idea.
>
>> have an alternate with a leading 'T' to indicate a time limit ratherv
>> than
>> revision count - I'm expecting such a format would be an error for
>> existing
>> servers).
>>
>> My other thought was this style of cut limit list may also allow a
>> big file
>> limit to do a similar process of listing objects (e.g. blobs) that
>> are
>> size-shallow in the repo, though it maybe a long list on some repos,
>> or with
>> a small size limit.
>
> This one, on the other hand, changes the "shape" of the repo (now with
> holes) and might need to go through the same process we do with this
> series. Maybe we should prepare for it now. Do you have a use case for
> size-based filtering? What can we do with a repo with some arbitrary
> blobs missing? Another form of this is narrow clone, where we cut by
> paths, not by blob size. Narrow clone sounds more useful to me because
> it's easier to control what we leave out.
In some sense a project with a sub-module is a narrow clone, split at a
'commit' object. There have been comments on the git-user list about the
problem of accidental adding of large files which then make the repo's
foot print pretty large as one use case [Git is consuming very much
RAM]. The bigFileThreshold being one way of spotting such files as
separate objects, and 'trimming' them.
It doesn't feel right to 'track files and directories` as paths for
doing a narrow clone - it'd probably fall into the same trap as tracking
file renames. However if one tracks trees and blobs (as a list of sha1
values, possibly with their source path) then it should it should be
possible to allow work on the repo with those empty directories/files in
the same manner as is used for sub-modules, possibly with some form of
git-link file as an alternate marker.
The thought process is to map sub-module working onto the other object
types (blobs and trees). The user would be unable to edit the trimmed
files/directories anyway, so its sha1 value can't change, allowing it to
be included in the next commit in the branch series.
Philip
> --
> Duy
> --
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-23 22:33 ` Philip Oakley
@ 2013-07-24 1:57 ` Duy Nguyen
2013-07-24 7:38 ` Philip Oakley
2013-07-24 8:30 ` Piotr Krukowiecki
0 siblings, 2 replies; 44+ messages in thread
From: Duy Nguyen @ 2013-07-24 1:57 UTC (permalink / raw)
To: Philip Oakley; +Cc: Git Mailing List, Junio C Hamano
On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley <philipoakley@iee.org> wrote:
> In some sense a project with a sub-module is a narrow clone, split at a
> 'commit' object.
Yes, except narrow clone is more flexible. You have to decide the
split boundary at commit time for sub-module, while you decide the
same at clone time for narrow clone.
> There have been comments on the git-user list about the
> problem of accidental adding of large files which then make the repo's foot
> print pretty large as one use case [Git is consuming very much RAM]. The
> bigFileThreshold being one way of spotting such files as separate objects,
> and 'trimming' them.
I think rewriting history to remove those accidents is better than
working around it (the same for accidentally committing password). We
might be able to spot problems early, maybe warn user at commit time
that they have added an exceptionally large blob, maybe before push
time..
The "Git is consuming very much RAM" part is not right. We try to keep
memory usage under a limit regardless of the size of a blob. There may
be some cases we haven't fixed yet. Reports are welcome.
--
Duy
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-24 1:57 ` Duy Nguyen
@ 2013-07-24 7:38 ` Philip Oakley
2013-07-24 8:30 ` Piotr Krukowiecki
1 sibling, 0 replies; 44+ messages in thread
From: Philip Oakley @ 2013-07-24 7:38 UTC (permalink / raw)
To: Duy Nguyen; +Cc: Git Mailing List, Junio C Hamano
From: "Duy Nguyen" <pclouds@gmail.com>
Sent: Wednesday, July 24, 2013 2:57 AM
> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley <philipoakley@iee.org>
> wrote:
>> In some sense a project with a sub-module is a narrow clone, split at
>> a
>> 'commit' object.
>
> Yes, except narrow clone is more flexible. You have to decide the
> split boundary at commit time for sub-module, while you decide the
> same at clone time for narrow clone.
>
True. It was the thought experiment part I was referring to.
>> There have been comments on the git-user list about the
>> problem of accidental adding of large files which then make the
>> repo's foot
>> print pretty large as one use case [Git is consuming very much RAM].
>> The
>> bigFileThreshold being one way of spotting such files as separate
>> objects,
>> and 'trimming' them.
>
> I think rewriting history to remove those accidents is better than
> working around it (the same for accidentally committing password). We
> might be able to spot problems early, maybe warn user at commit time
> that they have added an exceptionally large blob, maybe before push
> time..
Again, it was a thought experiment which related to a recent git-user
list comment.
I'd expect a real use case could be a team where one member who is
preparing documentation adds a [large] video to his branch and others
then get a bit concerned when they try to track it / pull it as they
really don't want it yet. The guy may have many versions on the central
repo before a final rebase has a single compressed version. Colleagues
may want to review the text surrounding it but not pull the video
itself. (remembering 50 % of 'idiots' are twice as dumb as the
average... ;-)
>
> The "Git is consuming very much RAM" part is not right. We try to keep
> memory usage under a limit regardless of the size of a blob. There may
> be some cases we haven't fixed yet. Reports are welcome.
I think this was a Windows user, but reports do pop up every now and
again. Some times its disc pressure, or just perceived slowness (from
others)
> --
> Duy
Philip
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-24 1:57 ` Duy Nguyen
2013-07-24 7:38 ` Philip Oakley
@ 2013-07-24 8:30 ` Piotr Krukowiecki
2013-07-24 10:35 ` Duy Nguyen
1 sibling, 1 reply; 44+ messages in thread
From: Piotr Krukowiecki @ 2013-07-24 8:30 UTC (permalink / raw)
To: Duy Nguyen; +Cc: Philip Oakley, Git Mailing List, Junio C Hamano
(resending, as my phone mail client decided to send it in html, sorry
about that)
On Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley <philipoakley@iee.org> wrote:
>> There have been comments on the git-user list about the
>> problem of accidental adding of large files which then make the repo's foot
>> print pretty large as one use case [Git is consuming very much RAM]. The
>> bigFileThreshold being one way of spotting such files as separate objects,
>> and 'trimming' them.
>
> I think rewriting history to remove those accidents is better than
> working around it (the same for accidentally committing password). We
> might be able to spot problems early, maybe warn user at commit time
> that they have added an exceptionally large blob, maybe before push
> time..
I can imagine a situation where large files were part of the project
at some point in history (they were required to build/use it) and
later were removed because build/project has changed.
It would be useful to have the history for log/blame/etc even if you
could not build/use old versions. A warning when checking
out/branching such incomplete tree would be needed.
--
Piotr Krukowiecki
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-24 8:30 ` Piotr Krukowiecki
@ 2013-07-24 10:35 ` Duy Nguyen
2013-07-24 16:50 ` Piotr Krukowiecki
0 siblings, 1 reply; 44+ messages in thread
From: Duy Nguyen @ 2013-07-24 10:35 UTC (permalink / raw)
To: Piotr Krukowiecki; +Cc: Philip Oakley, Git Mailing List, Junio C Hamano
On Wed, Jul 24, 2013 at 3:30 PM, Piotr Krukowiecki
<piotr.krukowiecki@gmail.com> wrote:
> (resending, as my phone mail client decided to send it in html, sorry
> about that)
>
> On Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley <philipoakley@iee.org> wrote:
>>> There have been comments on the git-user list about the
>>> problem of accidental adding of large files which then make the repo's foot
>>> print pretty large as one use case [Git is consuming very much RAM]. The
>>> bigFileThreshold being one way of spotting such files as separate objects,
>>> and 'trimming' them.
>>
>> I think rewriting history to remove those accidents is better than
>> working around it (the same for accidentally committing password). We
>> might be able to spot problems early, maybe warn user at commit time
>> that they have added an exceptionally large blob, maybe before push
>> time..
>
> I can imagine a situation where large files were part of the project
> at some point in history (they were required to build/use it) and
> later were removed because build/project has changed.
>
> It would be useful to have the history for log/blame/etc even if you
> could not build/use old versions. A warning when checking
> out/branching such incomplete tree would be needed.
That's what shallow clone is for. You fetch the latest (not including
old large blobs) and work on top. For archaeology, make a full clone.
Or do you mean log/blame/etc other paths that don't touch big blobs,
and the clone is still incomplete?
--
Duy
^ permalink raw reply [flat|nested] 44+ messages in thread
* Re: [PATCH v2 00/16] First class shallow clone
2013-07-24 10:35 ` Duy Nguyen
@ 2013-07-24 16:50 ` Piotr Krukowiecki
0 siblings, 0 replies; 44+ messages in thread
From: Piotr Krukowiecki @ 2013-07-24 16:50 UTC (permalink / raw)
To: Duy Nguyen; +Cc: Philip Oakley, Git Mailing List, Junio C Hamano
Duy Nguyen <pclouds@gmail.com> napisał:
>On Wed, Jul 24, 2013 at 3:30 PM, Piotr Krukowiecki
><piotr.krukowiecki@gmail.com> wrote:
>> (resending, as my phone mail client decided to send it in html, sorry
>> about that)
>>
>> On Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen <pclouds@gmail.com>
>wrote:
>>> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley
><philipoakley@iee.org> wrote:
>>>> There have been comments on the git-user list about the
>>>> problem of accidental adding of large files which then make the
>repo's foot
>>>> print pretty large as one use case [Git is consuming very much
>RAM]. The
>>>> bigFileThreshold being one way of spotting such files as separate
>objects,
>>>> and 'trimming' them.
>>>
>>> I think rewriting history to remove those accidents is better than
>>> working around it (the same for accidentally committing password).
>We
>>> might be able to spot problems early, maybe warn user at commit time
>>> that they have added an exceptionally large blob, maybe before push
>>> time..
>>
>> I can imagine a situation where large files were part of the project
>> at some point in history (they were required to build/use it) and
>> later were removed because build/project has changed.
>>
>> It would be useful to have the history for log/blame/etc even if you
>> could not build/use old versions. A warning when checking
>> out/branching such incomplete tree would be needed.
>
>That's what shallow clone is for. You fetch the latest (not including
>old large blobs) and work on top. For archaeology, make a full clone.
>Or do you mean log/blame/etc other paths that don't touch big blobs,
>and the clone is still incomplete?
Yes, for example if large files were removed recently the last-n-commits-shallow would be useless from blame/log POV.
^ permalink raw reply [flat|nested] 44+ messages in thread
end of thread, other threads:[~2013-07-24 16:51 UTC | newest]
Thread overview: 44+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-07-17 12:47 [PATCH 0/7] First class shallow clone Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 1/7] transport.h: remove send_pack prototype, already defined in send-pack.h Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 2/7] {receive,upload}-pack: advertise shallow graft information Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 3/7] connect.c: teach get_remote_heads to parse "shallow" lines Nguyễn Thái Ngọc Duy
2013-07-20 3:27 ` Junio C Hamano
2013-07-17 12:47 ` [PATCH 4/7] Move setup_alternate_shallow and write_shallow_commits to shallow.c Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 5/7] fetch-pack: support fetching from a shallow repository Nguyễn Thái Ngọc Duy
2013-07-20 3:17 ` Junio C Hamano
2013-07-17 12:47 ` [PATCH 6/7] {send,receive}-pack: support pushing from a shallow clone Nguyễn Thái Ngọc Duy
2013-07-17 12:47 ` [PATCH 7/7] send-pack: support pushing to " Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 00/16] First class " Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 01/16] send-pack: forbid pushing from a shallow repository Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 02/16] {receive,upload}-pack: advertise shallow graft information Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 03/16] connect.c: teach get_remote_heads to parse "shallow" lines Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 04/16] Move setup_alternate_shallow and write_shallow_commits to shallow.c Nguyễn Thái Ngọc Duy
2013-07-20 9:57 ` [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository Nguyễn Thái Ngọc Duy
2013-07-22 19:10 ` Philip Oakley
2013-07-23 2:06 ` Duy Nguyen
2013-07-23 14:39 ` Duy Nguyen
2013-07-20 9:58 ` [PATCH v2 06/16] {send,receive}-pack: support pushing from a shallow clone Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 07/16] send-pack: support pushing to " Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 08/16] upload-pack: let pack-objects do the object counting in shallow case Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 09/16] pack-protocol.txt: a bit about smart http Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 10/16] Add document for command arguments for supporting " Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 11/16] {fetch,upload}-pack: support fetching from a shallow clone via " Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 12/16] receive-pack: support pushing to a shallow clone via http Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 13/16] send-pack: support pushing from " Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 14/16] git-clone.txt: remove shallow clone limitations Nguyễn Thái Ngọc Duy
2013-07-20 9:58 ` [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one Nguyễn Thái Ngọc Duy
2013-07-22 19:23 ` Philip Oakley
2013-07-23 1:28 ` Duy Nguyen
2013-07-23 4:06 ` Junio C Hamano
2013-07-23 19:44 ` Philip Oakley
2013-07-20 9:58 ` [PATCH v2 16/16] clone: use git protocol for cloning shallow repo locally Nguyễn Thái Ngọc Duy
2013-07-22 23:41 ` [PATCH v2 00/16] First class shallow clone Philip Oakley
2013-07-23 1:20 ` Duy Nguyen
2013-07-23 4:08 ` Junio C Hamano
2013-07-23 5:01 ` Duy Nguyen
2013-07-23 22:33 ` Philip Oakley
2013-07-24 1:57 ` Duy Nguyen
2013-07-24 7:38 ` Philip Oakley
2013-07-24 8:30 ` Piotr Krukowiecki
2013-07-24 10:35 ` Duy Nguyen
2013-07-24 16:50 ` Piotr Krukowiecki
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).