From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jonathan Nieder Subject: [PATCH 3/4] fast-import: let importers retrieve blobs Date: Fri, 19 Nov 2010 03:47:38 -0600 Message-ID: <20101119094738.GD19061@burratino> References: <1287147256-9457-1-git-send-email-david.barr@cordelta.com> <1287147256-9457-2-git-send-email-david.barr@cordelta.com> <20101119093530.GA19061@burratino> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Git Mailing List , Ramkumar Ramachandra , Sverre Rabbelier , "Shawn O. Pearce" , vcs-fast-import-devs@lists.launchpad.net To: David Barr X-From: git-owner@vger.kernel.org Fri Nov 19 10:48:35 2010 Return-path: Envelope-to: gcvg-git-2@lo.gmane.org Received: from vger.kernel.org ([209.132.180.67]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1PJNa5-0004Fs-41 for gcvg-git-2@lo.gmane.org; Fri, 19 Nov 2010 10:48:33 +0100 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752114Ab0KSJs2 (ORCPT ); Fri, 19 Nov 2010 04:48:28 -0500 Received: from mail-yx0-f174.google.com ([209.85.213.174]:40833 "EHLO mail-yx0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751559Ab0KSJsZ (ORCPT ); Fri, 19 Nov 2010 04:48:25 -0500 Received: by yxf34 with SMTP id 34so2533239yxf.19 for ; Fri, 19 Nov 2010 01:48:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:date:from:to:cc:subject :message-id:references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=QoDY4qgA+qE6rrVFb/QKjZnty/f+f3+VRW3rkBxVHd0=; b=g682KzecNDAErBbNSnmQ5fVxozAtJKM7nVXqcukzDQhlpsPA6wa0t6K7QL4UTTn1lU xnVO3lq44GfXMwrWsKQMKYfrDZDobQiThDI+ZsKSA2FStTHyC8VhJv9DM/Wt7rqQEn0P 5ukKYTAbsZQX6ZWA4542skDCtxD0DCu2Won7w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=vBkzNMhbORdVwyTcCGtZi7PKupSCgl3eKu7RulSeWCebV8IgbEuXmI9bL6+6Nu5XiS mbrZEM2MRMVORgaUn8QuIbFw0J58EBl9UYFj8ZUd30HUg51DgQKwB3nsAdZH/SjFAeXd 3sW/38nOBhYEnV4gZNqnQ37zO1+OAdtUwLIHY= Received: by 10.150.148.19 with SMTP id v19mr3177012ybd.342.1290160104855; Fri, 19 Nov 2010 01:48:24 -0800 (PST) Received: from burratino ([68.255.106.176]) by mx.google.com with ESMTPS id p20sm2237781ybe.17.2010.11.19.01.48.22 (version=SSLv3 cipher=RC4-MD5); Fri, 19 Nov 2010 01:48:24 -0800 (PST) Content-Disposition: inline In-Reply-To: <20101119093530.GA19061@burratino> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Archived-At: From: David Barr New objects written by fast-import are not available immediately. Until a checkpoint has been started and finishes writing the pack index, any new blobs will not be accessible using standard git tools. So introduce a new way to access them: a "cat-blob" command in the command stream requests for fast-import to print a blob to stdout or a file descriptor specified by the argument to --cat-blob-fd. The value for cat-blob-fd cannot be specified in the stream because that would be a layering violation: the decision of where to direct a stream has to be made when fast-import is started anyway, so we might as well make the stream format is independent of that detail. Output uses the same format as "git cat-file --batch". Thanks to Sverre Rabbelier and Sam Vilain for guidance in designing the protocol. Based-on-patch-by: Jonathan Nieder Signed-off-by: David Barr Acked-by: Ramkumar Ramachandra Signed-off-by: Jonathan Nieder --- If you use this, you might want to do something like: blob mark :1 data <:: + Specify the file descriptor that will be written to + when the `cat-blob` command is encountered in the stream. + The default behaviour is to write to `stdout`. + --export-pack-edges=:: After creating a packfile, print a line of data to listing the filename of the packfile and the last @@ -320,6 +325,11 @@ and control the current import process. More detailed discussion standard output. This command is optional and is not needed to perform an import. +`cat-blob`:: + Causes fast-import to print a blob in 'cat-file --batch' + format to the file descriptor set with `--cat-blob-fd` or + `stdout` if unspecified. + `feature`:: Require that fast-import supports the specified feature, or abort if it does not. @@ -872,6 +882,29 @@ Placing a `progress` command immediately after a `checkpoint` will inform the reader when the `checkpoint` has been completed and it can safely access the refs that fast-import updated. +`cat-blob` +~~~~~~~~~~ +Causes fast-import to print a blob to a file descriptor previously +arranged with the `--cat-blob-fd` argument. The command otherwise +has no impact on the current import; its main purpose is to +retrieve blobs that may be in fast-import's memory but not +accessible from the target repository. + +.... + 'cat-blob' SP LF +.... + +The `` can be either a mark reference (`:`) +set previously or a full 40-byte SHA-1 of a Git blob, preexisting or +ready to be written. + +output uses the same format as `git cat-file --batch`: + +==== + SP 'blob' SP LF + LF +==== + `feature` ~~~~~~~~~ Require that fast-import supports the specified feature, or abort if @@ -898,6 +931,13 @@ import-marks:: second, an --import-marks= command-line option overrides any "feature import-marks" command in the stream. +cat-blob:: + Ignored. Versions of fast-import not supporting the + "cat-blob" command will exit with a message indicating so. + This lets the import error out early with a clear message, + rather than wasting time on the early part of an import + before the unsupported command is detected. + `option` ~~~~~~~~ Processes the specified option so that git fast-import behaves in a @@ -923,6 +963,7 @@ not be passed as option: * date-format * import-marks * export-marks +* cat-blob-fd * force Crash Reports diff --git a/fast-import.c b/fast-import.c index 959afef..88547c6 100644 --- a/fast-import.c +++ b/fast-import.c @@ -55,6 +55,8 @@ Format of STDIN stream: ('from' sp committish lf)? lf?; + cat_blob ::= 'cat-blob' sp (hexsha1 | idnum) lf; + checkpoint ::= 'checkpoint' lf lf?; @@ -361,6 +363,9 @@ static uintmax_t next_mark; static struct strbuf new_data = STRBUF_INIT; static int seen_data_command; +/* Where to write output of cat-blob commands */ +static int cat_blob_fd = STDOUT_FILENO; + static void parse_argv(void); static void write_branch_report(FILE *rpt, struct branch *b) @@ -2689,6 +2694,79 @@ static void parse_reset_branch(void) unread_command_buf = 1; } +static void cat_blob_write(const char *buf, unsigned long size) +{ + if (write_in_full(cat_blob_fd, buf, size) != size) + die_errno("Write to frontend failed"); +} + +static void cat_blob(struct object_entry *oe, unsigned char sha1[20]) +{ + struct strbuf line = STRBUF_INIT; + unsigned long size; + enum object_type type = 0; + char *buf; + + if (!oe || oe->pack_id == MAX_PACK_ID) { + buf = read_sha1_file(sha1, &type, &size); + } else { + type = oe->type; + buf = gfi_unpack_entry(oe, &size); + } + + /* + * Output based on batch_one_object() from cat-file.c. + */ + if (type <= 0) { + strbuf_reset(&line); + strbuf_addf(&line, "%s missing\n", sha1_to_hex(sha1)); + cat_blob_write(line.buf, line.len); + free(buf); + return; + } + if (!buf) + die("Can't read object %s", sha1_to_hex(sha1)); + if (type != OBJ_BLOB) + die("Object %s is a %s but a blob was expected.", + sha1_to_hex(sha1), typename(type)); + strbuf_reset(&line); + strbuf_addf(&line, "%s %s %lu\n", sha1_to_hex(sha1), + typename(type), size); + cat_blob_write(line.buf, line.len); + cat_blob_write(buf, size); + cat_blob_write("\n", 1); + free(buf); +} + +static void parse_cat_blob(void) +{ + const char *p; + struct object_entry *oe = oe; + unsigned char sha1[20]; + + /* cat-blob SP LF */ + p = command_buf.buf + strlen("cat-blob "); + if (*p == ':') { + char *x; + oe = find_mark(strtoumax(p + 1, &x, 10)); + if (x == p + 1) + die("Invalid mark: %s", command_buf.buf); + if (!oe) + die("Unknown mark: %s", command_buf.buf); + if (*x) + die("Garbage after mark: %s", command_buf.buf); + hashcpy(sha1, oe->idx.sha1); + } else { + if (get_sha1_hex(p, sha1)) + die("Invalid SHA1: %s", command_buf.buf); + if (p[40]) + die("Garbage after SHA1: %s", command_buf.buf); + oe = find_object(sha1); + } + + cat_blob(oe, sha1); +} + static void parse_checkpoint(void) { if (object_count) { @@ -2771,6 +2849,14 @@ static void option_export_marks(const char *marks) export_marks_file = make_fast_import_path(marks); } +static void option_cat_blob_fd(const char *fd) +{ + unsigned long n = ulong_arg("--cat-blob-fd", fd); + if (n > (unsigned long) INT_MAX) + die("--cat-blob-fd cannot exceed %d", INT_MAX); + cat_blob_fd = (int) n; +} + static void option_export_pack_edges(const char *edges) { if (pack_edges) @@ -2824,6 +2910,8 @@ static int parse_one_feature(const char *feature, int from_stream) option_import_marks(feature + 13, from_stream); } else if (!prefixcmp(feature, "export-marks=")) { option_export_marks(feature + 13); + } else if (!strcmp(feature, "cat-blob")) { + ; /* Don't die - this feature is supported */ } else if (!prefixcmp(feature, "relative-marks")) { relative_marks_paths = 1; } else if (!prefixcmp(feature, "no-relative-marks")) { @@ -2918,6 +3006,11 @@ static void parse_argv(void) if (parse_one_feature(a + 2, 0)) continue; + if (!prefixcmp(a + 2, "cat-blob-fd=")) { + option_cat_blob_fd(a + 2 + strlen("cat-blob-fd=")); + continue; + } + die("unknown option %s", a); } if (i != global_argc) @@ -2969,6 +3062,8 @@ int main(int argc, const char **argv) parse_new_tag(); else if (!prefixcmp(command_buf.buf, "reset ")) parse_reset_branch(); + else if (!prefixcmp(command_buf.buf, "cat-blob ")) + parse_cat_blob(); else if (!strcmp("checkpoint", command_buf.buf)) parse_checkpoint(); else if (!prefixcmp(command_buf.buf, "progress ")) diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh index 2c27da6..3e2741b 100755 --- a/t/t9300-fast-import.sh +++ b/t/t9300-fast-import.sh @@ -23,11 +23,18 @@ file5_data='an inline file. file6_data='#!/bin/sh echo "$@"' +>empty + ### ### series A ### test_tick + +test_expect_success 'empty stream succeeds' ' + git fast-import input <expect <<-EOF && + ${blob} blob 11 + yes it can + + EOF + echo "cat-blob $blob" | + git fast-import --cat-blob-fd=6 6>actual && + test_cmp expect actual +' + +test_expect_success 'R: in-stream cat-blob-fd not respected' ' + echo hello >greeting && + blob=$(git hash-object -w greeting) && + cat >expect <<-EOF && + ${blob} blob 6 + hello + + EOF + git fast-import --cat-blob-fd=3 3>actual.3 >actual.1 <<-EOF && + cat-blob $blob + EOF + test_cmp expect actual.3 && + test_cmp empty actual.1 && + git fast-import 3>actual.3 >actual.1 <<-EOF && + option cat-blob-fd=3 + cat-blob $blob + EOF + test_cmp empty actual.3 && + test_cmp expect actual.1 +' + +test_expect_success 'R: print new blob' ' + blob=$(echo "yep yep yep" | git hash-object --stdin) && + cat >expect <<-EOF && + ${blob} blob 12 + yep yep yep + + EOF + git fast-import --cat-blob-fd=6 6>actual <<-\EOF && + blob + mark :1 + data <expect <<-EOF && + ${blob} blob 25 + a new blob named by sha1 + + EOF + git fast-import --cat-blob-fd=6 6>actual <<-EOF && + blob + data <big && + for i in 1 2 3 + do + cat big big big big >bigger && + cat bigger bigger bigger bigger >big || + exit + done + ) +' + +test_expect_success 'R: print two blobs to stdout' ' + blob1=$(git hash-object big) && + blob1_len=$(wc -c expect && + { + cat <<-\END_PART1 && + blob + mark :1 + data <actual && + test_cmp expect actual +' + +test_expect_success 'setup: have pipes?' ' + rm -f frob && + if mkfifo frob + then + test_set_prereq PIPE + fi +' + +test_expect_success PIPE 'R: copy using cat-file' ' + expect_id=$(git hash-object big) && + expect_len=$(wc -c expect.response && + + rm -f blobs && + cat >frontend <<-\FRONTEND_END && + #!/bin/sh + cat <response && + dd if=/dev/stdin of=blob bs=$size count=1 <&3 && + read newline <&3 && + + cat < $GIT_COMMITTER_DATE + data <blobs + ) && + git show copied:file3 >actual && + test_cmp expect.response response && + test_cmp big actual +' + cat >input << EOF option git quiet blob @@ -1509,8 +1700,6 @@ hi EOF -touch empty - test_expect_success 'R: quiet option results in no stats being output' ' cat input | git fast-import 2> output && test_cmp empty output -- 1.7.2.3