[PATCH/RFC] Teach fast-import to import subtrees named by tree id

[PATCH/RFC] Teach fast-import to import subtrees named by tree id

[Jonathan Nieder]

To simulate the svn cp command, it would be very useful to be
replace an arbitrary file in the current revision by an
arbitrary directory from a previous one.  Modify the filemodify
command to allow that:

 M 040000 <tree id> pathname

This would be most useful in combination with a facility to
print the commit ids for new revisions as they are written.

Cc: Shawn O. Pearce <spearce@spearce.org>
Cc: Sverre Rabbelier <srabbelier@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
I actually thought fast-import already did this until David
mentioned that no, it does not.  Well, live and learn.

This and Sverre’s --print-marks command would allow svn-fe
to be simplified a great deal.

I was not sure whether to add a "feature" specification for
this, so I’ll try that as a separate patch.

Thoughts?
Jonathan

 Documentation/git-fast-import.txt |    8 ++++-
 fast-import.c                     |   24 ++++++++++------
 t/t9300-fast-import.sh            |   54 +++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+), 11 deletions(-)

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 19082b0..f4d9aeb 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -482,9 +482,11 @@ External data format::
 	'M' SP <mode> SP <dataref> SP <path> LF
 ....
 +
-Here `<dataref>` can be either a mark reference (`:<idnum>`)
+Here usually `<dataref>` must be either a mark reference (`:<idnum>`)
 set by a prior `blob` command, or a full 40-byte SHA-1 of an
-existing Git blob object.
+existing Git blob object.  If `<mode>` is `040000`` then
+`<dataref>` must be the full 40-byte SHA-1 of an existing
+Git tree object or a mark reference set with `--import-marks`.
 
 Inline data format::
 	The data content for the file has not been supplied yet.
@@ -509,6 +511,8 @@ in octal.  Git only supports the following modes:
 * `160000`: A gitlink, SHA-1 of the object refers to a commit in
   another repository. Git links can only be specified by SHA or through
   a commit mark. They are used to implement submodules.
+* `040000`: A subdirectory.  Subdirectories can only be specified by
+  SHA or through a tree mark set with `--import-marks`.
 
 In both formats `<path>` is the complete path of the file to be added
 (if not already existing) or modified (if already existing).
diff --git a/fast-import.c b/fast-import.c
index 1e5d66e..ad6843a 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -2131,6 +2131,7 @@ static void file_change_m(struct branch *b)
 	case S_IFREG | 0644:
 	case S_IFREG | 0755:
 	case S_IFLNK:
+	case S_IFDIR:
 	case S_IFGITLINK:
 		/* ok */
 		break;
@@ -2176,23 +2177,28 @@ static void file_change_m(struct branch *b)
 		 * another repository.
 		 */
 	} else if (inline_data) {
+		if (S_ISDIR(mode))
+			die("Directories cannot be specified 'inline': %s",
+				command_buf.buf);
 		if (p != uq.buf) {
 			strbuf_addstr(&uq, p);
 			p = uq.buf;
 		}
 		read_next_command();
 		parse_and_store_blob(&last_blob, sha1, 0);
-	} else if (oe) {
-		if (oe->type != OBJ_BLOB)
-			die("Not a blob (actually a %s): %s",
-				typename(oe->type), command_buf.buf);
 	} else {
-		enum object_type type = sha1_object_info(sha1, NULL);
+		enum object_type expected = S_ISDIR(mode) ?
+						OBJ_TREE: OBJ_BLOB;
+		enum object_type type = oe ? oe->type :
+					sha1_object_info(sha1, NULL);
 		if (type < 0)
-			die("Blob not found: %s", command_buf.buf);
-		if (type != OBJ_BLOB)
-			die("Not a blob (actually a %s): %s",
-			    typename(type), command_buf.buf);
+			die("%s not found: %s",
+					S_ISDIR(mode) ?  "Tree" : "Blob",
+					command_buf.buf);
+		if (type != expected)
+			die("Not a %s (actually a %s): %s",
+				typename(expected), typename(type),
+				command_buf.buf);
 	}
 
 	tree_content_set(&b->branch_tree, p, sha1, mode, NULL);
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 131f032..50d5913 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -796,6 +796,60 @@ test_expect_success \
 	'git fast-import <input &&
 	 test `git rev-parse N2^{tree}` = `git rev-parse N3^{tree}`'
 
+test_expect_success \
+	'N: copy directory by id' \
+	'cat >expect <<-\EOF &&
+	:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file3/newf
+	:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100	file2/oldf	file3/oldf
+	EOF
+	 subdir=$(git rev-parse refs/heads/branch^0:file2) &&
+	 cat >input <<-INPUT_END &&
+	commit refs/heads/N4
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	copy by tree hash
+	COMMIT
+
+	from refs/heads/branch^0
+	M 040000 $subdir file3
+	INPUT_END
+	 git fast-import <input &&
+	 git diff-tree -C --find-copies-harder -r N4^ N4 >actual &&
+	 compare_diff_raw expect actual'
+
+test_expect_success \
+	'N: modify copied tree' \
+	'cat >expect <<-\EOF &&
+	:100644 100644 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 C100	newdir/interesting	file3/file5
+	:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file3/newf
+	:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100	file2/oldf	file3/oldf
+	EOF
+	 subdir=$(git rev-parse refs/heads/branch^0:file2) &&
+	 cat >input <<-INPUT_END &&
+	commit refs/heads/N5
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	copy by tree hash
+	COMMIT
+
+	from refs/heads/branch^0
+	M 040000 $subdir file3
+
+	commit refs/heads/N5
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	modify directory copy
+	COMMIT
+
+	M 644 inline file3/file5
+	data <<EOF
+	$file5_data
+	EOF
+	INPUT_END
+	 git fast-import <input &&
+	 git diff-tree -C --find-copies-harder -r N5^^ N5 >actual &&
+	 compare_diff_raw expect actual'
+
 ###
 ### series O
 ###
-- 
1.7.1.1

[WIP/PATCH] Teach fast-import to print the id of each imported commit

[Jonathan Nieder]

For the svn importer, it would be useful to have a map from
subversion revision numbers to git commits.  This is particularly
relevant because the subversion api sometimes represents as "copy
this directory from this revision", and the importer needs to be
able to access the corresponding trees.  So (optionally) print
each commit id when the corresponding object is written.

Unfortunately when each commit object is written, it is not yet
accessible to the caller.  The corresponding pack index
and header are not written until the next checkpoint finishes.

Should fast-import accept lines of the form

 M 100644 <commit id>:<path> <path>

and

 M 040000 <commit id>:<path> <path>

to allow the caller to use commits before they are accessible
through the git object database?

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
but definitely not ready for inclusion
---
 fast-import.c          |   18 ++++++++++++++++++
 t/t9300-fast-import.sh |   45 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 0 deletions(-)

diff --git a/fast-import.c b/fast-import.c
index ad6843a..869da7f 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -361,6 +361,9 @@ static uintmax_t next_mark;
 static struct strbuf new_data = STRBUF_INIT;
 static int seen_data_command;
 
+/* Where to report commits */
+static int commits_fd = -1;
+
 static void parse_argv(void);
 
 static void write_branch_report(FILE *rpt, struct branch *b)
@@ -2563,6 +2566,11 @@ static void parse_new_commit(void)
 
 	if (!store_object(OBJ_COMMIT, &new_data, NULL, b->sha1, next_mark))
 		b->pack_id = pack_id;
+	if (commits_fd != -1) {
+		char *buf = sha1_to_hex(b->sha1);
+		buf[40] = '\n';
+		write_or_die(commits_fd, buf, 41);
+	}
 	b->last_commit = object_count_by_type[OBJ_COMMIT];
 }
 
@@ -2747,6 +2755,14 @@ static void option_export_marks(const char *marks)
 	safe_create_leading_directories_const(export_marks_file);
 }
 
+static void option_print_commits(const char *fd)
+{
+	unsigned long n = strtoul(fd, NULL, 0);
+	if (n > (unsigned long) INT_MAX)
+		die("--print-commits cannot exceed %d", INT_MAX);
+	commits_fd = (int) n;
+}
+
 static void option_export_pack_edges(const char *edges)
 {
 	if (pack_edges)
@@ -2800,6 +2816,8 @@ static int parse_one_feature(const char *feature, int from_stream)
 		option_import_marks(feature + 13, from_stream);
 	} else if (!prefixcmp(feature, "export-marks=")) {
 		option_export_marks(feature + 13);
+	} else if (!prefixcmp(feature, "print-commits=")) {
+		option_print_commits(feature + strlen("print-commits="));
 	} else if (!prefixcmp(feature, "relative-marks")) {
 		relative_marks_paths = 1;
 	} else if (!prefixcmp(feature, "no-relative-marks")) {
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 50d5913..5cb949f 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1555,6 +1555,51 @@ test_expect_success 'R: feature no-relative-marks should be honoured' '
     test_cmp marks.new non-relative.out
 '
 
+test_expect_success 'R: print-commits magic' '
+	mkfifo commits &&
+	cat >caller <<-\CALLER_END
+		#!/bin/sh
+		cat <<EOF &&
+		feature print-commits=3
+		commit refs/heads/printed
+		committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+		data <<COMMIT
+		to be printed
+		COMMIT
+
+		from refs/heads/master
+		D file3
+
+		EOF
+
+		read cid <&3 &&
+		! git rev-parse $cid: >/dev/null &&
+		echo checkpoint &&
+		sleep 1 &&
+		tree=$(git rev-parse $cid:) &&
+		cat <<EOF
+		commit refs/heads/printed
+		committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+		data <<COMMIT
+		to be printed
+		COMMIT
+
+		from refs/heads/printed^0
+		M 040000 $tree old
+
+		EOF
+		sleep 1
+	CALLER_END
+	chmod +x caller &&
+	{
+		sh caller 3<commits ||
+		echo fail
+	} |
+	git fast-import 3>commits &&
+	git rev-parse printed^ printed &&
+	git show printed:
+'
+
 cat >input << EOF
 option git quiet
 blob
-- 
1.7.1.1

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Sverre Rabbelier]

Heya,

On Thu, Jul 1, 2010 at 07:48, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Unfortunately when each commit object is written, it is not yet
> accessible to the caller.

This I don't understand, as soon as the object is written wouldn't it
be available in .git/objects?

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH/RFC] Teach fast-import to import subtrees named by tree id

[Sverre Rabbelier]

Heya,

On Thu, Jul 1, 2010 at 05:18, Jonathan Nieder <jrnieder@gmail.com> wrote:
> I was not sure whether to add a "feature" specification for
> this, so I’ll try that as a separate patch.

Yes, that'd be a good idea. That should be as easy as adding an option though?

-- 
Cheers,

Sverre Rabbelier

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Jonathan Nieder]

Sverre Rabbelier wrote:

> This I don't understand, as soon as the object is written wouldn't it
> be available in .git/objects?

fast-import writes objects directly to pack --- part of the secret to
its speed.

See store_object() in fast-import.c, or you can play around with the
test from my WIP patch to try it out (removing the ‘sleep 1’).

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Sverre Rabbelier]

Heya,

On Fri, Jul 2, 2010 at 05:41, Jonathan Nieder <jrnieder@gmail.com> wrote:
> fast-import writes objects directly to pack --- part of the secret to
> its speed.

Aah, so only after a 'checkpoint', I see.

> See store_object() in fast-import.c, or you can play around with the
> test from my WIP patch to try it out (removing the ‘sleep 1’).

Okay, in that case I think your other patch is made of even more win :)

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH/RFC] Teach fast-import to import subtrees named by tree id

[Jonathan Nieder]

Sverre Rabbelier wrote:
> On Thu, Jul 1, 2010 at 05:18, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> I was not sure whether to add a "feature" specification for
>> this, so I’ll try that as a separate patch.
[...]
> That should be as easy as adding an option though?

Yes, I think so.

-- 8< --
Subject: fast-import: And a feature option for importing subtrees

The "feature external-trees" command avoids wasting time
importing with old backends only to error out on a later
"M 040000 <tree hash> <path>".

Its use is completely optional.  In particular, if backends
without the external-trees feature become extinct, then there
will be no reason to keep using it.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Documentation/git-fast-import.txt |   17 +++++++++++++----
 fast-import.c                     |    3 +++
 2 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 5edf059..54a3ea1 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -484,9 +484,12 @@ External data format::
 +
 Here usually `<dataref>` must be either a mark reference (`:<idnum>`)
 set by a prior `blob` command, or a full 40-byte SHA-1 of an
-existing Git blob object.  If `<mode>` is `040000` then
-`<dataref>` must be the full 40-byte SHA-1 of an existing
-Git tree object or a mark reference set with `--import-marks`.
+existing Git blob object.
++
+If `<mode>` is `040000` then `<dataref>` must be the full 40-byte
+SHA-1 of an existing Git tree object or a mark reference set with
+`--import-marks`.  Frontends can check for a backend supporting
+this mode with the `feature external-trees` command.
 
 Inline data format::
 	The data content for the file has not been supplied yet.
@@ -889,7 +892,8 @@ The <feature> part of the command may be any string matching
 ^[a-zA-Z][a-zA-Z-]*$ and should be understood by fast-import.
 
 Feature work identical as their option counterparts with the
-exception of the import-marks feature, see below.
+exception of the import-marks and external-trees features;
+see below.
 
 The following features are currently supported:
 
@@ -898,6 +902,7 @@ The following features are currently supported:
 * export-marks
 * relative-marks
 * no-relative-marks
+* external-trees
 * force
 
 The import-marks behaves differently from when it is specified as
@@ -905,6 +910,10 @@ commandline option in that only one "feature import-marks" is allowed
 per stream. Also, any --import-marks= specified on the commandline
 will override those from the stream (if any).
 
+The `feature external-trees` command can be used to abort early
+if the backend does not support `M 040000 <tree hash> <path>`.
+It has no effect in recent fast-import verisons.
+
 `option`
 ~~~~~~~~
 Processes the specified option so that git fast-import behaves in a
diff --git a/fast-import.c b/fast-import.c
index ad6843a..290d4b1 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -2820,6 +2820,9 @@ static void parse_feature(void)
 	if (seen_data_command)
 		die("Got feature command '%s' after data command", feature);
 
+	if (!prefixcmp(feature, "external-trees"))
+		/* Yes, we do support the M 040000 command. */
+		return;
 	if (parse_one_feature(feature, 1))
 		return;
 
-- 
1.7.1.1

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Jonathan Nieder]

Jonathan Nieder wrote:

> Should fast-import accept lines of the form
> 
>  M 100644 <commit id>:<path> <path>
> 
> and
> 
>  M 040000 <commit id>:<path> <path>
> 
> to allow the caller to use commits before they are accessible
> through the git object database?

A ‘cat’ command (suggested by David) would also be useful, so the
caller can read trees and blobs before they are accessible through
the object db.  The svn importer would use this when applying
(svndiff0-format) deltas to import changes to regular files.

Re: [PATCH/RFC] Teach fast-import to import subtrees named by tree id

[Ramkumar Ramachandra]

Hi Jonathan,

Jonathan Nieder writes:
> To simulate the svn cp command, it would be very useful to be
> replace an arbitrary file in the current revision by an
> arbitrary directory from a previous one.  Modify the filemodify
> command to allow that:
> 
>  M 040000 <tree id> pathname
> 
> This would be most useful in combination with a facility to
> print the commit ids for new revisions as they are written.

Thanks for this patch! I applied and tested it: works as
expected. It's ready for inclusion, yes?

Reviewed-by: Ramkumar Ramachandra <artagnon@gmail.com>

> ---
> I actually thought fast-import already did this until David
> mentioned that no, it does not.  Well, live and learn.
> 
> This and Sverre’s --print-marks command would allow svn-fe
> to be simplified a great deal.
> 
> I was not sure whether to add a "feature" specification for
> this, so I’ll try that as a separate patch.
> 
> Thoughts?
> Jonathan
> 
>  Documentation/git-fast-import.txt |    8 ++++-
>  fast-import.c                     |   24 ++++++++++------
>  t/t9300-fast-import.sh            |   54 +++++++++++++++++++++++++++++++++++++
>  3 files changed, 75 insertions(+), 11 deletions(-)
> 
> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
> index 19082b0..f4d9aeb 100644
> --- a/Documentation/git-fast-import.txt
> +++ b/Documentation/git-fast-import.txt
> @@ -482,9 +482,11 @@ External data format::
>  	'M' SP <mode> SP <dataref> SP <path> LF
>  ....
>  +
> -Here `<dataref>` can be either a mark reference (`:<idnum>`)
> +Here usually `<dataref>` must be either a mark reference (`:<idnum>`)
>  set by a prior `blob` command, or a full 40-byte SHA-1 of an
> -existing Git blob object.
> +existing Git blob object.  If `<mode>` is `040000`` then
> +`<dataref>` must be the full 40-byte SHA-1 of an existing
> +Git tree object or a mark reference set with `--import-marks`.
>  
>  Inline data format::
>  	The data content for the file has not been supplied yet.
> @@ -509,6 +511,8 @@ in octal.  Git only supports the following modes:
>  * `160000`: A gitlink, SHA-1 of the object refers to a commit in
>    another repository. Git links can only be specified by SHA or through
>    a commit mark. They are used to implement submodules.
> +* `040000`: A subdirectory.  Subdirectories can only be specified by
> +  SHA or through a tree mark set with `--import-marks`.
>  
>  In both formats `<path>` is the complete path of the file to be added
>  (if not already existing) or modified (if already existing).
> diff --git a/fast-import.c b/fast-import.c
> index 1e5d66e..ad6843a 100644
> --- a/fast-import.c
> +++ b/fast-import.c
> @@ -2131,6 +2131,7 @@ static void file_change_m(struct branch *b)
>  	case S_IFREG | 0644:
>  	case S_IFREG | 0755:
>  	case S_IFLNK:
> +	case S_IFDIR:
>  	case S_IFGITLINK:
>  		/* ok */
>  		break;
> @@ -2176,23 +2177,28 @@ static void file_change_m(struct branch *b)
>  		 * another repository.
>  		 */
>  	} else if (inline_data) {
> +		if (S_ISDIR(mode))
> +			die("Directories cannot be specified 'inline': %s",
> +				command_buf.buf);

Okay. Since you've passed S_IFDIR in the earlier switch-case, you've
made sure that directories aren't specified inline here.

>  		if (p != uq.buf) {
>  			strbuf_addstr(&uq, p);
>  			p = uq.buf;
>  		}
>  		read_next_command();
>  		parse_and_store_blob(&last_blob, sha1, 0);
> -	} else if (oe) {
> -		if (oe->type != OBJ_BLOB)
> -			die("Not a blob (actually a %s): %s",
> -				typename(oe->type), command_buf.buf);
>  	} else {
> -		enum object_type type = sha1_object_info(sha1, NULL);
> +		enum object_type expected = S_ISDIR(mode) ?
> +						OBJ_TREE: OBJ_BLOB;
> +		enum object_type type = oe ? oe->type :
> +					sha1_object_info(sha1, NULL);

Instead allowing just blobs, you've allowed both blob and tree objects
to be specified here. I don't see any tree writing code in your
change, so I'm assuming blob and tree writing is just handled
transparently in Git.

>  		if (type < 0)
> -			die("Blob not found: %s", command_buf.buf);
> -		if (type != OBJ_BLOB)
> -			die("Not a blob (actually a %s): %s",
> -			    typename(type), command_buf.buf);
> +			die("%s not found: %s",
> +					S_ISDIR(mode) ?  "Tree" : "Blob",
> +					command_buf.buf);
> +		if (type != expected)
> +			die("Not a %s (actually a %s): %s",
> +				typename(expected), typename(type),
> +				command_buf.buf);
>  	}
>  	tree_content_set(&b->branch_tree, p, sha1, mode, NULL);

Conditionally printing either "blob" or "tree" in the error message. Okay.

> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
> index 131f032..50d5913 100755
> --- a/t/t9300-fast-import.sh
> +++ b/t/t9300-fast-import.sh

Okay. Test passes.

-- Ram

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Sverre Rabbelier]

Heya,

On Fri, Jul 2, 2010 at 07:12, Jonathan Nieder <jrnieder@gmail.com> wrote:
> A ‘cat’ command (suggested by David) would also be useful, so the
> caller can read trees and blobs before they are accessible through
> the object db.  The svn importer would use this when applying
> (svndiff0-format) deltas to import changes to regular files.

How would the output be delimited? The same way fast-import expects
its blob input?

-- 
Cheers,

Sverre Rabbelier

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Jonathan Nieder]

Sverre Rabbelier wrote:
> On Fri, Jul 2, 2010 at 07:12, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> A ‘cat’ command (suggested by David) would also be useful, so the
>> caller can read trees and blobs before they are accessible through
>> the object db.  The svn importer would use this when applying
>> (svndiff0-format) deltas to import changes to regular files.
>
> How would the output be delimited? The same way fast-import expects
> its blob input?

Right.  I was imagining something like

 <blob hash> blob <size>
 ... content ...
 <blank line>

à la cat-file --batch.  Hopefully by the end of this weekend, if no
one else gets to it first.

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Sverre Rabbelier]

Heya,

On Fri, Jul 2, 2010 at 17:40, Jonathan Nieder <jrnieder@gmail.com> wrote:
> à la cat-file --batch.  Hopefully by the end of this weekend, if no
> one else gets to it first.

Okay, sounds like an excellent plan. Perhaps you can even borrow from
git cat-file's code?

-- 
Cheers,

Sverre Rabbelier

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Sam Vilain]

On Fri, 2010-07-02 at 10:40 -0500, Jonathan Nieder wrote:
> Sverre Rabbelier wrote:
> > On Fri, Jul 2, 2010 at 07:12, Jonathan Nieder <jrnieder@gmail.com> wrote:
> 
> >> A ‘cat’ command (suggested by David) would also be useful, so the
> >> caller can read trees and blobs before they are accessible through
> >> the object db.  The svn importer would use this when applying
> >> (svndiff0-format) deltas to import changes to regular files.
> >
> > How would the output be delimited? The same way fast-import expects
> > its blob input?
> 
> Right.  I was imagining something like
> 
>  <blob hash> blob <size>
>  ... content ...
>  <blank line>
> 
> à la cat-file --batch.  Hopefully by the end of this weekend, if no
> one else gets to it first.

It should be very simple to implement --batch-verify as well, which is
essentially the same thing, just only printing the header and not the
contents.

How hard do you think it would be to implement a 'rev-parse' command?
Perhaps you can see where this is going... :-)

Sam

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Jonathan Nieder]

Sam Vilain wrote:

> It should be very simple to implement --batch-verify as well, which is
> essentially the same thing, just only printing the header and not the
> contents.
> 
> How hard do you think it would be to implement a 'rev-parse' command?
> Perhaps you can see where this is going... :-)

Well, I think I understand what you are saying, but it’s not quite
like that.  fast-import is already doing a lot of work to keep track
of this collection of objects in a pack that is not yet accessible to
git because its index has not been written yet.

svn-fe needs those objects (well, it at least needs a few blobs when
it learns to apply a delta to the latest revision).  So what to do?

One option is to make fast-import into a library, or copy the code we
need into svn-fe.  And that would be possible!  But it would make
svn-fe much more complicated, and there would be this second
fast-import interface to maintain.

Or svn-fe could keep the full text of all files from the latest
revision in a temporary directory.  To get blobs for an _older_
revision, it would still need to ask git, but such copy operations are
uncommon enough that the time to request a checkpoint and wait for
things to settle could be tolerable.  The main downside: in a huge
repository like ASF’s, that’s a good amount of wasted space.

Or one can teach fast-import to supply exactly what we need from it,
which is the raw content of some blobs (and perhaps trees) it has
written to pack and not indexed yet.

Jonathan

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Sam Vilain]

On Sat, 2010-07-03 at 19:35 -0500, Jonathan Nieder wrote:
> Sam Vilain wrote:
> 
> > It should be very simple to implement --batch-verify as well, which is
> > essentially the same thing, just only printing the header and not the
> > contents.
> > 
> > How hard do you think it would be to implement a 'rev-parse' command?
> > Perhaps you can see where this is going... :-)
> 
> Well, I think I understand what you are saying, but it’s not quite
> like that.  fast-import is already doing a lot of work to keep track
> of this collection of objects in a pack that is not yet accessible to
> git because its index has not been written yet.
  [...]
> Or one can teach fast-import to supply exactly what we need from it,
> which is the raw content of some blobs (and perhaps trees) it has
> written to pack and not indexed yet.

Let me explain further.  If in fast-import.c:new_object, if it were to
make a struct object (see object.h), and make sure it was put in
obj_hash (see object.c, particularly create_object()), then you might
find a whole load of plumbing would magically start working and be able
to work with the new objects that you are trying to load.  Of course
there may be a couple of other functions which might need to change.
Primarily object.c:read_object, which needs to be able to check the
packfile being spooled by git fast-import.

If you did this, then to implement this feature you could in principle
just call read_sha1_file()

Sam

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Jonathan Nieder]

Sam Vilain wrote:

> Let me explain further.  If in fast-import.c:new_object, if it were to
> make a struct object (see object.h), and make sure it was put in
> obj_hash (see object.c, particularly create_object()), then you might
> find a whole load of plumbing would magically start working and be able
> to work with the new objects that you are trying to load.  Of course
> there may be a couple of other functions which might need to change.
> Primarily object.c:read_object, which needs to be able to check the
> packfile being spooled by git fast-import.
> 
> If you did this, then to implement this feature you could in principle
> just call read_sha1_file()

Oh, to implement the ‘cat’ feature, I was planning to call
gfi_unpack_entry; the hard part is finding a spare moment to do that
(if others have more spare moments, I would be happy to find that
step unnecessary).

What you describe sounds more useful as part of a longer-term plan.
If fast-import and its caller live in the same process, then being
able to use the usual object access APIs would be convenient indeed.
In other words, it sounds like an incentive to libify fast-import.

Well, everything in due time.

Thanks for the comments,
Jonathan

Re: [WIP/PATCH] Teach fast-import to print the id of each imported commit

[Ramkumar Ramachandra]

Hi,

Jonathan Nieder writes:
> For the svn importer, it would be useful to have a map from
> subversion revision numbers to git commits.  This is particularly
> relevant because the subversion api sometimes represents as "copy
> this directory from this revision", and the importer needs to be
> able to access the corresponding trees.  So (optionally) print
> each commit id when the corresponding object is written.
> 
> Unfortunately when each commit object is written, it is not yet
> accessible to the caller.  The corresponding pack index
> and header are not written until the next checkpoint finishes.
> 
> Should fast-import accept lines of the form
> 
>  M 100644 <commit id>:<path> <path>
> 
> and
> 
>  M 040000 <commit id>:<path> <path>
> 
> to allow the caller to use commits before they are accessible
> through the git object database?
> 
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> but definitely not ready for inclusion

I'm resurrecting this thread. We need to finish this feature before we
can finish the dumpfilev3 support in svn-dump-fast-export.

-- Ram

[RFC/PATCH 0/3] fast-import: give importers access to the object store

[Jonathan Nieder]

Hi Ram,

Ramkumar Ramachandra wrote:

> I'm resurrecting this thread. We need to finish this feature before we
> can finish the dumpfilev3 support in svn-dump-fast-export.

Ok.

To recap, because fast-import does not write a pack index until the
checkpoint, frontends cannot necessarily read back what they have
written immediately.  The obvious application to allowing that is as a
sanity check, but more important to us is that it allows the lazy
frontend to forget previous revisions even if they are required for
later "file/directory copy" operations in the import.  The frontend
can be secure in the knowledge that the backend remembers everything
now.

It works like this:

frontend:
	feature report-fd=3
	commit refs/heads/master
	... revision 1 ...

importer:
	abc7856cba76bca87a65bca76bca8bca98bca78bca76

frontend:
	commit refs/heads/master
	... revision 2 ...

importer:
	fcdafecfdacba667cd5a4da6dca5fa68ca897dc65178

...

frontend:
	commit refs/heads/master
	... revision 50 ...

	cat abc7856cba76bca87a65bca76bca8bca98bca78

importer:
	c78a67d9987da089cd89ac879dacd897acd879acd76
	commit abc7856cba76bca87a65bca76bca8bca98bca78bca76 274819
	tree <... tree name ...>
	author <... author name ...>
	...

I suspect that these patches are not in their final form.
In particular, the interface is kind of klunky: to name a
blob by path, you have to supply a *tree* which contains that
blob as well as the pathname.  So retrieving, say,
v1.7.1:Documentation/git-fast-import.txt, would require
three round-trips: one to dereference the tag, one to dereference
the commit, and then one to finally retrieve the blob.

Another possible concern is that this is very much git specific.
Other fast-import backends are just not going to be able to do
it with the same format.  Is there a convention for naming
options like that?

Still, I hope it is useful to start with.  Thoughts?  Ideas?
Improvements?

Jonathan Nieder (3):
  t9300 (fast-import): style tweaks
  Teach fast-import to print the id of each imported commit
  fast-import: Let importers retrieve the objects being written

 Documentation/git-fast-import.txt |   45 +
 fast-import.c                     |  126 ++
 t/t9300-fast-import.sh            | 3371 ++++++++++++++++++++-----------------
 3 files changed, 1960 insertions(+), 1582 deletions(-)

-- 
1.7.2.3

[PATCH 1/3] t9300 (fast-import): style tweaks

[Jonathan Nieder]

Clarify dependencies between tests to make the fast-import test
script more approachable.  In particular:

 - guard setup commands with test assertions (for robustness and to
   make it easier to identify where each test starts and ends);

 - mention "setup" in the descriptions of tests that are used to
   prepare for later ones;

 - avoid hard-coded object names.  Although compare_diff_raw means
   most of them do not affect the test result, the reader can
   benefit more from other object descriptions;

 - tweak whitespace style: each test now starts with a test assertion,
   description, and opening quote on a single line, followed by test
   code that checks a single claim;

While at it:

 - introduce a verify_packs() helper, avoiding some repetition;

 - use test_cmp instead of test $foo = $bar for nicer output with -v;

 - use multiple separate commands instead of pipelines when that
   helps debugability;

 - use $(...) command substitutions in preference to ``; the former
   are less error-prone with respect to quoting and easier to notice
   when looking for $variable interpolations.

 - do not let tests exit the entire test script when they fail.

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh | 3207 ++++++++++++++++++++++++------------------------
 1 files changed, 1629 insertions(+), 1578 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 96d07f1..5c274e7 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -7,860 +7,890 @@ test_description='test git fast-import utility'
 . ./test-lib.sh
 . "$TEST_DIRECTORY"/diff-lib.sh ;# test-lib chdir's into trash
 
-file2_data='file2
-second line of EOF'
-
-file3_data='EOF
-in 3rd file
- END'
-
-file4_data=abcd
-file4_len=4
-
-file5_data='an inline file.
-  we should see it later.'
-
-file6_data='#!/bin/sh
-echo "$@"'
-
-###
-### series A
-###
-
-test_tick
-cat >input <<INPUT_END
-blob
-mark :2
-data <<EOF
-$file2_data
-EOF
-
-blob
-mark :3
-data <<END
-$file3_data
-END
-
-blob
-mark :4
-data $file4_len
-$file4_data
-commit refs/heads/master
-mark :5
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-initial
-COMMIT
-
-M 644 :2 file2
-M 644 :3 file3
-M 755 :4 file4
-
-tag series-A
-from :5
-data <<EOF
-An annotated tag without a tagger
-EOF
-
-INPUT_END
-test_expect_success \
-    'A: create pack from stdin' \
-    'git fast-import --export-marks=marks.out <input &&
-	 git whatchanged master'
-test_expect_success \
-	'A: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-
-cat >expect <<EOF
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-initial
-EOF
-test_expect_success \
-	'A: verify commit' \
-	'git cat-file commit master | sed 1d >actual &&
-	test_cmp expect actual'
-
-cat >expect <<EOF
-100644 blob file2
-100644 blob file3
-100755 blob file4
-EOF
-test_expect_success \
-	'A: verify tree' \
-	'git cat-file -p master^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
-
-echo "$file2_data" >expect
-test_expect_success \
-	'A: verify file2' \
-	'git cat-file blob master:file2 >actual && test_cmp expect actual'
-
-echo "$file3_data" >expect
-test_expect_success \
-	'A: verify file3' \
-	'git cat-file blob master:file3 >actual && test_cmp expect actual'
-
-printf "$file4_data" >expect
-test_expect_success \
-	'A: verify file4' \
-	'git cat-file blob master:file4 >actual && test_cmp expect actual'
-
-cat >expect <<EOF
-object $(git rev-parse refs/heads/master)
-type commit
-tag series-A
-
-An annotated tag without a tagger
-EOF
+test_expect_success 'setup: verify_packs helper' '
+	verify_packs () {
+		for p in .git/objects/pack/*.pack
+		do
+			git verify-pack $p ||
+			return
+		done
+	}
+'
+
+test_expect_success 'setup' '
+	zeroes=0000000000000000000000000000000000000000 &&
+
+	# Use command substitution to strip off final newlines.
+	file2_data=$(cat <<-\END_FILE2_DATA
+		file2
+		second line of EOF
+		END_FILE2_DATA
+	) &&
+	file2_id=$(echo "$file2_data" | git hash-object --stdin) &&
+	file3_data=$(cat <<-\END_FILE3_DATA
+		EOF
+		in 3rd file
+		 END
+		END_FILE3_DATA
+	) &&
+	file3_id=$(echo "$file3_data" | git hash-object --stdin) &&
+	file4_data=abcd &&
+	file4_len=4 &&
+	file4_id=$(printf "%s" "$file4_data" | git hash-object --stdin) &&
+	file5_data=$(cat <<-\END_FILE5_DATA
+		an inline file.
+		  we should see it later.
+		END_FILE5_DATA
+	) &&
+	file5_id=$(echo "$file5_data" | git hash-object --stdin) &&
+	file6_data=$(cat <<-\END_FILE6_DATA
+		#!/bin/sh
+		echo "$@"
+		END_FILE6_DATA
+	) &&
+	file6_id=$(echo "$file6_data" | git hash-object --stdin)
+'
+
+test_expect_success 'setup: series A' '
+	test_tick &&
+
+	cat >input <<-INPUT_END &&
+	blob
+	mark :2
+	data <<EOF
+	$file2_data
+	EOF
+
+	blob
+	mark :3
+	data <<END
+	$file3_data
+	END
+
+	blob
+	mark :4
+	data $file4_len
+	$file4_data
+	commit refs/heads/master
+	mark :5
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	initial
+	COMMIT
+
+	M 644 :2 file2
+	M 644 :3 file3
+	M 755 :4 file4
+
+	tag series-A
+	from :5
+	data <<EOF
+	An annotated tag without a tagger
+	EOF
+
+	INPUT_END
+
+	git fast-import --export-marks=marks.out <input &&
+	git whatchanged master
+'
+
+test_expect_success 'fast-import writes valid packs' '
+	verify_packs
+'
+
+test_expect_success 'A: verify commit' '
+	cat >expect <<-EOF &&
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	initial
+	EOF
+	git cat-file commit master >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'A: verify tree' '
+	cat >expect <<-\EOF &&
+	100644 blob file2
+	100644 blob file3
+	100755 blob file4
+	EOF
+	git cat-file -p master^{tree} >tree &&
+	sed "s/ [0-9a-f]*	/ /" <tree >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'A: verify file2' '
+	echo "$file2_data" >expect &&
+	git cat-file blob master:file2 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'A: verify file3' '
+	echo "$file3_data" >expect &&
+	git cat-file blob master:file3 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'A: verify file4' '
+	printf "$file4_data" >expect &&
+	git cat-file blob master:file4 >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'A: verify tag/series-A' '
+	master=$(git rev-parse refs/heads/master) &&
+	cat >expect <<-EOF &&
+	object $master
+	type commit
+	tag series-A
+
+	An annotated tag without a tagger
+	EOF
 	git cat-file tag tags/series-A >actual &&
 	test_cmp expect actual
 '
 
-cat >expect <<EOF
-:2 `git rev-parse --verify master:file2`
-:3 `git rev-parse --verify master:file3`
-:4 `git rev-parse --verify master:file4`
-:5 `git rev-parse --verify master^0`
-EOF
-test_expect_success \
-	'A: verify marks output' \
-	'test_cmp expect marks.out'
+test_expect_success 'A: verify marks output' '
+	cat >expect <<-EOF &&
+	:2 $(git rev-parse --verify master:file2)
+	:3 $(git rev-parse --verify master:file3)
+	:4 $(git rev-parse --verify master:file4)
+	:5 $(git rev-parse --verify master^0)
+	EOF
+	test_cmp expect marks.out
+'
 
-test_expect_success \
-	'A: verify marks import' \
-	'git fast-import \
+test_expect_success 'A: import marks' '
+	cat >expect <<-EOF &&
+	:2 $(git rev-parse --verify master:file2)
+	:3 $(git rev-parse --verify master:file3)
+	:4 $(git rev-parse --verify master:file4)
+	:5 $(git rev-parse --verify master^0)
+	EOF
+	git fast-import \
 		--import-marks=marks.out \
 		--export-marks=marks.new \
 		</dev/null &&
-	test_cmp expect marks.new'
-
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/verify--import-marks
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-recreate from :5
-COMMIT
-
-from :5
-M 755 :2 copy-of-file2
-
-INPUT_END
-test_expect_success \
-	'A: verify marks import does not crash' \
-	'git fast-import --import-marks=marks.out <input &&
-	 git whatchanged verify--import-marks'
-test_expect_success \
-	'A: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-cat >expect <<EOF
-:000000 100755 0000000000000000000000000000000000000000 7123f7f44e39be127c5eb701e5968176ee9d78b1 A	copy-of-file2
-EOF
-git diff-tree -M -r master verify--import-marks >actual
-test_expect_success \
-	'A: verify diff' \
-	'compare_diff_raw expect actual &&
-	 test `git rev-parse --verify master:file2` \
-	    = `git rev-parse --verify verify--import-marks:copy-of-file2`'
-
-test_tick
-mt=$(git hash-object --stdin < /dev/null)
-: >input.blob
-: >marks.exp
-: >tree.exp
-
-cat >input.commit <<EOF
-commit refs/heads/verify--dump-marks
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-test the sparse array dumping routines with exponentially growing marks
-COMMIT
-EOF
-
-i=0
-l=4
-m=6
-n=7
-while test "$i" -lt 27; do
-    cat >>input.blob <<EOF
-blob
-mark :$l
-data 0
-blob
-mark :$m
-data 0
-blob
-mark :$n
-data 0
-EOF
-    echo "M 100644 :$l l$i" >>input.commit
-    echo "M 100644 :$m m$i" >>input.commit
-    echo "M 100644 :$n n$i" >>input.commit
-
-    echo ":$l $mt" >>marks.exp
-    echo ":$m $mt" >>marks.exp
-    echo ":$n $mt" >>marks.exp
-
-    printf "100644 blob $mt\tl$i\n" >>tree.exp
-    printf "100644 blob $mt\tm$i\n" >>tree.exp
-    printf "100644 blob $mt\tn$i\n" >>tree.exp
-
-    l=$(($l + $l))
-    m=$(($m + $m))
-    n=$(($l + $n))
-
-    i=$((1 + $i))
-done
-
-sort tree.exp > tree.exp_s
+	test_cmp expect marks.new
+'
+
+test_expect_success 'setup: A: verify marks import' '
+	test_tick &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/verify--import-marks
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	recreate from :5
+	COMMIT
+
+	from :5
+	M 755 :2 copy-of-file2
+
+	INPUT_END
+
+	git fast-import --import-marks=marks.out <input &&
+	verify_packs &&
+	git whatchanged verify--import-marks
+'
+
+test_expect_success 'A: verify diff' '
+	echo ":000000 100755 $zeroes $file2_id A	copy-of-file2" >expect &&
+	echo $file2_id >expect.copy &&
+	git diff-tree -M -r master verify--import-marks >actual &&
+	git rev-parse --verify verify--import-marks:copy-of-file2 >actual.copy &&
+	compare_diff_raw expect actual &&
+	test_cmp expect.copy actual.copy
+'
 
 test_expect_success 'A: export marks with large values' '
-	cat input.blob input.commit | git fast-import --export-marks=marks.large &&
+	test_tick &&
+	mt=$(git hash-object --stdin </dev/null) &&
+	>input.blob &&
+	>marks.exp &&
+	>tree.exp &&
+
+	cat >input.commit <<-EOF &&
+	commit refs/heads/verify--dump-marks
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	test the sparse array dumping routines with exponentially growing marks
+	COMMIT
+	EOF
+
+	(
+		i=0 &&
+		l=4 &&
+		m=6 &&
+		n=7 &&
+		while test "$i" -lt 27
+		do
+			cat >>input.blob <<-EOF &&
+			blob
+			mark :$l
+			data 0
+			blob
+			mark :$m
+			data 0
+			blob
+			mark :$n
+			data 0
+			EOF
+			{
+				echo "M 100644 :$l l$i" &&
+				echo "M 100644 :$m m$i" &&
+				echo "M 100644 :$n n$i"
+			} >>input.commit &&
+			{
+				echo ":$l $mt" &&
+				echo ":$m $mt" &&
+				echo ":$n $mt"
+			} >>marks.exp &&
+			{
+				printf "100644 blob $mt\tl$i\n" &&
+				printf "100644 blob $mt\tm$i\n" &&
+				printf "100644 blob $mt\tn$i\n"
+			} >>tree.exp &&
+
+			l=$(($l + $l)) &&
+			m=$(($m + $m)) &&
+			n=$(($l + $n)) &&
+			i=$((1 + $i)) ||
+			exit
+		done
+	) &&
+	sort tree.exp >tree.exp_s &&
+	cat input.blob input.commit |
+	git fast-import --export-marks=marks.large &&
 	git ls-tree refs/heads/verify--dump-marks >tree.out &&
 	test_cmp tree.exp_s tree.out &&
-	test_cmp marks.exp marks.large'
+	test_cmp marks.exp marks.large
+'
 
-###
-### series B
-###
-
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/branch
-mark :1
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-corrupt
-COMMIT
-
-from refs/heads/master
-M 755 0000000000000000000000000000000000000001 zero1
-
-INPUT_END
 test_expect_success 'B: fail on invalid blob sha1' '
-    test_must_fail git fast-import <input
+	test_tick &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/branch
+	mark :1
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	corrupt
+	COMMIT
+
+	from refs/heads/master
+	M 755 0000000000000000000000000000000000000001 zero1
+
+	INPUT_END
+
+	test_must_fail git fast-import <input
 '
-rm -f .git/objects/pack_* .git/objects/index_*
 
-cat >input <<INPUT_END
-commit .badbranchname
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-corrupt
-COMMIT
-
-from refs/heads/master
-
-INPUT_END
 test_expect_success 'B: fail on invalid branch name ".badbranchname"' '
-    test_must_fail git fast-import <input
+	rm -f .git/objects/pack_* .git/objects/index_* &&
+
+	cat >input <<-INPUT_END &&
+	commit .badbranchname
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	corrupt
+	COMMIT
+
+	from refs/heads/master
+
+	INPUT_END
+
+	test_must_fail git fast-import <input
 '
-rm -f .git/objects/pack_* .git/objects/index_*
 
-cat >input <<INPUT_END
-commit bad[branch]name
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-corrupt
-COMMIT
-
-from refs/heads/master
-
-INPUT_END
 test_expect_success 'B: fail on invalid branch name "bad[branch]name"' '
-    test_must_fail git fast-import <input
+	rm -f .git/objects/pack_* .git/objects/index_* &&
+
+	cat >input <<-INPUT_END &&
+	commit bad[branch]name
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	corrupt
+	COMMIT
+
+	from refs/heads/master
+
+	INPUT_END
+
+	test_must_fail git fast-import <input
+'
+
+test_expect_success 'B: accept branch name "TEMP_TAG"' '
+	git rev-parse master >expect &&
+	rm -f .git/objects/pack_* .git/objects/index_* &&
+
+	cat >input <<-INPUT_END &&
+	commit TEMP_TAG
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	tag base
+	COMMIT
+
+	from refs/heads/master
+
+	INPUT_END
+
+	test_when_finished "rm -f .git/TEMP_TAG" &&
+	git fast-import <input &&
+	test -f .git/TEMP_TAG &&
+	git rev-parse TEMP_TAG^ >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'setup: series C' '
+	newf=$(echo hi newf | git hash-object -w --stdin) &&
+	oldf=$(git rev-parse --verify master:file2) &&
+	echo $newf >expect.new &&
+	echo $oldf >expect.old &&
+	test_tick &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/branch
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	second
+	COMMIT
+
+	from refs/heads/master
+	M 644 $oldf file2/oldf
+	M 755 $newf file2/newf
+	D file3
+
+	INPUT_END
+
+	git fast-import <input &&
+	verify_packs &&
+	git whatchanged branch
+'
+
+test_expect_success 'C: reuse of existing blob' '
+	git rev-parse --verify branch:file2/newf >actual.new &&
+	git rev-parse --verify branch:file2/oldf >actual.old &&
+	test_cmp expect.new actual.new &&
+	test_cmp expect.old actual.old
+'
+
+test_expect_success 'C: verify commit' '
+	cat >expect <<-EOF &&
+	parent `git rev-parse --verify master^0`
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	second
+	EOF
+	git cat-file commit branch >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
 '
-rm -f .git/objects/pack_* .git/objects/index_*
-
-cat >input <<INPUT_END
-commit TEMP_TAG
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-tag base
-COMMIT
-
-from refs/heads/master
-
-INPUT_END
-test_expect_success \
-    'B: accept branch name "TEMP_TAG"' \
-    'git fast-import <input &&
-	 test -f .git/TEMP_TAG &&
-	 test `git rev-parse master` = `git rev-parse TEMP_TAG^`'
-rm -f .git/TEMP_TAG
-
-###
-### series C
-###
-
-newf=`echo hi newf | git hash-object -w --stdin`
-oldf=`git rev-parse --verify master:file2`
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/branch
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-second
-COMMIT
-
-from refs/heads/master
-M 644 $oldf file2/oldf
-M 755 $newf file2/newf
-D file3
-
-INPUT_END
-test_expect_success \
-    'C: incremental import create pack from stdin' \
-    'git fast-import <input &&
-	 git whatchanged branch'
-test_expect_success \
-	'C: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-test_expect_success \
-	'C: validate reuse existing blob' \
-	'test $newf = `git rev-parse --verify branch:file2/newf`
-	 test $oldf = `git rev-parse --verify branch:file2/oldf`'
-
-cat >expect <<EOF
-parent `git rev-parse --verify master^0`
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-second
-EOF
-test_expect_success \
-	'C: verify commit' \
-	'git cat-file commit branch | sed 1d >actual &&
-	 test_cmp expect actual'
-
-cat >expect <<EOF
-:000000 100755 0000000000000000000000000000000000000000 f1fb5da718392694d0076d677d6d0e364c79b0bc A	file2/newf
-:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 R100	file2	file2/oldf
-:100644 000000 0d92e9f3374ae2947c23aa477cbc68ce598135f1 0000000000000000000000000000000000000000 D	file3
-EOF
-git diff-tree -M -r master branch >actual
-test_expect_success \
-	'C: validate rename result' \
-	'compare_diff_raw expect actual'
-
-###
-### series D
-###
-
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/branch
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-third
-COMMIT
-
-from refs/heads/branch^0
-M 644 inline newdir/interesting
-data <<EOF
-$file5_data
-EOF
-
-M 755 inline newdir/exec.sh
-data <<EOF
-$file6_data
-EOF
-
-INPUT_END
-test_expect_success \
-    'D: inline data in commit' \
-    'git fast-import <input &&
-	 git whatchanged branch'
-test_expect_success \
-	'D: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-
-cat >expect <<EOF
-:000000 100755 0000000000000000000000000000000000000000 35a59026a33beac1569b1c7f66f3090ce9c09afc A	newdir/exec.sh
-:000000 100644 0000000000000000000000000000000000000000 046d0371e9220107917db0d0e030628de8a1de9b A	newdir/interesting
-EOF
-git diff-tree -M -r branch^ branch >actual
-test_expect_success \
-	'D: validate new files added' \
-	'compare_diff_raw expect actual'
-
-echo "$file5_data" >expect
-test_expect_success \
-	'D: verify file5' \
-	'git cat-file blob branch:newdir/interesting >actual &&
-	 test_cmp expect actual'
-
-echo "$file6_data" >expect
-test_expect_success \
-	'D: verify file6' \
-	'git cat-file blob branch:newdir/exec.sh >actual &&
-	 test_cmp expect actual'
-
-###
-### series E
-###
-
-cat >input <<INPUT_END
-commit refs/heads/branch
-author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> Tue Feb 6 11:22:18 2007 -0500
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> Tue Feb 6 12:35:02 2007 -0500
-data <<COMMIT
-RFC 2822 type date
-COMMIT
-
-from refs/heads/branch^0
-
-INPUT_END
+
+test_expect_success 'C: validate rename result' '
+	cat >expect <<-EOF &&
+	:000000 100755 $zeroes $newf A	file2/newf
+	:100644 100644 $file2_id $file2_id R100	file2	file2/oldf
+	:100644 000000 $file3_id 0000000000000000000000000000000000000000 D	file3
+	EOF
+	git diff-tree -M -r master branch >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'setup: D: inline data in commit' '
+	test_tick &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/branch
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	third
+	COMMIT
+
+	from refs/heads/branch^0
+	M 644 inline newdir/interesting
+	data <<EOF
+	$file5_data
+	EOF
+
+	M 755 inline newdir/exec.sh
+	data <<EOF
+	$file6_data
+	EOF
+
+	INPUT_END
+
+	git fast-import <input &&
+	verify_packs &&
+	git whatchanged branch
+'
+
+test_expect_success 'D: validate new files added' '
+	cat >expect <<-EOF &&
+	:000000 100755 $zeroes $file6_id A	newdir/exec.sh
+	:000000 100644 $zeroes $file5_id A	newdir/interesting
+	EOF
+	git diff-tree -M -r branch^ branch >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'D: verify file5' '
+	echo "$file5_data" >expect &&
+	git cat-file blob branch:newdir/interesting >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'D: verify file6' '
+	echo "$file6_data" >expect &&
+	git cat-file blob branch:newdir/exec.sh >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'E: rfc2822 date, --date-format=raw' '
-    test_must_fail git fast-import --date-format=raw <input
+	cat >input <<-INPUT_END &&
+	commit refs/heads/branch
+	author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> Tue Feb 6 11:22:18 2007 -0500
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> Tue Feb 6 12:35:02 2007 -0500
+	data <<COMMIT
+	RFC 2822 type date
+	COMMIT
+
+	from refs/heads/branch^0
+
+	INPUT_END
+	test_must_fail git fast-import --date-format=raw <input
 '
-test_expect_success \
-    'E: rfc2822 date, --date-format=rfc2822' \
-    'git fast-import --date-format=rfc2822 <input'
-test_expect_success \
-	'E: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-
-cat >expect <<EOF
-author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> 1170778938 -0500
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1170783302 -0500
-
-RFC 2822 type date
-EOF
-test_expect_success \
-	'E: verify commit' \
-	'git cat-file commit branch | sed 1,2d >actual &&
-	test_cmp expect actual'
-
-###
-### series F
-###
-
-old_branch=`git rev-parse --verify branch^0`
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/branch
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-losing things already?
-COMMIT
-
-from refs/heads/branch~1
-
-reset refs/heads/other
-from refs/heads/branch
-
-INPUT_END
-test_expect_success \
-    'F: non-fast-forward update skips' \
-    'if git fast-import <input
-	 then
-		echo BAD gfi did not fail
-		return 1
-	 else
-		if test $old_branch = `git rev-parse --verify branch^0`
-		then
-			: branch unaffected and failure returned
-			return 0
-		else
-			echo BAD gfi changed branch $old_branch
-			return 1
-		fi
-	 fi
-	'
-test_expect_success \
-	'F: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-
-cat >expect <<EOF
-tree `git rev-parse branch~1^{tree}`
-parent `git rev-parse branch~1`
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-losing things already?
-EOF
-test_expect_success \
-	'F: verify other commit' \
-	'git cat-file commit other >actual &&
-	test_cmp expect actual'
-
-###
-### series G
-###
-
-old_branch=`git rev-parse --verify branch^0`
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/branch
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-losing things already?
-COMMIT
-
-from refs/heads/branch~1
-
-INPUT_END
-test_expect_success \
-    'G: non-fast-forward update forced' \
-    'git fast-import --force <input'
-test_expect_success \
-	'G: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-test_expect_success \
-	'G: branch changed, but logged' \
-	'test $old_branch != `git rev-parse --verify branch^0` &&
-	 test $old_branch = `git rev-parse --verify branch@{1}`'
-
-###
-### series H
-###
-
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/H
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-third
-COMMIT
-
-from refs/heads/branch^0
-M 644 inline i-will-die
-data <<EOF
-this file will never exist.
-EOF
-
-deleteall
-M 644 inline h/e/l/lo
-data <<EOF
-$file5_data
-EOF
-
-INPUT_END
-test_expect_success \
-    'H: deletall, add 1' \
-    'git fast-import <input &&
-	 git whatchanged H'
-test_expect_success \
-	'H: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-
-cat >expect <<EOF
-:100755 000000 f1fb5da718392694d0076d677d6d0e364c79b0bc 0000000000000000000000000000000000000000 D	file2/newf
-:100644 000000 7123f7f44e39be127c5eb701e5968176ee9d78b1 0000000000000000000000000000000000000000 D	file2/oldf
-:100755 000000 85df50785d62d3b05ab03d9cbf7e4a0b49449730 0000000000000000000000000000000000000000 D	file4
-:100644 100644 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 R100	newdir/interesting	h/e/l/lo
-:100755 000000 e74b7d465e52746be2b4bae983670711e6e66657 0000000000000000000000000000000000000000 D	newdir/exec.sh
-EOF
-git diff-tree -M -r H^ H >actual
-test_expect_success \
-	'H: validate old files removed, new files added' \
-	'compare_diff_raw expect actual'
-
-echo "$file5_data" >expect
-test_expect_success \
-	'H: verify file' \
-	'git cat-file blob H:h/e/l/lo >actual &&
-	 test_cmp expect actual'
-
-###
-### series I
-###
-
-cat >input <<INPUT_END
-commit refs/heads/export-boundary
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-we have a border.  its only 40 characters wide.
-COMMIT
-
-from refs/heads/branch
-
-INPUT_END
-test_expect_success \
-    'I: export-pack-edges' \
-    'git fast-import --export-pack-edges=edges.list <input'
-
-cat >expect <<EOF
-.git/objects/pack/pack-.pack: `git rev-parse --verify export-boundary`
-EOF
-test_expect_success \
-	'I: verify edge list' \
-	'sed -e s/pack-.*pack/pack-.pack/ edges.list >actual &&
-	 test_cmp expect actual'
-
-###
-### series J
-###
-
-cat >input <<INPUT_END
-commit refs/heads/J
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-create J
-COMMIT
-
-from refs/heads/branch
-
-reset refs/heads/J
-
-commit refs/heads/J
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-initialize J
-COMMIT
-
-INPUT_END
-test_expect_success \
-    'J: reset existing branch creates empty commit' \
-    'git fast-import <input'
-test_expect_success \
-	'J: branch has 1 commit, empty tree' \
-	'test 1 = `git rev-list J | wc -l` &&
-	 test 0 = `git ls-tree J | wc -l`'
-
-###
-### series K
-###
-
-cat >input <<INPUT_END
-commit refs/heads/K
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-create K
-COMMIT
-
-from refs/heads/branch
-
-commit refs/heads/K
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-redo K
-COMMIT
-
-from refs/heads/branch^1
-
-INPUT_END
-test_expect_success \
-    'K: reinit branch with from' \
-    'git fast-import <input'
-test_expect_success \
-    'K: verify K^1 = branch^1' \
-    'test `git rev-parse --verify branch^1` \
-		= `git rev-parse --verify K^1`'
-
-###
-### series L
-###
-
-cat >input <<INPUT_END
-blob
-mark :1
-data <<EOF
-some data
-EOF
-
-blob
-mark :2
-data <<EOF
-other data
-EOF
-
-commit refs/heads/L
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-create L
-COMMIT
-
-M 644 :1 b.
-M 644 :1 b/other
-M 644 :1 ba
-
-commit refs/heads/L
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-update L
-COMMIT
-
-M 644 :2 b.
-M 644 :2 b/other
-M 644 :2 ba
-INPUT_END
-
-cat >expect <<EXPECT_END
-:100644 100644 4268632... 55d3a52... M	b.
-:040000 040000 0ae5cac... 443c768... M	b
-:100644 100644 4268632... 55d3a52... M	ba
-EXPECT_END
-
-test_expect_success \
-    'L: verify internal tree sorting' \
-	'git fast-import <input &&
-	 git diff-tree --abbrev --raw L^ L >output &&
-	 test_cmp expect output'
-
-###
-### series M
-###
-
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/M1
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-file rename
-COMMIT
-
-from refs/heads/branch^0
-R file2/newf file2/n.e.w.f
-
-INPUT_END
-
-cat >expect <<EOF
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100	file2/newf	file2/n.e.w.f
-EOF
-test_expect_success \
-	'M: rename file in same subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -M -r M1^ M1 >actual &&
-	 compare_diff_raw expect actual'
-
-cat >input <<INPUT_END
-commit refs/heads/M2
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-file rename
-COMMIT
-
-from refs/heads/branch^0
-R file2/newf i/am/new/to/you
-
-INPUT_END
-
-cat >expect <<EOF
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100	file2/newf	i/am/new/to/you
-EOF
-test_expect_success \
-	'M: rename file to new subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -M -r M2^ M2 >actual &&
-	 compare_diff_raw expect actual'
-
-cat >input <<INPUT_END
-commit refs/heads/M3
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-file rename
-COMMIT
-
-from refs/heads/M2^0
-R i other/sub
-
-INPUT_END
-
-cat >expect <<EOF
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100	i/am/new/to/you	other/sub/am/new/to/you
-EOF
-test_expect_success \
-	'M: rename subdirectory to new subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -M -r M3^ M3 >actual &&
-	 compare_diff_raw expect actual'
-
-###
-### series N
-###
-
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/N1
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-file copy
-COMMIT
-
-from refs/heads/branch^0
-C file2/newf file2/n.e.w.f
-
-INPUT_END
-
-cat >expect <<EOF
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file2/n.e.w.f
-EOF
-test_expect_success \
-	'N: copy file in same subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -C --find-copies-harder -r N1^ N1 >actual &&
-	 compare_diff_raw expect actual'
-
-cat >input <<INPUT_END
-commit refs/heads/N2
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-clean directory copy
-COMMIT
-
-from refs/heads/branch^0
-C file2 file3
-
-commit refs/heads/N2
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-modify directory copy
-COMMIT
-
-M 644 inline file3/file5
-data <<EOF
-$file5_data
-EOF
-
-INPUT_END
-
-cat >expect <<EOF
-:100644 100644 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 C100	newdir/interesting	file3/file5
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file3/newf
-:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100	file2/oldf	file3/oldf
-EOF
-test_expect_success \
-	'N: copy then modify subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -C --find-copies-harder -r N2^^ N2 >actual &&
-	 compare_diff_raw expect actual'
-
-cat >input <<INPUT_END
-commit refs/heads/N3
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-dirty directory copy
-COMMIT
-
-from refs/heads/branch^0
-M 644 inline file2/file5
-data <<EOF
-$file5_data
-EOF
-
-C file2 file3
-D file2/file5
-
-INPUT_END
-
-test_expect_success \
-	'N: copy dirty subdirectory' \
-	'git fast-import <input &&
-	 test `git rev-parse N2^{tree}` = `git rev-parse N3^{tree}`'
-
-test_expect_success \
-	'N: copy directory by id' \
-	'cat >expect <<-\EOF &&
-	:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file3/newf
-	:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100	file2/oldf	file3/oldf
+
+test_expect_success 'E: rfc2822 date, --date-format=rfc2822' '
+	cat >expect <<-EOF &&
+	author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> 1170778938 -0500
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1170783302 -0500
+
+	RFC 2822 type date
+	EOF
+	git fast-import --date-format=rfc2822 <input &&
+	verify_packs &&
+	git cat-file commit branch >commit &&
+	sed 1,2d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'setup: F: non-fast-forward update' '
+	test_tick &&
+
+	cat >input <<-INPUT_END
+	commit refs/heads/branch
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	losing things already?
+	COMMIT
+
+	from refs/heads/branch~1
+
+	reset refs/heads/other
+	from refs/heads/branch
+
+	INPUT_END
+
+	old_branch=$(git rev-parse --verify branch^0) &&
+	test_must_fail git fast-import <input &&
+	verify_packs
+'
+
+test_expect_success 'F: non-fast-forward update skips' '
+	echo $old_branch >expect &&
+	git rev-parse --verify branch^0 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'F: verify other commit' '
+	cat >expect <<-EOF &&
+	tree $(git rev-parse branch~1^{tree})
+	parent $(git rev-parse branch~1)
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	losing things already?
+	EOF
+	git cat-file commit other >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'setup: G: forced non-fast-forward update' '
+	test_tick &&
+	cat >input <<-INPUT_END &&
+	commit refs/heads/branch
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	losing things already?
+	COMMIT
+
+	from refs/heads/branch~1
+
+	INPUT_END
+
+	git rev-parse --verify branch^0 >old_branch &&
+	git fast-import --force <input &&
+	verify_packs
+'
+
+test_expect_success 'G: branch changed, but logged' '
+	git rev-parse --verify branch^0 >branch &&
+	git rev-parse --verify branch@{1} >prev &&
+	! test_cmp old_branch branch &&
+	test_cmp old_branch prev
+'
+
+test_expect_success 'setup: H: deleteall, add one' '
+	test_tick &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/H
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	third
+	COMMIT
+
+	from refs/heads/branch^0
+	M 644 inline i-will-die
+	data <<EOF
+	this file will never exist.
+	EOF
+
+	deleteall
+	M 644 inline h/e/l/lo
+	data <<EOF
+	$file5_data
+	EOF
+
+	INPUT_END
+
+	git fast-import <input &&
+	verify_packs &&
+	git whatchanged H
+'
+
+test_expect_success 'H: validate old files removed, new files added' '
+	cat >expect <<-EOF &&
+	:100755 000000 $newf $zeroes D	file2/newf
+	:100644 000000 $oldf $zeroes D	file2/oldf
+	:100755 000000 $file4_id $zeroes D	file4
+	:100644 100644 $file5_id $file5_id R100	newdir/interesting	h/e/l/lo
+	:100755 000000 $file6_id $zeroes D	newdir/exec.sh
+	EOF
+	git diff-tree -M -r H^ H >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'H: verify file' '
+	echo "$file5_data" >expect &&
+	git cat-file blob H:h/e/l/lo >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'I: --export-pack-edges' '
+	echo ".git/objects/pack/pack-.pack: EDGE" >expect &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/export-boundary
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	we have a border.  its only 40 characters wide.
+	COMMIT
+
+	from refs/heads/branch
+
+	INPUT_END
+
+	git fast-import --export-pack-edges=edges.list <input &&
+	tip=$(git rev-parse --verify export-boundary) &&
+	sed -e "
+			s/pack-.*pack/pack-.pack/
+			s/$tip/EDGE/
+		" edges.list >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'J: reset existing branch creates empty commit' '
+	echo SHA >one &&
+	>empty &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/J
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	create J
+	COMMIT
+
+	from refs/heads/branch
+
+	reset refs/heads/J
+
+	commit refs/heads/J
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	initialize J
+	COMMIT
+
+	INPUT_END
+
+	git fast-import <input &&
+	git rev-list J >commitlist &&
+	git ls-tree J >listing &&
+
+	sed -e "s/$_x40/SHA/" commitlist >num-commits &&
+	test_cmp one num-commits &&
+	test_cmp empty listing
+'
+
+test_expect_success 'K: reinit branch with from' '
+	git rev-parse --verify branch^1 >expect &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/K
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	create K
+	COMMIT
+
+	from refs/heads/branch
+
+	commit refs/heads/K
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	redo K
+	COMMIT
+
+	from refs/heads/branch^1
+
+	INPUT_END
+
+	git fast-import <input &&
+	git rev-parse --verify K^1 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'L: trees sort correctly' '
+	some_data=$(echo some data | git hash-object -w --stdin) &&
+	other_data=$(echo other data | git hash-object -w --stdin) &&
+	some_tree=$(
+		rm -f tmp_index &&
+		echo "100644 blob $some_data	other" |
+		GIT_INDEX_FILE=tmp_index git update-index --index-info &&
+		GIT_INDEX_FILE=tmp_index git write-tree
+	) &&
+	other_tree=$(
+		rm -f tmp_index &&
+		echo "100644 blob $other_data	other" |
+		GIT_INDEX_FILE=tmp_index git update-index --index-info &&
+		GIT_INDEX_FILE=tmp_index git write-tree
+	) &&
+	cat >expect <<-EXPECT_END &&
+	:100644 100644 $some_data $other_data M	b.
+	:040000 040000 $some_tree $other_tree M	b
+	:100644 100644 $some_data $other_data M	ba
+	EXPECT_END
+
+	cat >input <<-INPUT_END &&
+	blob
+	mark :1
+	data <<EOF
+	some data
+	EOF
+
+	blob
+	mark :2
+	data <<EOF
+	other data
+	EOF
+
+	commit refs/heads/L
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	create L
+	COMMIT
+
+	M 644 :1 b.
+	M 644 :1 b/other
+	M 644 :1 ba
+
+	commit refs/heads/L
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	update L
+	COMMIT
+
+	M 644 :2 b.
+	M 644 :2 b/other
+	M 644 :2 ba
+	INPUT_END
+	git fast-import <input &&
+	git diff-tree L^ L >output &&
+	test_cmp expect output
+'
+
+test_expect_success 'M: rename file within subdirectory' '
+	test_tick &&
+	cat >expect <<-EOF &&
+	:100755 100755 $newf $newf R100	file2/newf	file2/n.e.w.f
+	EOF
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/M1
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	file rename
+	COMMIT
+
+	from refs/heads/branch^0
+	R file2/newf file2/n.e.w.f
+
+	INPUT_END
+
+	git fast-import <input &&
+	git diff-tree -M -r M1^ M1 >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'M: rename file to new subdirectory (and set up M2)' '
+	cat >expect <<-EOF &&
+	:100755 100755 $newf $newf R100	file2/newf	i/am/new/to/you
+	EOF
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/M2
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	file rename
+	COMMIT
+
+	from refs/heads/branch^0
+	R file2/newf i/am/new/to/you
+
+	INPUT_END
+
+	git fast-import <input &&
+	git diff-tree -M -r M2^ M2 >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'M: rename subdirectory to new subdirectory' '
+	cat >expect <<-EOF &&
+	:100755 100755 $newf $newf R100	i/am/new/to/you	other/sub/am/new/to/you
+	EOF
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/M3
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	file rename
+	COMMIT
+
+	from refs/heads/M2^0
+	R i other/sub
+
+	INPUT_END
+
+	git fast-import <input &&
+	git diff-tree -M -r M3^ M3 >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'N: copy file in same subdirectory' '
+	test_tick &&
+	cat >expect <<-EOF &&
+	:100755 100755 $newf $newf C100	file2/newf	file2/n.e.w.f
+	EOF
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/N1
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	file copy
+	COMMIT
+
+	from refs/heads/branch^0
+	C file2/newf file2/n.e.w.f
+
+	INPUT_END
+
+	git fast-import <input &&
+	git diff-tree -C --find-copies-harder -r N1^ N1 >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'N: copy then modify subdirectory (and set up N2)' '
+	cat >expect <<-EOF &&
+	:100644 100644 $file5_id $file5_id C100	newdir/interesting	file3/file5
+	:100755 100755 $newf $newf C100	file2/newf	file3/newf
+	:100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
+	EOF
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/N2
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	clean directory copy
+	COMMIT
+
+	from refs/heads/branch^0
+	C file2 file3
+
+	commit refs/heads/N2
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	modify directory copy
+	COMMIT
+
+	M 644 inline file3/file5
+	data <<EOF
+	$file5_data
 	EOF
-	 subdir=$(git rev-parse refs/heads/branch^0:file2) &&
-	 cat >input <<-INPUT_END &&
+
+	INPUT_END
+
+	git fast-import <input &&
+	git diff-tree -C --find-copies-harder -r N2^^ N2 >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'N: copy dirty subdirectory (and set up N3)' '
+	git rev-parse N2^{tree} >expect &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/N3
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	dirty directory copy
+	COMMIT
+
+	from refs/heads/branch^0
+	M 644 inline file2/file5
+	data <<EOF
+	$file5_data
+	EOF
+
+	C file2 file3
+	D file2/file5
+
+	INPUT_END
+
+	git fast-import <input &&
+	git rev-parse N3^{tree} >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'N: copy directory by id' '
+	subdir=$(git rev-parse --verify refs/heads/branch^0:file2) &&
+	cat >expect <<-EOF &&
+	:100755 100755 $newf $newf C100	file2/newf	file3/newf
+	:100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
+	EOF
+
+	cat >input <<-INPUT_END &&
 	commit refs/heads/N4
 	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 	data <<COMMIT
@@ -870,19 +900,21 @@ test_expect_success \
 	from refs/heads/branch^0
 	M 040000 $subdir file3
 	INPUT_END
-	 git fast-import <input &&
-	 git diff-tree -C --find-copies-harder -r N4^ N4 >actual &&
-	 compare_diff_raw expect actual'
 
-test_expect_success \
-	'N: modify copied tree' \
-	'cat >expect <<-\EOF &&
-	:100644 100644 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 C100	newdir/interesting	file3/file5
-	:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file3/newf
-	:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100	file2/oldf	file3/oldf
+	git fast-import <input &&
+	git diff-tree -C --find-copies-harder -r N4^ N4 >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'N: modify copied tree' '
+	subdir=$(git rev-parse refs/heads/branch^0:file2) &&
+	cat >expect <<-EOF &&
+	:100644 100644 $file5_id $file5_id C100	newdir/interesting	file3/file5
+	:100755 100755 $newf $newf C100	file2/newf	file3/newf
+	:100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
 	EOF
-	 subdir=$(git rev-parse refs/heads/branch^0:file2) &&
-	 cat >input <<-INPUT_END &&
+
+	cat >input <<-INPUT_END &&
 	commit refs/heads/N5
 	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 	data <<COMMIT
@@ -903,794 +935,813 @@ test_expect_success \
 	$file5_data
 	EOF
 	INPUT_END
-	 git fast-import <input &&
-	 git diff-tree -C --find-copies-harder -r N5^^ N5 >actual &&
-	 compare_diff_raw expect actual'
-
-###
-### series O
-###
-
-cat >input <<INPUT_END
-#we will
-commit refs/heads/O1
-# -- ignore all of this text
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-# $GIT_COMMITTER_NAME has inserted here for his benefit.
-data <<COMMIT
-dirty directory copy
-COMMIT
-
-# don't forget the import blank line!
-#
-# yes, we started from our usual base of branch^0.
-# i like branch^0.
-from refs/heads/branch^0
-# and we need to reuse file2/file5 from N3 above.
-M 644 inline file2/file5
-# otherwise the tree will be different
-data <<EOF
-$file5_data
-EOF
-
-# don't forget to copy file2 to file3
-C file2 file3
-#
-# or to delete file5 from file2.
-D file2/file5
-# are we done yet?
-
-INPUT_END
-
-test_expect_success \
-	'O: comments are all skipped' \
-	'git fast-import <input &&
-	 test `git rev-parse N3` = `git rev-parse O1`'
-
-cat >input <<INPUT_END
-commit refs/heads/O2
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-dirty directory copy
-COMMIT
-from refs/heads/branch^0
-M 644 inline file2/file5
-data <<EOF
-$file5_data
-EOF
-C file2 file3
-D file2/file5
-
-INPUT_END
-
-test_expect_success \
-	'O: blank lines not necessary after data commands' \
-	'git fast-import <input &&
-	 test `git rev-parse N3` = `git rev-parse O2`'
-
-test_expect_success \
-	'O: repack before next test' \
-	'git repack -a -d'
-
-cat >input <<INPUT_END
-commit refs/heads/O3
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-zstring
-COMMIT
-commit refs/heads/O3
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-zof
-COMMIT
-checkpoint
-commit refs/heads/O3
-mark :5
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-zempty
-COMMIT
-checkpoint
-commit refs/heads/O3
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-zcommits
-COMMIT
-reset refs/tags/O3-2nd
-from :5
-reset refs/tags/O3-3rd
-from :5
-INPUT_END
-
-cat >expect <<INPUT_END
-string
-of
-empty
-commits
-INPUT_END
-test_expect_success \
-	'O: blank lines not necessary after other commands' \
-	'git fast-import <input &&
-	 test 8 = `find .git/objects/pack -type f | wc -l` &&
-	 test `git rev-parse refs/tags/O3-2nd` = `git rev-parse O3^` &&
-	 git log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&
-	 test_cmp expect actual'
-
-cat >input <<INPUT_END
-commit refs/heads/O4
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-zstring
-COMMIT
-commit refs/heads/O4
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-zof
-COMMIT
-progress Two commits down, 2 to go!
-commit refs/heads/O4
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-zempty
-COMMIT
-progress Three commits down, 1 to go!
-commit refs/heads/O4
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-zcommits
-COMMIT
-progress I'm done!
-INPUT_END
-test_expect_success \
-	'O: progress outputs as requested by input' \
-	'git fast-import <input >actual &&
-	 grep "progress " <input >expect &&
-	 test_cmp expect actual'
-
-###
-### series P (gitlinks)
-###
-
-cat >input <<INPUT_END
-blob
-mark :1
-data 10
-test file
-
-reset refs/heads/sub
-commit refs/heads/sub
-mark :2
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data 12
-sub_initial
-M 100644 :1 file
-
-blob
-mark :3
-data <<DATAEND
-[submodule "sub"]
-	path = sub
-	url = "`pwd`/sub"
-DATAEND
-
-commit refs/heads/subuse1
-mark :4
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data 8
-initial
-from refs/heads/master
-M 100644 :3 .gitmodules
-M 160000 :2 sub
-
-blob
-mark :5
-data 20
-test file
-more data
-
-commit refs/heads/sub
-mark :6
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data 11
-sub_second
-from :2
-M 100644 :5 file
-
-commit refs/heads/subuse1
-mark :7
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data 7
-second
-from :4
-M 160000 :6 sub
-
-INPUT_END
-
-test_expect_success \
-	'P: supermodule & submodule mix' \
-	'git fast-import <input &&
-	 git checkout subuse1 &&
-	 rm -rf sub && mkdir sub && cd sub &&
-	 git init &&
-	 git fetch --update-head-ok .. refs/heads/sub:refs/heads/master &&
-	 git checkout master &&
-	 cd .. &&
-	 git submodule init &&
-	 git submodule update'
-
-SUBLAST=$(git rev-parse --verify sub)
-SUBPREV=$(git rev-parse --verify sub^)
-
-cat >input <<INPUT_END
-blob
-mark :1
-data <<DATAEND
-[submodule "sub"]
-	path = sub
-	url = "`pwd`/sub"
-DATAEND
-
-commit refs/heads/subuse2
-mark :2
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data 8
-initial
-from refs/heads/master
-M 100644 :1 .gitmodules
-M 160000 $SUBPREV sub
-
-commit refs/heads/subuse2
-mark :3
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data 7
-second
-from :2
-M 160000 $SUBLAST sub
-
-INPUT_END
-
-test_expect_success \
-	'P: verbatim SHA gitlinks' \
-	'git branch -D sub &&
-	 git gc && git prune &&
-	 git fast-import <input &&
-	 test $(git rev-parse --verify subuse2) = $(git rev-parse --verify subuse1)'
-
-test_tick
-cat >input <<INPUT_END
-commit refs/heads/subuse3
-mark :1
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-corrupt
-COMMIT
-
-from refs/heads/subuse2
-M 160000 inline sub
-data <<DATA
-$SUBPREV
-DATA
-
-INPUT_END
+
+	git fast-import <input &&
+	git diff-tree -C --find-copies-harder -r N5^^ N5 >actual &&
+	compare_diff_raw expect actual
+'
+
+test_expect_success 'O: comments are all skipped' '
+	git rev-parse N3 >expect &&
+
+	cat >input <<-INPUT_END &&
+	#we will
+	commit refs/heads/O1
+	# -- ignore all of this text
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	# $GIT_COMMITTER_NAME has inserted here for his benefit.
+	data <<COMMIT
+	dirty directory copy
+	COMMIT
+
+	# don'\''t forget the import blank line!
+	#
+	# yes, we started from our usual base of branch^0.
+	# i like branch^0.
+	from refs/heads/branch^0
+	# and we need to reuse file2/file5 from N3 above.
+	M 644 inline file2/file5
+	# otherwise the tree will be different
+	data <<EOF
+	$file5_data
+	EOF
+
+	# don'\''t forget to copy file2 to file3
+	C file2 file3
+	#
+	# or to delete file5 from file2.
+	D file2/file5
+	# are we done yet?
+
+	INPUT_END
+
+	git fast-import <input &&
+	git rev-parse O1 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'O: blank lines not necessary after data commands' '
+	git rev-parse N3 >expect &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/O2
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	dirty directory copy
+	COMMIT
+	from refs/heads/branch^0
+	M 644 inline file2/file5
+	data <<EOF
+	$file5_data
+	EOF
+	C file2 file3
+	D file2/file5
+
+	INPUT_END
+
+	git fast-import <input &&
+	git rev-parse O2 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'O: blank lines not necessary after other commands' '
+	cat >expect <<-\INPUT_END &&
+	string
+	of
+	empty
+	commits
+	INPUT_END
+
+	git repack -a -d &&
+
+	cat >input <<-INPUT_END &&
+	commit refs/heads/O3
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	zstring
+	COMMIT
+	commit refs/heads/O3
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	zof
+	COMMIT
+	checkpoint
+	commit refs/heads/O3
+	mark :5
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	zempty
+	COMMIT
+	checkpoint
+	commit refs/heads/O3
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	zcommits
+	COMMIT
+	reset refs/tags/O3-2nd
+	from :5
+	reset refs/tags/O3-3rd
+	from :5
+	INPUT_END
+
+	git fast-import <input &&
+	git log --reverse --pretty=oneline O3 >log &&
+	git rev-parse O3^ >o3-parent &&
+	git rev-parse refs/tags/O3-2nd >o3-2nd &&
+
+	sed s/^.*z// log >actual &&
+	test_cmp expect actual &&
+	test 8 = `find .git/objects/pack -type f | wc -l` &&
+	test_cmp o3-2nd o3-parent
+'
+
+test_expect_success 'O: progress outputs as requested by input' '
+	cat >input <<-INPUT_END &&
+	commit refs/heads/O4
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	zstring
+	COMMIT
+	commit refs/heads/O4
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	zof
+	COMMIT
+	progress Two commits down, 2 to go!
+	commit refs/heads/O4
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	zempty
+	COMMIT
+	progress Three commits down, 1 to go!
+	commit refs/heads/O4
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	zcommits
+	COMMIT
+	progress I'\''m done!
+	INPUT_END
+	grep "progress " <input >expect &&
+
+	git fast-import <input >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'setup: P: supermodule & submodule mix' '
+	cat >input <<-INPUT_END &&
+	blob
+	mark :1
+	data 10
+	test file
+
+	reset refs/heads/sub
+	commit refs/heads/sub
+	mark :2
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data 12
+	sub_initial
+	M 100644 :1 file
+
+	blob
+	mark :3
+	data <<DATAEND
+	[submodule "sub"]
+		path = sub
+		url = "$(pwd)/sub"
+	DATAEND
+
+	commit refs/heads/subuse1
+	mark :4
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data 8
+	initial
+	from refs/heads/master
+	M 100644 :3 .gitmodules
+	M 160000 :2 sub
+
+	blob
+	mark :5
+	data 20
+	test file
+	more data
+
+	commit refs/heads/sub
+	mark :6
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data 11
+	sub_second
+	from :2
+	M 100644 :5 file
+
+	commit refs/heads/subuse1
+	mark :7
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data 7
+	second
+	from :4
+	M 160000 :6 sub
+
+	INPUT_END
+
+	git fast-import <input &&
+	git checkout subuse1 &&
+	rm -rf sub &&
+	mkdir sub &&
+	(
+		cd sub &&
+		git init &&
+		git fetch --update-head-ok .. refs/heads/sub:refs/heads/master &&
+		git checkout master
+	) &&
+	git submodule init &&
+	git submodule update
+'
+
+test_expect_success 'P: verbatim SHA gitlinks (and set up subuse2)' '
+	SUBLAST=$(git rev-parse --verify sub) &&
+	SUBPREV=$(git rev-parse --verify sub^) &&
+	git rev-parse --verify subuse1 >expect &&
+
+	cat >input <<-INPUT_END &&
+	blob
+	mark :1
+	data <<DATAEND
+	[submodule "sub"]
+		path = sub
+		url = "$(pwd)/sub"
+	DATAEND
+
+	commit refs/heads/subuse2
+	mark :2
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data 8
+	initial
+	from refs/heads/master
+	M 100644 :1 .gitmodules
+	M 160000 $SUBPREV sub
+
+	commit refs/heads/subuse2
+	mark :3
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data 7
+	second
+	from :2
+	M 160000 $SUBLAST sub
+
+	INPUT_END
+
+	git branch -D sub &&
+	git gc && git prune &&
+	git fast-import <input &&
+	git rev-parse --verify subuse2 >actual &&
+	test_cmp expect actual
+'
 
 test_expect_success 'P: fail on inline gitlink' '
-    test_must_fail git fast-import <input'
+	test_tick &&
 
-test_tick
-cat >input <<INPUT_END
-blob
-mark :1
-data <<DATA
-$SUBPREV
-DATA
+	cat >input <<-INPUT_END &&
+	commit refs/heads/subuse3
+	mark :1
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	corrupt
+	COMMIT
 
-commit refs/heads/subuse3
-mark :2
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-corrupt
-COMMIT
+	from refs/heads/subuse2
+	M 160000 inline sub
+	data <<DATA
+	$SUBPREV
+	DATA
 
-from refs/heads/subuse2
-M 160000 :1 sub
+	INPUT_END
 
-INPUT_END
+	test_must_fail git fast-import <input
+'
 
 test_expect_success 'P: fail on blob mark in gitlink' '
-    test_must_fail git fast-import <input'
-
-###
-### series Q (notes)
-###
-
-note1_data="The first note for the first commit"
-note2_data="The first note for the second commit"
-note3_data="The first note for the third commit"
-note1b_data="The second note for the first commit"
-note1c_data="The third note for the first commit"
-note2b_data="The second note for the second commit"
-
-test_tick
-cat >input <<INPUT_END
-blob
-mark :2
-data <<EOF
-$file2_data
-EOF
-
-commit refs/heads/notes-test
-mark :3
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-first (:3)
-COMMIT
-
-M 644 :2 file2
-
-blob
-mark :4
-data $file4_len
-$file4_data
-commit refs/heads/notes-test
-mark :5
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-second (:5)
-COMMIT
-
-M 644 :4 file4
-
-commit refs/heads/notes-test
-mark :6
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-third (:6)
-COMMIT
-
-M 644 inline file5
-data <<EOF
-$file5_data
-EOF
-
-M 755 inline file6
-data <<EOF
-$file6_data
-EOF
-
-blob
-mark :7
-data <<EOF
-$note1_data
-EOF
-
-blob
-mark :8
-data <<EOF
-$note2_data
-EOF
-
-commit refs/notes/foobar
-mark :9
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-notes (:9)
-COMMIT
-
-N :7 :3
-N :8 :5
-N inline :6
-data <<EOF
-$note3_data
-EOF
-
-commit refs/notes/foobar
-mark :10
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-notes (:10)
-COMMIT
-
-N inline :3
-data <<EOF
-$note1b_data
-EOF
-
-commit refs/notes/foobar2
-mark :11
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-notes (:11)
-COMMIT
-
-N inline :3
-data <<EOF
-$note1c_data
-EOF
-
-commit refs/notes/foobar
-mark :12
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-notes (:12)
-COMMIT
-
-deleteall
-N inline :5
-data <<EOF
-$note2b_data
-EOF
-
-INPUT_END
-
-test_expect_success \
-	'Q: commit notes' \
-	'git fast-import <input &&
-	 git whatchanged notes-test'
-test_expect_success \
-	'Q: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
-
-commit1=$(git rev-parse notes-test~2)
-commit2=$(git rev-parse notes-test^)
-commit3=$(git rev-parse notes-test)
-
-cat >expect <<EOF
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-first (:3)
-EOF
-test_expect_success \
-	'Q: verify first commit' \
-	'git cat-file commit notes-test~2 | sed 1d >actual &&
-	test_cmp expect actual'
-
-cat >expect <<EOF
-parent $commit1
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-second (:5)
-EOF
-test_expect_success \
-	'Q: verify second commit' \
-	'git cat-file commit notes-test^ | sed 1d >actual &&
-	test_cmp expect actual'
-
-cat >expect <<EOF
-parent $commit2
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-third (:6)
-EOF
-test_expect_success \
-	'Q: verify third commit' \
-	'git cat-file commit notes-test | sed 1d >actual &&
-	test_cmp expect actual'
-
-cat >expect <<EOF
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-notes (:9)
-EOF
-test_expect_success \
-	'Q: verify first notes commit' \
-	'git cat-file commit refs/notes/foobar~2 | sed 1d >actual &&
-	test_cmp expect actual'
-
-cat >expect.unsorted <<EOF
-100644 blob $commit1
-100644 blob $commit2
-100644 blob $commit3
-EOF
-cat expect.unsorted | sort >expect
-test_expect_success \
-	'Q: verify first notes tree' \
-	'git cat-file -p refs/notes/foobar~2^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
-
-echo "$note1_data" >expect
-test_expect_success \
-	'Q: verify first note for first commit' \
-	'git cat-file blob refs/notes/foobar~2:$commit1 >actual && test_cmp expect actual'
-
-echo "$note2_data" >expect
-test_expect_success \
-	'Q: verify first note for second commit' \
-	'git cat-file blob refs/notes/foobar~2:$commit2 >actual && test_cmp expect actual'
-
-echo "$note3_data" >expect
-test_expect_success \
-	'Q: verify first note for third commit' \
-	'git cat-file blob refs/notes/foobar~2:$commit3 >actual && test_cmp expect actual'
-
-cat >expect <<EOF
-parent `git rev-parse --verify refs/notes/foobar~2`
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-notes (:10)
-EOF
-test_expect_success \
-	'Q: verify second notes commit' \
-	'git cat-file commit refs/notes/foobar^ | sed 1d >actual &&
-	test_cmp expect actual'
-
-cat >expect.unsorted <<EOF
-100644 blob $commit1
-100644 blob $commit2
-100644 blob $commit3
-EOF
-cat expect.unsorted | sort >expect
-test_expect_success \
-	'Q: verify second notes tree' \
-	'git cat-file -p refs/notes/foobar^^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
-
-echo "$note1b_data" >expect
-test_expect_success \
-	'Q: verify second note for first commit' \
-	'git cat-file blob refs/notes/foobar^:$commit1 >actual && test_cmp expect actual'
-
-echo "$note2_data" >expect
-test_expect_success \
-	'Q: verify first note for second commit' \
-	'git cat-file blob refs/notes/foobar^:$commit2 >actual && test_cmp expect actual'
-
-echo "$note3_data" >expect
-test_expect_success \
-	'Q: verify first note for third commit' \
-	'git cat-file blob refs/notes/foobar^:$commit3 >actual && test_cmp expect actual'
-
-cat >expect <<EOF
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-notes (:11)
-EOF
-test_expect_success \
-	'Q: verify third notes commit' \
-	'git cat-file commit refs/notes/foobar2 | sed 1d >actual &&
-	test_cmp expect actual'
-
-cat >expect.unsorted <<EOF
-100644 blob $commit1
-EOF
-cat expect.unsorted | sort >expect
-test_expect_success \
-	'Q: verify third notes tree' \
-	'git cat-file -p refs/notes/foobar2^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
-
-echo "$note1c_data" >expect
-test_expect_success \
-	'Q: verify third note for first commit' \
-	'git cat-file blob refs/notes/foobar2:$commit1 >actual && test_cmp expect actual'
-
-cat >expect <<EOF
-parent `git rev-parse --verify refs/notes/foobar^`
-author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-
-notes (:12)
-EOF
-test_expect_success \
-	'Q: verify fourth notes commit' \
-	'git cat-file commit refs/notes/foobar | sed 1d >actual &&
-	test_cmp expect actual'
-
-cat >expect.unsorted <<EOF
-100644 blob $commit2
-EOF
-cat expect.unsorted | sort >expect
-test_expect_success \
-	'Q: verify fourth notes tree' \
-	'git cat-file -p refs/notes/foobar^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
-
-echo "$note2b_data" >expect
-test_expect_success \
-	'Q: verify second note for second commit' \
-	'git cat-file blob refs/notes/foobar:$commit2 >actual && test_cmp expect actual'
-
-###
-### series R (feature and option)
-###
-
-cat >input <<EOF
-feature no-such-feature-exists
-EOF
+	test_tick &&
+
+	cat >input <<-INPUT_END &&
+	blob
+	mark :1
+	data <<DATA
+	$SUBPREV
+	DATA
+
+	commit refs/heads/subuse3
+	mark :2
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	corrupt
+	COMMIT
+
+	from refs/heads/subuse2
+	M 160000 :1 sub
+
+	INPUT_END
+
+	test_must_fail git fast-import <input
+'
+
+test_expect_success 'setup: series Q (notes)' '
+	test_tick &&
+
+	note1_data="The first note for the first commit" &&
+	note2_data="The first note for the second commit" &&
+	note3_data="The first note for the third commit" &&
+	note1b_data="The second note for the first commit" &&
+	note1c_data="The third note for the first commit" &&
+	note2b_data="The second note for the second commit" &&
+
+	cat >input <<-INPUT_END &&
+	blob
+	mark :2
+	data <<EOF
+	$file2_data
+	EOF
+
+	commit refs/heads/notes-test
+	mark :3
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	first (:3)
+	COMMIT
+
+	M 644 :2 file2
+
+	blob
+	mark :4
+	data $file4_len
+	$file4_data
+	commit refs/heads/notes-test
+	mark :5
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	second (:5)
+	COMMIT
+
+	M 644 :4 file4
+
+	commit refs/heads/notes-test
+	mark :6
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	third (:6)
+	COMMIT
+
+	M 644 inline file5
+	data <<EOF
+	$file5_data
+	EOF
+
+	M 755 inline file6
+	data <<EOF
+	$file6_data
+	EOF
+
+	blob
+	mark :7
+	data <<EOF
+	$note1_data
+	EOF
+
+	blob
+	mark :8
+	data <<EOF
+	$note2_data
+	EOF
+
+	commit refs/notes/foobar
+	mark :9
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	notes (:9)
+	COMMIT
+
+	N :7 :3
+	N :8 :5
+	N inline :6
+	data <<EOF
+	$note3_data
+	EOF
+
+	commit refs/notes/foobar
+	mark :10
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	notes (:10)
+	COMMIT
+
+	N inline :3
+	data <<EOF
+	$note1b_data
+	EOF
+
+	commit refs/notes/foobar2
+	mark :11
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	notes (:11)
+	COMMIT
+
+	N inline :3
+	data <<EOF
+	$note1c_data
+	EOF
+
+	commit refs/notes/foobar
+	mark :12
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	notes (:12)
+	COMMIT
+
+	deleteall
+	N inline :5
+	data <<EOF
+	$note2b_data
+	EOF
+
+	INPUT_END
+
+	git fast-import <input &&
+	verify_packs &&
+	git whatchanged notes-test
+'
+
+test_expect_success 'Q: verify first commit' '
+	cat >expect <<-EOF &&
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	first (:3)
+	EOF
+
+	git cat-file commit notes-test~2 >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify second commit' '
+	commit1=$(git rev-parse notes-test~2) &&
+	cat >expect <<-EOF
+	parent $commit1
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	second (:5)
+	EOF
+	git cat-file commit notes-test^ >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify third commit' '
+	commit2=$(git rev-parse notes-test^) &&
+	cat >expect <<-EOF &&
+	parent $commit2
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	third (:6)
+	EOF
+	git cat-file commit notes-test >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify first notes commit' '
+	cat >expect <<-EOF &&
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	notes (:9)
+	EOF
+	git cat-file commit refs/notes/foobar~2 >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify first notes tree' '
+	commit1=$(git rev-parse notes-test~2) &&
+	commit2=$(git rev-parse notes-test^) &&
+	commit3=$(git rev-parse notes-test) &&
+	sort >expect <<-EOF &&
+	100644 blob $commit1
+	100644 blob $commit2
+	100644 blob $commit3
+	EOF
+
+	git cat-file -p refs/notes/foobar~2^{tree} >tree &&
+	sed "s/ [0-9a-f]*	/ /" <tree >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify first note for first commit' '
+	echo "$note1_data" >expect &&
+	git cat-file blob refs/notes/foobar~2:$commit1 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify first note for second commit' '
+	echo "$note2_data" >expect &&
+	git cat-file blob refs/notes/foobar~2:$commit2 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify first note for third commit' '
+	echo "$note3_data" >expect &&
+	git cat-file blob refs/notes/foobar~2:$commit3 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify second notes commit' '
+	cat >expect <<-EOF &&
+	parent `git rev-parse --verify refs/notes/foobar~2`
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	notes (:10)
+	EOF
+	git cat-file commit refs/notes/foobar^ >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify second notes tree' '
+	sort >expect <<-EOF
+	100644 blob $commit1
+	100644 blob $commit2
+	100644 blob $commit3
+	EOF
+	git cat-file -p refs/notes/foobar^^{tree} >tree &&
+	sed "s/ [0-9a-f]*	/ /" <tree >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify second note for first commit' '
+	echo "$note1b_data" >expect &&
+	git cat-file blob refs/notes/foobar^:$commit1 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify first note for second commit' '
+	echo "$note2_data" >expect &&
+	git cat-file blob refs/notes/foobar^:$commit2 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify first note for third commit' '
+	echo "$note3_data" >expect &&
+	git cat-file blob refs/notes/foobar^:$commit3 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify third notes commit' '
+	cat >expect <<-EOF &&
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	notes (:11)
+	EOF
+	git cat-file commit refs/notes/foobar2 >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify third notes tree' '
+	echo "100644 blob $commit1" >expect &&
+	git cat-file -p refs/notes/foobar2^{tree} >tree &&
+	sed "s/ [0-9a-f]*	/ /" <tree >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify third note for first commit' '
+	echo "$note1c_data" >expect
+	git cat-file blob refs/notes/foobar2:$commit1 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify fourth notes commit' '
+	parent=$(git rev-parse --verify refs/notes/foobar^) &&
+	cat >expect <<-EOF &&
+	parent $parent
+	author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+
+	notes (:12)
+	EOF
+	git cat-file commit refs/notes/foobar >commit &&
+	sed 1d <commit >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify fourth notes tree' '
+	echo "100644 blob $commit2" >expect &&
+	git cat-file -p refs/notes/foobar^{tree} >tree &&
+	sed "s/ [0-9a-f]*	/ /" <tree >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'Q: verify second note for second commit' '
+	echo "$note2b_data" >expect
+	git cat-file blob refs/notes/foobar:$commit2 >actual &&
+	test_cmp expect actual
+'
 
 test_expect_success 'R: abort on unsupported feature' '
+	cat >input <<-EOF &&
+	feature no-such-feature-exists
+	EOF
+
 	test_must_fail git fast-import <input
 '
 
-cat >input <<EOF
-feature date-format=now
-EOF
-
 test_expect_success 'R: supported feature is accepted' '
+	cat >input <<-EOF &&
+	feature date-format=now
+	EOF
+
 	git fast-import <input
 '
 
-cat >input << EOF
-blob
-data 3
-hi
-feature date-format=now
-EOF
-
 test_expect_success 'R: abort on receiving feature after data command' '
+	cat >input <<-EOF &&
+	blob
+	data 3
+	hi
+	feature date-format=now
+	EOF
+
 	test_must_fail git fast-import <input
 '
 
-cat >input << EOF
-feature import-marks=git.marks
-feature import-marks=git2.marks
-EOF
-
 test_expect_success 'R: only one import-marks feature allowed per stream' '
+	cat >input <<-\EOF &&
+	feature import-marks=git.marks
+	feature import-marks=git2.marks
+	EOF
+
 	test_must_fail git fast-import <input
 '
 
-cat >input << EOF
-feature export-marks=git.marks
-blob
-mark :1
-data 3
-hi
-
-EOF
-
-test_expect_success \
-    'R: export-marks feature results in a marks file being created' \
-    'cat input | git fast-import &&
-    grep :1 git.marks'
-
-test_expect_success \
-    'R: export-marks options can be overriden by commandline options' \
-    'cat input | git fast-import --export-marks=other.marks &&
-    grep :1 other.marks'
-
-cat >input << EOF
-feature import-marks=marks.out
-feature export-marks=marks.new
-EOF
-
-test_expect_success \
-    'R: import to output marks works without any content' \
-    'cat input | git fast-import &&
-    test_cmp marks.out marks.new'
-
-cat >input <<EOF
-feature import-marks=nonexistant.marks
-feature export-marks=marks.new
-EOF
+test_expect_success 'setup: R: stream using export-marks feature' '
+	cat >input <<-\EOF
+	feature export-marks=git.marks
+	blob
+	mark :1
+	data 3
+	hi
+
+	EOF
+'
 
-test_expect_success \
-    'R: import marks prefers commandline marks file over the stream' \
-    'cat input | git fast-import --import-marks=marks.out &&
-    test_cmp marks.out marks.new'
+test_expect_success 'R: export-marks feature results in a marks file being created' '
+	git fast-import <input &&
+	grep :1 git.marks
+'
 
+test_expect_success 'R: export-marks options can be overriden by commandline options' '
+	git fast-import --export-marks=other.marks <input &&
+	grep :1 other.marks
+'
 
-cat >input <<EOF
-feature import-marks=nonexistant.marks
-feature export-marks=combined.marks
-EOF
+test_expect_success 'R: import to output marks works without any content' '
+	cat >input <<-\EOF &&
+	feature import-marks=marks.out
+	feature export-marks=marks.new
+	EOF
 
-test_expect_success 'R: multiple --import-marks= should be honoured' '
-    head -n2 marks.out > one.marks &&
-    tail -n +3 marks.out > two.marks &&
-    git fast-import --import-marks=one.marks --import-marks=two.marks <input &&
-    test_cmp marks.out combined.marks
+	git fast-import <input &&
+	test_cmp marks.out marks.new
 '
 
-cat >input <<EOF
-feature relative-marks
-feature import-marks=relative.in
-feature export-marks=relative.out
-EOF
-
-test_expect_success 'R: feature relative-marks should be honoured' '
-    mkdir -p .git/info/fast-import/ &&
-    cp marks.new .git/info/fast-import/relative.in &&
-    git fast-import <input &&
-    test_cmp marks.new .git/info/fast-import/relative.out
+test_expect_success 'R: import marks prefers commandline marks file over the stream' '
+	cat >input <<-EOF &&
+	feature import-marks=nonexistant.marks
+	feature export-marks=marks.new
+	EOF
+	git fast-import --import-marks=marks.out <input &&
+	test_cmp marks.out marks.new
 '
 
-cat >input <<EOF
-feature relative-marks
-feature import-marks=relative.in
-feature no-relative-marks
-feature export-marks=non-relative.out
-EOF
+test_expect_success 'R: multiple --import-marks= are honoured' '
+	cat >input <<-EOF &&
+	feature import-marks=nonexistant.marks
+	feature export-marks=combined.marks
+	EOF
 
-test_expect_success 'R: feature no-relative-marks should be honoured' '
-    git fast-import <input &&
-    test_cmp marks.new non-relative.out
+	head -n2 marks.out >one.marks &&
+	tail -n +3 marks.out >two.marks &&
+	git fast-import --import-marks=one.marks --import-marks=two.marks <input &&
+	test_cmp marks.out combined.marks
 '
 
-cat >input << EOF
-option git quiet
-blob
-data 3
-hi
+test_expect_success 'R: feature relative-marks is honoured' '
+	cat >input <<-\EOF &&
+	feature relative-marks
+	feature import-marks=relative.in
+	feature export-marks=relative.out
+	EOF
+
+	mkdir -p .git/info/fast-import/ &&
+	cp marks.out .git/info/fast-import/relative.in &&
+	git fast-import <input &&
+	test_cmp marks.out .git/info/fast-import/relative.out
+'
 
-EOF
+test_expect_success 'R: feature no-relative-marks is honoured' '
+	cat >input <<-\EOF &&
+	feature relative-marks
+	feature import-marks=relative.in
+	feature no-relative-marks
+	feature export-marks=non-relative.out
+	EOF
 
-touch empty
+	git fast-import <input &&
+	test_cmp marks.out non-relative.out
+'
 
 test_expect_success 'R: quiet option results in no stats being output' '
-    cat input | git fast-import 2> output &&
-    test_cmp empty output
+	>empty &&
+	cat >input <<-\EOF &&
+	option git quiet
+	blob
+	data 3
+	hi
+
+	EOF
+
+	git fast-import <input 2>output &&
+	test_cmp empty output
 '
 
-cat >input <<EOF
-option git non-existing-option
-EOF
-
 test_expect_success 'R: die on unknown option' '
-    test_must_fail git fast-import <input
+	cat >input <<-\EOF &&
+	option git non-existing-option
+	EOF
+
+	test_must_fail git fast-import <input
 '
 
 test_expect_success 'R: unknown commandline options are rejected' '\
-    test_must_fail git fast-import --non-existing-option < /dev/null
+	test_must_fail git fast-import --non-existing-option </dev/null
 '
 
-cat >input <<EOF
-option non-existing-vcs non-existing-option
-EOF
-
 test_expect_success 'R: ignore non-git options' '
-    git fast-import <input
+	cat >input <<-\EOF &&
+	option non-existing-vcs non-existing-option
+	EOF
+
+	git fast-import <input
 '
 
-##
-## R: very large blobs
-##
-blobsize=$((2*1024*1024 + 53))
-test-genrandom bar $blobsize >expect
-cat >input <<INPUT_END
-commit refs/heads/big-file
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-R - big file
-COMMIT
-
-M 644 inline big1
-data $blobsize
-INPUT_END
-cat expect >>input
-cat >>input <<INPUT_END
-M 644 inline big2
-data $blobsize
-INPUT_END
-cat expect >>input
-echo >>input
-
-test_expect_success \
-	'R: blob bigger than threshold' \
-	'test_create_repo R &&
-	 git --git-dir=R/.git fast-import --big-file-threshold=1 <input'
-test_expect_success \
-	'R: verify created pack' \
-	': >verify &&
-	 for p in R/.git/objects/pack/*.pack;
-	 do
-	   git verify-pack -v $p >>verify || exit;
-	 done'
-test_expect_success \
-	'R: verify written objects' \
-	'git --git-dir=R/.git cat-file blob big-file:big1 >actual &&
-	 test_cmp expect actual &&
-	 a=$(git --git-dir=R/.git rev-parse big-file:big1) &&
-	 b=$(git --git-dir=R/.git rev-parse big-file:big2) &&
-	 test $a = $b'
-test_expect_success \
-	'R: blob appears only once' \
-	'n=$(grep $a verify | wc -l) &&
-	 test 1 = $n'
+test_expect_success 'R: blob bigger than threshold' '
+	blobsize=$((2*1024*1024 + 53)) &&
+	test-genrandom bar $blobsize >expect &&
+	echo ONE | wc -l >expect.count &&
+
+	{
+		cat <<-INPUT_END &&
+		commit refs/heads/big-file
+		committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+		data <<COMMIT
+		R - big file
+		COMMIT
+
+		M 644 inline big1
+		data $blobsize
+		INPUT_END
+
+		cat expect &&
+		cat <<-INPUT_END &&
+		M 644 inline big2
+		data $blobsize
+		INPUT_END
+
+		cat expect &&
+		echo
+	} >input &&
+
+	test_create_repo R &&
+	git --git-dir=R/.git fast-import --big-file-threshold=1 <input &&
+	(
+		for p in R/.git/objects/pack/*.pack
+		do
+			git verify-pack -v $p ||
+			exit
+		done
+	) >verify &&
+	git --git-dir=R/.git cat-file blob big-file:big1 >actual &&
+	git --git-dir=R/.git rev-parse big-file:big1 >a &&
+	git --git-dir=R/.git rev-parse big-file:big2 >b &&
+	grep $(cat a) verify | wc -l >count &&
+
+	test_cmp expect actual &&
+	test_cmp a b &&
+	# blob only appears once
+	test_cmp expect.count count
+'
 
 test_done
-- 
1.7.2.3

[PATCH 2/3] Teach fast-import to print the id of each imported commit

[Jonathan Nieder]

For the svn importer, it would be useful to build a map from
subversion revision numbers to git commits as the import takes place.
This is particularly relevant because the subversion api sometimes
represents as "copy this directory from this revision", and the
importer needs to be able to access the corresponding trees.  So
(optionally) print each commit id when the corresponding object is
written.

Unfortunately when each commit object is written, it is not yet
accessible to the caller until a checkpoint has finished.  A later
patch will teach fast-import to directly pass on the relevant data on
request, using the same channel.

Cc: Shawn O. Pearce <spearce@spearce.org>
Cc: Sverre Rabbelier <srabbelier@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
I am not happy with the interface:

. the name --report-fd is a bit silly.  Maybe next round will
  have a better one.
. some frontends may want to use "cat" without the commit id noise
. it is easy to trigger SIGPIPE by ignoring the last couple of
  commit ids
. communication needs to be serialized.  What will happen if a commit
  gets finished while a "cat" command is being fulfilled?

 Documentation/git-fast-import.txt |   13 +++++++++++
 fast-import.c                     |   18 +++++++++++++++
 t/t9300-fast-import.sh            |   42 +++++++++++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 966ba4f..e217635 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -92,6 +92,18 @@ OPTIONS
 	--(no-)-relative-marks= with the --(import|export)-marks=
 	options.
 
+--report-fd=<fd>::
+	Print the 40-character object name for each commit to
+	the specified file descriptor before writing it to the
+	pack.  This information may be useful if the importer
+	needs to maintain a map from revisions in the source
+	repository to commit ids in the target repository
+	during the import.
++
+The described objects are not necessarily accessible
+using standard git plumbing tools until a little while
+after the next checkpoint.
+
 --export-pack-edges=<file>::
 	After creating a packfile, print a line of data to
 	<file> listing the filename of the packfile and the last
@@ -896,6 +908,7 @@ The following features are currently supported:
 * date-format
 * import-marks
 * export-marks
+* report-fd
 * relative-marks
 * no-relative-marks
 * force
diff --git a/fast-import.c b/fast-import.c
index 2317b0f..ef0cee7 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -361,6 +361,9 @@ static uintmax_t next_mark;
 static struct strbuf new_data = STRBUF_INIT;
 static int seen_data_command;
 
+/* Where to report commits */
+static int report_fd = -1;
+
 static void parse_argv(void);
 
 static void write_branch_report(FILE *rpt, struct branch *b)
@@ -2571,6 +2574,11 @@ static void parse_new_commit(void)
 
 	if (!store_object(OBJ_COMMIT, &new_data, NULL, b->sha1, next_mark))
 		b->pack_id = pack_id;
+	if (report_fd != -1) {
+		char *buf = sha1_to_hex(b->sha1);
+		buf[40] = '\n';
+		write_or_die(report_fd, buf, 41);
+	}
 	b->last_commit = object_count_by_type[OBJ_COMMIT];
 }
 
@@ -2755,6 +2763,14 @@ static void option_export_marks(const char *marks)
 	safe_create_leading_directories_const(export_marks_file);
 }
 
+static void option_report_fd(const char *fd)
+{
+	unsigned long n = strtoul(fd, NULL, 0);
+	if (n > (unsigned long) INT_MAX)
+		die("--report-fd cannot exceed %d", INT_MAX);
+	report_fd = (int) n;
+}
+
 static void option_export_pack_edges(const char *edges)
 {
 	if (pack_edges)
@@ -2808,6 +2824,8 @@ static int parse_one_feature(const char *feature, int from_stream)
 		option_import_marks(feature + 13, from_stream);
 	} else if (!prefixcmp(feature, "export-marks=")) {
 		option_export_marks(feature + 13);
+	} else if (!prefixcmp(feature, "report-fd=")) {
+		option_report_fd(feature + strlen("report-fd="));
 	} else if (!prefixcmp(feature, "relative-marks")) {
 		relative_marks_paths = 1;
 	} else if (!prefixcmp(feature, "no-relative-marks")) {
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 5c274e7..610e7a5 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1663,6 +1663,48 @@ test_expect_success 'R: feature no-relative-marks is honoured' '
 	test_cmp marks.out non-relative.out
 '
 
+test_expect_success 'have pipes?' '
+	test_when_finished "rm -f frob" &&
+	if mkfifo frob
+	then
+		test_set_prereq PIPE
+	fi
+'
+
+test_expect_success PIPE 'R: feature report-fd is honoured' '
+	mkfifo commits &&
+	test_when_finished "rm -f commits" &&
+	cat >frontend <<-\FRONTEND_END &&
+		#!/bin/sh
+		cat <<EOF &&
+		feature report-fd=3
+		commit refs/heads/printed
+		committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+		data <<COMMIT
+		to be printed
+		COMMIT
+
+		from refs/heads/master
+		D file3
+
+		EOF
+
+		read cid <&3 &&
+		echo "$cid" >received
+		EOF
+	FRONTEND_END
+	chmod +x frontend &&
+	(
+		{
+			sh frontend 3<commits ||
+			exit
+		} |
+		git fast-import 3>commits
+	) &&
+	git rev-parse --verify printed >real &&
+	test_cmp real received
+'
+
 test_expect_success 'R: quiet option results in no stats being output' '
 	>empty &&
 	cat >input <<-\EOF &&
-- 
1.7.2.3

[PATCH 3/3] fast-import: Let importers retrieve the objects being written

[Jonathan Nieder]

As the description of the "progress" option in git-fast-import.1
hints, there is no convenient way to immediately access the objects
written to a new repository through fast-import.  Until a checkpoint
has been started and finishes writing the pack index, any new blobs,
trees, and commits will not be accessible using standard git tools.

So introduce another way: a "cat" command introduced in the command
stream requests for fast-import to print an object to the same
report-fd stream used to report commits being written.

The output uses the same format as "git cat-file --batch".

Like cat-file --batch, this does not provide an option to dereference
objects to a type of the requestor's choosing.  Tags are presented
as tags, commits as commits, and trees as trees.

Objects can be specified by path within a tree as well, using a

 cat TREE "PATH"

syntax.  With this syntax, also, the tree can only be specified by
:n marker or 40-digit tree id.

Cc: Shawn O. Pearce <spearce@spearce.org>
Cc: David Barr <david.barr@cordelta.com>
Cc: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
That's the end of the series.  As with patch 2, I'm not thrilled
with the interface, but I hope it can get the job done for now.

 Documentation/git-fast-import.txt |   34 +++++++++++-
 fast-import.c                     |  108 +++++++++++++++++++++++++++++++++++
 t/t9300-fast-import.sh            |  114 +++++++++++++++++++++++++++++++++++++
 3 files changed, 255 insertions(+), 1 deletions(-)

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index e217635..2cf48f5 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -102,7 +102,8 @@ OPTIONS
 +
 The described objects are not necessarily accessible
 using standard git plumbing tools until a little while
-after the next checkpoint.
+after the next checkpoint.  To request access to the
+objects before then, use `cat` lines in the command stream.
 
 --export-pack-edges=<file>::
 	After creating a packfile, print a line of data to
@@ -332,6 +333,10 @@ and control the current import process.  More detailed discussion
 	standard output.  This command is optional and is not needed
 	to perform an import.
 
+`cat`::
+	Causes fast-import to print an object in 'cat-file --batch'
+	format to the file descriptor set with "feature report-fd".
+
 `feature`::
 	Require that fast-import supports the specified feature, or
 	abort if it does not.
@@ -888,6 +893,33 @@ Placing a `progress` command immediately after a `checkpoint` will
 inform the reader when the `checkpoint` has been completed and it
 can safely access the refs that fast-import updated.
 
+`cat`
+~~~~~
+Causes fast-import to print an object to a file descriptor
+previously arranged with the `--report-fd` option.  The command
+otherwise has no impact on the current import; its main purpose is to
+retrieve objects that may be in fast-import's memory but not
+accessible from the target repository a little quicker than by the
+method suggested by the description of the `progress` option.
+
+....
+	'cat' SP <dataref> LF
+....
+
+The `<dataref>` can be either a mark reference (`:<idnum>`)
+set previously, or a full 40-byte SHA-1 of any Git object,
+preexisting or ready to be written.
+
+If `<dataref>` refers to a tree object, it may be followed by
+a path within that tree to retrieve a subtree or blob.
+
+....
+	'cat' SP <treeref> SP <path> LF
+....
+
+A `<path>` string should be surrounded with quotation marks and
+use C-style escaping.
+
 `feature`
 ~~~~~~~~~
 Require that fast-import supports the specified feature, or abort if
diff --git a/fast-import.c b/fast-import.c
index ef0cee7..b7fa9ae 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -55,6 +55,9 @@ Format of STDIN stream:
     ('from' sp committish lf)?
     lf?;
 
+  cat_request ::= 'cat' sp (hexsha1 | idnum) lf
+    | 'cat' sp hexsha1 sp path_str lf;
+
   checkpoint ::= 'checkpoint' lf
     lf?;
 
@@ -2688,6 +2691,109 @@ static void parse_reset_branch(void)
 		unread_command_buf = 1;
 }
 
+static void quoted_path_sha1(unsigned char sha1[20], struct tree_entry *root,
+				const char *path, const char *line)
+{
+	struct strbuf uq = STRBUF_INIT;
+	struct tree_entry leaf = {0};
+	const char *x;
+
+	if (unquote_c_style(&uq, path, &x))
+		die("Invalid path: %s", line);
+	if (*x)
+		die("Garbage after path: %s", line);
+	tree_content_get(root, uq.buf, &leaf);
+	if (!leaf.versions[1].mode)
+		die("Path %s not in branch", uq.buf);
+	hashcpy(sha1, leaf.versions[1].sha1);
+}
+
+static void sendreport(const char *buf, unsigned long size)
+{
+	if (write_in_full(report_fd, buf, size) != size)
+		die_errno("Write to frontend failed");
+}
+
+static void cat_object(struct object_entry *oe, unsigned char sha1[20])
+{
+	struct strbuf line = STRBUF_INIT;
+	unsigned long size;
+	enum object_type type = 0;
+	char *buf;
+
+	if (report_fd < 0)
+		die("Internal error: bad report_fd %d", report_fd);
+	if (oe && oe->pack_id != MAX_PACK_ID) {
+		type = oe->type;
+		buf = gfi_unpack_entry(oe, &size);
+	} else {
+		buf = read_sha1_file(sha1, &type, &size);
+	}
+	if (!buf)
+		die("Can't read object %s", sha1_to_hex(sha1));
+
+	/*
+	 * Output based on batch_one_object() from cat-file.c.
+	 */
+	if (type <= 0) {
+		strbuf_reset(&line);
+		strbuf_addf(&line, "%s missing\n", sha1_to_hex(sha1));
+		if (write_in_full(report_fd, line.buf, line.len) != line.len)
+			die_errno("Write to frontend failed 1");
+	}
+	strbuf_reset(&line);
+	strbuf_addf(&line, "%s %s %lu\n", sha1_to_hex(sha1),
+						typename(type), size);
+	sendreport(line.buf, line.len);
+	sendreport(buf, size);
+	sendreport("\n", 1);
+	free(buf);
+}
+
+
+static void parse_cat_request(void)
+{
+	const char *p;
+	struct object_entry *oe = oe;
+	unsigned char sha1[20];
+	struct tree_entry root = {0};
+
+	/* cat SP <object> */
+	p = command_buf.buf + strlen("cat ");
+	if (report_fd < 0)
+		die("The cat command features the report-fd feature.");
+	if (*p == ':') {
+		char *x;
+		oe = find_mark(strtoumax(p + 1, &x, 10));
+		if (x == p + 1)
+			die("Invalid mark: %s", command_buf.buf);
+		if (!oe)
+			die("Unknown mark: %s", command_buf.buf);
+		p = x;
+		hashcpy(sha1, oe->idx.sha1);
+	} else {
+		if (get_sha1_hex(p, sha1))
+			die("Invalid SHA1: %s", command_buf.buf);
+		p += 40;
+		if (!*p)
+			oe = find_object(sha1);
+	}
+
+	/* [ SP "<path>" ] */
+	if (*p) {
+		if (*p++ != ' ')
+			die("Missing space after SHA1: %s", command_buf.buf);
+
+		/* cat <tree> "<path>" form. */
+		hashcpy(root.versions[1].sha1, sha1);
+		load_tree(&root);
+		quoted_path_sha1(sha1, &root, p, command_buf.buf);
+		oe = find_object(sha1);
+	}
+
+	cat_object(oe, sha1);
+}
+
 static void parse_checkpoint(void)
 {
 	if (object_count) {
@@ -2971,6 +3077,8 @@ int main(int argc, const char **argv)
 			parse_new_tag();
 		else if (!prefixcmp(command_buf.buf, "reset "))
 			parse_reset_branch();
+		else if (!prefixcmp(command_buf.buf, "cat "))
+			parse_cat_request();
 		else if (!strcmp("checkpoint", command_buf.buf))
 			parse_checkpoint();
 		else if (!prefixcmp(command_buf.buf, "progress "))
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 610e7a5..82e03e8 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1705,6 +1705,120 @@ test_expect_success PIPE 'R: feature report-fd is honoured' '
 	test_cmp real received
 '
 
+test_expect_success PIPE 'R: report-fd: can feed back printed tree' '
+	cat >frontend <<-\FRONTEND_END &&
+		#!/bin/sh
+		cat <<EOF &&
+		feature report-fd=3
+		commit refs/heads/printed
+		committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+		data <<COMMIT
+		to be printed
+		COMMIT
+
+		from refs/heads/master
+		D file3
+
+		EOF
+
+		read commit_id <&3 &&
+		echo "$commit_id" >printed &&
+		echo "$commit_id commit" >expect.response &&
+		echo "cat $commit_id" &&
+		read cid2 type size <&3 &&
+		echo "$cid2 $type" >response &&
+		dd if=/dev/stdin of=commit bs=1 count=$size <&3 &&
+		read newline <&3 &&
+		read tree tree_id <commit &&
+
+		cat <<EOF &&
+		commit refs/heads/printed
+		committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+		data <<COMMIT
+		to be printed
+		COMMIT
+
+		from refs/heads/printed^0
+		M 040000 $tree_id old
+
+		EOF
+		read cid <&3
+	FRONTEND_END
+	chmod +x frontend &&
+
+	mkfifo commits &&
+	test_when_finished "rm -f commits" &&
+	(
+		{
+			sh frontend 3<commits ||
+			exit
+		} |
+		git fast-import 3>commits
+	) &&
+	git rev-parse printed^ >expect.printed &&
+	git cat-file commit printed^ >expect.commit &&
+
+	test_cmp expect.printed printed &&
+	test_cmp expect.response response &&
+	test_cmp expect.commit commit
+'
+
+test_expect_success PIPE 'R: report-fd: can feed back printed blob' '
+	cat >expect <<-EOF &&
+	:100755 100644 $file6_id $file6_id C100	newdir/exec.sh	file6
+	EOF
+
+	cat >frontend <<-\FRONTEND_END &&
+		#!/bin/sh
+
+		branch=$(git rev-parse --verify refs/heads/branch) &&
+		cat <<EOF &&
+		feature report-fd=3
+		cat $branch
+		EOF
+
+		read commit_id type size <&3 &&
+		dd if=/dev/stdin of=commit bs=1 count=$size <&3 &&
+		read newline <&3 &&
+		read tree tree_id <commit &&
+
+		echo "cat $tree_id \"newdir/exec.sh\"" &&
+		read blob_id type size <&3 &&
+		dd if=/dev/stdin of=blob bs=1 count=$size <&3 &&
+		read newline <&3 &&
+
+		cat <<EOF &&
+		commit refs/heads/copyblob
+		committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+		data <<COMMIT
+		copy file6 to top level
+		COMMIT
+
+		from refs/heads/branch^0
+		M 644 inline "file6"
+		data $size
+		EOF
+		cat blob &&
+		echo &&
+		echo &&
+
+		read cid <&3
+	FRONTEND_END
+	chmod +x frontend &&
+
+	mkfifo commits &&
+	test_when_finished "rm -f commits" &&
+	(
+		{
+			sh frontend 3<commits ||
+			exit
+		} |
+		git fast-import 3>commits
+	) &&
+	git diff-tree -C --find-copies-harder -r copyblob^ copyblob >actual &&
+	compare_diff_raw expect actual
+'
+
 test_expect_success 'R: quiet option results in no stats being output' '
 	>empty &&
 	cat >input <<-\EOF &&
-- 
1.7.2.3

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Ramkumar Ramachandra]

Hi Jonathan,

Jonathan Nieder writes:
> To recap, because fast-import does not write a pack index until the
> checkpoint, frontends cannot necessarily read back what they have
> written immediately.  The obvious application to allowing that is as a
> sanity check, but more important to us is that it allows the lazy
> frontend to forget previous revisions even if they are required for
> later "file/directory copy" operations in the import.  The frontend
> can be secure in the knowledge that the backend remembers everything
> now.

Excellent timing! I finished writing tests for the svndiff0 applier
and started modifying svn-fe to use it. I've applied your patches and
rebased my work on top of it.

> I suspect that these patches are not in their final form.
> In particular, the interface is kind of klunky: to name a
> blob by path, you have to supply a *tree* which contains that
> blob as well as the pathname.  So retrieving, say,
> v1.7.1:Documentation/git-fast-import.txt, would require
> three round-trips: one to dereference the tag, one to dereference
> the commit, and then one to finally retrieve the blob.

Ideally, we'd like to be able to supply a commit and a pathname. I'll
look into the difficulties in implementing this interface.

> Another possible concern is that this is very much git specific.
> Other fast-import backends are just not going to be able to do
> it with the same format.  Is there a convention for naming
> options like that?

Why is this a concern? We aren't even breaking backward
compatibility. This bidi interface should only be used by SVN-like
frontends.

> Still, I hope it is useful to start with.  Thoughts?  Ideas?
> Improvements?

We'll get ideas to improve this as svn-fe gets built up. On a related
note, we should probably extend the git-remote-testgit helper to use
this feature in future to facilitate writing unittests.

Thanks.

-- Ram

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Sverre Rabbelier]

Heya,

On Sun, Sep 5, 2010 at 01:08, Ramkumar Ramachandra <artagnon@gmail.com> wrote:
>> Another possible concern is that this is very much git specific.
>> Other fast-import backends are just not going to be able to do
>> it with the same format.  Is there a convention for naming
>> options like that?
>
> Why is this a concern? We aren't even breaking backward
> compatibility. This bidi interface should only be used by SVN-like
> frontends.

I don't agree, imagine implementing hg-remote-svn (or the conceptual
equivalent at least), it would require the same functionality, yes?
You need to retrieve something from fast-import that you previously
gave it.

> We'll get ideas to improve this as svn-fe gets built up. On a related
> note, we should probably extend the git-remote-testgit helper to use
> this feature in future to facilitate writing unittests.

Hmmm, that would be somewhat difficult I suspect, the reason
git-remote-testgit was so easy to write is because git-fast-export and
git-fast-import do exactly what I wanted to test. You'd have to teach
munge the fast-export stream even more to insert this feature somehow,
I suspect that'll be a non-trivial feat. Of course, you will need to
test it somehow, I suspect that Jonathan's current approach (using
hardcoded streams) might be the easiest.

-- 
Cheers,

Sverre Rabbelier

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Ramkumar Ramachandra]

Hi Sverre,

Sverre Rabbelier writes:
> On Sun, Sep 5, 2010 at 01:08, Ramkumar Ramachandra <artagnon@gmail.com> wrote:
> >> Another possible concern is that this is very much git specific.
> >> Other fast-import backends are just not going to be able to do
> >> it with the same format.  Is there a convention for naming
> >> options like that?
> >
> > Why is this a concern? We aren't even breaking backward
> > compatibility. This bidi interface should only be used by SVN-like
> > frontends.
> 
> I don't agree, imagine implementing hg-remote-svn (or the conceptual
> equivalent at least), it would require the same functionality, yes?
> You need to retrieve something from fast-import that you previously
> gave it.

Ah, yes. I didn't think of that. If fast-import is already mature,
it's probably a good idea to design a new protocol altogether based on
this and not call it fast-import?

> > We'll get ideas to improve this as svn-fe gets built up. On a related
> > note, we should probably extend the git-remote-testgit helper to use
> > this feature in future to facilitate writing unittests.
> 
> Hmmm, that would be somewhat difficult I suspect, the reason
> git-remote-testgit was so easy to write is because git-fast-export and
> git-fast-import do exactly what I wanted to test. You'd have to teach
> munge the fast-export stream even more to insert this feature somehow,
> I suspect that'll be a non-trivial feat. Of course, you will need to
> test it somehow, I suspect that Jonathan's current approach (using
> hardcoded streams) might be the easiest.

Hm, ok. Whatever makes testing easier.

-- Ram

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Sverre Rabbelier]

Heya,

On Sun, Sep 5, 2010 at 03:47, Ramkumar Ramachandra <artagnon@gmail.com> wrote:
> Sverre Rabbelier writes:
>> I don't agree, imagine implementing hg-remote-svn (or the conceptual
>> equivalent at least), it would require the same functionality, yes?
>> You need to retrieve something from fast-import that you previously
>> gave it.
>
> Ah, yes. I didn't think of that. If fast-import is already mature,
> it's probably a good idea to design a new protocol altogether based on
> this and not call it fast-import?

That's not what I meant at all. We should still use fast-import, we
should just design this extension to the fast-import protocol in such
a way that it would also be useful when implementing something like
hg-remote-svn ;).

-- 
Cheers,

Sverre Rabbelier

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Jonathan Nieder]

Ramkumar Ramachandra wrote:

> I finished writing tests for the svndiff0 applier
> and started modifying svn-fe to use it. I've applied your patches and
> rebased my work on top of it.

Nice.

> Ideally, we'd like to be able to supply a commit and a pathname.

To expand on this a little, avoiding dwim-ery of this kind was part
of the spirit I was going for: minimal as possible.  Either the
frontend or the backend is going to have to read the first line of
the commit:

 "tree ...tree name...
  parent [etc]"

and I thought, why not let the frontend do that for now?

Hopefully after using something just barely sufficient like this, we
will know what primitives are needed to make code clear and
efficient (ls?  cat --batch-check?  cat byte range?).  And such
primitives could be at a higher level of abstraction, which might make
the protocol guardians happier.

> On a related
> note, we should probably extend the git-remote-testgit helper to use
> this feature in future to facilitate writing unittests.

Hopefully t/t9011-svn-helper.sh will exercise it in the future.

Thanks for the comments,
Jonathan

[PATCH 4/3] fast-import: typofix

[Jonathan Nieder]

Noticed-by: Sverre Rabbelier <srabbelier@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
To be squashed with patch 3 in the next round.  Thanks,
Sverre.

 fast-import.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fast-import.c b/fast-import.c
index b7fa9ae..099f63e 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -2761,7 +2761,7 @@ static void parse_cat_request(void)
 	/* cat SP <object> */
 	p = command_buf.buf + strlen("cat ");
 	if (report_fd < 0)
-		die("The cat command features the report-fd feature.");
+		die("The cat command requires the report-fd feature.");
 	if (*p == ':') {
 		char *x;
 		oe = find_mark(strtoumax(p + 1, &x, 10));
-- 
1.7.3.rc0.6.g7505a.dirty

[PATCH 5/3] fast-import: allow cat command with empty path

[Jonathan Nieder]

Rather than erroring out, treat an empty path as the path to
the root of a tree so frontends can be simplified a little.

While at it, fix a typo in an error message: the cat command is used
to examine paths within trees, not branches.

Cc: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Would this be helpful?  I imagine it would make tree access by
pathname a bit simpler, and I found myself tempted to try it.

The test from the next patch exercises this.

 fast-import.c |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/fast-import.c b/fast-import.c
index 099f63e..f3c4123 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -2702,9 +2702,13 @@ static void quoted_path_sha1(unsigned char sha1[20], struct tree_entry *root,
 		die("Invalid path: %s", line);
 	if (*x)
 		die("Garbage after path: %s", line);
+	if (uq.len == 0) {
+		hashcpy(sha1, root->versions[1].sha1);
+		return;
+	}
 	tree_content_get(root, uq.buf, &leaf);
 	if (!leaf.versions[1].mode)
-		die("Path %s not in branch", uq.buf);
+		die("Path %s not in tree", uq.buf);
 	hashcpy(sha1, leaf.versions[1].sha1);
 }
 
-- 
1.7.3.rc0.6.g7505a.dirty

[PATCH 6/3] fast-import: Allow cat requests at arbitrary points in stream

[Jonathan Nieder]

Rather than planning in advance and saving up a bunch of objects
before a "commit" command, frontends might want to wait until the
middle of a commit and request objects then.

Allow them to.

The new rule: a "cat" request can be inserted wherever a comment
is allowed, which means at the start of any line except in the
middle of a "data" command.

Cc: Ramkumar Ramachandra <artagnon@gmail.com>
Cc: David Barr <david.barr@cordelta.com>
Cc: Sverre Rabbelier <srabbelier@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
That's it for today.  I'll be on vacation for the next couple of weeks
so I can't promise to be responsive.  Comments are still welcome,
though (nothing is really different except that the latency may be
longer).

 Documentation/git-fast-import.txt |    4 ++
 fast-import.c                     |   27 ++++++++-----
 t/t9300-fast-import.sh            |   74 +++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+), 11 deletions(-)

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 2cf48f5..b670c7d 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -920,6 +920,10 @@ a path within that tree to retrieve a subtree or blob.
 A `<path>` string should be surrounded with quotation marks and
 use C-style escaping.
 
+The `cat` command can be used anywhere that comments are
+acceptable.  In particular, the `cat` command can be used in the
+middle of a commit, but not in the middle of a `data` stream.
+
 `feature`
 ~~~~~~~~~
 Require that fast-import supports the specified feature, or abort if
diff --git a/fast-import.c b/fast-import.c
index f3c4123..87246d5 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -55,9 +55,6 @@ Format of STDIN stream:
     ('from' sp committish lf)?
     lf?;
 
-  cat_request ::= 'cat' sp (hexsha1 | idnum) lf
-    | 'cat' sp hexsha1 sp path_str lf;
-
   checkpoint ::= 'checkpoint' lf
     lf?;
 
@@ -135,14 +132,18 @@ Format of STDIN stream:
   ts    ::= # time since the epoch in seconds, ascii base10 notation;
   tz    ::= # GIT style timezone;
 
-     # note: comments may appear anywhere in the input, except
-     # within a data command.  Any form of the data command
-     # always escapes the related input from comment processing.
+     # note: comments and cat requests may appear anywhere
+     # in the input, except within a data command.  Any form
+     # of the data command always escapes the related input
+     # from comment processing.
      #
      # In case it is not clear, the '#' that starts the comment
      # must be the first character on that line (an lf
      # preceded it).
      #
+  cat_request ::= 'cat' sp (hexsha1 | idnum) lf
+    | 'cat' sp hexsha1 sp path_str lf;
+
   comment ::= '#' not_lf* lf;
   not_lf  ::= # Any byte that is not ASCII newline (LF);
 */
@@ -368,6 +369,7 @@ static int seen_data_command;
 static int report_fd = -1;
 
 static void parse_argv(void);
+static void parse_cat_request(void);
 
 static void write_branch_report(FILE *rpt, struct branch *b)
 {
@@ -1777,7 +1779,6 @@ static void read_marks(void)
 	fclose(f);
 }
 
-
 static int read_next_command(void)
 {
 	static int stdin_eof = 0;
@@ -1787,7 +1788,7 @@ static int read_next_command(void)
 		return EOF;
 	}
 
-	do {
+	for (;;) {
 		if (unread_command_buf) {
 			unread_command_buf = 0;
 		} else {
@@ -1820,7 +1821,13 @@ static int read_next_command(void)
 			rc->prev->next = rc;
 			cmd_tail = rc;
 		}
-	} while (command_buf.buf[0] == '#');
+		if (!prefixcmp(command_buf.buf, "cat ")) {
+			parse_cat_request();
+			continue;
+		}
+		if (command_buf.buf[0] != '#')
+			return 0;
+	}
 
 	return 0;
 }
@@ -3081,8 +3088,6 @@ int main(int argc, const char **argv)
 			parse_new_tag();
 		else if (!prefixcmp(command_buf.buf, "reset "))
 			parse_reset_branch();
-		else if (!prefixcmp(command_buf.buf, "cat "))
-			parse_cat_request();
 		else if (!strcmp("checkpoint", command_buf.buf))
 			parse_checkpoint();
 		else if (!prefixcmp(command_buf.buf, "progress "))
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 82e03e8..dcdfbd4 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1819,6 +1819,80 @@ test_expect_success PIPE 'R: report-fd: can feed back printed blob' '
 	compare_diff_raw expect actual
 '
 
+test_expect_success PIPE 'R: report-fd: cat commands mid-commit' '
+	cat >expect <<-EOF &&
+	:100755 100644 $file4_id $file4_id C100 file4	file4-copy
+	EOF
+	git cat-file commit refs/heads/branch >expect.commit &&
+	echo tree >expect.type &&
+	git cat-file tree refs/heads/branch >expect.tree &&
+
+	cat >frontend <<-\FRONTEND_END &&
+		#!/bin/sh
+
+		branch=$(git rev-parse --verify refs/heads/branch) &&
+		cat <<EOF &&
+		feature report-fd=3
+		commit refs/heads/catdemo
+		cat $branch
+		EOF
+
+		read commit_id type size <&3 &&
+		dd if=/dev/stdin of=commit bs=1 count=$size <&3 &&
+		read newline <&3 &&
+		read tree tree_id <commit &&
+
+		echo "cat $tree_id" &&
+
+		read tree_id2 type size <&3 &&
+		dd if=/dev/stdin of=tree bs=1 count=$size <&3 &&
+		read newline <&3 &&
+		echo $type >type &&
+
+		grep ^committer commit &&
+		cat <<EOF &&
+		data <<COMMIT
+		copy file4 as file4-copy
+		COMMIT
+
+		from refs/heads/branch^0
+		cat $tree_id "file4"
+		EOF
+
+		read blob_id type size <&3 &&
+		dd if=/dev/stdin of=/dev/null bs=1 count=$size <&3 &&
+		read newline <&3 &&
+
+		cat <<EOF &&
+		M 644 $blob_id "file4-copy"
+		cat $tree_id ""
+
+		EOF
+
+		read tree_id3 type size <&3 &&
+		dd if=/dev/stdin of=tree2 bs=1 count=$size <&3 &&
+		read newline <&3 &&
+		read cid <&3
+	FRONTEND_END
+	chmod +x frontend &&
+
+	mkfifo commits &&
+	test_when_finished "rm -f commits" &&
+	(
+		{
+			sh frontend 3<commits ||
+			exit
+		} |
+		git fast-import 3>commits
+	) &&
+	git diff-tree -C catdemo^ catdemo >actual &&
+	compare_diff_raw expect actual
+	test_cmp expect.commit commit &&
+	test_cmp expect.type type &&
+	test_cmp expect.tree tree &&
+	test_cmp expect.tree tree2
+'
+
 test_expect_success 'R: quiet option results in no stats being output' '
 	>empty &&
 	cat >input <<-\EOF &&
-- 
1.7.3.rc0.6.g7505a.dirty

Re: [PATCH 6/3] fast-import: Allow cat requests at arbitrary points in stream

[Sverre Rabbelier]

Heya,

On Tue, Sep 7, 2010 at 22:27, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Rather than planning in advance and saving up a bunch of objects
> before a "commit" command, frontends might want to wait until the
> middle of a commit and request objects then.

I'm not sure I understand the justification of this patch. Is the
purpose of this patch to save you from having to loop over everything
you want to commit in the next commit and cat-ing that in advance?

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH 6/3] fast-import: Allow cat requests at arbitrary points in stream

[Jonathan Nieder]

Sverre Rabbelier wrote:
> On Tue, Sep 7, 2010 at 22:27, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> Rather than planning in advance and saving up a bunch of objects
>> before a "commit" command, frontends might want to wait until the
>> middle of a commit and request objects then.
>
> I'm not sure I understand the justification of this patch. Is the
> purpose of this patch to save you from having to loop over everything
> you want to commit in the next commit and cat-ing that in advance?

Yes.  In particular, an importer that wants to apply deltas
to generate the content for

 M 100644 inline foo

commands might use this.

Probably not relevant for svn-fe in its current design, since it uses
blob commands and remembers the tree as it builds it up.

Re: [PATCH 6/3] fast-import: Allow cat requests at arbitrary points in stream

[Ramkumar Ramachandra]

Hi Jonathan,

Jonathan Nieder writes:
> Rather than planning in advance and saving up a bunch of objects
> before a "commit" command, frontends might want to wait until the
> middle of a commit and request objects then.
> 
> Allow them to.
> 
> The new rule: a "cat" request can be inserted wherever a comment
> is allowed, which means at the start of any line except in the
> middle of a "data" command.

Excellent idea. This will certainly be useful when we start working on
the zero-tree refactor.

> That's it for today.  I'll be on vacation for the next couple of weeks
> so I can't promise to be responsive.  Comments are still welcome,
> though (nothing is really different except that the latency may be
> longer).

Quick status update: David has started working on the IO backend
that's necessary to get svn-fe to use svn-da. We'll start testing/
reviewing this series once that's done.

I'm busy studying for exams next week.

-- Ram

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Sam Vilain]

On Sat, 2010-09-04 at 22:15 -0500, Jonathan Nieder wrote:
> It works like this:
> 
> frontend:
> 	feature report-fd=3
> 	commit refs/heads/master
> 	... revision 1 ...
> 
> importer:
> 	abc7856cba76bca87a65bca76bca8bca98bca78bca76

This is probably quite a late comment, but I don't think that
'report-fd=3' is a good idea in a protocol like this.  It should not
take an argument and just respond down the appropriate selected file
descriptor.  ie, default to standard output.  If standard input is a
socket, then use that bidirectionally.  If --report-fd is used on the
command line, use that.

Sam

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Sverre Rabbelier]

Heya,

On Thu, Sep 16, 2010 at 02:14, Sam Vilain <sam@vilain.net> wrote:
> This is probably quite a late comment, but I don't think that
> 'report-fd=3' is a good idea in a protocol like this.  It should not
> take an argument and just respond down the appropriate selected file
> descriptor.  ie, default to standard output.  If standard input is a
> socket, then use that bidirectionally.  If --report-fd is used on the
> command line, use that.

I understand your point, the generated stream is not very reusable if
the fd is 'hardcoded'. On the other hand, the primary use for this
feature is the case where an exporter feeds directly into an importer.
What use case do you see for re-use of a stream where the report-fd is
used? Or are your objections based on some other ground?

-- 
Cheers,

Sverre Rabbelier

[PATCH/RFC 00/24] Re: [PATCH 1/3] t9300 (fast-import): style tweaks

[Jonathan Nieder]

Hi,

Jonathan Nieder wrote:

> Clarify dependencies between tests to make the fast-import test
> script more approachable.  In particular:
... many things ...
> While at it:
... more things ...

The patch was a lazy way for me to add new assertions to the
fast-import test script without going crazy.  But it really was lazy:
it has almost nothing to do with the "fast-import protocol
experiments" series that it headed, and worse, that one patch did so
many things at once that it was basically guaranteed that (1) no one
would like all of it and (2) it bitrotted in a couple of days.

Oh well.  Tomorrow I would like to re-roll the fast-import experiment
so the svn-fe that understands deltas can get more attention, and of
course that series does not require these style fixes at all.

So why resend them?  I end up mentally making these changes every time
I add a new test to that script, so I imagine it would be nicer to
make the changes once.  Maybe it would help newcomers to dive into the
wonderful world of fast-import testing.

So here is a small chunk of that monster patch, ejected from the
original series and split up.

Patch 1 introduces a verify_packs () helper that makes the script much
easier to read (by including only two copies of an unpleasant loop).
The nominal justification is that giving the

	for each pack
	do
		git verify-pack $pack ||
		exit
	done

loop its own function allows use to write "return" instead of "exit",
resulting in better behavior when a test fails.

Patch 2 is the most important one to me.  It gets rid of some
hardcoded tree and blob names, most of which were not doing any
harm except to scare me.  At first glance it is not obvious when
a stray test_tick, for example, will ruin later tests (it turns out
never but there is at one test that does depend on the choice of
hash function), and at first glance, it is not obvious what is
actually the expected diff when a raw diff is presented as expected
output.

The approach adopted is to introduce some symbolic constants, like

	empty_blob=$(git hash-object --stdin </dev/null)

and use them in the expected output.

Patches 3-6 just pick nits.  The dividends are better output
with -v and more robust checking for failure of git commands.

Patch 7 changes some 4-space indents to tabs (since the latter
is predominant in the file).

Patches 8-24 change from traditional

	test_expect_failure \
		'description' \
		'commands &&
		 more commands'

to modern

	test_expect_success 'description' '
		commands &&
		more commands
	'

style.  They are meant to be squashed together and are only split
in tiny pieces for easier review.

Let's take whatever is useful and forget about the rest.

Thanks,
Jonathan Nieder (24):
  t9300 (fast-import): avoid exiting early on failure
  t9300 (fast-import): avoid hard-coded object names
  t9300 (fast-import): guard "export large marks" test
  t9300 (fast-import): check exit status from upstream of pipes
  t9300 (fast-import): check exit status from command substitution
  t9300 (fast-import): use test_cmp in place of test $(foo) = $(bar)
  t9300 (fast-import): use tabs to indent
  t9300 (fast-import), series A: re-indent
  t9300 (fast-import), series B: re-indent
  t9300 (fast-import), series C: re-indent
  t9300 (fast-import), series D: re-indent
  t9300 (fast-import), series E: re-indent
  t9300 (fast-import), series F: re-indent
  t9300 (fast-import), series H: re-indent
  t9300 (fast-import), series I: re-indent
  t9300 (fast-import), series J: re-indent
  t9300 (fast-import), series K: re-indent
  t9300 (fast-import), series L: re-indent
  t9300 (fast-import), series M: re-indent
  t9300 (fast-import), series N: re-indent
  t9300 (fast-import), series O: re-indent
  t9300 (fast-import), series P: re-indent
  t9300 (fast-import), series Q: re-indent
  t9300 (fast-import), series R: re-indent

 t/t9300-fast-import.sh | 1010 ++++++++++++++++++++++++++----------------------
 1 files changed, 539 insertions(+), 471 deletions(-)

-- 
1.7.2.3

[PATCH 01/24] t9300 (fast-import): avoid exiting early on failure

[Jonathan Nieder]

Exiting is not an appropriate way to signal a test failure, since it
bypasses the usual expect_success/expect_failure logic and prevents
later tests that might be able to help diagnose the problem from
running.

Still, exiting is a temptingly easy way to catch errors in shell
loops.  Alternatives:

 1) Ignore errors in the loop body.  This can leave problems hidden
    and undiagnosed so it is not really an option.

 2) Enclose the loop in a subshell which can catch the exit on
    failure.

 3) Enclose the loop in a function and return the exit code on
    failure.

 4) More ad-hoc methods.

Most instances in t9300 lend themselves well to (3), with the nice
side effect of avoiding some repetitive code.  Introduce a new
verify_packs () helper in this spirit.

One instance uses "verify-pack -v", making it slightly different from
the others; rather than spending effort on making the helper general
enough to be usable for it, too, enclose the strange loop in
a subshell (strategy (2)).  While at it, simplify the code a little
by redirecting stdout of the loop instead of its body.

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   38 ++++++++++++++++++++++++--------------
 1 files changed, 24 insertions(+), 14 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 7c05920..1eef926 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -23,6 +23,14 @@ file5_data='an inline file.
 file6_data='#!/bin/sh
 echo "$@"'
 
+verify_packs () {
+	for p in .git/objects/pack/*.pack
+	do
+		git verify-pack $p ||
+		return
+	done
+}
+
 ###
 ### series A
 ###
@@ -69,7 +77,7 @@ test_expect_success \
 	 git whatchanged master'
 test_expect_success \
 	'A: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 
 cat >expect <<EOF
 author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
@@ -155,7 +163,7 @@ test_expect_success \
 	 git whatchanged verify--import-marks'
 test_expect_success \
 	'A: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 cat >expect <<EOF
 :000000 100755 0000000000000000000000000000000000000000 7123f7f44e39be127c5eb701e5968176ee9d78b1 A	copy-of-file2
 EOF
@@ -318,7 +326,7 @@ test_expect_success \
 	 git whatchanged branch'
 test_expect_success \
 	'C: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 test_expect_success \
 	'C: validate reuse existing blob' \
 	'test $newf = `git rev-parse --verify branch:file2/newf`
@@ -376,7 +384,7 @@ test_expect_success \
 	 git whatchanged branch'
 test_expect_success \
 	'D: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 
 cat >expect <<EOF
 :000000 100755 0000000000000000000000000000000000000000 35a59026a33beac1569b1c7f66f3090ce9c09afc A	newdir/exec.sh
@@ -422,7 +430,7 @@ test_expect_success \
     'git fast-import --date-format=rfc2822 <input'
 test_expect_success \
 	'E: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 
 cat >expect <<EOF
 author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> 1170778938 -0500
@@ -473,7 +481,7 @@ test_expect_success \
 	'
 test_expect_success \
 	'F: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 
 cat >expect <<EOF
 tree `git rev-parse branch~1^{tree}`
@@ -509,7 +517,7 @@ test_expect_success \
     'git fast-import --force <input'
 test_expect_success \
 	'G: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 test_expect_success \
 	'G: branch changed, but logged' \
 	'test $old_branch != `git rev-parse --verify branch^0` &&
@@ -546,7 +554,7 @@ test_expect_success \
 	 git whatchanged H'
 test_expect_success \
 	'H: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 
 cat >expect <<EOF
 :100755 000000 f1fb5da718392694d0076d677d6d0e364c79b0bc 0000000000000000000000000000000000000000 D	file2/newf
@@ -1327,7 +1335,7 @@ test_expect_success \
 	 git whatchanged notes-test'
 test_expect_success \
 	'Q: verify pack' \
-	'for p in .git/objects/pack/*.pack;do git verify-pack $p||exit;done'
+	'verify_packs'
 
 commit1=$(git rev-parse notes-test~2)
 commit2=$(git rev-parse notes-test^)
@@ -1675,11 +1683,13 @@ test_expect_success \
 	 git --git-dir=R/.git fast-import --big-file-threshold=1 <input'
 test_expect_success \
 	'R: verify created pack' \
-	': >verify &&
-	 for p in R/.git/objects/pack/*.pack;
-	 do
-	   git verify-pack -v $p >>verify || exit;
-	 done'
+	'(
+		for p in R/.git/objects/pack/*.pack
+		do
+			git verify-pack -v $p ||
+			exit
+		done
+	 ) >verify'
 test_expect_success \
 	'R: verify written objects' \
 	'git --git-dir=R/.git cat-file blob big-file:big1 >actual &&
-- 
1.7.2.3

[PATCH 02/24] t9300 (fast-import): avoid hard-coded object names

[Jonathan Nieder]

compare_diff_raw already ensures most (though not all) of the tests
do not rely on any particular hash function, but still the attentive
reader might be comforted by some less opaque object descriptions.

Tested with

	sed -i s/compare_diff_raw/test_cmp/g t9300-fast-import.sh &&
	sh t9300-fast-import.sh -v -i

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   91 ++++++++++++++++++++++++++++++++---------------
 1 files changed, 62 insertions(+), 29 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 1eef926..ed37a4e 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -7,6 +7,8 @@ test_description='test git fast-import utility'
 . ./test-lib.sh
 . "$TEST_DIRECTORY"/diff-lib.sh ;# test-lib chdir's into trash
 
+zeroes=0000000000000000000000000000000000000000
+
 file2_data='file2
 second line of EOF'
 
@@ -31,6 +33,14 @@ verify_packs () {
 	done
 }
 
+test_expect_success 'setup' '
+	file2_id=$(echo "$file2_data" | git hash-object --stdin) &&
+	file3_id=$(echo "$file3_data" | git hash-object --stdin) &&
+	file4_id=$(printf "%s" "$file4_data" | git hash-object --stdin) &&
+	file5_id=$(echo "$file5_data" | git hash-object --stdin) &&
+	file6_id=$(echo "$file6_data" | git hash-object --stdin)
+'
+
 ###
 ### series A
 ###
@@ -165,7 +175,7 @@ test_expect_success \
 	'A: verify pack' \
 	'verify_packs'
 cat >expect <<EOF
-:000000 100755 0000000000000000000000000000000000000000 7123f7f44e39be127c5eb701e5968176ee9d78b1 A	copy-of-file2
+:000000 100755 $zeroes $file2_id A	copy-of-file2
 EOF
 git diff-tree -M -r master verify--import-marks >actual
 test_expect_success \
@@ -345,9 +355,9 @@ test_expect_success \
 	 test_cmp expect actual'
 
 cat >expect <<EOF
-:000000 100755 0000000000000000000000000000000000000000 f1fb5da718392694d0076d677d6d0e364c79b0bc A	file2/newf
-:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 R100	file2	file2/oldf
-:100644 000000 0d92e9f3374ae2947c23aa477cbc68ce598135f1 0000000000000000000000000000000000000000 D	file3
+:000000 100755 $zeroes $newf A	file2/newf
+:100644 100644 $file2_id $file2_id R100	file2	file2/oldf
+:100644 000000 $file3_id $zeroes D	file3
 EOF
 git diff-tree -M -r master branch >actual
 test_expect_success \
@@ -387,8 +397,8 @@ test_expect_success \
 	'verify_packs'
 
 cat >expect <<EOF
-:000000 100755 0000000000000000000000000000000000000000 35a59026a33beac1569b1c7f66f3090ce9c09afc A	newdir/exec.sh
-:000000 100644 0000000000000000000000000000000000000000 046d0371e9220107917db0d0e030628de8a1de9b A	newdir/interesting
+:000000 100755 $zeroes $file6_id A	newdir/exec.sh
+:000000 100644 $zeroes $file5_id A	newdir/interesting
 EOF
 git diff-tree -M -r branch^ branch >actual
 test_expect_success \
@@ -557,11 +567,11 @@ test_expect_success \
 	'verify_packs'
 
 cat >expect <<EOF
-:100755 000000 f1fb5da718392694d0076d677d6d0e364c79b0bc 0000000000000000000000000000000000000000 D	file2/newf
-:100644 000000 7123f7f44e39be127c5eb701e5968176ee9d78b1 0000000000000000000000000000000000000000 D	file2/oldf
-:100755 000000 85df50785d62d3b05ab03d9cbf7e4a0b49449730 0000000000000000000000000000000000000000 D	file4
-:100644 100644 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 R100	newdir/interesting	h/e/l/lo
-:100755 000000 e74b7d465e52746be2b4bae983670711e6e66657 0000000000000000000000000000000000000000 D	newdir/exec.sh
+:100755 000000 $newf $zeroes D	file2/newf
+:100644 000000 $oldf $zeroes D	file2/oldf
+:100755 000000 $file4_id $zeroes D	file4
+:100644 100644 $file5_id $file5_id R100	newdir/interesting	h/e/l/lo
+:100755 000000 $file6_id $zeroes D	newdir/exec.sh
 EOF
 git diff-tree -M -r H^ H >actual
 test_expect_success \
@@ -698,16 +708,39 @@ M 644 :2 b/other
 M 644 :2 ba
 INPUT_END
 
+test_expect_success 'setup: object names for expected diff' '
+	some_data=$(
+		echo some data |
+		git hash-object -w --stdin
+	) &&
+	other_data=$(
+		echo other data |
+		git hash-object -w --stdin
+	) &&
+	some_tree=$(
+		rm -f tmp_index &&
+		echo "100644 blob $some_data	other" |
+		GIT_INDEX_FILE=tmp_index git update-index --index-info &&
+		GIT_INDEX_FILE=tmp_index git write-tree
+	) &&
+	other_tree=$(
+		rm -f tmp_index &&
+		echo "100644 blob $other_data	other" |
+		GIT_INDEX_FILE=tmp_index git update-index --index-info &&
+		GIT_INDEX_FILE=tmp_index git write-tree
+	)
+'
+
 cat >expect <<EXPECT_END
-:100644 100644 4268632... 55d3a52... M	b.
-:040000 040000 0ae5cac... 443c768... M	b
-:100644 100644 4268632... 55d3a52... M	ba
+:100644 100644 $some_data $other_data M	b.
+:040000 040000 $some_tree $other_tree M	b
+:100644 100644 $some_data $other_data M	ba
 EXPECT_END
 
 test_expect_success \
     'L: verify internal tree sorting' \
 	'git fast-import <input &&
-	 git diff-tree --abbrev --raw L^ L >output &&
+	 git diff-tree L^ L >output &&
 	 test_cmp expect output'
 
 ###
@@ -728,7 +761,7 @@ R file2/newf file2/n.e.w.f
 INPUT_END
 
 cat >expect <<EOF
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100	file2/newf	file2/n.e.w.f
+:100755 100755 $newf $newf R100	file2/newf	file2/n.e.w.f
 EOF
 test_expect_success \
 	'M: rename file in same subdirectory' \
@@ -749,7 +782,7 @@ R file2/newf i/am/new/to/you
 INPUT_END
 
 cat >expect <<EOF
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100	file2/newf	i/am/new/to/you
+:100755 100755 $newf $newf R100	file2/newf	i/am/new/to/you
 EOF
 test_expect_success \
 	'M: rename file to new subdirectory' \
@@ -770,7 +803,7 @@ R i other/sub
 INPUT_END
 
 cat >expect <<EOF
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100	i/am/new/to/you	other/sub/am/new/to/you
+:100755 100755 $newf $newf R100	i/am/new/to/you	other/sub/am/new/to/you
 EOF
 test_expect_success \
 	'M: rename subdirectory to new subdirectory' \
@@ -796,7 +829,7 @@ C file2/newf file2/n.e.w.f
 INPUT_END
 
 cat >expect <<EOF
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file2/n.e.w.f
+:100755 100755 $newf $newf C100	file2/newf	file2/n.e.w.f
 EOF
 test_expect_success \
 	'N: copy file in same subdirectory' \
@@ -828,9 +861,9 @@ EOF
 INPUT_END
 
 cat >expect <<EOF
-:100644 100644 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 C100	newdir/interesting	file3/file5
-:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file3/newf
-:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100	file2/oldf	file3/oldf
+:100644 100644 $file5_id $file5_id C100	newdir/interesting	file3/file5
+:100755 100755 $newf $newf C100	file2/newf	file3/newf
+:100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
 EOF
 test_expect_success \
 	'N: copy then modify subdirectory' \
@@ -863,9 +896,9 @@ test_expect_success \
 
 test_expect_success \
 	'N: copy directory by id' \
-	'cat >expect <<-\EOF &&
-	:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file3/newf
-	:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100	file2/oldf	file3/oldf
+	'cat >expect <<-EOF &&
+	:100755 100755 $newf $newf C100	file2/newf	file3/newf
+	:100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
 	EOF
 	 subdir=$(git rev-parse refs/heads/branch^0:file2) &&
 	 cat >input <<-INPUT_END &&
@@ -884,10 +917,10 @@ test_expect_success \
 
 test_expect_success \
 	'N: modify copied tree' \
-	'cat >expect <<-\EOF &&
-	:100644 100644 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 fcf778cda181eaa1cbc9e9ce3a2e15ee9f9fe791 C100	newdir/interesting	file3/file5
-	:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100	file2/newf	file3/newf
-	:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100	file2/oldf	file3/oldf
+	'cat >expect <<-EOF &&
+	:100644 100644 $file5_id $file5_id C100	newdir/interesting	file3/file5
+	:100755 100755 $newf $newf C100	file2/newf	file3/newf
+	:100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
 	EOF
 	 subdir=$(git rev-parse refs/heads/branch^0:file2) &&
 	 cat >input <<-INPUT_END &&
-- 
1.7.2.3

[PATCH 03/24] t9300 (fast-import): guard "export large marks" test setup

[Jonathan Nieder]

Since the setup for the large marks test involves more complex code
than just a "cat", "echo", or assignment, make it part of the
test_expect_success assertion.  This way, the test script will report
a failure if the setup breaks from a coding mistake, unusual shell, or
other unforseen circumstance.

While at it, make some other small simplifications (grouping commands
that write to the same file with { }; using ">file" instead of ": >file"
to empty a file).

Cc: Shawn O. Pearce <spearce@spearce.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |  106 +++++++++++++++++++++++++-----------------------
 1 files changed, 55 insertions(+), 51 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index ed37a4e..735ac83 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -184,58 +184,62 @@ test_expect_success \
 	 test `git rev-parse --verify master:file2` \
 	    = `git rev-parse --verify verify--import-marks:copy-of-file2`'
 
-test_tick
-mt=$(git hash-object --stdin < /dev/null)
-: >input.blob
-: >marks.exp
-: >tree.exp
-
-cat >input.commit <<EOF
-commit refs/heads/verify--dump-marks
-committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
-data <<COMMIT
-test the sparse array dumping routines with exponentially growing marks
-COMMIT
-EOF
-
-i=0
-l=4
-m=6
-n=7
-while test "$i" -lt 27; do
-    cat >>input.blob <<EOF
-blob
-mark :$l
-data 0
-blob
-mark :$m
-data 0
-blob
-mark :$n
-data 0
-EOF
-    echo "M 100644 :$l l$i" >>input.commit
-    echo "M 100644 :$m m$i" >>input.commit
-    echo "M 100644 :$n n$i" >>input.commit
-
-    echo ":$l $mt" >>marks.exp
-    echo ":$m $mt" >>marks.exp
-    echo ":$n $mt" >>marks.exp
-
-    printf "100644 blob $mt\tl$i\n" >>tree.exp
-    printf "100644 blob $mt\tm$i\n" >>tree.exp
-    printf "100644 blob $mt\tn$i\n" >>tree.exp
-
-    l=$(($l + $l))
-    m=$(($m + $m))
-    n=$(($l + $n))
-
-    i=$((1 + $i))
-done
-
-sort tree.exp > tree.exp_s
-
 test_expect_success 'A: export marks with large values' '
+	test_tick &&
+	mt=$(git hash-object --stdin </dev/null) &&
+	>input.blob &&
+	>marks.exp &&
+	>tree.exp &&
+
+	cat >input.commit <<-EOF &&
+	commit refs/heads/verify--dump-marks
+	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	test the sparse array dumping routines with exponentially growing marks
+	COMMIT
+	EOF
+
+	(
+		i=0 &&
+		l=4 &&
+		m=6 &&
+		n=7 &&
+		while test "$i" -lt 27
+		do
+			cat >>input.blob <<-EOF &&
+			blob
+			mark :$l
+			data 0
+			blob
+			mark :$m
+			data 0
+			blob
+			mark :$n
+			data 0
+			EOF
+			{
+				echo "M 100644 :$l l$i" &&
+				echo "M 100644 :$m m$i" &&
+				echo "M 100644 :$n n$i"
+			} >>input.commit &&
+			{
+				echo ":$l $mt" &&
+				echo ":$m $mt" &&
+				echo ":$n $mt"
+			} >>marks.exp &&
+			{
+				printf "100644 blob $mt\tl$i\n" &&
+				printf "100644 blob $mt\tm$i\n" &&
+				printf "100644 blob $mt\tn$i\n"
+			} >>tree.exp &&
+			l=$(($l + $l)) &&
+			m=$(($m + $m)) &&
+			n=$(($l + $n)) &&
+			i=$((1 + $i)) ||
+			exit
+		done
+	) &&
+	sort tree.exp >tree.exp_s &&
 	cat input.blob input.commit | git fast-import --export-marks=marks.large &&
 	git ls-tree refs/heads/verify--dump-marks >tree.out &&
 	test_cmp tree.exp_s tree.out &&
-- 
1.7.2.3

[PATCH 04/24] t9300 (fast-import): check exit status from upstream of pipes

[Jonathan Nieder]

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 735ac83..47339a3 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -97,7 +97,8 @@ initial
 EOF
 test_expect_success \
 	'A: verify commit' \
-	'git cat-file commit master | sed 1d >actual &&
+	'git cat-file commit master >commit &&
+	sed 1d <commit >actual &&
 	test_cmp expect actual'
 
 cat >expect <<EOF
@@ -107,7 +108,8 @@ cat >expect <<EOF
 EOF
 test_expect_success \
 	'A: verify tree' \
-	'git cat-file -p master^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
+	'git cat-file -p master^{tree} >tree &&
+	 sed "s/ [0-9a-f]*	/ /" <tree >actual &&
 	 test_cmp expect actual'
 
 echo "$file2_data" >expect
-- 
1.7.2.3

[PATCH 05/24] t9300 (fast-import): check exit status from command substitutions

[Jonathan Nieder]

If "git rev-parse" starts to crash or misbehave, we want to
notice.

Cc: Shawn O. Pearce <spearce@spearce.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   34 +++++++++++++++++++++-------------
 1 files changed, 21 insertions(+), 13 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 47339a3..c113ddf 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -127,24 +127,32 @@ test_expect_success \
 	'A: verify file4' \
 	'git cat-file blob master:file4 >actual && test_cmp expect actual'
 
-cat >expect <<EOF
-object $(git rev-parse refs/heads/master)
-type commit
-tag series-A
-
-An annotated tag without a tagger
-EOF
 test_expect_success 'A: verify tag/series-A' '
+	master=$(git rev-parse --verify refs/heads/master) &&
+	cat >expect <<-EOF &&
+	object $master
+	type commit
+	tag series-A
+
+	An annotated tag without a tagger
+	EOF
 	git cat-file tag tags/series-A >actual &&
 	test_cmp expect actual
 '
 
-cat >expect <<EOF
-:2 `git rev-parse --verify master:file2`
-:3 `git rev-parse --verify master:file3`
-:4 `git rev-parse --verify master:file4`
-:5 `git rev-parse --verify master^0`
-EOF
+test_expect_success 'setup: compute expected marks' '
+	{
+		printf ":2 " &&
+		git rev-parse --verify master:file2 &&
+		printf ":3 " &&
+		git rev-parse --verify master:file3 &&
+		printf ":4 " &&
+		git rev-parse --verify master:file4 &&
+		printf ":5 " &&
+		git rev-parse --verify master^0
+	} >expect
+'
+
 test_expect_success \
 	'A: verify marks output' \
 	'test_cmp expect marks.out'
-- 
1.7.2.3

[PATCH 06/24] t9300 (fast-import): use test_cmp in place of test $(foo) = $(bar)

[Jonathan Nieder]

This has two benefits: it ensures that the exit status from the
output-producing commands is checked and it produces more helpful
output when the output does not match and tests are being run with -v.

Cc: Shawn O. Pearce <spearce@spearce.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   13 ++++++-------
 1 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index c113ddf..8d418e4 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -184,15 +184,14 @@ test_expect_success \
 test_expect_success \
 	'A: verify pack' \
 	'verify_packs'
-cat >expect <<EOF
-:000000 100755 $zeroes $file2_id A	copy-of-file2
-EOF
-git diff-tree -M -r master verify--import-marks >actual
 test_expect_success \
 	'A: verify diff' \
-	'compare_diff_raw expect actual &&
-	 test `git rev-parse --verify master:file2` \
-	    = `git rev-parse --verify verify--import-marks:copy-of-file2`'
+	'echo ":000000 100755 $zeroes $file2_id A	copy-of-file2" >expect &&
+	 echo $file2_id >expect.copy &&
+	 git diff-tree -M -r master verify--import-marks >actual &&
+	 git rev-parse --verify verify--import-marks:copy-of-file2 >actual.copy &&
+	 compare_diff_raw expect actual &&
+	 test_cmp expect.copy actual.copy'
 
 test_expect_success 'A: export marks with large values' '
 	test_tick &&
-- 
1.7.2.3

[PATCH 07/24] t9300 (fast-import): use tabs to indent

[Jonathan Nieder]

The test script mostly uses tabs to indent, but some 4-space indents
snuck in.

Cc: Shawn O. Pearce <spearce@spearce.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |  118 ++++++++++++++++++++++++------------------------
 1 files changed, 59 insertions(+), 59 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 8d418e4..0059298 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -82,8 +82,8 @@ EOF
 
 INPUT_END
 test_expect_success \
-    'A: create pack from stdin' \
-    'git fast-import --export-marks=marks.out <input &&
+	'A: create pack from stdin' \
+	'git fast-import --export-marks=marks.out <input &&
 	 git whatchanged master'
 test_expect_success \
 	'A: verify pack' \
@@ -272,7 +272,7 @@ M 755 0000000000000000000000000000000000000001 zero1
 
 INPUT_END
 test_expect_success 'B: fail on invalid blob sha1' '
-    test_must_fail git fast-import <input
+	test_must_fail git fast-import <input
 '
 rm -f .git/objects/pack_* .git/objects/index_*
 
@@ -287,7 +287,7 @@ from refs/heads/master
 
 INPUT_END
 test_expect_success 'B: fail on invalid branch name ".badbranchname"' '
-    test_must_fail git fast-import <input
+	test_must_fail git fast-import <input
 '
 rm -f .git/objects/pack_* .git/objects/index_*
 
@@ -302,7 +302,7 @@ from refs/heads/master
 
 INPUT_END
 test_expect_success 'B: fail on invalid branch name "bad[branch]name"' '
-    test_must_fail git fast-import <input
+	test_must_fail git fast-import <input
 '
 rm -f .git/objects/pack_* .git/objects/index_*
 
@@ -317,8 +317,8 @@ from refs/heads/master
 
 INPUT_END
 test_expect_success \
-    'B: accept branch name "TEMP_TAG"' \
-    'git fast-import <input &&
+	'B: accept branch name "TEMP_TAG"' \
+	'git fast-import <input &&
 	 test -f .git/TEMP_TAG &&
 	 test `git rev-parse master` = `git rev-parse TEMP_TAG^`'
 rm -f .git/TEMP_TAG
@@ -344,8 +344,8 @@ D file3
 
 INPUT_END
 test_expect_success \
-    'C: incremental import create pack from stdin' \
-    'git fast-import <input &&
+	'C: incremental import create pack from stdin' \
+	'git fast-import <input &&
 	 git whatchanged branch'
 test_expect_success \
 	'C: verify pack' \
@@ -402,8 +402,8 @@ EOF
 
 INPUT_END
 test_expect_success \
-    'D: inline data in commit' \
-    'git fast-import <input &&
+	'D: inline data in commit' \
+	'git fast-import <input &&
 	 git whatchanged branch'
 test_expect_success \
 	'D: verify pack' \
@@ -446,11 +446,11 @@ from refs/heads/branch^0
 
 INPUT_END
 test_expect_success 'E: rfc2822 date, --date-format=raw' '
-    test_must_fail git fast-import --date-format=raw <input
+	test_must_fail git fast-import --date-format=raw <input
 '
 test_expect_success \
-    'E: rfc2822 date, --date-format=rfc2822' \
-    'git fast-import --date-format=rfc2822 <input'
+	'E: rfc2822 date, --date-format=rfc2822' \
+	'git fast-import --date-format=rfc2822 <input'
 test_expect_success \
 	'E: verify pack' \
 	'verify_packs'
@@ -486,8 +486,8 @@ from refs/heads/branch
 
 INPUT_END
 test_expect_success \
-    'F: non-fast-forward update skips' \
-    'if git fast-import <input
+	'F: non-fast-forward update skips' \
+	'if git fast-import <input
 	 then
 		echo BAD gfi did not fail
 		return 1
@@ -536,8 +536,8 @@ from refs/heads/branch~1
 
 INPUT_END
 test_expect_success \
-    'G: non-fast-forward update forced' \
-    'git fast-import --force <input'
+	'G: non-fast-forward update forced' \
+	'git fast-import --force <input'
 test_expect_success \
 	'G: verify pack' \
 	'verify_packs'
@@ -572,8 +572,8 @@ EOF
 
 INPUT_END
 test_expect_success \
-    'H: deletall, add 1' \
-    'git fast-import <input &&
+	'H: deletall, add 1' \
+	'git fast-import <input &&
 	 git whatchanged H'
 test_expect_success \
 	'H: verify pack' \
@@ -612,8 +612,8 @@ from refs/heads/branch
 
 INPUT_END
 test_expect_success \
-    'I: export-pack-edges' \
-    'git fast-import --export-pack-edges=edges.list <input'
+	'I: export-pack-edges' \
+	'git fast-import --export-pack-edges=edges.list <input'
 
 cat >expect <<EOF
 .git/objects/pack/pack-.pack: `git rev-parse --verify export-boundary`
@@ -646,8 +646,8 @@ COMMIT
 
 INPUT_END
 test_expect_success \
-    'J: reset existing branch creates empty commit' \
-    'git fast-import <input'
+	'J: reset existing branch creates empty commit' \
+	'git fast-import <input'
 test_expect_success \
 	'J: branch has 1 commit, empty tree' \
 	'test 1 = `git rev-list J | wc -l` &&
@@ -676,12 +676,12 @@ from refs/heads/branch^1
 
 INPUT_END
 test_expect_success \
-    'K: reinit branch with from' \
-    'git fast-import <input'
+	'K: reinit branch with from' \
+	'git fast-import <input'
 test_expect_success \
-    'K: verify K^1 = branch^1' \
-    'test `git rev-parse --verify branch^1` \
-		= `git rev-parse --verify K^1`'
+	'K: verify K^1 = branch^1' \
+	'test `git rev-parse --verify branch^1` \
+	    = `git rev-parse --verify K^1`'
 
 ###
 ### series L
@@ -751,7 +751,7 @@ cat >expect <<EXPECT_END
 EXPECT_END
 
 test_expect_success \
-    'L: verify internal tree sorting' \
+	'L: verify internal tree sorting' \
 	'git fast-import <input &&
 	 git diff-tree L^ L >output &&
 	 test_cmp expect output'
@@ -1227,7 +1227,7 @@ DATA
 INPUT_END
 
 test_expect_success 'P: fail on inline gitlink' '
-    test_must_fail git fast-import <input'
+	test_must_fail git fast-import <input'
 
 test_tick
 cat >input <<INPUT_END
@@ -1250,7 +1250,7 @@ M 160000 :1 sub
 INPUT_END
 
 test_expect_success 'P: fail on blob mark in gitlink' '
-    test_must_fail git fast-import <input'
+	test_must_fail git fast-import <input'
 
 ###
 ### series Q (notes)
@@ -1598,14 +1598,14 @@ hi
 EOF
 
 test_expect_success \
-    'R: export-marks feature results in a marks file being created' \
-    'cat input | git fast-import &&
-    grep :1 git.marks'
+	'R: export-marks feature results in a marks file being created' \
+	'cat input | git fast-import &&
+	grep :1 git.marks'
 
 test_expect_success \
-    'R: export-marks options can be overriden by commandline options' \
-    'cat input | git fast-import --export-marks=other.marks &&
-    grep :1 other.marks'
+	'R: export-marks options can be overriden by commandline options' \
+	'cat input | git fast-import --export-marks=other.marks &&
+	grep :1 other.marks'
 
 cat >input << EOF
 feature import-marks=marks.out
@@ -1613,9 +1613,9 @@ feature export-marks=marks.new
 EOF
 
 test_expect_success \
-    'R: import to output marks works without any content' \
-    'cat input | git fast-import &&
-    test_cmp marks.out marks.new'
+	'R: import to output marks works without any content' \
+	'cat input | git fast-import &&
+	test_cmp marks.out marks.new'
 
 cat >input <<EOF
 feature import-marks=nonexistant.marks
@@ -1623,9 +1623,9 @@ feature export-marks=marks.new
 EOF
 
 test_expect_success \
-    'R: import marks prefers commandline marks file over the stream' \
-    'cat input | git fast-import --import-marks=marks.out &&
-    test_cmp marks.out marks.new'
+	'R: import marks prefers commandline marks file over the stream' \
+	'cat input | git fast-import --import-marks=marks.out &&
+	test_cmp marks.out marks.new'
 
 
 cat >input <<EOF
@@ -1634,10 +1634,10 @@ feature export-marks=combined.marks
 EOF
 
 test_expect_success 'R: multiple --import-marks= should be honoured' '
-    head -n2 marks.out > one.marks &&
-    tail -n +3 marks.out > two.marks &&
-    git fast-import --import-marks=one.marks --import-marks=two.marks <input &&
-    test_cmp marks.out combined.marks
+	head -n2 marks.out > one.marks &&
+	tail -n +3 marks.out > two.marks &&
+	git fast-import --import-marks=one.marks --import-marks=two.marks <input &&
+	test_cmp marks.out combined.marks
 '
 
 cat >input <<EOF
@@ -1647,10 +1647,10 @@ feature export-marks=relative.out
 EOF
 
 test_expect_success 'R: feature relative-marks should be honoured' '
-    mkdir -p .git/info/fast-import/ &&
-    cp marks.new .git/info/fast-import/relative.in &&
-    git fast-import <input &&
-    test_cmp marks.new .git/info/fast-import/relative.out
+	mkdir -p .git/info/fast-import/ &&
+	cp marks.new .git/info/fast-import/relative.in &&
+	git fast-import <input &&
+	test_cmp marks.new .git/info/fast-import/relative.out
 '
 
 cat >input <<EOF
@@ -1661,8 +1661,8 @@ feature export-marks=non-relative.out
 EOF
 
 test_expect_success 'R: feature no-relative-marks should be honoured' '
-    git fast-import <input &&
-    test_cmp marks.new non-relative.out
+	git fast-import <input &&
+	test_cmp marks.new non-relative.out
 '
 
 cat >input << EOF
@@ -1676,8 +1676,8 @@ EOF
 touch empty
 
 test_expect_success 'R: quiet option results in no stats being output' '
-    cat input | git fast-import 2> output &&
-    test_cmp empty output
+	cat input | git fast-import 2> output &&
+	test_cmp empty output
 '
 
 cat >input <<EOF
@@ -1685,11 +1685,11 @@ option git non-existing-option
 EOF
 
 test_expect_success 'R: die on unknown option' '
-    test_must_fail git fast-import <input
+	test_must_fail git fast-import <input
 '
 
 test_expect_success 'R: unknown commandline options are rejected' '\
-    test_must_fail git fast-import --non-existing-option < /dev/null
+	test_must_fail git fast-import --non-existing-option < /dev/null
 '
 
 cat >input <<EOF
@@ -1697,7 +1697,7 @@ option non-existing-vcs non-existing-option
 EOF
 
 test_expect_success 'R: ignore non-git options' '
-    git fast-import <input
+	git fast-import <input
 '
 
 ##
-- 
1.7.2.3

[PATCH 08/24] t9300 (fast-import), series A: re-indent

[Jonathan Nieder]

This was originally an old-fashioned test script, with formatting
that might now look unfamiliar:

	test_expect_success \
		'series A: foo bar baz' \
		'test commands &&
		 more test commands'

The apostrophes after tab make indenting more trouble than it ought to
be and the alignment cannot be preserved with the <<- operator
working.  The initial and final apostrophes on command lines makes
patches that add new lines to a test harder to read.  The title
aligned with the test code makes the test description harder to
take in at a glance.

So switch to the usual modern style:

	test_expect_success 'series A: foo bar baz' '
		test commands &&
		more test commands
	'

No other change intended.

Cc: Shawn O. Pearce <spearce@spearce.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   99 ++++++++++++++++++++++++-----------------------
 1 files changed, 51 insertions(+), 48 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 0059298..10dc720 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -81,13 +81,12 @@ An annotated tag without a tagger
 EOF
 
 INPUT_END
-test_expect_success \
-	'A: create pack from stdin' \
-	'git fast-import --export-marks=marks.out <input &&
-	 git whatchanged master'
-test_expect_success \
-	'A: verify pack' \
-	'verify_packs'
+test_expect_success 'A: create pack from stdin' '
+	git fast-import --export-marks=marks.out <input &&
+	git whatchanged master'
+test_expect_success 'A: verify pack' '
+	verify_packs
+'
 
 cat >expect <<EOF
 author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
@@ -95,37 +94,40 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 initial
 EOF
-test_expect_success \
-	'A: verify commit' \
-	'git cat-file commit master >commit &&
+test_expect_success 'A: verify commit' '
+	git cat-file commit master >commit &&
 	sed 1d <commit >actual &&
-	test_cmp expect actual'
+	test_cmp expect actual
+'
 
 cat >expect <<EOF
 100644 blob file2
 100644 blob file3
 100755 blob file4
 EOF
-test_expect_success \
-	'A: verify tree' \
-	'git cat-file -p master^{tree} >tree &&
-	 sed "s/ [0-9a-f]*	/ /" <tree >actual &&
-	 test_cmp expect actual'
+test_expect_success 'A: verify tree' '
+	git cat-file -p master^{tree} >tree &&
+	sed "s/ [0-9a-f]*	/ /" <tree >actual &&
+	test_cmp expect actual
+'
 
 echo "$file2_data" >expect
-test_expect_success \
-	'A: verify file2' \
-	'git cat-file blob master:file2 >actual && test_cmp expect actual'
+test_expect_success 'A: verify file2' '
+	git cat-file blob master:file2 >actual &&
+	test_cmp expect actual
+'
 
 echo "$file3_data" >expect
-test_expect_success \
-	'A: verify file3' \
-	'git cat-file blob master:file3 >actual && test_cmp expect actual'
+test_expect_success 'A: verify file3' '
+	git cat-file blob master:file3 >actual &&
+	test_cmp expect actual
+'
 
 printf "$file4_data" >expect
-test_expect_success \
-	'A: verify file4' \
-	'git cat-file blob master:file4 >actual && test_cmp expect actual'
+test_expect_success 'A: verify file4' '
+	git cat-file blob master:file4 >actual &&
+	test_cmp expect actual
+'
 
 test_expect_success 'A: verify tag/series-A' '
 	master=$(git rev-parse --verify refs/heads/master) &&
@@ -153,17 +155,17 @@ test_expect_success 'setup: compute expected marks' '
 	} >expect
 '
 
-test_expect_success \
-	'A: verify marks output' \
-	'test_cmp expect marks.out'
+test_expect_success 'A: verify marks output' '
+	test_cmp expect marks.out
+'
 
-test_expect_success \
-	'A: verify marks import' \
-	'git fast-import \
+test_expect_success 'A: verify marks import' '
+	git fast-import \
 		--import-marks=marks.out \
 		--export-marks=marks.new \
 		</dev/null &&
-	test_cmp expect marks.new'
+	test_cmp expect marks.new
+'
 
 test_tick
 cat >input <<INPUT_END
@@ -177,21 +179,21 @@ from :5
 M 755 :2 copy-of-file2
 
 INPUT_END
-test_expect_success \
-	'A: verify marks import does not crash' \
-	'git fast-import --import-marks=marks.out <input &&
-	 git whatchanged verify--import-marks'
-test_expect_success \
-	'A: verify pack' \
-	'verify_packs'
-test_expect_success \
-	'A: verify diff' \
-	'echo ":000000 100755 $zeroes $file2_id A	copy-of-file2" >expect &&
-	 echo $file2_id >expect.copy &&
-	 git diff-tree -M -r master verify--import-marks >actual &&
-	 git rev-parse --verify verify--import-marks:copy-of-file2 >actual.copy &&
-	 compare_diff_raw expect actual &&
-	 test_cmp expect.copy actual.copy'
+test_expect_success 'A: verify marks import does not crash' '
+	git fast-import --import-marks=marks.out <input &&
+	git whatchanged verify--import-marks
+'
+test_expect_success 'A: verify pack' '
+	verify_packs
+'
+test_expect_success 'A: verify diff' '
+	echo ":000000 100755 $zeroes $file2_id A	copy-of-file2" >expect &&
+	echo $file2_id >expect.copy &&
+	git diff-tree -M -r master verify--import-marks >actual &&
+	git rev-parse --verify verify--import-marks:copy-of-file2 >actual.copy &&
+	compare_diff_raw expect actual &&
+	test_cmp expect.copy actual.copy
+'
 
 test_expect_success 'A: export marks with large values' '
 	test_tick &&
@@ -252,7 +254,8 @@ test_expect_success 'A: export marks with large values' '
 	cat input.blob input.commit | git fast-import --export-marks=marks.large &&
 	git ls-tree refs/heads/verify--dump-marks >tree.out &&
 	test_cmp tree.exp_s tree.out &&
-	test_cmp marks.exp marks.large'
+	test_cmp marks.exp marks.large
+'
 
 ###
 ### series B
-- 
1.7.2.3

[PATCH 09/24] t9300 (fast-import), series B: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 10dc720..a2b8950 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -319,11 +319,11 @@ COMMIT
 from refs/heads/master
 
 INPUT_END
-test_expect_success \
-	'B: accept branch name "TEMP_TAG"' \
-	'git fast-import <input &&
-	 test -f .git/TEMP_TAG &&
-	 test `git rev-parse master` = `git rev-parse TEMP_TAG^`'
+test_expect_success 'B: accept branch name "TEMP_TAG"' '
+	git fast-import <input &&
+	test -f .git/TEMP_TAG &&
+	test `git rev-parse master` = `git rev-parse TEMP_TAG^`
+'
 rm -f .git/TEMP_TAG
 
 ###
-- 
1.7.2.3

[PATCH 10/24] t9300 (fast-import), series C: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   36 ++++++++++++++++++------------------
 1 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index a2b8950..f8856b4 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -346,17 +346,17 @@ M 755 $newf file2/newf
 D file3
 
 INPUT_END
-test_expect_success \
-	'C: incremental import create pack from stdin' \
-	'git fast-import <input &&
-	 git whatchanged branch'
-test_expect_success \
-	'C: verify pack' \
-	'verify_packs'
-test_expect_success \
-	'C: validate reuse existing blob' \
-	'test $newf = `git rev-parse --verify branch:file2/newf`
-	 test $oldf = `git rev-parse --verify branch:file2/oldf`'
+test_expect_success 'C: incremental import create pack from stdin' '
+	git fast-import <input &&
+	git whatchanged branch
+'
+test_expect_success 'C: verify pack' '
+	verify_packs
+'
+test_expect_success 'C: validate reuse existing blob' '
+	test $newf = `git rev-parse --verify branch:file2/newf`
+	test $oldf = `git rev-parse --verify branch:file2/oldf`
+'
 
 cat >expect <<EOF
 parent `git rev-parse --verify master^0`
@@ -365,10 +365,10 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 second
 EOF
-test_expect_success \
-	'C: verify commit' \
-	'git cat-file commit branch | sed 1d >actual &&
-	 test_cmp expect actual'
+test_expect_success 'C: verify commit' '
+	git cat-file commit branch | sed 1d >actual &&
+	test_cmp expect actual
+'
 
 cat >expect <<EOF
 :000000 100755 $zeroes $newf A	file2/newf
@@ -376,9 +376,9 @@ cat >expect <<EOF
 :100644 000000 $file3_id $zeroes D	file3
 EOF
 git diff-tree -M -r master branch >actual
-test_expect_success \
-	'C: validate rename result' \
-	'compare_diff_raw expect actual'
+test_expect_success 'C: validate rename result' '
+	compare_diff_raw expect actual
+'
 
 ###
 ### series D
-- 
1.7.2.3

[PATCH 11/24] t9300 (fast-import), series D: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   36 ++++++++++++++++++------------------
 1 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index f8856b4..78ea775 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -404,34 +404,34 @@ $file6_data
 EOF
 
 INPUT_END
-test_expect_success \
-	'D: inline data in commit' \
-	'git fast-import <input &&
-	 git whatchanged branch'
-test_expect_success \
-	'D: verify pack' \
-	'verify_packs'
+test_expect_success 'D: inline data in commit' '
+	git fast-import <input &&
+	git whatchanged branch
+'
+test_expect_success 'D: verify pack' '
+	verify_packs
+'
 
 cat >expect <<EOF
 :000000 100755 $zeroes $file6_id A	newdir/exec.sh
 :000000 100644 $zeroes $file5_id A	newdir/interesting
 EOF
 git diff-tree -M -r branch^ branch >actual
-test_expect_success \
-	'D: validate new files added' \
-	'compare_diff_raw expect actual'
+test_expect_success 'D: validate new files added' '
+	compare_diff_raw expect actual
+'
 
 echo "$file5_data" >expect
-test_expect_success \
-	'D: verify file5' \
-	'git cat-file blob branch:newdir/interesting >actual &&
-	 test_cmp expect actual'
+test_expect_success 'D: verify file5' '
+	git cat-file blob branch:newdir/interesting >actual &&
+	test_cmp expect actual
+'
 
 echo "$file6_data" >expect
-test_expect_success \
-	'D: verify file6' \
-	'git cat-file blob branch:newdir/exec.sh >actual &&
-	 test_cmp expect actual'
+test_expect_success 'D: verify file6' '
+	git cat-file blob branch:newdir/exec.sh >actual &&
+	test_cmp expect actual
+'
 
 ###
 ### series E
-- 
1.7.2.3

[PATCH 12/24] t9300 (fast-import), series E: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 78ea775..1324157 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -451,12 +451,12 @@ INPUT_END
 test_expect_success 'E: rfc2822 date, --date-format=raw' '
 	test_must_fail git fast-import --date-format=raw <input
 '
-test_expect_success \
-	'E: rfc2822 date, --date-format=rfc2822' \
-	'git fast-import --date-format=rfc2822 <input'
-test_expect_success \
-	'E: verify pack' \
-	'verify_packs'
+test_expect_success 'E: rfc2822 date, --date-format=rfc2822' '
+	git fast-import --date-format=rfc2822 <input
+'
+test_expect_success 'E: verify pack' '
+	verify_packs
+'
 
 cat >expect <<EOF
 author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> 1170778938 -0500
@@ -464,10 +464,10 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1170783302 -0500
 
 RFC 2822 type date
 EOF
-test_expect_success \
-	'E: verify commit' \
-	'git cat-file commit branch | sed 1,2d >actual &&
-	test_cmp expect actual'
+test_expect_success 'E: verify commit' '
+	git cat-file commit branch | sed 1,2d >actual &&
+	test_cmp expect actual
+'
 
 ###
 ### series F
-- 
1.7.2.3

[PATCH 13/24] t9300 (fast-import), series F: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   47 +++++++++++++++++++++++------------------------
 1 files changed, 23 insertions(+), 24 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 1324157..ab317d6 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -488,13 +488,12 @@ reset refs/heads/other
 from refs/heads/branch
 
 INPUT_END
-test_expect_success \
-	'F: non-fast-forward update skips' \
-	'if git fast-import <input
-	 then
+test_expect_success 'F: non-fast-forward update skips' '
+	if git fast-import <input
+	then
 		echo BAD gfi did not fail
 		return 1
-	 else
+	else
 		if test $old_branch = `git rev-parse --verify branch^0`
 		then
 			: branch unaffected and failure returned
@@ -503,11 +502,11 @@ test_expect_success \
 			echo BAD gfi changed branch $old_branch
 			return 1
 		fi
-	 fi
-	'
-test_expect_success \
-	'F: verify pack' \
-	'verify_packs'
+	fi
+'
+test_expect_success 'F: verify pack' '
+	verify_packs
+'
 
 cat >expect <<EOF
 tree `git rev-parse branch~1^{tree}`
@@ -517,10 +516,10 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 losing things already?
 EOF
-test_expect_success \
-	'F: verify other commit' \
-	'git cat-file commit other >actual &&
-	test_cmp expect actual'
+test_expect_success 'F: verify other commit' '
+	git cat-file commit other >actual &&
+	test_cmp expect actual
+'
 
 ###
 ### series G
@@ -538,16 +537,16 @@ COMMIT
 from refs/heads/branch~1
 
 INPUT_END
-test_expect_success \
-	'G: non-fast-forward update forced' \
-	'git fast-import --force <input'
-test_expect_success \
-	'G: verify pack' \
-	'verify_packs'
-test_expect_success \
-	'G: branch changed, but logged' \
-	'test $old_branch != `git rev-parse --verify branch^0` &&
-	 test $old_branch = `git rev-parse --verify branch@{1}`'
+test_expect_success 'G: non-fast-forward update forced' '
+	git fast-import --force <input
+'
+test_expect_success 'G: verify pack' '
+	verify_packs
+'
+test_expect_success 'G: branch changed, but logged' '
+	test $old_branch != `git rev-parse --verify branch^0` &&
+	test $old_branch = `git rev-parse --verify branch@{1}`
+'
 
 ###
 ### series H
-- 
1.7.2.3

Re: [PATCH 08/24] t9300 (fast-import), series A: re-indent

[Sverre Rabbelier]

Heya,

On Fri, Sep 24, 2010 at 09:16, Jonathan Nieder <jrnieder@gmail.com> wrote:
>        test_expect_success \
>                'series A: foo bar baz' \

Sometimes this style is used if the test description is otherwise too long.

>        test_expect_success 'series A: foo bar baz' '

So what do you do in the case that the line becomes too long?

-- 
Cheers,

Sverre Rabbelier

[PATCH 14/24] t9300 (fast-import), series H: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   28 ++++++++++++++--------------
 1 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index ab317d6..875a951 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -573,13 +573,13 @@ $file5_data
 EOF
 
 INPUT_END
-test_expect_success \
-	'H: deletall, add 1' \
-	'git fast-import <input &&
-	 git whatchanged H'
-test_expect_success \
-	'H: verify pack' \
-	'verify_packs'
+test_expect_success 'H: deletall, add 1' '
+	git fast-import <input &&
+	git whatchanged H
+'
+test_expect_success 'H: verify pack' '
+	verify_packs
+'
 
 cat >expect <<EOF
 :100755 000000 $newf $zeroes D	file2/newf
@@ -589,15 +589,15 @@ cat >expect <<EOF
 :100755 000000 $file6_id $zeroes D	newdir/exec.sh
 EOF
 git diff-tree -M -r H^ H >actual
-test_expect_success \
-	'H: validate old files removed, new files added' \
-	'compare_diff_raw expect actual'
+test_expect_success 'H: validate old files removed, new files added' '
+	compare_diff_raw expect actual
+'
 
 echo "$file5_data" >expect
-test_expect_success \
-	'H: verify file' \
-	'git cat-file blob H:h/e/l/lo >actual &&
-	 test_cmp expect actual'
+test_expect_success 'H: verify file' '
+	git cat-file blob H:h/e/l/lo >actual &&
+	test_cmp expect actual
+'
 
 ###
 ### series I
-- 
1.7.2.3

[PATCH 15/24] t9300 (fast-import), series I: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 875a951..fda1911 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -613,17 +613,17 @@ COMMIT
 from refs/heads/branch
 
 INPUT_END
-test_expect_success \
-	'I: export-pack-edges' \
-	'git fast-import --export-pack-edges=edges.list <input'
+test_expect_success 'I: export-pack-edges' '
+	git fast-import --export-pack-edges=edges.list <input
+'
 
 cat >expect <<EOF
 .git/objects/pack/pack-.pack: `git rev-parse --verify export-boundary`
 EOF
-test_expect_success \
-	'I: verify edge list' \
-	'sed -e s/pack-.*pack/pack-.pack/ edges.list >actual &&
-	 test_cmp expect actual'
+test_expect_success 'I: verify edge list' '
+	sed -e s/pack-.*pack/pack-.pack/ edges.list >actual &&
+	test_cmp expect actual
+'
 
 ###
 ### series J
-- 
1.7.2.3

[PATCH 16/24] t9300 (fast-import), series J: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index fda1911..2a2f969 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -647,13 +647,13 @@ initialize J
 COMMIT
 
 INPUT_END
-test_expect_success \
-	'J: reset existing branch creates empty commit' \
-	'git fast-import <input'
-test_expect_success \
-	'J: branch has 1 commit, empty tree' \
-	'test 1 = `git rev-list J | wc -l` &&
-	 test 0 = `git ls-tree J | wc -l`'
+test_expect_success 'J: reset existing branch creates empty commit' '
+	git fast-import <input
+'
+test_expect_success 'J: branch has 1 commit, empty tree' '
+	test 1 = `git rev-list J | wc -l` &&
+	test 0 = `git ls-tree J | wc -l`
+'
 
 ###
 ### series K
-- 
1.7.2.3

[PATCH 17/24] t9300 (fast-import), series K: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 2a2f969..82647f8 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -677,13 +677,13 @@ COMMIT
 from refs/heads/branch^1
 
 INPUT_END
-test_expect_success \
-	'K: reinit branch with from' \
-	'git fast-import <input'
-test_expect_success \
-	'K: verify K^1 = branch^1' \
-	'test `git rev-parse --verify branch^1` \
-	    = `git rev-parse --verify K^1`'
+test_expect_success 'K: reinit branch with from' '
+	git fast-import <input
+'
+test_expect_success 'K: verify K^1 = branch^1' '
+	test `git rev-parse --verify branch^1` \
+	   = `git rev-parse --verify K^1`
+'
 
 ###
 ### series L
-- 
1.7.2.3

[PATCH 18/24] t9300 (fast-import), series L: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 82647f8..614c5b5 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -752,11 +752,11 @@ cat >expect <<EXPECT_END
 :100644 100644 $some_data $other_data M	ba
 EXPECT_END
 
-test_expect_success \
-	'L: verify internal tree sorting' \
-	'git fast-import <input &&
-	 git diff-tree L^ L >output &&
-	 test_cmp expect output'
+test_expect_success 'L: verify internal tree sorting' '
+	git fast-import <input &&
+	git diff-tree L^ L >output &&
+	test_cmp expect output
+'
 
 ###
 ### series M
-- 
1.7.2.3

[PATCH 19/24] t9300 (fast-import), series M: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   30 +++++++++++++++---------------
 1 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 614c5b5..5237afc 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -778,11 +778,11 @@ INPUT_END
 cat >expect <<EOF
 :100755 100755 $newf $newf R100	file2/newf	file2/n.e.w.f
 EOF
-test_expect_success \
-	'M: rename file in same subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -M -r M1^ M1 >actual &&
-	 compare_diff_raw expect actual'
+test_expect_success 'M: rename file in same subdirectory' '
+	git fast-import <input &&
+	git diff-tree -M -r M1^ M1 >actual &&
+	compare_diff_raw expect actual
+'
 
 cat >input <<INPUT_END
 commit refs/heads/M2
@@ -799,11 +799,11 @@ INPUT_END
 cat >expect <<EOF
 :100755 100755 $newf $newf R100	file2/newf	i/am/new/to/you
 EOF
-test_expect_success \
-	'M: rename file to new subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -M -r M2^ M2 >actual &&
-	 compare_diff_raw expect actual'
+test_expect_success 'M: rename file to new subdirectory' '
+	git fast-import <input &&
+	git diff-tree -M -r M2^ M2 >actual &&
+	compare_diff_raw expect actual
+'
 
 cat >input <<INPUT_END
 commit refs/heads/M3
@@ -820,11 +820,11 @@ INPUT_END
 cat >expect <<EOF
 :100755 100755 $newf $newf R100	i/am/new/to/you	other/sub/am/new/to/you
 EOF
-test_expect_success \
-	'M: rename subdirectory to new subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -M -r M3^ M3 >actual &&
-	 compare_diff_raw expect actual'
+test_expect_success 'M: rename subdirectory to new subdirectory' '
+	git fast-import <input &&
+	git diff-tree -M -r M3^ M3 >actual &&
+	compare_diff_raw expect actual
+'
 
 ###
 ### series N
-- 
1.7.2.3

[PATCH 20/24] t9300 (fast-import), series N: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   60 ++++++++++++++++++++++++------------------------
 1 files changed, 30 insertions(+), 30 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 5237afc..7bd2bd0 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -846,11 +846,11 @@ INPUT_END
 cat >expect <<EOF
 :100755 100755 $newf $newf C100	file2/newf	file2/n.e.w.f
 EOF
-test_expect_success \
-	'N: copy file in same subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -C --find-copies-harder -r N1^ N1 >actual &&
-	 compare_diff_raw expect actual'
+test_expect_success 'N: copy file in same subdirectory' '
+	git fast-import <input &&
+	git diff-tree -C --find-copies-harder -r N1^ N1 >actual &&
+	compare_diff_raw expect actual
+'
 
 cat >input <<INPUT_END
 commit refs/heads/N2
@@ -880,11 +880,11 @@ cat >expect <<EOF
 :100755 100755 $newf $newf C100	file2/newf	file3/newf
 :100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
 EOF
-test_expect_success \
-	'N: copy then modify subdirectory' \
-	'git fast-import <input &&
-	 git diff-tree -C --find-copies-harder -r N2^^ N2 >actual &&
-	 compare_diff_raw expect actual'
+test_expect_success 'N: copy then modify subdirectory' '
+	git fast-import <input &&
+	git diff-tree -C --find-copies-harder -r N2^^ N2 >actual &&
+	compare_diff_raw expect actual
+'
 
 cat >input <<INPUT_END
 commit refs/heads/N3
@@ -904,19 +904,18 @@ D file2/file5
 
 INPUT_END
 
-test_expect_success \
-	'N: copy dirty subdirectory' \
-	'git fast-import <input &&
-	 test `git rev-parse N2^{tree}` = `git rev-parse N3^{tree}`'
+test_expect_success 'N: copy dirty subdirectory' '
+	git fast-import <input &&
+	test `git rev-parse N2^{tree}` = `git rev-parse N3^{tree}`
+'
 
-test_expect_success \
-	'N: copy directory by id' \
-	'cat >expect <<-EOF &&
+test_expect_success 'N: copy directory by id' '
+	cat >expect <<-EOF &&
 	:100755 100755 $newf $newf C100	file2/newf	file3/newf
 	:100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
 	EOF
-	 subdir=$(git rev-parse refs/heads/branch^0:file2) &&
-	 cat >input <<-INPUT_END &&
+	subdir=$(git rev-parse refs/heads/branch^0:file2) &&
+	cat >input <<-INPUT_END &&
 	commit refs/heads/N4
 	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 	data <<COMMIT
@@ -926,19 +925,19 @@ test_expect_success \
 	from refs/heads/branch^0
 	M 040000 $subdir file3
 	INPUT_END
-	 git fast-import <input &&
-	 git diff-tree -C --find-copies-harder -r N4^ N4 >actual &&
-	 compare_diff_raw expect actual'
+	git fast-import <input &&
+	git diff-tree -C --find-copies-harder -r N4^ N4 >actual &&
+	compare_diff_raw expect actual
+'
 
-test_expect_success \
-	'N: modify copied tree' \
-	'cat >expect <<-EOF &&
+test_expect_success 'N: modify copied tree' '
+	cat >expect <<-EOF &&
 	:100644 100644 $file5_id $file5_id C100	newdir/interesting	file3/file5
 	:100755 100755 $newf $newf C100	file2/newf	file3/newf
 	:100644 100644 $oldf $oldf C100	file2/oldf	file3/oldf
 	EOF
-	 subdir=$(git rev-parse refs/heads/branch^0:file2) &&
-	 cat >input <<-INPUT_END &&
+	subdir=$(git rev-parse refs/heads/branch^0:file2) &&
+	cat >input <<-INPUT_END &&
 	commit refs/heads/N5
 	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 	data <<COMMIT
@@ -959,9 +958,10 @@ test_expect_success \
 	$file5_data
 	EOF
 	INPUT_END
-	 git fast-import <input &&
-	 git diff-tree -C --find-copies-harder -r N5^^ N5 >actual &&
-	 compare_diff_raw expect actual'
+	git fast-import <input &&
+	git diff-tree -C --find-copies-harder -r N5^^ N5 >actual &&
+	compare_diff_raw expect actual
+'
 
 ###
 ### series O
-- 
1.7.2.3

[PATCH 21/24] t9300 (fast-import), series O: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   46 +++++++++++++++++++++++-----------------------
 1 files changed, 23 insertions(+), 23 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 7bd2bd0..04e4f6d 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -998,10 +998,10 @@ D file2/file5
 
 INPUT_END
 
-test_expect_success \
-	'O: comments are all skipped' \
-	'git fast-import <input &&
-	 test `git rev-parse N3` = `git rev-parse O1`'
+test_expect_success 'O: comments are all skipped' '
+	git fast-import <input &&
+	test `git rev-parse N3` = `git rev-parse O1`
+'
 
 cat >input <<INPUT_END
 commit refs/heads/O2
@@ -1019,14 +1019,14 @@ D file2/file5
 
 INPUT_END
 
-test_expect_success \
-	'O: blank lines not necessary after data commands' \
-	'git fast-import <input &&
-	 test `git rev-parse N3` = `git rev-parse O2`'
+test_expect_success 'O: blank lines not necessary after data commands' '
+	git fast-import <input &&
+	test `git rev-parse N3` = `git rev-parse O2`
+'
 
-test_expect_success \
-	'O: repack before next test' \
-	'git repack -a -d'
+test_expect_success 'O: repack before next test' '
+	git repack -a -d
+'
 
 cat >input <<INPUT_END
 commit refs/heads/O3
@@ -1064,13 +1064,13 @@ of
 empty
 commits
 INPUT_END
-test_expect_success \
-	'O: blank lines not necessary after other commands' \
-	'git fast-import <input &&
-	 test 8 = `find .git/objects/pack -type f | wc -l` &&
-	 test `git rev-parse refs/tags/O3-2nd` = `git rev-parse O3^` &&
-	 git log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&
-	 test_cmp expect actual'
+test_expect_success 'O: blank lines not necessary after other commands' '
+	git fast-import <input &&
+	test 8 = `find .git/objects/pack -type f | wc -l` &&
+	test `git rev-parse refs/tags/O3-2nd` = `git rev-parse O3^` &&
+	git log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&
+	test_cmp expect actual
+'
 
 cat >input <<INPUT_END
 commit refs/heads/O4
@@ -1097,11 +1097,11 @@ zcommits
 COMMIT
 progress I'm done!
 INPUT_END
-test_expect_success \
-	'O: progress outputs as requested by input' \
-	'git fast-import <input >actual &&
-	 grep "progress " <input >expect &&
-	 test_cmp expect actual'
+test_expect_success 'O: progress outputs as requested by input' '
+	git fast-import <input >actual &&
+	grep "progress " <input >expect &&
+	test_cmp expect actual
+'
 
 ###
 ### series P (gitlinks)
-- 
1.7.2.3

[PATCH 22/24] t9300 (fast-import), series P: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |   38 ++++++++++++++++++++------------------
 1 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 04e4f6d..ef432e7 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1162,16 +1162,16 @@ M 160000 :6 sub
 
 INPUT_END
 
-test_expect_success \
-	'P: supermodule & submodule mix' \
-	'git fast-import <input &&
-	 git checkout subuse1 &&
-	 rm -rf sub && mkdir sub && (cd sub &&
-	 git init &&
-	 git fetch --update-head-ok .. refs/heads/sub:refs/heads/master &&
-	 git checkout master) &&
-	 git submodule init &&
-	 git submodule update'
+test_expect_success 'P: supermodule & submodule mix' '
+	git fast-import <input &&
+	git checkout subuse1 &&
+	rm -rf sub && mkdir sub && (cd sub &&
+	git init &&
+	git fetch --update-head-ok .. refs/heads/sub:refs/heads/master &&
+	git checkout master) &&
+	git submodule init &&
+	git submodule update
+'
 
 SUBLAST=$(git rev-parse --verify sub)
 SUBPREV=$(git rev-parse --verify sub^)
@@ -1204,12 +1204,12 @@ M 160000 $SUBLAST sub
 
 INPUT_END
 
-test_expect_success \
-	'P: verbatim SHA gitlinks' \
-	'git branch -D sub &&
-	 git gc && git prune &&
-	 git fast-import <input &&
-	 test $(git rev-parse --verify subuse2) = $(git rev-parse --verify subuse1)'
+test_expect_success 'P: verbatim SHA gitlinks' '
+	git branch -D sub &&
+	git gc && git prune &&
+	git fast-import <input &&
+	test $(git rev-parse --verify subuse2) = $(git rev-parse --verify subuse1)
+'
 
 test_tick
 cat >input <<INPUT_END
@@ -1229,7 +1229,8 @@ DATA
 INPUT_END
 
 test_expect_success 'P: fail on inline gitlink' '
-	test_must_fail git fast-import <input'
+	test_must_fail git fast-import <input
+'
 
 test_tick
 cat >input <<INPUT_END
@@ -1252,7 +1253,8 @@ M 160000 :1 sub
 INPUT_END
 
 test_expect_success 'P: fail on blob mark in gitlink' '
-	test_must_fail git fast-import <input'
+	test_must_fail git fast-import <input
+'
 
 ###
 ### series Q (notes)
-- 
1.7.2.3

[PATCH 23/24] t9300 (fast-import), series Q: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9300-fast-import.sh |  158 +++++++++++++++++++++++++-----------------------
 1 files changed, 83 insertions(+), 75 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index ef432e7..970f594 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1379,13 +1379,13 @@ EOF
 
 INPUT_END
 
-test_expect_success \
-	'Q: commit notes' \
-	'git fast-import <input &&
-	 git whatchanged notes-test'
-test_expect_success \
-	'Q: verify pack' \
-	'verify_packs'
+test_expect_success 'Q: commit notes' '
+	git fast-import <input &&
+	git whatchanged notes-test
+'
+test_expect_success 'Q: verify pack' '
+	verify_packs
+'
 
 commit1=$(git rev-parse notes-test~2)
 commit2=$(git rev-parse notes-test^)
@@ -1397,10 +1397,10 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 first (:3)
 EOF
-test_expect_success \
-	'Q: verify first commit' \
-	'git cat-file commit notes-test~2 | sed 1d >actual &&
-	test_cmp expect actual'
+test_expect_success 'Q: verify first commit' '
+	git cat-file commit notes-test~2 | sed 1d >actual &&
+	test_cmp expect actual
+'
 
 cat >expect <<EOF
 parent $commit1
@@ -1409,10 +1409,10 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 second (:5)
 EOF
-test_expect_success \
-	'Q: verify second commit' \
-	'git cat-file commit notes-test^ | sed 1d >actual &&
-	test_cmp expect actual'
+test_expect_success 'Q: verify second commit' '
+	git cat-file commit notes-test^ | sed 1d >actual &&
+	test_cmp expect actual
+'
 
 cat >expect <<EOF
 parent $commit2
@@ -1421,10 +1421,10 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 third (:6)
 EOF
-test_expect_success \
-	'Q: verify third commit' \
-	'git cat-file commit notes-test | sed 1d >actual &&
-	test_cmp expect actual'
+test_expect_success 'Q: verify third commit' '
+	git cat-file commit notes-test | sed 1d >actual &&
+	test_cmp expect actual
+'
 
 cat >expect <<EOF
 author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
@@ -1432,10 +1432,10 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 notes (:9)
 EOF
-test_expect_success \
-	'Q: verify first notes commit' \
-	'git cat-file commit refs/notes/foobar~2 | sed 1d >actual &&
-	test_cmp expect actual'
+test_expect_success 'Q: verify first notes commit' '
+	git cat-file commit refs/notes/foobar~2 | sed 1d >actual &&
+	test_cmp expect actual
+'
 
 cat >expect.unsorted <<EOF
 100644 blob $commit1
@@ -1443,25 +1443,28 @@ cat >expect.unsorted <<EOF
 100644 blob $commit3
 EOF
 cat expect.unsorted | sort >expect
-test_expect_success \
-	'Q: verify first notes tree' \
-	'git cat-file -p refs/notes/foobar~2^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
+test_expect_success 'Q: verify first notes tree' '
+	git cat-file -p refs/notes/foobar~2^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
+	test_cmp expect actual
+'
 
 echo "$note1_data" >expect
-test_expect_success \
-	'Q: verify first note for first commit' \
-	'git cat-file blob refs/notes/foobar~2:$commit1 >actual && test_cmp expect actual'
+test_expect_success 'Q: verify first note for first commit' '
+	git cat-file blob refs/notes/foobar~2:$commit1 >actual &&
+	test_cmp expect actual
+'
 
 echo "$note2_data" >expect
-test_expect_success \
-	'Q: verify first note for second commit' \
-	'git cat-file blob refs/notes/foobar~2:$commit2 >actual && test_cmp expect actual'
+test_expect_success 'Q: verify first note for second commit' '
+	git cat-file blob refs/notes/foobar~2:$commit2 >actual &&
+	test_cmp expect actual
+'
 
 echo "$note3_data" >expect
-test_expect_success \
-	'Q: verify first note for third commit' \
-	'git cat-file blob refs/notes/foobar~2:$commit3 >actual && test_cmp expect actual'
+test_expect_success 'Q: verify first note for third commit' '
+	git cat-file blob refs/notes/foobar~2:$commit3 >actual &&
+	test_cmp expect actual
+'
 
 cat >expect <<EOF
 parent `git rev-parse --verify refs/notes/foobar~2`
@@ -1470,10 +1473,10 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 notes (:10)
 EOF
-test_expect_success \
-	'Q: verify second notes commit' \
-	'git cat-file commit refs/notes/foobar^ | sed 1d >actual &&
-	test_cmp expect actual'
+test_expect_success 'Q: verify second notes commit' '
+	git cat-file commit refs/notes/foobar^ | sed 1d >actual &&
+	test_cmp expect actual
+'
 
 cat >expect.unsorted <<EOF
 100644 blob $commit1
@@ -1481,25 +1484,28 @@ cat >expect.unsorted <<EOF
 100644 blob $commit3
 EOF
 cat expect.unsorted | sort >expect
-test_expect_success \
-	'Q: verify second notes tree' \
-	'git cat-file -p refs/notes/foobar^^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
+test_expect_success 'Q: verify second notes tree' '
+	git cat-file -p refs/notes/foobar^^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
+	test_cmp expect actual
+'
 
 echo "$note1b_data" >expect
-test_expect_success \
-	'Q: verify second note for first commit' \
-	'git cat-file blob refs/notes/foobar^:$commit1 >actual && test_cmp expect actual'
+test_expect_success 'Q: verify second note for first commit' '
+	git cat-file blob refs/notes/foobar^:$commit1 >actual &&
+	test_cmp expect actual
+'
 
 echo "$note2_data" >expect
-test_expect_success \
-	'Q: verify first note for second commit' \
-	'git cat-file blob refs/notes/foobar^:$commit2 >actual && test_cmp expect actual'
+test_expect_success 'Q: verify first note for second commit' '
+	git cat-file blob refs/notes/foobar^:$commit2 >actual &&
+	test_cmp expect actual
+'
 
 echo "$note3_data" >expect
-test_expect_success \
-	'Q: verify first note for third commit' \
-	'git cat-file blob refs/notes/foobar^:$commit3 >actual && test_cmp expect actual'
+test_expect_success 'Q: verify first note for third commit' '
+	git cat-file blob refs/notes/foobar^:$commit3 >actual &&
+	test_cmp expect actual
+'
 
 cat >expect <<EOF
 author $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
@@ -1507,24 +1513,25 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 notes (:11)
 EOF
-test_expect_success \
-	'Q: verify third notes commit' \
-	'git cat-file commit refs/notes/foobar2 | sed 1d >actual &&
-	test_cmp expect actual'
+test_expect_success 'Q: verify third notes commit' '
+	git cat-file commit refs/notes/foobar2 | sed 1d >actual &&
+	test_cmp expect actual
+'
 
 cat >expect.unsorted <<EOF
 100644 blob $commit1
 EOF
 cat expect.unsorted | sort >expect
-test_expect_success \
-	'Q: verify third notes tree' \
-	'git cat-file -p refs/notes/foobar2^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
+test_expect_success 'Q: verify third notes tree' '
+	git cat-file -p refs/notes/foobar2^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
+	test_cmp expect actual
+'
 
 echo "$note1c_data" >expect
-test_expect_success \
-	'Q: verify third note for first commit' \
-	'git cat-file blob refs/notes/foobar2:$commit1 >actual && test_cmp expect actual'
+test_expect_success 'Q: verify third note for first commit' '
+	git cat-file blob refs/notes/foobar2:$commit1 >actual &&
+	test_cmp expect actual
+'
 
 cat >expect <<EOF
 parent `git rev-parse --verify refs/notes/foobar^`
@@ -1533,24 +1540,25 @@ committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
 
 notes (:12)
 EOF
-test_expect_success \
-	'Q: verify fourth notes commit' \
-	'git cat-file commit refs/notes/foobar | sed 1d >actual &&
-	test_cmp expect actual'
+test_expect_success 'Q: verify fourth notes commit' '
+	git cat-file commit refs/notes/foobar | sed 1d >actual &&
+	test_cmp expect actual
+'
 
 cat >expect.unsorted <<EOF
 100644 blob $commit2
 EOF
 cat expect.unsorted | sort >expect
-test_expect_success \
-	'Q: verify fourth notes tree' \
-	'git cat-file -p refs/notes/foobar^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
-	 test_cmp expect actual'
+test_expect_success 'Q: verify fourth notes tree' '
+	git cat-file -p refs/notes/foobar^{tree} | sed "s/ [0-9a-f]*	/ /" >actual &&
+	test_cmp expect actual
+'
 
 echo "$note2b_data" >expect
-test_expect_success \
-	'Q: verify second note for second commit' \
-	'git cat-file blob refs/notes/foobar:$commit2 >actual && test_cmp expect actual'
+test_expect_success 'Q: verify second note for second commit' '
+	git cat-file blob refs/notes/foobar:$commit2 >actual &&
+	test_cmp expect actual
+'
 
 ###
 ### series R (feature and option)
-- 
1.7.2.3

[PATCH 24/24] t9300 (fast-import), series R: re-indent

[Jonathan Nieder]

See "t9300 (fast-import), series A: re-indent".

Cc: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
That's all for now.  Sorry for the dull read.

Be seeing you,
Jonathan

 t/t9300-fast-import.sh |   72 ++++++++++++++++++++++++------------------------
 1 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 970f594..6c629b4 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -1609,35 +1609,35 @@ hi
 
 EOF
 
-test_expect_success \
-	'R: export-marks feature results in a marks file being created' \
-	'cat input | git fast-import &&
-	grep :1 git.marks'
+test_expect_success 'R: export-marks feature results in a marks file being created' '
+	cat input | git fast-import &&
+	grep :1 git.marks
+'
 
-test_expect_success \
-	'R: export-marks options can be overriden by commandline options' \
-	'cat input | git fast-import --export-marks=other.marks &&
-	grep :1 other.marks'
+test_expect_success 'R: export-marks options can be overriden by commandline options' '
+	cat input | git fast-import --export-marks=other.marks &&
+	grep :1 other.marks
+'
 
 cat >input << EOF
 feature import-marks=marks.out
 feature export-marks=marks.new
 EOF
 
-test_expect_success \
-	'R: import to output marks works without any content' \
-	'cat input | git fast-import &&
-	test_cmp marks.out marks.new'
+test_expect_success 'R: import to output marks works without any content' '
+	cat input | git fast-import &&
+	test_cmp marks.out marks.new
+'
 
 cat >input <<EOF
 feature import-marks=nonexistant.marks
 feature export-marks=marks.new
 EOF
 
-test_expect_success \
-	'R: import marks prefers commandline marks file over the stream' \
-	'cat input | git fast-import --import-marks=marks.out &&
-	test_cmp marks.out marks.new'
+test_expect_success 'R: import marks prefers commandline marks file over the stream' '
+	cat input | git fast-import --import-marks=marks.out &&
+	test_cmp marks.out marks.new
+'
 
 
 cat >input <<EOF
@@ -1700,7 +1700,7 @@ test_expect_success 'R: die on unknown option' '
 	test_must_fail git fast-import <input
 '
 
-test_expect_success 'R: unknown commandline options are rejected' '\
+test_expect_success 'R: unknown commandline options are rejected' '
 	test_must_fail git fast-import --non-existing-option < /dev/null
 '
 
@@ -1735,29 +1735,29 @@ INPUT_END
 cat expect >>input
 echo >>input
 
-test_expect_success \
-	'R: blob bigger than threshold' \
-	'test_create_repo R &&
-	 git --git-dir=R/.git fast-import --big-file-threshold=1 <input'
-test_expect_success \
-	'R: verify created pack' \
-	'(
+test_expect_success 'R: blob bigger than threshold' '
+	test_create_repo R &&
+	git --git-dir=R/.git fast-import --big-file-threshold=1 <input
+'
+test_expect_success 'R: verify created pack' '
+	(
 		for p in R/.git/objects/pack/*.pack
 		do
 			git verify-pack -v $p ||
 			exit
 		done
+	) >verify
+'
-	 ) >verify'
-test_expect_success \
-	'R: verify written objects' \
-	'git --git-dir=R/.git cat-file blob big-file:big1 >actual &&
-	 test_cmp expect actual &&
-	 a=$(git --git-dir=R/.git rev-parse big-file:big1) &&
-	 b=$(git --git-dir=R/.git rev-parse big-file:big2) &&
-	 test $a = $b'
-test_expect_success \
-	'R: blob appears only once' \
-	'n=$(grep $a verify | wc -l) &&
-	 test 1 = $n'
+test_expect_success 'R: verify written objects' '
+	git --git-dir=R/.git cat-file blob big-file:big1 >actual &&
+	test_cmp expect actual &&
+	a=$(git --git-dir=R/.git rev-parse big-file:big1) &&
+	b=$(git --git-dir=R/.git rev-parse big-file:big2) &&
+	test $a = $b
+'
+test_expect_success 'R: blob appears only once' '
+	n=$(grep $a verify | wc -l) &&
+	test 1 = $n
+'
 
 test_done
-- 
1.7.2.3

Re: [PATCH 08/24] t9300 (fast-import), series A: re-indent

[Jonathan Nieder]

Sverre Rabbelier wrote:
> On Fri, Sep 24, 2010 at 09:16, Jonathan Nieder <jrnieder@gmail.com> wrote:

>>        test_expect_success \
>>                'series A: foo bar baz' \
>
> Sometimes this style is used if the test description is otherwise too long.
>
>>        test_expect_success 'series A: foo bar baz' '
>
> So what do you do in the case that the line becomes too long?

In this series that never happens.

In general, either

 test_expect_success \
		'series A: foo bar baz' '
	... commands ...
 '

or

 test_expect_success \
	'series A: foo bar baz' '
	... commands ...
 '

looks fine to me for those cases where no shorter test description
comes quickly to mind.

Re: [PATCH 07/24] t9300 (fast-import): use tabs to indent

[Ramkumar Ramachandra]

Hi Jonathan,

Jonathan Nieder writes:
> The test script mostly uses tabs to indent, but some 4-space indents
> snuck in.

Patches 7-24 are boring non-functional changes that involve re-styling
and re-indenting. Maybe put them all in one large commit? It's fairly
trivial to review these changes anyway.

-- Ram

Re: [PATCH 07/24] t9300 (fast-import): use tabs to indent

[Jonathan Nieder]

Ramkumar Ramachandra wrote:

> Patches 7-24 are boring non-functional changes that involve re-styling
> and re-indenting. Maybe put them all in one large commit? It's fairly
> trivial to review these changes anyway.

Yeah, sorry for the mailbomb.  Whichever changes are wanted are meant
for squashing indeed.

Re: [PATCH 03/24] t9300 (fast-import): guard "export large marks" test setup

[Ramkumar Ramachandra]

Hi,

Jonathan Nieder writes:
>  test_expect_success 'A: export marks with large values' '
> +	test_tick &&
> +	mt=$(git hash-object --stdin </dev/null) &&
> +	>input.blob &&
> +	>marks.exp &&
> +	>tree.exp &&
> +
> +	cat >input.commit <<-EOF &&
> +	commit refs/heads/verify--dump-marks
> +	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
> +	data <<COMMIT
> +	test the sparse array dumping routines with exponentially growing marks

Ok, so we want marks to grow exponentially.

> +	COMMIT
> +	EOF
> +
> +	(
> +		i=0 &&
> +		l=4 &&
> +		m=6 &&
> +		n=7 &&

Maybe use slightly less cryptic variable names?

> +		while test "$i" -lt 27

Something like for i in `seq 1 10` will atleast make it clear that i
is just a loop variable. That's one arbitrary variable less.

> +		do
> +			cat >>input.blob <<-EOF &&
> +			blob
> +			mark :$l
> +			data 0
> +			blob
> +			mark :$m
> +			data 0
> +			blob
> +			mark :$n
> +			data 0
> +			EOF
> +			{
> +				echo "M 100644 :$l l$i" &&
> +				echo "M 100644 :$m m$i" &&
> +				echo "M 100644 :$n n$i"
> +			} >>input.commit &&
> +			{
> +				echo ":$l $mt" &&
> +				echo ":$m $mt" &&
> +				echo ":$n $mt"
> +			} >>marks.exp &&
> +			{
> +				printf "100644 blob $mt\tl$i\n" &&
> +				printf "100644 blob $mt\tm$i\n" &&
> +				printf "100644 blob $mt\tn$i\n"
> +			} >>tree.exp &&
> +			l=$(($l + $l)) &&
> +			m=$(($m + $m)) &&
> +			n=$(($l + $n)) &&

Maybe l=$(($l * 2)) and similarly for m to emphasize that they're
doubling in every loop iteration?

-- Ram

Re: [PATCH 03/24] t9300 (fast-import): guard "export large marks" test setup

[Ramkumar Ramachandra]

Hi again,

Jonathan Nieder writes:
>  test_expect_success 'A: export marks with large values' '
> +	test_tick &&
> +	mt=$(git hash-object --stdin </dev/null) &&
> +	>input.blob &&
> +	>marks.exp &&
> +	>tree.exp &&
> +
> +	cat >input.commit <<-EOF &&
> +	commit refs/heads/verify--dump-marks

Nit: Maybe change this to verify--export-marks corresponding to the
`--export-marks` feature of fast-import? The ref for testing the
`--import-marks` feature is already called `verify--import-marks`.

-- Ram

Re: [PATCH 03/24] t9300 (fast-import): guard "export large marks" test setup

[Raja R Harinath]

Hi,

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> Jonathan Nieder writes:
[snip]
>> +	cat >input.commit <<-EOF &&
>> +	commit refs/heads/verify--dump-marks
>> +	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
>> +	data <<COMMIT
>> +	test the sparse array dumping routines with exponentially growing marks
>
> Ok, so we want marks to grow exponentially.

Just to clarify, this code is already in the tree.  Jonathan is only
re-indenting it and moving it inside a test_expect_success.

[snip]
>> +		i=0 &&
>> +		l=4 &&
>> +		m=6 &&
>> +		n=7 &&
>
> Maybe use slightly less cryptic variable names?

I expect these are fairly idiomatic, except maybe for 'n', which could
be replaced by 'u', I guess.

[snip]
>> +			l=$(($l + $l)) &&
>> +			m=$(($m + $m)) &&
>> +			n=$(($l + $n)) &&
>
> Maybe l=$(($l * 2)) and similarly for m to emphasize that they're
> doubling in every loop iteration?

Hmm, $l appears twice, that's literally doubling, isn't it ;-)

- Hari

Re: [PATCH 03/24] t9300 (fast-import): guard "export large marks" test setup

[Raja R Harinath]

Hi,

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> Jonathan Nieder writes:
>>  test_expect_success 'A: export marks with large values' '
>> +	test_tick &&
>> +	mt=$(git hash-object --stdin </dev/null) &&
>> +	>input.blob &&
>> +	>marks.exp &&
>> +	>tree.exp &&
>> +
>> +	cat >input.commit <<-EOF &&
>> +	commit refs/heads/verify--dump-marks
>
> Nit: Maybe change this to verify--export-marks corresponding to the
> `--export-marks` feature of fast-import? The ref for testing the
> `--import-marks` feature is already called `verify--import-marks`.

Fair enough.  I just went with the name of the routine that had the bug:
dump_marks_helper.

- Hari

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Jonathan Nieder]

Hi Sam,

Sam Vilain wrote:
> On Sat, 2010-09-04 at 22:15 -0500, Jonathan Nieder wrote:

>> It works like this:
>> 
>> frontend:
>> 	feature report-fd=3
>> 	commit refs/heads/master
>> 	... revision 1 ...
>> 
>> importer:
>> 	abc7856cba76bca87a65bca76bca8bca98bca78bca76
>
> This is probably quite a late comment, but I don't think that
> 'report-fd=3' is a good idea in a protocol like this.

Yes, I guess "feature report-fd=3" is a protocol layering violation.
Unfortunately import-marks and export-marks have already set a
precedent.

How about a "feature report-fd" (with no argument) that checks if the
report-fd was set and errors out if not?

Well-behaved streams could use that and rely on the fd to be set on
the command line, while poorly-behaved streams could still use
"feature report-fd=whatever" to get the effect of --report-fd=whatever
and avoid breaking UI consistency.

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Sverre Rabbelier]

Heya,

On Fri, Sep 24, 2010 at 21:43, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Yes, I guess "feature report-fd=3" is a protocol layering violation.
> Unfortunately import-marks and export-marks have already set a
> precedent.

I don't agree, they (can) use a relative path name, allow the stream
to be re-used at a different location.

> Well-behaved streams could use that and rely on the fd to be set on
> the command line, while poorly-behaved streams could still use
> "feature report-fd=whatever" to get the effect of --report-fd=whatever
> and avoid breaking UI consistency.

Additionally you could have the commandline override whatever the
stream sets, so that the stream can be re-used as long as the user
specifies the appropriate commandline arguments?

-- 
Cheers,

Sverre Rabbelier

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Jonathan Nieder]

Sverre Rabbelier wrote:
> On Fri, Sep 24, 2010 at 21:43, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> Yes, I guess "feature report-fd=3" is a protocol layering violation.
>> Unfortunately import-marks and export-marks have already set a
>> precedent.
>
> I don't agree, they (can) use a relative path name, allow the stream
> to be re-used at a different location.

Sorry, I had confused myself.  Especially with relative-marks, using
relative paths without worrying about where they point makes a lot
of sense.  I was thinking of a frontend that reads or writes the marks
file itself, but import/export-marks features can also be used to just
save/restore marks opaquely.

(So iiuc one could imagine a backend that writes the marks within a
table somewhere instead of a text file when relative-marks is
requested, and most frontends could cope with that.)

Thus I have no precedent to fall back on. :)

>> Well-behaved streams could use that and rely on the fd to be set on
>> the command line, while poorly-behaved streams could still use
>> "feature report-fd=whatever" to get the effect of --report-fd=whatever
>> and avoid breaking UI consistency.
>
> Additionally you could have the commandline override whatever the
> stream sets, so that the stream can be re-used as long as the user
> specifies the appropriate commandline arguments?

Yep, that would work.  Still I don't think it makes a lot of sense to
allow "feature report-fd=4" in the fast-import stream.  If I can
ensure that fast-import has file descriptor 4 mapped to the right
place, then I am in control of the process that starts fast-import, so
a command-line option would be easy enough to use, no?

Re: [RFC/PATCH 0/3] fast-import: give importers access to the object store

[Sverre Rabbelier]

Heya,

On Sat, Sep 25, 2010 at 02:01, Jonathan Nieder <jrnieder@gmail.com> wrote:
> I was thinking of a frontend that reads or writes the marks
> file itself, but import/export-marks features can also be used to just
> save/restore marks opaquely.

That's exactly what git-remote-hg does :). The hg exporter and
importer doesn't care about where git stores it's marks. As long as
they both use the same marks it all works.

> Thus I have no precedent to fall back on. :)

Seems like it.

> Yep, that would work.  Still I don't think it makes a lot of sense to
> allow "feature report-fd=4" in the fast-import stream.  If I can
> ensure that fast-import has file descriptor 4 mapped to the right
> place, then I am in control of the process that starts fast-import, so
> a command-line option would be easy enough to use, no?

True. Perhaps some more input from others would help.

-- 
Cheers,

Sverre Rabbelier

svn-fe status

[Jonathan Nieder]

Jonathan Nieder wrote:

> Tomorrow I would like to re-roll the fast-import experiment
> so the svn-fe that understands deltas can get more attention

but I did not get as far as I would like.  Interested parties can find
some very rough patches in the git repository at:
 git://repo.or.cz/git/jrn.git tp/svn-fe-wip

The top two patches are completely bogus.  No doubt some of the
earlier ones are, too, which is why I do not think they are ready for
review yet, except by extra curious people.

Now the features that would be most useful from fast-import become a
bit clearer.  Since a single svn revision can make multiple changes to
a file, the ability to read back the currently staged content for the
current revision would be helpful, like this:

  commit refs/remotes/origin/root
  mark :100
  committer localuser <localuser@machine.example.com>
  data <<END_MESSAGE
  
  from refs/remotes/origin/root^0
  M 100644 :72:"trunk/README" "branches/topic/README"
  cat "branches/topic/README"
  M 100755 inline branches/topic/README
  data <<END_FILE
  ... output from delta application goes here ...
  END_FILE
  ... etc ...

In other words, it would be nice to be able to

 1. use ':<mark>:<path>' references to retrieve data from a
    previous revision
 2. use 'cat <path>' references to retrieve data from the
    current revision.

By using these two features, I think we could eliminate the repo_tree
module completely.

David's and Ram's discussions and patches from the last two weeks (or
at least the part I have understood) have been very helpful.

David Barr (3):
      svndiff: Give caller responsibility for initializing line_buffer
      vcs-svn: Reduce memory allocation churn in move_window
      vcs-svn: extend svndump to parse version 3 format

Jonathan Nieder (14):
      Makefile: declare vcs-svn test dependencies
      line_buffer: Allow character-oriented input
      line_buffer: Let caller peek ahead to find stream end
      line_buffer: Add binary-safe read() function
      vcs-svn: Add svn delta parser
      svndiff: Handle truncated data in deltas
      svndiff: Handle truncated source file
      svndiff: Cap length of delta read
      Teach fast-import to print the id of each imported commit
      fast-import: Let importers retrieve the objects being written
      fast-import: Allow cat command with empty path
      fast-import: Allow cat requests at arbitrary points in stream
      [WIP] svn-fe: Use the --report-fd feature
      wip

Ramkumar Ramachandra (1):
      Add a sample user for the svndiff library

 Documentation/git-fast-import.txt |   49 ++++
 Makefile                          |   10 +-
 contrib/svn-fe/.gitignore         |    1 +
 contrib/svn-fe/Makefile           |   30 ++-
 contrib/svn-fe/svn-da.c           |   26 ++
 contrib/svn-fe/svn-da.txt         |   24 ++
 contrib/svn-fe/svn-fe.txt         |    6 +-
 fast-import.c                     |  147 ++++++++++-
 t/t9010-svn-fe.sh                 |   37 +++-
 t/t9010/newdata.diff0             |  Bin 0 -> 19392 bytes
 t/t9010/newdata.done              |  522 +++++++++++++++++++++++++++++++++++++
 t/t9010/src.diff0                 |  Bin 0 -> 74 bytes
 t/t9010/src.done                  |  522 +++++++++++++++++++++++++++++++++++++
 t/t9300-fast-import.sh            |  210 +++++++++++++++
 test-svn-fe.c                     |   38 +++-
 vcs-svn/LICENSE                   |    2 +
 vcs-svn/fast_export.c             |  158 +++++++++++-
 vcs-svn/fast_export.h             |   11 +-
 vcs-svn/line_buffer.c             |   24 ++
 vcs-svn/line_buffer.h             |    3 +
 vcs-svn/repo_tree.c               |    9 +-
 vcs-svn/svndiff.c                 |  477 +++++++++++++++++++++++++++++++++
 vcs-svn/svndiff.h                 |   11 +
 vcs-svn/svndump.c                 |   69 +++++-
 24 files changed, 2340 insertions(+), 46 deletions(-)
 create mode 100644 contrib/svn-fe/svn-da.c
 create mode 100644 contrib/svn-fe/svn-da.txt
 create mode 100644 t/t9010/blank.done
 create mode 100644 t/t9010/newdata.diff0
 create mode 100644 t/t9010/newdata.done
 create mode 100644 t/t9010/src.diff0
 create mode 100644 t/t9010/src.done
 create mode 100644 vcs-svn/svndiff.c
 create mode 100644 vcs-svn/svndiff.h

Re: svn-fe status

[Sverre Rabbelier]

Heya,

On Sat, Sep 25, 2010 at 07:19, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Now the features that would be most useful from fast-import become a
> bit clearer.  Since a single svn revision can make multiple changes to
> a file, the ability to read back the currently staged content for the
> current revision would be helpful, like this:

Yes, that makes sense.

>  from refs/remotes/origin/root^0
>  M 100644 :72:"trunk/README" "branches/topic/README"

Shouldn't that be 'C' for copy? Of course, this should also be guarded
by a feature.

>  1. use ':<mark>:<path>' references to retrieve data from a
>    previous revision

I like this a lot better than the previous "catting the contents of a
commit to find the tree to find the file hash to cat the file
contents". I don't think there needs to be a colon after the mark
though, I'd prefer reserving the colon character for marks and instead
introduce a copy operation.

>  2. use 'cat <path>' references to retrieve data from the
>    current revision.

You'll thank yourself later if you add an (optional?) mark to cat, so
that you can cat previous versions of a file too? Doesn't svn ever
give you a diff against -. ... never mind! You don't need that, since
you can just copy the contents you want over first! Ok, great, that
works :).

> David's and Ram's discussions and patches from the last two weeks (or
> at least the part I have understood) have been very helpful.

Keep up the great work!

-- 
Cheers,

Sverre Rabbelier

Re: svn-fe status

[Jonathan Nieder]

Sverre Rabbelier wrote:

> Shouldn't that be 'C' for copy?

Yep, that is better.

 feature copyfromrev
 ...
 C :72 "trunk/README" "branches/topic/README"

We can peek ahead to make sure

 C :72 branches/topic/README

still copies a file named ":72".

> You'll thank yourself later if you add an (optional?) mark to cat, so
> that you can cat previous versions of a file too? Doesn't svn ever
> give you a diff against -. ... never mind!

For completeness it probably does make sense to allow

 cat <dataref> <path>

too, with <dataref> pointing to a tree (as before) or to a tag or
commit.  I just suspect svn-fe would not use it.

Re: svn-fe status

[Sverre Rabbelier]

Heya,

On Mon, Sep 27, 2010 at 04:54, Jonathan Nieder <jrnieder@gmail.com> wrote:
> We can peek ahead to make sure
>
>  C :72 branches/topic/README
>
> still copies a file named ":72".

Yes, I guess so, we have the entire line available when parsing, so
that shouldn't be a problem.

> For completeness it probably does make sense to allow
>
>  cat <dataref> <path>
>
> too, with <dataref> pointing to a tree (as before) or to a tag or
> commit.  I just suspect svn-fe would not use it.

I'd prefer to have one way to do it, then again, I don't use perl ;).

-- 
Cheers,

Sverre Rabbelier