From: Jonathan Nieder <jrnieder@gmail.com>
To: git@vger.kernel.org
Cc: David Barr <david.barr@cordelta.com>,
Ramkumar Ramachandra <artagnon@gmail.com>,
Sverre Rabbelier <srabbelier@gmail.com>,
Sam Vilain <sam@vilain.net>, Stephen Bash <bash@genarts.com>,
Tomas Carnecky <tom@dbservice.com>
Subject: [PATCH 08/12] vcs-svn: set up channel to read fast-import cat-blob response
Date: Sun, 6 Mar 2011 17:11:30 -0600 [thread overview]
Message-ID: <20110306231130.GJ24327@elie> (raw)
In-Reply-To: <20110306225419.GA24327@elie>
From: David Barr <david.barr@cordelta.com>
Date: Sat, 5 Mar 2011 13:30:23 +1100
Set up some plumbing: teach the svndump lib to pass a file descriptor
number to the fast_export lib, representing where cat-blob/ls
responses can be read from, and add a get_response_line helper
function to the fast_export lib to read a line from that file.
Unfortunately this means that svn-fe needs file descriptor 3 to be
redirected from somewhere (preferrably the cat-blob stream of a
fast-import backend); otherwise it will fail:
$ svndump <path> | svn-fe
fatal: cannot read from file descriptor 3: Bad file descriptor
For the moment, "svn-fe 3</dev/null" works as a workaround but it
will not work for very long. A fast-import backend that can retrieve
old commits is needed in order to be able to fulfill svn
"Node-copyfrom-rev" requests that refer to revs from a previous run.
[jn: with new change description]
Based-on-patch-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
contrib/svn-fe/svn-fe.txt | 6 ++-
t/t9010-svn-fe.sh | 118 +++++++++++++++++++++++++-------------------
vcs-svn/fast_export.c | 28 +++++++++++
vcs-svn/fast_export.h | 4 ++
vcs-svn/svndump.c | 5 ++
5 files changed, 109 insertions(+), 52 deletions(-)
diff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt
index cd075b9..85f7b83 100644
--- a/contrib/svn-fe/svn-fe.txt
+++ b/contrib/svn-fe/svn-fe.txt
@@ -7,7 +7,11 @@ svn-fe - convert an SVN "dumpfile" to a fast-import stream
SYNOPSIS
--------
-svnadmin dump --incremental REPO | svn-fe [url] | git fast-import
+[verse]
+mkfifo backchannel &&
+svnadmin dump --incremental REPO |
+ svn-fe [url] 3<backchannel |
+ git fast-import --cat-blob-fd=3 3>backchannel
DESCRIPTION
-----------
diff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh
index 5a6a4b9..2ae5374 100755
--- a/t/t9010-svn-fe.sh
+++ b/t/t9010-svn-fe.sh
@@ -5,8 +5,26 @@ test_description='check svn dumpfile importer'
. ./test-lib.sh
reinit_git () {
+ if ! test_declared_prereq PIPE
+ then
+ echo >&4 "reinit_git: need to declare PIPE prerequisite"
+ return 127
+ fi
rm -fr .git &&
- git init
+ rm -f stream backflow &&
+ git init &&
+ mkfifo stream backflow
+}
+
+try_dump () {
+ input=$1 &&
+ maybe_fail=${2:+test_$2} &&
+
+ {
+ $maybe_fail test-svn-fe "$input" >stream 3<backflow &
+ } &&
+ git fast-import --cat-blob-fd=3 <stream 3>backflow &&
+ wait $!
}
properties () {
@@ -35,21 +53,27 @@ text_no_props () {
>empty
-test_expect_success 'empty dump' '
+test_expect_success 'setup: have pipes?' '
+ rm -f frob &&
+ if mkfifo frob
+ then
+ test_set_prereq PIPE
+ fi
+'
+
+test_expect_success PIPE 'empty dump' '
reinit_git &&
echo "SVN-fs-dump-format-version: 2" >input &&
- test-svn-fe input >stream &&
- git fast-import <stream
+ try_dump input
'
-test_expect_success 'v4 dumps not supported' '
+test_expect_success PIPE 'v4 dumps not supported' '
reinit_git &&
echo "SVN-fs-dump-format-version: 4" >v4.dump &&
- test_must_fail test-svn-fe v4.dump >stream &&
- test_cmp empty stream
+ try_dump v4.dump must_fail
'
-test_expect_failure 'empty revision' '
+test_expect_failure PIPE 'empty revision' '
reinit_git &&
printf "rev <nobody, nobody@local>: %s\n" "" "" >expect &&
cat >emptyrev.dump <<-\EOF &&
@@ -64,13 +88,12 @@ test_expect_failure 'empty revision' '
Content-length: 0
EOF
- test-svn-fe emptyrev.dump >stream &&
- git fast-import <stream &&
+ try_dump emptyrev.dump &&
git log -p --format="rev <%an, %ae>: %s" HEAD >actual &&
test_cmp expect actual
'
-test_expect_success 'empty properties' '
+test_expect_success PIPE 'empty properties' '
reinit_git &&
printf "rev <nobody, nobody@local>: %s\n" "" "" >expect &&
cat >emptyprop.dump <<-\EOF &&
@@ -88,13 +111,12 @@ test_expect_success 'empty properties' '
PROPS-END
EOF
- test-svn-fe emptyprop.dump >stream &&
- git fast-import <stream &&
+ try_dump emptyprop.dump &&
git log -p --format="rev <%an, %ae>: %s" HEAD >actual &&
test_cmp expect actual
'
-test_expect_success 'author name and commit message' '
+test_expect_success PIPE 'author name and commit message' '
reinit_git &&
echo "<author@example.com, author@example.com@local>" >expect.author &&
cat >message <<-\EOF &&
@@ -121,15 +143,14 @@ test_expect_success 'author name and commit message' '
echo &&
cat props
} >log.dump &&
- test-svn-fe log.dump >stream &&
- git fast-import <stream &&
+ try_dump log.dump &&
git log -p --format="%B" HEAD >actual.log &&
git log --format="<%an, %ae>" >actual.author &&
test_cmp message actual.log &&
test_cmp expect.author actual.author
'
-test_expect_success 'unsupported properties are ignored' '
+test_expect_success PIPE 'unsupported properties are ignored' '
reinit_git &&
echo author >expect &&
cat >extraprop.dump <<-\EOF &&
@@ -149,13 +170,12 @@ test_expect_success 'unsupported properties are ignored' '
author
PROPS-END
EOF
- test-svn-fe extraprop.dump >stream &&
- git fast-import <stream &&
+ try_dump extraprop.dump &&
git log -p --format=%an HEAD >actual &&
test_cmp expect actual
'
-test_expect_failure 'timestamp and empty file' '
+test_expect_failure PIPE 'timestamp and empty file' '
echo author@example.com >expect.author &&
echo 1999-01-01 >expect.date &&
echo file >expect.files &&
@@ -186,8 +206,7 @@ test_expect_failure 'timestamp and empty file' '
EOF
} >emptyfile.dump &&
- test-svn-fe emptyfile.dump >stream &&
- git fast-import <stream &&
+ try_dump emptyfile.dump &&
git log --format=%an HEAD >actual.author &&
git log --date=short --format=%ad HEAD >actual.date &&
git ls-tree -r --name-only HEAD >actual.files &&
@@ -198,7 +217,7 @@ test_expect_failure 'timestamp and empty file' '
test_cmp empty file
'
-test_expect_success 'directory with files' '
+test_expect_success PIPE 'directory with files' '
reinit_git &&
printf "%s\n" directory/file1 directory/file2 >expect.files &&
echo hi >hi &&
@@ -242,8 +261,7 @@ test_expect_success 'directory with files' '
EOF
text_no_props hi
} >directory.dump &&
- test-svn-fe directory.dump >stream &&
- git fast-import <stream &&
+ try_dump directory.dump &&
git ls-tree -r --name-only HEAD >actual.files &&
git checkout HEAD directory &&
@@ -252,7 +270,8 @@ test_expect_success 'directory with files' '
test_cmp hi directory/file2
'
-test_expect_success 'node without action' '
+test_expect_success PIPE 'node without action' '
+ reinit_git &&
cat >inaction.dump <<-\EOF &&
SVN-fs-dump-format-version: 3
@@ -269,10 +288,11 @@ test_expect_success 'node without action' '
PROPS-END
EOF
- test_must_fail test-svn-fe inaction.dump
+ try_dump inaction.dump must_fail
'
-test_expect_success 'action: add node without text' '
+test_expect_success PIPE 'action: add node without text' '
+ reinit_git &&
cat >textless.dump <<-\EOF &&
SVN-fs-dump-format-version: 3
@@ -290,10 +310,10 @@ test_expect_success 'action: add node without text' '
PROPS-END
EOF
- test_must_fail test-svn-fe textless.dump
+ try_dump textless.dump must_fail
'
-test_expect_failure 'change file mode but keep old content' '
+test_expect_failure PIPE 'change file mode but keep old content' '
reinit_git &&
cat >expect <<-\EOF &&
OBJID
@@ -356,8 +376,7 @@ test_expect_failure 'change file mode but keep old content' '
PROPS-END
EOF
- test-svn-fe filemode.dump >stream &&
- git fast-import <stream &&
+ try_dump filemode.dump &&
{
git rev-list HEAD |
git diff-tree --root --stdin |
@@ -370,7 +389,7 @@ test_expect_failure 'change file mode but keep old content' '
test_cmp hello actual.target
'
-test_expect_success 'change file mode and reiterate content' '
+test_expect_success PIPE 'change file mode and reiterate content' '
reinit_git &&
cat >expect <<-\EOF &&
OBJID
@@ -382,7 +401,7 @@ test_expect_success 'change file mode and reiterate content' '
EOF
echo "link hello" >expect.blob &&
echo hello >hello &&
- cat >filemode.dump <<-\EOF &&
+ cat >filemode2.dump <<-\EOF &&
SVN-fs-dump-format-version: 3
Revision-number: 1
@@ -437,8 +456,7 @@ test_expect_success 'change file mode and reiterate content' '
PROPS-END
link hello
EOF
- test-svn-fe filemode.dump >stream &&
- git fast-import <stream &&
+ try_dump filemode2.dump &&
{
git rev-list HEAD |
git diff-tree --root --stdin |
@@ -451,7 +469,8 @@ test_expect_success 'change file mode and reiterate content' '
test_cmp hello actual.target
'
-test_expect_success 'deltas not supported' '
+test_expect_success PIPE 'deltas not supported' '
+ reinit_git &&
{
# (old) h + (inline) ello + (old) \n
printf "SVNQ%b%b%s" "Q\003\006\005\004" "\001Q\0204\001\002" "ello" |
@@ -511,10 +530,10 @@ test_expect_success 'deltas not supported' '
echo PROPS-END &&
cat delta
} >delta.dump &&
- test_must_fail test-svn-fe delta.dump
+ test_must_fail try_dump delta.dump
'
-test_expect_success 'property deltas supported' '
+test_expect_success PIPE 'property deltas supported' '
reinit_git &&
cat >expect <<-\EOF &&
OBJID
@@ -570,8 +589,7 @@ test_expect_success 'property deltas supported' '
PROPS-END
EOF
} >propdelta.dump &&
- test-svn-fe propdelta.dump >stream &&
- git fast-import <stream &&
+ try_dump propdelta.dump &&
{
git rev-list HEAD |
git diff-tree --stdin |
@@ -580,7 +598,7 @@ test_expect_success 'property deltas supported' '
test_cmp expect actual
'
-test_expect_success 'properties on /' '
+test_expect_success PIPE 'properties on /' '
reinit_git &&
cat <<-\EOF >expect &&
OBJID
@@ -625,8 +643,7 @@ test_expect_success 'properties on /' '
PROPS-END
EOF
- test-svn-fe changeroot.dump >stream &&
- git fast-import <stream &&
+ try_dump changeroot.dump &&
{
git rev-list HEAD |
git diff-tree --root --always --stdin |
@@ -635,7 +652,7 @@ test_expect_success 'properties on /' '
test_cmp expect actual
'
-test_expect_success 'deltas for typechange' '
+test_expect_success PIPE 'deltas for typechange' '
reinit_git &&
cat >expect <<-\EOF &&
OBJID
@@ -711,8 +728,7 @@ test_expect_success 'deltas for typechange' '
PROPS-END
link testing 321
EOF
- test-svn-fe deleteprop.dump >stream &&
- git fast-import <stream &&
+ try_dump deleteprop.dump &&
{
git rev-list HEAD |
git diff-tree --root --stdin |
@@ -736,12 +752,12 @@ test_expect_success 'set up svn repo' '
fi
'
-test_expect_success SVNREPO 't9135/svn.dump' '
- git init simple-git &&
- test-svn-fe "$TEST_DIRECTORY/t9135/svn.dump" >simple.fe &&
+test_expect_success SVNREPO,PIPE 't9135/svn.dump' '
+ mkdir -p simple-git &&
(
cd simple-git &&
- git fast-import <../simple.fe
+ reinit_git &&
+ try_dump "$TEST_DIRECTORY/t9135/svn.dump"
) &&
(
cd simple-svnco &&
diff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c
index 5a105ad..8786ed2 100644
--- a/vcs-svn/fast_export.c
+++ b/vcs-svn/fast_export.c
@@ -12,6 +12,24 @@
#define MAX_GITSVN_LINE_LEN 4096
static uint32_t first_commit_done;
+static struct line_buffer report_buffer = LINE_BUFFER_INIT;
+
+void fast_export_init(int fd)
+{
+ if (buffer_fdinit(&report_buffer, fd))
+ die_errno("cannot read from file descriptor %d", fd);
+}
+
+void fast_export_deinit(void)
+{
+ if (buffer_deinit(&report_buffer))
+ die_errno("error closing fast-import feedback stream");
+}
+
+void fast_export_reset(void)
+{
+ buffer_reset(&report_buffer);
+}
void fast_export_delete(uint32_t depth, uint32_t *path)
{
@@ -69,6 +87,16 @@ void fast_export_commit(uint32_t revision, uint32_t author, char *log,
printf("progress Imported commit %"PRIu32".\n\n", revision);
}
+static const char *get_response_line(void)
+{
+ const char *line = buffer_read_line(&report_buffer);
+ if (line)
+ return line;
+ if (buffer_ferror(&report_buffer))
+ die_errno("error reading from fast-import");
+ die("unexpected end of fast-import feedback");
+}
+
void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len, struct line_buffer *input)
{
if (mode == REPO_MODE_LNK) {
diff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h
index aff8005..09b2033 100644
--- a/vcs-svn/fast_export.h
+++ b/vcs-svn/fast_export.h
@@ -3,6 +3,10 @@
#include "line_buffer.h"
+void fast_export_init(int fd);
+void fast_export_deinit(void);
+void fast_export_reset(void);
+
void fast_export_delete(uint32_t depth, uint32_t *path);
void fast_export_modify(uint32_t depth, uint32_t *path, uint32_t mode,
uint32_t mark);
diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
index a384996..3cc4135 100644
--- a/vcs-svn/svndump.c
+++ b/vcs-svn/svndump.c
@@ -14,6 +14,8 @@
#include "obj_pool.h"
#include "string_pool.h"
+#define REPORT_FILENO 3
+
#define NODEACT_REPLACE 4
#define NODEACT_DELETE 3
#define NODEACT_ADD 2
@@ -382,6 +384,7 @@ int svndump_init(const char *filename)
if (buffer_init(&input, filename))
return error("cannot open %s: %s", filename, strerror(errno));
repo_init();
+ fast_export_init(REPORT_FILENO);
reset_dump_ctx(~0);
reset_rev_ctx(0);
reset_node_ctx(NULL);
@@ -392,6 +395,7 @@ int svndump_init(const char *filename)
void svndump_deinit(void)
{
log_reset();
+ fast_export_deinit();
repo_reset();
reset_dump_ctx(~0);
reset_rev_ctx(0);
@@ -405,6 +409,7 @@ void svndump_deinit(void)
void svndump_reset(void)
{
log_reset();
+ fast_export_reset();
buffer_reset(&input);
repo_reset();
reset_dump_ctx(~0);
--
1.7.4.1
next prev parent reply other threads:[~2011-03-06 23:11 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-12-10 10:20 [RFC/PATCH 00/10] vcs-svn: prepare for (implement?) incremental import Jonathan Nieder
2010-12-10 10:21 ` [PATCH 01/10] vcs-svn: use higher mark numbers for blobs Jonathan Nieder
2010-12-10 10:22 ` [PATCH 02/10] vcs-svn: save marks for imported commits Jonathan Nieder
2011-03-06 11:15 ` Jonathan Nieder
2010-12-10 10:23 ` [PATCH 03/10] vcs-svn: introduce cat_mark function to retrieve a marked blob Jonathan Nieder
2010-12-10 10:23 ` [PATCH 04/10] vcs-svn: make apply_delta caller retrieve preimage Jonathan Nieder
2010-12-10 10:25 ` [PATCH 05/10] vcs-svn: split off function to export result from delta application Jonathan Nieder
2010-12-10 10:26 ` [PATCH 06/10] vcs-svn: do not rely on marks for old blobs Jonathan Nieder
2010-12-10 10:27 ` [PATCH 07/10] vcs-svn: split off function to make 'ls' requests Jonathan Nieder
2010-12-10 10:28 ` [PATCH 08/10] vcs-svn: prepare to eliminate repo_tree structure Jonathan Nieder
2011-03-06 12:52 ` [PATCH v2] " Jonathan Nieder
2011-03-06 20:41 ` David Barr
2010-12-10 10:30 ` [PATCH 09/10] vcs-svn: simplifications for repo_modify_path et al Jonathan Nieder
2010-12-10 10:33 ` [PATCH 10/10] vcs-svn: eliminate repo_tree structure Jonathan Nieder
[not found] ` <C59168D0-B409-4A83-B96C-8CCD42D0B62F@cordelta.com>
[not found] ` <20101211184654.GA17464@burratino>
2010-12-11 22:47 ` [RFC/PATCH] fast-import: treat filemodify with empty tree as delete Jonathan Nieder
2010-12-11 23:00 ` [PATCH db/vcs-svn-incremental] vcs-svn: avoid git-isms in fast-import stream Jonathan Nieder
2010-12-11 23:04 ` [PATCH 12/10] vcs-svn: quote paths correctly for ls command David Michael Barr
2010-12-11 23:11 ` [PATCH db/vcs-svn-incremental] vcs-svn: quote all paths passed to fast-import Jonathan Nieder
2010-12-12 9:32 ` [PATCH 13/10] vcs-svn: use mark from previous import for parent commit David Michael Barr
2010-12-12 17:06 ` Jonathan Nieder
2011-03-06 22:54 ` [PATCH v2 00/12] vcs-svn: incremental import Jonathan Nieder
2011-03-06 23:03 ` [PATCH 01/12] vcs-svn: use higher mark numbers for blobs Jonathan Nieder
2011-03-08 19:08 ` Junio C Hamano
2011-03-09 6:55 ` Jonathan Nieder
2011-03-06 23:04 ` [PATCH 02/12] vcs-svn: save marks for imported commits Jonathan Nieder
2011-03-06 23:07 ` [PATCH 03/12] vcs-svn: introduce repo_read_path to check the content at a path Jonathan Nieder
2011-03-06 23:08 ` [PATCH 04/12] vcs-svn: handle_node: use repo_read_path Jonathan Nieder
2011-03-06 23:09 ` [PATCH 05/12] vcs-svn: simplify repo_modify_path and repo_copy Jonathan Nieder
2011-03-06 23:09 ` [PATCH 06/12] vcs-svn: add a comment before each commit Jonathan Nieder
2011-03-06 23:10 ` [PATCH 07/12] vcs-svn: allow input errors to be detected promptly Jonathan Nieder
2011-03-06 23:11 ` Jonathan Nieder [this message]
2011-03-06 23:12 ` [PATCH 09/12] vcs-svn: eliminate repo_tree structure Jonathan Nieder
2011-03-06 23:12 ` [PATCH 10/12] vcs-svn: quote paths correctly for ls command Jonathan Nieder
2011-03-06 23:13 ` [PATCH 11/12] vcs-svn: handle filenames with dq correctly Jonathan Nieder
2011-03-06 23:16 ` [PATCH 12/12] vcs-svn: use mark from previous import for parent commit Jonathan Nieder
2011-03-07 12:24 ` [PATCH v2 00/12] vcs-svn: incremental import Sverre Rabbelier
2011-03-07 21:23 ` Jonathan Nieder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110306231130.GJ24327@elie \
--to=jrnieder@gmail.com \
--cc=artagnon@gmail.com \
--cc=bash@genarts.com \
--cc=david.barr@cordelta.com \
--cc=git@vger.kernel.org \
--cc=sam@vilain.net \
--cc=srabbelier@gmail.com \
--cc=tom@dbservice.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).