git svn dcommit error: Cannot accept non-LF line endings in 'svn:log' property

git svn dcommit error: Cannot accept non-LF line endings in 'svn:log' property

[Bennett, Brian]

Environment:

Desktop: Windows 7 Enterprise 64-bit
svn client (if applicable): 1.8.8 from Apache
git (https://git-for-windows.github.io/): git version 2.10.1.windows.1
GitTfs (https://github.com/git-tfs/git-tfs): git-tfs version 0.27.0.0 (TFS client library 14.0.0.0 (MS)) (32-bit)
Team Foundation Server: 2010
Visual Studio installation: 2010 and 2015

All processing is being done on my desktop described above. My goal is to migrate Team Foundation Server source into git and then from git into a local SVN repository. The specific source from Team Foundation Server has only 2 changesets made against it and neither have a commit message associated with them.

Steps I'm taking:

1. Open a Windows (cmd.exe) command shell and put git-tfs into Path:
Set Path=C:\Users\brbennett\Downloads\GitTfs-0.27.0;%Path%

2. Create empty folder C:\TEMP\gitclone\Project_Elevation_Request and make this the working folder.

3. git-tfs clone -d "http://tfs:8080/tfs/collection" $/Folder1/Production/Elevation_Request
I can see the only 2 TFS changesets being cloned into the local git repository.
C423 = 6dfefb6160b53da7f580f24f2ce41af04f508b8a
C424 = e8d6573b5a6e29db78fd420f843ca7ad7480eda2

4. Move working folder to C:\TEMP\gitclone\Project_Elevation_Request\Elevation_Request

5. Create empty folder C:\TEMP\SVN\repos and create empty SVN repository:
svnadmin --compatible-version 1.8 create C:\TEMP\SVN\repos\Elevation_Request

6. Start up local SVN server in a different shell:
svnserve -d -r C:\TEMP\SVN\repos

7. Create trunk in local 
svn mkdir --parents svn:///C:/TEMP/SVN/repos/Elevation_Request/trunk -m "Importing git repo"
Committed revision 1.

8. git svn init svn:///C:/TEMP/SVN/repos/%PROJNAME% -s

9. git svn fetch
R1 for the trunk created earlier is retrieved
r1 = 5efc0da5f5af4cd62fde660a4402e3a751c2b003 (refs/remotes/origin/trunk)

10. git rebase origin/trunk
First, rewinding head to replay your work on top of it...
Applying:
Applying:

11. git svn dcommit
Committing to svn:///C:/TEMP/SVN/repos/Source_elevation_tool/trunk ...
        A       Source_elevation_tool.sln
        A       Source_elevation_tool/Form1.Designer.cs
        A       Source_elevation_tool/Form1.cs
        A       Source_elevation_tool/Form1.resx

ERROR from SVN:
Wrong or unexpected property value: Cannot accept non-LF line endings in 'svn:log' property
W: 40ff09d157bcbbf1e6deefd2222e38ed499ec8ac and refs/remotes/origin/trunk differ, using rebase -v:
:100644 000000 8222efb4e5a055b3b0b41ab91972f07dd71e4b10 0000000000000000000000000000000000000000 D      Source_elevation_tool.sln
:100644 000000 794f014c920a6aee2f21ee348e30f38a458e9c7d 0000000000000000000000000000000000000000 D      Source_elevation_tool.vssscc
:040000 000000 0acaa94fa910fe974019ae4c2dcbf9a620437758 0000000000000000000000000000000000000000 D      Source_elevation_tool
Current branch master is up to date.
ERROR: Not all changes have been committed into SVN, however the committed
ones (if any) seem to be successfully integrated into the working tree.
Please see the above messages for details.

I've researched enough to believe that the commit message being used by git svn contains a carriage return character (x'0D') and that has not been allowed in Subversion since version 1.6 (I can replicate this specific error message using an SVN dump file that contains x'0D' characters in the log messages.). However, I cannot find where I have any control over the log message that git svn is trying to use nor can I observe it. Note that I've also used the '-v' switch with the 'git svn dcommit', but do not receive anything other than what I am showing above.

Brian Bennett | Supv System Admin & Support, TA TECH Change Mgmt/Production Support
o: 319-355-7602 | c: 319-533-1094
e: brian.bennett@transamerica.com | w: www.transamerica.com

Transamerica
6400 C St. SW, Cedar Rapids, IA 52404 MS-2410
Facebook | LinkedIn

[PATCH] git-svn: convert CRLF to LF in commit message to SVN

[Eric Wong]

"Bennett, Brian" <Brian.Bennett@Transamerica.com> wrote:
> Environment:
> 
> Desktop: Windows 7 Enterprise 64-bit
> svn client (if applicable): 1.8.8 from Apache
> git (https://git-for-windows.github.io/): git version 2.10.1.windows.1
> GitTfs (https://github.com/git-tfs/git-tfs): git-tfs version 0.27.0.0 (TFS client library 14.0.0.0 (MS)) (32-bit)
> Team Foundation Server: 2010
> Visual Studio installation: 2010 and 2015

<snip>

Thanks for the report and research!

> I've researched enough to believe that the commit message
> being used by git svn contains a carriage return character
> (x'0D') and that has not been allowed in Subversion since
> version 1.6 (I can replicate this specific error message using
> an SVN dump file that contains x'0D' characters in the log
> messages.). However, I cannot find where I have any control
> over the log message that git svn is trying to use nor can I
> observe it. Note that I've also used the '-v' switch with the
> 'git svn dcommit', but do not receive anything other than what
> I am showing above.

Maybe git-for-windows isn't filtering CRLF into LF as "git commit"
does on GNU/Linux when the original commit was made?

I had to use "git commit-tree" to reproduce the error in testing
(instead of "git commit)"

Anyways, the one-line fix below should be enough for you.
Care to give it a shot?  Thanks again.


Junio: please pull when Brian confirms, thanks.

The following changes since commit 95ec6b1b3393eb6e26da40c565520a8db9796e9f:

  RelNotes: the eighth batch (2017-12-06 09:29:50 -0800)

are available in the git repository at:

  git://bogomips.org/git-svn.git svn-crlf

for you to fetch changes up to 95450bbbaaacaf2d603a4fbded25d55243dfb291:

  git-svn: convert CRLF to LF in commit message to SVN (2017-12-14 00:09:38 +0000)

----------------------------------------------------------------
Eric Wong (1):
      git-svn: convert CRLF to LF in commit message to SVN

 git-svn.perl                    |  1 +
 t/t9169-git-svn-dcommit-crlf.sh | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)
 create mode 100755 t/t9169-git-svn-dcommit-crlf.sh

------8<--------
Subject: [PATCH] git-svn: convert CRLF to LF in commit message to SVN

Subversion since 1.6 does not accept CR characters in the commit
message, so filter it out on our end before 'git svn dcommit' sets
the svn:log property.

Reported-by: Brian Bennett <Brian.Bennett@Transamerica.com>
Signed-off-by: Eric Wong <e@80x24.org>
---
 git-svn.perl                    |  1 +
 t/t9169-git-svn-dcommit-crlf.sh | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)
 create mode 100755 t/t9169-git-svn-dcommit-crlf.sh

diff --git a/git-svn.perl b/git-svn.perl
index d2404184ba..aa242d4f4f 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1865,6 +1865,7 @@ sub get_commit_entry {
 			}
 		}
 		$msgbuf =~ s/\s+$//s;
+		$msgbuf =~ s/\r\n/\n/sg; # SVN 1.6+ disallows CRLF
 		if ($Git::SVN::_add_author_from && defined($author)
 		    && !$saw_from) {
 			$msgbuf .= "\n\nFrom: $author";
diff --git a/t/t9169-git-svn-dcommit-crlf.sh b/t/t9169-git-svn-dcommit-crlf.sh
new file mode 100755
index 0000000000..54b1f61a2a
--- /dev/null
+++ b/t/t9169-git-svn-dcommit-crlf.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+test_description='git svn dcommit CRLF'
+. ./lib-git-svn.sh
+
+test_expect_success 'setup commit repository' '
+	svn_cmd mkdir -m "$test_description" "$svnrepo/dir" &&
+	git svn clone "$svnrepo" work &&
+	(
+		cd work &&
+		echo foo >>foo &&
+		git update-index --add foo &&
+		printf "a\\r\\n\\r\\nb\\r\\nc\\r\\n" >cmt &&
+		p=$(git rev-parse HEAD) &&
+		t=$(git write-tree) &&
+		cmt=$(git commit-tree -p $p $t <cmt) &&
+		git update-ref refs/heads/master $cmt &&
+		git cat-file commit HEAD | tail -n4 >out &&
+		test_cmp cmt out &&
+		git svn dcommit &&
+		printf "a\\n\\nb\\nc\\n" >exp &&
+		git cat-file commit HEAD | sed -ne 6,9p >out &&
+		test_cmp exp out
+	)
+'
+
+test_done
-- 
EW

RE: [PATCH] git-svn: convert CRLF to LF in commit message to SVN

[Bennett, Brian]

Thank you for your fast response,

I haven't done a build of this type before (so I could test the patch first) so I'm trying to do that and get this far:
1. git clone https://github.com/msysgit/msysgit.git "c:\temp\git\guitbuild"
2. git clone https://github.com/git-for-windows/git.git "c:\temp\git\guitbuild\git"
3. c:\temp\git\guitbuild\msys.bat
-------------------------------------------------------
Building and Installing Git
-------------------------------------------------------
    CC archive-tar.o
In file included from run-command.h:5,
                 from archive-tar.c:9:
compat/win32/pthread.h:38: error: expected ')' before 'ConditionVariable'
compat/win32/pthread.h:40: error: expected ')' before 'ConditionVariable'
compat/win32/pthread.h:42: error: expected ')' before 'ConditionVariable'
compat/win32/pthread.h:44: error: expected ')' before 'ConditionVariable'
make: *** [archive-tar.o] Error 1

The applicable lines from compat/win32/pthread.h:
37: WINBASEAPI VOID WINAPI
38: InitializeConditionVariable(PCONDITION_VARIABLE ConditionVariable);
39: WINBASEAPI VOID WINAPI
40: WakeConditionVariable(PCONDITION_VARIABLE ConditionVariable);
41: WINBASEAPI VOID WINAPI
42: WakeAllConditionVariable(PCONDITION_VARIABLE ConditionVariable);
43: WINBASEAPI BOOL WINAPI
44: SleepConditionVariableCS(PCONDITION_VARIABLE ConditionVariable,
45:                         PCRITICAL_SECTION CriticalSection,
46:                         DWORD dwMilliseconds);

I don't want to drag out testing the patch, so if either of you are able to quickly guide me on what I am doing incorrectly I am willing to get the build done so I can test it. If not, could one of you build with the patch and somehow get that to me so I could test?

Brian Bennett | Supv System Admin & Support, TA TECH Change Mgmt/Production Support
o: 319-355-7602 | c: 319-533-1094
e: brian.bennett@transamerica.com | w: www.transamerica.com

Transamerica
6400 C St. SW, Cedar Rapids, IA 52404 MS-2410
Facebook | LinkedIn

-----Original Message-----
From: Eric Wong [mailto:e@80x24.org] 
Sent: Wednesday, December 13, 2017 6:21 PM
To: Bennett, Brian <Brian.Bennett@Transamerica.com>; Junio C Hamano <gitster@pobox.com>
Cc: git@vger.kernel.org
Subject: [PATCH] git-svn: convert CRLF to LF in commit message to SVN

"Bennett, Brian" <Brian.Bennett@Transamerica.com> wrote:
> Environment:
> 
> Desktop: Windows 7 Enterprise 64-bit
> svn client (if applicable): 1.8.8 from Apache git 
> (https://urldefense.proofpoint.com/v2/url?u=https-3A__git-2Dfor-2Dwind
> ows.github.io_&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=CorEYR_fG6hKwP1xRO7
> dkFFJM6UfxLGgypqJT0q3mO4&m=f1K2uzEyLbtIX-0te07VlclknjdUztTvbgDMA0thROs
> &s=3AqxH_SEQG48PhnwuCD8udYta0mqXfgKKlmAWMfSlfE&e=): git version 
> 2.10.1.windows.1 GitTfs 
> (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_git-2
> Dtfs_git-2Dtfs&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=CorEYR_fG6hKwP1xRO7
> dkFFJM6UfxLGgypqJT0q3mO4&m=f1K2uzEyLbtIX-0te07VlclknjdUztTvbgDMA0thROs
> &s=C2gZ6zgihigH5eMpa5UVgj1mglbQbGN1HG0blMqjmsY&e=): git-tfs version 
> 0.27.0.0 (TFS client library 14.0.0.0 (MS)) (32-bit) Team Foundation 
> Server: 2010 Visual Studio installation: 2010 and 2015

<snip>

Thanks for the report and research!

> I've researched enough to believe that the commit message being used 
> by git svn contains a carriage return character
> (x'0D') and that has not been allowed in Subversion since version 1.6 
> (I can replicate this specific error message using an SVN dump file 
> that contains x'0D' characters in the log messages.). However, I 
> cannot find where I have any control over the log message that git svn 
> is trying to use nor can I observe it. Note that I've also used the 
> '-v' switch with the 'git svn dcommit', but do not receive anything 
> other than what I am showing above.

Maybe git-for-windows isn't filtering CRLF into LF as "git commit"
does on GNU/Linux when the original commit was made?

I had to use "git commit-tree" to reproduce the error in testing (instead of "git commit)"

Anyways, the one-line fix below should be enough for you.
Care to give it a shot?  Thanks again.


Junio: please pull when Brian confirms, thanks.

The following changes since commit 95ec6b1b3393eb6e26da40c565520a8db9796e9f:

  RelNotes: the eighth batch (2017-12-06 09:29:50 -0800)

are available in the git repository at:

  git://bogomips.org/git-svn.git svn-crlf

for you to fetch changes up to 95450bbbaaacaf2d603a4fbded25d55243dfb291:

  git-svn: convert CRLF to LF in commit message to SVN (2017-12-14 00:09:38 +0000)

----------------------------------------------------------------
Eric Wong (1):
      git-svn: convert CRLF to LF in commit message to SVN

 git-svn.perl                    |  1 +
 t/t9169-git-svn-dcommit-crlf.sh | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)
 create mode 100755 t/t9169-git-svn-dcommit-crlf.sh

------8<--------
Subject: [PATCH] git-svn: convert CRLF to LF in commit message to SVN

Subversion since 1.6 does not accept CR characters in the commit message, so filter it out on our end before 'git svn dcommit' sets the svn:log property.

Reported-by: Brian Bennett <Brian.Bennett@Transamerica.com>
Signed-off-by: Eric Wong <e@80x24.org>
---
 git-svn.perl                    |  1 +
 t/t9169-git-svn-dcommit-crlf.sh | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)
 create mode 100755 t/t9169-git-svn-dcommit-crlf.sh

diff --git a/git-svn.perl b/git-svn.perl index d2404184ba..aa242d4f4f 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1865,6 +1865,7 @@ sub get_commit_entry {
 			}
 		}
 		$msgbuf =~ s/\s+$//s;
+		$msgbuf =~ s/\r\n/\n/sg; # SVN 1.6+ disallows CRLF
 		if ($Git::SVN::_add_author_from && defined($author)
 		    && !$saw_from) {
 			$msgbuf .= "\n\nFrom: $author";
diff --git a/t/t9169-git-svn-dcommit-crlf.sh b/t/t9169-git-svn-dcommit-crlf.sh new file mode 100755 index 0000000000..54b1f61a2a
--- /dev/null
+++ b/t/t9169-git-svn-dcommit-crlf.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+test_description='git svn dcommit CRLF'
+. ./lib-git-svn.sh
+
+test_expect_success 'setup commit repository' '
+	svn_cmd mkdir -m "$test_description" "$svnrepo/dir" &&
+	git svn clone "$svnrepo" work &&
+	(
+		cd work &&
+		echo foo >>foo &&
+		git update-index --add foo &&
+		printf "a\\r\\n\\r\\nb\\r\\nc\\r\\n" >cmt &&
+		p=$(git rev-parse HEAD) &&
+		t=$(git write-tree) &&
+		cmt=$(git commit-tree -p $p $t <cmt) &&
+		git update-ref refs/heads/master $cmt &&
+		git cat-file commit HEAD | tail -n4 >out &&
+		test_cmp cmt out &&
+		git svn dcommit &&
+		printf "a\\n\\nb\\nc\\n" >exp &&
+		git cat-file commit HEAD | sed -ne 6,9p >out &&
+		test_cmp exp out
+	)
+'
+
+test_done
--
EW

Re: [PATCH] git-svn: convert CRLF to LF in commit message to SVN

[Todd Zullinger]

Hi Brian,

Bennett, Brian wrote:
> Thank you for your fast response,
> 
> I haven't done a build of this type before (so I could
> test the patch first) so I'm trying to do that and get
> this far:
...
> I don't want to drag out testing the patch, so if either
> of you are able to quickly guide me on what I am doing
> incorrectly I am willing to get the build done so I can
> test it. If not, could one of you build with the patch and
> somehow get that to me so I could test?

I don't know about building git for windows, but since the
git-svn command is a perl script, it might be easier to just
patch that file.  I think you can find the path where
git-svn is installed using: git --exec-path

For this one-liner, I'd just manually apply it.

(If you want to use 'git apply' or the patch command, you'll
have to edit the patch to adjust the name of the file, as
it's git-svn.perl in the git tree.  The .perl suffix is
dropped in the installed version.)

Hopefully that makes it easier for you to test Eric's patch.

-- 
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I am in shape.  Round is a shape.

RE: [PATCH] git-svn: convert CRLF to LF in commit message to SVN

[Bennett, Brian]

Thank you all for your guidance,

I have completed my test this morning with the patch and the 'git svn dcommit' is now SUCCESSFUl!

Thank you again for all of your help and I'll await to see when the patch is posted.

Brian Bennett | Supv System Admin & Support, TA TECH Change Mgmt/Production Support
o: 319-355-7602 | c: 319-533-1094
e: brian.bennett@transamerica.com | w: www.transamerica.com

Transamerica
6400 C St. SW, Cedar Rapids, IA 52404 MS-2410
Facebook | LinkedIn


-----Original Message-----
From: Todd Zullinger [mailto:todd.zullinger@gmail.com] On Behalf Of Todd Zullinger
Sent: Thursday, December 14, 2017 5:50 PM
To: Bennett, Brian <Brian.Bennett@Transamerica.com>
Cc: Eric Wong <e@80x24.org>; Junio C Hamano <gitster@pobox.com>; git@vger.kernel.org
Subject: Re: [PATCH] git-svn: convert CRLF to LF in commit message to SVN

Hi Brian,

Bennett, Brian wrote:
> Thank you for your fast response,
> 
> I haven't done a build of this type before (so I could test the patch 
> first) so I'm trying to do that and get this far:
...
> I don't want to drag out testing the patch, so if either of you are 
> able to quickly guide me on what I am doing incorrectly I am willing 
> to get the build done so I can test it. If not, could one of you build 
> with the patch and somehow get that to me so I could test?

I don't know about building git for windows, but since the git-svn command is a perl script, it might be easier to just patch that file.  I think you can find the path where git-svn is installed using: git --exec-path

For this one-liner, I'd just manually apply it.

(If you want to use 'git apply' or the patch command, you'll have to edit the patch to adjust the name of the file, as it's git-svn.perl in the git tree.  The .perl suffix is dropped in the installed version.)

Hopefully that makes it easier for you to test Eric's patch.

--
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I am in shape.  Round is a shape.

Re: [PATCH] git-svn: convert CRLF to LF in commit message to SVN

[Junio C Hamano]

"Bennett, Brian" <Brian.Bennett@Transamerica.com> writes:

> Thank you all for your guidance,
>
> I have completed my test this morning with the patch and the 'git
> svn dcommit' is now SUCCESSFUl!

Thanks, all.