Re: using git-svn with --no-metadata

Re: using git-svn with --no-metadata

[Eric Wong]

Vadim Zeitlin <vz-git@zeitlins.org> wrote:
>  Hello,
> 
>  First of all, let me use this opportunity to thank you for writing
> git-svn, it's the best thing since, well, probably git itself! Second,
> I'm sorry to bother you directly but I simply couldn't find any mailing
> list for git-svn and I wasn't sure if this was appropriate for the git
> mailing list itself. If I was wrong about not posting this there, please
> feel free to redirect/reply to git list if you prefer to discuss it there
> (I'd appreciate if you could still cc me in this case though).

Hello Vadim, you're welcome!

git-svn has always used the git mailing list.  Feel free to Cc
maintainers of particular subsections even when posting to the mailing
list, too.  The mailing list gets a lot of traffic, so we always
Cc folks who are relevant to the discussion.

>  Now to my question: I thought it would be a good idea to import an svn
> repository with --no-metadata option as I hoped that this would avoid the
> rewriting of commits which happen when git topic branches are checked in
> into svn with "git svn dcommit". The documentation (for my 1.7.1 version)
> did say that:

dcommit would have to rewrite commits anyways, since we sync the
usernames and commit times from SVN.  You can't avoid rewriting commits
unless you use "git svn set-tree" (on the other hand, set-tree can be
unfriendly to other SVN users).

> 	If you lose your .git/svn/git-svn/.rev_db file, git svn will not be
> 	able to rebuild it and you won't be able to fetch again, either.

> but I understood it as meaning that I wouldn't be able to fetch again if
> the file was lost (the file name appears to be wrong BTW, since the switch
> to rev_map from rev_db some time ago) and thought that it was ok as long as
> I took care to backup the .git/svn contents regularly. Apparently I was
> wrong as "git svn fetch" doesn't work even immediately after the import and
> looking at the code I see that working_head_info() only ever uses
> git-svn-id: lines in the commit messages and never rev_{db,map} files.
> 
>  So -- finally -- the questions are:
> 
> 1. Is this indeed the case, i.e. is importing from svn really impossible
>    with --no-metadata even if no files are lost? If so, I think it would
>    be nice to make the property description in the man page more clear.

rev_map (and rev_db before it) are designed for mapping SVN revision
numbers into git SHA1 identifiers.  rev_db used a simple but
space-inefficient offset lookup based on the SVN revision number.
rev_map uses a binary search also based on the SVN revision number.

> 2. Is it possible to implement getting the working head info from the
>    cached revision mappings instead of extracting it from the comments?
>    If so, would it be worth looking into doing this or is it something
>    that is doomed to fail anyhow?

It's possible to do a reverse mapping inefficiently (O(n)) by scanning
the values of the SHA1 commits.  You can add a hash database on disk,
too.  I don't think either is worth the effort in maintenance, though.
Nowadays git-filter-branch is in mainline git, and I recommend
using that for projects ready to drop SVN/git-svn entirely for git.

I'll update the docs accordingly.

-- 
Eric Wong

Re: using git-svn with --no-metadata

[Eric Wong]

Eric Wong <normalperson@yhbt.net> wrote:
> Vadim Zeitlin <vz-git@zeitlins.org> wrote:
> > The documentation (for my 1.7.1 version) did say that:
> >
> > 	If you lose your .git/svn/git-svn/.rev_db file, git svn will not be
> > 	able to rebuild it and you won't be able to fetch again, either.
> >
> > but I understood it as meaning that I wouldn't be able to fetch again if
> > the file was lost (the file name appears to be wrong BTW, since the switch
> > to rev_map from rev_db some time ago)

> I'll update the docs accordingly.

Hopefully the following patch makes sense, let me know what you think,
thanks.

>From 4c169c696bee06472ca3511a0f58c1c1b52b9fe9 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Sat, 21 Aug 2010 18:52:14 +0000
Subject: [PATCH] Documentation/git-svn: discourage "noMetadata"

"noMetadata" is a limited and sometimes harmful option.
Recommend git-filter-branch instead (if at all) for one-shot
imports instead.

This also fixes an out-of-date reference to "rev_db" format,
noticed by Vadim Zeitlin.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 Documentation/git-svn.txt |   14 ++++++++++++--
 1 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt
index 4b84d08..8abbcdc 100644
--- a/Documentation/git-svn.txt
+++ b/Documentation/git-svn.txt
@@ -56,6 +56,8 @@ COMMANDS
 	as well, they take precedence.
 --no-metadata;;
 	Set the 'noMetadata' option in the [svn-remote] config.
+	This option is not recommended, please read the 'svn.noMetadata'
+	section of this manpage before using this option.
 --use-svm-props;;
 	Set the 'useSvmProps' option in the [svn-remote] config.
 --use-svnsync-props;;
@@ -597,13 +599,21 @@ svn.noMetadata::
 svn-remote.<name>.noMetadata::
 	This gets rid of the 'git-svn-id:' lines at the end of every commit.
 +
-If you lose your .git/svn/git-svn/.rev_db file, 'git svn' will not
+If you lose your .git/svn/**/.rev_map.* files, 'git svn' will not
 be able to rebuild it and you won't be able to fetch again,
-either.  This is fine for one-shot imports.
+either.  This is sometimes useful for one-shot imports.
 +
 The 'git svn log' command will not work on repositories using
 this, either.  Using this conflicts with the 'useSvmProps'
 option for (hopefully) obvious reasons.
++
+This option is NOT recommended as it makes it difficult to track down
+old references to SVN revision numbers in existing documentation, bug
+reports and archives.  If you plan to eventually migrate from SVN to git
+and are certain about dropping SVN history, consider
+linkgit:git-filter-branch[1] instead.  filter-branch also allows
+reformating of metadata for ease-of-reading and rewriting authorship
+info for non-"svn.authorsFile" users.
 
 svn.useSvmProps::
 svn-remote.<name>.useSvmProps::
-- 
Eric Wong

Re: using git-svn with --no-metadata

[Andreas Schwab]

Eric Wong <normalperson@yhbt.net> writes:

> "noMetadata" is a limited and sometimes harmful option.
> Recommend git-filter-branch instead (if at all) for one-shot
> imports instead.

s/instead//

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re[2]: using git-svn with --no-metadata

[Vadim Zeitlin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4532 bytes --]

On Sat, 21 Aug 2010 18:37:11 +0000 Eric Wong <normalperson@yhbt.net> wrote:

EW> >  Now to my question: I thought it would be a good idea to import an svn
EW> > repository with --no-metadata option as I hoped that this would avoid the
EW> > rewriting of commits which happen when git topic branches are checked in
EW> > into svn with "git svn dcommit". The documentation (for my 1.7.1 version)
EW> > did say that:
EW> 
EW> dcommit would have to rewrite commits anyways, since we sync the
EW> usernames and commit times from SVN.  You can't avoid rewriting commits
EW> unless you use "git svn set-tree" (on the other hand, set-tree can be
EW> unfriendly to other SVN users).

 Thank you for this information, I didn't think about commit times at all
but if a rewrite is unavoidable because of them anyhow I'll just have to
[continue to] live with it.

EW> Nowadays git-filter-branch is in mainline git, and I recommend
EW> using that for projects ready to drop SVN/git-svn entirely for git.

 Unfortunately life is not that perfect yet (but I'm working on it ;-) and
I need to continue to use svn while I prefer, of course, use git for my own
development.

 Just in case you're curious, let me describe the problem I have with
commit rewriting. First, my setup is such that I have a single git-svn
clone (under Linux) but as most of the projects I'm working on are
cross-platform I also have git clones of this repository under Windows, Mac
&c. And when I implement some feature under, say, Windows, on a branch
win-work I then push it to Linux machine (where I can test that it didn't
break anything under Unix) and then use "git svn dcommit" from there. So
far all is well but the problem is that when I update my Windows repository
master branch, it has different commits from the ones on win-work branch.
So I can't e.g. "git branch -d win-work" but need to use "branch -D" (if I
am perfectly sure I committed everything) or, usually, "git rebase master
win-work" and only then delete the branch after git tells me that there are
no differences left.

 All this is hardly catastrophic but if I could avoid it somehow it would
be even better. Now that I know that "svn clone --no-metadata" won't help
me with this I wonder if the others have any better ways of working in such
setup?

EW> >From 4c169c696bee06472ca3511a0f58c1c1b52b9fe9 Mon Sep 17 00:00:00 2001
EW> From: Eric Wong <normalperson@yhbt.net>
EW> Date: Sat, 21 Aug 2010 18:52:14 +0000
EW> Subject: [PATCH] Documentation/git-svn: discourage "noMetadata"
EW> 
EW> "noMetadata" is a limited and sometimes harmful option.
EW> Recommend git-filter-branch instead (if at all) for one-shot
EW> imports instead.
EW> 
EW> This also fixes an out-of-date reference to "rev_db" format,
EW> noticed by Vadim Zeitlin.
EW> 
EW> Signed-off-by: Eric Wong <normalperson@yhbt.net>
EW> ---
EW>  Documentation/git-svn.txt |   14 ++++++++++++--
EW>  1 files changed, 12 insertions(+), 2 deletions(-)
EW> 
EW> diff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt
EW> index 4b84d08..8abbcdc 100644
EW> --- a/Documentation/git-svn.txt
EW> +++ b/Documentation/git-svn.txt
EW> @@ -56,6 +56,8 @@ COMMANDS
EW>  	as well, they take precedence.
EW>  --no-metadata;;
EW>  	Set the 'noMetadata' option in the [svn-remote] config.
EW> +	This option is not recommended, please read the 'svn.noMetadata'
EW> +	section of this manpage before using this option.
EW>  --use-svm-props;;
EW>  	Set the 'useSvmProps' option in the [svn-remote] config.
EW>  --use-svnsync-props;;
EW> @@ -597,13 +599,21 @@ svn.noMetadata::
EW>  svn-remote.<name>.noMetadata::
EW>  	This gets rid of the 'git-svn-id:' lines at the end of every commit.
EW>  +
EW> -If you lose your .git/svn/git-svn/.rev_db file, 'git svn' will not
EW> +If you lose your .git/svn/**/.rev_map.* files, 'git svn' will not
EW>  be able to rebuild it and you won't be able to fetch again,

 s/it/them/

EW> -either.  This is fine for one-shot imports.
EW> +either.  This is sometimes useful for one-shot imports.

 Sorry if I sound like a broken record but this still seems to be unclear
to me for the same reasons as the original text, i.e. because there is
still the "if you lose ... then you won't be able to fetch again"
implication (at least to me). Wouldn't something like

	This option can only be used for one-shot imports as 'git svn'
	will not be able to fetch again without metadata. Additionally,
	if you lose your .git/svn/**/.rev_map.* files, 'git svn' will not
	be able to rebuild them.

be more clear?

 Thanks again,
VZ

[-- Attachment #2: Type: APPLICATION/PGP-SIGNATURE, Size: 196 bytes --]

Re: using git-svn with --no-metadata

[Eric Wong]

Vadim Zeitlin <vz-git@zeitlins.org> wrote:
>  Just in case you're curious, let me describe the problem I have with
> commit rewriting. First, my setup is such that I have a single git-svn
> clone (under Linux) but as most of the projects I'm working on are
> cross-platform I also have git clones of this repository under Windows, Mac
> &c. And when I implement some feature under, say, Windows, on a branch
> win-work I then push it to Linux machine (where I can test that it didn't
> break anything under Unix) and then use "git svn dcommit" from there. So
> far all is well but the problem is that when I update my Windows repository
> master branch, it has different commits from the ones on win-work branch.
> So I can't e.g. "git branch -d win-work" but need to use "branch -D" (if I
> am perfectly sure I committed everything) or, usually, "git rebase master
> win-work" and only then delete the branch after git tells me that there are
> no differences left.
> 
>  All this is hardly catastrophic but if I could avoid it somehow it would
> be even better. Now that I know that "svn clone --no-metadata" won't help
> me with this I wonder if the others have any better ways of working in such
> setup?

I've favored a rebase-heavy workflow for many years now.  When working
with SVN, I found "git svn rebase" much more convenient to use than
normal "git rebase".  I'm not at all hesitant with "branch -D" since I
know reflogs/fsck will protect me from mistakes.

On pure git workflows, I still use rebase often and mail patches to
myself to transfer works-in-progress between machines.  If the machines
don't have a properly configured MTA, then I use format-patch and scp
patches/mboxes around.

>  Sorry if I sound like a broken record but this still seems to be unclear
> to me for the same reasons as the original text, i.e. because there is
> still the "if you lose ... then you won't be able to fetch again"
> implication (at least to me). Wouldn't something like
> 
> 	This option can only be used for one-shot imports as 'git svn'
> 	will not be able to fetch again without metadata. Additionally,
> 	if you lose your .git/svn/**/.rev_map.* files, 'git svn' will not
> 	be able to rebuild them.
> 
> be more clear?

Yes, much better.  I'll use your words instead :>

>From b7ddc7100f897d5b6a661a4aa57948608e4c3585 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Sat, 21 Aug 2010 18:52:14 +0000
Subject: [PATCH] Documentation/git-svn: discourage "noMetadata"

"noMetadata" is a sometimes harmful option, so better document
its behavior and limitations.

Suggested-by: Vadim Zeitlin
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 Documentation/git-svn.txt |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt
index 4b84d08..be8a51f 100644
--- a/Documentation/git-svn.txt
+++ b/Documentation/git-svn.txt
@@ -56,6 +56,8 @@ COMMANDS
 	as well, they take precedence.
 --no-metadata;;
 	Set the 'noMetadata' option in the [svn-remote] config.
+	This option is not recommended, please read the 'svn.noMetadata'
+	section of this manpage before using this option.
 --use-svm-props;;
 	Set the 'useSvmProps' option in the [svn-remote] config.
 --use-svnsync-props;;
@@ -597,13 +599,22 @@ svn.noMetadata::
 svn-remote.<name>.noMetadata::
 	This gets rid of the 'git-svn-id:' lines at the end of every commit.
 +
-If you lose your .git/svn/git-svn/.rev_db file, 'git svn' will not
-be able to rebuild it and you won't be able to fetch again,
-either.  This is fine for one-shot imports.
+This option can only be used for one-shot imports as 'git svn'
+will not be able to fetch again without metadata. Additionally,
+if you lose your .git/svn/**/.rev_map.* files, 'git svn' will not
+be able to rebuild them.
 +
 The 'git svn log' command will not work on repositories using
 this, either.  Using this conflicts with the 'useSvmProps'
 option for (hopefully) obvious reasons.
++
+This option is NOT recommended as it makes it difficult to track down
+old references to SVN revision numbers in existing documentation, bug
+reports and archives.  If you plan to eventually migrate from SVN to git
+and are certain about dropping SVN history, consider
+linkgit:git-filter-branch[1] instead.  filter-branch also allows
+reformating of metadata for ease-of-reading and rewriting authorship
+info for non-"svn.authorsFile" users.
 
 svn.useSvmProps::
 svn-remote.<name>.useSvmProps::
-- 
Eric Wong