How to perform efficient incremental update of a git repo from a large SVN repo? Bug or clueless user?

How to perform efficient incremental update of a git repo from a large SVN repo? Bug or clueless user?

[James Mason]

I have a large and active SVN repository.  More than 36,000 revisions,
thousands of branches - new ones created daily - and a non-standard
layout.  Also, a secondary git repository that serves as a faithful
"git" copy of the work accumulating in SVN (git version 2.9.5).  I seek
an efficient way to routinely (nightly?) update the git repo with new
changes accumulating in SVN.

Most of what I have found on the web suggests that, when properly
configured, incremental update will be correctly accomplished by "git
svn fetch".  My experience so far is that - while that will eventually
work - it is remarkably slow.  It's true that "git svn fetch" doesn't
redundantly add content - but it doesn't resume without appearing to
walk over all the old SVN revisions (watch the contents of
svn/.metadata).  For a large SVN repository - this can take days.

I initially posed this as a question on stack overflow (see "Routinely
updating a git repo from a large svn repo" ).  It appeared that I had
learned something about how to use "--revision" to speed things along -
so I recently added that as an answer to my own question (update now
taking routinely less than an hour).  Still that all seemed really odd.
"--revision" isn't documented as an option to "git svn fetch", you need
to dig START out of .git/svn/.metadata and END out of the SVN repository.

Today I searched on the string "branches-MaxRev" and found something
more explicitly at odds with my experience (in the git-svn document):

     Note that git-svn keeps track of the highest revision in which a
branch or tag has appeared. If the subset of
     branches or tags is changed after fetching, then
$GIT_DIR/svn/.metadata must be manually edited to remove
     (or reset) branches-maxRev and/or tags-maxRev as appropriate."

I read the documentation as saying that "git svn fetch" tracks where it
left off (which was my assumption before experiencing otherwise).   So
what I'm seeing seems like a bug - but I'm worried that my use of
"--revision" may not be entirely legitimate.

Any assistance, advice, etc., would be most appreciated.

-jrm

This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.

If you are not one of the named recipients or have received this email in error,

(i) you should not read, disclose, or copy it,

(ii) please notify sender of your receipt by reply email and delete this email and all attachments,

(iii) Dassault Systèmes does not accept or assume any liability or responsibility for any use of or reliance on this email.


Please be informed that your personal data are processed according to our data privacy policy as described on our website. Should you have any questions related to personal data protection, please contact 3DS Data Protection Officer at 3DS.compliance-privacy@3ds.com<mailto:3DS.compliance-privacy@3ds.com>


For other languages, go to https://www.3ds.com/terms/email-disclaimer

Re: How to perform efficient incremental update of a git repo from a large SVN repo? Bug or clueless user?

[Mateusz Loskot]

On Wed, 12 Dec 2018 at 21:13, James Mason <James.Mason@3ds.com> wrote:
>
> I have a large and active SVN repository.  More than 36,000 revisions,
> thousands of branches - new ones created daily - and a non-standard
> layout.  Also, a secondary git repository that serves as a faithful
> "git" copy of the work accumulating in SVN (git version 2.9.5).  I seek
> an efficient way to routinely (nightly?) update the git repo with new
> changes accumulating in SVN.

SubGit [1]

I've been using it to translate SVN repository with ~40K revisions,
producing ~25GB Git bare repo.
SubGit is x10-x100... faster than `git svn`.

[1] https://subgit.com/

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net