git-svn and a nested branches folder

git-svn and a nested branches folder

[Russ Brown]

Hi,

I'm having some trouble with using git-svn to fetch a repository, and I
think it's because the repository doesn't store branches as a flat list
directly under the 'branches' directory.

Basically, we have a structure like this:

|
+-trunk
+-tags
+-branches
  + category-a
    + branch-a
    + branch-b
  + category-b
    + branch-c
    + branch-d

etc. category-a and category-b are simple directories created using svn
mkdir. The branches are created using svn cp.

It helps us to organise the branches better, but the rationale is
besides the point. The problem is that git-svn seems to want to treat
category-a and category-b as branches, which isn't right at all. As a
result, git-svn seems to skip most (if not all) revisions that occur in
these directories and creates a lot of entries in unhandled.log.

I've also encountered an index corruption in one of the
.git/svn/<branch>/index files which I think it probably related.

I've had a quick look at the source myself, but perl isn't my strong
point. What I think it should do is something like recurse down the tree
from the directory given looking for folders that copy from trunk (or
some other branch that already exists). That would work perfectly well
for the default flat branch storage method as well as the one we use.

The only other problem is in branch naming, which could clash if you
only use the outer-most directory name, so I'd suggest something that
involves concatenating the folders in the path relative to 'branches' to
keep them unique (if git can handle slashes in branch names then all the
better).

Thanks for reading.

-- 

Russ

Re: git-svn and a nested branches folder

[David Kastrup]

Russ Brown <pickscrape@gmail.com> writes:

> I'm having some trouble with using git-svn to fetch a repository, and I
> think it's because the repository doesn't store branches as a flat list
> directly under the 'branches' directory.
>
> Basically, we have a structure like this:
>
> |
> +-trunk
> +-tags
> +-branches
>   + category-a
>     + branch-a
>     + branch-b
>   + category-b
>     + branch-c
>     + branch-d
>
> etc. category-a and category-b are simple directories created using svn
> mkdir. The branches are created using svn cp.
>
> It helps us to organise the branches better, but the rationale is
> besides the point. The problem is that git-svn seems to want to
> treat category-a and category-b as branches, which isn't right at
> all. As a result, git-svn seems to skip most (if not all) revisions
> that occur in these directories and creates a lot of entries in
> unhandled.log.

So what did you specify in your .git/config file regarding the svn
structure?

-- 
David Kastrup

Re: git-svn and a nested branches folder

[Russ Brown]

David Kastrup wrote:
> Russ Brown <pickscrape@gmail.com> writes:
> 
>> I'm having some trouble with using git-svn to fetch a repository, and I
>> think it's because the repository doesn't store branches as a flat list
>> directly under the 'branches' directory.
>>
>> Basically, we have a structure like this:
>>
>> |
>> +-trunk
>> +-tags
>> +-branches
>>   + category-a
>>     + branch-a
>>     + branch-b
>>   + category-b
>>     + branch-c
>>     + branch-d
>>
>> etc. category-a and category-b are simple directories created using svn
>> mkdir. The branches are created using svn cp.
>>
>> It helps us to organise the branches better, but the rationale is
>> besides the point. The problem is that git-svn seems to want to
>> treat category-a and category-b as branches, which isn't right at
>> all. As a result, git-svn seems to skip most (if not all) revisions
>> that occur in these directories and creates a lot of entries in
>> unhandled.log.
> 
> So what did you specify in your .git/config file regarding the svn
> structure?
> 

I specified the 'branches' directory, but that's because earlier in the
life of the repo we did just do the flat branch layout, but decided to
make it more structured once that got unwieldy.

Is it possible to specify more than one folder for the branches option?

-- 

Russ

Re: git-svn and a nested branches folder

[David Kastrup]

Russ Brown <pickscrape@gmail.com> writes:

> David Kastrup wrote:
>> Russ Brown <pickscrape@gmail.com> writes:
>> 
>>> I'm having some trouble with using git-svn to fetch a repository, and I
>>> think it's because the repository doesn't store branches as a flat list
>>> directly under the 'branches' directory.
>>>
>>> Basically, we have a structure like this:
>>>
>>> |
>>> +-trunk
>>> +-tags
>>> +-branches
>>>   + category-a
>>>     + branch-a
>>>     + branch-b
>>>   + category-b
>>>     + branch-c
>>>     + branch-d
>>>
>>> etc. category-a and category-b are simple directories created using svn
>>> mkdir. The branches are created using svn cp.
>>>
>>> It helps us to organise the branches better, but the rationale is
>>> besides the point. The problem is that git-svn seems to want to
>>> treat category-a and category-b as branches, which isn't right at
>>> all. As a result, git-svn seems to skip most (if not all) revisions
>>> that occur in these directories and creates a lot of entries in
>>> unhandled.log.
>> 
>> So what did you specify in your .git/config file regarding the svn
>> structure?
>
> I specified the 'branches' directory, but that's because earlier in
> the life of the repo we did just do the flat branch layout, but
> decided to make it more structured once that got unwieldy.

Cough, cough.  _What_ did you specify in your .git/config file
regarding the svn structure?  Please quote the section.

> Is it possible to specify more than one folder for the branches
> option?

It is possible to adapt the config section to the actual layout.  If
not otherwise, by starting with
git svn init
with a clean slate, editing the config file, and only then actually
fetching stuff.

However, git-svn will not magically start guessing that you changed
your structure around.  You have to edit the configuration
appropriately.

-- 
David Kastrup

Re: git-svn and a nested branches folder

[Russ Brown]

David Kastrup wrote:
> Russ Brown <pickscrape@gmail.com> writes:
> 
>> David Kastrup wrote:
>>> Russ Brown <pickscrape@gmail.com> writes:
>>>
>>>> I'm having some trouble with using git-svn to fetch a repository, and I
>>>> think it's because the repository doesn't store branches as a flat list
>>>> directly under the 'branches' directory.
>>>>
>>>> Basically, we have a structure like this:
>>>>
>>>> |
>>>> +-trunk
>>>> +-tags
>>>> +-branches
>>>>   + category-a
>>>>     + branch-a
>>>>     + branch-b
>>>>   + category-b
>>>>     + branch-c
>>>>     + branch-d
>>>>
>>>> etc. category-a and category-b are simple directories created using svn
>>>> mkdir. The branches are created using svn cp.
>>>>
>>>> It helps us to organise the branches better, but the rationale is
>>>> besides the point. The problem is that git-svn seems to want to
>>>> treat category-a and category-b as branches, which isn't right at
>>>> all. As a result, git-svn seems to skip most (if not all) revisions
>>>> that occur in these directories and creates a lot of entries in
>>>> unhandled.log.
>>> So what did you specify in your .git/config file regarding the svn
>>> structure?
>> I specified the 'branches' directory, but that's because earlier in
>> the life of the repo we did just do the flat branch layout, but
>> decided to make it more structured once that got unwieldy.
> 
> Cough, cough.  _What_ did you specify in your .git/config file
> regarding the svn structure?  Please quote the section.
> 

Erm, sorry.

[svn-remote "svn"]
        url = svn://svn.<name>.com
        fetch = trunk:refs/remotes/trunk
        branches = branches/*:refs/remotes/*
        tags = tags/*:refs/remotes/tags/*

(URL changed in case it annoys my employers)

I didn't write this by hand: it was generated by git-svn init.

>> Is it possible to specify more than one folder for the branches
>> option?
> 
> It is possible to adapt the config section to the actual layout.  If
> not otherwise, by starting with
> git svn init
> with a clean slate, editing the config file, and only then actually
> fetching stuff.
> 
> However, git-svn will not magically start guessing that you changed
> your structure around.  You have to edit the configuration
> appropriately.
> 

That's why I suggested that a method involving detecting branches based
on whether the directory is a copy of trunk or another branch might
'magically' work in all scenarios. I've used a similar branch 'scanning'
technique before for a different reason. But I realise there may be
technical reasons as to why that might not be possible.

-- 

Russ

Re: git-svn and a nested branches folder

[Peter Baumann]

On Tue, Sep 04, 2007 at 10:21:22AM -0500, Russ Brown wrote:
> David Kastrup wrote:
> > Russ Brown <pickscrape@gmail.com> writes:
> > 
> >> David Kastrup wrote:
> >>> Russ Brown <pickscrape@gmail.com> writes:
> >>>
> >>>> I'm having some trouble with using git-svn to fetch a repository, and I
> >>>> think it's because the repository doesn't store branches as a flat list
> >>>> directly under the 'branches' directory.
> >>>>
> >>>> Basically, we have a structure like this:
> >>>>
> >>>> |
> >>>> +-trunk
> >>>> +-tags
> >>>> +-branches
> >>>>   + category-a
> >>>>     + branch-a
> >>>>     + branch-b
> >>>>   + category-b
> >>>>     + branch-c
> >>>>     + branch-d
> >>>>
> >>>> etc. category-a and category-b are simple directories created using svn
> >>>> mkdir. The branches are created using svn cp.
> >>>>
> >>>> It helps us to organise the branches better, but the rationale is
> >>>> besides the point. The problem is that git-svn seems to want to
> >>>> treat category-a and category-b as branches, which isn't right at
> >>>> all. As a result, git-svn seems to skip most (if not all) revisions
> >>>> that occur in these directories and creates a lot of entries in
> >>>> unhandled.log.
> >>> So what did you specify in your .git/config file regarding the svn
> >>> structure?
> >> I specified the 'branches' directory, but that's because earlier in
> >> the life of the repo we did just do the flat branch layout, but
> >> decided to make it more structured once that got unwieldy.
> > 
> > Cough, cough.  _What_ did you specify in your .git/config file
> > regarding the svn structure?  Please quote the section.
> > 
> 
> Erm, sorry.
> 
> [svn-remote "svn"]
>         url = svn://svn.<name>.com
>         fetch = trunk:refs/remotes/trunk
>         branches = branches/*:refs/remotes/*
>         tags = tags/*:refs/remotes/tags/*
> 
> (URL changed in case it annoys my employers)
> 
> I didn't write this by hand: it was generated by git-svn init.
> 

Try something like this:

[svn-remote "svn"]
	# trunk
	fetch = trunk:refs/remotes/trunk

	# branches
	fetch = branches/category-a/branch_a:refs/remotes/svn/branch_a
	fetch = branches/category-a/branch_b:refs/remotes/svn/branch_b
	fetch = branches/category-b/branch_c:refs/remotes/svn/branch_c

	# tags
	tags = tags/*:refs/remotes/tags/*


(Not sure if wildcards will work here, but I'm sure you could experiment and try
 it out :-)

-Peter

Re: git-svn and a nested branches folder

[Russ Brown]

Peter Baumann wrote:
> On Tue, Sep 04, 2007 at 10:21:22AM -0500, Russ Brown wrote:
>> David Kastrup wrote:
>>> Russ Brown <pickscrape@gmail.com> writes:
>>>
>>>> David Kastrup wrote:
>>>>> Russ Brown <pickscrape@gmail.com> writes:
>>>>>
>>>>>> I'm having some trouble with using git-svn to fetch a repository, and I
>>>>>> think it's because the repository doesn't store branches as a flat list
>>>>>> directly under the 'branches' directory.
>>>>>>
>>>>>> Basically, we have a structure like this:
>>>>>>
>>>>>> |
>>>>>> +-trunk
>>>>>> +-tags
>>>>>> +-branches
>>>>>>   + category-a
>>>>>>     + branch-a
>>>>>>     + branch-b
>>>>>>   + category-b
>>>>>>     + branch-c
>>>>>>     + branch-d
>>>>>>
>>>>>> etc. category-a and category-b are simple directories created using svn
>>>>>> mkdir. The branches are created using svn cp.
>>>>>>
>>>>>> It helps us to organise the branches better, but the rationale is
>>>>>> besides the point. The problem is that git-svn seems to want to
>>>>>> treat category-a and category-b as branches, which isn't right at
>>>>>> all. As a result, git-svn seems to skip most (if not all) revisions
>>>>>> that occur in these directories and creates a lot of entries in
>>>>>> unhandled.log.
>>>>> So what did you specify in your .git/config file regarding the svn
>>>>> structure?
>>>> I specified the 'branches' directory, but that's because earlier in
>>>> the life of the repo we did just do the flat branch layout, but
>>>> decided to make it more structured once that got unwieldy.
>>> Cough, cough.  _What_ did you specify in your .git/config file
>>> regarding the svn structure?  Please quote the section.
>>>
>> Erm, sorry.
>>
>> [svn-remote "svn"]
>>         url = svn://svn.<name>.com
>>         fetch = trunk:refs/remotes/trunk
>>         branches = branches/*:refs/remotes/*
>>         tags = tags/*:refs/remotes/tags/*
>>
>> (URL changed in case it annoys my employers)
>>
>> I didn't write this by hand: it was generated by git-svn init.
>>
> 
> Try something like this:
> 
> [svn-remote "svn"]
> 	# trunk
> 	fetch = trunk:refs/remotes/trunk
> 
> 	# branches
> 	fetch = branches/category-a/branch_a:refs/remotes/svn/branch_a
> 	fetch = branches/category-a/branch_b:refs/remotes/svn/branch_b
> 	fetch = branches/category-b/branch_c:refs/remotes/svn/branch_c
> 
> 	# tags
> 	tags = tags/*:refs/remotes/tags/*
> 
> 
> (Not sure if wildcards will work here, but I'm sure you could experiment and try
>  it out :-)
> 

Thanks a lot! I'll try it out.

I wonder if I'm going to have to fetch in stages here. i.e. configuring
the branches for the original layout, fetching up the point where the
layout changed a bit and reconfigure, fetch some more, tweak again etc.

> -Peter


-- 

Russ

Re: git-svn and a nested branches folder

[Eric Wong]

Russ Brown <pickscrape@gmail.com> wrote:
> Hi,

Hi Russ,

> I'm having some trouble with using git-svn to fetch a repository, and I
> think it's because the repository doesn't store branches as a flat list
> directly under the 'branches' directory.
> 
> Basically, we have a structure like this:
> 
> |
> +-trunk
> +-tags
> +-branches
>   + category-a
>     + branch-a
>     + branch-b
>   + category-b
>     + branch-c
>     + branch-d
> 
> etc. category-a and category-b are simple directories created using svn
> mkdir. The branches are created using svn cp.
> 
> It helps us to organise the branches better, but the rationale is
> besides the point. The problem is that git-svn seems to want to treat
> category-a and category-b as branches, which isn't right at all. As a
> result, git-svn seems to skip most (if not all) revisions that occur in
> these directories and creates a lot of entries in unhandled.log.

This is a known problem with the way git-svn handles branches and tags.
Nested branch (and tags) structures aren't supported with globbing and
so you can't have more than one branches/tags specification in your
config.

> I've also encountered an index corruption in one of the
> .git/svn/<branch>/index files which I think it probably related.

Not sure about that.  git-svn should detect and attempt to fix index
corruptions (which happened for me since I interrupt git-svn quite
often when working on it).

> I've had a quick look at the source myself, but perl isn't my strong
> point. What I think it should do is something like recurse down the tree
> from the directory given looking for folders that copy from trunk (or
> some other branch that already exists). That would work perfectly well
> for the default flat branch storage method as well as the one we use.

The globbing functionality of branches in git-svn is some of the ugliest
code I've ever written in my life.  Somehow I got it working for the
simple general cases but I've been afraid to touch it for a while...

> The only other problem is in branch naming, which could clash if you
> only use the outer-most directory name, so I'd suggest something that
> involves concatenating the folders in the path relative to 'branches' to
> keep them unique (if git can handle slashes in branch names then all the
> better).

As Peter suggested, disable globbing for branches and use explicit
fetch refspecs for now...

-- 
Eric Wong

Re: git-svn and a nested branches folder

[Russ Brown]

Eric Wong wrote:
> Russ Brown <pickscrape@gmail.com> wrote:
>> Hi,
> 
> Hi Russ,
> 
>> I'm having some trouble with using git-svn to fetch a repository, and I
>> think it's because the repository doesn't store branches as a flat list
>> directly under the 'branches' directory.
>>
>> Basically, we have a structure like this:
>>
>> |
>> +-trunk
>> +-tags
>> +-branches
>>   + category-a
>>     + branch-a
>>     + branch-b
>>   + category-b
>>     + branch-c
>>     + branch-d
>>
>> etc. category-a and category-b are simple directories created using svn
>> mkdir. The branches are created using svn cp.
>>
>> It helps us to organise the branches better, but the rationale is
>> besides the point. The problem is that git-svn seems to want to treat
>> category-a and category-b as branches, which isn't right at all. As a
>> result, git-svn seems to skip most (if not all) revisions that occur in
>> these directories and creates a lot of entries in unhandled.log.
> 
> This is a known problem with the way git-svn handles branches and tags.
> Nested branch (and tags) structures aren't supported with globbing and
> so you can't have more than one branches/tags specification in your
> config.
> 
>> I've also encountered an index corruption in one of the
>> .git/svn/<branch>/index files which I think it probably related.
> 
> Not sure about that.  git-svn should detect and attempt to fix index
> corruptions (which happened for me since I interrupt git-svn quite
> often when working on it).
> 

I've since trashes that repo and it's not happened again, so it must
have been cosmic particles or something. :)

>> I've had a quick look at the source myself, but perl isn't my strong
>> point. What I think it should do is something like recurse down the tree
>> from the directory given looking for folders that copy from trunk (or
>> some other branch that already exists). That would work perfectly well
>> for the default flat branch storage method as well as the one we use.
> 
> The globbing functionality of branches in git-svn is some of the ugliest
> code I've ever written in my life.  Somehow I got it working for the
> simple general cases but I've been afraid to touch it for a while...
> 

Yeah, I know what you mean. :) A really nasty thing to get working and
once you do you don't want to touch it again. :)

>> The only other problem is in branch naming, which could clash if you
>> only use the outer-most directory name, so I'd suggest something that
>> involves concatenating the folders in the path relative to 'branches' to
>> keep them unique (if git can handle slashes in branch names then all the
>> better).
> 
> As Peter suggested, disable globbing for branches and use explicit
> fetch refspecs for now...
> 

I've actually knocked up a rough script which generates a list of
refspec lines for you given a repo URL, trunk reference and branches
directory. It uses svn log -v --xml and pipes it through a couple of
XSLT templates, and basically looks for all copies that copy from trunk
(recursively: so it includes branches of branches too). I can post it to
the list if you'd find it useful or interesting.

It's generating output that looks sensible to me, but the results aren't
quite what I'd expected. I'll paste a sample in here in case there's
anything obvious someone might spot that I've missed

# This line was generated by git-svn init, and I kept it
fetch = all/trunk:refs/remotes/trunk

# These lines generated by my tool, dirnames replaced for security reasons:

fetch = branches/folder/projecta:refs/remotes/svn/folder/projecta
fetch = branches/folder/projectb:refs/remotes/svn/folder/projectb
fetch = branches/folder/projectc:refs/remotes/svn/folder/projectc
fetch = branches/folder/projectd:refs/remotes/svn/folder/projectd
fetch = branches/folder/projecte:refs/remotes/svn/folder/projecte
fetch = branches/folder/projectf:refs/remotes/svn/folder/projectf
fetch = branches/folder/projectg:refs/remotes/svn/folder/projectg

git branch -a doesn't list any of those branches after fetch completes.
Looking back at the output from fetch, all revisions applied were to trunk.

Anything wrong with those fetch lines?

Thanks for your time.

-- 

Russ

Re: git-svn and a nested branches folder

[Eric Wong]

Russ Brown <pickscrape@gmail.com> wrote:
> Eric Wong wrote:
> > Russ Brown <pickscrape@gmail.com> wrote:
> >> Basically, we have a structure like this:
> >>
> >> |
> >> +-trunk
> >> +-tags
> >> +-branches
> >>   + category-a
> >>     + branch-a
> >>     + branch-b
> >>   + category-b
> >>     + branch-c
> >>     + branch-d

> >> The only other problem is in branch naming, which could clash if you
> >> only use the outer-most directory name, so I'd suggest something that
> >> involves concatenating the folders in the path relative to 'branches' to
> >> keep them unique (if git can handle slashes in branch names then all the
> >> better).
> > 
> > As Peter suggested, disable globbing for branches and use explicit
> > fetch refspecs for now...
> > 
> 
> I've actually knocked up a rough script which generates a list of
> refspec lines for you given a repo URL, trunk reference and branches
> directory. It uses svn log -v --xml and pipes it through a couple of
> XSLT templates, and basically looks for all copies that copy from trunk
> (recursively: so it includes branches of branches too). I can post it to
> the list if you'd find it useful or interesting.
> 
> It's generating output that looks sensible to me, but the results aren't
> quite what I'd expected. I'll paste a sample in here in case there's
> anything obvious someone might spot that I've missed
> 
> # This line was generated by git-svn init, and I kept it
> fetch = all/trunk:refs/remotes/trunk
> 
> # These lines generated by my tool, dirnames replaced for security reasons:
> 
> fetch = branches/folder/projecta:refs/remotes/svn/folder/projecta
> fetch = branches/folder/projectb:refs/remotes/svn/folder/projectb
> fetch = branches/folder/projectc:refs/remotes/svn/folder/projectc
> fetch = branches/folder/projectd:refs/remotes/svn/folder/projectd
> fetch = branches/folder/projecte:refs/remotes/svn/folder/projecte
> fetch = branches/folder/projectf:refs/remotes/svn/folder/projectf
> fetch = branches/folder/projectg:refs/remotes/svn/folder/projectg
> 
> git branch -a doesn't list any of those branches after fetch completes.
> Looking back at the output from fetch, all revisions applied were to trunk.
> 
> Anything wrong with those fetch lines?

>From your tree diagram, it seemed that trunk/ and branches/ were at
the same depth in your SVN repository.  However, in your generated
fetch lines they all started with "branches/" in front, yet your
trunk fetch line had "all/" in front of trunk, so maybe prefixing
the generated ones with "all/" helps?

> Thanks for your time.

No problem.

-- 
Eric Wong

Re: git-svn and a nested branches folder

[Russ Brown]

Eric Wong wrote:
> Russ Brown <pickscrape@gmail.com> wrote:
>> Eric Wong wrote:
>>> Russ Brown <pickscrape@gmail.com> wrote:
>>>> Basically, we have a structure like this:
>>>>
>>>> |
>>>> +-trunk
>>>> +-tags
>>>> +-branches
>>>>   + category-a
>>>>     + branch-a
>>>>     + branch-b
>>>>   + category-b
>>>>     + branch-c
>>>>     + branch-d
> 
>>>> The only other problem is in branch naming, which could clash if you
>>>> only use the outer-most directory name, so I'd suggest something that
>>>> involves concatenating the folders in the path relative to 'branches' to
>>>> keep them unique (if git can handle slashes in branch names then all the
>>>> better).
>>> As Peter suggested, disable globbing for branches and use explicit
>>> fetch refspecs for now...
>>>
>> I've actually knocked up a rough script which generates a list of
>> refspec lines for you given a repo URL, trunk reference and branches
>> directory. It uses svn log -v --xml and pipes it through a couple of
>> XSLT templates, and basically looks for all copies that copy from trunk
>> (recursively: so it includes branches of branches too). I can post it to
>> the list if you'd find it useful or interesting.
>>
>> It's generating output that looks sensible to me, but the results aren't
>> quite what I'd expected. I'll paste a sample in here in case there's
>> anything obvious someone might spot that I've missed
>>
>> # This line was generated by git-svn init, and I kept it
>> fetch = all/trunk:refs/remotes/trunk
>>
>> # These lines generated by my tool, dirnames replaced for security reasons:
>>
>> fetch = branches/folder/projecta:refs/remotes/svn/folder/projecta
>> fetch = branches/folder/projectb:refs/remotes/svn/folder/projectb
>> fetch = branches/folder/projectc:refs/remotes/svn/folder/projectc
>> fetch = branches/folder/projectd:refs/remotes/svn/folder/projectd
>> fetch = branches/folder/projecte:refs/remotes/svn/folder/projecte
>> fetch = branches/folder/projectf:refs/remotes/svn/folder/projectf
>> fetch = branches/folder/projectg:refs/remotes/svn/folder/projectg
>>
>> git branch -a doesn't list any of those branches after fetch completes.
>> Looking back at the output from fetch, all revisions applied were to trunk.
>>
>> Anything wrong with those fetch lines?
> 
> From your tree diagram, it seemed that trunk/ and branches/ were at
> the same depth in your SVN repository.  However, in your generated
> fetch lines they all started with "branches/" in front, yet your
> trunk fetch line had "all/" in front of trunk, so maybe prefixing
> the generated ones with "all/" helps?
> 

Ah! I think I know what I've done. I generated the fetch list from the
svn repository itself, but ran svn-init against an svk mirror (using
--use-svm-props). Nicely spotted. :)

Thanks again!

>> Thanks for your time.
> 
> No problem.
> 


-- 

Russ