[PATCH v2 0/8] remote-bzr: patches for next

[PATCH v2 0/8] remote-bzr: patches for next

[Felipe Contreras]

Hi,

Minor fixes since last time.

These patches have been cooking in my github repository, and improve the
situation when bzr servers don't support repositories properly.

Felipe Contreras (8):
  remote-bzr: recover from failed clones
  remote-bzr: fix for files with spaces
  remote-bzr: simplify get_remote_branch()
  remote-bzr: delay cloning/pulling
  remote-bzr: change global repo
  remote-bzr: trivial cleanups
  remote-bzr: reorganize the way 'wanted' works
  remote-bzr: add fallback check for a partial clone

 contrib/remote-helpers/git-remote-bzr | 112 +++++++++++++++++-----------------
 1 file changed, 55 insertions(+), 57 deletions(-)

-- 
1.8.3.rc3.312.g47657de

[PATCH v2 1/8] remote-bzr: recover from failed clones

[Felipe Contreras]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 10300c6..8a4df51 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -769,22 +769,24 @@ def get_remote_branch(origin, remote_branch, name):
     global dirname, peers
 
     branch_path = os.path.join(dirname, 'clone', name)
-    if os.path.exists(branch_path):
-        # pull
+
+    try:
         d = bzrlib.bzrdir.BzrDir.open(branch_path)
         branch = d.open_branch()
-        try:
-            branch.pull(remote_branch, [], None, False)
-        except bzrlib.errors.DivergedBranches:
-            # use remote branch for now
-            return remote_branch
-    else:
+    except bzrlib.errors.NotBranchError:
         # clone
         d = origin.sprout(branch_path, None,
                 hardlink=True, create_tree_if_local=False,
                 force_new_repo=False,
                 source_branch=remote_branch)
         branch = d.open_branch()
+    else:
+        # pull
+        try:
+            branch.pull(remote_branch, [], None, False)
+        except bzrlib.errors.DivergedBranches:
+            # use remote branch for now
+            return remote_branch
 
     return branch
 
-- 
1.8.3.rc3.312.g47657de

[PATCH v2 2/8] remote-bzr: fix for files with spaces

[Felipe Contreras]

Set the maximum number of splits to make when dividing the diff stat
lines based on space characters.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 8a4df51..35664c6 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -621,7 +621,7 @@ def parse_commit(parser):
             mark = int(mark_ref[1:])
             f = { 'mode' : m, 'mark' : mark }
         elif parser.check('D'):
-            t, path = line.split(' ')
+            t, path = line.split(' ', 1)
             f = { 'deleted' : True }
         else:
             die('Unknown file command: %s' % line)
-- 
1.8.3.rc3.312.g47657de

[PATCH v2 3/8] remote-bzr: simplify get_remote_branch()

[Felipe Contreras]

No need for 'origin', it's only needed for the bzrdir 'sprout' method,
which can be greatly simplified.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 35664c6..5c4201a 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -765,25 +765,26 @@ def do_list(parser):
     print "@refs/heads/%s HEAD" % master_branch
     print
 
-def get_remote_branch(origin, remote_branch, name):
+def clone(path, remote_branch):
+    bdir = bzrlib.bzrdir.BzrDir.create(path)
+    repo = bdir.find_repository()
+    repo.fetch(remote_branch.repository)
+    return remote_branch.sprout(bdir, repository=repo)
+
+def get_remote_branch(remote_branch, name):
     global dirname, peers
 
     branch_path = os.path.join(dirname, 'clone', name)
 
     try:
-        d = bzrlib.bzrdir.BzrDir.open(branch_path)
-        branch = d.open_branch()
+        branch = bzrlib.branch.Branch.open(branch_path)
     except bzrlib.errors.NotBranchError:
         # clone
-        d = origin.sprout(branch_path, None,
-                hardlink=True, create_tree_if_local=False,
-                force_new_repo=False,
-                source_branch=remote_branch)
-        branch = d.open_branch()
+        branch = clone(branch_path, remote_branch)
     else:
         # pull
         try:
-            branch.pull(remote_branch, [], None, False)
+            branch.pull(remote_branch, overwrite=True)
         except bzrlib.errors.DivergedBranches:
             # use remote branch for now
             return remote_branch
@@ -856,7 +857,7 @@ def get_repo(url, alias):
 
         if not is_local:
             peers[name] = remote_branch.base
-            branch = get_remote_branch(origin, remote_branch, name)
+            branch = get_remote_branch(remote_branch, name)
         else:
             branch = remote_branch
 
@@ -874,7 +875,7 @@ def get_repo(url, alias):
 
             if not is_local:
                 peers[name] = remote_branch.base
-                branch = get_remote_branch(origin, remote_branch, name)
+                branch = get_remote_branch(remote_branch, name)
             else:
                 branch = remote_branch
 
-- 
1.8.3.rc3.312.g47657de

[PATCH v2 4/8] remote-bzr: delay cloning/pulling

[Felipe Contreras]

Until the branch is actually going to be used.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 5c4201a..202a4f7 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -278,7 +278,7 @@ def export_branch(repo, name):
     ref = '%s/heads/%s' % (prefix, name)
     tip = marks.get_tip(name)
 
-    branch = bzrlib.branch.Branch.open(branches[name])
+    branch = get_remote_branch(name)
     repo = branch.repository
 
     branch.lock_read()
@@ -590,7 +590,7 @@ def parse_commit(parser):
 
     if ref.startswith('refs/heads/'):
         name = ref[len('refs/heads/'):]
-        branch = bzrlib.branch.Branch.open(branches[name])
+        branch = get_remote_branch(name)
     else:
         die('unknown ref')
 
@@ -692,7 +692,7 @@ def do_export(parser):
     for ref, revid in parsed_refs.iteritems():
         if ref.startswith('refs/heads/'):
             name = ref[len('refs/heads/'):]
-            branch = bzrlib.branch.Branch.open(branches[name])
+            branch = get_remote_branch(name)
             branch.generate_revision_history(revid, marks.get_tip(name))
 
             if name in peers:
@@ -749,7 +749,7 @@ def do_list(parser):
             master_branch = name
         print "? refs/heads/%s" % name
 
-    branch = bzrlib.branch.Branch.open(branches[master_branch])
+    branch = get_remote_branch(master_branch)
     branch.lock_read()
     for tag, revid in branch.tags.get_tag_dict().items():
         try:
@@ -771,8 +771,12 @@ def clone(path, remote_branch):
     repo.fetch(remote_branch.repository)
     return remote_branch.sprout(bdir, repository=repo)
 
-def get_remote_branch(remote_branch, name):
-    global dirname, peers
+def get_remote_branch(name):
+    global dirname, branches
+
+    remote_branch = bzrlib.branch.Branch.open(branches[name])
+    if isinstance(remote_branch.user_transport, bzrlib.transport.local.LocalTransport):
+        return remote_branch
 
     branch_path = os.path.join(dirname, 'clone', name)
 
@@ -857,13 +861,10 @@ def get_repo(url, alias):
 
         if not is_local:
             peers[name] = remote_branch.base
-            branch = get_remote_branch(remote_branch, name)
-        else:
-            branch = remote_branch
 
-        branches[name] = branch.base
+        branches[name] = remote_branch.base
 
-        return branch.repository
+        return remote_branch.repository
     else:
         # repository
 
@@ -875,11 +876,8 @@ def get_repo(url, alias):
 
             if not is_local:
                 peers[name] = remote_branch.base
-                branch = get_remote_branch(remote_branch, name)
-            else:
-                branch = remote_branch
 
-            branches[name] = branch.base
+            branches[name] = remote_branch.base
 
         return repo
 
-- 
1.8.3.rc3.312.g47657de

[PATCH v2 5/8] remote-bzr: change global repo

[Felipe Contreras]

It's not used anyway.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 202a4f7..80ed59f 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -864,7 +864,7 @@ def get_repo(url, alias):
 
         branches[name] = remote_branch.base
 
-        return remote_branch.repository
+        return origin
     else:
         # repository
 
@@ -879,7 +879,7 @@ def get_repo(url, alias):
 
             branches[name] = remote_branch.base
 
-        return repo
+        return origin
 
 def fix_path(alias, orig_url):
     url = urlparse.urlparse(orig_url, 'file')
-- 
1.8.3.rc3.312.g47657de

[PATCH v2 6/8] remote-bzr: trivial cleanups

[Felipe Contreras]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 80ed59f..34025c3 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -815,7 +815,7 @@ def find_branches(repo, wanted):
         except bzrlib.errors.NotBranchError:
             continue
         else:
-            yield name, branch
+            yield name, branch.base
 
 def get_repo(url, alias):
     global dirname, peer, branches
@@ -857,12 +857,12 @@ def get_repo(url, alias):
         # branch
 
         name = 'master'
-        remote_branch = origin.open_branch()
+        branch = origin.open_branch().base
 
         if not is_local:
-            peers[name] = remote_branch.base
+            peers[name] = branch
 
-        branches[name] = remote_branch.base
+        branches[name] = branch
 
         return origin
     else:
@@ -872,12 +872,12 @@ def get_repo(url, alias):
         # stupid python
         wanted = [e for e in wanted if e]
 
-        for name, remote_branch in find_branches(repo, wanted):
+        for name, branch in find_branches(repo, wanted):
 
             if not is_local:
-                peers[name] = remote_branch.base
+                peers[name] = branch
 
-            branches[name] = remote_branch.base
+            branches[name] = branch
 
         return origin
 
-- 
1.8.3.rc3.312.g47657de

[PATCH v2 7/8] remote-bzr: reorganize the way 'wanted' works

[Felipe Contreras]

If the user specified a list of branches, we ignore what the remote
repository lists, and simply use the branches directly. Since some
remotes don't report the branches correctly, this is useful.

Otherwise either fetch the repo, or the branch.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 58 ++++++++++++++++-------------------
 1 file changed, 26 insertions(+), 32 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 34025c3..3248586 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -795,7 +795,7 @@ def get_remote_branch(name):
 
     return branch
 
-def find_branches(repo, wanted):
+def find_branches(repo):
     transport = repo.bzrdir.root_transport
 
     for fn in transport.iter_files_recursive():
@@ -806,9 +806,6 @@ def find_branches(repo, wanted):
         name = name if name != '' else 'master'
         name = name.replace('/', '+')
 
-        if wanted and not name in wanted:
-            continue
-
         try:
             cur = transport.clone(subdir)
             branch = bzrlib.branch.Branch.open_from_transport(cur)
@@ -848,38 +845,35 @@ def get_repo(url, alias):
             except bzrlib.errors.NoRepositoryPresent:
                 pass
 
-    try:
-        repo = origin.open_repository()
-        if not repo.user_transport.listable():
-            # this repository is not usable for us
-            raise bzrlib.errors.NoRepositoryPresent(repo.bzrdir)
-    except bzrlib.errors.NoRepositoryPresent:
-        # branch
-
-        name = 'master'
-        branch = origin.open_branch().base
-
-        if not is_local:
-            peers[name] = branch
+    wanted = get_config('remote-bzr.branches').rstrip().split(', ')
+    # stupid python
+    wanted = [e for e in wanted if e]
 
-        branches[name] = branch
-
-        return origin
+    if not wanted:
+        try:
+            repo = origin.open_repository()
+            if not repo.user_transport.listable():
+                # this repository is not usable for us
+                raise bzrlib.errors.NoRepositoryPresent(repo.bzrdir)
+        except bzrlib.errors.NoRepositoryPresent:
+            wanted = ['master']
+
+    if wanted:
+        def list_wanted(url, wanted):
+            for name in wanted:
+                subdir = name if name != 'master' else ''
+                yield name, bzrlib.urlutils.join(url, subdir)
+
+        branch_list = list_wanted(url, wanted)
     else:
-        # repository
-
-        wanted = get_config('remote-bzr.branches').rstrip().split(', ')
-        # stupid python
-        wanted = [e for e in wanted if e]
+        branch_list = find_branches(repo)
 
-        for name, branch in find_branches(repo, wanted):
-
-            if not is_local:
-                peers[name] = branch
-
-            branches[name] = branch
+    for name, url in branch_list:
+        if not is_local:
+            peers[name] = url
+        branches[name] = url
 
-        return origin
+    return origin
 
 def fix_path(alias, orig_url):
     url = urlparse.urlparse(orig_url, 'file')
-- 
1.8.3.rc3.312.g47657de

[PATCH v2 8/8] remote-bzr: add fallback check for a partial clone

[Felipe Contreras]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 3248586..3cd6572 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -766,7 +766,10 @@ def do_list(parser):
     print
 
 def clone(path, remote_branch):
-    bdir = bzrlib.bzrdir.BzrDir.create(path)
+    try:
+        bdir = bzrlib.bzrdir.BzrDir.create(path)
+    except bzrlib.errors.AlreadyControlDirError:
+        bdir = bzrlib.bzrdir.BzrDir.open(path)
     repo = bdir.find_repository()
     repo.fetch(remote_branch.repository)
     return remote_branch.sprout(bdir, repository=repo)
-- 
1.8.3.rc3.312.g47657de

Re: [PATCH v2 7/8] remote-bzr: reorganize the way 'wanted' works

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> +    wanted = get_config('remote-bzr.branches').rstrip().split(', ')

Two minor nits and one design suggestion:

 - Why rstrip() not strip()?  It appears that this only is helping
   an end-user "mistake" like this:

	git config remote-bzr.branches 'trunk, devel, test '

   without helping people who have done this:

	git config remote-bzr.branches 'trunk,  devel, test'

 - Is

     git config remote-bzr.branches trunk,devel,test

   a grave sin?

   In other words, wouldn't we want something like this instead?

	map(lambda s: s.strip(), get_config('...').split(','))

 - Doesn't allowing multi-valued variable, e.g.

	[remote-bzr]
            branches = trunk
            branches = devel
            branches = test

   make it easier for the user to manage this configuration by
   e.g. selectively removing or adding tracked branches?

Re: [PATCH v2 7/8] remote-bzr: reorganize the way 'wanted' works

[Felipe Contreras]

Junio C Hamano <gitster@pobox.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
> 
> > +    wanted = get_config('remote-bzr.branches').rstrip().split(', ')
> 
> Two minor nits and one design suggestion:
> 
>  - Why rstrip() not strip()?

The purpose of the strip is to remove the _single_ "\n" at the end that
subprocess communicate. Maybe get_config() should do that.

> It appears that this only is helping
>    an end-user "mistake" like this:
> 
> 	git config remote-bzr.branches 'trunk, devel, test '
> 
>    without helping people who have done this:
> 
> 	git config remote-bzr.branches 'trunk,  devel, test'

No, that's tnot it.

>  - Is
> 
>      git config remote-bzr.branches trunk,devel,test
> 
>    a grave sin?
> 
>    In other words, wouldn't we want something like this instead?
> 
> 	map(lambda s: s.strip(), get_config('...').split(','))

Yeah, that might make sense.

>  - Doesn't allowing multi-valued variable, e.g.
> 
> 	[remote-bzr]
>             branches = trunk
>             branches = devel
>             branches = test
> 
>    make it easier for the user to manage this configuration by
>    e.g. selectively removing or adding tracked branches?

How would the 'git config' command look like?

Either way, that's orthogonal to this patch.

-- 
Felipe Contreras

Re: [PATCH v2 7/8] remote-bzr: reorganize the way 'wanted' works

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> Junio C Hamano <gitster@pobox.com> wrote:
>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>> 
>> > +    wanted = get_config('remote-bzr.branches').rstrip().split(', ')
>> 
>> Two minor nits and one design suggestion:
>> 
>>  - Why rstrip() not strip()?
>
> The purpose of the strip is to remove the _single_ "\n" at the end that
> subprocess communicate. Maybe get_config() should do that.
>
>> It appears that this only is helping
>>    an end-user "mistake" like this:
>> 
>> 	git config remote-bzr.branches 'trunk, devel, test '
>> 
>>    without helping people who have done this:
>> 
>> 	git config remote-bzr.branches 'trunk,  devel, test'
>
> No, that's tnot it.

Yes, rstrip() will also lose LF at the end.

But it also is true that your code also removes the trailing extra
SP in the first example above, while not losing the extra SP in the
middle in the second example, no?

So where does "that's tnot it" come from?  Is it true or false that
the former is helped while the latter is not?

>>  - Is
>> 
>>      git config remote-bzr.branches trunk,devel,test
>> 
>>    a grave sin?
>> 
>>    In other words, wouldn't we want something like this instead?
>> 
>> 	map(lambda s: s.strip(), get_config('...').split(','))
>
> Yeah, that might make sense.

If you go that route, you do not even have to even say "stupid
python".  You can write a more meaningful list comprehension, e.g.

	wanted = [s.strip() for s in get_config('...').split(',')]

without an unsightly lambda in it.

>>  - Doesn't allowing multi-valued variable, e.g.
>> 
>> 	[remote-bzr]
>>             branches = trunk
>>             branches = devel
>>             branches = test
>> 
>>    make it easier for the user to manage this configuration by
>>    e.g. selectively removing or adding tracked branches?
>
> How would the 'git config' command look like?
>
> Either way, that's orthogonal to this patch.

Yeah, I notice that "single variable, split at comma" comes way
before this series anyway, so it is too late (and it is not worth)
to fix it using multi-valued variables.  It was just an "if we were
designing this from scratch" kind of suggestion.

Re: [PATCH v2 7/8] remote-bzr: reorganize the way 'wanted' works

[Felipe Contreras]

On Wed, May 29, 2013 at 11:42 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> Junio C Hamano <gitster@pobox.com> wrote:
>>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>>
>>> > +    wanted = get_config('remote-bzr.branches').rstrip().split(', ')
>>>
>>> Two minor nits and one design suggestion:
>>>
>>>  - Why rstrip() not strip()?
>>
>> The purpose of the strip is to remove the _single_ "\n" at the end that
>> subprocess communicate. Maybe get_config() should do that.
>>
>>> It appears that this only is helping
>>>    an end-user "mistake" like this:
>>>
>>>      git config remote-bzr.branches 'trunk, devel, test '
>>>
>>>    without helping people who have done this:
>>>
>>>      git config remote-bzr.branches 'trunk,  devel, test'
>>
>> No, that's tnot it.
>
> Yes, rstrip() will also lose LF at the end.
>
> But it also is true that your code also removes the trailing extra
> SP in the first example above, while not losing the extra SP in the
> middle in the second example, no?
>
> So where does "that's tnot it" come from?  Is it true or false that
> the former is helped while the latter is not?

You said it is *only* helping the end-user with mistakes, but that's
not true, it _also_  gets rid of the new line, which is not only
helping, but it's required for the code to work, and it's actually the
purpose behind the code. The side-effect of removing extra spaces if
the user makes mistakes is irrelevant.

>>>  - Is
>>>
>>>      git config remote-bzr.branches trunk,devel,test
>>>
>>>    a grave sin?
>>>
>>>    In other words, wouldn't we want something like this instead?
>>>
>>>      map(lambda s: s.strip(), get_config('...').split(','))
>>
>> Yeah, that might make sense.
>
> If you go that route, you do not even have to even say "stupid
> python".  You can write a more meaningful list comprehension, e.g.
>
>         wanted = [s.strip() for s in get_config('...').split(',')]
>
> without an unsightly lambda in it.

Python would still do the stupid thing if there's no such configuration:

['']

But we can add 'if s' at the end, so the code to fix python's
stupidness is much smaller.

I wonder what 's' means. In C I use 'i', because that's what everybody
uses, but in more functional-like code we don't use an index, we
iterate directly with the element of an enumerable, so I say 'e' for
short.

>>>  - Doesn't allowing multi-valued variable, e.g.
>>>
>>>      [remote-bzr]
>>>             branches = trunk
>>>             branches = devel
>>>             branches = test
>>>
>>>    make it easier for the user to manage this configuration by
>>>    e.g. selectively removing or adding tracked branches?
>>
>> How would the 'git config' command look like?
>>
>> Either way, that's orthogonal to this patch.
>
> Yeah, I notice that "single variable, split at comma" comes way
> before this series anyway, so it is too late (and it is not worth)
> to fix it using multi-valued variables.  It was just an "if we were
> designing this from scratch" kind of suggestion.

It might be worth, I'm not sure, I'm not familiar with those, and I
don't know what the user would have to type, but my guess is that it
won't be as user friendly as 'git config foo one,two,three'.

-- 
Felipe Contreras