[PATCH 00/20] remote-bzr: massive changes

[PATCH 00/20] remote-bzr: massive changes

[Felipe Contreras]

Hi,

After being contacted by the emacs developers and others who are stuck with
Bazaar, which at this point seems to be utterly abandoned, I realized the
current implementation is too crude.

Bazaar branches are very simplistic, and our support for them is the same; we
need to create one remote per branch. This works nicely if you work on small
projects with few branches, but doesn't scale.

Big projects like emacs take a lot of space, and creating a remote per branch
is unrealistic, because each remote will have the whole Bazaar repository
copied, wasting space, and time each time a remote (for a bzr branch) is set
up.

Moreoever, a developer needs to constantly reset the master branch to the
commit he/she wants to push before pushing, since the transport-helper
infraestructure doesn't support pushing with refspecs (xwidget:master).

That is of course, if pushing actually worked (which in many cases doesn't).

In short, our support for real-world projects suck.

These patches fix all the issues I encountrered.

 1) First of all, there are several improvements for pushing. Before, we failed
    when trying to push a merge, now, even tricky merges work.
 
 2) Secondly, bzr branches are tied to a transport, so they time out if not
    used for a period of time, and importing/exporting huge chunks of a
    repository do take some time. So now they are only opened when they are
    about to be used.

 3) Then the big one; now bzr repositories are supported. They are very simple:
    basically an object store with no notion of branches, so to find the
    branches we need to traverse a directory (sometimes) remote, to find them.
    This is how Bazaar does it, ableit very slowly. Naturally, a lot of code
    had to be changed to support more than one branch.

 4) In addition, now remotes share all the objects, so adding a new remote
    doesn't imply fetchng a bunch of duplicated objects. They are just re-used
    automagically.

 5) Since the bzr objects are shared now, it only makes sense to share the
    remote-bzr marks, so we don't have to fast-import them again.

 6) The code was also reorganized to keep referenced as few objects as
    possible, since Bazaar seems to be need *a ton* of memory for them.

Finally, after all these changes I was finally able to clone the whole emacs
repository, all 130685 commits, and 56 branches without running out of memory
in my modest laptop.

Since the purpose of remote-bzr is to actually be usable for the poor souls
stucks in DSCMs that are not git, these changes are a must. I propose they be
merged for the next major version of git (v1.8.3) if no issues are found. They
changes pass all the tests, and work on various repositories I've tried. I'll
ask the emacs developers to give them a try, and let's see how it goes.

Cheers.

Felipe Contreras (20):
  remote-bzr: cleanup CustomTree
  remote-bzr: delay blob fetching until the very end
  remote-bzr: fix order of locking in CustomTree
  remote-bzr: always try to update the worktree
  remote-bzr: add support to push merges
  remote-bzr: fixes for branch diverge
  remote-bzr: fix partially pushed merge
  remote-bzr: use branch variable when appropriate
  remote-bzr: add support for bzr repos
  remote-bzr: fix branch names
  remote-bzr: add support for shared repo
  remote-bzr: split marks file
  remote-bzr: improve author sanitazion
  remote-bzr: add custom method to find branches
  remote-bzr: add option to specify branches
  remote-bzr: improve progress reporting
  remote-bzr: iterate revisions properly
  remote-bzr: delay peer branch usage
  remote-bzr: access branches only when needed
  remote-bzr: export HEAD

 contrib/remote-helpers/git-remote-bzr | 365 +++++++++++++++++++++++++---------
 contrib/remote-helpers/test-bzr.sh    |  36 ++++
 2 files changed, 302 insertions(+), 99 deletions(-)

-- 
1.8.2.1.884.g3532a8d

[PATCH 01/20] remote-bzr: cleanup CustomTree

[Felipe Contreras]

This code was not used at all.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index c19ed0e..7452a57 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -385,9 +385,6 @@ class CustomTree():
     def __init__(self, repo, revid, parents, files):
         global files_cache
 
-        self.repo = repo
-        self.revid = revid
-        self.parents = parents
         self.updates = {}
 
         def copy_tree(revid):
@@ -435,7 +432,7 @@ class CustomTree():
             if basename == '':
                 return None
             fid = bzrlib.generate_ids.gen_file_id(path)
-            d = add_entry(fid, dirname, 'directory')
+            add_entry(fid, dirname, 'directory')
             return fid
 
         def add_entry(fid, path, kind, mode = None):
@@ -458,7 +455,6 @@ class CustomTree():
                     (None, executable))
             self.files[path] = change[0]
             changes.append(change)
-            return change
 
         def update_entry(fid, path, kind, mode = None):
             dirname, basename = os.path.split(path)
@@ -480,7 +476,6 @@ class CustomTree():
                     (None, executable))
             self.files[path] = change[0]
             changes.append(change)
-            return change
 
         def remove_entry(fid, path, kind):
             dirname, basename = os.path.split(path)
@@ -495,7 +490,6 @@ class CustomTree():
                     (None, None))
             del self.files[path]
             changes.append(change)
-            return change
 
         for fid, f in self.updates.iteritems():
             path = f['path']
-- 
1.8.2.1.884.g3532a8d

[PATCH 02/20] remote-bzr: delay blob fetching until the very end

[Felipe Contreras]

Might be more efficient, but the real reason to use the marks will be
revealed in upcoming patches.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 7452a57..91b5cda 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -506,10 +506,12 @@ class CustomTree():
         return changes
 
     def get_file_with_stat(self, file_id, path=None):
-        return (StringIO.StringIO(self.updates[file_id]['data']), None)
+        mark = self.updates[file_id]['mark']
+        return (StringIO.StringIO(blob_marks[mark]), None)
 
     def get_symlink_target(self, file_id):
-        return self.updates[file_id]['data']
+        mark = self.updates[file_id]['mark']
+        return blob_marks[mark]
 
 def c_style_unescape(string):
     if string[0] == string[-1] == '"':
@@ -553,7 +555,7 @@ def parse_commit(parser):
         if parser.check('M'):
             t, m, mark_ref, path = line.split(' ', 3)
             mark = int(mark_ref[1:])
-            f = { 'mode' : m, 'data' : blob_marks[mark] }
+            f = { 'mode' : m, 'mark' : mark }
         elif parser.check('D'):
             t, path = line.split(' ')
             f = { 'deleted' : True }
-- 
1.8.2.1.884.g3532a8d

[PATCH 03/20] remote-bzr: fix order of locking in CustomTree

[Felipe Contreras]

It doesn't seem to make any difference, but revision_tree() requires a
lock.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 91b5cda..b428244 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -389,8 +389,8 @@ class CustomTree():
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
-            tree = repo.repository.revision_tree(revid)
             repo.lock_read()
+            tree = repo.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
                     files[path] = entry.file_id
-- 
1.8.2.1.884.g3532a8d

[PATCH 04/20] remote-bzr: always try to update the worktree

[Felipe Contreras]

And fail properly when we can't.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index b428244..eb91d28 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -641,9 +641,13 @@ def do_export(parser):
                 except bzrlib.errors.DivergedBranches:
                     print "error %s non-fast forward" % ref
                     continue
-            else:
+
+            try:
                 wt = repo.bzrdir.open_workingtree()
                 wt.update()
+            except bzrlib.errors.NoWorkingTree:
+                pass
+
         print "ok %s" % ref
 
     print
-- 
1.8.2.1.884.g3532a8d

[PATCH 05/20] remote-bzr: add support to push merges

[Felipe Contreras]

In order to do that, we need to store the marks of every file, so that
they can be fetched when needed. Unfortunately we can't tell bazaar that
nothing changed, we need to send the data so that it can figure it out
by itself.

And since it will be requesting a bunch of information by the file_id,
it's better to have a helper dict (rev_files), so that we can fetch it
quickly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 29 ++++++++++++++++++++--------
 contrib/remote-helpers/test-bzr.sh    | 36 +++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+), 8 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index eb91d28..6a7f836 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -393,7 +393,7 @@ class CustomTree():
             tree = repo.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
-                    files[path] = entry.file_id
+                    files[path] = [entry.file_id, None]
             finally:
                 repo.unlock()
             return files
@@ -408,12 +408,18 @@ class CustomTree():
                 self.base_files = copy_tree(self.base_id)
 
         self.files = files_cache[revid] = self.base_files.copy()
+        self.rev_files = {}
+
+        for path, data in self.files.iteritems():
+            fid, mark = data
+            self.rev_files[fid] = [path, mark]
 
         for path, f in files.iteritems():
-            fid = self.files.get(path, None)
+            fid, mark = self.files.get(path, [None, None])
             if not fid:
                 fid = bzrlib.generate_ids.gen_file_id(path)
             f['path'] = path
+            self.rev_files[fid] = [path, mark]
             self.updates[fid] = f
 
     def last_revision(self):
@@ -423,10 +429,10 @@ class CustomTree():
         changes = []
 
         def get_parent(dirname, basename):
-            parent_fid = self.base_files.get(dirname, None)
+            parent_fid, mark = self.base_files.get(dirname, [None, None])
             if parent_fid:
                 return parent_fid
-            parent_fid = self.files.get(dirname, None)
+            parent_fid, mark = self.files.get(dirname, [None, None])
             if parent_fid:
                 return parent_fid
             if basename == '':
@@ -453,7 +459,7 @@ class CustomTree():
                     (None, basename),
                     (None, kind),
                     (None, executable))
-            self.files[path] = change[0]
+            self.files[path] = [change[0], None]
             changes.append(change)
 
         def update_entry(fid, path, kind, mode = None):
@@ -474,7 +480,7 @@ class CustomTree():
                     (None, basename),
                     (None, kind),
                     (None, executable))
-            self.files[path] = change[0]
+            self.files[path] = [change[0], None]
             changes.append(change)
 
         def remove_entry(fid, path, kind):
@@ -503,16 +509,23 @@ class CustomTree():
             else:
                 add_entry(fid, path, 'file', f['mode'])
 
+            self.files[path][1] = f['mark']
+            self.rev_files[fid][1] = f['mark']
+
         return changes
 
     def get_file_with_stat(self, file_id, path=None):
-        mark = self.updates[file_id]['mark']
+        path, mark = self.rev_files[file_id]
         return (StringIO.StringIO(blob_marks[mark]), None)
 
     def get_symlink_target(self, file_id):
-        mark = self.updates[file_id]['mark']
+        path, mark = self.rev_files[file_id]
         return blob_marks[mark]
 
+    def id2path(self, file_id):
+        path, mark = self.rev_files[file_id]
+        return path
+
 def c_style_unescape(string):
     if string[0] == string[-1] == '"':
         return string.decode('string-escape')[1:-1]
diff --git a/contrib/remote-helpers/test-bzr.sh b/contrib/remote-helpers/test-bzr.sh
index 34666e1..eca3476 100755
--- a/contrib/remote-helpers/test-bzr.sh
+++ b/contrib/remote-helpers/test-bzr.sh
@@ -228,4 +228,40 @@ test_expect_success 'push utf-8 filenames' '
   test_cmp expected actual
 '
 
+test_expect_success 'pushing a merge' '
+  mkdir -p tmp && cd tmp &&
+  test_when_finished "cd .. && rm -rf tmp" &&
+
+  (
+  bzr init bzrrepo &&
+  cd bzrrepo &&
+  echo one > content &&
+  bzr add content &&
+  bzr commit -m one
+  ) &&
+
+  git clone "bzr::$PWD/bzrrepo" gitrepo &&
+
+  (
+  cd bzrrepo &&
+  echo two > content &&
+  bzr commit -m two
+  ) &&
+
+  (
+  cd gitrepo &&
+  echo three > content &&
+  git commit -a -m three &&
+  git fetch &&
+  git merge origin/master || true &&
+  echo three > content &&
+  git commit -a --no-edit &&
+  git push
+  ) &&
+
+  echo three > expected &&
+  cat bzrrepo/content > actual &&
+  test_cmp expected actual
+'
+
 test_done
-- 
1.8.2.1.884.g3532a8d

[PATCH 06/20] remote-bzr: fixes for branch diverge

[Felipe Contreras]

If the branches diverge we want to reset the pointer to where the remote
actually is. Since we can access remote branches just as easily as local
ones, let's do so.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 6a7f836..bf254a0 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -354,6 +354,7 @@ def do_import(parser):
     if os.path.exists(path):
         print "feature import-marks=%s" % path
     print "feature export-marks=%s" % path
+    print "feature force"
     sys.stdout.flush()
 
     while parser.check('import'):
@@ -716,7 +717,12 @@ def get_repo(url, alias):
             # pull
             d = bzrlib.bzrdir.BzrDir.open(clone_path)
             branch = d.open_branch()
-            result = branch.pull(remote_branch, [], None, False)
+            try:
+                result = branch.pull(remote_branch, [], None, False)
+            except bzrlib.errors.DivergedBranches:
+                # use remote branch for now
+                peer = None
+                return remote_branch
         else:
             # clone
             d = origin.sprout(clone_path, None,
-- 
1.8.2.1.884.g3532a8d

[PATCH 07/20] remote-bzr: fix partially pushed merge

[Felipe Contreras]

If part of the merge was already pushed, we don't have the blob_marks
available, however, the commits are already stored in bazaar, so we can
use the revision_tree to fetch the contents.

We want to do this only when there's no other option.

There's no easy way to test this.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index bf254a0..fdead31 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -387,6 +387,7 @@ class CustomTree():
         global files_cache
 
         self.updates = {}
+        self.branch = repo
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
@@ -515,13 +516,21 @@ class CustomTree():
 
         return changes
 
-    def get_file_with_stat(self, file_id, path=None):
+    def get_content(self, file_id):
         path, mark = self.rev_files[file_id]
-        return (StringIO.StringIO(blob_marks[mark]), None)
+        if mark:
+            return blob_marks[mark]
+
+        # last resort
+        tree = self.branch.repository.revision_tree(self.base_id)
+        return tree.get_file_text(file_id)
+
+    def get_file_with_stat(self, file_id, path=None):
+        content = self.get_content(file_id)
+        return (StringIO.StringIO(content), None)
 
     def get_symlink_target(self, file_id):
-        path, mark = self.rev_files[file_id]
-        return blob_marks[mark]
+        return self.get_content(file_id)
 
     def id2path(self, file_id):
         path, mark = self.rev_files[file_id]
-- 
1.8.2.1.884.g3532a8d

[PATCH 08/20] remote-bzr: use branch variable when appropriate

[Felipe Contreras]

There should be no functional changes. Basically we want to reserve the
'repo' variable.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index fdead31..0c67236 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -257,7 +257,8 @@ def export_branch(branch, name):
     tip = marks.get_tip(name)
 
     repo = branch.repository
-    repo.lock_read()
+
+    branch.lock_read()
     revs = branch.iter_merge_sorted_revisions(None, tip, 'exclude', 'forward')
     count = 0
 
@@ -325,7 +326,7 @@ def export_branch(branch, name):
             print "progress revision %s (%d/%d)" % (revid, count, len(revs))
             print "#############################################################"
 
-    repo.unlock()
+    branch.unlock()
 
     revid = branch.last_revision()
 
@@ -383,21 +384,21 @@ def parse_blob(parser):
 
 class CustomTree():
 
-    def __init__(self, repo, revid, parents, files):
+    def __init__(self, branch, revid, parents, files):
         global files_cache
 
         self.updates = {}
-        self.branch = repo
+        self.branch = branch
 
         def copy_tree(revid):
             files = files_cache[revid] = {}
-            repo.lock_read()
-            tree = repo.repository.revision_tree(revid)
+            branch.lock_read()
+            tree = branch.repository.revision_tree(revid)
             try:
                 for path, entry in tree.iter_entries_by_dir():
                     files[path] = [entry.file_id, None]
             finally:
-                repo.unlock()
+                branch.unlock()
             return files
 
         if len(parents) == 0:
@@ -587,20 +588,20 @@ def parse_commit(parser):
         path = c_style_unescape(path).decode('utf-8')
         files[path] = f
 
-    repo = parser.repo
+    branch = parser.repo
 
     committer, date, tz = committer
     parents = [str(mark_to_rev(p)) for p in parents]
     revid = bzrlib.generate_ids.gen_revision_id(committer, date)
     props = {}
-    props['branch-nick'] = repo.nick
+    props['branch-nick'] = branch.nick
 
-    mtree = CustomTree(repo, revid, parents, files)
+    mtree = CustomTree(branch, revid, parents, files)
     changes = mtree.iter_changes()
 
-    repo.lock_write()
+    branch.lock_write()
     try:
-        builder = repo.get_commit_builder(parents, None, date, tz, committer, props, revid)
+        builder = branch.get_commit_builder(parents, None, date, tz, committer, props, revid)
         try:
             list(builder.record_iter_changes(mtree, mtree.last_revision(), changes))
             builder.finish_inventory()
@@ -609,7 +610,7 @@ def parse_commit(parser):
             builder.abort()
             raise
     finally:
-        repo.unlock()
+        branch.unlock()
 
     parsed_refs[ref] = revid
     marks.new_mark(revid, commit_mark)
@@ -653,20 +654,20 @@ def do_export(parser):
         else:
             die('unhandled export command: %s' % line)
 
-    repo = parser.repo
+    branch = parser.repo
 
     for ref, revid in parsed_refs.iteritems():
         if ref == 'refs/heads/master':
-            repo.generate_revision_history(revid, marks.get_tip('master'))
+            branch.generate_revision_history(revid, marks.get_tip('master'))
             if peer:
                 try:
-                    repo.push(peer, stop_revision=revid)
+                    branch.push(peer, stop_revision=revid)
                 except bzrlib.errors.DivergedBranches:
                     print "error %s non-fast forward" % ref
                     continue
 
             try:
-                wt = repo.bzrdir.open_workingtree()
+                wt = branch.bzrdir.open_workingtree()
                 wt.update()
             except bzrlib.errors.NoWorkingTree:
                 pass
-- 
1.8.2.1.884.g3532a8d

[PATCH 09/20] remote-bzr: add support for bzr repos

[Felipe Contreras]

In bazaar, a repository can contain multiple branches, and previously we
were supporting only one branch at a time. Now we fetch them all.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 158 ++++++++++++++++++++++------------
 1 file changed, 102 insertions(+), 56 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 0c67236..3b33061 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -27,6 +27,7 @@ import bzrlib.generate_ids
 import bzrlib.transport
 import bzrlib.errors
 import bzrlib.ui
+import bzrlib.urlutils
 
 import sys
 import os
@@ -250,12 +251,13 @@ def export_files(tree, files):
 
     return final
 
-def export_branch(branch, name):
+def export_branch(repo, name):
     global prefix
 
     ref = '%s/heads/%s' % (prefix, name)
     tip = marks.get_tip(name)
 
+    branch = branches[name]
     repo = branch.repository
 
     branch.lock_read()
@@ -323,7 +325,7 @@ def export_branch(branch, name):
 
         count += 1
         if (count % 100 == 0):
-            print "progress revision %s (%d/%d)" % (revid, count, len(revs))
+            print "progress revision %s '%s' (%d/%d)" % (revid, name, count, len(revs))
             print "#############################################################"
 
     branch.unlock()
@@ -348,7 +350,7 @@ def export_tag(repo, name):
 def do_import(parser):
     global dirname
 
-    branch = parser.repo
+    repo = parser.repo
     path = os.path.join(dirname, 'marks-git')
 
     print "feature done"
@@ -362,10 +364,10 @@ def do_import(parser):
         ref = parser[1]
         if ref.startswith('refs/heads/'):
             name = ref[len('refs/heads/'):]
-            export_branch(branch, name)
+            export_branch(repo, name)
         if ref.startswith('refs/tags/'):
             name = ref[len('refs/tags/'):]
-            export_tag(branch, name)
+            export_tag(repo, name)
         parser.next()
 
     print 'done'
@@ -551,8 +553,11 @@ def parse_commit(parser):
     ref = parser[1]
     parser.next()
 
-    if ref != 'refs/heads/master':
-        die("bzr doesn't support multiple branches; use 'master'")
+    if ref.startswith('refs/heads/'):
+        name = ref[len('refs/heads/'):]
+        branch = branches[name]
+    else:
+        die('unknown ref')
 
     commit_mark = parser.get_mark()
     parser.next()
@@ -588,8 +593,6 @@ def parse_commit(parser):
         path = c_style_unescape(path).decode('utf-8')
         files[path] = f
 
-    branch = parser.repo
-
     committer, date, tz = committer
     parents = [str(mark_to_rev(p)) for p in parents]
     revid = bzrlib.generate_ids.gen_revision_id(committer, date)
@@ -621,9 +624,6 @@ def parse_reset(parser):
     ref = parser[1]
     parser.next()
 
-    if ref != 'refs/heads/master':
-        die("bzr doesn't support multiple branches; use 'master'")
-
     # ugh
     if parser.check('commit'):
         parse_commit(parser)
@@ -636,7 +636,7 @@ def parse_reset(parser):
     parsed_refs[ref] = mark_to_rev(from_mark)
 
 def do_export(parser):
-    global parsed_refs, dirname, peer
+    global parsed_refs, dirname
 
     parser.next()
 
@@ -654,23 +654,24 @@ def do_export(parser):
         else:
             die('unhandled export command: %s' % line)
 
-    branch = parser.repo
-
     for ref, revid in parsed_refs.iteritems():
-        if ref == 'refs/heads/master':
-            branch.generate_revision_history(revid, marks.get_tip('master'))
-            if peer:
-                try:
-                    branch.push(peer, stop_revision=revid)
-                except bzrlib.errors.DivergedBranches:
-                    print "error %s non-fast forward" % ref
-                    continue
+        name = ref[len('refs/heads/'):]
+        branch = branches[name]
+        branch.generate_revision_history(revid, marks.get_tip(name))
 
+        if name in peers:
+            peer = peers[name]
             try:
-                wt = branch.bzrdir.open_workingtree()
-                wt.update()
-            except bzrlib.errors.NoWorkingTree:
-                pass
+                peer.bzrdir.push_branch(branch, revision_id=revid)
+            except bzrlib.errors.DivergedBranches:
+                print "error %s non-fast forward" % ref
+                continue
+
+        try:
+            wt = branch.bzrdir.open_workingtree()
+            wt.update()
+        except bzrlib.errors.NoWorkingTree:
+            pass
 
         print "ok %s" % ref
 
@@ -697,9 +698,15 @@ def ref_is_valid(name):
 
 def do_list(parser):
     global tags
-    print "? refs/heads/%s" % 'master'
 
-    branch = parser.repo
+    master_branch = None
+
+    for name in branches:
+        if not master_branch:
+            master_branch = name
+        print "? refs/heads/%s" % name
+
+    branch = branches[master_branch]
     branch.lock_read()
     for tag, revid in branch.tags.get_tag_dict().items():
         try:
@@ -711,41 +718,77 @@ def do_list(parser):
         print "? refs/tags/%s" % tag
         tags[tag] = revid
     branch.unlock()
-    print "@refs/heads/%s HEAD" % 'master'
+
+    print "@refs/heads/%s HEAD" % master_branch
     print
 
+def get_remote_branch(origin, remote_branch, name):
+    global dirname, peers
+
+    branch_path = os.path.join(dirname, 'clone', name)
+    if os.path.exists(branch_path):
+        # pull
+        d = bzrlib.bzrdir.BzrDir.open(branch_path)
+        branch = d.open_branch()
+        try:
+            branch.pull(remote_branch, [], None, False)
+        except bzrlib.errors.DivergedBranches:
+            # use remote branch for now
+            return remote_branch
+    else:
+        # clone
+        d = origin.sprout(branch_path, None,
+                hardlink=True, create_tree_if_local=False,
+                force_new_repo=False,
+                source_branch=remote_branch)
+        branch = d.open_branch()
+
+    return branch
+
 def get_repo(url, alias):
-    global dirname, peer
+    global dirname, peer, branches
 
+    normal_url = bzrlib.urlutils.normalize_url(url)
     origin = bzrlib.bzrdir.BzrDir.open(url)
-    branch = origin.open_branch()
-
-    if not isinstance(origin.transport, bzrlib.transport.local.LocalTransport):
-        clone_path = os.path.join(dirname, 'clone')
-        remote_branch = branch
-        if os.path.exists(clone_path):
-            # pull
-            d = bzrlib.bzrdir.BzrDir.open(clone_path)
-            branch = d.open_branch()
-            try:
-                result = branch.pull(remote_branch, [], None, False)
-            except bzrlib.errors.DivergedBranches:
-                # use remote branch for now
-                peer = None
-                return remote_branch
+    is_local = isinstance(origin.transport, bzrlib.transport.local.LocalTransport)
+
+    clone_path = os.path.join(dirname, 'clone')
+
+    try:
+        repo = origin.open_repository()
+    except bzrlib.errors.NoRepositoryPresent:
+        # branch
+
+        name = 'master'
+        branch = origin.open_branch()
+
+        if not is_local:
+            if not os.path.exists(clone_path):
+                os.mkdir(clone_path)
+            peers[name] = branch
+            branches[name] = get_remote_branch(origin, branch, name)
         else:
-            # clone
-            d = origin.sprout(clone_path, None,
-                    hardlink=True, create_tree_if_local=False,
-                    source_branch=remote_branch)
-            branch = d.open_branch()
-            branch.bind(remote_branch)
-
-        peer = remote_branch
+            branches[name] = branch
+
+        return branch.repository
     else:
-        peer = None
+        # repository
 
-    return branch
+        if not is_local and not os.path.exists(clone_path):
+            clonedir = bzrlib.bzrdir.BzrDir.create(clone_path)
+
+        for branch in repo.find_branches():
+
+            name = repo.user_transport.relpath(branch.base)
+            name = name if name != '' else 'master'
+
+            if not is_local:
+                peers[name] = branch
+                branches[name] = get_remote_branch(origin, branch, name)
+            else:
+                branches[name] = branch
+
+        return repo
 
 def fix_path(alias, orig_url):
     url = urlparse.urlparse(orig_url, 'file')
@@ -762,6 +805,7 @@ def main(args):
     global parsed_refs
     global files_cache
     global is_tmp
+    global branches, peers
 
     alias = args[1]
     url = args[2]
@@ -772,6 +816,8 @@ def main(args):
     parsed_refs = {}
     files_cache = {}
     marks = None
+    branches = {}
+    peers = {}
 
     if alias[5:] == url:
         is_tmp = True
-- 
1.8.2.1.884.g3532a8d

[PATCH 10/20] remote-bzr: fix branch names

[Felipe Contreras]

When branches have '/' in their name (aka. sub-branches), bazaar seems
to choke while creating the new directory.

Also, git cannot have both 'foo' and 'foo/bar'.

So let's replace slashes with a plus sign.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 3b33061..38cec07 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -781,6 +781,7 @@ def get_repo(url, alias):
 
             name = repo.user_transport.relpath(branch.base)
             name = name if name != '' else 'master'
+            name = name.replace('/', '+')
 
             if not is_local:
                 peers[name] = branch
-- 
1.8.2.1.884.g3532a8d

[PATCH 11/20] remote-bzr: add support for shared repo

[Felipe Contreras]

This way all the remotes share the same data, so adding multiple
remotes, or renaming them doesn't create extra overhead.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 38cec07..9fe830e 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -752,7 +752,20 @@ def get_repo(url, alias):
     origin = bzrlib.bzrdir.BzrDir.open(url)
     is_local = isinstance(origin.transport, bzrlib.transport.local.LocalTransport)
 
-    clone_path = os.path.join(dirname, 'clone')
+    shared_path = os.path.join(gitdir, 'bzr')
+    try:
+        shared_dir = bzrlib.bzrdir.BzrDir.open(shared_path)
+    except bzrlib.errors.NotBranchError:
+        shared_dir = bzrlib.bzrdir.BzrDir.create(shared_path)
+    try:
+        shared_repo = shared_dir.open_repository()
+    except bzrlib.errors.NoRepositoryPresent:
+        shared_repo = shared_dir.create_repository(shared=True)
+
+    if not is_local:
+        clone_path = os.path.join(dirname, 'clone')
+        if not os.path.exists(clone_path):
+            os.mkdir(clone_path)
 
     try:
         repo = origin.open_repository()
@@ -763,8 +776,6 @@ def get_repo(url, alias):
         branch = origin.open_branch()
 
         if not is_local:
-            if not os.path.exists(clone_path):
-                os.mkdir(clone_path)
             peers[name] = branch
             branches[name] = get_remote_branch(origin, branch, name)
         else:
@@ -774,9 +785,6 @@ def get_repo(url, alias):
     else:
         # repository
 
-        if not is_local and not os.path.exists(clone_path):
-            clonedir = bzrlib.bzrdir.BzrDir.create(clone_path)
-
         for branch in repo.find_branches():
 
             name = repo.user_transport.relpath(branch.base)
@@ -800,7 +808,7 @@ def fix_path(alias, orig_url):
     subprocess.call(cmd)
 
 def main(args):
-    global marks, prefix, dirname
+    global marks, prefix, gitdir, dirname
     global tags, filenodes
     global blob_marks
     global parsed_refs
-- 
1.8.2.1.884.g3532a8d

[PATCH 12/20] remote-bzr: split marks file

[Felipe Contreras]

This way all the remotes can share the same git objects, and the same
marks. The information we want to store per remote is very small.

The code transparently converts from one organization of marks, to the
other. It's rather smooth and there shouldn't be any issues.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 58 ++++++++++++++++++++++++++---------
 1 file changed, 43 insertions(+), 15 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 9fe830e..fcd8e41 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -52,22 +52,42 @@ def gittz(tz):
 
 class Marks:
 
-    def __init__(self, path):
-        self.path = path
+    def __init__(self, gitdir, dirname):
+        self.marks_path = os.path.join(gitdir, 'bzr', '.marks-bzr')
+        self.remote_path = os.path.join(dirname, '.marks-bzr')
+        old_path = os.path.join(dirname, 'marks-int')
+
         self.tips = {}
         self.marks = {}
         self.rev_marks = {}
         self.last_mark = 0
-        self.load()
+
+        if not os.path.exists(old_path):
+            self.load()
+        else:
+            self.old_load(old_path)
+            os.remove(old_path)
 
     def load(self):
-        if not os.path.exists(self.path):
-            return
 
-        tmp = json.load(open(self.path))
+        if os.path.exists(self.marks_path):
+            tmp = json.load(open(self.marks_path))
+            self.marks = tmp['marks']
+            self.last_mark = tmp['last-mark']
+
+            for rev, mark in self.marks.iteritems():
+                self.rev_marks[mark] = rev
+
+        if os.path.exists(self.remote_path):
+            tmp = json.load(open(self.remote_path))
+            self.tips = tmp['tips']
+
+    def old_load(self, path):
+
+        tmp = json.load(open(path))
         self.tips = tmp['tips']
-        self.marks = tmp['marks']
-        self.last_mark = tmp['last-mark']
+        self.marks = tmp['marks'] # TODO remove
+        self.last_mark = tmp['last-mark'] # TODO remove
 
         for rev, mark in self.marks.iteritems():
             self.rev_marks[mark] = rev
@@ -76,7 +96,10 @@ class Marks:
         return { 'tips': self.tips, 'marks': self.marks, 'last-mark' : self.last_mark }
 
     def store(self):
-        json.dump(self.dict(), open(self.path, 'w'))
+        d = { 'marks': self.marks, 'last-mark' : self.last_mark }
+        json.dump(d, open(self.marks_path, 'w'))
+        d = { 'tips': self.tips }
+        json.dump(d, open(self.remote_path, 'w'))
 
     def __str__(self):
         return str(self.dict())
@@ -348,10 +371,10 @@ def export_tag(repo, name):
     print
 
 def do_import(parser):
-    global dirname
+    global gitdir
 
     repo = parser.repo
-    path = os.path.join(dirname, 'marks-git')
+    path = os.path.join(gitdir, 'bzr', '.marks-git')
 
     print "feature done"
     if os.path.exists(path):
@@ -678,14 +701,14 @@ def do_export(parser):
     print
 
 def do_capabilities(parser):
-    global dirname
+    global gitdir
 
     print "import"
     print "export"
     print "refspec refs/heads/*:%s/heads/*" % prefix
     print "refspec refs/tags/*:%s/tags/*" % prefix
 
-    path = os.path.join(dirname, 'marks-git')
+    path = os.path.join(gitdir, 'bzr', '.marks-git')
 
     if os.path.exists(path):
         print "*import-marks %s" % path
@@ -848,8 +871,13 @@ def main(args):
 
     repo = get_repo(url, alias)
 
-    marks_path = os.path.join(dirname, 'marks-int')
-    marks = Marks(marks_path)
+    marks = Marks(gitdir, dirname)
+
+    # move old marks
+    old_gitmarks = os.path.join(dirname, 'marks-git')
+    new_gitmarks = os.path.join(gitdir, 'bzr', '.marks-git')
+    if os.path.exists(old_gitmarks):
+        os.rename(old_gitmarks, new_gitmarks)
 
     parser = Parser(repo)
     for line in parser:
-- 
1.8.2.1.884.g3532a8d

[PATCH 13/20] remote-bzr: improve author sanitazion

[Felipe Contreras]

So that we don't end up with '<None>', and also synchronize it with the
one from remote-hg.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index fcd8e41..5c25b06 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -38,6 +38,7 @@ import atexit, shutil, hashlib, urlparse, subprocess
 
 NAME_RE = re.compile('^([^<>]+)')
 AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$')
+EMAIL_RE = re.compile('^([^<>]+[^ \\\t<>])?\\b(?:[ \\t<>]*?)\\b([^ \\t<>]+@[^ \\t<>]+)')
 RAW_AUTHOR_RE = re.compile('^(\w+) (.+)? <(.*)> (\d+) ([+-]\d+)')
 
 def die(msg, *args):
@@ -198,9 +199,19 @@ def fixup_user(user):
         name = m.group(1)
         mail = m.group(2).strip()
     else:
-        m = NAME_RE.match(user)
+        m = EMAIL_RE.match(user)
         if m:
-            name = m.group(1).strip()
+            name = m.group(1)
+            mail = m.group(2)
+        else:
+            m = NAME_RE.match(user)
+            if m:
+                name = m.group(1).strip()
+
+    if not name:
+        name = 'unknown'
+    if not mail:
+        mail = 'Unknown'
 
     return '%s <%s>' % (name, mail)
 
-- 
1.8.2.1.884.g3532a8d

[PATCH 14/20] remote-bzr: add custom method to find branches

[Felipe Contreras]

The official method is incredibly inefficient and slow.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 5c25b06..5155b93 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -779,6 +779,25 @@ def get_remote_branch(origin, remote_branch, name):
 
     return branch
 
+def find_branches(repo):
+    transport = repo.user_transport
+
+    for fn in transport.iter_files_recursive():
+        if not fn.endswith('.bzr/branch-format'):
+            continue
+
+        name = subdir = fn[:-len('/.bzr/branch-format')]
+        name = name if name != '' else 'master'
+        name = name.replace('/', '+')
+
+        try:
+            cur = transport.clone(subdir)
+            branch = bzrlib.branch.Branch.open_from_transport(cur)
+        except bzrlib.errors.NotBranchError:
+            continue
+        else:
+            yield name, branch
+
 def get_repo(url, alias):
     global dirname, peer, branches
 
@@ -819,11 +838,7 @@ def get_repo(url, alias):
     else:
         # repository
 
-        for branch in repo.find_branches():
-
-            name = repo.user_transport.relpath(branch.base)
-            name = name if name != '' else 'master'
-            name = name.replace('/', '+')
+        for name, branch in find_branches(repo):
 
             if not is_local:
                 peers[name] = branch
-- 
1.8.2.1.884.g3532a8d

[PATCH 15/20] remote-bzr: add option to specify branches

[Felipe Contreras]

We might not want all the branches. And branch handling in bazaar is
rather tricky, so it's safer to simply specify them.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 5155b93..06d9d24 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -13,6 +13,9 @@
 # or
 # % git clone bzr::lp:myrepo
 #
+# If you want to specify which branches you want track (per repo):
+# git config remote-bzr.branches 'trunk, devel, test'
+#
 
 import sys
 
@@ -51,6 +54,12 @@ def warn(msg, *args):
 def gittz(tz):
     return '%+03d%02d' % (tz / 3600, tz % 3600 / 60)
 
+def get_config(config):
+    cmd = ['git', 'config', '--get', config]
+    process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+    output, _ = process.communicate()
+    return output
+
 class Marks:
 
     def __init__(self, gitdir, dirname):
@@ -779,7 +788,7 @@ def get_remote_branch(origin, remote_branch, name):
 
     return branch
 
-def find_branches(repo):
+def find_branches(repo, wanted):
     transport = repo.user_transport
 
     for fn in transport.iter_files_recursive():
@@ -790,6 +799,9 @@ def find_branches(repo):
         name = name if name != '' else 'master'
         name = name.replace('/', '+')
 
+        if wanted and not name in wanted:
+            continue
+
         try:
             cur = transport.clone(subdir)
             branch = bzrlib.branch.Branch.open_from_transport(cur)
@@ -838,7 +850,11 @@ def get_repo(url, alias):
     else:
         # repository
 
-        for name, branch in find_branches(repo):
+        wanted = get_config('remote-bzr.branches').rstrip().split(', ')
+        # stupid python
+        wanted = [e for e in wanted if e]
+
+        for name, branch in find_branches(repo, wanted):
 
             if not is_local:
                 peers[name] = branch
-- 
1.8.2.1.884.g3532a8d

[PATCH 16/20] remote-bzr: improve progress reporting

[Felipe Contreras]

No need to manually count the revisions, and also, this would help to
iterate more properly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 06d9d24..7d3052a 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -305,13 +305,16 @@ def export_branch(repo, name):
 
     branch.lock_read()
     revs = branch.iter_merge_sorted_revisions(None, tip, 'exclude', 'forward')
-    count = 0
+    tip_revno = branch.revision_id_to_revno(tip)
+    last_revno, _ = branch.last_revision_info()
+    total = last_revno - tip_revno
 
-    revs = [revid for revid, _, _, _ in revs if not marks.is_marked(revid)]
+    revs = [revid, seq for revid, _, seq, _ in revs if not marks.is_marked(revid)]
 
-    for revid in revs:
+    for revid, seq in revs:
 
         rev = repo.get_revision(revid)
+        revno = seq[0]
 
         parents = rev.parent_ids
         time = rev.timestamp
@@ -366,10 +369,13 @@ def export_branch(repo, name):
             print "M %s :%u %s" % f
         print
 
-        count += 1
-        if (count % 100 == 0):
-            print "progress revision %s '%s' (%d/%d)" % (revid, name, count, len(revs))
-            print "#############################################################"
+        if len(seq) > 1:
+            # let's skip branch revisions from the progress report
+            continue
+
+        progress = (revno - tip_revno)
+        if (progress % 100 == 0):
+            print "progress revision %d '%s' (%d/%d)" % (revno, name, progress, total)
 
     branch.unlock()
 
-- 
1.8.2.1.884.g3532a8d

[PATCH 17/20] remote-bzr: iterate revisions properly

[Felipe Contreras]

This way we don't need to store the list of all the revisions, which
doesn't seem to be very memory efficient with bazaar's design, for
whatever reason.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 7d3052a..5f1b9c0 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -309,9 +309,10 @@ def export_branch(repo, name):
     last_revno, _ = branch.last_revision_info()
     total = last_revno - tip_revno
 
-    revs = [revid, seq for revid, _, seq, _ in revs if not marks.is_marked(revid)]
+    for revid, _, seq, _ in revs:
 
-    for revid, seq in revs:
+        if marks.is_marked(revid):
+            continue
 
         rev = repo.get_revision(revid)
         revno = seq[0]
-- 
1.8.2.1.884.g3532a8d

[PATCH 18/20] remote-bzr: delay peer branch usage

[Felipe Contreras]

So it doesn't time out.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 5f1b9c0..7517645 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -710,7 +710,7 @@ def do_export(parser):
         branch.generate_revision_history(revid, marks.get_tip(name))
 
         if name in peers:
-            peer = peers[name]
+            peer = bzrlib.branch.Branch.open(peers[name])
             try:
                 peer.bzrdir.push_branch(branch, revision_id=revid)
             except bzrlib.errors.DivergedBranches:
@@ -848,7 +848,7 @@ def get_repo(url, alias):
         branch = origin.open_branch()
 
         if not is_local:
-            peers[name] = branch
+            peers[name] = branch.base
             branches[name] = get_remote_branch(origin, branch, name)
         else:
             branches[name] = branch
@@ -864,7 +864,7 @@ def get_repo(url, alias):
         for name, branch in find_branches(repo, wanted):
 
             if not is_local:
-                peers[name] = branch
+                peers[name] = branch.base
                 branches[name] = get_remote_branch(origin, branch, name)
             else:
                 branches[name] = branch
-- 
1.8.2.1.884.g3532a8d

[PATCH 19/20] remote-bzr: access branches only when needed

[Felipe Contreras]

Bazaar doesn't seem to be tested for multiple usage of branches, so
resources seem to be leaked all over. Let's try to minimize this by
accessing the Branch objects only when needed.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 7517645..3c59a18 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -300,7 +300,7 @@ def export_branch(repo, name):
     ref = '%s/heads/%s' % (prefix, name)
     tip = marks.get_tip(name)
 
-    branch = branches[name]
+    branch = bzrlib.branch.Branch.open(branches[name])
     repo = branch.repository
 
     branch.lock_read()
@@ -605,7 +605,7 @@ def parse_commit(parser):
 
     if ref.startswith('refs/heads/'):
         name = ref[len('refs/heads/'):]
-        branch = branches[name]
+        branch = bzrlib.branch.Branch.open(branches[name])
     else:
         die('unknown ref')
 
@@ -706,7 +706,7 @@ def do_export(parser):
 
     for ref, revid in parsed_refs.iteritems():
         name = ref[len('refs/heads/'):]
-        branch = branches[name]
+        branch = bzrlib.branch.Branch.open(branches[name])
         branch.generate_revision_history(revid, marks.get_tip(name))
 
         if name in peers:
@@ -756,7 +756,7 @@ def do_list(parser):
             master_branch = name
         print "? refs/heads/%s" % name
 
-    branch = branches[master_branch]
+    branch = bzrlib.branch.Branch.open(branches[master_branch])
     branch.lock_read()
     for tag, revid in branch.tags.get_tag_dict().items():
         try:
@@ -845,13 +845,15 @@ def get_repo(url, alias):
         # branch
 
         name = 'master'
-        branch = origin.open_branch()
+        remote_branch = origin.open_branch()
 
         if not is_local:
-            peers[name] = branch.base
-            branches[name] = get_remote_branch(origin, branch, name)
+            peers[name] = remote_branch.base
+            branch = get_remote_branch(origin, remote_branch, name)
         else:
-            branches[name] = branch
+            branch = remote_branch
+
+        branches[name] = branch.base
 
         return branch.repository
     else:
@@ -861,13 +863,15 @@ def get_repo(url, alias):
         # stupid python
         wanted = [e for e in wanted if e]
 
-        for name, branch in find_branches(repo, wanted):
+        for name, remote_branch in find_branches(repo, wanted):
 
             if not is_local:
-                peers[name] = branch.base
-                branches[name] = get_remote_branch(origin, branch, name)
+                peers[name] = remote_branch.base
+                branch = get_remote_branch(origin, remote_branch, name)
             else:
-                branches[name] = branch
+                branch = remote_branch
+
+            branches[name] = branch.base
 
         return repo
 
-- 
1.8.2.1.884.g3532a8d

[PATCH 20/20] remote-bzr: export HEAD

[Felipe Contreras]

It probably doesn't make any difference, since we usually fetch along
the 'master' branch, and when we don't probably the current
transport-helper code doesn't work correctly.

But if it did work correctly, this might be useful.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index 3c59a18..41c26a7 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -412,6 +412,8 @@ def do_import(parser):
 
     while parser.check('import'):
         ref = parser[1]
+        if ref == 'HEAD':
+            export_branch(repo, 'master')
         if ref.startswith('refs/heads/'):
             name = ref[len('refs/heads/'):]
             export_branch(repo, name)
-- 
1.8.2.1.884.g3532a8d

Re: [PATCH 00/20] remote-bzr: massive changes

[Felipe Contreras]

On Thu, Apr 25, 2013 at 8:07 PM, Felipe Contreras
<felipe.contreras@gmail.com> wrote:

I forgot to mention; these apply on top of the previous 'fixes and cleanups'.

> Felipe Contreras (20):
>   remote-bzr: cleanup CustomTree
>   remote-bzr: delay blob fetching until the very end
>   remote-bzr: fix order of locking in CustomTree
>   remote-bzr: always try to update the worktree
>   remote-bzr: add support to push merges
>   remote-bzr: fixes for branch diverge
>   remote-bzr: fix partially pushed merge
>   remote-bzr: use branch variable when appropriate
>   remote-bzr: add support for bzr repos
>   remote-bzr: fix branch names
>   remote-bzr: add support for shared repo
>   remote-bzr: split marks file
>   remote-bzr: improve author sanitazion
>   remote-bzr: add custom method to find branches
>   remote-bzr: add option to specify branches
>   remote-bzr: improve progress reporting
>   remote-bzr: iterate revisions properly
>   remote-bzr: delay peer branch usage
>   remote-bzr: access branches only when needed
>   remote-bzr: export HEAD
>
>  contrib/remote-helpers/git-remote-bzr | 365 +++++++++++++++++++++++++---------
>  contrib/remote-helpers/test-bzr.sh    |  36 ++++
>  2 files changed, 302 insertions(+), 99 deletions(-)

-- 
Felipe Contreras

Re: [PATCH 00/20] remote-bzr: massive changes

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> On Thu, Apr 25, 2013 at 8:07 PM, Felipe Contreras
> <felipe.contreras@gmail.com> wrote:
>
> I forgot to mention; these apply on top of the previous 'fixes and cleanups'.

Then I should probably wait for v2 6/9 of that series is rerolled.

There could be ones that you may want to reroll other than 6/9 in
that, I didn't have time to look at the backlog. I'll be whipping
the -rc0 into shape today and it is unlikely I'll have time to look
at this or the previous one today.

Thanks.

Re: [PATCH 00/20] remote-bzr: massive changes

[Felipe Contreras]

On Fri, Apr 26, 2013 at 11:51 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> On Thu, Apr 25, 2013 at 8:07 PM, Felipe Contreras
>> <felipe.contreras@gmail.com> wrote:
>>
>> I forgot to mention; these apply on top of the previous 'fixes and cleanups'.
>
> Then I should probably wait for v2 6/9 of that series is rerolled.

Done.

> There could be ones that you may want to reroll other than 6/9 in
> that, I didn't have time to look at the backlog. I'll be whipping
> the -rc0 into shape today and it is unlikely I'll have time to look
> at this or the previous one today.

I don't see any need to reroll more. All the concerns of v1 and v2
have been addressed in one way or the other.

-- 
Felipe Contreras

Re: [PATCH 12/20] remote-bzr: split marks file

[Felipe Contreras]

On Thu, Apr 25, 2013 at 8:08 PM, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
> This way all the remotes can share the same git objects, and the same
> marks. The information we want to store per remote is very small.
>
> The code transparently converts from one organization of marks, to the
> other. It's rather smooth and there shouldn't be any issues.

Please drop this patch. While testing essentially the same
functionality in remote-hg I noticed that it doesn't work when dealing
with more than one remote. It's not clear if the marks can be shared
at all, and if possible, it would be very tricky.

Better drop it for now. I've tested that dropping this patch doesn't
cause an conflicts for the rest of the series.

And in case anybody is thinking that remote-bzr is really a too fast
moving target; even if this managed to land in 'master', it's likely
that people were not able to push at all, and in fact, many were not
even able to clone in 1.8.2. So, hardly could be considered a
regression. Nevertheless, I caught it in time.

Cheers.

-- 
Felipe Contreras

Re: [PATCH 12/20] remote-bzr: split marks file

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> And in case anybody is thinking that remote-bzr is really a too fast
> moving target; even if this managed to land in 'master', it's likely
> that people were not able to push at all, and in fact, many were not
> even able to clone in 1.8.2. So, hardly could be considered a
> regression. Nevertheless, I caught it in time.

You didn't.  I am already way too deep into today's 1.8.3-rc0
integration cycle and I won't waste a couple of hours work just to
revert this.

But from your description it sounds like with or without the patch
the helper is equally broken and it does not make that much of a
difference.

Re: [PATCH 12/20] remote-bzr: split marks file

[Felipe Contreras]

On Fri, Apr 26, 2013 at 7:17 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> And in case anybody is thinking that remote-bzr is really a too fast
>> moving target; even if this managed to land in 'master', it's likely
>> that people were not able to push at all, and in fact, many were not
>> even able to clone in 1.8.2. So, hardly could be considered a
>> regression. Nevertheless, I caught it in time.
>
> You didn't.  I am already way too deep into today's 1.8.3-rc0
> integration cycle and I won't waste a couple of hours work just to
> revert this.

All right, if you already merged this patch series, you could do what
I did in my public branch: revert it, it's a single command, doesn't
take hours.

But I don't see this series in origin/master, am I missing something?

> But from your description it sounds like with or without the patch
> the helper is equally broken and it does not make that much of a
> difference.

Indeed. Many people would need this patch series to push, and this
patch breaks the push for more than one remote. So if you merged the
whole thing quite likely their situation would improve; they could at
least push to one remote, instead of zero. Of course, if you remove
this patch, they would be able to push to many.

But I don't see it merged.

Cheers.

-- 
Felipe Contreras

Re: [PATCH 12/20] remote-bzr: split marks file

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> And in case anybody is thinking that remote-bzr is really a too fast
>> moving target; even if this managed to land in 'master', it's likely
>> that people were not able to push at all, and in fact, many were not
>> even able to clone in 1.8.2. So, hardly could be considered a
>> regression. Nevertheless, I caught it in time.
>
> You didn't.  I am already way too deep into today's 1.8.3-rc0
> integration cycle and I won't waste a couple of hours work just to
> revert this.

Oh, I was lucky ;-) I mistook this with the other 9-patch bzr
clean-up series that I applied to 'master' for -rc0.

Pushing out a tagged-tip takes a lot longer than the normal tip
because a lot more than what people see have to happen on my end.

Reverting a single patch is simple, but we do not want to do that on
top of "Git 1.8.3-rc0" commit and move the unpublished tag to point
at the revert.

Which means pretty much everything needs to be redone (one example
among many is that the tagname will propagate to the htmldocs and
manpages repositories, so their unpublished histories need to be
rewound).

But I didn't have to do that in the end ;-)

Re: [PATCH 12/20] remote-bzr: split marks file

[Felipe Contreras]

On Sun, Apr 28, 2013 at 1:47 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Junio C Hamano <gitster@pobox.com> writes:
>
>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>
>>> And in case anybody is thinking that remote-bzr is really a too fast
>>> moving target; even if this managed to land in 'master', it's likely
>>> that people were not able to push at all, and in fact, many were not
>>> even able to clone in 1.8.2. So, hardly could be considered a
>>> regression. Nevertheless, I caught it in time.
>>
>> You didn't.  I am already way too deep into today's 1.8.3-rc0
>> integration cycle and I won't waste a couple of hours work just to
>> revert this.
>
> Oh, I was lucky ;-) I mistook this with the other 9-patch bzr
> clean-up series that I applied to 'master' for -rc0.
>
> Pushing out a tagged-tip takes a lot longer than the normal tip
> because a lot more than what people see have to happen on my end.
>
> Reverting a single patch is simple, but we do not want to do that on
> top of "Git 1.8.3-rc0" commit and move the unpublished tag to point
> at the revert.
>
> Which means pretty much everything needs to be redone (one example
> among many is that the tagname will propagate to the htmldocs and
> manpages repositories, so their unpublished histories need to be
> rewound).
>
> But I didn't have to do that in the end ;-)

Yeah, I realized you were talking about that one later on. I haven't
heard anything bad from this new branch from emacs developers, so I
think it's OK to merge it.

Cheers.

-- 
Felipe Contreras