publishing a forked^W cloned directory with ancestry

publishing a forked^W cloned directory with ancestry

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 4058 bytes --]

Dear list,

I am the mdadm (software RAID management) maintainer for Debian and
a recent git convert. While preparing a new mdadm package tonight,
I found myself screaming too often at SVN [0] and decided to convert
Debian mdadm package maintenance to git. Upstream also uses git, so
I said to myself "jolly good", clapped hands, and… froze as the
pieces weren't fitting together nicely, at least from my
understanding of how git works.

There are multiple ways to maintain a Debian package and I shall
today only concentrate on tracking an upstream VCS repo and
packaging it for Debian "when I please" (others call that
snapshotting). Thus, I don't wait for upstream to publish tarballs,
I make the package from a given HEAD. There are pros and cons to
this, but let's just assume that it's "the way".

So I clone upstream and find that git-branch -r includes
upstream/master (s/origin/upstream/ for clarity). I then branch
'debian' off upstream/master and make some required changes. With
utter enjoyment of git, I wrap it up and package a new mdadm.deb.
Yay.

And then I wonder: how do I now publish this result of my work? I'd
like to push my repository to git.debian.org so that others can
clone it and help or submit patches against the debianised upstream.

But the remote branch upstream/master only really exists in
$GIT_DIR, which is local and can't be pushed. Or well, even if
I tried, the people cloning from the push location wouldn't see it
[1].

I saw two solutions to this: using two separate upstreams/origins,
and submodules:

1. I could tell my $GIT_DIR/config that upstream/* comes from mdadm
upstream and debian/* comes from git.debian.org and then merge
happily across branches locally and be done with it. However, John
Doe, who on a rainy Saturday afternoon has two hours to spend and
wants to fix some mdadm bugs would have to jump through hoops to
replicate the setup: all the ties between upstream and the
git.debian.org repo are local to my machine and can't be pushed
anywhere (except to verbose documentation).

2. Let's assume for a moment that all Debian changes go to ./debian,
then submodules seem to want to save the day. Except that Debian
changes are not confined to ./debian, and from the perspective of
the Debian mdadm maintainer and with the semantics of git-submodules
in the back of my head [2], I'd rather want it the other way around:
upstream be a submodule of my Debian-specific repo; but upstream
needs to live in . and TTBOMK, submodules can't live in .

So neither of these work and thus I am turning to you. I want to
publish my Debian "fork" of mdadm in such a way that people can
easily clone it, without pulling all the components together. How
would you do it?

I guess the cleanest solution I can come up with is to branch off
upstream/master into branch "upstream" whenever *I* decide it's time
to snapshot. Then, people using my repo would basically be confined
to the state of the tree as it was the last time I rebased
"upstream", but could work freely on the Debian-specific stuff.
I think this is actually quite okay, but I am still interested in
any comments you may have.

Cheers,
m


Footnotes:

[0] fine "too often" tonight meant "once" but it's once too many.
    Thanks Linus and Junio, since I touched git I can't work most
    other VCS anymore (this is sarcastic... not).
[1] I tried cloning B from A, then cloning C from B. Within C, there
    is no reference to A's master branch, so unless B pulled changes
    from A and C pulled changes from B, C could not be updated.
[2] The submodule's commit ID is stored in the supermodules index
    and a commit to the submodule also requires a commit to the
    supermodule to restore consistency.

-- 
"if builders built buildings the way
 programmers wrote programs,
 then the first woodpecker that came along
 would destroy civilization."
                                                  -- gerald weinberg
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: publishing a forked^W cloned directory with ancestry

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 2130 bytes --]

also sprach martin f krafft <madduck@madduck.net> [2007.08.30.2125 +0200]:
> And then I wonder: how do I now publish this result of my work? I'd
> like to push my repository to git.debian.org so that others can
> clone it and help or submit patches against the debianised upstream.

With this I mean: it would just be nice if people cloning the
git.debian.org repo could also use the upstream refs. I am aware
that the pack they download is complete in the sense that they can
just build after cloning (thanks to Harri Ilari Tapio Liusvaara
though for clearing this up for me a bit). After all, this is what
I meant when I wrote:

> I guess the cleanest solution I can come up with is to branch off
> upstream/master into branch "upstream" whenever *I* decide it's time
> to snapshot. Then, people using my repo would basically be confined
> to the state of the tree as it was the last time I rebased
> "upstream", but could work freely on the Debian-specific stuff.

But I guess in the Debian world, a bug may be fixed upstream before
it is fixed in Debian (not only because Debian is allegedly
outdated, also because our users and developers often cooperate
directly with upstream), and then when someone jumps in for me to
release a new mdadm package, they might just need to rebase to the
latest upstream HEAD.

Is it possible to enable this with git without asking those people
to first set up their local repo to reference git.debian.org *and*
upstream, all of which is likely going to be more than four
commands?

Harri hinted at using Makefiles, and sure, I can use the Makefile to
set up upstream if it's not already present, but I'd much rather
have a standard way that's going to be the same across all
git-maintained Debian packages.

Comments welcome,

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
"no woman should ever be quite accurate about her age.
 it looks so calculating."
                                                        -- oscar wilde
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: publishing a forked^W cloned directory with ancestry

[J. Bruce Fields]

On Thu, Aug 30, 2007 at 09:25:33PM +0200, martin f krafft wrote:
> So I clone upstream and find that git-branch -r includes
> upstream/master (s/origin/upstream/ for clarity). I then branch
> 'debian' off upstream/master and make some required changes. With
> utter enjoyment of git, I wrap it up and package a new mdadm.deb.
> Yay.
> 
> And then I wonder: how do I now publish this result of my work? I'd
> like to push my repository to git.debian.org so that others can
> clone it and help or submit patches against the debianised upstream.

So in the setup you describe if they clone your repo then they'll get a
single branch called 'debian' with your work in it.  That sounds fine to
me, actually.

> But the remote branch upstream/master only really exists in
> $GIT_DIR, which is local and can't be pushed. Or well, even if
> I tried, the people cloning from the push location wouldn't see it

They can always just fetch from upstream as well if they'd like.  They
could do something like:

	git clone git://coolproject.org/cool.git
	cd cool
	git remote add debian git://git.debian.org/cool.git
	git fetch debian

Then they have a repository where git-branch -r reports something like

	origin/master
	debian/debian

Or they could do it the other way around, with "origin" pointing to you
and an "upstream" remote pointing to coolproject.org.  The naming's
obviously up to them.

> 1. I could tell my $GIT_DIR/config that upstream/* comes from mdadm
> upstream and debian/* comes from git.debian.org and then merge
> happily across branches locally and be done with it. However, John
> Doe, who on a rainy Saturday afternoon has two hours to spend and
> wants to fix some mdadm bugs would have to jump through hoops to
> replicate the setup: all the ties between upstream and the
> git.debian.org repo are local to my machine and can't be pushed
> anywhere (except to verbose documentation).

Maybe the one extra "git remote add ...; git remote fetch" isn't such a
big deal?

> I guess the cleanest solution I can come up with is to branch off
> upstream/master into branch "upstream" whenever *I* decide it's time
> to snapshot. Then, people using my repo would basically be confined
> to the state of the tree as it was the last time I rebased
> "upstream", but could work freely on the Debian-specific stuff.
> I think this is actually quite okay, but I am still interested in
> any comments you may have.

Sure, you can do that.  I don't think it's really necessary.

My local kernel repository, for example, currently knows about five
other repos:

	$ git remote
	labiaga		# server pnfs work
	linux-nfs	# my public repo
	origin		# Linus's repo
	richterd	# a coworker's nfs work
	trond		# Trond's nfs stuff

Sure, each of those could add a "linus" branch that tracked upstream, so
I could still get some idea what Linus's tree was even if I didn't
happen to already have it.  But then I'd end up with 4 different
slightly-out-of-date pointers to the head of linus's repo in each of
those trees, which would end up being just be a bunch of cruft that I'd
have to ignore whenever I looked at them.

--b.

Re: publishing a forked^W cloned directory with ancestry

[Bart Trojanowski]

* martin f krafft <madduck@madduck.net> [070830 15:25]:
> [1] I tried cloning B from A, then cloning C from B. Within C, there
>     is no reference to A's master branch, so unless B pulled changes
>     from A and C pulled changes from B, C could not be updated.

I think a .git/config like this will do what you want:

[remote "upstream"]
        url = git://mdadm-upstream-repo
        fetch = +refs/heads/*:refs/remotes/upstream/*

[remote "debian"]
        url = git://debian-repo-you-want-to-publish-to
        fetch = +refs/heads/*:refs/remotes/debian/*
        push = refs/remotes/upstream/master:refs/heads/upstream
        push = refs/heads/master:refs/heads/master

Now when you 'git push debian' it will populate the 'upstream' and
'master' branches properly.

When someone clones your repo, they will get origin/master (your branch)
and origin/upstream (the official mdadm branch).

Did I understand the problem correctly?

-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/

Re: publishing a forked^W cloned directory with ancestry

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 2508 bytes --]

also sprach J. Bruce Fields <bfields@fieldses.org> [2007.08.30.2149 +0200]:
> Maybe the one extra "git remote add ...; git remote fetch" isn't
> such a big deal?

[...]

> Sure, each of those could add a "linus" branch that tracked
> upstream, so I could still get some idea what Linus's tree was
> even if I didn't happen to already have it.  But then I'd end up
> with 4 different slightly-out-of-date pointers to the head of
> linus's repo in each of those trees, which would end up being just
> be a bunch of cruft that I'd have to ignore whenever I looked at
> them.

Thanks to you and everyone else who replied! I spent some time off
computers and still found myself thinking about this as I was hiking
around woods and mountains and in the end, it's perfectly obvious:

I am trying to make it easy for third parties to contribute to my
git-managed project and I (somewhat rightfully) assumed that
I needed to shield my poor contributors from git's complex
tentacles. In doing so, I find myself going backwards and turning
git into something svn-like, thus losing much of its power.

The solution is clear: maintain only my Debian branch on
git.debian.org and expect those cloning it to add the upstream
remote themselves in true distributed manners. A concise file in the
project root with instructions is a nice add-on.



also sprach Bart Trojanowski <bart@jukie.net> [2007.08.30.2227 +0200]:
> Now when you 'git push debian' it will populate the 'upstream' and
> 'master' branches properly.
> 
> When someone clones your repo, they will get origin/master (your
> branch) and origin/upstream (the official mdadm branch).

This is the solution I had stuck in my head for the longest time
(after you proposed it), but I eventually discard it for one simple
reason: the danger that someone pushes to the upstream branch and
thus diverges it from the real upstream is just too high and would
result in a pretty nasty mess, as far as I can tell.

> Did I understand the problem correctly?

I think you did. And thanks for that. I hope my reasoning above also
makes sense.

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
"the surest way to corrupt a youth is to instruct him to hold in
 higher esteem those who think alike than those who think
 differently."
                                              -- friedrich nietzsche
 
spamtraps: madduck.bogus@madduck.net

[-- Attachment #2: Digital signature (see http://martin-krafft.net/gpg/) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]