[RFC] Stgit - patch history / add extra parents

[RFC] Stgit - patch history / add extra parents

[Jan Veldeman]

Hi,

I like stgit very much, but I feel there is still something missing:
stgit is very handy when you use it for patches which should be pushed to
mainline rather quickly. But for pacthes which won't be pushed immediately
to mainline, it would be usefull to have a history of the patches itself.

The patch below, together with the following script could be used to
make snapshots of the patch stack (I call it freeze, as I thought snapshot
was already going to be used for something else):

#!/bin/bash
# stg-freeze
set -ex
top=$(stg top)

# Freeze the current stack
branchname=$(basename $(readlink .git/HEAD))
while read patch
do
	cp .git/patches/$branchname/$patch/top .git/patches/$branchname/$patch/parent
done < .git/patches/$branchname/applied

# Refresh the patches
stg pop -a
stg push -t $top
# end stg-freeze

Note: this is a proof of concept, when/if it would be incorporated, it
should be implemented in stgit itself, and a bit more efficient, especially
the refreshing of the patches ;-)

This is how it works: when not doing anything, stgit works as normal.
Once stg-freeze is called, it creates .git/patches/$branchname/$patch/parent
which contains the SHA1 id of the "frozen" patch. By refreshing the stack,
all patches now include the corresponding frozen patch as a parent.

The following script is a test I use. Add stg-freeze to the path and
source/run it in an empty directory, view with gitk afterwards:

###
echo "Initial commit" | cg-init
stg init

stg new a -mpatch-1
echo a > a
stg add a
stg refresh -mpatch-1
stg new b -mpatch-2
echo b > b
stg add b
stg refresh -mpatch-2
stg-freeze
stg pop
echo a2 >> a
stg refresh -mpatch-1-update
stg push
stg-freeze
echo b2 >> b
stg refresh -mpatch-2-update


stg pop -a
echo c > c
cg-add c
cg-commit -m"Mainline advance"
stg push -a

stg-freeze

stg pop -a
echo d > d
cg-add d
cg-commit -m"Mainline advance 2"
stg push -a

stg pop -a
echo e > e
cg-add e
cg-commit -m"Mainline advance 3"
stg push -a

###

Comments/remarks are very welcome.

---
 stgit/stack.py |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)


diff --git a/stgit/stack.py b/stgit/stack.py
--- a/stgit/stack.py
+++ b/stgit/stack.py
@@ -140,6 +140,13 @@ class Patch:
         elif os.path.isfile(fname):
             os.remove(fname)
 
+    def get_parents(self):
+        parents=[]
+        if (self.__get_field('parent') != None):
+                parents = parents + [self.__get_field('parent')]
+        parents = parents + [ self.get_bottom() ]
+        return parents
+
     def get_bottom(self):
         return self.__get_field('bottom')
 
@@ -362,7 +369,7 @@ class Series:
         if not committer_email:
             committer_email = patch.get_commemail()
 
-        commit_id = git.commit(message = descr, parents = [patch.get_bottom()],
+        commit_id = git.commit(message = descr, parents = patch.get_parents(),
                                allowempty = True,
                                author_name = author_name,
                                author_email = author_email,

Re: [RFC] Stgit - patch history / add extra parents

[Catalin Marinas]

Jan Veldeman <jan.veldeman@gmail.com> wrote:
> I like stgit very much, but I feel there is still something missing:
> stgit is very handy when you use it for patches which should be pushed to
> mainline rather quickly. But for pacthes which won't be pushed immediately
> to mainline, it would be usefull to have a history of the patches
> itself.

The patch history feature was available in StGIT 0.1/0.2 releases
where you should have run a 'stg commit' before 'stg refresh'. The
commit was handling all the history changes, with separate commit
messages and refresh was updating the main commit with 2 parents. I
removed it in 0.3 after some people (I think it was Paul Jackson and
Daniel Barkalow) convinced me that this would make it yet another SCM
interface on top of GIT, which wasn't really my intention.

The main problem with having multiple parents for a commit object
corresponding to a patch is the upstream merging via 'git pull'. In
general you don't want a gatekeeper to pull the history of your patch
but the patch only.

> The patch below, together with the following script could be used to
> make snapshots of the patch stack (I call it freeze, as I thought snapshot
> was already going to be used for something else):

'snapshot' is not yet used for anything and I'm not sure how it is
best to be implemented. I thought about simply saving the current HEAD
into some .git/refs/heads/<file>, without preserving any history for
the patch. A gitk on this file would show the patches as they were on
the time of the snapshot creation. A new snapshot would remove this.

It might be best for a per-patch history to have a separate file in
<branch>/<patch>/, maybe called freeze, which keeps this history
information. The top one should remain unchanged. Its hash could be
accessed with the 'stg id /freeze' command (implemented
yesterday). This file would only be updated via the 'freeze' command
and its parent would be the previous freeze value.

Would this be close to what you need?

-- 
Catalin

Re: [RFC] Stgit - patch history / add extra parents

[Jan Veldeman]

Catalin Marinas wrote:

> 
> The patch history feature was available in StGIT 0.1/0.2 releases
> where you should have run a 'stg commit' before 'stg refresh'. The
> commit was handling all the history changes, with separate commit
> messages and refresh was updating the main commit with 2 parents. I
> removed it in 0.3 after some people (I think it was Paul Jackson and
> Daniel Barkalow) convinced me that this would make it yet another SCM
> interface on top of GIT, which wasn't really my intention.

hmm, I must have misted those threads, I'll try to find and read them.

> 
> The main problem with having multiple parents for a commit object
> corresponding to a patch is the upstream merging via 'git pull'. In
> general you don't want a gatekeeper to pull the history of your patch
> but the patch only.

I agree that such history should not be imported into the mainline, but such
history would still be very usefull when these patches won't be pushed to
mailine immediately. Also, when pushing to mainline, this history can easily
be removed by removing the branch/patch/parent files and refreshing (this
should off course be automated)

> 
> > The patch below, together with the following script could be used to
> > make snapshots of the patch stack (I call it freeze, as I thought snapshot
> > was already going to be used for something else):
> 
> 'snapshot' is not yet used for anything and I'm not sure how it is
> best to be implemented. I thought about simply saving the current HEAD
> into some .git/refs/heads/<file>, without preserving any history for
> the patch. A gitk on this file would show the patches as they were on
> the time of the snapshot creation. A new snapshot would remove this.
> 
> It might be best for a per-patch history to have a separate file in
> <branch>/<patch>/, maybe called freeze, which keeps this history
> information. The top one should remain unchanged. Its hash could be
> accessed with the 'stg id /freeze' command (implemented
> yesterday). This file would only be updated via the 'freeze' command
> and its parent would be the previous freeze value.
> 
> Would this be close to what you need?
> 

hmm, not exactly, for example, when reordering the patches (including the
top one), I would like to see this in gitk.
Or when a patch has been dropped (amongst a lot of patches), it should be
easily spotted.


But even if the "stg-freeze" would not be incorporated into stgit, would it
still be possible to include some sort of extra parents directory. So that
the freeze can be implemented on top of stgit?

TIA

Best regards,
Jan

Re: [RFC] Stgit - patch history / add extra parents

[Jan Veldeman]

Catalin Marinas wrote:

> 
> The patch history feature was available in StGIT 0.1/0.2 releases
> where you should have run a 'stg commit' before 'stg refresh'. The
> commit was handling all the history changes, with separate commit
> messages and refresh was updating the main commit with 2 parents. I
> removed it in 0.3 after some people (I think it was Paul Jackson and
> Daniel Barkalow) convinced me that this would make it yet another SCM
> interface on top of GIT, which wasn't really my intention.

I've quickly reread the threads about stg commit. Am I right to assume that
_all_ history was being recorded? Because this is not what I want. The
person controlling the archive should specify when to record the history.

So for example, you only tag (freeze) the history when exporting the
patches.  When an error is being reported on that version, it's easy to view
it and also view the progress that was already been made on those patches.


Best regards,
Jan

Re: [RFC] Stgit - patch history / add extra parents

[Catalin Marinas]

On Fri, 2005-08-19 at 20:27 +0200, Jan Veldeman wrote:
> hmm, not exactly, for example, when reordering the patches (including the
> top one), I would like to see this in gitk.
> Or when a patch has been dropped (amongst a lot of patches), it should be
> easily spotted.

I tried your patch but the gitk image confused me. I will look again at
it tomorrow (it's quite late in the UK now).

One immediate thing I noticed was that the commits directly accessible
via .git/HEAD are shown as empty by gitk and you would need to follow
the parents to see what they contain. For every freeze, the patches
expand to the right in gitk and the graph could get very complex after
several freeze commands.

-- 
Catalin

Re: [RFC] Stgit - patch history / add extra parents

[Catalin Marinas]

On Fri, 2005-08-19 at 21:48 +0200, Jan Veldeman wrote:
> I've quickly reread the threads about stg commit. Am I right to assume that
> _all_ history was being recorded? Because this is not what I want. The
> person controlling the archive should specify when to record the history.

True, I was talking about the full history, but I don't find the idea
much different. There was another mail on this issue -
http://article.gmane.org/gmane.comp.version-control.git/5191

> So for example, you only tag (freeze) the history when exporting the
> patches.  When an error is being reported on that version, it's easy to view
> it and also view the progress that was already been made on those patches.

I agree that it is a useful feature to be able to individually tag the
patches. The problem is how to do this best. Your approach looks to me
like it's not following the GIT DAG structure recommendation. Maybe the
GIT designers could further comment on this but a commit object with
multiple parents should be a result of a merge operation. A commit with
a single parent should represent a transition of the tree from one state
to another. With the freeze command you proposed, a commit with multiple
parents is no longer a result of a merge operation, but just a
convenience for tracking the patch history with gitk.

I tried your patch and test script but, if you try to reorder the
patches, the DAG structure become quite hard to understand and that's
only for a small set of frozen states.

I think the best approach would be to keep a list of tags for each
patch. They won't be browsable with gitk but the DAG structure would be
clean. We might get another tool like gquilt to be able to show all the
saved states of a patch.

Sorry for not including your patch but, while I agree with the general
idea of frozen patch states, I disagree with its implementation.

-- 
Catalin

Re: [RFC] Stgit - patch history / add extra parents

[Jan Veldeman]

Catalin Marinas wrote:

> > So for example, you only tag (freeze) the history when exporting the
> > patches.  When an error is being reported on that version, it's easy to view
> > it and also view the progress that was already been made on those patches.
> 
> I agree that it is a useful feature to be able to individually tag the
> patches. The problem is how to do this best. Your approach looks to me
> like it's not following the GIT DAG structure recommendation. Maybe the
> GIT designers could further comment on this but a commit object with
> multiple parents should be a result of a merge operation. A commit with
> a single parent should represent a transition of the tree from one state
> to another. With the freeze command you proposed, a commit with multiple
> parents is no longer a result of a merge operation, but just a
> convenience for tracking the patch history with gitk.

My interpretation of parents is broader than only merges, and reading the
README file, I believe it also the intension to do so (snippet from README
file):

A "commit" object ties such directory hierarchies together into
a DAG of revisions - each "commit" is associated with exactly one tree
(the directory hierarchy at the time of the commit). In addition, a
"commit" refers to one or more "parent" commit objects that describe the
history of how we arrived at that directory hierarchy.


Another advantage of this approach is when pushing the top of your
repository to others, less rebasing would be noticed:
When pushing the top of your stack out to others, the new one won't be a
(possibly indirect) child of the previous push. This makes updating such a
head more dificult. With this approach, the new head will be a (possibly
indirect) child of the previous one (except when the top patch has been
removed, but this corner case could also be solved).

Because of this less rebasing, this aproach also facilitates co-development
on stgit patch stacks: by using a base:top id pair, the stgit patch stack
could be initialized, including its exported history. By following the
parent/child relations, 2 patch stacks could be compared and merged.

> 
> I tried your patch and test script but, if you try to reorder the
> patches, the DAG structure become quite hard to understand and that's
> only for a small set of frozen states.

I acknowledge this problem, but the solution for this would be to give
gitk another way of viewing such history: only the latest description of a
patch should be visible and the history should become more horizontal
(because the older descriptions wouldn't be shown, the history of those
patches could be viewed on the line with the latest description)

But changing gitk would only be usefull when this approach would be taken.

> 
> I think the best approach would be to keep a list of tags for each
> patch. They won't be browsable with gitk but the DAG structure would be
> clean. We might get another tool like gquilt to be able to show all the
> saved states of a patch.
> 
> Sorry for not including your patch but, while I agree with the general
> idea of frozen patch states, I disagree with its implementation.
> 

I understand your objections when limiting parents only to merges, but I see
them broader. So as you said: we indeed need a git designer to clear this up
for us.

Best regards,
Jan

Re: [RFC] Stgit - patch history / add extra parents

[Daniel Barkalow]

On Sun, 21 Aug 2005, Jan Veldeman wrote:

> Catalin Marinas wrote:
>
> > > So for example, you only tag (freeze) the history when exporting the
> > > patches.  When an error is being reported on that version, it's easy to view
> > > it and also view the progress that was already been made on those patches.
> >
> > I agree that it is a useful feature to be able to individually tag the
> > patches. The problem is how to do this best. Your approach looks to me
> > like it's not following the GIT DAG structure recommendation. Maybe the
> > GIT designers could further comment on this but a commit object with
> > multiple parents should be a result of a merge operation. A commit with
> > a single parent should represent a transition of the tree from one state
> > to another. With the freeze command you proposed, a commit with multiple
> > parents is no longer a result of a merge operation, but just a
> > convenience for tracking the patch history with gitk.
>
> My interpretation of parents is broader than only merges, and reading the
> README file, I believe it also the intension to do so (snippet from README
> file):
>
> A "commit" object ties such directory hierarchies together into
> a DAG of revisions - each "commit" is associated with exactly one tree
> (the directory hierarchy at the time of the commit). In addition, a
> "commit" refers to one or more "parent" commit objects that describe the
> history of how we arrived at that directory hierarchy.

One factor not mentioned there is that, as things move upstream, we often
want to discard a lot of history; if someone commits constantly to deal
with editor malfunction or something, we don't really want to take all of
this junk into the project history when it is cleaned up and accepted.

So the point is that there are things which are, in fact, parents, but we
don't want to list them, because it's not desired information.

Probably the right thing is to have two views of the stack: the internal
view, showing what actually happened, and the external view, showing what
would have happened if the developers had done everything right the first
time. When you make changes to the series, this adds to the internal view
and entirely replaces the external view.

I think that users will also want to discard the commits from the stack
before rebasing in favor of the commits after, because (a) rebasing isn't
all that interesting, especially if there's minimal merging, and (b)
otherwise you'd get a ton of boring commits that obscure the interesting
ones.

I think that the best rule would be that, when you modify a patch, the
previous version is the new version's parent, and when you rebase a
series, you include as a parent any parent of the input that isn't also in
the input (but never include the input itself as a parent of the output;
the point of rebasing is to pretend that it was the newer mainline that
you modified). This should mean that the internal history of a patch
consists of the present version, based on each version that was replaced
due to changing the patch rather than rebasing it.

Of course, there's an interesting situation with the commits earlier in a
series from a patch that was changed not being ancestors of the newer
versions of those patches (because they weren't interesting in the
development of those patches) but accessible as the commits that an
interesting patch was based on.

A possible solution is just to consider the revision of any patch a
significant event in the history of the whole stack, causing all of the
patches to get a new retained version.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Stgit - patch history / add extra parents

[Junio C Hamano]

Jan Veldeman <jan.veldeman@gmail.com> writes:

> ... So as you said: we indeed need a git designer to clear this up
> for us.

I understand I am counted as one of the "git designers"; I have
been aware of this thread but I have to admit that I am way
behind and have not caught up with the issues.  I'll try to
comment on them when I feel I know enough about the issues
involved, but not right now.  Sorry.

Please do not start explain them to me yet, I am behind due to
lack of time and you would end up wasting your time.

Re: [RFC] Stgit - patch history / add extra parents

[Catalin Marinas]

Daniel Barkalow <barkalow@iabervon.org> wrote:
> One factor not mentioned there is that, as things move upstream, we often
> want to discard a lot of history; if someone commits constantly to deal
> with editor malfunction or something, we don't really want to take all of
> this junk into the project history when it is cleaned up and
> accepted.

That's true but Jan's proposal is to choose which commits to preserve
in the history via a 'freeze' command. That's a bit difficult to
implement in a clean way since the patches are floating on top of the
stack base and they change every time the base changes. It is possible
that a previous frozen state might no longer apply on top of a new
base.

> So the point is that there are things which are, in fact, parents, but we
> don't want to list them, because it's not desired information.

What's the definition of a parent in GIT terms? What are the
restriction for a commit object to be a parent? Can a parent be an
arbitrarily chosen commit?

> Probably the right thing is to have two views of the stack: the internal
> view, showing what actually happened, and the external view, showing what
> would have happened if the developers had done everything right the first
> time. When you make changes to the series, this adds to the internal view
> and entirely replaces the external view.

That's what I've been thinking. StGIT currently only implements the
external view.

An StGIT patch is a represented by a top and bottom commit
objects. The bottom one is the same as the parent of the top
commit. The patch is the diff between the top's tree id and the
bottom's tree id.

Jan's proposal is to allow a freeze command to save the current top
hash and later be used as a second parent for the newly generated
top. The problem I see with this approach is that (even for the
internal view you described) the newly generated top will have two
parents, new-bottom and old-top, but only the diff between new-top and
new-bottom is meaningful. The diff between new-top and old-top (as a
parent-child relation) wouldn't contain anything relevant to the patch
but all the new changes to the base of the stack.

Is the above an acceptable usage of the GIT DAG structure?

> I think that users will also want to discard the commits from the stack
> before rebasing in favor of the commits after, because (a) rebasing isn't
> all that interesting, especially if there's minimal merging, and (b)
> otherwise you'd get a ton of boring commits that obscure the interesting
> ones.

StGIT does this currently. The old commit is no longer available and
can be pruned (I still need to save the commits corresponding to the
unpushed patches since prune would remove those as well).

> I think that the best rule would be that, when you modify a patch, the
> previous version is the new version's parent, and when you rebase a
> series, you include as a parent any parent of the input that isn't also in
> the input (but never include the input itself as a parent of the output;
> the point of rebasing is to pretend that it was the newer mainline that
> you modified). This should mean that the internal history of a patch
> consists of the present version, based on each version that was replaced
> due to changing the patch rather than rebasing it.

Since you proposed this, my above usage of the DAG structure would be
fine then.

> Of course, there's an interesting situation with the commits earlier in a
> series from a patch that was changed not being ancestors of the newer
> versions of those patches (because they weren't interesting in the
> development of those patches) but accessible as the commits that an
> interesting patch was based on.
>
> A possible solution is just to consider the revision of any patch a
> significant event in the history of the whole stack, causing all of the
> patches to get a new retained version.

That's another idea to think about.

-- 
Catalin


P.S. I'll be away until next week and not able to follow-up the
discussions.

Re: [RFC] Stgit - patch history / add extra parents

[Daniel Barkalow]

On Tue, 23 Aug 2005, Catalin Marinas wrote:

> > So the point is that there are things which are, in fact, parents, but we
> > don't want to list them, because it's not desired information.
>
> What's the definition of a parent in GIT terms? What are the
> restriction for a commit object to be a parent? Can a parent be an
> arbitrarily chosen commit?

Something is legitimate as a parent if someone took that commit and did
something to it to get the new commit. The operation which caused the
change is not specified. But you only want to include it if anyone cares
about the parent.

(For example, I often start with a chunk of work that does multiple things
and is committed; I take mainline and generate a series of commits from
there. It would be legitimate to list my development commit as a parent of
each of these, since I did actually take it and strip out the unrelated
changes. This would be a bit confusing in the log, but would make merges
between something based on the "messy" version and something based on the
"refined" version work well. On the other hand, I don't want to report the
existance of the messy version, so I don't include it.)

> An StGIT patch is a represented by a top and bottom commit
> objects. The bottom one is the same as the parent of the top
> commit. The patch is the diff between the top's tree id and the
> bottom's tree id.
>
> Jan's proposal is to allow a freeze command to save the current top
> hash and later be used as a second parent for the newly generated
> top. The problem I see with this approach is that (even for the
> internal view you described) the newly generated top will have two
> parents, new-bottom and old-top, but only the diff between new-top and
> new-bottom is meaningful. The diff between new-top and old-top (as a
> parent-child relation) wouldn't contain anything relevant to the patch
> but all the new changes to the base of the stack.

Having a useful diff isn't really a requirement for a parent; the diff in
the case of a merge is going to be the total of everything that happened
elsewhere. The point is to be able to reach some commits between which
there are interesting diffs.

This also depends on how exactly freeze is used; if you use it before
commiting a modification to the patch without rebasing, you get:

old-top -> new-top
      ^    ^
       \  /
      bottom

bottom to old-top is the old patch
bottom to new-top is the new patch
old-top to new-top is the change to the patch

Then you want to keep new-top as a parent for rebasings until one of these
is frozen. These links are not interesting to look at, but preserve the
path to the old-top:new-top change, which is interesting.

Ignoring the links to the corresponding bottoms, the development therefore
looks like:

local1 -> local2 -> merge -> local3 -> merge
^                   ^                  ^
mainline---->-->--------->------>-->----->

And this is how development is normally supposed to look. The trick is to
only include a minimal number of merges.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Stgit - patch history / add extra parents

[Jan Veldeman]

Daniel Barkalow wrote:

> On Tue, 23 Aug 2005, Catalin Marinas wrote:
> 
> Something is legitimate as a parent if someone took that commit and did
> something to it to get the new commit. The operation which caused the
> change is not specified. But you only want to include it if anyone cares
> about the parent.

This is indeed what I thought a parent should be used for. As an adition,
I'll try to explain why I would sometimes want to care about some parents:

I want to track a mailine tree, but have quite a few changes, which shoudn't
be commited to the mainline immediately (let's call it my development tree).
This is why I would use stgit. But I would also want to colaborate with
other developers on this development tree, so I sometimes want to make
updates available of this development tree to the others. This is where
current stgit falls short. To easily share this development tree, I want
some history (not all, only the ones I choose) of this development tree
included, so that the other developers can easily follow my development.

The parents which should be visible to the outside, will always be versions
of my development tree, which I have previously pushed out. My way of
working would become:
* make changes, all over the place, using stgit
* still make changes (none of these gets tracked, intermittent versions are
  lost)
* having a good day: changes looks good, I want to push this out:
  * push my tree out
  * stgit-free (which makes the pushed out commits, the new parents of my
    stgit patches)
* restart from top

[...]
> This also depends on how exactly freeze is used; if you use it before
> commiting a modification to the patch without rebasing, you get:
> 
> old-top -> new-top
>       ^    ^
>        \  /
>       bottom
> 
> bottom to old-top is the old patch
> bottom to new-top is the new patch
> old-top to new-top is the change to the patch
> 
> Then you want to keep new-top as a parent for rebasings until one of these
> is frozen. These links are not interesting to look at, but preserve the
> path to the old-top:new-top change, which is interesting.

my proposal does something like this, but a little more: not only does it
keep track of the link between old-top and new-top, it also keeps track of
the links between old-patch-in-between and new-patch-in-between.
(This makes sense when the top is being removed or reordered)

I hope this kind of clarifies my intension.


Thank you for clarifying the meaning of parents.

Best regards,
Jan

Re: [RFC] Stgit - patch history / add extra parents

[Daniel Barkalow]

On Tue, 23 Aug 2005, Jan Veldeman wrote:

> Daniel Barkalow wrote:
>
> > On Tue, 23 Aug 2005, Catalin Marinas wrote:
> >
> > Something is legitimate as a parent if someone took that commit and did
> > something to it to get the new commit. The operation which caused the
> > change is not specified. But you only want to include it if anyone cares
> > about the parent.
>
> This is indeed what I thought a parent should be used for. As an adition,
> I'll try to explain why I would sometimes want to care about some parents:
>
> I want to track a mailine tree, but have quite a few changes, which shoudn't
> be commited to the mainline immediately (let's call it my development tree).
> This is why I would use stgit. But I would also want to colaborate with
> other developers on this development tree, so I sometimes want to make
> updates available of this development tree to the others. This is where
> current stgit falls short. To easily share this development tree, I want
> some history (not all, only the ones I choose) of this development tree
> included, so that the other developers can easily follow my development.
>
> The parents which should be visible to the outside, will always be versions
> of my development tree, which I have previously pushed out. My way of
> working would become:
> * make changes, all over the place, using stgit
> * still make changes (none of these gets tracked, intermittent versions are
>   lost)
> * having a good day: changes looks good, I want to push this out:
>   * push my tree out
>   * stgit-free (which makes the pushed out commits, the new parents of my
>     stgit patches)
> * restart from top

I'm not sure how applicable to this situation stgit really is; I see stgit
as optimized for the case of a patch set which is basically done, where
you want to keep it applicable to the mainline as the mainline advances.

For your application, I'd just have a git branch full of various stuff,
and then generate clean commits by branching mainline, diffing development
against it, cutting the diff down to just what I want to push, and
applying that. Then the clean patch goes into stgit.

> [...]
> > This also depends on how exactly freeze is used; if you use it before
> > commiting a modification to the patch without rebasing, you get:
> >
> > old-top -> new-top
> >       ^    ^
> >        \  /
> >       bottom
> >
> > bottom to old-top is the old patch
> > bottom to new-top is the new patch
> > old-top to new-top is the change to the patch
> >
> > Then you want to keep new-top as a parent for rebasings until one of these
> > is frozen. These links are not interesting to look at, but preserve the
> > path to the old-top:new-top change, which is interesting.
>
> my proposal does something like this, but a little more: not only does it
> keep track of the link between old-top and new-top, it also keeps track of
> the links between old-patch-in-between and new-patch-in-between.
> (This makes sense when the top is being removed or reordered)

I was thinking of this as being the top and bottom commits for a single
tracked patch, not as a whole series. I think patches lower wouldn't be
affected, and patches higher would see this as a rebase.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Stgit - patch history / add extra parents

[Junio C Hamano]

Catalin Marinas <catalin.marinas@gmail.com> writes:

> What's the definition of a parent in GIT terms? What are the
> restriction for a commit object to be a parent? Can a parent be an
> arbitrarily chosen commit?

Yes it can.  GIT does not care if the commit ancestry does not
make sense in contents terms (i.e. you can record one tree
object in a commit object, and record another, completely
unrelated tree object in a commit object that has the first
commit object as its parent).  The "git-diff-tree" output from
comparing those two commits may not make _any_ sense at all to
the human, though, but that is not a problem for GIT to do its
work.

> That's what I've been thinking. StGIT currently only implements the
> external view.
>
> An StGIT patch is a represented by a top and bottom commit
> objects. The bottom one is the same as the parent of the top
> commit. The patch is the diff between the top's tree id and the
> bottom's tree id.
>
> Jan's proposal is to allow a freeze command to save the current top
> hash and later be used as a second parent for the newly generated
> top. The problem I see with this approach is that (even for the
> internal view you described) the newly generated top will have two
> parents, new-bottom and old-top, but only the diff between new-top and
> new-bottom is meaningful. The diff between new-top and old-top (as a
> parent-child relation) wouldn't contain anything relevant to the patch
> but all the new changes to the base of the stack.
>
> Is the above an acceptable usage of the GIT DAG structure?

Surely, as long as it stays internal as StGIT patch stacks.
Even when it is exported (i.e. you manage to have other non
StGIT managed tree to pull and merge with such a commit), the
development history _might_ look funny and complicated, but it
should not be a problem for GIT to handle.  After all, you are
doing complicated thing in the StGIT patch stacks when you pop,
push and freeze number of times, so your history is complicated.
If you want to record that history at least for StGIT internal
bookkeeping purposes, GIT should not prevent you from doing it.

Having said that, I would 100% agree with you that having a
cleansed external view would be a good idea.  It makes merging
with StGIT managed repositories much more acceptable to ordinary
GIT managed history.

Re: [RFC] Stgit - patch history / add extra parents

[Linus Torvalds]

On Tue, 23 Aug 2005, Junio C Hamano wrote:
> 
> Yes it can.  GIT does not care if the commit ancestry does not
> make sense in contents terms (i.e. you can record one tree
> object in a commit object, and record another, completely
> unrelated tree object in a commit object that has the first
> commit object as its parent).  The "git-diff-tree" output from
> comparing those two commits may not make _any_ sense at all to
> the human, though, but that is not a problem for GIT to do its
> work.

One issue is later merges.

If you have incorrect parent information, trying to do a merge may prove
impossible and/or ignore changes from the "already merged" stream. By
marking another head as a parent, you basically say "I have merged
_everything_ leading up to that other head", and if you have only actually
done a partial merge (and gotten just a part of the changes, ignoring the
others), you'll have to notice that yourself, and forward-port the rest by
hand.

For stgit, this probably doesn't matter.

		Linus

Re: [RFC] Stgit - patch history / add extra parents

[Jan Veldeman]

Daniel Barkalow wrote:

[...]
> > The parents which should be visible to the outside, will always be versions
> > of my development tree, which I have previously pushed out. My way of
> > working would become:
> > * make changes, all over the place, using stgit
> > * still make changes (none of these gets tracked, intermittent versions are
> >   lost)
> > * having a good day: changes looks good, I want to push this out:
> >   * push my tree out
> >   * stgit-free (which makes the pushed out commits, the new parents of my
> >     stgit patches)
> > * restart from top
> 
> I'm not sure how applicable to this situation stgit really is; I see stgit
> as optimized for the case of a patch set which is basically done, where
> you want to keep it applicable to the mainline as the mainline advances.

Maybe I forgot to mention this: I would also like to have my development
tree split up in a patch stack. The separate patches makes tracking the
mainline a lot easier (conflicts are a lot easier to solve)

> 
> For your application, I'd just have a git branch full of various stuff,
> and then generate clean commits by branching mainline, diffing development
> against it, cutting the diff down to just what I want to push, and
> applying that. Then the clean patch goes into stgit.

But this would assume that once the patch goes into stgit, it won't
change except when the parent gets updated. I think we will still change
the patches quite a bit and simultanious by a couple of people.

> 
> > [...]
> > my proposal does something like this, but a little more: not only does it
> > keep track of the link between old-top and new-top, it also keeps track of
> > the links between old-patch-in-between and new-patch-in-between.
> > (This makes sense when the top is being removed or reordered)
> 
> I was thinking of this as being the top and bottom commits for a single
> tracked patch, not as a whole series. I think patches lower wouldn't be
> affected, and patches higher would see this as a rebase.
> 

ah, ok, I misunderstood that part

Best regards,
Jan

PS. sorry if my responses are sometimes a bit late, I'm trying to find
more time to spend on this list ;-)

Re: [RFC] Stgit - patch history / add extra parents

[Daniel Barkalow]

On Thu, 25 Aug 2005, Jan Veldeman wrote:

> Daniel Barkalow wrote:
>
> > I'm not sure how applicable to this situation stgit really is; I see stgit
> > as optimized for the case of a patch set which is basically done, where
> > you want to keep it applicable to the mainline as the mainline advances.
>
> Maybe I forgot to mention this: I would also like to have my development
> tree split up in a patch stack. The separate patches makes tracking the
> mainline a lot easier (conflicts are a lot easier to solve)

I just try to keep things in this state sufficiently briefly that it
doesn't become a problem. I also split things up into a bunch of branches,
rather than into a stack of patches, and only work on parallel development
before I've actually got a candidate for a series.

> But this would assume that once the patch goes into stgit, it won't
> change except when the parent gets updated. I think we will still change
> the patches quite a bit and simultanious by a couple of people.

The extension I had proposed to stgit should work for this; it would let
you version control each patch just like other git projects. I just think
it wouldn't work so well before the group has agreed on what patches there
are.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Stgit - patch history / add extra parents

[Catalin Marinas]

Back from holiday. Thanks to all who replied to this thread.

On Tue, 2005-08-23 at 14:05 -0400, Daniel Barkalow wrote:
> Having a useful diff isn't really a requirement for a parent; the diff in
> the case of a merge is going to be the total of everything that happened
> elsewhere. The point is to be able to reach some commits between which
> there are interesting diffs.
> 
> This also depends on how exactly freeze is used; if you use it before
> commiting a modification to the patch without rebasing, you get:
> 
> old-top -> new-top
>       ^    ^
>        \  /
>       bottom
> 
> bottom to old-top is the old patch
> bottom to new-top is the new patch
> old-top to new-top is the change to the patch
> 
> Then you want to keep new-top as a parent for rebasings until one of these
> is frozen. These links are not interesting to look at, but preserve the
> path to the old-top:new-top change, which is interesting.

This was my initial StGIT implementation (up to version 0.3), only that
there was no freeze command. Since I want an StGIT tree to be clean to
the outside world, I wouldn't keep multiple parents for the visible top
of a patch.

As I understand from Junio's and Linus' e-mails (on the 23rd of August),
there might be problems with merging the HEAD of an StGIT-managed tree
if the above method is accessible via HEAD.

> Ignoring the links to the corresponding bottoms, the development therefore
> looks like:
> 
> local1 -> local2 -> merge -> local3 -> merge
> ^                   ^                  ^
> mainline---->-->--------->------>-->----->
> 
> And this is how development is normally supposed to look. The trick is to
> only include a minimal number of merges.

A merge occurs every time a patch is rebased. Anyway, having the bottoms
in the graph (which is the main idea of StGIT) together with the old-top
(or frozen state) parents make the graph pretty complicated.

-- 
Catalin

Re: [RFC] Stgit - patch history / add extra parents

[Jan Veldeman]

Catalin, Daniel,

thank you both for your input.
I'll try your recommendations (when time is available ;-) and see how it
goes.

Best regards,
Jan

Re: [RFC] Stgit - patch history / add extra parents

[Catalin Marinas]

Jan Veldeman <jan.veldeman@gmail.com> wrote:
> The parents which should be visible to the outside, will always be versions
> of my development tree, which I have previously pushed out. My way of
> working would become:
> * make changes, all over the place, using stgit
> * still make changes (none of these gets tracked, intermittent versions are
>   lost)
> * having a good day: changes looks good, I want to push this out:
>   * push my tree out
>   * stgit-free (which makes the pushed out commits, the new parents of my
>     stgit patches)

When the tree status looks good, you could tag it and the whole stack
(the commits corresponding to the patches) would be available via this
tag.

As I said, I agree with the idea of freezing a patch and even having
multiple parents to a commit but this commit should not be visible via
HEAD. We could have another id for a patch, 'frozen', which stores the
last frozen state with the parents being the previous frozen
states. Its SHA1 value could be accessed via 'stg id <patch>/frozen'
(in the same way as /top and /bottom ids).

-- 
Catalin

Re: [RFC] Stgit - patch history / add extra parents

[Catalin Marinas]

Daniel Barkalow <barkalow@iabervon.org> wrote:
> On Tue, 23 Aug 2005, Jan Veldeman wrote:
>> The parents which should be visible to the outside, will always be versions
>> of my development tree, which I have previously pushed out. My way of
>> working would become:
>> * make changes, all over the place, using stgit
>> * still make changes (none of these gets tracked, intermittent versions are
>>   lost)
>> * having a good day: changes looks good, I want to push this out:
>>   * push my tree out
>>   * stgit-free (which makes the pushed out commits, the new parents of my
>>     stgit patches)
>> * restart from top
>
> I'm not sure how applicable to this situation stgit really is; I see stgit
> as optimized for the case of a patch set which is basically done, where
> you want to keep it applicable to the mainline as the mainline advances.
>
> For your application, I'd just have a git branch full of various stuff,
> and then generate clean commits by branching mainline, diffing development
> against it, cutting the diff down to just what I want to push, and
> applying that. Then the clean patch goes into stgit.

StGIT has the 'refresh' command which allows a patch to be
indefinitely modified (that's pretty much how I use StGIT). I use it
for the case where I would like to keep the changes in separate
logical entities (patches) but they are not independent enough to
create separate branches.

Take for example a new platform port, you can have some of the
existing code re-worked in a patch, some CPU support in the next
patch, basic platform support in another patch, some device drivers
specific to this platform in yet another patch. Subsequent patches are
dependent on the previous ones. Branching from the mainline for each
of these features might not make sense and keeping all of them in the
same branch can make maintenance a bit more difficult.

-- 
Catalin

Re: [RFC] Stgit - patch history / add extra parents

[Daniel Barkalow]

On Tue, 30 Aug 2005, Catalin Marinas wrote:

> Back from holiday. Thanks to all who replied to this thread.
> 
> On Tue, 2005-08-23 at 14:05 -0400, Daniel Barkalow wrote:
> > Having a useful diff isn't really a requirement for a parent; the diff in
> > the case of a merge is going to be the total of everything that happened
> > elsewhere. The point is to be able to reach some commits between which
> > there are interesting diffs.
> > 
> > This also depends on how exactly freeze is used; if you use it before
> > commiting a modification to the patch without rebasing, you get:
> > 
> > old-top -> new-top
> >       ^    ^
> >        \  /
> >       bottom
> > 
> > bottom to old-top is the old patch
> > bottom to new-top is the new patch
> > old-top to new-top is the change to the patch
> > 
> > Then you want to keep new-top as a parent for rebasings until one of these
> > is frozen. These links are not interesting to look at, but preserve the
> > path to the old-top:new-top change, which is interesting.
> 
> This was my initial StGIT implementation (up to version 0.3), only that
> there was no freeze command. Since I want an StGIT tree to be clean to
> the outside world, I wouldn't keep multiple parents for the visible top
> of a patch.
> 
> As I understand from Junio's and Linus' e-mails (on the 23rd of August),
> there might be problems with merging the HEAD of an StGIT-managed tree
> if the above method is accessible via HEAD.

Right, you'd want a separate head which is what you ask people to merge; 
the rest is only visible to people who are working on preparing the patch. 
But you could keep both sets of stuff (sharing tree objects but not 
commits).

> > Ignoring the links to the corresponding bottoms, the development therefore
> > looks like:
> > 
> > local1 -> local2 -> merge -> local3 -> merge
> > ^                   ^                  ^
> > mainline---->-->--------->------>-->----->
> > 
> > And this is how development is normally supposed to look. The trick is to
> > only include a minimal number of merges.
> 
> A merge occurs every time a patch is rebased. Anyway, having the bottoms
> in the graph (which is the main idea of StGIT) together with the old-top
> (or frozen state) parents make the graph pretty complicated.

It should be possible to drop merges such that there's only one between 
any pair of local changes. That is, if you rebase at the end of the line 
above, it would get as parents local3 and the new bottom, not the last 
merge and the new bottom.

The mainline changes only come in through the bottoms, so higher levels 
should look the same, but with the lower levels in the place of mainline.

	-Daniel
*This .sig left intentionally blank*