Help designing work flow

Help designing work flow

[John Dlugosz]

I know we (my group) should use "topic" branches and apply them back to
the dev branch when done.  There is concern that the commit history gets
too full of detailed stuff, especially with several developers, that you
can't tell what "really changed".  So, our dev branch should appear to
contain only commit nodes representing completed assignments; not every
day's checkpoint and trying to keep one's own stuff on top to avoid
merging later.

I guess that's how it is on these Open Source projects where patches are
submitted by email and applied by the maintainer.  We don't see the
details, just the final patch.  But, my situation will be developers
gathered around an in-house master repo, and everyone should be able to
push their own changes as assignments are completed.

What is the best procedure to achieve that?  Or what are some good
alternatives with trade-offs?

I see that if a topic branch is merged (disabling FF if applicable), the
main line (leftmost parent) will be a history of completed topics.  But,
we don't need to keep the detailed side-branches forever, and even if
gitk and other visualization  tools can be told to just show the main
line, advanced use such as git log this..that will forever be packed
with the micro-details.

So, unless someone has more input along that line, I'm assuming that we
want to apply the completed topic as a single-parent commit.  That is
the natural result if preparing patches and then applying them, but is
there a simpler, direct way to do that in git?

The detailed topic branches can be kept around for a while, for the
original author to extend if it needs to be returned to, and to examine
if the gestalt change in the single commit is too overwhelming to
understand, or to help figure out what might have broken.  But after a
while they can be deleted and then gc will free up the disk space.

Anything else I should look into?

--John


TradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD) of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE, FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and subscription company, and TradeStation Europe Limited, a United Kingdom, FSA-authorized introducing brokerage firm. None of these companies provides trading or investment advice, recommendations or endorsements of any kind. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited.
  If you received this in error, please contact the sender and delete the material from any computer.

Re: Help designing work flow

[Andreas Ericsson]

John Dlugosz wrote:
> I know we (my group) should use "topic" branches and apply them back to
> the dev branch when done.  There is concern that the commit history gets
> too full of detailed stuff, especially with several developers, that you
> can't tell what "really changed".  So, our dev branch should appear to
> contain only commit nodes representing completed assignments; not every
> day's checkpoint and trying to keep one's own stuff on top to avoid
> merging later.
> 
> I guess that's how it is on these Open Source projects where patches are
> submitted by email and applied by the maintainer.  We don't see the
> details, just the final patch.  But, my situation will be developers
> gathered around an in-house master repo, and everyone should be able to
> push their own changes as assignments are completed.
> 
> What is the best procedure to achieve that?  Or what are some good
> alternatives with trade-offs?
> 

Use topic-branches and let someone merge them into master after having
verified that they work properly.

We usually commit simple bugfixes directly to master and then have
developers rebase their changes onto master when they're ready to
integrate it. They push their development branches though, and further
changes to the topic are done on the topic-branches which can get
merged to master (or stable) many times. Once a topic is merged, we
always "git commit --amend" the merge commit to write a proper
commit-message for it, adding a link to our bug- and feature-tracker
so we get the at-a-glance information quickly and can dig up the
entire discussion history around the bug/feature later. Each topic
should be complete with documentation and test-cases before it's
merged.


> I see that if a topic branch is merged (disabling FF if applicable), the
> main line (leftmost parent) will be a history of completed topics.  But,
> we don't need to keep the detailed side-branches forever, and even if
> gitk and other visualization  tools can be told to just show the main
> line, advanced use such as git log this..that will forever be packed
> with the micro-details.
> 

You can tell "git log" to only show one line of history too, but besides
that, micro-details are good. You definitely want to be able to search
the micro-details when things go awry (and they will), so you see exactly
why some particular algorithm changed later.

> So, unless someone has more input along that line, I'm assuming that we
> want to apply the completed topic as a single-parent commit.  That is
> the natural result if preparing patches and then applying them, but is
> there a simpler, direct way to do that in git?
> 

You do not want to do that. We did it for a while, and it was hell when
we found out that one of them broke down. The really, really *nice* thing
about git is called "git bisect". What makes it so awesomely nice is
that, instead of looking at a 100k diff-file knowing that somewhere in
there a bug was introduced, you get (with good discipline using small
commits), a 1-40 line patch with a clear and concise message of why
those changes were thought necessary at that time. Applying a topic
branch as a single patch would rob you of that functionality, and you
will regret it. Trust me on this.

> The detailed topic branches can be kept around for a while, for the
> original author to extend if it needs to be returned to, and to examine
> if the gestalt change in the single commit is too overwhelming to
> understand, or to help figure out what might have broken.  But after a
> while they can be deleted and then gc will free up the disk space.
> 

But if they do need to be returned to, you cannot merge them again if
you've already applied the topic-patch (ugh), since you'd get conflicts
if any of the sections touched by the patch have been changed since.

We use topic-branches quite a lot. When we're done with them we delete
the branch-pointers but I wouldn't, ever, dream of re-cooking them as
mega-patches when applying them to master.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Help designing work flow

[John Tapsell]

2009/3/9 Andreas Ericsson <ae@op5.se>:
> John Dlugosz wrote:
>>
>> I know we (my group) should use "topic" branches and apply them back to
>> the dev branch when done.  There is concern that the commit history gets
>> too full of detailed stuff, especially with several developers, that you
>> can't tell what "really changed".  So, our dev branch should appear to
>> contain only commit nodes representing completed assignments; not every
>> day's checkpoint and trying to keep one's own stuff on top to avoid
>> merging later.
>>
>> I guess that's how it is on these Open Source projects where patches are
>> submitted by email and applied by the maintainer.  We don't see the
>> details, just the final patch.  But, my situation will be developers
>> gathered around an in-house master repo, and everyone should be able to
>> push their own changes as assignments are completed.
>>
>> What is the best procedure to achieve that?  Or what are some good
>> alternatives with trade-offs?
>>
>
> Use topic-branches and let someone merge them into master after having
> verified that they work properly.
>
> We usually commit simple bugfixes directly to master and then have
> developers rebase their changes onto master when they're ready to

The trouble with rebasing is that it then you end up with lots of
patches that haven't been tested.  You can end up with lots of
uncompiling commits.

Although merging is no better either.  Then you end up with one single
commit that tries to merge two trees, making it almost impossible to
track down bugs that resulted from the merge.

> integrate it. They push their development branches though, and further
> changes to the topic are done on the topic-branches which can get
> merged to master (or stable) many times. Once a topic is merged, we
> always "git commit --amend" the merge commit to write a proper
> commit-message for it, adding a link to our bug- and feature-tracker
> so we get the at-a-glance information quickly and can dig up the
> entire discussion history around the bug/feature later. Each topic
> should be complete with documentation and test-cases before it's
> merged.
>
>
>> I see that if a topic branch is merged (disabling FF if applicable), the
>> main line (leftmost parent) will be a history of completed topics.  But,
>> we don't need to keep the detailed side-branches forever, and even if
>> gitk and other visualization  tools can be told to just show the main
>> line, advanced use such as git log this..that will forever be packed
>> with the micro-details.
>>
>
> You can tell "git log" to only show one line of history too, but besides
> that, micro-details are good. You definitely want to be able to search
> the micro-details when things go awry (and they will), so you see exactly
> why some particular algorithm changed later.
>
>> So, unless someone has more input along that line, I'm assuming that we
>> want to apply the completed topic as a single-parent commit.  That is
>> the natural result if preparing patches and then applying them, but is
>> there a simpler, direct way to do that in git?
>>
>
> You do not want to do that. We did it for a while, and it was hell when
> we found out that one of them broke down. The really, really *nice* thing
> about git is called "git bisect". What makes it so awesomely nice is
> that, instead of looking at a 100k diff-file knowing that somewhere in
> there a bug was introduced, you get (with good discipline using small
> commits), a 1-40 line patch with a clear and concise message of why
> those changes were thought necessary at that time. Applying a topic
> branch as a single patch would rob you of that functionality, and you
> will regret it. Trust me on this.
>
>> The detailed topic branches can be kept around for a while, for the
>> original author to extend if it needs to be returned to, and to examine
>> if the gestalt change in the single commit is too overwhelming to
>> understand, or to help figure out what might have broken.  But after a
>> while they can be deleted and then gc will free up the disk space.
>>
>
> But if they do need to be returned to, you cannot merge them again if
> you've already applied the topic-patch (ugh), since you'd get conflicts
> if any of the sections touched by the patch have been changed since.
>
> We use topic-branches quite a lot. When we're done with them we delete
> the branch-pointers but I wouldn't, ever, dream of re-cooking them as
> mega-patches when applying them to master.
>
> --
> Andreas Ericsson                   andreas.ericsson@op5.se
> OP5 AB                             www.op5.se
> Tel: +46 8-230225                  Fax: +46 8-230231
>
> Considering the successes of the wars on alcohol, poverty, drugs and
> terror, I think we should give some serious thought to declaring war
> on peace.
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Help designing work flow

[Andreas Ericsson]

John Tapsell wrote:
> 2009/3/9 Andreas Ericsson <ae@op5.se>:
>> John Dlugosz wrote:
>>> I know we (my group) should use "topic" branches and apply them back to
>>> the dev branch when done.  There is concern that the commit history gets
>>> too full of detailed stuff, especially with several developers, that you
>>> can't tell what "really changed".  So, our dev branch should appear to
>>> contain only commit nodes representing completed assignments; not every
>>> day's checkpoint and trying to keep one's own stuff on top to avoid
>>> merging later.
>>>
>>> I guess that's how it is on these Open Source projects where patches are
>>> submitted by email and applied by the maintainer.  We don't see the
>>> details, just the final patch.  But, my situation will be developers
>>> gathered around an in-house master repo, and everyone should be able to
>>> push their own changes as assignments are completed.
>>>
>>> What is the best procedure to achieve that?  Or what are some good
>>> alternatives with trade-offs?
>>>
>> Use topic-branches and let someone merge them into master after having
>> verified that they work properly.
>>
>> We usually commit simple bugfixes directly to master and then have
>> developers rebase their changes onto master when they're ready to
> 
> The trouble with rebasing is that it then you end up with lots of
> patches that haven't been tested.  You can end up with lots of
> uncompiling commits.
> 

Not really, no. Unit-tests can still run just fine, and integration
testing still needs to be done after each feature is completed.

> Although merging is no better either.  Then you end up with one single
> commit that tries to merge two trees, making it almost impossible to
> track down bugs that resulted from the merge.
> 

Not really. If bugs are in "unrelated" areas (if the topic changed some
API without changing its' other callers, fe), you can backstep between
each commit on the merged branch, remerge that commit (instead of the
tip) and then run the tests again. But really, such bugs should have
been detected prior to merging the branch, and in any case "git bisect"
will find the commit that introduced the bug for you either way.


For next time, please cut away those parts of the email you didn't
reply to. I had to scroll down to the bottom to make sure you hadn't
written more.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Help designing work flow

[John Tapsell]

2009/3/9 Andreas Ericsson <ae@op5.se>:
> John Tapsell wrote:

> Not really. If bugs are in "unrelated" areas (if the topic changed some
> API without changing its' other callers, fe), you can backstep between
> each commit on the merged branch, remerge that commit (instead of the
> tip) and then run the tests again.

Doing that manually?  Sounds really complicated, especially if the
merge is nontrivial.

> But really, such bugs should have
> been detected prior to merging the branch, and in any case "git bisect"
> will find the commit that introduced the bug for you either way.

How will you detect bugs that arise from merging two trees, before you
merge them?

> For next time, please cut away those parts of the email you didn't
> reply to. I had to scroll down to the bottom to make sure you hadn't
> written more.

I'll try sorry.  Most email clients will hide the quoted text - there
might be an option for that in your client?

John

Re: Help designing work flow

[Andreas Ericsson]

John Tapsell wrote:
> 2009/3/9 Andreas Ericsson <ae@op5.se>:
>> John Tapsell wrote:
> 
>> Not really. If bugs are in "unrelated" areas (if the topic changed some
>> API without changing its' other callers, fe), you can backstep between
>> each commit on the merged branch, remerge that commit (instead of the
>> tip) and then run the tests again.
> 
> Doing that manually?  Sounds really complicated, especially if the
> merge is nontrivial.
> 

Well, using git bisect should work.

>> But really, such bugs should have
>> been detected prior to merging the branch, and in any case "git bisect"
>> will find the commit that introduced the bug for you either way.
> 
> How will you detect bugs that arise from merging two trees, before you
> merge them?
> 

Assuming you don't start each topic-branch from a completely empty tree
and that you have *some* tests, you can run those tests at every single
commit just as if you had done those on the 'master' branch. API changes
that break things should break earlier, unless another branch made those
changes.

Aside from that, it's quite simple to run tests before publishing the
results of the merge, and it's absolutely trivial to undo the merge
in case tests don't pass.

>> For next time, please cut away those parts of the email you didn't
>> reply to. I had to scroll down to the bottom to make sure you hadn't
>> written more.
> 
> I'll try sorry.  Most email clients will hide the quoted text - there
> might be an option for that in your client?
> 

I'm sure there is, but then I wouldn't have seen what you were replying
to, so that's not a very good option. In the space between me sending
and you replying, I wrote six other emails. Keeping track of them all
in my head is not something I'm prepared to even consider trying.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

RE: Help designing work flow

[John Dlugosz]

Thank you very much for your thoughts, especially the specific commands
etc. that I can show the team as reasons.

=== Re: ===
You can tell "git log" to only show one line of history too, but besides
that, micro-details are good. You definitely want to be able to search
the micro-details when things go awry (and they will), so you see
exactly
why some particular algorithm changed later.
=== end ===

Is there a way that someone (for example, product manager) can view only
the main line consisting of topic merges, and not see each one separated
by many many lines of detail changes?  Especially using the GUI?

--John
(please excuse the footer; it's not my idea)

TradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD) of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE, FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and subscription company, and TradeStation Europe Limited, a United Kingdom, FSA-authorized introducing brokerage firm. None of these companies provides trading or investment advice, recommendations or endorsements of any kind. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited.
  If you received this in error, please contact the sender and delete the material from any computer.

RE: Help designing work flow

[John Dlugosz]

> You can tell "git log" to only show one line of history too, but
> besides
> that, micro-details are good. You definitely want to be able to search
> the micro-details when things go awry (and they will), so you see
> exactly
> why some particular algorithm changed later.
> 

I misread that the first time.  I thought you meant that you can tell
git log to follow down the left parents only.

So, how would you do that?  List the completed merged topic nodes only,
not the detailed nodes that make it up?

--John

TradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD) of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE, FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and subscription company, and TradeStation Europe Limited, a United Kingdom, FSA-authorized introducing brokerage firm. None of these companies provides trading or investment advice, recommendations or endorsements of any kind. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited.
  If you received this in error, please contact the sender and delete the material from any computer.