SCSI trees, merges and git status

SCSI trees, merges and git status

[James Bottomley]

As of today, I have two SCSI git trees operational:

rsync://www.parisc-linux.org/~jejb/scsi-rc-fixes-2.6.git

and

rsync://www.parisc-linux.org/~jejb/scsi-misc-2.6.git

The latter has a non trivial merge in it because of a conflict in
scsi_device.h, so merges actually do work ...

The trees are exported from BK a changeset at a time (except the merge
bits, which were done manually).  I'll continue to accumulate patches in
the BK trees for the time being since we don't have a nice web browser
interface for the git trees (and also my commit scripts are all BK
based).

Linus, the rc-fixes repo is ready for applying ... it's the same one I
announced on linux-scsi and lkml a while ago just with the git date
information updated to be correct (the misc one should wait until after
2.6.12 is final).

James

Re: SCSI trees, merges and git status

[Linus Torvalds]

On Mon, 18 Apr 2005, James Bottomley wrote:
>
> As of today, I have two SCSI git trees operational:
> 
> rsync://www.parisc-linux.org/~jejb/scsi-rc-fixes-2.6.git

Merged. Here's the command line history:

	~/git/git-pull-script rsync://www.parisc-linux.org/~jejb/scsi-rc-fixes-2.6.git
	merge-cache ~/git/git-merge-one-file-script -a
	write-tree 
	commit-tree 2c8de70faf92af971667a26a6a397052fc572add -p $(cat .git/HEAD) -p $(cat .git/MERGE_HEAD)

ie the "git-pull-script" failed due to the content merge (I didn't trust 
the merge-cache stuff enough to put that into it), but then doing the 
automated merge was successful without any editing, so I just wrote the 
tree and committed it.

Again, if anybody wants to reproduce this, you'll need to then do the

	checkout-cache -f -a
	update-cace --refresh

afterwards to make your working area match the merged tree.

> Linus, the rc-fixes repo is ready for applying ... it's the same one I
> announced on linux-scsi and lkml a while ago just with the git date
> information updated to be correct (the misc one should wait until after
> 2.6.12 is final).

Ok. Can you verify? I did a "git diff" between your old head and my new
head, and it did not show any SCSI files (only the expected arm etc stuff
that you didn't have in your), so it all _looks_ good. But hey, just to
make sure that I didn't do anything stupid..

		Linus

Re: SCSI trees, merges and git status

[James Bottomley]

On Mon, 2005-04-18 at 14:39 -0700, Linus Torvalds wrote:
> > Linus, the rc-fixes repo is ready for applying ... it's the same one I
> > announced on linux-scsi and lkml a while ago just with the git date
> > information updated to be correct (the misc one should wait until after
> > 2.6.12 is final).
> 
> Ok. Can you verify? I did a "git diff" between your old head and my new
> head, and it did not show any SCSI files (only the expected arm etc stuff
> that you didn't have in your), so it all _looks_ good. But hey, just to
> make sure that I didn't do anything stupid..

Actually, the verify fails, according to bitkeeper.

It looks like the merge tree has contamination from the scsi-misc-2.6
tree ... possibly because the hosting system got the merged objects when
I pushed.

Could you strip it back and I'll check out the repos on www.parisc-
linux.org?

Thanks,

James

Re: SCSI trees, merges and git status

[Linus Torvalds]

On Mon, 18 Apr 2005, James Bottomley wrote:
> 
> It looks like the merge tree has contamination from the scsi-misc-2.6
> tree ... possibly because the hosting system got the merged objects when
> I pushed.

Nope, the way I merge, if I get a few objects it shouldn't matter at all. 
I'll just look at your HEAD, and merge with the objects that represents.

Afterwards, if I have extra objects, I'll see them with fsck-cache. 

> Could you strip it back and I'll check out the repos on www.parisc-
> linux.org?

Git does work like BK in the way that you cannot remove history when you
have distributed it. Once it's there, it's there.

The patches from you I have in my tree are:

	scsi: add DID_REQUEUE to the error handling
	zfcp: add point-2-point support
	[PATCH] Convert i2o to compat_ioctl
	[PATCH] kill old EH constants
	[PATCH] scsi: remove meaningless scsi_cmnd->serial_number_at_timeout field
	[PATCH] scsi: remove unused scsi_cmnd->internal_timeout field
	[PATCH] remove outdated print_* functions
	[PATCH] consolidate timeout defintions in scsi.h

or at least that's what they claim in their changelogs.

Oh, and here's the diffstat that matches "scsi":

 drivers/block/scsi_ioctl.c                 |    5 -
 drivers/s390/scsi/zfcp_aux.c               |    4 -
 drivers/s390/scsi/zfcp_def.h               |    5 +
 drivers/s390/scsi/zfcp_erp.c               |   20 +++++
 drivers/s390/scsi/zfcp_fsf.c               |   38 ++++++++--
 drivers/s390/scsi/zfcp_fsf.h               |    6 +
 drivers/s390/scsi/zfcp_sysfs_adapter.c     |    6 +
 drivers/scsi/53c7xx.c                      |   23 +++---
 drivers/scsi/BusLogic.c                    |    7 -
 drivers/scsi/NCR5380.c                     |    9 +-
 drivers/scsi/advansys.c                    |    7 -
 drivers/scsi/aha152x.c                     |   17 ++--
 drivers/scsi/arm/acornscsi.c               |    9 +-
 drivers/scsi/arm/fas216.c                  |    9 +-
 drivers/scsi/arm/scsi.h                    |    2 
 drivers/scsi/atari_NCR5380.c               |    9 +-
 drivers/scsi/constants.c                   |    2 
 drivers/scsi/ips.c                         |    7 -
 drivers/scsi/ncr53c8xx.c                   |   14 ---
 drivers/scsi/pci2000.c                     |    4 -
 drivers/scsi/qla2xxx/qla_dbg.c             |    6 -
 drivers/scsi/scsi.c                        |    5 -
 drivers/scsi/scsi.h                        |   43 -----------
 drivers/scsi/scsi_error.c                  |   11 ---
 drivers/scsi/scsi_ioctl.c                  |    5 -
 drivers/scsi/scsi_lib.c                    |    2 
 drivers/scsi/scsi_obsolete.h               |  106 -----------------------------
 drivers/scsi/scsi_priv.h                   |    5 -
 drivers/scsi/seagate.c                     |    5 -
 drivers/scsi/sg.c                          |    3 
 drivers/scsi/sun3_NCR5380.c                |    9 +-
 drivers/scsi/sym53c8xx_2/sym_glue.c        |    6 -
 drivers/scsi/ultrastor.c                   |    4 -

so it doesn't look like there's a _lot_ wrong. Send in a patch to revert 
anything that needs reverting..

		Linus

Re: SCSI trees, merges and git status

[David Woodhouse]

On Mon, 2005-04-18 at 17:03 -0700, Linus Torvalds wrote:
> Git does work like BK in the way that you cannot remove history when you
> have distributed it. Once it's there, it's there.

But older history can be pruned, and there's really no reason why an
http-based 'git pull' couldn't simply refrain from fetching commits
older than a certain threshold.

However, we can't _add_ the history if the current commits don't refer
to it. I really think we should take the imported git history and make
our 'current' tree refer to it -- even if just by having an appropriate
'parent' record in what is currently the oldest changeset in our tree;
the 2.6.12-rc2 import.

It doesn't matter that our oldest commit object refers to a nonexistent
parent, but that does allow us to import historical data if we _want_
to, and have it all work properly.

We should have the full historical git repo available within a day or
so, I believe. It would be really useful if we could make the current
trees refer back to that, instead of starting at 2.6.12-rc2.

-- 
dwmw2

Re: SCSI trees, merges and git status

[James Bottomley]

On Tue, 2005-04-19 at 10:10 +1000, David Woodhouse wrote:
> On Mon, 2005-04-18 at 17:03 -0700, Linus Torvalds wrote:
> > Git does work like BK in the way that you cannot remove history when you
> > have distributed it. Once it's there, it's there.
> 
> But older history can be pruned, and there's really no reason why an
> http-based 'git pull' couldn't simply refrain from fetching commits
> older than a certain threshold.

Yes, that's what I did to get back to the commit just before the merge:

fsck-cache --unreachable 54ff646c589dcc35182d01c5b557806759301aa3|awk
'/^unreachable /{print $2}'|sed 's:^\(..\):.git/objects/\1/:'|xargs rm

removes all the objects from the tree prior to the bogus commit---it's
based on your (Linus') git-prune-script.

James

Re: SCSI trees, merges and git status

[David Woodhouse]

On Mon, 2005-04-18 at 19:16 -0500, James Bottomley wrote:
> Yes, that's what I did to get back to the commit just before the
> merge:
> 
> fsck-cache --unreachable 54ff646c589dcc35182d01c5b557806759301aa3|awk
> '/^unreachable /{print $2}'|sed 's:^\(..\):.git/objects/\1/:'|xargs rm

I was actually digressing and talking about pruning ancient history
which _is_ theoretically reachable. It's not being 'undone'; it's just
being omitted from the current _working_ tree. The whole point is that
in a fully-populated tree the history _should_ be accessible all the way
back.

We're trying to get the older history available on kernel.org ASAP. The
blobs are rsyncing to ~dwmw2/git/kernel-tglx1; the trees and commit
objects will be coming soon. 

Theoretically all Linus actually needs in order to rebuild his current
tree is the sha1 hash of the final commit in that historical tree, which
corresponds to 2.6.12-rc2.

-- 
dwmw2

Re: SCSI trees, merges and git status

[James Bottomley]

On Mon, 2005-04-18 at 17:03 -0700, Linus Torvalds wrote:
> The patches from you I have in my tree are:
> 
> 	scsi: add DID_REQUEUE to the error handling
> 	zfcp: add point-2-point support
> 	[PATCH] Convert i2o to compat_ioctl
> 	[PATCH] kill old EH constants
> 	[PATCH] scsi: remove meaningless scsi_cmnd->serial_number_at_timeout field
> 	[PATCH] scsi: remove unused scsi_cmnd->internal_timeout field
> 	[PATCH] remove outdated print_* functions
> 	[PATCH] consolidate timeout defintions in scsi.h

Those are a subset of patches from my scsi-misc-2.6 tree .. that's the
problem.  The actual patches should be:

  o zfcp: convert to compat_ioctl
  o sg.c: update
  o updates for CFQ oops fix
  o finally fix 53c700 to use the generic iomem infrastructure
  o fix NMI lockup with CFQ scheduler

I've redone the scsi-rc-fixes-2.6 tree to remove all the contamination
and reset the head correctly.

I've verified that if I strip your tree back to 

54ff646c589dcc35182d01c5b557806759301aa3

and then do a 

git-pull-script rsync://www.parisc-linux.org/~jejb/scsi-rc-fixes-2.6.git

Then the git-pull... script actually does the merge and the resulting
tree checks out against BK

Sorry for the screw up.

James


> or at least that's what they claim in their changelogs.
> 
> Oh, and here's the diffstat that matches "scsi":
> 
>  drivers/block/scsi_ioctl.c                 |    5 -
>  drivers/s390/scsi/zfcp_aux.c               |    4 -
>  drivers/s390/scsi/zfcp_def.h               |    5 +
>  drivers/s390/scsi/zfcp_erp.c               |   20 +++++
>  drivers/s390/scsi/zfcp_fsf.c               |   38 ++++++++--
>  drivers/s390/scsi/zfcp_fsf.h               |    6 +
>  drivers/s390/scsi/zfcp_sysfs_adapter.c     |    6 +
>  drivers/scsi/53c7xx.c                      |   23 +++---
>  drivers/scsi/BusLogic.c                    |    7 -
>  drivers/scsi/NCR5380.c                     |    9 +-
>  drivers/scsi/advansys.c                    |    7 -
>  drivers/scsi/aha152x.c                     |   17 ++--
>  drivers/scsi/arm/acornscsi.c               |    9 +-
>  drivers/scsi/arm/fas216.c                  |    9 +-
>  drivers/scsi/arm/scsi.h                    |    2 
>  drivers/scsi/atari_NCR5380.c               |    9 +-
>  drivers/scsi/constants.c                   |    2 
>  drivers/scsi/ips.c                         |    7 -
>  drivers/scsi/ncr53c8xx.c                   |   14 ---
>  drivers/scsi/pci2000.c                     |    4 -
>  drivers/scsi/qla2xxx/qla_dbg.c             |    6 -
>  drivers/scsi/scsi.c                        |    5 -
>  drivers/scsi/scsi.h                        |   43 -----------
>  drivers/scsi/scsi_error.c                  |   11 ---
>  drivers/scsi/scsi_ioctl.c                  |    5 -
>  drivers/scsi/scsi_lib.c                    |    2 
>  drivers/scsi/scsi_obsolete.h               |  106 -----------------------------
>  drivers/scsi/scsi_priv.h                   |    5 -
>  drivers/scsi/seagate.c                     |    5 -
>  drivers/scsi/sg.c                          |    3 
>  drivers/scsi/sun3_NCR5380.c                |    9 +-
>  drivers/scsi/sym53c8xx_2/sym_glue.c        |    6 -
>  drivers/scsi/ultrastor.c                   |    4 -
> 
> so it doesn't look like there's a _lot_ wrong. Send in a patch to revert 
> anything that needs reverting..
> 
> 		Linus
> -
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: SCSI trees, merges and git status

[Linus Torvalds]

On Mon, 18 Apr 2005, James Bottomley wrote:
> 
> Then the git-pull... script actually does the merge and the resulting
> tree checks out against BK

So?

What do you intend to do with all the other stuff I've already put on top?

Yes, I can undo my tree, but my tree has had more stuff in it since I 
pulled from you, so not only will that confuse everybody who already got 
the up-to-date tree, it will also undo stuff that was correct.

In other words, HISTORY CANNOT BE UNDONE.

That's the rule, and it's a damn good one. It was the rule when we used
BK, and it's the rule now. The fact that you can undo your history in
_your_ tree doesn't change anything at all.

So I can merge with your new tree, but that won't actually help any: I'll 
just get a superset, the way you did things. 

The way to remove patches is to explicitly revert them (effectively
applying a reverse diff), but I'm wondering if it's worth it in this case. 
I looked at the patches I did get, and they didn't look horribly bad per 
se. Are they dangerous?

2.6.12 is some time away, if for no other reason than the fact that this 
SCM thing has obviously eaten two weeks of my time. So I'd be inclined to 
chalk this up as a "learning experience" with git, and just go forward.

		Linus

Re: SCSI trees, merges and git status

[James Bottomley]

On Mon, 2005-04-18 at 17:29 -0700, Linus Torvalds wrote:
> 2.6.12 is some time away, if for no other reason than the fact that this 
> SCM thing has obviously eaten two weeks of my time. So I'd be inclined to 
> chalk this up as a "learning experience" with git, and just go forward.

Fair enough.  If you pull from

rsync://www.parisc-linux.org/~jejb/scsi-misc-2.6.git

That will pull in the rest of my scsi-misc-2.6 tree (which includes all
of the rc fixes).  I've done a test pull and merge and checked the
resulting against BK, so hopefully there should be no more screw ups.

Doing this exposed two bugs in your merge script:

1) It doesn't like a completely new directory (the misc tree contains a
new drivers/scsi/lpfc)
2) the merge testing logic is wrong.  You only want to exit 1 if the
merge fails.

James

git-merge-one-file-script: bec009e2c37bacc9e6f9cad1cfa5fd56752c7bf1
--- a/git-merge-one-file-script
+++ b/git-merge-one-file-script
@@ -13,6 +13,11 @@
 # do any merges that migth change the tree layout
 #
 
+# if the directory is newly added in a branch, it might not exist
+# in the current tree
+dir=$(dirname "$4")
+mkdir -p "$dir"
+
 case "${1:-.}${2:-.}${3:-.}" in
 #
 # deleted in both, or deleted in one and unchanged in the other
@@ -40,7 +45,11 @@ case "${1:-.}${2:-.}${3:-.}" in
 	orig=$(unpack-file $1)
 	src1=$(unpack-file $2)
 	src2=$(unpack-file $3)
-	merge "$src2" "$orig" "$src1" || echo Leaving conflict merge in $src2 && exit 1
+	merge "$src2" "$orig" "$src1"
+	if [ $? -ne 0 ]; then
+		echo Leaving conflict merge in $src2
+		exit 1
+	fi
 	cp "$src2" "$4" && update-cache --add -- "$4" && exit 0
 	;;
 

Re: SCSI trees, merges and git status

[Linus Torvalds]

On Mon, 18 Apr 2005, James Bottomley wrote:
> 
> Fair enough.  If you pull from
> 
> rsync://www.parisc-linux.org/~jejb/scsi-misc-2.6.git

Thanks. Pulled and pushed out.

> Doing this exposed two bugs in your merge script:
> 
> 1) It doesn't like a completely new directory (the misc tree contains a
> new drivers/scsi/lpfc)
> 2) the merge testing logic is wrong.  You only want to exit 1 if the
> merge fails.

Applied.

		Linus

Re: SCSI trees, merges and git status

[Junio C Hamano]

>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:

LT> Merged. Here's the command line history:

LT> 	~/git/git-pull-script \
LT>        rsync://www.parisc-linux.org/~jejb/scsi-rc-fixes-2.6.git

Maybe it is just me, but I have this setup:

    $ /bin/ls -lF .git
    total 20
    -rw-rw-r--  1 junio src     41 Apr 18 16:03 HEAD
    -rw-rw-r--  1 junio junio   41 Apr 18 15:07 MERGE_HEAD
    -rw-------  1 junio src   2720 Apr 18 16:03 index
    lrwxrwxrwx  1 junio src     18 Apr 18 15:55 objects -> ../../.git/objects/

My point being that .git/objects is a symbolic link and shares
object database with somewhere else.

However the "Getting object database" part trashed this symlink
when I tried to pull from my other repo locally.  I am wondering
it the following might be a better alternative.  A possible
downside in this approach is that you would not pull .git/heads
and .git/tags (i.e. Pesky stuff) from the remote anymore.  Is it
a problem (I am also CC'ing Petr to hear his opinion on this).

If not, please apply.

[PATCH] Do not let rsync obliterate .git/object symbolic link.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---
git-pull-script: e27215d3978635558c63859495d97f8114b4ece3
--- a/git-pull-script
+++ b/git-pull-script
@@ -6,7 +6,7 @@
 merge_repo=$1
 
 echo "Getting object database"
-rsync -avz --ignore-existing $merge_repo/ .git/
+rsync -avz --ignore-existing $merge_repo/objects/. .git/objects/.
 
 echo "Getting remote head"
 rsync -avz $merge_repo/HEAD .git/MERGE_HEAD

Re: SCSI trees, merges and git status

[Petr Baudis]

Dear diary, on Tue, Apr 19, 2005 at 01:23:24AM CEST, I got a letter
where Junio C Hamano <junkio@cox.net> told me that...
> However the "Getting object database" part trashed this symlink
> when I tried to pull from my other repo locally.  I am wondering
> it the following might be a better alternative.  A possible
> downside in this approach is that you would not pull .git/heads
> and .git/tags (i.e. Pesky stuff) from the remote anymore.  Is it
> a problem (I am also CC'ing Petr to hear his opinion on this).

Getting tags is probably nice. You should definitively not get
.git/heads, though. Those are your private stuff mostly, and the HEAD
you "export" is .git/HEAD.

I'm thinking about this yet, since it might be useful to be able to
export multiple branches without needing to set up multiple rsync
URLs... you still don't want the heads/ directory en block, though.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor