Kernel nightly snapshots..

Kernel nightly snapshots..

[Linus Torvalds]

I forget who it is that used to do the nightly snapshots for the BK
kernels, but I _think_ it was David Woodhouse (every time I've blamed it
on somebody, I've blamed the wrong person, so I'm probably off on this one
too, but maybe I finally got it right).

I was wondering how to get that re-started.. It should be technically
pretty easy, except I realized that my tree doesn't even have plain 2.6.11
in it. But I just fixed that in the tree, since I need such a baseline 
myself for my next release..

Anyway, I just pushed out a kernel tree that contains a tag of a _tree_ 
that points to the tree at the point of 2.6.11. I also had to teach 
fsck-cache about the fact that you can give it any kind of object to start 
your references at, and to make fsck-cache happy, you need to

	git-fsck-cache --unreachable HEAD v2.6.11

to tell it that the kernel tree now has an unconnected tree (described by
the tag "v2.6.11-tree", and I made the appropriate entry for it in
.git/refs/tags).

I also updated git-prune-script to not remove these kinds of things.

With this, it should be trivial to create snapshots with

	git-diff-tree -p v2.6.11 HEAD

or similar.

		Linus

Re: Kernel nightly snapshots..

[H. Peter Anvin]

It used to be Jeff Garzik, but David Woodhouse volunteered to take this 
over a few days ago, so yesterday I chowned those directories to him.

	-hpa

Linus Torvalds wrote:
> I forget who it is that used to do the nightly snapshots for the BK
> kernels, but I _think_ it was David Woodhouse (every time I've blamed it
> on somebody, I've blamed the wrong person, so I'm probably off on this one
> too, but maybe I finally got it right).
> 
> I was wondering how to get that re-started.. It should be technically
> pretty easy, except I realized that my tree doesn't even have plain 2.6.11
> in it. But I just fixed that in the tree, since I need such a baseline 
> myself for my next release..
> 
> Anyway, I just pushed out a kernel tree that contains a tag of a _tree_ 
> that points to the tree at the point of 2.6.11. I also had to teach 
> fsck-cache about the fact that you can give it any kind of object to start 
> your references at, and to make fsck-cache happy, you need to
> 
> 	git-fsck-cache --unreachable HEAD v2.6.11
> 
> to tell it that the kernel tree now has an unconnected tree (described by
> the tag "v2.6.11-tree", and I made the appropriate entry for it in
> .git/refs/tags).
> 
> I also updated git-prune-script to not remove these kinds of things.
> 
> With this, it should be trivial to create snapshots with
> 
> 	git-diff-tree -p v2.6.11 HEAD
> 
> or similar.
> 
> 		Linus

Re: Kernel nightly snapshots..

[David Woodhouse]

On Wed, 2005-05-04 at 19:06 -0700, H. Peter Anvin wrote:
> It used to be Jeff Garzik, but David Woodhouse volunteered to take
> this over a few days ago, so yesterday I chowned those directories to
> him.

I still have a problem with tags....

hera /home/dwmw2/git/snapshot-2.6 $ cg-init /pub/scm/linux/kernel/git/torvalds/linux-2.6.git &> ../asd
hera /home/dwmw2/git/snapshot-2.6 $ cg-tag-ls
v2.6.11 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
v2.6.11-tree    5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
v2.6.12-rc2     9e734775f7c22d2f89943ad6c745571f1930105f
v2.6.12-rc3     0397236d43e48e821cce5bbe6a80a1a56bb7cc3a
hera /home/dwmw2/git/snapshot-2.6 $ git-cat-file -t 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a
.git/objects/03/97236d43e48e821cce5bbe6a80a1a56bb7cc3a: No such file or directory
fatal: git-cat-file 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a: bad file


-- 
dwmw2

Re: Kernel nightly snapshots..

[Linus Torvalds]

On Thu, 5 May 2005, David Woodhouse wrote:
> 
> hera /home/dwmw2/git/snapshot-2.6 $ cg-init /pub/scm/linux/kernel/git/torvalds/linux-2.6.git &> ../asd
> hera /home/dwmw2/git/snapshot-2.6 $ cg-tag-ls
> v2.6.11 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
> v2.6.11-tree    5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c
> v2.6.12-rc2     9e734775f7c22d2f89943ad6c745571f1930105f
> v2.6.12-rc3     0397236d43e48e821cce5bbe6a80a1a56bb7cc3a
> hera /home/dwmw2/git/snapshot-2.6 $ git-cat-file -t 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a
> .git/objects/03/97236d43e48e821cce5bbe6a80a1a56bb7cc3a: No such file or directory

Looks like cg uses git-http-pull instead of rsync, and doesn't download 
anything but the required objects. 

In which case you probably don't have the v2.6.11 tree either, in fact, 
since it's not required to get a working copy of HEAD.

If you fetch the _whole_ object database (with rsync), you should get 
them.

		Linus

Re: Kernel nightly snapshots..

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Thu, 2005-05-05 at 07:44 -0700, Linus Torvalds wrote:
> If you fetch the _whole_ object database (with rsync), you should get 
> them.

OK, I've changed my 'origin' to an rsync URL referring to the same
place, to make sure I get tags correctly in future. 2.6.12-rc3-git1 is
in the process of being built; if the attached script works and
continues working when invoked from cron, we might even see nightly
snapshots again as requested...

-- 
dwmw2

[-- Attachment #2: git-snapshot.sh --]
[-- Type: application/x-shellscript, Size: 1724 bytes --]

Re: Kernel nightly snapshots..

[David Woodhouse]

On Thu, 2005-05-05 at 16:10 +0100, David Woodhouse wrote:
> 2.6.12-rc3-git1 is in the process of being built; if the attached
> script works and continues working when invoked from cron, we might
> even see nightly snapshots again as requested...

Eep. That'll be a whole lot more useful if I do
	cg-diff -r $RELTREE:$CURTREE
instead of 
	cg-diff -r $CURTREE:$RELTREE
... but it should be OK now. This is the md5sum of the correct one:

21a0ee13c539e8b016682c1542bb9167  patch-2.6.12-rc3-git1.gz

-- 
dwmw2

Re: Kernel nightly snapshots..

[Linus Torvalds]

On Thu, 5 May 2005, David Woodhouse wrote:
> 
> Eep. That'll be a whole lot more useful if I do
> 	cg-diff -r $RELTREE:$CURTREE
> instead of 
> 	cg-diff -r $CURTREE:$RELTREE
> ... but it should be OK now. This is the md5sum of the correct one:
> 
> 21a0ee13c539e8b016682c1542bb9167  patch-2.6.12-rc3-git1.gz

Btw, if you want to, you could simplify and optimize things heavily by 
doign some very specific git internal mucking around, and using the new 
"-m" flag I added to "git-diff-cache".

The "-m" flag says that a file that isn't checked out will always match 
the cache, which means that you can create your magic diffs that modify 
the Makefile with something like this:

	export SNAP_TAG_DIRECTORY=/pub/linux/kernel/v2.6/snapshots/tags
	export BASE_DIRECTORY=/pub/scm/linux/kernel/git/torvalds/linux-2.6
	export SHA1_FILE_DIRECTORY=$BASE_DIRECTORY/objects

	RELNAME=`ls $BASE_DIRECTORY/refs/tags | tail -1`
	RELCOMM=`cat $BASE_DIRECTORY/refs/tags/$RELNAME`
	CURCOMM=`cat $BASE_DIRECTORY/HEAD`

	SNAPNAME=`ls $SNAP_TAG_DIRECTORY | grep -- "$RELNAME-git" | tail -1`
	if [ "$SNAPNAME" ]; then
		SNAPCOMM=`cat $SNAP_TAG_DIRECTORY/$SNAPNAME`
		[ "$SNAPCOMM" == "$CURCOMM" ] && exit 0
		OLDGITNUM=`echo $SNAPNAME | sed s/^.*-git//`
		NEWGITNUM=`expr $OLDGITNUM + 1`
		CURNAME="$RELNAME-git$NEWGITNUM"
	else
		CURNAME="$RELNAME-git1"
	fi

	EXTRAVERSION=-`echo $CURNAME | cut -f2- -d-`

	rm -rf tmp-empty-tree
	mkdir -p tmp-empty-tree/.git
	( cd tmp-empty-tree
	  git-read-tree $CURCOMM
	  git-checkout-cache Makefile
	  perl -pi -e "s/EXTRAVERSION =.*/EXTRAVERSION = $EXTRAVERSION/" Makefile
	  git-diff-cache -m -p $RELCOMM ) | gzip -9 > patch-$CURNAME.gz

	echo CURCOMM > $SNAP_TAG_DIRECTORY/$CURNAME

but it's entirely untested, and only written in my mail editor, so..

Anyway, here's explaining what it does:

 - it creates a new "tmp-empty-tree" thing, and points the object database 
   to _my_ tree (which is unwritable for you, but you don't care, you just
   want to read it)

 - it picks up my last tag as the release, and your last git tag from your 
   private snapshot tag directory as the last git thing. Then it takes my 
   current HEAD as the new thing to diff.

 - we read that current tree into this empty tree, but we don't check any
   of it out except for the Makefile. So the "tmp-empty-tree" should 
   literally just have the files "Makefile" and ".git/index" in it.

 - we edit the Makefile, and then we do "git-diff-cache -m -p" between the 
   result and the old base.

This _should_ work. Modulo bugs. And it should be reasonably efficient. It 
just needs my current git tree as of five minutes ago..

		Linus

Re: Kernel nightly snapshots..

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Thu, 2005-05-05 at 09:40 -0700, Linus Torvalds wrote:
> Btw, if you want to, you could simplify and optimize things heavily by 
> doign some very specific git internal mucking around, and using the new 
> "-m" flag I added to "git-diff-cache".

OK, this is in place now. I also made sure it was going to treat a real
'2.6.12' release as newer than '2.6.12-rcX'. The selection of the tags
and the output of the log are still chronological, which is of course
wrong -- but it'll do for now.

As an added bonus I've also fixed the script to refrain from creating a
-git1 snapshot even when there have been no commits since the last
release. And deleted 2.6.12-rc4-git1.

-- 
dwmw2

[-- Attachment #2: git-snapshot.sh --]
[-- Type: application/x-shellscript, Size: 2499 bytes --]

Re: Kernel nightly snapshots..

[H. Peter Anvin]

David Woodhouse wrote:
> On Thu, 2005-05-05 at 16:10 +0100, David Woodhouse wrote:
> 
>>2.6.12-rc3-git1 is in the process of being built; if the attached
>>script works and continues working when invoked from cron, we might
>>even see nightly snapshots again as requested...
> 
> Eep. That'll be a whole lot more useful if I do
> 	cg-diff -r $RELTREE:$CURTREE
> instead of 
> 	cg-diff -r $CURTREE:$RELTREE
> ... but it should be OK now. This is the md5sum of the correct one:
> 
> 21a0ee13c539e8b016682c1542bb9167  patch-2.6.12-rc3-git1.gz
> 

Hi David,

Could you add that to 2.4 as well, too?

	-hpa

Re: Kernel nightly snapshots..

[David Woodhouse]

On Thu, 2005-05-05 at 17:28 -0700, H. Peter Anvin wrote:
> Could you add that to 2.4 as well, too?

Is there a 2.4 git tree?

-- 
dwmw2

Re: Kernel nightly snapshots..

[H. Peter Anvin]

David Woodhouse wrote:
> On Thu, 2005-05-05 at 17:28 -0700, H. Peter Anvin wrote:
> 
>>Could you add that to 2.4 as well, too?
> 
> Is there a 2.4 git tree?
> 

I thought so, but now I can't find it.  Marcelo?

	-hpa

Re: Kernel nightly snapshots..

[Marcelo Tosatti]

On Fri, May 06, 2005 at 02:50:17PM -0700, H. Peter Anvin wrote:
> David Woodhouse wrote:
> >On Thu, 2005-05-05 at 17:28 -0700, H. Peter Anvin wrote:
> >
> >>Could you add that to 2.4 as well, too?
> >
> >Is there a 2.4 git tree?
> >
> 
> I thought so, but now I can't find it.  Marcelo?

Not yet. I should be able to put a "beta" version 
next week.

David, can I bug you to add nightly snapshotting 
as soon as the v2.4 git tree is created ? 

I hope there aren't many further changes in v2.4.x...

Re: Kernel nightly snapshots..

[Thomas Glanzmann]

Hello,

> Not yet. I should be able to put a "beta" version 
> next week.

do you have a timeframe? A friend of mine just asked me were to find the
tree he wants to merge his jornada handheld pda stuff.

	Thomas

Re: Kernel nightly snapshots..

[David Woodhouse]

On Sun, 2005-05-22 at 13:50 +0200, Thomas Glanzmann wrote:
> do you have a timeframe? A friend of mine just asked me were to find the
> tree he wants to merge his jornada handheld pda stuff.

That should be done against the 2.6 kernel, and that tree is already
available.

-- 
dwmw2

Re: Kernel nightly snapshots..

[Thomas Glanzmann]

Hello,

> That should be done against the 2.6 kernel, and that tree is already
> available.

it is a shame that 2.6 doesn't even boot on this machine[1]. And even if
the patch doesn't do its way upstream its good to have something like
git available to to vendortracking.

	Thomas

[1] http://bugzilla.kernel.org/show_bug.cgi?id=4415

Re: Kernel nightly snapshots..

[Russell King]

On Sun, May 22, 2005 at 02:09:54PM +0200, Thomas Glanzmann wrote:
> Hello,
> 
> > That should be done against the 2.6 kernel, and that tree is already
> > available.
> 
> it is a shame that 2.6 doesn't even boot on this machine[1]. And even if
> the patch doesn't do its way upstream its good to have something like
> git available to to vendortracking.
> 
> 	Thomas
> 
> [1] http://bugzilla.kernel.org/show_bug.cgi?id=4415

This bugzilla entry unfortunately gives me two problems:

1. I don't know what a 2.6.x-hh kernel is, or what modifications there
   are from the 2.6 mainline kernel.

2. I don't have visibility of this jornada list, nor what diagnosis
   has been done so far.

It's also worth pointing out that the patch there is for a 2.4 kernel,
and therefore probably isn't suitable for 2.6 as is.

What I can say is that there's been very little flow of patches into
mainline for Jornada 7xx - the last attempt was back in January 2002.
Many of the problems from that time stemmed from me having a separate
tree, which caused the expectation that I would handle integration of
_anything_.  Since the -rmk tree is no longer a public offering, and
usable ARM support is merged into mainline, this expectation has been
removed.

What this does mean, however, is that from my point of view, Jornada
is a dead and unmaintained machine as far as 2.6 goes, and is ripe for
removal from the kernel source - the only reason it builds today is
because I've been tweaking it to keep it building.  Whether it actually
works or not is a completely different matter.

The way to stop it from being removed is for someone to step up and
maintain the thing in mainline.  How easy this would be is also
something I have no view on, since I've no idea what's outstanding,
nor what unmerged handhelds features Jornada 7xx depends on.

All in all, due to the above, I think submitting bugs for -hh kernels
to the mainline bugzilla is a worthless exercise unless they can be
proven to be relevant to mainline kernels.

Sorry.

-- 
Russell King

Re: Kernel nightly snapshots..

[Linus Torvalds]

On Thu, 5 May 2005, David Woodhouse wrote:
> 
> OK, I've changed my 'origin' to an rsync URL referring to the same
> place, to make sure I get tags correctly in future. 2.6.12-rc3-git1 is
> in the process of being built; if the attached script works and
> continues working when invoked from cron, we might even see nightly
> snapshots again as requested...

	RELNAME=`cg-tag-ls | grep -v 'git' | tail -n1 | cut -f1 | sed s/^v//`

this seems to assume that cg-tag-ls outputs things in the right order, 
which I'm not at all sure it does.

It looks like cg-tag-ls just does

	cd .git/refs/tags
	for tag in *; do
		...

which means that the ordering will depend on the shell ordering or 
globbing.

Bash does globbing alphabetically sorted (and documents that), and it's
quite possible that all other shells do too. Even so, that doesn't
actually mean that it would be sorted by release, since v2.6.2 will sort
_after_ v2.6.12.

So this _should_ work for a while, but will eventually (before a -rc10
happens or we get to v2.6.100 ;^) need something better. That something
better probably being to ignore any tags that don't point to commits, and
then sorting by the date of the commit object.

		Linus

Re: Kernel nightly snapshots..

[David Woodhouse]

On Thu, 2005-05-05 at 08:46 -0700, Linus Torvalds wrote:
> So this _should_ work for a while, but will eventually (before a -rc10
> happens or we get to v2.6.100 ;^) need something better. That something
> better probably being to ignore any tags that don't point to commits, and
> then sorting by the date of the commit object.

Enough with the dates already. I don't want to be doing snapshots
against 2.6.11.9 just because that happened to get tagged more recently
than 2.6.12-rc4. Time is an illusion; especially so when looking at
branches which may not even have been merged.

The tag I want is the one which refers to a commit which is made by
yourself and is accessible from the current HEAD in the lowest number of
steps. And I don't _care_ if you tagged it 'Woozy Numbat'.

-- 
dwmw2