git fsck messages - what to do about it?

git fsck messages - what to do about it?

[Wolfgang Denk]

When running git fsck I get some messages:

-> git-fsck-cache --unreachable $(cat .git/HEAD .git/refs/heads/*)
unreachable commit 08cf4121ad704faa5ae7c7ae7b3dca4476ef2770
unreachable tree 29a4c18afd958cc6cee938f0fe7fa59764418097
unreachable tree 3b722a7278e02b272ddc8aefbc7be22703d8a94f
unreachable tree 4ebc50d240d99b4236952d13d5044691f9078e3c
unreachable tree 63af5944e130b449338cc907ee77c51f4a69e339
unreachable tree 79c9a6bb02a80df15c7d829bdac0901f88c5e301
unreachable blob 85dcf4b5c4c0f6833cd63fa1d0fd29ef250162af
unreachable commit c1b779fbce762c970e3ace4332bb6ddc0de65e61
unreachable blob d70d1d44f12db7b2a3976c2a5313c90568a62c25
unreachable commit e1bcd2a5d53bceef2679737474487e32d87e7f2c

I have to admit that I don't know what to do next...

a) I understand that these  messages  are  indications  of  problems,
   right? What can I do to clean this up?

b) Is there any way to find out what I did wrong in the first place?

In case it matters: the tree in question is available at
{rsync,http}://source.denx.net/git/u-boot.git

Thanks in advance,

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"This isn't brain surgery; it's just television."   - David Letterman

Re: git fsck messages - what to do about it?

[Catalin Marinas]

Wolfgang Denk <wd@denx.de> wrote:
> -> git-fsck-cache --unreachable $(cat .git/HEAD .git/refs/heads/*)
> unreachable commit 08cf4121ad704faa5ae7c7ae7b3dca4476ef2770
> unreachable tree 29a4c18afd958cc6cee938f0fe7fa59764418097
> unreachable tree 3b722a7278e02b272ddc8aefbc7be22703d8a94f
> unreachable tree 4ebc50d240d99b4236952d13d5044691f9078e3c
> unreachable tree 63af5944e130b449338cc907ee77c51f4a69e339
> unreachable tree 79c9a6bb02a80df15c7d829bdac0901f88c5e301
> unreachable blob 85dcf4b5c4c0f6833cd63fa1d0fd29ef250162af
> unreachable commit c1b779fbce762c970e3ace4332bb6ddc0de65e61
> unreachable blob d70d1d44f12db7b2a3976c2a5313c90568a62c25
> unreachable commit e1bcd2a5d53bceef2679737474487e32d87e7f2c
>
> I have to admit that I don't know what to do next...
>
> a) I understand that these  messages  are  indications  of  problems,
>    right? What can I do to clean this up?
>
> b) Is there any way to find out what I did wrong in the first place?

Did you use StGIT on your tree? StGIT can leave unreachable
objects. Otherwise, git-prune should remove them.

Note that if you use StGIT, git-prune will also remove the commit
objects for the patches which are not pushed onto the stack. I will
add a 'stg prune' commands which takes care of these.

-- 
Catalin

Re: git fsck messages - what to do about it?

[Petr Baudis]

Dear diary, on Fri, Sep 16, 2005 at 12:11:38PM CEST, I got a letter
where Wolfgang Denk <wd@denx.de> told me that...
> When running git fsck I get some messages:
> 
> -> git-fsck-cache --unreachable $(cat .git/HEAD .git/refs/heads/*)
> unreachable commit 08cf4121ad704faa5ae7c7ae7b3dca4476ef2770
> unreachable tree 29a4c18afd958cc6cee938f0fe7fa59764418097
> unreachable tree 3b722a7278e02b272ddc8aefbc7be22703d8a94f
> unreachable tree 4ebc50d240d99b4236952d13d5044691f9078e3c
> unreachable tree 63af5944e130b449338cc907ee77c51f4a69e339
> unreachable tree 79c9a6bb02a80df15c7d829bdac0901f88c5e301
> unreachable blob 85dcf4b5c4c0f6833cd63fa1d0fd29ef250162af
> unreachable commit c1b779fbce762c970e3ace4332bb6ddc0de65e61
> unreachable blob d70d1d44f12db7b2a3976c2a5313c90568a62c25
> unreachable commit e1bcd2a5d53bceef2679737474487e32d87e7f2c
> 
> I have to admit that I don't know what to do next...
> 
> a) I understand that these  messages  are  indications  of  problems,
>    right? What can I do to clean this up?
> 
> b) Is there any way to find out what I did wrong in the first place?

In case you were using Cogito, cg-admin-uncommit will leave unreachable
objects in the database - it's safe to prune them (unless you'll decide
later that you after all wanted the uncommitted stuff, of course).

But note that IT IS NOT SAFE to prune repositories cloned with -l or
_being_ cloned with -l. Perhaps one solution to this would be having
.git/objects/info/repousers/ or something, which would contain symlinks
back to all the repositories using the given objects database. Opinions?

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
If you want the holes in your knowledge showing up try teaching
someone.  -- Alan Cox

Re: git fsck messages - what to do about it?

[Wolfgang Denk]

In message <tnx3bo59tm8.fsf@arm.com> you wrote:
>
> > -> git-fsck-cache --unreachable $(cat .git/HEAD .git/refs/heads/*)
> > unreachable commit 08cf4121ad704faa5ae7c7ae7b3dca4476ef2770
> > unreachable tree 29a4c18afd958cc6cee938f0fe7fa59764418097
> > unreachable tree 3b722a7278e02b272ddc8aefbc7be22703d8a94f
> > unreachable tree 4ebc50d240d99b4236952d13d5044691f9078e3c
> > unreachable tree 63af5944e130b449338cc907ee77c51f4a69e339
> > unreachable tree 79c9a6bb02a80df15c7d829bdac0901f88c5e301
> > unreachable blob 85dcf4b5c4c0f6833cd63fa1d0fd29ef250162af
> > unreachable commit c1b779fbce762c970e3ace4332bb6ddc0de65e61
> > unreachable blob d70d1d44f12db7b2a3976c2a5313c90568a62c25
> > unreachable commit e1bcd2a5d53bceef2679737474487e32d87e7f2c
> >
> > I have to admit that I don't know what to do next...
> >
> > a) I understand that these  messages  are  indications  of  problems,
> >    right? What can I do to clean this up?
> >
> > b) Is there any way to find out what I did wrong in the first place?
> 
> Did you use StGIT on your tree? StGIT can leave unreachable

No, I did not. I *think* these unreachable commits resulted from "git
resolve"s  which  resulted  in  merge  conflicts,  which  I  resolved
manually. After this, I ran "git commit" manually, but probably I did
something wrong. I just don't understand what this might have been...

> objects. Otherwise, git-prune should remove them.

Ummm.... I hesitate to  do  this  before  understanding  where  these
messages come from and what sort of data is affected.

> Note that if you use StGIT, git-prune will also remove the commit

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Remember that Beethoven wrote his first symphony in C ...

Re: git fsck messages - what to do about it?

[Junio C Hamano]

Wolfgang Denk <wd@denx.de> writes:

> Ummm.... I hesitate to  do  this  before  understanding  where  these
> messages come from and what sort of data is affected.

Does git-fsck-cache without --unreachable report dangling
commits?  If so I would do something like this (this is my
'git-lost-found' script):

        #!/bin/sh

        . git-sh-setup-script || die "Not a git archive."

        laf="$GIT_DIR/refs/lost-found"
        mkdir -p "$laf" || exit

        git fsck-cache |
        while read dangling type sha1
        do
                case "$dangling" in
                dangling)
                        echo "$sha1" >"$laf/$sha1"
                        case "$type" in
                        commit)
                                git-show-branch "$sha1" ;;
                        esac
                        ;;
                esac
        done

and run 'gitk --all' to see what these dangling commits are, and
where in your commit ancestry mesh they fit in.  

After you understand what they are, do not forget to "rm -f
$GIT_DIR/refs/lost-found/*"; you made them reachable just to
make inspecting them easier, but you probably do not want them.

I can sort of understand dangling blobs can come from manually
resolved merges, but I do not think that would create dangling
commits.

Have you rewound and rebased your history?  'git reset --hard'
obviously throws the tip of the current branch away and 'git
rebase' does it in order to recreate an alternate history.
If that is the case, and if you are satisfied with the history
your repository currently have, then obviously these can be
safely pruned.

Another possibility is what Pasky suggested -- do you have a
symlink .git/objects in another repository that points at the
.git/objects directory of this repository?  These dangling
things may be objects the other repository needs.  If that is
the case you obviously cannot prune them -- you would corrupt
the other repository.

Re: git fsck messages - what to do about it?

[Linus Torvalds]

On Fri, 16 Sep 2005, Wolfgang Denk wrote:
>
> When running git fsck I get some messages:
> 
> -> git-fsck-cache --unreachable $(cat .git/HEAD .git/refs/heads/*)

Btw, you shouldn't need to list the heads explicitly any more: if you 
don't give any heads, git-fsck-cache by default will pick up all of them. 
In fact, it will generally do a better job of it that you did, since it 
will look at _all_ the refs in .git/refs/.

The above is very old-fashioned ;)

> unreachable commit 08cf4121ad704faa5ae7c7ae7b3dca4476ef2770
> unreachable tree 29a4c18afd958cc6cee938f0fe7fa59764418097
> unreachable tree 3b722a7278e02b272ddc8aefbc7be22703d8a94f
> unreachable tree 4ebc50d240d99b4236952d13d5044691f9078e3c
> unreachable tree 63af5944e130b449338cc907ee77c51f4a69e339
> unreachable tree 79c9a6bb02a80df15c7d829bdac0901f88c5e301
> unreachable blob 85dcf4b5c4c0f6833cd63fa1d0fd29ef250162af
> unreachable commit c1b779fbce762c970e3ace4332bb6ddc0de65e61
> unreachable blob d70d1d44f12db7b2a3976c2a5313c90568a62c25
> unreachable commit e1bcd2a5d53bceef2679737474487e32d87e7f2c
> 
> I have to admit that I don't know what to do next...
> 
> a) I understand that these  messages  are  indications  of  problems,
>    right? What can I do to clean this up?

First off, not necessarily. Especially if you have mixed rsync and the 
"native git pack" transports, you may have packs that refer to unpacked 
objects, and then since git-fsck-cache by default doesn't look into packs 
(it assumes that they are fine and self-sufficient), you can get this 
error.

The same is true if you have "alternates" directories: git-fsck-cache 
won't do the reachability in the other directories by default.

If you add the "--full" flag, git-fsck-cache will also look at all those 
things.

Now, I assume that this isn't the case, and what you have is actually real 
unreachable objects. And that's perfectly normal too: it will happen if 
you ever "drop" a commit (undo) by resetting a branch to its parent. For 
example, if I notice I've made a mistake after a commit, I just do

	git reset HEAD^

which effectively undoes the commit. I can then fix things up and 
re-commit. And this is all _normal_.

And what it results in is that the commit objects that I undid are
obviously not reachable from anything else. git-fsck-cache will say that
they are unreachable, and "git prune" will get rid of them.

The same thing happens if you do "git-update-cache" on a file, but instead 
of committing that blob object, you realize that you need to edit it a bit 
more, and do _another_ git-update-cache and commit only the final blob. 
The first blob will be unreachable - nothing ends up pointing to it.

So an unreachable object is not necessarily an error at all. Git - very 
much on purpose - will leave the stale objects around, because it's too 
expensive to try to be exact, so it's much better to just let them be, and 
do an occasional "cleanup" with "git prune".

There are other reasons to have "unreachable" objects, even more benign. 

Any time you use "rsync" as the fetch protocol, you'll get _all_ the 
objects from the other end. Regardless of whether you actually fetch all 
the _refs_ from the other end. In particular, if the other end has 
multiple branches, and you only pull _one_ of them, you'll still get the 
objects for all the others. And git-fsck-cache will correctly say that 
those objects are unreachable.

(And _yet_ another reason for unreachable objects: you share the object 
directory with some other project - those objects will be unreachable from 
_your_ project, but maybe they're reachable from another one?)

> b) Is there any way to find out what I did wrong in the first place?

Well, it probably isn't even an error, but there are nice tools to check 
it out. For example, you can just list the commits it lists to gitk, and 
you'll get a nice graphical view of where they are in the tree. That 
probably makes you go "Duh, yes, I did a 'git reset' to undo those".

Or maybe you recognize that the commit was clearly part of a chain that
you hadn't even committed, but came from another tree that you had 
rsync'ed with.

And if that is the case, just ignore them. Or do "git prune" to keep your 
tree nice and clean, and to make "git-fsck-cache --unreachable" happy.

		Linus

Re: git fsck messages - what to do about it?

[Linus Torvalds]

On Fri, 16 Sep 2005, Linus Torvalds wrote:
> 
> > b) Is there any way to find out what I did wrong in the first place?
> 
> Well, it probably isn't even an error, but there are nice tools to check 
> it out. For example, you can just list the commits it lists to gitk, and 
> you'll get a nice graphical view of where they are in the tree. That 
> probably makes you go "Duh, yes, I did a 'git reset' to undo those".

Doing exactly that:

	gitk 08cf4121ad704faa5ae7c7ae7b3dca4476ef2770 \
		8493c04f1b8c9da0c5dc83091cabbcd0b98954ed \
		e1bcd2a5d53bceef2679737474487e32d87e7f2c \
		c1b779fbce762c970e3ace4332bb6ddc0de65e61

says that three of the commits were the same one, redone:

	Added support for mtddevnum and mtddevname variables.

in one form or another (quick look at the diffs says that one was 
apparently re-done because you forgot the changelog entry, I didn't look 
any closer).

So I suspect that's you just undoing a commit, and redoing it. I assume
that somewhere in the _reachable_ tree is one final commit that actually
did that, but I didn't bother to search for it (but that's easy too if you
want to:just go "gitk --all" and then search for that "Added support for
mtddevnum"...)

The fourth commit is 8493c

	Rebased: Rewrite of NAND code based on what is in 2.6.12 Linux kernel

and considering that you seem to have other undo's in the tree, I'll just
assume this is one more, but you would know.

		Linus

Re: git fsck messages - what to do about it?

[Horst von Brand]

Wolfgang Denk <wd@denx.de> wrote:
> When running git fsck I get some messages:
> 
> -> git-fsck-cache --unreachable $(cat .git/HEAD .git/refs/heads/*)
> unreachable commit 08cf4121ad704faa5ae7c7ae7b3dca4476ef2770
> unreachable tree 29a4c18afd958cc6cee938f0fe7fa59764418097

[...]

> I have to admit that I don't know what to do next...
> 
> a) I understand that these  messages  are  indications  of  problems,
>    right? What can I do to clean this up?

One of the nits I'd like to pick is that git-fsck does only check, not
(like its filesystem counterparts) fix the problems it finds.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: git fsck messages - what to do about it?

[Linus Torvalds]

On Fri, 16 Sep 2005, Linus Torvalds wrote:
>
> > b) Is there any way to find out what I did wrong in the first place?
> 
> Well, it probably isn't even an error, but there are nice tools to check 
> it out. For example, you can just list the commits it lists to gitk, and 
> you'll get a nice graphical view of where they are in the tree. That 
> probably makes you go "Duh, yes, I did a 'git reset' to undo those".

Oh, btw, one final note on this problem.

The "--unreachable" flag is meant to show a list of all objects that 
aren't reachable from the refs (either the explicit ones, or the ones 
git-fsck-objects finds on its own). That's very useful for when you 
actually want to remove them, ie "git prune" uses that flag.

However, it's often not very interesting to see the full list of 
unreachable objects, because you are probably _not_ interested in the 
trees that are reachable from the unreachable commit - you're really 
interested in the "tips of unreachability".

And that's what git-fsck-object actually reports _without_ any flags, as 
"dangling" objects. 

So let's say that you have one commit that you've undone.

If you don't give any flags at all, git-fsck-objects will report

	dangling commit ...

but if you use "--unreachable" then you'll get the full list of objects, 
so you'll get

	unreachable tree ...
	unreachable blob ...
	unreachable commit ...
	unreachable blob ...

which is usually _not_ what you want.

Similarly, let's say that you undid a series of three commits. There is 
only one "dangling" commit (the "tip" of the series), but there are three 
unreachable commits (and any number of unreachable trees/blobs associated 
with those commits). Again, what a human tends to be interested in is the 
"dangling tip", not the whole collection of unreachable objects that can 
be reached through that tip.

So using "--unreachable" gives much more raw data, but it usually tends to
hide the real _information_, and the end result is thus much harder to
understand.

			Linus

Re: git fsck messages - what to do about it?

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> So using "--unreachable" gives much more raw data, but it usually tends to
> hide the real _information_, and the end result is thus much harder to
> understand.

Yup.  That's exactly why the 'git-lost-found' sample script
given in <7vll1xndxi.fsf@assigned-by-dhcp.cox.net> runs
git-fsck-objects output without --unreachable.