egit problem with sym linked eclipse project dirs

egit problem with sym linked eclipse project dirs

[Stephen Bannasch]

To hack around Eclipse's problem with projects in nested directories 
I clone a repo and then create a new top-level dir that just has 
symbolic links to all the project directories.

However when I Team/Share with git using egit the resources become 
untracked when displayed in Eclipse -- they still are tracked however 
when I check from the command line.

---- more details ----

using v0.4.0.200901041117 of egit

I've got a large dir structure in a git repo in which there are many 
eclipse projects at different levels of the tree.

However Eclipse doesn't work with projects at different levels in a 
nested dir structure (bad Eclipse) -- see: Eclipse bug: 
https://bugs.eclipse.org/bugs/show_bug.cgi?id=35973

When we work with these projects with svn we checkout each one 
individually into it's own dir to create the flat hierarchy Eclipse 
needs.  Updating all these separate projects takes forever -- at 
least compared to using Git.

I created a Ruby script to create a new dir named 'workspace' at the 
top level of a checkout with symbolic links to all the enclosed 
directories that have a .project file inside them.

http://svn.concord.org/svn/projects/trunk/common/flatten_projects.rb

or here in the git mirror:

http://gitorious.org/projects/otrunk-examples/repos/mainline/blobs/raw/5c5ca6ff830f4ad3e4d427df92c1a5dff64831b5/flatten_projects.rb

(warning: repos are public but very big)

The result is a dir: workspace/ that emulates the flat project space 
that Eclipse needs:

   http://gist.github.com/4658 (a bunch of these are legacy projects)

Then I open Eclipse, switch to a new Workspace, select the 
newly-created 'workspace' dir and import the projects to work on.

This works fine for Eclipse. The projects build and reference each other.

But when I then turn on the team/sharing/git for each project ...

The operation completes -- the git decorator appears showing [Git @ master] ...

but the files are all untracked.

Screenshots showing the sequence after importing just 7 projects:

   apple-support, framework, frameworkview, otrunk, swing, thirdparty-jars

share projects:
   http://skitch.com/stepheneb/bbnqf/skitched-20090113-144412

select git:
   http://img.skitch.com/20090113-peu5y38a1yqm4kfqasf6hh9r9t.png

iterate through each project:
   http://img.skitch.com/20090113-k6wf8xgs2bc4ehy9siscdyteqr.png
   http://img.skitch.com/20090113-jughm7eatnuum43gfuedmctrw5.png

showing untracked files on master branch:
   http://skitch.com/stepheneb/bbnqc/skitched-20090113-144646

However from the command line git shows the appropriate untracked 
files after an Eclipse build:

$ git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	java/core/apple-support/bin/
#	java/core/framework/lib/
#	java/core/frameworkview/lib/
#	java/core/swing/lib/
#	java/otrunk/otrunk/lib/
#	java/thirdparty/phet-simulations-java/simulations/cck/cck
#	rails/portal/portal
#	workspace/

Re: egit problem with sym linked eclipse project dirs

[Robin Rosenberg]

> To hack around Eclipse's problem with projects in nested directories 
> I clone a repo and then create a new top-level dir that just has 
> symbolic links to all the project directories.
> 
> However when I Team/Share with git using egit the resources become 
> untracked when displayed in Eclipse -- they still are tracked however 
> when I check from the command line.

I've seen this too, but so far not got myself to actually fix it. I think it has do with
that we ask eclipse project for it's path end there we gets off track. Thanks
for providing a good example that help "someone" to fix it. 

Could you add an issue at http://code.google.com/p/egit/issues/list and maybe
include a tar  ball with a simple project displaying this behaviour? (with a note
that the symlinks makes this impossible to recreate on Windows).

-- robin

Re: egit problem with sym linked eclipse project dirs

[Stephen Bannasch]

At 10:37 PM +0100 1/13/09, Robin Rosenberg wrote:
>  > To hack around Eclipse's problem with projects in nested directories
>>  I clone a repo and then create a new top-level dir that just has
>>  symbolic links to all the project directories.
>>
>>  However when I Team/Share with git using egit the resources become
>>  untracked when displayed in Eclipse -- they still are tracked however
>>  when I check from the command line.
>
>I've seen this too, but so far not got myself to actually fix it. I 
>think it has do with
>that we ask eclipse project for it's path end there we gets off track. Thanks
>for providing a good example that help "someone" to fix it.
>
>Could you add an issue at http://code.google.com/p/egit/issues/list and maybe
>include a tar  ball with a simple project displaying this behaviour? 
>(with a note
>that the symlinks makes this impossible to recreate on Windows).

Done: http://code.google.com/p/egit/issues/detail?id=52

Fxing this will make using Eclipse and Git together SO much nicer!

Re: egit problem with sym linked eclipse project dirs

[Stephen Bannasch]

At 6:27 PM -0500 1/13/09, Stephen Bannasch wrote:
>At 10:37 PM +0100 1/13/09, Robin Rosenberg wrote:
>>  > To hack around Eclipse's problem with projects in nested directories
>>>  I clone a repo and then create a new top-level dir that just has
>>>  symbolic links to all the project directories.
>>>
>>>  However when I Team/Share with git using egit the resources become
>>>  untracked when displayed in Eclipse -- they still are tracked however
>>>  when I check from the command line.
>>
>>I've seen this too, but so far not got myself to actually fix it. I 
>>think it has do with
>>that we ask eclipse project for it's path end there we gets off track. Thanks
>>for providing a good example that help "someone" to fix it.
>>
>>Could you add an issue at http://code.google.com/p/egit/issues/list and maybe
>>include a tar  ball with a simple project displaying this 
>>behaviour? (with a note
>>that the symlinks makes this impossible to recreate on Windows).
>
>Done: http://code.google.com/p/egit/issues/detail?id=52
>
>Fxing this will make using Eclipse and Git together SO much nicer!

Robin and Shawn,

Can you suggest where in the egit code I should look to see if I can fix this?

Re: egit problem with sym linked eclipse project dirs

[Robin Rosenberg]

fredag 23 januari 2009 02:56:16 skrev Stephen Bannasch:
> At 6:27 PM -0500 1/13/09, Stephen Bannasch wrote:
...
> >Done: http://code.google.com/p/egit/issues/detail?id=52
> >
> >Fxing this will make using Eclipse and Git together SO much nicer!
> 
> Robin and Shawn,
> 
> Can you suggest where in the egit code I should look to see if I can fix this?

There may be more than one place. 

Take a look at GitResourceDecorator. 

We map from resources using code like this:
	RepositoryMapping.getMapping(rsrc) // <-- prime suspect
	Repository.getRepoRelativePath(rsrc)
	Repository.getWorkDir
	Repository.isResourceChanged

I'm not entirely sure this is the correct way, but it probably has no
bearing on your problem.

	GitResourceDecorator decorator = getActiveDecorator();

It would be nice with a unit test for this. (Not necessarily the decorator,
but the methods used for it).

-- robin

[EGIT RFC PATCH(was Re: egit problem with sym linked eclipse project dirs)] Add some support for symlinked projects.

[Robin Rosenberg]

If a project is not in therepository via the path we found it,  we look for its canonical path and
see if that path is in the repository.

We only do this lookup at the project level. Odd linking at resource level
will not be recognized because this lookup is expensive.
---
 .../egit/core/project/RepositoryMapping.java       |   54 ++++++++++++++++++-
 1 files changed, 51 insertions(+), 3 deletions(-)

fredag 23 januari 2009 22:33:59 skrev Robin Rosenberg <robin.rosenberg.lists@dewire.com>:
> fredag 23 januari 2009 02:56:16 skrev Stephen Bannasch:
> > At 6:27 PM -0500 1/13/09, Stephen Bannasch wrote:
> ...
> > >Done: http://code.google.com/p/egit/issues/detail?id=52
> > >
> > >Fxing this will make using Eclipse and Git together SO much nicer!

Since I started looking at this, I may as well post what I came up with before
I lost interest in the issue.

This patch solved some of the problem. I'm not sure it doesn't cause more
problem than it solves, e.g. should the solution support more fine grained
handling of links, be faster or.. 

Shawn: don't  push in this state. 
Stephen: come up with a test suite and pro's cons, potholes etc.

-- robin

diff --git a/org.spearce.egit.core/src/org/spearce/egit/core/project/RepositoryMapping.java b/org.spearce.egit.core/src/org/spearce/egit/core/project/RepositoryMapping.java
index e9df630..f2a0977 100644
--- a/org.spearce.egit.core/src/org/spearce/egit/core/project/RepositoryMapping.java
+++ b/org.spearce.egit.core/src/org/spearce/egit/core/project/RepositoryMapping.java
@@ -23,6 +23,7 @@
 import org.eclipse.core.runtime.IPath;
 import org.eclipse.core.runtime.Path;
 import org.eclipse.team.core.RepositoryProvider;
+import org.spearce.egit.core.Activator;
 import org.spearce.egit.core.GitProvider;
 import org.spearce.jgit.lib.Constants;
 import org.spearce.jgit.lib.GitIndex;
@@ -49,6 +50,10 @@ static boolean isInitialKey(final String key) {
 
 	private IContainer container;
 
+	private String replace;
+
+	private String replaceWith;
+
 	/**
 	 * Construct a {@link RepositoryMapping} for a previously connected project.
 	 *
@@ -97,6 +102,45 @@ public RepositoryMapping(final IContainer mappedContainer, final File gitDir) {
 		}
 	}
 
+	private void checkIsLinked() {
+		File containerFile = container.getLocation().toFile();
+		String absoluteContainer = containerFile.getAbsoluteFile().getPath();
+		String canonicalContainer;
+		try {
+			canonicalContainer = containerFile.getCanonicalFile().getPath();
+			if (!absoluteContainer.equals(canonicalContainer)) {
+				int ic = canonicalContainer.length() - 1;
+				int ac = absoluteContainer.length() - 1;
+				char lastCh = File.separatorChar;
+				while (ic >= 0 && ac >= 0) {
+					char curchar = canonicalContainer.charAt(ic);
+					if (curchar != absoluteContainer.charAt(ac)) {
+						if (lastCh == File.separatorChar) {
+							String linkPointSrc = absoluteContainer.substring(
+									0, ac + 1);
+							String linkPointDest = canonicalContainer
+									.substring(workdirPrefix.length(), ic + 1);
+							replace = linkPointSrc.substring(
+									workdirPrefix.length()).replace(
+									File.separatorChar, '/');
+							replaceWith = linkPointDest.replace(
+									File.separatorChar, '/');
+							break;
+						}
+					}
+					lastCh = curchar;
+					ic--;
+					ac--;
+				}
+				if (ic == 0 || ac == 0)
+					Activator.logError(String.format("Cannot find the canonical location of %s within the repo %s", containerFile.toString(), gitdirPath), null);
+
+			}
+		} catch (IOException e) {
+			Activator.logError("Cannot get canonical path for container " + container.toString(), e);
+		}
+	}
+
 	IPath getContainerPath() {
 		return Path.fromPortableString(containerPath);
 	}
@@ -136,6 +180,7 @@ synchronized void setRepository(final Repository r) {
 		workdirPrefix = workdirPrefix.replace('\\', '/');
 		if (!workdirPrefix.endsWith("/"))
 			workdirPrefix += "/";
+		checkIsLinked();
 	}
 
 	/**
@@ -209,9 +254,12 @@ public String getRepoRelativePath(final IResource rsrc) {
 		final int pfxLen = workdirPrefix.length();
 		final String p = rsrc.getLocation().toString();
 		final int pLen = p.length();
-		if (pLen > pfxLen)
+		if (pLen > pfxLen) {
+			if (replace != null) {
+				return replaceWith + "/" + p.substring(pfxLen + replace.length() + 1);
+			}
 			return p.substring(pfxLen);
-		else if (p.length() == pfxLen - 1)
+		} else if (p.length() == pfxLen - 1)
 			return "";
 		return null;
 	}
-- 
1.6.1.285.g35d8b

Re: [EGIT RFC PATCH(was Re: egit problem with sym linked eclipse project dirs)] Add some support for symlinked projects.

[Stephen Bannasch]

OK ... I'm a bit confused now because I no longer have Git listed as 
a repository type for Team Sharing.

I deleted the existing org.spearce.* eclipse plugins

   [eclipse]$ rm -f plugins/org.spearce.*

pulled from git://repo.or.cz/egit.git

created a new branch for this test and applied your patch

   [egit.git (linksbranch)]$ git log
   commit 2b86cd4e27a9d9158092305271d6fb25ab27846e
   Author: Stephen Bannasch <stephen.bannasch@gmail.com>
   Date:   Wed Mar 11 22:29:26 2009 -0400

       check through links for repositories
       Robin Rosenberg's patch
       see: http://marc.info/?l=git&m=123681033214178&w=2
       and: http://code.google.com/p/egit/issues/detail?id=52

   commit 341b9c1abadd2ac0ec9ecc7c597990070612e058
   Author: Ruth Alkema <ruth@diasoft.nl>
   Date:   Wed Mar 11 16:42:45 2009 +0100

       Better fix for 'negative position' error on large pack files

       If position is really big and the window size is fairly small, id can
       be negative.  Instead of producing a negative position by extending
       id to a long, mask out the low bits from the original position value
       to compute the window start.

       [sp: message by me, as I had already applied v1 of the patch
            and pushed it (see immediate parent)]

       Signed-off-by: Ruth Alkema <ruth@diasoft.nl>
       Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

   ...

Opened Eclipse and switched to the egit workspace. Refreshed the egit 
projects and exported the org.spearce.* packages as deployable 
plugins and fragments to the Eclipse dir.

The new plugins are there:

   [eclipse]$ ls plugins/org.spearce.*
   plugins/org.spearce.egit.core.test_0.4.0.200903112237.jar
   plugins/org.spearce.egit_0.4.0.200903112237.jar
   plugins/org.spearce.egit.core_0.4.0.200903112237.jar
   plugins/org.spearce.jgit_0.4.0.200903112237.jar
   plugins/org.spearce.egit.ui_0.4.0.200903112237.jar

Quit and restarted Eclipse.

When I select a project with an existing git repository and try to 
enable team/sharing only CVS and SVN are listed.

Re: [EGIT RFC PATCH(was Re: egit problem with sym linked eclipse project dirs)] Add some support for symlinked projects.

[Robin Rosenberg]

torsdag 12 mars 2009 03:57:09 skrev Stephen Bannasch <stephen.bannasch@deanbrook.org>:
> OK ... I'm a bit confused now because I no longer have Git listed as 
> a repository type for Team Sharing.
> 
> I deleted the existing org.spearce.* eclipse plugins
> 
>    [eclipse]$ rm -f plugins/org.spearce.*
> 
> pulled from git://repo.or.cz/egit.git
> 
> created a new branch for this test and applied your patch
> 
>    [egit.git (linksbranch)]$ git log
>    commit 2b86cd4e27a9d9158092305271d6fb25ab27846e
>    Author: Stephen Bannasch <stephen.bannasch@gmail.com>
>    Date:   Wed Mar 11 22:29:26 2009 -0400
> 
>        check through links for repositories
>        Robin Rosenberg's patch
>        see: http://marc.info/?l=git&m=123681033214178&w=2
>        and: http://code.google.com/p/egit/issues/detail?id=52
> 
>    commit 341b9c1abadd2ac0ec9ecc7c597990070612e058
>    Author: Ruth Alkema <ruth@diasoft.nl>
>    Date:   Wed Mar 11 16:42:45 2009 +0100
> 
>        Better fix for 'negative position' error on large pack files
> 
>        If position is really big and the window size is fairly small, id can
>        be negative.  Instead of producing a negative position by extending
>        id to a long, mask out the low bits from the original position value
>        to compute the window start.
> 
>        [sp: message by me, as I had already applied v1 of the patch
>             and pushed it (see immediate parent)]
> 
>        Signed-off-by: Ruth Alkema <ruth@diasoft.nl>
>        Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
> 
>    ...
> 
> Opened Eclipse and switched to the egit workspace. Refreshed the egit 
> projects and exported the org.spearce.* packages as deployable 
> plugins and fragments to the Eclipse dir.
> 
> The new plugins are there:
> 
>    [eclipse]$ ls plugins/org.spearce.*
>    plugins/org.spearce.egit.core.test_0.4.0.200903112237.jar
>    plugins/org.spearce.egit_0.4.0.200903112237.jar
>    plugins/org.spearce.egit.core_0.4.0.200903112237.jar
>    plugins/org.spearce.jgit_0.4.0.200903112237.jar
>    plugins/org.spearce.egit.ui_0.4.0.200903112237.jar
> 
> Quit and restarted Eclipse.
> 
> When I select a project with an existing git repository and try to 
> enable team/sharing only CVS and SVN are listed.

You don't have the a matching feature. That could be it, but I'm not sure. 
You can also try starting eclipse witth the -clean switch. Looking at the
<workspace>/.metadata/.log could also give you some hints.

For debugging/testing in general it is often easier to launch Eclipse from
eclipse (Run As) without reinstalling. 

-- robin

Re: [EGIT RFC PATCH(was Re: egit problem with sym linked eclipse project dirs)] Add some support for symlinked projects.

[Stephen Bannasch]

At 7:56 AM +0100 3/12/09, Robin Rosenberg wrote:
>torsdag 12 mars 2009 03:57:09 skrev Stephen Bannasch 
><stephen.bannasch@deanbrook.org>:
>>  OK ... I'm a bit confused now because I no longer have Git listed as
>>  a repository type for Team Sharing.
>>
>  > I deleted the existing org.spearce.* eclipse plugins

...

>  > The new plugins are there:
>>
>>     [eclipse]$ ls plugins/org.spearce.*
>>     plugins/org.spearce.egit.core.test_0.4.0.200903112237.jar
>>     plugins/org.spearce.egit_0.4.0.200903112237.jar
>>     plugins/org.spearce.egit.core_0.4.0.200903112237.jar
>>     plugins/org.spearce.jgit_0.4.0.200903112237.jar
>>     plugins/org.spearce.egit.ui_0.4.0.200903112237.jar
>>
>>  Quit and restarted Eclipse.
>>
>>  When I select a project with an existing git repository and try to
>>  enable team/sharing only CVS and SVN are listed.
>
>You don't have the a matching feature. That could be it, but I'm not sure.
>You can also try starting eclipse witth the -clean switch. Looking at the
><workspace>/.metadata/.log could also give you some hints.

I figured out the problem I had when testing the patched egit plugin. 
If I had installed egit from the update site I had to do more than 
just delete the jars -- I had to delete the whole feature from within 
Eclipse -- and then install the new plugin built from source.

In figuring this out I ended up testing the original problem with the 
master branch from earlier today and didn't see the issue I 
originally reported.

Did you already integrate the experimental patch?

Here's a documentation patch:

 From feb1ae0ccf7f671506853f4f49e9041950404b3d Mon Sep 17 00:00:00 2001
From: Stephen Bannasch <stephen.bannasch@gmail.com>
Date: Wed, 18 Mar 2009 11:51:56 -0400
Subject: [PATCH] clarify how to delete egit plugin when installing new build

---
  EGIT_INSTALL |   11 ++++++++++-
  1 files changed, 10 insertions(+), 1 deletions(-)

diff --git a/EGIT_INSTALL b/EGIT_INSTALL
index 3a8f249..3a9c209 100644
--- a/EGIT_INSTALL
+++ b/EGIT_INSTALL
@@ -21,7 +21,16 @@ things.

  INSTALLATION INSTRUCTIONS

-- Delete any old versions of the plugin in the 
<eclipse-path>/plugins/org.spearce.*
+First remove any existing egit plugin.
+
+If you have installed egit from the egit update site 
http://www.jgit.org/update-site:
+
+  delete the plugin from Software Updates and Add-ons from within  Eclipse
+
+If you have installed the plugin from source:
+
+  delete any old versions of the plugin jars in the 
<eclipse-path>/plugins/org.spearce.*
+
  - Start eclipse
  - Make sure a recent JDK 1.5.0_11 or JDK 1.6.x is among your 
installed JRE's. Which
    one is the default should not matter but Java 6 is recommended.
--
1.6.1.2


>For debugging/testing in general it is often easier to launch Eclipse from
>eclipse (Run As) without reinstalling.

As part of this test I need to switch to another workspace and I 
couldn't get that working with "Run As".

Re: [EGIT RFC PATCH(was Re: egit problem with sym linked eclipse project dirs)] Add some support for symlinked projects.

[Robin Rosenberg]

onsdag 18 mars 2009 17:02:15 skrev Stephen Bannasch <stephen.bannasch@deanbrook.org>:
> At 7:56 AM +0100 3/12/09, Robin Rosenberg wrote:
> >torsdag 12 mars 2009 03:57:09 skrev Stephen Bannasch 
> ><stephen.bannasch@deanbrook.org>:
> >>  OK ... I'm a bit confused now because I no longer have Git listed as
> >>  a repository type for Team Sharing.
> >>
> >  > I deleted the existing org.spearce.* eclipse plugins
> 
> ...
> 
> >  > The new plugins are there:
> >>
> >>     [eclipse]$ ls plugins/org.spearce.*
> >>     plugins/org.spearce.egit.core.test_0.4.0.200903112237.jar
> >>     plugins/org.spearce.egit_0.4.0.200903112237.jar
> >>     plugins/org.spearce.egit.core_0.4.0.200903112237.jar
> >>     plugins/org.spearce.jgit_0.4.0.200903112237.jar
> >>     plugins/org.spearce.egit.ui_0.4.0.200903112237.jar
> >>
> >>  Quit and restarted Eclipse.
> >>
> >>  When I select a project with an existing git repository and try to
> >>  enable team/sharing only CVS and SVN are listed.
> >
> >You don't have the a matching feature. That could be it, but I'm not sure.
> >You can also try starting eclipse witth the -clean switch. Looking at the
> ><workspace>/.metadata/.log could also give you some hints.
> 
> I figured out the problem I had when testing the patched egit plugin. 
> If I had installed egit from the update site I had to do more than 
> just delete the jars -- I had to delete the whole feature from within 
> Eclipse -- and then install the new plugin built from source.
> 
> In figuring this out I ended up testing the original problem with the 
> master branch from earlier today and didn't see the issue I 
> originally reported.
> 
> Did you already integrate the experimental patch?
> 
> Here's a documentation patch:
> 
>  From feb1ae0ccf7f671506853f4f49e9041950404b3d Mon Sep 17 00:00:00 2001
> From: Stephen Bannasch <stephen.bannasch@gmail.com>
> Date: Wed, 18 Mar 2009 11:51:56 -0400
> Subject: [PATCH] clarify how to delete egit plugin when installing new build

Thanks. Deleteing from the update manager deletes the feature from the features
directory plus configuration data.

> >For debugging/testing in general it is often easier to launch Eclipse from
> >eclipse (Run As) without reinstalling.
> 
> As part of this test I need to switch to another workspace and I 
> couldn't get that working with "Run As".

I do it all the time. Come to think of it, just selecting Run As requires that an appropriate
plugin project is selected in the package explorer. Select the org.spearce.egit.ui project
and then Run As > Eclipse Application. You can modify which workspace to open in
the launch configuration. If you want to work on an existing workspace you have to 
modify the launch configuration since the launched eclipse won't let you (maybe it
does, but not by default).

-- robin

Re: [EGIT RFC PATCH(was Re: egit problem with sym linked eclipse project dirs)] Add some support for symlinked projects.

[Robin Rosenberg]

onsdag 18 mars 2009 17:02:15 skrev Stephen Bannasch <stephen.bannasch@deanbrook.org>:
> In figuring this out I ended up testing the original problem with the 
> master branch from earlier today and didn't see the issue I 
> originally reported.
> 
> Did you already integrate the experimental patch?

No, so maybe you are running the plugin + the experimental patch
anyway.

-- robim