[git-users] Problem using detached worktrees with commands implemented in scripts

[git-users] Problem using detached worktrees with commands implemented in scripts

[Dale R. Worley]

In Git, one can set up a repository with a "detached worktree", where
the .git directory is not a subdirectory of the top directory of the
work tree.

In general, Git commands on a repository with a detached worktree can
be executed by cd'ing into the directory containing the .git
directory, and executing the Git command there.  E.g., "git add" and
"git commit" execute as one would expect.  (I think they can also be
executed by cd'ing to the worktree and setting GIT_DIR.)

However, this approach does not work with "git filter-branch", which
objects with "You need to run this command from the toplevel of the
working tree."

I suspect that it does not work with other Git commands that are
implemented with shell scripts.  The problem appears to be in the
git-sh-setup script, which is called by the Git shell scripts to set
up the environment and do preliminary tests.

It seems to me that this inconsistency between the script commands and
the binary commands can be fixed by updating git-sh-setup in this way:

--- git-sh-setup.Custom.orig	2013-06-20 12:59:45.000000000 -0400
+++ git-sh-setup	2013-10-07 22:34:06.719946134 -0400
@@ -297,14 +297,18 @@
 # if we require to be in a git repository.
 if test -z "$NONGIT_OK"
 then
-	GIT_DIR=$(git rev-parse --git-dir) || exit
+	export GIT_DIR=$(git rev-parse --git-dir) || exit
 	if [ -z "$SUBDIRECTORY_OK" ]
 	then
-		test -z "$(git rev-parse --show-cdup)" || {
-			exit=$?
-			echo >&2 "You need to run this command from the toplevel of the working tree."
-			exit $exit
-		}
+		cdup="$(git rev-parse --show-cdup)"
+		if [ -n "$cdup" ]
+		then
+			# Current directory is not the toplevel.
+			# Set GIT_DIR to the absolute path of the repository.
+			GIT_DIR=$(cd "$GIT_DIR" && pwd)
+			# cd to the toplevel.
+			cd $cdup
+		fi
 	fi
 	test -n "$GIT_DIR" && GIT_DIR=$(cd "$GIT_DIR" && pwd) || {
 		echo >&2 "Unable to determine absolute path of git directory"

What this change does is, when a command is invoked from a directory
containing a repository with a detached worktree, is to set GIT_DIR to
the directory of the repository, then cd to the top of the worktree.
After that, the script command should work as expected.

I am far from being an expert in Git internals, so I don't know
whether this is the correct approach to take to this problem or not.

Does anyone have any feedback on this?

Dale

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Junio C Hamano]

worley@alum.mit.edu (Dale R. Worley) writes:

> In general, Git commands on a repository with a detached worktree can
> be executed by cd'ing into the directory containing the .git
> directory, ...

Eh?  News to me; it might happened to have appeared to work by
accident, but that is not by design.

IIRC, the intended use pattern (i.e. the change that introduced
GIT_DIR and GIT_WORK_TREE environment variables was designed to
support) for such a working tree is to:

 - export GIT_DIR that points at the correct .git directory;

 - export GIT_WORK_TREE that points at the correct top-level of such
   a working tree; and then

 - run the commands anywhere in the working tree, as if you did not
   export these two environment variables and instead had the .git
   directory at the usual place in the working tree.

It _is_ possible that we may have broken this canonical use pattern
over time with more recent updates; I do not think we have extensive
test coverage for "detached worktree" use case in the first place.

> Does anyone have any feedback on this?

Not exporting GIT_DIR variable in sh-setup was done not by accident
but as a very deliberate design choice, IIRC.

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Philip Oakley]

From: "Junio C Hamano" <gitster@pobox.com>
> worley@alum.mit.edu (Dale R. Worley) writes:
>
>> In general, Git commands on a repository with a detached worktree can
>> be executed by cd'ing into the directory containing the .git
>> directory, ...
>
> Eh?  News to me; it might happened to have appeared to work by
> accident, but that is not by design.

I think it is this part in Dale's original email

"However, this approach does not work with "git filter-branch", which
objects with "You need to run this command from the toplevel of the
working tree."

that is the problem Dale has seen. IIRC there are a few commands that do 
require to be run from the toplevel ('git bisect' I think is another), 
and the detection process for 'toplevel' may not work properly when in a 
separated work-tree environment.

Perhaps something to consider.

Philip

>
> IIRC, the intended use pattern (i.e. the change that introduced
> GIT_DIR and GIT_WORK_TREE environment variables was designed to
> support) for such a working tree is to:
>
> - export GIT_DIR that points at the correct .git directory;
>
> - export GIT_WORK_TREE that points at the correct top-level of such
>   a working tree; and then
>
> - run the commands anywhere in the working tree, as if you did not
>   export these two environment variables and instead had the .git
>   directory at the usual place in the working tree.
>
> It _is_ possible that we may have broken this canonical use pattern
> over time with more recent updates; I do not think we have extensive
> test coverage for "detached worktree" use case in the first place.
>
>> Does anyone have any feedback on this?
>
> Not exporting GIT_DIR variable in sh-setup was done not by accident
> but as a very deliberate design choice, IIRC.
> --

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Junio C Hamano]

"Philip Oakley" <philipoakley@iee.org> writes:

> ... and the detection process for 'toplevel' may not work
> properly when in a separated work-tree environment.

Without GIT_WORK_TREE exported to point at the top-level, there is
nothing that lets us "detect" it, as the working tree does not have
".git" directory to tell us to stop, no?

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Dale R. Worley]

> From: Junio C Hamano <gitster@pobox.com>
> 
> worley@alum.mit.edu (Dale R. Worley) writes:
> 
> > In general, Git commands on a repository with a detached worktree can
> > be executed by cd'ing into the directory containing the .git
> > directory, ...
> 
> Eh?  News to me; it might happened to have appeared to work by
> accident, but that is not by design.

I must admit I've never seen the design (and I personally doubt that
the design has ever been written down).  But at least the following
commands work correctly on a detached worktree if the current
directory contains the .git directory, because I am using them in a
production manner:

    git add
    git cat-file
    git commit
    git commit-tree
    git config
    git gc
    git log
    git ls-tree
    git reset
    git rev-list
    git update-ref

In my situation, the worktree is not, in my mind, dependent on the
repository; the repository is intended to keep backups of the contents
of the directories that are worktree.  Indeed, one could establish
several detached repositories to back up different subsets of the same
worktree.  So it is conceptually natural to execute Git in the
repository directory.  And, after all, the current directory
identifies the repository and the repository contains a pointer to the
worktree.

> Not exporting GIT_DIR variable in sh-setup was done not by accident
> but as a very deliberate design choice, IIRC.

The intention of my change is that it appears that all of the failures
of my use pattern are when the command is implemented by a shell
script, and it appears that all shell scripts initially invoke
git-sh-setup.

The change specifically detects my use pattern and, for the remainder
of the script, changes the use pattern into a pattern closely related
to the one that Junio documents:

 - export GIT_DIR that points at the correct .git directory;

 - GIT_WORK_TREE is left unset

 - set the current directory to the top of the working tree

Perhaps the change should also set GIT_WORK_TREE.  I haven't noticed
setting GIT_WORK_TREE to be necessary for "git filter-branch", but
perhaps that is coincidental.

It seems to me that this change would uniformly support the use
pattern I use without affecting Git's behavior in any other case.

Dale

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Junio C Hamano]

worley@alum.mit.edu (Dale R. Worley) writes:

> I must admit I've never seen the design (and I personally doubt that
> the design has ever been written down).  But at least the following
> commands work correctly on a detached worktree if the current
> directory contains the .git directory, because I am using them in a
> production manner:
>
>     git add

If you have this:

	/repositories/proj.git/{refs,objects,...}
	/working/trees/proj-wt1/

where proj-wt1 is a working tree for that proj.git repository, the
idea was to set these:

	GIT_DIR=/repositories/proj.git
        GIT_WORK_TREE=/working/trees/proj-wt1
        export GIT_DIR GIT_WORK_TREE

and then working in /working/trees/proj-wt1 or any of its
subdirectory should work as if you did not have these two
environment variables and had /working/trees/proj-wt1/.git instead
of /repositories/proj.git as the repository.  To make that use case
work was the motivation behind these environment variables.

	Side note: without GIT_WORK_TREE environment (or
	core.worktree), there is no way to tell where the top level
	is, so you were limited to always be at the top level of
	your working tree if you used GIT_DIR to refer to a
	repository that is not embedded in your working tree.  There
	were some changes in this area, but I do not recall the
	details offhand.

Now, when you say "the cwd contains the .git directory", do you mean

	cd /repositories
        git add ../working/trees/proj-wt1/file

updates "file" in the /repositories/proj.git/index?  Or do you mean
this?

	cd /repositories/proj.git
        git add ../../working/trees/proj-wt1/file

Or this?

	cd /repositories
	edit ../working/trees/proj-wt1/file
	git add file

Most of the commands you listed do not need to look at the actual
working tree files, so I would expect e.g. "git log" or "git log
paths..." to work but I am wondering what your definition of "works"
with respect to the pathspecs, especially when you talk about
starting Git command _outside_ the working tree (whether the working
tree has its repository embedded in it is not very relevant).

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Philip Oakley]

From: "Junio C Hamano" <gitster@pobox.com>
> "Philip Oakley" <philipoakley@iee.org> writes:
>
>> ... and the detection process for 'toplevel' may not work
>> properly when in a separated work-tree environment.
>
> Without GIT_WORK_TREE exported to point at the top-level, there is
> nothing that lets us "detect" it, as the working tree does not have
> ".git" directory to tell us to stop, no?
>

"No", but not in that way.

My point (to Dale) was, as you state, that the "cd to top level" was 
(IIUC) the probable causes of the fault, and that a documentation update 
would probably be appropriate for the discussion on exporting 
GIT_WORK_TREE, and that it would specifically mention those git commands 
that needed to "cd to top level", and hence would not work in such an 
environment. (I wasn't sure where the appropriate "cd to top level" 
function was)

An explanation here on the list wouldn't solve the problems for others 
who are yet to make the same mistake, hence the implied suggestion.

Philip 

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Junio C Hamano]

"Philip Oakley" <philipoakley@iee.org> writes:

> From: "Junio C Hamano" <gitster@pobox.com>
>> "Philip Oakley" <philipoakley@iee.org> writes:
>>
>>> ... and the detection process for 'toplevel' may not work
>>> properly when in a separated work-tree environment.
>>
>> Without GIT_WORK_TREE exported to point at the top-level, there is
>> nothing that lets us "detect" it, as the working tree does not have
>> ".git" directory to tell us to stop, no?
>>
>
> "No", but not in that way.
>
> My point (to Dale) was, as you state, that the "cd to top level" was
> (IIUC) the probable causes of the fault, and that a documentation
> update would probably be appropriate for the discussion on exporting
> GIT_WORK_TREE, and that it would specifically mention those git
> commands that needed to "cd to top level", and hence would not work in
> such an environment. (I wasn't sure where the appropriate "cd to top
> level" function was)
>
> An explanation here on the list wouldn't solve the problems for others
> who are yet to make the same mistake, hence the implied suggestion.

I understand what you mean by these last two lines. It was unclear
to me which part of our documentation needs updating and how, and
that was (and still is) what I was primarily interested in finding
out.

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Philip Oakley]

From: "Junio C Hamano" <gitster@pobox.com>
> "Philip Oakley" <philipoakley@iee.org> writes:
>
>> From: "Junio C Hamano" <gitster@pobox.com>
>>> "Philip Oakley" <philipoakley@iee.org> writes:
>>>
>>>> ... and the detection process for 'toplevel' may not work
>>>> properly when in a separated work-tree environment.
>>>
>>> Without GIT_WORK_TREE exported to point at the top-level, there is
>>> nothing that lets us "detect" it, as the working tree does not have
>>> ".git" directory to tell us to stop, no?
>>>
>>
>> "No", but not in that way.
>>
>> My point (to Dale) was, as you state, that the "cd to top level" was
>> (IIUC) the probable causes of the fault, and that a documentation
>> update would probably be appropriate for the discussion on exporting
>> GIT_WORK_TREE, and that it would specifically mention those git
>> commands that needed to "cd to top level", and hence would not work 
>> in
>> such an environment. (I wasn't sure where the appropriate "cd to top
>> level" function was)
>>
>> An explanation here on the list wouldn't solve the problems for 
>> others
>> who are yet to make the same mistake, hence the implied suggestion.
>
> I understand what you mean by these last two lines. It was unclear
> to me which part of our documentation needs updating and how, and
> that was (and still is) what I was primarily interested in finding
> out.
>
I was expecting that the places would be in git(1) [git.txt] and 
config(1) [config.txt], in the enironment variables GIT_WORK_TREE 
section and core.worktree sections repectively. However what the right 
text would be hasn't been fully determined yet, as it should be clear 
about which commands don't follow the stated 'rules'. Dale's use case 
does appear to be stretching...

Philip 

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Philip Oakley]

From: "Philip Oakley" <philipoakley@iee.org>
> From: "Junio C Hamano" <gitster@pobox.com>
>> "Philip Oakley" <philipoakley@iee.org> writes:
>>
>>> From: "Junio C Hamano" <gitster@pobox.com>
>>>> "Philip Oakley" <philipoakley@iee.org> writes:
>>>>
>>>>> ... and the detection process for 'toplevel' may not work
>>>>> properly when in a separated work-tree environment.
>>>>
>>>> Without GIT_WORK_TREE exported to point at the top-level, there is
>>>> nothing that lets us "detect" it, as the working tree does not have
>>>> ".git" directory to tell us to stop, no?
>>>>
>>>
>>> "No", but not in that way.
>>>
>>> My point (to Dale) was, as you state, that the "cd to top level" was
>>> (IIUC) the probable causes of the fault, and that a documentation
>>> update would probably be appropriate for the discussion on exporting
>>> GIT_WORK_TREE, and that it would specifically mention those git
>>> commands that needed to "cd to top level", and hence would not work
>>> in
>>> such an environment. (I wasn't sure where the appropriate "cd to top
>>> level" function was)
>>>
>>> An explanation here on the list wouldn't solve the problems for
>>> others
>>> who are yet to make the same mistake, hence the implied suggestion.
>>
>> I understand what you mean by these last two lines. It was unclear
>> to me which part of our documentation needs updating and how, and
>> that was (and still is) what I was primarily interested in finding
>> out.
>>
> I was expecting that the places would be in git(1) [git.txt] and
> config(1) [config.txt], in the enironment variables GIT_WORK_TREE
> section and core.worktree sections repectively. However what the right
> text would be hasn't been fully determined yet, as it should be clear
> about which commands don't follow the stated 'rules'. Dale's use case
> does appear to be stretching...
>
> Philip

A bit more looking gave that the cd_to_toplevel () in git-sh-setup.sh
directly uses `git rev-parse --show-toplevel`, which simply returns
work_tree (static char *work_tree; in environment.c, with comment /*
This is set by setup_git_dir_gently() and/or git_default_config() */), 
apparently without a check for the GIT_WORK_TREE.

One option may be to either protect the cd_to_toplevel  code with a
check of `git rev-parse --local-env-vars` to see if GIT_WORK_TREE is
present. Or create `git rev-parse --work-dir` to match `--git-dir`. This 
would be a code level fix. This makes the assumption that if a deteched 
GIT_WORK_TREE is set then it is the top level.

In terms of command scripts that use git-sh-setup.sh we have a longish 
list, so a full list in the documentation is probably unreasonable 
(which suggests that a code fix would be more apprpriate)

commands:

git-am
git-bisect
git-filter-branch
git-instaweb
git-lost-found
git-merge-one-file
git-mergetool
git-pull
git-quiltimport
git-rebase
git-repack
git-request-pull
git-stash
git-submodule
git-web--browse

git\contrib\*various*



Philip

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Jonathan Nieder]

Philip Oakley wrote:

> A bit more looking gave that the cd_to_toplevel () in git-sh-setup.sh
> directly uses `git rev-parse --show-toplevel`, which simply returns
> work_tree (static char *work_tree; in environment.c, with comment /*
> This is set by setup_git_dir_gently() and/or git_default_config()
> */), apparently without a check for the GIT_WORK_TREE.

Getting closer. :)

The usual way to use GIT_WORK_TREE is along with GIT_DIR.  That takes
you into the setup_explicit_git_dir() codepath, which does respect
GIT_WORK_TREE as it should.  (setup_discovered_git_dir does, too.)

The strange behavior you ran into is that unlike, say, git-pull.sh and
git-am.sh, filter-branch does not set SUBDIRECTORY_OK, store the
prefix from 'git rev-parse --show-prefix', and then cd_to_toplevel at
the top of the script.  In other words, nobody bothered to make it
work from anywhere other than the toplevel of the worktree to begin
with, and nobody wanted it enough to fix it later.

Hope that helps,
Jonathan

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Philip Oakley]

From: "Jonathan Nieder" <jrnieder@gmail.com>
> Philip Oakley wrote:

It was Dale that had the problem, I was just suggesting where he might 
want to look... ;-)

>
>> A bit more looking gave that the cd_to_toplevel () in git-sh-setup.sh
>> directly uses `git rev-parse --show-toplevel`, which simply returns
>> work_tree (static char *work_tree; in environment.c, with comment /*
>> This is set by setup_git_dir_gently() and/or git_default_config()
>> */), apparently without a check for the GIT_WORK_TREE.
>
> Getting closer. :)
>
> The usual way to use GIT_WORK_TREE is along with GIT_DIR.

When re-checking the manual's git(1) env variable section the comment 
that implies this didn't read well "The value will not be used in 
combination...". The section probably needs that to be stated explicitly 
("an exported GIT_WORK_TREE is ignored if GIT_DIR is not set").

>   That takes
> you into the setup_explicit_git_dir() codepath, which does respect
> GIT_WORK_TREE as it should.  (setup_discovered_git_dir does, too.)
>
> The strange behavior you ran into is that unlike, say, git-pull.sh and
> git-am.sh, filter-branch does not set SUBDIRECTORY_OK, store the
> prefix from 'git rev-parse --show-prefix', and then cd_to_toplevel at
> the top of the script.  In other words, nobody bothered to make it
> work from anywhere other than the toplevel of the worktree to begin
> with, and nobody wanted it enough to fix it later.
>

I maybe wrong, but I thought that in Dale's case he was already at the 
same level as the GIT_WORK_TREE he had set, so may not have expected to 
need a cd_to_toplevel


Dale did propose a patch in 
http://thread.gmane.org/gmane.comp.version-control.git/236260 as the 
error was from later in the setup script.

> Hope that helps,
> Jonathan
>

Philip 

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Dale R. Worley]

> From: Junio C Hamano <gitster@pobox.com>

> 	Side note: without GIT_WORK_TREE environment (or
> 	core.worktree), there is no way to tell where the top level
> 	is, so you were limited to always be at the top level of
> 	your working tree if you used GIT_DIR to refer to a
> 	repository that is not embedded in your working tree.  There
> 	were some changes in this area, but I do not recall the
> 	details offhand.

That's not true.  The core.worktree config variable tells the top of
the worktree, so once you've located the repository, you know where
the worktree is.

Indeed, it's not clear why GIT_WORK_TREE exists, as that allows the
user to set GIT_WORK_TREE inconsistently with core.worktree.  (What
happens if you do that?)

Dale

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Junio C Hamano]

worley@alum.mit.edu (Dale R. Worley) writes:

>> From: Junio C Hamano <gitster@pobox.com>
>
>> 	Side note: without GIT_WORK_TREE environment (or
>> 	core.worktree), there is no way to tell where the top level
>> 	is, so you were limited to always be at the top level of
>> 	your working tree if you used GIT_DIR to refer to a
>> 	repository that is not embedded in your working tree.  There
>> 	were some changes in this area, but I do not recall the
>> 	details offhand.
>
> That's not true.  The core.worktree config variable tells the top of
> the worktree, so once you've located the repository, you know where
> the worktree is.

Read the second line again, perhaps?

> ... it's not clear why GIT_WORK_TREE exists, ...

The configuration item came _way_ later than the environment, and we
need to keep users and scripts from old world working, that is why.

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Dale R. Worley]

> From: Junio C Hamano <gitster@pobox.com>

> Now, when you say "the cwd contains the .git directory", do you mean
> 
> 	cd /repositories
>         git add ../working/trees/proj-wt1/file
> 
> updates "file" in the /repositories/proj.git/index?  Or do you mean
> this?

The pattern I use is to have this:

	/repository/.git
	/working/...

then

	cd /repository
	git add /working/x/y/z

works as you'd expect it to.  "git rm" seems to work correctly under
these circumstances as well.

I seem to recall that using relative <path> values doesn't work under
some conditions involving symbolic links, but I can't recall the
details right now.

> you talk about starting Git command _outside_ the working tree
> (whether the working tree has its repository embedded in it is not
> very relevant).

The above pattern is what I mean, where the cwd is not within the work
tree.

Dale

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Dale R. Worley]

> From: Junio C Hamano <gitster@pobox.com>

> It was unclear to me which part of our documentation needs updating
> and how, and that was (and still is) what I was primarily interested
> in finding out.

It seems to me that what is missing is a description of the
circumstances under which Git can be run.  With Subversion (the only
other source control system I know in detail), the working tree that
is operated on is at and below the cwd, and the working tree always
points to the repository.  (A subdirectory of a working tree is also a
valid working tree.)

With Git, it seems that the basic usage is that Git searches upward
from the cwd to find the top of the work tree, which is distinguished
by having a .git subdirectory.  The rules when the worktree is
detached are more complicated, and don't seem to be written in any
single place.

Dale

Re: [git-users] Problem using detached worktrees with commands implemented in scripts

[Dale R. Worley]

> From: Junio C Hamano <gitster@pobox.com>

> > ... it's not clear why GIT_WORK_TREE exists, ...
> 
> The configuration item came _way_ later than the environment, and we
> need to keep users and scripts from old world working, that is why.

OK, that explains a great deal.  IIRC, I first became aware that
detached worktrees are possible through the documentation of
core.worktree.  As Git's architecture has a tight binding between the
repository and the worktree, it made a great deal of sense to me that
the repository points to the detached worktree.  And the absence of
core.worktree, a non-detached worktree, is essentially equivalent to
having core.worktree specify the directory containing the .git
directory.

So the obvious way (to me) to invoke Git with a detached worktree is
to set GIT_DIR to point to the repository, and the repository points
to the root of the worktree.  If the command operates on the worktree,
Git can compare the cwd with the worktree root to determine the
relative path of files.

(And you can see that in this situation, Git doesn't have to search
upward to try to determine where the worktree root is.)

What you're saying is that there's an older mode of operation where
the repository does not point to the worktree.  Instead, the caller
has to set GIT_DIR to locate the repository and GIT_WORK_TREE to
locate the worktree.  This would be prone to error, because the user
is responsible for always pairing the repository with the correct
worktree; it doesn't enforce the architectural assumption that the
repository is paired with a work tree.

Dale