git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
* [PATCH v6 0/5] teach submodules to know they're submodules
@ 2021-11-17  0:56 Emily Shaffer
  2021-11-17  0:56 ` [PATCH v6 1/5] t7400-submodule-basic: modernize inspect() helper Emily Shaffer
                   ` (7 more replies)
  0 siblings, 8 replies; 64+ messages in thread
From: Emily Shaffer @ 2021-11-17  0:56 UTC (permalink / raw)
  To: git
  Cc: Emily Shaffer, Albert Cui, Phillip Wood, Johannes Schindelin,
	Ævar Arnfjörð Bjarmason, Junio C Hamano,
	Matheus Tavares Bernardino, Jonathan Nieder, Jacob Keller,
	Atharva Raykar, Derrick Stolee, Jonathan Tan

For the original cover letter, see
https://lore.kernel.org/git/20210611225428.1208973-1-emilyshaffer%40google.com.

CI run: https://github.com/nasamuffin/git/actions/runs/1469463328

Since v5:

A couple things. Firstly, a semantics change *back* to the semantics of
v3 - we map from gitdir to gitdir, *not* from common dir to common dir,
so that theoretically a submodule with multiple worktrees in multiple
superproject worktrees will be able to figure out which worktree of the
superproject it's in. (Realistically, that's not really possible right
now, but I'd like to change that soon.)

Secondly, a rewording of comments and commit messages to indicate that
this isn't a cache of some expensive operation, but rather intended to
be the source of truth for all submodules. I also added a fifth commit
rewriting `git rev-parse --show-superproject-working-tree` to
demonstrate what that means in practice - but from a practical
standpoint, I'm a little worried about that fifth patch. More details in
the patch 5 description.

I did discuss Ævar's idea of relying on in-process filesystem digging to
find the superproject's gitdir with the rest of the Google team, but in
the end decided that there are some worries about filesystem digging in
this way (namely, some ugly interactions with network drives that are
actually already an issue for Googler Linux machines). Plus, the allure
of being able to definitively know that we're a submodule is pretty
strong. ;) But overall, this is the direction I'd prefer to keep going
in, rather than trying to guess from the filesystem going forward.

Since v4:

The only real change here is a slight semantics change to map from
<submodule gitdir> to <superproject common git dir>. In every case
*except* for when the superproject has a worktree, this changes nothing.
For the case when the superproject has a worktree, this means that now
submodules will refer to the general superproject common dir (e.g. no
worktree-specific refs or configs or whatnot).

I *think* that because a submodule should exist in the context of the
common dir, not the worktree gitdir, that is ok. However, it does mean
it would be difficult to do something like sharing a config specific to
the worktree (the initial goal of this series).

$ROOT/.git
$ROOT/.git/config.superproject <- shared by $ROOT/.git/modules/sub
$ROOT/.git/modules/sub <- points to $ROOT/.git
$ROOT/.git/worktrees/wt
$ROOT/.git/worktrees/wt/config.superproject <- contains a certain config-based pre-commit hook

If the submodule only knows about the common dir, that is tough, because
the submodule would basically have to guess which worktree it's in from
its own path. There would be no way for '$WT/sub' to inherit
'$ROOT/.git/worktrees/wt/config.superproject'.

That said... right now, we don't support submodules in worktrees very
well at all. A submodule in a worktree will get a brand new gitdir in
$ROOT/.git/worktrees/modules/ (and that brand new gitdir would point to
the super's common dir). So I think we can punt on this entire question
until we teach submodules and worktrees to play more gracefully together
(it's on my long list...), and at that time we can probably introduce a
pointer from $ROOT/.git/modules/sub/worktrees/wt/ to
$ROOT/.git/worktrees/wt/....

Or, to summarize the long ramble above: "this is still kind of weird
with worktrees, but let's fix it later when we fix worktrees more
thoroughly".

(More rambling about worktree weirdness here:
https://lore.kernel.org/git/YYRaII8YWVxlBqsF%40google.com )


Since v3, a pretty major change: the semantics of
submodule.superprojectGitDir has changed, to point from the submodule's
gitdir to the superproject's gitdir (in v3 and earlier, we kept a path
from the submodule's *worktree* to the superproject's gitdir instead).
This cleans up some of the confusions about the behavior when a
submodule worktree moves around in the superproject's tree, or in a
future when we support submodules having multiple worktrees.

I also tried to simplify the tests to use 'test-tool path-utils
relative_path' everywhere - I think that makes them much more clear for
a test reader, but if you're reviewing and it isn't obvious what we're
testing for, please speak up.

I think this is pretty mature and there was a lot of general agreement
that the gitdir->gitdir association was the way to go, so please be
brutal and look for nits, leaks, etc. this round ;)
[/v4 cover letter]

Emily Shaffer (5):
  t7400-submodule-basic: modernize inspect() helper
  introduce submodule.superprojectGitDir record
  submodule: record superproject gitdir during absorbgitdirs
  submodule: record superproject gitdir during 'update'
  submodule: use config to find superproject worktree

 Documentation/config/submodule.txt |  12 ++++
 builtin/submodule--helper.c        |  11 +++
 git-submodule.sh                   |  15 ++++
 submodule.c                        | 108 ++++++++++++++++++++++++++++-
 t/t1500-rev-parse.sh               |   9 +++
 t/t7400-submodule-basic.sh         |  54 ++++++++-------
 t/t7406-submodule-update.sh        |  27 ++++++++
 t/t7412-submodule-absorbgitdirs.sh |  82 +++++++++++++++++++++-
 8 files changed, 290 insertions(+), 28 deletions(-)

Range-diff against v5:
1:  6ff10beaf2 = 1:  f1b08a7057 t7400-submodule-basic: modernize inspect() helper
2:  d4f4627585 ! 2:  d46c8439ab introduce submodule.superprojectGitDir record
    @@ Commit message
     
         By using a relative path instead of an absolute path, we can move the
         superproject directory around on the filesystem without breaking the
    -    submodule's cache. And by using the path from gitdir to gitdir, we can
    +    submodule's pointer. And by using the path from gitdir to gitdir, we can
         move the submodule within the superproject's tree structure without
    -    breaking the submodule's cache, too. Finally, by pointing at
    -    "get_git_common_dir()" instead of "get_git_dir()", we ensure the link
    -    will refer to the parent repo, not to a specific worktree.
    +    breaking the submodule's pointer, too. Finally, by pointing at the
    +    superproject's worktree gitdir (if it exists), we ensure that we can
    +    tell which worktree contains our submodule.
     
         Since this hint value is only introduced during new submodule creation
         via `git submodule add`, though, there is more work to do to allow the
         record to be created at other times.
     
    -    If the new config is present, we can do some optional value-added
    -    behavior, like letting "git status" print additional info about the
    -    submodule's status in relation to its superproject, or like letting the
    -    superproject and submodule share an additional config file separate from
    -    either one's local config.
    +    Once this new config is reliably in place, we can use it to know
    +    definitively that we are working in a submodule, and to know which
    +    superproject we are a submodule of. This allows us to do some
    +    value-added behavior, like letting "git status" print additional info
    +    about the submodule's status in relation to its superproject, or like
    +    letting the superproject and submodule share an additional config file
    +    separate from either one's local config.
     
         Signed-off-by: Emily Shaffer <emilyshaffer@google.com>
         Helped-by: Junio C Hamano <gitster@pobox.com>
    @@ Documentation/config/submodule.txt: submodule.alternateErrorStrategy::
     +
     +submodule.superprojectGitDir::
     +	The relative path from the submodule's gitdir to its superproject's
    -+	common dir. When Git is run in a repository, it usually makes no
    ++	gitdir. When Git is run in a repository, it usually makes no
     +	difference whether this repository is standalone or a submodule, but if
     +	this configuration variable is present, additional behavior may be
     +	possible, such as "git status" printing additional information about
    @@ builtin/submodule--helper.c: static int clone_submodule(struct module_clone_data
      		git_config_set_in_file(p, "submodule.alternateErrorStrategy",
      				       error_strategy);
      
    -+	git_config_set_in_file(p, "submodule.superprojectGitdir",
    -+			       relative_path(absolute_path(get_git_common_dir()),
    ++	/*
    ++	 * Set the path from submodule's new gitdir to superproject's gitdir.
    ++	 * The latter may be a worktree gitdir. However, it is not possible for
    ++	 * the submodule to have a worktree-specific gitdir or config at clone
    ++	 * time, because "extensions.worktreeConfig" is only valid when set in
    ++	 * the local gitconfig, which the brand new submodule does not have yet.
    ++	 */
    ++	git_config_set_in_file(p, "submodule.superprojectGitDir",
    ++			       relative_path(absolute_path(get_git_dir()),
     +					     sm_gitdir, &sb));
     +
      	free(sm_alternate);
    @@ t/t7400-submodule-basic.sh: submodurl=$(pwd -P)
     +	# Ensure that submodule.superprojectGitDir contains the path from the
     +	# submodule's gitdir to the superproject's gitdir.
     +
    -+	super_abs_gitdir=$(git -C "$super_dir" rev-parse --path-format=absolute --git-common-dir) &&
    -+	sub_abs_gitdir=$(git -C "$sub_dir" rev-parse --path-format=absolute --git-common-dir) &&
    ++	super_abs_gitdir=$(git -C "$super_dir" rev-parse --absolute-git-dir) &&
    ++	sub_abs_gitdir=$(git -C "$sub_dir" rev-parse --absolute-git-dir) &&
     +
     +	[ "$(git -C "$sub_dir" config --get submodule.superprojectGitDir)" = \
     +	  "$(test-tool path-utils relative_path "$super_abs_gitdir" \
3:  2dae297943 ! 3:  63ddaf5608 submodule: record superproject gitdir during absorbgitdirs
    @@ submodule.c: static void relocate_single_git_dir_into_superproject(const char *p
      	char *old_git_dir = NULL, *real_old_git_dir = NULL, *real_new_git_dir = NULL;
      	struct strbuf new_gitdir = STRBUF_INIT;
      	const struct submodule *sub;
    ++	struct config_set sub_cs;
     +	struct strbuf config_path = STRBUF_INIT, sb = STRBUF_INIT;
    ++	int tmp;
      
      	if (submodule_uses_worktrees(path))
      		die(_("relocate_gitdir for submodule '%s' with "
    @@ submodule.c: static void relocate_single_git_dir_into_superproject(const char *p
      
      	relocate_gitdir(path, real_old_git_dir, real_new_git_dir);
      
    -+	/* cache pointer to superproject's gitdir */
    ++	/*
    ++	 * Note location of superproject's gitdir. Because the submodule already
    ++	 * has a gitdir and local config, we can store this pointer from
    ++	 * worktree config to worktree config, if the submodule has
    ++	 * extensions.worktreeConfig set.
    ++	 */
     +	strbuf_addf(&config_path, "%s/config", real_new_git_dir);
    ++	git_configset_init(&sub_cs);
    ++	git_configset_add_file(&sub_cs, config_path.buf);
    ++	/* return 0 indicates config was found - we have a worktree config */
    ++	if (!git_configset_get_bool(&sub_cs, "extensions.worktreeConfig", &tmp))
    ++		strbuf_addstr(&config_path, ".worktree");
    ++
     +	git_config_set_in_file(config_path.buf, "submodule.superprojectGitdir",
    -+			       relative_path(absolute_path(get_git_common_dir()),
    ++			       relative_path(absolute_path(get_git_dir()),
     +					     real_new_git_dir, &sb));
     +
    ++	git_configset_clear(&sub_cs);
     +	strbuf_release(&config_path);
     +	strbuf_release(&sb);
      	free(old_git_dir);
    @@ t/t7412-submodule-absorbgitdirs.sh: test_expect_success 'absorbing fails for a s
     +	# absorb the git dir
     +	git submodule absorbgitdirs sub4 &&
     +
    -+	# make sure the submodule cached the superproject gitdir correctly
    -+	submodule_gitdir="$(git -C sub4 rev-parse --path-format=absolute --git-common-dir)" &&
    -+	superproject_gitdir="$(git rev-parse --path-format=absolute --git-common-dir)" &&
    ++	# make sure the submodule noted the superproject gitdir correctly
    ++	submodule_gitdir="$(git -C sub4 rev-parse --absolute-git-dir)" &&
    ++	superproject_gitdir="$(git rev-parse --absolute-git-dir)" &&
     +
     +	test-tool path-utils relative_path "$superproject_gitdir" \
     +		"$submodule_gitdir" >expect &&
    @@ t/t7412-submodule-absorbgitdirs.sh: test_expect_success 'absorbing fails for a s
     +	test_cmp expect actual
     +	)
     +'
    ++
    ++test_expect_success 'absorbgitdirs works with a submodule with worktree config' '
    ++	# reuse the worktree of the superproject
    ++	(
    ++	cd wt &&
    ++
    ++	# create a new unembedded git dir
    ++	git init sub5 &&
    ++	test_commit -C sub5 first &&
    ++	git submodule add ./sub5 &&
    ++	test_tick &&
    ++
    ++	# turn on worktree configs for submodule
    ++	git -C sub5 config extensions.worktreeConfig true &&
    ++
    ++	# absorb the git dir
    ++	git submodule absorbgitdirs sub5 &&
    ++
    ++	# make sure the submodule noted the superproject gitdir correctly
    ++	submodule_gitdir="$(git -C sub5 rev-parse --absolute-git-dir)" &&
    ++	superproject_gitdir="$(git rev-parse --absolute-git-dir)" &&
    ++
    ++	test-tool path-utils relative_path "$superproject_gitdir" \
    ++		"$submodule_gitdir" >expect &&
    ++	git -C sub5 config submodule.superprojectGitDir >actual &&
    ++
    ++	test_cmp expect actual &&
    ++
    ++	# make sure the config went into the submodule config.worktree
    ++	test_file_not_empty "$submodule_gitdir/config.worktree"
    ++	)
    ++'
     +
      test_done
4:  f0412d6d34 ! 4:  33a582ef13 submodule: record superproject gitdir during 'update'
    @@ Metadata
      ## Commit message ##
         submodule: record superproject gitdir during 'update'
     
    -    A recorded hint path to the superproject's gitdir might be added during
    +    A recorded path to the superproject's gitdir might be added during
         'git submodule add', but in some cases - like submodules which were
         created before 'git submodule add' learned to record that info - it might
         be useful to update the hint. Let's do it during 'git submodule
    @@ git-submodule.sh: cmd_update()
      			;;
      		esac
      
    -+		# Cache a pointer to the superproject's common dir. This may have
    -+		# changed, unless it's a fresh clone. Writes it to worktree
    -+		# if applicable, otherwise to local.
    ++		# Store a poitner to the superproject's gitdir. This may have
    ++		# changed, unless it's a fresh clone. Write to worktree if
    ++		# applicable, and point to superproject's worktree gitdir if
    ++		# applicable.
     +		if test -z "$just_cloned"
     +		then
     +			sm_gitdir="$(git -C "$sm_path" rev-parse --absolute-git-dir)"
     +			relative_gitdir="$(git rev-parse --path-format=relative \
     +							 --prefix "${sm_gitdir}" \
    -+							 --git-common-dir)"
    ++							 --git-dir)"
     +
     +			git -C "$sm_path" config --worktree \
     +				submodule.superprojectgitdir "$relative_gitdir"
    @@ t/t7406-submodule-update.sh: test_expect_success 'submodule update --quiet passe
     +	 git -C submodule config --unset submodule.superprojectGitdir &&
     +	 git submodule update &&
     +	 test-tool path-utils relative_path \
    -+		"$(git rev-parse --path-format=absolute --git-common-dir)" \
    -+		"$(git -C submodule rev-parse --path-format=absolute --git-common-dir)" >expect &&
    ++		"$(git rev-parse --absolute-git-dir)" \
    ++		"$(git -C submodule rev-parse --absolute-git-dir)" >expect &&
     +	 git -C submodule config submodule.superprojectGitdir >actual &&
     +	 test_cmp expect actual
     +	)
     +'
    ++
    ++test_expect_success 'submodule update uses config.worktree if applicable' '
    ++	(cd super &&
    ++	 git -C submodule config --unset submodule.superprojectGitDir &&
    ++	 git -C submodule config extensions.worktreeConfig true &&
    ++	 git submodule update &&
    ++	 test-tool path-utils relative_path \
    ++		"$(git rev-parse --absolute-git-dir)" \
    ++		"$(git -C submodule rev-parse --absolute-git-dir)" >expect &&
    ++	 git -C submodule config submodule.superprojectGitdir >actual &&
    ++	 test_cmp expect actual &&
    ++
    ++	 test_file_not_empty "$(git -C submodule rev-parse --absolute-git-dir)/config.worktree"
    ++	)
    ++'
     +
      test_done
-:  ---------- > 5:  a8b5d40a77 submodule: use config to find superproject worktree
-- 
2.34.0.rc1.387.gb447b232ab-goog


^ permalink raw reply	[flat|nested] 64+ messages in thread
end of thread, other threads:[~2022-03-15 21:19 UTC | newest]

Thread overview: 64+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-17  0:56 [PATCH v6 0/5] teach submodules to know they're submodules Emily Shaffer
2021-11-17  0:56 ` [PATCH v6 1/5] t7400-submodule-basic: modernize inspect() helper Emily Shaffer
2021-11-17  0:56 ` [PATCH v6 2/5] introduce submodule.superprojectGitDir record Emily Shaffer
2021-11-17 23:43   ` Jonathan Tan
2021-11-17  0:56 ` [PATCH v6 3/5] submodule: record superproject gitdir during absorbgitdirs Emily Shaffer
2021-11-17  0:57 ` [PATCH v6 4/5] submodule: record superproject gitdir during 'update' Emily Shaffer
2021-11-17  0:57 ` [PATCH v6 5/5] submodule: use config to find superproject worktree Emily Shaffer
2021-11-17 11:43 ` [RFC PATCH 0/2] submodule: test what happens if submodule.superprojectGitDir isn't around Ævar Arnfjörð Bjarmason
2021-11-17 11:43   ` [RFC PATCH 1/2] submodule tests: fix potentially broken "config .. --unset" Ævar Arnfjörð Bjarmason
2021-11-17 11:43   ` [RFC PATCH 2/2] submodule: add test mode for checking absence of "superProjectGitDir" Ævar Arnfjörð Bjarmason
2021-11-23 20:08   ` [RFC PATCH 0/2] submodule: test what happens if submodule.superprojectGitDir isn't around Emily Shaffer
2021-11-24  1:38     ` Ævar Arnfjörð Bjarmason
2021-11-17 23:28 ` [PATCH v6 0/5] teach submodules to know they're submodules Jonathan Tan
2021-11-23 20:28   ` Emily Shaffer
2022-02-03 21:59 ` Emily Shaffer
2022-02-03 21:59   ` [PATCH v7 1/4] t7400-submodule-basic: modernize inspect() helper Emily Shaffer
2022-02-03 21:59   ` [PATCH v7 2/4] introduce submodule.superprojectGitDir record Emily Shaffer
2022-02-03 21:59   ` [PATCH v7 3/4] submodule: record superproject gitdir during absorbgitdirs Emily Shaffer
2022-02-03 21:59   ` [PATCH v7 4/4] submodule: record superproject gitdir during 'update' Emily Shaffer
2022-02-03 22:39   ` [PATCH v6 0/5] teach submodules to know they're submodules Junio C Hamano
2022-02-04  1:15   ` Ævar Arnfjörð Bjarmason
2022-02-04 16:20     ` Junio C Hamano
2022-02-07 19:56     ` Jonathan Nieder
2022-02-07 23:21       ` Junio C Hamano
2022-02-08  1:18         ` Jonathan Nieder
2022-02-08 18:24           ` Junio C Hamano
2022-02-10 22:12             ` Emily Shaffer
2022-02-10 22:53               ` Jonathan Nieder
2022-02-12 20:35       ` Ævar Arnfjörð Bjarmason
2022-02-13  6:25         ` Junio C Hamano
2022-03-01  0:26   ` [PATCH v8 0/3] " Emily Shaffer
2022-03-01  0:26     ` [PATCH v8 1/3] t7400-submodule-basic: modernize inspect() helper Emily Shaffer
2022-03-01  0:26     ` [PATCH v8 2/3] introduce submodule.hasSuperproject record Emily Shaffer
2022-03-01  7:00       ` Junio C Hamano
2022-03-08 20:04         ` Emily Shaffer
2022-03-08 22:13       ` Glen Choo
2022-03-08 22:29         ` Glen Choo
2022-03-01  0:26     ` [PATCH v8 3/3] rev-parse: short-circuit superproject worktree when config unset Emily Shaffer
2022-03-01  7:06       ` Junio C Hamano
2022-03-09  0:38         ` Emily Shaffer
2022-03-01  3:08     ` [PATCH v8 0/3] teach submodules to know they're submodules Junio C Hamano
2022-03-08 18:54       ` Emily Shaffer
2022-03-10  0:44     ` [PATCH v9 " Emily Shaffer
2022-03-10  0:44       ` [PATCH v9 1/3] t7400-submodule-basic: modernize inspect() helper Emily Shaffer
2022-03-10  0:44       ` [PATCH v9 2/3] introduce submodule.hasSuperproject record Emily Shaffer
2022-03-10  2:09         ` Junio C Hamano
2022-03-10 21:29           ` Glen Choo
2022-03-10 21:40           ` Glen Choo
2022-03-10 22:10             ` Junio C Hamano
2022-03-10 23:42               ` Glen Choo
2022-03-10 23:53                 ` Glen Choo
2022-03-15 20:48                   ` Emily Shaffer
2022-03-15 20:56                     ` Emily Shaffer
2022-03-15 21:19                       ` Glen Choo
2022-03-15 18:39               ` Emily Shaffer
2022-03-15 19:19                 ` Junio C Hamano
2022-03-10  2:32         ` Junio C Hamano
2022-03-10 21:54         ` Glen Choo
2022-03-15 18:27           ` Emily Shaffer
2022-03-10  0:44       ` [PATCH v9 3/3] rev-parse: short-circuit superproject worktree when config unset Emily Shaffer
2022-03-10  1:47         ` Junio C Hamano
2022-03-10  4:39           ` Eric Sunshine
2022-03-11  9:09       ` [PATCH v9 0/3] teach submodules to know they're submodules Ævar Arnfjörð Bjarmason
2022-03-13  5:43         ` Junio C Hamano

Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).