Re: [PATCH v2 02/10] submodule--helper: don't use global --super-prefix in "absorbgitdirs"

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Mon, Nov 14 2022, Glen Choo wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> The "--super-prefix" facility was introduced in [1] has always been a
>> transitory hack, which is why we've made it an error to supply it as
>> an option to "git" to commands that don't know about it.
>>
>> That's been a good goal, as it has a global effect we haven't wanted
>> calls to get_super_prefix() from built-ins we didn't expect.
>>
>> But it has meant that when we've had chains of different built-ins
>> using it all of the processes in that "chain" have needed to support
>> it, and worse processes that don't need it have needed to ask for
>> "SUPPORT_SUPER_PREFIX" because their parent process needs it.
>>
>> That's how "fsmonitor--daemon" ended up with it, per [2] it's called
>> from (among other things) "submodule--helper absorbgitdirs", but as we
>> declared "submodule--helper" as "SUPPORT_SUPER_PREFIX" we needed to
>> declare "fsmonitor--daemon" as accepting it too, even though it
>> doesn't care about it.
>>
>> But in the case of "absorbgitdirs" it only needed "--super-prefix" to
>> invoke itself recursively, and we'd never have another "in-between"
>> process in the chain. So we didn't need the bigger hammer of "git
>> --super-prefix", and the "setenv(GIT_SUPER_PREFIX_ENVIRONMENT, ...)"
>> that it entails.
>>
>> Let's instead accept a hidden "--super-prefix" option to
>> "submodule--helper absorbgitdirs" itself.
>>
>> Eventually (as with all other "--super-prefix" users) we'll want to
>> clean this code up so that this all happens in-process. I.e. needing
>> any variant of "--super-prefix" is itself a hack around our various
>> global state, and implicit reliance on "the_repository". This stepping
>> stone makes such an eventual change easier, as we'll need to deal with
>> less global state at that point.
>>
>> The "fsmonitor--daemon" test adjusted here was added in [3]. The
>> comment added in that commit has been out-of-date from the beginning,
>> and the "have_t2_error_event()" was being overly specific in testing
>> for a bug that we *don't* have. Let's instead test for the stdout and
>> stderr that we *do have*.
>
> I didn't understand this bit initially, because I read this as
> "have_t2_error_event() isn't catching bugs", which isn't true. But I see
> what you mean after inspecting the test_cmp output:
>
>   @@ -1 +1,2 @@
>   Migrating git directory of 'dir_1/dir_2/sub' from 'dir_1/dir_2/sub/.git' to '.git/modules/dir_1/dir_2/sub'
>   +fatal: fsmonitor--daemon doesn't support --super-prefix
>
> IOW, it doesn't make sense to inspecting the tr2 output for stray
> warnings when our stderr is so obviously broken. But at the end of the
> series, I don't think we even need this test at all because if we don't
> have a global "--super-prefix", there's literally no reason for
> fsmonitor--daemon to worry about absorbgitdirs or "supporting" the super
> prefix. That's why I removed it in [1].
>
> Wondering aloud, that stderr suggests that the "git submodule absorbgitdirs"
> invocation failed. I wonder why the test didn't catch a bad exit code
> (and resulting in us inspecting stderr unnecessarily). I think it's not
> worth looking too closely at right now, but we could revisit this after
> we get builtin/submodule.c.
>
> [1] https://lore.kernel.org/git/20221109004708.97668-5-chooglen@google.com

I'm prepping a re-roll where I'll keep some of this test, as testing
fsmonitor+absorbgitdirs is worthhwile, but the commit message etc. is
rephrased.

>> diff --git a/git.c b/git.c
>> index 10202a7f126..b1b7e1a837e 100644
>> --- a/git.c
>> +++ b/git.c
>> @@ -539,7 +539,7 @@ static struct cmd_struct commands[] = {
>>  	{ "format-patch", cmd_format_patch, RUN_SETUP },
>>  	{ "fsck", cmd_fsck, RUN_SETUP },
>>  	{ "fsck-objects", cmd_fsck, RUN_SETUP },
>> -	{ "fsmonitor--daemon", cmd_fsmonitor__daemon, SUPPORT_SUPER_PREFIX | RUN_SETUP },
>> +	{ "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP },
>>  	{ "gc", cmd_gc, RUN_SETUP },
>>  	{ "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT },
>>  	{ "grep", cmd_grep, RUN_SETUP_GENTLY },
>> diff --git a/parse-options.h b/parse-options.h
>> index b6ef86e0d15..50d852f2991 100644
>> --- a/parse-options.h
>> +++ b/parse-options.h
>> @@ -369,6 +369,10 @@ int parse_opt_tracking_mode(const struct option *, const char *, int);
>>  	{ OPTION_CALLBACK, 0, "abbrev", (var), N_("n"),	\
>>  	  N_("use <n> digits to display object names"),	\
>>  	  PARSE_OPT_OPTARG, &parse_opt_abbrev_cb, 0 }
>> +#define OPT__SUPER_PREFIX(var) \
>> +	OPT_STRING_F(0, "super-prefix", (var), N_("prefix"), \
>> +		N_("prefixed path to initial superproject"), PARSE_OPT_HIDDEN)
>> +
>
> Could we default to "" instead of NULL? (possibly via a callback). I
> think there's never any good reason to have NULL instead of "", e.g.
> since this is internal, we don't care to distinguish between
> "--super-prefix=''" and not passing "--super-prefix" at all...

I saw this comment on another patch, I think I'll keep it for now, but I
updated the reason: It's because the "prefix" is either NULL or a
non-zero length string, so we're doing the same here to be consistent
with it.

I think that makes sense, but ultimately it's an arbitrary choice, but
as they're sibling variables in the cases they're used doing the same
seems worthwhile...

>>  #define OPT__COLOR(var, h) \
>>  	OPT_COLOR_FLAG(0, "color", (var), (h))
>>  #define OPT_COLUMN(s, l, v, h) \
>> diff --git a/submodule.c b/submodule.c
>> index c47358097fd..d9fd0af81b6 100644
>> --- a/submodule.c
>> +++ b/submodule.c
>> @@ -2268,7 +2268,8 @@ int validate_submodule_git_dir(char *git_dir, const char *submodule_name)
>>   * Embeds a single submodules git directory into the superprojects git dir,
>>   * non recursively.
>>   */
>> -static void relocate_single_git_dir_into_superproject(const char *path)
>> +static void relocate_single_git_dir_into_superproject(const char *path,
>> +						      const char *super_prefix)
>>  {
>>  	char *old_git_dir = NULL, *real_old_git_dir = NULL, *real_new_git_dir = NULL;
>>  	struct strbuf new_gitdir = STRBUF_INIT;
>> @@ -2302,7 +2303,7 @@ static void relocate_single_git_dir_into_superproject(const char *path)
>>  	       real_old_git_dir[off] == real_new_git_dir[off])
>>  		off++;
>>  	fprintf(stderr, _("Migrating git directory of '%s%s' from '%s' to '%s'\n"),
>> -		get_super_prefix_or_empty(), path,
>> +		(super_prefix ? super_prefix : ""), path,
>>  		real_old_git_dir + off, real_new_git_dir + off);
>
>
> Which would make sites like these a bit cleaner.
>
> Tangentially, 'default to "" if NULL' sounds like a common pattern. Is
> there a good reason not to have a macro or inline function to do that?
> e.g. we think the ternary expression is good enough?)

See above.

>>  
>>  	relocate_gitdir(path, real_old_git_dir, real_new_git_dir);
>> @@ -2313,7 +2314,8 @@ static void relocate_single_git_dir_into_superproject(const char *path)
>>  	strbuf_release(&new_gitdir);
>>  }
>>  
>> -static void absorb_git_dir_into_superproject_recurse(const char *path)
>> +static void absorb_git_dir_into_superproject_recurse(const char *path,
>> +						     const char *super_prefix)
>>  {
>>  
>>  	struct child_process cp = CHILD_PROCESS_INIT;
>> @@ -2321,10 +2323,11 @@ static void absorb_git_dir_into_superproject_recurse(const char *path)
>>  	cp.dir = path;
>>  	cp.git_cmd = 1;
>>  	cp.no_stdin = 1;
>> -	strvec_pushf(&cp.args, "--super-prefix=%s%s/",
>> -		     get_super_prefix_or_empty(), path);
>>  	strvec_pushl(&cp.args, "submodule--helper",
>>  		     "absorbgitdirs", NULL);
>> +	strvec_pushf(&cp.args, "--super-prefix=%s%s/", super_prefix ?
>> +		     super_prefix : "", path);
>> +
>>  	prepare_submodule_repo_env(&cp.env);
>>  	if (run_command(&cp))
>>  		die(_("could not recurse into submodule '%s'"), path);
>> @@ -2335,7 +2338,8 @@ static void absorb_git_dir_into_superproject_recurse(const char *path)
>>   * having its git directory within the working tree to the git dir nested
>>   * in its superprojects git dir under modules/.
>>   */
>> -void absorb_git_dir_into_superproject(const char *path)
>> +void absorb_git_dir_into_superproject_sp(const char *path,
>> +					 const char *super_prefix)
>>  {
>>  	int err_code;
>>  	const char *sub_git_dir;
>> @@ -2377,14 +2381,14 @@ void absorb_git_dir_into_superproject(const char *path)
>>  		char *real_common_git_dir = real_pathdup(get_git_common_dir(), 1);
>>  
>>  		if (!starts_with(real_sub_git_dir, real_common_git_dir))
>> -			relocate_single_git_dir_into_superproject(path);
>> +			relocate_single_git_dir_into_superproject(path, super_prefix);
>>  
>>  		free(real_sub_git_dir);
>>  		free(real_common_git_dir);
>>  	}
>>  	strbuf_release(&gitdir);
>>  
>> -	absorb_git_dir_into_superproject_recurse(path);
>> +	absorb_git_dir_into_superproject_recurse(path, super_prefix);
>>  }
>>  
>>  int get_superproject_working_tree(struct strbuf *buf)
>> diff --git a/submodule.h b/submodule.h
>> index b52a4ff1e73..e5ee13fb06a 100644
>> --- a/submodule.h
>> +++ b/submodule.h
>> @@ -164,7 +164,12 @@ void submodule_unset_core_worktree(const struct submodule *sub);
>>   */
>>  void prepare_submodule_repo_env(struct strvec *env);
>>  
>> -void absorb_git_dir_into_superproject(const char *path);
>> +void absorb_git_dir_into_superproject_sp(const char *path,
>> +					 const char *super_prefix);
>> +static inline void absorb_git_dir_into_superproject(const char *path)
>> +{
>> +	absorb_git_dir_into_superproject_sp(path, NULL);
>> +}
>
> Is there a reason you chose to go with _sp() instead of changing the
> original function signature? I tested out that change, and it seems
> rather small (absorb_git_dir_into_superproject() only has 4 call sites).

I've just changed that, so no more *_sp()...

> Crucially, changing the signature catches a "git read-tree" call site:
>
>   diff --git a/submodule.c b/submodule.c
>   index d9fd0af81b..e79a04d3e3 100644
>   --- a/submodule.c
>   +++ b/submodule.c
>   @@ -2139,7 +2139,9 @@ int submodule_move_head(const char *path,
>           if (!(flags & SUBMODULE_MOVE_HEAD_DRY_RUN)) {
>                   if (old_head) {
>                           if (!submodule_uses_gitfile(path))
>   -                               absorb_git_dir_into_superproject(path);
>   +                               /* Pass super_prefix properly later. */
>   +                               absorb_git_dir_into_superproject(path,
>   +                                                                get_super_prefix());
>                   } else {
>                           struct strbuf gitdir = STRBUF_INIT;
>                           submodule_name_to_gitdir(&gitdir, the_repository,
>
> which would otherwise be broken since we used to read the global super
> prefix, but we don't do that in this patch. This has no test coverage
> (bleh), but it shouldn't be too hard, something like:
>
>   git init super &&
>   cd super &&
>   # Create an unabsorbed submodule right in the worktree
>   git init sub &&
>   test_commit -C sub "foo" &&
>   git add sub &&
>   git commit -m "Add submodule" &&
>   test_commit "bar" &&
>   # This should invoke "git read-tree" and absorb the git dir.
>   # Or maybe we should invoke "git read-tree" directly?
>   git checkout --recurse-submodules HEAD^ 2>err &&
>   # Search for the shortened message
>   grep "Migrating submodule sub from sub to .git/modules/...." err

Ah! Yeah that is a good catch, and a great reason to do it this way.

I tried to come up with a test case that actually tested this. I.e. I
set a "here = 1" at that absorb_git_dir_into_superproject() in the diff
above, and then if BUG()'d out in that function, but the part where
we're about to recurse.

I couldn't get it to trigger, but I didn't look at it very
long. I.e. doesn't this require not this sort of test, but something
where it's being absorbed, *and* we'd walk and absorb it recursiveyl?

At least with checkout we then run into an error (can't remember the
message, sorry, this was earlier), and I haven't found a way to trigger
it.

But I'll poke it some more, but if you happen to have a test case for it
that would really help...