nested submodules detection w/o .gitmodules file

nested submodules detection w/o .gitmodules file

[Andry]

Hello Git,

I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules:

`svn externals replacement` : https://github.com/chronoxor/gil/issues/6
`svn complete replacement for externals` : https://github.com/dirk-thomas/vcstool/issues/243

And stumbled on this discussion:

`nested submodules detection w/o .gitmodules file` : https://github.com/gitextensions/gitextensions/discussions/10644 (https://github.com/gitextensions/gitextensions/issues/10642)

The main question here is that, could the git have has submodules without `.submodules` file?

If no, then all side projects which utilizes it's own input file for the externals may subsequentially fail:

https://github.com/chronoxor/gil
https://github.com/dirk-thomas/vcstool
https://github.com/ingydotnet/git-subrepo

If yes, then other projects which does rely on the `.submodules` would have not actual or even invalid state:

https://github.com/gitextensions/gitextensions

Or even the github itself: 

`Zip archive to include submodule` : https://github.com/dear-github/dear-github/issues/214

(`[PATCH] archive: add –recurse-submodules to git-archive command` : https://git.github.io/rev_news/2022/11/30/edition-93/, https://lore.kernel.org/git/pull.1359.git.git.1665597148042.gitgitgadget@gmail.com/)

Mine point here is that:

Git database is a primary storage. The `.gitmodules` file is not a primary storage, so can be in not an actual or desync state with the database.
And any application or a 3dparty project must read the database directly.

But another problem here is that the git still does not have a stable API for that.
For example, a submodule can be declared directly from the `.git/config` file in a working copy: 

https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-init--ltpathgt82308203
https://git-scm.com/docs/gitsubmodules#_active_submodules

So, who is right and what is wrong here?

Re: nested submodules detection w/o .gitmodules file

[Emily Shaffer]

On Sun, Jan 22, 2023 at 12:45 PM Andry <andry@inbox.ru> wrote:
>
> Hello Git,
>
> I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules:
>
> `svn externals replacement` : https://github.com/chronoxor/gil/issues/6
> `svn complete replacement for externals` : https://github.com/dirk-thomas/vcstool/issues/243
>
> And stumbled on this discussion:
>
> `nested submodules detection w/o .gitmodules file` : https://github.com/gitextensions/gitextensions/discussions/10644 (https://github.com/gitextensions/gitextensions/issues/10642)
>
> The main question here is that, could the git have has submodules without `.submodules` file?

There is a little nuance here. Git can have nested repositories in a
few different ways; submodules are just one of them. A submodule is
the combination of a gitlink object in the object graph *and* a
corresponding entry in the .gitmodules file. It's certainly possible
to embed a nested repository in other ways, such as by ignoring the
.gitmodules file, but then your nested repository is no longer a
submodule, and operations which recurse over submodules will not
consider that nested repository. The reason is that we need to
understand where to clone the submodule from - that information isn't
contained in the superproject's repository URL, and it can't be
contained in the gitlink directly (which is in essence just a commit
object, but one that exists in the nested repository, not in the
superproject repository). If we didn't have a way of writing down the
submodule's remote URL and version controlling it, we wouldn't have a
way to populate the submodules when a user is cloning.

>
> If no, then all side projects which utilizes it's own input file for the externals may subsequentially fail:
>
> https://github.com/chronoxor/gil
> https://github.com/dirk-thomas/vcstool
> https://github.com/ingydotnet/git-subrepo
>
> If yes, then other projects which does rely on the `.submodules` would have not actual or even invalid state:
>
> https://github.com/gitextensions/gitextensions
>
> Or even the github itself:
>
> `Zip archive to include submodule` : https://github.com/dear-github/dear-github/issues/214
>
> (`[PATCH] archive: add –recurse-submodules to git-archive command` : https://git.github.io/rev_news/2022/11/30/edition-93/, https://lore.kernel.org/git/pull.1359.git.git.1665597148042.gitgitgadget@gmail.com/)
>
> Mine point here is that:
>
> Git database is a primary storage. The `.gitmodules` file is not a primary storage, so can be in not an actual or desync state with the database.
> And any application or a 3dparty project must read the database directly.

The .gitmodules exists to help at clone time; it's possible, as I
think you're pointing out, to have some intermediate state locally.
But this file is what needs to be the source of truth for putting
together the repository on a new machine for the first time.

>
> But another problem here is that the git still does not have a stable API for that.
> For example, a submodule can be declared directly from the `.git/config` file in a working copy:
>
> https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-init--ltpathgt82308203
> https://git-scm.com/docs/gitsubmodules#_active_submodules
>
> So, who is right and what is wrong here?
>

Re: nested submodules detection w/o .gitmodules file

[Andry]

Hello Emily,



Monday, January 23, 2023, 11:12:32 PM, you wrote:

ES> There is a little nuance here. Git can have nested repositories in a few different ways; submodules are just one of them.
ES> A submodule is the combination of a gitlink object in the object graph *and* a corresponding entry in the .gitmodules file.
ES> It's certainly possible to embed a nested repository in other ways, such as by ignoring the .gitmodules file, but then your
ES> nested repository is no longer a submodule, and operations which recurse over submodules will not consider that nested repository.

But it does consider. See the output of the `git submodule summary` command:
https://github.com/gitextensions/gitextensions/discussions/10644#discussioncomment-4688533

ES> The reason is that we need to understand where to clone the submodule from - that information isn't contained in the superproject's
ES> repository URL

Yes, it does clone "externally", but after a clone it is contained in the `.git/config` of a subproject:

[submodule]
        active = .
[remote "origin"]
        url = ...

ES> , and it can't be contained in the gitlink directly (which is in essence just a commit object, but one that exists
ES> in the nested repository, not in the superproject repository). If we didn't have a way of writing down the submodule's remote URL
ES> and version controlling it, we wouldn't have a way to populate the submodules when a user is cloning.

The `git submodule summary` nevertheless can populate submodules after that, so this is not the matter.

ES> The .gitmodules exists to help at clone time; it's possible, as I
ES> think you're pointing out, to have some intermediate state locally.
ES> But this file is what needs to be the source of truth for putting
ES> together the repository on a new machine for the first time.

Yes, the original clone command is superseded in those projects.

Re: nested submodules detection w/o .gitmodules file

[Andry]

Hello Git,



Sunday, January 22, 2023, 11:34:03 PM, you wrote:

A> Hello Git,

A> I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules:
A> ...
A> So, who is right and what is wrong here?

Is nobody interested in the issue?