git@vger.kernel.org mailing list mirror (one of many)
 help / Atom feed
Search results ordered by [date|relevance]  view[summary|threaded|Atom feed]

* [GSoC] Update: Week-13
@ 2017-08-15 23:04 Prathamesh Chavan
  0 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-15 23:04 UTC (permalink / raw)
  To: git; +Cc: Christian Couder, Stefan Beller

SUMMARY OF MY PROJECT:

Git submodule subcommands are currently implemented by using shell script
'git-submodule.sh'. There are several reasons why we'll prefer not to
use the shell script. My project intends to convert the subcommands into
C code, thus making them builtins. This will increase Git's portability
and hence the efficiency of working with the git-submodule commands.
Link to the complete proposal: [1]

Mentors:
Stefan Beller <sbeller@google.com>
Christian Couder <christian.couder@gmail.com>

UPDATES:

Following are the updates about my ongoing project:

* add: most of the time of the week was spent in debugging the
  ported submodule subcommand functions. But, even after so,
  currently the ported functions fail in total 15 tests from
  the test suit. The WIP patches have been updated regularly,
  and currently the patch is under discussion with the mentors
  as well. The current status of the patch is pushed on github as
  well, and can be viewed at:[2]

Since the rest of the patches were almost the same as that in the
previous update(except for the 'summary' patch, which was last
updated after Christian's review), the haven't been uploaded
again to avoid unnecessary floating patches.
The previous updated series is available at: [3]
But given that, next week I plan to float a separate patch
series, containing the initial patches from the subcommand porting
series (till deinit), which IMO, are ready for maintainer's
review.

Also, this week's update was one-day late, since I was traveling
on the previous day. But a prior idea about this was given to the
mentors.

PLAN FOR WEEK-14 (15 August 2017 to 21 August 2017):

* patches: Float a separate series, till deinit, and ask the maintainer
  for its review.

* add: The main aim of the next week is to resolve the issues with
  the current patch, and get all the tests pass.

* update: it is the last remaining subcommand to be ported. I aim to
  atleast start with this in the following week.

The work till week-13 is pushed on Github and is available at: [4].

[1]: https://docs.google.com/document/d/1krxVLooWl--75Pot3dazhfygR3wCUUWZWzTXtK1L-xU/
[2]: https://github.com/pratham-pc/git/commits/sub-add
[3]: https://public-inbox.org/git/20170807211900.15001-1-pc44800@gmail.com/
[4]: https://github.com/pratham-pc/git/commits/week-13

^ permalink raw reply	[relevance 16%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-15 22:43 ` [PATCH] push: do not add submodule odb as an alternate when recursing on demand Stefan Beller
@ 2017-08-15 23:10   ` Jonathan Nieder
  2017-08-15 23:14   ` Jonathan Nieder
  2017-08-15 23:23   ` Junio C Hamano
  2 siblings, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-08-15 23:10 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster

Stefan Beller wrote:

> "git push --recurse-submodules=on-demand" adds each submodule as an
> alternate with add_submodule_odb before checking whether the
> submodule has anything to push and pushing it if so.
>
> However, it never accesses any objects from the submodule.  In the
> parent process it uses the submodule's ref database to see if there
> is anything to push.  The actual push (which does rely on objects)
> occurs in a child process.
>
> The same was true when this call was originally added in
> v1.7.11-rc0~111^2 (push: teach --recurse-submodules the on-demand
> option, 2012-03-29).  Most likely it was added by analogy with
> fetch --recurse-submodules=on-demand, which did use the submodule's
> object database.
>
> Use is_submodule_populated_gently instead, which is simpler and
> cheaper.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  submodule.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Thanks for picking up this loose end.

> diff --git a/submodule.c b/submodule.c
> index 111a3007fc..e20216bc0f 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -966,7 +966,9 @@ static int push_submodule(const char *path,
>  			  const struct string_list *push_options,
>  			  int dry_run)
>  {
> -	if (add_submodule_odb(path))
> +	int code;
> +
> +	if (!is_submodule_populated_gently(path, &code))
>  		return 1;
>  
>  	if (for_each_remote_ref_submodule(path, has_remote, NULL) > 0) {

^ permalink raw reply	[relevance 15%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-15 22:43 ` [PATCH] push: do not add submodule odb as an alternate when recursing on demand Stefan Beller
  2017-08-15 23:10   ` Jonathan Nieder
@ 2017-08-15 23:14   ` Jonathan Nieder
  2017-08-15 23:27     ` Stefan Beller
  2017-08-15 23:23   ` Junio C Hamano
  2 siblings, 1 reply; 200+ results
From: Jonathan Nieder @ 2017-08-15 23:14 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster

Stefan Beller wrote:

> Use is_submodule_populated_gently instead, which is simpler and
> cheaper.
[...]
> --- a/submodule.c
> +++ b/submodule.c
> @@ -966,7 +966,9 @@ static int push_submodule(const char *path,
>  			  const struct string_list *push_options,
>  			  int dry_run)
>  {
> -	if (add_submodule_odb(path))
> +	int code;
> +
> +	if (!is_submodule_populated_gently(path, &code))
>  		return 1;

Ah, I forgot about this detail.  I don't think it should block this
patch (so my Reviewed-by still stands), but I wonder why this needs to
be gentle.  add_submodule_odb is gentle so that is the conservative
thing to do, but that doesn't mean it is the *right* thing to do.

If this passed NULL instead of &code as the second argument, would
anything break?

Could there be a comment explaining what kind of error we are
expecting and why it is okay to continue when that error is
encountered without any error handling?

Thanks,
Jonathan

^ permalink raw reply	[relevance 15%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-15 22:43 ` [PATCH] push: do not add submodule odb as an alternate when recursing on demand Stefan Beller
  2017-08-15 23:10   ` Jonathan Nieder
  2017-08-15 23:14   ` Jonathan Nieder
@ 2017-08-15 23:23   ` Junio C Hamano
  2017-08-15 23:31     ` Stefan Beller
  2 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2017-08-15 23:23 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, jrnieder

Stefan Beller <sbeller@google.com> writes:

> "git push --recurse-submodules=on-demand" adds each submodule as an
> alternate with add_submodule_odb before checking whether the
> submodule has anything to push and pushing it if so.
>
> However, it never accesses any objects from the submodule.
> ...
> Use is_submodule_populated_gently instead, which is simpler and
> cheaper.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>
>  Originally I intended to send this out as part of a larger series,
>  but the series is getting too large series, sending all things in smaller
>  units!

This vaguely reminds me that you sent something imilar perhaps for a
different codepath.

Is "is it populated" a good thing to check here, though?  IIRC,
add-submodule-odb allows you to add the object database of an
inactivated submodule, so this seems to change the behaviour.  I do
not know if the behaviour change is a good thing (i.e. bugfix) or
not (i.e. regression) offhand, though.

Thanks.

> diff --git a/submodule.c b/submodule.c
> index 111a3007fc..e20216bc0f 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -966,7 +966,9 @@ static int push_submodule(const char *path,
>  			  const struct string_list *push_options,
>  			  int dry_run)
>  {
> -	if (add_submodule_odb(path))
> +	int code;
> +
> +	if (!is_submodule_populated_gently(path, &code))
>  		return 1;
>  
>  	if (for_each_remote_ref_submodule(path, has_remote, NULL) > 0) {

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-15 23:14   ` Jonathan Nieder
@ 2017-08-15 23:27     ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-15 23:27 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: git, Junio C Hamano

On Tue, Aug 15, 2017 at 4:14 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Stefan Beller wrote:
>
>> Use is_submodule_populated_gently instead, which is simpler and
>> cheaper.
> [...]
>> --- a/submodule.c
>> +++ b/submodule.c
>> @@ -966,7 +966,9 @@ static int push_submodule(const char *path,
>>                         const struct string_list *push_options,
>>                         int dry_run)
>>  {
>> -     if (add_submodule_odb(path))
>> +     int code;
>> +
>> +     if (!is_submodule_populated_gently(path, &code))
>>               return 1;
>
> Ah, I forgot about this detail.  I don't think it should block this
> patch (so my Reviewed-by still stands), but I wonder why this needs to
> be gentle.  add_submodule_odb is gentle so that is the conservative
> thing to do, but that doesn't mean it is the *right* thing to do.
>
> If this passed NULL instead of &code as the second argument, would
> anything break?

push_submodule, which is called by push_unpushed_submodules
just issues warnings on submodule push error, (which happen all
before the main push,) such that any submodule error does not prevent
the main push.

We could tighten that, but I would suggest another patch for that.

> Could there be a comment explaining what kind of error we are
> expecting and why it is okay to continue when that error is
> encountered without any error handling?

The same as in add_submodule_odb, digging through the
logs. I up to now we did not care about the submodule succeeding
as much, it was just an aid for the main repo push.

>
> Thanks,
> Jonathan

^ permalink raw reply	[relevance 25%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-15 23:23   ` Junio C Hamano
@ 2017-08-15 23:31     ` Stefan Beller
  2017-08-16  0:11       ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-15 23:31 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git, Jonathan Nieder

On Tue, Aug 15, 2017 at 4:23 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> "git push --recurse-submodules=on-demand" adds each submodule as an
>> alternate with add_submodule_odb before checking whether the
>> submodule has anything to push and pushing it if so.
>>
>> However, it never accesses any objects from the submodule.
>> ...
>> Use is_submodule_populated_gently instead, which is simpler and
>> cheaper.
>>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> ---
>>
>>  Originally I intended to send this out as part of a larger series,
>>  but the series is getting too large series, sending all things in smaller
>>  units!
>
> This vaguely reminds me that you sent something imilar perhaps for a
> different codepath.

https://public-inbox.org/git/xmqqh8xzq6td.fsf@gitster.mtv.corp.google.com/

> Is "is it populated" a good thing to check here, though?  IIRC,
> add-submodule-odb allows you to add the object database of an
> inactivated submodule, so this seems to change the behaviour.  I do
> not know if the behaviour change is a good thing (i.e. bugfix) or
> not (i.e. regression) offhand, though.

Good point, we should be able to push non-populated, even inactive(?)
submodules. For that we strictly need add_submodule_odb here
(or the repo object of the submodule, eventually).

So let's retract this patch for now.

Thanks.

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-15 23:31     ` Stefan Beller
@ 2017-08-16  0:11       ` Junio C Hamano
  2017-08-16  1:05         ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2017-08-16  0:11 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git\, Jonathan Nieder

Stefan Beller <sbeller@google.com> writes:

>> Is "is it populated" a good thing to check here, though?  IIRC,
>> add-submodule-odb allows you to add the object database of an
>> inactivated submodule, so this seems to change the behaviour.  I do
>> not know if the behaviour change is a good thing (i.e. bugfix) or
>> not (i.e. regression) offhand, though.
>
> Good point, we should be able to push non-populated, even inactive(?)
> submodules. For that we strictly need add_submodule_odb here
> (or the repo object of the submodule, eventually).
>
> So let's retract this patch for now.

Not so fast.  

I am not convinced "push --recursive" should touch a submodule that
was once cloned from the upstream and then deactivated, so using
add-submodule-odb to decide if the push should go through may be a
bug that we may want to fix, in which case the diff of the patch in
question may be good as-is.  We need to sell it as a bugfix to the
users, who may complain about behaviour change (if there is one).

On the other hand, even if it were desirable for such a deactivated
submodule to be pushed, as your log message explained, there is no
reason to contaminate the in-core object hash by calling the
add-submodule-odb helper, when the only thing we care about is "do
we have the refs and object store for this submodule? we do not care
if it is activated or not".  Perhaps there is a more appropriate
helper in submodule.c that answers that question that we should be
using instead of add-submodule-odb, and if there is not yet such a
helper, perhaps this indicates that we need to add such a helper,
which essentially is the early half of what add-submodule-odb does,
i.e. ask git_path_submodule() for the object store and check if that
directory exists.

Thanks.

^ permalink raw reply	[relevance 25%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-16  0:11       ` Junio C Hamano
@ 2017-08-16  1:05         ` Stefan Beller
  2017-08-16  2:08           ` Jonathan Nieder
  2017-08-16 16:35           ` Heiko Voigt
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2017-08-16  1:05 UTC (permalink / raw)
  To: Junio C Hamano, Jonathan Tan; +Cc: git, Jonathan Nieder

On Tue, Aug 15, 2017 at 5:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>>> Is "is it populated" a good thing to check here, though?  IIRC,
>>> add-submodule-odb allows you to add the object database of an
>>> inactivated submodule, so this seems to change the behaviour.  I do
>>> not know if the behaviour change is a good thing (i.e. bugfix) or
>>> not (i.e. regression) offhand, though.
>>
>> Good point, we should be able to push non-populated, even inactive(?)
>> submodules. For that we strictly need add_submodule_odb here
>> (or the repo object of the submodule, eventually).
>>
>> So let's retract this patch for now.
>
> Not so fast.

Ok, I took another look at the code.

While we may desire that un-populated submodules can be pushed
(due to checking out another revision where the submodule
doesn't exist, before pushing), this is not supported currently, because
the call to run the push in the submodule assumes there is a
"<path>/.git" on which the child process can operate.
So for now we HAVE to have the submodule populated.

  In the future we may have the a lighter version just checking the
  object store of the submodule. Maybe this use case in the submodule
  can be covered by the .remote/.imported mechanism as well, such
  that we'd know if we have any local objects.

> I am not convinced "push --recursive" should touch a submodule that
> was once cloned from the upstream and then deactivated, so using
> add-submodule-odb to decide if the push should go through may be a
> bug that we may want to fix, in which case the diff of the patch in
> question may be good as-is.  We need to sell it as a bugfix to the
> users, who may complain about behaviour change (if there is one).

correct. Coupled with the observation above, we want really need
both "is active and populated" (your paragraph suggest not-active
submodules don't need pushing, the populated bit comes from the
observation above)

> On the other hand, even if it were desirable for such a deactivated
> submodule to be pushed, as your log message explained, there is no
> reason to contaminate the in-core object hash by calling the
> add-submodule-odb helper, when the only thing we care about is "do
> we have the refs and object store for this submodule? we do not care
> if it is activated or not".  Perhaps there is a more appropriate
> helper in submodule.c that answers that question that we should be
> using instead of add-submodule-odb, and if there is not yet such a
> helper, perhaps this indicates that we need to add such a helper,
> which essentially is the early half of what add-submodule-odb does,
> i.e. ask git_path_submodule() for the object store and check if that
> directory exists.

I think due to the constraint of also needing a worktree the is_populated
helper is the correct choice here.


>
> Thanks.

^ permalink raw reply	[relevance 26%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-16  1:05         ` Stefan Beller
@ 2017-08-16  2:08           ` Jonathan Nieder
  2017-08-16  5:52             ` Stefan Beller
  2017-08-16 16:35           ` Heiko Voigt
  1 sibling, 1 reply; 200+ results
From: Jonathan Nieder @ 2017-08-16  2:08 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, Jonathan Tan, git

Hi,

Stefan Beller wrote:
> On Tue, Aug 15, 2017 at 5:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Stefan Beller <sbeller@google.com> writes:
>>> Junio C Hamano wrote:

>>>> Is "is it populated" a good thing to check here, though?  IIRC,
>>>> add-submodule-odb allows you to add the object database of an
>>>> inactivated submodule, so this seems to change the behaviour.  I do
>>>> not know if the behaviour change is a good thing (i.e. bugfix) or
>>>> not (i.e. regression) offhand, though.
>>>
>>> Good point, we should be able to push non-populated, even inactive(?)
>>> submodules. For that we strictly need add_submodule_odb here
>>> (or the repo object of the submodule, eventually).
>>>
>>> So let's retract this patch for now.
>>
>> Not so fast.
>
> Ok, I took another look at the code.
>
> While we may desire that un-populated submodules can be pushed
> (due to checking out another revision where the submodule
> doesn't exist, before pushing), this is not supported currently, because
> the call to run the push in the submodule assumes there is a
> "<path>/.git" on which the child process can operate.
> So for now we HAVE to have the submodule populated.

It was not immediately obvious to me that this is just "for now".

I would be really confused if I had deactivated a submodule and
"git push --recurse-submodules" pushed from it anyway.  If the
submodule is active but not populated, then the question becomes "Why
wasn't it populated?"

If this is a bare repository, then nothing is populated, and pushing
from an active-but-unpopulated submodule sounds like a plausible wish.
But in a non-bare repository, I'm having trouble imagining the use
case that brings this situation about.

And where people have been needing this so far has been non-bare
repositories.  In that context, the check "is active and populated"
does not seem unusual or provisional.  Are you hinting that replacing
the check with "is active" would make it work well in bare
repositories?  I think I agree, though you'd have to be careful about
the case where the submodule is active but hasn't been fetched to
$GIT_DIR/modules yet.

Thanks,
Jonathan

^ permalink raw reply	[relevance 25%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-16  2:08           ` Jonathan Nieder
@ 2017-08-16  5:52             ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-16  5:52 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: Junio C Hamano, Jonathan Tan, git

On Tue, Aug 15, 2017 at 7:08 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi,
>
> Stefan Beller wrote:
>> On Tue, Aug 15, 2017 at 5:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>> Stefan Beller <sbeller@google.com> writes:
>>>> Junio C Hamano wrote:
>
>>>>> Is "is it populated" a good thing to check here, though?  IIRC,
>>>>> add-submodule-odb allows you to add the object database of an
>>>>> inactivated submodule, so this seems to change the behaviour.  I do
>>>>> not know if the behaviour change is a good thing (i.e. bugfix) or
>>>>> not (i.e. regression) offhand, though.
>>>>
>>>> Good point, we should be able to push non-populated, even inactive(?)
>>>> submodules. For that we strictly need add_submodule_odb here
>>>> (or the repo object of the submodule, eventually).
>>>>
>>>> So let's retract this patch for now.
>>>
>>> Not so fast.
>>
>> Ok, I took another look at the code.
>>
>> While we may desire that un-populated submodules can be pushed
>> (due to checking out another revision where the submodule
>> doesn't exist, before pushing), this is not supported currently, because
>> the call to run the push in the submodule assumes there is a
>> "<path>/.git" on which the child process can operate.
>> So for now we HAVE to have the submodule populated.
>
> It was not immediately obvious to me that this is just "for now".
>
> I would be really confused if I had deactivated a submodule and
> "git push --recurse-submodules" pushed from it anyway.

agreed.

>  If the
> submodule is active but not populated, then the question becomes "Why
> wasn't it populated?"

because you checked out a different revision where the sub is not
part of the working tree, for example. As long as you have locally
interesting "stuff" you want to transfer it. (IMHO even when the sub
is not active)

> If this is a bare repository, then nothing is populated,

Currently bare repos do not support having submodules
AT ALL, i.e. there is no modules/<name> directory
where the submodules would have any valuable information.
instead we *only* have the .gitmodules files at e.g. HEAD.

> and pushing
> from an active-but-unpopulated submodule sounds like a plausible wish.

it seems we're on the same page here.

> But in a non-bare repository, I'm having trouble imagining the use
> case that brings this situation about.

git checkout revision-with-sub
# hack.. hack.. hack..
"git commit --recurse-submodules -m bugfix"
# ^ not yet implemented
git checkout revision-without-sub
# Oh! I forgot to push the bugfix
git push --recurse-submodule <bugfix>

> And where people have been needing this so far has been non-bare
> repositories.  In that context, the check "is active and populated"
> does not seem unusual or provisional.  Are you hinting that replacing
> the check with "is active" would make it work well in bare
> repositories?

No I was saying:
* currently the code only works when "is populated"
* currently "is active" seems not to be considered

* in the future the code shall work even when not "is populated",
  (but submodule git dir is there at the very least)
* we may want to discuss if we want to care about "is active"
  additionally

>  I think I agree, though you'd have to be careful about
> the case where the submodule is active but hasn't been fetched to
> $GIT_DIR/modules yet.

When not fetched yet, there is no modules/<name> git dir and
we are sure there is no locally interesting information, so we
want to omit the push. (see above "has git dir" is the minimum
requirement, as opposed to current "is populated".

> Thanks,
> Jonathan

^ permalink raw reply	[relevance 25%]

* Re: [PATCH] push: do not add submodule odb as an alternate when recursing on demand
  2017-08-16  1:05         ` Stefan Beller
  2017-08-16  2:08           ` Jonathan Nieder
@ 2017-08-16 16:35           ` Heiko Voigt
  1 sibling, 0 replies; 200+ results
From: Heiko Voigt @ 2017-08-16 16:35 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, Jonathan Tan, git, Jonathan Nieder

Hi,

was about to write that we are maybe overly cautious here. Because the
current way a submodule ends up in the list to be pushed is through:

    find_unpushed_submodules()

that itself collects all changed submodules when submodule_needs_pushing() is
true. In there we have this:

    if (!submodule_has_commits(path, commits))
        /*
         * NOTE: We do consider it safe to return "no" here. The
         * correct answer would be "We do not know" instead of
         * "No push needed", but it is quite hard to change
         * the submodule pointer without having the submodule
         * around. If a user did however change the submodules
         * without having the submodule around, this indicates
         * an expert who knows what they are doing or a
         * maintainer integrating work from other people. In
         * both cases it should be safe to skip this check.
         */
        return 0;

So if the check, whether a submodule has commits, fails for any reason it will
not end up in the list to be pushed.

As a side note: inside submodule_has_commits() there is an add_submodule_odb()
followed by a process to really make sure that the commits are in the
submodule.

So IMO at this point we can be sure that the *database* exists and this extra
check could be dropped if we said that a caller to push_submodule() should make
sure that the submodule exists. The current ones are doing it already (if I did
not miss anything).

On Tue, Aug 15, 2017 at 06:05:25PM -0700, Stefan Beller wrote:
> On Tue, Aug 15, 2017 at 5:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
> > Stefan Beller <sbeller@google.com> writes:
> >
> >>> Is "is it populated" a good thing to check here, though?  IIRC,
> >>> add-submodule-odb allows you to add the object database of an
> >>> inactivated submodule, so this seems to change the behaviour.  I do
> >>> not know if the behaviour change is a good thing (i.e. bugfix) or
> >>> not (i.e. regression) offhand, though.
> >>
> >> Good point, we should be able to push non-populated, even inactive(?)
> >> submodules. For that we strictly need add_submodule_odb here
> >> (or the repo object of the submodule, eventually).
> >>
> >> So let's retract this patch for now.
> >
> > Not so fast.
> 
> Ok, I took another look at the code.
> 
> While we may desire that un-populated submodules can be pushed
> (due to checking out another revision where the submodule
> doesn't exist, before pushing), this is not supported currently, because
> the call to run the push in the submodule assumes there is a
> "<path>/.git" on which the child process can operate.
> So for now we HAVE to have the submodule populated.

That is a good point though. In the current form of push_submodule() we need to
have a populated submodule. So IMO to check whether the submodule is actually
*populated* instead of adding the odb is correct and a possible bug fix.

Cheers Heiko

^ permalink raw reply	[relevance 24%]

* Re: Submodule regression in 2.14?
  2017-08-16 18:35 Submodule regression in 2.14? Lars Schneider
@ 2017-08-16 18:51 ` Stefan Beller
  2017-08-16 18:53   ` Stefan Beller
  2017-08-17 21:21   ` Lars Schneider
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2017-08-16 18:51 UTC (permalink / raw)
  To: Lars Schneider; +Cc: git, Junio C Hamano

On Wed, Aug 16, 2017 at 11:35 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
> Hi,
>
> I think we discovered a regression in Git 2.14.1 today.
> It looks like as if "git submodule update --init --recursive" removes
> the "skip submodules" config.
>
> Consider the following steps:
>
>     git clone https://server/repo.git
>     cd repo
>     git config --local submodule.some/other/repo.update none
>     git submodule update --init --recursive
>     git pull --recurse-submodules
>
> With Git 2.14 the last "git pull" will clone the "some/other/repo"
> submodule. This did not happen with Git 2.13.
>
> Bug or feature? I don't have anymore time for Git today. I am happy to
> provide a proper test case tomorrow, though.

$ git log --oneline v2.13.0..v2.14.1 -- git-submodule.sh
532139940c add: warn when adding an embedded repository
(I am confident this is not the suspect, let's keep searching.
Not a lot happened in submodule land apparently)

Looking through all commits v2.13..v2.14 doesn't have me
suspect any of them.

Any chance the "did not happen with 2.13" was not
freshly cloned but tested on an existing repo? If so a hot
candidate for suspicion is a93dcb0a56 (Merge branch
'bw/submodule-is-active', 2017-03-30), IMHO, just
gut feeling, though.

Oh, wait.
$ git log --oneline v2.13.0..v2.14.1 -- builtin/pull.c
c9c63ee558 Merge branch 'sb/pull-rebase-submodule'
a6d7eb2c7a pull: optionally rebase submodules (remote submodule changes only)
could also be a culprit. Do you have pull.rebase set?

>
> Cheers,
> Lars

^ permalink raw reply	[relevance 26%]

* Re: Submodule regression in 2.14?
  2017-08-16 18:51 ` Stefan Beller
@ 2017-08-16 18:53   ` Stefan Beller
  2017-08-17 21:21   ` Lars Schneider
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-16 18:53 UTC (permalink / raw)
  To: Lars Schneider; +Cc: git, Junio C Hamano

On Wed, Aug 16, 2017 at 11:51 AM, Stefan Beller <sbeller@google.com> wrote:

> Any chance the "did not happen with 2.13" was not
> freshly cloned but tested on an existing repo? If so a hot
> candidate for suspicion is a93dcb0a56 (Merge branch
> 'bw/submodule-is-active', 2017-03-30), IMHO, just
> gut feeling, though.

which makes it a feature, I should add.

^ permalink raw reply	[relevance 24%]

* [PATCH] submodule.sh: remove unused variable
@ 2017-08-16 22:50 Stefan Beller
  2017-08-16 22:59 ` Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-16 22:50 UTC (permalink / raw)
  To: jrnieder; +Cc: git, Stefan Beller

This could have been part of 48308681b0 (git submodule update: have a
dedicated helper for cloning, 2016-02-29).

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 git-submodule.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index e131760eec..9dcec7b356 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -611,7 +611,6 @@ cmd_update()
 		die_if_unmatched "$mode" "$sha1"
 
 		name=$(git submodule--helper name "$sm_path") || exit
-		url=$(git config submodule."$name".url)
 		if ! test -z "$update"
 		then
 			update_module=$update
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 35%]

* Re: [PATCH] submodule.sh: remove unused variable
  2017-08-16 22:50 [PATCH] submodule.sh: remove unused variable Stefan Beller
@ 2017-08-16 22:59 ` Jonathan Nieder
  0 siblings, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-08-16 22:59 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Stefan Beller wrote:

> This could have been part of 48308681b0 (git submodule update: have a
> dedicated helper for cloning, 2016-02-29).
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  git-submodule.sh | 1 -
>  1 file changed, 1 deletion(-)

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Thanks for cleaning up.

> diff --git a/git-submodule.sh b/git-submodule.sh
> index e131760eec..9dcec7b356 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -611,7 +611,6 @@ cmd_update()
>  		die_if_unmatched "$mode" "$sha1"
>  
>  		name=$(git submodule--helper name "$sm_path") || exit
> -		url=$(git config submodule."$name".url)
>  		if ! test -z "$update"
>  		then
>  			update_module=$update
> -- 
> 2.14.0.rc0.3.g6c2e499285
> 

^ permalink raw reply	[relevance 15%]

* Re: [RFC PATCH 1/2] implement fetching of moved submodules
  2017-08-17 10:53 [RFC PATCH 1/2] implement fetching of moved submodules Heiko Voigt
  2017-08-17 11:00 ` [RFC PATCH 2/2] submodule: simplify decision tree whether to or not to fetch Heiko Voigt
@ 2017-08-17 17:20 ` Stefan Beller
  2017-08-18 16:06   ` Heiko Voigt
  1 sibling, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-17 17:20 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Thu, Aug 17, 2017 at 3:53 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> We store the changed submodules paths to calculate which submodule needs
> fetching. This does not work for moved submodules since their paths do
> not stay the same in case of a moved submodules. In case of new
> submodules we do not have a path in the current checkout, since they
> just appeared in this fetch.
>
> It is more general to collect the submodule names for changes instead of
> their paths to include the above cases.
>
> With the change described above we implement 'on-demand' fetching of
> changes in moved submodules.

This sounds as if this would also enable fetching new submodules
eventually?

> Note: This does only work when repositories have a .gitmodules file. In
> other words: It breaks if we do not get a name for a repository.
> IIRC, consensus was that this is a requirement to get nice submodule
> handling these days?

I think that should have been the consensus since ~1.7.8 (since the
submodules git dir can live inside the superprojects
<gitdir>/module/<name>).

A gitlink entry without corresponding .gitmodules entry is just a gitlink.
If we happen to have a repository at that path of the gitlink, we can
be nice and pretend like it is a functional submodule, but it really is
not. It's just another repo inside the superproject that happen to live
at the path of a gitlink.

> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
>
> I updated the leftover code from my series implementing recursive fetch
> for moved submodules[1] to the current master.
>
> This breaks t5531 and t5545 because they do not use a .gitmodules file.
>
> I also have some code leftover that does fallback on paths in case no
> submodule names can be found. But I do not really like it. The question
> here is how far do we support not using .gitmodules. Is it e.g.
> reasonable to say: "For --recurse-submodules=on-demand you need a
> .gitmodules file?"

I would not intentionally break users here, but any new functionality can
safely assume (a) we have a proper .gitmodules entry or (b) it is not a
submodule, so do nothing/be extra careful.

For example in recursive diff sort of makes sense to also handle
non-submodule gitlinks, but fetch is harder to tell.

(just last night I was rereading
https://public-inbox.org/git/CAJo=hJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg@mail.gmail.com/
which I think is a super cute application of gitlinks. If you happen
to checkout such
a tree, you don't want to fetch all of the fake submodules)

>
> [1] https://public-inbox.org/git/f5baa2acc09531a16f4f693eebbe60706bb8ed1e.1361751905.git.hvoigt@hvoigt.net/

Oha, that is from way back in the time. :)

>  submodule.c                 | 92 +++++++++++++++++++++++++--------------------
>  t/t5526-fetch-submodules.sh | 35 +++++++++++++++++
>  2 files changed, 86 insertions(+), 41 deletions(-)
>
> diff --git a/submodule.c b/submodule.c
> index 27de65a..3ed78ac 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -23,7 +23,7 @@
>  static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;
>  static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;
>  static int parallel_jobs = 1;
> -static struct string_list changed_submodule_paths = STRING_LIST_INIT_DUP;
> +static struct string_list changed_submodule_names = STRING_LIST_INIT_DUP;
>  static int initialized_fetch_ref_tips;
>  static struct oid_array ref_tips_before_fetch;
>  static struct oid_array ref_tips_after_fetch;
> @@ -742,11 +742,11 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)
>  }
>
>  static struct oid_array *submodule_commits(struct string_list *submodules,
> -                                          const char *path)
> +                                          const char *name)
>  {
>         struct string_list_item *item;
>
> -       item = string_list_insert(submodules, path);
> +       item = string_list_insert(submodules, name);
>         if (item->util)
>                 return (struct oid_array *) item->util;
>
> @@ -755,39 +755,34 @@ static struct oid_array *submodule_commits(struct string_list *submodules,
>         return (struct oid_array *) item->util;
>  }
>
> +struct collect_changed_submodules_cb_data {
> +       struct string_list *changed;

Here a comment would be helpful or a more concise variable name.
(What is changed?)

> +       const struct object_id *commit_oid;
> +};
> +
>  static void collect_changed_submodules_cb(struct diff_queue_struct *q,
>                                           struct diff_options *options,
>                                           void *data)
>  {
> +       struct collect_changed_submodules_cb_data *me = data;
> +       struct string_list *changed = me->changed;
> +       const struct object_id *commit_oid = me->commit_oid;
>         int i;
> -       struct string_list *changed = data;
>
>         for (i = 0; i < q->nr; i++) {
>                 struct diff_filepair *p = q->queue[i];
>                 struct oid_array *commits;
> +               const struct submodule *submodule;

(optional style nit, personal opinion, feel free to ignore)
I personally prefer to not name variables exactly as their type.
Also most (all) of the struct submodule uses used 'sub' as
the variable name, maybe keep it consistent?

> +
>                 if (!S_ISGITLINK(p->two->mode))
>                         continue;
>
> -               if (S_ISGITLINK(p->one->mode)) {
> -                       /*
> -                        * NEEDSWORK: We should honor the name configured in
> -                        * the .gitmodules file of the commit we are examining
> -                        * here to be able to correctly follow submodules
> -                        * being moved around.
> -                        */
> -                       commits = submodule_commits(changed, p->two->path);
> -                       oid_array_append(commits, &p->two->oid);
> -               } else {
> -                       /* Submodule is new or was moved here */
> -                       /*
> -                        * NEEDSWORK: When the .git directories of submodules
> -                        * live inside the superprojects .git directory some
> -                        * day we should fetch new submodules directly into
> -                        * that location too when config or options request
> -                        * that so they can be checked out from there.
> -                        */
> +               submodule = submodule_from_path(commit_oid, p->two->path);
> +               if (!submodule)
>                         continue;
> -               }
> +
> +               commits = submodule_commits(changed, submodule->name);
> +               oid_array_append(commits, &p->two->oid);
>         }
>  }
>
> @@ -810,11 +805,14 @@ static void collect_changed_submodules(struct string_list *changed,
>
>         while ((commit = get_revision(&rev))) {
>                 struct rev_info diff_rev;
> +               struct collect_changed_submodules_cb_data data;
> +               data.changed = changed;
> +               data.commit_oid = &commit->object.oid;
>
>                 init_revisions(&diff_rev, NULL);
>                 diff_rev.diffopt.output_format |= DIFF_FORMAT_CALLBACK;
>                 diff_rev.diffopt.format_callback = collect_changed_submodules_cb;
> -               diff_rev.diffopt.format_callback_data = changed;
> +               diff_rev.diffopt.format_callback_data = &data;
>                 diff_tree_combined_merge(commit, 1, &diff_rev);
>         }
>
> @@ -871,6 +869,7 @@ static int submodule_has_commits(const char *path, struct oid_array *commits)
>         oid_array_for_each_unique(commits, check_has_commit, &has_commit);
>
>         if (has_commit) {
> +

stray new line

>                 /*
>                  * Even if the submodule is checked out and the commit is
>                  * present, make sure it exists in the submodule's object store
> @@ -945,7 +944,7 @@ int find_unpushed_submodules(struct oid_array *commits,
>                 const char *remotes_name, struct string_list *needs_pushing)
>  {
>         struct string_list submodules = STRING_LIST_INIT_DUP;
> -       struct string_list_item *submodule;
> +       struct string_list_item *name;
>         struct argv_array argv = ARGV_ARRAY_INIT;
>
>         /* argv.argv[0] will be ignored by setup_revisions */
> @@ -956,12 +955,16 @@ int find_unpushed_submodules(struct oid_array *commits,
>
>         collect_changed_submodules(&submodules, &argv);
>
> -       for_each_string_list_item(submodule, &submodules) {
> -               struct oid_array *commits = submodule->util;
> -               const char *path = submodule->string;
> +       for_each_string_list_item(name, &submodules) {
> +               struct oid_array *commits = name->util;
> +               const struct submodule *submodule;
> +
> +               submodule = submodule_from_name(&null_oid, name->string);
> +               if (!submodule)
> +                       continue;
>
> -               if (submodule_needs_pushing(path, commits))
> -                       string_list_insert(needs_pushing, path);
> +               if (submodule_needs_pushing(submodule->path, commits))
> +                       string_list_insert(needs_pushing, submodule->path);

eventually we can also migrate to name here as well.
In a later patch.

>         }
>
>         free_submodules_oids(&submodules);
> @@ -1104,7 +1107,7 @@ static void calculate_changed_submodule_paths(void)
>  {
>         struct argv_array argv = ARGV_ARRAY_INIT;
>         struct string_list changed_submodules = STRING_LIST_INIT_DUP;
> -       const struct string_list_item *item;
> +       const struct string_list_item *name;
>
>         /* No need to check if there are no submodules configured */
>         if (!submodule_from_path(NULL, NULL))
> @@ -1119,16 +1122,20 @@ static void calculate_changed_submodule_paths(void)
>
>         /*
>          * Collect all submodules (whether checked out or not) for which new
> -        * commits have been recorded upstream in "changed_submodule_paths".
> +        * commits have been recorded upstream in "changed_submodule_names".
>          */
>         collect_changed_submodules(&changed_submodules, &argv);
>
> -       for_each_string_list_item(item, &changed_submodules) {
> -               struct oid_array *commits = item->util;
> -               const char *path = item->string;
> +       for_each_string_list_item(name, &changed_submodules) {
> +               struct oid_array *commits = name->util;
> +               const struct submodule *submodule;
> +
> +               submodule = submodule_from_name(&null_oid, name->string);
> +               if (!submodule)
> +                       continue;
>
> -               if (!submodule_has_commits(path, commits))
> -                       string_list_append(&changed_submodule_paths, path);
> +               if (!submodule_has_commits(submodule->path, commits))
> +                       string_list_append(&changed_submodule_names, name->string);
>         }
>
>         free_submodules_oids(&changed_submodules);
> @@ -1206,7 +1213,8 @@ static int get_next_submodule(struct child_process *cp,
>                                         continue;
>                                 if (submodule->fetch_recurse ==
>                                                 RECURSE_SUBMODULES_ON_DEMAND) {
> -                                       if (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))
> +                                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> +                                                                        submodule->name))
>                                                 continue;
>                                         default_argv = "on-demand";
>                                 }
> @@ -1215,13 +1223,15 @@ static int get_next_submodule(struct child_process *cp,
>                                     gitmodules_is_unmerged)
>                                         continue;
>                                 if (config_fetch_recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND) {
> -                                       if (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))
> +                                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> +                                                                        submodule->name))
>                                                 continue;
>                                         default_argv = "on-demand";
>                                 }
>                         }
>                 } else if (spf->command_line_option == RECURSE_SUBMODULES_ON_DEMAND) {
> -                       if (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))
> +                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> +                                                        submodule->name))
>                                 continue;
>                         default_argv = "on-demand";
>                 }
> @@ -1315,7 +1325,7 @@ int fetch_populated_submodules(const struct argv_array *options,
>
>         argv_array_clear(&spf.args);
>  out:
> -       string_list_clear(&changed_submodule_paths, 1);
> +       string_list_clear(&changed_submodule_names, 1);
>         return spf.result;
>  }
>
> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
> index 162baf1..ce788e9 100755
> --- a/t/t5526-fetch-submodules.sh
> +++ b/t/t5526-fetch-submodules.sh
> @@ -530,4 +530,39 @@ test_expect_success 'fetching submodule into a broken repository' '
>         test_must_fail git -C dst fetch --recurse-submodules
>  '
>
> +test_expect_success "fetch new commits when submodule got renamed" '
> +       git clone . downstream_rename &&
> +       (
> +               cd downstream_rename &&
> +               git submodule update --init &&
> +# NEEDSWORK: we omitted --recursive for the submodule update here since
> +# that does not work. See test 7001 for mv "moving nested submodules"
> +# for details. Once that is fixed we should add the --recursive option
> +# here.
> +               git checkout -b rename &&
> +               git mv submodule submodule_renamed &&
> +               (
> +                       cd submodule_renamed &&
> +                       git checkout -b rename_sub &&
> +                       echo a >a &&
> +                       git add a &&
> +                       git commit -ma &&
> +                       git push origin rename_sub &&
> +                       git rev-parse HEAD >../../expect
> +               ) &&
> +               git add submodule_renamed &&
> +               git commit -m "update renamed submodule" &&
> +               git push origin rename
> +       ) &&
> +       (
> +               cd downstream &&
> +               git fetch --recurse-submodules=on-demand &&
> +               (
> +                       cd submodule &&
> +                       git rev-parse origin/rename_sub >../../actual
> +               )
> +       ) &&
> +       test_cmp expect actual
> +'
> +
>  test_done
> --
> 2.0.0.274.g6b2cd91
>

^ permalink raw reply	[relevance 24%]

* Re: [RFC PATCH 2/2] submodule: simplify decision tree whether to or not to fetch
  2017-08-17 11:00 ` [RFC PATCH 2/2] submodule: simplify decision tree whether to or not to fetch Heiko Voigt
@ 2017-08-17 17:24   ` Stefan Beller
  2017-08-17 17:50     ` Brandon Williams
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-17 17:24 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Thu, Aug 17, 2017 at 4:00 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> To make extending this logic later easier.
>
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
> I am quite sure I replicated the same logic but a few more eyes would be
> appreciated.

A code cleanup is appreciated!

I thought Brandon had a series in flight doing a very similar cleanup here,
but in master..pu there is nothing to be found.

> Cheers Heiko

The code looks good to me.

Cheers!
Stefan

>
>  submodule.c | 55 +++++++++++++++++++++++++++----------------------------
>  1 file changed, 27 insertions(+), 28 deletions(-)
>
> diff --git a/submodule.c b/submodule.c
> index 3ed78ac..a1011f4 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -1171,6 +1171,21 @@ int submodule_touches_in_range(struct object_id *excl_oid,
>         return ret;
>  }
>
> +static int get_fetch_recurse_config(const struct submodule *submodule, int command_line_option)
> +{
> +       if (command_line_option != RECURSE_SUBMODULES_DEFAULT)
> +               return command_line_option;
> +
> +       if (submodule && submodule->fetch_recurse != RECURSE_SUBMODULES_NONE)
> +               /* local config overrules everything except commandline */
> +               return submodule->fetch_recurse;
> +
> +       if (gitmodules_is_unmerged)
> +               return RECURSE_SUBMODULES_OFF;
> +
> +       return config_fetch_recurse_submodules;
> +}
> +
>  struct submodule_parallel_fetch {
>         int count;
>         struct argv_array args;
> @@ -1203,37 +1218,21 @@ static int get_next_submodule(struct child_process *cp,
>                 if (!submodule)
>                         submodule = submodule_from_name(&null_oid, ce->name);
>
> -               default_argv = "yes";
> -               if (spf->command_line_option == RECURSE_SUBMODULES_DEFAULT) {
> -                       if (submodule &&
> -                           submodule->fetch_recurse !=
> -                                               RECURSE_SUBMODULES_NONE) {
> -                               if (submodule->fetch_recurse ==
> -                                               RECURSE_SUBMODULES_OFF)
> -                                       continue;
> -                               if (submodule->fetch_recurse ==
> -                                               RECURSE_SUBMODULES_ON_DEMAND) {
> -                                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> -                                                                        submodule->name))
> -                                               continue;
> -                                       default_argv = "on-demand";
> -                               }
> -                       } else {
> -                               if ((config_fetch_recurse_submodules == RECURSE_SUBMODULES_OFF) ||
> -                                   gitmodules_is_unmerged)
> -                                       continue;
> -                               if (config_fetch_recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND) {
> -                                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> -                                                                        submodule->name))
> -                                               continue;
> -                                       default_argv = "on-demand";
> -                               }
> -                       }
> -               } else if (spf->command_line_option == RECURSE_SUBMODULES_ON_DEMAND) {
> -                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> +               switch (get_fetch_recurse_config(submodule, spf->command_line_option))
> +               {
> +               default:
> +               case RECURSE_SUBMODULES_DEFAULT:
> +               case RECURSE_SUBMODULES_ON_DEMAND:
> +                       if (!submodule || !unsorted_string_list_lookup(&changed_submodule_names,
>                                                          submodule->name))
>                                 continue;
>                         default_argv = "on-demand";
> +                       break;
> +               case RECURSE_SUBMODULES_ON:
> +                       default_argv = "yes";
> +                       break;
> +               case RECURSE_SUBMODULES_OFF:
> +                       continue;
>                 }
>
>                 strbuf_addf(&submodule_path, "%s/%s", spf->work_tree, ce->name);
> --
> 2.0.0.274.g6b2cd91
>

^ permalink raw reply	[relevance 16%]

* Re: [PATCH] t5526: fix some broken && chains
  2017-08-17 10:36 [PATCH] t5526: fix some broken && chains Heiko Voigt
@ 2017-08-17 17:44 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-17 17:44 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Junio C Hamano, git

On Thu, Aug 17, 2017 at 3:36 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>

Reviewed-by: Stefan Beller <sbeller@google.com>

Thanks,
Stefan

> ---
>  t/t5526-fetch-submodules.sh | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
> index ce788e9..22a7358 100755
> --- a/t/t5526-fetch-submodules.sh
> +++ b/t/t5526-fetch-submodules.sh
> @@ -193,7 +193,7 @@ test_expect_success "recurseSubmodules=true propagates into submodules" '
>         add_upstream_commit &&
>         (
>                 cd downstream &&
> -               git config fetch.recurseSubmodules true
> +               git config fetch.recurseSubmodules true &&
>                 git fetch >../actual.out 2>../actual.err
>         ) &&
>         test_must_be_empty actual.out &&
> @@ -218,7 +218,7 @@ test_expect_success "--no-recurse-submodules overrides config setting" '
>         add_upstream_commit &&
>         (
>                 cd downstream &&
> -               git config fetch.recurseSubmodules true
> +               git config fetch.recurseSubmodules true &&
>                 git fetch --no-recurse-submodules >../actual.out 2>../actual.err
>         ) &&
>         ! test -s actual.out &&
> @@ -232,7 +232,7 @@ test_expect_success "Recursion doesn't happen when no new commits are fetched in
>                         cd submodule &&
>                         git config --unset fetch.recurseSubmodules
>                 ) &&
> -               git config --unset fetch.recurseSubmodules
> +               git config --unset fetch.recurseSubmodules &&
>                 git fetch >../actual.out 2>../actual.err
>         ) &&
>         ! test -s actual.out &&
> @@ -312,7 +312,7 @@ test_expect_success "Recursion picks up all submodules when necessary" '
>                 ) &&
>                 head1=$(git rev-parse --short HEAD^) &&
>                 git add subdir/deepsubmodule &&
> -               git commit -m "new deepsubmodule"
> +               git commit -m "new deepsubmodule" &&
>                 head2=$(git rev-parse --short HEAD) &&
>                 echo "Fetching submodule submodule" > ../expect.err.sub &&
>                 echo "From $pwd/submodule" >> ../expect.err.sub &&
> --
> 2.0.0.274.g6b2cd91
>

^ permalink raw reply	[relevance 17%]

* Re: [RFC PATCH 2/2] submodule: simplify decision tree whether to or not to fetch
  2017-08-17 17:24   ` Stefan Beller
@ 2017-08-17 17:50     ` Brandon Williams
  2017-08-18 16:06       ` Heiko Voigt
  0 siblings, 1 reply; 200+ results
From: Brandon Williams @ 2017-08-17 17:50 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Heiko Voigt, git, Jonathan Nieder, Jens Lehmann

On 08/17, Stefan Beller wrote:
> On Thu, Aug 17, 2017 at 4:00 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > To make extending this logic later easier.
> >
> > Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> > ---
> > I am quite sure I replicated the same logic but a few more eyes would be
> > appreciated.
> 
> A code cleanup is appreciated!
> 
> I thought Brandon had a series in flight doing a very similar cleanup here,
> but in master..pu there is nothing to be found.

Yeah there are 2 series in flight which will probably conflict here.
bw/grep-recurse-submodules and bw/submodule-config-cleanup

> 
> > Cheers Heiko
> 
> The code looks good to me.
> 
> Cheers!
> Stefan
> 
> >
> >  submodule.c | 55 +++++++++++++++++++++++++++----------------------------
> >  1 file changed, 27 insertions(+), 28 deletions(-)
> >
> > diff --git a/submodule.c b/submodule.c
> > index 3ed78ac..a1011f4 100644
> > --- a/submodule.c
> > +++ b/submodule.c
> > @@ -1171,6 +1171,21 @@ int submodule_touches_in_range(struct object_id *excl_oid,
> >         return ret;
> >  }
> >
> > +static int get_fetch_recurse_config(const struct submodule *submodule, int command_line_option)
> > +{
> > +       if (command_line_option != RECURSE_SUBMODULES_DEFAULT)
> > +               return command_line_option;
> > +
> > +       if (submodule && submodule->fetch_recurse != RECURSE_SUBMODULES_NONE)
> > +               /* local config overrules everything except commandline */
> > +               return submodule->fetch_recurse;
> > +
> > +       if (gitmodules_is_unmerged)
> > +               return RECURSE_SUBMODULES_OFF;
> > +
> > +       return config_fetch_recurse_submodules;
> > +}
> > +
> >  struct submodule_parallel_fetch {
> >         int count;
> >         struct argv_array args;
> > @@ -1203,37 +1218,21 @@ static int get_next_submodule(struct child_process *cp,
> >                 if (!submodule)
> >                         submodule = submodule_from_name(&null_oid, ce->name);
> >
> > -               default_argv = "yes";
> > -               if (spf->command_line_option == RECURSE_SUBMODULES_DEFAULT) {
> > -                       if (submodule &&
> > -                           submodule->fetch_recurse !=
> > -                                               RECURSE_SUBMODULES_NONE) {
> > -                               if (submodule->fetch_recurse ==
> > -                                               RECURSE_SUBMODULES_OFF)
> > -                                       continue;
> > -                               if (submodule->fetch_recurse ==
> > -                                               RECURSE_SUBMODULES_ON_DEMAND) {
> > -                                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> > -                                                                        submodule->name))
> > -                                               continue;
> > -                                       default_argv = "on-demand";
> > -                               }
> > -                       } else {
> > -                               if ((config_fetch_recurse_submodules == RECURSE_SUBMODULES_OFF) ||
> > -                                   gitmodules_is_unmerged)
> > -                                       continue;
> > -                               if (config_fetch_recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND) {
> > -                                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> > -                                                                        submodule->name))
> > -                                               continue;
> > -                                       default_argv = "on-demand";
> > -                               }
> > -                       }
> > -               } else if (spf->command_line_option == RECURSE_SUBMODULES_ON_DEMAND) {
> > -                       if (!unsorted_string_list_lookup(&changed_submodule_names,
> > +               switch (get_fetch_recurse_config(submodule, spf->command_line_option))
> > +               {
> > +               default:
> > +               case RECURSE_SUBMODULES_DEFAULT:
> > +               case RECURSE_SUBMODULES_ON_DEMAND:
> > +                       if (!submodule || !unsorted_string_list_lookup(&changed_submodule_names,
> >                                                          submodule->name))
> >                                 continue;
> >                         default_argv = "on-demand";
> > +                       break;
> > +               case RECURSE_SUBMODULES_ON:
> > +                       default_argv = "yes";
> > +                       break;
> > +               case RECURSE_SUBMODULES_OFF:
> > +                       continue;
> >                 }
> >
> >                 strbuf_addf(&submodule_path, "%s/%s", spf->work_tree, ce->name);
> > --
> > 2.0.0.274.g6b2cd91
> >

-- 
Brandon Williams

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] add test for bug in git-mv with nested submodules
  2017-08-17 10:34 [PATCH] add test for bug in git-mv with nested submodules Heiko Voigt
@ 2017-08-17 19:05 ` Stefan Beller
  2017-08-18 16:06   ` Heiko Voigt
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-17 19:05 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Junio C Hamano, git

On Thu, Aug 17, 2017 at 3:34 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> When using git-mv with a submodule it will detect that and update the
> paths for its configurations (.gitmodules, worktree and gitfile). This
> does not work for nested submodules where a user renames the root
> submodule.
>
> We discovered this fact when working on on-demand fetch for renamed
> submodules. Lets add a test to document.
>
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
>  t/t7001-mv.sh | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> index e365d1f..39f8aed 100755
> --- a/t/t7001-mv.sh
> +++ b/t/t7001-mv.sh
> @@ -491,4 +491,13 @@ test_expect_success 'moving a submodule in nested directories' '
>         test_cmp actual expect
>  '
>
> +test_expect_failure 'moving nested submodules' '
> +       git commit -am "cleanup commit" &&
> +       git submodule add ./. sub_nested &&

If possible, I would avoid adding the repo itself
as a submodule as it is unrealistic in the wild.

While it may be ok for the test here, later down the road
other tests making use of it it may become an issue with
the URL of the submodule.

> +       git commit -m "add sub_nested" &&
> +       git submodule update --init --recursive &&
> +       git mv sub_nested sub_nested_moved &&
> +       git status
> +'
> +
>  test_done
> --
> 2.0.0.274.g6b2cd91
>

^ permalink raw reply	[relevance 24%]

* Re: Submodule regression in 2.14?
  2017-08-16 18:51 ` Stefan Beller
  2017-08-16 18:53   ` Stefan Beller
@ 2017-08-17 21:21   ` Lars Schneider
  2017-08-17 21:55     ` Stefan Beller
  1 sibling, 1 reply; 200+ results
From: Lars Schneider @ 2017-08-17 21:21 UTC (permalink / raw)
  To: Stefan Beller, bmwill; +Cc: git, Junio C Hamano


> On 16 Aug 2017, at 20:51, Stefan Beller <sbeller@google.com> wrote:
> 
> On Wed, Aug 16, 2017 at 11:35 AM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> Hi,
>> 
>> I think we discovered a regression in Git 2.14.1 today.
>> It looks like as if "git submodule update --init --recursive" removes
>> the "skip submodules" config.
>> 
>> Consider the following steps:
>> 
>>    git clone https://server/repo.git
>>    cd repo
>>    git config --local submodule.some/other/repo.update none
>>    git submodule update --init --recursive
>>    git pull --recurse-submodules
>> 
>> With Git 2.14 the last "git pull" will clone the "some/other/repo"
>> submodule. This did not happen with Git 2.13.
>> 
>> Bug or feature? I don't have anymore time for Git today. I am happy to
>> provide a proper test case tomorrow, though.
> 
> $ git log --oneline v2.13.0..v2.14.1 -- git-submodule.sh
> 532139940c add: warn when adding an embedded repository
> (I am confident this is not the suspect, let's keep searching.
> Not a lot happened in submodule land apparently)
> 
> Looking through all commits v2.13..v2.14 doesn't have me
> suspect any of them.
> 
> Any chance the "did not happen with 2.13" was not
> freshly cloned but tested on an existing repo? If so a hot
> candidate for suspicion is a93dcb0a56 (Merge branch
> 'bw/submodule-is-active', 2017-03-30), IMHO, just
> gut feeling, though.
> 
> Oh, wait.
> $ git log --oneline v2.13.0..v2.14.1 -- builtin/pull.c
> c9c63ee558 Merge branch 'sb/pull-rebase-submodule'
> a6d7eb2c7a pull: optionally rebase submodules (remote submodule changes only)
> could also be a culprit. Do you have pull.rebase set?

I bisected the problem today and "a6d7eb2c7a pull: optionally rebase submodules 
(remote submodule changes only)" is indeed the culprit.

The commit seems to break the following test case.

diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index dcac364c5f..24f9729015 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1289,4 +1289,19 @@ test_expect_success 'init properly sets the config' '
 	test_must_fail git -C multisuper_clone config --get submodule.sub1.active
 '
 
+test_expect_success 'submodule update and git pull with disabled submodule' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	(
+		cd multisuper_clone &&
+		git config --local submodule.sub0.update none &&
+		git submodule update --init --recursive &&
+		git pull --recurse-submodules &&
+		git submodule status | cut -c 1,43- >actual
+	) &&
+	ls &&
+	test_cmp expect multisuper_clone/actual
+'
+
 test_done


I am not familiar with the code. Does anyone see the problem
right away?

Thanks,
Lars



^ permalink raw reply	[relevance 17%]

* Re: Submodule regression in 2.14?
  2017-08-17 21:21   ` Lars Schneider
@ 2017-08-17 21:55     ` Stefan Beller
  2017-08-18  2:13       ` Junio C Hamano
  2017-08-18 13:12       ` Lars Schneider
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2017-08-17 21:55 UTC (permalink / raw)
  To: Lars Schneider; +Cc: Brandon Williams, git, Junio C Hamano

On Thu, Aug 17, 2017 at 2:21 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
>> Oh, wait.
>> $ git log --oneline v2.13.0..v2.14.1 -- builtin/pull.c
>> c9c63ee558 Merge branch 'sb/pull-rebase-submodule'
>> a6d7eb2c7a pull: optionally rebase submodules (remote submodule changes only)
>> could also be a culprit. Do you have pull.rebase set?
>
> I bisected the problem today and "a6d7eb2c7a pull: optionally rebase submodules
> (remote submodule changes only)" is indeed the culprit.
>
> The commit seems to break the following test case.
>
> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
> index dcac364c5f..24f9729015 100755
> --- a/t/t7400-submodule-basic.sh
> +++ b/t/t7400-submodule-basic.sh
> @@ -1289,4 +1289,19 @@ test_expect_success 'init properly sets the config' '
>         test_must_fail git -C multisuper_clone config --get submodule.sub1.active
>  '
>
> +test_expect_success 'submodule update and git pull with disabled submodule' '
> +       test_when_finished "rm -rf multisuper_clone" &&
> +       pwd=$(pwd) &&
> +       git clone file://"$pwd"/multisuper multisuper_clone &&
> +       (
> +               cd multisuper_clone &&
> +               git config --local submodule.sub0.update none &&
> +               git submodule update --init --recursive &&
> +               git pull --recurse-submodules &&
> +               git submodule status | cut -c 1,43- >actual
> +       ) &&
> +       ls &&
> +       test_cmp expect multisuper_clone/actual
> +'

Thanks for providing this test.

cd trash directory.t7400-submodule-basic/multisuper_clone
cat .git/config
[submodule "sub0"]
  update = none
  active = true
  url = file:///.../t/trash directory.t7400-submodule-basic/sub1


submodule.<name>.update
    The default update procedure for a submodule.
    This variable is populated by git submodule init
    from the gitmodules(5) file. See description of
    update command in git-submodule(1).

The first sentence of .update is misleading IMHO as the
these settings should strictly apply to the "submodule update"
command. So "pull --recurse-submodules" ought to ignore it,
instead the pull can do whatever it wants, namely treat the
submodule roughly like a tree and either merge/rebase
inside the submodule as well. The user *asked* for recursive
pull after all.

Are you saying this might be a design mistake and
the .update ought to be respected by all the other
commands? For example
    git reset --recurse-submodules
should ignore the .update= none?

When designing these new recursive submodule functionality
outside the "submodule" command, I'd want submodules
to behave as much as possible like trees.

ideas?

Thanks,
Stefan

> +
>  test_done
>
>
> I am not familiar with the code. Does anyone see the problem
> right away?
>
> Thanks,
> Lars
>
>

^ permalink raw reply	[relevance 26%]

* Re: Submodule regression in 2.14?
  2017-08-17 21:55     ` Stefan Beller
@ 2017-08-18  2:13       ` Junio C Hamano
  2017-08-18  4:02         ` Stefan Beller
  2017-08-18 13:12       ` Lars Schneider
  1 sibling, 1 reply; 200+ results
From: Junio C Hamano @ 2017-08-18  2:13 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Lars Schneider, Brandon Williams, git\

Stefan Beller <sbeller@google.com> writes:

> Are you saying this might be a design mistake and
> the .update ought to be respected by all the other
> commands? For example
>     git reset --recurse-submodules
> should ignore the .update= none?

I have been under the impression that that has been the traditional
desire of what .update ought to mean.  I personally do not have a
strong opinion---at least not yet.

> When designing these new recursive submodule functionality
> outside the "submodule" command, I'd want submodules
> to behave as much as possible like trees.

I think that is sensible as long as the user does not explicitly say
"this and that submodule behave differently" by giving configuration
variables.  Perhaps .update is one of those that should countermand
the default behaviour of "--recurse-submodules"?

^ permalink raw reply	[relevance 24%]

* Re: Submodule regression in 2.14?
  2017-08-18  2:13       ` Junio C Hamano
@ 2017-08-18  4:02         ` Stefan Beller
  2017-08-18 16:50           ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-18  4:02 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Lars Schneider, Brandon Williams, git

On Thu, Aug 17, 2017 at 7:13 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> Are you saying this might be a design mistake and
>> the .update ought to be respected by all the other
>> commands? For example
>>     git reset --recurse-submodules
>> should ignore the .update= none?
>
> I have been under the impression that that has been the traditional
> desire of what .update ought to mean.  I personally do not have a
> strong opinion---at least not yet.

In this context note v2.14.0-rc1-34-g7463e2ec3
(bw/submodule-config-cleanup~7, "unpack-trees:
don't respect submodule.update") that is going opposite of
your impression.

>> When designing these new recursive submodule functionality
>> outside the "submodule" command, I'd want submodules
>> to behave as much as possible like trees.
>
> I think that is sensible as long as the user does not explicitly say
> "this and that submodule behave differently" by giving configuration
> variables.  Perhaps .update is one of those that should countermand
> the default behaviour of "--recurse-submodules"?

Maybe, I'll think about it. However there is no such
equivalent for trees (and AFAICT never came up) to
treat a specific directory other than the rest in worktree
operations.

The problem with the issue in question is however:
git-pull is a combination of two other high level commands
(fetch/merge), the fetch component already had
a recursive behavior, and that commit in question
added a bit for the merge component, so the UX
is hard to get right for both of them:

    git pull --recurse=fetch-only
    git pull --recurse=merge-respects-update-strategy

is what I'd want to avoid.

So maybe we can just respect the update strategy
before starting the local part.

Stefan

^ permalink raw reply	[relevance 24%]

* Re: Submodule regression in 2.14?
  2017-08-17 21:55     ` Stefan Beller
  2017-08-18  2:13       ` Junio C Hamano
@ 2017-08-18 13:12       ` Lars Schneider
  2017-08-18 17:16         ` Stefan Beller
  1 sibling, 1 reply; 200+ results
From: Lars Schneider @ 2017-08-18 13:12 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Brandon Williams, git, Junio C Hamano


> On 17 Aug 2017, at 23:55, Stefan Beller <sbeller@google.com> wrote:
> 
> On Thu, Aug 17, 2017 at 2:21 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> 
>>> Oh, wait.
>>> $ git log --oneline v2.13.0..v2.14.1 -- builtin/pull.c
>>> c9c63ee558 Merge branch 'sb/pull-rebase-submodule'
>>> a6d7eb2c7a pull: optionally rebase submodules (remote submodule changes only)
>>> could also be a culprit. Do you have pull.rebase set?
>> 
>> I bisected the problem today and "a6d7eb2c7a pull: optionally rebase submodules
>> (remote submodule changes only)" is indeed the culprit.
>> 
>> The commit seems to break the following test case.
>> 
>> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
>> index dcac364c5f..24f9729015 100755
>> --- a/t/t7400-submodule-basic.sh
>> +++ b/t/t7400-submodule-basic.sh
>> @@ -1289,4 +1289,19 @@ test_expect_success 'init properly sets the config' '
>>        test_must_fail git -C multisuper_clone config --get submodule.sub1.active
>> '
>> 
>> +test_expect_success 'submodule update and git pull with disabled submodule' '
>> +       test_when_finished "rm -rf multisuper_clone" &&
>> +       pwd=$(pwd) &&
>> +       git clone file://"$pwd"/multisuper multisuper_clone &&
>> +       (
>> +               cd multisuper_clone &&
>> +               git config --local submodule.sub0.update none &&
>> +               git submodule update --init --recursive &&
>> +               git pull --recurse-submodules &&
>> +               git submodule status | cut -c 1,43- >actual
>> +       ) &&
>> +       ls &&
>> +       test_cmp expect multisuper_clone/actual
>> +'
> 
> Thanks for providing this test.
> 
> cd trash directory.t7400-submodule-basic/multisuper_clone
> cat .git/config
> [submodule "sub0"]
>  update = none
>  active = true
>  url = file:///.../t/trash directory.t7400-submodule-basic/sub1
> 
> 
> submodule.<name>.update
>    The default update procedure for a submodule.
>    This variable is populated by git submodule init
>    from the gitmodules(5) file. See description of
>    update command in git-submodule(1).
> 
> The first sentence of .update is misleading IMHO as the
> these settings should strictly apply to the "submodule update"
> command. So "pull --recurse-submodules" ought to ignore it,
> instead the pull can do whatever it wants, namely treat the
> submodule roughly like a tree and either merge/rebase
> inside the submodule as well. The user *asked* for recursive
> pull after all.
> 
> Are you saying this might be a design mistake and
> the .update ought to be respected by all the other
> commands? For example
>    git reset --recurse-submodules
> should ignore the .update= none?
> 
> When designing these new recursive submodule functionality
> outside the "submodule" command, I'd want submodules
> to behave as much as possible like trees.

In the past "submodule.<name>.update=none" was an easy way
to selectively disable certain Submodules.

How would I do this with Git 2.14?

My gut feeling is that all commands should respect the
"submodule.<name>.update=none" setting.

- Lars

^ permalink raw reply	[relevance 16%]

* Re: [PATCH] add test for bug in git-mv with nested submodules
  2017-08-17 19:05 ` Stefan Beller
@ 2017-08-18 16:06   ` Heiko Voigt
  2017-08-18 19:04     ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Heiko Voigt @ 2017-08-18 16:06 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, git

On Thu, Aug 17, 2017 at 12:05:56PM -0700, Stefan Beller wrote:
> On Thu, Aug 17, 2017 at 3:34 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > When using git-mv with a submodule it will detect that and update the
> > paths for its configurations (.gitmodules, worktree and gitfile). This
> > does not work for nested submodules where a user renames the root
> > submodule.
> >
> > We discovered this fact when working on on-demand fetch for renamed
> > submodules. Lets add a test to document.
> >
> > Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> > ---
> >  t/t7001-mv.sh | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> > index e365d1f..39f8aed 100755
> > --- a/t/t7001-mv.sh
> > +++ b/t/t7001-mv.sh
> > @@ -491,4 +491,13 @@ test_expect_success 'moving a submodule in nested directories' '
> >         test_cmp actual expect
> >  '
> >
> > +test_expect_failure 'moving nested submodules' '
> > +       git commit -am "cleanup commit" &&
> > +       git submodule add ./. sub_nested &&
> 
> If possible, I would avoid adding the repo itself
> as a submodule as it is unrealistic in the wild.
> 
> While it may be ok for the test here, later down the road
> other tests making use of it it may become an issue with
> the URL of the submodule.

I just copied the shortcut that they were adding themselfes as submodule
in 'setup submodule'. The whole setup of submodules in this test is like
this. This way we already had a nested submodule structure which I could
just add.

I agree that this is unrealistic so I can change that in the test I am
adding. But from what I have seen, this shortcut is taken in quite some
places when dealing with submodules.

Cheers Heiko

^ permalink raw reply	[relevance 26%]

* Re: [RFC PATCH 1/2] implement fetching of moved submodules
  2017-08-17 17:20 ` [RFC PATCH 1/2] implement fetching of moved submodules Stefan Beller
@ 2017-08-18 16:06   ` Heiko Voigt
  0 siblings, 0 replies; 200+ results
From: Heiko Voigt @ 2017-08-18 16:06 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Thu, Aug 17, 2017 at 10:20:13AM -0700, Stefan Beller wrote:
> On Thu, Aug 17, 2017 at 3:53 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > We store the changed submodules paths to calculate which submodule needs
> > fetching. This does not work for moved submodules since their paths do
> > not stay the same in case of a moved submodules. In case of new
> > submodules we do not have a path in the current checkout, since they
> > just appeared in this fetch.
> >
> > It is more general to collect the submodule names for changes instead of
> > their paths to include the above cases.
> >
> > With the change described above we implement 'on-demand' fetching of
> > changes in moved submodules.
> 
> This sounds as if this would also enable fetching new submodules
> eventually?

Yes that was the goal when starting with these changes back then. But it
took more time than I had back then. So instead of letting these changes
sit bitrot again lets see if we can get them integrated.

For new submodules we need to change the iteration somehow. Currently we
are iterating through the index. But new submodules obviously do not
have an index entry (otherwise they would not be new). So instead of the
index we will need to create another list that contains "all"
submodules. Maybe something like: all submodules from the index plus all
submodules that changed / are new? We could also go further and inspect
all submodules from all ref tips to handle submodules on other branches
configured to 'yes'. But I think we should leave that for later if need
arises.

Some merge of index and additional submodules is needed, because for
--recurse-submodules=yes or submodule.<name>.fetchRecurseSubmodules=yes
we always need to run fetch inside the submodule. That would break if we
only looked at submodules that are collected as changed.

> > Note: This does only work when repositories have a .gitmodules file. In
> > other words: It breaks if we do not get a name for a repository.
> > IIRC, consensus was that this is a requirement to get nice submodule
> > handling these days?
> 
> I think that should have been the consensus since ~1.7.8 (since the
> submodules git dir can live inside the superprojects
> <gitdir>/module/<name>).

I agree but since we started without it, we kind of have a mixed state.

> A gitlink entry without corresponding .gitmodules entry is just a gitlink.
> If we happen to have a repository at that path of the gitlink, we can
> be nice and pretend like it is a functional submodule, but it really is
> not. It's just another repo inside the superproject that happen to live
> at the path of a gitlink.

Yeah but at the moment we are handling 'on-demand' fetches and stuff for
such just gitlink submodules. If we were firm on that requirement we
would just skip those but that is not the case with the current
implementation.

> > Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> > ---
> >
> > I updated the leftover code from my series implementing recursive fetch
> > for moved submodules[1] to the current master.
> >
> > This breaks t5531 and t5545 because they do not use a .gitmodules file.
> >
> > I also have some code leftover that does fallback on paths in case no
> > submodule names can be found. But I do not really like it. The question
> > here is how far do we support not using .gitmodules. Is it e.g.
> > reasonable to say: "For --recurse-submodules=on-demand you need a
> > .gitmodules file?"
> 
> I would not intentionally break users here, but any new functionality can
> safely assume (a) we have a proper .gitmodules entry or (b) it is not a
> submodule, so do nothing/be extra careful.
> 
> For example in recursive diff sort of makes sense to also handle
> non-submodule gitlinks, but fetch is harder to tell.

Well we have a few different cases for gitlinks without .gitmodule
entry:

 1. New gitlink: We can not handle since we do not know where to clone
                 from.

 2. Removed gitlink: No need to do anything in fetch

 3. Changed (but same name) gitlink: We can / and currently do run fetch
                                     in it

 4. Renamed: We currently skip those. We could probably do something to
             track the rename and run fetch in case of gitlink changes.
             In my current approach only the ones with a name are
             handled.

So I guess I will add a fallback to paths for 3. so we do not
unnecessarily break users using the current implementation.

> (just last night I was rereading
> https://public-inbox.org/git/CAJo=hJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg@mail.gmail.com/
> which I think is a super cute application of gitlinks. If you happen
> to checkout such
> a tree, you don't want to fetch all of the fake submodules)
> 
> >
> > [1] https://public-inbox.org/git/f5baa2acc09531a16f4f693eebbe60706bb8ed1e.1361751905.git.hvoigt@hvoigt.net/
> 
> Oha, that is from way back in the time. :)

Yeah this code did go through some proper bitrotting :)

> >  submodule.c                 | 92 +++++++++++++++++++++++++--------------------
> >  t/t5526-fetch-submodules.sh | 35 +++++++++++++++++
> >  2 files changed, 86 insertions(+), 41 deletions(-)
> >
> > diff --git a/submodule.c b/submodule.c
> > index 27de65a..3ed78ac 100644
> > --- a/submodule.c
> > +++ b/submodule.c
> > @@ -23,7 +23,7 @@
> >  static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;
> >  static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;
> >  static int parallel_jobs = 1;
> > -static struct string_list changed_submodule_paths = STRING_LIST_INIT_DUP;
> > +static struct string_list changed_submodule_names = STRING_LIST_INIT_DUP;
> >  static int initialized_fetch_ref_tips;
> >  static struct oid_array ref_tips_before_fetch;
> >  static struct oid_array ref_tips_after_fetch;
> > @@ -742,11 +742,11 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)
> >  }
> >
> >  static struct oid_array *submodule_commits(struct string_list *submodules,
> > -                                          const char *path)
> > +                                          const char *name)
> >  {
> >         struct string_list_item *item;
> >
> > -       item = string_list_insert(submodules, path);
> > +       item = string_list_insert(submodules, name);
> >         if (item->util)
> >                 return (struct oid_array *) item->util;
> >
> > @@ -755,39 +755,34 @@ static struct oid_array *submodule_commits(struct string_list *submodules,
> >         return (struct oid_array *) item->util;
> >  }
> >
> > +struct collect_changed_submodules_cb_data {
> > +       struct string_list *changed;
> 
> Here a comment would be helpful or a more concise variable name.
> (What is changed?)

I'll change that to 'changed_submodules' the caller who is passing this
in called this changed. It is the list of changed submodules to be
filled.

> 
> > +       const struct object_id *commit_oid;

What about this name? It is the commit_oid in the superproject of the
current revision under investigation. IMO is easy to get confused what
commits are referenced superproject or submodule. Maybe
'super_commit_oid' would be more clear?

> > +};
> > +
> >  static void collect_changed_submodules_cb(struct diff_queue_struct *q,
> >                                           struct diff_options *options,
> >                                           void *data)
> >  {
> > +       struct collect_changed_submodules_cb_data *me = data;
> > +       struct string_list *changed = me->changed;
> > +       const struct object_id *commit_oid = me->commit_oid;
> >         int i;
> > -       struct string_list *changed = data;
> >
> >         for (i = 0; i < q->nr; i++) {
> >                 struct diff_filepair *p = q->queue[i];
> >                 struct oid_array *commits;
> > +               const struct submodule *submodule;
> 
> (optional style nit, personal opinion, feel free to ignore)
> I personally prefer to not name variables exactly as their type.
> Also most (all) of the struct submodule uses used 'sub' as
> the variable name, maybe keep it consistent?

Well I understand that and its similar for me but I personally I do not
like abbreviations for variable names since I like to be able to read
code natually. So that took precendence over naming submodule
differently than its type here :) Counting submodule vs. sub in
submodule.c I see 4 vs. 5 occurrences...  I'll think about it.

> > +
> >                 if (!S_ISGITLINK(p->two->mode))
> >                         continue;
> >
> > -               if (S_ISGITLINK(p->one->mode)) {
> > -                       /*
> > -                        * NEEDSWORK: We should honor the name configured in
> > -                        * the .gitmodules file of the commit we are examining
> > -                        * here to be able to correctly follow submodules
> > -                        * being moved around.
> > -                        */
> > -                       commits = submodule_commits(changed, p->two->path);
> > -                       oid_array_append(commits, &p->two->oid);
> > -               } else {
> > -                       /* Submodule is new or was moved here */
> > -                       /*
> > -                        * NEEDSWORK: When the .git directories of submodules
> > -                        * live inside the superprojects .git directory some
> > -                        * day we should fetch new submodules directly into
> > -                        * that location too when config or options request
> > -                        * that so they can be checked out from there.
> > -                        */
> > +               submodule = submodule_from_path(commit_oid, p->two->path);
> > +               if (!submodule)
> >                         continue;
> > -               }
> > +
> > +               commits = submodule_commits(changed, submodule->name);
> > +               oid_array_append(commits, &p->two->oid);
> >         }
> >  }
> >
> > @@ -810,11 +805,14 @@ static void collect_changed_submodules(struct string_list *changed,
> >
> >         while ((commit = get_revision(&rev))) {
> >                 struct rev_info diff_rev;
> > +               struct collect_changed_submodules_cb_data data;
> > +               data.changed = changed;
> > +               data.commit_oid = &commit->object.oid;
> >
> >                 init_revisions(&diff_rev, NULL);
> >                 diff_rev.diffopt.output_format |= DIFF_FORMAT_CALLBACK;
> >                 diff_rev.diffopt.format_callback = collect_changed_submodules_cb;
> > -               diff_rev.diffopt.format_callback_data = changed;
> > +               diff_rev.diffopt.format_callback_data = &data;
> >                 diff_tree_combined_merge(commit, 1, &diff_rev);
> >         }
> >
> > @@ -871,6 +869,7 @@ static int submodule_has_commits(const char *path, struct oid_array *commits)
> >         oid_array_for_each_unique(commits, check_has_commit, &has_commit);
> >
> >         if (has_commit) {
> > +
> 
> stray new line

Yeah saw that too late. Will fix.

> >                 /*
> >                  * Even if the submodule is checked out and the commit is
> >                  * present, make sure it exists in the submodule's object store
> > @@ -945,7 +944,7 @@ int find_unpushed_submodules(struct oid_array *commits,
> >                 const char *remotes_name, struct string_list *needs_pushing)
> >  {
> >         struct string_list submodules = STRING_LIST_INIT_DUP;
> > -       struct string_list_item *submodule;
> > +       struct string_list_item *name;
> >         struct argv_array argv = ARGV_ARRAY_INIT;
> >
> >         /* argv.argv[0] will be ignored by setup_revisions */
> > @@ -956,12 +955,16 @@ int find_unpushed_submodules(struct oid_array *commits,
> >
> >         collect_changed_submodules(&submodules, &argv);
> >
> > -       for_each_string_list_item(submodule, &submodules) {
> > -               struct oid_array *commits = submodule->util;
> > -               const char *path = submodule->string;
> > +       for_each_string_list_item(name, &submodules) {
> > +               struct oid_array *commits = name->util;
> > +               const struct submodule *submodule;
> > +
> > +               submodule = submodule_from_name(&null_oid, name->string);
> > +               if (!submodule)
> > +                       continue;
> >
> > -               if (submodule_needs_pushing(path, commits))
> > -                       string_list_insert(needs_pushing, path);
> > +               if (submodule_needs_pushing(submodule->path, commits))
> > +                       string_list_insert(needs_pushing, submodule->path);
> 
> eventually we can also migrate to name here as well.
> In a later patch.

Yeah, then it would be possible to also push submodules without a
populated worktree.

There was this other thread about the is-populated-check in
push_submodules() where it was consensus that it does not make much
sense but for a maintainer integrating others work it might be useful to
not always have all submodules populated.

Cheers Heiko

^ permalink raw reply	[relevance 22%]

* Re: [RFC PATCH 2/2] submodule: simplify decision tree whether to or not to fetch
  2017-08-17 17:50     ` Brandon Williams
@ 2017-08-18 16:06       ` Heiko Voigt
  0 siblings, 0 replies; 200+ results
From: Heiko Voigt @ 2017-08-18 16:06 UTC (permalink / raw)
  To: Brandon Williams; +Cc: Stefan Beller, git, Jonathan Nieder, Jens Lehmann

On Thu, Aug 17, 2017 at 10:50:07AM -0700, Brandon Williams wrote:
> On 08/17, Stefan Beller wrote:
> > On Thu, Aug 17, 2017 at 4:00 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > > To make extending this logic later easier.
> > >
> > > Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> > > ---
> > > I am quite sure I replicated the same logic but a few more eyes would be
> > > appreciated.
> > 
> > A code cleanup is appreciated!
> > 
> > I thought Brandon had a series in flight doing a very similar cleanup here,
> > but in master..pu there is nothing to be found.
> 
> Yeah there are 2 series in flight which will probably conflict here.
> bw/grep-recurse-submodules and bw/submodule-config-cleanup

Ok then I will wait until those are in and then see if I can base the
cleanup on top. I think it is only necessary as a preparation for the
fully fledged fetch configuration logic mess we will get into once we
get to the full recursive submodule fetch implementation. Not
necessarily needed for the moved submodules.

> > 
> > The code looks good to me.

Thanks.

Cheers Heiko

^ permalink raw reply	[relevance 15%]

* Re: Submodule regression in 2.14?
  2017-08-18  4:02         ` Stefan Beller
@ 2017-08-18 16:50           ` Junio C Hamano
  2017-08-18 19:09             ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2017-08-18 16:50 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Lars Schneider, Brandon Williams, git\

Stefan Beller <sbeller@google.com> writes:

> On Thu, Aug 17, 2017 at 7:13 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Stefan Beller <sbeller@google.com> writes:
>>
>>> Are you saying this might be a design mistake and
>>> the .update ought to be respected by all the other
>>> commands? For example
>>>     git reset --recurse-submodules
>>> should ignore the .update= none?
>>
>> I have been under the impression that that has been the traditional
>> desire of what .update ought to mean.  I personally do not have a
>> strong opinion---at least not yet.
>
> In this context note v2.14.0-rc1-34-g7463e2ec3
> (bw/submodule-config-cleanup~7, "unpack-trees:
> don't respect submodule.update") that is going opposite of
> your impression.

Exactly.  We are in agreement that recent developments seem to go
against the traditional desire and it is understandable Lars sees
this as a regression.  I still do not have a strong opinion either
way, if this is a regression or a progress.

> Maybe, I'll think about it. However there is no such
> equivalent for trees (and AFAICT never came up) to
> treat a specific directory other than the rest in worktree
> operations.

I am not sure if I follow.  Submodules are not trees and one of the
reasons people may want to separate things into different modules is
so that they can treat them differently.  If submodules allow you
a richer set of operations than a tree that is part of a monolithic
project, is that necessarily a bad thing?

^ permalink raw reply	[relevance 24%]

* Re: Submodule regression in 2.14?
  2017-08-18 13:12       ` Lars Schneider
@ 2017-08-18 17:16         ` Stefan Beller
  2017-08-18 19:10           ` Junio C Hamano
  2017-08-19 18:24           ` Submodule regression in 2.14? Lars Schneider
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2017-08-18 17:16 UTC (permalink / raw)
  To: Lars Schneider; +Cc: Brandon Williams, git, Junio C Hamano

> In the past "submodule.<name>.update=none" was an easy way
> to selectively disable certain Submodules.
>
> How would I do this with Git 2.14?

    submodule.<name>.active = false

> My gut feeling is that all commands should respect the
> "submodule.<name>.update=none" setting.

Well my gut feeling was that the "update" part of the name
reponds to the subcommand, not the generic action.

For example when you set update=none, git-status,
recursive git-diff still reported the submodule.

>
> - Lars

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] add test for bug in git-mv with nested submodules
  2017-08-18 16:06   ` Heiko Voigt
@ 2017-08-18 19:04     ` Stefan Beller
  2017-09-15 11:50       ` [PATCH v2] add test for bug in git-mv for recursive " Heiko Voigt
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-18 19:04 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Junio C Hamano, git

> I just copied the shortcut that they were adding themselfes as submodule
> in 'setup submodule'. The whole setup of submodules in this test is like
> this. This way we already had a nested submodule structure which I could
> just add.
>
> I agree that this is unrealistic so I can change that in the test I am
> adding. But from what I have seen, this shortcut is taken in quite some
> places when dealing with submodules.

Please do not make it worse.
Once upon a time (late '16 IIRC) I had a series floating on the
list removing all occurrences, but there were issues with the
series and it did not land.

^ permalink raw reply	[relevance 16%]

* Re: Submodule regression in 2.14?
  2017-08-18 16:50           ` Junio C Hamano
@ 2017-08-18 19:09             ` Stefan Beller
  2017-08-19  6:51               ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-18 19:09 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Lars Schneider, Brandon Williams, git

On Fri, Aug 18, 2017 at 9:50 AM, Junio C Hamano <gitster@pobox.com> wrote:
> I am not sure if I follow.  Submodules are not trees and one of the
> reasons people may want to separate things into different modules is
> so that they can treat them differently.  If submodules allow you
> a richer set of operations than a tree that is part of a monolithic
> project, is that necessarily a bad thing?

It is not a bad thing on its own, but we have to consider which
additional actions are useful.

Jonathan brought up the following very long term vision:
Eventually the everyday git commands do not treat submodules
any special than trees, even the submodules git directory
may be non existent (everything is absorbed into the superproject);
so it really feels like a monorepo.
When you want to work on a submodule individually, you have to
make a new working tree.

^ permalink raw reply	[relevance 25%]

* Re: Submodule regression in 2.14?
  2017-08-18 17:16         ` Stefan Beller
@ 2017-08-18 19:10           ` Junio C Hamano
  2017-08-18 22:04             ` [PATCH] pull: respect submodule update configuration Stefan Beller
  2017-08-19 18:24           ` Submodule regression in 2.14? Lars Schneider
  1 sibling, 1 reply; 200+ results
From: Junio C Hamano @ 2017-08-18 19:10 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Lars Schneider, Brandon Williams, git\

Stefan Beller <sbeller@google.com> writes:

>> In the past "submodule.<name>.update=none" was an easy way
>> to selectively disable certain Submodules.
>>
>> How would I do this with Git 2.14?
>
>     submodule.<name>.active = false
>
>> My gut feeling is that all commands should respect the
>> "submodule.<name>.update=none" setting.
>
> Well my gut feeling was that the "update" part of the name
> reponds to the subcommand, not the generic action.
>
> For example when you set update=none, git-status,
> recursive git-diff still reported the submodule.

Both status and diff are read-only operations, so this smells like a
bit bogus argument made by comparing apples and oranges.

I think Lars is more interested in operations that actually affects
the state of submodules by updating them---"submodule update" may be
a prime example as it goes down to run fetch, pull and/or checkout.
It may have been the only thing that affected the state of
submodules before the "--recurse-submodules" option was added to
commands that affect the state of the (super)project, but I would
think that it is not so wrong to expect that these state-affecting
operations running in the "recurse into submodules" mode to honor
"do not update this submodule" that used to be honored only by
"submodule update".

^ permalink raw reply	[relevance 26%]

* Re: [bug] Git submodule command interprets switch as argument and switch
  2017-08-18  5:16 [bug] Git submodule command interprets switch as argument and switch R0b0t1
@ 2017-08-18 19:33 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-18 19:33 UTC (permalink / raw)
  To: R0b0t1, Prathamesh Chavan; +Cc: git

On Thu, Aug 17, 2017 at 10:16 PM, R0b0t1 <r030t1@gmail.com> wrote:
> The issue is as follows:
>
> R0b0t1@host:~/devel/project$ git submodule add
> https://github.com/user/project -f
> Cloning into '/home/R0b0t1/devel/project/-f'...
>
> My .gitignore's first line is *, and then I explicitly allow things.
> Despite the presence of "project/" in the .gitignore the submodule
> command says it is ignored.

That might indicate that another submodule command doesn't
cope with submodule names that look like a common flag.

> The "force" flag is interpreted as a flag
> and also as the destination directory.
>
> It is possible the argument parsing code for other commands exhibits this error.

Yes, though these other commands are in C, not in shell.
Note that Prathamesh is currently porting the "git submodule"
command to C, which would allow us to fix this bug easily.

Also note that the -f is ambigious, what if the user meant
to have the submodule at path "-f" ? (This issue comes
up in many other commands, for example when a path
and a branch name is accepted, the path of a potentially
deleted file.

To solve this git accepts a double dash, which signals git
that anything after the double dash there are arguments not
to be interpreted as a command line flag.


>
> R0b0t1.

^ permalink raw reply	[relevance 26%]

* [PATCH] pull: respect submodule update configuration
  2017-08-18 19:10           ` Junio C Hamano
@ 2017-08-18 22:04             ` Stefan Beller
  2017-08-18 22:05               ` Stefan Beller
  2017-08-19  6:24               ` Junio C Hamano
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2017-08-18 22:04 UTC (permalink / raw)
  To: gitster; +Cc: bmwill, git, larsxschneider, sbeller

From: Lars Schneider <larsxschneider@gmail.com>

Do not override the submodule configuration in the call to update
the submodules, but give a weaker default.

Reported-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
  
Personally I dislike this patch, but I have no better idea for the time
being.

Thanks,
Stefan 

 builtin/pull.c             |  6 ++++--
 git-submodule.sh           |  7 ++++++-
 t/t7400-submodule-basic.sh | 22 ++++++++++++++++++++++
 3 files changed, 32 insertions(+), 3 deletions(-)

diff --git a/builtin/pull.c b/builtin/pull.c
index 9b86e519b1..be4f74d764 100644
--- a/builtin/pull.c
+++ b/builtin/pull.c
@@ -553,7 +553,8 @@ static int rebase_submodules(void)
 	cp.git_cmd = 1;
 	cp.no_stdin = 1;
 	argv_array_pushl(&cp.args, "submodule", "update",
-				   "--recursive", "--rebase", NULL);
+				   "--recursive", "--default-update",
+				   "rebase", NULL);
 
 	return run_command(&cp);
 }
@@ -565,7 +566,8 @@ static int update_submodules(void)
 	cp.git_cmd = 1;
 	cp.no_stdin = 1;
 	argv_array_pushl(&cp.args, "submodule", "update",
-				   "--recursive", "--checkout", NULL);
+				   "--recursive", "--default-update",
+				   "checkout", NULL);
 
 	return run_command(&cp);
 }
diff --git a/git-submodule.sh b/git-submodule.sh
index e131760eec..6dbc32e686 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -511,6 +511,7 @@ fetch_in_submodule () (
 cmd_update()
 {
 	# parse $args after "submodule ... update".
+	default_update="checkout"
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -552,6 +553,10 @@ cmd_update()
 		--checkout)
 			update="checkout"
 			;;
+		--default-update)
+			default_update="$2"
+			shift
+			;;
 		--recommend-shallow)
 			recommend_shallow="--recommend-shallow"
 			;;
@@ -619,7 +624,7 @@ cmd_update()
 			update_module=$(git config submodule."$name".update)
 			if test -z "$update_module"
 			then
-				update_module="checkout"
+				update_module="$default_update"
 			fi
 		fi
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index dcac364c5f..ff64bf8528 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1289,4 +1289,26 @@ test_expect_success 'init properly sets the config' '
 	test_must_fail git -C multisuper_clone config --get submodule.sub1.active
 '
 
+test_expect_success 'submodule update and git pull with disabled submodule' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	cat <<-\EOF >expect &&
+	-sub0
+	 sub1 (test2)
+	 sub2 (test2)
+	 sub3 (test2)
+	 sub4 (test2)
+	 sub5 (test2)
+	EOF
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	(
+		cd multisuper_clone &&
+		git config --local submodule.sub0.update none &&
+		git submodule update --init --recursive &&
+		git pull --recurse-submodules &&
+		git submodule status | cut -c 1,43- >actual
+	) &&
+	test_cmp expect multisuper_clone/actual
+'
+
 test_done
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 30%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-18 22:04             ` [PATCH] pull: respect submodule update configuration Stefan Beller
@ 2017-08-18 22:05               ` Stefan Beller
  2017-08-19  6:17                 ` Junio C Hamano
  2017-08-19  6:24               ` Junio C Hamano
  1 sibling, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-18 22:05 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Brandon Williams, git, Lars Schneider, Stefan Beller

On Fri, Aug 18, 2017 at 3:04 PM, Stefan Beller <sbeller@google.com> wrote:
> From: Lars Schneider <larsxschneider@gmail.com>

eh, that is what I get for amending to Lars patch.

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-18 22:05               ` Stefan Beller
@ 2017-08-19  6:17                 ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-08-19  6:17 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Brandon Williams, git\, Lars Schneider

Stefan Beller <sbeller@google.com> writes:

> On Fri, Aug 18, 2017 at 3:04 PM, Stefan Beller <sbeller@google.com> wrote:
>> From: Lars Schneider <larsxschneider@gmail.com>
>
> eh, that is what I get for amending to Lars patch.

Sorry, I do not understand this remark.  

If you started from a patch by Lars (I do not recall seeing it but
the list is high volume so it is entirely plausible that I may have
missed it) and tweaked it, it is more than OK to keep the original
author and record it in an in-body From: header like you did,
instead of taking authorship over.



^ permalink raw reply	[relevance 15%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-18 22:04             ` [PATCH] pull: respect submodule update configuration Stefan Beller
  2017-08-18 22:05               ` Stefan Beller
@ 2017-08-19  6:24               ` Junio C Hamano
  2017-08-21 16:20                 ` Heiko Voigt
  1 sibling, 1 reply; 200+ results
From: Junio C Hamano @ 2017-08-19  6:24 UTC (permalink / raw)
  To: Stefan Beller; +Cc: bmwill, git, larsxschneider

Stefan Beller <sbeller@google.com> writes:

> From: Lars Schneider <larsxschneider@gmail.com>
>
> Do not override the submodule configuration in the call to update
> the submodules, but give a weaker default.
>
> Reported-by: Lars Schneider <larsxschneider@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>   
> Personally I dislike this patch, but I have no better idea for the time
> being.

The patch text from a cursory look seems reasonable to me.

It's not like you have 47 different codepaths that need to pay
attention to the .update config and they all have to pass the new
--default-update option, this is merely to fix one of them that
relates to the problem reported by Lars, and you need a similar fix
to other 46, right?

If you want the "--recurse-submodules" thing to always do the
"weaker default" thing in your project, you can choose not to set
.update to custom values in any of your submodules, so I do not
think the reason why you dislike this change is because it would
affect your use of submodules.

So I am a bit curious to learn which part of this change you dislike
and why.


>  builtin/pull.c             |  6 ++++--
>  git-submodule.sh           |  7 ++++++-
>  t/t7400-submodule-basic.sh | 22 ++++++++++++++++++++++
>  3 files changed, 32 insertions(+), 3 deletions(-)
>
> diff --git a/builtin/pull.c b/builtin/pull.c
> index 9b86e519b1..be4f74d764 100644
> --- a/builtin/pull.c
> +++ b/builtin/pull.c
> @@ -553,7 +553,8 @@ static int rebase_submodules(void)
>  	cp.git_cmd = 1;
>  	cp.no_stdin = 1;
>  	argv_array_pushl(&cp.args, "submodule", "update",
> -				   "--recursive", "--rebase", NULL);
> +				   "--recursive", "--default-update",
> +				   "rebase", NULL);
>  
>  	return run_command(&cp);
>  }
> @@ -565,7 +566,8 @@ static int update_submodules(void)
>  	cp.git_cmd = 1;
>  	cp.no_stdin = 1;
>  	argv_array_pushl(&cp.args, "submodule", "update",
> -				   "--recursive", "--checkout", NULL);
> +				   "--recursive", "--default-update",
> +				   "checkout", NULL);
>  
>  	return run_command(&cp);
>  }
> diff --git a/git-submodule.sh b/git-submodule.sh
> index e131760eec..6dbc32e686 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -511,6 +511,7 @@ fetch_in_submodule () (
>  cmd_update()
>  {
>  	# parse $args after "submodule ... update".
> +	default_update="checkout"
>  	while test $# -ne 0
>  	do
>  		case "$1" in
> @@ -552,6 +553,10 @@ cmd_update()
>  		--checkout)
>  			update="checkout"
>  			;;
> +		--default-update)
> +			default_update="$2"
> +			shift
> +			;;
>  		--recommend-shallow)
>  			recommend_shallow="--recommend-shallow"
>  			;;
> @@ -619,7 +624,7 @@ cmd_update()
>  			update_module=$(git config submodule."$name".update)
>  			if test -z "$update_module"
>  			then
> -				update_module="checkout"
> +				update_module="$default_update"
>  			fi
>  		fi
>  
> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
> index dcac364c5f..ff64bf8528 100755
> --- a/t/t7400-submodule-basic.sh
> +++ b/t/t7400-submodule-basic.sh
> @@ -1289,4 +1289,26 @@ test_expect_success 'init properly sets the config' '
>  	test_must_fail git -C multisuper_clone config --get submodule.sub1.active
>  '
>  
> +test_expect_success 'submodule update and git pull with disabled submodule' '
> +	test_when_finished "rm -rf multisuper_clone" &&
> +	pwd=$(pwd) &&
> +	cat <<-\EOF >expect &&
> +	-sub0
> +	 sub1 (test2)
> +	 sub2 (test2)
> +	 sub3 (test2)
> +	 sub4 (test2)
> +	 sub5 (test2)
> +	EOF
> +	git clone file://"$pwd"/multisuper multisuper_clone &&
> +	(
> +		cd multisuper_clone &&
> +		git config --local submodule.sub0.update none &&
> +		git submodule update --init --recursive &&
> +		git pull --recurse-submodules &&
> +		git submodule status | cut -c 1,43- >actual
> +	) &&
> +	test_cmp expect multisuper_clone/actual
> +'
> +
>  test_done

^ permalink raw reply	[relevance 25%]

* Re: Submodule regression in 2.14?
  2017-08-18 19:09             ` Stefan Beller
@ 2017-08-19  6:51               ` Junio C Hamano
  2017-08-21 16:05                 ` Heiko Voigt
  2017-08-21 16:46                 ` Stefan Beller
  0 siblings, 2 replies; 200+ results
From: Junio C Hamano @ 2017-08-19  6:51 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Lars Schneider, Brandon Williams, git\

Stefan Beller <sbeller@google.com> writes:

> Jonathan brought up the following very long term vision:
> Eventually the everyday git commands do not treat submodules
> any special than trees, even the submodules git directory
> may be non existent (everything is absorbed into the superproject);
> so it really feels like a monorepo.

That may be one valid option to have but I do not see a reason why
it needs to be the only valid option.  So I do not see why you are
bringing it up in this thread.

But that is a good starting point to discuss one possible future.
Let's think aloud how that world would look like.

 * When you "git clone" a superproject (perhaps implicitly with the
   "--recurse-submodules" option), the top-level project and all of
   its submodules will be checked out on the same branch (presumably
   the 'master' branch, which is the default).

 * Your attempt to "git commit", "git branch", "git checkout -b",
   etc. inside a submodule will either fail, or will implicitly
   chdir up to the top-level superproject and turn into the
   corresponding command with "--recurse-submodules".

 * "git commit --recurse-submodules -a" from the top-level will grab
   all the local changes, depth-first and recursively, in
   submodules, makes a commit, binds the new commit to the index of
   the superproject that immediately contains the submodule and
   makes a commit in it, until it percolates all the way up to the
   superproject.  When working in this mode, branches in submodules
   do not really matter; the gitlink in the superproject is the only
   thing that matters.

 * It naturally follows that between two adjacent commits C and C~1
   in the superproject's history, the commit in a submodule bound to
   C and the commit in a submodule bound to C~1 are either the same
   (i.e. superproject made a commit but there was no change in the
   submodule), or they are in direct parent-child relationship
   (i.e. the local changes made to the submodule was recorded as a
   single commit when the superproject made the commit).

 * "git push --recurse-submodules" from the top-level will push the
   history of the superproject out, together with the history of the
   submodules.

I think it is doable, but a mechanism to enumerate all the commits
bound from submodules to a range of superproject's commits needs to
be invented to drive the pack-objects for efficient object transfer.
Having it would also help in fsck and gc---as branches are immaterial
in the submodule repositories, commits in superprojects that are
reachable from refs will have to serve as the connectivity anchors
for commit DAG in the submodule histories.

As long as we are talking about idealized future world (well, at
least an idea of somebody's "ideal", not necessarily shared by
everybody), I wonder if there is even any need to have commits in
submodules in such a world.  To realize such a "monorepo" world, you
might be better off allowing a gitlink in the superproject to
directly point at a tree object in a submodule repository (making
them physically a single repository is an optional implementation
detail I choose to ignore in this discussion).

^ permalink raw reply	[relevance 23%]

* Re: Submodule regression in 2.14?
  2017-08-18 17:16         ` Stefan Beller
  2017-08-18 19:10           ` Junio C Hamano
@ 2017-08-19 18:24           ` Lars Schneider
  1 sibling, 0 replies; 200+ results
From: Lars Schneider @ 2017-08-19 18:24 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Brandon Williams, git, Junio C Hamano


> On 18 Aug 2017, at 19:16, Stefan Beller <sbeller@google.com> wrote:
> 
>> In the past "submodule.<name>.update=none" was an easy way
>> to selectively disable certain Submodules.
>> 
>> How would I do this with Git 2.14?
> 
>    submodule.<name>.active = false

That's what I thought after your first response. However,
this test case fails for me, too:


diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index dcac364c5f..24f9729015 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1289,4 +1289,19 @@ test_expect_success 'init properly sets the config' '
	test_must_fail git -C multisuper_clone config --get submodule.sub1.active
'

+test_expect_success 'submodule update and git pull with disabled submodule' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	(
+		cd multisuper_clone &&
+		git config --local submodule.sub0.update none &&
+		git config --local submodule.sub0.active false &&
+		git submodule update --init --recursive &&
+		git pull --recurse-submodules &&
+		git submodule status | cut -c 1,43- >actual
+	) &&
+	ls &&
+	test_cmp expect multisuper_clone/actual
+'
+
test_done


Here is the relevant part of the Git config:

	[submodule "sub0"]
		update = none
		active = false

Is this a bug?


>> My gut feeling is that all commands should respect the
>> "submodule.<name>.update=none" setting.
> 
> Well my gut feeling was that the "update" part of the name
> reponds to the subcommand, not the generic action.

I see. But wouldn't that be inconsistent with the config
"active" then? Following that logic "active" would only
respond to the (non-existent) "active" subcommand, no?


> For example when you set update=none, git-status,
> recursive git-diff still reported the submodule.

My understanding is this:

(1) "active" controls if a submodule is considered by
    Git at all.

(2) "update" controls how and if the submodule pointer
    modified

Is this your intention? What would be the use case for
"active=true" and "update=none"? 


- Lars

^ permalink raw reply	[relevance 17%]

* [GSoC][PATCH 1/4] submodule--helper: introduce get_submodule_displaypath()
@ 2017-08-21 16:15 Prathamesh Chavan
  2017-08-21 16:15 ` [GSoC][PATCH 2/4] submodule--helper: introduce for_each_submodule_list() Prathamesh Chavan
                   ` (2 more replies)
  0 siblings, 3 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-21 16:15 UTC (permalink / raw)
  To: git; +Cc: sbeller, christian.couder, gitster, Prathamesh Chavan

Introduce function get_submodule_displaypath() to replace the code
occurring in submodule_init() for generating displaypath of the
submodule with a call to it.

This new function will also be used in other parts of the system
in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
As said in the previous update,
a short patch series is floated for the maintainer's review,
and is consisting of the following changes:
* introduce function get_submodule_displaypath()
* introduce function for_each_submodule_list()
* port function set_name_rev() from shell to C
* port submodule subcommand 'status' from shell to C

The complete build report for the above series of patches is available
at:
https://travis-ci.org/pratham-pc/git/builds
Branch: week-14-1
Build #158
The above changes are also push on github and are available at:
https://github.com/pratham-pc/git/commits/week-14-1

The above changes were based on master branch.
But along with the above changes, the same series was also applied
to a separate branch based on the 'next' branch. The changes were
applicable without any additional changes to the patches.
Complete build report of that is also available at:
https://travis-ci.org/pratham-pc/git/builds
Branch: week-14-1-next
Build #159
The above changes are also push on github and are available at:
https://github.com/pratham-pc/git/commits/week-14-1-next

 builtin/submodule--helper.c | 33 ++++++++++++++++++++++-----------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 84562ec83..dcdbde963 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -220,6 +220,27 @@ static int resolve_relative_url_test(int argc, const char **argv, const char *pr
 	return 0;
 }
 
+static char *get_submodule_displaypath(const char *path, const char *prefix)
+{
+	const char *super_prefix = get_super_prefix();
+
+	if (prefix && super_prefix) {
+		BUG("cannot have prefix '%s' and superprefix '%s'",
+		    prefix, super_prefix);
+	} else if (prefix) {
+		struct strbuf sb = STRBUF_INIT;
+		char *displaypath = xstrdup(relative_path(path, prefix, &sb));
+		strbuf_release(&sb);
+		return displaypath;
+	} else if (super_prefix) {
+		int len = strlen(super_prefix);
+		const char *format = is_dir_sep(super_prefix[len - 1]) ? "%s%s" : "%s/%s";
+		return xstrfmt(format, super_prefix, path);
+	} else {
+		return xstrdup(path);
+	}
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -339,16 +360,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 
 	/* Only loads from .gitmodules, no overlay with .git/config */
 	gitmodules_config();
-
-	if (prefix && get_super_prefix())
-		die("BUG: cannot have prefix and superprefix");
-	else if (prefix)
-		displaypath = xstrdup(relative_path(path, prefix, &sb));
-	else if (get_super_prefix()) {
-		strbuf_addf(&sb, "%s%s", get_super_prefix(), path);
-		displaypath = strbuf_detach(&sb, NULL);
-	} else
-		displaypath = xstrdup(path);
+	displaypath = get_submodule_displaypath(path, prefix);
 
 	sub = submodule_from_path(&null_oid, path);
 
@@ -363,7 +375,6 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * Set active flag for the submodule being initialized
 	 */
 	if (!is_submodule_active(the_repository, path)) {
-		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
 	}
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH 2/4] submodule--helper: introduce for_each_submodule_list()
  2017-08-21 16:15 [GSoC][PATCH 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
@ 2017-08-21 16:15 ` Prathamesh Chavan
  2017-08-21 16:15 ` [GSoC][PATCH 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
  2017-08-21 16:15 ` [GSoC][PATCH 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  2 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-21 16:15 UTC (permalink / raw)
  To: git; +Cc: sbeller, christian.couder, gitster, Prathamesh Chavan

Introduce function for_each_submodule_list() and
replace a loop in module_init() with a call to it.

The new function will also be used in other parts of the
system in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 39 +++++++++++++++++++++++++++++----------
 1 file changed, 29 insertions(+), 10 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index dcdbde963..7803457ba 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -14,6 +14,9 @@
 #include "refs.h"
 #include "connect.h"
 
+typedef void (*submodule_list_func_t)(const struct cache_entry *list_item,
+				      void *cb_data);
+
 static char *get_default_remote(void)
 {
 	char *dest = NULL, *ret;
@@ -352,17 +355,30 @@ static int module_list(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
-static void init_submodule(const char *path, const char *prefix, int quiet)
+static void for_each_submodule_list(const struct module_list list,
+				    submodule_list_func_t fn, void *cb_data)
 {
+	int i;
+	for (i = 0; i < list.nr; i++)
+		fn(list.entries[i], cb_data);
+}
+
+struct init_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+};
+#define INIT_CB_INIT { NULL, 0 }
+
+static void init_submodule(const struct cache_entry *list_item, void *cb_data)
+{
+	struct init_cb *info = cb_data;
 	const struct submodule *sub;
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	/* Only loads from .gitmodules, no overlay with .git/config */
-	gitmodules_config();
-	displaypath = get_submodule_displaypath(path, prefix);
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(&null_oid, list_item->name);
 
 	if (!sub)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
@@ -374,7 +390,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 *
 	 * Set active flag for the submodule being initialized
 	 */
-	if (!is_submodule_active(the_repository, path)) {
+	if (!is_submodule_active(the_repository, list_item->name)) {
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
 	}
@@ -416,7 +432,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		if (git_config_set_gently(sb.buf, url))
 			die(_("Failed to register url for submodule path '%s'"),
 			    displaypath);
-		if (!quiet)
+		if (!info->quiet)
 			fprintf(stderr,
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
@@ -445,10 +461,10 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 
 static int module_init(int argc, const char **argv, const char *prefix)
 {
+	struct init_cb info = INIT_CB_INIT;
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
-	int i;
 
 	struct option module_init_options[] = {
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
@@ -473,8 +489,11 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
 
-	for (i = 0; i < list.nr; i++)
-		init_submodule(list.entries[i]->name, prefix, quiet);
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+
+	gitmodules_config();
+	for_each_submodule_list(list, init_submodule, &info);
 
 	return 0;
 }
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH 3/4] submodule: port set_name_rev() from shell to C
  2017-08-21 16:15 [GSoC][PATCH 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
  2017-08-21 16:15 ` [GSoC][PATCH 2/4] submodule--helper: introduce for_each_submodule_list() Prathamesh Chavan
@ 2017-08-21 16:15 ` Prathamesh Chavan
  2017-08-21 16:47   ` Heiko Voigt
  2017-08-21 16:15 ` [GSoC][PATCH 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  2 siblings, 1 reply; 200+ results
From: Prathamesh Chavan @ 2017-08-21 16:15 UTC (permalink / raw)
  To: git; +Cc: sbeller, christian.couder, gitster, Prathamesh Chavan

Function set_name_rev() is ported from git-submodule to the
submodule--helper builtin. The function get_name_rev() generates the
value of the revision name as required, and the function
print_name_rev() handles the formating and printing of the obtained
revision name.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 63 +++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            | 16 ++----------
 2 files changed, 65 insertions(+), 14 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 7803457ba..a4bff3f38 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -244,6 +244,68 @@ static char *get_submodule_displaypath(const char *path, const char *prefix)
 	}
 }
 
+static char *get_name_rev(const char *sub_path, const char* object_id)
+{
+	struct strbuf sb = STRBUF_INIT;
+	const char ***d;
+
+	static const char *describe_bare[] = {
+		NULL
+	};
+
+	static const char *describe_tags[] = {
+		"--tags", NULL
+	};
+
+	static const char *describe_contains[] = {
+		"--contains", NULL
+	};
+
+	static const char *describe_all_always[] = {
+		"--all", "--always", NULL
+	};
+
+	static const char **describe_argv[] = {
+		describe_bare, describe_tags, describe_contains,
+		describe_all_always, NULL
+	};
+
+	for (d = describe_argv; *d; d++) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+		prepare_submodule_repo_env(&cp.env_array);
+		cp.dir = sub_path;
+		cp.git_cmd = 1;
+		cp.no_stderr = 1;
+
+		argv_array_push(&cp.args, "describe");
+		argv_array_pushv(&cp.args, *d);
+		argv_array_push(&cp.args, object_id);
+
+		if (!capture_command(&cp, &sb, 0) && sb.len) {
+			strbuf_strip_suffix(&sb, "\n");
+			return strbuf_detach(&sb, NULL);
+		}
+	}
+
+	strbuf_release(&sb);
+	return NULL;
+}
+
+static int print_name_rev(int argc, const char **argv, const char *prefix)
+{
+	char *namerev;
+	if (argc != 3)
+		die("print-name-rev only accepts two arguments: <path> <sha1>");
+
+	namerev = get_name_rev(argv[1], argv[2]);
+	if (namerev && namerev[0])
+		printf(" (%s)", namerev);
+	printf("\n");
+
+	free(namerev);
+	return 0;
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -1242,6 +1304,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"print-name-rev", print_name_rev, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index e131760ee..e988167e0 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -759,18 +759,6 @@ cmd_update()
 	}
 }
 
-set_name_rev () {
-	revname=$( (
-		sanitize_submodule_env
-		cd "$1" && {
-			git describe "$2" 2>/dev/null ||
-			git describe --tags "$2" 2>/dev/null ||
-			git describe --contains "$2" 2>/dev/null ||
-			git describe --all --always "$2"
-		}
-	) )
-	test -z "$revname" || revname=" ($revname)"
-}
 #
 # Show commit summary for submodules in index or working tree
 #
@@ -1042,14 +1030,14 @@ cmd_status()
 		fi
 		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
 		then
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper print-name-rev "$sm_path" "$sha1")
 			say " $sha1 $displaypath$revname"
 		else
 			if test -z "$cached"
 			then
 				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
 			fi
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper print-name-rev "$sm_path" "$sha1")
 			say "+$sha1 $displaypath$revname"
 		fi
 
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH 4/4] submodule: port submodule subcommand 'status' from shell to C
  2017-08-21 16:15 [GSoC][PATCH 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
  2017-08-21 16:15 ` [GSoC][PATCH 2/4] submodule--helper: introduce for_each_submodule_list() Prathamesh Chavan
  2017-08-21 16:15 ` [GSoC][PATCH 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
@ 2017-08-21 16:15 ` " Prathamesh Chavan
  2 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-21 16:15 UTC (permalink / raw)
  To: git; +Cc: sbeller, christian.couder, gitster, Prathamesh Chavan

This aims to make git-submodule 'status' a built-in. Hence, the function
cmd_status() is ported from shell to C. This is done by introducing
three functions: module_status(), submodule_status() and print_status().

The function module_status() acts as the front-end of the subcommand.
It parses subcommand's options and then calls the function
module_list_compute() for computing the list of submodules. Then
this functions calls for_each_submodule_list() looping through the
list obtained.

Then for_each_submodule_list() calls submodule_status() for each of the
submodule in its list. The function submodule_status() is responsible
for generating the status each submodule it is called for, and
then calls print_status().

Finally, the function print_status() handles the printing of submodule's
status.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
The patch below underwent some changes after its previous version.

In the shell script, the submodules, which had merge-conflicts are
identified by checking if the $stage variable has the value 'U'
Till the last patch series, this was done by checking if ce_flags have
a non-zero value. In this new patch series, this was changed and
instead ce_stage() was called on the list_item(cache_entry), and then
we check whether it has a non-zero value.

 builtin/submodule--helper.c | 156 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  49 +-------------
 2 files changed, 157 insertions(+), 48 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index a4bff3f38..831370d6e 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -560,6 +560,161 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+struct status_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+	unsigned int recursive: 1;
+	unsigned int cached: 1;
+};
+#define STATUS_CB_INIT { NULL, 0, 0, 0 }
+
+static void print_status(struct status_cb *info, char state, const char *path,
+			 const struct object_id *oid, const char *displaypath)
+{
+	if (info->quiet)
+		return;
+
+	printf("%c%s %s", state, oid_to_hex(oid), displaypath);
+
+	if (state == ' ' || state == '+') {
+		struct argv_array name_rev_args = ARGV_ARRAY_INIT;
+
+		argv_array_pushl(&name_rev_args, "print-name-rev",
+				 path, oid_to_hex(oid), NULL);
+		print_name_rev(name_rev_args.argc, name_rev_args.argv,
+			       info->prefix);
+
+		argv_array_clear(&name_rev_args);
+	} else {
+		printf("\n");
+	}
+}
+
+static int handle_submodule_head_ref(const char *refname,
+				     const struct object_id *oid, int flags,
+				     void *cb_data)
+{
+	struct object_id *output = cb_data;
+	if (oid)
+		oidcpy(output, oid);
+
+	return 0;
+}
+
+static void status_submodule(const struct cache_entry *list_item, void *cb_data)
+{
+	struct status_cb *info = cb_data;
+	char *displaypath;
+	struct argv_array diff_files_args = ARGV_ARRAY_INIT;
+
+	if (!submodule_from_path(&null_oid, list_item->name))
+		die(_("no submodule mapping found in .gitmodules for path '%s'"),
+		      list_item->name);
+
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
+
+	if (ce_stage(list_item)) {
+		print_status(info, 'U', list_item->name,
+			     &null_oid, displaypath);
+		goto cleanup;
+	}
+
+	if (!is_submodule_active(the_repository, list_item->name)) {
+		print_status(info, '-', list_item->name, &list_item->oid,
+			     displaypath);
+		goto cleanup;
+	}
+
+	argv_array_pushl(&diff_files_args, "diff-files",
+			 "--ignore-submodules=dirty", "--quiet", "--",
+			 list_item->name, NULL);
+
+	if (!cmd_diff_files(diff_files_args.argc, diff_files_args.argv,
+			    info->prefix)) {
+		print_status(info, ' ', list_item->name, &list_item->oid,
+			     displaypath);
+	} else {
+		if (!info->cached) {
+			struct object_id oid;
+
+			if (head_ref_submodule(list_item->name,
+					       handle_submodule_head_ref, &oid))
+				die(_("could not resolve HEAD ref inside the"
+				      "submodule '%s'"), list_item->name);
+
+			print_status(info, '+', list_item->name, &oid,
+				     displaypath);
+		} else {
+			print_status(info, '+', list_item->name,
+				     &list_item->oid, displaypath);
+		}
+	}
+
+	if (info->recursive) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = list_item->name;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_pushl(&cpr.args, "--super-prefix", displaypath,
+				 "submodule--helper", "status", "--recursive",
+				 NULL);
+
+		if (info->cached)
+			argv_array_push(&cpr.args, "--cached");
+
+		if (info->quiet)
+			argv_array_push(&cpr.args, "--quiet");
+
+		if (run_command(&cpr))
+			die(_("failed to recurse into submodule '%s'"),
+			      list_item->name);
+	}
+
+cleanup:
+	argv_array_clear(&diff_files_args);
+	free(displaypath);
+}
+
+static int module_status(int argc, const char **argv, const char *prefix)
+{
+	struct status_cb info = STATUS_CB_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+	int quiet = 0;
+	int cached = 0;
+	int recursive = 0;
+
+	struct option module_status_options[] = {
+		OPT__QUIET(&quiet, N_("Suppress submodule status output")),
+		OPT_BOOL(0, "cached", &cached, N_("Use commit stored in the index instead of the one stored in the submodule HEAD")),
+		OPT_BOOL(0, "recursive", &recursive, N_("Recurse into nested submodules")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule status [--quiet] [--cached] [--recursive] [<path>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_status_options,
+			     git_submodule_helper_usage, 0);
+
+	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+	info.recursive = !!recursive;
+	info.cached = !!cached;
+
+	gitmodules_config();
+	for_each_submodule_list(list, status_submodule, &info);
+
+	return 0;
+}
+
 static int module_name(int argc, const char **argv, const char *prefix)
 {
 	const struct submodule *sub;
@@ -1306,6 +1461,7 @@ static struct cmd_struct commands[] = {
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
 	{"print-name-rev", print_name_rev, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
+	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
diff --git a/git-submodule.sh b/git-submodule.sh
index e988167e0..51b057d82 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1005,54 +1005,7 @@ cmd_status()
 		shift
 	done
 
-	{
-		git submodule--helper list --prefix "$wt_prefix" "$@" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path") || exit
-		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-		if test "$stage" = U
-		then
-			say "U$sha1 $displaypath"
-			continue
-		fi
-		if ! git submodule--helper is-active "$sm_path" ||
-		{
-			! test -d "$sm_path"/.git &&
-			! test -f "$sm_path"/.git
-		}
-		then
-			say "-$sha1 $displaypath"
-			continue;
-		fi
-		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
-		then
-			revname=$(git submodule--helper print-name-rev "$sm_path" "$sha1")
-			say " $sha1 $displaypath$revname"
-		else
-			if test -z "$cached"
-			then
-				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
-			fi
-			revname=$(git submodule--helper print-name-rev "$sm_path" "$sha1")
-			say "+$sha1 $displaypath$revname"
-		fi
-
-		if test -n "$recursive"
-		then
-			(
-				prefix="$displaypath/"
-				sanitize_submodule_env
-				wt_prefix=
-				cd "$sm_path" &&
-				eval cmd_status
-			) ||
-			die "$(eval_gettext "Failed to recurse into submodule path '\$sm_path'")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper status ${GIT_QUIET:+--quiet} ${cached:+--cached} ${recursive:+--recursive} "$@"
 }
 #
 # Sync remote urls for submodules
-- 
2.13.0


^ permalink raw reply	[relevance 20%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-19  6:24               ` Junio C Hamano
@ 2017-08-21 16:20                 ` Heiko Voigt
  2017-08-21 16:55                   ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Heiko Voigt @ 2017-08-21 16:20 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Stefan Beller, bmwill, git, larsxschneider

On Fri, Aug 18, 2017 at 11:24:47PM -0700, Junio C Hamano wrote:
> Stefan Beller <sbeller@google.com> writes:
> 
> > From: Lars Schneider <larsxschneider@gmail.com>
> >
> > Do not override the submodule configuration in the call to update
> > the submodules, but give a weaker default.
> >
> > Reported-by: Lars Schneider <larsxschneider@gmail.com>
> > Signed-off-by: Stefan Beller <sbeller@google.com>
> > ---
> >   
> > Personally I dislike this patch, but I have no better idea for the time
> > being.
> 
> The patch text from a cursory look seems reasonable to me.
> 
> It's not like you have 47 different codepaths that need to pay
> attention to the .update config and they all have to pass the new
> --default-update option, this is merely to fix one of them that
> relates to the problem reported by Lars, and you need a similar fix
> to other 46, right?
> 
> If you want the "--recurse-submodules" thing to always do the
> "weaker default" thing in your project, you can choose not to set
> .update to custom values in any of your submodules, so I do not
> think the reason why you dislike this change is because it would
> affect your use of submodules.
> 
> So I am a bit curious to learn which part of this change you dislike
> and why.

I am also curious. Isn't this the same strategy we are using in other
places?

Cheers Heiko

^ permalink raw reply	[relevance 7%]

* Re: Submodule regression in 2.14?
  2017-08-21 16:05                 ` Heiko Voigt
@ 2017-08-21 16:42                   ` Stefan Beller
  2017-08-22 15:33                     ` Heiko Voigt
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-21 16:42 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Junio C Hamano, Lars Schneider, Brandon Williams, git

On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
>> As long as we are talking about idealized future world (well, at
>> least an idea of somebody's "ideal", not necessarily shared by
>> everybody), I wonder if there is even any need to have commits in
>> submodules in such a world.  To realize such a "monorepo" world, you
>> might be better off allowing a gitlink in the superproject to
>> directly point at a tree object in a submodule repository (making
>> them physically a single repository is an optional implementation
>> detail I choose to ignore in this discussion).
>
> IMO this is one step to far. One main use of submodules are shared
> repositories that are used by many superprojects. The reason you want to
> have commits in the submodule are so that you can push them
> independently and all other users can pick up the changes. You could get
> by by Using the superproject commits for the submodule once you push or
> something but those do not necessarily make sense in the context of the
> submodule.
>
> So I think it is important that there are commits in the submodule so
> its history makes sense independently for others.
>
> Or how would you push out the history in the submodule in your idea?
> Maybe I am missing something? What would be your use case with gitlinks
> pointing to trees?

Well there are still commits, but in the superproject the UX feels more
as if the submodules were special trees. So if you want to interact with
the submodule specifically, you could do things like

    git add /path/inside/sub
    # works seamlessly from the superproject tree

    git commit --submodule-commit-only
    # When the flag is not give, you may get an editor
    # asking for two commit messages, (sub+super)

    git fetch --submodule
    # When the flag is not given, we'd fetch superproject and
    # on-demand

    # You feel the superproject is in the way?
    git worktree add --for-submodule <path/to/sub> ...
    # The new submodule worktree puts the
    # submodule only UX first. so it feels like its own
    # repository, no need for specific flags.


>
> Cheers Heiko

^ permalink raw reply	[relevance 26%]

* Re: Submodule regression in 2.14?
  2017-08-19  6:51               ` Junio C Hamano
  2017-08-21 16:05                 ` Heiko Voigt
@ 2017-08-21 16:46                 ` Stefan Beller
  2017-08-21 22:45                   ` Junio C Hamano
  1 sibling, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-21 16:46 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Lars Schneider, Brandon Williams, git

On Fri, Aug 18, 2017 at 11:51 PM, Junio C Hamano <gitster@pobox.com> wrote:

> As long as we are talking about idealized future world (well, at
> least an idea of somebody's "ideal", not necessarily shared by
> everybody), I wonder if there is even any need to have commits in
> submodules in such a world.  To realize such a "monorepo" world, you
> might be better off allowing a gitlink in the superproject to
> directly point at a tree object in a submodule repository (making
> them physically a single repository is an optional implementation
> detail I choose to ignore in this discussion).

Then the sharing between superprojects (e.g. send an Android's linux
patch upstream or to another distro that also uses a superproject),
becomes cumbersome as the commit messages are missing and
potentially not specific to that subtree.

^ permalink raw reply	[relevance 16%]

* Re: [GSoC][PATCH 3/4] submodule: port set_name_rev() from shell to C
  2017-08-21 16:15 ` [GSoC][PATCH 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
@ 2017-08-21 16:47   ` Heiko Voigt
  2017-08-21 17:24     ` Prathamesh Chavan
  0 siblings, 1 reply; 200+ results
From: Heiko Voigt @ 2017-08-21 16:47 UTC (permalink / raw)
  To: Prathamesh Chavan; +Cc: git, sbeller, christian.couder, gitster

On Mon, Aug 21, 2017 at 09:45:14PM +0530, Prathamesh Chavan wrote:
> Function set_name_rev() is ported from git-submodule to the
> submodule--helper builtin. The function get_name_rev() generates the
> value of the revision name as required, and the function
> print_name_rev() handles the formating and printing of the obtained
> revision name.
> 
> Mentored-by: Christian Couder <christian.couder@gmail.com>
> Mentored-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
> ---
>  builtin/submodule--helper.c | 63 +++++++++++++++++++++++++++++++++++++++++++++
>  git-submodule.sh            | 16 ++----------
>  2 files changed, 65 insertions(+), 14 deletions(-)
> 
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index 7803457ba..a4bff3f38 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c

[...]

> @@ -1242,6 +1304,7 @@ static struct cmd_struct commands[] = {
>  	{"relative-path", resolve_relative_path, 0},
>  	{"resolve-relative-url", resolve_relative_url, 0},
>  	{"resolve-relative-url-test", resolve_relative_url_test, 0},
> +	{"print-name-rev", print_name_rev, 0},

I see the function in git-submodule.sh was named kind of reverse. How
about we do it more naturally here and call this 'rev-name' instead?
That makes is more clear to me and is also similar to the used variable
name 'revname'.

I would also prefix it differently like 'get' or 'calculate' instead of
'print' since it tries to find a name and is not just a simple lookup.

So in summary I would prefer something like 'get-rev-name' as a name for
the subcommand here.

>  	{"init", module_init, SUPPORT_SUPER_PREFIX},
>  	{"remote-branch", resolve_remote_submodule_branch, 0},
>  	{"push-check", push_check, 0},
> diff --git a/git-submodule.sh b/git-submodule.sh
> index e131760ee..e988167e0 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -759,18 +759,6 @@ cmd_update()
>  	}
>  }
>  
> -set_name_rev () {
> -	revname=$( (
> -		sanitize_submodule_env
> -		cd "$1" && {
> -			git describe "$2" 2>/dev/null ||
> -			git describe --tags "$2" 2>/dev/null ||
> -			git describe --contains "$2" 2>/dev/null ||
> -			git describe --all --always "$2"
> -		}
> -	) )
> -	test -z "$revname" || revname=" ($revname)"
> -}
>  #
>  # Show commit summary for submodules in index or working tree
>  #
> @@ -1042,14 +1030,14 @@ cmd_status()
>  		fi
>  		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
>  		then
> -			set_name_rev "$sm_path" "$sha1"
> +			revname=$(git submodule--helper print-name-rev "$sm_path" "$sha1")
>  			say " $sha1 $displaypath$revname"
>  		else
>  			if test -z "$cached"
>  			then
>  				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
>  			fi
> -			set_name_rev "$sm_path" "$sha1"
> +			revname=$(git submodule--helper print-name-rev "$sm_path" "$sha1")
>  			say "+$sha1 $displaypath$revname"
>  		fi
>  
> -- 
> 2.13.0
> 

^ permalink raw reply	[relevance 14%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-21 16:20                 ` Heiko Voigt
@ 2017-08-21 16:55                   ` Stefan Beller
  2017-08-21 17:20                     ` Lars Schneider
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-21 16:55 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Junio C Hamano, Brandon Williams, git, Lars Schneider

On Mon, Aug 21, 2017 at 9:20 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:

>> So I am a bit curious to learn which part of this change you dislike
>> and why.
>
> I am also curious. Isn't this the same strategy we are using in other
> places?
>

I dislike it because the UX feels crude.  When reading the documentation,
it seems to me as if submodule.<name> can be one of the following

    (none, checkout, rebase, merge, !<custom-command>)

This is perfect for "submodule-update", whose primary goal is
to update submodules *somehow*. However other commands

    git rebase --recurse
    git merge --recurse
    git checkout --recurse

have a different primary mode of operation (note how their name
is one of the modes from the set above), so it may get confusing
for a user.

'none'  and '!<custom-command>' seem like they would be okay
for any of the commands above but then:

    git config submodule.<name>.update "!..."
    git reset --hard --recurse
    git status
    # submodule is reported, because "!..." did not 'reset'.

Anyway. That dislike is just a minor gut feeling about the UX/UI
being horrible. I wrote the patch to keep the conversation going,
and if it fixes Lars problem, let's take it for now.

Thanks,
Stefan

^ permalink raw reply	[relevance 25%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-21 16:55                   ` Stefan Beller
@ 2017-08-21 17:20                     ` Lars Schneider
  2017-08-21 17:48                       ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Lars Schneider @ 2017-08-21 17:20 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Heiko Voigt, Junio C Hamano, Brandon Williams, git


> On 21 Aug 2017, at 18:55, Stefan Beller <sbeller@google.com> wrote:
> 
> On Mon, Aug 21, 2017 at 9:20 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> 
>>> So I am a bit curious to learn which part of this change you dislike
>>> and why.
>> 
>> I am also curious. Isn't this the same strategy we are using in other
>> places?
>> 
> 
> I dislike it because the UX feels crude.  When reading the documentation,
> it seems to me as if submodule.<name> can be one of the following
> 
>    (none, checkout, rebase, merge, !<custom-command>)
> 
> This is perfect for "submodule-update", whose primary goal is
> to update submodules *somehow*. However other commands
> 
>    git rebase --recurse
>    git merge --recurse
>    git checkout --recurse
> 
> have a different primary mode of operation (note how their name
> is one of the modes from the set above), so it may get confusing
> for a user.
> 
> 'none'  and '!<custom-command>' seem like they would be okay
> for any of the commands above but then:
> 
>    git config submodule.<name>.update "!..."
>    git reset --hard --recurse
>    git status
>    # submodule is reported, because "!..." did not 'reset'.
> 
> Anyway. That dislike is just a minor gut feeling about the UX/UI
> being horrible. I wrote the patch to keep the conversation going,
> and if it fixes Lars problem, let's take it for now.

Well, I need just a way to disable certain Submodules completely.
If you show me how "git config --local submodule.sub.active false"
works then I don't need this patch.

I tried to make it work here:
https://public-inbox.org/git/89AB8AA3-8E19-46BA-B169-D1EA4CF4ABE7@gmail.com/

Thanks,
Lars

^ permalink raw reply	[relevance 15%]

* Re: [GSoC][PATCH 3/4] submodule: port set_name_rev() from shell to C
  2017-08-21 16:47   ` Heiko Voigt
@ 2017-08-21 17:24     ` Prathamesh Chavan
  0 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-21 17:24 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: git, Stefan Beller, Christian Couder, Junio C Hamano

On Mon, Aug 21, 2017 at 10:17 PM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> On Mon, Aug 21, 2017 at 09:45:14PM +0530, Prathamesh Chavan wrote:
>> Function set_name_rev() is ported from git-submodule to the
>> submodule--helper builtin. The function get_name_rev() generates the
>> value of the revision name as required, and the function
>> print_name_rev() handles the formating and printing of the obtained
>> revision name.
>>
>> Mentored-by: Christian Couder <christian.couder@gmail.com>
>> Mentored-by: Stefan Beller <sbeller@google.com>
>> Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
>> ---
>>  builtin/submodule--helper.c | 63 +++++++++++++++++++++++++++++++++++++++++++++
>>  git-submodule.sh            | 16 ++----------
>>  2 files changed, 65 insertions(+), 14 deletions(-)
>>
>> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
>> index 7803457ba..a4bff3f38 100644
>> --- a/builtin/submodule--helper.c
>> +++ b/builtin/submodule--helper.c
>
> [...]
>
>> @@ -1242,6 +1304,7 @@ static struct cmd_struct commands[] = {
>>       {"relative-path", resolve_relative_path, 0},
>>       {"resolve-relative-url", resolve_relative_url, 0},
>>       {"resolve-relative-url-test", resolve_relative_url_test, 0},
>> +     {"print-name-rev", print_name_rev, 0},
>
> I see the function in git-submodule.sh was named kind of reverse. How
> about we do it more naturally here and call this 'rev-name' instead?
> That makes is more clear to me and is also similar to the used variable
> name 'revname'.

The functions 'print_name_rev()' and 'get_name_rev()' are the ported
C functions of the function 'set_name_rev()'
Their names were assigned so due to the existing function's name,
and hence to faithfully port the functions.

But, thanks for the above suggestion, and also for reviewing
the patch. I will update the names as print_rev_name() and
get_rev_name() respectively.
>
> I would also prefix it differently like 'get' or 'calculate' instead of
> 'print' since it tries to find a name and is not just a simple lookup.

The former function from the shell script, 'set_name_rev()' is split
into two functions, namely: 'print_name_rev()' and 'get_name_rev()'
The function print_name_rev() is just the front_end of the function,
and exists to printf the return value of the get_name_rev() function
is the required format.
Calculation of the value is actually done by the function
get_name_rev().
Hence, I named the functions the way they are.

Thanks,
Prathamesh Chavan

^ permalink raw reply	[relevance 7%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-21 17:20                     ` Lars Schneider
@ 2017-08-21 17:48                       ` Stefan Beller
  2017-08-21 18:21                         ` Brandon Williams
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-21 17:48 UTC (permalink / raw)
  To: Lars Schneider; +Cc: Heiko Voigt, Junio C Hamano, Brandon Williams, git

On Mon, Aug 21, 2017 at 10:20 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
>> On 21 Aug 2017, at 18:55, Stefan Beller <sbeller@google.com> wrote:
>>
>> On Mon, Aug 21, 2017 at 9:20 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>>
>>>> So I am a bit curious to learn which part of this change you dislike
>>>> and why.
>>>
>>> I am also curious. Isn't this the same strategy we are using in other
>>> places?
>>>
>>
>> I dislike it because the UX feels crude.  When reading the documentation,
>> it seems to me as if submodule.<name> can be one of the following
>>
>>    (none, checkout, rebase, merge, !<custom-command>)
>>
>> This is perfect for "submodule-update", whose primary goal is
>> to update submodules *somehow*. However other commands
>>
>>    git rebase --recurse
>>    git merge --recurse
>>    git checkout --recurse
>>
>> have a different primary mode of operation (note how their name
>> is one of the modes from the set above), so it may get confusing
>> for a user.
>>
>> 'none'  and '!<custom-command>' seem like they would be okay
>> for any of the commands above but then:
>>
>>    git config submodule.<name>.update "!..."
>>    git reset --hard --recurse
>>    git status
>>    # submodule is reported, because "!..." did not 'reset'.
>>
>> Anyway. That dislike is just a minor gut feeling about the UX/UI
>> being horrible. I wrote the patch to keep the conversation going,
>> and if it fixes Lars problem, let's take it for now.
>
> Well, I need just a way to disable certain Submodules completely.
> If you show me how "git config --local submodule.sub.active false"
> works then I don't need this patch.
>
> I tried to make it work here:
> https://public-inbox.org/git/89AB8AA3-8E19-46BA-B169-D1EA4CF4ABE7@gmail.com/

(A) you need to set expect there as well, to have sub{2,4,5} be expected
there as well.

(B) That may hint at another (UX) bug.

The test case there uses "git submodule update --init".
The init flag will set all submodules to active.

Maybe you want

    git config submodule.active ":(exclude)sub0"
    git config --add submodule.active ":(exclude)sub2"
    git config --add submodule.active "."
    # Read: anything except sub0 and sub2 are interesting

    git submodule update
    # no init flag, needed even for new submodules IIUC

^ permalink raw reply	[relevance 26%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-21 17:48                       ` Stefan Beller
@ 2017-08-21 18:21                         ` Brandon Williams
  2017-08-22 14:50                           ` Lars Schneider
  0 siblings, 1 reply; 200+ results
From: Brandon Williams @ 2017-08-21 18:21 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Lars Schneider, Heiko Voigt, Junio C Hamano, git

On 08/21, Stefan Beller wrote:
> On Mon, Aug 21, 2017 at 10:20 AM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
> >
> >> On 21 Aug 2017, at 18:55, Stefan Beller <sbeller@google.com> wrote:
> >>
> >> On Mon, Aug 21, 2017 at 9:20 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> >>
> >>>> So I am a bit curious to learn which part of this change you dislike
> >>>> and why.
> >>>
> >>> I am also curious. Isn't this the same strategy we are using in other
> >>> places?
> >>>
> >>
> >> I dislike it because the UX feels crude.  When reading the documentation,
> >> it seems to me as if submodule.<name> can be one of the following
> >>
> >>    (none, checkout, rebase, merge, !<custom-command>)
> >>
> >> This is perfect for "submodule-update", whose primary goal is
> >> to update submodules *somehow*. However other commands
> >>
> >>    git rebase --recurse
> >>    git merge --recurse
> >>    git checkout --recurse
> >>
> >> have a different primary mode of operation (note how their name
> >> is one of the modes from the set above), so it may get confusing
> >> for a user.
> >>
> >> 'none'  and '!<custom-command>' seem like they would be okay
> >> for any of the commands above but then:
> >>
> >>    git config submodule.<name>.update "!..."
> >>    git reset --hard --recurse
> >>    git status
> >>    # submodule is reported, because "!..." did not 'reset'.
> >>
> >> Anyway. That dislike is just a minor gut feeling about the UX/UI
> >> being horrible. I wrote the patch to keep the conversation going,
> >> and if it fixes Lars problem, let's take it for now.
> >
> > Well, I need just a way to disable certain Submodules completely.
> > If you show me how "git config --local submodule.sub.active false"
> > works then I don't need this patch.

Yeah if you want to completely disable a submodule (as in not even check
it out) then setting .active to false would do that.  But as stefan
pointed out and IIRC 'submodule update --init' with no pathspec sets all
submodules to be active.  Perhaps it should only init submodules who
don't already have an explicit active flag set.

> >
> > I tried to make it work here:
> > https://public-inbox.org/git/89AB8AA3-8E19-46BA-B169-D1EA4CF4ABE7@gmail.com/
> 
> (A) you need to set expect there as well, to have sub{2,4,5} be expected
> there as well.
> 
> (B) That may hint at another (UX) bug.
> 
> The test case there uses "git submodule update --init".
> The init flag will set all submodules to active.
> 
> Maybe you want
> 
>     git config submodule.active ":(exclude)sub0"
>     git config --add submodule.active ":(exclude)sub2"
>     git config --add submodule.active "."
>     # Read: anything except sub0 and sub2 are interesting
> 
>     git submodule update
>     # no init flag, needed even for new submodules IIUC

-- 
Brandon Williams

^ permalink raw reply	[relevance 25%]

* Re: [RFC PATCH 0/6] Hash Abstraction
  2017-08-21  0:00 [RFC PATCH 0/6] Hash Abstraction brian m. carlson
  2017-08-21  0:00 ` [RFC PATCH 5/6] Integrate hash algorithm support with repo setup brian m. carlson
@ 2017-08-21 20:48 ` Stefan Beller
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-21 20:48 UTC (permalink / raw)
  To: brian m. carlson; +Cc: git

On Sun, Aug 20, 2017 at 5:00 PM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> = Overview
>
> This is an RFC series proposing a basic abstraction for hash functions.
>
> As we get closer to converting the remainder of the codebase to use
> struct object_id, we should think about the design we want our hash
> function abstraction to take.  This series is a proposal for one idea to
> start discussion.  Input on any aspect of this proposal is welcome.
>
> This series exposes a struct git_hash_algo that contains basic
> information about a given hash algorithm that distinguishes it from
> other algorithms: name, lengths, implementing functions, and empty tree
> and blob constants.  It also exposes an array of hash algorithms, and a
> constant for indexing them.
>
> The series also demonstrates a simple conversion using the abstraction
> over empty blob and tree values.
>
> In order to avoid conflicting with the struct repository work and with
> the goal of avoiding global variables as much as possible, I've pushed
> the hash algorithm into struct repository and exposed it via a #define.
> This necessitiates pulling repository.h into cache.h, which I don't
> think is fatal.  Doing that, in turn, necessitated some work on the
> Subversion code to avoid conflicts.
>
> It should be fine for Junio to pick up the first two patches from this
> series, as they're relatively independent and valuable without the rest
> of the series.  The rest should not be applied immediately, although
> they do pass the testsuite.
>
> I proposed this series now as it will inform the way we go about
> converting other parts of the codebase, especially some of the pack
> algorithms.  Because we share some hash computation code between pack
> checksums and object hashing, we need to decide whether to expose pack
> checksums as struct object_id, even though they are technically not
> object IDs.  Furthermore, if we end up needing to stuff an algorithm
> value into struct object_id, we'll no longer be able to directly
> reference object IDs in a pack without a copy.

Note that the checksum hash could be different from actual object ids,
so it may be easiest to introduce a "new pack format", which is just
s/sha1/<new hash>/g for any object ids, though I guess the work needed
is just the same for converting checksums to the new hash, too.


> This series is available from the usual places as branch hash-struct,
> based against master.
>
> = Naming
>
> The length names are similar to the current constant names
> intentionally.  I've used the "hash_algo" name for both the integer
> constant and the pointer to struct, although we could change the latter
> to "hash_impl" or such as people like.
>
> I chose to name the define "current_hash" and expose no other defines.
> The name is relatively short since we're going to be typing it a lot.
> However, if people like, we can capitalize it or expose other defines
> (say, a GIT_HASH_RAWSZ or GIT_HASH_HEXSZ) instead of or in addition to
> current_hash, which would make this name less interesting.
>
> Feel free to propose alternatives to the naming of anything in this
> series.
>
> = Open Issues
>
> I originally decided to convert hex.c as an example, but quickly found
> out that this caused segfaults.  As part of setup, we call
> is_git_directory, which calls validate_headref, which ends up calling
> get_sha1_hex.  Obviously, we don't have a repository, so the hash
> algorithm isn't set up yet.  This is an area we'll need to consider
> making hash function agnostic, and we may also need to consider
> inserting a hash constant integer into struct object_id if we're going
> to do that.  Alternatively, we could just paper over this issue as a
> special case.

We could introduce new files .sha1 or .sha256 in the config dir, whose
existence can be checked prior to validating the HEAD.

Maybe the HEAD validation can be non-strict and could check
if we'd have at least 'n' hex chars and at most 'm' hex chars. 'n' and 'm'
depend on the hash functions supported of that version of git. (Maybe
we'd validate later again to have the correct hash function length,
or we could also allow abbreviated hash names for HEAD)

> Clearly we're going to want to expose some sort of lookup functionality
> for hash algorithms.  We'll need to expose lookup by name (for the
> .git/config file and any command-line options), but we may want other
> functions as well.  What functions should those be?  Should we expose
> the structure or the constant for those lookup functions?  If the
> structure, we'll probably need to expose the constant in the structure
> as well for easy use.
>
> Should we avoid exposing the array of structure altogether and wrap this
> in a function?
>
> We could expose a union of hash context structures and take that as the
> pointer type for the API calls.  That would probably obviate the need
> for ctxsz.
>
> We could expose hex versions of the blob constants if desired.  This
> might make converting the remaining pieces of code that use them easier.
>
> There are probably dozens of other things I haven't thought of yet as
> well.
>
> brian m. carlson (6):
>   vcs-svn: remove unused prototypes
>   vcs-svn: rename repo functions to "svn_repo"
>   setup: expose enumerated repo info
>   Add structure representing hash algorithm
>   Integrate hash algorithm support with repo setup
>   Switch empty tree and blob lookups to use hash abstraction
>
>  builtin/am.c        |  2 +-
>  builtin/checkout.c  |  2 +-
>  builtin/diff.c      |  2 +-
>  builtin/pull.c      |  2 +-
>  cache.h             | 48 ++++++++++++++++++++++++++++++++++++++++++++----
>  diff-lib.c          |  2 +-
>  merge-recursive.c   |  2 +-
>  notes-merge.c       |  2 +-
>  repository.c        |  7 +++++++
>  repository.h        |  5 +++++
>  sequencer.c         |  6 +++---
>  setup.c             | 48 +++++++++++++++++++++++++++---------------------
>  sha1_file.c         | 29 +++++++++++++++++++++++++++++
>  submodule.c         |  2 +-
>  vcs-svn/repo_tree.c |  6 +++---
>  vcs-svn/repo_tree.h | 13 +++----------
>  vcs-svn/svndump.c   |  8 ++++----
>  17 files changed, 133 insertions(+), 53 deletions(-)
>

^ permalink raw reply	[relevance 8%]

* [GSoC] Update: Week 14
@ 2017-08-21 20:48 Prathamesh Chavan
  0 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-21 20:48 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller, Christian Couder

SUMMARY OF MY PROJECT:

Git submodule subcommands are currently implemented by using shell script
'git-submodule.sh'. There are several reasons why we'll prefer not to
use the shell script. My project intends to convert the subcommands into
C code, thus making them builtins. This will increase Git's portability
and hence the efficiency of working with the git-submodule commands.
Link to the complete proposal: [1]

Mentors:
Stefan Beller <sbeller@google.com>
Christian Couder <christian.couder@gmail.com>

UPDATES:

Following are the updates about my ongoing project:

* add: Although much time was spent by me on trying to debug the
  patch, the attempts made were unsuccessful or were
  rather partially successful (since while fixing one issue,
  they invoked a different one). After getting   suggestions and
  help from mentors for the remaining tests, now there are only
  two tests which the patch is currently failing at.
  The current status of the patch can be viewed at [2].

* update: porting this subcommand is already underway, and
  porting of the functions is_tip_reachable() and
  fetch_in_submodule() as completed.

* patches: a smaller patch series is floated on the mailing list
  for the maintainer's review.

The small patch series which recently was floated is pushed on Github
and is available at [3].

PLAN FOR WEEK-15 (22 August 2017 to 28 August 2017):

* add: One of the main aims of the next week is to resolve the
  the two tests which the current patch fails at.

* patches: will be working on getting the smaller series merged, and
  once its done, will prepare and float the next patch for the
  same purpose. Till then, I aim to stay more active on the mailing
  list as well for responding to the reviews.

Since I am a bit busier the following week (due to college academics),
I will resume porting submodule subcommand 'update' in the following week.

[1]: https://docs.google.com/document/d/1krxVLooWl--75Pot3dazhfygR3wCUUWZWzTXtK1L-xU/
[2]: https://github.com/pratham-pc/git/commits/sub-add
[3]: https://github.com/pratham-pc/git/commits/week-14-1

^ permalink raw reply	[relevance 16%]

* Re: [RFC PATCH 5/6] Integrate hash algorithm support with repo setup
  2017-08-21  0:00 ` [RFC PATCH 5/6] Integrate hash algorithm support with repo setup brian m. carlson
@ 2017-08-21 21:16   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-21 21:16 UTC (permalink / raw)
  To: brian m. carlson, Brandon Williams; +Cc: git

On Sun, Aug 20, 2017 at 5:00 PM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> In future versions of Git, we plan to support an additional hash
> algorithm.  Integrate the enumeration of hash algorithms with repository
> setup, and store a pointer to the enumerated data in struct repository.
> Of course, we currently only support SHA-1, so hard-code this value in
> read_repository_format.  In the future, we'll enumerate this value from
> the configuration.
>
> Add a constant, current_hash, which points to the hash_algo structure
> pointer in the repository global.  Include repository.h in cache.h since
> we now need to have access to these struct and variable definitions.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>

The new code seems to fit in nicely with the repository struct.

In another series I also included repository.h in cache.h
(not yet sent out), so I think that is a good idea.

> ---
>  cache.h      | 4 ++++
>  repository.c | 7 +++++++
>  repository.h | 5 +++++
>  setup.c      | 2 ++
>  4 files changed, 18 insertions(+)
>
> diff --git a/cache.h b/cache.h
> index 375a7fb15e..d759824803 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -13,6 +13,7 @@
>  #include "hash.h"
>  #include "path.h"
>  #include "sha1-array.h"
> +#include "repository.h"
>
>  #ifndef platform_SHA_CTX
>  /*
> @@ -112,6 +113,8 @@ struct git_hash_algo {
>  };
>  extern const struct git_hash_algo hash_algos[1];
>
> +#define current_hash the_repository->hash_algo
> +
>  #if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)
>  #define DTYPE(de)      ((de)->d_type)
>  #else
> @@ -894,6 +897,7 @@ struct repository_format {
>         int version;
>         int precious_objects;
>         int is_bare;
> +       int hash_algo;
>         char *work_tree;
>         struct string_list unknown_extensions;
>  };
> diff --git a/repository.c b/repository.c
> index 1617467568..37764f627a 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -62,6 +62,11 @@ void repo_set_gitdir(struct repository *repo, const char *path)
>         repo_setup_env(repo);
>  }
>
> +void repo_set_hash_algo(struct repository *repo, int hash_algo)
> +{
> +       repo->hash_algo = &hash_algos[hash_algo];
> +}
> +
>  /*
>   * Attempt to resolve and set the provided 'gitdir' for repository 'repo'.
>   * Return 0 upon success and a non-zero value upon failure.
> @@ -134,6 +139,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
>         if (read_and_verify_repository_format(&format, repo->commondir))
>                 goto error;
>
> +       repo->hash_algo = &hash_algos[format.hash_algo];
> +
>         if (worktree)
>                 repo_set_worktree(repo, worktree);
>
> diff --git a/repository.h b/repository.h
> index 417787f3ef..f171172150 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -4,6 +4,7 @@
>  struct config_set;
>  struct index_state;
>  struct submodule_cache;
> +struct git_hash_algo;
>
>  struct repository {
>         /* Environment */
> @@ -67,6 +68,9 @@ struct repository {
>          */
>         struct index_state *index;
>
> +       /* Repository's current hash algorithm. */
> +       const struct git_hash_algo *hash_algo;
> +
>         /* Configurations */
>         /*
>          * Bit used during initialization to indicate if repository state (like
> @@ -86,6 +90,7 @@ extern struct repository *the_repository;
>
>  extern void repo_set_gitdir(struct repository *repo, const char *path);
>  extern void repo_set_worktree(struct repository *repo, const char *path);
> +extern void repo_set_hash_algo(struct repository *repo, int algo);
>  extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);
>  extern int repo_submodule_init(struct repository *submodule,
>                                struct repository *superproject,
> diff --git a/setup.c b/setup.c
> index 115e70a4e8..289e24811c 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -491,6 +491,7 @@ int read_repository_format(struct repository_format *format, const char *path)
>         memset(format, 0, sizeof(*format));
>         format->version = -1;
>         format->is_bare = -1;
> +       format->hash_algo = GIT_HASH_SHA1;
>         string_list_init(&format->unknown_extensions, 1);
>         git_config_from_file(check_repo_format, path, format);
>         return format->version;
> @@ -1125,6 +1126,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
>                         repo_set_gitdir(the_repository, gitdir);
>                         setup_git_env();
>                 }
> +               repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
>         }
>
>         strbuf_release(&dir);

^ permalink raw reply	[relevance 8%]

* Re: Submodule regression in 2.14?
  2017-08-21 16:46                 ` Stefan Beller
@ 2017-08-21 22:45                   ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-08-21 22:45 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Lars Schneider, Brandon Williams, git\

Stefan Beller <sbeller@google.com> writes:

> On Fri, Aug 18, 2017 at 11:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
>> As long as we are talking about idealized future world (well, at
>> least an idea of somebody's "ideal", not necessarily shared by
>> everybody), I wonder if there is even any need to have commits in
>> submodules in such a world.  To realize such a "monorepo" world, you
>> might be better off allowing a gitlink in the superproject to
>> directly point at a tree object in a submodule repository (making
>> them physically a single repository is an optional implementation
>> detail I choose to ignore in this discussion).
>
> Then the sharing between superprojects (e.g. send an Android's linux
> patch upstream or to another distro that also uses a superproject),
> becomes cumbersome as the commit messages are missing and
> potentially not specific to that subtree.

Indeed.  That is a problem "git commit --recurse-submodules" has.
Socratic method seem to have worked well to convince you that it is
not necessarily a good idea to make submodules "just like a tree".


^ permalink raw reply	[relevance 24%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-21 18:21                         ` Brandon Williams
@ 2017-08-22 14:50                           ` Lars Schneider
  2017-08-22 17:51                             ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Lars Schneider @ 2017-08-22 14:50 UTC (permalink / raw)
  To: Brandon Williams; +Cc: Stefan Beller, Heiko Voigt, Junio C Hamano, git


> On 21 Aug 2017, at 20:21, Brandon Williams <bmwill@google.com> wrote:
> 
> On 08/21, Stefan Beller wrote:
>> On Mon, Aug 21, 2017 at 10:20 AM, Lars Schneider
>> <larsxschneider@gmail.com> wrote:
>>> 
>>>> On 21 Aug 2017, at 18:55, Stefan Beller <sbeller@google.com> wrote:
>>>> 
>>>> On Mon, Aug 21, 2017 at 9:20 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>>>> 
>>>>>> So I am a bit curious to learn which part of this change you dislike
>>>>>> and why.
>>>>> 
>>>>> I am also curious. Isn't this the same strategy we are using in other
>>>>> places?
>>>>> 
>>>> 
>>>> I dislike it because the UX feels crude.  When reading the documentation,
>>>> it seems to me as if submodule.<name> can be one of the following
>>>> 
>>>>   (none, checkout, rebase, merge, !<custom-command>)
>>>> 
>>>> This is perfect for "submodule-update", whose primary goal is
>>>> to update submodules *somehow*. However other commands
>>>> 
>>>>   git rebase --recurse
>>>>   git merge --recurse
>>>>   git checkout --recurse
>>>> 
>>>> have a different primary mode of operation (note how their name
>>>> is one of the modes from the set above), so it may get confusing
>>>> for a user.
>>>> 
>>>> 'none'  and '!<custom-command>' seem like they would be okay
>>>> for any of the commands above but then:
>>>> 
>>>>   git config submodule.<name>.update "!..."
>>>>   git reset --hard --recurse
>>>>   git status
>>>>   # submodule is reported, because "!..." did not 'reset'.
>>>> 
>>>> Anyway. That dislike is just a minor gut feeling about the UX/UI
>>>> being horrible. I wrote the patch to keep the conversation going,
>>>> and if it fixes Lars problem, let's take it for now.
>>> 
>>> Well, I need just a way to disable certain Submodules completely.
>>> If you show me how "git config --local submodule.sub.active false"
>>> works then I don't need this patch.
> 
> Yeah if you want to completely disable a submodule (as in not even check
> it out) then setting .active to false would do that.  But as stefan
> pointed out and IIRC 'submodule update --init' with no pathspec sets all
> submodules to be active.  Perhaps it should only init submodules who
> don't already have an explicit active flag set.

OK. I change my scripts to use ".active" and it seems to work nicely.

I noticed one oddity, though:

If I clone a repo using `git clone --recursive <url>` then the local
Git config of the repo gets the following entry:

[submodule]
	active = .

Is this intentional? Something in the git/git test harness seems to prevent
that. I was not able to write a test to replicate the issue.

Any idea?

Thanks,
Lars

^ permalink raw reply	[relevance 14%]

* Re: Submodule regression in 2.14?
  2017-08-21 16:42                   ` Stefan Beller
@ 2017-08-22 15:33                     ` Heiko Voigt
  2017-08-22 18:10                       ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Heiko Voigt @ 2017-08-22 15:33 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, Lars Schneider, Brandon Williams, git

On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
> On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
> >> As long as we are talking about idealized future world (well, at
> >> least an idea of somebody's "ideal", not necessarily shared by
> >> everybody), I wonder if there is even any need to have commits in
> >> submodules in such a world.  To realize such a "monorepo" world, you
> >> might be better off allowing a gitlink in the superproject to
> >> directly point at a tree object in a submodule repository (making
> >> them physically a single repository is an optional implementation
> >> detail I choose to ignore in this discussion).
> >
> > IMO this is one step to far. One main use of submodules are shared
> > repositories that are used by many superprojects. The reason you want to
> > have commits in the submodule are so that you can push them
> > independently and all other users can pick up the changes. You could get
> > by by Using the superproject commits for the submodule once you push or
> > something but those do not necessarily make sense in the context of the
> > submodule.
> >
> > So I think it is important that there are commits in the submodule so
> > its history makes sense independently for others.
> >
> > Or how would you push out the history in the submodule in your idea?
> > Maybe I am missing something? What would be your use case with gitlinks
> > pointing to trees?
> 
> Well there are still commits, but in the superproject the UX feels more
> as if the submodules were special trees.

Ah ok then I misunderstood. So they only feel like trees.

> So if you want to interact with
> the submodule specifically, you could do things like
> 
>     git add /path/inside/sub
>     # works seamlessly from the superproject tree

Would that mean that we need to loosen/keep the requirement loose for a
name from .gitmodules? I am asking because of my series for on-demand
fetch of renamed submodules. For the full functionality I would require
a name.

Would that be in a scenario where the user would then e.g. push the
submodule into the superproject?

Ah wait I misunderstood again. You mean a file in an existing
submodule right? Not adding submodule from a repository a user moved
there?

>     git commit --submodule-commit-only
>     # When the flag is not give, you may get an editor
>     # asking for two commit messages, (sub+super)

Or maybe something like

    git commit --submodule path/to/submodule

so the user can specify which submodule she wants. I first wrote it
without the switch but but that collides with listing files which should
be added. IMO this shorter option is also more intuitive to understand
what it does (for this case).

>     git fetch --submodule
>     # When the flag is not given, we'd fetch superproject and
>     # on-demand

Yes like above we should add the path to the submodule right?

>     # You feel the superproject is in the way?
>     git worktree add --for-submodule <path/to/sub> ...
>     # The new submodule worktree puts the
>     # submodule only UX first. so it feels like its own
>     # repository, no need for specific flags.

I am not sure I understand this one. What would that do? Put a worktree
for submodule path/to/sub to ...?

Overall I like the direction of these ideas.

Cheers Heiko

^ permalink raw reply	[relevance 25%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-22 14:50                           ` Lars Schneider
@ 2017-08-22 17:51                             ` Stefan Beller
  2017-08-22 18:55                               ` Brandon Williams
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-22 17:51 UTC (permalink / raw)
  To: Lars Schneider; +Cc: Brandon Williams, Heiko Voigt, Junio C Hamano, git

On Tue, Aug 22, 2017 at 7:50 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
> OK. I change my scripts to use ".active" and it seems to work nicely.
>
> I noticed one oddity, though:
>
> If I clone a repo using `git clone --recursive <url>` then the local
> Git config of the repo gets the following entry:
>
> [submodule]
>         active = .

bb62e0a99f (clone: teach --recurse-submodules to optionally take a
pathspec, 2017-03-17) makes it clear that this is intentional for
--recurse-submodules, but doesn't exactly state that --recurse will
behave the same. The idea here is that at clone time you can already
give

    git clone --recurse=:(exclude)sub0 <url> <path>

and have your desired set of submodules there.
Combined with the changes in the attr system, b0db704652
(pathspec: allow querying for attributes, 2017-03-13)
you could make up things like this:

  $ cat .gitattributes
  /sub0 label0
  /sub1
  /sub2 label1 label2
  /sub3 label1
  /platform-specifc-subs/* label1 label2

and then get a clone via

    git clone --recurse=:(attr:label2). <url> <path>

for example. The labeling via the attributes allows for
complex patterns, but a relatively easy command line, that you
can share with coworkers.

> Is this intentional? Something in the git/git test harness seems to prevent
> that. I was not able to write a test to replicate the issue.
>
> Any idea?

I do not seem to understand the perceived bug?
The setting of submodule.active=<pathspec> seems intentional to me,
but how would you not reproduce it? Maybe Brandon has an idea.

Thanks,
Stefan

^ permalink raw reply	[relevance 25%]

* Re: Submodule regression in 2.14?
  2017-08-22 15:33                     ` Heiko Voigt
@ 2017-08-22 18:10                       ` Stefan Beller
  2017-08-25  9:10                         ` Heiko Voigt
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-22 18:10 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Junio C Hamano, Lars Schneider, Brandon Williams, git

On Tue, Aug 22, 2017 at 8:33 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
>> On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>> > On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
>> >> As long as we are talking about idealized future world (well, at
>> >> least an idea of somebody's "ideal", not necessarily shared by
>> >> everybody), I wonder if there is even any need to have commits in
>> >> submodules in such a world.  To realize such a "monorepo" world, you
>> >> might be better off allowing a gitlink in the superproject to
>> >> directly point at a tree object in a submodule repository (making
>> >> them physically a single repository is an optional implementation
>> >> detail I choose to ignore in this discussion).
>> >
>> > IMO this is one step to far. One main use of submodules are shared
>> > repositories that are used by many superprojects. The reason you want to
>> > have commits in the submodule are so that you can push them
>> > independently and all other users can pick up the changes. You could get
>> > by by Using the superproject commits for the submodule once you push or
>> > something but those do not necessarily make sense in the context of the
>> > submodule.
>> >
>> > So I think it is important that there are commits in the submodule so
>> > its history makes sense independently for others.
>> >
>> > Or how would you push out the history in the submodule in your idea?
>> > Maybe I am missing something? What would be your use case with gitlinks
>> > pointing to trees?
>>
>> Well there are still commits, but in the superproject the UX feels more
>> as if the submodules were special trees.
>
> Ah ok then I misunderstood. So they only feel like trees.
>
>> So if you want to interact with
>> the submodule specifically, you could do things like
>>
>>     git add /path/inside/sub
>>     # works seamlessly from the superproject tree
>
> Would that mean that we need to loosen/keep the requirement loose for a
> name from .gitmodules? I am asking because of my series for on-demand
> fetch of renamed submodules. For the full functionality I would require
> a name.
>
> Would that be in a scenario where the user would then e.g. push the
> submodule into the superproject?
>
> Ah wait I misunderstood again. You mean a file in an existing
> submodule right? Not adding submodule from a repository a user moved
> there?

Assuming the submodule is at  /path in this example, the effect of
that command could be achieved today via

    git -C /path add inside/sub

(i.e. for git-add we "just" detect that there is a submodule boundary
and run the git-add inside the submodule)

>
>>     git commit --submodule-commit-only
>>     # When the flag is not give, you may get an editor
>>     # asking for two commit messages, (sub+super)
>
> Or maybe something like
>
>     git commit --submodule path/to/submodule

Yes. In todays UX, you do

    git -C path/to/submodule commit

for the command that you proposed. For a plain
git-commit in the superproject, we could envision
this sequence of todays commands:

    git -C submodule commit
    git add submodule
    git commit

>
> so the user can specify which submodule she wants. I first wrote it
> without the switch but but that collides with listing files which should
> be added. IMO this shorter option is also more intuitive to understand
> what it does (for this case).
>
>>     git fetch --submodule
>>     # When the flag is not given, we'd fetch superproject and
>>     # on-demand
>
> Yes like above we should add the path to the submodule right?

yes.

>
>>     # You feel the superproject is in the way?
>>     git worktree add --for-submodule <path/to/sub> ...
>>     # The new submodule worktree puts the
>>     # submodule only UX first. so it feels like its own
>>     # repository, no need for specific flags.
>
> I am not sure I understand this one. What would that do? Put a worktree
> for submodule path/to/sub to ...?

Yes, and at "..." you would have the UX of the submodule being
its own repository, no interaction with the superproject.

>
> Overall I like the direction of these ideas.
>
> Cheers Heiko

^ permalink raw reply	[relevance 26%]

* Re: [PATCH] pull: respect submodule update configuration
  2017-08-22 17:51                             ` Stefan Beller
@ 2017-08-22 18:55                               ` Brandon Williams
  0 siblings, 0 replies; 200+ results
From: Brandon Williams @ 2017-08-22 18:55 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Lars Schneider, Heiko Voigt, Junio C Hamano, git

On 08/22, Stefan Beller wrote:
> On Tue, Aug 22, 2017 at 7:50 AM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
> >
> > OK. I change my scripts to use ".active" and it seems to work nicely.
> >
> > I noticed one oddity, though:
> >
> > If I clone a repo using `git clone --recursive <url>` then the local
> > Git config of the repo gets the following entry:
> >
> > [submodule]
> >         active = .
> 
> bb62e0a99f (clone: teach --recurse-submodules to optionally take a
> pathspec, 2017-03-17) makes it clear that this is intentional for
> --recurse-submodules, but doesn't exactly state that --recurse will
> behave the same. The idea here is that at clone time you can already
> give
> 
>     git clone --recurse=:(exclude)sub0 <url> <path>
> 
> and have your desired set of submodules there.
> Combined with the changes in the attr system, b0db704652
> (pathspec: allow querying for attributes, 2017-03-13)
> you could make up things like this:
> 
>   $ cat .gitattributes
>   /sub0 label0
>   /sub1
>   /sub2 label1 label2
>   /sub3 label1
>   /platform-specifc-subs/* label1 label2
> 
> and then get a clone via
> 
>     git clone --recurse=:(attr:label2). <url> <path>
> 
> for example. The labeling via the attributes allows for
> complex patterns, but a relatively easy command line, that you
> can share with coworkers.
> 
> > Is this intentional? Something in the git/git test harness seems to prevent
> > that. I was not able to write a test to replicate the issue.
> >
> > Any idea?
> 
> I do not seem to understand the perceived bug?
> The setting of submodule.active=<pathspec> seems intentional to me,
> but how would you not reproduce it? Maybe Brandon has an idea.
> 

When adding '.active' we wanted it to be as flexible as possible.  So
you can either use a pathspec with 'submodule.active' to catch multiple
submodules as being active or you can turn on/off individual submodules
with 'submodule.<name>.active' (this has precedent over the more general
'submodule.active' config).

The intent was if a user supplies --recurse-submodules (I believe i
removed the docs for --recursive in order to make the CLI more consistent
with other commands, so --recursive is just a synonym for
--recurse-submodules) then they clearly wanted all the submodules cloned
and checked out.  With the '.active' config the way to specify this
is to make 'submodule.active = .'.  In the old world every submodule
would need to have its URL copied into the config.  This way the config
is kept cleaner as it only has a single entry added.

As stefan mentioned you can specify a value for 'submodule.active' to
take as an arg to --recurse-submodules (the default being '.' or all
submodules) so you can do clever things like group submodules using
attributes, you can even repeat the flag to provided a more complex
pathspec.

Hopefully that answers your question :D

-- 
Brandon Williams

^ permalink raw reply	[relevance 26%]

* [GSoC][PATCH v2 1/4] submodule--helper: introduce get_submodule_displaypath()
  2017-08-23 18:15 ` [GSoC][PATCH v2 0/4] submodule: Incremental rewrite of git-submodules Prathamesh Chavan
@ 2017-08-23 18:15   ` Prathamesh Chavan
  2017-08-23 18:15   ` [GSoC][PATCH v2 2/4] submodule--helper: introduce for_each_submodule() Prathamesh Chavan
                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-23 18:15 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Introduce function get_submodule_displaypath() to replace the code
occurring in submodule_init() for generating displaypath of the
submodule with a call to it.

This new function will also be used in other parts of the system
in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 38 +++++++++++++++++++++++++-------------
 1 file changed, 25 insertions(+), 13 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 84562ec83..e666f84ba 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -220,6 +220,28 @@ static int resolve_relative_url_test(int argc, const char **argv, const char *pr
 	return 0;
 }
 
+static char *get_submodule_displaypath(const char *path, const char *prefix)
+{
+	const char *super_prefix = get_super_prefix();
+
+	if (prefix && super_prefix) {
+		BUG("cannot have prefix '%s' and superprefix '%s'",
+		    prefix, super_prefix);
+	} else if (prefix) {
+		struct strbuf sb = STRBUF_INIT;
+		char *displaypath = xstrdup(relative_path(path, prefix, &sb));
+		strbuf_release(&sb);
+		return displaypath;
+	} else if (super_prefix) {
+		int len = strlen(super_prefix);
+		const char *format = is_dir_sep(super_prefix[len - 1]) ? "%s%s" : "%s/%s";
+
+		return xstrfmt(format, super_prefix, path);
+	} else {
+		return xstrdup(path);
+	}
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -339,16 +361,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 
 	/* Only loads from .gitmodules, no overlay with .git/config */
 	gitmodules_config();
-
-	if (prefix && get_super_prefix())
-		die("BUG: cannot have prefix and superprefix");
-	else if (prefix)
-		displaypath = xstrdup(relative_path(path, prefix, &sb));
-	else if (get_super_prefix()) {
-		strbuf_addf(&sb, "%s%s", get_super_prefix(), path);
-		displaypath = strbuf_detach(&sb, NULL);
-	} else
-		displaypath = xstrdup(path);
+	displaypath = get_submodule_displaypath(path, prefix);
 
 	sub = submodule_from_path(&null_oid, path);
 
@@ -363,9 +376,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * Set active flag for the submodule being initialized
 	 */
 	if (!is_submodule_active(the_repository, path)) {
-		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
+		strbuf_reset(&sb);
 	}
 
 	/*
@@ -373,7 +386,6 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * To look up the url in .git/config, we must not fall back to
 	 * .gitmodules, so look it up directly.
 	 */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.url", sub->name);
 	if (git_config_get_string(sb.buf, &url)) {
 		if (!sub->url)
@@ -410,9 +422,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
 	}
+	strbuf_reset(&sb);
 
 	/* Copy "update" setting when it is not set yet */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.update", sub->name);
 	if (git_config_get_string(sb.buf, &upd) &&
 	    sub->update_strategy.type != SM_UPDATE_UNSPECIFIED) {
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v2 2/4] submodule--helper: introduce for_each_submodule()
  2017-08-23 18:15 ` [GSoC][PATCH v2 0/4] submodule: Incremental rewrite of git-submodules Prathamesh Chavan
  2017-08-23 18:15   ` [GSoC][PATCH v2 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
@ 2017-08-23 18:15   ` Prathamesh Chavan
  2017-08-23 19:13     ` Junio C Hamano
  2017-08-23 18:15   ` [GSoC][PATCH v2 3/4] submodule: port set_name_rev() " Prathamesh Chavan
  2017-08-23 18:15   ` [GSoC][PATCH v2 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  3 siblings, 1 reply; 200+ results
From: Prathamesh Chavan @ 2017-08-23 18:15 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Introduce function for_each_submodule() and replace a loop
in module_init() with a call to it.

The new function will also be used in other parts of the
system in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 39 +++++++++++++++++++++++++++++----------
 1 file changed, 29 insertions(+), 10 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index e666f84ba..847fba854 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -14,6 +14,9 @@
 #include "refs.h"
 #include "connect.h"
 
+typedef void (*submodule_list_func_t)(const struct cache_entry *list_item,
+				      void *cb_data);
+
 static char *get_default_remote(void)
 {
 	char *dest = NULL, *ret;
@@ -353,17 +356,30 @@ static int module_list(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
-static void init_submodule(const char *path, const char *prefix, int quiet)
+static void for_each_submodule(const struct module_list *list,
+			       submodule_list_func_t fn, void *cb_data)
+{
+	int i;
+	for (i = 0; i < list->nr; i++)
+		fn(list->entries[i], cb_data);
+}
+
+struct init_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+};
+#define INIT_CB_INIT { NULL, 0 }
+
+static void init_submodule(const struct cache_entry *list_item, void *cb_data)
 {
+	struct init_cb *info = cb_data;
 	const struct submodule *sub;
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	/* Only loads from .gitmodules, no overlay with .git/config */
-	gitmodules_config();
-	displaypath = get_submodule_displaypath(path, prefix);
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(&null_oid, list_item->name);
 
 	if (!sub)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
@@ -375,7 +391,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 *
 	 * Set active flag for the submodule being initialized
 	 */
-	if (!is_submodule_active(the_repository, path)) {
+	if (!is_submodule_active(the_repository, list_item->name)) {
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
 		strbuf_reset(&sb);
@@ -417,7 +433,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		if (git_config_set_gently(sb.buf, url))
 			die(_("Failed to register url for submodule path '%s'"),
 			    displaypath);
-		if (!quiet)
+		if (!info->quiet)
 			fprintf(stderr,
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
@@ -446,10 +462,10 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 
 static int module_init(int argc, const char **argv, const char *prefix)
 {
+	struct init_cb info = INIT_CB_INIT;
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
-	int i;
 
 	struct option module_init_options[] = {
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
@@ -474,8 +490,11 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
 
-	for (i = 0; i < list.nr; i++)
-		init_submodule(list.entries[i]->name, prefix, quiet);
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+
+	gitmodules_config();
+	for_each_submodule(&list, init_submodule, &info);
 
 	return 0;
 }
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v2 3/4] submodule: port set_name_rev() from shell to C
  2017-08-23 18:15 ` [GSoC][PATCH v2 0/4] submodule: Incremental rewrite of git-submodules Prathamesh Chavan
  2017-08-23 18:15   ` [GSoC][PATCH v2 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
  2017-08-23 18:15   ` [GSoC][PATCH v2 2/4] submodule--helper: introduce for_each_submodule() Prathamesh Chavan
@ 2017-08-23 18:15   ` " Prathamesh Chavan
  2017-08-23 18:15   ` [GSoC][PATCH v2 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-23 18:15 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Function set_name_rev() is ported from git-submodule to the
submodule--helper builtin. The function compute_rev_name() generates the
value of the revision name as required.
The function get_rev_name() calls compute_rev_name() and receives the
revision name, and later handles its formating and printing.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 63 +++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            | 16 ++----------
 2 files changed, 65 insertions(+), 14 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 847fba854..6ae93ce38 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -245,6 +245,68 @@ static char *get_submodule_displaypath(const char *path, const char *prefix)
 	}
 }
 
+static char *compute_rev_name(const char *sub_path, const char* object_id)
+{
+	struct strbuf sb = STRBUF_INIT;
+	const char ***d;
+
+	static const char *describe_bare[] = {
+		NULL
+	};
+
+	static const char *describe_tags[] = {
+		"--tags", NULL
+	};
+
+	static const char *describe_contains[] = {
+		"--contains", NULL
+	};
+
+	static const char *describe_all_always[] = {
+		"--all", "--always", NULL
+	};
+
+	static const char **describe_argv[] = {
+		describe_bare, describe_tags, describe_contains,
+		describe_all_always, NULL
+	};
+
+	for (d = describe_argv; *d; d++) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+		prepare_submodule_repo_env(&cp.env_array);
+		cp.dir = sub_path;
+		cp.git_cmd = 1;
+		cp.no_stderr = 1;
+
+		argv_array_push(&cp.args, "describe");
+		argv_array_pushv(&cp.args, *d);
+		argv_array_push(&cp.args, object_id);
+
+		if (!capture_command(&cp, &sb, 0) && sb.len) {
+			strbuf_strip_suffix(&sb, "\n");
+			return strbuf_detach(&sb, NULL);
+		}
+	}
+
+	strbuf_release(&sb);
+	return NULL;
+}
+
+static int get_rev_name(int argc, const char **argv, const char *prefix)
+{
+	char *revname;
+	if (argc != 3)
+		die("get-rev-name only accepts two arguments: <path> <sha1>");
+
+	revname = compute_rev_name(argv[1], argv[2]);
+	if (revname && revname[0])
+		printf(" (%s)", revname);
+	printf("\n");
+
+	free(revname);
+	return 0;
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -1243,6 +1305,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"get-rev-name", get_rev_name, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index e131760ee..91f043ec6 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -759,18 +759,6 @@ cmd_update()
 	}
 }
 
-set_name_rev () {
-	revname=$( (
-		sanitize_submodule_env
-		cd "$1" && {
-			git describe "$2" 2>/dev/null ||
-			git describe --tags "$2" 2>/dev/null ||
-			git describe --contains "$2" 2>/dev/null ||
-			git describe --all --always "$2"
-		}
-	) )
-	test -z "$revname" || revname=" ($revname)"
-}
 #
 # Show commit summary for submodules in index or working tree
 #
@@ -1042,14 +1030,14 @@ cmd_status()
 		fi
 		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
 		then
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
 			say " $sha1 $displaypath$revname"
 		else
 			if test -z "$cached"
 			then
 				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
 			fi
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
 			say "+$sha1 $displaypath$revname"
 		fi
 
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v2 4/4] submodule: port submodule subcommand 'status' from shell to C
  2017-08-23 18:15 ` [GSoC][PATCH v2 0/4] submodule: Incremental rewrite of git-submodules Prathamesh Chavan
                     ` (2 preceding siblings ...)
  2017-08-23 18:15   ` [GSoC][PATCH v2 3/4] submodule: port set_name_rev() " Prathamesh Chavan
@ 2017-08-23 18:15   ` " Prathamesh Chavan
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-23 18:15 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

This aims to make git-submodule 'status' a built-in. Hence, the function
cmd_status() is ported from shell to C. This is done by introducing
three functions: module_status(), submodule_status() and print_status().

The function module_status() acts as the front-end of the subcommand.
It parses subcommand's options and then calls the function
module_list_compute() for computing the list of submodules. Then
this functions calls for_each_submodule() looping through the
list obtained.

Then for_each_submodule() calls submodule_status() for each of the
submodule in its list. The function submodule_status() is responsible
for generating the status each submodule it is called for, and
then calls print_status().

Finally, the function print_status() handles the printing of submodule's
status.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 156 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  49 +-------------
 2 files changed, 157 insertions(+), 48 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 6ae93ce38..933073251 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -561,6 +561,161 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+struct status_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+	unsigned int recursive: 1;
+	unsigned int cached: 1;
+};
+#define STATUS_CB_INIT { NULL, 0, 0, 0 }
+
+static void print_status(struct status_cb *info, char state, const char *path,
+			 const struct object_id *oid, const char *displaypath)
+{
+	if (info->quiet)
+		return;
+
+	printf("%c%s %s", state, oid_to_hex(oid), displaypath);
+
+	if (state == ' ' || state == '+') {
+		struct argv_array get_rev_args = ARGV_ARRAY_INIT;
+
+		argv_array_pushl(&get_rev_args, "get-rev-name",
+				 path, oid_to_hex(oid), NULL);
+		get_rev_name(get_rev_args.argc, get_rev_args.argv,
+			     info->prefix);
+
+		argv_array_clear(&get_rev_args);
+	} else {
+		printf("\n");
+	}
+}
+
+static int handle_submodule_head_ref(const char *refname,
+				     const struct object_id *oid, int flags,
+				     void *cb_data)
+{
+	struct object_id *output = cb_data;
+	if (oid)
+		oidcpy(output, oid);
+
+	return 0;
+}
+
+static void status_submodule(const struct cache_entry *list_item, void *cb_data)
+{
+	struct status_cb *info = cb_data;
+	char *displaypath;
+	struct argv_array diff_files_args = ARGV_ARRAY_INIT;
+
+	if (!submodule_from_path(&null_oid, list_item->name))
+		die(_("no submodule mapping found in .gitmodules for path '%s'"),
+		      list_item->name);
+
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
+
+	if (ce_stage(list_item)) {
+		print_status(info, 'U', list_item->name,
+			     &null_oid, displaypath);
+		goto cleanup;
+	}
+
+	if (!is_submodule_active(the_repository, list_item->name)) {
+		print_status(info, '-', list_item->name, &list_item->oid,
+			     displaypath);
+		goto cleanup;
+	}
+
+	argv_array_pushl(&diff_files_args, "diff-files",
+			 "--ignore-submodules=dirty", "--quiet", "--",
+			 list_item->name, NULL);
+
+	if (!cmd_diff_files(diff_files_args.argc, diff_files_args.argv,
+			    info->prefix)) {
+		print_status(info, ' ', list_item->name, &list_item->oid,
+			     displaypath);
+	} else {
+		if (!info->cached) {
+			struct object_id oid;
+
+			if (head_ref_submodule(list_item->name,
+					       handle_submodule_head_ref, &oid))
+				die(_("could not resolve HEAD ref inside the"
+				      "submodule '%s'"), list_item->name);
+
+			print_status(info, '+', list_item->name, &oid,
+				     displaypath);
+		} else {
+			print_status(info, '+', list_item->name,
+				     &list_item->oid, displaypath);
+		}
+	}
+
+	if (info->recursive) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = list_item->name;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_pushl(&cpr.args, "--super-prefix", displaypath,
+				 "submodule--helper", "status", "--recursive",
+				 NULL);
+
+		if (info->cached)
+			argv_array_push(&cpr.args, "--cached");
+
+		if (info->quiet)
+			argv_array_push(&cpr.args, "--quiet");
+
+		if (run_command(&cpr))
+			die(_("failed to recurse into submodule '%s'"),
+			      list_item->name);
+	}
+
+cleanup:
+	argv_array_clear(&diff_files_args);
+	free(displaypath);
+}
+
+static int module_status(int argc, const char **argv, const char *prefix)
+{
+	struct status_cb info = STATUS_CB_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+	int quiet = 0;
+	int cached = 0;
+	int recursive = 0;
+
+	struct option module_status_options[] = {
+		OPT__QUIET(&quiet, N_("Suppress submodule status output")),
+		OPT_BOOL(0, "cached", &cached, N_("Use commit stored in the index instead of the one stored in the submodule HEAD")),
+		OPT_BOOL(0, "recursive", &recursive, N_("Recurse into nested submodules")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule status [--quiet] [--cached] [--recursive] [<path>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_status_options,
+			     git_submodule_helper_usage, 0);
+
+	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+	info.recursive = !!recursive;
+	info.cached = !!cached;
+
+	gitmodules_config();
+	for_each_submodule(&list, status_submodule, &info);
+
+	return 0;
+}
+
 static int module_name(int argc, const char **argv, const char *prefix)
 {
 	const struct submodule *sub;
@@ -1307,6 +1462,7 @@ static struct cmd_struct commands[] = {
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
 	{"get-rev-name", get_rev_name, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
+	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
diff --git a/git-submodule.sh b/git-submodule.sh
index 91f043ec6..51b057d82 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1005,54 +1005,7 @@ cmd_status()
 		shift
 	done
 
-	{
-		git submodule--helper list --prefix "$wt_prefix" "$@" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path") || exit
-		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-		if test "$stage" = U
-		then
-			say "U$sha1 $displaypath"
-			continue
-		fi
-		if ! git submodule--helper is-active "$sm_path" ||
-		{
-			! test -d "$sm_path"/.git &&
-			! test -f "$sm_path"/.git
-		}
-		then
-			say "-$sha1 $displaypath"
-			continue;
-		fi
-		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
-		then
-			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
-			say " $sha1 $displaypath$revname"
-		else
-			if test -z "$cached"
-			then
-				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
-			fi
-			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
-			say "+$sha1 $displaypath$revname"
-		fi
-
-		if test -n "$recursive"
-		then
-			(
-				prefix="$displaypath/"
-				sanitize_submodule_env
-				wt_prefix=
-				cd "$sm_path" &&
-				eval cmd_status
-			) ||
-			die "$(eval_gettext "Failed to recurse into submodule path '\$sm_path'")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper status ${GIT_QUIET:+--quiet} ${cached:+--cached} ${recursive:+--recursive} "$@"
 }
 #
 # Sync remote urls for submodules
-- 
2.13.0


^ permalink raw reply	[relevance 20%]

* Re: [PATCH v4 02/16] refs.c: use is_dir_sep() in resolve_gitlink_ref()
  2017-08-23 12:36 ` [PATCH v4 02/16] refs.c: use is_dir_sep() in resolve_gitlink_ref() Nguyễn Thái Ngọc Duy
@ 2017-08-23 19:14   ` Stefan Beller
  2017-09-06 11:08     ` Duy Nguyen
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-23 19:14 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy; +Cc: git, Junio C Hamano, Michael Haggerty

On Wed, Aug 23, 2017 at 5:36 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> The "submodule" argument in this function is a path, which can have
> either '/' or '\\' as a separator. Use is_dir_sep() to support both.
>
> Noticed-by: Johannes Sixt <j6t@kdbg.org>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>

Immediate questions that come to mind:
* Could this go in as an independent bug fix?
  -> If so do we have any test that fails or stops failing on Windows?
  -> If not, do we need one?
* Assuming this is not an independent bug fix:
  Why do we need this patch in this series here?
  (I thought this is about worktrees, not submodules.
  So does this fix a potential bug that will be exposed
  later in this series, but was harmless up to now?)

I'll read the next patches to see if I will be enlightened.

Thanks,
Stefan

> ---
>  refs.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/refs.c b/refs.c
> index 3d549a8970..dec899a57a 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -1507,7 +1507,7 @@ int resolve_gitlink_ref(const char *submodule, const char *refname,
>         struct ref_store *refs;
>         int flags;
>
> -       while (len && submodule[len - 1] == '/')
> +       while (len && is_dir_sep(submodule[len - 1]))
>                 len--;
>
>         if (!len)
> --
> 2.11.0.157.gd943d85
>

^ permalink raw reply	[relevance 15%]

* Re: [PATCH v4 04/16] revision.c: --indexed-objects add objects from all worktrees
  2017-08-23 12:36 ` [PATCH v4 04/16] revision.c: --indexed-objects add objects from all worktrees Nguyễn Thái Ngọc Duy
@ 2017-08-23 19:25   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-23 19:25 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy; +Cc: git, Junio C Hamano, Michael Haggerty

On Wed, Aug 23, 2017 at 5:36 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> This is the result of single_worktree flag never being set (no way to up
> until now). To get objects from current index only, set single_worktree.
>
> The other add_index_objects_to_pending's caller is mark_reachable_objects()
> (e.g. "git prune") which also mark objects from all indexes.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  revision.c       | 21 +++++++++++++++++++++
>  t/t5304-prune.sh |  9 +++++++++
>  2 files changed, 30 insertions(+)
>
> diff --git a/revision.c b/revision.c
> index 6c46ad6c8a..f35cb49af5 100644
> --- a/revision.c
> +++ b/revision.c
> @@ -19,6 +19,7 @@
>  #include "dir.h"
>  #include "cache-tree.h"
>  #include "bisect.h"
> +#include "worktree.h"
>
>  volatile show_early_output_fn_t show_early_output;
>
> @@ -1290,8 +1291,28 @@ static void do_add_index_objects_to_pending(struct rev_info *revs,
>
>  void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)
>  {
> +       struct worktree **worktrees, **p;
> +
>         read_cache();
>         do_add_index_objects_to_pending(revs, &the_index);
> +
> +       if (revs->single_worktree)
> +               return;
> +
> +       worktrees = get_worktrees(0);
> +       for (p = worktrees; *p; p++) {
> +               struct worktree *wt = *p;
> +               struct index_state istate = { NULL };
> +
> +               if (wt->is_current)
> +                       continue; /* current index already taken care of */
> +
> +               if (read_index_from(&istate,
> +                                   worktree_git_path(wt, "index")) > 0)
> +                       do_add_index_objects_to_pending(revs, &istate);
> +               discard_index(&istate);
> +       }
> +       free_worktrees(worktrees);
>  }

The commit (message and code) looks acceptable and easy to read.
I wonder though if another approach would be more feasible:

* factor out anything after the early return into a new function
  "add_wt_index_objects_to_pending", maybe?
* introduce a new function
    for_each_wt_except_current(function, callback_cookie)
  such that this code could use this predefined iterator.
  (I have the impression I have seen this code pattern before,
  hence it may be useful to have such a foreach function; we
  have foreach functions already for various ref things, string
  list items, reflogs, and soon submodule lists)

>
>  static int add_parents_only(struct rev_info *revs, const char *arg_, int flags,
> diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
> index 133b5842b1..cba45c7be9 100755
> --- a/t/t5304-prune.sh
> +++ b/t/t5304-prune.sh
> @@ -283,4 +283,13 @@ test_expect_success 'prune: handle alternate object database' '
>         git -C B prune
>  '
>
> +test_expect_success 'prune: handle index in multiple worktrees' '
> +       git worktree add second-worktree &&
> +       echo "new blob for second-worktree" >second-worktree/blob &&
> +       git -C second-worktree add blob &&
> +       git prune --expire=now &&
> +       git -C second-worktree show :blob >actual &&
> +       test_cmp second-worktree/blob actual
> +'
> +
>  test_done
> --
> 2.11.0.157.gd943d85
>

^ permalink raw reply	[relevance 15%]

* Re: [GSoC][PATCH v2 2/4] submodule--helper: introduce for_each_submodule()
  2017-08-23 19:13     ` Junio C Hamano
@ 2017-08-23 19:31       ` Stefan Beller
  2017-08-23 19:52         ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-23 19:31 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Prathamesh Chavan, Christian Couder, git

On Wed, Aug 23, 2017 at 12:13 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Prathamesh Chavan <pc44800@gmail.com> writes:
>
>> +typedef void (*submodule_list_func_t)(const struct cache_entry *list_item,
>> +                                   void *cb_data);
>> +
>>  static char *get_default_remote(void)
>>  {
>>       char *dest = NULL, *ret;
>> @@ -353,17 +356,30 @@ static int module_list(int argc, const char **argv, const char *prefix)
>>       return 0;
>>  }
>>
>> -static void init_submodule(const char *path, const char *prefix, int quiet)
>> +static void for_each_submodule(const struct module_list *list,
>> +                            submodule_list_func_t fn, void *cb_data)
>
> In the output from
>
>         $ git grep for_each \*.h
>
> we find that the convention is that an interator over a group of X
> is for_each_X,

... which this is...

> the callback function that is given to for_each_X is
> of type each_X_fn.

So you suggest s/submodule_list_func_t/each_submodule_fn/

> An interator over a subset of group of X that
> has trait Y, for_each_Y_X() iterates and calls back a function of
> type each_X_fn (e.g. for_each_tag_ref() still calls each_ref_fn).

This reads as a suggestion for for_each_listed_submodule
as the name.

> I do not offhand think of a reason why the above code need to
> deviate from that pattern.

^ permalink raw reply	[relevance 16%]

* Re: [PATCH v4 05/16] refs.c: refactor get_submodule_ref_store(), share common free block
  2017-08-23 12:36 ` [PATCH v4 05/16] refs.c: refactor get_submodule_ref_store(), share common free block Nguyễn Thái Ngọc Duy
@ 2017-08-23 19:34   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-23 19:34 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy; +Cc: git, Junio C Hamano, Michael Haggerty

On Wed, Aug 23, 2017 at 5:36 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  refs.c | 17 ++++++-----------
>  1 file changed, 6 insertions(+), 11 deletions(-)
>
> diff --git a/refs.c b/refs.c
> index dec899a57a..522c4ab74f 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -1636,7 +1636,6 @@ struct ref_store *get_submodule_ref_store(const char *submodule)
>  {
>         struct strbuf submodule_sb = STRBUF_INIT;
>         struct ref_store *refs;

Do we need to assign NULL here, to cover the early outs?

> -       int ret;
>
>         if (!submodule || !*submodule) {
>                 /*
> @@ -1648,19 +1647,14 @@ struct ref_store *get_submodule_ref_store(const char *submodule)
>
>         refs = lookup_ref_store_map(&submodule_ref_stores, submodule);

Answering the question from above, no we do not need to
assign NULL as we have an unconditional assignment
here and any early outs are after this.

Thanks,
Stefan

^ permalink raw reply	[relevance 8%]

* Re: [GSoC][PATCH v2 2/4] submodule--helper: introduce for_each_submodule()
  2017-08-23 19:31       ` Stefan Beller
@ 2017-08-23 19:52         ` Junio C Hamano
  2017-08-24 19:50           ` [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2017-08-23 19:52 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Prathamesh Chavan, Christian Couder, git\

Stefan Beller <sbeller@google.com> writes:

> On Wed, Aug 23, 2017 at 12:13 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Prathamesh Chavan <pc44800@gmail.com> writes:
>>
>>> +typedef void (*submodule_list_func_t)(const struct cache_entry *list_item,
>>> +                                   void *cb_data);
>>> +
>>>  static char *get_default_remote(void)
>>>  {
>>>       char *dest = NULL, *ret;
>>> @@ -353,17 +356,30 @@ static int module_list(int argc, const char **argv, const char *prefix)
>>>       return 0;
>>>  }
>>>
>>> -static void init_submodule(const char *path, const char *prefix, int quiet)
>>> +static void for_each_submodule(const struct module_list *list,
>>> +                            submodule_list_func_t fn, void *cb_data)
>>
>> In the output from
>>
>>         $ git grep for_each \*.h
>>
>> we find that the convention is that an interator over a group of X
>> is for_each_X,
>
> ... which this is...
>
>> the callback function that is given to for_each_X is
>> of type each_X_fn.
>
> So you suggest s/submodule_list_func_t/each_submodule_fn/

It's not _I_ suggest---the remainder of the codebase screams that
the above name is wrong ;-).  Didn't it for the mentors while they
were reading this code?

>> An interator over a subset of group of X that
>> has trait Y, for_each_Y_X() iterates and calls back a function of
>> type each_X_fn (e.g. for_each_tag_ref() still calls each_ref_fn).
>
> This reads as a suggestion for for_each_listed_submodule
> as the name.

If you need to have two ways to iterate over them, i.e. (1) over all
submodules and (2) over only the listed ones (whatever that means),
then yes, for_each_listed_submodule() would be a good name for the
latter which would be a complement to for_each_submodule() that is
the former.

>
>> I do not offhand think of a reason why the above code need to
>> deviate from that pattern.

^ permalink raw reply	[relevance 22%]

* Re: [PATCH v4 14/16] rev-list: expose and document --single-worktree
  2017-08-23 12:37 ` [PATCH v4 14/16] rev-list: expose and document --single-worktree Nguyễn Thái Ngọc Duy
@ 2017-08-23 20:45   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-23 20:45 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy; +Cc: git, Junio C Hamano, Michael Haggerty

On Wed, Aug 23, 2017 at 5:37 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  Documentation/rev-list-options.txt | 8 ++++++++
>  revision.c                         | 2 ++
>  2 files changed, 10 insertions(+)
>
> diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt
> index a6cf9eb380..7d860bfca1 100644
> --- a/Documentation/rev-list-options.txt
> +++ b/Documentation/rev-list-options.txt
> @@ -184,6 +184,14 @@ explicitly.
>         Pretend as if all objects mentioned by reflogs are listed on the
>         command line as `<commit>`.
>
> +--single-worktree::
> +       By default, all working trees will be examined by the
> +       following options when there are more than one (see
> +       linkgit:git-worktree[1]): `--all`, `--reflog` and
> +       `--indexed-objects`.
> +       This option forces them to examine the current working tree
> +       only.
> +
>  --ignore-missing::
>         Upon seeing an invalid object name in the input, pretend as if
>         the bad input was not given.
> diff --git a/revision.c b/revision.c
> index d100b3a3be..6eba4131b4 100644
> --- a/revision.c
> +++ b/revision.c
> @@ -2251,6 +2251,8 @@ static int handle_revision_pseudo_opt(const char *submodule,
>                         return error("invalid argument to --no-walk");
>         } else if (!strcmp(arg, "--do-walk")) {
>                 revs->no_walk = 0;
> +       } else if (!strcmp(arg, "--single-worktree")) {
> +               revs->single_worktree = 1;

This is in handle_revision_pseudo_opt, that has the note

/*
* NOTE!
*
* Commands like "git shortlog" will not accept the options below
* unless parse_revision_opt queues them (as opposed to erroring
* out).
*
* When implementing your new pseudo-option, remember to
* register it in the list at the top of handle_revision_opt.
*/

The registration needs to be done at around line 1700.

But come to think of it, is it really a pseudo opt?
Could it be a "real" (non pseudo) opt in handle_revision_opt?
The reasoning (either way) would be of interest in the
commit message, IMHO.

^ permalink raw reply	[relevance 8%]

* [GSoC][PATCH v3 1/4] submodule--helper: introduce get_submodule_displaypath()
  2017-08-24 19:50           ` [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
@ 2017-08-24 19:50             ` Prathamesh Chavan
  2017-08-24 19:50             ` [GSoC][PATCH v3 2/4] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
                               ` (3 subsequent siblings)
  4 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-24 19:50 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Introduce function get_submodule_displaypath() to replace the code
occurring in submodule_init() for generating displaypath of the
submodule with a call to it.

This new function will also be used in other parts of the system
in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 38 +++++++++++++++++++++++++-------------
 1 file changed, 25 insertions(+), 13 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 84562ec83..e666f84ba 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -220,6 +220,28 @@ static int resolve_relative_url_test(int argc, const char **argv, const char *pr
 	return 0;
 }
 
+static char *get_submodule_displaypath(const char *path, const char *prefix)
+{
+	const char *super_prefix = get_super_prefix();
+
+	if (prefix && super_prefix) {
+		BUG("cannot have prefix '%s' and superprefix '%s'",
+		    prefix, super_prefix);
+	} else if (prefix) {
+		struct strbuf sb = STRBUF_INIT;
+		char *displaypath = xstrdup(relative_path(path, prefix, &sb));
+		strbuf_release(&sb);
+		return displaypath;
+	} else if (super_prefix) {
+		int len = strlen(super_prefix);
+		const char *format = is_dir_sep(super_prefix[len - 1]) ? "%s%s" : "%s/%s";
+
+		return xstrfmt(format, super_prefix, path);
+	} else {
+		return xstrdup(path);
+	}
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -339,16 +361,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 
 	/* Only loads from .gitmodules, no overlay with .git/config */
 	gitmodules_config();
-
-	if (prefix && get_super_prefix())
-		die("BUG: cannot have prefix and superprefix");
-	else if (prefix)
-		displaypath = xstrdup(relative_path(path, prefix, &sb));
-	else if (get_super_prefix()) {
-		strbuf_addf(&sb, "%s%s", get_super_prefix(), path);
-		displaypath = strbuf_detach(&sb, NULL);
-	} else
-		displaypath = xstrdup(path);
+	displaypath = get_submodule_displaypath(path, prefix);
 
 	sub = submodule_from_path(&null_oid, path);
 
@@ -363,9 +376,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * Set active flag for the submodule being initialized
 	 */
 	if (!is_submodule_active(the_repository, path)) {
-		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
+		strbuf_reset(&sb);
 	}
 
 	/*
@@ -373,7 +386,6 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * To look up the url in .git/config, we must not fall back to
 	 * .gitmodules, so look it up directly.
 	 */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.url", sub->name);
 	if (git_config_get_string(sb.buf, &url)) {
 		if (!sub->url)
@@ -410,9 +422,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
 	}
+	strbuf_reset(&sb);
 
 	/* Copy "update" setting when it is not set yet */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.update", sub->name);
 	if (git_config_get_string(sb.buf, &upd) &&
 	    sub->update_strategy.type != SM_UPDATE_UNSPECIFIED) {
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v3 2/4] submodule--helper: introduce for_each_listed_submodule()
  2017-08-24 19:50           ` [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
  2017-08-24 19:50             ` [GSoC][PATCH v3 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
@ 2017-08-24 19:50             ` Prathamesh Chavan
  2017-08-24 19:50             ` [GSoC][PATCH v3 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
                               ` (2 subsequent siblings)
  4 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-24 19:50 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Introduce function for_each_listed_submodule() and replace a loop
in module_init() with a call to it.

The new function will also be used in other parts of the
system in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 39 +++++++++++++++++++++++++++++----------
 1 file changed, 29 insertions(+), 10 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index e666f84ba..8cd81b144 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -14,6 +14,9 @@
 #include "refs.h"
 #include "connect.h"
 
+typedef void (*each_submodule_fn)(const struct cache_entry *list_item,
+				  void *cb_data);
+
 static char *get_default_remote(void)
 {
 	char *dest = NULL, *ret;
@@ -353,17 +356,30 @@ static int module_list(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
-static void init_submodule(const char *path, const char *prefix, int quiet)
+static void for_each_listed_submodule(const struct module_list *list,
+				      each_submodule_fn fn, void *cb_data)
+{
+	int i;
+	for (i = 0; i < list->nr; i++)
+		fn(list->entries[i], cb_data);
+}
+
+struct init_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+};
+#define INIT_CB_INIT { NULL, 0 }
+
+static void init_submodule(const struct cache_entry *list_item, void *cb_data)
 {
+	struct init_cb *info = cb_data;
 	const struct submodule *sub;
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	/* Only loads from .gitmodules, no overlay with .git/config */
-	gitmodules_config();
-	displaypath = get_submodule_displaypath(path, prefix);
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(&null_oid, list_item->name);
 
 	if (!sub)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
@@ -375,7 +391,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 *
 	 * Set active flag for the submodule being initialized
 	 */
-	if (!is_submodule_active(the_repository, path)) {
+	if (!is_submodule_active(the_repository, list_item->name)) {
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
 		strbuf_reset(&sb);
@@ -417,7 +433,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		if (git_config_set_gently(sb.buf, url))
 			die(_("Failed to register url for submodule path '%s'"),
 			    displaypath);
-		if (!quiet)
+		if (!info->quiet)
 			fprintf(stderr,
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
@@ -446,10 +462,10 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 
 static int module_init(int argc, const char **argv, const char *prefix)
 {
+	struct init_cb info = INIT_CB_INIT;
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
-	int i;
 
 	struct option module_init_options[] = {
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
@@ -474,8 +490,11 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
 
-	for (i = 0; i < list.nr; i++)
-		init_submodule(list.entries[i]->name, prefix, quiet);
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+
+	gitmodules_config();
+	for_each_listed_submodule(&list, init_submodule, &info);
 
 	return 0;
 }
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v3 3/4] submodule: port set_name_rev() from shell to C
  2017-08-24 19:50           ` [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
  2017-08-24 19:50             ` [GSoC][PATCH v3 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
  2017-08-24 19:50             ` [GSoC][PATCH v3 2/4] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
@ 2017-08-24 19:50             ` Prathamesh Chavan
  2017-08-24 19:50             ` [GSoC][PATCH v3 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  2017-08-25 18:51             ` [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules Junio C Hamano
  4 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-24 19:50 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Function set_name_rev() is ported from git-submodule to the
submodule--helper builtin. The function compute_rev_name() generates the
value of the revision name as required.
The function get_rev_name() calls compute_rev_name() and receives the
revision name, and later handles its formating and printing.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 63 +++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            | 16 ++----------
 2 files changed, 65 insertions(+), 14 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 8cd81b144..6ea6408c2 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -245,6 +245,68 @@ static char *get_submodule_displaypath(const char *path, const char *prefix)
 	}
 }
 
+static char *compute_rev_name(const char *sub_path, const char* object_id)
+{
+	struct strbuf sb = STRBUF_INIT;
+	const char ***d;
+
+	static const char *describe_bare[] = {
+		NULL
+	};
+
+	static const char *describe_tags[] = {
+		"--tags", NULL
+	};
+
+	static const char *describe_contains[] = {
+		"--contains", NULL
+	};
+
+	static const char *describe_all_always[] = {
+		"--all", "--always", NULL
+	};
+
+	static const char **describe_argv[] = {
+		describe_bare, describe_tags, describe_contains,
+		describe_all_always, NULL
+	};
+
+	for (d = describe_argv; *d; d++) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+		prepare_submodule_repo_env(&cp.env_array);
+		cp.dir = sub_path;
+		cp.git_cmd = 1;
+		cp.no_stderr = 1;
+
+		argv_array_push(&cp.args, "describe");
+		argv_array_pushv(&cp.args, *d);
+		argv_array_push(&cp.args, object_id);
+
+		if (!capture_command(&cp, &sb, 0) && sb.len) {
+			strbuf_strip_suffix(&sb, "\n");
+			return strbuf_detach(&sb, NULL);
+		}
+	}
+
+	strbuf_release(&sb);
+	return NULL;
+}
+
+static int get_rev_name(int argc, const char **argv, const char *prefix)
+{
+	char *revname;
+	if (argc != 3)
+		die("get-rev-name only accepts two arguments: <path> <sha1>");
+
+	revname = compute_rev_name(argv[1], argv[2]);
+	if (revname && revname[0])
+		printf(" (%s)", revname);
+	printf("\n");
+
+	free(revname);
+	return 0;
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -1243,6 +1305,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"get-rev-name", get_rev_name, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index e131760ee..91f043ec6 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -759,18 +759,6 @@ cmd_update()
 	}
 }
 
-set_name_rev () {
-	revname=$( (
-		sanitize_submodule_env
-		cd "$1" && {
-			git describe "$2" 2>/dev/null ||
-			git describe --tags "$2" 2>/dev/null ||
-			git describe --contains "$2" 2>/dev/null ||
-			git describe --all --always "$2"
-		}
-	) )
-	test -z "$revname" || revname=" ($revname)"
-}
 #
 # Show commit summary for submodules in index or working tree
 #
@@ -1042,14 +1030,14 @@ cmd_status()
 		fi
 		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
 		then
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
 			say " $sha1 $displaypath$revname"
 		else
 			if test -z "$cached"
 			then
 				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
 			fi
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
 			say "+$sha1 $displaypath$revname"
 		fi
 
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v3 4/4] submodule: port submodule subcommand 'status' from shell to C
  2017-08-24 19:50           ` [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
                               ` (2 preceding siblings ...)
  2017-08-24 19:50             ` [GSoC][PATCH v3 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
@ 2017-08-24 19:50             ` " Prathamesh Chavan
  2017-08-25 18:51             ` [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules Junio C Hamano
  4 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-24 19:50 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

This aims to make git-submodule 'status' a built-in. Hence, the function
cmd_status() is ported from shell to C. This is done by introducing
three functions: module_status(), submodule_status() and print_status().

The function module_status() acts as the front-end of the subcommand.
It parses subcommand's options and then calls the function
module_list_compute() for computing the list of submodules. Then
this functions calls for_each_listed_submodule() looping through the
list obtained.

Then for_each_listed_submodule() calls submodule_status() for each of the
submodule in its list. The function submodule_status() is responsible
for generating the status each submodule it is called for, and
then calls print_status().

Finally, the function print_status() handles the printing of submodule's
status.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 156 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  49 +-------------
 2 files changed, 157 insertions(+), 48 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 6ea6408c2..577494e31 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -561,6 +561,161 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+struct status_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+	unsigned int recursive: 1;
+	unsigned int cached: 1;
+};
+#define STATUS_CB_INIT { NULL, 0, 0, 0 }
+
+static void print_status(struct status_cb *info, char state, const char *path,
+			 const struct object_id *oid, const char *displaypath)
+{
+	if (info->quiet)
+		return;
+
+	printf("%c%s %s", state, oid_to_hex(oid), displaypath);
+
+	if (state == ' ' || state == '+') {
+		struct argv_array get_rev_args = ARGV_ARRAY_INIT;
+
+		argv_array_pushl(&get_rev_args, "get-rev-name",
+				 path, oid_to_hex(oid), NULL);
+		get_rev_name(get_rev_args.argc, get_rev_args.argv,
+			     info->prefix);
+
+		argv_array_clear(&get_rev_args);
+	} else {
+		printf("\n");
+	}
+}
+
+static int handle_submodule_head_ref(const char *refname,
+				     const struct object_id *oid, int flags,
+				     void *cb_data)
+{
+	struct object_id *output = cb_data;
+	if (oid)
+		oidcpy(output, oid);
+
+	return 0;
+}
+
+static void status_submodule(const struct cache_entry *list_item, void *cb_data)
+{
+	struct status_cb *info = cb_data;
+	char *displaypath;
+	struct argv_array diff_files_args = ARGV_ARRAY_INIT;
+
+	if (!submodule_from_path(&null_oid, list_item->name))
+		die(_("no submodule mapping found in .gitmodules for path '%s'"),
+		      list_item->name);
+
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
+
+	if (ce_stage(list_item)) {
+		print_status(info, 'U', list_item->name,
+			     &null_oid, displaypath);
+		goto cleanup;
+	}
+
+	if (!is_submodule_active(the_repository, list_item->name)) {
+		print_status(info, '-', list_item->name, &list_item->oid,
+			     displaypath);
+		goto cleanup;
+	}
+
+	argv_array_pushl(&diff_files_args, "diff-files",
+			 "--ignore-submodules=dirty", "--quiet", "--",
+			 list_item->name, NULL);
+
+	if (!cmd_diff_files(diff_files_args.argc, diff_files_args.argv,
+			    info->prefix)) {
+		print_status(info, ' ', list_item->name, &list_item->oid,
+			     displaypath);
+	} else {
+		if (!info->cached) {
+			struct object_id oid;
+
+			if (head_ref_submodule(list_item->name,
+					       handle_submodule_head_ref, &oid))
+				die(_("could not resolve HEAD ref inside the"
+				      "submodule '%s'"), list_item->name);
+
+			print_status(info, '+', list_item->name, &oid,
+				     displaypath);
+		} else {
+			print_status(info, '+', list_item->name,
+				     &list_item->oid, displaypath);
+		}
+	}
+
+	if (info->recursive) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = list_item->name;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_pushl(&cpr.args, "--super-prefix", displaypath,
+				 "submodule--helper", "status", "--recursive",
+				 NULL);
+
+		if (info->cached)
+			argv_array_push(&cpr.args, "--cached");
+
+		if (info->quiet)
+			argv_array_push(&cpr.args, "--quiet");
+
+		if (run_command(&cpr))
+			die(_("failed to recurse into submodule '%s'"),
+			      list_item->name);
+	}
+
+cleanup:
+	argv_array_clear(&diff_files_args);
+	free(displaypath);
+}
+
+static int module_status(int argc, const char **argv, const char *prefix)
+{
+	struct status_cb info = STATUS_CB_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+	int quiet = 0;
+	int cached = 0;
+	int recursive = 0;
+
+	struct option module_status_options[] = {
+		OPT__QUIET(&quiet, N_("Suppress submodule status output")),
+		OPT_BOOL(0, "cached", &cached, N_("Use commit stored in the index instead of the one stored in the submodule HEAD")),
+		OPT_BOOL(0, "recursive", &recursive, N_("Recurse into nested submodules")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule status [--quiet] [--cached] [--recursive] [<path>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_status_options,
+			     git_submodule_helper_usage, 0);
+
+	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+	info.recursive = !!recursive;
+	info.cached = !!cached;
+
+	gitmodules_config();
+	for_each_listed_submodule(&list, status_submodule, &info);
+
+	return 0;
+}
+
 static int module_name(int argc, const char **argv, const char *prefix)
 {
 	const struct submodule *sub;
@@ -1307,6 +1462,7 @@ static struct cmd_struct commands[] = {
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
 	{"get-rev-name", get_rev_name, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
+	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
diff --git a/git-submodule.sh b/git-submodule.sh
index 91f043ec6..51b057d82 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1005,54 +1005,7 @@ cmd_status()
 		shift
 	done
 
-	{
-		git submodule--helper list --prefix "$wt_prefix" "$@" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path") || exit
-		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-		if test "$stage" = U
-		then
-			say "U$sha1 $displaypath"
-			continue
-		fi
-		if ! git submodule--helper is-active "$sm_path" ||
-		{
-			! test -d "$sm_path"/.git &&
-			! test -f "$sm_path"/.git
-		}
-		then
-			say "-$sha1 $displaypath"
-			continue;
-		fi
-		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
-		then
-			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
-			say " $sha1 $displaypath$revname"
-		else
-			if test -z "$cached"
-			then
-				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
-			fi
-			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
-			say "+$sha1 $displaypath$revname"
-		fi
-
-		if test -n "$recursive"
-		then
-			(
-				prefix="$displaypath/"
-				sanitize_submodule_env
-				wt_prefix=
-				cd "$sm_path" &&
-				eval cmd_status
-			) ||
-			die "$(eval_gettext "Failed to recurse into submodule path '\$sm_path'")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper status ${GIT_QUIET:+--quiet} ${cached:+--cached} ${recursive:+--recursive} "$@"
 }
 #
 # Sync remote urls for submodules
-- 
2.13.0


^ permalink raw reply	[relevance 20%]

* Re: Submodule regression in 2.14?
  2017-08-22 18:10                       ` Stefan Beller
@ 2017-08-25  9:10                         ` Heiko Voigt
  2017-08-25 16:38                           ` Stefan Beller
  2017-08-25 16:53                           ` Junio C Hamano
  0 siblings, 2 replies; 200+ results
From: Heiko Voigt @ 2017-08-25  9:10 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, Lars Schneider, Brandon Williams, git

On Tue, Aug 22, 2017 at 11:10:52AM -0700, Stefan Beller wrote:
> On Tue, Aug 22, 2017 at 8:33 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
> >> On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> >> > On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
> >>     # You feel the superproject is in the way?
> >>     git worktree add --for-submodule <path/to/sub> ...
> >>     # The new submodule worktree puts the
> >>     # submodule only UX first. so it feels like its own
> >>     # repository, no need for specific flags.
> >
> > I am not sure I understand this one. What would that do? Put a worktree
> > for submodule path/to/sub to ...?
> 
> Yes, and at "..." you would have the UX of the submodule being
> its own repository, no interaction with the superproject.

Are you sure that is what Junio meant? IMO that would be 'git worktree
clone' or 'git worktree checkout', no?  In todays git terminology an
'add' is something that puts data into the repository / the index. That
is why I was/am confused.

Cheers Heiko

^ permalink raw reply	[relevance 15%]

* Re: Submodule regression in 2.14?
  2017-08-25  9:10                         ` Heiko Voigt
@ 2017-08-25 16:38                           ` Stefan Beller
  2017-08-25 16:53                           ` Junio C Hamano
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-25 16:38 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Junio C Hamano, Lars Schneider, Brandon Williams, git

On Fri, Aug 25, 2017 at 2:10 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> On Tue, Aug 22, 2017 at 11:10:52AM -0700, Stefan Beller wrote:
>> On Tue, Aug 22, 2017 at 8:33 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>> > On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
>> >> On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>> >> > On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
>> >>     # You feel the superproject is in the way?
>> >>     git worktree add --for-submodule <path/to/sub> ...
>> >>     # The new submodule worktree puts the
>> >>     # submodule only UX first. so it feels like its own
>> >>     # repository, no need for specific flags.
>> >
>> > I am not sure I understand this one. What would that do? Put a worktree
>> > for submodule path/to/sub to ...?
>>
>> Yes, and at "..." you would have the UX of the submodule being
>> its own repository, no interaction with the superproject.
>
> Are you sure that is what Junio meant?

I am not sure at all, what Junio thought about this.

> IMO that would be 'git worktree
> clone' or 'git worktree checkout', no?  In todays git terminology an
> 'add' is something that puts data into the repository / the index. That
> is why I was/am confused.

I went with current gits:
$ git worktree
usage: git worktree add [<options>] <path> [<branch>]
   or: git worktree list [<options>]
   or: git worktree lock [<options>] <path>
   or: git worktree prune [<options>]
   or: git worktree unlock <path>

where the 'add' adds a new worktree (my mental emphasis was on the
option that says: "Make the new worktree for the submodule, such
that in that worktree the submodule feels like the toplevel
repository". If 'add' is a bad naming choice, we may want to
discuss&deprecate it in a new thread as that is related to the
worktree area?)

Thanks,
Stefan

>
> Cheers Heiko

^ permalink raw reply	[relevance 24%]

* Re: Submodule regression in 2.14?
  2017-08-25  9:10                         ` Heiko Voigt
  2017-08-25 16:38                           ` Stefan Beller
@ 2017-08-25 16:53                           ` Junio C Hamano
  1 sibling, 0 replies; 200+ results
From: Junio C Hamano @ 2017-08-25 16:53 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Stefan Beller, Lars Schneider, Brandon Williams, git\

Heiko Voigt <hvoigt@hvoigt.net> writes:

> On Tue, Aug 22, 2017 at 11:10:52AM -0700, Stefan Beller wrote:
>> On Tue, Aug 22, 2017 at 8:33 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>> > On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
>> >> On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>> >> > On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
>> >>     # You feel the superproject is in the way?
>> >>     git worktree add --for-submodule <path/to/sub> ...
>> >>     # The new submodule worktree puts the
>> >>     # submodule only UX first. so it feels like its own
>> >>     # repository, no need for specific flags.
>> >
>> > I am not sure I understand this one. What would that do? Put a worktree
>> > for submodule path/to/sub to ...?
>> 
>> Yes, and at "..." you would have the UX of the submodule being
>> its own repository, no interaction with the superproject.
>
> Are you sure that is what Junio meant?

I am sure it was not.  

If adding a separate worktree only for one submodule, you should be
able to "cd $path-to-submodule" and run "git worktree add" to create
a new worktree, and I do not see any reason for the superproject to
get involved in that process.  Does it have more information ready
than the end user has to help the process of creating a new worktree?

While I can understand how --for-submodule above may be implemented,
I do not see the point of adding such an option.





^ permalink raw reply	[relevance 16%]

* Re: [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules
  2017-08-25 18:51             ` [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules Junio C Hamano
@ 2017-08-25 19:15               ` Stefan Beller
  2017-08-25 20:32                 ` Junio C Hamano
  2017-08-28 11:55               ` [GSoC][PATCH v4 " Prathamesh Chavan
  1 sibling, 1 reply; 200+ results
From: Stefan Beller @ 2017-08-25 19:15 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Prathamesh Chavan, Christian Couder, git

On Fri, Aug 25, 2017 at 11:51 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Thanks.  I'll try to queue these before I'll go offline.
>
> Mentors may want to help the student further in adjusting the patch
> series to the more recent codebase; unfortunately the area the GSoC
> project touches is a bit fluid these days.  I resolved the conflicts
> with nd/pune-in-worktree and bw/submodule-config-cleanup topics so
> that the result would compile, but I am not sure if the resolution
> is correct (e.g. there may be a new leak I introduced while doing
> so).
>
> Thanks.

Ok, noted.

Presumably I'll review your dirty merge then for this series?
(And later parts might go on top of the new dirty merge)

^ permalink raw reply	[relevance 16%]

* Re: [GSoC][PATCH v3 0/4] Incremental rewrite of git-submodules
  2017-08-25 19:15               ` Stefan Beller
@ 2017-08-25 20:32                 ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-08-25 20:32 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Prathamesh Chavan, Christian Couder, git\

Stefan Beller <sbeller@google.com> writes:

> On Fri, Aug 25, 2017 at 11:51 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Thanks.  I'll try to queue these before I'll go offline.
>>
>> Mentors may want to help the student further in adjusting the patch
>> series to the more recent codebase; unfortunately the area the GSoC
>> project touches is a bit fluid these days.  I resolved the conflicts
>> with nd/pune-in-worktree and bw/submodule-config-cleanup topics so
>> that the result would compile, but I am not sure if the resolution
>> is correct (e.g. there may be a new leak I introduced while doing
>> so).
>>
>> Thanks.
>
> Ok, noted.
>
> Presumably I'll review your dirty merge then for this series?
> (And later parts might go on top of the new dirty merge)

In my mind, GSoC mentors are there to help the student improve as a
developer, more than they are to help improving the immediate output
of the student.  So in that sense, I was hoping that you'd teach how
to work well together with other developers, when one's topic has
interactions with topics by others.  Making sure an inevitable evil
merge is made correctly is of course needed for the current topic,
but a more important skill to learn is to avoid the need to ask an
evil merge to be made by the integrator in the first place.



^ permalink raw reply	[relevance 15%]

* [GSoC][PATCH v4 1/4] submodule--helper: introduce get_submodule_displaypath()
  2017-08-28 11:55               ` [GSoC][PATCH v4 " Prathamesh Chavan
@ 2017-08-28 11:55                 ` Prathamesh Chavan
  2017-08-28 11:55                 ` [GSoC][PATCH v4 2/4] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
                                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-28 11:55 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Introduce function get_submodule_displaypath() to replace the code
occurring in submodule_init() for generating displaypath of the
submodule with a call to it.

This new function will also be used in other parts of the system
in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 37 +++++++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 12 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 818fe74f0..e25854371 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -220,6 +220,28 @@ static int resolve_relative_url_test(int argc, const char **argv, const char *pr
 	return 0;
 }
 
+static char *get_submodule_displaypath(const char *path, const char *prefix)
+{
+	const char *super_prefix = get_super_prefix();
+
+	if (prefix && super_prefix) {
+		BUG("cannot have prefix '%s' and superprefix '%s'",
+		    prefix, super_prefix);
+	} else if (prefix) {
+		struct strbuf sb = STRBUF_INIT;
+		char *displaypath = xstrdup(relative_path(path, prefix, &sb));
+		strbuf_release(&sb);
+		return displaypath;
+	} else if (super_prefix) {
+		int len = strlen(super_prefix);
+		const char *format = is_dir_sep(super_prefix[len - 1]) ? "%s%s" : "%s/%s";
+
+		return xstrfmt(format, super_prefix, path);
+	} else {
+		return xstrdup(path);
+	}
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -335,15 +357,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	if (prefix && get_super_prefix())
-		die("BUG: cannot have prefix and superprefix");
-	else if (prefix)
-		displaypath = xstrdup(relative_path(path, prefix, &sb));
-	else if (get_super_prefix()) {
-		strbuf_addf(&sb, "%s%s", get_super_prefix(), path);
-		displaypath = strbuf_detach(&sb, NULL);
-	} else
-		displaypath = xstrdup(path);
+	displaypath = get_submodule_displaypath(path, prefix);
 
 	sub = submodule_from_path(&null_oid, path);
 
@@ -358,9 +372,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * Set active flag for the submodule being initialized
 	 */
 	if (!is_submodule_active(the_repository, path)) {
-		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
+		strbuf_reset(&sb);
 	}
 
 	/*
@@ -368,7 +382,6 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * To look up the url in .git/config, we must not fall back to
 	 * .gitmodules, so look it up directly.
 	 */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.url", sub->name);
 	if (git_config_get_string(sb.buf, &url)) {
 		if (!sub->url)
@@ -405,9 +418,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
 	}
+	strbuf_reset(&sb);
 
 	/* Copy "update" setting when it is not set yet */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.update", sub->name);
 	if (git_config_get_string(sb.buf, &upd) &&
 	    sub->update_strategy.type != SM_UPDATE_UNSPECIFIED) {
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v4 2/4] submodule--helper: introduce for_each_listed_submodule()
  2017-08-28 11:55               ` [GSoC][PATCH v4 " Prathamesh Chavan
  2017-08-28 11:55                 ` [GSoC][PATCH v4 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
@ 2017-08-28 11:55                 ` Prathamesh Chavan
  2017-08-28 11:55                 ` [GSoC][PATCH v4 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
  2017-08-28 11:55                 ` [GSoC][PATCH v4 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-28 11:55 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Introduce function for_each_listed_submodule() and replace a loop
in module_init() with a call to it.

The new function will also be used in other parts of the
system in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 36 ++++++++++++++++++++++++++++--------
 1 file changed, 28 insertions(+), 8 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index e25854371..ea99d8e39 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -14,6 +14,9 @@
 #include "refs.h"
 #include "connect.h"
 
+typedef void (*each_submodule_fn)(const struct cache_entry *list_item,
+				  void *cb_data);
+
 static char *get_default_remote(void)
 {
 	char *dest = NULL, *ret;
@@ -351,15 +354,30 @@ static int module_list(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
-static void init_submodule(const char *path, const char *prefix, int quiet)
+static void for_each_listed_submodule(const struct module_list *list,
+				      each_submodule_fn fn, void *cb_data)
+{
+	int i;
+	for (i = 0; i < list->nr; i++)
+		fn(list->entries[i], cb_data);
+}
+
+struct init_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+};
+#define INIT_CB_INIT { NULL, 0 }
+
+static void init_submodule(const struct cache_entry *list_item, void *cb_data)
 {
+	struct init_cb *info = cb_data;
 	const struct submodule *sub;
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	displaypath = get_submodule_displaypath(path, prefix);
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(&null_oid, list_item->name);
 
 	if (!sub)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
@@ -371,7 +389,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 *
 	 * Set active flag for the submodule being initialized
 	 */
-	if (!is_submodule_active(the_repository, path)) {
+	if (!is_submodule_active(the_repository, list_item->name)) {
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
 		strbuf_reset(&sb);
@@ -413,7 +431,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		if (git_config_set_gently(sb.buf, url))
 			die(_("Failed to register url for submodule path '%s'"),
 			    displaypath);
-		if (!quiet)
+		if (!info->quiet)
 			fprintf(stderr,
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
@@ -442,10 +460,10 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 
 static int module_init(int argc, const char **argv, const char *prefix)
 {
+	struct init_cb info = INIT_CB_INIT;
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
-	int i;
 
 	struct option module_init_options[] = {
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
@@ -470,8 +488,10 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
 
-	for (i = 0; i < list.nr; i++)
-		init_submodule(list.entries[i]->name, prefix, quiet);
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+
+	for_each_listed_submodule(&list, init_submodule, &info);
 
 	return 0;
 }
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v4 3/4] submodule: port set_name_rev() from shell to C
  2017-08-28 11:55               ` [GSoC][PATCH v4 " Prathamesh Chavan
  2017-08-28 11:55                 ` [GSoC][PATCH v4 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
  2017-08-28 11:55                 ` [GSoC][PATCH v4 2/4] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
@ 2017-08-28 11:55                 ` Prathamesh Chavan
  2017-08-28 11:55                 ` [GSoC][PATCH v4 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-28 11:55 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

Function set_name_rev() is ported from git-submodule to the
submodule--helper builtin. The function compute_rev_name() generates the
value of the revision name as required.
The function get_rev_name() calls compute_rev_name() and receives the
revision name, and later handles its formating and printing.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 63 +++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            | 16 ++----------
 2 files changed, 65 insertions(+), 14 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index ea99d8e39..85df11129 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -245,6 +245,68 @@ static char *get_submodule_displaypath(const char *path, const char *prefix)
 	}
 }
 
+static char *compute_rev_name(const char *sub_path, const char* object_id)
+{
+	struct strbuf sb = STRBUF_INIT;
+	const char ***d;
+
+	static const char *describe_bare[] = {
+		NULL
+	};
+
+	static const char *describe_tags[] = {
+		"--tags", NULL
+	};
+
+	static const char *describe_contains[] = {
+		"--contains", NULL
+	};
+
+	static const char *describe_all_always[] = {
+		"--all", "--always", NULL
+	};
+
+	static const char **describe_argv[] = {
+		describe_bare, describe_tags, describe_contains,
+		describe_all_always, NULL
+	};
+
+	for (d = describe_argv; *d; d++) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+		prepare_submodule_repo_env(&cp.env_array);
+		cp.dir = sub_path;
+		cp.git_cmd = 1;
+		cp.no_stderr = 1;
+
+		argv_array_push(&cp.args, "describe");
+		argv_array_pushv(&cp.args, *d);
+		argv_array_push(&cp.args, object_id);
+
+		if (!capture_command(&cp, &sb, 0) && sb.len) {
+			strbuf_strip_suffix(&sb, "\n");
+			return strbuf_detach(&sb, NULL);
+		}
+	}
+
+	strbuf_release(&sb);
+	return NULL;
+}
+
+static int get_rev_name(int argc, const char **argv, const char *prefix)
+{
+	char *revname;
+	if (argc != 3)
+		die("get-rev-name only accepts two arguments: <path> <sha1>");
+
+	revname = compute_rev_name(argv[1], argv[2]);
+	if (revname && revname[0])
+		printf(" (%s)", revname);
+	printf("\n");
+
+	free(revname);
+	return 0;
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -1292,6 +1354,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"get-rev-name", get_rev_name, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index 66d1ae8ef..5211361c5 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -758,18 +758,6 @@ cmd_update()
 	}
 }
 
-set_name_rev () {
-	revname=$( (
-		sanitize_submodule_env
-		cd "$1" && {
-			git describe "$2" 2>/dev/null ||
-			git describe --tags "$2" 2>/dev/null ||
-			git describe --contains "$2" 2>/dev/null ||
-			git describe --all --always "$2"
-		}
-	) )
-	test -z "$revname" || revname=" ($revname)"
-}
 #
 # Show commit summary for submodules in index or working tree
 #
@@ -1041,14 +1029,14 @@ cmd_status()
 		fi
 		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
 		then
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
 			say " $sha1 $displaypath$revname"
 		else
 			if test -z "$cached"
 			then
 				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
 			fi
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
 			say "+$sha1 $displaypath$revname"
 		fi
 
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [GSoC][PATCH v4 4/4] submodule: port submodule subcommand 'status' from shell to C
  2017-08-28 11:55               ` [GSoC][PATCH v4 " Prathamesh Chavan
                                   ` (2 preceding siblings ...)
  2017-08-28 11:55                 ` [GSoC][PATCH v4 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
@ 2017-08-28 11:55                 ` " Prathamesh Chavan
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-08-28 11:55 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, pc44800, sbeller

This aims to make git-submodule 'status' a built-in. Hence, the function
cmd_status() is ported from shell to C. This is done by introducing
three functions: module_status(), submodule_status() and print_status().

The function module_status() acts as the front-end of the subcommand.
It parses subcommand's options and then calls the function
module_list_compute() for computing the list of submodules. Then
this functions calls for_each_listed_submodule() looping through the
list obtained.

Then for_each_listed_submodule() calls submodule_status() for each of the
submodule in its list. The function submodule_status() is responsible
for generating the status each submodule it is called for, and
then calls print_status().

Finally, the function print_status() handles the printing of submodule's
status.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 155 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  49 +-------------
 2 files changed, 156 insertions(+), 48 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 85df11129..abf5c126a 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -558,6 +558,160 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+struct status_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+	unsigned int recursive: 1;
+	unsigned int cached: 1;
+};
+#define STATUS_CB_INIT { NULL, 0, 0, 0 }
+
+static void print_status(struct status_cb *info, char state, const char *path,
+			 const struct object_id *oid, const char *displaypath)
+{
+	if (info->quiet)
+		return;
+
+	printf("%c%s %s", state, oid_to_hex(oid), displaypath);
+
+	if (state == ' ' || state == '+') {
+		struct argv_array get_rev_args = ARGV_ARRAY_INIT;
+
+		argv_array_pushl(&get_rev_args, "get-rev-name",
+				 path, oid_to_hex(oid), NULL);
+		get_rev_name(get_rev_args.argc, get_rev_args.argv,
+			     info->prefix);
+
+		argv_array_clear(&get_rev_args);
+	} else {
+		printf("\n");
+	}
+}
+
+static int handle_submodule_head_ref(const char *refname,
+				     const struct object_id *oid, int flags,
+				     void *cb_data)
+{
+	struct object_id *output = cb_data;
+	if (oid)
+		oidcpy(output, oid);
+
+	return 0;
+}
+
+static void status_submodule(const struct cache_entry *list_item, void *cb_data)
+{
+	struct status_cb *info = cb_data;
+	char *displaypath;
+	struct argv_array diff_files_args = ARGV_ARRAY_INIT;
+
+	if (!submodule_from_path(&null_oid, list_item->name))
+		die(_("no submodule mapping found in .gitmodules for path '%s'"),
+		      list_item->name);
+
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
+
+	if (ce_stage(list_item)) {
+		print_status(info, 'U', list_item->name,
+			     &null_oid, displaypath);
+		goto cleanup;
+	}
+
+	if (!is_submodule_active(the_repository, list_item->name)) {
+		print_status(info, '-', list_item->name, &list_item->oid,
+			     displaypath);
+		goto cleanup;
+	}
+
+	argv_array_pushl(&diff_files_args, "diff-files",
+			 "--ignore-submodules=dirty", "--quiet", "--",
+			 list_item->name, NULL);
+
+	if (!cmd_diff_files(diff_files_args.argc, diff_files_args.argv,
+			    info->prefix)) {
+		print_status(info, ' ', list_item->name, &list_item->oid,
+			     displaypath);
+	} else {
+		if (!info->cached) {
+			struct object_id oid;
+
+			if (head_ref_submodule(list_item->name,
+					       handle_submodule_head_ref, &oid))
+				die(_("could not resolve HEAD ref inside the"
+				      "submodule '%s'"), list_item->name);
+
+			print_status(info, '+', list_item->name, &oid,
+				     displaypath);
+		} else {
+			print_status(info, '+', list_item->name,
+				     &list_item->oid, displaypath);
+		}
+	}
+
+	if (info->recursive) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = list_item->name;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_pushl(&cpr.args, "--super-prefix", displaypath,
+				 "submodule--helper", "status", "--recursive",
+				 NULL);
+
+		if (info->cached)
+			argv_array_push(&cpr.args, "--cached");
+
+		if (info->quiet)
+			argv_array_push(&cpr.args, "--quiet");
+
+		if (run_command(&cpr))
+			die(_("failed to recurse into submodule '%s'"),
+			      list_item->name);
+	}
+
+cleanup:
+	argv_array_clear(&diff_files_args);
+	free(displaypath);
+}
+
+static int module_status(int argc, const char **argv, const char *prefix)
+{
+	struct status_cb info = STATUS_CB_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+	int quiet = 0;
+	int cached = 0;
+	int recursive = 0;
+
+	struct option module_status_options[] = {
+		OPT__QUIET(&quiet, N_("Suppress submodule status output")),
+		OPT_BOOL(0, "cached", &cached, N_("Use commit stored in the index instead of the one stored in the submodule HEAD")),
+		OPT_BOOL(0, "recursive", &recursive, N_("Recurse into nested submodules")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule status [--quiet] [--cached] [--recursive] [<path>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_status_options,
+			     git_submodule_helper_usage, 0);
+
+	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+	info.recursive = !!recursive;
+	info.cached = !!cached;
+
+	for_each_listed_submodule(&list, status_submodule, &info);
+
+	return 0;
+}
+
 static int module_name(int argc, const char **argv, const char *prefix)
 {
 	const struct submodule *sub;
@@ -1356,6 +1510,7 @@ static struct cmd_struct commands[] = {
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
 	{"get-rev-name", get_rev_name, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
+	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
diff --git a/git-submodule.sh b/git-submodule.sh
index 5211361c5..156255a9e 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1004,54 +1004,7 @@ cmd_status()
 		shift
 	done
 
-	{
-		git submodule--helper list --prefix "$wt_prefix" "$@" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path") || exit
-		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-		if test "$stage" = U
-		then
-			say "U$sha1 $displaypath"
-			continue
-		fi
-		if ! git submodule--helper is-active "$sm_path" ||
-		{
-			! test -d "$sm_path"/.git &&
-			! test -f "$sm_path"/.git
-		}
-		then
-			say "-$sha1 $displaypath"
-			continue;
-		fi
-		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
-		then
-			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
-			say " $sha1 $displaypath$revname"
-		else
-			if test -z "$cached"
-			then
-				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
-			fi
-			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
-			say "+$sha1 $displaypath$revname"
-		fi
-
-		if test -n "$recursive"
-		then
-			(
-				prefix="$displaypath/"
-				sanitize_submodule_env
-				wt_prefix=
-				cd "$sm_path" &&
-				eval cmd_status
-			) ||
-			die "$(eval_gettext "Failed to recurse into submodule path '\$sm_path'")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper status ${GIT_QUIET:+--quiet} ${cached:+--cached} ${recursive:+--recursive} "$@"
 }
 #
 # Sync remote urls for submodules
-- 
2.13.0


^ permalink raw reply	[relevance 20%]

* [PATCH 00/39] per-repository object store, part 1
@ 2017-08-30  6:46 Jonathan Nieder
  2017-08-30  6:52 ` [PATCH 02/39] repository: introduce object store field Jonathan Nieder
                   ` (2 more replies)
  0 siblings, 3 replies; 200+ results
From: Jonathan Nieder @ 2017-08-30  6:46 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller, brian m. carlson

Hi,

Most of the credit for this series should go to Stefan Beller.  I just
decided to pull the trigger on sending out what we have so far.

This series is about API.  It makes no functional change yet.

Today, when a git command wants to operate on some objects from another
repository (e.g., a submodule), it has two choices:

 A. Use run_command to operate on that repository in a separate process.

 B. Use add_to_alternates_memory to pretend the repository is an
    alternate.  This has a number of downsides.  Aside from aesthetics,
    one particularly painful consequence is that as alternates
    accumulate, the number of packs git has to check for objects
    increases, which can cause significant slowdowns.

Brandon Williams's recent work to introduce "struct repository" points
to a better way.  Encapsulating object access in struct repository
would mean:

  i. The API for accessing objects in another repository becomes more
     simple and familiar (instead of using the CLI or abusing alternates).

  ii. Operations on one repository do not interfere with another,
      neither in semantics (e.g. replace objects do not work correctly
      with the approach (B) above) nor performance (already described
      above).

 iii. Resources associated with access to a repository could be freed
      when done with that repo.

  iv. Thread-safe multiple readers to a single repository also become
      straightforward, by using multiple repository objects for the same
      repo.

This series is a small step in that direction.

At the end of this series, sha1_loose_object_info takes a repository
argument and can be independently called for multiple repositories.
Not incredibly useful on its own, but a future series will do the same
for sha1_object_info, which will be enough to migrate a caller in
submodule.c (which uses the object store for commit existence checks).

This series has a few phases:

 1. Patch 1 is a cleanup that made some of the later patches easier.

 2. Patches 2-6 create a struct object_store field inside struct
    repository and move some globals to it.

 3. Patches 7-24 are mechanical changes that update some functions to
    accept a repository argument. The only goal is to make the later
    patches that teach these functions to actual handle a repository
    other than the_repository easier to review.  The patches enforce
    at compile time that no caller passes a repository other than
    the_repository --- see patch 7 in particular for details on how
    that works.

 4. Patches 25-39 update the implementations of those functions to
    handle a repository other than the_repository.  This means the
    safety check introduced in phase 3 goes away completely --- all
    functions that gained a repository argument are safe to use with
    a repository argument other than the_repository.

Patches 2-6 and 25-39 should be the most interesting to review.  I'd
particularly appreciate if people can look over 25-39 carefully.  We
were careful not to leave any calls to functions that assume they are
operating on the_repository, but a triple-check is always welcome.

Thanks as well to brian m. carlson, who showed us how such a long and
potentially tedius series can be made bearable for reviewers.

Thoughts of all kinds welcome, as always.

Thanks,
Jonathan Nieder (24):
  pack: make packed_git_mru global a value instead of a pointer
  object-store: move packed_git and packed_git_mru to object store
    struct
  pack: move prepare_packed_git_run_once to object store struct
  pack: move approximate object count to object store struct
  pack: add repository argument to install_packed_git
  pack: add repository argument to prepare_packed_git_one
  pack: add repository argument to rearrange_packed_git
  pack: add repository argument to prepare_packed_git_mru
  pack: add repository argument to prepare_packed_git
  pack: add repository argument to reprepare_packed_git
  pack: add repository argument to sha1_file_name
  pack: add repository argument to map_sha1_file
  pack: allow install_packed_git to handle arbitrary repositories
  pack: allow rearrange_packed_git to handle arbitrary repositories
  pack: allow prepare_packed_git_mru to handle arbitrary repositories
  pack: allow prepare_packed_git_one to handle arbitrary repositories
  pack: allow prepare_packed_git to handle arbitrary repositories
  pack: allow reprepare_packed_git to handle arbitrary repositories
  pack: allow sha1_file_name to handle arbitrary repositories
  pack: allow stat_sha1_file to handle arbitrary repositories
  pack: allow open_sha1_file to handle arbitrary repositories
  pack: allow map_sha1_file_1 to handle arbitrary repositories
  pack: allow map_sha1_file to handle arbitrary repositories
  pack: allow sha1_loose_object_info to handle arbitrary repositories

Stefan Beller (15):
  repository: introduce object store field
  object-store: move alt_odb_list and alt_odb_tail to object store
    struct
  sha1_file: add repository argument to alt_odb_usable
  sha1_file: add repository argument to link_alt_odb_entry
  sha1_file: add repository argument to read_info_alternates
  sha1_file: add repository argument to link_alt_odb_entries
  sha1_file: add repository argument to stat_sha1_file
  sha1_file: add repository argument to open_sha1_file
  sha1_file: add repository argument to map_sha1_file_1
  sha1_file: add repository argument to sha1_loose_object_info
  object-store: add repository argument to prepare_alt_odb
  object-store: add repository argument to foreach_alt_odb
  sha1_file: allow alt_odb_usable to handle arbitrary repositories
  object-store: allow prepare_alt_odb to handle arbitrary repositories
  object-store: allow foreach_alt_odb to handle arbitrary repositories

 builtin/count-objects.c             |  10 ++-
 builtin/fsck.c                      |  15 ++--
 builtin/gc.c                        |   8 +-
 builtin/index-pack.c                |   1 +
 builtin/pack-objects.c              |  23 +++--
 builtin/pack-redundant.c            |   8 +-
 builtin/receive-pack.c              |   4 +-
 builtin/submodule--helper.c         |   4 +-
 bulk-checkin.c                      |   3 +-
 cache.h                             |  50 ++---------
 contrib/coccinelle/packed_git.cocci |  15 ++++
 fast-import.c                       |  10 ++-
 fetch-pack.c                        |   3 +-
 http-backend.c                      |   8 +-
 http-push.c                         |   1 +
 http-walker.c                       |   4 +-
 http.c                              |   9 +-
 mru.h                               |   1 +
 object-store.h                      |  71 ++++++++++++++++
 pack-bitmap.c                       |   6 +-
 pack-check.c                        |   1 +
 pack-revindex.c                     |   1 +
 packfile.c                          |  94 ++++++++++----------
 packfile.h                          |   6 +-
 reachable.c                         |   1 +
 repository.c                        |   4 +-
 repository.h                        |   7 ++
 server-info.c                       |   8 +-
 sha1_file.c                         | 165 ++++++++++++++++++++----------------
 sha1_name.c                         |  11 ++-
 streaming.c                         |   5 +-
 transport.c                         |   4 +-
 32 files changed, 344 insertions(+), 217 deletions(-)
 create mode 100644 contrib/coccinelle/packed_git.cocci
 create mode 100644 object-store.h

-- 
2.14.1.581.gf28d330327


^ permalink raw reply	[relevance 15%]

* [PATCH 02/39] repository: introduce object store field
  2017-08-30  6:46 [PATCH 00/39] per-repository object store, part 1 Jonathan Nieder
@ 2017-08-30  6:52 ` Jonathan Nieder
  2017-08-30  7:02 ` [PATCH 16/39] object-store: add repository argument to foreach_alt_odb Jonathan Nieder
  2017-08-30 23:07 ` [PATCH 00/39] per-repository object store, part 1 Brandon Williams
  2 siblings, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-08-30  6:52 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller, brian m. carlson

From: Stefan Beller <sbeller@google.com>

The object store field will contain any objects needed for access to
objects in a given repository.

This patch introduces the object store but for now it is empty.  C99
forbids empty structs, but common C compilers cope well with them and
this struct will gain members very soon (starting with the next
patch).

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Are there any straightforward ways to avoid the empty struct?  Should
the struct have a placeholder member to avoid language issues with an
object that would have size 0?

 object-store.h | 8 ++++++++
 repository.c   | 4 +++-
 repository.h   | 7 +++++++
 3 files changed, 18 insertions(+), 1 deletion(-)
 create mode 100644 object-store.h

diff --git a/object-store.h b/object-store.h
new file mode 100644
index 0000000000..05722cdde0
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,8 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct object_store {
+};
+#define OBJECT_STORE_INIT {}
+
+#endif /* OBJECT_STORE_H */
diff --git a/repository.c b/repository.c
index f107af7d76..566753ed4b 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,13 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, 0, 0
+	NULL, NULL, NULL, OBJECT_STORE_INIT,
+	NULL, NULL, NULL, NULL, NULL, NULL, &the_index, 0, 0
 };
 struct repository *the_repository = &the_repo;
 
diff --git a/repository.h b/repository.h
index 7f5e24a0a2..9633ad10f5 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -25,6 +27,11 @@ struct repository {
 	 */
 	char *objectdir;
 
+	/*
+	 * Holds any information related to the object store.
+	 */
+	struct object_store objects;
+
 	/*
 	 * Path to the repository's graft file.
 	 * Cannot be NULL after initialization.
-- 
2.14.1.581.gf28d330327


^ permalink raw reply	[relevance 15%]

* [PATCH 16/39] object-store: add repository argument to foreach_alt_odb
  2017-08-30  6:46 [PATCH 00/39] per-repository object store, part 1 Jonathan Nieder
  2017-08-30  6:52 ` [PATCH 02/39] repository: introduce object store field Jonathan Nieder
@ 2017-08-30  7:02 ` Jonathan Nieder
  2017-08-30 23:07 ` [PATCH 00/39] per-repository object store, part 1 Brandon Williams
  2 siblings, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-08-30  7:02 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller, brian m. carlson

From: Stefan Beller <sbeller@google.com>

Add a repository argument to allow foreach_alt_odb callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c     | 2 +-
 builtin/submodule--helper.c | 4 +++-
 cache.h                     | 2 --
 object-store.h              | 4 ++++
 sha1_file.c                 | 4 ++--
 transport.c                 | 4 +++-
 6 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 9334648dcc..843a7d7d7e 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -155,7 +155,7 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		printf("prune-packable: %lu\n", packed_loose);
 		printf("garbage: %lu\n", garbage);
 		printf("size-garbage: %s\n", garbage_buf.buf);
-		foreach_alt_odb(print_alternate, NULL);
+		foreach_alt_odb(the_repository, print_alternate, NULL);
 		strbuf_release(&loose_buf);
 		strbuf_release(&pack_buf);
 		strbuf_release(&garbage_buf);
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 818fe74f0a..299a71b90c 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -6,6 +6,7 @@
 #include "quote.h"
 #include "pathspec.h"
 #include "dir.h"
+#include "object-store.h"
 #include "submodule.h"
 #include "submodule-config.h"
 #include "string-list.h"
@@ -598,7 +599,8 @@ static void prepare_possible_alternates(const char *sm_name,
 		die(_("Value '%s' for submodule.alternateErrorStrategy is not recognized"), error_strategy);
 
 	if (!strcmp(sm_alternate, "superproject"))
-		foreach_alt_odb(add_possible_reference_from_superproject, &sas);
+		foreach_alt_odb(the_repository,
+				add_possible_reference_from_superproject, &sas);
 	else if (!strcmp(sm_alternate, "no"))
 		; /* do nothing */
 	else
diff --git a/cache.h b/cache.h
index 5770224167..abec62dc9b 100644
--- a/cache.h
+++ b/cache.h
@@ -1524,8 +1524,6 @@ struct alternate_object_database {
 	char path[FLEX_ARRAY];
 };
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
-typedef int alt_odb_fn(struct alternate_object_database *, void *);
-extern int foreach_alt_odb(alt_odb_fn, void*);
 
 /*
  * Allocate a "struct alternate_object_database" but do _not_ actually
diff --git a/object-store.h b/object-store.h
index e749c952d5..eef8d3b653 100644
--- a/object-store.h
+++ b/object-store.h
@@ -57,4 +57,8 @@ struct packed_git {
 #define prepare_alt_odb(r) prepare_alt_odb_##r()
 extern void prepare_alt_odb_the_repository(void);
 
+typedef int alt_odb_fn(struct alternate_object_database *, void *);
+#define foreach_alt_odb(r, fn, cb) foreach_alt_odb_##r(fn, cb)
+extern int foreach_alt_odb_the_repository(alt_odb_fn, void*);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 9211c73920..8c599dbfc2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -606,7 +606,7 @@ char *compute_alternate_path(const char *path, struct strbuf *err)
 	return ref_git;
 }
 
-int foreach_alt_odb(alt_odb_fn fn, void *cb)
+int foreach_alt_odb_the_repository(alt_odb_fn fn, void *cb)
 {
 	struct alternate_object_database *ent;
 	int r = 0;
@@ -2031,7 +2031,7 @@ int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)
 
 	alt.cb = cb;
 	alt.data = data;
-	return foreach_alt_odb(loose_from_alt_odb, &alt);
+	return foreach_alt_odb(the_repository, loose_from_alt_odb, &alt);
 }
 
 static int check_stream_sha1(git_zstream *stream,
diff --git a/transport.c b/transport.c
index d75ff0514d..426a3a3a80 100644
--- a/transport.c
+++ b/transport.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "config.h"
 #include "transport.h"
+#include "repository.h"
 #include "run-command.h"
 #include "pkt-line.h"
 #include "fetch-pack.h"
@@ -11,6 +12,7 @@
 #include "bundle.h"
 #include "dir.h"
 #include "refs.h"
+#include "object-store.h"
 #include "branch.h"
 #include "url.h"
 #include "submodule.h"
@@ -1286,5 +1288,5 @@ void for_each_alternate_ref(alternate_ref_fn fn, void *data)
 	struct alternate_refs_data cb;
 	cb.fn = fn;
 	cb.data = data;
-	foreach_alt_odb(refs_from_alternate_cb, &cb);
+	foreach_alt_odb(the_repository, refs_from_alternate_cb, &cb);
 }
-- 
2.14.1.581.gf28d330327


^ permalink raw reply	[relevance 17%]

* Re: [PATCH 00/39] per-repository object store, part 1
  2017-08-30  6:46 [PATCH 00/39] per-repository object store, part 1 Jonathan Nieder
  2017-08-30  6:52 ` [PATCH 02/39] repository: introduce object store field Jonathan Nieder
  2017-08-30  7:02 ` [PATCH 16/39] object-store: add repository argument to foreach_alt_odb Jonathan Nieder
@ 2017-08-30 23:07 ` Brandon Williams
  2 siblings, 0 replies; 200+ results
From: Brandon Williams @ 2017-08-30 23:07 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: git, Stefan Beller, brian m. carlson

On 08/29, Jonathan Nieder wrote:
> Hi,
> 
> Most of the credit for this series should go to Stefan Beller.  I just
> decided to pull the trigger on sending out what we have so far.
> 
> This series is about API.  It makes no functional change yet.
> 
> Today, when a git command wants to operate on some objects from another
> repository (e.g., a submodule), it has two choices:
> 
>  A. Use run_command to operate on that repository in a separate process.
> 
>  B. Use add_to_alternates_memory to pretend the repository is an
>     alternate.  This has a number of downsides.  Aside from aesthetics,
>     one particularly painful consequence is that as alternates
>     accumulate, the number of packs git has to check for objects
>     increases, which can cause significant slowdowns.
> 
> Brandon Williams's recent work to introduce "struct repository" points
> to a better way.  Encapsulating object access in struct repository
> would mean:
> 
>   i. The API for accessing objects in another repository becomes more
>      simple and familiar (instead of using the CLI or abusing alternates).
> 
>   ii. Operations on one repository do not interfere with another,
>       neither in semantics (e.g. replace objects do not work correctly
>       with the approach (B) above) nor performance (already described
>       above).
> 
>  iii. Resources associated with access to a repository could be freed
>       when done with that repo.
> 
>   iv. Thread-safe multiple readers to a single repository also become
>       straightforward, by using multiple repository objects for the same
>       repo.
> 
> This series is a small step in that direction.
> 
> At the end of this series, sha1_loose_object_info takes a repository
> argument and can be independently called for multiple repositories.
> Not incredibly useful on its own, but a future series will do the same
> for sha1_object_info, which will be enough to migrate a caller in
> submodule.c (which uses the object store for commit existence checks).
> 
> This series has a few phases:
> 
>  1. Patch 1 is a cleanup that made some of the later patches easier.
> 
>  2. Patches 2-6 create a struct object_store field inside struct
>     repository and move some globals to it.
> 
>  3. Patches 7-24 are mechanical changes that update some functions to
>     accept a repository argument. The only goal is to make the later
>     patches that teach these functions to actual handle a repository
>     other than the_repository easier to review.  The patches enforce
>     at compile time that no caller passes a repository other than
>     the_repository --- see patch 7 in particular for details on how
>     that works.
> 
>  4. Patches 25-39 update the implementations of those functions to
>     handle a repository other than the_repository.  This means the
>     safety check introduced in phase 3 goes away completely --- all
>     functions that gained a repository argument are safe to use with
>     a repository argument other than the_repository.
> 
> Patches 2-6 and 25-39 should be the most interesting to review.  I'd
> particularly appreciate if people can look over 25-39 carefully.  We
> were careful not to leave any calls to functions that assume they are
> operating on the_repository, but a triple-check is always welcome.
> 
> Thanks as well to brian m. carlson, who showed us how such a long and
> potentially tedius series can be made bearable for reviewers.
> 
> Thoughts of all kinds welcome, as always.

Just finished looking through the series.  Thanks for keeping each
commit very short and to the point, it made reviewing it much easier.  I
couldn't see anything wrong these transformations and I am very happy to
see this work getting done.

One thing that needs to be noted is that currently the object_store is
only really being used by the_repository so this series didn't need to
create any object_store_init() or object_store_clear() type functions.
So these types of functions will need to be added once submodules are
using their own object store, in their own struct repository.

> 
> Thanks,
> Jonathan Nieder (24):
>   pack: make packed_git_mru global a value instead of a pointer
>   object-store: move packed_git and packed_git_mru to object store
>     struct
>   pack: move prepare_packed_git_run_once to object store struct
>   pack: move approximate object count to object store struct
>   pack: add repository argument to install_packed_git
>   pack: add repository argument to prepare_packed_git_one
>   pack: add repository argument to rearrange_packed_git
>   pack: add repository argument to prepare_packed_git_mru
>   pack: add repository argument to prepare_packed_git
>   pack: add repository argument to reprepare_packed_git
>   pack: add repository argument to sha1_file_name
>   pack: add repository argument to map_sha1_file
>   pack: allow install_packed_git to handle arbitrary repositories
>   pack: allow rearrange_packed_git to handle arbitrary repositories
>   pack: allow prepare_packed_git_mru to handle arbitrary repositories
>   pack: allow prepare_packed_git_one to handle arbitrary repositories
>   pack: allow prepare_packed_git to handle arbitrary repositories
>   pack: allow reprepare_packed_git to handle arbitrary repositories
>   pack: allow sha1_file_name to handle arbitrary repositories
>   pack: allow stat_sha1_file to handle arbitrary repositories
>   pack: allow open_sha1_file to handle arbitrary repositories
>   pack: allow map_sha1_file_1 to handle arbitrary repositories
>   pack: allow map_sha1_file to handle arbitrary repositories
>   pack: allow sha1_loose_object_info to handle arbitrary repositories
> 
> Stefan Beller (15):
>   repository: introduce object store field
>   object-store: move alt_odb_list and alt_odb_tail to object store
>     struct
>   sha1_file: add repository argument to alt_odb_usable
>   sha1_file: add repository argument to link_alt_odb_entry
>   sha1_file: add repository argument to read_info_alternates
>   sha1_file: add repository argument to link_alt_odb_entries
>   sha1_file: add repository argument to stat_sha1_file
>   sha1_file: add repository argument to open_sha1_file
>   sha1_file: add repository argument to map_sha1_file_1
>   sha1_file: add repository argument to sha1_loose_object_info
>   object-store: add repository argument to prepare_alt_odb
>   object-store: add repository argument to foreach_alt_odb
>   sha1_file: allow alt_odb_usable to handle arbitrary repositories
>   object-store: allow prepare_alt_odb to handle arbitrary repositories
>   object-store: allow foreach_alt_odb to handle arbitrary repositories
> 
>  builtin/count-objects.c             |  10 ++-
>  builtin/fsck.c                      |  15 ++--
>  builtin/gc.c                        |   8 +-
>  builtin/index-pack.c                |   1 +
>  builtin/pack-objects.c              |  23 +++--
>  builtin/pack-redundant.c            |   8 +-
>  builtin/receive-pack.c              |   4 +-
>  builtin/submodule--helper.c         |   4 +-
>  bulk-checkin.c                      |   3 +-
>  cache.h                             |  50 ++---------
>  contrib/coccinelle/packed_git.cocci |  15 ++++
>  fast-import.c                       |  10 ++-
>  fetch-pack.c                        |   3 +-
>  http-backend.c                      |   8 +-
>  http-push.c                         |   1 +
>  http-walker.c                       |   4 +-
>  http.c                              |   9 +-
>  mru.h                               |   1 +
>  object-store.h                      |  71 ++++++++++++++++
>  pack-bitmap.c                       |   6 +-
>  pack-check.c                        |   1 +
>  pack-revindex.c                     |   1 +
>  packfile.c                          |  94 ++++++++++----------
>  packfile.h                          |   6 +-
>  reachable.c                         |   1 +
>  repository.c                        |   4 +-
>  repository.h                        |   7 ++
>  server-info.c                       |   8 +-
>  sha1_file.c                         | 165 ++++++++++++++++++++----------------
>  sha1_name.c                         |  11 ++-
>  streaming.c                         |   5 +-
>  transport.c                         |   4 +-
>  32 files changed, 344 insertions(+), 217 deletions(-)
>  create mode 100644 contrib/coccinelle/packed_git.cocci
>  create mode 100644 object-store.h
> 
> -- 
> 2.14.1.581.gf28d330327
> 

-- 
Brandon Williams

^ permalink raw reply	[relevance 6%]

* Re: [PATCH 2/3] merge-recursive: remove return value from get_files_dirs
  2017-08-29 15:58     ` Kevin Willford
@ 2017-08-31 18:12       ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-08-31 18:12 UTC (permalink / raw)
  To: Kevin Willford; +Cc: Jeff King, git, gitster

On Tue, Aug 29, 2017 at 8:58 AM, Kevin Willford <kewillf@microsoft.com> wrote:
>>
>> On Mon, Aug 28, 2017 at 02:28:28PM -0600, Kevin Willford wrote:
>>
>> > The return value of the get_files_dirs call is never being used.
>> > Looking at the history of the file and it was originally only
>> > being used for debug output statements.  Also when
>> > read_tree_recursive return value is non zero it is changed to
>> > zero.  This leads me to believe that it doesn't matter if
>> > read_tree_recursive gets an error.
>>
>> Or that the function is buggy. :)
>
> That was one of my questions as well.  Should read_tree_recursive
> be propagating a -1 and merge_trees be checking for that and bail
> when the call to get_files_dirs return is < 0?  I made a commit with
> this change and ran the tests and they all still passed so either this
> return really doesn't matter or there are not sufficient tests covering
> it.
>
> I went with this change because it was not changing any of the
> current functionality and if we find a case where it matters that
> read_tree_recursive fails due to bad tree or something else we
> can address it then.
>
>>
>> I'm tempted to say that we should probably die() when
>> read_tree_recursive fails. This should only happen if we fail to parse
>> the tree, or if our callback (save_files_dirs here) returns failure, and
>> the latter looks like it never happens.
>>
>> > Since the debug output has been removed and the caller isn't
>> > checking the return value there is no reason to keep calulating
>> > and returning a value.
>>
>> Agreed, and I'm happy to see dead code go.
>>
>> Minor nit: s/calulating/calculating/ in your commit message.
>
> When will that spell checker for git messages be ready? ;)

Different workflows need different setups apparently.
(me, as a heavy user of git-gui, likes the builtin spell checker,
though I need to find a way to train it to accept git lingo, such
as "submodule", or "gitlink")

Maybe:
https://tarasalenin.wordpress.com/2010/09/11/configure-git-gui-spell-checker-on-windows/

If you do not use git-gui... you are at the merci of your $EDITOR
($GIT_EDITOR, or core.editor) to have spell checking.

^ permalink raw reply	[relevance 15%]

* Re: [PATCH] pull: honor submodule.recurse config option
  2017-09-01  7:29 ` [PATCH] pull: honor submodule.recurse config option Nicolas Morey-Chaisemartin
@ 2017-09-01 17:11   ` Stefan Beller
  2017-09-01 18:15     ` René Scharfe
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-01 17:11 UTC (permalink / raw)
  To: Nicolas Morey-Chaisemartin; +Cc: git, magnus

On Fri, Sep 1, 2017 at 12:29 AM, Nicolas Morey-Chaisemartin
<nicolas@morey-chaisemartin.com> wrote:
> git pull used to not parse the submodule.recurse config option and simply
> consider the --recurse-submodules CLI option.
> When using the config option, submodules would only be fetched recursively
> while the CLi option would tigger both fetch and update/merge.
>
> Reported-by: Magnus Homann <magnus@homann.se>
> Signed-off-by: Nicolas Morey-Chaisemartin <nicolas@morey-chaisemartin.com>

Reviewed-by: Stefan Beller <sbeller@google.com>

Thanks,
Stefan


> ---
>  builtin/pull.c | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/builtin/pull.c b/builtin/pull.c
> index 7fe281414..e4edf23c5 100644
> --- a/builtin/pull.c
> +++ b/builtin/pull.c
> @@ -326,6 +326,11 @@ static int git_pull_config(const char *var, const char *value, void *cb)
>                 config_autostash = git_config_bool(var, value);
>                 return 0;
>         }
> +       if (!strcmp(var, "submodule.recurse")) {
> +               int r = git_config_bool(var, value) ?
> +                       RECURSE_SUBMODULES_ON : RECURSE_SUBMODULES_OFF;
> +               recurse_submodules = r;
> +       }
>         return git_default_config(var, value, cb);
>  }
>
> --
> 2.14.1.460.g196d2604f
>

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] pull: honor submodule.recurse config option
  2017-09-01 17:11   ` Stefan Beller
@ 2017-09-01 18:15     ` René Scharfe
  0 siblings, 0 replies; 200+ results
From: René Scharfe @ 2017-09-01 18:15 UTC (permalink / raw)
  To: Stefan Beller, Nicolas Morey-Chaisemartin; +Cc: git, magnus

Am 01.09.2017 um 19:11 schrieb Stefan Beller:
> On Fri, Sep 1, 2017 at 12:29 AM, Nicolas Morey-Chaisemartin
> <nicolas@morey-chaisemartin.com> wrote:
>> git pull used to not parse the submodule.recurse config option and simply
>> consider the --recurse-submodules CLI option.
>> When using the config option, submodules would only be fetched recursively
>> while the CLi option would tigger both fetch and update/merge.
>>
>> Reported-by: Magnus Homann <magnus@homann.se>
>> Signed-off-by: Nicolas Morey-Chaisemartin <nicolas@morey-chaisemartin.com>
> 
> Reviewed-by: Stefan Beller <sbeller@google.com>
> 
> Thanks,
> Stefan
> 
> 
>> ---
>>   builtin/pull.c | 5 +++++
>>   1 file changed, 5 insertions(+)
>>
>> diff --git a/builtin/pull.c b/builtin/pull.c
>> index 7fe281414..e4edf23c5 100644
>> --- a/builtin/pull.c
>> +++ b/builtin/pull.c
>> @@ -326,6 +326,11 @@ static int git_pull_config(const char *var, const char *value, void *cb)
>>                  config_autostash = git_config_bool(var, value);
>>                  return 0;
>>          }
>> +       if (!strcmp(var, "submodule.recurse")) {
>> +               int r = git_config_bool(var, value) ?
>> +                       RECURSE_SUBMODULES_ON : RECURSE_SUBMODULES_OFF;>> +               recurse_submodules = r;

A few nits to pick:

Why not assign directly to recurse_submodules?  The variable r is only
set once and read once, and doesn't have a particularly descriptive name
that would justify having it.

builtin/fetch.c::git_fetch_config(), builtin/push.c::git_push_config()
and submodule.c::git_default_submodule_config() do the same, and I can't
infer why for them either.

And why fall through to git_default_config() here even though we know
that it won't match "submodule.recurse" again?  Config functions are
usually exit early on finding a match, as the "rebase.autostash" handler
above does.

>> +       }
>>          return git_default_config(var, value, cb);
>>   }
>>
>> --
>> 2.14.1.460.g196d2604f
>>

^ permalink raw reply	[relevance 24%]

* Re: [PATCH v4 02/16] refs.c: use is_dir_sep() in resolve_gitlink_ref()
  2017-08-23 19:14   ` Stefan Beller
@ 2017-09-06 11:08     ` Duy Nguyen
  2017-09-06 17:41       ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Duy Nguyen @ 2017-09-06 11:08 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Junio C Hamano, Michael Haggerty

On Thu, Aug 24, 2017 at 2:14 AM, Stefan Beller <sbeller@google.com> wrote:
> On Wed, Aug 23, 2017 at 5:36 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
>> The "submodule" argument in this function is a path, which can have
>> either '/' or '\\' as a separator. Use is_dir_sep() to support both.
>>
>> Noticed-by: Johannes Sixt <j6t@kdbg.org>
>> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
>
> Immediate questions that come to mind:
> * Could this go in as an independent bug fix?

It probably could. But it may depend on other submodule changes in this series.

>   -> If so do we have any test that fails or stops failing on Windows?

It was spotted during the review [1] so I guess no test fails on
Windows (not that I would know because I can't run tests on Windows)

>   -> If not, do we need one?

Since I don't use Windows, I don't particularly care. And I can't add
one anyway because I can't run it.

[1] https://public-inbox.org/git/%3Ca74cf309-fb16-2f45-8189-d1d0c655dea4@kdbg.org%3E/

> * Assuming this is not an independent bug fix:
>   Why do we need this patch in this series here?
>   (I thought this is about worktrees, not submodules.
>   So does this fix a potential bug that will be exposed
>   later in this series, but was harmless up to now?)

The series could probably be split in two. The first part is about
using the new refs_* API in revision.c. This helps clean up refs API a
bit, all *_submodule() functions will be one. Not strictly needed to
be part of this, it's just a continuation of my previous series that
introduces *_refs. Since submodule code is touched, this is found.

The second part is actually fixing the prune bug.

Should I split the series in two? I think I separated two parts in the
past (maybe I misremember) at least in the description...
-- 
Duy

^ permalink raw reply	[relevance 16%]

* Re: [PATCH v4 02/16] refs.c: use is_dir_sep() in resolve_gitlink_ref()
  2017-09-06 11:08     ` Duy Nguyen
@ 2017-09-06 17:41       ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-09-06 17:41 UTC (permalink / raw)
  To: Duy Nguyen; +Cc: git, Junio C Hamano, Michael Haggerty

On Wed, Sep 6, 2017 at 4:08 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Thu, Aug 24, 2017 at 2:14 AM, Stefan Beller <sbeller@google.com> wrote:
>> On Wed, Aug 23, 2017 at 5:36 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
>>> The "submodule" argument in this function is a path, which can have
>>> either '/' or '\\' as a separator. Use is_dir_sep() to support both.
>>>
>>> Noticed-by: Johannes Sixt <j6t@kdbg.org>
>>> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
>>
>> Immediate questions that come to mind:
>> * Could this go in as an independent bug fix?
>
> It probably could. But it may depend on other submodule changes in this series.
>
>>   -> If so do we have any test that fails or stops failing on Windows?
>
> It was spotted during the review [1] so I guess no test fails on
> Windows (not that I would know because I can't run tests on Windows)
>
>>   -> If not, do we need one?
>
> Since I don't use Windows, I don't particularly care. And I can't add
> one anyway because I can't run it.
>
> [1] https://public-inbox.org/git/%3Ca74cf309-fb16-2f45-8189-d1d0c655dea4@kdbg.org%3E/

IIRC I asked these questions as I was genuinely confused,
I do not mind too much either.

>
>> * Assuming this is not an independent bug fix:
>>   Why do we need this patch in this series here?
>>   (I thought this is about worktrees, not submodules.
>>   So does this fix a potential bug that will be exposed
>>   later in this series, but was harmless up to now?)
>
> The series could probably be split in two. The first part is about
> using the new refs_* API in revision.c. This helps clean up refs API a
> bit, all *_submodule() functions will be one. Not strictly needed to
> be part of this, it's just a continuation of my previous series that
> introduces *_refs. Since submodule code is touched, this is found.
>
> The second part is actually fixing the prune bug.
>
> Should I split the series in two? I think I separated two parts in the
> past (maybe I misremember) at least in the description...

I had to reread the coverletter
https://public-inbox.org/git/20170823123704.16518-1-pclouds@gmail.com/
to get that part. I would not be opposed to splitting the series, but
I'll review an unsplit series as well.

Thanks,
Stefan

> --
> Duy

^ permalink raw reply	[relevance 8%]

* Re: gitmodules below root directory
  2017-09-06 13:53 gitmodules below root directory Robert Dailey
@ 2017-09-06 18:35 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-09-06 18:35 UTC (permalink / raw)
  To: Robert Dailey, Prathamesh Chavan; +Cc: Git

On Wed, Sep 6, 2017 at 6:53 AM, Robert Dailey <rcdailey.lists@gmail.com> wrote:
> The gitmodules documentation[1] states that the .gitmodules file is at
> the root. However, it would be nice if this could be supported in any
> directory similar to how .gitignore works. Right now git-subrepo does
> not support submodules inside of a subrepo[2] (I suspect subtrees
> would have the same problem, but I did not verify). I think this is a
> limitation of git, rather than subrepo itself. Perhaps there are
> reasons why .gitmodules must be at the root, but I at least wanted to
> point it out and see if this could be supported.
>
> [1]: https://git-scm.com/docs/gitmodules
> [2]: https://github.com/ingydotnet/git-subrepo/issues/262

I agree that subtree likely suffers the same problem.
And at first it seems reasonable to want to have .gitmodules
at deeper trees supported, as that would fix subtree and subrepo
(and others) with ease.

Historically the need to store submodule URLs were the motivation
for having the .gitmodules file. An absolute URL for a submodule would
work fine no matter where the .gitmodules file would be located.
Relative URLs are currently defined as relative to the top level
of the project, which we would need to inspect if the anchor
is chosen well at the root or if we would want to allow anchoring
the relative URL within the tree. (This is no reason against
.gitmodules in deep trees, just pointing out the work required).

But does the URL still make sense? For absolute URLs this is likely
the case, for relative URLs my bets are off. Maybe?

It turned out that people want to e.g. move, delete and re-introduce
submodules, which is why the location of a submodule git directory
was moved to be either inside the tree (to keep supporting existing
git repos with submodules) as well as interned in the superproject.

In the example given in [2], the git dir of the submodule
("folder B") may be located at .git/modules/nameB as seen
from the root of RepoX:

    RepoX
    + folder A
    + folder B (submodule)
    + .gitmodules
    + .git # regular RepoX git dir
       + modules/<nameB>

An important mechanism of the .gitmodules file is
the resolution of the "name" and the "path" of the
submodule. (Given the path of a gitlink entry, where do
I find the git repository for the submodule? vice versa is slightly
less relevant: Given this git repository deep inside my own git
directory, where is the working tree)

So in the example we'd have

    RepoY
    + RepoX (subrepo)
      + folder A
      + folder B (submodule)
     +.gitmodules

The path entry in the .gitmodules file would not change via
subtree/subrepo merge, such that Git would need to know
that the actual path to the submodule is the
concatenation of 'path to tree in which the .gitmodules file is'
and the given path inside the .gitmodules file. Seems doable so far.

What about the name of a submodule? The .gitmodules file
follows the syntax of git config files, such that names cannot
occur twice as the names are stored as the section name:

  [submodule "nameB"]
    path = "folder B"

And I would think the property of having unique names
is important, such that each submodule has its unique
place to put its git dir inside the superprojects
"$GIT_DIR/modules/".

With multiple .gitmodules files, we would loose the
uniqueness property. (It may not be too bad, maybe
even a clever hack, haven't thought about it deeply,
but it seems ugly at first)

As said above, the name<->path resolution is
important, (and shall be unique, deterministic and simple),
so how do we do it? What about the case where we have

  .gitmodules "name" -> dir/path
  dir/.gitmodules "name" -> ./path

In this case we'd have the same mapping, but using this
mechanism we can map multiple names at the same path,
and we could choose to resolve a given path in different
.gitmodules files, which is cumbersome.

  anotherdir/.gitmodules "name" -> ../dir/path

seems crazy, too.

What about moving submodules?
Consider the example as in [2] again:

  $ git mv RepoX/folderB dir/sub
  $ git commit -m "move submodule"
  # ok fine, we can come up with a plan
  # where to put the submodule configuration,
  # maybe in dir/.gitmodules?

  $ git rm RepoX
  $ git commit -m "don't need the rest of RepoX"
  # observation: we would not want
  # RepoX/.gitmodules to still have impact on
  # the submodule.

  $ git revert HEAD^^ # undo the initial move
  # we'd move the .gitmodules file back to RepoX/.

tl;dr: I think this idea produces lots of interesting
corner cases in the data model, let's not go there
without having an idea how to solve them.

From an implementation stand point:
The submodule-config API could easily enhanced
to support reading multiple .gitmodules files (in case
their location is well defined, we would not want to
walk the whole tree recursively). This API is only
easily accessible from within C, such that current
implementing this idea in git-submodule.sh would
be a hassle to do. Prathamesh made good progress
in his GSoC project porting most of git-submodule.sh to
C, though, so once that is merged I'd claim that the
actual implementation of this idea is "rather easy".

Thanks,
Stefan

^ permalink raw reply	[relevance 16%]

* [PATCH 0/4] Fixes from the per-repository object store series
@ 2017-09-12 17:23 Jonathan Nieder
  2017-09-12 17:30 ` [PATCH 2/4] push, fetch: error out for submodule entries not pointing to commits Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Jonathan Nieder @ 2017-09-12 17:23 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller

Hi,

This is a continuation of the series at [1].  That was part 1 ---
you can think of this as part 0, since it contains the simplest and
least controversial part of the series --- each patch in this series
is a bugfix in its own right.

Patch 1 should be familiar if you reviewed the series [1].  It is
unchanged from the patch there, except to note Peff's ack.

Patches 2-4 have not yet visited the list but are fixes (or, in the
case of patch 4, cleanups) noticed as part of the same process of
teaching object store code to handle multiple repositories.

We hope that splitting these out should make them easier to review
and for people to benefit from these fixes without having to wait
for the rest of the series to settle.

Thoughts of all kinds welcome.

Jonathan Nieder (1):
  pack: make packed_git_mru global a value instead of a pointer

Stefan Beller (3):
  push, fetch: error out for submodule entries not pointing to commits
  replace-objects: evaluate replacement refs without using the object
    store
  packed refs: pass .git dir instead of packed-refs path to init_fn

 builtin/pack-objects.c         |  4 ++--
 cache.h                        |  4 ++--
 packfile.c                     | 12 +++++-------
 refs.c                         |  2 +-
 refs/files-backend.c           |  4 ++--
 refs/packed-backend.c          |  4 ++--
 submodule.c                    | 33 +++++++++++++++++++++++++--------
 t/t5531-deep-submodule-push.sh | 10 ++++++++++
 8 files changed, 49 insertions(+), 24 deletions(-)

[1] https://public-inbox.org/git/20170830064634.GA153983@aiede.mtv.corp.google.com/

^ permalink raw reply	[relevance 17%]

* [PATCH 2/4] push, fetch: error out for submodule entries not pointing to commits
  2017-09-12 17:23 [PATCH 0/4] Fixes from the per-repository object store series Jonathan Nieder
@ 2017-09-12 17:30 ` Jonathan Nieder
  2017-09-14 12:45   ` Heiko Voigt
  2017-09-18 16:52   ` Brandon Williams
  0 siblings, 2 replies; 200+ results
From: Jonathan Nieder @ 2017-09-12 17:30 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller, Brandon Williams, Heiko Voigt

From: Stefan Beller <sbeller@google.com>

The check_has_commit helper uses resolves a submodule entry to a
commit, when validating its existence. As a side effect this means
tolerates a submodule entry pointing to a tag, which is not a valid
submodule entry that git commands would know how to cope with.

Tighten the check to require an actual commit, not a tag pointing to a
commit.

Also improve the error handling when a submodule entry points to
non-commit (e.g., a blob) to error out instead of warning and
pretending the pointed to object doesn't exist.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 submodule.c                    | 33 +++++++++++++++++++++++++--------
 t/t5531-deep-submodule-push.sh | 10 ++++++++++
 2 files changed, 35 insertions(+), 8 deletions(-)

diff --git a/submodule.c b/submodule.c
index 3cea8221e0..e0da55920d 100644
--- a/submodule.c
+++ b/submodule.c
@@ -767,19 +767,36 @@ static int append_oid_to_argv(const struct object_id *oid, void *data)
 	return 0;
 }
 
+struct has_commit_data {
+	int result;
+	const char *path;
+};
+
 static int check_has_commit(const struct object_id *oid, void *data)
 {
-	int *has_commit = data;
+	struct has_commit_data *cb = data;
 
-	if (!lookup_commit_reference(oid))
-		*has_commit = 0;
+	enum object_type type = sha1_object_info(oid->hash, NULL);
 
-	return 0;
+	switch (type) {
+	case OBJ_COMMIT:
+		return 0;
+	case OBJ_BAD:
+		/*
+		 * Object is missing or invalid. If invalid, an error message
+		 * has already been printed.
+		 */
+		cb->result = 0;
+		return 0;
+	default:
+		die(_("submodule entry '%s' (%s) is a %s, not a commit"),
+		    cb->path, oid_to_hex(oid), typename(type));
+	}
 }
 
 static int submodule_has_commits(const char *path, struct oid_array *commits)
 {
-	int has_commit = 1;
+	struct has_commit_data has_commit = { 1, path };
 
 	/*
 	 * Perform a cheap, but incorrect check for the existence of 'commits'.
@@ -795,7 +812,7 @@ static int submodule_has_commits(const char *path, struct oid_array *commits)
 
 	oid_array_for_each_unique(commits, check_has_commit, &has_commit);
 
-	if (has_commit) {
+	if (has_commit.result) {
 		/*
 		 * Even if the submodule is checked out and the commit is
 		 * present, make sure it exists in the submodule's object store
@@ -814,12 +831,12 @@ static int submodule_has_commits(const char *path, struct oid_array *commits)
 		cp.dir = path;
 
 		if (capture_command(&cp, &out, GIT_MAX_HEXSZ + 1) || out.len)
-			has_commit = 0;
+			has_commit.result = 0;
 
 		strbuf_release(&out);
 	}
 
-	return has_commit;
+	return has_commit.result;
 }
 
 static int submodule_needs_pushing(const char *path, struct oid_array *commits)
diff --git a/t/t5531-deep-submodule-push.sh b/t/t5531-deep-submodule-push.sh
index 0f84a53146..39cb2c1c34 100755
--- a/t/t5531-deep-submodule-push.sh
+++ b/t/t5531-deep-submodule-push.sh
@@ -298,6 +298,16 @@ test_expect_success 'push succeeds if submodule commit disabling recursion from
 	)
 '
 
+test_expect_success 'submodule entry pointing at a tag is error' '
+	git -C work/gar/bage tag -a test1 -m "tag" &&
+	tag=$(git -C work/gar/bage rev-parse test1^{tag}) &&
+	git -C work update-index --cacheinfo 160000 "$tag" gar/bage &&
+	git -C work commit -m "bad commit" &&
+	test_when_finished "git -C work reset --hard HEAD^" &&
+	test_must_fail git -C work push --recurse-submodules=on-demand ../pub.git master 2>err &&
+	test_i18ngrep "is a tag, not a commit" err
+'
+
 test_expect_success 'push fails if recurse submodules option passed as yes' '
 	(
 		cd work/gar/bage &&
-- 
2.14.1.690.gbb1197296e


^ permalink raw reply	[relevance 23%]

* Re: [PATCH 2/4] push, fetch: error out for submodule entries not pointing to commits
  2017-09-12 17:30 ` [PATCH 2/4] push, fetch: error out for submodule entries not pointing to commits Jonathan Nieder
@ 2017-09-14 12:45   ` Heiko Voigt
  2017-09-18 16:52   ` Brandon Williams
  1 sibling, 0 replies; 200+ results
From: Heiko Voigt @ 2017-09-14 12:45 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: git, Stefan Beller, Brandon Williams

On Tue, Sep 12, 2017 at 10:30:27AM -0700, Jonathan Nieder wrote:
> From: Stefan Beller <sbeller@google.com>
> 
> The check_has_commit helper uses resolves a submodule entry to a
> commit, when validating its existence. As a side effect this means
> tolerates a submodule entry pointing to a tag, which is not a valid
> submodule entry that git commands would know how to cope with.
> 
> Tighten the check to require an actual commit, not a tag pointing to a
> commit.
> 
> Also improve the error handling when a submodule entry points to
> non-commit (e.g., a blob) to error out instead of warning and
> pretending the pointed to object doesn't exist.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Looks good to me.

Cheers Heiko

^ permalink raw reply	[relevance 7%]

* [PATCH v2] add test for bug in git-mv for recursive submodules
  2017-08-18 19:04     ` Stefan Beller
@ 2017-09-15 11:50       ` " Heiko Voigt
  2017-09-17  0:46         ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Heiko Voigt @ 2017-09-15 11:50 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Stefan Beller, git

When using git-mv with a submodule it will detect that and update the
paths for its configurations (.gitmodules, worktree and gitfile). This
does not work for recursive submodules where a user renames the root
submodule.

We discovered this fact when working on on-demand fetch for renamed
submodules. Lets add a test to document.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
On Fri, Aug 18, 2017 at 12:04:03PM -0700, Stefan Beller wrote:
> > I just copied the shortcut that they were adding themselfes as submodule
> > in 'setup submodule'. The whole setup of submodules in this test is like
> > this. This way we already had a nested submodule structure which I could
> > just add.
> >
> > I agree that this is unrealistic so I can change that in the test I am
> > adding. But from what I have seen, this shortcut is taken in quite some
> > places when dealing with submodules.
> 
> Please do not make it worse.
> Once upon a time (late '16 IIRC) I had a series floating on the
> list removing all occurrences, but there were issues with the
> series and it did not land.

Took a little while but here is a more clean patch creating individual
submodules for the nesting.

Cheers Heiko

 t/t7001-mv.sh | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
index e365d1ff77..cbc5fb37fe 100755
--- a/t/t7001-mv.sh
+++ b/t/t7001-mv.sh
@@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
 	test_cmp actual expect
 '
 
+test_expect_failure 'moving nested submodules' '
+	git commit -am "cleanup commit" &&
+	mkdir sub_nested_nested &&
+	(cd sub_nested_nested &&
+		touch nested_level2 &&
+		git init &&
+		git add . &&
+		git commit -m "nested level 2"
+	) &&
+	mkdir sub_nested &&
+	(cd sub_nested &&
+		touch nested_level1 &&
+		git init &&
+		git add . &&
+		git commit -m "nested level 1"
+		git submodule add ../sub_nested_nested &&
+		git commit -m "add nested level 2"
+	) &&
+	git submodule add ./sub_nested nested_move &&
+	git commit -m "add nested_move" &&
+	git submodule update --init --recursive &&
+	git mv nested_move sub_nested_moved &&
+	git status
+'
+
 test_done
-- 
2.14.1.145.gb3622a4


^ permalink raw reply	[relevance 24%]

* Re: [PATCH v2] add test for bug in git-mv for recursive submodules
  2017-09-15 11:50       ` [PATCH v2] add test for bug in git-mv for recursive " Heiko Voigt
@ 2017-09-17  0:46         ` Junio C Hamano
  2017-09-18 20:03           ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2017-09-17  0:46 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: Stefan Beller, git\

Heiko Voigt <hvoigt@hvoigt.net> writes:

> When using git-mv with a submodule it will detect that and update the
> paths for its configurations (.gitmodules, worktree and gitfile). This
> does not work for recursive submodules where a user renames the root
> submodule.
>
> We discovered this fact when working on on-demand fetch for renamed
> submodules. Lets add a test to document.
>
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
> On Fri, Aug 18, 2017 at 12:04:03PM -0700, Stefan Beller wrote:
>> > I just copied the shortcut that they were adding themselfes as submodule
>> > in 'setup submodule'. The whole setup of submodules in this test is like
>> > this. This way we already had a nested submodule structure which I could
>> > just add.
>> >
>> > I agree that this is unrealistic so I can change that in the test I am
>> > adding. But from what I have seen, this shortcut is taken in quite some
>> > places when dealing with submodules.
>> 
>> Please do not make it worse.
>> Once upon a time (late '16 IIRC) I had a series floating on the
>> list removing all occurrences, but there were issues with the
>> series and it did not land.
>
> Took a little while but here is a more clean patch creating individual
> submodules for the nesting.
>
> Cheers Heiko

Thanks.  Stefan, does this look good to you now?

It is not quite clear which step is expected to fail with the
current code by reading the test or the proposed log message.  Does
"mv" refuse to work and we do not get to run "status", or does
"status" report a failure, or do we fail well before that?

The log message that only says "This does not work when ..." is not
helpful in figuring it out, either.  Something like "This does not
work and fails to update the paths for its configurations" or
whatever that describes "what actually happens" (in contrast to
"what ought to happen", which you described clearly) should be
there.  

Description on how you happened to have discovered the issue feels a
lot less relevant compared to that, and it is totally useless if it
is unclear what the issue is in the first place.

>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
>
> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> index e365d1ff77..cbc5fb37fe 100755
> --- a/t/t7001-mv.sh
> +++ b/t/t7001-mv.sh
> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
>  	test_cmp actual expect
>  '
>  
> +test_expect_failure 'moving nested submodules' '
> +	git commit -am "cleanup commit" &&
> +	mkdir sub_nested_nested &&
> +	(cd sub_nested_nested &&
> +		touch nested_level2 &&
> +		git init &&
> +		git add . &&
> +		git commit -m "nested level 2"
> +	) &&
> +	mkdir sub_nested &&
> +	(cd sub_nested &&
> +		touch nested_level1 &&
> +		git init &&
> +		git add . &&
> +		git commit -m "nested level 1"
> +		git submodule add ../sub_nested_nested &&
> +		git commit -m "add nested level 2"
> +	) &&
> +	git submodule add ./sub_nested nested_move &&
> +	git commit -m "add nested_move" &&
> +	git submodule update --init --recursive &&
> +	git mv nested_move sub_nested_moved &&
> +	git status
> +'
> +
>  test_done

^ permalink raw reply	[relevance 7%]

* Re: [PATCH 2/4] push, fetch: error out for submodule entries not pointing to commits
  2017-09-12 17:30 ` [PATCH 2/4] push, fetch: error out for submodule entries not pointing to commits Jonathan Nieder
  2017-09-14 12:45   ` Heiko Voigt
@ 2017-09-18 16:52   ` Brandon Williams
  1 sibling, 0 replies; 200+ results
From: Brandon Williams @ 2017-09-18 16:52 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: git, Stefan Beller, Heiko Voigt

On 09/12, Jonathan Nieder wrote:
> From: Stefan Beller <sbeller@google.com>
> 
> The check_has_commit helper uses resolves a submodule entry to a
> commit, when validating its existence. As a side effect this means
> tolerates a submodule entry pointing to a tag, which is not a valid
> submodule entry that git commands would know how to cope with.
> 
> Tighten the check to require an actual commit, not a tag pointing to a
> commit.
> 
> Also improve the error handling when a submodule entry points to
> non-commit (e.g., a blob) to error out instead of warning and
> pretending the pointed to object doesn't exist.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Looks good!

> ---
>  submodule.c                    | 33 +++++++++++++++++++++++++--------
>  t/t5531-deep-submodule-push.sh | 10 ++++++++++
>  2 files changed, 35 insertions(+), 8 deletions(-)
> 
> diff --git a/submodule.c b/submodule.c
> index 3cea8221e0..e0da55920d 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -767,19 +767,36 @@ static int append_oid_to_argv(const struct object_id *oid, void *data)
>  	return 0;
>  }
>  
> +struct has_commit_data {
> +	int result;
> +	const char *path;
> +};
> +
>  static int check_has_commit(const struct object_id *oid, void *data)
>  {
> -	int *has_commit = data;
> +	struct has_commit_data *cb = data;
>  
> -	if (!lookup_commit_reference(oid))
> -		*has_commit = 0;
> +	enum object_type type = sha1_object_info(oid->hash, NULL);
>  
> -	return 0;
> +	switch (type) {
> +	case OBJ_COMMIT:
> +		return 0;
> +	case OBJ_BAD:
> +		/*
> +		 * Object is missing or invalid. If invalid, an error message
> +		 * has already been printed.
> +		 */
> +		cb->result = 0;
> +		return 0;
> +	default:
> +		die(_("submodule entry '%s' (%s) is a %s, not a commit"),
> +		    cb->path, oid_to_hex(oid), typename(type));
> +	}
>  }
>  
>  static int submodule_has_commits(const char *path, struct oid_array *commits)
>  {
> -	int has_commit = 1;
> +	struct has_commit_data has_commit = { 1, path };
>  
>  	/*
>  	 * Perform a cheap, but incorrect check for the existence of 'commits'.
> @@ -795,7 +812,7 @@ static int submodule_has_commits(const char *path, struct oid_array *commits)
>  
>  	oid_array_for_each_unique(commits, check_has_commit, &has_commit);
>  
> -	if (has_commit) {
> +	if (has_commit.result) {
>  		/*
>  		 * Even if the submodule is checked out and the commit is
>  		 * present, make sure it exists in the submodule's object store
> @@ -814,12 +831,12 @@ static int submodule_has_commits(const char *path, struct oid_array *commits)
>  		cp.dir = path;
>  
>  		if (capture_command(&cp, &out, GIT_MAX_HEXSZ + 1) || out.len)
> -			has_commit = 0;
> +			has_commit.result = 0;
>  
>  		strbuf_release(&out);
>  	}
>  
> -	return has_commit;
> +	return has_commit.result;
>  }
>  
>  static int submodule_needs_pushing(const char *path, struct oid_array *commits)
> diff --git a/t/t5531-deep-submodule-push.sh b/t/t5531-deep-submodule-push.sh
> index 0f84a53146..39cb2c1c34 100755
> --- a/t/t5531-deep-submodule-push.sh
> +++ b/t/t5531-deep-submodule-push.sh
> @@ -298,6 +298,16 @@ test_expect_success 'push succeeds if submodule commit disabling recursion from
>  	)
>  '
>  
> +test_expect_success 'submodule entry pointing at a tag is error' '
> +	git -C work/gar/bage tag -a test1 -m "tag" &&
> +	tag=$(git -C work/gar/bage rev-parse test1^{tag}) &&
> +	git -C work update-index --cacheinfo 160000 "$tag" gar/bage &&
> +	git -C work commit -m "bad commit" &&
> +	test_when_finished "git -C work reset --hard HEAD^" &&
> +	test_must_fail git -C work push --recurse-submodules=on-demand ../pub.git master 2>err &&
> +	test_i18ngrep "is a tag, not a commit" err
> +'
> +
>  test_expect_success 'push fails if recurse submodules option passed as yes' '
>  	(
>  		cd work/gar/bage &&
> -- 
> 2.14.1.690.gbb1197296e
> 

-- 
Brandon Williams

^ permalink raw reply	[relevance 7%]

* Re: [PATCH v2] add test for bug in git-mv for recursive submodules
  2017-09-17  0:46         ` Junio C Hamano
@ 2017-09-18 20:03           ` Stefan Beller
  2017-09-20 13:46             ` Heiko Voigt
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-18 20:03 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Heiko Voigt, git

>> Took a little while but here is a more clean patch creating individual
>> submodules for the nesting.
>>
>> Cheers Heiko

Thanks for writing this test!

>
> Thanks.  Stefan, does this look good to you now?

Yes, though there are nits below.

> It is not quite clear which step is expected to fail with the
> current code by reading the test or the proposed log message.  Does
> "mv" refuse to work and we do not get to run "status", or does
> "status" report a failure, or do we fail well before that?

git-mv failing seems like a new possibility without incurring
another process spawn with the new repository object.
(Though then we could also just fix the recursed submodule)

> The log message that only says "This does not work when ..." is not
> helpful in figuring it out, either.  Something like "This does not
> work and fails to update the paths for its configurations" or
> whatever that describes "what actually happens" (in contrast to
> "what ought to happen", which you described clearly) should be
> there.
>
> Description on how you happened to have discovered the issue feels a
> lot less relevant compared to that, and it is totally useless if it
> is unclear what the issue is in the first place.
>
>>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
>>  1 file changed, 25 insertions(+)
>>
>> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
>> index e365d1ff77..cbc5fb37fe 100755
>> --- a/t/t7001-mv.sh
>> +++ b/t/t7001-mv.sh
>> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
>>       test_cmp actual expect
>>  '
>>
>> +test_expect_failure 'moving nested submodules' '
>> +     git commit -am "cleanup commit" &&
>> +     mkdir sub_nested_nested &&
>> +     (cd sub_nested_nested &&

We seem to have different styles for nested shell. I prefer

  outside command &&
  (
      first nested command here &&
      ...

as that aligns indentation to the nesting level. I have seen
the style you use a lot in the  test suite, and we do not have
a guideline in Documentation/CodingGuidelines, so I do not
complain too loudly. ;)


>> +             touch nested_level2 &&
>> +             git init &&
>> +             git add . &&
>> +             git commit -m "nested level 2"
>> +     ) &&
>> +     mkdir sub_nested &&
>> +     (cd sub_nested &&
>> +             touch nested_level1 &&
>> +             git init &&
>> +             git add . &&
>> +             git commit -m "nested level 1"
>> +             git submodule add ../sub_nested_nested &&
>> +             git commit -m "add nested level 2"
>> +     ) &&
>> +     git submodule add ./sub_nested nested_move &&
>> +     git commit -m "add nested_move" &&
>> +     git submodule update --init --recursive &&

So far a nice setup!

>> +     git mv nested_move sub_nested_moved &&

This is the offending command that produces the bug,
as it will break most subsequent commands, such as

>> +     git status

git-status is one of the basic commands. Without
status to function, I think it is hard to recover your repo without
a lot of in-depth knowledge of Git (submodules).

I wonder if git-status should complain more gracefully
and fallback to one of --ignore-submodules={dirty, all},
that actually still works.

Maybe we could introduce a new default mode for this
flag, that is "none-except-on-error", though this sounds
as if we're fixing symptoms instead of the root cause.

^ permalink raw reply	[relevance 24%]

* Re: [PATCH v2] add test for bug in git-mv for recursive submodules
  2017-09-18 20:03           ` Stefan Beller
@ 2017-09-20 13:46             ` Heiko Voigt
  0 siblings, 0 replies; 200+ results
From: Heiko Voigt @ 2017-09-20 13:46 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, git

On Mon, Sep 18, 2017 at 01:03:32PM -0700, Stefan Beller wrote:
> >> Took a little while but here is a more clean patch creating individual
> >> submodules for the nesting.
> >>
> >> Cheers Heiko
> 
> Thanks for writing this test!

No worries. :)

> > Thanks.  Stefan, does this look good to you now?
> 
> Yes, though there are nits below.
> 
> > It is not quite clear which step is expected to fail with the
> > current code by reading the test or the proposed log message.  Does
> > "mv" refuse to work and we do not get to run "status", or does
> > "status" report a failure, or do we fail well before that?
> 
> git-mv failing seems like a new possibility without incurring
> another process spawn with the new repository object.
> (Though then we could also just fix the recursed submodule)

It is mv that fails to update everything necessary when using it with
recursively nested submodules. So the git-mv command does not report a
failure here. As an interim fix it could maybe report an error when
encountering nested submodules but the real fix would be to teach it to
recursively spawn the appropriate git-mv commands.

> > The log message that only says "This does not work when ..." is not
> > helpful in figuring it out, either.  Something like "This does not
> > work and fails to update the paths for its configurations" or
> > whatever that describes "what actually happens" (in contrast to
> > "what ought to happen", which you described clearly) should be
> > there.
> >
> > Description on how you happened to have discovered the issue feels a
> > lot less relevant compared to that, and it is totally useless if it
> > is unclear what the issue is in the first place.

Sorry about being a bit brief here. How about dropping that information
how I discovered the bug then and change the commit message to something
like this:

    add test for bug in git-mv for recursive submodules

    When using git-mv with a submodule it will detect that and update
    the paths for its configurations (.gitmodules, worktree and
    gitfile). This does not work in case it encounters nested
    submodules. In that case it only updates the configurations for the
    submodule directly underneath the superproject and fails to update
    the paths for the submodules nested more deeply. This in turn leads
    to the symptom that git status reports that it can not chdir to the
    nested submodule in its old location.

    Lets add a test to document.

?

> >>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
> >>  1 file changed, 25 insertions(+)
> >>
> >> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> >> index e365d1ff77..cbc5fb37fe 100755
> >> --- a/t/t7001-mv.sh
> >> +++ b/t/t7001-mv.sh
> >> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
> >>       test_cmp actual expect
> >>  '
> >>
> >> +test_expect_failure 'moving nested submodules' '
> >> +     git commit -am "cleanup commit" &&
> >> +     mkdir sub_nested_nested &&
> >> +     (cd sub_nested_nested &&
> 
> We seem to have different styles for nested shell. I prefer
> 
>   outside command &&
>   (
>       first nested command here &&
>       ...
> 
> as that aligns indentation to the nesting level. I have seen
> the style you use a lot in the  test suite, and we do not have
> a guideline in Documentation/CodingGuidelines, so I do not
> complain too loudly. ;)

Yeah we have some different styles it seems ;) So here some reasoning
behind my style:

I actually would agree on your style if 'first nested command' was any
arbitrary command but when I use my style it is always when I use a
nested shell for changing into some directory, doing something there and
then being able to return to the previous directory by closing the nested
shell. So for me the 'cd somewhere' belongs to the brackets similarly
like a condition definition belongs to the if it is used with.

> >> +             touch nested_level2 &&
> >> +             git init &&
> >> +             git add . &&
> >> +             git commit -m "nested level 2"
> >> +     ) &&
> >> +     mkdir sub_nested &&
> >> +     (cd sub_nested &&
> >> +             touch nested_level1 &&
> >> +             git init &&
> >> +             git add . &&
> >> +             git commit -m "nested level 1"
> >> +             git submodule add ../sub_nested_nested &&
> >> +             git commit -m "add nested level 2"
> >> +     ) &&
> >> +     git submodule add ./sub_nested nested_move &&
> >> +     git commit -m "add nested_move" &&
> >> +     git submodule update --init --recursive &&
> 
> So far a nice setup!

Thanks.

> >> +     git mv nested_move sub_nested_moved &&
> 
> This is the offending command that produces the bug,
> as it will break most subsequent commands, such as

Yes.

> >> +     git status
> 
> git-status is one of the basic commands. Without
> status to function, I think it is hard to recover your repo without
> a lot of in-depth knowledge of Git (submodules).
> 
> I wonder if git-status should complain more gracefully
> and fallback to one of --ignore-submodules={dirty, all},
> that actually still works.
> 
> Maybe we could introduce a new default mode for this
> flag, that is "none-except-on-error", though this sounds
> as if we're fixing symptoms instead of the root cause.

I think we should rather fix the root cause. For me git-mv is actually
breaking the repository and as described above one possible interim
solution for me would be for 'git-mv' to error out and tell the user
that it does currently not work on recursively nested submodules.

Cheers Heiko

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] git: add --no-optional-locks option
  2017-09-22  4:25   ` Jeff King
@ 2017-09-22 20:09     ` Stefan Beller
  2017-09-22 21:25       ` Jeff King
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-22 20:09 UTC (permalink / raw)
  To: Jeff King; +Cc: Johannes Sixt, Johannes Schindelin, git

On Thu, Sep 21, 2017 at 9:25 PM, Jeff King <peff@peff.net> wrote:

>
> But imagine that "git status" learns to recurse into submodules and run
> "git status" inside them. Surely we would want the submodule repos to
> also avoid taking any unnecessary locks?

You can teach Git to recurse into submodules already at home,
just 'git config status.submoduleSummary none'. ;)

It occurs to me that the config name is badly choosen, as it stores
an argument for git status --ignore-submodules[=mode]

^ permalink raw reply	[relevance 17%]

* Re: [PATCH] git: add --no-optional-locks option
  2017-09-22 20:09     ` Stefan Beller
@ 2017-09-22 21:25       ` Jeff King
  2017-09-22 21:41         ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Jeff King @ 2017-09-22 21:25 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Johannes Sixt, Johannes Schindelin, git

On Fri, Sep 22, 2017 at 01:09:32PM -0700, Stefan Beller wrote:

> On Thu, Sep 21, 2017 at 9:25 PM, Jeff King <peff@peff.net> wrote:
> 
> >
> > But imagine that "git status" learns to recurse into submodules and run
> > "git status" inside them. Surely we would want the submodule repos to
> > also avoid taking any unnecessary locks?
> 
> You can teach Git to recurse into submodules already at home,
> just 'git config status.submoduleSummary none'. ;)
> 
> It occurs to me that the config name is badly choosen, as it stores
> an argument for git status --ignore-submodules[=mode]

Ah, thanks. I _thought_ we could already do that but when I went looking
for the standard --recursive option I couldn't find it.

So yes, I would think we would want this option to apply recursively in
that case, even when we cross repository boundaries.

-Peff

^ permalink raw reply	[relevance 8%]

* [PATCH] Documentation/config: clarify the meaning of submodule.<name>.update
@ 2017-09-22 21:28 Stefan Beller
  2017-09-22 21:37 ` Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-22 21:28 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller

With more commands (that potentially change a submodule) paying attention
to submodules as well as the recent discussion[1] on submodule.<name>.update,
let's spell out that submodule.<name>.update is strictly to be used
for configuring the "submodule update" command and not to be obeyed
by other commands.

These other commands usually have a strict meaning of what they should
do (i.e. checkout, reset, rebase, merge) as well as have their name
overlapping with the modes possible for submodule.<name>.update.

[1] https://public-inbox.org/git/4283F0B0-BC1C-4ED1-8126-7E512D84484B@gmail.com/

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/config.txt | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index dc4e3f58a2..b0ded777fe 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -3085,10 +3085,9 @@ submodule.<name>.url::
 	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
 
 submodule.<name>.update::
-	The default update procedure for a submodule. This variable
-	is populated by `git submodule init` from the
-	linkgit:gitmodules[5] file. See description of 'update'
-	command in linkgit:git-submodule[1].
+	The method how a submodule is updated via 'git submodule update'.
+	It is populated by `git submodule init` from the linkgit:gitmodules[5]
+	file. See description of 'update' command in linkgit:git-submodule[1].
 
 submodule.<name>.branch::
 	The remote branch name for a submodule, used by `git submodule
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 32%]

* Re: [PATCH] Documentation/config: clarify the meaning of submodule.<name>.update
  2017-09-22 21:28 [PATCH] Documentation/config: clarify the meaning of submodule.<name>.update Stefan Beller
@ 2017-09-22 21:37 ` Jonathan Nieder
  2017-09-22 22:52   ` [PATCHv2] " Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Jonathan Nieder @ 2017-09-22 21:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Hi,

Stefan Beller wrote:

> With more commands (that potentially change a submodule) paying attention
> to submodules as well as the recent discussion[1] on submodule.<name>.update,
> let's spell out that submodule.<name>.update is strictly to be used
> for configuring the "submodule update" command and not to be obeyed
> by other commands.

Good idea, thank you.

You'll want to update Documentation/gitmodules.txt, too.

I think this can go further: it should say explicitly that commands
like "git checkout --recurse-submodules" do not pay attention to this
option.

> These other commands usually have a strict meaning of what they should
> do (i.e. checkout, reset, rebase, merge) as well as have their name
> overlapping with the modes possible for submodule.<name>.update.
>
> [1] https://public-inbox.org/git/4283F0B0-BC1C-4ED1-8126-7E512D84484B@gmail.com/

Can you summarize what this discussion concluded with so the reader
does not have to look far to understand it?

> Signed-off-by: Stefan Beller <sbeller@google.com>

Reported-by: Lars Schneider <larsxschneider@gmail.com>

> ---
>  Documentation/config.txt | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index dc4e3f58a2..b0ded777fe 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -3085,10 +3085,9 @@ submodule.<name>.url::
>  	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
>  
>  submodule.<name>.update::
> -	The default update procedure for a submodule. This variable
> -	is populated by `git submodule init` from the
> -	linkgit:gitmodules[5] file. See description of 'update'
> -	command in linkgit:git-submodule[1].
> +	The method how a submodule is updated via 'git submodule update'.
> +	It is populated by `git submodule init` from the linkgit:gitmodules[5]
> +	file. See description of 'update' command in linkgit:git-submodule[1].
>  

Wording nits: s/The method how/The method by which/; s/via/by/

More importantly, can this be more explicit about how it is meant to
be used?  E.g. to say

 1. This only affects "git submodule update" and doesn't affect
    commands like "git checkout --recurse-submodules".

 2. It exists for historical reasons; settings like submodule.active
    and pull.rebase are more likely to be what someone is looking for.

Thanks and hope that helps,
Jonathan

^ permalink raw reply	[relevance 25%]

* Re: [PATCH] git: add --no-optional-locks option
  2017-09-22 21:25       ` Jeff King
@ 2017-09-22 21:41         ` Stefan Beller
  2017-09-23  3:34           ` Jeff King
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-22 21:41 UTC (permalink / raw)
  To: Jeff King; +Cc: Johannes Sixt, Johannes Schindelin, git

On Fri, Sep 22, 2017 at 2:25 PM, Jeff King <peff@peff.net> wrote:
> On Fri, Sep 22, 2017 at 01:09:32PM -0700, Stefan Beller wrote:
>
>> On Thu, Sep 21, 2017 at 9:25 PM, Jeff King <peff@peff.net> wrote:
>>
>> >
>> > But imagine that "git status" learns to recurse into submodules and run
>> > "git status" inside them. Surely we would want the submodule repos to
>> > also avoid taking any unnecessary locks?
>>
>> You can teach Git to recurse into submodules already at home,
>> just 'git config status.submoduleSummary none'. ;)
>>
>> It occurs to me that the config name is badly choosen, as it stores
>> an argument for git status --ignore-submodules[=mode]
>
> Ah, thanks. I _thought_ we could already do that but when I went looking
> for the standard --recursive option I couldn't find it.

Thanks for checking for submodules there.

I personally prefer --recurse-submodules despite the longer name,
because a plain recursive option can mean anything in a sufficiently
complex program such as Git (recurse into the tree (c.f. ls-tree), or
for the algorithm used (c.f. merge, diff) or yet another dimension
I did not think of).

> So yes, I would think we would want this option to apply recursively in
> that case, even when we cross repository boundaries.

Regarding the actual patch which is heavily aimed at coping with IDEs
despite the command line being used, I wonder how many IDEs pass
--ignore-submodules and recurse themselves (if needed). Reason for
my suspicion is [1] which does pay attention to submodules:

>    Our application calls status including the following flags:
>    --porcelain=v2 --ignored --untracked-files=all --ignore-submodules=none

[1] https://public-inbox.org/git/2bbb1d0f-ae06-1878-d185-112bd51f75c9@gmail.com/

There might be another option to cope with the situation:

 4. Teach all commands to spinlock / busywait shortly for important
     locks instead of giving up. In that case git-status rewriting
     the index ought to be fine?

Thanks,
Stefan

^ permalink raw reply	[relevance 18%]

* [PATCHv2] Documentation/config: clarify the meaning of submodule.<name>.update
  2017-09-22 21:37 ` Jonathan Nieder
@ 2017-09-22 22:52   ` " Stefan Beller
  2017-09-22 22:58     ` Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-22 22:52 UTC (permalink / raw)
  To: jrnieder; +Cc: git, sbeller

With more commands (that potentially change a submodule) paying attention
to submodules as well as the recent discussion[1] on
submodule.<name>.update, let's spell out that submodule.<name>.update
is strictly to be used for configuring the "submodule update" command
and not to be obeyed by other commands.

These other commands usually have a strict meaning of what they should
do (i.e. checkout, reset, rebase, merge) as well as have their name
overlapping with the modes possible for submodule.<name>.update.

[1] https://public-inbox.org/git/4283F0B0-BC1C-4ED1-8126-7E512D84484B@gmail.com/
    submodule.<name>.update was set to "none", triggering unexpected
    behavior as the submodule was thought to never be touched.
    However a newer version of Git taught 'git pull --rebase' to also
    populate and rebase submodules if they were active.
    The newer options such as submodule.active and command specific
    flags would not have triggered unexpected behavior.

Reported-by: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/config.txt | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

Jonathan writes:
> You'll want to update Documentation/gitmodules.txt, too.

No. /grumpycat

It should already be fine, as I read it as if it is only relevant to 
"git submodule update" there, already:

  submodule.<name>.update::
	Defines the default update procedure for the named submodule,
	i.e. how the submodule is updated by "git submodule update"
	command in the superproject. This is only used by `git
	submodule init` to initialize the configuration variable of
	the same name. Allowed values here are 'checkout', 'rebase',
	'merge' or 'none'. See description of 'update' command in
	linkgit:git-submodule[1] for their meaning. Note that the
	'!command' form is intentionally ignored here for security
	reasons.
 

diff --git a/Documentation/config.txt b/Documentation/config.txt
index dc4e3f58a2..1ac0ae6adb 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -3085,10 +3085,14 @@ submodule.<name>.url::
 	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
 
 submodule.<name>.update::
-	The default update procedure for a submodule. This variable
-	is populated by `git submodule init` from the
-	linkgit:gitmodules[5] file. See description of 'update'
-	command in linkgit:git-submodule[1].
+	The method by which a submodule is updated by 'git submodule update',
+	which is the only affected command, others such as
+	'git checkout --recurse-submodules' are unaffected. It exists for
+	historical reasons, when 'git submodule' was the only command to
+	interact with submodules; settings like `submodule.active`
+	and `pull.rebase` are more specific. It is populated by
+	`git submodule init` from the linkgit:gitmodules[5] file.
+	See description of 'update' command in linkgit:git-submodule[1].
 
 submodule.<name>.branch::
 	The remote branch name for a submodule, used by `git submodule
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 29%]

* Re: [PATCHv2] Documentation/config: clarify the meaning of submodule.<name>.update
  2017-09-22 22:52   ` [PATCHv2] " Stefan Beller
@ 2017-09-22 22:58     ` Jonathan Nieder
  2017-09-23 23:52       ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Jonathan Nieder @ 2017-09-22 22:58 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Stefan Beller wrote:

> Reported-by: Lars Schneider <larsxschneider@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  Documentation/config.txt | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
>
> Jonathan writes:

>> You'll want to update Documentation/gitmodules.txt, too.
>
> No. /grumpycat
>
> It should already be fine, as I read it as if it is only relevant to 
> "git submodule update" there, already:
>
>   submodule.<name>.update::
> 	Defines the default update procedure for the named submodule,
> 	i.e. how the submodule is updated by "git submodule update"
> 	command in the superproject. This is only used by `git
> 	submodule init` to initialize the configuration variable of
> 	the same name. Allowed values here are 'checkout', 'rebase',
> 	'merge' or 'none'. See description of 'update' command in
> 	linkgit:git-submodule[1] for their meaning. Note that the
> 	'!command' form is intentionally ignored here for security
> 	reasons.

I disagree.  Actually, I think the git-config(1) blurb could just
point here, and that the text here ought to be clear about what
commands it affects and how an end user would use it.

^ permalink raw reply	[relevance 15%]

* Re: [PATCH] git: add --no-optional-locks option
  2017-09-22 21:41         ` Stefan Beller
@ 2017-09-23  3:34           ` Jeff King
  0 siblings, 0 replies; 200+ results
From: Jeff King @ 2017-09-23  3:34 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Johannes Sixt, Johannes Schindelin, git

On Fri, Sep 22, 2017 at 02:41:06PM -0700, Stefan Beller wrote:

> > Ah, thanks. I _thought_ we could already do that but when I went looking
> > for the standard --recursive option I couldn't find it.
> 
> Thanks for checking for submodules there.
> 
> I personally prefer --recurse-submodules despite the longer name,
> because a plain recursive option can mean anything in a sufficiently
> complex program such as Git (recurse into the tree (c.f. ls-tree), or
> for the algorithm used (c.f. merge, diff) or yet another dimension
> I did not think of).

Yeah, I agree that's a better name (mentioning --recursive is probably
just showing my general ignorance of submodules).

> > So yes, I would think we would want this option to apply recursively in
> > that case, even when we cross repository boundaries.
> 
> Regarding the actual patch which is heavily aimed at coping with IDEs
> despite the command line being used, I wonder how many IDEs pass
> --ignore-submodules and recurse themselves (if needed). Reason for
> my suspicion is [1] which does pay attention to submodules:
> 
> >    Our application calls status including the following flags:
> >    --porcelain=v2 --ignored --untracked-files=all --ignore-submodules=none
> 
> [1] https://public-inbox.org/git/2bbb1d0f-ae06-1878-d185-112bd51f75c9@gmail.com/

Yeah, I think that's probably Visual Studio (which is what Johannes
presumably wrote the original patch for). GitHub Desktop does not
currently pass it, though perhaps should consider doing so (though of
course the user could tweak their config, too).

> There might be another option to cope with the situation:
> 
>  4. Teach all commands to spinlock / busywait shortly for important
>      locks instead of giving up. In that case git-status rewriting
>      the index ought to be fine?

We do have all the infrastructure in place to do a reasonable busywait
with backoff. I think the patch is roughly just:

diff --git a/read-cache.c b/read-cache.c
index 65f4fe8375..fc1ba122a3 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1563,7 +1563,8 @@ static int read_index_extension(struct index_state *istate,
 
 int hold_locked_index(struct lock_file *lk, int lock_flags)
 {
-	return hold_lock_file_for_update(lk, get_index_file(), lock_flags);
+	return hold_lock_file_for_update_timeout(lk, get_index_file(),
+						 lock_flags, 500);
 }
 
 int read_index(struct index_state *istate)

though I think there are a few sites which manually call
hold_lock_file_for_update() on the index that would need similar
treatment.

I suspect it would work OK in practice, unless your index is so big that
500ms isn't enough. The user may also see minor stalls when there's lock
contention. I'm not sure how annoying that would be.

-Peff

^ permalink raw reply	[relevance 14%]

* Re: [PATCHv2] Documentation/config: clarify the meaning of submodule.<name>.update
  2017-09-22 22:58     ` Jonathan Nieder
@ 2017-09-23 23:52       ` Junio C Hamano
  2017-09-25 19:10         ` [PATCH] Documentation: consolidate " Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2017-09-23 23:52 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: Stefan Beller, git

Jonathan Nieder <jrnieder@gmail.com> writes:

> Stefan Beller wrote:
>
>> Reported-by: Lars Schneider <larsxschneider@gmail.com>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> ---
>>  Documentation/config.txt | 12 ++++++++----
>>  1 file changed, 8 insertions(+), 4 deletions(-)
>>
>> Jonathan writes:
>
>>> You'll want to update Documentation/gitmodules.txt, too.
>>
>> No. /grumpycat
>>
>> It should already be fine, as I read it as if it is only relevant to 
>> "git submodule update" there, already:
>>
>>   submodule.<name>.update::
>> 	Defines the default update procedure for the named submodule,
>> 	i.e. how the submodule is updated by "git submodule update"
>> 	command in the superproject. This is only used by `git
>> 	submodule init` to initialize the configuration variable of
>> 	the same name. Allowed values here are 'checkout', 'rebase',
>> 	'merge' or 'none'. See description of 'update' command in
>> 	linkgit:git-submodule[1] for their meaning. Note that the
>> 	'!command' form is intentionally ignored here for security
>> 	reasons.
>
> I disagree.  Actually, I think the git-config(1) blurb could just
> point here, and that the text here ought to be clear about what
> commands it affects and how an end user would use it.

I tend to agree with the consolidation.  It's not like this patch
makes things worse, but as we've already diverted our attention to
this area and took time to review, if it is a simple update we can
make before we apply, that is better than having to remember and
come back to patch the result of this step further.

In any case, thanks for reporting, attempting to improve and
reviewing, all.

^ permalink raw reply	[relevance 7%]

* [PATCH v5 1/4] submodule--helper: introduce get_submodule_displaypath()
  2017-09-24 12:08 ` [PATCH v5 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
@ 2017-09-24 12:08   ` Prathamesh Chavan
  2017-09-25  3:35     ` Junio C Hamano
  2017-09-24 12:08   ` [PATCH v5 2/4] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 200+ results
From: Prathamesh Chavan @ 2017-09-24 12:08 UTC (permalink / raw)
  To: hanwen; +Cc: christian.couder, git, gitster, pc44800, sbeller

Introduce function get_submodule_displaypath() to replace the code
occurring in submodule_init() for generating displaypath of the
submodule with a call to it.

This new function will also be used in other parts of the system
in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 38 ++++++++++++++++++++++++++------------
 1 file changed, 26 insertions(+), 12 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 818fe74f0..d24ac9028 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -220,6 +220,29 @@ static int resolve_relative_url_test(int argc, const char **argv, const char *pr
 	return 0;
 }
 
+/* the result should be freed by the caller. */
+static char *get_submodule_displaypath(const char *path, const char *prefix)
+{
+	const char *super_prefix = get_super_prefix();
+
+	if (prefix && super_prefix) {
+		BUG("cannot have prefix '%s' and superprefix '%s'",
+		    prefix, super_prefix);
+	} else if (prefix) {
+		struct strbuf sb = STRBUF_INIT;
+		char *displaypath = xstrdup(relative_path(path, prefix, &sb));
+		strbuf_release(&sb);
+		return displaypath;
+	} else if (super_prefix) {
+		int len = strlen(super_prefix);
+		const char *format = (len > 0 && is_dir_sep(super_prefix[len - 1])) ? "%s%s" : "%s/%s";
+
+		return xstrfmt(format, super_prefix, path);
+	} else {
+		return xstrdup(path);
+	}
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -335,15 +358,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	if (prefix && get_super_prefix())
-		die("BUG: cannot have prefix and superprefix");
-	else if (prefix)
-		displaypath = xstrdup(relative_path(path, prefix, &sb));
-	else if (get_super_prefix()) {
-		strbuf_addf(&sb, "%s%s", get_super_prefix(), path);
-		displaypath = strbuf_detach(&sb, NULL);
-	} else
-		displaypath = xstrdup(path);
+	displaypath = get_submodule_displaypath(path, prefix);
 
 	sub = submodule_from_path(&null_oid, path);
 
@@ -358,9 +373,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * Set active flag for the submodule being initialized
 	 */
 	if (!is_submodule_active(the_repository, path)) {
-		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
+		strbuf_reset(&sb);
 	}
 
 	/*
@@ -368,7 +383,6 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * To look up the url in .git/config, we must not fall back to
 	 * .gitmodules, so look it up directly.
 	 */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.url", sub->name);
 	if (git_config_get_string(sb.buf, &url)) {
 		if (!sub->url)
@@ -405,9 +419,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
 	}
+	strbuf_reset(&sb);
 
 	/* Copy "update" setting when it is not set yet */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.update", sub->name);
 	if (git_config_get_string(sb.buf, &upd) &&
 	    sub->update_strategy.type != SM_UPDATE_UNSPECIFIED) {
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [PATCH v5 2/4] submodule--helper: introduce for_each_listed_submodule()
  2017-09-24 12:08 ` [PATCH v5 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
  2017-09-24 12:08   ` [PATCH v5 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
@ 2017-09-24 12:08   ` Prathamesh Chavan
  2017-09-25  3:43     ` Junio C Hamano
  2017-09-24 12:08   ` [PATCH v5 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
  2017-09-24 12:08   ` [PATCH v5 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  3 siblings, 1 reply; 200+ results
From: Prathamesh Chavan @ 2017-09-24 12:08 UTC (permalink / raw)
  To: hanwen; +Cc: christian.couder, git, gitster, pc44800, sbeller

Introduce function for_each_listed_submodule() and replace a loop
in module_init() with a call to it.

The new function will also be used in other parts of the
system in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 36 ++++++++++++++++++++++++++++--------
 1 file changed, 28 insertions(+), 8 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index d24ac9028..d12790b5c 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -14,6 +14,9 @@
 #include "refs.h"
 #include "connect.h"
 
+typedef void (*each_submodule_fn)(const struct cache_entry *list_item,
+				  void *cb_data);
+
 static char *get_default_remote(void)
 {
 	char *dest = NULL, *ret;
@@ -352,15 +355,30 @@ static int module_list(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
-static void init_submodule(const char *path, const char *prefix, int quiet)
+static void for_each_listed_submodule(const struct module_list *list,
+				      each_submodule_fn fn, void *cb_data)
+{
+	int i;
+	for (i = 0; i < list->nr; i++)
+		fn(list->entries[i], cb_data);
+}
+
+struct init_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+};
+#define INIT_CB_INIT { NULL, 0 }
+
+static void init_submodule(const struct cache_entry *list_item, void *cb_data)
 {
+	struct init_cb *info = cb_data;
 	const struct submodule *sub;
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	displaypath = get_submodule_displaypath(path, prefix);
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(&null_oid, list_item->name);
 
 	if (!sub)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
@@ -372,7 +390,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 *
 	 * Set active flag for the submodule being initialized
 	 */
-	if (!is_submodule_active(the_repository, path)) {
+	if (!is_submodule_active(the_repository, list_item->name)) {
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
 		strbuf_reset(&sb);
@@ -414,7 +432,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		if (git_config_set_gently(sb.buf, url))
 			die(_("Failed to register url for submodule path '%s'"),
 			    displaypath);
-		if (!quiet)
+		if (!info->quiet)
 			fprintf(stderr,
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
@@ -443,10 +461,10 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 
 static int module_init(int argc, const char **argv, const char *prefix)
 {
+	struct init_cb info = INIT_CB_INIT;
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
-	int i;
 
 	struct option module_init_options[] = {
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
@@ -471,8 +489,10 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
 
-	for (i = 0; i < list.nr; i++)
-		init_submodule(list.entries[i]->name, prefix, quiet);
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+
+	for_each_listed_submodule(&list, init_submodule, &info);
 
 	return 0;
 }
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [PATCH v5 3/4] submodule: port set_name_rev() from shell to C
  2017-09-24 12:08 ` [PATCH v5 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
  2017-09-24 12:08   ` [PATCH v5 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
  2017-09-24 12:08   ` [PATCH v5 2/4] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
@ 2017-09-24 12:08   ` Prathamesh Chavan
  2017-09-24 12:08   ` [PATCH v5 4/4] submodule: port submodule subcommand 'status' " Prathamesh Chavan
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-09-24 12:08 UTC (permalink / raw)
  To: hanwen; +Cc: christian.couder, git, gitster, pc44800, sbeller

Function set_name_rev() is ported from git-submodule to the
submodule--helper builtin. The function compute_rev_name() generates the
value of the revision name as required.
The function get_rev_name() calls compute_rev_name() and receives the
revision name, and later handles its formatting and printing.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 63 +++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            | 16 ++----------
 2 files changed, 65 insertions(+), 14 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index d12790b5c..7ca8e8153 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -246,6 +246,68 @@ static char *get_submodule_displaypath(const char *path, const char *prefix)
 	}
 }
 
+static char *compute_rev_name(const char *sub_path, const char* object_id)
+{
+	struct strbuf sb = STRBUF_INIT;
+	const char ***d;
+
+	static const char *describe_bare[] = {
+		NULL
+	};
+
+	static const char *describe_tags[] = {
+		"--tags", NULL
+	};
+
+	static const char *describe_contains[] = {
+		"--contains", NULL
+	};
+
+	static const char *describe_all_always[] = {
+		"--all", "--always", NULL
+	};
+
+	static const char **describe_argv[] = {
+		describe_bare, describe_tags, describe_contains,
+		describe_all_always, NULL
+	};
+
+	for (d = describe_argv; *d; d++) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+		prepare_submodule_repo_env(&cp.env_array);
+		cp.dir = sub_path;
+		cp.git_cmd = 1;
+		cp.no_stderr = 1;
+
+		argv_array_push(&cp.args, "describe");
+		argv_array_pushv(&cp.args, *d);
+		argv_array_push(&cp.args, object_id);
+
+		if (!capture_command(&cp, &sb, 0) && sb.len) {
+			strbuf_strip_suffix(&sb, "\n");
+			return strbuf_detach(&sb, NULL);
+		}
+	}
+
+	strbuf_release(&sb);
+	return NULL;
+}
+
+static int get_rev_name(int argc, const char **argv, const char *prefix)
+{
+	char *revname;
+	if (argc != 3)
+		die("get-rev-name only accepts two arguments: <path> <sha1>");
+
+	revname = compute_rev_name(argv[1], argv[2]);
+	if (revname && revname[0])
+		printf(" (%s)", revname);
+	printf("\n");
+
+	free(revname);
+	return 0;
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -1293,6 +1355,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"get-rev-name", get_rev_name, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index 66d1ae8ef..5211361c5 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -758,18 +758,6 @@ cmd_update()
 	}
 }
 
-set_name_rev () {
-	revname=$( (
-		sanitize_submodule_env
-		cd "$1" && {
-			git describe "$2" 2>/dev/null ||
-			git describe --tags "$2" 2>/dev/null ||
-			git describe --contains "$2" 2>/dev/null ||
-			git describe --all --always "$2"
-		}
-	) )
-	test -z "$revname" || revname=" ($revname)"
-}
 #
 # Show commit summary for submodules in index or working tree
 #
@@ -1041,14 +1029,14 @@ cmd_status()
 		fi
 		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
 		then
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
 			say " $sha1 $displaypath$revname"
 		else
 			if test -z "$cached"
 			then
 				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
 			fi
-			set_name_rev "$sm_path" "$sha1"
+			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
 			say "+$sha1 $displaypath$revname"
 		fi
 
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [PATCH v5 4/4] submodule: port submodule subcommand 'status' from shell to C
  2017-09-24 12:08 ` [PATCH v5 0/4] Incremental rewrite of git-submodules Prathamesh Chavan
                     ` (2 preceding siblings ...)
  2017-09-24 12:08   ` [PATCH v5 3/4] submodule: port set_name_rev() from shell to C Prathamesh Chavan
@ 2017-09-24 12:08   ` " Prathamesh Chavan
  3 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-09-24 12:08 UTC (permalink / raw)
  To: hanwen; +Cc: christian.couder, git, gitster, pc44800, sbeller

This aims to make git-submodule 'status' a built-in. Hence, the function
cmd_status() is ported from shell to C. This is done by introducing
three functions: module_status(), submodule_status() and print_status().

The function module_status() acts as the front-end of the subcommand.
It parses subcommand's options and then calls the function
module_list_compute() for computing the list of submodules. Then
this functions calls for_each_listed_submodule() looping through the
list obtained.

Then for_each_listed_submodule() calls submodule_status() for each of the
submodule in its list. The function submodule_status() is responsible
for generating the status each submodule it is called for, and
then calls print_status().

Finally, the function print_status() handles the printing of submodule's
status.

Helper function get_rev_name() removed after porting the above subcommand
as it is no longer used.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 162 +++++++++++++++++++++++++++++++++++++++-----
 git-submodule.sh            |  49 +-------------
 2 files changed, 147 insertions(+), 64 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 7ca8e8153..8876a4a08 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -293,21 +293,6 @@ static char *compute_rev_name(const char *sub_path, const char* object_id)
 	return NULL;
 }
 
-static int get_rev_name(int argc, const char **argv, const char *prefix)
-{
-	char *revname;
-	if (argc != 3)
-		die("get-rev-name only accepts two arguments: <path> <sha1>");
-
-	revname = compute_rev_name(argv[1], argv[2]);
-	if (revname && revname[0])
-		printf(" (%s)", revname);
-	printf("\n");
-
-	free(revname);
-	return 0;
-}
-
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -559,6 +544,151 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+struct status_cb {
+	const char *prefix;
+	unsigned int quiet: 1;
+	unsigned int recursive: 1;
+	unsigned int cached: 1;
+};
+#define STATUS_CB_INIT { NULL, 0, 0, 0 }
+
+static void print_status(struct status_cb *info, char state, const char *path,
+			 const struct object_id *oid, const char *displaypath)
+{
+	if (info->quiet)
+		return;
+
+	printf("%c%s %s", state, oid_to_hex(oid), displaypath);
+
+	if (state == ' ' || state == '+')
+		printf(" (%s)", compute_rev_name(path, oid_to_hex(oid)));
+
+	printf("\n");
+}
+
+static int handle_submodule_head_ref(const char *refname,
+				     const struct object_id *oid, int flags,
+				     void *cb_data)
+{
+	struct object_id *output = cb_data;
+	if (oid)
+		oidcpy(output, oid);
+
+	return 0;
+}
+
+static void status_submodule(const struct cache_entry *list_item, void *cb_data)
+{
+	struct status_cb *info = cb_data;
+	char *displaypath;
+	struct argv_array diff_files_args = ARGV_ARRAY_INIT;
+
+	if (!submodule_from_path(&null_oid, list_item->name))
+		die(_("no submodule mapping found in .gitmodules for path '%s'"),
+		      list_item->name);
+
+	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
+
+	if (ce_stage(list_item)) {
+		print_status(info, 'U', list_item->name,
+			     &null_oid, displaypath);
+		goto cleanup;
+	}
+
+	if (!is_submodule_active(the_repository, list_item->name)) {
+		print_status(info, '-', list_item->name, &list_item->oid,
+			     displaypath);
+		goto cleanup;
+	}
+
+	argv_array_pushl(&diff_files_args, "diff-files",
+			 "--ignore-submodules=dirty", "--quiet", "--",
+			 list_item->name, NULL);
+
+	if (!cmd_diff_files(diff_files_args.argc, diff_files_args.argv,
+			    info->prefix)) {
+		print_status(info, ' ', list_item->name, &list_item->oid,
+			     displaypath);
+	} else {
+		if (!info->cached) {
+			struct object_id oid;
+
+			if (refs_head_ref(get_submodule_ref_store(list_item->name), handle_submodule_head_ref, &oid))
+				die(_("could not resolve HEAD ref inside the"
+				      "submodule '%s'"), list_item->name);
+
+			print_status(info, '+', list_item->name, &oid,
+				     displaypath);
+		} else {
+			print_status(info, '+', list_item->name,
+				     &list_item->oid, displaypath);
+		}
+	}
+
+	if (info->recursive) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = list_item->name;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_pushl(&cpr.args, "--super-prefix", displaypath,
+				 "submodule--helper", "status", "--recursive",
+				 NULL);
+
+		if (info->cached)
+			argv_array_push(&cpr.args, "--cached");
+
+		if (info->quiet)
+			argv_array_push(&cpr.args, "--quiet");
+
+		if (run_command(&cpr))
+			die(_("failed to recurse into submodule '%s'"),
+			      list_item->name);
+	}
+
+cleanup:
+	argv_array_clear(&diff_files_args);
+	free(displaypath);
+}
+
+static int module_status(int argc, const char **argv, const char *prefix)
+{
+	struct status_cb info = STATUS_CB_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+	int quiet = 0;
+	int cached = 0;
+	int recursive = 0;
+
+	struct option module_status_options[] = {
+		OPT__QUIET(&quiet, N_("Suppress submodule status output")),
+		OPT_BOOL(0, "cached", &cached, N_("Use commit stored in the index instead of the one stored in the submodule HEAD")),
+		OPT_BOOL(0, "recursive", &recursive, N_("Recurse into nested submodules")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule status [--quiet] [--cached] [--recursive] [<path>...]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_status_options,
+			     git_submodule_helper_usage, 0);
+
+	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.prefix = prefix;
+	info.quiet = !!quiet;
+	info.recursive = !!recursive;
+	info.cached = !!cached;
+
+	for_each_listed_submodule(&list, status_submodule, &info);
+
+	return 0;
+}
+
 static int module_name(int argc, const char **argv, const char *prefix)
 {
 	const struct submodule *sub;
@@ -1355,8 +1485,8 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
-	{"get-rev-name", get_rev_name, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
+	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
diff --git a/git-submodule.sh b/git-submodule.sh
index 5211361c5..156255a9e 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1004,54 +1004,7 @@ cmd_status()
 		shift
 	done
 
-	{
-		git submodule--helper list --prefix "$wt_prefix" "$@" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path") || exit
-		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-		if test "$stage" = U
-		then
-			say "U$sha1 $displaypath"
-			continue
-		fi
-		if ! git submodule--helper is-active "$sm_path" ||
-		{
-			! test -d "$sm_path"/.git &&
-			! test -f "$sm_path"/.git
-		}
-		then
-			say "-$sha1 $displaypath"
-			continue;
-		fi
-		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
-		then
-			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
-			say " $sha1 $displaypath$revname"
-		else
-			if test -z "$cached"
-			then
-				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
-			fi
-			revname=$(git submodule--helper get-rev-name "$sm_path" "$sha1")
-			say "+$sha1 $displaypath$revname"
-		fi
-
-		if test -n "$recursive"
-		then
-			(
-				prefix="$displaypath/"
-				sanitize_submodule_env
-				wt_prefix=
-				cd "$sm_path" &&
-				eval cmd_status
-			) ||
-			die "$(eval_gettext "Failed to recurse into submodule path '\$sm_path'")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper status ${GIT_QUIET:+--quiet} ${cached:+--cached} ${recursive:+--recursive} "$@"
 }
 #
 # Sync remote urls for submodules
-- 
2.13.0


^ permalink raw reply	[relevance 20%]

* Re: [PATCH v5 1/4] submodule--helper: introduce get_submodule_displaypath()
  2017-09-24 12:08   ` [PATCH v5 1/4] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
@ 2017-09-25  3:35     ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-09-25  3:35 UTC (permalink / raw)
  To: Prathamesh Chavan; +Cc: hanwen, christian.couder, git, sbeller

Prathamesh Chavan <pc44800@gmail.com> writes:

> Introduce function get_submodule_displaypath() to replace the code
> occurring in submodule_init() for generating displaypath of the
> submodule with a call to it.
>
> This new function will also be used in other parts of the system
> in later patches.
>
> Mentored-by: Christian Couder <christian.couder@gmail.com>
> Mentored-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
> ---
>  builtin/submodule--helper.c | 38 ++++++++++++++++++++++++++------------
>  1 file changed, 26 insertions(+), 12 deletions(-)
>
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index 818fe74f0..d24ac9028 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -220,6 +220,29 @@ static int resolve_relative_url_test(int argc, const char **argv, const char *pr
>  	return 0;
>  }
>  
> +/* the result should be freed by the caller. */
> +static char *get_submodule_displaypath(const char *path, const char *prefix)
> +{
> +	const char *super_prefix = get_super_prefix();
> +
> +	if (prefix && super_prefix) {
> +		BUG("cannot have prefix '%s' and superprefix '%s'",
> +		    prefix, super_prefix);
> +	} else if (prefix) {
> +		struct strbuf sb = STRBUF_INIT;
> +		char *displaypath = xstrdup(relative_path(path, prefix, &sb));
> +		strbuf_release(&sb);
> +		return displaypath;
> +	} else if (super_prefix) {
> +		int len = strlen(super_prefix);
> +		const char *format = (len > 0 && is_dir_sep(super_prefix[len - 1])) ? "%s%s" : "%s/%s";
> +
> +		return xstrfmt(format, super_prefix, path);
> +	} else {
> +		return xstrdup(path);
> +	}
> +}

Looks like a fairly faithful rewrite of the original below, with a
reasonable clean-up (e.g. use of xstrfmt() instead of addf()).

One thing I noticed is that future callers of this function, unlike
init_submodule() which is its original caller, are allowed to have
super-prefix that does not end in a slash, because the helper
automatically supplies one when it is missing.

I am not sure the added leniency is desirable [*1*], but in any
case, the added leniency deserves a mention in the log message, I
would think.


[Footnote]

*1* Often it is harder to introduce bugs when the internal rules are
stricter, e.g. "prefix must end with slash", than when they are
looser, e.g. "prefix may or may not end with slash", because
allowing different codepaths to have the same thing in different
representations would hide bugs that is only uncovered when these
codepaths eventually meet (e.g. one codepath has a path with and the
other without trailing slash and they both think that is the
prefix---then they use strcmp() to see if they have the same prefix,
which would be a bug, which may not be noticed for a long time).

>  struct module_list {
>  	const struct cache_entry **entries;
>  	int alloc, nr;
> @@ -335,15 +358,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>  	struct strbuf sb = STRBUF_INIT;
>  	char *upd = NULL, *url = NULL, *displaypath;
>  
> -	if (prefix && get_super_prefix())
> -		die("BUG: cannot have prefix and superprefix");
> -	else if (prefix)
> -		displaypath = xstrdup(relative_path(path, prefix, &sb));
> -	else if (get_super_prefix()) {
> -		strbuf_addf(&sb, "%s%s", get_super_prefix(), path);
> -		displaypath = strbuf_detach(&sb, NULL);
> -	} else
> -		displaypath = xstrdup(path);
> +	displaypath = get_submodule_displaypath(path, prefix);
>  
>  	sub = submodule_from_path(&null_oid, path);
>  
> @@ -358,9 +373,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>  	 * Set active flag for the submodule being initialized
>  	 */
>  	if (!is_submodule_active(the_repository, path)) {
> -		strbuf_reset(&sb);
>  		strbuf_addf(&sb, "submodule.%s.active", sub->name);
>  		git_config_set_gently(sb.buf, "true");
> +		strbuf_reset(&sb);
>  	}
>  
>  	/*
> @@ -368,7 +383,6 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>  	 * To look up the url in .git/config, we must not fall back to
>  	 * .gitmodules, so look it up directly.
>  	 */
> -	strbuf_reset(&sb);
>  	strbuf_addf(&sb, "submodule.%s.url", sub->name);
>  	if (git_config_get_string(sb.buf, &url)) {
>  		if (!sub->url)
> @@ -405,9 +419,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>  				_("Submodule '%s' (%s) registered for path '%s'\n"),
>  				sub->name, url, displaypath);
>  	}
> +	strbuf_reset(&sb);
>  
>  	/* Copy "update" setting when it is not set yet */
> -	strbuf_reset(&sb);
>  	strbuf_addf(&sb, "submodule.%s.update", sub->name);
>  	if (git_config_get_string(sb.buf, &upd) &&
>  	    sub->update_strategy.type != SM_UPDATE_UNSPECIFIED) {

^ permalink raw reply	[relevance 7%]

* Re: [PATCH v5 2/4] submodule--helper: introduce for_each_listed_submodule()
  2017-09-24 12:08   ` [PATCH v5 2/4] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
@ 2017-09-25  3:43     ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-09-25  3:43 UTC (permalink / raw)
  To: Prathamesh Chavan; +Cc: hanwen, christian.couder, git, sbeller

Prathamesh Chavan <pc44800@gmail.com> writes:

> Introduce function for_each_listed_submodule() and replace a loop
> in module_init() with a call to it.
>
> The new function will also be used in other parts of the
> system in later patches.
>
> Mentored-by: Christian Couder <christian.couder@gmail.com>
> Mentored-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
> ---
>  builtin/submodule--helper.c | 36 ++++++++++++++++++++++++++++--------
>  1 file changed, 28 insertions(+), 8 deletions(-)

This looks more or less perfectly done, but I say this basing it on
an assumption that nobody ever would want to initialize a single
submodule without going through module_init() interface.

If there will be a caller that would have made a direct call to
init_submodule() if this patch did not exist, then I think this
patch is better done by keeping the external interface to the
init_submodule() function intact, and introducing an intermediate
helper init_submodule_cb() function that is to be used as the
callback used in for_each_listed_submodule() API.  In general, we
should make sure that these callback functions that take "void
*cb_data" parameters are called _only_ by these iterators that
defined the callback (in this case, for_each_listed_submodule())
and not other functions.

> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index d24ac9028..d12790b5c 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -14,6 +14,9 @@
>  #include "refs.h"
>  #include "connect.h"
>  
> +typedef void (*each_submodule_fn)(const struct cache_entry *list_item,
> +				  void *cb_data);
> +
>  static char *get_default_remote(void)
>  {
>  	char *dest = NULL, *ret;
> @@ -352,15 +355,30 @@ static int module_list(int argc, const char **argv, const char *prefix)
>  	return 0;
>  }
>  
> -static void init_submodule(const char *path, const char *prefix, int quiet)
> +static void for_each_listed_submodule(const struct module_list *list,
> +				      each_submodule_fn fn, void *cb_data)
> +{
> +	int i;
> +	for (i = 0; i < list->nr; i++)
> +		fn(list->entries[i], cb_data);
> +}
> +
> +struct init_cb {
> +	const char *prefix;
> +	unsigned int quiet: 1;
> +};
> +#define INIT_CB_INIT { NULL, 0 }
> +
> +static void init_submodule(const struct cache_entry *list_item, void *cb_data)
>  {
> +	struct init_cb *info = cb_data;
>  	const struct submodule *sub;
>  	struct strbuf sb = STRBUF_INIT;
>  	char *upd = NULL, *url = NULL, *displaypath;
>  
> -	displaypath = get_submodule_displaypath(path, prefix);
> +	displaypath = get_submodule_displaypath(list_item->name, info->prefix);
>  
> -	sub = submodule_from_path(&null_oid, path);
> +	sub = submodule_from_path(&null_oid, list_item->name);
>  
>  	if (!sub)
>  		die(_("No url found for submodule path '%s' in .gitmodules"),
> @@ -372,7 +390,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>  	 *
>  	 * Set active flag for the submodule being initialized
>  	 */
> -	if (!is_submodule_active(the_repository, path)) {
> +	if (!is_submodule_active(the_repository, list_item->name)) {
>  		strbuf_addf(&sb, "submodule.%s.active", sub->name);
>  		git_config_set_gently(sb.buf, "true");
>  		strbuf_reset(&sb);
> @@ -414,7 +432,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>  		if (git_config_set_gently(sb.buf, url))
>  			die(_("Failed to register url for submodule path '%s'"),
>  			    displaypath);
> -		if (!quiet)
> +		if (!info->quiet)
>  			fprintf(stderr,
>  				_("Submodule '%s' (%s) registered for path '%s'\n"),
>  				sub->name, url, displaypath);
> @@ -443,10 +461,10 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>  
>  static int module_init(int argc, const char **argv, const char *prefix)
>  {
> +	struct init_cb info = INIT_CB_INIT;
>  	struct pathspec pathspec;
>  	struct module_list list = MODULE_LIST_INIT;
>  	int quiet = 0;
> -	int i;
>  
>  	struct option module_init_options[] = {
>  		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
> @@ -471,8 +489,10 @@ static int module_init(int argc, const char **argv, const char *prefix)
>  	if (!argc && git_config_get_value_multi("submodule.active"))
>  		module_list_active(&list);
>  
> -	for (i = 0; i < list.nr; i++)
> -		init_submodule(list.entries[i]->name, prefix, quiet);
> +	info.prefix = prefix;
> +	info.quiet = !!quiet;
> +
> +	for_each_listed_submodule(&list, init_submodule, &info);
>  
>  	return 0;
>  }

^ permalink raw reply	[relevance 14%]

* Re: What's cooking in git.git (Sep 2017, #04; Mon, 25)
  2017-09-25  7:25 What's cooking in git.git (Sep 2017, #04; Mon, 25) Junio C Hamano
@ 2017-09-25 18:30 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-09-25 18:30 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git

>
> * sb/parse-options-blank-line-before-option-list (2017-08-25) 1 commit
>  - usage_with_options: omit double new line on empty option list
>
>  "git worktree" with no option and no subcommand showed too many
>  blank lines in its help text, which has been reduced.
>
>  Superseded by bc/rev-parse-parseopt-fix?

yes it is, please retract this one.


> * sb/submodule-recursive-checkout-detach-head (2017-07-28) 2 commits
>  - Documentation/checkout: clarify submodule HEADs to be detached
>  - recursive submodules: detach HEAD from new state
>
>  "git checkout --recursive" may overwrite and rewind the history of
>  the branch that happens to be checked out in submodule
>  repositories, which might not be desirable.  Detach the HEAD but
>  still allow the recursive checkout to succeed in such a case.
>
>  Undecided.
>  This needs justification in a larger picture; it is unclear why
>  this is better than rejecting recursive checkout, for example.

I'll revisit this with fresh eyes, the original plan was to get
jn/per-repo-obj-store done to have more fundamental operations
available to do the right thing, whatever it is, though.

>
> * jn/per-repo-obj-store (2017-09-07) 39 commits

We intend to get this one rolling again.

^ permalink raw reply	[relevance 8%]

* [PATCH] Documentation: consolidate submodule.<name>.update
  2017-09-23 23:52       ` Junio C Hamano
@ 2017-09-25 19:10         ` " Stefan Beller
  2017-09-25 19:17           ` Brandon Williams
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-25 19:10 UTC (permalink / raw)
  To: gitster; +Cc: git, jrnieder, sbeller

Have one place to explain the effects of setting submodule.<name>.update
instead of two.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
>> I disagree.  Actually, I think the git-config(1) blurb could just
>> point here, and that the text here ought to be clear about what
>> commands it affects and how an end user would use it.
>
> I tend to agree with the consolidation.

Something like this?

 Documentation/config.txt     |  9 +--------
 Documentation/gitmodules.txt | 20 +++++++++++---------
 2 files changed, 12 insertions(+), 17 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 1ac0ae6adb..0d5a296b6c 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -3085,14 +3085,7 @@ submodule.<name>.url::
 	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
 
 submodule.<name>.update::
-	The method by which a submodule is updated by 'git submodule update',
-	which is the only affected command, others such as
-	'git checkout --recurse-submodules' are unaffected. It exists for
-	historical reasons, when 'git submodule' was the only command to
-	interact with submodules; settings like `submodule.active`
-	and `pull.rebase` are more specific. It is populated by
-	`git submodule init` from the linkgit:gitmodules[5] file.
-	See description of 'update' command in linkgit:git-submodule[1].
+	See `submodule.<name>.update` in linkgit:gitmodules[5].
 
 submodule.<name>.branch::
 	The remote branch name for a submodule, used by `git submodule
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index db5d47eb19..d156dee387 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt
@@ -38,15 +38,17 @@ submodule.<name>.url::
 In addition, there are a number of optional keys:
 
 submodule.<name>.update::
-	Defines the default update procedure for the named submodule,
-	i.e. how the submodule is updated by "git submodule update"
-	command in the superproject. This is only used by `git
-	submodule init` to initialize the configuration variable of
-	the same name. Allowed values here are 'checkout', 'rebase',
-	'merge' or 'none'. See description of 'update' command in
-	linkgit:git-submodule[1] for their meaning. Note that the
-	'!command' form is intentionally ignored here for security
-	reasons.
+	The method by which a submodule is updated by 'git submodule update',
+	which is the only affected command, others such as
+	'git checkout --recurse-submodules' are unaffected. It exists for
+	historical reasons, when 'git submodule' was the only command to
+	interact with submodules; settings like `submodule.active`
+	and `pull.rebase` are more specific. It is copied to the config
+	by `git submodule init` from the .gitmodules file.
+	Allowed values here are 'checkout', 'rebase', 'merge' or 'none'.
+	See description of 'update' command in linkgit:git-submodule[1]
+	for their meaning. Note that the '!command' form is intentionally
+	ignored here for security reasons.
 
 submodule.<name>.branch::
 	A remote branch name for tracking updates in the upstream submodule.
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 28%]

* Re: [PATCH] Documentation: consolidate submodule.<name>.update
  2017-09-25 19:10         ` [PATCH] Documentation: consolidate " Stefan Beller
@ 2017-09-25 19:17           ` Brandon Williams
  2017-09-25 22:18             ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Brandon Williams @ 2017-09-25 19:17 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, jrnieder

On 09/25, Stefan Beller wrote:
> Have one place to explain the effects of setting submodule.<name>.update
> instead of two.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> >> I disagree.  Actually, I think the git-config(1) blurb could just
> >> point here, and that the text here ought to be clear about what
> >> commands it affects and how an end user would use it.
> >
> > I tend to agree with the consolidation.
> 
> Something like this?

I like the consolidation, its easier to keep up to date when its only in
one place.

> 
>  Documentation/config.txt     |  9 +--------
>  Documentation/gitmodules.txt | 20 +++++++++++---------
>  2 files changed, 12 insertions(+), 17 deletions(-)
> 
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 1ac0ae6adb..0d5a296b6c 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -3085,14 +3085,7 @@ submodule.<name>.url::
>  	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
>  
>  submodule.<name>.update::
> -	The method by which a submodule is updated by 'git submodule update',
> -	which is the only affected command, others such as
> -	'git checkout --recurse-submodules' are unaffected. It exists for
> -	historical reasons, when 'git submodule' was the only command to
> -	interact with submodules; settings like `submodule.active`
> -	and `pull.rebase` are more specific. It is populated by
> -	`git submodule init` from the linkgit:gitmodules[5] file.
> -	See description of 'update' command in linkgit:git-submodule[1].
> +	See `submodule.<name>.update` in linkgit:gitmodules[5].
>  
>  submodule.<name>.branch::
>  	The remote branch name for a submodule, used by `git submodule
> diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
> index db5d47eb19..d156dee387 100644
> --- a/Documentation/gitmodules.txt
> +++ b/Documentation/gitmodules.txt
> @@ -38,15 +38,17 @@ submodule.<name>.url::
>  In addition, there are a number of optional keys:
>  
>  submodule.<name>.update::
> -	Defines the default update procedure for the named submodule,
> -	i.e. how the submodule is updated by "git submodule update"
> -	command in the superproject. This is only used by `git
> -	submodule init` to initialize the configuration variable of
> -	the same name. Allowed values here are 'checkout', 'rebase',
> -	'merge' or 'none'. See description of 'update' command in
> -	linkgit:git-submodule[1] for their meaning. Note that the
> -	'!command' form is intentionally ignored here for security
> -	reasons.
> +	The method by which a submodule is updated by 'git submodule update',
> +	which is the only affected command, others such as
> +	'git checkout --recurse-submodules' are unaffected. It exists for
> +	historical reasons, when 'git submodule' was the only command to
> +	interact with submodules; settings like `submodule.active`
> +	and `pull.rebase` are more specific. It is copied to the config
> +	by `git submodule init` from the .gitmodules file.
> +	Allowed values here are 'checkout', 'rebase', 'merge' or 'none'.
> +	See description of 'update' command in linkgit:git-submodule[1]
> +	for their meaning. Note that the '!command' form is intentionally
> +	ignored here for security reasons.

This probably needs to be tweaked a bit to say that the '!command' form
is ignored by submodule init, in that it isn't copied over from the
.gitmodules file, but if it is configured in your config it will be
respected by 'submodule update'.

>  
>  submodule.<name>.branch::
>  	A remote branch name for tracking updates in the upstream submodule.
> -- 
> 2.14.0.rc0.3.g6c2e499285
> 

-- 
Brandon Williams

^ permalink raw reply	[relevance 24%]

* [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
@ 2017-09-25 19:55 Stefan Beller
  2017-09-25 20:04 ` Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-25 19:55 UTC (permalink / raw)
  To: git; +Cc: jrnieder, Stefan Beller

submodule.<name>.update can be assigned an arbitrary command via setting
it to "!command". When this command is found in the regular config, Git
ought to just run that command instead of other update mechanisms.

However if that command is just found in the .gitmodules file, it is
potentially untrusted, which is why we do not run it.  Add a test
confirming the behavior.

Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 t/t7406-submodule-update.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 034914a14f..780af4e6f5 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -406,6 +406,16 @@ test_expect_success 'submodule update - command in .git/config' '
 	)
 '
 
+test_expect_success 'submodule update - command in .gitmodules is ignored' '
+	test_when_finished "git -C super reset --hard HEAD^" &&
+
+	git -C super config -f .gitmodules submodule.submodule.update "!false || echo >bad" &&
+	git -C super commit -a -m "add command to .gitmodules file" &&
+	git -C super/submodule reset --hard $submodulesha1^ &&
+	git -C super submodule update submodule 2>../actual &&
+	test_path_is_missing super/bad
+'
+
 cat << EOF >expect
 Execution of 'false $submodulesha1' failed in submodule path 'submodule'
 EOF
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 33%]

* Re: [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
  2017-09-25 19:55 [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules Stefan Beller
@ 2017-09-25 20:04 ` Jonathan Nieder
  2017-09-25 22:50   ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Jonathan Nieder @ 2017-09-25 20:04 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Stefan Beller wrote:

> submodule.<name>.update can be assigned an arbitrary command via setting
> it to "!command". When this command is found in the regular config, Git
> ought to just run that command instead of other update mechanisms.
> 
> However if that command is just found in the .gitmodules file, it is
> potentially untrusted, which is why we do not run it.  Add a test
> confirming the behavior.
> 
> Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  t/t7406-submodule-update.sh | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
> index 034914a14f..780af4e6f5 100755
> --- a/t/t7406-submodule-update.sh
> +++ b/t/t7406-submodule-update.sh
> @@ -406,6 +406,16 @@ test_expect_success 'submodule update - command in .git/config' '
>  	)
>  '
>  
> +test_expect_success 'submodule update - command in .gitmodules is ignored' '
> +	test_when_finished "git -C super reset --hard HEAD^" &&
> +
> +	git -C super config -f .gitmodules submodule.submodule.update "!false || echo >bad" &&

What does the '!false || echo >bad' do?

Ideally we want this test to be super robust: e.g. if it runs the
command but from a different directory, we still want the test to fail,
and if it runs the command but using exec instead of a shell, we still
want the test to fail.

Maybe write_script would help with this.  E.g. would something like

	test_when_finished ... &&
	write_script must_not_run.sh <<-EOF &&
	>$TEST_DIRECTORY/bad
	EOF

	git -C super config -f .gitmodules submodule.submodule.update \
		"!$TEST_DIRECTORY/must_not_run.sh" &&
	...

work?

Thanks,
Jonathan

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] Documentation: consolidate submodule.<name>.update
  2017-09-25 19:17           ` Brandon Williams
@ 2017-09-25 22:18             ` Stefan Beller
  2017-09-25 22:22               ` Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-25 22:18 UTC (permalink / raw)
  To: Brandon Williams; +Cc: Junio C Hamano, git, Jonathan Nieder

On Mon, Sep 25, 2017 at 12:17 PM, Brandon Williams <bmwill@google.com> wrote:
> On 09/25, Stefan Beller wrote:
>> Have one place to explain the effects of setting submodule.<name>.update
>> instead of two.
>>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> ---
>> >> I disagree.  Actually, I think the git-config(1) blurb could just
>> >> point here, and that the text here ought to be clear about what
>> >> commands it affects and how an end user would use it.
>> >
>> > I tend to agree with the consolidation.
>>
>> Something like this?
>
> I like the consolidation, its easier to keep up to date when its only in
> one place.

After thinking about it further, I'd like to withdraw this patch
for now.

The reason is that this would also forward point to
git-submodule.txt, such that we'd still have 2 places
in which it is explained. The current situation with explaining
it in 3 places is not too bad as each place hints at their specific
circumstance:
git-submodule.txt explains the values to set itself.
gitmodules.txt explains what the possible fields in that file are,
  and regarding this field it points out it is ignored in the init call.
config.txt: actually describe the effects of the setting.

I think we can keep it as-is for now.

Thanks,
Stefan

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] Documentation: consolidate submodule.<name>.update
  2017-09-25 22:18             ` Stefan Beller
@ 2017-09-25 22:22               ` Jonathan Nieder
  2017-09-25 22:42                 ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Jonathan Nieder @ 2017-09-25 22:22 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Brandon Williams, Junio C Hamano, git

Stefan Beller wrote:
> On Mon, Sep 25, 2017 at 12:17 PM, Brandon Williams <bmwill@google.com> wrote:
>> On 09/25, Stefan Beller wrote:

>>> Have one place to explain the effects of setting submodule.<name>.update
>>> instead of two.
>>>
>>> Signed-off-by: Stefan Beller <sbeller@google.com>
>>> ---
>>>>> I disagree.  Actually, I think the git-config(1) blurb could just
>>>>> point here, and that the text here ought to be clear about what
>>>>> commands it affects and how an end user would use it.
>>>>
>>>> I tend to agree with the consolidation.
>>>
>>> Something like this?
>>
>> I like the consolidation, its easier to keep up to date when its only in
>> one place.
>
> After thinking about it further, I'd like to withdraw this patch
> for now.
>
> The reason is that this would also forward point to
> git-submodule.txt, such that we'd still have 2 places
> in which it is explained. The current situation with explaining
> it in 3 places is not too bad as each place hints at their specific
> circumstance:
> git-submodule.txt explains the values to set itself.
> gitmodules.txt explains what the possible fields in that file are,
>   and regarding this field it points out it is ignored in the init call.
> config.txt: actually describe the effects of the setting.
>
> I think we can keep it as-is for now.

Sorry, I got lost.  Which state is as-is?

As a user, how do I find out what submodule.*.update is going to do
and which commands will respect it?

Maybe we should work on first wordsmithing the doc and then figuring
out where it goes?  The current state of the doc with (?) three
different texts, all wrong, doesn't seem like a good state to
preserve.

Thanks,
Jonathan

^ permalink raw reply	[relevance 22%]

* Re: [PATCH] Documentation: consolidate submodule.<name>.update
  2017-09-25 22:22               ` Jonathan Nieder
@ 2017-09-25 22:42                 ` Stefan Beller
  2017-09-25 23:44                   ` Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-25 22:42 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: Brandon Williams, Junio C Hamano, git

On Mon, Sep 25, 2017 at 3:22 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Stefan Beller wrote:
>> On Mon, Sep 25, 2017 at 12:17 PM, Brandon Williams <bmwill@google.com> wrote:
>>> On 09/25, Stefan Beller wrote:
>
>>>> Have one place to explain the effects of setting submodule.<name>.update
>>>> instead of two.
>>>>
>>>> Signed-off-by: Stefan Beller <sbeller@google.com>
>>>> ---
>>>>>> I disagree.  Actually, I think the git-config(1) blurb could just
>>>>>> point here, and that the text here ought to be clear about what
>>>>>> commands it affects and how an end user would use it.
>>>>>
>>>>> I tend to agree with the consolidation.
>>>>
>>>> Something like this?
>>>
>>> I like the consolidation, its easier to keep up to date when its only in
>>> one place.
>>
>> After thinking about it further, I'd like to withdraw this patch
>> for now.
>>
>> The reason is that this would also forward point to
>> git-submodule.txt, such that we'd still have 2 places
>> in which it is explained. The current situation with explaining
>> it in 3 places is not too bad as each place hints at their specific
>> circumstance:
>> git-submodule.txt explains the values to set itself.
>> gitmodules.txt explains what the possible fields in that file are,
>>   and regarding this field it points out it is ignored in the init call.
>> config.txt: actually describe the effects of the setting.
>>
>> I think we can keep it as-is for now.
>
> Sorry, I got lost.  Which state is as-is?

By as-is I refer to origin/pu.

> As a user, how do I find out what submodule.*.update is going to do
> and which commands will respect it?

The user would discover it via 'man git-config' I would assume, which
covers any config variable? It also directs to git-submodule which is
more detailed, but the text there is suitable for the casual reader.
(pu has origin/sb/doc-config-submodule-update)

> Maybe we should work on first wordsmithing the doc and then figuring
> out where it goes?  The current state of the doc with (?) three
> different texts,

such that each different text highlights each locations purpose.

> all wrong,

Care to spell out this bold claim?

Thanks,
Stefan

^ permalink raw reply	[relevance 24%]

* [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
  2017-09-25 20:04 ` Jonathan Nieder
@ 2017-09-25 22:50   ` Stefan Beller
  2017-09-26  0:01     ` Jonathan Nieder
  2017-09-26  5:37     ` Johannes Sixt
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2017-09-25 22:50 UTC (permalink / raw)
  To: jrnieder; +Cc: git, sbeller

submodule.<name>.update can be assigned an arbitrary command via setting
it to "!command". When this command is found in the regular config, Git
ought to just run that command instead of other update mechanisms.

However if that command is just found in the .gitmodules file, it is
potentially untrusted, which is why we do not run it.  Add a test
confirming the behavior.

Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---

 updated to use the super robust script.
 Thanks Jonathan,
 
 Stefan

 t/t7406-submodule-update.sh | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 034914a14f..d718cb00e7 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -406,6 +406,20 @@ test_expect_success 'submodule update - command in .git/config' '
 	)
 '
 
+test_expect_success 'submodule update - command in .gitmodules is ignored' '
+	test_when_finished "git -C super reset --hard HEAD^" &&
+
+	write_script must_not_run.sh <<-EOF &&
+	>$TEST_DIRECTORY/bad
+	EOF
+
+	git -C super config -f .gitmodules submodule.submodule.update "!$TEST_DIRECTORY/must_not_run.sh" &&
+	git -C super commit -a -m "add command to .gitmodules file" &&
+	git -C super/submodule reset --hard $submodulesha1^ &&
+	git -C super submodule update submodule &&
+	test_path_is_missing bad
+'
+
 cat << EOF >expect
 Execution of 'false $submodulesha1' failed in submodule path 'submodule'
 EOF
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 33%]

* Re: [PATCH] Documentation: consolidate submodule.<name>.update
  2017-09-25 22:42                 ` Stefan Beller
@ 2017-09-25 23:44                   ` Jonathan Nieder
  0 siblings, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-09-25 23:44 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Brandon Williams, Junio C Hamano, git

Stefan Beller wrote:

> By as-is I refer to origin/pu.

Ah, okay.  I'm not in that habit because pu frequently loses topics
--- it's mostly useful as a tool to (1) distribute topic branches and
(2) check whether they are compatible with each other.

[...]
> On Mon, Sep 25, 2017 at 3:22 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> Maybe we should work on first wordsmithing the doc and then figuring
>> out where it goes?  The current state of the doc with (?) three
>> different texts,
>
> such that each different text highlights each locations purpose.
>
>> all wrong,
>
> Care to spell out this bold claim?

In the state of "master", "man git-config" tells me that

	[submodule "<name>"]
		update = ...

sets the "default update procedure for a submodule".  It suggests that
I read about "git submodule update" in git-submodule(1) for more
details.  This is problematic because

 1. It doesn't tell me when I should and should not set it.
 2. I would expect "git checkout --recurse-submodules" to respect the
    default update procedure.
 3. It doesn't tell me what commands like "git fetch
    --recurse-submodules" will do.
 4. It doesn't point me to better alternatives, when this
    configuration doesn't fit my use case.

"man git-submodule" doesn't have a section on submodule.<name>.update.
It has references scattered throughout the manpage.  It only tells me
how the setting affects the "git submodule update" command and doesn't
address the problems (1)-(4).

"man gitmodules" also refers to this setting as the "default update
procedure for the named submodule".  It doesn't answer the questions
(1)-(4) either.

Thanks and hope that helps,
Jonathan

^ permalink raw reply	[relevance 26%]

* Re: [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
  2017-09-25 22:50   ` Stefan Beller
@ 2017-09-26  0:01     ` Jonathan Nieder
  2017-09-26  5:37     ` Johannes Sixt
  1 sibling, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-09-26  0:01 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Stefan Beller wrote:

> submodule.<name>.update can be assigned an arbitrary command via setting
> it to "!command". When this command is found in the regular config, Git
> ought to just run that command instead of other update mechanisms.
>
> However if that command is just found in the .gitmodules file, it is
> potentially untrusted, which is why we do not run it.  Add a test
> confirming the behavior.
>
> Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  t/t7406-submodule-update.sh | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
> index 034914a14f..d718cb00e7 100755
> --- a/t/t7406-submodule-update.sh
> +++ b/t/t7406-submodule-update.sh
> @@ -406,6 +406,20 @@ test_expect_success 'submodule update - command in .git/config' '
>  	)
>  '
>  
> +test_expect_success 'submodule update - command in .gitmodules is ignored' '
> +	test_when_finished "git -C super reset --hard HEAD^" &&
> +
> +	write_script must_not_run.sh <<-EOF &&
> +	>$TEST_DIRECTORY/bad
> +	EOF
> +
> +	git -C super config -f .gitmodules submodule.submodule.update "!$TEST_DIRECTORY/must_not_run.sh" &&

Long line, but I don't think I care.  I wish there were a tool like
"make style" to format shell scripts.

> +	git -C super commit -a -m "add command to .gitmodules file" &&
> +	git -C super/submodule reset --hard $submodulesha1^ &&
> +	git -C super submodule update submodule &&
> +	test_path_is_missing bad
> +'

Per offline discussion, you tested that this fails when you use
.git/config instead of .gitmodules, so there aren't any subtle typos
here. :)

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Thanks for writing it.

^ permalink raw reply	[relevance 15%]

* Re: [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
  2017-09-25 22:50   ` Stefan Beller
  2017-09-26  0:01     ` Jonathan Nieder
@ 2017-09-26  5:37     ` Johannes Sixt
  1 sibling, 0 replies; 200+ results
From: Johannes Sixt @ 2017-09-26  5:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: jrnieder, git

Am 26.09.2017 um 00:50 schrieb Stefan Beller:
> submodule.<name>.update can be assigned an arbitrary command via setting
> it to "!command". When this command is found in the regular config, Git
> ought to just run that command instead of other update mechanisms.
> 
> However if that command is just found in the .gitmodules file, it is
> potentially untrusted, which is why we do not run it.  Add a test
> confirming the behavior.
> 
> Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> 
>   updated to use the super robust script.
>   Thanks Jonathan,
>   
>   Stefan
> 
>   t/t7406-submodule-update.sh | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 
> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
> index 034914a14f..d718cb00e7 100755
> --- a/t/t7406-submodule-update.sh
> +++ b/t/t7406-submodule-update.sh
> @@ -406,6 +406,20 @@ test_expect_success 'submodule update - command in .git/config' '
>   	)
>   '
>   
> +test_expect_success 'submodule update - command in .gitmodules is ignored' '
> +	test_when_finished "git -C super reset --hard HEAD^" &&
> +
> +	write_script must_not_run.sh <<-EOF &&
> +	>$TEST_DIRECTORY/bad
> +	EOF

I am pretty confident that this does not test what you intend to test. 
Notice that $TEST_DIRECTORY is expanded when the script is written. But 
that path contains a blank, and we have something like this in the test 
script:

	#!/bin/sh
	>/the/build/directory/t/trash directory.t7406/bad

If you inject the bug against which this test protects into 
git-submodule, you should find a file "trash" in your t directory, and 
the file "bad" still absent. Not to mention that the script fails 
because it cannot run "directory.t7406/bad".

To fix that, you should use and exported variable and access that from 
the test script, for example:

	write_script must_not_run.sh <<-\EOF &&
	>"$TEST_DIRECTORY"/bad
	EOF
...
	(
		export TEST_DIRECTORY &&
		git -C super submodule update submodule
	) &&
	test_path_is_missing bad

> +
> +	git -C super config -f .gitmodules submodule.submodule.update "!$TEST_DIRECTORY/must_not_run.sh" &&
> +	git -C super commit -a -m "add command to .gitmodules file" &&
> +	git -C super/submodule reset --hard $submodulesha1^ &&
> +	git -C super submodule update submodule &&
> +	test_path_is_missing bad
> +'
> +
>   cat << EOF >expect
>   Execution of 'false $submodulesha1' failed in submodule path 'submodule'
>   EOF
> 

-- Hannes

^ permalink raw reply	[relevance 25%]

* [ANNOUNCE] Git v2.14.2
@ 2017-09-26  6:09 Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-09-26  6:09 UTC (permalink / raw)
  To: git; +Cc: Linux Kernel

The latest maintenance release Git v2.14.2 is now available at the
usual places.  In addition to the cvsserver related fixes that
appear in 2.10.5, 2.11.4, 2.12.5 and 2.13.6 announced separately,
this also includes various fixes that were merged already to the
'master' branch.

The tarballs are found at:

    https://www.kernel.org/pub/software/scm/git/

The following public repositories all have a copy of the 'v2.14.2'
tag and the 'maint' branch that the tag points at:

  url = https://kernel.googlesource.com/pub/scm/git/git
  url = git://repo.or.cz/alt-git.git
  url = https://github.com/gitster/git

----------------------------------------------------------------

Git v2.14.2 Release Notes
=========================

Fixes since v2.14.1
-------------------

 * Because recent Git for Windows do come with a real msgfmt, the
   build procedure for git-gui has been updated to use it instead of a
   hand-rolled substitute.

 * "%C(color name)" in the pretty print format always produced ANSI
   color escape codes, which was an early design mistake.  They now
   honor the configuration (e.g. "color.ui = never") and also tty-ness
   of the output medium.

 * The http.{sslkey,sslCert} configuration variables are to be
   interpreted as a pathname that honors "~[username]/" prefix, but
   weren't, which has been fixed.

 * Numerous bugs in walking of reflogs via "log -g" and friends have
   been fixed.

 * "git commit" when seeing an totally empty message said "you did not
   edit the message", which is clearly wrong.  The message has been
   corrected.

 * When a directory is not readable, "gitweb" fails to build the
   project list.  Work this around by skipping such a directory.

 * A recently added test for the "credential-cache" helper revealed
   that EOF detection done around the time the connection to the cache
   daemon is torn down were flaky.  This was fixed by reacting to
   ECONNRESET and behaving as if we got an EOF.

 * Some versions of GnuPG fail to kill gpg-agent it auto-spawned
   and such a left-over agent can interfere with a test.  Work it
   around by attempting to kill one before starting a new test.

 * "git log --tag=no-such-tag" showed log starting from HEAD, which
   has been fixed---it now shows nothing.

 * The "tag.pager" configuration variable was useless for those who
   actually create tag objects, as it interfered with the use of an
   editor.  A new mechanism has been introduced for commands to enable
   pager depending on what operation is being carried out to fix this,
   and then "git tag -l" is made to run pager by default.

 * "git push --recurse-submodules $there HEAD:$target" was not
   propagated down to the submodules, but now it is.

 * Commands like "git rebase" accepted the --rerere-autoupdate option
   from the command line, but did not always use it.  This has been
   fixed.

 * "git clone --recurse-submodules --quiet" did not pass the quiet
   option down to submodules.

 * "git am -s" has been taught that some input may end with a trailer
   block that is not Signed-off-by: and it should refrain from adding
   an extra blank line before adding a new sign-off in such a case.

 * "git svn" used with "--localtime" option did not compute the tz
   offset for the timestamp in question and instead always used the
   current time, which has been corrected.

 * Memory leaks in a few error codepaths have been plugged.

 * bash 4.4 or newer gave a warning on NUL byte in command
   substitution done in "git stash"; this has been squelched.

 * "git grep -L" and "git grep --quiet -L" reported different exit
   codes; this has been corrected.

 * When handshake with a subprocess filter notices that the process
   asked for an unknown capability, Git did not report what program
   the offending subprocess was running.  This has been corrected.

 * "git apply" that is used as a better "patch -p1" failed to apply a
   taken from a file with CRLF line endings to a file with CRLF line
   endings.  The root cause was because it misused convert_to_git()
   that tried to do "safe-crlf" processing by looking at the index
   entry at the same path, which is a nonsense---in that mode, "apply"
   is not working on the data in (or derived from) the index at all.
   This has been fixed.

 * Killing "git merge --edit" before the editor returns control left
   the repository in a state with MERGE_MSG but without MERGE_HEAD,
   which incorrectly tells the subsequent "git commit" that there was
   a squash merge in progress.  This has been fixed.

 * "git archive" did not work well with pathspecs and the
   export-ignore attribute.

 * "git cvsserver" no longer is invoked by "git shell" by default,
   as it is old and largely unmaintained.

 * Various Perl scripts did not use safe_pipe_capture() instead of
   backticks, leaving them susceptible to end-user input.  They have
   been corrected.

Also contains various documentation updates and code clean-ups.

Credits go to joernchen <joernchen@phenoelit.de> for finding the
unsafe constructs in "git cvsserver", and to Jeff King at GitHub for
finding and fixing instances of the same issue in other scripts.

----------------------------------------------------------------

Changes since v2.14.1 are as follows:

Andreas Heiduk (2):
      doc: add missing values "none" and "default" for diff.wsErrorHighlight
      doc: clarify "config --bool" behaviour with empty string

Anthony Sottile (1):
      git-grep: correct exit code with --quiet and -L

Brandon Williams (2):
      submodule--helper: teach push-check to handle HEAD
      clone: teach recursive clones to respect -q

Christian Couder (2):
      refs: use skip_prefix() in ref_is_hidden()
      sub-process: print the cmd when a capability is unsupported

Dimitrios Christidis (1):
      fmt-merge-msg: fix coding style

Heiko Voigt (1):
      t5526: fix some broken && chains

Hielke Christian Braun (1):
      gitweb: skip unreadable subdirectories

Jeff King (32):
      t1414: document some reflog-walk oddities
      revision: disallow reflog walking with revs->limited
      log: clarify comment about reflog cycles
      log: do not free parents when walking reflog
      get_revision_1(): replace do-while with an early return
      rev-list: check reflog_info before showing usage
      reflog-walk: stop using fake parents
      reflog-walk: apply --since/--until to reflog dates
      check return value of verify_ref_format()
      docs/for-each-ref: update pointer to color syntax
      t: use test_decode_color rather than literal ANSI codes
      ref-filter: simplify automatic color reset
      ref-filter: abstract ref format into its own struct
      ref-filter: move need_color_reset_at_eol into ref_format
      ref-filter: provide a function for parsing sort options
      ref-filter: make parse_ref_filter_atom a private function
      ref-filter: factor out the parsing of sorting atoms
      ref-filter: pass ref_format struct to atom parsers
      color: check color.ui in git_default_config()
      for-each-ref: load config earlier
      rev-list: pass diffopt->use_colors through to pretty-print
      pretty: respect color settings for %C placeholders
      ref-filter: consult want_color() before emitting colors
      t6018: flesh out empty input/output rev-list tests
      revision: add rev_input_given flag
      rev-list: don't show usage when we see empty ref patterns
      revision: do not fallback to default when rev_input_given is set
      hashcmp: use memcmp instead of open-coded loop
      doc: fix typo in sendemail.identity
      shell: drop git-cvsserver support by default
      archimport: use safe_pipe_capture for user input
      cvsimport: shell-quote variable used in backticks

Johannes Schindelin (2):
      run_processes_parallel: change confusing task_cb convention
      git-gui (MinGW): make use of MSys2's msgfmt

Jonathan Nieder (4):
      vcs-svn: remove more unused prototypes and declarations
      vcs-svn: remove custom mode constants
      vcs-svn: remove repo_delete wrapper function
      vcs-svn: move remaining repo_tree functions to fast_export.h

Jonathan Tan (7):
      fsck: remove redundant parse_tree() invocation
      object: remove "used" field from struct object
      fsck: cleanup unused variable
      Documentation: migrate sub-process docs to header
      sub-process: refactor handshake to common function
      tests: ensure fsck fails on corrupt packfiles
      Doc: clarify that pack-objects makes packs, plural

Junio C Hamano (11):
      http.c: http.sslcert and http.sslkey are both pathnames
      perl/Git.pm: typofix in a comment
      Prepare for 2.14.2
      RelNotes: further fixes for 2.14.2 from the master front
      cvsserver: move safe_pipe_capture() to the main package
      cvsserver: use safe_pipe_capture for `constant commands` as well
      Git 2.10.5
      Git 2.11.4
      Git 2.12.5
      Git 2.13.6
      Git 2.14.2

Kaartic Sivaraam (1):
      commit: check for empty message before the check for untouched template

Kevin Daudt (1):
      stash: prevent warning about null bytes in input

Lars Schneider (7):
      t0021: keep filter log files on comparison
      t0021: make debug log file name configurable
      t0021: write "OUT <size>" only on success
      convert: put the flags field before the flag itself for consistent style
      convert: move multiple file filter error handling to separate function
      convert: refactor capabilities negotiation
      convert: add "status=delayed" to filter process protocol

Martin Ågren (7):
      builtin.h: take over documentation from api-builtin.txt
      git.c: let builtins opt for handling `pager.foo` themselves
      git.c: provide setup_auto_pager()
      t7006: add tests for how git tag paginates
      tag: respect `pager.tag` in list-mode only
      tag: change default of `pager.tag` to "on"
      git.c: ignore pager.* when launching builtin as dashed external

Michael Forney (1):
      scripts: use "git foo" not "git-foo"

Michael J Gruber (6):
      Documentation: use proper wording for ref format strings
      Documentation/git-for-each-ref: clarify peeling of tags for --format
      Documentation/git-merge: explain --continue
      merge: clarify call chain
      merge: split write_merge_state in two
      merge: save merge state earlier

Philip Oakley (4):
      git-gui: remove duplicate entries from .gitconfig's gui.recentrepo
      git gui: cope with duplicates in _get_recentrepo
      git gui: de-dup selected repo from recentrepo history
      git gui: allow for a long recentrepo list

Phillip Wood (7):
      am: remember --rerere-autoupdate setting
      rebase: honor --rerere-autoupdate
      rebase -i: honor --rerere-autoupdate
      t3504: use test_commit
      cherry-pick/revert: remember --rerere-autoupdate
      cherry-pick/revert: reject --rerere-autoupdate when continuing
      am: fix signoff when other trailers are present

Ramsay Jones (2):
      credential-cache: interpret an ECONNRESET as an EOF
      builtin/add: add detail to a 'cannot chmod' error message

René Scharfe (23):
      bswap: convert to unsigned before shifting in get_be32
      bswap: convert get_be16, get_be32 and put_be32 to inline functions
      add MOVE_ARRAY
      use MOVE_ARRAY
      apply: use COPY_ARRAY and MOVE_ARRAY in update_image()
      ls-files: don't try to prune an empty index
      dir: support platforms that require aligned reads
      pack-objects: remove unnecessary NULL check
      t0001: skip test with restrictive permissions if getpwd(3) respects them
      test-path-utils: handle const parameter of basename and dirname
      t3700: fix broken test under !POSIXPERM
      t4062: use less than 256 repetitions in regex
      sha1_file: avoid comparison if no packed hash matches the first byte
      apply: remove prefix_length member from apply_state
      merge: use skip_prefix()
      win32: plug memory leak on realloc() failure in syslog()
      fsck: free buffers on error in fsck_obj()
      sha1_file: release delta_stack on error in unpack_entry()
      t1002: stop using sum(1)
      t5001: add tests for export-ignore attributes and exclude pathspecs
      archive: factor out helper functions for handling attributes
      archive: don't queue excluded directories
      commit: remove unused inline function single_parent()

Santiago Torres (1):
      t: lib-gpg: flush gpg agent on startup

Stefan Beller (3):
      t8008: rely on rev-parse'd HEAD instead of sha1 value
      sha1_file: make read_info_alternates static
      submodule.sh: remove unused variable

Torsten Bögershausen (2):
      convert: add SAFE_CRLF_KEEP_CRLF
      apply: file commited with CRLF should roundtrip diff and apply

Urs Thuermann (1):
      git svn fetch: Create correct commit timestamp when using --localtime

brian m. carlson (2):
      vcs-svn: remove unused prototypes
      vcs-svn: rename repo functions to "svn_repo"

joernchen (1):
      cvsserver: use safe_pipe_capture instead of backticks

Ævar Arnfjörð Bjarmason (1):
      tests: don't give unportable ">" to "test" built-in, use -gt



^ permalink raw reply	[relevance 11%]

* [PATCH] submodule: correct error message for missing commits.
@ 2017-09-26 18:27 Stefan Beller
  2017-09-26 23:37 ` Jacob Keller
  2017-09-27  0:52 ` Junio C Hamano
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2017-09-26 18:27 UTC (permalink / raw)
  To: git; +Cc: jacob.keller, Stefan Beller

When a submodule diff should be displayed we currently just add the
submodule objects to the main object store and then e.g. walk the
revision graph and create a summary for that submodule.

It is possible that we are missing the submodule either completely or
partially, which we currently differentiate with different error messages
depending on whether (1) the whole submodule object store is missing or
(2) just the needed for this particular diff. (1) is reported as
"not initialized", and (2) is reported as "commits not present".

If a submodule is deinit'ed its repository data is still around inside
the superproject, such that the diff can still be produced. In that way
the error message (1) is misleading as we can have a diff despite the
submodule being not initialized.

Downgrade the error message (1) to be the same as (2) and just say
the commits are not present, as that is the true reason why the diff
cannot be shown.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c                               | 2 +-
 t/t4059-diff-submodule-not-initialized.sh | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/submodule.c b/submodule.c
index 6531c5d609..280c246477 100644
--- a/submodule.c
+++ b/submodule.c
@@ -567,7 +567,7 @@ static void show_submodule_header(FILE *f, const char *path,
 
 	if (add_submodule_odb(path)) {
 		if (!message)
-			message = "(not initialized)";
+			message = "(commits not present)";
 		goto output_header;
 	}
 
diff --git a/t/t4059-diff-submodule-not-initialized.sh b/t/t4059-diff-submodule-not-initialized.sh
index cd70fd5192..49bca7b48d 100755
--- a/t/t4059-diff-submodule-not-initialized.sh
+++ b/t/t4059-diff-submodule-not-initialized.sh
@@ -95,7 +95,7 @@ test_expect_success 'submodule not initialized in new clone' '
 	git clone . sm3 &&
 	git -C sm3 diff-tree -p --no-commit-id --submodule=log HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 $smhead1...$smhead2 (not initialized)
+	Submodule sm1 $smhead1...$smhead2 (commits not present)
 	EOF
 	test_cmp expected actual
 '
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 32%]

* [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
  2017-09-26  6:28 [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules Junio C Hamano
@ 2017-09-26 18:54 ` Stefan Beller
  2017-09-26 19:46   ` Johannes Sixt
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-26 18:54 UTC (permalink / raw)
  To: gitster; +Cc: git, j6t, jrnieder, sbeller

submodule.<name>.update can be assigned an arbitrary command via setting
it to "!command". When this command is found in the regular config, Git
ought to just run that command instead of other update mechanisms.

However if that command is just found in the .gitmodules file, it is
potentially untrusted, which is why we do not run it.  Add a test
confirming the behavior.

Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---

Johannes wrote:
> I am pretty confident that this does not test what you intend to
> test. Notice that $TEST_DIRECTORY is expanded when the script is
> written. But that path contains a blank, and we have something like
> this in the test script:
>
>       #!/bin/sh
>       >/the/build/directory/t/trash directory.t7406/bad

I can confirm that.

Instead of mucking around with writing a script, "to make it robust",
I decided to got the simplest route and just have the command "!false",
which would make "git submodule update submodule" return an error.

Thanks,
Stefan


 t/t7406-submodule-update.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 034914a14f..a9ea098e55 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -406,6 +406,16 @@ test_expect_success 'submodule update - command in .git/config' '
 	)
 '
 
+test_expect_success 'submodule update - command in .gitmodules is ignored' '
+	test_when_finished "git -C super reset --hard HEAD^" &&
+
+	git -C super config -f .gitmodules submodule.submodule.update "!false" &&
+	git -C super commit -a -m "add command to .gitmodules file" &&
+	git -C super/submodule reset --hard $submodulesha1^ &&
+	git -C super submodule update submodule &&
+	test_path_is_missing bad
+'
+
 cat << EOF >expect
 Execution of 'false $submodulesha1' failed in submodule path 'submodule'
 EOF
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 32%]

* Re: [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
  2017-09-26 18:54 ` Stefan Beller
@ 2017-09-26 19:46   ` Johannes Sixt
  2017-09-26 19:54     ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Johannes Sixt @ 2017-09-26 19:46 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, jrnieder

Am 26.09.2017 um 20:54 schrieb Stefan Beller:
> +test_expect_success 'submodule update - command in .gitmodules is ignored' '
> +	test_when_finished "git -C super reset --hard HEAD^" &&
> +
> +	git -C super config -f .gitmodules submodule.submodule.update "!false" &&
> +	git -C super commit -a -m "add command to .gitmodules file" &&
> +	git -C super/submodule reset --hard $submodulesha1^ &&
> +	git -C super submodule update submodule &&
> +	test_path_is_missing bad

This test for a missing file is certainly a remnant from the previous 
iteration, isn't it?

-- Hannes

^ permalink raw reply	[relevance 15%]

* [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
  2017-09-26 19:46   ` Johannes Sixt
@ 2017-09-26 19:54     ` Stefan Beller
  2017-09-27  3:21       ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-09-26 19:54 UTC (permalink / raw)
  To: j6t; +Cc: git, gitster, jrnieder, sbeller

submodule.<name>.update can be assigned an arbitrary command via setting
it to "!command". When this command is found in the regular config, Git
ought to just run that command instead of other update mechanisms.

However if that command is just found in the .gitmodules file, it is
potentially untrusted, which is why we do not run it.  Add a test
confirming the behavior.

Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---

> This test for a missing file is certainly a remnant from the
> previous iteration, isn't it?

Yes. This is a good indicator I need some vacation.

Thanks,
Stefan

 t/t7406-submodule-update.sh | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 034914a14f..6f083c4d68 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -406,6 +406,14 @@ test_expect_success 'submodule update - command in .git/config' '
 	)
 '
 
+test_expect_success 'submodule update - command in .gitmodules is ignored' '
+	test_when_finished "git -C super reset --hard HEAD^" &&
+	git -C super config -f .gitmodules submodule.submodule.update "!false" &&
+	git -C super commit -a -m "add command to .gitmodules file" &&
+	git -C super/submodule reset --hard $submodulesha1^ &&
+	git -C super submodule update submodule
+'
+
 cat << EOF >expect
 Execution of 'false $submodulesha1' failed in submodule path 'submodule'
 EOF
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 33%]

* [PATCH] technical doc: add a design doc for hash function transition
  2017-09-26 22:11 RFC v3: Another proposed hash function transition plan Johannes Schindelin
@ 2017-09-26 22:25 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-09-26 22:25 UTC (permalink / raw)
  To: johannes.schindelin; +Cc: bmwill, david, demerphq, git, gitster, jason, jonathantanmy, jrnieder, peff, sandals, sbeller, torvalds, Jonathan Nieder

From: Jonathan Nieder <jrn@google.com>

This is "RFC v3: Another proposed hash function transition plan" from
the git mailing list.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Brandon Williams <bmwill@google.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---

 This takes the original Google Doc[1] and adds it to our history,
 such that the discussion can be on on list and in the commit messages.
 
 * replaced SHA3-256 with NEWHASH, sha3 with newhash
 * added section 'Implementation plan'
 * added section 'Future work'
 * added section 'Agreed-upon criteria for selecting NewHash'
 
 As the discussion restarts again, here is our attempt
 to add value to the discussion, we planned to polish it more, but as the
 discussion is restarting, we might just post it as-is.
  
 Thanks.

[1] https://docs.google.com/document/d/18hYAQCTsDgaFUo-VJGhT0UqyetL2LbAzkWNK1fYS8R0/edit

 Documentation/Makefile                             |   1 +
 .../technical/hash-function-transition.txt         | 571 +++++++++++++++++++++
 2 files changed, 572 insertions(+)
 create mode 100644 Documentation/technical/hash-function-transition.txt

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 2415e0d657..471bb29725 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -67,6 +67,7 @@ SP_ARTICLES += howto/maintain-git
 API_DOCS = $(patsubst %.txt,%,$(filter-out technical/api-index-skel.txt technical/api-index.txt, $(wildcard technical/api-*.txt)))
 SP_ARTICLES += $(API_DOCS)
 
+TECH_DOCS += technical/hash-function-transition
 TECH_DOCS += technical/http-protocol
 TECH_DOCS += technical/index-format
 TECH_DOCS += technical/pack-format
diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt
new file mode 100644
index 0000000000..0ac751d600
--- /dev/null
+++ b/Documentation/technical/hash-function-transition.txt
@@ -0,0 +1,571 @@
+Git hash function transition
+============================
+
+Objective
+---------
+Migrate Git from SHA-1 to a stronger hash function.
+
+Background
+----------
+At its core, the Git version control system is a content addressable
+filesystem. It uses the SHA-1 hash function to name content. For
+example, files, directories, and revisions are referred to by hash
+values unlike in other traditional version control systems where files
+or versions are referred to via sequential numbers. The use of a hash
+function to address its content delivers a few advantages:
+
+* Integrity checking is easy. Bit flips, for example, are easily
+  detected, as the hash of corrupted content does not match its name.
+* Lookup of objects is fast.
+
+Using a cryptographically secure hash function brings additional
+advantages:
+
+* Object names can be signed and third parties can trust the hash to
+  address the signed object and all objects it references.
+* Communication using Git protocol and out of band communication
+  methods have a short reliable string that can be used to reliably
+  address stored content.
+
+Over time some flaws in SHA-1 have been discovered by security
+researchers. https://shattered.io demonstrated a practical SHA-1 hash
+collision. As a result, SHA-1 cannot be considered cryptographically
+secure any more. This impacts the communication of hash values because
+we cannot trust that a given hash value represents the known good
+version of content that the speaker intended.
+
+SHA-1 still possesses the other properties such as fast object lookup
+and safe error checking, but other hash functions are equally suitable
+that are believed to be cryptographically secure.
+
+Goals
+-----
+1. The transition to NEWHASH can be done one local repository at a time.
+   a. Requiring no action by any other party.
+   b. A NEWHASH repository can communicate with SHA-1 Git servers
+      (push/fetch).
+   c. Users can use SHA-1 and NEWHASH identifiers for objects
+      interchangeably.
+   d. New signed objects make use of a stronger hash function than
+      SHA-1 for their security guarantees.
+2. Allow a complete transition away from SHA-1.
+   a. Local metadata for SHA-1 compatibility can be removed from a
+      repository if compatibility with SHA-1 is no longer needed.
+3. Maintainability throughout the process.
+   a. The object format is kept simple and consistent.
+   b. Creation of a generalized repository conversion tool.
+
+Non-Goals
+---------
+1. Add NEWHASH support to Git protocol. This is valuable and the
+   logical next step but it is out of scope for this initial design.
+2. Transparently improving the security of existing SHA-1 signed
+   objects.
+3. Intermixing objects using multiple hash functions in a single
+   repository.
+4. Taking the opportunity to fix other bugs in git's formats and
+   protocols.
+5. Shallow clones and fetches into a NEWHASH repository. (This will
+   change when we add NEWHASH support to Git protocol.)
+6. Skip fetching some submodules of a project into a NEWHASH
+   repository. (This also depends on NEWHASH support in Git
+   protocol.)
+
+Overview
+--------
+We introduce a new repository format extension `newhash`. Repositories
+with this extension enabled use NEWHASH instead of SHA-1 to name
+their objects. This affects both object names and object content ---
+both the names of objects and all references to other objects within
+an object are switched to the new hash function.
+
+newhash repositories cannot be read by older versions of Git.
+
+Alongside the packfile, a newhash repository stores a bidirectional
+mapping between newhash and sha1 object names in a new format of .idx files.
+The mapping is generated locally and can be verified using "git fsck".
+Object lookups use this mapping to allow naming objects using either
+their sha1 and newhash names interchangeably.
+
+"git cat-file" and "git hash-object" gain options to display an object
+in its sha1 form and write an object given its sha1 form. This
+requires all objects referenced by that object to be present in the
+object database so that they can be named using the appropriate name
+(using the bidirectional hash mapping).
+
+Fetches from a SHA-1 based server convert the fetched objects into
+newhash form and record the mapping in the bidirectional mapping table
+(see below for details). Pushes to a SHA-1 based server convert the
+objects being pushed into sha1 form so the server does not have to be
+aware of the hash function the client is using.
+
+Detailed Design
+---------------
+Object names
+~~~~~~~~~~~~
+Objects can be named by their 40 hexadecimal digit sha1-name or <n>
+hexadecimal digit newhash-name, plus names derived from those (see
+gitrevisions(7)).
+
+The sha1-name of an object is the SHA-1 of the concatenation of its
+type, length, a nul byte, and the object's sha1-content. This is the
+traditional <sha1> used in Git to name objects.
+
+The newhash-name of an object is the NEWHASH of the concatenation of its
+type, length, a nul byte, and the object's newhash-content.
+
+Object format
+~~~~~~~~~~~~~
+The content as a byte sequence of a tag, commit, or tree object named
+by sha1 and newhash differ because an object named by newhash-name refers to
+other objects by their newhash-names and an object named by sha1-name
+refers to other objects by their sha1-names.
+
+The newhash-content of an object is the same as its sha1-content, except
+that objects referenced by the object are named using their newhash-names
+instead of sha1-names. Because a blob object does not refer to any
+other object, its sha1-content and newhash-content are the same.
+
+The format allows round-trip conversion between newhash-content and
+sha1-content.
+
+Object storage
+~~~~~~~~~~~~~~
+Loose objects use zlib compression and packed objects use the packed
+format described in Documentation/technical/pack-format.txt, just like
+today. The content that is compressed and stored uses newhash-content
+instead of sha1-content.
+
+Translation table
+~~~~~~~~~~~~~~~~~
+A fast bidirectional mapping between sha1-names and newhash-names of all
+local objects in the repository is kept on disk.
+
+For pack files, upgrade the .idx file to be as follows:
+
+  4 magic bytes
+  header, containing pointers to the 3 lists below
+
+  list of
+  abbrev sha1 -> ordinal, sorted by sha1
+
+  list of
+  abbrev newhash -> ordinal, sorted by newhash
+
+  list of
+  ordinal, complete sha1, complete new hash,
+  sorted by ordinal, such that a lookup can be computed after looking into
+  one of the first lists.
+
+For unpacked objects, keep a simple list
+  sha1 -> newhash
+around at $OBJECT_DIR/loose-lookup
+
+All operations that make new objects (e.g., "git commit") add the new
+objects to the translation table.
+
+(This work could have been deferred to push time, but that would
+significantly complicate and slow down pushes. Calculating the
+sha1-name at object creation time at the same time it is being
+streamed to disk and having its newhash-name calculated should be an
+acceptable cost.)
+
+Reading an object's sha1-content
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The sha1-content of an object can be read by converting all newhash-names
+its newhash-content references to sha1-names using the translation table.
+
+Fetch
+~~~~~
+Fetching from a SHA-1 based server requires translating between SHA-1
+and NEWHASH based representations on the fly.
+
+SHA-1s named in the ref advertisement that are present on the client
+can be translated to NEWHASH and looked up as local objects using the
+translation table.
+
+Negotiation proceeds as today. Any "have"s generated locally are
+converted to SHA-1 before being sent to the server, and SHA-1s
+mentioned by the server are converted to NEWHASH when looking them up
+locally.
+
+After negotiation, the server sends a packfile containing the
+requested objects. We convert the packfile to NEWHASH format using
+the following steps:
+
+1. index-pack: inflate each object in the packfile and compute its
+   SHA-1. Objects can contain deltas in OBJ_REF_DELTA format against
+   objects the client has locally. These objects can be looked up
+   using the translation table and their sha1-content read as
+   described above to resolve the deltas.
+2. topological sort: starting at the "want"s from the negotiation
+   phase, walk through objects in the pack and emit a list of them,
+   excluding blobs, in reverse topologically sorted order, with each
+   object coming later in the list than all objects it references.
+   (This list only contains objects reachable from the "wants". If the
+   pack from the server contained additional extraneous objects, then
+   they will be discarded.)
+3. convert to newhash: open a new (newhash) packfile. Read the topologically
+   sorted list just generated. For each object, inflate its
+   sha1-content, convert to newhash-content, and write it to the newhash
+   pack. Include the new sha1<->newhash mapping entry in the translation
+   table.
+4. sort: reorder entries in the new pack to match the order of objects
+   in the pack the server generated and include blobs. Write a newhash idx
+   file.
+5. clean up: remove the SHA-1 based pack file, index, and
+   topologically sorted list obtained from the server and steps 1
+   and 2.
+
+Step 3 requires every object referenced by the new object to be in the
+translation table. This is why the topological sort step is necessary.
+
+As an optimization, step 1 could write a file describing what non-blob
+objects each object it has inflated from the packfile references. This
+makes the topological sort in step 2 possible without inflating the
+objects in the packfile for a second time. The objects need to be
+inflated again in step 3, for a total of two inflations.
+
+Step 4 is probably necessary for good read-time performance. "git
+pack-objects" on the server optimizes the pack file for good data
+locality (see Documentation/technical/pack-heuristics.txt).
+
+Details of this process are likely to change. It will take some
+experimenting to get this to perform well.
+
+Push
+~~~~
+Push is simpler than fetch because the objects referenced by the
+pushed objects are already in the translation table. The sha1-content
+of each object being pushed can be read as described in the "Reading
+an object's sha1-content" section to generate the pack written by git
+send-pack.
+
+Signed Commits
+~~~~~~~~~~~~~~
+We add a new field "gpgsig-newhash" to the commit object format to allow
+signing commits without relying on SHA-1. It is similar to the
+existing "gpgsig" field. Its signed payload is the newhash-content of the
+commit object with any "gpgsig" and "gpgsig-newhash" fields removed.
+
+This means commits can be signed
+1. using SHA-1 only, as in existing signed commit objects
+2. using both SHA-1 and NEWHASH, by using both gpgsig-newhash and gpgsig
+   fields.
+3. using only NEWHASH, by only using the gpgsig-newhash field.
+
+Old versions of "git verify-commit" can verify the gpgsig signature in
+cases (1) and (2) without modifications and view case (3) as an
+ordinary unsigned commit.
+
+Signed Tags
+~~~~~~~~~~~
+We add a new field "gpgsig-newhash" to the tag object format to allow
+signing tags without relying on SHA-1. Its signed payload is the
+newhash-content of the tag with its gpgsig-newhash field and "-----BEGIN PGP
+SIGNATURE-----" delimited in-body signature removed.
+
+This means tags can be signed
+1. using SHA-1 only, as in existing signed tag objects
+2. using both SHA-1 and NEWHASH, by using gpgsig-newhash and an in-body
+   signature.
+3. using only NEWHASH, by only using the gpgsig-newhash field.
+
+Mergetag embedding
+~~~~~~~~~~~~~~~~~~
+The mergetag field in the sha1-content of a commit contains the
+sha1-content of a tag that was merged by that commit.
+
+The mergetag field in the newhash-content of the same commit contains the
+newhash-content of the same tag.
+
+Submodules
+~~~~~~~~~~
+To convert recorded submodule pointers, you need to have the converted
+submodule repository in place. The translation table of the submodule
+can be used to look up the new hash.
+
+Caveats
+-------
+Invalid objects
+~~~~~~~~~~~~~~~
+The conversion from sha1-content to newhash-content retains any
+brokenness in the original object (e.g., tree entry modes encoded with
+leading 0, tree objects whose paths are not sorted correctly, and
+commit objects without an author or committer). This is a deliberate
+feature of the design to allow the conversion to round-trip.
+
+More profoundly broken objects (e.g., a commit with a truncated "tree"
+header line) cannot be converted but were not usable by current Git
+anyway.
+
+Shallow clone and submodules
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Because it requires all referenced objects to be available in the
+locally generated translation table, this design does not support
+shallow clone or unfetched submodules. Protocol improvements might
+allow lifting this restriction.
+
+Alternates
+~~~~~~~~~~
+For the same reason, a newhash repository cannot borrow objects from a
+sha1 repository using objects/info/alternates or
+$GIT_ALTERNATE_OBJECT_REPOSITORIES.
+
+git notes
+~~~~~~~~~
+The "git notes" tool annotates objects using their sha1-name as key.
+This design does not describe a way to migrate notes trees to use
+newhash-names. That migration is expected to happen separately (for
+example using a file at the root of the notes tree to describe which
+hash it uses).
+
+Server-side cost
+~~~~~~~~~~~~~~~~
+Until Git protocol gains NEWHASH support, using newhash based storage on
+public-facing Git servers is strongly discouraged. Once Git protocol
+gains NEWHASH support, newhash based servers are likely not to support
+sha1 compatibility, to avoid what may be a very expensive hash
+reencode during clone and to encourage peers to modernize.
+
+The design described here allows fetches by SHA-1 clients of a
+personal NEWHASH repository because it's not much more difficult than
+allowing pushes from that repository. This support needs to be guarded
+by a configuration option --- servers like git.kernel.org that serve a
+large number of clients would not be expected to bear that cost.
+
+Meaning of signatures
+~~~~~~~~~~~~~~~~~~~~~
+The signed payload for signed commits and tags does not explicitly
+name the hash used to identify objects. If some day Git adopts a new
+hash function with the same length as the current SHA-1 (40
+hexadecimal digit) or NEWHASH (64 hexadecimal digit) objects then the
+intent behind the PGP signed payload in an object signature is
+unclear:
+
+	object e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7
+	type commit
+	tag v2.12.0
+	tagger Junio C Hamano <gitster@pobox.com> 1487962205 -0800
+
+	Git 2.12
+
+Does this mean Git v2.12.0 is the commit with sha1-name
+e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7 or the commit with
+new-40-digit-hash-name e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7?
+
+Fortunately NEWHASH and SHA-1 have different lengths. If Git starts
+using another hash with the same length to name objects, then it will
+need to change the format of signed payloads using that hash to
+address this issue.
+
+Alternatives considered
+-----------------------
+Upgrading everyone working on a particular project on a flag day
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Projects like the Linux kernel are large and complex enough that
+flipping the switch for all projects based on the repository at once
+is infeasible.
+
+Not only would all developers and server operators supporting
+developers have to switch on the same flag day, but supporting tooling
+(continuous integration, code review, bug trackers, etc) would have to
+be adapted as well. This also makes it difficult to get early feedback
+from some project participants testing before it is time for mass
+adoption.
+
+Using hash functions in parallel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+(e.g. https://public-inbox.org/git/22708.8913.864049.452252@chiark.greenend.org.uk/ )
+Objects newly created would be addressed by the new hash, but inside
+such an object (e.g. commit) it is still possible to address objects
+using the old hash function.
+* You cannot trust its history (needed for bisectability) in the
+  future without further work
+* Maintenance burden as the number of supported hash functions grows
+  (they will never go away, so they accumulate). In this proposal, by
+  comparison, converted objects lose all references to SHA-1.
+
+Signed objects with multiple hashes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Instead of introducing the gpgsig-newhash field in commit and tag objects
+for newhash-content based signatures, an earlier version of this design
+added "hash newhash <newhash-name>" fields to strengthen the existing
+sha1-content based signatures.
+
+In other words, a single signature was used to attest to the object
+content using both hash functions. This had some advantages:
+* Using one signature instead of two speeds up the signing process.
+* Having one signed payload with both hashes allows the signer to
+  attest to the sha1-name and newhash-name referring to the same object.
+* All users consume the same signature. Broken signatures are likely
+  to be detected quickly using current versions of git.
+
+However, it also came with disadvantages:
+* Verifying a signed object requires access to the sha1-names of all
+  objects it references, even after the transition is complete and
+  translation table is no longer needed for anything else. To support
+  this, the design added fields such as "hash sha1 tree <sha1-name>"
+  and "hash sha1 parent <sha1-name>" to the newhash-content of a signed
+  commit, complicating the conversion process.
+* Allowing signed objects without a sha1 (for after the transition is
+  complete) complicated the design further, requiring a "nohash sha1"
+  field to suppress including "hash sha1" fields in the newhash-content
+  and signed payload.
+
+
+Implementation plan
+-------------------
+
+Here's a rough list of some useful tasks, in no particular order:
+
+1. bc/object-id: This patch series continues, eliminating assumptions
+   about the size of object ids by encapsulating them in a struct.
+   One straightforward way to find code that still needs to be
+   converted is to grep for "sha" --- often the conversion patches
+   change function and variable names to refer to oid_ where they used
+   to use sha1_, making the stragglers easier to spot.
+
+2. Hard-coded object ids in tests: Many tests beyond t00* make assumptions
+   about the exact values of object ids.  That's bad for maintainability
+   for other reasons beyond the hash function transition, too.
+
+   It should be possible to suss them out by patching git's sha1
+   routine to use the ones-complement of sha1 (~sha1) instead and
+   seeing which tests fail.
+
+3. Repository format extension to use a different hash function: we
+   want git to be able to work with two hash functions: sha1 and
+   something else.  For interoperability and simplity, it is useful
+   for a single git binary to support both hash functions.
+
+   That means a repository needs to be able to specify what hash
+   function is used for the objects in that repository.  This can be
+   configured by setting '[core] repositoryformatversion=1' (to avoid
+   confusing old versions of git) and
+   '[extensions] experimentalNewHashFunction = true'.
+   Documentation/technical/repository-version.txt has more details.
+
+   We can start experimenting with this using e.g. the ~sha1 function
+   described at (2), or the 160-bit hash of the patch author's choice
+   (e.g. truncated blake2bp-256).
+
+4. When choosing a hash function, people may argue about performance.
+   It would be useful for run some benchmarks for git (running
+   the test suite, t/perf tests, etc) using a variety of hash
+   functions as input to such a discussion.
+
+5. Longer hash: Even once all object id references in git use struct
+   object_id (see (1)), we need to tackle other assumptions about
+   object id size in git and its tests.
+
+   It should be possible to suss them out by replacing git's sha1
+   routine with a 40-byte hash: sha1 with each byte repeated (sha1+sha1)
+   and seeing what fails.
+
+6. Repository format extension for longer hash: As in (3), we could
+   add a repository format extension to experiment with using the
+   sha1+sha1 function.
+
+7. Avoiding wasted memory from unused hash functions: struct object_id
+   has definition 'unsigned char hash[GIT_MAX_RAWSZ]', where
+   GIT_MAX_RAWSZ is the size of the largest supported hash function.
+   When operating on a repository that only uses sha1, this wastes
+   memory.
+
+   Avoid that by making object identifiers variable-sized.  That is,
+   something like
+
+     struct object_id {
+        union {
+           unsigned char hash20[20];
+           unsigned char hash32[32];
+        } *hash;
+     }
+
+   or
+
+     struct object_id {
+       unsigned char *hash;
+     }
+
+   The hard part is that allocation and destruction have to be
+   explicit instead of happening automatically when an object_id is an
+   automatic variable.
+
+8. Implementation of this plan (roughly in order):
+   - abstract the hash computation to be able to plug in another hash
+   - make the choice of hash dependant on repository extension
+   - implement the new .idx format
+   - implement cat-file's flag to show things in old/new hash
+   - convert fetch, push
+
+9. We can use help from security experts in all of this.  Fuzzing,
+   analysis of how we use cryptography, security review of other parts
+   of the design, and information to help choose a hash function are
+   all appreciated.
+
+Agreed-upon criteria for selecting NewHash
+------------------------------------------
+
+The discussion which hash function to use is going in circles, so let's
+first argree on criteria on how to select the new hash function. These
+could include:
+* cryptografic strength
+* performance
+* other cryptografic aspects(?)
+* portability / availability of properly licensed implementations
+
+Future work
+-----------
+
+* other compression instead of zlib (this is a stated non goal, though!)
+* rehash discussion whether to include generation numbers natively
+  (this is a stated non goal, though!)
+* describing (1) the possibility of caching translated objects
+* and (2) protocol changes.
+* other format changes
+
+Document History
+----------------
+
+2017-03-03
+bmwill@google.com, jonathantanmy@google.com, jrnieder@gmail.com,
+sbeller@google.com
+
+Initial version sent to
+http://public-inbox.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com
+
+2017-03-03 jrnieder@gmail.com
+Incorporated suggestions from jonathantanmy and sbeller:
+* describe purpose of signed objects with each hash type
+* redefine signed object verification using object content under the
+  first hash function
+
+2017-03-06 jrnieder@gmail.com
+* Use SHA3-256 instead of SHA2 (thanks, Linus and brian m. carlson).[1][2]
+* Make sha3-based signatures a separate field, avoiding the need for
+  "hash" and "nohash" fields (thanks to peff[3]).
+* Add a sorting phase to fetch (thanks to Junio for noticing the need
+  for this).
+* Omit blobs from the topological sort during fetch (thanks to peff).
+* Discuss alternates, git notes, and git servers in the caveats
+  section (thanks to Junio Hamano, brian m. carlson[4], and Shawn
+  Pearce).
+* Clarify language throughout (thanks to various commenters,
+  especially Junio).
+
+[1] http://public-inbox.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/
+[2] http://public-inbox.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/
+[3] http://public-inbox.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/
+[4] http://public-inbox.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net
+
+2017-09-25
+* replaced SHA3-256 with NEWHASH, sha3 with newhash
+* added section 'Implementation plan'
+* added section 'Future work'
+
+* This version is sent to the list; to be incorporated into git.git, such
+  that further document history is found using git-log.
+
+
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 7%]

* Re: [PATCH] submodule: correct error message for missing commits.
  2017-09-26 18:27 [PATCH] submodule: correct error message for missing commits Stefan Beller
@ 2017-09-26 23:37 ` Jacob Keller
  2017-09-27  0:52 ` Junio C Hamano
  1 sibling, 0 replies; 200+ results
From: Jacob Keller @ 2017-09-26 23:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Git mailing list

On Tue, Sep 26, 2017 at 11:27 AM, Stefan Beller <sbeller@google.com> wrote:
> When a submodule diff should be displayed we currently just add the
> submodule objects to the main object store and then e.g. walk the
> revision graph and create a summary for that submodule.
>
> It is possible that we are missing the submodule either completely or
> partially, which we currently differentiate with different error messages
> depending on whether (1) the whole submodule object store is missing or
> (2) just the needed for this particular diff. (1) is reported as
> "not initialized", and (2) is reported as "commits not present".
>
> If a submodule is deinit'ed its repository data is still around inside
> the superproject, such that the diff can still be produced. In that way
> the error message (1) is misleading as we can have a diff despite the
> submodule being not initialized.
>
> Downgrade the error message (1) to be the same as (2) and just say
> the commits are not present, as that is the true reason why the diff
> cannot be shown.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>

This makes sense to me.

Thanks,
Jake

> ---
>  submodule.c                               | 2 +-
>  t/t4059-diff-submodule-not-initialized.sh | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/submodule.c b/submodule.c
> index 6531c5d609..280c246477 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -567,7 +567,7 @@ static void show_submodule_header(FILE *f, const char *path,
>
>         if (add_submodule_odb(path)) {
>                 if (!message)
> -                       message = "(not initialized)";
> +                       message = "(commits not present)";
>                 goto output_header;
>         }
>
> diff --git a/t/t4059-diff-submodule-not-initialized.sh b/t/t4059-diff-submodule-not-initialized.sh
> index cd70fd5192..49bca7b48d 100755
> --- a/t/t4059-diff-submodule-not-initialized.sh
> +++ b/t/t4059-diff-submodule-not-initialized.sh
> @@ -95,7 +95,7 @@ test_expect_success 'submodule not initialized in new clone' '
>         git clone . sm3 &&
>         git -C sm3 diff-tree -p --no-commit-id --submodule=log HEAD >actual &&
>         cat >expected <<-EOF &&
> -       Submodule sm1 $smhead1...$smhead2 (not initialized)
> +       Submodule sm1 $smhead1...$smhead2 (commits not present)
>         EOF
>         test_cmp expected actual
>  '
> --
> 2.14.0.rc0.3.g6c2e499285
>

^ permalink raw reply	[relevance 15%]

* Re: [PATCH] submodule: correct error message for missing commits.
  2017-09-26 18:27 [PATCH] submodule: correct error message for missing commits Stefan Beller
  2017-09-26 23:37 ` Jacob Keller
@ 2017-09-27  0:52 ` Junio C Hamano
  1 sibling, 0 replies; 200+ results
From: Junio C Hamano @ 2017-09-27  0:52 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, jacob.keller

Stefan Beller <sbeller@google.com> writes:

> When a submodule diff should be displayed we currently just add the
> submodule objects to the main object store and then e.g. walk the
> revision graph and create a summary for that submodule.
>
> It is possible that we are missing the submodule either completely or
> partially, which we currently differentiate with different error messages
> depending on whether (1) the whole submodule object store is missing or
> (2) just the needed for this particular diff. (1) is reported as
> "not initialized", and (2) is reported as "commits not present".
>
> If a submodule is deinit'ed its repository data is still around inside
> the superproject, such that the diff can still be produced. In that way
> the error message (1) is misleading as we can have a diff despite the
> submodule being not initialized.
>
> Downgrade the error message (1) to be the same as (2) and just say
> the commits are not present, as that is the true reason why the diff
> cannot be shown.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---

Sounds good.  Thanks for working on it.


^ permalink raw reply	[relevance 15%]

* Re: [PATCH] t7406: submodule.<name>.update command must not be run from .gitmodules
  2017-09-26 19:54     ` Stefan Beller
@ 2017-09-27  3:21       ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-09-27  3:21 UTC (permalink / raw)
  To: Stefan Beller; +Cc: j6t, git, jrnieder

Stefan Beller <sbeller@google.com> writes:

> submodule.<name>.update can be assigned an arbitrary command via setting
> it to "!command". When this command is found in the regular config, Git
> ought to just run that command instead of other update mechanisms.
>
> However if that command is just found in the .gitmodules file, it is
> potentially untrusted, which is why we do not run it.  Add a test
> confirming the behavior.
>
> Suggested-by: Jonathan Nieder <jrnieder@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---

Earlier, we saw:

    Ideally we want this test to be super robust: e.g. if it runs the
    command but from a different directory, we still want the test to fail,
    and if it runs the command but using exec instead of a shell, we still
    want the test to fail.

and this one (i.e. signal that it is a command by prefixing with
'!', and then have a single command that would fail whether it is
run via run_command() with or without shell) would satisfy that
criteria, I would think.

>> This test for a missing file is certainly a remnant from the
>> previous iteration, isn't it?
>
> Yes. This is a good indicator I need some vacation.

Or just take a deep breath before making a knee-jerk reaction public
and instead double-check before sending things out ;-)

Will queue.  Thanks.

>
> Thanks,
> Stefan
>
>  t/t7406-submodule-update.sh | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
> index 034914a14f..6f083c4d68 100755
> --- a/t/t7406-submodule-update.sh
> +++ b/t/t7406-submodule-update.sh
> @@ -406,6 +406,14 @@ test_expect_success 'submodule update - command in .git/config' '
>  	)
>  '
>  
> +test_expect_success 'submodule update - command in .gitmodules is ignored' '
> +	test_when_finished "git -C super reset --hard HEAD^" &&
> +	git -C super config -f .gitmodules submodule.submodule.update "!false" &&
> +	git -C super commit -a -m "add command to .gitmodules file" &&
> +	git -C super/submodule reset --hard $submodulesha1^ &&
> +	git -C super submodule update submodule
> +'
> +
>  cat << EOF >expect
>  Execution of 'false $submodulesha1' failed in submodule path 'submodule'
>  EOF

^ permalink raw reply	[relevance 15%]

* [PATCH v4] technical doc: add a design doc for hash function transition
  2017-03-09 19:14     ` Shawn Pearce
@ 2017-09-28  4:43       ` Jonathan Nieder
  2017-09-29  6:06         ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Jonathan Nieder @ 2017-09-28  4:43 UTC (permalink / raw)
  To: Shawn Pearce; +Cc: Linus Torvalds, Git Mailing List, Stefan Beller, bmwill, Jonathan Tan, Jeff King, David Lang, brian m. carlson, Masaya Suzuki, demerphq, The Keccak Team, Johannes Schindelin

This document describes what a transition to a new hash function for
Git would look like.  Add it to Documentation/technical/ as the plan
of record so that future changes can be recorded as patches.

Also-by: Brandon Williams <bmwill@google.com>
Also-by: Jonathan Tan <jonathantanmy@google.com>
Also-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
On Thu, Mar 09, 2017 at 11:14 AM, Shawn Pearce wrote:
> On Mon, Mar 6, 2017 at 4:17 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> Thanks for the kind words on what had quite a few flaws still.  Here's
>> a new draft.  I think the next version will be a patch against
>> Documentation/technical/.
>
> FWIW, I like this approach.

Okay, here goes.

Instead of sharding the loose object translation tables by first byte,
we went for a single table.  It simplifies the design and we need to
keep the number of loose objects under control anyway.

We also included a description of the transition plan and tried to
include a summary of what has been agreed upon so far about the choice
of hash function.

Thanks to Junio for reviving the discussion and in particular to Dscho
for pushing this forward and making the missing pieces clearer.

Thoughts of all kinds welcome, as always.

 Documentation/Makefile                             |   1 +
 .../technical/hash-function-transition.txt         | 797 +++++++++++++++++++++
 2 files changed, 798 insertions(+)
 create mode 100644 Documentation/technical/hash-function-transition.txt

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 2415e0d657..471bb29725 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -67,6 +67,7 @@ SP_ARTICLES += howto/maintain-git
 API_DOCS = $(patsubst %.txt,%,$(filter-out technical/api-index-skel.txt technical/api-index.txt, $(wildcard technical/api-*.txt)))
 SP_ARTICLES += $(API_DOCS)
 
+TECH_DOCS += technical/hash-function-transition
 TECH_DOCS += technical/http-protocol
 TECH_DOCS += technical/index-format
 TECH_DOCS += technical/pack-format
diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt
new file mode 100644
index 0000000000..417ba491d0
--- /dev/null
+++ b/Documentation/technical/hash-function-transition.txt
@@ -0,0 +1,797 @@
+Git hash function transition
+============================
+
+Objective
+---------
+Migrate Git from SHA-1 to a stronger hash function.
+
+Background
+----------
+At its core, the Git version control system is a content addressable
+filesystem. It uses the SHA-1 hash function to name content. For
+example, files, directories, and revisions are referred to by hash
+values unlike in other traditional version control systems where files
+or versions are referred to via sequential numbers. The use of a hash
+function to address its content delivers a few advantages:
+
+* Integrity checking is easy. Bit flips, for example, are easily
+  detected, as the hash of corrupted content does not match its name.
+* Lookup of objects is fast.
+
+Using a cryptographically secure hash function brings additional
+advantages:
+
+* Object names can be signed and third parties can trust the hash to
+  address the signed object and all objects it references.
+* Communication using Git protocol and out of band communication
+  methods have a short reliable string that can be used to reliably
+  address stored content.
+
+Over time some flaws in SHA-1 have been discovered by security
+researchers. https://shattered.io demonstrated a practical SHA-1 hash
+collision. As a result, SHA-1 cannot be considered cryptographically
+secure any more. This impacts the communication of hash values because
+we cannot trust that a given hash value represents the known good
+version of content that the speaker intended.
+
+SHA-1 still possesses the other properties such as fast object lookup
+and safe error checking, but other hash functions are equally suitable
+that are believed to be cryptographically secure.
+
+Goals
+-----
+Where NewHash is a strong 256-bit hash function to replace SHA-1 (see
+"Selection of a New Hash", below):
+
+1. The transition to NewHash can be done one local repository at a time.
+   a. Requiring no action by any other party.
+   b. A NewHash repository can communicate with SHA-1 Git servers
+      (push/fetch).
+   c. Users can use SHA-1 and NewHash identifiers for objects
+      interchangeably (see "Object names on the command line", below).
+   d. New signed objects make use of a stronger hash function than
+      SHA-1 for their security guarantees.
+2. Allow a complete transition away from SHA-1.
+   a. Local metadata for SHA-1 compatibility can be removed from a
+      repository if compatibility with SHA-1 is no longer needed.
+3. Maintainability throughout the process.
+   a. The object format is kept simple and consistent.
+   b. Creation of a generalized repository conversion tool.
+
+Non-Goals
+---------
+1. Add NewHash support to Git protocol. This is valuable and the
+   logical next step but it is out of scope for this initial design.
+2. Transparently improving the security of existing SHA-1 signed
+   objects.
+3. Intermixing objects using multiple hash functions in a single
+   repository.
+4. Taking the opportunity to fix other bugs in Git's formats and
+   protocols.
+5. Shallow clones and fetches into a NewHash repository. (This will
+   change when we add NewHash support to Git protocol.)
+6. Skip fetching some submodules of a project into a NewHash
+   repository. (This also depends on NewHash support in Git
+   protocol.)
+
+Overview
+--------
+We introduce a new repository format extension. Repositories with this
+extension enabled use NewHash instead of SHA-1 to name their objects.
+This affects both object names and object content --- both the names
+of objects and all references to other objects within an object are
+switched to the new hash function.
+
+NewHash repositories cannot be read by older versions of Git.
+
+Alongside the packfile, a NewHash repository stores a bidirectional
+mapping between NewHash and SHA-1 object names. The mapping is generated
+locally and can be verified using "git fsck". Object lookups use this
+mapping to allow naming objects using either their SHA-1 and NewHash names
+interchangeably.
+
+"git cat-file" and "git hash-object" gain options to display an object
+in its sha1 form and write an object given its sha1 form. This
+requires all objects referenced by that object to be present in the
+object database so that they can be named using the appropriate name
+(using the bidirectional hash mapping).
+
+Fetches from a SHA-1 based server convert the fetched objects into
+NewHash form and record the mapping in the bidirectional mapping table
+(see below for details). Pushes to a SHA-1 based server convert the
+objects being pushed into sha1 form so the server does not have to be
+aware of the hash function the client is using.
+
+Detailed Design
+---------------
+Repository format extension
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A NewHash repository uses repository format version `1` (see
+Documentation/technical/repository-version.txt) with extensions
+`objectFormat` and `compatObjectFormat`:
+
+	[core]
+		repositoryFormatVersion = 1
+	[extensions]
+		objectFormat = newhash
+		compatObjectFormat = sha1
+
+Specifying a repository format extension ensures that versions of Git
+not aware of NewHash do not try to operate on these repositories,
+instead producing an error message:
+
+	$ git status
+	fatal: unknown repository extensions found:
+		objectformat
+		compatobjectformat
+
+See the "Transition plan" section below for more details on these
+repository extensions.
+
+Object names
+~~~~~~~~~~~~
+Objects can be named by their 40 hexadecimal digit sha1-name or 64
+hexadecimal digit newhash-name, plus names derived from those (see
+gitrevisions(7)).
+
+The sha1-name of an object is the SHA-1 of the concatenation of its
+type, length, a nul byte, and the object's sha1-content. This is the
+traditional <sha1> used in Git to name objects.
+
+The newhash-name of an object is the NewHash of the concatenation of its
+type, length, a nul byte, and the object's newhash-content.
+
+Object format
+~~~~~~~~~~~~~
+The content as a byte sequence of a tag, commit, or tree object named
+by sha1 and newhash differ because an object named by newhash-name refers to
+other objects by their newhash-names and an object named by sha1-name
+refers to other objects by their sha1-names.
+
+The newhash-content of an object is the same as its sha1-content, except
+that objects referenced by the object are named using their newhash-names
+instead of sha1-names. Because a blob object does not refer to any
+other object, its sha1-content and newhash-content are the same.
+
+The format allows round-trip conversion between newhash-content and
+sha1-content.
+
+Object storage
+~~~~~~~~~~~~~~
+Loose objects use zlib compression and packed objects use the packed
+format described in Documentation/technical/pack-format.txt, just like
+today. The content that is compressed and stored uses newhash-content
+instead of sha1-content.
+
+Pack index
+~~~~~~~~~~
+Pack index (.idx) files use a new v3 format that supports multiple
+hash functions. They have the following format (all integers are in
+network byte order):
+
+- A header appears at the beginning and consists of the following:
+  - The 4-byte pack index signature: '\377t0c'
+  - 4-byte version number: 3
+  - 4-byte length of the header section, including the signature and
+    version number
+  - 4-byte number of objects contained in the pack
+  - 4-byte number of object formats in this pack index: 2
+  - For each object format:
+    - 4-byte format identifier (e.g., 'sha1' for SHA-1)
+    - 4-byte length in bytes of shortened object names. This is the
+      shortest possible length needed to make names in the shortened
+      object name table unambiguous.
+    - 4-byte integer, recording where tables relating to this format
+      are stored in this index file, as an offset from the beginning.
+  - 4-byte offset to the trailer from the beginning of this file.
+  - Zero or more additional key/value pairs (4-byte key, 4-byte
+    value). Only one key is supported: 'PSRC'. See the "Loose objects
+    and unreachable objects" section for supported values and how this
+    is used.  All other keys are reserved. Readers must ignore
+    unrecognized keys.
+- Zero or more NUL bytes. This can optionally be used to improve the
+  alignment of the full object name table below.
+- Tables for the first object format:
+  - A sorted table of shortened object names.  These are prefixes of
+    the names of all objects in this pack file, packed together
+    without offset values to reduce the cache footprint of the binary
+    search for a specific object name.
+
+  - A table of full object names in pack order. This allows resolving
+    a reference to "the nth object in the pack file" (from a
+    reachability bitmap or from the next table of another object
+    format) to its object name.
+
+  - A table of 4-byte values mapping object name order to pack order.
+    For an object in the table of sorted shortened object names, the
+    value at the corresponding index in this table is the index in the
+    previous table for that same object.
+
+    This can be used to look up the object in reachability bitmaps or
+    to look up its name in another object format.
+
+  - A table of 4-byte CRC32 values of the packed object data, in the
+    order that the objects appear in the pack file. This is to allow
+    compressed data to be copied directly from pack to pack during
+    repacking without undetected data corruption.
+
+  - A table of 4-byte offset values. For an object in the table of
+    sorted shortened object names, the value at the corresponding
+    index in this table indicates where that object can be found in
+    the pack file. These are usually 31-bit pack file offsets, but
+    large offsets are encoded as an index into the next table with the
+    most significant bit set.
+
+  - A table of 8-byte offset entries (empty for pack files less than
+    2 GiB). Pack files are organized with heavily used objects toward
+    the front, so most object references should not need to refer to
+    this table.
+- Zero or more NUL bytes.
+- Tables for the second object format, with the same layout as above,
+  up to and not including the table of CRC32 values.
+- Zero or more NUL bytes.
+- The trailer consists of the following:
+  - A copy of the 20-byte NewHash checksum at the end of the
+    corresponding packfile.
+
+  - 20-byte NewHash checksum of all of the above.
+
+Loose object index
+~~~~~~~~~~~~~~~~~~
+A new file $GIT_OBJECT_DIR/loose-object-idx contains information about
+all loose objects. Its format is
+
+  # loose-object-idx
+  (newhash-name SP sha1-name LF)*
+
+where the object names are in hexadecimal format. The file is not
+sorted.
+
+The loose object index is protected against concurrent writes by a
+lock file $GIT_OBJECT_DIR/loose-object-idx.lock. To add a new loose
+object:
+
+1. Write the loose object to a temporary file, like today.
+2. Open loose-object-idx.lock with O_CREAT | O_EXCL to acquire the lock.
+3. Rename the loose object into place.
+4. Open loose-object-idx with O_APPEND and write the new object
+5. Unlink loose-object-idx.lock to release the lock.
+
+To remove entries (e.g. in "git pack-refs" or "git-prune"):
+
+1. Open loose-object-idx.lock with O_CREAT | O_EXCL to acquire the
+   lock.
+2. Write the new content to loose-object-idx.lock.
+3. Unlink any loose objects being removed.
+4. Rename to replace loose-object-idx, releasing the lock.
+
+Translation table
+~~~~~~~~~~~~~~~~~
+The index files support a bidirectional mapping between sha1-names
+and newhash-names. The lookup proceeds similarly to ordinary object
+lookups. For example, to convert a sha1-name to a newhash-name:
+
+ 1. Look for the object in idx files. If a match is present in the
+    idx's sorted list of truncated sha1-names, then:
+    a. Read the corresponding entry in the sha1-name order to pack
+       name order mapping.
+    b. Read the corresponding entry in the full sha1-name table to
+       verify we found the right object. If it is, then
+    c. Read the corresponding entry in the full newhash-name table.
+       That is the object's newhash-name.
+ 2. Check for a loose object. Read lines from loose-object-idx until
+    we find a match.
+
+Step (1) takes the same amount of time as an ordinary object lookup:
+O(number of packs * log(objects per pack)). Step (2) takes O(number of
+loose objects) time. To maintain good performance it will be necessary
+to keep the number of loose objects low. See the "Loose objects and
+unreachable objects" section below for more details.
+
+Since all operations that make new objects (e.g., "git commit") add
+the new objects to the corresponding index, this mapping is possible
+for all objects in the object store.
+
+Reading an object's sha1-content
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The sha1-content of an object can be read by converting all newhash-names
+its newhash-content references to sha1-names using the translation table.
+
+Fetch
+~~~~~
+Fetching from a SHA-1 based server requires translating between SHA-1
+and NewHash based representations on the fly.
+
+SHA-1s named in the ref advertisement that are present on the client
+can be translated to NewHash and looked up as local objects using the
+translation table.
+
+Negotiation proceeds as today. Any "have"s generated locally are
+converted to SHA-1 before being sent to the server, and SHA-1s
+mentioned by the server are converted to NewHash when looking them up
+locally.
+
+After negotiation, the server sends a packfile containing the
+requested objects. We convert the packfile to NewHash format using
+the following steps:
+
+1. index-pack: inflate each object in the packfile and compute its
+   SHA-1. Objects can contain deltas in OBJ_REF_DELTA format against
+   objects the client has locally. These objects can be looked up
+   using the translation table and their sha1-content read as
+   described above to resolve the deltas.
+2. topological sort: starting at the "want"s from the negotiation
+   phase, walk through objects in the pack and emit a list of them,
+   excluding blobs, in reverse topologically sorted order, with each
+   object coming later in the list than all objects it references.
+   (This list only contains objects reachable from the "wants". If the
+   pack from the server contained additional extraneous objects, then
+   they will be discarded.)
+3. convert to newhash: open a new (newhash) packfile. Read the topologically
+   sorted list just generated. For each object, inflate its
+   sha1-content, convert to newhash-content, and write it to the newhash
+   pack. Record the new sha1<->newhash mapping entry for use in the idx.
+4. sort: reorder entries in the new pack to match the order of objects
+   in the pack the server generated and include blobs. Write a newhash idx
+   file
+5. clean up: remove the SHA-1 based pack file, index, and
+   topologically sorted list obtained from the server in steps 1
+   and 2.
+
+Step 3 requires every object referenced by the new object to be in the
+translation table. This is why the topological sort step is necessary.
+
+As an optimization, step 1 could write a file describing what non-blob
+objects each object it has inflated from the packfile references. This
+makes the topological sort in step 2 possible without inflating the
+objects in the packfile for a second time. The objects need to be
+inflated again in step 3, for a total of two inflations.
+
+Step 4 is probably necessary for good read-time performance. "git
+pack-objects" on the server optimizes the pack file for good data
+locality (see Documentation/technical/pack-heuristics.txt).
+
+Details of this process are likely to change. It will take some
+experimenting to get this to perform well.
+
+Push
+~~~~
+Push is simpler than fetch because the objects referenced by the
+pushed objects are already in the translation table. The sha1-content
+of each object being pushed can be read as described in the "Reading
+an object's sha1-content" section to generate the pack written by git
+send-pack.
+
+Signed Commits
+~~~~~~~~~~~~~~
+We add a new field "gpgsig-newhash" to the commit object format to allow
+signing commits without relying on SHA-1. It is similar to the
+existing "gpgsig" field. Its signed payload is the newhash-content of the
+commit object with any "gpgsig" and "gpgsig-newhash" fields removed.
+
+This means commits can be signed
+1. using SHA-1 only, as in existing signed commit objects
+2. using both SHA-1 and NewHash, by using both gpgsig-newhash and gpgsig
+   fields.
+3. using only NewHash, by only using the gpgsig-newhash field.
+
+Old versions of "git verify-commit" can verify the gpgsig signature in
+cases (1) and (2) without modifications and view case (3) as an
+ordinary unsigned commit.
+
+Signed Tags
+~~~~~~~~~~~
+We add a new field "gpgsig-newhash" to the tag object format to allow
+signing tags without relying on SHA-1. Its signed payload is the
+newhash-content of the tag with its gpgsig-newhash field and "-----BEGIN PGP
+SIGNATURE-----" delimited in-body signature removed.
+
+This means tags can be signed
+1. using SHA-1 only, as in existing signed tag objects
+2. using both SHA-1 and NewHash, by using gpgsig-newhash and an in-body
+   signature.
+3. using only NewHash, by only using the gpgsig-newhash field.
+
+Mergetag embedding
+~~~~~~~~~~~~~~~~~~
+The mergetag field in the sha1-content of a commit contains the
+sha1-content of a tag that was merged by that commit.
+
+The mergetag field in the newhash-content of the same commit contains the
+newhash-content of the same tag.
+
+Submodules
+~~~~~~~~~~
+To convert recorded submodule pointers, you need to have the converted
+submodule repository in place. The translation table of the submodule
+can be used to look up the new hash.
+
+Loose objects and unreachable objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Fast lookups in the loose-object-idx require that the number of loose
+objects not grow too high.
+
+"git gc --auto" currently waits for there to be 6700 loose objects
+present before consolidating them into a packfile. We will need to
+measure to find a more appropriate threshold for it to use.
+
+"git gc --auto" currently waits for there to be 50 packs present
+before combining packfiles. Packing loose objects more aggressively
+may cause the number of pack files to grow too quickly. This can be
+mitigated by using a strategy similar to Martin Fick's exponential
+rolling garbage collection script:
+https://gerrit-review.googlesource.com/c/gerrit/+/35215
+
+"git gc" currently expels any unreachable objects it encounters in
+pack files to loose objects in an attempt to prevent a race when
+pruning them (in case another process is simultaneously writing a new
+object that refers to the about-to-be-deleted object). This leads to
+an explosion in the number of loose objects present and disk space
+usage due to the objects in delta form being replaced with independent
+loose objects.  Worse, the race is still present for loose objects.
+
+Instead, "git gc" will need to move unreachable objects to a new
+packfile marked as UNREACHABLE_GARBAGE (using the PSRC field; see
+below). To avoid the race when writing new objects referring to an
+about-to-be-deleted object, code paths that write new objects will
+need to copy any objects from UNREACHABLE_GARBAGE packs that they
+refer to to new, non-UNREACHABLE_GARBAGE packs (or loose objects).
+UNREACHABLE_GARBAGE are then safe to delete if their creation time (as
+indicated by the file's mtime) is long enough ago.
+
+To avoid a proliferation of UNREACHABLE_GARBAGE packs, they can be
+combined under certain circumstances. If "gc.garbageTtl" is set to
+greater than one day, then packs created within a single calendar day,
+UTC, can be coalesced together. The resulting packfile would have an
+mtime before midnight on that day, so this makes the effective maximum
+ttl the garbageTtl + 1 day. If "gc.garbageTtl" is less than one day,
+then we divide the calendar day into intervals one-third of that ttl
+in duration. Packs created within the same interval can be coalesced
+together. The resulting packfile would have an mtime before the end of
+the interval, so this makes the effective maximum ttl equal to the
+garbageTtl * 4/3.
+
+This rule comes from Thirumala Reddy Mutchukota's JGit change
+https://git.eclipse.org/r/90465.
+
+The UNREACHABLE_GARBAGE setting goes in the PSRC field of the pack
+index. More generally, that field indicates where a pack came from:
+
+ - 1 (PACK_SOURCE_RECEIVE) for a pack received over the network
+ - 2 (PACK_SOURCE_AUTO) for a pack created by a lightweight
+   "gc --auto" operation
+ - 3 (PACK_SOURCE_GC) for a pack created by a full gc
+ - 4 (PACK_SOURCE_UNREACHABLE_GARBAGE) for potential garbage
+   discovered by gc
+ - 5 (PACK_SOURCE_INSERT) for locally created objects that were
+   written directly to a pack file, e.g. from "git add ."
+
+This information can be useful for debugging and for "gc --auto" to
+make appropriate choices about which packs to coalesce.
+
+Caveats
+-------
+Invalid objects
+~~~~~~~~~~~~~~~
+The conversion from sha1-content to newhash-content retains any
+brokenness in the original object (e.g., tree entry modes encoded with
+leading 0, tree objects whose paths are not sorted correctly, and
+commit objects without an author or committer). This is a deliberate
+feature of the design to allow the conversion to round-trip.
+
+More profoundly broken objects (e.g., a commit with a truncated "tree"
+header line) cannot be converted but were not usable by current Git
+anyway.
+
+Shallow clone and submodules
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Because it requires all referenced objects to be available in the
+locally generated translation table, this design does not support
+shallow clone or unfetched submodules. Protocol improvements might
+allow lifting this restriction.
+
+Alternates
+~~~~~~~~~~
+For the same reason, a newhash repository cannot borrow objects from a
+sha1 repository using objects/info/alternates or
+$GIT_ALTERNATE_OBJECT_REPOSITORIES.
+
+git notes
+~~~~~~~~~
+The "git notes" tool annotates objects using their sha1-name as key.
+This design does not describe a way to migrate notes trees to use
+newhash-names. That migration is expected to happen separately (for
+example using a file at the root of the notes tree to describe which
+hash it uses).
+
+Server-side cost
+~~~~~~~~~~~~~~~~
+Until Git protocol gains NewHash support, using NewHash based storage
+on public-facing Git servers is strongly discouraged. Once Git
+protocol gains NewHash support, NewHash based servers are likely not
+to support SHA-1 compatibility, to avoid what may be a very expensive
+hash reencode during clone and to encourage peers to modernize.
+
+The design described here allows fetches by SHA-1 clients of a
+personal NewHash repository because it's not much more difficult than
+allowing pushes from that repository. This support needs to be guarded
+by a configuration option --- servers like git.kernel.org that serve a
+large number of clients would not be expected to bear that cost.
+
+Meaning of signatures
+~~~~~~~~~~~~~~~~~~~~~
+The signed payload for signed commits and tags does not explicitly
+name the hash used to identify objects. If some day Git adopts a new
+hash function with the same length as the current SHA-1 (40
+hexadecimal digit) or NewHash (64 hexadecimal digit) objects then the
+intent behind the PGP signed payload in an object signature is
+unclear:
+
+	object e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7
+	type commit
+	tag v2.12.0
+	tagger Junio C Hamano <gitster@pobox.com> 1487962205 -0800
+
+	Git 2.12
+
+Does this mean Git v2.12.0 is the commit with sha1-name
+e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7 or the commit with
+new-40-digit-hash-name e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7?
+
+Fortunately NewHash and SHA-1 have different lengths. If Git starts
+using another hash with the same length to name objects, then it will
+need to change the format of signed payloads using that hash to
+address this issue.
+
+Object names on the command line
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+To support the transition (see Transition plan below), this design
+supports four different modes of operation:
+
+ 1. ("dark launch") Treat object names input by the user as SHA-1 and
+    convert any object names written to output to SHA-1, but store
+    objects using NewHash.  This allows users to test the code with no
+    visible behavior change except for performance.  This allows
+    allows running even tests that assume the SHA-1 hash function, to
+    sanity-check the behavior of the new mode.
+
+ 2. ("early transition") Allow both SHA-1 and NewHash object names in
+    input. Any object names written to output use SHA-1. This allows
+    users to continue to make use of SHA-1 to communicate with peers
+    (e.g. by email) that have not migrated yet and prepares for mode 3.
+
+ 3. ("late transition") Allow both SHA-1 and NewHash object names in
+    input. Any object names written to output use NewHash. In this
+    mode, users are using a more secure object naming method by
+    default.  The disruption is minimal as long as most of their peers
+    are in mode 2 or mode 3.
+
+ 4. ("post-transition") Treat object names input by the user as
+    NewHash and write output using NewHash. This is safer than mode 3
+    because there is less risk that input is incorrectly interpreted
+    using the wrong hash function.
+
+The mode is specified in configuration.
+
+The user can also explicitly specify which format to use for a
+particular revision specifier and for output, overriding the mode. For
+example:
+
+git --output-format=sha1 log abac87a^{sha1}..f787cac^{newhash}
+
+Selection of a New Hash
+-----------------------
+In early 2005, around the time that Git was written,  Xiaoyun Wang,
+Yiqun Lisa Yin, and Hongbo Yu announced an attack finding SHA-1
+collisions in 2^69 operations. In August they published details.
+Luckily, no practical demonstrations of a collision in full SHA-1 were
+published until 10 years later, in 2017.
+
+The hash function NewHash to replace SHA-1 should be stronger than
+SHA-1 was: we would like it to be trustworthy and useful in practice
+for at least 10 years.
+
+Some other relevant properties:
+
+1. A 256-bit hash (long enough to match common security practice; not
+   excessively long to hurt performance and disk usage).
+
+2. High quality implementations should be widely available (e.g. in
+   OpenSSL).
+
+3. The hash function's properties should match Git's needs (e.g. Git
+   requires collision and 2nd preimage resistance and does not require
+   length extension resistance).
+
+4. As a tiebreaker, the hash should be fast to compute (fortunately
+   many contenders are faster than SHA-1).
+
+Some hashes under consideration are SHA-256, SHA-512/256, SHA-256x16,
+K12, and BLAKE2bp-256.
+
+Transition plan
+---------------
+Some initial steps can be implemented independently of one another:
+- adding a hash function API (vtable)
+- teaching fsck to tolerate the gpgsig-newhash field
+- excluding gpgsig-* from the fields copied by "git commit --amend"
+- annotating tests that depend on SHA-1 values with a SHA1 test
+  prerequisite
+- using "struct object_id", GIT_MAX_RAWSZ, and GIT_MAX_HEXSZ
+  consistently instead of "unsigned char *" and the hardcoded
+  constants 20 and 40.
+- introducing index v3
+- adding support for the PSRC field and safer object pruning
+
+
+The first user-visible change is the introduction of the objectFormat
+extension (without compatObjectFormat). This requires:
+- implementing the loose-object-idx
+- teaching fsck about this mode of operation
+- using the hash function API (vtable) when computing object names
+- signing objects and verifying signatures
+- rejecting attempts to fetch from or push to an incompatible
+  repository
+
+Next comes introduction of compatObjectFormat:
+- translating object names between object formats
+- translating object content between object formats
+- generating and verifying signatures in the compat format
+- adding appropriate index entries when adding a new object to the
+  object store
+- --output-format option
+- ^{sha1} and ^{newhash} revision notation
+- configuration to specify default input and output format (see
+  "Object names on the command line" above)
+
+The next step is supporting fetches and pushes to SHA-1 repositories:
+- allow pushes to a repository using the compat format
+- generate a topologically sorted list of the SHA-1 names of fetched
+  objects
+- convert the fetched packfile to newhash format and generate an idx
+  file
+- re-sort to match the order of objects in the fetched packfile
+
+The infrastructure supporting fetch also allows converting an existing
+repository. In converted repositories and new clones, end users can
+gain support for the new hash function without any visible change in
+behavior (see "dark launch" in the "Object names on the command line"
+section). In particular this allows users to verify NewHash signatures
+on objects in the repository, and it should ensure the transition code
+is stable in production in preparation for using it more widely.
+
+Over time projects would encourage their users to adopt the "early
+transition" and then "late transition" modes to take advantage of the
+new, more futureproof NewHash object names.
+
+When objectFormat and compatObjectFormat are both set, commands
+generating signatures would generate both SHA-1 and NewHash signatures
+by default to support both new and old users.
+
+In projects using NewHash heavily, users could be encouraged to adopt
+the "post-transition" mode to avoid accidentally making implicit use
+of SHA-1 object names.
+
+Once a critical mass of users have upgraded to a version of Git that
+can verify NewHash signatures and have converted their existing
+repositories to support verifying them, we can add support for a
+setting to generate only NewHash signatures. This is expected to be at
+least a year later.
+
+That is also a good moment to advertise the ability to convert
+repositories to use NewHash only, stripping out all SHA-1 related
+metadata. This improves performance by eliminating translation
+overhead and security by avoiding the possibility of accidentally
+relying on the safety of SHA-1.
+
+Updating Git's protocols to allow a server to specify which hash
+functions it supports is also an important part of this transition. It
+is not discussed in detail in this document but this transition plan
+assumes it happens. :)
+
+Alternatives considered
+-----------------------
+Upgrading everyone working on a particular project on a flag day
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Projects like the Linux kernel are large and complex enough that
+flipping the switch for all projects based on the repository at once
+is infeasible.
+
+Not only would all developers and server operators supporting
+developers have to switch on the same flag day, but supporting tooling
+(continuous integration, code review, bug trackers, etc) would have to
+be adapted as well. This also makes it difficult to get early feedback
+from some project participants testing before it is time for mass
+adoption.
+
+Using hash functions in parallel
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+(e.g. https://public-inbox.org/git/22708.8913.864049.452252@chiark.greenend.org.uk/ )
+Objects newly created would be addressed by the new hash, but inside
+such an object (e.g. commit) it is still possible to address objects
+using the old hash function.
+* You cannot trust its history (needed for bisectability) in the
+  future without further work
+* Maintenance burden as the number of supported hash functions grows
+  (they will never go away, so they accumulate). In this proposal, by
+  comparison, converted objects lose all references to SHA-1.
+
+Signed objects with multiple hashes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Instead of introducing the gpgsig-newhash field in commit and tag objects
+for newhash-content based signatures, an earlier version of this design
+added "hash newhash <newhash-name>" fields to strengthen the existing
+sha1-content based signatures.
+
+In other words, a single signature was used to attest to the object
+content using both hash functions. This had some advantages:
+* Using one signature instead of two speeds up the signing process.
+* Having one signed payload with both hashes allows the signer to
+  attest to the sha1-name and newhash-name referring to the same object.
+* All users consume the same signature. Broken signatures are likely
+  to be detected quickly using current versions of git.
+
+However, it also came with disadvantages:
+* Verifying a signed object requires access to the sha1-names of all
+  objects it references, even after the transition is complete and
+  translation table is no longer needed for anything else. To support
+  this, the design added fields such as "hash sha1 tree <sha1-name>"
+  and "hash sha1 parent <sha1-name>" to the newhash-content of a signed
+  commit, complicating the conversion process.
+* Allowing signed objects without a sha1 (for after the transition is
+  complete) complicated the design further, requiring a "nohash sha1"
+  field to suppress including "hash sha1" fields in the newhash-content
+  and signed payload.
+
+Lazily populated translation table
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Some of the work of building the translation table could be deferred to
+push time, but that would significantly complicate and slow down pushes.
+Calculating the sha1-name at object creation time at the same time it is
+being streamed to disk and having its newhash-name calculated should be
+an acceptable cost.
+
+Document History
+----------------
+
+2017-03-03
+bmwill@google.com, jonathantanmy@google.com, jrnieder@gmail.com,
+sbeller@google.com
+
+Initial version sent to
+http://public-inbox.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com
+
+2017-03-03 jrnieder@gmail.com
+Incorporated suggestions from jonathantanmy and sbeller:
+* describe purpose of signed objects with each hash type
+* redefine signed object verification using object content under the
+  first hash function
+
+2017-03-06 jrnieder@gmail.com
+* Use SHA3-256 instead of SHA2 (thanks, Linus and brian m. carlson).[1][2]
+* Make sha3-based signatures a separate field, avoiding the need for
+  "hash" and "nohash" fields (thanks to peff[3]).
+* Add a sorting phase to fetch (thanks to Junio for noticing the need
+  for this).
+* Omit blobs from the topological sort during fetch (thanks to peff).
+* Discuss alternates, git notes, and git servers in the caveats
+  section (thanks to Junio Hamano, brian m. carlson[4], and Shawn
+  Pearce).
+* Clarify language throughout (thanks to various commenters,
+  especially Junio).
+
+2017-09-27 jrnieder@gmail.com, sbeller@google.com
+* use placeholder NewHash instead of SHA3-256
+* describe criteria for picking a hash function.
+* include a transition plan (thanks especially to Brandon Williams
+  for fleshing these ideas out)
+* define the translation table (thanks, Shawn Pearce[5], Jonathan
+  Tan, and Masaya Suzuki)
+* avoid loose object overhead by packing more aggressively in
+  "git gc --auto"
+
+[1] http://public-inbox.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/
+[2] http://public-inbox.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/
+[3] http://public-inbox.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/
+[4] http://public-inbox.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net
+[5] https://public-inbox.org/git/CAJo=hJtoX9=AyLHHpUJS7fueV9ciZ_MNpnEPHUz8Whui6g9F0A@mail.gmail.com/
-- 
2.14.2.822.g60be5d43e6-goog


^ permalink raw reply	[relevance 4%]

* Re: [RFC] clang-format: outline the git project's coding style
  2017-08-09 22:33       ` Johannes Schindelin
@ 2017-09-28 11:41         ` Johannes Schindelin
  2017-09-28 17:16           ` Brandon Williams
  0 siblings, 1 reply; 200+ results
From: Johannes Schindelin @ 2017-09-28 11:41 UTC (permalink / raw)
  To: Brandon Williams; +Cc: Stefan Beller, git

Hi all,

On Thu, 10 Aug 2017, Johannes Schindelin wrote:

> On Tue, 8 Aug 2017, Brandon Williams wrote:
> 
> > On 08/08, Stefan Beller wrote:
> > > On Tue, Aug 8, 2017 at 5:05 AM, Johannes Schindelin
> > > <Johannes.Schindelin@gmx.de> wrote:
> > > >
> > > > On Mon, 7 Aug 2017, Brandon Williams wrote:
> > > >
> > > >> Add a '.clang-format' file which outlines the git project's
> > > >> coding style.  This can be used with clang-format to auto-format
> > > >> .c and .h files to conform with git's style.
> > > >>
> > > >> Signed-off-by: Brandon Williams <bmwill@google.com>
> > > >> ---
> > > >>
> > > >> I'm sure this sort of thing comes up every so often on the list
> > > >> but back at git-merge I mentioned how it would be nice to not
> > > >> have to worry about style when reviewing patches as that is
> > > >> something mechanical and best left to a machine (for the most
> > > >> part).
> > > >
> > > > Amen.
> > > >
> > > > If I never have to see a review mentioning an unwrapped line, I am quite
> > > > certain I will be quite content.
> > > 
> > > As a thought experiment I'd like to propose to take it one step further:
> > > 
> > >   If the code was formatted perfectly in one style such that a
> > >   formatter for this style would not produce changes when rerun
> > >   again on the code, then each individual could have a clean/smudge
> > >   filter to work in their preferred style, and only the exchange and
> > >   storage of code is in a mutual agreed style. If the mutually
> > >   agreed style is close to what I prefer, I don't have to use
> > >   clean/smudge filters.
> > > 
> > > Additionally to this patch, we'd want to either put a note into
> > > SubmittingPatches or Documentation/gitworkflows.txt to hint at
> > > how to use this formatting to just affect the patch that is currently
> > > worked on or rather a pre-commit hook?
> > 
> > I'm sure both of these things will probably happen if we decide to make
> > use of a code-formatter.  This RFC is really just trying to ask the
> > question: "Is this something we want to entertain doing?"
> > My response would be 'Yes' but there's many other opinions to consider
> > first :)
> 
> I am sure that something even better will be possible: a Continuous
> "Integration" that fixes the coding style automatically by using
> `filter-branch` (avoiding the merge conflicts that would arise if `rebase
> -i` was used).

FWIW I just set this up in my VSTS account, with the following build step
(performed in one of the Hosted Linux agents, i.e. I do not even have to
have a VM dedicated to the task):

-- snip --
die () {
    echo "$*" >&2
    exit 1
}

head=$(git rev-parse HEAD) ||
die "Could not determine HEAD"

test -n "$BUILD_SOURCEBRANCHNAME" ||
die "Need a source branch name to work with"

base=$(git merge-base origin/core/master HEAD) &&
count=$(git rev-list --count $base..) &&
test 0 -lt $count ||
die "Could not determine commits to clean up (count: $count)"

test -f .clang-format ||
git show origin/core/pu/.clang-format >.clang-format ||
die "Need a .clang-format"

sudo add-apt-repository 'deb http://apt.llvm.org/xenial/
llvm-toolchain-xenial main' &&
sudo apt-get update &&
sudo apt-get --allow-unauthenticated -y install clang-format-6.0 ||
die "Could not install clang-format 6.0"

git filter-branch -f --tree-filter \
    'git diff HEAD^.. |
     clang-format-diff-6.0 -p 1 -i &&

     git update-index --refresh --ignore-submodules &&
     git diff-files --quiet --ignore-submodules ||
     git commit --amend -C HEAD' $base.. ||
die "Could not rewrite branch"

if test "$head" = "$(git rev-parse HEAD)"
then
    echo "No changes in $BUILD_SOURCEBRANCHNAME introduced by clang-format" >&2
else
    git push vsts +HEAD:refs/heads/clang-format/"$BUILD_SOURCEBRANCHNAME" ||
    die "Could not push clang-format/$BUILD_SOURCEBRANCHNAME"
    echo "Clean branch pushed as clang-format/$BUILD_SOURCEBRANCHNAME" >&2
    exit 123
fi
-- snap --

A couple of notes for the interested:

- you can easily set this up for yourself, as Visual Studio Team Services
  is free for small teams up to five people (including single developer
  "teams"): https://www.visualstudio.com/team-services/, and of course you
  can back it by your own git.git fork on GitHub, no need to host the code
  in VSTS.

  Disclaimer: I work closely with the developers behind Visual Studio Team
  Services, and I am a genuine fan, yet I understand if anybody thinks of
  this as advertising the service, so this will be the only time I mention
  this.

- the script assumes that there is a `core/master` tracking upstream Git's
  master branch, then reformats the commits in the current branch that are
  not also reachable from core/master.

- The push credentials to push the result at the end are of course not
  included in the script, they need to be provided separately.

- the exit code 123 when the branch needed to be rewritten indicates to
  any consumer that the build "failed". The reason is that I want to
  integrate this into a system where I open a PR in my own account, which
  triggers the build automatically contingent on the base branch being
  core/master, and if the build fails, the PR gets "blocked", providing a
  very easy way to see that there is still work to be done.

- for the moment, I do not push back to the original branch, even if I
  could. The reason is that already my first test produced a dubious
  result, see below.

I am reasonably happy with the way this build job works right now,
especially given that I do not have to mess up any other setup I have just
to get the bleeding edge version of Clang.

Now for the dubious result.

I took my most recent contribution, the lazyload one (which you can easily
get yourself by fetching the lazyload-v2 tag from
https://github.com/dscho/git), because it was pretty self-contained and
small, only one patch. With the current .clang-format as per git.git's
master (or for that matter, pu, as they are identical), the output `git
show | clang-format-diff-6.0 -p 1` ends in this hunk:

-- snip --
@@ -43,8 +43,7 @@
        if (!proc->initialized) {
                HANDLE hnd;
                proc->initialized = 1;
-               hnd = LoadLibraryExA(proc->dll, NULL,
-                                    LOAD_LIBRARY_SEARCH_SYSTEM32);
+               hnd = LoadLibraryExA(proc->dll, NULL, LOAD_LIBRARY_SEARCH_SYSTEM32);
                if (hnd)
                        proc->pfunction = GetProcAddress(hnd, proc->function);
        }
-- snap --

(tabs intentionally converted to spaces, to show the extent of the damage
clearly)

In other words, despite the column limit of 80 (with a tab width of 8
spaces), it takes a perfectly well formatted pair of lines and combines
them into a single line that now has 84 columns. Absolutely not what we
want.

Even worse: if I replace the column limit of 80 by 79, like so:

-- snip --
diff --git a/.clang-format b/.clang-format
index 3ede2628d2d..9f686c1ed5a 100644
--- a/.clang-format
+++ b/.clang-format
@@ -6,7 +6,7 @@ UseTab: Always
 TabWidth: 8
 IndentWidth: 8
 ContinuationIndentWidth: 8
-ColumnLimit: 80
+ColumnLimit: 79
 
 # C Language specifics
 Language: Cpp
-- snap --

then that hunk vanishes and clang-format leaves the LoadLibraryEx() lines
alone!

Even stranger, if I revert to 80 columns, copy the offending line above
the conditional block so that the indentation level is different (based on
a hunch that this may have something to do with clang's understanding of
tab widths) and extend the last parameter artificially, it still breaks at
exactly 84 columns, i.e. if I make the line 84 columns long, it keeps it
as one line, if I extend it to 85 columns, it breaks the line into two.

In fact, the same holds true even with no indentation at all: if I turn
this line into a static variable assignment outside of the function, it
again breaks the line as soon as I extend it to 85 columns.

Then I repeated the exercise with clang-format-6.0 instead of
clang-format-diff-6.0, and the finding still holds. 85 columns, despite
the explicit ColumnLimit: 80 in .clang-format.

I then tried to format a file containing only the line "int i123, j123;"
with various values for ColumnLimit, and could not get it to break at all.

Any insights?

Thanks,
Dscho

^ permalink raw reply	[relevance 6%]

* Re: [RFC] clang-format: outline the git project's coding style
  2017-09-28 11:41         ` Johannes Schindelin
@ 2017-09-28 17:16           ` Brandon Williams
  0 siblings, 0 replies; 200+ results
From: Brandon Williams @ 2017-09-28 17:16 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: Stefan Beller, git

On 09/28, Johannes Schindelin wrote:
> Hi all,
>
> On Thu, 10 Aug 2017, Johannes Schindelin wrote:
>
> > On Tue, 8 Aug 2017, Brandon Williams wrote:
> >
> > > On 08/08, Stefan Beller wrote:
> > > > On Tue, Aug 8, 2017 at 5:05 AM, Johannes Schindelin
> > > > <Johannes.Schindelin@gmx.de> wrote:
> > > > >
> > > > > On Mon, 7 Aug 2017, Brandon Williams wrote:
> > > > >
> > > > >> Add a '.clang-format' file which outlines the git project's
> > > > >> coding style.  This can be used with clang-format to auto-format
> > > > >> .c and .h files to conform with git's style.
> > > > >>
> > > > >> Signed-off-by: Brandon Williams <bmwill@google.com>
> > > > >> ---
> > > > >>
> > > > >> I'm sure this sort of thing comes up every so often on the list
> > > > >> but back at git-merge I mentioned how it would be nice to not
> > > > >> have to worry about style when reviewing patches as that is
> > > > >> something mechanical and best left to a machine (for the most
> > > > >> part).
> > > > >
> > > > > Amen.
> > > > >
> > > > > If I never have to see a review mentioning an unwrapped line, I am quite
> > > > > certain I will be quite content.
> > > >
> > > > As a thought experiment I'd like to propose to take it one step further:
> > > >
> > > >   If the code was formatted perfectly in one style such that a
> > > >   formatter for this style would not produce changes when rerun
> > > >   again on the code, then each individual could have a clean/smudge
> > > >   filter to work in their preferred style, and only the exchange and
> > > >   storage of code is in a mutual agreed style. If the mutually
> > > >   agreed style is close to what I prefer, I don't have to use
> > > >   clean/smudge filters.
> > > >
> > > > Additionally to this patch, we'd want to either put a note into
> > > > SubmittingPatches or Documentation/gitworkflows.txt to hint at
> > > > how to use this formatting to just affect the patch that is currently
> > > > worked on or rather a pre-commit hook?
> > >
> > > I'm sure both of these things will probably happen if we decide to make
> > > use of a code-formatter.  This RFC is really just trying to ask the
> > > question: "Is this something we want to entertain doing?"
> > > My response would be 'Yes' but there's many other opinions to consider
> > > first :)
> >
> > I am sure that something even better will be possible: a Continuous
> > "Integration" that fixes the coding style automatically by using
> > `filter-branch` (avoiding the merge conflicts that would arise if `rebase
> > -i` was used).
>
> FWIW I just set this up in my VSTS account, with the following build step
> (performed in one of the Hosted Linux agents, i.e. I do not even have to
> have a VM dedicated to the task):
>
> -- snip --
> die () {
>     echo "$*" >&2
>     exit 1
> }
>
> head=$(git rev-parse HEAD) ||
> die "Could not determine HEAD"
>
> test -n "$BUILD_SOURCEBRANCHNAME" ||
> die "Need a source branch name to work with"
>
> base=$(git merge-base origin/core/master HEAD) &&
> count=$(git rev-list --count $base..) &&
> test 0 -lt $count ||
> die "Could not determine commits to clean up (count: $count)"
>
> test -f .clang-format ||
> git show origin/core/pu/.clang-format >.clang-format ||
> die "Need a .clang-format"
>
> sudo add-apt-repository 'deb http://apt.llvm.org/xenial/
> llvm-toolchain-xenial main' &&
> sudo apt-get update &&
> sudo apt-get --allow-unauthenticated -y install clang-format-6.0 ||
> die "Could not install clang-format 6.0"
>
> git filter-branch -f --tree-filter \
>     'git diff HEAD^.. |
>      clang-format-diff-6.0 -p 1 -i &&
>
>      git update-index --refresh --ignore-submodules &&
>      git diff-files --quiet --ignore-submodules ||
>      git commit --amend -C HEAD' $base.. ||
> die "Could not rewrite branch"
>
> if test "$head" = "$(git rev-parse HEAD)"
> then
>     echo "No changes in $BUILD_SOURCEBRANCHNAME introduced by clang-format" >&2
> else
>     git push vsts +HEAD:refs/heads/clang-format/"$BUILD_SOURCEBRANCHNAME" ||
>     die "Could not push clang-format/$BUILD_SOURCEBRANCHNAME"
>     echo "Clean branch pushed as clang-format/$BUILD_SOURCEBRANCHNAME" >&2
>     exit 123
> fi
> -- snap --
>
> A couple of notes for the interested:
>
> - you can easily set this up for yourself, as Visual Studio Team Services
>   is free for small teams up to five people (including single developer
>   "teams"): https://www.visualstudio.com/team-services/, and of course you
>   can back it by your own git.git fork on GitHub, no need to host the code
>   in VSTS.
>
>   Disclaimer: I work closely with the developers behind Visual Studio Team
>   Services, and I am a genuine fan, yet I understand if anybody thinks of
>   this as advertising the service, so this will be the only time I mention
>   this.
>
> - the script assumes that there is a `core/master` tracking upstream Git's
>   master branch, then reformats the commits in the current branch that are
>   not also reachable from core/master.
>
> - The push credentials to push the result at the end are of course not
>   included in the script, they need to be provided separately.
>
> - the exit code 123 when the branch needed to be rewritten indicates to
>   any consumer that the build "failed". The reason is that I want to
>   integrate this into a system where I open a PR in my own account, which
>   triggers the build automatically contingent on the base branch being
>   core/master, and if the build fails, the PR gets "blocked", providing a
>   very easy way to see that there is still work to be done.
>
> - for the moment, I do not push back to the original branch, even if I
>   could. The reason is that already my first test produced a dubious
>   result, see below.
>
> I am reasonably happy with the way this build job works right now,
> especially given that I do not have to mess up any other setup I have just
> to get the bleeding edge version of Clang.
>
> Now for the dubious result.
>
> I took my most recent contribution, the lazyload one (which you can easily
> get yourself by fetching the lazyload-v2 tag from
> https://github.com/dscho/git), because it was pretty self-contained and
> small, only one patch. With the current .clang-format as per git.git's
> master (or for that matter, pu, as they are identical), the output `git
> show | clang-format-diff-6.0 -p 1` ends in this hunk:
>
> -- snip --
> @@ -43,8 +43,7 @@
>         if (!proc->initialized) {
>                 HANDLE hnd;
>                 proc->initialized = 1;
> -               hnd = LoadLibraryExA(proc->dll, NULL,
> -                                    LOAD_LIBRARY_SEARCH_SYSTEM32);
> +               hnd = LoadLibraryExA(proc->dll, NULL, LOAD_LIBRARY_SEARCH_SYSTEM32);
>                 if (hnd)
>                         proc->pfunction = GetProcAddress(hnd, proc->function);
>         }
> -- snap --
>
> (tabs intentionally converted to spaces, to show the extent of the damage
> clearly)
>
> In other words, despite the column limit of 80 (with a tab width of 8
> spaces), it takes a perfectly well formatted pair of lines and combines
> them into a single line that now has 84 columns. Absolutely not what we
> want.
>
> Even worse: if I replace the column limit of 80 by 79, like so:
>
> -- snip --
> diff --git a/.clang-format b/.clang-format
> index 3ede2628d2d..9f686c1ed5a 100644
> --- a/.clang-format
> +++ b/.clang-format
> @@ -6,7 +6,7 @@ UseTab: Always
>  TabWidth: 8
>  IndentWidth: 8
>  ContinuationIndentWidth: 8
> -ColumnLimit: 80
> +ColumnLimit: 79
>
>  # C Language specifics
>  Language: Cpp
> -- snap --
>
> then that hunk vanishes and clang-format leaves the LoadLibraryEx() lines
> alone!
>
> Even stranger, if I revert to 80 columns, copy the offending line above
> the conditional block so that the indentation level is different (based on
> a hunch that this may have something to do with clang's understanding of
> tab widths) and extend the last parameter artificially, it still breaks at
> exactly 84 columns, i.e. if I make the line 84 columns long, it keeps it
> as one line, if I extend it to 85 columns, it breaks the line into two.
>
> In fact, the same holds true even with no indentation at all: if I turn
> this line into a static variable assignment outside of the function, it
> again breaks the line as soon as I extend it to 85 columns.
>
> Then I repeated the exercise with clang-format-6.0 instead of
> clang-format-diff-6.0, and the finding still holds. 85 columns, despite
> the explicit ColumnLimit: 80 in .clang-format.
>
> I then tried to format a file containing only the line "int i123, j123;"
> with various values for ColumnLimit, and could not get it to break at all.
>
> Any insights?

I think I know what is happening here, and this is something that we
will need to tweak based on more people beginning to use the formatting
tool.

In the .clang-format file there are a number of parameters at the end:

    # Penalties
    # This decides what order things should be done if a line is too long
    PenaltyBreakAssignment: 100
    PenaltyBreakBeforeFirstCallParameter: 100
    PenaltyBreakComment: 100
    PenaltyBreakFirstLessLess: 0
    PenaltyBreakString: 100
    PenaltyExcessCharacter: 5
    PenaltyReturnTypeOnItsOwnLine: 0

These penalties are used to determine which line breaking events should be
done based on some programmable weight (or penalty).  I thing that if we
bump up the penalty on excess characters to something higher than '5'
(which i think is quite low) then it may format more like you were
expecting.

--
Brandon Williams

^ permalink raw reply	[relevance 1%]

* Re: [PATCH v4] technical doc: add a design doc for hash function transition
  2017-09-28  4:43       ` [PATCH v4] technical doc: add a design doc for hash function transition Jonathan Nieder
@ 2017-09-29  6:06         ` Junio C Hamano
  2017-09-29 17:34           ` Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2017-09-29  6:06 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: Shawn Pearce, Linus Torvalds, Git Mailing List, Stefan Beller, bmwill, Jonathan Tan, Jeff King, David Lang, brian m. carlson, Masaya Suzuki, demerphq, The Keccak Team, Johannes Schindelin

Jonathan Nieder <jrnieder@gmail.com> writes:

> This document describes what a transition to a new hash function for
> Git would look like.  Add it to Documentation/technical/ as the plan
> of record so that future changes can be recorded as patches.
>
> Also-by: Brandon Williams <bmwill@google.com>
> Also-by: Jonathan Tan <jonathantanmy@google.com>
> Also-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---

Shoudln't these all be s-o-b: (with a note immediately before that
to say all four contributed equally or something)?

> +Background
> +----------
> +At its core, the Git version control system is a content addressable
> +filesystem. It uses the SHA-1 hash function to name content. For
> +example, files, directories, and revisions are referred to by hash
> +values unlike in other traditional version control systems where files
> +or versions are referred to via sequential numbers. The use of a hash

Traditional systems refer to files via numbers???  Perhaps "where
versions of files are referred to via sequential numbers" or
something?

> +function to address its content delivers a few advantages:
> +
> +* Integrity checking is easy. Bit flips, for example, are easily
> +  detected, as the hash of corrupted content does not match its name.
> +* Lookup of objects is fast.

* There is no ambiguity what the object's name should be, given its
  content.

* Deduping the same content copied across versions and paths is
  automatic.

> +SHA-1 still possesses the other properties such as fast object lookup
> +and safe error checking, but other hash functions are equally suitable
> +that are believed to be cryptographically secure.

s/secure/more &/, perhaps?

> +Goals
> +-----
> +...
> +   c. Users can use SHA-1 and NewHash identifiers for objects
> +      interchangeably (see "Object names on the command line", below).

Mental note.  This needs to extend to the "index X..Y" lines in the
patch output, which is used by "apply -3" and "am -3".

> +2. Allow a complete transition away from SHA-1.
> +   a. Local metadata for SHA-1 compatibility can be removed from a
> +      repository if compatibility with SHA-1 is no longer needed.

I like the emphasis on "Local" here.  Metadata for compatiblity that
is embedded in the objects obviously cannot be removed.

From that point of view, one of the goals ought to be "make sure
that as much SHA-1 compatibility metadata as possible is local and
outside the object".  This goal may not be able to say more than "as
much as possible", as signed objects that came from SHA-1 world
needs to carry the compatibility metadata somewhere somehow.  

Or perhaps we could.  There is nothing that says a signed tag
created in the SHA-1 world must have the PGP/SHA-1 signature in the
NewHash payload---it could be split off of the object data and
stored in a local metadata cache, to be used only when we need to
convert it back to the SHA-1 world.

But I am getting ahead of myself before reading the proposal
through.

> +Non-Goals
> +---------
> ...
> +6. Skip fetching some submodules of a project into a NewHash
> +   repository. (This also depends on NewHash support in Git
> +   protocol.)

It is unclear what this means.  Around submodule support, one thing
I can think of is that a NewHash tree in a superproject would record
a gitlink that is a NewHash commit object name in it, therefore it
cannot refer to an unconverted SHA-1 submodule repository.  But it
is unclear if the above description refers to the same issue, or
something else.

> +Overview
> +--------
> +We introduce a new repository format extension. Repositories with this
> +extension enabled use NewHash instead of SHA-1 to name their objects.
> +This affects both object names and object content --- both the names
> +of objects and all references to other objects within an object are
> +switched to the new hash function.
> +
> +NewHash repositories cannot be read by older versions of Git.
> +
> +Alongside the packfile, a NewHash repository stores a bidirectional
> +mapping between NewHash and SHA-1 object names. The mapping is generated
> +locally and can be verified using "git fsck". Object lookups use this
> +mapping to allow naming objects using either their SHA-1 and NewHash names
> +interchangeably.
> +
> +"git cat-file" and "git hash-object" gain options to display an object
> +in its sha1 form and write an object given its sha1 form.

Both of these are somewhat unclear.  I am guessing that "git
cat-file --convert-to=sha1 <type> <NewHashName>" would emit the
object contents converted from their NewHash payload to SHA-1
payload (blobs are unchanged, trees, commits and tags get their
outgoing references converted from NewHash to their SHA-1
counterparts), and that is what you mean by "options to display an
object in its sha1 form".  

I am not sure how "git hash-object" with the option would work,
though.  Do you give an option "--hash=sha1 --stdout --stdin -t
<type>" to feed a NewHash contents (file, tree, commit or tag) to
the command, convert it to the SHA-1 content (hmm, how's that
different from the cat-file's new option???) and then write out its
loose object representation suitable to be used in the SHA-1 workd?
Where do you write it to?  It won't be in the repository, as we
rejected mixed repository in our Non-Goals section.

> +Object names
> +~~~~~~~~~~~~
> +Objects can be named by their 40 hexadecimal digit sha1-name or 64
> +hexadecimal digit newhash-name, plus names derived from those (see
> +gitrevisions(7)).
> +
> +The sha1-name of an object is the SHA-1 of the concatenation of its
> +type, length, a nul byte, and the object's sha1-content. This is the
> +traditional <sha1> used in Git to name objects.
> +
> +The newhash-name of an object is the NewHash of the concatenation of its
> +type, length, a nul byte, and the object's newhash-content.

It makes me wonder if we want to add the hashname in this object
header.  "length" would be different for non-blob objects anyway,
and it is not "compat metadata" we want to avoid baked in, yet it
would help diagnose a mistake of attempting to use a "mixed" objects
in a single repository.  Not a big issue, though.

> +The format allows round-trip conversion between newhash-content and
> +sha1-content.

If it is a goal to eventually be able to lose SHA-1 compatibility
metadata from the objects, then we might want to remove SHA-1 based
signature bits (e.g. PGP trailer in signed tag, gpgsig header in the
commit object) from NewHash contents, and instead have them stored
in a side "metadata" table, only to be used while converting back.
I dunno if that is desirable.

> +Pack index
> +~~~~~~~~~~
> +Pack index (.idx) files use a new v3 format that supports multiple
> +hash functions. They have the following format (all integers are in
> +network byte order):
> +
> +- A header appears at the beginning and consists of the following:
> +  - The 4-byte pack index signature: '\377t0c'
> +  - 4-byte version number: 3
> +  - 4-byte length of the header section, including the signature and
> +    version number
> +  - 4-byte number of objects contained in the pack
> +  - 4-byte number of object formats in this pack index: 2
> +  - For each object format:
> +    - 4-byte format identifier (e.g., 'sha1' for SHA-1)
> +    - 4-byte length in bytes of shortened object names. This is the
> +      shortest possible length needed to make names in the shortened
> +      object name table unambiguous.
> +    - 4-byte integer, recording where tables relating to this format
> +      are stored in this index file, as an offset from the beginning.
> +  - 4-byte offset to the trailer from the beginning of this file.
> +  - Zero or more additional key/value pairs (4-byte key, 4-byte
> +    value). Only one key is supported: 'PSRC'. See the "Loose objects
> +    and unreachable objects" section for supported values and how this
> +    is used.  All other keys are reserved. Readers must ignore
> +    unrecognized keys.
> +- Zero or more NUL bytes. This can optionally be used to improve the
> +  alignment of the full object name table below.
> +- Tables for the first object format:
> +  - A sorted table of shortened object names.  These are prefixes of
> +    the names of all objects in this pack file, packed together
> +    without offset values to reduce the cache footprint of the binary
> +    search for a specific object name.

I take it to mean that the stride is defined in the "length in bytes
of shortened object names" in the file header.  If so, I can see how
this would work.  This "sorted table", unlike the next one, does not
say how it is sorted, but I assume this is just the object name
order (as opposed to the pack location order the next table uses)?

> +  - A table of full object names in pack order. This allows resolving
> +    a reference to "the nth object in the pack file" (from a
> +    reachability bitmap or from the next table of another object
> +    format) to its object name.
> +
> +  - A table of 4-byte values mapping object name order to pack order.
> +    For an object in the table of sorted shortened object names, the
> +    value at the corresponding index in this table is the index in the
> +    previous table for that same object.
> +
> +    This can be used to look up the object in reachability bitmaps or
> +    to look up its name in another object format.

And this is a separate table because the short-name table wants to
be as compact as possible for binary search?  Otherwise an entry in
the short-name table could be <pack order number, n-bytes that is
short unique prefix>.

> +  - A table of 4-byte CRC32 values of the packed object data, in the
> +    order that the objects appear in the pack file. This is to allow
> +    compressed data to be copied directly from pack to pack during
> +    repacking without undetected data corruption.

An obvious alternative would be to have the CRC32 checksum near
(e.g. immediately before) the object data in the packfile (as
opposed to the .idx file like this document specifies).  I am not
sure what the pros and cons are between the two, though, and that is
why I mention the possiblity here.

Hmm, as the corresponding packfile stores object data only in
NewHash content format, it is somewhat curious that this table that
stores CRC32 of the data appears in the "Tables for each object
format" section, as they would be identical, no?  Unless I am
grossly misleading the spec, the checksum should either go outside
the "Tables for each object format" section but still in .idx, or
should be eliminated and become part of the packdata stream instead,
perhaps?

> +  - A table of 4-byte offset values. For an object in the table of
> +    sorted shortened object names, the value at the corresponding
> +    index in this table indicates where that object can be found in
> +    the pack file. These are usually 31-bit pack file offsets, but
> +    large offsets are encoded as an index into the next table with the
> +    most significant bit set.

Oy.  So we can go from a short prefix to the pack location by first
finding it via binsearch in the short-name table, realize that it is
nth object in the object name order, and consulting this table.
When we know the pack-order of an object, there is no direct way to
go to its location (short of reversing the name-order-to-pack-order
table)?

> +  - A table of 8-byte offset entries (empty for pack files less than
> +    2 GiB). Pack files are organized with heavily used objects toward
> +    the front, so most object references should not need to refer to
> +    this table.

> +- Zero or more NUL bytes.

... for padding/aligning.

> +- Tables for the second object format, with the same layout as above,
> +  up to and not including the table of CRC32 values.
> +- Zero or more NUL bytes.
> +- The trailer consists of the following:
> +  - A copy of the 20-byte NewHash checksum at the end of the
> +    corresponding packfile.
> +
> +  - 20-byte NewHash checksum of all of the above.

When did NewHash shrink to 20-byte suddenly?  I think the above two
are both "32-byte"?

> +Loose object index
> +~~~~~~~~~~~~~~~~~~
> +A new file $GIT_OBJECT_DIR/loose-object-idx contains information about
> +all loose objects. Its format is
> +
> +  # loose-object-idx
> +  (newhash-name SP sha1-name LF)*
> +
> +where the object names are in hexadecimal format. The file is not
> +sorted.

Shouldn't the file somehow say what hashes are involved to allow us
match it with extension.{objectFormat,compatObjectFormat}, perhaps
at the end of the "# loose-object-idx" line?

> +The loose object index is protected against concurrent writes by a
> +lock file $GIT_OBJECT_DIR/loose-object-idx.lock. To add a new loose
> +object:
> +
> +1. Write the loose object to a temporary file, like today.
> +2. Open loose-object-idx.lock with O_CREAT | O_EXCL to acquire the lock.
> +3. Rename the loose object into place.
> +4. Open loose-object-idx with O_APPEND and write the new object

"write the new entry, fsync and close"?

> +Translation table
> +~~~~~~~~~~~~~~~~~
> +The index files support a bidirectional mapping between sha1-names
> +and newhash-names. The lookup proceeds similarly to ordinary object
> +lookups. For example, to convert a sha1-name to a newhash-name:
> +
> + 1. Look for the object in idx files. If a match is present in the
> +    idx's sorted list of truncated sha1-names, then:
> +    a. Read the corresponding entry in the sha1-name order to pack
> +       name order mapping.
> +    b. Read the corresponding entry in the full sha1-name table to
> +       verify we found the right object. If it is, then
> +    c. Read the corresponding entry in the full newhash-name table.
> +       That is the object's newhash-name.

c. is possible because b. and c. are sorted the same way, i.e. the
index used to consult the full sha1-name table, which is the pack
order number, can be used to find its full newhash in the "full
newhash sorted by pack order" table?

> +Reading an object's sha1-content
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I'd stop here and continue in a separate message.  Thanks for a
detailed write-up.

^ permalink raw reply	[relevance 5%]

* [PATCH v6 1/3] submodule--helper: introduce get_submodule_displaypath()
  2017-09-29  9:44 ` [PATCH v6 0/3] Incremental rewrite of git-submodules Prathamesh Chavan
@ 2017-09-29  9:44   ` Prathamesh Chavan
  2017-09-29  9:44   ` [PATCH v6 2/3] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
  2017-09-29  9:44   ` [PATCH v6 3/3] submodule: port submodule subcommand 'status' from shell to C Prathamesh Chavan
  2 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-09-29  9:44 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, hanwen, sbeller, Prathamesh Chavan

Introduce function get_submodule_displaypath() to replace the code
occurring in submodule_init() for generating displaypath of the
submodule with a call to it.

This new function will also be used in other parts of the system
in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 35 +++++++++++++++++++++++------------
 1 file changed, 23 insertions(+), 12 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 818fe74f0..cdae54426 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -220,6 +220,26 @@ static int resolve_relative_url_test(int argc, const char **argv, const char *pr
 	return 0;
 }
 
+/* the result should be freed by the caller. */
+static char *get_submodule_displaypath(const char *path, const char *prefix)
+{
+	const char *super_prefix = get_super_prefix();
+
+	if (prefix && super_prefix) {
+		BUG("cannot have prefix '%s' and superprefix '%s'",
+		    prefix, super_prefix);
+	} else if (prefix) {
+		struct strbuf sb = STRBUF_INIT;
+		char *displaypath = xstrdup(relative_path(path, prefix, &sb));
+		strbuf_release(&sb);
+		return displaypath;
+	} else if (super_prefix) {
+		return xstrfmt("%s%s", super_prefix, path);
+	} else {
+		return xstrdup(path);
+	}
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -335,15 +355,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	if (prefix && get_super_prefix())
-		die("BUG: cannot have prefix and superprefix");
-	else if (prefix)
-		displaypath = xstrdup(relative_path(path, prefix, &sb));
-	else if (get_super_prefix()) {
-		strbuf_addf(&sb, "%s%s", get_super_prefix(), path);
-		displaypath = strbuf_detach(&sb, NULL);
-	} else
-		displaypath = xstrdup(path);
+	displaypath = get_submodule_displaypath(path, prefix);
 
 	sub = submodule_from_path(&null_oid, path);
 
@@ -358,9 +370,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * Set active flag for the submodule being initialized
 	 */
 	if (!is_submodule_active(the_repository, path)) {
-		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
+		strbuf_reset(&sb);
 	}
 
 	/*
@@ -368,7 +380,6 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * To look up the url in .git/config, we must not fall back to
 	 * .gitmodules, so look it up directly.
 	 */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.url", sub->name);
 	if (git_config_get_string(sb.buf, &url)) {
 		if (!sub->url)
@@ -405,9 +416,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
 	}
+	strbuf_reset(&sb);
 
 	/* Copy "update" setting when it is not set yet */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.update", sub->name);
 	if (git_config_get_string(sb.buf, &upd) &&
 	    sub->update_strategy.type != SM_UPDATE_UNSPECIFIED) {
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [PATCH v6 2/3] submodule--helper: introduce for_each_listed_submodule()
  2017-09-29  9:44 ` [PATCH v6 0/3] Incremental rewrite of git-submodules Prathamesh Chavan
  2017-09-29  9:44   ` [PATCH v6 1/3] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
@ 2017-09-29  9:44   ` Prathamesh Chavan
  2017-10-02  0:55     ` Junio C Hamano
  2017-09-29  9:44   ` [PATCH v6 3/3] submodule: port submodule subcommand 'status' from shell to C Prathamesh Chavan
  2 siblings, 1 reply; 200+ results
From: Prathamesh Chavan @ 2017-09-29  9:44 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, hanwen, sbeller, Prathamesh Chavan

Introduce function for_each_listed_submodule() and replace a loop
in module_init() with a call to it.

The new function will also be used in other parts of the
system in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 39 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 34 insertions(+), 5 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index cdae54426..20a1ef868 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -14,6 +14,11 @@
 #include "refs.h"
 #include "connect.h"
 
+#define CB_OPT_QUIET		(1<<0)
+
+typedef void (*each_submodule_fn)(const struct cache_entry *list_item,
+				  void *cb_data);
+
 static char *get_default_remote(void)
 {
 	char *dest = NULL, *ret;
@@ -349,7 +354,22 @@ static int module_list(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
-static void init_submodule(const char *path, const char *prefix, int quiet)
+static void for_each_listed_submodule(const struct module_list *list,
+				      each_submodule_fn fn, void *cb_data)
+{
+	int i;
+	for (i = 0; i < list->nr; i++)
+		fn(list->entries[i], cb_data);
+}
+
+struct init_cb {
+	const char *prefix;
+	unsigned int cb_flags;
+};
+#define INIT_CB_INIT { NULL, 0 }
+
+static void init_submodule(const char *path, const char *prefix,
+			   unsigned int cb_flags)
 {
 	const struct submodule *sub;
 	struct strbuf sb = STRBUF_INIT;
@@ -411,7 +431,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		if (git_config_set_gently(sb.buf, url))
 			die(_("Failed to register url for submodule path '%s'"),
 			    displaypath);
-		if (!quiet)
+		if (!(cb_flags & CB_OPT_QUIET))
 			fprintf(stderr,
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
@@ -438,12 +458,18 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	free(upd);
 }
 
+static void init_submodule_cb(const struct cache_entry *list_item, void *cb_data)
+{
+	struct init_cb *info = cb_data;
+	init_submodule(list_item->name, info->prefix, info->cb_flags);
+}
+
 static int module_init(int argc, const char **argv, const char *prefix)
 {
+	struct init_cb info = INIT_CB_INIT;
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
-	int i;
 
 	struct option module_init_options[] = {
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
@@ -468,8 +494,11 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
 
-	for (i = 0; i < list.nr; i++)
-		init_submodule(list.entries[i]->name, prefix, quiet);
+	info.prefix = prefix;
+	if (quiet)
+		info.cb_flags |= CB_OPT_QUIET;
+
+	for_each_listed_submodule(&list, init_submodule_cb, &info);
 
 	return 0;
 }
-- 
2.13.0


^ permalink raw reply	[relevance 21%]

* [PATCH v6 3/3] submodule: port submodule subcommand 'status' from shell to C
  2017-09-29  9:44 ` [PATCH v6 0/3] Incremental rewrite of git-submodules Prathamesh Chavan
  2017-09-29  9:44   ` [PATCH v6 1/3] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
  2017-09-29  9:44   ` [PATCH v6 2/3] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
@ 2017-09-29  9:44   ` Prathamesh Chavan
  2 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-09-29  9:44 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, hanwen, sbeller, Prathamesh Chavan

This aims to make git-submodule 'status' a built-in. Hence, the function
cmd_status() is ported from shell to C. This is done by introducing
four functions: module_status(), submodule_status_cb(),
submodule_status() and print_status().

The function module_status() acts as the front-end of the subcommand.
It parses subcommand's options and then calls the function
module_list_compute() for computing the list of submodules. Then
this functions calls for_each_listed_submodule() looping through the
list obtained.

Then for_each_listed_submodule() calls submodule_status_cb() for each of
the submodule in its list. The function submodule_status_cb() calls
submodule_status() after passing appropriate arguments to the funciton.
Function submodule_status() is responsible for generating the status
each submodule it is called for, and then calls print_status().

Finally, the function print_status() handles the printing of submodule's
status.

Function set_name_rev() is also ported from git-submodule to the
submodule--helper builtin function compute_rev_name(), which now
generates the value of the revision name as required.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 207 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  61 +------------
 2 files changed, 208 insertions(+), 60 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 20a1ef868..50d38fc20 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -13,8 +13,13 @@
 #include "remote.h"
 #include "refs.h"
 #include "connect.h"
+#include "revision.h"
+#include "diffcore.h"
+#include "diff.h"
 
 #define CB_OPT_QUIET		(1<<0)
+#define CB_OPT_CACHED		(1<<1)
+#define CB_OPT_RECURSIVE	(1<<2)
 
 typedef void (*each_submodule_fn)(const struct cache_entry *list_item,
 				  void *cb_data);
@@ -245,6 +250,53 @@ static char *get_submodule_displaypath(const char *path, const char *prefix)
 	}
 }
 
+static char *compute_rev_name(const char *sub_path, const char* object_id)
+{
+	struct strbuf sb = STRBUF_INIT;
+	const char ***d;
+
+	static const char *describe_bare[] = {
+		NULL
+	};
+
+	static const char *describe_tags[] = {
+		"--tags", NULL
+	};
+
+	static const char *describe_contains[] = {
+		"--contains", NULL
+	};
+
+	static const char *describe_all_always[] = {
+		"--all", "--always", NULL
+	};
+
+	static const char **describe_argv[] = {
+		describe_bare, describe_tags, describe_contains,
+		describe_all_always, NULL
+	};
+
+	for (d = describe_argv; *d; d++) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+		prepare_submodule_repo_env(&cp.env_array);
+		cp.dir = sub_path;
+		cp.git_cmd = 1;
+		cp.no_stderr = 1;
+
+		argv_array_push(&cp.args, "describe");
+		argv_array_pushv(&cp.args, *d);
+		argv_array_push(&cp.args, object_id);
+
+		if (!capture_command(&cp, &sb, 0)) {
+			strbuf_strip_suffix(&sb, "\n");
+			return strbuf_detach(&sb, NULL);
+		}
+	}
+
+	strbuf_release(&sb);
+	return NULL;
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -503,6 +555,160 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+struct status_cb {
+	const char *prefix;
+	unsigned int cb_flags;
+};
+#define STATUS_CB_INIT { NULL, 0 }
+
+static void print_status(unsigned int flags, char state, const char *path,
+			 const struct object_id *oid, const char *displaypath)
+{
+	if (flags & CB_OPT_QUIET)
+		return;
+
+	printf("%c%s %s", state, oid_to_hex(oid), displaypath);
+
+	if (state == ' ' || state == '+')
+		printf(" (%s)", compute_rev_name(path, oid_to_hex(oid)));
+
+	printf("\n");
+}
+
+static int handle_submodule_head_ref(const char *refname,
+				     const struct object_id *oid, int flags,
+				     void *cb_data)
+{
+	struct object_id *output = cb_data;
+	if (oid)
+		oidcpy(output, oid);
+
+	return 0;
+}
+
+static void status_submodule(const char *path, const struct object_id *ce_oid,
+			     unsigned int ce_flags, const char *prefix,
+			     unsigned int cb_flags)
+{
+	char *displaypath;
+	struct argv_array diff_files_args = ARGV_ARRAY_INIT;
+	struct rev_info rev;
+	int diff_files_result;
+
+	if (!submodule_from_path(&null_oid, path))
+		die(_("no submodule mapping found in .gitmodules for path '%s'"),
+		      path);
+
+	displaypath = get_submodule_displaypath(path, prefix);
+
+	if ((CE_STAGEMASK & ce_flags) >> CE_STAGESHIFT) {
+		print_status(cb_flags, 'U', path, &null_oid, displaypath);
+		goto cleanup;
+	}
+
+	if (!is_submodule_active(the_repository, path)) {
+		print_status(cb_flags, '-', path, ce_oid, displaypath);
+		goto cleanup;
+	}
+
+	argv_array_pushl(&diff_files_args, "diff-files",
+			 "--ignore-submodules=dirty", "--quiet", "--",
+			 path, NULL);
+
+	git_config(git_diff_basic_config, NULL);
+	init_revisions(&rev, prefix);
+	rev.abbrev = 0;
+	precompose_argv(diff_files_args.argc, diff_files_args.argv);
+	diff_files_args.argc = setup_revisions(diff_files_args.argc,
+					       diff_files_args.argv,
+					       &rev, NULL);
+	diff_files_result = run_diff_files(&rev, 0);
+
+	if (!diff_result_code(&rev.diffopt, diff_files_result)) {
+		print_status(cb_flags, ' ', path, ce_oid,
+			     displaypath);
+	} else if (!(cb_flags & CB_OPT_CACHED)) {
+		struct object_id oid;
+
+		if (refs_head_ref(get_submodule_ref_store(path),
+				  handle_submodule_head_ref, &oid))
+			die(_("could not resolve HEAD ref inside the"
+			      "submodule '%s'"), path);
+
+		print_status(cb_flags, '+', path, &oid, displaypath);
+	} else {
+		print_status(cb_flags, '+', path, ce_oid, displaypath);
+	}
+
+	if (cb_flags & CB_OPT_RECURSIVE) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = path;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_push(&cpr.args, "--super-prefix");
+		argv_array_pushf(&cpr.args, "%s/", displaypath);
+		argv_array_pushl(&cpr.args, "submodule--helper", "status",
+				 "--recursive", NULL);
+
+		if (cb_flags & CB_OPT_CACHED)
+			argv_array_push(&cpr.args, "--cached");
+
+		if (cb_flags & CB_OPT_QUIET)
+			argv_array_push(&cpr.args, "--quiet");
+
+		if (run_command(&cpr))
+			die(_("failed to recurse into submodule '%s'"), path);
+	}
+
+cleanup:
+	argv_array_clear(&diff_files_args);
+	free(displaypath);
+}
+
+static void status_submodule_cb(const struct cache_entry *list_item,
+				void *cb_data)
+{
+	struct status_cb *info = cb_data;
+	status_submodule(list_item->name, &list_item->oid, list_item->ce_flags,
+			 info->prefix, info->cb_flags);
+}
+
+static int module_status(int argc, const char **argv, const char *prefix)
+{
+	struct status_cb info = STATUS_CB_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+	int quiet = 0;
+
+	struct option module_status_options[] = {
+		OPT__QUIET(&quiet, N_("Suppress submodule status output")),
+		OPT_BIT(0, "cached", &info.cb_flags, N_("Use commit stored in the index instead of the one stored in the submodule HEAD"), CB_OPT_CACHED),
+		OPT_BIT(0, "recursive", &info.cb_flags, N_("recurse into nested submodules"), CB_OPT_RECURSIVE),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule status [--quiet] [--cached] [--recursive] [<path>...]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_status_options,
+			     git_submodule_helper_usage, 0);
+
+	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.prefix = prefix;
+	if (quiet)
+		info.cb_flags |= CB_OPT_QUIET;
+
+	for_each_listed_submodule(&list, status_submodule_cb, &info);
+
+	return 0;
+}
+
 static int module_name(int argc, const char **argv, const char *prefix)
 {
 	const struct submodule *sub;
@@ -1300,6 +1506,7 @@ static struct cmd_struct commands[] = {
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
+	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
diff --git a/git-submodule.sh b/git-submodule.sh
index 66d1ae8ef..156255a9e 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -758,18 +758,6 @@ cmd_update()
 	}
 }
 
-set_name_rev () {
-	revname=$( (
-		sanitize_submodule_env
-		cd "$1" && {
-			git describe "$2" 2>/dev/null ||
-			git describe --tags "$2" 2>/dev/null ||
-			git describe --contains "$2" 2>/dev/null ||
-			git describe --all --always "$2"
-		}
-	) )
-	test -z "$revname" || revname=" ($revname)"
-}
 #
 # Show commit summary for submodules in index or working tree
 #
@@ -1016,54 +1004,7 @@ cmd_status()
 		shift
 	done
 
-	{
-		git submodule--helper list --prefix "$wt_prefix" "$@" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path") || exit
-		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-		if test "$stage" = U
-		then
-			say "U$sha1 $displaypath"
-			continue
-		fi
-		if ! git submodule--helper is-active "$sm_path" ||
-		{
-			! test -d "$sm_path"/.git &&
-			! test -f "$sm_path"/.git
-		}
-		then
-			say "-$sha1 $displaypath"
-			continue;
-		fi
-		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
-		then
-			set_name_rev "$sm_path" "$sha1"
-			say " $sha1 $displaypath$revname"
-		else
-			if test -z "$cached"
-			then
-				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
-			fi
-			set_name_rev "$sm_path" "$sha1"
-			say "+$sha1 $displaypath$revname"
-		fi
-
-		if test -n "$recursive"
-		then
-			(
-				prefix="$displaypath/"
-				sanitize_submodule_env
-				wt_prefix=
-				cd "$sm_path" &&
-				eval cmd_status
-			) ||
-			die "$(eval_gettext "Failed to recurse into submodule path '\$sm_path'")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper status ${GIT_QUIET:+--quiet} ${cached:+--cached} ${recursive:+--recursive} "$@"
 }
 #
 # Sync remote urls for submodules
-- 
2.13.0


^ permalink raw reply	[relevance 19%]

* Re: [PATCH v4] technical doc: add a design doc for hash function transition
  2017-09-29  6:06         ` Junio C Hamano
@ 2017-09-29 17:34           ` Jonathan Nieder
  0 siblings, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-09-29 17:34 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Shawn Pearce, Linus Torvalds, Git Mailing List, Stefan Beller, bmwill, Jonathan Tan, Jeff King, David Lang, brian m. carlson, Masaya Suzuki, demerphq, The Keccak Team, Johannes Schindelin

Junio C Hamano wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:

>> This document describes what a transition to a new hash function for
>> Git would look like.  Add it to Documentation/technical/ as the plan
>> of record so that future changes can be recorded as patches.
>>
>> Also-by: Brandon Williams <bmwill@google.com>
>> Also-by: Jonathan Tan <jonathantanmy@google.com>
>> Also-by: Stefan Beller <sbeller@google.com>
>> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
>> ---
>
> Shoudln't these all be s-o-b: (with a note immediately before that
> to say all four contributed equally or something)?

I don't want to get lost in the weeds in the question of how to
represent such a collaborative effort in git's metadata.

You're right that I should collect their sign-offs!  Your approach of
using text instead of machine-readable data for common authorship also
seems okay.

In any event, this is indeed

Signed-off-by: Brandon Williams <bmwill@google.com>
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Stefan Beller <sbeller@google.com>

(I just checked :)).

>> +Background
>> +----------
>> +At its core, the Git version control system is a content addressable
>> +filesystem. It uses the SHA-1 hash function to name content. For
>> +example, files, directories, and revisions are referred to by hash
>> +values unlike in other traditional version control systems where files
>> +or versions are referred to via sequential numbers. The use of a hash
>
> Traditional systems refer to files via numbers???  Perhaps "where
> versions of files are referred to via sequential numbers" or
> something?

Good point.  The wording you suggested will work well.

>> +function to address its content delivers a few advantages:
>> +
>> +* Integrity checking is easy. Bit flips, for example, are easily
>> +  detected, as the hash of corrupted content does not match its name.
>> +* Lookup of objects is fast.
>
> * There is no ambiguity what the object's name should be, given its
>   content.
>
> * Deduping the same content copied across versions and paths is
>   automatic.

:)  Yep, these are nice too, especially that second one.

It also is how we make diff-ing fast.

>> +SHA-1 still possesses the other properties such as fast object lookup
>> +and safe error checking, but other hash functions are equally suitable
>> +that are believed to be cryptographically secure.
>
> s/secure/more &/, perhaps?

We were looking for a phrase meaning that it should be a cryptographic
hash function in good standing, which SHA-1 is at least approaching
not being.

"more secure" should work fine.  Let's go with that.

>> +Goals
>> +-----
>> +...
>> +   c. Users can use SHA-1 and NewHash identifiers for objects
>> +      interchangeably (see "Object names on the command line", below).
>
> Mental note.  This needs to extend to the "index X..Y" lines in the
> patch output, which is used by "apply -3" and "am -3".

Will add a note about this to "Object names on the command line".  Stefan
had already pointed out that that section should really be renamed to
something like "Object names in input and output".

>> +2. Allow a complete transition away from SHA-1.
>> +   a. Local metadata for SHA-1 compatibility can be removed from a
>> +      repository if compatibility with SHA-1 is no longer needed.
>
> I like the emphasis on "Local" here.  Metadata for compatiblity that
> is embedded in the objects obviously cannot be removed.
>
> From that point of view, one of the goals ought to be "make sure
> that as much SHA-1 compatibility metadata as possible is local and
> outside the object".  This goal may not be able to say more than "as
> much as possible", as signed objects that came from SHA-1 world
> needs to carry the compatibility metadata somewhere somehow.
>
> Or perhaps we could.  There is nothing that says a signed tag
> created in the SHA-1 world must have the PGP/SHA-1 signature in the
> NewHash payload---it could be split off of the object data and
> stored in a local metadata cache, to be used only when we need to
> convert it back to the SHA-1 world.

That would break round-tripping and would mean that multiple SHA-1
objects could have the same NewHash name.  In other words, from
my point of view there is something that says that such data must
be preserved.

Another way to put it: even after removing all SHA-1 compatibility
metadata, one nice feature of this design is that it can be recovered
if I change my mind, from data in the NewHash based repository alone.

[...]
>> +Non-Goals
>> +---------
>> ...
>> +6. Skip fetching some submodules of a project into a NewHash
>> +   repository. (This also depends on NewHash support in Git
>> +   protocol.)
>
> It is unclear what this means.  Around submodule support, one thing
> I can think of is that a NewHash tree in a superproject would record
> a gitlink that is a NewHash commit object name in it, therefore it
> cannot refer to an unconverted SHA-1 submodule repository.  But it
> is unclear if the above description refers to the same issue, or
> something else.

It refers to that issue.

[...]
>> +Overview
>> +--------
>> +We introduce a new repository format extension. Repositories with this
>> +extension enabled use NewHash instead of SHA-1 to name their objects.
>> +This affects both object names and object content --- both the names
>> +of objects and all references to other objects within an object are
>> +switched to the new hash function.
>> +
>> +NewHash repositories cannot be read by older versions of Git.
>> +
>> +Alongside the packfile, a NewHash repository stores a bidirectional
>> +mapping between NewHash and SHA-1 object names. The mapping is generated
>> +locally and can be verified using "git fsck". Object lookups use this
>> +mapping to allow naming objects using either their SHA-1 and NewHash names
>> +interchangeably.
>> +
>> +"git cat-file" and "git hash-object" gain options to display an object
>> +in its sha1 form and write an object given its sha1 form.
>
> Both of these are somewhat unclear.

I think we can delete this paragraph.  It was written before the
"Object names on the command line" section that goes into such issues
in more detail.

[...]
>> +Object names
>> +~~~~~~~~~~~~
>> +Objects can be named by their 40 hexadecimal digit sha1-name or 64
>> +hexadecimal digit newhash-name, plus names derived from those (see
>> +gitrevisions(7)).
>> +
>> +The sha1-name of an object is the SHA-1 of the concatenation of its
>> +type, length, a nul byte, and the object's sha1-content. This is the
>> +traditional <sha1> used in Git to name objects.
>> +
>> +The newhash-name of an object is the NewHash of the concatenation of its
>> +type, length, a nul byte, and the object's newhash-content.
>
> It makes me wonder if we want to add the hashname in this object
> header.  "length" would be different for non-blob objects anyway,
> and it is not "compat metadata" we want to avoid baked in, yet it
> would help diagnose a mistake of attempting to use a "mixed" objects
> in a single repository.  Not a big issue, though.

Do you mean that adding the hashname into the computation that
produces the object name would help in some use case?

Or do you mean storing the hashname on disk somewhere, even if it
doesn't enter into the object name?  For the latter, we store the
hashname in the .git/config extensions.* configuration and the pack
index files.  You also suggested storing the hash name in
.git/objects/loose-object-idx, which seems to me like a good idea.

We didn't touch on the .pack format but we probably need to (if only
because of the size of REF_DELTAs and the cksum trailer), and it would
also need to name what object format it is using.

For loose objects, it would be nice to name the hash in the file, so
that "file" can understand what is happening if someone accidentally
mixes types using "cp".  The only downside is losing the ability to
copy blobs (which have the same content despite being named using
different hashes) between repositories after determining their new
names.  That doesn't seem like a strong downside --- it's pretty
harmless to include the hash type in loose object files, too.  I think
I would prefer this to be a "magic number" instead of part of the
zlib-deflated payload, since this way "file" can discover it more
easily.

>> +The format allows round-trip conversion between newhash-content and
>> +sha1-content.
>
> If it is a goal to eventually be able to lose SHA-1 compatibility
> metadata from the objects, then we might want to remove SHA-1 based
> signature bits (e.g. PGP trailer in signed tag, gpgsig header in the
> commit object) from NewHash contents, and instead have them stored
> in a side "metadata" table, only to be used while converting back.
> I dunno if that is desirable.

I don't consider that desirable.

A SHA-1 based signature is still of historical interest even if my
centuries-newer version of Git is not able to verify it.

[...]
> I take it to mean that the stride is defined in the "length in bytes
> of shortened object names" in the file header.  If so, I can see how
> this would work.  This "sorted table", unlike the next one, does not
> say how it is sorted, but I assume this is just the object name
> order (as opposed to the pack location order the next table uses)?

Yes.  Will clarify.

>> +  - A table of full object names in pack order. This allows resolving
>> +    a reference to "the nth object in the pack file" (from a
>> +    reachability bitmap or from the next table of another object
>> +    format) to its object name.
>> +
>> +  - A table of 4-byte values mapping object name order to pack order.
>> +    For an object in the table of sorted shortened object names, the
>> +    value at the corresponding index in this table is the index in the
>> +    previous table for that same object.
>> +
>> +    This can be used to look up the object in reachability bitmaps or
>> +    to look up its name in another object format.
>
> And this is a separate table because the short-name table wants to
> be as compact as possible for binary search?  Otherwise an entry in
> the short-name table could be <pack order number, n-bytes that is
> short unique prefix>.

Yes.  The idx v2 format has a similar design.

>> +  - A table of 4-byte CRC32 values of the packed object data, in the
>> +    order that the objects appear in the pack file. This is to allow
>> +    compressed data to be copied directly from pack to pack during
>> +    repacking without undetected data corruption.
>
> An obvious alternative would be to have the CRC32 checksum near
> (e.g. immediately before) the object data in the packfile (as
> opposed to the .idx file like this document specifies).  I am not
> sure what the pros and cons are between the two, though, and that is
> why I mention the possiblity here.

As you mentioned under separate cover, it is useful for derived data
like this to be outside the packfile.

> Hmm, as the corresponding packfile stores object data only in
> NewHash content format, it is somewhat curious that this table that
> stores CRC32 of the data appears in the "Tables for each object
> format" section, as they would be identical, no?  Unless I am
> grossly misleading the spec, the checksum should either go outside
> the "Tables for each object format" section but still in .idx, or
> should be eliminated and become part of the packdata stream instead,
> perhaps?

It's actually only present for the first object format.  Will find a
better way to describe this.

>> +  - A table of 4-byte offset values. For an object in the table of
>> +    sorted shortened object names, the value at the corresponding
>> +    index in this table indicates where that object can be found in
>> +    the pack file. These are usually 31-bit pack file offsets, but
>> +    large offsets are encoded as an index into the next table with the
>> +    most significant bit set.
>
> Oy.  So we can go from a short prefix to the pack location by first
> finding it via binsearch in the short-name table, realize that it is
> nth object in the object name order, and consulting this table.
> When we know the pack-order of an object, there is no direct way to
> go to its location (short of reversing the name-order-to-pack-order
> table)?

An earlier version of the design also had a pack-order-to-pack-offset
table, but we weren't able to think of any cases where that would be
used without also looking up the object name that can be used to
verify the integrity of the inflated object.

Do you have an application in mind?

[...]
>> +- Tables for the second object format, with the same layout as above,
>> +  up to and not including the table of CRC32 values.
>> +- Zero or more NUL bytes.
>> +- The trailer consists of the following:
>> +  - A copy of the 20-byte NewHash checksum at the end of the
>> +    corresponding packfile.
>> +
>> +  - 20-byte NewHash checksum of all of the above.
>
> When did NewHash shrink to 20-byte suddenly?  I think the above two
> are both "32-byte"?

Yes, good catch.

[...]
>> +Loose object index
>> +~~~~~~~~~~~~~~~~~~
>> +A new file $GIT_OBJECT_DIR/loose-object-idx contains information about
>> +all loose objects. Its format is
>> +
>> +  # loose-object-idx
>> +  (newhash-name SP sha1-name LF)*
>> +
>> +where the object names are in hexadecimal format. The file is not
>> +sorted.
>
> Shouldn't the file somehow say what hashes are involved to allow us
> match it with extension.{objectFormat,compatObjectFormat}, perhaps
> at the end of the "# loose-object-idx" line?

Good idea!

[...]
>> +The loose object index is protected against concurrent writes by a
>> +lock file $GIT_OBJECT_DIR/loose-object-idx.lock. To add a new loose
>> +object:
>> +
>> +1. Write the loose object to a temporary file, like today.
>> +2. Open loose-object-idx.lock with O_CREAT | O_EXCL to acquire the lock.
>> +3. Rename the loose object into place.
>> +4. Open loose-object-idx with O_APPEND and write the new object
>
> "write the new entry, fsync and close"?

Yes, I think we do need to fsync. :/

[...]
>> +Translation table
>> +~~~~~~~~~~~~~~~~~
>> +The index files support a bidirectional mapping between sha1-names
>> +and newhash-names. The lookup proceeds similarly to ordinary object
>> +lookups. For example, to convert a sha1-name to a newhash-name:
>> +
>> + 1. Look for the object in idx files. If a match is present in the
>> +    idx's sorted list of truncated sha1-names, then:
>> +    a. Read the corresponding entry in the sha1-name order to pack
>> +       name order mapping.
>> +    b. Read the corresponding entry in the full sha1-name table to
>> +       verify we found the right object. If it is, then
>> +    c. Read the corresponding entry in the full newhash-name table.
>> +       That is the object's newhash-name.
>
> c. is possible because b. and c. are sorted the same way, i.e. the
> index used to consult the full sha1-name table, which is the pack
> order number, can be used to find its full newhash in the "full
> newhash sorted by pack order" table?

Yes.

>> +Reading an object's sha1-content
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> I'd stop here and continue in a separate message.  Thanks for a
> detailed write-up.

Thanks for looking it over.

Jonathan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH v6 2/3] submodule--helper: introduce for_each_listed_submodule()
  2017-09-29  9:44   ` [PATCH v6 2/3] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
@ 2017-10-02  0:55     ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-10-02  0:55 UTC (permalink / raw)
  To: Prathamesh Chavan; +Cc: christian.couder, git, hanwen, sbeller

Prathamesh Chavan <pc44800@gmail.com> writes:

> Introduce function for_each_listed_submodule() and replace a loop
> in module_init() with a call to it.
>
> The new function will also be used in other parts of the
> system in later patches.
>
> Mentored-by: Christian Couder <christian.couder@gmail.com>
> Mentored-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
> ---
>  builtin/submodule--helper.c | 39 ++++++++++++++++++++++++++++++++++-----
>  1 file changed, 34 insertions(+), 5 deletions(-)
>
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index cdae54426..20a1ef868 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -14,6 +14,11 @@
>  #include "refs.h"
>  #include "connect.h"
>  
> +#define CB_OPT_QUIET		(1<<0)

Is the purpose of this bit to make the callback quiet?  I do not
think so.  Is there a reason why we cannot call it just OPT_QUIET or
something instead?

When the set of functions that pay attention to these flags include
both ones that are callable for a single submodule and ones meant as
callbacks for for-each interface, having to flip bit whose name
screams "CallBack!" in a caller of a single-short version feels very
wrong.

"make style" tells me to format the above like so:

	#define OPT_QUIET (1 << 0)

and I think I agree.

> @@ -349,7 +354,22 @@ static int module_list(int argc, const char **argv, const char *prefix)
>  	return 0;
>  }
>  
> -static void init_submodule(const char *path, const char *prefix, int quiet)
> +static void for_each_listed_submodule(const struct module_list *list,
> +				      each_submodule_fn fn, void *cb_data)
> +{
> +	int i;
> +	for (i = 0; i < list->nr; i++)
> +		fn(list->entries[i], cb_data);
> +}

Good.

> +struct init_cb {

I take it is a short-hand for "submodule init callback"?  As long as
the name stays inside this file, I think we are OK.

> +	const char *prefix;
> +	unsigned int cb_flags;

Call this just "flags"; call-back ness is plenty clear from the fact
that it lives in a structure meant as a callback interface already.

> +};

Blank line here?

> +#define INIT_CB_INIT { NULL, 0 }
> +
> +static void init_submodule(const char *path, const char *prefix,
> +			   unsigned int cb_flags)

Call this also "flags"; a direct caller of this function that wants
to initialize a single submodule without going thru the for-each
callback interface would not be passing "callback flags"--they are
just passing a set of flags.

^ permalink raw reply	[relevance 16%]

* Re: git submodule add fails when using both --branch and --depth
  2017-09-30 17:20 git submodule add fails when using both --branch and --depth Thadeus Fleming
@ 2017-10-02 19:55 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-10-02 19:55 UTC (permalink / raw)
  To: Thadeus Fleming; +Cc: git

On Sat, Sep 30, 2017 at 10:20 AM, Thadeus Fleming
<thadeus.j.fleming@gmail.com> wrote:
> I'm running git 2.14.2 on Ubuntu 16.04.
>
> Compare the behavior of
>
>> git clone --branch pu --depth 1 https://github.com/git/git git-pu
>
> which clones only the latest commit of the pu branch and
>
>> mkdir tmp && cd tmp && git init
>> git submodule add --branch pu --depth 1 https://github.com/git/git \
>   git-pu
>
> which gives the error
>
> fatal: 'origin/pu' is not a commit and a branch 'pu' cannot be created
> from it
> Unable to checkout submodule 'git-pu'
>
> Investigating further, there is indeed only one commit in the local repo:
>
>> cd git-pu
>> git log --oneline | wc -l
> 1
>
> But that commit is the head of master.
>
>> git branch -a
> * master                                 remotes/origin/master
>   remotes/origin/HEAD -> origin/master
>
> This appears to be because git-submodule--helper does not accept a
> --branch option. Using the --depth N option causes it to only clone N
> commits from the default branch, which generally do not include the
> desired branch. Thus, the next step,
>
> git checkout -f -q -B "$branch" "origin/$branch"
>
> fails, and provides the rather confusing error message above.
>
> I'd suggest that git-submodule--helper learn a --branch option
> consistent with git clone, and if that is impossible, that
> git submodule add rejects the simultaneous use of both the --branch and
> --depth options.

Adding the branch field to the submodule helper is a great idea.

>
>
>
> --tjf

^ permalink raw reply	[relevance 23%]

* Re: Security of .git/config and .git/hooks
  2017-10-03 12:32 ` Jeff King
@ 2017-10-03 15:10   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-10-03 15:10 UTC (permalink / raw)
  To: Jeff King; +Cc: Jonathan Nieder, git, Loic Guelorget, Sitaram Chamarty

So once upon a time we compared Gits security model with a
web browser. A web browser lets you execute 3rd party code
(e.g. javascript) and it is supposedly safe to look at malicious sites.

Currently Git only promises to have the clone/fetch operation safe,
not the "here is a zip of my whole repo" case, which sounds more
like the web browser experience ("here is a site with js, even zipped
in transfer"). Tightening the security model of Git towards this seems
like a good idea to me.

>>  1. Introduce a (configurable) list of "safe" configuration items that
>>     can be set in .git/config and don't respect any others.
>
> A whitelist is obviously safer than a blacklist. Though I also feel like
> some of the options may give an unexpectedly wide attack surface. I.e.,
> I wouldn't be surprised if some innocent-looking option ends up being
> used in a tricky way to gain more access. E.g., submodule config
> pointing to paths outside of the repository.
>
> Do you plan to run in safe-mode all the time? What if safe-mode was a
> lot more extreme, and simply avoided reading repo-level config at all
> (except for check_repository_format(), which should be pretty innocent).
>
> I have a feeling there are some features (like submodules) that would
> simply be broken in safe-mode.

I would think that the essential submodule things would be "safe"
to look at. But e.g. submodule.<name>.update = "!rm -rf /" would be
not ok, hence the .update configuration would be in the unsafe space.

Any unsafe config option would need to be set outside the actual
repository (~/.config/git/<repo-id>/config ?)

>
>>  2. But what if I want to set a different pager per-repository?
>>     I think we could do this using configuration "profiles".
>>     My ~/.config/git/profiles/ directory would contain git-style
>>     config files for repositories to include.  Repositories could
>>     then contain
>>
>>       [include]
>>               path = ~/.config/git/profiles/fancy-log-pager
>>
>>     to make use of those settings.  The facility (1) would
>>     special-case this directory to allow it to set "unsafe" settings
>>     since files there are assumed not to be under the control of an
>>     attacker.
>
> You can do something quite similar already:
>
>   git config --global \
>     include.gitdir:/path/to/specific/repo.path
>     .gitconfig-fancy-log-pager
>
> The main difference is that the profile inclusion is done by path rather
> than riding along with the repository directory if it gets moved. In
> practice I doubt that matters much, and I think the security model for
> include.gitdir is a lot simpler.

I am not sure if this works so well for the submodule.<name>.update
config (that we want to deprecate anyway, but still)

>> For backward compatibility, this facility would be controlled by a
>> global configuration setting.  If that setting is not enabled, then
>> the current, less safe behavior would remain.
>
> Are config and symlinks everything we need to care about? I can't think
> of anything else, but Git really has quite a large attack surface when
> accessing a local repo. Right now the safest thing you can do is "git
> clone --no-local" an untrusted repo and then look only at the clone. Of
> course nobody _actually_ does that, so any "safe" mode seems like it
> would be an improvement. But would claiming to have a "safe" mode
> encourage people to use it to look at risky repositories, exacerbating
> any holes (e.g., exploiting a bug in the index format)? I don't know.

Good point. Though we only care about the case of breaking out and
executing untrusted code; most of the index exploits would rather
trigger a segfault or infinite lop (in my imagination at least).

>
> -Peff

^ permalink raw reply	[relevance 17%]

* [ANNOUNCE] Git v2.15.0-rc0
@ 2017-10-05  5:55 Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-10-05  5:55 UTC (permalink / raw)
  To: git; +Cc: Linux Kernel

An early preview release Git v2.15.0-rc0 is now available for
testing at the usual places.  It is comprised of 672 non-merge
commits since v2.14.0, contributed by 66 people, 20 of which are
new faces.

The tarballs are found at:

    https://www.kernel.org/pub/software/scm/git/testing/

The following public repositories all have a copy of the
'v2.15.0-rc0' tag and the 'master' branch that the tag points at:

  url = https://kernel.googlesource.com/pub/scm/git/git
  url = git://repo.or.cz/alt-git.git
  url = https://github.com/gitster/git

New contributors whose contributions weren't in v2.14.0 are as follows.
Welcome to the Git development community!

  Ann T Ropea, Daniel Watkins, Dimitrios Christidis, Eric Rannaud,
  Evan Zacks, Hielke Christian Braun, Ian Campbell, Ilya Kantor,
  Jameson Miller, Job Snijders, Joel Teichroeb, joernchen,
  Łukasz Gryglicki, Manav Rathi, Martin Ågren, Michael Forney,
  Patryk Obara, Rene Scharfe, Ross Kabus, and Urs Thuermann.

Returning contributors who helped this release are as follows.
Thanks for your continued support.

  Adam Dinwoodie, Ævar Arnfjörð Bjarmason, Andreas Heiduk,
  Anthony Sottile, Ben Boeckel, Brandon Casey, Brandon Williams,
  brian m. carlson, Christian Couder, Eric Blake, Han-Wen Nienhuys,
  Heiko Voigt, Jeff Hostetler, Jeff King, Johannes Schindelin,
  Jonathan Nieder, Jonathan Tan, Junio C Hamano, Kaartic Sivaraam,
  Kevin Daudt, Kevin Willford, Lars Schneider, Martin Koegler,
  Matthieu Moy, Max Kirillov, Michael Haggerty, Michael J Gruber,
  Nguyễn Thái Ngọc Duy, Nicolas Morey-Chaisemartin, Øystein
  Walle, Paolo Bonzini, Pat Thoyts, Philip Oakley, Phillip
  Wood, Raman Gupta, Ramsay Jones, René Scharfe, Sahil Dua,
  Santiago Torres, Stefan Beller, Stephan Beyer, Takashi Iwai,
  Thomas Gummerer, Tom G. Christensen, Torsten Bögershausen,
  and William Duclot.

----------------------------------------------------------------

Git 2.15 Release Notes (draft)
==============================

Backward compatibility notes and other notable changes.

 * Use of an empty string as a pathspec element that is used for
   'everything matches' is still warned and Git asks users to use a
   more explicit '.' for that instead.  The hope is that existing
   users will not mind this change, and eventually the warning can be
   turned into a hard error, upgrading the deprecation into removal of
   this (mis)feature.  That is now scheduled to happen in the upcoming
   release.

 * Git now avoids blindly falling back to ".git" when the setup
   sequence said we are _not_ in Git repository.  A corner case that
   happens to work right now may be broken by a call to die("BUG").
   We've tried hard to locate such cases and fixed them, but there
   might still be cases that need to be addressed--bug reports are
   greatly appreciated.

 * "branch --set-upstream" that has been deprecated in Git 1.8 has
   finally been retired.


Updates since v2.14
-------------------

UI, Workflows & Features

 * An example that is now obsolete has been removed from a sample hook,
   and an old example in it that added a sign-off manually has been
   improved to use the interpret-trailers command.

 * The advice message given when "git rebase" stops for conflicting
   changes has been improved.

 * The "rerere-train" script (in contrib/) learned the "--overwrite"
   option to allow overwriting existing recorded resolutions.

 * "git contacts" (in contrib/) now lists the address on the
   "Reported-by:" trailer to its output, in addition to those on
   S-o-b: and other trailers, to make it easier to notify (and thank)
   the original bug reporter.

 * "git rebase", especially when it is run by mistake and ends up
   trying to replay many changes, spent long time in silence.  The
   command has been taught to show progress report when it spends
   long time preparing these many changes to replay (which would give
   the user a chance to abort with ^C).

 * "git merge" learned a "--signoff" option to add the Signed-off-by:
   trailer with the committer's name.

 * "git diff" learned to optionally paint new lines that are the same
   as deleted lines elsewhere differently from genuinely new lines.

 * "git interpret-trailers" learned to take the trailer specifications
   from the command line that overrides the configured values.

 * "git interpret-trailers" has been taught a "--parse" and a few
   other options to make it easier for scripts to grab existing
   trailer lines from a commit log message.

 * "gitweb" shows a link to visit the 'raw' contents of blbos in the
   history overview page.

 * "[gc] rerereResolved = 5.days" used to be invalid, as the variable
   is defined to take an integer counting the number of days.  It now
   is allowed.

 * The code to acquire a lock on a reference (e.g. while accepting a
   push from a client) used to immediately fail when the reference is
   already locked---now it waits for a very short while and retries,
   which can make it succeed if the lock holder was holding it during
   a read-only operation.

 * "branch --set-upstream" that has been deprecated in Git 1.8 has
   finally been retired.

 * The codepath to call external process filter for smudge/clean
   operation learned to show the progress meter.

 * "git rev-parse" learned "--is-shallow-repository", that is to be
   used in a way similar to existing "--is-bare-repository" and
   friends.

 * "git describe --match <pattern>" has been taught to play well with
   the "--all" option.

 * "git branch" learned "-c/-C" to create a new branch by copying an
   existing one.

 * Some commands (most notably "git status") makes an opportunistic
   update when performing a read-only operation to help optimize later
   operations in the same repository.  The new "--no-optional-locks"
   option can be passed to Git to disable them.


Performance, Internal Implementation, Development Support etc.

 * Conversion from uchar[20] to struct object_id continues.

 * Start using selected c99 constructs in small, stable and
   essentialpart of the system to catch people who care about
   older compilers that do not grok them.

 * The filter-process interface learned to allow a process with long
   latency give a "delayed" response.

 * Many uses of comparision callback function the hashmap API uses
   cast the callback function type when registering it to
   hashmap_init(), which defeats the compile time type checking when
   the callback interface changes (e.g. gaining more parameters).
   The callback implementations have been updated to take "void *"
   pointers and cast them to the type they expect instead.

 * Because recent Git for Windows do come with a real msgfmt, the
   build procedure for git-gui has been updated to use it instead of a
   hand-rolled substitute.

 * "git grep --recurse-submodules" has been reworked to give a more
   consistent output across submodule boundary (and do its thing
   without having to fork a separate process).

 * A helper function to read a single whole line into strbuf
   mistakenly triggered OOM error at EOF under certain conditions,
   which has been fixed.
   (merge 642956cf45 rs/strbuf-getwholeline-fix later to maint).

 * The "ref-store" code reorganization continues.

 * "git commit" used to discard the index and re-read from the filesystem
   just in case the pre-commit hook has updated it in the middle; this
   has been optimized out when we know we do not run the pre-commit hook.
   (merge 680ee550d7 kw/commit-keep-index-when-pre-commit-is-not-run later to maint).

 * Updates to the HTTP layer we made recently unconditionally used
   features of libCurl without checking the existence of them, causing
   compilation errors, which has been fixed.  Also migrate the code to
   check feature macros, not version numbers, to cope better with
   libCurl that vendor ships with backported features.

 * The API to start showing progress meter after a short delay has
   been simplified.
   (merge 8aade107dd jc/simplify-progress later to maint).

 * Code clean-up to avoid mixing values read from the .gitmodules file
   and values read from the .git/config file.

 * We used to spend more than necessary cycles allocating and freeing
   piece of memory while writing each index entry out.  This has been
   optimized.

 * Platforms that ship with a separate sha1 with collision detection
   library can link to it instead of using the copy we ship as part of
   our source tree.

 * Code around "notes" have been cleaned up.
   (merge 3964281524 mh/notes-cleanup later to maint).

 * The long-standing rule that an in-core lockfile instance, once it
   is used, must not be freed, has been lifted and the lockfile and
   tempfile APIs have been updated to reduce the chance of programming
   errors.

 * Our hashmap implementation in hashmap.[ch] is not thread-safe when
   adding a new item needs to expand the hashtable by rehashing; add
   an API to disable the automatic rehashing to work it around.

 * Many of our programs consider that it is OK to release dynamic
   storage that is used throughout the life of the program by simply
   exiting, but this makes it harder to leak detection tools to avoid
   reporting false positives.  Plug many existing leaks and introduce
   a mechanism for developers to mark that the region of memory
   pointed by a pointer is not lost/leaking to help these tools.

 * As "git commit" to conclude a conflicted "git merge" honors the
   commit-msg hook, "git merge" that records a merge commit that
   cleanly auto-merges should, but it didn't.

 * The codepath for "git merge-recursive" has been cleaned up.

 * Many leaks of strbuf have been fixed.

 * "git imap-send" has our own implementation of the protocol and also
   can use more recent libCurl with the imap protocol support.  Update
   the latter so that it can use the credential subsystem, and then
   make it the default option to use, so that we can eventually
   deprecate and remove the former.

 * "make style" runs git-clang-format to help developers by pointing
   out coding style issues.

 * A test to demonstrate "git mv" failing to adjust nested submodules
   has been added.
   (merge c514167df2 hv/mv-nested-submodules-test later to maint).

 * On Cygwin, "ulimit -s" does not report failure but it does not work
   at all, which causes an unexpected success of some tests that
   expect failures under a limited stack situation.  This has been
   fixed.

 * Many codepaths have been updated to squelch -Wimplicit-fallthrough
   warnings from Gcc 7 (which is a good code hygiene).

 * Add a helper for DLL loading in anticipation for its need in a
   future topic RSN.

 * "git status --ignored", when noticing that a directory without any
   tracked path is ignored, still enumerated all the ignored paths in
   the directory, which is unnecessary.  The codepath has been
   optimized to avoid this overhead.

 * The final batch to "git rebase -i" updates to move more code from
   the shell script to C has been merged.

 * Operations that do not touch (majority of) packed refs have been
   optimized by making accesses to packed-refs file lazy; we no longer
   pre-parse everything, and an access to a single ref in the
   packed-refs does not touch majority of irrelevant refs, either.

 * Add comment to clarify that the style file is meant to be used with
   clang-5 and the rules are still work in progress.

Also contains various documentation updates and code clean-ups.


Fixes since v2.14
-----------------

 * "%C(color name)" in the pretty print format always produced ANSI
   color escape codes, which was an early design mistake.  They now
   honor the configuration (e.g. "color.ui = never") and also tty-ness
   of the output medium.

 * The http.{sslkey,sslCert} configuration variables are to be
   interpreted as a pathname that honors "~[username]/" prefix, but
   weren't, which has been fixed.

 * Numerous bugs in walking of reflogs via "log -g" and friends have
   been fixed.

 * "git commit" when seeing an totally empty message said "you did not
   edit the message", which is clearly wrong.  The message has been
   corrected.

 * When a directory is not readable, "gitweb" fails to build the
   project list.  Work this around by skipping such a directory.

 * Some versions of GnuPG fails to kill gpg-agent it auto-spawned
   and such a left-over agent can interfere with a test.  Work it
   around by attempting to kill one before starting a new test.

 * A recently added test for the "credential-cache" helper revealed
   that EOF detection done around the time the connection to the cache
   daemon is torn down were flaky.  This was fixed by reacting to
   ECONNRESET and behaving as if we got an EOF.

 * "git log --tag=no-such-tag" showed log starting from HEAD, which
   has been fixed---it now shows nothing.

 * The "tag.pager" configuration variable was useless for those who
   actually create tag objects, as it interfered with the use of an
   editor.  A new mechanism has been introduced for commands to enable
   pager depending on what operation is being carried out to fix this,
   and then "git tag -l" is made to run pager by default.

 * "git push --recurse-submodules $there HEAD:$target" was not
   propagated down to the submodules, but now it is.

 * Commands like "git rebase" accepted the --rerere-autoupdate option
   from the command line, but did not always use it.  This has been
   fixed.

 * "git clone --recurse-submodules --quiet" did not pass the quiet
   option down to submodules.

 * Test portability fix for OBSD.

 * Portability fix for OBSD.

 * "git am -s" has been taught that some input may end with a trailer
   block that is not Signed-off-by: and it should refrain from adding
   an extra blank line before adding a new sign-off in such a case.

 * "git svn" used with "--localtime" option did not compute the tz
   offset for the timestamp in question and instead always used the
   current time, which has been corrected.

 * Memory leak in an error codepath has been plugged.

 * "git stash -u" used the contents of the committed version of the
   ".gitignore" file to decide which paths are ignored, even when the
   file has local changes.  The command has been taught to instead use
   the locally modified contents.

 * bash 4.4 or newer gave a warning on NUL byte in command
   substitution done in "git stash"; this has been squelched.

 * "git grep -L" and "git grep --quiet -L" reported different exit
   codes; this has been corrected.

 * When handshake with a subprocess filter notices that the process
   asked for an unknown capability, Git did not report what program
   the offending subprocess was running.  This has been corrected.

 * "git apply" that is used as a better "patch -p1" failed to apply a
   taken from a file with CRLF line endings to a file with CRLF line
   endings.  The root cause was because it misused convert_to_git()
   that tried to do "safe-crlf" processing by looking at the index
   entry at the same path, which is a nonsense---in that mode, "apply"
   is not working on the data in (or derived from) the index at all.
   This has been fixed.

 * Killing "git merge --edit" before the editor returns control left
   the repository in a state with MERGE_MSG but without MERGE_HEAD,
   which incorrectly tells the subsequent "git commit" that there was
   a squash merge in progress.  This has been fixed.

 * "git archive" did not work well with pathspecs and the
   export-ignore attribute.

 * In addition to "cc: <a@dd.re.ss> # cruft", "cc: a@dd.re.ss # cruft"
   was taught to "git send-email" as a valid way to tell it that it
   needs to also send a carbon copy to <a@dd.re.ss> in the trailer
   section.
   (merge cc90750677 mm/send-email-cc-cruft later to maint).

 * "git branch -M a b" while on a branch that is completely unrelated
   to either branch a or branch b misbehaved when multiple worktree
   was in use.  This has been fixed.
   (merge 31824d180d nd/worktree-kill-parse-ref later to maint).

 * "git gc" and friends when multiple worktrees are used off of a
   single repository did not consider the index and per-worktree refs
   of other worktrees as the root for reachability traversal, making
   objects that are in use only in other worktrees to be subject to
   garbage collection.

 * A regression to "gitk --bisect" by a recent update has been fixed.
   (merge 1d0538e486 mh/packed-ref-store-prep later to maint).

 * "git -c submodule.recurse=yes pull" did not work as if the
   "--recurse-submodules" option was given from the command line.
   This has been corrected.

 * Unlike "git commit-tree < file", "git commit-tree -F file" did not
   pass the contents of the file verbatim and instead completed an
   incomplete line at the end, if exists.  The latter has been updated
   to match the behaviour of the former.
   (merge c818e74332 rk/commit-tree-make-F-verbatim later to maint).

 * Many codepaths did not diagnose write failures correctly when disks
   go full, due to their misuse of write_in_full() helper function,
   which have been corrected.
   (merge f48ecd38cb jk/write-in-full-fix later to maint).

 * "git help co" now says "co is aliased to ...", not "git co is".
   (merge b3a8076e0d ks/help-alias-label later to maint).

 * "git archive", especially when used with pathspec, stored an empty
   directory in its output, even though Git itself never does so.
   This has been fixed.
   (merge 4318094047 rs/archive-excluded-directory later to maint).

 * API error-proofing which happens to also squelch warnings from GCC.
   (merge c788c54cde tg/refs-allowed-flags later to maint).

 * The explanation of the cut-line in the commit log editor has been
   slightly tweaked.
   (merge 8c4b1a3593 ks/commit-do-not-touch-cut-line later to maint).

 * "git gc" tries to avoid running two instances at the same time by
   reading and writing pid/host from and to a lock file; it used to
   use an incorrect fscanf() format when reading, which has been
   corrected.
   (merge afe2fab72c aw/gc-lockfile-fscanf-fix later to maint).

 * The scripts to drive TravisCI has been reorganized and then an
   optimization to avoid spending cycles on a branch whose tip is
   tagged has been implemented.
   (merge 8376eb4a8f ls/travis-scriptify later to maint).

 * The test linter has been taught that we do not like "echo -e".
   (merge 1a6d46895d tb/test-lint-echo-e later to maint).

 * Code cmp.std.c nitpick.
   (merge ac7da78ede mh/for-each-string-list-item-empty-fix later to maint).

 * A regression fix for 2.11 that made the code to read the list of
   alternate object stores overrun the end of the string.
   (merge f0f7bebef7 jk/info-alternates-fix later to maint).

 * "git describe --match" learned to take multiple patterns in v2.13
   series, but the feature ignored the patterns after the first one
   and did not work at all.  This has been fixed.
   (merge da769d2986 jk/describe-omit-some-refs later to maint).

 * "git filter-branch" cannot reproduce a history with a tag without
   the tagger field, which only ancient versions of Git allowed to be
   created.  This has been corrected.
   (merge b2c1ca6b4b ic/fix-filter-branch-to-handle-tag-without-tagger later to maint).

 * "git cat-file --textconv" started segfaulting recently, which
   has been corrected.
   (merge cc0ea7c9e5 jk/diff-blob later to maint).

 * The built-in pattern to detect the "function header" for HTML did
   not match <H1>..<H6> elements without any attributes, which has
   been fixed.
   (merge 9c03caca2c ik/userdiff-html-h-element-fix later to maint).

 * "git mailinfo" was loose in decoding quoted printable and produced
   garbage when the two letters after the equal sign are not
   hexadecimal.  This has been fixed.
   (merge c8cf423eab rs/mailinfo-qp-decode-fix later to maint).

 * The machinery to create xdelta used in pack files received the
   sizes of the data in size_t, but lost the higher bits of them by
   storing them in "unsigned int" during the computation, which is
   fixed.

 * The delta format used in the packfile cannot reference data at
   offset larger than what can be expressed in 4-byte, but the
   generator for the data failed to make sure the offset does not
   overflow.  This has been corrected.

 * The documentation for '-X<option>' for merges was misleadingly
   written to suggest that "-s theirs" exists, which is not the case.
   (merge c25d98b2a7 jc/merge-x-theirs-docfix later to maint).

 * "git fast-export" with -M/-C option issued "copy" instruction on a
   path that is simultaneously modified, which was incorrect.
   (merge b3e8ca89cf jt/fast-export-copy-modify-fix later to maint).

 * Many codepaths have been updated to squelch -Wsign-compare
   warnings.
   (merge 071bcaab64 rj/no-sign-compare later to maint).

 * Memory leaks in various codepaths have been plugged.
   (merge 4d01a7fa65 ma/leakplugs later to maint).

 * Recent versions of "git rev-parse --parseopt" did not parse the
   option specification that does not have the optional flags (*=?!)
   correctly, which has been corrected.
   (merge a6304fa4c2 bc/rev-parse-parseopt-fix later to maint).

 * The checkpoint command "git fast-import" did not flush updates to
   refs and marks unless at least one object was created since the
   last checkpoint, which has been corrected, as these things can
   happen without any new object getting created.
   (merge 30e215a65c er/fast-import-dump-refs-on-checkpoint later to maint).

 * Spell the name of our system as "Git" in the output from
   request-pull script.
   (merge e66d7c37a5 ar/request-pull-phrasofix later to maint).

 * Other minor doc, test and build updates and code cleanups.
   (merge f094b89a4d ma/parse-maybe-bool later to maint).
   (merge 39b00fa4d4 jk/drop-sha1-entry-pos later to maint).
   (merge 6cdf8a7929 ma/ts-cleanups later to maint).
   (merge 7560f547e6 ma/up-to-date later to maint).
   (merge 0db3dc75f3 rs/apply-epoch later to maint).
   (merge 74f1bd912b dw/diff-highlight-makefile-fix later to maint).
   (merge f991761eb8 jk/config-lockfile-leak-fix later to maint).
   (merge 150efef1e7 ma/pkt-line-leakfix later to maint).
   (merge 5554451de6 mg/timestamp-t-fix later to maint).
   (merge 276d0e35c0 ma/split-symref-update-fix later to maint).
   (merge 3bc4b8f7c7 bb/doc-eol-dirty later to maint).
   (merge c1bb33c99c jk/system-path-cleanup later to maint).
   (merge ab46e6fc72 cc/subprocess-handshake-missing-capabilities later to maint).
   (merge f7a32dd97f kd/doc-for-each-ref later to maint).
   (merge be94568bc7 ez/doc-duplicated-words-fix later to maint).
   (merge 01e4be6c3d ks/test-readme-phrasofix later to maint).
   (merge 217bb56d4f hn/typofix later to maint).
   (merge c08fd6388c jk/doc-read-tree-table-asciidoctor-fix later to maint).
   (merge c3342b362e ks/doc-use-camelcase-for-config-name later to maint).
   (merge 0bca165fdb jk/validate-headref-fix later to maint).
   (merge 93dbefb389 mr/doc-negative-pathspec later to maint).
   (merge 5e633326e4 ad/doc-markup-fix later to maint).
   (merge 9ca356fa8b rs/cocci-de-paren-call-params later to maint).
   (merge 7099153e8d rs/tag-null-pointer-arith-fix later to maint).
   (merge 0e187d758c rs/run-command-use-alloc-array later to maint).

----------------------------------------------------------------

Changes since v2.14.0 are as follows:

Adam Dinwoodie (1):
      doc: correct command formatting

Andreas Heiduk (2):
      doc: add missing values "none" and "default" for diff.wsErrorHighlight
      doc: clarify "config --bool" behaviour with empty string

Ann T Ropea (1):
      request-pull: capitalise "Git" to make it a proper noun

Anthony Sottile (1):
      git-grep: correct exit code with --quiet and -L

Ben Boeckel (1):
      Documentation: mention that `eol` can change the dirty status of paths

Brandon Casey (7):
      t1502: demonstrate rev-parse --parseopt option mis-parsing
      rev-parse parseopt: do not search help text for flag chars
      rev-parse parseopt: interpret any whitespace as start of help text
      git-rebase: don't ignore unexpected command line arguments
      t0040,t1502: Demonstrate parse_options bugs
      parse-options: write blank line to correct output stream
      parse-options: only insert newline in help text if needed

Brandon Williams (29):
      repo_read_index: don't discard the index
      repository: have the_repository use the_index
      submodule--helper: teach push-check to handle HEAD
      cache.h: add GITMODULES_FILE macro
      config: add config_from_gitmodules
      submodule: remove submodule.fetchjobs from submodule-config parsing
      submodule: remove fetch.recursesubmodules from submodule-config parsing
      submodule: check for unstaged .gitmodules outside of config parsing
      submodule: check for unmerged .gitmodules outside of config parsing
      submodule: merge repo_read_gitmodules and gitmodules_config
      grep: recurse in-process using 'struct repository'
      t7411: check configuration parsing errors
      submodule: don't use submodule_from_name
      add, reset: ensure submodules can be added or reset
      submodule--helper: don't overlay config in remote_submodule_branch
      submodule--helper: don't overlay config in update-clone
      fetch: don't overlay config with submodule-config
      submodule: don't rely on overlayed config when setting diffopts
      unpack-trees: don't respect submodule.update
      submodule: remove submodule_config callback routine
      diff: stop allowing diff to have submodules configured in .git/config
      submodule-config: remove support for overlaying repository config
      submodule-config: move submodule-config functions to submodule-config.c
      submodule-config: lazy-load a repository's .gitmodules file
      unpack-trees: improve loading of .gitmodules
      submodule: remove gitmodules_config
      clone: teach recursive clones to respect -q
      clang-format: outline the git project's coding style
      Makefile: add style build rule

Christian Couder (3):
      refs: use skip_prefix() in ref_is_hidden()
      sub-process: print the cmd when a capability is unsupported
      sha1-lookup: remove sha1_entry_pos() from header file

Daniel Watkins (1):
      diff-highlight: add clean target to Makefile

Dimitrios Christidis (1):
      fmt-merge-msg: fix coding style

Eric Blake (1):
      git-contacts: also recognise "Reported-by:"

Eric Rannaud (1):
      fast-import: checkpoint: dump branches/tags/marks even if object_count==0

Evan Zacks (1):
      doc: fix minor typos (extra/duplicated words)

Han-Wen Nienhuys (5):
      submodule.h: typofix
      submodule.c: describe submodule_to_gitdir() in a new comment
      real_path: clarify return value ownership
      read_gitfile_gently: clarify return value ownership.
      string-list.h: move documentation from Documentation/api/ into header

Heiko Voigt (2):
      t5526: fix some broken && chains
      add test for bug in git-mv for recursive submodules

Hielke Christian Braun (1):
      gitweb: skip unreadable subdirectories

Ian Campbell (4):
      filter-branch: reset $GIT_* before cleaning up
      filter-branch: preserve and restore $GIT_AUTHOR_* and $GIT_COMMITTER_*
      filter-branch: stash away ref map in a branch
      filter-branch: use hash-object instead of mktag

Ilya Kantor (1):
      userdiff: fix HTML hunk header regexp

Jameson Miller (1):
      Improve performance of git status --ignored

Jeff Hostetler (1):
      hashmap: add API to disable item counting when threaded

Jeff King (110):
      t1414: document some reflog-walk oddities
      revision: disallow reflog walking with revs->limited
      log: clarify comment about reflog cycles
      log: do not free parents when walking reflog
      get_revision_1(): replace do-while with an early return
      rev-list: check reflog_info before showing usage
      reflog-walk: stop using fake parents
      reflog-walk: apply --since/--until to reflog dates
      check return value of verify_ref_format()
      docs/for-each-ref: update pointer to color syntax
      t: use test_decode_color rather than literal ANSI codes
      ref-filter: simplify automatic color reset
      ref-filter: abstract ref format into its own struct
      ref-filter: move need_color_reset_at_eol into ref_format
      ref-filter: provide a function for parsing sort options
      ref-filter: make parse_ref_filter_atom a private function
      ref-filter: factor out the parsing of sorting atoms
      ref-filter: pass ref_format struct to atom parsers
      color: check color.ui in git_default_config()
      for-each-ref: load config earlier
      rev-list: pass diffopt->use_colors through to pretty-print
      pretty: respect color settings for %C placeholders
      ref-filter: consult want_color() before emitting colors
      strbuf: use designated initializers in STRBUF_INIT
      t/lib-proto-disable: restore protocol.allow after config tests
      t5813: add test for hostname starting with dash
      connect: factor out "looks like command line option" check
      connect: reject dashed arguments for proxy commands
      connect: reject paths that look like command line options
      t6018: flesh out empty input/output rev-list tests
      revision: add rev_input_given flag
      rev-list: don't show usage when we see empty ref patterns
      revision: do not fallback to default when rev_input_given is set
      hashcmp: use memcmp instead of open-coded loop
      sha1_file: drop experimental GIT_USE_LOOKUP search
      trailer: put process_trailers() options into a struct
      interpret-trailers: add an option to show only the trailers
      interpret-trailers: add an option to show only existing trailers
      interpret-trailers: add an option to unfold values
      interpret-trailers: add --parse convenience option
      pretty: move trailer formatting to trailer.c
      t4205: refactor %(trailers) tests
      pretty: support normalization options for %(trailers)
      doc: fix typo in sendemail.identity
      config: use a static lock_file struct
      write_index_as_tree: cleanup tempfile on error
      setup_temporary_shallow: avoid using inactive tempfile
      setup_temporary_shallow: move tempfile struct into function
      verify_signed_buffer: prefer close_tempfile() to close()
      always check return value of close_tempfile
      tempfile: do not delete tempfile on failed close
      lockfile: do not rollback lock on failed close
      tempfile: prefer is_tempfile_active to bare access
      tempfile: handle NULL tempfile pointers gracefully
      tempfile: replace die("BUG") with BUG()
      tempfile: factor out activation
      tempfile: factor out deactivation
      tempfile: robustify cleanup handler
      tempfile: release deactivated strbufs instead of resetting
      tempfile: use list.h for linked list
      tempfile: remove deactivated list entries
      tempfile: auto-allocate tempfiles on heap
      lockfile: update lifetime requirements in documentation
      ref_lock: stop leaking lock_files
      stop leaking lock structs in some simple cases
      test-lib: --valgrind should not override --verbose-log
      test-lib: set LSAN_OPTIONS to abort by default
      add: free leaked pathspec after add_files_to_cache()
      update-index: fix cache entry leak in add_one_file()
      config: plug user_config leak
      reset: make tree counting less confusing
      reset: free allocated tree buffers
      repository: free fields before overwriting them
      set_git_dir: handle feeding gitdir to itself
      rev-parse: don't trim bisect refnames
      system_path: move RUNTIME_PREFIX to a sub-function
      git_extract_argv0_path: do nothing without RUNTIME_PREFIX
      add UNLEAK annotation for reducing leak false positives
      shortlog: skip format/parse roundtrip for internal traversal
      shell: drop git-cvsserver support by default
      archimport: use safe_pipe_capture for user input
      cvsimport: shell-quote variable used in backticks
      config: avoid "write_in_full(fd, buf, len) < len" pattern
      get-tar-commit-id: check write_in_full() return against 0
      avoid "write_in_full(fd, buf, len) != len" pattern
      convert less-trivial versions of "write_in_full() != len"
      pkt-line: check write_in_full() errors against "< 0"
      notes-merge: use ssize_t for write_in_full() return value
      config: flip return value of store_write_*()
      read_pack_header: handle signed/unsigned comparison in read result
      prefix_ref_iterator: break when we leave the prefix
      read_info_alternates: read contents into strbuf
      read_info_alternates: warn on non-trivial errors
      revision: replace "struct cmdline_pathspec" with argv_array
      cat-file: handle NULL object_context.path
      test-line-buffer: simplify command parsing
      curl_trace(): eliminate switch fallthrough
      consistently use "fallthrough" comments in switches
      doc: put literal block delimiter around table
      files-backend: prefer "0" for write_in_full() error check
      notes-merge: drop dead zero-write code
      prefer "!=" when checking read_in_full() result
      avoid looking at errno for short read_in_full() returns
      distinguish error versus short read from read_in_full()
      worktree: use xsize_t to access file size
      worktree: check the result of read_in_full()
      validate_headref: NUL-terminate HEAD buffer
      validate_headref: use skip_prefix for symref parsing
      validate_headref: use get_oid_hex for detached HEADs
      git: add --no-optional-locks option

Job Snijders (1):
      gitweb: add 'raw' blob_plain link in history overview

Joel Teichroeb (3):
      stash: add a test for stash create with no files
      stash: add a test for when apply fails during stash branch
      stash: add a test for stashing in a detached state

Johannes Schindelin (14):
      run_processes_parallel: change confusing task_cb convention
      git-gui (MinGW): make use of MSys2's msgfmt
      t3415: verify that an empty instructionFormat is handled as before
      rebase -i: generate the script via rebase--helper
      rebase -i: remove useless indentation
      rebase -i: do not invent onelines when expanding/collapsing SHA-1s
      rebase -i: also expand/collapse the SHA-1s via the rebase--helper
      t3404: relax rebase.missingCommitsCheck tests
      rebase -i: check for missing commits in the rebase--helper
      rebase -i: skip unnecessary picks using the rebase--helper
      t3415: test fixup with wrapped oneline
      rebase -i: rearrange fixup/squash lines using the rebase--helper
      Win32: simplify loading of DLL functions
      clang-format: adjust line break penalties

Jonathan Nieder (6):
      vcs-svn: remove more unused prototypes and declarations
      vcs-svn: remove custom mode constants
      vcs-svn: remove repo_delete wrapper function
      vcs-svn: move remaining repo_tree functions to fast_export.h
      pack: make packed_git_mru global a value instead of a pointer
      pathspec doc: parse_pathspec does not maintain references to args

Jonathan Tan (39):
      fsck: remove redundant parse_tree() invocation
      object: remove "used" field from struct object
      fsck: cleanup unused variable
      Documentation: migrate sub-process docs to header
      sub-process: refactor handshake to common function
      tests: ensure fsck fails on corrupt packfiles
      sha1_file: set whence in storage-specific info fn
      sha1_file: remove read_packed_sha1()
      diff: avoid redundantly clearing a flag
      diff: respect MIN_BLOCK_LENGTH for last block
      diff: define block by number of alphanumeric chars
      Doc: clarify that pack-objects makes packs, plural
      pack: move pack name-related functions
      pack: move static state variables
      pack: move pack_report()
      pack: move open_pack_index(), parse_pack_index()
      pack: move release_pack_memory()
      pack: move pack-closing functions
      pack: move use_pack()
      pack: move unuse_pack()
      pack: move add_packed_git()
      pack: move install_packed_git()
      pack: move {,re}prepare_packed_git and approximate_object_count
      pack: move unpack_object_header_buffer()
      pack: move get_size_from_delta()
      pack: move unpack_object_header()
      pack: move clear_delta_base_cache(), packed_object_info(), unpack_entry()
      pack: move nth_packed_object_{sha1,oid}
      pack: move check_pack_index_ptr(), nth_packed_object_offset()
      pack: move find_pack_entry_one(), is_pack_valid()
      pack: move find_sha1_pack()
      pack: move find_pack_entry() and make it global
      pack: move has_sha1_pack()
      pack: move has_pack_index()
      pack: move for_each_packed_object()
      Remove inadvertently added outgoing/packfile.h
      Add t/helper/test-write-cache to .gitignore
      git-compat-util: make UNLEAK less error-prone
      fast-export: do not copy from modified file

Junio C Hamano (49):
      t1408: add a test of stale packed refs covered by loose refs
      clean.c: use designated initializer
      http.c: http.sslcert and http.sslkey are both pathnames
      connect: reject ssh hostname that begins with a dash
      Git 2.7.6
      Git 2.8.6
      Git 2.9.5
      Git 2.10.4
      Git 2.11.3
      Git 2.12.4
      Git 2.13.5
      Git 2.14.1
      Start post 2.14 cycle
      perl/Git.pm: typofix in a comment
      The first batch of topics after the 2.14 cycle
      diff: retire sane_truncate_fn
      progress: simplify "delayed" progress API
      The second batch post 2.14
      t4200: give us a clean slate after "rerere gc" tests
      t4200: make "rerere gc" test more robust
      t4200: gather "rerere gc" together
      t4200: parameterize "rerere gc" custom expiry test
      rerere: represent time duration in timestamp_t internally
      rerere: allow approxidate in gc.rerereResolved/gc.rerereUnresolved
      The third batch post 2.14
      Prepare for 2.14.2
      The fourth batch post 2.14
      The fifth batch post 2.14
      The sixth batch post 2.14
      RelNotes: further fixes for 2.14.2 from the master front
      The seventh batch post 2.14
      travis: dedent a few scripts that are indented overly deeply
      subprocess: loudly die when subprocess asks for an unsupported capability
      cvsserver: move safe_pipe_capture() to the main package
      cvsserver: use safe_pipe_capture for `constant commands` as well
      gc: call fscanf() with %<len>s, not %<len>c, when reading hostname
      The eighth batch for 2.15
      Git 2.10.5
      Git 2.11.4
      Git 2.12.5
      Git 2.13.6
      Git 2.14.2
      branch: fix "copy" to never touch HEAD
      merge-strategies: avoid implying that "-s theirs" exists
      The ninth batch for 2.15
      The tenth batch for 2.15
      The eleventh batch for 2.15
      The twelfth batch for 2.15
      Git 2.15-rc0

Kaartic Sivaraam (13):
      hook: cleanup script
      hook: name the positional variables
      hook: add sign-off using "interpret-trailers"
      hook: add a simple first example
      commit: check for empty message before the check for untouched template
      hook: use correct logical variable
      t3200: cleanup cruft of a test
      builtin/branch: stop supporting the "--set-upstream" option
      branch: quote branch/ref names to improve readability
      help: change a message to be more precise
      commit-template: change a message to be more intuitive
      t/README: fix typo and grammatically improve a sentence
      doc: camelCase the config variables to improve readability

Kevin Daudt (3):
      stash: prevent warning about null bytes in input
      doc/for-each-ref: consistently use '=' to between argument names and values
      doc/for-each-ref: explicitly specify option names

Kevin Willford (9):
      format-patch: have progress option while generating patches
      rebase: turn on progress option by default for format-patch
      commit: skip discarding the index if there is no pre-commit hook
      perf: add test for writing the index
      read-cache: fix memory leak in do_write_index
      read-cache: avoid allocating every ondisk entry when writing
      merge-recursive: fix memory leak
      merge-recursive: remove return value from get_files_dirs
      merge-recursive: change current file dir string_lists to hashmap

Lars Schneider (11):
      t0021: keep filter log files on comparison
      t0021: make debug log file name configurable
      t0021: write "OUT <size>" only on success
      convert: put the flags field before the flag itself for consistent style
      convert: move multiple file filter error handling to separate function
      convert: refactor capabilities negotiation
      convert: add "status=delayed" to filter process protocol
      convert: display progress for filtered objects that have been delayed
      travis-ci: move Travis CI code into dedicated scripts
      travis-ci: skip a branch build if equal tag is present
      travis-ci: fix "skip_branch_tip_with_tag()" string comparison

Manav Rathi (1):
      docs: improve discoverability of exclude pathspec

Martin Koegler (2):
      diff-delta: fix encoding size that would not fit in "unsigned int"
      diff-delta: do not allow delta offset truncation

Martin Ågren (32):
      builtin.h: take over documentation from api-builtin.txt
      git.c: let builtins opt for handling `pager.foo` themselves
      git.c: provide setup_auto_pager()
      t7006: add tests for how git tag paginates
      tag: respect `pager.tag` in list-mode only
      tag: change default of `pager.tag` to "on"
      git.c: ignore pager.* when launching builtin as dashed external
      Doc/git-{push,send-pack}: correct --sign= to --signed=
      t5334: document that git push --signed=1 does not work
      config: introduce git_parse_maybe_bool_text
      config: make git_{config,parse}_maybe_bool equivalent
      treewide: deprecate git_config_maybe_bool, use git_parse_maybe_bool
      parse_decoration_style: drop unused argument `var`
      doc/interpret-trailers: fix "the this" typo
      convert: always initialize attr_action in convert_attrs
      pack-objects: take lock before accessing `remaining`
      strbuf_setlen: don't write to strbuf_slopbuf
      ThreadSanitizer: add suppressions
      Documentation/user-manual: update outdated example output
      treewide: correct several "up-to-date" to "up to date"
      pkt-line: re-'static'-ify buffer in packet_write_fmt_1()
      config: remove git_config_maybe_bool
      refs/files-backend: add longer-scoped copy of string to list
      refs/files-backend: fix memory leak in lock_ref_for_update
      refs/files-backend: correct return value in lock_ref_for_update
      refs/files-backend: add `refname`, not "HEAD", to list
      builtin/commit: fix memory leak in `prepare_index()`
      commit: fix memory leak in `reduce_heads()`
      leak_pending: use `object_array_clear()`, not `free()`
      object_array: use `object_array_clear()`, not `free()`
      object_array: add and use `object_array_pop()`
      pack-bitmap[-write]: use `object_array_clear()`, don't leak

Matthieu Moy (2):
      send-email: fix garbage removal after address
      send-email: don't use Mail::Address, even if available

Max Kirillov (2):
      describe: fix matching to actually match all patterns
      describe: teach --match to handle branches and remotes

Michael Forney (1):
      scripts: use "git foo" not "git-foo"

Michael Haggerty (77):
      add_packed_ref(): teach function to overwrite existing refs
      packed_ref_store: new struct
      packed_ref_store: move `packed_refs_path` here
      packed_ref_store: move `packed_refs_lock` member here
      clear_packed_ref_cache(): take a `packed_ref_store *` parameter
      validate_packed_ref_cache(): take a `packed_ref_store *` parameter
      get_packed_ref_cache(): take a `packed_ref_store *` parameter
      get_packed_refs(): take a `packed_ref_store *` parameter
      add_packed_ref(): take a `packed_ref_store *` parameter
      lock_packed_refs(): take a `packed_ref_store *` parameter
      commit_packed_refs(): take a `packed_ref_store *` parameter
      rollback_packed_refs(): take a `packed_ref_store *` parameter
      get_packed_ref(): take a `packed_ref_store *` parameter
      repack_without_refs(): take a `packed_ref_store *` parameter
      packed_peel_ref(): new function, extracted from `files_peel_ref()`
      packed_ref_store: support iteration
      packed_read_raw_ref(): new function, replacing `resolve_packed_ref()`
      packed-backend: new module for handling packed references
      packed_ref_store: make class into a subclass of `ref_store`
      commit_packed_refs(): report errors rather than dying
      commit_packed_refs(): use a staging file separate from the lockfile
      packed_refs_lock(): function renamed from lock_packed_refs()
      packed_refs_lock(): report errors via a `struct strbuf *err`
      packed_refs_unlock(), packed_refs_is_locked(): new functions
      clear_packed_ref_cache(): don't protest if the lock is held
      commit_packed_refs(): remove call to `packed_refs_unlock()`
      repack_without_refs(): don't lock or unlock the packed refs
      t3210: add some tests of bogus packed-refs file contents
      read_packed_refs(): die if `packed-refs` contains bogus data
      packed_ref_store: handle a packed-refs file that is a symlink
      files-backend: cheapen refname_available check when locking refs
      refs: retry acquiring reference locks for 100ms
      notes: make GET_NIBBLE macro more robust
      load_subtree(): remove unnecessary conditional
      load_subtree(): reduce the scope of some local variables
      load_subtree(): fix incorrect comment
      load_subtree(): separate logic for internal vs. terminal entries
      load_subtree(): check earlier whether an internal node is a tree entry
      load_subtree(): only consider blobs to be potential notes
      get_oid_hex_segment(): return 0 on success
      load_subtree(): combine some common code
      get_oid_hex_segment(): don't pad the rest of `oid`
      hex_to_bytes(): simpler replacement for `get_oid_hex_segment()`
      load_subtree(): declare some variables to be `size_t`
      load_subtree(): check that `prefix_len` is in the expected range
      packed-backend: don't adjust the reference count on lock/unlock
      struct ref_transaction: add a place for backends to store data
      packed_ref_store: implement reference transactions
      packed_delete_refs(): implement method
      files_pack_refs(): use a reference transaction to write packed refs
      prune_refs(): also free the linked list
      files_initial_transaction_commit(): use a transaction for packed refs
      t1404: demonstrate two problems with reference transactions
      files_ref_store: use a transaction to update packed refs
      packed-backend: rip out some now-unused code
      files_transaction_finish(): delete reflogs before references
      ref_iterator: keep track of whether the iterator output is ordered
      packed_ref_cache: add a backlink to the associated `packed_ref_store`
      die_unterminated_line(), die_invalid_line(): new functions
      read_packed_refs(): use mmap to read the `packed-refs` file
      read_packed_refs(): only check for a header at the top of the file
      read_packed_refs(): make parsing of the header line more robust
      for_each_string_list_item: avoid undefined behavior for empty list
      read_packed_refs(): read references with minimal copying
      packed_ref_cache: remember the file-wide peeling state
      mmapped_ref_iterator: add iterator over a packed-refs file
      mmapped_ref_iterator_advance(): no peeled value for broken refs
      packed-backend.c: reorder some definitions
      packed_ref_cache: keep the `packed-refs` file mmapped if possible
      read_packed_refs(): ensure that references are ordered when read
      packed_ref_iterator_begin(): iterate using `mmapped_ref_iterator`
      packed_read_raw_ref(): read the reference from the mmapped buffer
      ref_store: implement `refs_peel_ref()` generically
      packed_ref_store: get rid of the `ref_cache` entirely
      ref_cache: remove support for storing peeled values
      mmapped_ref_iterator: inline into `packed_ref_iterator`
      packed-backend.c: rename a bunch of things and update comments

Michael J Gruber (11):
      Documentation: use proper wording for ref format strings
      Documentation/git-for-each-ref: clarify peeling of tags for --format
      Documentation/git-merge: explain --continue
      merge: clarify call chain
      merge: split write_merge_state in two
      merge: save merge state earlier
      name-rev: change ULONG_MAX to TIME_MAX
      t7004: move limited stack prereq to test-lib
      t6120: test name-rev --all and --stdin
      t6120: clean up state after breaking repo
      t6120: test describe and name-rev with deep repos

Nguyễn Thái Ngọc Duy (17):
      branch: fix branch renaming not updating HEADs correctly
      revision.h: new flag in struct rev_info wrt. worktree-related refs
      refs.c: use is_dir_sep() in resolve_gitlink_ref()
      revision.c: refactor add_index_objects_to_pending()
      revision.c: --indexed-objects add objects from all worktrees
      refs.c: refactor get_submodule_ref_store(), share common free block
      refs: move submodule slash stripping code to get_submodule_ref_store
      refs: add refs_head_ref()
      revision.c: use refs_for_each*() instead of for_each_*_submodule()
      refs.c: move for_each_remote_ref_submodule() to submodule.c
      refs: remove dead for_each_*_submodule()
      revision.c: --all adds HEAD from all worktrees
      files-backend: make reflog iterator go through per-worktree reflog
      revision.c: --reflog add HEAD reflog from all worktrees
      rev-list: expose and document --single-worktree
      refs.c: remove fallback-to-main-store code get_submodule_ref_store()
      refs.c: reindent get_submodule_ref_store()

Nicolas Morey-Chaisemartin (7):
      stash: clean untracked files before reset
      pull: fix cli and config option parsing order
      pull: honor submodule.recurse config option
      imap-send: return with error if curl failed
      imap-send: add wrapper to get server credentials if needed
      imap_send: setup_curl: retreive credentials if not set in config file
      imap-send: use curl by default when possible

Paolo Bonzini (4):
      trailers: export action enums and corresponding lookup functions
      trailers: introduce struct new_trailer_item
      interpret-trailers: add options for actions
      interpret-trailers: fix documentation typo

Patryk Obara (10):
      sha1_file: fix definition of null_sha1
      commit: replace the raw buffer with strbuf in read_graft_line
      commit: allocate array using object_id size
      commit: rewrite read_graft_line
      builtin/hash-object: convert to struct object_id
      read-cache: convert to struct object_id
      sha1_file: convert index_path to struct object_id
      sha1_file: convert index_fd to struct object_id
      sha1_file: convert hash_sha1_file_literally to struct object_id
      sha1_file: convert index_stream to struct object_id

Philip Oakley (4):
      git-gui: remove duplicate entries from .gitconfig's gui.recentrepo
      git gui: cope with duplicates in _get_recentrepo
      git gui: de-dup selected repo from recentrepo history
      git gui: allow for a long recentrepo list

Phillip Wood (7):
      am: remember --rerere-autoupdate setting
      rebase: honor --rerere-autoupdate
      rebase -i: honor --rerere-autoupdate
      t3504: use test_commit
      cherry-pick/revert: remember --rerere-autoupdate
      cherry-pick/revert: reject --rerere-autoupdate when continuing
      am: fix signoff when other trailers are present

Raman Gupta (1):
      contrib/rerere-train: optionally overwrite existing resolutions

Ramsay Jones (9):
      credential-cache: interpret an ECONNRESET as an EOF
      builtin/add: add detail to a 'cannot chmod' error message
      test-lib: don't use ulimit in test prerequisites on cygwin
      test-lib: use more compact expression in PIPE prerequisite
      t9010-*.sh: skip all tests if the PIPE prereq is missing
      git-compat-util.h: xsize_t() - avoid -Wsign-compare warnings
      commit-slab.h: avoid -Wsign-compare warnings
      cache.h: hex2chr() - avoid -Wsign-compare warnings
      ALLOC_GROW: avoid -Wsign-compare warnings

Rene Scharfe (34):
      am: release strbufs after use in detect_patch_format()
      am: release strbuf on error return in hg_patch_to_mail()
      am: release strbuf after use in safe_to_abort()
      check-ref-format: release strbuf after use in check_ref_format_branch()
      clean: release strbuf after use in remove_dirs()
      clone: release strbuf after use in remove_junk()
      commit: release strbuf on error return in commit_tree_extended()
      connect: release strbuf on error return in git_connect()
      convert: release strbuf on error return in filter_buffer_or_fd()
      diff: release strbuf after use in diff_summary()
      diff: release strbuf after use in show_rename_copy()
      diff: release strbuf after use in show_stats()
      help: release strbuf on error return in exec_man_konqueror()
      help: release strbuf on error return in exec_man_man()
      help: release strbuf on error return in exec_woman_emacs()
      mailinfo: release strbuf after use in handle_from()
      mailinfo: release strbuf on error return in handle_boundary()
      merge: release strbuf after use in save_state()
      merge: release strbuf after use in write_merge_heads()
      notes: release strbuf after use in notes_copy_from_stdin()
      refs: release strbuf on error return in write_pseudoref()
      remote: release strbuf after use in read_remote_branches()
      remote: release strbuf after use in migrate_file()
      remote: release strbuf after use in set_url()
      send-pack: release strbuf on error return in send_pack()
      sha1_file: release strbuf on error return in index_path()
      shortlog: release strbuf after use in insert_one_record()
      sequencer: release strbuf after use in save_head()
      transport-helper: release strbuf after use in process_connect_service()
      userdiff: release strbuf after use in userdiff_get_textconv()
      utf8: release strbuf on error return in strbuf_utf8_replace()
      vcs-svn: release strbuf after use in end_revision()
      wt-status: release strbuf after use in read_rebase_todolist()
      wt-status: release strbuf after use in wt_longstatus_print_tracking()

René Scharfe (44):
      tree-diff: don't access hash of NULL object_id pointer
      notes: don't access hash of NULL object_id pointer
      receive-pack: don't access hash of NULL object_id pointer
      bswap: convert to unsigned before shifting in get_be32
      bswap: convert get_be16, get_be32 and put_be32 to inline functions
      add MOVE_ARRAY
      use MOVE_ARRAY
      apply: use COPY_ARRAY and MOVE_ARRAY in update_image()
      ls-files: don't try to prune an empty index
      dir: support platforms that require aligned reads
      pack-objects: remove unnecessary NULL check
      t0001: skip test with restrictive permissions if getpwd(3) respects them
      test-path-utils: handle const parameter of basename and dirname
      t3700: fix broken test under !POSIXPERM
      t4062: use less than 256 repetitions in regex
      sha1_file: avoid comparison if no packed hash matches the first byte
      apply: remove prefix_length member from apply_state
      merge: use skip_prefix()
      win32: plug memory leak on realloc() failure in syslog()
      strbuf: clear errno before calling getdelim(3)
      fsck: free buffers on error in fsck_obj()
      sha1_file: release delta_stack on error in unpack_entry()
      tree-walk: convert fill_tree_descriptor() to object_id
      t1002: stop using sum(1)
      t5001: add tests for export-ignore attributes and exclude pathspecs
      archive: factor out helper functions for handling attributes
      archive: don't queue excluded directories
      commit: remove unused inline function single_parent()
      apply: check date of potential epoch timestamps first
      apply: remove epoch date from regex
      archive: don't add empty directories to archives
      refs: make sha1 output parameter of refs_resolve_ref_unsafe() optional
      refs: pass NULL to refs_resolve_ref_unsafe() if hash is not needed
      refs: pass NULL to resolve_ref_unsafe() if hash is not needed
      mailinfo: don't decode invalid =XY quoted-printable sequences
      refs: pass NULL to refs_resolve_refdup() if hash is not needed
      refs: pass NULL to resolve_refdup() if hash is not needed
      coccinelle: remove parentheses that become unnecessary
      path: use strbuf_add_real_path()
      use strbuf_addstr() for adding strings to strbufs
      graph: use strbuf_addchars() to add spaces
      tag: avoid NULL pointer arithmetic
      repository: use FREE_AND_NULL
      run-command: use ALLOC_ARRAY

Ross Kabus (1):
      commit-tree: do not complete line in -F input

Sahil Dua (2):
      config: create a function to format section headers
      branch: add a --copy (-c) option to go with --move (-m)

Santiago Torres (1):
      t: lib-gpg: flush gpg agent on startup

Stefan Beller (49):
      diff.c: readability fix
      diff.c: move line ending check into emit_hunk_header
      diff.c: factor out diff_flush_patch_all_file_pairs
      diff.c: introduce emit_diff_symbol
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_CONTEXT_MARKER
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_CONTEXT_FRAGINFO
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_NO_LF_EOF
      diff.c: migrate emit_line_checked to use emit_diff_symbol
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_WORDS[_PORCELAIN]
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_CONTEXT_INCOMPLETE
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_FILEPAIR_{PLUS, MINUS}
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_HEADER
      diff.c: emit_diff_symbol learns about DIFF_SYMBOL_BINARY_FILES
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_REWRITE_DIFF
      submodule.c: migrate diff output to use emit_diff_symbol
      diff.c: convert emit_binary_diff_body to use emit_diff_symbol
      diff.c: convert show_stats to use emit_diff_symbol
      diff.c: convert word diffing to use emit_diff_symbol
      diff.c: emit_diff_symbol learns about DIFF_SYMBOL_STAT_SEP
      diff.c: emit_diff_symbol learns about DIFF_SYMBOL_SUMMARY
      diff.c: buffer all output if asked to
      diff.c: color moved lines differently
      diff.c: color moved lines differently, plain mode
      diff.c: add dimming to moved line detection
      diff: document the new --color-moved setting
      attr.c: drop hashmap_cmp_fn cast
      builtin/difftool.c: drop hashmap_cmp_fn cast
      builtin/describe: drop hashmap_cmp_fn cast
      config.c: drop hashmap_cmp_fn cast
      convert/sub-process: drop cast to hashmap_cmp_fn
      patch-ids.c: drop hashmap_cmp_fn cast
      remote.c: drop hashmap_cmp_fn cast
      submodule-config.c: drop hashmap_cmp_fn cast
      name-hash.c: drop hashmap_cmp_fn cast
      t/helper/test-hashmap: use custom data instead of duplicate cmp functions
      commit: convert lookup_commit_graft to struct object_id
      tag: convert gpg_verify_tag to use struct object_id
      t8008: rely on rev-parse'd HEAD instead of sha1 value
      t1200: remove t1200-tutorial.sh
      sha1_file: make read_info_alternates static
      submodule.sh: remove unused variable
      builtin/merge: honor commit-msg hook for merges
      push, fetch: error out for submodule entries not pointing to commits
      replace-objects: evaluate replacement refs without using the object store
      Documentation/githooks: mention merge in commit-msg hook
      Documentation/config: clarify the meaning of submodule.<name>.update
      t7406: submodule.<name>.update command must not be run from .gitmodules
      diff: correct newline in summary for renamed files
      submodule: correct error message for missing commits

Stephan Beyer (1):
      clang-format: add a comment about the meaning/status of the

Takashi Iwai (2):
      sha1dc: build git plumbing code more explicitly
      sha1dc: allow building with the external sha1dc library

Thomas Gummerer (2):
      read-cache: fix index corruption with index v4
      refs: strip out not allowed flags from ref_transaction_update

Tom G. Christensen (2):
      http: fix handling of missing CURLPROTO_*
      http: use a feature check to enable GSSAPI delegation control

Torsten Bögershausen (3):
      convert: add SAFE_CRLF_KEEP_CRLF
      apply: file commited with CRLF should roundtrip diff and apply
      test-lint: echo -e (or -E) is not portable

Urs Thuermann (1):
      git svn fetch: Create correct commit timestamp when using --localtime

William Duclot (1):
      rebase: make resolve message clearer for inexperienced users

brian m. carlson (14):
      builtin/fsck: convert remaining caller of get_sha1 to object_id
      builtin/merge-tree: convert remaining caller of get_sha1 to object_id
      submodule: convert submodule config lookup to use object_id
      remote: convert struct push_cas to struct object_id
      sequencer: convert to struct object_id
      builtin/update_ref: convert to struct object_id
      bisect: convert bisect_checkout to struct object_id
      builtin/unpack-file: convert to struct object_id
      Convert remaining callers of get_sha1 to get_oid.
      sha1_name: convert get_sha1* to get_oid*
      sha1_name: convert GET_SHA1* flags to GET_OID*
      sha1_name: convert uses of 40 to GIT_SHA1_HEXSZ
      vcs-svn: remove unused prototypes
      vcs-svn: rename repo functions to "svn_repo"

joernchen (1):
      cvsserver: use safe_pipe_capture instead of backticks

Ævar Arnfjörð Bjarmason (2):
      branch: add test for -m renaming multiple config sections
      tests: don't give unportable ">" to "test" built-in, use -gt

Øystein Walle (1):
      rev-parse: rev-parse: add --is-shallow-repository

Łukasz Gryglicki (1):
      merge: add a --signoff flag


^ permalink raw reply	[relevance 11%]

* Re: [PATCH 2/6] Update documentation for new directory and status logic
  2017-10-05 20:54 ` [PATCH 2/6] Update documentation for new directory and status logic jameson.miller81
@ 2017-10-05 21:51   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-10-05 21:51 UTC (permalink / raw)
  To: Jameson Miller; +Cc: git, Junio C Hamano, Jeff King, Brandon Williams, Jameson Miller

On Thu, Oct 5, 2017 at 1:54 PM,  <jameson.miller81@gmail.com> wrote:
> From: Jameson Miller <jamill@microsoft.com>
>
> Signed-off-by: Jameson Miller <jamill@microsoft.com>
> ---
>  Documentation/git-status.txt                      | 22 +++++++++++++++++-
>  Documentation/technical/api-directory-listing.txt | 28 +++++++++++++++++++----
>  2 files changed, 45 insertions(+), 5 deletions(-)
>
> diff --git a/Documentation/git-status.txt b/Documentation/git-status.txt
> index 9f3a78a36c..7d1410ff3f 100644
> --- a/Documentation/git-status.txt
> +++ b/Documentation/git-status.txt
> @@ -97,8 +97,28 @@ configuration variable documented in linkgit:git-config[1].
>         (and suppresses the output of submodule summaries when the config option
>         `status.submoduleSummary` is set).
>
> ---ignored::
> +--ignored[=<mode>]::
>         Show ignored files as well.
> ++
> +The mode parameter is used to specify the handling of ignored files.
> +It is optional: it defaults to 'default', and if specified, it must be
> +stuck to the option (e.g. '--ignored=default`, but not `--ignored default`).

Is there a rationale for not accepting `--ignored default`?
(It's just the way OPTION_STRING inputs seem to work,
but in other cases [c.f. man git gc, search "--prune=" ] this is
just implied). Or rather: no need to spell this out explicitly IMHO,
as that draws the users attention to it, which might be confusing.

> ++
> +The possible options are:
> ++
> +       - 'traditional' - Shows ignored files and directories, unless
> +                         --untracked-files=all is specifed, in which case
> +                         individual files in ignored directories are
> +                         displayed.
> +       - 'no'          - Show no ignored files.
> +       - 'matching'    - Shows ignored files and directories matching an
> +                         ignore pattern.
> ++
> +When 'matching' mode is specified, paths that explicity match an
> +ignored pattern are shown. If a directory matches an ignore pattern,
> +then it is shown, but not paths contained in the ignored directory. If
> +a directory does not match an ignore pattern, but all contents are
> +ignored, then the directory is not shown, but all contents are shown.
>
>  -z::
>         Terminate entries with NUL, instead of LF.  This implies
> diff --git a/Documentation/technical/api-directory-listing.txt b/Documentation/technical/api-directory-listing.txt
> index 6c77b4920c..86c981c29c 100644
> --- a/Documentation/technical/api-directory-listing.txt
> +++ b/Documentation/technical/api-directory-listing.txt
> @@ -22,16 +22,20 @@ The notable options are:
>
>  `flags`::
>
> -       A bit-field of options (the `*IGNORED*` flags are mutually exclusive):
> +       A bit-field of options:
>
>  `DIR_SHOW_IGNORED`:::
>
> -       Return just ignored files in `entries[]`, not untracked files.
> +       Return just ignored files in `entries[]`, not untracked
> +       files. This is flag is mutually exclusive with

"is flag is" ?


> +       `DIR_SHOW_IGNORED_TOO`.
>
>  `DIR_SHOW_IGNORED_TOO`:::
>
> -       Similar to `DIR_SHOW_IGNORED`, but return ignored files in `ignored[]`
> -       in addition to untracked files in `entries[]`.
> +       Similar to `DIR_SHOW_IGNORED`, but return ignored files in
> +       `ignored[]` in addition to untracked files in
> +       `entries[]`. This flag is mutually exclusive with
> +       `DIR_SHOW_IGNORED`.
>
>  `DIR_KEEP_UNTRACKED_CONTENTS`:::
>
> @@ -39,6 +43,22 @@ The notable options are:
>         untracked contents of untracked directories are also returned in
>         `entries[]`.
>
> +`DIR_SHOW_IGNORED_TOO_MODE_MATCHING`:::
> +
> +       Only has meaning if `DIR_SHOW_IGNORED_TOO` is also set; if
> +       this is set, returns ignored files and directories that match
> +       an exclude pattern. If a directory matches an exclude pattern,
> +       then the directory is returned and the contained paths are
> +       not. A directory that does not match an exclude pattern will
> +       not be returned even if all of its contents are ignored. In
> +       this case, the contents are returned as individual entries.
> +

I think to make the asciidoc happy, you'd put a '+ LF' as paragraph
delimiter and the subsequent paragraphs are not indented.
C.f. Documentation/git-gc.txt "--auto".
Oh, screw that. It turns out, this place is the only place in our docs
where we use triple colons. So I don't know what I am talking about.

> +If this is set, files and directories
> +       that explicity match an ignore pattern are reported. Implicity
> +       ignored directories (directories that do not match an ignore
> +       pattern, but whose contents are all ignored) are not reported,
> +       instead all of the contents are reported.
> +
>  `DIR_COLLECT_IGNORED`:::
>
>         Special mode for git-add. Return ignored files in `ignored[]` and
> --
> 2.13.6
>

^ permalink raw reply	[relevance 8%]

* [PATCH v7 1/3] submodule--helper: introduce get_submodule_displaypath()
  2017-10-06 13:24 ` [PATCH v7 0/3] Incremental rewrite of git-submodules Prathamesh Chavan
@ 2017-10-06 13:24   ` Prathamesh Chavan
  2017-10-06 13:24   ` [PATCH v7 2/3] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
  2017-10-06 13:24   ` [PATCH v7 3/3] submodule: port submodule subcommand 'status' from shell to C Prathamesh Chavan
  2 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-10-06 13:24 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, hanwen, pc44800, sbeller

Introduce function get_submodule_displaypath() to replace the code
occurring in submodule_init() for generating displaypath of the
submodule with a call to it.

This new function will also be used in other parts of the system
in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 35 +++++++++++++++++++++++------------
 1 file changed, 23 insertions(+), 12 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 06ed02f99..56c1c52e2 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -219,6 +219,26 @@ static int resolve_relative_url_test(int argc, const char **argv, const char *pr
 	return 0;
 }
 
+/* the result should be freed by the caller. */
+static char *get_submodule_displaypath(const char *path, const char *prefix)
+{
+	const char *super_prefix = get_super_prefix();
+
+	if (prefix && super_prefix) {
+		BUG("cannot have prefix '%s' and superprefix '%s'",
+		    prefix, super_prefix);
+	} else if (prefix) {
+		struct strbuf sb = STRBUF_INIT;
+		char *displaypath = xstrdup(relative_path(path, prefix, &sb));
+		strbuf_release(&sb);
+		return displaypath;
+	} else if (super_prefix) {
+		return xstrfmt("%s%s", super_prefix, path);
+	} else {
+		return xstrdup(path);
+	}
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -334,15 +354,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	struct strbuf sb = STRBUF_INIT;
 	char *upd = NULL, *url = NULL, *displaypath;
 
-	if (prefix && get_super_prefix())
-		die("BUG: cannot have prefix and superprefix");
-	else if (prefix)
-		displaypath = xstrdup(relative_path(path, prefix, &sb));
-	else if (get_super_prefix()) {
-		strbuf_addf(&sb, "%s%s", get_super_prefix(), path);
-		displaypath = strbuf_detach(&sb, NULL);
-	} else
-		displaypath = xstrdup(path);
+	displaypath = get_submodule_displaypath(path, prefix);
 
 	sub = submodule_from_path(&null_oid, path);
 
@@ -357,9 +369,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * Set active flag for the submodule being initialized
 	 */
 	if (!is_submodule_active(the_repository, path)) {
-		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
 		git_config_set_gently(sb.buf, "true");
+		strbuf_reset(&sb);
 	}
 
 	/*
@@ -367,7 +379,6 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	 * To look up the url in .git/config, we must not fall back to
 	 * .gitmodules, so look it up directly.
 	 */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.url", sub->name);
 	if (git_config_get_string(sb.buf, &url)) {
 		if (!sub->url)
@@ -404,9 +415,9 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
 	}
+	strbuf_reset(&sb);
 
 	/* Copy "update" setting when it is not set yet */
-	strbuf_reset(&sb);
 	strbuf_addf(&sb, "submodule.%s.update", sub->name);
 	if (git_config_get_string(sb.buf, &upd) &&
 	    sub->update_strategy.type != SM_UPDATE_UNSPECIFIED) {
-- 
2.14.2


^ permalink raw reply	[relevance 21%]

* [PATCH v7 2/3] submodule--helper: introduce for_each_listed_submodule()
  2017-10-06 13:24 ` [PATCH v7 0/3] Incremental rewrite of git-submodules Prathamesh Chavan
  2017-10-06 13:24   ` [PATCH v7 1/3] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
@ 2017-10-06 13:24   ` Prathamesh Chavan
  2017-10-06 13:24   ` [PATCH v7 3/3] submodule: port submodule subcommand 'status' from shell to C Prathamesh Chavan
  2 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-10-06 13:24 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, hanwen, pc44800, sbeller

Introduce function for_each_listed_submodule() and replace a loop
in module_init() with a call to it.

The new function will also be used in other parts of the
system in later patches.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 40 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 5 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 56c1c52e2..29e3fde16 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -14,6 +14,11 @@
 #include "refs.h"
 #include "connect.h"
 
+#define OPT_QUIET (1 << 0)
+
+typedef void (*each_submodule_fn)(const struct cache_entry *list_item,
+				  void *cb_data);
+
 static char *get_default_remote(void)
 {
 	char *dest = NULL, *ret;
@@ -348,7 +353,23 @@ static int module_list(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
-static void init_submodule(const char *path, const char *prefix, int quiet)
+static void for_each_listed_submodule(const struct module_list *list,
+				      each_submodule_fn fn, void *cb_data)
+{
+	int i;
+	for (i = 0; i < list->nr; i++)
+		fn(list->entries[i], cb_data);
+}
+
+struct init_cb {
+	const char *prefix;
+	unsigned int flags;
+};
+
+#define INIT_CB_INIT { NULL, 0 }
+
+static void init_submodule(const char *path, const char *prefix,
+			   unsigned int flags)
 {
 	const struct submodule *sub;
 	struct strbuf sb = STRBUF_INIT;
@@ -410,7 +431,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		if (git_config_set_gently(sb.buf, url))
 			die(_("Failed to register url for submodule path '%s'"),
 			    displaypath);
-		if (!quiet)
+		if (!(flags & OPT_QUIET))
 			fprintf(stderr,
 				_("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
@@ -437,12 +458,18 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 	free(upd);
 }
 
+static void init_submodule_cb(const struct cache_entry *list_item, void *cb_data)
+{
+	struct init_cb *info = cb_data;
+	init_submodule(list_item->name, info->prefix, info->flags);
+}
+
 static int module_init(int argc, const char **argv, const char *prefix)
 {
+	struct init_cb info = INIT_CB_INIT;
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
-	int i;
 
 	struct option module_init_options[] = {
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
@@ -467,8 +494,11 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
 
-	for (i = 0; i < list.nr; i++)
-		init_submodule(list.entries[i]->name, prefix, quiet);
+	info.prefix = prefix;
+	if (quiet)
+		info.flags |= OPT_QUIET;
+
+	for_each_listed_submodule(&list, init_submodule_cb, &info);
 
 	return 0;
 }
-- 
2.14.2


^ permalink raw reply	[relevance 21%]

* [PATCH v7 3/3] submodule: port submodule subcommand 'status' from shell to C
  2017-10-06 13:24 ` [PATCH v7 0/3] Incremental rewrite of git-submodules Prathamesh Chavan
  2017-10-06 13:24   ` [PATCH v7 1/3] submodule--helper: introduce get_submodule_displaypath() Prathamesh Chavan
  2017-10-06 13:24   ` [PATCH v7 2/3] submodule--helper: introduce for_each_listed_submodule() Prathamesh Chavan
@ 2017-10-06 13:24   ` Prathamesh Chavan
  2 siblings, 0 replies; 200+ results
From: Prathamesh Chavan @ 2017-10-06 13:24 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, hanwen, pc44800, sbeller

This aims to make git-submodule 'status' a built-in. Hence, the function
cmd_status() is ported from shell to C. This is done by introducing
four functions: module_status(), submodule_status_cb(),
submodule_status() and print_status().

The function module_status() acts as the front-end of the subcommand.
It parses subcommand's options and then calls the function
module_list_compute() for computing the list of submodules. Then
this functions calls for_each_listed_submodule() looping through the
list obtained.

Then for_each_listed_submodule() calls submodule_status_cb() for each of
the submodule in its list. The function submodule_status_cb() calls
submodule_status() after passing appropriate arguments to the funciton.
Function submodule_status() is responsible for generating the status
each submodule it is called for, and then calls print_status().

Finally, the function print_status() handles the printing of submodule's
status.

Function set_name_rev() is also ported from git-submodule to the
submodule--helper builtin function compute_rev_name(), which now
generates the value of the revision name as required.

Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
---
 builtin/submodule--helper.c | 198 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  61 +-------------
 2 files changed, 199 insertions(+), 60 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 29e3fde16..d366e8e7b 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -13,8 +13,13 @@
 #include "remote.h"
 #include "refs.h"
 #include "connect.h"
+#include "revision.h"
+#include "diffcore.h"
+#include "diff.h"
 
 #define OPT_QUIET (1 << 0)
+#define OPT_CACHED (1 << 1)
+#define OPT_RECURSIVE (1 << 2)
 
 typedef void (*each_submodule_fn)(const struct cache_entry *list_item,
 				  void *cb_data);
@@ -244,6 +249,44 @@ static char *get_submodule_displaypath(const char *path, const char *prefix)
 	}
 }
 
+static char *compute_rev_name(const char *sub_path, const char* object_id)
+{
+	struct strbuf sb = STRBUF_INIT;
+	const char ***d;
+
+	static const char *describe_bare[] = { NULL };
+
+	static const char *describe_tags[] = { "--tags", NULL };
+
+	static const char *describe_contains[] = { "--contains", NULL };
+
+	static const char *describe_all_always[] = { "--all", "--always", NULL };
+
+	static const char **describe_argv[] = { describe_bare, describe_tags,
+						describe_contains,
+						describe_all_always, NULL };
+
+	for (d = describe_argv; *d; d++) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+		prepare_submodule_repo_env(&cp.env_array);
+		cp.dir = sub_path;
+		cp.git_cmd = 1;
+		cp.no_stderr = 1;
+
+		argv_array_push(&cp.args, "describe");
+		argv_array_pushv(&cp.args, *d);
+		argv_array_push(&cp.args, object_id);
+
+		if (!capture_command(&cp, &sb, 0)) {
+			strbuf_strip_suffix(&sb, "\n");
+			return strbuf_detach(&sb, NULL);
+		}
+	}
+
+	strbuf_release(&sb);
+	return NULL;
+}
+
 struct module_list {
 	const struct cache_entry **entries;
 	int alloc, nr;
@@ -503,6 +546,160 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+struct status_cb {
+	const char *prefix;
+	unsigned int flags;
+};
+
+#define STATUS_CB_INIT { NULL, 0 }
+
+static void print_status(unsigned int flags, char state, const char *path,
+			 const struct object_id *oid, const char *displaypath)
+{
+	if (flags & OPT_QUIET)
+		return;
+
+	printf("%c%s %s", state, oid_to_hex(oid), displaypath);
+
+	if (state == ' ' || state == '+')
+		printf(" (%s)", compute_rev_name(path, oid_to_hex(oid)));
+
+	printf("\n");
+}
+
+static int handle_submodule_head_ref(const char *refname,
+				     const struct object_id *oid, int flags,
+				     void *cb_data)
+{
+	struct object_id *output = cb_data;
+	if (oid)
+		oidcpy(output, oid);
+
+	return 0;
+}
+
+static void status_submodule(const char *path, const struct object_id *ce_oid,
+			     unsigned int ce_flags, const char *prefix,
+			     unsigned int flags)
+{
+	char *displaypath;
+	struct argv_array diff_files_args = ARGV_ARRAY_INIT;
+	struct rev_info rev;
+	int diff_files_result;
+
+	if (!submodule_from_path(&null_oid, path))
+		die(_("no submodule mapping found in .gitmodules for path '%s'"),
+		      path);
+
+	displaypath = get_submodule_displaypath(path, prefix);
+
+	if ((CE_STAGEMASK & ce_flags) >> CE_STAGESHIFT) {
+		print_status(flags, 'U', path, &null_oid, displaypath);
+		goto cleanup;
+	}
+
+	if (!is_submodule_active(the_repository, path)) {
+		print_status(flags, '-', path, ce_oid, displaypath);
+		goto cleanup;
+	}
+
+	argv_array_pushl(&diff_files_args, "diff-files",
+			 "--ignore-submodules=dirty", "--quiet", "--",
+			 path, NULL);
+
+	git_config(git_diff_basic_config, NULL);
+	init_revisions(&rev, prefix);
+	rev.abbrev = 0;
+	diff_files_args.argc = setup_revisions(diff_files_args.argc,
+					       diff_files_args.argv,
+					       &rev, NULL);
+	diff_files_result = run_diff_files(&rev, 0);
+
+	if (!diff_result_code(&rev.diffopt, diff_files_result)) {
+		print_status(flags, ' ', path, ce_oid,
+			     displaypath);
+	} else if (!(flags & OPT_CACHED)) {
+		struct object_id oid;
+
+		if (refs_head_ref(get_submodule_ref_store(path),
+				  handle_submodule_head_ref, &oid))
+			die(_("could not resolve HEAD ref inside the"
+			      "submodule '%s'"), path);
+
+		print_status(flags, '+', path, &oid, displaypath);
+	} else {
+		print_status(flags, '+', path, ce_oid, displaypath);
+	}
+
+	if (flags & OPT_RECURSIVE) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = path;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_push(&cpr.args, "--super-prefix");
+		argv_array_pushf(&cpr.args, "%s/", displaypath);
+		argv_array_pushl(&cpr.args, "submodule--helper", "status",
+				 "--recursive", NULL);
+
+		if (flags & OPT_CACHED)
+			argv_array_push(&cpr.args, "--cached");
+
+		if (flags & OPT_QUIET)
+			argv_array_push(&cpr.args, "--quiet");
+
+		if (run_command(&cpr))
+			die(_("failed to recurse into submodule '%s'"), path);
+	}
+
+cleanup:
+	argv_array_clear(&diff_files_args);
+	free(displaypath);
+}
+
+static void status_submodule_cb(const struct cache_entry *list_item,
+				void *cb_data)
+{
+	struct status_cb *info = cb_data;
+	status_submodule(list_item->name, &list_item->oid, list_item->ce_flags,
+			 info->prefix, info->flags);
+}
+
+static int module_status(int argc, const char **argv, const char *prefix)
+{
+	struct status_cb info = STATUS_CB_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+	int quiet = 0;
+
+	struct option module_status_options[] = {
+		OPT__QUIET(&quiet, N_("Suppress submodule status output")),
+		OPT_BIT(0, "cached", &info.flags, N_("Use commit stored in the index instead of the one stored in the submodule HEAD"), OPT_CACHED),
+		OPT_BIT(0, "recursive", &info.flags, N_("recurse into nested submodules"), OPT_RECURSIVE),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule status [--quiet] [--cached] [--recursive] [<path>...]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_status_options,
+			     git_submodule_helper_usage, 0);
+
+	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.prefix = prefix;
+	if (quiet)
+		info.flags |= OPT_QUIET;
+
+	for_each_listed_submodule(&list, status_submodule_cb, &info);
+
+	return 0;
+}
+
 static int module_name(int argc, const char **argv, const char *prefix)
 {
 	const struct submodule *sub;
@@ -1300,6 +1497,7 @@ static struct cmd_struct commands[] = {
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
+	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"push-check", push_check, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
diff --git a/git-submodule.sh b/git-submodule.sh
index 66d1ae8ef..156255a9e 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -758,18 +758,6 @@ cmd_update()
 	}
 }
 
-set_name_rev () {
-	revname=$( (
-		sanitize_submodule_env
-		cd "$1" && {
-			git describe "$2" 2>/dev/null ||
-			git describe --tags "$2" 2>/dev/null ||
-			git describe --contains "$2" 2>/dev/null ||
-			git describe --all --always "$2"
-		}
-	) )
-	test -z "$revname" || revname=" ($revname)"
-}
 #
 # Show commit summary for submodules in index or working tree
 #
@@ -1016,54 +1004,7 @@ cmd_status()
 		shift
 	done
 
-	{
-		git submodule--helper list --prefix "$wt_prefix" "$@" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path") || exit
-		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-		if test "$stage" = U
-		then
-			say "U$sha1 $displaypath"
-			continue
-		fi
-		if ! git submodule--helper is-active "$sm_path" ||
-		{
-			! test -d "$sm_path"/.git &&
-			! test -f "$sm_path"/.git
-		}
-		then
-			say "-$sha1 $displaypath"
-			continue;
-		fi
-		if git diff-files --ignore-submodules=dirty --quiet -- "$sm_path"
-		then
-			set_name_rev "$sm_path" "$sha1"
-			say " $sha1 $displaypath$revname"
-		else
-			if test -z "$cached"
-			then
-				sha1=$(sanitize_submodule_env; cd "$sm_path" && git rev-parse --verify HEAD)
-			fi
-			set_name_rev "$sm_path" "$sha1"
-			say "+$sha1 $displaypath$revname"
-		fi
-
-		if test -n "$recursive"
-		then
-			(
-				prefix="$displaypath/"
-				sanitize_submodule_env
-				wt_prefix=
-				cd "$sm_path" &&
-				eval cmd_status
-			) ||
-			die "$(eval_gettext "Failed to recurse into submodule path '\$sm_path'")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper status ${GIT_QUIET:+--quiet} ${cached:+--cached} ${recursive:+--recursive} "$@"
 }
 #
 # Sync remote urls for submodules
-- 
2.14.2


^ permalink raw reply	[relevance 19%]

* [PATCH 2/2] tests: fix diff order arguments in test_cmp
  2017-10-06 19:00 [PATCH 1/2] tests: use shell negation instead of test_must_fail for test_cmp Stefan Beller
@ 2017-10-06 19:00 ` " Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2017-10-06 19:00 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller

Fix the argument order for test_cmp. When given the expected
result first the diff shows the actual output with '+' and the
expectation with '-', which is the convention for our tests.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 t/t1004-read-tree-m-u-wf.sh          |  2 +-
 t/t4015-diff-whitespace.sh           |  4 ++--
 t/t4205-log-pretty-formats.sh        |  2 +-
 t/t6007-rev-list-cherry-pick-file.sh | 32 ++++++++++++++++----------------
 t/t6013-rev-list-reverse-parents.sh  |  4 ++--
 t/t7001-mv.sh                        |  2 +-
 t/t7005-editor.sh                    |  6 +++---
 t/t7102-reset.sh                     |  4 ++--
 t/t7201-co.sh                        |  4 ++--
 t/t7400-submodule-basic.sh           |  2 +-
 t/t7405-submodule-merge.sh           |  2 +-
 t/t7506-status-submodule.sh          |  4 ++--
 t/t7600-merge.sh                     |  6 +++---
 t/t7610-mergetool.sh                 |  4 ++--
 t/t9001-send-email.sh                |  6 +++---
 15 files changed, 42 insertions(+), 42 deletions(-)

diff --git a/t/t1004-read-tree-m-u-wf.sh b/t/t1004-read-tree-m-u-wf.sh
index c70cf42300..c7ce5d8bb5 100755
--- a/t/t1004-read-tree-m-u-wf.sh
+++ b/t/t1004-read-tree-m-u-wf.sh
@@ -218,7 +218,7 @@ test_expect_success 'D/F' '
 		echo "100644 $a 2	subdir/file2"
 		echo "100644 $b 3	subdir/file2/another"
 	) >expect &&
-	test_cmp actual expect
+	test_cmp expect actual
 
 '
 
diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh
index 12d182dc1b..1709e4578b 100755
--- a/t/t4015-diff-whitespace.sh
+++ b/t/t4015-diff-whitespace.sh
@@ -155,7 +155,7 @@ test_expect_success 'ignore-blank-lines: only new lines' '
 " >x &&
 	git diff --ignore-blank-lines >out &&
 	>expect &&
-	test_cmp out expect
+	test_cmp expect out
 '
 
 test_expect_success 'ignore-blank-lines: only new lines with space' '
@@ -165,7 +165,7 @@ test_expect_success 'ignore-blank-lines: only new lines with space' '
  " >x &&
 	git diff -w --ignore-blank-lines >out &&
 	>expect &&
-	test_cmp out expect
+	test_cmp expect out
 '
 
 test_expect_success 'ignore-blank-lines: after change' '
diff --git a/t/t4205-log-pretty-formats.sh b/t/t4205-log-pretty-formats.sh
index ec5f530102..42f584f8b3 100755
--- a/t/t4205-log-pretty-formats.sh
+++ b/t/t4205-log-pretty-formats.sh
@@ -590,7 +590,7 @@ test_expect_success '%(trailers:unfold) unfolds trailers' '
 test_expect_success ':only and :unfold work together' '
 	git log --no-walk --pretty="%(trailers:only:unfold)" >actual &&
 	git log --no-walk --pretty="%(trailers:unfold:only)" >reverse &&
-	test_cmp actual reverse &&
+	test_cmp reverse actual &&
 	{
 		grep -v patch.description <trailers | unfold &&
 		echo
diff --git a/t/t6007-rev-list-cherry-pick-file.sh b/t/t6007-rev-list-cherry-pick-file.sh
index 2959745196..f0268372d2 100755
--- a/t/t6007-rev-list-cherry-pick-file.sh
+++ b/t/t6007-rev-list-cherry-pick-file.sh
@@ -57,7 +57,7 @@ test_expect_success '--left-right' '
 	git rev-list --left-right B...C > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 test_expect_success '--count' '
@@ -77,14 +77,14 @@ test_expect_success '--cherry-pick bar does not come up empty' '
 	git rev-list --left-right --cherry-pick B...C -- bar > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 test_expect_success 'bar does not come up empty' '
 	git rev-list --left-right B...C -- bar > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 cat >expect <<EOF
@@ -96,14 +96,14 @@ test_expect_success '--cherry-pick bar does not come up empty (II)' '
 	git rev-list --left-right --cherry-pick F...E -- bar > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 test_expect_success 'name-rev multiple --refs combine inclusive' '
 	git rev-list --left-right --cherry-pick F...E -- bar >actual &&
 	git name-rev --stdin --name-only --refs="*tags/F" --refs="*tags/E" \
 		<actual >actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 cat >expect <<EOF
@@ -115,7 +115,7 @@ test_expect_success 'name-rev --refs excludes non-matched patterns' '
 	git rev-list --left-right --cherry-pick F...E -- bar >actual &&
 	git name-rev --stdin --name-only --refs="*tags/F" \
 		<actual >actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 cat >expect <<EOF
@@ -127,14 +127,14 @@ test_expect_success 'name-rev --exclude excludes matched patterns' '
 	git rev-list --left-right --cherry-pick F...E -- bar >actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" --exclude="*E" \
 		<actual >actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 test_expect_success 'name-rev --no-refs clears the refs list' '
 	git rev-list --left-right --cherry-pick F...E -- bar >expect &&
 	git name-rev --stdin --name-only --refs="*tags/F" --refs="*tags/E" --no-refs --refs="*tags/G" \
 		<expect >actual &&
-	test_cmp actual expect
+	test_cmp expect actual
 '
 
 cat >expect <<EOF
@@ -148,7 +148,7 @@ test_expect_success '--cherry-mark' '
 	git rev-list --cherry-mark F...E -- bar > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 cat >expect <<EOF
@@ -162,7 +162,7 @@ test_expect_success '--cherry-mark --left-right' '
 	git rev-list --cherry-mark --left-right F...E -- bar > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 cat >expect <<EOF
@@ -173,14 +173,14 @@ test_expect_success '--cherry-pick --right-only' '
 	git rev-list --cherry-pick --right-only F...E -- bar > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 test_expect_success '--cherry-pick --left-only' '
 	git rev-list --cherry-pick --left-only E...F -- bar > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 cat >expect <<EOF
@@ -192,7 +192,7 @@ test_expect_success '--cherry' '
 	git rev-list --cherry F...E -- bar > actual &&
 	git name-rev --stdin --name-only --refs="*tags/*" \
 		< actual > actual.named &&
-	test_cmp actual.named expect
+	test_cmp expect actual.named
 '
 
 cat >expect <<EOF
@@ -201,7 +201,7 @@ EOF
 
 test_expect_success '--cherry --count' '
 	git rev-list --cherry --count F...E -- bar > actual &&
-	test_cmp actual expect
+	test_cmp expect actual
 '
 
 cat >expect <<EOF
@@ -210,7 +210,7 @@ EOF
 
 test_expect_success '--cherry-mark --count' '
 	git rev-list --cherry-mark --count F...E -- bar > actual &&
-	test_cmp actual expect
+	test_cmp expect actual
 '
 
 cat >expect <<EOF
@@ -219,7 +219,7 @@ EOF
 
 test_expect_success '--cherry-mark --left-right --count' '
 	git rev-list --cherry-mark --left-right --count F...E -- bar > actual &&
-	test_cmp actual expect
+	test_cmp expect actual
 '
 
 test_expect_success '--cherry-pick with independent, but identical branches' '
diff --git a/t/t6013-rev-list-reverse-parents.sh b/t/t6013-rev-list-reverse-parents.sh
index 59fc2f06e0..89458d370f 100755
--- a/t/t6013-rev-list-reverse-parents.sh
+++ b/t/t6013-rev-list-reverse-parents.sh
@@ -28,7 +28,7 @@ test_expect_success '--reverse --parents --full-history combines correctly' '
 		perl -e "print reverse <>" > expected &&
 	git rev-list --reverse --parents --full-history master -- foo \
 		> actual &&
-	test_cmp actual expected
+	test_cmp expected actual
 	'
 
 test_expect_success '--boundary does too' '
@@ -36,7 +36,7 @@ test_expect_success '--boundary does too' '
 		perl -e "print reverse <>" > expected &&
 	git rev-list --boundary --reverse --parents --full-history \
 		master ^root -- foo > actual &&
-	test_cmp actual expected
+	test_cmp expected actual
 	'
 
 test_done
diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
index cbc5fb37fe..f5929c46f3 100755
--- a/t/t7001-mv.sh
+++ b/t/t7001-mv.sh
@@ -488,7 +488,7 @@ test_expect_success 'moving a submodule in nested directories' '
 		git config -f ../.gitmodules submodule.deep/directory/hierarchy/sub.path >../actual &&
 		echo "directory/hierarchy/sub" >../expect
 	) &&
-	test_cmp actual expect
+	test_cmp expect actual
 '
 
 test_expect_failure 'moving nested submodules' '
diff --git a/t/t7005-editor.sh b/t/t7005-editor.sh
index 1b530b5022..29e5043b94 100755
--- a/t/t7005-editor.sh
+++ b/t/t7005-editor.sh
@@ -38,7 +38,7 @@ test_expect_success setup '
 	test_commit "$msg" &&
 	echo "$msg" >expect &&
 	git show -s --format=%s > actual &&
-	test_cmp actual expect
+	test_cmp expect actual
 
 '
 
@@ -85,7 +85,7 @@ do
 		git --exec-path=. commit --amend &&
 		git show -s --pretty=oneline |
 		sed -e "s/^[0-9a-f]* //" >actual &&
-		test_cmp actual expect
+		test_cmp expect actual
 	'
 done
 
@@ -107,7 +107,7 @@ do
 		git --exec-path=. commit --amend &&
 		git show -s --pretty=oneline |
 		sed -e "s/^[0-9a-f]* //" >actual &&
-		test_cmp actual expect
+		test_cmp expect actual
 	'
 done
 
diff --git a/t/t7102-reset.sh b/t/t7102-reset.sh
index 86f23be34a..95653a08ca 100755
--- a/t/t7102-reset.sh
+++ b/t/t7102-reset.sh
@@ -428,9 +428,9 @@ test_expect_success 'test --mixed <paths>' '
 	git reset HEAD -- file1 file2 file3 &&
 	test_must_fail git diff --quiet &&
 	git diff > output &&
-	test_cmp output expect &&
+	test_cmp expect output &&
 	git diff --cached > output &&
-	test_cmp output cached_expect
+	test_cmp cached_expect output
 '
 
 test_expect_success 'test resetting the index at give paths' '
diff --git a/t/t7201-co.sh b/t/t7201-co.sh
index d4b217b0ee..76c223c967 100755
--- a/t/t7201-co.sh
+++ b/t/t7201-co.sh
@@ -187,7 +187,7 @@ test_expect_success 'format of merge conflict from checkout -m' '
 	d
 	>>>>>>> local
 	EOF
-	test_cmp two expect
+	test_cmp expect two
 '
 
 test_expect_success 'checkout --merge --conflict=diff3 <branch>' '
@@ -213,7 +213,7 @@ test_expect_success 'checkout --merge --conflict=diff3 <branch>' '
 	d
 	>>>>>>> local
 	EOF
-	test_cmp two expect
+	test_cmp expect two
 '
 
 test_expect_success 'switch to another branch while carrying a deletion' '
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 6f8337ffb5..a39e69a3eb 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1211,7 +1211,7 @@ test_expect_success 'clone --recurse-submodules with a pathspec works' '
 
 	git clone --recurse-submodules="sub0" multisuper multisuper_clone &&
 	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
-	test_cmp actual expected
+	test_cmp expected actual
 '
 
 test_expect_success 'clone with multiple --recurse-submodules options' '
diff --git a/t/t7405-submodule-merge.sh b/t/t7405-submodule-merge.sh
index 0d5b42a25b..7bfb2f498d 100755
--- a/t/t7405-submodule-merge.sh
+++ b/t/t7405-submodule-merge.sh
@@ -119,7 +119,7 @@ test_expect_success 'merge with one side as a fast-forward of the other' '
 	 git ls-tree test-forward sub | cut -f1 | cut -f3 -d" " > actual &&
 	 (cd sub &&
 	  git rev-parse sub-d > ../expect) &&
-	 test_cmp actual expect)
+	 test_cmp expect actual)
 '
 
 test_expect_success 'merging should conflict for non fast-forward' '
diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh
index 055c90736e..9edf6572ed 100755
--- a/t/t7506-status-submodule.sh
+++ b/t/t7506-status-submodule.sh
@@ -306,7 +306,7 @@ test_expect_success 'diff with merge conflict in .gitmodules' '
 		cd super &&
 		git diff >../diff_actual 2>&1
 	) &&
-	test_cmp diff_actual diff_expect
+	test_cmp diff_expect diff_actual
 '
 
 test_expect_success 'diff --submodule with merge conflict in .gitmodules' '
@@ -314,7 +314,7 @@ test_expect_success 'diff --submodule with merge conflict in .gitmodules' '
 		cd super &&
 		git diff --submodule >../diff_submodule_actual 2>&1
 	) &&
-	test_cmp diff_submodule_actual diff_submodule_expect
+	test_cmp diff_submodule_expect diff_submodule_actual
 '
 
 # We'll setup different cases for further testing:
diff --git a/t/t7600-merge.sh b/t/t7600-merge.sh
index 80194b79f9..dfde6a675a 100755
--- a/t/t7600-merge.sh
+++ b/t/t7600-merge.sh
@@ -697,7 +697,7 @@ test_expect_success 'merge --no-ff --edit' '
 	git cat-file commit HEAD >raw &&
 	grep "work done on the side branch" raw &&
 	sed "1,/^$/d" >actual raw &&
-	test_cmp actual expected
+	test_cmp expected actual
 '
 
 test_expect_success GPG 'merge --ff-only tag' '
@@ -709,7 +709,7 @@ test_expect_success GPG 'merge --ff-only tag' '
 	git merge --ff-only signed &&
 	git rev-parse signed^0 >expect &&
 	git rev-parse HEAD >actual &&
-	test_cmp actual expect
+	test_cmp expect actual
 '
 
 test_expect_success GPG 'merge --no-edit tag should skip editor' '
@@ -721,7 +721,7 @@ test_expect_success GPG 'merge --no-edit tag should skip editor' '
 	EDITOR=false git merge --no-edit signed &&
 	git rev-parse signed^0 >expect &&
 	git rev-parse HEAD^2 >actual &&
-	test_cmp actual expect
+	test_cmp expect actual
 '
 
 test_expect_success 'set up mod-256 conflict scenario' '
diff --git a/t/t7610-mergetool.sh b/t/t7610-mergetool.sh
index 381b7df452..1a430b9c40 100755
--- a/t/t7610-mergetool.sh
+++ b/t/t7610-mergetool.sh
@@ -621,7 +621,7 @@ test_expect_success 'file with no base' '
 	test_must_fail git merge master &&
 	git mergetool --no-prompt --tool mybase -- both &&
 	>expected &&
-	test_cmp both expected
+	test_cmp expected both
 '
 
 test_expect_success 'custom commands override built-ins' '
@@ -632,7 +632,7 @@ test_expect_success 'custom commands override built-ins' '
 	test_must_fail git merge master &&
 	git mergetool --no-prompt --tool defaults -- both &&
 	echo master both added >expected &&
-	test_cmp both expected
+	test_cmp expected both
 '
 
 test_expect_success 'filenames seen by tools start with ./' '
diff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh
index f30980895c..4d261c2a9c 100755
--- a/t/t9001-send-email.sh
+++ b/t/t9001-send-email.sh
@@ -1266,7 +1266,7 @@ test_expect_success $PREREQ 'asks about and fixes 8bit encodings' '
 	grep email-using-8bit stdout &&
 	grep "Which 8bit encoding" stdout &&
 	egrep "Content|MIME" msgtxt1 >actual &&
-	test_cmp actual content-type-decl
+	test_cmp content-type-decl actual
 '
 
 test_expect_success $PREREQ 'sendemail.8bitEncoding works' '
@@ -1277,7 +1277,7 @@ test_expect_success $PREREQ 'sendemail.8bitEncoding works' '
 			--smtp-server="$(pwd)/fake.sendmail" \
 			email-using-8bit >stdout &&
 	egrep "Content|MIME" msgtxt1 >actual &&
-	test_cmp actual content-type-decl
+	test_cmp content-type-decl actual
 '
 
 test_expect_success $PREREQ '--8bit-encoding overrides sendemail.8bitEncoding' '
@@ -1289,7 +1289,7 @@ test_expect_success $PREREQ '--8bit-encoding overrides sendemail.8bitEncoding' '
 			--8bit-encoding=UTF-8 \
 			email-using-8bit >stdout &&
 	egrep "Content|MIME" msgtxt1 >actual &&
-	test_cmp actual content-type-decl
+	test_cmp content-type-decl actual
 '
 
 test_expect_success $PREREQ 'setup expect' '
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 14%]

* Re: [RFC PATCH v3 0/4] implement fetching of moved submodules
  2017-10-06 22:25 [RFC PATCH v3 0/4] implement fetching of moved submodules Heiko Voigt
  2017-10-06 22:32 ` [RFC PATCH 2/4] change submodule push test to use proper repository setup Heiko Voigt
@ 2017-10-06 22:57 ` Stefan Beller
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2017-10-06 22:57 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Fri, Oct 6, 2017 at 3:25 PM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> The last iteration can be found here:
>
> https://public-inbox.org/git/20170817105349.GC52233@book.hvoigt.net/
>
> This is mainly a status update and to let people know that I am still
> working on this.

Cool. :)

> I struggled quite a bit with reviving my original test for the path
> based recursive fetch (first patch). The behavior seems to haved changed
> and simply setting the submodule configuration in .git/config without
> one in .gitmodules does not work anymore. I did not have time to
> investigate whether this was a deliberate change or a maybe a bug?

I think it is deliberate. (We wrote this man page "gitsubmodules"[1] and there
was so much discussion on "What is a submodule?". Key take away is this:
* a gitlink alone is not a submodule.
* a submodule consists of at least
  -> the gitlink in the superproject
  -> a mapping of path -> name via
      $(git config -f .gitmodules submodule.<name>.path)
  -> Depending on config in .git/config or the existence of its git directory,
      it may be [in]active or [de]initialized.

[1] not to be confused with "gitmodules" or "git-submodule"

Sometimes we accept a plain git-link without the config in .gitmodules,
(a) due to historic reasons or (b) because it seems sane even for
a repo "that just happens to exist at the gitlinks location"
(example git-diff)

> So the solution for now is that I write my fake configuration (to avoid
> skipping submodule handling altogether) into a .gitmodules file.

I'll try to spot what is fake about the config.

> The second patch (cleanup of a submodule push testcase) was written
> because that currently is the only test failing. It is not meant for
> inclusion but rather as a demonstration of what might be happening when
> we cleanup testcases: Because of the behavioral change above, on first
> sight, it seemed like there was a shortcut in fetch and so on-demand
> fetch without submodule configuration would not be supported anymore.
>
> IIRC there were a lot more tests failing before when I implemented my
> patch without the fallback on paths. So my guess is that some tests have
> been cleaned up to use proper (.gitmodules) submodule setup.

I don't remember any large recent activity for submodule things lately.

> So the thing here is: If we want to make sure that we stay backwards
> compatible by supporting the setup with gitlinks without configuration.
> Then we also should keep tests around that have the plain manual setup
> without .gitmodules files. Just something, I think, we should keep in
> mind.

But do we want this?

Without the name<->path mapping, we can only have the "old style"
submodules, that have their git repo inside its tree instead of inside
the superprojects git dir.
So renaming/moving "old style with no name<->path mapping" will not
work. (That may be an acceptable trade off. But then again, just providing
the mapping, such that the superproject can absorb the git directory
of the submodule for this use case, doesn't seem like a big deal to me.
Something you want to have anyway, for ease of use of the superproject
w.r.t. cloning for example)

So while I do not try to deliberately break these old behaviors, I'd rather
want us to go forward with a saner model than "if we happen to have
enough data around, the operation succeeds", i.e. ignore anything
that is not following the rather strict definition of a submodule.

FYI: Once upon a time I found "fake submodules"
http://debuggable.com/posts/git-fake-submodules:4b563ee4-f3cc-4061-967e-0e48cbdd56cb
Last time this was discussed on list, this was considered a bug not
worth fixing instead of a feature IIRC. (Personally I think this is
a rather cool hack, which we may want to abuse ourselves for
things like "convert a subtree into a submodule and back again",
but we could also go without this hack)

> Apart from the tests nothing has been added in this iteration. Since I
> finally have a working test now I will continue with reviving the
> fallback to paths.

I'll have a look.

Cheers,
Stefan

^ permalink raw reply	[relevance 24%]

* Re: [PATCH] submodule: avoid sentence-lego in translated string
  2017-10-09  4:14 ` Martin Ågren
@ 2017-10-09 16:47   ` Stefan Beller
  2017-10-09 21:11     ` Jean-Noël AVILA
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-10-09 16:47 UTC (permalink / raw)
  To: Martin Ågren; +Cc: Junio C Hamano, brian m. carlson, Philip Oakley, Git Mailing List, Jiang Xin, Alexander Shopov, Jordi Mas, Ralf Thielow, Christopher Díaz, Jean-Noël Avila, Marco Paolone, Changwoo Ryu, Vasco Almeida, Dimitriy Ryazantcev, Peter Krefting, Trần Ngọc Quân, Jacques Viviers, m4sk1n

On Sun, Oct 8, 2017 at 9:14 PM, Martin Ågren <martin.agren@gmail.com> wrote:
> In this particular case, we could have three specific messages plus one default
> message (which at the time shouldn't ever occur). Or we have one specific
> message for the "tag"-case, which seems to have been Stefan's original
> motivation, and a generic message for the other cases.

I think the original patch is an improvement for the translations,
but I'd really want to deliver as much information to the user,
(it's an error message after all, so tell as much as possible that can
be helpful,)
which is why I would prefer to keep `typename(type)` in there.

I always assumed that translators are aware of these issues and sort of
work around this somehow, maybe like this:

  "submodule entry '%s' (%s) is not a commit. It is of type %s"

Thanks,
Stefan

^ permalink raw reply	[relevance 22%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-06 22:32 ` [RFC PATCH 2/4] change submodule push test to use proper repository setup Heiko Voigt
@ 2017-10-09 18:20   ` Stefan Beller
  2017-10-10 13:03     ` Heiko Voigt
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-10-09 18:20 UTC (permalink / raw)
  To: Heiko Voigt; +Cc: git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Fri, Oct 6, 2017 at 3:32 PM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> NOTE: The argument in this message is not correct, see description in
> cover letter.
>
> The setup of the repositories in this test is using gitlinks without the
> .gitmodules infrastructure. It is however testing convenience features
> like --recurse-submodules=on-demand. These features are already not
> supported by fetch without a .gitmodules file. This leads us to the
> conclusion that it is not really used here as well.
>
> Let's use the usual submodule commands to setup the repository in a
> typical way. This also has the advantage that we are testing with a
> repository structure that is more similar to one we could expect on a
> users setup.
>
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
>
> As mentioned in the cover letter. This seems to be the only test that
> ensures that we stay compatible with setups without .gitmodules. Maybe
> we should add/revive some?

An interesting discussion covering this topic is found at
https://public-inbox.org/git/20170606035650.oykbz2uc4xkr3cr2@sigill.intra.peff.net/

^ permalink raw reply	[relevance 16%]

* Re: [PATCH] submodule: avoid sentence-lego in translated string
  2017-10-09 16:47   ` Stefan Beller
@ 2017-10-09 21:11     ` Jean-Noël AVILA
  2017-10-10  1:26       ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Jean-Noël AVILA @ 2017-10-09 21:11 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Martin Ågren, Junio C Hamano, brian m. carlson, Philip Oakley, Git Mailing List, Jiang Xin, Alexander Shopov, Jordi Mas, Ralf Thielow, Christopher Díaz, Marco Paolone, Changwoo Ryu, Vasco Almeida, Dimitriy Ryazantcev, Peter Krefting, Trần Ngọc Quân, Jacques Viviers, m4sk1n

On Monday, 9 October 2017, 09:47:26 CEST Stefan Beller wrote:

> I always assumed that translators are aware of these issues and sort of
> work around this somehow, maybe like this:
> 
>   "submodule entry '%s' (%s) is not a commit. It is of type %s"

Translators can be aware of the issue if the coder commented the 
internationalization string with some possible candidates for the placeholders 
when it is not clear unless you check in the source code. Much effort was 
poured into translating the technical terms in other parts of Git; it seems 
awkward to just step back in this occurence.


^ permalink raw reply	[relevance 15%]

* Re: Branch switching with submodules where the submodule replaces a folder aborts unexpectedly
  2017-10-09 21:29 Branch switching with submodules where the submodule replaces a folder aborts unexpectedly Thomas Braun
@ 2017-10-09 21:59 ` Stefan Beller
  2017-10-12 11:48   ` Thomas Braun
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2017-10-09 21:59 UTC (permalink / raw)
  To: Thomas Braun; +Cc: git

On Mon, Oct 9, 2017 at 2:29 PM, Thomas Braun
<thomas.braun@virtuell-zuhause.de> wrote:
> Hi,
>
> I'm currently in the progress of pulling some subprojects in a git repository of mine into their own repositories and adding these subprojects back as submodules.
>
> While doing this I enountered a potential bug as checkout complains on branch switching that a file already exists.

(And I presume you know about --recurse-submodules as a flag for git-checkout)

This is consistent with our tests, unfortunately.

git/t$ ./t2013-checkout-submodule.sh
...
not ok 15 - git checkout --recurse-submodules: replace submodule with
a directory # TODO known breakage
...

> If I'm misusing git here I'm glad for any advice.

You are not.

Apart from this bug report, would you think that such filtering of
trees into submodules (and back again) might be an interesting feature
of Git or are these cases rare and special?

Thanks,
Stefan

^ permalink raw reply	[relevance 28%]

* [PATCH] submodule: spell out API of submodule_move_head
@ 2017-10-09 22:06 Stefan Beller
  2017-10-09 22:22 ` Jonathan Nieder
  2017-10-10  0:16 ` Junio C Hamano
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2017-10-09 22:06 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller, Jonathan Nieder

This way users of this function do not need to read the implementation to
know what it will do.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.c |  5 -----
 submodule.h | 18 ++++++++++++++++++
 2 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/submodule.c b/submodule.c
index c1edc91a13..d12e4ea96c 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1507,11 +1507,6 @@ static void submodule_reset_index(const char *path)
 		die("could not reset submodule index");
 }
 
-/**
- * Moves a submodule at a given path from a given head to another new head.
- * For edge cases (a submodule coming into existence or removing a submodule)
- * pass NULL for old or new respectively.
- */
 int submodule_move_head(const char *path,
 			 const char *old,
 			 const char *new,
diff --git a/submodule.h b/submodule.h
index f0da0277a4..3c15977be9 100644
--- a/submodule.h
+++ b/submodule.h
@@ -111,6 +111,24 @@ extern void connect_work_tree_and_git_dir(const char *work_tree, const char *git
  */
 int submodule_to_gitdir(struct strbuf *buf, const char *submodule);
 
+/**
+ * Move the HEAD and content of the active submodule at 'path' from object id
+ * 'old' to 'new'.
+ *
+ * Updates the submodule at 'path' and files in its work tree to commit
+ * 'new'. The commit previously pointed to by the submodule is named by
+ * 'old'. This updates the submodule's HEAD, index, and work tree but
+ * does not touch its gitlink entry in the superproject.
+ *
+ * If the submodule did not previously exist, then 'old' should be NULL.
+ * Similarly, if the submodule is to be removed, 'new' should be NULL.
+ *
+ * If updating the submodule would cause local changes to be overwritten,
+ * then instead of updating the submodule, this function prints an error
+ * message and returns -1.
+ *
+ * If the submodule is not active, this does nothing and returns 0.
+ */
 #define SUBMODULE_MOVE_HEAD_DRY_RUN (1<<0)
 #define SUBMODULE_MOVE_HEAD_FORCE   (1<<1)
 extern int submodule_move_head(const char *path,
-- 
2.14.0.rc0.3.g6c2e499285


^ permalink raw reply	[relevance 30%]

* Re: [PATCH] submodule: spell out API of submodule_move_head
  2017-10-09 22:06 [PATCH] submodule: spell out API of submodule_move_head Stefan Beller
@ 2017-10-09 22:22 ` Jonathan Nieder
  2017-10-10  0:16 ` Junio C Hamano
  1 sibling, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-10-09 22:22 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Stefan Beller wrote:

> This way users of this function do not need to read the implementation to
> know what it will do.
>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  submodule.c |  5 -----
>  submodule.h | 18 ++++++++++++++++++
>  2 files changed, 18 insertions(+), 5 deletions(-)

Looks good (well, you'd expect me to think that :)).  Thanks.

^ permalink raw reply	[relevance 15%]

* Re: [PATCH] submodule: spell out API of submodule_move_head
  2017-10-09 22:06 [PATCH] submodule: spell out API of submodule_move_head Stefan Beller
  2017-10-09 22:22 ` Jonathan Nieder
@ 2017-10-10  0:16 ` Junio C Hamano
  2017-10-10  0:27   ` Jonathan Nieder
  1 sibling, 1 reply; 200+ results
From: Junio C Hamano @ 2017-10-10  0:16 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Jonathan Nieder

Stefan Beller <sbeller@google.com> writes:

> +/**
> + * Move the HEAD and content of the active submodule at 'path' from object id
> + * 'old' to 'new'.
> + *
> + * Updates the submodule at 'path' and files in its work tree to commit
> + * 'new'. The commit previously pointed to by the submodule is named by
> + * 'old'. This updates the submodule's HEAD, index, and work tree but
> + * does not touch its gitlink entry in the superproject.
> + *
> + * If the submodule did not previously exist, then 'old' should be NULL.
> + * Similarly, if the submodule is to be removed, 'new' should be NULL.
> + *
> + * If updating the submodule would cause local changes to be overwritten,
> + * then instead of updating the submodule, this function prints an error
> + * message and returns -1.

This is not a new issue (the removed comment did not mention this at
all), but is it correct to say that updates to "index and work tree"
was as if we did "git -C $path checkout new" (and of course, HEAD in
the $path submodule must be at 'old')?

What should happen if 'old' does not match reality (i.e. old is NULL
but HEAD does point at some commit, old and HEAD are different,
etc.)?  Should the call be aborted?

> + * If the submodule is not active, this does nothing and returns 0.
> + */
>  #define SUBMODULE_MOVE_HEAD_DRY_RUN (1<<0)
>  #define SUBMODULE_MOVE_HEAD_FORCE   (1<<1)
>  extern int submodule_move_head(const char *path,

^ permalink raw reply	[relevance 22%]

* Re: [PATCH] submodule: spell out API of submodule_move_head
  2017-10-10  0:16 ` Junio C Hamano
@ 2017-10-10  0:27   ` Jonathan Nieder
  0 siblings, 0 replies; 200+ results
From: Jonathan Nieder @ 2017-10-10  0:27 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Stefan Beller, git

Junio C Hamano wrote:
> Stefan Beller <sbeller@google.com> writes:

>> +/**
>> + * Move the HEAD and content of the active submodule at 'path' from object id
>> + * 'old' to 'new'.
>> + *
>> + * Updates the submodule at 'path' and files in its work tree to commit
>> + * 'new'. The commit previously pointed to by the submodule is named by
>> + * 'old'. This updates the submodule's HEAD, index, and work tree but
>> + * does not touch its gitlink entry in the superproject.
>> + *
>> + * If the submodule did not previously exist, then 'old' should be NULL.
>> + * Similarly, if the submodule is to be removed, 'new' should be NULL.
>> + *
>> + * If updating the submodule would cause local changes to be overwritten,
>> + * then instead of updating the submodule, this function prints an error
>> + * message and returns -1.
>
> This is not a new issue (the removed comment did not mention this at
> all), but is it correct to say that updates to "index and work tree"
> was as if we did "git -C $path checkout new" (and of course, HEAD in
> the $path submodule must be at 'old')?

I don't understand the question.  This comment doesn't say it's like
"git checkout" --- are you saying it should?

The function is more like "git read-tree -m -u" (or --reset when
SUBMODULE_MOVE_HEAD_FORCE is passed) than like "git checkout".
Perhaps what you are hinting at is that read-tree --recurse-submodules
is not necessarily the right primitive to implement "git checkout"
with.  But that's a separate issue from documenting the current
behavior of the function.

> What should happen if 'old' does not match reality (i.e. old is NULL
> but HEAD does point at some commit, old and HEAD are different,
> etc.)?  Should the call be aborted?

No.

Thanks,
Jonathan

>> + * If the submodule is not active, this does nothing and returns 0.
>> + */
>>  #define SUBMODULE_MOVE_HEAD_DRY_RUN (1<<0)
>>  #define SUBMODULE_MOVE_HEAD_FORCE   (1<<1)
>>  extern int submodule_move_head(const char *path,

^ permalink raw reply	[relevance 14%]

* Re: [PATCH] submodule: avoid sentence-lego in translated string
  2017-10-09 21:11     ` Jean-Noël AVILA
@ 2017-10-10  1:26       ` Junio C Hamano
  2017-10-10  2:35         ` Changwoo Ryu
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2017-10-10  1:26 UTC (permalink / raw)
  To: Jean-Noël AVILA; +Cc: Stefan Beller, Martin Ågren, brian m. carlson, Philip Oakley, Git Mailing List, Jiang Xin, Alexander Shopov, Jordi Mas, Ralf Thielow, Christopher Díaz, Marco Paolone, Changwoo Ryu, Vasco Almeida, Dimitriy Ryazantcev, Peter Krefting, Trần Ngọc Quân, Jacques Viviers, m4sk1n

Jean-Noël AVILA <jn.avila@free.fr> writes:

> On Monday, 9 October 2017, 09:47:26 CEST Stefan Beller wrote:
>
>> I always assumed that translators are aware of these issues and sort of
>> work around this somehow, maybe like this:
>> 
>>   "submodule entry '%s' (%s) is not a commit. It is of type %s"
>
> Translators can be aware of the issue if the coder commented the 
> internationalization string with some possible candidates for the placeholders 
> when it is not clear unless you check in the source code. Much effort was 
> poured into translating the technical terms in other parts of Git; it seems 
> awkward to just step back in this occurence.

I do not see this particular case as "stepping back", though.

Our users do not spell "git cat-file -t commit v2.0^{commit}" with
'commit' translated to their language, right?  Shouldn't an error
message output use the same phrase the input side requests users to
use?

^ permalink raw reply	[relevance 7%]

* Re: [PATCH] submodule: avoid sentence-lego in translated string
  2017-10-10  1:26       ` Junio C Hamano
@ 2017-10-10  2:35         ` Changwoo Ryu
  2017-10-10 18:14           ` Martin Ågren
  0 siblings, 1 reply; 200+ results
From: Changwoo Ryu @ 2017-10-10  2:35 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Jean-Noël AVILA, Stefan Beller, Martin Ågren, brian m. carlson, Philip Oakley, Git Mailing List, Jiang Xin, Alexander Shopov, Jordi Mas, Ralf Thielow, Christopher Díaz, Marco Paolone, Vasco Almeida, Dimitriy Ryazantcev, Peter Krefting, Trần Ngọc Quân, Jacques Viviers, m4sk1n

2017-10-10 10:26 GMT+09:00 Junio C Hamano <gitster@pobox.com>:
> Jean-Noël AVILA <jn.avila@free.fr> writes:
>
>> On Monday, 9 October 2017, 09:47:26 CEST Stefan Beller wrote:
>>
>>> I always assumed that translators are aware of these issues and sort of
>>> work around this somehow, maybe like this:
>>>
>>>   "submodule entry '%s' (%s) is not a commit. It is of type %s"
>>
>> Translators can be aware of the issue if the coder commented the
>> internationalization string with some possible candidates for the placeholders
>> when it is not clear unless you check in the source code. Much effort was
>> poured into translating the technical terms in other parts of Git; it seems
>> awkward to just step back in this occurence.
>
> I do not see this particular case as "stepping back", though.
>
> Our users do not spell "git cat-file -t commit v2.0^{commit}" with
> 'commit' translated to their language, right?  Shouldn't an error
> message output use the same phrase the input side requests users to
> use?

Users know the limit of command-line translation. They type "commit"
to commit but they see translated "commit" in output messages. It is
actually confusing. But the untranslated English literals in the
middle of translated sentences does not remove the confusion but
increase it in a different way.

^ permalink raw reply	[relevance 7%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-09 18:20   ` Stefan Beller
@ 2017-10-10 13:03     ` Heiko Voigt
  2017-10-10 18:39       ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Heiko Voigt @ 2017-10-10 13:03 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Jonathan Nieder, Jens Lehmann, Brandon Williams

Hi,

On Mon, Oct 09, 2017 at 11:20:51AM -0700, Stefan Beller wrote:
> On Fri, Oct 6, 2017 at 3:32 PM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > NOTE: The argument in this message is not correct, see description in
> > cover letter.
> >
> > The setup of the repositories in this test is using gitlinks without the
> > .gitmodules infrastructure. It is however testing convenience features
> > like --recurse-submodules=on-demand. These features are already not
> > supported by fetch without a .gitmodules file. This leads us to the
> > conclusion that it is not really used here as well.
> >
> > Let's use the usual submodule commands to setup the repository in a
> > typical way. This also has the advantage that we are testing with a
> > repository structure that is more similar to one we could expect on a
> > users setup.
> >
> > Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> > ---
> >
> > As mentioned in the cover letter. This seems to be the only test that
> > ensures that we stay compatible with setups without .gitmodules. Maybe
> > we should add/revive some?
> 
> An interesting discussion covering this topic is found at
> https://public-inbox.org/git/20170606035650.oykbz2uc4xkr3cr2@sigill.intra.peff.net/

Thanks for that pointer. So in that discussion Junio said that the
recursive operations should succeed if we have everything necessary at
hand. I kind of agree because why should we limit usage when not
necessary. On the other hand we want git to be easy to use. And that
example from Peff is perfect as a demonstration of a incosistency we
currently have:

git clone git://some.where.git/submodule.git
git add submodule

is an operation I remember, I did, when first getting in contact with
submodules (many years back), since that is one intuitive way. And the
thing is: It works, kind of... Only later I discovered that one actually
needs to us a special submodule command to get everything approriately
setup to work together with others.

If everyone agrees that submodules are the default way of handling
repositories insided repositories, IMO, 'git add' should also alter
.gitmodules by default. We could provide a switch to avoid doing that.

An intermediate solution would be to warn but in the long run my goal
for submodules is and always was: Make them behave as close to files as
possible. And why should a 'git add submodule' not magically do
everything it can to make submodules just work? I can look into a patch
for that if people agree here...

Regarding handling of gitlinks with or without .gitmodules:

Currently we are actually in some intermediate state:

 * If there is no .gitmodules file: No submodule processing on any
   gitlinks (AFAIK)
 * If there is a .gitmodules files with some submodule configured: Do
   recursive fetch and push as far as possible on gitlinks.

So I am not sure whether there are actually many users (knowingly)
using a mix of some submodules configured and some not and then relying
on the submodule infrastructure.

I would rather expect two sorts of users:

  1. Those that do use .gitmodules

  2. Those that do *not* use .gitmodules

Users that do not use any .gitmodules file will currently (AFAIK) not
get any submodule handling. So the question is are there really many
"mixed users"? My guess would be no.
Because without those using this mixed we could switch to saying: "You
need to have a .gitmodules file for submodule handling" without much
fallout from breaking users use cases.

Maybe we can test this out somehow? My patch series would be ready in
that case, just had to drop the first patch and adjust the commit
message of this one.

Cheers Heiko

^ permalink raw reply	[relevance 23%]

* Re: [PATCH] submodule: avoid sentence-lego in translated string
  2017-10-10  2:35         ` Changwoo Ryu
@ 2017-10-10 18:14           ` Martin Ågren
  0 siblings, 0 replies; 200+ results
From: Martin Ågren @ 2017-10-10 18:14 UTC (permalink / raw)
  To: Changwoo Ryu; +Cc: Junio C Hamano, Jean-Noël AVILA, Stefan Beller, brian m. carlson, Philip Oakley, Git Mailing List, Jiang Xin, Alexander Shopov, Jordi Mas, Ralf Thielow, Christopher Díaz, Marco Paolone, Vasco Almeida, Dimitriy Ryazantcev, Peter Krefting, Trần Ngọc Quân, Jacques Viviers, m4sk1n

On 10 October 2017 at 04:35, Changwoo Ryu <cwryu@debian.org> wrote:
> 2017-10-10 10:26 GMT+09:00 Junio C Hamano <gitster@pobox.com>:
>> Jean-Noël AVILA <jn.avila@free.fr> writes:
>>
>>> On Monday, 9 October 2017, 09:47:26 CEST Stefan Beller wrote:
>>>
>>>> I always assumed that translators are aware of these issues and sort of
>>>> work around this somehow, maybe like this:
>>>>
>>>>   "submodule entry '%s' (%s) is not a commit. It is of type %s"
>>>
>>> Translators can be aware of the issue if the coder commented the
>>> internationalization string with some possible candidates for the placeholders
>>> when it is not clear unless you check in the source code. Much effort was
>>> poured into translating the technical terms in other parts of Git; it seems
>>> awkward to just step back in this occurence.
>>
>> I do not see this particular case as "stepping back", though.
>>
>> Our users do not spell "git cat-file -t commit v2.0^{commit}" with
>> 'commit' translated to their language, right?  Shouldn't an error
>> message output use the same phrase the input side requests users to
>> use?

I thought Jean-Noël meant at least partially the translator-experience,
not just the user-experience, but I might be wrong.

I prepared a patch to give a TRANSLATORS:-comment, but then I realized
that we have more instances like this with `typename()`. Actually, quite
often we avoid the issue (intentionally or unintentionally) by writing
"of type %s", but other times, we do the "is a %s". So I don't know,
maybe it all works anyway. The regular translators have now received 10
mails (11 counting this) so might be aware of this particular string by
now. :-/

> Users know the limit of command-line translation. They type "commit"
> to commit but they see translated "commit" in output messages. It is
> actually confusing. But the untranslated English literals in the
> middle of translated sentences does not remove the confusion but
> increase it in a different way.

What you describe seems plausible, but I have to admit that I don't use
i18n-ized software myself.

Martin

^ permalink raw reply	[relevance 6%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-10 13:03     ` Heiko Voigt
@ 2017-10-10 18:39       ` Stefan Beller
  2017-10-10 23:31         ` Junio C Hamano
                           ` (2 more replies)
  0 siblings, 3 replies; 200+ results
From: Stefan Beller @ 2017-10-10 18:39 UTC (permalink / raw)
  To: Heiko Voigt, Josh Triplett; +Cc: git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Tue, Oct 10, 2017 at 6:03 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:

>> > As mentioned in the cover letter. This seems to be the only test that
>> > ensures that we stay compatible with setups without .gitmodules. Maybe
>> > we should add/revive some?
>>
>> An interesting discussion covering this topic is found at
>> https://public-inbox.org/git/20170606035650.oykbz2uc4xkr3cr2@sigill.intra.peff.net/
>
> Thanks for that pointer. So in that discussion Junio said that the
> recursive operations should succeed if we have everything necessary at
> hand. I kind of agree because why should we limit usage when not
> necessary. On the other hand we want git to be easy to use. And that
> example from Peff is perfect as a demonstration of a incosistency we
> currently have:
>
> git clone git://some.where.git/submodule.git
> git add submodule
>
> is an operation I remember, I did, when first getting in contact with
> submodules (many years back), since that is one intuitive way. And the
> thing is: It works, kind of... Only later I discovered that one actually
> needs to us a special submodule command to get everything approriately
> setup to work together with others.

I agree that we ought to not block off users "because we can", but rather
perform the operation if possible with the data at hand.

Note that the result of the discussion `jk/warn-add-gitlink actually`
warns about adding a raw gitlink now, such that we hint at using
"git submodule add", directly.

So you propose to make git-add behave like "git submodule add"
(i.e. also add the .gitmodules entry for name/path/URL), which I
like from a submodule perspective.

However other users of gitlinks might be confused[1], which is why
I refrained from "making every gitlink into a submodule". Specifically
the more powerful a submodule operation is (the more fluff adds),
the harder it should be for people to mis-use it.

[1] https://github.com/git-series/git-series/blob/master/INTERNALS.md
     "git-series uses gitlinks to store pointer to commits in its own repo."

> If everyone agrees that submodules are the default way of handling
> repositories insided repositories, IMO, 'git add' should also alter
> .gitmodules by default. We could provide a switch to avoid doing that.

I wonder if that switch should be default-on (i.e. not treat a gitlink as
a submodule initially, behavior as-is, and then eventually we will
die() on unconfigured repos, expecting the user to make the decision)

> An intermediate solution would be to warn

That is already implemented by Peff.

> but in the long run my goal
> for submodules is and always was: Make them behave as close to files as
> possible. And why should a 'git add submodule' not magically do
> everything it can to make submodules just work? I can look into a patch
> for that if people agree here...

I'd love to see this implemented. I cc'd Josh (the author of git-series), who
may disagree with this, or has some good input how to go forward without
breaking git-series.

> Regarding handling of gitlinks with or without .gitmodules:
>
> Currently we are actually in some intermediate state:
>
>  * If there is no .gitmodules file: No submodule processing on any
>    gitlinks (AFAIK)

AFAIK this is true.

>  * If there is a .gitmodules files with some submodule configured: Do
>    recursive fetch and push as far as possible on gitlinks.

* If submodule.recurse is set, then we also treat submodules like files
  for checkout, reset, read-tree.

> So I am not sure whether there are actually many users (knowingly)
> using a mix of some submodules configured and some not and then relying
> on the submodule infrastructure.
>
> I would rather expect two sorts of users:
>
>   1. Those that do use .gitmodules

Those want to reap all benefits of good submodules.

>
>   2. Those that do *not* use .gitmodules

As said above, we don't know if those users are
"holding submodules wrong" or are using gitlinks for
magic tricks (unrelated to submodules).

>
> Users that do not use any .gitmodules file will currently (AFAIK) not
> get any submodule handling. So the question is are there really many
> "mixed users"? My guess would be no.

I hope that there are few (if any) users of these mixed setups.

> Because without those using this mixed we could switch to saying: "You
> need to have a .gitmodules file for submodule handling" without much
> fallout from breaking users use cases.

That seems reasonable to me, actually.

> Maybe we can test this out somehow? My patch series would be ready in
> that case, just had to drop the first patch and adjust the commit
> message of this one.

I wonder how we would test this, though? Do you have any idea
(even vague) how we'd accomplish such a measurement?
I fear we'll have to go this way blindly.

Cheers,
Stefan

>
> Cheers Heiko

^ permalink raw reply	[relevance 24%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-10 18:39       ` Stefan Beller
@ 2017-10-10 23:31         ` Junio C Hamano
  2017-10-10 23:41           ` Stefan Beller
  2017-10-11 14:56           ` Heiko Voigt
  2017-10-11 14:52         ` Heiko Voigt
  2017-10-11 15:10         ` Josh Triplett
  2 siblings, 2 replies; 200+ results
From: Junio C Hamano @ 2017-10-10 23:31 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Heiko Voigt, Josh Triplett, git\, Jonathan Nieder, Jens Lehmann, Brandon Williams

Stefan Beller <sbeller@google.com> writes:

> So you propose to make git-add behave like "git submodule add"
> (i.e. also add the .gitmodules entry for name/path/URL), which I
> like from a submodule perspective.
>
> However other users of gitlinks might be confused[1], which is why
> I refrained from "making every gitlink into a submodule". Specifically
> the more powerful a submodule operation is (the more fluff adds),
> the harder it should be for people to mis-use it.

A few questions that come to mind are:

 - Does "git add sub/" have enough information to populate
   .gitmodules?  If we have reasonable "default" values for
   .gitmodules entries (e.g. missing URL means we won't fetch when
   asked to go recursively fetch), perhaps we can leave everything
   other than "submodule.$name.path" undefined.

 - Can't we help those who have gitlinks without .gitmodules entries
   exactly the same way as above, i.e. when we see a gitlink and try
   to treat it as a submodule, we'd first try to look it up from
   .gitmodules (by going from path to name and then to
   submodule.$name.$var); the above "'git add sub/' would add an
   entry for .gitmodules" wish is based on the assumption that there
   are reasonable "default" values for each of these $var--so by
   basing on the same assumption, we can "pretend" as if these
   submodule.$name.$var were in .gitmodules file when we see
   gitlinks without .gitmodules entries.  IOW, if "git add sub/" can
   add .gitmodules to help people without having to type "git
   submodule add sub/", then we can give exactly the same degree of
   help without even modifying .gitmodules when "git add sub/" is
   run.

 - Even if we could solve it with "git add sub/" that adds to
   .gitmodules, is it a good solution, when we can solve the same
   thing without having to do so?




^ permalink raw reply	[relevance 24%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-10 23:31         ` Junio C Hamano
@ 2017-10-10 23:41           ` Stefan Beller
  2017-10-11 14:56           ` Heiko Voigt
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2017-10-10 23:41 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Heiko Voigt, Josh Triplett, git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Tue, Oct 10, 2017 at 4:31 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> So you propose to make git-add behave like "git submodule add"
>> (i.e. also add the .gitmodules entry for name/path/URL), which I
>> like from a submodule perspective.
>>
>> However other users of gitlinks might be confused[1], which is why
>> I refrained from "making every gitlink into a submodule". Specifically
>> the more powerful a submodule operation is (the more fluff adds),
>> the harder it should be for people to mis-use it.
>
> A few questions that come to mind are:
>
>  - Does "git add sub/" have enough information to populate
>    .gitmodules?  If we have reasonable "default" values for
>    .gitmodules entries (e.g. missing URL means we won't fetch when
>    asked to go recursively fetch), perhaps we can leave everything
>    other than "submodule.$name.path" undefined.

I think we would want to populate path and URL only.

>
>  - Can't we help those who have gitlinks without .gitmodules entries
>    exactly the same way as above, i.e. when we see a gitlink and try
>    to treat it as a submodule, we'd first try to look it up from
>    .gitmodules (by going from path to name and then to
>    submodule.$name.$var); the above "'git add sub/' would add an
>    entry for .gitmodules" wish is based on the assumption that there
>    are reasonable "default" values for each of these $var--so by
>    basing on the same assumption, we can "pretend" as if these
>    submodule.$name.$var were in .gitmodules file when we see
>    gitlinks without .gitmodules entries.  IOW, if "git add sub/" can
>    add .gitmodules to help people without having to type "git
>    submodule add sub/", then we can give exactly the same degree of
>    help without even modifying .gitmodules when "git add sub/" is
>    run.

I do not understand the gist of this paragraph, other then:

  "When git-add <repository> encounters a section submodule.<name>.*,
   do not modify it; We can assume it is sane already."

>  - Even if we could solve it with "git add sub/" that adds to
>    .gitmodules, is it a good solution, when we can solve the same
>    thing without having to do so?

I am confused even more.

So you suggest that "git add [--gitlink=submodule]" taking on the
responsibilities of "git submodule add" is a bad idea?

I thought we had the same transition from "git remote update" to
"git fetch", which eventually superseded the former.

^ permalink raw reply	[relevance 25%]

* [ANNOUNCE] Git v2.15.0-rc1
@ 2017-10-11  6:35 Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2017-10-11  6:35 UTC (permalink / raw)
  To: git; +Cc: Linux Kernel

A release candidate Git v2.15.0-rc1 is now available for testing
at the usual places.  It is comprised of 721 non-merge commits
since v2.14.0, contributed by 72 people, 22 of which are new faces.

The tarballs are found at:

    https://www.kernel.org/pub/software/scm/git/testing/

The following public repositories all have a copy of the
'v2.15.0-rc1' tag and the 'master' branch that the tag points at:

  url = https://kernel.googlesource.com/pub/scm/git/git
  url = git://repo.or.cz/alt-git.git
  url = https://github.com/gitster/git

New contributors whose contributions weren't in v2.14.0 are as follows.
Welcome to the Git development community!

  Ann T Ropea, Daniel Watkins, Derrick Stolee, Dimitrios
  Christidis, Eric Rannaud, Evan Zacks, Hielke Christian Braun,
  Ian Campbell, Ilya Kantor, Jameson Miller, Job Snijders, Joel
  Teichroeb, joernchen, Łukasz Gryglicki, Manav Rathi, Martin
  Ågren, Michael Forney, Patryk Obara, Randall S. Becker, Ross
  Kabus, Taylor Blau, and Urs Thuermann.

Returning contributors who helped this release are as follows.
Thanks for your continued support.

  Adam Dinwoodie, Ævar Arnfjörð Bjarmason, Andreas Heiduk,
  Anthony Sottile, Ben Boeckel, Brandon Casey, Brandon Williams,
  brian m. carlson, Christian Couder, Eric Blake, Han-Wen Nienhuys,
  Heiko Voigt, Jean-Noel Avila, Jeff Hostetler, Jeff King, Johannes
  Schindelin, Johannes Sixt, Jonathan Nieder, Jonathan Tan,
  Junio C Hamano, Kaartic Sivaraam, Kevin Daudt, Kevin Willford,
  Lars Schneider, Martin Koegler, Matthieu Moy, Max Kirillov,
  Michael Haggerty, Michael J Gruber, Nguyễn Thái Ngọc Duy,
  Nicolas Morey-Chaisemartin, Øystein Walle, Paolo Bonzini,
  Pat Thoyts, Philip Oakley, Phillip Wood, Raman Gupta, Ramsay
  Jones, René Scharfe, Sahil Dua, Santiago Torres, Stefan Beller,
  Stephan Beyer, Takashi Iwai, Thomas Braun, Thomas Gummerer,
  Todd Zullinger, Tom G. Christensen, Torsten Bögershausen,
  and William Duclot.

----------------------------------------------------------------

Git 2.15 Release Notes (draft)
==============================

Backward compatibility notes and other notable changes.

 * Use of an empty string as a pathspec element that is used for
   'everything matches' is still warned and Git asks users to use a
   more explicit '.' for that instead.  The hope is that existing
   users will not mind this change, and eventually the warning can be
   turned into a hard error, upgrading the deprecation into removal of
   this (mis)feature.  That is now scheduled to happen in Git v2.16,
   the next major release after this one.

 * Git now avoids blindly falling back to ".git" when the setup
   sequence said we are _not_ in Git repository.  A corner case that
   happens to work right now may be broken by a call to BUG().
   We've tried hard to locate such cases and fixed them, but there
   might still be cases that need to be addressed--bug reports are
   greatly appreciated.

 * "branch --set-upstream" that has been deprecated in Git 1.8 has
   finally been retired.


Updates since v2.14
-------------------

UI, Workflows & Features

 * An example that is now obsolete has been removed from a sample hook,
   and an old example in it that added a sign-off manually has been
   improved to use the interpret-trailers command.

 * The advice message given when "git rebase" stops for conflicting
   changes has been improved.

 * The "rerere-train" script (in contrib/) learned the "--overwrite"
   option to allow overwriting existing recorded resolutions.

 * "git contacts" (in contrib/) now lists the address on the
   "Reported-by:" trailer to its output, in addition to those on
   S-o-b: and other trailers, to make it easier to notify (and thank)
   the original bug reporter.

 * "git rebase", especially when it is run by mistake and ends up
   trying to replay many changes, spent long time in silence.  The
   command has been taught to show progress report when it spends
   long time preparing these many changes to replay (which would give
   the user a chance to abort with ^C).

 * "git merge" learned a "--signoff" option to add the Signed-off-by:
   trailer with the committer's name.

 * "git diff" learned to optionally paint new lines that are the same
   as deleted lines elsewhere differently from genuinely new lines.

 * "git interpret-trailers" learned to take the trailer specifications
   from the command line that overrides the configured values.

 * "git interpret-trailers" has been taught a "--parse" and a few
   other options to make it easier for scripts to grab existing
   trailer lines from a commit log message.

 * The "--format=%(trailers)" option "git log" and its friends take
   learned to take the 'unfold' and 'only' modifiers to normalize its
   output, e.g. "git log --format=%(trailers:only,unfold)".

 * "gitweb" shows a link to visit the 'raw' contents of blbos in the
   history overview page.

 * "[gc] rerereResolved = 5.days" used to be invalid, as the variable
   is defined to take an integer counting the number of days.  It now
   is allowed.

 * The code to acquire a lock on a reference (e.g. while accepting a
   push from a client) used to immediately fail when the reference is
   already locked---now it waits for a very short while and retries,
   which can make it succeed if the lock holder was holding it during
   a read-only operation.

 * "branch --set-upstream" that has been deprecated in Git 1.8 has
   finally been retired.

 * The codepath to call external process filter for smudge/clean
   operation learned to show the progress meter.

 * "git rev-parse" learned "--is-shallow-repository", that is to be
   used in a way similar to existing "--is-bare-repository" and
   friends.

 * "git describe --match <pattern>" has been taught to play well with
   the "--all" option.

 * "git branch" learned "-c/-C" to create a new branch by copying an
   existing one.

 * Some commands (most notably "git status") makes an opportunistic
   update when performing a read-only operation to help optimize later
   operations in the same repository.  The new "--no-optional-locks"
   option can be passed to Git to disable them.

 * "git for-each-ref --format=..." learned a new format element,
   %(trailers), to show only the commit log trailer part of the log
   message.


Performance, Internal Implementation, Development Support etc.

 * Conversion from uchar[20] to struct object_id continues.

 * Start using selected c99 constructs in small, stable and
   essentialpart of the system to catch people who care about
   older compilers that do not grok them.

 * The filter-process interface learned to allow a process with long
   latency give a "delayed" response.

 * Many uses of comparision callback function the hashmap API uses
   cast the callback function type when registering it to
   hashmap_init(), which defeats the compile time type checking when
   the callback interface changes (e.g. gaining more parameters).
   The callback implementations have been updated to take "void *"
   pointers and cast them to the type they expect instead.

 * Because recent Git for Windows do come with a real msgfmt, the
   build procedure for git-gui has been updated to use it instead of a
   hand-rolled substitute.

 * "git grep --recurse-submodules" has been reworked to give a more
   consistent output across submodule boundary (and do its thing
   without having to fork a separate process).

 * A helper function to read a single whole line into strbuf
   mistakenly triggered OOM error at EOF under certain conditions,
   which has been fixed.
   (merge 642956cf45 rs/strbuf-getwholeline-fix later to maint).

 * The "ref-store" code reorganization continues.

 * "git commit" used to discard the index and re-read from the filesystem
   just in case the pre-commit hook has updated it in the middle; this
   has been optimized out when we know we do not run the pre-commit hook.
   (merge 680ee550d7 kw/commit-keep-index-when-pre-commit-is-not-run later to maint).

 * Updates to the HTTP layer we made recently unconditionally used
   features of libCurl without checking the existence of them, causing
   compilation errors, which has been fixed.  Also migrate the code to
   check feature macros, not version numbers, to cope better with
   libCurl that vendor ships with backported features.

 * The API to start showing progress meter after a short delay has
   been simplified.
   (merge 8aade107dd jc/simplify-progress later to maint).

 * Code clean-up to avoid mixing values read from the .gitmodules file
   and values read from the .git/config file.

 * We used to spend more than necessary cycles allocating and freeing
   piece of memory while writing each index entry out.  This has been
   optimized.

 * Platforms that ship with a separate sha1 with collision detection
   library can link to it instead of using the copy we ship as part of
   our source tree.

 * Code around "notes" have been cleaned up.
   (merge 3964281524 mh/notes-cleanup later to maint).

 * The long-standing rule that an in-core lockfile instance, once it
   is used, must not be freed, has been lifted and the lockfile and
   tempfile APIs have been updated to reduce the chance of programming
   errors.

 * Our hashmap implementation in hashmap.[ch] is not thread-safe when
   adding a new item needs to expand the hashtable by rehashing; add
   an API to disable the automatic rehashing to work it around.

 * Many of our programs consider that it is OK to release dynamic
   storage that is used throughout the life of the program by simply
   exiting, but this makes it harder to leak detection tools to avoid
   reporting false positives.  Plug many existing leaks and introduce
   a mechanism for developers to mark that the region of memory
   pointed by a pointer is not lost/leaking to help these tools.

 * As "git commit" to conclude a conflicted "git merge" honors the
   commit-msg hook, "git merge" that records a merge commit that
   cleanly auto-merges should, but it didn't.

 * The codepath for "git merge-recursive" has been cleaned up.

 * Many leaks of strbuf have been fixed.

 * "git imap-send" has our own implementation of the protocol and also
   can use more recent libCurl with the imap protocol support.  Update
   the latter so that it can use the credential subsystem, and then
   make it the default option to use, so that we can eventually
   deprecate and remove the former.

 * "make style" runs git-clang-format to help developers by pointing
   out coding style issues.

 * A test to demonstrate "git mv" failing to adjust nested submodules
   has been added.
   (merge c514167df2 hv/mv-nested-submodules-test later to maint).

 * On Cygwin, "ulimit -s" does not report failure but it does not work
   at all, which causes an unexpected success of some tests that
   expect failures under a limited stack situation.  This has been
   fixed.

 * Many codepaths have been updated to squelch -Wimplicit-fallthrough
   warnings from Gcc 7 (which is a good code hygiene).

 * Add a helper for DLL loading in anticipation for its need in a
   future topic RSN.

 * "git status --ignored", when noticing that a directory without any
   tracked path is ignored, still enumerated all the ignored paths in
   the directory, which is unnecessary.  The codepath has been
   optimized to avoid this overhead.

 * The final batch to "git rebase -i" updates to move more code from
   the shell script to C has been merged.

 * Operations that do not touch (majority of) packed refs have been
   optimized by making accesses to packed-refs file lazy; we no longer
   pre-parse everything, and an access to a single ref in the
   packed-refs does not touch majority of irrelevant refs, either.

 * Add comment to clarify that the style file is meant to be used with
   clang-5 and the rules are still work in progress.

 * Many variables that points at a region of memory that will live
   throughout the life of the program have been marked with UNLEAK
   marker to help the leak checkers concentrate on real leaks..

 * Plans for weaning us off of SHA-1 has been documented.

 * A new "oidmap" API has been introduced and oidset API has been
   rewritten to use it.


Also contains various documentation updates and code clean-ups.


Fixes since v2.14
-----------------

 * "%C(color name)" in the pretty print format always produced ANSI
   color escape codes, which was an early design mistake.  They now
   honor the configuration (e.g. "color.ui = never") and also tty-ness
   of the output medium.

 * The http.{sslkey,sslCert} configuration variables are to be
   interpreted as a pathname that honors "~[username]/" prefix, but
   weren't, which has been fixed.

 * Numerous bugs in walking of reflogs via "log -g" and friends have
   been fixed.

 * "git commit" when seeing an totally empty message said "you did not
   edit the message", which is clearly wrong.  The message has been
   corrected.

 * When a directory is not readable, "gitweb" fails to build the
   project list.  Work this around by skipping such a directory.

 * Some versions of GnuPG fails to kill gpg-agent it auto-spawned
   and such a left-over agent can interfere with a test.  Work it
   around by attempting to kill one before starting a new test.

 * A recently added test for the "credential-cache" helper revealed
   that EOF detection done around the time the connection to the cache
   daemon is torn down were flaky.  This was fixed by reacting to
   ECONNRESET and behaving as if we got an EOF.

 * "git log --tag=no-such-tag" showed log starting from HEAD, which
   has been fixed---it now shows nothing.

 * The "tag.pager" configuration variable was useless for those who
   actually create tag objects, as it interfered with the use of an
   editor.  A new mechanism has been introduced for commands to enable
   pager depending on what operation is being carried out to fix this,
   and then "git tag -l" is made to run pager by default.

 * "git push --recurse-submodules $there HEAD:$target" was not
   propagated down to the submodules, but now it is.

 * Commands like "git rebase" accepted the --rerere-autoupdate option
   from the command line, but did not always use it.  This has been
   fixed.

 * "git clone --recurse-submodules --quiet" did not pass the quiet
   option down to submodules.

 * Test portability fix for OBSD.

 * Portability fix for OBSD.

 * "git am -s" has been taught that some input may end with a trailer
   block that is not Signed-off-by: and it should refrain from adding
   an extra blank line before adding a new sign-off in such a case.

 * "git svn" used with "--localtime" option did not compute the tz
   offset for the timestamp in question and instead always used the
   current time, which has been corrected.

 * Memory leak in an error codepath has been plugged.

 * "git stash -u" used the contents of the committed version of the
   ".gitignore" file to decide which paths are ignored, even when the
   file has local changes.  The command has been taught to instead use
   the locally modified contents.

 * bash 4.4 or newer gave a warning on NUL byte in command
   substitution done in "git stash"; this has been squelched.

 * "git grep -L" and "git grep --quiet -L" reported different exit
   codes; this has been corrected.

 * When handshake with a subprocess filter notices that the process
   asked for an unknown capability, Git did not report what program
   the offending subprocess was running.  This has been corrected.

 * "git apply" that is used as a better "patch -p1" failed to apply a
   taken from a file with CRLF line endings to a file with CRLF line
   endings.  The root cause was because it misused convert_to_git()
   that tried to do "safe-crlf" processing by looking at the index
   entry at the same path, which is a nonsense---in that mode, "apply"
   is not working on the data in (or derived from) the index at all.
   This has been fixed.

 * Killing "git merge --edit" before the editor returns control left
   the repository in a state with MERGE_MSG but without MERGE_HEAD,
   which incorrectly tells the subsequent "git commit" that there was
   a squash merge in progress.  This has been fixed.

 * "git archive" did not work well with pathspecs and the
   export-ignore attribute.

 * In addition to "cc: <a@dd.re.ss> # cruft", "cc: a@dd.re.ss # cruft"
   was taught to "git send-email" as a valid way to tell it that it
   needs to also send a carbon copy to <a@dd.re.ss> in the trailer
   section.
   (merge cc90750677 mm/send-email-cc-cruft later to maint).

 * "git branch -M a b" while on a branch that is completely unrelated
   to either branch a or branch b misbehaved when multiple worktree
   was in use.  This has been fixed.
   (merge 31824d180d nd/worktree-kill-parse-ref later to maint).

 * "git gc" and friends when multiple worktrees are used off of a
   single repository did not consider the index and per-worktree refs
   of other worktrees as the root for reachability traversal, making
   objects that are in use only in other worktrees to be subject to
   garbage collection.

 * A regression to "gitk --bisect" by a recent update has been fixed.
   (merge 1d0538e486 mh/packed-ref-store-prep later to maint).

 * "git -c submodule.recurse=yes pull" did not work as if the
   "--recurse-submodules" option was given from the command line.
   This has been corrected.

 * Unlike "git commit-tree < file", "git commit-tree -F file" did not
   pass the contents of the file verbatim and instead completed an
   incomplete line at the end, if exists.  The latter has been updated
   to match the behaviour of the former.
   (merge c818e74332 rk/commit-tree-make-F-verbatim later to maint).

 * Many codepaths did not diagnose write failures correctly when disks
   go full, due to their misuse of write_in_full() helper function,
   which have been corrected.
   (merge f48ecd38cb jk/write-in-full-fix later to maint).

 * "git help co" now says "co is aliased to ...", not "git co is".
   (merge b3a8076e0d ks/help-alias-label later to maint).

 * "git archive", especially when used with pathspec, stored an empty
   directory in its output, even though Git itself never does so.
   This has been fixed.
   (merge 4318094047 rs/archive-excluded-directory later to maint).

 * API error-proofing which happens to also squelch warnings from GCC.
   (merge c788c54cde tg/refs-allowed-flags later to maint).

 * The explanation of the cut-line in the commit log editor has been
   slightly tweaked.
   (merge 8c4b1a3593 ks/commit-do-not-touch-cut-line later to maint).

 * "git gc" tries to avoid running two instances at the same time by
   reading and writing pid/host from and to a lock file; it used to
   use an incorrect fscanf() format when reading, which has been
   corrected.
   (merge afe2fab72c aw/gc-lockfile-fscanf-fix later to maint).

 * The scripts to drive TravisCI has been reorganized and then an
   optimization to avoid spending cycles on a branch whose tip is
   tagged has been implemented.
   (merge 8376eb4a8f ls/travis-scriptify later to maint).

 * The test linter has been taught that we do not like "echo -e".
   (merge 1a6d46895d tb/test-lint-echo-e later to maint).

 * Code cmp.std.c nitpick.
   (merge ac7da78ede mh/for-each-string-list-item-empty-fix later to maint).

 * A regression fix for 2.11 that made the code to read the list of
   alternate object stores overrun the end of the string.
   (merge f0f7bebef7 jk/info-alternates-fix later to maint).

 * "git describe --match" learned to take multiple patterns in v2.13
   series, but the feature ignored the patterns after the first one
   and did not work at all.  This has been fixed.
   (merge da769d2986 jk/describe-omit-some-refs later to maint).

 * "git filter-branch" cannot reproduce a history with a tag without
   the tagger field, which only ancient versions of Git allowed to be
   created.  This has been corrected.
   (merge b2c1ca6b4b ic/fix-filter-branch-to-handle-tag-without-tagger later to maint).

 * "git cat-file --textconv" started segfaulting recently, which
   has been corrected.
   (merge cc0ea7c9e5 jk/diff-blob later to maint).

 * The built-in pattern to detect the "function header" for HTML did
   not match <H1>..<H6> elements without any attributes, which has
   been fixed.
   (merge 9c03caca2c ik/userdiff-html-h-element-fix later to maint).

 * "git mailinfo" was loose in decoding quoted printable and produced
   garbage when the two letters after the equal sign are not
   hexadecimal.  This has been fixed.
   (merge c8cf423eab rs/mailinfo-qp-decode-fix later to maint).

 * The machinery to create xdelta used in pack files received the
   sizes of the data in size_t, but lost the higher bits of them by
   storing them in "unsigned int" during the computation, which is
   fixed.

 * The delta format used in the packfile cannot reference data at
   offset larger than what can be expressed in 4-byte, but the
   generator for the data failed to make sure the offset does not
   overflow.  This has been corrected.

 * The documentation for '-X<option>' for merges was misleadingly
   written to suggest that "-s theirs" exists, which is not the case.
   (merge c25d98b2a7 jc/merge-x-theirs-docfix later to maint).

 * "git fast-export" with -M/-C option issued "copy" instruction on a
   path that is simultaneously modified, which was incorrect.
   (merge b3e8ca89cf jt/fast-export-copy-modify-fix later to maint).

 * Many codepaths have been updated to squelch -Wsign-compare
   warnings.
   (merge 071bcaab64 rj/no-sign-compare later to maint).

 * Memory leaks in various codepaths have been plugged.
   (merge 4d01a7fa65 ma/leakplugs later to maint).

 * Recent versions of "git rev-parse --parseopt" did not parse the
   option specification that does not have the optional flags (*=?!)
   correctly, which has been corrected.
   (merge a6304fa4c2 bc/rev-parse-parseopt-fix later to maint).

 * The checkpoint command "git fast-import" did not flush updates to
   refs and marks unless at least one object was created since the
   last checkpoint, which has been corrected, as these things can
   happen without any new object getting created.
   (merge 30e215a65c er/fast-import-dump-refs-on-checkpoint later to maint).

 * Spell the name of our system as "Git" in the output from
   request-pull script.
   (merge e66d7c37a5 ar/request-pull-phrasofix later to maint).

 * Fixes for a handful memory access issues identified by valgrind.
   (merge 2944a94c6b tg/memfixes later to maint).

 * Backports a moral equivalent of 2015 fix to the poll() emulation
   from the upstream gnulib to fix occasional breakages on HPE NonStop.
   (merge 61b2a1acaa rb/compat-poll-fix later to maint).

 * Users with "color.ui = always" in their configuration were broken
   by a recent change that made plumbing commands to pay attention to
   them as the patch created internally by "git add -p" were colored
   (heh) and made unusable.  Fix this regression by redefining
   'always' to mean the same thing as 'auto'.
   (merge 6be4595edb jk/ui-color-always-to-auto-maint later to maint).

 * In the "--format=..." option of the "git for-each-ref" command (and
   its friends, i.e. the listing mode of "git branch/tag"), "%(atom:)"
   (e.g. "%(refname:)", "%(body:)" used to error out.  Instead, treat
   them as if the colon and an empty string that follows it were not
   there.
   (merge bea4dbeafd tb/ref-filter-empty-modifier later to maint).

 * An ancient bug that made Git misbehave with creation/renaming of
   refs has been fixed.

 * Other minor doc, test and build updates and code cleanups.
   (merge f094b89a4d ma/parse-maybe-bool later to maint).
   (merge 39b00fa4d4 jk/drop-sha1-entry-pos later to maint).
   (merge 6cdf8a7929 ma/ts-cleanups later to maint).
   (merge 7560f547e6 ma/up-to-date later to maint).
   (merge 0db3dc75f3 rs/apply-epoch later to maint).
   (merge 74f1bd912b dw/diff-highlight-makefile-fix later to maint).
   (merge f991761eb8 jk/config-lockfile-leak-fix later to maint).
   (merge 150efef1e7 ma/pkt-line-leakfix later to maint).
   (merge 5554451de6 mg/timestamp-t-fix later to maint).
   (merge 276d0e35c0 ma/split-symref-update-fix later to maint).
   (merge 3bc4b8f7c7 bb/doc-eol-dirty later to maint).
   (merge c1bb33c99c jk/system-path-cleanup later to maint).
   (merge ab46e6fc72 cc/subprocess-handshake-missing-capabilities later to maint).
   (merge f7a32dd97f kd/doc-for-each-ref later to maint).
   (merge be94568bc7 ez/doc-duplicated-words-fix later to maint).
   (merge 01e4be6c3d ks/test-readme-phrasofix later to maint).
   (merge 217bb56d4f hn/typofix later to maint).
   (merge c08fd6388c jk/doc-read-tree-table-asciidoctor-fix later to maint).
   (merge c3342b362e ks/doc-use-camelcase-for-config-name later to maint).
   (merge 0bca165fdb jk/validate-headref-fix later to maint).
   (merge 93dbefb389 mr/doc-negative-pathspec later to maint).
   (merge 5e633326e4 ad/doc-markup-fix later to maint).
   (merge 9ca356fa8b rs/cocci-de-paren-call-params later to maint).
   (merge 7099153e8d rs/tag-null-pointer-arith-fix later to maint).
   (merge 0e187d758c rs/run-command-use-alloc-array later to maint).
   (merge e0222159fa jn/strbuf-doc-re-reuse later to maint).
   (merge 97487ea11a rs/qsort-s later to maint).
   (merge a9155c50bd sb/branch-avoid-repeated-strbuf-release later to maint).
   (merge f777623514 ks/branch-tweak-error-message-for-extra-args later to maint).
   (merge 33f3c683ec ks/verify-filename-non-option-error-message-tweak later to maint).
   (merge b3ea7dd32d jk/sha1-loose-object-info-fix later to maint).
   (merge 2720f6db5d rs/fsck-null-return-from-lookup later to maint).
   (merge 99b7b687a6 rs/rs-mailmap later to maint).
   (merge 7823655082 tb/complete-describe later to maint).
   (merge 7cbbf9d6a2 ls/filter-process-delayed later to maint).

----------------------------------------------------------------

Changes since v2.14.0 are as follows:

Adam Dinwoodie (1):
      doc: correct command formatting

Andreas Heiduk (2):
      doc: add missing values "none" and "default" for diff.wsErrorHighlight
      doc: clarify "config --bool" behaviour with empty string

Ann T Ropea (1):
      request-pull: capitalise "Git" to make it a proper noun

Anthony Sottile (1):
      git-grep: correct exit code with --quiet and -L

Ben Boeckel (1):
      Documentation: mention that `eol` can change the dirty status of paths

Brandon Casey (7):
      t1502: demonstrate rev-parse --parseopt option mis-parsing
      rev-parse parseopt: do not search help text for flag chars
      rev-parse parseopt: interpret any whitespace as start of help text
      git-rebase: don't ignore unexpected command line arguments
      t0040,t1502: Demonstrate parse_options bugs
      parse-options: write blank line to correct output stream
      parse-options: only insert newline in help text if needed

Brandon Williams (29):
      repo_read_index: don't discard the index
      repository: have the_repository use the_index
      submodule--helper: teach push-check to handle HEAD
      cache.h: add GITMODULES_FILE macro
      config: add config_from_gitmodules
      submodule: remove submodule.fetchjobs from submodule-config parsing
      submodule: remove fetch.recursesubmodules from submodule-config parsing
      submodule: check for unstaged .gitmodules outside of config parsing
      submodule: check for unmerged .gitmodules outside of config parsing
      submodule: merge repo_read_gitmodules and gitmodules_config
      grep: recurse in-process using 'struct repository'
      t7411: check configuration parsing errors
      submodule: don't use submodule_from_name
      add, reset: ensure submodules can be added or reset
      submodule--helper: don't overlay config in remote_submodule_branch
      submodule--helper: don't overlay config in update-clone
      fetch: don't overlay config with submodule-config
      submodule: don't rely on overlayed config when setting diffopts
      unpack-trees: don't respect submodule.update
      submodule: remove submodule_config callback routine
      diff: stop allowing diff to have submodules configured in .git/config
      submodule-config: remove support for overlaying repository config
      submodule-config: move submodule-config functions to submodule-config.c
      submodule-config: lazy-load a repository's .gitmodules file
      unpack-trees: improve loading of .gitmodules
      submodule: remove gitmodules_config
      clone: teach recursive clones to respect -q
      clang-format: outline the git project's coding style
      Makefile: add style build rule

Christian Couder (3):
      refs: use skip_prefix() in ref_is_hidden()
      sub-process: print the cmd when a capability is unsupported
      sha1-lookup: remove sha1_entry_pos() from header file

Daniel Watkins (1):
      diff-highlight: add clean target to Makefile

Derrick Stolee (1):
      cleanup: fix possible overflow errors in binary search

Dimitrios Christidis (1):
      fmt-merge-msg: fix coding style

Eric Blake (1):
      git-contacts: also recognise "Reported-by:"

Eric Rannaud (1):
      fast-import: checkpoint: dump branches/tags/marks even if object_count==0

Evan Zacks (1):
      doc: fix minor typos (extra/duplicated words)

Han-Wen Nienhuys (5):
      submodule.h: typofix
      submodule.c: describe submodule_to_gitdir() in a new comment
      real_path: clarify return value ownership
      read_gitfile_gently: clarify return value ownership.
      string-list.h: move documentation from Documentation/api/ into header

Heiko Voigt (2):
      t5526: fix some broken && chains
      add test for bug in git-mv for recursive submodules

Hielke Christian Braun (1):
      gitweb: skip unreadable subdirectories

Ian Campbell (4):
      filter-branch: reset $GIT_* before cleaning up
      filter-branch: preserve and restore $GIT_AUTHOR_* and $GIT_COMMITTER_*
      filter-branch: stash away ref map in a branch
      filter-branch: use hash-object instead of mktag

Ilya Kantor (1):
      userdiff: fix HTML hunk header regexp

Jameson Miller (1):
      Improve performance of git status --ignored

Jean-Noel Avila (1):
      i18n: add a missing space in message

Jeff Hostetler (1):
      hashmap: add API to disable item counting when threaded

Jeff King (129):
      t1414: document some reflog-walk oddities
      revision: disallow reflog walking with revs->limited
      log: clarify comment about reflog cycles
      log: do not free parents when walking reflog
      get_revision_1(): replace do-while with an early return
      rev-list: check reflog_info before showing usage
      reflog-walk: stop using fake parents
      reflog-walk: apply --since/--until to reflog dates
      check return value of verify_ref_format()
      docs/for-each-ref: update pointer to color syntax
      t: use test_decode_color rather than literal ANSI codes
      ref-filter: simplify automatic color reset
      ref-filter: abstract ref format into its own struct
      ref-filter: move need_color_reset_at_eol into ref_format
      ref-filter: provide a function for parsing sort options
      ref-filter: make parse_ref_filter_atom a private function
      ref-filter: factor out the parsing of sorting atoms
      ref-filter: pass ref_format struct to atom parsers
      color: check color.ui in git_default_config()
      for-each-ref: load config earlier
      rev-list: pass diffopt->use_colors through to pretty-print
      pretty: respect color settings for %C placeholders
      ref-filter: consult want_color() before emitting colors
      strbuf: use designated initializers in STRBUF_INIT
      t/lib-proto-disable: restore protocol.allow after config tests
      t5813: add test for hostname starting with dash
      connect: factor out "looks like command line option" check
      connect: reject dashed arguments for proxy commands
      connect: reject paths that look like command line options
      t6018: flesh out empty input/output rev-list tests
      revision: add rev_input_given flag
      rev-list: don't show usage when we see empty ref patterns
      revision: do not fallback to default when rev_input_given is set
      hashcmp: use memcmp instead of open-coded loop
      sha1_file: drop experimental GIT_USE_LOOKUP search
      trailer: put process_trailers() options into a struct
      interpret-trailers: add an option to show only the trailers
      interpret-trailers: add an option to show only existing trailers
      interpret-trailers: add an option to unfold values
      interpret-trailers: add --parse convenience option
      pretty: move trailer formatting to trailer.c
      t4205: refactor %(trailers) tests
      pretty: support normalization options for %(trailers)
      doc: fix typo in sendemail.identity
      config: use a static lock_file struct
      write_index_as_tree: cleanup tempfile on error
      setup_temporary_shallow: avoid using inactive tempfile
      setup_temporary_shallow: move tempfile struct into function
      verify_signed_buffer: prefer close_tempfile() to close()
      always check return value of close_tempfile
      tempfile: do not delete tempfile on failed close
      lockfile: do not rollback lock on failed close
      tempfile: prefer is_tempfile_active to bare access
      tempfile: handle NULL tempfile pointers gracefully
      tempfile: replace die("BUG") with BUG()
      tempfile: factor out activation
      tempfile: factor out deactivation
      tempfile: robustify cleanup handler
      tempfile: release deactivated strbufs instead of resetting
      tempfile: use list.h for linked list
      tempfile: remove deactivated list entries
      tempfile: auto-allocate tempfiles on heap
      lockfile: update lifetime requirements in documentation
      ref_lock: stop leaking lock_files
      stop leaking lock structs in some simple cases
      test-lib: --valgrind should not override --verbose-log
      test-lib: set LSAN_OPTIONS to abort by default
      add: free leaked pathspec after add_files_to_cache()
      update-index: fix cache entry leak in add_one_file()
      config: plug user_config leak
      reset: make tree counting less confusing
      reset: free allocated tree buffers
      repository: free fields before overwriting them
      set_git_dir: handle feeding gitdir to itself
      rev-parse: don't trim bisect refnames
      system_path: move RUNTIME_PREFIX to a sub-function
      git_extract_argv0_path: do nothing without RUNTIME_PREFIX
      add UNLEAK annotation for reducing leak false positives
      shortlog: skip format/parse roundtrip for internal traversal
      shell: drop git-cvsserver support by default
      archimport: use safe_pipe_capture for user input
      cvsimport: shell-quote variable used in backticks
      config: avoid "write_in_full(fd, buf, len) < len" pattern
      get-tar-commit-id: check write_in_full() return against 0
      avoid "write_in_full(fd, buf, len) != len" pattern
      convert less-trivial versions of "write_in_full() != len"
      pkt-line: check write_in_full() errors against "< 0"
      notes-merge: use ssize_t for write_in_full() return value
      config: flip return value of store_write_*()
      read_pack_header: handle signed/unsigned comparison in read result
      prefix_ref_iterator: break when we leave the prefix
      read_info_alternates: read contents into strbuf
      read_info_alternates: warn on non-trivial errors
      revision: replace "struct cmdline_pathspec" with argv_array
      cat-file: handle NULL object_context.path
      test-line-buffer: simplify command parsing
      curl_trace(): eliminate switch fallthrough
      consistently use "fallthrough" comments in switches
      doc: put literal block delimiter around table
      files-backend: prefer "0" for write_in_full() error check
      notes-merge: drop dead zero-write code
      prefer "!=" when checking read_in_full() result
      avoid looking at errno for short read_in_full() returns
      distinguish error versus short read from read_in_full()
      worktree: use xsize_t to access file size
      worktree: check the result of read_in_full()
      validate_headref: NUL-terminate HEAD buffer
      validate_headref: use skip_prefix for symref parsing
      validate_headref: use get_oid_hex for detached HEADs
      git: add --no-optional-locks option
      test-terminal: set TERM=vt100
      t4015: prefer --color to -c color.diff=always
      t3701: use test-terminal to collect color output
      t7508: use test_terminal for color output
      t7502: use diff.noprefix for --verbose test
      t6006: drop "always" color config tests
      t3203: drop "always" color test
      t3205: use --color instead of color.branch=always
      provide --color option for all ref-filter users
      color: make "always" the same as "auto" in config
      t4015: use --color with --color-moved
      t7301: use test_terminal to check color
      path.c: fix uninitialized memory access
      sha1_loose_object_info: handle errors from unpack_sha1_rest
      t3308: create a real ref directory/file conflict
      refs_resolve_ref_unsafe: handle d/f conflicts for writes
      write_entry: fix leak when retrying delayed filter
      write_entry: avoid reading blobs in CE_RETRY case
      write_entry: untangle symlink and regular-file cases

Job Snijders (1):
      gitweb: add 'raw' blob_plain link in history overview

Joel Teichroeb (3):
      stash: add a test for stash create with no files
      stash: add a test for when apply fails during stash branch
      stash: add a test for stashing in a detached state

Johannes Schindelin (14):
      run_processes_parallel: change confusing task_cb convention
      git-gui (MinGW): make use of MSys2's msgfmt
      t3415: verify that an empty instructionFormat is handled as before
      rebase -i: generate the script via rebase--helper
      rebase -i: remove useless indentation
      rebase -i: do not invent onelines when expanding/collapsing SHA-1s
      rebase -i: also expand/collapse the SHA-1s via the rebase--helper
      t3404: relax rebase.missingCommitsCheck tests
      rebase -i: check for missing commits in the rebase--helper
      rebase -i: skip unnecessary picks using the rebase--helper
      t3415: test fixup with wrapped oneline
      rebase -i: rearrange fixup/squash lines using the rebase--helper
      Win32: simplify loading of DLL functions
      clang-format: adjust line break penalties

Johannes Sixt (1):
      sub-process: use child_process.args instead of child_process.argv

Jonathan Nieder (8):
      vcs-svn: remove more unused prototypes and declarations
      vcs-svn: remove custom mode constants
      vcs-svn: remove repo_delete wrapper function
      vcs-svn: move remaining repo_tree functions to fast_export.h
      pack: make packed_git_mru global a value instead of a pointer
      pathspec doc: parse_pathspec does not maintain references to args
      technical doc: add a design doc for hash function transition
      strbuf doc: reuse after strbuf_release is fine

Jonathan Tan (40):
      fsck: remove redundant parse_tree() invocation
      object: remove "used" field from struct object
      fsck: cleanup unused variable
      Documentation: migrate sub-process docs to header
      sub-process: refactor handshake to common function
      tests: ensure fsck fails on corrupt packfiles
      sha1_file: set whence in storage-specific info fn
      sha1_file: remove read_packed_sha1()
      diff: avoid redundantly clearing a flag
      diff: respect MIN_BLOCK_LENGTH for last block
      diff: define block by number of alphanumeric chars
      Doc: clarify that pack-objects makes packs, plural
      pack: move pack name-related functions
      pack: move static state variables
      pack: move pack_report()
      pack: move open_pack_index(), parse_pack_index()
      pack: move release_pack_memory()
      pack: move pack-closing functions
      pack: move use_pack()
      pack: move unuse_pack()
      pack: move add_packed_git()
      pack: move install_packed_git()
      pack: move {,re}prepare_packed_git and approximate_object_count
      pack: move unpack_object_header_buffer()
      pack: move get_size_from_delta()
      pack: move unpack_object_header()
      pack: move clear_delta_base_cache(), packed_object_info(), unpack_entry()
      pack: move nth_packed_object_{sha1,oid}
      pack: move check_pack_index_ptr(), nth_packed_object_offset()
      pack: move find_pack_entry_one(), is_pack_valid()
      pack: move find_sha1_pack()
      pack: move find_pack_entry() and make it global
      pack: move has_sha1_pack()
      pack: move has_pack_index()
      pack: move for_each_packed_object()
      Remove inadvertently added outgoing/packfile.h
      Add t/helper/test-write-cache to .gitignore
      git-compat-util: make UNLEAK less error-prone
      fast-export: do not copy from modified file
      oidmap: map with OID as key

Junio C Hamano (51):
      t1408: add a test of stale packed refs covered by loose refs
      clean.c: use designated initializer
      http.c: http.sslcert and http.sslkey are both pathnames
      connect: reject ssh hostname that begins with a dash
      Git 2.7.6
      Git 2.8.6
      Git 2.9.5
      Git 2.10.4
      Git 2.11.3
      Git 2.12.4
      Git 2.13.5
      Git 2.14.1
      Start post 2.14 cycle
      perl/Git.pm: typofix in a comment
      The first batch of topics after the 2.14 cycle
      diff: retire sane_truncate_fn
      progress: simplify "delayed" progress API
      The second batch post 2.14
      t4200: give us a clean slate after "rerere gc" tests
      t4200: make "rerere gc" test more robust
      t4200: gather "rerere gc" together
      t4200: parameterize "rerere gc" custom expiry test
      rerere: represent time duration in timestamp_t internally
      rerere: allow approxidate in gc.rerereResolved/gc.rerereUnresolved
      The third batch post 2.14
      Prepare for 2.14.2
      The fourth batch post 2.14
      The fifth batch post 2.14
      The sixth batch post 2.14
      RelNotes: further fixes for 2.14.2 from the master front
      The seventh batch post 2.14
      travis: dedent a few scripts that are indented overly deeply
      subprocess: loudly die when subprocess asks for an unsupported capability
      cvsserver: move safe_pipe_capture() to the main package
      cvsserver: use safe_pipe_capture for `constant commands` as well
      gc: call fscanf() with %<len>s, not %<len>c, when reading hostname
      The eighth batch for 2.15
      Git 2.10.5
      Git 2.11.4
      Git 2.12.5
      Git 2.13.6
      Git 2.14.2
      branch: fix "copy" to never touch HEAD
      merge-strategies: avoid implying that "-s theirs" exists
      The ninth batch for 2.15
      The tenth batch for 2.15
      The eleventh batch for 2.15
      The twelfth batch for 2.15
      Git 2.15-rc0
      Prepare for -rc1
      Git 2.15-rc1

Kaartic Sivaraam (15):
      hook: cleanup script
      hook: name the positional variables
      hook: add sign-off using "interpret-trailers"
      hook: add a simple first example
      commit: check for empty message before the check for untouched template
      hook: use correct logical variable
      t3200: cleanup cruft of a test
      builtin/branch: stop supporting the "--set-upstream" option
      branch: quote branch/ref names to improve readability
      help: change a message to be more precise
      commit-template: change a message to be more intuitive
      t/README: fix typo and grammatically improve a sentence
      doc: camelCase the config variables to improve readability
      branch: change the error messages to be more meaningful
      setup: update error message to be more meaningful

Kevin Daudt (3):
      stash: prevent warning about null bytes in input
      doc/for-each-ref: consistently use '=' to between argument names and values
      doc/for-each-ref: explicitly specify option names

Kevin Willford (9):
      format-patch: have progress option while generating patches
      rebase: turn on progress option by default for format-patch
      commit: skip discarding the index if there is no pre-commit hook
      perf: add test for writing the index
      read-cache: fix memory leak in do_write_index
      read-cache: avoid allocating every ondisk entry when writing
      merge-recursive: fix memory leak
      merge-recursive: remove return value from get_files_dirs
      merge-recursive: change current file dir string_lists to hashmap

Lars Schneider (13):
      t0021: keep filter log files on comparison
      t0021: make debug log file name configurable
      t0021: write "OUT <size>" only on success
      convert: put the flags field before the flag itself for consistent style
      convert: move multiple file filter error handling to separate function
      convert: refactor capabilities negotiation
      convert: add "status=delayed" to filter process protocol
      convert: display progress for filtered objects that have been delayed
      travis-ci: move Travis CI code into dedicated scripts
      travis-ci: skip a branch build if equal tag is present
      travis-ci: fix "skip_branch_tip_with_tag()" string comparison
      entry.c: update cache entry only for existing files
      entry.c: check if file exists after checkout

Manav Rathi (1):
      docs: improve discoverability of exclude pathspec

Martin Koegler (2):
      diff-delta: fix encoding size that would not fit in "unsigned int"
      diff-delta: do not allow delta offset truncation

Martin Ågren (33):
      builtin.h: take over documentation from api-builtin.txt
      git.c: let builtins opt for handling `pager.foo` themselves
      git.c: provide setup_auto_pager()
      t7006: add tests for how git tag paginates
      tag: respect `pager.tag` in list-mode only
      tag: change default of `pager.tag` to "on"
      git.c: ignore pager.* when launching builtin as dashed external
      Doc/git-{push,send-pack}: correct --sign= to --signed=
      t5334: document that git push --signed=1 does not work
      config: introduce git_parse_maybe_bool_text
      config: make git_{config,parse}_maybe_bool equivalent
      treewide: deprecate git_config_maybe_bool, use git_parse_maybe_bool
      parse_decoration_style: drop unused argument `var`
      doc/interpret-trailers: fix "the this" typo
      convert: always initialize attr_action in convert_attrs
      pack-objects: take lock before accessing `remaining`
      strbuf_setlen: don't write to strbuf_slopbuf
      ThreadSanitizer: add suppressions
      Documentation/user-manual: update outdated example output
      treewide: correct several "up-to-date" to "up to date"
      pkt-line: re-'static'-ify buffer in packet_write_fmt_1()
      config: remove git_config_maybe_bool
      refs/files-backend: add longer-scoped copy of string to list
      refs/files-backend: fix memory leak in lock_ref_for_update
      refs/files-backend: correct return value in lock_ref_for_update
      refs/files-backend: add `refname`, not "HEAD", to list
      builtin/commit: fix memory leak in `prepare_index()`
      commit: fix memory leak in `reduce_heads()`
      leak_pending: use `object_array_clear()`, not `free()`
      object_array: use `object_array_clear()`, not `free()`
      object_array: add and use `object_array_pop()`
      pack-bitmap[-write]: use `object_array_clear()`, don't leak
      builtin/: add UNLEAKs

Matthieu Moy (2):
      send-email: fix garbage removal after address
      send-email: don't use Mail::Address, even if available

Max Kirillov (2):
      describe: fix matching to actually match all patterns
      describe: teach --match to handle branches and remotes

Michael Forney (1):
      scripts: use "git foo" not "git-foo"

Michael Haggerty (77):
      add_packed_ref(): teach function to overwrite existing refs
      packed_ref_store: new struct
      packed_ref_store: move `packed_refs_path` here
      packed_ref_store: move `packed_refs_lock` member here
      clear_packed_ref_cache(): take a `packed_ref_store *` parameter
      validate_packed_ref_cache(): take a `packed_ref_store *` parameter
      get_packed_ref_cache(): take a `packed_ref_store *` parameter
      get_packed_refs(): take a `packed_ref_store *` parameter
      add_packed_ref(): take a `packed_ref_store *` parameter
      lock_packed_refs(): take a `packed_ref_store *` parameter
      commit_packed_refs(): take a `packed_ref_store *` parameter
      rollback_packed_refs(): take a `packed_ref_store *` parameter
      get_packed_ref(): take a `packed_ref_store *` parameter
      repack_without_refs(): take a `packed_ref_store *` parameter
      packed_peel_ref(): new function, extracted from `files_peel_ref()`
      packed_ref_store: support iteration
      packed_read_raw_ref(): new function, replacing `resolve_packed_ref()`
      packed-backend: new module for handling packed references
      packed_ref_store: make class into a subclass of `ref_store`
      commit_packed_refs(): report errors rather than dying
      commit_packed_refs(): use a staging file separate from the lockfile
      packed_refs_lock(): function renamed from lock_packed_refs()
      packed_refs_lock(): report errors via a `struct strbuf *err`
      packed_refs_unlock(), packed_refs_is_locked(): new functions
      clear_packed_ref_cache(): don't protest if the lock is held
      commit_packed_refs(): remove call to `packed_refs_unlock()`
      repack_without_refs(): don't lock or unlock the packed refs
      t3210: add some tests of bogus packed-refs file contents
      read_packed_refs(): die if `packed-refs` contains bogus data
      packed_ref_store: handle a packed-refs file that is a symlink
      files-backend: cheapen refname_available check when locking refs
      refs: retry acquiring reference locks for 100ms
      notes: make GET_NIBBLE macro more robust
      load_subtree(): remove unnecessary conditional
      load_subtree(): reduce the scope of some local variables
      load_subtree(): fix incorrect comment
      load_subtree(): separate logic for internal vs. terminal entries
      load_subtree(): check earlier whether an internal node is a tree entry
      load_subtree(): only consider blobs to be potential notes
      get_oid_hex_segment(): return 0 on success
      load_subtree(): combine some common code
      get_oid_hex_segment(): don't pad the rest of `oid`
      hex_to_bytes(): simpler replacement for `get_oid_hex_segment()`
      load_subtree(): declare some variables to be `size_t`
      load_subtree(): check that `prefix_len` is in the expected range
      packed-backend: don't adjust the reference count on lock/unlock
      struct ref_transaction: add a place for backends to store data
      packed_ref_store: implement reference transactions
      packed_delete_refs(): implement method
      files_pack_refs(): use a reference transaction to write packed refs
      prune_refs(): also free the linked list
      files_initial_transaction_commit(): use a transaction for packed refs
      t1404: demonstrate two problems with reference transactions
      files_ref_store: use a transaction to update packed refs
      packed-backend: rip out some now-unused code
      files_transaction_finish(): delete reflogs before references
      ref_iterator: keep track of whether the iterator output is ordered
      packed_ref_cache: add a backlink to the associated `packed_ref_store`
      die_unterminated_line(), die_invalid_line(): new functions
      read_packed_refs(): use mmap to read the `packed-refs` file
      read_packed_refs(): only check for a header at the top of the file
      read_packed_refs(): make parsing of the header line more robust
      for_each_string_list_item: avoid undefined behavior for empty list
      read_packed_refs(): read references with minimal copying
      packed_ref_cache: remember the file-wide peeling state
      mmapped_ref_iterator: add iterator over a packed-refs file
      mmapped_ref_iterator_advance(): no peeled value for broken refs
      packed-backend.c: reorder some definitions
      packed_ref_cache: keep the `packed-refs` file mmapped if possible
      read_packed_refs(): ensure that references are ordered when read
      packed_ref_iterator_begin(): iterate using `mmapped_ref_iterator`
      packed_read_raw_ref(): read the reference from the mmapped buffer
      ref_store: implement `refs_peel_ref()` generically
      packed_ref_store: get rid of the `ref_cache` entirely
      ref_cache: remove support for storing peeled values
      mmapped_ref_iterator: inline into `packed_ref_iterator`
      packed-backend.c: rename a bunch of things and update comments

Michael J Gruber (11):
      Documentation: use proper wording for ref format strings
      Documentation/git-for-each-ref: clarify peeling of tags for --format
      Documentation/git-merge: explain --continue
      merge: clarify call chain
      merge: split write_merge_state in two
      merge: save merge state earlier
      name-rev: change ULONG_MAX to TIME_MAX
      t7004: move limited stack prereq to test-lib
      t6120: test name-rev --all and --stdin
      t6120: clean up state after breaking repo
      t6120: test describe and name-rev with deep repos

Nguyễn Thái Ngọc Duy (17):
      branch: fix branch renaming not updating HEADs correctly
      revision.h: new flag in struct rev_info wrt. worktree-related refs
      refs.c: use is_dir_sep() in resolve_gitlink_ref()
      revision.c: refactor add_index_objects_to_pending()
      revision.c: --indexed-objects add objects from all worktrees
      refs.c: refactor get_submodule_ref_store(), share common free block
      refs: move submodule slash stripping code to get_submodule_ref_store
      refs: add refs_head_ref()
      revision.c: use refs_for_each*() instead of for_each_*_submodule()
      refs.c: move for_each_remote_ref_submodule() to submodule.c
      refs: remove dead for_each_*_submodule()
      revision.c: --all adds HEAD from all worktrees
      files-backend: make reflog iterator go through per-worktree reflog
      revision.c: --reflog add HEAD reflog from all worktrees
      rev-list: expose and document --single-worktree
      refs.c: remove fallback-to-main-store code get_submodule_ref_store()
      refs.c: reindent get_submodule_ref_store()

Nicolas Morey-Chaisemartin (7):
      stash: clean untracked files before reset
      pull: fix cli and config option parsing order
      pull: honor submodule.recurse config option
      imap-send: return with error if curl failed
      imap-send: add wrapper to get server credentials if needed
      imap_send: setup_curl: retreive credentials if not set in config file
      imap-send: use curl by default when possible

Paolo Bonzini (4):
      trailers: export action enums and corresponding lookup functions
      trailers: introduce struct new_trailer_item
      interpret-trailers: add options for actions
      interpret-trailers: fix documentation typo

Patryk Obara (10):
      sha1_file: fix definition of null_sha1
      commit: replace the raw buffer with strbuf in read_graft_line
      commit: allocate array using object_id size
      commit: rewrite read_graft_line
      builtin/hash-object: convert to struct object_id
      read-cache: convert to struct object_id
      sha1_file: convert index_path to struct object_id
      sha1_file: convert index_fd to struct object_id
      sha1_file: convert hash_sha1_file_literally to struct object_id
      sha1_file: convert index_stream to struct object_id

Philip Oakley (4):
      git-gui: remove duplicate entries from .gitconfig's gui.recentrepo
      git gui: cope with duplicates in _get_recentrepo
      git gui: de-dup selected repo from recentrepo history
      git gui: allow for a long recentrepo list

Phillip Wood (7):
      am: remember --rerere-autoupdate setting
      rebase: honor --rerere-autoupdate
      rebase -i: honor --rerere-autoupdate
      t3504: use test_commit
      cherry-pick/revert: remember --rerere-autoupdate
      cherry-pick/revert: reject --rerere-autoupdate when continuing
      am: fix signoff when other trailers are present

Raman Gupta (1):
      contrib/rerere-train: optionally overwrite existing resolutions

Ramsay Jones (9):
      credential-cache: interpret an ECONNRESET as an EOF
      builtin/add: add detail to a 'cannot chmod' error message
      test-lib: don't use ulimit in test prerequisites on cygwin
      test-lib: use more compact expression in PIPE prerequisite
      t9010-*.sh: skip all tests if the PIPE prereq is missing
      git-compat-util.h: xsize_t() - avoid -Wsign-compare warnings
      commit-slab.h: avoid -Wsign-compare warnings
      cache.h: hex2chr() - avoid -Wsign-compare warnings
      ALLOC_GROW: avoid -Wsign-compare warnings

Randall S. Becker (1):
      poll.c: always set revents, even if to zero

René Scharfe (81):
      tree-diff: don't access hash of NULL object_id pointer
      notes: don't access hash of NULL object_id pointer
      receive-pack: don't access hash of NULL object_id pointer
      bswap: convert to unsigned before shifting in get_be32
      bswap: convert get_be16, get_be32 and put_be32 to inline functions
      add MOVE_ARRAY
      use MOVE_ARRAY
      apply: use COPY_ARRAY and MOVE_ARRAY in update_image()
      ls-files: don't try to prune an empty index
      dir: support platforms that require aligned reads
      pack-objects: remove unnecessary NULL check
      t0001: skip test with restrictive permissions if getpwd(3) respects them
      test-path-utils: handle const parameter of basename and dirname
      t3700: fix broken test under !POSIXPERM
      t4062: use less than 256 repetitions in regex
      sha1_file: avoid comparison if no packed hash matches the first byte
      apply: remove prefix_length member from apply_state
      merge: use skip_prefix()
      win32: plug memory leak on realloc() failure in syslog()
      strbuf: clear errno before calling getdelim(3)
      fsck: free buffers on error in fsck_obj()
      sha1_file: release delta_stack on error in unpack_entry()
      tree-walk: convert fill_tree_descriptor() to object_id
      t1002: stop using sum(1)
      t5001: add tests for export-ignore attributes and exclude pathspecs
      archive: factor out helper functions for handling attributes
      archive: don't queue excluded directories
      commit: remove unused inline function single_parent()
      apply: check date of potential epoch timestamps first
      apply: remove epoch date from regex
      am: release strbufs after use in detect_patch_format()
      am: release strbuf on error return in hg_patch_to_mail()
      am: release strbuf after use in safe_to_abort()
      check-ref-format: release strbuf after use in check_ref_format_branch()
      clean: release strbuf after use in remove_dirs()
      clone: release strbuf after use in remove_junk()
      commit: release strbuf on error return in commit_tree_extended()
      connect: release strbuf on error return in git_connect()
      convert: release strbuf on error return in filter_buffer_or_fd()
      diff: release strbuf after use in diff_summary()
      diff: release strbuf after use in show_rename_copy()
      diff: release strbuf after use in show_stats()
      help: release strbuf on error return in exec_man_konqueror()
      help: release strbuf on error return in exec_man_man()
      help: release strbuf on error return in exec_woman_emacs()
      mailinfo: release strbuf after use in handle_from()
      mailinfo: release strbuf on error return in handle_boundary()
      merge: release strbuf after use in save_state()
      merge: release strbuf after use in write_merge_heads()
      notes: release strbuf after use in notes_copy_from_stdin()
      refs: release strbuf on error return in write_pseudoref()
      remote: release strbuf after use in read_remote_branches()
      remote: release strbuf after use in migrate_file()
      remote: release strbuf after use in set_url()
      send-pack: release strbuf on error return in send_pack()
      sha1_file: release strbuf on error return in index_path()
      shortlog: release strbuf after use in insert_one_record()
      sequencer: release strbuf after use in save_head()
      transport-helper: release strbuf after use in process_connect_service()
      userdiff: release strbuf after use in userdiff_get_textconv()
      utf8: release strbuf on error return in strbuf_utf8_replace()
      vcs-svn: release strbuf after use in end_revision()
      wt-status: release strbuf after use in read_rebase_todolist()
      wt-status: release strbuf after use in wt_longstatus_print_tracking()
      archive: don't add empty directories to archives
      refs: make sha1 output parameter of refs_resolve_ref_unsafe() optional
      refs: pass NULL to refs_resolve_ref_unsafe() if hash is not needed
      refs: pass NULL to resolve_ref_unsafe() if hash is not needed
      mailinfo: don't decode invalid =XY quoted-printable sequences
      refs: pass NULL to refs_resolve_refdup() if hash is not needed
      refs: pass NULL to resolve_refdup() if hash is not needed
      coccinelle: remove parentheses that become unnecessary
      path: use strbuf_add_real_path()
      use strbuf_addstr() for adding strings to strbufs
      graph: use strbuf_addchars() to add spaces
      tag: avoid NULL pointer arithmetic
      repository: use FREE_AND_NULL
      run-command: use ALLOC_ARRAY
      test-stringlist: avoid buffer underrun when sorting nothing
      fsck: handle NULL return of lookup_blob() and lookup_tree()
      .mailmap: normalize name for René Scharfe

Ross Kabus (1):
      commit-tree: do not complete line in -F input

Sahil Dua (2):
      config: create a function to format section headers
      branch: add a --copy (-c) option to go with --move (-m)

Santiago Torres (1):
      t: lib-gpg: flush gpg agent on startup

Stefan Beller (51):
      diff.c: readability fix
      diff.c: move line ending check into emit_hunk_header
      diff.c: factor out diff_flush_patch_all_file_pairs
      diff.c: introduce emit_diff_symbol
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_CONTEXT_MARKER
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_CONTEXT_FRAGINFO
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_NO_LF_EOF
      diff.c: migrate emit_line_checked to use emit_diff_symbol
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_WORDS[_PORCELAIN]
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_CONTEXT_INCOMPLETE
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_FILEPAIR_{PLUS, MINUS}
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_HEADER
      diff.c: emit_diff_symbol learns about DIFF_SYMBOL_BINARY_FILES
      diff.c: emit_diff_symbol learns DIFF_SYMBOL_REWRITE_DIFF
      submodule.c: migrate diff output to use emit_diff_symbol
      diff.c: convert emit_binary_diff_body to use emit_diff_symbol
      diff.c: convert show_stats to use emit_diff_symbol
      diff.c: convert word diffing to use emit_diff_symbol
      diff.c: emit_diff_symbol learns about DIFF_SYMBOL_STAT_SEP
      diff.c: emit_diff_symbol learns about DIFF_SYMBOL_SUMMARY
      diff.c: buffer all output if asked to
      diff.c: color moved lines differently
      diff.c: color moved lines differently, plain mode
      diff.c: add dimming to moved line detection
      diff: document the new --color-moved setting
      attr.c: drop hashmap_cmp_fn cast
      builtin/difftool.c: drop hashmap_cmp_fn cast
      builtin/describe: drop hashmap_cmp_fn cast
      config.c: drop hashmap_cmp_fn cast
      convert/sub-process: drop cast to hashmap_cmp_fn
      patch-ids.c: drop hashmap_cmp_fn cast
      remote.c: drop hashmap_cmp_fn cast
      submodule-config.c: drop hashmap_cmp_fn cast
      name-hash.c: drop hashmap_cmp_fn cast
      t/helper/test-hashmap: use custom data instead of duplicate cmp functions
      commit: convert lookup_commit_graft to struct object_id
      tag: convert gpg_verify_tag to use struct object_id
      t8008: rely on rev-parse'd HEAD instead of sha1 value
      t1200: remove t1200-tutorial.sh
      sha1_file: make read_info_alternates static
      submodule.sh: remove unused variable
      builtin/merge: honor commit-msg hook for merges
      push, fetch: error out for submodule entries not pointing to commits
      replace-objects: evaluate replacement refs without using the object store
      Documentation/githooks: mention merge in commit-msg hook
      Documentation/config: clarify the meaning of submodule.<name>.update
      t7406: submodule.<name>.update command must not be run from .gitmodules
      diff: correct newline in summary for renamed files
      submodule: correct error message for missing commits
      branch: reset instead of release a strbuf
      tests: fix diff order arguments in test_cmp

Stephan Beyer (1):
      clang-format: add a comment about the meaning/status of the

Takashi Iwai (2):
      sha1dc: build git plumbing code more explicitly
      sha1dc: allow building with the external sha1dc library

Taylor Blau (8):
      pretty.c: delimit "%(trailers)" arguments with ","
      t4205: unfold across multiple lines
      doc: 'trailers' is the preferred way to format trailers
      doc: use "`<literal>`"-style quoting for literal strings
      t6300: refactor %(trailers) tests
      ref-filter.c: use trailer_opts to format trailers
      ref-filter.c: parse trailers arguments with %(contents) atom
      ref-filter.c: pass empty-string as NULL to atom parsers

Thomas Braun (1):
      completion: add --broken and --dirty to describe

Thomas Gummerer (3):
      read-cache: fix index corruption with index v4
      refs: strip out not allowed flags from ref_transaction_update
      http-push: fix construction of hex value from path

Todd Zullinger (1):
      api-argv-array.txt: remove broken link to string-list API

Tom G. Christensen (2):
      http: fix handling of missing CURLPROTO_*
      http: use a feature check to enable GSSAPI delegation control

Torsten Bögershausen (3):
      convert: add SAFE_CRLF_KEEP_CRLF
      apply: file commited with CRLF should roundtrip diff and apply
      test-lint: echo -e (or -E) is not portable

Urs Thuermann (1):
      git svn fetch: Create correct commit timestamp when using --localtime

William Duclot (1):
      rebase: make resolve message clearer for inexperienced users

brian m. carlson (14):
      builtin/fsck: convert remaining caller of get_sha1 to object_id
      builtin/merge-tree: convert remaining caller of get_sha1 to object_id
      submodule: convert submodule config lookup to use object_id
      remote: convert struct push_cas to struct object_id
      sequencer: convert to struct object_id
      builtin/update_ref: convert to struct object_id
      bisect: convert bisect_checkout to struct object_id
      builtin/unpack-file: convert to struct object_id
      Convert remaining callers of get_sha1 to get_oid.
      sha1_name: convert get_sha1* to get_oid*
      sha1_name: convert GET_SHA1* flags to GET_OID*
      sha1_name: convert uses of 40 to GIT_SHA1_HEXSZ
      vcs-svn: remove unused prototypes
      vcs-svn: rename repo functions to "svn_repo"

joernchen (1):
      cvsserver: use safe_pipe_capture instead of backticks

Ævar Arnfjörð Bjarmason (2):
      branch: add test for -m renaming multiple config sections
      tests: don't give unportable ">" to "test" built-in, use -gt

Øystein Walle (1):
      rev-parse: rev-parse: add --is-shallow-repository

Łukasz Gryglicki (1):
      merge: add a --signoff flag


^ permalink raw reply	[relevance 11%]

* Re: Enhancement request: git-push: Allow (configurable) default push-option
  2017-10-04 15:20   ` Marius Paliga
@ 2017-10-11  7:14     ` Marius Paliga
  2017-10-11  9:18       ` Marius Paliga
  0 siblings, 1 reply; 200+ results
From: Marius Paliga @ 2017-10-11  7:14 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Including proposed patch...


Signed-off-by: Marius Paliga <marius.paliga@gmail.com>
---
 Documentation/git-push.txt |  3 +++
 builtin/push.c             | 11 ++++++++++-
 t/t5545-push-options.sh    | 48 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3e76e99f3..133c42183 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -161,6 +161,9 @@ already exists on the remote side.
     Transmit the given string to the server, which passes them to
     the pre-receive as well as the post-receive hook. The given string
     must not contain a NUL or LF character.
+    Default push options can also be specified with configuration
+    variable `push.optiondefault`. String(s) specified here will always
+    be passed to the server without need to specify it using `--push-option`

 --receive-pack=<git-receive-pack>::
 --exec=<git-receive-pack>::
diff --git a/builtin/push.c b/builtin/push.c
index 2ac810422..4dd5d6f0e 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -32,6 +32,8 @@ static const char **refspec;
 static int refspec_nr;
 static int refspec_alloc;

+static struct string_list push_options = STRING_LIST_INIT_DUP;
+
 static void add_refspec(const char *ref)
 {
     refspec_nr++;
@@ -467,6 +469,8 @@ static int git_push_config(const char *k, const
char *v, void *cb)
 {
     int *flags = cb;
     int status;
+    const struct string_list *default_push_options;
+    struct string_list_item *item;

     status = git_gpg_config(k, v, NULL);
     if (status)
@@ -505,6 +509,12 @@ static int git_push_config(const char *k, const
char *v, void *cb)
         recurse_submodules = val;
     }

+    default_push_options = git_config_get_value_multi("push.optiondefault");
+    if (default_push_options)
+        for_each_string_list_item(item, default_push_options)
+            if (!string_list_has_string(&push_options, item->string))
+                string_list_append(&push_options, item->string);
+
     return git_default_config(k, v, NULL);
 }

@@ -515,7 +525,6 @@ int cmd_push(int argc, const char **argv, const
char *prefix)
     int push_cert = -1;
     int rc;
     const char *repo = NULL;    /* default repository */
-    struct string_list push_options = STRING_LIST_INIT_DUP;
     const struct string_list_item *item;

     struct option options[] = {
diff --git a/t/t5545-push-options.sh b/t/t5545-push-options.sh
index 90a4b0d2f..575f3dc38 100755
--- a/t/t5545-push-options.sh
+++ b/t/t5545-push-options.sh
@@ -140,6 +140,54 @@ test_expect_success 'push options and submodules' '
     test_cmp expect parent_upstream/.git/hooks/post-receive.push_options
 '

+test_expect_success 'default push option' '
+    mk_repo_pair &&
+    git -C upstream config receive.advertisePushOptions true &&
+    (
+        cd workbench &&
+        test_commit one &&
+        git push --mirror up &&
+        test_commit two &&
+        git -c push.optiondefault=default push up master
+    ) &&
+    test_refs master master &&
+    echo "default" >expect &&
+    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
+    test_cmp expect upstream/.git/hooks/post-receive.push_options
+'
+
+test_expect_success 'two default push options' '
+    mk_repo_pair &&
+    git -C upstream config receive.advertisePushOptions true &&
+    (
+        cd workbench &&
+        test_commit one &&
+        git push --mirror up &&
+        test_commit two &&
+        git -c push.optiondefault=default1 -c
push.optiondefault=default2 push up master
+    ) &&
+    test_refs master master &&
+    printf "default1\ndefault2\n" >expect &&
+    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
+    test_cmp expect upstream/.git/hooks/post-receive.push_options
+'
+
+test_expect_success 'default and manual push options' '
+    mk_repo_pair &&
+    git -C upstream config receive.advertisePushOptions true &&
+    (
+        cd workbench &&
+        test_commit one &&
+        git push --mirror up &&
+        test_commit two &&
+        git -c push.optiondefault=default push --push-option=manual up master
+    ) &&
+    test_refs master master &&
+    printf "default\nmanual\n" >expect &&
+    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
+    test_cmp expect upstream/.git/hooks/post-receive.push_options
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd

-- 
2.14.1


2017-10-04 17:20 GMT+02:00 Marius Paliga <marius.paliga@gmail.com>:
> Hi Stefan,
>
> I will look at it.
>
> Thanks,
> Marius
>
>
> 2017-10-03 18:53 GMT+02:00 Stefan Beller <sbeller@google.com>:
>> On Tue, Oct 3, 2017 at 3:15 AM, Marius Paliga <marius.paliga@gmail.com> wrote:
>>> There is a need to pass predefined push-option during "git push"
>>> without need to specify it explicitly.
>>>
>>> In another words we need to have a new "git config" variable to
>>> specify string that will be automatically passed as "--push-option"
>>> when pushing to remote.
>>>
>>> Something like the following:
>>>
>>> git config push.optionDefault AllowMultipleCommits
>>>
>>> and then command
>>>   git push
>>> would silently run
>>>   git push --push-option "AllowMultipleCommits"
>>
>> We would need to
>> * design this feature (seems like you already have a good idea what you need)
>> * implement it (see builtin/push.c):
>>  - move "struct string_list push_options = STRING_LIST_INIT_DUP;"
>>   to be a file-static variable, such that we have access to it outside
>> of cmd_push.
>>  - In git_push_config in builtin/push.c that parses the config, we'd
>> need to check
>>   for "push.optionDefault" and add these to the push_options (I assume multiple
>>   are allowed)
>> * document it (Documentation/git-push.txt)
>> * add a test for it ? (t/t5545-push-options.sh)
>>
>> Care to write a patch? Otherwise I'd mark it up as part of
>> #leftoverbits for now,
>> as it seems like a good starter project.
>>
>> Thanks,
>> Stefan

^ permalink raw reply	[relevance 4%]

* Re: Enhancement request: git-push: Allow (configurable) default push-option
  2017-10-11  7:14     ` Marius Paliga
@ 2017-10-11  9:18       ` Marius Paliga
  0 siblings, 0 replies; 200+ results
From: Marius Paliga @ 2017-10-11  9:18 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Found one possible issue when looking for duplicates, we need to use
  "unsorted_string_list_has_string" instead of "string_list_has_string"

-                       if (!string_list_has_string(&push_options,
item->string))
+                       if
(!unsorted_string_list_has_string(&push_options, item->string)) {

New (fixed) patch follows...


Signed-off-by: Marius Paliga <marius.paliga@gmail.com>
---
 Documentation/git-push.txt |  3 +++
 builtin/push.c             | 11 ++++++++++-
 t/t5545-push-options.sh    | 48 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3e76e99f3..133c42183 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -161,6 +161,9 @@ already exists on the remote side.
     Transmit the given string to the server, which passes them to
     the pre-receive as well as the post-receive hook. The given string
     must not contain a NUL or LF character.
+    Default push options can also be specified with configuration
+    variable `push.optiondefault`. String(s) specified here will always
+    be passed to the server without need to specify it using `--push-option`

 --receive-pack=<git-receive-pack>::
 --exec=<git-receive-pack>::
diff --git a/builtin/push.c b/builtin/push.c
index 2ac810422..ab458419a 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -32,6 +32,8 @@ static const char **refspec;
 static int refspec_nr;
 static int refspec_alloc;

+static struct string_list push_options = STRING_LIST_INIT_DUP;
+
 static void add_refspec(const char *ref)
 {
     refspec_nr++;
@@ -467,6 +469,8 @@ static int git_push_config(const char *k, const
char *v, void *cb)
 {
     int *flags = cb;
     int status;
+    const struct string_list *default_push_options;
+    struct string_list_item *item;

     status = git_gpg_config(k, v, NULL);
     if (status)
@@ -505,6 +509,12 @@ static int git_push_config(const char *k, const
char *v, void *cb)
         recurse_submodules = val;
     }

+    default_push_options = git_config_get_value_multi("push.optiondefault");
+    if (default_push_options)
+        for_each_string_list_item(item, default_push_options)
+            if (!unsorted_string_list_has_string(&push_options, item->string))
+                string_list_append(&push_options, item->string);
+
     return git_default_config(k, v, NULL);
 }

@@ -515,7 +525,6 @@ int cmd_push(int argc, const char **argv, const
char *prefix)
     int push_cert = -1;
     int rc;
     const char *repo = NULL;    /* default repository */
-    struct string_list push_options = STRING_LIST_INIT_DUP;
     const struct string_list_item *item;

     struct option options[] = {
diff --git a/t/t5545-push-options.sh b/t/t5545-push-options.sh
index 90a4b0d2f..575f3dc38 100755
--- a/t/t5545-push-options.sh
+++ b/t/t5545-push-options.sh
@@ -140,6 +140,54 @@ test_expect_success 'push options and submodules' '
     test_cmp expect parent_upstream/.git/hooks/post-receive.push_options
 '

+test_expect_success 'default push option' '
+    mk_repo_pair &&
+    git -C upstream config receive.advertisePushOptions true &&
+    (
+        cd workbench &&
+        test_commit one &&
+        git push --mirror up &&
+        test_commit two &&
+        git -c push.optiondefault=default push up master
+    ) &&
+    test_refs master master &&
+    echo "default" >expect &&
+    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
+    test_cmp expect upstream/.git/hooks/post-receive.push_options
+'
+
+test_expect_success 'two default push options' '
+    mk_repo_pair &&
+    git -C upstream config receive.advertisePushOptions true &&
+    (
+        cd workbench &&
+        test_commit one &&
+        git push --mirror up &&
+        test_commit two &&
+        git -c push.optiondefault=default1 -c
push.optiondefault=default2 push up master
+    ) &&
+    test_refs master master &&
+    printf "default1\ndefault2\n" >expect &&
+    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
+    test_cmp expect upstream/.git/hooks/post-receive.push_options
+'
+
+test_expect_success 'default and manual push options' '
+    mk_repo_pair &&
+    git -C upstream config receive.advertisePushOptions true &&
+    (
+        cd workbench &&
+        test_commit one &&
+        git push --mirror up &&
+        test_commit two &&
+        git -c push.optiondefault=default push --push-option=manual up master
+    ) &&
+    test_refs master master &&
+    printf "default\nmanual\n" >expect &&
+    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
+    test_cmp expect upstream/.git/hooks/post-receive.push_options
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd

-- 
2.14.1

2017-10-11 9:14 GMT+02:00 Marius Paliga <marius.paliga@gmail.com>:
> Including proposed patch...
>
>
> Signed-off-by: Marius Paliga <marius.paliga@gmail.com>
> ---
>  Documentation/git-push.txt |  3 +++
>  builtin/push.c             | 11 ++++++++++-
>  t/t5545-push-options.sh    | 48 ++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 61 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
> index 3e76e99f3..133c42183 100644
> --- a/Documentation/git-push.txt
> +++ b/Documentation/git-push.txt
> @@ -161,6 +161,9 @@ already exists on the remote side.
>      Transmit the given string to the server, which passes them to
>      the pre-receive as well as the post-receive hook. The given string
>      must not contain a NUL or LF character.
> +    Default push options can also be specified with configuration
> +    variable `push.optiondefault`. String(s) specified here will always
> +    be passed to the server without need to specify it using `--push-option`
>
>  --receive-pack=<git-receive-pack>::
>  --exec=<git-receive-pack>::
> diff --git a/builtin/push.c b/builtin/push.c
> index 2ac810422..4dd5d6f0e 100644
> --- a/builtin/push.c
> +++ b/builtin/push.c
> @@ -32,6 +32,8 @@ static const char **refspec;
>  static int refspec_nr;
>  static int refspec_alloc;
>
> +static struct string_list push_options = STRING_LIST_INIT_DUP;
> +
>  static void add_refspec(const char *ref)
>  {
>      refspec_nr++;
> @@ -467,6 +469,8 @@ static int git_push_config(const char *k, const
> char *v, void *cb)
>  {
>      int *flags = cb;
>      int status;
> +    const struct string_list *default_push_options;
> +    struct string_list_item *item;
>
>      status = git_gpg_config(k, v, NULL);
>      if (status)
> @@ -505,6 +509,12 @@ static int git_push_config(const char *k, const
> char *v, void *cb)
>          recurse_submodules = val;
>      }
>
> +    default_push_options = git_config_get_value_multi("push.optiondefault");
> +    if (default_push_options)
> +        for_each_string_list_item(item, default_push_options)
> +            if (!string_list_has_string(&push_options, item->string))
> +                string_list_append(&push_options, item->string);
> +
>      return git_default_config(k, v, NULL);
>  }
>
> @@ -515,7 +525,6 @@ int cmd_push(int argc, const char **argv, const
> char *prefix)
>      int push_cert = -1;
>      int rc;
>      const char *repo = NULL;    /* default repository */
> -    struct string_list push_options = STRING_LIST_INIT_DUP;
>      const struct string_list_item *item;
>
>      struct option options[] = {
> diff --git a/t/t5545-push-options.sh b/t/t5545-push-options.sh
> index 90a4b0d2f..575f3dc38 100755
> --- a/t/t5545-push-options.sh
> +++ b/t/t5545-push-options.sh
> @@ -140,6 +140,54 @@ test_expect_success 'push options and submodules' '
>      test_cmp expect parent_upstream/.git/hooks/post-receive.push_options
>  '
>
> +test_expect_success 'default push option' '
> +    mk_repo_pair &&
> +    git -C upstream config receive.advertisePushOptions true &&
> +    (
> +        cd workbench &&
> +        test_commit one &&
> +        git push --mirror up &&
> +        test_commit two &&
> +        git -c push.optiondefault=default push up master
> +    ) &&
> +    test_refs master master &&
> +    echo "default" >expect &&
> +    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
> +    test_cmp expect upstream/.git/hooks/post-receive.push_options
> +'
> +
> +test_expect_success 'two default push options' '
> +    mk_repo_pair &&
> +    git -C upstream config receive.advertisePushOptions true &&
> +    (
> +        cd workbench &&
> +        test_commit one &&
> +        git push --mirror up &&
> +        test_commit two &&
> +        git -c push.optiondefault=default1 -c
> push.optiondefault=default2 push up master
> +    ) &&
> +    test_refs master master &&
> +    printf "default1\ndefault2\n" >expect &&
> +    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
> +    test_cmp expect upstream/.git/hooks/post-receive.push_options
> +'
> +
> +test_expect_success 'default and manual push options' '
> +    mk_repo_pair &&
> +    git -C upstream config receive.advertisePushOptions true &&
> +    (
> +        cd workbench &&
> +        test_commit one &&
> +        git push --mirror up &&
> +        test_commit two &&
> +        git -c push.optiondefault=default push --push-option=manual up master
> +    ) &&
> +    test_refs master master &&
> +    printf "default\nmanual\n" >expect &&
> +    test_cmp expect upstream/.git/hooks/pre-receive.push_options &&
> +    test_cmp expect upstream/.git/hooks/post-receive.push_options
> +'
> +
>  . "$TEST_DIRECTORY"/lib-httpd.sh
>  start_httpd
>
> --
> 2.14.1
>
>
> 2017-10-04 17:20 GMT+02:00 Marius Paliga <marius.paliga@gmail.com>:
>> Hi Stefan,
>>
>> I will look at it.
>>
>> Thanks,
>> Marius
>>
>>
>> 2017-10-03 18:53 GMT+02:00 Stefan Beller <sbeller@google.com>:
>>> On Tue, Oct 3, 2017 at 3:15 AM, Marius Paliga <marius.paliga@gmail.com> wrote:
>>>> There is a need to pass predefined push-option during "git push"
>>>> without need to specify it explicitly.
>>>>
>>>> In another words we need to have a new "git config" variable to
>>>> specify string that will be automatically passed as "--push-option"
>>>> when pushing to remote.
>>>>
>>>> Something like the following:
>>>>
>>>> git config push.optionDefault AllowMultipleCommits
>>>>
>>>> and then command
>>>>   git push
>>>> would silently run
>>>>   git push --push-option "AllowMultipleCommits"
>>>
>>> We would need to
>>> * design this feature (seems like you already have a good idea what you need)
>>> * implement it (see builtin/push.c):
>>>  - move "struct string_list push_options = STRING_LIST_INIT_DUP;"
>>>   to be a file-static variable, such that we have access to it outside
>>> of cmd_push.
>>>  - In git_push_config in builtin/push.c that parses the config, we'd
>>> need to check
>>>   for "push.optionDefault" and add these to the push_options (I assume multiple
>>>   are allowed)
>>> * document it (Documentation/git-push.txt)
>>> * add a test for it ? (t/t5545-push-options.sh)
>>>
>>> Care to write a patch? Otherwise I'd mark it up as part of
>>> #leftoverbits for now,
>>> as it seems like a good starter project.
>>>
>>> Thanks,
>>> Stefan

^ permalink raw reply	[relevance 4%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-10 18:39       ` Stefan Beller
  2017-10-10 23:31         ` Junio C Hamano
@ 2017-10-11 14:52         ` Heiko Voigt
  2017-10-11 15:10         ` Josh Triplett
  2 siblings, 0 replies; 200+ results
From: Heiko Voigt @ 2017-10-11 14:52 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Josh Triplett, git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Tue, Oct 10, 2017 at 11:39:21AM -0700, Stefan Beller wrote:
> So you propose to make git-add behave like "git submodule add"
> (i.e. also add the .gitmodules entry for name/path/URL), which I
> like from a submodule perspective.

Well more like: clone and add will behave like "git submodule add" but
basically yes.

> However other users of gitlinks might be confused[1], which is why
> I refrained from "making every gitlink into a submodule". Specifically
> the more powerful a submodule operation is (the more fluff adds),
> the harder it should be for people to mis-use it.
> 
> [1] https://github.com/git-series/git-series/blob/master/INTERNALS.md
>      "git-series uses gitlinks to store pointer to commits in its own repo."

But would those users use

    git add

to add a gitlink? From the description in that file I read that it
points to commits in its own repository. Will there also be files
checked out like submodules at that location?

Otherwise I would propose that 'git add' could detect whether a gitlink
is a submodule by trying to read its git configuration. If we do not
find that we simply do not do anything.

> > If everyone agrees that submodules are the default way of handling
> > repositories insided repositories, IMO, 'git add' should also alter
> > .gitmodules by default. We could provide a switch to avoid doing that.
> 
> I wonder if that switch should be default-on (i.e. not treat a gitlink as
> a submodule initially, behavior as-is, and then eventually we will
> die() on unconfigured repos, expecting the user to make the decision)
> 
> > An intermediate solution would be to warn
> 
> That is already implemented by Peff.

Ah ok, thanks I suspected so when I realized that this discussion was
older.

> > but in the long run my goal
> > for submodules is and always was: Make them behave as close to files as
> > possible. And why should a 'git add submodule' not magically do
> > everything it can to make submodules just work? I can look into a patch
> > for that if people agree here...
> 
> I'd love to see this implemented. I cc'd Josh (the author of git-series), who
> may disagree with this, or has some good input how to go forward without
> breaking git-series.

Yeah, lets see if, as described above, that actually would break
git-series.

> > Regarding handling of gitlinks with or without .gitmodules:
> >
> > Currently we are actually in some intermediate state:
> >
> >  * If there is no .gitmodules file: No submodule processing on any
> >    gitlinks (AFAIK)
> 
> AFAIK this is true.
> 
> >  * If there is a .gitmodules files with some submodule configured: Do
> >    recursive fetch and push as far as possible on gitlinks.
> 
> * If submodule.recurse is set, then we also treat submodules like files
>   for checkout, reset, read-tree.

To clarify: If submodule.recurse is set but there is no .gitmodules file
we do submodule processing for the above commands?

> > So I am not sure whether there are actually many users (knowingly)
> > using a mix of some submodules configured and some not and then relying
> > on the submodule infrastructure.
> >
> > I would rather expect two sorts of users:
> >
> >   1. Those that do use .gitmodules
> 
> Those want to reap all benefits of good submodules.
> 
> >
> >   2. Those that do *not* use .gitmodules
> 
> As said above, we don't know if those users are
> "holding submodules wrong" or are using gitlinks for
> magic tricks (unrelated to submodules).

I did not want to say that they are "holding submodules wrong" but
rather that if they do not use .gitmodules they do that knowingly and
thus consistently not use .gitmodules for any gitlink.

> > Users that do not use any .gitmodules file will currently (AFAIK) not
> > get any submodule handling. So the question is are there really many
> > "mixed users"? My guess would be no.
> 
> I hope that there are few (if any) users of these mixed setups.

That sounds promising.

> > Because without those using this mixed we could switch to saying: "You
> > need to have a .gitmodules file for submodule handling" without much
> > fallout from breaking users use cases.
> 
> That seems reasonable to me, actually.

Nice.

> > Maybe we can test this out somehow? My patch series would be ready in
> > that case, just had to drop the first patch and adjust the commit
> > message of this one.
> 
> I wonder how we would test this, though? Do you have any idea
> (even vague) how we'd accomplish such a measurement?
> I fear we'll have to go this way blindly.

One idea would be to expose this somewhere to a limited amount of users.
I remember Jonathan was suggesting, back when Jens was working on the
recursive checkout, that he could add the series to the debian package
and see what happens. Or we could use Junios next branch? Something like
that. If we get complaints we know the assumption was wrong and we need
a fallback.

Cheers Heiko

^ permalink raw reply	[relevance 24%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-10 23:31         ` Junio C Hamano
  2017-10-10 23:41           ` Stefan Beller
@ 2017-10-11 14:56           ` Heiko Voigt
  1 sibling, 0 replies; 200+ results
From: Heiko Voigt @ 2017-10-11 14:56 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Stefan Beller, Josh Triplett, git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Wed, Oct 11, 2017 at 08:31:37AM +0900, Junio C Hamano wrote:
> Stefan Beller <sbeller@google.com> writes:
> 
> > So you propose to make git-add behave like "git submodule add"
> > (i.e. also add the .gitmodules entry for name/path/URL), which I
> > like from a submodule perspective.
> >
> > However other users of gitlinks might be confused[1], which is why
> > I refrained from "making every gitlink into a submodule". Specifically
> > the more powerful a submodule operation is (the more fluff adds),
> > the harder it should be for people to mis-use it.
> 
> A few questions that come to mind are:
> 
>  - Does "git add sub/" have enough information to populate
>    .gitmodules?  If we have reasonable "default" values for
>    .gitmodules entries (e.g. missing URL means we won't fetch when
>    asked to go recursively fetch), perhaps we can leave everything
>    other than "submodule.$name.path" undefined.

My suggestion would be: If we do not have them we do not populate them.
We could even go further and say: If we do not have the set "git
submodule add" would populate then we do not add anything to .gitmodules
and warn the user.

>  - Can't we help those who have gitlinks without .gitmodules entries
>    exactly the same way as above, i.e. when we see a gitlink and try
>    to treat it as a submodule, we'd first try to look it up from
>    .gitmodules (by going from path to name and then to
>    submodule.$name.$var); the above "'git add sub/' would add an
>    entry for .gitmodules" wish is based on the assumption that there
>    are reasonable "default" values for each of these $var--so by
>    basing on the same assumption, we can "pretend" as if these
>    submodule.$name.$var were in .gitmodules file when we see
>    gitlinks without .gitmodules entries.  IOW, if "git add sub/" can
>    add .gitmodules to help people without having to type "git
>    submodule add sub/", then we can give exactly the same degree of
>    help without even modifying .gitmodules when "git add sub/" is
>    run.

This "default" value thing got me thinking in a different direction. We
could use a scheme like that to get names (and values) for submodules
that are missing from the .gitmodules file. If we decide that we need to
handle them.

Cheers Heiko

^ permalink raw reply	[relevance 15%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-10 18:39       ` Stefan Beller
  2017-10-10 23:31         ` Junio C Hamano
  2017-10-11 14:52         ` Heiko Voigt
@ 2017-10-11 15:10         ` Josh Triplett
  2017-10-12 16:17           ` Brandon Williams
  2 siblings, 1 reply; 200+ results
From: Josh Triplett @ 2017-10-11 15:10 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Heiko Voigt, git, Jonathan Nieder, Jens Lehmann, Brandon Williams

On Tue, Oct 10, 2017 at 11:39:21AM -0700, Stefan Beller wrote:
> On Tue, Oct 10, 2017 at 6:03 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > but in the long run my goal
> > for submodules is and always was: Make them behave as close to files as
> > possible. And why should a 'git add submodule' not magically do
> > everything it can to make submodules just work? I can look into a patch
> > for that if people agree here...
> 
> I'd love to see this implemented. I cc'd Josh (the author of git-series), who
> may disagree with this, or has some good input how to go forward without
> breaking git-series.

git-series doesn't use the git-submodule command at all, nor does it
construct series trees using git-add or any other git command-line tool;
it constructs gitlinks directly. Most of the time, it doesn't even make
sense to `git checkout` a series branch. Modifying commands like git-add
and similar to automatically manage .gitmodules won't cause any issue at
all, as long as git itself doesn't start rejecting or complaining about
repositories that have gitlinks without a .gitmodules file.

- Josh Triplett

^ permalink raw reply	[relevance 22%]

* Re: Branch switching with submodules where the submodule replaces a folder aborts unexpectedly
  2017-10-09 21:59 ` Stefan Beller
@ 2017-10-12 11:48   ` Thomas Braun
  0 siblings, 0 replies; 200+ results
From: Thomas Braun @ 2017-10-12 11:48 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

> Stefan Beller <sbeller@google.com> hat am 9. Oktober 2017 um 23:59 geschrieben:
> 
> 
> On Mon, Oct 9, 2017 at 2:29 PM, Thomas Braun
> <thomas.braun@virtuell-zuhause.de> wrote:
> > Hi,
> >
> > I'm currently in the progress of pulling some subprojects in a git repository of mine into their
> > own repositories and adding these subprojects back as submodules.
> >
> > While doing this I enountered a potential bug as checkout complains on branch switching that a 
> > file already exists.
> 
> (And I presume you know about --recurse-submodules as a flag for git-checkout)

No I did not know about it. I tend to not know options which don't complete in my shell (patch follows for that).

> This is consistent with our tests, unfortunately.
> 
> git/t$ ./t2013-checkout-submodule.sh
> ...
> not ok 15 - git checkout --recurse-submodules: replace submodule with
> a directory # TODO known breakage
> ...
> 
> > If I'm misusing git here I'm glad for any advice.
> 
> You are not.

Glad to know that.

> Apart from this bug report, would you think that such filtering of
> trees into submodules (and back again) might be an interesting feature
> of Git or are these cases rare and special?

For me not particularly. In my case it is a one time thing going from an embedded project folder to a submodule.

^ permalink raw reply	[relevance 16%]

* Re: [RFC PATCH 2/4] change submodule push test to use proper repository setup
  2017-10-11 15:10         ` Josh Triplett
@ 2017-10-12 16:17           ` Brandon Williams
  0 siblings, 0 replies; 200+ results
From: Brandon Williams @ 2017-10-12 16:17 UTC (permalink / raw)
  To: Josh Triplett; +Cc: Stefan Beller, Heiko Voigt, git, Jonathan Nieder, Jens Lehmann

On 10/11, Josh Triplett wrote:
> On Tue, Oct 10, 2017 at 11:39:21AM -0700, Stefan Beller wrote:
> > On Tue, Oct 10, 2017 at 6:03 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > > but in the long run my goal
> > > for submodules is and always was: Make them behave as close to files as
> > > possible. And why should a 'git add submodule' not magically do
> > > everything it can to make submodules just work? I can look into a patch
> > > for that if people agree here...
> > 
> > I'd love to see this implemented. I cc'd Josh (the author of git-series), who
> > may disagree with this, or has some good input how to go forward without
> > breaking git-series.
> 
> git-series doesn't use the git-submodule command at all, nor does it
> construct series trees using git-add or any other git command-line tool;
> it constructs gitlinks directly. Most of the time, it doesn't even make
> sense to `git checkout` a series branch. Modifying commands like git-add
> and similar to automatically manage .gitmodules won't cause any issue at
> all, as long as git itself doesn't start rejecting or complaining about
> repositories that have gitlinks without a .gitmodules file.

That's good to know!  And from what I remember, with the commands we've
begun teaching to understand submodules we have been requiring a
.gitmodules entry for a submodule in order to do the recursion, and a
gitlink without a .gitmodules entry would simply be ignored or skipped.

-- 
Brandon Williams

^ permalink raw reply	[relevanc