[PATCH] push: make `--force-with-lease[=<ref>]` safer

[PATCH] push: make `--force-with-lease[=<ref>]` safer

[Srinidhi Kaushik]

The `--force-with-lease` option in `git-push`, makes sure that
refs on remote aren't clobbered by unexpected changes when the
"<expect>" ref value is explicitly specified.

For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip
of the remote tracking branch is populated as the "<expect>" value,
there is a possibility of allowing unwanted overwrites on the remote
side when some tools that implicitly fetch remote-tracking refs in
the background are used with the repository. If a remote-tracking ref
was updated when a rewrite is happening locally and if those changes
are pushed by omitting the "<expect>" value in `--force-with-lease`,
any new changes from the updated tip will be lost locally and will
be overwritten on the remote.

This problem can be addressed by checking the `reflog` of the branch
that is being pushed and verify if there in a entry with the remote
tracking ref. By running this check, we can ensure that refs being
are fetched in the background while a "lease" is being held are not
overlooked before a push, and any new changes can be acknowledged
and (if necessary) integrated locally.

The new check will cause `git-push` to fail if it detects the presence
of any updated refs that we do not have locally and reject the push
stating `implicit fetch` as the reason.

An experimental configuration setting: `push.rejectImplicitFetch`
which defaults to `true` (when `features.experimental` is enabled)
has been added, to allow `git-push` to reject a push if the check
fails.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---

Hello,
I picked this up from #leftoverbits over at GitHub [1] from the open
issues list. This idea [2], for a safer `--force-with-lease` was
originally proposed by Johannes on the mailing list.

[1]: https://github.com/gitgitgadget/git/issues/640
[2]: https://lore.kernel.org/git/nycvar.QRO.7.76.6.1808272306271.73@tvgsbejvaqbjf.bet/

Thanks.

 Documentation/config/feature.txt |  3 +
 Documentation/config/push.txt    | 14 +++++
 Documentation/git-push.txt       |  6 ++
 builtin/send-pack.c              |  5 ++
 remote.c                         | 96 +++++++++++++++++++++++++++++---
 remote.h                         |  4 +-
 send-pack.c                      |  1 +
 t/t5533-push-cas.sh              | 86 ++++++++++++++++++++++++++++
 transport-helper.c               |  5 ++
 transport.c                      |  5 ++
 10 files changed, 217 insertions(+), 8 deletions(-)

diff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt
index c0cbf2bb1c..f93e9fd898 100644
--- a/Documentation/config/feature.txt
+++ b/Documentation/config/feature.txt
@@ -18,6 +18,9 @@ skipping more commits at a time, reducing the number of round trips.
 * `protocol.version=2` speeds up fetches from repositories with many refs by
 allowing the client to specify which refs to list before the server lists
 them.
++
+* `push.rejectImplicitFetch=true` runs additional checks for linkgit:git-push[1]
+`--force-with-lease` to mitigate implicit updates of remote-tracking refs.

 feature.manyFiles::
 	Enable config options that optimize for repos with many files in the
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..1a7184034d 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,17 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.rejectImplicitFetch::
+	If set to `true`, runs additional checks for the `--force-with-lease`
+	option when used with linkgit:git-push[1] if the expected value for
+	the remote ref is unspecified (`--force-with-lease[=<ref>]`), and
+	instead asked depend on the current value of the remote-tracking ref.
+	The check ensures that the commit at the tip of the remote-tracking
+	branch -- which may have been implicitly updated by tools that fetch
+	remote refs by running linkgit:git-fetch[1] in the background -- has
+	been integrated locally, when holding the "lease". If the new changes
+	from such remote-tracking refs have not been updated locally before
+	pushing, linkgit:git-push[1] will fail indicating the reject reason
+	as `implicit fetch`. Enabling `feature.experimental` makes this option
+	default to `true`.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..2176a743f3 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, setting the (experimental) `push.rejectImplicitFetch` option
+to `true` will ensure changes from remote-tracking refs that are updated in the
+background using linkgit:git-fetch[1] are accounted for (either by integrating
+them locally, or explicitly specifying an overwrite), by rejecting to update
+such refs.

 -f::
 --force::
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 2b9610f121..6500a8267a 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;

+		case REF_STATUS_REJECT_IMPLICIT_FETCH:
+			res = "error";
+			msg = "implicit fetch";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/remote.c b/remote.c
index c5ed74f91c..ee2dedd15b 100644
--- a/remote.c
+++ b/remote.c
@@ -49,6 +49,8 @@ static const char *pushremote_name;
 static struct rewrites rewrites;
 static struct rewrites rewrites_push;

+static struct object_id cas_reflog_check_oid;
+
 static int valid_remote(const struct remote *remote)
 {
 	return (!!remote->url) || (!!remote->foreign_vcs);
@@ -1446,6 +1448,22 @@ int match_push_refs(struct ref *src, struct ref **dst,
 	return 0;
 }

+/*
+ * Consider `push.rejectImplicitFetch` to be set to true if experimental
+ * features are enabled; use user-defined value if set explicitly.
+ */
+int reject_implicit_fetch()
+{
+	int conf = 0;
+	if (!git_config_get_bool("push.rejectImplicitFetch", &conf))
+		return conf;
+
+	if (!git_config_get_bool("feature.experimental", &conf))
+		return conf;
+
+	return conf;
+}
+
 void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 			     int force_update)
 {
@@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 		 * If the remote ref has moved and is now different
 		 * from what we expect, reject any push.
 		 *
-		 * It also is an error if the user told us to check
-		 * with the remote-tracking branch to find the value
-		 * to expect, but we did not have such a tracking
-		 * branch.
+		 * It also is an error if the user told us to check with the
+		 * remote-tracking branch to find the value to expect, but we
+		 * did not have such a tracking branch, or we have one that
+		 * has new changes.
 		 */
 		if (ref->expect_old_sha1) {
 			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
 				reject_reason = REF_STATUS_REJECT_STALE;
+			else if (reject_implicit_fetch() && ref->implicit_fetch)
+				reject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;
 			else
-				/* If the ref isn't stale then force the update. */
+				/*
+				 * If the ref isn't stale, or there was no
+				 * implicit fetch, force the update.
+				 */
 				force_ref_update = 1;
 		}

@@ -2272,23 +2295,67 @@ static int remote_tracking(struct remote *remote, const char *refname,
 	return 0;
 }

+static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,
+			     const char *ident, timestamp_t timestamp, int tz,
+			     const char *message, void *cb_data)
+{
+	return oideq(noid, &cas_reflog_check_oid);
+}
+
+/*
+ * Iterate through the reflog of a local branch and check if the tip of the
+ * remote-tracking branch is reachable from one of the entries.
+ */
+static int remote_ref_in_reflog(const struct object_id *r_oid,
+				const struct object_id *l_oid,
+				const char *local_ref_name)
+{
+	int ret = 0;
+	cas_reflog_check_oid = *r_oid;
+
+	struct commit *r_commit, *l_commit;
+	l_commit = lookup_commit_reference(the_repository, l_oid);
+	r_commit = lookup_commit_reference(the_repository, r_oid);
+
+	/*
+	 * If the remote-tracking ref is an ancestor of the local ref (a merge,
+	 * for instance) there is no need to iterate through the reflog entries
+	 * to ensure reachability; it can be skipped to return early instead.
+	 */
+	ret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;
+	if (ret)
+		goto skip;
+
+	ret = for_each_reflog_ent_reverse(local_ref_name,
+					  oid_in_reflog_ent,
+					  NULL);
+skip:
+	return ret;
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
 {
-	int i;
+	int i, do_reflog_check = 0;
+	struct object_id oid;
+	struct ref *local_ref = get_local_ref(ref->name);

 	/* Find an explicit --<option>=<name>[:<value>] entry */
 	for (i = 0; i < cas->nr; i++) {
 		struct push_cas *entry = &cas->entry[i];
 		if (!refname_match(entry->refname, ref->name))
 			continue;
+
 		ref->expect_old_sha1 = 1;
 		if (!entry->use_tracking)
 			oidcpy(&ref->old_oid_expect, &entry->expect);
 		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 			oidclr(&ref->old_oid_expect);
-		return;
+		else
+			do_reflog_check = 1;
+
+		goto reflog_check;
 	}

 	/* Are we using "--<option>" to cover all? */
@@ -2298,6 +2365,21 @@ static void apply_cas(struct push_cas_option *cas,
 	ref->expect_old_sha1 = 1;
 	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 		oidclr(&ref->old_oid_expect);
+	else
+		do_reflog_check = 1;
+
+reflog_check:
+	/*
+	 * For cases where "--force-with-lease[=<refname>]" i.e., when the
+	 * "<expect>" value is unspecified, run additional checks to verify
+	 * if the tip of the remote-tracking branch (if implicitly updated
+	 * when a "lease" is being held) is reachable from at least one entry
+	 * in the reflog of the local branch that is being pushed, ensuring
+	 * new changes (if any) have been integrated locally.
+	 */
+	if (do_reflog_check && local_ref && !read_ref(local_ref->name, &oid))
+		ref->implicit_fetch = !remote_ref_in_reflog(&ref->old_oid, &oid,
+							    local_ref->name);
 }

 void apply_push_cas(struct push_cas_option *cas,
diff --git a/remote.h b/remote.h
index 5e3ea5a26d..f859fa5fed 100644
--- a/remote.h
+++ b/remote.h
@@ -104,7 +104,8 @@ struct ref {
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		implicit_fetch:1;

 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -133,6 +134,7 @@ struct ref {
 		REF_STATUS_REJECT_FETCH_FIRST,
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
+		REF_STATUS_REJECT_IMPLICIT_FETCH,
 		REF_STATUS_REJECT_SHALLOW,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
diff --git a/send-pack.c b/send-pack.c
index 632f1580ca..fe7f14add4 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_IMPLICIT_FETCH:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..840b2a95f9 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -13,6 +13,41 @@ setup_srcdst_basic () {
 	)
 }

+setup_implicit_fetch () {
+	rm -fr src dup dst &&
+	git init --bare dst &&
+	git clone --no-local dst src &&
+	git clone --no-local dst dup
+	(
+		cd src &&
+		test_commit A &&
+		git push
+	) &&
+	(
+		cd dup &&
+		git fetch &&
+		git merge origin/master &&
+		test_commit B &&
+		git switch -c branch master~1 &&
+		test_commit C &&
+		test_commit D &&
+		git push --all
+	) &&
+	(
+		cd src &&
+		git switch master &&
+		git fetch --all &&
+		git branch branch --track origin/branch &&
+		git merge origin/master
+	) &&
+	(
+		cd dup &&
+		git switch master &&
+		test_commit E &&
+		git push origin master:master
+	)
+}
+
 test_expect_success setup '
 	# create template repository
 	test_commit A &&
@@ -256,4 +291,55 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '

+test_expect_success 'implicit updates to remote-tracking refs with `push.rejectImplicitFetch` set (protected, all refs)' '
+	setup_implicit_fetch &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch master &&
+		test_commit G &&
+		git switch branch &&
+		test_commit H &&
+		git fetch --all &&
+		git config --local feature.experimental true &&
+		test_must_fail git push --force-with-lease --all 2>err &&
+		grep "implicit fetch" err
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch &&
+	(
+		cd src &&
+		git config --local feature.experimental false &&
+		git push --force-with-lease --all 2>err &&
+		grep "forced update" err
+	)
+'
+
+test_expect_success 'implicit updates to remote-tracking refs with `push.rejectImplicitFetch` set (protected, specific ref)' '
+	setup_implicit_fetch &&
+	git ls-remote dst refs/heads/master >actual &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit F &&
+		git switch master &&
+		test_commit G &&
+		git fetch  &&
+		git config --local push.rejectImplicitFetch true &&
+		test_must_fail git push --force-with-lease=master --all 2>err &&
+		grep "implicit fetch" err
+	) &&
+	git ls-remote dst refs/heads/master >expect &&
+	test_cmp expect actual &&
+	(
+		cd src &&
+		git push --force --force-with-lease --all 2>err &&
+		grep "forced update" err
+	)
+'
+
 test_done
diff --git a/transport-helper.c b/transport-helper.c
index c52c99d829..75b4c1b758 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "ignored fetch")) {
+			status = REF_STATUS_REJECT_IMPLICIT_FETCH;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -896,6 +900,7 @@ static int push_refs_with_push(struct transport *transport,
 		switch (ref->status) {
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
+		case REF_STATUS_REJECT_IMPLICIT_FETCH:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
diff --git a/transport.c b/transport.c
index 43e24bf1e5..588575498f 100644
--- a/transport.c
+++ b/transport.c
@@ -567,6 +567,10 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "stale info", porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_IMPLICIT_FETCH:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "implicit fetch", porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -1101,6 +1105,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_IMPLICIT_FETCH) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;

 		strbuf_reset(&buf);
--
2.28.0

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Phillip Wood]

Hi Srinidhi

Thanks for working on this, making --force-with-lease safer would be a 
valuable contribution

On 04/09/2020 19:51, Srinidhi Kaushik wrote:
> The `--force-with-lease` option in `git-push`, makes sure that
> refs on remote aren't clobbered by unexpected changes when the
> "<expect>" ref value is explicitly specified.

I think it would help to write out 
`--force-with-lease[=<refname>[:<expect>]]` so readers know what 
"<expect>" is referring to

> For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip
> of the remote tracking branch is populated as the "<expect>" value,
> there is a possibility of allowing unwanted overwrites on the remote
> side when some tools that implicitly fetch remote-tracking refs in
> the background are used with the repository. If a remote-tracking ref
> was updated when a rewrite is happening locally and if those changes
> are pushed by omitting the "<expect>" value in `--force-with-lease`,
> any new changes from the updated tip will be lost locally and will
> be overwritten on the remote.
> 
> This problem can be addressed by checking the `reflog` of the branch
> that is being pushed and verify if there in a entry with the remote
> tracking ref. By running this check, we can ensure that refs being
> are fetched in the background while a "lease" is being held are not
> overlooked before a push, and any new changes can be acknowledged
> and (if necessary) integrated locally.

An addition safety measure would be to check the reflog of the local 
commit and the tip of the remote tracking branch dates overlap. 
Otherwise if there is an implicit fetch of a remote head that has been 
rewound we still push the local branch when we shouldn't.

> The new check will cause `git-push` to fail if it detects the presence
> of any updated refs that we do not have locally and reject the push
> stating `implicit fetch` as the reason.

'implicit fetch' is a rather terse message - can we say something along 
the lines of "the remote has been updated since the last merge/push"?

> An experimental configuration setting: `push.rejectImplicitFetch`
> which defaults to `true` (when `features.experimental` is enabled)
> has been added, to allow `git-push` to reject a push if the check
> fails.

Making this available with features.experimental initially is probably a 
good idea, I hope it will become the default if in future versions.

> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
> ---
> 
> Hello,
> I picked this up from #leftoverbits over at GitHub [1] from the open
> issues list. This idea [2], for a safer `--force-with-lease` was
> originally proposed by Johannes on the mailing list.
> 
> [1]: https://github.com/gitgitgadget/git/issues/640
> [2]: https://lore.kernel.org/git/nycvar.QRO.7.76.6.1808272306271.73@tvgsbejvaqbjf.bet/
> 
> Thanks.
> 
>   Documentation/config/feature.txt |  3 +
>   Documentation/config/push.txt    | 14 +++++
>   Documentation/git-push.txt       |  6 ++
>   builtin/send-pack.c              |  5 ++
>   remote.c                         | 96 +++++++++++++++++++++++++++++---
>   remote.h                         |  4 +-
>   send-pack.c                      |  1 +
>   t/t5533-push-cas.sh              | 86 ++++++++++++++++++++++++++++
>   transport-helper.c               |  5 ++
>   transport.c                      |  5 ++
>   10 files changed, 217 insertions(+), 8 deletions(-)
> 
> diff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt
> index c0cbf2bb1c..f93e9fd898 100644
> --- a/Documentation/config/feature.txt
> +++ b/Documentation/config/feature.txt
> @@ -18,6 +18,9 @@ skipping more commits at a time, reducing the number of round trips.
>   * `protocol.version=2` speeds up fetches from repositories with many refs by
>   allowing the client to specify which refs to list before the server lists
>   them.
> ++
> +* `push.rejectImplicitFetch=true` runs additional checks for linkgit:git-push[1]
> +`--force-with-lease` to mitigate implicit updates of remote-tracking refs.
> 
>   feature.manyFiles::
>   	Enable config options that optimize for repos with many files in the
> diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
> index f5e5b38c68..1a7184034d 100644
> --- a/Documentation/config/push.txt
> +++ b/Documentation/config/push.txt
> @@ -114,3 +114,17 @@ push.recurseSubmodules::
>   	specifying '--recurse-submodules=check|on-demand|no'.
>   	If not set, 'no' is used by default, unless 'submodule.recurse' is
>   	set (in which case a 'true' value means 'on-demand').
> +
> +push.rejectImplicitFetch::
> +	If set to `true`, runs additional checks for the `--force-with-lease`
> +	option when used with linkgit:git-push[1] if the expected value for
> +	the remote ref is unspecified (`--force-with-lease[=<ref>]`), and
> +	instead asked depend on the current value of the remote-tracking ref.
> +	The check ensures that the commit at the tip of the remote-tracking
> +	branch -- which may have been implicitly updated by tools that fetch
> +	remote refs by running linkgit:git-fetch[1] in the background -- has
> +	been integrated locally, when holding the "lease". If the new changes
> +	from such remote-tracking refs have not been updated locally before
> +	pushing, linkgit:git-push[1] will fail indicating the reject reason
> +	as `implicit fetch`. Enabling `feature.experimental` makes this option
> +	default to `true`.
> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
> index 3b8053447e..2176a743f3 100644
> --- a/Documentation/git-push.txt
> +++ b/Documentation/git-push.txt
> @@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally
>   force push changes to `master` if the remote version is still at
>   `base`, regardless of what your local `remotes/origin/master` has been
>   updated to in the background.
> ++
> +Alternatively, setting the (experimental) `push.rejectImplicitFetch` option
> +to `true` will ensure changes from remote-tracking refs that are updated in the
> +background using linkgit:git-fetch[1] are accounted for (either by integrating
> +them locally, or explicitly specifying an overwrite), by rejecting to update
> +such refs.
> 
>   -f::
>   --force::
> diff --git a/builtin/send-pack.c b/builtin/send-pack.c
> index 2b9610f121..6500a8267a 100644
> --- a/builtin/send-pack.c
> +++ b/builtin/send-pack.c
> @@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)
>   			msg = "stale info";
>   			break;
> 
> +		case REF_STATUS_REJECT_IMPLICIT_FETCH:
> +			res = "error";
> +			msg = "implicit fetch";
> +			break;
> +
>   		case REF_STATUS_REJECT_ALREADY_EXISTS:
>   			res = "error";
>   			msg = "already exists";
> diff --git a/remote.c b/remote.c
> index c5ed74f91c..ee2dedd15b 100644
> --- a/remote.c
> +++ b/remote.c
> @@ -49,6 +49,8 @@ static const char *pushremote_name;
>   static struct rewrites rewrites;
>   static struct rewrites rewrites_push;
> 
> +static struct object_id cas_reflog_check_oid;
> +

rather than using a global variable I think it would be better just to 
pass this value around using the cb_data argument of the reflog callback 
function

>   static int valid_remote(const struct remote *remote)
>   {
>   	return (!!remote->url) || (!!remote->foreign_vcs);
> @@ -1446,6 +1448,22 @@ int match_push_refs(struct ref *src, struct ref **dst,
>   	return 0;
>   }
> 
> +/*
> + * Consider `push.rejectImplicitFetch` to be set to true if experimental
> + * features are enabled; use user-defined value if set explicitly.
> + */
> +int reject_implicit_fetch()
> +{
> +	int conf = 0;
> +	if (!git_config_get_bool("push.rejectImplicitFetch", &conf))
> +		return conf;
> +
> +	if (!git_config_get_bool("feature.experimental", &conf))
> +		return conf;
> +
> +	return conf;
> +}
> +
>   void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
>   			     int force_update)
>   {
> @@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
>   		 * If the remote ref has moved and is now different
>   		 * from what we expect, reject any push.
>   		 *
> -		 * It also is an error if the user told us to check
> -		 * with the remote-tracking branch to find the value
> -		 * to expect, but we did not have such a tracking
> -		 * branch.
> +		 * It also is an error if the user told us to check with the
> +		 * remote-tracking branch to find the value to expect, but we
> +		 * did not have such a tracking branch, or we have one that
> +		 * has new changes.
>   		 */
>   		if (ref->expect_old_sha1) {
>   			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
>   				reject_reason = REF_STATUS_REJECT_STALE;
> +			else if (reject_implicit_fetch() && ref->implicit_fetch)
> +				reject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;
>   			else
> -				/* If the ref isn't stale then force the update. */
> +				/*
> +				 * If the ref isn't stale, or there was no
> +				 * implicit fetch, force the update.
> +				 */
>   				force_ref_update = 1;
>   		}
> 
> @@ -2272,23 +2295,67 @@ static int remote_tracking(struct remote *remote, const char *refname,
>   	return 0;
>   }
> 
> +static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,
> +			     const char *ident, timestamp_t timestamp, int tz,
> +			     const char *message, void *cb_data)
> +{

using the callback data we would have something like

struct oid *remote_head = cb_data;
return oideq(noid, remote_head);

> +	return oideq(noid, &cas_reflog_check_oid);
> +}
> +
> +/*
> + * Iterate through the reflog of a local branch and check if the tip of the
> + * remote-tracking branch is reachable from one of the entries.
> + */
> +static int remote_ref_in_reflog(const struct object_id *r_oid,
> +				const struct object_id *l_oid,
> +				const char *local_ref_name)
> +{
> +	int ret = 0;
> +	cas_reflog_check_oid = *r_oid;
> +
> +	struct commit *r_commit, *l_commit;

Our coding style is to declare all variables before any statements, so 
this should come above `cas_reflog_check_oid = *r_oid` but that line 
wants to go away anyway.

> +	l_commit = lookup_commit_reference(the_repository, l_oid);
> +	r_commit = lookup_commit_reference(the_repository, r_oid);
> +
> +	/*
> +	 * If the remote-tracking ref is an ancestor of the local ref (a merge,
> +	 * for instance) there is no need to iterate through the reflog entries
> +	 * to ensure reachability; it can be skipped to return early instead.
> +	 */
> +	ret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;
> +	if (ret)
> +		goto skip;

Rather than using a goto it would perhaps be better to do

if (!ret)
	ret = for_each_reflog_...

> +
> +	ret = for_each_reflog_ent_reverse(local_ref_name,
> +					  oid_in_reflog_ent,
> +					  NULL);

using the callback data we'd pass r_oid rather than NULL as the last 
argument

> +skip:
> +	return ret;
> +}
> +
>   static void apply_cas(struct push_cas_option *cas,
>   		      struct remote *remote,
>   		      struct ref *ref)
>   {
> -	int i;
> +	int i, do_reflog_check = 0;
> +	struct object_id oid;
> +	struct ref *local_ref = get_local_ref(ref->name);
> 
>   	/* Find an explicit --<option>=<name>[:<value>] entry */
>   	for (i = 0; i < cas->nr; i++) {
>   		struct push_cas *entry = &cas->entry[i];
>   		if (!refname_match(entry->refname, ref->name))
>   			continue;
> +
>   		ref->expect_old_sha1 = 1;
>   		if (!entry->use_tracking)
>   			oidcpy(&ref->old_oid_expect, &entry->expect);
>   		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
>   			oidclr(&ref->old_oid_expect);
> -		return;
> +		else
> +			do_reflog_check = 1;
> +
> +		goto reflog_check;

I'm not too keen in jumping here, can't we just check `do_reflog_check` 
below?

Best Wishes

Phillip

>   	}
> 
>   	/* Are we using "--<option>" to cover all? */
> @@ -2298,6 +2365,21 @@ static void apply_cas(struct push_cas_option *cas,
>   	ref->expect_old_sha1 = 1;
>   	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
>   		oidclr(&ref->old_oid_expect);
> +	else
> +		do_reflog_check = 1;
> +
> +reflog_check:
> +	/*
> +	 * For cases where "--force-with-lease[=<refname>]" i.e., when the
> +	 * "<expect>" value is unspecified, run additional checks to verify
> +	 * if the tip of the remote-tracking branch (if implicitly updated
> +	 * when a "lease" is being held) is reachable from at least one entry
> +	 * in the reflog of the local branch that is being pushed, ensuring
> +	 * new changes (if any) have been integrated locally.
> +	 */
> +	if (do_reflog_check && local_ref && !read_ref(local_ref->name, &oid))
> +		ref->implicit_fetch = !remote_ref_in_reflog(&ref->old_oid, &oid,
> +							    local_ref->name);
>   }
> 
>   void apply_push_cas(struct push_cas_option *cas,
> diff --git a/remote.h b/remote.h
> index 5e3ea5a26d..f859fa5fed 100644
> --- a/remote.h
> +++ b/remote.h
> @@ -104,7 +104,8 @@ struct ref {
>   		forced_update:1,
>   		expect_old_sha1:1,
>   		exact_oid:1,
> -		deletion:1;
> +		deletion:1,
> +		implicit_fetch:1;
> 
>   	enum {
>   		REF_NOT_MATCHED = 0, /* initial value */
> @@ -133,6 +134,7 @@ struct ref {
>   		REF_STATUS_REJECT_FETCH_FIRST,
>   		REF_STATUS_REJECT_NEEDS_FORCE,
>   		REF_STATUS_REJECT_STALE,
> +		REF_STATUS_REJECT_IMPLICIT_FETCH,
>   		REF_STATUS_REJECT_SHALLOW,
>   		REF_STATUS_UPTODATE,
>   		REF_STATUS_REMOTE_REJECT,
> diff --git a/send-pack.c b/send-pack.c
> index 632f1580ca..fe7f14add4 100644
> --- a/send-pack.c
> +++ b/send-pack.c
> @@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
>   	case REF_STATUS_REJECT_FETCH_FIRST:
>   	case REF_STATUS_REJECT_NEEDS_FORCE:
>   	case REF_STATUS_REJECT_STALE:
> +	case REF_STATUS_REJECT_IMPLICIT_FETCH:
>   	case REF_STATUS_REJECT_NODELETE:
>   		return CHECK_REF_STATUS_REJECTED;
>   	case REF_STATUS_UPTODATE:
> diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
> index 0b0eb1d025..840b2a95f9 100755
> --- a/t/t5533-push-cas.sh
> +++ b/t/t5533-push-cas.sh
> @@ -13,6 +13,41 @@ setup_srcdst_basic () {
>   	)
>   }
> 
> +setup_implicit_fetch () {
> +	rm -fr src dup dst &&
> +	git init --bare dst &&
> +	git clone --no-local dst src &&
> +	git clone --no-local dst dup
> +	(
> +		cd src &&
> +		test_commit A &&
> +		git push
> +	) &&
> +	(
> +		cd dup &&
> +		git fetch &&
> +		git merge origin/master &&
> +		test_commit B &&
> +		git switch -c branch master~1 &&
> +		test_commit C &&
> +		test_commit D &&
> +		git push --all
> +	) &&
> +	(
> +		cd src &&
> +		git switch master &&
> +		git fetch --all &&
> +		git branch branch --track origin/branch &&
> +		git merge origin/master
> +	) &&
> +	(
> +		cd dup &&
> +		git switch master &&
> +		test_commit E &&
> +		git push origin master:master
> +	)
> +}
> +
>   test_expect_success setup '
>   	# create template repository
>   	test_commit A &&
> @@ -256,4 +291,55 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
>   	)
>   '
> 
> +test_expect_success 'implicit updates to remote-tracking refs with `push.rejectImplicitFetch` set (protected, all refs)' '
> +	setup_implicit_fetch &&
> +	test_when_finished "rm -fr dst src dup" &&
> +	git ls-remote dst refs/heads/master >expect.master &&
> +	git ls-remote dst refs/heads/master >expect.branch &&
> +	(
> +		cd src &&
> +		git switch master &&
> +		test_commit G &&
> +		git switch branch &&
> +		test_commit H &&
> +		git fetch --all &&
> +		git config --local feature.experimental true &&
> +		test_must_fail git push --force-with-lease --all 2>err &&
> +		grep "implicit fetch" err
> +	) &&
> +	git ls-remote dst refs/heads/master >actual.master &&
> +	git ls-remote dst refs/heads/master >actual.branch &&
> +	test_cmp expect.master actual.master &&
> +	test_cmp expect.branch actual.branch &&
> +	(
> +		cd src &&
> +		git config --local feature.experimental false &&
> +		git push --force-with-lease --all 2>err &&
> +		grep "forced update" err
> +	)
> +'
> +
> +test_expect_success 'implicit updates to remote-tracking refs with `push.rejectImplicitFetch` set (protected, specific ref)' '
> +	setup_implicit_fetch &&
> +	git ls-remote dst refs/heads/master >actual &&
> +	(
> +		cd src &&
> +		git switch branch &&
> +		test_commit F &&
> +		git switch master &&
> +		test_commit G &&
> +		git fetch  &&
> +		git config --local push.rejectImplicitFetch true &&
> +		test_must_fail git push --force-with-lease=master --all 2>err &&
> +		grep "implicit fetch" err
> +	) &&
> +	git ls-remote dst refs/heads/master >expect &&
> +	test_cmp expect actual &&
> +	(
> +		cd src &&
> +		git push --force --force-with-lease --all 2>err &&
> +		grep "forced update" err
> +	)
> +'
> +
>   test_done
> diff --git a/transport-helper.c b/transport-helper.c
> index c52c99d829..75b4c1b758 100644
> --- a/transport-helper.c
> +++ b/transport-helper.c
> @@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,
>   			status = REF_STATUS_REJECT_STALE;
>   			FREE_AND_NULL(msg);
>   		}
> +		else if (!strcmp(msg, "ignored fetch")) {
> +			status = REF_STATUS_REJECT_IMPLICIT_FETCH;
> +			FREE_AND_NULL(msg);
> +		}
>   		else if (!strcmp(msg, "forced update")) {
>   			forced = 1;
>   			FREE_AND_NULL(msg);
> @@ -896,6 +900,7 @@ static int push_refs_with_push(struct transport *transport,
>   		switch (ref->status) {
>   		case REF_STATUS_REJECT_NONFASTFORWARD:
>   		case REF_STATUS_REJECT_STALE:
> +		case REF_STATUS_REJECT_IMPLICIT_FETCH:
>   		case REF_STATUS_REJECT_ALREADY_EXISTS:
>   			if (atomic) {
>   				reject_atomic_push(remote_refs, mirror);
> diff --git a/transport.c b/transport.c
> index 43e24bf1e5..588575498f 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -567,6 +567,10 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,
>   		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
>   				 "stale info", porcelain, summary_width);
>   		break;
> +	case REF_STATUS_REJECT_IMPLICIT_FETCH:
> +		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
> +				 "implicit fetch", porcelain, summary_width);
> +		break;
>   	case REF_STATUS_REJECT_SHALLOW:
>   		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
>   				 "new shallow roots not allowed",
> @@ -1101,6 +1105,7 @@ static int run_pre_push_hook(struct transport *transport,
>   		if (!r->peer_ref) continue;
>   		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
>   		if (r->status == REF_STATUS_REJECT_STALE) continue;
> +		if (r->status == REF_STATUS_REJECT_IMPLICIT_FETCH) continue;
>   		if (r->status == REF_STATUS_UPTODATE) continue;
> 
>   		strbuf_reset(&buf);
> --
> 2.28.0
> 

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> The `--force-with-lease` option in `git-push`, makes sure that
> refs on remote aren't clobbered by unexpected changes when the
> "<expect>" ref value is explicitly specified.
>
> For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip
> of the remote tracking branch is populated as the "<expect>" value,
> there is a possibility of allowing unwanted overwrites on the remote
> side when some tools that implicitly fetch remote-tracking refs in
> the background are used with the repository. If a remote-tracking ref
> was updated when a rewrite is happening locally and if those changes
> are pushed by omitting the "<expect>" value in `--force-with-lease`,
> any new changes from the updated tip will be lost locally and will
> be overwritten on the remote.

Hmph, I am not sure if we are on the same page as the problem with
the form of force-with-lease without <expect>.

In this sequence of end-user operation

    $ git checkout --detach origin/target
    ... edit working tree files ...
    $ git commit --amend -a
    $ git push origin +HEAD:target

the user wanted to fix the topmost commit at the 'target' branch at
the origin repository, and force-update it.

The --force-with-lease is a way to make sure that the only commit
being lost by the force-update is the commit the user wanted to
amend.  If other users pushed to the 'target' branch in the
meantime, the forced push at the last step will lose it.

    $ git checkout --detach origin/target
    $ TO_LOSE=$(git rev-parse HEAD)
    ... edit working tree files ...
    $ git commit --amend -a
    $ git push origin --force-with-lease=target:$TO_LOSE HEAD:target

So we say "I knew, when I started working on the replacement, I
started at the commit $TO_LOSE; please stop my forced push if the
tip of 'target' was moved by somebody else, away from $TO_LOSE".

The force-with-lease without the exact <expect> object name, i.e.

    $ git push origin --force-with-lease=target HEAD:target

would break if 'origin/target' was updated anytime between the time
when the first "git checkout --detach" step finishes and the time
the last "git push" is run, because 'origin/target' would be
different from $TO_LOSE and things that were pushed to 'origin/target'
by others in the meantime will be lost, in addition to $TO_LOSE commit
that the user is willing to discard and replace.

> This problem can be addressed by checking the `reflog` of the branch
> that is being pushed and verify if there in a entry with the remote
> tracking ref.

Sorry, but it is unclear how reflog would help.

Before the "git checkout" step in the example, there would have been
a "git fetch" from the origin that brought the remote-tracking
branch 'origin/target' to the current state with a reflog entry for
it.  If an automated background process makes another fetch while
the user is editing files in the working tree, such a fetch may also
add another reflog entry for that action.

Unless you make a snapshot of the reflog state immediately after you
do "git checkout" in the example, you wouldn't know if there were
unexpected updates to the remote-tracking branch even if you check
the reflog.

Besides, you do not control the parenthood relationship between the
commits _other_ people push and update to 'target' branch at the
'origin' repository, so you cannot rely on the topology among them
to make any decision.  Other people may be force pushing to the
branch while you are preparing the commit to replace $TO_LOSE by
force pushing.

> +	The check ensures that the commit at the tip of the remote-tracking
> +	branch -- which may have been implicitly updated by tools that fetch
> +	remote refs by running linkgit:git-fetch[1] in the background -- has
> +	been integrated locally, when holding the "lease".

The problem with "expect-less" form is that it does not hold the
lease at all.  The point of "hold the lease" by giving an explicit
$TO_LOSE is to force a push that does not fast-forward, so if you
iterate over the remote-tracking branch at the time of "push", if
you find more commits built on top of $TO_LOSE because a background
fetch updated the branch, or if you find NO commits on top of $TO_LOSE
because no background fetch happened in the meantime, what would be
pushed would not fast-forward.  So I am not sure what the point of
iterating over reflog.

If we want an option to force a safer behaviour, I think adding a
configuration variable to forbid the form of force-with-lease
without <expect> would be one way to help.  Perhaps make it the
default, while allowing those who want to live dangerously to
explicitly turn it off.

Thanks.

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Johannes Schindelin]

Hi Srinidhi,

On Sat, 5 Sep 2020, Srinidhi Kaushik wrote:

> The `--force-with-lease` option in `git-push`, makes sure that
> refs on remote aren't clobbered by unexpected changes when the
> "<expect>" ref value is explicitly specified.
>
> For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip
> of the remote tracking branch is populated as the "<expect>" value,
> there is a possibility of allowing unwanted overwrites on the remote
> side when some tools that implicitly fetch remote-tracking refs in
> the background are used with the repository. If a remote-tracking ref
> was updated when a rewrite is happening locally and if those changes
> are pushed by omitting the "<expect>" value in `--force-with-lease`,
> any new changes from the updated tip will be lost locally and will
> be overwritten on the remote.
>
> This problem can be addressed by checking the `reflog` of the branch
> that is being pushed and verify if there in a entry with the remote
> tracking ref. By running this check, we can ensure that refs being
> are fetched in the background while a "lease" is being held are not
> overlooked before a push, and any new changes can be acknowledged
> and (if necessary) integrated locally.
>
> The new check will cause `git-push` to fail if it detects the presence
> of any updated refs that we do not have locally and reject the push
> stating `implicit fetch` as the reason.
>
> An experimental configuration setting: `push.rejectImplicitFetch`
> which defaults to `true` (when `features.experimental` is enabled)
> has been added, to allow `git-push` to reject a push if the check
> fails.
>
> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
> ---
>
> Hello,
> I picked this up from #leftoverbits over at GitHub [1] from the open
> issues list. This idea [2], for a safer `--force-with-lease` was
> originally proposed by Johannes on the mailing list.
>
> [1]: https://github.com/gitgitgadget/git/issues/640
> [2]: https://lore.kernel.org/git/nycvar.QRO.7.76.6.1808272306271.73@tvgsbejvaqbjf.bet/

First of all: thank you for picking this up! The contribution is
pleasantly well-written, thank you also for that.

Now, to be honest, I thought that this mode would merit a new option
rather than piggy-backing on top of `--force-with-lease`. The reason is
that `--force-with-lease` targets a slightly different use case than mine:
it makes sure that we do not overwrite remote refs unless we already had a
chance to inspect them.

In contrast, my workflow uses `git pull --rebase` in two or more separate
worktrees, e.g. when developing a patch on two different Operating
Systems, I frequently forget to pull (to my public repository) on one
side, and I want to avoid force-pushing in that case, even if VS Code (or
I, via `git remote update`) fetched the ref (but failing to rebase the
local branch on top of it).

However, in other scenarios I very much do _not_ want to incorporate the
remote ref. For example, I often fetch
https://github.com/git-for-windows/git.wiki.git to check for the
occasional bogus change. Whenever I see such a bogus change, and it is at
the tip of the branch, I want to force-push _without_ incorporating the
bogus change into the local branch, yet I _do_ want to use
`--force-with-lease` because an independent change could have come in via
the Wiki in the meantime.

So I think that the original `--force-with-lease` and the mode you
implemented target subtly different use cases that are both valid, and
therefore I would like to request a separate option for the latter.

However, I have to admit that I could not think of a good name for that
option. "Implicit fetch" seems a bit too vague here, because the local
branch was not fetched, and certainly not implicitly, yet the logic
revolves around the local branch having been rebased to the
remote-tracking ref at some stage.

Even if we went with the config option to modify `--force-with-lease`'s
behavior, I would recommend separating out the `feature.experimental`
changes into their own patch, so that they can be reverted easily in case
the experimental feature is made the default.

A couple more comments:

> @@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
>  		 * If the remote ref has moved and is now different
>  		 * from what we expect, reject any push.
>  		 *
> -		 * It also is an error if the user told us to check
> -		 * with the remote-tracking branch to find the value
> -		 * to expect, but we did not have such a tracking
> -		 * branch.
> +		 * It also is an error if the user told us to check with the
> +		 * remote-tracking branch to find the value to expect, but we
> +		 * did not have such a tracking branch, or we have one that
> +		 * has new changes.

If I were you, I would try to keep the original formatting, so that it
becomes more obvious that the part ", or we have [...]" was appended.

>  		if (ref->expect_old_sha1) {
>  			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
>  				reject_reason = REF_STATUS_REJECT_STALE;
> +			else if (reject_implicit_fetch() && ref->implicit_fetch)
> +				reject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;
>  			else
> -				/* If the ref isn't stale then force the update. */
> +				/*
> +				 * If the ref isn't stale, or there was no

Should this "or" not be an "and" instead?

> +				 * implicit fetch, force the update.
> +				 */
>  				force_ref_update = 1;
>  		}
> [...]
>  static void apply_cas(struct push_cas_option *cas,
>  		      struct remote *remote,
>  		      struct ref *ref)
>  {
> -	int i;
> +	int i, do_reflog_check = 0;
> +	struct object_id oid;
> +	struct ref *local_ref = get_local_ref(ref->name);
>
>  	/* Find an explicit --<option>=<name>[:<value>] entry */
>  	for (i = 0; i < cas->nr; i++) {
>  		struct push_cas *entry = &cas->entry[i];
>  		if (!refname_match(entry->refname, ref->name))
>  			continue;
> +
>  		ref->expect_old_sha1 = 1;
>  		if (!entry->use_tracking)
>  			oidcpy(&ref->old_oid_expect, &entry->expect);
>  		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
>  			oidclr(&ref->old_oid_expect);
> -		return;
> +		else
> +			do_reflog_check = 1;
> +
> +		goto reflog_check;

Hmm. I do not condemn `goto` statements in general, but this one makes the
flow harder to follow. I would prefer something like this:

-- snip --
 		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 			oidclr(&ref->old_oid_expect);
+		else if (local_ref && !read_ref(local_ref->name, &oid))
+			ref->implicit_fetch =
+				!remote_ref_in_reflog(&ref->old_oid, &oid,
+						      local_ref->name);
 		return;
-- snap --

Again, thank you so much for working on this!

Ciao,
Dscho

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Srinidhi Kaushik]

Hi Phillip,

On 09/07/2020 16:23, Phillip Wood wrote:
> [...]
> Thanks for working on this, making --force-with-lease safer would be a
> valuable contribution

:)

> > The `--force-with-lease` option in `git-push`, makes sure that
> > refs on remote aren't clobbered by unexpected changes when the
> > "<expect>" ref value is explicitly specified.
>
> I think it would help to write out
> `--force-with-lease[=<refname>[:<expect>]]` so readers know what
> "<expect>" is referring to

That makes sense; noted.

> > For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip
> > of the remote tracking branch is populated as the "<expect>" value,
> > there is a possibility of allowing unwanted overwrites on the remote
> > side when some tools that implicitly fetch remote-tracking refs in
> > the background are used with the repository. If a remote-tracking ref
> > was updated when a rewrite is happening locally and if those changes
> > are pushed by omitting the "<expect>" value in `--force-with-lease`,
> > any new changes from the updated tip will be lost locally and will
> > be overwritten on the remote.
> >
> > This problem can be addressed by checking the `reflog` of the branch
> > that is being pushed and verify if there in a entry with the remote
> > tracking ref. By running this check, we can ensure that refs being
> > are fetched in the background while a "lease" is being held are not
> > overlooked before a push, and any new changes can be acknowledged
> > and (if necessary) integrated locally.

> An addition safety measure would be to check the reflog of the local
> commit and the tip of the remote tracking branch dates overlap.
> Otherwise if there is an implicit fetch of a remote head that has been
> rewound we still push the local branch when we shouldn't.

This sounds much better. My initial description of the check was perhaps
a bit confusing.

> > The new check will cause `git-push` to fail if it detects the presence
> > of any updated refs that we do not have locally and reject the push
> > stating `implicit fetch` as the reason.
>
> 'implicit fetch' is a rather terse message - can we say something along
> the lines of "the remote has been updated since the last merge/push"?

I was going by the "two-word" approach like "stale info", "fetch first",
"no match", and so on. But, I'll look into wording the reject reason
along those lines.

> > An experimental configuration setting: `push.rejectImplicitFetch`
> > which defaults to `true` (when `features.experimental` is enabled)
> > has been added, to allow `git-push` to reject a push if the check
> > fails.
>
> Making this available with features.experimental initially is probably a
> good idea, I hope it will become the default if in future versions.

I hope so. :)

> > [...]
> > +		case REF_STATUS_REJECT_IMPLICIT_FETCH:
> > +			res = "error";
> > +			msg = "implicit fetch";
> > +			break;
> > +
> >   		case REF_STATUS_REJECT_ALREADY_EXISTS:
> >   			res = "error";
> >   			msg = "already exists";
> > diff --git a/remote.c b/remote.c
> > index c5ed74f91c..ee2dedd15b 100644
> > --- a/remote.c
> > +++ b/remote.c
> > @@ -49,6 +49,8 @@ static const char *pushremote_name;
> >   static struct rewrites rewrites;
> >   static struct rewrites rewrites_push;
> >
> > +static struct object_id cas_reflog_check_oid;
> > +
>
> rather than using a global variable I think it would be better just to
> pass this value around using the cb_data argument of the reflog callback
> function

I have to admit that I was hesitant to use the global variable when
writing this. For some reason I thought the callback data was used to
store results from the called function only and not pass arguments.
Will fix that in v2.

> > [...]
> > +static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,
> > +			     const char *ident, timestamp_t timestamp, int tz,
> > +			     const char *message, void *cb_data)
> > +{
>
> using the callback data we would have something like
>
> struct oid *remote_head = cb_data;
> return oideq(noid, remote_head);

Got it; this and the callback argument to the reflog entry function
will be updated accordingly.

> > [...]
> > +{
> > +	int ret = 0;
> > +	cas_reflog_check_oid = *r_oid;
> > +
> > +	struct commit *r_commit, *l_commit;
>
> Our coding style is to declare all variables before any statements, so
> this should come above `cas_reflog_check_oid = *r_oid` but that line
> wants to go away anyway.

Noted; `cas_reflog_check_oid` will go away as suggested above.

> > [...]
> > +	ret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;
> > +	if (ret)
> > +		goto skip;
>
> Rather than using a goto it would perhaps be better to do
>
> if (!ret)
>	ret = for_each_reflog_...
>
>

OK, yes. The `goto` can be avoided here.

> > +
> > +	ret = for_each_reflog_ent_reverse(local_ref_name,
> > +					  oid_in_reflog_ent,
> > +					  NULL);
>
> using the callback data we'd pass r_oid rather than NULL as the last
> argument


> > [...]
> >   		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
> >   			oidclr(&ref->old_oid_expect);
> > -		return;
> > +		else
> > +			do_reflog_check = 1;
> > +
> > +		goto reflog_check;
>
> I'm not too keen in jumping here, can't we just check `do_reflog_check`
> below?

Yes, of course. I was trying conserve the original flow of returning
from the function right after the loop. Since this is just one condition
to check if `do_reflog_check` is set to 1; we can get rid of the `goto`
and use a `break` instead.

> [...]

Thanks for a thorough review.
--
Srinidhi Kaushik

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Now, to be honest, I thought that this mode would merit a new option
> rather than piggy-backing on top of `--force-with-lease`. The reason is
> that `--force-with-lease` targets a slightly different use case than mine:
> it makes sure that we do not overwrite remote refs unless we already had a
> chance to inspect them.
>
> In contrast, my workflow uses `git pull --rebase` in two or more separate
> worktrees, e.g. when developing a patch on two different Operating
> Systems, I frequently forget to pull (to my public repository) on one
> side, and I want to avoid force-pushing in that case, even if VS Code (or
> I, via `git remote update`) fetched the ref (but failing to rebase the
> local branch on top of it).
>
> So I think that the original `--force-with-lease` and the mode you
> implemented target subtly different use cases that are both valid, and
> therefore I would like to request a separate option for the latter.

I tend to agree that that particular use case does not fit what the
"force with lease" option is meant to solve.  It should be a different
option, as you do not even want to be forcing.

But probably I am not getting your use case well enough to give a
good name suggestion.  "git push" without any form of "force" would
safely fail in your "I rebased on one side, and the other one is now
out of sync" situation already, so...

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Srinidhi Kaushik]

Hi Junio,
Thanks for taking the time to review this patch.

On 09/07/2020 09:14, Junio C Hamano wrote:
>> The `--force-with-lease` option in `git-push`, makes sure that
>> refs on remote aren't clobbered by unexpected changes when the
>> "<expect>" ref value is explicitly specified.
>>
>> For other cases (i.e., `--force-with-lease[=<ref>]`) where the tip
>> of the remote tracking branch is populated as the "<expect>" value,
>> there is a possibility of allowing unwanted overwrites on the remote
>> side when some tools that implicitly fetch remote-tracking refs in
>> the background are used with the repository. If a remote-tracking ref
>> was updated when a rewrite is happening locally and if those changes
>> are pushed by omitting the "<expect>" value in `--force-with-lease`,
>> any new changes from the updated tip will be lost locally and will
>> be overwritten on the remote.
>
> Hmph, I am not sure if we are on the same page as the problem with
> the form of force-with-lease without <expect>.
>
> In this sequence of end-user operation
>
>    $ git checkout --detach origin/target
>    ... edit working tree files ...
>    $ git commit --amend -a
>    $ git push origin +HEAD:target
>
> the user wanted to fix the topmost commit at the 'target' branch at
> the origin repository, and force-update it.
>
> The --force-with-lease is a way to make sure that the only commit
> being lost by the force-update is the commit the user wanted to
> amend.  If other users pushed to the 'target' branch in the
> meantime, the forced push at the last step will lose it.
>
>    $ git checkout --detach origin/target
>    $ TO_LOSE=$(git rev-parse HEAD)
>    ... edit working tree files ...
>    $ git commit --amend -a
>    $ git push origin --force-with-lease=target:$TO_LOSE HEAD:target
>
> So we say "I knew, when I started working on the replacement, I
> started at the commit $TO_LOSE; please stop my forced push if the
> tip of 'target' was moved by somebody else, away from $TO_LOSE".
>
> The force-with-lease without the exact <expect> object name, i.e.
>
>    $ git push origin --force-with-lease=target HEAD:target
>
> would break if 'origin/target' was updated anytime between the time
> when the first "git checkout --detach" step finishes and the time
> the last "git push" is run, because 'origin/target' would be
> different from $TO_LOSE and things that were pushed to 'origin/target'
> by others in the meantime will be lost, in addition to $TO_LOSE commit
> that the user is willing to discard and replace.

Sorry, that commit message should have been worded in a better way.
What I originally meant to say was: losing any _new_ changes that
were made on the remote during a rewrite, i.e., when we have a starting
point ($TO_LOSE) from the remote-tracking ref that we want to base our
rewrite on (after checkout) and then if the remote-tracking ref gets
updated by a push from another user, _and_ we don't use an "<expect>"
for `--force-with-lease`, there is a possibility that changes from the
other push would be lost because we're basing our rewrite on an older
version (?) of the remote-tracking ref, and our push would overwrite
its new changes because the value "<expect>" when omitted would point
to the updated tip of the remote instead. The condition:

	if (!oideq(&ref->old_oid, &ref->old_oid_expect))

would evaluate false since we're using `use_tracking`. This essentially
reduces the behavior of `--force-with-lease=<ref>` to `--force` in this
scenario.

>> This problem can be addressed by checking the `reflog` of the branch
>> that is being pushed and verify if there in a entry with the remote
>> tracking ref.
> Sorry, but it is unclear how reflog would help.
>
> Before the "git checkout" step in the example, there would have been
> a "git fetch" from the origin that brought the remote-tracking
> branch 'origin/target' to the current state with a reflog entry for
> it.  If an automated background process makes another fetch while
> the user is editing files in the working tree, such a fetch may also
> add another reflog entry for that action.
>
> Unless you make a snapshot of the reflog state immediately after you
> do "git checkout" in the example, you wouldn't know if there were
> unexpected updates to the remote-tracking branch even if you check
> the reflog.
>
> Besides, you do not control the parenthood relationship between the
> commits _other_ people push and update to 'target' branch at the
> 'origin' repository, so you cannot rely on the topology among them
> to make any decision.  Other people may be force pushing to the
> branch while you are preparing the commit to replace $TO_LOSE by
> force pushing.
>
>> +     The check ensures that the commit at the tip of the remote-tracking
>> +     branch -- which may have been implicitly updated by tools that fetch
>> +     remote refs by running linkgit:git-fetch[1] in the background -- has
>> +     been integrated locally, when holding the "lease".
>
> The problem with "expect-less" form is that it does not hold the
> lease at all.  The point of "hold the lease" by giving an explicit
> $TO_LOSE is to force a push that does not fast-forward, so if you
> iterate over the remote-tracking branch at the time of "push", if
> you find more commits built on top of $TO_LOSE because a background
> fetch updated the branch, or if you find NO commits on top of $TO_LOSE
> because no background fetch happened in the meantime, what would be
> pushed would not fast-forward.  So I am not sure what the point of
> iterating over reflog.

Right, I agree with what is described above. But, in this patch, we are
looking at the reflog of the _local_ branch that is going to be updated
on the remote side. The point of going through the reflog is to see if
the current tip its remote-tracking branch is present in one of the
reflog entries implying that any new changes (pushes from another user)
in the meantime aren't ignored and overwritten with our push.
Would that be an incorrect assumption?

> If we want an option to force a safer behaviour, I think adding a
> configuration variable to forbid the form of force-with-lease
> without <expect> would be one way to help.  Perhaps make it the
> default, while allowing those who want to live dangerously to
> explicitly turn it off.

Yes, that sounds like a good way to mitigate this issue; but that being
said, setups where `--force-with-lease` is being used as an alias for
`--force` should probably be taken into consideration though.

Thanks.
--
Srinidhi Kaushik

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Srinidhi Kaushik]

Hi Johannes,

On 09/07/2020 21:45, Johannes Schindelin wrote:
>> [...]
> First of all: thank you for picking this up! The contribution is
> pleasantly well-written, thank you also for that.

Thank you for the kind words.

> Now, to be honest, I thought that this mode would merit a new option
> rather than piggy-backing on top of `--force-with-lease`. The reason is
> that `--force-with-lease` targets a slightly different use case than mine:
> it makes sure that we do not overwrite remote refs unless we already had a
> chance to inspect them.
>
> In contrast, my workflow uses `git pull --rebase` in two or more separate
> worktrees, e.g. when developing a patch on two different Operating
> Systems, I frequently forget to pull (to my public repository) on one
> side, and I want to avoid force-pushing in that case, even if VS Code (or
> I, via `git remote update`) fetched the ref (but failing to rebase the
> local branch on top of it).
>
> However, in other scenarios I very much do _not_ want to incorporate the
> remote ref. For example, I often fetch
> https://github.com/git-for-windows/git.wiki.git to check for the
> occasional bogus change. Whenever I see such a bogus change, and it is at
> the tip of the branch, I want to force-push _without_ incorporating the
> bogus change into the local branch, yet I _do_ want to use
> `--force-with-lease` because an independent change could have come in via
> the Wiki in the meantime.

I realize that this new check would not be helpful if we deliberately
choose not to include an unwanted change from the updated remote's tip.
In that case, we would have to use `--force` make it work, and that
defeats the use of `--force-with-lease`.

> So I think that the original `--force-with-lease` and the mode you
> implemented target subtly different use cases that are both valid, and
> therefore I would like to request a separate option for the latter.

OK. So, I am assuming that you are suggesting to add a new function that
is separate from `apply_push_cas()` and run the check on each of the
remote refs. Would that be correct?

If that's the case, how does it work along with `--force-with-lease`?
On one hand we have `--force-with-lease` to ensure we rewrite the remote
that we have _already_ seen, and on the other, a new option that checks
reflog of the local branch to see if it is missing any updates from the
remote that may have happened in the meantime. If we pass both of them
for `push` and if the former doesn't complain, and the latter check
fails, should the `push` still go through?

I feel that this check included with `--force-with-lease` only when
the `use_tracking` or `use_tracking_for_rest` options are enabled
would give a heads-up the the user about the background fetch. If
they decide that they don't need new updates, then supplying the
new "<expect>" value in the next push would imply they've seen the
new update, and choose to overwrite it anyway. The check would not
run in this case. But again, I wonder if the this "two-step" process
makes `push` cumbersome.

> However, I have to admit that I could not think of a good name for that
> option. "Implicit fetch" seems a bit too vague here, because the local
> branch was not fetched, and certainly not implicitly, yet the logic
> revolves around the local branch having been rebased to the
> remote-tracking ref at some stage.

The message "implicit fetch" was in context of the remote ref. But yes,
the current reject reason is not clear and implies that local branch
was fetched, which isn't the case.

> Even if we went with the config option to modify `--force-with-lease`'s
> behavior, I would recommend separating out the `feature.experimental`
> changes into their own patch, so that they can be reverted easily in case
> the experimental feature is made the default.

Good idea!

> A couple more comments:
>
>> @@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
>>                * If the remote ref has moved and is now different
>>                * from what we expect, reject any push.
>>                *
>> -              * It also is an error if the user told us to check
>> -              * with the remote-tracking branch to find the value
>> -              * to expect, but we did not have such a tracking
>> -              * branch.
>> +              * It also is an error if the user told us to check with the
>> +              * remote-tracking branch to find the value to expect, but we
>> +              * did not have such a tracking branch, or we have one that
>> +              * has new changes.
>
> If I were you, I would try to keep the original formatting, so that it
> becomes more obvious that the part ", or we have [...]" was appended.

Alright, I will append the new comment in a new line instead.

>
>>               if (ref->expect_old_sha1) {
>>                       if (!oideq(&ref->old_oid, &ref->old_oid_expect))
>>                               reject_reason = REF_STATUS_REJECT_STALE;
>> +                     else if (reject_implicit_fetch() && ref->implicit_fetch)
>> +                             reject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;
>>                       else
>> -                             /* If the ref isn't stale then force the update. */
>> +                             /*
>> +                              * If the ref isn't stale, or there was no
>
> Should this "or" not be an "and" instead?

D'oh, you are right. It should have been an "and".

>
>> +                              * implicit fetch, force the update.
>> +                              */
>>                               force_ref_update = 1;
>>               }
>> [...]
>>  static void apply_cas(struct push_cas_option *cas,
>>                     struct remote *remote,
>>                     struct ref *ref)
>>  {
>> -     int i;
>> +     int i, do_reflog_check = 0;
>> +     struct object_id oid;
>> +     struct ref *local_ref = get_local_ref(ref->name);
>>
>>       /* Find an explicit --<option>=<name>[:<value>] entry */
>>       for (i = 0; i < cas->nr; i++) {
>>               struct push_cas *entry = &cas->entry[i];
>>               if (!refname_match(entry->refname, ref->name))
>>                       continue;
>> +
>>               ref->expect_old_sha1 = 1;
>>               if (!entry->use_tracking)
>>                       oidcpy(&ref->old_oid_expect, &entry->expect);
>>               else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
>>                       oidclr(&ref->old_oid_expect);
>> -             return;
>> +             else
>> +                     do_reflog_check = 1;
>> +
>> +             goto reflog_check;
>
> Hmm. I do not condemn `goto` statements in general, but this one makes the
> flow harder to follow. I would prefer something like this:
>
> -- snip --
>               else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
>                       oidclr(&ref->old_oid_expect);
> +             else if (local_ref && !read_ref(local_ref->name, &oid))
> +                     ref->implicit_fetch =
> +                             !remote_ref_in_reflog(&ref->old_oid, &oid,
> +                                                   local_ref->name);
>               return;
> -- snap --

Adding this condition looks cleaner instead of the `goto`. A similar
suggestion was made in the other thread [1] as well; this will be
addressed in v2.

> Again, thank you so much for working on this!

Thanks again, for taking the time to review this.

[1]: https://public-inbox.org/git/624d9e35-29b8-4012-a3d6-e9b00a9e4485@gmail.com/
--
Srinidhi Kaushik

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Now, to be honest, I thought that this mode would merit a new option
> rather than piggy-backing on top of `--force-with-lease`. The reason is
> that `--force-with-lease` targets a slightly different use case than mine:
> it makes sure that we do not overwrite remote refs unless we already had a
> chance to inspect them.
>
> In contrast, my workflow uses `git pull --rebase` in two or more separate
> worktrees, e.g. when developing a patch on two different Operating
> Systems, I frequently forget to pull (to my public repository) on one
> side, and I want to avoid force-pushing in that case, even if VS Code (or
> I, via `git remote update`) fetched the ref (but failing to rebase the
> local branch on top of it).
> ...
> So I think that the original `--force-with-lease` and the mode you
> implemented target subtly different use cases that are both valid, and
> therefore I would like to request a separate option for the latter.

I agree that the use case in the second paragraph above does not fit
what the "force with lease" option is meant to solve.  You do not
even want to be forcing in the workflow so "--force-with-anything"
is a bad name for the mode of operation, if I am reading you right.

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> to the updated tip of the remote instead. The condition:
>
> 	if (!oideq(&ref->old_oid, &ref->old_oid_expect))
>
> would evaluate false since we're using `use_tracking`. This essentially
> reduces the behavior of `--force-with-lease=<ref>` to `--force` in this
> scenario.

Yes, that is exactly why I have kept saying that the form without
<expect> is not safe and cannot be made safer especially when
auto-fetching is involved.

> Right, I agree with what is described above. But, in this patch, we are
> looking at the reflog of the _local_ branch that is going to be updated
> on the remote side. The point of going through the reflog is to see if
> the current tip its remote-tracking branch is present in one of the
> reflog entries implying that any new changes (pushes from another user)
> in the meantime aren't ignored and overwritten with our push.
> Would that be an incorrect assumption?

I am afraid it is.  You may have looked at it and even kept it
locally, to be looked at later, without letting it affect what you
are going ot force-push in any way.  You might later come back and
resurrect their change on top of what you force pushed, or more
likely you may simply forget, especially if the forced push goes
through without failing.

> Yes, that sounds like a good way to mitigate this issue; but that being
> said, setups where `--force-with-lease` is being used as an alias for
> `--force` should probably be taken into consideration though.

We cannot help, and it is not our job to dispel, misconfigurations
and misconceptions caused by following bad pieces of advice other
people gave our users, though.

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Johannes Schindelin]

Hi Junio,

On Tue, 8 Sep 2020, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> > Now, to be honest, I thought that this mode would merit a new option
> > rather than piggy-backing on top of `--force-with-lease`. The reason is
> > that `--force-with-lease` targets a slightly different use case than mine:
> > it makes sure that we do not overwrite remote refs unless we already had a
> > chance to inspect them.
> >
> > In contrast, my workflow uses `git pull --rebase` in two or more separate
> > worktrees, e.g. when developing a patch on two different Operating
> > Systems, I frequently forget to pull (to my public repository) on one
> > side, and I want to avoid force-pushing in that case, even if VS Code (or
> > I, via `git remote update`) fetched the ref (but failing to rebase the
> > local branch on top of it).
> >
> > So I think that the original `--force-with-lease` and the mode you
> > implemented target subtly different use cases that are both valid, and
> > therefore I would like to request a separate option for the latter.
>
> I tend to agree that that particular use case does not fit what the
> "force with lease" option is meant to solve.  It should be a different
> option, as you do not even want to be forcing.

Actually, I _do_ want to be forcing, but _only_ if the recorded remote tip
has been integrated into the local branch at some point (even if it has
been rebased or amended away).

And even then, I want to only force when the recorded remote tip still
matches the actual remote tip, i.e. I want to force "with lease".

> But probably I am not getting your use case well enough to give a
> good name suggestion.  "git push" without any form of "force" would
> safely fail in your "I rebased on one side, and the other one is now
> out of sync" situation already, so...

Yes, it would safely fail, but it would also fail in situations where I do
_not_ want it to fail.

Imagine that you and I work together on a patch series, and we actually
share the same public repository. Let's say that we also want to allow for
some crossing emails that might be missed by the other when pushing our
changes.

Since we are working on a patch series, there will be a lot of amending
and rewording and rebasing going on.

So let's assume that you just did another round of amending and rewording
and want to push the branch. What you will want to avoid is to overwrite
commits that I might have just pushed. `--force-with-lease` to the rescue!

Since you did not fetch in between pulling and amending, that is safe.

But what if, maybe even by mistake, you called `git fetch` in between? And
then you _did_ get my updates, but only in the remote-tracking branch (the
"lease"), yet those updates never made it into the local branch. And since
`git push --force-with-lease` did not fail, you failed to notice that I
had made some edits, didn't `range-diff` and took care of amending your
local changes accordingly, and therefore you overwrote my changes.

That is the scenario this new mode wants to address.

Note that a similar scenario can occur if you work on a patch series on
Linux, then push it to your public repository, then try to make it work
on, say, an HP/UX machine you happen to have access to (maybe because your
employer needs you to make sure that this particular patch series solves a
particular problem on that particular machine), and you need to amend the
changes there. If you're very disciplined, you will only work on one
machine until you're done, then push, then fetch on the other machine.
However, if you're less disciplined (or "just need to check out something
real quick on the other machine"), this new mode will come in quite handy.

Ciao,
Dscho

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Johannes Schindelin]

Hi Junio,

On Tue, 8 Sep 2020, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> > Now, to be honest, I thought that this mode would merit a new option
> > rather than piggy-backing on top of `--force-with-lease`. The reason is
> > that `--force-with-lease` targets a slightly different use case than mine:
> > it makes sure that we do not overwrite remote refs unless we already had a
> > chance to inspect them.
> >
> > In contrast, my workflow uses `git pull --rebase` in two or more separate
> > worktrees, e.g. when developing a patch on two different Operating
> > Systems, I frequently forget to pull (to my public repository) on one
> > side, and I want to avoid force-pushing in that case, even if VS Code (or
> > I, via `git remote update`) fetched the ref (but failing to rebase the
> > local branch on top of it).
> > ...
> > So I think that the original `--force-with-lease` and the mode you
> > implemented target subtly different use cases that are both valid, and
> > therefore I would like to request a separate option for the latter.
>
> I agree that the use case in the second paragraph above does not fit
> what the "force with lease" option is meant to solve.  You do not
> even want to be forcing in the workflow so "--force-with-anything"
> is a bad name for the mode of operation, if I am reading you right.

No, you _have_ to force the push.

If you don't have to force the push, i.e. if it is a fast-forward, there
absolutely is no need for any of this.

In contrast, when you want to make sure that you _actually_ incorporated
the revision that is currently the remote tip, e.g. via `git pull
--rebase` with a possible additional rebase on top that makes this _not_ a
fast-forward, you totally have to force the push, otherwise it won't work.

Ciao,
Dscho

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Johannes Schindelin]

Hi,

On Wed, 9 Sep 2020, Johannes Schindelin wrote:

> On Tue, 8 Sep 2020, Junio C Hamano wrote:
>
> > Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> >
> > > Now, to be honest, I thought that this mode would merit a new option
> > > rather than piggy-backing on top of `--force-with-lease`. The reason is
> > > that `--force-with-lease` targets a slightly different use case than mine:
> > > it makes sure that we do not overwrite remote refs unless we already had a
> > > chance to inspect them.
> > >
> > > In contrast, my workflow uses `git pull --rebase` in two or more separate
> > > worktrees, e.g. when developing a patch on two different Operating
> > > Systems, I frequently forget to pull (to my public repository) on one
> > > side, and I want to avoid force-pushing in that case, even if VS Code (or
> > > I, via `git remote update`) fetched the ref (but failing to rebase the
> > > local branch on top of it).
> > > ...
> > > So I think that the original `--force-with-lease` and the mode you
> > > implemented target subtly different use cases that are both valid, and
> > > therefore I would like to request a separate option for the latter.
> >
> > I agree that the use case in the second paragraph above does not fit
> > what the "force with lease" option is meant to solve.  You do not
> > even want to be forcing in the workflow so "--force-with-anything"
> > is a bad name for the mode of operation, if I am reading you right.
>
> No, you _have_ to force the push.
>
> If you don't have to force the push, i.e. if it is a fast-forward, there
> absolutely is no need for any of this.
>
> In contrast, when you want to make sure that you _actually_ incorporated
> the revision that is currently the remote tip, e.g. via `git pull
> --rebase` with a possible additional rebase on top that makes this _not_ a
> fast-forward, you totally have to force the push, otherwise it won't work.

Maybe `--force-if-incorporated`? Originally, I had in mind to call it
`--safe-force`, but that might be too vague.

BTW I think the patch needs to cover a bit more, still: after I run `git
pull --rebase`, the local branch will never have been at the same revision
as the fetched one: `git rebase` moves to an unnamed branch before
replaying the patches. So I think we need to see whether the remote tip
was _reachable_ from (not necessarily identical to) any of the reflog's
revisions.

Ciao,
Dscho

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Srinidhi Kaushik]

Hello,

On 09/10/2020 12:22, Johannes Schindelin wrote:
> Hi
>> [...] 
> Maybe `--force-if-incorporated`? Originally, I had in mind to call it
> `--safe-force`, but that might be too vague.

That's nice. I haven't been able to come up with a good name for this
option. So far, I have: `--check-updated-remote-refs` which is really
long and is probably confusing.
 
> BTW I think the patch needs to cover a bit more, still: after I run `git
> pull --rebase`, the local branch will never have been at the same revision
> as the fetched one: `git rebase` moves to an unnamed branch before
> replaying the patches. So I think we need to see whether the remote tip
> was _reachable_ from (not necessarily identical to) any of the reflog's
> revisions.

Good catch. Would adding in_merge_bases() along with checking if OIDs
are equal for each reflog entry in oid_in_reflog_ent() address the
problem? That way, we would check if remote ref is reachable from
one of the entries?

Thanks.

-- >8 --
+ static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,
+			     const char *ident, timestamp_t timestamp, int tz,
+			     const char *message, void *cb_data)
+ {
+	struct object_id *remote_oid = cb_data;
+	struct commit *a = lookup_commit_reference(the_repository, noid);
+	struct commit *b = lookup_commit_reference(the_repository, remote_oid);
+	return oideq(noid, remote_oid) || in_merge_bases(b, a);
+ }

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

>> In contrast, when you want to make sure that you _actually_ incorporated
>> the revision that is currently the remote tip, e.g. via `git pull
>> --rebase` with a possible additional rebase on top that makes this _not_ a
>> fast-forward, you totally have to force the push, otherwise it won't work.
>
> Maybe `--force-if-incorporated`? Originally, I had in mind to call it
> `--safe-force`, but that might be too vague.

Yup.  "safe force" indeed feels like a misnomer.  The assumption of
safety relies heavily on the workflow.

I might even say --force-if-merged even if the way the to-be-lost
changes have become part of what you are pushing out is not
technically a merge, but there may be shorter and sweeter way to
express it than 'merge' and 'incorporate'.

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Johannes Schindelin]

Hi Srinidhi,

On Thu, 10 Sep 2020, Srinidhi Kaushik wrote:

> On 09/10/2020 12:22, Johannes Schindelin wrote:
>
> > BTW I think the patch needs to cover a bit more, still: after I run `git
> > pull --rebase`, the local branch will never have been at the same revision
> > as the fetched one: `git rebase` moves to an unnamed branch before
> > replaying the patches. So I think we need to see whether the remote tip
> > was _reachable_ from (not necessarily identical to) any of the reflog's
> > revisions.
>
> Good catch. Would adding in_merge_bases() along with checking if OIDs
> are equal for each reflog entry in oid_in_reflog_ent() address the
> problem? That way, we would check if remote ref is reachable from
> one of the entries?
>
> Thanks.
>
> -- >8 --
> + static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,
> +			     const char *ident, timestamp_t timestamp, int tz,
> +			     const char *message, void *cb_data)
> + {
> +	struct object_id *remote_oid = cb_data;
> +	struct commit *a = lookup_commit_reference(the_repository, noid);
> +	struct commit *b = lookup_commit_reference(the_repository, remote_oid);
> +	return oideq(noid, remote_oid) || in_merge_bases(b, a);
> + }

Since `in_merge_bases()` is quite a bit more expensive than `oideq()`,
personally, I would actually walk the reflog with the `oideq()` check
first (stopping early in case a match was found), and only fall back to
looking for a merge base in the reflog if the first reflog walk did not
find a match.

Ciao,
Dscho

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Johannes Schindelin]

Hi Junio,

On Thu, 10 Sep 2020, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> >> In contrast, when you want to make sure that you _actually_ incorporated
> >> the revision that is currently the remote tip, e.g. via `git pull
> >> --rebase` with a possible additional rebase on top that makes this _not_ a
> >> fast-forward, you totally have to force the push, otherwise it won't work.
> >
> > Maybe `--force-if-incorporated`? Originally, I had in mind to call it
> > `--safe-force`, but that might be too vague.
>
> Yup.  "safe force" indeed feels like a misnomer.  The assumption of
> safety relies heavily on the workflow.
>
> I might even say --force-if-merged even if the way the to-be-lost
> changes have become part of what you are pushing out is not
> technically a merge, but there may be shorter and sweeter way to
> express it than 'merge' and 'incorporate'.

You're right, `--force-if-merged` is a much better way to put it.

Thanks,
Dscho

[PATCH v2 0/2] push: make "--force-with-lease" safer

[Srinidhi Kaushik]

The `--force-with-lease[=<refname>[:<expect]]` option in `git-push`
makes sure that refs on remote aren't clobbered by unexpected changes
when the "<refname>" and "<expect>" ref values are explicitly specified.

For setups where the remote-tracking refs are implicitly updated by
tools that run in the background, not specifying an "<expect>" value
for force updates may result in loss of remote updates.

Let's say, we have a local branch that is based on a remote ref
that may be updated implicitly with a background fetch. If we decide
to rewrite changes on the remote, and base our local branch on the
current tip of the remote-tracking ref. If the remote ref was updated
by a push from another user, and it was fetched -- right after the
checkout, during rewrite or before push -- in the background, and if
we decide to force update our rewritten local changes on the remote
with `--force-with-lease[=<refname>]` (i.e, without specifying an
"<expect>" value) to `git-push`, the remote changes pushed from
another user during our rewrite may be lost.

The new option `--force-if-includes` will allow forcing an update only
if the tip of the remote-tracking ref has been integrated locally.
Using this along with `--force-with-lease`, during the time of push
can help preventing unintended remote overwrites.

Srinidhi Kaushik (2):
  push: add "--[no-]force-if-includes"
  push: enable "forceIfIncludesWithLease" by default

 Documentation/config/advice.txt   |   4 +
 Documentation/config/feature.txt  |   6 ++
 Documentation/config/push.txt     |   8 ++
 Documentation/git-push.txt        |  22 +++++
 advice.c                          |   3 +
 advice.h                          |   2 +
 builtin/push.c                    |  38 +++++++-
 builtin/send-pack.c               |  13 ++-
 remote.c                          | 129 ++++++++++++++++++++++++---
 remote.h                          |  14 ++-
 send-pack.c                       |   1 +
 t/t5533-push-cas.sh               |  53 ++++++++++++
 t/t5549-push-force-if-includes.sh | 139 ++++++++++++++++++++++++++++++
 transport-helper.c                |   5 ++
 transport.c                       |  24 +++++-
 transport.h                       |  12 +--
 16 files changed, 451 insertions(+), 22 deletions(-)
 create mode 100755 t/t5549-push-force-if-includes.sh

base-commit: 54e85e7af1ac9e9a92888060d6811ae767fea1bc
--
2.28.0

[PATCH v2 1/2] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: `--force-if-includes` to `git-push` where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref on which a local branch has based on (for a rewrite), is reachable
from at least one of the `reflog` entries of the local branch about
to be updated by force on the remote.

This option can also be used with `--force-with-lease` in setups
where the remote-tracking refs of the repository are implicitly
updated in the background.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with `--force-with-lease[=<refname>[:expect]]` without specifying the
"<refname>" or "<expect>" values, can cause the update that happened
in-between the checkout and forced push to be lost.

Specifying `--force-with-includes` with `--force-with-lease` as an
ancillary argument at the time of push, ensures that any new updates
to the remote-tracking refs are integrated locally before allowing a
forced update. This behavior can enabled by default if the configuration
option `push.forceIfIncludesWithLease` is set to `true`.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt   |   4 +
 Documentation/config/push.txt     |   8 ++
 Documentation/git-push.txt        |  22 +++++
 advice.c                          |   3 +
 advice.h                          |   2 +
 builtin/push.c                    |  27 +++++-
 builtin/send-pack.c               |  13 ++-
 remote.c                          | 129 ++++++++++++++++++++++++---
 remote.h                          |  14 ++-
 send-pack.c                       |   1 +
 t/t5533-push-cas.sh               |  53 ++++++++++++
 t/t5549-push-force-if-includes.sh | 139 ++++++++++++++++++++++++++++++
 transport-helper.c                |   5 ++
 transport.c                       |  24 +++++-
 transport.h                       |  12 +--
 15 files changed, 434 insertions(+), 22 deletions(-)
 create mode 100755 t/t5549-push-force-if-includes.sh

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..f48aed49e8 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -41,6 +41,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..1b4948faa0 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,11 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.forceIfIncludesWithLease::
+	If set to `true`, adds `--force-if-includes` as an ancillary argument
+	to `--force-with-lease[=<refname>[:<expect>]]`, when "<refname>" or
+	"<expect>" values are unspecified at the time of push.
++
+Note: Specifying `--no-force-if-includes` to linkgit:git-push[1] as an
+argument during the time of push does _not_ override this configuration.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..199c601bd4 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -320,6 +320,15 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying `--force-if-includes` an an ancillary option along
+with `--force-with-lease[=<refname>[:expect]]` (when "<refname>" or "<expect>"
+values are unspecified) at the time of `push` will verify if updates from
+the remote-tracking refs that may have been implicitly updated in the
+background (via linkgit:git-fetch[1], and the like) are integrated locally
+before allowing a forced update. This behavior can be enabled by default if
+the configuration option `push.forceIfIncludesWithLease` to `true`
+in linkgit:git-config[1].
 
 -f::
 --force::
@@ -341,6 +350,19 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option verifies if the tip of the remote-tracking ref on which
+a local branch has based on (for a rewrite), is reachable from at
+least one of the `reflog` entries of the local branch about to be
+updated by force on the remote. The check ensures that any updates
+from the remote have been incorporated locally by rejecting a push
+if that is not the case.
++
+Specifying `--no-force-if-includes` disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index bc94078e72..7fb07eb38e 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -300,6 +300,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git rebase ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -335,6 +341,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -384,8 +397,9 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
-
 	return 1;
 }
 
@@ -520,8 +534,14 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		if (!v)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
-	}
+	} else if (!strcmp(k, "push.forceifincludeswithlease")) {
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
 
+		return 0;
+	}
 	return git_default_config(k, v, NULL);
 }
 
@@ -551,6 +571,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, "force-if-includes", &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 2b9610f121..ee5d7af00c 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
@@ -155,6 +160,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
 	struct packet_reader reader;
+	unsigned int force_if_inc;
 
 	struct option options[] = {
 		OPT__VERBOSITY(&verbose),
@@ -179,6 +185,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 		  N_("require old value of ref to be at this value"),
 		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
+		OPT_BOOL(0, "force-if-includes", &force_if_inc,
+			 N_("require remote updates to be integrated locally")),
 		OPT_END()
 	};
 
@@ -278,7 +286,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		return -1;
 
 	if (!is_empty_cas(&cas))
-		apply_push_cas(&cas, remote, remote_refs);
+		apply_push_cas(&cas, remote, remote_refs, force_if_inc);
+
+	if (is_empty_cas(&cas) && force_if_inc)
+		run_local_reflog_check(remote_refs);
 
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
diff --git a/remote.c b/remote.c
index 420150837b..71af6d3073 100644
--- a/remote.c
+++ b/remote.c
@@ -1484,6 +1484,36 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 				force_ref_update = 1;
 		}
 
+		/*
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
+		 *
+		 * There is no need to check for reachability, if the
+		 * ref is marked for deletion.
+		 */
+		if (ref->if_includes && !ref->deletion) {
+			/*
+			 * If `force_ref_update' was previously set by
+			 * "compare-and-swap", and we have to run this
+			 * check, reset it back to the original value
+			 * and update it depending on the status of this
+			 * check.
+			 */
+			force_ref_update = ref->force || force_update;
+
+			if (ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
+			else
+				/*
+				 * If updates from the remote-tracking ref
+				 * have been integrated locally; force the
+				 * update.
+				 */
+				force_ref_update = 1;
+		}
+
 		/*
 		 * If the update isn't already rejected then check
 		 * the usual "must fast-forward" rules.
@@ -2272,11 +2302,76 @@ static int remote_tracking(struct remote *remote, const char *refname,
 	return 0;
 }
 
+static int ref_reachable(struct object_id *o_oid, struct object_id *n_oid,
+			 const char *ident, timestamp_t timestamp, int tz,
+			 const char *message, void *cb_data)
+{
+	int ret = 0;
+	struct object_id *r_oid = cb_data;
+
+	ret = oideq(n_oid, r_oid);
+	if (!ret) {
+		struct commit *loc = lookup_commit_reference(the_repository,
+							     n_oid);
+		struct commit *rem = lookup_commit_reference(the_repository,
+							     r_oid);
+		ret = (loc && rem) ? in_merge_bases(rem, loc) : 0;
+	}
+
+	return ret;
+}
+
+/*
+ * Iterate through the reflog of a local branch and check
+ * if the tip of the remote-tracking branch is reachable
+ * from one of the entries.
+ */
+static int ref_reachable_from_reflog(const struct object_id *r_oid,
+				     const struct object_id *l_oid,
+				     const char *local_ref_name)
+{
+	int ret = 0;
+	struct commit *r_commit, *l_commit;
+
+	l_commit = lookup_commit_reference(the_repository, l_oid);
+	r_commit = lookup_commit_reference(the_repository, r_oid);
+
+	/*
+	 * If the remote-tracking ref is an ancestor of the local
+	 * ref (a merge, for instance) there is no need to iterate
+	 * through the reflog entries to ensure reachability; it
+	 * can be skipped to return early instead.
+	 */
+	ret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;
+	if (!ret)
+		ret = for_each_reflog_ent_reverse(local_ref_name, ref_reachable,
+						  (struct object_id *)r_oid);
+
+	return ret;
+}
+
+/*
+ * Check for reachability of a remote-tracking ref in its local
+ * ref's reflog entries.
+ */
+void check_reflog_for_ref(struct ref *r_ref)
+{
+	struct ref *l_ref = get_local_ref(r_ref->name);
+	struct object_id r_oid;
+
+	r_ref->if_includes = 1;
+	if (l_ref && !read_ref(l_ref->name, &r_oid))
+		r_ref->unreachable = !ref_reachable_from_reflog(&r_ref->old_oid,
+								&r_oid,
+								l_ref->name);
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
-		      struct ref *ref)
+		      struct ref *ref,
+		      int if_includes)
 {
-	int i;
+	int i, check_reflog = 0;
 
 	/* Find an explicit --<option>=<name>[:<value>] entry */
 	for (i = 0; i < cas->nr; i++) {
@@ -2288,23 +2383,37 @@ static void apply_cas(struct push_cas_option *cas,
 			oidcpy(&ref->old_oid_expect, &entry->expect);
 		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 			oidclr(&ref->old_oid_expect);
-		return;
+		else
+			check_reflog = 1;
+		break;
 	}
 
 	/* Are we using "--<option>" to cover all? */
-	if (!cas->use_tracking_for_rest)
-		return;
+	if (cas->use_tracking_for_rest) {
+		ref->expect_old_sha1 = 1;
+		if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+			oidclr(&ref->old_oid_expect);
+		else
+			check_reflog = 1;
+	}
 
-	ref->expect_old_sha1 = 1;
-	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
-		oidclr(&ref->old_oid_expect);
+	if (if_includes && check_reflog)
+		check_reflog_for_ref(ref);
 }
 
 void apply_push_cas(struct push_cas_option *cas,
 		    struct remote *remote,
-		    struct ref *remote_refs)
+		    struct ref *remote_refs,
+		    int if_includes)
+{
+	struct ref *ref;
+	for (ref = remote_refs; ref; ref = ref->next)
+		apply_cas(cas, remote, ref, if_includes);
+}
+
+void run_local_reflog_check(struct ref *remote_refs)
 {
 	struct ref *ref;
 	for (ref = remote_refs; ref; ref = ref->next)
-		apply_cas(cas, remote, ref);
+		check_reflog_for_ref(ref);
 }
diff --git a/remote.h b/remote.h
index 5e3ea5a26d..c4a648ed6d 100644
--- a/remote.h
+++ b/remote.h
@@ -104,7 +104,9 @@ struct ref {
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		if_includes:1, /* For "--force-if-includes". */
+		unreachable:1; /* Used by "if_includes". */
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -134,6 +136,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -344,6 +347,13 @@ struct push_cas_option {
 int parseopt_push_cas_option(const struct option *, const char *arg, int unset);
 
 int is_empty_cas(const struct push_cas_option *);
-void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);
+void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *, int);
+
+/*
+ * Check if the remote-tracking ref was updated (since checkout)
+ * implicitly in the background and verify that changes from the
+ * updated tip have been integrated locally, before pushing.
+ */
+void run_local_reflog_check(struct ref*);
 
 #endif
diff --git a/send-pack.c b/send-pack.c
index 632f1580ca..956306e8e8 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..dee422acaa 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -256,4 +256,57 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates of REMOTE can be mitigated with "--force-if-includes"' '
+	rm -rf src dst &&
+	git init --bare src.bare &&
+	test_when_finished "rm -rf src.bare" &&
+	git clone --no-local src.bare dst &&
+	test_when_finished "rm -rf dst" &&
+	(
+		cd dst &&
+		test_commit G &&
+		git push origin master:master
+	) &&
+	git clone --no-local src.bare dst2 &&
+	test_when_finished "rm -rf dst2" &&
+	(
+		cd dst2 &&
+		test_commit H &&
+		git push
+	) &&
+	(
+		cd dst &&
+		test_commit I &&
+		git fetch origin &&
+		test_must_fail git push --force-with-lease --force-if-includes origin
+	)
+'
+
+test_expect_success 'background updates of REMOTE can be mitigated with "push.forceIfIncludesWithLease"' '
+	rm -rf src dst &&
+	git init --bare src.bare &&
+	test_when_finished "rm -rf src.bare" &&
+	git clone --no-local src.bare dst &&
+	test_when_finished "rm -rf dst" &&
+	(
+		cd dst &&
+		test_commit G &&
+		git push origin master:master
+	) &&
+	git clone --no-local src.bare dst2 &&
+	test_when_finished "rm -rf dst2" &&
+	(
+		cd dst2 &&
+		test_commit H &&
+		git push
+	) &&
+	(
+		cd dst &&
+		test_commit I &&
+		git fetch origin &&
+		git config --local push.forceIfIncludesWithLease true &&
+		test_must_fail git push --force-with-lease origin
+	)
+'
+
 test_done
diff --git a/t/t5549-push-force-if-includes.sh b/t/t5549-push-force-if-includes.sh
new file mode 100755
index 0000000000..2e2ca79266
--- /dev/null
+++ b/t/t5549-push-force-if-includes.sh
@@ -0,0 +1,139 @@
+test_description='Test push "--force-if-includes" forced update safety.'
+
+. ./test-lib.sh
+
+setup_src_dup_dst () {
+	rm -fr src dup dst &&
+	git init --bare dst &&
+	git clone --no-local dst src &&
+	git clone --no-local dst dup
+	(
+		cd src &&
+		test_commit A &&
+		git push
+	) &&
+	(
+		cd dup &&
+		git fetch &&
+		git merge origin/master &&
+		test_commit B &&
+		git switch -c branch master~1 &&
+		test_commit C &&
+		test_commit D &&
+		git push --all
+	) &&
+	(
+		cd src &&
+		git switch master &&
+		git fetch --all &&
+		git branch branch --track origin/branch &&
+		git rebase origin/master
+	) &&
+	(
+		cd dup &&
+		git switch master &&
+		test_commit E &&
+		git switch branch &&
+		test_commit F &&
+		git push origin --all
+	)
+}
+
+test_expect_success 'reject push if remote changes are not integrated locally (protected, all refs)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit H &&
+		git switch master &&
+		test_commit I &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes --all
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success 'reject push if remote changes are not integrated locally (protected, specific ref)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit H &&
+		git switch master &&
+		test_commit I &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes origin master
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success 'allow force push if "--force" is specified' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit H &&
+		git switch master &&
+		test_commit I &&
+		git fetch --all &&
+		git push --force --force-if-includes origin --all 2>err &&
+		grep "forced update" err
+	)
+'
+
+test_expect_success 'allow force push if "--delete" is specified' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit H &&
+		git switch master &&
+		test_commit I &&
+		git fetch --all &&
+		git push --delete --force-if-includes origin branch 2>err &&
+		grep "deleted" err
+	)
+'
+
+test_expect_success 'honor specified refspecs (force)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit H &&
+		git switch master &&
+		test_commit I &&
+		git fetch --all &&
+		git push --force-if-includes origin +branch 2>err &&
+		grep "forced update" err
+	)
+'
+
+test_expect_success 'honor specified refspecs (delete)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit H &&
+		git switch master &&
+		test_commit I &&
+		git fetch --all &&
+		git push --force-if-includes origin :branch 2>err &&
+		grep "deleted" err
+	)
+'
+
+test_done
diff --git a/transport-helper.c b/transport-helper.c
index c52c99d829..6fd74c2c65 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
diff --git a/transport.c b/transport.c
index 43e24bf1e5..3f9e697cd1 100644
--- a/transport.c
+++ b/transport.c
@@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "stale info", porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote updated since checkout",
+				 porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -659,6 +664,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
@@ -1101,6 +1108,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
@@ -1151,6 +1159,8 @@ int transport_push(struct repository *r,
 		int pretend = flags & TRANSPORT_PUSH_DRY_RUN;
 		int push_ret, ret, err;
 		struct strvec ref_prefixes = STRVEC_INIT;
+		int empty_cas = 1;
+		int if_includes = flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES;
 
 		if (check_push_refs(local_refs, rs) < 0)
 			return -1;
@@ -1178,9 +1188,19 @@ int transport_push(struct repository *r,
 
 		if (transport->smart_options &&
 		    transport->smart_options->cas &&
-		    !is_empty_cas(transport->smart_options->cas))
+		    !is_empty_cas(transport->smart_options->cas)) {
+			empty_cas = 0;
 			apply_push_cas(transport->smart_options->cas,
-				       transport->remote, remote_refs);
+				       transport->remote, remote_refs,
+				       if_includes);
+		}
+
+		/*
+		 * Run the check for all refs when "--force-if-includes"
+		 * is specified without "--force-with-lease".
+		 */
+		if (empty_cas && if_includes)
+			run_local_reflog_check(remote_refs);
 
 		set_ref_status_for_push(remote_refs,
 			flags & TRANSPORT_PUSH_MIRROR,
diff --git a/transport.h b/transport.h
index ca409ea1e4..729bb6979e 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -217,11 +218,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v2 2/2] push: enable "forceIfIncludesWithLease" by default

[Srinidhi Kaushik]

Set `push.forceIfIncludesWithLease` to `true` if experimental
features are enabled.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/feature.txt |  6 ++++++
 builtin/push.c                   | 11 +++++++++++
 2 files changed, 17 insertions(+)

diff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt
index c0cbf2bb1c..4a8a386132 100644
--- a/Documentation/config/feature.txt
+++ b/Documentation/config/feature.txt
@@ -18,6 +18,12 @@ skipping more commits at a time, reducing the number of round trips.
 * `protocol.version=2` speeds up fetches from repositories with many refs by
 allowing the client to specify which refs to list before the server lists
 them.
++
+* `push.forceIfIncludesWithLease=true` adds `--force-if-includes` as an
+additional option along with `--force-with-lease[=<refname>[:<expect>]`
+when "<refname>" or "<expect>" valus are unspecified for linkgit:git-push[1].
+Allows forced updates only if the local branch has incorporated changes
+from the remote-tracking ref.
 
 feature.manyFiles::
 	Enable config options that optimize for repos with many files in the
diff --git a/builtin/push.c b/builtin/push.c
index 7fb07eb38e..658694edff 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -540,6 +540,17 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		else
 			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
 
+		return 0;
+	} else if (!strcmp(k, "feature.experimental")) {
+		/*
+		 * Set `push.forceIfIncludesWithLease` to true,
+		 * if experimental features are enabled.
+		 */
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+
 		return 0;
 	}
 	return git_default_config(k, v, NULL);
-- 
2.28.0

Re: [PATCH v2 0/2] push: make "--force-with-lease" safer

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> The `--force-with-lease[=<refname>[:<expect]]` option in `git-push`
> makes sure that refs on remote aren't clobbered by unexpected changes
> when the "<refname>" and "<expect>" ref values are explicitly specified.

If you did a feature with different semantics to satisfy Dscho's
need, then this is no longer "make force-with-lease safer", I would
think.  Hopefully it is just the cover letter.

> The new option `--force-if-includes` will allow forcing an update only
> if the tip of the remote-tracking ref has been integrated locally.
> Using this along with `--force-with-lease`, during the time of push
> can help preventing unintended remote overwrites.

"if-includes" sounds quite sensible.  I think you want to lose the
word "lease" from the configuration variable name.  I do not think
it should be on by default, though.

Thanks.

Re: [PATCH v2 1/2] push: add "--[no-]force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Add a new option: `--force-if-includes` to `git-push` where forced
> updates are allowed only if the tip of the remote-tracking ref has
> been integrated locally, by verifying if the tip of the remote-tracking
> ref on which a local branch has based on (for a rewrite), is reachable
> from at least one of the `reflog` entries of the local branch about
> to be updated by force on the remote.
>
> This option can also be used with `--force-with-lease` in setups
> where the remote-tracking refs of the repository are implicitly
> updated in the background.
>
> If a local branch is based on a remote ref for a rewrite, and if that
> remote-tracking ref is updated by a push from another repository after
> it has been checked out locally, force updating that branch to remote
> with `--force-with-lease[=<refname>[:expect]]` without specifying the
> "<refname>" or "<expect>" values, can cause the update that happened
> in-between the checkout and forced push to be lost.
>
> Specifying `--force-with-includes` with `--force-with-lease` as an
> ancillary argument at the time of push, ensures that any new updates
> to the remote-tracking refs are integrated locally before allowing a
> forced update. This behavior can enabled by default if the configuration
> option `push.forceIfIncludesWithLease` is set to `true`.

This step seems to do too many things at once.  Split into logical
progression of improvements, or nobody can sensibly review it, I am
afraid.

> diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
> index f5e5b38c68..1b4948faa0 100644
> --- a/Documentation/config/push.txt
> +++ b/Documentation/config/push.txt
> @@ -114,3 +114,11 @@ push.recurseSubmodules::
>  	specifying '--recurse-submodules=check|on-demand|no'.
>  	If not set, 'no' is used by default, unless 'submodule.recurse' is
>  	set (in which case a 'true' value means 'on-demand').
> +
> +push.forceIfIncludesWithLease::
> +	If set to `true`, adds `--force-if-includes` as an ancillary argument
> +	to `--force-with-lease[=<refname>[:<expect>]]`, when "<refname>" or
> +	"<expect>" values are unspecified at the time of push.
> ++
> +Note: Specifying `--no-force-if-includes` to linkgit:git-push[1] as an
> +argument during the time of push does _not_ override this configuration.

I do not see why you still want to link these two unrelated
features.  I may want to do forced push with lease when I know I am
rewinding, and I may want to do a if-included force when needed, but
I do not see why I want to ask the former and implicitly see it also
trigger the latter.

I haven't seen the details, but if we severe the (unnecessary?)
entanglement of these two features, perhaps this patch will become a
lot smaller and more focused?  I dunno.

Re: [PATCH v2 2/2] push: enable "forceIfIncludesWithLease" by default

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Set `push.forceIfIncludesWithLease` to `true` if experimental
> features are enabled.

Why?  

We enable something to the guinea-pig audience if we consider that
it should become the default eventually for everybody.  I did not
see enough justification why this should be on for everybody
someday.

Re: [PATCH v2 0/2] push: make "--force-with-lease" safer

[Srinidhi Kaushik]

Hi Junio,

On 09/12/2020 11:15, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> > The `--force-with-lease[=<refname>[:<expect]]` option in `git-push`
> > makes sure that refs on remote aren't clobbered by unexpected changes
> > when the "<refname>" and "<expect>" ref values are explicitly specified.
> 
> If you did a feature with different semantics to satisfy Dscho's
> need, then this is no longer "make force-with-lease safer", I would
> think.  Hopefully it is just the cover letter.

Yes, this patch is about the new option, but I thought of keeping the
original reason for introducing it  in the cover letter for context.
I will add this as a note and change subjject cover letter in v3.
 
> > The new option `--force-if-includes` will allow forcing an update only
> > if the tip of the remote-tracking ref has been integrated locally.
> > Using this along with `--force-with-lease`, during the time of push
> > can help preventing unintended remote overwrites.
> 
> "if-includes" sounds quite sensible.  I think you want to lose the
> word "lease" from the configuration variable name.  I do not think
> it should be on by default, though.

Thanks; that makes sense. I am thinking of  just adding the option
as a command line argument without a configuration option. Will change
this in the next patch-set.

> [...]

Thanks again, for reviewing this.
-- 
Srinidhi Kaushik

Re: [PATCH v2 1/2] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

On 09/12/2020 11:20, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> > Add a new option: `--force-if-includes` to `git-push` where forced
> > updates are allowed only if the tip of the remote-tracking ref has
> > been integrated locally, by verifying if the tip of the remote-tracking
> > ref on which a local branch has based on (for a rewrite), is reachable
> > from at least one of the `reflog` entries of the local branch about
> > to be updated by force on the remote.
> >
> > This option can also be used with `--force-with-lease` in setups
> > where the remote-tracking refs of the repository are implicitly
> > updated in the background.
> >
> > If a local branch is based on a remote ref for a rewrite, and if that
> > remote-tracking ref is updated by a push from another repository after
> > it has been checked out locally, force updating that branch to remote
> > with `--force-with-lease[=<refname>[:expect]]` without specifying the
> > "<refname>" or "<expect>" values, can cause the update that happened
> > in-between the checkout and forced push to be lost.
> >
> > Specifying `--force-with-includes` with `--force-with-lease` as an
> > ancillary argument at the time of push, ensures that any new updates
> > to the remote-tracking refs are integrated locally before allowing a
> > forced update. This behavior can enabled by default if the configuration
> > option `push.forceIfIncludesWithLease` is set to `true`.
> 
> This step seems to do too many things at once.  Split into logical
> progression of improvements, or nobody can sensibly review it, I am
> afraid.

OK, I will break the commit up into smaller change-sets in the next
patch.
 
> > diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
> > index f5e5b38c68..1b4948faa0 100644
> > --- a/Documentation/config/push.txt
> > +++ b/Documentation/config/push.txt
> > @@ -114,3 +114,11 @@ push.recurseSubmodules::
> >  	specifying '--recurse-submodules=check|on-demand|no'.
> >  	If not set, 'no' is used by default, unless 'submodule.recurse' is
> >  	set (in which case a 'true' value means 'on-demand').
> > +
> > +push.forceIfIncludesWithLease::
> > +	If set to `true`, adds `--force-if-includes` as an ancillary argument
> > +	to `--force-with-lease[=<refname>[:<expect>]]`, when "<refname>" or
> > +	"<expect>" values are unspecified at the time of push.
> > ++
> > +Note: Specifying `--no-force-if-includes` to linkgit:git-push[1] as an
> > +argument during the time of push does _not_ override this configuration.
> 
> I do not see why you still want to link these two unrelated
> features.  I may want to do forced push with lease when I know I am
> rewinding, and I may want to do a if-included force when needed, but
> I do not see why I want to ask the former and implicitly see it also
> trigger the latter.

I am actually thinking of getting rid of the configuration option
altogether. Sadly, I realized that after I sent out the patch.

> I haven't seen the details, but if we severe the (unnecessary?)
> entanglement of these two features, perhaps this patch will become a
> lot smaller and more focused?  I dunno.

Other than the configuration setting (which will be removed), tthe only
parts that overlap are checking to see if "--force-with-lease" specified
with `use_tracking` and `use_tracking_for_test` modes only. The new
option does not modify the "strict" version of "compare-and-swap"
because in that case, it is not needed at all.

However, I think it can be cleaned up a bit more; will look into it.

Thanks.
-- 
Srinidhi Kaushik

[PATCH v3 0/7] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: "--force-if-includes" to `git-push` where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref -- on which a local branch has based on -- is reachable from at
least one of the "reflog" entries of the branch about to be updated
by force on the remote.

This option can also be used with `--force-with-lease` with setups
where the remote-tracking refs of the repository are implicitly
updated in the background to help prevent unintended remote
overwrites.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with `--force-with-lease[=<refname>[:expect]]` without specifying the
"<refname>" or "<expect>" values, can cause the update that happened
in-between the checkout and forced push to be lost.

Changes since v2:
  * Removed configuration option for setting "--force-if-includes"
    with "--force-with-lease".
  * Broke up the patch into smaller commits.

base-commit: 54e85e7af1ac9e9a92888060d6811ae767fea1bc

Srinidhi Kaushik (7):
  remote: add reflog check for "force-if-includes"
  transport: add flag for "--[no-]force-if-includes"
  send-pack: check ref status for "force-if-includes"
  transport-helper: update ref status for "force-if-includes"
  builtin/push: add option "--[no-]force-if-includes"
  doc: add reference for "--[no-]force-if-includes"
  t: add tests for "force-if-includes"

 Documentation/config/advice.txt   |   9 +-
 Documentation/git-push.txt        |  19 ++++
 advice.c                          |   3 +
 advice.h                          |   2 +
 builtin/push.c                    |  20 +++-
 builtin/send-pack.c               |   5 +
 remote.c                          | 135 +++++++++++++++++++++++--
 remote.h                          |  14 ++-
 send-pack.c                       |   1 +
 t/t5533-push-cas.sh               |  26 +++++
 t/t5549-push-force-if-includes.sh | 161 ++++++++++++++++++++++++++++++
 transport-helper.c                |   5 +
 transport.c                       |  18 +++-
 transport.h                       |  12 ++-
 14 files changed, 411 insertions(+), 19 deletions(-)
 create mode 100755 t/t5549-push-force-if-includes.sh

--
2.28.0

[PATCH v3 1/7] remote: add reflog check for "force-if-includes"

[Srinidhi Kaushik]

Add a check to verify if the remote-tracking ref of the local branch is
reachable from one of its "reflog" entries; `set_ref_status_for_push`
updated to add a reject reason and disallow the forced push if the
check fails.

When a local branch that is based on a remote ref, has been rewound and
is to be force pushed on the remote, "apply_push_force_if_includes()"
runs a check that ensure any updates to remote-tracking refs that may
have happened (by push from another repository) in-between the time of
the last checkout, and right before the time of push by rejecting the
forced update.

The struct "ref" has three new bit-fields:
  * if_includes: Set when we have to run the new check on the ref.
  * is_tracking: Set when the remote ref was marked as "use_tracking"
                 or "use_tracking_for_rest" by compare-and-swap.
  * unreachable: Set if the ref is unreachable from any of the "reflog"
                 entries of its local counterpart.

When "--force-with-includes" is used along with "--force-with-lease",
the check is run only for refs marked as "is_tracking".

The enum "status" updated to include "REF_STATUS_REJECT_REMOTE_UPDATED"
to imply that the ref failed the check.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 remote.c | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
 remote.h |  14 +++++-
 2 files changed, 141 insertions(+), 8 deletions(-)

diff --git a/remote.c b/remote.c
index 420150837b..e4b2d85a6f 100644
--- a/remote.c
+++ b/remote.c
@@ -1484,6 +1484,36 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 				force_ref_update = 1;
 		}
 
+		/*
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
+		 *
+		 * There is no need to check for reachability, if the
+		 * ref is marked for deletion.
+		 */
+		if (ref->if_includes && !ref->deletion) {
+			/*
+			 * If `force_ref_update' was previously set by
+			 * "compare-and-swap", and we have to run this
+			 * check, reset it back to the original value
+			 * and update it depending on the status of this
+			 * check.
+			 */
+			force_ref_update = ref->force || force_update;
+
+			if (ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
+			else
+				/*
+				 * If updates from the remote-tracking ref
+				 * have been integrated locally; force the
+				 * update.
+				 */
+				force_ref_update = 1;
+		}
+
 		/*
 		 * If the update isn't already rejected then check
 		 * the usual "must fast-forward" rules.
@@ -2272,11 +2302,74 @@ static int remote_tracking(struct remote *remote, const char *refname,
 	return 0;
 }
 
+static int ref_reachable(struct object_id *o_oid, struct object_id *n_oid,
+			 const char *ident, timestamp_t timestamp, int tz,
+			 const char *message, void *cb_data)
+{
+	int ret = 0;
+	struct object_id *r_oid = cb_data;
+
+	ret = oideq(n_oid, r_oid);
+	if (!ret) {
+		struct commit *loc = lookup_commit_reference(the_repository,
+							     n_oid);
+		struct commit *rem = lookup_commit_reference(the_repository,
+							     r_oid);
+		ret = (loc && rem) ? in_merge_bases(rem, loc) : 0;
+	}
+
+	return ret;
+}
+
+/*
+ * Iterate through the reflog of a local branch and check
+ * if the tip of the remote-tracking branch is reachable
+ * from one of the entries.
+ */
+static int ref_reachable_from_reflog(const struct object_id *r_oid,
+				     const struct object_id *l_oid,
+				     const char *local_ref_name)
+{
+	int ret = 0;
+	struct commit *r_commit, *l_commit;
+
+	l_commit = lookup_commit_reference(the_repository, l_oid);
+	r_commit = lookup_commit_reference(the_repository, r_oid);
+
+	/*
+	 * If the remote-tracking ref is an ancestor of the local
+	 * ref (a merge, for instance) there is no need to iterate
+	 * through the reflog entries to ensure reachability; it
+	 * can be skipped to return early instead.
+	 */
+	ret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;
+	if (!ret)
+		ret = for_each_reflog_ent_reverse(local_ref_name, ref_reachable,
+						  (struct object_id *)r_oid);
+
+	return ret;
+}
+
+/*
+ * Check for reachability of a remote-tracking
+ * ref in the reflog entries of its local ref.
+ */
+void check_reflog_for_ref(struct ref *r_ref)
+{
+	struct object_id r_oid;
+	struct ref *l_ref = get_local_ref(r_ref->name);
+
+	if (r_ref->if_includes && l_ref && !read_ref(l_ref->name, &r_oid))
+		r_ref->unreachable = !ref_reachable_from_reflog(&r_ref->old_oid,
+								&r_oid,
+								l_ref->name);
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
 {
-	int i;
+	int i, is_tracking = 0;
 
 	/* Find an explicit --<option>=<name>[:<value>] entry */
 	for (i = 0; i < cas->nr; i++) {
@@ -2288,16 +2381,26 @@ static void apply_cas(struct push_cas_option *cas,
 			oidcpy(&ref->old_oid_expect, &entry->expect);
 		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 			oidclr(&ref->old_oid_expect);
-		return;
+		else
+			is_tracking = 1;
+		break;
 	}
 
 	/* Are we using "--<option>" to cover all? */
-	if (!cas->use_tracking_for_rest)
-		return;
+	if (cas->use_tracking_for_rest) {
+		ref->expect_old_sha1 = 1;
+		if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+			oidclr(&ref->old_oid_expect);
+		else
+			is_tracking = 1;
+	}
+
+	/*
+	 * Mark this ref to be checked if "--force-if-includes" is
+	 * specified as an argument along with "compare-and-swap".
+	 */
+	ref->is_tracking = is_tracking;
 
-	ref->expect_old_sha1 = 1;
-	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
-		oidclr(&ref->old_oid_expect);
 }
 
 void apply_push_cas(struct push_cas_option *cas,
@@ -2308,3 +2411,21 @@ void apply_push_cas(struct push_cas_option *cas,
 	for (ref = remote_refs; ref; ref = ref->next)
 		apply_cas(cas, remote, ref);
 }
+
+void apply_push_force_if_includes(struct ref *remote_refs, int used_with_cas)
+{
+	struct ref *ref;
+	for (ref = remote_refs; ref; ref = ref->next) {
+		/*
+		 * If "compare-and-swap" is used along with option, run the
+		 * check on refs that have been marked to do so. Otherwise,
+		 * all refs will be checked.
+		 */
+		if (used_with_cas)
+			ref->if_includes = ref->is_tracking;
+		else
+			ref->if_includes = 1;
+
+		check_reflog_for_ref(ref);
+	}
+}
diff --git a/remote.h b/remote.h
index 5e3ea5a26d..1618ba892b 100644
--- a/remote.h
+++ b/remote.h
@@ -104,7 +104,10 @@ struct ref {
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		if_includes:1, /* If "--force-with-includes" was specified.  */
+		is_tracking:1, /* If "use_tracking[_for_rest]" is set (CAS). */
+		unreachable:1; /* For "if_includes"; unreachable in reflog.  */
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -134,6 +137,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -346,4 +350,12 @@ int parseopt_push_cas_option(const struct option *, const char *arg, int unset);
 int is_empty_cas(const struct push_cas_option *);
 void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);
 
+/*
+ * Runs when "--force-if-includes" is specified.
+ * Checks if the remote-tracking ref was updated (since checkout)
+ * implicitly in the background and verify that changes from the
+ * updated tip have been integrated locally, before pushing.
+ */
+void apply_push_force_if_includes(struct ref*, int);
+
 #endif
-- 
2.28.0

[PATCH v3 2/7] transport: add flag for "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES", which is
set if "--force-if-includes" is specified in the command line
and calls "apply_push_force_if_includes()" if set.

Update "print_one_push_status()" to catch and print the ref
status if set to "REF_STATUS_REJECT_REMOTE_UPDATED"; add a
reject reason: "REJECT_REF_NEEDS_UPDATE" to map it new ref
status.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 transport.c | 18 +++++++++++++++++-
 transport.h | 12 +++++++-----
 2 files changed, 24 insertions(+), 6 deletions(-)

diff --git a/transport.c b/transport.c
index 43e24bf1e5..7f5020bba3 100644
--- a/transport.c
+++ b/transport.c
@@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "stale info", porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote ref updated since checkout",
+				 porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -659,6 +664,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
@@ -1101,6 +1108,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
@@ -1151,6 +1159,8 @@ int transport_push(struct repository *r,
 		int pretend = flags & TRANSPORT_PUSH_DRY_RUN;
 		int push_ret, ret, err;
 		struct strvec ref_prefixes = STRVEC_INIT;
+		int is_cas = 0;
+		int if_includes = flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES;
 
 		if (check_push_refs(local_refs, rs) < 0)
 			return -1;
@@ -1178,9 +1188,15 @@ int transport_push(struct repository *r,
 
 		if (transport->smart_options &&
 		    transport->smart_options->cas &&
-		    !is_empty_cas(transport->smart_options->cas))
+		    !is_empty_cas(transport->smart_options->cas)) {
+			is_cas = 1;
 			apply_push_cas(transport->smart_options->cas,
 				       transport->remote, remote_refs);
+		}
+
+		/* When "--force-if-includes" is specified. */
+		if (if_includes)
+			apply_push_force_if_includes(remote_refs, is_cas);
 
 		set_ref_status_for_push(remote_refs,
 			flags & TRANSPORT_PUSH_MIRROR,
diff --git a/transport.h b/transport.h
index ca409ea1e4..729bb6979e 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -217,11 +218,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v3 3/7] send-pack: check ref status for "force-if-includes"

[Srinidhi Kaushik]

Update "print_helper_status" in send-pack to catch the ref
status ("REF_STATUS_REJECT_REMOTE_UPDATED") that is set if
checks for "--force-if-includes" fail, and add a string
equivalent of the ref status for printing.

Update "check_to_send_update" to catch the new ref status
when setting "CHECK_REF_STATUS_REJECTED".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 builtin/send-pack.c | 5 +++++
 send-pack.c         | 1 +
 2 files changed, 6 insertions(+)

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 2b9610f121..4d76727edb 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote ref updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/send-pack.c b/send-pack.c
index 632f1580ca..956306e8e8 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
-- 
2.28.0

[PATCH v3 4/7] transport-helper: update ref status for "force-if-includes"

[Srinidhi Kaushik]

Update "push_update_refs_status" to map the string equivalent
of the ref status "REF_STATUS_REJECT_REMOTE_UPDATED" when set
if a `git-push` fails for "--force-if-includes".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 transport-helper.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/transport-helper.c b/transport-helper.c
index c52c99d829..e547e21199 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote ref updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
-- 
2.28.0

[PATCH v3 5/7] builtin/push: add option "--[no-]force-if-includes"

[Srinidhi Kaushik]

Update the option parser to set the flag for "force-if-includes";
add an (optional) advice message to print when the push fails and
the reject reason is "REJECT_REF_NEEDS_UPDATE".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 advice.c       |  3 +++
 advice.h       |  2 ++
 builtin/push.c | 20 ++++++++++++++++++--
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index bc94078e72..d23b4678b4 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -300,6 +300,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git rebase ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -335,6 +341,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -384,8 +397,9 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
-
 	return 1;
 }
 
@@ -521,7 +535,6 @@ static int git_push_config(const char *k, const char *v, void *cb)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
 	}
-
 	return git_default_config(k, v, NULL);
 }
 
@@ -551,6 +564,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, "force-if-includes", &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
-- 
2.28.0

[PATCH v3 6/7] doc: add reference for "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add documentation for using the new option; append notes for
"--force-with-lease" about using the new option to prevent
unintended remote overwrites when being used in setups where a
tool implicitly updates remote-tracking refs in the background.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt |  9 ++++++---
 Documentation/git-push.txt      | 19 +++++++++++++++++++
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..acbd0c09aa 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -10,9 +10,8 @@ advice.*::
 		that the check is disabled.
 	pushUpdateRejected::
 		Set this variable to 'false' if you want to disable
-		'pushNonFFCurrent',
-		'pushNonFFMatching', 'pushAlreadyExists',
-		'pushFetchFirst', and 'pushNeedsForce'
+		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
+		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
 		simultaneously.
 	pushNonFFCurrent::
 		Advice shown when linkgit:git-push[1] fails due to a
@@ -41,6 +40,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..b40fe7e7cf 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying `--force-if-includes` an an ancillary option along
+with `--force-with-lease[=<refname>[:expect]]` (when "<refname>" or "<expect>"
+values are unspecified) at the time of `push` will verify if updates from the
+remote-tracking refs that may have been implicitly updated in the background
+are integrated locally before allowing a forced update.
 
 -f::
 --force::
@@ -341,6 +347,19 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option verifies if the tip of the remote-tracking ref on which
+a local branch has based on (for a rewrite), is reachable from at
+least one of the `reflog` entries of the local branch about to be
+updated by force on the remote. The check ensures that any updates
+from the remote have been incorporated locally by rejecting a push
+if that is not the case.
++
+Specifying `--no-force-if-includes` disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
-- 
2.28.0

[PATCH v3 7/7] t: add tests for "force-if-includes"

[Srinidhi Kaushik]

t/5533:
  * Add a test cases to verify when "--force-if-includes" is used
    along with "--force-with-lease[=<refname>[:expect]]" (when the
    "<expect>" value is unspecified) can help prevent unintended
    remote overwrites when remote refs are updated in the background.

t/t5549:
  * Add test for the new option to cover the following scenarios:
    - Reject forced updates to remote, if the remote ref is updated
      in-between the time of checkout, rewrite and before the push,
      with cases for a specific ref, and "--all".
    - Allow forced updates for "--force", or if the refspec is
      prepended with a "+".
    - Allow deletes on the remote for "--delete", or if refspec is
      specified as ":<ref>".
    - Skip the reflog check introduced by the new option if `git-push`
      is specified with "--force-with-lease=<refname>:<expect>".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 t/t5533-push-cas.sh               |  26 +++++
 t/t5549-push-force-if-includes.sh | 161 ++++++++++++++++++++++++++++++
 2 files changed, 187 insertions(+)
 create mode 100755 t/t5549-push-force-if-includes.sh

diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..6580aab49c 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -256,4 +256,30 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates of REMOTE can be mitigated with "--force-if-includes"' '
+	rm -rf src dst &&
+	git init --bare src.bare &&
+	test_when_finished "rm -rf src.bare" &&
+	git clone --no-local src.bare dst &&
+	test_when_finished "rm -rf dst" &&
+	(
+		cd dst &&
+		test_commit G &&
+		git push origin master:master
+	) &&
+	git clone --no-local src.bare dst2 &&
+	test_when_finished "rm -rf dst2" &&
+	(
+		cd dst2 &&
+		test_commit H &&
+		git push
+	) &&
+	(
+		cd dst &&
+		test_commit I &&
+		git fetch origin &&
+		test_must_fail git push --force-with-lease --force-if-includes origin
+	)
+'
+
 test_done
diff --git a/t/t5549-push-force-if-includes.sh b/t/t5549-push-force-if-includes.sh
new file mode 100755
index 0000000000..e5d1675478
--- /dev/null
+++ b/t/t5549-push-force-if-includes.sh
@@ -0,0 +1,161 @@
+test_description='Test push "--force-if-includes" forced update safety.'
+
+. ./test-lib.sh
+
+setup_src_dup_dst () {
+	rm -fr src dup dst &&
+	git init --bare dst &&
+	git clone --no-local dst src &&
+	git clone --no-local dst dup
+	(
+		cd src &&
+		test_commit foo &&
+		git push
+	) &&
+	(
+		cd dup &&
+		git fetch &&
+		git merge origin/master &&
+		test_commit bar &&
+		git switch -c branch master~1 &&
+		test_commit baz &&
+		test_commit D &&
+		git push --all
+	) &&
+	(
+		cd src &&
+		git switch master &&
+		git fetch --all &&
+		git branch branch --track origin/branch &&
+		git rebase origin/master
+	) &&
+	(
+		cd dup &&
+		git switch master &&
+		test_commit qux &&
+		git switch branch &&
+		test_commit quux &&
+		git push origin --all
+	)
+}
+
+test_expect_success 'reject push if remote changes are not integrated locally (protected, all refs)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit wobble &&
+		git switch master &&
+		test_commit wubble &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes --all
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success 'reject push if remote changes are not integrated locally (protected, specific ref)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit wobble &&
+		git switch master &&
+		test_commit wubble &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes origin master
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success 'allow force push if "--force" is specified (forced, all refs)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit wobble &&
+		git switch master &&
+		test_commit wubble &&
+		git fetch --all &&
+		git push --force --force-if-includes origin --all 2>err &&
+		grep "forced update" err
+	)
+'
+
+test_expect_success 'allow force push if "--delete" is specified' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit wobble &&
+		git switch master &&
+		test_commit wubble &&
+		git fetch --all &&
+		git push --delete --force-if-includes origin branch 2>err &&
+		grep "deleted" err
+	)
+'
+
+test_expect_success 'allow forced updates if specified with refspec (forced, specific ref)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit wobble &&
+		git switch master &&
+		test_commit wubble &&
+		git fetch --all &&
+		git push --force-if-includes origin +branch 2>err &&
+		grep "forced update" err
+	)
+'
+
+test_expect_success 'allow deletes if specified with refspec (delete, specific ref)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit wobble &&
+		git switch master &&
+		test_commit wubble &&
+		git fetch --all &&
+		git push --force-if-includes origin :branch 2>err &&
+		grep "deleted" err
+	)
+'
+
+test_expect_success 'must be disabled for --force-with-lease="<ref>:<expect>" (protected, specific ref)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit wobble &&
+		git switch master &&
+		test_commit wubble &&
+		r_head="$(git rev-parse refs/remotes/origin/master)" &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes --force-with-lease="master:$r_head" 2>err &&
+		grep "stale info" err
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_done
-- 
2.28.0

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Srinidhi Kaushik]

Hi Johannes,

On 09/12/2020 00:16, Johannes Schindelin wrote:
> > + static int oid_in_reflog_ent(struct object_id *ooid, struct object_id *noid,
> > +			     const char *ident, timestamp_t timestamp, int tz,
> > +			     const char *message, void *cb_data)
> > + {
> > +	struct object_id *remote_oid = cb_data;
> > +	struct commit *a = lookup_commit_reference(the_repository, noid);
> > +	struct commit *b = lookup_commit_reference(the_repository, remote_oid);
> > +	return oideq(noid, remote_oid) || in_merge_bases(b, a);
> > + }
> 
> Since `in_merge_bases()` is quite a bit more expensive than `oideq()`,
> personally, I would actually walk the reflog with the `oideq()` check
> first (stopping early in case a match was found), and only fall back to
> looking for a merge base in the reflog if the first reflog walk did not
> find a match.

Thanks for the suggestion; the updated the patch set (v3) - [1] does
something similar to that, along with some other changes. Can you please
take a look?

[1]: https://public-inbox.org/git/20200913145413.18351-2-shrinidhi.kaushik@gmail.com/

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi Junio,
>
> On Thu, 10 Sep 2020, Junio C Hamano wrote:
>
>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>>
>> >> In contrast, when you want to make sure that you _actually_ incorporated
>> >> the revision that is currently the remote tip, e.g. via `git pull
>> >> --rebase` with a possible additional rebase on top that makes this _not_ a
>> >> fast-forward, you totally have to force the push, otherwise it won't work.
>> >
>> > Maybe `--force-if-incorporated`? Originally, I had in mind to call it
>> > `--safe-force`, but that might be too vague.
>>
>> Yup.  "safe force" indeed feels like a misnomer.  The assumption of
>> safety relies heavily on the workflow.
>>
>> I might even say --force-if-merged even if the way the to-be-lost
>> changes have become part of what you are pushing out is not
>> technically a merge, but there may be shorter and sweeter way to
>> express it than 'merge' and 'incorporate'.
>
> You're right, `--force-if-merged` is a much better way to put it.

I am pretty happy that Srinidhi chose 'included', which is what
seems the best description without being a white-lie that is
technically incorrect.

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi Srinidhi,
>
> On Thu, 10 Sep 2020, Srinidhi Kaushik wrote:
>
>> On 09/10/2020 12:22, Johannes Schindelin wrote:
>>
>> > BTW I think the patch needs to cover a bit more, still: after I run `git
>> > pull --rebase`, the local branch will never have been at the same revision
>> > as the fetched one: `git rebase` moves to an unnamed branch before
>> > replaying the patches. So I think we need to see whether the remote tip
>> > was _reachable_ from (not necessarily identical to) any of the reflog's
>> > revisions.
>>
>> Good catch. Would adding in_merge_bases() along with checking if OIDs
>> are equal for each reflog entry in oid_in_reflog_ent() address the
>> problem? That way, we would check if remote ref is reachable from
>> one of the entries?

That sounds very expensive.  

If we switched to check the reflog of HEAD (and not any particular
local branch's reflog), then "has this ever been checked out", tests
would suffice, no?  We detach at the tip of the remote-tracking
branch and then reapply our work one by one in such a case, so we
should have the record of having their tip in the working tree at
some point (i.e. at the beginning of the "rebase" phase of the "pull
--rebase" operation).

Re: [PATCH v3 1/7] remote: add reflog check for "force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Add a check to verify if the remote-tracking ref of the local branch is
> reachable from one of its "reflog" entries; `set_ref_status_for_push`
> updated to add a reject reason and disallow the forced push if the
> check fails.

I have to wonder (not objecting to, just wondering about) if it is a
good assumption that the current branch must be where we should have
seen the tip of the other side we are about to lose.  I ask because
when I do a large rewrite I often am on a detached HEAD most of the
time, and after everything looks sensible in the rewritten result,
I "checkout -B" the local branch.

We could reduce the rate of false positive ("no you've not looked at
what you are about to discard, so we won't let you force") by
checking reflogs of all the local branches and HEAD, but that may be
too much.  I wonder if checking reflog entries only for HEAD (and
not any of the current local branches) would be a good compromise.

Re: [PATCH v3 1/7] remote: add reflog check for "force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> The struct "ref" has three new bit-fields:
>   * if_includes: Set when we have to run the new check on the ref.
>   * is_tracking: Set when the remote ref was marked as "use_tracking"
>                  or "use_tracking_for_rest" by compare-and-swap.

... meaning that --force-with-lease with an explicit "the tip must
still be at this exact commit" won't use the extra check to loosen
the condition?  That sounds sensible.

>   * unreachable: Set if the ref is unreachable from any of the "reflog"
>                  entries of its local counterpart.

The same comment applies on "reflog of which branch should we
check"; I suspect that checking HEAD and without checking merge base
may prove to be a good way to go.

> + */
> +void apply_push_force_if_includes(struct ref*, int);

SP between ref and '*'?

Re: [PATCH v3 6/7] doc: add reference for "--[no-]force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
> index 3b8053447e..b40fe7e7cf 100644
> --- a/Documentation/git-push.txt
> +++ b/Documentation/git-push.txt
> @@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally
>  force push changes to `master` if the remote version is still at
>  `base`, regardless of what your local `remotes/origin/master` has been
>  updated to in the background.
> ++
> +Alternatively, specifying `--force-if-includes` an an ancillary option along
> +with `--force-with-lease[=<refname>[:expect]]` (when "<refname>" or "<expect>"
> +values are unspecified) at the time of `push` will verify if updates from the
> +remote-tracking refs that may have been implicitly updated in the background
> +are integrated locally before allowing a forced update.

You cannot omit <refname> without omitting <expect>, so

	... with "--force-with-lease[=<refname>]" (i.e. without
	saying what exact commit the ref on the remote side must be
	pointing at, or which refs on the remote side are being
	protected) at the time of ...

would be more appropriate.

> +--[no-]force-if-includes::
> +	Force an update only if the tip of the remote-tracking ref
> +	has been integrated locally.
> ++
> +This option verifies if the tip of the remote-tracking ref on which
> +a local branch has based on (for a rewrite), is reachable from at
> +least one of the `reflog` entries of the local branch about to be

If we take the "we don't have to look at a local branch's reflog;
just check HEAD's and rebase will automatically be handled without
expensive merge-base" approach, then

    ... if the tip of the remote-tracking ref was once checked out
    to the working tree (for a rewrite) by seeing if it appears in
    the reflog of "HEAD" ...

Re: [PATCH v3 1/7] remote: add reflog check for "force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> +/*
> + * Check for reachability of a remote-tracking
> + * ref in the reflog entries of its local ref.
> + */
> +void check_reflog_for_ref(struct ref *r_ref)

Make it file-scope static.

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Srinidhi Kaushik]

Hi Junio,

On 09/14/2020 13:08, Junio C Hamano wrote:
>>> [...]
>>>
>>> Good catch. Would adding in_merge_bases() along with checking if OIDs
>>> are equal for each reflog entry in oid_in_reflog_ent() address the
>>> problem? That way, we would check if remote ref is reachable from
>>> one of the entries?
> 
> That sounds very expensive.  

Yes, you're right about that.
 
> If we switched to check the reflog of HEAD (and not any particular
> local branch's reflog), then "has this ever been checked out", tests
> would suffice, no?  We detach at the tip of the remote-tracking
> branch and then reapply our work one by one in such a case, so we
> should have the record of having their tip in the working tree at
> some point (i.e. at the beginning of the "rebase" phase of the "pull
> --rebase" operation).
 
Interesting, I think that might work! Since HEAD moves around and
records all the movements, if the remote was ever checked out there
should be an entry in the reflog. I guess we could stick to "oideq()"
for each entry if we're going this way. However, I think we should
keep that one "in_merge_bases()" call at the beginning to check if
we can can skip checking the reflog if the remote ref is an ancestor
of the local ref.

>>>> On 09/10/2020 12:22, Johannes Schindelin wrote:
>>>>
>>>> BTW I think the patch needs to cover a bit more, still: after I run `git
>>>> pull --rebase`, the local branch will never have been at the same revision
>>>> as the fetched one: `git rebase` moves to an unnamed branch before
>>>> replaying the patches. So I think we need to see whether the remote tip
>>>> was _reachable_ from (not necessarily identical to) any of the reflog's
>>>> revision

Also, there would be an entry for what Johannes describes above -- when
a `git pull --rebase` is run. Will change to this in the next series.

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH v3 6/7] doc: add reference for "--[no-]force-if-includes"

[Srinidhi Kaushik]

Hi Junio,

On 09/14/2020 14:01, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> > diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
> > index 3b8053447e..b40fe7e7cf 100644
> > --- a/Documentation/git-push.txt
> > +++ b/Documentation/git-push.txt
> > @@ -320,6 +320,12 @@ seen and are willing to overwrite, then rewrite history, and finally
> >  force push changes to `master` if the remote version is still at
> >  `base`, regardless of what your local `remotes/origin/master` has been
> >  updated to in the background.
> > ++
> > +Alternatively, specifying `--force-if-includes` an an ancillary option along
> > +with `--force-with-lease[=<refname>[:expect]]` (when "<refname>" or "<expect>"
> > +values are unspecified) at the time of `push` will verify if updates from the
> > +remote-tracking refs that may have been implicitly updated in the background
> > +are integrated locally before allowing a forced update.
> 
> You cannot omit <refname> without omitting <expect>, so
> 
> 	... with "--force-with-lease[=<refname>]" (i.e. without
> 	saying what exact commit the ref on the remote side must be
> 	pointing at, or which refs on the remote side are being
> 	protected) at the time of ...
> 
> would be more appropriate.

OK, that makes sense, will update.
 
> > +--[no-]force-if-includes::
> > +	Force an update only if the tip of the remote-tracking ref
> > +	has been integrated locally.
> > ++
> > +This option verifies if the tip of the remote-tracking ref on which
> > +a local branch has based on (for a rewrite), is reachable from at
> > +least one of the `reflog` entries of the local branch about to be
> 
> If we take the "we don't have to look at a local branch's reflog;
> just check HEAD's and rebase will automatically be handled without
> expensive merge-base" approach, then
> 
>     ... if the tip of the remote-tracking ref was once checked out
>     to the working tree (for a rewrite) by seeing if it appears in
>     the reflog of "HEAD" ...
> 

Right, as mentioned in the other thread, we can go with this route,
will change in the next series.

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Johannes Schindelin]

Hi,

On Wed, 16 Sep 2020, Srinidhi Kaushik wrote:

> On 09/14/2020 13:08, Junio C Hamano wrote:
> >>> [...]
> >>>
> >>> Good catch. Would adding in_merge_bases() along with checking if OIDs
> >>> are equal for each reflog entry in oid_in_reflog_ent() address the
> >>> problem? That way, we would check if remote ref is reachable from
> >>> one of the entries?
> >
> > That sounds very expensive.
>
> Yes, you're right about that.
>
> > If we switched to check the reflog of HEAD (and not any particular
> > local branch's reflog), then "has this ever been checked out", tests
> > would suffice, no?  We detach at the tip of the remote-tracking
> > branch and then reapply our work one by one in such a case, so we
> > should have the record of having their tip in the working tree at
> > some point (i.e. at the beginning of the "rebase" phase of the "pull
> > --rebase" operation).
>
> Interesting, I think that might work! Since HEAD moves around and
> records all the movements, if the remote was ever checked out there
> should be an entry in the reflog.

No, I don't think that would suffice. For several reasons:

- the user could have checked out the remote-tracking branch directly
  (`git switch --detach origin/my-branch`), as opposed to actually
  integrating the revision into the branch at some stage. I know that I do
  that sometimes.

- even if the remote-tracking branch has not been checked out directly, it
  might have been `git pull --rebase`d into a _different_ branch, in which
  case the reflog of `HEAD` would say "yep, I saw this commit!" but that
  would not mean that it was integrated into the branch we want to
  (force-)push.

- the reflog of the `HEAD` is worktree-specific, and if the user pushes
  from a different worktree (e.g. to push multiple branches at the same
  time, or because the push is scripted and run from a different working
  directory), it would be missed.

- and if we take a little step back, we do see that the reflog of `HEAD`
  _does_ answer a different question from what we actually asked.

Sure, it would be faster, but is that worth the consequences?

I really think we need to stick to looking at the reflog of the asked-for
branch. And to make that faster, we should have a first loop using
`oideq()` and failing that check, run a second loop using
`is_in_merge_bases()`.

Ciao,
Dscho

Re: [PATCH v3 1/7] remote: add reflog check for "force-if-includes"

[Srinidhi Kaushik]

Hello,

On 09/14/2020 13:17, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> > Add a check to verify if the remote-tracking ref of the local branch is
> > reachable from one of its "reflog" entries; `set_ref_status_for_push`
> > updated to add a reject reason and disallow the forced push if the
> > check fails.
> 
> I have to wonder (not objecting to, just wondering about) if it is a
> good assumption that the current branch must be where we should have
> seen the tip of the other side we are about to lose.  I ask because
> when I do a large rewrite I often am on a detached HEAD most of the
> time, and after everything looks sensible in the rewritten result,
> I "checkout -B" the local branch.
> 
> We could reduce the rate of false positive ("no you've not looked at
> what you are about to discard, so we won't let you force") by
> checking reflogs of all the local branches and HEAD, but that may be
> too much.  I wonder if checking reflog entries only for HEAD (and
> not any of the current local branches) would be a good compromise.

One scenario I can think of is when there are multiple local branches
that track the same remote. Let's say we have two branches "A" and "B"
and they both track "A" on the remote. If we are currently on "A",
and then we decide to rebase on "origin/A" (after a push from another
repository). Then, if we (accidentally) switch to "B", and force update
with `--force-if-includes` it will _not_ be rejected because HEAD's
reflog has a record of the checkout and there will be an overwrite
if we check only HEAD's reflog.

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Johannes Schindelin]

Hi Srinidhi,

On Tue, 8 Sep 2020, Srinidhi Kaushik wrote:

> On 09/07/2020 21:45, Johannes Schindelin wrote:
> >> [...]
>
> > Now, to be honest, I thought that this mode would merit a new option
> > rather than piggy-backing on top of `--force-with-lease`. The reason is
> > that `--force-with-lease` targets a slightly different use case than mine:
> > it makes sure that we do not overwrite remote refs unless we already had a
> > chance to inspect them.
> >
> > In contrast, my workflow uses `git pull --rebase` in two or more separate
> > worktrees, e.g. when developing a patch on two different Operating
> > Systems, I frequently forget to pull (to my public repository) on one
> > side, and I want to avoid force-pushing in that case, even if VS Code (or
> > I, via `git remote update`) fetched the ref (but failing to rebase the
> > local branch on top of it).
> >
> > However, in other scenarios I very much do _not_ want to incorporate the
> > remote ref. For example, I often fetch
> > https://github.com/git-for-windows/git.wiki.git to check for the
> > occasional bogus change. Whenever I see such a bogus change, and it is at
> > the tip of the branch, I want to force-push _without_ incorporating the
> > bogus change into the local branch, yet I _do_ want to use
> > `--force-with-lease` because an independent change could have come in via
> > the Wiki in the meantime.
>
> I realize that this new check would not be helpful if we deliberately
> choose not to include an unwanted change from the updated remote's tip.
> In that case, we would have to use `--force` make it work, and that
> defeats the use of `--force-with-lease`.

I would characterize it as "somewhat stricter" than `--force-with-lease`.
The check would not make sense without the lease.

> > So I think that the original `--force-with-lease` and the mode you
> > implemented target subtly different use cases that are both valid, and
> > therefore I would like to request a separate option for the latter.
>
> OK. So, I am assuming that you are suggesting to add a new function that
> is separate from `apply_push_cas()` and run the check on each of the
> remote refs. Would that be correct?

Oh, I don't really know the code well enough to make a suggestion. I guess
if it is easy enough to modify the existing code path (e.g. extend the
function signature in a minimal manner), then that's what I would go for,
rather than a completely separate code path.

> If that's the case, how does it work along with `--force-with-lease`?

In my mind, the new mode implies `--force-with-lease`.

Thanks,
Dscho

> On one hand we have `--force-with-lease` to ensure we rewrite the remote
> that we have _already_ seen, and on the other, a new option that checks
> reflog of the local branch to see if it is missing any updates from the
> remote that may have happened in the meantime. If we pass both of them
> for `push` and if the former doesn't complain, and the latter check
> fails, should the `push` still go through?
>
> I feel that this check included with `--force-with-lease` only when
> the `use_tracking` or `use_tracking_for_rest` options are enabled
> would give a heads-up the the user about the background fetch. If
> they decide that they don't need new updates, then supplying the
> new "<expect>" value in the next push would imply they've seen the
> new update, and choose to overwrite it anyway. The check would not
> run in this case. But again, I wonder if the this "two-step" process
> makes `push` cumbersome.
>
> > However, I have to admit that I could not think of a good name for that
> > option. "Implicit fetch" seems a bit too vague here, because the local
> > branch was not fetched, and certainly not implicitly, yet the logic
> > revolves around the local branch having been rebased to the
> > remote-tracking ref at some stage.
>
> The message "implicit fetch" was in context of the remote ref. But yes,
> the current reject reason is not clear and implies that local branch
> was fetched, which isn't the case.
>
> > Even if we went with the config option to modify `--force-with-lease`'s
> > behavior, I would recommend separating out the `feature.experimental`
> > changes into their own patch, so that they can be reverted easily in case
> > the experimental feature is made the default.
>
> Good idea!
>
> > A couple more comments:
> >
> >> @@ -1471,16 +1489,21 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
> >>                * If the remote ref has moved and is now different
> >>                * from what we expect, reject any push.
> >>                *
> >> -              * It also is an error if the user told us to check
> >> -              * with the remote-tracking branch to find the value
> >> -              * to expect, but we did not have such a tracking
> >> -              * branch.
> >> +              * It also is an error if the user told us to check with the
> >> +              * remote-tracking branch to find the value to expect, but we
> >> +              * did not have such a tracking branch, or we have one that
> >> +              * has new changes.
> >
> > If I were you, I would try to keep the original formatting, so that it
> > becomes more obvious that the part ", or we have [...]" was appended.
>
> Alright, I will append the new comment in a new line instead.
>
> >
> >>               if (ref->expect_old_sha1) {
> >>                       if (!oideq(&ref->old_oid, &ref->old_oid_expect))
> >>                               reject_reason = REF_STATUS_REJECT_STALE;
> >> +                     else if (reject_implicit_fetch() && ref->implicit_fetch)
> >> +                             reject_reason = REF_STATUS_REJECT_IMPLICIT_FETCH;
> >>                       else
> >> -                             /* If the ref isn't stale then force the update. */
> >> +                             /*
> >> +                              * If the ref isn't stale, or there was no
> >
> > Should this "or" not be an "and" instead?
>
> D'oh, you are right. It should have been an "and".
>
> >
> >> +                              * implicit fetch, force the update.
> >> +                              */
> >>                               force_ref_update = 1;
> >>               }
> >> [...]
> >>  static void apply_cas(struct push_cas_option *cas,
> >>                     struct remote *remote,
> >>                     struct ref *ref)
> >>  {
> >> -     int i;
> >> +     int i, do_reflog_check = 0;
> >> +     struct object_id oid;
> >> +     struct ref *local_ref = get_local_ref(ref->name);
> >>
> >>       /* Find an explicit --<option>=<name>[:<value>] entry */
> >>       for (i = 0; i < cas->nr; i++) {
> >>               struct push_cas *entry = &cas->entry[i];
> >>               if (!refname_match(entry->refname, ref->name))
> >>                       continue;
> >> +
> >>               ref->expect_old_sha1 = 1;
> >>               if (!entry->use_tracking)
> >>                       oidcpy(&ref->old_oid_expect, &entry->expect);
> >>               else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
> >>                       oidclr(&ref->old_oid_expect);
> >> -             return;
> >> +             else
> >> +                     do_reflog_check = 1;
> >> +
> >> +             goto reflog_check;
> >
> > Hmm. I do not condemn `goto` statements in general, but this one makes the
> > flow harder to follow. I would prefer something like this:
> >
> > -- snip --
> >               else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
> >                       oidclr(&ref->old_oid_expect);
> > +             else if (local_ref && !read_ref(local_ref->name, &oid))
> > +                     ref->implicit_fetch =
> > +                             !remote_ref_in_reflog(&ref->old_oid, &oid,
> > +                                                   local_ref->name);
> >               return;
> > -- snap --
>
> Adding this condition looks cleaner instead of the `goto`. A similar
> suggestion was made in the other thread [1] as well; this will be
> addressed in v2.
>
> > Again, thank you so much for working on this!
>
> Thanks again, for taking the time to review this.
>
> [1]: https://public-inbox.org/git/624d9e35-29b8-4012-a3d6-e9b00a9e4485@gmail.com/
> --
> Srinidhi Kaushik
>

Re: [PATCH v3 1/7] remote: add reflog check for "force-if-includes"

[Johannes Schindelin]

Hi Srinidhi,

On Sun, 13 Sep 2020, Srinidhi Kaushik wrote:

> diff --git a/remote.c b/remote.c
> index 420150837b..e4b2d85a6f 100644
> --- a/remote.c
> +++ b/remote.c
> @@ -1484,6 +1484,36 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
>  				force_ref_update = 1;
>  		}
>
> +		/*
> +		 * If the tip of the remote-tracking ref is unreachable
> +		 * from any reflog entry of its local ref indicating a
> +		 * possible update since checkout; reject the push.
> +		 *
> +		 * There is no need to check for reachability, if the
> +		 * ref is marked for deletion.
> +		 */
> +		if (ref->if_includes && !ref->deletion) {
> +			/*
> +			 * If `force_ref_update' was previously set by
> +			 * "compare-and-swap", and we have to run this
> +			 * check, reset it back to the original value
> +			 * and update it depending on the status of this
> +			 * check.
> +			 */
> +			force_ref_update = ref->force || force_update;
> +
> +			if (ref->unreachable)
> +				reject_reason =
> +					REF_STATUS_REJECT_REMOTE_UPDATED;
> +			else
> +				/*
> +				 * If updates from the remote-tracking ref
> +				 * have been integrated locally; force the
> +				 * update.
> +				 */
> +				force_ref_update = 1;
> +		}
> +
>  		/*
>  		 * If the update isn't already rejected then check
>  		 * the usual "must fast-forward" rules.
> @@ -2272,11 +2302,74 @@ static int remote_tracking(struct remote *remote, const char *refname,
>  	return 0;
>  }
>
> +static int ref_reachable(struct object_id *o_oid, struct object_id *n_oid,
> +			 const char *ident, timestamp_t timestamp, int tz,
> +			 const char *message, void *cb_data)
> +{
> +	int ret = 0;
> +	struct object_id *r_oid = cb_data;
> +
> +	ret = oideq(n_oid, r_oid);
> +	if (!ret) {

Rather than having the largest part of the actual code statements in this
function indented, it would make more sense to write

	if (oideq(n_oid, r_oid))
		return 1;

	if (!(local = lookup...) ||
	    !(remote = lookup...))
		return 0;

	return in_merge_bases(remote, local);

> +		struct commit *loc = lookup_commit_reference(the_repository,
> +							     n_oid);
> +		struct commit *rem = lookup_commit_reference(the_repository,
> +							     r_oid);
> +		ret = (loc && rem) ? in_merge_bases(rem, loc) : 0;
> +	}

This chooses the strategy of iterating over the reflog just once, and at
every step first testing whether the respective reflog entry is identical
to the remote-tracking branch tip. Only when they are different do we test
whether the remote-tracking branch tip is at least reachable from the
reflog entry.

Let's assume that our local branch has 20 reflog entries, and the 4th one
is identical to the current tip of the remote-tracking branch. Then we
tested reachability 3 times. But that test is rather expensive.

Therefore, I would have preferred to have a call to
`for_each_reflog_ent_reverse()` with a callback function that only returns
the `oideq()` result, and only if the return value of that call is 0, I
would have wanted to see another call to `for_each_reflog_ent_reverse()`
to go through, this time looking for reachability.

> +
> +	return ret;
> +}
> +
> +/*
> + * Iterate through the reflog of a local branch and check
> + * if the tip of the remote-tracking branch is reachable
> + * from one of the entries.
> + */
> +static int ref_reachable_from_reflog(const struct object_id *r_oid,
> +				     const struct object_id *l_oid,
> +				     const char *local_ref_name)
> +{
> +	int ret = 0;
> +	struct commit *r_commit, *l_commit;
> +
> +	l_commit = lookup_commit_reference(the_repository, l_oid);
> +	r_commit = lookup_commit_reference(the_repository, r_oid);

At this point, we already LOOked up `r_commit`. But we don't pass that to
`ref_reachable()` at any point (instead passing only `r_oid`), so we have
to perform the lookup again.

That's wasteful. Shouldn't we pass `r_commit` directly?

With the two-pass strategy I outlined above, the first pass would use
`r_oid`, and only when the second pass is necessary would we resort to
calling the expensive reachability check.

> +
> +	/*
> +	 * If the remote-tracking ref is an ancestor of the local
> +	 * ref (a merge, for instance) there is no need to iterate
> +	 * through the reflog entries to ensure reachability; it
> +	 * can be skipped to return early instead.
> +	 */
> +	ret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;

Correct me if I am wrong, but isn't the first reflog entry
(`<remote-branch>@{0}`) identical to `r_commit`? In that case, the first
iteration of the second pass over the reflog would trivially perform this
check, and we do not need to duplicate the logic here.

> +	if (!ret)
> +		ret = for_each_reflog_ent_reverse(local_ref_name, ref_reachable,
> +						  (struct object_id *)r_oid);
> +
> +	return ret;
> +}
> +
> +/*
> + * Check for reachability of a remote-tracking
> + * ref in the reflog entries of its local ref.
> + */
> +void check_reflog_for_ref(struct ref *r_ref)
> +{
> +	struct object_id r_oid;
> +	struct ref *l_ref = get_local_ref(r_ref->name);
> +
> +	if (r_ref->if_includes && l_ref && !read_ref(l_ref->name, &r_oid))

If `r_ref->if_includes` is 0, we do not even have to get the local ref,
correct? It would make `check_reflog_for_ref()` much easier to read for me
if it was only called when that flag was already verified to be 1, and
then followed this structure:

	if (!l_ref)
		return;
	if (read_ref(...))
		warning(_("ignoring stale remote branch information ..."));
	else
		r_ref->unreachable = ...

Also, it might make a lot more sense to rename `check_reflog_for_ref()` to
`check_if_includes_upstream()`, and to rename `r_ref` to `local` and
`l_ref` to `remote_tracking` or something like that: nothing is inherently
"left" or "right" about those refs.

> +		r_ref->unreachable = !ref_reachable_from_reflog(&r_ref->old_oid,
> +								&r_oid,
> +								l_ref->name);
> +}
> +
>  static void apply_cas(struct push_cas_option *cas,
>  		      struct remote *remote,
>  		      struct ref *ref)
>  {
> -	int i;
> +	int i, is_tracking = 0;
>
>  	/* Find an explicit --<option>=<name>[:<value>] entry */
>  	for (i = 0; i < cas->nr; i++) {
> @@ -2288,16 +2381,26 @@ static void apply_cas(struct push_cas_option *cas,
>  			oidcpy(&ref->old_oid_expect, &entry->expect);
>  		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
>  			oidclr(&ref->old_oid_expect);
> -		return;
> +		else
> +			is_tracking = 1;

As part of `remote_tracking()`, we already looked up the branch name.
Since we need it in the `is_tracking` case, maybe this should not be a
Boolean anymore but store a copy of the remote-tracking branch name
instead?

Oh, following the code path all the way down to
`match_name_with_pattern()`, it seems that `remote_tracking()`'s `dst`
variable _already_ contains a copy (which means that that memory is
leaked, right?).

> +		break;
>  	}
>
>  	/* Are we using "--<option>" to cover all? */
> -	if (!cas->use_tracking_for_rest)
> -		return;
> +	if (cas->use_tracking_for_rest) {
> +		ref->expect_old_sha1 = 1;
> +		if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
> +			oidclr(&ref->old_oid_expect);
> +		else
> +			is_tracking = 1;
> +	}
> +
> +	/*
> +	 * Mark this ref to be checked if "--force-if-includes" is
> +	 * specified as an argument along with "compare-and-swap".
> +	 */
> +	ref->is_tracking = is_tracking;
>
> -	ref->expect_old_sha1 = 1;
> -	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
> -		oidclr(&ref->old_oid_expect);
>  }
>
>  void apply_push_cas(struct push_cas_option *cas,
> @@ -2308,3 +2411,21 @@ void apply_push_cas(struct push_cas_option *cas,
>  	for (ref = remote_refs; ref; ref = ref->next)
>  		apply_cas(cas, remote, ref);
>  }
> +
> +void apply_push_force_if_includes(struct ref *remote_refs, int used_with_cas)
> +{
> +	struct ref *ref;
> +	for (ref = remote_refs; ref; ref = ref->next) {
> +		/*
> +		 * If "compare-and-swap" is used along with option, run the
> +		 * check on refs that have been marked to do so. Otherwise,
> +		 * all refs will be checked.
> +		 */
> +		if (used_with_cas)
> +			ref->if_includes = ref->is_tracking;
> +		else
> +			ref->if_includes = 1;
> +
> +		check_reflog_for_ref(ref);
> +	}
> +}
> diff --git a/remote.h b/remote.h
> index 5e3ea5a26d..1618ba892b 100644
> --- a/remote.h
> +++ b/remote.h
> @@ -104,7 +104,10 @@ struct ref {
>  		forced_update:1,
>  		expect_old_sha1:1,
>  		exact_oid:1,
> -		deletion:1;
> +		deletion:1,
> +		if_includes:1, /* If "--force-with-includes" was specified.  */
> +		is_tracking:1, /* If "use_tracking[_for_rest]" is set (CAS). */
> +		unreachable:1; /* For "if_includes"; unreachable in reflog.  */
>
>  	enum {
>  		REF_NOT_MATCHED = 0, /* initial value */
> @@ -134,6 +137,7 @@ struct ref {
>  		REF_STATUS_REJECT_NEEDS_FORCE,
>  		REF_STATUS_REJECT_STALE,
>  		REF_STATUS_REJECT_SHALLOW,
> +		REF_STATUS_REJECT_REMOTE_UPDATED,
>  		REF_STATUS_UPTODATE,
>  		REF_STATUS_REMOTE_REJECT,
>  		REF_STATUS_EXPECTING_REPORT,
> @@ -346,4 +350,12 @@ int parseopt_push_cas_option(const struct option *, const char *arg, int unset);
>  int is_empty_cas(const struct push_cas_option *);
>  void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);
>
> +/*
> + * Runs when "--force-if-includes" is specified.
> + * Checks if the remote-tracking ref was updated (since checkout)
> + * implicitly in the background and verify that changes from the
> + * updated tip have been integrated locally, before pushing.
> + */
> +void apply_push_force_if_includes(struct ref*, int);

This function is not even hooked up in this patch, right? I don't think
that it makes sense to introduce it without a caller, in particular since
it makes it harder to guess what those parameters might be used for.

In general, it appears to me as if the code worked way too hard to
accomplish something that should be a lot simpler: when
`--force-if-includes` is passed, it should piggy-back on top of the
`--force-with-lease` code path, and just add yet another check on top.

With that in mind, I would have expected something more in line with this:

-- snip --
 struct push_cas_option {
 	unsigned use_tracking_for_rest:1;
 	struct push_cas {
 		struct object_id expect;
-		unsigned use_tracking:1;
+		enum {
+			PUSH_NO_CAS = 0,
+			PUSH_CAS_USE_TRACKING,
+			PUSH_CAS_IF_INCLUDED
+		} mode;
 		char *refname;
 	} *entry;
 	int nr;
 	int alloc;
 };
-- snap --

and then adjusting the respective code paths accordingly.

Ciao,
Dscho

> +
>  #endif
> --
> 2.28.0
>
>

Re: [PATCH v3 5/7] builtin/push: add option "--[no-]force-if-includes"

[Johannes Schindelin]

Hi Srinidhi,

On Sun, 13 Sep 2020, Srinidhi Kaushik wrote:

> diff --git a/builtin/push.c b/builtin/push.c
> index bc94078e72..d23b4678b4 100644
> --- a/builtin/push.c
> +++ b/builtin/push.c
> @@ -300,6 +300,12 @@ static const char message_advice_ref_needs_force[] =
>  	   "or update a remote ref to make it point at a non-commit object,\n"
>  	   "without using the '--force' option.\n");
>
> +static const char message_advice_ref_needs_update[] =
> +	N_("Updates were rejected because the tip of the remote-tracking\n"
> +	   "branch has been updated since the last checkout. You may want\n"
> +	   "to integrate those changes locally (e.g., 'git rebase ...')\n"

Shouldn't this talk about `git pull` instead of `git rebase`?

Ciao,
Dscho

Re: [PATCH v3 0/7] push: add "--[no-]force-if-includes"

[Johannes Schindelin]

Hi Srinidhi,

On Sun, 13 Sep 2020, Srinidhi Kaushik wrote:

> Add a new option: "--force-if-includes" to `git-push` where forced
> updates are allowed only if the tip of the remote-tracking ref has
> been integrated locally, by verifying if the tip of the remote-tracking
> ref -- on which a local branch has based on -- is reachable from at
> least one of the "reflog" entries of the branch about to be updated
> by force on the remote.
>
> This option can also be used with `--force-with-lease` with setups
> where the remote-tracking refs of the repository are implicitly
> updated in the background to help prevent unintended remote
> overwrites.
>
> If a local branch is based on a remote ref for a rewrite, and if that
> remote-tracking ref is updated by a push from another repository after
> it has been checked out locally, force updating that branch to remote
> with `--force-with-lease[=<refname>[:expect]]` without specifying the
> "<refname>" or "<expect>" values, can cause the update that happened
> in-between the checkout and forced push to be lost.

Thank you for working on this! I gave this an incomplete look-over, and
offered some suggestions that you hopefully find useful.

> Changes since v2:
>   * Removed configuration option for setting "--force-if-includes"
>     with "--force-with-lease".
>   * Broke up the patch into smaller commits.

While the commits all seem to be compiling individually, I am not really a
fan of introducing a function without a caller that shows how it is
supposed to work. I'd rather see some incremental story, and in this case,
I think if _I_ were to submit this patch series, I would probably have
only two commits: one that extends the already-existing code path to turn
that `use_tracking` flag into that `enum`, and the second patch which
wires up the option, adds the documentation and the tests.

However, please do not let my tastes dictate how you want to present the
work, although I hope that my suggestion inspires you ;-)

Ciao,
Dscho

Re: [PATCH v3 1/7] remote: add reflog check for "force-if-includes"

[Srinidhi Kaushik]

Hi Johannes,

On 09/16/2020 14:35, Johannes Schindelin wrote:

> > [...]
> > +		struct commit *loc = lookup_commit_reference(the_repository,
> > +							     n_oid);
> > +		struct commit *rem = lookup_commit_reference(the_repository,
> > +							     r_oid);
> > +		ret = (loc && rem) ? in_merge_bases(rem, loc) : 0;
> > +	}
> 
> This chooses the strategy of iterating over the reflog just once, and at
> every step first testing whether the respective reflog entry is identical
> to the remote-tracking branch tip. Only when they are different do we test
> whether the remote-tracking branch tip is at least reachable from the
> reflog entry.
> 
> Let's assume that our local branch has 20 reflog entries, and the 4th one
> is identical to the current tip of the remote-tracking branch. Then we
> tested reachability 3 times. But that test is rather expensive.
> 
> Therefore, I would have preferred to have a call to
> `for_each_reflog_ent_reverse()` with a callback function that only returns
> the `oideq()` result, and only if the return value of that call is 0, I
> would have wanted to see another call to `for_each_reflog_ent_reverse()`
> to go through, this time looking for reachability.

OK, you're right about this. Will update check to use the two-step
approach.

> > [...] 
> > +	int ret = 0;
> > +	struct commit *r_commit, *l_commit;
> > +
> > +	l_commit = lookup_commit_reference(the_repository, l_oid);
> > +	r_commit = lookup_commit_reference(the_repository, r_oid);
> 
> At this point, we already LOOked up `r_commit`. But we don't pass that to
> `ref_reachable()` at any point (instead passing only `r_oid`), so we have
> to perform the lookup again.
> 
> That's wasteful. Shouldn't we pass `r_commit` directly?

Hmm, I suppose we can. Not sure why I did that in the first place. 

> With the two-pass strategy I outlined above, the first pass would use
> `r_oid`, and only when the second pass is necessary would we resort to
> calling the expensive reachability check.
> 
> > +
> > +	/*
> > +	 * If the remote-tracking ref is an ancestor of the local
> > +	 * ref (a merge, for instance) there is no need to iterate
> > +	 * through the reflog entries to ensure reachability; it
> > +	 * can be skipped to return early instead.
> > +	 */
> > +	ret = (r_commit && l_commit) ? in_merge_bases(r_commit, l_commit) : 0;
> 
> Correct me if I am wrong, but isn't the first reflog entry
> (`<remote-branch>@{0}`) identical to `r_commit`? In that case, the first
> iteration of the second pass over the reflog would trivially perform this
> check, and we do not need to duplicate the logic here.

That's a correct assumption; the second pass will check for this.

> > +	if (!ret)
> > +		ret = for_each_reflog_ent_reverse(local_ref_name, ref_reachable,
> > +						  (struct object_id *)r_oid);
> > +
> > +	return ret;
> > +}
> > +
> > +/*
> > + * Check for reachability of a remote-tracking
> > + * ref in the reflog entries of its local ref.
> > + */
> > +void check_reflog_for_ref(struct ref *r_ref)
> > +{
> > +	struct object_id r_oid;
> > +	struct ref *l_ref = get_local_ref(r_ref->name);
> > +
> > +	if (r_ref->if_includes && l_ref && !read_ref(l_ref->name, &r_oid))
> 
> If `r_ref->if_includes` is 0, we do not even have to get the local ref,
> correct? It would make `check_reflog_for_ref()` much easier to read for me
> if it was only called when that flag was already verified to be 1, and
> then followed this structure:
> 
> 	if (!l_ref)
> 		return;
> 	if (read_ref(...))
> 		warning(_("ignoring stale remote branch information ..."));
> 	else
> 		r_ref->unreachable = ...
> 
> Also, it might make a lot more sense to rename `check_reflog_for_ref()` to
> `check_if_includes_upstream()`, and to rename `r_ref` to `local` and
> `l_ref` to `remote_tracking` or something like that: nothing is inherently
> "left" or "right" about those refs.

Now that I think about it, we don't need te call to "read_ref()" at all
because the reflog will have the OID we need for checking. As to
"{l,r}_ref", they were meant to be [l]ocal and [r]emote. :)

> > +		r_ref->unreachable = !ref_reachable_from_reflog(&r_ref->old_oid,
> > +								&r_oid,
> > +								l_ref->name);
> > +}
> > +
> >  static void apply_cas(struct push_cas_option *cas,
> >  		      struct remote *remote,
> >  		      struct ref *ref)
> >  {
> > -	int i;
> > +	int i, is_tracking = 0;
> >
> >  	/* Find an explicit --<option>=<name>[:<value>] entry */
> >  	for (i = 0; i < cas->nr; i++) {
> > @@ -2288,16 +2381,26 @@ static void apply_cas(struct push_cas_option *cas,
> >  			oidcpy(&ref->old_oid_expect, &entry->expect);
> >  		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
> >  			oidclr(&ref->old_oid_expect);
> > -		return;
> > +		else
> > +			is_tracking = 1;
> 
> As part of `remote_tracking()`, we already looked up the branch name.
> Since we need it in the `is_tracking` case, maybe this should not be a
> Boolean anymore but store a copy of the remote-tracking branch name
> instead?
> 
> Oh, following the code path all the way down to
> `match_name_with_pattern()`, it seems that `remote_tracking()`'s `dst`
> variable _already_ contains a copy (which means that that memory is
> leaked, right?).

I'm not sure I understand what you mean. The call to "remote_tracking()"
gets the OID of the remote ref and writes it to "old_oid_expect".
We don't need "dst" from there because "old_oid" is sufficient for
checking the reflog entries. However, calling "get_local_ref()" is
necessary to get the local branch name. Also, why does "is_tracking"
have to be a Boolean? We already have "ref->name", right?

Anyway, I think we can get rid of "is_tracking" altogether and just use
"if_includes".

> > [...] 
> > +/*
> > + * Runs when "--force-if-includes" is specified.
> > + * Checks if the remote-tracking ref was updated (since checkout)
> > + * implicitly in the background and verify that changes from the
> > + * updated tip have been integrated locally, before pushing.
> > + */
> > +void apply_push_force_if_includes(struct ref*, int);
> 
> This function is not even hooked up in this patch, right? I don't think
> that it makes sense to introduce it without a caller, in particular since
> it makes it harder to guess what those parameters might be used for.
> 
> In general, it appears to me as if the code worked way too hard to
> accomplish something that should be a lot simpler: when
> `--force-if-includes` is passed, it should piggy-back on top of the
> `--force-with-lease` code path, and just add yet another check on top.

Well, it isn't included in this commit, because it was called from
"transport.c" and "send-pack.c", I decided to put that in another
commit. I will fix that in the next patch series.

> With that in mind, I would have expected something more in line with this:
> 
> -- snip --
>  struct push_cas_option {
>  	unsigned use_tracking_for_rest:1;
>  	struct push_cas {
>  		struct object_id expect;
> -		unsigned use_tracking:1;
> +		enum {
> +			PUSH_NO_CAS = 0,
> +			PUSH_CAS_USE_TRACKING,
> +			PUSH_CAS_IF_INCLUDED
> +		} mode;
>  		char *refname;
>  	} *entry;
>  	int nr;
>  	int alloc;
>  };
> -- snap --
> 
> and then adjusting the respective code paths accordingly.

OK, I will clean it up and get rid of the redundant/wasteful
calls and data being passed around. Regarding the new "enum",
I feel that just having a new bit-field "use_force_if_includes"
might be sufficient for "push_cas_option". The idea is to set it
to "1" when "--force-if-includes" is specified along with
"--force-with-lease", and "apply_push_cas()" will run the check
and set "ref->unreachable" depending on the check status.
To clarify, you would have to specify something like this to
enable the check:

   git push --force-if-includes --force-with-lease=master [...]

Then, having a configuration setting "push.useForceIfIncludes" would
be the same as running "git-push" like mentioned above. This new option
will be a "no-op" if specified otherwise. Would that be acceptable?

Thanks again, for a thorough review. 
-- 
Srinidhi Kaushik

[PATCH v4 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: "--force-if-includes" to "git-push" where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref -- on which a local branch has based on -- is reachable from at
least one of the "reflog" entries of the branch about to be updated
by force on the remote.

This option can be used with "--force-with-lease" with setups where
the remote-tracking refs of the repository are implicitly updated in
the background to help prevent unintended remote overwrites.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with "--force-with-lease[=<refname>[:<expect>]]" without specifying
the "<expect>" value, can cause the update that happened in-between the
checkout and forced push to be lost.

Changes since v3:
  - Switch to the two-step approach: check the "reflog" the first
    time to see if the remote ref exists; if not, go through it one
    more time to see if it is reachable (using "in_merge_bases()")
    from any of the entries. This is a necessary step as explained
    here [1].

  - Added back the configuration setting "push.useForceIfIncludes", now
    that it does not affect the functionality of "--force-with-lease"
    when the expected ref is specified.

  - Update "remote-curl" and "send-pack" to recognize the new option;
    this was missing in the previous patches.

  - Broke up the patch-set into medium sized commits.

  - Clean-up unnecessary function calls.

Srinidhi Kaushik (3):
  push: add reflog check for "--force-if-includes"
  push: parse and set flag for "--force-if-includes"
  t, doc: update tests, reference for "--force-if-includes"

 Documentation/config/advice.txt |  9 ++-
 Documentation/config/push.txt   |  5 ++
 Documentation/git-push.txt      | 20 +++++++
 advice.c                        |  3 +
 advice.h                        |  2 +
 builtin/push.c                  | 27 +++++++++
 builtin/send-pack.c             | 11 ++++
 remote-curl.c                   | 14 ++++-
 remote.c                        | 97 ++++++++++++++++++++++++++++++++-
 remote.h                        | 12 +++-
 send-pack.c                     |  1 +
 t/t5533-push-cas.sh             | 53 ++++++++++++++++++
 transport-helper.c              | 11 ++++
 transport.c                     |  8 +++
 transport.h                     | 15 +++--
 15 files changed, 276 insertions(+), 12 deletions(-)

base-commit: 385c171a018f2747b329bcfa6be8eda1709e5abd

[1]: https://public-inbox.org/git/nycvar.QRO.7.76.6.2009161214030.56@tvgsbejvaqbjf.bet/

--
2.28.0

[PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Adds a check to verify if the remote-tracking ref of the local branch
is reachable from one of its "reflog" entries.

When a local branch that is based on a remote ref, has been rewound
and is to be force pushed on the remote, "--force-if-includes" runs
a check that ensures any updates to remote-tracking refs that may have
happened (by push from another repository) in-between the time of the
last checkout, and right before the time of push, have been integrated
locally before allowing a forced updated.

A new field "use_force_if_includes" has been added to "push_cas_option",
which is set to "1" when "--force-if-includes" is specified as an
argument in the command line or set as a configuration option.

The struct "ref" has two new bit-fields:
  - if_includes:
    Set when we have to run the new check on the ref, and the remote
    ref was marked as "use_tracking" or "use_tracking_for_rest" by
    compare-and-swap (if the "the remote tip must be at the expected
    commit" condition is not specified); "apply_push_cas()" has been
    updated to check if this field is set and run the check.

  - unreachable:
    Set if the ref is unreachable from any of the "reflog" entries of
    its local counterpart.

"REF_STATUS_REJECT_REMOTE_UPDATED" has been added to the "status"
enum to imply that the ref failed the check; "case" statements in
"send-pack", "transport" and "transport-helper" have been updated
accordingly to catch this status when set.

When "--force-with-includes" is used along with "--force-with-lease",
the check runs only for refs marked as "if_includes". If the option
is passed without specifying "--force-with-lease", or specified along
with "--force-with-lease=<refname>:<expect>" it is a "no-op".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 builtin/send-pack.c |  5 +++
 remote.c            | 92 ++++++++++++++++++++++++++++++++++++++++++++-
 remote.h            |  6 ++-
 send-pack.c         |  1 +
 transport-helper.c  |  5 +++
 transport.c         |  6 +++
 6 files changed, 112 insertions(+), 3 deletions(-)

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 2b9610f121..4d76727edb 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote ref updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/remote.c b/remote.c
index eafc14cbe7..60d681a885 100644
--- a/remote.c
+++ b/remote.c
@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 		 * with the remote-tracking branch to find the value
 		 * to expect, but we did not have such a tracking
 		 * branch.
+		 *
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
 		 */
 		if (ref->expect_old_sha1) {
 			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
 				reject_reason = REF_STATUS_REJECT_STALE;
+			else if (ref->if_includes && ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
 			else
-				/* If the ref isn't stale then force the update. */
+				/*
+				 * If the ref isn't stale, and is reachable
+				 * from from one of the reflog entries of
+				 * the local branch, force the update.
+				 */
 				force_ref_update = 1;
 		}
 
@@ -2268,6 +2279,70 @@ static int remote_tracking(struct remote *remote, const char *refname,
 	return 0;
 }
 
+/* Checks if the ref exists in the reflog entry. */
+static int reflog_entry_exists(struct object_id *o_oid,
+				  struct object_id *n_oid,
+				  const char *ident, timestamp_t timestamp,
+				  int tz, const char *message, void *cb_data)
+{
+	struct object_id *remote_oid = cb_data;
+	return oideq(n_oid, remote_oid);
+}
+
+/* Checks if the ref is reachable from the reflog entry. */
+static int reflog_entry_reachable(struct object_id *o_oid,
+			       struct object_id *n_oid,
+			       const char *ident, timestamp_t timestamp,
+			       int tz, const char *message, void *cb_data)
+{
+	struct commit *local_commit;
+	struct commit *remote_commit = cb_data;
+
+	local_commit = lookup_commit_reference(the_repository, n_oid);
+	if (local_commit)
+		return in_merge_bases(remote_commit, local_commit);
+
+	return 0;
+}
+
+/*
+ * Iterate through he reflog entries of the local branch to check
+ * if the remote-tracking ref exists in on of the entries; if not,
+ * go through the entries once more, but this time check if the
+ * remote-tracking ref is reachable from any of the entries.
+ */
+static int is_reachable_in_reflog(const char *local_ref_name,
+				  const struct object_id *remote_oid)
+{
+	struct commit *remote_commit;
+
+	if (for_each_reflog_ent_reverse(local_ref_name, reflog_entry_exists,
+					(struct object_id *)remote_oid))
+		return 1;
+
+	remote_commit = lookup_commit_reference(the_repository, remote_oid);
+	if (remote_commit)
+		return for_each_reflog_ent_reverse(local_ref_name,
+						   reflog_entry_reachable,
+						   remote_commit);
+	return 0;
+}
+
+/*
+ * Check for reachability of a remote-tracking
+ * ref in the reflog entries of its local ref.
+ */
+static void check_if_includes_upstream(struct ref *remote_ref)
+{
+	struct ref *local_ref = get_local_ref(remote_ref->name);
+
+	if (!local_ref)
+		return;
+
+	if (!is_reachable_in_reflog(local_ref->name, &remote_ref->old_oid))
+		remote_ref->unreachable = 1;
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
@@ -2284,6 +2359,8 @@ static void apply_cas(struct push_cas_option *cas,
 			oidcpy(&ref->old_oid_expect, &entry->expect);
 		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 			oidclr(&ref->old_oid_expect);
+		else
+			ref->if_includes = cas->use_force_if_includes;
 		return;
 	}
 
@@ -2294,6 +2371,8 @@ static void apply_cas(struct push_cas_option *cas,
 	ref->expect_old_sha1 = 1;
 	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 		oidclr(&ref->old_oid_expect);
+	else
+		ref->if_includes = cas->use_force_if_includes;
 }
 
 void apply_push_cas(struct push_cas_option *cas,
@@ -2301,6 +2380,15 @@ void apply_push_cas(struct push_cas_option *cas,
 		    struct ref *remote_refs)
 {
 	struct ref *ref;
-	for (ref = remote_refs; ref; ref = ref->next)
+	for (ref = remote_refs; ref; ref = ref->next) {
 		apply_cas(cas, remote, ref);
+
+		/*
+		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
+		 * mode, and if "--foce-if-includes" was specified, run
+		 * the check.
+		 */
+		if (ref->if_includes)
+			check_if_includes_upstream(ref);
+	}
 }
diff --git a/remote.h b/remote.h
index 5e3ea5a26d..38ab8539e2 100644
--- a/remote.h
+++ b/remote.h
@@ -104,7 +104,9 @@ struct ref {
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		if_includes:1, /* If "--force-with-includes" was specified. */
+		unreachable:1; /* For "if_includes"; unreachable in reflog. */
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -134,6 +136,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -332,6 +335,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);
 
 struct push_cas_option {
 	unsigned use_tracking_for_rest:1;
+	unsigned use_force_if_includes:1;
 	struct push_cas {
 		struct object_id expect;
 		unsigned use_tracking:1;
diff --git a/send-pack.c b/send-pack.c
index 632f1580ca..956306e8e8 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/transport-helper.c b/transport-helper.c
index c52c99d829..e547e21199 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote ref updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
diff --git a/transport.c b/transport.c
index 43e24bf1e5..99fe6233a3 100644
--- a/transport.c
+++ b/transport.c
@@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "stale info", porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote ref updated since checkout",
+				 porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -1101,6 +1106,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
-- 
2.28.0

[PATCH v4 2/3] push: parse and set flag for "--force-if-includes"

[Srinidhi Kaushik]

Adds a flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES" to indicate that the new
option was passed from the command line of via configuration settings;
update command line and configuration parsers to set the new flag
accordingly.

Introduces a new configuration option "push.useForceIfIncludes", which
is equivalent to setting "--force-if-includes" in the command line.

Updates "remote-curl" to recognize and pass this option to "send-pack"
when enabled.

Updates "advise" to catch the reject reason "REJECT_REF_NEEDS_UPDATE",
which is set when the ref status is "REF_STATUS_REJECT_REMOTE_UPDATED"
and (optionally) print a help message when the push fails.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 advice.c            |  3 +++
 advice.h            |  2 ++
 builtin/push.c      | 27 +++++++++++++++++++++++++++
 builtin/send-pack.c |  6 ++++++
 remote-curl.c       | 14 +++++++++++++-
 remote.c            |  5 +++++
 remote.h            |  6 ++++++
 transport-helper.c  |  6 ++++++
 transport.c         |  2 ++
 transport.h         | 15 ++++++++++-----
 10 files changed, 80 insertions(+), 6 deletions(-)

diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index 0eeb2c8dd5..c007b19360 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git pull ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
 
 	return 1;
@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		if (!v)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
+	} else if (!strcmp(k, "push.useforceifincludes")) {
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		return 0;
 	}
 
 	return git_default_config(k, v, NULL);
@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
@@ -593,6 +617,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		set_refspecs(argv + 1, argc - 1, repo);
 	}
 
+	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
+		push_set_force_if_includes(&cas);
+
 	remote = pushremote_get(repo);
 	if (!remote) {
 		if (repo)
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 4d76727edb..9289c0eecb 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -159,6 +159,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int progress = -1;
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
+	unsigned int force_if_includes = 0;
 	struct packet_reader reader;
 
 	struct option options[] = {
@@ -184,6 +185,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 		  N_("require old value of ref to be at this value"),
 		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
+		OPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,
+			 N_("require remote updates to be integrated locally")),
 		OPT_END()
 	};
 
@@ -282,6 +285,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	if (match_push_refs(local_refs, &remote_refs, &rs, flags))
 		return -1;
 
+	if (force_if_includes)
+		push_set_force_if_includes(&cas);
+
 	if (!is_empty_cas(&cas))
 		apply_push_cas(&cas, remote, remote_refs);
 
diff --git a/remote-curl.c b/remote-curl.c
index 32cc4a0c55..0290b04891 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -44,7 +44,8 @@ struct options {
 		from_promisor : 1,
 
 		atomic : 1,
-		object_format : 1;
+		object_format : 1,
+		force_if_includes : 1;
 	const struct git_hash_algo *hash_algo;
 };
 static struct options options;
@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)
 		string_list_append(&cas_options, val.buf);
 		strbuf_release(&val);
 		return 0;
+	} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {
+		if (!strcmp(value, "true"))
+			options.force_if_includes = 1;
+		else if (!strcmp(value, "false"))
+			options.force_if_includes = 0;
+		else
+			return -1;
+		return 0;
 	} else if (!strcmp(name, "cloning")) {
 		if (!strcmp(value, "true"))
 			options.cloning = 1;
@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)
 		strvec_push(&args, cas_option->string);
 	strvec_push(&args, url.buf);
 
+	if (options.force_if_includes)
+		strvec_push(&args, "--force-if-includes");
+
 	strvec_push(&args, "--stdin");
 	for (i = 0; i < nr_spec; i++)
 		packet_buf_write(&preamble, "%s\n", specs[i]);
diff --git a/remote.c b/remote.c
index 60d681a885..7679be9213 100644
--- a/remote.c
+++ b/remote.c
@@ -2255,6 +2255,11 @@ int parseopt_push_cas_option(const struct option *opt, const char *arg, int unse
 	return parse_push_cas_option(opt->value, arg, unset);
 }
 
+void push_set_force_if_includes(struct push_cas_option *cas)
+{
+	cas->use_force_if_includes = 1;
+}
+
 int is_empty_cas(const struct push_cas_option *cas)
 {
 	return !cas->use_tracking_for_rest && !cas->nr;
diff --git a/remote.h b/remote.h
index 38ab8539e2..72c374d539 100644
--- a/remote.h
+++ b/remote.h
@@ -350,4 +350,10 @@ int parseopt_push_cas_option(const struct option *, const char *arg, int unset);
 int is_empty_cas(const struct push_cas_option *);
 void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);
 
+/*
+ * Sets "use_force_if_includes" for "compare-and-swap"
+ * when "--force-if-includes" is specified.
+ */
+void push_set_force_if_includes(struct push_cas_option *);
+
 #endif
diff --git a/transport-helper.c b/transport-helper.c
index e547e21199..2a4436dd79 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -868,6 +868,12 @@ static void set_common_push_options(struct transport *transport,
 		if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
 			die(_("helper %s does not support --atomic"), name);
 
+	/* If called with "--force-if-includes". */
+	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
+		if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
+			die(_("helper %s does not support --%s"),
+			    name, TRANS_OPT_FORCE_IF_INCLUDES);
+
 	if (flags & TRANSPORT_PUSH_OPTIONS) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, transport->push_options)
diff --git a/transport.c b/transport.c
index 99fe6233a3..da98ebf50e 100644
--- a/transport.c
+++ b/transport.c
@@ -664,6 +664,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
diff --git a/transport.h b/transport.h
index ca409ea1e4..24558c027d 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);
 /* Request atomic (all-or-nothing) updates when pushing */
 #define TRANS_OPT_ATOMIC "atomic"
 
+/* Require remote changes to be integrated locally. */
+#define TRANS_OPT_FORCE_IF_INCLUDES "force-if-includes"
+
 /**
  * Returns 0 if the option was used, non-zero otherwise. Prints a
  * message to stderr if the option is not used.
@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v4 3/3] t, doc: update tests, reference for "--force-if-includes"

[Srinidhi Kaushik]

* t/t5533-push-cas.sh:
    Updates test cases for "compare-and-swap" when used along with
    "--force-if-includes" helps mitigate overwrites when remote
    ref are updated in the background.

* Documentation:
    Adds reference for the new option, configuration setting
    ("push.useForceIfIncludes") and advise messages.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt |  9 ++++--
 Documentation/config/push.txt   |  5 ++++
 Documentation/git-push.txt      | 20 +++++++++++++
 t/t5533-push-cas.sh             | 53 +++++++++++++++++++++++++++++++++
 4 files changed, 84 insertions(+), 3 deletions(-)

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..acbd0c09aa 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -10,9 +10,8 @@ advice.*::
 		that the check is disabled.
 	pushUpdateRejected::
 		Set this variable to 'false' if you want to disable
-		'pushNonFFCurrent',
-		'pushNonFFMatching', 'pushAlreadyExists',
-		'pushFetchFirst', and 'pushNeedsForce'
+		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
+		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
 		simultaneously.
 	pushNonFFCurrent::
 		Advice shown when linkgit:git-push[1] fails due to a
@@ -41,6 +40,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..6e57a27608 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,8 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.useForceIfIncludes::
+	If set to "true", it is equivalent to specifying "--force-if-includes"
+	as an argument to linkgit:git-push[1]. Adding "--no-force-if-includes"
+	the time of push overrides this configuration setting.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..dc1215a5ad 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -320,6 +320,13 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying "--force-if-includes" an an ancillary option along
+with "--force-with-lease[=<refname>]" (i.e., without saying what exact commit
+the ref on the remote side must be pointing at, or which refs on the remote
+side are being protected) at the time of "push" will verify if updates from the
+remote-tracking refs that may have been implicitly updated in the background
+are integrated locally before allowing a forced update.
 
 -f::
 --force::
@@ -341,6 +348,19 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option verifies if the tip of the remote-tracking ref on which
+a local branch has based on (for a rewrite), is reachable from at
+least one of the "reflog" entries of the local branch about to be
+updated by force on the remote. The check ensures that any updates
+from the remote have been incorporated locally by rejecting a push
+if that is not the case.
++
+Specifying "--no-force-if-includes" disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..60dd7d2f1c 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -256,4 +256,57 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates of REMOTE can be mitigated with "--force-if-includes"' '
+	rm -rf src dst &&
+	git init --bare src.bare &&
+	test_when_finished "rm -rf src.bare" &&
+	git clone --no-local src.bare dst &&
+	test_when_finished "rm -rf dst" &&
+	(
+		cd dst &&
+		test_commit G &&
+		git push origin master:master
+	) &&
+	git clone --no-local src.bare dst2 &&
+	test_when_finished "rm -rf dst2" &&
+	(
+		cd dst2 &&
+		test_commit H &&
+		git push
+	) &&
+	(
+		cd dst &&
+		test_commit I &&
+		git fetch origin &&
+		test_must_fail git push --force-with-lease --force-if-includes origin
+	)
+'
+
+test_expect_success 'background updates of REMOTE can be mitigated with "push.useForceIfIncludes"' '
+	rm -rf src dst &&
+	git init --bare src.bare &&
+	test_when_finished "rm -rf src.bare" &&
+	git clone --no-local src.bare dst &&
+	test_when_finished "rm -rf dst" &&
+	(
+		cd dst &&
+		test_commit G &&
+		git push origin master:master
+	) &&
+	git clone --no-local src.bare dst2 &&
+	test_when_finished "rm -rf dst2" &&
+	(
+		cd dst2 &&
+		test_commit H &&
+		git push
+	) &&
+	(
+		cd dst &&
+		test_commit I &&
+		git fetch origin &&
+		git config --local push.useForceIfIncludes "true" &&
+		test_must_fail git push --force-if-includes origin
+	)
+'
+
 test_done
-- 
2.28.0

Re: [PATCH] push: make `--force-with-lease[=<ref>]` safer

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> No, I don't think that would suffice. For several reasons:
>
> - the user could have checked out the remote-tracking branch directly
>   (`git switch --detach origin/my-branch`), as opposed to actually
>   integrating the revision into the branch at some stage. I know that I do
>   that sometimes.
>
> - even if the remote-tracking branch has not been checked out directly, it
>   might have been `git pull --rebase`d into a _different_ branch, in which
>   case the reflog of `HEAD` would say "yep, I saw this commit!" but that
>   would not mean that it was integrated into the branch we want to
>   (force-)push.
>
> - the reflog of the `HEAD` is worktree-specific, and if the user pushes
>   from a different worktree (e.g. to push multiple branches at the same
>   time, or because the push is scripted and run from a different working
>   directory), it would be missed.
>
> - and if we take a little step back, we do see that the reflog of `HEAD`
>   _does_ answer a different question from what we actually asked.
>
> Sure, it would be faster, but is that worth the consequences?

It is not about "would be faster" but is trying to be more inclusive
of different workflows.

One minor downside consequence of not looking at the reflog of HEAD
is that it does not make the new option useful for those who could
benefit from it.

They detach HEAD at the tip of the remote-tracking branch, work on
the replacement history while on the detached HEAD, and then force
update the local branch from there once they are done (this is a
variant to the first one you said you "do sometimes").  This is a
handy technique to allow them to always compare the previous round
of their effort (which is untouched while they prepare the new
iteration on detached HEAD) with what they achieved so far on HEAD.

After finishing, they would go on to deal with other branches, so
the local branch prepared to be force-pushed may no longer be the
current one.  Without checking with the reflog of HEAD, there is no
way to notice that the local branch to be force-pushed (which no
longer is the current branch) was built using the commit at the tip
of the remote-tracking branch we are losing, because the branch's
reflog would have only one entry that records where it was before
adjusting for the updated remote-tracking branch, and where it is
now after rebuilding.

It is inevitable for a heuristics like the one that was proposed to
work only for those with some workflows while excluding those with
some other workflows, and that is why I said it is "minor" downside
that it does not make the feature useful for some people who could
potentially benefit and also that is why I think it is perfectly OK
to draw the line there and not support them.  After all, the line
has to be drawn somewhere, and the looser the heuristics are, the
wider the potential audience *and* at the same time the wider the
potential of false positives.

Where we drew the line and its consequences must be communicated
clearly in the documentation, though.

Thanks.

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Adds a check to verify if the remote-tracking ref of the local branch
> is reachable from one of its "reflog" entries.

s/Adds/Add/

> When "--force-with-includes" is used along with "--force-with-lease",

A misspelt name for the new option is found here.

> +/* Checks if the ref is reachable from the reflog entry. */
> +static int reflog_entry_reachable(struct object_id *o_oid,
> +			       struct object_id *n_oid,
> +			       const char *ident, timestamp_t timestamp,
> +			       int tz, const char *message, void *cb_data)
> +{
> +	struct commit *local_commit;
> +	struct commit *remote_commit = cb_data;
> +
> +	local_commit = lookup_commit_reference(the_repository, n_oid);
> +	if (local_commit)
> +		return in_merge_bases(remote_commit, local_commit);
> +
> +	return 0;
> +}

Makes me wonder, if in_merge_bases() is so expensive that it makes
sense to split the "were we exactly at the tip?" and "is one of the
commits we were at a descendant of the tip?" into separate phases,
if this part should be calling in_merge_bases() one by one.

Would it make more sense to iterate over reflog entries from newer
to older, collect them in an array of pointers to "struct commit" in
a batch of say 8 commits or less, and then ask in_merge_bases_many()
if the remote_commit is an ancestor of one of these local commits?

The traversal cost to start from one "local commit" to see if
remote_commit is an ancestor of it using in_merge_bases() and
in_merge_bases_many() should be the same and an additional traversal
cost to start from more local commits should be negligible compared
to the traversal itself, so making a call to in_merge_bases() for
each local_commit smells somewhat suboptimal.

If we were talking about older parts of the history, optional
generation numbers could change the equation somewhat, but the
common case for the workflow this series is trying to help is that
these local commits ane the remote tip are relatively new and it is
not unlikely that the generation numbers have not been computed for
them, which is why I suspect that in_merges_many may be a win.

> @@ -2301,6 +2380,15 @@ void apply_push_cas(struct push_cas_option *cas,
>  		    struct ref *remote_refs)
>  {
>  	struct ref *ref;
> -	for (ref = remote_refs; ref; ref = ref->next)
> +	for (ref = remote_refs; ref; ref = ref->next) {
>  		apply_cas(cas, remote, ref);
> +
> +		/*
> +		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
> +		 * mode, and if "--foce-if-includes" was specified, run
> +		 * the check.
> +		 */
> +		if (ref->if_includes)
> +			check_if_includes_upstream(ref);

s/foce/force/; 

I can see that the code is checking "and if force-if-includes was
specified" part, but it is not immediately clear where the code
checks if "--force-with-lease" is used with "tracking" and not with
"the other side must be exactly this commit" mode here.

    ... goes and looks ...

Ah, ok, I found out. 

The field name "if_includes", and the comment for the field in
remote.h, are both misleading.  It gives an impression that the
field being true means "--force-if-included is in use", but in
reality the field means a lot more.  When it is true, it signals
that "--force-if-included" is in use *and* for this ref we were told
to use the "--force-with-lease" without an exact object name.  And
that logic is not here, but has already happened in apply_cas().

Which makes the above comment correct.  We however need a better
name for this field and/or an explanation for the field in the
header file, or both, to avoid misleading readers.

> diff --git a/remote.h b/remote.h
> index 5e3ea5a26d..38ab8539e2 100644
> --- a/remote.h
> +++ b/remote.h
> @@ -104,7 +104,9 @@ struct ref {
>  		forced_update:1,
>  		expect_old_sha1:1,
>  		exact_oid:1,
> -		deletion:1;
> +		deletion:1,
> +		if_includes:1, /* If "--force-with-includes" was specified. */

The description needs to be tightened.

> +		unreachable:1; /* For "if_includes"; unreachable in reflog. */


Thanks.

Re: [PATCH v4 2/3] push: parse and set flag for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Adds a flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES" to indicate that the new
> option was passed from the command line of via configuration settings;
> update command line and configuration parsers to set the new flag
> accordingly.

s/Adds/Add/;

> Introduces a new configuration option "push.useForceIfIncludes", which
> is equivalent to setting "--force-if-includes" in the command line.

s/Introduces/Introduce/; (I won't repeat).

>
> Updates "remote-curl" to recognize and pass this option to "send-pack"
> when enabled.
>
> Updates "advise" to catch the reject reason "REJECT_REF_NEEDS_UPDATE",
> which is set when the ref status is "REF_STATUS_REJECT_REMOTE_UPDATED"
> and (optionally) print a help message when the push fails.

All of the above say what were done.  A summarizing sentence before
all of the above would make the proposed commit log message perfect,
perhaps:

    The previous step added the necessary machinery to implement the
    "--force-if-includes" protection, when "--force-with-lease" is
    used without giving exact object the remote still ought to have.
    Surface the feature by adding a command line option and a
    configuration variable to enable it.

    - Add a flag ... to indicate that ...

    - Introduce a configuration option ...

    - Update 'remote-curl' to ...

    ...


Also, in the proposed log message for [1/3], especially near its
end, how "--force-if-includes" interacts with "--force-with-lease"
was described.  The description should be added to the log message
of this change, as it is what introduces the end-user facing
feature.  The description can also be in the log for [1/3] as well,
but not having it here for [2/3] is unfriendly to the readers.

> diff --git a/builtin/send-pack.c b/builtin/send-pack.c
> index 4d76727edb..9289c0eecb 100644
> --- a/builtin/send-pack.c
> +++ b/builtin/send-pack.c
> @@ -159,6 +159,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
>  	int progress = -1;
>  	int from_stdin = 0;
>  	struct push_cas_option cas = {0};
> +	unsigned int force_if_includes = 0;

I think OPT_BOOL takes a pointer to int, not unsigned, as it is
OPT_SET_INT in disguise, and you can see that a near-by 'progress'
that also is fed to OPT_BOOL() is 'int' so you can mimic it.

> @@ -184,6 +185,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
>  		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
>  		  N_("require old value of ref to be at this value"),
>  		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
> +		OPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,
> +			 N_("require remote updates to be integrated locally")),
>  		OPT_END()
>  	};

> diff --git a/remote.h b/remote.h
> index 38ab8539e2..72c374d539 100644
> --- a/remote.h
> +++ b/remote.h
> @@ -350,4 +350,10 @@ int parseopt_push_cas_option(const struct option *, const char *arg, int unset);
>  int is_empty_cas(const struct push_cas_option *);
>  void apply_push_cas(struct push_cas_option *, struct remote *, struct ref *);
>  
> +/*
> + * Sets "use_force_if_includes" for "compare-and-swap"
> + * when "--force-if-includes" is specified.
> + */
> +void push_set_force_if_includes(struct push_cas_option *);

Let's not add this helper function.  Instead just open-code a single
liner at its two callers.  It makes it easier to read and understand
the flow and the logic in cmd_push() and cmd_send_pack().

> diff --git a/transport-helper.c b/transport-helper.c
> index e547e21199..2a4436dd79 100644
> --- a/transport-helper.c
> +++ b/transport-helper.c
> @@ -868,6 +868,12 @@ static void set_common_push_options(struct transport *transport,
>  		if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
>  			die(_("helper %s does not support --atomic"), name);
>  
> +	/* If called with "--force-if-includes". */

The comment does not add any value as you are already using a
descriptive constant name.  Drop it to follow suit of existing if
statements nearby.

> +	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
> +		if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
> +			die(_("helper %s does not support --%s"),
> +			    name, TRANS_OPT_FORCE_IF_INCLUDES);
> +
>  	if (flags & TRANSPORT_PUSH_OPTIONS) {
>  		struct string_list_item *item;
>  		for_each_string_list_item(item, transport->push_options)

Thanks.

Re: [PATCH v4 3/3] t, doc: update tests, reference for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Subject: Re: [PATCH v4 3/3] t, doc: update tests, reference for "--force-if-includes"

good.

> * t/t5533-push-cas.sh:
>     Updates test cases for "compare-and-swap" when used along with
>     "--force-if-includes" helps mitigate overwrites when remote
>     ref are updated in the background.
>
> * Documentation:
>     Adds reference for the new option, configuration setting
>     ("push.useForceIfIncludes") and advise messages.

s/Updates/Update/; s/Adds/Add/;

> +push.useForceIfIncludes::
> +	If set to "true", it is equivalent to specifying "--force-if-includes"
> +	as an argument to linkgit:git-push[1]. Adding "--no-force-if-includes"

s/as an argument to/on the command line of/ would be better.

Some readers differenciate arguments and options and we are
discussing an option, not an argument.

> +Alternatively, specifying "--force-if-includes" an an ancillary option along
> +with "--force-with-lease[=<refname>]" (i.e., without saying what exact commit
> +the ref on the remote side must be pointing at, or which refs on the remote
> +side are being protected) at the time of "push" will verify if updates from the
> +remote-tracking refs that may have been implicitly updated in the background
> +are integrated locally before allowing a forced update.

OK.  A user who wants to know more can refer to --force-if-includes
from here pretty easily.

> @@ -341,6 +348,19 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
>  origin +master` to force a push to the `master` branch). See the
>  `<refspec>...` section above for details.
>  
> +--[no-]force-if-includes::
> +	Force an update only if the tip of the remote-tracking ref
> +	has been integrated locally.
> ++
> +This option verifies if the tip of the remote-tracking ref on which
> +a local branch has based on (for a rewrite), is reachable from at
> +least one of the "reflog" entries of the local branch about to be
> +updated by force on the remote.

The latter half of this sentence is quite a mouthful, and after
reading it three times, it is not quite clear.

> +The check ensures that any updates
> +from the remote have been incorporated locally by rejecting a push
> +if that is not the case.

OK.

> ++
> +Specifying "--no-force-if-includes" disables this behavior.

Do we want to add:

    It is a no-op unless "--force-with-lease[=<refname>]" without exact
    object name is used at the same time.

here or somewhere nearby?


> +test_expect_success 'background updates of REMOTE can be mitigated with "--force-if-includes"' '
> +	rm -rf src dst &&
> +	git init --bare src.bare &&
> +	test_when_finished "rm -rf src.bare" &&
> +	git clone --no-local src.bare dst &&
> +	test_when_finished "rm -rf dst" &&
> +	(
> +		cd dst &&
> +		test_commit G &&
> +		git push origin master:master
> +	) &&
> +	git clone --no-local src.bare dst2 &&
> +	test_when_finished "rm -rf dst2" &&
> +	(
> +		cd dst2 &&
> +		test_commit H &&
> +		git push
> +	) &&
> +	(
> +		cd dst &&
> +		test_commit I &&
> +		git fetch origin &&
> +		test_must_fail git push --force-with-lease --force-if-includes origin

I briefly wondered if it makes sense to also check if
--force-with-lease alone (or with --no-force-if-includes)
successfully pushes in this case, but I think we are OK without such
a test.  After all, we won't test "--force" alone, either, as we
expect that to work (and should be tested elsewhere).

> +	)
> +'
> +
> +test_expect_success 'background updates of REMOTE can be mitigated with "push.useForceIfIncludes"' '
> +	rm -rf src dst &&
> +	git init --bare src.bare &&
> +	test_when_finished "rm -rf src.bare" &&
> +	git clone --no-local src.bare dst &&
> +	test_when_finished "rm -rf dst" &&
> +	(
> +		cd dst &&
> +		test_commit G &&
> +		git push origin master:master
> +	) &&
> +	git clone --no-local src.bare dst2 &&
> +	test_when_finished "rm -rf dst2" &&
> +	(
> +		cd dst2 &&
> +		test_commit H &&
> +		git push
> +	) &&
> +	(
> +		cd dst &&
> +		test_commit I &&
> +		git fetch origin &&
> +		git config --local push.useForceIfIncludes "true" &&
> +		test_must_fail git push --force-if-includes origin

I am not sure what is tested here.  I thought with or without the
configuration variable, the feature is a no-op unless a lazy
force-with-lease is in use?

Perhaps you meant to test

		test_must_fail git push --force-with-lease origin

instead?

> +	)
> +'
> +
>  test_done

Thanks.

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Hi Junio,

On 09/19/2020 13:03, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> > Adds a check to verify if the remote-tracking ref of the local branch
> > is reachable from one of its "reflog" entries.
> 
> s/Adds/Add/

Gotcha, I will reword the commit messages for all three commits.
in the patch series.

> > When "--force-with-includes" is used along with "--force-with-lease",
> 
> A misspelt name for the new option is found here.

*Facepalm.* Thanks, will update.

> > [...]
> Makes me wonder, if in_merge_bases() is so expensive that it makes
> sense to split the "were we exactly at the tip?" and "is one of the
> commits we were at a descendant of the tip?" into separate phases,
> if this part should be calling in_merge_bases() one by one.
> 
> Would it make more sense to iterate over reflog entries from newer
> to older, collect them in an array of pointers to "struct commit" in
> a batch of say 8 commits or less, and then ask in_merge_bases_many()
> if the remote_commit is an ancestor of one of these local commits?
>
> The traversal cost to start from one "local commit" to see if
> remote_commit is an ancestor of it using in_merge_bases() and
> in_merge_bases_many() should be the same and an additional traversal
> cost to start from more local commits should be negligible compared
> to the traversal itself, so making a call to in_merge_bases() for
> each local_commit smells somewhat suboptimal.
> 
> If we were talking about older parts of the history, optional
> generation numbers could change the equation somewhat, but the
> common case for the workflow this series is trying to help is that
> these local commits ane the remote tip are relatively new and it is
> not unlikely that the generation numbers have not been computed for
> them, which is why I suspect that in_merges_many may be a win.

Nice! We can definitely try batching commits from the reflog and
pass it along to "in_merge_bases_many()". As for being faster than
calling "in_merge_bases()" for each commit entry in the reflog --
I am not familiar with how the former works. Do we still keep the
"reflog_entry_exists()" part? It might still be faster to go through
the entries once to check with "oideq()" in the first run.

Also, I was wondering if it is worth considering this:
  - check if the reflog of the HEAD has the remote ref
  - check if the reflog of the local branch has the remote ref
  - check if the remote ref is reachable from any of the local ref's
    "reflog" entries using "in_merge_bases_many()" in batches as
    suggested here.

The first two (we can even skip the second one) runs are relatively
fast, and the third one might be faster than checking "in_merge_bases()"
for each reflog entry. I suppose adding these three steps would make
the process slower overall, though. For context, I was referring to
your message [1] on the other thread regarding checking the HEAD's
reflog.

> > [...]
> > +		/*
> > +		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
> > +		 * mode, and if "--foce-if-includes" was specified, run
> > +		 * the check.
> > +		 */
> > +		if (ref->if_includes)
> > +			check_if_includes_upstream(ref);
> 
> s/foce/force/; 

Yes, sorry about that; will update.
 
> I can see that the code is checking "and if force-if-includes was
> specified" part, but it is not immediately clear where the code
> checks if "--force-with-lease" is used with "tracking" and not with
> "the other side must be exactly this commit" mode here.
> 
>     ... goes and looks ...
> 
> Ah, ok, I found out. 
> 
> The field name "if_includes", and the comment for the field in
> remote.h, are both misleading.  It gives an impression that the
> field being true means "--force-if-included is in use", but in
> reality the field means a lot more.  When it is true, it signals
> that "--force-if-included" is in use *and* for this ref we were told
> to use the "--force-with-lease" without an exact object name.  And
> that logic is not here, but has already happened in apply_cas().
> 
> Which makes the above comment correct.  We however need a better
> name for this field and/or an explanation for the field in the
> header file, or both, to avoid misleading readers.
>
> > diff --git a/remote.h b/remote.h
> > index 5e3ea5a26d..38ab8539e2 100644
> > --- a/remote.h
> > +++ b/remote.h
> > @@ -104,7 +104,9 @@ struct ref {
> >  		forced_update:1,
> >  		expect_old_sha1:1,
> >  		exact_oid:1,
> > -		deletion:1;
> > +		deletion:1,
> > +		if_includes:1, /* If "--force-with-includes" was specified. */
> 
> The description needs to be tightened.
> 
> > +		unreachable:1; /* For "if_includes"; unreachable in reflog. */

OK, you're right. Perhaps, we could rename it to something like
"if_includes_for_tracking" and update the comment description
with saying something along the lines of:

+  /*
+   * Set when "--force-if-includes" is enabled, and
+   * if "compare-and-swap" is not provided with the
+   * exact commit to be expected on the remote (in
+   * "use_tracking" or use_tracking_for_rest" mode).
+   */


[1]: https://public-inbox.org/git/xmqqsgbdk69b.fsf@gitster.c.googlers.com

Thanks again, for taking the time to review this.
-- 
Srinidhi Kaushik

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Phillip Wood]

On 19/09/2020 21:03, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
>> +/* Checks if the ref is reachable from the reflog entry. */
>> +static int reflog_entry_reachable(struct object_id *o_oid,
>> +			       struct object_id *n_oid,
>> +			       const char *ident, timestamp_t timestamp,
>> +			       int tz, const char *message, void *cb_data)
>> +{
>> +	struct commit *local_commit;
>> +	struct commit *remote_commit = cb_data;
>> +
>> +	local_commit = lookup_commit_reference(the_repository, n_oid);
>> +	if (local_commit)
>> +		return in_merge_bases(remote_commit, local_commit);
>> +
>> +	return 0;
>> +}
> 
> Makes me wonder, if in_merge_bases() is so expensive that it makes
> sense to split the "were we exactly at the tip?" and "is one of the
> commits we were at a descendant of the tip?" into separate phases,
> if this part should be calling in_merge_bases() one by one.
> 
> Would it make more sense to iterate over reflog entries from newer
> to older, collect them in an array of pointers to "struct commit" in
> a batch of say 8 commits or less, and then ask in_merge_bases_many()
> if the remote_commit is an ancestor of one of these local commits?

As I said before[1] I think we should also be checking the reflog dates 
so that we do not look at any local reflog entries that are older than 
the most recent reflog entry for the remote tracking branch. This 
protects against a background fetch when the remote has been rewound and 
it would also reduce the number of calls to in_merge_bases().

Best Wishes

Phillip

[1] 
https://lore.kernel.org/git/624d9e35-29b8-4012-a3d6-e9b00a9e4485@gmail.com/

> The traversal cost to start from one "local commit" to see if
> remote_commit is an ancestor of it using in_merge_bases() and
> in_merge_bases_many() should be the same and an additional traversal
> cost to start from more local commits should be negligible compared
> to the traversal itself, so making a call to in_merge_bases() for
> each local_commit smells somewhat suboptimal.
> 
> If we were talking about older parts of the history, optional
> generation numbers could change the equation somewhat, but the
> common case for the workflow this series is trying to help is that
> these local commits ane the remote tip are relatively new and it is
> not unlikely that the generation numbers have not been computed for
> them, which is why I suspect that in_merges_many may be a win.
> 
>> @@ -2301,6 +2380,15 @@ void apply_push_cas(struct push_cas_option *cas,
>>   		    struct ref *remote_refs)
>>   {
>>   	struct ref *ref;
>> -	for (ref = remote_refs; ref; ref = ref->next)
>> +	for (ref = remote_refs; ref; ref = ref->next) {
>>   		apply_cas(cas, remote, ref);
>> +
>> +		/*
>> +		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
>> +		 * mode, and if "--foce-if-includes" was specified, run
>> +		 * the check.
>> +		 */
>> +		if (ref->if_includes)
>> +			check_if_includes_upstream(ref);
> 
> s/foce/force/;
> 
> I can see that the code is checking "and if force-if-includes was
> specified" part, but it is not immediately clear where the code
> checks if "--force-with-lease" is used with "tracking" and not with
> "the other side must be exactly this commit" mode here.
> 
>      ... goes and looks ...
> 
> Ah, ok, I found out.
> 
> The field name "if_includes", and the comment for the field in
> remote.h, are both misleading.  It gives an impression that the
> field being true means "--force-if-included is in use", but in
> reality the field means a lot more.  When it is true, it signals
> that "--force-if-included" is in use *and* for this ref we were told
> to use the "--force-with-lease" without an exact object name.  And
> that logic is not here, but has already happened in apply_cas().
> 
> Which makes the above comment correct.  We however need a better
> name for this field and/or an explanation for the field in the
> header file, or both, to avoid misleading readers.
> 
>> diff --git a/remote.h b/remote.h
>> index 5e3ea5a26d..38ab8539e2 100644
>> --- a/remote.h
>> +++ b/remote.h
>> @@ -104,7 +104,9 @@ struct ref {
>>   		forced_update:1,
>>   		expect_old_sha1:1,
>>   		exact_oid:1,
>> -		deletion:1;
>> +		deletion:1,
>> +		if_includes:1, /* If "--force-with-includes" was specified. */
> 
> The description needs to be tightened.
> 
>> +		unreachable:1; /* For "if_includes"; unreachable in reflog. */
> 
> 
> Thanks.
> 

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> As I said before[1] I think we should also be checking the reflog
> dates so that we do not look at any local reflog entries that are
> older than the most recent reflog entry for the remote tracking
> branch. This protects against a background fetch when the remote has
> been rewound and it would also reduce the number of calls to
> in_merge_bases().

Meaning we first check the timestamp of the topmost reflog entry of
remote-tracking branch (i.e. the time *we* acquired the tip commit
that we are about to lose), and leverage on the fact that no local
commit older than that timestamp can possibly be written with the
knowledge of that remote work?  Assuming that local timestamp is
monotonically increasing, it is a quite valid optimization (and the
clock skew we often talk about in the context of revision traversal
are often discrepancy between matchines).

Having said that.

The new generation number based on (adjusted) timestamp is being
worked in, and that work is supposed to bring such an optimization
to us automatically (at least on the reachability's side, i.e. logic
that uses get_merge_bases()), I think, so we probably do *not* want
to add such a heuristics specifically for this codepath.

Thanks.

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Meaning we first check the timestamp of the topmost reflog entry of
> remote-tracking branch (i.e. the time *we* acquired the tip commit
> that we are about to lose), and leverage on the fact that no local
> commit older than that timestamp can possibly be written with the
> knowledge of that remote work?  Assuming that local timestamp is
> monotonically increasing, it is a quite valid optimization (and the
> clock skew we often talk about in the context of revision traversal
> are often discrepancy between matchines).
>
> Having said that.
>
> The new generation number based on (adjusted) timestamp is being
> worked in, and that work is supposed to bring such an optimization
> to us automatically (at least on the reachability's side, i.e. logic
> that uses get_merge_bases()), I think, so we probably do *not* want
> to add such a heuristics specifically for this codepath.

Eh, I spoke too soon before I thought it through.  I do not think we
will gain "assume that any commit whose timestamp is older than this
externally given one will never reach the other commit" even with
the reachability index based on (adjusted) timestamp.  At least,
stopping the traversal of reflog entries of the local side at the
timestamp of the topmost reflog entry of remote-tracking branch in
question would be an easy, worthwhile and sensible optimization.

Thanks.

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

>> If we were talking about older parts of the history, optional
>> generation numbers could change the equation somewhat, but the
>> common case for the workflow this series is trying to help is that
>> these local commits ane the remote tip are relatively new and it is
>> not unlikely that the generation numbers have not been computed for
>> them, which is why I suspect that in_merges_many may be a win.
>
> Nice! We can definitely try batching commits from the reflog and
> pass it along to "in_merge_bases_many()". As for being faster than
> calling "in_merge_bases()" for each commit entry in the reflog --
> I am not familiar with how the former works. Do we still keep the
> "reflog_entry_exists()" part? It might still be faster to go through
> the entries once to check with "oideq()" in the first run.

That is what I meant.  You go through local reflog entries until you
find one that is older than the timestamp of the reflog entry of the
remote-tracking branch, check with oideq() to see if the tip was ever
directly checked out.  Then, using these same local reflog entries,
you can make in_merge_bases_many() tranversal to see if any of them
reach the tip.  I suspect that the number of local reflog entries you
need to examine would not be too many, so if you can put them all in
a single array of "struct commit *" pointers in the first "oideq()"
phase, you may be able to do just a single in_merge_bases_many() batch
to check for the reachability.

> Also, I was wondering if it is worth considering this:
>   - check if the reflog of the HEAD has the remote ref

It would help the workflow I had in mind, but it would raise the
risk of false positives according to Dscho and I tend to agree, so
I do not know if it is overall a good idea.

>   - check if the reflog of the local branch has the remote ref

Isn't that the oideq() test?

>   - check if the remote ref is reachable from any of the local ref's
>     "reflog" entries using "in_merge_bases_many()" in batches as
>     suggested here.

I think it amounts to the same as "does any reflog entry of HEAD
reach it?" and shares the same issues with false positives as the
first one.

>> > +		deletion:1,
>> > +		if_includes:1, /* If "--force-with-includes" was specified. */
>> 
>> The description needs to be tightened.
>> 
>> > +		unreachable:1; /* For "if_includes"; unreachable in reflog. */
>
> OK, you're right. Perhaps, we could rename it to something like
> "if_includes_for_tracking" and update the comment description
> with saying something along the lines of:

That is overlong.  Let me try:

		/* need to check if local reflog reaches the remote tip */
		check_reachable:1,

		/* local reflog does not reach the remote tip */
		unreachable:1;

Thans.

[PATCH v5 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: "--force-if-includes" to "git-push" where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref -- on which a local branch has based on -- is reachable from at
least one of the "reflog" entries of the branch about to be updated
by force on the remote.

This option can be used with "--force-with-lease" with setups where
the remote-tracking refs of the repository are implicitly updated in
the background to help prevent unintended remote overwrites.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with "--force-with-lease[=<refname>[:<expect>]]" without specifying
the "<expect>" value, can cause the update that happened in-between the
checkout and forced push to be lost.

Changes since v4:
  - Collect commits during the reflog iteration step and use
    "in_merge_bases_many()" once on the list of collected commits
    instead of using "in_merge_bases()" for every entry [1].

  - Include reflog timestamps to reduce the number of reflog entry
    iterations by stopping early if we see an entry that is older
    than the remote tip's commit date [2]. This, coupled with a
    single call of "in_merge_bases_many()" should help make the
    check faster [3].

  - Reword commit messages, documentation; add a note to explain
    when the option is a "no-op".

  - Fix a mistake in the test script (t5533).

Srinidhi Kaushik (3):
  push: add reflog check for "--force-if-includes"
  push: parse and set flag for "--force-if-includes"
  t, doc: update tests, reference for "--force-if-includes"

 Documentation/config/advice.txt |   9 +-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 +++++-
 advice.c                        |   3 +
 advice.h                        |   2 +
 builtin/push.c                  |  27 ++++++
 builtin/send-pack.c             |  12 +++
 remote-curl.c                   |  14 +++-
 remote.c                        | 140 +++++++++++++++++++++++++++++++-
 remote.h                        |   8 +-
 send-pack.c                     |   1 +
 t/t5533-push-cas.sh             |  53 ++++++++++++
 transport-helper.c              |  10 +++
 transport.c                     |   8 ++
 transport.h                     |  15 ++--
 15 files changed, 321 insertions(+), 13 deletions(-)

base-commit: 385c171a018f2747b329bcfa6be8eda1709e5abd

Thanks to Johannes, Junio and Phillip for continually suggesting
improvements and patiently reviewing the patch series.

[1]: https://public-inbox.org/git/xmqqft7djzz0.fsf@gitster.c.googlers.com/
[2]: https://public-inbox.org/git/f6bb2b1f-0f1b-f196-59f1-893580430cf2@gmail.com/
[3]: https://public-inbox.org/git/xmqqimc7ezk2.fsf@gitster.c.googlers.com/
--
2.28.0

[PATCH v5 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Add a check to verify if the remote-tracking ref of the local branch
is reachable from one of its "reflog" entries.

When a local branch that is based on a remote ref, has been rewound
and is to be force pushed on the remote, "--force-if-includes" runs
a check that ensures any updates to remote-tracking refs that may have
happened (by push from another repository) in-between the time of the
last checkout, and right before the time of push, have been integrated
locally before allowing a forced updated.

A new field "use_force_if_includes" has been added to "push_cas_option",
which is set to "1" when "--force-if-includes" is specified as an
option in the command line or as a configuration setting.

The struct "ref" has two new bit-fields:
  - check_reachable:
    Set when we have to run the new check on the ref, and the remote
    ref was marked as "use_tracking" or "use_tracking_for_rest" by
    compare-and-swap (if the "the remote tip must be at the expected
    commit" condition is not specified); "apply_push_cas()" has been
    updated to check if this field is set and run the check.

  - unreachable:
    Set if the ref is unreachable from any of the "reflog" entries of
    its local counterpart.

"REF_STATUS_REJECT_REMOTE_UPDATED" has been added to the "status"
enum to imply that the ref failed the check; "case" statements in
"send-pack", "transport" and "transport-helper" have been updated
accordingly to catch this status when set.

When "--force-is-includes" is used along with "--force-with-lease",
the check runs only for refs marked as "if_includes". If the option
is passed without specifying "--force-with-lease", or specified along
with "--force-with-lease=<refname>:<expect>" it is a "no-op".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 builtin/send-pack.c |   5 ++
 remote.c            | 140 +++++++++++++++++++++++++++++++++++++++++++-
 remote.h            |   8 ++-
 send-pack.c         |   1 +
 transport-helper.c  |   5 ++
 transport.c         |   6 ++
 6 files changed, 162 insertions(+), 3 deletions(-)

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 2b9610f121..4d76727edb 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote ref updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/remote.c b/remote.c
index eafc14cbe7..0dcac4ab8e 100644
--- a/remote.c
+++ b/remote.c
@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 		 * with the remote-tracking branch to find the value
 		 * to expect, but we did not have such a tracking
 		 * branch.
+		 *
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
 		 */
 		if (ref->expect_old_sha1) {
 			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
 				reject_reason = REF_STATUS_REJECT_STALE;
+			else if (ref->check_reachable && ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
 			else
-				/* If the ref isn't stale then force the update. */
+				/*
+				 * If the ref isn't stale, and is reachable
+				 * from from one of the reflog entries of
+				 * the local branch, force the update.
+				 */
 				force_ref_update = 1;
 		}
 
@@ -2268,6 +2279,118 @@ static int remote_tracking(struct remote *remote, const char *refname,
 	return 0;
 }
 
+/*
+ * The struct "reflog_commit_list" and related helper functions
+ * for list manipulation are used for collecting commits into a
+ * list during reflog traversals in "if_exists_or_grab_until()".
+ */
+struct reflog_commit_list {
+	struct commit **items;
+	size_t nr, alloc;
+};
+
+/* Adds a commit to list. */
+static void add_commit(struct reflog_commit_list *list, struct commit *commit)
+{
+	ALLOC_GROW(list->items, list->nr + 1, list->alloc);
+	list->items[list->nr++] = commit;
+}
+
+/* Free and reset the list. */
+static void free_reflog_commit_list(struct reflog_commit_list *list)
+{
+	FREE_AND_NULL(list->items);
+	list->nr = list->alloc = 0;
+}
+
+struct check_and_collect_until_cb_data {
+	struct commit *remote_commit;
+	struct reflog_commit_list *local_commits;
+};
+
+
+static int check_and_collect_until(struct object_id *o_oid,
+				   struct object_id *n_oid,
+				   const char *ident, timestamp_t timestamp,
+				   int tz, const char *message, void *cb_data)
+{
+	struct commit *commit;
+	struct check_and_collect_until_cb_data *cb = cb_data;
+
+	/*
+	 * If the reflog entry timestamp is older than the
+	 * remote commit date, there is no need to check or
+	 * collect entries older than this one.
+	 */
+	if (timestamp < cb->remote_commit->date)
+		return -1;
+
+	/* An entry was found. */
+	if (oideq(n_oid, &cb->remote_commit->object.oid))
+		return 1;
+
+	/* Lookup the commit and append it to the list. */
+	if ((commit = lookup_commit_reference(the_repository, n_oid)))
+		add_commit(cb->local_commits, commit);
+
+	return 0;
+}
+
+/*
+ * Iterate through the reflog of a local ref to check if there is an entry for
+ * the given remote-tracking ref (i.e., if it was checked out); runs until the
+ * timestamp of an entry is older than the commit date of the remote-tracking
+ * ref. Any commits that seen along the way are collected into a list to check
+ * if the remote-tracking ref is reachable from any of them.
+ */
+static int is_reachable_in_reflog(const char *local_ref_name,
+				  const struct object_id *remote_oid)
+{
+	struct commit *remote_commit;
+	struct reflog_commit_list list = { NULL, 0, 0 };
+	struct check_and_collect_until_cb_data cb;
+	int ret = 0;
+
+	remote_commit = lookup_commit_reference(the_repository, remote_oid);
+	if (!remote_commit)
+		goto cleanup_return;
+
+	cb.remote_commit = remote_commit;
+	cb.local_commits = &list;
+	ret = for_each_reflog_ent_reverse(local_ref_name,
+					  check_and_collect_until, &cb);
+
+	/* We found an entry in the reflog. */
+	if (ret > 0)
+		goto cleanup_return;
+
+	/*
+	 * Check if "remote_commit" is reachable from
+	 * any of the commits in the collected list.
+	 */
+	if (list.nr > 0)
+		ret = in_merge_bases_many(remote_commit, list.nr, list.items);
+
+cleanup_return:
+	free_reflog_commit_list(&list);
+	return ret;
+}
+
+/*
+ * Check for reachability of a remote-tracking
+ * ref in the reflog entries of its local ref.
+ */
+static void check_if_includes_upstream(struct ref *remote_ref)
+{
+	struct ref *local_ref = get_local_ref(remote_ref->name);
+
+	if (!local_ref)
+		return;
+
+	if (!is_reachable_in_reflog(local_ref->name, &remote_ref->old_oid))
+		remote_ref->unreachable = 1;
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
@@ -2284,6 +2407,8 @@ static void apply_cas(struct push_cas_option *cas,
 			oidcpy(&ref->old_oid_expect, &entry->expect);
 		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 			oidclr(&ref->old_oid_expect);
+		else
+			ref->check_reachable = cas->use_force_if_includes;
 		return;
 	}
 
@@ -2294,6 +2419,8 @@ static void apply_cas(struct push_cas_option *cas,
 	ref->expect_old_sha1 = 1;
 	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
 		oidclr(&ref->old_oid_expect);
+	else
+		ref->check_reachable = cas->use_force_if_includes;
 }
 
 void apply_push_cas(struct push_cas_option *cas,
@@ -2301,6 +2428,15 @@ void apply_push_cas(struct push_cas_option *cas,
 		    struct ref *remote_refs)
 {
 	struct ref *ref;
-	for (ref = remote_refs; ref; ref = ref->next)
+	for (ref = remote_refs; ref; ref = ref->next) {
 		apply_cas(cas, remote, ref);
+
+		/*
+		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
+		 * mode, and if "--force-if-includes" was specified, run
+		 * the check.
+		 */
+		if (ref->check_reachable)
+			check_if_includes_upstream(ref);
+	}
 }
diff --git a/remote.h b/remote.h
index 5e3ea5a26d..7c5e59770e 100644
--- a/remote.h
+++ b/remote.h
@@ -104,7 +104,11 @@ struct ref {
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		/* Need to check if local reflog reaches the remote tip. */
+		check_reachable:1,
+		/* The local reflog does not reach the remote tip. */
+		unreachable:1;
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -134,6 +138,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -332,6 +337,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);
 
 struct push_cas_option {
 	unsigned use_tracking_for_rest:1;
+	unsigned use_force_if_includes:1;
 	struct push_cas {
 		struct object_id expect;
 		unsigned use_tracking:1;
diff --git a/send-pack.c b/send-pack.c
index 632f1580ca..956306e8e8 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/transport-helper.c b/transport-helper.c
index c52c99d829..e547e21199 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote ref updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
diff --git a/transport.c b/transport.c
index 43e24bf1e5..99fe6233a3 100644
--- a/transport.c
+++ b/transport.c
@@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "stale info", porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote ref updated since checkout",
+				 porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -1101,6 +1106,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
-- 
2.28.0

[PATCH v5 2/3] push: parse and set flag for "--force-if-includes"

[Srinidhi Kaushik]

The previous commit added the necessary machinery to implement the
"--force-if-includes" protection, when "--force-with-lease" is used
without giving exact object the remote still ought to have. Surface
the feature by adding a command line option and a configuration
variable to enable it.

 - Add a flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES" to indicate that the
   new option was passed from the command line of via configuration
   settings; update command line and configuration parsers to set the
   new flag accordingly.

 - Introduce a new configuration option "push.useForceIfIncludes", which
   is equivalent to setting "--force-if-includes" in the command line.

 - Update "remote-curl" to recognize and pass this option to "send-pack"
   when enabled.

 - Update "advise" to catch the reject reason "REJECT_REF_NEEDS_UPDATE",
   set when the ref status is "REF_STATUS_REJECT_REMOTE_UPDATED" and
   (optionally) print a help message when the push fails.

 - The new option is a "no-op" in the following scenarios:
    * When used without "--force-with-lease".
    * When used with "--force-with-lease", and if the expected commit
      on the remote side is specified as an argument.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 advice.c            |  3 +++
 advice.h            |  2 ++
 builtin/push.c      | 27 +++++++++++++++++++++++++++
 builtin/send-pack.c |  7 +++++++
 remote-curl.c       | 14 +++++++++++++-
 transport-helper.c  |  5 +++++
 transport.c         |  2 ++
 transport.h         | 15 ++++++++++-----
 8 files changed, 69 insertions(+), 6 deletions(-)

diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index 0eeb2c8dd5..908b557edb 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git pull ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
 
 	return 1;
@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		if (!v)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
+	} else if (!strcmp(k, "push.useforceifincludes")) {
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		return 0;
 	}
 
 	return git_default_config(k, v, NULL);
@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 	if ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))
 		die(_("--all and --mirror are incompatible"));
 
+	if (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))
+		cas.use_force_if_includes = 1;
+
 	for_each_string_list_item(item, push_options)
 		if (strchr(item->string, '\n'))
 			die(_("push options must not have new line characters"));
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 4d76727edb..c115a4a1ac 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -159,6 +159,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int progress = -1;
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
+	int force_if_includes = 0;
 	struct packet_reader reader;
 
 	struct option options[] = {
@@ -184,6 +185,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 		  N_("require old value of ref to be at this value"),
 		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
+		OPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,
+			 N_("require remote updates to be integrated locally")),
 		OPT_END()
 	};
 
@@ -285,6 +288,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	if (!is_empty_cas(&cas))
 		apply_push_cas(&cas, remote, remote_refs);
 
+	if (!is_empty_cas(&cas) && force_if_includes)
+		cas.use_force_if_includes = 1;
+
+
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
 
diff --git a/remote-curl.c b/remote-curl.c
index 32cc4a0c55..0290b04891 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -44,7 +44,8 @@ struct options {
 		from_promisor : 1,
 
 		atomic : 1,
-		object_format : 1;
+		object_format : 1,
+		force_if_includes : 1;
 	const struct git_hash_algo *hash_algo;
 };
 static struct options options;
@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)
 		string_list_append(&cas_options, val.buf);
 		strbuf_release(&val);
 		return 0;
+	} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {
+		if (!strcmp(value, "true"))
+			options.force_if_includes = 1;
+		else if (!strcmp(value, "false"))
+			options.force_if_includes = 0;
+		else
+			return -1;
+		return 0;
 	} else if (!strcmp(name, "cloning")) {
 		if (!strcmp(value, "true"))
 			options.cloning = 1;
@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)
 		strvec_push(&args, cas_option->string);
 	strvec_push(&args, url.buf);
 
+	if (options.force_if_includes)
+		strvec_push(&args, "--force-if-includes");
+
 	strvec_push(&args, "--stdin");
 	for (i = 0; i < nr_spec; i++)
 		packet_buf_write(&preamble, "%s\n", specs[i]);
diff --git a/transport-helper.c b/transport-helper.c
index e547e21199..298e814bf7 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -868,6 +868,11 @@ static void set_common_push_options(struct transport *transport,
 		if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
 			die(_("helper %s does not support --atomic"), name);
 
+	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
+		if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
+			die(_("helper %s does not support --%s"),
+			    name, TRANS_OPT_FORCE_IF_INCLUDES);
+
 	if (flags & TRANSPORT_PUSH_OPTIONS) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, transport->push_options)
diff --git a/transport.c b/transport.c
index 99fe6233a3..da98ebf50e 100644
--- a/transport.c
+++ b/transport.c
@@ -664,6 +664,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
diff --git a/transport.h b/transport.h
index ca409ea1e4..24558c027d 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);
 /* Request atomic (all-or-nothing) updates when pushing */
 #define TRANS_OPT_ATOMIC "atomic"
 
+/* Require remote changes to be integrated locally. */
+#define TRANS_OPT_FORCE_IF_INCLUDES "force-if-includes"
+
 /**
  * Returns 0 if the option was used, non-zero otherwise. Prints a
  * message to stderr if the option is not used.
@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v5 3/3] t, doc: update tests, reference for "--force-if-includes"

[Srinidhi Kaushik]

Update test cases for the new option, and document its usage
and update related references.

 - t/t5533-push-cas.sh:
   Update test cases for "compare-and-swap" when used along with
   "--force-if-includes" helps mitigate overwrites when remote
   ref are updated in the background.

 - Documentation:
   Add reference for the new option, configuration setting
   ("push.useForceIfIncludes") and advise messages.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt |  9 ++++--
 Documentation/config/push.txt   |  6 ++++
 Documentation/git-push.txt      | 26 +++++++++++++++-
 t/t5533-push-cas.sh             | 53 +++++++++++++++++++++++++++++++++
 4 files changed, 90 insertions(+), 4 deletions(-)

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..acbd0c09aa 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -10,9 +10,8 @@ advice.*::
 		that the check is disabled.
 	pushUpdateRejected::
 		Set this variable to 'false' if you want to disable
-		'pushNonFFCurrent',
-		'pushNonFFMatching', 'pushAlreadyExists',
-		'pushFetchFirst', and 'pushNeedsForce'
+		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
+		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
 		simultaneously.
 	pushNonFFCurrent::
 		Advice shown when linkgit:git-push[1] fails due to a
@@ -41,6 +40,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..fd981f7808 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,9 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.useForceIfIncludes::
+	If set to "true", it is equivalent to specifying
+	"--force-if-includes" as an option to linkgit:git-push[1]
+	in the command line. Adding "--no-force-if-includes" at the
+	time of push overrides this configuration setting.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..706380d263 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -13,7 +13,7 @@ SYNOPSIS
 	   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
 	   [-u | --set-upstream] [-o <string> | --push-option=<string>]
 	   [--[no-]signed|--signed=(true|false|if-asked)]
-	   [--force-with-lease[=<refname>[:<expect>]]]
+	   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]
 	   [--no-verify] [<repository> [<refspec>...]]
 
 DESCRIPTION
@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying "--force-if-includes" an an ancillary option
+along with "--force-with-lease[=<refname>]" (i.e., without saying what
+exact commit the ref on the remote side must be pointing at, or which
+refs on the remote side are being protected) at the time of "push" will
+verify if updates from the remote-tracking refs that may have been
+implicitly updated in the background are integrated locally before
+allowing a forced update.
 
 -f::
 --force::
@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option enables a check that verifies if the tip of the
+remote-tracking ref is reachable from one of the "reflog" entries of
+the local branch based in it for a rewrite. The check ensures that any
+updates from the remote have been incorporated locally by rejecting the
+forced update if that is not the case.
++
+If the option is passed without specifying "--force-with-lease", or
+specified along with "--force-with-lease=<refname>:<expect>", it is
+a "no-op".
++
+Specifying "--no-force-if-includes" disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..620d101f50 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -256,4 +256,57 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates of REMOTE can be mitigated with "--force-if-includes"' '
+	rm -rf src dst &&
+	git init --bare src.bare &&
+	test_when_finished "rm -rf src.bare" &&
+	git clone --no-local src.bare dst &&
+	test_when_finished "rm -rf dst" &&
+	(
+		cd dst &&
+		test_commit G &&
+		git push origin master:master
+	) &&
+	git clone --no-local src.bare dst2 &&
+	test_when_finished "rm -rf dst2" &&
+	(
+		cd dst2 &&
+		test_commit H &&
+		git push
+	) &&
+	(
+		cd dst &&
+		test_commit I &&
+		git fetch origin &&
+		test_must_fail git push --force-with-lease --force-if-includes origin
+	)
+'
+
+test_expect_success 'background updates of REMOTE can be mitigated with "push.useForceIfIncludes"' '
+	rm -rf src dst &&
+	git init --bare src.bare &&
+	test_when_finished "rm -rf src.bare" &&
+	git clone --no-local src.bare dst &&
+	test_when_finished "rm -rf dst" &&
+	(
+		cd dst &&
+		test_commit G &&
+		git push origin master:master
+	) &&
+	git clone --no-local src.bare dst2 &&
+	test_when_finished "rm -rf dst2" &&
+	(
+		cd dst2 &&
+		test_commit H &&
+		git push
+	) &&
+	(
+		cd dst &&
+		test_commit I &&
+		git fetch origin &&
+		git config --local push.useForceIfIncludes "true" &&
+		test_must_fail git push --force-with-lease origin
+	)
+'
+
 test_done
-- 
2.28.0

Re: [PATCH v5 1/3] push: add reflog check for "--force-if-includes"

[Phillip Wood]

Hi Srinidhi

I think this is moving forward in the right direction, I've got a couple 
of comments below. Note I've only looked at the first part of the patch 
as I'm not that familiar with the rest of the code.

On 23/09/2020 08:30, Srinidhi Kaushik wrote:
> Add a check to verify if the remote-tracking ref of the local branch
> is reachable from one of its "reflog" entries.
> 
> When a local branch that is based on a remote ref, has been rewound
> and is to be force pushed on the remote, "--force-if-includes" runs
> a check that ensures any updates to remote-tracking refs that may have

nit pick - there is only one remote tracking ref for each local ref

> happened (by push from another repository) in-between the time of the

s/push/fetch/ ?

> last checkout,

more generally it is the last time we updated the local branch to 
incorporate any fetched changes in the remote tracking branch, this 
includes `pull --rebase` `pull --merge` as well as checking out the 
remote ref

> and right before the time of push, have been integrated
> locally before allowing a forced updated.
> 
> A new field "use_force_if_includes" has been added to "push_cas_option",
> which is set to "1" when "--force-if-includes" is specified as an
> option in the command line or as a configuration setting.
> 
> The struct "ref" has two new bit-fields:
>    - check_reachable:
>      Set when we have to run the new check on the ref, and the remote
>      ref was marked as "use_tracking" or "use_tracking_for_rest" by
>      compare-and-swap (if the "the remote tip must be at the expected
>      commit" condition is not specified); "apply_push_cas()" has been
>      updated to check if this field is set and run the check.
> 
>    - unreachable:
>      Set if the ref is unreachable from any of the "reflog" entries of
>      its local counterpart.
> 
> "REF_STATUS_REJECT_REMOTE_UPDATED" has been added to the "status"
> enum to imply that the ref failed the check; "case" statements in
> "send-pack", "transport" and "transport-helper" have been updated
> accordingly to catch this status when set.

This is quite a long description of the implementation, I think it would 
be more helpful to the reader to concentrate on what the new feature is 
and why it is useful.

> 
> When "--force-is-includes" is used along with "--force-with-lease",

s/-is-/-if-/

> the check runs only for refs marked as "if_includes". If the option
> is passed without specifying "--force-with-lease", or specified along
> with "--force-with-lease=<refname>:<expect>" it is a "no-op".

If I've understood this correctly `--force-if-includes` does nothing on 
its own - I had hoped it would imply --force-with-lease

> 
> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
> ---
>   builtin/send-pack.c |   5 ++
>   remote.c            | 140 +++++++++++++++++++++++++++++++++++++++++++-
>   remote.h            |   8 ++-
>   send-pack.c         |   1 +
>   transport-helper.c  |   5 ++
>   transport.c         |   6 ++
>   6 files changed, 162 insertions(+), 3 deletions(-)
> 
> diff --git a/builtin/send-pack.c b/builtin/send-pack.c
> index 2b9610f121..4d76727edb 100644
> --- a/builtin/send-pack.c
> +++ b/builtin/send-pack.c
> @@ -69,6 +69,11 @@ static void print_helper_status(struct ref *ref)
>   			msg = "stale info";
>   			break;
>   
> +		case REF_STATUS_REJECT_REMOTE_UPDATED:
> +			res = "error";
> +			msg = "remote ref updated since checkout";
> +			break;
> +
>   		case REF_STATUS_REJECT_ALREADY_EXISTS:
>   			res = "error";
>   			msg = "already exists";
> diff --git a/remote.c b/remote.c
> index eafc14cbe7..0dcac4ab8e 100644
> --- a/remote.c
> +++ b/remote.c
> @@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
>   		 * with the remote-tracking branch to find the value
>   		 * to expect, but we did not have such a tracking
>   		 * branch.
> +		 *
> +		 * If the tip of the remote-tracking ref is unreachable
> +		 * from any reflog entry of its local ref indicating a
> +		 * possible update since checkout; reject the push.
>   		 */
>   		if (ref->expect_old_sha1) {
>   			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
>   				reject_reason = REF_STATUS_REJECT_STALE;
> +			else if (ref->check_reachable && ref->unreachable)
> +				reject_reason =
> +					REF_STATUS_REJECT_REMOTE_UPDATED;
>   			else
> -				/* If the ref isn't stale then force the update. */
> +				/*
> +				 * If the ref isn't stale, and is reachable
> +				 * from from one of the reflog entries of
> +				 * the local branch, force the update.
> +				 */
>   				force_ref_update = 1;
>   		}
>   
> @@ -2268,6 +2279,118 @@ static int remote_tracking(struct remote *remote, const char *refname,
>   	return 0;
>   }
>   
> +/*
> + * The struct "reflog_commit_list" and related helper functions
> + * for list manipulation are used for collecting commits into a
> + * list during reflog traversals in "if_exists_or_grab_until()".
> + */
> +struct reflog_commit_list {
> +	struct commit **items;
> +	size_t nr, alloc;
> +};
> +
> +/* Adds a commit to list. */
> +static void add_commit(struct reflog_commit_list *list, struct commit *commit)
> +{
> +	ALLOC_GROW(list->items, list->nr + 1, list->alloc);
> +	list->items[list->nr++] = commit;
> +}
> +
> +/* Free and reset the list. */
> +static void free_reflog_commit_list(struct reflog_commit_list *list)
> +{
> +	FREE_AND_NULL(list->items);
> +	list->nr = list->alloc = 0;
> +}
> +
> +struct check_and_collect_until_cb_data {
> +	struct commit *remote_commit;
> +	struct reflog_commit_list *local_commits;
> +};
> +
> +
> +static int check_and_collect_until(struct object_id *o_oid,
> +				   struct object_id *n_oid,
> +				   const char *ident, timestamp_t timestamp,
> +				   int tz, const char *message, void *cb_data)
> +{
> +	struct commit *commit;
> +	struct check_and_collect_until_cb_data *cb = cb_data;
> +
> +	/*
> +	 * If the reflog entry timestamp is older than the
> +	 * remote commit date, there is no need to check or
> +	 * collect entries older than this one.
> +	 */
> +	if (timestamp < cb->remote_commit->date)

It's great that you've incorporated a date check, however I think we 
need to check the local reflog timestamp against the last time the 
remote ref was updated (i.e. the remote reflog timestamp), not the 
commit date of the commit that the remote ref points too. We are 
interested in whether the local branch has incorporated the remote 
branch since the last time the remote branch was updated.

> +		return -1;
> +
> +	/* An entry was found. */
> +	if (oideq(n_oid, &cb->remote_commit->object.oid))
> +		return 1;
> +
> +	/* Lookup the commit and append it to the list. */
> +	if ((commit = lookup_commit_reference(the_repository, n_oid)))
> +		add_commit(cb->local_commits, commit);

I think Junio suggested batching the commits for the merge base check 
into small groups, rather than checking them all at once

Best Wishes

Phillip

> +
> +	return 0;
> +}
> +
> +/*
> + * Iterate through the reflog of a local ref to check if there is an entry for
> + * the given remote-tracking ref (i.e., if it was checked out); runs until the
> + * timestamp of an entry is older than the commit date of the remote-tracking
> + * ref. Any commits that seen along the way are collected into a list to check
> + * if the remote-tracking ref is reachable from any of them.
> + */
> +static int is_reachable_in_reflog(const char *local_ref_name,
> +				  const struct object_id *remote_oid)
> +{
> +	struct commit *remote_commit;
> +	struct reflog_commit_list list = { NULL, 0, 0 };
> +	struct check_and_collect_until_cb_data cb;
> +	int ret = 0;
> +
> +	remote_commit = lookup_commit_reference(the_repository, remote_oid);
> +	if (!remote_commit)
> +		goto cleanup_return;
> +
> +	cb.remote_commit = remote_commit;
> +	cb.local_commits = &list;
> +	ret = for_each_reflog_ent_reverse(local_ref_name,
> +					  check_and_collect_until, &cb);
> +
> +	/* We found an entry in the reflog. */
> +	if (ret > 0)
> +		goto cleanup_return;
> +
> +	/*
> +	 * Check if "remote_commit" is reachable from
> +	 * any of the commits in the collected list.
> +	 */
> +	if (list.nr > 0)
> +		ret = in_merge_bases_many(remote_commit, list.nr, list.items);
> +
> +cleanup_return:
> +	free_reflog_commit_list(&list);
> +	return ret;
> +}
> +
> +/*
> + * Check for reachability of a remote-tracking
> + * ref in the reflog entries of its local ref.
> + */
> +static void check_if_includes_upstream(struct ref *remote_ref)
> +{
> +	struct ref *local_ref = get_local_ref(remote_ref->name);
> +
> +	if (!local_ref)
> +		return;
> +
> +	if (!is_reachable_in_reflog(local_ref->name, &remote_ref->old_oid))
> +		remote_ref->unreachable = 1;
> +}
> +
>   static void apply_cas(struct push_cas_option *cas,
>   		      struct remote *remote,
>   		      struct ref *ref)
> @@ -2284,6 +2407,8 @@ static void apply_cas(struct push_cas_option *cas,
>   			oidcpy(&ref->old_oid_expect, &entry->expect);
>   		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
>   			oidclr(&ref->old_oid_expect);
> +		else
> +			ref->check_reachable = cas->use_force_if_includes;
>   		return;
>   	}
>   
> @@ -2294,6 +2419,8 @@ static void apply_cas(struct push_cas_option *cas,
>   	ref->expect_old_sha1 = 1;
>   	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
>   		oidclr(&ref->old_oid_expect);
> +	else
> +		ref->check_reachable = cas->use_force_if_includes;
>   }
>   
>   void apply_push_cas(struct push_cas_option *cas,
> @@ -2301,6 +2428,15 @@ void apply_push_cas(struct push_cas_option *cas,
>   		    struct ref *remote_refs)
>   {
>   	struct ref *ref;
> -	for (ref = remote_refs; ref; ref = ref->next)
> +	for (ref = remote_refs; ref; ref = ref->next) {
>   		apply_cas(cas, remote, ref);
> +
> +		/*
> +		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
> +		 * mode, and if "--force-if-includes" was specified, run
> +		 * the check.
> +		 */
> +		if (ref->check_reachable)
> +			check_if_includes_upstream(ref);
> +	}
>   }
> diff --git a/remote.h b/remote.h
> index 5e3ea5a26d..7c5e59770e 100644
> --- a/remote.h
> +++ b/remote.h
> @@ -104,7 +104,11 @@ struct ref {
>   		forced_update:1,
>   		expect_old_sha1:1,
>   		exact_oid:1,
> -		deletion:1;
> +		deletion:1,
> +		/* Need to check if local reflog reaches the remote tip. */
> +		check_reachable:1,
> +		/* The local reflog does not reach the remote tip. */
> +		unreachable:1;
>   
>   	enum {
>   		REF_NOT_MATCHED = 0, /* initial value */
> @@ -134,6 +138,7 @@ struct ref {
>   		REF_STATUS_REJECT_NEEDS_FORCE,
>   		REF_STATUS_REJECT_STALE,
>   		REF_STATUS_REJECT_SHALLOW,
> +		REF_STATUS_REJECT_REMOTE_UPDATED,
>   		REF_STATUS_UPTODATE,
>   		REF_STATUS_REMOTE_REJECT,
>   		REF_STATUS_EXPECTING_REPORT,
> @@ -332,6 +337,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);
>   
>   struct push_cas_option {
>   	unsigned use_tracking_for_rest:1;
> +	unsigned use_force_if_includes:1;
>   	struct push_cas {
>   		struct object_id expect;
>   		unsigned use_tracking:1;
> diff --git a/send-pack.c b/send-pack.c
> index 632f1580ca..956306e8e8 100644
> --- a/send-pack.c
> +++ b/send-pack.c
> @@ -240,6 +240,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
>   	case REF_STATUS_REJECT_FETCH_FIRST:
>   	case REF_STATUS_REJECT_NEEDS_FORCE:
>   	case REF_STATUS_REJECT_STALE:
> +	case REF_STATUS_REJECT_REMOTE_UPDATED:
>   	case REF_STATUS_REJECT_NODELETE:
>   		return CHECK_REF_STATUS_REJECTED;
>   	case REF_STATUS_UPTODATE:
> diff --git a/transport-helper.c b/transport-helper.c
> index c52c99d829..e547e21199 100644
> --- a/transport-helper.c
> +++ b/transport-helper.c
> @@ -779,6 +779,10 @@ static int push_update_ref_status(struct strbuf *buf,
>   			status = REF_STATUS_REJECT_STALE;
>   			FREE_AND_NULL(msg);
>   		}
> +		else if (!strcmp(msg, "remote ref updated since checkout")) {
> +			status = REF_STATUS_REJECT_REMOTE_UPDATED;
> +			FREE_AND_NULL(msg);
> +		}
>   		else if (!strcmp(msg, "forced update")) {
>   			forced = 1;
>   			FREE_AND_NULL(msg);
> @@ -897,6 +901,7 @@ static int push_refs_with_push(struct transport *transport,
>   		case REF_STATUS_REJECT_NONFASTFORWARD:
>   		case REF_STATUS_REJECT_STALE:
>   		case REF_STATUS_REJECT_ALREADY_EXISTS:
> +		case REF_STATUS_REJECT_REMOTE_UPDATED:
>   			if (atomic) {
>   				reject_atomic_push(remote_refs, mirror);
>   				string_list_clear(&cas_options, 0);
> diff --git a/transport.c b/transport.c
> index 43e24bf1e5..99fe6233a3 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -567,6 +567,11 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count,
>   		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
>   				 "stale info", porcelain, summary_width);
>   		break;
> +	case REF_STATUS_REJECT_REMOTE_UPDATED:
> +		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
> +				 "remote ref updated since checkout",
> +				 porcelain, summary_width);
> +		break;
>   	case REF_STATUS_REJECT_SHALLOW:
>   		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
>   				 "new shallow roots not allowed",
> @@ -1101,6 +1106,7 @@ static int run_pre_push_hook(struct transport *transport,
>   		if (!r->peer_ref) continue;
>   		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
>   		if (r->status == REF_STATUS_REJECT_STALE) continue;
> +		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
>   		if (r->status == REF_STATUS_UPTODATE) continue;
>   
>   		strbuf_reset(&buf);
> 

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Hi Junio,

On 09/21/2020 11:48, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> >> If we were talking about older parts of the history, optional
> >> generation numbers could change the equation somewhat, but the
> >> common case for the workflow this series is trying to help is that
> >> these local commits ane the remote tip are relatively new and it is
> >> not unlikely that the generation numbers have not been computed for
> >> them, which is why I suspect that in_merges_many may be a win.
> >
> > Nice! We can definitely try batching commits from the reflog and
> > pass it along to "in_merge_bases_many()". As for being faster than
> > calling "in_merge_bases()" for each commit entry in the reflog --
> > I am not familiar with how the former works. Do we still keep the
> > "reflog_entry_exists()" part? It might still be faster to go through
> > the entries once to check with "oideq()" in the first run.
> 
> That is what I meant.  You go through local reflog entries until you
> find one that is older than the timestamp of the reflog entry of the
> remote-tracking branch, check with oideq() to see if the tip was ever
> directly checked out.  Then, using these same local reflog entries,
> you can make in_merge_bases_many() tranversal to see if any of them
> reach the tip.  I suspect that the number of local reflog entries you
> need to examine would not be too many, so if you can put them all in
> a single array of "struct commit *" pointers in the first "oideq()"
> phase, you may be able to do just a single in_merge_bases_many() batch
> to check for the reachability.

Gotcha.

> > Also, I was wondering if it is worth considering this:
> >   - check if the reflog of the HEAD has the remote ref
> 
> It would help the workflow I had in mind, but it would raise the
> risk of false positives according to Dscho and I tend to agree, so
> I do not know if it is overall a good idea.

Oh, right. This doesn't work when a "git pull --rebase" is run on
a different branch (and a few other cases, as mentioned by Johannes).

> >   - check if the reflog of the local branch has the remote ref
> 
> Isn't that the oideq() test?

Yes.

> >   - check if the remote ref is reachable from any of the local ref's
> >     "reflog" entries using "in_merge_bases_many()" in batches as
> >     suggested here.
> 
> I think it amounts to the same as "does any reflog entry of HEAD
> reach it?" and shares the same issues with false positives as the
> first one.

Hmm, isn't this the same as what was mentioned by you earlier (without
the timestamp:

> [...] Then, using these same local reflog entries,
> you can make in_merge_bases_many() tranversal to see if any of them
> reach the tip.

In v5 (the new patch) [1], the check does this:
  - go through the local reflog until it hits an entry with a timestamp
    older than the remote commit, and doing an "oideq()" check and
    collecting commits into a list along the way.

  - if an exact entry was found, the test passes; otherwise use
    the commit list and make a call to "in_merge_bases_many()" to
    check for reachability, and report it.

> >> > +		deletion:1,
> >> > +		if_includes:1, /* If "--force-with-includes" was specified. */
> >> 
> >> The description needs to be tightened.
> >> 
> >> > +		unreachable:1; /* For "if_includes"; unreachable in reflog. */
> >
> > OK, you're right. Perhaps, we could rename it to something like
> > "if_includes_for_tracking" and update the comment description
> > with saying something along the lines of:
> 
> That is overlong.  Let me try:
> 
> 		/* need to check if local reflog reaches the remote tip */
> 		check_reachable:1,
> 
> 		/* local reflog does not reach the remote tip */
> 		unreachable:1;
> 

I have updated the description in v5 [1]; thanks!

[1]: https://public-inbox.org/git/20200923073022.61293-1-shrinidhi.kaushik@gmail.com/

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH v5 3/3] t, doc: update tests, reference for "--force-if-includes"

[Phillip Wood]

Hi Srinidhi

On 23/09/2020 08:30, Srinidhi Kaushik wrote:
> Update test cases for the new option, and document its usage
> and update related references.
> 
>   - t/t5533-push-cas.sh:
>     Update test cases for "compare-and-swap" when used along with
>     "--force-if-includes" helps mitigate overwrites when remote
>     ref are updated in the background.
> 
>   - Documentation:
>     Add reference for the new option, configuration setting
>     ("push.useForceIfIncludes") and advise messages.
> 
> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
> ---
>   Documentation/config/advice.txt |  9 ++++--
>   Documentation/config/push.txt   |  6 ++++
>   Documentation/git-push.txt      | 26 +++++++++++++++-
>   t/t5533-push-cas.sh             | 53 +++++++++++++++++++++++++++++++++
>   4 files changed, 90 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
> index bdd37c3eaa..acbd0c09aa 100644
> --- a/Documentation/config/advice.txt
> +++ b/Documentation/config/advice.txt
> @@ -10,9 +10,8 @@ advice.*::
>   		that the check is disabled.
>   	pushUpdateRejected::
>   		Set this variable to 'false' if you want to disable
> -		'pushNonFFCurrent',
> -		'pushNonFFMatching', 'pushAlreadyExists',
> -		'pushFetchFirst', and 'pushNeedsForce'
> +		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
> +		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
>   		simultaneously.
>   	pushNonFFCurrent::
>   		Advice shown when linkgit:git-push[1] fails due to a
> @@ -41,6 +40,10 @@ advice.*::
>   		we can still suggest that the user push to either
>   		refs/heads/* or refs/tags/* based on the type of the
>   		source object.
> +	pushRefNeedsUpdate::
> +		Shown when linkgit:git-push[1] rejects a forced update of
> +		a branch when its remote-tracking ref has updates that we
> +		do not have locally.
>   	statusAheadBehind::
>   		Shown when linkgit:git-status[1] computes the ahead/behind
>   		counts for a local ref compared to its remote tracking ref,
> diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
> index f5e5b38c68..fd981f7808 100644
> --- a/Documentation/config/push.txt
> +++ b/Documentation/config/push.txt
> @@ -114,3 +114,9 @@ push.recurseSubmodules::
>   	specifying '--recurse-submodules=check|on-demand|no'.
>   	If not set, 'no' is used by default, unless 'submodule.recurse' is
>   	set (in which case a 'true' value means 'on-demand').
> +
> +push.useForceIfIncludes::
> +	If set to "true", it is equivalent to specifying
> +	"--force-if-includes" as an option to linkgit:git-push[1]

I think we normally use backquotes for option names in the documentation 
so this would be `--force-if-includes`

> +	in the command line. Adding "--no-force-if-includes" at the
> +	time of push overrides this configuration setting.
> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
> index 3b8053447e..706380d263 100644
> --- a/Documentation/git-push.txt
> +++ b/Documentation/git-push.txt
> @@ -13,7 +13,7 @@ SYNOPSIS
>   	   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
>   	   [-u | --set-upstream] [-o <string> | --push-option=<string>]
>   	   [--[no-]signed|--signed=(true|false|if-asked)]
> -	   [--force-with-lease[=<refname>[:<expect>]]]
> +	   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]
>   	   [--no-verify] [<repository> [<refspec>...]]
>   
>   DESCRIPTION
> @@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally
>   force push changes to `master` if the remote version is still at
>   `base`, regardless of what your local `remotes/origin/master` has been
>   updated to in the background.
> ++
> +Alternatively, specifying "--force-if-includes" an an ancillary option
> +along with "--force-with-lease[=<refname>]" (i.e., without saying what
> +exact commit the ref on the remote side must be pointing at, or which
> +refs on the remote side are being protected) at the time of "push" will
> +verify if updates from the remote-tracking refs that may have been
> +implicitly updated in the background are integrated locally before
> +allowing a forced update.
>   
>   -f::
>   --force::
> @@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
>   origin +master` to force a push to the `master` branch). See the
>   `<refspec>...` section above for details.
>   
> +--[no-]force-if-includes::
> +	Force an update only if the tip of the remote-tracking ref
> +	has been integrated locally.
> ++
> +This option enables a check that verifies if the tip of the
> +remote-tracking ref is reachable from one of the "reflog" entries of
> +the local branch based in it for a rewrite. The check ensures that any
> +updates from the remote have been incorporated locally by rejecting the
> +forced update if that is not the case.
> ++
> +If the option is passed without specifying "--force-with-lease", or
> +specified along with "--force-with-lease=<refname>:<expect>", it is
> +a "no-op".
> ++
> +Specifying "--no-force-if-includes" disables this behavior.
> +
>   --repo=<repository>::
>   	This option is equivalent to the <repository> argument. If both
>   	are specified, the command-line argument takes precedence.
> diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
> index 0b0eb1d025..620d101f50 100755
> --- a/t/t5533-push-cas.sh
> +++ b/t/t5533-push-cas.sh
> @@ -256,4 +256,57 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
>   	)
>   '

It's great to see a couple of tests, however I think it would be useful 
to have some more - one to check that we do not force when the remote is 
rewound and a couple to show that we do force when we should. In 
particular we should still force if we `pull --rebase` and then rewrite 
the local history using `rebase -i` so that the remote ref is no longer 
an ancestor of the local HEAD.

Best Wishes

Phillip

> +test_expect_success 'background updates of REMOTE can be mitigated with "--force-if-includes"' '
> +	rm -rf src dst &&
> +	git init --bare src.bare &&
> +	test_when_finished "rm -rf src.bare" &&
> +	git clone --no-local src.bare dst &&
> +	test_when_finished "rm -rf dst" &&
> +	(
> +		cd dst &&
> +		test_commit G &&
> +		git push origin master:master
> +	) &&
> +	git clone --no-local src.bare dst2 &&
> +	test_when_finished "rm -rf dst2" &&
> +	(
> +		cd dst2 &&
> +		test_commit H &&
> +		git push
> +	) &&
> +	(
> +		cd dst &&
> +		test_commit I &&
> +		git fetch origin &&
> +		test_must_fail git push --force-with-lease --force-if-includes origin
> +	)
> +'
> +
> +test_expect_success 'background updates of REMOTE can be mitigated with "push.useForceIfIncludes"' '
> +	rm -rf src dst &&
> +	git init --bare src.bare &&
> +	test_when_finished "rm -rf src.bare" &&
> +	git clone --no-local src.bare dst &&
> +	test_when_finished "rm -rf dst" &&
> +	(
> +		cd dst &&
> +		test_commit G &&
> +		git push origin master:master
> +	) &&
> +	git clone --no-local src.bare dst2 &&
> +	test_when_finished "rm -rf dst2" &&
> +	(
> +		cd dst2 &&
> +		test_commit H &&
> +		git push
> +	) &&
> +	(
> +		cd dst &&
> +		test_commit I &&
> +		git fetch origin &&
> +		git config --local push.useForceIfIncludes "true" &&
> +		test_must_fail git push --force-with-lease origin
> +	)
> +'
> +
>   test_done
> 

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Hi Phillip,

On 09/21/2020 14:19, Phillip Wood wrote:
> As I said before[1] I think we should also be checking the reflog dates so
> that we do not look at any local reflog entries that are older than the most
> recent reflog entry for the remote tracking branch. This protects against a
> background fetch when the remote has been rewound and it would also reduce
> the number of calls to in_merge_bases().

Thanks for suggesting this; I must have missed it earlier (sorry).
The latest patch [1], has been updated to take timestamps into
account which helps reduce the number of reflog iterations.

[1]: https://public-inbox.org/git/20200923073022.61293-1-shrinidhi.kaushik@gmail.com/

Thanks.
--
Srinidhi Kaushik

Re: [PATCH v5 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Hi Phillip,

On 09/23/2020 11:18, Phillip Wood wrote:
> Hi Srinidhi
> 
> I think this is moving forward in the right direction, I've got a couple of
> comments below. Note I've only looked at the first part of the patch as I'm
> not that familiar with the rest of the code.
> 
> On 23/09/2020 08:30, Srinidhi Kaushik wrote:
> > Add a check to verify if the remote-tracking ref of the local branch
> > is reachable from one of its "reflog" entries.
> > 
> > When a local branch that is based on a remote ref, has been rewound
> > and is to be force pushed on the remote, "--force-if-includes" runs
> > a check that ensures any updates to remote-tracking refs that may have
> 
> nit pick - there is only one remote tracking ref for each local ref

You're right, I will correct that. 
> > happened (by push from another repository) in-between the time of the
> 
> s/push/fetch/ ?

Well, I was hoping it implies that there was a push made by another
person to the ref while a rewrite is happening locally. 
 
> > last checkout,
>
> more generally it is the last time we updated the local branch to
> incorporate any fetched changes in the remote tracking branch, this includes
> `pull --rebase` `pull --merge` as well as checking out the remote ref
> 
> > and right before the time of push, have been integrated
> > locally before allowing a forced updated.
> > 
> > A new field "use_force_if_includes" has been added to "push_cas_option",
> > which is set to "1" when "--force-if-includes" is specified as an
> > option in the command line or as a configuration setting.
> > 
> > The struct "ref" has two new bit-fields:
> >    - check_reachable:
> >      Set when we have to run the new check on the ref, and the remote
> >      ref was marked as "use_tracking" or "use_tracking_for_rest" by
> >      compare-and-swap (if the "the remote tip must be at the expected
> >      commit" condition is not specified); "apply_push_cas()" has been
> >      updated to check if this field is set and run the check.
> > 
> >    - unreachable:
> >      Set if the ref is unreachable from any of the "reflog" entries of
> >      its local counterpart.
> > 
> > "REF_STATUS_REJECT_REMOTE_UPDATED" has been added to the "status"
> > enum to imply that the ref failed the check; "case" statements in
> > "send-pack", "transport" and "transport-helper" have been updated
> > accordingly to catch this status when set.
> 
> This is quite a long description of the implementation, I think it would be
> more helpful to the reader to concentrate on what the new feature is and why
> it is useful.

OK, I will try to minimize the description a bit further.
 
> > 
> > When "--force-is-includes" is used along with "--force-with-lease",
> 
> s/-is-/-if-/

Oh, this happened again. Sorry!

> > the check runs only for refs marked as "if_includes". If the option
> > is passed without specifying "--force-with-lease", or specified along
> > with "--force-with-lease=<refname>:<expect>" it is a "no-op".

There's another typo: s/if_includes/check_reachable/.
 
> If I've understood this correctly `--force-if-includes` does nothing on its
> own - I had hoped it would imply --force-with-lease

Yes, it does nothing if not used with  "--force-with-lease", or when
used with it and the exact commit to be expected is specified for
"--force-with-lease" (i.e., with --force-with-lease=<ref>:<expect>).

> > [...]
> It's great that you've incorporated a date check, however I think we need to
> check the local reflog timestamp against the last time the remote ref was
> updated (i.e. the remote reflog timestamp), not the commit date of the
> commit that the remote ref points too. We are interested in whether the
> local branch has incorporated the remote branch since the last time the
> remote branch was updated.

Got it. I misread the requirement here; will update.

> > +		return -1;
> > +
> > +	/* An entry was found. */
> > +	if (oideq(n_oid, &cb->remote_commit->object.oid))
> > +		return 1;
> > +
> > +	/* Lookup the commit and append it to the list. */
> > +	if ((commit = lookup_commit_reference(the_repository, n_oid)))
> > +		add_commit(cb->local_commits, commit);
> 
> I think Junio suggested batching the commits for the merge base check into
> small groups, rather than checking them all at once

Doesn't stopping the iteration early with the date check collect fewer
commits? Would batching still be necessary?

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH v5 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> Hi Srinidhi
>
> I think this is moving forward in the right direction, I've got a
> couple of comments below. Note I've only looked at the first part of
> the patch as I'm not that familiar with the rest of the code.
>
> On 23/09/2020 08:30, Srinidhi Kaushik wrote:
>> Add a check to verify if the remote-tracking ref of the local branch
>> is reachable from one of its "reflog" entries.
>> When a local branch that is based on a remote ref, has been rewound
>> and is to be force pushed on the remote, "--force-if-includes" runs
>> a check that ensures any updates to remote-tracking refs that may have
>
> nit pick - there is only one remote tracking ref for each local ref

Yup, "updates to the remote-tracking ref".

> This is quite a long description of the implementation, I think it
> would be more helpful to the reader to concentrate on what the new
> feature is and why it is useful.

Thanks for pointing it out.

> It's great that you've incorporated a date check, however I think we
> need to check the local reflog timestamp against the last time the 
> remote ref was updated (i.e. the remote reflog timestamp), not the
> commit date of the commit that the remote ref points too. We are 
> interested in whether the local branch has incorporated the remote
> branch since the last time the remote branch was updated.

Yes, exactly.

>
>> +		return -1;
>> +
>> +	/* An entry was found. */
>> +	if (oideq(n_oid, &cb->remote_commit->object.oid))
>> +		return 1;
>> +
>> +	/* Lookup the commit and append it to the list. */
>> +	if ((commit = lookup_commit_reference(the_repository, n_oid)))
>> +		add_commit(cb->local_commits, commit);
>
> I think Junio suggested batching the commits for the merge base check
> into small groups, rather than checking them all at once

That was before you suggested to cut the reflog traversal using the
reflog timestamp.  The code in earlier rounds of this series wanted
to read the reflog entries all the way down to the beginning of time,
and it made sense to batch in order to limit the traversal.

Now we found out that using only the local reflog entries more
recent than the latest update to the remote-tracking branch, and
because these local reflog entries will already be read in the above
code anyway, I do not see a point in batching.

Even if we were to batch calls to git_merge_bases_many(), it won't
happen in the quoted part of the code, where it first tries to see
any of the recent local reflog entries directly point at the object
with oideq().  While it is doing so, the only thing it does for the
later merge-base phase is to collect them in local_commits[] array.
Chunking can, if desired, be done by carving the array into pieces
and calling git_merge_bases_many() in is_reachable_in_reflog()
function.

Thanks for a careful review.

Re: [PATCH v5 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> diff --git a/remote.h b/remote.h
> index 5e3ea5a26d..7c5e59770e 100644
> --- a/remote.h
> +++ b/remote.h
> @@ -104,7 +104,11 @@ struct ref {
>  		forced_update:1,
>  		expect_old_sha1:1,
>  		exact_oid:1,
> -		deletion:1;
> +		deletion:1,
> +		/* Need to check if local reflog reaches the remote tip. */
> +		check_reachable:1,
> +		/* The local reflog does not reach the remote tip. */
> +		unreachable:1;

I know the comment text above is what I suggested, but it may be
more useful to readers to explain the latter as

		/* ... the result of the above check */
		unreachable:1;

Thanks.

Re: [PATCH v4 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

>> >   - check if the remote ref is reachable from any of the local ref's
>> >     "reflog" entries using "in_merge_bases_many()" in batches as
>> >     suggested here.
>> 
>> I think it amounts to the same as "does any reflog entry of HEAD
>> reach it?" and shares the same issues with false positives as the
>> first one.
>
> Hmm, isn't this the same as what was mentioned by you earlier (without
> the timestamp:

I misunderstood what you meant with the "any of the local ref" part
of the sentence and mistook it as "enumerate all the local refs and
HEAD, and collect all reflog entries of these local refs, and see if
any of them reach the remote ref".  Re-reading the sentence with
that "any of" refers to "entries" (of a single local ref) in mind,
it is pretty much the same as we agreed is a good way to go.

Thanks.

[PATCH v6 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: "--force-if-includes" to "git-push" where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref -- on which a local branch has based on -- is reachable from at
least one of the "reflog" entries of the branch about to be updated
by force on the remote.

This option can be used with "--force-with-lease" with setups where
the remote-tracking refs of the repository are implicitly updated in
the background to help prevent unintended remote overwrites.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with "--force-with-lease[=<refname>[:<expect>]]" without specifying
the "<expect>" value, can cause the update that happened in-between
the checkout and forced push to be lost.

Changes since v5:
  - In the previous patch, the stopping condition for "reflog"
    iteration used the commit date of the remote ref. That has
    been changed to use the the timestamp of the latest entry
    of the remote ref's "reflog"; calls "in_merge_bases_many()"
    are batched when the number of commits collected in the list
    is more than 8.

  - Add more tests for t5533 to include scenarios where forced
    updates should be allowed for "pull --rebase" and local
    rebase operations.

  - Reword commit messages, fix typos, and update documentation
    to use "`" where necessary.

Srinidhi Kaushik (3):
  push: add reflog check for "--force-if-includes"
  push: parse and set flag for "--force-if-includes"
  t, doc: update tests, reference for "--force-if-includes"

 Documentation/config/advice.txt |   9 +-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 ++++-
 advice.c                        |   3 +
 advice.h                        |   2 +
 builtin/push.c                  |  27 +++++
 builtin/send-pack.c             |  12 +++
 remote-curl.c                   |  14 ++-
 remote.c                        | 185 ++++++++++++++++++++++++++++++--
 remote.h                        |  12 ++-
 send-pack.c                     |   1 +
 t/t5533-push-cas.sh             | 129 ++++++++++++++++++++++
 transport-helper.c              |  10 ++
 transport.c                     |   8 ++
 transport.h                     |  15 ++-
 15 files changed, 442 insertions(+), 17 deletions(-)

base-commit: 9bc233ae1cf19a49e51842c7959d80a675dbd1c0
--
2.28.0

[PATCH v6 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Add a check to verify if the remote-tracking ref of the local branch
is reachable from one of its "reflog" entries.

The check iterates through the local ref's reflog to see if there
is an entry for the remote-tracking ref and collecting any commits
that are seen, into a list; the iteration stops if an entry in the
reflog matches the remote ref or if the entry timestamp is older
the latest entry of the remote ref's "reflog". If there wasn't an
entry found for the remote ref, "in_merge_bases_many()" is called
to check if it is reachable from the list of collected commits.

When a local branch that is based on a remote ref, has been rewound
and is to be force pushed on the remote, "--force-if-includes" runs
a check that ensures any updates to the remote-tracking ref that may
have happened (by push from another repository) in-between the time
of the last update to the local branch (via "git-pull", for instance)
and right before the time of push, have been integrated locally
before allowing a forced update.

If the new option is passed without specifying "--force-with-lease",
or specified along with "--force-with-lease=<refname>:<expect>" it
is a "no-op".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 builtin/send-pack.c |   5 ++
 remote.c            | 185 ++++++++++++++++++++++++++++++++++++++++++--
 remote.h            |  12 ++-
 send-pack.c         |   1 +
 transport-helper.c  |   5 ++
 transport.c         |   6 ++
 6 files changed, 207 insertions(+), 7 deletions(-)

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 7af148d733..516cba7336 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote ref updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/remote.c b/remote.c
index eafc14cbe7..a696243247 100644
--- a/remote.c
+++ b/remote.c
@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 		 * with the remote-tracking branch to find the value
 		 * to expect, but we did not have such a tracking
 		 * branch.
+		 *
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
 		 */
 		if (ref->expect_old_sha1) {
 			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
 				reject_reason = REF_STATUS_REJECT_STALE;
+			else if (ref->check_reachable && ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
 			else
-				/* If the ref isn't stale then force the update. */
+				/*
+				 * If the ref isn't stale, and is reachable
+				 * from from one of the reflog entries of
+				 * the local branch, force the update.
+				 */
 				force_ref_update = 1;
 		}
 
@@ -2252,11 +2263,11 @@ int is_empty_cas(const struct push_cas_option *cas)
 /*
  * Look at remote.fetch refspec and see if we have a remote
  * tracking branch for the refname there.  Fill its current
- * value in sha1[].
+ * value in sha1[], and as a string.
  * If we cannot do so, return negative to signal an error.
  */
 static int remote_tracking(struct remote *remote, const char *refname,
-			   struct object_id *oid)
+			   struct object_id *oid, char **dst_refname)
 {
 	char *dst;
 
@@ -2265,9 +2276,154 @@ static int remote_tracking(struct remote *remote, const char *refname,
 		return -1; /* no tracking ref for refname at remote */
 	if (read_ref(dst, oid))
 		return -1; /* we know what the tracking ref is but we cannot read it */
+
+	*dst_refname = dst;
+	return 0;
+}
+
+/*
+ * The struct "reflog_commit_list" and related helper functions
+ * for list manipulation are used for collecting commits into a
+ * list during reflog traversals in "if_exists_or_grab_until()".
+ */
+struct reflog_commit_list {
+	struct commit **items;
+	size_t nr, alloc;
+};
+
+/* Add a commit to list. */
+static void add_commit(struct reflog_commit_list *list, struct commit *commit)
+{
+	ALLOC_GROW(list->items, list->nr + 1, list->alloc);
+	list->items[list->nr++] = commit;
+}
+
+/* Free and reset the list. */
+static void free_reflog_commit_list(struct reflog_commit_list *list)
+{
+	FREE_AND_NULL(list->items);
+	list->nr = list->alloc = 0;
+}
+
+struct check_and_collect_until_cb_data {
+	struct commit *remote_commit;
+	struct reflog_commit_list *local_commits;
+	timestamp_t remote_reflog_timestamp;
+};
+
+/* Get the timestamp of the latest entry. */
+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,
+		       const char *ident, timestamp_t timestamp,
+		       int tz, const char *message, void *cb_data)
+{
+	timestamp_t *ts = cb_data;
+	*ts = timestamp;
+	return 1;
+}
+
+static int check_and_collect_until(struct object_id *o_oid,
+				   struct object_id *n_oid,
+				   const char *ident, timestamp_t timestamp,
+				   int tz, const char *message, void *cb_data)
+{
+	struct commit *commit;
+	struct check_and_collect_until_cb_data *cb = cb_data;
+
+	/*
+	 * If the reflog entry timestamp is older than the remote ref's
+	 * latest reflog entry, there is no need to check or collect
+	 * entries older than this one.
+	 */
+	if (timestamp < cb->remote_reflog_timestamp)
+		return -1;
+
+	/* An entry was found. */
+	if (oideq(n_oid, &cb->remote_commit->object.oid))
+		return 1;
+
+	/* Look-up the commit and append it to the list. */
+	if ((commit = lookup_commit_reference(the_repository, n_oid)))
+		add_commit(cb->local_commits, commit);
+
 	return 0;
 }
 
+#define MERGE_BASES_BATCH_SIZE 8
+
+/*
+ * Iterate through the reflog of the local ref to check if there is an entry
+ * for the given remote-tracking ref; runs until the timestamp of an entry is
+ * older than latest timestamp of remote-tracking ref's reflog. Any commits
+ * are that seen along the way are collected into a list to check if the
+ * remote-tracking ref is reachable from any of them.
+ */
+static int is_reachable_in_reflog(const char *local, const struct ref *remote)
+{
+	timestamp_t date;
+	struct commit *commit;
+	struct commit **chunk;
+	struct check_and_collect_until_cb_data cb;
+	struct reflog_commit_list list = { NULL, 0, 0 };
+	size_t count = 0, batch_size = 0;
+	int ret = 0;
+
+	commit = lookup_commit_reference(the_repository, &remote->old_oid);
+	if (!commit)
+		goto cleanup_return;
+
+	/*
+	 * Get the timestamp from the latest entry
+	 * of the remote-tracking ref's reflog.
+	 */
+	for_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);
+
+	cb.remote_commit = commit;
+	cb.local_commits = &list;
+	cb.remote_reflog_timestamp = date;
+	ret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);
+
+	/* We found an entry in the reflog. */
+	if (ret > 0)
+		goto cleanup_return;
+
+	/*
+	 * Check if the remote commit is reachable from any
+	 * of the commits in the collected list, in batches.
+	 */
+	chunk = list.items;
+	while (count < list.nr) {
+		batch_size = MERGE_BASES_BATCH_SIZE;
+
+		/* For any leftover entries. */
+		if ((count + MERGE_BASES_BATCH_SIZE) > list.nr)
+			batch_size = list.nr - count;
+
+		if ((ret = in_merge_bases_many(commit, batch_size, chunk)))
+			break;
+
+		chunk += batch_size;
+		count += MERGE_BASES_BATCH_SIZE;
+	}
+
+cleanup_return:
+	free_reflog_commit_list(&list);
+	return ret;
+}
+
+/*
+ * Check for reachability of a remote-tracking
+ * ref in the reflog entries of its local ref.
+ */
+static void check_if_includes_upstream(struct ref *remote)
+{
+	struct ref *local = get_local_ref(remote->name);
+	if (!local)
+		return;
+
+	if (is_reachable_in_reflog(local->name, remote) <= 0)
+		remote->unreachable = 1;
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
@@ -2282,8 +2438,12 @@ static void apply_cas(struct push_cas_option *cas,
 		ref->expect_old_sha1 = 1;
 		if (!entry->use_tracking)
 			oidcpy(&ref->old_oid_expect, &entry->expect);
-		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+		else if (remote_tracking(remote, ref->name,
+					 &ref->old_oid_expect,
+					 &ref->tracking_ref))
 			oidclr(&ref->old_oid_expect);
+		else
+			ref->check_reachable = cas->use_force_if_includes;
 		return;
 	}
 
@@ -2292,8 +2452,12 @@ static void apply_cas(struct push_cas_option *cas,
 		return;
 
 	ref->expect_old_sha1 = 1;
-	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+	if (remote_tracking(remote, ref->name,
+			    &ref->old_oid_expect,
+			    &ref->tracking_ref))
 		oidclr(&ref->old_oid_expect);
+	else
+		ref->check_reachable = cas->use_force_if_includes;
 }
 
 void apply_push_cas(struct push_cas_option *cas,
@@ -2301,6 +2465,15 @@ void apply_push_cas(struct push_cas_option *cas,
 		    struct ref *remote_refs)
 {
 	struct ref *ref;
-	for (ref = remote_refs; ref; ref = ref->next)
+	for (ref = remote_refs; ref; ref = ref->next) {
 		apply_cas(cas, remote, ref);
+
+		/*
+		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
+		 * mode, and if "--force-if-includes" was specified, run
+		 * the check.
+		 */
+		if (ref->check_reachable)
+			check_if_includes_upstream(ref);
+	}
 }
diff --git a/remote.h b/remote.h
index eb62a47044..2d5391d281 100644
--- a/remote.h
+++ b/remote.h
@@ -107,12 +107,20 @@ struct ref {
 	struct object_id new_oid;
 	struct object_id old_oid_expect; /* used by expect-old */
 	char *symref;
+	char *tracking_ref;
 	unsigned int
 		force:1,
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		/* Need to check if local reflog reaches the remote tip. */
+		check_reachable:1,
+		/*
+		 * Store the result of the check enabled by "check_reachable";
+		 * implies the local reflog does not reach the remote tip.
+		 */
+		unreachable:1;
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -142,6 +150,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);
 
 struct push_cas_option {
 	unsigned use_tracking_for_rest:1;
+	unsigned use_force_if_includes:1;
 	struct push_cas {
 		struct object_id expect;
 		unsigned use_tracking:1;
diff --git a/send-pack.c b/send-pack.c
index c9698070fc..eb4a44270b 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/transport-helper.c b/transport-helper.c
index b573b6c730..6157de30c7 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote ref updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
diff --git a/transport.c b/transport.c
index ffe2115845..75f1786aca 100644
--- a/transport.c
+++ b/transport.c
@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,
 				 "stale info",
 				 report, porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote ref updated since checkout",
+				 porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
-- 
2.28.0

[PATCH v6 2/3] push: parse and set flag for "--force-if-includes"

[Srinidhi Kaushik]

The previous commit added the necessary machinery to implement the
"--force-if-includes" protection, when "--force-with-lease" is used
without giving exact object the remote still ought to have. Surface
the feature by adding a command line option and a configuration
variable to enable it.

 - Add a flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES" to indicate that the
   new option was passed from the command line of via configuration
   settings; update command line and configuration parsers to set the
   new flag accordingly.

 - Introduce a new configuration option "push.useForceIfIncludes", which
   is equivalent to setting "--force-if-includes" in the command line.

 - Update "remote-curl" to recognize and pass this option to "send-pack"
   when enabled.

 - Update "advise" to catch the reject reason "REJECT_REF_NEEDS_UPDATE",
   set when the ref status is "REF_STATUS_REJECT_REMOTE_UPDATED" and
   (optionally) print a help message when the push fails.

 - The new option is a "no-op" in the following scenarios:
    * When used without "--force-with-lease".
    * When used with "--force-with-lease", and if the expected commit
      on the remote side is specified as an argument.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 advice.c            |  3 +++
 advice.h            |  2 ++
 builtin/push.c      | 27 +++++++++++++++++++++++++++
 builtin/send-pack.c |  7 +++++++
 remote-curl.c       | 14 +++++++++++++-
 transport-helper.c  |  5 +++++
 transport.c         |  2 ++
 transport.h         | 15 ++++++++++-----
 8 files changed, 69 insertions(+), 6 deletions(-)

diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index 0eeb2c8dd5..908b557edb 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git pull ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
 
 	return 1;
@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		if (!v)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
+	} else if (!strcmp(k, "push.useforceifincludes")) {
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		return 0;
 	}
 
 	return git_default_config(k, v, NULL);
@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 	if ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))
 		die(_("--all and --mirror are incompatible"));
 
+	if (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))
+		cas.use_force_if_includes = 1;
+
 	for_each_string_list_item(item, push_options)
 		if (strchr(item->string, '\n'))
 			die(_("push options must not have new line characters"));
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 516cba7336..a284ada051 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int progress = -1;
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
+	int force_if_includes = 0;
 	struct packet_reader reader;
 
 	struct option options[] = {
@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 		  N_("require old value of ref to be at this value"),
 		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
+		OPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,
+			 N_("require remote updates to be integrated locally")),
 		OPT_END()
 	};
 
@@ -304,6 +307,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	if (!is_empty_cas(&cas))
 		apply_push_cas(&cas, remote, remote_refs);
 
+	if (!is_empty_cas(&cas) && force_if_includes)
+		cas.use_force_if_includes = 1;
+
+
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
 
diff --git a/remote-curl.c b/remote-curl.c
index 32cc4a0c55..0290b04891 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -44,7 +44,8 @@ struct options {
 		from_promisor : 1,
 
 		atomic : 1,
-		object_format : 1;
+		object_format : 1,
+		force_if_includes : 1;
 	const struct git_hash_algo *hash_algo;
 };
 static struct options options;
@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)
 		string_list_append(&cas_options, val.buf);
 		strbuf_release(&val);
 		return 0;
+	} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {
+		if (!strcmp(value, "true"))
+			options.force_if_includes = 1;
+		else if (!strcmp(value, "false"))
+			options.force_if_includes = 0;
+		else
+			return -1;
+		return 0;
 	} else if (!strcmp(name, "cloning")) {
 		if (!strcmp(value, "true"))
 			options.cloning = 1;
@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)
 		strvec_push(&args, cas_option->string);
 	strvec_push(&args, url.buf);
 
+	if (options.force_if_includes)
+		strvec_push(&args, "--force-if-includes");
+
 	strvec_push(&args, "--stdin");
 	for (i = 0; i < nr_spec; i++)
 		packet_buf_write(&preamble, "%s\n", specs[i]);
diff --git a/transport-helper.c b/transport-helper.c
index 6157de30c7..5f6e0b3bd8 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,
 		if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
 			die(_("helper %s does not support --atomic"), name);
 
+	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
+		if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
+			die(_("helper %s does not support --%s"),
+			    name, TRANS_OPT_FORCE_IF_INCLUDES);
+
 	if (flags & TRANSPORT_PUSH_OPTIONS) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, transport->push_options)
diff --git a/transport.c b/transport.c
index 75f1786aca..5cfb7360e9 100644
--- a/transport.c
+++ b/transport.c
@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
diff --git a/transport.h b/transport.h
index ca409ea1e4..24558c027d 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);
 /* Request atomic (all-or-nothing) updates when pushing */
 #define TRANS_OPT_ATOMIC "atomic"
 
+/* Require remote changes to be integrated locally. */
+#define TRANS_OPT_FORCE_IF_INCLUDES "force-if-includes"
+
 /**
  * Returns 0 if the option was used, non-zero otherwise. Prints a
  * message to stderr if the option is not used.
@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v6 3/3] t, doc: update tests, reference for "--force-if-includes"

[Srinidhi Kaushik]

Update test cases for the new option, and document its usage
and update related references.

Update test cases for the new option, and document its usage
and update related references.

 - t/t5533-push-cas.sh:
   Update test cases for "compare-and-swap" when used along with
   "--force-if-includes" helps mitigate overwrites when remote
   refs are updated in the background; allows forced updates when
   changes from remote are integrated locally.

 - Documentation:
   Add reference for the new option, configuration setting
   ("push.useForceIfIncludes") and advise messages.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt |   9 ++-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 ++++++-
 t/t5533-push-cas.sh             | 129 ++++++++++++++++++++++++++++++++
 4 files changed, 166 insertions(+), 4 deletions(-)

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..acbd0c09aa 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -10,9 +10,8 @@ advice.*::
 		that the check is disabled.
 	pushUpdateRejected::
 		Set this variable to 'false' if you want to disable
-		'pushNonFFCurrent',
-		'pushNonFFMatching', 'pushAlreadyExists',
-		'pushFetchFirst', and 'pushNeedsForce'
+		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
+		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
 		simultaneously.
 	pushNonFFCurrent::
 		Advice shown when linkgit:git-push[1] fails due to a
@@ -41,6 +40,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..21b256e0a4 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,9 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.useForceIfIncludes::
+	If set to "true", it is equivalent to specifying
+	`--force-if-includes` as an option to linkgit:git-push[1]
+	in the command line. Adding `--no-force-if-includes` at the
+	time of push overrides this configuration setting.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..fb3a220386 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -13,7 +13,7 @@ SYNOPSIS
 	   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
 	   [-u | --set-upstream] [-o <string> | --push-option=<string>]
 	   [--[no-]signed|--signed=(true|false|if-asked)]
-	   [--force-with-lease[=<refname>[:<expect>]]]
+	   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]
 	   [--no-verify] [<repository> [<refspec>...]]
 
 DESCRIPTION
@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying `--force-if-includes` an an ancillary option
+along with `--force-with-lease[=<refname>]` (i.e., without saying what
+exact commit the ref on the remote side must be pointing at, or which
+refs on the remote side are being protected) at the time of "push" will
+verify if updates from the remote-tracking refs that may have been
+implicitly updated in the background are integrated locally before
+allowing a forced update.
 
 -f::
 --force::
@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option enables a check that verifies if the tip of the
+remote-tracking ref is reachable from one of the "reflog" entries of
+the local branch based in it for a rewrite. The check ensures that any
+updates from the remote have been incorporated locally by rejecting the
+forced update if that is not the case.
++
+If the option is passed without specifying `--force-with-lease`, or
+specified along with `--force-with-lease=<refname>:<expect>`, it is
+a "no-op".
++
+Specifying `--no-force-if-includes` disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..ed44ddc654 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -13,6 +13,46 @@ setup_srcdst_basic () {
 	)
 }
 
+# For tests with "--force-if-includes".
+setup_src_dup_dst () {
+	rm -fr src dup dst &&
+	git init --bare dst &&
+	git clone --no-local dst src &&
+	git clone --no-local dst dup
+	(
+		cd src &&
+		test_commit A &&
+		test_commit B &&
+		test_commit C &&
+		git push
+	) &&
+	(
+		cd dup &&
+		git fetch &&
+		git merge origin/master &&
+		git switch -c branch master~2 &&
+		test_commit D &&
+		test_commit E &&
+		git push --all
+	) &&
+	(
+		cd src &&
+		git switch master &&
+		git fetch --all &&
+		git branch branch --track origin/branch &&
+		git rebase origin/master
+	) &&
+	(
+		cd dup &&
+		git switch master &&
+		test_commit F &&
+		test_commit G &&
+		git switch branch &&
+		test_commit H &&
+		git push origin --all
+	)
+}
+
 test_expect_success setup '
 	# create template repository
 	test_commit A &&
@@ -256,4 +296,93 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates to remote can be mitigated with "--force-if-includes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		test_must_fail git push --force-with-lease --force-if-includes --all
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success 'background updates to remote can be mitigated with "push.useForceIfIncludes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		git config --local push.useForceIfIncludes true &&
+		test_must_fail git push --force-with-lease=master origin master
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success '"--force-if-includes" should be disabled for --force-with-lease="<refname>:<expect>"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		remote_head="$(git rev-parse refs/remotes/origin/master)" &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes --force-with-lease="master:$remote_head" 2>err &&
+		grep "stale info" err
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase")' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase", local rebase)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git rebase --onto HEAD~4 HEAD~1 &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
 test_done
-- 
2.28.0

Re: [PATCH v6 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Hello,
Please disregard this patch series. It does not compile with the
latest commit on "master" (9bc233ae1cf19a49e51842c7959d80a675dbd1c0).
I will fix those errors and retest in v7.

Apologies for the noise.

Thanks.
-- 
Srinidhi Kaushik

[PATCH v7 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: "--force-if-includes" to "git-push" where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref -- on which a local branch has based on -- is reachable from at
least one of the "reflog" entries of the branch about to be updated
by force on the remote.

This option can be used with "--force-with-lease" with setups where
the remote-tracking refs of the repository are implicitly updated in
the background to help prevent unintended remote overwrites.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with "--force-with-lease[=<refname>[:<expect>]]" without specifying
the "<expect>" value, can cause the update that happened in-between
the checkout and forced push to be lost.

Changes since v5:
  - In the previous patch, the stopping condition for "reflog"
    iteration used the commit date of the remote ref. That has
    been changed to use the the timestamp of the latest entry
    of the remote ref's "reflog"; calls "in_merge_bases_many()"
    are batched when the number of commits collected in the list
    is more than 8.

  - Add more tests for t5533 to include scenarios where forced
    updates should be allowed for "pull --rebase" and local
    rebase operations.

  - Reword commit messages, fix typos, and update documentation
    to use "`" where necessary.

Changes since v6:
  - Compilation failed because there was an argument missing when
    calling "print_ref_status()"; it has been updated.

Srinidhi Kaushik (3):
  push: add reflog check for "--force-if-includes"
  push: parse and set flag for "--force-if-includes"
  t, doc: update tests, reference for "--force-if-includes"

 Documentation/config/advice.txt |   9 +-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 ++++-
 advice.c                        |   3 +
 advice.h                        |   2 +
 builtin/push.c                  |  27 +++++
 builtin/send-pack.c             |  12 +++
 remote-curl.c                   |  14 ++-
 remote.c                        | 185 ++++++++++++++++++++++++++++++--
 remote.h                        |  12 ++-
 send-pack.c                     |   1 +
 t/t5533-push-cas.sh             | 129 ++++++++++++++++++++++
 transport-helper.c              |  10 ++
 transport.c                     |   8 ++
 transport.h                     |  15 ++-
 15 files changed, 442 insertions(+), 17 deletions(-)

base-commit: 9bc233ae1cf19a49e51842c7959d80a675dbd1c0
--
2.28.0

[PATCH v7 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Add a check to verify if the remote-tracking ref of the local branch
is reachable from one of its "reflog" entries.

The check iterates through the local ref's reflog to see if there
is an entry for the remote-tracking ref and collecting any commits
that are seen, into a list; the iteration stops if an entry in the
reflog matches the remote ref or if the entry timestamp is older
the latest entry of the remote ref's "reflog". If there wasn't an
entry found for the remote ref, "in_merge_bases_many()" is called
to check if it is reachable from the list of collected commits.

When a local branch that is based on a remote ref, has been rewound
and is to be force pushed on the remote, "--force-if-includes" runs
a check that ensures any updates to the remote-tracking ref that may
have happened (by push from another repository) in-between the time
of the last update to the local branch (via "git-pull", for instance)
and right before the time of push, have been integrated locally
before allowing a forced update.

If the new option is passed without specifying "--force-with-lease",
or specified along with "--force-with-lease=<refname>:<expect>" it
is a "no-op".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 builtin/send-pack.c |   5 ++
 remote.c            | 185 ++++++++++++++++++++++++++++++++++++++++++--
 remote.h            |  12 ++-
 send-pack.c         |   1 +
 transport-helper.c  |   5 ++
 transport.c         |   6 ++
 6 files changed, 207 insertions(+), 7 deletions(-)

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 7af148d733..516cba7336 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote ref updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/remote.c b/remote.c
index eafc14cbe7..a696243247 100644
--- a/remote.c
+++ b/remote.c
@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 		 * with the remote-tracking branch to find the value
 		 * to expect, but we did not have such a tracking
 		 * branch.
+		 *
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
 		 */
 		if (ref->expect_old_sha1) {
 			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
 				reject_reason = REF_STATUS_REJECT_STALE;
+			else if (ref->check_reachable && ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
 			else
-				/* If the ref isn't stale then force the update. */
+				/*
+				 * If the ref isn't stale, and is reachable
+				 * from from one of the reflog entries of
+				 * the local branch, force the update.
+				 */
 				force_ref_update = 1;
 		}
 
@@ -2252,11 +2263,11 @@ int is_empty_cas(const struct push_cas_option *cas)
 /*
  * Look at remote.fetch refspec and see if we have a remote
  * tracking branch for the refname there.  Fill its current
- * value in sha1[].
+ * value in sha1[], and as a string.
  * If we cannot do so, return negative to signal an error.
  */
 static int remote_tracking(struct remote *remote, const char *refname,
-			   struct object_id *oid)
+			   struct object_id *oid, char **dst_refname)
 {
 	char *dst;
 
@@ -2265,9 +2276,154 @@ static int remote_tracking(struct remote *remote, const char *refname,
 		return -1; /* no tracking ref for refname at remote */
 	if (read_ref(dst, oid))
 		return -1; /* we know what the tracking ref is but we cannot read it */
+
+	*dst_refname = dst;
+	return 0;
+}
+
+/*
+ * The struct "reflog_commit_list" and related helper functions
+ * for list manipulation are used for collecting commits into a
+ * list during reflog traversals in "if_exists_or_grab_until()".
+ */
+struct reflog_commit_list {
+	struct commit **items;
+	size_t nr, alloc;
+};
+
+/* Add a commit to list. */
+static void add_commit(struct reflog_commit_list *list, struct commit *commit)
+{
+	ALLOC_GROW(list->items, list->nr + 1, list->alloc);
+	list->items[list->nr++] = commit;
+}
+
+/* Free and reset the list. */
+static void free_reflog_commit_list(struct reflog_commit_list *list)
+{
+	FREE_AND_NULL(list->items);
+	list->nr = list->alloc = 0;
+}
+
+struct check_and_collect_until_cb_data {
+	struct commit *remote_commit;
+	struct reflog_commit_list *local_commits;
+	timestamp_t remote_reflog_timestamp;
+};
+
+/* Get the timestamp of the latest entry. */
+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,
+		       const char *ident, timestamp_t timestamp,
+		       int tz, const char *message, void *cb_data)
+{
+	timestamp_t *ts = cb_data;
+	*ts = timestamp;
+	return 1;
+}
+
+static int check_and_collect_until(struct object_id *o_oid,
+				   struct object_id *n_oid,
+				   const char *ident, timestamp_t timestamp,
+				   int tz, const char *message, void *cb_data)
+{
+	struct commit *commit;
+	struct check_and_collect_until_cb_data *cb = cb_data;
+
+	/*
+	 * If the reflog entry timestamp is older than the remote ref's
+	 * latest reflog entry, there is no need to check or collect
+	 * entries older than this one.
+	 */
+	if (timestamp < cb->remote_reflog_timestamp)
+		return -1;
+
+	/* An entry was found. */
+	if (oideq(n_oid, &cb->remote_commit->object.oid))
+		return 1;
+
+	/* Look-up the commit and append it to the list. */
+	if ((commit = lookup_commit_reference(the_repository, n_oid)))
+		add_commit(cb->local_commits, commit);
+
 	return 0;
 }
 
+#define MERGE_BASES_BATCH_SIZE 8
+
+/*
+ * Iterate through the reflog of the local ref to check if there is an entry
+ * for the given remote-tracking ref; runs until the timestamp of an entry is
+ * older than latest timestamp of remote-tracking ref's reflog. Any commits
+ * are that seen along the way are collected into a list to check if the
+ * remote-tracking ref is reachable from any of them.
+ */
+static int is_reachable_in_reflog(const char *local, const struct ref *remote)
+{
+	timestamp_t date;
+	struct commit *commit;
+	struct commit **chunk;
+	struct check_and_collect_until_cb_data cb;
+	struct reflog_commit_list list = { NULL, 0, 0 };
+	size_t count = 0, batch_size = 0;
+	int ret = 0;
+
+	commit = lookup_commit_reference(the_repository, &remote->old_oid);
+	if (!commit)
+		goto cleanup_return;
+
+	/*
+	 * Get the timestamp from the latest entry
+	 * of the remote-tracking ref's reflog.
+	 */
+	for_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);
+
+	cb.remote_commit = commit;
+	cb.local_commits = &list;
+	cb.remote_reflog_timestamp = date;
+	ret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);
+
+	/* We found an entry in the reflog. */
+	if (ret > 0)
+		goto cleanup_return;
+
+	/*
+	 * Check if the remote commit is reachable from any
+	 * of the commits in the collected list, in batches.
+	 */
+	chunk = list.items;
+	while (count < list.nr) {
+		batch_size = MERGE_BASES_BATCH_SIZE;
+
+		/* For any leftover entries. */
+		if ((count + MERGE_BASES_BATCH_SIZE) > list.nr)
+			batch_size = list.nr - count;
+
+		if ((ret = in_merge_bases_many(commit, batch_size, chunk)))
+			break;
+
+		chunk += batch_size;
+		count += MERGE_BASES_BATCH_SIZE;
+	}
+
+cleanup_return:
+	free_reflog_commit_list(&list);
+	return ret;
+}
+
+/*
+ * Check for reachability of a remote-tracking
+ * ref in the reflog entries of its local ref.
+ */
+static void check_if_includes_upstream(struct ref *remote)
+{
+	struct ref *local = get_local_ref(remote->name);
+	if (!local)
+		return;
+
+	if (is_reachable_in_reflog(local->name, remote) <= 0)
+		remote->unreachable = 1;
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
@@ -2282,8 +2438,12 @@ static void apply_cas(struct push_cas_option *cas,
 		ref->expect_old_sha1 = 1;
 		if (!entry->use_tracking)
 			oidcpy(&ref->old_oid_expect, &entry->expect);
-		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+		else if (remote_tracking(remote, ref->name,
+					 &ref->old_oid_expect,
+					 &ref->tracking_ref))
 			oidclr(&ref->old_oid_expect);
+		else
+			ref->check_reachable = cas->use_force_if_includes;
 		return;
 	}
 
@@ -2292,8 +2452,12 @@ static void apply_cas(struct push_cas_option *cas,
 		return;
 
 	ref->expect_old_sha1 = 1;
-	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+	if (remote_tracking(remote, ref->name,
+			    &ref->old_oid_expect,
+			    &ref->tracking_ref))
 		oidclr(&ref->old_oid_expect);
+	else
+		ref->check_reachable = cas->use_force_if_includes;
 }
 
 void apply_push_cas(struct push_cas_option *cas,
@@ -2301,6 +2465,15 @@ void apply_push_cas(struct push_cas_option *cas,
 		    struct ref *remote_refs)
 {
 	struct ref *ref;
-	for (ref = remote_refs; ref; ref = ref->next)
+	for (ref = remote_refs; ref; ref = ref->next) {
 		apply_cas(cas, remote, ref);
+
+		/*
+		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
+		 * mode, and if "--force-if-includes" was specified, run
+		 * the check.
+		 */
+		if (ref->check_reachable)
+			check_if_includes_upstream(ref);
+	}
 }
diff --git a/remote.h b/remote.h
index eb62a47044..2d5391d281 100644
--- a/remote.h
+++ b/remote.h
@@ -107,12 +107,20 @@ struct ref {
 	struct object_id new_oid;
 	struct object_id old_oid_expect; /* used by expect-old */
 	char *symref;
+	char *tracking_ref;
 	unsigned int
 		force:1,
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		/* Need to check if local reflog reaches the remote tip. */
+		check_reachable:1,
+		/*
+		 * Store the result of the check enabled by "check_reachable";
+		 * implies the local reflog does not reach the remote tip.
+		 */
+		unreachable:1;
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -142,6 +150,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);
 
 struct push_cas_option {
 	unsigned use_tracking_for_rest:1;
+	unsigned use_force_if_includes:1;
 	struct push_cas {
 		struct object_id expect;
 		unsigned use_tracking:1;
diff --git a/send-pack.c b/send-pack.c
index c9698070fc..eb4a44270b 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/transport-helper.c b/transport-helper.c
index b573b6c730..6157de30c7 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote ref updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
diff --git a/transport.c b/transport.c
index ffe2115845..65fcd22b20 100644
--- a/transport.c
+++ b/transport.c
@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,
 				 "stale info",
 				 report, porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote ref updated since checkout",
+				 report, porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
-- 
2.28.0

[PATCH v7 2/3] push: parse and set flag for "--force-if-includes"

[Srinidhi Kaushik]

The previous commit added the necessary machinery to implement the
"--force-if-includes" protection, when "--force-with-lease" is used
without giving exact object the remote still ought to have. Surface
the feature by adding a command line option and a configuration
variable to enable it.

 - Add a flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES" to indicate that the
   new option was passed from the command line of via configuration
   settings; update command line and configuration parsers to set the
   new flag accordingly.

 - Introduce a new configuration option "push.useForceIfIncludes", which
   is equivalent to setting "--force-if-includes" in the command line.

 - Update "remote-curl" to recognize and pass this option to "send-pack"
   when enabled.

 - Update "advise" to catch the reject reason "REJECT_REF_NEEDS_UPDATE",
   set when the ref status is "REF_STATUS_REJECT_REMOTE_UPDATED" and
   (optionally) print a help message when the push fails.

 - The new option is a "no-op" in the following scenarios:
    * When used without "--force-with-lease".
    * When used with "--force-with-lease", and if the expected commit
      on the remote side is specified as an argument.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 advice.c            |  3 +++
 advice.h            |  2 ++
 builtin/push.c      | 27 +++++++++++++++++++++++++++
 builtin/send-pack.c |  7 +++++++
 remote-curl.c       | 14 +++++++++++++-
 transport-helper.c  |  5 +++++
 transport.c         |  2 ++
 transport.h         | 15 ++++++++++-----
 8 files changed, 69 insertions(+), 6 deletions(-)

diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index 0eeb2c8dd5..908b557edb 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git pull ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
 
 	return 1;
@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		if (!v)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
+	} else if (!strcmp(k, "push.useforceifincludes")) {
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		return 0;
 	}
 
 	return git_default_config(k, v, NULL);
@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 	if ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))
 		die(_("--all and --mirror are incompatible"));
 
+	if (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))
+		cas.use_force_if_includes = 1;
+
 	for_each_string_list_item(item, push_options)
 		if (strchr(item->string, '\n'))
 			die(_("push options must not have new line characters"));
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 516cba7336..a284ada051 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int progress = -1;
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
+	int force_if_includes = 0;
 	struct packet_reader reader;
 
 	struct option options[] = {
@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 		  N_("require old value of ref to be at this value"),
 		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
+		OPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,
+			 N_("require remote updates to be integrated locally")),
 		OPT_END()
 	};
 
@@ -304,6 +307,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	if (!is_empty_cas(&cas))
 		apply_push_cas(&cas, remote, remote_refs);
 
+	if (!is_empty_cas(&cas) && force_if_includes)
+		cas.use_force_if_includes = 1;
+
+
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
 
diff --git a/remote-curl.c b/remote-curl.c
index 32cc4a0c55..0290b04891 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -44,7 +44,8 @@ struct options {
 		from_promisor : 1,
 
 		atomic : 1,
-		object_format : 1;
+		object_format : 1,
+		force_if_includes : 1;
 	const struct git_hash_algo *hash_algo;
 };
 static struct options options;
@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)
 		string_list_append(&cas_options, val.buf);
 		strbuf_release(&val);
 		return 0;
+	} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {
+		if (!strcmp(value, "true"))
+			options.force_if_includes = 1;
+		else if (!strcmp(value, "false"))
+			options.force_if_includes = 0;
+		else
+			return -1;
+		return 0;
 	} else if (!strcmp(name, "cloning")) {
 		if (!strcmp(value, "true"))
 			options.cloning = 1;
@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)
 		strvec_push(&args, cas_option->string);
 	strvec_push(&args, url.buf);
 
+	if (options.force_if_includes)
+		strvec_push(&args, "--force-if-includes");
+
 	strvec_push(&args, "--stdin");
 	for (i = 0; i < nr_spec; i++)
 		packet_buf_write(&preamble, "%s\n", specs[i]);
diff --git a/transport-helper.c b/transport-helper.c
index 6157de30c7..5f6e0b3bd8 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,
 		if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
 			die(_("helper %s does not support --atomic"), name);
 
+	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
+		if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
+			die(_("helper %s does not support --%s"),
+			    name, TRANS_OPT_FORCE_IF_INCLUDES);
+
 	if (flags & TRANSPORT_PUSH_OPTIONS) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, transport->push_options)
diff --git a/transport.c b/transport.c
index 65fcd22b20..47da955e4f 100644
--- a/transport.c
+++ b/transport.c
@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
diff --git a/transport.h b/transport.h
index ca409ea1e4..24558c027d 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);
 /* Request atomic (all-or-nothing) updates when pushing */
 #define TRANS_OPT_ATOMIC "atomic"
 
+/* Require remote changes to be integrated locally. */
+#define TRANS_OPT_FORCE_IF_INCLUDES "force-if-includes"
+
 /**
  * Returns 0 if the option was used, non-zero otherwise. Prints a
  * message to stderr if the option is not used.
@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v7 3/3] t, doc: update tests, reference for "--force-if-includes"

[Srinidhi Kaushik]

Update test cases for the new option, and document its usage
and update related references.

Update test cases for the new option, and document its usage
and update related references.

 - t/t5533-push-cas.sh:
   Update test cases for "compare-and-swap" when used along with
   "--force-if-includes" helps mitigate overwrites when remote
   refs are updated in the background; allows forced updates when
   changes from remote are integrated locally.

 - Documentation:
   Add reference for the new option, configuration setting
   ("push.useForceIfIncludes") and advise messages.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt |   9 ++-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 ++++++-
 t/t5533-push-cas.sh             | 129 ++++++++++++++++++++++++++++++++
 4 files changed, 166 insertions(+), 4 deletions(-)

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..acbd0c09aa 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -10,9 +10,8 @@ advice.*::
 		that the check is disabled.
 	pushUpdateRejected::
 		Set this variable to 'false' if you want to disable
-		'pushNonFFCurrent',
-		'pushNonFFMatching', 'pushAlreadyExists',
-		'pushFetchFirst', and 'pushNeedsForce'
+		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
+		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
 		simultaneously.
 	pushNonFFCurrent::
 		Advice shown when linkgit:git-push[1] fails due to a
@@ -41,6 +40,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..21b256e0a4 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,9 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.useForceIfIncludes::
+	If set to "true", it is equivalent to specifying
+	`--force-if-includes` as an option to linkgit:git-push[1]
+	in the command line. Adding `--no-force-if-includes` at the
+	time of push overrides this configuration setting.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..fb3a220386 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -13,7 +13,7 @@ SYNOPSIS
 	   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
 	   [-u | --set-upstream] [-o <string> | --push-option=<string>]
 	   [--[no-]signed|--signed=(true|false|if-asked)]
-	   [--force-with-lease[=<refname>[:<expect>]]]
+	   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]
 	   [--no-verify] [<repository> [<refspec>...]]
 
 DESCRIPTION
@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying `--force-if-includes` an an ancillary option
+along with `--force-with-lease[=<refname>]` (i.e., without saying what
+exact commit the ref on the remote side must be pointing at, or which
+refs on the remote side are being protected) at the time of "push" will
+verify if updates from the remote-tracking refs that may have been
+implicitly updated in the background are integrated locally before
+allowing a forced update.
 
 -f::
 --force::
@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option enables a check that verifies if the tip of the
+remote-tracking ref is reachable from one of the "reflog" entries of
+the local branch based in it for a rewrite. The check ensures that any
+updates from the remote have been incorporated locally by rejecting the
+forced update if that is not the case.
++
+If the option is passed without specifying `--force-with-lease`, or
+specified along with `--force-with-lease=<refname>:<expect>`, it is
+a "no-op".
++
+Specifying `--no-force-if-includes` disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..ed44ddc654 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -13,6 +13,46 @@ setup_srcdst_basic () {
 	)
 }
 
+# For tests with "--force-if-includes".
+setup_src_dup_dst () {
+	rm -fr src dup dst &&
+	git init --bare dst &&
+	git clone --no-local dst src &&
+	git clone --no-local dst dup
+	(
+		cd src &&
+		test_commit A &&
+		test_commit B &&
+		test_commit C &&
+		git push
+	) &&
+	(
+		cd dup &&
+		git fetch &&
+		git merge origin/master &&
+		git switch -c branch master~2 &&
+		test_commit D &&
+		test_commit E &&
+		git push --all
+	) &&
+	(
+		cd src &&
+		git switch master &&
+		git fetch --all &&
+		git branch branch --track origin/branch &&
+		git rebase origin/master
+	) &&
+	(
+		cd dup &&
+		git switch master &&
+		test_commit F &&
+		test_commit G &&
+		git switch branch &&
+		test_commit H &&
+		git push origin --all
+	)
+}
+
 test_expect_success setup '
 	# create template repository
 	test_commit A &&
@@ -256,4 +296,93 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates to remote can be mitigated with "--force-if-includes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		test_must_fail git push --force-with-lease --force-if-includes --all
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success 'background updates to remote can be mitigated with "push.useForceIfIncludes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		git config --local push.useForceIfIncludes true &&
+		test_must_fail git push --force-with-lease=master origin master
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success '"--force-if-includes" should be disabled for --force-with-lease="<refname>:<expect>"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		remote_head="$(git rev-parse refs/remotes/origin/master)" &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes --force-with-lease="master:$remote_head" 2>err &&
+		grep "stale info" err
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase")' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase", local rebase)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git rebase --onto HEAD~4 HEAD~1 &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
 test_done
-- 
2.28.0

Re: [PATCH v7 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> @@ -2252,11 +2263,11 @@ int is_empty_cas(const struct push_cas_option *cas)
>  /*
>   * Look at remote.fetch refspec and see if we have a remote
>   * tracking branch for the refname there.  Fill its current
> - * value in sha1[].
> + * value in sha1[], and as a string.

I think the array being referred to was renamed to oid[] sometime
ago.  "and as a string" makes it sound as if sha1[] gets the value
as 40-hex object name text, but that is not what is being done.

    Fill the name of the remote-tracking branch in *dst_refname,
    and the name of the commit object at tis tip in oid[].

perhaps?

>   * If we cannot do so, return negative to signal an error.
>   */
>  static int remote_tracking(struct remote *remote, const char *refname,
> -			   struct object_id *oid)
> +			   struct object_id *oid, char **dst_refname)
>  {
>  	char *dst;
>  
> @@ -2265,9 +2276,154 @@ static int remote_tracking(struct remote *remote, const char *refname,
>  		return -1; /* no tracking ref for refname at remote */
>  	if (read_ref(dst, oid))
>  		return -1; /* we know what the tracking ref is but we cannot read it */
> +
> +	*dst_refname = dst;
> +	return 0;
> +}
> +
> +/*
> + * The struct "reflog_commit_list" and related helper functions
> + * for list manipulation are used for collecting commits into a
> + * list during reflog traversals in "if_exists_or_grab_until()".

Has the name of that function changed since this comment was
written?

> + */
> +struct reflog_commit_list {
> +	struct commit **items;

Name an array in singular when its primary use is to work on an
element at a time---that will let you say item[4] to call the 4-th
item, instead of items[4] that smells awkward.

An array that is used mostly to pass around a collection as a whole
is easier to think about when given a plural name, though.

> +	size_t nr, alloc;
> +};
> +
> +/* Add a commit to list. */
> +static void add_commit(struct reflog_commit_list *list, struct commit *commit)
> +{
> +	ALLOC_GROW(list->items, list->nr + 1, list->alloc);
> +	list->items[list->nr++] = commit;
> +}
> +
> +/* Free and reset the list. */
> +static void free_reflog_commit_list(struct reflog_commit_list *list)
> +{
> +	FREE_AND_NULL(list->items);
> +	list->nr = list->alloc = 0;
> +}
> +
> +struct check_and_collect_until_cb_data {
> +	struct commit *remote_commit;
> +	struct reflog_commit_list *local_commits;
> +	timestamp_t remote_reflog_timestamp;
> +};
> +
> +/* Get the timestamp of the latest entry. */
> +static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,
> +		       const char *ident, timestamp_t timestamp,
> +		       int tz, const char *message, void *cb_data)
> +{
> +	timestamp_t *ts = cb_data;
> +	*ts = timestamp;
> +	return 1;
> +}

The idea is to use a callback that immediately says "no more" to
grab the data from the first item in the iteration.  It feels
somewhat awkward but because there is no "give us the Nth entry" API
function, it is the cleanest way we can do this.

> +static int check_and_collect_until(struct object_id *o_oid,
> +				   struct object_id *n_oid,
> +				   const char *ident, timestamp_t timestamp,
> +				   int tz, const char *message, void *cb_data)
> +{
> +	struct commit *commit;
> +	struct check_and_collect_until_cb_data *cb = cb_data;
> +
> +	/*
> +	 * If the reflog entry timestamp is older than the remote ref's
> +	 * latest reflog entry, there is no need to check or collect
> +	 * entries older than this one.
> +	 */
> +	if (timestamp < cb->remote_reflog_timestamp)
> +		return -1;
> +
> +	/* An entry was found. */
> +	if (oideq(n_oid, &cb->remote_commit->object.oid))
> +		return 1;
> +
> +	/* Look-up the commit and append it to the list. */
> +	if ((commit = lookup_commit_reference(the_repository, n_oid)))
> +		add_commit(cb->local_commits, commit);

This is merely a minor naming thing, but if you rename add_commit()
to append_commit(), you probably do not even need the comment before
this statement.

>  	return 0;
>  }
>  
> +#define MERGE_BASES_BATCH_SIZE 8

Hmph.  Do we still need batching?

> +/*
> + * Iterate through the reflog of the local ref to check if there is an entry
> + * for the given remote-tracking ref; runs until the timestamp of an entry is
> + * older than latest timestamp of remote-tracking ref's reflog. Any commits
> + * are that seen along the way are collected into a list to check if the
> + * remote-tracking ref is reachable from any of them.
> + */
> +static int is_reachable_in_reflog(const char *local, const struct ref *remote)
> +{
> +	timestamp_t date;
> +	struct commit *commit;
> +	struct commit **chunk;
> +	struct check_and_collect_until_cb_data cb;
> +	struct reflog_commit_list list = { NULL, 0, 0 };
> +	size_t count = 0, batch_size = 0;
> +	int ret = 0;
> +
> +	commit = lookup_commit_reference(the_repository, &remote->old_oid);
> +	if (!commit)
> +		goto cleanup_return;
> +
> +	/*
> +	 * Get the timestamp from the latest entry
> +	 * of the remote-tracking ref's reflog.
> +	 */
> +	for_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);
> +
> +	cb.remote_commit = commit;
> +	cb.local_commits = &list;
> +	cb.remote_reflog_timestamp = date;
> +	ret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);
> +
> +	/* We found an entry in the reflog. */
> +	if (ret > 0)
> +		goto cleanup_return;

Good.  So '1' from the callback is "we found one, no need to look
further and no need to do merge-base", and '-1' from the callback is
"we looked at all entries that are young enough to matter and we
didn't find exact match".  Makes sense.

> +	/*
> +	 * Check if the remote commit is reachable from any
> +	 * of the commits in the collected list, in batches.
> +	 */

I do not know if batching would help (have you measured it?), but if
we were to batch, it is more common to arrange the loop like this:

	for (chunk = list.items;
             chunk < list.items + list.nr;
	     chunk += size) {
             	size = list.items + list.nr - chunk;
                if (MERGE_BASES_BATCH_SIZE < size)
			size = MERGE_BASES_BATCH_SIZE;
		... use chunk[0..size] ...
		chunk += size;
	}

That is, assume that we can grab everything during this round, and
if that bites off too many, clamp it to the maximum value.  If you
are not comfortable with pointer arithmetic, it is also fine to use
an auxiliary variable 'count', but ...

> +	chunk = list.items;
> +	while (count < list.nr) {
> +		batch_size = MERGE_BASES_BATCH_SIZE;
> +
> +		/* For any leftover entries. */
> +		if ((count + MERGE_BASES_BATCH_SIZE) > list.nr)
> +			batch_size = list.nr - count;
> +
> +		if ((ret = in_merge_bases_many(commit, batch_size, chunk)))
> +			break;
> +
> +		chunk += batch_size;
> +		count += MERGE_BASES_BATCH_SIZE;

... you are risking chunk and count to go out of sync here.

It does not matter within this loop (count will point beyond the end
of list.item[] while chunk will never go past the array), but future
developers can be confused into thinking that they can use chunk and
count interchangeably after this loop exits, and at that point the
discrepancy may start to matter.

But all of the above matters if it is a good idea to batch.  Does it
make a difference?

    ... goes and looks at in_merge_bases_many() ...

Ah, it probably would.  

I thought in_merge_bases_many() would stop early as soon as any of
the traversal from chunk[] reaches commit, but it uses a rather more
generic paint_down_to_common() so extra items in chunk[] that are
topologically older than commit would result in additional traversal
from commit down to them, which would not contribute much to the end
result.  It may be a good #leftovebit idea for future improvement to
teach in_merge_bases_many() to use a custom replacement for
paint_down_to_common() that stops early as soon as we find the
answer is true.

> +	}
> +
> +cleanup_return:
> +	free_reflog_commit_list(&list);
> +	return ret;
> +}
> +

Thanks.

Re: [PATCH v7 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Hi Junio,

On 09/26/2020 16:42, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> > @@ -2252,11 +2263,11 @@ int is_empty_cas(const struct push_cas_option *cas)
> >  /*
> >   * Look at remote.fetch refspec and see if we have a remote
> >   * tracking branch for the refname there.  Fill its current
> > - * value in sha1[].
> > + * value in sha1[], and as a string.
> 
> I think the array being referred to was renamed to oid[] sometime
> ago.  "and as a string" makes it sound as if sha1[] gets the value
> as 40-hex object name text, but that is not what is being done.
> 
>     Fill the name of the remote-tracking branch in *dst_refname,
>     and the name of the commit object at tis tip in oid[].
> 
> perhaps?

Of course, that sounds better; will update.
 
> > + * The struct "reflog_commit_list" and related helper functions
> > + * for list manipulation are used for collecting commits into a
> > + * list during reflog traversals in "if_exists_or_grab_until()".
> 
> Has the name of that function changed since this comment was
> written?

Heh, it sure has. It should have been "check_and_collect_until()".
 
> > + */
> > +struct reflog_commit_list {
> > +	struct commit **items;
> 
> Name an array in singular when its primary use is to work on an
> element at a time---that will let you say item[4] to call the 4-th
> item, instead of items[4] that smells awkward.
> 
> An array that is used mostly to pass around a collection as a whole
> is easier to think about when given a plural name, though.

Yup.

> > +
> > +/* Get the timestamp of the latest entry. */
> > +static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,
> > +		       const char *ident, timestamp_t timestamp,
> > +		       int tz, const char *message, void *cb_data)
> > +{
> > +	timestamp_t *ts = cb_data;
> > +	*ts = timestamp;
> > +	return 1;
> > +}
> 
> The idea is to use a callback that immediately says "no more" to
> grab the data from the first item in the iteration.  It feels
> somewhat awkward but because there is no "give us the Nth entry" API
> function, it is the cleanest way we can do this.

I considered using "grab_1st_entry_timestamp()" briefy, but
"peek_reflog" is shorter compared to that.

> > +	/* Look-up the commit and append it to the list. */
> > +	if ((commit = lookup_commit_reference(the_repository, n_oid)))
> > +		add_commit(cb->local_commits, commit);
> 
> This is merely a minor naming thing, but if you rename add_commit()
> to append_commit(), you probably do not even need the comment before
> this statement.

Will do.

> >  	return 0;
> >  }
> >  
> > +#define MERGE_BASES_BATCH_SIZE 8
> 
> Hmph.  Do we still need batching?
> 
> > +/*
> > + * Iterate through the reflog of the local ref to check if there is an entry
> > + * for the given remote-tracking ref; runs until the timestamp of an entry is
> > + * older than latest timestamp of remote-tracking ref's reflog. Any commits
> > + * are that seen along the way are collected into a list to check if the
> > + * remote-tracking ref is reachable from any of them.
> > + */
> > +static int is_reachable_in_reflog(const char *local, const struct ref *remote)
> > +{
> > +	timestamp_t date;
> > +	struct commit *commit;
> > +	struct commit **chunk;
> > +	struct check_and_collect_until_cb_data cb;
> > +	struct reflog_commit_list list = { NULL, 0, 0 };
> > +	size_t count = 0, batch_size = 0;
> > +	int ret = 0;
> > +
> > +	commit = lookup_commit_reference(the_repository, &remote->old_oid);
> > +	if (!commit)
> > +		goto cleanup_return;
> > +
> > +	/*
> > +	 * Get the timestamp from the latest entry
> > +	 * of the remote-tracking ref's reflog.
> > +	 */
> > +	for_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);
> > +
> > +	cb.remote_commit = commit;
> > +	cb.local_commits = &list;
> > +	cb.remote_reflog_timestamp = date;
> > +	ret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);
> > +
> > +	/* We found an entry in the reflog. */
> > +	if (ret > 0)
> > +		goto cleanup_return;
> 
> Good.  So '1' from the callback is "we found one, no need to look
> further and no need to do merge-base", and '-1' from the callback is
> "we looked at all entries that are young enough to matter and we
> didn't find exact match".  Makes sense.
> 
> > +	/*
> > +	 * Check if the remote commit is reachable from any
> > +	 * of the commits in the collected list, in batches.
> > +	 */
> 
> I do not know if batching would help (have you measured it?), but if
> we were to batch, it is more common to arrange the loop like this:
> 
> 	for (chunk = list.items;
>              chunk < list.items + list.nr;
> 	     chunk += size) {
>              	size = list.items + list.nr - chunk;
>                 if (MERGE_BASES_BATCH_SIZE < size)
> 			size = MERGE_BASES_BATCH_SIZE;
> 		... use chunk[0..size] ...
> 		chunk += size;
> 	}
> 
> That is, assume that we can grab everything during this round, and
> if that bites off too many, clamp it to the maximum value.  If you
> are not comfortable with pointer arithmetic, it is also fine to use
> an auxiliary variable 'count', but ...

Actually, the "for" version looks much cleaner and avoids the use
of "count". However, I think ...

>               chunk += size;

... should be skipped because "for ( ... ; chunk += size)" is already
doing it for us; otherwise we would offset 16 entries instead of 8
per iteration, no?

> > +	chunk = list.items;
> > +	while (count < list.nr) {
> > +		batch_size = MERGE_BASES_BATCH_SIZE;
> > +
> > +		/* For any leftover entries. */
> > +		if ((count + MERGE_BASES_BATCH_SIZE) > list.nr)
> > +			batch_size = list.nr - count;
> > +
> > +		if ((ret = in_merge_bases_many(commit, batch_size, chunk)))
> > +			break;
> > +
> > +		chunk += batch_size;
> > +		count += MERGE_BASES_BATCH_SIZE;
> 
> ... you are risking chunk and count to go out of sync here.
> 
> It does not matter within this loop (count will point beyond the end
> of list.item[] while chunk will never go past the array), but future
> developers can be confused into thinking that they can use chunk and
> count interchangeably after this loop exits, and at that point the
> discrepancy may start to matter.

I agree, it should have been "count += batch_size;". But, I think the
"for" version looks cleaner; I will change it to that the next set.
 
> But all of the above matters if it is a good idea to batch.  Does it
> make a difference?
> 
>     ... goes and looks at in_merge_bases_many() ...
> 
> Ah, it probably would.  
> 
> I thought in_merge_bases_many() would stop early as soon as any of
> the traversal from chunk[] reaches commit, but it uses a rather more
> generic paint_down_to_common() so extra items in chunk[] that are
> topologically older than commit would result in additional traversal
> from commit down to them, which would not contribute much to the end
> result.  It may be a good #leftovebit idea for future improvement to
> teach in_merge_bases_many() to use a custom replacement for
> paint_down_to_common() that stops early as soon as we find the
> answer is true.

If we consider the amount of time it takes when "in_merge_bases_many()"
has to be run for all the entries, there isn't much of a difference in
performance between batching and non-batching -- they took about the
same. But, as you said if the remote is reachable in the first few
entries, batching would help with returning early if a descendant is
found.

Making the function stop early when a descendent is found
does sound like a good #leftoverbits idea. :)

Thanks again, for a detailed review.
-- 
Srinidhi Kaushik

[PATCH v8 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: "--force-if-includes" to "git-push" where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref -- on which a local branch has based on -- is reachable from at
least one of the "reflog" entries of the branch about to be updated
by force on the remote.

This option can be used with "--force-with-lease" with setups where
the remote-tracking refs of the repository are implicitly updated in
the background to help prevent unintended remote overwrites.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with "--force-with-lease[=<refname>[:<expect>]]" without specifying
the "<expect>" value, can cause the update that happened in-between
the checkout and forced push to be lost.

Changes since v7:
  - Clean up the way batching is done for "in_merge_bases_many()".

  - The timestamp check during "reflog" iteration has been moved
    to the end of the function because we should stop collecting
    entries older than the current one (i.e., in the next round).

  - Add a test case to show deletes should be allowed with the
    new option.

  - Minor changes to comments and function names.

Srinidhi Kaushik (3):
  push: add reflog check for "--force-if-includes"
  push: parse and set flag for "--force-if-includes"
  t, doc: update tests, reference for "--force-if-includes"

 Documentation/config/advice.txt |   9 +-
 Documentation/config/push.txt   |   6 +
 Documentation/git-push.txt      |  26 ++++-
 advice.c                        |   3 +
 advice.h                        |   2 +
 builtin/push.c                  |  27 +++++
 builtin/send-pack.c             |  12 ++
 remote-curl.c                   |  14 ++-
 remote.c                        | 188 ++++++++++++++++++++++++++++++--
 remote.h                        |  12 +-
 send-pack.c                     |   1 +
 t/t5533-push-cas.sh             | 140 ++++++++++++++++++++++++
 transport-helper.c              |  10 ++
 transport.c                     |   8 ++
 transport.h                     |  15 ++-
 15 files changed, 455 insertions(+), 18 deletions(-)

base-commit: 9bc233ae1cf19a49e51842c7959d80a675dbd1c0
--
2.28.0

[PATCH v8 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Add a check to verify if the remote-tracking ref of the local branch
is reachable from one of its "reflog" entries.

The check iterates through the local ref's reflog to see if there
is an entry for the remote-tracking ref and collecting any commits
that are seen, into a list; the iteration stops if an entry in the
reflog matches the remote ref or if the entry timestamp is older
the latest entry of the remote ref's "reflog". If there wasn't an
entry found for the remote ref, "in_merge_bases_many()" is called
to check if it is reachable from the list of collected commits.

When a local branch that is based on a remote ref, has been rewound
and is to be force pushed on the remote, "--force-if-includes" runs
a check that ensures any updates to the remote-tracking ref that may
have happened (by push from another repository) in-between the time
of the last update to the local branch (via "git-pull", for instance)
and right before the time of push, have been integrated locally
before allowing a forced update.

If the new option is passed without specifying "--force-with-lease",
or specified along with "--force-with-lease=<refname>:<expect>" it
is a "no-op".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 builtin/send-pack.c |   5 ++
 remote.c            | 188 ++++++++++++++++++++++++++++++++++++++++++--
 remote.h            |  12 ++-
 send-pack.c         |   1 +
 transport-helper.c  |   5 ++
 transport.c         |   6 ++
 6 files changed, 209 insertions(+), 8 deletions(-)

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 7af148d733..516cba7336 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote ref updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/remote.c b/remote.c
index eafc14cbe7..4781680409 100644
--- a/remote.c
+++ b/remote.c
@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 		 * with the remote-tracking branch to find the value
 		 * to expect, but we did not have such a tracking
 		 * branch.
+		 *
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
 		 */
 		if (ref->expect_old_sha1) {
 			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
 				reject_reason = REF_STATUS_REJECT_STALE;
+			else if (ref->check_reachable && ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
 			else
-				/* If the ref isn't stale then force the update. */
+				/*
+				 * If the ref isn't stale, and is reachable
+				 * from from one of the reflog entries of
+				 * the local branch, force the update.
+				 */
 				force_ref_update = 1;
 		}
 
@@ -2251,12 +2262,13 @@ int is_empty_cas(const struct push_cas_option *cas)
 
 /*
  * Look at remote.fetch refspec and see if we have a remote
- * tracking branch for the refname there.  Fill its current
- * value in sha1[].
+ * tracking branch for the refname there. Fill the name of
+ * the remote-tracking branch in *dst_refname, and the name
+ * of the commit object at its tip in oid[].
  * If we cannot do so, return negative to signal an error.
  */
 static int remote_tracking(struct remote *remote, const char *refname,
-			   struct object_id *oid)
+			   struct object_id *oid, char **dst_refname)
 {
 	char *dst;
 
@@ -2265,9 +2277,154 @@ static int remote_tracking(struct remote *remote, const char *refname,
 		return -1; /* no tracking ref for refname at remote */
 	if (read_ref(dst, oid))
 		return -1; /* we know what the tracking ref is but we cannot read it */
+
+	*dst_refname = dst;
+	return 0;
+}
+
+/*
+ * The struct "reflog_commit_list" and related helper functions
+ * for list manipulation are used for collecting commits into a
+ * list during reflog traversals in "check_and_collect_until()".
+ */
+struct reflog_commit_list {
+	struct commit **items;
+	size_t nr, alloc;
+};
+
+/* Append a commit to the list. */
+static void append_commit(struct reflog_commit_list *list,
+			  struct commit *commit)
+{
+	ALLOC_GROW(list->items, list->nr + 1, list->alloc);
+	list->items[list->nr++] = commit;
+}
+
+/* Free and reset the list. */
+static void free_reflog_commit_list(struct reflog_commit_list *list)
+{
+	FREE_AND_NULL(list->items);
+	list->nr = list->alloc = 0;
+}
+
+struct check_and_collect_until_cb_data {
+	struct commit *remote_commit;
+	struct reflog_commit_list *local_commits;
+	timestamp_t remote_reflog_timestamp;
+};
+
+/* Get the timestamp of the latest entry. */
+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,
+		       const char *ident, timestamp_t timestamp,
+		       int tz, const char *message, void *cb_data)
+{
+	timestamp_t *ts = cb_data;
+	*ts = timestamp;
+	return 1;
+}
+
+static int check_and_collect_until(struct object_id *o_oid,
+				   struct object_id *n_oid,
+				   const char *ident, timestamp_t timestamp,
+				   int tz, const char *message, void *cb_data)
+{
+	struct commit *commit;
+	struct check_and_collect_until_cb_data *cb = cb_data;
+
+	/* An entry was found. */
+	if (oideq(n_oid, &cb->remote_commit->object.oid))
+		return 1;
+
+	if ((commit = lookup_commit_reference(the_repository, n_oid)))
+		append_commit(cb->local_commits, commit);
+
+	/*
+	 * If the reflog entry timestamp is older than the remote ref's
+	 * latest reflog entry, there is no need to check or collect
+	 * entries older than this one.
+	 */
+	if (timestamp < cb->remote_reflog_timestamp)
+		return -1;
+
 	return 0;
 }
 
+#define MERGE_BASES_BATCH_SIZE 8
+
+/*
+ * Iterate through the reflog of the local ref to check if there is an entry
+ * for the given remote-tracking ref; runs until the timestamp of an entry is
+ * older than latest timestamp of remote-tracking ref's reflog. Any commits
+ * are that seen along the way are collected into a list to check if the
+ * remote-tracking ref is reachable from any of them.
+ */
+static int is_reachable_in_reflog(const char *local, const struct ref *remote)
+{
+	timestamp_t date;
+	struct commit *commit;
+	struct commit **chunk;
+	struct check_and_collect_until_cb_data cb;
+	struct reflog_commit_list list = { NULL, 0, 0 };
+	size_t size = 0;
+	int ret = 0;
+
+	commit = lookup_commit_reference(the_repository, &remote->old_oid);
+	if (!commit)
+		goto cleanup_return;
+
+	/*
+	 * Get the timestamp from the latest entry
+	 * of the remote-tracking ref's reflog.
+	 */
+	for_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);
+
+	cb.remote_commit = commit;
+	cb.local_commits = &list;
+	cb.remote_reflog_timestamp = date;
+	ret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);
+
+	/* We found an entry in the reflog. */
+	if (ret > 0)
+		goto cleanup_return;
+
+	/*
+	 * Check if the remote commit is reachable from any
+	 * of the commits in the collected list, in batches.
+	 */
+	for (chunk = list.items; chunk < list.items + list.nr; chunk += size) {
+		size = list.items + list.nr - chunk;
+
+		/*
+		 * If the number of remaining entries is more than
+		 * the batch size, clamp the number of items passed
+		 * to the maximum batch size.
+		 */
+		if (MERGE_BASES_BATCH_SIZE < size)
+			size = MERGE_BASES_BATCH_SIZE;
+
+		if ((ret = in_merge_bases_many(commit, size, chunk)))
+			break;
+	}
+
+cleanup_return:
+	free_reflog_commit_list(&list);
+	return ret;
+}
+
+/*
+ * Check for reachability of a remote-tracking
+ * ref in the reflog entries of its local ref.
+ */
+static void check_if_includes_upstream(struct ref *remote)
+{
+	struct ref *local = get_local_ref(remote->name);
+	if (!local)
+		return;
+
+	if (is_reachable_in_reflog(local->name, remote) <= 0)
+		remote->unreachable = 1;
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
@@ -2282,8 +2439,12 @@ static void apply_cas(struct push_cas_option *cas,
 		ref->expect_old_sha1 = 1;
 		if (!entry->use_tracking)
 			oidcpy(&ref->old_oid_expect, &entry->expect);
-		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+		else if (remote_tracking(remote, ref->name,
+					 &ref->old_oid_expect,
+					 &ref->tracking_ref))
 			oidclr(&ref->old_oid_expect);
+		else
+			ref->check_reachable = cas->use_force_if_includes;
 		return;
 	}
 
@@ -2292,8 +2453,12 @@ static void apply_cas(struct push_cas_option *cas,
 		return;
 
 	ref->expect_old_sha1 = 1;
-	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+	if (remote_tracking(remote, ref->name,
+			    &ref->old_oid_expect,
+			    &ref->tracking_ref))
 		oidclr(&ref->old_oid_expect);
+	else
+		ref->check_reachable = cas->use_force_if_includes;
 }
 
 void apply_push_cas(struct push_cas_option *cas,
@@ -2301,6 +2466,15 @@ void apply_push_cas(struct push_cas_option *cas,
 		    struct ref *remote_refs)
 {
 	struct ref *ref;
-	for (ref = remote_refs; ref; ref = ref->next)
+	for (ref = remote_refs; ref; ref = ref->next) {
 		apply_cas(cas, remote, ref);
+
+		/*
+		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
+		 * mode, and if "--force-if-includes" was specified, run
+		 * the check.
+		 */
+		if (ref->check_reachable)
+			check_if_includes_upstream(ref);
+	}
 }
diff --git a/remote.h b/remote.h
index eb62a47044..2d5391d281 100644
--- a/remote.h
+++ b/remote.h
@@ -107,12 +107,20 @@ struct ref {
 	struct object_id new_oid;
 	struct object_id old_oid_expect; /* used by expect-old */
 	char *symref;
+	char *tracking_ref;
 	unsigned int
 		force:1,
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		/* Need to check if local reflog reaches the remote tip. */
+		check_reachable:1,
+		/*
+		 * Store the result of the check enabled by "check_reachable";
+		 * implies the local reflog does not reach the remote tip.
+		 */
+		unreachable:1;
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -142,6 +150,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);
 
 struct push_cas_option {
 	unsigned use_tracking_for_rest:1;
+	unsigned use_force_if_includes:1;
 	struct push_cas {
 		struct object_id expect;
 		unsigned use_tracking:1;
diff --git a/send-pack.c b/send-pack.c
index c9698070fc..eb4a44270b 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/transport-helper.c b/transport-helper.c
index b573b6c730..6157de30c7 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote ref updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
diff --git a/transport.c b/transport.c
index ffe2115845..65fcd22b20 100644
--- a/transport.c
+++ b/transport.c
@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,
 				 "stale info",
 				 report, porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote ref updated since checkout",
+				 report, porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
-- 
2.28.0

[PATCH v8 2/3] push: parse and set flag for "--force-if-includes"

[Srinidhi Kaushik]

The previous commit added the necessary machinery to implement the
"--force-if-includes" protection, when "--force-with-lease" is used
without giving exact object the remote still ought to have. Surface
the feature by adding a command line option and a configuration
variable to enable it.

 - Add a flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES" to indicate that the
   new option was passed from the command line of via configuration
   settings; update command line and configuration parsers to set the
   new flag accordingly.

 - Introduce a new configuration option "push.useForceIfIncludes", which
   is equivalent to setting "--force-if-includes" in the command line.

 - Update "remote-curl" to recognize and pass this option to "send-pack"
   when enabled.

 - Update "advise" to catch the reject reason "REJECT_REF_NEEDS_UPDATE",
   set when the ref status is "REF_STATUS_REJECT_REMOTE_UPDATED" and
   (optionally) print a help message when the push fails.

 - The new option is a "no-op" in the following scenarios:
    * When used without "--force-with-lease".
    * When used with "--force-with-lease", and if the expected commit
      on the remote side is specified as an argument.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 advice.c            |  3 +++
 advice.h            |  2 ++
 builtin/push.c      | 27 +++++++++++++++++++++++++++
 builtin/send-pack.c |  7 +++++++
 remote-curl.c       | 14 +++++++++++++-
 transport-helper.c  |  5 +++++
 transport.c         |  2 ++
 transport.h         | 15 ++++++++++-----
 8 files changed, 69 insertions(+), 6 deletions(-)

diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index 0eeb2c8dd5..908b557edb 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git pull ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
 
 	return 1;
@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		if (!v)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
+	} else if (!strcmp(k, "push.useforceifincludes")) {
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		return 0;
 	}
 
 	return git_default_config(k, v, NULL);
@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 	if ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))
 		die(_("--all and --mirror are incompatible"));
 
+	if (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))
+		cas.use_force_if_includes = 1;
+
 	for_each_string_list_item(item, push_options)
 		if (strchr(item->string, '\n'))
 			die(_("push options must not have new line characters"));
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 516cba7336..a284ada051 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int progress = -1;
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
+	int force_if_includes = 0;
 	struct packet_reader reader;
 
 	struct option options[] = {
@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 		  N_("require old value of ref to be at this value"),
 		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
+		OPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,
+			 N_("require remote updates to be integrated locally")),
 		OPT_END()
 	};
 
@@ -304,6 +307,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	if (!is_empty_cas(&cas))
 		apply_push_cas(&cas, remote, remote_refs);
 
+	if (!is_empty_cas(&cas) && force_if_includes)
+		cas.use_force_if_includes = 1;
+
+
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
 
diff --git a/remote-curl.c b/remote-curl.c
index 32cc4a0c55..0290b04891 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -44,7 +44,8 @@ struct options {
 		from_promisor : 1,
 
 		atomic : 1,
-		object_format : 1;
+		object_format : 1,
+		force_if_includes : 1;
 	const struct git_hash_algo *hash_algo;
 };
 static struct options options;
@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)
 		string_list_append(&cas_options, val.buf);
 		strbuf_release(&val);
 		return 0;
+	} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {
+		if (!strcmp(value, "true"))
+			options.force_if_includes = 1;
+		else if (!strcmp(value, "false"))
+			options.force_if_includes = 0;
+		else
+			return -1;
+		return 0;
 	} else if (!strcmp(name, "cloning")) {
 		if (!strcmp(value, "true"))
 			options.cloning = 1;
@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)
 		strvec_push(&args, cas_option->string);
 	strvec_push(&args, url.buf);
 
+	if (options.force_if_includes)
+		strvec_push(&args, "--force-if-includes");
+
 	strvec_push(&args, "--stdin");
 	for (i = 0; i < nr_spec; i++)
 		packet_buf_write(&preamble, "%s\n", specs[i]);
diff --git a/transport-helper.c b/transport-helper.c
index 6157de30c7..5f6e0b3bd8 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,
 		if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
 			die(_("helper %s does not support --atomic"), name);
 
+	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
+		if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
+			die(_("helper %s does not support --%s"),
+			    name, TRANS_OPT_FORCE_IF_INCLUDES);
+
 	if (flags & TRANSPORT_PUSH_OPTIONS) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, transport->push_options)
diff --git a/transport.c b/transport.c
index 65fcd22b20..47da955e4f 100644
--- a/transport.c
+++ b/transport.c
@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
diff --git a/transport.h b/transport.h
index ca409ea1e4..24558c027d 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);
 /* Request atomic (all-or-nothing) updates when pushing */
 #define TRANS_OPT_ATOMIC "atomic"
 
+/* Require remote changes to be integrated locally. */
+#define TRANS_OPT_FORCE_IF_INCLUDES "force-if-includes"
+
 /**
  * Returns 0 if the option was used, non-zero otherwise. Prints a
  * message to stderr if the option is not used.
@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v8 3/3] t, doc: update tests, reference for "--force-if-includes"

[Srinidhi Kaushik]

Update test cases for the new option, and document its usage
and update related references.

Update test cases for the new option, and document its usage
and update related references.

 - t/t5533-push-cas.sh:
   Update test cases for "compare-and-swap" when used along with
   "--force-if-includes" helps mitigate overwrites when remote
   refs are updated in the background; allows forced updates when
   changes from remote are integrated locally.

 - Documentation:
   Add reference for the new option, configuration setting
   ("push.useForceIfIncludes") and advise messages.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt |   9 +-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 +++++-
 t/t5533-push-cas.sh             | 140 ++++++++++++++++++++++++++++++++
 4 files changed, 177 insertions(+), 4 deletions(-)

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..acbd0c09aa 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -10,9 +10,8 @@ advice.*::
 		that the check is disabled.
 	pushUpdateRejected::
 		Set this variable to 'false' if you want to disable
-		'pushNonFFCurrent',
-		'pushNonFFMatching', 'pushAlreadyExists',
-		'pushFetchFirst', and 'pushNeedsForce'
+		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
+		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
 		simultaneously.
 	pushNonFFCurrent::
 		Advice shown when linkgit:git-push[1] fails due to a
@@ -41,6 +40,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..21b256e0a4 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,9 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.useForceIfIncludes::
+	If set to "true", it is equivalent to specifying
+	`--force-if-includes` as an option to linkgit:git-push[1]
+	in the command line. Adding `--no-force-if-includes` at the
+	time of push overrides this configuration setting.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..fb3a220386 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -13,7 +13,7 @@ SYNOPSIS
 	   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
 	   [-u | --set-upstream] [-o <string> | --push-option=<string>]
 	   [--[no-]signed|--signed=(true|false|if-asked)]
-	   [--force-with-lease[=<refname>[:<expect>]]]
+	   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]
 	   [--no-verify] [<repository> [<refspec>...]]
 
 DESCRIPTION
@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying `--force-if-includes` an an ancillary option
+along with `--force-with-lease[=<refname>]` (i.e., without saying what
+exact commit the ref on the remote side must be pointing at, or which
+refs on the remote side are being protected) at the time of "push" will
+verify if updates from the remote-tracking refs that may have been
+implicitly updated in the background are integrated locally before
+allowing a forced update.
 
 -f::
 --force::
@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option enables a check that verifies if the tip of the
+remote-tracking ref is reachable from one of the "reflog" entries of
+the local branch based in it for a rewrite. The check ensures that any
+updates from the remote have been incorporated locally by rejecting the
+forced update if that is not the case.
++
+If the option is passed without specifying `--force-with-lease`, or
+specified along with `--force-with-lease=<refname>:<expect>`, it is
+a "no-op".
++
+Specifying `--no-force-if-includes` disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..a36b371aeb 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -13,6 +13,46 @@ setup_srcdst_basic () {
 	)
 }
 
+# For tests with "--force-if-includes".
+setup_src_dup_dst () {
+	rm -fr src dup dst &&
+	git init --bare dst &&
+	git clone --no-local dst src &&
+	git clone --no-local dst dup
+	(
+		cd src &&
+		test_commit A &&
+		test_commit B &&
+		test_commit C &&
+		git push
+	) &&
+	(
+		cd dup &&
+		git fetch &&
+		git merge origin/master &&
+		git switch -c branch master~2 &&
+		test_commit D &&
+		test_commit E &&
+		git push --all
+	) &&
+	(
+		cd src &&
+		git switch master &&
+		git fetch --all &&
+		git branch branch --track origin/branch &&
+		git rebase origin/master
+	) &&
+	(
+		cd dup &&
+		git switch master &&
+		test_commit F &&
+		test_commit G &&
+		git switch branch &&
+		test_commit H &&
+		git push origin --all
+	)
+}
+
 test_expect_success setup '
 	# create template repository
 	test_commit A &&
@@ -256,4 +296,104 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates to remote can be mitigated with "--force-if-includes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		test_must_fail git push --force-with-lease --force-if-includes --all
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success 'background updates to remote can be mitigated with "push.useForceIfIncludes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		git config --local push.useForceIfIncludes true &&
+		test_must_fail git push --force-with-lease=master origin master
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success '"--force-if-includes" should be disabled for --force-with-lease="<refname>:<expect>"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/master >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		remote_head="$(git rev-parse refs/remotes/origin/master)" &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes --force-with-lease="master:$remote_head" 2>err &&
+		grep "stale info" err
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/master >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase")' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase", local rebase)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git rebase --onto HEAD~4 HEAD~1 &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
+test_expect_success '"--force-if-includes" should allow deletes ' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		git pull --rebase origin branch &&
+		git push --force-if-includes --force-with-lease=branch origin :branch
+	)
+'
+
 test_done
-- 
2.28.0

Re: [PATCH v8 0/3] push: add "--[no-]force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Add a new option: "--force-if-includes" to "git-push" where forced
> updates are allowed only if the tip of the remote-tracking ref has
> been integrated locally, by verifying if the tip of the remote-tracking
> ref -- on which a local branch has based on -- is reachable from at
> least one of the "reflog" entries of the branch about to be updated
> by force on the remote.

https://travis-ci.org/github/git/git/jobs/730962458 is a build of
'seen' with this topic, and the same 'seen' without this topic is
https://travis-ci.org/github/git/git/builds/730857608 that passes
all the jobs.  It is curious why one particular job fails while
others in the same build is OK.  I've seen this pattern for this
topic in the past few days.

The failure in t5533-push-cas.sh is sort-of understandable as the
topic directly touches the area of the code the failing test
exercises, but the failure in t3701 is totally unexpected.

You can go down to the bottom of the page and click on the
ci/print-test-failures.sh line to see which test piece fails.

Anything rings a bell?

Thanks.

Re: [PATCH v8 0/3] push: add "--[no-]force-if-includes"

[SZEDER Gábor]

On Mon, Sep 28, 2020 at 10:31:34AM -0700, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> > Add a new option: "--force-if-includes" to "git-push" where forced
> > updates are allowed only if the tip of the remote-tracking ref has
> > been integrated locally, by verifying if the tip of the remote-tracking
> > ref -- on which a local branch has based on -- is reachable from at
> > least one of the "reflog" entries of the branch about to be updated
> > by force on the remote.
> 
> https://travis-ci.org/github/git/git/jobs/730962458 is a build of
> 'seen' with this topic, and the same 'seen' without this topic is
> https://travis-ci.org/github/git/git/builds/730857608 that passes
> all the jobs.  It is curious why one particular job fails while
> others in the same build is OK.

That build runs the test suite with a bunch of GIT_TEST_* knobs
enabled, and the last two tests added in this series fail when run as:

  GIT_TEST_COMMIT_GRAPH=1 ./t5533-push-cas.sh

> The failure in t5533-push-cas.sh is sort-of understandable as the
> topic directly touches the area of the code the failing test
> exercises, but the failure in t3701 is totally unexpected.

That's not a failure, but the fix of a known breakage: we expect
failure from the scripted 'git add -i' in two tests, but the builtin
'git add -i' fixes those issues, and those two tests succeed with
GIT_TEST_ADD_I_USE_BUILTIN=1.

Re: [PATCH v8 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Hello,

On 09/28/2020 19:46, SZEDER Gábor wrote:
> On Mon, Sep 28, 2020 at 10:31:34AM -0700, Junio C Hamano wrote:
> > Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> > 
> > > Add a new option: "--force-if-includes" to "git-push" where forced
> > > updates are allowed only if the tip of the remote-tracking ref has
> > > been integrated locally, by verifying if the tip of the remote-tracking
> > > ref -- on which a local branch has based on -- is reachable from at
> > > least one of the "reflog" entries of the branch about to be updated
> > > by force on the remote.
> > 
> > https://travis-ci.org/github/git/git/jobs/730962458 is a build of
> > 'seen' with this topic, and the same 'seen' without this topic is
> > https://travis-ci.org/github/git/git/builds/730857608 that passes
> > all the jobs.  It is curious why one particular job fails while
> > others in the same build is OK.
> 
> That build runs the test suite with a bunch of GIT_TEST_* knobs
> enabled, and the last two tests added in this series fail when run as:
> 
>   GIT_TEST_COMMIT_GRAPH=1 ./t5533-push-cas.sh
 
Thanks for the heads-up. It turns out that "in_merge_bases_many()"
returns different results depending on "GIT_TEST_COMMIT_GRAPH".
Initially I thought that it might be related to batching the entries,
but that is not the case.

One of the tests that is failing is:
  cd src &&
  git switch branch &&
  test_commit I &&
  git switch master &&
  test_commit J &&
  git pull --rebase origin master &&
  git push --force-if-includes --force-with-lease="master"

Here, we are testing to check if forced updates are allowed after
the remote changes have been incorporated locally, which is true
in this case and should pass.

"in_merge_bases_many()" used in the check as follows:

  for (chunk = list.items; chunk < list.items + list.nr; chunk += size) {
	  size = list.items + list.nr - chunk;
	  if (MERGE_BASES_BATCH_SIZE < size)
	        size = MERGE_BASES_BATCH_SIZE;

	  if ((ret = in_merge_bases_many(commit, size, chunk)))
	        break;
  }

In "repo_in_merge_bases_many()" [1], the following condition evaluates
to true when "GIT_TEST_COMMIT_GRAPH" is 1.

	generation = commit_graph_generation(commit);
	if (generation > min_generation)
		return ret;

Unfortunately, I am unfamiliar with the code, and not sure why this
happens; I remember Junio mention [2] something about generation
numbers could it be related to that?

A possible "workaround" is to use "in_merge_bases()" for each of the
commits we collect in the list, and the tests seem to pass with
"GIT_TEST_COMMIT_GRAPH" being set; but I wonder if that's the right
way to fix this.

[1] https://git.kernel.org/pub/scm/git/git.git/tree/commit-reach.c#n319
[2] https://public-inbox.org/git/xmqqft7djzz0.fsf@gitster.c.googlers.com/

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH v8 0/3] push: add "--[no-]force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

>> That build runs the test suite with a bunch of GIT_TEST_* knobs
>> enabled, and the last two tests added in this series fail when run as:
>> 
>>   GIT_TEST_COMMIT_GRAPH=1 ./t5533-push-cas.sh
>  
> Thanks for the heads-up. It turns out that "in_merge_bases_many()"
> returns different results depending on "GIT_TEST_COMMIT_GRAPH".
> Initially I thought that it might be related to batching the entries,
> but that is not the case.
>
> One of the tests that is failing is:
>   cd src &&
>   git switch branch &&
>   test_commit I &&
>   git switch master &&
>   test_commit J &&
>   git pull --rebase origin master &&
>   git push --force-if-includes --force-with-lease="master"
>
> Here, we are testing to check if forced updates are allowed after
> the remote changes have been incorporated locally, which is true
> in this case and should pass.
>
> "in_merge_bases_many()" used in the check as follows:
> ...
> Unfortunately, I am unfamiliar with the code, and not sure why this
> happens; I remember Junio mention [2] something about generation
> numbers could it be related to that?

Now it's time to summon the commit-graph folks.  I think we should
assume that it a bug in the code with commit-graph if it produces a
result that is different from the code without, until we prove
otherwise (e.g. in a history with clock-skew, traditional traversal
of A..B could give a wrong result where commit-graph may produce a
correct result.  I however think the topology-based merge-base
computation does not suffer from the same issue).

Thanks.  

Re: [PATCH v8 0/3] push: add "--[no-]force-if-includes"

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> That build runs the test suite with a bunch of GIT_TEST_* knobs
> enabled,...

Ahh, OK, I knew there was one "funny" job but did not realize
linux-gcc was that one.  It is understandable why that particular
job behaves differently from others ;-)

Thanks.

Re: [PATCH v8 3/3] t, doc: update tests, reference for "--force-if-includes"

[Philip Oakley]

Hi, spelling nit.

On 27/09/2020 15:17, Srinidhi Kaushik wrote:
> Update test cases for the new option, and document its usage
> and update related references.
>
> Update test cases for the new option, and document its usage
> and update related references.
>
>  - t/t5533-push-cas.sh:
>    Update test cases for "compare-and-swap" when used along with
>    "--force-if-includes" helps mitigate overwrites when remote
>    refs are updated in the background; allows forced updates when
>    changes from remote are integrated locally.
>
>  - Documentation:
>    Add reference for the new option, configuration setting
>    ("push.useForceIfIncludes") and advise messages.
>
> Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
> ---
>  Documentation/config/advice.txt |   9 +-
>  Documentation/config/push.txt   |   6 ++
>  Documentation/git-push.txt      |  26 +++++-
>  t/t5533-push-cas.sh             | 140 ++++++++++++++++++++++++++++++++
>  4 files changed, 177 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
> index bdd37c3eaa..acbd0c09aa 100644
> --- a/Documentation/config/advice.txt
> +++ b/Documentation/config/advice.txt
> @@ -10,9 +10,8 @@ advice.*::
>  		that the check is disabled.
>  	pushUpdateRejected::
>  		Set this variable to 'false' if you want to disable
> -		'pushNonFFCurrent',
> -		'pushNonFFMatching', 'pushAlreadyExists',
> -		'pushFetchFirst', and 'pushNeedsForce'
> +		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
> +		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
>  		simultaneously.
>  	pushNonFFCurrent::
>  		Advice shown when linkgit:git-push[1] fails due to a
> @@ -41,6 +40,10 @@ advice.*::
>  		we can still suggest that the user push to either
>  		refs/heads/* or refs/tags/* based on the type of the
>  		source object.
> +	pushRefNeedsUpdate::
> +		Shown when linkgit:git-push[1] rejects a forced update of
> +		a branch when its remote-tracking ref has updates that we
> +		do not have locally.
>  	statusAheadBehind::
>  		Shown when linkgit:git-status[1] computes the ahead/behind
>  		counts for a local ref compared to its remote tracking ref,
> diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
> index f5e5b38c68..21b256e0a4 100644
> --- a/Documentation/config/push.txt
> +++ b/Documentation/config/push.txt
> @@ -114,3 +114,9 @@ push.recurseSubmodules::
>  	specifying '--recurse-submodules=check|on-demand|no'.
>  	If not set, 'no' is used by default, unless 'submodule.recurse' is
>  	set (in which case a 'true' value means 'on-demand').
> +
> +push.useForceIfIncludes::
> +	If set to "true", it is equivalent to specifying
> +	`--force-if-includes` as an option to linkgit:git-push[1]
> +	in the command line. Adding `--no-force-if-includes` at the
> +	time of push overrides this configuration setting.
> diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
> index 3b8053447e..fb3a220386 100644
> --- a/Documentation/git-push.txt
> +++ b/Documentation/git-push.txt
> @@ -13,7 +13,7 @@ SYNOPSIS
>  	   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
>  	   [-u | --set-upstream] [-o <string> | --push-option=<string>]
>  	   [--[no-]signed|--signed=(true|false|if-asked)]
> -	   [--force-with-lease[=<refname>[:<expect>]]]
> +	   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]
>  	   [--no-verify] [<repository> [<refspec>...]]
>  
>  DESCRIPTION
> @@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally
>  force push changes to `master` if the remote version is still at
>  `base`, regardless of what your local `remotes/origin/master` has been
>  updated to in the background.
> ++
> +Alternatively, specifying `--force-if-includes` an an ancillary option

s/ an an / as an /

> +along with `--force-with-lease[=<refname>]` (i.e., without saying what
> +exact commit the ref on the remote side must be pointing at, or which
> +refs on the remote side are being protected) at the time of "push" will
> +verify if updates from the remote-tracking refs that may have been
> +implicitly updated in the background are integrated locally before
> +allowing a forced update.
>  
>  -f::
>  --force::
> @@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
>  origin +master` to force a push to the `master` branch). See the
>  `<refspec>...` section above for details.
>  
> +--[no-]force-if-includes::
> +	Force an update only if the tip of the remote-tracking ref
> +	has been integrated locally.
> ++
> +This option enables a check that verifies if the tip of the
> +remote-tracking ref is reachable from one of the "reflog" entries of
> +the local branch based in it for a rewrite. The check ensures that any
> +updates from the remote have been incorporated locally by rejecting the
> +forced update if that is not the case.
> ++
> +If the option is passed without specifying `--force-with-lease`, or
> +specified along with `--force-with-lease=<refname>:<expect>`, it is
> +a "no-op".
> ++
> +Specifying `--no-force-if-includes` disables this behavior.
> +
>  --repo=<repository>::
>  	This option is equivalent to the <repository> argument. If both
>  	are specified, the command-line argument takes precedence.
> diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
> index 0b0eb1d025..a36b371aeb 100755
> --- a/t/t5533-push-cas.sh
> +++ b/t/t5533-push-cas.sh
> @@ -13,6 +13,46 @@ setup_srcdst_basic () {
>  	)
>  }
>  
> +# For tests with "--force-if-includes".
> +setup_src_dup_dst () {
> +	rm -fr src dup dst &&
> +	git init --bare dst &&
> +	git clone --no-local dst src &&
> +	git clone --no-local dst dup
> +	(
> +		cd src &&
> +		test_commit A &&
> +		test_commit B &&
> +		test_commit C &&
> +		git push
> +	) &&
> +	(
> +		cd dup &&
> +		git fetch &&
> +		git merge origin/master &&
> +		git switch -c branch master~2 &&
> +		test_commit D &&
> +		test_commit E &&
> +		git push --all
> +	) &&
> +	(
> +		cd src &&
> +		git switch master &&
> +		git fetch --all &&
> +		git branch branch --track origin/branch &&
> +		git rebase origin/master
> +	) &&
> +	(
> +		cd dup &&
> +		git switch master &&
> +		test_commit F &&
> +		test_commit G &&
> +		git switch branch &&
> +		test_commit H &&
> +		git push origin --all
> +	)
> +}
> +
>  test_expect_success setup '
>  	# create template repository
>  	test_commit A &&
> @@ -256,4 +296,104 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
>  	)
>  '
>  
> +test_expect_success 'background updates to remote can be mitigated with "--force-if-includes"' '
> +	setup_src_dup_dst &&
> +	test_when_finished "rm -fr dst src dup" &&
> +	git ls-remote dst refs/heads/master >expect.master &&
> +	git ls-remote dst refs/heads/master >expect.branch &&
> +	(
> +		cd src &&
> +		git switch branch &&
> +		test_commit I &&
> +		git switch master &&
> +		test_commit J &&
> +		git fetch --all &&
> +		test_must_fail git push --force-with-lease --force-if-includes --all
> +	) &&
> +	git ls-remote dst refs/heads/master >actual.master &&
> +	git ls-remote dst refs/heads/master >actual.branch &&
> +	test_cmp expect.master actual.master &&
> +	test_cmp expect.branch actual.branch
> +'
> +
> +test_expect_success 'background updates to remote can be mitigated with "push.useForceIfIncludes"' '
> +	setup_src_dup_dst &&
> +	test_when_finished "rm -fr dst src dup" &&
> +	git ls-remote dst refs/heads/master >expect.master &&
> +	(
> +		cd src &&
> +		git switch branch &&
> +		test_commit I &&
> +		git switch master &&
> +		test_commit J &&
> +		git fetch --all &&
> +		git config --local push.useForceIfIncludes true &&
> +		test_must_fail git push --force-with-lease=master origin master
> +	) &&
> +	git ls-remote dst refs/heads/master >actual.master &&
> +	test_cmp expect.master actual.master
> +'
> +
> +test_expect_success '"--force-if-includes" should be disabled for --force-with-lease="<refname>:<expect>"' '
> +	setup_src_dup_dst &&
> +	test_when_finished "rm -fr dst src dup" &&
> +	git ls-remote dst refs/heads/master >expect.master &&
> +	git ls-remote dst refs/heads/master >expect.branch &&
> +	(
> +		cd src &&
> +		git switch branch &&
> +		test_commit I &&
> +		git switch master &&
> +		test_commit J &&
> +		remote_head="$(git rev-parse refs/remotes/origin/master)" &&
> +		git fetch --all &&
> +		test_must_fail git push --force-if-includes --force-with-lease="master:$remote_head" 2>err &&
> +		grep "stale info" err
> +	) &&
> +	git ls-remote dst refs/heads/master >actual.master &&
> +	git ls-remote dst refs/heads/master >actual.branch &&
> +	test_cmp expect.master actual.master &&
> +	test_cmp expect.branch actual.branch
> +'
> +
> +test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase")' '
> +	setup_src_dup_dst &&
> +	test_when_finished "rm -fr dst src dup" &&
> +	(
> +		cd src &&
> +		git switch branch &&
> +		test_commit I &&
> +		git switch master &&
> +		test_commit J &&
> +		git pull --rebase origin master &&
> +		git push --force-if-includes --force-with-lease="master"
> +	)
> +'
> +
> +test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase", local rebase)' '
> +	setup_src_dup_dst &&
> +	test_when_finished "rm -fr dst src dup" &&
> +	(
> +		cd src &&
> +		git switch branch &&
> +		test_commit I &&
> +		git switch master &&
> +		test_commit J &&
> +		git pull --rebase origin master &&
> +		git rebase --onto HEAD~4 HEAD~1 &&
> +		git push --force-if-includes --force-with-lease="master"
> +	)
> +'
> +
> +test_expect_success '"--force-if-includes" should allow deletes ' '
> +	setup_src_dup_dst &&
> +	test_when_finished "rm -fr dst src dup" &&
> +	(
> +		cd src &&
> +		git switch branch &&
> +		git pull --rebase origin branch &&
> +		git push --force-if-includes --force-with-lease=branch origin :branch
> +	)
> +'
> +
>  test_done

Re: [PATCH v8 3/3] t, doc: update tests, reference for "--force-if-includes"

[Srinidhi Kaushik]

Hi Philip,

On 09/30/2020 13:54, Philip Oakley wrote:
> Hi, spelling nit.
> 
> > On 27/09/2020 15:17, Srinidhi Kaushik wrote:
> > [...]
> > +Alternatively, specifying `--force-if-includes` an an ancillary option
> 
> s/ an an / as an /

Thanks! I will fix this in the next set.
-- 
Srinidhi Kaushik

[PATCH v9 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: "--force-if-includes" to "git-push" where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref -- on which a local branch has based on -- is reachable from at
least one of the "reflog" entries of the branch about to be updated
by force on the remote.

This option can be used with "--force-with-lease" with setups where
the remote-tracking refs of the repository are implicitly updated in
the background to help prevent unintended remote overwrites.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with "--force-with-lease[=<refname>[:<expect>]]" without specifying
the "<expect>" value, can cause the update that happened in-between
the checkout and forced push to be lost.

Changes since v8:
  - Disable "commit-graph" when "in_merge_bases_many()" is called
    for this check, because it returns different results depending
    on whether "commit-graph" is enabled [1].

  - Rename the commit list name, remove redundant comments and fix
    some typos.

Srinidhi Kaushik (3):
  push: add reflog check for "--force-if-includes"
  push: parse and set flag for "--force-if-includes"
  t, doc: update tests, reference for "--force-if-includes"

 Documentation/config/advice.txt |   9 +-
 Documentation/config/push.txt   |   6 +
 Documentation/git-push.txt      |  26 ++++-
 advice.c                        |   3 +
 advice.h                        |   2 +
 builtin/push.c                  |  27 +++++
 builtin/send-pack.c             |  12 ++
 remote-curl.c                   |  14 ++-
 remote.c                        | 198 ++++++++++++++++++++++++++++++--
 remote.h                        |  12 +-
 send-pack.c                     |   1 +
 t/t5533-push-cas.sh             | 137 ++++++++++++++++++++++
 transport-helper.c              |  10 ++
 transport.c                     |   8 ++
 transport.h                     |  15 ++-
 15 files changed, 462 insertions(+), 18 deletions(-)

base-commit: 306ee63a703ad67c54ba1209dc11dd9ea500dc1f
[1] https://public-inbox.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com

--
2.28.0

[PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Add a check to verify if the remote-tracking ref of the local branch
is reachable from one of its "reflog" entries.

The check iterates through the local ref's reflog to see if there
is an entry for the remote-tracking ref and collecting any commits
that are seen, into a list; the iteration stops if an entry in the
reflog matches the remote ref or if the entry timestamp is older
the latest entry of the remote ref's "reflog". If there wasn't an
entry found for the remote ref, "in_merge_bases_many()" is called
to check if it is reachable from the list of collected commits.

When a local branch that is based on a remote ref, has been rewound
and is to be force pushed on the remote, "--force-if-includes" runs
a check that ensures any updates to the remote-tracking ref that may
have happened (by push from another repository) in-between the time
of the last update to the local branch (via "git-pull", for instance)
and right before the time of push, have been integrated locally
before allowing a forced update.

If the new option is passed without specifying "--force-with-lease",
or specified along with "--force-with-lease=<refname>:<expect>" it
is a "no-op".

Calls to "in_merge_bases_many()" return different results depending
on whether the "commit-graph" feature is enabled or not -- it is
temporarily disabled when the check runs [1].

[1] https://lore.kernel.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 builtin/send-pack.c |   5 ++
 remote.c            | 198 ++++++++++++++++++++++++++++++++++++++++++--
 remote.h            |  12 ++-
 send-pack.c         |   1 +
 transport-helper.c  |   5 ++
 transport.c         |   6 ++
 6 files changed, 219 insertions(+), 8 deletions(-)

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 7af148d733..516cba7336 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote ref updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/remote.c b/remote.c
index eafc14cbe7..98a578f5dc 100644
--- a/remote.c
+++ b/remote.c
@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 		 * with the remote-tracking branch to find the value
 		 * to expect, but we did not have such a tracking
 		 * branch.
+		 *
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
 		 */
 		if (ref->expect_old_sha1) {
 			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
 				reject_reason = REF_STATUS_REJECT_STALE;
+			else if (ref->check_reachable && ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
 			else
-				/* If the ref isn't stale then force the update. */
+				/*
+				 * If the ref isn't stale, and is reachable
+				 * from from one of the reflog entries of
+				 * the local branch, force the update.
+				 */
 				force_ref_update = 1;
 		}
 
@@ -2251,12 +2262,13 @@ int is_empty_cas(const struct push_cas_option *cas)
 
 /*
  * Look at remote.fetch refspec and see if we have a remote
- * tracking branch for the refname there.  Fill its current
- * value in sha1[].
+ * tracking branch for the refname there. Fill the name of
+ * the remote-tracking branch in *dst_refname, and the name
+ * of the commit object at its tip in oid[].
  * If we cannot do so, return negative to signal an error.
  */
 static int remote_tracking(struct remote *remote, const char *refname,
-			   struct object_id *oid)
+			   struct object_id *oid, char **dst_refname)
 {
 	char *dst;
 
@@ -2265,9 +2277,164 @@ static int remote_tracking(struct remote *remote, const char *refname,
 		return -1; /* no tracking ref for refname at remote */
 	if (read_ref(dst, oid))
 		return -1; /* we know what the tracking ref is but we cannot read it */
+
+	*dst_refname = dst;
 	return 0;
 }
 
+/*
+ * The struct "reflog_commit_list" and related helper functions
+ * for list manipulation are used for collecting commits into a
+ * list during reflog traversals in "check_and_collect_until()".
+ */
+struct reflog_commit_list {
+	struct commit **item;
+	size_t nr, alloc;
+};
+
+/* Append a commit to the list. */
+static void append_commit(struct reflog_commit_list *list,
+			  struct commit *commit)
+{
+	ALLOC_GROW(list->item, list->nr + 1, list->alloc);
+	list->item[list->nr++] = commit;
+}
+
+/* Free and reset the list. */
+static void free_reflog_commit_list(struct reflog_commit_list *list)
+{
+	FREE_AND_NULL(list->item);
+	list->nr = list->alloc = 0;
+}
+
+struct check_and_collect_until_cb_data {
+	struct commit *remote_commit;
+	struct reflog_commit_list *local_commits;
+	timestamp_t remote_reflog_timestamp;
+};
+
+/* Get the timestamp of the latest entry. */
+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,
+		       const char *ident, timestamp_t timestamp,
+		       int tz, const char *message, void *cb_data)
+{
+	timestamp_t *ts = cb_data;
+	*ts = timestamp;
+	return 1;
+}
+
+static int check_and_collect_until(struct object_id *o_oid,
+				   struct object_id *n_oid,
+				   const char *ident, timestamp_t timestamp,
+				   int tz, const char *message, void *cb_data)
+{
+	struct commit *commit;
+	struct check_and_collect_until_cb_data *cb = cb_data;
+
+	/* An entry was found. */
+	if (oideq(n_oid, &cb->remote_commit->object.oid))
+		return 1;
+
+	if ((commit = lookup_commit_reference(the_repository, n_oid)))
+		append_commit(cb->local_commits, commit);
+
+	/*
+	 * If the reflog entry timestamp is older than the remote ref's
+	 * latest reflog entry, there is no need to check or collect
+	 * entries older than this one.
+	 */
+	if (timestamp < cb->remote_reflog_timestamp)
+		return -1;
+
+	return 0;
+}
+
+#define MERGE_BASES_BATCH_SIZE 8
+
+/*
+ * Iterate through the reflog of the local ref to check if there is an entry
+ * for the given remote-tracking ref; runs until the timestamp of an entry is
+ * older than latest timestamp of remote-tracking ref's reflog. Any commits
+ * are that seen along the way are collected into a list to check if the
+ * remote-tracking ref is reachable from any of them.
+ */
+static int is_reachable_in_reflog(const char *local, const struct ref *remote)
+{
+	timestamp_t date;
+	struct commit *commit;
+	struct commit **chunk;
+	struct check_and_collect_until_cb_data cb;
+	struct reflog_commit_list list = { NULL, 0, 0 };
+	size_t size = 0;
+	int ret = 0;
+
+	commit = lookup_commit_reference(the_repository, &remote->old_oid);
+	if (!commit)
+		goto cleanup_return;
+
+	/*
+	 * Get the timestamp from the latest entry
+	 * of the remote-tracking ref's reflog.
+	 */
+	for_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);
+
+	cb.remote_commit = commit;
+	cb.local_commits = &list;
+	cb.remote_reflog_timestamp = date;
+	ret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);
+
+	/* We found an entry in the reflog. */
+	if (ret > 0)
+		goto cleanup_return;
+
+	/*
+	 * Check if the remote commit is reachable from any
+	 * of the commits in the collected list, in batches.
+	 */
+	for (chunk = list.item; chunk < list.item + list.nr; chunk += size) {
+		size = list.item + list.nr - chunk;
+		if (MERGE_BASES_BATCH_SIZE < size)
+			size = MERGE_BASES_BATCH_SIZE;
+
+		if ((ret = in_merge_bases_many(commit, size, chunk)))
+			break;
+	}
+
+cleanup_return:
+	free_reflog_commit_list(&list);
+	return ret;
+}
+
+/* Toggle the "commit-graph" feature; return the previously set state. */
+static int toggle_commit_graph(struct repository *repo, int disable) {
+	int prev = repo->commit_graph_disabled;
+	repo->commit_graph_disabled = disable;
+	return prev;
+}
+
+/*
+ * Check for reachability of a remote-tracking
+ * ref in the reflog entries of its local ref.
+ */
+static void check_if_includes_upstream(struct ref *remote)
+{
+	int prev;
+	struct ref *local = get_local_ref(remote->name);
+	if (!local)
+		return;
+
+	/*
+	 * TODO: Remove "toggle_commit_graph()" calls around the check.
+	 * Depending on whether "commit-graph" enabled or not,
+	 * "in_merge_bases_many()" returns different results;
+	 * disable it temporarily when the check runs.
+	 */
+	prev = toggle_commit_graph(the_repository, 1);
+	if (is_reachable_in_reflog(local->name, remote) <= 0)
+		remote->unreachable = 1;
+	toggle_commit_graph(the_repository, prev);
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
@@ -2282,8 +2449,12 @@ static void apply_cas(struct push_cas_option *cas,
 		ref->expect_old_sha1 = 1;
 		if (!entry->use_tracking)
 			oidcpy(&ref->old_oid_expect, &entry->expect);
-		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+		else if (remote_tracking(remote, ref->name,
+					 &ref->old_oid_expect,
+					 &ref->tracking_ref))
 			oidclr(&ref->old_oid_expect);
+		else
+			ref->check_reachable = cas->use_force_if_includes;
 		return;
 	}
 
@@ -2292,8 +2463,12 @@ static void apply_cas(struct push_cas_option *cas,
 		return;
 
 	ref->expect_old_sha1 = 1;
-	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+	if (remote_tracking(remote, ref->name,
+			    &ref->old_oid_expect,
+			    &ref->tracking_ref))
 		oidclr(&ref->old_oid_expect);
+	else
+		ref->check_reachable = cas->use_force_if_includes;
 }
 
 void apply_push_cas(struct push_cas_option *cas,
@@ -2301,6 +2476,15 @@ void apply_push_cas(struct push_cas_option *cas,
 		    struct ref *remote_refs)
 {
 	struct ref *ref;
-	for (ref = remote_refs; ref; ref = ref->next)
+	for (ref = remote_refs; ref; ref = ref->next) {
 		apply_cas(cas, remote, ref);
+
+		/*
+		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
+		 * mode, and if "--force-if-includes" was specified, run
+		 * the check.
+		 */
+		if (ref->check_reachable)
+			check_if_includes_upstream(ref);
+	}
 }
diff --git a/remote.h b/remote.h
index eb62a47044..2d5391d281 100644
--- a/remote.h
+++ b/remote.h
@@ -107,12 +107,20 @@ struct ref {
 	struct object_id new_oid;
 	struct object_id old_oid_expect; /* used by expect-old */
 	char *symref;
+	char *tracking_ref;
 	unsigned int
 		force:1,
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		/* Need to check if local reflog reaches the remote tip. */
+		check_reachable:1,
+		/*
+		 * Store the result of the check enabled by "check_reachable";
+		 * implies the local reflog does not reach the remote tip.
+		 */
+		unreachable:1;
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -142,6 +150,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);
 
 struct push_cas_option {
 	unsigned use_tracking_for_rest:1;
+	unsigned use_force_if_includes:1;
 	struct push_cas {
 		struct object_id expect;
 		unsigned use_tracking:1;
diff --git a/send-pack.c b/send-pack.c
index c9698070fc..eb4a44270b 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/transport-helper.c b/transport-helper.c
index b573b6c730..6157de30c7 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote ref updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
diff --git a/transport.c b/transport.c
index ffe2115845..65fcd22b20 100644
--- a/transport.c
+++ b/transport.c
@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,
 				 "stale info",
 				 report, porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote ref updated since checkout",
+				 report, porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
-- 
2.28.0

[PATCH v9 2/3] push: parse and set flag for "--force-if-includes"

[Srinidhi Kaushik]

The previous commit added the necessary machinery to implement the
"--force-if-includes" protection, when "--force-with-lease" is used
without giving exact object the remote still ought to have. Surface
the feature by adding a command line option and a configuration
variable to enable it.

 - Add a flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES" to indicate that the
   new option was passed from the command line of via configuration
   settings; update command line and configuration parsers to set the
   new flag accordingly.

 - Introduce a new configuration option "push.useForceIfIncludes", which
   is equivalent to setting "--force-if-includes" in the command line.

 - Update "remote-curl" to recognize and pass this option to "send-pack"
   when enabled.

 - Update "advise" to catch the reject reason "REJECT_REF_NEEDS_UPDATE",
   set when the ref status is "REF_STATUS_REJECT_REMOTE_UPDATED" and
   (optionally) print a help message when the push fails.

 - The new option is a "no-op" in the following scenarios:
    * When used without "--force-with-lease".
    * When used with "--force-with-lease", and if the expected commit
      on the remote side is specified as an argument.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 advice.c            |  3 +++
 advice.h            |  2 ++
 builtin/push.c      | 27 +++++++++++++++++++++++++++
 builtin/send-pack.c |  7 +++++++
 remote-curl.c       | 14 +++++++++++++-
 transport-helper.c  |  5 +++++
 transport.c         |  2 ++
 transport.h         | 15 ++++++++++-----
 8 files changed, 69 insertions(+), 6 deletions(-)

diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index 0eeb2c8dd5..908b557edb 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git pull ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
 
 	return 1;
@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		if (!v)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
+	} else if (!strcmp(k, "push.useforceifincludes")) {
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		return 0;
 	}
 
 	return git_default_config(k, v, NULL);
@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 	if ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))
 		die(_("--all and --mirror are incompatible"));
 
+	if (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))
+		cas.use_force_if_includes = 1;
+
 	for_each_string_list_item(item, push_options)
 		if (strchr(item->string, '\n'))
 			die(_("push options must not have new line characters"));
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 516cba7336..a284ada051 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int progress = -1;
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
+	int force_if_includes = 0;
 	struct packet_reader reader;
 
 	struct option options[] = {
@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 		  N_("require old value of ref to be at this value"),
 		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
+		OPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,
+			 N_("require remote updates to be integrated locally")),
 		OPT_END()
 	};
 
@@ -304,6 +307,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	if (!is_empty_cas(&cas))
 		apply_push_cas(&cas, remote, remote_refs);
 
+	if (!is_empty_cas(&cas) && force_if_includes)
+		cas.use_force_if_includes = 1;
+
+
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
 
diff --git a/remote-curl.c b/remote-curl.c
index 32cc4a0c55..0290b04891 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -44,7 +44,8 @@ struct options {
 		from_promisor : 1,
 
 		atomic : 1,
-		object_format : 1;
+		object_format : 1,
+		force_if_includes : 1;
 	const struct git_hash_algo *hash_algo;
 };
 static struct options options;
@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)
 		string_list_append(&cas_options, val.buf);
 		strbuf_release(&val);
 		return 0;
+	} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {
+		if (!strcmp(value, "true"))
+			options.force_if_includes = 1;
+		else if (!strcmp(value, "false"))
+			options.force_if_includes = 0;
+		else
+			return -1;
+		return 0;
 	} else if (!strcmp(name, "cloning")) {
 		if (!strcmp(value, "true"))
 			options.cloning = 1;
@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)
 		strvec_push(&args, cas_option->string);
 	strvec_push(&args, url.buf);
 
+	if (options.force_if_includes)
+		strvec_push(&args, "--force-if-includes");
+
 	strvec_push(&args, "--stdin");
 	for (i = 0; i < nr_spec; i++)
 		packet_buf_write(&preamble, "%s\n", specs[i]);
diff --git a/transport-helper.c b/transport-helper.c
index 6157de30c7..5f6e0b3bd8 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,
 		if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
 			die(_("helper %s does not support --atomic"), name);
 
+	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
+		if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
+			die(_("helper %s does not support --%s"),
+			    name, TRANS_OPT_FORCE_IF_INCLUDES);
+
 	if (flags & TRANSPORT_PUSH_OPTIONS) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, transport->push_options)
diff --git a/transport.c b/transport.c
index 65fcd22b20..47da955e4f 100644
--- a/transport.c
+++ b/transport.c
@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
diff --git a/transport.h b/transport.h
index ca409ea1e4..24558c027d 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);
 /* Request atomic (all-or-nothing) updates when pushing */
 #define TRANS_OPT_ATOMIC "atomic"
 
+/* Require remote changes to be integrated locally. */
+#define TRANS_OPT_FORCE_IF_INCLUDES "force-if-includes"
+
 /**
  * Returns 0 if the option was used, non-zero otherwise. Prints a
  * message to stderr if the option is not used.
@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v9 3/3] t, doc: update tests, reference for "--force-if-includes"

[Srinidhi Kaushik]

Update test cases for the new option, and document its usage
and update related references.

Update test cases for the new option, and document its usage
and update related references.

 - t/t5533-push-cas.sh:
   Update test cases for "compare-and-swap" when used along with
   "--force-if-includes" helps mitigate overwrites when remote
   refs are updated in the background; allows forced updates when
   changes from remote are integrated locally.

 - Documentation:
   Add reference for the new option, configuration setting
   ("push.useForceIfIncludes") and advise messages.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt |   9 ++-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 +++++-
 t/t5533-push-cas.sh             | 137 ++++++++++++++++++++++++++++++++
 4 files changed, 174 insertions(+), 4 deletions(-)

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..acbd0c09aa 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -10,9 +10,8 @@ advice.*::
 		that the check is disabled.
 	pushUpdateRejected::
 		Set this variable to 'false' if you want to disable
-		'pushNonFFCurrent',
-		'pushNonFFMatching', 'pushAlreadyExists',
-		'pushFetchFirst', and 'pushNeedsForce'
+		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
+		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
 		simultaneously.
 	pushNonFFCurrent::
 		Advice shown when linkgit:git-push[1] fails due to a
@@ -41,6 +40,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..21b256e0a4 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,9 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.useForceIfIncludes::
+	If set to "true", it is equivalent to specifying
+	`--force-if-includes` as an option to linkgit:git-push[1]
+	in the command line. Adding `--no-force-if-includes` at the
+	time of push overrides this configuration setting.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..ab103c82cf 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -13,7 +13,7 @@ SYNOPSIS
 	   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
 	   [-u | --set-upstream] [-o <string> | --push-option=<string>]
 	   [--[no-]signed|--signed=(true|false|if-asked)]
-	   [--force-with-lease[=<refname>[:<expect>]]]
+	   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]
 	   [--no-verify] [<repository> [<refspec>...]]
 
 DESCRIPTION
@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying `--force-if-includes` as an ancillary option
+along with `--force-with-lease[=<refname>]` (i.e., without saying what
+exact commit the ref on the remote side must be pointing at, or which
+refs on the remote side are being protected) at the time of "push" will
+verify if updates from the remote-tracking refs that may have been
+implicitly updated in the background are integrated locally before
+allowing a forced update.
 
 -f::
 --force::
@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option enables a check that verifies if the tip of the
+remote-tracking ref is reachable from one of the "reflog" entries of
+the local branch based in it for a rewrite. The check ensures that any
+updates from the remote have been incorporated locally by rejecting the
+forced update if that is not the case.
++
+If the option is passed without specifying `--force-with-lease`, or
+specified along with `--force-with-lease=<refname>:<expect>`, it is
+a "no-op".
++
+Specifying `--no-force-if-includes` disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..7813e8470e 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -13,6 +13,46 @@ setup_srcdst_basic () {
 	)
 }
 
+# For tests with "--force-if-includes".
+setup_src_dup_dst () {
+	rm -fr src dup dst &&
+	git init --bare dst &&
+	git clone --no-local dst src &&
+	git clone --no-local dst dup
+	(
+		cd src &&
+		test_commit A &&
+		test_commit B &&
+		test_commit C &&
+		git push origin
+	) &&
+	(
+		cd dup &&
+		git fetch &&
+		git merge origin/master &&
+		git switch -c branch master~2 &&
+		test_commit D &&
+		test_commit E &&
+		git push origin --all
+	) &&
+	(
+		cd src &&
+		git switch master &&
+		git fetch --all &&
+		git branch branch --track origin/branch &&
+		git rebase origin/master
+	) &&
+	(
+		cd dup &&
+		git switch master &&
+		test_commit F &&
+		test_commit G &&
+		git switch branch &&
+		test_commit H &&
+		git push origin --all
+	)
+}
+
 test_expect_success setup '
 	# create template repository
 	test_commit A &&
@@ -256,4 +296,101 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates to remote can be mitigated with "--force-if-includes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/branch >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		test_must_fail git push --force-with-lease --force-if-includes --all
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/branch >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success 'background updates to remote can be mitigated with "push.useForceIfIncludes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		git config --local push.useForceIfIncludes true &&
+		test_must_fail git push --force-with-lease=master origin master
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success '"--force-if-includes" should be disabled for --force-with-lease="<refname>:<expect>"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		remote_head="$(git rev-parse refs/remotes/origin/master)" &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes --force-with-lease="master:$remote_head" 2>err &&
+		grep "stale info" err
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase")' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase", local rebase)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git rebase --onto HEAD~4 HEAD~1 &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
+test_expect_success '"--force-if-includes" should allow deletes' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		git pull --rebase origin branch &&
+		git push --force-if-includes --force-with-lease="branch" origin :branch
+	)
+'
+
 test_done
-- 
2.28.0

Re: [PATCH v9 0/3] push: add "--[no-]force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Changes since v8:
>   - Disable "commit-graph" when "in_merge_bases_many()" is called
>     for this check, because it returns different results depending
>     on whether "commit-graph" is enabled [1].

Is that a wise move, though?  If the "different results" is
expected, then it is a different story, but I would think it is a
bug in commit-graph codepath if it produces a result different from
what the callers expect, and disabling from the caller's end would
mean that we lose one opportunity to help commit-graph folks to go
and fix their bugs, no?

Other than that, I think the topic is in quite a good shape.  Thanks
for working on polishing it.

Re: [PATCH v9 0/3] push: add "--[no-]force-if-includes"

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
>
>> Changes since v8:
>>   - Disable "commit-graph" when "in_merge_bases_many()" is called
>>     for this check, because it returns different results depending
>>     on whether "commit-graph" is enabled [1].
>
> Is that a wise move, though?  If the "different results" is
> expected, then it is a different story, but I would think it is a
> bug in commit-graph codepath if it produces a result different from
> what the callers expect, and disabling from the caller's end would
> mean that we lose one opportunity to help commit-graph folks to go
> and fix their bugs, no?
>
> Other than that, I think the topic is in quite a good shape.  Thanks
> for working on polishing it.

In other words, how about doing it like so.

In an ideal world, folks who know more about commit-graph than we do
will find what's broken in in_merge_bases_many() when commit-graph
is in use, before I finish lecturing against hiding a breakage under
the rug.  Let's see if another call for help helps ;-)

 remote.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git i/remote.c w/remote.c
index 98a578f5dc..361b8f1c0e 100644
--- i/remote.c
+++ w/remote.c
@@ -2408,7 +2408,20 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)
 /* Toggle the "commit-graph" feature; return the previously set state. */
 static int toggle_commit_graph(struct repository *repo, int disable) {
 	int prev = repo->commit_graph_disabled;
-	repo->commit_graph_disabled = disable;
+	static int should_toggle = -1;
+
+	if (should_toggle < 0) {
+		/*
+		 * The in_merge_bases_many() seems to misbehave when
+		 * the commit-graph feature is in use.  Disable it for
+		 * normal users, but keep it enabled when specifically
+		 * testing the feature.
+		 */
+		should_toggle = !git_env_bool("GIT_TEST_COMMIT_GRAPH", 0);
+	}
+
+	if (should_toggle)
+		repo->commit_graph_disabled = disable;
 	return prev;
 }
 

Re: [PATCH v9 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Hello,

On 10/01/2020 10:12, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> >
> >> Changes since v8:
> >>   - Disable "commit-graph" when "in_merge_bases_many()" is called
> >>     for this check, because it returns different results depending
> >>     on whether "commit-graph" is enabled [1].
> >
> > Is that a wise move, though?  If the "different results" is
> > expected, then it is a different story, but I would think it is a
> > bug in commit-graph codepath if it produces a result different from
> > what the callers expect, and disabling from the caller's end would
> > mean that we lose one opportunity to help commit-graph folks to go
> > and fix their bugs, no?

I didn't want want to cause a delay with this patch. Since the new
option was seemingly working without it, I decided to disable it here
and added a "TODO" in the comments to remove "toggle_commit_graph()"
in the future. We can definitely put this on hold and wait until the
with and without "commit-graph" result disparities are clarified.

> > Other than that, I think the topic is in quite a good shape.  Thanks
> > for working on polishing it.

Thanks, I appreciate it!
 
> In other words, how about doing it like so.
> 
> In an ideal world, folks who know more about commit-graph than we do
> will find what's broken in in_merge_bases_many() when commit-graph
> is in use, before I finish lecturing against hiding a breakage under
> the rug.  Let's see if another call for help helps ;-)

:)
 
>  remote.c | 15 ++++++++++++++-
>  1 file changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git i/remote.c w/remote.c
> index 98a578f5dc..361b8f1c0e 100644
> --- i/remote.c
> +++ w/remote.c
> @@ -2408,7 +2408,20 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)
>  /* Toggle the "commit-graph" feature; return the previously set state. */
>  static int toggle_commit_graph(struct repository *repo, int disable) {
>  	int prev = repo->commit_graph_disabled;
> -	repo->commit_graph_disabled = disable;
> +	static int should_toggle = -1;
> +
> +	if (should_toggle < 0) {
> +		/*
> +		 * The in_merge_bases_many() seems to misbehave when
> +		 * the commit-graph feature is in use.  Disable it for
> +		 * normal users, but keep it enabled when specifically
> +		 * testing the feature.
> +		 */
> +		should_toggle = !git_env_bool("GIT_TEST_COMMIT_GRAPH", 0);
> +	}
> +
> +	if (should_toggle)
> +		repo->commit_graph_disabled = disable;
>  	return prev;
>  }
>  

Ah, this will keep a record of the behavior in the tests; nice,
will update in the next set.

I looked around a little bit trying to understand what was happening
here. As mentioned before [1], "repo_in_merge_bases_many()" returns
early because of this:

  for (i = 0; i < nr_reference; i++) {
	  if (repo_parse_commit(r, reference[i]))
		  return ret;

	  generation = commit_graph_generation(reference[i]);
	  if (generation < min_generation)
		  min_generation = generation;
	  fprintf(stderr,
		  "[%s]: (local): generation: %u, min_generation: %u\n",
		  oid_to_hex(&reference[i]->object.oid),
		  generation,
		  min_generation);
  }

  generation = commit_graph_generation(commit);
  fprintf(stderr, "[%s]: (remote) generation: %u, min_generation: %u\n",
	  oid_to_hex(&commit->object.oid), generation, min_generation);
  if (generation > min_generation) {
	  return ret;
  }


I made some changes locally to display the generation numbers of all
the commits that were collected during the "reflog" traversal.

In the beginning we get the minimum generation number of the list of
the commits in "reference[]" to see if the "commit" is reachable from
one of the items in "reference[]". Since in this case, the generation
number of "commit" is higher than minimum amongst the members of
"reference", does it mean it cannot be reached?

When "GIT_TEST_COMMIT_GRAPH" is turned off, all of the commits in
"reference" as well as "commit" have "GENERATION_NUMBER_INFINITY",
and the path moves forward to "paint_down_to_common()" and correctly
identifies the reachability.

I ran the test again, but this time with running "git-commit-graph"
right before pushing:

  (a) git commit-graph write --reachable, and the commit's generation
      number was "GENERATION_NUMBER_INFINITY".

  (b) git-show-ref -s | git commit-graph write --stdin-commits, and
      the commit's generation number was 5.

and since generation number of "commit" was always higher than the
minimum -- it returns that it is not reachable from any of "reference".

One of the failing tests in t5533 was modified (for debugging), and
the following shows the difference in behavior of
"repo_in_merge_bases_many()". The test checks if the remote tip is
reachable from the reflog entries of the local branch after
"git pull --rebase" is run.

  $ git log --graph --oneline # (before "pull --rebase")
   * be2465f J
   * 157d38b C
   * f9a2dac B
   * 112d1ac A

  $ git log --graph --oneline # (after "pull --rebase")
   * 7c16010 J
   * b995a30 G
   * 00b6961 F
   * 157d38b C
   * f9a2dac B
   * 112d1ac A

  $ git reflog master
  7c16010 master@{0}: pull --rebase origin master (finish): refs/heads/master onto b995a30227dd14b34b6f7728201aefac8cc12296
  be2465f master@{1}: commit: J
  157d38b master@{2}: commit: C
  f9a2dac master@{3}: commit: B
  112d1ac master@{4}: commit (initial): A

  - (a) is run.
    $ git push --force-if-includes --force-with-lease=master
    [7c16010bad84d8b53873875c2e242890920360f2]: (local):  generation: 4294967295, min_generation: 4294967295
    [be2465f6a155acb8f5eb9ee876bad730e0f656cf]: (local):  generation: 4, min_generation: 4
    [157d38b4112d457e6645c7c4e9a486e6189be435]: (local):  generation: 3, min_generation: 3
    [f9a2dac17e4f8cafaa9655d40cb86c53094da8d6]: (local):  generation: 2, min_generation: 2
    [112d1ac551b908f10b995d7e41456f4cd8f071c5]: (local):  generation: 1, min_generation: 1
    [b995a30227dd14b34b6f7728201aefac8cc12296]: (remote): generation: 4294967295, min_generation: 1
    [...] fail.

  - "git fetch --all" and (b) is run.
    $ git push --force-if-includes --force-with-lease=master
    [7c16010bad84d8b53873875c2e242890920360f2]: (local):  generation: 4294967295, min_generation: 4294967295
    [be2465f6a155acb8f5eb9ee876bad730e0f656cf]: (local):  generation: 4, min_generation: 4
    [157d38b4112d457e6645c7c4e9a486e6189be435]: (local):  generation: 3, min_generation: 3
    [f9a2dac17e4f8cafaa9655d40cb86c53094da8d6]: (local):  generation: 2, min_generation: 2
    [112d1ac551b908f10b995d7e41456f4cd8f071c5]: (local):  generation: 1, min_generation: 1
    [b995a30227dd14b34b6f7728201aefac8cc12296]: (remote): generation: 5, min_generation: 1
    [...] fail.

  - neither (a), nor (b) are run, and "core.commitGraph" is disabled.
    $ git push --force-if-includes --force-with-lease=master
    [7c16010bad84d8b53873875c2e242890920360f2]: (local):  generation: 4294967295, min_generation: 4294967295
    [be2465f6a155acb8f5eb9ee876bad730e0f656cf]: (local):  generation: 4294967295, min_generation: 4294967295
    [157d38b4112d457e6645c7c4e9a486e6189be435]: (local):  generation: 4294967295, min_generation: 4294967295
    [f9a2dac17e4f8cafaa9655d40cb86c53094da8d6]: (local):  generation: 4294967295, min_generation: 4294967295
    [112d1ac551b908f10b995d7e41456f4cd8f071c5]: (local):  generation: 4294967295, min_generation: 4294967295
    [b995a30227dd14b34b6f7728201aefac8cc12296]: (remote): generation: 4294967295, min_generation: 4294967295
    [...] pass.

It looks like G (b995a30) is reachable from J (7c16010), but
"repo_in_merge_bases_many()" disagrees when "GIT_TEST_COMMIT_GRAPH"
is enabled. I hope this information helps a little to understand
why this happens and whether it is a bug or not.

[1] https://public-inbox.org/git/20200928193400.GA88208@mail.clickyotomy.dev

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH v9 0/3] push: add "--[no-]force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Ah, this will keep a record of the behavior in the tests; nice,
> will update in the next set.

I actually think this kind of "sweep the breakage under the rug"
belongs not in the code that uses the (suspected broken) API but
in the code that is suspected to be broken, i.e. somewhere in
commit.c that is involved in in_merge_bases_many() codepath.

So if you are going to reroll, I would rather see you drop this part
entirely, and add a similar "this code is broken so we won't run in
the real user's environment" to protect all code that call
in_merge_bases_many() from the breakage, not just your new one.

> One of the failing tests in t5533 was modified (for debugging), and
> the following shows the difference in behavior of
> "repo_in_merge_bases_many()". The test checks if the remote tip is
> reachable from the reflog entries of the local branch after
> "git pull --rebase" is run.
> ...
> It looks like G (b995a30) is reachable from J (7c16010), but
> "repo_in_merge_bases_many()" disagrees when "GIT_TEST_COMMIT_GRAPH"
> is enabled. I hope this information helps a little to understand
> why this happens and whether it is a bug or not.

Wonderful.  With this good start, perhaps we may not have to discuss
disabling the commit-graph code in your code or in commit.c for all
callers ;-)

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Johannes Schindelin]

Hi Srinidhi,


On Thu, 1 Oct 2020, Srinidhi Kaushik wrote:

> Add a check to verify if the remote-tracking ref of the local branch
> is reachable from one of its "reflog" entries.
>
> The check iterates through the local ref's reflog to see if there
> is an entry for the remote-tracking ref and collecting any commits
> that are seen, into a list; the iteration stops if an entry in the
> reflog matches the remote ref or if the entry timestamp is older
> the latest entry of the remote ref's "reflog". If there wasn't an
> entry found for the remote ref, "in_merge_bases_many()" is called
> to check if it is reachable from the list of collected commits.
>
> When a local branch that is based on a remote ref, has been rewound
> and is to be force pushed on the remote, "--force-if-includes" runs
> a check that ensures any updates to the remote-tracking ref that may
> have happened (by push from another repository) in-between the time
> of the last update to the local branch (via "git-pull", for instance)
> and right before the time of push, have been integrated locally
> before allowing a forced update.
>
> If the new option is passed without specifying "--force-with-lease",
> or specified along with "--force-with-lease=<refname>:<expect>" it
> is a "no-op".
>
> Calls to "in_merge_bases_many()" return different results depending
> on whether the "commit-graph" feature is enabled or not -- it is
> temporarily disabled when the check runs [1].
>
> [1] https://lore.kernel.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com

I can verify that the multiple calls to `in_merge_bases_many()` lead to a
problem, and I intend to debug this further, but it is the wrong function
to call to begin with.

With these two patches, the tests pass for me, and they also reduce the
complexity quite a bit (Junio, could I ask you to put them on top of
sk/force-if-includes?):

-- snipsnap --
From 0e7bd31c4cb0ae08ad772ac230eea2dd7a884886 Mon Sep 17 00:00:00 2001
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Fri, 2 Oct 2020 15:33:05 +0200
Subject: [PATCH 1/2] fixup??? push: add reflog check for "--force-if-includes"

This follows the pattern used elsewhere.

Maybe we should also rename this to `commit_array`? It is not a linked
list, after all.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 remote.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/remote.c b/remote.c
index 37533cafc44..2c6c63aa906 100644
--- a/remote.c
+++ b/remote.c
@@ -2441,6 +2441,7 @@ struct reflog_commit_list {
 	struct commit **item;
 	size_t nr, alloc;
 };
+#define REFLOG_COMMIT_LIST_INIT { NULL, 0, 0 }

 /* Append a commit to the list. */
 static void append_commit(struct reflog_commit_list *list,
@@ -2514,7 +2515,7 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)
 	struct commit *commit;
 	struct commit **chunk;
 	struct check_and_collect_until_cb_data cb;
-	struct reflog_commit_list list = { NULL, 0, 0 };
+	struct reflog_commit_list list = REFLOG_COMMIT_LIST_INIT;
 	size_t size = 0;
 	int ret = 0;

--
2.28.0.windows.1.18.g5300e52e185


From 10ea5640015f4bc7144e8e5b025e31294329c600 Mon Sep 17 00:00:00 2001
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Fri, 2 Oct 2020 15:35:58 +0200
Subject: [PATCH 2/2] fixup??? push: add reflog check for "--force-if-includes"

We should not call `in_merge_bases_many()` repeatedly: there is a much
better API for that: `get_reachable_subset()`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 remote.c | 43 ++++---------------------------------------
 1 file changed, 4 insertions(+), 39 deletions(-)

diff --git a/remote.c b/remote.c
index 2c6c63aa906..881415921e2 100644
--- a/remote.c
+++ b/remote.c
@@ -2513,10 +2513,9 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)
 {
 	timestamp_t date;
 	struct commit *commit;
-	struct commit **chunk;
 	struct check_and_collect_until_cb_data cb;
 	struct reflog_commit_list list = REFLOG_COMMIT_LIST_INIT;
-	size_t size = 0;
+	struct commit_list *reachable;
 	int ret = 0;

 	commit = lookup_commit_reference(the_repository, &remote->old_oid);
@@ -2542,61 +2541,27 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)
 	 * Check if the remote commit is reachable from any
 	 * of the commits in the collected list, in batches.
 	 */
-	for (chunk = list.item; chunk < list.item + list.nr; chunk += size) {
-		size = list.item + list.nr - chunk;
-		if (MERGE_BASES_BATCH_SIZE < size)
-			size = MERGE_BASES_BATCH_SIZE;
-
-		if ((ret = in_merge_bases_many(commit, size, chunk)))
-			break;
-	}
+	reachable = get_reachable_subset(list.item, list.nr, &commit, 1, 0);
+	ret = !!reachable;
+	free_commit_list(reachable);

 cleanup_return:
 	free_reflog_commit_list(&list);
 	return ret;
 }

-/* Toggle the "commit-graph" feature; return the previously set state. */
-static int toggle_commit_graph(struct repository *repo, int disable) {
-	int prev = repo->commit_graph_disabled;
-	static int should_toggle = -1;
-
-	if (should_toggle < 0) {
-		/*
-		 * The in_merge_bases_many() seems to misbehave when
-		 * the commit-graph feature is in use.  Disable it for
-		 * normal users, but keep it enabled when specifically
-		 * testing the feature.
-		 */
-		should_toggle = !git_env_bool("GIT_TEST_COMMIT_GRAPH", 0);
-	}
-
-	if (should_toggle)
-		repo->commit_graph_disabled = disable;
-	return prev;
-}
-
 /*
  * Check for reachability of a remote-tracking
  * ref in the reflog entries of its local ref.
  */
 static void check_if_includes_upstream(struct ref *remote)
 {
-	int prev;
 	struct ref *local = get_local_ref(remote->name);
 	if (!local)
 		return;

-	/*
-	 * TODO: Remove "toggle_commit_graph()" calls around the check.
-	 * Depending on whether "commit-graph" enabled or not,
-	 * "in_merge_bases_many()" returns different results;
-	 * disable it temporarily when the check runs.
-	 */
-	prev = toggle_commit_graph(the_repository, 1);
 	if (is_reachable_in_reflog(local->name, remote) <= 0)
 		remote->unreachable = 1;
-	toggle_commit_graph(the_repository, prev);
 }

 static void apply_cas(struct push_cas_option *cas,
--
2.28.0.windows.1.18.g5300e52e185

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Johannes Schindelin]

Hi,

On Fri, 2 Oct 2020, Johannes Schindelin wrote:

>
> On Thu, 1 Oct 2020, Srinidhi Kaushik wrote:
>
> > Add a check to verify if the remote-tracking ref of the local branch
> > is reachable from one of its "reflog" entries.
> >
> > The check iterates through the local ref's reflog to see if there
> > is an entry for the remote-tracking ref and collecting any commits
> > that are seen, into a list; the iteration stops if an entry in the
> > reflog matches the remote ref or if the entry timestamp is older
> > the latest entry of the remote ref's "reflog". If there wasn't an
> > entry found for the remote ref, "in_merge_bases_many()" is called
> > to check if it is reachable from the list of collected commits.
> >
> > When a local branch that is based on a remote ref, has been rewound
> > and is to be force pushed on the remote, "--force-if-includes" runs
> > a check that ensures any updates to the remote-tracking ref that may
> > have happened (by push from another repository) in-between the time
> > of the last update to the local branch (via "git-pull", for instance)
> > and right before the time of push, have been integrated locally
> > before allowing a forced update.
> >
> > If the new option is passed without specifying "--force-with-lease",
> > or specified along with "--force-with-lease=<refname>:<expect>" it
> > is a "no-op".
> >
> > Calls to "in_merge_bases_many()" return different results depending
> > on whether the "commit-graph" feature is enabled or not -- it is
> > temporarily disabled when the check runs [1].
> >
> > [1] https://lore.kernel.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com
>
> I can verify that the multiple calls to `in_merge_bases_many()` lead to a
> problem, and I intend to debug this further, but it is the wrong function
> to call to begin with.

It was actually Stolee who figured this out: the shortcut at the beginning
of `in_merge_bases_many()` which tries to exit early when the generation
number of the `commit` indicates that it cannot be reached from the
`reference` commits (because their generation numbers are smaller) has a
bug in the logic. Obviously, the generation numbers are only used when
commit-graph is used, therefore things broke only in the `linux-gcc` job.

Stolee will send out a patch shortly.

Having said that, the change I suggested (to use `get_reachable_subset()`
instead of repeated `in_merge_bases_many()`) is _still_ the right thing to
do: we are not actually interested in the merge bases at all, but in
reachability, and in the future there might be more efficient ways to
determine that than painting down all the way to merge bases.

Thanks,
Dscho

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Hi Johannes,

On 10/02/2020 15:52, Johannes Schindelin wrote:
> Hi Srinidhi,
> 
> 
> On Thu, 1 Oct 2020, Srinidhi Kaushik wrote:
> 
> > Add a check to verify if the remote-tracking ref of the local branch
> > is reachable from one of its "reflog" entries.
> >
> > The check iterates through the local ref's reflog to see if there
> > is an entry for the remote-tracking ref and collecting any commits
> > that are seen, into a list; the iteration stops if an entry in the
> > reflog matches the remote ref or if the entry timestamp is older
> > the latest entry of the remote ref's "reflog". If there wasn't an
> > entry found for the remote ref, "in_merge_bases_many()" is called
> > to check if it is reachable from the list of collected commits.
> >
> > When a local branch that is based on a remote ref, has been rewound
> > and is to be force pushed on the remote, "--force-if-includes" runs
> > a check that ensures any updates to the remote-tracking ref that may
> > have happened (by push from another repository) in-between the time
> > of the last update to the local branch (via "git-pull", for instance)
> > and right before the time of push, have been integrated locally
> > before allowing a forced update.
> >
> > If the new option is passed without specifying "--force-with-lease",
> > or specified along with "--force-with-lease=<refname>:<expect>" it
> > is a "no-op".
> >
> > Calls to "in_merge_bases_many()" return different results depending
> > on whether the "commit-graph" feature is enabled or not -- it is
> > temporarily disabled when the check runs [1].
> >
> > [1] https://lore.kernel.org/git/xmqqtuvhn6yx.fsf@gitster.c.googlers.com
> 
> I can verify that the multiple calls to `in_merge_bases_many()` lead to a
> problem, and I intend to debug this further, but it is the wrong function
> to call to begin with.
> 
> With these two patches, the tests pass for me, and they also reduce the
> complexity quite a bit (Junio, could I ask you to put them on top of
> sk/force-if-includes?):

Thanks for looking into this. :)
 
> -- snipsnap --
> From 0e7bd31c4cb0ae08ad772ac230eea2dd7a884886 Mon Sep 17 00:00:00 2001
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
> Date: Fri, 2 Oct 2020 15:33:05 +0200
> Subject: [PATCH 1/2] fixup??? push: add reflog check for "--force-if-includes"
> 
> This follows the pattern used elsewhere.
> 
> Maybe we should also rename this to `commit_array`? It is not a linked
> list, after all.

Makes sense. I'll change it.
 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  remote.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/remote.c b/remote.c
> index 37533cafc44..2c6c63aa906 100644
> --- a/remote.c
> +++ b/remote.c
> @@ -2441,6 +2441,7 @@ struct reflog_commit_list {
>  	struct commit **item;
>  	size_t nr, alloc;
>  };
> +#define REFLOG_COMMIT_LIST_INIT { NULL, 0, 0 }
> 
>  /* Append a commit to the list. */
>  static void append_commit(struct reflog_commit_list *list,
> @@ -2514,7 +2515,7 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)
>  	struct commit *commit;
>  	struct commit **chunk;
>  	struct check_and_collect_until_cb_data cb;
> -	struct reflog_commit_list list = { NULL, 0, 0 };
> +	struct reflog_commit_list list = REFLOG_COMMIT_LIST_INIT;
>  	size_t size = 0;
>  	int ret = 0;
> 
> --
> 2.28.0.windows.1.18.g5300e52e185
> 
> 
> From 10ea5640015f4bc7144e8e5b025e31294329c600 Mon Sep 17 00:00:00 2001
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
> Date: Fri, 2 Oct 2020 15:35:58 +0200
> Subject: [PATCH 2/2] fixup??? push: add reflog check for "--force-if-includes"
> 
> We should not call `in_merge_bases_many()` repeatedly: there is a much
> better API for that: `get_reachable_subset()`.

Perfect. I wasn't aware of this.

> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  remote.c | 43 ++++---------------------------------------
>  1 file changed, 4 insertions(+), 39 deletions(-)
> 
> diff --git a/remote.c b/remote.c
> index 2c6c63aa906..881415921e2 100644
> --- a/remote.c
> +++ b/remote.c
> @@ -2513,10 +2513,9 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)
>  {
>  	timestamp_t date;
>  	struct commit *commit;
> -	struct commit **chunk;
>  	struct check_and_collect_until_cb_data cb;
>  	struct reflog_commit_list list = REFLOG_COMMIT_LIST_INIT;
> -	size_t size = 0;
> +	struct commit_list *reachable;
>  	int ret = 0;
> 
>  	commit = lookup_commit_reference(the_repository, &remote->old_oid);
> @@ -2542,61 +2541,27 @@ static int is_reachable_in_reflog(const char *local, const struct ref *remote)
>  	 * Check if the remote commit is reachable from any
>  	 * of the commits in the collected list, in batches.
>  	 */
> -	for (chunk = list.item; chunk < list.item + list.nr; chunk += size) {
> -		size = list.item + list.nr - chunk;
> -		if (MERGE_BASES_BATCH_SIZE < size)
> -			size = MERGE_BASES_BATCH_SIZE;
> -
> -		if ((ret = in_merge_bases_many(commit, size, chunk)))
> -			break;
> -	}
> +	reachable = get_reachable_subset(list.item, list.nr, &commit, 1, 0);
> +	ret = !!reachable;
> +	free_commit_list(reachable);
> 
>  cleanup_return:
>  	free_reflog_commit_list(&list);
>  	return ret;
>  }
> 
> -/* Toggle the "commit-graph" feature; return the previously set state. */
> -static int toggle_commit_graph(struct repository *repo, int disable) {
> -	int prev = repo->commit_graph_disabled;
> -	static int should_toggle = -1;
> -
> -	if (should_toggle < 0) {
> -		/*
> -		 * The in_merge_bases_many() seems to misbehave when
> -		 * the commit-graph feature is in use.  Disable it for
> -		 * normal users, but keep it enabled when specifically
> -		 * testing the feature.
> -		 */
> -		should_toggle = !git_env_bool("GIT_TEST_COMMIT_GRAPH", 0);
> -	}
> -
> -	if (should_toggle)
> -		repo->commit_graph_disabled = disable;
> -	return prev;
> -}
> -

OK. The tests are passing with or without "GIT_TEST_COMMIT_GRAPH"
by switching to "get_reachable_subset()" we don't have to toggle
te feature during the check.

>  /*
>   * Check for reachability of a remote-tracking
>   * ref in the reflog entries of its local ref.
>   */
>  static void check_if_includes_upstream(struct ref *remote)
>  {
> -	int prev;
>  	struct ref *local = get_local_ref(remote->name);
>  	if (!local)
>  		return;
> 
> -	/*
> -	 * TODO: Remove "toggle_commit_graph()" calls around the check.
> -	 * Depending on whether "commit-graph" enabled or not,
> -	 * "in_merge_bases_many()" returns different results;
> -	 * disable it temporarily when the check runs.
> -	 */
> -	prev = toggle_commit_graph(the_repository, 1);
>  	if (is_reachable_in_reflog(local->name, remote) <= 0)
>  		remote->unreachable = 1;
> -	toggle_commit_graph(the_repository, prev);
>  }
> 
>  static void apply_cas(struct push_cas_option *cas,
> --
> 2.28.0.windows.1.18.g5300e52e185
> 

Again, thank you so much working on this! If you'd like, I can go ahead
and apply these patches and rename "reflog_commit_list" to "commit_array"
in the next series (v10).

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Having said that, the change I suggested (to use `get_reachable_subset()`
> instead of repeated `in_merge_bases_many()`) is _still_ the right thing to
> do: we are not actually interested in the merge bases at all, but in
> reachability, and in the future there might be more efficient ways to
> determine that than painting down all the way to merge bases.

I agree with you that the age-old implementation has an obvious room
for optimization.  I think I already pointed out a #leftoverbit that
we can invent a version of paint_down_to_common() that can
short-circuit and return immediately after one side (the "commit"
side) gets painted, so that in_merge_bases_many() can stop
immediately after finding out that the answer is "true".

The function is *not* about computing the merge base across the
commits on the "reference" side but finding out if "commit" is
reachable from any in the "reference" side, so (1) it has a wrong
name and more importantly (2) it wants to do something quite similar
to get_reachable_subset(), but it is much less ambitious.

get_reachable_subset() is capable of doing a lot more.  Unlike the
older in_merge_bases_many() that allowed only one commit on the
candidate for an ancestor side, it can throw a set and ask "which
ones among these are reachable from the other set".

So from the "semantics" point of view, get_reachable_subset() is
overkill and less suitable than in_merge_bases_many() for this
particular application.  We know we have only one candidate, and we
want to ask "is this reachable, or not?" a single bit question.  In
any case, they should yield the right answer from correctness point
of view ;-)

Having said that.

I do not think in the longer term we should keep both.  Clearly the
get_reachable_subset() function can handle more general cases, so it
would make a lot of sense to make in_merge_bases_many() into a thin
wrapper that feeds just a single commit array on one side to be
filtered while feeding the "reference" commits to the other side, as
long as we can demonstrate that the result is just as correct as,
and it is not slower than, the current implementation.  That may be
a bit larger than a typical #leftoverbit but would be a good clean-up
project.

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> I can verify that the multiple calls to `in_merge_bases_many()` lead to a
> problem ...

Is it because it forgets to clear the marks?  I do not think so.
Besides the sample history demonstrated by the failing test were so
small that it didn't need multiple calls, IIRC.  There was a trivial
bug in the function when commit-graph was enabled and there is no
reason to avoid calling the function multiple times, right?

I just wanted to make sure that a "in-merge-bases-many cannot be
called twice" myth does not get etched in the archive.

Thanks for solving the puzzle with Derrick, by the way.  Very much
appreciated.  I was wondering if my CC'ing commit-graph folks were
somehow not reaching the intended recipients.

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

>> We should not call `in_merge_bases_many()` repeatedly: there is a much
>> better API for that: `get_reachable_subset()`.
>
> Perfect. I wasn't aware of this.

This is possibly a piece of misinformation.  in_merge_bases_many()
is designed to be callable more than once.  get_reachable_subset()
may be an overkill as we only are interested in a single "is this
one an ancestor of any of these?", not "which ones among these are
ancestors of the other set?".

> OK. The tests are passing with or without "GIT_TEST_COMMIT_GRAPH"
> by switching to "get_reachable_subset()" we don't have to toggle
> te feature during the check.

Correct.  Once the "see if this one is reachable from any of these"
is fixed (either by correcting the broken in_merge_bases_many() or
using get_reachable_subset()), we can get rid of this hack.

> Again, thank you so much working on this! If you'd like, I can go ahead
> and apply these patches and rename "reflog_commit_list" to "commit_array"
> in the next series (v10).

I like the s/list/array/ change, but I do not think switching to
get_reachable_subset() and having to receive a commit list only to
free the list is warranted.

Derrick sent a fix to in_merge_bases_many() in the near-by thread.

Thanks.

Re: [PATCH v9 0/3] push: add "--[no-]force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> I didn't want want to cause a delay with this patch. Since the new
> option was seemingly working without it,...

It is a good example to help other new contributors to understand an
important point in how the development in common works, so let me
say this.

I did very much wanted to keep the bug exposed at least to the test
suite.  Since the broken helper were designed to be used in many
other places in the code, and we had a simple reproduction recipe in
this topic, using it as an opening to help debug and fix bugs in the
broken helper had higher priority than adding the "--force-if-includes"
feature.

We help the contributors who have been involved in the broken helper
by delaying this topic a bit and leaving the reproduction readily
available to them, so that they help us who are working on a piece
of code that wants to see the broken helper fixed.  

That way everybody benefits.

It's not like a corporate development where your interest lies in
shipping your piece regardless of the work done by other teams,
where it might serve you better by using the second best tool for
the task, to avoid the tool that ought to be best but does not work
well *and* you do not want to help the team that manages that best
tool, even if helping them may benefit the whole organization.

So, let's play well together.  Yield a bit to help others and let
others also help you.

Thanks.

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Hi Junio,

On 10/02/2020 09:41, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> >> We should not call `in_merge_bases_many()` repeatedly: there is a much
> >> better API for that: `get_reachable_subset()`.
> >
> > Perfect. I wasn't aware of this.
> 
> This is possibly a piece of misinformation.  in_merge_bases_many()
> is designed to be callable more than once.  get_reachable_subset()
> may be an overkill as we only are interested in a single "is this
> one an ancestor of any of these?", not "which ones among these are
> ancestors of the other set?".

Noted; even though "get_reachable_subset()" and "in_merge_bases_many()"
(after the commit-graph fix) return the same result, I suppose the
latter was designed for this specific use-case.

> > OK. The tests are passing with or without "GIT_TEST_COMMIT_GRAPH"
> > by switching to "get_reachable_subset()" we don't have to toggle
> > te feature during the check.
> 
> Correct.  Once the "see if this one is reachable from any of these"
> is fixed (either by correcting the broken in_merge_bases_many() or
> using get_reachable_subset()), we can get rid of this hack.

OK. Shall I update the next set by reverting the "disable commit-graph"
change, s/list/array/ and leaving the rest as is -- if we decide to go
forward with "in_merge_bases_many()", that is?

> > Again, thank you so much working on this! If you'd like, I can go ahead
> > and apply these patches and rename "reflog_commit_list" to "commit_array"
> > in the next series (v10).
> 
> I like the s/list/array/ change, but I do not think switching to
> get_reachable_subset() and having to receive a commit list only to
> free the list is warranted.
> 
> Derrick sent a fix to in_merge_bases_many() in the near-by thread.

Nice! Will take a look.

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH v9 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Hi Junio,

On 10/02/2020 09:50, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> 
> > I didn't want want to cause a delay with this patch. Since the new
> > option was seemingly working without it,...
> 
> It is a good example to help other new contributors to understand an
> important point in how the development in common works, so let me
> say this.
> 
> I did very much wanted to keep the bug exposed at least to the test
> suite.  Since the broken helper were designed to be used in many
> other places in the code, and we had a simple reproduction recipe in
> this topic, using it as an opening to help debug and fix bugs in the
> broken helper had higher priority than adding the "--force-if-includes"
> feature.
> 
> We help the contributors who have been involved in the broken helper
> by delaying this topic a bit and leaving the reproduction readily
> available to them, so that they help us who are working on a piece
> of code that wants to see the broken helper fixed.  
> 
> That way everybody benefits.
> 
> It's not like a corporate development where your interest lies in
> shipping your piece regardless of the work done by other teams,
> where it might serve you better by using the second best tool for
> the task, to avoid the tool that ought to be best but does not work
> well *and* you do not want to help the team that manages that best
> tool, even if helping them may benefit the whole organization.
> 
> So, let's play well together.  Yield a bit to help others and let
> others also help you.
> 
> Thanks.

Thank you for pointing this out. You're right; I should not have
rushed to disabling the feature because it wasn't working with my
patch instead of waiting for the issue to be investigated. This is
valuable advice, and I will keep this in mind when making future
contributions.

Thanks.
-- 
Srinidhi Kaushik

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Hi Junio,
>
> On 10/02/2020 09:41, Junio C Hamano wrote:
>> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
>> 
>> >> We should not call `in_merge_bases_many()` repeatedly: there is a much
>> >> better API for that: `get_reachable_subset()`.
>> >
>> > Perfect. I wasn't aware of this.
>> 
>> This is possibly a piece of misinformation.  in_merge_bases_many()
>> is designed to be callable more than once.  get_reachable_subset()
>> may be an overkill as we only are interested in a single "is this
>> one an ancestor of any of these?", not "which ones among these are
>> ancestors of the other set?".
>
> Noted; even though "get_reachable_subset()" and "in_merge_bases_many()"
> (after the commit-graph fix) return the same result, I suppose the
> latter was designed for this specific use-case.

Yes, in_merge_bases_many() was invented first in 4c4b27e8 (commit.c:
add in_merge_bases_many(), 2013-03-04) for this exact use case.  For
use cases where callers have multiple "these may be ancestors"
candidates, instead of having to iterate over them and calling
in_merge_bases_many() multiple times, get_reachable_subset() was
added much later at fcb2c076 (commit-reach: implement
get_reachable_subset, 2018-11-02).

> OK. Shall I update the next set by reverting the "disable commit-graph"
> change, s/list/array/ and leaving the rest as is -- if we decide to go
> forward with "in_merge_bases_many()", that is?

Yes, that would be the ideal endgame.  What I pushed out to 'seen'
has the removal of "disable" bit as a SQUASH??? commit at the tip,
but not s/list/array renaming.

Thanks.

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Hello,

On 10/02/2020 13:14, Junio C Hamano wrote:
> Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:
> > Noted; even though "get_reachable_subset()" and "in_merge_bases_many()"
> > (after the commit-graph fix) return the same result, I suppose the
> > latter was designed for this specific use-case.
> 
> Yes, in_merge_bases_many() was invented first in 4c4b27e8 (commit.c:
> add in_merge_bases_many(), 2013-03-04) for this exact use case.  For
> use cases where callers have multiple "these may be ancestors"
> candidates, instead of having to iterate over them and calling
> in_merge_bases_many() multiple times, get_reachable_subset() was
> added much later at fcb2c076 (commit-reach: implement
> get_reachable_subset, 2018-11-02).

Got it. Thanks for the detailed explanation and reference.
 
> > OK. Shall I update the next set by reverting the "disable commit-graph"
> > change, s/list/array/ and leaving the rest as is -- if we decide to go
> > forward with "in_merge_bases_many()", that is?
> 
> Yes, that would be the ideal endgame.  What I pushed out to 'seen'
> has the removal of "disable" bit as a SQUASH??? commit at the tip,
> but not s/list/array renaming.
> 
> Thanks.

Alright, I will add those changes in the next set. Also, I saw in the
other thread that you tested commit-graph fix on this series and the
tests are passing -- thanks for checking.
-- 
Srinidhi Kaushik

Re: [PATCH v9 1/3] push: add reflog check for "--force-if-includes"

[Junio C Hamano]

Srinidhi Kaushik <shrinidhi.kaushik@gmail.com> writes:

> Alright, I will add those changes in the next set. Also, I saw in the
> other thread that you tested commit-graph fix on this series and the
> tests are passing -- thanks for checking.

What I pushed out on 'seen':

 - starts building sk/force-if-includes topic on 6c430a647cb99 (at
   the tip of master sometime ago)

 - merges Derrick's fix ds/in-merge-bases-many-optim-bug topic

 - and then applies the previous three patches on top

 - and finally have the SQUASH??? fix-up.

Rebuilding your series on top of this commit

    aed0800ca6 Merge branch 'ds/in-merge-bases-many-optim-bug' into sk/force-if-includes

would be the most appropriate.

Thanks.

[PATCH v10 0/3] push: add "--[no-]force-if-includes"

[Srinidhi Kaushik]

Add a new option: "--force-if-includes" to "git-push" where forced
updates are allowed only if the tip of the remote-tracking ref has
been integrated locally, by verifying if the tip of the remote-tracking
ref -- on which a local branch has based on -- is reachable from at
least one of the "reflog" entries of the branch about to be updated
by force on the remote.

This option can be used with "--force-with-lease" with setups where
the remote-tracking refs of the repository are implicitly updated in
the background to help prevent unintended remote overwrites.

If a local branch is based on a remote ref for a rewrite, and if that
remote-tracking ref is updated by a push from another repository after
it has been checked out locally, force updating that branch to remote
with "--force-with-lease[=<refname>[:<expect>]]" without specifying
the "<expect>" value, can cause the update that happened in-between
the checkout and forced push to be lost.

Changes since v9:
  - Remove "toggle_commit_graph()" because "in_merge_bases_many()"
    has been fixed [1].

  - Rename "reflog_commit_list" to "reflog_commit_array", and related
    comments.

Srinidhi Kaushik (3):
  push: add reflog check for "--force-if-includes"
  push: parse and set flag for "--force-if-includes"
  t, doc: update tests, reference for "--force-if-includes"

 Documentation/config/advice.txt |   9 +-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 ++++-
 advice.c                        |   3 +
 advice.h                        |   2 +
 builtin/push.c                  |  27 +++++
 builtin/send-pack.c             |  11 ++
 remote-curl.c                   |  14 ++-
 remote.c                        | 184 ++++++++++++++++++++++++++++++--
 remote.h                        |  12 ++-
 send-pack.c                     |   1 +
 t/t5533-push-cas.sh             | 137 ++++++++++++++++++++++++
 transport-helper.c              |  10 ++
 transport.c                     |   8 ++
 transport.h                     |  15 ++-
 15 files changed, 447 insertions(+), 18 deletions(-)

base-commit: aed0800ca6 (Merge branch 'ds/in-merge-bases-many-optim-bug'
			 into sk/force-if-includes, 2020-10-02)

[1] https://public-inbox.org/git/pull.739.git.1601650736489.gitgitgadget@gmail.com
--
2.28.0

[PATCH v10 1/3] push: add reflog check for "--force-if-includes"

[Srinidhi Kaushik]

Add a check to verify if the remote-tracking ref of the local branch
is reachable from one of its "reflog" entries.

The check iterates through the local ref's reflog to see if there
is an entry for the remote-tracking ref and collecting any commits
that are seen, into a list; the iteration stops if an entry in the
reflog matches the remote ref or if the entry timestamp is older
the latest entry of the remote ref's "reflog". If there wasn't an
entry found for the remote ref, "in_merge_bases_many()" is called
to check if it is reachable from the list of collected commits.

When a local branch that is based on a remote ref, has been rewound
and is to be force pushed on the remote, "--force-if-includes" runs
a check that ensures any updates to the remote-tracking ref that may
have happened (by push from another repository) in-between the time
of the last update to the local branch (via "git-pull", for instance)
and right before the time of push, have been integrated locally
before allowing a forced update.

If the new option is passed without specifying "--force-with-lease",
or specified along with "--force-with-lease=<refname>:<expect>" it
is a "no-op".

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 builtin/send-pack.c |   5 ++
 remote.c            | 184 ++++++++++++++++++++++++++++++++++++++++++--
 remote.h            |  12 ++-
 send-pack.c         |   1 +
 transport-helper.c  |   5 ++
 transport.c         |   6 ++
 6 files changed, 205 insertions(+), 8 deletions(-)

diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 7af148d733..516cba7336 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -71,6 +71,11 @@ static void print_helper_status(struct ref *ref)
 			msg = "stale info";
 			break;
 
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
+			res = "error";
+			msg = "remote ref updated since checkout";
+			break;
+
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
 			res = "error";
 			msg = "already exists";
diff --git a/remote.c b/remote.c
index eafc14cbe7..10e0be9097 100644
--- a/remote.c
+++ b/remote.c
@@ -1471,12 +1471,23 @@ void set_ref_status_for_push(struct ref *remote_refs, int send_mirror,
 		 * with the remote-tracking branch to find the value
 		 * to expect, but we did not have such a tracking
 		 * branch.
+		 *
+		 * If the tip of the remote-tracking ref is unreachable
+		 * from any reflog entry of its local ref indicating a
+		 * possible update since checkout; reject the push.
 		 */
 		if (ref->expect_old_sha1) {
 			if (!oideq(&ref->old_oid, &ref->old_oid_expect))
 				reject_reason = REF_STATUS_REJECT_STALE;
+			else if (ref->check_reachable && ref->unreachable)
+				reject_reason =
+					REF_STATUS_REJECT_REMOTE_UPDATED;
 			else
-				/* If the ref isn't stale then force the update. */
+				/*
+				 * If the ref isn't stale, and is reachable
+				 * from from one of the reflog entries of
+				 * the local branch, force the update.
+				 */
 				force_ref_update = 1;
 		}
 
@@ -2251,12 +2262,13 @@ int is_empty_cas(const struct push_cas_option *cas)
 
 /*
  * Look at remote.fetch refspec and see if we have a remote
- * tracking branch for the refname there.  Fill its current
- * value in sha1[].
+ * tracking branch for the refname there. Fill the name of
+ * the remote-tracking branch in *dst_refname, and the name
+ * of the commit object at its tip in oid[].
  * If we cannot do so, return negative to signal an error.
  */
 static int remote_tracking(struct remote *remote, const char *refname,
-			   struct object_id *oid)
+			   struct object_id *oid, char **dst_refname)
 {
 	char *dst;
 
@@ -2265,9 +2277,150 @@ static int remote_tracking(struct remote *remote, const char *refname,
 		return -1; /* no tracking ref for refname at remote */
 	if (read_ref(dst, oid))
 		return -1; /* we know what the tracking ref is but we cannot read it */
+
+	*dst_refname = dst;
 	return 0;
 }
 
+/*
+ * The struct "reflog_commit_array" and related helper functions
+ * are used for collecting commits into an array during reflog
+ * traversals in "check_and_collect_until()".
+ */
+struct reflog_commit_array {
+	struct commit **item;
+	size_t nr, alloc;
+};
+
+#define REFLOG_COMMIT_ARRAY_INIT { NULL, 0, 0 }
+
+/* Append a commit to the array. */
+static void append_commit(struct reflog_commit_array *arr,
+			  struct commit *commit)
+{
+	ALLOC_GROW(arr->item, arr->nr + 1, arr->alloc);
+	arr->item[arr->nr++] = commit;
+}
+
+/* Free and reset the array. */
+static void free_commit_array(struct reflog_commit_array *arr)
+{
+	FREE_AND_NULL(arr->item);
+	arr->nr = arr->alloc = 0;
+}
+
+struct check_and_collect_until_cb_data {
+	struct commit *remote_commit;
+	struct reflog_commit_array *local_commits;
+	timestamp_t remote_reflog_timestamp;
+};
+
+/* Get the timestamp of the latest entry. */
+static int peek_reflog(struct object_id *o_oid, struct object_id *n_oid,
+		       const char *ident, timestamp_t timestamp,
+		       int tz, const char *message, void *cb_data)
+{
+	timestamp_t *ts = cb_data;
+	*ts = timestamp;
+	return 1;
+}
+
+static int check_and_collect_until(struct object_id *o_oid,
+				   struct object_id *n_oid,
+				   const char *ident, timestamp_t timestamp,
+				   int tz, const char *message, void *cb_data)
+{
+	struct commit *commit;
+	struct check_and_collect_until_cb_data *cb = cb_data;
+
+	/* An entry was found. */
+	if (oideq(n_oid, &cb->remote_commit->object.oid))
+		return 1;
+
+	if ((commit = lookup_commit_reference(the_repository, n_oid)))
+		append_commit(cb->local_commits, commit);
+
+	/*
+	 * If the reflog entry timestamp is older than the remote ref's
+	 * latest reflog entry, there is no need to check or collect
+	 * entries older than this one.
+	 */
+	if (timestamp < cb->remote_reflog_timestamp)
+		return -1;
+
+	return 0;
+}
+
+#define MERGE_BASES_BATCH_SIZE 8
+
+/*
+ * Iterate through the reflog of the local ref to check if there is an entry
+ * for the given remote-tracking ref; runs until the timestamp of an entry is
+ * older than latest timestamp of remote-tracking ref's reflog. Any commits
+ * are that seen along the way are collected into an array to check if the
+ * remote-tracking ref is reachable from any of them.
+ */
+static int is_reachable_in_reflog(const char *local, const struct ref *remote)
+{
+	timestamp_t date;
+	struct commit *commit;
+	struct commit **chunk;
+	struct check_and_collect_until_cb_data cb;
+	struct reflog_commit_array arr = REFLOG_COMMIT_ARRAY_INIT;
+	size_t size = 0;
+	int ret = 0;
+
+	commit = lookup_commit_reference(the_repository, &remote->old_oid);
+	if (!commit)
+		goto cleanup_return;
+
+	/*
+	 * Get the timestamp from the latest entry
+	 * of the remote-tracking ref's reflog.
+	 */
+	for_each_reflog_ent_reverse(remote->tracking_ref, peek_reflog, &date);
+
+	cb.remote_commit = commit;
+	cb.local_commits = &arr;
+	cb.remote_reflog_timestamp = date;
+	ret = for_each_reflog_ent_reverse(local, check_and_collect_until, &cb);
+
+	/* We found an entry in the reflog. */
+	if (ret > 0)
+		goto cleanup_return;
+
+	/*
+	 * Check if the remote commit is reachable from any
+	 * of the commits in the collected array, in batches.
+	 */
+	for (chunk = arr.item; chunk < arr.item + arr.nr; chunk += size) {
+		size = arr.item + arr.nr - chunk;
+		if (MERGE_BASES_BATCH_SIZE < size)
+			size = MERGE_BASES_BATCH_SIZE;
+
+		if ((ret = in_merge_bases_many(commit, size, chunk)))
+			break;
+	}
+
+cleanup_return:
+	free_commit_array(&arr);
+	return ret;
+}
+
+/*
+ * Check for reachability of a remote-tracking
+ * ref in the reflog entries of its local ref.
+ */
+static void check_if_includes_upstream(struct ref *remote)
+{
+	struct ref *local = get_local_ref(remote->name);
+	if (!local)
+		return;
+
+	if (is_reachable_in_reflog(local->name, remote) <= 0)
+		remote->unreachable = 1;
+}
+
 static void apply_cas(struct push_cas_option *cas,
 		      struct remote *remote,
 		      struct ref *ref)
@@ -2282,8 +2435,12 @@ static void apply_cas(struct push_cas_option *cas,
 		ref->expect_old_sha1 = 1;
 		if (!entry->use_tracking)
 			oidcpy(&ref->old_oid_expect, &entry->expect);
-		else if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+		else if (remote_tracking(remote, ref->name,
+					 &ref->old_oid_expect,
+					 &ref->tracking_ref))
 			oidclr(&ref->old_oid_expect);
+		else
+			ref->check_reachable = cas->use_force_if_includes;
 		return;
 	}
 
@@ -2292,8 +2449,12 @@ static void apply_cas(struct push_cas_option *cas,
 		return;
 
 	ref->expect_old_sha1 = 1;
-	if (remote_tracking(remote, ref->name, &ref->old_oid_expect))
+	if (remote_tracking(remote, ref->name,
+			    &ref->old_oid_expect,
+			    &ref->tracking_ref))
 		oidclr(&ref->old_oid_expect);
+	else
+		ref->check_reachable = cas->use_force_if_includes;
 }
 
 void apply_push_cas(struct push_cas_option *cas,
@@ -2301,6 +2462,15 @@ void apply_push_cas(struct push_cas_option *cas,
 		    struct ref *remote_refs)
 {
 	struct ref *ref;
-	for (ref = remote_refs; ref; ref = ref->next)
+	for (ref = remote_refs; ref; ref = ref->next) {
 		apply_cas(cas, remote, ref);
+
+		/*
+		 * If "compare-and-swap" is in "use_tracking[_for_rest]"
+		 * mode, and if "--force-if-includes" was specified, run
+		 * the check.
+		 */
+		if (ref->check_reachable)
+			check_if_includes_upstream(ref);
+	}
 }
diff --git a/remote.h b/remote.h
index eb62a47044..2d5391d281 100644
--- a/remote.h
+++ b/remote.h
@@ -107,12 +107,20 @@ struct ref {
 	struct object_id new_oid;
 	struct object_id old_oid_expect; /* used by expect-old */
 	char *symref;
+	char *tracking_ref;
 	unsigned int
 		force:1,
 		forced_update:1,
 		expect_old_sha1:1,
 		exact_oid:1,
-		deletion:1;
+		deletion:1,
+		/* Need to check if local reflog reaches the remote tip. */
+		check_reachable:1,
+		/*
+		 * Store the result of the check enabled by "check_reachable";
+		 * implies the local reflog does not reach the remote tip.
+		 */
+		unreachable:1;
 
 	enum {
 		REF_NOT_MATCHED = 0, /* initial value */
@@ -142,6 +150,7 @@ struct ref {
 		REF_STATUS_REJECT_NEEDS_FORCE,
 		REF_STATUS_REJECT_STALE,
 		REF_STATUS_REJECT_SHALLOW,
+		REF_STATUS_REJECT_REMOTE_UPDATED,
 		REF_STATUS_UPTODATE,
 		REF_STATUS_REMOTE_REJECT,
 		REF_STATUS_EXPECTING_REPORT,
@@ -341,6 +350,7 @@ struct ref *get_stale_heads(struct refspec *rs, struct ref *fetch_map);
 
 struct push_cas_option {
 	unsigned use_tracking_for_rest:1;
+	unsigned use_force_if_includes:1;
 	struct push_cas {
 		struct object_id expect;
 		unsigned use_tracking:1;
diff --git a/send-pack.c b/send-pack.c
index 2d2f9997ac..b4d4a10a71 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -299,6 +299,7 @@ static int check_to_send_update(const struct ref *ref, const struct send_pack_ar
 	case REF_STATUS_REJECT_FETCH_FIRST:
 	case REF_STATUS_REJECT_NEEDS_FORCE:
 	case REF_STATUS_REJECT_STALE:
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
 	case REF_STATUS_REJECT_NODELETE:
 		return CHECK_REF_STATUS_REJECTED;
 	case REF_STATUS_UPTODATE:
diff --git a/transport-helper.c b/transport-helper.c
index b573b6c730..6157de30c7 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -827,6 +827,10 @@ static int push_update_ref_status(struct strbuf *buf,
 			status = REF_STATUS_REJECT_STALE;
 			FREE_AND_NULL(msg);
 		}
+		else if (!strcmp(msg, "remote ref updated since checkout")) {
+			status = REF_STATUS_REJECT_REMOTE_UPDATED;
+			FREE_AND_NULL(msg);
+		}
 		else if (!strcmp(msg, "forced update")) {
 			forced = 1;
 			FREE_AND_NULL(msg);
@@ -967,6 +971,7 @@ static int push_refs_with_push(struct transport *transport,
 		case REF_STATUS_REJECT_NONFASTFORWARD:
 		case REF_STATUS_REJECT_STALE:
 		case REF_STATUS_REJECT_ALREADY_EXISTS:
+		case REF_STATUS_REJECT_REMOTE_UPDATED:
 			if (atomic) {
 				reject_atomic_push(remote_refs, mirror);
 				string_list_clear(&cas_options, 0);
diff --git a/transport.c b/transport.c
index ffe2115845..65fcd22b20 100644
--- a/transport.c
+++ b/transport.c
@@ -633,6 +633,11 @@ static int print_one_push_report(struct ref *ref, const char *dest, int count,
 				 "stale info",
 				 report, porcelain, summary_width);
 		break;
+	case REF_STATUS_REJECT_REMOTE_UPDATED:
+		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
+				 "remote ref updated since checkout",
+				 report, porcelain, summary_width);
+		break;
 	case REF_STATUS_REJECT_SHALLOW:
 		print_ref_status('!', "[rejected]", ref, ref->peer_ref,
 				 "new shallow roots not allowed",
@@ -1185,6 +1190,7 @@ static int run_pre_push_hook(struct transport *transport,
 		if (!r->peer_ref) continue;
 		if (r->status == REF_STATUS_REJECT_NONFASTFORWARD) continue;
 		if (r->status == REF_STATUS_REJECT_STALE) continue;
+		if (r->status == REF_STATUS_REJECT_REMOTE_UPDATED) continue;
 		if (r->status == REF_STATUS_UPTODATE) continue;
 
 		strbuf_reset(&buf);
-- 
2.28.0

[PATCH v10 2/3] push: parse and set flag for "--force-if-includes"

[Srinidhi Kaushik]

The previous commit added the necessary machinery to implement the
"--force-if-includes" protection, when "--force-with-lease" is used
without giving exact object the remote still ought to have. Surface
the feature by adding a command line option and a configuration
variable to enable it.

 - Add a flag: "TRANSPORT_PUSH_FORCE_IF_INCLUDES" to indicate that the
   new option was passed from the command line of via configuration
   settings; update command line and configuration parsers to set the
   new flag accordingly.

 - Introduce a new configuration option "push.useForceIfIncludes", which
   is equivalent to setting "--force-if-includes" in the command line.

 - Update "remote-curl" to recognize and pass this option to "send-pack"
   when enabled.

 - Update "advise" to catch the reject reason "REJECT_REF_NEEDS_UPDATE",
   set when the ref status is "REF_STATUS_REJECT_REMOTE_UPDATED" and
   (optionally) print a help message when the push fails.

 - The new option is a "no-op" in the following scenarios:
    * When used without "--force-with-lease".
    * When used with "--force-with-lease", and if the expected commit
      on the remote side is specified as an argument.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 advice.c            |  3 +++
 advice.h            |  2 ++
 builtin/push.c      | 27 +++++++++++++++++++++++++++
 builtin/send-pack.c |  6 ++++++
 remote-curl.c       | 14 +++++++++++++-
 transport-helper.c  |  5 +++++
 transport.c         |  2 ++
 transport.h         | 15 ++++++++++-----
 8 files changed, 68 insertions(+), 6 deletions(-)

diff --git a/advice.c b/advice.c
index f0a3d32d20..164742305f 100644
--- a/advice.c
+++ b/advice.c
@@ -11,6 +11,7 @@ int advice_push_already_exists = 1;
 int advice_push_fetch_first = 1;
 int advice_push_needs_force = 1;
 int advice_push_unqualified_ref_name = 1;
+int advice_push_ref_needs_update = 1;
 int advice_status_hints = 1;
 int advice_status_u_option = 1;
 int advice_status_ahead_behind_warning = 1;
@@ -72,6 +73,7 @@ static struct {
 	{ "pushFetchFirst", &advice_push_fetch_first },
 	{ "pushNeedsForce", &advice_push_needs_force },
 	{ "pushUnqualifiedRefName", &advice_push_unqualified_ref_name },
+	{ "pushRefNeedsUpdate", &advice_push_ref_needs_update },
 	{ "statusHints", &advice_status_hints },
 	{ "statusUoption", &advice_status_u_option },
 	{ "statusAheadBehindWarning", &advice_status_ahead_behind_warning },
@@ -116,6 +118,7 @@ static struct {
 	[ADVICE_PUSH_ALREADY_EXISTS]			= { "pushAlreadyExists", 1 },
 	[ADVICE_PUSH_FETCH_FIRST]			= { "pushFetchFirst", 1 },
 	[ADVICE_PUSH_NEEDS_FORCE]			= { "pushNeedsForce", 1 },
+	[ADVICE_PUSH_REF_NEEDS_UPDATE]			= { "pushRefNeedsUpdate", 1 },
 
 	/* make this an alias for backward compatibility */
 	[ADVICE_PUSH_UPDATE_REJECTED_ALIAS]		= { "pushNonFastForward", 1 },
diff --git a/advice.h b/advice.h
index 16f2c11642..bc2432980a 100644
--- a/advice.h
+++ b/advice.h
@@ -11,6 +11,7 @@ extern int advice_push_already_exists;
 extern int advice_push_fetch_first;
 extern int advice_push_needs_force;
 extern int advice_push_unqualified_ref_name;
+extern int advice_push_ref_needs_update;
 extern int advice_status_hints;
 extern int advice_status_u_option;
 extern int advice_status_ahead_behind_warning;
@@ -60,6 +61,7 @@ extern int advice_add_empty_pathspec;
 	ADVICE_PUSH_UNQUALIFIED_REF_NAME,
 	ADVICE_PUSH_UPDATE_REJECTED_ALIAS,
 	ADVICE_PUSH_UPDATE_REJECTED,
+	ADVICE_PUSH_REF_NEEDS_UPDATE,
 	ADVICE_RESET_QUIET_WARNING,
 	ADVICE_RESOLVE_CONFLICT,
 	ADVICE_RM_HINTS,
diff --git a/builtin/push.c b/builtin/push.c
index 0eeb2c8dd5..908b557edb 100644
--- a/builtin/push.c
+++ b/builtin/push.c
@@ -290,6 +290,12 @@ static const char message_advice_ref_needs_force[] =
 	   "or update a remote ref to make it point at a non-commit object,\n"
 	   "without using the '--force' option.\n");
 
+static const char message_advice_ref_needs_update[] =
+	N_("Updates were rejected because the tip of the remote-tracking\n"
+	   "branch has been updated since the last checkout. You may want\n"
+	   "to integrate those changes locally (e.g., 'git pull ...')\n"
+	   "before forcing an update.\n");
+
 static void advise_pull_before_push(void)
 {
 	if (!advice_push_non_ff_current || !advice_push_update_rejected)
@@ -325,6 +331,13 @@ static void advise_ref_needs_force(void)
 	advise(_(message_advice_ref_needs_force));
 }
 
+static void advise_ref_needs_update(void)
+{
+	if (!advice_push_ref_needs_update || !advice_push_update_rejected)
+		return;
+	advise(_(message_advice_ref_needs_update));
+}
+
 static int push_with_options(struct transport *transport, struct refspec *rs,
 			     int flags)
 {
@@ -374,6 +387,8 @@ static int push_with_options(struct transport *transport, struct refspec *rs,
 		advise_ref_fetch_first();
 	} else if (reject_reasons & REJECT_NEEDS_FORCE) {
 		advise_ref_needs_force();
+	} else if (reject_reasons & REJECT_REF_NEEDS_UPDATE) {
+		advise_ref_needs_update();
 	}
 
 	return 1;
@@ -510,6 +525,12 @@ static int git_push_config(const char *k, const char *v, void *cb)
 		if (!v)
 			return config_error_nonbool(k);
 		return color_parse(v, push_colors[slot]);
+	} else if (!strcmp(k, "push.useforceifincludes")) {
+		if (git_config_bool(k, v))
+			*flags |= TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		else
+			*flags &= ~TRANSPORT_PUSH_FORCE_IF_INCLUDES;
+		return 0;
 	}
 
 	return git_default_config(k, v, NULL);
@@ -541,6 +562,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 			       N_("require old value of ref to be at this value"),
 			       PARSE_OPT_OPTARG | PARSE_OPT_LITERAL_ARGHELP, parseopt_push_cas_option),
+		OPT_BIT(0, TRANS_OPT_FORCE_IF_INCLUDES, &flags,
+			N_("require remote updates to be integrated locally"),
+			TRANSPORT_PUSH_FORCE_IF_INCLUDES),
 		OPT_CALLBACK(0, "recurse-submodules", &recurse_submodules, "(check|on-demand|no)",
 			     N_("control recursive pushing of submodules"), option_parse_recurse_submodules),
 		OPT_BOOL_F( 0 , "thin", &thin, N_("use thin pack"), PARSE_OPT_NOCOMPLETE),
@@ -625,6 +649,9 @@ int cmd_push(int argc, const char **argv, const char *prefix)
 	if ((flags & TRANSPORT_PUSH_ALL) && (flags & TRANSPORT_PUSH_MIRROR))
 		die(_("--all and --mirror are incompatible"));
 
+	if (!is_empty_cas(&cas) && (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES))
+		cas.use_force_if_includes = 1;
+
 	for_each_string_list_item(item, push_options)
 		if (strchr(item->string, '\n'))
 			die(_("push options must not have new line characters"));
diff --git a/builtin/send-pack.c b/builtin/send-pack.c
index 516cba7336..a7e01667b0 100644
--- a/builtin/send-pack.c
+++ b/builtin/send-pack.c
@@ -178,6 +178,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	int progress = -1;
 	int from_stdin = 0;
 	struct push_cas_option cas = {0};
+	int force_if_includes = 0;
 	struct packet_reader reader;
 
 	struct option options[] = {
@@ -203,6 +204,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, CAS_OPT_NAME, &cas, N_("<refname>:<expect>"),
 		  N_("require old value of ref to be at this value"),
 		  PARSE_OPT_OPTARG, parseopt_push_cas_option),
+		OPT_BOOL(0, TRANS_OPT_FORCE_IF_INCLUDES, &force_if_includes,
+			 N_("require remote updates to be integrated locally")),
 		OPT_END()
 	};
 
@@ -304,6 +307,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)
 	if (!is_empty_cas(&cas))
 		apply_push_cas(&cas, remote, remote_refs);
 
+	if (!is_empty_cas(&cas) && force_if_includes)
+		cas.use_force_if_includes = 1;
+
 	set_ref_status_for_push(remote_refs, args.send_mirror,
 		args.force_update);
 
diff --git a/remote-curl.c b/remote-curl.c
index 32cc4a0c55..0290b04891 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -44,7 +44,8 @@ struct options {
 		from_promisor : 1,
 
 		atomic : 1,
-		object_format : 1;
+		object_format : 1,
+		force_if_includes : 1;
 	const struct git_hash_algo *hash_algo;
 };
 static struct options options;
@@ -131,6 +132,14 @@ static int set_option(const char *name, const char *value)
 		string_list_append(&cas_options, val.buf);
 		strbuf_release(&val);
 		return 0;
+	} else if (!strcmp(name, TRANS_OPT_FORCE_IF_INCLUDES)) {
+		if (!strcmp(value, "true"))
+			options.force_if_includes = 1;
+		else if (!strcmp(value, "false"))
+			options.force_if_includes = 0;
+		else
+			return -1;
+		return 0;
 	} else if (!strcmp(name, "cloning")) {
 		if (!strcmp(value, "true"))
 			options.cloning = 1;
@@ -1318,6 +1327,9 @@ static int push_git(struct discovery *heads, int nr_spec, const char **specs)
 		strvec_push(&args, cas_option->string);
 	strvec_push(&args, url.buf);
 
+	if (options.force_if_includes)
+		strvec_push(&args, "--force-if-includes");
+
 	strvec_push(&args, "--stdin");
 	for (i = 0; i < nr_spec; i++)
 		packet_buf_write(&preamble, "%s\n", specs[i]);
diff --git a/transport-helper.c b/transport-helper.c
index 6157de30c7..5f6e0b3bd8 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -938,6 +938,11 @@ static void set_common_push_options(struct transport *transport,
 		if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
 			die(_("helper %s does not support --atomic"), name);
 
+	if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
+		if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
+			die(_("helper %s does not support --%s"),
+			    name, TRANS_OPT_FORCE_IF_INCLUDES);
+
 	if (flags & TRANSPORT_PUSH_OPTIONS) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, transport->push_options)
diff --git a/transport.c b/transport.c
index 65fcd22b20..47da955e4f 100644
--- a/transport.c
+++ b/transport.c
@@ -748,6 +748,8 @@ void transport_print_push_status(const char *dest, struct ref *refs,
 			*reject_reasons |= REJECT_FETCH_FIRST;
 		} else if (ref->status == REF_STATUS_REJECT_NEEDS_FORCE) {
 			*reject_reasons |= REJECT_NEEDS_FORCE;
+		} else if (ref->status == REF_STATUS_REJECT_REMOTE_UPDATED) {
+			*reject_reasons |= REJECT_REF_NEEDS_UPDATE;
 		}
 	}
 	free(head);
diff --git a/transport.h b/transport.h
index ca409ea1e4..24558c027d 100644
--- a/transport.h
+++ b/transport.h
@@ -136,6 +136,7 @@ struct transport {
 #define TRANSPORT_PUSH_ATOMIC			(1<<13)
 #define TRANSPORT_PUSH_OPTIONS			(1<<14)
 #define TRANSPORT_RECURSE_SUBMODULES_ONLY	(1<<15)
+#define TRANSPORT_PUSH_FORCE_IF_INCLUDES	(1<<16)
 
 int transport_summary_width(const struct ref *refs);
 
@@ -208,6 +209,9 @@ void transport_check_allowed(const char *type);
 /* Request atomic (all-or-nothing) updates when pushing */
 #define TRANS_OPT_ATOMIC "atomic"
 
+/* Require remote changes to be integrated locally. */
+#define TRANS_OPT_FORCE_IF_INCLUDES "force-if-includes"
+
 /**
  * Returns 0 if the option was used, non-zero otherwise. Prints a
  * message to stderr if the option is not used.
@@ -217,11 +221,12 @@ int transport_set_option(struct transport *transport, const char *name,
 void transport_set_verbosity(struct transport *transport, int verbosity,
 	int force_progress);
 
-#define REJECT_NON_FF_HEAD     0x01
-#define REJECT_NON_FF_OTHER    0x02
-#define REJECT_ALREADY_EXISTS  0x04
-#define REJECT_FETCH_FIRST     0x08
-#define REJECT_NEEDS_FORCE     0x10
+#define REJECT_NON_FF_HEAD      0x01
+#define REJECT_NON_FF_OTHER     0x02
+#define REJECT_ALREADY_EXISTS   0x04
+#define REJECT_FETCH_FIRST      0x08
+#define REJECT_NEEDS_FORCE      0x10
+#define REJECT_REF_NEEDS_UPDATE 0x20
 
 int transport_push(struct repository *repo,
 		   struct transport *connection,
-- 
2.28.0

[PATCH v10 3/3] t, doc: update tests, reference for "--force-if-includes"

[Srinidhi Kaushik]

Update test cases for the new option, and document its usage
and update related references.

Update test cases for the new option, and document its usage
and update related references.

 - t/t5533-push-cas.sh:
   Update test cases for "compare-and-swap" when used along with
   "--force-if-includes" helps mitigate overwrites when remote
   refs are updated in the background; allows forced updates when
   changes from remote are integrated locally.

 - Documentation:
   Add reference for the new option, configuration setting
   ("push.useForceIfIncludes") and advise messages.

Signed-off-by: Srinidhi Kaushik <shrinidhi.kaushik@gmail.com>
---
 Documentation/config/advice.txt |   9 ++-
 Documentation/config/push.txt   |   6 ++
 Documentation/git-push.txt      |  26 +++++-
 t/t5533-push-cas.sh             | 137 ++++++++++++++++++++++++++++++++
 4 files changed, 174 insertions(+), 4 deletions(-)

diff --git a/Documentation/config/advice.txt b/Documentation/config/advice.txt
index bdd37c3eaa..acbd0c09aa 100644
--- a/Documentation/config/advice.txt
+++ b/Documentation/config/advice.txt
@@ -10,9 +10,8 @@ advice.*::
 		that the check is disabled.
 	pushUpdateRejected::
 		Set this variable to 'false' if you want to disable
-		'pushNonFFCurrent',
-		'pushNonFFMatching', 'pushAlreadyExists',
-		'pushFetchFirst', and 'pushNeedsForce'
+		'pushNonFFCurrent', 'pushNonFFMatching', 'pushAlreadyExists',
+		'pushFetchFirst', 'pushNeedsForce', and 'pushRefNeedsUpdate'
 		simultaneously.
 	pushNonFFCurrent::
 		Advice shown when linkgit:git-push[1] fails due to a
@@ -41,6 +40,10 @@ advice.*::
 		we can still suggest that the user push to either
 		refs/heads/* or refs/tags/* based on the type of the
 		source object.
+	pushRefNeedsUpdate::
+		Shown when linkgit:git-push[1] rejects a forced update of
+		a branch when its remote-tracking ref has updates that we
+		do not have locally.
 	statusAheadBehind::
 		Shown when linkgit:git-status[1] computes the ahead/behind
 		counts for a local ref compared to its remote tracking ref,
diff --git a/Documentation/config/push.txt b/Documentation/config/push.txt
index f5e5b38c68..21b256e0a4 100644
--- a/Documentation/config/push.txt
+++ b/Documentation/config/push.txt
@@ -114,3 +114,9 @@ push.recurseSubmodules::
 	specifying '--recurse-submodules=check|on-demand|no'.
 	If not set, 'no' is used by default, unless 'submodule.recurse' is
 	set (in which case a 'true' value means 'on-demand').
+
+push.useForceIfIncludes::
+	If set to "true", it is equivalent to specifying
+	`--force-if-includes` as an option to linkgit:git-push[1]
+	in the command line. Adding `--no-force-if-includes` at the
+	time of push overrides this configuration setting.
diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index 3b8053447e..ab103c82cf 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -13,7 +13,7 @@ SYNOPSIS
 	   [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
 	   [-u | --set-upstream] [-o <string> | --push-option=<string>]
 	   [--[no-]signed|--signed=(true|false|if-asked)]
-	   [--force-with-lease[=<refname>[:<expect>]]]
+	   [--force-with-lease[=<refname>[:<expect>]] [--force-if-includes]]
 	   [--no-verify] [<repository> [<refspec>...]]
 
 DESCRIPTION
@@ -320,6 +320,14 @@ seen and are willing to overwrite, then rewrite history, and finally
 force push changes to `master` if the remote version is still at
 `base`, regardless of what your local `remotes/origin/master` has been
 updated to in the background.
++
+Alternatively, specifying `--force-if-includes` as an ancillary option
+along with `--force-with-lease[=<refname>]` (i.e., without saying what
+exact commit the ref on the remote side must be pointing at, or which
+refs on the remote side are being protected) at the time of "push" will
+verify if updates from the remote-tracking refs that may have been
+implicitly updated in the background are integrated locally before
+allowing a forced update.
 
 -f::
 --force::
@@ -341,6 +349,22 @@ one branch, use a `+` in front of the refspec to push (e.g `git push
 origin +master` to force a push to the `master` branch). See the
 `<refspec>...` section above for details.
 
+--[no-]force-if-includes::
+	Force an update only if the tip of the remote-tracking ref
+	has been integrated locally.
++
+This option enables a check that verifies if the tip of the
+remote-tracking ref is reachable from one of the "reflog" entries of
+the local branch based in it for a rewrite. The check ensures that any
+updates from the remote have been incorporated locally by rejecting the
+forced update if that is not the case.
++
+If the option is passed without specifying `--force-with-lease`, or
+specified along with `--force-with-lease=<refname>:<expect>`, it is
+a "no-op".
++
+Specifying `--no-force-if-includes` disables this behavior.
+
 --repo=<repository>::
 	This option is equivalent to the <repository> argument. If both
 	are specified, the command-line argument takes precedence.
diff --git a/t/t5533-push-cas.sh b/t/t5533-push-cas.sh
index 0b0eb1d025..7813e8470e 100755
--- a/t/t5533-push-cas.sh
+++ b/t/t5533-push-cas.sh
@@ -13,6 +13,46 @@ setup_srcdst_basic () {
 	)
 }
 
+# For tests with "--force-if-includes".
+setup_src_dup_dst () {
+	rm -fr src dup dst &&
+	git init --bare dst &&
+	git clone --no-local dst src &&
+	git clone --no-local dst dup
+	(
+		cd src &&
+		test_commit A &&
+		test_commit B &&
+		test_commit C &&
+		git push origin
+	) &&
+	(
+		cd dup &&
+		git fetch &&
+		git merge origin/master &&
+		git switch -c branch master~2 &&
+		test_commit D &&
+		test_commit E &&
+		git push origin --all
+	) &&
+	(
+		cd src &&
+		git switch master &&
+		git fetch --all &&
+		git branch branch --track origin/branch &&
+		git rebase origin/master
+	) &&
+	(
+		cd dup &&
+		git switch master &&
+		test_commit F &&
+		test_commit G &&
+		git switch branch &&
+		test_commit H &&
+		git push origin --all
+	)
+}
+
 test_expect_success setup '
 	# create template repository
 	test_commit A &&
@@ -256,4 +296,101 @@ test_expect_success 'background updates of REMOTE can be mitigated with a non-up
 	)
 '
 
+test_expect_success 'background updates to remote can be mitigated with "--force-if-includes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	git ls-remote dst refs/heads/branch >expect.branch &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		test_must_fail git push --force-with-lease --force-if-includes --all
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	git ls-remote dst refs/heads/branch >actual.branch &&
+	test_cmp expect.master actual.master &&
+	test_cmp expect.branch actual.branch
+'
+
+test_expect_success 'background updates to remote can be mitigated with "push.useForceIfIncludes"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git fetch --all &&
+		git config --local push.useForceIfIncludes true &&
+		test_must_fail git push --force-with-lease=master origin master
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success '"--force-if-includes" should be disabled for --force-with-lease="<refname>:<expect>"' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	git ls-remote dst refs/heads/master >expect.master &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		remote_head="$(git rev-parse refs/remotes/origin/master)" &&
+		git fetch --all &&
+		test_must_fail git push --force-if-includes --force-with-lease="master:$remote_head" 2>err &&
+		grep "stale info" err
+	) &&
+	git ls-remote dst refs/heads/master >actual.master &&
+	test_cmp expect.master actual.master
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase")' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
+test_expect_success '"--force-if-includes" should allow forced update after a rebase ("pull --rebase", local rebase)' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		test_commit I &&
+		git switch master &&
+		test_commit J &&
+		git pull --rebase origin master &&
+		git rebase --onto HEAD~4 HEAD~1 &&
+		git push --force-if-includes --force-with-lease="master"
+	)
+'
+
+test_expect_success '"--force-if-includes" should allow deletes' '
+	setup_src_dup_dst &&
+	test_when_finished "rm -fr dst src dup" &&
+	(
+		cd src &&
+		git switch branch &&
+		git pull --rebase origin branch &&
+		git push --force-if-includes --force-with-lease="branch" origin :branch
+	)
+'
+
 test_done
-- 
2.28.0