[PATCH v13 00/14] libify apply and use lib in am, part 3

[PATCH v13 00/14] libify apply and use lib in am, part 3

[Christian Couder]

Goal
~~~~

This is a patch series about libifying `git apply` functionality, and
using this libified functionality in `git am`, so that no 'git apply'
process is spawn anymore. This makes `git am` significantly faster, so
`git rebase`, when it uses the am backend, is also significantly
faster.

Previous discussions and patches series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This has initially been discussed in the following thread:

  http://thread.gmane.org/gmane.comp.version-control.git/287236/

Then the following patch series were sent:

RFC: http://thread.gmane.org/gmane.comp.version-control.git/288489/
v1: http://thread.gmane.org/gmane.comp.version-control.git/292324/
v2: http://thread.gmane.org/gmane.comp.version-control.git/294248/
v3: http://thread.gmane.org/gmane.comp.version-control.git/295429/
v4: http://thread.gmane.org/gmane.comp.version-control.git/296350/
v5: http://thread.gmane.org/gmane.comp.version-control.git/296490/
v6: http://thread.gmane.org/gmane.comp.version-control.git/297024/
v7: http://thread.gmane.org/gmane.comp.version-control.git/297193/
v8: https://public-inbox.org/git/20160627182429.31550-1-chriscool%40tuxfamily.org/
v9: https://public-inbox.org/git/20160730172509.22939-1-chriscool%40tuxfamily.org/
v10: https://public-inbox.org/git/20160808210337.5038-1-chriscool%40tuxfamily.org/
v11: https://public-inbox.org/git/20160811085229.19017-1-chriscool%40tuxfamily.org/
v12: https://public-inbox.org/git/20160811184501.384-1-chriscool@tuxfamily.org/

Highlevel view of the patches in the series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is "part 3" of the full patch series. I am resending only the
last 14 patches of the full series as "part 3", because I don't want
to resend the first 27 patches of v10 nearly as is.

(So "part 2" is made of patch 01/40 from v11 and patches from 02/40 to
27/40 from v10. And "part 1" has been in "master" for some time now.)

  - Patch 01/14 to 04/14 were v1, v2, v6, v7, v8, v9, v10, v11 and v12.

They haven't changed since v12.

They finish libifying the apply functionality that was in
builtin/apply.c and move it into apply.{c,h}, but the libified
functionality is not yet used in `git am`.

  - Patch 05/14 was in v6, v7, v8, v9, v10 and v12 and hasn't changed.

It replaces some calls to error() with calls to error_errno().

  - Patches 06/14 to 10/14 were in v2, v6, v7, v8, v9 and v10.

They haven't changed since v10.

They implement a way to make the libified apply code silent by
changing the bool `apply_verbosely` into a tristate enum called
`apply_verbosity`, that can be one of `verbosity_verbose`,
`verbosity_normal` or `verbosity_silent`.

This is because "git am", since it was a shell script, has been
silencing the apply functionality by redirecting file descriptors to
/dev/null, but this is not acceptable in C.

  - Patch 11/14 was in v9, v10 and v12, and hasn't changed.

It refactors `git apply` option parsing to make it possible for `git
am` to easily pass some command line options to the libified applied
code.

  - Patch 12/14 is new.

It is a refactoring to prepare for some new changes in patch 13/14.

  - Patch 13/14 was in v12.

It replaces patch 33/40 in v10 (environment: add set_index_file())
that was a hack to make it possible for `git am` to use the libified
apply functionality on a different index file.

For this purpose in this patch we add a "const char *index_file" into
"struct apply_state", and when "index_file" is set, we use
hold_lock_file_for_update(), passing it "state->index_file", instead
of using hold_locked_index(), as it is not possible to pass an index
filename in hold_locked_index().

This patch was changed in this version to also use
"read_cache_from(state->index_file)" instead of "read_cache()" when
state->index_file is set.

  - Patch 14/14 was in v1, v2, v6, v7, v8, v9, v10 and v12, and hasn't
    changed since v12.

This patch makes `git am` use the libified functionality.

General comments
~~~~~~~~~~~~~~~~

Now this patch series is shorter than previously. Hopefully also the
early part of this series until 05/14 or 11/14 will be ready soon to
be moved to next and master, and I may only need to resend the last 3
patches if anything.

I will send a diff between this version and v12 as a reply to this
email.

The benefits are not just related to not creating new processes. When
`git am` launched a `git apply` process, this new process had to read
the index from disk. Then after the `git apply`process had terminated,
`git am` dropped its index and read the index from disk to get the
index that had been modified by the `git apply`process. This was
inefficient and also prevented the split-index mechanism to provide
many performance benefits.

By the way, current work is ongoing to make it possible to use
split-index more easily by adding a config variable, see:

https://public-inbox.org/git/20160711172254.13439-1-chriscool%40tuxfamily.org/

Using an earlier version of this series as rebase material, Duy
explained split-index benefits along with this patch series like this:

 > Without the series, the picture is not so surprising. We run git-apply
 > 80+ times, each consists of this sequence
 >
 > read index
 > write index (cache tree updates only)
 > read index again
 > optionally initialize name hash (when new entries are added, I guess)
 > read packed-refs
 > write index
 >
 > With this series, we run a single git-apply which does
 >
 > read index (and sharedindex too if in split-index mode)
 > initialize name hash
 > write index 80+ times

(See: http://thread.gmane.org/gmane.comp.version-control.git/292324/focus=292460)

Links
~~~~~

This patch series is available here:

https://github.com/chriscool/git/commits/libify-apply-use-in-am

The previous versions are available there:

v1: https://github.com/chriscool/git/commits/libify-apply-use-in-am25 
v2: https://github.com/chriscool/git/commits/libify-apply-use-in-am54
v6: https://github.com/chriscool/git/commits/libify-apply-use-in-am65
v7: https://github.com/chriscool/git/commits/libify-apply-use-in-am75
v8: https://github.com/chriscool/git/commits/libify-apply-use-in-am97
v9: https://github.com/chriscool/git/commits/libify-apply-use-in-am106
v10: https://github.com/chriscool/git/commits/libify-apply-use-in-am114
v11: https://github.com/chriscool/git/commits/libify-apply-use-in-am116
v12: https://github.com/chriscool/git/commits/libify-apply-use-in-am118

Performance numbers
~~~~~~~~~~~~~~~~~~~

Numbers are only available for tests that have been performed on Linux
using a very early version of this series, though Johannes Sixt
reported great improvements on Windows. It could be interesting to get
detailed numbers on other platforms like Windows and OSX.

  - Around mid April Ævar did a huge many-hundred commit rebase on the
    kernel with untracked cache.

command: git rebase --onto 1993b17 52bef0c 29dde7c

Vanilla "next" without split index:                1m54.953s
Vanilla "next" with split index:                   1m22.476s
This series on top of "next" without split index:  1m12.034s
This series on top of "next" with split index:     0m15.678s

Ævar used his Debian laptop with SSD.

  - Around mid April I tested rebasing 13 commits in Booking.com's
    monorepo on a Red Hat 6.5 server with split-index and
    GIT_TRACE_PERFORMANCE=1.

With Git v2.8.0, the rebase took 6.375888383 s, with the git am
command launched by the rebase command taking 3.705677431 s.

With this series on top of next, the rebase took 3.044529494 s, with
the git am command launched by the rebase command taking 0.583521168
s.

Christian Couder (14):
  builtin/apply: rename option parsing functions
  apply: rename and move opt constants to apply.h
  Move libified code from builtin/apply.c to apply.{c,h}
  apply: make some parsing functions static again
  apply: use error_errno() where possible
  apply: make it possible to silently apply
  apply: don't print on stdout in verbosity_silent mode
  usage: add set_warn_routine()
  usage: add get_error_routine() and get_warn_routine()
  apply: change error_routine when silent
  apply: refactor `git apply` option parsing
  apply: pass apply state to build_fake_ancestor()
  apply: learn to use a different index file
  builtin/am: use apply API in run_apply()

 apply.c           | 4871 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 apply.h           |   34 +-
 builtin/am.c      |   65 +-
 builtin/apply.c   | 4814 +---------------------------------------------------
 git-compat-util.h |    3 +
 usage.c           |   15 +
 6 files changed, 4961 insertions(+), 4841 deletions(-)

-- 
2.9.2.770.g14ff7d2

[PATCH v13 01/14] builtin/apply: rename option parsing functions

[Christian Couder]

As these functions are going to be part of the libified
apply API, let's give them a name that is more specific
to the apply API.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 builtin/apply.c | 40 ++++++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/builtin/apply.c b/builtin/apply.c
index ad403f8..429fe44 100644
--- a/builtin/apply.c
+++ b/builtin/apply.c
@@ -4588,16 +4588,16 @@ static int apply_patch(struct apply_state *state,
 	return res;
 }
 
-static int option_parse_exclude(const struct option *opt,
-				const char *arg, int unset)
+static int apply_option_parse_exclude(const struct option *opt,
+				      const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	add_name_limit(state, arg, 1);
 	return 0;
 }
 
-static int option_parse_include(const struct option *opt,
-				const char *arg, int unset)
+static int apply_option_parse_include(const struct option *opt,
+				      const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	add_name_limit(state, arg, 0);
@@ -4605,9 +4605,9 @@ static int option_parse_include(const struct option *opt,
 	return 0;
 }
 
-static int option_parse_p(const struct option *opt,
-			  const char *arg,
-			  int unset)
+static int apply_option_parse_p(const struct option *opt,
+				const char *arg,
+				int unset)
 {
 	struct apply_state *state = opt->value;
 	state->p_value = atoi(arg);
@@ -4615,8 +4615,8 @@ static int option_parse_p(const struct option *opt,
 	return 0;
 }
 
-static int option_parse_space_change(const struct option *opt,
-				     const char *arg, int unset)
+static int apply_option_parse_space_change(const struct option *opt,
+					   const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	if (unset)
@@ -4626,8 +4626,8 @@ static int option_parse_space_change(const struct option *opt,
 	return 0;
 }
 
-static int option_parse_whitespace(const struct option *opt,
-				   const char *arg, int unset)
+static int apply_option_parse_whitespace(const struct option *opt,
+					 const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	state->whitespace_option = arg;
@@ -4636,8 +4636,8 @@ static int option_parse_whitespace(const struct option *opt,
 	return 0;
 }
 
-static int option_parse_directory(const struct option *opt,
-				  const char *arg, int unset)
+static int apply_option_parse_directory(const struct option *opt,
+					const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	strbuf_reset(&state->root);
@@ -4755,13 +4755,13 @@ int cmd_apply(int argc, const char **argv, const char *prefix)
 	struct option builtin_apply_options[] = {
 		{ OPTION_CALLBACK, 0, "exclude", &state, N_("path"),
 			N_("don't apply changes matching the given path"),
-			0, option_parse_exclude },
+			0, apply_option_parse_exclude },
 		{ OPTION_CALLBACK, 0, "include", &state, N_("path"),
 			N_("apply changes matching the given path"),
-			0, option_parse_include },
+			0, apply_option_parse_include },
 		{ OPTION_CALLBACK, 'p', NULL, &state, N_("num"),
 			N_("remove <num> leading slashes from traditional diff paths"),
-			0, option_parse_p },
+			0, apply_option_parse_p },
 		OPT_BOOL(0, "no-add", &state.no_add,
 			N_("ignore additions made by the patch")),
 		OPT_BOOL(0, "stat", &state.diffstat,
@@ -4793,13 +4793,13 @@ int cmd_apply(int argc, const char **argv, const char *prefix)
 				N_("ensure at least <n> lines of context match")),
 		{ OPTION_CALLBACK, 0, "whitespace", &state, N_("action"),
 			N_("detect new or modified lines that have whitespace errors"),
-			0, option_parse_whitespace },
+			0, apply_option_parse_whitespace },
 		{ OPTION_CALLBACK, 0, "ignore-space-change", &state, NULL,
 			N_("ignore changes in whitespace when finding context"),
-			PARSE_OPT_NOARG, option_parse_space_change },
+			PARSE_OPT_NOARG, apply_option_parse_space_change },
 		{ OPTION_CALLBACK, 0, "ignore-whitespace", &state, NULL,
 			N_("ignore changes in whitespace when finding context"),
-			PARSE_OPT_NOARG, option_parse_space_change },
+			PARSE_OPT_NOARG, apply_option_parse_space_change },
 		OPT_BOOL('R', "reverse", &state.apply_in_reverse,
 			N_("apply the patch in reverse")),
 		OPT_BOOL(0, "unidiff-zero", &state.unidiff_zero,
@@ -4817,7 +4817,7 @@ int cmd_apply(int argc, const char **argv, const char *prefix)
 			RECOUNT),
 		{ OPTION_CALLBACK, 0, "directory", &state, N_("root"),
 			N_("prepend <root> to all filenames"),
-			0, option_parse_directory },
+			0, apply_option_parse_directory },
 		OPT_END()
 	};
 
-- 
2.9.2.770.g14ff7d2

[PATCH v13 02/14] apply: rename and move opt constants to apply.h

[Christian Couder]

The constants for the "inaccurate-eof" and the "recount" options will
be used in both "apply.c" and "builtin/apply.c", so they need to go
into "apply.h", and therefore they need a name that is more specific
to the API they belong to.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.h         |  3 +++
 builtin/apply.c | 11 ++++-------
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/apply.h b/apply.h
index 53f09b5..ca1dcee 100644
--- a/apply.h
+++ b/apply.h
@@ -108,4 +108,7 @@ extern int init_apply_state(struct apply_state *state,
 extern void clear_apply_state(struct apply_state *state);
 extern int check_apply_state(struct apply_state *state, int force_apply);
 
+#define APPLY_OPT_INACCURATE_EOF	(1<<0)
+#define APPLY_OPT_RECOUNT		(1<<1)
+
 #endif
diff --git a/builtin/apply.c b/builtin/apply.c
index 429fe44..9c396bb 100644
--- a/builtin/apply.c
+++ b/builtin/apply.c
@@ -4463,9 +4463,6 @@ static int write_out_results(struct apply_state *state, struct patch *list)
 
 static struct lock_file lock_file;
 
-#define INACCURATE_EOF	(1<<0)
-#define RECOUNT		(1<<1)
-
 /*
  * Try to apply a patch.
  *
@@ -4495,8 +4492,8 @@ static int apply_patch(struct apply_state *state,
 		int nr;
 
 		patch = xcalloc(1, sizeof(*patch));
-		patch->inaccurate_eof = !!(options & INACCURATE_EOF);
-		patch->recount =  !!(options & RECOUNT);
+		patch->inaccurate_eof = !!(options & APPLY_OPT_INACCURATE_EOF);
+		patch->recount =  !!(options & APPLY_OPT_RECOUNT);
 		nr = parse_chunk(state, buf.buf + offset, buf.len - offset, patch);
 		if (nr < 0) {
 			free_patch(patch);
@@ -4811,10 +4808,10 @@ int cmd_apply(int argc, const char **argv, const char *prefix)
 		OPT__VERBOSE(&state.apply_verbosely, N_("be verbose")),
 		OPT_BIT(0, "inaccurate-eof", &options,
 			N_("tolerate incorrectly detected missing new-line at the end of file"),
-			INACCURATE_EOF),
+			APPLY_OPT_INACCURATE_EOF),
 		OPT_BIT(0, "recount", &options,
 			N_("do not trust the line counts in the hunk headers"),
-			RECOUNT),
+			APPLY_OPT_RECOUNT),
 		{ OPTION_CALLBACK, 0, "directory", &state, N_("root"),
 			N_("prepend <root> to all filenames"),
 			0, apply_option_parse_directory },
-- 
2.9.2.770.g14ff7d2

[PATCH v13 04/14] apply: make some parsing functions static again

[Christian Couder]

Some parsing functions that were used in both "apply.c" and
"builtin/apply.c" are now only used in the former, so they
can be made static to "apply.c".

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.c | 6 +++---
 apply.h | 5 -----
 2 files changed, 3 insertions(+), 8 deletions(-)

diff --git a/apply.c b/apply.c
index 7b96130..c0cb3f5 100644
--- a/apply.c
+++ b/apply.c
@@ -27,7 +27,7 @@ static void git_apply_config(void)
 	git_config(git_default_config, NULL);
 }
 
-int parse_whitespace_option(struct apply_state *state, const char *option)
+static int parse_whitespace_option(struct apply_state *state, const char *option)
 {
 	if (!option) {
 		state->ws_error_action = warn_on_ws_error;
@@ -57,8 +57,8 @@ int parse_whitespace_option(struct apply_state *state, const char *option)
 	return error(_("unrecognized whitespace option '%s'"), option);
 }
 
-int parse_ignorewhitespace_option(struct apply_state *state,
-				  const char *option)
+static int parse_ignorewhitespace_option(struct apply_state *state,
+						 const char *option)
 {
 	if (!option || !strcmp(option, "no") ||
 	    !strcmp(option, "false") || !strcmp(option, "never") ||
diff --git a/apply.h b/apply.h
index 5ec022c..df44b51 100644
--- a/apply.h
+++ b/apply.h
@@ -97,11 +97,6 @@ struct apply_state {
 	int applied_after_fixing_ws;
 };
 
-extern int parse_whitespace_option(struct apply_state *state,
-				   const char *option);
-extern int parse_ignorewhitespace_option(struct apply_state *state,
-					 const char *option);
-
 extern int apply_option_parse_exclude(const struct option *opt,
 				      const char *arg, int unset);
 extern int apply_option_parse_include(const struct option *opt,
-- 
2.9.2.770.g14ff7d2

[PATCH v13 05/14] apply: use error_errno() where possible

[Christian Couder]

To avoid possible mistakes and to uniformly show the errno
related messages, let's use error_errno() where possible.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/apply.c b/apply.c
index c0cb3f5..41a33d3 100644
--- a/apply.c
+++ b/apply.c
@@ -3497,7 +3497,7 @@ static int load_current(struct apply_state *state,
 	ce = active_cache[pos];
 	if (lstat(name, &st)) {
 		if (errno != ENOENT)
-			return error(_("%s: %s"), name, strerror(errno));
+			return error_errno("%s", name);
 		if (checkout_target(&the_index, ce, &st))
 			return -1;
 	}
@@ -3647,7 +3647,7 @@ static int check_preimage(struct apply_state *state,
 	} else if (!state->cached) {
 		stat_ret = lstat(old_name, st);
 		if (stat_ret && errno != ENOENT)
-			return error(_("%s: %s"), old_name, strerror(errno));
+			return error_errno("%s", old_name);
 	}
 
 	if (state->check_index && !previous) {
@@ -3669,7 +3669,7 @@ static int check_preimage(struct apply_state *state,
 	} else if (stat_ret < 0) {
 		if (patch->is_new < 0)
 			goto is_new;
-		return error(_("%s: %s"), old_name, strerror(errno));
+		return error_errno("%s", old_name);
 	}
 
 	if (!state->cached && !previous)
@@ -3728,7 +3728,7 @@ static int check_to_create(struct apply_state *state,
 
 		return EXISTS_IN_WORKTREE;
 	} else if ((errno != ENOENT) && (errno != ENOTDIR)) {
-		return error("%s: %s", new_name, strerror(errno));
+		return error_errno("%s", new_name);
 	}
 	return 0;
 }
@@ -4247,9 +4247,9 @@ static int add_index_file(struct apply_state *state,
 		if (!state->cached) {
 			if (lstat(path, &st) < 0) {
 				free(ce);
-				return error(_("unable to stat newly "
-					       "created file '%s': %s"),
-					     path, strerror(errno));
+				return error_errno(_("unable to stat newly "
+						     "created file '%s'"),
+						   path);
 			}
 			fill_stat_cache_info(ce, &st);
 		}
@@ -4306,7 +4306,7 @@ static int try_create_file(const char *path, unsigned int mode, const char *buf,
 	strbuf_release(&nbuf);
 
 	if (close(fd) < 0 && !res)
-		return error(_("closing file '%s': %s"), path, strerror(errno));
+		return error_errno(_("closing file '%s'"), path);
 
 	return res ? -1 : 0;
 }
@@ -4503,7 +4503,7 @@ static int write_out_one_reject(struct apply_state *state, struct patch *patch)
 
 	rej = fopen(namebuf, "w");
 	if (!rej)
-		return error(_("cannot open %s: %s"), namebuf, strerror(errno));
+		return error_errno(_("cannot open %s"), namebuf);
 
 	/* Normal git tools never deal with .rej, so do not pretend
 	 * this is a git patch by saying --git or giving extended
-- 
2.9.2.770.g14ff7d2

[PATCH v13 06/14] apply: make it possible to silently apply

[Christian Couder]

This changes 'int apply_verbosely' into 'enum apply_verbosity', and
changes the possible values of the variable from a bool to
a tristate.

The previous 'false' state is changed into 'verbosity_normal'.
The previous 'true' state is changed into 'verbosity_verbose'.

The new added state is 'verbosity_silent'. It should prevent
anything to be printed on both stderr and stdout.

This is needed because `git am` wants to first call apply
functionality silently, if it can then fall back on 3-way merge
in case of error.

Printing on stdout, and calls to warning() or error() are not
taken care of in this patch, as that will be done in following
patches.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.c         | 62 +++++++++++++++++++++++++++++++++++++--------------------
 apply.h         |  8 +++++++-
 builtin/apply.c |  2 +-
 3 files changed, 48 insertions(+), 24 deletions(-)

diff --git a/apply.c b/apply.c
index 41a33d3..df85cbc 100644
--- a/apply.c
+++ b/apply.c
@@ -125,8 +125,11 @@ int check_apply_state(struct apply_state *state, int force_apply)
 			return error(_("--3way outside a repository"));
 		state->check_index = 1;
 	}
-	if (state->apply_with_reject)
-		state->apply = state->apply_verbosely = 1;
+	if (state->apply_with_reject) {
+		state->apply = 1;
+		if (state->apply_verbosity == verbosity_normal)
+			state->apply_verbosity = verbosity_verbose;
+	}
 	if (!force_apply && (state->diffstat || state->numstat || state->summary || state->check || state->fake_ancestor))
 		state->apply = 0;
 	if (state->check_index && is_not_gitdir)
@@ -1620,8 +1623,9 @@ static void record_ws_error(struct apply_state *state,
 		return;
 
 	err = whitespace_error_string(result);
-	fprintf(stderr, "%s:%d: %s.\n%.*s\n",
-		state->patch_input_file, linenr, err, len, line);
+	if (state->apply_verbosity > verbosity_silent)
+		fprintf(stderr, "%s:%d: %s.\n%.*s\n",
+			state->patch_input_file, linenr, err, len, line);
 	free(err);
 }
 
@@ -1816,7 +1820,7 @@ static int parse_single_patch(struct apply_state *state,
 		return error(_("new file %s depends on old contents"), patch->new_name);
 	if (0 < patch->is_delete && newlines)
 		return error(_("deleted file %s still has contents"), patch->old_name);
-	if (!patch->is_delete && !newlines && context)
+	if (!patch->is_delete && !newlines && context && state->apply_verbosity > verbosity_silent)
 		fprintf_ln(stderr,
 			   _("** warning: "
 			     "file %s becomes empty but is not deleted"),
@@ -2911,7 +2915,7 @@ static int apply_one_fragment(struct apply_state *state,
 			/* Ignore it, we already handled it */
 			break;
 		default:
-			if (state->apply_verbosely)
+			if (state->apply_verbosity > verbosity_normal)
 				error(_("invalid start of line: '%c'"), first);
 			applied_pos = -1;
 			goto out;
@@ -3026,7 +3030,7 @@ static int apply_one_fragment(struct apply_state *state,
 				state->apply = 0;
 		}
 
-		if (state->apply_verbosely && applied_pos != pos) {
+		if (state->apply_verbosity > verbosity_normal && applied_pos != pos) {
 			int offset = applied_pos - pos;
 			if (state->apply_in_reverse)
 				offset = 0 - offset;
@@ -3041,14 +3045,14 @@ static int apply_one_fragment(struct apply_state *state,
 		 * Warn if it was necessary to reduce the number
 		 * of context lines.
 		 */
-		if ((leading != frag->leading) ||
-		    (trailing != frag->trailing))
+		if ((leading != frag->leading ||
+		     trailing != frag->trailing) && state->apply_verbosity > verbosity_silent)
 			fprintf_ln(stderr, _("Context reduced to (%ld/%ld)"
 					     " to apply fragment at %d"),
 				   leading, trailing, applied_pos+1);
 		update_image(state, img, applied_pos, &preimage, &postimage);
 	} else {
-		if (state->apply_verbosely)
+		if (state->apply_verbosity > verbosity_normal)
 			error(_("while searching for:\n%.*s"),
 			      (int)(old - oldlines), oldlines);
 	}
@@ -3539,7 +3543,8 @@ static int try_threeway(struct apply_state *state,
 		 read_blob_object(&buf, pre_sha1, patch->old_mode))
 		return error("repository lacks the necessary blob to fall back on 3-way merge.");
 
-	fprintf(stderr, "Falling back to three-way merge...\n");
+	if (state->apply_verbosity > verbosity_silent)
+		fprintf(stderr, "Falling back to three-way merge...\n");
 
 	img = strbuf_detach(&buf, &len);
 	prepare_image(&tmp_image, img, len, 1);
@@ -3569,7 +3574,9 @@ static int try_threeway(struct apply_state *state,
 	status = three_way_merge(image, patch->new_name,
 				 pre_sha1, our_sha1, post_sha1);
 	if (status < 0) {
-		fprintf(stderr, "Failed to fall back on three-way merge...\n");
+		if (state->apply_verbosity > verbosity_silent)
+			fprintf(stderr,
+				"Failed to fall back on three-way merge...\n");
 		return status;
 	}
 
@@ -3581,9 +3588,15 @@ static int try_threeway(struct apply_state *state,
 			hashcpy(patch->threeway_stage[0].hash, pre_sha1);
 		hashcpy(patch->threeway_stage[1].hash, our_sha1);
 		hashcpy(patch->threeway_stage[2].hash, post_sha1);
-		fprintf(stderr, "Applied patch to '%s' with conflicts.\n", patch->new_name);
+		if (state->apply_verbosity > verbosity_silent)
+			fprintf(stderr,
+				"Applied patch to '%s' with conflicts.\n",
+				patch->new_name);
 	} else {
-		fprintf(stderr, "Applied patch to '%s' cleanly.\n", patch->new_name);
+		if (state->apply_verbosity > verbosity_silent)
+			fprintf(stderr,
+				"Applied patch to '%s' cleanly.\n",
+				patch->new_name);
 	}
 	return 0;
 }
@@ -3956,7 +3969,7 @@ static int check_patch_list(struct apply_state *state, struct patch *patch)
 	prepare_fn_table(state, patch);
 	while (patch) {
 		int res;
-		if (state->apply_verbosely)
+		if (state->apply_verbosity > verbosity_normal)
 			say_patch_name(stderr,
 				       _("Checking patch %s..."), patch);
 		res = check_patch(state, patch);
@@ -4472,7 +4485,7 @@ static int write_out_one_reject(struct apply_state *state, struct patch *patch)
 	}
 
 	if (!cnt) {
-		if (state->apply_verbosely)
+		if (state->apply_verbosity > verbosity_normal)
 			say_patch_name(stderr,
 				       _("Applied patch %s cleanly."), patch);
 		return 0;
@@ -4489,7 +4502,8 @@ static int write_out_one_reject(struct apply_state *state, struct patch *patch)
 			    "Applying patch %%s with %d rejects...",
 			    cnt),
 		    cnt);
-	say_patch_name(stderr, sb.buf, patch);
+	if (state->apply_verbosity > verbosity_silent)
+		say_patch_name(stderr, sb.buf, patch);
 	strbuf_release(&sb);
 
 	cnt = strlen(patch->new_name);
@@ -4516,10 +4530,12 @@ static int write_out_one_reject(struct apply_state *state, struct patch *patch)
 	     frag;
 	     cnt++, frag = frag->next) {
 		if (!frag->rejected) {
-			fprintf_ln(stderr, _("Hunk #%d applied cleanly."), cnt);
+			if (state->apply_verbosity > verbosity_silent)
+				fprintf_ln(stderr, _("Hunk #%d applied cleanly."), cnt);
 			continue;
 		}
-		fprintf_ln(stderr, _("Rejected hunk #%d."), cnt);
+		if (state->apply_verbosity > verbosity_silent)
+			fprintf_ln(stderr, _("Rejected hunk #%d."), cnt);
 		fprintf(rej, "%.*s", frag->size, frag->patch);
 		if (frag->patch[frag->size-1] != '\n')
 			fputc('\n', rej);
@@ -4568,8 +4584,10 @@ static int write_out_results(struct apply_state *state, struct patch *list)
 		struct string_list_item *item;
 
 		string_list_sort(&cpath);
-		for_each_string_list_item(item, &cpath)
-			fprintf(stderr, "U %s\n", item->string);
+		if (state->apply_verbosity > verbosity_silent) {
+			for_each_string_list_item(item, &cpath)
+				fprintf(stderr, "U %s\n", item->string);
+		}
 		string_list_clear(&cpath, 0);
 
 		rerere(0);
@@ -4626,7 +4644,7 @@ static int apply_patch(struct apply_state *state,
 			listp = &patch->next;
 		}
 		else {
-			if (state->apply_verbosely)
+			if (state->apply_verbosity > verbosity_normal)
 				say_patch_name(stderr, _("Skipped patch '%s'."), patch);
 			free_patch(patch);
 			skipped_patch++;
diff --git a/apply.h b/apply.h
index df44b51..bd4eb6d 100644
--- a/apply.h
+++ b/apply.h
@@ -13,6 +13,12 @@ enum apply_ws_ignore {
 	ignore_ws_change
 };
 
+enum apply_verbosity {
+	verbosity_silent = -1,
+	verbosity_normal = 0,
+	verbosity_verbose = 1
+};
+
 /*
  * We need to keep track of how symlinks in the preimage are
  * manipulated by the patches.  A patch to add a/b/c where a/b
@@ -51,13 +57,13 @@ struct apply_state {
 	int allow_overlap;
 	int apply_in_reverse;
 	int apply_with_reject;
-	int apply_verbosely;
 	int no_add;
 	int threeway;
 	int unidiff_zero;
 	int unsafe_paths;
 
 	/* Other non boolean parameters */
+	enum apply_verbosity apply_verbosity;
 	const char *fake_ancestor;
 	const char *patch_input_file;
 	int line_termination;
diff --git a/builtin/apply.c b/builtin/apply.c
index 9c66474..7338701 100644
--- a/builtin/apply.c
+++ b/builtin/apply.c
@@ -74,7 +74,7 @@ int cmd_apply(int argc, const char **argv, const char *prefix)
 			N_("leave the rejected hunks in corresponding *.rej files")),
 		OPT_BOOL(0, "allow-overlap", &state.allow_overlap,
 			N_("allow overlapping hunks")),
-		OPT__VERBOSE(&state.apply_verbosely, N_("be verbose")),
+		OPT__VERBOSE(&state.apply_verbosity, N_("be verbose")),
 		OPT_BIT(0, "inaccurate-eof", &options,
 			N_("tolerate incorrectly detected missing new-line at the end of file"),
 			APPLY_OPT_INACCURATE_EOF),
-- 
2.9.2.770.g14ff7d2

[PATCH v13 07/14] apply: don't print on stdout in verbosity_silent mode

[Christian Couder]

When apply_verbosity is set to verbosity_silent nothing should be
printed on both stderr and stdout.

To avoid printing on stdout, we can just skip calling the following
functions:

	- stat_patch_list(),
	- numstat_patch_list(),
	- summary_patch_list().

It is safe to do that because the above functions have no side
effects other than printing:

- stat_patch_list() only computes some local values and then call
show_stats() and print_stat_summary(), those two functions only
compute local values and call printing functions,
- numstat_patch_list() also only computes local values and calls
printing functions,
- summary_patch_list() calls show_file_mode_name(), printf(),
show_rename_copy(), show_mode_change() that are only printing.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/apply.c b/apply.c
index df85cbc..ddbb0a2 100644
--- a/apply.c
+++ b/apply.c
@@ -4702,13 +4702,13 @@ static int apply_patch(struct apply_state *state,
 		goto end;
 	}
 
-	if (state->diffstat)
+	if (state->diffstat && state->apply_verbosity > verbosity_silent)
 		stat_patch_list(state, list);
 
-	if (state->numstat)
+	if (state->numstat && state->apply_verbosity > verbosity_silent)
 		numstat_patch_list(state, list);
 
-	if (state->summary)
+	if (state->summary && state->apply_verbosity > verbosity_silent)
 		summary_patch_list(list);
 
 end:
-- 
2.9.2.770.g14ff7d2

[PATCH v13 08/14] usage: add set_warn_routine()

[Christian Couder]

There are already set_die_routine() and set_error_routine(),
so let's add set_warn_routine() as this will be needed in a
following commit.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 git-compat-util.h | 1 +
 usage.c           | 5 +++++
 2 files changed, 6 insertions(+)

diff --git a/git-compat-util.h b/git-compat-util.h
index 590bfdd..c7a51f8 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -440,6 +440,7 @@ static inline int const_error(void)
 
 extern void set_die_routine(NORETURN_PTR void (*routine)(const char *err, va_list params));
 extern void set_error_routine(void (*routine)(const char *err, va_list params));
+extern void set_warn_routine(void (*routine)(const char *warn, va_list params));
 extern void set_die_is_recursing_routine(int (*routine)(void));
 extern void set_error_handle(FILE *);
 
diff --git a/usage.c b/usage.c
index 1dad03f..67e5526 100644
--- a/usage.c
+++ b/usage.c
@@ -70,6 +70,11 @@ void set_error_routine(void (*routine)(const char *err, va_list params))
 	error_routine = routine;
 }
 
+void set_warn_routine(void (*routine)(const char *warn, va_list params))
+{
+	warn_routine = routine;
+}
+
 void set_die_is_recursing_routine(int (*routine)(void))
 {
 	die_is_recursing = routine;
-- 
2.9.2.770.g14ff7d2

[PATCH v13 09/14] usage: add get_error_routine() and get_warn_routine()

[Christian Couder]

Let's make it possible to get the current error_routine and warn_routine,
so that we can store them before using set_error_routine() or
set_warn_routine() to use new ones.

This way we will be able put back the original routines, when we are done
with using new ones.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 git-compat-util.h |  2 ++
 usage.c           | 10 ++++++++++
 2 files changed, 12 insertions(+)

diff --git a/git-compat-util.h b/git-compat-util.h
index c7a51f8..de04df1 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -440,7 +440,9 @@ static inline int const_error(void)
 
 extern void set_die_routine(NORETURN_PTR void (*routine)(const char *err, va_list params));
 extern void set_error_routine(void (*routine)(const char *err, va_list params));
+extern void (*get_error_routine(void))(const char *err, va_list params);
 extern void set_warn_routine(void (*routine)(const char *warn, va_list params));
+extern void (*get_warn_routine(void))(const char *warn, va_list params);
 extern void set_die_is_recursing_routine(int (*routine)(void));
 extern void set_error_handle(FILE *);
 
diff --git a/usage.c b/usage.c
index 67e5526..2fd3045 100644
--- a/usage.c
+++ b/usage.c
@@ -70,11 +70,21 @@ void set_error_routine(void (*routine)(const char *err, va_list params))
 	error_routine = routine;
 }
 
+void (*get_error_routine(void))(const char *err, va_list params)
+{
+	return error_routine;
+}
+
 void set_warn_routine(void (*routine)(const char *warn, va_list params))
 {
 	warn_routine = routine;
 }
 
+void (*get_warn_routine(void))(const char *warn, va_list params)
+{
+	return warn_routine;
+}
+
 void set_die_is_recursing_routine(int (*routine)(void))
 {
 	die_is_recursing = routine;
-- 
2.9.2.770.g14ff7d2

[PATCH v13 10/14] apply: change error_routine when silent

[Christian Couder]

To avoid printing anything when applying with
`state->apply_verbosity == verbosity_silent`, let's save the
existing warn and error routines before applying, and let's
replace them with a routine that does nothing.

Then after applying, let's restore the saved routines.

Helped-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.c | 21 ++++++++++++++++++++-
 apply.h |  8 ++++++++
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/apply.c b/apply.c
index ddbb0a2..bf81b70 100644
--- a/apply.c
+++ b/apply.c
@@ -112,6 +112,11 @@ void clear_apply_state(struct apply_state *state)
 	/* &state->fn_table is cleared at the end of apply_patch() */
 }
 
+static void mute_routine(const char *bla, va_list params)
+{
+	/* do nothing */
+}
+
 int check_apply_state(struct apply_state *state, int force_apply)
 {
 	int is_not_gitdir = !startup_info->have_repository;
@@ -144,6 +149,13 @@ int check_apply_state(struct apply_state *state, int force_apply)
 	if (!state->lock_file)
 		return error("BUG: state->lock_file should not be NULL");
 
+	if (state->apply_verbosity <= verbosity_silent) {
+		state->saved_error_routine = get_error_routine();
+		state->saved_warn_routine = get_warn_routine();
+		set_error_routine(mute_routine);
+		set_warn_routine(mute_routine);
+	}
+
 	return 0;
 }
 
@@ -4864,7 +4876,7 @@ int apply_all_patches(struct apply_state *state,
 		state->newfd = -1;
 	}
 
-	return !!errs;
+	res = !!errs;
 
 end:
 	if (state->newfd >= 0) {
@@ -4872,5 +4884,12 @@ int apply_all_patches(struct apply_state *state,
 		state->newfd = -1;
 	}
 
+	if (state->apply_verbosity <= verbosity_silent) {
+		set_error_routine(state->saved_error_routine);
+		set_warn_routine(state->saved_warn_routine);
+	}
+
+	if (res > -1)
+		return res;
 	return (res == -1 ? 1 : 128);
 }
diff --git a/apply.h b/apply.h
index bd4eb6d..5cde641 100644
--- a/apply.h
+++ b/apply.h
@@ -94,6 +94,14 @@ struct apply_state {
 	 */
 	struct string_list fn_table;
 
+	/*
+	 * This is to save reporting routines before using
+	 * set_error_routine() or set_warn_routine() to install muting
+	 * routines when in verbosity_silent mode.
+	 */
+	void (*saved_error_routine)(const char *err, va_list params);
+	void (*saved_warn_routine)(const char *warn, va_list params);
+
 	/* These control whitespace errors */
 	enum apply_ws_error_action ws_error_action;
 	enum apply_ws_ignore ws_ignore_action;
-- 
2.9.2.770.g14ff7d2

[PATCH v13 11/14] apply: refactor `git apply` option parsing

[Christian Couder]

Parsing `git apply` options can be useful to other commands that
want to call the libified apply functionality, because this way
they can easily pass some options from their own command line to
the libified apply functionality.

This will be used by `git am` in a following patch.

To make this possible, let's refactor the `git apply` option
parsing code into a new libified apply_parse_options() function.

Doing that makes it possible to remove some functions definitions
from "apply.h" and make them static in "apply.c".

Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.c         | 103 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 apply.h         |  18 +++-------
 builtin/apply.c |  74 ++--------------------------------------
 3 files changed, 97 insertions(+), 98 deletions(-)

diff --git a/apply.c b/apply.c
index bf81b70..2ec2a8a 100644
--- a/apply.c
+++ b/apply.c
@@ -4730,16 +4730,16 @@ static int apply_patch(struct apply_state *state,
 	return res;
 }
 
-int apply_option_parse_exclude(const struct option *opt,
-			       const char *arg, int unset)
+static int apply_option_parse_exclude(const struct option *opt,
+				      const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	add_name_limit(state, arg, 1);
 	return 0;
 }
 
-int apply_option_parse_include(const struct option *opt,
-			       const char *arg, int unset)
+static int apply_option_parse_include(const struct option *opt,
+				      const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	add_name_limit(state, arg, 0);
@@ -4747,9 +4747,9 @@ int apply_option_parse_include(const struct option *opt,
 	return 0;
 }
 
-int apply_option_parse_p(const struct option *opt,
-			 const char *arg,
-			 int unset)
+static int apply_option_parse_p(const struct option *opt,
+				const char *arg,
+				int unset)
 {
 	struct apply_state *state = opt->value;
 	state->p_value = atoi(arg);
@@ -4757,8 +4757,8 @@ int apply_option_parse_p(const struct option *opt,
 	return 0;
 }
 
-int apply_option_parse_space_change(const struct option *opt,
-				    const char *arg, int unset)
+static int apply_option_parse_space_change(const struct option *opt,
+					   const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	if (unset)
@@ -4768,8 +4768,8 @@ int apply_option_parse_space_change(const struct option *opt,
 	return 0;
 }
 
-int apply_option_parse_whitespace(const struct option *opt,
-				  const char *arg, int unset)
+static int apply_option_parse_whitespace(const struct option *opt,
+					 const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	state->whitespace_option = arg;
@@ -4778,8 +4778,8 @@ int apply_option_parse_whitespace(const struct option *opt,
 	return 0;
 }
 
-int apply_option_parse_directory(const struct option *opt,
-				 const char *arg, int unset)
+static int apply_option_parse_directory(const struct option *opt,
+					const char *arg, int unset)
 {
 	struct apply_state *state = opt->value;
 	strbuf_reset(&state->root);
@@ -4893,3 +4893,80 @@ int apply_all_patches(struct apply_state *state,
 		return res;
 	return (res == -1 ? 1 : 128);
 }
+
+int apply_parse_options(int argc, const char **argv,
+			struct apply_state *state,
+			int *force_apply, int *options,
+			const char * const *apply_usage)
+{
+	struct option builtin_apply_options[] = {
+		{ OPTION_CALLBACK, 0, "exclude", state, N_("path"),
+			N_("don't apply changes matching the given path"),
+			0, apply_option_parse_exclude },
+		{ OPTION_CALLBACK, 0, "include", state, N_("path"),
+			N_("apply changes matching the given path"),
+			0, apply_option_parse_include },
+		{ OPTION_CALLBACK, 'p', NULL, state, N_("num"),
+			N_("remove <num> leading slashes from traditional diff paths"),
+			0, apply_option_parse_p },
+		OPT_BOOL(0, "no-add", &state->no_add,
+			N_("ignore additions made by the patch")),
+		OPT_BOOL(0, "stat", &state->diffstat,
+			N_("instead of applying the patch, output diffstat for the input")),
+		OPT_NOOP_NOARG(0, "allow-binary-replacement"),
+		OPT_NOOP_NOARG(0, "binary"),
+		OPT_BOOL(0, "numstat", &state->numstat,
+			N_("show number of added and deleted lines in decimal notation")),
+		OPT_BOOL(0, "summary", &state->summary,
+			N_("instead of applying the patch, output a summary for the input")),
+		OPT_BOOL(0, "check", &state->check,
+			N_("instead of applying the patch, see if the patch is applicable")),
+		OPT_BOOL(0, "index", &state->check_index,
+			N_("make sure the patch is applicable to the current index")),
+		OPT_BOOL(0, "cached", &state->cached,
+			N_("apply a patch without touching the working tree")),
+		OPT_BOOL(0, "unsafe-paths", &state->unsafe_paths,
+			N_("accept a patch that touches outside the working area")),
+		OPT_BOOL(0, "apply", force_apply,
+			N_("also apply the patch (use with --stat/--summary/--check)")),
+		OPT_BOOL('3', "3way", &state->threeway,
+			 N_( "attempt three-way merge if a patch does not apply")),
+		OPT_FILENAME(0, "build-fake-ancestor", &state->fake_ancestor,
+			N_("build a temporary index based on embedded index information")),
+		/* Think twice before adding "--nul" synonym to this */
+		OPT_SET_INT('z', NULL, &state->line_termination,
+			N_("paths are separated with NUL character"), '\0'),
+		OPT_INTEGER('C', NULL, &state->p_context,
+				N_("ensure at least <n> lines of context match")),
+		{ OPTION_CALLBACK, 0, "whitespace", state, N_("action"),
+			N_("detect new or modified lines that have whitespace errors"),
+			0, apply_option_parse_whitespace },
+		{ OPTION_CALLBACK, 0, "ignore-space-change", state, NULL,
+			N_("ignore changes in whitespace when finding context"),
+			PARSE_OPT_NOARG, apply_option_parse_space_change },
+		{ OPTION_CALLBACK, 0, "ignore-whitespace", state, NULL,
+			N_("ignore changes in whitespace when finding context"),
+			PARSE_OPT_NOARG, apply_option_parse_space_change },
+		OPT_BOOL('R', "reverse", &state->apply_in_reverse,
+			N_("apply the patch in reverse")),
+		OPT_BOOL(0, "unidiff-zero", &state->unidiff_zero,
+			N_("don't expect at least one line of context")),
+		OPT_BOOL(0, "reject", &state->apply_with_reject,
+			N_("leave the rejected hunks in corresponding *.rej files")),
+		OPT_BOOL(0, "allow-overlap", &state->allow_overlap,
+			N_("allow overlapping hunks")),
+		OPT__VERBOSE(&state->apply_verbosity, N_("be verbose")),
+		OPT_BIT(0, "inaccurate-eof", options,
+			N_("tolerate incorrectly detected missing new-line at the end of file"),
+			APPLY_OPT_INACCURATE_EOF),
+		OPT_BIT(0, "recount", options,
+			N_("do not trust the line counts in the hunk headers"),
+			APPLY_OPT_RECOUNT),
+		{ OPTION_CALLBACK, 0, "directory", state, N_("root"),
+			N_("prepend <root> to all filenames"),
+			0, apply_option_parse_directory },
+		OPT_END()
+	};
+
+	return parse_options(argc, argv, state->prefix, builtin_apply_options, apply_usage, 0);
+}
diff --git a/apply.h b/apply.h
index 5cde641..e2b89e8 100644
--- a/apply.h
+++ b/apply.h
@@ -111,20 +111,10 @@ struct apply_state {
 	int applied_after_fixing_ws;
 };
 
-extern int apply_option_parse_exclude(const struct option *opt,
-				      const char *arg, int unset);
-extern int apply_option_parse_include(const struct option *opt,
-				      const char *arg, int unset);
-extern int apply_option_parse_p(const struct option *opt,
-				const char *arg,
-				int unset);
-extern int apply_option_parse_whitespace(const struct option *opt,
-					 const char *arg, int unset);
-extern int apply_option_parse_directory(const struct option *opt,
-					const char *arg, int unset);
-extern int apply_option_parse_space_change(const struct option *opt,
-					   const char *arg, int unset);
-
+extern int apply_parse_options(int argc, const char **argv,
+			       struct apply_state *state,
+			       int *force_apply, int *options,
+			       const char * const *apply_usage);
 extern int init_apply_state(struct apply_state *state,
 			    const char *prefix,
 			    struct lock_file *lock_file);
diff --git a/builtin/apply.c b/builtin/apply.c
index 7338701..81b9a61 100644
--- a/builtin/apply.c
+++ b/builtin/apply.c
@@ -18,80 +18,12 @@ int cmd_apply(int argc, const char **argv, const char *prefix)
 	int ret;
 	struct apply_state state;
 
-	struct option builtin_apply_options[] = {
-		{ OPTION_CALLBACK, 0, "exclude", &state, N_("path"),
-			N_("don't apply changes matching the given path"),
-			0, apply_option_parse_exclude },
-		{ OPTION_CALLBACK, 0, "include", &state, N_("path"),
-			N_("apply changes matching the given path"),
-			0, apply_option_parse_include },
-		{ OPTION_CALLBACK, 'p', NULL, &state, N_("num"),
-			N_("remove <num> leading slashes from traditional diff paths"),
-			0, apply_option_parse_p },
-		OPT_BOOL(0, "no-add", &state.no_add,
-			N_("ignore additions made by the patch")),
-		OPT_BOOL(0, "stat", &state.diffstat,
-			N_("instead of applying the patch, output diffstat for the input")),
-		OPT_NOOP_NOARG(0, "allow-binary-replacement"),
-		OPT_NOOP_NOARG(0, "binary"),
-		OPT_BOOL(0, "numstat", &state.numstat,
-			N_("show number of added and deleted lines in decimal notation")),
-		OPT_BOOL(0, "summary", &state.summary,
-			N_("instead of applying the patch, output a summary for the input")),
-		OPT_BOOL(0, "check", &state.check,
-			N_("instead of applying the patch, see if the patch is applicable")),
-		OPT_BOOL(0, "index", &state.check_index,
-			N_("make sure the patch is applicable to the current index")),
-		OPT_BOOL(0, "cached", &state.cached,
-			N_("apply a patch without touching the working tree")),
-		OPT_BOOL(0, "unsafe-paths", &state.unsafe_paths,
-			N_("accept a patch that touches outside the working area")),
-		OPT_BOOL(0, "apply", &force_apply,
-			N_("also apply the patch (use with --stat/--summary/--check)")),
-		OPT_BOOL('3', "3way", &state.threeway,
-			 N_( "attempt three-way merge if a patch does not apply")),
-		OPT_FILENAME(0, "build-fake-ancestor", &state.fake_ancestor,
-			N_("build a temporary index based on embedded index information")),
-		/* Think twice before adding "--nul" synonym to this */
-		OPT_SET_INT('z', NULL, &state.line_termination,
-			N_("paths are separated with NUL character"), '\0'),
-		OPT_INTEGER('C', NULL, &state.p_context,
-				N_("ensure at least <n> lines of context match")),
-		{ OPTION_CALLBACK, 0, "whitespace", &state, N_("action"),
-			N_("detect new or modified lines that have whitespace errors"),
-			0, apply_option_parse_whitespace },
-		{ OPTION_CALLBACK, 0, "ignore-space-change", &state, NULL,
-			N_("ignore changes in whitespace when finding context"),
-			PARSE_OPT_NOARG, apply_option_parse_space_change },
-		{ OPTION_CALLBACK, 0, "ignore-whitespace", &state, NULL,
-			N_("ignore changes in whitespace when finding context"),
-			PARSE_OPT_NOARG, apply_option_parse_space_change },
-		OPT_BOOL('R', "reverse", &state.apply_in_reverse,
-			N_("apply the patch in reverse")),
-		OPT_BOOL(0, "unidiff-zero", &state.unidiff_zero,
-			N_("don't expect at least one line of context")),
-		OPT_BOOL(0, "reject", &state.apply_with_reject,
-			N_("leave the rejected hunks in corresponding *.rej files")),
-		OPT_BOOL(0, "allow-overlap", &state.allow_overlap,
-			N_("allow overlapping hunks")),
-		OPT__VERBOSE(&state.apply_verbosity, N_("be verbose")),
-		OPT_BIT(0, "inaccurate-eof", &options,
-			N_("tolerate incorrectly detected missing new-line at the end of file"),
-			APPLY_OPT_INACCURATE_EOF),
-		OPT_BIT(0, "recount", &options,
-			N_("do not trust the line counts in the hunk headers"),
-			APPLY_OPT_RECOUNT),
-		{ OPTION_CALLBACK, 0, "directory", &state, N_("root"),
-			N_("prepend <root> to all filenames"),
-			0, apply_option_parse_directory },
-		OPT_END()
-	};
-
 	if (init_apply_state(&state, prefix, &lock_file))
 		exit(128);
 
-	argc = parse_options(argc, argv, state.prefix, builtin_apply_options,
-			apply_usage, 0);
+	argc = apply_parse_options(argc, argv,
+				   &state, &force_apply, &options,
+				   apply_usage);
 
 	if (check_apply_state(&state, force_apply))
 		exit(128);
-- 
2.9.2.770.g14ff7d2

[PATCH v13 12/14] apply: pass apply state to build_fake_ancestor()

[Christian Couder]

To libify git apply functionality, we will need to read from a
different index file in get_current_sha1(). This index file will be
stored in "struct apply_state", so let's pass the state to
build_fake_ancestor() which will later pass it to get_current_sha1().

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/apply.c b/apply.c
index 2ec2a8a..276e4af 100644
--- a/apply.c
+++ b/apply.c
@@ -4042,7 +4042,7 @@ static int preimage_sha1_in_gitlink_patch(struct patch *p, unsigned char sha1[20
 }
 
 /* Build an index that contains the just the files needed for a 3way merge */
-static int build_fake_ancestor(struct patch *list, const char *filename)
+static int build_fake_ancestor(struct apply_state *state, struct patch *list)
 {
 	struct patch *patch;
 	struct index_state result = { NULL };
@@ -4089,12 +4089,13 @@ static int build_fake_ancestor(struct patch *list, const char *filename)
 		}
 	}
 
-	hold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);
+	hold_lock_file_for_update(&lock, state->fake_ancestor, LOCK_DIE_ON_ERROR);
 	res = write_locked_index(&result, &lock, COMMIT_LOCK);
 	discard_index(&result);
 
 	if (res)
-		return error("Could not write temporary index to %s", filename);
+		return error("Could not write temporary index to %s",
+			     state->fake_ancestor);
 
 	return 0;
 }
@@ -4709,7 +4710,7 @@ static int apply_patch(struct apply_state *state,
 	}
 
 	if (state->fake_ancestor &&
-	    build_fake_ancestor(list, state->fake_ancestor)) {
+	    build_fake_ancestor(state, list)) {
 		res = -128;
 		goto end;
 	}
-- 
2.9.2.770.g14ff7d2

[PATCH v13 13/14] apply: learn to use a different index file

[Christian Couder]

Sometimes we want to apply in a different index file.

Before the apply functionality was libified it was possible to
use the GIT_INDEX_FILE environment variable, for this purpose.

But now, as the apply functionality has been libified, it should
be possible to do that in a libified way.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 apply.c | 27 +++++++++++++++++++++------
 apply.h |  1 +
 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/apply.c b/apply.c
index 276e4af..5cd037d 100644
--- a/apply.c
+++ b/apply.c
@@ -3993,12 +3993,21 @@ static int check_patch_list(struct apply_state *state, struct patch *patch)
 	return err;
 }
 
+static int read_apply_cache(struct apply_state *state)
+{
+	if (state->index_file)
+		return read_cache_from(state->index_file);
+	else
+		return read_cache();
+}
+
 /* This function tries to read the sha1 from the current index */
-static int get_current_sha1(const char *path, unsigned char *sha1)
+static int get_current_sha1(struct apply_state *state, const char *path,
+			    unsigned char *sha1)
 {
 	int pos;
 
-	if (read_cache() < 0)
+	if (read_apply_cache(state) < 0)
 		return -1;
 	pos = cache_name_pos(path, strlen(path));
 	if (pos < 0)
@@ -4071,7 +4080,7 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)
 			; /* ok */
 		} else if (!patch->lines_added && !patch->lines_deleted) {
 			/* mode-only change: update the current */
-			if (get_current_sha1(patch->old_name, sha1))
+			if (get_current_sha1(state, patch->old_name, sha1))
 				return error("mode change for %s, which is not "
 					     "in current HEAD", name);
 		} else
@@ -4675,10 +4684,16 @@ static int apply_patch(struct apply_state *state,
 		state->apply = 0;
 
 	state->update_index = state->check_index && state->apply;
-	if (state->update_index && state->newfd < 0)
-		state->newfd = hold_locked_index(state->lock_file, 1);
+	if (state->update_index && state->newfd < 0) {
+		if (state->index_file)
+			state->newfd = hold_lock_file_for_update(state->lock_file,
+								 state->index_file,
+								 LOCK_DIE_ON_ERROR);
+		else
+			state->newfd = hold_locked_index(state->lock_file, 1);
+	}
 
-	if (state->check_index && read_cache() < 0) {
+	if (state->check_index && read_apply_cache(state) < 0) {
 		error(_("unable to read index file"));
 		res = -128;
 		goto end;
diff --git a/apply.h b/apply.h
index e2b89e8..1ba4f8d 100644
--- a/apply.h
+++ b/apply.h
@@ -63,6 +63,7 @@ struct apply_state {
 	int unsafe_paths;
 
 	/* Other non boolean parameters */
+	const char *index_file;
 	enum apply_verbosity apply_verbosity;
 	const char *fake_ancestor;
 	const char *patch_input_file;
-- 
2.9.2.770.g14ff7d2

[PATCH v13 14/14] builtin/am: use apply API in run_apply()

[Christian Couder]

This replaces run_apply() implementation with a new one that
uses the apply API that has been previously prepared in
apply.c and apply.h.

This shoud improve performance a lot in certain cases.

As the previous implementation was creating a new `git apply`
process to apply each patch, it could be slow on systems like
Windows where it is costly to create new processes.

Also the new `git apply` process had to read the index from
disk, and when the process was done the calling process
discarded its own index and read back from disk the new
index that had been created by the `git apply` process.

This could be very inefficient with big repositories that
have big index files, especially when the system decided
that it was a good idea to run the `git apply` processes on
a different processor core.

Also eliminating index reads enables further performance
improvements by using:

`git update-index --split-index`

For example here is a benchmark of a multi hundred commit
rebase on the Linux kernel on a Debian laptop with SSD:

command: git rebase --onto 1993b17 52bef0c 29dde7c

Vanilla "next" without split index:                1m54.953s
Vanilla "next" with split index:                   1m22.476s
This series on top of "next" without split index:  1m12.034s
This series on top of "next" with split index:     0m15.678s

(using branch "next" from mid April 2016.)

Benchmarked-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 builtin/am.c | 65 ++++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 43 insertions(+), 22 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index 739b34d..0e5d384 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -28,6 +28,7 @@
 #include "rerere.h"
 #include "prompt.h"
 #include "mailinfo.h"
+#include "apply.h"
 
 /**
  * Returns 1 if the file is empty or does not exist, 0 otherwise.
@@ -1522,39 +1523,59 @@ static int parse_mail_rebase(struct am_state *state, const char *mail)
  */
 static int run_apply(const struct am_state *state, const char *index_file)
 {
-	struct child_process cp = CHILD_PROCESS_INIT;
-
-	cp.git_cmd = 1;
-
-	if (index_file)
-		argv_array_pushf(&cp.env_array, "GIT_INDEX_FILE=%s", index_file);
+	struct argv_array apply_paths = ARGV_ARRAY_INIT;
+	struct argv_array apply_opts = ARGV_ARRAY_INIT;
+	struct apply_state apply_state;
+	int res, opts_left;
+	static struct lock_file lock_file;
+	int force_apply = 0;
+	int options = 0;
+
+	if (init_apply_state(&apply_state, NULL, &lock_file))
+		die("BUG: init_apply_state() failed");
+
+	argv_array_push(&apply_opts, "apply");
+	argv_array_pushv(&apply_opts, state->git_apply_opts.argv);
+
+	opts_left = apply_parse_options(apply_opts.argc, apply_opts.argv,
+					&apply_state, &force_apply, &options,
+					NULL);
+
+	if (opts_left != 0)
+		die("unknown option passed thru to git apply");
+
+	if (index_file) {
+		apply_state.index_file = index_file;
+		apply_state.cached = 1;
+	} else
+		apply_state.check_index = 1;
 
 	/*
 	 * If we are allowed to fall back on 3-way merge, don't give false
 	 * errors during the initial attempt.
 	 */
-	if (state->threeway && !index_file) {
-		cp.no_stdout = 1;
-		cp.no_stderr = 1;
-	}
+	if (state->threeway && !index_file)
+		apply_state.apply_verbosity = verbosity_silent;
 
-	argv_array_push(&cp.args, "apply");
+	if (check_apply_state(&apply_state, force_apply))
+		die("BUG: check_apply_state() failed");
 
-	argv_array_pushv(&cp.args, state->git_apply_opts.argv);
+	argv_array_push(&apply_paths, am_path(state, "patch"));
 
-	if (index_file)
-		argv_array_push(&cp.args, "--cached");
-	else
-		argv_array_push(&cp.args, "--index");
+	res = apply_all_patches(&apply_state, apply_paths.argc, apply_paths.argv, options);
 
-	argv_array_push(&cp.args, am_path(state, "patch"));
+	argv_array_clear(&apply_paths);
+	argv_array_clear(&apply_opts);
+	clear_apply_state(&apply_state);
 
-	if (run_command(&cp))
-		return -1;
+	if (res)
+		return res;
 
-	/* Reload index as git-apply will have modified it. */
-	discard_cache();
-	read_cache_from(index_file ? index_file : get_index_file());
+	if (index_file) {
+		/* Reload index as apply_all_patches() will have modified it. */
+		discard_cache();
+		read_cache_from(index_file);
+	}
 
 	return 0;
 }
-- 
2.9.2.770.g14ff7d2

Re: [PATCH v13 00/14] libify apply and use lib in am, part 3

[Christian Couder]

On Sat, Aug 27, 2016 at 8:45 PM, Christian Couder
<christian.couder@gmail.com> wrote:
>
> I will send a diff between this version and v12 as a reply to this
> email.

Here is the diff:

diff --git a/apply.c b/apply.c
index 7e561a4..5cd037d 100644
--- a/apply.c
+++ b/apply.c
@@ -3993,12 +3993,21 @@ static int check_patch_list(struct apply_state
*state, struct patch *patch)
     return err;
 }

+static int read_apply_cache(struct apply_state *state)
+{
+    if (state->index_file)
+        return read_cache_from(state->index_file);
+    else
+        return read_cache();
+}
+
 /* This function tries to read the sha1 from the current index */
-static int get_current_sha1(const char *path, unsigned char *sha1)
+static int get_current_sha1(struct apply_state *state, const char *path,
+                unsigned char *sha1)
 {
     int pos;

-    if (read_cache() < 0)
+    if (read_apply_cache(state) < 0)
         return -1;
     pos = cache_name_pos(path, strlen(path));
     if (pos < 0)
@@ -4042,7 +4051,7 @@ static int preimage_sha1_in_gitlink_patch(struct
patch *p, unsigned char sha1[20
 }

 /* Build an index that contains the just the files needed for a 3way merge */
-static int build_fake_ancestor(struct patch *list, const char *filename)
+static int build_fake_ancestor(struct apply_state *state, struct patch *list)
 {
     struct patch *patch;
     struct index_state result = { NULL };
@@ -4071,7 +4080,7 @@ static int build_fake_ancestor(struct patch
*list, const char *filename)
             ; /* ok */
         } else if (!patch->lines_added && !patch->lines_deleted) {
             /* mode-only change: update the current */
-            if (get_current_sha1(patch->old_name, sha1))
+            if (get_current_sha1(state, patch->old_name, sha1))
                 return error("mode change for %s, which is not "
                          "in current HEAD", name);
         } else
@@ -4089,12 +4098,13 @@ static int build_fake_ancestor(struct patch
*list, const char *filename)
         }
     }

-    hold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);
+    hold_lock_file_for_update(&lock, state->fake_ancestor, LOCK_DIE_ON_ERROR);
     res = write_locked_index(&result, &lock, COMMIT_LOCK);
     discard_index(&result);

     if (res)
-        return error("Could not write temporary index to %s", filename);
+        return error("Could not write temporary index to %s",
+                 state->fake_ancestor);

     return 0;
 }
@@ -4683,7 +4693,7 @@ static int apply_patch(struct apply_state *state,
             state->newfd = hold_locked_index(state->lock_file, 1);
     }

-    if (state->check_index && read_cache() < 0) {
+    if (state->check_index && read_apply_cache(state) < 0) {
         error(_("unable to read index file"));
         res = -128;
         goto end;
@@ -4715,7 +4725,7 @@ static int apply_patch(struct apply_state *state,
     }

     if (state->fake_ancestor &&
-        build_fake_ancestor(list, state->fake_ancestor)) {
+        build_fake_ancestor(state, list)) {
         res = -128;
         goto end;
     }

Re: [PATCH v13 00/14] libify apply and use lib in am, part 3

[Junio C Hamano]

Christian Couder <christian.couder@gmail.com> writes:

> Highlevel view of the patches in the series
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> This is "part 3" of the full patch series. I am resending only the
> last 14 patches of the full series as "part 3", because I don't want
> to resend the first 27 patches of v10 nearly as is.

Just to make sure, you are sending the first 11 of these 14 exactly
as-is, right?  I didn't spot anything different other than 12 and 13
that replaced the "alternate index" step from the previous round.

Re: [PATCH v13 00/14] libify apply and use lib in am, part 3

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Christian Couder <christian.couder@gmail.com> writes:
>
>> Highlevel view of the patches in the series
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> This is "part 3" of the full patch series. I am resending only the
>> last 14 patches of the full series as "part 3", because I don't want
>> to resend the first 27 patches of v10 nearly as is.
>
> Just to make sure, you are sending the first 11 of these 14 exactly
> as-is, right?  I didn't spot anything different other than 12 and 13
> that replaced the "alternate index" step from the previous round.

In any case, I didn't spot anything glaringly wrong in the updated
part of the patches.  I'll wait for a few days and then mark it as
'Will merge to next' unless we hear comments from others.

Thanks.

Re: [PATCH v13 00/14] libify apply and use lib in am, part 3

[Christian Couder]

On Mon, Aug 29, 2016 at 9:04 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Christian Couder <christian.couder@gmail.com> writes:
>
>> Highlevel view of the patches in the series
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> This is "part 3" of the full patch series. I am resending only the
>> last 14 patches of the full series as "part 3", because I don't want
>> to resend the first 27 patches of v10 nearly as is.
>
> Just to make sure, you are sending the first 11 of these 14 exactly
> as-is, right?  I didn't spot anything different other than 12 and 13
> that replaced the "alternate index" step from the previous round.

Yeah, the first 11 of the 14 patches have no change compared to v12.
I didn't want to create a "part 4" as that could be confusing, and
sending the first 11 patches gives them another chance to be reviewed
again.

Re: [PATCH v13 02/14] apply: rename and move opt constants to apply.h

[Stefan Beller]

On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
<christian.couder@gmail.com> wrote:
>  extern int check_apply_state(struct apply_state *state, int force_apply);
>

With greater scope comes greater responsibility. Nit of the day:
In case a reroll is necessary, consider putting a comment here.
(What are these constants? what do they control? How/where do I use them?)

> +#define APPLY_OPT_INACCURATE_EOF       (1<<0)
> +#define APPLY_OPT_RECOUNT              (1<<1)
> +

Re: [PATCH v13 03/14] Move libified code from builtin/apply.c to apply.{c,h}

[Stefan Beller]

On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
<christian.couder@gmail.com> wrote:
> As most of the apply code in builtin/apply.c has been libified by a number of
> previous commits, it can now be moved to apply.{c,h}, so that more code can
> use it.
>
> Helped-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
> ---
>  apply.c         | 4731 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  apply.h         |   19 +
>  builtin/apply.c | 4733 +------------------------------------------------------

I deduce by roughly the same line count in the .c files, it is just
moving files over.
(On my todo list I have an idea how to make reviewing patches like this easier.)

>  3 files changed, 4751 insertions(+), 4732 deletions(-)
>
> diff --git a/apply.c b/apply.c
> index 2eac3e3..7b96130 100644
> --- a/apply.c
> +++ b/apply.c
> @@ -1,5 +1,23 @@
> +/*
> + * apply.c
> + *
> + * Copyright (C) Linus Torvalds, 2005

We're very inconsistent with the intellectual property log.
Sometimes we have that at the top of a file, sometimes we don't
and rather point at the git history to find out who touched the code.
I'd rather use the history instead of having a bunch of copyright lines.
So maybe consider to drop this introductory comment as a
{preparatory, follow up} cleanup?

(This is also a nit that doesn't require a reroll on its own)

Re: [PATCH v13 04/14] apply: make some parsing functions static again

[Stefan Beller]

On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
<christian.couder@gmail.com> wrote:
> Some parsing functions that were used in both "apply.c" and
> "builtin/apply.c" are now only used in the former, so they
> can be made static to "apply.c".
>
> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
> ---
>  apply.c | 6 +++---
>  apply.h | 5 -----
>  2 files changed, 3 insertions(+), 8 deletions(-)

Looks good.

Re: [PATCH v13 05/14] apply: use error_errno() where possible

[Stefan Beller]

On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
<christian.couder@gmail.com> wrote:
> To avoid possible mistakes and to uniformly show the errno
> related messages, let's use error_errno() where possible.
>
> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
> ---

Looks good,
Thanks for these cleanups!
Stefan

Re: [PATCH v13 06/14] apply: make it possible to silently apply

[Stefan Beller]

> Printing on stdout, and calls to warning() or error() are not
> taken care of in this patch, as that will be done in following
> patches.

> -               if (state->apply_verbosely)
> +               if (state->apply_verbosity > verbosity_normal)
>                         error(_("while searching for:\n%.*s"),
>                               (int)(old - oldlines), oldlines);

But this is an error(..) ?

Have you considered to replace all these print functions (error, warning,
fprintf, printf, fprintf_ln) with another custom

    int say_when_at_least(verbosity level, const char *fmt,...)

? (I guess that would be more invasive, but the result would be more
consistent.)

Re: [PATCH v13 00/14] libify apply and use lib in am, part 3

[Junio C Hamano]

Christian Couder <christian.couder@gmail.com> writes:

> On Mon, Aug 29, 2016 at 9:04 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Christian Couder <christian.couder@gmail.com> writes:
>>
>>> Highlevel view of the patches in the series
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> This is "part 3" of the full patch series. I am resending only the
>>> last 14 patches of the full series as "part 3", because I don't want
>>> to resend the first 27 patches of v10 nearly as is.
>>
>> Just to make sure, you are sending the first 11 of these 14 exactly
>> as-is, right?  I didn't spot anything different other than 12 and 13
>> that replaced the "alternate index" step from the previous round.
>
> Yeah, the first 11 of the 14 patches have no change compared to v12.
> I didn't want to create a "part 4" as that could be confusing, and
> sending the first 11 patches gives them another chance to be reviewed
> again.

Hmph.

But most likely, you made sure that those who _could_ review the
first 11 are miniscule minority by omitting the earlier steps before
these 14 patches -- unless they are familiar with them, the first 11
patches are not much use to them.  And those who are familiar have
already seen the first 11, too.  That was why I wondered who the
target audience was when seeing only the last 14, among which 11 of
them were identical to the previous.

Re: [PATCH v13 10/14] apply: change error_routine when silent

[Stefan Beller]

On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
<christian.couder@gmail.com> wrote:
> To avoid printing anything when applying with
> `state->apply_verbosity == verbosity_silent`, let's save the
> existing warn and error routines before applying, and let's
> replace them with a routine that does nothing.
>
> Then after applying, let's restore the saved routines.
>
> Helped-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
> ---
>  apply.c | 21 ++++++++++++++++++++-
>  apply.h |  8 ++++++++
>  2 files changed, 28 insertions(+), 1 deletion(-)
>
> diff --git a/apply.c b/apply.c
> index ddbb0a2..bf81b70 100644
> --- a/apply.c
> +++ b/apply.c
> @@ -112,6 +112,11 @@ void clear_apply_state(struct apply_state *state)
>         /* &state->fn_table is cleared at the end of apply_patch() */
>  }
>
> +static void mute_routine(const char *bla, va_list params)

Instead of 'bla' you could go with 'format' as the man page for
[f]printf puts it.
Or you could leave it empty, i.e.

    static void mute_routine(const char *, va_list)
    ...

I personally do not mind bla, as I know that the first parameter is
supposed to be the format, but let's not add unneeded information.
(Side question: Is there a culture that doesn't recognize 'bla' as a
fill word?)



> +{
> +       /* do nothing */
> +}
> +
>  int check_apply_state(struct apply_state *state, int force_apply)
>  {
>         int is_not_gitdir = !startup_info->have_repository;
> @@ -144,6 +149,13 @@ int check_apply_state(struct apply_state *state, int force_apply)
>         if (!state->lock_file)
>                 return error("BUG: state->lock_file should not be NULL");
>
> +       if (state->apply_verbosity <= verbosity_silent) {
> +               state->saved_error_routine = get_error_routine();
> +               state->saved_warn_routine = get_warn_routine();
> +               set_error_routine(mute_routine);
> +               set_warn_routine(mute_routine);
> +       }
> +
>         return 0;
>  }
>
> @@ -4864,7 +4876,7 @@ int apply_all_patches(struct apply_state *state,
>                 state->newfd = -1;
>         }
>
> -       return !!errs;
> +       res = !!errs;

I am trying to understand this and the next chunk (they work together?)

>
>  end:
>         if (state->newfd >= 0) {
> @@ -4872,5 +4884,12 @@ int apply_all_patches(struct apply_state *state,
>                 state->newfd = -1;
>         }
>
> +       if (state->apply_verbosity <= verbosity_silent) {
> +               set_error_routine(state->saved_error_routine);
> +               set_warn_routine(state->saved_warn_routine);
> +       }
> +
> +       if (res > -1)
> +               return res;
>         return (res == -1 ? 1 : 128);

So anything > -1 is returned as is, and == -1 returns 1 and <-1  returns 128 ?

So I guess a reminder/explanation on why we need to fiddle with return codes
in the commit message would help. (I do not understand how the
verbosity influences return codes.)

Re: [PATCH v13 14/14] builtin/am: use apply API in run_apply()

[Stefan Beller]

On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
<christian.couder@gmail.com> wrote:
> +
> +       if (opts_left != 0)
> +               die("unknown option passed thru to git apply");

Through and thru are different spellings of the same word.
Thru is the less preferred form, however, and it might be
considered out of place outside the most informal contexts.

[Source: The Internet]

"git grep thru" confirms we only use it in comments or function
names, both are not exposed to our dear users.

The rest looks good.

Re: [PATCH v13 14/14] builtin/am: use apply API in run_apply()

[Christian Couder]

On Thu, Sep 1, 2016 at 12:33 AM, Stefan Beller <sbeller@google.com> wrote:
> On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
> <christian.couder@gmail.com> wrote:
>> +
>> +       if (opts_left != 0)
>> +               die("unknown option passed thru to git apply");
>
> Through and thru are different spellings of the same word.
> Thru is the less preferred form, however, and it might be
> considered out of place outside the most informal contexts.

Ok I will change to "through"...

> [Source: The Internet]
>
> "git grep thru" confirms we only use it in comments or function
> names, both are not exposed to our dear users.

...but I find it strange to use different kind of language for our
users, who by the way are likely to be developers, than for our own
developers.

> The rest looks good.

Thanks for your review,
Christian.

Re: [PATCH v13 03/14] Move libified code from builtin/apply.c to apply.{c,h}

[Christian Couder]

On Wed, Aug 31, 2016 at 11:57 PM, Stefan Beller <sbeller@google.com> wrote:
> On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
> <christian.couder@gmail.com> wrote:
>> As most of the apply code in builtin/apply.c has been libified by a number of
>> previous commits, it can now be moved to apply.{c,h}, so that more code can
>> use it.
>>
>> Helped-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
>> Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
>> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
>> ---
>>  apply.c         | 4731 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  apply.h         |   19 +
>>  builtin/apply.c | 4733 +------------------------------------------------------
>
> I deduce by roughly the same line count in the .c files, it is just
> moving files over.
> (On my todo list I have an idea how to make reviewing patches like this easier.)

Yeah, at one point I used some special options to try to get a smaller
diff, but it got lost along the way.

>>  3 files changed, 4751 insertions(+), 4732 deletions(-)
>>
>> diff --git a/apply.c b/apply.c
>> index 2eac3e3..7b96130 100644
>> --- a/apply.c
>> +++ b/apply.c
>> @@ -1,5 +1,23 @@
>> +/*
>> + * apply.c
>> + *
>> + * Copyright (C) Linus Torvalds, 2005
>
> We're very inconsistent with the intellectual property log.
> Sometimes we have that at the top of a file, sometimes we don't
> and rather point at the git history to find out who touched the code.
> I'd rather use the history instead of having a bunch of copyright lines.
> So maybe consider to drop this introductory comment as a
> {preparatory, follow up} cleanup?
>
> (This is also a nit that doesn't require a reroll on its own)

Yeah, if people are ok with it I may od it in a follow up patch.

Re: [PATCH v13 06/14] apply: make it possible to silently apply

[Christian Couder]

On Thu, Sep 1, 2016 at 12:07 AM, Stefan Beller <sbeller@google.com> wrote:
>> Printing on stdout, and calls to warning() or error() are not
>> taken care of in this patch, as that will be done in following
>> patches.
>
>> -               if (state->apply_verbosely)
>> +               if (state->apply_verbosity > verbosity_normal)
>>                         error(_("while searching for:\n%.*s"),
>>                               (int)(old - oldlines), oldlines);
>
> But this is an error(..) ?

Do you mean that it was a bug in the original code to print this error
only in verbose mode?

> Have you considered to replace all these print functions (error, warning,
> fprintf, printf, fprintf_ln) with another custom
>
>     int say_when_at_least(verbosity level, const char *fmt,...)
>
> ? (I guess that would be more invasive, but the result would be more
> consistent.)

My opinion is that there is a reason (or there should have been a
reason) why people decided to use error() instead of warning() for
example.
If I use say_when_at_least(verbosity level, const char *fmt,...) like
you suggest, how do I decide if error() or warning() is used to
actually print the error message?
Another parameter to this function (severity level?) is needed.

Anyway I don't think such a refactoring is needed.

Re: [PATCH v13 02/14] apply: rename and move opt constants to apply.h

[Christian Couder]

On Wed, Aug 31, 2016 at 11:46 PM, Stefan Beller <sbeller@google.com> wrote:
> On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
> <christian.couder@gmail.com> wrote:
>>  extern int check_apply_state(struct apply_state *state, int force_apply);
>>
>
> With greater scope comes greater responsibility. Nit of the day:
> In case a reroll is necessary, consider putting a comment here.
> (What are these constants? what do they control? How/where do I use them?)
>
>> +#define APPLY_OPT_INACCURATE_EOF       (1<<0)
>> +#define APPLY_OPT_RECOUNT              (1<<1)

Ok I will reroll with some added comments.

Re: [PATCH v13 10/14] apply: change error_routine when silent

[Christian Couder]

On Thu, Sep 1, 2016 at 12:20 AM, Stefan Beller <sbeller@google.com> wrote:
> On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
> <christian.couder@gmail.com> wrote:
>> To avoid printing anything when applying with
>> `state->apply_verbosity == verbosity_silent`, let's save the
>> existing warn and error routines before applying, and let's
>> replace them with a routine that does nothing.
>>
>> Then after applying, let's restore the saved routines.
>>
>> Helped-by: Stefan Beller <sbeller@google.com>
>> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
>> ---
>>  apply.c | 21 ++++++++++++++++++++-
>>  apply.h |  8 ++++++++
>>  2 files changed, 28 insertions(+), 1 deletion(-)
>>
>> diff --git a/apply.c b/apply.c
>> index ddbb0a2..bf81b70 100644
>> --- a/apply.c
>> +++ b/apply.c
>> @@ -112,6 +112,11 @@ void clear_apply_state(struct apply_state *state)
>>         /* &state->fn_table is cleared at the end of apply_patch() */
>>  }
>>
>> +static void mute_routine(const char *bla, va_list params)
>
> Instead of 'bla' you could go with 'format' as the man page for
> [f]printf puts it.
> Or you could leave it empty, i.e.
>
>     static void mute_routine(const char *, va_list)
>     ...

Ok to do that.

> I personally do not mind bla, as I know that the first parameter is
> supposed to be the format, but let's not add unneeded information.
> (Side question: Is there a culture that doesn't recognize 'bla' as a
> fill word?)
>
>> @@ -4864,7 +4876,7 @@ int apply_all_patches(struct apply_state *state,
>>                 state->newfd = -1;
>>         }
>>
>> -       return !!errs;
>> +       res = !!errs;
>
> I am trying to understand this and the next chunk (they work together?)

Yes, they work together.

>>  end:
>>         if (state->newfd >= 0) {
>> @@ -4872,5 +4884,12 @@ int apply_all_patches(struct apply_state *state,
>>                 state->newfd = -1;
>>         }
>>
>> +       if (state->apply_verbosity <= verbosity_silent) {
>> +               set_error_routine(state->saved_error_routine);
>> +               set_warn_routine(state->saved_warn_routine);
>> +       }
>> +
>> +       if (res > -1)
>> +               return res;
>>         return (res == -1 ? 1 : 128);
>
> So anything > -1 is returned as is, and == -1 returns 1 and <-1  returns 128 ?
>
> So I guess a reminder/explanation on why we need to fiddle with return codes
> in the commit message would help. (I do not understand how the
> verbosity influences return codes.)

It doesn't influence return codes, but we need to restore error
routines just before returning, so we cannot return early anymore.
I will add something to the commit message.

Re: [PATCH v13 00/14] libify apply and use lib in am, part 3

[Christian Couder]

On Thu, Sep 1, 2016 at 12:15 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Christian Couder <christian.couder@gmail.com> writes:
>
>> On Mon, Aug 29, 2016 at 9:04 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>> Christian Couder <christian.couder@gmail.com> writes:
>>>
>>>> Highlevel view of the patches in the series
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>
>>>> This is "part 3" of the full patch series. I am resending only the
>>>> last 14 patches of the full series as "part 3", because I don't want
>>>> to resend the first 27 patches of v10 nearly as is.
>>>
>>> Just to make sure, you are sending the first 11 of these 14 exactly
>>> as-is, right?  I didn't spot anything different other than 12 and 13
>>> that replaced the "alternate index" step from the previous round.
>>
>> Yeah, the first 11 of the 14 patches have no change compared to v12.
>> I didn't want to create a "part 4" as that could be confusing, and
>> sending the first 11 patches gives them another chance to be reviewed
>> again.
>
> Hmph.
>
> But most likely, you made sure that those who _could_ review the
> first 11 are miniscule minority by omitting the earlier steps before
> these 14 patches -- unless they are familiar with them, the first 11
> patches are not much use to them.  And those who are familiar have
> already seen the first 11, too.  That was why I wondered who the
> target audience was when seeing only the last 14, among which 11 of
> them were identical to the previous.

Following Stefan's review, it looks like I will need to resend at
least 02/14, 10/14 and 14/14.
What do you prefer me to resend:
1) all the last 40 or so patches
2) the last 14 patches
3) only the few patches that changed
?

Re: [PATCH v13 06/14] apply: make it possible to silently apply

[Stefan Beller]

On Thu, Sep 1, 2016 at 1:01 AM, Christian Couder
<christian.couder@gmail.com> wrote:
> On Thu, Sep 1, 2016 at 12:07 AM, Stefan Beller <sbeller@google.com> wrote:
>>> Printing on stdout, and calls to warning() or error() are not
>>> taken care of in this patch, as that will be done in following
>>> patches.
>>
>>> -               if (state->apply_verbosely)
>>> +               if (state->apply_verbosity > verbosity_normal)
>>>                         error(_("while searching for:\n%.*s"),
>>>                               (int)(old - oldlines), oldlines);
>>
>> But this is an error(..) ?
>
> Do you mean that it was a bug in the original code to print this error
> only in verbose mode?

Oh never mind.

I meant to point out the inconsistency between the commit message, that
said: "error() are not taken care of in this patch" and modifying a
condition for
an error call. However we need to fix them as you renamed apply_verbosely
to apply_verbosity and made it an enum. So I spoke too early.

>
> Anyway I don't think such a refactoring is needed.

great :)

Re: [PATCH v13 00/14] libify apply and use lib in am, part 3

[Junio C Hamano]

Christian Couder <christian.couder@gmail.com> writes:

> Following Stefan's review, it looks like I will need to resend at
> least 02/14, 10/14 and 14/14.
> What do you prefer me to resend:
> 1) all the last 40 or so patches
> 2) the last 14 patches
> 3) only the few patches that changed

If this reroll is to be the candidate to be the final one, I'd
prefer to see 1 to give an easy access to more sets of eyes, but if
you just send them without giving these patches one more read-over
before sending them out, it is not as valuable as it would be.

I think 2 is of the least value.

If you do 3., pointing people at where the remainder of the series
can be found is necessary.

Re: [PATCH v13 10/14] apply: change error_routine when silent

[Christian Couder]

On Thu, Sep 1, 2016 at 10:19 AM, Christian Couder
<christian.couder@gmail.com> wrote:
> On Thu, Sep 1, 2016 at 12:20 AM, Stefan Beller <sbeller@google.com> wrote:
>> On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
>> <christian.couder@gmail.com> wrote:
>>> To avoid printing anything when applying with
>>> `state->apply_verbosity == verbosity_silent`, let's save the
>>> existing warn and error routines before applying, and let's
>>> replace them with a routine that does nothing.
>>>
>>> Then after applying, let's restore the saved routines.
>>>
>>> Helped-by: Stefan Beller <sbeller@google.com>
>>> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
>>> ---
>>>  apply.c | 21 ++++++++++++++++++++-
>>>  apply.h |  8 ++++++++
>>>  2 files changed, 28 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/apply.c b/apply.c
>>> index ddbb0a2..bf81b70 100644
>>> --- a/apply.c
>>> +++ b/apply.c
>>> @@ -112,6 +112,11 @@ void clear_apply_state(struct apply_state *state)
>>>         /* &state->fn_table is cleared at the end of apply_patch() */
>>>  }
>>>
>>> +static void mute_routine(const char *bla, va_list params)
>>
>> Instead of 'bla' you could go with 'format' as the man page for
>> [f]printf puts it.
>> Or you could leave it empty, i.e.
>>
>>     static void mute_routine(const char *, va_list)
>>     ...
>
> Ok to do that.

Actually I get the following error when doing that:

apply.c: In function ‘mute_routine’:
apply.c:115:1: error: parameter name omitted
 static void mute_routine(const char *, va_list)
 ^
apply.c:115:1: error: parameter name omitted
make: *** [apply.o] Error 1

So I will leave it as is.

Re: [PATCH v13 10/14] apply: change error_routine when silent

[Ramsay Jones]

On 04/09/16 11:54, Christian Couder wrote:
> On Thu, Sep 1, 2016 at 10:19 AM, Christian Couder
> <christian.couder@gmail.com> wrote:
>> On Thu, Sep 1, 2016 at 12:20 AM, Stefan Beller <sbeller@google.com> wrote:
>>> On Sat, Aug 27, 2016 at 11:45 AM, Christian Couder
>>> <christian.couder@gmail.com> wrote:
>>>> To avoid printing anything when applying with
>>>> `state->apply_verbosity == verbosity_silent`, let's save the
>>>> existing warn and error routines before applying, and let's
>>>> replace them with a routine that does nothing.
>>>>
>>>> Then after applying, let's restore the saved routines.
>>>>
>>>> Helped-by: Stefan Beller <sbeller@google.com>
>>>> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
>>>> ---
>>>>  apply.c | 21 ++++++++++++++++++++-
>>>>  apply.h |  8 ++++++++
>>>>  2 files changed, 28 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/apply.c b/apply.c
>>>> index ddbb0a2..bf81b70 100644
>>>> --- a/apply.c
>>>> +++ b/apply.c
>>>> @@ -112,6 +112,11 @@ void clear_apply_state(struct apply_state *state)
>>>>         /* &state->fn_table is cleared at the end of apply_patch() */
>>>>  }
>>>>
>>>> +static void mute_routine(const char *bla, va_list params)
>>>
>>> Instead of 'bla' you could go with 'format' as the man page for
>>> [f]printf puts it.
>>> Or you could leave it empty, i.e.
>>>
>>>     static void mute_routine(const char *, va_list)
>>>     ...
>>
>> Ok to do that.
> 
> Actually I get the following error when doing that:
> 
> apply.c: In function ‘mute_routine’:
> apply.c:115:1: error: parameter name omitted
>  static void mute_routine(const char *, va_list)
>  ^
> apply.c:115:1: error: parameter name omitted
> make: *** [apply.o] Error 1

Yes, this is not C++. ;-)

> So I will leave it as is.

I think I would prefer to see:

    static void mute_routine(const char *msg, va_list params)

given that it would either be an error-msg or a warning-msg.

ATB,
Ramsay Jones

Re: [PATCH v13 10/14] apply: change error_routine when silent

[Christian Couder]

On Sun, Sep 4, 2016 at 6:31 PM, Ramsay Jones
<ramsay@ramsayjones.plus.com> wrote:
>
>
> On 04/09/16 11:54, Christian Couder wrote:
>> On Thu, Sep 1, 2016 at 10:19 AM, Christian Couder
>> <christian.couder@gmail.com> wrote:
>>> On Thu, Sep 1, 2016 at 12:20 AM, Stefan Beller <sbeller@google.com> wrote:
>>>>>
>>>>> +static void mute_routine(const char *bla, va_list params)
>>>>
>>>> Instead of 'bla' you could go with 'format' as the man page for
>>>> [f]printf puts it.
>>>> Or you could leave it empty, i.e.
>>>>
>>>>     static void mute_routine(const char *, va_list)
>>>>     ...
>>>
>>> Ok to do that.
>>
>> Actually I get the following error when doing that:
>>
>> apply.c: In function ‘mute_routine’:
>> apply.c:115:1: error: parameter name omitted
>>  static void mute_routine(const char *, va_list)
>>  ^
>> apply.c:115:1: error: parameter name omitted
>> make: *** [apply.o] Error 1
>
> Yes, this is not C++. ;-)
>
>> So I will leave it as is.
>
> I think I would prefer to see:
>
>     static void mute_routine(const char *msg, va_list params)
>
> given that it would either be an error-msg or a warning-msg.

Ok for "msg".

Re: [PATCH v13 00/14] libify apply and use lib in am, part 3

[Christian Couder]

On Thu, Sep 1, 2016 at 7:48 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Christian Couder <christian.couder@gmail.com> writes:
>
>> Following Stefan's review, it looks like I will need to resend at
>> least 02/14, 10/14 and 14/14.
>> What do you prefer me to resend:
>> 1) all the last 40 or so patches
>> 2) the last 14 patches
>> 3) only the few patches that changed
>
> If this reroll is to be the candidate to be the final one, I'd
> prefer to see 1 to give an easy access to more sets of eyes, but if
> you just send them without giving these patches one more read-over
> before sending them out, it is not as valuable as it would be.

Ok, I will send the 41 patches very soon after having given them one
more read over and implementing the few changes suggested by Stefan
and Ramsay and one improvement I spotted.

> I think 2 is of the least value.
>
> If you do 3., pointing people at where the remainder of the series
> can be found is necessary.

There is always a pointer to the full patch series on GitHub anyway in
the cover letter, but I will send everything anyway.