[RFC PATCH 00/24] Add backup log

[RFC PATCH 00/24] Add backup log

[Nguyễn Thái Ngọc Duy]

"Backup log" is similar to reflog. But instead of keeping track of ref
changes, it keeps track of file content changes. These could be from
the index (e.g. "git add" replacing something in the index), worktree
("git reset --hard" deleting everything) or in gitdir ("git config
--edit", or deleted reflog).

Backup log, when enabled, keeps the backup versions so you can undo if
needed. Head for 01/24 to have a better picture what it does, when
backups are made... This series adds a new plumbing command 'git
backup-log' to manage these backups.

A couple things left to do:

- high level UI design, including maybe extended SHA-1 syntax
- whether "git checkout <paths>" should keep backups. I think doing it
  unconditionally may be too much, but maybe keep backups of files
  with "precious" attribute on
- a UI to edit $GIT_DIR/info/excludes and gitattributes so we can make
  backups of them
- whether we should keep command causing the changes in the backup log
  (e.g. this change is made by git-add, that one git-rebase...).
  Reflog has this. I did not add it because it complicates the parsing
  a bit and not sure if it's worth it.

Nguyễn Thái Ngọc Duy (24):
  doc: introduce new "backup log" concept
  backup-log: add "update" subcommand
  read-cache.c: new flag for add_index_entry() to write to backup log
  add: support backup log
  update-index: support backup log with --keep-backup
  commit: support backup log
  apply: support backup log with --keep-backup
  add--interactive: support backup log
  backup-log.c: add API for walking backup log
  backup-log: add cat command
  backup-log: add diff command
  backup-log: add log command
  backup-log: add prune command
  gc: prune backup logs
  backup-log: keep all blob references around
  sha1-file.c: let index_path() accept NULL istate
  config --edit: support backup log
  refs: keep backup of deleted reflog
  unpack-trees.c: keep backup of ignored files being overwritten
  reset --hard: keep backup of overwritten files
  checkout -f: keep backup of overwritten files
  am: keep backup of overwritten files on --skip or --abort
  rebase: keep backup of overwritten files on --skip or --abort
  FIXME

 .gitignore                         |   1 +
 Documentation/config/core.txt      |   5 +
 Documentation/git-apply.txt        |   3 +
 Documentation/git-backup-log.txt   | 109 ++++++++
 Documentation/git-update-index.txt |   3 +
 Makefile                           |   2 +
 apply.c                            |  38 ++-
 apply.h                            |   1 +
 backup-log.c                       | 388 +++++++++++++++++++++++++++++
 backup-log.h                       |  38 +++
 builtin.h                          |   1 +
 builtin/add.c                      |   5 +
 builtin/am.c                       |   3 +
 builtin/backup-log.c               | 371 +++++++++++++++++++++++++++
 builtin/checkout.c                 |   4 +
 builtin/commit.c                   |  16 +-
 builtin/config.c                   |  27 +-
 builtin/gc.c                       |   3 +
 builtin/pack-objects.c             |   9 +-
 builtin/rebase.c                   |   6 +-
 builtin/repack.c                   |   1 +
 builtin/reset.c                    |   2 +
 builtin/update-index.c             |   7 +
 cache.h                            |   2 +
 command-list.txt                   |   1 +
 git-add--interactive.perl          |  14 +-
 git.c                              |   1 +
 merge-recursive.c                  |   2 +-
 merge.c                            |   2 +
 parse-options.c                    |   2 +-
 reachable.c                        |   3 +
 read-cache.c                       |  49 +++-
 refs/files-backend.c               |  32 +++
 revision.c                         |   3 +
 sha1-file.c                        |   8 +-
 t/t2080-backup-log.sh              | 228 +++++++++++++++++
 unpack-trees.c                     | 143 +++++++++--
 unpack-trees.h                     |   6 +-
 38 files changed, 1488 insertions(+), 51 deletions(-)
 create mode 100644 Documentation/git-backup-log.txt
 create mode 100644 backup-log.c
 create mode 100644 backup-log.h
 create mode 100644 builtin/backup-log.c
 create mode 100755 t/t2080-backup-log.sh

-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 01/24] doc: introduce new "backup log" concept

[Nguyễn Thái Ngọc Duy]

A reflog file records changes of a certain ref over time. A backup log
file does a similar job, but it records changes of some files over time.
This is the main idea of it.

This is added so that we can support undoing certain changes. For
example, if you have carefully prepared your index with "git add -p"
then accidentally do "git commit -a", your well crafted index is lost
with no easy way to recover it. We could go the other way and make
"git commit -a" complain loudly, but that has other bad side effects,
the big one is we may complain too loudly at the wrong time and too
often. This "do the right things (most of the time) but allow the
user to undo when we get it wrong" approach seems more inline with how
git handles other things.

The current plan is to have three backup log files:

- $GIT_DIR/index.bkl contains "interesting" changes made in the index.

- $GIT_DIR/worktree.bkl contains "interesting" changes made in
  worktree.

- $GIT_DIR/common/gitdir.bkl contains changes made in other files
  inside $GIT_DIR such as config, those in info/ directory, or reflog
  file deletion (aka the mystical reflog graveyard)

All these only record "interesting" changes which will be defined
later. But a few examples of them are: "git add -p" is interesting,
but "git reset HEAD" is not. Similarly changes made in $GIT_DIR/config
are usually interesting.

This patch does none of that! It adds a new man page for a new
plumbing command "git backup-log" which does show what this
functionality looks like.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/config/core.txt    |   5 ++
 Documentation/git-backup-log.txt | 107 +++++++++++++++++++++++++++++++
 2 files changed, 112 insertions(+)
 create mode 100644 Documentation/git-backup-log.txt

diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index d0e6635fe0..63b78cc048 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -316,6 +316,11 @@ This value is true by default in a repository that has
 a working directory associated with it, and false by
 default in a bare repository.
 
+core.backupLog::
+	If true, many commands will keep backup content in object database
+	before they modify some file. See linkgit:git-backup-log[1] for more
+	information.
+
 core.repositoryFormatVersion::
 	Internal variable identifying the repository format and layout
 	version.
diff --git a/Documentation/git-backup-log.txt b/Documentation/git-backup-log.txt
new file mode 100644
index 0000000000..98998156c1
--- /dev/null
+++ b/Documentation/git-backup-log.txt
@@ -0,0 +1,107 @@
+git-backup-log(1)
+=================
+
+NAME
+----
+git-backup-log - Manage backup log files
+
+
+SYNOPSIS
+--------
+[verse]
+'git backup-log' [--id=<id> | --path=<path>] log [<rev-options>] [[--] <path>...]
+'git backup-log' [--id=<id> | --path=<path>] cat [--before] [--hash] <change-id> <path>
+'git backup-log' [--id=<id> | --path=<path>] diff [<diff-options>] <change-id>
+'git backup-log' [--id=<id> | --path=<path>] prune [--expire=<time>]
+'git backup-log' [--id=<id> | --path=<path>] update <path> <old-hash> <new-hash>
+
+DESCRIPTION
+-----------
+Backup log records changes of certain files in the object database so
+that if some file is overwritten by accident, you could still get the
+original content back.
+
+Backup log is enabled by setting core.backupLog to true and the
+following commands will save backups:
+
+- linkgit:git-add[1] keeps all index changes. File removal is not
+  recorded.
+- linkgit:git-commit[1] keep all index changes in `-a` or partial
+  commit mode.
+- linkgit:git-apply[1] and linkgit:git-update-index[1] will keep
+  changes if `--keep-backup` is specified
+- Changes of `$GIT_DIR/config` made by `git config --edit` are kept.
+- Deleted reflogs are kept. References from this deleted reflog will
+  not be kept at the next garbage collection though. This is mostly
+  meant to immediately undo an accidental branch deletion.
+- linkgit:git-checkout[1] when switching branches and
+  linkgit:git-merge[1] will make a backup before overwriting
+  ignored files.
+- linkgit:git-checkout[1] with `--force`, linkgit:git-reset[1] with
+  `--hard` or linkgit:git-am[1] and linkgit:git-rebase[1] with
+  `--skip` or `--abort` will make a backup before overwriting non
+  up-to-date files.
+
+Backups are split in three groups, changes related in the index, in
+working directory or in $GIT_DIR. These can be selected with `--id`
+parameter as `index`, `worktree` and `gitdir` respectively.
+Alternatively file path of these are `$GIT_DIR/index.bkl`,
+`$GIT_DIR/worktree.bkl` and `$GIT_DIR/common/gitdir.bkl` which could
+be specified with `--path`
+
+This command is split into subcommands:
+
+update::
+	Add a new change associated with `<path>` from `<old-hash>` to
+	`<new-hash>` to the selected backup log file. `<path>` must be
+	normalized.
+
+log::
+	View the selected backup log (optionally filtered by pathspec).
+	By default, the diff of the change is shown.
+
+cat::
+	Output the file content or their object name of a specific
+	change.
+
+diff::
+	Output the diff of a specific change.
+
+prune::
+	Prune the backup log, delete updates older than a specific
+	time or invalid entries. The actual backup content is still in
+	the object database and may need to be pruned separatedly.
+
+OPTIONS
+-------
+
+--id=<id>::
+	The name of the the backup log file. Supported names are
+	`index`, `worktree` and `gitdir`.
+
+--path=<path>::
+	The path of a backup log file.
+
+--before::
+	Show the version before the change instead. This is most
+	useful when the change is a file deletion.
+
+--hash::
+	Show the object name instead of content.
+
+--expire=<time>::
+	The cutoff time for pruning backup log. The default is three
+	months ago.
+
+<path>::
+	The path of the file where the change is made.
+
+<old-hash>::
+	The blob hash of the content before the change.
+
+<old-hash>::
+	The blob hash of the content after the change.
+
+GIT
+---
+Part of the linkgit:git[1] suite
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 02/24] backup-log: add "update" subcommand

[Nguyễn Thái Ngọc Duy]

This defines backup log file format and adds basic support for writing
new entries to backup log files. The format is the same as reflog
except that "message" field becomes "path".

Similar to reflog, updating is done by appending to the end of the file
instead of creating a branch new file and do an atomic rename. If the
backup log file gets large, regenerating whole file could take longer,
and we should keep backup log overhead to minimum since it will be
called by a bunch of commands later.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 .gitignore            |  1 +
 Makefile              |  2 ++
 backup-log.c          | 46 ++++++++++++++++++++++++
 backup-log.h          | 13 +++++++
 builtin.h             |  1 +
 builtin/backup-log.c  | 81 +++++++++++++++++++++++++++++++++++++++++++
 command-list.txt      |  1 +
 git.c                 |  1 +
 parse-options.c       |  2 +-
 t/t2080-backup-log.sh | 20 +++++++++++
 10 files changed, 167 insertions(+), 1 deletion(-)
 create mode 100644 backup-log.c
 create mode 100644 backup-log.h
 create mode 100644 builtin/backup-log.c
 create mode 100755 t/t2080-backup-log.sh

diff --git a/.gitignore b/.gitignore
index 0d77ea5894..9c0427c709 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,6 +20,7 @@
 /git-apply
 /git-archimport
 /git-archive
+/git-backup-log
 /git-bisect
 /git-bisect--helper
 /git-blame
diff --git a/Makefile b/Makefile
index 1a44c811aa..d1e1a925c3 100644
--- a/Makefile
+++ b/Makefile
@@ -836,6 +836,7 @@ LIB_OBJS += archive-tar.o
 LIB_OBJS += archive-zip.o
 LIB_OBJS += argv-array.o
 LIB_OBJS += attr.o
+LIB_OBJS += backup-log.o
 LIB_OBJS += base85.o
 LIB_OBJS += bisect.o
 LIB_OBJS += blame.o
@@ -1027,6 +1028,7 @@ BUILTIN_OBJS += builtin/am.o
 BUILTIN_OBJS += builtin/annotate.o
 BUILTIN_OBJS += builtin/apply.o
 BUILTIN_OBJS += builtin/archive.o
+BUILTIN_OBJS += builtin/backup-log.o
 BUILTIN_OBJS += builtin/bisect--helper.o
 BUILTIN_OBJS += builtin/blame.o
 BUILTIN_OBJS += builtin/branch.o
diff --git a/backup-log.c b/backup-log.c
new file mode 100644
index 0000000000..c1e805b09e
--- /dev/null
+++ b/backup-log.c
@@ -0,0 +1,46 @@
+#include "cache.h"
+#include "backup-log.h"
+#include "lockfile.h"
+#include "strbuf.h"
+
+void bkl_append(struct strbuf *output, const char *path,
+		const struct object_id *from,
+		const struct object_id *to)
+{
+	if (oideq(from, to))
+		return;
+
+	strbuf_addf(output, "%s %s %s\t%s\n", oid_to_hex(from),
+		    oid_to_hex(to), git_committer_info(0), path);
+}
+
+static int bkl_write_unlocked(const char *path, struct strbuf *new_log)
+{
+	int fd = open(path, O_CREAT | O_WRONLY | O_APPEND, 0666);
+	if (fd == -1)
+		return error_errno(_("unable to open %s"), path);
+	if (write_in_full(fd, new_log->buf, new_log->len) < 0) {
+		close(fd);
+		return error_errno(_("unable to update %s"), path);
+	}
+	close(fd);
+	return 0;
+}
+
+int bkl_write(const char *path, struct strbuf *new_log)
+{
+	struct lock_file lk;
+	int ret;
+
+	ret = hold_lock_file_for_update(&lk, path, LOCK_REPORT_ON_ERROR);
+	if (ret == -1)
+		return -1;
+	ret = bkl_write_unlocked(path, new_log);
+	/*
+	 * We do not write the the .lock file and append to the real one
+	 * instead to reduce update cost. So we can't commit even in
+	 * successful case.
+	 */
+	rollback_lock_file(&lk);
+	return ret;
+}
diff --git a/backup-log.h b/backup-log.h
new file mode 100644
index 0000000000..5e475d6f35
--- /dev/null
+++ b/backup-log.h
@@ -0,0 +1,13 @@
+#ifndef __BACKUP_LOG_H__
+#define __BACKUP_LOG_H__
+
+struct object_id;
+struct strbuf;
+
+void bkl_append(struct strbuf *output, const char *path,
+		const struct object_id *from,
+		const struct object_id *to);
+
+int bkl_write(const char *path, struct strbuf *new_log);
+
+#endif
diff --git a/builtin.h b/builtin.h
index 6538932e99..4e16c52bfa 100644
--- a/builtin.h
+++ b/builtin.h
@@ -132,6 +132,7 @@ extern int cmd_am(int argc, const char **argv, const char *prefix);
 extern int cmd_annotate(int argc, const char **argv, const char *prefix);
 extern int cmd_apply(int argc, const char **argv, const char *prefix);
 extern int cmd_archive(int argc, const char **argv, const char *prefix);
+extern int cmd_backup_log(int argc, const char **argv, const char *prefix);
 extern int cmd_bisect__helper(int argc, const char **argv, const char *prefix);
 extern int cmd_blame(int argc, const char **argv, const char *prefix);
 extern int cmd_branch(int argc, const char **argv, const char *prefix);
diff --git a/builtin/backup-log.c b/builtin/backup-log.c
new file mode 100644
index 0000000000..75a02c8878
--- /dev/null
+++ b/builtin/backup-log.c
@@ -0,0 +1,81 @@
+#include "cache.h"
+#include "builtin.h"
+#include "backup-log.h"
+#include "parse-options.h"
+
+static char const * const backup_log_usage[] = {
+	N_("git backup-log [--path=<path> | --id=<id>] update <path> <old-hash> <new-hash>"),
+	NULL
+};
+
+static int update(int argc, const char **argv,
+		  const char *prefix, const char *log_path)
+{
+	struct strbuf sb = STRBUF_INIT;
+	struct object_id oid_from, oid_to;
+	const char *path;
+	int ret;
+
+	if (argc != 4)
+		usage_with_options(backup_log_usage, NULL);
+
+	path = argv[1];
+
+	if (get_oid(argv[2], &oid_from))
+		die(_("not a valid object name: %s"), argv[2]);
+
+	if (get_oid(argv[3], &oid_to))
+		die(_("not a valid object name: %s"), argv[3]);
+
+	bkl_append(&sb, path, &oid_from, &oid_to);
+	ret = bkl_write(log_path, &sb);
+	strbuf_release(&sb);
+
+	return ret;
+}
+
+static char *log_id_to_path(const char *id)
+{
+	if (!strcmp(id, "index"))
+		return git_pathdup("index.bkl");
+	else if (!strcmp(id, "worktree"))
+		return git_pathdup("worktree.bkl");
+	else if (!strcmp(id, "gitdir"))
+		return git_pathdup("common/gitdir.bkl");
+	else
+		die(_("backup log id '%s' is not recognized"), id);
+}
+
+int cmd_backup_log(int argc, const char **argv, const char *prefix)
+{
+	const char *log_id = NULL;
+	const char *log_path = NULL;
+	char *to_free = NULL;
+	struct option common_opts[] = {
+		OPT_STRING(0, "id", &log_id, N_("id"), N_("backup log file id")),
+		OPT_FILENAME(0, "path", &log_path, N_("backup log file path")),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, prefix, common_opts,
+			     backup_log_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
+	if (!argc)
+		die(_("expected a subcommand"));
+
+	if (log_id) {
+		if (log_path)
+			die(_("--id and --path are incompatible"));
+		log_path = to_free = log_id_to_path(log_id);
+	}
+	if (!log_path)
+		die(_("either --id or --path is required"));
+
+	if (!strcmp(argv[0], "update"))
+		return update(argc, argv, prefix, log_path);
+	else
+		die(_("unknown subcommand: %s"), argv[0]);
+
+	free(to_free);
+	return 0;
+}
diff --git a/command-list.txt b/command-list.txt
index 3a9af104b5..d4162f73d5 100644
--- a/command-list.txt
+++ b/command-list.txt
@@ -51,6 +51,7 @@ git-annotate                            ancillaryinterrogators
 git-apply                               plumbingmanipulators            complete
 git-archimport                          foreignscminterface
 git-archive                             mainporcelain
+git-backup-log                          plumbinginterrogators
 git-bisect                              mainporcelain           info
 git-blame                               ancillaryinterrogators          complete
 git-branch                              mainporcelain           history
diff --git a/git.c b/git.c
index 2f604a41ea..d46d22c8b1 100644
--- a/git.c
+++ b/git.c
@@ -447,6 +447,7 @@ static struct cmd_struct commands[] = {
 	{ "annotate", cmd_annotate, RUN_SETUP | NO_PARSEOPT },
 	{ "apply", cmd_apply, RUN_SETUP_GENTLY },
 	{ "archive", cmd_archive, RUN_SETUP_GENTLY },
+	{ "backup-log", cmd_backup_log, RUN_SETUP_GENTLY },
 	{ "bisect--helper", cmd_bisect__helper, RUN_SETUP },
 	{ "blame", cmd_blame, RUN_SETUP },
 	{ "branch", cmd_branch, RUN_SETUP | DELAY_PAGER_CONFIG },
diff --git a/parse-options.c b/parse-options.c
index 3b874a83a0..e1fc080cc3 100644
--- a/parse-options.c
+++ b/parse-options.c
@@ -705,7 +705,7 @@ static int usage_with_options_internal(struct parse_opt_ctx_t *ctx,
 
 	need_newline = 1;
 
-	for (; opts->type != OPTION_END; opts++) {
+	for (; opts && opts->type != OPTION_END; opts++) {
 		size_t pos;
 		int pad;
 
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
new file mode 100755
index 0000000000..267c34bb25
--- /dev/null
+++ b/t/t2080-backup-log.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+
+test_description='backup-log'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit initial &&
+	mkdir sub
+'
+
+test_expect_success 'backup-log add new item' '
+	ID=$(git rev-parse HEAD:./initial.t) &&
+	test_tick &&
+	git -C sub backup-log --id=index update foo $ZERO_OID $ID &&
+	echo "$ZERO_OID $ID $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	foo" >expected &&
+	test_cmp expected .git/index.bkl
+'
+
+test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 03/24] read-cache.c: new flag for add_index_entry() to write to backup log

[Nguyễn Thái Ngọc Duy]

Index update API is updated to write to backup log if requested. The
entry deletion API is not updated because file removal is not
"interesting" from the undo point of view.

Note, we do double locking when writing $GIT_DIR/index now:

- $GIT_DIR/index.lock is created
- then $GIT_DIR/index.bkl.lock is created

Nobody will lock these in reverse order so no chance for dead lock
(yet)

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h      |  2 ++
 read-cache.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/cache.h b/cache.h
index ca36b44ee0..51ffae7961 100644
--- a/cache.h
+++ b/cache.h
@@ -336,6 +336,7 @@ struct index_state {
 	uint64_t fsmonitor_last_update;
 	struct ewah_bitmap *fsmonitor_dirty;
 	struct mem_pool *ce_mem_pool;
+	struct strbuf *backup_log;
 };
 
 extern struct index_state the_index;
@@ -745,6 +746,7 @@ extern int index_name_pos(const struct index_state *, const char *name, int name
 #define ADD_CACHE_JUST_APPEND 8		/* Append only; tree.c::read_tree() */
 #define ADD_CACHE_NEW_ONLY 16		/* Do not replace existing ones */
 #define ADD_CACHE_KEEP_CACHE_TREE 32	/* Do not invalidate cache-tree */
+#define ADD_CACHE_LOG_UPDATES 64	/* Log changes in $GIT_DIR/index.bkl */
 extern int add_index_entry(struct index_state *, struct cache_entry *ce, int option);
 extern void rename_index_entry_at(struct index_state *, int pos, const char *new_name);
 
diff --git a/read-cache.c b/read-cache.c
index bd45dc3e24..a53cfabc2e 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -25,6 +25,7 @@
 #include "fsmonitor.h"
 #include "thread-utils.h"
 #include "progress.h"
+#include "backup-log.h"
 
 /* Mask for the name length in ce_flags in the on-disk index */
 
@@ -691,6 +692,21 @@ void set_object_name_for_intent_to_add_entry(struct cache_entry *ce)
 	oidcpy(&ce->oid, &oid);
 }
 
+static void update_backup_log(struct index_state *istate,
+			      const struct object_id *prev,
+			      const struct cache_entry *ce)
+{
+	struct strbuf *sb = istate->backup_log;
+
+	if (!sb) {
+		sb = xmalloc(sizeof(*sb));
+		strbuf_init(sb, 0);
+		istate->backup_log = sb;
+	}
+
+	bkl_append(sb, ce->name, prev, &ce->oid);
+}
+
 int add_to_index(struct index_state *istate, const char *path, struct stat *st, int flags)
 {
 	int namelen, was_same;
@@ -765,6 +781,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,
 			discard_cache_entry(ce);
 			return error("unable to index file %s", path);
 		}
+		add_option |= flags & ADD_CACHE_LOG_UPDATES;
 	} else
 		set_object_name_for_intent_to_add_entry(ce);
 
@@ -1257,6 +1274,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e
 	int ok_to_replace = option & ADD_CACHE_OK_TO_REPLACE;
 	int skip_df_check = option & ADD_CACHE_SKIP_DFCHECK;
 	int new_only = option & ADD_CACHE_NEW_ONLY;
+	struct object_id backup_prev;
 
 	if (!(option & ADD_CACHE_KEEP_CACHE_TREE))
 		cache_tree_invalidate_path(istate, ce->name);
@@ -1273,8 +1291,12 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e
 
 	/* existing match? Just replace it. */
 	if (pos >= 0) {
-		if (!new_only)
-			replace_index_entry(istate, pos, ce);
+		if (new_only)
+			return 0;
+
+		if (option & ADD_CACHE_LOG_UPDATES)
+			update_backup_log(istate, &istate->cache[pos]->oid, ce);
+		replace_index_entry(istate, pos, ce);
 		return 0;
 	}
 	pos = -pos-1;
@@ -1282,6 +1304,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e
 	if (!(option & ADD_CACHE_KEEP_CACHE_TREE))
 		untracked_cache_add_to_index(istate, ce->name);
 
+	oidclr(&backup_prev);
 	/*
 	 * Inserting a merged entry ("stage 0") into the index
 	 * will always replace all non-merged entries..
@@ -1289,6 +1312,8 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e
 	if (pos < istate->cache_nr && ce_stage(ce) == 0) {
 		while (ce_same_name(istate->cache[pos], ce)) {
 			ok_to_add = 1;
+			if (ce_stage(ce) == 2)
+				oidcpy(&backup_prev, &istate->cache[pos]->oid);
 			if (!remove_index_entry_at(istate, pos))
 				break;
 		}
@@ -1307,6 +1332,8 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e
 		pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));
 		pos = -pos-1;
 	}
+	if (option & ADD_CACHE_LOG_UPDATES)
+		update_backup_log(istate, &backup_prev, ce);
 	return pos + 1;
 }
 
@@ -2323,6 +2350,10 @@ int discard_index(struct index_state *istate)
 	discard_split_index(istate);
 	free_untracked_cache(istate->untracked);
 	istate->untracked = NULL;
+	if (istate->backup_log) {
+		strbuf_release(istate->backup_log);
+		FREE_AND_NULL(istate->backup_log);
+	}
 
 	if (istate->ce_mem_pool) {
 		mem_pool_discard(istate->ce_mem_pool, should_validate_cache_entries());
@@ -3157,6 +3188,20 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
 	if (istate->fsmonitor_last_update)
 		fill_fsmonitor_bitmap(istate);
 
+	if (istate->backup_log && istate->backup_log->len) {
+		struct strbuf sb = STRBUF_INIT;
+		char *path = get_locked_file_path(lock);
+
+		strbuf_addf(&sb, "%s.bkl", path);
+		free(path);
+		if (bkl_write(sb.buf, istate->backup_log)) {
+			strbuf_release(&sb);
+			return -1;
+		}
+		strbuf_reset(istate->backup_log);
+		strbuf_release(&sb);
+	}
+
 	if (!si || alternate_index_output ||
 	    (istate->cache_changed & ~EXTMASK)) {
 		if (si)
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 04/24] add: support backup log

[Nguyễn Thái Ngọc Duy]

There is not much to say about the actual change in this patch, which
is straightforward. There is something to say about the lack of change
though.

The definition of "interesting" changes to keep in backup log
previously is "file modification, except file removal". It is further
refined: only changes coming from worktree (for from the user to be
more accurate) are interesting. Changes in the index from object
database (e.g. merging, switching branches, resetting...) are not
interesting because the actual content is already in the object
database and can be recovered (provided that you still have the
history of commands you used, of course)

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/add.c         |  5 +++++
 t/t2080-backup-log.sh | 11 +++++++++++
 2 files changed, 16 insertions(+)

diff --git a/builtin/add.c b/builtin/add.c
index f65c172299..f21d9efdd9 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -393,6 +393,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 	int require_pathspec;
 	char *seen = NULL;
 	struct lock_file lock_file = LOCK_INIT;
+	int core_backup_log = 0;
 
 	git_config(add_config, NULL);
 
@@ -439,6 +440,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)
 		 (!(addremove || take_worktree_changes)
 		  ? ADD_CACHE_IGNORE_REMOVAL : 0));
 
+	repo_config_get_bool(the_repository, "core.backuplog", &core_backup_log);
+	if (core_backup_log)
+		flags |= ADD_CACHE_LOG_UPDATES;
+
 	if (require_pathspec && argc == 0) {
 		fprintf(stderr, _("Nothing specified, nothing added.\n"));
 		fprintf(stderr, _("Maybe you wanted to say 'git add .'?\n"));
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index 267c34bb25..f7bdaaa3f6 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -17,4 +17,15 @@ test_expect_success 'backup-log add new item' '
 	test_cmp expected .git/index.bkl
 '
 
+test_expect_success 'add writes backup log' '
+	test_tick &&
+	echo update >>initial.t &&
+	OLD=$(git rev-parse :./initial.t) &&
+	NEW=$(git hash-object initial.t) &&
+	git -c core.backupLog=true add initial.t &&
+	echo "$OLD $NEW $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	initial.t" >expected &&
+	tail -n1 .git/index.bkl >actual &&
+	test_cmp expected actual
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 05/24] update-index: support backup log with --keep-backup

[Nguyễn Thái Ngọc Duy]

Since this is a plumbing command, backup log support remains off by
default and only active when both --keep-backup and core.backupLog=true
are specified.

The check of core.backupLog is mostly for convenient, the calling script
does not have to explicitly check core.backupLog every time it executes
update-index. Truly disabling backup log must be done with something
like

    git -c core.backupLog=false update-index ...

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-update-index.txt |  3 +++
 builtin/update-index.c             |  7 +++++++
 t/t2080-backup-log.sh              | 11 +++++++++++
 3 files changed, 21 insertions(+)

diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt
index 1c4d146a41..31fe330c88 100644
--- a/Documentation/git-update-index.txt
+++ b/Documentation/git-update-index.txt
@@ -218,6 +218,9 @@ will remove the intended effect of the option.
 	the configured value will take effect next time the index is
 	read and this will remove the intended effect of the option.
 
+--keep-backup::
+	Enable index backup log.
+
 \--::
 	Do not interpret any more arguments as options.
 
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 31e7cce301..295b5f5277 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -28,6 +28,8 @@
 static int allow_add;
 static int allow_remove;
 static int allow_replace;
+static int update_backup_log;
+static int core_backup_log;
 static int info_only;
 static int force_remove;
 static int verbose;
@@ -289,6 +291,7 @@ static int add_one_path(const struct cache_entry *old, const char *path, int len
 	}
 	option = allow_add ? ADD_CACHE_OK_TO_ADD : 0;
 	option |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;
+	option |= update_backup_log && core_backup_log ? ADD_CACHE_LOG_UPDATES : 0;
 	if (add_cache_entry(ce, option)) {
 		discard_cache_entry(ce);
 		return error("%s: cannot add to the index - missing --add option?", path);
@@ -419,6 +422,7 @@ static int add_cacheinfo(unsigned int mode, const struct object_id *oid,
 		ce->ce_flags |= CE_VALID;
 	option = allow_add ? ADD_CACHE_OK_TO_ADD : 0;
 	option |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;
+	option |= update_backup_log && core_backup_log ? ADD_CACHE_LOG_UPDATES : 0;
 	if (add_cache_entry(ce, option))
 		return error("%s: cannot add to the index - missing --add option?",
 			     path);
@@ -969,6 +973,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 			N_("let files replace directories and vice-versa"), 1),
 		OPT_SET_INT(0, "remove", &allow_remove,
 			N_("notice files missing from worktree"), 1),
+		OPT_SET_INT(0, "keep-backup", &update_backup_log,
+			N_("update index backup log if core.backupLog is set"), 1),
 		OPT_BIT(0, "unmerged", &refresh_args.flags,
 			N_("refresh even if index contains unmerged entries"),
 			REFRESH_UNMERGED),
@@ -1060,6 +1066,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 		usage_with_options(update_index_usage, options);
 
 	git_config(git_default_config, NULL);
+	repo_config_get_bool(the_repository, "core.backupLog", &core_backup_log);
 
 	/* we will diagnose later if it turns out that we need to update it */
 	newfd = hold_locked_index(&lock_file, 0);
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index f7bdaaa3f6..6b3814c172 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -28,4 +28,15 @@ test_expect_success 'add writes backup log' '
 	test_cmp expected actual
 '
 
+test_expect_success 'update-index --keep-backup writes backup log' '
+	test_tick &&
+	echo update-index >>initial.t &&
+	OLD=$(git rev-parse :./initial.t) &&
+	git -c core.backupLog=true update-index --keep-backup initial.t &&
+	NEW=$(git hash-object initial.t) &&
+	echo "$OLD $NEW $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	initial.t" >expected &&
+	tail -n1 .git/index.bkl >actual &&
+	test_cmp expected actual
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 06/24] commit: support backup log

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/commit.c      | 16 +++++++++++-----
 t/t2080-backup-log.sh | 24 ++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 5 deletions(-)

diff --git a/builtin/commit.c b/builtin/commit.c
index c021b119bb..2bdbeff4a2 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -268,7 +268,7 @@ static int list_paths(struct string_list *list, const char *with_tree,
 	return ret;
 }
 
-static void add_remove_files(struct string_list *list)
+static void add_remove_files(struct string_list *list, int flags)
 {
 	int i;
 	for (i = 0; i < list->nr; i++) {
@@ -280,7 +280,7 @@ static void add_remove_files(struct string_list *list)
 			continue;
 
 		if (!lstat(p->string, &st)) {
-			if (add_to_cache(p->string, &st, 0))
+			if (add_to_cache(p->string, &st, flags))
 				die(_("updating files failed"));
 		} else
 			remove_file_from_cache(p->string);
@@ -331,8 +331,14 @@ static const char *prepare_index(int argc, const char **argv, const char *prefix
 	struct string_list partial = STRING_LIST_INIT_DUP;
 	struct pathspec pathspec;
 	int refresh_flags = REFRESH_QUIET;
+	int add_flags = 0;
+	int core_backup_log = 0;
 	const char *ret;
 
+	repo_config_get_bool(the_repository, "core.backuplog", &core_backup_log);
+	if (core_backup_log)
+		add_flags = ADD_CACHE_LOG_UPDATES;
+
 	if (is_status)
 		refresh_flags |= REFRESH_UNMERGED;
 	parse_pathspec(&pathspec, 0,
@@ -391,7 +397,7 @@ static const char *prepare_index(int argc, const char **argv, const char *prefix
 	 */
 	if (all || (also && pathspec.nr)) {
 		hold_locked_index(&index_lock, LOCK_DIE_ON_ERROR);
-		add_files_to_cache(also ? prefix : NULL, &pathspec, 0);
+		add_files_to_cache(also ? prefix : NULL, &pathspec, add_flags);
 		refresh_cache_or_die(refresh_flags);
 		update_main_cache_tree(WRITE_TREE_SILENT);
 		if (write_locked_index(&the_index, &index_lock, 0))
@@ -460,7 +466,7 @@ static const char *prepare_index(int argc, const char **argv, const char *prefix
 		die(_("cannot read the index"));
 
 	hold_locked_index(&index_lock, LOCK_DIE_ON_ERROR);
-	add_remove_files(&partial);
+	add_remove_files(&partial, add_flags);
 	refresh_cache(REFRESH_QUIET);
 	update_main_cache_tree(WRITE_TREE_SILENT);
 	if (write_locked_index(&the_index, &index_lock, 0))
@@ -472,7 +478,7 @@ static const char *prepare_index(int argc, const char **argv, const char *prefix
 				  LOCK_DIE_ON_ERROR);
 
 	create_base_index(current_head);
-	add_remove_files(&partial);
+	add_remove_files(&partial, 0);
 	refresh_cache(REFRESH_QUIET);
 
 	if (write_locked_index(&the_index, &false_lock, 0))
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index 6b3814c172..b19e26a807 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -39,4 +39,28 @@ test_expect_success 'update-index --keep-backup writes backup log' '
 	test_cmp expected actual
 '
 
+test_expect_success 'commit -a writes backup log' '
+	test_tick &&
+	echo update-again >>initial.t &&
+	OLD=$(git rev-parse :./initial.t) &&
+	NEW=$(git hash-object initial.t) &&
+	git -c core.backupLog=true commit -am update-again &&
+	echo "$OLD $NEW $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	initial.t" >expected &&
+	tail -n1 .git/index.bkl >actual &&
+	test_cmp expected actual
+'
+
+test_expect_success 'partial commit writes backup log' '
+	rm -f .git/index.bkl &&
+	test_tick &&
+	echo rising >>new-file &&
+	git add -N new-file &&
+	OLD=$EMPTY_BLOB &&
+	NEW=$(git hash-object new-file) &&
+	git -c core.backupLog=true commit -m update-once-more -- new-file initial.t &&
+	echo "$OLD $NEW $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	new-file" >expected &&
+	# expect file deletion to not log anything &&
+	test_cmp expected .git/index.bkl
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 07/24] apply: support backup log with --keep-backup

[Nguyễn Thái Ngọc Duy]

Normally changes from git-apply are not worth logging because they
come from a file and can be recovered from there. This option will
mostly be used by add--interactive.pl where patches are generated
during an interactive add session and are very much worth logging.

The logging is a bit more complicated because an update is done in two
phases. The old version of all patches is removed first then the new
one added. We need to keep track of the old version in order to make a
backup.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-apply.txt |  3 +++
 apply.c                     | 38 +++++++++++++++++++++++++++++++++++--
 apply.h                     |  1 +
 t/t2080-backup-log.sh       | 25 ++++++++++++++++++++++++
 4 files changed, 65 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-apply.txt b/Documentation/git-apply.txt
index b9aa39000f..7b0ae790db 100644
--- a/Documentation/git-apply.txt
+++ b/Documentation/git-apply.txt
@@ -250,6 +250,9 @@ When `git apply` is used as a "better GNU patch", the user can pass
 the `--unsafe-paths` option to override this safety check.  This option
 has no effect when `--index` or `--cached` is in use.
 
+--keep-backup::
+	Enable index backup log when `--cached` or `--index` is used.
+
 CONFIGURATION
 -------------
 
diff --git a/apply.c b/apply.c
index 01793d6126..716c9a658f 100644
--- a/apply.c
+++ b/apply.c
@@ -21,6 +21,7 @@
 #include "quote.h"
 #include "rerere.h"
 #include "apply.h"
+#include "backup-log.h"
 
 static void git_apply_config(void)
 {
@@ -226,6 +227,7 @@ struct patch {
 	char old_oid_prefix[GIT_MAX_HEXSZ + 1];
 	char new_oid_prefix[GIT_MAX_HEXSZ + 1];
 	struct patch *next;
+	struct object_id old_oid;
 
 	/* three-way fallback result */
 	struct object_id threeway_stage[3];
@@ -4261,6 +4263,16 @@ static void patch_stats(struct apply_state *state, struct patch *patch)
 static int remove_file(struct apply_state *state, struct patch *patch, int rmdir_empty)
 {
 	if (state->update_index && !state->ita_only) {
+		if (state->backup_log) {
+			int pos = index_name_pos(state->repo->index,
+						 patch->old_name,
+						 strlen(patch->old_name));
+			if (pos >= 0)
+				oidcpy(&patch->old_oid,
+				       &state->repo->index->cache[pos]->oid);
+			else
+				oidclr(&patch->old_oid);
+		}
 		if (remove_file_from_index(state->repo->index, patch->old_name) < 0)
 			return error(_("unable to remove %s from index"), patch->old_name);
 	}
@@ -4276,7 +4288,8 @@ static int add_index_file(struct apply_state *state,
 			  const char *path,
 			  unsigned mode,
 			  void *buf,
-			  unsigned long size)
+			  unsigned long size,
+			  const struct object_id *old_oid)
 {
 	struct stat st;
 	struct cache_entry *ce;
@@ -4314,6 +4327,16 @@ static int add_index_file(struct apply_state *state,
 				       "for newly created file %s"), path);
 		}
 	}
+	if (state->backup_log) {
+		struct strbuf *sb = state->repo->index->backup_log;
+
+		if (!sb) {
+			sb = xmalloc(sizeof(*sb));
+			strbuf_init(sb, 0);
+			state->repo->index->backup_log = sb;
+		}
+		bkl_append(sb, ce->name, old_oid, &ce->oid);
+	}
 	if (add_index_entry(state->repo->index, ce, ADD_CACHE_OK_TO_ADD) < 0) {
 		discard_cache_entry(ce);
 		return error(_("unable to add cache entry for %s"), path);
@@ -4484,7 +4507,8 @@ static int create_file(struct apply_state *state, struct patch *patch)
 	if (patch->conflicted_threeway)
 		return add_conflicted_stages_file(state, patch);
 	else if (state->update_index)
-		return add_index_file(state, path, mode, buf, size);
+		return add_index_file(state, path, mode, buf, size,
+				      &patch->old_oid);
 	return 0;
 }
 
@@ -4659,6 +4683,7 @@ static int apply_patch(struct apply_state *state,
 	struct patch *list = NULL, **listp = &list;
 	int skipped_patch = 0;
 	int res = 0;
+	int core_backup_log = 0;
 
 	state->patch_input_file = filename;
 	if (read_patch_file(&buf, fd) < 0)
@@ -4721,6 +4746,13 @@ static int apply_patch(struct apply_state *state,
 		goto end;
 	}
 
+	if (state->backup_log &&
+	    (!state->update_index ||
+	     repo_config_get_bool(state->repo, "core.backupLog",
+				  &core_backup_log) ||
+	     !core_backup_log))
+		state->backup_log = 0;
+
 	if (state->check || state->apply) {
 		int r = check_patch_list(state, list);
 		if (r == -128) {
@@ -4982,6 +5014,8 @@ int apply_parse_options(int argc, const char **argv,
 			N_("mark new files with `git add --intent-to-add`")),
 		OPT_BOOL(0, "cached", &state->cached,
 			N_("apply a patch without touching the working tree")),
+		OPT_BOOL(0, "keep-backup", &state->backup_log,
+			 N_("log index changes if the feature is enabled")),
 		OPT_BOOL_F(0, "unsafe-paths", &state->unsafe_paths,
 			   N_("accept a patch that touches outside the working area"),
 			   PARSE_OPT_NOCOMPLETE),
diff --git a/apply.h b/apply.h
index 5948348133..d4de2ebcb9 100644
--- a/apply.h
+++ b/apply.h
@@ -51,6 +51,7 @@ struct apply_state {
 	int check_index; /* preimage must match the indexed version */
 	int update_index; /* check_index && apply */
 	int ita_only;	  /* add intent-to-add entries to the index */
+	int backup_log;	  /* enable backup log */
 
 	/* These control cosmetic aspect of the output */
 	int diffstat; /* just show a diffstat, and don't actually apply */
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index b19e26a807..8d1c8c5935 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -63,4 +63,29 @@ test_expect_success 'partial commit writes backup log' '
 	test_cmp expected .git/index.bkl
 '
 
+test_expect_success 'apply --cached writes backup log' '
+	rm -f .git/index.bkl &&
+	git reset --hard &&
+	test_tick &&
+	echo to-be-deleted >to-be-deleted &&
+	echo to-be-modified >to-be-modified &&
+	OLD_M=$(git hash-object to-be-modified) &&
+	git add . &&
+	git commit -m first &&
+	rm to-be-deleted &&
+	echo modified >>to-be-modified &&
+	NEW_M=$(git hash-object to-be-modified) &&
+	OLD_A=$ZERO_OID &&
+	echo to-be-added >to-be-added &&
+	NEW_A=$(git hash-object to-be-added) &&
+	git add . &&
+	git commit -m second &&
+	git diff HEAD^ HEAD >patch &&
+	git reset --hard HEAD^ &&
+	git -c core.backupLog=true apply --cached --keep-backup patch &&
+	echo "$OLD_A $NEW_A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	to-be-added" >expected &&
+	echo "$OLD_M $NEW_M $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	to-be-modified" >>expected &&
+	test_cmp expected .git/index.bkl
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 08/24] add--interactive: support backup log

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 git-add--interactive.perl | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/git-add--interactive.perl b/git-add--interactive.perl
index 20eb81cc92..730133f57e 100755
--- a/git-add--interactive.perl
+++ b/git-add--interactive.perl
@@ -102,28 +102,28 @@ sub colored {
 my %patch_modes = (
 	'stage' => {
 		DIFF => 'diff-files -p',
-		APPLY => sub { apply_patch 'apply --cached', @_; },
+		APPLY => sub { apply_patch 'apply --cached --keep-backup', @_; },
 		APPLY_CHECK => 'apply --cached',
 		FILTER => 'file-only',
 		IS_REVERSE => 0,
 	},
 	'stash' => {
 		DIFF => 'diff-index -p HEAD',
-		APPLY => sub { apply_patch 'apply --cached', @_; },
+		APPLY => sub { apply_patch 'apply --cached --keep-backup', @_; },
 		APPLY_CHECK => 'apply --cached',
 		FILTER => undef,
 		IS_REVERSE => 0,
 	},
 	'reset_head' => {
 		DIFF => 'diff-index -p --cached',
-		APPLY => sub { apply_patch 'apply -R --cached', @_; },
+		APPLY => sub { apply_patch 'apply -R --cached --keep-backup', @_; },
 		APPLY_CHECK => 'apply -R --cached',
 		FILTER => 'index-only',
 		IS_REVERSE => 1,
 	},
 	'reset_nothead' => {
 		DIFF => 'diff-index -R -p --cached',
-		APPLY => sub { apply_patch 'apply --cached', @_; },
+		APPLY => sub { apply_patch 'apply --cached --keep-backup', @_; },
 		APPLY_CHECK => 'apply --cached',
 		FILTER => 'index-only',
 		IS_REVERSE => 0,
@@ -628,7 +628,7 @@ sub update_cmd {
 				       HEADER => $status_head, },
 				     @mods);
 	if (@update) {
-		system(qw(git update-index --add --remove --),
+		system(qw(git update-index --add --remove --keep-backup --),
 		       map { $_->{VALUE} } @update);
 		say_n_paths('updated', @update);
 	}
@@ -648,7 +648,7 @@ sub revert_cmd {
 			my @lines = run_cmd_pipe(qw(git ls-tree HEAD --),
 						 map { $_->{VALUE} } @update);
 			my $fh;
-			open $fh, '| git update-index --index-info'
+			open $fh, '| git update-index --keep-backup --index-info'
 			    or die;
 			for (@lines) {
 				print $fh $_;
@@ -673,7 +673,7 @@ sub add_untracked_cmd {
 	my @add = list_and_choose({ PROMPT => __('Add untracked') },
 				  list_untracked());
 	if (@add) {
-		system(qw(git update-index --add --), @add);
+		system(qw(git update-index --keep-backup --add --), @add);
 		say_n_paths('added', @add);
 	} else {
 		print __("No untracked files.\n");
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 09/24] backup-log.c: add API for walking backup log

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 backup-log.c | 173 +++++++++++++++++++++++++++++++++++++++++++++++++++
 backup-log.h |  21 ++++++-
 2 files changed, 193 insertions(+), 1 deletion(-)

diff --git a/backup-log.c b/backup-log.c
index c1e805b09e..49f2ce68fe 100644
--- a/backup-log.c
+++ b/backup-log.c
@@ -44,3 +44,176 @@ int bkl_write(const char *path, struct strbuf *new_log)
 	rollback_lock_file(&lk);
 	return ret;
 }
+
+int bkl_parse_entry(struct strbuf *sb, struct bkl_entry *re)
+{
+	char *email_end, *message;
+	const char *p = sb->buf;
+
+	/* old SP new SP name <email> SP time TAB msg LF */
+	if (!sb->len || sb->buf[sb->len - 1] != '\n' ||
+	    parse_oid_hex(p, &re->old_oid, &p) || *p++ != ' ' ||
+	    parse_oid_hex(p, &re->new_oid, &p) || *p++ != ' ' ||
+	    !(email_end = strchr(p, '>')) ||
+	    email_end[1] != ' ')
+		return -1;	/* corrupt? */
+	re->email = p;
+	re->timestamp = parse_timestamp(email_end + 2, &message, 10);
+	if (!re->timestamp ||
+	    !message || message[0] != ' ' ||
+	    (message[1] != '+' && message[1] != '-') ||
+	    !isdigit(message[2]) || !isdigit(message[3]) ||
+	    !isdigit(message[4]) || !isdigit(message[5]))
+		return -1; /* corrupt? */
+	email_end[1] = '\0';
+	re->tz = strtol(message + 1, NULL, 10);
+	if (message[6] != '\t')
+		message += 6;
+	else
+		message += 7;
+	sb->buf[sb->len - 1] = '\0'; /* no LF */
+	re->path = message;
+	return 0;
+}
+
+static char *find_beginning_of_line(char *bob, char *scan)
+{
+	while (bob < scan && *(--scan) != '\n')
+		; /* keep scanning backwards */
+	/*
+	 * Return either beginning of the buffer, or LF at the end of
+	 * the previous line.
+	 */
+	return scan;
+}
+
+int bkl_parse_file_reverse(const char *path,
+			   int (*parse)(struct strbuf *line, void *data),
+			   void *data)
+{
+	struct strbuf sb = STRBUF_INIT;
+	FILE *logfp;
+	long pos;
+	int ret = 0, at_tail = 1;
+
+	logfp = fopen(path, "r");
+	if (!logfp) {
+		if (errno == ENOENT || errno == ENOTDIR)
+			return 0;
+		return -1;
+	}
+
+	/* Jump to the end */
+	if (fseek(logfp, 0, SEEK_END) < 0)
+		ret = error_errno(_("cannot seek back in %s"), path);
+	pos = ftell(logfp);
+	while (!ret && 0 < pos) {
+		int cnt;
+		size_t nread;
+		char buf[BUFSIZ];
+		char *endp, *scanp;
+
+		/* Fill next block from the end */
+		cnt = (sizeof(buf) < pos) ? sizeof(buf) : pos;
+		if (fseek(logfp, pos - cnt, SEEK_SET)) {
+			ret = error_errno(_("cannot seek back in %s"), path);
+			break;
+		}
+		nread = fread(buf, cnt, 1, logfp);
+		if (nread != 1) {
+			ret = error_errno(_("cannot read %d bytes from %s"),
+					  cnt, path);
+			break;
+		}
+		pos -= cnt;
+
+		scanp = endp = buf + cnt;
+		if (at_tail && scanp[-1] == '\n')
+			/* Looking at the final LF at the end of the file */
+			scanp--;
+		at_tail = 0;
+
+		while (buf < scanp) {
+			/*
+			 * terminating LF of the previous line, or the beginning
+			 * of the buffer.
+			 */
+			char *bp;
+
+			bp = find_beginning_of_line(buf, scanp);
+
+			if (*bp == '\n') {
+				/*
+				 * The newline is the end of the previous line,
+				 * so we know we have complete line starting
+				 * at (bp + 1). Prefix it onto any prior data
+				 * we collected for the line and process it.
+				 */
+				strbuf_splice(&sb, 0, 0, bp + 1, endp - (bp + 1));
+				scanp = bp;
+				endp = bp + 1;
+				ret = parse(&sb, data);
+				strbuf_reset(&sb);
+				if (ret)
+					break;
+			} else if (!pos) {
+				/*
+				 * We are at the start of the buffer, and the
+				 * start of the file; there is no previous
+				 * line, and we have everything for this one.
+				 * Process it, and we can end the loop.
+				 */
+				strbuf_splice(&sb, 0, 0, buf, endp - buf);
+				ret = parse(&sb, data);
+				strbuf_reset(&sb);
+				break;
+			}
+
+			if (bp == buf) {
+				/*
+				 * We are at the start of the buffer, and there
+				 * is more file to read backwards. Which means
+				 * we are in the middle of a line. Note that we
+				 * may get here even if *bp was a newline; that
+				 * just means we are at the exact end of the
+				 * previous line, rather than some spot in the
+				 * middle.
+				 *
+				 * Save away what we have to be combined with
+				 * the data from the next read.
+				 */
+				strbuf_splice(&sb, 0, 0, buf, endp - buf);
+				break;
+			}
+		}
+
+	}
+	if (!ret && sb.len)
+		BUG("reverse reflog parser had leftover data");
+
+	fclose(logfp);
+	strbuf_release(&sb);
+	return ret;
+}
+
+int bkl_parse_file(const char *path,
+		   int (*parse)(struct strbuf *line, void *data),
+		   void *data)
+{
+	struct strbuf sb = STRBUF_INIT;
+	FILE *logfp;
+	int ret = 0;
+
+	logfp = fopen(path, "r");
+	if (!logfp) {
+		if (errno == ENOENT || errno == ENOTDIR)
+			return 0;
+		return -1;
+	}
+
+	while (!ret && !strbuf_getwholeline(&sb, logfp, '\n'))
+		ret = parse(&sb, data);
+	fclose(logfp);
+	strbuf_release(&sb);
+	return ret;
+}
diff --git a/backup-log.h b/backup-log.h
index 5e475d6f35..c9de9c687c 100644
--- a/backup-log.h
+++ b/backup-log.h
@@ -1,13 +1,32 @@
 #ifndef __BACKUP_LOG_H__
 #define __BACKUP_LOG_H__
 
-struct object_id;
+#include "cache.h"
+
 struct strbuf;
 
+struct bkl_entry
+{
+	struct object_id old_oid;
+	struct object_id new_oid;
+	const char *email;
+	timestamp_t timestamp;
+	int tz;
+	const char *path;
+};
+
 void bkl_append(struct strbuf *output, const char *path,
 		const struct object_id *from,
 		const struct object_id *to);
 
 int bkl_write(const char *path, struct strbuf *new_log);
 
+int bkl_parse_entry(struct strbuf *sb, struct bkl_entry *re);
+int bkl_parse_file_reverse(const char *path,
+			   int (*parse)(struct strbuf *line, void *data),
+			   void *data);
+int bkl_parse_file(const char *path,
+		   int (*parse)(struct strbuf *line, void *data),
+		   void *data);
+
 #endif
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 10/24] backup-log: add cat command

[Nguyễn Thái Ngọc Duy]

This command introduces a new concept, "change id". This is similar to
"n" in reflog sha-1 extended syntax @{n}. I'm trying to group changes of
the same second together, and this timestamp becomes "change id", so
you view roughly a snapshot of changes. Of course it's not 100%
accurate. But it works most of the time and it keeps log update low.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/backup-log.c  | 83 +++++++++++++++++++++++++++++++++++++++++++
 t/t2080-backup-log.sh | 17 +++++++++
 2 files changed, 100 insertions(+)

diff --git a/builtin/backup-log.c b/builtin/backup-log.c
index 75a02c8878..b370ff9d27 100644
--- a/builtin/backup-log.c
+++ b/builtin/backup-log.c
@@ -1,10 +1,13 @@
 #include "cache.h"
 #include "builtin.h"
 #include "backup-log.h"
+#include "dir.h"
+#include "object-store.h"
 #include "parse-options.h"
 
 static char const * const backup_log_usage[] = {
 	N_("git backup-log [--path=<path> | --id=<id>] update <path> <old-hash> <new-hash>"),
+	N_("git backup-log [--path=<path> | --id=<id>] cat [<options>] <change-id> <path>"),
 	NULL
 };
 
@@ -34,6 +37,84 @@ static int update(int argc, const char **argv,
 	return ret;
 }
 
+struct cat_options {
+	timestamp_t id;
+	char *path;
+	int before;
+	int show_hash;
+};
+
+static int cat_parse(struct strbuf *line, void *data)
+{
+	struct cat_options *opts = data;
+	struct bkl_entry entry;
+	struct object_id *oid;
+	void *buf;
+	unsigned long size;
+	enum object_type type;
+
+	if (bkl_parse_entry(line, &entry))
+		return -1;
+
+	if (entry.timestamp < opts->id)
+		return 2;
+
+	if (entry.timestamp != opts->id ||
+	    fspathcmp(entry.path, opts->path))
+		return 0;
+
+	if (opts->before)
+		oid = &entry.old_oid;
+	else
+		oid = &entry.new_oid;
+
+	if (opts->show_hash) {
+		puts(oid_to_hex(oid));
+		return 1;
+	}
+
+	if (is_null_oid(oid))
+		return 1;	/* treat null oid like empty blob */
+
+	buf = read_object_file(oid, &type, &size);
+	if (!buf)
+		die(_("object not found: %s"), oid_to_hex(oid));
+	if (type != OBJ_BLOB)
+		die(_("not a blob: %s"), oid_to_hex(oid));
+
+	write_in_full(1, buf, size);
+	free(buf);
+
+	return 1;
+}
+
+static int cat(int argc, const char **argv,
+		const char *prefix, const char *log_path)
+{
+	struct cat_options opts;
+	struct option options[] = {
+		OPT_BOOL(0, "before", &opts.before, N_("select the version before the update")),
+		OPT_BOOL(0, "hash", &opts.show_hash, N_("show the hash instead of the content")),
+		OPT_END()
+	};
+	char *end = NULL;
+	int ret;
+
+	memset(&opts, 0, sizeof(opts));
+	argc = parse_options(argc, argv, prefix, options, backup_log_usage, 0);
+	if (argc != 2)
+		usage_with_options(backup_log_usage, options);
+	opts.id = strtol(argv[0], &end, 10);
+	if (!end || *end)
+		die(_("not a valid change id: %s"), argv[0]);
+	opts.path = prefix_path(prefix, prefix ? strlen(prefix) : 0, argv[1]);
+	setup_pager();
+	ret = bkl_parse_file_reverse(log_path, cat_parse, &opts);
+	if (ret < 0)
+		die(_("failed to parse '%s'"), log_path);
+	return ret - 1;
+}
+
 static char *log_id_to_path(const char *id)
 {
 	if (!strcmp(id, "index"))
@@ -73,6 +154,8 @@ int cmd_backup_log(int argc, const char **argv, const char *prefix)
 
 	if (!strcmp(argv[0], "update"))
 		return update(argc, argv, prefix, log_path);
+	else if (!strcmp(argv[0], "cat"))
+		return cat(argc, argv, prefix, log_path);
 	else
 		die(_("unknown subcommand: %s"), argv[0]);
 
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index 8d1c8c5935..9c004c9727 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -88,4 +88,21 @@ test_expect_success 'apply --cached writes backup log' '
 	test_cmp expected .git/index.bkl
 '
 
+test_expect_success 'backup-log cat' '
+	OLD=$(git rev-parse :./initial.t) &&
+	echo update >>initial.t &&
+	test_tick &&
+	git -c core.backupLog=true add initial.t &&
+	NEW=$(git rev-parse :./initial.t) &&
+	git backup-log --id=index cat --before --hash $test_tick initial.t >actual &&
+	echo $OLD >expected &&
+	test_cmp expected actual &&
+	git backup-log --id=index cat --hash $test_tick initial.t >actual &&
+	echo $NEW >expected &&
+	test_cmp expected actual &&
+	git backup-log --id=index cat $test_tick initial.t >actual &&
+	git cat-file blob $NEW >expected &&
+	test_cmp expected actual
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 11/24] backup-log: add diff command

[Nguyễn Thái Ngọc Duy]

This gives you a patch so you can apply if you want.

FIXME pathspec support?

PS. the file list in diff queue is not sorted like a normal "git
diff". If this is a big deal, we probably can sort it later.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/backup-log.c  | 100 ++++++++++++++++++++++++++++++++++++++++++
 t/t2080-backup-log.sh |  25 +++++++++++
 2 files changed, 125 insertions(+)

diff --git a/builtin/backup-log.c b/builtin/backup-log.c
index b370ff9d27..f8029e0011 100644
--- a/builtin/backup-log.c
+++ b/builtin/backup-log.c
@@ -1,6 +1,8 @@
 #include "cache.h"
 #include "builtin.h"
 #include "backup-log.h"
+#include "diff.h"
+#include "diffcore.h"
 #include "dir.h"
 #include "object-store.h"
 #include "parse-options.h"
@@ -8,6 +10,7 @@
 static char const * const backup_log_usage[] = {
 	N_("git backup-log [--path=<path> | --id=<id>] update <path> <old-hash> <new-hash>"),
 	N_("git backup-log [--path=<path> | --id=<id>] cat [<options>] <change-id> <path>"),
+	N_("git backup-log [--path=<path> | --id=<id>] diff [<diff-options>] <change-id>"),
 	NULL
 };
 
@@ -115,6 +118,101 @@ static int cat(int argc, const char **argv,
 	return ret - 1;
 }
 
+static void queue_blk_entry_for_diff(const struct bkl_entry *e)
+{
+	unsigned mode = canon_mode(S_IFREG | 0644);
+	struct diff_filespec *one, *two;
+	const struct object_id *old, *new;
+
+	if (is_null_oid(&e->old_oid))
+		old = the_hash_algo->empty_blob;
+	else
+		old = &e->old_oid;
+
+	if (is_null_oid(&e->new_oid))
+		new = the_hash_algo->empty_blob;
+	else
+		new = &e->new_oid;
+
+	if (oideq(old, new))
+		return;
+
+	one = alloc_filespec(e->path);
+	two = alloc_filespec(e->path);
+
+	fill_filespec(one, old, 1, mode);
+	fill_filespec(two, new, 1, mode);
+	diff_queue(&diff_queued_diff, one, two);
+}
+
+static int diff_parse(struct strbuf *line, void *data)
+{
+	timestamp_t id = *(timestamp_t *)data;
+	struct bkl_entry entry;
+
+	if (bkl_parse_entry(line, &entry))
+		return -1;
+
+	if (entry.timestamp < id)
+		return 1;
+
+	if (entry.timestamp > id)
+		return 0;
+
+	queue_blk_entry_for_diff(&entry);
+	return 0;
+}
+
+static int diff(int argc, const char **argv,
+		const char *prefix, const char *log_path)
+{
+	char *end = NULL;
+	struct diff_options diffopt;
+	timestamp_t id = -1;
+	int ret, i, found_dash_dash = 0;
+
+	repo_diff_setup(the_repository, &diffopt);
+	i = 1;
+	while (i < argc) {
+		const char *arg = argv[i];
+
+		if (!found_dash_dash && *arg == '-') {
+			if (!strcmp(arg, "--")) {
+				found_dash_dash = 1;
+				i++;
+				continue;
+			}
+			ret = diff_opt_parse(&diffopt, argv + i, argc - i, prefix);
+			if (ret < 0)
+				exit(128);
+			i += ret;
+			continue;
+		}
+
+		id = strtol(arg, &end, 10);
+		if (!end || *end)
+			die(_("not a valid change id: %s"), arg);
+		i++;
+		break;
+	}
+	if (!diffopt.output_format)
+		diffopt.output_format = DIFF_FORMAT_PATCH;
+	diff_setup_done(&diffopt);
+	if (i != argc || id == -1)
+		usage_with_options(backup_log_usage, NULL);
+
+	ret = bkl_parse_file_reverse(log_path, diff_parse, &id);
+	if (ret < 0)
+		die(_("failed to parse '%s'"), log_path);
+
+	if (ret == 1) {
+		setup_pager();
+		diffcore_std(&diffopt);
+		diff_flush(&diffopt);
+	}
+	return ret - 1;
+}
+
 static char *log_id_to_path(const char *id)
 {
 	if (!strcmp(id, "index"))
@@ -156,6 +254,8 @@ int cmd_backup_log(int argc, const char **argv, const char *prefix)
 		return update(argc, argv, prefix, log_path);
 	else if (!strcmp(argv[0], "cat"))
 		return cat(argc, argv, prefix, log_path);
+	else if (!strcmp(argv[0], "diff"))
+		return diff(argc, argv, prefix, log_path);
 	else
 		die(_("unknown subcommand: %s"), argv[0]);
 
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index 9c004c9727..cd4288d386 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -105,4 +105,29 @@ test_expect_success 'backup-log cat' '
 	test_cmp expected actual
 '
 
+test_expect_success 'backup-log diff' '
+	echo first >initial.t &&
+	git add initial.t &&
+	echo second >initial.t &&
+	test_tick &&
+	git -c core.backupLog=true add initial.t &&
+	git backup-log --id=index diff $test_tick >actual &&
+	cat >expected <<-\EOF &&
+	diff --git a/initial.t b/initial.t
+	index 9c59e24..e019be0 100644
+	--- a/initial.t
+	+++ b/initial.t
+	@@ -1 +1 @@
+	-first
+	+second
+	EOF
+	test_cmp expected actual &&
+	git backup-log --id=index diff --stat $test_tick >stat.actual &&
+	cat >stat.expected <<-\EOF &&
+	 initial.t | 2 +-
+	 1 file changed, 1 insertion(+), 1 deletion(-)
+	EOF
+	test_cmp stat.expected stat.actual
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 12/24] backup-log: add log command

[Nguyễn Thái Ngọc Duy]

One note about the default relative mode. Since this is mostly used
after a panic moment "oops I did it again!". The user would most
likely look for latest changes of a certain path and relative dates
make it easier to spot if the change is within a time frame.

Default to --patch too because the output is pretty useless otherwise.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/backup-log.c  | 90 +++++++++++++++++++++++++++++++++++++++++++
 t/t2080-backup-log.sh | 34 ++++++++++++++++
 2 files changed, 124 insertions(+)

diff --git a/builtin/backup-log.c b/builtin/backup-log.c
index f8029e0011..2496d73ba5 100644
--- a/builtin/backup-log.c
+++ b/builtin/backup-log.c
@@ -6,11 +6,13 @@
 #include "dir.h"
 #include "object-store.h"
 #include "parse-options.h"
+#include "revision.h"
 
 static char const * const backup_log_usage[] = {
 	N_("git backup-log [--path=<path> | --id=<id>] update <path> <old-hash> <new-hash>"),
 	N_("git backup-log [--path=<path> | --id=<id>] cat [<options>] <change-id> <path>"),
 	N_("git backup-log [--path=<path> | --id=<id>] diff [<diff-options>] <change-id>"),
+	N_("git backup-log [--path=<path> | --id=<id>] log [<options>] [--] [<path>]"),
 	NULL
 };
 
@@ -213,6 +215,92 @@ static int diff(int argc, const char **argv,
 	return ret - 1;
 }
 
+struct log_options {
+	struct rev_info revs;
+	timestamp_t last_time;
+	int last_tz;
+};
+
+static void dump(struct bkl_entry *entry, struct log_options *opts)
+{
+	timestamp_t last_time = opts->last_time;
+	int last_tz = opts->last_tz;
+
+	if (entry) {
+		opts->last_time = entry->timestamp;
+		opts->last_tz = entry->tz;
+	}
+
+	if (last_time != -1 &&
+	    ((!entry && diff_queued_diff.nr) ||
+	     (entry && last_time != entry->timestamp))) {
+		printf("Change-Id: %"PRItime"\n", last_time);
+		printf("Date: %s\n", show_date(last_time, last_tz, &opts->revs.date_mode));
+		printf("\n--\n");
+		diffcore_std(&opts->revs.diffopt);
+		diff_flush(&opts->revs.diffopt);
+		printf("\n");
+	}
+}
+
+static int log_parse(struct strbuf *line, void *data)
+{
+	struct log_options *opts = data;
+	struct bkl_entry entry;
+
+	if (bkl_parse_entry(line, &entry))
+		return -1;
+
+	if (opts->revs.max_age != -1 && entry.timestamp < opts->revs.max_age)
+		return 1;
+
+	if (!match_pathspec(the_repository->index, &opts->revs.prune_data,
+			    entry.path, strlen(entry.path),
+			    0, NULL, 0))
+		return 0;
+
+	dump(&entry, opts);
+
+	queue_blk_entry_for_diff(&entry);
+
+	return 0;
+}
+
+static int log_(int argc, const char **argv,
+		const char *prefix, const char *log_path)
+{
+	struct log_options opts;
+	int ret;
+
+	memset(&opts, 0, sizeof(opts));
+	opts.last_time = -1;
+	opts.last_tz = -1;
+
+	repo_init_revisions(the_repository, &opts.revs, prefix);
+	opts.revs.date_mode.type = DATE_RELATIVE;
+	argc = setup_revisions(argc, argv, &opts.revs, NULL);
+	if (!opts.revs.diffopt.output_format) {
+		opts.revs.diffopt.output_format = DIFF_FORMAT_PATCH;
+		diff_setup_done(&opts.revs.diffopt);
+	}
+
+	setup_pager();
+	if (opts.revs.reverse) {
+		/*
+		 * Default order is reading file from the bottom. --reverse
+		 * makes it read from the top instead (i.e. forward)
+		 */
+		ret = bkl_parse_file(log_path, log_parse, &opts);
+	} else {
+		ret = bkl_parse_file_reverse(log_path, log_parse, &opts);
+	}
+	if (ret < 0)
+		die(_("failed to parse '%s'"), log_path);
+	dump(NULL, &opts);	/* flush */
+
+	return ret;
+}
+
 static char *log_id_to_path(const char *id)
 {
 	if (!strcmp(id, "index"))
@@ -256,6 +344,8 @@ int cmd_backup_log(int argc, const char **argv, const char *prefix)
 		return cat(argc, argv, prefix, log_path);
 	else if (!strcmp(argv[0], "diff"))
 		return diff(argc, argv, prefix, log_path);
+	else if (!strcmp(argv[0], "log"))
+		return log_(argc, argv, prefix, log_path);
 	else
 		die(_("unknown subcommand: %s"), argv[0]);
 
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index cd4288d386..3cdabc2ba2 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -130,4 +130,38 @@ test_expect_success 'backup-log diff' '
 	test_cmp stat.expected stat.actual
 '
 
+test_expect_success 'backup-log log' '
+	rm -f .git/index.bkl &&
+	echo first >initial.t &&
+	git add initial.t &&
+	echo second >initial.t &&
+	test_tick &&
+	git -c core.backupLog=true add initial.t &&
+	echo third >initial.t &&
+	# note, same tick!
+	git -c core.backupLog=true add initial.t &&
+	test_tick &&
+	echo forth >initial.t &&
+	git -c core.backupLog=true add initial.t &&
+	git backup-log --id=index log --stat --date=iso >actual &&
+	cat >expected <<-\EOF &&
+	Change-Id: 1112912593
+	Date: 2005-04-07 15:23:13 -0700
+	
+	--
+	 initial.t | 2 +-
+	 1 file changed, 1 insertion(+), 1 deletion(-)
+	
+	Change-Id: 1112912533
+	Date: 2005-04-07 15:22:13 -0700
+	
+	--
+	 initial.t | 2 +-
+	 initial.t | 2 +-
+	 2 files changed, 2 insertions(+), 2 deletions(-)
+	
+	EOF
+	test_cmp expected actual
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 13/24] backup-log: add prune command

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 backup-log.c         | 71 ++++++++++++++++++++++++++++++++++++++++++++
 backup-log.h         |  3 ++
 builtin/backup-log.c | 17 +++++++++++
 3 files changed, 91 insertions(+)

diff --git a/backup-log.c b/backup-log.c
index 49f2ce68fe..5e38725981 100644
--- a/backup-log.c
+++ b/backup-log.c
@@ -1,6 +1,8 @@
 #include "cache.h"
 #include "backup-log.h"
+#include "blob.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "strbuf.h"
 
 void bkl_append(struct strbuf *output, const char *path,
@@ -217,3 +219,72 @@ int bkl_parse_file(const char *path,
 	strbuf_release(&sb);
 	return ret;
 }
+
+struct prune_options {
+	struct repository *repo;
+	FILE *fp;
+	timestamp_t expire;
+	struct strbuf copy;
+};
+
+static int good_oid(struct repository *r, const struct object_id *oid)
+{
+	if (is_null_oid(oid))
+		return 1;
+
+	return oid_object_info(r, oid, NULL) == OBJ_BLOB;
+}
+
+static int prune_parse(struct strbuf *line, void *data)
+{
+	struct prune_options *opts = data;
+	struct bkl_entry entry;
+
+	strbuf_reset(&opts->copy);
+	strbuf_addbuf(&opts->copy, line);
+
+	if (bkl_parse_entry(line, &entry))
+		return -1;
+
+	if (entry.timestamp < opts->expire)
+		return 0;
+
+	if (oideq(&entry.old_oid, &entry.new_oid))
+		return 0;
+
+	if (!good_oid(opts->repo, &entry.old_oid) ||
+	    !good_oid(opts->repo, &entry.new_oid))
+		return 0;
+
+	if (!opts->fp)
+		return -1;
+
+	fputs(opts->copy.buf, opts->fp);
+	return 0;
+}
+
+int bkl_prune(struct repository *r, const char *path, timestamp_t expire)
+{
+	struct lock_file lk;
+	struct prune_options opts;
+	int ret;
+
+	ret = hold_lock_file_for_update(&lk, path, 0);
+	if (ret == -1) {
+		if (errno == ENOTDIR || errno == ENOENT)
+			return 0;
+		return error(_("failed to lock '%s'"), path);
+	}
+	opts.repo = r;
+	opts.expire = expire;
+	opts.fp = fdopen_lock_file(&lk, "w");
+	strbuf_init(&opts.copy, 0);
+
+	ret = bkl_parse_file(path, prune_parse, &opts);
+	if (ret < 0)
+		rollback_lock_file(&lk);
+	else
+		ret = commit_lock_file(&lk);
+	strbuf_release(&opts.copy);
+	return ret;
+}
diff --git a/backup-log.h b/backup-log.h
index c9de9c687c..06fe706f81 100644
--- a/backup-log.h
+++ b/backup-log.h
@@ -3,6 +3,7 @@
 
 #include "cache.h"
 
+struct repository;
 struct strbuf;
 
 struct bkl_entry
@@ -29,4 +30,6 @@ int bkl_parse_file(const char *path,
 		   int (*parse)(struct strbuf *line, void *data),
 		   void *data);
 
+int bkl_prune(struct repository *r, const char *id, timestamp_t expire);
+
 #endif
diff --git a/builtin/backup-log.c b/builtin/backup-log.c
index 2496d73ba5..2291124c38 100644
--- a/builtin/backup-log.c
+++ b/builtin/backup-log.c
@@ -301,6 +301,21 @@ static int log_(int argc, const char **argv,
 	return ret;
 }
 
+static int prune(int argc, const char **argv,
+		 const char *prefix, const char *log_path)
+{
+	timestamp_t expire = time(NULL) - 90 * 24 * 3600;
+	struct option options[] = {
+		OPT_EXPIRY_DATE(0, "expire", &expire,
+				N_("expire objects older than <time>")),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, prefix, options, backup_log_usage, 0);
+
+	return bkl_prune(the_repository, log_path, expire);
+}
+
 static char *log_id_to_path(const char *id)
 {
 	if (!strcmp(id, "index"))
@@ -346,6 +361,8 @@ int cmd_backup_log(int argc, const char **argv, const char *prefix)
 		return diff(argc, argv, prefix, log_path);
 	else if (!strcmp(argv[0], "log"))
 		return log_(argc, argv, prefix, log_path);
+	else if (!strcmp(argv[0], "prune"))
+		return prune(argc, argv, prefix, log_path);
 	else
 		die(_("unknown subcommand: %s"), argv[0]);
 
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 14/24] gc: prune backup logs

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 backup-log.c | 33 +++++++++++++++++++++++++++++++++
 backup-log.h |  1 +
 builtin/gc.c |  3 +++
 3 files changed, 37 insertions(+)

diff --git a/backup-log.c b/backup-log.c
index 5e38725981..dbb6d5487e 100644
--- a/backup-log.c
+++ b/backup-log.c
@@ -4,6 +4,7 @@
 #include "lockfile.h"
 #include "object-store.h"
 #include "strbuf.h"
+#include "worktree.h"
 
 void bkl_append(struct strbuf *output, const char *path,
 		const struct object_id *from,
@@ -288,3 +289,35 @@ int bkl_prune(struct repository *r, const char *path, timestamp_t expire)
 	strbuf_release(&opts.copy);
 	return ret;
 }
+
+void bkl_prune_all_or_die(struct repository *r, timestamp_t expire)
+{
+	struct worktree **worktrees, **p;
+	char *bkl_path;
+
+	bkl_path = repo_git_path(r, "common/gitdir.bkl");
+	if (bkl_prune(r, bkl_path, expire))
+		die(_("failed to prune %s"), "gitdir.bkl");
+	free(bkl_path);
+
+	worktrees = get_worktrees(0);
+	for (p = worktrees; *p; p++) {
+		struct worktree *wt = *p;
+
+		if (bkl_prune(r, worktree_git_path(wt, "index.bkl"), expire)) {
+			if (wt->id)
+				die(_("failed to prune %s on working tree '%s'"),
+				    "index.bkl", wt->id);
+			else
+				die(_("failed to prune %s"), "index.bkl");
+		}
+		if (bkl_prune(r, worktree_git_path(wt, "worktree.bkl"), expire)) {
+			if (wt->id)
+				die(_("failed to prune %s on working tree '%s'"),
+				    "worktree.bkl", wt->id);
+			else
+				die(_("failed to prune %s"), "worktree.bkl");
+		}
+	}
+	free_worktrees(worktrees);
+}
diff --git a/backup-log.h b/backup-log.h
index 06fe706f81..6572ce9c93 100644
--- a/backup-log.h
+++ b/backup-log.h
@@ -31,5 +31,6 @@ int bkl_parse_file(const char *path,
 		   void *data);
 
 int bkl_prune(struct repository *r, const char *id, timestamp_t expire);
+void bkl_prune_all_or_die(struct repository *r, timestamp_t expire);
 
 #endif
diff --git a/builtin/gc.c b/builtin/gc.c
index 871a56f1c5..50a5d46abb 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -27,6 +27,7 @@
 #include "pack-objects.h"
 #include "blob.h"
 #include "tree.h"
+#include "backup-log.h"
 
 #define FAILED_RUN "failed to run %s"
 
@@ -657,6 +658,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	if (run_command_v_opt(rerere.argv, RUN_GIT_CMD))
 		die(FAILED_RUN, rerere.argv[0]);
 
+	bkl_prune_all_or_die(the_repository, time(NULL) - 90 * 24 * 3600);
+
 	report_garbage = report_pack_garbage;
 	reprepare_packed_git(the_repository);
 	if (pack_garbage.nr > 0)
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 15/24] backup-log: keep all blob references around

[Nguyễn Thái Ngọc Duy]

The four commands prune, rev-list, pack-objects and repack are updated
to not consider backup-log's blobs as unreachable and either delete
them outright or not repack them.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 backup-log.c           | 65 ++++++++++++++++++++++++++++++++++++++++++
 backup-log.h           |  2 ++
 builtin/pack-objects.c |  9 +++++-
 builtin/repack.c       |  1 +
 reachable.c            |  3 ++
 revision.c             |  3 ++
 6 files changed, 82 insertions(+), 1 deletion(-)

diff --git a/backup-log.c b/backup-log.c
index dbb6d5487e..37fa71e4e0 100644
--- a/backup-log.c
+++ b/backup-log.c
@@ -3,6 +3,7 @@
 #include "blob.h"
 #include "lockfile.h"
 #include "object-store.h"
+#include "revision.h"
 #include "strbuf.h"
 #include "worktree.h"
 
@@ -321,3 +322,67 @@ void bkl_prune_all_or_die(struct repository *r, timestamp_t expire)
 	}
 	free_worktrees(worktrees);
 }
+
+struct pending_cb {
+	struct rev_info *revs;
+	unsigned flags;
+};
+
+static void add_blob_to_pending(const struct object_id *oid,
+				const char *path,
+				struct pending_cb *cb)
+{
+	struct blob *blob;
+
+	if (!good_oid(cb->revs->repo, oid))
+		return;
+
+	blob = lookup_blob(cb->revs->repo, oid);
+	blob->object.flags |= cb->flags;
+	add_pending_object(cb->revs, &blob->object, path);
+}
+
+static int add_pending(struct strbuf *line, void *cb)
+{
+	struct bkl_entry entry;
+
+	if (bkl_parse_entry(line, &entry))
+		return -1;
+
+	add_blob_to_pending(&entry.old_oid, entry.path, cb);
+	add_blob_to_pending(&entry.new_oid, entry.path, cb);
+	return 0;
+}
+
+static void add_backup_log_to_pending(const char *path, struct pending_cb *cb)
+{
+	bkl_parse_file(path, add_pending, cb);
+}
+
+void add_backup_logs_to_pending(struct rev_info *revs, unsigned flags)
+{
+	struct worktree **worktrees, **p;
+	char *path;
+	struct pending_cb cb;
+
+	cb.revs = revs;
+	cb.flags = flags;
+
+	worktrees = get_worktrees(0);
+	for (p = worktrees; *p; p++) {
+		struct worktree *wt = *p;
+
+		path = xstrdup(worktree_git_path(wt, "index.bkl"));
+		add_backup_log_to_pending(path, &cb);
+		free(path);
+
+		path = xstrdup(worktree_git_path(wt, "worktree.bkl"));
+		add_backup_log_to_pending(path, &cb);
+		free(path);
+	}
+	free_worktrees(worktrees);
+
+	path = git_pathdup("common/gitdir.bkl");
+	add_backup_log_to_pending(path, &cb);
+	free(path);
+}
diff --git a/backup-log.h b/backup-log.h
index 6572ce9c93..aaa76b7fe2 100644
--- a/backup-log.h
+++ b/backup-log.h
@@ -4,6 +4,7 @@
 #include "cache.h"
 
 struct repository;
+struct rev_info;
 struct strbuf;
 
 struct bkl_entry
@@ -30,6 +31,7 @@ int bkl_parse_file(const char *path,
 		   int (*parse)(struct strbuf *line, void *data),
 		   void *data);
 
+void add_backup_logs_to_pending(struct rev_info *revs, unsigned flags);
 int bkl_prune(struct repository *r, const char *id, timestamp_t expire);
 void bkl_prune_all_or_die(struct repository *r, timestamp_t expire);
 
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 411aefd687..940eb0c768 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -3230,7 +3230,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	int all_progress_implied = 0;
 	struct argv_array rp = ARGV_ARRAY_INIT;
 	int rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;
-	int rev_list_index = 0;
+	int rev_list_index = 0, rev_list_backuplog = 0;
 	struct string_list keep_pack_list = STRING_LIST_INIT_NODUP;
 	struct option pack_objects_options[] = {
 		OPT_SET_INT('q', "quiet", &progress,
@@ -3278,6 +3278,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		OPT_SET_INT_F(0, "reflog", &rev_list_reflog,
 			      N_("include objects referred by reflog entries"),
 			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "backup-log", &rev_list_backuplog,
+			      N_("include objects referred by backup-log entries"),
+			      1, PARSE_OPT_NONEG),
 		OPT_SET_INT_F(0, "indexed-objects", &rev_list_index,
 			      N_("include objects referred to by the index"),
 			      1, PARSE_OPT_NONEG),
@@ -3366,6 +3369,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		use_internal_rev_list = 1;
 		argv_array_push(&rp, "--reflog");
 	}
+	if (rev_list_backuplog) {
+		use_internal_rev_list = 1;
+		argv_array_push(&rp, "--backup-log");
+	}
 	if (rev_list_index) {
 		use_internal_rev_list = 1;
 		argv_array_push(&rp, "--indexed-objects");
diff --git a/builtin/repack.c b/builtin/repack.c
index 45583683ee..a8a9fad9c6 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -365,6 +365,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 	argv_array_push(&cmd.args, "--non-empty");
 	argv_array_push(&cmd.args, "--all");
 	argv_array_push(&cmd.args, "--reflog");
+	argv_array_push(&cmd.args, "--backup-log");
 	argv_array_push(&cmd.args, "--indexed-objects");
 	if (repository_format_partial_clone)
 		argv_array_push(&cmd.args, "--exclude-promisor-objects");
diff --git a/reachable.c b/reachable.c
index 6e9b810d2a..61f6560b54 100644
--- a/reachable.c
+++ b/reachable.c
@@ -12,6 +12,7 @@
 #include "packfile.h"
 #include "worktree.h"
 #include "object-store.h"
+#include "backup-log.h"
 
 struct connectivity_progress {
 	struct progress *progress;
@@ -185,6 +186,8 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	if (mark_reflog)
 		add_reflogs_to_pending(revs, 0);
 
+	add_backup_logs_to_pending(revs, 0);
+
 	cp.progress = progress;
 	cp.count = 0;
 
diff --git a/revision.c b/revision.c
index 13e0519c02..755edea61e 100644
--- a/revision.c
+++ b/revision.c
@@ -27,6 +27,7 @@
 #include "commit-reach.h"
 #include "commit-graph.h"
 #include "prio-queue.h"
+#include "backup-log.h"
 
 volatile show_early_output_fn_t show_early_output;
 
@@ -2286,6 +2287,8 @@ static int handle_revision_pseudo_opt(const char *submodule,
 		clear_ref_exclusion(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--reflog")) {
 		add_reflogs_to_pending(revs, *flags);
+	} else if (!strcmp(arg, "--backup-log")) {
+		add_backup_logs_to_pending(revs, *flags);
 	} else if (!strcmp(arg, "--indexed-objects")) {
 		add_index_objects_to_pending(revs, *flags);
 	} else if (!strcmp(arg, "--not")) {
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 16/24] sha1-file.c: let index_path() accept NULL istate

[Nguyễn Thái Ngọc Duy]

istate is needed for content conversion. Allow to pass NULL istate
(which implies that the caller does not care at all about file content
conversion).

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 sha1-file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1-file.c b/sha1-file.c
index 5bd11c85bc..214d62f3b5 100644
--- a/sha1-file.c
+++ b/sha1-file.c
@@ -1829,7 +1829,8 @@ static int index_mem(struct index_state *istate,
 	 */
 	if ((type == OBJ_BLOB) && path) {
 		struct strbuf nbuf = STRBUF_INIT;
-		if (convert_to_git(istate, path, buf, size, &nbuf,
+		if (istate &&
+		    convert_to_git(istate, path, buf, size, &nbuf,
 				   get_conv_flags(flags))) {
 			buf = strbuf_detach(&nbuf, &size);
 			re_allocated = 1;
@@ -1957,12 +1958,13 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 	 * Call xsize_t() only when needed to avoid potentially unnecessary
 	 * die() for large files.
 	 */
-	if (type == OBJ_BLOB && path && would_convert_to_git_filter_fd(istate, path))
+	if (type == OBJ_BLOB && path && istate &&
+	    would_convert_to_git_filter_fd(istate, path))
 		ret = index_stream_convert_blob(istate, oid, fd, path, flags);
 	else if (!S_ISREG(st->st_mode))
 		ret = index_pipe(istate, oid, fd, type, path, flags);
 	else if (st->st_size <= big_file_threshold || type != OBJ_BLOB ||
-		 (path && would_convert_to_git(istate, path)))
+		 (path && istate && would_convert_to_git(istate, path)))
 		ret = index_core(istate, oid, fd, xsize_t(st->st_size),
 				 type, path, flags);
 	else
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 17/24] config --edit: support backup log

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/config.c      | 27 ++++++++++++++++++++++++++-
 t/t2080-backup-log.sh | 14 ++++++++++++++
 2 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/builtin/config.c b/builtin/config.c
index 84385ef165..a42044d03e 100644
--- a/builtin/config.c
+++ b/builtin/config.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "backup-log.h"
 #include "config.h"
 #include "color.h"
 #include "parse-options.h"
@@ -593,6 +594,15 @@ static char *default_user_config(void)
 	return strbuf_detach(&buf, NULL);
 }
 
+static void hash_one_path(const char *path, struct object_id *oid)
+{
+	struct stat st;
+
+	if (lstat(path, &st) ||
+	    index_path(NULL, oid, path, &st, HASH_WRITE_OBJECT))
+		oidclr(oid);
+}
+
 int cmd_config(int argc, const char **argv, const char *prefix)
 {
 	int nongit = !startup_info->have_repository;
@@ -735,6 +745,8 @@ int cmd_config(int argc, const char **argv, const char *prefix)
 	}
 	else if (actions == ACTION_EDIT) {
 		char *config_file;
+		int core_backup_log = 0;
+		struct object_id old, new;
 
 		check_argc(argc, 0, 0);
 		if (!given_config_source.file && nongit)
@@ -747,6 +759,9 @@ int cmd_config(int argc, const char **argv, const char *prefix)
 		config_file = given_config_source.file ?
 				xstrdup(given_config_source.file) :
 				git_pathdup("config");
+		repo_config_get_bool(the_repository, "core.backuplog",
+				     &core_backup_log);
+		oidclr(&old);
 		if (use_global_config) {
 			int fd = open(config_file, O_CREAT | O_EXCL | O_WRONLY, 0666);
 			if (fd >= 0) {
@@ -757,9 +772,19 @@ int cmd_config(int argc, const char **argv, const char *prefix)
 			}
 			else if (errno != EEXIST)
 				die_errno(_("cannot create configuration file %s"), config_file);
-		}
+		} else if (!given_config_source.file && core_backup_log)
+			hash_one_path(git_path("config"), &old);
 		launch_editor(config_file, NULL, NULL);
 		free(config_file);
+		if (!is_null_oid(&old)) {
+			struct strbuf sb = STRBUF_INIT;
+
+			hash_one_path(git_path("config"), &new);
+			bkl_append(&sb, "config", &old, &new);
+			mkdir(git_path("common"), 0777);
+			bkl_write(git_path("common/gitdir.bkl"), &sb);
+			strbuf_release(&sb);
+		}
 	}
 	else if (actions == ACTION_SET) {
 		int ret;
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index 3cdabc2ba2..dbd19db757 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -164,4 +164,18 @@ test_expect_success 'backup-log log' '
 	test_cmp expected actual
 '
 
+test_expect_success 'config --edit makes a backup' '
+	cat >edit.sh <<-EOF &&
+	#!$SHELL_PATH
+	echo ";Edited" >>"\$1"
+	EOF
+	chmod +x edit.sh &&
+	OLD=$(git hash-object .git/config) &&
+	EDITOR=./edit.sh git -c core.backupLog=true config --edit &&
+	NEW=$(git hash-object .git/config) &&
+	grep config .git/common/gitdir.bkl &&
+	grep ^$OLD .git/common/gitdir.bkl &&
+	grep $NEW .git/common/gitdir.bkl
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 18/24] refs: keep backup of deleted reflog

[Nguyễn Thái Ngọc Duy]

As noted in git-backup-log.txt a long time ago, this is mostly meant
for recovering a branch immediately after an accidental deletion.

References from the deleted reflog will not be included in
reachability tests and they will be deleted at the next gc. At that
point, this deleted reflog becomes useless.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 refs/files-backend.c  | 32 ++++++++++++++++++++++++++++++++
 t/t2080-backup-log.sh | 12 ++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/refs/files-backend.c b/refs/files-backend.c
index dd8abe9185..9afb274075 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -11,6 +11,7 @@
 #include "../dir.h"
 #include "../chdir-notify.h"
 #include "worktree.h"
+#include "backup-log.h"
 
 /*
  * This backend uses the following flags in `ref_update::flags` for
@@ -1873,6 +1874,35 @@ static int files_reflog_exists(struct ref_store *ref_store,
 	return ret;
 }
 
+static void backup_reflog(struct files_ref_store *refs,
+			  const char *reflog_path,
+			  const char *refname)
+{
+	int core_backup_log = 0;
+	struct stat st;
+	struct strbuf line = STRBUF_INIT;
+	struct strbuf path = STRBUF_INIT;
+	struct object_id old, new;
+
+	repo_config_get_bool(the_repository, "core.backuplog",
+			     &core_backup_log);
+	if (!core_backup_log)
+		return;
+	if (ref_type(refname) != REF_TYPE_NORMAL)
+		return;
+	if (lstat(reflog_path, &st) ||
+	    index_path(NULL, &old, reflog_path, &st, HASH_WRITE_OBJECT))
+		return;
+
+	strbuf_addf(&path, "logs/%s", refname);
+	oidclr(&new);
+	bkl_append(&line, path.buf, &old, &new);
+	strbuf_release(&path);
+	mkdir_in_gitdir(git_path("common"));
+	bkl_write(git_path("common/gitdir.bkl"), &line);
+	strbuf_release(&line);
+}
+
 static int files_delete_reflog(struct ref_store *ref_store,
 			       const char *refname)
 {
@@ -1882,6 +1912,7 @@ static int files_delete_reflog(struct ref_store *ref_store,
 	int ret;
 
 	files_reflog_path(refs, &sb, refname);
+	backup_reflog(refs, sb.buf, refname);
 	ret = remove_path(sb.buf);
 	strbuf_release(&sb);
 	return ret;
@@ -2797,6 +2828,7 @@ static int files_transaction_finish(struct ref_store *ref_store,
 		    !(update->flags & REF_IS_PRUNING)) {
 			strbuf_reset(&sb);
 			files_reflog_path(refs, &sb, update->refname);
+			backup_reflog(refs, sb.buf, update->refname);
 			if (!unlink_or_warn(sb.buf))
 				try_remove_empty_parents(refs, update->refname,
 							 REMOVE_EMPTY_PARENTS_REFLOG);
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index dbd19db757..710df1ec8b 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -178,4 +178,16 @@ test_expect_success 'config --edit makes a backup' '
 	grep $NEW .git/common/gitdir.bkl
 '
 
+test_expect_success 'deleted reflog is kept' '
+	git checkout -b foo &&
+	git commit -am everything-else &&
+	test_commit two &&
+	test_commit three &&
+	git checkout -f master &&
+	OLD=$(git hash-object .git/logs/refs/heads/foo) &&
+	git -c core.backupLog=true branch -D foo &&
+	grep logs/refs/heads/foo .git/common/gitdir.bkl &&
+	grep ^$OLD .git/common/gitdir.bkl
+'
+
 test_done
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 19/24] unpack-trees.c: keep backup of ignored files being overwritten

[Nguyễn Thái Ngọc Duy]

Ignored files are usually machine generated (e.g. *.o) and not worth
keeping, so when a merge or branch switch happens and need to
overwrite them, we just go ahead and do it. Occasionally though
ignored files _can_ have valuable content.

We will likely have a separate mechanism to protect these "precious"
ignored files, but a surprise is still a surprise. Keep a backup of
ignored files when they are overwritten (until we are explicitly told
"rubbish" by said mechanism). This lets the user recover the content
and start telling git that the file is precious so it won't happen
again.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/checkout.c    |  2 ++
 merge.c               |  2 ++
 t/t2080-backup-log.sh | 21 ++++++++++++
 unpack-trees.c        | 77 +++++++++++++++++++++++++++++++++++--------
 unpack-trees.h        |  3 ++
 5 files changed, 92 insertions(+), 13 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index acdafc6e4c..b5e27a5f6d 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -620,6 +620,8 @@ static int merge_working_tree(const struct checkout_opts *opts,
 			topts.dir = xcalloc(1, sizeof(*topts.dir));
 			topts.dir->flags |= DIR_SHOW_IGNORED;
 			setup_standard_excludes(topts.dir);
+			repo_config_get_bool(the_repository, "core.backupLog",
+					     &topts.keep_backup);
 		}
 		tree = parse_tree_indirect(old_branch_info->commit ?
 					   &old_branch_info->commit->object.oid :
diff --git a/merge.c b/merge.c
index 91008f7602..9f20305d7a 100644
--- a/merge.c
+++ b/merge.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "config.h"
 #include "diff.h"
 #include "diffcore.h"
 #include "lockfile.h"
@@ -85,6 +86,7 @@ int checkout_fast_forward(struct repository *r,
 		dir.flags |= DIR_SHOW_IGNORED;
 		setup_standard_excludes(&dir);
 		opts.dir = &dir;
+		repo_config_get_bool(r, "core.backupLog", &opts.keep_backup);
 	}
 
 	opts.head_idx = 1;
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index 710df1ec8b..a283528912 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -190,4 +190,25 @@ test_expect_success 'deleted reflog is kept' '
 	grep ^$OLD .git/common/gitdir.bkl
 '
 
+test_expect_success 'overwritten ignored file is backed up' '
+	git init overwrite-ignore &&
+	(
+		cd overwrite-ignore &&
+		echo ignored-overwritten >ignored &&
+		NEW=$(git hash-object ignored) &&
+		git add ignored &&
+		git commit -m ignored &&
+		git rm --cached ignored &&
+		echo /ignored >.gitignore &&
+		git add .gitignore &&
+		git commit -m first-commit-no-ignored &&
+		echo precious >ignored &&
+		OLD=$(git hash-object ignored) &&
+		test_tick &&
+		git -c core.backupLog=true checkout --detach HEAD^ &&
+		echo "$OLD $NEW $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	ignored" >expected &&
+		test_cmp expected .git/worktree.bkl
+	)
+'
+
 test_done
diff --git a/unpack-trees.c b/unpack-trees.c
index 7570df481b..8d7273af2b 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1,6 +1,7 @@
 #define NO_THE_INDEX_COMPATIBILITY_MACROS
 #include "cache.h"
 #include "argv-array.h"
+#include "backup-log.h"
 #include "repository.h"
 #include "config.h"
 #include "dir.h"
@@ -191,6 +192,23 @@ void clear_unpack_trees_porcelain(struct unpack_trees_options *opts)
 	memset(opts->msgs, 0, sizeof(opts->msgs));
 }
 
+static void make_backup(const struct cache_entry *ce,
+			const struct object_id *old_hash,
+			const struct object_id *new_hash,
+			struct unpack_trees_options *o)
+{
+	struct object_id null_hash;
+
+	if (!o->keep_backup || is_null_oid(old_hash))
+		return;
+
+	if (!new_hash) {
+		oidclr(&null_hash);
+		new_hash = &null_hash;
+	}
+	bkl_append(&o->backup_log, ce->name, old_hash, new_hash);
+}
+
 static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,
 			 unsigned int set, unsigned int clear)
 {
@@ -462,6 +480,9 @@ static int check_updates(struct unpack_trees_options *o)
 	if (o->clone)
 		report_collided_checkout(index);
 
+	if (o->backup_log.len)
+		bkl_write(git_path("worktree.bkl"), &o->backup_log);
+
 	trace_performance_leave("check_updates");
 	return errs != 0;
 }
@@ -1460,7 +1481,8 @@ static void mark_new_skip_worktree(struct exclude_list *el,
 
 static int verify_absent(const struct cache_entry *,
 			 enum unpack_trees_error_types,
-			 struct unpack_trees_options *);
+			 struct unpack_trees_options *,
+			 struct object_id *);
 /*
  * N-way merge "len" trees.  Returns 0 on success, -1 on failure to manipulate the
  * resulting index, -2 on failure to reflect the changes to the work tree.
@@ -1489,6 +1511,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 		free(sparse);
 	}
 
+	strbuf_init(&o->backup_log, 0);
+	if (!o->update)
+		o->keep_backup = 0;
+
 	memset(&o->result, 0, sizeof(o->result));
 	o->result.initialized = 1;
 	o->result.timestamp.sec = o->src_index->timestamp.sec;
@@ -1596,7 +1622,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 			 * correct CE_NEW_SKIP_WORKTREE
 			 */
 			if (ce->ce_flags & CE_ADDED &&
-			    verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {
+			    verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o, NULL)) {
 				if (!o->show_all_errors)
 					goto return_failed;
 				ret = -1;
@@ -1646,6 +1672,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 	o->src_index = NULL;
 
 done:
+	strbuf_release(&o->backup_log);
 	trace_performance_leave("unpack_trees");
 	clear_exclude_list(&el);
 	return ret;
@@ -1880,7 +1907,8 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le
 static int check_ok_to_remove(const char *name, int len, int dtype,
 			      const struct cache_entry *ce, struct stat *st,
 			      enum unpack_trees_error_types error_type,
-			      struct unpack_trees_options *o)
+			      struct unpack_trees_options *o,
+			      struct object_id *old_hash)
 {
 	const struct cache_entry *result;
 
@@ -1895,12 +1923,16 @@ static int check_ok_to_remove(const char *name, int len, int dtype,
 		return 0;
 
 	if (o->dir &&
-	    is_excluded(o->dir, o->src_index, name, &dtype))
+	    is_excluded(o->dir, o->src_index, name, &dtype)) {
+		if (o->keep_backup && old_hash)
+			index_path(NULL, old_hash, name, st,
+				   HASH_WRITE_OBJECT);
 		/*
 		 * ce->name is explicitly excluded, so it is Ok to
 		 * overwrite it.
 		 */
 		return 0;
+	}
 	if (S_ISDIR(st->st_mode)) {
 		/*
 		 * We are checking out path "foo" and
@@ -1935,7 +1967,8 @@ static int check_ok_to_remove(const char *name, int len, int dtype,
  */
 static int verify_absent_1(const struct cache_entry *ce,
 			   enum unpack_trees_error_types error_type,
-			   struct unpack_trees_options *o)
+			   struct unpack_trees_options *o,
+			   struct object_id *old_hash)
 {
 	int len;
 	struct stat st;
@@ -1960,7 +1993,7 @@ static int verify_absent_1(const struct cache_entry *ce,
 								NULL, o);
 			else
 				ret = check_ok_to_remove(path, len, DT_UNKNOWN, NULL,
-							 &st, error_type, o);
+							 &st, error_type, o, old_hash);
 		}
 		free(path);
 		return ret;
@@ -1975,17 +2008,20 @@ static int verify_absent_1(const struct cache_entry *ce,
 
 		return check_ok_to_remove(ce->name, ce_namelen(ce),
 					  ce_to_dtype(ce), ce, &st,
-					  error_type, o);
+					  error_type, o, old_hash);
 	}
 }
 
 static int verify_absent(const struct cache_entry *ce,
 			 enum unpack_trees_error_types error_type,
-			 struct unpack_trees_options *o)
+			 struct unpack_trees_options *o,
+			 struct object_id *old_hash)
 {
+	if (o->keep_backup && old_hash)
+		oidclr(old_hash);
 	if (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))
 		return 0;
-	return verify_absent_1(ce, error_type, o);
+	return verify_absent_1(ce, error_type, o, old_hash);
 }
 
 static int verify_absent_sparse(const struct cache_entry *ce,
@@ -1996,7 +2032,7 @@ static int verify_absent_sparse(const struct cache_entry *ce,
 	if (orphaned_error == ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN)
 		orphaned_error = ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN;
 
-	return verify_absent_1(ce, orphaned_error, o);
+	return verify_absent_1(ce, orphaned_error, o, NULL);
 }
 
 static int merged_entry(const struct cache_entry *ce,
@@ -2007,6 +2043,8 @@ static int merged_entry(const struct cache_entry *ce,
 	struct cache_entry *merge = dup_cache_entry(ce, &o->result);
 
 	if (!old) {
+		struct object_id old_hash;
+
 		/*
 		 * New index entries. In sparse checkout, the following
 		 * verify_absent() will be delayed until after
@@ -2023,10 +2061,15 @@ static int merged_entry(const struct cache_entry *ce,
 		merge->ce_flags |= CE_NEW_SKIP_WORKTREE;
 
 		if (verify_absent(merge,
-				  ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {
+				  ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN,
+				  o, &old_hash)) {
 			discard_cache_entry(merge);
 			return -1;
 		}
+		if (o->keep_backup)
+			bkl_append(&o->backup_log, merge->name,
+				   &old_hash, &merge->oid);
+
 		invalidate_ce_path(merge, o);
 
 		if (submodule_from_ce(ce)) {
@@ -2083,8 +2126,12 @@ static int deleted_entry(const struct cache_entry *ce,
 {
 	/* Did it exist in the index? */
 	if (!old) {
-		if (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))
+		struct object_id old_hash;
+
+		if (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED,
+				  o, &old_hash))
 			return -1;
+		make_backup(ce, &old_hash, NULL, o);
 		return 0;
 	}
 	if (!(old->ce_flags & CE_CONFLICTED) && verify_uptodate(old, o))
@@ -2236,8 +2283,12 @@ int threeway_merge(const struct cache_entry * const *stages,
 			if (index)
 				return deleted_entry(index, index, o);
 			if (ce && !head_deleted) {
-				if (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED, o))
+				struct object_id old_hash;
+
+				if (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED,
+						  o, &old_hash))
 					return -1;
+				make_backup(ce, &old_hash, NULL, o);
 			}
 			return 0;
 		}
diff --git a/unpack-trees.h b/unpack-trees.h
index 0135080a7b..e2a64e2401 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -60,6 +60,7 @@ struct unpack_trees_options {
 		     exiting_early,
 		     show_all_errors,
 		     dry_run;
+	int keep_backup;
 	const char *prefix;
 	int cache_bottom;
 	struct dir_struct *dir;
@@ -83,6 +84,8 @@ struct unpack_trees_options {
 	struct index_state *src_index;
 	struct index_state result;
 
+	struct strbuf backup_log;
+
 	struct exclude_list *el; /* for internal use */
 };
 
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 20/24] reset --hard: keep backup of overwritten files

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/reset.c       |  2 ++
 merge-recursive.c     |  2 +-
 t/t2080-backup-log.sh | 14 +++++++++
 unpack-trees.c        | 70 +++++++++++++++++++++++++++++++++----------
 unpack-trees.h        |  3 +-
 5 files changed, 74 insertions(+), 17 deletions(-)

diff --git a/builtin/reset.c b/builtin/reset.c
index 58166964f8..517a27dce5 100644
--- a/builtin/reset.c
+++ b/builtin/reset.c
@@ -67,6 +67,8 @@ static int reset_index(const struct object_id *oid, int reset_type, int quiet)
 		break;
 	case HARD:
 		opts.update = 1;
+		repo_config_get_bool(the_repository, "core.backupLog",
+				     &opts.keep_backup);
 		/* fallthrough */
 	default:
 		opts.reset = 1;
diff --git a/merge-recursive.c b/merge-recursive.c
index acc2f64a4e..10a9d3180a 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -896,7 +896,7 @@ static int was_dirty(struct merge_options *o, const char *path)
 
 	ce = index_file_exists(o->unpack_opts.src_index,
 			       path, strlen(path), ignore_case);
-	dirty = verify_uptodate(ce, &o->unpack_opts) != 0;
+	dirty = verify_uptodate(ce, &o->unpack_opts, NULL) != 0;
 	return dirty;
 }
 
diff --git a/t/t2080-backup-log.sh b/t/t2080-backup-log.sh
index a283528912..901755ce93 100755
--- a/t/t2080-backup-log.sh
+++ b/t/t2080-backup-log.sh
@@ -211,4 +211,18 @@ test_expect_success 'overwritten ignored file is backed up' '
 	)
 '
 
+test_expect_success 'overwritten out-of-date file is backed up' '
+	git init overwrite-outofdate &&
+	(
+		cd overwrite-outofdate &&
+		test_commit haha &&
+		NEW=$(git hash-object haha.t) &&
+		echo bad >>haha.t &&
+		OLD=$(git hash-object haha.t) &&
+		git -c core.backupLog reset --hard &&
+		echo "$OLD $NEW $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $test_tick -0700	haha.t" >expected &&
+		test_cmp expected .git/worktree.bkl
+	)
+'
+
 test_done
diff --git a/unpack-trees.c b/unpack-trees.c
index 8d7273af2b..221869b47c 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1715,7 +1715,8 @@ static int same(const struct cache_entry *a, const struct cache_entry *b)
  */
 static int verify_uptodate_1(const struct cache_entry *ce,
 			     struct unpack_trees_options *o,
-			     enum unpack_trees_error_types error_type)
+			     enum unpack_trees_error_types error_type,
+			     struct object_id *old_hash)
 {
 	struct stat st;
 
@@ -1727,10 +1728,16 @@ static int verify_uptodate_1(const struct cache_entry *ce,
 	 * if this entry is truly up-to-date because this file may be
 	 * overwritten.
 	 */
-	if ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))
+	if ((ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {
 		; /* keep checking */
-	else if (o->reset || ce_uptodate(ce))
+	} else if (o->reset) {
+		if (o->keep_backup && old_hash && !lstat(ce->name, &st))
+			index_path(NULL, old_hash, ce->name, &st,
+				   HASH_WRITE_OBJECT);
+		return 0;
+	} else if (ce_uptodate(ce)) {
 		return 0;
+	}
 
 	if (!lstat(ce->name, &st)) {
 		int flags = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE;
@@ -1764,17 +1771,20 @@ static int verify_uptodate_1(const struct cache_entry *ce,
 }
 
 int verify_uptodate(const struct cache_entry *ce,
-		    struct unpack_trees_options *o)
+		    struct unpack_trees_options *o,
+		    struct object_id *old_hash)
 {
+	if (o->keep_backup && old_hash)
+		oidclr(old_hash);
 	if (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))
 		return 0;
-	return verify_uptodate_1(ce, o, ERROR_NOT_UPTODATE_FILE);
+	return verify_uptodate_1(ce, o, ERROR_NOT_UPTODATE_FILE, old_hash);
 }
 
 static int verify_uptodate_sparse(const struct cache_entry *ce,
 				  struct unpack_trees_options *o)
 {
-	return verify_uptodate_1(ce, o, ERROR_SPARSE_NOT_UPTODATE_FILE);
+	return verify_uptodate_1(ce, o, ERROR_SPARSE_NOT_UPTODATE_FILE, NULL);
 }
 
 /*
@@ -1862,8 +1872,11 @@ static int verify_clean_subdirectory(const struct cache_entry *ce,
 		 * removed.
 		 */
 		if (!ce_stage(ce2)) {
-			if (verify_uptodate(ce2, o))
+			struct object_id old_hash;
+
+			if (verify_uptodate(ce2, o, &old_hash))
 				return -1;
+			make_backup(ce2, &old_hash, NULL, o);
 			add_entry(o, ce2, CE_REMOVE, 0);
 			invalidate_ce_path(ce, o);
 			mark_ce_used(ce2, o);
@@ -1973,8 +1986,13 @@ static int verify_absent_1(const struct cache_entry *ce,
 	int len;
 	struct stat st;
 
-	if (o->index_only || o->reset || !o->update)
+	if (o->index_only || o->reset || !o->update) {
+		if (o->reset && o->keep_backup &&
+		    old_hash && !lstat(ce->name, &st))
+			index_path(NULL, old_hash, ce->name, &st,
+				   HASH_WRITE_OBJECT);
 		return 0;
+	}
 
 	len = check_leading_path(ce->name, ce_namelen(ce));
 	if (!len)
@@ -2092,10 +2110,12 @@ static int merged_entry(const struct cache_entry *ce,
 			copy_cache_entry(merge, old);
 			update = 0;
 		} else {
-			if (verify_uptodate(old, o)) {
+			struct object_id old_hash;
+			if (verify_uptodate(old, o, &old_hash)) {
 				discard_cache_entry(merge);
 				return -1;
 			}
+			make_backup(old, &old_hash, &merge->oid, o);
 			/* Migrate old flags over */
 			update |= old->ce_flags & (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);
 			invalidate_ce_path(old, o);
@@ -2124,18 +2144,20 @@ static int deleted_entry(const struct cache_entry *ce,
 			 const struct cache_entry *old,
 			 struct unpack_trees_options *o)
 {
+	struct object_id old_hash;
+
 	/* Did it exist in the index? */
 	if (!old) {
-		struct object_id old_hash;
-
 		if (verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_REMOVED,
 				  o, &old_hash))
 			return -1;
 		make_backup(ce, &old_hash, NULL, o);
 		return 0;
 	}
-	if (!(old->ce_flags & CE_CONFLICTED) && verify_uptodate(old, o))
+	if (!(old->ce_flags & CE_CONFLICTED) &&
+	    verify_uptodate(old, o, &old_hash))
 		return -1;
+	make_backup(ce, &old_hash, NULL, o);
 	add_entry(o, ce, CE_REMOVE, 0);
 	invalidate_ce_path(ce, o);
 	return 1;
@@ -2305,8 +2327,16 @@ int threeway_merge(const struct cache_entry * const *stages,
 	 * conflict resolution files.
 	 */
 	if (index) {
-		if (verify_uptodate(index, o))
+		struct object_id old_hash;
+
+		if (verify_uptodate(index, o, &old_hash))
 			return -1;
+		/*
+		 * A new conflict appears. We could make a backup from
+		 * worktree version to stage 2 or 3. But neither makes much
+		 * sense. Make a deletion backup instead.
+		 */
+		make_backup(index, &old_hash, NULL, o);
 	}
 
 	o->nontrivial_merge = 1;
@@ -2447,16 +2477,26 @@ int oneway_merge(const struct cache_entry * const *src,
 		return deleted_entry(old, old, o);
 
 	if (old && same(old, a)) {
+		struct object_id old_hash;
 		int update = 0;
+
+		oidclr(&old_hash);
 		if (o->reset && o->update && !ce_uptodate(old) && !ce_skip_worktree(old)) {
 			struct stat st;
+
 			if (lstat(old->name, &st) ||
-			    ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))
+			    ie_match_stat(o->src_index, old, &st,
+					  CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE))
 				update |= CE_UPDATE;
+
+			if (update & CE_UPDATE && o->keep_backup)
+				index_path(NULL, &old_hash, old->name, &st,
+					   HASH_WRITE_OBJECT);
 		}
 		if (o->update && S_ISGITLINK(old->ce_mode) &&
-		    should_update_submodules() && !verify_uptodate(old, o))
+		    should_update_submodules() && !verify_uptodate(old, o, NULL))
 			update |= CE_UPDATE;
+		make_backup(old, &old_hash, &old->oid, o);
 		add_entry(o, old, update, 0);
 		return 0;
 	}
diff --git a/unpack-trees.h b/unpack-trees.h
index e2a64e2401..a453def564 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -93,7 +93,8 @@ int unpack_trees(unsigned n, struct tree_desc *t,
 		 struct unpack_trees_options *options);
 
 int verify_uptodate(const struct cache_entry *ce,
-		    struct unpack_trees_options *o);
+		    struct unpack_trees_options *o,
+		    struct object_id *old_hash);
 
 int threeway_merge(const struct cache_entry * const *stages,
 		   struct unpack_trees_options *o);
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 21/24] checkout -f: keep backup of overwritten files

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/checkout.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index b5e27a5f6d..3ae001ae35 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -438,6 +438,8 @@ static int reset_tree(struct tree *tree, const struct checkout_opts *o,
 	opts.verbose_update = o->show_progress;
 	opts.src_index = &the_index;
 	opts.dst_index = &the_index;
+	repo_config_get_bool(the_repository, "core.backupLog",
+			     &opts.keep_backup);
 	parse_tree(tree);
 	init_tree_desc(&tree_desc, tree->buffer, tree->size);
 	switch (unpack_trees(1, &tree_desc, &opts)) {
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 22/24] am: keep backup of overwritten files on --skip or --abort

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/am.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/builtin/am.c b/builtin/am.c
index 8f27f3375b..098bbaab39 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1886,6 +1886,9 @@ static int fast_forward_to(struct tree *head, struct tree *remote, int reset)
 	opts.update = 1;
 	opts.merge = 1;
 	opts.reset = reset;
+	if (opts.reset)
+		repo_config_get_bool(the_repository, "core.backupLog",
+				     &opts.keep_backup);
 	opts.fn = twoway_merge;
 	init_tree_desc(&t[0], head->buffer, head->size);
 	init_tree_desc(&t[1], remote->buffer, remote->size);
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 23/24] rebase: keep backup of overwritten files on --skip or --abort

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/rebase.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/builtin/rebase.c b/builtin/rebase.c
index b5c99ec10c..5c7b223843 100644
--- a/builtin/rebase.c
+++ b/builtin/rebase.c
@@ -573,8 +573,12 @@ static int reset_head(struct object_id *oid, const char *action,
 	unpack_tree_opts.fn = reset_hard ? oneway_merge : twoway_merge;
 	unpack_tree_opts.update = 1;
 	unpack_tree_opts.merge = 1;
-	if (!detach_head)
+	if (!detach_head) {
 		unpack_tree_opts.reset = 1;
+		repo_config_get_bool(the_repository, "core.backupLog",
+				     &unpack_tree_opts.keep_backup);
+	}
+
 
 	if (read_index_unmerged(the_repository->index) < 0) {
 		ret = error(_("could not read index"));
-- 
2.20.0.rc2.486.g9832c05c3d

[PATCH 24/24] FIXME

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-backup-log.txt | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Documentation/git-backup-log.txt b/Documentation/git-backup-log.txt
index 98998156c1..fe03726337 100644
--- a/Documentation/git-backup-log.txt
+++ b/Documentation/git-backup-log.txt
@@ -41,6 +41,8 @@ following commands will save backups:
   `--hard` or linkgit:git-am[1] and linkgit:git-rebase[1] with
   `--skip` or `--abort` will make a backup before overwriting non
   up-to-date files.
+- FIXME perhaps `git checkout <paths>` only makes backups on
+  "precious" paths only?
 
 Backups are split in three groups, changes related in the index, in
 working directory or in $GIT_DIR. These can be selected with `--id`
-- 
2.20.0.rc2.486.g9832c05c3d