git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "William Baker via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: williamtbakeremail@gmail.com, stolee@gmail.com,
	Johannes.Schindelin@gmx.de, jeffhost@microsoft.com,
	Junio C Hamano <gitster@pobox.com>,
	William Baker <William.Baker@microsoft.com>
Subject: [PATCH v2 1/1] fsmonitor: don't fill bitmap with entries to be removed
Date: Wed, 09 Oct 2019 14:00:12 -0700 (PDT)	[thread overview]
Message-ID: <08741d986c2b51828f115ab50f178d62e9982978.1570654810.git.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.372.v2.git.gitgitgadget@gmail.com>

From: William Baker <William.Baker@microsoft.com>

While doing some testing with fsmonitor enabled I found
that git commands would segfault after staging and
unstaging an untracked file.  Looking at the crash it
appeared that fsmonitor_ewah_callback was attempting to
adjust bits beyond the bounds of the index cache.

Digging into how this could happen it became clear that
the fsmonitor extension must have been written with
more bits than there were entries in the index.  The
root cause ended up being that fill_fsmonitor_bitmap was
populating fsmonitor_dirty with bits for all entries in
the index, even those that had been marked for removal.

To solve this problem fill_fsmonitor_bitmap has been
updated to skip entries with the the CE_REMOVE flag set.
With this change the bits written for the fsmonitor
extension will be consistent with the index entries
written by do_write_index.  Additionally, BUG checks
have been added to detect if the number of bits in
fsmonitor_dirty should ever exceed the number of
entries in the index again.

Another option that was considered was moving the call
to fill_fsmonitor_bitmap closer to where the index is
written (and where the fsmonitor extension itself is
written).  However, that did not work as the
fsmonitor_dirty bitmap must be filled before the index
is split during writing.

Signed-off-by: William Baker <William.Baker@microsoft.com>
---
 fsmonitor.c                 | 29 ++++++++++++++++++++++++-----
 t/t7519-status-fsmonitor.sh | 13 +++++++++++++
 t/t7519/fsmonitor-env       | 24 ++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 5 deletions(-)
 create mode 100755 t/t7519/fsmonitor-env

diff --git a/fsmonitor.c b/fsmonitor.c
index 231e83a94d..7e27839278 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -14,8 +14,13 @@ struct trace_key trace_fsmonitor = TRACE_KEY_INIT(FSMONITOR);
 static void fsmonitor_ewah_callback(size_t pos, void *is)
 {
 	struct index_state *istate = (struct index_state *)is;
-	struct cache_entry *ce = istate->cache[pos];
+	struct cache_entry *ce;
+	
+	if (pos >= istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
+		    (uintmax_t)pos, istate->cache_nr);
 
+	ce = istate->cache[pos];
 	ce->ce_flags &= ~CE_FSMONITOR_VALID;
 }
 
@@ -50,17 +55,24 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
 	}
 	istate->fsmonitor_dirty = fsmonitor_dirty;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
+
 	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
 	return 0;
 }
 
 void fill_fsmonitor_bitmap(struct index_state *istate)
 {
-	unsigned int i;
+	unsigned int i, skipped = 0;
 	istate->fsmonitor_dirty = ewah_new();
-	for (i = 0; i < istate->cache_nr; i++)
-		if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
-			ewah_set(istate->fsmonitor_dirty, i);
+	for (i = 0; i < istate->cache_nr; i++) {
+		if (istate->cache[i]->ce_flags & CE_REMOVE)
+			skipped++;
+		else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
+			ewah_set(istate->fsmonitor_dirty, i - skipped);
+	}
 }
 
 void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
@@ -71,6 +83,10 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
 	uint32_t ewah_size = 0;
 	int fixup = 0;
 
+	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
+
 	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
 	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
 
@@ -236,6 +252,9 @@ void tweak_fsmonitor(struct index_state *istate)
 			}
 
 			/* Mark all previously saved entries as dirty */
+			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
 			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
 
 			/* Now mark the untracked cache for fsmonitor usage */
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index 81a375fa0f..87042470ab 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -354,4 +354,17 @@ test_expect_success 'discard_index() also discards fsmonitor info' '
 	test_cmp expect actual
 '
 
+# This test covers staging/unstaging files that appear at the end of the index.
+# Test files with names beginning with 'z' are used under the assumption that
+# earlier tests do not add/leave index entries that sort below them. 
+test_expect_success 'status succeeds after staging/unstaging ' '
+	test_commit initial &&
+	removed=$(test_seq 1 100 | sed "s/^/z/") &&
+	touch $removed &&
+	git add $removed &&
+	test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+	FSMONITOR_LIST="$removed" git restore -S $removed &&
+	FSMONITOR_LIST="$removed" git status
+'
+
 test_done
diff --git a/t/t7519/fsmonitor-env b/t/t7519/fsmonitor-env
new file mode 100755
index 0000000000..8f1f7ab164
--- /dev/null
+++ b/t/t7519/fsmonitor-env
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# An test hook script to integrate with git to test fsmonitor.
+#
+# The hook is passed a version (currently 1) and a time in nanoseconds
+# formatted as a string and outputs to stdout all files that have been
+# modified since the given time. Paths must be relative to the root of
+# the working tree and separated by a single NUL.
+#
+#echo "$0 $*" >&2
+
+if test "$#" -ne 2
+then
+	echo "$0: exactly 2 arguments expected" >&2
+	exit 2
+fi
+
+if test "$1" != 1
+then
+	echo "Unsupported core.fsmonitor hook version." >&2
+	exit 1
+fi
+
+printf '%s\n' $FSMONITOR_LIST
-- 
gitgitgadget

  reply	other threads:[~2019-10-09 21:00 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-03 19:49 [PATCH 0/1] fsmonitor: don't fill bitmap with entries to be removed William Baker via GitGitGadget
2019-10-03 19:49 ` [PATCH 1/1] " William Baker via GitGitGadget
2019-10-03 23:36   ` Junio C Hamano
2019-10-07 18:10     ` William Baker
2019-10-03 21:08 ` [PATCH 0/1] " Johannes Schindelin
2019-10-03 23:17   ` Junio C Hamano
2019-10-09 21:00 ` [PATCH v2 " William Baker via GitGitGadget
2019-10-09 21:00   ` William Baker via GitGitGadget [this message]
2019-10-10 11:07     ` [PATCH v2 1/1] " SZEDER Gábor
2019-10-10 11:22       ` SZEDER Gábor
2019-10-11 16:38         ` William Baker
2019-10-10  1:56   ` [PATCH v2 0/1] " Junio C Hamano
2019-10-10 12:02     ` Johannes Schindelin
2019-10-11 20:11   ` [PATCH v3 " William Baker via GitGitGadget
2019-10-11 20:11     ` [PATCH v3 1/1] " William Baker via GitGitGadget
2019-10-12  1:26       ` Junio C Hamano
2019-10-15 19:07         ` William Baker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=08741d986c2b51828f115ab50f178d62e9982978.1570654810.git.gitgitgadget@gmail.com \
    --to=gitgitgadget@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=William.Baker@microsoft.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jeffhost@microsoft.com \
    --cc=stolee@gmail.com \
    --cc=williamtbakeremail@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).