git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
* [PATCH 00/23] Builtin FSMonitor Part 3
@ 2022-02-15 15:59 Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 01/23] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
                   ` (25 more replies)
  0 siblings, 26 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler

Here is part 3 of my builtin FSMonitor series.

Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in
"seen", so this series should be considered a preview until part 2 moves to
"next". (And part 2 should not graduate to "master" without this part.)

Part 2 established the client code (used by commands like git status) and an
MVP implementation of the FSMonitor daemon. This was sufficient to test the
concepts and basic functionality.

Part 3 finishes the daemon and adds additional tests. This includes:

 * On Windows, handle short- and long-name aliasing.
 * On Mac, handle Unicode aliasing.
 * Mark bare, virtual, and remote working directories incompatible with
   FSMonitor.
 * On Mac, ignore xattr change FSEvents.
 * On Windows, cd out of the working directory root.
 * Create a thread to monitor the health and shutdown the daemon if
   necessary.
 * Speed up handling of directory notification events.
 * Test directory move and rename events.
 * Add performance test.

Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a
synthetic repo containing 1M files on a Macbook Pro. It shows the effects of
the untracked cache (uc) and FSMonitor (fsm) on git status.

$ ./p7527-builtin-fsmonitor.sh 
# passed all 67 test(s)
1..67
Test                                                                 this tree         
---------------------------------------------------------------------------------------
7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12) 
7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24) 
7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47) 
7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05) 
7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)
7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31) 
7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75) 
7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36) 

7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84) 
7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86) 
7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70) 
7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10) 
7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91) 
7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89) 
7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71) 
7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78) 

7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11) 
7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)
7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96) 
7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72) 
7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25) 
7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68) 
7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99) 
7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19) 

7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)   
7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64) 
7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92) 
7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)  
7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)   
7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49) 
7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)   
7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)   


Here is performance data from t/perf/p7519-fsmonitor.sh on the same
synthetic repo containing 1M files on a Macbook Pro. This test performs the
same series of commands on all three FSMonitor options: (1) using the hook
interface to talk to Watchman, (2) no FSMonitor, and (3) the builtin
FSMonitor daemon.

$ ./p7519-fsmonitor.sh 
# passed all 42 test(s)
1..42
Test                                                          this tree        
-------------------------------------------------------------------------------
7519.4: status (fsmonitor=fsmonitor-watchman)                  0.89(0.63+0.17)  
7519.5: status -uno (fsmonitor=fsmonitor-watchman)             0.56(0.43+0.13)  
7519.6: status -uall (fsmonitor=fsmonitor-watchman)           24.52(1.82+9.90) 
7519.7: status (dirty) (fsmonitor=fsmonitor-watchman)         32.40(6.54+9.94) 
7519.8: diff (fsmonitor=fsmonitor-watchman)                    0.66(0.38+0.42)  
7519.9: diff HEAD (fsmonitor=fsmonitor-watchman)               1.60(0.69+0.80)  
7519.10: diff -- 0_files (fsmonitor=fsmonitor-watchman)        0.37(0.29+0.08)  
7519.11: diff -- 10_files (fsmonitor=fsmonitor-watchman)       0.42(0.32+0.09)  
7519.12: diff -- 100_files (fsmonitor=fsmonitor-watchman)      0.42(0.32+0.10)  
7519.13: diff -- 1000_files (fsmonitor=fsmonitor-watchman)     0.41(0.31+0.09)  
7519.14: diff -- 10000_files (fsmonitor=fsmonitor-watchman)    0.44(0.32+0.12)  
7519.15: add (fsmonitor=fsmonitor-watchman)                   24.81(2.44+10.06)

7519.18: status (fsmonitor=disabled)                          11.18(1.80+74.68)
7519.19: status -uno (fsmonitor=disabled)                      7.11(1.50+73.95) 
7519.20: status -uall (fsmonitor=disabled)                    30.09(3.01+91.11)
7519.21: status (dirty) (fsmonitor=disabled)                  31.46(5.03+77.75)
7519.22: diff (fsmonitor=disabled)                             5.79(1.27+62.97) 
7519.23: diff HEAD (fsmonitor=disabled)                        7.29(1.57+76.03) 
7519.24: diff -- 0_files (fsmonitor=disabled)                  0.68(0.30+0.09)  
7519.25: diff -- 10_files (fsmonitor=disabled)                 0.32(0.29+0.07)  
7519.26: diff -- 100_files (fsmonitor=disabled)                0.32(0.29+0.07)  
7519.27: diff -- 1000_files (fsmonitor=disabled)               0.35(0.29+0.09)  
7519.28: diff -- 10000_files (fsmonitor=disabled)              0.56(0.29+0.21)  
7519.29: add (fsmonitor=disabled)                             29.49(3.52+85.88)

7519.31: status (builtin fsmonitor--daemon)                    0.67(0.54+0.09)  
7519.32: status -uno (builtin fsmonitor--daemon)               0.42(0.35+0.08)  
7519.33: status -uall (builtin fsmonitor--daemon)             22.77(1.71+10.35)
7519.34: status (dirty) (builtin fsmonitor--daemon)           21.79(4.18+9.97) 
7519.35: diff (builtin fsmonitor--daemon)                      0.52(0.34+0.36)  
7519.36: diff HEAD (builtin fsmonitor--daemon)                 1.55(0.66+0.80)  
7519.37: diff -- 0_files (builtin fsmonitor--daemon)           0.37(0.31+0.08)  
7519.38: diff -- 10_files (builtin fsmonitor--daemon)          0.35(0.29+0.07)  
7519.39: diff -- 100_files (builtin fsmonitor--daemon)         0.35(0.29+0.07)  
7519.40: diff -- 1000_files (builtin fsmonitor--daemon)        0.35(0.29+0.07)  
7519.41: diff -- 10000_files (builtin fsmonitor--daemon)       0.37(0.29+0.09)  
7519.42: add (builtin fsmonitor--daemon)                      24.64(2.40+10.04)


Jeff Hostetler (23):
  fsm-listen-win32: handle shortnames
  t7527: test FS event reporing on macOS WRT case and Unicode
  t7527: test builtin FSMonitor watching repos with unicode paths
  t/helper/fsmonitor-client: create stress test
  fsmonitor-settings: bare repos are incompatible with FSMonitor
  fsmonitor-settings: stub in platform-specific incompatibility checking
  fsmonitor-settings: virtual repos are incompatible with FSMonitor
  fsmonitor-settings: stub in macOS-specific incompatibility checking
  fsmonitor-settings: remote repos on macOS are incompatible with
    FSMonitor
  fsmonitor-settings: remote repos on Windows are incompatible with
    FSMonitor
  unpack-trees: initialize fsmonitor_has_run_once in o->result
  fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
  fsmonitor--daemon: print start message only if
    fsmonitor.announceStartup
  fsmonitor--daemon: cd out of worktree root
  fsmonitor--daemon: prepare for adding health thread
  fsmonitor--daemon: rename listener thread related variables
  fsmonitor--daemon: stub in health thread
  fsm-health-win32: add framework to monitor daemon health
  fsm-health-win32: force shutdown daemon if worktree root moves
  fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  fsmonitor: optimize processing of directory events
  t7527: FSMonitor tests for directory moves
  t/perf/p7527: add perf test for builtin FSMonitor

 Makefile                               |  19 +-
 builtin/fsmonitor--daemon.c            | 138 ++++++++-
 builtin/update-index.c                 |   9 +
 compat/fsmonitor/fsm-health-darwin.c   |  24 ++
 compat/fsmonitor/fsm-health-win32.c    | 257 ++++++++++++++++
 compat/fsmonitor/fsm-health.h          |  47 +++
 compat/fsmonitor/fsm-listen-darwin.c   |  89 +++++-
 compat/fsmonitor/fsm-listen-win32.c    | 409 ++++++++++++++++++++-----
 compat/fsmonitor/fsm-listen.h          |   2 +-
 compat/fsmonitor/fsm-settings-darwin.c |  75 +++++
 compat/fsmonitor/fsm-settings-win32.c  | 137 +++++++++
 config.mak.uname                       |   5 +
 contrib/buildsystems/CMakeLists.txt    |   8 +
 fsmonitor--daemon.h                    |  11 +-
 fsmonitor-settings.c                   |  84 +++++
 fsmonitor-settings.h                   |  26 ++
 fsmonitor.c                            |  71 ++++-
 t/helper/test-fsmonitor-client.c       | 105 +++++++
 t/perf/p7527-builtin-fsmonitor.sh      | 257 ++++++++++++++++
 t/t7519-status-fsmonitor.sh            |  35 +++
 t/t7527-builtin-fsmonitor.sh           | 157 ++++++++++
 unpack-trees.c                         |   1 +
 22 files changed, 1838 insertions(+), 128 deletions(-)
 create mode 100644 compat/fsmonitor/fsm-health-darwin.c
 create mode 100644 compat/fsmonitor/fsm-health-win32.c
 create mode 100644 compat/fsmonitor/fsm-health.h
 create mode 100644 compat/fsmonitor/fsm-settings-darwin.c
 create mode 100644 compat/fsmonitor/fsm-settings-win32.c
 create mode 100755 t/perf/p7527-builtin-fsmonitor.sh


base-commit: ce8aa175364c7a02115c4b4f91d5674f4d6d8ae8
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1143
-- 
gitgitgadget

^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 01/23] fsm-listen-win32: handle shortnames
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 14:48   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode Jeff Hostetler via GitGitGadget
                   ` (24 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach FSMonitor daemon on Windows to recognize shortname paths as
aliases of normal longname paths.  FSMonitor clients, such as `git
status`, should receive the longname spelling of changed files (when
possible).

Sometimes we receive FS events using the shortname, such as when a CMD
shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1".  The FS notification
arrives using whatever combination of long and shortnames were used by
the other process.  (Shortnames do seem to be case normalized,
however.)

Use Windows GetLongPathNameW() to try to map the pathname spelling in
the notification event into the normalized longname spelling.  (This
can fail if the file/directory is deleted, moved, or renamed, because
we are asking the FS for the mapping in response to the event and
after it has already happened, but we try.)

Special case the shortname spelling of ".git" to avoid under-reporting
these events.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-win32.c | 359 +++++++++++++++++++++++-----
 t/t7527-builtin-fsmonitor.sh        |  65 +++++
 2 files changed, 370 insertions(+), 54 deletions(-)

diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index c2d11acbc1e..eb407b0748c 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -25,6 +25,9 @@ struct one_watch
 	DWORD count;
 
 	struct strbuf path;
+	wchar_t wpath_longname[MAX_PATH + 1];
+	DWORD wpath_longname_len;
+
 	HANDLE hDir;
 	HANDLE hEvent;
 	OVERLAPPED overlapped;
@@ -34,6 +37,21 @@ struct one_watch
 	 * need to later call GetOverlappedResult() and possibly CancelIoEx().
 	 */
 	BOOL is_active;
+
+	/*
+	 * Are shortnames enabled on the containing drive?  This is
+	 * always true for "C:/" drives and usually never true for
+	 * other drives.
+	 *
+	 * We only set this for the worktree because we only need to
+	 * convert shortname paths to longname paths for items we send
+	 * to clients.  (We don't care about shortname expansion for
+	 * paths inside a GITDIR because we never send them to
+	 * clients.)
+	 */
+	BOOL has_shortnames;
+	BOOL has_tilda;
+	wchar_t dotgit_shortname[16]; /* for 8.3 name */
 };
 
 struct fsmonitor_daemon_backend_data
@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data
 };
 
 /*
- * Convert the WCHAR path from the notification into UTF8 and
- * then normalize it.
+ * Convert the WCHAR path from the event into UTF8 and normalize it.
+ *
+ * `wpath_len` is in WCHARS not bytes.
  */
-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,
 				  struct strbuf *normalized_path)
 {
 	int reserve;
 	int len = 0;
 
 	strbuf_reset(normalized_path);
-	if (!info->FileNameLength)
+	if (!wpath_len)
 		goto normalize;
 
 	/*
@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
 	 * sequence of 2 UTF8 characters.  That should let us
 	 * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.
 	 */
-	reserve = info->FileNameLength + 1;
+	reserve = 2 * wpath_len + 1;
 	strbuf_grow(normalized_path, reserve);
 
 	for (;;) {
-		len = WideCharToMultiByte(CP_UTF8, 0, info->FileName,
-					  info->FileNameLength / sizeof(WCHAR),
+		len = WideCharToMultiByte(CP_UTF8, 0,
+					  wpath, wpath_len,
 					  normalized_path->buf,
 					  strbuf_avail(normalized_path) - 1,
 					  NULL, NULL);
@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
 			goto normalize;
 		if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {
 			error("[GLE %ld] could not convert path to UTF-8: '%.*ls'",
-			      GetLastError(),
-			      (int)(info->FileNameLength / sizeof(WCHAR)),
-			      info->FileName);
+			      GetLastError(), (int)wpath_len, wpath);
 			return -1;
 		}
 
@@ -98,6 +115,148 @@ normalize:
 	return strbuf_normalize_path(normalized_path);
 }
 
+/*
+ * See if the worktree root directory has shortnames enabled.
+ * This will help us decide if we need to do an expensive shortname
+ * to longname conversion on every notification event.
+ *
+ * We do not want to create a file to test this, so we assume that the
+ * root directory contains a ".git" file or directory.  (Out caller
+ * only calls us for the worktree root, so this should be fine.)
+ *
+ * Remember the spelling of the shortname for ".git" if it exists.
+ */
+static void check_for_shortnames(struct one_watch *watch)
+{
+	wchar_t buf_in[MAX_PATH + 1];
+	wchar_t buf_out[MAX_PATH + 1];
+	wchar_t *last_slash = NULL;
+	wchar_t *last_bslash = NULL;
+	wchar_t *last;
+
+	/* build L"<wt-root-path>/.git" */
+	wcscpy(buf_in, watch->wpath_longname);
+	wcscpy(buf_in + watch->wpath_longname_len, L".git");
+
+	if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))
+		return;
+
+	last_slash = wcsrchr(buf_out, L'/');
+	last_bslash = wcsrchr(buf_out, L'\\');
+	if (last_slash > last_bslash)
+		last = last_slash + 1;
+	else if (last_bslash)
+		last = last_bslash + 1;
+	else
+		last = buf_out;
+
+	if (!wcscmp(last, L".git"))
+		return;
+
+	watch->has_shortnames = 1;
+	wcsncpy(watch->dotgit_shortname, last,
+		ARRAY_SIZE(watch->dotgit_shortname));
+
+	/*
+	 * The shortname for ".git" is usually of the form "GIT~1", so
+	 * we should be able to avoid shortname to longname mapping on
+	 * every notification event if the source string does not
+	 * contain a "~".
+	 *
+	 * However, the documentation for GetLongPathNameW() says
+	 * that there are filesystems that don't follow that pattern
+	 * and warns against this optimization.
+	 *
+	 * Lets test this.
+	 */
+	if (wcschr(watch->dotgit_shortname, L'~'))
+		watch->has_tilda = 1;
+}
+
+enum get_relative_result {
+	GRR_NO_CONVERSION_NEEDED,
+	GRR_HAVE_CONVERSION,
+	GRR_SHUTDOWN,
+};
+
+/*
+ * Info notification paths are relative to the root of the watch.
+ * If our CWD is still at the root, then we can use relative paths
+ * to convert from shortnames to longnames.  If our process has a
+ * different CWD, then we need to construct an absolute path, do
+ * the conversion, and then return the root-relative portion.
+ *
+ * We use the longname form of the root as our basis and assume that
+ * it already has a trailing slash.
+ *
+ * `wpath_len` is in WCHARS not bytes.
+ */
+static enum get_relative_result get_relative_longname(
+	struct one_watch *watch,
+	const wchar_t *wpath, DWORD wpath_len,
+	wchar_t *wpath_longname)
+{
+	wchar_t buf_in[2 * MAX_PATH + 1];
+	wchar_t buf_out[MAX_PATH + 1];
+	DWORD root_len;
+
+	/* Build L"<wt-root-path>/<event-rel-path>" */
+	root_len = watch->wpath_longname_len;
+	wcsncpy(buf_in, watch->wpath_longname, root_len);
+	wcsncpy(buf_in + root_len, wpath, wpath_len);
+	buf_in[root_len + wpath_len] = 0;
+
+	/*
+	 * We don't actually know if the source pathname is a
+	 * shortname or a longname.  This routine allows either to be
+	 * given as input.
+	 */
+	if (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {
+		/*
+		 * The shortname to longname conversion can fail for
+		 * various reasons, for example if the file has been
+		 * deleted.  (That is, if we just received a
+		 * delete-file notification event and the file is
+		 * already gone, we can't ask the file system to
+		 * lookup the longname for it.  Likewise, for moves
+		 * and renames where we are given the old name.)
+		 *
+		 * NEEDSWORK: Since deleting or moving a file or
+		 * directory by shortname is rather obscure, I'm going
+		 * ignore the failure and ask the caller to report the
+		 * original relative path.  This seemds kinder than
+		 * failing here and forcing a resync.
+		 */
+		return GRR_NO_CONVERSION_NEEDED;
+	}
+
+	if (!wcscmp(buf_in, buf_out)) {
+		/*
+		 * The path does not have a shortname alias.
+		 */
+		return GRR_NO_CONVERSION_NEEDED;
+	}
+
+	if (wcsncmp(buf_in, buf_out, root_len)) {
+		/*
+		 * The spelling of the root directory portion of the computed
+		 * longname has changed.  This should not happen.  Basically,
+		 * it means that we don't know where (without recomputing the
+		 * longname of just the root directory) to split out the
+		 * relative path.  Since this should not happen, I'm just
+		 * going to let this fail and force a shutdown (because all
+		 * subsequent events are probably going to see the same
+		 * mismatch).
+		 */
+		return GRR_SHUTDOWN;
+	}
+
+	/* Return the worktree root-relative portion of the longname. */
+
+	wcscpy(wpath_longname, buf_out + root_len);
+	return GRR_HAVE_CONVERSION;
+}
+
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
 	SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);
@@ -111,7 +270,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 	DWORD share_mode =
 		FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;
 	HANDLE hDir;
-	wchar_t wpath[MAX_PATH];
+	DWORD len_longname;
+	wchar_t wpath[MAX_PATH + 1];
+	wchar_t wpath_longname[MAX_PATH + 1];
 
 	if (xutftowcs_path(wpath, path) < 0) {
 		error(_("could not convert to wide characters: '%s'"), path);
@@ -128,6 +289,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 		return NULL;
 	}
 
+	if (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {
+		error(_("[GLE %ld] could not get longname of '%s'"),
+		      GetLastError(), path);
+		CloseHandle(hDir);
+		return NULL;
+	}
+
+	len_longname = wcslen(wpath_longname);
+	if (wpath_longname[len_longname - 1] != L'/' &&
+	    wpath_longname[len_longname - 1] != L'\\') {
+		wpath_longname[len_longname++] = L'/';
+		wpath_longname[len_longname] = 0;
+	}
+
 	CALLOC_ARRAY(watch, 1);
 
 	watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */
@@ -135,6 +310,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 	strbuf_init(&watch->path, 0);
 	strbuf_addstr(&watch->path, path);
 
+	wcscpy(watch->wpath_longname, wpath_longname);
+	watch->wpath_longname_len = len_longname;
+
 	watch->hDir = hDir;
 	watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
 
@@ -258,6 +436,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)
 	watch->is_active = FALSE;
 }
 
+/*
+ * Process a single relative pathname event.
+ * Return 1 if we should shutdown.
+ */
+static int process_1_worktree_event(
+	struct string_list *cookie_list,
+	struct fsmonitor_batch **batch,
+	const struct strbuf *path,
+	enum fsmonitor_path_type t,
+	DWORD info_action)
+{
+	const char *slash;
+
+	switch (t) {
+	case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
+		/* special case cookie files within .git */
+
+		/* Use just the filename of the cookie file. */
+		slash = find_last_dir_sep(path->buf);
+		string_list_append(cookie_list,
+				   slash ? slash + 1 : path->buf);
+		break;
+
+	case IS_INSIDE_DOT_GIT:
+		/* ignore everything inside of "<worktree>/.git/" */
+		break;
+
+	case IS_DOT_GIT:
+		/* "<worktree>/.git" was deleted (or renamed away) */
+		if ((info_action == FILE_ACTION_REMOVED) ||
+		    (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {
+			trace2_data_string("fsmonitor", NULL,
+					   "fsm-listen/dotgit",
+					   "removed");
+			return 1;
+		}
+		break;
+
+	case IS_WORKDIR_PATH:
+		/* queue normal pathname */
+		if (!*batch)
+			*batch = fsmonitor_batch__new();
+		fsmonitor_batch__add_path(*batch, path->buf);
+		break;
+
+	case IS_GITDIR:
+	case IS_INSIDE_GITDIR:
+	case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
+	default:
+		BUG("unexpected path classification '%d' for '%s'",
+		    t, path->buf);
+	}
+
+	return 0;
+}
+
 /*
  * Process filesystem events that happen anywhere (recursively) under the
  * <worktree> root directory.  For a normal working directory, this includes
@@ -274,6 +508,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
 	struct fsmonitor_batch *batch = NULL;
 	const char *p = watch->buffer;
+	wchar_t wpath_longname[MAX_PATH + 1];
 
 	/*
 	 * If the kernel gets more events than will fit in the kernel
@@ -306,54 +541,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)
 	 */
 	for (;;) {
 		FILE_NOTIFY_INFORMATION *info = (void *)p;
-		const char *slash;
+		wchar_t *wpath = info->FileName;
+		DWORD wpath_len = info->FileNameLength / sizeof(WCHAR);
 		enum fsmonitor_path_type t;
+		enum get_relative_result grr;
+
+		if (watch->has_shortnames) {
+			if (!wcscmp(wpath, watch->dotgit_shortname)) {
+				/*
+				 * This event exactly matches the
+				 * spelling of the shortname of
+				 * ".git", so we can skip some steps.
+				 *
+				 * (This case is odd because the user
+				 * can "rm -rf GIT~1" and we cannot
+				 * use the filesystem to map it back
+				 * to ".git".)
+				 */
+				strbuf_reset(&path);
+				strbuf_addstr(&path, ".git");
+				t = IS_DOT_GIT;
+				goto process_it;
+			}
 
-		strbuf_reset(&path);
-		if (normalize_path_in_utf8(info, &path) == -1)
-			goto skip_this_path;
-
-		t = fsmonitor_classify_path_workdir_relative(path.buf);
-
-		switch (t) {
-		case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
-			/* special case cookie files within .git */
-
-			/* Use just the filename of the cookie file. */
-			slash = find_last_dir_sep(path.buf);
-			string_list_append(&cookie_list,
-					   slash ? slash + 1 : path.buf);
-			break;
-
-		case IS_INSIDE_DOT_GIT:
-			/* ignore everything inside of "<worktree>/.git/" */
-			break;
+			if (watch->has_tilda && !wcschr(wpath, L'~')) {
+				/*
+				 * Shortnames on this filesystem have tildas
+				 * and the notification path does not have
+				 * one, so we assume that it is a longname.
+				 */
+				goto normalize_it;
+			}
 
-		case IS_DOT_GIT:
-			/* "<worktree>/.git" was deleted (or renamed away) */
-			if ((info->Action == FILE_ACTION_REMOVED) ||
-			    (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {
-				trace2_data_string("fsmonitor", NULL,
-						   "fsm-listen/dotgit",
-						   "removed");
+			grr = get_relative_longname(watch, wpath, wpath_len,
+						    wpath_longname);
+			switch (grr) {
+			case GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */
+				break;
+			case GRR_HAVE_CONVERSION:
+				wpath = wpath_longname;
+				wpath_len = wcslen(wpath);
+				break;
+			default:
+			case GRR_SHUTDOWN:
 				goto force_shutdown;
 			}
-			break;
+		}
 
-		case IS_WORKDIR_PATH:
-			/* queue normal pathname */
-			if (!batch)
-				batch = fsmonitor_batch__new();
-			fsmonitor_batch__add_path(batch, path.buf);
-			break;
+normalize_it:
+		if (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)
+			goto skip_this_path;
 
-		case IS_GITDIR:
-		case IS_INSIDE_GITDIR:
-		case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
-		default:
-			BUG("unexpected path classification '%d' for '%s'",
-			    t, path.buf);
-		}
+		t = fsmonitor_classify_path_workdir_relative(path.buf);
+
+process_it:
+		if (process_1_worktree_event(&cookie_list, &batch, &path, t,
+					     info->Action))
+			goto force_shutdown;
 
 skip_this_path:
 		if (!info->NextEntryOffset)
@@ -382,6 +626,9 @@ force_shutdown:
  * Note that we DO NOT get filesystem events on the external <gitdir>
  * itself (it is not inside something that we are watching).  In particular,
  * we do not get an event if the external <gitdir> is deleted.
+ *
+ * Also, we do not care about shortnames within the external <gitdir>, since
+ * we never send these paths to clients.
  */
 static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 {
@@ -403,8 +650,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 		const char *slash;
 		enum fsmonitor_path_type t;
 
-		strbuf_reset(&path);
-		if (normalize_path_in_utf8(info, &path) == -1)
+		if (normalize_path_in_utf8(
+			    info->FileName,
+			    info->FileNameLength / sizeof(WCHAR),
+			    &path) == -1)
 			goto skip_this_path;
 
 		t = fsmonitor_classify_path_gitdir_relative(path.buf);
@@ -538,6 +787,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 	if (!data->watch_worktree)
 		goto failed;
 
+	check_for_shortnames(data->watch_worktree);
+
 	if (state->nr_paths_watching > 1) {
 		data->watch_gitdir = create_watch(state,
 						  state->path_gitdir_watch.buf);
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 0ccbfb9616f..dbca7f835eb 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -123,6 +123,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '
 	test_must_fail git -C test_implicit_2 fsmonitor--daemon status
 '
 
+# File systems on Windows may or may not have shortnames.
+# This is a volume-specific setting on modern systems.
+# "C:/" drives are required to have them enabled.  Other
+# hard drives default to disabled.
+#
+# This is a crude test to see if shortnames are enabled
+# on the volume containing the test directory.  It is
+# crude, but it does not require elevation like `fsutil`.
+#
+test_lazy_prereq SHORTNAMES '
+	mkdir .foo &&
+	test -d "FOO~1"
+'
+
+# Here we assume that the shortname of ".git" is "GIT~1".
+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '
+	test_when_finished "stop_daemon_delete_repo test_implicit_1s" &&
+
+	git init test_implicit_1s &&
+
+	start_daemon test_implicit_1s &&
+
+	# renaming the .git directory will implicitly stop the daemon.
+	# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.
+	# the rename-from FS Event will contain the shortname.
+	#
+	mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&
+
+	sleep 1 &&
+	# put it back so that our status will not crawl out to our
+	# parent directory.
+	# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.
+	mv test_implicit_1s/.gitxyz test_implicit_1s/.git &&
+
+	test_must_fail git -C test_implicit_1s fsmonitor--daemon status
+'
+
+# Here we first create a file with LONGNAME of "GIT~1" before
+# we create the repo.  This will cause the shortname of ".git"
+# to be "GIT~2".
+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '
+	test_when_finished "stop_daemon_delete_repo test_implicit_1s2" &&
+
+	mkdir test_implicit_1s2 &&
+	echo HELLO >test_implicit_1s2/GIT~1 &&
+	git init test_implicit_1s2 &&
+
+	test_path_is_file test_implicit_1s2/GIT~1 &&
+	test_path_is_dir  test_implicit_1s2/GIT~2 &&
+
+	start_daemon test_implicit_1s2 &&
+
+	# renaming the .git directory will implicitly stop the daemon.
+	# the rename-from FS Event will contain the shortname.
+	#
+	mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&
+
+	sleep 1 &&
+	# put it back so that our status will not crawl out to our
+	# parent directory.
+	mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&
+
+	test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status
+'
+
 test_expect_success 'cannot start multiple daemons' '
 	test_when_finished "stop_daemon_delete_repo test_multiple" &&
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 01/23] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 14:52   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 03/23] t7527: test builtin FSMonitor watching repos with unicode paths Jeff Hostetler via GitGitGadget
                   ` (23 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Confirm that macOS FS events are reported with a normalized spelling.

APFS (and/or HFS+) is case-insensitive.  This means that case-independent
lookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed.  But that
doesn't tell us how FS events are reported if we try "rm -rf .git" versus
"rm -rf .GIT".  Are the events reported using the on-disk spelling of the
pathname or in the spelling used by the command.

NEEDSWORK: I was only able to test case.  It would be nice to add tests
that use different Unicode spellings/normalizations and understand the
differences between APFS and HFS+ in this area.  We should confirm that
the spelling of the workdir paths that the daemon sends to clients are
always properly normalized.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index dbca7f835eb..1fdabfc4f1e 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -188,6 +188,36 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '
 	test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status
 '
 
+# Confirm that MacOS hides all of the Unicode normalization and/or
+# case folding from the FS events.  That is, are the pathnames in the
+# FS events reported using the spelling on the disk or in the spelling
+# used by the other process.
+#
+# Note that we assume that the filesystem is set to case insensitive.
+#
+# NEEDSWORK: APFS handles Unicode and Unicode normalization
+# differently than HFS+.  I only have an APFS partition, so
+# more testing here would be helpful.
+#
+
+# Rename .git using alternate spelling and confirm that the daemon
+# sees the event using the correct spelling and shutdown.
+test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '
+	test_when_finished "stop_daemon_delete_repo test_apfs" &&
+
+	git init test_apfs &&
+	start_daemon test_apfs &&
+
+	test_path_is_dir test_apfs/.git &&
+	test_path_is_dir test_apfs/.GIT &&
+
+	mv test_apfs/.GIT test_apfs/.FOO &&
+	sleep 1 &&
+	mv test_apfs/.FOO test_apfs/.git &&
+
+	test_must_fail git -C test_apfs fsmonitor--daemon status
+'
+
 test_expect_success 'cannot start multiple daemons' '
 	test_when_finished "stop_daemon_delete_repo test_multiple" &&
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 03/23] t7527: test builtin FSMonitor watching repos with unicode paths
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 01/23] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 04/23] t/helper/fsmonitor-client: create stress test Jeff Hostetler via GitGitGadget
                   ` (22 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create some test repos with UTF8 pathnames and verify that
the builtin FSMonitor can watch them.  This test is mainly
for Windows where we need to avoid `*A()` routines.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 1fdabfc4f1e..c0145544ccb 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -696,4 +696,27 @@ do
 	done
 done
 
+# Test Unicode UTF-8 characters in the pathname of the working
+# directory.  Use of "*A()" routines rather than "*W()" routines
+# on Windows can sometimes lead to odd failures.
+#
+u1=$(printf "u_c3_a6__\xC3\xA6")
+u2=$(printf "u_e2_99_ab__\xE2\x99\xAB")
+u_values="$u1 $u2"
+for u in $u_values
+do
+	test_expect_success "Unicode path: $u" '
+		test_when_finished "stop_daemon_delete_repo $u" &&
+
+		git init "$u" &&
+		echo 1 >"$u"/file1 &&
+		git -C "$u" add file1 &&
+		git -C "$u" config core.fsmonitor true &&
+
+		start_daemon "$u" &&
+		git -C "$u" status >actual &&
+		grep "new file:   file1" actual
+	'
+done
+
 test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 04/23] t/helper/fsmonitor-client: create stress test
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (2 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 03/23] t7527: test builtin FSMonitor watching repos with unicode paths Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 14:58   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
                   ` (21 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create a stress test to hammer on the fsmonitor daemon.
Create a client-side thread pool of n threads and have
each of them make m requests as fast as they can.

NEEDSWORK: This is just the client-side thread pool and
is useful for interactive testing and experimentation.
We need to add a script test to drive this.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/helper/test-fsmonitor-client.c | 105 +++++++++++++++++++++++++++++++
 1 file changed, 105 insertions(+)

diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c
index f7a5b3a32fa..9dd2f9af553 100644
--- a/t/helper/test-fsmonitor-client.c
+++ b/t/helper/test-fsmonitor-client.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "parse-options.h"
 #include "fsmonitor-ipc.h"
+#include "thread-utils.h"
+#include "trace2.h"
 
 #ifndef HAVE_FSMONITOR_DAEMON_BACKEND
 int cmd__fsmonitor_client(int argc, const char **argv)
@@ -79,20 +81,120 @@ static int do_send_flush(void)
 	return 0;
 }
 
+struct hammer_thread_data
+{
+	pthread_t pthread_id;
+	int thread_nr;
+
+	int nr_requests;
+	const char *token;
+
+	int sum_successful;
+	int sum_errors;
+};
+
+static void *hammer_thread_proc(void *_hammer_thread_data)
+{
+	struct hammer_thread_data *data = _hammer_thread_data;
+	struct strbuf answer = STRBUF_INIT;
+	int k;
+	int ret;
+
+	trace2_thread_start("hammer");
+
+	for (k = 0; k < data->nr_requests; k++) {
+		strbuf_reset(&answer);
+
+		ret = fsmonitor_ipc__send_query(data->token, &answer);
+		if (ret < 0)
+			data->sum_errors++;
+		else
+			data->sum_successful++;
+	}
+
+	strbuf_release(&answer);
+	trace2_thread_exit();
+	return NULL;
+}
+
+/*
+ * Start a pool of client threads that will each send a series of
+ * commands to the daemon.
+ *
+ * The goal is to overload the daemon with a sustained series of
+ * concurrent requests.
+ */
+static int do_hammer(const char *token, int nr_threads, int nr_requests)
+{
+	struct hammer_thread_data *data = NULL;
+	int k;
+	int sum_join_errors = 0;
+	int sum_commands = 0;
+	int sum_errors = 0;
+
+	if (!token || !*token)
+		token = get_token_from_index();
+	if (nr_threads < 1)
+		nr_threads = 1;
+	if (nr_requests < 1)
+		nr_requests = 1;
+
+	CALLOC_ARRAY(data, nr_threads);
+
+	for (k = 0; k < nr_threads; k++) {
+		struct hammer_thread_data *p = &data[k];
+		p->thread_nr = k;
+		p->nr_requests = nr_requests;
+		p->token = token;
+
+		if (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {
+			warning("failed to create thread[%d] skipping remainder", k);
+			nr_threads = k;
+			break;
+		}
+	}
+
+	for (k = 0; k < nr_threads; k++) {
+		struct hammer_thread_data *p = &data[k];
+
+		if (pthread_join(p->pthread_id, NULL))
+			sum_join_errors++;
+		sum_commands += p->sum_successful;
+		sum_errors += p->sum_errors;
+	}
+
+	fprintf(stderr, "HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\n",
+		nr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);
+
+	free(data);
+
+	/*
+	 * TODO Decide if/when to return an error or call die().
+	 */
+	return 0;
+}
+
 int cmd__fsmonitor_client(int argc, const char **argv)
 {
 	const char *subcmd;
 	const char *token = NULL;
+	int nr_threads = 1;
+	int nr_requests = 1;
 
 	const char * const fsmonitor_client_usage[] = {
 		N_("test-helper fsmonitor-client query [<token>]"),
 		N_("test-helper fsmonitor-client flush"),
+		N_("test-helper fsmonitor-client hammer [<token>] [<threads>] [<requests>]"),
 		NULL,
 	};
 
 	struct option options[] = {
 		OPT_STRING(0, "token", &token, N_("token"),
 			   N_("command token to send to the server")),
+
+		OPT_INTEGER(0, "threads", &nr_threads, N_("number of client threads")),
+		OPT_INTEGER(0, "requests", &nr_requests, N_("number of requests per thread")),
+
 		OPT_END()
 	};
 
@@ -116,6 +218,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)
 	if (!strcmp(subcmd, "flush"))
 		return !!do_send_flush();
 
+	if (!strcmp(subcmd, "hammer"))
+		return !!do_hammer(token, nr_threads, nr_requests);
+
 	die("Unhandled subcommand: '%s'", subcmd);
 }
 #endif
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (3 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 04/23] t/helper/fsmonitor-client: create stress test Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-25 20:42   ` Ævar Arnfjörð Bjarmason
  2022-02-15 15:59 ` [PATCH 06/23] fsmonitor-settings: stub in platform-specific incompatibility checking Jeff Hostetler via GitGitGadget
                   ` (20 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Bare repos do not have a worktree, so there is nothing for the
daemon watch.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c | 11 +++++++
 builtin/update-index.c      |  9 ++++++
 fsmonitor-settings.c        | 62 +++++++++++++++++++++++++++++++++++++
 fsmonitor-settings.h        | 11 +++++++
 t/t7519-status-fsmonitor.sh | 26 ++++++++++++++++
 5 files changed, 119 insertions(+)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 6011fe42ee0..899355c55aa 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1424,6 +1424,17 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
 		die(_("invalid 'ipc-threads' value (%d)"),
 		    fsmonitor__ipc_threads);
 
+	prepare_repo_settings(the_repository);
+	fsm_settings__set_ipc(the_repository);
+
+	if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {
+		struct strbuf buf_reason = STRBUF_INIT;
+		fsm_settings__get_reason(the_repository, &buf_reason);
+		error("%s '%s'", buf_reason.buf, xgetcwd());
+		strbuf_release(&buf_reason);
+		return -1;
+	}
+
 	if (!strcmp(subcmd, "start"))
 		return !!try_to_start_background_daemon();
 
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 1232f0d214e..9a2c45f2e56 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1215,6 +1215,15 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 
 	if (fsmonitor > 0) {
 		enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
+
+		if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
+			struct strbuf buf_reason = STRBUF_INIT;
+			fsm_settings__get_reason(r, &buf_reason);
+			error("%s", buf_reason.buf);
+			strbuf_release(&buf_reason);
+			return -1;
+		}
+
 		if (fsm_mode == FSMONITOR_MODE_DISABLED) {
 			advise(_("core.fsmonitor is unset; "
 				 "set it if you really want to "
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index eb4d661c81e..0fc5566eb8a 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -9,9 +9,33 @@
  */
 struct fsmonitor_settings {
 	enum fsmonitor_mode mode;
+	enum fsmonitor_reason reason;
 	char *hook_path;
 };
 
+static void set_incompatible(struct repository *r,
+			     enum fsmonitor_reason reason)
+{
+	struct fsmonitor_settings *s = r->settings.fsmonitor;
+
+	s->mode = FSMONITOR_MODE_INCOMPATIBLE;
+	s->reason = reason;
+}
+
+static int check_for_incompatible(struct repository *r)
+{
+	if (!r->worktree) {
+		/*
+		 * Bare repositories don't have a working directory and
+		 * therefore have nothing to watch.
+		 */
+		set_incompatible(r, FSMONITOR_REASON_BARE);
+		return 1;
+	}
+
+	return 0;
+}
+
 static void lookup_fsmonitor_settings(struct repository *r)
 {
 	struct fsmonitor_settings *s;
@@ -78,6 +102,9 @@ void fsm_settings__set_ipc(struct repository *r)
 {
 	lookup_fsmonitor_settings(r);
 
+	if (check_for_incompatible(r))
+		return;
+
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 }
@@ -86,6 +113,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path)
 {
 	lookup_fsmonitor_settings(r);
 
+	if (check_for_incompatible(r))
+		return;
+
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 	r->settings.fsmonitor->hook_path = strdup(path);
@@ -96,5 +126,37 @@ void fsm_settings__set_disabled(struct repository *r)
 	lookup_fsmonitor_settings(r);
 
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;
+	r->settings.fsmonitor->reason = FSMONITOR_REASON_ZERO;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 }
+
+static void create_reason_message(struct repository *r,
+				  struct strbuf *buf_reason)
+{
+	struct fsmonitor_settings *s = r->settings.fsmonitor;
+
+	switch (s->reason) {
+	case FSMONITOR_REASON_ZERO:
+		return;
+
+	case FSMONITOR_REASON_BARE:
+		strbuf_addstr(buf_reason,
+			      _("bare repos are incompatible with fsmonitor"));
+		return;
+
+	default:
+		BUG("Unhandled case in create_reason_message '%d'", s->reason);
+	}
+}
+
+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
+					       struct strbuf *buf_reason)
+{
+	lookup_fsmonitor_settings(r);
+
+	strbuf_reset(buf_reason);
+	if (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)
+		create_reason_message(r, buf_reason);
+
+	return r->settings.fsmonitor->reason;
+}
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index a4c5d7b4889..e5f145a2f79 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -4,17 +4,28 @@
 struct repository;
 
 enum fsmonitor_mode {
+	FSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */
 	FSMONITOR_MODE_DISABLED = 0,
 	FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor=<hook_path> */
 	FSMONITOR_MODE_IPC = 2,  /* core.fsmonitor=<true> */
 };
 
+/*
+ * Incompatibility reasons.
+ */
+enum fsmonitor_reason {
+	FSMONITOR_REASON_ZERO = 0,
+	FSMONITOR_REASON_BARE = 1,
+};
+
 void fsm_settings__set_ipc(struct repository *r);
 void fsm_settings__set_hook(struct repository *r, const char *path);
 void fsm_settings__set_disabled(struct repository *r);
 
 enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);
 const char *fsm_settings__get_hook_path(struct repository *r);
+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
+					       struct strbuf *buf_reason);
 
 struct fsmonitor_settings;
 
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index f488d930dfd..3c4e6f5f89c 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -55,6 +55,32 @@ test_lazy_prereq UNTRACKED_CACHE '
 	test $ret -ne 1
 '
 
+# Test that we detect and disallow repos that are incompatible with FSMonitor.
+test_expect_success 'incompatible bare repo' '
+	test_when_finished "rm -rf ./bare-clone actual expect" &&
+	git init --bare bare-clone &&
+	cat >expect <<-\EOF &&
+	error: bare repos are incompatible with fsmonitor
+	EOF
+
+	test_must_fail \
+		git -C ./bare-clone -c core.fsmonitor=foo \
+			update-index --fsmonitor 2>actual &&
+	test_cmp expect actual &&
+
+	test_must_fail \
+		git -C ./bare-clone -c core.fsmonitor=true \
+			update-index --fsmonitor 2>actual &&
+	test_cmp expect actual
+'
+
+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
+	test_when_finished "rm -rf ./bare-clone actual" &&
+	git init --bare bare-clone &&
+	test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&
+	grep "bare repos are incompatible with fsmonitor" actual
+'
+
 test_expect_success 'setup' '
 	mkdir -p .git/hooks &&
 	: >tracked &&
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 06/23] fsmonitor-settings: stub in platform-specific incompatibility checking
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (4 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 15:05   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
                   ` (19 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Extend generic incompatibility checkout with platform-specific
mechanism.  Stub in Win32 version.

In the existing fsmonitor-settings code we have a way to mark
types of repos as incompatible with fsmonitor (whether via the
hook and ipc APIs).  For example, we do this for bare repos,
since there are no files to watch.

Extend this exclusion mechanism for platfor-specific reasons.
This commit just creates the framework and adds a stub for Win32.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 Makefile                              | 13 +++++++++++++
 compat/fsmonitor/fsm-settings-win32.c |  9 +++++++++
 config.mak.uname                      |  4 ++++
 contrib/buildsystems/CMakeLists.txt   |  3 +++
 fsmonitor-settings.c                  | 12 ++++++++++++
 fsmonitor-settings.h                  | 13 +++++++++++++
 6 files changed, 54 insertions(+)
 create mode 100644 compat/fsmonitor/fsm-settings-win32.c

diff --git a/Makefile b/Makefile
index 5ad0cef69f4..ed3d5231ea1 100644
--- a/Makefile
+++ b/Makefile
@@ -470,6 +470,11 @@ all::
 # `compat/fsmonitor/fsm-listen-<name>.c` that implements the
 # `fsm_listen__*()` routines.
 #
+# If your platform has os-specific ways to tell if a repo is incompatible with
+# fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS
+# to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c`
+# that implements the `fsm_os_settings__*()` routines.
+#
 # Define DEVELOPER to enable more compiler warnings. Compiler version
 # and family are auto detected, but could be overridden by defining
 # COMPILER_FEATURES (see config.mak.dev). You can still set
@@ -1941,6 +1946,11 @@ ifdef FSMONITOR_DAEMON_BACKEND
 	COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
 endif
 
+ifdef FSMONITOR_OS_SETTINGS
+	COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
+	COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
+endif
+
 ifeq ($(TCLTK_PATH),)
 NO_TCLTK = NoThanks
 endif
@@ -2819,6 +2829,9 @@ GIT-BUILD-OPTIONS: FORCE
 ifdef FSMONITOR_DAEMON_BACKEND
 	@echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+
 endif
+ifdef FSMONITOR_OS_SETTINGS
+	@echo FSMONITOR_OS_SETTINGS=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_OS_SETTINGS)))'\' >>$@+
+endif
 ifdef TEST_OUTPUT_DIRECTORY
 	@echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+
 endif
diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
new file mode 100644
index 00000000000..176a6f5726c
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -0,0 +1,9 @@
+#include "cache.h"
+#include "config.h"
+#include "repository.h"
+#include "fsmonitor-settings.h"
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
+{
+	return FSMONITOR_REASON_ZERO;
+}
diff --git a/config.mak.uname b/config.mak.uname
index 81f21d9e201..219a04b4c66 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -441,6 +441,8 @@ ifeq ($(uname_S),Windows)
 	# These are always available, so we do not have to conditionally
 	# support it.
 	FSMONITOR_DAEMON_BACKEND = win32
+	FSMONITOR_OS_SETTINGS = win32
+
 	NO_SVN_TESTS = YesPlease
 	RUNTIME_PREFIX = YesPlease
 	HAVE_WPGMPTR = YesWeDo
@@ -629,6 +631,8 @@ ifeq ($(uname_S),MINGW)
 	# These are always available, so we do not have to conditionally
 	# support it.
 	FSMONITOR_DAEMON_BACKEND = win32
+	FSMONITOR_OS_SETTINGS = win32
+
 	RUNTIME_PREFIX = YesPlease
 	HAVE_WPGMPTR = YesWeDo
 	NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 41395705ac6..7b8ea41967b 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)
 	if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+
+		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 0fc5566eb8a..e445572354e 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r)
 		return 1;
 	}
 
+#ifdef HAVE_FSMONITOR_OS_SETTINGS
+	{
+		enum fsmonitor_reason reason;
+
+		reason = fsm_os__incompatible(r);
+		if (reason != FSMONITOR_REASON_ZERO) {
+			set_incompatible(r, reason);
+			return 1;
+		}
+	}
+#endif
+
 	return 0;
 }
 
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index e5f145a2f79..b52bf8edaf1 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -29,4 +29,17 @@ enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
 
 struct fsmonitor_settings;
 
+#ifdef HAVE_FSMONITOR_OS_SETTINGS
+/*
+ * Ask platform-specific code whether the repository is incompatible
+ * with fsmonitor (both hook and ipc modes).  For example, if the working
+ * directory is on a remote volume and mounted via a technology that does
+ * not support notification events.
+ *
+ * fsm_os__* routines should considered private to fsm_settings__
+ * routines.
+ */
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);
+#endif /* HAVE_FSMONITOR_OS_SETTINGS */
+
 #endif /* FSMONITOR_SETTINGS_H */
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (5 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 06/23] fsmonitor-settings: stub in platform-specific incompatibility checking Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 15:11   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 08/23] fsmonitor-settings: stub in macOS-specific incompatibility checking Jeff Hostetler via GitGitGadget
                   ` (18 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Virtual repos, such as GVFS (aka VFS for Git), are incompatible
with FSMonitor.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++
 fsmonitor-settings.c                  |  5 +++++
 fsmonitor-settings.h                  |  1 +
 t/t7519-status-fsmonitor.sh           |  9 +++++++++
 4 files changed, 41 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
index 176a6f5726c..7caa79570af 100644
--- a/compat/fsmonitor/fsm-settings-win32.c
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -3,7 +3,33 @@
 #include "repository.h"
 #include "fsmonitor-settings.h"
 
+/*
+ * GVFS (aka VFS for Git) is incompatible with FSMonitor.
+ *
+ * Granted, core Git does not know anything about GVFS and we
+ * shouldn't make assumptions about a downstream feature, but users
+ * can install both versions.  And this can lead to incorrect results
+ * from core Git commands.  So, without bringing in any of the GVFS
+ * code, do a simple config test for a published config setting.  (We
+ * do not look at the various *_TEST_* environment variables.)
+ */
+static enum fsmonitor_reason is_virtual(struct repository *r)
+{
+	const char *const_str;
+
+	if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str))
+		return FSMONITOR_REASON_VIRTUAL;
+
+	return FSMONITOR_REASON_ZERO;
+}
+
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
+	enum fsmonitor_reason reason;
+
+	reason = is_virtual(r);
+	if (reason)
+		return reason;
+
 	return FSMONITOR_REASON_ZERO;
 }
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index e445572354e..bb2ddd2457f 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -156,6 +156,11 @@ static void create_reason_message(struct repository *r,
 			      _("bare repos are incompatible with fsmonitor"));
 		return;
 
+	case FSMONITOR_REASON_VIRTUAL:
+		strbuf_addstr(buf_reason,
+			      _("virtual repos are incompatible with fsmonitor"));
+		return;
+
 	default:
 		BUG("Unhandled case in create_reason_message '%d'", s->reason);
 	}
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index b52bf8edaf1..c169683bf2d 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -16,6 +16,7 @@ enum fsmonitor_mode {
 enum fsmonitor_reason {
 	FSMONITOR_REASON_ZERO = 0,
 	FSMONITOR_REASON_BARE = 1,
+	FSMONITOR_REASON_VIRTUAL = 2,
 };
 
 void fsm_settings__set_ipc(struct repository *r);
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index 3c4e6f5f89c..d7540931a16 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -81,6 +81,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
 	grep "bare repos are incompatible with fsmonitor" actual
 '
 
+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '
+	test_when_finished "rm -rf ./fake-virtual-clone actual" &&
+	git init fake-virtual-clone &&
+	test_must_fail git -C ./fake-virtual-clone \
+			   -c core.virtualfilesystem=true \
+			   fsmonitor--daemon run 2>actual &&
+	grep "virtual repos are incompatible with fsmonitor" actual
+'
+
 test_expect_success 'setup' '
 	mkdir -p .git/hooks &&
 	: >tracked &&
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 08/23] fsmonitor-settings: stub in macOS-specific incompatibility checking
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (6 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
                   ` (17 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++
 config.mak.uname                       | 1 +
 contrib/buildsystems/CMakeLists.txt    | 3 +++
 3 files changed, 13 insertions(+)
 create mode 100644 compat/fsmonitor/fsm-settings-darwin.c

diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
new file mode 100644
index 00000000000..176a6f5726c
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -0,0 +1,9 @@
+#include "cache.h"
+#include "config.h"
+#include "repository.h"
+#include "fsmonitor-settings.h"
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
+{
+	return FSMONITOR_REASON_ZERO;
+}
diff --git a/config.mak.uname b/config.mak.uname
index 219a04b4c66..529d58aae5d 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -159,6 +159,7 @@ ifeq ($(uname_S),Darwin)
 	ifndef NO_PTHREADS
 	ifndef NO_UNIX_SOCKETS
 	FSMONITOR_DAEMON_BACKEND = darwin
+	FSMONITOR_OS_SETTINGS = darwin
 	endif
 	endif
 
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 7b8ea41967b..fc70dd2fc1d 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
+
+		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
 	endif()
 endif()
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (7 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 08/23] fsmonitor-settings: stub in macOS-specific incompatibility checking Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 15:26   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 10/23] fsmonitor-settings: remote repos on Windows " Jeff Hostetler via GitGitGadget
                   ` (16 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to detect remote working directories on macOS and mark them as
incompatible with FSMonitor.

With this, `git fsmonitor--daemon run` will error out with a message
like it does for bare repos.

Client commands, like `git status`, will not attempt to start the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++
 fsmonitor-settings.c                   |  5 ++
 fsmonitor-settings.h                   |  1 +
 3 files changed, 72 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 176a6f5726c..c3b719d6fb0 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -2,8 +2,74 @@
 #include "config.h"
 #include "repository.h"
 #include "fsmonitor-settings.h"
+#include "fsmonitor.h"
+#include <sys/param.h>
+#include <sys/mount.h>
+
+/*
+ * Remote working directories are problematic for FSMonitor.
+ *
+ * The underlying file system on the server machine and/or the remote
+ * mount type (NFS, SAMBA, etc.) dictates whether notification events
+ * are available at all to remote client machines.
+ *
+ * Kernel differences between the server and client machines also
+ * dictate the how (buffering, frequency, de-dup) the events are
+ * delivered to client machine processes.
+ *
+ * A client machine (such as a laptop) may choose to suspend/resume
+ * and it is unclear (without lots of testing) whether the watcher can
+ * resync after a resume.  We might be able to treat this as a normal
+ * "events were dropped by the kernel" event and do our normal "flush
+ * and resync" --or-- we might need to close the existing (zombie?)
+ * notification fd and create a new one.
+ *
+ * In theory, the above issues need to be addressed whether we are
+ * using the Hook or IPC API.
+ *
+ * For the builtin FSMonitor, we create the Unix domain socket for the
+ * IPC in the .git directory.  If the working directory is remote,
+ * then the socket will be created on the remote file system.  This
+ * can fail if the remote file system does not support UDS file types
+ * (e.g. smbfs to a Windows server) or if the remote kernel does not
+ * allow a non-local process to bind() the socket.  (These problems
+ * could be fixed by moving the UDS out of the .git directory and to a
+ * well-known local directory on the client machine, but care should
+ * be taken to ensure that $HOME is actually local and not a managed
+ * file share.)
+ *
+ * So (for now at least), mark remote working directories as
+ * incompatible.
+ */
+static enum fsmonitor_reason is_remote(struct repository *r)
+{
+	struct statfs fs;
+
+	if (statfs(r->worktree, &fs) == -1) {
+		int saved_errno = errno;
+		trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s",
+				 r->worktree, strerror(saved_errno));
+		errno = saved_errno;
+		return FSMONITOR_REASON_ZERO;
+	}
+
+	trace_printf_key(&trace_fsmonitor,
+			 "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'",
+			 r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);
+
+	if (!(fs.f_flags & MNT_LOCAL))
+		return FSMONITOR_REASON_REMOTE;
+
+	return FSMONITOR_REASON_ZERO;
+}
 
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
+	enum fsmonitor_reason reason;
+
+	reason = is_remote(r);
+	if (reason)
+		return reason;
+
 	return FSMONITOR_REASON_ZERO;
 }
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index bb2ddd2457f..de69ace246a 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -161,6 +161,11 @@ static void create_reason_message(struct repository *r,
 			      _("virtual repos are incompatible with fsmonitor"));
 		return;
 
+	case FSMONITOR_REASON_REMOTE:
+		strbuf_addstr(buf_reason,
+			      _("remote repos are incompatible with fsmonitor"));
+		return;
+
 	default:
 		BUG("Unhandled case in create_reason_message '%d'", s->reason);
 	}
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index c169683bf2d..fca25887c0f 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -17,6 +17,7 @@ enum fsmonitor_reason {
 	FSMONITOR_REASON_ZERO = 0,
 	FSMONITOR_REASON_BARE = 1,
 	FSMONITOR_REASON_VIRTUAL = 2,
+	FSMONITOR_REASON_REMOTE = 3,
 };
 
 void fsm_settings__set_ipc(struct repository *r);
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 10/23] fsmonitor-settings: remote repos on Windows are incompatible with FSMonitor
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (8 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 15:56   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 11/23] unpack-trees: initialize fsmonitor_has_run_once in o->result Jeff Hostetler via GitGitGadget
                   ` (15 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to detect remote working directories on Windows and mark them as
incompatible with FSMonitor.

With this `git fsmonitor--daemon run` will error out with a message like it
does for bare repos.

Client commands, such as `git status`, will not attempt to start the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++
 1 file changed, 102 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
index 7caa79570af..77156c0ca20 100644
--- a/compat/fsmonitor/fsm-settings-win32.c
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -2,6 +2,7 @@
 #include "config.h"
 #include "repository.h"
 #include "fsmonitor-settings.h"
+#include "fsmonitor.h"
 
 /*
  * GVFS (aka VFS for Git) is incompatible with FSMonitor.
@@ -23,6 +24,103 @@ static enum fsmonitor_reason is_virtual(struct repository *r)
 	return FSMONITOR_REASON_ZERO;
 }
 
+/*
+ * Remote working directories are problematic for FSMonitor.
+ *
+ * The underlying file system on the server machine and/or the remote
+ * mount type dictates whether notification events are available at
+ * all to remote client machines.
+ *
+ * Kernel differences between the server and client machines also
+ * dictate the how (buffering, frequency, de-dup) the events are
+ * delivered to client machine processes.
+ *
+ * A client machine (such as a laptop) may choose to suspend/resume
+ * and it is unclear (without lots of testing) whether the watcher can
+ * resync after a resume.  We might be able to treat this as a normal
+ * "events were dropped by the kernel" event and do our normal "flush
+ * and resync" --or-- we might need to close the existing (zombie?)
+ * notification fd and create a new one.
+ *
+ * In theory, the above issues need to be addressed whether we are
+ * using the Hook or IPC API.
+ *
+ * So (for now at least), mark remote working directories as
+ * incompatible.
+ *
+ * Notes for testing:
+ *
+ * (a) Windows allows a network share to be mapped to a drive letter.
+ *     (This is the normal method to access it.)
+ *
+ *     $ NET USE Z: \\server\share
+ *     $ git -C Z:/repo status
+ *
+ * (b) Windows allows a network share to be referenced WITHOUT mapping
+ *     it to drive letter.
+ *
+ *     $ NET USE \\server\share\dir
+ *     $ git -C //server/share/repo status
+ *
+ * (c) Windows allows "SUBST" to create a fake drive mapping to an
+ *     arbitrary path (which may be remote)
+ *
+ *     $ SUBST Q: Z:\repo
+ *     $ git -C Q:/ status
+ *
+ * (d) Windows allows a directory symlink to be created on a local
+ *     file system that points to a remote repo.
+ *
+ *     $ mklink /d ./link //server/share/repo
+ *     $ git -C ./link status
+ */
+static enum fsmonitor_reason is_remote(struct repository *r)
+{
+	wchar_t wpath[MAX_PATH];
+	wchar_t wfullpath[MAX_PATH];
+	size_t wlen;
+	UINT driveType;
+
+	/*
+	 * Do everything in wide chars because the drive letter might be
+	 * a multi-byte sequence.  See win32_has_dos_drive_prefix().
+	 */
+	if (xutftowcs_path(wpath, r->worktree) < 0)
+		return FSMONITOR_REASON_ZERO;
+
+	/*
+	 * GetDriveTypeW() requires a final slash.  We assume that the
+	 * worktree pathname points to an actual directory.
+	 */
+	wlen = wcslen(wpath);
+	if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') {
+		wpath[wlen++] = L'\\';
+		wpath[wlen] = 0;
+	}
+
+	/*
+	 * Normalize the path.  If nothing else, this converts forward
+	 * slashes to backslashes.  This is essential to get GetDriveTypeW()
+	 * correctly handle some UNC "\\server\share\..." paths.
+	 */
+	if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))
+		return FSMONITOR_REASON_ZERO;
+
+	driveType = GetDriveTypeW(wfullpath);
+	trace_printf_key(&trace_fsmonitor,
+			 "DriveType '%s' L'%ls' (%u)",
+			 r->worktree, wfullpath, driveType);
+
+	if (driveType == DRIVE_REMOTE) {
+		trace_printf_key(&trace_fsmonitor,
+				 "is_remote('%s') true",
+				 r->worktree);
+		return FSMONITOR_REASON_REMOTE;
+	}
+
+	return FSMONITOR_REASON_ZERO;
+}
+
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
 	enum fsmonitor_reason reason;
@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 	if (reason)
 		return reason;
 
+	reason = is_remote(r);
+	if (reason)
+		return reason;
+
 	return FSMONITOR_REASON_ZERO;
 }
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 11/23] unpack-trees: initialize fsmonitor_has_run_once in o->result
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (9 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 10/23] fsmonitor-settings: remote repos on Windows " Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 12/23] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Jeff Hostetler via GitGitGadget
                   ` (14 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Initialize `o->result.fsmonitor_has_run_once` based upon value
in `o->src_index->fsmonitor_has_run_once` to prevent a second
fsmonitor query during the tree traversal and possibly getting
a skewed view of the working directory.

The checkout code has already talked to the fsmonitor and the
traversal is updating the index as it traverses, so there is
no need to query the fsmonitor.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 unpack-trees.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/unpack-trees.c b/unpack-trees.c
index a7e1712d236..13e2e5de82c 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1756,6 +1756,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 
 	o->result.fsmonitor_last_update =
 		xstrdup_or_null(o->src_index->fsmonitor_last_update);
+	o->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;
 
 	/*
 	 * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 12/23] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (10 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 11/23] unpack-trees: initialize fsmonitor_has_run_once in o->result Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 13/23] fsmonitor--daemon: print start message only if fsmonitor.announceStartup Jeff Hostetler via GitGitGadget
                   ` (13 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Ignore FSEvents resulting from `xattr` changes.  Git does not care about
xattr's or changes to xattr's, so don't waste time collecting these
events in the daemon nor transmitting them to clients.

Various security tools add xattrs to files and/or directories, such as
to mark them as having been downloaded.  We should ignore these events
since it doesn't affect the content of the file/directory or the normal
meta-data that Git cares about.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 2aefdc14d89..79d08517d7b 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -172,7 +172,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)
 	if (flag & kFSEventStreamEventFlagItemCloned)
 		strbuf_addstr(&msg, "ItemCloned|");
 
-	trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s",
+	trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s",
 			 path, flag, msg.buf);
 
 	strbuf_release(&msg);
@@ -197,6 +197,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)
 		ef & kFSEventStreamEventFlagUserDropped);
 }
 
+/*
+ * If an `xattr` change is the only reason we received this event,
+ * then silently ignore it.  Git doesn't care about xattr's.  We
+ * have to be careful here because the kernel can combine multiple
+ * events for a single path.  And because events always have certain
+ * bits set, such as `ItemIsFile` or `ItemIsDir`.
+ *
+ * Return 1 if we should ignore it.
+ */
+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)
+{
+	static const FSEventStreamEventFlags mask =
+		kFSEventStreamEventFlagItemChangeOwner |
+		kFSEventStreamEventFlagItemCreated |
+		kFSEventStreamEventFlagItemFinderInfoMod |
+		kFSEventStreamEventFlagItemInodeMetaMod |
+		kFSEventStreamEventFlagItemModified |
+		kFSEventStreamEventFlagItemRemoved |
+		kFSEventStreamEventFlagItemRenamed |
+		kFSEventStreamEventFlagItemXattrMod |
+		kFSEventStreamEventFlagItemCloned;
+
+	return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);
+}
+
 static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			     void *ctx,
 			     size_t num_of_events,
@@ -262,6 +287,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			continue;
 		}
 
+		if (ef_ignore_xattr(event_flags[k])) {
+			trace_printf_key(&trace_fsmonitor,
+					 "ignore-xattr: '%s', flags=0x%x",
+					 path_k, event_flags[k]);
+			continue;
+		}
+
 		switch (fsmonitor_classify_path_absolute(state, path_k)) {
 
 		case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 13/23] fsmonitor--daemon: print start message only if fsmonitor.announceStartup
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (11 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 12/23] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 14/23] fsmonitor--daemon: cd out of worktree root Jeff Hostetler via GitGitGadget
                   ` (12 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach fsmonitor--daemon to print a startup message only when
`fsmonitor.announceStartup` is true.  This setting is false by default
so that the output of client commands, like `git status`, is not
changed if the daemon is implicitly started.

The message is conditionally printed by "run" and "start" subcommands
and is sent to stderr.  It contains the path to the work tree root.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 899355c55aa..dd0561cfc51 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -27,6 +27,9 @@ static int fsmonitor__ipc_threads = 8;
 #define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout"
 static int fsmonitor__start_timeout_sec = 60;
 
+#define FSMONITOR__ANNOUNCE_STARTUP "fsmonitor.announcestartup"
+static int fsmonitor__announce_startup = 0;
+
 static int fsmonitor_config(const char *var, const char *value, void *cb)
 {
 	if (!strcmp(var, FSMONITOR__IPC_THREADS)) {
@@ -47,6 +50,16 @@ static int fsmonitor_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) {
+		int is_bool;
+		int i = git_config_bool_or_int(var, value, &is_bool);
+		if (i < 0)
+			return error(_("value of '%s' not bool or int: %d"),
+				     var, i);
+		fsmonitor__announce_startup = i;
+		return 0;
+	}
+
 	return git_default_config(var, value, cb);
 }
 
@@ -1307,9 +1320,11 @@ static int try_to_run_foreground_daemon(int free_console)
 		die("fsmonitor--daemon is already running '%s'",
 		    the_repository->worktree);
 
-	printf(_("running fsmonitor-daemon in '%s'\n"),
-	       the_repository->worktree);
-	fflush(stdout);
+	if (fsmonitor__announce_startup) {
+		fprintf(stderr, _("running fsmonitor-daemon in '%s'\n"),
+			the_repository->worktree);
+		fflush(stderr);
+	}
 
 #ifdef GIT_WINDOWS_NATIVE
 	if (free_console)
@@ -1360,9 +1375,11 @@ static int try_to_start_background_daemon(void)
 		die("fsmonitor--daemon is already running '%s'",
 		    the_repository->worktree);
 
-	printf(_("starting fsmonitor-daemon in '%s'\n"),
-	       the_repository->worktree);
-	fflush(stdout);
+	if (fsmonitor__announce_startup) {
+		fprintf(stderr, _("starting fsmonitor-daemon in '%s'\n"),
+			the_repository->worktree);
+		fflush(stderr);
+	}
 
 	cp.git_cmd = 1;
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 14/23] fsmonitor--daemon: cd out of worktree root
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (12 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 13/23] fsmonitor--daemon: print start message only if fsmonitor.announceStartup Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 15/23] fsmonitor--daemon: prepare for adding health thread Jeff Hostetler via GitGitGadget
                   ` (11 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach the fsmonitor--daemon to CD outside of the worktree
before starting up.

The common Git startup mechanism causes the CWD of the daemon process
to be in the root of the worktree.  On Windows, this causes the daemon
process to hold a locked handle on the CWD and prevents other
processes from moving or deleting the worktree while the daemon is
running.

CD to HOME before entering main event loops.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c         | 32 +++++++++++++++++++++++++++--
 compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------
 fsmonitor--daemon.h                 |  1 +
 3 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index dd0561cfc51..e9b3f44d791 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1169,11 +1169,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	 * before we need it.
 	 */
 	if (ipc_server_run_async(&state->ipc_server_data,
-				 fsmonitor_ipc__get_path(), &ipc_opts,
+				 state->path_ipc.buf, &ipc_opts,
 				 handle_client, state))
 		return error_errno(
 			_("could not start IPC thread pool on '%s'"),
-			fsmonitor_ipc__get_path());
+			state->path_ipc.buf);
 
 	/*
 	 * Start the fsmonitor listener thread to collect filesystem
@@ -1208,6 +1208,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 static int fsmonitor_run_daemon(void)
 {
 	struct fsmonitor_daemon_state state;
+	const char *home;
 	int err;
 
 	memset(&state, 0, sizeof(state));
@@ -1277,6 +1278,15 @@ static int fsmonitor_run_daemon(void)
 
 	strbuf_addch(&state.path_cookie_prefix, '/');
 
+	/*
+	 * We create a named-pipe or unix domain socket inside of the
+	 * ".git" directory.  (Well, on Windows, we base our named
+	 * pipe in the NPFS on the absolute path of the git
+	 * directory.)
+	 */
+	strbuf_init(&state.path_ipc, 0);
+	strbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));
+
 	/*
 	 * Confirm that we can create platform-specific resources for the
 	 * filesystem listener before we bother starting all the threads.
@@ -1286,6 +1296,23 @@ static int fsmonitor_run_daemon(void)
 		goto done;
 	}
 
+	/*
+	 * CD out of the worktree root directory.
+	 *
+	 * The common Git startup mechanism causes our CWD to be the
+	 * root of the worktree.  On Windows, this causes our process
+	 * to hold a locked handle on the CWD.  This prevents the
+	 * worktree from being moved or deleted while the daemon is
+	 * running.
+	 *
+	 * We assume that our FS and IPC listener threads have either
+	 * opened all of the handles that they need or will do
+	 * everything using absolute paths.
+	 */
+	home = getenv("HOME");
+	if (home && *home && chdir(home))
+		die_errno("could not cd home '%s'", home);
+
 	err = fsmonitor_run_daemon_1(&state);
 
 done:
@@ -1298,6 +1325,7 @@ done:
 	strbuf_release(&state.path_worktree_watch);
 	strbuf_release(&state.path_gitdir_watch);
 	strbuf_release(&state.path_cookie_prefix);
+	strbuf_release(&state.path_ipc);
 
 	/*
 	 * NEEDSWORK: Consider "rm -rf <gitdir>/<fsmonitor-dir>"
diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index eb407b0748c..6985903c95b 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -398,12 +398,22 @@ static int recv_rdcw_watch(struct one_watch *watch)
 	}
 
 	/*
-	 * NEEDSWORK: If an external <gitdir> is deleted, the above
-	 * returns an error.  I'm not sure that there's anything that
-	 * we can do here other than failing -- the <worktree>/.git
-	 * link file would be broken anyway.  We might try to check
-	 * for that and return a better error message, but I'm not
-	 * sure it is worth it.
+	 * GetOverlappedResult() fails if the watched directory is
+	 * deleted while we were waiting for an overlapped IO to
+	 * complete.  The documentation did not list specific errors,
+	 * but I observed ERROR_ACCESS_DENIED (0x05) errors during
+	 * testing.
+	 *
+	 * Note that we only get notificaiton events for events
+	 * *within* the directory, not *on* the directory itself.
+	 * (These might be properies of the parent directory, for
+	 * example).
+	 *
+	 * NEEDSWORK: We might try to check for the deleted directory
+	 * case and return a better error message, but I'm not sure it
+	 * is worth it.
+	 *
+	 * Shutdown if we get any error.
 	 */
 
 	error("GetOverlappedResult failed on '%s' [GLE %ld]",
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index c16ef095688..a777c3a0590 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {
 	struct fsmonitor_daemon_backend_data *backend_data;
 
 	struct ipc_server_data *ipc_server_data;
+	struct strbuf path_ipc;
 };
 
 /*
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 15/23] fsmonitor--daemon: prepare for adding health thread
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (13 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 14/23] fsmonitor--daemon: cd out of worktree root Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 16/23] fsmonitor--daemon: rename listener thread related variables Jeff Hostetler via GitGitGadget
                   ` (10 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Refactor daemon thread startup to make it easier to start
a third thread class to monitor the health of the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index e9b3f44d791..f42fb2ab626 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1162,6 +1162,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 		 */
 		.uds_disallow_chdir = 0
 	};
+	int listener_started = 0;
+	int err = 0;
 
 	/*
 	 * Start the IPC thread pool before the we've started the file
@@ -1182,15 +1184,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	if (pthread_create(&state->listener_thread, NULL,
 			   fsm_listen__thread_proc, state) < 0) {
 		ipc_server_stop_async(state->ipc_server_data);
-		ipc_server_await(state->ipc_server_data);
-
-		return error(_("could not start fsmonitor listener thread"));
+		err = error(_("could not start fsmonitor listener thread"));
+		goto cleanup;
 	}
+	listener_started = 1;
 
 	/*
 	 * The daemon is now fully functional in background threads.
+	 * Our primary thread should now just wait while the threads
+	 * do all the work.
+	 */
+cleanup:
+	/*
 	 * Wait for the IPC thread pool to shutdown (whether by client
-	 * request or from filesystem activity).
+	 * request, from filesystem activity, or an error).
 	 */
 	ipc_server_await(state->ipc_server_data);
 
@@ -1199,10 +1206,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	 * event from the IPC thread pool, but it doesn't hurt to tell
 	 * it again.  And wait for it to shutdown.
 	 */
-	fsm_listen__stop_async(state);
-	pthread_join(state->listener_thread, NULL);
+	if (listener_started) {
+		fsm_listen__stop_async(state);
+		pthread_join(state->listener_thread, NULL);
+	}
 
-	return state->error_code;
+	if (err)
+		return err;
+	if (state->error_code)
+		return state->error_code;
+	return 0;
 }
 
 static int fsmonitor_run_daemon(void)
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 16/23] fsmonitor--daemon: rename listener thread related variables
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (14 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 15/23] fsmonitor--daemon: prepare for adding health thread Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 17/23] fsmonitor--daemon: stub in health thread Jeff Hostetler via GitGitGadget
                   ` (9 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Rename platform-specific listener thread related variables
and data types as we prepare to add another backend thread
type.

[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`
[] `state->backend_data` becomes `state->listen_data`
[] `state->error_code` becomes `state->listen_error_code`

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c          |  6 +++---
 compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------
 compat/fsmonitor/fsm-listen-win32.c  | 28 +++++++++++++-------------
 compat/fsmonitor/fsm-listen.h        |  2 +-
 fsmonitor--daemon.h                  |  6 +++---
 5 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index f42fb2ab626..f3fb865a9d8 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1213,8 +1213,8 @@ cleanup:
 
 	if (err)
 		return err;
-	if (state->error_code)
-		return state->error_code;
+	if (state->listen_error_code)
+		return state->listen_error_code;
 	return 0;
 }
 
@@ -1229,7 +1229,7 @@ static int fsmonitor_run_daemon(void)
 	hashmap_init(&state.cookies, cookies_cmp, NULL, 0);
 	pthread_mutex_init(&state.main_lock, NULL);
 	pthread_cond_init(&state.cookies_cond, NULL);
-	state.error_code = 0;
+	state.listen_error_code = 0;
 	state.current_token_data = fsmonitor_new_token_data();
 
 	/* Prepare to (recursively) watch the <worktree-root> directory. */
diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 79d08517d7b..87a8476b09f 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -99,7 +99,7 @@ void FSEventStreamRelease(FSEventStreamRef stream);
 #include "fsm-listen.h"
 #include "fsmonitor--daemon.h"
 
-struct fsmonitor_daemon_backend_data
+struct fsm_listen_data
 {
 	CFStringRef cfsr_worktree_path;
 	CFStringRef cfsr_gitdir_path;
@@ -230,7 +230,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			     const FSEventStreamEventId event_ids[])
 {
 	struct fsmonitor_daemon_state *state = ctx;
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	char **paths = (char **)event_paths;
 	struct fsmonitor_batch *batch = NULL;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -419,11 +419,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 		NULL,
 		NULL
 	};
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 	const void *dir_array[2];
 
 	CALLOC_ARRAY(data, 1);
-	state->backend_data = data;
+	state->listen_data = data;
 
 	data->cfsr_worktree_path = CFStringCreateWithCString(
 		NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);
@@ -455,18 +455,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 failed:
 	error("Unable to create FSEventStream.");
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 	return -1;
 }
 
 void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	if (!state || !state->backend_data)
+	if (!state || !state->listen_data)
 		return;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	if (data->stream) {
 		if (data->stream_started)
@@ -476,14 +476,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 		FSEventStreamRelease(data->stream);
 	}
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 }
 
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	data = state->backend_data;
+	data = state->listen_data;
 	data->shutdown_style = SHUTDOWN_EVENT;
 
 	CFRunLoopStop(data->rl);
@@ -491,9 +491,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 
 void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	data->rl = CFRunLoopGetCurrent();
 
@@ -510,7 +510,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 
 	switch (data->shutdown_style) {
 	case FORCE_ERROR_STOP:
-		state->error_code = -1;
+		state->listen_error_code = -1;
 		/* fall thru */
 	case FORCE_SHUTDOWN:
 		ipc_server_stop_async(state->ipc_server_data);
@@ -522,7 +522,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 	return;
 
 force_error_stop_without_loop:
-	state->error_code = -1;
+	state->listen_error_code = -1;
 	ipc_server_stop_async(state->ipc_server_data);
 	return;
 }
diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index 6985903c95b..8406f8277dc 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -54,7 +54,7 @@ struct one_watch
 	wchar_t dotgit_shortname[16]; /* for 8.3 name */
 };
 
-struct fsmonitor_daemon_backend_data
+struct fsm_listen_data
 {
 	struct one_watch *watch_worktree;
 	struct one_watch *watch_gitdir;
@@ -259,7 +259,7 @@ static enum get_relative_result get_relative_longname(
 
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
-	SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);
+	SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);
 }
 
 static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
@@ -333,7 +333,7 @@ static void destroy_watch(struct one_watch *watch)
 	free(watch);
 }
 
-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,
+static int start_rdcw_watch(struct fsm_listen_data *data,
 			    struct one_watch *watch)
 {
 	DWORD dwNotifyFilter =
@@ -512,7 +512,7 @@ static int process_1_worktree_event(
  */
 static int process_worktree_events(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	struct one_watch *watch = data->watch_worktree;
 	struct strbuf path = STRBUF_INIT;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -642,7 +642,7 @@ force_shutdown:
  */
 static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	struct one_watch *watch = data->watch_gitdir;
 	struct strbuf path = STRBUF_INIT;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -700,11 +700,11 @@ skip_this_path:
 
 void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	DWORD dwWait;
 	int result;
 
-	state->error_code = 0;
+	state->listen_error_code = 0;
 
 	if (start_rdcw_watch(data, data->watch_worktree) == -1)
 		goto force_error_stop;
@@ -769,7 +769,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 	}
 
 force_error_stop:
-	state->error_code = -1;
+	state->listen_error_code = -1;
 
 force_shutdown:
 	/*
@@ -786,7 +786,7 @@ clean_shutdown:
 
 int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
 	CALLOC_ARRAY(data, 1);
 
@@ -819,7 +819,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 		data->nr_listener_handles++;
 	}
 
-	state->backend_data = data;
+	state->listen_data = data;
 	return 0;
 
 failed:
@@ -832,16 +832,16 @@ failed:
 
 void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	if (!state || !state->backend_data)
+	if (!state || !state->listen_data)
 		return;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	CloseHandle(data->hEventShutdown);
 	destroy_watch(data->watch_worktree);
 	destroy_watch(data->watch_gitdir);
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 }
diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h
index f0539349baf..41650bf8972 100644
--- a/compat/fsmonitor/fsm-listen.h
+++ b/compat/fsmonitor/fsm-listen.h
@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);
  * do so if the listener thread receives a normal shutdown signal from
  * the IPC layer.)
  *
- * It should set `state->error_code` to -1 if the daemon should exit
+ * It should set `state->listen_error_code` to -1 if the daemon should exit
  * with an error.
  */
 void fsm_listen__loop(struct fsmonitor_daemon_state *state);
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index a777c3a0590..f7de7882517 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);
  */
 void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);
 
-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */
+struct fsm_listen_data; /* opaque platform-specific data for listener thread */
 
 struct fsmonitor_daemon_state {
 	pthread_t listener_thread;
@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {
 	int cookie_seq;
 	struct hashmap cookies;
 
-	int error_code;
-	struct fsmonitor_daemon_backend_data *backend_data;
+	int listen_error_code;
+	struct fsm_listen_data *listen_data;
 
 	struct ipc_server_data *ipc_server_data;
 	struct strbuf path_ipc;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 17/23] fsmonitor--daemon: stub in health thread
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (15 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 16/23] fsmonitor--daemon: rename listener thread related variables Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 16:04   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 18/23] fsm-health-win32: add framework to monitor daemon health Jeff Hostetler via GitGitGadget
                   ` (8 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create another thread to watch over the daemon process and
automatically shut it down if necessary.

This commit creates the basic framework for a "health" thread
to monitor the daemon and/or the file system.  Later commits
will add platform-specific code to do the actual work.

The "health" thread is intended to monitor conditions that
would be difficult to track inside the IPC thread pool and/or
the file system listener threads.  For example, when there are
file system events outside of the watched worktree root or if
we want to have an idle-timeout auto-shutdown feature.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 Makefile                             |  6 ++-
 builtin/fsmonitor--daemon.c          | 39 +++++++++++++++
 compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++
 compat/fsmonitor/fsm-health-win32.c  | 72 ++++++++++++++++++++++++++++
 compat/fsmonitor/fsm-health.h        | 47 ++++++++++++++++++
 contrib/buildsystems/CMakeLists.txt  |  2 +
 fsmonitor--daemon.h                  |  4 ++
 7 files changed, 192 insertions(+), 2 deletions(-)
 create mode 100644 compat/fsmonitor/fsm-health-darwin.c
 create mode 100644 compat/fsmonitor/fsm-health-win32.c
 create mode 100644 compat/fsmonitor/fsm-health.h

diff --git a/Makefile b/Makefile
index ed3d5231ea1..85a61934f21 100644
--- a/Makefile
+++ b/Makefile
@@ -467,8 +467,9 @@ all::
 #
 # If your platform supports a built-in fsmonitor backend, set
 # FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding
-# `compat/fsmonitor/fsm-listen-<name>.c` that implements the
-# `fsm_listen__*()` routines.
+# `compat/fsmonitor/fsm-listen-<name>.c` and
+# `compat/fsmonitor/fsm-health-<name>.c` files
+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.
 #
 # If your platform has os-specific ways to tell if a repo is incompatible with
 # fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS
@@ -1944,6 +1945,7 @@ endif
 ifdef FSMONITOR_DAEMON_BACKEND
 	COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
 	COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
+	COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
 endif
 
 ifdef FSMONITOR_OS_SETTINGS
diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index f3fb865a9d8..591433e897d 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -3,6 +3,7 @@
 #include "parse-options.h"
 #include "fsmonitor.h"
 #include "fsmonitor-ipc.h"
+#include "compat/fsmonitor/fsm-health.h"
 #include "compat/fsmonitor/fsm-listen.h"
 #include "fsmonitor--daemon.h"
 #include "simple-ipc.h"
@@ -1124,6 +1125,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,
 	pthread_mutex_unlock(&state->main_lock);
 }
 
+static void *fsm_health__thread_proc(void *_state)
+{
+	struct fsmonitor_daemon_state *state = _state;
+
+	trace2_thread_start("fsm-health");
+
+	fsm_health__loop(state);
+
+	trace2_thread_exit();
+	return NULL;
+}
+
 static void *fsm_listen__thread_proc(void *_state)
 {
 	struct fsmonitor_daemon_state *state = _state;
@@ -1162,6 +1175,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 		 */
 		.uds_disallow_chdir = 0
 	};
+	int health_started = 0;
 	int listener_started = 0;
 	int err = 0;
 
@@ -1189,6 +1203,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	}
 	listener_started = 1;
 
+	/*
+	 * Start the health thread to watch over our process.
+	 */
+	if (pthread_create(&state->health_thread, NULL,
+			   fsm_health__thread_proc, state) < 0) {
+		ipc_server_stop_async(state->ipc_server_data);
+		err = error(_("could not start fsmonitor health thread"));
+		goto cleanup;
+	}
+	health_started = 1;
+
 	/*
 	 * The daemon is now fully functional in background threads.
 	 * Our primary thread should now just wait while the threads
@@ -1211,10 +1236,17 @@ cleanup:
 		pthread_join(state->listener_thread, NULL);
 	}
 
+	if (health_started) {
+		fsm_health__stop_async(state);
+		pthread_join(state->health_thread, NULL);
+	}
+
 	if (err)
 		return err;
 	if (state->listen_error_code)
 		return state->listen_error_code;
+	if (state->health_error_code)
+		return state->health_error_code;
 	return 0;
 }
 
@@ -1230,6 +1262,7 @@ static int fsmonitor_run_daemon(void)
 	pthread_mutex_init(&state.main_lock, NULL);
 	pthread_cond_init(&state.cookies_cond, NULL);
 	state.listen_error_code = 0;
+	state.health_error_code = 0;
 	state.current_token_data = fsmonitor_new_token_data();
 
 	/* Prepare to (recursively) watch the <worktree-root> directory. */
@@ -1309,6 +1342,11 @@ static int fsmonitor_run_daemon(void)
 		goto done;
 	}
 
+	if (fsm_health__ctor(&state)) {
+		err = error(_("could not initialize health thread"));
+		goto done;
+	}
+
 	/*
 	 * CD out of the worktree root directory.
 	 *
@@ -1332,6 +1370,7 @@ done:
 	pthread_cond_destroy(&state.cookies_cond);
 	pthread_mutex_destroy(&state.main_lock);
 	fsm_listen__dtor(&state);
+	fsm_health__dtor(&state);
 
 	ipc_server_free(state.ipc_server_data);
 
diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c
new file mode 100644
index 00000000000..b9f709e8548
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-darwin.c
@@ -0,0 +1,24 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+	return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+	return;
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+	return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+}
diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
new file mode 100644
index 00000000000..94b1d020f25
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -0,0 +1,72 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+struct fsm_health_data
+{
+	HANDLE hEventShutdown;
+
+	HANDLE hHandles[1]; /* the array does not own these handles */
+#define HEALTH_SHUTDOWN 0
+	int nr_handles; /* number of active event handles */
+};
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data;
+
+	CALLOC_ARRAY(data, 1);
+
+	data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);
+
+	data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;
+	data->nr_handles++;
+
+	state->health_data = data;
+	return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data;
+
+	if (!state || !state->health_data)
+		return;
+
+	data = state->health_data;
+
+	CloseHandle(data->hEventShutdown);
+
+	FREE_AND_NULL(state->health_data);
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data = state->health_data;
+
+	for (;;) {
+		DWORD dwWait = WaitForMultipleObjects(data->nr_handles,
+						      data->hHandles,
+						      FALSE, INFINITE);
+
+		if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)
+			goto clean_shutdown;
+
+		error(_("health thread wait failed [GLE %ld]"),
+		      GetLastError());
+		goto force_error_stop;
+	}
+
+force_error_stop:
+	state->health_error_code = -1;
+	ipc_server_stop_async(state->ipc_server_data);
+clean_shutdown:
+	return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+	SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);
+}
diff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h
new file mode 100644
index 00000000000..45547ba9380
--- /dev/null
+++ b/compat/fsmonitor/fsm-health.h
@@ -0,0 +1,47 @@
+#ifndef FSM_HEALTH_H
+#define FSM_HEALTH_H
+
+/* This needs to be implemented by each backend */
+
+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
+
+struct fsmonitor_daemon_state;
+
+/*
+ * Initialize platform-specific data for the fsmonitor health thread.
+ * This will be called from the main thread PRIOR to staring the
+ * thread.
+ *
+ * Returns 0 if successful.
+ * Returns -1 otherwise.
+ */
+int fsm_health__ctor(struct fsmonitor_daemon_state *state);
+
+/*
+ * Cleanup platform-specific data for the health thread.
+ * This will be called from the main thread AFTER joining the thread.
+ */
+void fsm_health__dtor(struct fsmonitor_daemon_state *state);
+
+/*
+ * The main body of the platform-specific event loop to monitor the
+ * health of the daemon process.  This will run in the health thread.
+ *
+ * The health thread should call `ipc_server_stop_async()` if it needs
+ * to cause a shutdown.  (It should NOT do so if it receives a shutdown
+ * shutdown signal.)
+ *
+ * It should set `state->health_error_code` to -1 if the daemon should exit
+ * with an error.
+ */
+void fsm_health__loop(struct fsmonitor_daemon_state *state);
+
+/*
+ * Gently request that the health thread shutdown.
+ * It does not wait for it to stop.  The caller should do a JOIN
+ * to wait for it.
+ */
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);
+
+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */
+#endif /* FSM_HEALTH_H */
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index fc70dd2fc1d..80704406895 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)
 	if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
 
 		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
 
 		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index f7de7882517..716e0e4d28d 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);
 void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);
 
 struct fsm_listen_data; /* opaque platform-specific data for listener thread */
+struct fsm_health_data; /* opaque platform-specific data for health thread */
 
 struct fsmonitor_daemon_state {
 	pthread_t listener_thread;
+	pthread_t health_thread;
 	pthread_mutex_t main_lock;
 
 	struct strbuf path_worktree_watch;
@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {
 	struct hashmap cookies;
 
 	int listen_error_code;
+	int health_error_code;
 	struct fsm_listen_data *listen_data;
+	struct fsm_health_data *health_data;
 
 	struct ipc_server_data *ipc_server_data;
 	struct strbuf path_ipc;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 18/23] fsm-health-win32: add framework to monitor daemon health
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (16 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 17/23] fsmonitor--daemon: stub in health thread Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 16:05   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves Jeff Hostetler via GitGitGadget
                   ` (7 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create framework in Win32 version of the "health" thread to
periodically inspect the system and shutdown if warranted.

This version just includes the setup for the timeout in
WaitForMultipleObjects() and calls the (currently empty) table
of functions.

A later commit will add functions to the table to actually
inspect the system.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-health-win32.c | 54 ++++++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 1 deletion(-)

diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
index 94b1d020f25..3c3453369cd 100644
--- a/compat/fsmonitor/fsm-health-win32.c
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -4,6 +4,40 @@
 #include "fsm-health.h"
 #include "fsmonitor--daemon.h"
 
+/*
+ * Every minute wake up and test our health.
+ */
+#define WAIT_FREQ_MS (60 * 1000)
+
+enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };
+
+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,
+			  enum interval_fn_ctx ctx);
+
+static interval_fn *table[] = {
+	NULL, /* must be last */
+};
+
+/*
+ * Call all of the functions in the table.
+ * Shortcut and return first error.
+ *
+ * Return 0 if all succeeded.
+ */
+static int call_all(struct fsmonitor_daemon_state *state,
+		    enum interval_fn_ctx ctx)
+{
+	int k;
+
+	for (k = 0; table[k]; k++) {
+		int r = table[k](state, ctx);
+		if (r)
+			return r;
+	}
+
+	return 0;
+}
+
 struct fsm_health_data
 {
 	HANDLE hEventShutdown;
@@ -45,15 +79,31 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)
 void fsm_health__loop(struct fsmonitor_daemon_state *state)
 {
 	struct fsm_health_data *data = state->health_data;
+	int r;
+
+	r = call_all(state, CTX_INIT);
+	if (r < 0)
+		goto force_error_stop;
+	if (r > 0)
+		goto force_shutdown;
 
 	for (;;) {
 		DWORD dwWait = WaitForMultipleObjects(data->nr_handles,
 						      data->hHandles,
-						      FALSE, INFINITE);
+						      FALSE, WAIT_FREQ_MS);
 
 		if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)
 			goto clean_shutdown;
 
+		if (dwWait == WAIT_TIMEOUT) {
+			r = call_all(state, CTX_TIMER);
+			if (r < 0)
+				goto force_error_stop;
+			if (r > 0)
+				goto force_shutdown;
+			continue;
+		}
+
 		error(_("health thread wait failed [GLE %ld]"),
 		      GetLastError());
 		goto force_error_stop;
@@ -61,8 +111,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)
 
 force_error_stop:
 	state->health_error_code = -1;
+force_shutdown:
 	ipc_server_stop_async(state->ipc_server_data);
 clean_shutdown:
+	call_all(state, CTX_TERM);
 	return;
 }
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (17 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 18/23] fsm-health-win32: add framework to monitor daemon health Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 16:09   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 20/23] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Jeff Hostetler via GitGitGadget
                   ` (6 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Force shutdown fsmonitor daemon if the worktree root directory
is moved, renamed, or deleted.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-health-win32.c | 133 ++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)

diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
index 3c3453369cd..2526ad9194f 100644
--- a/compat/fsmonitor/fsm-health-win32.c
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -14,7 +14,10 @@ enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };
 typedef int (interval_fn)(struct fsmonitor_daemon_state *state,
 			  enum interval_fn_ctx ctx);
 
+static interval_fn has_worktree_moved;
+
 static interval_fn *table[] = {
+	has_worktree_moved,
 	NULL, /* must be last */
 };
 
@@ -45,6 +48,12 @@ struct fsm_health_data
 	HANDLE hHandles[1]; /* the array does not own these handles */
 #define HEALTH_SHUTDOWN 0
 	int nr_handles; /* number of active event handles */
+
+	struct wt_moved
+	{
+		wchar_t wpath[MAX_PATH + 1];
+		BY_HANDLE_FILE_INFORMATION bhfi;
+	} wt_moved;
 };
 
 int fsm_health__ctor(struct fsmonitor_daemon_state *state)
@@ -76,6 +85,130 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)
 	FREE_AND_NULL(state->health_data);
 }
 
+static int lookup_bhfi(wchar_t *wpath,
+		       BY_HANDLE_FILE_INFORMATION *bhfi)
+{
+	DWORD desired_access = FILE_LIST_DIRECTORY;
+	DWORD share_mode =
+		FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;
+	HANDLE hDir;
+
+	hDir = CreateFileW(wpath, desired_access, share_mode, NULL,
+			   OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);
+	if (hDir == INVALID_HANDLE_VALUE) {
+		error(_("[GLE %ld] health thread could not open '%ls'"),
+		      GetLastError(), wpath);
+		return -1;
+	}
+
+	if (!GetFileInformationByHandle(hDir, bhfi)) {
+		error(_("[GLE %ld] health thread getting BHFI for '%ls'"),
+		      GetLastError(), wpath);
+		CloseHandle(hDir);
+		return -1;
+	}
+
+	CloseHandle(hDir);
+	return 0;
+}
+
+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,
+		   const BY_HANDLE_FILE_INFORMATION *bhfi_2)
+{
+	return (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&
+		bhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&
+		bhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);
+}
+
+/*
+ * Shutdown if the original worktree root directory been deleted,
+ * moved, or renamed?
+ *
+ * Since the main thread did a "chdir(getenv($HOME))" and our CWD
+ * is not in the worktree root directory and because the listener
+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible
+ * for the root directory to be moved or deleted while we are still
+ * watching it.  We want to detect that here and force a shutdown.
+ *
+ * Granted, a delete MAY cause some operations to fail, such as
+ * GetOverlappedResult(), but it is not guaranteed.  And because
+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the
+ * directory, not changes *ON* the directory, our watch will not
+ * receive a delete event for it.
+ *
+ * A move/rename of the worktree root will also not generate an event.
+ * And since the listener thread already has an open handle, it may
+ * continue to receive events for events within the directory.
+ * However, the pathname of the named-pipe was constructed using the
+ * original location of the worktree root.  (Remember named-pipes are
+ * stored in the NPFS and not in the actual file system.)  Clients
+ * trying to talk to the worktree after the move/rename will not
+ * reach our daemon process, since we're still listening on the
+ * pipe with original path.
+ *
+ * Furthermore, if the user does something like:
+ *
+ *   $ mv repo repo.old
+ *   $ git init repo
+ *
+ * A new daemon cannot be started in the new instance of "repo"
+ * because the named-pipe is still being used by the daemon on
+ * the original instance.
+ *
+ * So, detect move/rename/delete and shutdown.  This should also
+ * handle unsafe drive removal.
+ *
+ * We use the file system unique ID to distinguish the original
+ * directory instance from a new instance and force a shutdown
+ * if the unique ID changes.
+ *
+ * Since a worktree move/rename/delete/unmount doesn't happen
+ * that often (and we can't get an immediate event anyway), we
+ * use a timeout and periodically poll it.
+ */
+static int has_worktree_moved(struct fsmonitor_daemon_state *state,
+			      enum interval_fn_ctx ctx)
+{
+	struct fsm_health_data *data = state->health_data;
+	BY_HANDLE_FILE_INFORMATION bhfi;
+	int r;
+
+	switch (ctx) {
+	case CTX_TERM:
+		return 0;
+
+	case CTX_INIT:
+		if (xutftowcs_path(data->wt_moved.wpath,
+				   state->path_worktree_watch.buf) < 0) {
+			error(_("could not convert to wide characters: '%s'"),
+			      state->path_worktree_watch.buf);
+			return -1;
+		}
+
+		/*
+		 * On the first call we lookup the unique sequence ID for
+		 * the worktree root directory.
+		 */
+		return lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);
+
+	case CTX_TIMER:
+		r = lookup_bhfi(data->wt_moved.wpath, &bhfi);
+		if (r)
+			return r;
+		if (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {
+			error(_("BHFI changed '%ls'"), data->wt_moved.wpath);
+			return -1;
+		}
+		return 0;
+
+	default:
+		die("unhandled case in 'has_worktree_moved': %d",
+		    (int)ctx);
+	}
+
+	return 0;
+}
+
 void fsm_health__loop(struct fsmonitor_daemon_state *state)
 {
 	struct fsm_health_data *data = state->health_data;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 20/23] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (18 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 16:10   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 21/23] fsmonitor: optimize processing of directory events Jeff Hostetler via GitGitGadget
                   ` (5 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach the listener thread to shutdown the daemon if the spelling of the
worktree root directory changes.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 87a8476b09f..d3afbbc53d6 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -178,6 +178,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)
 	strbuf_release(&msg);
 }
 
+static int ef_is_root_changed(const FSEventStreamEventFlags ef)
+{
+	return (ef & kFSEventStreamEventFlagRootChanged);
+}
+
 static int ef_is_root_delete(const FSEventStreamEventFlags ef)
 {
 	return (ef & kFSEventStreamEventFlagItemIsDir &&
@@ -287,6 +292,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			continue;
 		}
 
+		if (ef_is_root_changed(event_flags[k])) {
+			/*
+			 * The spelling of the pathname of the root directory
+			 * has changed.  This includes the name of the root
+			 * directory itself of of any parent directory in the
+			 * path.
+			 *
+			 * (There may be other conditions that throw this,
+			 * but I couldn't find any information on it.)
+			 *
+			 * Force a shutdown now and avoid things getting
+			 * out of sync.  The Unix domain socket is inside
+			 * the .git directory and a spelling change will make
+			 * it hard for clients to rendezvous with us.
+			 */
+			trace_printf_key(&trace_fsmonitor,
+					 "event: root changed");
+			goto force_shutdown;
+		}
+
 		if (ef_ignore_xattr(event_flags[k])) {
 			trace_printf_key(&trace_fsmonitor,
 					 "ignore-xattr: '%s', flags=0x%x",
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 21/23] fsmonitor: optimize processing of directory events
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (19 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 20/23] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-24 16:13   ` Derrick Stolee
  2022-02-15 15:59 ` [PATCH 22/23] t7527: FSMonitor tests for directory moves Jeff Hostetler via GitGitGadget
                   ` (4 subsequent siblings)
  25 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to perform binary search over the cache-entries for a directory
notification and then linearly scan forward to find the immediate children.

Previously, when the FSMonitor reported a modified directory Git would
perform a linear search on the entire cache-entry array for all
entries matching that directory prefix and invalidate them.  Since the
cache-entry array is already sorted, we can use a binary search to
find the first matching entry and then only linearly walk forward and
invalidate entries until the prefix changes.

Also, the original code would invalidate anything having the same
directory prefix.  Since a directory event should only be received for
items that are immediately within the directory (and not within
sub-directories of it), only invalidate those entries and not the
whole subtree.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 54 insertions(+), 17 deletions(-)

diff --git a/fsmonitor.c b/fsmonitor.c
index 8e3499d0667..be08861a18e 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,
 static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
 {
 	int i, len = strlen(name);
-	if (name[len - 1] == '/') {
+	int pos = index_name_pos(istate, name, len);
+
+	trace_printf_key(&trace_fsmonitor,
+			 "fsmonitor_refresh_callback '%s' (pos %d)",
+			 name, pos);
 
+	if (name[len - 1] == '/') {
 		/*
-		 * TODO We should binary search to find the first path with
-		 * TODO this directory prefix.  Then linearly update entries
-		 * TODO while the prefix matches.  Taking care to search without
-		 * TODO the trailing slash -- because '/' sorts after a few
-		 * TODO interesting special chars, like '.' and ' '.
+		 * The daemon can decorate directory events, such as
+		 * moves or renames, with a trailing slash if the OS
+		 * FS Event contains sufficient information, such as
+		 * MacOS.
+		 *
+		 * Use this to invalidate the entire cone under that
+		 * directory.
+		 *
+		 * We do not expect an exact match because the index
+		 * does not normally contain directory entries, so we
+		 * start at the insertion point and scan.
 		 */
+		if (pos < 0)
+			pos = -pos - 1;
 
 		/* Mark all entries for the folder invalid */
-		for (i = 0; i < istate->cache_nr; i++) {
-			if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&
-			    starts_with(istate->cache[i]->name, name))
-				istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
+		for (i = pos; i < istate->cache_nr; i++) {
+			if (!starts_with(istate->cache[i]->name, name))
+				break;
+			istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
 		}
-		/* Need to remove the / from the path for the untracked cache */
+
+		/*
+		 * We need to remove the traling "/" from the path
+		 * for the untracked cache.
+		 */
 		name[len - 1] = '\0';
+	} else if (pos >= 0) {
+		/*
+		 * We have an exact match for this path and can just
+		 * invalidate it.
+		 */
+		istate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;
 	} else {
-		int pos = index_name_pos(istate, name, strlen(name));
-
-		if (pos >= 0) {
-			struct cache_entry *ce = istate->cache[pos];
-			ce->ce_flags &= ~CE_FSMONITOR_VALID;
+		/*
+		 * The path is not a tracked file -or- it is a
+		 * directory event on a platform that cannot
+		 * distinguish between file and directory events in
+		 * the event handler, such as Windows.
+		 *
+		 * Scan as if it is a directory and invalidate the
+		 * cone under it.  (But remember to ignore items
+		 * between "name" and "name/", such as "name-" and
+		 * "name.".
+		 */
+		pos = -pos - 1;
+
+		for (i = pos; i < istate->cache_nr; i++) {
+			if (!starts_with(istate->cache[i]->name, name))
+				break;
+			if ((unsigned char)istate->cache[i]->name[len] > '/')
+				break;
+			if (istate->cache[i]->name[len] == '/')
+				istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
 		}
 	}
 
@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
 	 * Mark the untracked cache dirty even if it wasn't found in the index
 	 * as it could be a new untracked file.
 	 */
-	trace_printf_key(&trace_fsmonitor, "fsmonitor_refresh_callback '%s'", name);
 	untracked_cache_invalidate_path(istate, name, 0);
 }
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 22/23] t7527: FSMonitor tests for directory moves
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (20 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 21/23] fsmonitor: optimize processing of directory events Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-15 15:59 ` [PATCH 23/23] t/perf/p7527: add perf test for builtin FSMonitor Jeff Hostetler via GitGitGadget
                   ` (3 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create unit tests to move a directory.  Verify that `git status`
gives the same result with and without FSMonitor enabled.

NEEDSWORK: This test exposes a bug in the untracked-cache on
Windows when FSMonitor is disabled.  These are commented out
for the moment.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 39 ++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index c0145544ccb..408d614b28b 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -261,6 +261,16 @@ test_expect_success 'setup' '
 	trace*
 	EOF
 
+	mkdir -p T1/T2/T3/T4 &&
+	echo 1 >T1/F1 &&
+	echo 1 >T1/T2/F1 &&
+	echo 1 >T1/T2/T3/F1 &&
+	echo 1 >T1/T2/T3/T4/F1 &&
+	echo 2 >T1/F2 &&
+	echo 2 >T1/T2/F2 &&
+	echo 2 >T1/T2/T3/F2 &&
+	echo 2 >T1/T2/T3/T4/F2 &&
+
 	git -c core.fsmonitor=false add . &&
 	test_tick &&
 	git -c core.fsmonitor=false commit -m initial &&
@@ -354,6 +364,19 @@ verify_status () {
 	echo HELLO AFTER
 }
 
+move_directory_contents_deeper() {
+	mkdir T1/_new_
+	mv T1/[A-Z]* T1/_new_
+}
+
+move_directory_up() {
+	mv T1/T2/T3 T1
+}
+
+move_directory() {
+	mv T1/T2/T3 T1/T2/NewT3
+}
+
 # The next few test cases confirm that our fsmonitor daemon sees each type
 # of OS filesystem notification that we care about.  At this layer we just
 # ensure we are getting the OS notifications and do not try to confirm what
@@ -685,6 +708,22 @@ do
 		matrix_try $uc_val $fsm_val file_to_directory
 		matrix_try $uc_val $fsm_val directory_to_file
 
+		# NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor
+		# is DISABLED.  Turn off a few test that cause it problems until
+		# we can debug it.
+		#
+		try_moves="true"
+		test_have_prereq UNTRACKED_CACHE,WINDOWS && \
+			test $uc_val = true && \
+			test $fsm_val = false && \
+			try_moves="false"
+		if test $try_moves = true
+		then
+			matrix_try $uc_val $fsm_val move_directory_contents_deeper
+			matrix_try $uc_val $fsm_val move_directory_up
+			matrix_try $uc_val $fsm_val move_directory
+		fi
+
 		if test $fsm_val = true
 		then
 			test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end" '
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH 23/23] t/perf/p7527: add perf test for builtin FSMonitor
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (21 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 22/23] t7527: FSMonitor tests for directory moves Jeff Hostetler via GitGitGadget
@ 2022-02-15 15:59 ` Jeff Hostetler via GitGitGadget
  2022-02-16  1:00 ` [PATCH 00/23] Builtin FSMonitor Part 3 Junio C Hamano
                   ` (2 subsequent siblings)
  25 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-02-15 15:59 UTC (permalink / raw)
  To: git; +Cc: Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++
 1 file changed, 257 insertions(+)
 create mode 100755 t/perf/p7527-builtin-fsmonitor.sh

diff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh
new file mode 100755
index 00000000000..9338b9ea008
--- /dev/null
+++ b/t/perf/p7527-builtin-fsmonitor.sh
@@ -0,0 +1,257 @@
+#!/bin/sh
+
+test_description="Perf test for the builtin FSMonitor"
+
+. ./perf-lib.sh
+
+if ! test_have_prereq FSMONITOR_DAEMON
+then
+	skip_all="fsmonitor--daemon is not supported on this platform"
+	test_done
+fi
+
+test_lazy_prereq UNTRACKED_CACHE '
+	{ git update-index --test-untracked-cache; ret=$?; } &&
+	test $ret -ne 1
+'
+
+# Lie to perf-lib and ask for a new empty repo and avoid
+# the complaints about GIT_PERF_REPO not being big enough
+# the perf hit when GIT_PERF_LARGE_REPO is copied into
+# the trash directory.
+#
+# NEEDSWORK: It would be nice if perf-lib had an option to
+# "borrow" an existing large repo (especially for gigantic
+# monorepos) and use it in-place.  For now, fake it here.
+#
+test_perf_fresh_repo
+
+
+# Use a generated synthetic monorepo.  If it doesn't exist, we will
+# generate it.  If it does exist, we will put it in a known state
+# before we start our timings.
+#
+PARAM_D=5
+PARAM_W=10
+PARAM_F=9
+
+PARAMS="$PARAM_D"."$PARAM_W"."$PARAM_F"
+
+BALLAST_BR=p0006-ballast
+export BALLAST_BR
+
+TMP_BR=tmp_br
+export TMP_BR
+
+REPO=../repos/gen-many-files-"$PARAMS".git
+export REPO
+
+if ! test -d $REPO
+then
+	(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)
+fi
+
+
+enable_uc () {
+	git -C $REPO config core.untrackedcache true
+	git -C $REPO update-index --untracked-cache
+	git -C $REPO status >/dev/null 2>&1
+}
+
+disable_uc () {
+	git -C $REPO config core.untrackedcache false
+	git -C $REPO update-index --no-untracked-cache
+	git -C $REPO status >/dev/null 2>&1
+}
+
+start_fsm () {
+	git -C $REPO fsmonitor--daemon start
+	git -C $REPO fsmonitor--daemon status
+	git -C $REPO config core.fsmonitor true
+	git -C $REPO update-index --fsmonitor
+	git -C $REPO status >/dev/null 2>&1
+}
+
+stop_fsm () {
+	git -C $REPO config --unset core.fsmonitor
+	git -C $REPO update-index --no-fsmonitor
+	test_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null
+	git -C $REPO status >/dev/null 2>&1
+}
+
+
+# Ensure that FSMonitor is turned off on the borrowed repo.
+#
+test_expect_success "Setup borrowed repo (fsm+uc)" "
+	stop_fsm &&
+	disable_uc
+"
+
+# Also ensure that it starts in a known state.
+#
+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time
+# the ballast checkout, since only the first invocation does any work and the
+# subsequent ones just print "already on branch" and quit, so the reported
+# time is not useful.
+#
+# Create a temp branch and do all work relative to it so that we don't
+# accidentially alter the real ballast branch.
+#
+test_expect_success "Setup borrowed repo (temp ballast branch)" "
+	test_might_fail git -C $REPO checkout $BALLAST_BR &&
+	test_might_fail git -C $REPO reset --hard &&
+	git -C $REPO clean -d -f &&
+	test_might_fail git -C $REPO branch -D $TMP_BR &&
+	git -C $REPO branch $TMP_BR $BALLAST_BR &&
+	git -C $REPO checkout $TMP_BR
+"
+
+
+echo Data >data.txt
+
+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1.  With
+# FSMonitor enabled, we can get a skewed view of status times, since
+# the index MAY (or may not) be updated after the first invocation
+# which will update the FSMonitor Token, so the subsequent invocations
+# may get a smaller response from the daemon.
+#
+do_status () {
+	msg=$1
+
+	test_perf "$msg" "
+		git -C $REPO status >/dev/null 2>&1
+	"
+}
+
+do_matrix () {
+	uc=$1
+	fsm=$2
+
+	t="[uc $uc][fsm $fsm]"
+	MATRIX_BR="$TMP_BR-$uc-$fsm"
+
+	test_expect_success "$t Setup matrix branch" "
+		git -C $REPO clean -d -f &&
+		git -C $REPO checkout $TMP_BR &&
+		test_might_fail git -C $REPO branch -D $MATRIX_BR &&
+		git -C $REPO branch $MATRIX_BR $TMP_BR &&
+		git -C $REPO checkout $MATRIX_BR
+	"
+
+	if test $uc = true
+	then
+		enable_uc
+	else
+		disable_uc
+	fi
+
+	if test $fsm = true
+	then
+		start_fsm
+	else
+		stop_fsm
+	fi
+
+	do_status "$t status after checkout"
+
+	# Modify many files in the matrix branch.
+	# Stage them.
+	# Commit them.
+	# Rollback.
+	#
+	test_expect_success "$t modify tracked files" "
+		find $REPO -name file1 -exec cp data.txt {} \\;
+	"
+
+	do_status "$t status after big change"
+
+	# Don't bother timing the "add" because _REPEAT_COUNT
+	# issue described above.
+	#
+	test_expect_success "$t add all" "
+		git -C $REPO add -A
+	"
+
+	do_status "$t status after add all"
+
+	test_expect_success "$t add dot" "
+		git -C $REPO add .
+	"
+
+	do_status "$t status after add dot"
+
+	test_expect_success "$t commit staged" "
+		git -C $REPO commit -a -m data
+	"
+
+	do_status "$t status after commit"
+
+	test_expect_success "$t reset HEAD~1 hard" "
+		git -C $REPO reset --hard HEAD~1 >/dev/null 2>&1
+	"
+
+	do_status "$t status after reset hard"
+
+	# Create some untracked files.
+	#
+	test_expect_success "$t create untracked files" "
+		cp -R $REPO/ballast/dir1 $REPO/ballast/xxx1
+	"
+
+	do_status "$t status after create untracked files"
+
+	# Remove the new untracked files.
+	#
+	test_expect_success "$t clean -df" "
+		git -C $REPO clean -d -f
+	"
+
+	do_status "$t status after clean"
+
+	if test $fsm = true
+	then
+		stop_fsm
+	fi
+}
+
+# Begin testing each case in the matrix that we care about.
+#
+uc_values="false"
+test_have_prereq UNTRACKED_CACHE && uc_values="false true"
+
+fsm_values="false true"
+
+for uc_val in $uc_values
+do
+	for fsm_val in $fsm_values
+	do
+		do_matrix $uc_val $fsm_val
+	done
+done
+
+cleanup () {
+	uc=$1
+	fsm=$2
+
+	MATRIX_BR="$TMP_BR-$uc-$fsm"
+
+	test_might_fail git -C $REPO branch -D $MATRIX_BR
+}
+
+
+# We're borrowing this repo.  We should leave it in a clean state.
+#
+test_expect_success "Cleanup temp and matrix branches" "
+	git -C $REPO clean -d -f &&
+	test_might_fail git -C $REPO checkout $BALLAST_BR &&
+	test_might_fail git -C $REPO branch -D $TMP_BR &&
+	for uc_val in $uc_values
+	do
+		for fsm_val in $fsm_values
+		do
+			cleanup $uc_val $fsm_val
+		done
+	done
+"
+
+test_done
-- 
gitgitgadget

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 00/23] Builtin FSMonitor Part 3
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (22 preceding siblings ...)
  2022-02-15 15:59 ` [PATCH 23/23] t/perf/p7527: add perf test for builtin FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-02-16  1:00 ` Junio C Hamano
  2022-02-16 14:04   ` Jeff Hostetler
  2022-02-24 16:21 ` Derrick Stolee
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
  25 siblings, 1 reply; 277+ messages in thread
From: Junio C Hamano @ 2022-02-16  1:00 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget; +Cc: git, Jeff Hostetler

"Jeff Hostetler via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Here is part 3 of my builtin FSMonitor series.
>
> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in
> "seen", so this series should be considered a preview until part 2 moves to
> "next". (And part 2 should not graduate to "master" without this part.)
>
> Part 2 established the client code (used by commands like git status) and an
> MVP implementation of the FSMonitor daemon. This was sufficient to test the
> concepts and basic functionality.

Sounds like part 2 is sufficiently done to be eligible for being in
'master', waiting for an improved daemon, no?  

Have people been reviewing the patches in part 2?  I haven't had a
chance to take a deeper look myself.

Thanks.


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 00/23] Builtin FSMonitor Part 3
  2022-02-16  1:00 ` [PATCH 00/23] Builtin FSMonitor Part 3 Junio C Hamano
@ 2022-02-16 14:04   ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-02-16 14:04 UTC (permalink / raw)
  To: Junio C Hamano, Jeff Hostetler via GitGitGadget; +Cc: git, Jeff Hostetler



On 2/15/22 8:00 PM, Junio C Hamano wrote:
> "Jeff Hostetler via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> Here is part 3 of my builtin FSMonitor series.
>>
>> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in
>> "seen", so this series should be considered a preview until part 2 moves to
>> "next". (And part 2 should not graduate to "master" without this part.)
>>
>> Part 2 established the client code (used by commands like git status) and an
>> MVP implementation of the FSMonitor daemon. This was sufficient to test the
>> concepts and basic functionality.
> 
> Sounds like part 2 is sufficiently done to be eligible for being in
> 'master', waiting for an improved daemon, no?
> 
> Have people been reviewing the patches in part 2?  I haven't had a
> chance to take a deeper look myself.
> 
> Thanks.
> 

Yes, I think Part 2 could advance if we wanted to.  I just didn't
want to presume that, so I was being conservative.  I've always
treated it and Part 3 as a unit.

On the other hand, we've been distributing Part 2 V4 and an
un-submitted version of Part 3 experimentally in git-for-windows
and microsoft/git.git (gvfs) since last summer, so I think we're
good.

It would be good to get some eyes on the Part 2 V5 changes having
to do with removing `core.useBuiltinFSMonitor` and overloading
the existing `core.fsmonitor` to take either a hook path or bool.
That was the only major change between V4 and V5.  And that V5
change has not yet been shipped in GFW nor GVFS.

Thanks
Jeff

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 01/23] fsm-listen-win32: handle shortnames
  2022-02-15 15:59 ` [PATCH 01/23] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
@ 2022-02-24 14:48   ` Derrick Stolee
  0 siblings, 0 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 14:48 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Teach FSMonitor daemon on Windows to recognize shortname paths as
> aliases of normal longname paths.  FSMonitor clients, such as `git
> status`, should receive the longname spelling of changed files (when
> possible).
> 
> Sometimes we receive FS events using the shortname, such as when a CMD
> shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1".  The FS notification
> arrives using whatever combination of long and shortnames were used by
> the other process.  (Shortnames do seem to be case normalized,
> however.)
> 
> Use Windows GetLongPathNameW() to try to map the pathname spelling in
> the notification event into the normalized longname spelling.  (This
> can fail if the file/directory is deleted, moved, or renamed, because
> we are asking the FS for the mapping in response to the event and
> after it has already happened, but we try.)

I recall when we first discovered that this was important for users.
I'm glad we haven't needed to worry about it since this fix.

> Special case the shortname spelling of ".git" to avoid under-reporting
> these events.

This is particularly critical due to the case of deleting the .git
directory being really important to catch, but GetLongPathNameW()
won't work in that case.

> +	if (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {
> +		/*
> +		 * The shortname to longname conversion can fail for
> +		 * various reasons, for example if the file has been
> +		 * deleted.  (That is, if we just received a
> +		 * delete-file notification event and the file is
> +		 * already gone, we can't ask the file system to
> +		 * lookup the longname for it.  Likewise, for moves
> +		 * and renames where we are given the old name.)
> +		 *
> +		 * NEEDSWORK: Since deleting or moving a file or
> +		 * directory by shortname is rather obscure, I'm going
> +		 * ignore the failure and ask the caller to report the
> +		 * original relative path.  This seemds kinder than

s/seemds/seems/

> +		 * failing here and forcing a resync.
> +		 */
> +		return GRR_NO_CONVERSION_NEEDED;
> +	}

This NEEDSWORK is something I saw while reading that makes it seem
like we _need_ to do something here. I think this is an appropriate
short-term solution that we can revisit if necessary. I don't
anticipate that actually being the case, though.

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode
  2022-02-15 15:59 ` [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode Jeff Hostetler via GitGitGadget
@ 2022-02-24 14:52   ` Derrick Stolee
  2022-02-24 17:33     ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
  0 siblings, 1 reply; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 14:52 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Confirm that macOS FS events are reported with a normalized spelling.
> 
> APFS (and/or HFS+) is case-insensitive.  This means that case-independent
> lookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed.  But that
> doesn't tell us how FS events are reported if we try "rm -rf .git" versus
> "rm -rf .GIT".  Are the events reported using the on-disk spelling of the
> pathname or in the spelling used by the command.

Was this last sentence supposed to be a question?
 
> NEEDSWORK: I was only able to test case.  It would be nice to add tests

"I was only able test the APFS case."?

> that use different Unicode spellings/normalizations and understand the
> differences between APFS and HFS+ in this area.  We should confirm that
> the spelling of the workdir paths that the daemon sends to clients are
> always properly normalized.

Are there any macOS experts out there who can help us find the answers
to these questions?

> +# Confirm that MacOS hides all of the Unicode normalization and/or
> +# case folding from the FS events.  That is, are the pathnames in the
> +# FS events reported using the spelling on the disk or in the spelling
> +# used by the other process.
> +#
> +# Note that we assume that the filesystem is set to case insensitive.
> +#
> +# NEEDSWORK: APFS handles Unicode and Unicode normalization
> +# differently than HFS+.  I only have an APFS partition, so
> +# more testing here would be helpful.
> +#
> +
> +# Rename .git using alternate spelling and confirm that the daemon
> +# sees the event using the correct spelling and shutdown.
> +test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '
> +	test_when_finished "stop_daemon_delete_repo test_apfs" &&
> +
> +	git init test_apfs &&
> +	start_daemon test_apfs &&
> +
> +	test_path_is_dir test_apfs/.git &&
> +	test_path_is_dir test_apfs/.GIT &&
> +
> +	mv test_apfs/.GIT test_apfs/.FOO &&
> +	sleep 1 &&

This sleep is unfortunate. Do we really need it? Or does this test
become flaky without it?

> +	mv test_apfs/.FOO test_apfs/.git &&
> +
> +	test_must_fail git -C test_apfs fsmonitor--daemon status
> +'
> +

This test is helpful in that it will help us discover if HFS+ or
any future filesystem would break these assumptions.

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 04/23] t/helper/fsmonitor-client: create stress test
  2022-02-15 15:59 ` [PATCH 04/23] t/helper/fsmonitor-client: create stress test Jeff Hostetler via GitGitGadget
@ 2022-02-24 14:58   ` Derrick Stolee
  2022-03-01 19:37     ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 14:58 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Create a stress test to hammer on the fsmonitor daemon.
> Create a client-side thread pool of n threads and have
> each of them make m requests as fast as they can.
> 
> NEEDSWORK: This is just the client-side thread pool and
> is useful for interactive testing and experimentation.
> We need to add a script test to drive this.

I haven't gotten far enough in the series to know if you
_do_ use this in a test eventually. If so, this NEEDSWORK
could be replaced with a mention of a future change.

> +	/*
> +	 * TODO Decide if/when to return an error or call die().
> +	 */
> +	return 0;

This TODO could be cleaned up.

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 06/23] fsmonitor-settings: stub in platform-specific incompatibility checking
  2022-02-15 15:59 ` [PATCH 06/23] fsmonitor-settings: stub in platform-specific incompatibility checking Jeff Hostetler via GitGitGadget
@ 2022-02-24 15:05   ` Derrick Stolee
  0 siblings, 0 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 15:05 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Extend generic incompatibility checkout with platform-specific
> mechanism.  Stub in Win32 version.
> 
> In the existing fsmonitor-settings code we have a way to mark
> types of repos as incompatible with fsmonitor (whether via the
> hook and ipc APIs).  For example, we do this for bare repos,
> since there are no files to watch.
> 
> Extend this exclusion mechanism for platfor-specific reasons.

s/platfor-specific/platform-specific/

> +# If your platform has os-specific ways to tell if a repo is incompatible with
> +# fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS
> +# to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c`
> +# that implements the `fsm_os_settings__*()` routines.

nit: It's just a comment, but I think "OS" and "IPC" should probably
be capitalized.

> +#ifdef HAVE_FSMONITOR_OS_SETTINGS
> +	{
> +		enum fsmonitor_reason reason;
> +
> +		reason = fsm_os__incompatible(r);
> +		if (reason != FSMONITOR_REASON_ZERO) {

A naming nit about FSMONITOR_REASON_ZERO. It seems named ZERO
on purpose so a non-zero reason signals incompatibility. That
would mean you could use

		if (reason)

here. Alternatively, FSMONITOR_REASON_COMPATIBLE would signal
the meaning better here:

		if (reason != FSMONITOR_REASON_COMPATIBLE)

> +			set_incompatible(r, reason);
> +			return 1;
> +		}

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor
  2022-02-15 15:59 ` [PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-02-24 15:11   ` Derrick Stolee
  2022-03-01 21:01     ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 15:11 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git
  Cc: Jeff Hostetler, Jonathan Nieder, Junio C Hamano

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Virtual repos, such as GVFS (aka VFS for Git), are incompatible
> with FSMonitor.

I would swap all of your "GVFS (aka VFS for Git)" for just
"VFS for Git".

> +/*
> + * GVFS (aka VFS for Git) is incompatible with FSMonitor.
> + *
> + * Granted, core Git does not know anything about GVFS and we
> + * shouldn't make assumptions about a downstream feature, but users
> + * can install both versions.  And this can lead to incorrect results
> + * from core Git commands.  So, without bringing in any of the GVFS
> + * code, do a simple config test for a published config setting.  (We
> + * do not look at the various *_TEST_* environment variables.)
> + */
> +static enum fsmonitor_reason is_virtual(struct repository *r)
> +{
> +	const char *const_str;
> +
> +	if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str))
> +		return FSMONITOR_REASON_VIRTUAL;
> +
> +	return FSMONITOR_REASON_ZERO;
> +}

This reason seems to be specific to a config setting that only
exists in the microsoft/git fork. Perhaps this patch should remain
there.

However, there is also the discussion of vfsd going on, so something
similar might be necessary for that system. Junio also mentioned
wanting to collaborate on a common indicator that virtualization was
being used, so maybe we _should_ make core.virtualFilesystem a config
setting in the core Git project.

The reason for the incompatibility here is that VFS for Git has its
own filesystem watcher and Git gets updates from it via custom code
that is a precursor to this FS Monitor feature. I don't know if vfsd
has plans for a similar setup. (It would probably be best to fit
into the FS Monitor client/server model and use a different daemon
for those virtualized repos, but I don't know enough details to be
sure.)

CC'ing Jonathan Nieder for thoughts on this.

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor
  2022-02-15 15:59 ` [PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-02-24 15:26   ` Derrick Stolee
  2022-03-01 21:30     ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 15:26 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Teach Git to detect remote working directories on macOS and mark them as
> incompatible with FSMonitor.
> 
> With this, `git fsmonitor--daemon run` will error out with a message
> like it does for bare repos.
> 
> Client commands, like `git status`, will not attempt to start the daemon.

...

> + * A client machine (such as a laptop) may choose to suspend/resume
> + * and it is unclear (without lots of testing) whether the watcher can
> + * resync after a resume.  We might be able to treat this as a normal
> + * "events were dropped by the kernel" event and do our normal "flush
> + * and resync" --or-- we might need to close the existing (zombie?)
> + * notification fd and create a new one.
> + *
> + * In theory, the above issues need to be addressed whether we are
> + * using the Hook or IPC API.

The only thing I can think about is a case where the filesystem
monitor is actually running on the remote machine and Git
communicates with it over the network. This is currently possible
with the hook, but I am not aware of a hook implementation that
does this.

We can find a way to update the hook interface to communicate to
Git that a remote disk is an appropriate case, but only if there
is a real need for that.

> + * For the builtin FSMonitor, we create the Unix domain socket for the
> + * IPC in the .git directory.  If the working directory is remote,
> + * then the socket will be created on the remote file system.  This

The socket is on the remote file system, but the daemon process is still
local, so I still see this a problem.

> + * can fail if the remote file system does not support UDS file types
> + * (e.g. smbfs to a Windows server) or if the remote kernel does not
> + * allow a non-local process to bind() the socket.  (These problems
> + * could be fixed by moving the UDS out of the .git directory and to a
> + * well-known local directory on the client machine, but care should
> + * be taken to ensure that $HOME is actually local and not a managed
> + * file share.)
> + *
> + * So (for now at least), mark remote working directories as
> + * incompatible.
> + */
> +static enum fsmonitor_reason is_remote(struct repository *r)
> +{
> +	struct statfs fs;
> +
> +	if (statfs(r->worktree, &fs) == -1) {
> +		int saved_errno = errno;
> +		trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s",
> +				 r->worktree, strerror(saved_errno));
> +		errno = saved_errno;
> +		return FSMONITOR_REASON_ZERO;

So if we fail to inspect the filesystem, we report it as compatible?
I suppose other things are likely to fail if checks like this are
fialing, but I wonder if we should preempt that by marking this as
incompatible due to filesystem errors.

> +	}
> +
> +	trace_printf_key(&trace_fsmonitor,
> +			 "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'",
> +			 r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);
> +
> +	if (!(fs.f_flags & MNT_LOCAL))
> +		return FSMONITOR_REASON_REMOTE;

I do see that we need a successful response to give this specific
reason for incompatibility.

> +	return FSMONITOR_REASON_ZERO;
> +}
>  
>  enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
>  {
> +	enum fsmonitor_reason reason;
> +
> +	reason = is_remote(r);
> +	if (reason)
> +		return reason;

This organization is looking like you want to short-circuit the
checks if you find an incompatibility, with the intent of having
multiple checks in the future.

But this can be done with simple || operators:

	return is_remote() ||
	       reason_check_2() ||
	       reason_check_3();

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 10/23] fsmonitor-settings: remote repos on Windows are incompatible with FSMonitor
  2022-02-15 15:59 ` [PATCH 10/23] fsmonitor-settings: remote repos on Windows " Jeff Hostetler via GitGitGadget
@ 2022-02-24 15:56   ` Derrick Stolee
  0 siblings, 0 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 15:56 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
...
> @@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
>  	if (reason)
>  		return reason;
>  
> +	reason = is_remote(r);
> +	if (reason)
> +		return reason;
> +
>  	return FSMONITOR_REASON_ZERO;

Just popping in to say that

	return is_virtual(r) ||
	       is_remote(r);

would work here, but we might remove is_virtual().

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread
  2022-02-15 15:59 ` [PATCH 17/23] fsmonitor--daemon: stub in health thread Jeff Hostetler via GitGitGadget
@ 2022-02-24 16:04   ` Derrick Stolee
  2022-02-24 16:15     ` Derrick Stolee
  2022-03-03 16:16     ` Jeff Hostetler
  0 siblings, 2 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 16:04 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Create another thread to watch over the daemon process and
> automatically shut it down if necessary.
> 
> This commit creates the basic framework for a "health" thread
> to monitor the daemon and/or the file system.  Later commits
> will add platform-specific code to do the actual work.

...

> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c
> new file mode 100644
> index 00000000000..b9f709e8548
> --- /dev/null
> +++ b/compat/fsmonitor/fsm-health-darwin.c
> @@ -0,0 +1,24 @@
> +#include "cache.h"
> +#include "config.h"
> +#include "fsmonitor.h"
> +#include "fsm-health.h"
> +#include "fsmonitor--daemon.h"
> +
> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)
> +{
> +	return 0;
> +}
> +
> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)
> +{
> +	return;
> +}
> +
> +void fsm_health__loop(struct fsmonitor_daemon_state *state)
> +{
> +	return;
> +}
> +
> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
> +{
> +}

The macOS implementation is stubbed, as advertised.

> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
> new file mode 100644
> index 00000000000..94b1d020f25
> --- /dev/null
> +++ b/compat/fsmonitor/fsm-health-win32.c
> @@ -0,0 +1,72 @@
> +#include "cache.h"
> +#include "config.h"
> +#include "fsmonitor.h"
> +#include "fsm-health.h"
> +#include "fsmonitor--daemon.h"
> +
> +struct fsm_health_data
> +{
> +	HANDLE hEventShutdown;
> +
> +	HANDLE hHandles[1]; /* the array does not own these handles */
> +#define HEALTH_SHUTDOWN 0
> +	int nr_handles; /* number of active event handles */
> +};
> +
> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)
> +{
> +	struct fsm_health_data *data;
> +
> +	CALLOC_ARRAY(data, 1);
> +
> +	data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);
> +
> +	data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;
> +	data->nr_handles++;
> +
> +	state->health_data = data;
> +	return 0;
> +}
> +
> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)
> +{
> +	struct fsm_health_data *data;
> +
> +	if (!state || !state->health_data)
> +		return;
> +
> +	data = state->health_data;
> +
> +	CloseHandle(data->hEventShutdown);
> +
> +	FREE_AND_NULL(state->health_data);
> +}
> +
> +void fsm_health__loop(struct fsmonitor_daemon_state *state)
> +{
> +	struct fsm_health_data *data = state->health_data;
> +
> +	for (;;) {
> +		DWORD dwWait = WaitForMultipleObjects(data->nr_handles,
> +						      data->hHandles,
> +						      FALSE, INFINITE);
> +
> +		if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)
> +			goto clean_shutdown;
> +
> +		error(_("health thread wait failed [GLE %ld]"),
> +		      GetLastError());
> +		goto force_error_stop;
> +	}
> +
> +force_error_stop:
> +	state->health_error_code = -1;
> +	ipc_server_stop_async(state->ipc_server_data);
> +clean_shutdown:
> +	return;
> +}
> +
> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
> +{
> +	SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);
> +}

But it apppears the Windows code is actually implemented. Did you
mean to do that as separate step, or should the commit message
mention that the Windows implementation is included?

Thanks,
-Stolee


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 18/23] fsm-health-win32: add framework to monitor daemon health
  2022-02-15 15:59 ` [PATCH 18/23] fsm-health-win32: add framework to monitor daemon health Jeff Hostetler via GitGitGadget
@ 2022-02-24 16:05   ` Derrick Stolee
  0 siblings, 0 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 16:05 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Create framework in Win32 version of the "health" thread to
> periodically inspect the system and shutdown if warranted.

This specific step makes sense to not be included in the
previous step.

> +/*
> + * Every minute wake up and test our health.
> + */
> +#define WAIT_FREQ_MS (60 * 1000)
> +
> +enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };

Please split out the values into their own lines.

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves
  2022-02-15 15:59 ` [PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves Jeff Hostetler via GitGitGadget
@ 2022-02-24 16:09   ` Derrick Stolee
  2022-03-03 18:00     ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 16:09 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
> index 3c3453369cd..2526ad9194f 100644
> --- a/compat/fsmonitor/fsm-health-win32.c
> +++ b/compat/fsmonitor/fsm-health-win32.c
> @@ -14,7 +14,10 @@ enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };
>  typedef int (interval_fn)(struct fsmonitor_daemon_state *state,
>  			  enum interval_fn_ctx ctx);
>  
> +static interval_fn has_worktree_moved;
> +
>  static interval_fn *table[] = {
> +	has_worktree_moved,
>  	NULL, /* must be last */
>  };

Looking at this now, I think table[] should be defined immediately
before fsm_health__loop() so it is easier to see how they interact.
It also avoids this static declaration of the function before its
implementation.

Or, is there a reason it is so high up in the file?

Thanks,
-Stolee


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 20/23] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  2022-02-15 15:59 ` [PATCH 20/23] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Jeff Hostetler via GitGitGadget
@ 2022-02-24 16:10   ` Derrick Stolee
  0 siblings, 0 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 16:10 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:

> +		if (ef_is_root_changed(event_flags[k])) {
> +			/*
> +			 * The spelling of the pathname of the root directory
> +			 * has changed.  This includes the name of the root
> +			 * directory itself of of any parent directory in the

s/of of/or of/

> +			 * path.
> +			 *
> +			 * (There may be other conditions that throw this,
> +			 * but I couldn't find any information on it.)
> +			 *
> +			 * Force a shutdown now and avoid things getting
> +			 * out of sync.  The Unix domain socket is inside
> +			 * the .git directory and a spelling change will make
> +			 * it hard for clients to rendezvous with us.
> +			 */
> +			trace_printf_key(&trace_fsmonitor,
> +					 "event: root changed");
> +			goto force_shutdown;
> +		}
> +

Thanks,
-Stolee


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 21/23] fsmonitor: optimize processing of directory events
  2022-02-15 15:59 ` [PATCH 21/23] fsmonitor: optimize processing of directory events Jeff Hostetler via GitGitGadget
@ 2022-02-24 16:13   ` Derrick Stolee
  0 siblings, 0 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 16:13 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Teach Git to perform binary search over the cache-entries for a directory
> notification and then linearly scan forward to find the immediate children.
> 
> Previously, when the FSMonitor reported a modified directory Git would
> perform a linear search on the entire cache-entry array for all
> entries matching that directory prefix and invalidate them.  Since the
> cache-entry array is already sorted, we can use a binary search to
> find the first matching entry and then only linearly walk forward and
> invalidate entries until the prefix changes.
> 
> Also, the original code would invalidate anything having the same
> directory prefix.  Since a directory event should only be received for
> items that are immediately within the directory (and not within
> sub-directories of it), only invalidate those entries and not the
> whole subtree.

This makes sense as an optimization. I've also taken a close look at
the code to be sure this makes sense if the file system events are
for a directory within a sparse directory entry, and the end result
will be that the sparse directory is marked as invalid, which is fine.

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread
  2022-02-24 16:04   ` Derrick Stolee
@ 2022-02-24 16:15     ` Derrick Stolee
  2022-03-03 16:40       ` Jeff Hostetler
  2022-03-03 16:50       ` Jeff Hostetler
  2022-03-03 16:16     ` Jeff Hostetler
  1 sibling, 2 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 16:15 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/24/2022 11:04 AM, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Create another thread to watch over the daemon process and
>> automatically shut it down if necessary.
>>
>> This commit creates the basic framework for a "health" thread
>> to monitor the daemon and/or the file system.  Later commits
>> will add platform-specific code to do the actual work.
> 
> ...
> 
>> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c
>> new file mode 100644
>> index 00000000000..b9f709e8548
>> --- /dev/null
>> +++ b/compat/fsmonitor/fsm-health-darwin.c
>> @@ -0,0 +1,24 @@
>> +#include "cache.h"
>> +#include "config.h"
>> +#include "fsmonitor.h"
>> +#include "fsm-health.h"
>> +#include "fsmonitor--daemon.h"
>> +
>> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)
>> +{
>> +	return 0;
>> +}
>> +
>> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)
>> +{
>> +	return;
>> +}
>> +
>> +void fsm_health__loop(struct fsmonitor_daemon_state *state)
>> +{
>> +	return;
>> +}
>> +
>> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
>> +{
>> +}
> 
> The macOS implementation is stubbed, as advertised.

After looking at the rest of the patch series, it seems that these
are never filled in. Are some of the win32 health monitors also
appropriate for macOS? (They would need platform-specific checks,
probably.)

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 00/23] Builtin FSMonitor Part 3
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (23 preceding siblings ...)
  2022-02-16  1:00 ` [PATCH 00/23] Builtin FSMonitor Part 3 Junio C Hamano
@ 2022-02-24 16:21 ` Derrick Stolee
  2022-03-07 21:23   ` Jeff Hostetler
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
  25 siblings, 1 reply; 277+ messages in thread
From: Derrick Stolee @ 2022-02-24 16:21 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> Here is part 3 of my builtin FSMonitor series.
> 
> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in
> "seen", so this series should be considered a preview until part 2 moves to
> "next". (And part 2 should not graduate to "master" without this part.)
> 
> Part 2 established the client code (used by commands like git status) and an
> MVP implementation of the FSMonitor daemon. This was sufficient to test the
> concepts and basic functionality.
> 
> Part 3 finishes the daemon and adds additional tests. This includes:
> 
>  * On Windows, handle short- and long-name aliasing.
>  * On Mac, handle Unicode aliasing.
>  * Mark bare, virtual, and remote working directories incompatible with
>    FSMonitor.
>  * On Mac, ignore xattr change FSEvents.
>  * On Windows, cd out of the working directory root.
>  * Create a thread to monitor the health and shutdown the daemon if
>    necessary.
>  * Speed up handling of directory notification events.
>  * Test directory move and rename events.
>  * Add performance test.

I had been more involved in early versions of parts 1 and 2, but was
not paying attention much as these updates for part 3 were coming into
git-for-windows/git and microsoft/git. (I do recall some of the support
issues that motivated them.)

Since these have been battle-tested in the wild for a while, I do not
doubt that they are solving the user needs they intend to solve.

Instead, I focused on looking at the patches for anything that might
be abnormal. The layout of the code is clear enough that I can easily
see how information from the platform-specific code is being handled,
so I do not worry about corner cases being an issue from the Git side
of the code. As for the platform-specific stuff, maybe there are cases
that are missed that require deep expertise in those platform calls. I
cannot speak to that until we get a user report of something even
stranger happening than the special cases targeted here.

With that said, I mostly found superficial things to improve the
series. Some typos, some rename possibilities, things like that.
 
> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a
> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of
> the untracked cache (uc) and FSMonitor (fsm) on git status.
> 
> $ ./p7527-builtin-fsmonitor.sh 
> # passed all 67 test(s)
> 1..67
> Test                                                                 this tree         
> ---------------------------------------------------------------------------------------
> 7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12) 
> 7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24) 
> 7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47) 
> 7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05) 
> 7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)
> 7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31) 
> 7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75) 
> 7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36) 
> 
> 7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84) 
> 7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86) 
> 7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70) 
> 7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10) 
> 7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91) 
> 7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89) 
> 7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71) 
> 7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78) 
> 
> 7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11) 
> 7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)
> 7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96) 
> 7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72) 
> 7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25) 
> 7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68) 
> 7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99) 
> 7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19) 
> 
> 7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)   
> 7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64) 
> 7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92) 
> 7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)  
> 7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)   
> 7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49) 
> 7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)   
> 7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)   

The other stylistic thing is this performance test. It would be nice if
these tests were grouped by the operation (like "status after checkout")
so it is easier to compare the same operation across the matrix definitions.

This would require reordering the test definition as well as allowing the
different cases to simultaneously live in different repositories. The
p2000-sparse-operations.sh has this kind of organization, but you'll need
more involved test cases than "run this command".

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode
  2022-02-24 14:52   ` Derrick Stolee
@ 2022-02-24 17:33     ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
  2022-03-04 23:40       ` Jeff Hostetler
  2022-03-04 23:47       ` Jeff Hostetler
  0 siblings, 2 replies; 277+ messages in thread
From: Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= @ 2022-02-24 17:33 UTC (permalink / raw)
  To: Derrick Stolee; +Cc: Jeff Hostetler via GitGitGadget, git, Jeff Hostetler

On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> > From: Jeff Hostetler <jeffhost@microsoft.com>
> >
> > Confirm that macOS FS events are reported with a normalized spelling.
> >
> > APFS (and/or HFS+) is case-insensitive.  This means that case-independent

This is not true, strictly speaking.
You can format a disk with "case sensitive" or "case-insenstive, case preserving".
Both APFS and HFS+  can be formated that way.
The default, which is what you get when you get a new machine,
is "case-insenstive, case preserving".
And I assume, that more 99% of all disks are formated that way.
The "core.ignorecase" is used in the same way as it is used under NTFS,
FAT or all other case-insenstive file systems.
(and even ext4 can be formated case-insensitive these days.)

An interesting article can be found here:
https://lwn.net/Articles/784041/

And to be technically correct, I think that even NTFS can be
"configured to be case insensitive in an empty directory".

In that sense, I would like to avoid this statement, which
file system is case insensitive, and which is not.
Git assumes that after probing the FS in "git init" we have
a valid configuration in core.ignorecase.

> > lookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed.  But that
> > doesn't tell us how FS events are reported if we try "rm -rf .git" versus
> > "rm -rf .GIT".  Are the events reported using the on-disk spelling of the
> > pathname or in the spelling used by the command.
>
> Was this last sentence supposed to be a question?
>
> > NEEDSWORK: I was only able to test case.  It would be nice to add tests
>
> "I was only able test the APFS case."?
>
> > that use different Unicode spellings/normalizations and understand the
> > differences between APFS and HFS+ in this area.  We should confirm that
> > the spelling of the workdir paths that the daemon sends to clients are
> > always properly normalized.
>
> Are there any macOS experts out there who can help us find the answers
> to these questions?

There is a difference between HFS+ and APFS.
HFS+  is "unicode decomposing" when you call readdir() - file names
are stored decomposed on disk once created.
However, opening  file in precompsed form succeds.
In that sense I would strongly suspect, that any monitors are "sending"
the decomposed form (on HFS+).

APFS does not manipulate file names in that way, it is
"unicode normalization preserving and ignoring".


>
> > +# Confirm that MacOS hides all of the Unicode normalization and/or
> > +# case folding from the FS events.  That is, are the pathnames in the
> > +# FS events reported using the spelling on the disk or in the spelling
> > +# used by the other process.
> > +#
> > +# Note that we assume that the filesystem is set to case insensitive.
> > +#
> > +# NEEDSWORK: APFS handles Unicode and Unicode normalization
> > +# differently than HFS+.  I only have an APFS partition, so
> > +# more testing here would be helpful.

I have an older system and may be able to help, (but am busy this week)
> > +#
> > +
> > +# Rename .git using alternate spelling and confirm that the daemon
> > +# sees the event using the correct spelling and shutdown.
> > +test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '

Isn't this precondition is the wrong one and CASE_INSENSITIVE_FS is a better one ?

[snip]

I diddn't follow you series, but if there is a need to test under MacOS with HFS+,
feel free to cc me in the next round

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor
  2022-02-15 15:59 ` [PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-02-25 20:42   ` Ævar Arnfjörð Bjarmason
  2022-03-02 21:09     ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-02-25 20:42 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget; +Cc: git, Jeff Hostetler


On Tue, Feb 15 2022, Jeff Hostetler via GitGitGadget wrote:

> +static void create_reason_message(struct repository *r,
> +				  struct strbuf *buf_reason)
> +{
> +	struct fsmonitor_settings *s = r->settings.fsmonitor;
> +
> +	switch (s->reason) {
> +	case FSMONITOR_REASON_ZERO:
> +		return;
> +
> +	case FSMONITOR_REASON_BARE:
> +		strbuf_addstr(buf_reason,
> +			      _("bare repos are incompatible with fsmonitor"));
> +		return;
> +
> +	default:
> +		BUG("Unhandled case in create_reason_message '%d'", s->reason);
> +	}
> +}
> +
> +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
> +					       struct strbuf *buf_reason)
> +{
> +	lookup_fsmonitor_settings(r);
> +
> +	strbuf_reset(buf_reason);
> +	if (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)
> +		create_reason_message(r, buf_reason);
> +
> +	return r->settings.fsmonitor->reason;
> +}

This API (just looking at one small bit discussed because related bits
conflict with another series) seems to require a lot of ceremony just to
get a const char * error.

I tried this on top of "seen", and the parts I compile on Linux (so not
the fsmonitor--daemon.c) were happy with it.

 builtin/fsmonitor--daemon.c | 14 +++++++-------
 builtin/update-index.c      |  9 ++++-----
 fsmonitor-settings.c        | 47 +++++++++++++++------------------------------
 fsmonitor-settings.h        |  5 +++--
 4 files changed, 29 insertions(+), 46 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 591433e897d..7ad7bc718b3 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1496,7 +1496,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
 {
 	const char *subcmd;
 	int free_console = 0;
-
+	enum fsmonitor_mode fsm_mode;
 	struct option options[] = {
 		OPT_BOOL(0, "free-console", &free_console, N_("free console")),
 		OPT_INTEGER(0, "ipc-threads",
@@ -1524,12 +1524,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
 	prepare_repo_settings(the_repository);
 	fsm_settings__set_ipc(the_repository);
 
-	if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {
-		struct strbuf buf_reason = STRBUF_INIT;
-		fsm_settings__get_reason(the_repository, &buf_reason);
-		error("%s '%s'", buf_reason.buf, xgetcwd());
-		strbuf_release(&buf_reason);
-		return -1;
+	fsm_mode = fsm_settings__get_mode(the_repository);
+	if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
+		enum fsmonitor_reason fsm_reason = fsm_settings__get_reason(the_repository);
+		const char *reason = fsm_settings_incompatible_reason_msg(fsm_mode, fsm_reason);
+
+		return error("%s", reason ? reason : "???");
 	}
 
 	if (!strcmp(subcmd, "start"))
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 61b0b98ccaf..f8f638d33d9 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1237,13 +1237,12 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 
 	if (fsmonitor > 0) {
 		enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
+		enum fsmonitor_reason fsm_reason = fsm_settings__get_reason(r);
 
 		if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
-			struct strbuf buf_reason = STRBUF_INIT;
-			fsm_settings__get_reason(r, &buf_reason);
-			error("%s", buf_reason.buf);
-			strbuf_release(&buf_reason);
-			return -1;
+			const char *reason = fsm_settings_incompatible_reason_msg(fsm_mode, fsm_reason);
+
+			return error("%s", reason ? reason : "???");
 		}
 
 		if (fsm_mode == FSMONITOR_MODE_DISABLED) {
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index de69ace246a..e37b342aa2b 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -103,6 +103,13 @@ enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)
 	return r->settings.fsmonitor->mode;
 }
 
+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)
+{
+	lookup_fsmonitor_settings(r);
+
+	return r->settings.fsmonitor->reason;
+}
+
 const char *fsm_settings__get_hook_path(struct repository *r)
 {
 	lookup_fsmonitor_settings(r);
@@ -142,43 +149,19 @@ void fsm_settings__set_disabled(struct repository *r)
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 }
 
-static void create_reason_message(struct repository *r,
-				  struct strbuf *buf_reason)
+const char *fsm_settings_incompatible_reason_msg(enum fsmonitor_mode mode,
+						 enum fsmonitor_reason reason)
 {
-	struct fsmonitor_settings *s = r->settings.fsmonitor;
+	assert(mode == FSMONITOR_MODE_INCOMPATIBLE);
 
-	switch (s->reason) {
+	switch (reason) {
 	case FSMONITOR_REASON_ZERO:
-		return;
-
+		return NULL;
 	case FSMONITOR_REASON_BARE:
-		strbuf_addstr(buf_reason,
-			      _("bare repos are incompatible with fsmonitor"));
-		return;
-
+		return _("bare repos are incompatible with fsmonitor");
 	case FSMONITOR_REASON_VIRTUAL:
-		strbuf_addstr(buf_reason,
-			      _("virtual repos are incompatible with fsmonitor"));
-		return;
-
+		return _("virtual repos are incompatible with fsmonitor");
 	case FSMONITOR_REASON_REMOTE:
-		strbuf_addstr(buf_reason,
-			      _("remote repos are incompatible with fsmonitor"));
-		return;
-
-	default:
-		BUG("Unhandled case in create_reason_message '%d'", s->reason);
+		return _("remote repos are incompatible with fsmonitor");
 	}
 }
-
-enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
-					       struct strbuf *buf_reason)
-{
-	lookup_fsmonitor_settings(r);
-
-	strbuf_reset(buf_reason);
-	if (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)
-		create_reason_message(r, buf_reason);
-
-	return r->settings.fsmonitor->reason;
-}
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index fca25887c0f..81be1ef1801 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -25,9 +25,10 @@ void fsm_settings__set_hook(struct repository *r, const char *path);
 void fsm_settings__set_disabled(struct repository *r);
 
 enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);
+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);
 const char *fsm_settings__get_hook_path(struct repository *r);
-enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
-					       struct strbuf *buf_reason);
+const char *fsm_settings_incompatible_reason_msg(enum fsmonitor_mode mode,
+						 enum fsmonitor_reason reason);
 
 struct fsmonitor_settings;
 


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 04/23] t/helper/fsmonitor-client: create stress test
  2022-02-24 14:58   ` Derrick Stolee
@ 2022-03-01 19:37     ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-01 19:37 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler



On 2/24/22 9:58 AM, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Create a stress test to hammer on the fsmonitor daemon.
>> Create a client-side thread pool of n threads and have
>> each of them make m requests as fast as they can.
>>
>> NEEDSWORK: This is just the client-side thread pool and
>> is useful for interactive testing and experimentation.
>> We need to add a script test to drive this.
> 
> I haven't gotten far enough in the series to know if you
> _do_ use this in a test eventually. If so, this NEEDSWORK
> could be replaced with a mention of a future change.

Right. I mainly use this test helper to hammer on the
daemon during interactive tests.  I don't have a script
to actually use.  I'm currently not sure what that would
look like given our test script framework.

> 
>> +	/*
>> +	 * TODO Decide if/when to return an error or call die().
>> +	 */
>> +	return 0;
> 
> This TODO could be cleaned up.

good catch.  thanks!

Jeff


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 07/23] fsmonitor-settings: virtual repos are incompatible with FSMonitor
  2022-02-24 15:11   ` Derrick Stolee
@ 2022-03-01 21:01     ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-01 21:01 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git
  Cc: Jeff Hostetler, Jonathan Nieder, Junio C Hamano



On 2/24/22 10:11 AM, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Virtual repos, such as GVFS (aka VFS for Git), are incompatible
>> with FSMonitor.
> 
> I would swap all of your "GVFS (aka VFS for Git)" for just
> "VFS for Git".
> 
>> +/*
>> + * GVFS (aka VFS for Git) is incompatible with FSMonitor.
>> + *
>> + * Granted, core Git does not know anything about GVFS and we
>> + * shouldn't make assumptions about a downstream feature, but users
>> + * can install both versions.  And this can lead to incorrect results
>> + * from core Git commands.  So, without bringing in any of the GVFS
>> + * code, do a simple config test for a published config setting.  (We
>> + * do not look at the various *_TEST_* environment variables.)
>> + */
>> +static enum fsmonitor_reason is_virtual(struct repository *r)
>> +{
>> +	const char *const_str;
>> +
>> +	if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str))
>> +		return FSMONITOR_REASON_VIRTUAL;
>> +
>> +	return FSMONITOR_REASON_ZERO;
>> +}
> 
> This reason seems to be specific to a config setting that only
> exists in the microsoft/git fork. Perhaps this patch should remain
> there.
> 
> However, there is also the discussion of vfsd going on, so something
> similar might be necessary for that system. Junio also mentioned
> wanting to collaborate on a common indicator that virtualization was
> being used, so maybe we _should_ make core.virtualFilesystem a config
> setting in the core Git project.
> 
> The reason for the incompatibility here is that VFS for Git has its
> own filesystem watcher and Git gets updates from it via custom code
> that is a precursor to this FS Monitor feature. I don't know if vfsd
> has plans for a similar setup. (It would probably be best to fit
> into the FS Monitor client/server model and use a different daemon
> for those virtualized repos, but I don't know enough details to be
> sure.)
> 
> CC'ing Jonathan Nieder for thoughts on this.

I was wondering whether this should be upstream or just in our
downstream forks, but I thought it better to include it so that
we don't try to monitor a virtualized repo and not mislead the
user.  It may be that we can correctly watch the repo and
generate correct results, but without knowing any details of
what the virtualization is doing behind the scenes, we would
be making some unsafe assumptions.  And since Windows users often
have multiple versions of Git installed (their CL tools and
whatever their IDE installed), promoting this check to upstream
felt important.

WRT the ongoing "vfsd" effort, I'm not sure what that looks like
yet and/or whether repos managed by "vfsd" have similar concerns.

I'll rename the variables in my patch here to be "vfs4git" rather
the generic "virtual".  This will avoid confusion later if another
case needs to be added for "vfsd".  My code is Win32-specific and
it is unclear it theirs will be Linux-only or cross-platform, so
hopefully with the rename we can coexist sanely.

Jeff



^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 09/23] fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor
  2022-02-24 15:26   ` Derrick Stolee
@ 2022-03-01 21:30     ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-01 21:30 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler



On 2/24/22 10:26 AM, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Teach Git to detect remote working directories on macOS and mark them as
>> incompatible with FSMonitor.
>>
>> With this, `git fsmonitor--daemon run` will error out with a message
>> like it does for bare repos.
>>
>> Client commands, like `git status`, will not attempt to start the daemon.
> 
> ...
> 
>> + * A client machine (such as a laptop) may choose to suspend/resume
>> + * and it is unclear (without lots of testing) whether the watcher can
>> + * resync after a resume.  We might be able to treat this as a normal
>> + * "events were dropped by the kernel" event and do our normal "flush
>> + * and resync" --or-- we might need to close the existing (zombie?)
>> + * notification fd and create a new one.
>> + *
>> + * In theory, the above issues need to be addressed whether we are
>> + * using the Hook or IPC API.
> 
> The only thing I can think about is a case where the filesystem
> monitor is actually running on the remote machine and Git
> communicates with it over the network. This is currently possible
> with the hook, but I am not aware of a hook implementation that
> does this.
> 
> We can find a way to update the hook interface to communicate to
> Git that a remote disk is an appropriate case, but only if there
> is a real need for that.

I'm not saying we can't support remote working directories.
We do get events from SMB for example, but there network
buffer limits and all the usual connectivity issues with a
local daemon reading an event stream from a remote file system.
So out of caution, I want to disable it for now.  We can always
revisit it later.

WRT the builtin IPC-based daemon, I have the socket/named pipe
restricted to local access only.  This prevents a client like
"git status" from talking to a remote daemon, but it also prevents
a remote bad guy from talking our daemon.  But those are secondary
concerns right now -- I'm mainly concerned with correctly getting
all of the (possibly high volume/speed) events to the client.

There's also the possible incompatibilities of vendor X client-side
OS and vendor Y server-side OS and how well notifications are
supported between them....

So I'd just like to shut it off for now.

> 
>> + * For the builtin FSMonitor, we create the Unix domain socket for the
>> + * IPC in the .git directory.  If the working directory is remote,
>> + * then the socket will be created on the remote file system.  This
> 
> The socket is on the remote file system, but the daemon process is still
> local, so I still see this a problem.
> 
>> + * can fail if the remote file system does not support UDS file types
>> + * (e.g. smbfs to a Windows server) or if the remote kernel does not
>> + * allow a non-local process to bind() the socket.  (These problems
>> + * could be fixed by moving the UDS out of the .git directory and to a
>> + * well-known local directory on the client machine, but care should
>> + * be taken to ensure that $HOME is actually local and not a managed
>> + * file share.)
>> + *
>> + * So (for now at least), mark remote working directories as
>> + * incompatible.
>> + */
>> +static enum fsmonitor_reason is_remote(struct repository *r)
>> +{
>> +	struct statfs fs;
>> +
>> +	if (statfs(r->worktree, &fs) == -1) {
>> +		int saved_errno = errno;
>> +		trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s",
>> +				 r->worktree, strerror(saved_errno));
>> +		errno = saved_errno;
>> +		return FSMONITOR_REASON_ZERO;
> 
> So if we fail to inspect the filesystem, we report it as compatible?
> I suppose other things are likely to fail if checks like this are
> fialing, but I wonder if we should preempt that by marking this as
> incompatible due to filesystem errors.

Good point.

> 
>> +	}
>> +
>> +	trace_printf_key(&trace_fsmonitor,
>> +			 "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'",
>> +			 r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);
>> +
>> +	if (!(fs.f_flags & MNT_LOCAL))
>> +		return FSMONITOR_REASON_REMOTE;
> 
> I do see that we need a successful response to give this specific
> reason for incompatibility.
> 
>> +	return FSMONITOR_REASON_ZERO;
>> +}
>>   
>>   enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
>>   {
>> +	enum fsmonitor_reason reason;
>> +
>> +	reason = is_remote(r);
>> +	if (reason)
>> +		return reason;
> 
> This organization is looking like you want to short-circuit the
> checks if you find an incompatibility, with the intent of having
> multiple checks in the future.
> 
> But this can be done with simple || operators:
> 
> 	return is_remote() ||
> 	       reason_check_2() ||
> 	       reason_check_3();

True.  That is a little shorter.

Thanks
Jeff

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 05/23] fsmonitor-settings: bare repos are incompatible with FSMonitor
  2022-02-25 20:42   ` Ævar Arnfjörð Bjarmason
@ 2022-03-02 21:09     ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-02 21:09 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason, Jeff Hostetler via GitGitGadget
  Cc: git, Jeff Hostetler



On 2/25/22 3:42 PM, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, Feb 15 2022, Jeff Hostetler via GitGitGadget wrote:
> 
>> +static void create_reason_message(struct repository *r,
>> +				  struct strbuf *buf_reason)
>> +{
>> +	struct fsmonitor_settings *s = r->settings.fsmonitor;
>> +
>> +	switch (s->reason) {
>> +	case FSMONITOR_REASON_ZERO:
>> +		return;
>> +
>> +	case FSMONITOR_REASON_BARE:
>> +		strbuf_addstr(buf_reason,
>> +			      _("bare repos are incompatible with fsmonitor"));
>> +		return;
>> +
>> +	default:
>> +		BUG("Unhandled case in create_reason_message '%d'", s->reason);
>> +	}
>> +}
>> +
>> +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
>> +					       struct strbuf *buf_reason)
>> +{
>> +	lookup_fsmonitor_settings(r);
>> +
>> +	strbuf_reset(buf_reason);
>> +	if (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)
>> +		create_reason_message(r, buf_reason);
>> +
>> +	return r->settings.fsmonitor->reason;
>> +}
> 
> This API (just looking at one small bit discussed because related bits
> conflict with another series) seems to require a lot of ceremony just to
> get a const char * error.

My thought at the time was that the __get_reason() code might want to
format a message and include details from the repo (such as the working
directory root) or the kind of objection (such as a remote SMB mount).
So I had it take a strbuf rather than a "const char *" return value.

But so far (in the successive commits) all of the reason messages have
been constant and I think I'm OK with having the non-specific messages.

So yes, I could simplify it.

Thanks
Jeff


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread
  2022-02-24 16:04   ` Derrick Stolee
  2022-02-24 16:15     ` Derrick Stolee
@ 2022-03-03 16:16     ` Jeff Hostetler
  1 sibling, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-03 16:16 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler



On 2/24/22 11:04 AM, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Create another thread to watch over the daemon process and
>> automatically shut it down if necessary.
>>
>> This commit creates the basic framework for a "health" thread
>> to monitor the daemon and/or the file system.  Later commits
>> will add platform-specific code to do the actual work.
> 
> ...
...
> 
>> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
>> new file mode 100644
>> index 00000000000..94b1d020f25
>> --- /dev/null
>> +++ b/compat/fsmonitor/fsm-health-win32.c
>> @@ -0,0 +1,72 @@
>> +#include "cache.h"
>> +#include "config.h"
>> +#include "fsmonitor.h"
>> +#include "fsm-health.h"
>> +#include "fsmonitor--daemon.h"
>> +
>> +struct fsm_health_data
>> +{
>> +	HANDLE hEventShutdown;
>> +
>> +	HANDLE hHandles[1]; /* the array does not own these handles */
>> +#define HEALTH_SHUTDOWN 0
>> +	int nr_handles; /* number of active event handles */
>> +};
>> +
>> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)
>> +{
>> +	struct fsm_health_data *data;
>> +
>> +	CALLOC_ARRAY(data, 1);
>> +
>> +	data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);
>> +
>> +	data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;
>> +	data->nr_handles++;
>> +
>> +	state->health_data = data;
>> +	return 0;
>> +}
>> +
>> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)
>> +{
>> +	struct fsm_health_data *data;
>> +
>> +	if (!state || !state->health_data)
>> +		return;
>> +
>> +	data = state->health_data;
>> +
>> +	CloseHandle(data->hEventShutdown);
>> +
>> +	FREE_AND_NULL(state->health_data);
>> +}
>> +
>> +void fsm_health__loop(struct fsmonitor_daemon_state *state)
>> +{
>> +	struct fsm_health_data *data = state->health_data;
>> +
>> +	for (;;) {
>> +		DWORD dwWait = WaitForMultipleObjects(data->nr_handles,
>> +						      data->hHandles,
>> +						      FALSE, INFINITE);
>> +
>> +		if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)
>> +			goto clean_shutdown;
>> +
>> +		error(_("health thread wait failed [GLE %ld]"),
>> +		      GetLastError());
>> +		goto force_error_stop;
>> +	}
>> +
>> +force_error_stop:
>> +	state->health_error_code = -1;
>> +	ipc_server_stop_async(state->ipc_server_data);
>> +clean_shutdown:
>> +	return;
>> +}
>> +
>> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
>> +{
>> +	SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);
>> +}
> 
> But it apppears the Windows code is actually implemented. Did you
> mean to do that as separate step, or should the commit message
> mention that the Windows implementation is included?

The Windows version stubs in just enough of the thread-proc to keep
the health thread alive and waiting for a shutdown event.  It doesn'ta
actually have any health monitor in it yet.

I'll update the commit message to clarify.

Thanks
Jeff


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread
  2022-02-24 16:15     ` Derrick Stolee
@ 2022-03-03 16:40       ` Jeff Hostetler
  2022-03-03 16:50       ` Jeff Hostetler
  1 sibling, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-03 16:40 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler



On 2/24/22 11:15 AM, Derrick Stolee wrote:
> On 2/24/2022 11:04 AM, Derrick Stolee wrote:
>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>>
>>> Create another thread to watch over the daemon process and
>>> automatically shut it down if necessary.
>>>
>>> This commit creates the basic framework for a "health" thread
>>> to monitor the daemon and/or the file system.  Later commits
>>> will add platform-specific code to do the actual work.
>>
>> ...
>>
>>> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c
>>> new file mode 100644
>>> index 00000000000..b9f709e8548
>>> --- /dev/null
>>> +++ b/compat/fsmonitor/fsm-health-darwin.c
>>> @@ -0,0 +1,24 @@
>>> +#include "cache.h"
>>> +#include "config.h"
>>> +#include "fsmonitor.h"
>>> +#include "fsm-health.h"
>>> +#include "fsmonitor--daemon.h"
>>> +
>>> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)
>>> +{
>>> +	return 0;
>>> +}
>>> +
>>> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)
>>> +{
>>> +	return;
>>> +}
>>> +
>>> +void fsm_health__loop(struct fsmonitor_daemon_state *state)
>>> +{
>>> +	return;
>>> +}
>>> +
>>> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
>>> +{
>>> +}
>>
>> The macOS implementation is stubbed, as advertised.
> 
> After looking at the rest of the patch series, it seems that these
> are never filled in. Are some of the win32 health monitors also
> appropriate for macOS? (They would need platform-specific checks,
> probably.)
> 
> Thanks,
> -Stolee
> 

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 17/23] fsmonitor--daemon: stub in health thread
  2022-02-24 16:15     ` Derrick Stolee
  2022-03-03 16:40       ` Jeff Hostetler
@ 2022-03-03 16:50       ` Jeff Hostetler
  1 sibling, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-03 16:50 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler



On 2/24/22 11:15 AM, Derrick Stolee wrote:
> On 2/24/2022 11:04 AM, Derrick Stolee wrote:
>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>>
>>> Create another thread to watch over the daemon process and
>>> automatically shut it down if necessary.
>>>
>>> This commit creates the basic framework for a "health" thread
>>> to monitor the daemon and/or the file system.  Later commits
>>> will add platform-specific code to do the actual work.
>>
>> ...
>>
>>> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c
>>> new file mode 100644
>>> index 00000000000..b9f709e8548
>>> --- /dev/null
>>> +++ b/compat/fsmonitor/fsm-health-darwin.c
>>> @@ -0,0 +1,24 @@
>>> +#include "cache.h"
>>> +#include "config.h"
>>> +#include "fsmonitor.h"
>>> +#include "fsm-health.h"
>>> +#include "fsmonitor--daemon.h"
>>> +
>>> +int fsm_health__ctor(struct fsmonitor_daemon_state *state)
>>> +{
>>> +	return 0;
>>> +}
>>> +
>>> +void fsm_health__dtor(struct fsmonitor_daemon_state *state)
>>> +{
>>> +	return;
>>> +}
>>> +
>>> +void fsm_health__loop(struct fsmonitor_daemon_state *state)
>>> +{
>>> +	return;
>>> +}
>>> +
>>> +void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
>>> +{
>>> +}
>>
>> The macOS implementation is stubbed, as advertised.
> 
> After looking at the rest of the patch series, it seems that these
> are never filled in. Are some of the win32 health monitors also
> appropriate for macOS? (They would need platform-specific checks,
> probably.)

Yes, there are some asymmetries here.  For example: On MacOS we
get a notification at part of the existing watch if the repo root
directory is moved or renamed, so (in a later commit) we can add
code to the listener thread to check for that; however, on Windows
we don't, so we have to poll for it in the health thread using
timers and the BY_HANDLE_FILE_INFORMATION data.

Here I'm just stubbing in a trivial health thread so that the framework
is complete for the supported platforms.

Thanks,
Jeff


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 19/23] fsm-health-win32: force shutdown daemon if worktree root moves
  2022-02-24 16:09   ` Derrick Stolee
@ 2022-03-03 18:00     ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-03 18:00 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler



On 2/24/22 11:09 AM, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>> diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
>> index 3c3453369cd..2526ad9194f 100644
>> --- a/compat/fsmonitor/fsm-health-win32.c
>> +++ b/compat/fsmonitor/fsm-health-win32.c
>> @@ -14,7 +14,10 @@ enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };
>>   typedef int (interval_fn)(struct fsmonitor_daemon_state *state,
>>   			  enum interval_fn_ctx ctx);
>>   
>> +static interval_fn has_worktree_moved;
>> +
>>   static interval_fn *table[] = {
>> +	has_worktree_moved,
>>   	NULL, /* must be last */
>>   };
> 
> Looking at this now, I think table[] should be defined immediately
> before fsm_health__loop() so it is easier to see how they interact.
> It also avoids this static declaration of the function before its
> implementation.
> 
> Or, is there a reason it is so high up in the file?

I don't think so.  I was trying to keep all of the public API
routines at the bottom, but all of the static stuff is pretty
much free to move around.

I'll revisit.

Thanks
Jeff

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode
  2022-02-24 17:33     ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
@ 2022-03-04 23:40       ` Jeff Hostetler
  2022-03-05  8:59         ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
  2022-03-04 23:47       ` Jeff Hostetler
  1 sibling, 1 reply; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-04 23:40 UTC (permalink / raw)
  To: Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=, Derrick Stolee
  Cc: Jeff Hostetler via GitGitGadget, git, Jeff Hostetler



On 2/24/22 12:33 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:
> On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:
>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>>
>>> Confirm that macOS FS events are reported with a normalized spelling.
>>>
>>> APFS (and/or HFS+) is case-insensitive.  This means that case-independent
> 
> This is not true, strictly speaking.
> You can format a disk with "case sensitive" or "case-insenstive, case preserving".
> Both APFS and HFS+  can be formated that way.
> The default, which is what you get when you get a new machine,
> is "case-insenstive, case preserving".
> And I assume, that more 99% of all disks are formated that way.
> The "core.ignorecase" is used in the same way as it is used under NTFS,
> FAT or all other case-insenstive file systems.
> (and even ext4 can be formated case-insensitive these days.)
> 
> An interesting article can be found here:
> https://lwn.net/Articles/784041/
> 
> And to be technically correct, I think that even NTFS can be
> "configured to be case insensitive in an empty directory".
> 
> In that sense, I would like to avoid this statement, which
> file system is case insensitive, and which is not.
> Git assumes that after probing the FS in "git init" we have
> a valid configuration in core.ignorecase.

You're right. I was incorrectly glossing over the differences
between APFS and HFS+ -- and conflating case and nfc/nfd
issues.

[...]
>>
>>> NEEDSWORK: I was only able to test case.  It would be nice to add tests
>>
>> "I was only able test the APFS case."?

I'm going to completely redo this commit in the next version.
I now have both APFS and HFS+ partitions on my machine and
can compare the differences in behaviors and will have a new
set of tests to cover this.

>>
>>> that use different Unicode spellings/normalizations and understand the
>>> differences between APFS and HFS+ in this area.  We should confirm that
>>> the spelling of the workdir paths that the daemon sends to clients are
>>> always properly normalized.
>>
>> Are there any macOS experts out there who can help us find the answers
>> to these questions?
> 
> There is a difference between HFS+ and APFS.
> HFS+  is "unicode decomposing" when you call readdir() - file names
> are stored decomposed on disk once created.
> However, opening  file in precompsed form succeds.
> In that sense I would strongly suspect, that any monitors are "sending"
> the decomposed form (on HFS+).
> 
> APFS does not manipulate file names in that way, it is
> "unicode normalization preserving and ignoring".

It took a few hours of poking to figure out what Apple is doing,
but yes on HFS+ they convert to NFD and use that as the on-disk
format.  And they do collision detection as they always have in
NFD-space.

Whereas on APFS, they preserve the NFC/NFD as given when the file
is created, but always do the same collision detection in NFD-space.
The net result is similar, but subtlety different.

FS Events from MacOS are sent using the on-disk format (NFD on HFS+
and whichever on APFS) and my FSMonitor daemon is sending them to
the client as received.

I'm not sure whether or not the daemon should respect the
`core.precompseUnicode` setting and when watching an HFS+
volume do the NFD-->NFC conversion for the client.  I'm not
sure whether that would be any more or less correct than just
reporting the paths as received.  I'm going to leave this as a
question for the future.


Thanks for all of your background information on this topic.
Jeff



^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode
  2022-02-24 17:33     ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
  2022-03-04 23:40       ` Jeff Hostetler
@ 2022-03-04 23:47       ` Jeff Hostetler
  1 sibling, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-04 23:47 UTC (permalink / raw)
  To: Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=, Derrick Stolee
  Cc: Jeff Hostetler via GitGitGadget, git, Jeff Hostetler



On 2/24/22 12:33 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:
> On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:
>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>>
>>> Confirm that macOS FS events are reported with a normalized spelling.
>>>
>>> APFS (and/or HFS+) is case-insensitive.  This means that case-independent
> 
[...]
> 
> An interesting article can be found here:
> https://lwn.net/Articles/784041/
> 
> And to be technically correct, I think that even NTFS can be
> "configured to be case insensitive in an empty directory".

Yes, it is now possible to have NTFS be case sensitive (on a
directory by directory basis).  I haven't had a chance to
experiment with this yet, but I'm hoping that if we can always
have the daemon report using the on-disk spelling, we can
avoid most of this insanity.

Thanks,
Jeff

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode
  2022-03-04 23:40       ` Jeff Hostetler
@ 2022-03-05  8:59         ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
  2022-03-07 20:45           ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= @ 2022-03-05  8:59 UTC (permalink / raw)
  To: Jeff Hostetler
  Cc: Derrick Stolee, Jeff Hostetler via GitGitGadget, git, Jeff Hostetler

On Fri, Mar 04, 2022 at 06:40:27PM -0500, Jeff Hostetler wrote:
>
>
> On 2/24/22 12:33 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:
> > On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:
> > > On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> > > > From: Jeff Hostetler <jeffhost@microsoft.com>
> > > >
> > > > Confirm that macOS FS events are reported with a normalized spelling.
> > > >
> > > > APFS (and/or HFS+) is case-insensitive.  This means that case-independent
> >
> > This is not true, strictly speaking.
> > You can format a disk with "case sensitive" or "case-insenstive, case preserving".
> > Both APFS and HFS+  can be formated that way.
> > The default, which is what you get when you get a new machine,
> > is "case-insenstive, case preserving".
> > And I assume, that more 99% of all disks are formated that way.
> > The "core.ignorecase" is used in the same way as it is used under NTFS,
> > FAT or all other case-insenstive file systems.
> > (and even ext4 can be formated case-insensitive these days.)
> >
> > An interesting article can be found here:
> > https://lwn.net/Articles/784041/
> >
> > And to be technically correct, I think that even NTFS can be
> > "configured to be case insensitive in an empty directory".
> >
> > In that sense, I would like to avoid this statement, which
> > file system is case insensitive, and which is not.
> > Git assumes that after probing the FS in "git init" we have
> > a valid configuration in core.ignorecase.
>
> You're right. I was incorrectly glossing over the differences
> between APFS and HFS+ -- and conflating case and nfc/nfd
> issues.
>
> [...]
> > >
> > > > NEEDSWORK: I was only able to test case.  It would be nice to add tests
> > >
> > > "I was only able test the APFS case."?
>
> I'm going to completely redo this commit in the next version.
> I now have both APFS and HFS+ partitions on my machine and
> can compare the differences in behaviors and will have a new
> set of tests to cover this.
>
> > >
> > > > that use different Unicode spellings/normalizations and understand the
> > > > differences between APFS and HFS+ in this area.  We should confirm that
> > > > the spelling of the workdir paths that the daemon sends to clients are
> > > > always properly normalized.
> > >
> > > Are there any macOS experts out there who can help us find the answers
> > > to these questions?
> >
> > There is a difference between HFS+ and APFS.
> > HFS+  is "unicode decomposing" when you call readdir() - file names
> > are stored decomposed on disk once created.
> > However, opening  file in precompsed form succeds.
> > In that sense I would strongly suspect, that any monitors are "sending"
> > the decomposed form (on HFS+).
> >
> > APFS does not manipulate file names in that way, it is
> > "unicode normalization preserving and ignoring".
>
> It took a few hours of poking to figure out what Apple is doing,
> but yes on HFS+ they convert to NFD and use that as the on-disk
> format.  And they do collision detection as they always have in
> NFD-space.
>
> Whereas on APFS, they preserve the NFC/NFD as given when the file
> is created, but always do the same collision detection in NFD-space.
> The net result is similar, but subtlety different.

That depends what you mean with "net result".
What Git sees after calling precompose_utf8_readdir() with
core.precomposeunicode=true ?


>
> FS Events from MacOS are sent using the on-disk format (NFD on HFS+
> and whichever on APFS) and my FSMonitor daemon is sending them to
> the client as received.
>
> I'm not sure whether or not the daemon should respect the
> `core.precompseUnicode` setting and when watching an HFS+
> volume do the NFD-->NFC conversion for the client.  I'm not
> sure whether that would be any more or less correct than just
> reporting the paths as received.  I'm going to leave this as a
> question for the future.

I think that I have a suggestion for an answer:
We still have HFS+ systems around, and we still have an NFD feature
in MacOs for USB sticks with FAT or SAMBA mounted network volumes.
Both return NFD in readdir().
Even if NFC is on disk for FAT or going over the wire for SAMBA.
Having a precompose() function in the FSMonitor would help to make
things consistent.
And the answer is yes.

>
>
> Thanks for all of your background information on this topic.
> Jeff
>

The pleasure is on my side.
Please feel free to cc me for the next round, so that I don't miss
to review them.

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 02/23] t7527: test FS event reporing on macOS WRT case and Unicode
  2022-03-05  8:59         ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
@ 2022-03-07 20:45           ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-07 20:45 UTC (permalink / raw)
  To: Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
  Cc: Derrick Stolee, Jeff Hostetler via GitGitGadget, git, Jeff Hostetler



On 3/5/22 3:59 AM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:
> On Fri, Mar 04, 2022 at 06:40:27PM -0500, Jeff Hostetler wrote:
>>
>>
>> On 2/24/22 12:33 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:
>>> On Thu, Feb 24, 2022 at 09:52:28AM -0500, Derrick Stolee wrote:
>>>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>>>>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>>>>
>>>>> Confirm that macOS FS events are reported with a normalized spelling.
[...]
>>
>>>>
>>>>> that use different Unicode spellings/normalizations and understand the
>>>>> differences between APFS and HFS+ in this area.  We should confirm that
>>>>> the spelling of the workdir paths that the daemon sends to clients are
>>>>> always properly normalized.
>>>>
>>>> Are there any macOS experts out there who can help us find the answers
>>>> to these questions?
>>>
>>> There is a difference between HFS+ and APFS.
>>> HFS+  is "unicode decomposing" when you call readdir() - file names
>>> are stored decomposed on disk once created.
>>> However, opening  file in precompsed form succeds.
>>> In that sense I would strongly suspect, that any monitors are "sending"
>>> the decomposed form (on HFS+).
>>>
>>> APFS does not manipulate file names in that way, it is
>>> "unicode normalization preserving and ignoring".
>>
>> It took a few hours of poking to figure out what Apple is doing,
>> but yes on HFS+ they convert to NFD and use that as the on-disk
>> format.  And they do collision detection as they always have in
>> NFD-space.
>>
>> Whereas on APFS, they preserve the NFC/NFD as given when the file
>> is created, but always do the same collision detection in NFD-space.
>> The net result is similar, but subtlety different.
> 
> That depends what you mean with "net result".
> What Git sees after calling precompose_utf8_readdir() with
> core.precomposeunicode=true ?

I just meant that the OS always does the collision detection
regardless of whether the filesystem is APFS or HFS. (They even
do it on a FAT32 thumb drive).  So the OS layer is always
composition insensitive while the underlying filesystem may
or may not be composition preserving.


>>
>> FS Events from MacOS are sent using the on-disk format (NFD on HFS+
>> and whichever on APFS) and my FSMonitor daemon is sending them to
>> the client as received.
>>
>> I'm not sure whether or not the daemon should respect the
>> `core.precompseUnicode` setting and when watching an HFS+
>> volume do the NFD-->NFC conversion for the client.  I'm not
>> sure whether that would be any more or less correct than just
>> reporting the paths as received.  I'm going to leave this as a
>> question for the future.
> 
> I think that I have a suggestion for an answer:
> We still have HFS+ systems around, and we still have an NFD feature
> in MacOs for USB sticks with FAT or SAMBA mounted network volumes.
> Both return NFD in readdir().
> Even if NFC is on disk for FAT or going over the wire for SAMBA.
> Having a precompose() function in the FSMonitor would help to make
> things consistent.
> And the answer is yes.

I agree.  I'm going to have the mac version always send the NFC
version (like core.precomposeUnicode suggests), but also send NFD
for *if* we get an NFD spelling from the FS event.  So on an HFS
volume we'll get two events for each path that have different
UTF8 spellings.  In the case of APFS we should only get the NFC
spelling -- unless the user really wants to use NFD spellings --
in which case we will respect that and send both.

This maybe be overkill, but the daemon (at the time that it
receives the FS event) doesn't know how the client process will
be configured when it makes a query at some point in the future.
So by recording both and reporting them to the client, the client
can decide how to process them.

Whichever spelling the client sees, it is only using it to mark
the cache-entry or untracked-cache data as possibly dirty so that
the original scanning code can actually check it.  This is a little
different from the diff-tree code that needs the official spelling
from the tree entry to match up items with the working directory.

I've overhauled all of the MacOS Unicode handling in V2.

Jeff


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 00/23] Builtin FSMonitor Part 3
  2022-02-24 16:21 ` Derrick Stolee
@ 2022-03-07 21:23   ` Jeff Hostetler
  2022-03-09 15:34     ` Derrick Stolee
  0 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-07 21:23 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler



On 2/24/22 11:21 AM, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>> Here is part 3 of my builtin FSMonitor series.
[...]
>> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a
>> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of
>> the untracked cache (uc) and FSMonitor (fsm) on git status.
>>
>> $ ./p7527-builtin-fsmonitor.sh
>> # passed all 67 test(s)
>> 1..67
>> Test                                                                 this tree
>> ---------------------------------------------------------------------------------------
>> 7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12)
>> 7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24)
>> 7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47)
>> 7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05)
>> 7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)
>> 7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31)
>> 7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75)
>> 7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36)
>>
>> 7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84)
>> 7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86)
>> 7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70)
>> 7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10)
>> 7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91)
>> 7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89)
>> 7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71)
>> 7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78)
>>
>> 7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11)
>> 7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)
>> 7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96)
>> 7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72)
>> 7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25)
>> 7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68)
>> 7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99)
>> 7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19)
>>
>> 7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)
>> 7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64)
>> 7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92)
>> 7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)
>> 7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)
>> 7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49)
>> 7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)
>> 7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)
> 
> The other stylistic thing is this performance test. It would be nice if
> these tests were grouped by the operation (like "status after checkout")
> so it is easier to compare the same operation across the matrix definitions.
> 
> This would require reordering the test definition as well as allowing the
> different cases to simultaneously live in different repositories. The
> p2000-sparse-operations.sh has this kind of organization, but you'll need
> more involved test cases than "run this command".

Yeah, it would be nice to turn this test inside-out so that
could group the outputs by test case rather than by (uc,fsm)
combination.  That would certainly make it easier to see how
the two terms affect things.

The problem is I'd either need 4 parallel repos that I could
setup with each (uc,fsm) pair or I'd need to start/stop the
daemon and swap out the {.git/index, .git/config} between
each step.  The former is a problem for monorepos.  The latter
is doable, but I'm not sure it is worth the effort right now.

Jeff

^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-02-15 15:59 [PATCH 00/23] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
                   ` (24 preceding siblings ...)
  2022-02-24 16:21 ` Derrick Stolee
@ 2022-03-08 22:15 ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 01/27] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
                     ` (30 more replies)
  25 siblings, 31 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler

Here is V2 of part 3 of my builtin FSMonitor series.

I think I have addressed all of the feedback from V1. This includes:

[] AEvar's suggestion to simplify the API to get the repo incompatibility
message.

[] I updated the daemon on MacOS to report both the NFC and NFD spellings of
a pathname when appropriate. This is a little more general than the
"core.precomposeUnicode" setting, since the daemon does not know how the
client has (or will have) it set when they make a query.

[] I replaced my Unicode NFC/NFD test for MacOS to focus exclusively on
Unicode composition/decomposition sensitivity and to not confuse that with
case sensitivity.

[] Added code on MacOS to mark repos on FAT32 and NTFS volumes as
incompatible with FSMonitor, since they don't support Unix domain sockets.

[] I reordered some of the static functions in the Windows version of the
health monitoring code to reduce the need for forward declarations.

[] Lots of typos and cleanup.

This version has been rebased upon V6 of Part 2 which is currently in
"next".

I'm not going to repeat the perf test results in the V2 version of the cover
letter; interested readers should look at the cover letter for V1.

Jeff Hostetler (27):
  fsm-listen-win32: handle shortnames
  t7527: test FSMonitor on repos with Unicode root paths
  t/helper/fsmonitor-client: create stress test
  fsmonitor-settings: bare repos are incompatible with FSMonitor
  fsmonitor-settings: stub in platform-specific incompatibility checking
  fsmonitor-settings: VFS for Git virtual repos are incompatible
  fsmonitor-settings: stub in macOS-specific incompatibility checking
  fsmonitor-settings: remote repos on macOS are incompatible
  fsmonitor-settings: remote repos on Windows are incompatible
  unpack-trees: initialize fsmonitor_has_run_once in o->result
  fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
  fsmonitor--daemon: cd out of worktree root
  fsmonitor--daemon: prepare for adding health thread
  fsmonitor--daemon: rename listener thread related variables
  fsmonitor--daemon: stub in health thread
  fsm-health-win32: add polling framework to monitor daemon health
  fsm-health-win32: force shutdown daemon if worktree root moves
  fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  fsmonitor: optimize processing of directory events
  t7527: FSMonitor tests for directory moves
  t/perf/p7527: add perf test for builtin FSMonitor
  fsmonitor: never set CE_FSMONITOR_VALID on submodules
  t7527: test FSMonitor on case insensitive+preserving file system
  fsmonitor: on macOS also emit NFC spelling for NFD pathname
  t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  t7527: test Unicode NFC/NFD handling on MacOS
  fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible

 Makefile                               |  19 +-
 builtin/fsmonitor--daemon.c            | 107 ++++++-
 builtin/update-index.c                 |   7 +
 compat/fsmonitor/fsm-health-darwin.c   |  24 ++
 compat/fsmonitor/fsm-health-win32.c    | 278 +++++++++++++++++
 compat/fsmonitor/fsm-health.h          |  47 +++
 compat/fsmonitor/fsm-listen-darwin.c   | 122 ++++++--
 compat/fsmonitor/fsm-listen-win32.c    | 413 ++++++++++++++++++++-----
 compat/fsmonitor/fsm-listen.h          |   2 +-
 compat/fsmonitor/fsm-settings-darwin.c |  89 ++++++
 compat/fsmonitor/fsm-settings-win32.c  | 137 ++++++++
 config.mak.uname                       |   5 +
 contrib/buildsystems/CMakeLists.txt    |   8 +
 fsmonitor--daemon.h                    |  11 +-
 fsmonitor-settings.c                   |  81 +++++
 fsmonitor-settings.h                   |  29 ++
 fsmonitor.c                            |  73 ++++-
 fsmonitor.h                            |  11 +
 t/helper/test-fsmonitor-client.c       | 106 +++++++
 t/lib-unicode-nfc-nfd.sh               | 159 ++++++++++
 t/perf/p7527-builtin-fsmonitor.sh      | 257 +++++++++++++++
 t/t7519-status-fsmonitor.sh            |  32 ++
 t/t7527-builtin-fsmonitor.sh           | 314 +++++++++++++++++++
 unpack-trees.c                         |   1 +
 24 files changed, 2208 insertions(+), 124 deletions(-)
 create mode 100644 compat/fsmonitor/fsm-health-darwin.c
 create mode 100644 compat/fsmonitor/fsm-health-win32.c
 create mode 100644 compat/fsmonitor/fsm-health.h
 create mode 100644 compat/fsmonitor/fsm-settings-darwin.c
 create mode 100644 compat/fsmonitor/fsm-settings-win32.c
 create mode 100755 t/lib-unicode-nfc-nfd.sh
 create mode 100755 t/perf/p7527-builtin-fsmonitor.sh


base-commit: 1a9241e1fee9d418e436849853f031329e792192
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1143

Range-diff vs v1:

  1:  23f38338cec !  1:  34619e0652b fsm-listen-win32: handle shortnames
     @@ compat/fsmonitor/fsm-listen-win32.c: normalize:
      + * to longname conversion on every notification event.
      + *
      + * We do not want to create a file to test this, so we assume that the
     -+ * root directory contains a ".git" file or directory.  (Out caller
     ++ * root directory contains a ".git" file or directory.  (Our caller
      + * only calls us for the worktree root, so this should be fine.)
      + *
      + * Remember the spelling of the shortname for ".git" if it exists.
     @@ compat/fsmonitor/fsm-listen-win32.c: normalize:
      +		 * lookup the longname for it.  Likewise, for moves
      +		 * and renames where we are given the old name.)
      +		 *
     -+		 * NEEDSWORK: Since deleting or moving a file or
     -+		 * directory by shortname is rather obscure, I'm going
     -+		 * ignore the failure and ask the caller to report the
     -+		 * original relative path.  This seemds kinder than
     -+		 * failing here and forcing a resync.
     ++		 * Since deleting or moving a file or directory by its
     ++		 * shortname is rather obscure, I'm going ignore the
     ++		 * failure and ask the caller to report the original
     ++		 * relative path.  This seems kinder than failing here
     ++		 * and forcing a resync.  Besides, forcing a resync on
     ++		 * every file/directory delete would effectively
     ++		 * cripple monitoring.
     ++		 *
     ++		 * We might revisit this in the future.
      +		 */
      +		return GRR_NO_CONVERSION_NEEDED;
      +	}
  3:  9af952e4d17 !  2:  3a0f30b849a t7527: test builtin FSMonitor watching repos with unicode paths
     @@ Metadata
      Author: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## Commit message ##
     -    t7527: test builtin FSMonitor watching repos with unicode paths
     +    t7527: test FSMonitor on repos with Unicode root paths
      
     -    Create some test repos with UTF8 pathnames and verify that
     -    the builtin FSMonitor can watch them.  This test is mainly
     -    for Windows where we need to avoid `*A()` routines.
     +    Create some test repos with UTF8 characters in the pathname of the
     +    root directory and verify that the builtin FSMonitor can watch them.
     +
     +    This test is mainly for Windows where we need to avoid `*A()`
     +    routines.
      
          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
      
     @@ t/t7527-builtin-fsmonitor.sh: do
       done
       
      +# Test Unicode UTF-8 characters in the pathname of the working
     -+# directory.  Use of "*A()" routines rather than "*W()" routines
     ++# directory root.  Use of "*A()" routines rather than "*W()" routines
      +# on Windows can sometimes lead to odd failures.
      +#
      +u1=$(printf "u_c3_a6__\xC3\xA6")
     @@ t/t7527-builtin-fsmonitor.sh: do
      +u_values="$u1 $u2"
      +for u in $u_values
      +do
     -+	test_expect_success "Unicode path: $u" '
     ++	test_expect_success "Unicode in repo root path: $u" '
      +		test_when_finished "stop_daemon_delete_repo $u" &&
      +
      +		git init "$u" &&
  4:  9efdbe28223 !  3:  87d1c0b6f2a t/helper/fsmonitor-client: create stress test
     @@ Commit message
          Create a client-side thread pool of n threads and have
          each of them make m requests as fast as they can.
      
     -    NEEDSWORK: This is just the client-side thread pool and
     -    is useful for interactive testing and experimentation.
     -    We need to add a script test to drive this.
     +    We do not currently inspect the contents of the response.
     +    We're only interested in placing a heavy request load on
     +    the daemon.
     +
     +    This test is useful for interactive testing and various
     +    experimentation.  For example, to place additional load
     +    on the daemon while another test is running.  We currently
     +    do not have a test script that actually uses this helper.
     +    We might add such a test in the future.
      
          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
      
     @@ t/helper/test-fsmonitor-client.c: static int do_send_flush(void)
      +	free(data);
      +
      +	/*
     -+	 * TODO Decide if/when to return an error or call die().
     ++	 * Return an error if any of the _send_query requests failed.
     ++	 * We don't care about thread create/join errors.
      +	 */
     -+	return 0;
     ++	return sum_errors > 0;
      +}
      +
       int cmd__fsmonitor_client(int argc, const char **argv)
  5:  44cc61e186c !  4:  8c4f90ae4fd fsmonitor-settings: bare repos are incompatible with FSMonitor
     @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar
      +	fsm_settings__set_ipc(the_repository);
      +
      +	if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {
     -+		struct strbuf buf_reason = STRBUF_INIT;
     -+		fsm_settings__get_reason(the_repository, &buf_reason);
     -+		error("%s '%s'", buf_reason.buf, xgetcwd());
     -+		strbuf_release(&buf_reason);
     -+		return -1;
     ++		const char *msg = fsm_settings__get_reason_msg(the_repository);
     ++
     ++		return error("%s '%s'", msg ? msg : "???", xgetcwd());
      +	}
      +
       	if (!strcmp(subcmd, "start"))
     @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const
       		enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
      +
      +		if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
     -+			struct strbuf buf_reason = STRBUF_INIT;
     -+			fsm_settings__get_reason(r, &buf_reason);
     -+			error("%s", buf_reason.buf);
     -+			strbuf_release(&buf_reason);
     -+			return -1;
     ++			const char *msg = fsm_settings__get_reason_msg(r);
     ++
     ++			return error("%s '%s'", msg ? msg : "???", xgetcwd());
      +		}
      +
       		if (fsm_mode == FSMONITOR_MODE_DISABLED) {
     @@ fsmonitor-settings.c
       {
       	struct fsmonitor_settings *s;
      @@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)
     - {
     + 
       	lookup_fsmonitor_settings(r);
       
      +	if (check_for_incompatible(r))
     @@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)
       	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
       }
      @@ fsmonitor-settings.c: void fsm_settings__set_hook(struct repository *r, const char *path)
     - {
     + 
       	lookup_fsmonitor_settings(r);
       
      +	if (check_for_incompatible(r))
     @@ fsmonitor-settings.c: void fsm_settings__set_disabled(struct repository *r)
       	lookup_fsmonitor_settings(r);
       
       	r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;
     -+	r->settings.fsmonitor->reason = FSMONITOR_REASON_ZERO;
     ++	r->settings.fsmonitor->reason = FSMONITOR_REASON_OK;
       	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
       }
      +
     -+static void create_reason_message(struct repository *r,
     -+				  struct strbuf *buf_reason)
     ++enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)
      +{
     -+	struct fsmonitor_settings *s = r->settings.fsmonitor;
     -+
     -+	switch (s->reason) {
     -+	case FSMONITOR_REASON_ZERO:
     -+		return;
     ++	if (!r)
     ++		r = the_repository;
      +
     -+	case FSMONITOR_REASON_BARE:
     -+		strbuf_addstr(buf_reason,
     -+			      _("bare repos are incompatible with fsmonitor"));
     -+		return;
     ++	lookup_fsmonitor_settings(r);
      +
     -+	default:
     -+		BUG("Unhandled case in create_reason_message '%d'", s->reason);
     -+	}
     ++	return r->settings.fsmonitor->reason;
      +}
      +
     -+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
     -+					       struct strbuf *buf_reason)
     ++const char *fsm_settings__get_reason_msg(struct repository *r)
      +{
     -+	lookup_fsmonitor_settings(r);
     ++	enum fsmonitor_reason reason = fsm_settings__get_reason(r);
      +
     -+	strbuf_reset(buf_reason);
     -+	if (r->settings.fsmonitor->mode == FSMONITOR_MODE_INCOMPATIBLE)
     -+		create_reason_message(r, buf_reason);
     ++	switch (reason) {
     ++	case FSMONITOR_REASON_OK:
     ++		return NULL;
      +
     -+	return r->settings.fsmonitor->reason;
     ++	case FSMONITOR_REASON_BARE:
     ++		return _("bare repos are incompatible with fsmonitor");
     ++	}
     ++
     ++	BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
     ++	    reason);
      +}
      
       ## fsmonitor-settings.h ##
     @@ fsmonitor-settings.h
      + * Incompatibility reasons.
      + */
      +enum fsmonitor_reason {
     -+	FSMONITOR_REASON_ZERO = 0,
     -+	FSMONITOR_REASON_BARE = 1,
     ++	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
     ++	FSMONITOR_REASON_BARE,
      +};
      +
       void fsm_settings__set_ipc(struct repository *r);
       void fsm_settings__set_hook(struct repository *r, const char *path);
       void fsm_settings__set_disabled(struct repository *r);
     - 
     +@@ fsmonitor-settings.h: void fsm_settings__set_disabled(struct repository *r);
       enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);
       const char *fsm_settings__get_hook_path(struct repository *r);
     -+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
     -+					       struct strbuf *buf_reason);
       
     ++enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);
     ++const char *fsm_settings__get_reason_msg(struct repository *r);
     ++
       struct fsmonitor_settings;
       
     + #endif /* FSMONITOR_SETTINGS_H */
      
       ## t/t7519-status-fsmonitor.sh ##
      @@ t/t7519-status-fsmonitor.sh: test_lazy_prereq UNTRACKED_CACHE '
     @@ t/t7519-status-fsmonitor.sh: test_lazy_prereq UNTRACKED_CACHE '
      +test_expect_success 'incompatible bare repo' '
      +	test_when_finished "rm -rf ./bare-clone actual expect" &&
      +	git init --bare bare-clone &&
     -+	cat >expect <<-\EOF &&
     -+	error: bare repos are incompatible with fsmonitor
     -+	EOF
      +
      +	test_must_fail \
      +		git -C ./bare-clone -c core.fsmonitor=foo \
      +			update-index --fsmonitor 2>actual &&
     -+	test_cmp expect actual &&
     ++	grep "bare repos are incompatible with fsmonitor" actual &&
      +
      +	test_must_fail \
      +		git -C ./bare-clone -c core.fsmonitor=true \
      +			update-index --fsmonitor 2>actual &&
     -+	test_cmp expect actual
     ++	grep "bare repos are incompatible with fsmonitor" actual
      +'
      +
      +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
  6:  4715677f85f !  5:  6329328d185 fsmonitor-settings: stub in platform-specific incompatibility checking
     @@ Commit message
      
          In the existing fsmonitor-settings code we have a way to mark
          types of repos as incompatible with fsmonitor (whether via the
     -    hook and ipc APIs).  For example, we do this for bare repos,
     +    hook and IPC APIs).  For example, we do this for bare repos,
          since there are no files to watch.
      
     -    Extend this exclusion mechanism for platfor-specific reasons.
     +    Extend this exclusion mechanism for platform-specific reasons.
          This commit just creates the framework and adds a stub for Win32.
      
          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
     @@ Makefile: all::
       # `compat/fsmonitor/fsm-listen-<name>.c` that implements the
       # `fsm_listen__*()` routines.
       #
     -+# If your platform has os-specific ways to tell if a repo is incompatible with
     -+# fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS
     ++# If your platform has OS-specific ways to tell if a repo is incompatible with
     ++# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS
      +# to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c`
      +# that implements the `fsm_os_settings__*()` routines.
      +#
     @@ compat/fsmonitor/fsm-settings-win32.c (new)
      +
      +enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
      +{
     -+	return FSMONITOR_REASON_ZERO;
     ++	return FSMONITOR_REASON_OK;
      +}
      
       ## config.mak.uname ##
     @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)
      +		enum fsmonitor_reason reason;
      +
      +		reason = fsm_os__incompatible(r);
     -+		if (reason != FSMONITOR_REASON_ZERO) {
     ++		if (reason != FSMONITOR_REASON_OK) {
      +			set_incompatible(r, reason);
      +			return 1;
      +		}
     @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)
       
      
       ## fsmonitor-settings.h ##
     -@@ fsmonitor-settings.h: enum fsmonitor_reason fsm_settings__get_reason(struct repository *r,
     +@@ fsmonitor-settings.h: const char *fsm_settings__get_reason_msg(struct repository *r);
       
       struct fsmonitor_settings;
       
     @@ fsmonitor-settings.h: enum fsmonitor_reason fsm_settings__get_reason(struct repo
      + * Ask platform-specific code whether the repository is incompatible
      + * with fsmonitor (both hook and ipc modes).  For example, if the working
      + * directory is on a remote volume and mounted via a technology that does
     -+ * not support notification events.
     ++ * not support notification events, then we should not pretend to watch it.
      + *
      + * fsm_os__* routines should considered private to fsm_settings__
      + * routines.
  7:  4e856d60e38 !  6:  fa9e86e7de7 fsmonitor-settings: virtual repos are incompatible with FSMonitor
     @@ Metadata
      Author: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## Commit message ##
     -    fsmonitor-settings: virtual repos are incompatible with FSMonitor
     +    fsmonitor-settings: VFS for Git virtual repos are incompatible
      
     -    Virtual repos, such as GVFS (aka VFS for Git), are incompatible
     -    with FSMonitor.
     +    VFS for Git virtual repositories are incompatible with FSMonitor.
     +
     +    VFS for Git is a downstream fork of Git.  It contains its own custom
     +    file system watcher that is aware of the virtualization.  If a working
     +    directory is being managed by VFS for Git, we should not try to watch
     +    it because we may get incomplete results.
     +
     +    We do not know anything about how VFS for Git works, but we do
     +    know that VFS for Git working directories contain a well-defined
     +    config setting.  If it is set, mark the working directory as
     +    incompatible.
      
          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
      
     @@ compat/fsmonitor/fsm-settings-win32.c
       #include "fsmonitor-settings.h"
       
      +/*
     -+ * GVFS (aka VFS for Git) is incompatible with FSMonitor.
     ++ * VFS for Git is incompatible with FSMonitor.
      + *
     -+ * Granted, core Git does not know anything about GVFS and we
     ++ * Granted, core Git does not know anything about VFS for Git and we
      + * shouldn't make assumptions about a downstream feature, but users
      + * can install both versions.  And this can lead to incorrect results
     -+ * from core Git commands.  So, without bringing in any of the GVFS
     -+ * code, do a simple config test for a published config setting.  (We
     -+ * do not look at the various *_TEST_* environment variables.)
     ++ * from core Git commands.  So, without bringing in any of the VFS for
     ++ * Git code, do a simple config test for a published config setting.
     ++ * (We do not look at the various *_TEST_* environment variables.)
      + */
     -+static enum fsmonitor_reason is_virtual(struct repository *r)
     ++static enum fsmonitor_reason check_vfs4git(struct repository *r)
      +{
      +	const char *const_str;
      +
      +	if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str))
     -+		return FSMONITOR_REASON_VIRTUAL;
     ++		return FSMONITOR_REASON_VFS4GIT;
      +
     -+	return FSMONITOR_REASON_ZERO;
     ++	return FSMONITOR_REASON_OK;
      +}
      +
       enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
       {
      +	enum fsmonitor_reason reason;
      +
     -+	reason = is_virtual(r);
     -+	if (reason)
     ++	reason = check_vfs4git(r);
     ++	if (reason != FSMONITOR_REASON_OK)
      +		return reason;
      +
     - 	return FSMONITOR_REASON_ZERO;
     + 	return FSMONITOR_REASON_OK;
       }
      
       ## fsmonitor-settings.c ##
     -@@ fsmonitor-settings.c: static void create_reason_message(struct repository *r,
     - 			      _("bare repos are incompatible with fsmonitor"));
     - 		return;
     +@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)
       
     -+	case FSMONITOR_REASON_VIRTUAL:
     -+		strbuf_addstr(buf_reason,
     -+			      _("virtual repos are incompatible with fsmonitor"));
     -+		return;
     + 	case FSMONITOR_REASON_BARE:
     + 		return _("bare repos are incompatible with fsmonitor");
      +
     - 	default:
     - 		BUG("Unhandled case in create_reason_message '%d'", s->reason);
     ++	case FSMONITOR_REASON_VFS4GIT:
     ++		return _("virtual repos are incompatible with fsmonitor");
       	}
     + 
     + 	BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
      
       ## fsmonitor-settings.h ##
      @@ fsmonitor-settings.h: enum fsmonitor_mode {
       enum fsmonitor_reason {
     - 	FSMONITOR_REASON_ZERO = 0,
     - 	FSMONITOR_REASON_BARE = 1,
     -+	FSMONITOR_REASON_VIRTUAL = 2,
     + 	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
     + 	FSMONITOR_REASON_BARE,
     ++	FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */
       };
       
       void fsm_settings__set_ipc(struct repository *r);
  8:  e5511ef0f8b !  7:  c1802410410 fsmonitor-settings: stub in macOS-specific incompatibility checking
     @@ compat/fsmonitor/fsm-settings-darwin.c (new)
      +
      +enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
      +{
     -+	return FSMONITOR_REASON_ZERO;
     ++	return FSMONITOR_REASON_OK;
      +}
      
       ## config.mak.uname ##
  9:  412fbc45868 !  8:  e3bfa0bd69d fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor
     @@ Metadata
      Author: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## Commit message ##
     -    fsmonitor-settings: remote repos on macOS are incompatible with FSMonitor
     +    fsmonitor-settings: remote repos on macOS are incompatible
      
          Teach Git to detect remote working directories on macOS and mark them as
          incompatible with FSMonitor.
     @@ compat/fsmonitor/fsm-settings-darwin.c
      + * So (for now at least), mark remote working directories as
      + * incompatible.
      + */
     -+static enum fsmonitor_reason is_remote(struct repository *r)
     ++static enum fsmonitor_reason check_remote(struct repository *r)
      +{
      +	struct statfs fs;
      +
     @@ compat/fsmonitor/fsm-settings-darwin.c
      +		trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s",
      +				 r->worktree, strerror(saved_errno));
      +		errno = saved_errno;
     -+		return FSMONITOR_REASON_ZERO;
     ++		return FSMONITOR_REASON_ERROR;
      +	}
      +
      +	trace_printf_key(&trace_fsmonitor,
     @@ compat/fsmonitor/fsm-settings-darwin.c
      +	if (!(fs.f_flags & MNT_LOCAL))
      +		return FSMONITOR_REASON_REMOTE;
      +
     -+	return FSMONITOR_REASON_ZERO;
     ++	return FSMONITOR_REASON_OK;
      +}
       
       enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
       {
      +	enum fsmonitor_reason reason;
      +
     -+	reason = is_remote(r);
     -+	if (reason)
     ++	reason = check_remote(r);
     ++	if (reason != FSMONITOR_REASON_OK)
      +		return reason;
      +
     - 	return FSMONITOR_REASON_ZERO;
     + 	return FSMONITOR_REASON_OK;
       }
      
       ## fsmonitor-settings.c ##
     -@@ fsmonitor-settings.c: static void create_reason_message(struct repository *r,
     - 			      _("virtual repos are incompatible with fsmonitor"));
     - 		return;
     +@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)
     + 	case FSMONITOR_REASON_BARE:
     + 		return _("bare repos are incompatible with fsmonitor");
       
     ++	case FSMONITOR_REASON_ERROR:
     ++		return _("repo incompatible with fsmonitor due to errors");
     ++
      +	case FSMONITOR_REASON_REMOTE:
     -+		strbuf_addstr(buf_reason,
     -+			      _("remote repos are incompatible with fsmonitor"));
     -+		return;
     ++		return _("remote repos are incompatible with fsmonitor");
      +
     - 	default:
     - 		BUG("Unhandled case in create_reason_message '%d'", s->reason);
     + 	case FSMONITOR_REASON_VFS4GIT:
     + 		return _("virtual repos are incompatible with fsmonitor");
       	}
      
       ## fsmonitor-settings.h ##
     -@@ fsmonitor-settings.h: enum fsmonitor_reason {
     - 	FSMONITOR_REASON_ZERO = 0,
     - 	FSMONITOR_REASON_BARE = 1,
     - 	FSMONITOR_REASON_VIRTUAL = 2,
     -+	FSMONITOR_REASON_REMOTE = 3,
     +@@ fsmonitor-settings.h: enum fsmonitor_mode {
     + enum fsmonitor_reason {
     + 	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
     + 	FSMONITOR_REASON_BARE,
     ++	FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */
     ++	FSMONITOR_REASON_REMOTE,
     + 	FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */
       };
       
     - void fsm_settings__set_ipc(struct repository *r);
 10:  ae09fb10c3a !  9:  e32da3118fb fsmonitor-settings: remote repos on Windows are incompatible with FSMonitor
     @@ Metadata
      Author: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## Commit message ##
     -    fsmonitor-settings: remote repos on Windows are incompatible with FSMonitor
     +    fsmonitor-settings: remote repos on Windows are incompatible
      
          Teach Git to detect remote working directories on Windows and mark them as
          incompatible with FSMonitor.
     @@ compat/fsmonitor/fsm-settings-win32.c
      +#include "fsmonitor.h"
       
       /*
     -  * GVFS (aka VFS for Git) is incompatible with FSMonitor.
     -@@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(struct repository *r)
     - 	return FSMONITOR_REASON_ZERO;
     +  * VFS for Git is incompatible with FSMonitor.
     +@@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason check_vfs4git(struct repository *r)
     + 	return FSMONITOR_REASON_OK;
       }
       
      +/*
     @@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(s
      + *     $ mklink /d ./link //server/share/repo
      + *     $ git -C ./link status
      + */
     -+static enum fsmonitor_reason is_remote(struct repository *r)
     ++static enum fsmonitor_reason check_remote(struct repository *r)
      +{
      +	wchar_t wpath[MAX_PATH];
      +	wchar_t wfullpath[MAX_PATH];
     @@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(s
      +	 * a multi-byte sequence.  See win32_has_dos_drive_prefix().
      +	 */
      +	if (xutftowcs_path(wpath, r->worktree) < 0)
     -+		return FSMONITOR_REASON_ZERO;
     ++		return FSMONITOR_REASON_ERROR;
      +
      +	/*
      +	 * GetDriveTypeW() requires a final slash.  We assume that the
     @@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(s
      +	 * correctly handle some UNC "\\server\share\..." paths.
      +	 */
      +	if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))
     -+		return FSMONITOR_REASON_ZERO;
     ++		return FSMONITOR_REASON_ERROR;
      +
      +	driveType = GetDriveTypeW(wfullpath);
      +	trace_printf_key(&trace_fsmonitor,
     @@ compat/fsmonitor/fsm-settings-win32.c: static enum fsmonitor_reason is_virtual(s
      +
      +	if (driveType == DRIVE_REMOTE) {
      +		trace_printf_key(&trace_fsmonitor,
     -+				 "is_remote('%s') true",
     ++				 "check_remote('%s') true",
      +				 r->worktree);
      +		return FSMONITOR_REASON_REMOTE;
      +	}
      +
     -+	return FSMONITOR_REASON_ZERO;
     ++	return FSMONITOR_REASON_OK;
      +}
      +
       enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
       {
       	enum fsmonitor_reason reason;
      @@ compat/fsmonitor/fsm-settings-win32.c: enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
     - 	if (reason)
     + 	if (reason != FSMONITOR_REASON_OK)
       		return reason;
       
     -+	reason = is_remote(r);
     -+	if (reason)
     ++	reason = check_remote(r);
     ++	if (reason != FSMONITOR_REASON_OK)
      +		return reason;
      +
     - 	return FSMONITOR_REASON_ZERO;
     + 	return FSMONITOR_REASON_OK;
       }
 11:  be1672e32b2 = 10:  f63de4eda31 unpack-trees: initialize fsmonitor_has_run_once in o->result
 12:  71babe7243a = 11:  fe305f5f287 fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
 14:  95cdff22be0 ! 12:  c8f3e251b1f fsmonitor--daemon: cd out of worktree root
     @@ builtin/fsmonitor--daemon.c: done:
       	strbuf_release(&state.path_cookie_prefix);
      +	strbuf_release(&state.path_ipc);
       
     - 	/*
     - 	 * NEEDSWORK: Consider "rm -rf <gitdir>/<fsmonitor-dir>"
     + 	return err;
     + }
      
       ## compat/fsmonitor/fsm-listen-win32.c ##
      @@ compat/fsmonitor/fsm-listen-win32.c: static int recv_rdcw_watch(struct one_watch *watch)
 15:  6b5642f7770 = 13:  71673be2da5 fsmonitor--daemon: prepare for adding health thread
 16:  63c502c20bd = 14:  5387baaf5d7 fsmonitor--daemon: rename listener thread related variables
 17:  4a77f5b1fde ! 15:  f78e4ad87c0 fsmonitor--daemon: stub in health thread
     @@ Commit message
          file system events outside of the watched worktree root or if
          we want to have an idle-timeout auto-shutdown feature.
      
     +    This commit creates the health thread itself, defines the thread-proc
     +    and sets up the thread's event loop.  It integrates this new thread
     +    into the existing IPC and Listener thread models.
     +
     +    This commit defines the API to the platform-specific code where all of
     +    the monitoring will actually happen.
     +
     +    The platform-specific code for MacOS is just stubs.  Meaning that the
     +    health thread will immediately exit on MacOS, but that is OK and
     +    expected.  Future work can define MacOS-specific monitoring.
     +
     +    The platform-specific code for Windows sets up enough of the
     +    WaitForMultipleObjects() machinery to watch for system and/or custom
     +    events.  Currently, the set of wait handles only includes our custom
     +    shutdown event (sent from our other theads).  Later commits in this
     +    series will extend the set of wait handles to monitor other
     +    conditions.
     +
          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## Makefile ##
     @@ Makefile: all::
      +# `compat/fsmonitor/fsm-health-<name>.c` files
      +# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.
       #
     - # If your platform has os-specific ways to tell if a repo is incompatible with
     - # fsmonitor (whether the hook or ipc daemon version), set FSMONITOR_OS_SETTINGS
     + # If your platform has OS-specific ways to tell if a repo is incompatible with
     + # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS
      @@ Makefile: endif
       ifdef FSMONITOR_DAEMON_BACKEND
       	COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
 18:  a398cdb8a04 ! 16:  bb72f911a05 fsm-health-win32: add framework to monitor daemon health
     @@ Metadata
      Author: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## Commit message ##
     -    fsm-health-win32: add framework to monitor daemon health
     +    fsm-health-win32: add polling framework to monitor daemon health
      
     -    Create framework in Win32 version of the "health" thread to
     -    periodically inspect the system and shutdown if warranted.
     +    Extend the Windows version of the "health" thread to periodically
     +    inspect the system and shutdown if warranted.
      
     -    This version just includes the setup for the timeout in
     -    WaitForMultipleObjects() and calls the (currently empty) table
     -    of functions.
     +    This commit updates the thread's wait loop to use a timeout and
     +    defines a (currently empty) table of functions to poll the system.
      
          A later commit will add functions to the table to actually
          inspect the system.
     @@ compat/fsmonitor/fsm-health-win32.c
      + */
      +#define WAIT_FREQ_MS (60 * 1000)
      +
     -+enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };
     ++/*
     ++ * State machine states for each of the interval functions
     ++ * used for polling our health.
     ++ */
     ++enum interval_fn_ctx {
     ++	CTX_INIT = 0,
     ++	CTX_TERM,
     ++	CTX_TIMER
     ++};
      +
      +typedef int (interval_fn)(struct fsmonitor_daemon_state *state,
      +			  enum interval_fn_ctx ctx);
      +
     + struct fsm_health_data
     + {
     + 	HANDLE hEventShutdown;
     +@@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daemon_state *state)
     + 	FREE_AND_NULL(state->health_data);
     + }
     + 
     ++/*
     ++ * A table of the polling functions.
     ++ */
      +static interval_fn *table[] = {
      +	NULL, /* must be last */
      +};
      +
      +/*
     -+ * Call all of the functions in the table.
     ++ * Call all of the polling functions in the table.
      + * Shortcut and return first error.
      + *
      + * Return 0 if all succeeded.
     @@ compat/fsmonitor/fsm-health-win32.c
      +	return 0;
      +}
      +
     - struct fsm_health_data
     - {
     - 	HANDLE hEventShutdown;
     -@@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daemon_state *state)
       void fsm_health__loop(struct fsmonitor_daemon_state *state)
       {
       	struct fsm_health_data *data = state->health_data;
 19:  023fcd6e2b1 ! 17:  baf8c031a97 fsm-health-win32: force shutdown daemon if worktree root moves
     @@ Commit message
          Force shutdown fsmonitor daemon if the worktree root directory
          is moved, renamed, or deleted.
      
     +    Use Windows low-level GetFileInformationByHandle() to get and
     +    compare the Windows system unique ID for the directory with a
     +    cached version when we started up.  This lets us detect the
     +    case where someone renames the directory that we are watching
     +    and then creates a new directory with the original pathname.
     +
     +    This is important because we are listening to a named pipe for
     +    requests and they are stored in the Named Pipe File System (NPFS)
     +    which a kernel-resident pseudo filesystem not associated with
     +    the actual NTFS directory.
     +
     +    For example, if the daemon was watching "~/foo/", it would have
     +    a directory-watch handle on that directory and a named-pipe
     +    handle for "//./pipe/...foo".  Moving the directory to "~/bar/"
     +    does not invalidate the directory handle.  (So the daemon would
     +    actually be watching "~/bar" but listening on "//./pipe/...foo".
     +    If the user then does "git init ~/foo" and causes another daemon
     +    to start, the first daemon will still have ownership of the pipe
     +    and the second daemon instance will fail to start.  "git status"
     +    clients in "~/foo" will ask "//./pipe/...foo" about changes and
     +    the first daemon instance will tell them about "~/bar".
     +
     +    This commit causes the first daemon to shutdown if the system unique
     +    ID for "~/foo" changes (changes from what it was when the daemon
     +    started).  Shutdown occurs after a periodic poll.  After the
     +    first daemon exits and releases the lock on the named pipe,
     +    subsequent Git commands may cause another daemon to be started
     +    on "~/foo".  Similarly, a subsequent Git command may cause another
     +    daemon to be started on "~/bar".
     +
          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## compat/fsmonitor/fsm-health-win32.c ##
     -@@ compat/fsmonitor/fsm-health-win32.c: enum interval_fn_ctx { CTX_INIT = 0, CTX_TERM, CTX_TIMER };
     - typedef int (interval_fn)(struct fsmonitor_daemon_state *state,
     - 			  enum interval_fn_ctx ctx);
     - 
     -+static interval_fn has_worktree_moved;
     -+
     - static interval_fn *table[] = {
     -+	has_worktree_moved,
     - 	NULL, /* must be last */
     - };
     - 
      @@ compat/fsmonitor/fsm-health-win32.c: struct fsm_health_data
       	HANDLE hHandles[1]; /* the array does not own these handles */
       #define HEALTH_SHUTDOWN 0
     @@ compat/fsmonitor/fsm-health-win32.c: struct fsm_health_data
      +	} wt_moved;
       };
       
     - int fsm_health__ctor(struct fsmonitor_daemon_state *state)
     -@@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daemon_state *state)
     - 	FREE_AND_NULL(state->health_data);
     - }
     - 
     ++/*
     ++ * Lookup the system unique ID for the path.  This is as close as
     ++ * we get to an inode number, but this also contains volume info,
     ++ * so it is a little stronger.
     ++ */
      +static int lookup_bhfi(wchar_t *wpath,
      +		       BY_HANDLE_FILE_INFORMATION *bhfi)
      +{
     @@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daem
      +	return 0;
      +}
      +
     ++/*
     ++ * Compare the relevant fields from two system unique IDs.
     ++ * We use this to see if two different handles to the same
     ++ * path actually refer to the same *instance* of the file
     ++ * or directory.
     ++ */
      +static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,
      +		   const BY_HANDLE_FILE_INFORMATION *bhfi_2)
      +{
     @@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daem
      +	return 0;
      +}
      +
     - void fsm_health__loop(struct fsmonitor_daemon_state *state)
     ++
     + int fsm_health__ctor(struct fsmonitor_daemon_state *state)
       {
     - 	struct fsm_health_data *data = state->health_data;
     + 	struct fsm_health_data *data;
     +@@ compat/fsmonitor/fsm-health-win32.c: void fsm_health__dtor(struct fsmonitor_daemon_state *state)
     +  * A table of the polling functions.
     +  */
     + static interval_fn *table[] = {
     ++	has_worktree_moved,
     + 	NULL, /* must be last */
     + };
     + 
 20:  496b3da35d0 ! 18:  796b6591393 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
     @@ compat/fsmonitor/fsm-listen-darwin.c: static void fsevent_callback(ConstFSEventS
      +			/*
      +			 * The spelling of the pathname of the root directory
      +			 * has changed.  This includes the name of the root
     -+			 * directory itself of of any parent directory in the
     ++			 * directory itself or of any parent directory in the
      +			 * path.
      +			 *
      +			 * (There may be other conditions that throw this,
 21:  07a9c7542be = 19:  24591920878 fsmonitor: optimize processing of directory events
 22:  f065e8c9a90 = 20:  06a32413854 t7527: FSMonitor tests for directory moves
 23:  e5a12832afa = 21:  4b59013cadd t/perf/p7527: add perf test for builtin FSMonitor
  -:  ----------- > 22:  524d449ed64 fsmonitor: never set CE_FSMONITOR_VALID on submodules
  2:  ad8cf6d9a47 ! 23:  c7264decaf6 t7527: test FS event reporing on macOS WRT case and Unicode
     @@ Metadata
      Author: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## Commit message ##
     -    t7527: test FS event reporing on macOS WRT case and Unicode
     +    t7527: test FSMonitor on case insensitive+preserving file system
      
     -    Confirm that macOS FS events are reported with a normalized spelling.
     -
     -    APFS (and/or HFS+) is case-insensitive.  This means that case-independent
     -    lookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed.  But that
     -    doesn't tell us how FS events are reported if we try "rm -rf .git" versus
     -    "rm -rf .GIT".  Are the events reported using the on-disk spelling of the
     -    pathname or in the spelling used by the command.
     -
     -    NEEDSWORK: I was only able to test case.  It would be nice to add tests
     -    that use different Unicode spellings/normalizations and understand the
     -    differences between APFS and HFS+ in this area.  We should confirm that
     -    the spelling of the workdir paths that the daemon sends to clients are
     -    always properly normalized.
     +    Test that FS events from the OS are received using the preserved,
     +    on-disk spelling of files/directories rather than spelling used
     +    to make the change.
      
          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## t/t7527-builtin-fsmonitor.sh ##
     -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '
     - 	test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status
     +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success "Submodule" '
     + 	my_match_and_clean
       '
       
     -+# Confirm that MacOS hides all of the Unicode normalization and/or
     -+# case folding from the FS events.  That is, are the pathnames in the
     -+# FS events reported using the spelling on the disk or in the spelling
     -+# used by the other process.
     -+#
     -+# Note that we assume that the filesystem is set to case insensitive.
     -+#
     -+# NEEDSWORK: APFS handles Unicode and Unicode normalization
     -+# differently than HFS+.  I only have an APFS partition, so
     -+# more testing here would be helpful.
     ++# On a case-insensitive file system, confirm that the daemon
     ++# notices when the .git directory is moved/renamed/deleted
     ++# regardless of how it is spelled in the the FS event.
     ++# That is, does the FS event receive the spelling of the
     ++# operation or does it receive the spelling preserved with
     ++# the file/directory.
      +#
     ++test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '
     ++#	test_when_finished "stop_daemon_delete_repo test_insensitive" &&
     ++
     ++	git init test_insensitive &&
     ++	(
     ++		GIT_TRACE_FSMONITOR="$(pwd)/insensitive.trace" &&
     ++		export GIT_TRACE_FSMONITOR &&
      +
     -+# Rename .git using alternate spelling and confirm that the daemon
     -+# sees the event using the correct spelling and shutdown.
     -+test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' '
     -+	test_when_finished "stop_daemon_delete_repo test_apfs" &&
     ++		start_daemon test_insensitive
     ++	) &&
      +
     -+	git init test_apfs &&
     -+	start_daemon test_apfs &&
     ++	mkdir -p test_insensitive/abc/def &&
     ++	echo xyz >test_insensitive/ABC/DEF/xyz &&
      +
     -+	test_path_is_dir test_apfs/.git &&
     -+	test_path_is_dir test_apfs/.GIT &&
     ++	test_path_is_dir test_insensitive/.git &&
     ++	test_path_is_dir test_insensitive/.GIT &&
      +
     -+	mv test_apfs/.GIT test_apfs/.FOO &&
     ++	# Rename .git using an alternate spelling to verify that that
     ++	# daemon detects it and automatically shuts down.
     ++	mv test_insensitive/.GIT test_insensitive/.FOO &&
      +	sleep 1 &&
     -+	mv test_apfs/.FOO test_apfs/.git &&
     ++	mv test_insensitive/.FOO test_insensitive/.git &&
     ++	test_must_fail git -C test_insensitive fsmonitor--daemon status &&
      +
     -+	test_must_fail git -C test_apfs fsmonitor--daemon status
     ++	# Verify that events were reported using on-disk spellings of the
     ++	# directories and files that we touched.  We may or may not get a
     ++	# trailing slash on modified directories.
     ++	#
     ++	egrep "^event: abc/?$"       ./insensitive.trace &&
     ++	egrep "^event: abc/def/?$"   ./insensitive.trace &&
     ++	egrep "^event: abc/def/xyz$" ./insensitive.trace
      +'
      +
     - test_expect_success 'cannot start multiple daemons' '
     - 	test_when_finished "stop_daemon_delete_repo test_multiple" &&
     - 
     + test_done
 13:  f19671f7def <  -:  ----------- fsmonitor--daemon: print start message only if fsmonitor.announceStartup
  -:  ----------- > 24:  95b9d4210d2 fsmonitor: on macOS also emit NFC spelling for NFD pathname
  -:  ----------- > 25:  5a0c1b7a287 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  -:  ----------- > 26:  a45c1fd3000 t7527: test Unicode NFC/NFD handling on MacOS
  -:  ----------- > 27:  e3e01677d93 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible

-- 
gitgitgadget

^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 01/27] fsm-listen-win32: handle shortnames
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 02/27] t7527: test FSMonitor on repos with Unicode root paths Jeff Hostetler via GitGitGadget
                     ` (29 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach FSMonitor daemon on Windows to recognize shortname paths as
aliases of normal longname paths.  FSMonitor clients, such as `git
status`, should receive the longname spelling of changed files (when
possible).

Sometimes we receive FS events using the shortname, such as when a CMD
shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1".  The FS notification
arrives using whatever combination of long and shortnames were used by
the other process.  (Shortnames do seem to be case normalized,
however.)

Use Windows GetLongPathNameW() to try to map the pathname spelling in
the notification event into the normalized longname spelling.  (This
can fail if the file/directory is deleted, moved, or renamed, because
we are asking the FS for the mapping in response to the event and
after it has already happened, but we try.)

Special case the shortname spelling of ".git" to avoid under-reporting
these events.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++-----
 t/t7527-builtin-fsmonitor.sh        |  65 +++++
 2 files changed, 374 insertions(+), 54 deletions(-)

diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index c2d11acbc1e..f4673d7d8b1 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -25,6 +25,9 @@ struct one_watch
 	DWORD count;
 
 	struct strbuf path;
+	wchar_t wpath_longname[MAX_PATH + 1];
+	DWORD wpath_longname_len;
+
 	HANDLE hDir;
 	HANDLE hEvent;
 	OVERLAPPED overlapped;
@@ -34,6 +37,21 @@ struct one_watch
 	 * need to later call GetOverlappedResult() and possibly CancelIoEx().
 	 */
 	BOOL is_active;
+
+	/*
+	 * Are shortnames enabled on the containing drive?  This is
+	 * always true for "C:/" drives and usually never true for
+	 * other drives.
+	 *
+	 * We only set this for the worktree because we only need to
+	 * convert shortname paths to longname paths for items we send
+	 * to clients.  (We don't care about shortname expansion for
+	 * paths inside a GITDIR because we never send them to
+	 * clients.)
+	 */
+	BOOL has_shortnames;
+	BOOL has_tilda;
+	wchar_t dotgit_shortname[16]; /* for 8.3 name */
 };
 
 struct fsmonitor_daemon_backend_data
@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data
 };
 
 /*
- * Convert the WCHAR path from the notification into UTF8 and
- * then normalize it.
+ * Convert the WCHAR path from the event into UTF8 and normalize it.
+ *
+ * `wpath_len` is in WCHARS not bytes.
  */
-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,
 				  struct strbuf *normalized_path)
 {
 	int reserve;
 	int len = 0;
 
 	strbuf_reset(normalized_path);
-	if (!info->FileNameLength)
+	if (!wpath_len)
 		goto normalize;
 
 	/*
@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
 	 * sequence of 2 UTF8 characters.  That should let us
 	 * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.
 	 */
-	reserve = info->FileNameLength + 1;
+	reserve = 2 * wpath_len + 1;
 	strbuf_grow(normalized_path, reserve);
 
 	for (;;) {
-		len = WideCharToMultiByte(CP_UTF8, 0, info->FileName,
-					  info->FileNameLength / sizeof(WCHAR),
+		len = WideCharToMultiByte(CP_UTF8, 0,
+					  wpath, wpath_len,
 					  normalized_path->buf,
 					  strbuf_avail(normalized_path) - 1,
 					  NULL, NULL);
@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
 			goto normalize;
 		if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {
 			error("[GLE %ld] could not convert path to UTF-8: '%.*ls'",
-			      GetLastError(),
-			      (int)(info->FileNameLength / sizeof(WCHAR)),
-			      info->FileName);
+			      GetLastError(), (int)wpath_len, wpath);
 			return -1;
 		}
 
@@ -98,6 +115,152 @@ normalize:
 	return strbuf_normalize_path(normalized_path);
 }
 
+/*
+ * See if the worktree root directory has shortnames enabled.
+ * This will help us decide if we need to do an expensive shortname
+ * to longname conversion on every notification event.
+ *
+ * We do not want to create a file to test this, so we assume that the
+ * root directory contains a ".git" file or directory.  (Our caller
+ * only calls us for the worktree root, so this should be fine.)
+ *
+ * Remember the spelling of the shortname for ".git" if it exists.
+ */
+static void check_for_shortnames(struct one_watch *watch)
+{
+	wchar_t buf_in[MAX_PATH + 1];
+	wchar_t buf_out[MAX_PATH + 1];
+	wchar_t *last_slash = NULL;
+	wchar_t *last_bslash = NULL;
+	wchar_t *last;
+
+	/* build L"<wt-root-path>/.git" */
+	wcscpy(buf_in, watch->wpath_longname);
+	wcscpy(buf_in + watch->wpath_longname_len, L".git");
+
+	if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))
+		return;
+
+	last_slash = wcsrchr(buf_out, L'/');
+	last_bslash = wcsrchr(buf_out, L'\\');
+	if (last_slash > last_bslash)
+		last = last_slash + 1;
+	else if (last_bslash)
+		last = last_bslash + 1;
+	else
+		last = buf_out;
+
+	if (!wcscmp(last, L".git"))
+		return;
+
+	watch->has_shortnames = 1;
+	wcsncpy(watch->dotgit_shortname, last,
+		ARRAY_SIZE(watch->dotgit_shortname));
+
+	/*
+	 * The shortname for ".git" is usually of the form "GIT~1", so
+	 * we should be able to avoid shortname to longname mapping on
+	 * every notification event if the source string does not
+	 * contain a "~".
+	 *
+	 * However, the documentation for GetLongPathNameW() says
+	 * that there are filesystems that don't follow that pattern
+	 * and warns against this optimization.
+	 *
+	 * Lets test this.
+	 */
+	if (wcschr(watch->dotgit_shortname, L'~'))
+		watch->has_tilda = 1;
+}
+
+enum get_relative_result {
+	GRR_NO_CONVERSION_NEEDED,
+	GRR_HAVE_CONVERSION,
+	GRR_SHUTDOWN,
+};
+
+/*
+ * Info notification paths are relative to the root of the watch.
+ * If our CWD is still at the root, then we can use relative paths
+ * to convert from shortnames to longnames.  If our process has a
+ * different CWD, then we need to construct an absolute path, do
+ * the conversion, and then return the root-relative portion.
+ *
+ * We use the longname form of the root as our basis and assume that
+ * it already has a trailing slash.
+ *
+ * `wpath_len` is in WCHARS not bytes.
+ */
+static enum get_relative_result get_relative_longname(
+	struct one_watch *watch,
+	const wchar_t *wpath, DWORD wpath_len,
+	wchar_t *wpath_longname)
+{
+	wchar_t buf_in[2 * MAX_PATH + 1];
+	wchar_t buf_out[MAX_PATH + 1];
+	DWORD root_len;
+
+	/* Build L"<wt-root-path>/<event-rel-path>" */
+	root_len = watch->wpath_longname_len;
+	wcsncpy(buf_in, watch->wpath_longname, root_len);
+	wcsncpy(buf_in + root_len, wpath, wpath_len);
+	buf_in[root_len + wpath_len] = 0;
+
+	/*
+	 * We don't actually know if the source pathname is a
+	 * shortname or a longname.  This routine allows either to be
+	 * given as input.
+	 */
+	if (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {
+		/*
+		 * The shortname to longname conversion can fail for
+		 * various reasons, for example if the file has been
+		 * deleted.  (That is, if we just received a
+		 * delete-file notification event and the file is
+		 * already gone, we can't ask the file system to
+		 * lookup the longname for it.  Likewise, for moves
+		 * and renames where we are given the old name.)
+		 *
+		 * Since deleting or moving a file or directory by its
+		 * shortname is rather obscure, I'm going ignore the
+		 * failure and ask the caller to report the original
+		 * relative path.  This seems kinder than failing here
+		 * and forcing a resync.  Besides, forcing a resync on
+		 * every file/directory delete would effectively
+		 * cripple monitoring.
+		 *
+		 * We might revisit this in the future.
+		 */
+		return GRR_NO_CONVERSION_NEEDED;
+	}
+
+	if (!wcscmp(buf_in, buf_out)) {
+		/*
+		 * The path does not have a shortname alias.
+		 */
+		return GRR_NO_CONVERSION_NEEDED;
+	}
+
+	if (wcsncmp(buf_in, buf_out, root_len)) {
+		/*
+		 * The spelling of the root directory portion of the computed
+		 * longname has changed.  This should not happen.  Basically,
+		 * it means that we don't know where (without recomputing the
+		 * longname of just the root directory) to split out the
+		 * relative path.  Since this should not happen, I'm just
+		 * going to let this fail and force a shutdown (because all
+		 * subsequent events are probably going to see the same
+		 * mismatch).
+		 */
+		return GRR_SHUTDOWN;
+	}
+
+	/* Return the worktree root-relative portion of the longname. */
+
+	wcscpy(wpath_longname, buf_out + root_len);
+	return GRR_HAVE_CONVERSION;
+}
+
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
 	SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);
@@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 	DWORD share_mode =
 		FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;
 	HANDLE hDir;
-	wchar_t wpath[MAX_PATH];
+	DWORD len_longname;
+	wchar_t wpath[MAX_PATH + 1];
+	wchar_t wpath_longname[MAX_PATH + 1];
 
 	if (xutftowcs_path(wpath, path) < 0) {
 		error(_("could not convert to wide characters: '%s'"), path);
@@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 		return NULL;
 	}
 
+	if (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {
+		error(_("[GLE %ld] could not get longname of '%s'"),
+		      GetLastError(), path);
+		CloseHandle(hDir);
+		return NULL;
+	}
+
+	len_longname = wcslen(wpath_longname);
+	if (wpath_longname[len_longname - 1] != L'/' &&
+	    wpath_longname[len_longname - 1] != L'\\') {
+		wpath_longname[len_longname++] = L'/';
+		wpath_longname[len_longname] = 0;
+	}
+
 	CALLOC_ARRAY(watch, 1);
 
 	watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */
@@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 	strbuf_init(&watch->path, 0);
 	strbuf_addstr(&watch->path, path);
 
+	wcscpy(watch->wpath_longname, wpath_longname);
+	watch->wpath_longname_len = len_longname;
+
 	watch->hDir = hDir;
 	watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
 
@@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)
 	watch->is_active = FALSE;
 }
 
+/*
+ * Process a single relative pathname event.
+ * Return 1 if we should shutdown.
+ */
+static int process_1_worktree_event(
+	struct string_list *cookie_list,
+	struct fsmonitor_batch **batch,
+	const struct strbuf *path,
+	enum fsmonitor_path_type t,
+	DWORD info_action)
+{
+	const char *slash;
+
+	switch (t) {
+	case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
+		/* special case cookie files within .git */
+
+		/* Use just the filename of the cookie file. */
+		slash = find_last_dir_sep(path->buf);
+		string_list_append(cookie_list,
+				   slash ? slash + 1 : path->buf);
+		break;
+
+	case IS_INSIDE_DOT_GIT:
+		/* ignore everything inside of "<worktree>/.git/" */
+		break;
+
+	case IS_DOT_GIT:
+		/* "<worktree>/.git" was deleted (or renamed away) */
+		if ((info_action == FILE_ACTION_REMOVED) ||
+		    (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {
+			trace2_data_string("fsmonitor", NULL,
+					   "fsm-listen/dotgit",
+					   "removed");
+			return 1;
+		}
+		break;
+
+	case IS_WORKDIR_PATH:
+		/* queue normal pathname */
+		if (!*batch)
+			*batch = fsmonitor_batch__new();
+		fsmonitor_batch__add_path(*batch, path->buf);
+		break;
+
+	case IS_GITDIR:
+	case IS_INSIDE_GITDIR:
+	case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
+	default:
+		BUG("unexpected path classification '%d' for '%s'",
+		    t, path->buf);
+	}
+
+	return 0;
+}
+
 /*
  * Process filesystem events that happen anywhere (recursively) under the
  * <worktree> root directory.  For a normal working directory, this includes
@@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
 	struct fsmonitor_batch *batch = NULL;
 	const char *p = watch->buffer;
+	wchar_t wpath_longname[MAX_PATH + 1];
 
 	/*
 	 * If the kernel gets more events than will fit in the kernel
@@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)
 	 */
 	for (;;) {
 		FILE_NOTIFY_INFORMATION *info = (void *)p;
-		const char *slash;
+		wchar_t *wpath = info->FileName;
+		DWORD wpath_len = info->FileNameLength / sizeof(WCHAR);
 		enum fsmonitor_path_type t;
+		enum get_relative_result grr;
+
+		if (watch->has_shortnames) {
+			if (!wcscmp(wpath, watch->dotgit_shortname)) {
+				/*
+				 * This event exactly matches the
+				 * spelling of the shortname of
+				 * ".git", so we can skip some steps.
+				 *
+				 * (This case is odd because the user
+				 * can "rm -rf GIT~1" and we cannot
+				 * use the filesystem to map it back
+				 * to ".git".)
+				 */
+				strbuf_reset(&path);
+				strbuf_addstr(&path, ".git");
+				t = IS_DOT_GIT;
+				goto process_it;
+			}
 
-		strbuf_reset(&path);
-		if (normalize_path_in_utf8(info, &path) == -1)
-			goto skip_this_path;
-
-		t = fsmonitor_classify_path_workdir_relative(path.buf);
-
-		switch (t) {
-		case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
-			/* special case cookie files within .git */
-
-			/* Use just the filename of the cookie file. */
-			slash = find_last_dir_sep(path.buf);
-			string_list_append(&cookie_list,
-					   slash ? slash + 1 : path.buf);
-			break;
-
-		case IS_INSIDE_DOT_GIT:
-			/* ignore everything inside of "<worktree>/.git/" */
-			break;
+			if (watch->has_tilda && !wcschr(wpath, L'~')) {
+				/*
+				 * Shortnames on this filesystem have tildas
+				 * and the notification path does not have
+				 * one, so we assume that it is a longname.
+				 */
+				goto normalize_it;
+			}
 
-		case IS_DOT_GIT:
-			/* "<worktree>/.git" was deleted (or renamed away) */
-			if ((info->Action == FILE_ACTION_REMOVED) ||
-			    (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {
-				trace2_data_string("fsmonitor", NULL,
-						   "fsm-listen/dotgit",
-						   "removed");
+			grr = get_relative_longname(watch, wpath, wpath_len,
+						    wpath_longname);
+			switch (grr) {
+			case GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */
+				break;
+			case GRR_HAVE_CONVERSION:
+				wpath = wpath_longname;
+				wpath_len = wcslen(wpath);
+				break;
+			default:
+			case GRR_SHUTDOWN:
 				goto force_shutdown;
 			}
-			break;
+		}
 
-		case IS_WORKDIR_PATH:
-			/* queue normal pathname */
-			if (!batch)
-				batch = fsmonitor_batch__new();
-			fsmonitor_batch__add_path(batch, path.buf);
-			break;
+normalize_it:
+		if (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)
+			goto skip_this_path;
 
-		case IS_GITDIR:
-		case IS_INSIDE_GITDIR:
-		case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
-		default:
-			BUG("unexpected path classification '%d' for '%s'",
-			    t, path.buf);
-		}
+		t = fsmonitor_classify_path_workdir_relative(path.buf);
+
+process_it:
+		if (process_1_worktree_event(&cookie_list, &batch, &path, t,
+					     info->Action))
+			goto force_shutdown;
 
 skip_this_path:
 		if (!info->NextEntryOffset)
@@ -382,6 +630,9 @@ force_shutdown:
  * Note that we DO NOT get filesystem events on the external <gitdir>
  * itself (it is not inside something that we are watching).  In particular,
  * we do not get an event if the external <gitdir> is deleted.
+ *
+ * Also, we do not care about shortnames within the external <gitdir>, since
+ * we never send these paths to clients.
  */
 static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 {
@@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 		const char *slash;
 		enum fsmonitor_path_type t;
 
-		strbuf_reset(&path);
-		if (normalize_path_in_utf8(info, &path) == -1)
+		if (normalize_path_in_utf8(
+			    info->FileName,
+			    info->FileNameLength / sizeof(WCHAR),
+			    &path) == -1)
 			goto skip_this_path;
 
 		t = fsmonitor_classify_path_gitdir_relative(path.buf);
@@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 	if (!data->watch_worktree)
 		goto failed;
 
+	check_for_shortnames(data->watch_worktree);
+
 	if (state->nr_paths_watching > 1) {
 		data->watch_gitdir = create_watch(state,
 						  state->path_gitdir_watch.buf);
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 0ccbfb9616f..dbca7f835eb 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -123,6 +123,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '
 	test_must_fail git -C test_implicit_2 fsmonitor--daemon status
 '
 
+# File systems on Windows may or may not have shortnames.
+# This is a volume-specific setting on modern systems.
+# "C:/" drives are required to have them enabled.  Other
+# hard drives default to disabled.
+#
+# This is a crude test to see if shortnames are enabled
+# on the volume containing the test directory.  It is
+# crude, but it does not require elevation like `fsutil`.
+#
+test_lazy_prereq SHORTNAMES '
+	mkdir .foo &&
+	test -d "FOO~1"
+'
+
+# Here we assume that the shortname of ".git" is "GIT~1".
+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '
+	test_when_finished "stop_daemon_delete_repo test_implicit_1s" &&
+
+	git init test_implicit_1s &&
+
+	start_daemon test_implicit_1s &&
+
+	# renaming the .git directory will implicitly stop the daemon.
+	# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.
+	# the rename-from FS Event will contain the shortname.
+	#
+	mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&
+
+	sleep 1 &&
+	# put it back so that our status will not crawl out to our
+	# parent directory.
+	# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.
+	mv test_implicit_1s/.gitxyz test_implicit_1s/.git &&
+
+	test_must_fail git -C test_implicit_1s fsmonitor--daemon status
+'
+
+# Here we first create a file with LONGNAME of "GIT~1" before
+# we create the repo.  This will cause the shortname of ".git"
+# to be "GIT~2".
+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '
+	test_when_finished "stop_daemon_delete_repo test_implicit_1s2" &&
+
+	mkdir test_implicit_1s2 &&
+	echo HELLO >test_implicit_1s2/GIT~1 &&
+	git init test_implicit_1s2 &&
+
+	test_path_is_file test_implicit_1s2/GIT~1 &&
+	test_path_is_dir  test_implicit_1s2/GIT~2 &&
+
+	start_daemon test_implicit_1s2 &&
+
+	# renaming the .git directory will implicitly stop the daemon.
+	# the rename-from FS Event will contain the shortname.
+	#
+	mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&
+
+	sleep 1 &&
+	# put it back so that our status will not crawl out to our
+	# parent directory.
+	mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&
+
+	test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status
+'
+
 test_expect_success 'cannot start multiple daemons' '
 	test_when_finished "stop_daemon_delete_repo test_multiple" &&
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 02/27] t7527: test FSMonitor on repos with Unicode root paths
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 01/27] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 03/27] t/helper/fsmonitor-client: create stress test Jeff Hostetler via GitGitGadget
                     ` (28 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create some test repos with UTF8 characters in the pathname of the
root directory and verify that the builtin FSMonitor can watch them.

This test is mainly for Windows where we need to avoid `*A()`
routines.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index dbca7f835eb..c2627f28865 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -666,4 +666,27 @@ do
 	done
 done
 
+# Test Unicode UTF-8 characters in the pathname of the working
+# directory root.  Use of "*A()" routines rather than "*W()" routines
+# on Windows can sometimes lead to odd failures.
+#
+u1=$(printf "u_c3_a6__\xC3\xA6")
+u2=$(printf "u_e2_99_ab__\xE2\x99\xAB")
+u_values="$u1 $u2"
+for u in $u_values
+do
+	test_expect_success "Unicode in repo root path: $u" '
+		test_when_finished "stop_daemon_delete_repo $u" &&
+
+		git init "$u" &&
+		echo 1 >"$u"/file1 &&
+		git -C "$u" add file1 &&
+		git -C "$u" config core.fsmonitor true &&
+
+		start_daemon "$u" &&
+		git -C "$u" status >actual &&
+		grep "new file:   file1" actual
+	'
+done
+
 test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 03/27] t/helper/fsmonitor-client: create stress test
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 01/27] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 02/27] t7527: test FSMonitor on repos with Unicode root paths Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
                     ` (27 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create a stress test to hammer on the fsmonitor daemon.
Create a client-side thread pool of n threads and have
each of them make m requests as fast as they can.

We do not currently inspect the contents of the response.
We're only interested in placing a heavy request load on
the daemon.

This test is useful for interactive testing and various
experimentation.  For example, to place additional load
on the daemon while another test is running.  We currently
do not have a test script that actually uses this helper.
We might add such a test in the future.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++
 1 file changed, 106 insertions(+)

diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c
index f7a5b3a32fa..985340ba719 100644
--- a/t/helper/test-fsmonitor-client.c
+++ b/t/helper/test-fsmonitor-client.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "parse-options.h"
 #include "fsmonitor-ipc.h"
+#include "thread-utils.h"
+#include "trace2.h"
 
 #ifndef HAVE_FSMONITOR_DAEMON_BACKEND
 int cmd__fsmonitor_client(int argc, const char **argv)
@@ -79,20 +81,121 @@ static int do_send_flush(void)
 	return 0;
 }
 
+struct hammer_thread_data
+{
+	pthread_t pthread_id;
+	int thread_nr;
+
+	int nr_requests;
+	const char *token;
+
+	int sum_successful;
+	int sum_errors;
+};
+
+static void *hammer_thread_proc(void *_hammer_thread_data)
+{
+	struct hammer_thread_data *data = _hammer_thread_data;
+	struct strbuf answer = STRBUF_INIT;
+	int k;
+	int ret;
+
+	trace2_thread_start("hammer");
+
+	for (k = 0; k < data->nr_requests; k++) {
+		strbuf_reset(&answer);
+
+		ret = fsmonitor_ipc__send_query(data->token, &answer);
+		if (ret < 0)
+			data->sum_errors++;
+		else
+			data->sum_successful++;
+	}
+
+	strbuf_release(&answer);
+	trace2_thread_exit();
+	return NULL;
+}
+
+/*
+ * Start a pool of client threads that will each send a series of
+ * commands to the daemon.
+ *
+ * The goal is to overload the daemon with a sustained series of
+ * concurrent requests.
+ */
+static int do_hammer(const char *token, int nr_threads, int nr_requests)
+{
+	struct hammer_thread_data *data = NULL;
+	int k;
+	int sum_join_errors = 0;
+	int sum_commands = 0;
+	int sum_errors = 0;
+
+	if (!token || !*token)
+		token = get_token_from_index();
+	if (nr_threads < 1)
+		nr_threads = 1;
+	if (nr_requests < 1)
+		nr_requests = 1;
+
+	CALLOC_ARRAY(data, nr_threads);
+
+	for (k = 0; k < nr_threads; k++) {
+		struct hammer_thread_data *p = &data[k];
+		p->thread_nr = k;
+		p->nr_requests = nr_requests;
+		p->token = token;
+
+		if (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {
+			warning("failed to create thread[%d] skipping remainder", k);
+			nr_threads = k;
+			break;
+		}
+	}
+
+	for (k = 0; k < nr_threads; k++) {
+		struct hammer_thread_data *p = &data[k];
+
+		if (pthread_join(p->pthread_id, NULL))
+			sum_join_errors++;
+		sum_commands += p->sum_successful;
+		sum_errors += p->sum_errors;
+	}
+
+	fprintf(stderr, "HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\n",
+		nr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);
+
+	free(data);
+
+	/*
+	 * Return an error if any of the _send_query requests failed.
+	 * We don't care about thread create/join errors.
+	 */
+	return sum_errors > 0;
+}
+
 int cmd__fsmonitor_client(int argc, const char **argv)
 {
 	const char *subcmd;
 	const char *token = NULL;
+	int nr_threads = 1;
+	int nr_requests = 1;
 
 	const char * const fsmonitor_client_usage[] = {
 		N_("test-helper fsmonitor-client query [<token>]"),
 		N_("test-helper fsmonitor-client flush"),
+		N_("test-helper fsmonitor-client hammer [<token>] [<threads>] [<requests>]"),
 		NULL,
 	};
 
 	struct option options[] = {
 		OPT_STRING(0, "token", &token, N_("token"),
 			   N_("command token to send to the server")),
+
+		OPT_INTEGER(0, "threads", &nr_threads, N_("number of client threads")),
+		OPT_INTEGER(0, "requests", &nr_requests, N_("number of requests per thread")),
+
 		OPT_END()
 	};
 
@@ -116,6 +219,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)
 	if (!strcmp(subcmd, "flush"))
 		return !!do_send_flush();
 
+	if (!strcmp(subcmd, "hammer"))
+		return !!do_hammer(token, nr_threads, nr_requests);
+
 	die("Unhandled subcommand: '%s'", subcmd);
 }
 #endif
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (2 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 03/27] t/helper/fsmonitor-client: create stress test Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-11  1:31     ` Ævar Arnfjörð Bjarmason
  2022-03-08 22:15   ` [PATCH v2 05/27] fsmonitor-settings: stub in platform-specific incompatibility checking Jeff Hostetler via GitGitGadget
                     ` (26 subsequent siblings)
  30 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Bare repos do not have a worktree, so there is nothing for the
daemon watch.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c |  9 ++++++
 builtin/update-index.c      |  7 +++++
 fsmonitor-settings.c        | 57 +++++++++++++++++++++++++++++++++++++
 fsmonitor-settings.h        | 12 ++++++++
 t/t7519-status-fsmonitor.sh | 23 +++++++++++++++
 5 files changed, 108 insertions(+)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 97ca2a356e5..00eaffbb945 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1443,6 +1443,15 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
 		die(_("invalid 'ipc-threads' value (%d)"),
 		    fsmonitor__ipc_threads);
 
+	prepare_repo_settings(the_repository);
+	fsm_settings__set_ipc(the_repository);
+
+	if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {
+		const char *msg = fsm_settings__get_reason_msg(the_repository);
+
+		return error("%s '%s'", msg ? msg : "???", xgetcwd());
+	}
+
 	if (!strcmp(subcmd, "start"))
 		return !!try_to_start_background_daemon();
 
diff --git a/builtin/update-index.c b/builtin/update-index.c
index d335f1ac72a..8f460e7195f 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1237,6 +1237,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 
 	if (fsmonitor > 0) {
 		enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
+
+		if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
+			const char *msg = fsm_settings__get_reason_msg(r);
+
+			return error("%s '%s'", msg ? msg : "???", xgetcwd());
+		}
+
 		if (fsm_mode == FSMONITOR_MODE_DISABLED) {
 			advise(_("core.fsmonitor is unset; "
 				 "set it if you really want to "
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 3b54e7a51f6..8410cc73404 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -9,9 +9,33 @@
  */
 struct fsmonitor_settings {
 	enum fsmonitor_mode mode;
+	enum fsmonitor_reason reason;
 	char *hook_path;
 };
 
+static void set_incompatible(struct repository *r,
+			     enum fsmonitor_reason reason)
+{
+	struct fsmonitor_settings *s = r->settings.fsmonitor;
+
+	s->mode = FSMONITOR_MODE_INCOMPATIBLE;
+	s->reason = reason;
+}
+
+static int check_for_incompatible(struct repository *r)
+{
+	if (!r->worktree) {
+		/*
+		 * Bare repositories don't have a working directory and
+		 * therefore have nothing to watch.
+		 */
+		set_incompatible(r, FSMONITOR_REASON_BARE);
+		return 1;
+	}
+
+	return 0;
+}
+
 static void lookup_fsmonitor_settings(struct repository *r)
 {
 	struct fsmonitor_settings *s;
@@ -87,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r)
 
 	lookup_fsmonitor_settings(r);
 
+	if (check_for_incompatible(r))
+		return;
+
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 }
@@ -98,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path)
 
 	lookup_fsmonitor_settings(r);
 
+	if (check_for_incompatible(r))
+		return;
+
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 	r->settings.fsmonitor->hook_path = strdup(path);
@@ -111,5 +141,32 @@ void fsm_settings__set_disabled(struct repository *r)
 	lookup_fsmonitor_settings(r);
 
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;
+	r->settings.fsmonitor->reason = FSMONITOR_REASON_OK;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 }
+
+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)
+{
+	if (!r)
+		r = the_repository;
+
+	lookup_fsmonitor_settings(r);
+
+	return r->settings.fsmonitor->reason;
+}
+
+const char *fsm_settings__get_reason_msg(struct repository *r)
+{
+	enum fsmonitor_reason reason = fsm_settings__get_reason(r);
+
+	switch (reason) {
+	case FSMONITOR_REASON_OK:
+		return NULL;
+
+	case FSMONITOR_REASON_BARE:
+		return _("bare repos are incompatible with fsmonitor");
+	}
+
+	BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
+	    reason);
+}
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index a4c5d7b4889..a1af058a287 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -4,11 +4,20 @@
 struct repository;
 
 enum fsmonitor_mode {
+	FSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */
 	FSMONITOR_MODE_DISABLED = 0,
 	FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor=<hook_path> */
 	FSMONITOR_MODE_IPC = 2,  /* core.fsmonitor=<true> */
 };
 
+/*
+ * Incompatibility reasons.
+ */
+enum fsmonitor_reason {
+	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
+	FSMONITOR_REASON_BARE,
+};
+
 void fsm_settings__set_ipc(struct repository *r);
 void fsm_settings__set_hook(struct repository *r, const char *path);
 void fsm_settings__set_disabled(struct repository *r);
@@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r);
 enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);
 const char *fsm_settings__get_hook_path(struct repository *r);
 
+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);
+const char *fsm_settings__get_reason_msg(struct repository *r);
+
 struct fsmonitor_settings;
 
 #endif /* FSMONITOR_SETTINGS_H */
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index a6308acf006..cf258d88f04 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '
 	test $ret -ne 1
 '
 
+# Test that we detect and disallow repos that are incompatible with FSMonitor.
+test_expect_success 'incompatible bare repo' '
+	test_when_finished "rm -rf ./bare-clone actual expect" &&
+	git init --bare bare-clone &&
+
+	test_must_fail \
+		git -C ./bare-clone -c core.fsmonitor=foo \
+			update-index --fsmonitor 2>actual &&
+	grep "bare repos are incompatible with fsmonitor" actual &&
+
+	test_must_fail \
+		git -C ./bare-clone -c core.fsmonitor=true \
+			update-index --fsmonitor 2>actual &&
+	grep "bare repos are incompatible with fsmonitor" actual
+'
+
+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
+	test_when_finished "rm -rf ./bare-clone actual" &&
+	git init --bare bare-clone &&
+	test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&
+	grep "bare repos are incompatible with fsmonitor" actual
+'
+
 test_expect_success 'setup' '
 	mkdir -p .git/hooks &&
 	: >tracked &&
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 05/27] fsmonitor-settings: stub in platform-specific incompatibility checking
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (3 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible Jeff Hostetler via GitGitGadget
                     ` (25 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Extend generic incompatibility checkout with platform-specific
mechanism.  Stub in Win32 version.

In the existing fsmonitor-settings code we have a way to mark
types of repos as incompatible with fsmonitor (whether via the
hook and IPC APIs).  For example, we do this for bare repos,
since there are no files to watch.

Extend this exclusion mechanism for platform-specific reasons.
This commit just creates the framework and adds a stub for Win32.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 Makefile                              | 13 +++++++++++++
 compat/fsmonitor/fsm-settings-win32.c |  9 +++++++++
 config.mak.uname                      |  4 ++++
 contrib/buildsystems/CMakeLists.txt   |  3 +++
 fsmonitor-settings.c                  | 12 ++++++++++++
 fsmonitor-settings.h                  | 13 +++++++++++++
 6 files changed, 54 insertions(+)
 create mode 100644 compat/fsmonitor/fsm-settings-win32.c

diff --git a/Makefile b/Makefile
index daa21bed6c3..93604fe8ef7 100644
--- a/Makefile
+++ b/Makefile
@@ -475,6 +475,11 @@ all::
 # `compat/fsmonitor/fsm-listen-<name>.c` that implements the
 # `fsm_listen__*()` routines.
 #
+# If your platform has OS-specific ways to tell if a repo is incompatible with
+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS
+# to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c`
+# that implements the `fsm_os_settings__*()` routines.
+#
 # Define DEVELOPER to enable more compiler warnings. Compiler version
 # and family are auto detected, but could be overridden by defining
 # COMPILER_FEATURES (see config.mak.dev). You can still set
@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND
 	COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
 endif
 
+ifdef FSMONITOR_OS_SETTINGS
+	COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
+	COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
+endif
+
 ifeq ($(TCLTK_PATH),)
 NO_TCLTK = NoThanks
 endif
@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE
 ifdef FSMONITOR_DAEMON_BACKEND
 	@echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+
 endif
+ifdef FSMONITOR_OS_SETTINGS
+	@echo FSMONITOR_OS_SETTINGS=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_OS_SETTINGS)))'\' >>$@+
+endif
 ifdef TEST_OUTPUT_DIRECTORY
 	@echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+
 endif
diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
new file mode 100644
index 00000000000..7fce32a3c5b
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -0,0 +1,9 @@
+#include "cache.h"
+#include "config.h"
+#include "repository.h"
+#include "fsmonitor-settings.h"
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
+{
+	return FSMONITOR_REASON_OK;
+}
diff --git a/config.mak.uname b/config.mak.uname
index 501970902da..cf224768ad6 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)
 	# These are always available, so we do not have to conditionally
 	# support it.
 	FSMONITOR_DAEMON_BACKEND = win32
+	FSMONITOR_OS_SETTINGS = win32
+
 	NO_SVN_TESTS = YesPlease
 	RUNTIME_PREFIX = YesPlease
 	HAVE_WPGMPTR = YesWeDo
@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)
 	# These are always available, so we do not have to conditionally
 	# support it.
 	FSMONITOR_DAEMON_BACKEND = win32
+	FSMONITOR_OS_SETTINGS = win32
+
 	RUNTIME_PREFIX = YesPlease
 	HAVE_WPGMPTR = YesWeDo
 	NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index ee0d7257b77..16705da2000 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)
 	if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+
+		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 8410cc73404..1b16b551d87 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r)
 		return 1;
 	}
 
+#ifdef HAVE_FSMONITOR_OS_SETTINGS
+	{
+		enum fsmonitor_reason reason;
+
+		reason = fsm_os__incompatible(r);
+		if (reason != FSMONITOR_REASON_OK) {
+			set_incompatible(r, reason);
+			return 1;
+		}
+	}
+#endif
+
 	return 0;
 }
 
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index a1af058a287..be25272c012 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -30,4 +30,17 @@ const char *fsm_settings__get_reason_msg(struct repository *r);
 
 struct fsmonitor_settings;
 
+#ifdef HAVE_FSMONITOR_OS_SETTINGS
+/*
+ * Ask platform-specific code whether the repository is incompatible
+ * with fsmonitor (both hook and ipc modes).  For example, if the working
+ * directory is on a remote volume and mounted via a technology that does
+ * not support notification events, then we should not pretend to watch it.
+ *
+ * fsm_os__* routines should considered private to fsm_settings__
+ * routines.
+ */
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);
+#endif /* HAVE_FSMONITOR_OS_SETTINGS */
+
 #endif /* FSMONITOR_SETTINGS_H */
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (4 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 05/27] fsmonitor-settings: stub in platform-specific incompatibility checking Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking Jeff Hostetler via GitGitGadget
                     ` (24 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

VFS for Git virtual repositories are incompatible with FSMonitor.

VFS for Git is a downstream fork of Git.  It contains its own custom
file system watcher that is aware of the virtualization.  If a working
directory is being managed by VFS for Git, we should not try to watch
it because we may get incomplete results.

We do not know anything about how VFS for Git works, but we do
know that VFS for Git working directories contain a well-defined
config setting.  If it is set, mark the working directory as
incompatible.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++
 fsmonitor-settings.c                  |  3 +++
 fsmonitor-settings.h                  |  1 +
 t/t7519-status-fsmonitor.sh           |  9 +++++++++
 4 files changed, 39 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
index 7fce32a3c5b..ee78bba38e3 100644
--- a/compat/fsmonitor/fsm-settings-win32.c
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -3,7 +3,33 @@
 #include "repository.h"
 #include "fsmonitor-settings.h"
 
+/*
+ * VFS for Git is incompatible with FSMonitor.
+ *
+ * Granted, core Git does not know anything about VFS for Git and we
+ * shouldn't make assumptions about a downstream feature, but users
+ * can install both versions.  And this can lead to incorrect results
+ * from core Git commands.  So, without bringing in any of the VFS for
+ * Git code, do a simple config test for a published config setting.
+ * (We do not look at the various *_TEST_* environment variables.)
+ */
+static enum fsmonitor_reason check_vfs4git(struct repository *r)
+{
+	const char *const_str;
+
+	if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str))
+		return FSMONITOR_REASON_VFS4GIT;
+
+	return FSMONITOR_REASON_OK;
+}
+
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
+	enum fsmonitor_reason reason;
+
+	reason = check_vfs4git(r);
+	if (reason != FSMONITOR_REASON_OK)
+		return reason;
+
 	return FSMONITOR_REASON_OK;
 }
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 1b16b551d87..ea3e365dfc4 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -177,6 +177,9 @@ const char *fsm_settings__get_reason_msg(struct repository *r)
 
 	case FSMONITOR_REASON_BARE:
 		return _("bare repos are incompatible with fsmonitor");
+
+	case FSMONITOR_REASON_VFS4GIT:
+		return _("virtual repos are incompatible with fsmonitor");
 	}
 
 	BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index be25272c012..7950529611b 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -16,6 +16,7 @@ enum fsmonitor_mode {
 enum fsmonitor_reason {
 	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
 	FSMONITOR_REASON_BARE,
+	FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */
 };
 
 void fsm_settings__set_ipc(struct repository *r);
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index cf258d88f04..285508fb67e 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
 	grep "bare repos are incompatible with fsmonitor" actual
 '
 
+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '
+	test_when_finished "rm -rf ./fake-virtual-clone actual" &&
+	git init fake-virtual-clone &&
+	test_must_fail git -C ./fake-virtual-clone \
+			   -c core.virtualfilesystem=true \
+			   fsmonitor--daemon run 2>actual &&
+	grep "virtual repos are incompatible with fsmonitor" actual
+'
+
 test_expect_success 'setup' '
 	mkdir -p .git/hooks &&
 	: >tracked &&
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (5 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 08/27] fsmonitor-settings: remote repos on macOS are incompatible Jeff Hostetler via GitGitGadget
                     ` (23 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++
 config.mak.uname                       | 1 +
 contrib/buildsystems/CMakeLists.txt    | 3 +++
 3 files changed, 13 insertions(+)
 create mode 100644 compat/fsmonitor/fsm-settings-darwin.c

diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
new file mode 100644
index 00000000000..7fce32a3c5b
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -0,0 +1,9 @@
+#include "cache.h"
+#include "config.h"
+#include "repository.h"
+#include "fsmonitor-settings.h"
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
+{
+	return FSMONITOR_REASON_OK;
+}
diff --git a/config.mak.uname b/config.mak.uname
index cf224768ad6..cf911d141f2 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)
 	ifndef NO_PTHREADS
 	ifndef NO_UNIX_SOCKETS
 	FSMONITOR_DAEMON_BACKEND = darwin
+	FSMONITOR_OS_SETTINGS = darwin
 	endif
 	endif
 
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 16705da2000..b8f9f7a0388 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
+
+		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
 	endif()
 endif()
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 08/27] fsmonitor-settings: remote repos on macOS are incompatible
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (6 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 09/27] fsmonitor-settings: remote repos on Windows " Jeff Hostetler via GitGitGadget
                     ` (22 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to detect remote working directories on macOS and mark them as
incompatible with FSMonitor.

With this, `git fsmonitor--daemon run` will error out with a message
like it does for bare repos.

Client commands, like `git status`, will not attempt to start the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++
 fsmonitor-settings.c                   |  6 +++
 fsmonitor-settings.h                   |  2 +
 3 files changed, 74 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 7fce32a3c5b..fdd762bf79d 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -2,8 +2,74 @@
 #include "config.h"
 #include "repository.h"
 #include "fsmonitor-settings.h"
+#include "fsmonitor.h"
+#include <sys/param.h>
+#include <sys/mount.h>
+
+/*
+ * Remote working directories are problematic for FSMonitor.
+ *
+ * The underlying file system on the server machine and/or the remote
+ * mount type (NFS, SAMBA, etc.) dictates whether notification events
+ * are available at all to remote client machines.
+ *
+ * Kernel differences between the server and client machines also
+ * dictate the how (buffering, frequency, de-dup) the events are
+ * delivered to client machine processes.
+ *
+ * A client machine (such as a laptop) may choose to suspend/resume
+ * and it is unclear (without lots of testing) whether the watcher can
+ * resync after a resume.  We might be able to treat this as a normal
+ * "events were dropped by the kernel" event and do our normal "flush
+ * and resync" --or-- we might need to close the existing (zombie?)
+ * notification fd and create a new one.
+ *
+ * In theory, the above issues need to be addressed whether we are
+ * using the Hook or IPC API.
+ *
+ * For the builtin FSMonitor, we create the Unix domain socket for the
+ * IPC in the .git directory.  If the working directory is remote,
+ * then the socket will be created on the remote file system.  This
+ * can fail if the remote file system does not support UDS file types
+ * (e.g. smbfs to a Windows server) or if the remote kernel does not
+ * allow a non-local process to bind() the socket.  (These problems
+ * could be fixed by moving the UDS out of the .git directory and to a
+ * well-known local directory on the client machine, but care should
+ * be taken to ensure that $HOME is actually local and not a managed
+ * file share.)
+ *
+ * So (for now at least), mark remote working directories as
+ * incompatible.
+ */
+static enum fsmonitor_reason check_remote(struct repository *r)
+{
+	struct statfs fs;
+
+	if (statfs(r->worktree, &fs) == -1) {
+		int saved_errno = errno;
+		trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s",
+				 r->worktree, strerror(saved_errno));
+		errno = saved_errno;
+		return FSMONITOR_REASON_ERROR;
+	}
+
+	trace_printf_key(&trace_fsmonitor,
+			 "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'",
+			 r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);
+
+	if (!(fs.f_flags & MNT_LOCAL))
+		return FSMONITOR_REASON_REMOTE;
+
+	return FSMONITOR_REASON_OK;
+}
 
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
+	enum fsmonitor_reason reason;
+
+	reason = check_remote(r);
+	if (reason != FSMONITOR_REASON_OK)
+		return reason;
+
 	return FSMONITOR_REASON_OK;
 }
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index ea3e365dfc4..7ff3f98964d 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -178,6 +178,12 @@ const char *fsm_settings__get_reason_msg(struct repository *r)
 	case FSMONITOR_REASON_BARE:
 		return _("bare repos are incompatible with fsmonitor");
 
+	case FSMONITOR_REASON_ERROR:
+		return _("repo incompatible with fsmonitor due to errors");
+
+	case FSMONITOR_REASON_REMOTE:
+		return _("remote repos are incompatible with fsmonitor");
+
 	case FSMONITOR_REASON_VFS4GIT:
 		return _("virtual repos are incompatible with fsmonitor");
 	}
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index 7950529611b..6aa9a00379b 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -16,6 +16,8 @@ enum fsmonitor_mode {
 enum fsmonitor_reason {
 	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
 	FSMONITOR_REASON_BARE,
+	FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */
+	FSMONITOR_REASON_REMOTE,
 	FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */
 };
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 09/27] fsmonitor-settings: remote repos on Windows are incompatible
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (7 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 08/27] fsmonitor-settings: remote repos on macOS are incompatible Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 10/27] unpack-trees: initialize fsmonitor_has_run_once in o->result Jeff Hostetler via GitGitGadget
                     ` (21 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to detect remote working directories on Windows and mark them as
incompatible with FSMonitor.

With this `git fsmonitor--daemon run` will error out with a message like it
does for bare repos.

Client commands, such as `git status`, will not attempt to start the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++
 1 file changed, 102 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
index ee78bba38e3..907655720bb 100644
--- a/compat/fsmonitor/fsm-settings-win32.c
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -2,6 +2,7 @@
 #include "config.h"
 #include "repository.h"
 #include "fsmonitor-settings.h"
+#include "fsmonitor.h"
 
 /*
  * VFS for Git is incompatible with FSMonitor.
@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)
 	return FSMONITOR_REASON_OK;
 }
 
+/*
+ * Remote working directories are problematic for FSMonitor.
+ *
+ * The underlying file system on the server machine and/or the remote
+ * mount type dictates whether notification events are available at
+ * all to remote client machines.
+ *
+ * Kernel differences between the server and client machines also
+ * dictate the how (buffering, frequency, de-dup) the events are
+ * delivered to client machine processes.
+ *
+ * A client machine (such as a laptop) may choose to suspend/resume
+ * and it is unclear (without lots of testing) whether the watcher can
+ * resync after a resume.  We might be able to treat this as a normal
+ * "events were dropped by the kernel" event and do our normal "flush
+ * and resync" --or-- we might need to close the existing (zombie?)
+ * notification fd and create a new one.
+ *
+ * In theory, the above issues need to be addressed whether we are
+ * using the Hook or IPC API.
+ *
+ * So (for now at least), mark remote working directories as
+ * incompatible.
+ *
+ * Notes for testing:
+ *
+ * (a) Windows allows a network share to be mapped to a drive letter.
+ *     (This is the normal method to access it.)
+ *
+ *     $ NET USE Z: \\server\share
+ *     $ git -C Z:/repo status
+ *
+ * (b) Windows allows a network share to be referenced WITHOUT mapping
+ *     it to drive letter.
+ *
+ *     $ NET USE \\server\share\dir
+ *     $ git -C //server/share/repo status
+ *
+ * (c) Windows allows "SUBST" to create a fake drive mapping to an
+ *     arbitrary path (which may be remote)
+ *
+ *     $ SUBST Q: Z:\repo
+ *     $ git -C Q:/ status
+ *
+ * (d) Windows allows a directory symlink to be created on a local
+ *     file system that points to a remote repo.
+ *
+ *     $ mklink /d ./link //server/share/repo
+ *     $ git -C ./link status
+ */
+static enum fsmonitor_reason check_remote(struct repository *r)
+{
+	wchar_t wpath[MAX_PATH];
+	wchar_t wfullpath[MAX_PATH];
+	size_t wlen;
+	UINT driveType;
+
+	/*
+	 * Do everything in wide chars because the drive letter might be
+	 * a multi-byte sequence.  See win32_has_dos_drive_prefix().
+	 */
+	if (xutftowcs_path(wpath, r->worktree) < 0)
+		return FSMONITOR_REASON_ERROR;
+
+	/*
+	 * GetDriveTypeW() requires a final slash.  We assume that the
+	 * worktree pathname points to an actual directory.
+	 */
+	wlen = wcslen(wpath);
+	if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') {
+		wpath[wlen++] = L'\\';
+		wpath[wlen] = 0;
+	}
+
+	/*
+	 * Normalize the path.  If nothing else, this converts forward
+	 * slashes to backslashes.  This is essential to get GetDriveTypeW()
+	 * correctly handle some UNC "\\server\share\..." paths.
+	 */
+	if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))
+		return FSMONITOR_REASON_ERROR;
+
+	driveType = GetDriveTypeW(wfullpath);
+	trace_printf_key(&trace_fsmonitor,
+			 "DriveType '%s' L'%ls' (%u)",
+			 r->worktree, wfullpath, driveType);
+
+	if (driveType == DRIVE_REMOTE) {
+		trace_printf_key(&trace_fsmonitor,
+				 "check_remote('%s') true",
+				 r->worktree);
+		return FSMONITOR_REASON_REMOTE;
+	}
+
+	return FSMONITOR_REASON_OK;
+}
+
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
 	enum fsmonitor_reason reason;
@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 	if (reason != FSMONITOR_REASON_OK)
 		return reason;
 
+	reason = check_remote(r);
+	if (reason != FSMONITOR_REASON_OK)
+		return reason;
+
 	return FSMONITOR_REASON_OK;
 }
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 10/27] unpack-trees: initialize fsmonitor_has_run_once in o->result
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (8 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 09/27] fsmonitor-settings: remote repos on Windows " Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 11/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Jeff Hostetler via GitGitGadget
                     ` (20 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Initialize `o->result.fsmonitor_has_run_once` based upon value
in `o->src_index->fsmonitor_has_run_once` to prevent a second
fsmonitor query during the tree traversal and possibly getting
a skewed view of the working directory.

The checkout code has already talked to the fsmonitor and the
traversal is updating the index as it traverses, so there is
no need to query the fsmonitor.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 unpack-trees.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/unpack-trees.c b/unpack-trees.c
index 360844bda3a..888cff81f9c 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 
 	o->result.fsmonitor_last_update =
 		xstrdup_or_null(o->src_index->fsmonitor_last_update);
+	o->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;
 
 	/*
 	 * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 11/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (9 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 10/27] unpack-trees: initialize fsmonitor_has_run_once in o->result Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 12/27] fsmonitor--daemon: cd out of worktree root Jeff Hostetler via GitGitGadget
                     ` (19 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Ignore FSEvents resulting from `xattr` changes.  Git does not care about
xattr's or changes to xattr's, so don't waste time collecting these
events in the daemon nor transmitting them to clients.

Various security tools add xattrs to files and/or directories, such as
to mark them as having been downloaded.  We should ignore these events
since it doesn't affect the content of the file/directory or the normal
meta-data that Git cares about.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 5c5de1ae702..f9b61b7b1c0 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -172,7 +172,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)
 	if (flag & kFSEventStreamEventFlagItemCloned)
 		strbuf_addstr(&msg, "ItemCloned|");
 
-	trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s",
+	trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s",
 			 path, flag, msg.buf);
 
 	strbuf_release(&msg);
@@ -197,6 +197,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)
 		ef & kFSEventStreamEventFlagUserDropped);
 }
 
+/*
+ * If an `xattr` change is the only reason we received this event,
+ * then silently ignore it.  Git doesn't care about xattr's.  We
+ * have to be careful here because the kernel can combine multiple
+ * events for a single path.  And because events always have certain
+ * bits set, such as `ItemIsFile` or `ItemIsDir`.
+ *
+ * Return 1 if we should ignore it.
+ */
+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)
+{
+	static const FSEventStreamEventFlags mask =
+		kFSEventStreamEventFlagItemChangeOwner |
+		kFSEventStreamEventFlagItemCreated |
+		kFSEventStreamEventFlagItemFinderInfoMod |
+		kFSEventStreamEventFlagItemInodeMetaMod |
+		kFSEventStreamEventFlagItemModified |
+		kFSEventStreamEventFlagItemRemoved |
+		kFSEventStreamEventFlagItemRenamed |
+		kFSEventStreamEventFlagItemXattrMod |
+		kFSEventStreamEventFlagItemCloned;
+
+	return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);
+}
+
 static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			     void *ctx,
 			     size_t num_of_events,
@@ -262,6 +287,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			continue;
 		}
 
+		if (ef_ignore_xattr(event_flags[k])) {
+			trace_printf_key(&trace_fsmonitor,
+					 "ignore-xattr: '%s', flags=0x%x",
+					 path_k, event_flags[k]);
+			continue;
+		}
+
 		switch (fsmonitor_classify_path_absolute(state, path_k)) {
 
 		case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 12/27] fsmonitor--daemon: cd out of worktree root
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (10 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 11/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 13/27] fsmonitor--daemon: prepare for adding health thread Jeff Hostetler via GitGitGadget
                     ` (18 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach the fsmonitor--daemon to CD outside of the worktree
before starting up.

The common Git startup mechanism causes the CWD of the daemon process
to be in the root of the worktree.  On Windows, this causes the daemon
process to hold a locked handle on the CWD and prevents other
processes from moving or deleting the worktree while the daemon is
running.

CD to HOME before entering main event loops.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c         | 32 +++++++++++++++++++++++++++--
 compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------
 fsmonitor--daemon.h                 |  1 +
 3 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 00eaffbb945..dbd37a2b3b8 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1175,11 +1175,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	 * before we need it.
 	 */
 	if (ipc_server_run_async(&state->ipc_server_data,
-				 fsmonitor_ipc__get_path(), &ipc_opts,
+				 state->path_ipc.buf, &ipc_opts,
 				 handle_client, state))
 		return error_errno(
 			_("could not start IPC thread pool on '%s'"),
-			fsmonitor_ipc__get_path());
+			state->path_ipc.buf);
 
 	/*
 	 * Start the fsmonitor listener thread to collect filesystem
@@ -1214,6 +1214,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 static int fsmonitor_run_daemon(void)
 {
 	struct fsmonitor_daemon_state state;
+	const char *home;
 	int err;
 
 	memset(&state, 0, sizeof(state));
@@ -1283,6 +1284,15 @@ static int fsmonitor_run_daemon(void)
 
 	strbuf_addch(&state.path_cookie_prefix, '/');
 
+	/*
+	 * We create a named-pipe or unix domain socket inside of the
+	 * ".git" directory.  (Well, on Windows, we base our named
+	 * pipe in the NPFS on the absolute path of the git
+	 * directory.)
+	 */
+	strbuf_init(&state.path_ipc, 0);
+	strbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));
+
 	/*
 	 * Confirm that we can create platform-specific resources for the
 	 * filesystem listener before we bother starting all the threads.
@@ -1292,6 +1302,23 @@ static int fsmonitor_run_daemon(void)
 		goto done;
 	}
 
+	/*
+	 * CD out of the worktree root directory.
+	 *
+	 * The common Git startup mechanism causes our CWD to be the
+	 * root of the worktree.  On Windows, this causes our process
+	 * to hold a locked handle on the CWD.  This prevents the
+	 * worktree from being moved or deleted while the daemon is
+	 * running.
+	 *
+	 * We assume that our FS and IPC listener threads have either
+	 * opened all of the handles that they need or will do
+	 * everything using absolute paths.
+	 */
+	home = getenv("HOME");
+	if (home && *home && chdir(home))
+		die_errno("could not cd home '%s'", home);
+
 	err = fsmonitor_run_daemon_1(&state);
 
 done:
@@ -1304,6 +1331,7 @@ done:
 	strbuf_release(&state.path_worktree_watch);
 	strbuf_release(&state.path_gitdir_watch);
 	strbuf_release(&state.path_cookie_prefix);
+	strbuf_release(&state.path_ipc);
 
 	return err;
 }
diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index f4673d7d8b1..948f0e63915 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch)
 	}
 
 	/*
-	 * NEEDSWORK: If an external <gitdir> is deleted, the above
-	 * returns an error.  I'm not sure that there's anything that
-	 * we can do here other than failing -- the <worktree>/.git
-	 * link file would be broken anyway.  We might try to check
-	 * for that and return a better error message, but I'm not
-	 * sure it is worth it.
+	 * GetOverlappedResult() fails if the watched directory is
+	 * deleted while we were waiting for an overlapped IO to
+	 * complete.  The documentation did not list specific errors,
+	 * but I observed ERROR_ACCESS_DENIED (0x05) errors during
+	 * testing.
+	 *
+	 * Note that we only get notificaiton events for events
+	 * *within* the directory, not *on* the directory itself.
+	 * (These might be properies of the parent directory, for
+	 * example).
+	 *
+	 * NEEDSWORK: We might try to check for the deleted directory
+	 * case and return a better error message, but I'm not sure it
+	 * is worth it.
+	 *
+	 * Shutdown if we get any error.
 	 */
 
 	error("GetOverlappedResult failed on '%s' [GLE %ld]",
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index bd09fffc176..223c2131b58 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {
 	struct fsmonitor_daemon_backend_data *backend_data;
 
 	struct ipc_server_data *ipc_server_data;
+	struct strbuf path_ipc;
 };
 
 /*
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 13/27] fsmonitor--daemon: prepare for adding health thread
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (11 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 12/27] fsmonitor--daemon: cd out of worktree root Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 14/27] fsmonitor--daemon: rename listener thread related variables Jeff Hostetler via GitGitGadget
                     ` (17 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Refactor daemon thread startup to make it easier to start
a third thread class to monitor the health of the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index dbd37a2b3b8..d5a8fc2ddc2 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1168,6 +1168,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 		 */
 		.uds_disallow_chdir = 0
 	};
+	int listener_started = 0;
+	int err = 0;
 
 	/*
 	 * Start the IPC thread pool before the we've started the file
@@ -1188,15 +1190,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	if (pthread_create(&state->listener_thread, NULL,
 			   fsm_listen__thread_proc, state) < 0) {
 		ipc_server_stop_async(state->ipc_server_data);
-		ipc_server_await(state->ipc_server_data);
-
-		return error(_("could not start fsmonitor listener thread"));
+		err = error(_("could not start fsmonitor listener thread"));
+		goto cleanup;
 	}
+	listener_started = 1;
 
 	/*
 	 * The daemon is now fully functional in background threads.
+	 * Our primary thread should now just wait while the threads
+	 * do all the work.
+	 */
+cleanup:
+	/*
 	 * Wait for the IPC thread pool to shutdown (whether by client
-	 * request or from filesystem activity).
+	 * request, from filesystem activity, or an error).
 	 */
 	ipc_server_await(state->ipc_server_data);
 
@@ -1205,10 +1212,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	 * event from the IPC thread pool, but it doesn't hurt to tell
 	 * it again.  And wait for it to shutdown.
 	 */
-	fsm_listen__stop_async(state);
-	pthread_join(state->listener_thread, NULL);
+	if (listener_started) {
+		fsm_listen__stop_async(state);
+		pthread_join(state->listener_thread, NULL);
+	}
 
-	return state->error_code;
+	if (err)
+		return err;
+	if (state->error_code)
+		return state->error_code;
+	return 0;
 }
 
 static int fsmonitor_run_daemon(void)
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 14/27] fsmonitor--daemon: rename listener thread related variables
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (12 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 13/27] fsmonitor--daemon: prepare for adding health thread Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 15/27] fsmonitor--daemon: stub in health thread Jeff Hostetler via GitGitGadget
                     ` (16 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Rename platform-specific listener thread related variables
and data types as we prepare to add another backend thread
type.

[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`
[] `state->backend_data` becomes `state->listen_data`
[] `state->error_code` becomes `state->listen_error_code`

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c          |  6 +++---
 compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------
 compat/fsmonitor/fsm-listen-win32.c  | 28 +++++++++++++-------------
 compat/fsmonitor/fsm-listen.h        |  2 +-
 fsmonitor--daemon.h                  |  6 +++---
 5 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index d5a8fc2ddc2..92c61d5b94d 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1219,8 +1219,8 @@ cleanup:
 
 	if (err)
 		return err;
-	if (state->error_code)
-		return state->error_code;
+	if (state->listen_error_code)
+		return state->listen_error_code;
 	return 0;
 }
 
@@ -1235,7 +1235,7 @@ static int fsmonitor_run_daemon(void)
 	hashmap_init(&state.cookies, cookies_cmp, NULL, 0);
 	pthread_mutex_init(&state.main_lock, NULL);
 	pthread_cond_init(&state.cookies_cond, NULL);
-	state.error_code = 0;
+	state.listen_error_code = 0;
 	state.current_token_data = fsmonitor_new_token_data();
 
 	/* Prepare to (recursively) watch the <worktree-root> directory. */
diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index f9b61b7b1c0..7c81bc7e5bf 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -99,7 +99,7 @@ void FSEventStreamRelease(FSEventStreamRef stream);
 #include "fsm-listen.h"
 #include "fsmonitor--daemon.h"
 
-struct fsmonitor_daemon_backend_data
+struct fsm_listen_data
 {
 	CFStringRef cfsr_worktree_path;
 	CFStringRef cfsr_gitdir_path;
@@ -230,7 +230,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			     const FSEventStreamEventId event_ids[])
 {
 	struct fsmonitor_daemon_state *state = ctx;
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	char **paths = (char **)event_paths;
 	struct fsmonitor_batch *batch = NULL;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -422,11 +422,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 		NULL,
 		NULL
 	};
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 	const void *dir_array[2];
 
 	CALLOC_ARRAY(data, 1);
-	state->backend_data = data;
+	state->listen_data = data;
 
 	data->cfsr_worktree_path = CFStringCreateWithCString(
 		NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);
@@ -458,18 +458,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 failed:
 	error("Unable to create FSEventStream.");
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 	return -1;
 }
 
 void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	if (!state || !state->backend_data)
+	if (!state || !state->listen_data)
 		return;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	if (data->stream) {
 		if (data->stream_started)
@@ -479,14 +479,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 		FSEventStreamRelease(data->stream);
 	}
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 }
 
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	data = state->backend_data;
+	data = state->listen_data;
 	data->shutdown_style = SHUTDOWN_EVENT;
 
 	CFRunLoopStop(data->rl);
@@ -494,9 +494,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 
 void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	data->rl = CFRunLoopGetCurrent();
 
@@ -513,7 +513,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 
 	switch (data->shutdown_style) {
 	case FORCE_ERROR_STOP:
-		state->error_code = -1;
+		state->listen_error_code = -1;
 		/* fall thru */
 	case FORCE_SHUTDOWN:
 		ipc_server_stop_async(state->ipc_server_data);
@@ -525,7 +525,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 	return;
 
 force_error_stop_without_loop:
-	state->error_code = -1;
+	state->listen_error_code = -1;
 	ipc_server_stop_async(state->ipc_server_data);
 	return;
 }
diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index 948f0e63915..a7c5d263940 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -54,7 +54,7 @@ struct one_watch
 	wchar_t dotgit_shortname[16]; /* for 8.3 name */
 };
 
-struct fsmonitor_daemon_backend_data
+struct fsm_listen_data
 {
 	struct one_watch *watch_worktree;
 	struct one_watch *watch_gitdir;
@@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname(
 
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
-	SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);
+	SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);
 }
 
 static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
@@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch)
 	free(watch);
 }
 
-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,
+static int start_rdcw_watch(struct fsm_listen_data *data,
 			    struct one_watch *watch)
 {
 	DWORD dwNotifyFilter =
@@ -516,7 +516,7 @@ static int process_1_worktree_event(
  */
 static int process_worktree_events(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	struct one_watch *watch = data->watch_worktree;
 	struct strbuf path = STRBUF_INIT;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -646,7 +646,7 @@ force_shutdown:
  */
 static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	struct one_watch *watch = data->watch_gitdir;
 	struct strbuf path = STRBUF_INIT;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -704,11 +704,11 @@ skip_this_path:
 
 void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	DWORD dwWait;
 	int result;
 
-	state->error_code = 0;
+	state->listen_error_code = 0;
 
 	if (start_rdcw_watch(data, data->watch_worktree) == -1)
 		goto force_error_stop;
@@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 	}
 
 force_error_stop:
-	state->error_code = -1;
+	state->listen_error_code = -1;
 
 force_shutdown:
 	/*
@@ -790,7 +790,7 @@ clean_shutdown:
 
 int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
 	CALLOC_ARRAY(data, 1);
 
@@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 		data->nr_listener_handles++;
 	}
 
-	state->backend_data = data;
+	state->listen_data = data;
 	return 0;
 
 failed:
@@ -836,16 +836,16 @@ failed:
 
 void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	if (!state || !state->backend_data)
+	if (!state || !state->listen_data)
 		return;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	CloseHandle(data->hEventShutdown);
 	destroy_watch(data->watch_worktree);
 	destroy_watch(data->watch_gitdir);
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 }
diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h
index f0539349baf..41650bf8972 100644
--- a/compat/fsmonitor/fsm-listen.h
+++ b/compat/fsmonitor/fsm-listen.h
@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);
  * do so if the listener thread receives a normal shutdown signal from
  * the IPC layer.)
  *
- * It should set `state->error_code` to -1 if the daemon should exit
+ * It should set `state->listen_error_code` to -1 if the daemon should exit
  * with an error.
  */
 void fsm_listen__loop(struct fsmonitor_daemon_state *state);
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index 223c2131b58..2c6fa1a5d91 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);
  */
 void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);
 
-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */
+struct fsm_listen_data; /* opaque platform-specific data for listener thread */
 
 struct fsmonitor_daemon_state {
 	pthread_t listener_thread;
@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {
 	int cookie_seq;
 	struct hashmap cookies;
 
-	int error_code;
-	struct fsmonitor_daemon_backend_data *backend_data;
+	int listen_error_code;
+	struct fsm_listen_data *listen_data;
 
 	struct ipc_server_data *ipc_server_data;
 	struct strbuf path_ipc;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 15/27] fsmonitor--daemon: stub in health thread
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (13 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 14/27] fsmonitor--daemon: rename listener thread related variables Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 16/27] fsm-health-win32: add polling framework to monitor daemon health Jeff Hostetler via GitGitGadget
                     ` (15 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create another thread to watch over the daemon process and
automatically shut it down if necessary.

This commit creates the basic framework for a "health" thread
to monitor the daemon and/or the file system.  Later commits
will add platform-specific code to do the actual work.

The "health" thread is intended to monitor conditions that
would be difficult to track inside the IPC thread pool and/or
the file system listener threads.  For example, when there are
file system events outside of the watched worktree root or if
we want to have an idle-timeout auto-shutdown feature.

This commit creates the health thread itself, defines the thread-proc
and sets up the thread's event loop.  It integrates this new thread
into the existing IPC and Listener thread models.

This commit defines the API to the platform-specific code where all of
the monitoring will actually happen.

The platform-specific code for MacOS is just stubs.  Meaning that the
health thread will immediately exit on MacOS, but that is OK and
expected.  Future work can define MacOS-specific monitoring.

The platform-specific code for Windows sets up enough of the
WaitForMultipleObjects() machinery to watch for system and/or custom
events.  Currently, the set of wait handles only includes our custom
shutdown event (sent from our other theads).  Later commits in this
series will extend the set of wait handles to monitor other
conditions.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 Makefile                             |  6 ++-
 builtin/fsmonitor--daemon.c          | 39 +++++++++++++++
 compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++
 compat/fsmonitor/fsm-health-win32.c  | 72 ++++++++++++++++++++++++++++
 compat/fsmonitor/fsm-health.h        | 47 ++++++++++++++++++
 contrib/buildsystems/CMakeLists.txt  |  2 +
 fsmonitor--daemon.h                  |  4 ++
 7 files changed, 192 insertions(+), 2 deletions(-)
 create mode 100644 compat/fsmonitor/fsm-health-darwin.c
 create mode 100644 compat/fsmonitor/fsm-health-win32.c
 create mode 100644 compat/fsmonitor/fsm-health.h

diff --git a/Makefile b/Makefile
index 93604fe8ef7..5f1623baadd 100644
--- a/Makefile
+++ b/Makefile
@@ -472,8 +472,9 @@ all::
 #
 # If your platform supports a built-in fsmonitor backend, set
 # FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding
-# `compat/fsmonitor/fsm-listen-<name>.c` that implements the
-# `fsm_listen__*()` routines.
+# `compat/fsmonitor/fsm-listen-<name>.c` and
+# `compat/fsmonitor/fsm-health-<name>.c` files
+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.
 #
 # If your platform has OS-specific ways to tell if a repo is incompatible with
 # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS
@@ -1982,6 +1983,7 @@ endif
 ifdef FSMONITOR_DAEMON_BACKEND
 	COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
 	COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
+	COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
 endif
 
 ifdef FSMONITOR_OS_SETTINGS
diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 92c61d5b94d..bebb3a292e2 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -3,6 +3,7 @@
 #include "parse-options.h"
 #include "fsmonitor.h"
 #include "fsmonitor-ipc.h"
+#include "compat/fsmonitor/fsm-health.h"
 #include "compat/fsmonitor/fsm-listen.h"
 #include "fsmonitor--daemon.h"
 #include "simple-ipc.h"
@@ -1130,6 +1131,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,
 	pthread_mutex_unlock(&state->main_lock);
 }
 
+static void *fsm_health__thread_proc(void *_state)
+{
+	struct fsmonitor_daemon_state *state = _state;
+
+	trace2_thread_start("fsm-health");
+
+	fsm_health__loop(state);
+
+	trace2_thread_exit();
+	return NULL;
+}
+
 static void *fsm_listen__thread_proc(void *_state)
 {
 	struct fsmonitor_daemon_state *state = _state;
@@ -1168,6 +1181,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 		 */
 		.uds_disallow_chdir = 0
 	};
+	int health_started = 0;
 	int listener_started = 0;
 	int err = 0;
 
@@ -1195,6 +1209,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	}
 	listener_started = 1;
 
+	/*
+	 * Start the health thread to watch over our process.
+	 */
+	if (pthread_create(&state->health_thread, NULL,
+			   fsm_health__thread_proc, state) < 0) {
+		ipc_server_stop_async(state->ipc_server_data);
+		err = error(_("could not start fsmonitor health thread"));
+		goto cleanup;
+	}
+	health_started = 1;
+
 	/*
 	 * The daemon is now fully functional in background threads.
 	 * Our primary thread should now just wait while the threads
@@ -1217,10 +1242,17 @@ cleanup:
 		pthread_join(state->listener_thread, NULL);
 	}
 
+	if (health_started) {
+		fsm_health__stop_async(state);
+		pthread_join(state->health_thread, NULL);
+	}
+
 	if (err)
 		return err;
 	if (state->listen_error_code)
 		return state->listen_error_code;
+	if (state->health_error_code)
+		return state->health_error_code;
 	return 0;
 }
 
@@ -1236,6 +1268,7 @@ static int fsmonitor_run_daemon(void)
 	pthread_mutex_init(&state.main_lock, NULL);
 	pthread_cond_init(&state.cookies_cond, NULL);
 	state.listen_error_code = 0;
+	state.health_error_code = 0;
 	state.current_token_data = fsmonitor_new_token_data();
 
 	/* Prepare to (recursively) watch the <worktree-root> directory. */
@@ -1315,6 +1348,11 @@ static int fsmonitor_run_daemon(void)
 		goto done;
 	}
 
+	if (fsm_health__ctor(&state)) {
+		err = error(_("could not initialize health thread"));
+		goto done;
+	}
+
 	/*
 	 * CD out of the worktree root directory.
 	 *
@@ -1338,6 +1376,7 @@ done:
 	pthread_cond_destroy(&state.cookies_cond);
 	pthread_mutex_destroy(&state.main_lock);
 	fsm_listen__dtor(&state);
+	fsm_health__dtor(&state);
 
 	ipc_server_free(state.ipc_server_data);
 
diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c
new file mode 100644
index 00000000000..b9f709e8548
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-darwin.c
@@ -0,0 +1,24 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+	return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+	return;
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+	return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+}
diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
new file mode 100644
index 00000000000..94b1d020f25
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -0,0 +1,72 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+struct fsm_health_data
+{
+	HANDLE hEventShutdown;
+
+	HANDLE hHandles[1]; /* the array does not own these handles */
+#define HEALTH_SHUTDOWN 0
+	int nr_handles; /* number of active event handles */
+};
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data;
+
+	CALLOC_ARRAY(data, 1);
+
+	data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);
+
+	data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;
+	data->nr_handles++;
+
+	state->health_data = data;
+	return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data;
+
+	if (!state || !state->health_data)
+		return;
+
+	data = state->health_data;
+
+	CloseHandle(data->hEventShutdown);
+
+	FREE_AND_NULL(state->health_data);
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data = state->health_data;
+
+	for (;;) {
+		DWORD dwWait = WaitForMultipleObjects(data->nr_handles,
+						      data->hHandles,
+						      FALSE, INFINITE);
+
+		if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)
+			goto clean_shutdown;
+
+		error(_("health thread wait failed [GLE %ld]"),
+		      GetLastError());
+		goto force_error_stop;
+	}
+
+force_error_stop:
+	state->health_error_code = -1;
+	ipc_server_stop_async(state->ipc_server_data);
+clean_shutdown:
+	return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+	SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);
+}
diff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h
new file mode 100644
index 00000000000..45547ba9380
--- /dev/null
+++ b/compat/fsmonitor/fsm-health.h
@@ -0,0 +1,47 @@
+#ifndef FSM_HEALTH_H
+#define FSM_HEALTH_H
+
+/* This needs to be implemented by each backend */
+
+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
+
+struct fsmonitor_daemon_state;
+
+/*
+ * Initialize platform-specific data for the fsmonitor health thread.
+ * This will be called from the main thread PRIOR to staring the
+ * thread.
+ *
+ * Returns 0 if successful.
+ * Returns -1 otherwise.
+ */
+int fsm_health__ctor(struct fsmonitor_daemon_state *state);
+
+/*
+ * Cleanup platform-specific data for the health thread.
+ * This will be called from the main thread AFTER joining the thread.
+ */
+void fsm_health__dtor(struct fsmonitor_daemon_state *state);
+
+/*
+ * The main body of the platform-specific event loop to monitor the
+ * health of the daemon process.  This will run in the health thread.
+ *
+ * The health thread should call `ipc_server_stop_async()` if it needs
+ * to cause a shutdown.  (It should NOT do so if it receives a shutdown
+ * shutdown signal.)
+ *
+ * It should set `state->health_error_code` to -1 if the daemon should exit
+ * with an error.
+ */
+void fsm_health__loop(struct fsmonitor_daemon_state *state);
+
+/*
+ * Gently request that the health thread shutdown.
+ * It does not wait for it to stop.  The caller should do a JOIN
+ * to wait for it.
+ */
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);
+
+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */
+#endif /* FSM_HEALTH_H */
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index b8f9f7a0388..16ace43d1c7 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)
 	if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
 
 		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
 
 		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index 2c6fa1a5d91..2102a5c9ff5 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);
 void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);
 
 struct fsm_listen_data; /* opaque platform-specific data for listener thread */
+struct fsm_health_data; /* opaque platform-specific data for health thread */
 
 struct fsmonitor_daemon_state {
 	pthread_t listener_thread;
+	pthread_t health_thread;
 	pthread_mutex_t main_lock;
 
 	struct strbuf path_worktree_watch;
@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {
 	struct hashmap cookies;
 
 	int listen_error_code;
+	int health_error_code;
 	struct fsm_listen_data *listen_data;
+	struct fsm_health_data *health_data;
 
 	struct ipc_server_data *ipc_server_data;
 	struct strbuf path_ipc;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 16/27] fsm-health-win32: add polling framework to monitor daemon health
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (14 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 15/27] fsmonitor--daemon: stub in health thread Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 17/27] fsm-health-win32: force shutdown daemon if worktree root moves Jeff Hostetler via GitGitGadget
                     ` (14 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Extend the Windows version of the "health" thread to periodically
inspect the system and shutdown if warranted.

This commit updates the thread's wait loop to use a timeout and
defines a (currently empty) table of functions to poll the system.

A later commit will add functions to the table to actually
inspect the system.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-
 1 file changed, 64 insertions(+), 1 deletion(-)

diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
index 94b1d020f25..24fc612bf02 100644
--- a/compat/fsmonitor/fsm-health-win32.c
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -4,6 +4,24 @@
 #include "fsm-health.h"
 #include "fsmonitor--daemon.h"
 
+/*
+ * Every minute wake up and test our health.
+ */
+#define WAIT_FREQ_MS (60 * 1000)
+
+/*
+ * State machine states for each of the interval functions
+ * used for polling our health.
+ */
+enum interval_fn_ctx {
+	CTX_INIT = 0,
+	CTX_TERM,
+	CTX_TIMER
+};
+
+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,
+			  enum interval_fn_ctx ctx);
+
 struct fsm_health_data
 {
 	HANDLE hEventShutdown;
@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)
 	FREE_AND_NULL(state->health_data);
 }
 
+/*
+ * A table of the polling functions.
+ */
+static interval_fn *table[] = {
+	NULL, /* must be last */
+};
+
+/*
+ * Call all of the polling functions in the table.
+ * Shortcut and return first error.
+ *
+ * Return 0 if all succeeded.
+ */
+static int call_all(struct fsmonitor_daemon_state *state,
+		    enum interval_fn_ctx ctx)
+{
+	int k;
+
+	for (k = 0; table[k]; k++) {
+		int r = table[k](state, ctx);
+		if (r)
+			return r;
+	}
+
+	return 0;
+}
+
 void fsm_health__loop(struct fsmonitor_daemon_state *state)
 {
 	struct fsm_health_data *data = state->health_data;
+	int r;
+
+	r = call_all(state, CTX_INIT);
+	if (r < 0)
+		goto force_error_stop;
+	if (r > 0)
+		goto force_shutdown;
 
 	for (;;) {
 		DWORD dwWait = WaitForMultipleObjects(data->nr_handles,
 						      data->hHandles,
-						      FALSE, INFINITE);
+						      FALSE, WAIT_FREQ_MS);
 
 		if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)
 			goto clean_shutdown;
 
+		if (dwWait == WAIT_TIMEOUT) {
+			r = call_all(state, CTX_TIMER);
+			if (r < 0)
+				goto force_error_stop;
+			if (r > 0)
+				goto force_shutdown;
+			continue;
+		}
+
 		error(_("health thread wait failed [GLE %ld]"),
 		      GetLastError());
 		goto force_error_stop;
@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)
 
 force_error_stop:
 	state->health_error_code = -1;
+force_shutdown:
 	ipc_server_stop_async(state->ipc_server_data);
 clean_shutdown:
+	call_all(state, CTX_TERM);
 	return;
 }
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 17/27] fsm-health-win32: force shutdown daemon if worktree root moves
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (15 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 16/27] fsm-health-win32: add polling framework to monitor daemon health Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 18/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Jeff Hostetler via GitGitGadget
                     ` (13 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Force shutdown fsmonitor daemon if the worktree root directory
is moved, renamed, or deleted.

Use Windows low-level GetFileInformationByHandle() to get and
compare the Windows system unique ID for the directory with a
cached version when we started up.  This lets us detect the
case where someone renames the directory that we are watching
and then creates a new directory with the original pathname.

This is important because we are listening to a named pipe for
requests and they are stored in the Named Pipe File System (NPFS)
which a kernel-resident pseudo filesystem not associated with
the actual NTFS directory.

For example, if the daemon was watching "~/foo/", it would have
a directory-watch handle on that directory and a named-pipe
handle for "//./pipe/...foo".  Moving the directory to "~/bar/"
does not invalidate the directory handle.  (So the daemon would
actually be watching "~/bar" but listening on "//./pipe/...foo".
If the user then does "git init ~/foo" and causes another daemon
to start, the first daemon will still have ownership of the pipe
and the second daemon instance will fail to start.  "git status"
clients in "~/foo" will ask "//./pipe/...foo" about changes and
the first daemon instance will tell them about "~/bar".

This commit causes the first daemon to shutdown if the system unique
ID for "~/foo" changes (changes from what it was when the daemon
started).  Shutdown occurs after a periodic poll.  After the
first daemon exits and releases the lock on the named pipe,
subsequent Git commands may cause another daemon to be started
on "~/foo".  Similarly, a subsequent Git command may cause another
daemon to be started on "~/bar".

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++
 1 file changed, 143 insertions(+)

diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
index 24fc612bf02..0132ca79305 100644
--- a/compat/fsmonitor/fsm-health-win32.c
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -29,8 +29,150 @@ struct fsm_health_data
 	HANDLE hHandles[1]; /* the array does not own these handles */
 #define HEALTH_SHUTDOWN 0
 	int nr_handles; /* number of active event handles */
+
+	struct wt_moved
+	{
+		wchar_t wpath[MAX_PATH + 1];
+		BY_HANDLE_FILE_INFORMATION bhfi;
+	} wt_moved;
 };
 
+/*
+ * Lookup the system unique ID for the path.  This is as close as
+ * we get to an inode number, but this also contains volume info,
+ * so it is a little stronger.
+ */
+static int lookup_bhfi(wchar_t *wpath,
+		       BY_HANDLE_FILE_INFORMATION *bhfi)
+{
+	DWORD desired_access = FILE_LIST_DIRECTORY;
+	DWORD share_mode =
+		FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;
+	HANDLE hDir;
+
+	hDir = CreateFileW(wpath, desired_access, share_mode, NULL,
+			   OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);
+	if (hDir == INVALID_HANDLE_VALUE) {
+		error(_("[GLE %ld] health thread could not open '%ls'"),
+		      GetLastError(), wpath);
+		return -1;
+	}
+
+	if (!GetFileInformationByHandle(hDir, bhfi)) {
+		error(_("[GLE %ld] health thread getting BHFI for '%ls'"),
+		      GetLastError(), wpath);
+		CloseHandle(hDir);
+		return -1;
+	}
+
+	CloseHandle(hDir);
+	return 0;
+}
+
+/*
+ * Compare the relevant fields from two system unique IDs.
+ * We use this to see if two different handles to the same
+ * path actually refer to the same *instance* of the file
+ * or directory.
+ */
+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,
+		   const BY_HANDLE_FILE_INFORMATION *bhfi_2)
+{
+	return (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&
+		bhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&
+		bhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);
+}
+
+/*
+ * Shutdown if the original worktree root directory been deleted,
+ * moved, or renamed?
+ *
+ * Since the main thread did a "chdir(getenv($HOME))" and our CWD
+ * is not in the worktree root directory and because the listener
+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible
+ * for the root directory to be moved or deleted while we are still
+ * watching it.  We want to detect that here and force a shutdown.
+ *
+ * Granted, a delete MAY cause some operations to fail, such as
+ * GetOverlappedResult(), but it is not guaranteed.  And because
+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the
+ * directory, not changes *ON* the directory, our watch will not
+ * receive a delete event for it.
+ *
+ * A move/rename of the worktree root will also not generate an event.
+ * And since the listener thread already has an open handle, it may
+ * continue to receive events for events within the directory.
+ * However, the pathname of the named-pipe was constructed using the
+ * original location of the worktree root.  (Remember named-pipes are
+ * stored in the NPFS and not in the actual file system.)  Clients
+ * trying to talk to the worktree after the move/rename will not
+ * reach our daemon process, since we're still listening on the
+ * pipe with original path.
+ *
+ * Furthermore, if the user does something like:
+ *
+ *   $ mv repo repo.old
+ *   $ git init repo
+ *
+ * A new daemon cannot be started in the new instance of "repo"
+ * because the named-pipe is still being used by the daemon on
+ * the original instance.
+ *
+ * So, detect move/rename/delete and shutdown.  This should also
+ * handle unsafe drive removal.
+ *
+ * We use the file system unique ID to distinguish the original
+ * directory instance from a new instance and force a shutdown
+ * if the unique ID changes.
+ *
+ * Since a worktree move/rename/delete/unmount doesn't happen
+ * that often (and we can't get an immediate event anyway), we
+ * use a timeout and periodically poll it.
+ */
+static int has_worktree_moved(struct fsmonitor_daemon_state *state,
+			      enum interval_fn_ctx ctx)
+{
+	struct fsm_health_data *data = state->health_data;
+	BY_HANDLE_FILE_INFORMATION bhfi;
+	int r;
+
+	switch (ctx) {
+	case CTX_TERM:
+		return 0;
+
+	case CTX_INIT:
+		if (xutftowcs_path(data->wt_moved.wpath,
+				   state->path_worktree_watch.buf) < 0) {
+			error(_("could not convert to wide characters: '%s'"),
+			      state->path_worktree_watch.buf);
+			return -1;
+		}
+
+		/*
+		 * On the first call we lookup the unique sequence ID for
+		 * the worktree root directory.
+		 */
+		return lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);
+
+	case CTX_TIMER:
+		r = lookup_bhfi(data->wt_moved.wpath, &bhfi);
+		if (r)
+			return r;
+		if (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {
+			error(_("BHFI changed '%ls'"), data->wt_moved.wpath);
+			return -1;
+		}
+		return 0;
+
+	default:
+		die("unhandled case in 'has_worktree_moved': %d",
+		    (int)ctx);
+	}
+
+	return 0;
+}
+
+
 int fsm_health__ctor(struct fsmonitor_daemon_state *state)
 {
 	struct fsm_health_data *data;
@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)
  * A table of the polling functions.
  */
 static interval_fn *table[] = {
+	has_worktree_moved,
 	NULL, /* must be last */
 };
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 18/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (16 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 17/27] fsm-health-win32: force shutdown daemon if worktree root moves Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 19/27] fsmonitor: optimize processing of directory events Jeff Hostetler via GitGitGadget
                     ` (12 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach the listener thread to shutdown the daemon if the spelling of the
worktree root directory changes.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 7c81bc7e5bf..b6c33f2cf3b 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -178,6 +178,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)
 	strbuf_release(&msg);
 }
 
+static int ef_is_root_changed(const FSEventStreamEventFlags ef)
+{
+	return (ef & kFSEventStreamEventFlagRootChanged);
+}
+
 static int ef_is_root_delete(const FSEventStreamEventFlags ef)
 {
 	return (ef & kFSEventStreamEventFlagItemIsDir &&
@@ -287,6 +292,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			continue;
 		}
 
+		if (ef_is_root_changed(event_flags[k])) {
+			/*
+			 * The spelling of the pathname of the root directory
+			 * has changed.  This includes the name of the root
+			 * directory itself or of any parent directory in the
+			 * path.
+			 *
+			 * (There may be other conditions that throw this,
+			 * but I couldn't find any information on it.)
+			 *
+			 * Force a shutdown now and avoid things getting
+			 * out of sync.  The Unix domain socket is inside
+			 * the .git directory and a spelling change will make
+			 * it hard for clients to rendezvous with us.
+			 */
+			trace_printf_key(&trace_fsmonitor,
+					 "event: root changed");
+			goto force_shutdown;
+		}
+
 		if (ef_ignore_xattr(event_flags[k])) {
 			trace_printf_key(&trace_fsmonitor,
 					 "ignore-xattr: '%s', flags=0x%x",
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 19/27] fsmonitor: optimize processing of directory events
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (17 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 18/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 20/27] t7527: FSMonitor tests for directory moves Jeff Hostetler via GitGitGadget
                     ` (11 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to perform binary search over the cache-entries for a directory
notification and then linearly scan forward to find the immediate children.

Previously, when the FSMonitor reported a modified directory Git would
perform a linear search on the entire cache-entry array for all
entries matching that directory prefix and invalidate them.  Since the
cache-entry array is already sorted, we can use a binary search to
find the first matching entry and then only linearly walk forward and
invalidate entries until the prefix changes.

Also, the original code would invalidate anything having the same
directory prefix.  Since a directory event should only be received for
items that are immediately within the directory (and not within
sub-directories of it), only invalidate those entries and not the
whole subtree.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 54 insertions(+), 17 deletions(-)

diff --git a/fsmonitor.c b/fsmonitor.c
index 292a6742b4f..e1229c289cf 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,
 static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
 {
 	int i, len = strlen(name);
-	if (name[len - 1] == '/') {
+	int pos = index_name_pos(istate, name, len);
+
+	trace_printf_key(&trace_fsmonitor,
+			 "fsmonitor_refresh_callback '%s' (pos %d)",
+			 name, pos);
 
+	if (name[len - 1] == '/') {
 		/*
-		 * TODO We should binary search to find the first path with
-		 * TODO this directory prefix.  Then linearly update entries
-		 * TODO while the prefix matches.  Taking care to search without
-		 * TODO the trailing slash -- because '/' sorts after a few
-		 * TODO interesting special chars, like '.' and ' '.
+		 * The daemon can decorate directory events, such as
+		 * moves or renames, with a trailing slash if the OS
+		 * FS Event contains sufficient information, such as
+		 * MacOS.
+		 *
+		 * Use this to invalidate the entire cone under that
+		 * directory.
+		 *
+		 * We do not expect an exact match because the index
+		 * does not normally contain directory entries, so we
+		 * start at the insertion point and scan.
 		 */
+		if (pos < 0)
+			pos = -pos - 1;
 
 		/* Mark all entries for the folder invalid */
-		for (i = 0; i < istate->cache_nr; i++) {
-			if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&
-			    starts_with(istate->cache[i]->name, name))
-				istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
+		for (i = pos; i < istate->cache_nr; i++) {
+			if (!starts_with(istate->cache[i]->name, name))
+				break;
+			istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
 		}
-		/* Need to remove the / from the path for the untracked cache */
+
+		/*
+		 * We need to remove the traling "/" from the path
+		 * for the untracked cache.
+		 */
 		name[len - 1] = '\0';
+	} else if (pos >= 0) {
+		/*
+		 * We have an exact match for this path and can just
+		 * invalidate it.
+		 */
+		istate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;
 	} else {
-		int pos = index_name_pos(istate, name, strlen(name));
-
-		if (pos >= 0) {
-			struct cache_entry *ce = istate->cache[pos];
-			ce->ce_flags &= ~CE_FSMONITOR_VALID;
+		/*
+		 * The path is not a tracked file -or- it is a
+		 * directory event on a platform that cannot
+		 * distinguish between file and directory events in
+		 * the event handler, such as Windows.
+		 *
+		 * Scan as if it is a directory and invalidate the
+		 * cone under it.  (But remember to ignore items
+		 * between "name" and "name/", such as "name-" and
+		 * "name.".
+		 */
+		pos = -pos - 1;
+
+		for (i = pos; i < istate->cache_nr; i++) {
+			if (!starts_with(istate->cache[i]->name, name))
+				break;
+			if ((unsigned char)istate->cache[i]->name[len] > '/')
+				break;
+			if (istate->cache[i]->name[len] == '/')
+				istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
 		}
 	}
 
@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
 	 * Mark the untracked cache dirty even if it wasn't found in the index
 	 * as it could be a new untracked file.
 	 */
-	trace_printf_key(&trace_fsmonitor, "fsmonitor_refresh_callback '%s'", name);
 	untracked_cache_invalidate_path(istate, name, 0);
 }
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 20/27] t7527: FSMonitor tests for directory moves
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (18 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 19/27] fsmonitor: optimize processing of directory events Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 21/27] t/perf/p7527: add perf test for builtin FSMonitor Jeff Hostetler via GitGitGadget
                     ` (10 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create unit tests to move a directory.  Verify that `git status`
gives the same result with and without FSMonitor enabled.

NEEDSWORK: This test exposes a bug in the untracked-cache on
Windows when FSMonitor is disabled.  These are commented out
for the moment.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 39 ++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index c2627f28865..8b63067cbf9 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -231,6 +231,16 @@ test_expect_success 'setup' '
 	trace*
 	EOF
 
+	mkdir -p T1/T2/T3/T4 &&
+	echo 1 >T1/F1 &&
+	echo 1 >T1/T2/F1 &&
+	echo 1 >T1/T2/T3/F1 &&
+	echo 1 >T1/T2/T3/T4/F1 &&
+	echo 2 >T1/F2 &&
+	echo 2 >T1/T2/F2 &&
+	echo 2 >T1/T2/T3/F2 &&
+	echo 2 >T1/T2/T3/T4/F2 &&
+
 	git -c core.fsmonitor=false add . &&
 	test_tick &&
 	git -c core.fsmonitor=false commit -m initial &&
@@ -324,6 +334,19 @@ verify_status () {
 	echo HELLO AFTER
 }
 
+move_directory_contents_deeper() {
+	mkdir T1/_new_
+	mv T1/[A-Z]* T1/_new_
+}
+
+move_directory_up() {
+	mv T1/T2/T3 T1
+}
+
+move_directory() {
+	mv T1/T2/T3 T1/T2/NewT3
+}
+
 # The next few test cases confirm that our fsmonitor daemon sees each type
 # of OS filesystem notification that we care about.  At this layer we just
 # ensure we are getting the OS notifications and do not try to confirm what
@@ -655,6 +678,22 @@ do
 		matrix_try $uc_val $fsm_val file_to_directory
 		matrix_try $uc_val $fsm_val directory_to_file
 
+		# NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor
+		# is DISABLED.  Turn off a few test that cause it problems until
+		# we can debug it.
+		#
+		try_moves="true"
+		test_have_prereq UNTRACKED_CACHE,WINDOWS && \
+			test $uc_val = true && \
+			test $fsm_val = false && \
+			try_moves="false"
+		if test $try_moves = true
+		then
+			matrix_try $uc_val $fsm_val move_directory_contents_deeper
+			matrix_try $uc_val $fsm_val move_directory_up
+			matrix_try $uc_val $fsm_val move_directory
+		fi
+
 		if test $fsm_val = true
 		then
 			test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end" '
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 21/27] t/perf/p7527: add perf test for builtin FSMonitor
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (19 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 20/27] t7527: FSMonitor tests for directory moves Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 22/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules Jeff Hostetler via GitGitGadget
                     ` (9 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++
 1 file changed, 257 insertions(+)
 create mode 100755 t/perf/p7527-builtin-fsmonitor.sh

diff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh
new file mode 100755
index 00000000000..9338b9ea008
--- /dev/null
+++ b/t/perf/p7527-builtin-fsmonitor.sh
@@ -0,0 +1,257 @@
+#!/bin/sh
+
+test_description="Perf test for the builtin FSMonitor"
+
+. ./perf-lib.sh
+
+if ! test_have_prereq FSMONITOR_DAEMON
+then
+	skip_all="fsmonitor--daemon is not supported on this platform"
+	test_done
+fi
+
+test_lazy_prereq UNTRACKED_CACHE '
+	{ git update-index --test-untracked-cache; ret=$?; } &&
+	test $ret -ne 1
+'
+
+# Lie to perf-lib and ask for a new empty repo and avoid
+# the complaints about GIT_PERF_REPO not being big enough
+# the perf hit when GIT_PERF_LARGE_REPO is copied into
+# the trash directory.
+#
+# NEEDSWORK: It would be nice if perf-lib had an option to
+# "borrow" an existing large repo (especially for gigantic
+# monorepos) and use it in-place.  For now, fake it here.
+#
+test_perf_fresh_repo
+
+
+# Use a generated synthetic monorepo.  If it doesn't exist, we will
+# generate it.  If it does exist, we will put it in a known state
+# before we start our timings.
+#
+PARAM_D=5
+PARAM_W=10
+PARAM_F=9
+
+PARAMS="$PARAM_D"."$PARAM_W"."$PARAM_F"
+
+BALLAST_BR=p0006-ballast
+export BALLAST_BR
+
+TMP_BR=tmp_br
+export TMP_BR
+
+REPO=../repos/gen-many-files-"$PARAMS".git
+export REPO
+
+if ! test -d $REPO
+then
+	(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)
+fi
+
+
+enable_uc () {
+	git -C $REPO config core.untrackedcache true
+	git -C $REPO update-index --untracked-cache
+	git -C $REPO status >/dev/null 2>&1
+}
+
+disable_uc () {
+	git -C $REPO config core.untrackedcache false
+	git -C $REPO update-index --no-untracked-cache
+	git -C $REPO status >/dev/null 2>&1
+}
+
+start_fsm () {
+	git -C $REPO fsmonitor--daemon start
+	git -C $REPO fsmonitor--daemon status
+	git -C $REPO config core.fsmonitor true
+	git -C $REPO update-index --fsmonitor
+	git -C $REPO status >/dev/null 2>&1
+}
+
+stop_fsm () {
+	git -C $REPO config --unset core.fsmonitor
+	git -C $REPO update-index --no-fsmonitor
+	test_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null
+	git -C $REPO status >/dev/null 2>&1
+}
+
+
+# Ensure that FSMonitor is turned off on the borrowed repo.
+#
+test_expect_success "Setup borrowed repo (fsm+uc)" "
+	stop_fsm &&
+	disable_uc
+"
+
+# Also ensure that it starts in a known state.
+#
+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time
+# the ballast checkout, since only the first invocation does any work and the
+# subsequent ones just print "already on branch" and quit, so the reported
+# time is not useful.
+#
+# Create a temp branch and do all work relative to it so that we don't
+# accidentially alter the real ballast branch.
+#
+test_expect_success "Setup borrowed repo (temp ballast branch)" "
+	test_might_fail git -C $REPO checkout $BALLAST_BR &&
+	test_might_fail git -C $REPO reset --hard &&
+	git -C $REPO clean -d -f &&
+	test_might_fail git -C $REPO branch -D $TMP_BR &&
+	git -C $REPO branch $TMP_BR $BALLAST_BR &&
+	git -C $REPO checkout $TMP_BR
+"
+
+
+echo Data >data.txt
+
+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1.  With
+# FSMonitor enabled, we can get a skewed view of status times, since
+# the index MAY (or may not) be updated after the first invocation
+# which will update the FSMonitor Token, so the subsequent invocations
+# may get a smaller response from the daemon.
+#
+do_status () {
+	msg=$1
+
+	test_perf "$msg" "
+		git -C $REPO status >/dev/null 2>&1
+	"
+}
+
+do_matrix () {
+	uc=$1
+	fsm=$2
+
+	t="[uc $uc][fsm $fsm]"
+	MATRIX_BR="$TMP_BR-$uc-$fsm"
+
+	test_expect_success "$t Setup matrix branch" "
+		git -C $REPO clean -d -f &&
+		git -C $REPO checkout $TMP_BR &&
+		test_might_fail git -C $REPO branch -D $MATRIX_BR &&
+		git -C $REPO branch $MATRIX_BR $TMP_BR &&
+		git -C $REPO checkout $MATRIX_BR
+	"
+
+	if test $uc = true
+	then
+		enable_uc
+	else
+		disable_uc
+	fi
+
+	if test $fsm = true
+	then
+		start_fsm
+	else
+		stop_fsm
+	fi
+
+	do_status "$t status after checkout"
+
+	# Modify many files in the matrix branch.
+	# Stage them.
+	# Commit them.
+	# Rollback.
+	#
+	test_expect_success "$t modify tracked files" "
+		find $REPO -name file1 -exec cp data.txt {} \\;
+	"
+
+	do_status "$t status after big change"
+
+	# Don't bother timing the "add" because _REPEAT_COUNT
+	# issue described above.
+	#
+	test_expect_success "$t add all" "
+		git -C $REPO add -A
+	"
+
+	do_status "$t status after add all"
+
+	test_expect_success "$t add dot" "
+		git -C $REPO add .
+	"
+
+	do_status "$t status after add dot"
+
+	test_expect_success "$t commit staged" "
+		git -C $REPO commit -a -m data
+	"
+
+	do_status "$t status after commit"
+
+	test_expect_success "$t reset HEAD~1 hard" "
+		git -C $REPO reset --hard HEAD~1 >/dev/null 2>&1
+	"
+
+	do_status "$t status after reset hard"
+
+	# Create some untracked files.
+	#
+	test_expect_success "$t create untracked files" "
+		cp -R $REPO/ballast/dir1 $REPO/ballast/xxx1
+	"
+
+	do_status "$t status after create untracked files"
+
+	# Remove the new untracked files.
+	#
+	test_expect_success "$t clean -df" "
+		git -C $REPO clean -d -f
+	"
+
+	do_status "$t status after clean"
+
+	if test $fsm = true
+	then
+		stop_fsm
+	fi
+}
+
+# Begin testing each case in the matrix that we care about.
+#
+uc_values="false"
+test_have_prereq UNTRACKED_CACHE && uc_values="false true"
+
+fsm_values="false true"
+
+for uc_val in $uc_values
+do
+	for fsm_val in $fsm_values
+	do
+		do_matrix $uc_val $fsm_val
+	done
+done
+
+cleanup () {
+	uc=$1
+	fsm=$2
+
+	MATRIX_BR="$TMP_BR-$uc-$fsm"
+
+	test_might_fail git -C $REPO branch -D $MATRIX_BR
+}
+
+
+# We're borrowing this repo.  We should leave it in a clean state.
+#
+test_expect_success "Cleanup temp and matrix branches" "
+	git -C $REPO clean -d -f &&
+	test_might_fail git -C $REPO checkout $BALLAST_BR &&
+	test_might_fail git -C $REPO branch -D $TMP_BR &&
+	for uc_val in $uc_values
+	do
+		for fsm_val in $fsm_values
+		do
+			cleanup $uc_val $fsm_val
+		done
+	done
+"
+
+test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 22/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (20 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 21/27] t/perf/p7527: add perf test for builtin FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 23/27] t7527: test FSMonitor on case insensitive+preserving file system Jeff Hostetler via GitGitGadget
                     ` (8 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Never set CE_FSMONITOR_VALID on the cache-entry of submodule
directories.

During a client command like 'git status', we may need to recurse
into each submodule to compute a status summary for the submodule.
Since the purpose of the ce_flag is to let Git avoid scanning a
cache-entry, setting the flag causes the recursive call to be
avoided and we report incorrect (no status) for the submodule.

We created an OS watch on the root directory of our working
directory and we receive events for everything in the cone
under it.  When submodules are present inside our working
directory, we receive events for both our repo (the super) and
any subs within it.  Since our index doesn't have any information
for items within the submodules, we can't use those events.

We could try to truncate the paths of those events back to the
submodule boundary and mark the GITLINK as dirty, but that
feels expensive since we would have to prefix compare every FS
event that we receive against a list of submodule roots.  And
it still wouldn't be sufficient to correctly report status on
the submodule, since we don't have any space in the cache-entry
to cache the submodule's status (the 'SCMU' bits in porcelain
V2 speak).  That is, the CE_FSMONITOR_VALID bit just says that
we don't need to scan/inspect it because we already know the
answer -- it doesn't say that the item is clean -- and we
don't have space in the cache-entry to store those answers.
So we should always do the recursive scan.

Therefore, we should never set the flag on GITLINK cache-entries.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 fsmonitor.c                  |  2 +
 fsmonitor.h                  | 11 +++++
 t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++
 3 files changed, 106 insertions(+)

diff --git a/fsmonitor.c b/fsmonitor.c
index e1229c289cf..57d6a483bee 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)
 		if (fsmonitor_enabled) {
 			/* Mark all entries valid */
 			for (i = 0; i < istate->cache_nr; i++) {
+				if (S_ISGITLINK(istate->cache[i]->ce_mode))
+					continue;
 				istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
 			}
 
diff --git a/fsmonitor.h b/fsmonitor.h
index 3f41f653691..edf7ce5203b 100644
--- a/fsmonitor.h
+++ b/fsmonitor.h
@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)
  * Set the given cache entries CE_FSMONITOR_VALID bit. This should be
  * called any time the cache entry has been updated to reflect the
  * current state of the file on disk.
+ *
+ * However, never mark submodules as valid.  When commands like "git
+ * status" run they might need to recurse into the submodule (using a
+ * child process) to get a summary of the submodule state.  We don't
+ * have (and don't want to create) the facility to translate every
+ * FS event that we receive and that happens to be deep inside of a
+ * submodule back to the submodule root, so we cannot correctly keep
+ * track of this bit on the gitlink directory.  Therefore, we never
+ * set it on submodules.
  */
 static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)
 {
@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache
 
 	if (fsm_mode > FSMONITOR_MODE_DISABLED &&
 	    !(ce->ce_flags & CE_FSMONITOR_VALID)) {
+		if (S_ISGITLINK(ce->ce_mode))
+			return;
 		istate->cache_changed = 1;
 		ce->ce_flags |= CE_FSMONITOR_VALID;
 		trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name);
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 8b63067cbf9..a18e077d375 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -728,4 +728,97 @@ do
 	'
 done
 
+# Test fsmonitor interaction with submodules.
+#
+# If we start the daemon in the super, it will see FS events for
+# everything in the working directory cone and this includes any
+# files/directories contained *within* the submodules.
+#
+# A `git status` at top level will get events for items within the
+# submodule and ignore them, since they aren't named in the index
+# of the super repo.  This makes the fsmonitor response a little
+# noisy, but it doesn't alter the correctness of the state of the
+# super-proper.
+#
+# When we have submodules, `git status` normally does a recursive
+# status on each of the submodules and adds a summary row for any
+# dirty submodules.  (See the "S..." bits in porcelain V2 output.)
+#
+# It is therefore important that the top level status not be tricked
+# by the FSMonitor response to skip those recursive calls.
+
+my_match_and_clean () {
+	git -C super --no-optional-locks status --porcelain=v2 >actual.with &&
+	git -C super --no-optional-locks -c core.fsmonitor=false \
+		status --porcelain=v2 >actual.without &&
+	test_cmp actual.with actual.without &&
+
+	git -C super/dir_1/dir_2/sub reset --hard &&
+	git -C super/dir_1/dir_2/sub clean -d -f
+}
+
+test_expect_success "Submodule" '
+	test_when_finished "git -C super fsmonitor--daemon stop" &&
+
+	git init "super" &&
+	echo x >super/file_1 &&
+	echo y >super/file_2 &&
+	echo z >super/file_3 &&
+	mkdir super/dir_1 &&
+	echo a >super/dir_1/file_11 &&
+	echo b >super/dir_1/file_12 &&
+	mkdir super/dir_1/dir_2 &&
+	echo a >super/dir_1/dir_2/file_21 &&
+	echo b >super/dir_1/dir_2/file_22 &&
+	git -C super add . &&
+	git -C super commit -m "initial super commit" &&
+
+	git init "sub" &&
+	echo x >sub/file_x &&
+	echo y >sub/file_y &&
+	echo z >sub/file_z &&
+	mkdir sub/dir_x &&
+	echo a >sub/dir_x/file_a &&
+	echo b >sub/dir_x/file_b &&
+	mkdir sub/dir_x/dir_y &&
+	echo a >sub/dir_x/dir_y/file_a &&
+	echo b >sub/dir_x/dir_y/file_b &&
+	git -C sub add . &&
+	git -C sub commit -m "initial sub commit" &&
+
+	git -C super submodule add ../sub ./dir_1/dir_2/sub &&
+	git -C super commit -m "add sub" &&
+
+	start_daemon super &&
+	git -C super config core.fsmonitor true &&
+	git -C super update-index --fsmonitor &&
+	git -C super status &&
+
+	# Now run pairs of commands w/ and w/o FSMonitor while we make
+	# some dirt in the submodule and confirm matching output.
+
+	# Completely clean status.
+	my_match_and_clean &&
+
+	# .M S..U
+	echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&
+	my_match_and_clean &&
+
+	# .M S.M.
+	echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&
+	git -C super/dir_1/dir_2/sub add . &&
+	my_match_and_clean &&
+
+	# .M S.M.
+	echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&
+	git -C super/dir_1/dir_2/sub add . &&
+	my_match_and_clean &&
+
+	# .M SC..
+	echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&
+	git -C super/dir_1/dir_2/sub add . &&
+	git -C super/dir_1/dir_2/sub commit -m "SC.." &&
+	my_match_and_clean
+'
+
 test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 23/27] t7527: test FSMonitor on case insensitive+preserving file system
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (21 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 22/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 24/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname Jeff Hostetler via GitGitGadget
                     ` (7 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Test that FS events from the OS are received using the preserved,
on-disk spelling of files/directories rather than spelling used
to make the change.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 40 ++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index a18e077d375..a41e37236b5 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -821,4 +821,44 @@ test_expect_success "Submodule" '
 	my_match_and_clean
 '
 
+# On a case-insensitive file system, confirm that the daemon
+# notices when the .git directory is moved/renamed/deleted
+# regardless of how it is spelled in the the FS event.
+# That is, does the FS event receive the spelling of the
+# operation or does it receive the spelling preserved with
+# the file/directory.
+#
+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '
+#	test_when_finished "stop_daemon_delete_repo test_insensitive" &&
+
+	git init test_insensitive &&
+	(
+		GIT_TRACE_FSMONITOR="$(pwd)/insensitive.trace" &&
+		export GIT_TRACE_FSMONITOR &&
+
+		start_daemon test_insensitive
+	) &&
+
+	mkdir -p test_insensitive/abc/def &&
+	echo xyz >test_insensitive/ABC/DEF/xyz &&
+
+	test_path_is_dir test_insensitive/.git &&
+	test_path_is_dir test_insensitive/.GIT &&
+
+	# Rename .git using an alternate spelling to verify that that
+	# daemon detects it and automatically shuts down.
+	mv test_insensitive/.GIT test_insensitive/.FOO &&
+	sleep 1 &&
+	mv test_insensitive/.FOO test_insensitive/.git &&
+	test_must_fail git -C test_insensitive fsmonitor--daemon status &&
+
+	# Verify that events were reported using on-disk spellings of the
+	# directories and files that we touched.  We may or may not get a
+	# trailing slash on modified directories.
+	#
+	egrep "^event: abc/?$"       ./insensitive.trace &&
+	egrep "^event: abc/def/?$"   ./insensitive.trace &&
+	egrep "^event: abc/def/xyz$" ./insensitive.trace
+'
+
 test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 24/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (22 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 23/27] t7527: test FSMonitor on case insensitive+preserving file system Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Jeff Hostetler via GitGitGadget
                     ` (6 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Emit NFC or NFC and NFD spellings of pathnames on macOS.

MacOS is Unicode composition insensitive, so NFC and NFD spellings are
treated as aliases and collide.  While the spelling of pathnames in
filesystem events depends upon the underlying filesystem, such as
APFS, HFS+ or FAT32, the OS enforces such collisions regardless of
filesystem.

Teach the daemon to always report the NFC spelling and to report
the NFD spelling when stored in that format on the disk.

This is slightly more general than "core.precomposeUnicode".

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--
 1 file changed, 31 insertions(+), 2 deletions(-)

diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index b6c33f2cf3b..3332d3b7792 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -227,6 +227,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)
 	return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);
 }
 
+/*
+ * On MacOS we have to adjust for Unicode composition insensitivity
+ * (where NFC and NFD spellings are not respected).  The different
+ * spellings are essentially aliases regardless of how the path is
+ * actually stored on the disk.
+ *
+ * This is related to "core.precomposeUnicode" (which wants to try
+ * to hide NFD completely and treat everything as NFC).  Here, we
+ * don't know what the value the client has (or will have) for this
+ * config setting when they make a query, so assume the worst and
+ * emit both when the OS gives us an NFD path.
+ */
+static void my_add_path(struct fsmonitor_batch *batch, const char *path)
+{
+	char *composed;
+
+	/* add the NFC or NFD path as received from the OS */
+	fsmonitor_batch__add_path(batch, path);
+
+	/* if NFD, also add the corresponding NFC spelling */
+	composed = (char *)precompose_string_if_needed(path);
+	if (!composed || composed == path)
+		return;
+
+	fsmonitor_batch__add_path(batch, composed);
+	free(composed);
+}
+
+
 static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			     void *ctx,
 			     size_t num_of_events,
@@ -377,7 +406,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 
 				if (!batch)
 					batch = fsmonitor_batch__new();
-				fsmonitor_batch__add_path(batch, rel);
+				my_add_path(batch, rel);
 			}
 
 			if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {
@@ -390,7 +419,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 
 				if (!batch)
 					batch = fsmonitor_batch__new();
-				fsmonitor_batch__add_path(batch, tmp.buf);
+				my_add_path(batch, tmp.buf);
 			}
 
 			break;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (23 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 24/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-09 18:40     ` Derrick Stolee
  2022-03-08 22:15   ` [PATCH v2 26/27] t7527: test Unicode NFC/NFD handling on MacOS Jeff Hostetler via GitGitGadget
                     ` (5 subsequent siblings)
  30 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create a set of prereqs to help understand how file names
are handled by the filesystem when they contain NFC and NFD
Unicode characters.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/lib-unicode-nfc-nfd.sh | 159 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 159 insertions(+)
 create mode 100755 t/lib-unicode-nfc-nfd.sh

diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh
new file mode 100755
index 00000000000..a09e910c302
--- /dev/null
+++ b/t/lib-unicode-nfc-nfd.sh
@@ -0,0 +1,159 @@
+# Help detect how Unicode NFC and NFD are handled on the filesystem.
+
+# A simple character that has a NFD form.
+#
+# NFC:       U+00e9 LATIN SMALL LETTER E WITH ACUTE
+# UTF8(NFC): \xc3 \xa9
+#
+# NFD:       U+0065 LATIN SMALL LETTER E
+#            U+0301 COMBINING ACUTE ACCENT
+# UTF8(NFD): \x65  +  \xcc \x81
+#
+utf8_nfc=$(printf "\xc3\xa9")
+utf8_nfd=$(printf "\x65\xcc\x81")
+
+# Is the OS or the filesystem "Unicode composition sensitive"?
+#
+# That is, does the OS or the filesystem allow files to exist with
+# both the NFC and NFD spellings?  Or, does the OS/FS lie to us and
+# tell us that the NFC and NFD forms are equivalent.
+#
+# This is or may be independent of what type of filesystem we have,
+# since it might be handled by the OS at a layer above the FS.
+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a
+# collision, for example.
+#
+# This does not tell us how the Unicode pathname will be spelled
+# on disk, but rather only that the two spelling "collide".  We
+# will examine the actual on disk spelling in a later prereq.
+#
+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '
+	mkdir trial_${utf8_nfc} &&
+	mkdir trial_${utf8_nfd}
+'
+
+# Is the spelling of an NFC pathname preserved on disk?
+#
+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD
+# and on APFS, NFC paths are preserved.  As we have established
+# above, this is independent of "composition sensitivity".
+#
+# 0000000 63 5f c3 a9
+#
+# (/usr/bin/od output contains different amount of whitespace
+# on different platforms, so we need the wildcards here.)
+#
+test_lazy_prereq UNICODE_NFC_PRESERVED '
+	mkdir c_${utf8_nfc} &&
+	ls | od -t x1 | grep "63 *5f *c3 *a9"
+'
+
+# Is the spelling of an NFD pathname preserved on disk?
+#
+# 0000000 64 5f 65 cc 81
+#
+test_lazy_prereq UNICODE_NFD_PRESERVED '
+	mkdir d_${utf8_nfd} &&
+	ls | od -t x1 | grep "64 *5f *65 *cc *81"
+'
+	mkdir c_${utf8_nfc} &&
+	mkdir d_${utf8_nfd} &&
+
+# The following _DOUBLE_ forms are more for my curiosity,
+# but there may be quirks lurking when there are multiple
+# combining characters in non-canonical order.
+
+# Unicode also allows multiple combining characters
+# that can be decomposed in pieces.
+#
+# NFC:        U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI
+# UTF8(NFC):  \xe1 \xbd \xa7
+#
+# NFD1:       U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA
+#             U+0342 COMBINING GREEK PERISPOMENI
+# UTF8(NFD1): \xe1 \xbd \xa1  +  \xcd \x82
+#
+# But U+1f61 decomposes into
+# NFD2:       U+03c9 GREEK SMALL LETTER OMEGA
+#             U+0314 COMBINING REVERSED COMMA ABOVE
+# UTF8(NFD2): \xcf \x89  +  \xcc \x94
+#
+# Yielding:   \xcf \x89  +  \xcc \x94  +  \xcd \x82
+#
+# Note that I've used the canonical ordering of the
+# combinining characters.  It is also possible to
+# swap them.  My testing shows that that non-standard
+# ordering also causes a collision in mkdir.  However,
+# the resulting names don't draw correctly on the
+# terminal (implying that the on-disk format also has
+# them out of order).
+#
+greek_nfc=$(printf "\xe1\xbd\xa7")
+greek_nfd1=$(printf "\xe1\xbd\xa1\xcd\x82")
+greek_nfd2=$(printf "\xcf\x89\xcc\x94\xcd\x82")
+
+# See if a double decomposition also collides.
+#
+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '
+	mkdir trial_${greek_nfc} &&
+	mkdir trial_${greek_nfd2}
+'
+
+# See if the NFC spelling appears on the disk.
+#
+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '
+	mkdir c_${greek_nfc} &&
+	ls | od -t x1 | grep "63 *5f *e1 *bd *a7"
+'
+
+# See if the NFD spelling appears on the disk.
+#
+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '
+	mkdir d_${greek_nfd2} &&
+	ls | od -t x1 | grep "64 *5f *cf *89 *cc *94 *cd *82"
+'
+
+if test $unicode_debug = 1
+then
+	if test_have_prereq UNICODE_COMPOSITION_SENSITIVE
+	then
+		echo NFC and NFD are distinct on this OS/filesystem.
+	else
+		echo NFC and NFD are aliases on this OS/filesystem.
+	fi
+
+	if test_have_prereq UNICODE_NFC_PRESERVED
+	then
+		echo NFC maintains original spelling.
+	else
+		echo NFC is modified.
+	fi
+
+	if test_have_prereq UNICODE_NFD_PRESERVED
+	then
+		echo NFD maintains original spelling.
+	else
+		echo NFD is modified.
+	fi
+
+	if test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE
+	then
+		echo DOUBLE NFC and NFD are distinct on this OS/filesystem.
+	else
+		echo DOUBLE NFC and NFD are aliases on this OS/filesystem.
+	fi
+
+	if test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED
+	then
+		echo Double NFC maintains original spelling.
+	else
+		echo Double NFC is modified.
+	fi
+
+	if test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED
+	then
+		echo Double NFD maintains original spelling.
+	else
+		echo Double NFD is modified.
+	fi
+fi
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 26/27] t7527: test Unicode NFC/NFD handling on MacOS
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (24 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-08 22:15   ` [PATCH v2 27/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible Jeff Hostetler via GitGitGadget
                     ` (4 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Confirm that the daemon reports events using the on-disk
spelling for Unicode NFC/NFD characters.  On APFS we still
have Unicode aliasing, so we cannot create two files that
only differ by NFC/NFD, but the on-disk format preserves
the spelling used to create the file.  On HFS+ we also
have aliasing, but the path is always stored on disk in
NFD.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 54 ++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index a41e37236b5..48c9125d8da 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -861,4 +861,58 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '
 	egrep "^event: abc/def/xyz$" ./insensitive.trace
 '
 
+unicode_debug=0
+. "$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh"
+
+# See if the OS or filesystem does NFC/NFD aliasing/munging.
+#
+# The daemon should err on the side of caution and send BOTH the
+# NFC and NFD forms.  It does not know the original spelling of
+# the pathname (how the user thinks it should be spelled), so
+# emit both and let the client decide (when necessary).  This is
+# similar to "core.precomposeUnicode".
+#
+test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' '
+	test_when_finished "stop_daemon_delete_repo test_unicode" &&
+
+	git init test_unicode &&
+	(
+		GIT_TRACE_FSMONITOR="$(pwd)/unicode.trace" &&
+		export GIT_TRACE_FSMONITOR &&
+
+		start_daemon test_unicode
+	) &&
+
+	# Create a directory using an NFC spelling.
+	#
+	mkdir test_unicode/nfc &&
+	mkdir test_unicode/nfc/c_${utf8_nfc} &&
+
+	# Create a directory using an NFD spelling.
+	#
+	mkdir test_unicode/nfd &&
+	mkdir test_unicode/nfd/d_${utf8_nfd} &&
+
+	git -C test_unicode fsmonitor--daemon stop &&
+
+	if test_have_prereq UNICODE_NFC_PRESERVED
+	then
+		# We should have seen NFC event from OS.
+		# We should not have synthesized an NFD event.
+		egrep    "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace &&
+		egrep -v "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace
+	else
+		# We should have seen NFD event from OS.
+		# We should have synthesized an NFC event.
+		egrep "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace &&
+		egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace
+	fi &&
+
+	# We assume UNICODE_NFD_PRESERVED.
+	# We should have seen explicit NFD from OS.
+	# We should have synthesized an NFC event.
+	egrep "^event: nfd/d_${utf8_nfd}/?$" ./unicode.trace &&
+	egrep "^event: nfd/d_${utf8_nfc}/?$" ./unicode.trace
+'
+
 test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v2 27/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (25 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 26/27] t7527: test Unicode NFC/NFD handling on MacOS Jeff Hostetler via GitGitGadget
@ 2022-03-08 22:15   ` Jeff Hostetler via GitGitGadget
  2022-03-09 18:48   ` [PATCH v2 00/27] Builtin FSMonitor Part 3 Derrick Stolee
                     ` (3 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-08 22:15 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

On MacOS mark repos on NTFS or FAT32 volumes as incompatible.

The builtin FSMonitor used Unix domain sockets on MacOS for IPC
with clients.  These sockets are kept in the .git directory.
Unix sockets are not supported by NTFS and FAT32, so the daemon
cannot start up.

Test for this during our compatibility checking so that client
commands do not keep trying to start the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---
 fsmonitor-settings.c                   |  3 +++
 fsmonitor-settings.h                   |  1 +
 3 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index fdd762bf79d..efc732c0f31 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -7,7 +7,7 @@
 #include <sys/mount.h>
 
 /*
- * Remote working directories are problematic for FSMonitor.
+ * [1] Remote working directories are problematic for FSMonitor.
  *
  * The underlying file system on the server machine and/or the remote
  * mount type (NFS, SAMBA, etc.) dictates whether notification events
@@ -40,8 +40,16 @@
  *
  * So (for now at least), mark remote working directories as
  * incompatible.
+ *
+ *
+ * [2] FAT32 and NTFS working directories are problematic too.
+ *
+ * The builtin FSMonitor uses a Unix domain socket in the .git
+ * directory for IPC.  These Windows drive formats do not support
+ * Unix domain sockets, so mark them as incompatible for the daemon.
+ *
  */
-static enum fsmonitor_reason check_remote(struct repository *r)
+static enum fsmonitor_reason check_volume(struct repository *r)
 {
 	struct statfs fs;
 
@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)
 	if (!(fs.f_flags & MNT_LOCAL))
 		return FSMONITOR_REASON_REMOTE;
 
+	if (!strcmp(fs.f_fstypename, "msdos")) /* aka FAT32 */
+		return FSMONITOR_REASON_NOSOCKETS;
+
+	if (!strcmp(fs.f_fstypename, "ntfs"))
+		return FSMONITOR_REASON_NOSOCKETS;
+
 	return FSMONITOR_REASON_OK;
 }
 
@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
 	enum fsmonitor_reason reason;
 
-	reason = check_remote(r);
+	reason = check_volume(r);
 	if (reason != FSMONITOR_REASON_OK)
 		return reason;
 
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 7ff3f98964d..5734c93baf9 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -186,6 +186,9 @@ const char *fsm_settings__get_reason_msg(struct repository *r)
 
 	case FSMONITOR_REASON_VFS4GIT:
 		return _("virtual repos are incompatible with fsmonitor");
+
+	case FSMONITOR_REASON_NOSOCKETS:
+		return _("repo filesystem does not support Unix sockets");
 	}
 
 	BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index 6aa9a00379b..af792313413 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -19,6 +19,7 @@ enum fsmonitor_reason {
 	FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */
 	FSMONITOR_REASON_REMOTE,
 	FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */
+	FSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */
 };
 
 void fsm_settings__set_ipc(struct repository *r);
-- 
gitgitgadget

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH 00/23] Builtin FSMonitor Part 3
  2022-03-07 21:23   ` Jeff Hostetler
@ 2022-03-09 15:34     ` Derrick Stolee
  0 siblings, 0 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-03-09 15:34 UTC (permalink / raw)
  To: Jeff Hostetler, Jeff Hostetler via GitGitGadget, git; +Cc: Jeff Hostetler

On 3/7/2022 4:23 PM, Jeff Hostetler wrote:
> 
> 
> On 2/24/22 11:21 AM, Derrick Stolee wrote:
>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>>> Here is part 3 of my builtin FSMonitor series.
> [...]
>>> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a
>>> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of
>>> the untracked cache (uc) and FSMonitor (fsm) on git status.
>>>
>>> $ ./p7527-builtin-fsmonitor.sh
>>> # passed all 67 test(s)
>>> 1..67
>>> Test                                                                 this tree
>>> ---------------------------------------------------------------------------------------
>>> 7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12)
>>> 7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24)
>>> 7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47)
>>> 7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05)
>>> 7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)
>>> 7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31)
>>> 7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75)
>>> 7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36)
>>>
>>> 7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84)
>>> 7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86)
>>> 7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70)
>>> 7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10)
>>> 7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91)
>>> 7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89)
>>> 7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71)
>>> 7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78)
>>>
>>> 7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11)
>>> 7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)
>>> 7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96)
>>> 7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72)
>>> 7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25)
>>> 7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68)
>>> 7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99)
>>> 7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19)
>>>
>>> 7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)
>>> 7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64)
>>> 7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92)
>>> 7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)
>>> 7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)
>>> 7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49)
>>> 7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)
>>> 7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)
>>
>> The other stylistic thing is this performance test. It would be nice if
>> these tests were grouped by the operation (like "status after checkout")
>> so it is easier to compare the same operation across the matrix definitions.
>>
>> This would require reordering the test definition as well as allowing the
>> different cases to simultaneously live in different repositories. The
>> p2000-sparse-operations.sh has this kind of organization, but you'll need
>> more involved test cases than "run this command".
> 
> Yeah, it would be nice to turn this test inside-out so that
> could group the outputs by test case rather than by (uc,fsm)
> combination.  That would certainly make it easier to see how
> the two terms affect things.
> 
> The problem is I'd either need 4 parallel repos that I could
> setup with each (uc,fsm) pair or I'd need to start/stop the
> daemon and swap out the {.git/index, .git/config} between
> each step.  The former is a problem for monorepos.  The latter
> is doable, but I'm not sure it is worth the effort right now.

Yeah, that extra data is not something to take lightly.

Reworking this test is not critical right now, so I'll leave it
for another time, if we need it. But, perhaps we could set up
the test to create four _worktrees_ so at least the only
duplication is the working tree and not the entire repository.

(The working tree is still a huge amount of data in some cases,
so this doesn't actually solve the problem, just reduces it.)

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  2022-03-08 22:15   ` [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Jeff Hostetler via GitGitGadget
@ 2022-03-09 18:40     ` Derrick Stolee
  2022-03-09 18:42       ` Derrick Stolee
  2022-03-10 14:23       ` Jeff Hostetler
  0 siblings, 2 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-03-09 18:40 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git
  Cc: Jeff Hostetler, Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler

On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <jeffhost@microsoft.com>
> 
> Create a set of prereqs to help understand how file names
> are handled by the filesystem when they contain NFC and NFD
> Unicode characters.

Prereqs look good and are well documented.

> +if test $unicode_debug = 1

Is this $unicode_debug something I should know from a previous
patch? or is it a leftover from local debugging?

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  2022-03-09 18:40     ` Derrick Stolee
@ 2022-03-09 18:42       ` Derrick Stolee
  2022-03-10 14:28         ` Jeff Hostetler
  2022-03-10 14:23       ` Jeff Hostetler
  1 sibling, 1 reply; 277+ messages in thread
From: Derrick Stolee @ 2022-03-09 18:42 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git
  Cc: Jeff Hostetler, Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler

On 3/9/2022 1:40 PM, Derrick Stolee wrote:
> On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Create a set of prereqs to help understand how file names
>> are handled by the filesystem when they contain NFC and NFD
>> Unicode characters.
> 
> Prereqs look good and are well documented.
> 
>> +if test $unicode_debug = 1
> 
> Is this $unicode_debug something I should know from a previous
> patch? or is it a leftover from local debugging?

I see that you set unicode_debug = 0 in a later patch, but I
suppose that we might want this output no matter what. Or, do
we think it will interrupt the output parsing of 'prove' and
other tools?

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (26 preceding siblings ...)
  2022-03-08 22:15   ` [PATCH v2 27/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible Jeff Hostetler via GitGitGadget
@ 2022-03-09 18:48   ` Derrick Stolee
  2022-03-10  5:31   ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
                     ` (2 subsequent siblings)
  30 siblings, 0 replies; 277+ messages in thread
From: Derrick Stolee @ 2022-03-09 18:48 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget, git
  Cc: Jeff Hostetler, Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler

On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:
> Here is V2 of part 3 of my builtin FSMonitor series.
> 
> I think I have addressed all of the feedback from V1. This includes:

> Range-diff vs v1:

>   -:  ----------- > 22:  524d449ed64 fsmonitor: never set CE_FSMONITOR_VALID on submodules
>   -:  ----------- > 24:  95b9d4210d2 fsmonitor: on macOS also emit NFC spelling for NFD pathname
>   -:  ----------- > 25:  5a0c1b7a287 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
>   -:  ----------- > 26:  a45c1fd3000 t7527: test Unicode NFC/NFD handling on MacOS
>   -:  ----------- > 27:  e3e01677d93 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible

I looked closely through the range-diff for the edits, then looked at
these new patches closely. Outside of one thought about some debug
output, I'm happy with this version.

Thanks,
-Stolee

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (27 preceding siblings ...)
  2022-03-09 18:48   ` [PATCH v2 00/27] Builtin FSMonitor Part 3 Derrick Stolee
@ 2022-03-10  5:31   ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
  2022-03-13 10:42   ` Torsten Bögershausen
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
  30 siblings, 0 replies; 277+ messages in thread
From: Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= @ 2022-03-10  5:31 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget
  Cc: git, Jeff Hostetler, Derrick Stolee, ??var Arnfj??r?? Bjarmason,
	Jeff Hostetler

On Tue, Mar 08, 2022 at 10:15:00PM +0000, Jeff Hostetler via GitGitGadget wrote:
> Here is V2 of part 3 of my builtin FSMonitor series.


Hej Jeff,

First of all, the new test case passes here on one machine.
I will do another round on another machine the next days.

Having said that, there are some small questions here and there,
I'll send them as soon as I find the time for a more proper review.

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  2022-03-09 18:40     ` Derrick Stolee
  2022-03-09 18:42       ` Derrick Stolee
@ 2022-03-10 14:23       ` Jeff Hostetler
  1 sibling, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-10 14:23 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git
  Cc: Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler



On 3/9/22 1:40 PM, Derrick Stolee wrote:
> On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Create a set of prereqs to help understand how file names
>> are handled by the filesystem when they contain NFC and NFD
>> Unicode characters.
> 
> Prereqs look good and are well documented.
> 
>> +if test $unicode_debug = 1
> 
> Is this $unicode_debug something I should know from a previous
> patch? or is it a leftover from local debugging?


I added that and all of the print statements to help
describe the characteristics of the (OS, FS) pair,
for example what happens on (MacOS, FAT32) and is that
any different from (MacOS, APFS).  I found this very
useful in trying to decipher the docs.

However, it is kinda noisy and appears directly on the
console.  Since most people don't need to see it (unless
they are working on Unicode/UTF8 issues), I decided to
turn it off for now.

I'm not sure if we have a way to handle such output or
not.  I thought about maybe hooking it into the -d or -x
options, but I'm not sure if that helps or not.  So I
just turned it off.

Also, by not always testing the prereqs just to print
the result here, we avoid actually doing the lazy evals
until a real test wants to use one of them.


I'll add a comment in the script documenting it.

Thanks
Jeff

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  2022-03-09 18:42       ` Derrick Stolee
@ 2022-03-10 14:28         ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-10 14:28 UTC (permalink / raw)
  To: Derrick Stolee, Jeff Hostetler via GitGitGadget, git
  Cc: Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	Jeff Hostetler



On 3/9/22 1:42 PM, Derrick Stolee wrote:
> On 3/9/2022 1:40 PM, Derrick Stolee wrote:
>> On 3/8/2022 5:15 PM, Jeff Hostetler via GitGitGadget wrote:
>>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>>
>>> Create a set of prereqs to help understand how file names
>>> are handled by the filesystem when they contain NFC and NFD
>>> Unicode characters.
>>
>> Prereqs look good and are well documented.
>>
>>> +if test $unicode_debug = 1
>>
>> Is this $unicode_debug something I should know from a previous
>> patch? or is it a leftover from local debugging?
> 
> I see that you set unicode_debug = 0 in a later patch, but I
> suppose that we might want this output no matter what. Or, do
> we think it will interrupt the output parsing of 'prove' and
> other tools?

I was afraid that it might interrupt tools like prove, but
I just tried it and it didn't.  But yeah it would be safer
to turn it off until someone actually wants to do some debugging
in this area.

Jeff



^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor
  2022-03-08 22:15   ` [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-03-11  1:31     ` Ævar Arnfjörð Bjarmason
  2022-03-11 22:25       ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-03-11  1:31 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget
  Cc: git, Jeff Hostetler, Derrick Stolee, Torsten Bögershausen,
	Jeff Hostetler


On Tue, Mar 08 2022, Jeff Hostetler via GitGitGadget wrote:

> From: Jeff Hostetler <jeffhost@microsoft.com>
> [...]
> +	prepare_repo_settings(the_repository);
> +	fsm_settings__set_ipc(the_repository);
> +
> +	if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {
> +		const char *msg = fsm_settings__get_reason_msg(the_repository);
> +
> +		return error("%s '%s'", msg ? msg : "???", xgetcwd());
> +	}
> +
>  	if (!strcmp(subcmd, "start"))
>  		return !!try_to_start_background_daemon();
>  
> diff --git a/builtin/update-index.c b/builtin/update-index.c
> index d335f1ac72a..8f460e7195f 100644
> --- a/builtin/update-index.c
> +++ b/builtin/update-index.c
> @@ -1237,6 +1237,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
>  
>  	if (fsmonitor > 0) {
>  		enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
> +
> +		if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
> +			const char *msg = fsm_settings__get_reason_msg(r);
> +
> +			return error("%s '%s'", msg ? msg : "???", xgetcwd());
> +		}
> +
>  		if (fsm_mode == FSMONITOR_MODE_DISABLED) {
>  			advise(_("core.fsmonitor is unset; "
>  				 "set it if you really want to "

Can w assert somewhere earlier that ->mode can't be
FSMONITOR_MODE_INCOMPATIBLE at the same time that ->reason ==
FSMONITOR_REASON_OK, should that ever happen?

Then we can get rid of the "???" case here.

The "%s '%s'" here should really be marked for translation, but just
"some reason '$path'" is a pretty confusing message. This will emit
e.g.:

    "bare repos are incompatible with fsmonitor '/some/path/to/repo'"

Since we always hand these to error maybe have the helper do e.g.:

    error(_("bare repository '%s' is incompatible with fsmonitor"), path);

I find the second-guessing in fsmonitor-settings.c really hard to
follow, i.e. how seemingly every function has some "not loaded yet? load
it" instead of a more typical "init it", "use it", "free it"
pattern. Including stuff like this:
	
	enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)
	{
	        if (!r)
	                r = the_repository;

But anyway, seeing as we do try really hard to load the_repository (or a
repository) can't we use the_repository->gitdir etc. here instead of
xgetcwd(), or the_repository->worktree when non-bare?

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor
  2022-03-11  1:31     ` Ævar Arnfjörð Bjarmason
@ 2022-03-11 22:25       ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-11 22:25 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason, Jeff Hostetler via GitGitGadget
  Cc: git, Derrick Stolee, Torsten Bögershausen, Jeff Hostetler



On 3/10/22 8:31 PM, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, Mar 08 2022, Jeff Hostetler via GitGitGadget wrote:
> 
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>> [...]
>> +	prepare_repo_settings(the_repository);
>> +	fsm_settings__set_ipc(the_repository);
>> +
>> +	if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {
>> +		const char *msg = fsm_settings__get_reason_msg(the_repository);
>> +
>> +		return error("%s '%s'", msg ? msg : "???", xgetcwd());
>> +	}
>> +
>>   	if (!strcmp(subcmd, "start"))
>>   		return !!try_to_start_background_daemon();
>>   
>> diff --git a/builtin/update-index.c b/builtin/update-index.c
>> index d335f1ac72a..8f460e7195f 100644
>> --- a/builtin/update-index.c
>> +++ b/builtin/update-index.c
>> @@ -1237,6 +1237,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
>>   
>>   	if (fsmonitor > 0) {
>>   		enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
>> +
>> +		if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
>> +			const char *msg = fsm_settings__get_reason_msg(r);
>> +
>> +			return error("%s '%s'", msg ? msg : "???", xgetcwd());
>> +		}
>> +
>>   		if (fsm_mode == FSMONITOR_MODE_DISABLED) {
>>   			advise(_("core.fsmonitor is unset; "
>>   				 "set it if you really want to "
> 
> Can w assert somewhere earlier that ->mode can't be
> FSMONITOR_MODE_INCOMPATIBLE at the same time that ->reason ==
> FSMONITOR_REASON_OK, should that ever happen?
> 
> Then we can get rid of the "???" case here.
> 

Yeah, it would be nice to assert() the pair and simplify things.  I'll
make a note to look at that.


> The "%s '%s'" here should really be marked for translation, but just
> "some reason '$path'" is a pretty confusing message. This will emit
> e.g.:

I already have translations in the code that looks up the message,
so doing it here for a pair of %s's felt wrong.

> 
>      "bare repos are incompatible with fsmonitor '/some/path/to/repo'"
> 
> Since we always hand these to error maybe have the helper do e.g.:
> 
>      error(_("bare repository '%s' is incompatible with fsmonitor"), path);
> 

I'm wondering now if we should just drop the path from the message
and use the error message from __get_reason_msg() as is.  (It was
useful during debugging, but I could see it going away.)


> I find the second-guessing in fsmonitor-settings.c really hard to
> follow, i.e. how seemingly every function has some "not loaded yet? load
> it" instead of a more typical "init it", "use it", "free it"
> pattern. Including stuff like this:
> 	
> 	enum fsmonitor_mode fsm_settings__get_mode(struct repository *r)
> 	{
> 	        if (!r)
> 	                r = the_repository;
> 
> But anyway, seeing as we do try really hard to load the_repository (or a
> repository) can't we use the_repository->gitdir etc. here instead of
> xgetcwd(), or the_repository->worktree when non-bare?
> 

I'll take a look and see if I can simplify it.

Thanks
Jeff

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (28 preceding siblings ...)
  2022-03-10  5:31   ` Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
@ 2022-03-13 10:42   ` Torsten Bögershausen
  2022-03-21 22:06     ` Jeff Hostetler
  2022-03-21 22:59     ` Jeff Hostetler
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
  30 siblings, 2 replies; 277+ messages in thread
From: Torsten Bögershausen @ 2022-03-13 10:42 UTC (permalink / raw)
  To: Jeff Hostetler via GitGitGadget
  Cc: git, Jeff Hostetler, Derrick Stolee, ??var Arnfj??r?? Bjarmason,
	Jeff Hostetler

Hej Jeff,

I tried your patch on both a newer Mac and an older machine (with HFS+)
The older machine doesn't have
kFSEventStreamEventFlagItemCloned
As it is an enum, and not a #define, I ended up here:

  diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
  index 3332d3b779..fa172a05c4 100644
  --- a/compat/fsmonitor/fsm-listen-darwin.c
  +++ b/compat/fsmonitor/fsm-listen-darwin.c
  @@ -169,8 +169,6 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)
                  strbuf_addstr(&msg, "ItemXattrMod|");
          if (flag & kFSEventStreamEventFlagOwnEvent)
                  strbuf_addstr(&msg, "OwnEvent|");
  -       if (flag & kFSEventStreamEventFlagItemCloned)
  -               strbuf_addstr(&msg, "ItemCloned|");

          trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s",
                           path, flag, msg.buf);
  @@ -221,8 +219,7 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)
                  kFSEventStreamEventFlagItemModified |
                  kFSEventStreamEventFlagItemRemoved |
                  kFSEventStreamEventFlagItemRenamed |
  -               kFSEventStreamEventFlagItemXattrMod |
  -               kFSEventStreamEventFlagItemCloned;
  +               kFSEventStreamEventFlagItemXattrMod ;

One other thing, I just add it here:
There is a new file, t/lib-unicode-nfc-nfd.sh, which helps us with this code:
test_lazy_prereq UNICODE_NFC_PRESERVED

The existing code uses a construct called
UTF8_NFD_TO_NFC

And now I have 2 questions:
- Do we need the UNICODE_NFC_PRESERVED at all ?
- And should the UTF8_NFD_TO_NFC better be called UTF8_NFC_TO_NFD,
  because that is what it checks.
- Do we need the UNICODE_NFD_PRESERVED at all ?

As there are no non-UNICODE_NFD_PRESERVED filesystems, as far as I know.
And the current code does no tests, just debug prints.
I dunno.

On Tue, Mar 08, 2022 at 10:15:00PM +0000, Jeff Hostetler via GitGitGadget wrote:
> Here is V2 of part 3 of my builtin FSMonitor series.

> [] I updated the daemon on MacOS to report both the NFC and NFD spellings of
> a pathname when appropriate. This is a little more general than the
> "core.precomposeUnicode" setting, since the daemon does not know how the
> client has (or will have) it set when they make a query.
>
> [] I replaced my Unicode NFC/NFD test for MacOS to focus exclusively on
> Unicode composition/decomposition sensitivity and to not confuse that with
> case sensitivity.

That is a good thing.

[snip]

^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-13 10:42   ` Torsten Bögershausen
@ 2022-03-21 22:06     ` Jeff Hostetler
  2022-03-21 23:18       ` rsbecker
  2022-03-21 22:59     ` Jeff Hostetler
  1 sibling, 1 reply; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-21 22:06 UTC (permalink / raw)
  To: Torsten Bögershausen, Jeff Hostetler via GitGitGadget
  Cc: git, Derrick Stolee, ??var Arnfj??r?? Bjarmason, Jeff Hostetler



On 3/13/22 6:42 AM, Torsten Bögershausen wrote:
> Hej Jeff,
> 
> I tried your patch on both a newer Mac and an older machine (with HFS+)
> The older machine doesn't have
> kFSEventStreamEventFlagItemCloned
> As it is an enum, and not a #define, I ended up here:
> 
>    diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
>    index 3332d3b779..fa172a05c4 100644
>    --- a/compat/fsmonitor/fsm-listen-darwin.c
>    +++ b/compat/fsmonitor/fsm-listen-darwin.c
>    @@ -169,8 +169,6 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)
>                    strbuf_addstr(&msg, "ItemXattrMod|");
>            if (flag & kFSEventStreamEventFlagOwnEvent)
>                    strbuf_addstr(&msg, "OwnEvent|");
>    -       if (flag & kFSEventStreamEventFlagItemCloned)
>    -               strbuf_addstr(&msg, "ItemCloned|");
> 
>            trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s",
>                             path, flag, msg.buf);
>    @@ -221,8 +219,7 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)
>                    kFSEventStreamEventFlagItemModified |
>                    kFSEventStreamEventFlagItemRemoved |
>                    kFSEventStreamEventFlagItemRenamed |
>    -               kFSEventStreamEventFlagItemXattrMod |
>    -               kFSEventStreamEventFlagItemCloned;
>    +               kFSEventStreamEventFlagItemXattrMod ;

It looks like the ...Cloned bit was added to the SDK in 10.13 [1].
All the other bits were defined sometime between 10.5 and 10.10.

I'll add something in V7 to guard that bit.  I think 10.10 is old enough
that we don't need to special case those bits too.

Thanks,
Jeff

[1] /Applications/Xcode.app/Contents/Developer/Platforms/ \
     MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/ \
     Library/Frameworks/CoreServices.framework/Frameworks/ \
     FSEvents.framework/Versions/Current/Headers/FSEvents.h


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-13 10:42   ` Torsten Bögershausen
  2022-03-21 22:06     ` Jeff Hostetler
@ 2022-03-21 22:59     ` Jeff Hostetler
  1 sibling, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-21 22:59 UTC (permalink / raw)
  To: Torsten Bögershausen, Jeff Hostetler via GitGitGadget
  Cc: git, Derrick Stolee, ??var Arnfj??r?? Bjarmason, Jeff Hostetler



On 3/13/22 6:42 AM, Torsten Bögershausen wrote:
> Hej Jeff,
> 
[...]
> 
> One other thing, I just add it here:
> There is a new file, t/lib-unicode-nfc-nfd.sh, which helps us with this code:
> test_lazy_prereq UNICODE_NFC_PRESERVED
> 
> The existing code uses a construct called
> UTF8_NFD_TO_NFC
> 
> And now I have 2 questions:
> - Do we need the UNICODE_NFC_PRESERVED at all ?
> - And should the UTF8_NFD_TO_NFC better be called UTF8_NFC_TO_NFD,
>    because that is what it checks.
> - Do we need the UNICODE_NFD_PRESERVED at all ?
> 
> As there are no non-UNICODE_NFD_PRESERVED filesystems, as far as I know.
> And the current code does no tests, just debug prints.
> I dunno.

I created t/lib-unicode-nfc-nfd.sh to help me understand
the issues.  I found the existing UTF8_NFD_TO_NFC prereq
confusing (and yes it seemed poorly named).

The existing prereq returned the same answer on APFS, HFS+,
and FAT32 (a thumbdrive).  I know they behave differently
and I found it odd that the prereq did not make any distinction.

I was hesitant to rename the existing prereq because it is
currently used by 5+ different tests and I didn't want to
expand the scope of my two already very large series.

Also, the existing prereq feels a little sloppy.  It creates
a file in NFC and does a lstat in the NFD spelling.  There
are several ways that the OS and/or FS can lie to us.  For
example, the prereq is satisfied on a FAT32 thumbdrive and
we know FAT32 doesn't do NFC-->NFD conversions.  So I'd like
to move away from that prereq definition at some point.


My new prereqs try to:

(1) independently confirm whether there is aliasing happening
     at all (whether at the FS or OS layer).

(2) determine if the actual on-disk spelling is altered by the
     FS (in both NFC and NFD cases).


We know that HFS+ does not preserve NFC spellings, but APFS
does.  (FAT32 also preserves NFC spelling under MacOS.)
So the UNICODE_NFC_PRESERVED lets me distinguish between HFS+
and APFS/FAT32.

I have not heard of any filesystems that convert NFD to NFC,
so technically we don't need the UNICODE_NFD_PRESERVED prereq,
but then again until I tested that, it was unclear how MacOS
did the aliasing on APFS (and FAT32).  On the basis of that
testing, we can say that MacOS -- at the MacOS layer -- is
responsible for the aliasing and that both NFC and NFD spellings
are preserved on APFS and FAT32.

So I'd rather keep the 3 prereqs that I have now.

The ones marked _DOUBLE_ are currently extra.  I have them to
help study how code points with multiple combining characters
are handled.  I have prereqs for the basic double chars, but
there are several opportunities for weird edge cases (non-
canonical ordering and other collisions) that I don't want to
get stuck on right now.  So we might make more use of them in
the future.


That's too long of an answer, but hopefully that explains
some of my paranoia. :-)

Jeff

> 
> On Tue, Mar 08, 2022 at 10:15:00PM +0000, Jeff Hostetler via GitGitGadget wrote:
>> Here is V2 of part 3 of my builtin FSMonitor series.
[...]

^ permalink raw reply	[flat|nested] 277+ messages in thread

* RE: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-21 22:06     ` Jeff Hostetler
@ 2022-03-21 23:18       ` rsbecker
  2022-03-22 14:10         ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: rsbecker @ 2022-03-21 23:18 UTC (permalink / raw)
  To: 'Jeff Hostetler', 'Torsten Bögershausen',
	'Jeff Hostetler via GitGitGadget'
  Cc: git, 'Derrick Stolee',
	'??var Arnfj??r?? Bjarmason', 'Jeff Hostetler'

On March 21, 2022 6:06 PM, Jeff Hostetler wrote:
>On 3/13/22 6:42 AM, Torsten Bögershausen wrote:
>> Hej Jeff,
>>
>> I tried your patch on both a newer Mac and an older machine (with
>> HFS+) The older machine doesn't have kFSEventStreamEventFlagItemCloned
>> As it is an enum, and not a #define, I ended up here:
>>
>>    diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-
>listen-darwin.c
>>    index 3332d3b779..fa172a05c4 100644
>>    --- a/compat/fsmonitor/fsm-listen-darwin.c
>>    +++ b/compat/fsmonitor/fsm-listen-darwin.c
>>    @@ -169,8 +169,6 @@ static void log_flags_set(const char *path, const
>FSEventStreamEventFlags flag)
>>                    strbuf_addstr(&msg, "ItemXattrMod|");
>>            if (flag & kFSEventStreamEventFlagOwnEvent)
>>                    strbuf_addstr(&msg, "OwnEvent|");
>>    -       if (flag & kFSEventStreamEventFlagItemCloned)
>>    -               strbuf_addstr(&msg, "ItemCloned|");
>>
>>            trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s",
>>                             path, flag, msg.buf);
>>    @@ -221,8 +219,7 @@ static int ef_ignore_xattr(const
>FSEventStreamEventFlags ef)
>>                    kFSEventStreamEventFlagItemModified |
>>                    kFSEventStreamEventFlagItemRemoved |
>>                    kFSEventStreamEventFlagItemRenamed |
>>    -               kFSEventStreamEventFlagItemXattrMod |
>>    -               kFSEventStreamEventFlagItemCloned;
>>    +               kFSEventStreamEventFlagItemXattrMod ;
>
>It looks like the ...Cloned bit was added to the SDK in 10.13 [1].
>All the other bits were defined sometime between 10.5 and 10.10.
>
>I'll add something in V7 to guard that bit.  I think 10.10 is old enough that we don't
>need to special case those bits too.

I realize it is a bit late in the game, but would you consider a pre-hook and post-hook that automatically run with fsmonitor kicks off/terminates. I am thinking about use cases where this is integrated into more complex processes and it would be nice to have notifications of what fsmonitor is doing and when.

Thanks,
Randall


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-21 23:18       ` rsbecker
@ 2022-03-22 14:10         ` Jeff Hostetler
  2022-03-22 14:25           ` rsbecker
  0 siblings, 1 reply; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-22 14:10 UTC (permalink / raw)
  To: rsbecker, 'Torsten Bögershausen',
	'Jeff Hostetler via GitGitGadget'
  Cc: git, 'Derrick Stolee',
	'??var Arnfj??r?? Bjarmason', 'Jeff Hostetler'



On 3/21/22 7:18 PM, rsbecker@nexbridge.com wrote:
> On March 21, 2022 6:06 PM, Jeff Hostetler wrote:
>> On 3/13/22 6:42 AM, Torsten Bögershausen wrote:
>>> Hej Jeff,
>>>
[...]
>>
>> It looks like the ...Cloned bit was added to the SDK in 10.13 [1].
>> All the other bits were defined sometime between 10.5 and 10.10.
>>
>> I'll add something in V7 to guard that bit.  I think 10.10 is old enough that we don't
>> need to special case those bits too.
> 
> I realize it is a bit late in the game, but would you consider a pre-hook and post-hook that automatically run with fsmonitor kicks off/terminates. I am thinking about use cases where this is integrated into more complex processes and it would be nice to have notifications of what fsmonitor is doing and when.
> 
> Thanks,
> Randall
> 

I hadn't really considered having a pre/post hook for the daemon.
I'm not opposed to it; I just hadn't thought about it.

By this I assume you mean something inside the fsmonitor--daemon
process that invokes the hooks when it is starting/stopping.
As opposed to something in a client command (like status) before
it implicitly started a daemon process.  The latter method would
not give you post-hook events because the daemon usually outlives
the client command.

Perhaps you could elaborate on what you would use these hooks for
or how they would be helpful.  It would be easy to add pre/post
hooks in the main thread of the daemon.  However, I worry about
the prehook slowing the startup of the daemon -- since the client
status command might be waiting for it to become ready.  I also
have a "health" thread in part3 that would be a candidate for
pre/post and any other periodic hooks that might be useful.
But again, before I suggest a design for this, it would be good
to know what kind of things you would want to do with them.

Jeff

^ permalink raw reply	[flat|nested] 277+ messages in thread

* RE: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-22 14:10         ` Jeff Hostetler
@ 2022-03-22 14:25           ` rsbecker
  2022-03-22 15:01             ` Jeff Hostetler
  0 siblings, 1 reply; 277+ messages in thread
From: rsbecker @ 2022-03-22 14:25 UTC (permalink / raw)
  To: 'Jeff Hostetler', 'Torsten Bögershausen',
	'Jeff Hostetler via GitGitGadget'
  Cc: git, 'Derrick Stolee',
	'??var Arnfj??r?? Bjarmason', 'Jeff Hostetler'

On March 22, 2022 10:11 AM, Jeff Hostetler wrote:
>On 3/21/22 7:18 PM, rsbecker@nexbridge.com wrote:
>> On March 21, 2022 6:06 PM, Jeff Hostetler wrote:
>>> On 3/13/22 6:42 AM, Torsten Bögershausen wrote:
>>>> Hej Jeff,
>>>>
>[...]
>>>
>>> It looks like the ...Cloned bit was added to the SDK in 10.13 [1].
>>> All the other bits were defined sometime between 10.5 and 10.10.
>>>
>>> I'll add something in V7 to guard that bit.  I think 10.10 is old
>>> enough that we don't need to special case those bits too.
>>
>> I realize it is a bit late in the game, but would you consider a pre-hook and post-
>hook that automatically run with fsmonitor kicks off/terminates. I am thinking
>about use cases where this is integrated into more complex processes and it
>would be nice to have notifications of what fsmonitor is doing and when.
>>
>> Thanks,
>> Randall
>>
>
>I hadn't really considered having a pre/post hook for the daemon.
>I'm not opposed to it; I just hadn't thought about it.
>
>By this I assume you mean something inside the fsmonitor--daemon process that
>invokes the hooks when it is starting/stopping.
>As opposed to something in a client command (like status) before it implicitly
>started a daemon process.  The latter method would not give you post-hook
>events because the daemon usually outlives the client command.
>
>Perhaps you could elaborate on what you would use these hooks for or how they
>would be helpful.  It would be easy to add pre/post hooks in the main thread of
>the daemon.  However, I worry about the prehook slowing the startup of the
>daemon -- since the client status command might be waiting for it to become
>ready.  I also have a "health" thread in part3 that would be a candidate for
>pre/post and any other periodic hooks that might be useful.
>But again, before I suggest a design for this, it would be good to know what kind of
>things you would want to do with them.

Some examples of what I have in mind. There are more, but this covers what I have in mind urgently:

1. Setting up a lock file (semaphore) just before fsmonitor runs that will cause any scripts that might change the state of the repository on the fly to suspend until fsmonitor is done.
2. Ensuring that in-flight scripts that do stuff are finished or not leaving the repo in a transitional state before fsmonitor runs - holding fsmonitor until the pre-hook finishes.
3. Notifying syslog or some other paging system if something has gone horribly wrong - as in fsmonitor found something bad in the index.
4. Clearing any semaphores created earlier (example 1).

Regards,
Randall


^ permalink raw reply	[flat|nested] 277+ messages in thread

* Re: [PATCH v2 00/27] Builtin FSMonitor Part 3
  2022-03-22 14:25           ` rsbecker
@ 2022-03-22 15:01             ` Jeff Hostetler
  0 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler @ 2022-03-22 15:01 UTC (permalink / raw)
  To: rsbecker, 'Torsten Bögershausen',
	'Jeff Hostetler via GitGitGadget'
  Cc: git, 'Derrick Stolee',
	'??var Arnfj??r?? Bjarmason', 'Jeff Hostetler'



On 3/22/22 10:25 AM, rsbecker@nexbridge.com wrote:
> On March 22, 2022 10:11 AM, Jeff Hostetler wrote:
>> On 3/21/22 7:18 PM, rsbecker@nexbridge.com wrote:
>>> On March 21, 2022 6:06 PM, Jeff Hostetler wrote:
>>>> On 3/13/22 6:42 AM, Torsten Bögershausen wrote:
>>>>> Hej Jeff,
>>>>>
>> [...]
>>>>
>>>> It looks like the ...Cloned bit was added to the SDK in 10.13 [1].
>>>> All the other bits were defined sometime between 10.5 and 10.10.
>>>>
>>>> I'll add something in V7 to guard that bit.  I think 10.10 is old
>>>> enough that we don't need to special case those bits too.
>>>
>>> I realize it is a bit late in the game, but would you consider a pre-hook and post-
>> hook that automatically run with fsmonitor kicks off/terminates. I am thinking
>> about use cases where this is integrated into more complex processes and it
>> would be nice to have notifications of what fsmonitor is doing and when.
>>>
>>> Thanks,
>>> Randall
>>>
>>
>> I hadn't really considered having a pre/post hook for the daemon.
>> I'm not opposed to it; I just hadn't thought about it.
>>
>> By this I assume you mean something inside the fsmonitor--daemon process that
>> invokes the hooks when it is starting/stopping.
>> As opposed to something in a client command (like status) before it implicitly
>> started a daemon process.  The latter method would not give you post-hook
>> events because the daemon usually outlives the client command.
>>
>> Perhaps you could elaborate on what you would use these hooks for or how they
>> would be helpful.  It would be easy to add pre/post hooks in the main thread of
>> the daemon.  However, I worry about the prehook slowing the startup of the
>> daemon -- since the client status command might be waiting for it to become
>> ready.  I also have a "health" thread in part3 that would be a candidate for
>> pre/post and any other periodic hooks that might be useful.
>> But again, before I suggest a design for this, it would be good to know what kind of
>> things you would want to do with them.
> 
> Some examples of what I have in mind. There are more, but this covers what I have in mind urgently:
> 
> 1. Setting up a lock file (semaphore) just before fsmonitor runs that will cause any scripts that might change the state of the repository on the fly to suspend until fsmonitor is done.

The builtin fsmonitor--daemon is a long-running process.  It is
either explicitly started by the user or implicitly started by
clients commands, like "git status", lazily after it is enabled
in the config.  It runs until explicitly stopped (or the workdir
is deleted).

It is designed to be running and watch the filesystem for changes.

Later client commands can ask it for what has changed on disk
since the previous request (checkpoint).  And the only way to
capture that info is to watch the file system as things happen.
(Unless we have a really deep journal, but that often requires
admin access, so we don't use that.)

The fsmonitor daemon is unlike other subordinate commands in Git.
For example, "git fetch" might synchronously invoke "git index-pack"
and communicate over the child's stdin/stdout.  And that child is
bound to a single parent process.

When the daemon starts, it disassociates from the console and
opens a socket or named pipe and listens for requests (REST-like) 
commands.  It is designed to respond to multiple clients concurrently
and over a long time period -- like a daemon or service process.

> 2. Ensuring that in-flight scripts that do stuff are finished or not leaving the repo in a transitional state before fsmonitor runs - holding fsmonitor until the pre-hook finishes.
> 3. Notifying syslog or some other paging system if something has gone horribly wrong - as in fsmonitor found something bad in the index.

fsmonitor doesn't read the index.  it's only watching and summarizing
file system events.  And can enumerate the list of changed paths between
two checkpoints in response to a client request.


> 4. Clearing any semaphores created earlier (example 1).

The daemon isn't designed to ever "be done" and hence there are no
on-disk lock files/semaphores.  There might be some opportunities
for startup/shutdown to do some cleanup (temp files and the like),
but I think it is premature to talk about that right now.

Hope this helps,
Jeff


> 
> Regards,
> Randall
> 

^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 00/27] Builtin FSMonitor Part 3
  2022-03-08 22:15 ` [PATCH v2 00/27] " Jeff Hostetler via GitGitGadget
                     ` (29 preceding siblings ...)
  2022-03-13 10:42   ` Torsten Bögershausen
@ 2022-03-22 18:22   ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 01/27] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
                       ` (28 more replies)
  30 siblings, 29 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler

Here is V3 of Part 3 of my builtin FSMonitor series.

I have addressed all of the feedback from Part 3 V2 and the mess that was
Part 2.5 (now obsolete). This version builds upon the new V7 of Part 2
(which includes 2.5).

This version includes: (1) fixup a few more "_()" calls in die() and error()
messages. (2) refactor how fsmonitor incompatibility error messages are
formatted (3) make use of new "start_daemon()" function t7527 to reduce
duplicated code. (4) improve documentation around Unicode/UTF8 testing on
MacOS.

Here is the range-diff from V2 to V3:

 1:  34619e0652 !  1:  779a15b38e fsm-listen-win32: handle shortnames
    @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI
     @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
                  goto normalize;
              if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {
    -             error("[GLE %ld] could not convert path to UTF-8: '%.*ls'",
    +             error(_("[GLE %ld] could not convert path to UTF-8: '%.*ls'"),
     -                  GetLastError(),
     -                  (int)(info->FileNameLength / sizeof(WCHAR)),
     -                  info->FileName);
    @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon stop (rename
     +
     +    git init test_implicit_1s &&
     +
    -+    start_daemon test_implicit_1s &&
    ++    start_daemon -C test_implicit_1s &&
     +
     +    # renaming the .git directory will implicitly stop the daemon.
     +    # this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.
    @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon stop (rename
     +    test_path_is_file test_implicit_1s2/GIT~1 &&
     +    test_path_is_dir  test_implicit_1s2/GIT~2 &&
     +
    -+    start_daemon test_implicit_1s2 &&
    ++    start_daemon -C test_implicit_1s2 &&
     +
     +    # renaming the .git directory will implicitly stop the daemon.
     +    # the rename-from FS Event will contain the shortname.
 2:  3a0f30b849 !  2:  11d4a17b69 t7527: test FSMonitor on repos with Unicode root paths
    @@ t/t7527-builtin-fsmonitor.sh: do
     +        git -C "$u" add file1 &&
     +        git -C "$u" config core.fsmonitor true &&
     +
    -+        start_daemon "$u" &&
    ++        start_daemon -C "$u" &&
     +        git -C "$u" status >actual &&
     +        grep "new file:   file1" actual
     +    '
 3:  87d1c0b6f2 !  3:  901fa32f6e t/helper/fsmonitor-client: create stress test
    @@ t/helper/test-fsmonitor-client.c: static int do_send_flush(void)
     +    int nr_requests = 1;
      
          const char * const fsmonitor_client_usage[] = {
    -         N_("test-helper fsmonitor-client query [<token>]"),
    -         N_("test-helper fsmonitor-client flush"),
    -+        N_("test-helper fsmonitor-client hammer [<token>] [<threads>] [<requests>]"),
    +         "test-tool fsmonitor-client query [<token>]",
    +         "test-tool fsmonitor-client flush",
    ++        "test-tool fsmonitor-client hammer [<token>] [<threads>] [<requests>]",
              NULL,
          };
      
          struct option options[] = {
    -         OPT_STRING(0, "token", &token, N_("token"),
    -                N_("command token to send to the server")),
    +         OPT_STRING(0, "token", &token, "token",
    +                "command token to send to the server"),
     +
    -+        OPT_INTEGER(0, "threads", &nr_threads, N_("number of client threads")),
    -+        OPT_INTEGER(0, "requests", &nr_requests, N_("number of requests per thread")),
    ++        OPT_INTEGER(0, "threads", &nr_threads, "number of client threads"),
    ++        OPT_INTEGER(0, "requests", &nr_requests, "number of requests per thread"),
     +
              OPT_END()
          };
 4:  8c4f90ae4f !  4:  a8f0b2a525 fsmonitor-settings: bare repos are incompatible with FSMonitor
    @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar
     +    prepare_repo_settings(the_repository);
     +    fsm_settings__set_ipc(the_repository);
     +
    -+    if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {
    -+        const char *msg = fsm_settings__get_reason_msg(the_repository);
    -+
    -+        return error("%s '%s'", msg ? msg : "???", xgetcwd());
    -+    }
    ++    if (fsm_settings__error_if_incompatible(the_repository))
    ++        return 1;
     +
          if (!strcmp(subcmd, "start"))
              return !!try_to_start_background_daemon();
    @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const
          if (fsmonitor > 0) {
              enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
     +
    -+        if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
    -+            const char *msg = fsm_settings__get_reason_msg(r);
    -+
    -+            return error("%s '%s'", msg ? msg : "???", xgetcwd());
    -+        }
    ++        if (fsm_settings__error_if_incompatible(the_repository))
    ++            return 1;
     +
              if (fsm_mode == FSMONITOR_MODE_DISABLED) {
    -             advise(_("core.fsmonitor is unset; "
    -                  "set it if you really want to "
    +             warning(_("core.fsmonitor is unset; "
    +                 "set it if you really want to "
     
      ## fsmonitor-settings.c ##
     @@
    @@ fsmonitor-settings.c
      static void lookup_fsmonitor_settings(struct repository *r)
      {
          struct fsmonitor_settings *s;
    +@@ fsmonitor-settings.c: static void lookup_fsmonitor_settings(struct repository *r)
    + 
    +     CALLOC_ARRAY(s, 1);
    +     s->mode = FSMONITOR_MODE_DISABLED;
    ++    s->reason = FSMONITOR_REASON_OK;
    + 
    +     r->settings.fsmonitor = s;
    + 
     @@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)
      
          lookup_fsmonitor_settings(r);
    @@ fsmonitor-settings.c: void fsm_settings__set_disabled(struct repository *r)
     +    return r->settings.fsmonitor->reason;
     +}
     +
    -+const char *fsm_settings__get_reason_msg(struct repository *r)
    ++int fsm_settings__error_if_incompatible(struct repository *r)
     +{
     +    enum fsmonitor_reason reason = fsm_settings__get_reason(r);
     +
     +    switch (reason) {
     +    case FSMONITOR_REASON_OK:
    -+        return NULL;
    ++        return 0;
     +
     +    case FSMONITOR_REASON_BARE:
    -+        return _("bare repos are incompatible with fsmonitor");
    ++        error(_("bare repository '%s' is incompatible with fsmonitor"),
    ++              xgetcwd());
    ++        return 1;
     +    }
     +
    -+    BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
    ++    BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
     +        reason);
     +}
     
    @@ fsmonitor-settings.h: void fsm_settings__set_disabled(struct repository *r);
      const char *fsm_settings__get_hook_path(struct repository *r);
      
     +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);
    -+const char *fsm_settings__get_reason_msg(struct repository *r);
    ++int fsm_settings__error_if_incompatible(struct repository *r);
     +
      struct fsmonitor_settings;
      
    @@ t/t7519-status-fsmonitor.sh: test_lazy_prereq UNTRACKED_CACHE '
     +    test_must_fail \
     +        git -C ./bare-clone -c core.fsmonitor=foo \
     +            update-index --fsmonitor 2>actual &&
    -+    grep "bare repos are incompatible with fsmonitor" actual &&
    ++    grep "bare repository .* is incompatible with fsmonitor" actual &&
     +
     +    test_must_fail \
     +        git -C ./bare-clone -c core.fsmonitor=true \
     +            update-index --fsmonitor 2>actual &&
    -+    grep "bare repos are incompatible with fsmonitor" actual
    ++    grep "bare repository .* is incompatible with fsmonitor" actual
     +'
     +
     +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
     +    test_when_finished "rm -rf ./bare-clone actual" &&
     +    git init --bare bare-clone &&
     +    test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&
    -+    grep "bare repos are incompatible with fsmonitor" actual
    ++    grep "bare repository .* is incompatible with fsmonitor" actual
     +'
     +
      test_expect_success 'setup' '
 5:  6329328d18 !  5:  e32a8a7ea7 fsmonitor-settings: stub in platform-specific incompatibility checking
    @@ Metadata
     Author: Jeff Hostetler <jeffhost@microsoft.com>
     
      ## Commit message ##
    -    fsmonitor-settings: stub in platform-specific incompatibility checking
    +    fsmonitor-settings: stub in Win32-specific incompatibility checking
     
         Extend generic incompatibility checkout with platform-specific
         mechanism.  Stub in Win32 version.
    @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)
      
     
      ## fsmonitor-settings.h ##
    -@@ fsmonitor-settings.h: const char *fsm_settings__get_reason_msg(struct repository *r);
    +@@ fsmonitor-settings.h: int fsm_settings__error_if_incompatible(struct repository *r);
      
      struct fsmonitor_settings;
      
 6:  fa9e86e7de !  6:  5546339d96 fsmonitor-settings: VFS for Git virtual repos are incompatible
    @@ compat/fsmonitor/fsm-settings-win32.c
      }
     
      ## fsmonitor-settings.c ##
    -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)
    - 
    -     case FSMONITOR_REASON_BARE:
    -         return _("bare repos are incompatible with fsmonitor");
    +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)
    +         error(_("bare repository '%s' is incompatible with fsmonitor"),
    +               xgetcwd());
    +         return 1;
     +
     +    case FSMONITOR_REASON_VFS4GIT:
    -+        return _("virtual repos are incompatible with fsmonitor");
    ++        error(_("virtual repository '%s' is incompatible with fsmonitor"),
    ++              r->worktree);
    ++        return 1;
          }
      
    -     BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
    +     BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
     
      ## fsmonitor-settings.h ##
     @@ fsmonitor-settings.h: enum fsmonitor_mode {
    @@ fsmonitor-settings.h: enum fsmonitor_mode {
     
      ## t/t7519-status-fsmonitor.sh ##
     @@ t/t7519-status-fsmonitor.sh: test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
    -     grep "bare repos are incompatible with fsmonitor" actual
    +     grep "bare repository .* is incompatible with fsmonitor" actual
      '
      
     +test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '
    @@ t/t7519-status-fsmonitor.sh: test_expect_success FSMONITOR_DAEMON 'run fsmonitor
     +    test_must_fail git -C ./fake-virtual-clone \
     +               -c core.virtualfilesystem=true \
     +               fsmonitor--daemon run 2>actual &&
    -+    grep "virtual repos are incompatible with fsmonitor" actual
    ++    grep "virtual repository .* is incompatible with fsmonitor" actual
     +'
     +
      test_expect_success 'setup' '
 7:  c180241041 =  7:  1d2877efda fsmonitor-settings: stub in macOS-specific incompatibility checking
 8:  e3bfa0bd69 !  8:  06d7f18676 fsmonitor-settings: remote repos on macOS are incompatible
    @@ compat/fsmonitor/fsm-settings-darwin.c
      }
     
      ## fsmonitor-settings.c ##
    -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)
    -     case FSMONITOR_REASON_BARE:
    -         return _("bare repos are incompatible with fsmonitor");
    +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)
    +               xgetcwd());
    +         return 1;
      
     +    case FSMONITOR_REASON_ERROR:
    -+        return _("repo incompatible with fsmonitor due to errors");
    ++        error(_("repository '%s' is incompatible with fsmonitor due to errors"),
    ++              r->worktree);
    ++        return 1;
     +
     +    case FSMONITOR_REASON_REMOTE:
    -+        return _("remote repos are incompatible with fsmonitor");
    ++        error(_("remote repository '%s' is incompatible with fsmonitor"),
    ++              r->worktree);
    ++        return 1;
     +
          case FSMONITOR_REASON_VFS4GIT:
    -         return _("virtual repos are incompatible with fsmonitor");
    -     }
    +         error(_("virtual repository '%s' is incompatible with fsmonitor"),
    +               r->worktree);
     
      ## fsmonitor-settings.h ##
     @@ fsmonitor-settings.h: enum fsmonitor_mode {
 9:  e32da3118f =  9:  5ca97f482d fsmonitor-settings: remote repos on Windows are incompatible
27:  e3e01677d9 ! 10:  6715143724 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible
    @@ compat/fsmonitor/fsm-settings-darwin.c: enum fsmonitor_reason fsm_os__incompatib
      
     
      ## fsmonitor-settings.c ##
    -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)
    - 
    -     case FSMONITOR_REASON_VFS4GIT:
    -         return _("virtual repos are incompatible with fsmonitor");
    +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)
    +         error(_("virtual repository '%s' is incompatible with fsmonitor"),
    +               r->worktree);
    +         return 1;
     +
     +    case FSMONITOR_REASON_NOSOCKETS:
    -+        return _("repo filesystem does not support Unix sockets");
    ++        error(_("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets"),
    ++              r->worktree);
    ++        return 1;
          }
      
    -     BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
    +     BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
     
      ## fsmonitor-settings.h ##
     @@ fsmonitor-settings.h: enum fsmonitor_reason {
10:  f63de4eda3 = 11:  ed1f723130 unpack-trees: initialize fsmonitor_has_run_once in o->result
11:  fe305f5f28 = 12:  35c77b854b fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
12:  c8f3e251b1 ! 13:  a5affb359c fsmonitor--daemon: cd out of worktree root
    @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
     +     */
     +    home = getenv("HOME");
     +    if (home && *home && chdir(home))
    -+        die_errno("could not cd home '%s'", home);
    ++        die_errno(_("could not cd home '%s'"), home);
     +
          err = fsmonitor_run_daemon_1(&state);
      
    @@ compat/fsmonitor/fsm-listen-win32.c: static int recv_rdcw_watch(struct one_watch
     +     * Shutdown if we get any error.
           */
      
    -     error("GetOverlappedResult failed on '%s' [GLE %ld]",
    +     error(_("GetOverlappedResult failed on '%s' [GLE %ld]"),
     
      ## fsmonitor--daemon.h ##
     @@ fsmonitor--daemon.h: struct fsmonitor_daemon_state {
13:  71673be2da = 14:  087af5dfb6 fsmonitor--daemon: prepare for adding health thread
14:  5387baaf5d ! 15:  e78eb20c1b fsmonitor--daemon: rename listener thread related variables
    @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
          /* Prepare to (recursively) watch the <worktree-root> directory. */
     
      ## compat/fsmonitor/fsm-listen-darwin.c ##
    -@@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef stream);
    +@@
      #include "fsm-listen.h"
      #include "fsmonitor--daemon.h"
      
    @@ compat/fsmonitor/fsm-listen-darwin.c: int fsm_listen__ctor(struct fsmonitor_daem
              NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);
     @@ compat/fsmonitor/fsm-listen-darwin.c: int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
      failed:
    -     error("Unable to create FSEventStream.");
    +     error(_("Unable to create FSEventStream."));
      
     -    FREE_AND_NULL(state->backend_data);
     +    FREE_AND_NULL(state->listen_data);
15:  f78e4ad87c = 16:  301fff5296 fsmonitor--daemon: stub in health thread
16:  bb72f911a0 = 17:  c6b5bdd25e fsm-health-win32: add polling framework to monitor daemon health
17:  baf8c031a9 ! 18:  13d11713a8 fsm-health-win32: force shutdown daemon if worktree root moves
    @@ compat/fsmonitor/fsm-health-win32.c: struct fsm_health_data
     +        return 0;
     +
     +    default:
    -+        die("unhandled case in 'has_worktree_moved': %d",
    ++        die(_("unhandled case in 'has_worktree_moved': %d"),
     +            (int)ctx);
     +    }
     +
18:  796b659139 = 19:  01c1a38c46 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
19:  2459192087 = 20:  0f0a5b5ca1 fsmonitor: optimize processing of directory events
20:  06a3241385 ! 21:  d8218d197a t7527: FSMonitor tests for directory moves
    @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '
          git -c core.fsmonitor=false add . &&
          test_tick &&
          git -c core.fsmonitor=false commit -m initial &&
    -@@ t/t7527-builtin-fsmonitor.sh: verify_status () {
    -     echo HELLO AFTER
    +@@ t/t7527-builtin-fsmonitor.sh: directory_to_file () {
    +     echo 1 >dir1
      }
      
     +move_directory_contents_deeper() {
    -+    mkdir T1/_new_
    ++    mkdir T1/_new_ &&
     +    mv T1/[A-Z]* T1/_new_
     +}
     +
21:  4b59013cad = 22:  79da369dcc t/perf/p7527: add perf test for builtin FSMonitor
22:  524d449ed6 ! 23:  4ab4306ada fsmonitor: never set CE_FSMONITOR_VALID on submodules
    @@ t/t7527-builtin-fsmonitor.sh: do
     +    git -C super submodule add ../sub ./dir_1/dir_2/sub &&
     +    git -C super commit -m "add sub" &&
     +
    -+    start_daemon super &&
    ++    start_daemon -C super &&
     +    git -C super config core.fsmonitor true &&
     +    git -C super update-index --fsmonitor &&
     +    git -C super status &&
23:  c7264decaf ! 24:  5d0fa19929 t7527: test FSMonitor on case insensitive+preserving file system
    @@ t/t7527-builtin-fsmonitor.sh: test_expect_success "Submodule" '
     +#    test_when_finished "stop_daemon_delete_repo test_insensitive" &&
     +
     +    git init test_insensitive &&
    -+    (
    -+        GIT_TRACE_FSMONITOR="$(pwd)/insensitive.trace" &&
    -+        export GIT_TRACE_FSMONITOR &&
     +
    -+        start_daemon test_insensitive
    -+    ) &&
    ++    start_daemon -C test_insensitive -tf "$PWD/insensitive.trace" &&
     +
     +    mkdir -p test_insensitive/abc/def &&
     +    echo xyz >test_insensitive/ABC/DEF/xyz &&
24:  95b9d4210d = 25:  264397e8bd fsmonitor: on macOS also emit NFC spelling for NFD pathname
25:  5a0c1b7a28 ! 26:  e6b621fb76 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
    @@ t/lib-unicode-nfc-nfd.sh (new)
     +    ls | od -t x1 | grep "64 *5f *cf *89 *cc *94 *cd *82"
     +'
     +
    -+if test $unicode_debug = 1
    ++# The following is for debugging. I found it useful when
    ++# trying to understand the various (OS, FS) quirks WRT
    ++# Unicode and how composition/decomposition is handled.
    ++# For example, when trying to understand how (macOS, APFS)
    ++# and (macOS, HFS) and (macOS, FAT32) compare.
    ++#
    ++# It is rather noisy, so it is disabled by default.
    ++#
    ++if test "$unicode_debug" = "true"
     +then
     +    if test_have_prereq UNICODE_COMPOSITION_SENSITIVE
     +    then
26:  a45c1fd300 ! 27:  aa96a849ce t7527: test Unicode NFC/NFD handling on MacOS
    @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse
          egrep "^event: abc/def/xyz$" ./insensitive.trace
      '
      
    -+unicode_debug=0
    ++# The variable "unicode_debug" is defined in the following library
    ++# script to dump information about how the (OS, FS) handles Unicode
    ++# composition.  Uncomment the following line if you want to enable it.
    ++#
    ++# unicode_debug=true
    ++
     +. "$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh"
     +
     +# See if the OS or filesystem does NFC/NFD aliasing/munging.
    @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse
     +    test_when_finished "stop_daemon_delete_repo test_unicode" &&
     +
     +    git init test_unicode &&
    -+    (
    -+        GIT_TRACE_FSMONITOR="$(pwd)/unicode.trace" &&
    -+        export GIT_TRACE_FSMONITOR &&
     +
    -+        start_daemon test_unicode
    -+    ) &&
    ++    start_daemon -C test_unicode -tf "$PWD/unicode.trace" &&
     +
     +    # Create a directory using an NFC spelling.
     +    #


Jeff Hostetler (27):
  fsm-listen-win32: handle shortnames
  t7527: test FSMonitor on repos with Unicode root paths
  t/helper/fsmonitor-client: create stress test
  fsmonitor-settings: bare repos are incompatible with FSMonitor
  fsmonitor-settings: stub in Win32-specific incompatibility checking
  fsmonitor-settings: VFS for Git virtual repos are incompatible
  fsmonitor-settings: stub in macOS-specific incompatibility checking
  fsmonitor-settings: remote repos on macOS are incompatible
  fsmonitor-settings: remote repos on Windows are incompatible
  fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible
  unpack-trees: initialize fsmonitor_has_run_once in o->result
  fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
  fsmonitor--daemon: cd out of worktree root
  fsmonitor--daemon: prepare for adding health thread
  fsmonitor--daemon: rename listener thread related variables
  fsmonitor--daemon: stub in health thread
  fsm-health-win32: add polling framework to monitor daemon health
  fsm-health-win32: force shutdown daemon if worktree root moves
  fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  fsmonitor: optimize processing of directory events
  t7527: FSMonitor tests for directory moves
  t/perf/p7527: add perf test for builtin FSMonitor
  fsmonitor: never set CE_FSMONITOR_VALID on submodules
  t7527: test FSMonitor on case insensitive+preserving file system
  fsmonitor: on macOS also emit NFC spelling for NFD pathname
  t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  t7527: test Unicode NFC/NFD handling on MacOS

 Makefile                               |  19 +-
 builtin/fsmonitor--daemon.c            | 104 ++++++-
 builtin/update-index.c                 |   4 +
 compat/fsmonitor/fsm-health-darwin.c   |  24 ++
 compat/fsmonitor/fsm-health-win32.c    | 278 +++++++++++++++++
 compat/fsmonitor/fsm-health.h          |  47 +++
 compat/fsmonitor/fsm-listen-darwin.c   | 122 ++++++--
 compat/fsmonitor/fsm-listen-win32.c    | 413 ++++++++++++++++++++-----
 compat/fsmonitor/fsm-listen.h          |   2 +-
 compat/fsmonitor/fsm-settings-darwin.c |  89 ++++++
 compat/fsmonitor/fsm-settings-win32.c  | 137 ++++++++
 config.mak.uname                       |   5 +
 contrib/buildsystems/CMakeLists.txt    |   8 +
 fsmonitor--daemon.h                    |  11 +-
 fsmonitor-settings.c                   |  92 ++++++
 fsmonitor-settings.h                   |  29 ++
 fsmonitor.c                            |  73 ++++-
 fsmonitor.h                            |  11 +
 t/helper/test-fsmonitor-client.c       | 106 +++++++
 t/lib-unicode-nfc-nfd.sh               | 167 ++++++++++
 t/perf/p7527-builtin-fsmonitor.sh      | 257 +++++++++++++++
 t/t7519-status-fsmonitor.sh            |  32 ++
 t/t7527-builtin-fsmonitor.sh           | 311 +++++++++++++++++++
 unpack-trees.c                         |   1 +
 24 files changed, 2218 insertions(+), 124 deletions(-)
 create mode 100644 compat/fsmonitor/fsm-health-darwin.c
 create mode 100644 compat/fsmonitor/fsm-health-win32.c
 create mode 100644 compat/fsmonitor/fsm-health.h
 create mode 100644 compat/fsmonitor/fsm-settings-darwin.c
 create mode 100644 compat/fsmonitor/fsm-settings-win32.c
 create mode 100755 t/lib-unicode-nfc-nfd.sh
 create mode 100755 t/perf/p7527-builtin-fsmonitor.sh


base-commit: e6cf84dc8eb4933220187849b84e5cb96cda185f
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/1143

Range-diff vs v2:

  1:  34619e0652b !  1:  779a15b38e8 fsm-listen-win32: handle shortnames
     @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTI
      @@ compat/fsmonitor/fsm-listen-win32.c: static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
       			goto normalize;
       		if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {
     - 			error("[GLE %ld] could not convert path to UTF-8: '%.*ls'",
     + 			error(_("[GLE %ld] could not convert path to UTF-8: '%.*ls'"),
      -			      GetLastError(),
      -			      (int)(info->FileNameLength / sizeof(WCHAR)),
      -			      info->FileName);
     @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon stop (rename
      +
      +	git init test_implicit_1s &&
      +
     -+	start_daemon test_implicit_1s &&
     ++	start_daemon -C test_implicit_1s &&
      +
      +	# renaming the .git directory will implicitly stop the daemon.
      +	# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.
     @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'implicit daemon stop (rename
      +	test_path_is_file test_implicit_1s2/GIT~1 &&
      +	test_path_is_dir  test_implicit_1s2/GIT~2 &&
      +
     -+	start_daemon test_implicit_1s2 &&
     ++	start_daemon -C test_implicit_1s2 &&
      +
      +	# renaming the .git directory will implicitly stop the daemon.
      +	# the rename-from FS Event will contain the shortname.
  2:  3a0f30b849a !  2:  11d4a17b692 t7527: test FSMonitor on repos with Unicode root paths
     @@ t/t7527-builtin-fsmonitor.sh: do
      +		git -C "$u" add file1 &&
      +		git -C "$u" config core.fsmonitor true &&
      +
     -+		start_daemon "$u" &&
     ++		start_daemon -C "$u" &&
      +		git -C "$u" status >actual &&
      +		grep "new file:   file1" actual
      +	'
  3:  87d1c0b6f2a !  3:  901fa32f6ea t/helper/fsmonitor-client: create stress test
     @@ t/helper/test-fsmonitor-client.c: static int do_send_flush(void)
      +	int nr_requests = 1;
       
       	const char * const fsmonitor_client_usage[] = {
     - 		N_("test-helper fsmonitor-client query [<token>]"),
     - 		N_("test-helper fsmonitor-client flush"),
     -+		N_("test-helper fsmonitor-client hammer [<token>] [<threads>] [<requests>]"),
     + 		"test-tool fsmonitor-client query [<token>]",
     + 		"test-tool fsmonitor-client flush",
     ++		"test-tool fsmonitor-client hammer [<token>] [<threads>] [<requests>]",
       		NULL,
       	};
       
       	struct option options[] = {
     - 		OPT_STRING(0, "token", &token, N_("token"),
     - 			   N_("command token to send to the server")),
     + 		OPT_STRING(0, "token", &token, "token",
     + 			   "command token to send to the server"),
      +
     -+		OPT_INTEGER(0, "threads", &nr_threads, N_("number of client threads")),
     -+		OPT_INTEGER(0, "requests", &nr_requests, N_("number of requests per thread")),
     ++		OPT_INTEGER(0, "threads", &nr_threads, "number of client threads"),
     ++		OPT_INTEGER(0, "requests", &nr_requests, "number of requests per thread"),
      +
       		OPT_END()
       	};
  4:  8c4f90ae4fd !  4:  a8f0b2a5256 fsmonitor-settings: bare repos are incompatible with FSMonitor
     @@ builtin/fsmonitor--daemon.c: int cmd_fsmonitor__daemon(int argc, const char **ar
      +	prepare_repo_settings(the_repository);
      +	fsm_settings__set_ipc(the_repository);
      +
     -+	if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) {
     -+		const char *msg = fsm_settings__get_reason_msg(the_repository);
     -+
     -+		return error("%s '%s'", msg ? msg : "???", xgetcwd());
     -+	}
     ++	if (fsm_settings__error_if_incompatible(the_repository))
     ++		return 1;
      +
       	if (!strcmp(subcmd, "start"))
       		return !!try_to_start_background_daemon();
     @@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const
       	if (fsmonitor > 0) {
       		enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
      +
     -+		if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) {
     -+			const char *msg = fsm_settings__get_reason_msg(r);
     -+
     -+			return error("%s '%s'", msg ? msg : "???", xgetcwd());
     -+		}
     ++		if (fsm_settings__error_if_incompatible(the_repository))
     ++			return 1;
      +
       		if (fsm_mode == FSMONITOR_MODE_DISABLED) {
     - 			advise(_("core.fsmonitor is unset; "
     - 				 "set it if you really want to "
     + 			warning(_("core.fsmonitor is unset; "
     + 				"set it if you really want to "
      
       ## fsmonitor-settings.c ##
      @@
     @@ fsmonitor-settings.c
       static void lookup_fsmonitor_settings(struct repository *r)
       {
       	struct fsmonitor_settings *s;
     +@@ fsmonitor-settings.c: static void lookup_fsmonitor_settings(struct repository *r)
     + 
     + 	CALLOC_ARRAY(s, 1);
     + 	s->mode = FSMONITOR_MODE_DISABLED;
     ++	s->reason = FSMONITOR_REASON_OK;
     + 
     + 	r->settings.fsmonitor = s;
     + 
      @@ fsmonitor-settings.c: void fsm_settings__set_ipc(struct repository *r)
       
       	lookup_fsmonitor_settings(r);
     @@ fsmonitor-settings.c: void fsm_settings__set_disabled(struct repository *r)
      +	return r->settings.fsmonitor->reason;
      +}
      +
     -+const char *fsm_settings__get_reason_msg(struct repository *r)
     ++int fsm_settings__error_if_incompatible(struct repository *r)
      +{
      +	enum fsmonitor_reason reason = fsm_settings__get_reason(r);
      +
      +	switch (reason) {
      +	case FSMONITOR_REASON_OK:
     -+		return NULL;
     ++		return 0;
      +
      +	case FSMONITOR_REASON_BARE:
     -+		return _("bare repos are incompatible with fsmonitor");
     ++		error(_("bare repository '%s' is incompatible with fsmonitor"),
     ++		      xgetcwd());
     ++		return 1;
      +	}
      +
     -+	BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
     ++	BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
      +	    reason);
      +}
      
     @@ fsmonitor-settings.h: void fsm_settings__set_disabled(struct repository *r);
       const char *fsm_settings__get_hook_path(struct repository *r);
       
      +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);
     -+const char *fsm_settings__get_reason_msg(struct repository *r);
     ++int fsm_settings__error_if_incompatible(struct repository *r);
      +
       struct fsmonitor_settings;
       
     @@ t/t7519-status-fsmonitor.sh: test_lazy_prereq UNTRACKED_CACHE '
      +	test_must_fail \
      +		git -C ./bare-clone -c core.fsmonitor=foo \
      +			update-index --fsmonitor 2>actual &&
     -+	grep "bare repos are incompatible with fsmonitor" actual &&
     ++	grep "bare repository .* is incompatible with fsmonitor" actual &&
      +
      +	test_must_fail \
      +		git -C ./bare-clone -c core.fsmonitor=true \
      +			update-index --fsmonitor 2>actual &&
     -+	grep "bare repos are incompatible with fsmonitor" actual
     ++	grep "bare repository .* is incompatible with fsmonitor" actual
      +'
      +
      +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
      +	test_when_finished "rm -rf ./bare-clone actual" &&
      +	git init --bare bare-clone &&
      +	test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&
     -+	grep "bare repos are incompatible with fsmonitor" actual
     ++	grep "bare repository .* is incompatible with fsmonitor" actual
      +'
      +
       test_expect_success 'setup' '
  5:  6329328d185 !  5:  e32a8a7ea7a fsmonitor-settings: stub in platform-specific incompatibility checking
     @@ Metadata
      Author: Jeff Hostetler <jeffhost@microsoft.com>
      
       ## Commit message ##
     -    fsmonitor-settings: stub in platform-specific incompatibility checking
     +    fsmonitor-settings: stub in Win32-specific incompatibility checking
      
          Extend generic incompatibility checkout with platform-specific
          mechanism.  Stub in Win32 version.
     @@ fsmonitor-settings.c: static int check_for_incompatible(struct repository *r)
       
      
       ## fsmonitor-settings.h ##
     -@@ fsmonitor-settings.h: const char *fsm_settings__get_reason_msg(struct repository *r);
     +@@ fsmonitor-settings.h: int fsm_settings__error_if_incompatible(struct repository *r);
       
       struct fsmonitor_settings;
       
  6:  fa9e86e7de7 !  6:  5546339d963 fsmonitor-settings: VFS for Git virtual repos are incompatible
     @@ compat/fsmonitor/fsm-settings-win32.c
       }
      
       ## fsmonitor-settings.c ##
     -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)
     - 
     - 	case FSMONITOR_REASON_BARE:
     - 		return _("bare repos are incompatible with fsmonitor");
     +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)
     + 		error(_("bare repository '%s' is incompatible with fsmonitor"),
     + 		      xgetcwd());
     + 		return 1;
      +
      +	case FSMONITOR_REASON_VFS4GIT:
     -+		return _("virtual repos are incompatible with fsmonitor");
     ++		error(_("virtual repository '%s' is incompatible with fsmonitor"),
     ++		      r->worktree);
     ++		return 1;
       	}
       
     - 	BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
     + 	BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
      
       ## fsmonitor-settings.h ##
      @@ fsmonitor-settings.h: enum fsmonitor_mode {
     @@ fsmonitor-settings.h: enum fsmonitor_mode {
      
       ## t/t7519-status-fsmonitor.sh ##
      @@ t/t7519-status-fsmonitor.sh: test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
     - 	grep "bare repos are incompatible with fsmonitor" actual
     + 	grep "bare repository .* is incompatible with fsmonitor" actual
       '
       
      +test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '
     @@ t/t7519-status-fsmonitor.sh: test_expect_success FSMONITOR_DAEMON 'run fsmonitor
      +	test_must_fail git -C ./fake-virtual-clone \
      +			   -c core.virtualfilesystem=true \
      +			   fsmonitor--daemon run 2>actual &&
     -+	grep "virtual repos are incompatible with fsmonitor" actual
     ++	grep "virtual repository .* is incompatible with fsmonitor" actual
      +'
      +
       test_expect_success 'setup' '
  7:  c1802410410 =  7:  1d2877efda0 fsmonitor-settings: stub in macOS-specific incompatibility checking
  8:  e3bfa0bd69d !  8:  06d7f18676d fsmonitor-settings: remote repos on macOS are incompatible
     @@ compat/fsmonitor/fsm-settings-darwin.c
       }
      
       ## fsmonitor-settings.c ##
     -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)
     - 	case FSMONITOR_REASON_BARE:
     - 		return _("bare repos are incompatible with fsmonitor");
     +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)
     + 		      xgetcwd());
     + 		return 1;
       
      +	case FSMONITOR_REASON_ERROR:
     -+		return _("repo incompatible with fsmonitor due to errors");
     ++		error(_("repository '%s' is incompatible with fsmonitor due to errors"),
     ++		      r->worktree);
     ++		return 1;
      +
      +	case FSMONITOR_REASON_REMOTE:
     -+		return _("remote repos are incompatible with fsmonitor");
     ++		error(_("remote repository '%s' is incompatible with fsmonitor"),
     ++		      r->worktree);
     ++		return 1;
      +
       	case FSMONITOR_REASON_VFS4GIT:
     - 		return _("virtual repos are incompatible with fsmonitor");
     - 	}
     + 		error(_("virtual repository '%s' is incompatible with fsmonitor"),
     + 		      r->worktree);
      
       ## fsmonitor-settings.h ##
      @@ fsmonitor-settings.h: enum fsmonitor_mode {
  9:  e32da3118fb =  9:  5ca97f482d0 fsmonitor-settings: remote repos on Windows are incompatible
 27:  e3e01677d93 ! 10:  67151437245 fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible
     @@ compat/fsmonitor/fsm-settings-darwin.c: enum fsmonitor_reason fsm_os__incompatib
       
      
       ## fsmonitor-settings.c ##
     -@@ fsmonitor-settings.c: const char *fsm_settings__get_reason_msg(struct repository *r)
     - 
     - 	case FSMONITOR_REASON_VFS4GIT:
     - 		return _("virtual repos are incompatible with fsmonitor");
     +@@ fsmonitor-settings.c: int fsm_settings__error_if_incompatible(struct repository *r)
     + 		error(_("virtual repository '%s' is incompatible with fsmonitor"),
     + 		      r->worktree);
     + 		return 1;
      +
      +	case FSMONITOR_REASON_NOSOCKETS:
     -+		return _("repo filesystem does not support Unix sockets");
     ++		error(_("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets"),
     ++		      r->worktree);
     ++		return 1;
       	}
       
     - 	BUG("Unhandled case in fsm_settings__get_reason_msg '%d'",
     + 	BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
      
       ## fsmonitor-settings.h ##
      @@ fsmonitor-settings.h: enum fsmonitor_reason {
 10:  f63de4eda31 = 11:  ed1f7231309 unpack-trees: initialize fsmonitor_has_run_once in o->result
 11:  fe305f5f287 = 12:  35c77b854bd fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
 12:  c8f3e251b1f ! 13:  a5affb359c4 fsmonitor--daemon: cd out of worktree root
     @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
      +	 */
      +	home = getenv("HOME");
      +	if (home && *home && chdir(home))
     -+		die_errno("could not cd home '%s'", home);
     ++		die_errno(_("could not cd home '%s'"), home);
      +
       	err = fsmonitor_run_daemon_1(&state);
       
     @@ compat/fsmonitor/fsm-listen-win32.c: static int recv_rdcw_watch(struct one_watch
      +	 * Shutdown if we get any error.
       	 */
       
     - 	error("GetOverlappedResult failed on '%s' [GLE %ld]",
     + 	error(_("GetOverlappedResult failed on '%s' [GLE %ld]"),
      
       ## fsmonitor--daemon.h ##
      @@ fsmonitor--daemon.h: struct fsmonitor_daemon_state {
 13:  71673be2da5 = 14:  087af5dfb63 fsmonitor--daemon: prepare for adding health thread
 14:  5387baaf5d7 ! 15:  e78eb20c1bf fsmonitor--daemon: rename listener thread related variables
     @@ builtin/fsmonitor--daemon.c: static int fsmonitor_run_daemon(void)
       	/* Prepare to (recursively) watch the <worktree-root> directory. */
      
       ## compat/fsmonitor/fsm-listen-darwin.c ##
     -@@ compat/fsmonitor/fsm-listen-darwin.c: void FSEventStreamRelease(FSEventStreamRef stream);
     +@@
       #include "fsm-listen.h"
       #include "fsmonitor--daemon.h"
       
     @@ compat/fsmonitor/fsm-listen-darwin.c: int fsm_listen__ctor(struct fsmonitor_daem
       		NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);
      @@ compat/fsmonitor/fsm-listen-darwin.c: int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
       failed:
     - 	error("Unable to create FSEventStream.");
     + 	error(_("Unable to create FSEventStream."));
       
      -	FREE_AND_NULL(state->backend_data);
      +	FREE_AND_NULL(state->listen_data);
 15:  f78e4ad87c0 = 16:  301fff5296a fsmonitor--daemon: stub in health thread
 16:  bb72f911a05 = 17:  c6b5bdd25e4 fsm-health-win32: add polling framework to monitor daemon health
 17:  baf8c031a97 ! 18:  13d11713a86 fsm-health-win32: force shutdown daemon if worktree root moves
     @@ compat/fsmonitor/fsm-health-win32.c: struct fsm_health_data
      +		return 0;
      +
      +	default:
     -+		die("unhandled case in 'has_worktree_moved': %d",
     ++		die(_("unhandled case in 'has_worktree_moved': %d"),
      +		    (int)ctx);
      +	}
      +
 18:  796b6591393 = 19:  01c1a38c462 fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
 19:  24591920878 = 20:  0f0a5b5ca16 fsmonitor: optimize processing of directory events
 20:  06a32413854 ! 21:  d8218d197ad t7527: FSMonitor tests for directory moves
     @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'setup' '
       	git -c core.fsmonitor=false add . &&
       	test_tick &&
       	git -c core.fsmonitor=false commit -m initial &&
     -@@ t/t7527-builtin-fsmonitor.sh: verify_status () {
     - 	echo HELLO AFTER
     +@@ t/t7527-builtin-fsmonitor.sh: directory_to_file () {
     + 	echo 1 >dir1
       }
       
      +move_directory_contents_deeper() {
     -+	mkdir T1/_new_
     ++	mkdir T1/_new_ &&
      +	mv T1/[A-Z]* T1/_new_
      +}
      +
 21:  4b59013cadd = 22:  79da369dcce t/perf/p7527: add perf test for builtin FSMonitor
 22:  524d449ed64 ! 23:  4ab4306adab fsmonitor: never set CE_FSMONITOR_VALID on submodules
     @@ t/t7527-builtin-fsmonitor.sh: do
      +	git -C super submodule add ../sub ./dir_1/dir_2/sub &&
      +	git -C super commit -m "add sub" &&
      +
     -+	start_daemon super &&
     ++	start_daemon -C super &&
      +	git -C super config core.fsmonitor true &&
      +	git -C super update-index --fsmonitor &&
      +	git -C super status &&
 23:  c7264decaf6 ! 24:  5d0fa19929d t7527: test FSMonitor on case insensitive+preserving file system
     @@ t/t7527-builtin-fsmonitor.sh: test_expect_success "Submodule" '
      +#	test_when_finished "stop_daemon_delete_repo test_insensitive" &&
      +
      +	git init test_insensitive &&
     -+	(
     -+		GIT_TRACE_FSMONITOR="$(pwd)/insensitive.trace" &&
     -+		export GIT_TRACE_FSMONITOR &&
      +
     -+		start_daemon test_insensitive
     -+	) &&
     ++	start_daemon -C test_insensitive -tf "$PWD/insensitive.trace" &&
      +
      +	mkdir -p test_insensitive/abc/def &&
      +	echo xyz >test_insensitive/ABC/DEF/xyz &&
 24:  95b9d4210d2 = 25:  264397e8bd4 fsmonitor: on macOS also emit NFC spelling for NFD pathname
 25:  5a0c1b7a287 ! 26:  e6b621fb766 t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
     @@ t/lib-unicode-nfc-nfd.sh (new)
      +	ls | od -t x1 | grep "64 *5f *cf *89 *cc *94 *cd *82"
      +'
      +
     -+if test $unicode_debug = 1
     ++# The following is for debugging. I found it useful when
     ++# trying to understand the various (OS, FS) quirks WRT
     ++# Unicode and how composition/decomposition is handled.
     ++# For example, when trying to understand how (macOS, APFS)
     ++# and (macOS, HFS) and (macOS, FAT32) compare.
     ++#
     ++# It is rather noisy, so it is disabled by default.
     ++#
     ++if test "$unicode_debug" = "true"
      +then
      +	if test_have_prereq UNICODE_COMPOSITION_SENSITIVE
      +	then
 26:  a45c1fd3000 ! 27:  aa96a849ce4 t7527: test Unicode NFC/NFD handling on MacOS
     @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse
       	egrep "^event: abc/def/xyz$" ./insensitive.trace
       '
       
     -+unicode_debug=0
     ++# The variable "unicode_debug" is defined in the following library
     ++# script to dump information about how the (OS, FS) handles Unicode
     ++# composition.  Uncomment the following line if you want to enable it.
     ++#
     ++# unicode_debug=true
     ++
      +. "$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh"
      +
      +# See if the OS or filesystem does NFC/NFD aliasing/munging.
     @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'case inse
      +	test_when_finished "stop_daemon_delete_repo test_unicode" &&
      +
      +	git init test_unicode &&
     -+	(
     -+		GIT_TRACE_FSMONITOR="$(pwd)/unicode.trace" &&
     -+		export GIT_TRACE_FSMONITOR &&
      +
     -+		start_daemon test_unicode
     -+	) &&
     ++	start_daemon -C test_unicode -tf "$PWD/unicode.trace" &&
      +
      +	# Create a directory using an NFC spelling.
      +	#

-- 
gitgitgadget

^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 01/27] fsm-listen-win32: handle shortnames
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 02/27] t7527: test FSMonitor on repos with Unicode root paths Jeff Hostetler via GitGitGadget
                       ` (27 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach FSMonitor daemon on Windows to recognize shortname paths as
aliases of normal longname paths.  FSMonitor clients, such as `git
status`, should receive the longname spelling of changed files (when
possible).

Sometimes we receive FS events using the shortname, such as when a CMD
shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1".  The FS notification
arrives using whatever combination of long and shortnames were used by
the other process.  (Shortnames do seem to be case normalized,
however.)

Use Windows GetLongPathNameW() to try to map the pathname spelling in
the notification event into the normalized longname spelling.  (This
can fail if the file/directory is deleted, moved, or renamed, because
we are asking the FS for the mapping in response to the event and
after it has already happened, but we try.)

Special case the shortname spelling of ".git" to avoid under-reporting
these events.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++-----
 t/t7527-builtin-fsmonitor.sh        |  65 +++++
 2 files changed, 374 insertions(+), 54 deletions(-)

diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index 5b928ab66e5..3f1b68267bd 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -25,6 +25,9 @@ struct one_watch
 	DWORD count;
 
 	struct strbuf path;
+	wchar_t wpath_longname[MAX_PATH + 1];
+	DWORD wpath_longname_len;
+
 	HANDLE hDir;
 	HANDLE hEvent;
 	OVERLAPPED overlapped;
@@ -34,6 +37,21 @@ struct one_watch
 	 * need to later call GetOverlappedResult() and possibly CancelIoEx().
 	 */
 	BOOL is_active;
+
+	/*
+	 * Are shortnames enabled on the containing drive?  This is
+	 * always true for "C:/" drives and usually never true for
+	 * other drives.
+	 *
+	 * We only set this for the worktree because we only need to
+	 * convert shortname paths to longname paths for items we send
+	 * to clients.  (We don't care about shortname expansion for
+	 * paths inside a GITDIR because we never send them to
+	 * clients.)
+	 */
+	BOOL has_shortnames;
+	BOOL has_tilda;
+	wchar_t dotgit_shortname[16]; /* for 8.3 name */
 };
 
 struct fsmonitor_daemon_backend_data
@@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data
 };
 
 /*
- * Convert the WCHAR path from the notification into UTF8 and
- * then normalize it.
+ * Convert the WCHAR path from the event into UTF8 and normalize it.
+ *
+ * `wpath_len` is in WCHARS not bytes.
  */
-static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
+static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len,
 				  struct strbuf *normalized_path)
 {
 	int reserve;
 	int len = 0;
 
 	strbuf_reset(normalized_path);
-	if (!info->FileNameLength)
+	if (!wpath_len)
 		goto normalize;
 
 	/*
@@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
 	 * sequence of 2 UTF8 characters.  That should let us
 	 * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time.
 	 */
-	reserve = info->FileNameLength + 1;
+	reserve = 2 * wpath_len + 1;
 	strbuf_grow(normalized_path, reserve);
 
 	for (;;) {
-		len = WideCharToMultiByte(CP_UTF8, 0, info->FileName,
-					  info->FileNameLength / sizeof(WCHAR),
+		len = WideCharToMultiByte(CP_UTF8, 0,
+					  wpath, wpath_len,
 					  normalized_path->buf,
 					  strbuf_avail(normalized_path) - 1,
 					  NULL, NULL);
@@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info,
 			goto normalize;
 		if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) {
 			error(_("[GLE %ld] could not convert path to UTF-8: '%.*ls'"),
-			      GetLastError(),
-			      (int)(info->FileNameLength / sizeof(WCHAR)),
-			      info->FileName);
+			      GetLastError(), (int)wpath_len, wpath);
 			return -1;
 		}
 
@@ -98,6 +115,152 @@ normalize:
 	return strbuf_normalize_path(normalized_path);
 }
 
+/*
+ * See if the worktree root directory has shortnames enabled.
+ * This will help us decide if we need to do an expensive shortname
+ * to longname conversion on every notification event.
+ *
+ * We do not want to create a file to test this, so we assume that the
+ * root directory contains a ".git" file or directory.  (Our caller
+ * only calls us for the worktree root, so this should be fine.)
+ *
+ * Remember the spelling of the shortname for ".git" if it exists.
+ */
+static void check_for_shortnames(struct one_watch *watch)
+{
+	wchar_t buf_in[MAX_PATH + 1];
+	wchar_t buf_out[MAX_PATH + 1];
+	wchar_t *last_slash = NULL;
+	wchar_t *last_bslash = NULL;
+	wchar_t *last;
+
+	/* build L"<wt-root-path>/.git" */
+	wcscpy(buf_in, watch->wpath_longname);
+	wcscpy(buf_in + watch->wpath_longname_len, L".git");
+
+	if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH))
+		return;
+
+	last_slash = wcsrchr(buf_out, L'/');
+	last_bslash = wcsrchr(buf_out, L'\\');
+	if (last_slash > last_bslash)
+		last = last_slash + 1;
+	else if (last_bslash)
+		last = last_bslash + 1;
+	else
+		last = buf_out;
+
+	if (!wcscmp(last, L".git"))
+		return;
+
+	watch->has_shortnames = 1;
+	wcsncpy(watch->dotgit_shortname, last,
+		ARRAY_SIZE(watch->dotgit_shortname));
+
+	/*
+	 * The shortname for ".git" is usually of the form "GIT~1", so
+	 * we should be able to avoid shortname to longname mapping on
+	 * every notification event if the source string does not
+	 * contain a "~".
+	 *
+	 * However, the documentation for GetLongPathNameW() says
+	 * that there are filesystems that don't follow that pattern
+	 * and warns against this optimization.
+	 *
+	 * Lets test this.
+	 */
+	if (wcschr(watch->dotgit_shortname, L'~'))
+		watch->has_tilda = 1;
+}
+
+enum get_relative_result {
+	GRR_NO_CONVERSION_NEEDED,
+	GRR_HAVE_CONVERSION,
+	GRR_SHUTDOWN,
+};
+
+/*
+ * Info notification paths are relative to the root of the watch.
+ * If our CWD is still at the root, then we can use relative paths
+ * to convert from shortnames to longnames.  If our process has a
+ * different CWD, then we need to construct an absolute path, do
+ * the conversion, and then return the root-relative portion.
+ *
+ * We use the longname form of the root as our basis and assume that
+ * it already has a trailing slash.
+ *
+ * `wpath_len` is in WCHARS not bytes.
+ */
+static enum get_relative_result get_relative_longname(
+	struct one_watch *watch,
+	const wchar_t *wpath, DWORD wpath_len,
+	wchar_t *wpath_longname)
+{
+	wchar_t buf_in[2 * MAX_PATH + 1];
+	wchar_t buf_out[MAX_PATH + 1];
+	DWORD root_len;
+
+	/* Build L"<wt-root-path>/<event-rel-path>" */
+	root_len = watch->wpath_longname_len;
+	wcsncpy(buf_in, watch->wpath_longname, root_len);
+	wcsncpy(buf_in + root_len, wpath, wpath_len);
+	buf_in[root_len + wpath_len] = 0;
+
+	/*
+	 * We don't actually know if the source pathname is a
+	 * shortname or a longname.  This routine allows either to be
+	 * given as input.
+	 */
+	if (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) {
+		/*
+		 * The shortname to longname conversion can fail for
+		 * various reasons, for example if the file has been
+		 * deleted.  (That is, if we just received a
+		 * delete-file notification event and the file is
+		 * already gone, we can't ask the file system to
+		 * lookup the longname for it.  Likewise, for moves
+		 * and renames where we are given the old name.)
+		 *
+		 * Since deleting or moving a file or directory by its
+		 * shortname is rather obscure, I'm going ignore the
+		 * failure and ask the caller to report the original
+		 * relative path.  This seems kinder than failing here
+		 * and forcing a resync.  Besides, forcing a resync on
+		 * every file/directory delete would effectively
+		 * cripple monitoring.
+		 *
+		 * We might revisit this in the future.
+		 */
+		return GRR_NO_CONVERSION_NEEDED;
+	}
+
+	if (!wcscmp(buf_in, buf_out)) {
+		/*
+		 * The path does not have a shortname alias.
+		 */
+		return GRR_NO_CONVERSION_NEEDED;
+	}
+
+	if (wcsncmp(buf_in, buf_out, root_len)) {
+		/*
+		 * The spelling of the root directory portion of the computed
+		 * longname has changed.  This should not happen.  Basically,
+		 * it means that we don't know where (without recomputing the
+		 * longname of just the root directory) to split out the
+		 * relative path.  Since this should not happen, I'm just
+		 * going to let this fail and force a shutdown (because all
+		 * subsequent events are probably going to see the same
+		 * mismatch).
+		 */
+		return GRR_SHUTDOWN;
+	}
+
+	/* Return the worktree root-relative portion of the longname. */
+
+	wcscpy(wpath_longname, buf_out + root_len);
+	return GRR_HAVE_CONVERSION;
+}
+
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
 	SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);
@@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 	DWORD share_mode =
 		FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;
 	HANDLE hDir;
-	wchar_t wpath[MAX_PATH];
+	DWORD len_longname;
+	wchar_t wpath[MAX_PATH + 1];
+	wchar_t wpath_longname[MAX_PATH + 1];
 
 	if (xutftowcs_path(wpath, path) < 0) {
 		error(_("could not convert to wide characters: '%s'"), path);
@@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 		return NULL;
 	}
 
+	if (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) {
+		error(_("[GLE %ld] could not get longname of '%s'"),
+		      GetLastError(), path);
+		CloseHandle(hDir);
+		return NULL;
+	}
+
+	len_longname = wcslen(wpath_longname);
+	if (wpath_longname[len_longname - 1] != L'/' &&
+	    wpath_longname[len_longname - 1] != L'\\') {
+		wpath_longname[len_longname++] = L'/';
+		wpath_longname[len_longname] = 0;
+	}
+
 	CALLOC_ARRAY(watch, 1);
 
 	watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */
@@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
 	strbuf_init(&watch->path, 0);
 	strbuf_addstr(&watch->path, path);
 
+	wcscpy(watch->wpath_longname, wpath_longname);
+	watch->wpath_longname_len = len_longname;
+
 	watch->hDir = hDir;
 	watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
 
@@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch)
 	watch->is_active = FALSE;
 }
 
+/*
+ * Process a single relative pathname event.
+ * Return 1 if we should shutdown.
+ */
+static int process_1_worktree_event(
+	struct string_list *cookie_list,
+	struct fsmonitor_batch **batch,
+	const struct strbuf *path,
+	enum fsmonitor_path_type t,
+	DWORD info_action)
+{
+	const char *slash;
+
+	switch (t) {
+	case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
+		/* special case cookie files within .git */
+
+		/* Use just the filename of the cookie file. */
+		slash = find_last_dir_sep(path->buf);
+		string_list_append(cookie_list,
+				   slash ? slash + 1 : path->buf);
+		break;
+
+	case IS_INSIDE_DOT_GIT:
+		/* ignore everything inside of "<worktree>/.git/" */
+		break;
+
+	case IS_DOT_GIT:
+		/* "<worktree>/.git" was deleted (or renamed away) */
+		if ((info_action == FILE_ACTION_REMOVED) ||
+		    (info_action == FILE_ACTION_RENAMED_OLD_NAME)) {
+			trace2_data_string("fsmonitor", NULL,
+					   "fsm-listen/dotgit",
+					   "removed");
+			return 1;
+		}
+		break;
+
+	case IS_WORKDIR_PATH:
+		/* queue normal pathname */
+		if (!*batch)
+			*batch = fsmonitor_batch__new();
+		fsmonitor_batch__add_path(*batch, path->buf);
+		break;
+
+	case IS_GITDIR:
+	case IS_INSIDE_GITDIR:
+	case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
+	default:
+		BUG("unexpected path classification '%d' for '%s'",
+		    t, path->buf);
+	}
+
+	return 0;
+}
+
 /*
  * Process filesystem events that happen anywhere (recursively) under the
  * <worktree> root directory.  For a normal working directory, this includes
@@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
 	struct fsmonitor_batch *batch = NULL;
 	const char *p = watch->buffer;
+	wchar_t wpath_longname[MAX_PATH + 1];
 
 	/*
 	 * If the kernel gets more events than will fit in the kernel
@@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state)
 	 */
 	for (;;) {
 		FILE_NOTIFY_INFORMATION *info = (void *)p;
-		const char *slash;
+		wchar_t *wpath = info->FileName;
+		DWORD wpath_len = info->FileNameLength / sizeof(WCHAR);
 		enum fsmonitor_path_type t;
+		enum get_relative_result grr;
+
+		if (watch->has_shortnames) {
+			if (!wcscmp(wpath, watch->dotgit_shortname)) {
+				/*
+				 * This event exactly matches the
+				 * spelling of the shortname of
+				 * ".git", so we can skip some steps.
+				 *
+				 * (This case is odd because the user
+				 * can "rm -rf GIT~1" and we cannot
+				 * use the filesystem to map it back
+				 * to ".git".)
+				 */
+				strbuf_reset(&path);
+				strbuf_addstr(&path, ".git");
+				t = IS_DOT_GIT;
+				goto process_it;
+			}
 
-		strbuf_reset(&path);
-		if (normalize_path_in_utf8(info, &path) == -1)
-			goto skip_this_path;
-
-		t = fsmonitor_classify_path_workdir_relative(path.buf);
-
-		switch (t) {
-		case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
-			/* special case cookie files within .git */
-
-			/* Use just the filename of the cookie file. */
-			slash = find_last_dir_sep(path.buf);
-			string_list_append(&cookie_list,
-					   slash ? slash + 1 : path.buf);
-			break;
-
-		case IS_INSIDE_DOT_GIT:
-			/* ignore everything inside of "<worktree>/.git/" */
-			break;
+			if (watch->has_tilda && !wcschr(wpath, L'~')) {
+				/*
+				 * Shortnames on this filesystem have tildas
+				 * and the notification path does not have
+				 * one, so we assume that it is a longname.
+				 */
+				goto normalize_it;
+			}
 
-		case IS_DOT_GIT:
-			/* "<worktree>/.git" was deleted (or renamed away) */
-			if ((info->Action == FILE_ACTION_REMOVED) ||
-			    (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) {
-				trace2_data_string("fsmonitor", NULL,
-						   "fsm-listen/dotgit",
-						   "removed");
+			grr = get_relative_longname(watch, wpath, wpath_len,
+						    wpath_longname);
+			switch (grr) {
+			case GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */
+				break;
+			case GRR_HAVE_CONVERSION:
+				wpath = wpath_longname;
+				wpath_len = wcslen(wpath);
+				break;
+			default:
+			case GRR_SHUTDOWN:
 				goto force_shutdown;
 			}
-			break;
+		}
 
-		case IS_WORKDIR_PATH:
-			/* queue normal pathname */
-			if (!batch)
-				batch = fsmonitor_batch__new();
-			fsmonitor_batch__add_path(batch, path.buf);
-			break;
+normalize_it:
+		if (normalize_path_in_utf8(wpath, wpath_len, &path) == -1)
+			goto skip_this_path;
 
-		case IS_GITDIR:
-		case IS_INSIDE_GITDIR:
-		case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
-		default:
-			BUG("unexpected path classification '%d' for '%s'",
-			    t, path.buf);
-		}
+		t = fsmonitor_classify_path_workdir_relative(path.buf);
+
+process_it:
+		if (process_1_worktree_event(&cookie_list, &batch, &path, t,
+					     info->Action))
+			goto force_shutdown;
 
 skip_this_path:
 		if (!info->NextEntryOffset)
@@ -382,6 +630,9 @@ force_shutdown:
  * Note that we DO NOT get filesystem events on the external <gitdir>
  * itself (it is not inside something that we are watching).  In particular,
  * we do not get an event if the external <gitdir> is deleted.
+ *
+ * Also, we do not care about shortnames within the external <gitdir>, since
+ * we never send these paths to clients.
  */
 static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 {
@@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 		const char *slash;
 		enum fsmonitor_path_type t;
 
-		strbuf_reset(&path);
-		if (normalize_path_in_utf8(info, &path) == -1)
+		if (normalize_path_in_utf8(
+			    info->FileName,
+			    info->FileNameLength / sizeof(WCHAR),
+			    &path) == -1)
 			goto skip_this_path;
 
 		t = fsmonitor_classify_path_gitdir_relative(path.buf);
@@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 	if (!data->watch_worktree)
 		goto failed;
 
+	check_for_shortnames(data->watch_worktree);
+
 	if (state->nr_paths_watching > 1) {
 		data->watch_gitdir = create_watch(state,
 						  state->path_gitdir_watch.buf);
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 389ebf431c6..d28a74feeb9 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -173,6 +173,71 @@ test_expect_success 'implicit daemon stop (rename .git)' '
 	test_must_fail git -C test_implicit_2 fsmonitor--daemon status
 '
 
+# File systems on Windows may or may not have shortnames.
+# This is a volume-specific setting on modern systems.
+# "C:/" drives are required to have them enabled.  Other
+# hard drives default to disabled.
+#
+# This is a crude test to see if shortnames are enabled
+# on the volume containing the test directory.  It is
+# crude, but it does not require elevation like `fsutil`.
+#
+test_lazy_prereq SHORTNAMES '
+	mkdir .foo &&
+	test -d "FOO~1"
+'
+
+# Here we assume that the shortname of ".git" is "GIT~1".
+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' '
+	test_when_finished "stop_daemon_delete_repo test_implicit_1s" &&
+
+	git init test_implicit_1s &&
+
+	start_daemon -C test_implicit_1s &&
+
+	# renaming the .git directory will implicitly stop the daemon.
+	# this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}.
+	# the rename-from FS Event will contain the shortname.
+	#
+	mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz &&
+
+	sleep 1 &&
+	# put it back so that our status will not crawl out to our
+	# parent directory.
+	# this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}.
+	mv test_implicit_1s/.gitxyz test_implicit_1s/.git &&
+
+	test_must_fail git -C test_implicit_1s fsmonitor--daemon status
+'
+
+# Here we first create a file with LONGNAME of "GIT~1" before
+# we create the repo.  This will cause the shortname of ".git"
+# to be "GIT~2".
+test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' '
+	test_when_finished "stop_daemon_delete_repo test_implicit_1s2" &&
+
+	mkdir test_implicit_1s2 &&
+	echo HELLO >test_implicit_1s2/GIT~1 &&
+	git init test_implicit_1s2 &&
+
+	test_path_is_file test_implicit_1s2/GIT~1 &&
+	test_path_is_dir  test_implicit_1s2/GIT~2 &&
+
+	start_daemon -C test_implicit_1s2 &&
+
+	# renaming the .git directory will implicitly stop the daemon.
+	# the rename-from FS Event will contain the shortname.
+	#
+	mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz &&
+
+	sleep 1 &&
+	# put it back so that our status will not crawl out to our
+	# parent directory.
+	mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git &&
+
+	test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status
+'
+
 test_expect_success 'cannot start multiple daemons' '
 	test_when_finished "stop_daemon_delete_repo test_multiple" &&
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 02/27] t7527: test FSMonitor on repos with Unicode root paths
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 01/27] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 03/27] t/helper/fsmonitor-client: create stress test Jeff Hostetler via GitGitGadget
                       ` (26 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create some test repos with UTF8 characters in the pathname of the
root directory and verify that the builtin FSMonitor can watch them.

This test is mainly for Windows where we need to avoid `*A()`
routines.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index d28a74feeb9..429029fcadd 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -656,4 +656,27 @@ do
 	done
 done
 
+# Test Unicode UTF-8 characters in the pathname of the working
+# directory root.  Use of "*A()" routines rather than "*W()" routines
+# on Windows can sometimes lead to odd failures.
+#
+u1=$(printf "u_c3_a6__\xC3\xA6")
+u2=$(printf "u_e2_99_ab__\xE2\x99\xAB")
+u_values="$u1 $u2"
+for u in $u_values
+do
+	test_expect_success "Unicode in repo root path: $u" '
+		test_when_finished "stop_daemon_delete_repo $u" &&
+
+		git init "$u" &&
+		echo 1 >"$u"/file1 &&
+		git -C "$u" add file1 &&
+		git -C "$u" config core.fsmonitor true &&
+
+		start_daemon -C "$u" &&
+		git -C "$u" status >actual &&
+		grep "new file:   file1" actual
+	'
+done
+
 test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 03/27] t/helper/fsmonitor-client: create stress test
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 01/27] fsm-listen-win32: handle shortnames Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 02/27] t7527: test FSMonitor on repos with Unicode root paths Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
                       ` (25 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create a stress test to hammer on the fsmonitor daemon.
Create a client-side thread pool of n threads and have
each of them make m requests as fast as they can.

We do not currently inspect the contents of the response.
We're only interested in placing a heavy request load on
the daemon.

This test is useful for interactive testing and various
experimentation.  For example, to place additional load
on the daemon while another test is running.  We currently
do not have a test script that actually uses this helper.
We might add such a test in the future.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++
 1 file changed, 106 insertions(+)

diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c
index 3062c8a3c2b..54a4856c48c 100644
--- a/t/helper/test-fsmonitor-client.c
+++ b/t/helper/test-fsmonitor-client.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "parse-options.h"
 #include "fsmonitor-ipc.h"
+#include "thread-utils.h"
+#include "trace2.h"
 
 #ifndef HAVE_FSMONITOR_DAEMON_BACKEND
 int cmd__fsmonitor_client(int argc, const char **argv)
@@ -79,20 +81,121 @@ static int do_send_flush(void)
 	return 0;
 }
 
+struct hammer_thread_data
+{
+	pthread_t pthread_id;
+	int thread_nr;
+
+	int nr_requests;
+	const char *token;
+
+	int sum_successful;
+	int sum_errors;
+};
+
+static void *hammer_thread_proc(void *_hammer_thread_data)
+{
+	struct hammer_thread_data *data = _hammer_thread_data;
+	struct strbuf answer = STRBUF_INIT;
+	int k;
+	int ret;
+
+	trace2_thread_start("hammer");
+
+	for (k = 0; k < data->nr_requests; k++) {
+		strbuf_reset(&answer);
+
+		ret = fsmonitor_ipc__send_query(data->token, &answer);
+		if (ret < 0)
+			data->sum_errors++;
+		else
+			data->sum_successful++;
+	}
+
+	strbuf_release(&answer);
+	trace2_thread_exit();
+	return NULL;
+}
+
+/*
+ * Start a pool of client threads that will each send a series of
+ * commands to the daemon.
+ *
+ * The goal is to overload the daemon with a sustained series of
+ * concurrent requests.
+ */
+static int do_hammer(const char *token, int nr_threads, int nr_requests)
+{
+	struct hammer_thread_data *data = NULL;
+	int k;
+	int sum_join_errors = 0;
+	int sum_commands = 0;
+	int sum_errors = 0;
+
+	if (!token || !*token)
+		token = get_token_from_index();
+	if (nr_threads < 1)
+		nr_threads = 1;
+	if (nr_requests < 1)
+		nr_requests = 1;
+
+	CALLOC_ARRAY(data, nr_threads);
+
+	for (k = 0; k < nr_threads; k++) {
+		struct hammer_thread_data *p = &data[k];
+		p->thread_nr = k;
+		p->nr_requests = nr_requests;
+		p->token = token;
+
+		if (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) {
+			warning("failed to create thread[%d] skipping remainder", k);
+			nr_threads = k;
+			break;
+		}
+	}
+
+	for (k = 0; k < nr_threads; k++) {
+		struct hammer_thread_data *p = &data[k];
+
+		if (pthread_join(p->pthread_id, NULL))
+			sum_join_errors++;
+		sum_commands += p->sum_successful;
+		sum_errors += p->sum_errors;
+	}
+
+	fprintf(stderr, "HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\n",
+		nr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors);
+
+	free(data);
+
+	/*
+	 * Return an error if any of the _send_query requests failed.
+	 * We don't care about thread create/join errors.
+	 */
+	return sum_errors > 0;
+}
+
 int cmd__fsmonitor_client(int argc, const char **argv)
 {
 	const char *subcmd;
 	const char *token = NULL;
+	int nr_threads = 1;
+	int nr_requests = 1;
 
 	const char * const fsmonitor_client_usage[] = {
 		"test-tool fsmonitor-client query [<token>]",
 		"test-tool fsmonitor-client flush",
+		"test-tool fsmonitor-client hammer [<token>] [<threads>] [<requests>]",
 		NULL,
 	};
 
 	struct option options[] = {
 		OPT_STRING(0, "token", &token, "token",
 			   "command token to send to the server"),
+
+		OPT_INTEGER(0, "threads", &nr_threads, "number of client threads"),
+		OPT_INTEGER(0, "requests", &nr_requests, "number of requests per thread"),
+
 		OPT_END()
 	};
 
@@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv)
 	if (!strcmp(subcmd, "flush"))
 		return !!do_send_flush();
 
+	if (!strcmp(subcmd, "hammer"))
+		return !!do_hammer(token, nr_threads, nr_requests);
+
 	die("Unhandled subcommand: '%s'", subcmd);
 }
 #endif
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (2 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 03/27] t/helper/fsmonitor-client: create stress test Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 05/27] fsmonitor-settings: stub in Win32-specific incompatibility checking Jeff Hostetler via GitGitGadget
                       ` (24 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Bare repos do not have a worktree, so there is nothing for the
daemon watch.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c |  6 ++++
 builtin/update-index.c      |  4 +++
 fsmonitor-settings.c        | 60 +++++++++++++++++++++++++++++++++++++
 fsmonitor-settings.h        | 12 ++++++++
 t/t7519-status-fsmonitor.sh | 23 ++++++++++++++
 5 files changed, 105 insertions(+)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index a30ecf6cfac..edca8a667ab 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1450,6 +1450,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix)
 		die(_("invalid 'ipc-threads' value (%d)"),
 		    fsmonitor__ipc_threads);
 
+	prepare_repo_settings(the_repository);
+	fsm_settings__set_ipc(the_repository);
+
+	if (fsm_settings__error_if_incompatible(the_repository))
+		return 1;
+
 	if (!strcmp(subcmd, "start"))
 		return !!try_to_start_background_daemon();
 
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 876112abb21..d29048f16f2 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1237,6 +1237,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 
 	if (fsmonitor > 0) {
 		enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
+
+		if (fsm_settings__error_if_incompatible(the_repository))
+			return 1;
+
 		if (fsm_mode == FSMONITOR_MODE_DISABLED) {
 			warning(_("core.fsmonitor is unset; "
 				"set it if you really want to "
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 757d230d538..86c09bd35fe 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -9,9 +9,33 @@
  */
 struct fsmonitor_settings {
 	enum fsmonitor_mode mode;
+	enum fsmonitor_reason reason;
 	char *hook_path;
 };
 
+static void set_incompatible(struct repository *r,
+			     enum fsmonitor_reason reason)
+{
+	struct fsmonitor_settings *s = r->settings.fsmonitor;
+
+	s->mode = FSMONITOR_MODE_INCOMPATIBLE;
+	s->reason = reason;
+}
+
+static int check_for_incompatible(struct repository *r)
+{
+	if (!r->worktree) {
+		/*
+		 * Bare repositories don't have a working directory and
+		 * therefore have nothing to watch.
+		 */
+		set_incompatible(r, FSMONITOR_REASON_BARE);
+		return 1;
+	}
+
+	return 0;
+}
+
 static void lookup_fsmonitor_settings(struct repository *r)
 {
 	struct fsmonitor_settings *s;
@@ -23,6 +47,7 @@ static void lookup_fsmonitor_settings(struct repository *r)
 
 	CALLOC_ARRAY(s, 1);
 	s->mode = FSMONITOR_MODE_DISABLED;
+	s->reason = FSMONITOR_REASON_OK;
 
 	r->settings.fsmonitor = s;
 
@@ -86,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r)
 
 	lookup_fsmonitor_settings(r);
 
+	if (check_for_incompatible(r))
+		return;
+
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 }
@@ -97,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path)
 
 	lookup_fsmonitor_settings(r);
 
+	if (check_for_incompatible(r))
+		return;
+
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 	r->settings.fsmonitor->hook_path = strdup(path);
@@ -110,5 +141,34 @@ void fsm_settings__set_disabled(struct repository *r)
 	lookup_fsmonitor_settings(r);
 
 	r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED;
+	r->settings.fsmonitor->reason = FSMONITOR_REASON_OK;
 	FREE_AND_NULL(r->settings.fsmonitor->hook_path);
 }
+
+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)
+{
+	if (!r)
+		r = the_repository;
+
+	lookup_fsmonitor_settings(r);
+
+	return r->settings.fsmonitor->reason;
+}
+
+int fsm_settings__error_if_incompatible(struct repository *r)
+{
+	enum fsmonitor_reason reason = fsm_settings__get_reason(r);
+
+	switch (reason) {
+	case FSMONITOR_REASON_OK:
+		return 0;
+
+	case FSMONITOR_REASON_BARE:
+		error(_("bare repository '%s' is incompatible with fsmonitor"),
+		      xgetcwd());
+		return 1;
+	}
+
+	BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
+	    reason);
+}
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index a4c5d7b4889..8654edf33d8 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -4,11 +4,20 @@
 struct repository;
 
 enum fsmonitor_mode {
+	FSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */
 	FSMONITOR_MODE_DISABLED = 0,
 	FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor=<hook_path> */
 	FSMONITOR_MODE_IPC = 2,  /* core.fsmonitor=<true> */
 };
 
+/*
+ * Incompatibility reasons.
+ */
+enum fsmonitor_reason {
+	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
+	FSMONITOR_REASON_BARE,
+};
+
 void fsm_settings__set_ipc(struct repository *r);
 void fsm_settings__set_hook(struct repository *r, const char *path);
 void fsm_settings__set_disabled(struct repository *r);
@@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r);
 enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);
 const char *fsm_settings__get_hook_path(struct repository *r);
 
+enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);
+int fsm_settings__error_if_incompatible(struct repository *r);
+
 struct fsmonitor_settings;
 
 #endif /* FSMONITOR_SETTINGS_H */
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index a6308acf006..9a8e21c5608 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE '
 	test $ret -ne 1
 '
 
+# Test that we detect and disallow repos that are incompatible with FSMonitor.
+test_expect_success 'incompatible bare repo' '
+	test_when_finished "rm -rf ./bare-clone actual expect" &&
+	git init --bare bare-clone &&
+
+	test_must_fail \
+		git -C ./bare-clone -c core.fsmonitor=foo \
+			update-index --fsmonitor 2>actual &&
+	grep "bare repository .* is incompatible with fsmonitor" actual &&
+
+	test_must_fail \
+		git -C ./bare-clone -c core.fsmonitor=true \
+			update-index --fsmonitor 2>actual &&
+	grep "bare repository .* is incompatible with fsmonitor" actual
+'
+
+test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
+	test_when_finished "rm -rf ./bare-clone actual" &&
+	git init --bare bare-clone &&
+	test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual &&
+	grep "bare repository .* is incompatible with fsmonitor" actual
+'
+
 test_expect_success 'setup' '
 	mkdir -p .git/hooks &&
 	: >tracked &&
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 05/27] fsmonitor-settings: stub in Win32-specific incompatibility checking
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (3 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible Jeff Hostetler via GitGitGadget
                       ` (23 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Extend generic incompatibility checkout with platform-specific
mechanism.  Stub in Win32 version.

In the existing fsmonitor-settings code we have a way to mark
types of repos as incompatible with fsmonitor (whether via the
hook and IPC APIs).  For example, we do this for bare repos,
since there are no files to watch.

Extend this exclusion mechanism for platform-specific reasons.
This commit just creates the framework and adds a stub for Win32.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 Makefile                              | 13 +++++++++++++
 compat/fsmonitor/fsm-settings-win32.c |  9 +++++++++
 config.mak.uname                      |  4 ++++
 contrib/buildsystems/CMakeLists.txt   |  3 +++
 fsmonitor-settings.c                  | 12 ++++++++++++
 fsmonitor-settings.h                  | 13 +++++++++++++
 6 files changed, 54 insertions(+)
 create mode 100644 compat/fsmonitor/fsm-settings-win32.c

diff --git a/Makefile b/Makefile
index daa21bed6c3..93604fe8ef7 100644
--- a/Makefile
+++ b/Makefile
@@ -475,6 +475,11 @@ all::
 # `compat/fsmonitor/fsm-listen-<name>.c` that implements the
 # `fsm_listen__*()` routines.
 #
+# If your platform has OS-specific ways to tell if a repo is incompatible with
+# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS
+# to the "<name>" of the corresponding `compat/fsmonitor/fsm-settings-<name>.c`
+# that implements the `fsm_os_settings__*()` routines.
+#
 # Define DEVELOPER to enable more compiler warnings. Compiler version
 # and family are auto detected, but could be overridden by defining
 # COMPILER_FEATURES (see config.mak.dev). You can still set
@@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND
 	COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
 endif
 
+ifdef FSMONITOR_OS_SETTINGS
+	COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
+	COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
+endif
+
 ifeq ($(TCLTK_PATH),)
 NO_TCLTK = NoThanks
 endif
@@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE
 ifdef FSMONITOR_DAEMON_BACKEND
 	@echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+
 endif
+ifdef FSMONITOR_OS_SETTINGS
+	@echo FSMONITOR_OS_SETTINGS=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_OS_SETTINGS)))'\' >>$@+
+endif
 ifdef TEST_OUTPUT_DIRECTORY
 	@echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+
 endif
diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
new file mode 100644
index 00000000000..7fce32a3c5b
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -0,0 +1,9 @@
+#include "cache.h"
+#include "config.h"
+#include "repository.h"
+#include "fsmonitor-settings.h"
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
+{
+	return FSMONITOR_REASON_OK;
+}
diff --git a/config.mak.uname b/config.mak.uname
index 501970902da..cf224768ad6 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows)
 	# These are always available, so we do not have to conditionally
 	# support it.
 	FSMONITOR_DAEMON_BACKEND = win32
+	FSMONITOR_OS_SETTINGS = win32
+
 	NO_SVN_TESTS = YesPlease
 	RUNTIME_PREFIX = YesPlease
 	HAVE_WPGMPTR = YesWeDo
@@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW)
 	# These are always available, so we do not have to conditionally
 	# support it.
 	FSMONITOR_DAEMON_BACKEND = win32
+	FSMONITOR_OS_SETTINGS = win32
+
 	RUNTIME_PREFIX = YesPlease
 	HAVE_WPGMPTR = YesWeDo
 	NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index ee0d7257b77..16705da2000 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC)
 	if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+
+		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 86c09bd35fe..8ff55f8c3fd 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r)
 		return 1;
 	}
 
+#ifdef HAVE_FSMONITOR_OS_SETTINGS
+	{
+		enum fsmonitor_reason reason;
+
+		reason = fsm_os__incompatible(r);
+		if (reason != FSMONITOR_REASON_OK) {
+			set_incompatible(r, reason);
+			return 1;
+		}
+	}
+#endif
+
 	return 0;
 }
 
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index 8654edf33d8..4b35f051fb1 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -30,4 +30,17 @@ int fsm_settings__error_if_incompatible(struct repository *r);
 
 struct fsmonitor_settings;
 
+#ifdef HAVE_FSMONITOR_OS_SETTINGS
+/*
+ * Ask platform-specific code whether the repository is incompatible
+ * with fsmonitor (both hook and ipc modes).  For example, if the working
+ * directory is on a remote volume and mounted via a technology that does
+ * not support notification events, then we should not pretend to watch it.
+ *
+ * fsm_os__* routines should considered private to fsm_settings__
+ * routines.
+ */
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r);
+#endif /* HAVE_FSMONITOR_OS_SETTINGS */
+
 #endif /* FSMONITOR_SETTINGS_H */
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (4 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 05/27] fsmonitor-settings: stub in Win32-specific incompatibility checking Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking Jeff Hostetler via GitGitGadget
                       ` (22 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

VFS for Git virtual repositories are incompatible with FSMonitor.

VFS for Git is a downstream fork of Git.  It contains its own custom
file system watcher that is aware of the virtualization.  If a working
directory is being managed by VFS for Git, we should not try to watch
it because we may get incomplete results.

We do not know anything about how VFS for Git works, but we do
know that VFS for Git working directories contain a well-defined
config setting.  If it is set, mark the working directory as
incompatible.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++
 fsmonitor-settings.c                  |  5 +++++
 fsmonitor-settings.h                  |  1 +
 t/t7519-status-fsmonitor.sh           |  9 +++++++++
 4 files changed, 41 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
index 7fce32a3c5b..ee78bba38e3 100644
--- a/compat/fsmonitor/fsm-settings-win32.c
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -3,7 +3,33 @@
 #include "repository.h"
 #include "fsmonitor-settings.h"
 
+/*
+ * VFS for Git is incompatible with FSMonitor.
+ *
+ * Granted, core Git does not know anything about VFS for Git and we
+ * shouldn't make assumptions about a downstream feature, but users
+ * can install both versions.  And this can lead to incorrect results
+ * from core Git commands.  So, without bringing in any of the VFS for
+ * Git code, do a simple config test for a published config setting.
+ * (We do not look at the various *_TEST_* environment variables.)
+ */
+static enum fsmonitor_reason check_vfs4git(struct repository *r)
+{
+	const char *const_str;
+
+	if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str))
+		return FSMONITOR_REASON_VFS4GIT;
+
+	return FSMONITOR_REASON_OK;
+}
+
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
+	enum fsmonitor_reason reason;
+
+	reason = check_vfs4git(r);
+	if (reason != FSMONITOR_REASON_OK)
+		return reason;
+
 	return FSMONITOR_REASON_OK;
 }
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 8ff55f8c3fd..1efb6e17a20 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -179,6 +179,11 @@ int fsm_settings__error_if_incompatible(struct repository *r)
 		error(_("bare repository '%s' is incompatible with fsmonitor"),
 		      xgetcwd());
 		return 1;
+
+	case FSMONITOR_REASON_VFS4GIT:
+		error(_("virtual repository '%s' is incompatible with fsmonitor"),
+		      r->worktree);
+		return 1;
 	}
 
 	BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index 4b35f051fb1..6361fcbf6b0 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -16,6 +16,7 @@ enum fsmonitor_mode {
 enum fsmonitor_reason {
 	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
 	FSMONITOR_REASON_BARE,
+	FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */
 };
 
 void fsm_settings__set_ipc(struct repository *r);
diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh
index 9a8e21c5608..156895f9437 100755
--- a/t/t7519-status-fsmonitor.sh
+++ b/t/t7519-status-fsmonitor.sh
@@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' '
 	grep "bare repository .* is incompatible with fsmonitor" actual
 '
 
+test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' '
+	test_when_finished "rm -rf ./fake-virtual-clone actual" &&
+	git init fake-virtual-clone &&
+	test_must_fail git -C ./fake-virtual-clone \
+			   -c core.virtualfilesystem=true \
+			   fsmonitor--daemon run 2>actual &&
+	grep "virtual repository .* is incompatible with fsmonitor" actual
+'
+
 test_expect_success 'setup' '
 	mkdir -p .git/hooks &&
 	: >tracked &&
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (5 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 08/27] fsmonitor-settings: remote repos on macOS are incompatible Jeff Hostetler via GitGitGadget
                       ` (21 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++
 config.mak.uname                       | 1 +
 contrib/buildsystems/CMakeLists.txt    | 3 +++
 3 files changed, 13 insertions(+)
 create mode 100644 compat/fsmonitor/fsm-settings-darwin.c

diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
new file mode 100644
index 00000000000..7fce32a3c5b
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -0,0 +1,9 @@
+#include "cache.h"
+#include "config.h"
+#include "repository.h"
+#include "fsmonitor-settings.h"
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
+{
+	return FSMONITOR_REASON_OK;
+}
diff --git a/config.mak.uname b/config.mak.uname
index cf224768ad6..cf911d141f2 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin)
 	ifndef NO_PTHREADS
 	ifndef NO_UNIX_SOCKETS
 	FSMONITOR_DAEMON_BACKEND = darwin
+	FSMONITOR_OS_SETTINGS = darwin
 	endif
 	endif
 
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 16705da2000..b8f9f7a0388 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
+
+		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
 	endif()
 endif()
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 08/27] fsmonitor-settings: remote repos on macOS are incompatible
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (6 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 09/27] fsmonitor-settings: remote repos on Windows " Jeff Hostetler via GitGitGadget
                       ` (20 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to detect remote working directories on macOS and mark them as
incompatible with FSMonitor.

With this, `git fsmonitor--daemon run` will error out with a message
like it does for bare repos.

Client commands, like `git status`, will not attempt to start the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++
 fsmonitor-settings.c                   | 10 ++++
 fsmonitor-settings.h                   |  2 +
 3 files changed, 78 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 7fce32a3c5b..fdd762bf79d 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -2,8 +2,74 @@
 #include "config.h"
 #include "repository.h"
 #include "fsmonitor-settings.h"
+#include "fsmonitor.h"
+#include <sys/param.h>
+#include <sys/mount.h>
+
+/*
+ * Remote working directories are problematic for FSMonitor.
+ *
+ * The underlying file system on the server machine and/or the remote
+ * mount type (NFS, SAMBA, etc.) dictates whether notification events
+ * are available at all to remote client machines.
+ *
+ * Kernel differences between the server and client machines also
+ * dictate the how (buffering, frequency, de-dup) the events are
+ * delivered to client machine processes.
+ *
+ * A client machine (such as a laptop) may choose to suspend/resume
+ * and it is unclear (without lots of testing) whether the watcher can
+ * resync after a resume.  We might be able to treat this as a normal
+ * "events were dropped by the kernel" event and do our normal "flush
+ * and resync" --or-- we might need to close the existing (zombie?)
+ * notification fd and create a new one.
+ *
+ * In theory, the above issues need to be addressed whether we are
+ * using the Hook or IPC API.
+ *
+ * For the builtin FSMonitor, we create the Unix domain socket for the
+ * IPC in the .git directory.  If the working directory is remote,
+ * then the socket will be created on the remote file system.  This
+ * can fail if the remote file system does not support UDS file types
+ * (e.g. smbfs to a Windows server) or if the remote kernel does not
+ * allow a non-local process to bind() the socket.  (These problems
+ * could be fixed by moving the UDS out of the .git directory and to a
+ * well-known local directory on the client machine, but care should
+ * be taken to ensure that $HOME is actually local and not a managed
+ * file share.)
+ *
+ * So (for now at least), mark remote working directories as
+ * incompatible.
+ */
+static enum fsmonitor_reason check_remote(struct repository *r)
+{
+	struct statfs fs;
+
+	if (statfs(r->worktree, &fs) == -1) {
+		int saved_errno = errno;
+		trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s",
+				 r->worktree, strerror(saved_errno));
+		errno = saved_errno;
+		return FSMONITOR_REASON_ERROR;
+	}
+
+	trace_printf_key(&trace_fsmonitor,
+			 "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'",
+			 r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);
+
+	if (!(fs.f_flags & MNT_LOCAL))
+		return FSMONITOR_REASON_REMOTE;
+
+	return FSMONITOR_REASON_OK;
+}
 
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
+	enum fsmonitor_reason reason;
+
+	reason = check_remote(r);
+	if (reason != FSMONITOR_REASON_OK)
+		return reason;
+
 	return FSMONITOR_REASON_OK;
 }
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 1efb6e17a20..0a1811ff004 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -180,6 +180,16 @@ int fsm_settings__error_if_incompatible(struct repository *r)
 		      xgetcwd());
 		return 1;
 
+	case FSMONITOR_REASON_ERROR:
+		error(_("repository '%s' is incompatible with fsmonitor due to errors"),
+		      r->worktree);
+		return 1;
+
+	case FSMONITOR_REASON_REMOTE:
+		error(_("remote repository '%s' is incompatible with fsmonitor"),
+		      r->worktree);
+		return 1;
+
 	case FSMONITOR_REASON_VFS4GIT:
 		error(_("virtual repository '%s' is incompatible with fsmonitor"),
 		      r->worktree);
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index 6361fcbf6b0..34391b583b3 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -16,6 +16,8 @@ enum fsmonitor_mode {
 enum fsmonitor_reason {
 	FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */
 	FSMONITOR_REASON_BARE,
+	FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */
+	FSMONITOR_REASON_REMOTE,
 	FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */
 };
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 09/27] fsmonitor-settings: remote repos on Windows are incompatible
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (7 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 08/27] fsmonitor-settings: remote repos on macOS are incompatible Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 10/27] fsmonitor-settings: NTFS and FAT32 on MacOS " Jeff Hostetler via GitGitGadget
                       ` (19 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to detect remote working directories on Windows and mark them as
incompatible with FSMonitor.

With this `git fsmonitor--daemon run` will error out with a message like it
does for bare repos.

Client commands, such as `git status`, will not attempt to start the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++
 1 file changed, 102 insertions(+)

diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
index ee78bba38e3..907655720bb 100644
--- a/compat/fsmonitor/fsm-settings-win32.c
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -2,6 +2,7 @@
 #include "config.h"
 #include "repository.h"
 #include "fsmonitor-settings.h"
+#include "fsmonitor.h"
 
 /*
  * VFS for Git is incompatible with FSMonitor.
@@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)
 	return FSMONITOR_REASON_OK;
 }
 
+/*
+ * Remote working directories are problematic for FSMonitor.
+ *
+ * The underlying file system on the server machine and/or the remote
+ * mount type dictates whether notification events are available at
+ * all to remote client machines.
+ *
+ * Kernel differences between the server and client machines also
+ * dictate the how (buffering, frequency, de-dup) the events are
+ * delivered to client machine processes.
+ *
+ * A client machine (such as a laptop) may choose to suspend/resume
+ * and it is unclear (without lots of testing) whether the watcher can
+ * resync after a resume.  We might be able to treat this as a normal
+ * "events were dropped by the kernel" event and do our normal "flush
+ * and resync" --or-- we might need to close the existing (zombie?)
+ * notification fd and create a new one.
+ *
+ * In theory, the above issues need to be addressed whether we are
+ * using the Hook or IPC API.
+ *
+ * So (for now at least), mark remote working directories as
+ * incompatible.
+ *
+ * Notes for testing:
+ *
+ * (a) Windows allows a network share to be mapped to a drive letter.
+ *     (This is the normal method to access it.)
+ *
+ *     $ NET USE Z: \\server\share
+ *     $ git -C Z:/repo status
+ *
+ * (b) Windows allows a network share to be referenced WITHOUT mapping
+ *     it to drive letter.
+ *
+ *     $ NET USE \\server\share\dir
+ *     $ git -C //server/share/repo status
+ *
+ * (c) Windows allows "SUBST" to create a fake drive mapping to an
+ *     arbitrary path (which may be remote)
+ *
+ *     $ SUBST Q: Z:\repo
+ *     $ git -C Q:/ status
+ *
+ * (d) Windows allows a directory symlink to be created on a local
+ *     file system that points to a remote repo.
+ *
+ *     $ mklink /d ./link //server/share/repo
+ *     $ git -C ./link status
+ */
+static enum fsmonitor_reason check_remote(struct repository *r)
+{
+	wchar_t wpath[MAX_PATH];
+	wchar_t wfullpath[MAX_PATH];
+	size_t wlen;
+	UINT driveType;
+
+	/*
+	 * Do everything in wide chars because the drive letter might be
+	 * a multi-byte sequence.  See win32_has_dos_drive_prefix().
+	 */
+	if (xutftowcs_path(wpath, r->worktree) < 0)
+		return FSMONITOR_REASON_ERROR;
+
+	/*
+	 * GetDriveTypeW() requires a final slash.  We assume that the
+	 * worktree pathname points to an actual directory.
+	 */
+	wlen = wcslen(wpath);
+	if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') {
+		wpath[wlen++] = L'\\';
+		wpath[wlen] = 0;
+	}
+
+	/*
+	 * Normalize the path.  If nothing else, this converts forward
+	 * slashes to backslashes.  This is essential to get GetDriveTypeW()
+	 * correctly handle some UNC "\\server\share\..." paths.
+	 */
+	if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))
+		return FSMONITOR_REASON_ERROR;
+
+	driveType = GetDriveTypeW(wfullpath);
+	trace_printf_key(&trace_fsmonitor,
+			 "DriveType '%s' L'%ls' (%u)",
+			 r->worktree, wfullpath, driveType);
+
+	if (driveType == DRIVE_REMOTE) {
+		trace_printf_key(&trace_fsmonitor,
+				 "check_remote('%s') true",
+				 r->worktree);
+		return FSMONITOR_REASON_REMOTE;
+	}
+
+	return FSMONITOR_REASON_OK;
+}
+
 enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
 	enum fsmonitor_reason reason;
@@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 	if (reason != FSMONITOR_REASON_OK)
 		return reason;
 
+	reason = check_remote(r);
+	if (reason != FSMONITOR_REASON_OK)
+		return reason;
+
 	return FSMONITOR_REASON_OK;
 }
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 10/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (8 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 09/27] fsmonitor-settings: remote repos on Windows " Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 11/27] unpack-trees: initialize fsmonitor_has_run_once in o->result Jeff Hostetler via GitGitGadget
                       ` (18 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

On MacOS mark repos on NTFS or FAT32 volumes as incompatible.

The builtin FSMonitor used Unix domain sockets on MacOS for IPC
with clients.  These sockets are kept in the .git directory.
Unix sockets are not supported by NTFS and FAT32, so the daemon
cannot start up.

Test for this during our compatibility checking so that client
commands do not keep trying to start the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++---
 fsmonitor-settings.c                   |  5 +++++
 fsmonitor-settings.h                   |  1 +
 3 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index fdd762bf79d..efc732c0f31 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -7,7 +7,7 @@
 #include <sys/mount.h>
 
 /*
- * Remote working directories are problematic for FSMonitor.
+ * [1] Remote working directories are problematic for FSMonitor.
  *
  * The underlying file system on the server machine and/or the remote
  * mount type (NFS, SAMBA, etc.) dictates whether notification events
@@ -40,8 +40,16 @@
  *
  * So (for now at least), mark remote working directories as
  * incompatible.
+ *
+ *
+ * [2] FAT32 and NTFS working directories are problematic too.
+ *
+ * The builtin FSMonitor uses a Unix domain socket in the .git
+ * directory for IPC.  These Windows drive formats do not support
+ * Unix domain sockets, so mark them as incompatible for the daemon.
+ *
  */
-static enum fsmonitor_reason check_remote(struct repository *r)
+static enum fsmonitor_reason check_volume(struct repository *r)
 {
 	struct statfs fs;
 
@@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r)
 	if (!(fs.f_flags & MNT_LOCAL))
 		return FSMONITOR_REASON_REMOTE;
 
+	if (!strcmp(fs.f_fstypename, "msdos")) /* aka FAT32 */
+		return FSMONITOR_REASON_NOSOCKETS;
+
+	if (!strcmp(fs.f_fstypename, "ntfs"))
+		return FSMONITOR_REASON_NOSOCKETS;
+
 	return FSMONITOR_REASON_OK;
 }
 
@@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
 {
 	enum fsmonitor_reason reason;
 
-	reason = check_remote(r);
+	reason = check_volume(r);
 	if (reason != FSMONITOR_REASON_OK)
 		return reason;
 
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 0a1811ff004..60d5eaee497 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -194,6 +194,11 @@ int fsm_settings__error_if_incompatible(struct repository *r)
 		error(_("virtual repository '%s' is incompatible with fsmonitor"),
 		      r->worktree);
 		return 1;
+
+	case FSMONITOR_REASON_NOSOCKETS:
+		error(_("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets"),
+		      r->worktree);
+		return 1;
 	}
 
 	BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'",
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index 34391b583b3..23d5676c8c8 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -19,6 +19,7 @@ enum fsmonitor_reason {
 	FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */
 	FSMONITOR_REASON_REMOTE,
 	FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */
+	FSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */
 };
 
 void fsm_settings__set_ipc(struct repository *r);
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 11/27] unpack-trees: initialize fsmonitor_has_run_once in o->result
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (9 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 10/27] fsmonitor-settings: NTFS and FAT32 on MacOS " Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 12/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Jeff Hostetler via GitGitGadget
                       ` (17 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Initialize `o->result.fsmonitor_has_run_once` based upon value
in `o->src_index->fsmonitor_has_run_once` to prevent a second
fsmonitor query during the tree traversal and possibly getting
a skewed view of the working directory.

The checkout code has already talked to the fsmonitor and the
traversal is updating the index as it traverses, so there is
no need to query the fsmonitor.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 unpack-trees.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/unpack-trees.c b/unpack-trees.c
index 360844bda3a..888cff81f9c 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
 
 	o->result.fsmonitor_last_update =
 		xstrdup_or_null(o->src_index->fsmonitor_last_update);
+	o->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once;
 
 	/*
 	 * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 12/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (10 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 11/27] unpack-trees: initialize fsmonitor_has_run_once in o->result Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 13/27] fsmonitor--daemon: cd out of worktree root Jeff Hostetler via GitGitGadget
                       ` (16 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Ignore FSEvents resulting from `xattr` changes.  Git does not care about
xattr's or changes to xattr's, so don't waste time collecting these
events in the daemon nor transmitting them to clients.

Various security tools add xattrs to files and/or directories, such as
to mark them as having been downloaded.  We should ignore these events
since it doesn't affect the content of the file/directory or the normal
meta-data that Git cares about.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 0741fe834c3..14105f45c18 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)
 	if (flag & kFSEventStreamEventFlagItemCloned)
 		strbuf_addstr(&msg, "ItemCloned|");
 
-	trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s",
+	trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s",
 			 path, flag, msg.buf);
 
 	strbuf_release(&msg);
@@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef)
 		ef & kFSEventStreamEventFlagUserDropped);
 }
 
+/*
+ * If an `xattr` change is the only reason we received this event,
+ * then silently ignore it.  Git doesn't care about xattr's.  We
+ * have to be careful here because the kernel can combine multiple
+ * events for a single path.  And because events always have certain
+ * bits set, such as `ItemIsFile` or `ItemIsDir`.
+ *
+ * Return 1 if we should ignore it.
+ */
+static int ef_ignore_xattr(const FSEventStreamEventFlags ef)
+{
+	static const FSEventStreamEventFlags mask =
+		kFSEventStreamEventFlagItemChangeOwner |
+		kFSEventStreamEventFlagItemCreated |
+		kFSEventStreamEventFlagItemFinderInfoMod |
+		kFSEventStreamEventFlagItemInodeMetaMod |
+		kFSEventStreamEventFlagItemModified |
+		kFSEventStreamEventFlagItemRemoved |
+		kFSEventStreamEventFlagItemRenamed |
+		kFSEventStreamEventFlagItemXattrMod |
+		kFSEventStreamEventFlagItemCloned;
+
+	return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);
+}
+
 static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			     void *ctx,
 			     size_t num_of_events,
@@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			continue;
 		}
 
+		if (ef_ignore_xattr(event_flags[k])) {
+			trace_printf_key(&trace_fsmonitor,
+					 "ignore-xattr: '%s', flags=0x%x",
+					 path_k, event_flags[k]);
+			continue;
+		}
+
 		switch (fsmonitor_classify_path_absolute(state, path_k)) {
 
 		case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 13/27] fsmonitor--daemon: cd out of worktree root
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (11 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 12/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 14/27] fsmonitor--daemon: prepare for adding health thread Jeff Hostetler via GitGitGadget
                       ` (15 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach the fsmonitor--daemon to CD outside of the worktree
before starting up.

The common Git startup mechanism causes the CWD of the daemon process
to be in the root of the worktree.  On Windows, this causes the daemon
process to hold a locked handle on the CWD and prevents other
processes from moving or deleting the worktree while the daemon is
running.

CD to HOME before entering main event loops.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c         | 32 +++++++++++++++++++++++++++--
 compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------
 fsmonitor--daemon.h                 |  1 +
 3 files changed, 47 insertions(+), 8 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index edca8a667ab..f17d5ed84c8 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1182,11 +1182,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	 * before we need it.
 	 */
 	if (ipc_server_run_async(&state->ipc_server_data,
-				 fsmonitor_ipc__get_path(), &ipc_opts,
+				 state->path_ipc.buf, &ipc_opts,
 				 handle_client, state))
 		return error_errno(
 			_("could not start IPC thread pool on '%s'"),
-			fsmonitor_ipc__get_path());
+			state->path_ipc.buf);
 
 	/*
 	 * Start the fsmonitor listener thread to collect filesystem
@@ -1221,6 +1221,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 static int fsmonitor_run_daemon(void)
 {
 	struct fsmonitor_daemon_state state;
+	const char *home;
 	int err;
 
 	memset(&state, 0, sizeof(state));
@@ -1290,6 +1291,15 @@ static int fsmonitor_run_daemon(void)
 
 	strbuf_addch(&state.path_cookie_prefix, '/');
 
+	/*
+	 * We create a named-pipe or unix domain socket inside of the
+	 * ".git" directory.  (Well, on Windows, we base our named
+	 * pipe in the NPFS on the absolute path of the git
+	 * directory.)
+	 */
+	strbuf_init(&state.path_ipc, 0);
+	strbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));
+
 	/*
 	 * Confirm that we can create platform-specific resources for the
 	 * filesystem listener before we bother starting all the threads.
@@ -1299,6 +1309,23 @@ static int fsmonitor_run_daemon(void)
 		goto done;
 	}
 
+	/*
+	 * CD out of the worktree root directory.
+	 *
+	 * The common Git startup mechanism causes our CWD to be the
+	 * root of the worktree.  On Windows, this causes our process
+	 * to hold a locked handle on the CWD.  This prevents the
+	 * worktree from being moved or deleted while the daemon is
+	 * running.
+	 *
+	 * We assume that our FS and IPC listener threads have either
+	 * opened all of the handles that they need or will do
+	 * everything using absolute paths.
+	 */
+	home = getenv("HOME");
+	if (home && *home && chdir(home))
+		die_errno(_("could not cd home '%s'"), home);
+
 	err = fsmonitor_run_daemon_1(&state);
 
 done:
@@ -1311,6 +1338,7 @@ done:
 	strbuf_release(&state.path_worktree_watch);
 	strbuf_release(&state.path_gitdir_watch);
 	strbuf_release(&state.path_cookie_prefix);
+	strbuf_release(&state.path_ipc);
 
 	return err;
 }
diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index 3f1b68267bd..c43d92b9620 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch)
 	}
 
 	/*
-	 * NEEDSWORK: If an external <gitdir> is deleted, the above
-	 * returns an error.  I'm not sure that there's anything that
-	 * we can do here other than failing -- the <worktree>/.git
-	 * link file would be broken anyway.  We might try to check
-	 * for that and return a better error message, but I'm not
-	 * sure it is worth it.
+	 * GetOverlappedResult() fails if the watched directory is
+	 * deleted while we were waiting for an overlapped IO to
+	 * complete.  The documentation did not list specific errors,
+	 * but I observed ERROR_ACCESS_DENIED (0x05) errors during
+	 * testing.
+	 *
+	 * Note that we only get notificaiton events for events
+	 * *within* the directory, not *on* the directory itself.
+	 * (These might be properies of the parent directory, for
+	 * example).
+	 *
+	 * NEEDSWORK: We might try to check for the deleted directory
+	 * case and return a better error message, but I'm not sure it
+	 * is worth it.
+	 *
+	 * Shutdown if we get any error.
 	 */
 
 	error(_("GetOverlappedResult failed on '%s' [GLE %ld]"),
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index bd09fffc176..223c2131b58 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -54,6 +54,7 @@ struct fsmonitor_daemon_state {
 	struct fsmonitor_daemon_backend_data *backend_data;
 
 	struct ipc_server_data *ipc_server_data;
+	struct strbuf path_ipc;
 };
 
 /*
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 14/27] fsmonitor--daemon: prepare for adding health thread
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (12 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 13/27] fsmonitor--daemon: cd out of worktree root Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 15/27] fsmonitor--daemon: rename listener thread related variables Jeff Hostetler via GitGitGadget
                       ` (14 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Refactor daemon thread startup to make it easier to start
a third thread class to monitor the health of the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index f17d5ed84c8..8033916aa63 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1175,6 +1175,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 		 */
 		.uds_disallow_chdir = 0
 	};
+	int listener_started = 0;
+	int err = 0;
 
 	/*
 	 * Start the IPC thread pool before the we've started the file
@@ -1195,15 +1197,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	if (pthread_create(&state->listener_thread, NULL,
 			   fsm_listen__thread_proc, state) < 0) {
 		ipc_server_stop_async(state->ipc_server_data);
-		ipc_server_await(state->ipc_server_data);
-
-		return error(_("could not start fsmonitor listener thread"));
+		err = error(_("could not start fsmonitor listener thread"));
+		goto cleanup;
 	}
+	listener_started = 1;
 
 	/*
 	 * The daemon is now fully functional in background threads.
+	 * Our primary thread should now just wait while the threads
+	 * do all the work.
+	 */
+cleanup:
+	/*
 	 * Wait for the IPC thread pool to shutdown (whether by client
-	 * request or from filesystem activity).
+	 * request, from filesystem activity, or an error).
 	 */
 	ipc_server_await(state->ipc_server_data);
 
@@ -1212,10 +1219,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	 * event from the IPC thread pool, but it doesn't hurt to tell
 	 * it again.  And wait for it to shutdown.
 	 */
-	fsm_listen__stop_async(state);
-	pthread_join(state->listener_thread, NULL);
+	if (listener_started) {
+		fsm_listen__stop_async(state);
+		pthread_join(state->listener_thread, NULL);
+	}
 
-	return state->error_code;
+	if (err)
+		return err;
+	if (state->error_code)
+		return state->error_code;
+	return 0;
 }
 
 static int fsmonitor_run_daemon(void)
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 15/27] fsmonitor--daemon: rename listener thread related variables
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (13 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 14/27] fsmonitor--daemon: prepare for adding health thread Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 16/27] fsmonitor--daemon: stub in health thread Jeff Hostetler via GitGitGadget
                       ` (13 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Rename platform-specific listener thread related variables
and data types as we prepare to add another backend thread
type.

[] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data`
[] `state->backend_data` becomes `state->listen_data`
[] `state->error_code` becomes `state->listen_error_code`

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 builtin/fsmonitor--daemon.c          |  6 +++---
 compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++--------------
 compat/fsmonitor/fsm-listen-win32.c  | 28 +++++++++++++-------------
 compat/fsmonitor/fsm-listen.h        |  2 +-
 fsmonitor--daemon.h                  |  6 +++---
 5 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 8033916aa63..af91bc2fb4c 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1226,8 +1226,8 @@ cleanup:
 
 	if (err)
 		return err;
-	if (state->error_code)
-		return state->error_code;
+	if (state->listen_error_code)
+		return state->listen_error_code;
 	return 0;
 }
 
@@ -1242,7 +1242,7 @@ static int fsmonitor_run_daemon(void)
 	hashmap_init(&state.cookies, cookies_cmp, NULL, 0);
 	pthread_mutex_init(&state.main_lock, NULL);
 	pthread_cond_init(&state.cookies_cond, NULL);
-	state.error_code = 0;
+	state.listen_error_code = 0;
 	state.current_token_data = fsmonitor_new_token_data();
 
 	/* Prepare to (recursively) watch the <worktree-root> directory. */
diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 14105f45c18..07113205a61 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -27,7 +27,7 @@
 #include "fsm-listen.h"
 #include "fsmonitor--daemon.h"
 
-struct fsmonitor_daemon_backend_data
+struct fsm_listen_data
 {
 	CFStringRef cfsr_worktree_path;
 	CFStringRef cfsr_gitdir_path;
@@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			     const FSEventStreamEventId event_ids[])
 {
 	struct fsmonitor_daemon_state *state = ctx;
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	char **paths = (char **)event_paths;
 	struct fsmonitor_batch *batch = NULL;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 		NULL,
 		NULL
 	};
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 	const void *dir_array[2];
 
 	CALLOC_ARRAY(data, 1);
-	state->backend_data = data;
+	state->listen_data = data;
 
 	data->cfsr_worktree_path = CFStringCreateWithCString(
 		NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8);
@@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 failed:
 	error(_("Unable to create FSEventStream."));
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 	return -1;
 }
 
 void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	if (!state || !state->backend_data)
+	if (!state || !state->listen_data)
 		return;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	if (data->stream) {
 		if (data->stream_started)
@@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 		FSEventStreamRelease(data->stream);
 	}
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 }
 
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	data = state->backend_data;
+	data = state->listen_data;
 	data->shutdown_style = SHUTDOWN_EVENT;
 
 	CFRunLoopStop(data->rl);
@@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 
 void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	data->rl = CFRunLoopGetCurrent();
 
@@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 
 	switch (data->shutdown_style) {
 	case FORCE_ERROR_STOP:
-		state->error_code = -1;
+		state->listen_error_code = -1;
 		/* fall thru */
 	case FORCE_SHUTDOWN:
 		ipc_server_stop_async(state->ipc_server_data);
@@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 	return;
 
 force_error_stop_without_loop:
-	state->error_code = -1;
+	state->listen_error_code = -1;
 	ipc_server_stop_async(state->ipc_server_data);
 	return;
 }
diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c
index c43d92b9620..be2d93f47b2 100644
--- a/compat/fsmonitor/fsm-listen-win32.c
+++ b/compat/fsmonitor/fsm-listen-win32.c
@@ -54,7 +54,7 @@ struct one_watch
 	wchar_t dotgit_shortname[16]; /* for 8.3 name */
 };
 
-struct fsmonitor_daemon_backend_data
+struct fsm_listen_data
 {
 	struct one_watch *watch_worktree;
 	struct one_watch *watch_gitdir;
@@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname(
 
 void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
 {
-	SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]);
+	SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]);
 }
 
 static struct one_watch *create_watch(struct fsmonitor_daemon_state *state,
@@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch)
 	free(watch);
 }
 
-static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data,
+static int start_rdcw_watch(struct fsm_listen_data *data,
 			    struct one_watch *watch)
 {
 	DWORD dwNotifyFilter =
@@ -516,7 +516,7 @@ static int process_1_worktree_event(
  */
 static int process_worktree_events(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	struct one_watch *watch = data->watch_worktree;
 	struct strbuf path = STRBUF_INIT;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -646,7 +646,7 @@ force_shutdown:
  */
 static int process_gitdir_events(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	struct one_watch *watch = data->watch_gitdir;
 	struct strbuf path = STRBUF_INIT;
 	struct string_list cookie_list = STRING_LIST_INIT_DUP;
@@ -704,11 +704,11 @@ skip_this_path:
 
 void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data = state->backend_data;
+	struct fsm_listen_data *data = state->listen_data;
 	DWORD dwWait;
 	int result;
 
-	state->error_code = 0;
+	state->listen_error_code = 0;
 
 	if (start_rdcw_watch(data, data->watch_worktree) == -1)
 		goto force_error_stop;
@@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state)
 	}
 
 force_error_stop:
-	state->error_code = -1;
+	state->listen_error_code = -1;
 
 force_shutdown:
 	/*
@@ -790,7 +790,7 @@ clean_shutdown:
 
 int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
 	CALLOC_ARRAY(data, 1);
 
@@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
 		data->nr_listener_handles++;
 	}
 
-	state->backend_data = data;
+	state->listen_data = data;
 	return 0;
 
 failed:
@@ -836,16 +836,16 @@ failed:
 
 void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
 {
-	struct fsmonitor_daemon_backend_data *data;
+	struct fsm_listen_data *data;
 
-	if (!state || !state->backend_data)
+	if (!state || !state->listen_data)
 		return;
 
-	data = state->backend_data;
+	data = state->listen_data;
 
 	CloseHandle(data->hEventShutdown);
 	destroy_watch(data->watch_worktree);
 	destroy_watch(data->watch_gitdir);
 
-	FREE_AND_NULL(state->backend_data);
+	FREE_AND_NULL(state->listen_data);
 }
diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h
index f0539349baf..41650bf8972 100644
--- a/compat/fsmonitor/fsm-listen.h
+++ b/compat/fsmonitor/fsm-listen.h
@@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state);
  * do so if the listener thread receives a normal shutdown signal from
  * the IPC layer.)
  *
- * It should set `state->error_code` to -1 if the daemon should exit
+ * It should set `state->listen_error_code` to -1 if the daemon should exit
  * with an error.
  */
 void fsm_listen__loop(struct fsmonitor_daemon_state *state);
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index 223c2131b58..2c6fa1a5d91 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);
  */
 void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);
 
-struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */
+struct fsm_listen_data; /* opaque platform-specific data for listener thread */
 
 struct fsmonitor_daemon_state {
 	pthread_t listener_thread;
@@ -50,8 +50,8 @@ struct fsmonitor_daemon_state {
 	int cookie_seq;
 	struct hashmap cookies;
 
-	int error_code;
-	struct fsmonitor_daemon_backend_data *backend_data;
+	int listen_error_code;
+	struct fsm_listen_data *listen_data;
 
 	struct ipc_server_data *ipc_server_data;
 	struct strbuf path_ipc;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 16/27] fsmonitor--daemon: stub in health thread
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (14 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 15/27] fsmonitor--daemon: rename listener thread related variables Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 17/27] fsm-health-win32: add polling framework to monitor daemon health Jeff Hostetler via GitGitGadget
                       ` (12 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create another thread to watch over the daemon process and
automatically shut it down if necessary.

This commit creates the basic framework for a "health" thread
to monitor the daemon and/or the file system.  Later commits
will add platform-specific code to do the actual work.

The "health" thread is intended to monitor conditions that
would be difficult to track inside the IPC thread pool and/or
the file system listener threads.  For example, when there are
file system events outside of the watched worktree root or if
we want to have an idle-timeout auto-shutdown feature.

This commit creates the health thread itself, defines the thread-proc
and sets up the thread's event loop.  It integrates this new thread
into the existing IPC and Listener thread models.

This commit defines the API to the platform-specific code where all of
the monitoring will actually happen.

The platform-specific code for MacOS is just stubs.  Meaning that the
health thread will immediately exit on MacOS, but that is OK and
expected.  Future work can define MacOS-specific monitoring.

The platform-specific code for Windows sets up enough of the
WaitForMultipleObjects() machinery to watch for system and/or custom
events.  Currently, the set of wait handles only includes our custom
shutdown event (sent from our other theads).  Later commits in this
series will extend the set of wait handles to monitor other
conditions.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 Makefile                             |  6 ++-
 builtin/fsmonitor--daemon.c          | 39 +++++++++++++++
 compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++
 compat/fsmonitor/fsm-health-win32.c  | 72 ++++++++++++++++++++++++++++
 compat/fsmonitor/fsm-health.h        | 47 ++++++++++++++++++
 contrib/buildsystems/CMakeLists.txt  |  2 +
 fsmonitor--daemon.h                  |  4 ++
 7 files changed, 192 insertions(+), 2 deletions(-)
 create mode 100644 compat/fsmonitor/fsm-health-darwin.c
 create mode 100644 compat/fsmonitor/fsm-health-win32.c
 create mode 100644 compat/fsmonitor/fsm-health.h

diff --git a/Makefile b/Makefile
index 93604fe8ef7..5f1623baadd 100644
--- a/Makefile
+++ b/Makefile
@@ -472,8 +472,9 @@ all::
 #
 # If your platform supports a built-in fsmonitor backend, set
 # FSMONITOR_DAEMON_BACKEND to the "<name>" of the corresponding
-# `compat/fsmonitor/fsm-listen-<name>.c` that implements the
-# `fsm_listen__*()` routines.
+# `compat/fsmonitor/fsm-listen-<name>.c` and
+# `compat/fsmonitor/fsm-health-<name>.c` files
+# that implement the `fsm_listen__*()` and `fsm_health__*()` routines.
 #
 # If your platform has OS-specific ways to tell if a repo is incompatible with
 # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS
@@ -1982,6 +1983,7 @@ endif
 ifdef FSMONITOR_DAEMON_BACKEND
 	COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
 	COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
+	COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
 endif
 
 ifdef FSMONITOR_OS_SETTINGS
diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index af91bc2fb4c..522904abf36 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -3,6 +3,7 @@
 #include "parse-options.h"
 #include "fsmonitor.h"
 #include "fsmonitor-ipc.h"
+#include "compat/fsmonitor/fsm-health.h"
 #include "compat/fsmonitor/fsm-listen.h"
 #include "fsmonitor--daemon.h"
 #include "simple-ipc.h"
@@ -1137,6 +1138,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state,
 	pthread_mutex_unlock(&state->main_lock);
 }
 
+static void *fsm_health__thread_proc(void *_state)
+{
+	struct fsmonitor_daemon_state *state = _state;
+
+	trace2_thread_start("fsm-health");
+
+	fsm_health__loop(state);
+
+	trace2_thread_exit();
+	return NULL;
+}
+
 static void *fsm_listen__thread_proc(void *_state)
 {
 	struct fsmonitor_daemon_state *state = _state;
@@ -1175,6 +1188,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 		 */
 		.uds_disallow_chdir = 0
 	};
+	int health_started = 0;
 	int listener_started = 0;
 	int err = 0;
 
@@ -1202,6 +1216,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state)
 	}
 	listener_started = 1;
 
+	/*
+	 * Start the health thread to watch over our process.
+	 */
+	if (pthread_create(&state->health_thread, NULL,
+			   fsm_health__thread_proc, state) < 0) {
+		ipc_server_stop_async(state->ipc_server_data);
+		err = error(_("could not start fsmonitor health thread"));
+		goto cleanup;
+	}
+	health_started = 1;
+
 	/*
 	 * The daemon is now fully functional in background threads.
 	 * Our primary thread should now just wait while the threads
@@ -1224,10 +1249,17 @@ cleanup:
 		pthread_join(state->listener_thread, NULL);
 	}
 
+	if (health_started) {
+		fsm_health__stop_async(state);
+		pthread_join(state->health_thread, NULL);
+	}
+
 	if (err)
 		return err;
 	if (state->listen_error_code)
 		return state->listen_error_code;
+	if (state->health_error_code)
+		return state->health_error_code;
 	return 0;
 }
 
@@ -1243,6 +1275,7 @@ static int fsmonitor_run_daemon(void)
 	pthread_mutex_init(&state.main_lock, NULL);
 	pthread_cond_init(&state.cookies_cond, NULL);
 	state.listen_error_code = 0;
+	state.health_error_code = 0;
 	state.current_token_data = fsmonitor_new_token_data();
 
 	/* Prepare to (recursively) watch the <worktree-root> directory. */
@@ -1322,6 +1355,11 @@ static int fsmonitor_run_daemon(void)
 		goto done;
 	}
 
+	if (fsm_health__ctor(&state)) {
+		err = error(_("could not initialize health thread"));
+		goto done;
+	}
+
 	/*
 	 * CD out of the worktree root directory.
 	 *
@@ -1345,6 +1383,7 @@ done:
 	pthread_cond_destroy(&state.cookies_cond);
 	pthread_mutex_destroy(&state.main_lock);
 	fsm_listen__dtor(&state);
+	fsm_health__dtor(&state);
 
 	ipc_server_free(state.ipc_server_data);
 
diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c
new file mode 100644
index 00000000000..b9f709e8548
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-darwin.c
@@ -0,0 +1,24 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+	return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+	return;
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+	return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+}
diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
new file mode 100644
index 00000000000..94b1d020f25
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -0,0 +1,72 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+struct fsm_health_data
+{
+	HANDLE hEventShutdown;
+
+	HANDLE hHandles[1]; /* the array does not own these handles */
+#define HEALTH_SHUTDOWN 0
+	int nr_handles; /* number of active event handles */
+};
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data;
+
+	CALLOC_ARRAY(data, 1);
+
+	data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL);
+
+	data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown;
+	data->nr_handles++;
+
+	state->health_data = data;
+	return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data;
+
+	if (!state || !state->health_data)
+		return;
+
+	data = state->health_data;
+
+	CloseHandle(data->hEventShutdown);
+
+	FREE_AND_NULL(state->health_data);
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+	struct fsm_health_data *data = state->health_data;
+
+	for (;;) {
+		DWORD dwWait = WaitForMultipleObjects(data->nr_handles,
+						      data->hHandles,
+						      FALSE, INFINITE);
+
+		if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)
+			goto clean_shutdown;
+
+		error(_("health thread wait failed [GLE %ld]"),
+		      GetLastError());
+		goto force_error_stop;
+	}
+
+force_error_stop:
+	state->health_error_code = -1;
+	ipc_server_stop_async(state->ipc_server_data);
+clean_shutdown:
+	return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+	SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]);
+}
diff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h
new file mode 100644
index 00000000000..45547ba9380
--- /dev/null
+++ b/compat/fsmonitor/fsm-health.h
@@ -0,0 +1,47 @@
+#ifndef FSM_HEALTH_H
+#define FSM_HEALTH_H
+
+/* This needs to be implemented by each backend */
+
+#ifdef HAVE_FSMONITOR_DAEMON_BACKEND
+
+struct fsmonitor_daemon_state;
+
+/*
+ * Initialize platform-specific data for the fsmonitor health thread.
+ * This will be called from the main thread PRIOR to staring the
+ * thread.
+ *
+ * Returns 0 if successful.
+ * Returns -1 otherwise.
+ */
+int fsm_health__ctor(struct fsmonitor_daemon_state *state);
+
+/*
+ * Cleanup platform-specific data for the health thread.
+ * This will be called from the main thread AFTER joining the thread.
+ */
+void fsm_health__dtor(struct fsmonitor_daemon_state *state);
+
+/*
+ * The main body of the platform-specific event loop to monitor the
+ * health of the daemon process.  This will run in the health thread.
+ *
+ * The health thread should call `ipc_server_stop_async()` if it needs
+ * to cause a shutdown.  (It should NOT do so if it receives a shutdown
+ * shutdown signal.)
+ *
+ * It should set `state->health_error_code` to -1 if the daemon should exit
+ * with an error.
+ */
+void fsm_health__loop(struct fsmonitor_daemon_state *state);
+
+/*
+ * Gently request that the health thread shutdown.
+ * It does not wait for it to stop.  The caller should do a JOIN
+ * to wait for it.
+ */
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state);
+
+#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */
+#endif /* FSM_HEALTH_H */
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index b8f9f7a0388..16ace43d1c7 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC)
 	if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
 
 		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
 	elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
 		add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
+		list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
 
 		add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
 		list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index 2c6fa1a5d91..2102a5c9ff5 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch);
 void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path);
 
 struct fsm_listen_data; /* opaque platform-specific data for listener thread */
+struct fsm_health_data; /* opaque platform-specific data for health thread */
 
 struct fsmonitor_daemon_state {
 	pthread_t listener_thread;
+	pthread_t health_thread;
 	pthread_mutex_t main_lock;
 
 	struct strbuf path_worktree_watch;
@@ -51,7 +53,9 @@ struct fsmonitor_daemon_state {
 	struct hashmap cookies;
 
 	int listen_error_code;
+	int health_error_code;
 	struct fsm_listen_data *listen_data;
+	struct fsm_health_data *health_data;
 
 	struct ipc_server_data *ipc_server_data;
 	struct strbuf path_ipc;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 17/27] fsm-health-win32: add polling framework to monitor daemon health
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (15 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 16/27] fsmonitor--daemon: stub in health thread Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 18/27] fsm-health-win32: force shutdown daemon if worktree root moves Jeff Hostetler via GitGitGadget
                       ` (11 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Extend the Windows version of the "health" thread to periodically
inspect the system and shutdown if warranted.

This commit updates the thread's wait loop to use a timeout and
defines a (currently empty) table of functions to poll the system.

A later commit will add functions to the table to actually
inspect the system.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++-
 1 file changed, 64 insertions(+), 1 deletion(-)

diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
index 94b1d020f25..24fc612bf02 100644
--- a/compat/fsmonitor/fsm-health-win32.c
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -4,6 +4,24 @@
 #include "fsm-health.h"
 #include "fsmonitor--daemon.h"
 
+/*
+ * Every minute wake up and test our health.
+ */
+#define WAIT_FREQ_MS (60 * 1000)
+
+/*
+ * State machine states for each of the interval functions
+ * used for polling our health.
+ */
+enum interval_fn_ctx {
+	CTX_INIT = 0,
+	CTX_TERM,
+	CTX_TIMER
+};
+
+typedef int (interval_fn)(struct fsmonitor_daemon_state *state,
+			  enum interval_fn_ctx ctx);
+
 struct fsm_health_data
 {
 	HANDLE hEventShutdown;
@@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)
 	FREE_AND_NULL(state->health_data);
 }
 
+/*
+ * A table of the polling functions.
+ */
+static interval_fn *table[] = {
+	NULL, /* must be last */
+};
+
+/*
+ * Call all of the polling functions in the table.
+ * Shortcut and return first error.
+ *
+ * Return 0 if all succeeded.
+ */
+static int call_all(struct fsmonitor_daemon_state *state,
+		    enum interval_fn_ctx ctx)
+{
+	int k;
+
+	for (k = 0; table[k]; k++) {
+		int r = table[k](state, ctx);
+		if (r)
+			return r;
+	}
+
+	return 0;
+}
+
 void fsm_health__loop(struct fsmonitor_daemon_state *state)
 {
 	struct fsm_health_data *data = state->health_data;
+	int r;
+
+	r = call_all(state, CTX_INIT);
+	if (r < 0)
+		goto force_error_stop;
+	if (r > 0)
+		goto force_shutdown;
 
 	for (;;) {
 		DWORD dwWait = WaitForMultipleObjects(data->nr_handles,
 						      data->hHandles,
-						      FALSE, INFINITE);
+						      FALSE, WAIT_FREQ_MS);
 
 		if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN)
 			goto clean_shutdown;
 
+		if (dwWait == WAIT_TIMEOUT) {
+			r = call_all(state, CTX_TIMER);
+			if (r < 0)
+				goto force_error_stop;
+			if (r > 0)
+				goto force_shutdown;
+			continue;
+		}
+
 		error(_("health thread wait failed [GLE %ld]"),
 		      GetLastError());
 		goto force_error_stop;
@@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state)
 
 force_error_stop:
 	state->health_error_code = -1;
+force_shutdown:
 	ipc_server_stop_async(state->ipc_server_data);
 clean_shutdown:
+	call_all(state, CTX_TERM);
 	return;
 }
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 18/27] fsm-health-win32: force shutdown daemon if worktree root moves
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (16 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 17/27] fsm-health-win32: add polling framework to monitor daemon health Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 19/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Jeff Hostetler via GitGitGadget
                       ` (10 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Force shutdown fsmonitor daemon if the worktree root directory
is moved, renamed, or deleted.

Use Windows low-level GetFileInformationByHandle() to get and
compare the Windows system unique ID for the directory with a
cached version when we started up.  This lets us detect the
case where someone renames the directory that we are watching
and then creates a new directory with the original pathname.

This is important because we are listening to a named pipe for
requests and they are stored in the Named Pipe File System (NPFS)
which a kernel-resident pseudo filesystem not associated with
the actual NTFS directory.

For example, if the daemon was watching "~/foo/", it would have
a directory-watch handle on that directory and a named-pipe
handle for "//./pipe/...foo".  Moving the directory to "~/bar/"
does not invalidate the directory handle.  (So the daemon would
actually be watching "~/bar" but listening on "//./pipe/...foo".
If the user then does "git init ~/foo" and causes another daemon
to start, the first daemon will still have ownership of the pipe
and the second daemon instance will fail to start.  "git status"
clients in "~/foo" will ask "//./pipe/...foo" about changes and
the first daemon instance will tell them about "~/bar".

This commit causes the first daemon to shutdown if the system unique
ID for "~/foo" changes (changes from what it was when the daemon
started).  Shutdown occurs after a periodic poll.  After the
first daemon exits and releases the lock on the named pipe,
subsequent Git commands may cause another daemon to be started
on "~/foo".  Similarly, a subsequent Git command may cause another
daemon to be started on "~/bar".

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++
 1 file changed, 143 insertions(+)

diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c
index 24fc612bf02..2ea08c1d4e8 100644
--- a/compat/fsmonitor/fsm-health-win32.c
+++ b/compat/fsmonitor/fsm-health-win32.c
@@ -29,8 +29,150 @@ struct fsm_health_data
 	HANDLE hHandles[1]; /* the array does not own these handles */
 #define HEALTH_SHUTDOWN 0
 	int nr_handles; /* number of active event handles */
+
+	struct wt_moved
+	{
+		wchar_t wpath[MAX_PATH + 1];
+		BY_HANDLE_FILE_INFORMATION bhfi;
+	} wt_moved;
 };
 
+/*
+ * Lookup the system unique ID for the path.  This is as close as
+ * we get to an inode number, but this also contains volume info,
+ * so it is a little stronger.
+ */
+static int lookup_bhfi(wchar_t *wpath,
+		       BY_HANDLE_FILE_INFORMATION *bhfi)
+{
+	DWORD desired_access = FILE_LIST_DIRECTORY;
+	DWORD share_mode =
+		FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE;
+	HANDLE hDir;
+
+	hDir = CreateFileW(wpath, desired_access, share_mode, NULL,
+			   OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);
+	if (hDir == INVALID_HANDLE_VALUE) {
+		error(_("[GLE %ld] health thread could not open '%ls'"),
+		      GetLastError(), wpath);
+		return -1;
+	}
+
+	if (!GetFileInformationByHandle(hDir, bhfi)) {
+		error(_("[GLE %ld] health thread getting BHFI for '%ls'"),
+		      GetLastError(), wpath);
+		CloseHandle(hDir);
+		return -1;
+	}
+
+	CloseHandle(hDir);
+	return 0;
+}
+
+/*
+ * Compare the relevant fields from two system unique IDs.
+ * We use this to see if two different handles to the same
+ * path actually refer to the same *instance* of the file
+ * or directory.
+ */
+static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1,
+		   const BY_HANDLE_FILE_INFORMATION *bhfi_2)
+{
+	return (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber &&
+		bhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh &&
+		bhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow);
+}
+
+/*
+ * Shutdown if the original worktree root directory been deleted,
+ * moved, or renamed?
+ *
+ * Since the main thread did a "chdir(getenv($HOME))" and our CWD
+ * is not in the worktree root directory and because the listener
+ * thread added FILE_SHARE_DELETE to the watch handle, it is possible
+ * for the root directory to be moved or deleted while we are still
+ * watching it.  We want to detect that here and force a shutdown.
+ *
+ * Granted, a delete MAY cause some operations to fail, such as
+ * GetOverlappedResult(), but it is not guaranteed.  And because
+ * ReadDirectoryChangesW() only reports on changes *WITHIN* the
+ * directory, not changes *ON* the directory, our watch will not
+ * receive a delete event for it.
+ *
+ * A move/rename of the worktree root will also not generate an event.
+ * And since the listener thread already has an open handle, it may
+ * continue to receive events for events within the directory.
+ * However, the pathname of the named-pipe was constructed using the
+ * original location of the worktree root.  (Remember named-pipes are
+ * stored in the NPFS and not in the actual file system.)  Clients
+ * trying to talk to the worktree after the move/rename will not
+ * reach our daemon process, since we're still listening on the
+ * pipe with original path.
+ *
+ * Furthermore, if the user does something like:
+ *
+ *   $ mv repo repo.old
+ *   $ git init repo
+ *
+ * A new daemon cannot be started in the new instance of "repo"
+ * because the named-pipe is still being used by the daemon on
+ * the original instance.
+ *
+ * So, detect move/rename/delete and shutdown.  This should also
+ * handle unsafe drive removal.
+ *
+ * We use the file system unique ID to distinguish the original
+ * directory instance from a new instance and force a shutdown
+ * if the unique ID changes.
+ *
+ * Since a worktree move/rename/delete/unmount doesn't happen
+ * that often (and we can't get an immediate event anyway), we
+ * use a timeout and periodically poll it.
+ */
+static int has_worktree_moved(struct fsmonitor_daemon_state *state,
+			      enum interval_fn_ctx ctx)
+{
+	struct fsm_health_data *data = state->health_data;
+	BY_HANDLE_FILE_INFORMATION bhfi;
+	int r;
+
+	switch (ctx) {
+	case CTX_TERM:
+		return 0;
+
+	case CTX_INIT:
+		if (xutftowcs_path(data->wt_moved.wpath,
+				   state->path_worktree_watch.buf) < 0) {
+			error(_("could not convert to wide characters: '%s'"),
+			      state->path_worktree_watch.buf);
+			return -1;
+		}
+
+		/*
+		 * On the first call we lookup the unique sequence ID for
+		 * the worktree root directory.
+		 */
+		return lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi);
+
+	case CTX_TIMER:
+		r = lookup_bhfi(data->wt_moved.wpath, &bhfi);
+		if (r)
+			return r;
+		if (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) {
+			error(_("BHFI changed '%ls'"), data->wt_moved.wpath);
+			return -1;
+		}
+		return 0;
+
+	default:
+		die(_("unhandled case in 'has_worktree_moved': %d"),
+		    (int)ctx);
+	}
+
+	return 0;
+}
+
+
 int fsm_health__ctor(struct fsmonitor_daemon_state *state)
 {
 	struct fsm_health_data *data;
@@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state)
  * A table of the polling functions.
  */
 static interval_fn *table[] = {
+	has_worktree_moved,
 	NULL, /* must be last */
 };
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 19/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (17 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 18/27] fsm-health-win32: force shutdown daemon if worktree root moves Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 20/27] fsmonitor: optimize processing of directory events Jeff Hostetler via GitGitGadget
                       ` (9 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach the listener thread to shutdown the daemon if the spelling of the
worktree root directory changes.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 07113205a61..83d38e8ac6c 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag)
 	strbuf_release(&msg);
 }
 
+static int ef_is_root_changed(const FSEventStreamEventFlags ef)
+{
+	return (ef & kFSEventStreamEventFlagRootChanged);
+}
+
 static int ef_is_root_delete(const FSEventStreamEventFlags ef)
 {
 	return (ef & kFSEventStreamEventFlagItemIsDir &&
@@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			continue;
 		}
 
+		if (ef_is_root_changed(event_flags[k])) {
+			/*
+			 * The spelling of the pathname of the root directory
+			 * has changed.  This includes the name of the root
+			 * directory itself or of any parent directory in the
+			 * path.
+			 *
+			 * (There may be other conditions that throw this,
+			 * but I couldn't find any information on it.)
+			 *
+			 * Force a shutdown now and avoid things getting
+			 * out of sync.  The Unix domain socket is inside
+			 * the .git directory and a spelling change will make
+			 * it hard for clients to rendezvous with us.
+			 */
+			trace_printf_key(&trace_fsmonitor,
+					 "event: root changed");
+			goto force_shutdown;
+		}
+
 		if (ef_ignore_xattr(event_flags[k])) {
 			trace_printf_key(&trace_fsmonitor,
 					 "ignore-xattr: '%s', flags=0x%x",
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 20/27] fsmonitor: optimize processing of directory events
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (18 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 19/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 21/27] t7527: FSMonitor tests for directory moves Jeff Hostetler via GitGitGadget
                       ` (8 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Teach Git to perform binary search over the cache-entries for a directory
notification and then linearly scan forward to find the immediate children.

Previously, when the FSMonitor reported a modified directory Git would
perform a linear search on the entire cache-entry array for all
entries matching that directory prefix and invalidate them.  Since the
cache-entry array is already sorted, we can use a binary search to
find the first matching entry and then only linearly walk forward and
invalidate entries until the prefix changes.

Also, the original code would invalidate anything having the same
directory prefix.  Since a directory event should only be received for
items that are immediately within the directory (and not within
sub-directories of it), only invalidate those entries and not the
whole subtree.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 54 insertions(+), 17 deletions(-)

diff --git a/fsmonitor.c b/fsmonitor.c
index 292a6742b4f..e1229c289cf 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r,
 static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
 {
 	int i, len = strlen(name);
-	if (name[len - 1] == '/') {
+	int pos = index_name_pos(istate, name, len);
+
+	trace_printf_key(&trace_fsmonitor,
+			 "fsmonitor_refresh_callback '%s' (pos %d)",
+			 name, pos);
 
+	if (name[len - 1] == '/') {
 		/*
-		 * TODO We should binary search to find the first path with
-		 * TODO this directory prefix.  Then linearly update entries
-		 * TODO while the prefix matches.  Taking care to search without
-		 * TODO the trailing slash -- because '/' sorts after a few
-		 * TODO interesting special chars, like '.' and ' '.
+		 * The daemon can decorate directory events, such as
+		 * moves or renames, with a trailing slash if the OS
+		 * FS Event contains sufficient information, such as
+		 * MacOS.
+		 *
+		 * Use this to invalidate the entire cone under that
+		 * directory.
+		 *
+		 * We do not expect an exact match because the index
+		 * does not normally contain directory entries, so we
+		 * start at the insertion point and scan.
 		 */
+		if (pos < 0)
+			pos = -pos - 1;
 
 		/* Mark all entries for the folder invalid */
-		for (i = 0; i < istate->cache_nr; i++) {
-			if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID &&
-			    starts_with(istate->cache[i]->name, name))
-				istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
+		for (i = pos; i < istate->cache_nr; i++) {
+			if (!starts_with(istate->cache[i]->name, name))
+				break;
+			istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
 		}
-		/* Need to remove the / from the path for the untracked cache */
+
+		/*
+		 * We need to remove the traling "/" from the path
+		 * for the untracked cache.
+		 */
 		name[len - 1] = '\0';
+	} else if (pos >= 0) {
+		/*
+		 * We have an exact match for this path and can just
+		 * invalidate it.
+		 */
+		istate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;
 	} else {
-		int pos = index_name_pos(istate, name, strlen(name));
-
-		if (pos >= 0) {
-			struct cache_entry *ce = istate->cache[pos];
-			ce->ce_flags &= ~CE_FSMONITOR_VALID;
+		/*
+		 * The path is not a tracked file -or- it is a
+		 * directory event on a platform that cannot
+		 * distinguish between file and directory events in
+		 * the event handler, such as Windows.
+		 *
+		 * Scan as if it is a directory and invalidate the
+		 * cone under it.  (But remember to ignore items
+		 * between "name" and "name/", such as "name-" and
+		 * "name.".
+		 */
+		pos = -pos - 1;
+
+		for (i = pos; i < istate->cache_nr; i++) {
+			if (!starts_with(istate->cache[i]->name, name))
+				break;
+			if ((unsigned char)istate->cache[i]->name[len] > '/')
+				break;
+			if (istate->cache[i]->name[len] == '/')
+				istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;
 		}
 	}
 
@@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
 	 * Mark the untracked cache dirty even if it wasn't found in the index
 	 * as it could be a new untracked file.
 	 */
-	trace_printf_key(&trace_fsmonitor, "fsmonitor_refresh_callback '%s'", name);
 	untracked_cache_invalidate_path(istate, name, 0);
 }
 
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 21/27] t7527: FSMonitor tests for directory moves
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (19 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 20/27] fsmonitor: optimize processing of directory events Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 22/27] t/perf/p7527: add perf test for builtin FSMonitor Jeff Hostetler via GitGitGadget
                       ` (7 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create unit tests to move a directory.  Verify that `git status`
gives the same result with and without FSMonitor enabled.

NEEDSWORK: This test exposes a bug in the untracked-cache on
Windows when FSMonitor is disabled.  These are commented out
for the moment.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 39 ++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 429029fcadd..bc66d8285a3 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -281,6 +281,16 @@ test_expect_success 'setup' '
 	trace*
 	EOF
 
+	mkdir -p T1/T2/T3/T4 &&
+	echo 1 >T1/F1 &&
+	echo 1 >T1/T2/F1 &&
+	echo 1 >T1/T2/T3/F1 &&
+	echo 1 >T1/T2/T3/T4/F1 &&
+	echo 2 >T1/F2 &&
+	echo 2 >T1/T2/F2 &&
+	echo 2 >T1/T2/T3/F2 &&
+	echo 2 >T1/T2/T3/T4/F2 &&
+
 	git -c core.fsmonitor=false add . &&
 	test_tick &&
 	git -c core.fsmonitor=false commit -m initial &&
@@ -363,6 +373,19 @@ directory_to_file () {
 	echo 1 >dir1
 }
 
+move_directory_contents_deeper() {
+	mkdir T1/_new_ &&
+	mv T1/[A-Z]* T1/_new_
+}
+
+move_directory_up() {
+	mv T1/T2/T3 T1
+}
+
+move_directory() {
+	mv T1/T2/T3 T1/T2/NewT3
+}
+
 # The next few test cases confirm that our fsmonitor daemon sees each type
 # of OS filesystem notification that we care about.  At this layer we just
 # ensure we are getting the OS notifications and do not try to confirm what
@@ -645,6 +668,22 @@ do
 		matrix_try $uc_val $fsm_val file_to_directory
 		matrix_try $uc_val $fsm_val directory_to_file
 
+		# NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor
+		# is DISABLED.  Turn off a few test that cause it problems until
+		# we can debug it.
+		#
+		try_moves="true"
+		test_have_prereq UNTRACKED_CACHE,WINDOWS && \
+			test $uc_val = true && \
+			test $fsm_val = false && \
+			try_moves="false"
+		if test $try_moves = true
+		then
+			matrix_try $uc_val $fsm_val move_directory_contents_deeper
+			matrix_try $uc_val $fsm_val move_directory_up
+			matrix_try $uc_val $fsm_val move_directory
+		fi
+
 		if test $fsm_val = true
 		then
 			test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end" '
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 22/27] t/perf/p7527: add perf test for builtin FSMonitor
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (20 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 21/27] t7527: FSMonitor tests for directory moves Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 23/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules Jeff Hostetler via GitGitGadget
                       ` (6 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++
 1 file changed, 257 insertions(+)
 create mode 100755 t/perf/p7527-builtin-fsmonitor.sh

diff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh
new file mode 100755
index 00000000000..9338b9ea008
--- /dev/null
+++ b/t/perf/p7527-builtin-fsmonitor.sh
@@ -0,0 +1,257 @@
+#!/bin/sh
+
+test_description="Perf test for the builtin FSMonitor"
+
+. ./perf-lib.sh
+
+if ! test_have_prereq FSMONITOR_DAEMON
+then
+	skip_all="fsmonitor--daemon is not supported on this platform"
+	test_done
+fi
+
+test_lazy_prereq UNTRACKED_CACHE '
+	{ git update-index --test-untracked-cache; ret=$?; } &&
+	test $ret -ne 1
+'
+
+# Lie to perf-lib and ask for a new empty repo and avoid
+# the complaints about GIT_PERF_REPO not being big enough
+# the perf hit when GIT_PERF_LARGE_REPO is copied into
+# the trash directory.
+#
+# NEEDSWORK: It would be nice if perf-lib had an option to
+# "borrow" an existing large repo (especially for gigantic
+# monorepos) and use it in-place.  For now, fake it here.
+#
+test_perf_fresh_repo
+
+
+# Use a generated synthetic monorepo.  If it doesn't exist, we will
+# generate it.  If it does exist, we will put it in a known state
+# before we start our timings.
+#
+PARAM_D=5
+PARAM_W=10
+PARAM_F=9
+
+PARAMS="$PARAM_D"."$PARAM_W"."$PARAM_F"
+
+BALLAST_BR=p0006-ballast
+export BALLAST_BR
+
+TMP_BR=tmp_br
+export TMP_BR
+
+REPO=../repos/gen-many-files-"$PARAMS".git
+export REPO
+
+if ! test -d $REPO
+then
+	(cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F)
+fi
+
+
+enable_uc () {
+	git -C $REPO config core.untrackedcache true
+	git -C $REPO update-index --untracked-cache
+	git -C $REPO status >/dev/null 2>&1
+}
+
+disable_uc () {
+	git -C $REPO config core.untrackedcache false
+	git -C $REPO update-index --no-untracked-cache
+	git -C $REPO status >/dev/null 2>&1
+}
+
+start_fsm () {
+	git -C $REPO fsmonitor--daemon start
+	git -C $REPO fsmonitor--daemon status
+	git -C $REPO config core.fsmonitor true
+	git -C $REPO update-index --fsmonitor
+	git -C $REPO status >/dev/null 2>&1
+}
+
+stop_fsm () {
+	git -C $REPO config --unset core.fsmonitor
+	git -C $REPO update-index --no-fsmonitor
+	test_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null
+	git -C $REPO status >/dev/null 2>&1
+}
+
+
+# Ensure that FSMonitor is turned off on the borrowed repo.
+#
+test_expect_success "Setup borrowed repo (fsm+uc)" "
+	stop_fsm &&
+	disable_uc
+"
+
+# Also ensure that it starts in a known state.
+#
+# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time
+# the ballast checkout, since only the first invocation does any work and the
+# subsequent ones just print "already on branch" and quit, so the reported
+# time is not useful.
+#
+# Create a temp branch and do all work relative to it so that we don't
+# accidentially alter the real ballast branch.
+#
+test_expect_success "Setup borrowed repo (temp ballast branch)" "
+	test_might_fail git -C $REPO checkout $BALLAST_BR &&
+	test_might_fail git -C $REPO reset --hard &&
+	git -C $REPO clean -d -f &&
+	test_might_fail git -C $REPO branch -D $TMP_BR &&
+	git -C $REPO branch $TMP_BR $BALLAST_BR &&
+	git -C $REPO checkout $TMP_BR
+"
+
+
+echo Data >data.txt
+
+# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1.  With
+# FSMonitor enabled, we can get a skewed view of status times, since
+# the index MAY (or may not) be updated after the first invocation
+# which will update the FSMonitor Token, so the subsequent invocations
+# may get a smaller response from the daemon.
+#
+do_status () {
+	msg=$1
+
+	test_perf "$msg" "
+		git -C $REPO status >/dev/null 2>&1
+	"
+}
+
+do_matrix () {
+	uc=$1
+	fsm=$2
+
+	t="[uc $uc][fsm $fsm]"
+	MATRIX_BR="$TMP_BR-$uc-$fsm"
+
+	test_expect_success "$t Setup matrix branch" "
+		git -C $REPO clean -d -f &&
+		git -C $REPO checkout $TMP_BR &&
+		test_might_fail git -C $REPO branch -D $MATRIX_BR &&
+		git -C $REPO branch $MATRIX_BR $TMP_BR &&
+		git -C $REPO checkout $MATRIX_BR
+	"
+
+	if test $uc = true
+	then
+		enable_uc
+	else
+		disable_uc
+	fi
+
+	if test $fsm = true
+	then
+		start_fsm
+	else
+		stop_fsm
+	fi
+
+	do_status "$t status after checkout"
+
+	# Modify many files in the matrix branch.
+	# Stage them.
+	# Commit them.
+	# Rollback.
+	#
+	test_expect_success "$t modify tracked files" "
+		find $REPO -name file1 -exec cp data.txt {} \\;
+	"
+
+	do_status "$t status after big change"
+
+	# Don't bother timing the "add" because _REPEAT_COUNT
+	# issue described above.
+	#
+	test_expect_success "$t add all" "
+		git -C $REPO add -A
+	"
+
+	do_status "$t status after add all"
+
+	test_expect_success "$t add dot" "
+		git -C $REPO add .
+	"
+
+	do_status "$t status after add dot"
+
+	test_expect_success "$t commit staged" "
+		git -C $REPO commit -a -m data
+	"
+
+	do_status "$t status after commit"
+
+	test_expect_success "$t reset HEAD~1 hard" "
+		git -C $REPO reset --hard HEAD~1 >/dev/null 2>&1
+	"
+
+	do_status "$t status after reset hard"
+
+	# Create some untracked files.
+	#
+	test_expect_success "$t create untracked files" "
+		cp -R $REPO/ballast/dir1 $REPO/ballast/xxx1
+	"
+
+	do_status "$t status after create untracked files"
+
+	# Remove the new untracked files.
+	#
+	test_expect_success "$t clean -df" "
+		git -C $REPO clean -d -f
+	"
+
+	do_status "$t status after clean"
+
+	if test $fsm = true
+	then
+		stop_fsm
+	fi
+}
+
+# Begin testing each case in the matrix that we care about.
+#
+uc_values="false"
+test_have_prereq UNTRACKED_CACHE && uc_values="false true"
+
+fsm_values="false true"
+
+for uc_val in $uc_values
+do
+	for fsm_val in $fsm_values
+	do
+		do_matrix $uc_val $fsm_val
+	done
+done
+
+cleanup () {
+	uc=$1
+	fsm=$2
+
+	MATRIX_BR="$TMP_BR-$uc-$fsm"
+
+	test_might_fail git -C $REPO branch -D $MATRIX_BR
+}
+
+
+# We're borrowing this repo.  We should leave it in a clean state.
+#
+test_expect_success "Cleanup temp and matrix branches" "
+	git -C $REPO clean -d -f &&
+	test_might_fail git -C $REPO checkout $BALLAST_BR &&
+	test_might_fail git -C $REPO branch -D $TMP_BR &&
+	for uc_val in $uc_values
+	do
+		for fsm_val in $fsm_values
+		do
+			cleanup $uc_val $fsm_val
+		done
+	done
+"
+
+test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 23/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (21 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 22/27] t/perf/p7527: add perf test for builtin FSMonitor Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 24/27] t7527: test FSMonitor on case insensitive+preserving file system Jeff Hostetler via GitGitGadget
                       ` (5 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Never set CE_FSMONITOR_VALID on the cache-entry of submodule
directories.

During a client command like 'git status', we may need to recurse
into each submodule to compute a status summary for the submodule.
Since the purpose of the ce_flag is to let Git avoid scanning a
cache-entry, setting the flag causes the recursive call to be
avoided and we report incorrect (no status) for the submodule.

We created an OS watch on the root directory of our working
directory and we receive events for everything in the cone
under it.  When submodules are present inside our working
directory, we receive events for both our repo (the super) and
any subs within it.  Since our index doesn't have any information
for items within the submodules, we can't use those events.

We could try to truncate the paths of those events back to the
submodule boundary and mark the GITLINK as dirty, but that
feels expensive since we would have to prefix compare every FS
event that we receive against a list of submodule roots.  And
it still wouldn't be sufficient to correctly report status on
the submodule, since we don't have any space in the cache-entry
to cache the submodule's status (the 'SCMU' bits in porcelain
V2 speak).  That is, the CE_FSMONITOR_VALID bit just says that
we don't need to scan/inspect it because we already know the
answer -- it doesn't say that the item is clean -- and we
don't have space in the cache-entry to store those answers.
So we should always do the recursive scan.

Therefore, we should never set the flag on GITLINK cache-entries.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 fsmonitor.c                  |  2 +
 fsmonitor.h                  | 11 +++++
 t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++
 3 files changed, 106 insertions(+)

diff --git a/fsmonitor.c b/fsmonitor.c
index e1229c289cf..57d6a483bee 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate)
 		if (fsmonitor_enabled) {
 			/* Mark all entries valid */
 			for (i = 0; i < istate->cache_nr; i++) {
+				if (S_ISGITLINK(istate->cache[i]->ce_mode))
+					continue;
 				istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
 			}
 
diff --git a/fsmonitor.h b/fsmonitor.h
index 3f41f653691..edf7ce5203b 100644
--- a/fsmonitor.h
+++ b/fsmonitor.h
@@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate)
  * Set the given cache entries CE_FSMONITOR_VALID bit. This should be
  * called any time the cache entry has been updated to reflect the
  * current state of the file on disk.
+ *
+ * However, never mark submodules as valid.  When commands like "git
+ * status" run they might need to recurse into the submodule (using a
+ * child process) to get a summary of the submodule state.  We don't
+ * have (and don't want to create) the facility to translate every
+ * FS event that we receive and that happens to be deep inside of a
+ * submodule back to the submodule root, so we cannot correctly keep
+ * track of this bit on the gitlink directory.  Therefore, we never
+ * set it on submodules.
  */
 static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)
 {
@@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache
 
 	if (fsm_mode > FSMONITOR_MODE_DISABLED &&
 	    !(ce->ce_flags & CE_FSMONITOR_VALID)) {
+		if (S_ISGITLINK(ce->ce_mode))
+			return;
 		istate->cache_changed = 1;
 		ce->ce_flags |= CE_FSMONITOR_VALID;
 		trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name);
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index bc66d8285a3..39efed42a69 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -718,4 +718,97 @@ do
 	'
 done
 
+# Test fsmonitor interaction with submodules.
+#
+# If we start the daemon in the super, it will see FS events for
+# everything in the working directory cone and this includes any
+# files/directories contained *within* the submodules.
+#
+# A `git status` at top level will get events for items within the
+# submodule and ignore them, since they aren't named in the index
+# of the super repo.  This makes the fsmonitor response a little
+# noisy, but it doesn't alter the correctness of the state of the
+# super-proper.
+#
+# When we have submodules, `git status` normally does a recursive
+# status on each of the submodules and adds a summary row for any
+# dirty submodules.  (See the "S..." bits in porcelain V2 output.)
+#
+# It is therefore important that the top level status not be tricked
+# by the FSMonitor response to skip those recursive calls.
+
+my_match_and_clean () {
+	git -C super --no-optional-locks status --porcelain=v2 >actual.with &&
+	git -C super --no-optional-locks -c core.fsmonitor=false \
+		status --porcelain=v2 >actual.without &&
+	test_cmp actual.with actual.without &&
+
+	git -C super/dir_1/dir_2/sub reset --hard &&
+	git -C super/dir_1/dir_2/sub clean -d -f
+}
+
+test_expect_success "Submodule" '
+	test_when_finished "git -C super fsmonitor--daemon stop" &&
+
+	git init "super" &&
+	echo x >super/file_1 &&
+	echo y >super/file_2 &&
+	echo z >super/file_3 &&
+	mkdir super/dir_1 &&
+	echo a >super/dir_1/file_11 &&
+	echo b >super/dir_1/file_12 &&
+	mkdir super/dir_1/dir_2 &&
+	echo a >super/dir_1/dir_2/file_21 &&
+	echo b >super/dir_1/dir_2/file_22 &&
+	git -C super add . &&
+	git -C super commit -m "initial super commit" &&
+
+	git init "sub" &&
+	echo x >sub/file_x &&
+	echo y >sub/file_y &&
+	echo z >sub/file_z &&
+	mkdir sub/dir_x &&
+	echo a >sub/dir_x/file_a &&
+	echo b >sub/dir_x/file_b &&
+	mkdir sub/dir_x/dir_y &&
+	echo a >sub/dir_x/dir_y/file_a &&
+	echo b >sub/dir_x/dir_y/file_b &&
+	git -C sub add . &&
+	git -C sub commit -m "initial sub commit" &&
+
+	git -C super submodule add ../sub ./dir_1/dir_2/sub &&
+	git -C super commit -m "add sub" &&
+
+	start_daemon -C super &&
+	git -C super config core.fsmonitor true &&
+	git -C super update-index --fsmonitor &&
+	git -C super status &&
+
+	# Now run pairs of commands w/ and w/o FSMonitor while we make
+	# some dirt in the submodule and confirm matching output.
+
+	# Completely clean status.
+	my_match_and_clean &&
+
+	# .M S..U
+	echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u &&
+	my_match_and_clean &&
+
+	# .M S.M.
+	echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m &&
+	git -C super/dir_1/dir_2/sub add . &&
+	my_match_and_clean &&
+
+	# .M S.M.
+	echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&
+	git -C super/dir_1/dir_2/sub add . &&
+	my_match_and_clean &&
+
+	# .M SC..
+	echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a &&
+	git -C super/dir_1/dir_2/sub add . &&
+	git -C super/dir_1/dir_2/sub commit -m "SC.." &&
+	my_match_and_clean
+'
+
 test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 24/27] t7527: test FSMonitor on case insensitive+preserving file system
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (22 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 23/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 25/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname Jeff Hostetler via GitGitGadget
                       ` (4 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Test that FS events from the OS are received using the preserved,
on-disk spelling of files/directories rather than spelling used
to make the change.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 39efed42a69..51cdf05e7ec 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -811,4 +811,40 @@ test_expect_success "Submodule" '
 	my_match_and_clean
 '
 
+# On a case-insensitive file system, confirm that the daemon
+# notices when the .git directory is moved/renamed/deleted
+# regardless of how it is spelled in the the FS event.
+# That is, does the FS event receive the spelling of the
+# operation or does it receive the spelling preserved with
+# the file/directory.
+#
+test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' '
+#	test_when_finished "stop_daemon_delete_repo test_insensitive" &&
+
+	git init test_insensitive &&
+
+	start_daemon -C test_insensitive -tf "$PWD/insensitive.trace" &&
+
+	mkdir -p test_insensitive/abc/def &&
+	echo xyz >test_insensitive/ABC/DEF/xyz &&
+
+	test_path_is_dir test_insensitive/.git &&
+	test_path_is_dir test_insensitive/.GIT &&
+
+	# Rename .git using an alternate spelling to verify that that
+	# daemon detects it and automatically shuts down.
+	mv test_insensitive/.GIT test_insensitive/.FOO &&
+	sleep 1 &&
+	mv test_insensitive/.FOO test_insensitive/.git &&
+	test_must_fail git -C test_insensitive fsmonitor--daemon status &&
+
+	# Verify that events were reported using on-disk spellings of the
+	# directories and files that we touched.  We may or may not get a
+	# trailing slash on modified directories.
+	#
+	egrep "^event: abc/?$"       ./insensitive.trace &&
+	egrep "^event: abc/def/?$"   ./insensitive.trace &&
+	egrep "^event: abc/def/xyz$" ./insensitive.trace
+'
+
 test_done
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 25/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (23 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 24/27] t7527: test FSMonitor on case insensitive+preserving file system Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:22     ` [PATCH v3 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Jeff Hostetler via GitGitGadget
                       ` (3 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Emit NFC or NFC and NFD spellings of pathnames on macOS.

MacOS is Unicode composition insensitive, so NFC and NFD spellings are
treated as aliases and collide.  While the spelling of pathnames in
filesystem events depends upon the underlying filesystem, such as
APFS, HFS+ or FAT32, the OS enforces such collisions regardless of
filesystem.

Teach the daemon to always report the NFC spelling and to report
the NFD spelling when stored in that format on the disk.

This is slightly more general than "core.precomposeUnicode".

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++--
 1 file changed, 31 insertions(+), 2 deletions(-)

diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 83d38e8ac6c..823cf63999e 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef)
 	return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod);
 }
 
+/*
+ * On MacOS we have to adjust for Unicode composition insensitivity
+ * (where NFC and NFD spellings are not respected).  The different
+ * spellings are essentially aliases regardless of how the path is
+ * actually stored on the disk.
+ *
+ * This is related to "core.precomposeUnicode" (which wants to try
+ * to hide NFD completely and treat everything as NFC).  Here, we
+ * don't know what the value the client has (or will have) for this
+ * config setting when they make a query, so assume the worst and
+ * emit both when the OS gives us an NFD path.
+ */
+static void my_add_path(struct fsmonitor_batch *batch, const char *path)
+{
+	char *composed;
+
+	/* add the NFC or NFD path as received from the OS */
+	fsmonitor_batch__add_path(batch, path);
+
+	/* if NFD, also add the corresponding NFC spelling */
+	composed = (char *)precompose_string_if_needed(path);
+	if (!composed || composed == path)
+		return;
+
+	fsmonitor_batch__add_path(batch, composed);
+	free(composed);
+}
+
+
 static void fsevent_callback(ConstFSEventStreamRef streamRef,
 			     void *ctx,
 			     size_t num_of_events,
@@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 
 				if (!batch)
 					batch = fsmonitor_batch__new();
-				fsmonitor_batch__add_path(batch, rel);
+				my_add_path(batch, rel);
 			}
 
 			if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) {
@@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
 
 				if (!batch)
 					batch = fsmonitor_batch__new();
-				fsmonitor_batch__add_path(batch, tmp.buf);
+				my_add_path(batch, tmp.buf);
 			}
 
 			break;
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (24 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 25/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:22     ` Jeff Hostetler via GitGitGadget
  2022-03-22 18:23     ` [PATCH v3 27/27] t7527: test Unicode NFC/NFD handling on MacOS Jeff Hostetler via GitGitGadget
                       ` (2 subsequent siblings)
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:22 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker, Jeff Hostetler, Jeff Hostetler

From: Jeff Hostetler <jeffhost@microsoft.com>

Create a set of prereqs to help understand how file names
are handled by the filesystem when they contain NFC and NFD
Unicode characters.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 167 insertions(+)
 create mode 100755 t/lib-unicode-nfc-nfd.sh

diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh
new file mode 100755
index 00000000000..cf9c26d1e22
--- /dev/null
+++ b/t/lib-unicode-nfc-nfd.sh
@@ -0,0 +1,167 @@
+# Help detect how Unicode NFC and NFD are handled on the filesystem.
+
+# A simple character that has a NFD form.
+#
+# NFC:       U+00e9 LATIN SMALL LETTER E WITH ACUTE
+# UTF8(NFC): \xc3 \xa9
+#
+# NFD:       U+0065 LATIN SMALL LETTER E
+#            U+0301 COMBINING ACUTE ACCENT
+# UTF8(NFD): \x65  +  \xcc \x81
+#
+utf8_nfc=$(printf "\xc3\xa9")
+utf8_nfd=$(printf "\x65\xcc\x81")
+
+# Is the OS or the filesystem "Unicode composition sensitive"?
+#
+# That is, does the OS or the filesystem allow files to exist with
+# both the NFC and NFD spellings?  Or, does the OS/FS lie to us and
+# tell us that the NFC and NFD forms are equivalent.
+#
+# This is or may be independent of what type of filesystem we have,
+# since it might be handled by the OS at a layer above the FS.
+# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a
+# collision, for example.
+#
+# This does not tell us how the Unicode pathname will be spelled
+# on disk, but rather only that the two spelling "collide".  We
+# will examine the actual on disk spelling in a later prereq.
+#
+test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE '
+	mkdir trial_${utf8_nfc} &&
+	mkdir trial_${utf8_nfd}
+'
+
+# Is the spelling of an NFC pathname preserved on disk?
+#
+# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD
+# and on APFS, NFC paths are preserved.  As we have established
+# above, this is independent of "composition sensitivity".
+#
+# 0000000 63 5f c3 a9
+#
+# (/usr/bin/od output contains different amount of whitespace
+# on different platforms, so we need the wildcards here.)
+#
+test_lazy_prereq UNICODE_NFC_PRESERVED '
+	mkdir c_${utf8_nfc} &&
+	ls | od -t x1 | grep "63 *5f *c3 *a9"
+'
+
+# Is the spelling of an NFD pathname preserved on disk?
+#
+# 0000000 64 5f 65 cc 81
+#
+test_lazy_prereq UNICODE_NFD_PRESERVED '
+	mkdir d_${utf8_nfd} &&
+	ls | od -t x1 | grep "64 *5f *65 *cc *81"
+'
+	mkdir c_${utf8_nfc} &&
+	mkdir d_${utf8_nfd} &&
+
+# The following _DOUBLE_ forms are more for my curiosity,
+# but there may be quirks lurking when there are multiple
+# combining characters in non-canonical order.
+
+# Unicode also allows multiple combining characters
+# that can be decomposed in pieces.
+#
+# NFC:        U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI
+# UTF8(NFC):  \xe1 \xbd \xa7
+#
+# NFD1:       U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA
+#             U+0342 COMBINING GREEK PERISPOMENI
+# UTF8(NFD1): \xe1 \xbd \xa1  +  \xcd \x82
+#
+# But U+1f61 decomposes into
+# NFD2:       U+03c9 GREEK SMALL LETTER OMEGA
+#             U+0314 COMBINING REVERSED COMMA ABOVE
+# UTF8(NFD2): \xcf \x89  +  \xcc \x94
+#
+# Yielding:   \xcf \x89  +  \xcc \x94  +  \xcd \x82
+#
+# Note that I've used the canonical ordering of the
+# combinining characters.  It is also possible to
+# swap them.  My testing shows that that non-standard
+# ordering also causes a collision in mkdir.  However,
+# the resulting names don't draw correctly on the
+# terminal (implying that the on-disk format also has
+# them out of order).
+#
+greek_nfc=$(printf "\xe1\xbd\xa7")
+greek_nfd1=$(printf "\xe1\xbd\xa1\xcd\x82")
+greek_nfd2=$(printf "\xcf\x89\xcc\x94\xcd\x82")
+
+# See if a double decomposition also collides.
+#
+test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE '
+	mkdir trial_${greek_nfc} &&
+	mkdir trial_${greek_nfd2}
+'
+
+# See if the NFC spelling appears on the disk.
+#
+test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED '
+	mkdir c_${greek_nfc} &&
+	ls | od -t x1 | grep "63 *5f *e1 *bd *a7"
+'
+
+# See if the NFD spelling appears on the disk.
+#
+test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED '
+	mkdir d_${greek_nfd2} &&
+	ls | od -t x1 | grep "64 *5f *cf *89 *cc *94 *cd *82"
+'
+
+# The following is for debugging. I found it useful when
+# trying to understand the various (OS, FS) quirks WRT
+# Unicode and how composition/decomposition is handled.
+# For example, when trying to understand how (macOS, APFS)
+# and (macOS, HFS) and (macOS, FAT32) compare.
+#
+# It is rather noisy, so it is disabled by default.
+#
+if test "$unicode_debug" = "true"
+then
+	if test_have_prereq UNICODE_COMPOSITION_SENSITIVE
+	then
+		echo NFC and NFD are distinct on this OS/filesystem.
+	else
+		echo NFC and NFD are aliases on this OS/filesystem.
+	fi
+
+	if test_have_prereq UNICODE_NFC_PRESERVED
+	then
+		echo NFC maintains original spelling.
+	else
+		echo NFC is modified.
+	fi
+
+	if test_have_prereq UNICODE_NFD_PRESERVED
+	then
+		echo NFD maintains original spelling.
+	else
+		echo NFD is modified.
+	fi
+
+	if test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE
+	then
+		echo DOUBLE NFC and NFD are distinct on this OS/filesystem.
+	else
+		echo DOUBLE NFC and NFD are aliases on this OS/filesystem.
+	fi
+
+	if test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED
+	then
+		echo Double NFC maintains original spelling.
+	else
+		echo Double NFC is modified.
+	fi
+
+	if test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED
+	then
+		echo Double NFD maintains original spelling.
+	else
+		echo Double NFD is modified.
+	fi
+fi
-- 
gitgitgadget


^ permalink raw reply	[flat|nested] 277+ messages in thread

* [PATCH v3 27/27] t7527: test Unicode NFC/NFD handling on MacOS
  2022-03-22 18:22   ` [PATCH v3 " Jeff Hostetler via GitGitGadget
                       ` (25 preceding siblings ...)
  2022-03-22 18:22     ` [PATCH v3 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Jeff Hostetler via GitGitGadget
@ 2022-03-22 18:23     ` Jeff Hostetler via GitGitGadget
  2022-03-24 16:50     ` [PATCH v4 00/27] Builtin FSMonitor Part 3 Jeff Hostetler via GitGitGadget
  2022-03-28 14:37     ` [PATCH v3 00/27] " Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=
  28 siblings, 0 replies; 277+ messages in thread
From: Jeff Hostetler via GitGitGadget @ 2022-03-22 18:23 UTC (permalink / raw)
  To: git
  Cc: Jeff Hostetler, Derrick Stolee,
	Ævar Arnfjörð Bjarmason,
	Torsten =?unknown-8bit?Q?B=C3=B6gershausen?=,
	rsbecker